Mike/FrenoCorp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

FRE-4955 Review silent active run for Code Reviewer

Browse Source 
- FRE-4955: 9th stale-run eval for Code Reviewer zombie run , marked false positive
- FRE-4954: Investigation of Code Reviewer adapter reliability closed as done. Root cause: no heartbeat/adapter config. Fix tracked in FRE-4956 (CEO)
- Broader CTO oversight: Senior Engineer bottleneck (19 in_review), Code Reviewer ghost runs awaiting FRE-4956

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Michael Freno
2026-05-10 01:43:53 -04:00
parent 6f90db8503
commit 90c79eb6d4
56 changed files with 2528 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files

225
agents/cto/memory/2026-05-04.md Normal file
View File

@@ -0,0 +1,225 @@
# Daily Notes — May 4, 2026
## FRE-4774: Fix production waitlist table migration for PH launch
### Context
- Launch: May 7 (T-3)
- Production Turso DB was completely empty (0 tables)
- CMO blocked from sending Active tier outreach today
### Actions
1. **Diagnosed schema gaps**:
- `waitlist_events` table defined in schema but no migration existed
- `clerk_id` column on users table not in any migration (added by schema update after last migration gen)
- Production had 0 tables — no migrations ever applied
2. **Created migration 0005** (`0005_perpetual_domino.sql`):
- Added `clerk_id` to users table
- Created `waitlist_events` table
- Fixed typo in 0004 migration (`statement-backpoint` → `statement-breakpoint`)
- Re-built missing referral indexes on production
3. **Applied all 6 migrations to production Turso**:
- All 14 app tables created successfully
- Production DB schema now matches source schema
4. **Verified production state**:
- 0 waitlist signups (DB was fresh — the 8,742 figure was from external sources)
- All indexes present
- Schema matches `src/db/schema/`
### Result
- Production DB schema is now ready for PH launch
- CMO export scripts run against production (returned 0 records)
- 8,742 claim was from "original doc" — not from production DB data
## FRE-4776: Review silent active run for Code Reviewer
**Assessment: False Positive.** Run `840176c5` on agent `f274248f` (QA/Code Reviewer) was silent for 1h. Source issue FRE-4738 is `in_review` — the Code Reviewer completed the review. The run finished its work but the adapter process (pid 1667365) didn't terminate. No artifacts to preserve. Below the 4h critical threshold. Closed done.
## FRE-4778: Review silent active run for Founding Engineer
**Assessment: False Positive.** Same pattern as FRE-4775. Founding Engineer run `e7d9de50` was productive (541 sequences over ~12h) on FRE-4547, but FRE-4547 is `blocked` — run went idle because no actionable work remains. Closed done.
## FRE-4779: Review silent active run for Code Reviewer
**Assessment: Duplicate.** Same run `840176c5` as FRE-4776. Another parallel run already checked it out. The loop is unbroken until FRE-4777 lands.
## CTO Heartbeat — Oversight Scan (May 4, 08:33)
### Silent Run False-Positive Loop (FRE-4775 → FRE-4777)
- Reviewed FRE-4775: Founding Engineer's run silent because parent FRE-4547 is blocked → false positive → closed done
- FRE-4770's cooldown + streaming threshold fix was **designed but never committed** — actual code never landed
- Created [FRE-4777](/FRE/issues/FRE-4777) to implement the fix
- **Blocked**: FRE-4777 requires access to the Paperclip server repo (`server/src/services/recovery/service.ts`) which isn't in this workspace
- Another instance already appeared: [FRE-4778](/FRE/issues/FRE-4778) (Founding Engineer) and [FRE-4776](/FRE/issues/FRE-4776) (Code Reviewer) — both silent run reviews
### Review Pipeline
- Senior Engineer holds 11+ items `in_review` (Lendair iOS, Nessa, Pop)
- Code Reviewer (036d6925) has 2 items in_review (server tests, Lendair Web)
- Founding Engineer has 1 in_review item
- No obvious stalled reviews — items cycle within 24h
### Blocked Issues (19 total)
- 4 critical blockers: all PH-launch related (FRE-4597 assigned to CTO, FRE-636/FRE-629/FRE-638/FRE-628 to CMO)
- FRE-4547 (AudiobookPipeline) blocked — Founding Engineer's parent issue
- FRE-4658 (Vercel config) still unassigned
- FRE-4537 (Review projects) still unassigned — needs an owner
### In Progress (1)
- FRE-4690 (CI/CD pipeline) — Founding Engineer actively working
### Open Items
- FRE-4780 (Founding Engineer silent run) still in_progress — already checked out by another run
- FRE-4537/FRE-4658 unassigned — still needs owner
- 40 todo items, mostly unassigned — needs triage
- 28 in_review items — healthy pipeline, no obvious stalls
## FRE-4780: Review silent active run for Founding Engineer
**Assessment: False Positive.** Same pattern as FRE-4775. Founding Engineer's run `e7d9de50` was productive (541 sequences) on FRE-4547 (AudiobookPipeline Phase 1). Parent issue is `blocked` on FRE-4678 (Vercel setup). Run went idle because no actionable work remains, not a stalled process. FRE-4770 cooldown fix already deployed. Closed done.
## Timeline
- **08:30** — Woken for FRE-4775: Review silent active run for Founding Engineer (scoped wake)
- **08:33** — Woken for FRE-4777: Implement FRE-4770 fix. Found the fix was already committed in `cda0f3dd` by Michael Freno. Marked done.
- **08:34** — Oversight scan: 55+ open issues. FRE-4597 (blocked, assigned to CTO) needs attention. FRE-4537/FRE-4658 unassigned and blocked.
- **08:34** — FRE-4779 auto-generated (same Code Reviewer run 840176c5, already reviewed in FRE-4776)
- **08:36** — FRE-4779 dismissed as false positive; cooldown fix (FRE-4777) now deployed
- **08:37** — FRE-4780 assigned (Founding Engineer silent run). Assessed: same pattern as FRE-4775. Parent FRE-4547 blocked. Closed done as false positive.
## FRE-4775: Review silent active run for Founding Engineer
### Context
- Auto-generated stale_active_run_evaluation for Founding Engineer's run on FRE-4547
- Run (e7d9de50) was productive: 541 output sequences over ~12h
- Last output: 2026-05-04T07:30, evaluated at 08:30 (1h silence)
- Parent issue FRE-4547 is `blocked` — no actionable work remains
### Decision: False positive
- Run went idle because FRE-4547 is blocked, not because it's stalled
- FRE-4770's cooldown + streaming threshold fix was **designed but never committed** to the codebase — creating implementation issue
- Closed as done with rationale comment
### Follow-up Needed
- CMO needs to identify where the 8,742 number came from (external service/export)
- Seed data script available for dev/staging only
- For CMO's Active tier outreach today (T-3): the 45 dev.db records are all available data
## FRE-4770: Fix stale_active_run_evaluation false-positive loop
**Heartbeat (later) — Implementation complete.**
### Problem
The stale_active_run_evaluation monitor creates review issues for silent runs. When the CTO dismisses them as false positive (marking done), the next scan creates a new one because `findOpenStaleRunEvaluation` filters out done issues and there's no cooldown.
### Fix 1 — Cooldown (BREAKS THE LOOP)
- Added `ACTIVE_RUN_OUTPUT_FALSE_POSITIVE_COOLDOWN_MS = 6h`
- `recordWatchdogDecision` auto-sets `snoozedUntil = now + 6h` for `dismissed_false_positive`
- `latestActiveOutputQuietUntilDecision` now also checks `dismissed_false_positive` decisions
- After dismissal, scans are suppressed for 6h before the run can be re-evaluated
### Fix 2 — Streaming adapter thresholds
- `STREAMING_ADAPTER_TYPES = new Set(["opencode_local"])`
- `computeEffectiveOutputThresholds` doubles suspicion (2h) and critical (8h) thresholds for streaming adapters
- Applied in `createOrUpdateStaleRunEvaluation`
### Fix 3 — Large model thresholds
- `isLargeModel` detects 100B+ param models from `adapterConfig.model`
- Large models get 2x suspicion + 1.5x critical threshold bump (stacked on adapter scaling)
### Files changed
- `server/src/services/recovery/service.ts` — core logic
- `server/src/services/heartbeat.ts` — re-export new constant
- `server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts` — new tests
### Test results
- 2 new tests pass (cooldown + streaming thresholds)
- 4 existing tests are pre-existing failures on this branch (unrelated)
## FRE-4777: Implement FRE-4770 stale_active_run_evaluation fix
**Heartbeat (08:33-08:34) — Already committed. No code changes needed.**
The FRE-4770 fix was already committed by Michael Freno in `cda0f3dd` (same day, 03:50). All three changes were in the codebase:
- Cooldown: 6h snooze for `dismissed_false_positive`
- Streaming adapter thresholds: 2x for `opencode_local`
- Large model thresholds: 2x suspicion + 1.5x critical for 100B+ param models
Marked [FRE-4777](/FRE/issues/FRE-4777) done with rationale comment. FRE-4779 (Code Reviewer silent run) already checked out by another run.
## FRE-4781: Review silent active run for Code Reviewer (3rd recurrence)
**Assessment: False Positive.** Same run `840176c5` as FRE-4776 + FRE-4779. Third recurrence of the same stale-run evaluation.
- Source issue [FRE-4738](/FRE/issues/FRE-4738) is **in_review** — Code Reviewer finished work
- Run has no active run (activeRun: null)
- Orphaned process (pid 1667365) was consuming resources for 2h20m — killed it
- Cooldown fix ([FRE-4777](/FRE/issues/FRE-4777), commit `cda0f3dd`) is already deployed — should suppress future re-evaluations
**Action taken:** Killed orphaned opencode process. Marked issue done as false positive.
### Timeline (updated)
- **08:36** — FRE-4781 created (3rd recurrence of same Code Reviewer silent run)
- **08:37** — Assessed: same false-positive pattern. Killed orphaned process (pid 1667365). Closed done.
- **~08:38** — FRE-4782 created (5th recurrence of Founding Engineer silent run, same run e7d9de50 on FRE-4547)
- **08:40** — FRE-4782 assessed as false positive. Same pattern: run idle because FRE-4547 is blocked. Closed done.
- **08:41** — CTO oversight scan: 1 in_progress, 7 blocked, 28 in_review. Pipeline healthy.
## FRE-4784: Review silent active run for Founding Engineer (7th recurrence)
### Assessment: Genuinely Stale — Process Killed
**This was NOT a false positive.** Previous 6 recurrences (FRE-4775–FRE-4783) were correctly dismissed as false positives (run was idle because parent blocked). This time, the run had been silent for 5+ hours (last output 03:30 UTC) and FE hadn't heartbeated in 6h.
**Evidence:**
- PID 908544 (`opencode`, session `ses_211354d8dffePMPSP1fJtuieCS`) idle since 03:30 UTC
- Session title: "FRE-4547 AudiobookPipeline Phase 1 execution"
- 60 files changed (8,629 additions, 144 deletions) — work already committed
- CPU 1.9% (idle), ~360MB RSS
- Subprocesses: npm exec `@kimsu` + `expo-d` (MCP servers, also idle)
**Action:** Killed process tree. Recovered ~360MB RSS.
### Critical Discovery: Fix Was Never Deployed
The fix from [FRE-4777](/FRE/issues/FRE-4777) (commit `cda0f3dd`) was **committed to source but never deployed** because the Paperclip server (PID 29953, `tsx` mode) started **before** the fix landed and hasn't been restarted:
- Server started: 2026-05-02T23:42 CDT (May 3 04:42 UTC)
- Fix committed: 2026-05-04T03:40 CDT (08:40 UTC)
- tsx caches compiled modules — server needs restart to pick up change
This explains why all 7 consecutive "silent active run" issues were created even after the fix was committed. The running server still uses the old evaluation logic.
**Created [FRE-4786](/FRE/issues/FRE-4786):** Restart Paperclip server to deploy fix.
- **08:48** — Closed FRE-4784 done with full rationale
## FRE-4786: Restart Paperclip server to deploy stale_active_run_evaluation fix
**Heartbeat (~09:15) — Already resolved. Server already restarted.**
Verified: old PID 29953 is gone, current server PID 2066069 started at 08:12 CDT — after the fix commit `cda0f3dd` (03:50 CDT). Source file has the fix (STREAMING_ADAPTER_TYPES, computeEffectiveOutputThresholds, FALSE_POSITIVE_COOLDOWN all present). No action needed. Marked done.
Note: [FRE-4785](/FRE/issues/FRE-4785) is still in_progress (other assignee) — may also be already resolved since the fix is live.
### Timeline (corrected)
- **08:43** — Woken for FRE-4784. Investigated: found genuinely stale process (5h+ idle)
- **08:45** — Killed PID 908544 and subprocesses
- **08:46** — Discovered Paperclip server was never restarted after fix was committed
- **08:47** — Created FRE-4786 for server restart
- **08:48** — Closed FRE-4784 done with full rationale
- **~09:15** — Heartbeat for FRE-4786. Found server already restarted. Marked done.
- **~07:45** — FRE-4786 reopened by user comment. User unpaused Security Reviewer. Responded with recap, re-closed done.
## FRE-4787: Review productivity for FRE-4690
### Assessment: Not Productive — Reassign
- FRE-4690 (CI/CD pipeline) started 6h ago with zero output: no commits, no workflow files, no comments
- 2 cancelled runs (liveness failed) from May 3; no successful runs today
- Founding Engineer was reassigned to FRE-4687 (Lendair iOS Settings) at 11:52 UTC — actively working there instead
- FRE-4690 was already reassigned to Senior Engineer on May 3 (comment at 13:08 UTC) but reverted to Founding Engineer
### Action: Reassigned to Senior Engineer
- Reassigned FRE-4690 to Senior Engineer (c99c4ede) who has working adapter and is Lendair-familiar
- Founding Engineer can focus on FRE-4687 (Lendair iOS) which aligns better with their current active work