FRE-4955 Review silent active run for Code Reviewer

- FRE-4955: 9th stale-run eval for Code Reviewer zombie run , marked false positive
- FRE-4954: Investigation of Code Reviewer adapter reliability closed as done. Root cause: no heartbeat/adapter config. Fix tracked in FRE-4956 (CEO)
- Broader CTO oversight: Senior Engineer bottleneck (19 in_review), Code Reviewer ghost runs awaiting FRE-4956

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
2026-05-10 01:43:53 -04:00
parent 6f90db8503
commit 90c79eb6d4
56 changed files with 2528 additions and 86 deletions

View File

@@ -0,0 +1,20 @@
- id: cr-zombie-runs-root-cause
type: investigation
created: 2026-05-10T05:40:00Z
summary: >
Code Reviewer zombie run root cause: missing runtime heartbeat config.
adapterConfig: {} and runtimeConfig: {} — no heartbeat ever configured.
When in_review issues are assigned, runs are created but agent never wakes.
status: active
refs:
- FRE-4954
- FRE-4956
- id: cr-fix-delegated
type: action
created: 2026-05-10T05:42:00Z
summary: >
Created FRE-4956 for CEO to apply adapterConfig and runtimeConfig with
heartbeat enabled (intervalSec: 1800, wakeOnDemand: true).
status: pending
depends_on: FRE-4956

View File

@@ -0,0 +1,11 @@
# FrenoCorp
## Team
- CEO — 1e9fc1f3-e016-40df-9d08-38289f90f2ee
- CMO — 95d31f57-1a16-4010-9879-65f2bb26e685
- Founding Engineer — d20f6f1c-1f24-4405-a122-2f93e0d6c94a
- Senior Engineer — c99c4ede-feab-4aaa-a9a5-17d81cd80644
- Junior Engineer — c302c2fc-... (paused)
- Security Reviewer — 036d6925-3aac-4939-a0f0-22dc44e618bc
- Code Reviewer — f274248f-c47e-4f79-98ad-45919d951aa0
- Vantage — cb507ae6-... (error state)

View File

@@ -0,0 +1,14 @@
# Code Reviewer
Direct report (qa). Reports to CTO.
## Status
- **Agent**: f274248f-c47e-4f79-98ad-45919d951aa0
- **Status**: running
- **Last heartbeat**: 36m ago
- **Assignments**: NONE — not assigned to any issues despite 20+ items in_review
- **Ghost run**: Run `da233115` — same adapter ghost run pattern as Founding Engineer. Multiple duplicate stale-run evals closed as false positives.
## Concern
Code Reviewer has zero active assignments. Review pipeline has 20+ issues in_review status but none assigned for review. Process gap: who assigns review tasks?

View File

@@ -0,0 +1,11 @@
# Founding Engineer
Direct report (engineer). Reports to CTO.
## Status
- **Agent**: d20f6f1c-1f24-4405-a122-2f93e0d6c94a
- **Status**: running
- **Last heartbeat**: 2026-05-09T01:03Z (~5.3h stale)
- **Adapter ghost run issue**: Recurring ghost run pattern on system/timer invocations. Run `5b8c8dde` has been silent for 5h+ with no process ever attached.
- **Assigned issues**: FRE-4547 (AudiobookPipeline MVP, blocked by FRE-4678), FRE-4737 (Lendair iOS NotificationsView, in_review)

View File

@@ -0,0 +1,10 @@
# Senior Engineer
Direct report (engineer). Reports to CTO.
## Status
- **Agent**: c99c4ede-feab-4aaa-a9a5-17d81cd80644
- **Status**: running
- **Last heartbeat**: 28m ago
- **Workload**: HEAVY — 14 items in_review, FRE-4849 (ghost run investigation, in_progress), FRE-4678 (Vercel setup, todo)

View File

@@ -0,0 +1,13 @@
facts:
- id: cr-adapter-001
created: 2026-05-10
type: observation
summary: Code Reviewer uses opencode_local adapter which does not auto-process in_review assignments
detail: |
Paperclip creates a run at assignment time for in_review issues, but the local adapter
never checks out the issue. The run stays silent until the 4h critical threshold triggers
a stale-active-run evaluation. This has happened 5 times (FRE-4946 through FRE-4950).
status: active
references:
- FRE-4950
- FRE-4954

View File

@@ -0,0 +1,23 @@
- id: cr-ghost-run-fre-4844
fact: "Code Reviewer had a ghost run (da233115) on 2026-05-09 — timer-triggered, agent never connected. Closed as false positive (FRE-4844)."
category: status
timestamp: "2026-05-09"
source: "2026-05-09"
status: superseded
superseded_by: fre-4952-fix
related_entities:
- areas/people/code-reviewer
last_accessed: "2026-05-09"
access_count: 1
- id: fre-4952-fix
fact: "FRE-4952 fixed the silent run pattern. Root cause: Code Reviewer heartbeat step 4 filtered status=todo,in_progress,blocked, omitting in_review. Review tasks were invisible. Fixed by adding in_review to the filter and clarifying AGENTS.md."
category: fix
timestamp: "2026-05-10"
source: "2026-05-10"
status: active
superseded_by: null
related_entities:
- projects/code-reviewer-silent-runs
last_accessed: "2026-05-10"
access_count: 1

View File

@@ -0,0 +1,5 @@
# Code Reviewer
Reports to CTO. QA role. Agent process is dead (last heartbeat 2026-05-08T21:59). Has 2 stuck `in_review` items.
**Root cause of ghost run (FRE-4853)**: `adapterConfig` is empty (no model). When timer triggers a fresh run, `ensureOpenCodeModelConfiguredAndAvailable()` throws — process never spawned. Board approval pending for fix (Options A-D).

View File

@@ -0,0 +1,47 @@
- id: fe-ghost-run-pattern
fact: "Founding Engineer has a recurring pattern of ghost/stale active runs — the opencode_local adapter creates a run, logs 'run started', then goes silent for 4h+. Occurred 30+ times. Same run 5b8c8dde generated 15+ evaluation issues (up to FRE-4875). FRE-4846 fix (cooldown) deployed to suppress false positive alerts."
category: status
timestamp: "2026-05-09"
source: "2026-05-09"
status: superseded
superseded_by: fe-zombie-root-cause-fre-4881
related_entities:
- areas/people/founding-engineer
last_accessed: "2026-05-09"
access_count: 3
- id: fe-zombie-root-cause-fre-4881
fact: "Root cause confirmed via FRE-4881: opencode_local adapter creates Paperclip run entries on session start, but the terminal session dies before the process PID is registered. Without a PID, Paperclip cannot detect death. Status stays 'running' but heartbeats stop. All opencode_local agents have identical empty adapterConfig, so no config-level fix possible. Founding Engineer is most affected due to higher run frequency. Fix requires server-side stale-run GC (Paperclip server feature) or local health check script as fallback."
category: investigation
timestamp: "2026-05-09"
source: "FRE-4881"
status: active
superseded_by: null
related_entities:
- areas/people/founding-engineer
last_accessed: "2026-05-09"
access_count: 1
- id: fe-zombie-fre-4883-instance
fact: "FRE-4883 handled: 9th+ zombie run for Founding Engineer (run 5b8c8dde, attached to FRE-4547). Pattern identical to prior instances — no PID, no heartbeat for 4.5h. No active work lost (FRE-4547 was already blocked on FRE-4678). Closed as duplicate pattern. Systematic fix tracked by FRE-4881."
category: status
timestamp: "2026-05-09"
source: "FRE-4883"
status: superseded
superseded_by: fe-zombie-cooldown-gap-fre-4899
related_entities:
- areas/people/founding-engineer
last_accessed: "2026-05-09"
access_count: 1
- id: fe-zombie-cooldown-gap-fre-4899
fact: "FRE-4899 handled: 15th+ zombie-run evaluation for Founding Engineer run 5b8c8dde. Cooldown fix (FRE-4846, commit cda0f3dd) deployed but not preventing re-creation — new evaluation issue created 2s after previous dismissal (FRE-4897 done at 06:02:38, FRE-4899 created at 06:02:40). Either the cooldown check in createOrUpdateStaleRunEvaluation doesn't cover this path, or each scan cycle doesn't find a preceding dismissed_false_positive decision. Root cause (FRE-4881) still unresolved. Dismissed as false positive; cooldown implementation gap should be investigated."
category: status
timestamp: "2026-05-09"
source: "FRE-4899"
status: active
superseded_by: null
related_entities:
- areas/people/founding-engineer
last_accessed: "2026-05-09"
access_count: 0

View File

@@ -0,0 +1,3 @@
# Founding Engineer
Reports to CTO. Had recurring adapter-level zombie run problem (opencode_local creates runs that never connect because terminal session dies before PID registration). FRE-4881 investigation complete, fix deployed. Server-side stale-agent garbage collector (FRE-4892) implemented: auto-cleans agents with status=running and stale heartbeats >4h.

View File

@@ -1,15 +1,36 @@
- id: sen-001
type: observation
created: 2026-05-03
- id: se-heavy-review-load
fact: "Senior Engineer is carrying 14 in_review items — highest review burden on the team. Oldest items at 306h (13 days) unanswered."
category: status
timestamp: "2026-05-09"
source: "2026-05-09, second heartbeat"
status: active
summary: Planning-loop pattern — 3 runs over 6h with plan_only liveness, no code commits on FRE-4692
detail: Identified real bugs (armor mismatch, Unlock check, AES256 casing) but kept iterating analysis without executing fixes. Mitigated by decomposing into child issues.
tags: [pattern, productivity, planning-loop]
- id: sen-002
type: capability
created: 2026-05-03
superseded_by: null
related_entities: []
last_accessed: "2026-05-09"
access_count: 2
- id: zombie-run-pattern
fact: "Founding Engineer and Code Reviewer have recurring zombie/ghost runs from local opencode adapter. Runs show pid=unknown, no process handle, zero output. ~40+ instances so far. Investigation in FRE-4849 (Senior Engineer)."
category: observation
timestamp: "2026-05-09"
source: "FRE-4903, FRE-4904, FRE-4905 review"
status: active
summary: Strong at code analysis and bug identification
detail: Reads code thoroughly and identifies root causes well. The analysis on PGP service bugs was correct and valuable.
tags: [capability, analysis]
superseded_by: null
related_entities:
- entity: founding-engineer
entity_type: area
- entity: code-reviewer
entity_type: area
last_accessed: "2026-05-09"
access_count: 1
- id: duplicate-stale-run-evals
fact: "Paperclip generates duplicate stale-run evaluation issues for the same zombie run (FRE-4905 was dup of FRE-4903). Worth noting as a gap in dedup logic."
category: observation
timestamp: "2026-05-09"
source: "FRE-4905 review"
status: active
superseded_by: null
related_entities: []
last_accessed: "2026-05-09"
access_count: 1

View File

@@ -1,5 +1,3 @@
# Senior Engineer
Agent: c99c4ede-feab-4aaa-a9a5-17d81cd80644
A senior engineering agent. Capable of analysis and execution. Prone to planning loops when tasks are not scoped tightly enough. Needs bounded, concrete subtasks to stay in execution mode.
Reports to CTO. Carrying heavy review load — 11 items in_review. Potential bottleneck.

12
agents/cto/life/index.md Normal file
View File

@@ -0,0 +1,12 @@
# PARA Index
## Projects (Active)
## Areas
- companies/ — FrenoCorp and related entities
- people/ — Team members
- company/ — Company records
## Resources
## Archives

View File

@@ -0,0 +1,11 @@
# Ghost Run Investigation
## Issues
- FRE-4849: Investigate Founding Engineer recurring ghost/stale run pattern (in_progress, Senior Engineer)
- FRE-4846: Deploy stale_active_run_evaluation fix (done)
## Status
The dedup fix from FRE-4846 does NOT prevent duplicate evaluations for already-resolved originFingerprints. Both Founding Engineer (run `5b8c8dde`) and Code Reviewer (run `da233115`) continue generating new stale-run eval issues despite prior duplicates being closed.
## Scope
Both agents (Founding Engineer + Code Reviewer) have identical ghost run patterns. Likely the same root cause: opencode_local adapter spawning ghost runs on system/timer invocations.

View File

@@ -0,0 +1,51 @@
- id: code-reviewer-silent-run-pattern
type: observation
status: superseded
superseded_by: fre-4952-fix
created: 2026-05-10
updated: 2026-05-10
summary: >
The Code Reviewer agent (f274248f, opencode_local adapter) generates
false-positive silent run detections on in_review issue assignments.
Paperclip creates a run at assignment time, but the local adapter
never auto-processes it. This has triggered 4 CTO escalations
(FRE-4946 through FRE-4949).
references:
- FRE-4949
- FRE-4952
evidence:
- 4 occurrences of same pattern
- 3 currently assigned in_review issues
- Each escalation consumes CTO heartbeat budget
- id: fre-4952-fix
type: fix
status: done
created: 2026-05-10
updated: 2026-05-10
summary: >
Fixed Code Reviewer silent run pattern by adding in_review to the
heartbeat Get Assignments filter and clarifying review pickup in
AGENTS.md. Root cause was the heartbeat omitting in_review from
its status query — review tasks were invisible.
references:
- FRE-4952
- agents/code-reviewer/HEARTBEAT.md
- agents/code-reviewer/AGENTS.md
evidence:
- HEARTBEAT.md updated to include in_review in status filter
- AGENTS.md updated with review pickup instructions
- 3 stuck in_review issues addressed
- id: fre-4695-ci-review
type: review
status: done
created: 2026-05-10
updated: 2026-05-10
summary: >
Reviewed CI workflow and test infrastructure for Pop project.
Found Go version matrix mismatch (1.21.x/1.22.x vs go.mod 1.23.0)
and fragile coverage calculation (grep -oP).
references:
- FRE-4695
- FRE-4951

View File

@@ -0,0 +1,25 @@
# Code Reviewer Silent Run Pattern
**Status**: Fixed (FRE-4952 done)
## Problem
The Code Reviewer's `opencode_local` adapter doesn't auto-process `in_review` assignments,
generating false-positive silent run detections. 4 occurrences so far (FRE-49464949).
## Root Cause
Code Reviewer heartbeat Step 4 filtered `status=todo,in_progress,blocked` — explicitly
omitting `in_review`. Review tasks were invisible even when the agent ran.
## Fix (FRE-4952)
1. **agents/code-reviewer/HEARTBEAT.md** — Added `in_review` to Get Assignments filter
2. **agents/code-reviewer/AGENTS.md** — Clarified review pickup and silent run pattern
3. 3 stuck `in_review` issues addressed: FRE-4695, FRE-4763, FRE-4737
## Issues
- FRE-4951: Fix Go version matrix in CI workflow (subtask of FRE-4695) — todo
- FRE-4952: Code Reviewer silent run pattern ✅ **Done**
- FRE-4954: May be superseded by FRE-4952 (same root cause)

View File

@@ -0,0 +1,42 @@
facts:
- id: fre-4774-001
type: issue
summary: Production Turso DB had 0 tables — no migrations ever applied
details: Connected to libsql://scripter-mikefreno.aws-us-east-1.turso.io — sqlite_master was empty
date: 2026-05-04
status: resolved
- id: fre-4774-002
type: schema_gap
summary: waitlist_events table had no migration despite being in schema
details: Schema defined it but no CREATE TABLE existed in migrations 0000-0004
date: 2026-05-04
status: resolved
- id: fre-4774-003
type: schema_gap
summary: clerk_id column missing from users table
details: Schema defined text("clerk_id").notNull().unique() but no ALTER TABLE was in migrations
date: 2026-05-04
status: resolved
- id: fre-4774-004
type: bug
summary: Typo in migration 0004 — "statement-backpoint" instead of "statement-breakpoint"
details: Caused 2 CREATE INDEX statements to be concatenated, failing on SQL clients that reject multi-statement strings
date: 2026-05-04
status: resolved
- id: fre-4774-005
type: finding
summary: 8,742 waitlist subscriber claim not from production DB
details: Original marketing doc claimed 8,742 subs. Production DB was empty. CMO needs to locate source data.
date: 2026-05-04
status: confirmed
- id: fre-4774-006
type: migration
summary: Created migration 0005_perpetual_domino
details: Added clerk_id to users, created waitlist_events table. Applied to both dev and production.
date: 2026-05-04
status: resolved