FRE-4955 Review silent active run for Code Reviewer
- FRE-4955: 9th stale-run eval for Code Reviewer zombie run , marked false positive - FRE-4954: Investigation of Code Reviewer adapter reliability closed as done. Root cause: no heartbeat/adapter config. Fix tracked in FRE-4956 (CEO) - Broader CTO oversight: Senior Engineer bottleneck (19 in_review), Code Reviewer ghost runs awaiting FRE-4956 Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
20
agents/cto/life/areas/agents/code-reviewer/items.yaml
Normal file
20
agents/cto/life/areas/agents/code-reviewer/items.yaml
Normal file
@@ -0,0 +1,20 @@
|
||||
- id: cr-zombie-runs-root-cause
|
||||
type: investigation
|
||||
created: 2026-05-10T05:40:00Z
|
||||
summary: >
|
||||
Code Reviewer zombie run root cause: missing runtime heartbeat config.
|
||||
adapterConfig: {} and runtimeConfig: {} — no heartbeat ever configured.
|
||||
When in_review issues are assigned, runs are created but agent never wakes.
|
||||
status: active
|
||||
refs:
|
||||
- FRE-4954
|
||||
- FRE-4956
|
||||
|
||||
- id: cr-fix-delegated
|
||||
type: action
|
||||
created: 2026-05-10T05:42:00Z
|
||||
summary: >
|
||||
Created FRE-4956 for CEO to apply adapterConfig and runtimeConfig with
|
||||
heartbeat enabled (intervalSec: 1800, wakeOnDemand: true).
|
||||
status: pending
|
||||
depends_on: FRE-4956
|
||||
11
agents/cto/life/areas/companies/FrenoCorp.md
Normal file
11
agents/cto/life/areas/companies/FrenoCorp.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# FrenoCorp
|
||||
|
||||
## Team
|
||||
- CEO — 1e9fc1f3-e016-40df-9d08-38289f90f2ee
|
||||
- CMO — 95d31f57-1a16-4010-9879-65f2bb26e685
|
||||
- Founding Engineer — d20f6f1c-1f24-4405-a122-2f93e0d6c94a
|
||||
- Senior Engineer — c99c4ede-feab-4aaa-a9a5-17d81cd80644
|
||||
- Junior Engineer — c302c2fc-... (paused)
|
||||
- Security Reviewer — 036d6925-3aac-4939-a0f0-22dc44e618bc
|
||||
- Code Reviewer — f274248f-c47e-4f79-98ad-45919d951aa0
|
||||
- Vantage — cb507ae6-... (error state)
|
||||
14
agents/cto/life/areas/people/Code%20Reviewer/summary.md
Normal file
14
agents/cto/life/areas/people/Code%20Reviewer/summary.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Code Reviewer
|
||||
|
||||
Direct report (qa). Reports to CTO.
|
||||
|
||||
## Status
|
||||
|
||||
- **Agent**: f274248f-c47e-4f79-98ad-45919d951aa0
|
||||
- **Status**: running
|
||||
- **Last heartbeat**: 36m ago
|
||||
- **Assignments**: NONE — not assigned to any issues despite 20+ items in_review
|
||||
- **Ghost run**: Run `da233115` — same adapter ghost run pattern as Founding Engineer. Multiple duplicate stale-run evals closed as false positives.
|
||||
|
||||
## Concern
|
||||
Code Reviewer has zero active assignments. Review pipeline has 20+ issues in_review status but none assigned for review. Process gap: who assigns review tasks?
|
||||
11
agents/cto/life/areas/people/Founding%20Engineer/summary.md
Normal file
11
agents/cto/life/areas/people/Founding%20Engineer/summary.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# Founding Engineer
|
||||
|
||||
Direct report (engineer). Reports to CTO.
|
||||
|
||||
## Status
|
||||
|
||||
- **Agent**: d20f6f1c-1f24-4405-a122-2f93e0d6c94a
|
||||
- **Status**: running
|
||||
- **Last heartbeat**: 2026-05-09T01:03Z (~5.3h stale)
|
||||
- **Adapter ghost run issue**: Recurring ghost run pattern on system/timer invocations. Run `5b8c8dde` has been silent for 5h+ with no process ever attached.
|
||||
- **Assigned issues**: FRE-4547 (AudiobookPipeline MVP, blocked by FRE-4678), FRE-4737 (Lendair iOS NotificationsView, in_review)
|
||||
10
agents/cto/life/areas/people/Senior%20Engineer/summary.md
Normal file
10
agents/cto/life/areas/people/Senior%20Engineer/summary.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# Senior Engineer
|
||||
|
||||
Direct report (engineer). Reports to CTO.
|
||||
|
||||
## Status
|
||||
|
||||
- **Agent**: c99c4ede-feab-4aaa-a9a5-17d81cd80644
|
||||
- **Status**: running
|
||||
- **Last heartbeat**: 28m ago
|
||||
- **Workload**: HEAVY — 14 items in_review, FRE-4849 (ghost run investigation, in_progress), FRE-4678 (Vercel setup, todo)
|
||||
13
agents/cto/life/areas/people/code-reviewer.yaml
Normal file
13
agents/cto/life/areas/people/code-reviewer.yaml
Normal file
@@ -0,0 +1,13 @@
|
||||
facts:
|
||||
- id: cr-adapter-001
|
||||
created: 2026-05-10
|
||||
type: observation
|
||||
summary: Code Reviewer uses opencode_local adapter which does not auto-process in_review assignments
|
||||
detail: |
|
||||
Paperclip creates a run at assignment time for in_review issues, but the local adapter
|
||||
never checks out the issue. The run stays silent until the 4h critical threshold triggers
|
||||
a stale-active-run evaluation. This has happened 5 times (FRE-4946 through FRE-4950).
|
||||
status: active
|
||||
references:
|
||||
- FRE-4950
|
||||
- FRE-4954
|
||||
23
agents/cto/life/areas/people/code-reviewer/items.yaml
Normal file
23
agents/cto/life/areas/people/code-reviewer/items.yaml
Normal file
@@ -0,0 +1,23 @@
|
||||
- id: cr-ghost-run-fre-4844
|
||||
fact: "Code Reviewer had a ghost run (da233115) on 2026-05-09 — timer-triggered, agent never connected. Closed as false positive (FRE-4844)."
|
||||
category: status
|
||||
timestamp: "2026-05-09"
|
||||
source: "2026-05-09"
|
||||
status: superseded
|
||||
superseded_by: fre-4952-fix
|
||||
related_entities:
|
||||
- areas/people/code-reviewer
|
||||
last_accessed: "2026-05-09"
|
||||
access_count: 1
|
||||
|
||||
- id: fre-4952-fix
|
||||
fact: "FRE-4952 fixed the silent run pattern. Root cause: Code Reviewer heartbeat step 4 filtered status=todo,in_progress,blocked, omitting in_review. Review tasks were invisible. Fixed by adding in_review to the filter and clarifying AGENTS.md."
|
||||
category: fix
|
||||
timestamp: "2026-05-10"
|
||||
source: "2026-05-10"
|
||||
status: active
|
||||
superseded_by: null
|
||||
related_entities:
|
||||
- projects/code-reviewer-silent-runs
|
||||
last_accessed: "2026-05-10"
|
||||
access_count: 1
|
||||
5
agents/cto/life/areas/people/code-reviewer/summary.md
Normal file
5
agents/cto/life/areas/people/code-reviewer/summary.md
Normal file
@@ -0,0 +1,5 @@
|
||||
# Code Reviewer
|
||||
|
||||
Reports to CTO. QA role. Agent process is dead (last heartbeat 2026-05-08T21:59). Has 2 stuck `in_review` items.
|
||||
|
||||
**Root cause of ghost run (FRE-4853)**: `adapterConfig` is empty (no model). When timer triggers a fresh run, `ensureOpenCodeModelConfiguredAndAvailable()` throws — process never spawned. Board approval pending for fix (Options A-D).
|
||||
47
agents/cto/life/areas/people/founding-engineer/items.yaml
Normal file
47
agents/cto/life/areas/people/founding-engineer/items.yaml
Normal file
@@ -0,0 +1,47 @@
|
||||
- id: fe-ghost-run-pattern
|
||||
fact: "Founding Engineer has a recurring pattern of ghost/stale active runs — the opencode_local adapter creates a run, logs 'run started', then goes silent for 4h+. Occurred 30+ times. Same run 5b8c8dde generated 15+ evaluation issues (up to FRE-4875). FRE-4846 fix (cooldown) deployed to suppress false positive alerts."
|
||||
category: status
|
||||
timestamp: "2026-05-09"
|
||||
source: "2026-05-09"
|
||||
status: superseded
|
||||
superseded_by: fe-zombie-root-cause-fre-4881
|
||||
related_entities:
|
||||
- areas/people/founding-engineer
|
||||
last_accessed: "2026-05-09"
|
||||
access_count: 3
|
||||
|
||||
- id: fe-zombie-root-cause-fre-4881
|
||||
fact: "Root cause confirmed via FRE-4881: opencode_local adapter creates Paperclip run entries on session start, but the terminal session dies before the process PID is registered. Without a PID, Paperclip cannot detect death. Status stays 'running' but heartbeats stop. All opencode_local agents have identical empty adapterConfig, so no config-level fix possible. Founding Engineer is most affected due to higher run frequency. Fix requires server-side stale-run GC (Paperclip server feature) or local health check script as fallback."
|
||||
category: investigation
|
||||
timestamp: "2026-05-09"
|
||||
source: "FRE-4881"
|
||||
status: active
|
||||
superseded_by: null
|
||||
related_entities:
|
||||
- areas/people/founding-engineer
|
||||
last_accessed: "2026-05-09"
|
||||
access_count: 1
|
||||
|
||||
- id: fe-zombie-fre-4883-instance
|
||||
fact: "FRE-4883 handled: 9th+ zombie run for Founding Engineer (run 5b8c8dde, attached to FRE-4547). Pattern identical to prior instances — no PID, no heartbeat for 4.5h. No active work lost (FRE-4547 was already blocked on FRE-4678). Closed as duplicate pattern. Systematic fix tracked by FRE-4881."
|
||||
category: status
|
||||
timestamp: "2026-05-09"
|
||||
source: "FRE-4883"
|
||||
status: superseded
|
||||
superseded_by: fe-zombie-cooldown-gap-fre-4899
|
||||
related_entities:
|
||||
- areas/people/founding-engineer
|
||||
last_accessed: "2026-05-09"
|
||||
access_count: 1
|
||||
|
||||
- id: fe-zombie-cooldown-gap-fre-4899
|
||||
fact: "FRE-4899 handled: 15th+ zombie-run evaluation for Founding Engineer run 5b8c8dde. Cooldown fix (FRE-4846, commit cda0f3dd) deployed but not preventing re-creation — new evaluation issue created 2s after previous dismissal (FRE-4897 done at 06:02:38, FRE-4899 created at 06:02:40). Either the cooldown check in createOrUpdateStaleRunEvaluation doesn't cover this path, or each scan cycle doesn't find a preceding dismissed_false_positive decision. Root cause (FRE-4881) still unresolved. Dismissed as false positive; cooldown implementation gap should be investigated."
|
||||
category: status
|
||||
timestamp: "2026-05-09"
|
||||
source: "FRE-4899"
|
||||
status: active
|
||||
superseded_by: null
|
||||
related_entities:
|
||||
- areas/people/founding-engineer
|
||||
last_accessed: "2026-05-09"
|
||||
access_count: 0
|
||||
@@ -0,0 +1,3 @@
|
||||
# Founding Engineer
|
||||
|
||||
Reports to CTO. Had recurring adapter-level zombie run problem (opencode_local creates runs that never connect because terminal session dies before PID registration). FRE-4881 investigation complete, fix deployed. Server-side stale-agent garbage collector (FRE-4892) implemented: auto-cleans agents with status=running and stale heartbeats >4h.
|
||||
@@ -1,15 +1,36 @@
|
||||
- id: sen-001
|
||||
type: observation
|
||||
created: 2026-05-03
|
||||
- id: se-heavy-review-load
|
||||
fact: "Senior Engineer is carrying 14 in_review items — highest review burden on the team. Oldest items at 306h (13 days) unanswered."
|
||||
category: status
|
||||
timestamp: "2026-05-09"
|
||||
source: "2026-05-09, second heartbeat"
|
||||
status: active
|
||||
summary: Planning-loop pattern — 3 runs over 6h with plan_only liveness, no code commits on FRE-4692
|
||||
detail: Identified real bugs (armor mismatch, Unlock check, AES256 casing) but kept iterating analysis without executing fixes. Mitigated by decomposing into child issues.
|
||||
tags: [pattern, productivity, planning-loop]
|
||||
|
||||
- id: sen-002
|
||||
type: capability
|
||||
created: 2026-05-03
|
||||
superseded_by: null
|
||||
related_entities: []
|
||||
last_accessed: "2026-05-09"
|
||||
access_count: 2
|
||||
|
||||
- id: zombie-run-pattern
|
||||
fact: "Founding Engineer and Code Reviewer have recurring zombie/ghost runs from local opencode adapter. Runs show pid=unknown, no process handle, zero output. ~40+ instances so far. Investigation in FRE-4849 (Senior Engineer)."
|
||||
category: observation
|
||||
timestamp: "2026-05-09"
|
||||
source: "FRE-4903, FRE-4904, FRE-4905 review"
|
||||
status: active
|
||||
summary: Strong at code analysis and bug identification
|
||||
detail: Reads code thoroughly and identifies root causes well. The analysis on PGP service bugs was correct and valuable.
|
||||
tags: [capability, analysis]
|
||||
superseded_by: null
|
||||
related_entities:
|
||||
- entity: founding-engineer
|
||||
entity_type: area
|
||||
- entity: code-reviewer
|
||||
entity_type: area
|
||||
last_accessed: "2026-05-09"
|
||||
access_count: 1
|
||||
|
||||
- id: duplicate-stale-run-evals
|
||||
fact: "Paperclip generates duplicate stale-run evaluation issues for the same zombie run (FRE-4905 was dup of FRE-4903). Worth noting as a gap in dedup logic."
|
||||
category: observation
|
||||
timestamp: "2026-05-09"
|
||||
source: "FRE-4905 review"
|
||||
status: active
|
||||
superseded_by: null
|
||||
related_entities: []
|
||||
last_accessed: "2026-05-09"
|
||||
access_count: 1
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
# Senior Engineer
|
||||
|
||||
Agent: c99c4ede-feab-4aaa-a9a5-17d81cd80644
|
||||
|
||||
A senior engineering agent. Capable of analysis and execution. Prone to planning loops when tasks are not scoped tightly enough. Needs bounded, concrete subtasks to stay in execution mode.
|
||||
Reports to CTO. Carrying heavy review load — 11 items in_review. Potential bottleneck.
|
||||
|
||||
12
agents/cto/life/index.md
Normal file
12
agents/cto/life/index.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# PARA Index
|
||||
|
||||
## Projects (Active)
|
||||
|
||||
## Areas
|
||||
- companies/ — FrenoCorp and related entities
|
||||
- people/ — Team members
|
||||
- company/ — Company records
|
||||
|
||||
## Resources
|
||||
|
||||
## Archives
|
||||
@@ -0,0 +1,11 @@
|
||||
# Ghost Run Investigation
|
||||
|
||||
## Issues
|
||||
- FRE-4849: Investigate Founding Engineer recurring ghost/stale run pattern (in_progress, Senior Engineer)
|
||||
- FRE-4846: Deploy stale_active_run_evaluation fix (done)
|
||||
|
||||
## Status
|
||||
The dedup fix from FRE-4846 does NOT prevent duplicate evaluations for already-resolved originFingerprints. Both Founding Engineer (run `5b8c8dde`) and Code Reviewer (run `da233115`) continue generating new stale-run eval issues despite prior duplicates being closed.
|
||||
|
||||
## Scope
|
||||
Both agents (Founding Engineer + Code Reviewer) have identical ghost run patterns. Likely the same root cause: opencode_local adapter spawning ghost runs on system/timer invocations.
|
||||
@@ -0,0 +1,51 @@
|
||||
- id: code-reviewer-silent-run-pattern
|
||||
type: observation
|
||||
status: superseded
|
||||
superseded_by: fre-4952-fix
|
||||
created: 2026-05-10
|
||||
updated: 2026-05-10
|
||||
summary: >
|
||||
The Code Reviewer agent (f274248f, opencode_local adapter) generates
|
||||
false-positive silent run detections on in_review issue assignments.
|
||||
Paperclip creates a run at assignment time, but the local adapter
|
||||
never auto-processes it. This has triggered 4 CTO escalations
|
||||
(FRE-4946 through FRE-4949).
|
||||
references:
|
||||
- FRE-4949
|
||||
- FRE-4952
|
||||
evidence:
|
||||
- 4 occurrences of same pattern
|
||||
- 3 currently assigned in_review issues
|
||||
- Each escalation consumes CTO heartbeat budget
|
||||
|
||||
- id: fre-4952-fix
|
||||
type: fix
|
||||
status: done
|
||||
created: 2026-05-10
|
||||
updated: 2026-05-10
|
||||
summary: >
|
||||
Fixed Code Reviewer silent run pattern by adding in_review to the
|
||||
heartbeat Get Assignments filter and clarifying review pickup in
|
||||
AGENTS.md. Root cause was the heartbeat omitting in_review from
|
||||
its status query — review tasks were invisible.
|
||||
references:
|
||||
- FRE-4952
|
||||
- agents/code-reviewer/HEARTBEAT.md
|
||||
- agents/code-reviewer/AGENTS.md
|
||||
evidence:
|
||||
- HEARTBEAT.md updated to include in_review in status filter
|
||||
- AGENTS.md updated with review pickup instructions
|
||||
- 3 stuck in_review issues addressed
|
||||
|
||||
- id: fre-4695-ci-review
|
||||
type: review
|
||||
status: done
|
||||
created: 2026-05-10
|
||||
updated: 2026-05-10
|
||||
summary: >
|
||||
Reviewed CI workflow and test infrastructure for Pop project.
|
||||
Found Go version matrix mismatch (1.21.x/1.22.x vs go.mod 1.23.0)
|
||||
and fragile coverage calculation (grep -oP).
|
||||
references:
|
||||
- FRE-4695
|
||||
- FRE-4951
|
||||
@@ -0,0 +1,25 @@
|
||||
# Code Reviewer Silent Run Pattern
|
||||
|
||||
**Status**: Fixed (FRE-4952 done)
|
||||
|
||||
## Problem
|
||||
|
||||
The Code Reviewer's `opencode_local` adapter doesn't auto-process `in_review` assignments,
|
||||
generating false-positive silent run detections. 4 occurrences so far (FRE-4946–4949).
|
||||
|
||||
## Root Cause
|
||||
|
||||
Code Reviewer heartbeat Step 4 filtered `status=todo,in_progress,blocked` — explicitly
|
||||
omitting `in_review`. Review tasks were invisible even when the agent ran.
|
||||
|
||||
## Fix (FRE-4952)
|
||||
|
||||
1. **agents/code-reviewer/HEARTBEAT.md** — Added `in_review` to Get Assignments filter
|
||||
2. **agents/code-reviewer/AGENTS.md** — Clarified review pickup and silent run pattern
|
||||
3. 3 stuck `in_review` issues addressed: FRE-4695, FRE-4763, FRE-4737
|
||||
|
||||
## Issues
|
||||
|
||||
- FRE-4951: Fix Go version matrix in CI workflow (subtask of FRE-4695) — todo
|
||||
- FRE-4952: Code Reviewer silent run pattern ✅ **Done**
|
||||
- FRE-4954: May be superseded by FRE-4952 (same root cause)
|
||||
@@ -0,0 +1,42 @@
|
||||
facts:
|
||||
- id: fre-4774-001
|
||||
type: issue
|
||||
summary: Production Turso DB had 0 tables — no migrations ever applied
|
||||
details: Connected to libsql://scripter-mikefreno.aws-us-east-1.turso.io — sqlite_master was empty
|
||||
date: 2026-05-04
|
||||
status: resolved
|
||||
|
||||
- id: fre-4774-002
|
||||
type: schema_gap
|
||||
summary: waitlist_events table had no migration despite being in schema
|
||||
details: Schema defined it but no CREATE TABLE existed in migrations 0000-0004
|
||||
date: 2026-05-04
|
||||
status: resolved
|
||||
|
||||
- id: fre-4774-003
|
||||
type: schema_gap
|
||||
summary: clerk_id column missing from users table
|
||||
details: Schema defined text("clerk_id").notNull().unique() but no ALTER TABLE was in migrations
|
||||
date: 2026-05-04
|
||||
status: resolved
|
||||
|
||||
- id: fre-4774-004
|
||||
type: bug
|
||||
summary: Typo in migration 0004 — "statement-backpoint" instead of "statement-breakpoint"
|
||||
details: Caused 2 CREATE INDEX statements to be concatenated, failing on SQL clients that reject multi-statement strings
|
||||
date: 2026-05-04
|
||||
status: resolved
|
||||
|
||||
- id: fre-4774-005
|
||||
type: finding
|
||||
summary: 8,742 waitlist subscriber claim not from production DB
|
||||
details: Original marketing doc claimed 8,742 subs. Production DB was empty. CMO needs to locate source data.
|
||||
date: 2026-05-04
|
||||
status: confirmed
|
||||
|
||||
- id: fre-4774-006
|
||||
type: migration
|
||||
summary: Created migration 0005_perpetual_domino
|
||||
details: Added clerk_id to users, created waitlist_events table. Applied to both dev and production.
|
||||
date: 2026-05-04
|
||||
status: resolved
|
||||
Reference in New Issue
Block a user