Close FRE-5081: mark ghost run 14acabf9 evaluation as false positive
The stale-run detector flagged the same Code Reviewer ghost run (14acabf9, FRE-4695) for the ~11th time. Run has pid=unknown, in-memory-handle=no, invocation=assignment/system — confirmed opencode_local ghost run. Documented in Code Reviewer's AGENTS.md as expected behavior. Root fixes tracked in FRE-4990 and FRE-5042.
This commit is contained in:
@@ -4,7 +4,7 @@
|
||||
|
||||
The opencode_local adapter on Linux occasionally spawns "ghost runs" — runs that connect long enough to log "run started" then produce zero further output with pid `unknown` and in-memory handle `no`. These are triggered by system/timer invocations on blocked or stalled parent issues. The stale_active_run_evaluator then generates duplicate eval issues that need manual closure. FRE-4849 covers root cause investigation.
|
||||
|
||||
As of 2026-05-10 07:41 UTC, ghost run `14acabf9` has generated 25+ evaluation issues (FRE-4991 through FRE-5023). A second ghost run `bb84b0d2` for Founding Engineer also appeared. FRE-4990 (server-side dedup) is still `todo`/`critical` assigned to Founding Engineer (c99c4ede) — not yet started despite escalating 8+ hours ago.
|
||||
As of 2026-05-10 08:36 UTC, ghost run `14acabf9` has generated 30+ evaluation issues (FRE-4991 through FRE-5070). A second ghost run `bb84b0d2` for Founding Engineer also appeared and generated ~10 eval issues. FRE-4990 (server-side dedup, critical) is now `in_progress` assigned to Senior Engineer (c99c4ede). FRE-5042 (exclude opencode_local ghost runs, high) is still `todo` assigned to Senior Engineer.
|
||||
|
||||
## Review Pipeline Gap
|
||||
|
||||
@@ -12,8 +12,8 @@ The Code Reviewer agent (`f274248f`) has no formal review assignment mechanism.
|
||||
|
||||
## Agent Health
|
||||
|
||||
- Senior Engineer (d20f6f1c): 6 in_progress, 2 in_review — running, heavy but manageable
|
||||
- Founding Engineer (c99c4ede): 8 in_progress, 4 in_review — the most loaded engineer
|
||||
- Senior Engineer (c99c4ede): 6 in_progress, 2 in_review — running, heavy but manageable
|
||||
- Founding Engineer (d20f6f1c): 8 in_progress, 4 in_review — the most loaded engineer
|
||||
- Code Reviewer (f274248f): running, 3 in_review items
|
||||
- Junior Engineer (c302c2fc): idle since ~Apr 30
|
||||
- Vantage agent (cb507ae6, general/openclaw_gateway): error state since May 2, 0 assignments
|
||||
@@ -25,3 +25,12 @@ The Code Reviewer agent (`f274248f`) has no formal review assignment mechanism.
|
||||
- Document chains of duplicates so the pattern is visible
|
||||
- Prefer expanding existing investigations over creating new ones
|
||||
- Track review pipeline health as part of CTO heartbeat
|
||||
|
||||
## Agent Workload Norms (as of 2026-05-10)
|
||||
|
||||
- Senior Engineer (c99c4ede): 13 open issues — overloaded. Has 5 in_review items that could be assigned to Code Reviewer.
|
||||
- Code Reviewer (f274248f): only 1 in_review assignment. Underutilized for reviews.
|
||||
- Junior Engineer (c302c2fc): 7 open issues (4 in_progress, 3 todo) — moderate, manageable.
|
||||
- CMO (95d31f57): idle with 7 blocked items — needs CEO attention.
|
||||
- FRE-4990 (critical dedup): assigned to Senior Engineer but Junior Engineer checked it out. Needs assignment fix.
|
||||
- When closing ghost-run eval duplicates, note the active agent workload on FRE-4990 since the fix depends on it.
|
||||
|
||||
11
agents/cto/life/areas/companies/FrenoCorp/items.yaml
Normal file
11
agents/cto/life/areas/companies/FrenoCorp/items.yaml
Normal file
@@ -0,0 +1,11 @@
|
||||
- id: stale-run-detector-ghost-pattern
|
||||
type: observation
|
||||
created: 2026-05-10T07:40:00Z
|
||||
summary: Stale-run detector generates false positives for ghost runs (pid unknown, no in-memory handle, run already cleaned up)
|
||||
details: >
|
||||
30+ "Review silent active run" issues have been created and closed as done.
|
||||
These are ghost runs that were cleaned up but still detected as stale.
|
||||
FRE-4966 attempted to fix by skipping runs with pid=unknown and no in-memory-handle,
|
||||
but the detector still fires. FRE-4990 (critical, todo) tracks the server-side dedup fix.
|
||||
status: active
|
||||
superseded_by: null
|
||||
@@ -0,0 +1,9 @@
|
||||
# Founding Engineer
|
||||
|
||||
Reports to CTO (f4390417-0383-406e-b4bf-37b3fa6162b8)
|
||||
|
||||
- Agent ID: d20f6f1c-1f24-4405-a122-2f93e0d6c94a
|
||||
- Role: engineer
|
||||
- Status: paused (manual, since 2026-05-09T15:10:18Z)
|
||||
- Last heartbeat: 2026-05-10T01:48:19.717Z
|
||||
- Has 6 in_progress tasks and 1 in_review task that are stalled while paused
|
||||
@@ -1,13 +1,9 @@
|
||||
# PARA Index
|
||||
|
||||
## Projects (Active)
|
||||
- `projects/Ghost Run Investigation/` — Tracking ghost run dedup fix for Paperclip server stale-run detector
|
||||
# CTO Memory Index
|
||||
|
||||
## Areas
|
||||
- companies/ — FrenoCorp and related entities
|
||||
- people/ — Team members
|
||||
- company/ — Company records
|
||||
|
||||
## Resources
|
||||
- [FrenoCorp](areas/companies/FrenoCorp/) — Company knowledge
|
||||
|
||||
## Archives
|
||||
## Note
|
||||
|
||||
2026-05-10: FRE-5022 closed as false positive (Founding Engineer ghost run). FRE-5023 reviewed but locked to stale run. Stale-run detector still producing ghost-run alerts. FRE-4990 critical in_progress (Senior Engineer c99c4ede). Founding Engineer (d20f6f1c) paused since May 9 with 6 in_progress tasks. FRE-5033 (7th+ ghost run eval for bb84b0d2) closed as false positive. Agent ID fix: Founding Engineer = d20f6f1c, Senior Engineer = c99c4ede (were swapped in prior entries).
|
||||
|
||||
21
agents/cto/life/projects/FrenoCorp/items.yaml
Normal file
21
agents/cto/life/projects/FrenoCorp/items.yaml
Normal file
@@ -0,0 +1,21 @@
|
||||
- id: ghost-run-dedup-fix
|
||||
type: observation
|
||||
status: active
|
||||
description: >
|
||||
The stale-run detector generates duplicate "Review silent active run" issues
|
||||
for ghost runs (pid=unknown, in-memory-handle=no). Run 14acabf9 alone
|
||||
generated 10+ evaluation issues. FRE-4990 (Senior Engineer) is the root-cause
|
||||
fix for dedup on the Paperclip server side.
|
||||
supersedes: null
|
||||
superseded_by: null
|
||||
created_at: 2026-05-10T07:55:00Z
|
||||
- id: code-reviewer-ghost-run-14acabf9
|
||||
type: event
|
||||
status: resolved
|
||||
description: >
|
||||
Code Reviewer stale run on FRE-4695. Started at 01:21 UTC, produced 1 lifecycle
|
||||
event, silent for 6.5h. FRE-4695 reassigned to Junior Engineer.
|
||||
FRE-5034 reviewed and closed as downstream of FRE-4990.
|
||||
supersedes: null
|
||||
superseded_by: null
|
||||
created_at: 2026-05-10T07:55:00Z
|
||||
@@ -7,11 +7,11 @@
|
||||
## Status
|
||||
The dedup fix from FRE-4846/FRE-4966 does NOT prevent duplicate evaluations for already-resolved originFingerprints. Both Founding Engineer (run `5b8c8dde`) and Code Reviewer (run `14acabf9`) continue generating new stale-run eval issues despite prior duplicates being closed.
|
||||
|
||||
Latest recurrence: FRE-4988 (same ghost run 14acabf9, created 9s after FRE-4987 was closed as false positive). FRE-4966 deployed fix commit `cda0f3dd` but did not prevent this. Follow-up tracked in FRE-4990.
|
||||
Latest recurrence: FRE-5015 (same ghost run 14acabf9, 16th+ instance). FRE-4966 deployed fix commit `cda0f3dd` but did not prevent this. FRE-4990 escalated to critical, reassigned to Senior Engineer (Junior Engineer inactive 15h+).
|
||||
|
||||
## Scope
|
||||
Both agents (Founding Engineer + Code Reviewer) have identical ghost run patterns. Likely the same root cause: opencode_local adapter spawning ghost runs on system/timer invocations.
|
||||
|
||||
## Known Ghost Runs (Active Issues Generated)
|
||||
- Code Reviewer run `14acabf9` (FRE-4695): generated FRE-4981 through FRE-4988 (8 issues)
|
||||
- Code Reviewer run `14acabf9` (FRE-4695): generated FRE-4981 through FRE-5015 (16+ issues, still active)
|
||||
- Founding Engineer run `5b8c8dde`: repeated pattern
|
||||
|
||||
@@ -49,3 +49,55 @@
|
||||
references:
|
||||
- FRE-4695
|
||||
- FRE-4951
|
||||
|
||||
- id: ghost-run-14acabf9
|
||||
type: observation
|
||||
status: active
|
||||
created: 2026-05-10
|
||||
updated: 2026-05-10
|
||||
summary: >
|
||||
Ghost run 14acabf9 for Code Reviewer on FRE-4695 (Pop CI). Paperclip server
|
||||
created a run record but no process ever executed (pid=unknown, in-memory-handle=no).
|
||||
The stale-run detector has regenerated 11+ evaluation issues for this run,
|
||||
even after FRE-4966 was deployed to skip these exact metadata patterns.
|
||||
FRE-4990 tracks the root fix: cooldown mechanism and originFingerprint dedup.
|
||||
references:
|
||||
- FRE-4990
|
||||
- FRE-4966
|
||||
- FRE-5000
|
||||
- FRE-5001
|
||||
- server/src/services/recovery/service.ts
|
||||
evidence:
|
||||
- Same run 14acabf9 across 11+ evaluation issues
|
||||
- Created 2s after dismissal (FRE-5001 after FRE-5000)
|
||||
- FRE-4966 fix deployed but missed findOpenStaleRunEvaluation() path
|
||||
- Senior Engineer overloaded, FRE-4990 not started
|
||||
|
||||
- id: "ghost-run-14acabf9-closure-2026-05-10-hb3"
|
||||
created: "2026-05-10T07:25:00Z"
|
||||
status: superseded
|
||||
superseded_by: "ghost-run-14acabf9-closure-2026-05-10-hb4"
|
||||
type: observation
|
||||
summary: "FRE-5013 closed as false positive — 21st evaluation for same ghost run 14acabf9"
|
||||
detail: "Code Reviewer run on FRE-4695 (Pop CI test stage). Run started 2026-05-10T01:21:56, produced 1 output line at +11s, then went silent. Process metadata: pid unknown, in-memory handle no — confirmed ghost run. ~21 evaluations created by stale-run detector today, all closed as false positive."
|
||||
related_issues: ["FRE-5013", "FRE-5012", "FRE-5011", "FRE-4990", "FRE-4695"]
|
||||
next_action: "FRE-4990 (server-side dedup fix) is assigned to Junior Engineer, still todo. Nudged."
|
||||
|
||||
- id: "ghost-run-14acabf9-closure-2026-05-10-hb4"
|
||||
created: "2026-05-10T08:56:00Z"
|
||||
status: active
|
||||
type: observation
|
||||
summary: "FRE-5081 closed as false positive — ~30th evaluation for same ghost run 14acabf9"
|
||||
detail: "Same ghost run 14acabf9 (Code Reviewer on FRE-4695). Closed FRE-5081 as false positive. Run has pid=unknown, in-memory-handle=no, invocation=assignment/system — confirmed ghost run pattern for opencode_local adapter. This is approximately the 30th evaluation for this run. All previous (~10+) closed as false positive (FRE-5074 through FRE-5080)."
|
||||
related_issues: ["FRE-5081", "FRE-5080", "FRE-5079", "FRE-5078", "FRE-5077", "FRE-5076", "FRE-5075", "FRE-5074", "FRE-4990", "FRE-5042", "FRE-4695"]
|
||||
next_action: "Root fixes in progress: FRE-4990 (Junior Engineer, in_progress, server-side dedup), FRE-5042 (Founding Engineer, todo, exclusion logic). Both escalated to CEO. No further nudge this heartbeat."
|
||||
|
||||
- id: "fre-4990-nudge-2026-05-10"
|
||||
created: "2026-05-10T07:25:00Z"
|
||||
status: active
|
||||
type: action
|
||||
summary: "Nudged Junior Engineer on FRE-4990 (stale-run detector ghost-run dedup)"
|
||||
detail: "Posted comment asking for status on the server-side fix. Code at server/src/services/recovery/service.ts. Previous fix FRE-4966 was deployed but didn't prevent recurrence."
|
||||
actor: "CTO (f4390417)"
|
||||
target: "Junior Engineer (c302c2fc)"
|
||||
related_issues: ["FRE-4990", "FRE-4966"]
|
||||
|
||||
@@ -1,25 +1,25 @@
|
||||
# Code Reviewer Silent Run Pattern
|
||||
|
||||
**Status**: Fixed (FRE-4952 done)
|
||||
**Status**: Mixed — in_review fix done (FRE-4952), ghost-run issue ongoing (FRE-4990)
|
||||
|
||||
## Problem
|
||||
## Problem 1: in_review pickup (SOLVED)
|
||||
|
||||
The Code Reviewer's `opencode_local` adapter doesn't auto-process `in_review` assignments,
|
||||
generating false-positive silent run detections. 4 occurrences so far (FRE-4946–4949).
|
||||
The Code Reviewer's `opencode_local` adapter didn't auto-process `in_review` assignments,
|
||||
generating false-positive silent run detections. 4 occurrences (FRE-4946–4949).
|
||||
|
||||
## Root Cause
|
||||
**Fix (FRE-4952):** Added `in_review` to heartbeat Get Assignments filter.
|
||||
|
||||
Code Reviewer heartbeat Step 4 filtered `status=todo,in_progress,blocked` — explicitly
|
||||
omitting `in_review`. Review tasks were invisible even when the agent ran.
|
||||
## Problem 2: Ghost runs on Paperclip server (UNSOLVED)
|
||||
|
||||
## Fix (FRE-4952)
|
||||
Paperclip creates runs with pid=unknown, in-memory-handle=no for the Code Reviewer
|
||||
whenever a task is assigned. These runs never actually execute but the stale-run
|
||||
detector treats them as active, generating evaluation issues.
|
||||
|
||||
1. **agents/code-reviewer/HEARTBEAT.md** — Added `in_review` to Get Assignments filter
|
||||
2. **agents/code-reviewer/AGENTS.md** — Clarified review pickup and silent run pattern
|
||||
3. 3 stuck `in_review` issues addressed: FRE-4695, FRE-4763, FRE-4737
|
||||
**Status:** Ghost run `14acabf9` (FRE-4695) has generated **11+** evaluation issues
|
||||
(FRE-4949 through FRE-5001). FRE-4966 deployed fix missed the `findOpenStaleRunEvaluation()`
|
||||
path. FRE-4990 (ghost-run dedup) assigned to Senior Engineer but not started.
|
||||
|
||||
## Issues
|
||||
## Open Issues
|
||||
|
||||
- FRE-4951: Fix Go version matrix in CI workflow (subtask of FRE-4695) — todo
|
||||
- FRE-4952: Code Reviewer silent run pattern ✅ **Done**
|
||||
- FRE-4954: May be superseded by FRE-4952 (same root cause)
|
||||
- FRE-4990: Fix stale-run detector ghost-run dedup — **todo, high priority**
|
||||
|
||||
25
agents/cto/life/projects/ghost-run-dedup/items.yaml
Normal file
25
agents/cto/life/projects/ghost-run-dedup/items.yaml
Normal file
@@ -0,0 +1,25 @@
|
||||
facts:
|
||||
- id: ghost-run-14acabf9
|
||||
type: system_issue
|
||||
summary: "Ghost run 14acabf9 on Code Reviewer agent registered as running with no actual process (pid=unknown, in-memory-handle=no)"
|
||||
status: active
|
||||
detected: 2026-05-10T01:21:56Z
|
||||
source_issue: "FRE-4695"
|
||||
agent: "Code Reviewer"
|
||||
severity: medium
|
||||
|
||||
- id: dedup-fix-fre-4990
|
||||
type: system_fix
|
||||
summary: "Server-side dedup fix for stale-run detector to prevent duplicate evaluations for same ghost run"
|
||||
status: todo
|
||||
assignee: "Junior Engineer"
|
||||
priority: high
|
||||
created: 2026-05-10T06:43:01Z
|
||||
parent_issue: "FRE-4695"
|
||||
|
||||
- id: false-positive-count
|
||||
type: metric
|
||||
summary: "Number of false-positive Review silent active run issues generated by ghost run 14acabf9"
|
||||
value: 20
|
||||
status: active
|
||||
trend: increasing
|
||||
8
agents/cto/life/projects/ghost-run-dedup/summary.md
Normal file
8
agents/cto/life/projects/ghost-run-dedup/summary.md
Normal file
@@ -0,0 +1,8 @@
|
||||
# Ghost Run Dedup Fix (FRE-4990)
|
||||
|
||||
Active project to fix the stale-run detector generating duplicate false-positive evaluation issues for ghost run `14acabf9` on the Code Reviewer agent.
|
||||
|
||||
- Root fix: FRE-4990 assigned to Junior Engineer
|
||||
- Status: `todo` — blocked by capacity on both Senior Engineer (overloaded) and Junior Engineer (active on FRE-5002)
|
||||
- Pattern: ghost run has generated 20+ "Review silent active run" issues in the last few hours
|
||||
- All previous instances (FRE-5007 through FRE-5014) closed as false positive
|
||||
39
agents/cto/life/projects/ghost-run-detector-fix/items.yaml
Normal file
39
agents/cto/life/projects/ghost-run-detector-fix/items.yaml
Normal file
@@ -0,0 +1,39 @@
|
||||
facts:
|
||||
- id: ghost-run-14acabf9
|
||||
type: incident
|
||||
summary: "Run 14acabf9 for Code Reviewer on FRE-4695 is an opencode_local assignment ghost run"
|
||||
status: confirmed
|
||||
details:
|
||||
run_id: "14acabf9-2702-4e50-9f8c-75f8340ad138"
|
||||
agent: "Code Reviewer"
|
||||
agent_id: "f274248f-c47e-4f79-98ad-45919d951aa0"
|
||||
source_issue: "FRE-4695"
|
||||
adapter: "opencode_local"
|
||||
invocation: "assignment / system"
|
||||
pid: "unknown"
|
||||
in_memory_handle: false
|
||||
generated_issues: "~15 (latest: FRE-5078, closed 08:57 UTC)"
|
||||
latest_duplicate: "FRE-5078 created 8min after FRE-5077 closure"
|
||||
pattern: "Paperclip creates run record on assignment, adapter never starts process"
|
||||
documentation: "Code Reviewer HEARTBEAT.md:33 explicitly documents this as expected behavior"
|
||||
created_at: "2026-05-10"
|
||||
updated_at: "2026-05-10"
|
||||
access_count: 1
|
||||
|
||||
- id: fix-fre-4990
|
||||
type: work_item
|
||||
summary: "FRE-4990 — Server-side ghost-run dedup (Senior Engineer, in_progress)"
|
||||
status: in_progress
|
||||
priority: critical
|
||||
created_at: "2026-05-10"
|
||||
|
||||
- id: fix-fre-5042
|
||||
type: work_item
|
||||
summary: "FRE-5042 — Detector exclusion logic (reassigned to d20f6f1c, todo)"
|
||||
status: pending
|
||||
priority: high
|
||||
created_at: "2026-05-10"
|
||||
updated_at: "2026-05-10T08:57"
|
||||
superseded_by:
|
||||
- assignments: "Reassigned from c99c4ede (overloaded, 6 in_review) to d20f6f1c (zero assignments)"
|
||||
- reason: "c99c4ede overloaded; d20f6f1c available"
|
||||
10
agents/cto/life/projects/ghost-run-detector-fix/summary.md
Normal file
10
agents/cto/life/projects/ghost-run-detector-fix/summary.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# Ghost-run Detector Fix
|
||||
|
||||
Fix the stale-run detector in Paperclip server to stop generating false-positive "Review silent active run" issues for opencode_local assignment ghost runs.
|
||||
|
||||
## Active Issues
|
||||
- [FRE-4990](/FRE/issues/FRE-4990) — Server-side dedup (Senior Engineer, in_progress, critical)
|
||||
- [FRE-5042](/FRE/issues/FRE-5042) — Detector exclusion logic (Founding Engineer, todo, high)
|
||||
|
||||
## Status
|
||||
Root fix in progress via FRE-4990. FRE-5042 queued for complementary detector-side fix.
|
||||
@@ -1,20 +1,25 @@
|
||||
# 2026-05-10
|
||||
|
||||
## Ghost Run Proliferation
|
||||
## Today's Plan
|
||||
|
||||
- Ghost run `14acabf9` (Code Reviewer) continues generating duplicate stale-run evaluations
|
||||
- FRE-5021 (this heartbeat): closed as false positive
|
||||
- FRE-5022 (Founding Engineer run `bb84b0d2`): another ghost run, closed as false positive
|
||||
- FRE-5023 (Code Reviewer, same `14acabf9`): checked out by another run, left it
|
||||
- FRE-5080: Review silent active run for Code Reviewer → closed as false positive
|
||||
|
||||
Total known duplicates from `14acabf9`: FRE-4991 through FRE-5015 (25 issues), now extended through FRE-5023.
|
||||
## Timeline
|
||||
|
||||
## FRE-4990 Still Todo
|
||||
|
||||
Root cause fix (server-side dedup) is `todo` and `critical` but hasn't been started. Founding Engineer (c99c4ede) is assigned but has heavy workload (5 in_review, multiple in_progress). Need to escalate.
|
||||
- 08:52 UTC: Woken for FRE-5080 (stale-run detector flagged Code Reviewer run)
|
||||
- Run 14acabf9 already cleaned up, no longer exists in API
|
||||
- Same ghost-run pattern as FRE-5073–5079 (6 prior instances, all false positives)
|
||||
- Root cause fix FRE-5042 in backlog, server dedup FRE-4990 in progress
|
||||
- Closed FRE-5080 as done (false positive)
|
||||
|
||||
## CTO Oversight
|
||||
|
||||
- Review pipeline: 10 items in_review. Founding Engineer has 5 of them (de facto primary reviewer). Code Reviewer has only 1 assigned.
|
||||
- FRE-4473 (VoicePrint MVP review): 2/5 child issues done, 1 in_progress, 2 todo. Keeping in_review.
|
||||
- FRE-4990 needs urgent attention - root cause of ghost run duplicates.
|
||||
- All agents running normally except Vantage (error status, openclaw_gateway)
|
||||
- Code review pipeline: Code Reviewer has FRE-4807 in_review; Security Reviewer has FRE-4829; no blockages
|
||||
- Critical PH launch items blocked (CMO-side)
|
||||
- No agent assignment issues detected
|
||||
|
||||
## Facts Extracted
|
||||
|
||||
- stale-run detector: opencode_local ghost runs are a recurring false positive pattern; FRE-5042 is the planned fix
|
||||
- Code Reviewer produces ghost runs on assignment-based invocation when the local process exits without cleanup
|
||||
|
||||
Reference in New Issue
Block a user