feat: sync watchdog recovery slice
This commit is contained in:
132
README.md
132
README.md
@@ -8,112 +8,72 @@
|
|||||||
- **dispatch receipt binding**
|
- **dispatch receipt binding**
|
||||||
- **anti-blackhole / completion-delivery watchdog groundwork**
|
- **anti-blackhole / completion-delivery watchdog groundwork**
|
||||||
|
|
||||||
目標是避免這種情況再次發生:
|
目標是避免兩類問題持續發生:
|
||||||
|
|
||||||
- 任務已完成
|
1. **continuity failure / auto-next break**
|
||||||
- 下一步其實已經明確
|
2. **subagent anti-blackhole / fake timeout**
|
||||||
- 但沒有真的 dispatch 下一個 task
|
|
||||||
- 最後流程卻還是收尾,造成 **auto-next break / continuity failure**
|
|
||||||
|
|
||||||
## 目前已完成
|
## 目前已完成
|
||||||
|
|
||||||
目前這個 repo 已經包含並驗證以下能力:
|
### A. Continuity hard-gate
|
||||||
|
- continuity evaluator
|
||||||
|
- dispatch receipt binding groundwork
|
||||||
|
- `derivedAction` continuity binding
|
||||||
|
- `dry_run_dispatch` 不得冒充真 receipt
|
||||||
|
- fake receipt authority 最小收緊
|
||||||
|
- hook integration 已接入
|
||||||
|
|
||||||
1. **continuity evaluator**
|
### B. Anti-blackhole watchdog recovery
|
||||||
- task 完成、next action 已知、但沒有 next dispatch receipt,且 closure 狀態又不是 `waiting_user` / `blocked` / `pending_verification` 時,會判定 `continuity_failure`。
|
- watchdog status recompute
|
||||||
|
- 最小 recovery decision 閉環:
|
||||||
2. **dispatch receipt binding groundwork**
|
- `fetch_history`
|
||||||
- 已有 continuity receipt storage 定義
|
- `respawn`
|
||||||
- 已有最小 dispatch receipt writer
|
- `blocked`
|
||||||
- 已有 continuity gate / dispatch binding 對應測試
|
- owner-visible reporting payload
|
||||||
|
- scenario matrix tests
|
||||||
3. **`derivedAction` 與 `nextDerivedAction` 一致納入 continuity 判定**
|
|
||||||
- 不再只有 `nextDerivedAction` 才受 gate 約束。
|
|
||||||
|
|
||||||
4. **`dry_run_dispatch` 不得冒充真 receipt**
|
|
||||||
- planner 的 dry-run 結果不再被 handler fallback 當成真實 dispatch receipt。
|
|
||||||
|
|
||||||
5. **fake receipt authority 已補強**
|
|
||||||
- continuity gate 不再接受任意 non-null `dispatchReceipt`
|
|
||||||
- 現在至少要求最小 receipt 欄位:
|
|
||||||
- `planId`
|
|
||||||
- `currentTask`
|
|
||||||
- `nextDerivedAction`
|
|
||||||
- `dispatchedAt`
|
|
||||||
|
|
||||||
6. **hook integration 已接入**
|
|
||||||
- continuity gate 已接進 `hooks/force-recall/handler.ts`
|
|
||||||
- 目前會透過 `[APPROVED_PLAN_CONTINUITY_GATE]` block 注入現行 flow
|
|
||||||
|
|
||||||
## 目前限制
|
## 目前限制
|
||||||
|
- continuity 仍偏 prompt-level hard-gate integration
|
||||||
這條線雖然已經接入現行 flow,但目前仍偏向 **prompt-level hard-gate integration**,而不是 engine-level abort。也就是說:
|
- watchdog recovery 目前驗收的是 decision / reporting / test slice,不是 live integration
|
||||||
|
|
||||||
- 已經不是只有規則文件
|
|
||||||
- 已經不是只有獨立腳本測試
|
|
||||||
- 但也還不是最底層 runtime/core 的絕對阻斷器
|
|
||||||
|
|
||||||
## 下一步建議
|
## 下一步建議
|
||||||
|
1. continuity runtime enforcement hardening
|
||||||
下一階段最合理的方向有兩條:
|
2. watchdog live recovery integration
|
||||||
|
3. escalation / receipt contract hardening
|
||||||
1. **把 continuity hard-gate 再往更硬的 runtime enforcement 推進**
|
|
||||||
2. **回頭補完 anti-blackhole / completion-delivery watchdog recovery 閉環**
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## English Description
|
## English Description
|
||||||
|
|
||||||
This repository is a focused export from a larger OpenClaw workspace. It captures a workflow hardening workstream around:
|
This repository is a focused export from a larger OpenClaw workspace covering:
|
||||||
|
|
||||||
- **approved plan continuity hard-gate**
|
- **approved plan continuity hard-gate**
|
||||||
- **dispatch receipt binding**
|
- **anti-blackhole / completion-delivery watchdog recovery**
|
||||||
- **anti-blackhole / completion-delivery watchdog groundwork**
|
|
||||||
|
|
||||||
The goal is to prevent this failure mode:
|
|
||||||
|
|
||||||
- a task is completed,
|
|
||||||
- the next step is already known,
|
|
||||||
- but the next task is never actually dispatched,
|
|
||||||
- and the flow still closes out as if continuity were preserved.
|
|
||||||
|
|
||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
The repo now includes and validates the following capabilities:
|
### A. Continuity hard-gate
|
||||||
|
- continuity evaluator
|
||||||
|
- dispatch receipt binding groundwork
|
||||||
|
- `derivedAction` continuity binding
|
||||||
|
- `dry_run_dispatch` no longer accepted as a real receipt
|
||||||
|
- fake receipt authority tightened
|
||||||
|
- hook integration present
|
||||||
|
|
||||||
1. **Continuity evaluator**
|
### B. Anti-blackhole watchdog recovery
|
||||||
- When a task is complete, the next action is known, and there is no next dispatch receipt, and the closure state is not `waiting_user`, `blocked`, or `pending_verification`, the flow is classified as `continuity_failure`.
|
- watchdog status recompute
|
||||||
|
- minimal recovery-decision loop:
|
||||||
|
- `fetch_history`
|
||||||
|
- `respawn`
|
||||||
|
- `blocked`
|
||||||
|
- owner-visible reporting payload
|
||||||
|
- scenario matrix tests
|
||||||
|
|
||||||
2. **Dispatch receipt binding groundwork**
|
## Current Limitations
|
||||||
- continuity receipt storage shape
|
- continuity remains prompt-level rather than engine-level
|
||||||
- minimal dispatch receipt writer
|
- watchdog recovery is validated as a decision/reporting/test slice, not live execution integration
|
||||||
- continuity gate / dispatch binding tests
|
|
||||||
|
|
||||||
3. **`derivedAction` is treated as a real next-action source**
|
|
||||||
- The gate no longer depends only on `nextDerivedAction`.
|
|
||||||
|
|
||||||
4. **`dry_run_dispatch` is no longer accepted as a real receipt**
|
|
||||||
- Planner dry-run output is no longer promoted into a real dispatch receipt by handler fallback logic.
|
|
||||||
|
|
||||||
5. **Fake receipt authority has been tightened**
|
|
||||||
- The continuity gate no longer accepts any arbitrary non-null `dispatchReceipt`
|
|
||||||
- It now requires at least these minimum fields:
|
|
||||||
- `planId`
|
|
||||||
- `currentTask`
|
|
||||||
- `nextDerivedAction`
|
|
||||||
- `dispatchedAt`
|
|
||||||
|
|
||||||
6. **Hook integration is now present**
|
|
||||||
- The continuity gate is integrated into `hooks/force-recall/handler.ts`
|
|
||||||
- It currently enters the live flow through the `[APPROVED_PLAN_CONTINUITY_GATE]` injected block
|
|
||||||
|
|
||||||
## Current Limitation
|
|
||||||
|
|
||||||
This workstream is now beyond pure documentation and beyond isolated script-level testing, but it still behaves more like a **prompt-level hard-gate integration** than a true engine-level abort mechanism.
|
|
||||||
|
|
||||||
## Suggested Next Steps
|
## Suggested Next Steps
|
||||||
|
1. continuity runtime enforcement hardening
|
||||||
Two reasonable follow-up directions remain:
|
2. watchdog live recovery integration
|
||||||
|
3. escalation / receipt contract hardening
|
||||||
1. **push continuity hard-gate further toward stronger runtime enforcement**
|
|
||||||
2. **return to anti-blackhole / completion-delivery watchdog recovery closure**
|
|
||||||
|
|||||||
@@ -231,6 +231,108 @@ function recomputeStatus(payload) {
|
|||||||
return 'active';
|
return 'active';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function decideRecoveryAction(payload, status) {
|
||||||
|
if (!payload || typeof payload !== 'object') {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (status !== 'done_but_not_forwarded') {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const attemptCountRaw = payload.recoveryAttemptCount;
|
||||||
|
const recoveryAttemptCount = Number.isFinite(attemptCountRaw)
|
||||||
|
? attemptCountRaw
|
||||||
|
: Number.parseInt(String(attemptCountRaw ?? '0'), 10);
|
||||||
|
|
||||||
|
if (!Number.isNaN(recoveryAttemptCount) && recoveryAttemptCount >= 2) {
|
||||||
|
return 'blocked';
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!Number.isNaN(recoveryAttemptCount) && recoveryAttemptCount >= 1) {
|
||||||
|
return 'respawn';
|
||||||
|
}
|
||||||
|
|
||||||
|
return 'fetch_history';
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildReportingPayload(payload, status, recoveryDecision) {
|
||||||
|
const detail = {
|
||||||
|
runId: typeof payload?.runId === 'string' ? payload.runId : null,
|
||||||
|
childSessionKey: typeof payload?.childSessionKey === 'string' ? payload.childSessionKey : null,
|
||||||
|
status,
|
||||||
|
recoveryDecision,
|
||||||
|
};
|
||||||
|
|
||||||
|
if (status === 'suspect_delivery_failure') {
|
||||||
|
return {
|
||||||
|
ownerVisible: true,
|
||||||
|
category: 'suspect_delivery_failure',
|
||||||
|
decision: 'report',
|
||||||
|
summary: 'Subagent delivery is suspected to have failed after crossing SLA.',
|
||||||
|
detail,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (status === 'done_but_not_forwarded' && recoveryDecision === 'fetch_history') {
|
||||||
|
return {
|
||||||
|
ownerVisible: true,
|
||||||
|
category: 'done_but_not_forwarded',
|
||||||
|
decision: 'fetch_history',
|
||||||
|
summary: 'Child run is done but no forwarded completion receipt is confirmed yet.',
|
||||||
|
detail,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (status === 'done_but_not_forwarded' && recoveryDecision === 'respawn') {
|
||||||
|
return {
|
||||||
|
ownerVisible: true,
|
||||||
|
category: 'done_but_not_forwarded',
|
||||||
|
decision: 'respawn',
|
||||||
|
summary: 'Child run is done but recovery already failed once; respawn is the next conservative step.',
|
||||||
|
detail,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (status === 'done_but_not_forwarded' && recoveryDecision === 'blocked') {
|
||||||
|
return {
|
||||||
|
ownerVisible: true,
|
||||||
|
category: 'done_but_not_forwarded',
|
||||||
|
decision: 'blocked',
|
||||||
|
summary: 'Child run is still not forwarded after repeated recovery attempts; owner attention is required.',
|
||||||
|
detail,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (status === 'completed') {
|
||||||
|
return {
|
||||||
|
ownerVisible: false,
|
||||||
|
category: 'completed',
|
||||||
|
decision: 'none',
|
||||||
|
summary: 'Completion receipt is present; no owner-visible report is needed.',
|
||||||
|
detail,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (status === 'active') {
|
||||||
|
return {
|
||||||
|
ownerVisible: false,
|
||||||
|
category: 'active',
|
||||||
|
decision: 'none',
|
||||||
|
summary: 'Dispatch is still within SLA; no owner-visible report is needed.',
|
||||||
|
detail,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
ownerVisible: false,
|
||||||
|
category: status,
|
||||||
|
decision: 'none',
|
||||||
|
summary: 'No owner-visible report is needed.',
|
||||||
|
detail,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
function main() {
|
function main() {
|
||||||
const args = parseArgs(process.argv.slice(2));
|
const args = parseArgs(process.argv.slice(2));
|
||||||
|
|
||||||
@@ -244,6 +346,8 @@ function main() {
|
|||||||
const dispatchWrite = writeDispatchReceiptState(inputPayload);
|
const dispatchWrite = writeDispatchReceiptState(inputPayload);
|
||||||
const completionWrite = writeCompletionReceiptState(inputPayload);
|
const completionWrite = writeCompletionReceiptState(inputPayload);
|
||||||
const status = recomputeStatus(inputPayload);
|
const status = recomputeStatus(inputPayload);
|
||||||
|
const recoveryDecision = decideRecoveryAction(inputPayload, status);
|
||||||
|
const reporting = buildReportingPayload(inputPayload, status, recoveryDecision);
|
||||||
|
|
||||||
if ('content' in input) {
|
if ('content' in input) {
|
||||||
delete input.content;
|
delete input.content;
|
||||||
@@ -260,7 +364,7 @@ function main() {
|
|||||||
const response = {
|
const response = {
|
||||||
ok: true,
|
ok: true,
|
||||||
tool: 'subagent_delivery_watchdog',
|
tool: 'subagent_delivery_watchdog',
|
||||||
version: 'skeleton-v4',
|
version: 'skeleton-v5',
|
||||||
mode: 'receipt-write',
|
mode: 'receipt-write',
|
||||||
args: {
|
args: {
|
||||||
compact: args.compact,
|
compact: args.compact,
|
||||||
@@ -270,8 +374,10 @@ function main() {
|
|||||||
result: {
|
result: {
|
||||||
status,
|
status,
|
||||||
message: status === 'not_implemented'
|
message: status === 'not_implemented'
|
||||||
? 'Dispatch and completion receipt writes are implemented; status recompute only handles basic active/suspect/completed states.'
|
? 'Dispatch and completion receipt writes are implemented; status recompute only handles basic active/suspect/completed states plus conservative recovery/reporting decisions.'
|
||||||
: 'Basic watchdog status recompute completed.',
|
: 'Basic watchdog status recompute completed.',
|
||||||
|
recoveryDecision,
|
||||||
|
reporting,
|
||||||
records,
|
records,
|
||||||
dispatchReceiptWrite: dispatchWrite,
|
dispatchReceiptWrite: dispatchWrite,
|
||||||
completionReceiptWrite: completionWrite,
|
completionReceiptWrite: completionWrite,
|
||||||
|
|||||||
@@ -59,6 +59,25 @@ function printResult(prefix, name, detail = '') {
|
|||||||
process.stdout.write(`${prefix} ${name}${suffix}\n`);
|
process.stdout.write(`${prefix} ${name}${suffix}\n`);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function runFixture(payloadInput, fixtureName = 'fixture.json') {
|
||||||
|
const runner = createFixtureRunner();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const inputPath = runner.writeFixture(fixtureName, payloadInput);
|
||||||
|
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
||||||
|
|
||||||
|
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}\n${result.stderr}`);
|
||||||
|
assert.equal(result.stderr, '');
|
||||||
|
|
||||||
|
return {
|
||||||
|
payload: JSON.parse(result.stdout),
|
||||||
|
inputPath,
|
||||||
|
};
|
||||||
|
} finally {
|
||||||
|
runner.cleanup();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
test('fixture runner can invoke watchdog skeleton with a generated input file', () => {
|
test('fixture runner can invoke watchdog skeleton with a generated input file', () => {
|
||||||
const runner = createFixtureRunner();
|
const runner = createFixtureRunner();
|
||||||
|
|
||||||
@@ -84,8 +103,6 @@ test('fixture runner can invoke watchdog skeleton with a generated input file',
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
test('watchdog reports active before SLA when dispatch exists and no completion receipt has arrived yet', () => {
|
test('watchdog reports active before SLA when dispatch exists and no completion receipt has arrived yet', () => {
|
||||||
const runner = createFixtureRunner();
|
const runner = createFixtureRunner();
|
||||||
|
|
||||||
@@ -100,8 +117,7 @@ test('watchdog reports active before SLA when dispatch exists and no completion
|
|||||||
|
|
||||||
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
||||||
|
|
||||||
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}
|
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}\n${result.stderr}`);
|
||||||
${result.stderr}`);
|
|
||||||
assert.equal(result.stderr, '');
|
assert.equal(result.stderr, '');
|
||||||
|
|
||||||
const payload = JSON.parse(result.stdout);
|
const payload = JSON.parse(result.stdout);
|
||||||
@@ -114,7 +130,6 @@ ${result.stderr}`);
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
||||||
test('watchdog reports suspect delivery failure after SLA when dispatch exists and no completion receipt has arrived yet', () => {
|
test('watchdog reports suspect delivery failure after SLA when dispatch exists and no completion receipt has arrived yet', () => {
|
||||||
const runner = createFixtureRunner();
|
const runner = createFixtureRunner();
|
||||||
|
|
||||||
@@ -129,8 +144,7 @@ test('watchdog reports suspect delivery failure after SLA when dispatch exists a
|
|||||||
|
|
||||||
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
||||||
|
|
||||||
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}
|
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}\n${result.stderr}`);
|
||||||
${result.stderr}`);
|
|
||||||
assert.equal(result.stderr, '');
|
assert.equal(result.stderr, '');
|
||||||
|
|
||||||
const payload = JSON.parse(result.stdout);
|
const payload = JSON.parse(result.stdout);
|
||||||
@@ -143,7 +157,6 @@ ${result.stderr}`);
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
||||||
test('watchdog reports completed when dispatch exists and completion receipt has arrived', () => {
|
test('watchdog reports completed when dispatch exists and completion receipt has arrived', () => {
|
||||||
const runner = createFixtureRunner();
|
const runner = createFixtureRunner();
|
||||||
|
|
||||||
@@ -159,8 +172,7 @@ test('watchdog reports completed when dispatch exists and completion receipt has
|
|||||||
|
|
||||||
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
||||||
|
|
||||||
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}
|
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}\n${result.stderr}`);
|
||||||
${result.stderr}`);
|
|
||||||
assert.equal(result.stderr, '');
|
assert.equal(result.stderr, '');
|
||||||
|
|
||||||
const payload = JSON.parse(result.stdout);
|
const payload = JSON.parse(result.stdout);
|
||||||
@@ -188,8 +200,7 @@ test('watchdog reports done but not forwarded when child run is marked done with
|
|||||||
|
|
||||||
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
||||||
|
|
||||||
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}
|
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}\n${result.stderr}`);
|
||||||
${result.stderr}`);
|
|
||||||
assert.equal(result.stderr, '');
|
assert.equal(result.stderr, '');
|
||||||
|
|
||||||
const payload = JSON.parse(result.stdout);
|
const payload = JSON.parse(result.stdout);
|
||||||
@@ -202,6 +213,302 @@ ${result.stderr}`);
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('watchdog prefers fetch_history recovery when child run is done but no forwarded completion receipt exists', () => {
|
||||||
|
const runner = createFixtureRunner();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const inputPath = runner.writeFixture('dispatch-done-not-forwarded-recovery.json', {
|
||||||
|
runId: 'fixture-run-done-not-forwarded-recovery',
|
||||||
|
childSessionKey: 'session:done-not-forwarded-recovery',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:05:00.000Z',
|
||||||
|
childRunStatus: 'done',
|
||||||
|
forwardedToMain: false,
|
||||||
|
recoveryAttemptCount: 0,
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
||||||
|
|
||||||
|
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}\n${result.stderr}`);
|
||||||
|
assert.equal(result.stderr, '');
|
||||||
|
|
||||||
|
const payload = JSON.parse(result.stdout);
|
||||||
|
assert.equal(payload.ok, true);
|
||||||
|
assert.equal(payload.result.status, 'done_but_not_forwarded');
|
||||||
|
assert.equal(payload.result.recoveryDecision, 'fetch_history');
|
||||||
|
} finally {
|
||||||
|
runner.cleanup();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('watchdog escalates to respawn when fetch_history recovery was already attempted and delivery is still not forwarded', () => {
|
||||||
|
const runner = createFixtureRunner();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const inputPath = runner.writeFixture('dispatch-done-not-forwarded-respawn.json', {
|
||||||
|
runId: 'fixture-run-done-not-forwarded-respawn',
|
||||||
|
childSessionKey: 'session:done-not-forwarded-respawn',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:06:00.000Z',
|
||||||
|
childRunStatus: 'done',
|
||||||
|
forwardedToMain: false,
|
||||||
|
recoveryAttemptCount: 1,
|
||||||
|
recoveryAction: 'fetch_history',
|
||||||
|
lastRecoveryAt: '2026-04-24T10:05:30.000Z',
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
||||||
|
|
||||||
|
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}\n${result.stderr}`);
|
||||||
|
assert.equal(result.stderr, '');
|
||||||
|
|
||||||
|
const payload = JSON.parse(result.stdout);
|
||||||
|
assert.equal(payload.ok, true);
|
||||||
|
assert.equal(payload.result.status, 'done_but_not_forwarded');
|
||||||
|
assert.equal(payload.result.recoveryDecision, 'respawn');
|
||||||
|
} finally {
|
||||||
|
runner.cleanup();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('watchdog escalates to blocked when respawn recovery was already attempted and delivery is still not forwarded', () => {
|
||||||
|
const runner = createFixtureRunner();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const inputPath = runner.writeFixture('dispatch-done-not-forwarded-blocked.json', {
|
||||||
|
runId: 'fixture-run-done-not-forwarded-blocked',
|
||||||
|
childSessionKey: 'session:done-not-forwarded-blocked',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:07:00.000Z',
|
||||||
|
childRunStatus: 'done',
|
||||||
|
forwardedToMain: false,
|
||||||
|
recoveryAttemptCount: 2,
|
||||||
|
recoveryAction: 'respawn',
|
||||||
|
lastRecoveryAt: '2026-04-24T10:06:30.000Z',
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = runner.runWatchdog(['--compact', '--input', inputPath]);
|
||||||
|
|
||||||
|
assert.equal(result.status, 0, `expected zero exit status, got ${result.status}\n${result.stderr}`);
|
||||||
|
assert.equal(result.stderr, '');
|
||||||
|
|
||||||
|
const payload = JSON.parse(result.stdout);
|
||||||
|
assert.equal(payload.ok, true);
|
||||||
|
assert.equal(payload.result.status, 'done_but_not_forwarded');
|
||||||
|
assert.equal(payload.result.recoveryDecision, 'blocked');
|
||||||
|
} finally {
|
||||||
|
runner.cleanup();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('owner-visible reporting marks suspect_delivery_failure as reportable with minimal payload', () => {
|
||||||
|
const { payload } = runFixture({
|
||||||
|
runId: 'fixture-run-report-suspect',
|
||||||
|
childSessionKey: 'session:report-suspect',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:15:00.000Z',
|
||||||
|
}, 'report-suspect.json');
|
||||||
|
|
||||||
|
assert.deepEqual(payload.result.reporting, {
|
||||||
|
ownerVisible: true,
|
||||||
|
category: 'suspect_delivery_failure',
|
||||||
|
decision: 'report',
|
||||||
|
summary: 'Subagent delivery is suspected to have failed after crossing SLA.',
|
||||||
|
detail: {
|
||||||
|
runId: 'fixture-run-report-suspect',
|
||||||
|
childSessionKey: 'session:report-suspect',
|
||||||
|
status: 'suspect_delivery_failure',
|
||||||
|
recoveryDecision: null,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('owner-visible reporting marks done_but_not_forwarded plus fetch_history as reportable with minimal payload', () => {
|
||||||
|
const { payload } = runFixture({
|
||||||
|
runId: 'fixture-run-report-fetch-history',
|
||||||
|
childSessionKey: 'session:report-fetch-history',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:05:00.000Z',
|
||||||
|
childRunStatus: 'done',
|
||||||
|
recoveryAttemptCount: 0,
|
||||||
|
}, 'report-fetch-history.json');
|
||||||
|
|
||||||
|
assert.deepEqual(payload.result.reporting, {
|
||||||
|
ownerVisible: true,
|
||||||
|
category: 'done_but_not_forwarded',
|
||||||
|
decision: 'fetch_history',
|
||||||
|
summary: 'Child run is done but no forwarded completion receipt is confirmed yet.',
|
||||||
|
detail: {
|
||||||
|
runId: 'fixture-run-report-fetch-history',
|
||||||
|
childSessionKey: 'session:report-fetch-history',
|
||||||
|
status: 'done_but_not_forwarded',
|
||||||
|
recoveryDecision: 'fetch_history',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('owner-visible reporting marks respawn as reportable with minimal payload', () => {
|
||||||
|
const { payload } = runFixture({
|
||||||
|
runId: 'fixture-run-report-respawn',
|
||||||
|
childSessionKey: 'session:report-respawn',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:06:00.000Z',
|
||||||
|
childRunStatus: 'done',
|
||||||
|
recoveryAttemptCount: 1,
|
||||||
|
recoveryAction: 'fetch_history',
|
||||||
|
}, 'report-respawn.json');
|
||||||
|
|
||||||
|
assert.deepEqual(payload.result.reporting, {
|
||||||
|
ownerVisible: true,
|
||||||
|
category: 'done_but_not_forwarded',
|
||||||
|
decision: 'respawn',
|
||||||
|
summary: 'Child run is done but recovery already failed once; respawn is the next conservative step.',
|
||||||
|
detail: {
|
||||||
|
runId: 'fixture-run-report-respawn',
|
||||||
|
childSessionKey: 'session:report-respawn',
|
||||||
|
status: 'done_but_not_forwarded',
|
||||||
|
recoveryDecision: 'respawn',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('owner-visible reporting marks blocked as reportable with minimal payload', () => {
|
||||||
|
const { payload } = runFixture({
|
||||||
|
runId: 'fixture-run-report-blocked',
|
||||||
|
childSessionKey: 'session:report-blocked',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:07:00.000Z',
|
||||||
|
childRunStatus: 'done',
|
||||||
|
recoveryAttemptCount: 2,
|
||||||
|
recoveryAction: 'respawn',
|
||||||
|
}, 'report-blocked.json');
|
||||||
|
|
||||||
|
assert.deepEqual(payload.result.reporting, {
|
||||||
|
ownerVisible: true,
|
||||||
|
category: 'done_but_not_forwarded',
|
||||||
|
decision: 'blocked',
|
||||||
|
summary: 'Child run is still not forwarded after repeated recovery attempts; owner attention is required.',
|
||||||
|
detail: {
|
||||||
|
runId: 'fixture-run-report-blocked',
|
||||||
|
childSessionKey: 'session:report-blocked',
|
||||||
|
status: 'done_but_not_forwarded',
|
||||||
|
recoveryDecision: 'blocked',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scenario matrix: normal completion stays non-owner-visible and carries no recovery action', () => {
|
||||||
|
const { payload } = runFixture({
|
||||||
|
runId: 'fixture-scenario-normal-completion',
|
||||||
|
childSessionKey: 'session:scenario-normal-completion',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:05:00.000Z',
|
||||||
|
completionReceiptAt: '2026-04-24T10:04:00.000Z',
|
||||||
|
forwardedToMain: true,
|
||||||
|
}, 'scenario-normal-completion.json');
|
||||||
|
|
||||||
|
assert.equal(payload.result.status, 'completed');
|
||||||
|
assert.equal(payload.result.recoveryDecision, null);
|
||||||
|
assert.deepEqual(payload.result.reporting, {
|
||||||
|
ownerVisible: false,
|
||||||
|
category: 'completed',
|
||||||
|
decision: 'none',
|
||||||
|
summary: 'Completion receipt is present; no owner-visible report is needed.',
|
||||||
|
detail: {
|
||||||
|
runId: 'fixture-scenario-normal-completion',
|
||||||
|
childSessionKey: 'session:scenario-normal-completion',
|
||||||
|
status: 'completed',
|
||||||
|
recoveryDecision: null,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scenario matrix: slow-but-active stays non-owner-visible before SLA', () => {
|
||||||
|
const { payload } = runFixture({
|
||||||
|
runId: 'fixture-scenario-slow-but-active',
|
||||||
|
childSessionKey: 'session:scenario-slow-but-active',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:09:59.000Z',
|
||||||
|
}, 'scenario-slow-but-active.json');
|
||||||
|
|
||||||
|
assert.equal(payload.result.status, 'active');
|
||||||
|
assert.equal(payload.result.recoveryDecision, null);
|
||||||
|
assert.deepEqual(payload.result.reporting, {
|
||||||
|
ownerVisible: false,
|
||||||
|
category: 'active',
|
||||||
|
decision: 'none',
|
||||||
|
summary: 'Dispatch is still within SLA; no owner-visible report is needed.',
|
||||||
|
detail: {
|
||||||
|
runId: 'fixture-scenario-slow-but-active',
|
||||||
|
childSessionKey: 'session:scenario-slow-but-active',
|
||||||
|
status: 'active',
|
||||||
|
recoveryDecision: null,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scenario matrix: done-but-not-forwarded resolves to fetch_history reporting decision', () => {
|
||||||
|
const { payload } = runFixture({
|
||||||
|
runId: 'fixture-scenario-fetch-history',
|
||||||
|
childSessionKey: 'session:scenario-fetch-history',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:05:00.000Z',
|
||||||
|
childRunStatus: 'done',
|
||||||
|
recoveryAttemptCount: 0,
|
||||||
|
}, 'scenario-fetch-history.json');
|
||||||
|
|
||||||
|
assert.equal(payload.result.status, 'done_but_not_forwarded');
|
||||||
|
assert.equal(payload.result.recoveryDecision, 'fetch_history');
|
||||||
|
assert.equal(payload.result.reporting.ownerVisible, true);
|
||||||
|
assert.equal(payload.result.reporting.decision, 'fetch_history');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scenario matrix: repeated failure escalates to respawn reporting decision', () => {
|
||||||
|
const { payload } = runFixture({
|
||||||
|
runId: 'fixture-scenario-respawn',
|
||||||
|
childSessionKey: 'session:scenario-respawn',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:06:00.000Z',
|
||||||
|
childRunStatus: 'done',
|
||||||
|
recoveryAttemptCount: 1,
|
||||||
|
recoveryAction: 'fetch_history',
|
||||||
|
}, 'scenario-respawn.json');
|
||||||
|
|
||||||
|
assert.equal(payload.result.status, 'done_but_not_forwarded');
|
||||||
|
assert.equal(payload.result.recoveryDecision, 'respawn');
|
||||||
|
assert.equal(payload.result.reporting.ownerVisible, true);
|
||||||
|
assert.equal(payload.result.reporting.decision, 'respawn');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scenario matrix: repeated failure can escalate to blocked reporting decision', () => {
|
||||||
|
const { payload } = runFixture({
|
||||||
|
runId: 'fixture-scenario-blocked',
|
||||||
|
childSessionKey: 'session:scenario-blocked',
|
||||||
|
dispatchAt: '2026-04-24T10:00:00.000Z',
|
||||||
|
expectedBy: '2026-04-24T10:10:00.000Z',
|
||||||
|
currentTime: '2026-04-24T10:07:00.000Z',
|
||||||
|
childRunStatus: 'done',
|
||||||
|
recoveryAttemptCount: 2,
|
||||||
|
recoveryAction: 'respawn',
|
||||||
|
}, 'scenario-blocked.json');
|
||||||
|
|
||||||
|
assert.equal(payload.result.status, 'done_but_not_forwarded');
|
||||||
|
assert.equal(payload.result.recoveryDecision, 'blocked');
|
||||||
|
assert.equal(payload.result.reporting.ownerVisible, true);
|
||||||
|
assert.equal(payload.result.reporting.decision, 'blocked');
|
||||||
|
});
|
||||||
|
|
||||||
test('fixture runner exposes missing-input behavior for future fail-first cases', () => {
|
test('fixture runner exposes missing-input behavior for future fail-first cases', () => {
|
||||||
const runner = createFixtureRunner();
|
const runner = createFixtureRunner();
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-run-done-not-forwarded-blocked",
|
||||||
|
"childSessionKey": "session:done-not-forwarded-blocked",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z",
|
||||||
|
"forwardedToMain": false
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-run-done-not-forwarded-recovery",
|
||||||
|
"childSessionKey": "session:done-not-forwarded-recovery",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z",
|
||||||
|
"forwardedToMain": false
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-run-done-not-forwarded-respawn",
|
||||||
|
"childSessionKey": "session:done-not-forwarded-respawn",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z",
|
||||||
|
"forwardedToMain": false
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-run-report-blocked",
|
||||||
|
"childSessionKey": "session:report-blocked",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z"
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-run-report-fetch-history",
|
||||||
|
"childSessionKey": "session:report-fetch-history",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z"
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-run-report-respawn",
|
||||||
|
"childSessionKey": "session:report-respawn",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z"
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-run-report-suspect",
|
||||||
|
"childSessionKey": "session:report-suspect",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z"
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-scenario-blocked",
|
||||||
|
"childSessionKey": "session:scenario-blocked",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z"
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-scenario-fetch-history",
|
||||||
|
"childSessionKey": "session:scenario-fetch-history",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z"
|
||||||
|
}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-scenario-normal-completion",
|
||||||
|
"childSessionKey": "session:scenario-normal-completion",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z",
|
||||||
|
"completionReceivedAt": "2026-04-24T10:04:00.000Z",
|
||||||
|
"forwardedToMain": true
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-scenario-respawn",
|
||||||
|
"childSessionKey": "session:scenario-respawn",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z"
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"runId": "fixture-scenario-slow-but-active",
|
||||||
|
"childSessionKey": "session:scenario-slow-but-active",
|
||||||
|
"dispatchAt": "2026-04-24T10:00:00.000Z",
|
||||||
|
"expectedBy": "2026-04-24T10:10:00.000Z"
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user