ADR-0105: Resumed session lifecycle and terminal notification¶

• Status: Accepted (2026-07-25; implementation merged)
• Kind: Aspirational (records the target state)
• Implementation-status: implemented (the lifecycle and notification behavior specified here is on main)
• Area: persistence-state
• Date: 2026-07-24
• Relations: extends ADR-0095

Context¶

li agent -r <branch_id> reopens an existing branch and runs another turn on it. Resume is how re-review rounds work: a second, third or fourth pass over the same PR reuses the branch so the reviewer keeps its context.

A background leg is expected to announce its own completion. li agent --notify "<cmd> --status {status}" registers a terminal callback bound to the run's session entity; when that session reaches a terminal status the callback fires and executes the command. For lionmcp-submitted jobs that command is lionmcp/notify.py, which does two things: stamps the job record terminal (jobs.mark_terminal) and delivers an inbox notice to the submitting seat.

Resume legs do neither. The leg finishes, and nothing announces it.

P1 — A resumed leg's terminal callback never fires. setup_agent_persist (lionagi/cli/_runs.py) creates a session row with "status": "running" on the new-branch path. On the resume path it looks up existing_branch, adopts its session_id, and never touches the session's status. That session was already terminalized to completed by the leg that created it. At teardown the resumed leg calls update_status(session, to_status="completed"), and the emit in lionagi/state/lifecycle/service.py is gated on the status actually changing:

if (
    transition_id is not None
    and previous_status != command.to_status
    and command.entity_type in EXECUTION_ENTITY_KINDS
    and command.to_status in policy.terminal_statuses
):
    envelope = _build_terminal_envelope(...)
    await self._terminal_callbacks.emit(envelope)

completed → completed is not a change, so no envelope is built and no callback runs. The gate is deliberate and documented in ADR-0095 ("a same-status reason append is not a new event"); it is not the defect.

P2 — The failure is silent and looks like a hang. A detached leg that ends without notifying is indistinguishable from one still running. The caller waits on an inbox that will never receive anything. This is precisely the state the notify hook exists to prevent, and it is worst in re-review rounds, which is where resume legs concentrate.

P3 — The job record is left non-terminal, from the same cause. lionmcp's notify.py is what calls jobs.mark_terminal, and it is executed by the terminal callback. No callback means the job record keeps status: running and a null finished_at. lionmcp later observes the pid is gone with no terminal record and reports exited (jobs.py, "pid gone, no terminal record captured"). That is honest reporting of someone else's silence, not a second bug.

Measured evidence¶

Across 18 consecutive lionmcp job records (2026-07-24), separation was total with no exceptions in either direction:

--notify was present in argv for the failing legs, so this is not a submit-side omission.

The row write itself succeeds. Session rows for the two resume legs carry started_at from the original leg and ended_at stamped to the resumed leg's own end time:

So the UPDATE lands and only the callback dispatch is skipped. An earlier hypothesis that the write was rejected by a CAS miss or the ADR-0035 terminal guard is wrong, and it matters: a fix aimed at the write path would have targeted working code.

Out of scope. The terminal-callback registry, envelope shape and delivery semantics (ADR-0095 owns them). The ADR-0035 terminal guard and CAS machinery. The defer_terminal auto-resume path, which already has a correct answer and is only described here for its interaction. Whether lionmcp should stamp from_actor at the source (separate, tracked outside this ADR).

Decision¶

D1 — A resumed session returns to running on reopen¶

When setup_agent_persist adopts an existing branch's session, it transitions that session back to running before the leg executes, through the same update_status path every other lifecycle write uses rather than a direct UPDATE.

Contract (lionagi/cli/_runs.py, resume path of setup_agent_persist):

existing_branch = await db.get_branch(branch_id)
if existing_branch:
    session_id = existing_branch["session_id"]
    existing_session = await db.get_session(session_id)
    # A session that is no longer there is not an error: maintenance removes old
    # terminal sessions, and there is then nothing to resume into. The leg records
    # itself under a new session rather than reading a row that no longer exists.
    # A reopened session is running again: its closing transition must be a real
    # status change, or the terminal callback that announces this leg never fires.
    if existing_session is None:
        existing_branch = None  # take the new-session path
    elif existing_session["status"] in SESSION_TERMINAL_STATUSES:
        await db.update_status(
            "session",
            session_id,
            new_status="running",
            reason_code=SessionReasons.REOPENED_BY_RESUME,
            reason_summary="branch resumed by a new leg",
            expected_statuses=SESSION_TERMINAL_STATUSES,
            extra_fields={
                "ended_at": None,
                "node_metadata": json.dumps({**node_metadata, **current_pid_markers()}),
            },
            override=True,
            override_actor="cli.resume",
            override_justification="branch resumed by a new leg; the session is executing again",
        )

update_status is the ADR-0035 machinery in lionagi/state/db.py; new_status and reason_code are required, and expected_statuses supplies the CAS guard. SESSION_TERMINAL_STATUSES is derived from the lifecycle policy registry rather than hand-maintained, and currently holds aborted, cancelled, completed, completed_empty, failed, timed_out.

The override is load-bearing and was missing from this ADR's first draft. The session policy declares exactly one edge, running → {terminal}, and the transition service rejects any move out of a terminal status unless an override is supplied (lionagi/state/lifecycle/policy.py, whose own comment states "No exit from a terminal status without override"; enforced at lionagi/state/lifecycle/service.py in the previous_status in policy.terminal_statuses branch). Without it this write does not land. Measured rather than reasoned about: StateDB.update_status raises TransitionRejectedError("refusing to write 'running' without override=True"), and setup_agent_persist catches Exception broadly, logs a warning, and returns None, which disables persistence for the entire run. So the version of D1 without an override does not merely fail to fix the missing notice; it costs every resumed leg all of its state persistence, behind one warning line. That consequence is worse than the defect and is the reason this correction is stated at the contract rather than left to implementation.

reason_code must also be a code registered in lionagi/state/reasons.py; an unregistered string raises before the transition is attempted. This ADR adds SessionReasons.REOPENED_BY_RESUME ("session.reopened.by_resume"), which satisfies the session policy's reason_prefixes.

Override rather than a new declared edge, deliberately. Declaring terminal → running in session_edges would legalize terminal-exit for every session writer in the system, and the finality of a terminal session is the property the reaper, the teardown guard, and li wait all rest on. The override keeps the exception scoped to the one caller that has earned it and, because override requires a non-empty actor and justification and emits a status_transition_override admin event, it makes each reopening attributable. A reopened session is a real event and should leave a record saying who reopened it and why; the declared-edge version would leave none.

ended_at is already in the session policy's patch_fields. node_metadata is added to it, and to EXTRA_STATUS_WRITE_FIELDS_BY_ENTITY_TYPE in lionagi/state/db.py, so that the process markers move in the same transaction as the status.

That is not a convenience. The sweeps select rows by status and then ask the recorded process markers whether the row is still alive, so status and markers are read together and have to agree at every instant a sweep could look. Splitting the two writes leaves a window in which the row reads running while still carrying the markers of the leg that already exited, which is precisely the shape li kill --all-stale cancels (lionagi/cli/kill.py: select status='running', age past the threshold, recorded PID not alive). The markers must also not be written on a lost race, because the row then belongs to a different leg and our markers would make that leg's liveness answer for our process. Both requirements are satisfied by the one guarded write: it either wins and installs status and markers together, or it loses and touches nothing. This is what the same-row allowlist exists for — its own comment says it keeps a caller from splitting a status change and a dependent column into two transactions.

Markers, not liveness in general: a terminal session is never checked for liveness, so stale markers on one were harmless. A running one is checked, which is what makes them load-bearing from the reopen onward.

Exact semantics:

• Session already terminal (any of the six above) → transitioned to running; the leg's closing transition is then a genuine change and emits normally.
• Session already running → left alone. This is a resume racing a live leg on the same branch; reopening is a no-op and the guard makes it observable rather than clobbering.
• Transition returns conflict (another process moved the row first) → the leg proceeds. A resumed leg must not fail because its bookkeeping lost a race; the cost is one missed notice, which is the status quo, not a regression.
• to_status="running" is not in policy.terminal_statuses, so reopening never itself emits a terminal envelope.

Why this way: the defect is that a session's status stops describing the session. Returning it to running restores the invariant that a session marked terminal is not currently executing, and every downstream consequence (the callback firing, the job record closing) follows from that invariant holding rather than from special handling.

D2 — The change-gate is not relaxed¶

previous_status != command.to_status stays exactly as it is.

Why: it is load-bearing idempotency for every caller, not a notify implementation detail. ADR-0095 introduced it so that a same-status reason append is not a new event. Relaxing it so terminal writes always emit would re-fire completion notices fleet-wide on any repeated terminal write — reaper sweeps, status refreshes, retried teardowns — turning a missing-notice defect into a duplicate-notice one affecting every consumer instead of one path. The narrow fix at the resume path costs one guarded write; the broad fix at the gate costs correctness everywhere.

D3 — started_at is preserved, ended_at is cleared on reopen¶

Reopening sets ended_at = NULL and leaves started_at untouched.

Semantics: started_at is when this session began, and resume continues a session rather than starting one, so it keeps the original value. ended_at must be null while running, because "has an end time" and "is still executing" cannot both be true — that is the same one-value-two-meanings failure this ADR is fixing, and leaving a stale ended_at on a running session would reintroduce it one column over.

Today's behaviour is the incoherent middle: started_at original, ended_at from whichever leg last closed, status terminal throughout. A reader cannot tell from the row whether the session ran once or four times.

Consequence a consumer must know: a session's ended_at may move later, and its status may go completed → running → completed. The first of those was already true in production, since resume legs were mutating ended_at on terminal sessions before this ADR. The second is genuinely new, so it is not asserted here on the strength of the first. The enumeration is in "Consumers of session finality" below.

D4 — The job record needs no separate repair¶

lionmcp's stuck status: running record is downstream of D1, not a second defect. notify.py performs both the record stamp and the notice delivery, and it is executed by the callback D1 restores. When the callback fires, both happen.

This is recorded as a decision rather than left implicit because the two symptoms travelling together is suggestive but not proof of one cause, and the alternative — having lionmcp reconcile pid-gone jobs itself — is a real option that is being declined. Declined because it would paper over a missing notification with an inferred one: lionmcp would report completed for a leg it never heard from, which is a worse failure than reporting exited honestly.

Consumers of session finality¶

D1 makes a session's status go terminal → running → terminal, which no consumer has seen before. Every live reader of session terminal-status and ended_at was enumerated by grep across lionagi/ and read at its call site, rather than reasoned about from the decision. Results:

No consumer breaks. Four findings need stating; two of them needed a fix.

The teardown terminal-skip guard starts working. cli/_runs.py skips its status write when the session was already terminal at teardown start, logging that the earlier terminal record is protected. Before D1 that meant a resumed leg's outcome was silently dropped. After D1 the session is running at teardown, so the write lands and the resume leg's outcome is recorded. This is correct and is a second defect D1 fixes, but it is a real semantic change: the resume leg's terminal status now replaces the original leg's on the row. The earlier one is not lost — every applied transition appends to status_transitions — but the row itself shows the latest leg.

A resumed session becomes eligible for phantom reaping. Both sweeps above select status = 'running', so a terminal session is invisible to them today. A reopened one is not. If a resume leg dies without writing a terminal status and the session then sits stale for PHANTOM_STALE_HOURS, the reaper transitions it to failed, and a session that had previously completed now reads failed. This is judged acceptable rather than handled: the session genuinely was re-run and the re-run genuinely died, running → failed is a declared edge that applies normally, and the earlier completed survives in status_transitions. It is recorded here because the derived row no longer shows it, and a reader of the row alone would draw the wrong conclusion about the first leg.

li state doctor was measuring the wrong thing, and D1 exposed it. The command sweeps sessions "stuck at status='running'" whose started_at is older than a threshold, and its own help text promised that "an actively-running CLI process is left alone". Age answers how long since this session first started, which equals how long this process has been running only for a session that ran once — the case that used to be the only one. A resumed session keeps its original started_at (D3) while its process is new, so a live resumed leg became sweepable from the moment D1 made it running, and at the default threshold a session first started more than a day ago is sweepable immediately and permanently, for as long as the leg runs.

This one is fixed rather than disclosed, because the promise was already written down and D1 would have falsified it. _doctor now requires both conditions: the age past the threshold and the recorded process gone, the same veto li kill --all-stale already applied. Sessions with no recorded PID are unchanged, so a crash that never wrote markers is still reapable, and the help text now states the predicate it enforces.

The general shape, worth naming because two consumers hit it in one change: a sweep that selects on status and thresholds on started_at is asking "is this stuck" and answering "is this old". Those were the same question while a session had exactly one leg.

The prune and a resume can now collide, and it had to be settled in both directions. prune_old_data removes terminal sessions past a retention window. It clears their associations and transition history first, then deletes the row, all inside one transaction. Before D1 a session selected as terminal stayed terminal for the length of that pass; after D1 it can be reopened partway through.

A resume landing mid-prune leaves the session with its row and without the history already cleared for it. Putting the terminal predicate on each destructive statement is necessary and not sufficient: every statement is then individually correct and the sequence can still land half-applied, because another transaction commits between two of them. The prune now writes to each candidate row before reading the status that decides the batch — the write is what takes the lock, on postgresql by holding the rows and on sqlite by escalating the transaction to a write. A resume that arrives before the lock is dropped at the re-read, which is a quiet zero-count pass. A post-condition after the delete re-reads the candidates and abandons the transaction if any survived, so the pass applies whole or not at all.

A prune landing mid-resume is the same race from the other side, and the lock decides it in the prune's favour: a resume already waiting is released to find its session gone. Two paths turned that into a leg with no persistence at all — the status write reports a missing row by raising, and persistence read the branch and its session as two separate reads and then dereferenced a row that no longer existed. Both now treat a missing session as "nothing to resume into": the leg records itself under a new session, exactly as a branch nobody has seen before does. The old session's history is gone either way, since removing it is what the prune was for; what is preserved is the current leg.

Alternatives considered¶

Create a new session per resume leg, linked to the branch. Each leg would own exactly one session, statuses would never resurrect, and started_at/ended_at would need no special rules — the cleanest model on paper. Rejected because session identity is the unit resume exists to preserve: the branch's message progression and artifact contract hang off the session, and find_branch resolution assumes one session per branch lineage. Changing that is a much larger migration for a defect whose fix is one guarded write. Worth revisiting if session-per-invocation is ever wanted for other reasons; this ADR does not foreclose it.

Emit a synthetic terminal envelope from the resume teardown. Bypass the gate by constructing an envelope directly when the leg knows it resumed. Rejected: it duplicates the emit path outside the service that owns it, so the two can drift, and it makes "was this a real transition" unanswerable from the transition log — the row would show no change while a consumer saw an event.

Register the notify callback on the invocation entity instead of the session. Invocations are per-leg, so a resumed leg would have its own fresh entity. Rejected: cli/agent.py documents why this is session-scoped — invocation records are finalized externally and would never fire. This trades a missing notice for a different missing notice.

Give resume and prune a shared claim or lease over a session. The prune would refuse to delete a row an in-flight resume had claimed, so neither side could lose. Rejected: a lease is only as good as its expiry, and the process that holds one is exactly the process that can die without releasing it — the failure it introduces (a session nothing will ever prune because a dead leg holds its claim) is quieter than the one it prevents. Letting the prune win and having the resume start a new session reaches the same end state without a second liveness problem to get right. The lock is held for one transaction by the database itself, which needs no expiry.

Have --notify fall back to running the command unconditionally at teardown. Simple and always fires. Rejected: it abandons the entity model entirely, would fire for deferred auto-resume legs that deliberately suppress their terminal (defer_terminal=will_auto_resume), and reintroduces double-delivery whenever the callback does fire.

Consequences¶

Easier: a detached resume leg announces itself like any other leg, so re-review rounds stop requiring polling, and the job record closes on its own. Session status becomes trustworthy as a description of whether the session is executing.

Harder: session status is no longer monotonic. A consumer that latched "terminal means finished forever" must tolerate reopening. The enumeration above found no such consumer, and the three that change behaviour do so in ways stated there. Reopening is also the system's only sanctioned exit from a terminal status, so it carries an override audit row rather than passing as an ordinary write.

Also harder, and the part most likely to bite a future change: a session's row now describes the current leg, while its start time still describes the session. Anything that derives "how long has this been running" from started_at is correct only for a session that ran once. Two sweeps made that assumption, both found here; a third written later would make it again, and the row gives no hint that the two ever differ.

New failure mode: a crashed resume leg leaves the session running with a null ended_at where previously it would have kept a stale terminal status. That is better for a reader and worse for anything counting running sessions, which is what the orphan-recovery path in ADR-0095 exists to reconcile.

Reversal cost: D1 is one guarded write and its regression test — cheap to revert. D3 is coupled to D1 and reverts with it. D2 changes nothing, so it has no reversal cost. D4 is a decision not to build something.

Verification¶

The regression must fail without D1 and must assert delivery rather than registration: run a resume leg with a --notify command that writes a file, and assert the file exists and the session's transition log records a real running → completed change. Asserting only that a callback was registered would pass against today's broken behaviour, since registration was never the problem.

A second regression covers the reopen write itself: assert that reopening a terminal session applies rather than returning rejected. Without the override this write is refused, and the notice regression above would fail for a reason unrelated to what it is testing — a silent rejection reads from the outside exactly like the defect. Pin them separately so a future change to the override path fails at the write, not at the notice three steps downstream.

A third regression covers the markers, and it takes two cases because there are two ways to get them wrong. Reopening must leave the row carrying this process's markers, merged with whatever unrelated metadata was already there; and it must install them through the status write rather than a second one, which is pinned by making any separate write fail the test. A version that writes them separately passes the first case and fails the second, which is the point: the window is the defect, not the value.

A fourth covers the sweep: a session past the age threshold whose recorded process is alive is left alone, and one whose process is gone is swept, in the same run. Asserting only the first would pass against a sweep that had stopped working. A session whose recorded number is alive but belongs to a different process is swept too, since a live PID alone would otherwise protect a stuck row for as long as that number stayed in use.

A fifth pins the shape of what reopening reads. node_metadata has held values that are not JSON objects, and reading one as an object raises: for a string that is not JSON at all, and for a JSON scalar or list when the object is merged. The resume must survive all of them, because the caller catches broadly and turns any raise here into a run with no state persistence at all — a far larger failure than the field that caused it.

A sixth covers the prune, from both sides, and each side needs more than one case. On the prune side: the reopen is driven past each destructive statement in turn, and every case asserts the session keeps its row, its transition history and its associations, so a guard that covers one statement and not the next fails. A separate case pins the order — the first statement the prune issues against sessions must be a write, since reading first leaves the decision resting on a value another transaction may change before the writes land. On the resume side: a session deleted between the read and the write must not raise, and a session deleted between the branch read and the session read must leave the leg with persistence under a new session rather than none at all. Both of those fail on the untreated code at the exact line that breaks, one raising and one dereferencing None.