Skip to main content
Chatticus

Pi harness

Design note for Kanbus story chatticus-6c4d24 (epic chatticus-9f09c5). Measured 2026-10-04 against @earendil-works/pi-durable 1.0.2 (Pi monorepo 2003871). The code is a throwaway spike in spikes/pi-durable-6c4d24. No product code depends on it.

What we want

One durable engine for a bot's conversation that fits the serverless shape in Design challenges and Messaging: no always-on process, a stream scoped to one turn, and a turn that can begin on a Lambda and continue on the computer.

The plan under test:

What pi-durable is

A durable agent harness from the Pi framework. Everything is a committed record before it is shown:

It talks to storage only through a 16-method Storage interface: commit(writes), mintId(), and reads. The Session serializes commits and assumes one process owns a storage at a time. It has no cross-process lock. That is the part the fence supplies.

The storage: S3 data, DynamoDB index

src/indexed-storage.ts. The first version (src/dynamodb-storage.ts, kept as the cost baseline) stored every record body in DynamoDB.

S3: one object per commit

s3://<bucket>/conversations/<storage>/commits/<seq:012>-<fence:08>.json
  { "seq": 12, "fence": 3, "token": "<uuid>", "writes": [ ...every StorageWrite of the commit... ] }

DynamoDB: the index

One table, pk/sk, plus three local secondary indexes (l1, l2, l3). Local indexes support strongly consistent reads, which the storage contract requires. One storage is one partition, pk = PI#<tenant>#<bot>#<channel>. Every record item carries a pointer c (commit seq), f (fence) and p (position of the write in the commit object).

sk Holds (besides the pointer) Index keys
META commit seq, ID high-water mark, last commit token --
OWNER current fence --
R#<id> conversation the whole record (ids only: parent, owner) l1 C#<id>, l2 CT#<ownerTask>#<id>, l3 CC#<ownerConversation>#<id>
R#<id> entry conversation id l1 E#<conversation>#<id>; with a head also l2 H#<conversation>#<id>
R#<id> task conversation, kind, abort and background flags l1 T#<id>, l2 TS#<status>#<id>
R#<id> submission conversation id l1 S#<id>, l2 SS#<status>#<id>, l3 SR#<conversation>#<hash(requestId)>#<id>
R#<id> document lifecycle record (kind, key, scope, created, retired), latest version and revision seq l1 D#<hash(scope)>#<id>, l2 DA#<hash(address)>#<id>
V#<document>#<seq> base or delta flag --

The commit protocol

  1. Prepare: read what validation needs from the index (document records and address occupancy; 1.2 to 2.1 read round trips per commit, measured).
  2. PutObject the commit object with If-None-Match: *, so a key is never overwritten. The fence in the key means only this owner ever writes it. If the key already exists with this commit's token, it is this commit's own earlier attempt. If it has another token, the storage reads META: when META.seq >= seq the object is committed data and the commit fails with OwnershipLost without deleting anything; otherwise it is this owner's orphan from a commit whose transaction failed, and it is deleted and replaced. An S3 409 ConditionalRequestConflict means a concurrent write to the same key is still in flight, which with the fence in the key can only be this owner's own earlier attempt; it is retried with backoff and resolves to a success or a same-token 412.
  3. One TransactWriteItems with the index items, the META update conditioned on seq = <last seq this owner saw> (which also stores the token), and, when fenced, a ConditionCheck on OWNER.fence.
  4. The transaction carries the token as ClientRequestToken. The storage itself retries throttling, timeouts and TransactionConflict with the same token. If a retry finds the META condition failed, it re-reads META: when the stored token is its own, the first attempt landed and only the response was lost, so the commit counts as done. The check has three outcomes: committed, not committed, or unknown (the read itself failed). Unknown throws CommitOutcomeUnknown, an OwnershipLost subtype that is fatal to the Session, and the storage does not delete the object: it may be committed, so it is left for the next owner to find and for the sweeper.
  5. The SDK client uses maxAttempts: 1. In the first version that was load-bearing: an SDK retry after a lost response would have come back as a false "another owner committed". Production must keep SDK retries off for TransactWriteItems (or make them reuse the token) and leave retrying to the storage as above.

Readers only follow the index. An object whose transaction never landed is invisible.

Failure cases

Failure What happens Measured
Transaction rejected (ID collision, size) StorageRejected; the storage deletes its own object Object gone (results/orphans.json)
Fence moved before the transaction OwnershipLost; the object is deleted Object gone
Crash between PutObject and the transaction The object stays as an orphan. The next commit by the same owner at that seq replaces it; another owner uses a different key Replaced and committed
PutObject fails Nothing committed; pi-durable sees a plain error and poisons the Session --
Transaction outcome unknown after retries The token check decides; if still unknown, CommitOutcomeUnknown poisons the Session and the object is kept. A landed commit is then found by the next owner Not reproducible on moto
Object deleted or unreadable after commit Reads of those records fail. This must not happen: no lifecycle rule may expire commits/ --

Orphan rule. The sweeper reads META and OWNER first, then checks references, so the values it acts on can only be older than the truth in the safe direction (both only rise). It may delete commits/<seq>-<fence>.json only when no index item points at (seq, fence) and either of these holds:

  1. META.seq >= seq. The commit's META condition needs META.seq = seq - 1, so a late commit of this seq can never succeed.
  2. OWNER.fence > fence. Fences only rise, so the commit's ConditionCheck on OWNER.fence = fence can never succeed.

There is no time-based clause. An object with seq > META.seq whose fence is still the current OWNER.fence may be an in-flight commit of a live (or momentarily stalled) owner and is kept; it becomes deletable by clause 1 or 2 as soon as the sequence passes it or a new owner raises the fence. results/orphans.json shows both clauses and the kept case. An S3 lifecycle rule cannot express "unreferenced", so the sweeper is a small scheduled job. Orphans arise only from a crash between steps 2 and 3, so it can be rare.

Ownership and the fence

Results

All runs used moto 5.2.3 for both DynamoDB and S3 (path-style, dummy credentials). Amazon DynamoDB Local was requested, but its container never answered on port 8000 in this session (the pull was stopped after 30 minutes). moto honors If-None-Match: * on PutObject (412 on the second write, results/s3-conditional.json).

Neither moto nor DynamoDB Local exercises TransactionConflict, throttling, SDK retries or per-partition throughput limits. The retry, idempotency-token and ownership-loss paths are therefore untested against real behavior. Only a run against a real table covers them.

Test Result
pi-durable storage conformance suite 23 of 23 pass for the production S3-plus-index storage in conversation/src/storage/indexed-storage.ts, in CI (job conversation) against moto. The spike's copy failed 10 cases on an index-name bug that the production copy fixes.
Owner 1 answers, owner 2 reopens and continues Pass. Owner 2 answered "teal" to "what is my favourite colour?"
Resubmit the same requestId The original submission id comes back, already done. 0 commits
Stale owner commits after a newer fence Rejected with OwnershipLost (Owner fence moved). Stale fence claim rejected
Fence raised while an owner is mid tool call The stale owner's next commit failed with OwnershipLost; it then did nothing more. The next owner reran the replay-safe tool and finished
Lambda-to-computer handoff Pass (below)
Kill mid model stream Pass. The partial (262 chars) became a pi.assistant entry with stopReason: aborted, then the same messages were resent and answered
Kill mid tool call, replay: "safe" The tool reran on the new owner: started twice, finished once. One result
Kill mid tool call, replay: "unsafe" No rerun. The model got an interrupted error result: "Tool run_terminal was interrupted and may have partially run"
Attributed message from another participant Pass. "Friday at 3pm, according to Bea."
Approval-style gate Pass. A beforeTool hook returned { block }. The model got "Tool call blocked: needs approval: ..." and said so
Non-owner message during a busy turn Pass through a mailbox (below). A direct commit from a second handle was rejected (OwnershipLost: Commit sequence moved)
Orphans A rejected or fenced-out commit leaves no object; a crashed attempt's object is replaced (results/orphans.json)
101 transaction items Rejected before sending (both storages). 410 KB entry and 4.2 MB commit: committed on S3-plus-index, rejected on DynamoDB-only

Per turn (gpt-5-nano, S3-plus-index storage)

Turn Commits Max items per commit Bytes written (index + objects) Stream batches Stream bytes
First turn, new conversation 8 13 20 KB -- --
Plain answer 7 9 15 to 16 KB 5 10 KB
One tool call (3 s tool, streamed output) 32 9 62 KB 19 34 KB
Tool call blocked 45 9 91 KB 27 52 KB
Tool call plus a mid-turn steer 21 10 49 KB 13 35 KB

Reopen

A new owner's Harness.open plus root() took 7 DynamoDB reads and 0 S3 GETs on both storages (results/cost.json): the Session loads lazily. Reading the whole transcript after two turns took 2 index queries and 9 S3 GETs (one per commit that holds an entry), fetched in parallel. The moto timings (43 ms reopen, 28 ms transcript) say nothing about AWS.

The cost that grows is not reopen but the first model request of each turn: it needs every visible entry, so a fresh owner fetches one object per commit that holds an entry, about 4 to 5 per turn of history. A periodic snapshot object (conversations/<id>/snapshots/<seq>.json holding the visible entries and current documents) would make that one GET plus the recent commits. It is not implemented in the spike; it should be before production. Request cost is flat per turn only with that snapshot object. Without it (the spike as built), a fresh owner's first model request fetches about 4 to 5 commit objects per turn of history, so GETs and first-token latency grow linearly with conversation length. The cost projections below count only the spike's two-turn runs and omit these growing GETs.

Cost: DynamoDB-only versus S3 data with a DynamoDB index

Prices, us-east-1, checked 2026-10-04 on DynamoDB on-demand pricing and S3 pricing:

Request units are estimated by the spike's meter from each request (moto does not report consumed capacity): 2 WRU per started KB per transaction item plus 1 per local index the item lands in, and 1 RRU per started 4 KB of strongly consistent reads. The meter undercounts index writes that move an item between index keys (a delete plus a put). One run each, same prompts (scripts/cost.ts, results/cost.json):

Turn Storage Commits WRU RRU S3 PUT Requests per 1,000 turns
Plain answer DynamoDB-only 7 120 46 0 $0.081
Plain answer S3 + index 7 99 42 7 $0.102
One tool call DynamoDB-only 31 479 182 0 $0.322
One tool call S3 + index 30 382 141 30 $0.406

Per commit, the S3-plus-index storage costs about $0.0000135 to $0.0000146 and the DynamoDB-only storage about $0.0000104 to $0.0000115: the index version costs about 25 to 30% more in requests. The reason: at these payload sizes (a few KB per commit) WRUs are set by the per-item minimum, not by the payload, so moving the payload out of DynamoDB saves only 2 to 3 WRU per commit, while each commit adds one S3 PUT ($0.000005, the price of 8 WRU).

Resident data after setup and both turns DynamoDB-only S3 + index
DynamoDB billable bytes (items + 100 bytes each, times local indexes) 62 KB 25 KB
S3 bytes 0 53 KB (38 objects)
Per 1,000 turns per month (setup spread over the two turns) $0.0073 $0.0035

Storage per GB-month of conversation: DynamoDB-only pays $0.25 on every billable byte, including the copies in three local indexes; the index version pays $0.25 on a third as many DynamoDB bytes and $0.023 on the rest.

So:

AWS run

Run on 2026-10-04 against real AWS in the Chatticus development account (us-east-1, the ThinTurn stack's region), approved by the human for this run only. A throwaway CDK app (spikes/pi-durable-6c4d24/aws/, stack ChatticusPiDurableSpike) deployed one on-demand table with the spike's index schema, one S3 bucket (block public access, SSL enforced), and one Node 22 arm64 Lambda (1024 MB, no VPC, 120 s timeout, the owner code bundled with esbuild). The Lambda read the existing development OpenAI parameter and ran gpt-5-nano; SDK maxAttempts was 1. The stack was destroyed afterwards (describe-stacks, head-bucket and describe-table all report not found). Raw invoke records are in results/aws/raw/, the roll-up in results/aws/summary.json. Conformance was not re-run on AWS.

Finding before any measurement: real DynamoDB rejected the table because local secondary index names must be at least 3 characters; moto accepted l1, l2 and l3. The indexes are now named l1-index and so on (the attribute names are unchanged).

Measurement moto (local) AWS (Lambda to DynamoDB and S3, us-east-1)
Cold start, init n/a 421 to 597 ms (3 runs)
Cold plain turn, handler duration n/a 3.79 s and 3.80 s (init 0.57 and 0.60 s; wall 5.0 s)
Cold tool turn, handler duration n/a 11.3 s (init 0.42 s)
SSM key fetch (cold only) n/a 145 to 156 ms
Open (claim fence, load, harness, root) 43 ms 138 to 156 ms warm (p50), 347 to 407 ms cold
Warm plain turn, duration n/a p50 3.2 s (2.5 to 4.7 s, 10 runs), 8 commits
Warm tool turn, duration n/a 7.4 to 10.6 s (3 runs), 27 to 29 commits (the tool was instant here)
Commit latency, p50 / p95 31 to 75 ms / 54 to 787 ms 64 ms / 103 ms (plain), 60 ms / 84 ms (tool); 164 commits pooled; max 131 ms
S3 PutObject p50 n/a 29 ms (p95 about 44 ms)
TransactWriteItems p50 n/a 23 ms (p95 about 35 to 39 ms)
Max Lambda memory n/a 160 to 161 MB of 1024 MB

Model time dominates every turn: a plain turn is about 8 commits of about 64 ms (roughly 0.5 s of the 3 s). Commit latency is a S3 PUT plus a transaction plus about 10 ms of owner work, and it stayed well below the 100 ms partial window at p95. Cold-start cost (about 0.5 s init plus about 0.2 s SSM and 0.25 s extra open) is small next to the model call.

Consumed capacity versus the meter (ReturnConsumedCapacity: TOTAL on every call, one fresh conversation per invoke, per turn without the open):

Turn Commits WRU actual (meter) RRU actual (meter) S3 PUT S3 GET
Plain, mean of 10 8 102 (99) 42 (42) 7 0
Tool, mean of 3 28 359 (346) 127 (127) 27 0

The meter's read estimate is exact and its write estimate is low by 3 to 4% (index-key moves, as predicted). Opening a new conversation additionally costs 39 WRU and 22 RRU. Updated request cost per 1,000 turns with actual capacity at $0.625/M WRU, $0.125/M RRU and $0.005 per 1,000 PUTs: plain $0.104 (was $0.102), tool $0.375 for 28 commits (was $0.406 for 30 commits; about $0.0000134 per commit, unchanged). The cost table stands; the meter can be trusted to within 5%.

Lost response (results/aws/raw/lost-*.json). A request handler hook let the third TransactWriteItems of a plain turn commit and then raised an error instead of returning the response, with maxAttempts: 1.

Forced conflict (results/aws/raw/conflict.json). 20 trials: two handles on the same conversation at the same sequence committed concurrently. In all 20, exactly one committed and the other raised OwnershipLost ("Commit sequence moved"); the final sequence was always 2. Real TransactionConflict cancellations happened in 13 of 20 trials (the storage retried, then saw the failed sequence condition); in the other 7 the loser was rejected by the sequence condition directly. moto never produced either behavior.

Bundle. The Lambda bundle with the AWS SDK clients included is 4.93 MB unminified and 2.05 MB minified; with @aws-sdk/* left to the Lambda runtime it is 737 KB minified. The pi-ai OpenAI provider did not drag in the other providers (subpath imports tree-shake).

Cost of the run: well under $1 (a few cents of Lambda, a few thousand requests, DynamoDB on-demand, minutes of storage); the stack's bucket and table were deleted, and the leftover log groups were deleted by hand.

Long-term storage

Decision: payloads stay in S3 and DynamoDB is only the index. The concern is not the cost of one turn but storage that compounds: every turn ever taken stays resident and billed every month. DynamoDB charges $0.25 per GB-month for every billable byte, including each local-index copy; S3 Standard charges $0.023.

Projection for one household

100 plain turns and 20 tool turns a day. Bytes per turn are the measured resident growth from scripts/cost.ts (results/cost.json, one run each, first-turn setup subtracted):

Per turn DynamoDB-only S3 + index: DynamoDB S3 + index: S3
Plain 21,144 - 8,632 = 12,512 B 12,848 - 8,357 = 4,491 B 12,965 - 1,250 = 11,715 B
Tool call (31 and 30 commits) 62,294 - 21,144 = 41,150 B 24,941 - 12,848 = 12,093 B 52,759 - 12,965 = 39,794 B
Per day (100 plain + 20 tool) 1,251,200 + 823,000 = 2,074,200 B 449,100 + 241,860 = 690,960 B 1,171,500 + 795,880 = 1,967,380 B
Per 30-day month, GiB (/ 2^30) 0.0580 0.0193 0.0550

The stored data after N months is N times the monthly growth. The bill for month N is that stock times the price. The cumulative bill is the price times the monthly growth times N(N+1)/2. Prices as cited above; the 25 GB DynamoDB free tier is per account and ignored.

After Stored: DynamoDB-only Stored: S3 + index Bill that month: DynamoDB-only Bill that month: S3 + index Cumulative: DynamoDB-only Cumulative: S3 + index
1 month 0.058 GiB 0.019 + 0.055 GiB 0.058 x 0.25 = $0.014 0.019 x 0.25 + 0.055 x 0.023 = $0.006 $0.014 $0.006
12 months 0.695 GiB 0.232 + 0.660 GiB $0.174 $0.073 0.25 x 0.0580 x 78 = $1.13 (0.25 x 0.0193 + 0.023 x 0.0550) x 78 = $0.48
36 months 2.086 GiB 0.695 + 1.979 GiB $0.522 $0.219 0.25 x 0.0580 x 666 = $9.65 (0.25 x 0.0193 + 0.023 x 0.0550) x 666 = $4.06

Requests for the same household are about $0.44 a month (DynamoDB-only) and $0.55 (S3 + index), flat over time only with the snapshot object (without it the S3 layout's GETs grow with history, at $0.0000004 each): 100 x $0.0000808 + 20 x $0.000322 a day against 100 x $0.000102 + 20 x $0.000406. The S3 layout's monthly bill (storage plus requests) drops below DynamoDB-only's around month 14 and its cumulative bill around month 26. That crossover comes earlier with any stage-2 item below; with item (2), the DynamoDB part stops growing.

Stage-2 requirements

  1. S3 lifecycle. Commit and segment objects of conversations that have been idle (for example 30 days) move to S3 Intelligent-Tiering or Glacier Instant Retrieval. Both keep millisecond reads, so a returning conversation needs no restore. Never expire commits/ or segments/.

  2. Compaction of idle conversations. When a conversation has been idle and has no live task, a job folds its commit objects into one segment object and replaces its per-record index items with one manifest item, so the DynamoDB footprint per conversation stays roughly constant.

    s3://<bucket>/conversations/<storage>/segments/<throughSeq:012>.json
      { "throughSeq": 4812, "records": { "<id>": <record> }, "revisions": { "<doc>": [<base>, <deltas>...] } }
    
    DynamoDB pk = PI#<storage>, sk = MANIFEST
      { "segment": "segments/000000004812.json", "throughSeq": 4812, "bytes": 1834221,
        "liveTasks": 0, "nextId": 90311, "requestIds": "<bloom or hash set of recent requestIds>" }
    

    Reads resolve through the manifest. An id with an R#<id> item is read as today, from the commit object it points at. Without one, and with id at or below the segment's range, it is read from the segment, which is fetched once and cached. Ordered scans (entries, tasks, submissions, documents) merge the segment's sorted lists with the index queries above throughSeq. The first commit after reopening writes index items again, only for new records. Compaction itself runs as an owner holding the fence: it writes the segment, then in one transaction deletes the folded index items (in batches of under 100) and writes MANIFEST. A crash leaves either the old index or the new manifest, never neither. Folded commit objects are deleted only after the manifest commits.

  3. DynamoDB Standard-IA for the index table. Standard-IA trades lower storage price for higher request price. Evaluate it once compaction exists, because the index's balance of request cost to storage cost decides it.

  4. Fewer commits per turn. Commits dominate request cost: 7 for a plain turn, 30 or more for a tool turn, most of them 100 ms partial and tool-output updates of pi.live. Options: a longer partial throttle in pi-durable, coalescing live updates when nobody is watching the stream, and fewer index items per commit (the pi.live revision and its record update are 2 of a typical 9).

The Lambda-to-computer handoff

This is the core product question: a Lambda starts a conversation, the computer comes up, and Pi continues the same conversation.

What ran (scripts/handoff.ts):

  1. Owner A (fence 1, "Lambda", no executor for computer tools) admitted the message with requestId. The model called run_terminal. A's tool implementation does not execute: it signals "parked" and waits for its invocation to be cancelled. A parked 2.6 s after opening and closed.
  2. After A closed, the generation task was waiting on tools and the tool task was running at checkpoint execute. Owner B saw that tool task as pending at execute.
  3. Owner B (fence 2, "computer") opened the same storage and called resume(). The tool ran once. The model got the result and answered with the real output, 3.8 s after B opened.
  4. One transcript: user, system, assistant (tool call), tool result, assistant. The provider session id (pi.provider) was the same for A and B, so the provider prompt cache carries over.
  5. A handle still holding fence 1 then tried to commit and was rejected with OwnershipLost.

"Exactly once" held here because owner A never reached the tool's real execute() work: it parked before doing anything. A real computer tool still needs its own idempotency marker, because a computer owner that crashes mid-execution looks identical to a parked call.

The pi-durable mechanisms that make this work:

What is missing:

Watching a turn: what the stream would carry

watchEvents() turns each commit into one batch of agent events. A per-turn SSE stream can forward these batches unchanged:

Event Maps to (Messaging)
submission, inbox_update channel.message.created, queued steer or follow-up
run_start / run_end turn.started / turn.completed
turn_start / turn_end one model round (no current Chatticus event)
message_start / message_update / message_end turn.token (text and thinking deltas, tool-call argument deltas)
tool_execution_start / _update / _end tool.call, tool output, tool.result
usage_changed budget metering
snapshot late join or a client that fell more than 100 batches behind

The initial snapshot holds the whole active transcript. It grew from 0.4 KB to 42 KB over four short turns. A per-turn stream should skip it, because the client already has the channel, and send only batches. Missing from the events: a turn.waiting reason (computer booting) and approval.required. Both would be Chatticus events derived from our own documents.

Non-owner mutations

While an owner holds a conversation, nobody else can commit to its storage. Measured: a second handle's commit during a busy turn was rejected. So a human posting or an approval decision cannot write Pi storage directly. Two ways in:

The open part is the edge between these two. The handler has to know whether an owner is live: with a fence lease or a heartbeat, or by always using the mailbox and having every owner drain it at open and before closing. Always using the mailbox is the simplest. It still needs a test with concurrent writers.

Approvals have the same shape. beforeTool can only allow, rewrite or block right now. A hook that waits for a human would keep the owner alive for hours. That rules it out for Lambda. An approval is therefore either a block plus a later re-request after the decision, or a parked tool (as in the handoff) whose next owner checks an approval document before executing.

The language boundary

Status note: this section records the boundary as it stood when the spike was written. The Python tree was later deleted from develop (Kanbus chatticus-f27222) and the whole control plane is TypeScript; see TypeScript control plane. Staging and production run the Python control plane until the develop to main promotion.

Recommendation

The spike holds. pi-durable runs on S3 commit objects with a small DynamoDB index (and, as a baseline, on DynamoDB alone), passes its own conformance suite on both, survives SIGKILL mid stream and mid tool, stops a fenced-out owner, and hands a parked tool call from a computerless owner to a computer owner with one execution and one transcript. The fence maps onto it cleanly.

One measured result contradicts the reason given for the S3 move: at Chatticus's payload sizes, keeping payloads out of DynamoDB does not lower per-turn cost; it raises request cost by about 25 to 30% and lowers storage cost by about 60%, breaking even after roughly 13 (plain) to 14 (tool) months of retention. The S3 layout is still the better default because it removes the 400 KB and 4 MB limits and its cost advantage grows with payload size and retention, but it should not be justified as a per-turn saving. The decision to keep it rests on long-term storage (see Long-term storage).

The staged plan stands:

  1. This spike proves pi-durable on S3 plus a DynamoDB index with one short-lived owner per turn, the Lambda-to-computer handoff and crash resume. No product code before this report. Before step 2, rerun conformance on DynamoDB Local and run one turn against a real table and bucket in the development account, to get real latency, consumed capacity, and the retry and conflict paths.
  2. If it holds, build a TypeScript conversation service that embeds pi-durable and owns everything conversational: messages, turns, approvals, tasks and grants as pi-durable documents, and the per-turn stream fed from watchEvents(). Python keeps organizations, auth, membership, budgets, routines and the computer lifecycle, and talks to it over HTTP. Only the TypeScript side reads or writes Pi storage.
  3. Later, decide whether the rest of the control plane moves to TypeScript, based on what remains and how stable the pi-durable API is. The package is marked experimental and "changes without notice". Pin it exactly and run the conformance suite in CI.

Snapshot objects and the sweeper

Built in conversation/src/storage/snapshot-object.ts, conversation/src/pi/sweeper.ts and the handler conversation/src/lambdas/orphan-sweeper.ts (not wired in infra). Specified in features/pi_session_snapshots.feature and features/pi_orphan_sweeper.feature; the 23 storage conformance cases also run with snapshots enabled (test/indexed-storage.conformance.test.ts).

Snapshot format

conversations/<storage>/snapshots/<seq:012>-<fence:08>.json, immutable, written with If-None-Match: *. A new SNAPSHOT index item (seq, fence) names the newest one and only moves forward (conditional update), so it is not part of the commit transaction and never contends with META.

{ "format": 1, "seq": 61, "fence": 1,
  "commits": { "<seq:012>-<fence:08>": { "<position>": <StorageWrite> } } }

It is a pack, not a summary: for every commit object that an index item still points at as of seq, it carries only the referenced writes, by position (the latest record of each entry, task and submission, and each document's revisions from its newest base to its newest revision). seq is META.seq read before the index is read, so every pointer at or below it is complete. A snapshot is built from the previous snapshot plus the commit objects that snapshot did not carry, so its cost grows with the commits since the last one.

A storage opened with a snapshot reference still costs 0 S3 GETs. The first read of a commit object at or below the snapshot's seq fetches the snapshot once and answers from it; keys above seq (or absent from the pack) are read from S3 as before. An unreadable snapshot degrades to commit reads and logs. The owner writes a snapshot by policy: every N commits and/or at close; a failed snapshot never fails the commit or the close.

Sweeper

sweepOrphans(deps, storageId) reads META, OWNER and SNAPSHOT, lists the commit objects, reads every (c, f) pointer of the partition once, and deletes an object only if no pointer names it, META.seq >= seq or OWNER.fence > fence holds, and its S3 last-modified time is at least the grace period before clock.now(). It also deletes snapshot objects older than the grace period that are not the one SNAPSHOT names and not newer than it. sweepAllStorages finds storages from the conversations/ common prefixes.

Verified on moto

Fact Result
Cold read of a 12-turn conversation (61 commits), no snapshot 25 S3 GETs (one per commit holding an entry)
Same, snapshot written at close 1 GET
Same, snapshot every 10 commits, owner closed without a final snapshot 2 GETs (snapshot plus the commits after it)
Snapshot at turn 6 plus 2 later turns by another owner 5 GETs, all 8 turns read in order
Model context with and without the snapshot (the SNAPSHOT item removed) byte-identical messages
pi-durable conformance, 23 cases, snapshots every 2 commits and at close, no owner cache 23 of 23 pass
Crashed commit (S3 put done, index transaction fails, injected at the DynamoDB client) object stays; deleted by the sweeper after the owner is replaced and the grace has passed, not before
Commit held between put and transaction, 2 days of clock kept; the commit then completes and reads back
Object at seq > META.seq under the current fence, 30 days old kept (no fence proof)
Object at seq <= META.seq under the current fence, never indexed deleted (clause 1 alone)
Committed objects of a replaced owner, 365 days of clock all kept (index pointers)

Findings:

Open questions

Measured for the turn executor

Facts the TypeScript turn executor (conversation/src/turn/) relies on, each checked against pi-durable and pi-ai 1.0.2 on moto.

Measured for the voice understanding step

The understand-the-user step is one non-durable completion outside a Harness (conversation/src/voice/understanding.ts). Verified against pi-ai 1.0.2 with the faux provider and with the real OpenAI provider whose HTTP fetch is replaced, so no request leaves the machine (conversation/test/voice-understanding.test.ts).

The tool gate (verified in TS-26)

Measured with the scripted provider (toolCall(...) then reply(...)) against @earendil-works/pi-durable 1.0.2, in conversation/src/pi/gate.ts and the scenarios of model_tool_loop_sinks.feature.

The computer handoff (verified in TS-29)

Measured with the scripted provider on moto against @earendil-works/pi-durable 1.0.2, first in the prototype conversation/test/computer-park-resume.test.ts and then through the production executor (features/computer_action_handoff.feature). The code is conversation/src/pi/computer-tools.ts (the tools), conversation/src/turn/park.ts (park, resume, probe settlement) and conversation/src/domain/actions.ts (the action).

Spike: handoff to a computer owner

Status: the recommendation below was built and rehearsed on development (TS-65 to TS-69). This section is the record of the spike's measurements; the built design, its proof and its limits are in Pi session handoff.

Measured 2026-10-07 against @earendil-works/pi-durable and pi-ai 1.0.2 on moto 5.2.3. The code is the throwaway spike in spikes/pi-handoff; the raw output of every run quoted here is in its results/ directory. Nothing in conversation/src changed. No AWS access, no image push.

The question: can a Pi session that a Lambda-like owner (A) started be handed to a container-like owner (B) that runs Pi's own coding tools on its own disk, and then be handed back to a Lambda-like owner (A2)? Every owner is a separate Node process. B also ran inside a node:22-bookworm-slim arm64 container. Two scenarios ran:

Answers

Q1. Can B take over mid-turn with a different tool implementation? Yes, with one rule.

B registers Result for A's pending write
Same names, replay: "safe" (safe) B ran it. Task completed. File written.
pi-durable's CodingTools as shipped (unsafe) Not rerun. The model got [error] Tool write was interrupted and may have partially run. No file. The later bash and edit calls ran in an empty workspace and failed.
No tools (missing) Same interrupted result. Later calls got Tool bash is not available.
write with another schema, an extra required mode (schema) B ran it with the stored arguments. The new schema was not applied to the stored call.

So the handoff needs B to register the same tool names with replay: "safe". The shipped tools default to unsafe.

Q2. Where do Pi's built-in coding tools come from? From pi-durable itself, already installed.

Q3. Does the existing executor run in B with only the tool set swapped? Yes, with a one-line seam.

Q4. What does B need? See the table and the proposal below.

AWS calls that B made, recorded by a middleware on its clients (aws.surface in the results):

Service Resource Calls B made
DynamoDB Conversations table (pi-durable index) GetItem, Query, TransactWriteItems, UpdateItem
DynamoDB Messaging table (turns, mailbox, actions, ledger, budgets, policy, bots, channels) GetItem, Query, BatchGetItem, TransactWriteItems, UpdateItem, DeleteItem
S3 conversations/<storageId>/commits/<seq>-<fence>.json GetObject, PutObject
SQS TurnRuns, TurnProbes, ComputerStarts (recorders in the spike, so not exercised) SendMessage, ChangeMessageVisibility
SSM the OpenAI key parameter (Lambda handler only) GetParameter
Model OPENAI_API_KEY in the environment, read by pi-ai HTTPS to api.openai.com

Two checks of what a model-chosen shell command can reach (results/q4-*.txt):

Smallest design that keeps the keys away from model-chosen commands (not built):

  1. Model-call proxy on the control plane. A route (Lambda function URL) that holds the OpenAI key, accepts a per-session bearer token (tenant, bot, turn, expiry of at most 15 minutes, signed by the control plane), forwards to OpenAI and writes the vendor ledger. B is configured with the proxy base URL and the token. B never holds the model key. This needs the pi-ai OpenAI provider to accept a base URL override. I did not test a real override.
  2. Storage credentials scoped to the conversation. At start the control plane calls sts:AssumeRole with session tags (the role's static policy reads them; see PI_SESSION_HANDOFF.md) and passes the temporary credentials (15 to 60 minutes) to B: S3 Get and Put on conversations/<encoded storageId>/* only, and DynamoDB on the Conversations table with dynamodb:LeadingKeys equal to the conversation's partition key. The Messaging table is the weak point: the executor uses its Dynamo stores directly, so B needs the tenant's turn, mailbox and action rows. Scope it with LeadingKeys on the tenant prefix. That is tenant isolation, not least privilege. The better end state replaces B's Messaging access with the existing host protocol over HTTPS (computer/host/src/protocol-client.ts). That is a larger change.
  3. The shell runs as another user. The owner process runs as owner, with the credentials in memory or in a file of mode 0400. Every bash call runs as an unprivileged agent user (through prepare and setpriv or su) with a scrubbed environment, /proc mounted hidepid=2 or a separate PID namespace, and /workspace as its only writable path. On Fargate also block 169.254.170.2 for the agent user (an iptables owner match), because the task role's credentials are served there to every process. A stronger variant runs the shell in a nested sandbox (gVisor or a second container). None of this was built or measured.

Q5. Round trip back to a Lambda-style owner: yes, with the full history.

A2 (fence 3) opened the same storage and answered "What did you just do in the workspace?". Its model request, recorded by the scripted provider, was user, system, assistant, toolResult, assistant, toolResult, assistant, toolResult, assistant, user. The tool results were B's, from the container disk: Successfully wrote to hello.txt, total 4 ... PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" git version 2.39.5 hello from Chatticus, Successfully replaced 1 block(s). The same held through the executor (a second turn, A2 on fence 3) and for gpt-5-nano (A2's answer described the work that B did).

Phase 2: B inside the container

What did not work, and what is unknown

Surprises (pi-durable and Pi)

  1. pi-durable ships its own read, write, edit, bash and an ExecutionEnv. The coding-agent package is not needed.
  2. Those tools default to replay: "unsafe", so a handoff does not run the parked call. The model gets "interrupted", and the work is lost without a visible error unless B overrides the policy.
  3. Replay does not re-validate stored arguments and does not compare implementations. A changed schema is ignored for the pending call. A missing tool turns the pending call into interrupted, not unavailable.
  4. The bash tool returns no content. Its output arrives only through api.output(). A wrapper that needs the text must capture it.
  5. The shipped bash inherits the owner's whole environment, and a same-user child can read /proc/$PPID/environ.
  6. Changing the tool set between owners adds a positional system entry to the transcript.
  7. Pi's call ids from OpenAI contain |fc_.... The scripted provider's call-N ids collide across processes; the spike offsets them.
  8. Harness.open takes env, but our openOwnerSession does not pass it.
  9. After a handoff the generation sits waiting on its tool task until some owner runs it. Nothing in pi-durable says "deferred to another executor". The only signal is replay: "safe" plus a closed owner, as in the TS-29 section above.

Recommendation: go, in steps, with the security design as a gate

Go. The handoff works across processes and in a container, the executor core needs one seam, the history survives the round trip, and the start of the owner itself is about one second on top of the compute start. No-go for shipping B with a model key or broad storage credentials in its environment: the security design in Q4 comes before any production use.

Minimal product design, in order:

  1. conversation/src (behavior first, in features/). A scenario "a computer owner finishes a parked turn". Then: an optional computerTools factory on ExecutorDeps in place of the hard-coded extension; an env option on openOwnerSession; a local-tools extension with the names and schemas of computerToolsExtension, replay: "safe", each call wrapped in the action ledger (claim, run, complete); deps.runVisibility and acknowledge supplied by B's job source. Keep one path: the parked-action protocol with an external host is replaced, not kept beside it.
  2. computer/host. An owner entry point in the image: reads a run job (SQS, or a lease over the host protocol), builds the deps from scoped credentials, calls resumeTurnForAction and consumeRunJob, and runs bash as the agent user. Add the esbuild bundle (about 4 MB) to the image; git is already there.
  3. Control plane. Mint the per-session token and the scoped STS credentials when a turn is handed to a computer. Add the model proxy route. Stop starting a host only to run actions.
  4. infra/. A role for the computer owner with no standing permissions beyond sts:AssumeRole to the scoped role; the scoped role with LeadingKeys and S3 prefix conditions; the proxy function URL; an SQS queue (or a lease route) for computer-owner jobs; remove the OpenAI key from anything the computer can read.
  5. Cutover. Delete the host-protocol action executors once B runs the tools. Run the conformance suite and the handoff scenarios against pi-durable 1.0.2 in CI.