Pi harness

Design note for Kanbus story chatticus-6c4d24 (epic chatticus-9f09c5). Measured 2026-10-04 against @earendil-works/pi-durable 1.0.2 (Pi monorepo 2003871). The code is a throwaway spike in spikes/pi-durable-6c4d24. No product code depends on it.

What we want

One durable engine for a bot's conversation that fits the serverless shape in Design challenges and Messaging: no always-on process, a stream scoped to one turn, and a turn that can begin on a Lambda and continue on the computer.

The plan under test:

  • One pi-durable storage per bot conversation (bot x channel). Its data lives in S3 as one immutable object per commit, found by convention. DynamoDB holds only the index that makes a commit visible.
  • Each turn is run by a short-lived owner (later a Lambda or the computer's worker). It claims the turn's fence, opens the storage, resumes unfinished tasks, submits the new message, runs, and closes.
  • Only the fence holder can commit.

What pi-durable is

A durable agent harness from the Pi framework. Everything is a committed record before it is shown:

  • Entries: the immutable transcript (pi.user, pi.assistant, pi.tool-result, pi.system, or our own kinds). Entries with a model field are what the model sees.
  • Tasks: durable state machines that checkpoint each step. A turn is a pi.generation task (prepare, request, tools) that owns one pi.tool task per tool call.
  • Submissions: admitted input with an optional requestId for deduplication. A submission to a busy conversation waits in the inbox as a steer or a follow-up.
  • Documents: typed JSON state committed with entries, stored as a base plus Chord deltas (pi.live holds the streaming partial and tool output, pi.inbox, pi.usage, pi.agent, pi.provider).

It talks to storage only through a 16-method Storage interface: commit(writes), mintId(), and reads. The Session serializes commits and assumes one process owns a storage at a time. It has no cross-process lock. That is the part the fence supplies.

The storage: S3 data, DynamoDB index

src/indexed-storage.ts. The first version (src/dynamodb-storage.ts, kept as the cost baseline) stored every record body in DynamoDB.

S3: one object per commit

s3://<bucket>/conversations/<storage>/commits/<seq:012>-<fence:08>.json
  { "seq": 12, "fence": 3, "token": "<uuid>", "writes": [ ...every StorageWrite of the commit... ] }
  • The key is derived from the commit sequence and the committing owner's fence, so a reader needs no stored path. Zero padding keeps a listing in commit order.
  • Objects are immutable. An owner caches every object it reads or writes for its lifetime and fetches missing ones in parallel.
  • Document copies (forks) are resolved to their base value before the object is written, so an object never refers to another object.
  • With the data in S3, DynamoDB's 400 KB item limit and 4 MB transaction limit no longer apply to payloads. Measured: a 410 KB entry and a 4.2 MB commit both committed (results/limits-indexed.json). Both are rejected by the DynamoDB-only storage.

DynamoDB: the index

One table, pk/sk, plus three local secondary indexes (l1, l2, l3). Local indexes support strongly consistent reads, which the storage contract requires. One storage is one partition, pk = PI#<tenant>#<bot>#<channel>. Every record item carries a pointer c (commit seq), f (fence) and p (position of the write in the commit object).

sk Holds (besides the pointer) Index keys
META commit seq, ID high-water mark, last commit token --
OWNER current fence --
R#<id> conversation the whole record (ids only: parent, owner) l1 C#<id>, l2 CT#<ownerTask>#<id>, l3 CC#<ownerConversation>#<id>
R#<id> entry conversation id l1 E#<conversation>#<id>; with a head also l2 H#<conversation>#<id>
R#<id> task conversation, kind, abort and background flags l1 T#<id>, l2 TS#<status>#<id>
R#<id> submission conversation id l1 S#<id>, l2 SS#<status>#<id>, l3 SR#<conversation>#<hash(requestId)>#<id>
R#<id> document lifecycle record (kind, key, scope, created, retired), latest version and revision seq l1 D#<hash(scope)>#<id>, l2 DA#<hash(address)>#<id>
V#<document>#<seq> base or delta flag --
  • Item size. The largest index item written was 361 bytes (measured over every phase script). Index items carry no message text, tool output or document value.
  • Global ID namespace. Every record is keyed R#<id>, so a collision is a key collision. Conversations, entries and documents are written with attribute_not_exists(pk). Tasks and submissions replace themselves with attribute_not_exists(pk) OR t = :type.
  • Ancestry-aware entries. Reads walk the fork chain (parent.at caps each ancestor) and query l1 newest-first per conversation. Filters for tasks and submissions run on index attributes, so a scan reads S3 only for the records it returns.
  • Point-in-time documents. A read queries V#<id># downward from the requested seq (for a current read, the record's latest revision seq) to the newest base, then fetches those commit objects in parallel and applies the deltas. Nothing is ever deleted, so this is a consistent point-in-time read. The DynamoDB-only version had to delete superseded revisions and could race a reader (see open questions).

The commit protocol

  1. Prepare: read what validation needs from the index (document records and address occupancy; 1.2 to 2.1 read round trips per commit, measured).
  2. PutObject the commit object with If-None-Match: *, so a key is never overwritten. The fence in the key means only this owner ever writes it. If the key already exists with this commit's token, it is this commit's own earlier attempt. If it has another token, the storage reads META: when META.seq >= seq the object is committed data and the commit fails with OwnershipLost without deleting anything; otherwise it is this owner's orphan from a commit whose transaction failed, and it is deleted and replaced. An S3 409 ConditionalRequestConflict means a concurrent write to the same key is still in flight, which with the fence in the key can only be this owner's own earlier attempt; it is retried with backoff and resolves to a success or a same-token 412.
  3. One TransactWriteItems with the index items, the META update conditioned on seq = <last seq this owner saw> (which also stores the token), and, when fenced, a ConditionCheck on OWNER.fence.
  4. The transaction carries the token as ClientRequestToken. The storage itself retries throttling, timeouts and TransactionConflict with the same token. If a retry finds the META condition failed, it re-reads META: when the stored token is its own, the first attempt landed and only the response was lost, so the commit counts as done. The check has three outcomes: committed, not committed, or unknown (the read itself failed). Unknown throws CommitOutcomeUnknown, an OwnershipLost subtype that is fatal to the Session, and the storage does not delete the object: it may be committed, so it is left for the next owner to find and for the sweeper.
  5. The SDK client uses maxAttempts: 1. In the first version that was load-bearing: an SDK retry after a lost response would have come back as a false "another owner committed". Production must keep SDK retries off for TransactWriteItems (or make them reuse the token) and leave retrying to the storage as above.

Readers only follow the index. An object whose transaction never landed is invisible.

Failure cases

Failure What happens Measured
Transaction rejected (ID collision, size) StorageRejected; the storage deletes its own object Object gone (results/orphans.json)
Fence moved before the transaction OwnershipLost; the object is deleted Object gone
Crash between PutObject and the transaction The object stays as an orphan. The next commit by the same owner at that seq replaces it; another owner uses a different key Replaced and committed
PutObject fails Nothing committed; pi-durable sees a plain error and poisons the Session --
Transaction outcome unknown after retries The token check decides; if still unknown, CommitOutcomeUnknown poisons the Session and the object is kept. A landed commit is then found by the next owner Not reproducible on moto
Object deleted or unreadable after commit Reads of those records fail. This must not happen: no lifecycle rule may expire commits/ --

Orphan rule. The sweeper reads META and OWNER first, then checks references, so the values it acts on can only be older than the truth in the safe direction (both only rise). It may delete commits/<seq>-<fence>.json only when no index item points at (seq, fence) and either of these holds:

  1. META.seq >= seq. The commit's META condition needs META.seq = seq - 1, so a late commit of this seq can never succeed.
  2. OWNER.fence > fence. Fences only rise, so the commit's ConditionCheck on OWNER.fence = fence can never succeed.

There is no time-based clause. An object with seq > META.seq whose fence is still the current OWNER.fence may be an in-flight commit of a live (or momentarily stalled) owner and is kept; it becomes deletable by clause 1 or 2 as soon as the sequence passes it or a new owner raises the fence. results/orphans.json shows both clauses and the kept case. An S3 lifecycle rule cannot express "unreferenced", so the sweeper is a small scheduled job. Orphans arise only from a crash between steps 2 and 3, so it can be rare.

Ownership and the fence

  • claimOwnership(fence) raises OWNER.fence if the new value is higher.
  • The storage then commits only with fence = mine. An owner that has lost the fence gets OwnershipLost: Owner fence moved and cannot write. So does an owner whose META condition fails for someone else's commit, or whose transaction keeps hitting TransactionConflict (for example against a concurrent claimOwnership).
  • OwnershipLost is deliberately not a StorageRejected. pi-durable treats StorageRejected as an ordinary rollback and keeps going; any other commit error poisons the Session. Measured (scripts/fence-loss.ts): the fence was raised while the stale owner's tool was running. Its next commit failed with OwnershipLost, after which it made no further commit attempt, started no tool and sent no model request in the 18 s observed. Its submission.wait() never settled, so a host must close a stale owner itself rather than wait for it. The next owner resumed and finished the turn.
  • The Chatticus per-turn fence maps onto this directly. An attempt claims the turn (attempt.claimed) and that fence value is the storage fence. Relinquishing the attempt is closing the Harness.

Results

All runs used moto 5.2.3 for both DynamoDB and S3 (path-style, dummy credentials). Amazon DynamoDB Local was requested, but its container never answered on port 8000 in this session (the pull was stopped after 30 minutes). moto honors If-None-Match: * on PutObject (412 on the second write, results/s3-conditional.json).

Neither moto nor DynamoDB Local exercises TransactionConflict, throttling, SDK retries or per-partition throughput limits. The retry, idempotency-token and ownership-loss paths are therefore untested against real behavior. Only a run against a real table covers them.

Test Result
pi-durable storage conformance suite 23 of 23 pass for the production S3-plus-index storage in conversation/src/storage/indexed-storage.ts, in CI (job conversation) against moto. The spike's copy failed 10 cases on an index-name bug that the production copy fixes.
Owner 1 answers, owner 2 reopens and continues Pass. Owner 2 answered "teal" to "what is my favourite colour?"
Resubmit the same requestId The original submission id comes back, already done. 0 commits
Stale owner commits after a newer fence Rejected with OwnershipLost (Owner fence moved). Stale fence claim rejected
Fence raised while an owner is mid tool call The stale owner's next commit failed with OwnershipLost; it then did nothing more. The next owner reran the replay-safe tool and finished
Lambda-to-computer handoff Pass (below)
Kill mid model stream Pass. The partial (262 chars) became a pi.assistant entry with stopReason: aborted, then the same messages were resent and answered
Kill mid tool call, replay: "safe" The tool reran on the new owner: started twice, finished once. One result
Kill mid tool call, replay: "unsafe" No rerun. The model got an interrupted error result: "Tool run_terminal was interrupted and may have partially run"
Attributed message from another participant Pass. "Friday at 3pm, according to Bea."
Approval-style gate Pass. A beforeTool hook returned { block }. The model got "Tool call blocked: needs approval: ..." and said so
Non-owner message during a busy turn Pass through a mailbox (below). A direct commit from a second handle was rejected (OwnershipLost: Commit sequence moved)
Orphans A rejected or fenced-out commit leaves no object; a crashed attempt's object is replaced (results/orphans.json)
101 transaction items Rejected before sending (both storages). 410 KB entry and 4.2 MB commit: committed on S3-plus-index, rejected on DynamoDB-only

Per turn (gpt-5-nano, S3-plus-index storage)

Turn Commits Max items per commit Bytes written (index + objects) Stream batches Stream bytes
First turn, new conversation 8 13 20 KB -- --
Plain answer 7 9 15 to 16 KB 5 10 KB
One tool call (3 s tool, streamed output) 32 9 62 KB 19 34 KB
Tool call blocked 45 9 91 KB 27 52 KB
Tool call plus a mid-turn steer 21 10 49 KB 13 35 KB
  • The commit count of a turn varies with the model: how many 100 ms progress windows its answer and the tool's output span, and how many model requests it makes. The same tool turn took 19 to 43 commits across runs.
  • No real commit came near 100 items. The item count grows with the number of tool calls in one model response (roughly calls + 4), so a response with about 95 parallel tool calls would exceed the limit. That limit remains with the index.
  • Commit latency against moto was 31 to 75 ms p50 and 54 to 787 ms p95 for the S3-plus-index storage (one PutObject plus one transaction), against 12 to 41 ms p50 for the DynamoDB-only storage. Neither is representative of AWS in a region. In a region expect roughly an S3 PUT plus a small transaction per commit, tens of milliseconds. Partials commit at most every 100 ms, so commit latency limits how fast the stream updates.

Reopen

A new owner's Harness.open plus root() took 7 DynamoDB reads and 0 S3 GETs on both storages (results/cost.json): the Session loads lazily. Reading the whole transcript after two turns took 2 index queries and 9 S3 GETs (one per commit that holds an entry), fetched in parallel. The moto timings (43 ms reopen, 28 ms transcript) say nothing about AWS.

The cost that grows is not reopen but the first model request of each turn: it needs every visible entry, so a fresh owner fetches one object per commit that holds an entry, about 4 to 5 per turn of history. A periodic snapshot object (conversations/<id>/snapshots/<seq>.json holding the visible entries and current documents) would make that one GET plus the recent commits. It is not implemented in the spike; it should be before production. Request cost is flat per turn only with that snapshot object. Without it (the spike as built), a fresh owner's first model request fetches about 4 to 5 commit objects per turn of history, so GETs and first-token latency grow linearly with conversation length. The cost projections below count only the spike's two-turn runs and omit these growing GETs.

Cost: DynamoDB-only versus S3 data with a DynamoDB index

Prices, us-east-1, checked 2026-10-04 on DynamoDB on-demand pricing and S3 pricing:

  • DynamoDB on-demand: $0.625 per million write request units, $0.125 per million read request units, $0.25 per GB-month (Standard table class; the first 25 GB are free). A transactional write costs 2 WRU per started KB. Writes to a local secondary index are billed as writes too.
  • S3 Standard: $0.005 per 1,000 PUT, $0.0004 per 1,000 GET, $0.023 per GB-month (first 50 TB).

Request units are estimated by the spike's meter from each request (moto does not report consumed capacity): 2 WRU per started KB per transaction item plus 1 per local index the item lands in, and 1 RRU per started 4 KB of strongly consistent reads. The meter undercounts index writes that move an item between index keys (a delete plus a put). One run each, same prompts (scripts/cost.ts, results/cost.json):

Turn Storage Commits WRU RRU S3 PUT Requests per 1,000 turns
Plain answer DynamoDB-only 7 120 46 0 $0.081
Plain answer S3 + index 7 99 42 7 $0.102
One tool call DynamoDB-only 31 479 182 0 $0.322
One tool call S3 + index 30 382 141 30 $0.406

Per commit, the S3-plus-index storage costs about $0.0000135 to $0.0000146 and the DynamoDB-only storage about $0.0000104 to $0.0000115: the index version costs about 25 to 30% more in requests. The reason: at these payload sizes (a few KB per commit) WRUs are set by the per-item minimum, not by the payload, so moving the payload out of DynamoDB saves only 2 to 3 WRU per commit, while each commit adds one S3 PUT ($0.000005, the price of 8 WRU).

Resident data after setup and both turns DynamoDB-only S3 + index
DynamoDB billable bytes (items + 100 bytes each, times local indexes) 62 KB 25 KB
S3 bytes 0 53 KB (38 objects)
Per 1,000 turns per month (setup spread over the two turns) $0.0073 $0.0035

Storage per GB-month of conversation: DynamoDB-only pays $0.25 on every billable byte, including the copies in three local indexes; the index version pays $0.25 on a third as many DynamoDB bytes and $0.023 on the rest.

So:

  • For short turns, the index version is not cheaper at first. Its extra request cost per tool turn ($0.000084) equals its storage saving ($0.0000059 per turn per month) after about 14 months of retention; for a plain turn ($0.000021 against $0.0000016 a month), after about 13 months. Formula: break-even months = extra request cost per turn / storage saved per turn per month. Extra request cost is S3 + index minus DynamoDB-only from results/cost.json at $0.625/M WRU, $0.125/M RRU, $0.005 per 1,000 PUTs: plain $0.000102125 - $0.00008075 = $0.0000214, tool $0.000406375 - $0.000322125 = $0.0000843. Storage saved per month is (DynamoDB-only bytes x $0.25 - index bytes x $0.25 - S3 bytes x $0.023) / 2^30 using the per-turn bytes in the projection table: plain (12,512 x 0.25 - 4,491 x 0.25
    • 11,715 x 0.023) / 2^30 = $0.0000016, tool (41,150 x 0.25 - 12,093 x 0.25
    • 39,794 x 0.023) / 2^30 = $0.0000059. Break-even: 0.0000214 / 0.0000016 = 13.2 months (plain), 0.0000843 / 0.0000059 = 14.2 months (tool). An earlier report of about 4 and 20 months was wrong and appears nowhere else; the figures here use 2^30 bytes per GB (with 10^9, 12.3 and 13.3 months). See the household projection below.
  • It wins when payloads are large or kept long: long answers, large tool output, images, and years of history. It also removes the 400 KB item and 4 MB transaction limits, which is a correctness gain, not a cost one.
  • Both are cheap in absolute terms: under half a dollar per 1,000 tool turns for storage requests, small next to the model cost of the same turns.
  • Cheapest next steps, if cost matters: fewer commits per turn (pi-durable commits partials every 100 ms; a longer throttle cuts commits roughly in proportion), fewer index items per commit (the pi.live revision plus its record update are 2 of the typical 9), and the snapshot object above so that cold reads do not grow with history.

AWS run

Run on 2026-10-04 against real AWS in the Chatticus development account (us-east-1, the ThinTurn stack's region), approved by the human for this run only. A throwaway CDK app (spikes/pi-durable-6c4d24/aws/, stack ChatticusPiDurableSpike) deployed one on-demand table with the spike's index schema, one S3 bucket (block public access, SSL enforced), and one Node 22 arm64 Lambda (1024 MB, no VPC, 120 s timeout, the owner code bundled with esbuild). The Lambda read the existing development OpenAI parameter and ran gpt-5-nano; SDK maxAttempts was 1. The stack was destroyed afterwards (describe-stacks, head-bucket and describe-table all report not found). Raw invoke records are in results/aws/raw/, the roll-up in results/aws/summary.json. Conformance was not re-run on AWS.

Finding before any measurement: real DynamoDB rejected the table because local secondary index names must be at least 3 characters; moto accepted l1, l2 and l3. The indexes are now named l1-index and so on (the attribute names are unchanged).

Measurement moto (local) AWS (Lambda to DynamoDB and S3, us-east-1)
Cold start, init n/a 421 to 597 ms (3 runs)
Cold plain turn, handler duration n/a 3.79 s and 3.80 s (init 0.57 and 0.60 s; wall 5.0 s)
Cold tool turn, handler duration n/a 11.3 s (init 0.42 s)
SSM key fetch (cold only) n/a 145 to 156 ms
Open (claim fence, load, harness, root) 43 ms 138 to 156 ms warm (p50), 347 to 407 ms cold
Warm plain turn, duration n/a p50 3.2 s (2.5 to 4.7 s, 10 runs), 8 commits
Warm tool turn, duration n/a 7.4 to 10.6 s (3 runs), 27 to 29 commits (the tool was instant here)
Commit latency, p50 / p95 31 to 75 ms / 54 to 787 ms 64 ms / 103 ms (plain), 60 ms / 84 ms (tool); 164 commits pooled; max 131 ms
S3 PutObject p50 n/a 29 ms (p95 about 44 ms)
TransactWriteItems p50 n/a 23 ms (p95 about 35 to 39 ms)
Max Lambda memory n/a 160 to 161 MB of 1024 MB

Model time dominates every turn: a plain turn is about 8 commits of about 64 ms (roughly 0.5 s of the 3 s). Commit latency is a S3 PUT plus a transaction plus about 10 ms of owner work, and it stayed well below the 100 ms partial window at p95. Cold-start cost (about 0.5 s init plus about 0.2 s SSM and 0.25 s extra open) is small next to the model call.

Consumed capacity versus the meter (ReturnConsumedCapacity: TOTAL on every call, one fresh conversation per invoke, per turn without the open):

Turn Commits WRU actual (meter) RRU actual (meter) S3 PUT S3 GET
Plain, mean of 10 8 102 (99) 42 (42) 7 0
Tool, mean of 3 28 359 (346) 127 (127) 27 0

The meter's read estimate is exact and its write estimate is low by 3 to 4% (index-key moves, as predicted). Opening a new conversation additionally costs 39 WRU and 22 RRU. Updated request cost per 1,000 turns with actual capacity at $0.625/M WRU, $0.125/M RRU and $0.005 per 1,000 PUTs: plain $0.104 (was $0.102), tool $0.375 for 28 commits (was $0.406 for 30 commits; about $0.0000134 per commit, unchanged). The cost table stands; the meter can be trusted to within 5%.

Lost response (results/aws/raw/lost-*.json). A request handler hook let the third TransactWriteItems of a plain turn commit and then raised an error instead of returning the response, with maxAttempts: 1.

  • Error TimeoutError (retryable): the storage retried with the same ClientRequestToken, DynamoDB returned success, the commit counted once (9 transactions sent, 8 commits, attempts per commit [1,1,2,1,...]).
  • Error with a non-retryable name: the storage read META, found its own token, and counted the commit as committed without resending (8 sent, 8 commits). Both turns finished done with sequence numbers unbroken, so no commit was lost or doubled.

Forced conflict (results/aws/raw/conflict.json). 20 trials: two handles on the same conversation at the same sequence committed concurrently. In all 20, exactly one committed and the other raised OwnershipLost ("Commit sequence moved"); the final sequence was always 2. Real TransactionConflict cancellations happened in 13 of 20 trials (the storage retried, then saw the failed sequence condition); in the other 7 the loser was rejected by the sequence condition directly. moto never produced either behavior.

Bundle. The Lambda bundle with the AWS SDK clients included is 4.93 MB unminified and 2.05 MB minified; with @aws-sdk/* left to the Lambda runtime it is 737 KB minified. The pi-ai OpenAI provider did not drag in the other providers (subpath imports tree-shake).

Cost of the run: well under $1 (a few cents of Lambda, a few thousand requests, DynamoDB on-demand, minutes of storage); the stack's bucket and table were deleted, and the leftover log groups were deleted by hand.

Long-term storage

Decision: payloads stay in S3 and DynamoDB is only the index. The concern is not the cost of one turn but storage that compounds: every turn ever taken stays resident and billed every month. DynamoDB charges $0.25 per GB-month for every billable byte, including each local-index copy; S3 Standard charges $0.023.

Projection for one household

100 plain turns and 20 tool turns a day. Bytes per turn are the measured resident growth from scripts/cost.ts (results/cost.json, one run each, first-turn setup subtracted):

Per turn DynamoDB-only S3 + index: DynamoDB S3 + index: S3
Plain 21,144 - 8,632 = 12,512 B 12,848 - 8,357 = 4,491 B 12,965 - 1,250 = 11,715 B
Tool call (31 and 30 commits) 62,294 - 21,144 = 41,150 B 24,941 - 12,848 = 12,093 B 52,759 - 12,965 = 39,794 B
Per day (100 plain + 20 tool) 1,251,200 + 823,000 = 2,074,200 B 449,100 + 241,860 = 690,960 B 1,171,500 + 795,880 = 1,967,380 B
Per 30-day month, GiB (/ 2^30) 0.0580 0.0193 0.0550

The stored data after N months is N times the monthly growth. The bill for month N is that stock times the price. The cumulative bill is the price times the monthly growth times N(N+1)/2. Prices as cited above; the 25 GB DynamoDB free tier is per account and ignored.

After Stored: DynamoDB-only Stored: S3 + index Bill that month: DynamoDB-only Bill that month: S3 + index Cumulative: DynamoDB-only Cumulative: S3 + index
1 month 0.058 GiB 0.019 + 0.055 GiB 0.058 x 0.25 = $0.014 0.019 x 0.25 + 0.055 x 0.023 = $0.006 $0.014 $0.006
12 months 0.695 GiB 0.232 + 0.660 GiB $0.174 $0.073 0.25 x 0.0580 x 78 = $1.13 (0.25 x 0.0193 + 0.023 x 0.0550) x 78 = $0.48
36 months 2.086 GiB 0.695 + 1.979 GiB $0.522 $0.219 0.25 x 0.0580 x 666 = $9.65 (0.25 x 0.0193 + 0.023 x 0.0550) x 666 = $4.06

Requests for the same household are about $0.44 a month (DynamoDB-only) and $0.55 (S3 + index), flat over time only with the snapshot object (without it the S3 layout's GETs grow with history, at $0.0000004 each): 100 x $0.0000808 + 20 x $0.000322 a day against 100 x $0.000102 + 20 x $0.000406. The S3 layout's monthly bill (storage plus requests) drops below DynamoDB-only's around month 14 and its cumulative bill around month 26. That crossover comes earlier with any stage-2 item below; with item (2), the DynamoDB part stops growing.

Stage-2 requirements

  1. S3 lifecycle. Commit and segment objects of conversations that have been idle (for example 30 days) move to S3 Intelligent-Tiering or Glacier Instant Retrieval. Both keep millisecond reads, so a returning conversation needs no restore. Never expire commits/ or segments/.

  2. Compaction of idle conversations. When a conversation has been idle and has no live task, a job folds its commit objects into one segment object and replaces its per-record index items with one manifest item, so the DynamoDB footprint per conversation stays roughly constant.

    s3://<bucket>/conversations/<storage>/segments/<throughSeq:012>.json
      { "throughSeq": 4812, "records": { "<id>": <record> }, "revisions": { "<doc>": [<base>, <deltas>...] } }
    
    DynamoDB pk = PI#<storage>, sk = MANIFEST
      { "segment": "segments/000000004812.json", "throughSeq": 4812, "bytes": 1834221,
        "liveTasks": 0, "nextId": 90311, "requestIds": "<bloom or hash set of recent requestIds>" }
    

    Reads resolve through the manifest. An id with an R#<id> item is read as today, from the commit object it points at. Without one, and with id at or below the segment's range, it is read from the segment, which is fetched once and cached. Ordered scans (entries, tasks, submissions, documents) merge the segment's sorted lists with the index queries above throughSeq. The first commit after reopening writes index items again, only for new records. Compaction itself runs as an owner holding the fence: it writes the segment, then in one transaction deletes the folded index items (in batches of under 100) and writes MANIFEST. A crash leaves either the old index or the new manifest, never neither. Folded commit objects are deleted only after the manifest commits.

  3. DynamoDB Standard-IA for the index table. Standard-IA trades lower storage price for higher request price. Evaluate it once compaction exists, because the index's balance of request cost to storage cost decides it.

  4. Fewer commits per turn. Commits dominate request cost: 7 for a plain turn, 30 or more for a tool turn, most of them 100 ms partial and tool-output updates of pi.live. Options: a longer partial throttle in pi-durable, coalescing live updates when nobody is watching the stream, and fewer index items per commit (the pi.live revision and its record update are 2 of a typical 9).

The Lambda-to-computer handoff

This is the core product question: a Lambda starts a conversation, the computer comes up, and Pi continues the same conversation.

What ran (scripts/handoff.ts):

  1. Owner A (fence 1, "Lambda", no executor for computer tools) admitted the message with requestId. The model called run_terminal. A's tool implementation does not execute: it signals "parked" and waits for its invocation to be cancelled. A parked 2.6 s after opening and closed.
  2. After A closed, the generation task was waiting on tools and the tool task was running at checkpoint execute. Owner B saw that tool task as pending at execute.
  3. Owner B (fence 2, "computer") opened the same storage and called resume(). The tool ran once. The model got the result and answered with the real output, 3.8 s after B opened.
  4. One transcript: user, system, assistant (tool call), tool result, assistant. The provider session id (pi.provider) was the same for A and B, so the provider prompt cache carries over.
  5. A handle still holding fence 1 then tried to commit and was rejected with OwnershipLost.

"Exactly once" held here because owner A never reached the tool's real execute() work: it parked before doing anything. A real computer tool still needs its own idempotency marker, because a computer owner that crashes mid-execution looks identical to a parked call.

The pi-durable mechanisms that make this work:

  • Intent before effect. The pi.tool task commits its checkpoint { phase: "execute", arguments, replay } before execute() runs.
  • Close is not abort. Harness.close() stops every invocation without writing an outcome (spec section 5.4). A throw after the invocation is signalled does not become a tool error. The task stays at execute, and running reads back as pending on reopen.
  • Recovery replays only when safe. On reopen, execute reruns the tool only if both the stored and the current policy are replay: "safe". Otherwise the model gets an interrupted error result.
  • The generation waits durably. It sits waiting on its tool tasks (phase: "tools") and resumes when they are terminal, on whichever owner is running then.

What is missing:

  • No explicit "deferred to another executor" state. Parking reuses crash recovery, so a computer tool must be declared replay: "safe". B cannot tell "A parked before running it" from "B crashed halfway through". A real computer tool needs its own idempotency marker (for example an action id in a task document) or a parking hook upstream.
  • beforeTool runs on A. Approval and argument rewriting happen on the owner that first sees the call. That is fine for policy hooks. It is wrong if the hook needs the computer.
  • The tool and its schema must be registered on A too, so the model sees the tool from the first request. Only execute() differs by capability.

Watching a turn: what the stream would carry

watchEvents() turns each commit into one batch of agent events. A per-turn SSE stream can forward these batches unchanged:

Event Maps to (Messaging)
submission, inbox_update channel.message.created, queued steer or follow-up
run_start / run_end turn.started / turn.completed
turn_start / turn_end one model round (no current Chatticus event)
message_start / message_update / message_end turn.token (text and thinking deltas, tool-call argument deltas)
tool_execution_start / _update / _end tool.call, tool output, tool.result
usage_changed budget metering
snapshot late join or a client that fell more than 100 batches behind

The initial snapshot holds the whole active transcript. It grew from 0.4 KB to 42 KB over four short turns. A per-turn stream should skip it, because the client already has the channel, and send only batches. Missing from the events: a turn.waiting reason (computer booting) and approval.required. Both would be Chatticus events derived from our own documents.

Non-owner mutations

While an owner holds a conversation, nobody else can commit to its storage. Measured: a second handle's commit during a busy turn was rejected. So a human posting or an approval decision cannot write Pi storage directly. Two ways in:

  • Mailbox (tested). The HTTP handler writes the message to a mailbox item outside Pi storage (MB#<storage>, keyed by message id). The owner drains the mailbox and calls submit({ type: "input", whenBusy: "steer", requestId: <message id> }). In the test, Bea's message arrived during a tool round, was placed after it, and the final answer used it. The requestId makes a re-drain after a crash harmless. A passive message (not addressed to the bot) would be submit({ type: "write", entry }).
  • Owner-free admission. When no owner is live, the handler can briefly become the owner itself (claim the next fence, submit, close) and enqueue a turn.

The open part is the edge between these two. The handler has to know whether an owner is live: with a fence lease or a heartbeat, or by always using the mailbox and having every owner drain it at open and before closing. Always using the mailbox is the simplest. It still needs a test with concurrent writers.

Approvals have the same shape. beforeTool can only allow, rewrite or block right now. A hook that waits for a human would keep the owner alive for hours. That rules it out for Lambda. An approval is therefore either a block plus a later re-request after the decision, or a parked tool (as in the handoff) whose next owner checks an approval document before executing.

The language boundary

Status note: this section records the boundary as it stood when the spike was written. The Python tree was later deleted from develop (Kanbus chatticus-f27222) and the whole control plane is TypeScript; see TypeScript control plane. Staging and production run the Python control plane until the develop to main promotion.

  • Then: the Python control plane owns organizations, auth, membership, budgets, routines, the computer lifecycle, turns and the transcript. web/, infra/ and the harnesses are TypeScript. The worker-to-control-plane boundary is already HTTP.
  • pi-durable, pi-ai and Chord are TypeScript only. Storage access is a TypeScript API, and the storage layout (Chord deltas, commit objects, ID namespace) is pi-durable's private format. Python reading or writing these items or objects would be a second reader of someone else's format. The working rules forbid dual readers.
  • So the cut follows the storage. Whatever reads or writes a Pi conversation is TypeScript. Everything else can stay where it is and talk to it over HTTP.

Recommendation

The spike holds. pi-durable runs on S3 commit objects with a small DynamoDB index (and, as a baseline, on DynamoDB alone), passes its own conformance suite on both, survives SIGKILL mid stream and mid tool, stops a fenced-out owner, and hands a parked tool call from a computerless owner to a computer owner with one execution and one transcript. The fence maps onto it cleanly.

One measured result contradicts the reason given for the S3 move: at Chatticus's payload sizes, keeping payloads out of DynamoDB does not lower per-turn cost; it raises request cost by about 25 to 30% and lowers storage cost by about 60%, breaking even after roughly 13 (plain) to 14 (tool) months of retention. The S3 layout is still the better default because it removes the 400 KB and 4 MB limits and its cost advantage grows with payload size and retention, but it should not be justified as a per-turn saving. The decision to keep it rests on long-term storage (see Long-term storage).

The staged plan stands:

  1. This spike proves pi-durable on S3 plus a DynamoDB index with one short-lived owner per turn, the Lambda-to-computer handoff and crash resume. No product code before this report. Before step 2, rerun conformance on DynamoDB Local and run one turn against a real table and bucket in the development account, to get real latency, consumed capacity, and the retry and conflict paths.
  2. If it holds, build a TypeScript conversation service that embeds pi-durable and owns everything conversational: messages, turns, approvals, tasks and grants as pi-durable documents, and the per-turn stream fed from watchEvents(). Python keeps organizations, auth, membership, budgets, routines and the computer lifecycle, and talks to it over HTTP. Only the TypeScript side reads or writes Pi storage.
  3. Later, decide whether the rest of the control plane moves to TypeScript, based on what remains and how stable the pi-durable API is. The package is marked experimental and "changes without notice". Pin it exactly and run the conformance suite in CI.

Snapshot objects and the sweeper

Built in conversation/src/storage/snapshot-object.ts, conversation/src/pi/sweeper.ts and the handler conversation/src/lambdas/orphan-sweeper.ts (not wired in infra). Specified in features/pi_session_snapshots.feature and features/pi_orphan_sweeper.feature; the 23 storage conformance cases also run with snapshots enabled (test/indexed-storage.conformance.test.ts).

Snapshot format

conversations/<storage>/snapshots/<seq:012>-<fence:08>.json, immutable, written with If-None-Match: *. A new SNAPSHOT index item (seq, fence) names the newest one and only moves forward (conditional update), so it is not part of the commit transaction and never contends with META.

{ "format": 1, "seq": 61, "fence": 1,
  "commits": { "<seq:012>-<fence:08>": { "<position>": <StorageWrite> } } }

It is a pack, not a summary: for every commit object that an index item still points at as of seq, it carries only the referenced writes, by position (the latest record of each entry, task and submission, and each document's revisions from its newest base to its newest revision). seq is META.seq read before the index is read, so every pointer at or below it is complete. A snapshot is built from the previous snapshot plus the commit objects that snapshot did not carry, so its cost grows with the commits since the last one.

A storage opened with a snapshot reference still costs 0 S3 GETs. The first read of a commit object at or below the snapshot's seq fetches the snapshot once and answers from it; keys above seq (or absent from the pack) are read from S3 as before. An unreadable snapshot degrades to commit reads and logs. The owner writes a snapshot by policy: every N commits and/or at close; a failed snapshot never fails the commit or the close.

Sweeper

sweepOrphans(deps, storageId) reads META, OWNER and SNAPSHOT, lists the commit objects, reads every (c, f) pointer of the partition once, and deletes an object only if no pointer names it, META.seq >= seq or OWNER.fence > fence holds, and its S3 last-modified time is at least the grace period before clock.now(). It also deletes snapshot objects older than the grace period that are not the one SNAPSHOT names and not newer than it. sweepAllStorages finds storages from the conversations/ common prefixes.

Verified on moto

Fact Result
Cold read of a 12-turn conversation (61 commits), no snapshot 25 S3 GETs (one per commit holding an entry)
Same, snapshot written at close 1 GET
Same, snapshot every 10 commits, owner closed without a final snapshot 2 GETs (snapshot plus the commits after it)
Snapshot at turn 6 plus 2 later turns by another owner 5 GETs, all 8 turns read in order
Model context with and without the snapshot (the SNAPSHOT item removed) byte-identical messages
pi-durable conformance, 23 cases, snapshots every 2 commits and at close, no owner cache 23 of 23 pass
Crashed commit (S3 put done, index transaction fails, injected at the DynamoDB client) object stays; deleted by the sweeper after the owner is replaced and the grace has passed, not before
Commit held between put and transaction, 2 days of clock kept; the commit then completes and reads back
Object at seq > META.seq under the current fence, 30 days old kept (no fence proof)
Object at seq <= META.seq under the current fence, never indexed deleted (clause 1 alone)
Committed objects of a replaced owner, 365 days of clock all kept (index pointers)

Findings:

  • A commit whose every record was later superseded (for example a task rewritten in a later commit) has no index pointer and satisfies clause 1, so the rule deletes it although it was once committed. Nothing can read it, so this is safe, but "committed" is not the same as "kept" for such an object. If compaction ever needs to read old commits, it must run before the sweeper.
  • S3 LastModified is the only age the sweeper can see (commit objects carry no timestamp), so tests set the fake clock to real now and advance it.
  • The handler works against moto through AWS_ENDPOINT_URL (test/orphan-sweeper-handler.test.ts).
  • A snapshot write by a stale owner is harmless: it packs committed, immutable data and SNAPSHOT only moves forward.

Open questions

  • Real AWS: latency of PutObject plus the transaction per commit, consumed capacity versus the spike's estimate, and the throttling, conflict and lost-response paths that no emulator exercises.
  • Snapshot objects and the orphan sweeper are built and verified on moto (see Snapshot objects and the sweeper). Still open: their behavior on a real table and bucket, the sweeper's schedule and infra wiring, and the grace period to use in production.
  • Parking: a first-class "defer to computer" path (a pi-durable hook or a tool-level idempotency key) instead of replay: "safe" plus close.
  • Mailbox versus owner-free admission when no owner is live, and the concurrent-writer test for it.
  • Partition growth: local indexes cap an item collection at 10 GB per conversation. With index items under 400 bytes that is tens of millions of records, far beyond one channel.
  • Fewer commits per turn: the 100 ms partial throttle is pi-durable's; a longer one would cut S3 PUTs and index writes.
  • The DynamoDB-only baseline's document reads were not point-in-time: revision cleanup could delete a base under a concurrent reader (it now re-reads up to three times). The S3-plus-index storage never deletes, so its reads are consistent.
  • API stability of an experimental 1.0.x package, and how far we depend on internals (the storage format) rather than the documented interface.

Measured for the turn executor

Facts the TypeScript turn executor (conversation/src/turn/) relies on, each checked against pi-durable and pi-ai 1.0.2 on moto.

  • A scripted model. fauxProvider replaces every answer's usage with an estimate from the prompt, so a scenario cannot script token counts with it. conversation/features-support/fakes/scripted-provider.ts builds a provider with createProvider and createAssistantMessageEventStream instead: text streamed in small deltas, tool calls, a held request, and failures shaped like the OpenAI SDK's.
  • A failed model call settles unanswered. The settled input has status: "unanswered", reason: "model_error" and the provider text in detail (for example OpenAI API error (429): {"message":...,"code":"insufficient_quota"}). It is not an answer whose stop reason is error. Pi does not retry 401, 403 and 429 insufficient_quota; it retries 429 rate_limit_exceeded and transport failures (four calls with maxRetries: 2) and then settles the same way.
  • watchEvents order for one answer with a tool round. message_start and message_end (user entry), submission, run_start, turn_start, the assistant message (message_start, message_update batches of text_delta, message_end), tool_execution_start, tool_execution_end, turn_end, turn_start, the final assistant message, turn_end, run_end, submission (settled), usage_changed. An assistant message_start already carries the text streamed so far, and updates arrive about every 100 ms, so a consumer that ignores the start loses the first words and a fast model may deliver no update at all.
  • Delivery is asynchronous and lossy at the edges. Batches reach the listener on a microtask after the commit, stop() discards undelivered batches, and a throwing listener ends the watch. The executor waits for the batch that carries the last input's settlement before it ends the turn.
  • A stale owner never settles. After another owner raises the fence, the stale owner's submission.wait() stays pending forever. Detect it by the lease renewal (a compare-and-set on the attempt) and close the harness.
  • A poisoned session is visible only to the next call. After CommitOutcomeUnknown (or any commit failure after storage admission) Pi rejects every later call with Session is poisoned by a failed commit after storage admission whose cause is the original error, while the waiting submission does not settle. A commit with no writes (harness.commit(() => undefined)) is free and throws that error, so the executor probes with it on every pass and unwraps the cause chain.
  • Usage of a turn is the sum of the assistant entries from the prompt entry on (root.entries({ minEntryId })); pi.usage is per conversation and spans turns, so it cannot be diffed against a per-turn ledger counter.
  • Idempotent resubmission. submit with a requestId that already exists returns that submission whatever content is passed, so a resumed owner can attach to the prompt of an interrupted turn with an empty content.

Measured for the voice understanding step

The understand-the-user step is one non-durable completion outside a Harness (conversation/src/voice/understanding.ts). Verified against pi-ai 1.0.2 with the faux provider and with the real OpenAI provider whose HTTP fetch is replaced, so no request leaves the machine (conversation/test/voice-understanding.test.ts).

  • Call. models.completeSimple(model, { systemPrompt, messages: [{ role: "user", content, timestamp }] }, options) on a Models from createModels() with openaiProvider() set; model is models.getModel("openai", "gpt-5-nano") (api openai-responses, undefined when the provider does not serve it). No Harness, session or fence.
  • Minimal reasoning. options.reasoning: "minimal" sends reasoning: { effort: "minimal", summary: "auto" }.
  • JSON output has no named option. onPayload: (payload) => ({ ...payload, text: { ...payload.text, format: { type: "json_object" } } }) puts the JSON object format on the Responses request; the reply text is then parsed.
  • Timeout. options.signal: AbortSignal.timeout(10_000). An aborted or failed call does not reject: completeSimple resolves an AssistantMessage with stopReason: "error" or "aborted" and errorMessage, so the caller must check stopReason and throw.
  • Result. The text is the concatenation of message.content blocks of type: "text". Usage is message.usage.input and message.usage.output (plus reasoning, a subset of output), which map to the ledger's inputTokens and outputTokens.
  • maxTokens is ignored for gpt-5-nano. The model's compat marks max_output_tokens unsupported, so the request carries no cap (the Python call sent max_completion_tokens: 400).
  • Offline test seam. options.fetch replaces the provider's HTTP client; the Responses stream is server-sent events (response.created, response.output_item.added, response.content_part.added, response.output_text.delta/.done, response.output_item.done, response.completed with usage). The faux provider (features-support/fakes/scripted-provider.ts) covers the same call without HTTP.

The tool gate (verified in TS-26)

Measured with the scripted provider (toolCall(...) then reply(...)) against @earendil-works/pi-durable 1.0.2, in conversation/src/pi/gate.ts and the scenarios of model_tool_loop_sinks.feature.

  • The gate is a beforeTool hook, registered once with hook(ToolTask, { beforeTool }) in its own extension. It sees every tool of every extension. wrapTool is the other seam (it decorates one tool's execute), but execute runs after the intent is recorded, so a gate there leaves a started tool behind. The hook runs before intent: a blocked call leaves no trace of having started.
  • Order inside the tool task. The tool is resolved first (an unregistered name never reaches the hook: the model gets tool_unavailable), then the arguments are validated against the TypeBox schema, then beforeTool runs. A tool the gate must judge therefore has to be registered, even when it can never execute (send is registered with replay: "unsafe" and an execute that throws).
  • A block is a tool result, not an error. { block: reason } makes Pi append a pi.tool-result entry the model sees as Tool call blocked: <reason> and the generation continues: the next model request answers the denial. The listener receives tool_execution_end for the call and no tool_execution_start; createAgentEventListener writes the missing tool.call before the tool.result, so the journal always pairs them by action_id.
  • The result text is wrapped. The turn-event body of a blocked call is <harness>\n[error] Tool call blocked: <reason>\n</harness>, because Pi renders the diagnostics into the content. Match on the substring.
  • The hook runs per call, so it reads state per call. The gate reads the turn's grant item every time; a member's PUT /turns/{id}/grant during a turn applies from the next call. A throw inside the hook blocks the call with the error text, so a failure to resolve the member's standing is reported as a denial, never as an allowed call.
  • Approvals. The hook cannot wait. A consequential tool whose grant and standing allow it is blocked with immutable approval required; the approval flow is a later re-request, as above.

The computer handoff (verified in TS-29)

Measured with the scripted provider on moto against @earendil-works/pi-durable 1.0.2, first in the prototype conversation/test/computer-park-resume.test.ts and then through the production executor (features/computer_action_handoff.feature). The code is conversation/src/pi/computer-tools.ts (the tools), conversation/src/turn/park.ts (park, resume, probe settlement) and conversation/src/domain/actions.ts (the action).

  • The park is a tool execute that never settles. It looks the Pi call id up, finds no answer, signals the executor and awaits the invocation's abortSignal. The tool is registered replay: "safe" because that execute is idempotent. The executor records the action, parks the turn, and closes the harness with Harness.close.
  • Close is not abort, again. After the owner closed, the pi.tool task stayed at its execute checkpoint and the generation stayed waiting on tools. The closed owner's own submission.wait() rejects with Harness is closed (an owner that is not closed but has been fenced out never settles, as before, and its next write fails with OwnershipLost; a commit with no writes is free and does not throw).
  • The call id is stable across owners. api.callId is the same on the new owner: the lookup log was [callId, callId], one per owner. The action is therefore keyed by it (act#{call_id} index item) and a resumed execute finds the host's answer by it.
  • The next owner needs no explicit resume. Harness.open with the next fence, root(), and submit with the same requestId (the turn's turn:<id>, empty content) enable scheduling and the parked tool task runs execute again. It ran once on the new owner and the generation went on to the answer.
  • The new owner sees tool_execution_end with no tool_execution_start for the resumed call (the first event of its stream, then message_start, message_end, turn_end, turn_start, the answer). The journal writer must not write a second tool.call for it: the resumed attempt reads the journal's tool.call action ids first (AgentEventDelivery.claimCallJournal).
  • A throw inside execute is a tool error, so a crash cannot be injected there. Parking therefore signals the executor, which creates the action and parks outside Pi, where crash windows (computer_action, computer_park, computer_start) are real. An answer the host marked as an error is thrown from execute so the model reads it as a tool error ([error] ...).
  • Policy answers stay inside execute. A spend ceiling refusal is returned as the tool result when no action exists yet; once an action exists the ceiling is not asked again, so a call already parked is not refused later.
  • Cost. The two-owner prototype takes about one second on moto, the same as the plain two-owner turn.

Spike: handoff to a computer owner

Status: the recommendation below was built and rehearsed on development (TS-65 to TS-69). This section is the record of the spike's measurements; the built design, its proof and its limits are in Pi session handoff.

Measured 2026-10-07 against @earendil-works/pi-durable and pi-ai 1.0.2 on moto 5.2.3. The code is the throwaway spike in spikes/pi-handoff; the raw output of every run quoted here is in its results/ directory. Nothing in conversation/src changed. No AWS access, no image push.

The question: can a Pi session that a Lambda-like owner (A) started be handed to a container-like owner (B) that runs Pi's own coding tools on its own disk, and then be handed back to a Lambda-like owner (A2)? Every owner is a separate Node process. B also ran inside a node:22-bookworm-slim arm64 container. Two scenarios ran:

  • Raw: pi-durable plus the production fenced IndexedStorage. A scripted model (faux), and one run with gpt-5-nano.
  • Executor: the production consumeRunJob and TurnAttempt, with the same Dynamo stores the TurnExecutor Lambda builds, a seeded bot and channel, and the front door's own postMessage. SQS is replaced by recorders.

Answers

Q1. Can B take over mid-turn with a different tool implementation? Yes, with one rule.

  • Fences and commits (raw, scripted): A fence 1 wrote seq 1-8 and parked. B fence 2 wrote seq 9-39. A2 fence 3 wrote seq 40-46. With gpt-5-nano: 1-10, 11-45, 46-53. Executor: 1-10, 11-40, 41-48.
  • At the park A's tasks were pi.generation waiting/tools and pi.tool running/execute. B read the same tool task as pending/execute. B ran the call with its own write (journal: tool.park in A, then tool.run.local in B, same call id call-1). The file appeared on B's disk. B saw tool_execution_end for the call and no tool_execution_start, as before.
  • pi-durable does not compare tool implementations. It keeps the checkpoint { phase: "execute", arguments, replay } and, on reopen, reruns the call only if the stored policy and the current registration are both replay: "safe". Four B variants:
B registers Result for A's pending write
Same names, replay: "safe" (safe) B ran it. Task completed. File written.
pi-durable's CodingTools as shipped (unsafe) Not rerun. The model got [error] Tool write was interrupted and may have partially run. No file. The later bash and edit calls ran in an empty workspace and failed.
No tools (missing) Same interrupted result. Later calls got Tool bash is not available.
write with another schema, an extra required mode (schema) B ran it with the stored arguments. The new schema was not applied to the stored call.

So the handoff needs B to register the same tool names with replay: "safe". The shipped tools default to unsafe.

Q2. Where do Pi's built-in coding tools come from? From pi-durable itself, already installed.

  • @earendil-works/pi-durable/tools exports CodingTools (an Extension with read, write, edit, bash) and createBashTool({ commandPrefix, prepare }). They are ToolRegistrations. They do their I/O through api.env, an ExecutionEnv (file system plus shell). @earendil-works/pi-durable/env/node provides NodeExecutionEnv({ cwd, shellPath, shellEnv }). Harness.open(storage, { models, registry, env }, ctx) builds the environment per call. Without env the tools return an error result. They register the way our computerToolsExtension does: registry.install(extension).
  • @earendil-works/pi-coding-agent is not installed at the root. It is published as 1.0.4 (the root pins 1.0.2). It exposes createCodingTools(cwd, options) with pluggable BashOperations, ReadOperations and so on, but its tools are AgentTools of pi-agent-core, not pi-durable ToolRegistrations, so they need an adapter. It pulled in 121 packages and 159 MB (pi-tui, pi-mcp, pi-codemode) and wants pi-ai and chord ^1.0.4, which do not match the root's 1.0.2. I installed it exact-pinned in spikes/pi-handoff/coding-agent-probe only. Its tools ran (write and bash in a temp directory). I did not use it. Use pi-durable's own tools.
  • The tools are rooted by the environment's cwd, not by a constructor argument. They do not confine paths: absolute paths outside cwd are allowed.

Q3. Does the existing executor run in B with only the tool set swapped? Yes, with a one-line seam.

  • TurnAttempt.execute hard-codes computerToolsExtension(this.computerHandoff()) in its extension list, so the tool set cannot be swapped from outside. The spike generates a copy of executor.ts with that one line replaced by an optional deps.computerToolsOverride (results/executor-seam.diff: one added import, one changed line). Nothing else changed: consumeRunJob, the drive loop, the gate, the mailbox, the journal, the finalizer.
  • Evidence (results/phase1-executor-host.txt, results/phase2-container-executor.txt): A ran the production executor unchanged and ended parked (attempt 1, fence 1, action requested). B resumed the turn (resumeTurnForAction), ran the seamed executor and ended done (attempt 2, fence 2, turn completed). A2 ran the unchanged executor for the next turn (fence 3, done).
  • B's tools keep the names and schemas of computerToolsExtension. The gate and the grant know those names; Pi's native names read, write and bash are not in the gate's vocabulary and would be denied as an unknown capability. Each call goes through the existing action ledger: look up the action by call id, claim it, run the pi-durable tool, complete it. A's parked write_workspace action went requested to done with claimedBy: container-owner-b. Calls that B started itself got new action records. This stops a host from running a requested action a second time.
  • The gate ran in B unchanged. It blocked run_terminal on the first attempt because the default household grant does not grant it (Tool call blocked: tool "run_terminal" is not granted). The scenario sets a grant that includes it.
  • What assumes a Lambda, and what B must supply:
    • deps.remainingMilliseconds (Lambda context). Optional. B passes none, so the executor never yields.
    • deps.runVisibility.extend (SQS ChangeMessageVisibility with the record's receipt handle). B is not the SQS consumer of that record. B needs a no-op, or its own job source with its own visibility rule.
    • The acknowledge argument of consumeRunJob. In the Lambda it does nothing (SQS deletes on return). B must delete its own message.
    • turnRuns and turnProbes (SQS send): the executor arms a deadline probe on every claim and enqueues on yield and on resume. B needs SQS send rights, or recorders.
    • computer.computerStarts: only used when a turn parks. B does not park.
    • resolveOpenAiApiKey() (SSM) lives in the Lambda handler, not the executor. B only needs OPENAI_API_KEY in its environment, or the proxy in Q4.
    • openOwnerSession does not pass env to Harness.open. The spike injected the environment through a proxy of the tool API. The product should add an env option.
    • A parked turn has waitingFor set and no attempt. Nothing can claim it until the wait is cleared. B called resumeTurnForAction itself. In the product the control plane or the host must do it before B claims.

Q4. What does B need? See the table and the proposal below.

AWS calls that B made, recorded by a middleware on its clients (aws.surface in the results):

Service Resource Calls B made
DynamoDB Conversations table (pi-durable index) GetItem, Query, TransactWriteItems, UpdateItem
DynamoDB Messaging table (turns, mailbox, actions, ledger, budgets, policy, bots, channels) GetItem, Query, BatchGetItem, TransactWriteItems, UpdateItem, DeleteItem
S3 conversations/<storageId>/commits/<seq>-<fence>.json GetObject, PutObject
SQS TurnRuns, TurnProbes, ComputerStarts (recorders in the spike, so not exercised) SendMessage, ChangeMessageVisibility
SSM the OpenAI key parameter (Lambda handler only) GetParameter
Model OPENAI_API_KEY in the environment, read by pi-ai HTTPS to api.openai.com

Two checks of what a model-chosen shell command can reach (results/q4-*.txt):

  • pi-durable's shipped bash passes the owner's environment to the command. With OPENAI_API_KEY and AWS_SECRET_ACCESS_KEY set in the owner, echo $OPENAI_API_KEY printed the value. The prepare hook of createBashTool can set inheritEnv = false and replace env, and then the command saw nothing.
  • That is not enough. In the container a child shell of the same user read the owner's secrets from /proc/$PPID/environ.

Smallest design that keeps the keys away from model-chosen commands (not built):

  1. Model-call proxy on the control plane. A route (Lambda function URL) that holds the OpenAI key, accepts a per-session bearer token (tenant, bot, turn, expiry of at most 15 minutes, signed by the control plane), forwards to OpenAI and writes the vendor ledger. B is configured with the proxy base URL and the token. B never holds the model key. This needs the pi-ai OpenAI provider to accept a base URL override. I did not test a real override.
  2. Storage credentials scoped to the conversation. At start the control plane calls sts:AssumeRole with session tags (the role's static policy reads them; see PI_SESSION_HANDOFF.md) and passes the temporary credentials (15 to 60 minutes) to B: S3 Get and Put on conversations/<encoded storageId>/* only, and DynamoDB on the Conversations table with dynamodb:LeadingKeys equal to the conversation's partition key. The Messaging table is the weak point: the executor uses its Dynamo stores directly, so B needs the tenant's turn, mailbox and action rows. Scope it with LeadingKeys on the tenant prefix. That is tenant isolation, not least privilege. The better end state replaces B's Messaging access with the existing host protocol over HTTPS (computer/host/src/protocol-client.ts). That is a larger change.
  3. The shell runs as another user. The owner process runs as owner, with the credentials in memory or in a file of mode 0400. Every bash call runs as an unprivileged agent user (through prepare and setpriv or su) with a scrubbed environment, /proc mounted hidepid=2 or a separate PID namespace, and /workspace as its only writable path. On Fargate also block 169.254.170.2 for the agent user (an iptables owner match), because the task role's credentials are served there to every process. A stronger variant runs the shell in a nested sandbox (gVisor or a second container). None of this was built or measured.

Q5. Round trip back to a Lambda-style owner: yes, with the full history.

A2 (fence 3) opened the same storage and answered "What did you just do in the workspace?". Its model request, recorded by the scripted provider, was user, system, assistant, toolResult, assistant, toolResult, assistant, toolResult, assistant, user. The tool results were B's, from the container disk: Successfully wrote to hello.txt, total 4 ... PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" git version 2.39.5 hello from Chatticus, Successfully replaced 1 block(s). The same held through the executor (a second turn, A2 on fence 3) and for gpt-5-nano (A2's answer described the work that B did).

Phase 2: B inside the container

  • Image pi-handoff-spike:phase2: arm64, node:22-bookworm-slim (Node 22.23.3, Debian 12) plus git 2.39.5, user node (uid 1000), esbuild bundles of both owner entry points. 359 MB: base about 247 MB, git layer 105 MB, bundles 7.5 MB (two bundles of 3.5 and 3.9 MB; the pi-ai providers are bundled).
  • Evidence from the container (results/phase2-container-raw.txt): whoami: node (uid 1000), PRETTY_NAME="Debian GNU/Linux 12 (bookworm)", git version 2.39.5, -rw-r--r-- 1 node node 21 ... hello.txt, content HELLO from Chatticus. docker cp read the file from the stopped container, and the host workspace directory stayed empty. B's bash tool printed those values inside the model's tool result, and A2 on the host then quoted them.
  • Cold start of B, 3 runs over a user-defined Docker network to moto: docker run to node start 0.59 to 0.64 s, node start to storage open 0.16 to 0.20 s, storage open to the first tool result 0.42 to 0.51 s, docker run to the first tool result 1.17 to 1.32 s. The whole B run (three tool calls, scripted model) took 4.7 to 5.3 s. A bare docker run --entrypoint true takes 0.75 to 1.05 s on this Mac. These numbers include no image pull and no compute start. The real cold start of a Fargate task (tens of seconds) was not measured.
  • Executor in the container: A parked, B done in 5.1 and 6.9 s (two runs), A2 done; three tool calls, three ledger records.

What did not work, and what is unknown

  • host.docker.internal was unreliable. On Docker Desktop, with the Mac under load (load average 15 to 58), about 8 of 21 container runs failed or stalled: connect ETIMEDOUT 192.168.65.2:5622, or a request hanging past 15 s. One of 200 plain fetches timed out, and a request took 25 to 36 ms. Over a user-defined bridge network to a moto container the same loop had 0 failures in 200 and 5.4 ms per request, and 5 of 5 later container runs passed. The runners use the bridge when SPIKE_DOCKER_NETWORK is set. Moto is not AWS, so none of these latencies predict AWS.
  • Two host-process B runs also stalled ("did not settle in 60 s") around the time another session restarted the shared Docker daemon and moto disappeared. I attribute them to that. They did not recur in the later host runs. Not proven. The storage clients have no request timeout; production should set one.
  • Not tested: real AWS and Fargate; B crashing in the middle of a bash call (the ledger lease and INTERRUPTED_ACTION_RESULT should answer it, but I did not kill B); two B owners racing for the fence; a real OpenAI base URL override for the proxy; the unprivileged-shell isolation; image size with the production computer's tools; B reading its job from SQS.
  • The executor seam is a generated copy, not a patch to conversation/src. The product change is the same one line.
  • pi-durable 1.0.4 and pi-coding-agent 1.0.4 exist; the repository pins 1.0.2. The spike did not test 1.0.4.

Surprises (pi-durable and Pi)

  1. pi-durable ships its own read, write, edit, bash and an ExecutionEnv. The coding-agent package is not needed.
  2. Those tools default to replay: "unsafe", so a handoff does not run the parked call. The model gets "interrupted", and the work is lost without a visible error unless B overrides the policy.
  3. Replay does not re-validate stored arguments and does not compare implementations. A changed schema is ignored for the pending call. A missing tool turns the pending call into interrupted, not unavailable.
  4. The bash tool returns no content. Its output arrives only through api.output(). A wrapper that needs the text must capture it.
  5. The shipped bash inherits the owner's whole environment, and a same-user child can read /proc/$PPID/environ.
  6. Changing the tool set between owners adds a positional system entry to the transcript.
  7. Pi's call ids from OpenAI contain |fc_.... The scripted provider's call-N ids collide across processes; the spike offsets them.
  8. Harness.open takes env, but our openOwnerSession does not pass it.
  9. After a handoff the generation sits waiting on its tool task until some owner runs it. Nothing in pi-durable says "deferred to another executor". The only signal is replay: "safe" plus a closed owner, as in the TS-29 section above.

Recommendation: go, in steps, with the security design as a gate

Go. The handoff works across processes and in a container, the executor core needs one seam, the history survives the round trip, and the start of the owner itself is about one second on top of the compute start. No-go for shipping B with a model key or broad storage credentials in its environment: the security design in Q4 comes before any production use.

Minimal product design, in order:

  1. conversation/src (behavior first, in features/). A scenario "a computer owner finishes a parked turn". Then: an optional computerTools factory on ExecutorDeps in place of the hard-coded extension; an env option on openOwnerSession; a local-tools extension with the names and schemas of computerToolsExtension, replay: "safe", each call wrapped in the action ledger (claim, run, complete); deps.runVisibility and acknowledge supplied by B's job source. Keep one path: the parked-action protocol with an external host is replaced, not kept beside it.
  2. computer/host. An owner entry point in the image: reads a run job (SQS, or a lease over the host protocol), builds the deps from scoped credentials, calls resumeTurnForAction and consumeRunJob, and runs bash as the agent user. Add the esbuild bundle (about 4 MB) to the image; git is already there.
  3. Control plane. Mint the per-session token and the scoped STS credentials when a turn is handed to a computer. Add the model proxy route. Stop starting a host only to run actions.
  4. infra/. A role for the computer owner with no standing permissions beyond sts:AssumeRole to the scoped role; the scoped role with LeadingKeys and S3 prefix conditions; the proxy function URL; an SQS queue (or a lease route) for computer-owner jobs; remove the OpenAI key from anything the computer can read.
  5. Cutover. Delete the host-protocol action executors once B runs the tools. Run the conformance suite and the handoff scenarios against pi-durable 1.0.2 in CI.