# Computer snapshots

> A Chatticus **computer** is a workplace identity (`computer_id`), not a particular Mac, Fargate task, or EC2 instance. Any machine that can run the computer image is a **host**. A garage Mac is a host in the same sens...

Canonical: https://chattic.us/wiki/computer-snapshots/

A Chatticus **computer** is a workplace identity (`computer_id`), not a
particular Mac, Fargate task, or EC2 instance. Any machine that can run the
computer image is a **host**. A garage Mac is a host in the same sense Fargate
is a host. It is cheaper when it is on. It is not a different kind of
computer.

Hosts do not share a running container. Live migration is out of scope. The
durable workplace is a **snapshot in S3**. A host that should run the
computer **hydrates** that snapshot onto local disk, runs, then **publishes**
again before someone else takes over.

For how hosts are chosen from declared work kinds (manifold; not implemented),
see [Computer manifold](/wiki/computer-manifold).

## What is stored where

The Ubuntu image (Xvfb, Chromium, worker, agent) lives in a **registry**
(ECR). Every host already pulls that image. Do not copy OS layers from a Mac
to AWS or back.

The snapshot is only the **durable workplace**:

| Object | In the snapshot | Not in the snapshot |
| --- | --- | --- |
| `/workspace` | yes | |
| Chromium profile (cookies, logins) | yes | |
| Image layers / apt packages | | registry image; treat extra packages as replaceable |
| Bot memory, chats, skills | | DynamoDB |
| Secrets | | Secrets Manager |

Canonical layout:

```
s3://chatticus-{env}/tenants/{tenant_id}/computers/{computer_id}/
  snapshot.tar.gz       workspace + browser profile (v2: one gzip member per file)
  index/{checksum}.json every entry: path, kind, size, mode, sha256, link target, member offset and length
  manifest.json         image digest, checksum, index_checksum, published_by, published_at
```

**Snapshot v2.** The pack is still a valid `.tar.gz`, so hydrate unpacks it
exactly as before, but each file is its own gzip member inside the tar stream.
The index (`conversation/src/snapshot/index-format.ts`) records, for every
entry, the byte range of its member, so the Front Door can list a computer's
files and read one file with a single ranged `GET` while the computer is off.
A publish writes the pack, then the index, then the manifest, which is written
last and names the index (`index_checksum`, the SHA-256 of the index bytes). A
manifest without `index_checksum` is a v1 snapshot: it still hydrates, and the
file routes answer `409 snapshot_not_indexed` until the computer publishes
again or `conversation/bin/reindex-snapshots.ts` runs. The pack is streamed to S3
as a multipart upload, so the owner's memory does not grow with the workspace.
Directories named `node_modules`, `.next`, `dist`, `build`, `target`, `.venv`,
`__pycache__` and `.cache` (at any depth), and any file over 50 MiB, are left out
of the pack and the index; a skipped file is logged as `snapshot_file_skipped`.
A publish scans size and modified time against the last index and hashes only
files that moved, so an unchanged disk is not packed again.

`manifest.json` names the image digest the snapshot was taken against. A
host whose local image does not match still hydrates the workplace files; it
does not docker-load a private OS.

## Why not "just leave it in S3 and mount it"

S3 is the object every host can **see**. It is a bad live disk for Chromium
and overlayfs: high latency, no POSIX semantics that a browser profile
needs, and concurrent writers from two hosts would split the brain.

Each running host has a **local cache** (Docker volume, bind-mount, or EBS).
S3 is the last published checkpoint. Using a computer without copying bytes
onto that host is not a goal. Seeing the same checkpoint without
administrator-to-administrator copies **is**.

EFS can back `/workspace` on AWS hosts that share a VPC. It does not attach
to a home Mac. The Mac still hydrates from S3. Do not design two durable
stores. S3 is canonical. EFS, if used, is an AWS-side cache of the same
files.

## Publish, hydrate, relocate

No magic mover. Three administrator-or-worker operations:

1. **Publish.** The host that currently has the live disk stops writes, packs
   `/workspace` and the browser profile, uploads to the computer's snapshot
   URI, and records the checksum. The computer is no longer dirty.
2. **Relocate.** An administrator names the next host. The control plane
   does not copy a container. It records `intended_host_worker_id` and
   `hydrate_required`. Turns for that computer go only to that host until it
   hydrates. Other hosts of the same `computer_id` do not run it in the
   meantime (no split brain).
3. **Hydrate.** That host downloads the snapshot (or hits a local cache with
   the same checksum), unpacks onto its volume, and clears
   `hydrate_required`. Prefer-local ranking resumes.

Copying "a bunch of containers from AWS to my Mac" is this sequence aimed
at the Mac: publish on AWS if needed, relocate to the Mac worker, the Mac
hydrates. The other direction is the same with Fargate or EC2 as the target.

Failover when a prefer-local Mac's heartbeat dies is the same hydrate path on
an AWS host, without an administrator picking the target. That is not live
migration. It is "start from the last snapshot." Work that was never
published is gone, same as an unsynced volume.

## Dirty disk

Writes on a host mark the computer **dirty**. Relocate is rejected until the
current host publishes. That is the administrator safety: do not point the
Mac at S3 and throw away the Fargate volume's last hour of cookies.

An explicit discard (not v1 kernel) is the only way to relocate without
publishing.

## What we will not build

- Live container move (CRIU, `docker checkpoint`, rsync of overlay2 while
  Chromium is running)
- `docker save` / `docker load` of the whole image as the relocation story
- Mounting the snapshot bucket as the container root
- Two hosts running the same `computer_id` with two live disks

The control-plane kernel records snapshot URI, checksum, dirty, intended
host, and hydrate-required. Hosts pack `/workspace` and the browser profile
into `snapshot.tar.gz` plus `manifest.json`.

A **filesystem object store** is the local stand-in. Production uses S3:

| Organization | Bucket | Who creates it |
| --- | --- | --- |
| Anthus-managed | CDK stack ``ChatticusSnapshots`` | Anthus deploy |
| Customer-owned | ``chatticus-snapshots-{OrganizationId}`` in the customer account | Customer runs ``infra/customer-role.yml`` |

Hosts set ``CHATTICUS_SNAPSHOT_BUCKET`` to the bucket name for their
organization. Anthus-managed hosts read the CDK ``SnapshotBucketName``
output. Customer Fargate hosts read the value wired by the
``ChatticusComputers`` stack parameter of the same name. Do not create
either bucket with the AWS CLI.

Object keys keep the canonical prefix
``tenants/{tenant_id}/computers/{computer_id}/`` inside whichever bucket
holds the organization.

If ``CHATTICUS_SNAPSHOT_BUCKET`` names a bucket that does not exist yet,
hydrate and publish skip without crashing the host. The customer must
re-run or update the cross-account template to create the bucket.

```bash
cd infra
npm install
npx cdk bootstrap
npx cdk deploy ChatticusSnapshots
export CHATTICUS_SNAPSHOT_BUCKET=<SnapshotBucketName>

node conversation/bin/snapshot.ts pack \
  --live-root ./var/hosts/fargate \
  --store s3 \
  --tenant anthus \
  --computer household-computer \
  --worker fargate-1

node conversation/bin/snapshot.ts hydrate \
  --live-root ./var/hosts/garage-mac \
  --store s3 \
  --tenant anthus \
  --computer household-computer
```

If the Mac already has a cache whose checksum matches the manifest, hydrate
does not download the pack again.

Inside the computer image, `/workspace` is a symlink to
`/var/lib/chatticus/computer/workspace`. Do not give two hosts the same live
volume. They share the snapshot store only.

Prove relocate with two running containers:

```bash
sh computer/test_relocate.sh
```