anbot

78ef64a / SPEC.md

13253 bytes

# ANBot-DSA self-hosting specification

The first ANBot-DSA node must publish its own committed source as signed ANProto
records and AndFS blocks. Another machine must be able to clone it with ordinary
Git, start an independent node from the clone, and publish the same source again.

## Publish locally

A node keeps an Ed25519 identity and a durable data directory outside the source
checkout. Identity creation and publication work offline.

Provide a CLI command that publishes a local branch from a SHA-1 Git repository.
Resolve the branch tip once and create a self-contained Git bundle advertising
that exact branch and commit, with all reachable history and no prerequisites.
Preserve native Git object IDs. Store the bundle in AndFS and write signed
`git-repo` and `git-update` records. Reuse the repository record for later
publications to the same clone URL.

After storing and verifying the bundle and its records durably, atomically select
the update by its record ID. Keep that selection in durable state. An interruption
before selection commits must leave the previous update usable and may leave
unselected records or blocks. Selection is the publication commit point. A fresh
CLI invocation may prepare a new signed update.

On success, print the repository ID, update ID, commit ID, and clone URL.

## Signed records

Use the existing ANProto envelope. Here `B64` is canonical padded standard
Base64, `H` is SHA-256, and `||` concatenates bytes.

- `author = B64(publicKey)`, with a 32-byte Ed25519 public key.
- `contentHash = B64(H(UTF8(content)))`.
- `payload = UTF8(timestamp + contentHash)`, with exactly 13 ASCII decimal digits
  for the Unix timestamp in milliseconds.
- `proof = Ed25519.sign(privateKey, payload)`, a 64-byte signature.
- `signature = author + B64(proof || payload)`.
- `id = B64(H(UTF8(signature)))`.

Store each record as a JSON object with string fields `id`, `signature`, and
`content`. Preserve the exact decoded strings. Content is valid Unicode encoded
as UTF-8, up to 65,536 bytes. To verify, check field types and canonical encodings,
split the 208-character envelope after the 44-character author, decode the
remaining 121 bytes, and verify the proof against the 57-byte payload. Check the
timestamp and hash syntax, recompute the content hash and record ID, and reject
any mismatch. Read the timestamp as a signer assertion. Explicit local
publication selects the served update.

The `content` string contains one of these JSON objects:

```json
{"anbot":1,"type":"git-repo","name":"anbot"}
```

```json
{
  "anbot": 1,
  "type": "git-update",
  "repo": {"format": "anproto", "id": "<git-repo record ID>"},
  "ref": "refs/heads/main",
  "objectFormat": "sha1",
  "commit": "<40 lowercase hexadecimal characters>",
  "bundle": {"format": "andfs-v1", "id": "<manifest hash>"},
  "previous": "<git-update record ID or null>"
}
```

The `previous` field is optional: it names the `git-update` record this
update follows, or is null for the first update of a repository. Records
without it are chain roots. Readers must ignore the field except when
walking the update chain during sync.

The repository ID is the `git-repo` record ID. Both records must verify
under the repository's publisher key: the author of the `git-repo` record.
A node signs its own publications with its publishing key; a node serving
records synced from another node verifies them against the publisher key,
not its own identity. The update must refer to the
repository served at the clone URL. Its `ref` must start with `refs/heads/`
and pass Git branch-ref validation. Reject duplicate member names in both
record containers and signed JSON content.

## AndFS bundle storage

Split the bundle into consecutive 262,144-byte chunks. Only the final chunk may
be shorter. Hash each chunk with SHA-256 and encode the hash as unpadded Base64url.
Store the chunks by hash and write this compact UTF-8 JSON manifest in the shown
property order:

```json
{"andfs":1,"size":N,"chunkSize":262144,"chunks":[...]}
```

Here `N` is the bundle size in bytes and `chunks` is the ordered list of chunk
hashes. The manifest ID is the unpadded Base64url SHA-256 hash of its exact bytes.
Store it by that ID. An empty file uses size zero and an empty list. A manifest
can contain at most 16,384 chunks and occupy at most 1 MiB.

On recovery, hash the original manifest bytes and validate its structure before
using it. Require `andfs` to equal `1`, a nonnegative integer size, the fixed
chunk size, and exactly `ceil(size / 262144)` chunk hashes. Verify each chunk's
hash and expected length and the reconstructed bundle size. Missing or corrupt data makes recovery fail.

## Serve one branch

Expose one branch for public reads over Git smart HTTP. The stable clone path is
`/git/<repository-hash>.git`, where the repository ID is encoded as unpadded
Base64url. Accept a nonempty, case-sensitive hash prefix if it uniquely
identifies a served repository. The MVP serves one repository per node. If a
node serves several repositories, an exact ID takes precedence and an
ambiguous prefix must be rejected. The first
publication sets the branch for that URL. Later publications can advance its
selected update at the same URL. Advertise only that branch and set the symbolic
`HEAD` to it so ordinary `git clone` checks it out. Support ordinary `git fetch`.

A bare repository may serve as a cache. Rebuild it from the durable selection,
signed records, manifest, and chunks. Before serving an update, verify its
records and blocks and use Git to verify bundle integrity, object connectivity,
absence of prerequisites, and the exact advertised branch and commit. Serve
without checking out published code or running repository hooks. The selected
publication remains recoverable after restart with the cache and original
checkout unavailable.

## Peer record and block exchange

A node syncs signed records and AndFS blocks directly from another node over
HTTP, verifies them, and serves the same history without a Git clone in
between. Sync is per-interest: follow the head of the repository you care
about, walk back until you reach records you already hold, then stop.

The server exposes two read-only endpoints next to the Git endpoints. Both
carry `Access-Control-Allow-Origin: *` so browser peers can fetch them
cross-origin. They serve only records and blocks the node holds, and never
accept unverified data.

`GET /gossip/poll?since=<cursor>` returns records after the cursor, paginated
at 100 records per page, in (payload-timestamp, record ID) order, shaped like
the Wiredove relay poll:

```json
{"records":[{"id":"...","signature":"...","content":"..."}],"nextSince":"..."}
```

The stream carries git records and signed social posts together: a post is
a record whose content is `{"anbot":1,"type":"post","text":"..."}`, with an
optional `replyTo` field naming the parent post's record ID, and text of at
most 4096 UTF-8 bytes. Both kinds carry verified payload timestamps, so the
same cursor pages across them.

The cursor is opaque Base64url JSON `[timestamp, recordId]`, where `timestamp`
is the 13-digit payload timestamp from the verified signed payload. `nextSince`
is the cursor for the following page, or null at the end of the records. An
invalid cursor is a 400. The server keeps no index: it scans its record and
post stores, verifies each record — posts against the post rules — skips
unverified ones with a log line, sorts the verified records, and filters by
the cursor, so the cursor stays correct across restarts.

`GET /blocks/<hash>` returns manifest or chunk bytes by hash. The hash must
decode to 32 bytes of Base64url; the bytes are verified against the hash
before serving. Responses carry `Cache-Control: public, max-age=31536000,
immutable`. Missing or malformed hashes are 404; corrupt bytes are 503.

The pull client verifies the envelope and content of every record before
storing it, deduplicates by record ID, and keeps only records attributed to
the repository being synced: the `git-repo` record with the repository ID and
`git-update` records whose `repo.id` matches it. Records that are posts are
verified against the post rules and stored in the post store, separately
from git records. Posts have no chain and no blocks: they never affect the
update walk, bundle verification, or selection advancement, and a pull that
brings only new posts still stores them. It walks the update chain
from the newest update via `previous` until it reaches an update already held,
fetches the manifest and any missing chunks only for the new chain segment,
and verifies the manifest structure, each chunk hash and length, and the
reconstructed size. The selection advances only after the bundle verifies:
on Node, through the existing materialize path; in the browser, by verifying
the manifest and chunks and holding the bundle bytes. Partial failure never
advances the selection, and unverified bytes are never stored under a verified
ID. The client keeps no cursor between pulls: every pull starts from the
beginning and deduplicates against held records.

A browser peer syncs, verifies, and holds records, posts, and blocks. It does not
materialize Git repositories or serve Git HTTP. Its store keeps the
verify-then-select rule, but browser storage has no fsync and no-overwrite is
best-effort across tabs; see `store-browser.mjs`.

## Web UI

Serve plain HTML pages for the published branch at the server root, alongside
the Git endpoints. The pages are read-only, use no JavaScript, and render from
the same verified bare repository that serves Git requests. Provide a
repository overview with the branch, HEAD commit, recent commits, and root
file listing; directory listings; file contents; and one commit's message and
changed files, each linking to that file's diff in the commit and to the
file's contents at the commit. Escape all repository content in the HTML. Reject paths that
escape the repository tree. Missing files, directories, and commits return
404. Public reads require no credentials.

## Authorized pushes

### Submit a post

`POST /posts` accepts a signed ANProto post record as JSON, up to 32 KiB.
The server verifies the post and checks its signer against the same allowlist
used for pushes. It stores the post before returning `201` with its ID. An
identical retry returns `200` with the same ID. An invalid signature returns
`400`; a verified but unapproved signer gets `403`. Replies keep their
signed parent IDs and can arrive before their parent.

### Publish a Git update

Push support is opt-in. The node accepts a Git push only after the agent proves
possession of an allowlisted ANBot Ed25519 private key. Authorization uses a
short-lived, single-use challenge over HTTPS. The agent signs the UTF-8 compact
JSON array `["anbot-git-push-v1", origin, repoId, nonce]`. The node verifies
the signature and issues a short-lived token for Git's write requests. Each
write request checks that the key is still allowlisted.

Receive into a disposable bare repository. Allow only fast-forward updates to
the served branch. Verify and publish the resulting bundle through the local
publication path before acknowledging a successful update. Use the server's
publishing identity for the resulting ANProto records. Receiving commits never
checks out or executes their content.

For a push that advances the branch, the agent signs a `git-push` record
containing the repository ID, branch ref, and proposed commit. The server
checks the record against the authenticated agent and the received tip. The
server's signed Git update embeds that record, the count and subjects of new
commits (up to 20 subjects), and the previous update ID. The feed follows the
selected publication history, so interrupted, rejected, and up-to-date pushes
produce no announcement.

## Acceptance

Use the actual committed ANBot source. Include startup instructions, runtime
requirements, and dependency locks or vendored dependencies in that source.
Document setup downloads and verify local operation offline after setup.

1. Publish from node A. Restart A with its cache deleted and its original source
   checkout unavailable. Clone from A with ordinary Git into an isolated
   environment and verify that the checked-out HEAD matches the published commit.
2. Start node B from the clone with a fresh identity and independent data
   directory. Publish B, take A offline, and clone B with ordinary Git. Verify
   the same commit and history survive. B may assign its own ANBot repository ID.
3. Make a development commit in B's checkout and publish it. Fetch from B at the
   same URL and verify the new commit and its ancestry.
4. Interrupt publication before the selection becomes durable. Restart with the
   cache deleted and verify that a fresh clone still gets the previous selected
   commit. Retry publication and verify that a clone gets the newly selected tip.

At handoff, record the source commit and the exact start, publish, clone, and
fetch commands with their observed results.

Following, historical adapters, and multi-publisher import follow this
milestone. Peer record and block exchange is specified above; the bootstrap
transfers source between nodes through Git or through sync.

Git · Social feed