t5de-catalog: store and serve community product files

A Go server that takes product uploads with hub upload tokens, checks them by t5de-api's pack rules, stores files by sha256, registers revisions with the hub (POST /api/v1/catalog/revisions), and serves files with immutable caching; files used only by adult revisions need an adult access token. Contract: t5de-hub's docs/products.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 18:59:16 -07:00
commit 8742829e77
@@ -0,0 +1,75 @@
+# t5de-catalog
+
+T5DE's catalog server: it stores community products' files by sha256, serves them, and registers uploads with
+T5DE's hub (git.t5de.dev/t5de/t5de-hub). Creators upload from the hub's website straight to a catalog, so large
+files never pass through the hub; T5DE's proxy then downloads the files from the catalogs the hub lists. Anyone
+can run a catalog and ask the hub to approve it.
+
+The contract (tokens, this API, the hub's API for catalogs) is t5de-hub's `docs/products.md`; the design and the
+reasons are in its `docs/design.md` ("Products and catalogs", "Catalogs (`t5de-catalog`)").
+
+Unofficial: not made by or affiliated with IMVU.
+
+## How it works
+
+- **Uploads** (`POST /api/v1/uploads`): the creator's browser sends the product as a zip (Create Mode's `.chkn`,
+  files at the root) with an upload token from the hub. The catalog checks the token (signed by the hub, for this
+  catalog), checks the files by t5de-api's pack rules (names, limits counted while reading, `index.xml` with
+  `ProductRating` GA or AP), stores each file under its sha256, and registers the revision with the hub
+  (`POST /api/v1/catalog/revisions`, `Authorization: Catalog <id>:<key>`). Only once the hub accepts it is the
+  revision recorded as hosted; files of a refused upload stay on disk but aren't served.
+- **Files** (`GET /api/v1/files/<sha256>`): immutable, so cached for a year (`ETag` is the hash; ranges work). A
+  file used only by adult revisions needs an access token from the hub with `adult: true`, and isn't cached by
+  shared caches.
+- **Tokens** are EdDSA (Ed25519) JWTs; the hub's keys come from `HUB_URL/.well-known/t5de-keys.json`, cached and
+  fetched again when a token names an unknown key (at most every 30 s).
+- **Storage**: `CATALOG_DATA/catalog.db` (SQLite: the revisions hosted and which files belong to them) and
+  `CATALOG_DATA/files/<first 2 hex>/<sha256>`, written whole (temp file, then rename). `CATALOG_DATA/tmp` holds
+  uploads being checked and is emptied at start.
+
+## API
+
+| | |
+| --- | --- |
+| `GET /health` | `{ok: true}` |
+| `POST /api/v1/uploads` | `Authorization: Bearer <t5de-upload token>`; multipart: `file` (the zip), `notes` (required for a new revision of an existing product). 201 `{product, revision}`. Errors are `{error, message}`: 400 bad files or form (`bad_files`, `notes_required`, `bad_request`), 401 bad token (`bad_token`), 413 over 300 MiB (`too_large`), 502 the hub refused (its error code, e.g. `unchanged`) or couldn't be reached (`hub_unavailable`). CORS for `WEB_ORIGIN`. |
+| `GET /api/v1/files/<sha256>` | the file; 401/403 for adult-only files without an adult access token; 404 if not hosted |
+| `GET /api/v1/revisions/<product>/<revision>` | `{product, revision, rating, files}` for revisions hosted here |
+
+## Running
+
+Register the catalog on the hub's website (`POST /api/v1/catalogs`) to get its ID and key, and have a hub
+moderator approve it. Then, for a hub running locally:
+
+    HUB_URL=http://localhost:4101 \
+    CATALOG_ID=<id> CATALOG_KEY=<key> \
+    CATALOG_PUBLIC_URL=http://localhost:4102 \
+    WEB_ORIGIN=http://localhost:4100 \
+    CATALOG_DATA=./data \
+    go run ./cmd/t5de-catalog
+
+| Variable | |
+| --- | --- |
+| `CATALOG_ADDR` | listen address (default `127.0.0.1:4102`) |
+| `CATALOG_DATA` | data folder (default `./data`) |
+| `CATALOG_PUBLIC_URL` | the catalog's public URL, as registered on the hub (logged) |
+| `HUB_URL` | the hub's API; also the tokens' issuer (required) |
+| `CATALOG_ID`, `CATALOG_KEY` | from registering on the hub (required) |
+| `WEB_ORIGIN` | the hub website's origin, allowed to upload from browsers |
+
+It logs to standard error and stops cleanly on Ctrl+C or SIGTERM, finishing requests in progress. Put it behind
+a TLS-terminating reverse proxy (or a CDN for `/api/v1/files/`) in production.
+
+With Docker:
+
+    docker build -t t5de-catalog .
+    docker run -p 4102:4102 -v catalog-data:/data -e HUB_URL=... -e CATALOG_ID=... -e CATALOG_KEY=... \
+      -e CATALOG_PUBLIC_URL=... -e WEB_ORIGIN=... t5de-catalog
+
+## Building
+
+    go test ./...
+    CGO_ENABLED=0 go build -trimpath -o dist/t5de-catalog ./cmd/t5de-catalog
+
+The SQLite driver is pure Go (modernc.org/sqlite), so builds are static. t5de-api comes from git.t5de.dev
+directly: set `GOPRIVATE=git.t5de.dev`.