t5de-catalog

T5DE's catalog server: it stores community products' files by sha256, serves them, and registers uploads with T5DE's hub (git.t5de.dev/t5de/t5de-hub). Creators upload from the hub's website straight to a catalog, so large files never pass through the hub; T5DE's proxy then downloads the files from the catalogs the hub lists. Anyone can run a catalog and ask the hub to approve it.

The contract (tokens, this API, the hub's API for catalogs) is t5de-hub's docs/products.md; the design and the reasons are in its docs/design.md ("Products and catalogs", "Catalogs (t5de-catalog)").

Unofficial: not made by or affiliated with IMVU.

How it works

• Uploads (POST /api/v1/uploads): the creator's browser sends the product as a zip (Create Mode's .chkn, files at the root) with an upload token from the hub. The catalog checks the token (signed by the hub, for this catalog), checks the files by t5de-api's pack rules (names, limits counted while reading, index.xml with ProductRating GA or AP), stores each file under its sha256, and registers the revision with the hub (POST /api/v1/catalog/revisions, Authorization: Catalog <id>:<key>). Only once the hub accepts it is the revision recorded as hosted; files of a refused upload stay on disk but aren't served.
• Files (GET /api/v1/files/<sha256>): immutable, so cached for a year (ETag is the hash; ranges work). A file used only by adult revisions needs an access token from the hub with adult: true, and isn't cached by shared caches.
• Tokens are EdDSA (Ed25519) JWTs; the hub's keys come from HUB_URL/.well-known/t5de-keys.json, cached and fetched again when a token names an unknown key (at most every 30 s).
• Storage: CATALOG_DATA/catalog.db (SQLite: the revisions hosted and which files belong to them) and CATALOG_DATA/files/<first 2 hex>/<sha256>, written whole (temp file, then rename). CATALOG_DATA/tmp holds uploads being checked and is emptied at start.

API

GET /health	{ok: true}
POST /api/v1/uploads	Authorization: Bearer <t5de-upload token>; multipart: file (the zip), notes (required for a new revision of an existing product). 201 {product, revision}. Errors are {error, message}: 400 bad files or form (bad_files, notes_required, bad_request), 401 bad token (bad_token), 413 over 300 MiB (too_large), 502 the hub refused (its error code, e.g. unchanged) or couldn't be reached (hub_unavailable). CORS for WEB_ORIGIN.
GET /api/v1/files/<sha256>	the file; 401/403 for adult-only files without an adult access token; 404 if not hosted
GET /api/v1/revisions/<product>/<revision>	{product, revision, rating, files} for revisions hosted here

Running

Register the catalog on the hub's website (POST /api/v1/catalogs) to get its ID and key, and have a hub moderator approve it. Then, for a hub running locally:

HUB_URL=http://localhost:4101 \
CATALOG_ID=<id> CATALOG_KEY=<key> \
CATALOG_PUBLIC_URL=http://localhost:4102 \
WEB_ORIGIN=http://localhost:4100 \
CATALOG_DATA=./data \
go run ./cmd/t5de-catalog

Variable
CATALOG_ADDR	listen address (default 127.0.0.1:4102)
CATALOG_DATA	data folder (default ./data)
CATALOG_PUBLIC_URL	the catalog's public URL, as registered on the hub (logged)
HUB_URL	the hub's API; also the tokens' issuer (required)
CATALOG_ID, CATALOG_KEY	from registering on the hub (required)
WEB_ORIGIN	the hub website's origin, allowed to upload from browsers

It logs to standard error and stops cleanly on Ctrl+C or SIGTERM, finishing requests in progress. Put it behind a TLS-terminating reverse proxy (or a CDN for /api/v1/files/) in production.

With Docker:

docker build -t t5de-catalog .
docker run -p 4102:4102 -v catalog-data:/data -e HUB_URL=... -e CATALOG_ID=... -e CATALOG_KEY=... \
  -e CATALOG_PUBLIC_URL=... -e WEB_ORIGIN=... t5de-catalog

Building

go test ./...
CGO_ENABLED=0 go build -trimpath -o dist/t5de-catalog ./cmd/t5de-catalog

The SQLite driver is pure Go (modernc.org/sqlite), so builds are static. t5de-api comes from git.t5de.dev directly: set GOPRIVATE=git.t5de.dev.