dhkatzandClaude Opus 5.5
8742829e77
CI / build (push) Successful in 54s
A Go server that takes product uploads with hub upload tokens, checks them by t5de-api's pack rules, stores files by sha256, registers revisions with the hub (POST /api/v1/catalog/revisions), and serves files with immutable caching; files used only by adult revisions need an adult access token. Contract: t5de-hub's docs/products.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
76 lines
4.2 KiB
Markdown
76 lines
4.2 KiB
Markdown
# t5de-catalog
|
|
|
|
T5DE's catalog server: it stores community products' files by sha256, serves them, and registers uploads with
|
|
T5DE's hub (git.t5de.dev/t5de/t5de-hub). Creators upload from the hub's website straight to a catalog, so large
|
|
files never pass through the hub; T5DE's proxy then downloads the files from the catalogs the hub lists. Anyone
|
|
can run a catalog and ask the hub to approve it.
|
|
|
|
The contract (tokens, this API, the hub's API for catalogs) is t5de-hub's `docs/products.md`; the design and the
|
|
reasons are in its `docs/design.md` ("Products and catalogs", "Catalogs (`t5de-catalog`)").
|
|
|
|
Unofficial: not made by or affiliated with IMVU.
|
|
|
|
## How it works
|
|
|
|
- **Uploads** (`POST /api/v1/uploads`): the creator's browser sends the product as a zip (Create Mode's `.chkn`,
|
|
files at the root) with an upload token from the hub. The catalog checks the token (signed by the hub, for this
|
|
catalog), checks the files by t5de-api's pack rules (names, limits counted while reading, `index.xml` with
|
|
`ProductRating` GA or AP), stores each file under its sha256, and registers the revision with the hub
|
|
(`POST /api/v1/catalog/revisions`, `Authorization: Catalog <id>:<key>`). Only once the hub accepts it is the
|
|
revision recorded as hosted; files of a refused upload stay on disk but aren't served.
|
|
- **Files** (`GET /api/v1/files/<sha256>`): immutable, so cached for a year (`ETag` is the hash; ranges work). A
|
|
file used only by adult revisions needs an access token from the hub with `adult: true`, and isn't cached by
|
|
shared caches.
|
|
- **Tokens** are EdDSA (Ed25519) JWTs; the hub's keys come from `HUB_URL/.well-known/t5de-keys.json`, cached and
|
|
fetched again when a token names an unknown key (at most every 30 s).
|
|
- **Storage**: `CATALOG_DATA/catalog.db` (SQLite: the revisions hosted and which files belong to them) and
|
|
`CATALOG_DATA/files/<first 2 hex>/<sha256>`, written whole (temp file, then rename). `CATALOG_DATA/tmp` holds
|
|
uploads being checked and is emptied at start.
|
|
|
|
## API
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| `GET /health` | `{ok: true}` |
|
|
| `POST /api/v1/uploads` | `Authorization: Bearer <t5de-upload token>`; multipart: `file` (the zip), `notes` (required for a new revision of an existing product). 201 `{product, revision}`. Errors are `{error, message}`: 400 bad files or form (`bad_files`, `notes_required`, `bad_request`), 401 bad token (`bad_token`), 413 over 300 MiB (`too_large`), 502 the hub refused (its error code, e.g. `unchanged`) or couldn't be reached (`hub_unavailable`). CORS for `WEB_ORIGIN`. |
|
|
| `GET /api/v1/files/<sha256>` | the file; 401/403 for adult-only files without an adult access token; 404 if not hosted |
|
|
| `GET /api/v1/revisions/<product>/<revision>` | `{product, revision, rating, files}` for revisions hosted here |
|
|
|
|
## Running
|
|
|
|
Register the catalog on the hub's website (`POST /api/v1/catalogs`) to get its ID and key, and have a hub
|
|
moderator approve it. Then, for a hub running locally:
|
|
|
|
HUB_URL=http://localhost:4101 \
|
|
CATALOG_ID=<id> CATALOG_KEY=<key> \
|
|
CATALOG_PUBLIC_URL=http://localhost:4102 \
|
|
WEB_ORIGIN=http://localhost:4100 \
|
|
CATALOG_DATA=./data \
|
|
go run ./cmd/t5de-catalog
|
|
|
|
| Variable | |
|
|
| --- | --- |
|
|
| `CATALOG_ADDR` | listen address (default `127.0.0.1:4102`) |
|
|
| `CATALOG_DATA` | data folder (default `./data`) |
|
|
| `CATALOG_PUBLIC_URL` | the catalog's public URL, as registered on the hub (logged) |
|
|
| `HUB_URL` | the hub's API; also the tokens' issuer (required) |
|
|
| `CATALOG_ID`, `CATALOG_KEY` | from registering on the hub (required) |
|
|
| `WEB_ORIGIN` | the hub website's origin, allowed to upload from browsers |
|
|
|
|
It logs to standard error and stops cleanly on Ctrl+C or SIGTERM, finishing requests in progress. Put it behind
|
|
a TLS-terminating reverse proxy (or a CDN for `/api/v1/files/`) in production.
|
|
|
|
With Docker:
|
|
|
|
docker build -t t5de-catalog .
|
|
docker run -p 4102:4102 -v catalog-data:/data -e HUB_URL=... -e CATALOG_ID=... -e CATALOG_KEY=... \
|
|
-e CATALOG_PUBLIC_URL=... -e WEB_ORIGIN=... t5de-catalog
|
|
|
|
## Building
|
|
|
|
go test ./...
|
|
CGO_ENABLED=0 go build -trimpath -o dist/t5de-catalog ./cmd/t5de-catalog
|
|
|
|
The SQLite driver is pure Go (modernc.org/sqlite), so builds are static. t5de-api comes from git.t5de.dev
|
|
directly: set `GOPRIVATE=git.t5de.dev`.
|