t5de-renderer/docs/architecture.md

# Architecture

The renderer is two programs:

- **The adapter**, `t5deRenderEngine.dll` (32-bit, `adapter/`): a renderer plugin IMVU Classic loads in place of its
  OpenGL renderer (selected by the `t5de-renderer` patch in the `t5de` repository). It implements IMVU's
  `PiRenderEngine` interface (`include/t5de/imvu/`, `docs/interface.md`) and passes what the client draws to the
  renderer process. It stays thin: IMVU's process is 32-bit (4 GB of address space shared with Python, Gecko and
  IMVU's own code) and tied to Visual C++ 2010's ABI.
- **The renderer process**, `t5de-renderer.exe` (64-bit, `renderer/`): draws, with
  [Diligent Engine](https://github.com/DiligentGraphics/DiligentCore) (Apache-2.0). Being 64-bit and separate, it can
  use any modern API and library, isn't limited by the client's address space, and can crash without taking the
  client down.

Why Diligent: one API over Direct3D 11 (every Windows PC, and software rendering on machines without a graphics
driver), Direct3D 12 and Vulkan (ray tracing, mesh shaders), and WebGPU (browsers, for a possible future client).
It's built with CMake and fetched by `FetchContent`, pinned to a commit (`renderer/CMakeLists.txt`); only the
Direct3D 11 backend is built for now.

## How they talk

`include/t5de/protocol/` is shared by both builds, so it uses fixed-size fields only. One renderer process draws all
of a client's 3D views, with one Direct3D device. The adapter starts `t5de-renderer.exe --control <name> --parent
<pid>` from its own folder with the first engine (again with the next one if it has exited), and creates a control
section (`Local\t5de-renderer-<pid>`, `protocol::Control`). For each engine the client creates (each 3D view: Home's
avatar, the room, ...), the adapter:

1. creates a shared memory channel (`<control name>-<n>`) holding IMVU's window handle, and its events;
2. counts it in the control section and sets `<control name>-attach`, and the renderer process opens it, with a
   child window, swap chain and scene of its own;
3. sends each frame's commands through it, and sets `closing` when the engine is destroyed (the renderer process
   then closes the view).

The renderer process draws into a child window it creates inside the client's window: window handles are valid in
every process, and Windows allows child windows across processes (Chrome's GPU process works this way). The child
is disabled so mouse input goes to the client's window under it. Its swap chain is a flip model one, which the
compositor shows above the client's own drawing. The client's window must not clip its children (`WS_CLIPCHILDREN`,
which the client creates it with, so the adapter clears it):
the client draws a frame when its window gets `WM_PAINT` (`SceneWindow.dll` invalidates it, and draws on the paint),
and a window entirely covered by clipped children never gets one, so the client fell to about one frame a second.
The process exits when the client exits.

## Status

The renderer process draws the whole 3D view in place of the client's renderer (the full view, the default). The
adapter still forwards state and resources to the client's Direct3D renderer (as a fallback, and for reading textures
back, which the renderer process doesn't do), but while the renderer process covers the whole view
(`Channel::covering`, set after each frame it presents) the client's renderer skips its clears and draws, decided per
frame; it draws again as soon as the renderer process stops covering the view (the view changes, or the process
fails). Checked in rooms with screenshots of both renderers side by side (the compare view): they match, particles
and the 2D overlay (name tags) included.

For comparing while the client runs (development):

- Views: `T5DE_RENDERER_VIEW` (off, split, full (the default) or compare) when the client starts, or
  `%TEMP%\t5de-renderer-view.txt` holding one of those at any time. Compare draws the left half's scene over the
  right half, so both renderers show the same thing side by side.
- `%TEMP%\t5de-renderer-shaders\fixed_function.hlsl`, if present, replaces the built-in shaders and is reloaded
  when it changes.
- The renderer process runs from a copy (`%TEMP%\t5de-renderer`), so `t5de-renderer.exe` can be replaced while the
  client runs; engines started afterwards run the new one (the room's view keeps its engine until the client
  restarts).
- The adapter's log (`%TEMP%\t5de-renderer.log`) sums up what the client draws every 300 frames (the first few
  times), and logs textures and render targets the client renderer made itself.

- The command stream (`include/t5de/protocol/commands.hpp`) carries state, transforms, lights, fog, textures,
  buffers (copied when created or updated) and draws, in chunks through the channel; the adapter waits for each to
  be taken, and gives up on a renderer process that stops taking them (the client's renderer carries on).
- The renderer process reproduces IMVU's Direct3D 9 setup (`renderer/shaders/fixed_function.hlsl`), particles
  included (point sprites, drawn as instanced squares), with a pipeline per combination of states.
- Speed: on a PC with a graphics card, drawing the whole view (3440x1059, a room of some 220 draws) takes 1.3 to
  3.8 ms a frame, and the time from the client finishing a frame to ours being presented is the same (frames don't
  queue, none skipped). Earlier, in the split view, the halves couldn't be told apart, even while the camera moved. On the development VM (software rendering, over Remote Desktop) drawing takes 8 to 60 ms, and
  the right half visibly trails, mostly from compositing another process's window over Remote Desktop. The
  renderer process draws only the newest frame when it falls behind, and keeps at most one frame queued.

Render targets are the renderer process's too, while it runs: the adapter makes its own (`PixmapTarget`,
`TextureTarget`), the renderer process draws frames into them (textures in the window's formats, so the same
pipelines draw both), and a pixmap target's frame is read back through the channel (`Channel::readSize`, the
`-read` event) into its Pixmap, as the client's renderer does, synchronously. A read back must fit in the channel's
data area (16 MB: the client's 2048x2048 snapshots just do). Checked with outfit pictures and photos.

Higher frame rates (in progress): the client steps its scene 30 times a second (SceneWindow.dll's `runApp`), and
draws a window's frame on WM_PAINT. The adapter's frame pacer (`adapter/frame_pacer.cpp`) asks the 3D windows to
repaint between steps at the user's rate (T5DE's settings, Graphics: `HKCU\Software\T5DE\Graphics\FrameRate`), only
when affordable: drawing shares the client's main thread with its input and interface. The extra frames show the
same scene until animations, the camera and particles are drawn at the time between steps.

Next: sharing pipelines between views (each view's scene compiles its own).

## Building

Two presets, one per architecture: `cmake --workflow --preset adapter` (32-bit, with the tests) and
`cmake --workflow --preset renderer` (64-bit; the first configure fetches and builds Diligent). Both programs go in
the client's folder.