dhkatzandClaude Opus 5.5
db344b7b95
CI / build (push) Successful in 1m30s
ModelInstance::step ramps the weights of animations fading in and out in tenths over 10 ticks; between ticks, the posing hook sets them partway to the next tick's weight, and puts them back with the times. Only if ModelInstance::step counts the fades where expected (checked when installing); otherwise crossfades stay at the client's rate. Co-Authored-By: Claude Opus 5.5 <[email protected]>
112 lines
8.6 KiB
Markdown
112 lines
8.6 KiB
Markdown
# Architecture
|
|
|
|
The renderer is two programs:
|
|
|
|
- **The adapter**, `t5deRenderEngine.dll` (32-bit, `adapter/`): a renderer plugin IMVU Classic loads in place of its
|
|
OpenGL renderer (selected by the `t5de-renderer` patch in the `t5de` repository). It implements IMVU's
|
|
`PiRenderEngine` interface (`include/t5de/imvu/`, `docs/interface.md`) and passes what the client draws to the
|
|
renderer process. It stays thin: IMVU's process is 32-bit (4 GB of address space shared with Python, Gecko and
|
|
IMVU's own code) and tied to Visual C++ 2010's ABI.
|
|
- **The renderer process**, `t5de-renderer.exe` (64-bit, `renderer/`): draws, with
|
|
[Diligent Engine](https://github.com/DiligentGraphics/DiligentCore) (Apache-2.0). Being 64-bit and separate, it can
|
|
use any modern API and library, isn't limited by the client's address space, and can crash without taking the
|
|
client down.
|
|
|
|
Why Diligent: one API over Direct3D 11 (every Windows PC, and software rendering on machines without a graphics
|
|
driver), Direct3D 12 and Vulkan (ray tracing, mesh shaders), and WebGPU (browsers, for a possible future client).
|
|
It's built with CMake and fetched by `FetchContent`, pinned to a commit (`renderer/CMakeLists.txt`); only the
|
|
Direct3D 11 backend is built for now.
|
|
|
|
## How they talk
|
|
|
|
`include/t5de/protocol/` is shared by both builds, so it uses fixed-size fields only. One renderer process draws all
|
|
of a client's 3D views, with one Direct3D device. The adapter starts `t5de-renderer.exe --control <name> --parent
|
|
<pid>` from its own folder with the first engine (again with the next one if it has exited), and creates a control
|
|
section (`Local\t5de-renderer-<pid>`, `protocol::Control`). For each engine the client creates (each 3D view: Home's
|
|
avatar, the room, ...), the adapter:
|
|
|
|
1. creates a shared memory channel (`<control name>-<n>`) holding IMVU's window handle, and its events;
|
|
2. counts it in the control section and sets `<control name>-attach`, and the renderer process opens it, with a
|
|
child window, swap chain and scene of its own;
|
|
3. sends each frame's commands through it, and sets `closing` when the engine is destroyed (the renderer process
|
|
then closes the view).
|
|
|
|
The renderer process draws into a child window it creates inside the client's window: window handles are valid in
|
|
every process, and Windows allows child windows across processes (Chrome's GPU process works this way). The child
|
|
is disabled so mouse input goes to the client's window under it. Its swap chain is a flip model one, which the
|
|
compositor shows above the client's own drawing. The client's window must not clip its children (`WS_CLIPCHILDREN`,
|
|
which the client creates it with, so the adapter clears it):
|
|
the client draws a frame when its window gets `WM_PAINT` (`SceneWindow.dll` invalidates it, and draws on the paint),
|
|
and a window entirely covered by clipped children never gets one, so the client fell to about one frame a second.
|
|
The process exits when the client exits.
|
|
|
|
## Status
|
|
|
|
The renderer process draws the whole 3D view in place of the client's renderer (the full view, the default). The
|
|
adapter still forwards state and resources to the client's Direct3D renderer (as a fallback, and for reading textures
|
|
back, which the renderer process doesn't do), but while the renderer process covers the whole view
|
|
(`Channel::covering`, set after each frame it presents) the client's renderer skips its clears and draws, decided per
|
|
frame; it draws again as soon as the renderer process stops covering the view (the view changes, or the process
|
|
fails). Checked in rooms with screenshots of both renderers side by side (the compare view): they match, particles
|
|
and the 2D overlay (name tags) included.
|
|
|
|
For comparing while the client runs (development):
|
|
|
|
- Views: `T5DE_RENDERER_VIEW` (off, split, full (the default) or compare) when the client starts, or
|
|
`%TEMP%\t5de-renderer-view.txt` holding one of those at any time. Compare draws the left half's scene over the
|
|
right half, so both renderers show the same thing side by side.
|
|
- `%TEMP%\t5de-renderer-shaders\fixed_function.hlsl`, if present, replaces the built-in shaders and is reloaded
|
|
when it changes.
|
|
- The renderer process runs from a copy (`%TEMP%\t5de-renderer`), so `t5de-renderer.exe` can be replaced while the
|
|
client runs; engines started afterwards run the new one (the room's view keeps its engine until the client
|
|
restarts).
|
|
- The adapter's log (`%TEMP%\t5de-renderer.log`) sums up what the client draws every 300 frames (the first few
|
|
times), and logs textures and render targets the client renderer made itself.
|
|
|
|
- The command stream (`include/t5de/protocol/commands.hpp`) carries state, transforms, lights, fog, textures,
|
|
buffers (copied when created or updated) and draws, in chunks through the channel; the adapter waits for each to
|
|
be taken, and gives up on a renderer process that stops taking them (the client's renderer carries on).
|
|
- The renderer process reproduces IMVU's Direct3D 9 setup (`renderer/shaders/fixed_function.hlsl`), particles
|
|
included (point sprites, drawn as instanced squares), with a pipeline per combination of states.
|
|
- Speed: on a PC with a graphics card, drawing the whole view (3440x1059, a room of some 220 draws) takes 1.3 to
|
|
3.8 ms a frame, and the time from the client finishing a frame to ours being presented is the same (frames don't
|
|
queue, none skipped). Earlier, in the split view, the halves couldn't be told apart, even while the camera moved. On the development VM (software rendering, over Remote Desktop) drawing takes 8 to 60 ms, and
|
|
the right half visibly trails, mostly from compositing another process's window over Remote Desktop. The
|
|
renderer process draws only the newest frame when it falls behind, and keeps at most one frame queued.
|
|
|
|
Render targets are the renderer process's too, while it runs: the adapter makes its own (`PixmapTarget`,
|
|
`TextureTarget`), the renderer process draws frames into them (textures in the window's formats, so the same
|
|
pipelines draw both), and a pixmap target's frame is read back through the channel (`Channel::readSize`, the
|
|
`-read` event) into its Pixmap, as the client's renderer does, synchronously. A read back must fit in the channel's
|
|
data area (16 MB: the client's 2048x2048 snapshots just do). Checked with outfit pictures and photos.
|
|
|
|
Higher frame rates (in progress): the client steps its scene 30 times a second (SceneWindow.dll's `runApp`), and
|
|
draws a window's frame on WM_PAINT. The adapter's frame pacer (`adapter/frame_pacer.cpp`) asks the 3D windows to
|
|
repaint between steps at the user's rate (T5DE's settings, Graphics: `HKCU\Software\T5DE\Graphics\FrameRate`), only
|
|
leaving the client idle time, and never holding up the next step: the adapter records the client's steps by replacing
|
|
`SceneViewer::step`'s entry in its exported vtable (`adapter/scene_hooks.cpp`, after checking it), and a frame that
|
|
wouldn't end before the step is due isn't asked for (a late step makes the client catch up two at once: hitches).
|
|
While the renderer process covers the view, the client's renderer doesn't begin or present frames (its present can
|
|
wait for the display). The renderer process takes each view's chunks on a thread of its own as they come, so the
|
|
client never waits for drawing; it draws the newest frame. On a PC with a graphics card a room frame takes the client
|
|
about 2.5 ms (the client's own work: posing avatars, which it does every frame, and walking the scene), and the
|
|
renderer process about 0.5 ms.
|
|
|
|
Between steps, the frames show the scene at the time between them (`adapter/scene_hooks.cpp`): avatars posed with
|
|
their animations a fraction of a frame on (ModelInstance::render's call to updateCalSkeleton, redirected; Cal3D poses
|
|
between keyframes; animations held, looping or starting are posed as they are), and the camera where its easing (a
|
|
share 0.4 of the way to its target per step) has brought it, `1 - 0.6^fraction` of the way (CameraController's
|
|
getCameraTransform, in its exported vtable). Nothing is predicted: both are what the next step would give, at the
|
|
fraction; and particles moved on by their velocity times the time since the step (ModelInstance::render's call to
|
|
ParticleSystem::render, redirected), then back by the same (the client sorts them by distance as it draws them, which
|
|
reorders them); and crossfades between animations (weights ramped in tenths over 10 steps by ModelInstance::step),
|
|
partway to the next step's weights. Still at 30 per second: gaze (heads turning to whoever's talking).
|
|
|
|
Next: sharing pipelines between views (each view's scene compiles its own).
|
|
|
|
## Building
|
|
|
|
Two presets, one per architecture: `cmake --workflow --preset adapter` (32-bit, with the tests) and
|
|
`cmake --workflow --preset renderer` (64-bit; the first configure fetches and builds Diligent). Both programs go in
|
|
the client's folder.
|