resolved/readme.md

resolved

Renders what a spacecraft camera sees: Earth, Moon, Sun and a target spacecraft loaded from a mesh. The inputs are an epoch, the observer and target states and attitudes, and a camera model. Images are radiometric: the renderer works in radiance and converts to electrons and then DN through the camera's optics and detector, so an image should match what a real camera with the same parameters records.

It is built for closed-loop simulation, where each step renders one frame and latency is the number that matters. The goal is high-fidelity 1–2 MP RGB in about 10 ms end to end on a mid-range GPU (the reference is an RTX 3060); 100 ms is the ceiling, not the aim. Lighting fidelity and camera fidelity are selected separately, and each tier is benchmarked, so a run can trade fidelity for speed knowing what each step costs.

Rendering goes through wgpu. On a machine without a GPU, the same shaders run on the CPU through Mesa's lavapipe (vulkan-swrast on Arch, mesa-vulkan-drivers on Debian and Ubuntu).

Performance and fidelity

Lighting and camera fidelity are chosen separately: lighting tiers L0 to L3, camera tiers C0 to C3, set with RendererOptions::fidelity in the library and --lighting and --camera on the command line. A tier is a fixed bundle of features, and a tier that needs data (L1 needs an image of Earth's surface) refuses to start without it. A row is added here as each tier is complete, with its measured cost and a sample image from the same scene, so the cost of each step up is visible at a glance.

Tier	Lighting	Camera	RTX 3060	lavapipe (CPU)	Sample
L0 / C0	Lambertian and glossy surfaces (glossy ones reflect only the Sun), Earth's and the Moon's limbs exact, eclipse by Earth and the Moon, no shadows on the target, no light from Earth	Pinhole, absolute radiometry to 12-bit DN, no noise or blur	0.9–1.3 ms	5–68 ms
L1 / C0	Adds Earth's surface from Blue Marble and the Moon's from the LRO Camera's map, the Moon's measured phase curve, shadows on the target, Earthshine and Moonshine, eclipses seen on Earth and the Moon, the Sun's limb darkening	Pinhole, absolute radiometry to 12-bit DN, no noise or blur	1.3–2.3 ms	12–114 ms
L1 / C1	As L1 above, with stars from Hipparcos, Tycho-2 and Gaia	Adds pixels that collect light over their area (the scene rendered 2 x 2 finer, then averaged), lens distortion, a Gaussian lens blur, the Sun, Moon and Earth at their apparent positions, shot, dark and read noise, PRNU, DSNU and hot pixels, blooming	4.1–5.2 ms	83–684 ms
L1 / C2	As L1 above, with stars	Adds diffraction (the Airy pattern over each channel's band) or a measured PSF, applied with its wings by FFT	35–37 ms	0.61–1.22 s

Times are one 1920 x 1080 RGB frame end to end (submit to wait_view, with the DN image ready to read), median of criterion runs, over models from 12 to 752,000 triangles: the box-wing, Landsat 8 and ICESat-2. The GPU's own work is 0.7 to 0.9 ms of that at L0 and 1.1 to 1.5 ms at L1, of which 0.48 ms is the sensor pass writing the image into memory the CPU reads. wait, which copies the 12 MB image into a new Vec, takes 1.8 to 2.0 ms on the RTX 3060 at L0 and 2.3 to 3.1 ms at L1, and on lavapipe 7 to 77 ms and 14 to 144 ms. L1's setup also loads the Earth and Moon maps, about 1 s, once. In the sample scene the GPU's work is 3.8 ms at C1, of which 1.3 ms is drawing Earth and the Moon at 2 x 2 and 1.0 ms the blur, and 33 ms at C2, of which 30 ms is the FFT convolution on a 4096 x 4096 grid. The C2 sample uses the example camera's Gaussian PSF, and looks like C1's: at f/4 the Airy core is smaller than a pixel, and its wings show around the Sun (examples/scenes/sun-glare.toml). lavapipe ran on the Intel Core Ultra 9 285K's 24 cores. Run to run, the RTX 3060's times vary by about 5 %, and lavapipe's by up to 15 % between sessions. The samples are the Landsat 8 example scene (examples/scenes/landsat8.toml), the preview (not --pretty) at half size, with the noise of seed 0. cargo bench -p resolved reproduces the numbers (L1 when the maps are in data/, and stars from C1 when the catalog is), and resolved render --timings N shows the per-pass split.

Planned: L2 adds measured materials and the atmosphere; L3 is path traced.

Usage

use resolved::nalgebra::{UnitQuaternion, Vector3};
use resolved::radiometry::AU;
use resolved::{CameraConfig, Mesh, Pose, Renderer, Scene, Target};

# fn main() -> Result<(), resolved::Error> {
// Setup, once: device, shaders, spectra folded into the camera's channels.
let mut renderer = Renderer::new(CameraConfig::example(), Default::default())?;

// A model, uploaded once. Usually `Mesh::load(path, &options)` on an STL or glTF.
let stl = "solid panel
facet normal 0 0 0
outer loop
vertex 0 -1 -1
vertex 0 1 -1
vertex 0 0 1
endloop
endfacet
endsolid panel";
let panel = renderer.load_model(&Mesh::from_stl(&mut std::io::Cursor::new(stl), "panel", &Default::default())?)?;

// Each step: inertial positions in meters, Earth at the origin.
let target = Vector3::new(7.0e6, 0.0, 0.0);
let moon = Vector3::new(0.0, 3.844e8, 0.0);
let scene = Scene {
    earth_attitude: UnitQuaternion::identity(),
    sun: Vector3::new(AU, 0.0, 0.0),
    moon,
    // Which way the Moon's surface map faces; an ephemeris gives it exactly.
    moon_attitude: Scene::moon_facing_earth(moon),
    targets: vec![Target { model: panel, pose: Pose::new(target, UnitQuaternion::identity()) }],
    // From camera C1, what makes positions apparent; an ephemeris gives Earth's and
    // the Moon's, and the observer's is the simulation's.
    velocities: Default::default(),
};
let observer = Pose::looking_at(target + Vector3::new(20.0, 0.0, 0.0), target, Vector3::z()).unwrap();

let pending = renderer.submit(&scene, &observer)?;
// ... the simulation's own work for this step overlaps the GPU's here ...
let frame = pending.wait()?;
// (`pending.wait_view()` lends the image where the GPU left it instead of copying it,
// saving 0.6 ms or more at 1920 x 1080; the next `submit` waits until it is dropped.)

// 12-bit DN, channels r, g, b interleaved. The panel faces the Sun and the camera.
assert!(frame.dn_at(960, 540, 1) > 1000);
# Ok(())
# }

Command line

resolved render renders a scene described in TOML: camera overrides, the observer, Sun and Moon positions (or an epoch, with the astrokit feature) and targets with their models. examples/scenes/ has commented examples: boxwing.toml uses a small generated model that is in the repository, lunar-eclipse.toml and solar-eclipse.toml show two real eclipses at lighting L1, full-moon.toml the Moon's map, crescent-moon.toml its night side lit by Earth, earthrise.toml and moonrise.toml put Earth, the Moon and a spacecraft in one view (from lunar orbit and from geostationary height), sun-glare.toml puts the Sun in the frame of a camera whose CCD blooms (camera C1), sunset.toml lights a spacecraft with the Sun setting behind Earth's atmosphere (lighting L1), and the rest use NASA spacecraft models (Landsat 8, ICESat-2, OSTM/Jason-2, an SSL-1300 and ACE) that scripts/fetch-models.sh downloads into examples/models/nasa/. That script needs Node (for npx), because most NASA models are Draco-compressed and are decoded on the way in. At lighting L1, Earth's surface comes from NASA's Blue Marble, which scripts/fetch-earth.sh downloads and resizes (with ImageMagick) into data/earth/, one image per month. The month comes from the scene's epoch, or from [appearance] earth_month in a scene without one. scripts/fetch-earth.sh --sharp 07 also makes July's sharp map (460 m per texel instead of 2.45 km; 330 MB to download and about 3 minutes to make 4.9 GB), which the CLI and the movie use when it is there. Finer still, a scene can name boxes of ground in [appearance] earth_regions to draw from ESA WorldCover's 10 m Sentinel-2 composite; scripts/fetch-earth-regions.py --scene FILE downloads the 1-degree cells that cover them (about 650 MB each) and makes each into data/earth/regions/ (194 MB of GPU memory a cell). Only the cells named are fetched, and only ahead of time: the CLI draws cells that are not there from the sharp map and names them. suez.toml looks straight down from 500 km where four such cells meet. Clouds over Earth come from the Blue Marble's cloud map (Terra MODIS, 2001 July 29), which scripts/fetch-clouds.sh downloads (420 MB) and resizes into data/clouds/blue-marble.png; [appearance] clouds = false leaves them out. Lighting L2 draws the clouds of the scene's day instead (its epoch's, or [appearance] cloud_date), which scripts/fetch-dated-clouds.py FIRST [LAST] makes from MODIS for each day in a range (about two minutes and 0.3 GB of downloads a day) into data/clouds/YYYY-MM-DD.png; the example scenes use 2024-07-15 and their epochs' days. The Moon's surface comes from the LRO Camera's normalized-reflectance mosaic, which scripts/fetch-moon.py downloads (1.2 GB) and assembles (with numpy and Pillow) into data/moon/wac-hapke-643.png. The Moon is turned by the scene's epoch, or by moon_attitude in a scene without one (by default facing Earth, which can be off by up to about 10 degrees). Also at lighting L1, and shown by camera C1, stars come from the Hipparcos, Tycho-2 and Gaia DR3 catalogs, which scripts/fetch-stars.py downloads (210 MB of files and 2.3 GB of Gaia queries, about an hour) and merges into data/stars/stars.bin (590 MB, 37 million stars to G 15); their proper motion is taken to the scene's epoch, or J2000 in a scene without one. The CLI loads only the stars bright enough to show at the camera's exposure. stars-pleiades.toml is a star field, and geo-telescope-*.toml a telescope on one geostationary satellite imaging another against the stars.

cargo run --release -p resolved-cli -- render \
    --scene examples/scenes/boxwing.toml \
    --out frame.png --preview preview.png --timings 50

scripts/make-gallery.sh renders every example scene at L0 and L1, with camera C1, into gallery/ with --pretty, skipping any whose model or data is missing, and scripts/serve-gallery.py serves the result as a web page on port 8000 of every interface, each scene captioned from its file.

scripts/gallery-delta.py shows what a change did to the images: it renders the example scenes at two revisions, noiseless, and the gallery's Delta tab shows each changed image before, after, and with its changed pixels colored (orange brighter, blue darker), with the change in DN. With no arguments it compares HEAD with uncommitted changes, or HEAD~1 with HEAD when there are none; --base and --head pick others. Each commit is built in a git worktree under target/delta/ and its renders are kept, so comparing the next commit reuses them. About 40 s on the RTX 3060 once built; it needs numpy, Pillow and ImageMagick.

crates/resolved-showcase makes a 30-second movie at lighting L1 and camera C1: an inspector holding station behind a Landsat 8 model 705 km up, which astrokit propagates with J2. Time runs fast over the daylit Earth, slows as the Moon rises over the limb ahead, and slows almost to real time as the spacecraft passes into Earth's shadow. The camera drifts on a small natural relative orbit around its station, and adjusts its exposure frame by frame, up to the flight time between frames, by at most 6 times per second of flight. It meters the first frame before taking it. In Earth's shadow at the end, exposures reach about 2 s, and show the target lit by the Moon and the stars above the limb. --out movie.mp4 encodes it with ffmpeg, rendering and tone mapping each 1920 x 1080 frame in about 9 ms on the RTX 3060. --serve 0.0.0.0:8010 renders it live, in step with the clock, and streams it as MJPEG at 60 frames per second (--fps; at 120 it skips up to 16 frames in 5 s), skipping about 9 more where the flight starts over, while it meters, to the gallery page's Movie tab, which also plays and offers the recorded file (make-gallery.sh writes gallery/movie.mp4):

cargo run --release -p resolved-showcase -- --serve 0.0.0.0:8010

• --out writes raw DN as a 16-bit PNG, shifted up to fill 16 bits as cameras usually store them: a 12-bit DN is the stored value divided by 16. It is linear, so it looks dark next to the preview.
• --preview writes an 8-bit sRGB image for looking at, with the bias as black and saturation as white. It is the signal as recorded, so a scene exposed to keep its brightest parts unsaturated looks dark.
• --pretty makes the preview look like a photograph instead: white balanced to sunlight, with a tone curve that brightens shadows and midtones by up to 3 times and rolls highlights off to white at saturation. --out is the same either way.
• --timings N renders N frames and reports setup and per-frame times.
• --lighting L0|L1 and --camera C0|C1|C2 pick the fidelity tiers (default L0 and C0). --supersampling N renders each pixel as N x N finer ones from C1 (default 2). --psf-reach N sets how far in pixels camera C2's PSF must reach (default 16; it reaches as far as its FFT grid allows, 64 px at 1920 x 1080 and 2 x 2). A tier the scene or the data cannot support stops with an error naming what is missing.
• --seed N and --frame N pick the detector noise at camera C1 (default 0 and 0): the same seed and frame give the same image. --noiseless leaves out the noise, dark current and fixed pattern, so two renders differ only where their light does.
• --data DIR says where the tiers' data is (earth/bmng-2004-MM.jpg, moon/wac-hapke-643.png and stars/stars.bin). Without it the CLI tries RESOLVED_DATA, then ./data, then the repository's data/.
• --adapter takes auto, gpu, cpu or part of an adapter's name; resolved adapters lists them.
• --kernels DIR says where astrokit's kernels are, for scenes given by epoch. Without it the CLI tries ASTROKIT_KERNELS, then data/kernels, then the sibling ../astrokit checkout it was built against.

Features

• astrokit: build a Scene from an epoch with astrokit's ephemerides (JPL DE440s for the Sun and Moon, ITRF93 for Earth's orientation), through ephemeris::Ephemeris. astrokit is not on crates.io, so this expects a checkout at ../astrokit next to this repository, and its kernels from its scripts/fetch-kernels.sh (or ASTROKIT_KERNELS pointing at them).

Status

Lighting L0 and L1 and cameras C0, C1 and C2 are in: the library renders Earth, Moon, Sun and targets to DN through Renderer, and the CLI renders scene files. Lighting L1 draws Earth's surface from Blue Marble and the Moon's from the LRO Camera's map, shadows on the target, light from Earth and the Moon on it, eclipses on Earth and the Moon, the Sun's limb darkening, and stars from Hipparcos, Tycho-2 and Gaia, which camera C1 shows. Camera C1 adds pixels that collect light over their area, lens distortion (Brown-Conrady), lens blur (a Gaussian PSF) applied to the scene rendered 2 x 2 finer than the pixels, the Sun, Moon and Earth at their apparent positions (light time and aberration), detector noise (shot, dark current and read noise, seeded and reproducible), the detector's fixed pattern (PRNU, DSNU and hot pixels), and blooming along columns or into neighbors. Camera C2 adds diffraction (the Airy pattern, averaged over each channel's band) and measured PSFs, applied with their wings by FFT. Planned, in order:

• Lighting: BRDF and spectral materials, atmospheric scattering, clouds, lunar terrain.
• Portable ray tracing, path tracing, and label outputs for ML.
• Validation against lunar, Earth and Mitsuba references; motion blur, rolling shutter, stray light; multi-band and 4k.

Conventions

• SI units throughout: meters, seconds, radians, watts.
• Scene state is f64 in an inertial frame (GCRF). The GPU only sees f32 coordinates relative to the camera.

Layout

• crates/resolved: the library.
• crates/resolved-cli: the resolved command, for rendering without writing a host program.
• examples/: scene files and the box-wing model (scripts/make-example-model.py regenerates it). The NASA scenes are generated by scripts/make-example-scenes.py.
• scripts/: generators for compiled-in data and examples.
• docs/optics.md: the optics and radiometry the renderer models, explained for readers without an optics background.

Development

Point git at the committed hooks once per clone:

git config core.hooksPath hooks

pre-commit runs formatting, clippy and the library unit tests; pre-push runs the full test suite, including doctests, and the documentation build.

The integration tests render on both a hardware GPU and lavapipe, so running them needs a Vulkan driver for the GPU and lavapipe installed. Set WGPU_BACKEND, WGPU_ADAPTER_NAME and the other standard wgpu environment variables to steer adapter discovery.

Benchmarks are end-to-end frame times (submit then wait, readback included) on the GPU and on lavapipe, at two resolutions, for the box-wing and, once fetched, Landsat 8 and ICESat-2:

cargo bench -p resolved

For where a frame's time goes, resolved render --timings N adds GPU timestamps around each pass (RendererOptions::profile in the library).

To see what the GPU tests render, set RESOLVED_TEST_IMAGES=1. Each test then writes a PNG per device to target/tmp/test-images/, scaled so a white surface facing the Sun is full white:

RESOLVED_TEST_IMAGES=1 cargo test --test gpu -- bodies:: mesh::

License

MIT