307 lines
16 KiB
Markdown
307 lines
16 KiB
Markdown
# resolved
|
||
|
||
Renders what a spacecraft camera sees: Earth, Moon, Sun and a target
|
||
spacecraft loaded from a mesh. The inputs are an epoch, the observer and target
|
||
states and attitudes, and a camera model. Images are radiometric: the renderer
|
||
works in radiance and converts to electrons and then DN through the camera's
|
||
optics and detector, so an image should match what a real camera with the same
|
||
parameters records.
|
||
|
||
It is built for closed-loop simulation, where each step renders one frame and
|
||
latency is the number that matters. The goal is high-fidelity 1–2 MP RGB in
|
||
about 10 ms end to end on a mid-range GPU (the reference is an RTX 3060);
|
||
100 ms is the ceiling, not the aim. Lighting fidelity and camera fidelity are
|
||
selected separately, and each tier is benchmarked, so a run can trade
|
||
fidelity for speed knowing what each step costs.
|
||
|
||
Rendering goes through wgpu. On a machine without a GPU, the same shaders run
|
||
on the CPU through Mesa's lavapipe (`vulkan-swrast` on Arch,
|
||
`mesa-vulkan-drivers` on Debian and Ubuntu).
|
||
|
||
## Performance and fidelity
|
||
|
||
Lighting and camera fidelity are chosen separately: lighting tiers L0 to L3,
|
||
camera tiers C0 to C3, set with `RendererOptions::fidelity` in the library and
|
||
`--lighting` and `--camera` on the command line. A tier is a fixed bundle of
|
||
features, and a tier that needs data (L1 needs an image of Earth's surface)
|
||
refuses to start without it. A row is added here as each tier is complete, with
|
||
its measured cost and a sample image from the same scene, so the cost of each
|
||
step up is visible at a glance.
|
||
|
||
| Tier | Lighting | Camera | RTX 3060 | lavapipe (CPU) | Sample |
|
||
|---|---|---|---|---|---|
|
||
| L0 / C0 | Lambertian surfaces, Earth's and the Moon's limbs exact, eclipse by Earth and the Moon, no shadows on the target, no light from Earth | Pinhole, absolute radiometry to 12-bit DN, no noise or blur | 0.9–1.3 ms | 5–68 ms | [](docs/images/l0-c0.png) |
|
||
| L1 / C0 | Adds Earth's surface from Blue Marble and the Moon's from the LRO Camera's map, the Moon's measured phase curve, shadows on the target, Earthshine and Moonshine, eclipses seen on Earth and the Moon, the Sun's limb darkening | Pinhole, absolute radiometry to 12-bit DN, no noise or blur | 1.3–2.3 ms | 12–114 ms | [](docs/images/l1-c0.png) |
|
||
| L1 / C1 | As L1 above, with stars from Hipparcos, Tycho-2 and Gaia | Adds pixels that collect light over their area (the scene rendered 2 x 2 finer, then averaged), lens distortion, a Gaussian lens blur, the Sun, Moon and Earth at their apparent positions, shot, dark and read noise, PRNU, DSNU and hot pixels, blooming | 4.1–5.2 ms | 83–684 ms | [](docs/images/l1-c1.png) |
|
||
| L1 / C2 | As L1 above, with stars | Adds diffraction (the Airy pattern over each channel's band) or a measured PSF, applied with its wings by FFT | 35–37 ms | 0.61–1.22 s | [](docs/images/l1-c2.png) |
|
||
|
||
Times are one 1920 x 1080 RGB frame end to end (`submit` to `wait_view`, with
|
||
the DN image ready to read), median of criterion runs, over models from 12 to
|
||
752,000 triangles: the box-wing, Landsat 8 and ICESat-2. The GPU's own work is
|
||
0.7 to 0.9 ms of that at L0 and 1.1 to 1.5 ms at L1, of which 0.48 ms is the
|
||
sensor pass writing the image into memory the CPU reads. `wait`, which copies
|
||
the 12 MB image into a new `Vec`, takes 1.8 to 2.0 ms on the RTX 3060 at L0
|
||
and 2.3 to 3.1 ms at L1, and on lavapipe 7 to 77 ms and 14 to 144 ms. L1's
|
||
setup also loads the Earth and Moon maps, about 1 s, once. In the sample scene
|
||
the GPU's work is 3.8 ms at C1, of which 1.3 ms is drawing Earth and the Moon
|
||
at 2 x 2 and 1.0 ms the blur, and 33 ms at C2, of which 30 ms is the FFT
|
||
convolution on a 4096 x 4096 grid. The C2 sample uses the example camera's
|
||
Gaussian PSF, and looks like C1's: at f/4 the Airy core is smaller than a
|
||
pixel, and its wings show around the Sun (`examples/scenes/sun-glare.toml`).
|
||
lavapipe ran on the Intel Core Ultra 9 285K's 24 cores. Run to run, the RTX
|
||
3060's times vary by about 5 %, and lavapipe's by up to 15 % between sessions.
|
||
The samples are the Landsat 8 example scene (`examples/scenes/landsat8.toml`),
|
||
the preview (not `--pretty`) at half size, with the noise of seed 0. `cargo
|
||
bench -p resolved` reproduces the numbers (L1 when the maps are in `data/`,
|
||
and stars from C1 when the catalog is), and `resolved render --timings N`
|
||
shows the per-pass split.
|
||
|
||
Planned: L2 adds measured materials and the atmosphere; L3 is path traced.
|
||
|
||
## Usage
|
||
|
||
```rust
|
||
use resolved::nalgebra::{UnitQuaternion, Vector3};
|
||
use resolved::radiometry::AU;
|
||
use resolved::{CameraConfig, Mesh, Pose, Renderer, Scene, Target};
|
||
|
||
# fn main() -> Result<(), resolved::Error> {
|
||
// Setup, once: device, shaders, spectra folded into the camera's channels.
|
||
let mut renderer = Renderer::new(CameraConfig::example(), Default::default())?;
|
||
|
||
// A model, uploaded once. Usually `Mesh::load(path, &options)` on an STL or glTF.
|
||
let stl = "solid panel
|
||
facet normal 0 0 0
|
||
outer loop
|
||
vertex 0 -1 -1
|
||
vertex 0 1 -1
|
||
vertex 0 0 1
|
||
endloop
|
||
endfacet
|
||
endsolid panel";
|
||
let panel = renderer.load_model(&Mesh::from_stl(&mut std::io::Cursor::new(stl), "panel", &Default::default())?)?;
|
||
|
||
// Each step: inertial positions in meters, Earth at the origin.
|
||
let target = Vector3::new(7.0e6, 0.0, 0.0);
|
||
let moon = Vector3::new(0.0, 3.844e8, 0.0);
|
||
let scene = Scene {
|
||
earth_attitude: UnitQuaternion::identity(),
|
||
sun: Vector3::new(AU, 0.0, 0.0),
|
||
moon,
|
||
// Which way the Moon's surface map faces; an ephemeris gives it exactly.
|
||
moon_attitude: Scene::moon_facing_earth(moon),
|
||
targets: vec![Target { model: panel, pose: Pose::new(target, UnitQuaternion::identity()) }],
|
||
// From camera C1, what makes positions apparent; an ephemeris gives Earth's and
|
||
// the Moon's, and the observer's is the simulation's.
|
||
velocities: Default::default(),
|
||
};
|
||
let observer = Pose::looking_at(target + Vector3::new(20.0, 0.0, 0.0), target, Vector3::z()).unwrap();
|
||
|
||
let pending = renderer.submit(&scene, &observer)?;
|
||
// ... the simulation's own work for this step overlaps the GPU's here ...
|
||
let frame = pending.wait()?;
|
||
// (`pending.wait_view()` lends the image where the GPU left it instead of copying it,
|
||
// saving 0.6 ms or more at 1920 x 1080; the next `submit` waits until it is dropped.)
|
||
|
||
// 12-bit DN, channels r, g, b interleaved. The panel faces the Sun and the camera.
|
||
assert!(frame.dn_at(960, 540, 1) > 1000);
|
||
# Ok(())
|
||
# }
|
||
```
|
||
|
||
## Command line
|
||
|
||
`resolved render` renders a scene described in TOML: camera overrides, the
|
||
observer, Sun and Moon positions (or an epoch, with the `astrokit` feature) and
|
||
targets with their models. `examples/scenes/` has commented examples:
|
||
`boxwing.toml` uses a small generated model that is in the repository, `lunar-eclipse.toml` and `solar-eclipse.toml` show two
|
||
real eclipses at lighting L1, `full-moon.toml` the Moon's map, `earthrise.toml`
|
||
and `moonrise.toml` put Earth, the Moon and a spacecraft in one view (from lunar
|
||
orbit and from geostationary height), `sun-glare.toml` puts the Sun in the frame
|
||
of a camera whose CCD blooms (camera C1), `sunset.toml` lights a spacecraft with
|
||
the Sun setting behind Earth's atmosphere (lighting L1), and the rest use NASA spacecraft models
|
||
(Landsat 8, ICESat-2, OSTM/Jason-2, an SSL-1300 and ACE) that
|
||
`scripts/fetch-models.sh` downloads into `examples/models/nasa/`. That script needs Node (for `npx`),
|
||
because most NASA models are Draco-compressed and are decoded on the way in.
|
||
At lighting L1, Earth's surface comes from NASA's Blue Marble, which
|
||
`scripts/fetch-earth.sh` downloads and resizes (with ImageMagick) into
|
||
`data/earth/`, one image per month. The month comes from the scene's epoch, or
|
||
from `[appearance] earth_month` in a scene without one. `scripts/fetch-earth.sh
|
||
--sharp 07` also makes July's sharp map (460 m per texel instead of 2.45 km; 330 MB
|
||
to download and about 3 minutes to make 4.9 GB), which the CLI and the movie use
|
||
when it is there. The Moon's surface
|
||
comes from the LRO Camera's normalized-reflectance mosaic, which
|
||
`scripts/fetch-moon.py` downloads (1.2 GB) and assembles (with numpy and Pillow)
|
||
into `data/moon/wac-hapke-643.png`. The Moon is turned by the scene's epoch, or
|
||
by `moon_attitude` in a scene without one (by default facing Earth, which can
|
||
be off by up to about 10 degrees). Also at lighting L1, and shown by camera C1,
|
||
stars come from the Hipparcos, Tycho-2 and Gaia DR3 catalogs, which `scripts/fetch-stars.py` downloads (210 MB of
|
||
files and 2.3 GB of Gaia queries, about an hour) and merges into
|
||
`data/stars/stars.bin` (590 MB, 37 million stars to G 15); their proper motion
|
||
is taken to the scene's epoch, or J2000 in a scene without one. The CLI loads
|
||
only the stars bright enough to show at the camera's exposure.
|
||
`stars-pleiades.toml` is a star field, and `geo-telescope-*.toml` a telescope
|
||
on one geostationary satellite imaging another against the stars.
|
||
|
||
```sh
|
||
cargo run --release -p resolved-cli -- render \
|
||
--scene examples/scenes/boxwing.toml \
|
||
--out frame.png --preview preview.png --timings 50
|
||
```
|
||
|
||
`scripts/make-gallery.sh` renders every example scene at L0 and L1, with
|
||
camera C1, into `gallery/` with `--pretty`, skipping any whose model or data
|
||
is missing, and `scripts/serve-gallery.py` serves the result as a web page on
|
||
port 8000 of every interface, each scene captioned from its file.
|
||
|
||
`scripts/gallery-delta.py` shows what a change did to the images: it renders
|
||
the example scenes at two revisions, noiseless, and the gallery's Delta tab
|
||
shows each changed image before, after, and with its changed pixels colored
|
||
(orange brighter, blue darker), with the change in DN. With no arguments it
|
||
compares HEAD with uncommitted changes, or HEAD~1 with HEAD when there are
|
||
none; `--base` and `--head` pick others. Each commit is built in a git worktree
|
||
under `target/delta/` and its renders are kept, so comparing the next commit
|
||
reuses them. About 40 s on the RTX 3060 once built; it needs numpy, Pillow and
|
||
ImageMagick.
|
||
|
||
`crates/resolved-showcase` makes a 30-second movie at lighting L1 and camera
|
||
C1: an inspector holding station behind a Landsat 8 model 705 km up, which
|
||
astrokit propagates with J2. Time runs fast over the daylit Earth, slows as
|
||
the Moon rises over the limb ahead, and slows almost to real time as the
|
||
spacecraft passes into Earth's shadow. The camera drifts on a small natural
|
||
relative orbit around its station, and adjusts its exposure frame by frame, up
|
||
to the flight time between frames, by at most 6 times per second of flight. It
|
||
meters the first frame before taking it. In Earth's shadow at the end,
|
||
exposures reach about 2 s, and show the target lit by the Moon and the stars
|
||
above the limb. `--out movie.mp4` encodes it with ffmpeg, rendering and tone
|
||
mapping each 1920 x 1080 frame in about 9 ms on the RTX 3060. `--serve
|
||
0.0.0.0:8010` renders it live, in step with the clock, and streams it as MJPEG
|
||
at 60 frames per second (`--fps`; at 120 it skips up to 16 frames in 5 s),
|
||
skipping about 9 more where the flight starts over, while it meters, to the
|
||
gallery page's Movie tab, which also plays and offers the recorded file
|
||
(`make-gallery.sh` writes `gallery/movie.mp4`):
|
||
|
||
```sh
|
||
cargo run --release -p resolved-showcase -- --serve 0.0.0.0:8010
|
||
```
|
||
|
||
- `--out` writes raw DN as a 16-bit PNG, shifted up to fill 16 bits as
|
||
cameras usually store them: a 12-bit DN is the stored value divided by 16.
|
||
It is linear, so it looks dark next to the preview.
|
||
- `--preview` writes an 8-bit sRGB image for looking at, with the bias as
|
||
black and saturation as white. It is the signal as recorded, so a scene
|
||
exposed to keep its brightest parts unsaturated looks dark.
|
||
- `--pretty` makes the preview look like a photograph instead: white balanced
|
||
to sunlight, with a tone curve that brightens shadows and midtones by up to
|
||
3 times and rolls highlights off to white at saturation. `--out` is the same
|
||
either way.
|
||
- `--timings N` renders N frames and reports setup and per-frame times.
|
||
- `--lighting L0|L1` and `--camera C0|C1|C2` pick the fidelity tiers (default
|
||
L0 and C0). `--supersampling N` renders each pixel as N x N finer ones from C1
|
||
(default 2). `--psf-reach N` sets how far in pixels camera C2's PSF must reach
|
||
(default 16; it reaches as far as its FFT grid allows, 64 px at 1920 x 1080 and
|
||
2 x 2). A tier the scene or the data cannot support stops with an error naming
|
||
what is missing.
|
||
- `--seed N` and `--frame N` pick the detector noise at camera C1 (default 0
|
||
and 0): the same seed and frame give the same image. `--noiseless` leaves out
|
||
the noise, dark current and fixed pattern, so two renders differ only where
|
||
their light does.
|
||
- `--data DIR` says where the tiers' data is (`earth/bmng-2004-MM.jpg`,
|
||
`moon/wac-hapke-643.png` and `stars/stars.bin`).
|
||
Without it the CLI tries `RESOLVED_DATA`, then `./data`, then the
|
||
repository's `data/`.
|
||
- `--adapter` takes `auto`, `gpu`, `cpu` or part of an adapter's name;
|
||
`resolved adapters` lists them.
|
||
- `--kernels DIR` says where astrokit's kernels are, for scenes given by
|
||
epoch. Without it the CLI tries `ASTROKIT_KERNELS`, then `data/kernels`,
|
||
then the sibling `../astrokit` checkout it was built against.
|
||
|
||
## Features
|
||
|
||
- `astrokit`: build a `Scene` from an epoch with
|
||
[astrokit](https://git.rcjohnstone.com/connor/astrokit)'s ephemerides (JPL
|
||
DE440s for the Sun and Moon, ITRF93 for Earth's orientation), through
|
||
`ephemeris::Ephemeris`. astrokit is not on crates.io, so this expects a
|
||
checkout at `../astrokit` next to this repository, and its kernels from its
|
||
`scripts/fetch-kernels.sh` (or `ASTROKIT_KERNELS` pointing at them).
|
||
|
||
## Status
|
||
|
||
Lighting L0 and L1 and cameras C0, C1 and C2 are in: the library renders
|
||
Earth, Moon, Sun and targets to DN through `Renderer`, and the CLI renders
|
||
scene files. Lighting L1 draws Earth's surface from Blue Marble and the Moon's
|
||
from the LRO Camera's map, shadows on the target, light from Earth and the
|
||
Moon on it, eclipses on Earth and the Moon, the Sun's limb darkening, and
|
||
stars from Hipparcos, Tycho-2 and Gaia, which camera C1 shows. Camera C1 adds
|
||
pixels that collect light over their area, lens distortion (Brown-Conrady),
|
||
lens blur (a Gaussian PSF) applied to the scene rendered 2 x 2 finer than the
|
||
pixels, the Sun, Moon and Earth at their apparent positions (light time and
|
||
aberration), detector noise (shot, dark current and read noise, seeded and
|
||
reproducible), the detector's fixed pattern (PRNU, DSNU and hot pixels), and
|
||
blooming along columns or into neighbors. Camera C2 adds diffraction (the Airy
|
||
pattern, averaged over each channel's band) and measured PSFs, applied with
|
||
their wings by FFT. Planned, in order:
|
||
|
||
- Lighting: BRDF and spectral materials, atmospheric scattering, clouds, lunar
|
||
terrain.
|
||
- Portable ray tracing, path tracing, and label outputs for ML.
|
||
- Validation against lunar, Earth and Mitsuba references; motion blur, rolling
|
||
shutter, stray light; multi-band and 4k.
|
||
|
||
## Conventions
|
||
|
||
- SI units throughout: meters, seconds, radians, watts.
|
||
- Scene state is f64 in an inertial frame (GCRF). The GPU only sees f32
|
||
coordinates relative to the camera.
|
||
|
||
## Layout
|
||
|
||
- `crates/resolved`: the library.
|
||
- `crates/resolved-cli`: the `resolved` command, for rendering without writing
|
||
a host program.
|
||
- `examples/`: scene files and the box-wing model
|
||
(`scripts/make-example-model.py` regenerates it). The NASA scenes are
|
||
generated by `scripts/make-example-scenes.py`.
|
||
- `scripts/`: generators for compiled-in data and examples.
|
||
- `docs/optics.md`: the optics and radiometry the renderer models, explained
|
||
for readers without an optics background.
|
||
|
||
## Development
|
||
|
||
Point git at the committed hooks once per clone:
|
||
|
||
```sh
|
||
git config core.hooksPath hooks
|
||
```
|
||
|
||
`pre-commit` runs formatting, clippy and the library unit tests; `pre-push`
|
||
runs the full test suite, including doctests, and the documentation build.
|
||
|
||
The integration tests render on both a hardware GPU and lavapipe, so running
|
||
them needs a Vulkan driver for the GPU and lavapipe installed. Set
|
||
`WGPU_BACKEND`, `WGPU_ADAPTER_NAME` and the other standard wgpu environment
|
||
variables to steer adapter discovery.
|
||
|
||
Benchmarks are end-to-end frame times (`submit` then `wait`, readback
|
||
included) on the GPU and on lavapipe, at two resolutions, for the box-wing and,
|
||
once fetched, Landsat 8 and ICESat-2:
|
||
|
||
```sh
|
||
cargo bench -p resolved
|
||
```
|
||
|
||
For where a frame's time goes, `resolved render --timings N` adds GPU
|
||
timestamps around each pass (`RendererOptions::profile` in the library).
|
||
|
||
To see what the GPU tests render, set `RESOLVED_TEST_IMAGES=1`. Each test then
|
||
writes a PNG per device to `target/tmp/test-images/`, scaled so a white
|
||
surface facing the Sun is full white:
|
||
|
||
```sh
|
||
RESOLVED_TEST_IMAGES=1 cargo test --test gpu -- bodies:: mesh::
|
||
```
|
||
|
||
## License
|
||
|
||
MIT
|