resolved/readme.md

# resolved

Renders what a spacecraft camera sees: Earth, Moon, Sun and a target
spacecraft loaded from a mesh. The inputs are an epoch, the observer and target
states and attitudes, and a camera model. Images are radiometric: the renderer
works in radiance and converts to electrons and then DN through the camera's
optics and detector, so an image should match what a real camera with the same
parameters records.

It is built for closed-loop simulation, where each step renders one frame and
latency is the number that matters. The goal is high-fidelity 1–2 MP RGB in
about 10 ms end to end on a mid-range GPU (the reference is an RTX 3060);
100 ms is the ceiling, not the aim. Lighting fidelity and camera fidelity are
selected separately, and each tier is benchmarked, so a run can trade
fidelity for speed knowing what each step costs.

Rendering goes through wgpu. On a machine without a GPU, the same shaders run
on the CPU through Mesa's lavapipe (`vulkan-swrast` on Arch,
`mesa-vulkan-drivers` on Debian and Ubuntu).

## Performance and fidelity

Lighting and camera fidelity are chosen separately: lighting tiers L0 to L3,
camera tiers C0 to C3, set with `RendererOptions::fidelity` in the library and
`--lighting` and `--camera` on the command line. A tier is a fixed bundle of
features, and a tier that needs data (L1 needs an image of Earth's surface)
refuses to start without it. A row is added here as each tier is complete, with
its measured cost and a sample image from the same scene, so the cost of each
step up is visible at a glance.

| Tier | Lighting | Camera | RTX 3060 | lavapipe (CPU) | Sample |
|---|---|---|---|---|---|
| L0 / C0 | Lambertian surfaces, Earth's and the Moon's limbs exact, eclipse by Earth and the Moon, no shadows on the target, no light from Earth | Pinhole, absolute radiometry to 12-bit DN, no noise or blur | 0.9–1.3 ms | 5–68 ms | [![L0/C0](docs/images/l0-c0.png)](docs/images/l0-c0.png) |
| L1 / C0 | Adds Earth's surface from Blue Marble and the Moon's from the LRO Camera's map, the Moon's measured phase curve, shadows on the target, Earthshine and Moonshine, eclipses seen on Earth and the Moon, the Sun's limb darkening | Pinhole, absolute radiometry to 12-bit DN, no noise or blur | 1.3–2.3 ms | 12–114 ms | [![L1/C0](docs/images/l1-c0.png)](docs/images/l1-c0.png) |

Times are one 1920 x 1080 RGB frame end to end (`submit` to `wait_view`, with
the DN image ready to read), median of criterion runs, over models from 12 to
752,000 triangles: the box-wing, Landsat 8 and ICESat-2. The GPU's own work is
0.7 to 0.9 ms of that at L0 and 1.1 to 1.5 ms at L1, of which 0.48 ms is the
sensor pass writing the image into memory the CPU reads. `wait`, which copies
the 12 MB image into a new `Vec`, takes 1.8 to 2.0 ms on the RTX 3060 at L0
and 2.3 to 3.1 ms at L1, and on lavapipe 7 to 77 ms and 14 to 144 ms. L1's
setup also loads the Earth and Moon maps, about 1 s, once. lavapipe ran on the
Intel Core Ultra 9 285K's 24 cores. Run to run, the RTX 3060's times vary by
about 5 %, and lavapipe's by up to 15 % between sessions. The samples are the
Landsat 8 example scene (`examples/scenes/landsat8.toml`). `cargo bench -p
resolved` reproduces the numbers (L1 when the maps are in `data/`), and
`resolved render --timings N` shows the per-pass split.

Planned: C1 and C2 add pixel-area sampling, lens distortion, blur and detector
noise; L2 adds measured materials and the atmosphere; L3 is path traced.

## Usage

```rust
use resolved::nalgebra::{UnitQuaternion, Vector3};
use resolved::radiometry::AU;
use resolved::{CameraConfig, Mesh, Pose, Renderer, Scene, Target};

# fn main() -> Result<(), resolved::Error> {
// Setup, once: device, shaders, spectra folded into the camera's channels.
let mut renderer = Renderer::new(CameraConfig::example(), Default::default())?;

// A model, uploaded once. Usually `Mesh::load(path, &options)` on an STL or glTF.
let stl = "solid panel
facet normal 0 0 0
outer loop
vertex 0 -1 -1
vertex 0 1 -1
vertex 0 0 1
endloop
endfacet
endsolid panel";
let panel = renderer.load_model(&Mesh::from_stl(&mut std::io::Cursor::new(stl), "panel", &Default::default())?)?;

// Each step: inertial positions in meters, Earth at the origin.
let target = Vector3::new(7.0e6, 0.0, 0.0);
let moon = Vector3::new(0.0, 3.844e8, 0.0);
let scene = Scene {
    earth_attitude: UnitQuaternion::identity(),
    sun: Vector3::new(AU, 0.0, 0.0),
    moon,
    // Which way the Moon's surface map faces; an ephemeris gives it exactly.
    moon_attitude: Scene::moon_facing_earth(moon),
    targets: vec![Target { model: panel, pose: Pose::new(target, UnitQuaternion::identity()) }],
};
let observer = Pose::looking_at(target + Vector3::new(20.0, 0.0, 0.0), target, Vector3::z()).unwrap();

let pending = renderer.submit(&scene, &observer)?;
// ... the simulation's own work for this step overlaps the GPU's here ...
let frame = pending.wait()?;
// (`pending.wait_view()` lends the image where the GPU left it instead of copying it,
// saving 0.6 ms or more at 1920 x 1080; the next `submit` waits until it is dropped.)

// 12-bit DN, channels r, g, b interleaved. The panel faces the Sun and the camera.
assert!(frame.dn_at(960, 540, 1) > 1000);
# Ok(())
# }
```

## Command line

`resolved render` renders a scene described in TOML: camera overrides, the
observer, Sun and Moon positions (or an epoch, with the `astrokit` feature) and
targets with their models. `examples/scenes/` has commented examples:
`boxwing.toml` uses a small generated model that is in the repository, `lunar-eclipse.toml` and `solar-eclipse.toml` show two
real eclipses at lighting L1, `full-moon.toml` the Moon's map, and the rest use NASA spacecraft models
(Landsat 8, ICESat-2, OSTM/Jason-2, an SSL-1300 and ACE) that
`scripts/fetch-models.sh` downloads into `examples/models/nasa/`. That script needs Node (for `npx`),
because most NASA models are Draco-compressed and are decoded on the way in.
At lighting L1, Earth's surface comes from NASA's Blue Marble, which
`scripts/fetch-earth.sh` downloads and resizes (with ImageMagick) into
`data/earth/`, one image per month. The month comes from the scene's epoch, or
from `[appearance] earth_month` in a scene without one. The Moon's surface
comes from the LRO Camera's normalized-reflectance mosaic, which
`scripts/fetch-moon.py` downloads (1.2 GB) and assembles (with numpy and Pillow)
into `data/moon/wac-hapke-643.png`. The Moon is turned by the scene's epoch, or
by `moon_attitude` in a scene without one (by default facing Earth, which can
be off by up to about 10 degrees).

```sh
cargo run --release -p resolved-cli -- render \
    --scene examples/scenes/boxwing.toml \
    --out frame.png --preview preview.png --timings 50
```

`scripts/make-gallery.sh` renders every example scene at L0 and L1 into
`gallery/`, skipping any whose model or data is missing, and
`scripts/serve-gallery.py` serves the result as a web page on port 8000 of
every interface, each scene captioned from its file.

- `--out` writes raw DN as a 16-bit PNG, shifted up to fill 16 bits as
  cameras usually store them: a 12-bit DN is the stored value divided by 16.
  It is linear, so it looks dark next to the preview.
- `--preview` writes an 8-bit sRGB image for looking at, with the bias as
  black and saturation as white.
- `--timings N` renders N frames and reports setup and per-frame times.
- `--lighting L0|L1` and `--camera C0` pick the fidelity tiers (default L0
  and C0). A tier the scene or the data cannot support stops with an error
  naming what is missing.
- `--data DIR` says where the tiers' data is (`earth/bmng-2004-MM.jpg` and
  `moon/wac-hapke-643.png`).
  Without it the CLI tries `RESOLVED_DATA`, then `./data`, then the
  repository's `data/`.
- `--adapter` takes `auto`, `gpu`, `cpu` or part of an adapter's name;
  `resolved adapters` lists them.
- `--kernels DIR` says where astrokit's kernels are, for scenes given by
  epoch. Without it the CLI tries `ASTROKIT_KERNELS`, then `data/kernels`,
  then the sibling `../astrokit` checkout it was built against.

## Features

- `astrokit`: build a `Scene` from an epoch with
  [astrokit](https://git.rcjohnstone.com/connor/astrokit)'s ephemerides (JPL
  DE440s for the Sun and Moon, ITRF93 for Earth's orientation), through
  `ephemeris::Ephemeris`. astrokit is not on crates.io, so this expects a
  checkout at `../astrokit` next to this repository, and its kernels from its
  `scripts/fetch-kernels.sh` (or `ASTROKIT_KERNELS` pointing at them).

## Status

Lighting L0 and L1 and camera C0 are in: the library renders Earth, Moon, Sun
and targets to DN through `Renderer`, and the CLI renders scene files. Lighting
L1 draws Earth's surface from Blue Marble and the Moon's from the LRO Camera's
map, shadows on the target, light from Earth and the Moon on it, eclipses on
Earth and the Moon, and the Sun's limb darkening. Planned, in order:

- Camera: pixel-area sampling, lens distortion, PSF, detector noise,
  saturation and blooming, stars, apparent positions.
- Lighting: BRDF and spectral materials, atmospheric scattering, clouds, lunar
  terrain.
- Portable ray tracing, path tracing, and label outputs for ML.
- Validation against lunar, Earth and Mitsuba references; motion blur, rolling
  shutter, stray light; multi-band and 4k.

## Conventions

- SI units throughout: meters, seconds, radians, watts.
- Scene state is f64 in an inertial frame (GCRF). The GPU only sees f32
  coordinates relative to the camera.

## Layout

- `crates/resolved`: the library.
- `crates/resolved-cli`: the `resolved` command, for rendering without writing
  a host program.
- `examples/`: scene files and the box-wing model
  (`scripts/make-example-model.py` regenerates it). The NASA scenes are
  generated by `scripts/make-example-scenes.py`.
- `scripts/`: generators for compiled-in data and examples.
- `docs/optics.md`: the optics and radiometry the renderer models, explained
  for readers without an optics background.

## Development

Point git at the committed hooks once per clone:

```sh
git config core.hooksPath hooks
```

`pre-commit` runs formatting, clippy and the library unit tests; `pre-push`
runs the full test suite, including doctests, and the documentation build.

The integration tests render on both a hardware GPU and lavapipe, so running
them needs a Vulkan driver for the GPU and lavapipe installed. Set
`WGPU_BACKEND`, `WGPU_ADAPTER_NAME` and the other standard wgpu environment
variables to steer adapter discovery.

Benchmarks are end-to-end frame times (`submit` then `wait`, readback
included) on the GPU and on lavapipe, at two resolutions, for the box-wing and,
once fetched, Landsat 8 and ICESat-2:

```sh
cargo bench -p resolved
```

For where a frame's time goes, `resolved render --timings N` adds GPU
timestamps around each pass (`RendererOptions::profile` in the library).

To see what the GPU tests render, set `RESOLVED_TEST_IMAGES=1`. Each test then
writes a PNG per device to `target/tmp/test-images/`, scaled so a white
surface facing the Sun is full white:

```sh
RESOLVED_TEST_IMAGES=1 cargo test --test gpu -- bodies:: mesh::
```

## License

MIT