1665 lines
89 KiB
Markdown
1665 lines
89 KiB
Markdown
# Optics and radiometry guide
|
|
|
|
This explains the physics the renderer models, for a reader who knows
|
|
orbital mechanics but not optics. Each section says which code implements
|
|
it. When a feature adds a new optical effect, it adds or updates a section
|
|
here in the same change.
|
|
|
|
Numbers in the examples come from `CameraConfig::example()` unless stated
|
|
otherwise: 1920 x 1080 pixels, 5.5 um pitch, 50 mm focal length, f/4, three
|
|
color channels, 12-bit output.
|
|
|
|
## The chain from Sun to number
|
|
|
|
Every pixel value comes from the same sequence:
|
|
|
|
1. The Sun emits power across a range of wavelengths.
|
|
2. Some of it lands on a surface (Earth, the Moon, the target), which
|
|
reflects part of it in various directions.
|
|
3. The fraction reflected toward the camera enters the lens aperture. The
|
|
optics focus it onto the detector, losing a little to absorption. Each
|
|
point of the scene lands as a small spot rather than a point.
|
|
4. The pixel converts some of the arriving photons into electrons. The
|
|
fraction depends on wavelength and on the color filter over that pixel.
|
|
5. The electrons pile up during the exposure, up to a limit.
|
|
6. The readout electronics turn the electron count into an integer, the
|
|
digital number (DN).
|
|
|
|
The renderer follows this chain with physical units at every step. Image
|
|
brightness therefore comes from the scene and the camera settings, not from
|
|
a tuned scale factor. Change the exposure time, and the image changes
|
|
exactly as a real camera's would.
|
|
|
|
## Geometry: the pinhole camera
|
|
|
|
Code: `camera.rs` (`Intrinsics`), `scene.rs`.
|
|
|
|
A pinhole camera maps each direction in space to one point on the focal
|
|
plane. Put the pinhole at the origin, looking along +Z. A point at camera
|
|
coordinates (x, y, z) lands on the focal plane at (f x / z, f y / z), where
|
|
f is the focal length. Dividing by the pixel pitch p converts focal-plane
|
|
distance to pixels:
|
|
|
|
```text
|
|
u = (f / p) * x / z + cx
|
|
v = (f / p) * y / z + cy
|
|
```
|
|
|
|
`f / p` is the focal length in pixels. (cx, cy) is the principal point,
|
|
where the boresight hits the image, normally the image center. For the
|
|
example camera, f / p = 50 mm / 5.5 um = 9091 pixels.
|
|
|
|
Some quantities that follow from that:
|
|
|
|
- **IFOV, the instantaneous field of view.** The angle one pixel covers:
|
|
p / f = 110 microradians. At 10 m that is 1.1 mm per pixel; at 7000 km
|
|
it is 770 m per pixel. This sets the smallest detail the camera can
|
|
resolve.
|
|
- **FOV, the field of view.** The angle the whole image covers:
|
|
2 atan(width / 2 / (f / p)), about 12.1 x 6.8 degrees here. A longer
|
|
focal length or smaller sensor gives a narrower field of view and finer
|
|
IFOV.
|
|
|
|
Real lenses are not perfect pinholes. They bend straight lines slightly
|
|
(distortion, next) and spread each point into a small blur (the point
|
|
spread function, below). The pinhole model is the starting point those
|
|
effects are added to.
|
|
|
|
### Lens distortion
|
|
|
|
Code: `camera.rs` (`BrownConrady`, `Intrinsics`); `gpu/warp.rs`,
|
|
`gpu/blur.rs`, `shaders/blur.wgsl`, `shaders/distortion.wgsl`.
|
|
|
|
A real lens moves each point a little away from where the pinhole puts it,
|
|
by an amount that grows toward the edge of the image. Straight lines in the
|
|
scene come out curved. When points move outward more the farther out they
|
|
are, the corners stretch (pincushion distortion); when they move inward,
|
|
the image bulges (barrel distortion). A lens element slightly off center
|
|
adds a smaller sideways shift (tangential distortion).
|
|
|
|
The standard model, Brown-Conrady, works on the pinhole's normalized
|
|
coordinates x = X / Z and y = Y / Z, with r^2 = x^2 + y^2:
|
|
|
|
```text
|
|
x_d = x (1 + k1 r^2 + k2 r^4 + k3 r^6) + 2 p1 x y + p2 (r^2 + 2 x^2)
|
|
y_d = y (1 + k1 r^2 + k2 r^4 + k3 r^6) + p1 (r^2 + 2 y^2) + 2 p2 x y
|
|
u = (f / p) x_d + cx, v = (f / p) y_d + cy
|
|
```
|
|
|
|
k1, k2 and k3 are the radial terms, positive for pincushion and negative
|
|
for barrel; p1 and p2 are the tangential terms. This is OpenCV's convention,
|
|
so the coefficients a camera calibration produces apply directly (OpenCV
|
|
lists them in the order k1, k2, p1, p2, k3). For the example camera, k1 =
|
|
0.1 alone moves the corners 1.6 px outward.
|
|
|
|
That matters for navigation, not just for looks. A limb fit, a star
|
|
identification or a target's centroid all assume where each pixel looks;
|
|
an unmodeled 1.6 px is a 0.2 mrad pointing error at the corner, which at
|
|
36,000 km is 6 km. `Intrinsics::project` and `Intrinsics::ray` include the
|
|
distortion (the inverse by Newton's method, to 1e-10 px), so navigation code
|
|
and the renderer share one model; the projection matches OpenCV's to 1e-9
|
|
px.
|
|
|
|
Rendering it, from camera tier C1 (C0 is always a pinhole): the GPU draws
|
|
the target's triangles with a linear projection, which keeps straight edges
|
|
straight, and moving their corners alone would leave long edges straight
|
|
when they should curve. So the renderer draws the whole frame without
|
|
distortion, the bodies and targets sorted together as usual, on a slightly
|
|
larger and offset area that covers everything the lens bends into view.
|
|
Then it resamples it: for each pixel of the focal plane it finds the
|
|
undistorted point the lens bends onto that pixel and reads the render
|
|
there, interpolating between the four nearest render pixels. The blur's
|
|
first pass (below) does this as it reads its input, so the distorted image
|
|
is never written out. Radiance is
|
|
the same at both ends of a ray, so moving it is all distortion does to an
|
|
extended scene. The resampling works on the finer image C1 renders (below),
|
|
before the blur, which is where the lens puts it in a real camera.
|
|
|
|
Two effects of distortion fall on the detector instead. The cos^4 falloff
|
|
uses each pixel's real line of sight, the undistorted one. And where the
|
|
lens stretches the image, the same light covers more pixels: each pixel's
|
|
signal is divided by how much the distortion stretches area there (the
|
|
determinant of its derivative), 0.6 % at the corners for k1 = 0.1.
|
|
|
|
Measured with coefficients of 0.3, -0.2, 0.1 (radial) and 0.001, -0.0008
|
|
(tangential), which move the corners about 4 px: a small Moon's light lands
|
|
centered where `Intrinsics::project` puts it to within 0.004 px across the
|
|
frame, and a long straight edge follows the predicted curve to within 0.07
|
|
px in every row and 0.02 px averaged over rows. The interpolation adds a
|
|
little blur where it falls between render pixels, at most a variance of
|
|
1/16 px^2 at 2 x 2 rendering (computed), which takes sigma 0.5 px to 0.56 px
|
|
at worst.
|
|
|
|
Distortion costs 0.25 to 0.32 ms per 1920 x 1080 frame on an RTX 3060 at
|
|
2 x 2 rendering, and 1.1 to 1.3 ms at 3840 x 2160, almost all of it the
|
|
blur's first pass reading through it, and under 0.1 ms the larger
|
|
undistorted render. A separate pass writing the distorted image out and the
|
|
blur reading it back cost 0.7 to 0.8 ms and 2.8 to 3.3 ms: at those sizes
|
|
both are limited by how fast the card moves the image through memory. On
|
|
lavapipe distortion adds 10 to 11 ms to 1920 x 1080 frames.
|
|
|
|
### Coordinate conventions
|
|
|
|
The camera frame has +Z along the boresight, +X toward increasing column u,
|
|
and +Y toward increasing row v. This is the computer-vision convention: row
|
|
0 is the top of the image, so +Y points down in the picture. Pixel (i, j)
|
|
covers u in [i, i+1) and v in [j, j+1), so its center is at (i + 0.5,
|
|
j + 0.5).
|
|
|
|
### Why the GPU sees camera-relative coordinates
|
|
|
|
GPUs compute in 32-bit floats, which carry about 7 significant digits. A
|
|
spacecraft 7000 km from Earth's center, stored in Earth-centered f32
|
|
coordinates, can only be placed to the nearest 0.5 m. A target 10 m away
|
|
would jitter by several pixels. So all positions stay in 64-bit floats on
|
|
the CPU. They are made relative to the camera there, and converted to
|
|
32-bit only after that. A target 10 m from the camera is then placed to
|
|
about a micrometer.
|
|
|
|
### Depth
|
|
|
|
When two things overlap in the image, the nearer one must win. The GPU
|
|
keeps a depth value per pixel for this. The renderer stores near / z,
|
|
which is 1 at the near clip distance and falls toward 0 with distance.
|
|
Combined with how floating-point numbers are spaced, this gives roughly the
|
|
same relative precision at every distance. That matters in a scene that
|
|
contains a 10 m target, Earth at thousands of kilometers, and the Sun at
|
|
150 million kilometers. The scheme is called reversed-Z with an infinite
|
|
far plane.
|
|
|
|
## Radiometric quantities
|
|
|
|
Code: `spectrum.rs`, `radiometry.rs`.
|
|
|
|
Light has a few distinct measures, and the camera measures a specific one.
|
|
|
|
- **Irradiance, E, in W/m^2.** Power arriving per unit area of a surface.
|
|
Sunlight at 1 AU delivers about 1361 W/m^2 to a surface facing the Sun.
|
|
It falls off with the square of the distance from the Sun. A surface
|
|
tilted by angle theta from the Sun direction receives E cos(theta).
|
|
- **Solid angle, in steradians (sr).** The 2D version of an angle: the area
|
|
a cone cuts out of a unit sphere. The full sphere is 4 pi sr. One pixel
|
|
of the example camera covers (110 urad)^2, about 1.2e-8 sr.
|
|
- **Radiance, L, in W/m^2/sr.** Power per unit area per unit solid angle,
|
|
traveling in a particular direction. It is what a camera pixel responds
|
|
to: how bright a surface looks from a given direction.
|
|
|
|
Radiance has a property that explains a lot of what a camera sees: it does
|
|
not change along a ray through empty space. Move a camera twice as far from
|
|
a wall, and each pixel sees four times as much wall area. But each square
|
|
meter now contributes a quarter of the power, so the pixel value is
|
|
unchanged. An object that covers many pixels has the same surface
|
|
brightness at any distance; it just covers fewer pixels farther away.
|
|
|
|
That stops being true once the object is smaller than a pixel. The pixel
|
|
then also sees dark space around the object, so the signal falls as one
|
|
over distance squared. That is the regime of a distant satellite seen as a
|
|
point (space situational awareness), as opposed to the resolved imagery
|
|
this renderer is mostly for.
|
|
|
|
### Reflection: albedo and Lambertian surfaces
|
|
|
|
A Lambertian surface reflects light equally in every direction. Matte
|
|
paint, the Moon (roughly) and cloud tops behave close to this. If such a
|
|
surface receives irradiance E and reflects a fraction rho (its albedo), its
|
|
radiance in every direction is:
|
|
|
|
```text
|
|
L = rho * E / pi
|
|
```
|
|
|
|
The pi comes from integrating the reflected radiance over the hemisphere
|
|
above the surface. A white Lambertian sheet facing the Sun at 1 AU, with
|
|
rho = 1, has a radiance of about 1361 / pi = 433 W/m^2/sr, summed over all
|
|
wavelengths. Earth's average albedo is about 0.3.
|
|
|
|
Real spacecraft surfaces are not Lambertian. Multilayer insulation (MLI)
|
|
and solar cells are partly mirror-like and produce bright glints at
|
|
particular angles. These are modeled by a BRDF, a function giving reflected
|
|
radiance for every pair of incoming and outgoing directions. BRDFs are not
|
|
modeled yet.
|
|
|
|
## Wavelength: spectra, channels and photons
|
|
|
|
Code: `spectrum.rs` (`Spectrum`), `camera.rs` (`Channel`).
|
|
|
|
Everything above varies with wavelength. Sunlight peaks around 500 nm and
|
|
extends from the ultraviolet into the infrared. Surfaces reflect different
|
|
wavelengths differently; that is what color is. Optics transmit some
|
|
wavelengths better than others. The sensor's sensitivity also depends on
|
|
wavelength.
|
|
|
|
The renderer describes each of these as a `Spectrum`: a value as a function
|
|
of wavelength, in meters.
|
|
|
|
- **Quantum efficiency (QE).** The fraction of arriving photons that become
|
|
electrons in a pixel. A color camera puts red, green and blue filters
|
|
over different pixels, so each channel has its own QE curve. The example
|
|
camera uses idealized Gaussian curves centered at 620, 540 and 460 nm,
|
|
80 nm wide at half height, peaking at 60 %.
|
|
- **Transmission (tau).** The fraction of light the lenses pass, about 0.9
|
|
for the example camera.
|
|
|
|
Detectors count photons, not energy. A photon of wavelength lambda carries
|
|
energy h c / lambda (h is Planck's constant, c the speed of light). So one
|
|
watt of red light delivers more photons than one watt of blue. The
|
|
renderer converts power to photons per wavelength before applying QE.
|
|
|
|
### The solar spectrum
|
|
|
|
Code: `radiometry.rs` (`solar_spectrum`), generated into `solar_data.rs` by
|
|
`scripts/fetch-solar-spectrum.sh`.
|
|
|
|
The renderer uses a measured solar spectrum, the TSIS-1 Hybrid Solar
|
|
Reference Spectrum (Coddington et al. 2021). It is the current
|
|
international reference, built mainly from the TSIS-1 instruments on the
|
|
International Space Station and filled in with a high-resolution model of
|
|
the solar absorption lines. It gives the irradiance at 1 AU, outside the atmosphere,
|
|
every 1 nm from 202 to 2730 nm. Its uncertainty is 0.3 % from 460 to
|
|
2365 nm. Over that range it sums to 1325.8 W/m^2. The other 35 W/m^2 of the
|
|
1361 W/m^2 total is at wavelengths silicon detectors do not see.
|
|
|
|
The Sun is often approximated as a 5772 K blackbody: a smooth curve with
|
|
the right total power. The real spectrum has thousands of absorption lines
|
|
from the Sun's outer layers, and its shape differs from the smooth curve.
|
|
For the example camera's channels, the blackbody gives 4.6 % (red), 7.5 %
|
|
(green) and 9.7 % (blue) fewer electrons than the measured spectrum. That
|
|
is far outside what absolute radiometry can tolerate.
|
|
|
|
### Folding spectra into per-channel constants
|
|
|
|
Code: `radiometry.rs` (`Radiometry`).
|
|
|
|
Doing all of this wavelength by wavelength for every pixel would be slow.
|
|
Most factors do not depend on the pixel: the solar spectrum, the optics,
|
|
and the QE curves. The renderer multiplies and integrates those over
|
|
wavelength once, when the camera is set up. That yields a few numbers per
|
|
channel, and the GPU only multiplies by them.
|
|
|
|
The central number is the solar flux of each channel k:
|
|
|
|
```text
|
|
S_k = integral of E_sun(lambda) * tau(lambda) * QE_k(lambda) * lambda / (h c) d lambda
|
|
```
|
|
|
|
It is the rate of electrons per square meter that sunlight at 1 AU would
|
|
produce if it fell straight onto the detector through the optics. For the
|
|
example camera, S_k is about 2.4e20 electrons per second per m^2 in red and
|
|
green, and 2.0e20 in blue.
|
|
|
|
The renderer then works in channel radiance: scene radiance weighted the
|
|
same way (transmission, QE, photons per joule), in electrons per second
|
|
per m^2 per steradian. A Lambertian surface with albedo rho, lit at angle i
|
|
from its normal, at distance d from the Sun in AU, has channel radiance:
|
|
|
|
```text
|
|
L_k = rho * cos(i) * S_k / (pi * d^2)
|
|
```
|
|
|
|
The solar disk has channel radiance S_k / Omega_sun, where Omega_sun is the
|
|
Sun's solid angle at 1 AU, 6.79e-5 sr (it is 0.53 degrees across). Seen
|
|
from anywhere, the Sun's surface has that same radiance, about 2.0e7
|
|
W/m^2/sr in total. Looked at directly, it puts about 5e8 electrons into a
|
|
pixel of the example camera's green channel in 0.1 ms, 26,000 times the
|
|
full well.
|
|
|
|
A surface whose reflectance varies with wavelength is reduced to an
|
|
effective albedo per channel. That is the albedo of a flat-spectrum surface
|
|
that would give the same signal under sunlight
|
|
(`Radiometry::effective_albedo`). This step is where the error of the
|
|
shortcut comes in. Two spectra that both vary inside one channel's band
|
|
interact in ways a single number per channel cannot capture. One example is
|
|
a surface whose reflectance changes steeply across green, lit partly by
|
|
sunlight and partly by bluish light scattered from Earth's atmosphere.
|
|
Rendering more, narrower channels reduces that error, and the code keeps
|
|
the channel count generic for that reason.
|
|
|
|
## From radiance to electrons: the camera equation
|
|
|
|
Code: `radiometry.rs` (`Radiometry::electrons`); `gpu/sensor.rs`, `shaders/sensor.wgsl`.
|
|
|
|
A lens with focal length f and aperture diameter D has f-number
|
|
N = f / D. The example camera is f/4: a 50 mm lens with a 12.5 mm aperture.
|
|
For a scene radiance L, the irradiance on the focal plane at the image
|
|
center is:
|
|
|
|
```text
|
|
E_image = pi * L * tau / (4 N^2)
|
|
```
|
|
|
|
The aperture collects light over a solid angle proportional to (D / f)^2 =
|
|
1 / N^2. So opening the aperture one "stop", which divides N by the square
|
|
root of 2, doubles the light. Image irradiance is independent of distance
|
|
for the same reason radiance is.
|
|
|
|
Away from the center, image irradiance falls off as cos^4(theta), where
|
|
theta is the angle from the boresight. Two factors come from the longer
|
|
path to the off-axis pixel. One comes from the aperture appearing
|
|
foreshortened. One comes from light reaching the pixel at a slant. At the
|
|
corner of the example camera, theta is 6.9 degrees, and the falloff is
|
|
3 %. Real lenses add more falloff of their own, called vignetting.
|
|
|
|
The electrons a pixel collects in one channel are then:
|
|
|
|
```text
|
|
electrons = E_image * (pixel area) * (exposure time) * (photons per joule) * QE
|
|
```
|
|
|
|
The last two factors are wavelength-dependent. In channel radiance they are
|
|
already folded in, so the camera equation reduces to
|
|
|
|
```text
|
|
electrons = L_k * pi * (pixel area) * (exposure time) / (4 N^2) * cos^4(theta)
|
|
```
|
|
|
|
For the example camera, pi * A * t / (4 N^2) is 1.48e-16 m^2 sr s.
|
|
|
|
## From electrons to DN: the detector
|
|
|
|
Code: `camera.rs` (`Detector::dn`, `Detector::variance`); `gpu/sensor.rs`,
|
|
`shaders/sensor.wgsl`.
|
|
|
|
- **Full well.** The maximum number of electrons a pixel holds. Beyond it
|
|
the pixel saturates and reads its maximum value whatever the light.
|
|
Brighter scenes do not look brighter. In real sensors, excess charge can
|
|
also spill into neighboring pixels (blooming, below).
|
|
- **Gain.** Electrons per DN. With a gain of 5, every 5 electrons add 1 to
|
|
the output.
|
|
- **Bias.** A fixed offset added to every pixel, so that noise around zero
|
|
signal does not get clipped at 0. The example uses 64 DN.
|
|
- **ADC bits.** The output resolution. 12 bits gives values 0 to 4095.
|
|
|
|
So DN = bias + min(electrons, full well) / gain, rounded and clipped to the
|
|
ADC range. The renderer does this on the GPU for every pixel and channel,
|
|
as the last step of a frame, and reads back only the resulting integers.
|
|
From camera tier C1, noise enters between the electrons and the gain (see
|
|
Noise below).
|
|
|
|
Two consequences of these numbers show up in every image:
|
|
|
|
- **Saturation is not always the ADC's maximum.** In the example camera the
|
|
full well fills first: 20,000 / 5 + 64 = 4064 DN, below the 12-bit maximum
|
|
of 4095. A saturated pixel there reads 4064. Real cameras are designed
|
|
either way, and an algorithm that detects saturation by looking for 4095
|
|
would miss it in this one.
|
|
- **Black is not zero.** A pixel that receives no light reads the bias,
|
|
64 DN, so empty space in a raw frame displays as dark gray.
|
|
|
|
Raw frames are also not white-balanced. The camera reports what each
|
|
channel collected, and the channels collect different amounts from the same
|
|
gray surface. In the example camera, sunlight on a gray surface gives about
|
|
15 % fewer electrons in blue than in red, so gray things look slightly
|
|
yellow-green in a raw image. A white balance step, applied after capture,
|
|
would scale the channels to make them equal. The renderer leaves it out,
|
|
because it simulates what the sensor records.
|
|
|
|
### Worked example
|
|
|
|
Take a white Lambertian surface facing the Sun at 1 AU, seen by the example
|
|
camera's green channel. Integrating the solar spectrum through the green QE
|
|
curve, f/4 optics, 5.5 um pixels and a 0.1 ms exposure gives 11,139
|
|
electrons. Red gets 11,408 and blue 9,626. (A test in `radiometry.rs`
|
|
checks the green figure.)
|
|
|
|
- DN = 64 + 11,139 / 5, about 2292, a little over half the 12-bit range.
|
|
- The full well is 20,000 electrons, so this surface is at 56 % of
|
|
saturation.
|
|
- Earth, with albedo near 0.3, reads about 64 + 3,342 / 5, about 732 DN.
|
|
- The same white surface at the image corner, 6.9 degrees off axis, gets
|
|
cos^4 of that, 97 %: 10,819 electrons, 2228 DN.
|
|
|
|
The sensor tests check the 2292, 2228, 732, 4064 and 64 DN figures
|
|
exactly, on the GPU and on lavapipe.
|
|
|
|
Picking the exposure time is the same trade a photographer makes. A longer
|
|
exposure saturates the sunlit target. A shorter one leaves the side of the
|
|
target lit only by Earthshine lost in noise.
|
|
|
|
### Noise
|
|
|
|
Code: `Detector::variance`; `sensor.wgsl` (`NOISE`).
|
|
|
|
A real image is noisy, and the noise decides what can be detected at all.
|
|
From camera tier C1 the renderer adds three kinds, following the EMVA 1288
|
|
standard's model of a camera:
|
|
|
|
- **Shot noise.** Light arrives as photons at random times, so a pixel
|
|
expecting N electrons collects a Poisson-distributed count with mean N and
|
|
standard deviation sqrt(N). It is a property of light, not of the camera:
|
|
a perfect detector still has it. A pixel expecting 10,000 electrons varies
|
|
by 100 (1 %); one expecting 100 varies by 10 (10 %).
|
|
- **Dark current.** The detector makes electrons of its own from heat,
|
|
`Detector::dark_current` per second, which add to the light's and carry
|
|
shot noise of their own. It grows with temperature, roughly doubling every
|
|
6 to 8 degrees C; the value given is for the detector's operating
|
|
temperature.
|
|
- **Read noise.** Reading a pixel out adds a few electrons of random error,
|
|
`Detector::read_noise` RMS, to every pixel in every frame, light or not.
|
|
|
|
In order: the light's electrons and the dark electrons together are drawn
|
|
as one Poisson count (a sum of Poisson counts is itself one); the count is
|
|
clipped to the full well; read noise is added; then the gain, bias and
|
|
rounding to DN as before. Below a mean of 50 electrons the count is drawn
|
|
exactly. Above it a Gaussian of the same mean and variance takes its place,
|
|
which a Poisson count's shape is already close to and which costs the same
|
|
at any level. With read noise the DN can fall below the bias, which is what
|
|
the bias is for.
|
|
|
|
Each pixel's variance in electrons^2 is then
|
|
|
|
variance = (electrons + dark current x exposure) + read noise^2 + gain^2 / 12
|
|
|
|
where the last term is the rounding to whole DN. Divided by gain^2 it is in
|
|
DN^2. Plotted against the mean signal in DN, the variance is a straight line
|
|
with slope 1 / gain and intercept (read noise^2 + dark shot noise) / gain^2:
|
|
the photon transfer curve, the standard way to measure a camera's gain and
|
|
read noise from flat fields alone.
|
|
|
|
For the example camera (5 electrons read noise, 20 electrons per second dark
|
|
current, gain 5) at 0.1 ms, the signal-to-noise ratio, signal over the
|
|
square root of the variance, is:
|
|
|
|
| What | Electrons | SNR |
|
|
|---|---|---|
|
|
| A white surface facing the Sun, green channel | 11,139 | 105 |
|
|
| Earth, albedo 0.3 | 3,342 | 58 |
|
|
| A white surface lit by 1 % of sunlight, as Earthshine can | 111 | 9.5 |
|
|
| A V = 0 star, over the 4 pixels it covers | 56 | 4.4 |
|
|
| A V = 4 star, over 4 pixels | 1.4 | 0.13 |
|
|
|
|
So at the exposures a sunlit target needs, the brightest stars are just
|
|
detectable and the rest are lost in read noise. At 50 ms, a star tracker's
|
|
exposure, a V = 6 star reaches an SNR of 7.5.
|
|
|
|
Dark current hardly matters at these exposures: 20 electrons per second is
|
|
0.002 electrons in 0.1 ms. It matters for long ones: 1 electron in 50 ms,
|
|
100 in 5 s, with 10 electrons of shot noise.
|
|
|
|
**Reproducible randomness.** The noise comes from a counter-based generator:
|
|
a hash (Jarzynski and Olano's pcg4d) of the pixel and channel, the frame
|
|
number and a 64-bit seed (`RendererOptions::noise_seed`), turned into
|
|
uniform numbers, then into Gaussians (Box-Muller) or a Poisson count (by
|
|
inverting its distribution). Nothing is carried from one pixel or frame to
|
|
the next, so every value is computed independently on the GPU. The renderer
|
|
numbers its frames from 0 (`Frame::number`), and
|
|
`Renderer::set_frame_number` renders a frame again with exactly the same
|
|
noise. Different frame numbers or seeds give independent noise. The hash is
|
|
identical on every device, but the logarithms and cosines that shape it may
|
|
differ in the last bit, so the GPU and lavapipe agree statistically rather
|
|
than pixel for pixel.
|
|
|
|
**Noise-free images.** A detector with no read noise, no dark current, shot
|
|
noise off and no fixed pattern (`Detector::noiseless`) renders C1 without
|
|
noise, exactly as before noise existed, for ground truth and ML labels. The
|
|
renderer then compiles the noise out entirely. At C0 there is never noise.
|
|
|
|
Measured, identically on the GPU and lavapipe: flat fields at six levels
|
|
from 100 to 16,000 electrons, two frames each, give half the variance of
|
|
their difference within 2 % of the formula (0.982 to 1.008 of it, against a
|
|
sampling error of 0.6 %), and the photon transfer curve's slope is 1 / gain
|
|
to 0.1 %. A dark frame of 100 electrons reads 20.005 DN above the bias, with
|
|
its variance within 0.6 %. Counts below 50 match the Poisson probabilities
|
|
count by count. Consecutive frames, different seeds and neighboring pixels
|
|
are uncorrelated (|r| at most 0.005 over 196,608 values). 400 faint stars of
|
|
300 electrons each vary as predicted to 0.4 %, an SNR of 7.35. Noise costs nothing measurable on an
|
|
RTX 3060 (0.476 ms for the sensor pass at 1920 x 1080 with or without it,
|
|
1.89 ms at 3840 x 2160); on lavapipe it adds 1.5 ms at 1920 x 1080 and
|
|
9.6 ms at 3840 x 2160.
|
|
|
|
### Fixed pattern: PRNU and DSNU
|
|
|
|
Code: `fixed_pattern.rs` (`FixedPattern`); `sensor.wgsl` (`PATTERN`).
|
|
|
|
No two pixels of a real detector are quite alike, and the differences are
|
|
the same in every frame. That sets them apart from the noise above, which is
|
|
new each frame:
|
|
|
|
- **PRNU (photo-response nonuniformity).** Each pixel converts light to
|
|
electrons a little more or less efficiently than average, typically by
|
|
0.5 to 1.5 % RMS. Under even light the image is mottled by that much.
|
|
- **DSNU (dark signal nonuniformity).** Each pixel's dark current differs
|
|
from the average, by tens of percent. A few pixels, from defects in the
|
|
silicon, make tens to hundreds of times the average: hot pixels.
|
|
|
|
From camera tier C1 each value (pixel and channel) has a response factor,
|
|
which multiplies its light's electrons (stars' included), and a dark factor,
|
|
which multiplies the dark current, before the shot noise and the full well.
|
|
The example camera has 1 % PRNU, 10 % DSNU, and 1 value in 10,000 hot at 50
|
|
times the dark current (`Detector::prnu`, `dsnu`, `hot_pixel_fraction`,
|
|
`hot_pixel_multiple`). Because the dark factor scales the dark current, the
|
|
dark pattern grows with the exposure, as a real one does: at 0.1 ms the
|
|
example camera's hot pixels make 0.1 electrons, invisible; over 5 s, 5,000.
|
|
|
|
The pattern comes from `Detector::pattern_seed`, apart from the noise's
|
|
seed, so one camera keeps its pattern whatever the frame or noise: the same
|
|
hash as the noise, of the seed and the value's index, gives a Gaussian
|
|
response and dark factor and decides whether the value is hot. The shader
|
|
computes it for each value as it goes, so it takes no memory and no setup.
|
|
`FixedPattern::generate` computes the same pattern on the CPU, for tests or
|
|
to correct images with. A pattern measured from a real camera can be used
|
|
instead (`FixedPattern::from_calibration`, `RendererOptions::fixed_pattern`):
|
|
a flat, a frame of even light, and a dark, a frame with none, each the
|
|
average of many so their own noise is small, bias removed; each is divided
|
|
by its mean.
|
|
|
|
This is why real images are flat-fielded and dark-subtracted before use:
|
|
divided by a flat and with a dark frame taken off, which removes the
|
|
pattern and leaves the noise. It matters for detection in particular. A hot
|
|
pixel in a long exposure is a bright, sharp point that stays put from frame
|
|
to frame, just like a faint star or a distant target; only its
|
|
single-pixel sharpness (a real point source is spread by the PSF) or a dark
|
|
frame gives it away.
|
|
|
|
Measured, on the GPU and lavapipe: with noise off, a flat field and dark
|
|
frames at two exposures read the CPU's pattern to the rounding (within
|
|
0.505 DN of the unrounded value), the hot pixels are the same ones, and the
|
|
spreads are the requested 1 % and 10 % to 2 %. With noise on, at 10,000
|
|
electrons the pattern and the noise are equal parts of the spread, and two
|
|
frames or two noise seeds correlate by the predicted 0.50, while another
|
|
pattern seed shares nothing. Calibration maps are applied to the rounding.
|
|
The pattern costs nothing measurable on an RTX 3060; on lavapipe it adds
|
|
about 2.3 ms at 1920 x 1080 and 6.3 ms at 3840 x 2160. A loaded pattern
|
|
takes 8 bytes per value, 199 MB at 3840 x 2160 in RGB.
|
|
|
|
### Blooming
|
|
|
|
Code: `Blooming`; `sensor.wgsl` (`BLOOMING`: `collect_charge`,
|
|
`bloom_columns`, `read_out`).
|
|
|
|
A saturated pixel's extra charge does not vanish. Where it goes depends on
|
|
the sensor, and matters whenever something in or near the frame is far past
|
|
the full well: the Sun at the example camera's 0.1 ms is about 26,000 full
|
|
wells at the center of its disk, and a sunlit specular part can be hundreds.
|
|
|
|
- **CCDs** move charge down columns to read it out, and the channels that
|
|
hold it are only walled off from each other across the columns. Excess
|
|
charge spills along the column, up and down, filling each pixel it
|
|
reaches before passing on: the familiar vertical streak through the Sun
|
|
or a glint, which can cross a whole frame and hide a target in it.
|
|
- **CMOS sensors** read each pixel on its own, and most have an
|
|
anti-blooming drain that carries excess charge away. What escapes it
|
|
spills into the pixels around.
|
|
|
|
From camera tier C1, `Detector::blooming` chooses: `None` (the default: the
|
|
pixel clips and the excess is lost), `Column { efficiency }` or
|
|
`Neighbors { efficiency }`, where the efficiency is the fraction of the
|
|
excess that spreads and the rest drains. Along a column, half goes up and
|
|
half down; each pixel it reaches takes what it has room for and passes the
|
|
rest on, until it runs out or leaves the frame. Into the neighbors, each of
|
|
the 8 takes an eighth, and what overfills them drains.
|
|
|
|
Blooming happens to the charge, so it comes after the shot noise (and the
|
|
fixed pattern) and before the read noise and the ADC. With blooming on, the
|
|
sensor pass works in three steps rather than one: it writes each value's
|
|
charge to a buffer, spreads it, and reads it out. Along columns the spread
|
|
is a walk up and down each column, one at a time; the first step marks the
|
|
columns that hold any excess, and the walk skips the rest, which in most
|
|
frames is all of them. The CPU reference in the tests does the same walk in
|
|
f64.
|
|
|
|
Measured, on the GPU and lavapipe: a single overfull pixel on an even
|
|
background fills the pixels it should along its column to the electron,
|
|
charge that leaves the frame is lost, and nothing else changes; the
|
|
neighbors share equally; below the full well, or at efficiency 0, blooming
|
|
changes no value. The Sun at 2e-5 efficiency, in a 256 x 256 frame of the
|
|
example optics, streaks 22 pixels each way from the middle of its disk,
|
|
matching the CPU column by column to 0 DN. Column blooming costs 0.23 ms per
|
|
1920 x 1080 frame on an RTX 3060 with nothing past the full well and 0.75 ms
|
|
with the Sun in the frame (1.0 and 2.2 ms at 3840 x 2160); into the
|
|
neighbors, 0.24 ms (1.0 ms). On lavapipe, about 3 ms and 10 to 12 ms at
|
|
1920 x 1080.
|
|
|
|
## How shapes become pixels
|
|
|
|
Two techniques turn 3D shapes into an image. The renderer uses one for the
|
|
bodies and the other for the target.
|
|
|
|
- **Mesh.** A surface approximated by many small flat triangles that share
|
|
corners (vertices). A cube needs 12 triangles. A curved surface needs
|
|
many more, and it is still faceted up close. Each vertex stores a position
|
|
and usually a normal, the direction the surface faces there. STL and glTF
|
|
files are meshes, and so is the target spacecraft.
|
|
- **Rasterization.** The GPU's built-in way of drawing a mesh. Each
|
|
triangle's corners are projected into the image with the pinhole
|
|
equations above. The GPU then finds every pixel whose center falls inside
|
|
the projected triangle, and runs a small program (a fragment shader) for
|
|
each of those pixels to compute its value. It is fast because it only
|
|
touches the pixels each triangle covers, and GPUs have dedicated hardware
|
|
for it. It can only draw triangles.
|
|
- **Ray casting.** The reverse direction: for each pixel, follow the line
|
|
of sight from the camera through the pixel center and solve for where it
|
|
first meets a surface. For a sphere or an ellipsoid that is a quadratic
|
|
equation with an exact answer, so there are no triangles and no facets.
|
|
Every pixel does the work, whether or not it ends up seeing the body.
|
|
|
|
The bodies use ray casting, because their shapes have exact equations and
|
|
their edges are what limb-based navigation measures. The target uses
|
|
rasterization, because a spacecraft model has no equation, only its mesh.
|
|
The meshes are drawn first, with their depth. The bodies pass then casts its
|
|
rays and compares depths at every pixel, so the two sort against each other
|
|
correctly: a target in front of Earth covers it, and a target behind the limb
|
|
is hidden. Ray casting a mesh is also possible, by testing rays against
|
|
its triangles. A ray tracer, planned, does that, because following rays
|
|
gives exact shadows and light bouncing between surfaces. Rasterization can
|
|
only approximate those, for example with the shadow maps lighting L1 uses
|
|
(below).
|
|
|
|
### Pixel area
|
|
|
|
At camera tier C0, both techniques decide each pixel's value from one
|
|
point, the pixel's center. A real pixel collects light over its whole area,
|
|
about 5.5 um square on the detector. Where a pixel straddles an edge, say
|
|
half on a sunlit panel and half on dark space, the real pixel reads about
|
|
half the panel's value. At C0 the renderer reads either the full value or
|
|
none, depending on which side the center falls.
|
|
|
|
This shows in two ways. Edges are stair-stepped instead of graded. And
|
|
anything thinner than a pixel, like the booms of the example satellite seen
|
|
from 40 m, breaks into dashes, because only some pixel centers land on it.
|
|
A regular pattern finer than the pixels, like the cells of a solar array,
|
|
beats against the pixel grid and shows as curved bands (moiré). For an
|
|
algorithm that finds edges or measures brightness at sub-pixel precision,
|
|
the graded edge is the signal, so this matters for limb navigation and
|
|
pose estimation.
|
|
|
|
Camera tier C1 samples each pixel at several points and averages them
|
|
(supersampling), in a way that costs little where nothing is crossing the
|
|
pixel:
|
|
|
|
- **Meshes** use the GPU's multisampling. The rasterizer tests each
|
|
triangle against 4 points in every pixel, and keeps a depth for each, but
|
|
runs the fragment shader once per pixel. A pixel half covered by a panel
|
|
then gets the panel's value in 2 of its 4 samples, and reads half. The
|
|
covered fraction comes in steps of a quarter, and the light is computed
|
|
at one point inside the covered part (the centroid of the covered
|
|
samples), so a pixel on an edge is still shaded as one surface.
|
|
- **Bodies** are ray cast, so the bodies pass first decides whether a body's
|
|
edge could cross the pixel: whether the line of sight through its center
|
|
passes within one pixel's angle of the edge of Earth, the Moon or the Sun.
|
|
Most pixels do not, and cast one ray as at C0. A pixel on an edge casts a
|
|
grid of 4 x 4 rays spread over its area and averages them. A body only a
|
|
few pixels across needs a finer grid, because its brightness falls off
|
|
toward its edge and a coarse grid misjudges that: a 4 x 4 grid gets a
|
|
full Moon 0.45 pixels wide 45 % wrong in total light. So the grid has at
|
|
least 8 rays across the body's radius, up to 64 x 64 for one smaller than
|
|
a pixel, which keeps the total within 1 %. Only a handful of pixels pay
|
|
for that.
|
|
- **Both together.** Where a target and a body share a pixel, each of the
|
|
grid's rays takes the mesh sample nearest to it, and keeps the target if
|
|
that sample is nearer than the body along the ray. So a target seen just
|
|
past Earth's limb takes its share of the pixel, and Earth keeps the rest.
|
|
|
|
A grid of 4 x 4 is still a coarse estimate of an area. For an edge nearly
|
|
aligned with the pixel grid, its 4 columns of rays resolve the covered
|
|
fraction in quarters, so a pixel on the limb can be off by up to an eighth
|
|
of the difference across the edge (measured 0.10 on Earth's limb from low
|
|
orbit, against up to a half at C0). Patterns that stagger the rays fix
|
|
aligned edges but do worse at other angles; the regular grid had the
|
|
smallest error on average over all angles. At C1 the point spread function
|
|
(below) then spreads every edge over a few pixels, and C1 also renders each
|
|
pixel as 2 x 2 finer ones by default, each sampled this way, which divides
|
|
these errors further.
|
|
|
|
At one rendered pixel per image pixel, C1 costs 0.4 to 0.5 ms more per
|
|
1920 x 1080 frame on an RTX 3060, and 1.2 to 1.4 ms at 3840 x 2160, measured on Landsat 8, ICESat-2 and the moonrise
|
|
scene. On the CPU through lavapipe it costs much more, 2.6 to 8 times the
|
|
mesh pass at C0 (ICESat-2's 752,000 triangles go from 74 to 580 ms per
|
|
frame), because lavapipe appears to run the fragment shader for every
|
|
sample rather than every pixel. The 4 samples of radiance and depth also
|
|
take 80 bytes per pixel of GPU memory: 166 MB at 1920 x 1080 and 660 MB at
|
|
3840 x 2160.
|
|
|
|
## Blur: the point spread function
|
|
|
|
Code: `camera.rs` (`Psf`); `gpu/blur.rs`, `shaders/blur.wgsl`.
|
|
|
|
No camera focuses a point to a point. Light from one point of the scene
|
|
lands on the detector as a small spot, and the shape of that spot is the
|
|
point spread function (PSF). Several things widen it:
|
|
|
|
- **Diffraction.** Light passing through a round aperture spreads into a
|
|
pattern of rings (the Airy pattern) whose size depends only on the
|
|
wavelength and the f-number. At f/4 and 550 nm the first dark ring is
|
|
2.7 um from the center, half a pixel of the example camera.
|
|
- **Aberrations and defocus.** Real lenses bring different parts of the
|
|
aperture, different wavelengths and different parts of the field to
|
|
slightly different points.
|
|
- **The detector.** Electrons freed deep in a pixel can drift into its
|
|
neighbors before they are collected (charge diffusion).
|
|
- **Jitter.** The camera shakes a little during the exposure.
|
|
|
|
Each of these blurs the image of the one before it, and many such blurs
|
|
together come out close to a Gaussian, a bell curve in 2D. So a common model
|
|
of a camera's PSF is a Gaussian with one width, its standard deviation
|
|
sigma, measured in pixels. A spot with sigma 0.5 px is 1.2 px wide at half
|
|
its peak, and a point centered on a pixel puts about 47 % of its light in
|
|
that pixel and the rest in its neighbors; at 0.7 px, 28 %. Longer
|
|
wavelengths diffract more, so sigma is set per channel and usually grows
|
|
toward the red.
|
|
|
|
How wide the spot should be depends on the camera. Published measurements,
|
|
converted to a Gaussian's sigma with the pixel's own area taken out:
|
|
|
|
- Earth-observation imagers are built sharp: about 0.2 px for Landsat 8's
|
|
OLI, the sharpest, and 0.35 to 0.55 px for WorldView-3, Pléiades Neo and
|
|
Sentinel-2's specification. The Lunar Reconnaissance Orbiter's cameras are
|
|
specified in the same range.
|
|
- Navigation and small-satellite cameras are softer: about 0.6 to 0.9 px
|
|
for DART's DRACO camera in flight, and 0.55 to 0.7 px for Satellogic's
|
|
imagers.
|
|
- Star trackers are defocused so each star covers several pixels. The one
|
|
measured in flight here (on Lunar Flashlight) has a sharp core and wide
|
|
wings that no single Gaussian matches: its spot is as wide at half its
|
|
peak as a 0.4 px Gaussian, and spreads its light as widely as a 1.25 px
|
|
one.
|
|
|
|
The imagers' numbers are for the whole system, including the atmosphere
|
|
and image processing, so the optics alone are somewhat sharper. The example
|
|
camera uses 0.5 px in every channel, a sharp camera, not derived from its
|
|
optics; diffraction alone at f/4 would give about 0.17 px.
|
|
|
|
Without blur, a star or a distant spacecraft lands in one pixel or in none,
|
|
and every edge is as sharp as the pixel grid allows. Algorithms that find
|
|
the center of a star or a target to a fraction of a pixel (centroiding)
|
|
need the spot spread over more than one pixel, so they cannot be tested on
|
|
an unblurred image.
|
|
|
|
The blur is applied to radiance on the focal plane, before light becomes
|
|
electrons. That order matters for anything bright enough to saturate. The
|
|
Sun's image is tens of thousands of times brighter than a full well.
|
|
Blurred first, its spot spreads that light into the neighboring pixels,
|
|
which also fill, so the saturated region is wider than the Sun's disk, as
|
|
in a real camera. Clipped first and then blurred, the saturated region
|
|
would shrink back inside the disk and its edge would fade over a pixel or
|
|
two, as if the Sun were barely brighter than the full well. The blur's
|
|
weights sum to 1, so it moves light without adding or removing any.
|
|
|
|
### Blurring before the pixels average
|
|
|
|
The blur has to act on the light where it lands, before each pixel averages
|
|
it. Blurring an image of pixel averages afterward is not the same once the
|
|
spot is about as small as a pixel: a point of light that fell anywhere in a
|
|
pixel has become that pixel's value, and where in the pixel it fell is
|
|
gone. Its blurred spot should lean toward wherever it was, and no blur of
|
|
the averages can tell which way. Measured with sigma 0.4 px, points of
|
|
light blurred after averaging put up to 51 % of their light in the wrong
|
|
pixel, and edges were off by up to 12 % of their step.
|
|
|
|
So from camera tier C1 the renderer draws the scene finer than the pixels,
|
|
n x n rendered pixels to each image pixel, blurs that by a Gaussian PSF set
|
|
in `CameraConfig::psf`, and only then averages each n x n block into one
|
|
pixel. Each rendered pixel still collects light over its own area the way
|
|
the previous section describes, so the finer image is itself made of
|
|
small-pixel averages, and the same error remains at a smaller scale:
|
|
|
|
| Rendered pixels per pixel | Points, worst pixel | Edges, worst pixel |
|
|
|---|---|---|
|
|
| 1 x 1 | 51 % of the light | 12 % of the step |
|
|
| 2 x 2 | 19 % | 2.2 % |
|
|
| 4 x 4 | 4.5 % | 0.5 % |
|
|
|
|
(sigma 0.4 px, points at 16 and edges at 129 positions within a pixel; the
|
|
worst edges lie on a pixel boundary.) C1 renders 2 x 2 by default;
|
|
`RendererOptions::supersampling` sets 1 to 4. Point sources, stars in
|
|
particular, need better than that, and will be drawn with the blurred spot
|
|
computed exactly over each pixel instead.
|
|
|
|
A 2D Gaussian is the product of a 1D Gaussian across and one down, and
|
|
averaging an n x n block is likewise an average across and one down. So the
|
|
blur runs as two passes, one along rows and one along columns, each taking
|
|
a weighted sum of the rendered pixels out to 4 sigma on either side
|
|
(beyond which a Gaussian holds less than 1e-4 of its light) and averaging
|
|
n of them into one pixel at once. Light from just outside the frame also
|
|
blurs into it, so the renderer draws a margin of that width around the
|
|
image, and the blur reads it and drops it. The weights are the Gaussian at
|
|
each rendered pixel's offset, which keeps a spot's width (its variance)
|
|
within 1 % of sigma squared once sigma is 0.65 rendered pixels or more:
|
|
0.33 px in image pixels at 2 x 2.
|
|
|
|
Rendering finer costs about n^2 times the drawing, and the blur grows with
|
|
it. To keep that down, the meshes are covered at 2 points per rendered pixel
|
|
rather than 4 when rendering finer: at 2 x 2 that still puts samples at 4
|
|
distinct offsets across and down each image pixel, as 4 samples did at
|
|
1 x 1, so mesh edges still grade in quarters or better. (A device that
|
|
cannot render 2 samples, such as lavapipe, uses 4.) GPU time per frame at
|
|
C1, from the first pass to the last, on an RTX 3060 (Landsat 8, ICESat-2
|
|
and the moonrise scene, at lighting L1, sigma 0.5 px):
|
|
|
|
| Rendered pixels per pixel | 1920 x 1080 | 3840 x 2160 |
|
|
|---|---|---|
|
|
| 1 x 1 | 1.9 to 2.5 ms | 6.1 to 6.9 ms |
|
|
| 2 x 2 (default) | 3.7 to 4.4 ms | 12.6 to 14.6 ms |
|
|
| 3 x 3 | 6.7 to 7.9 ms | 34 to 38 ms |
|
|
| 4 x 4 | 10.9 to 13.0 ms | 59 to 66 ms |
|
|
|
|
Of the 3.7 to 4.4 ms at 2 x 2, the bodies take 1.3 to 1.4 ms, the meshes
|
|
0.7 to 1.1 ms and the blur 1.0 ms. With 2 samples instead of 4 the meshes
|
|
and bodies took 0.3 to 0.6 ms less in all; with 1 sample, 0.7 to 1.0 ms
|
|
less, but a mesh edge then grades only in halves at 2 x 2, coarser than at
|
|
1 x 1. The rendered images take 60 bytes per rendered pixel (100 at 1 x 1,
|
|
with 4 samples), and the blur 16 bytes per image pixel per rendered row:
|
|
about 0.27, 0.60, 1.25 and 2.15 GB at 1920 x 1080 for 1 x 1 to 4 x 4, and
|
|
four times that at 3840 x 2160. The renderer reports running out of memory
|
|
as an error. On the CPU through lavapipe, 2 x 2 adds 9 to 13 ms to
|
|
1920 x 1080 frames of 116 to 680 ms, and 3 x 3 at 3840 x 2160 runs out of
|
|
memory.
|
|
|
|
## Earth, Moon and Sun
|
|
|
|
Code: `bodies.rs`, `gpu/bodies.rs`, `shaders/bodies.wgsl`.
|
|
|
|
The three bodies are not drawn as triangle meshes; they are ray cast. For
|
|
every pixel, the shader sends a ray from the camera through the pixel's
|
|
center and solves for where it meets each body's exact shape. The nearest hit wins. This
|
|
gives an exact edge at any zoom, at the cost of a few arithmetic operations
|
|
per pixel. A mesh would show its facets along the limb, the edge of the
|
|
disk, which is the feature horizon-based navigation measures.
|
|
|
|
- **Earth** is the WGS84 ellipsoid: 6378.137 km equatorial radius,
|
|
6356.752 km polar. The 21 km difference matters at fine pixel scales. A
|
|
camera with 5.5 urad pixels at 200,000 km sees the north limb 19 pixels
|
|
away from where a spherical Earth would put it. A test checks the limb
|
|
against the exact tangent to the ellipsoid to within a pixel.
|
|
- **The Moon** is a sphere of 1737.4 km radius. From Earth's distance it
|
|
is about 0.52 degrees across, 82 pixels in the example camera.
|
|
- **The Sun** is a sphere of 695,700 km radius: 0.53 degrees across, about
|
|
85 pixels in the example camera. Averaged over its disk it has the solar
|
|
radiance from the folding section above; from lighting L1 its center is
|
|
brighter than its edge (see "The Sun's disk" below). It is drawn only
|
|
where no nearer body covers it, so Earth eclipses it naturally.
|
|
|
|
### Shading
|
|
|
|
Earth and the Moon are Lambertian, at lighting L0 each with one reflectance
|
|
spectrum over the whole body, set by `Appearance`. The Moon's is a flat 0.12
|
|
(from L1 it comes from a map; see "The Moon's surface"). Earth's
|
|
default is a stand-in shaped like cloud-free ocean seen from space: about
|
|
0.3 in the blue falling to 0.07 in the red, then back up in the near
|
|
infrared. Most of that blue is sunlight scattered by the atmosphere above
|
|
the ocean, the same scattering that makes the sky blue; the ocean itself is
|
|
dark. In the example camera it gives channel albedos of 0.09, 0.15 and 0.26
|
|
(red, green, blue), so Earth renders blue. Its overall brightness is below
|
|
the real Earth's average of about 0.3, which includes bright clouds.
|
|
|
|
Each point's radiance is rho cos(i) S_k / (pi d^2), with i the angle between
|
|
the surface normal and the Sun. The night side has i past 90 degrees and is
|
|
black. The Sun's distance d is taken from each body's center, since the
|
|
difference across a body is a few parts in 10^5.
|
|
|
|
The total light a sphere sends a far-away camera has a closed form. A
|
|
Lambertian sphere of albedo rho and radius R, at distance D, delivers an
|
|
irradiance of (2/3) rho E (R/D)^2 Phi(alpha), where alpha is the phase angle
|
|
(Sun-sphere-camera) and Phi(alpha) = (sin alpha + (pi - alpha) cos alpha) / pi.
|
|
It falls from 1 at full phase to 0.61 at 60 degrees and 0.11 at 120 degrees.
|
|
Up close it is not exact: the near side of the sphere is nearer the camera
|
|
than its center, so it counts for more. At 40,000 km from the Moon (R/D =
|
|
0.043), the real total is 3 % above the formula. The renderer's image agrees
|
|
with the exact surface integral to 0.03 % or better (`tests/gpu/analytic.rs`).
|
|
|
|
This gets the overall brightness and the phase (how much of the disk is
|
|
lit) right. It misses some things that are visible in real images:
|
|
|
|
- Earth's oceans, land and clouds differ in brightness by a factor of
|
|
several. From lighting L1, Earth's surface comes from an image (next
|
|
section); clouds are not modeled yet.
|
|
- The full Moon looks almost uniformly bright out to its edge. A Lambertian
|
|
sphere instead darkens toward the edge, because the light there arrives
|
|
and leaves at grazing angles. Lunar soil scatters light back toward its
|
|
source much more strongly than Lambert predicts, which is also why it
|
|
brightens so steeply toward full Moon. At lighting L1 the Moon's total
|
|
brightness is scaled to its measured phase curve ("The Moon's surface"),
|
|
but its edge still darkens; a dedicated lunar reflectance model would fix
|
|
that. The Moon's maria (the dark plains of
|
|
old basalt lava, the "seas" seen with the naked eye) and its bright
|
|
highlands also differ in albedo by about a factor of two, which from
|
|
lighting L1 the Moon's map shows.
|
|
- Earth's atmosphere softens the limb and adds a thin bright band around it
|
|
(not modeled yet). Here Earth's edge is a sharp line.
|
|
|
|
### Earth's surface
|
|
|
|
Code: `earth.rs`, `shaders/bodies.wgsl`; data from `scripts/fetch-earth.sh`.
|
|
|
|
At lighting tier L1 (`RendererOptions::fidelity`, with the image in
|
|
`RendererOptions::earth_texture`; `--lighting L1` on the command line), Earth's
|
|
reflectance comes from an image of the whole surface instead of one spectrum. The image is NASA's
|
|
Blue Marble Next Generation: a cloud-free mosaic of the 2004 MODIS satellite
|
|
record, one image per month, so snow cover and vegetation follow the season.
|
|
The CLI picks the month from the scene's epoch.
|
|
|
|
The image is equirectangular: each column is a longitude and each row a
|
|
latitude. For every pixel that sees Earth, the shader finds the surface
|
|
point's geodetic latitude and longitude from the ellipsoid's normal there, in
|
|
Earth-fixed axes, and reads the image at that place. Earth's orientation from
|
|
the scene turns the continents with the epoch.
|
|
|
|
The full image is 500 m per pixel at the equator, resized to 660 m to fit the
|
|
largest texture lavapipe allows (16384 x 8192). Whether that is fine or coarse
|
|
depends on the view:
|
|
|
|
- From low orbit looking straight down, a camera pixel of the example camera
|
|
covers 77 m of ground, so each texture pixel spans about 8 image pixels and
|
|
looks soft.
|
|
- From geostationary orbit, a camera pixel covers about 4 km, so each image
|
|
pixel averages about 35 texture pixels (6 x 6).
|
|
|
|
For the second case the texture is stored with mipmaps: copies at half, a
|
|
quarter, an eighth of the size, and so on. The GPU reads from the copy whose
|
|
pixels best match the image pixel, so a far view shows the true average color
|
|
of a region rather than a shimmering sample of individual texture pixels.
|
|
|
|
Two details matter for brightness:
|
|
|
|
- Averaging has to happen on reflectance, not on the encoded values image files
|
|
store (sRGB, which compresses the darks). Averaging encoded black and white
|
|
gives a reflectance of 21 %, not 50 %. So the texture is decoded to linear
|
|
reflectance once at load and kept in 16-bit floating point, and both the
|
|
mipmaps and the GPU's blending between neighboring texture pixels work on
|
|
those values. A coastline seen up close blends correctly from sea to land.
|
|
- Blue Marble is surface reflectance with no atmosphere, and open ocean is
|
|
dark, a few percent. From orbit the ocean looks bright blue because the air
|
|
above it scatters sunlight toward the camera: Rayleigh scattering, which
|
|
grows as 1 / wavelength^4, which is also why the sky is blue. Until the
|
|
atmosphere is modeled, a stand-in adds a reflectance of that shape,
|
|
0.07 at 450 nm falling to 0.017 at 650 nm (`Appearance::earth_haze`). It
|
|
brightens and blues the whole lit disk evenly, where the real effect grows
|
|
toward the limb, where the line of sight crosses more air.
|
|
|
|
The texture's red, green and blue are reflectances in the same three bands a
|
|
glTF base color uses (see the target model section), folded into the camera's
|
|
channels once at setup as a small matrix, so any channel layout works.
|
|
|
|
Blue Marble is a visual product built from calibrated MODIS reflectances, not a
|
|
calibrated reflectance product itself, so its values are close but not exact
|
|
reflectances. A calibrated alternative (MODIS's albedo product) is a possible
|
|
later refinement.
|
|
|
|
### The Moon's surface
|
|
|
|
Code: `moon.rs`, `shaders/bodies.wgsl`; data from `scripts/fetch-moon.py`. From
|
|
lighting L1, with the map in `RendererOptions::moon_texture`.
|
|
|
|
At lighting L1 the Moon's reflectance comes from a map instead of one spectrum.
|
|
The map is the LRO Camera's Wide Angle Camera mosaic at 643 nm (red), built from
|
|
more than 100,000 images taken under every kind of lighting. The camera team
|
|
used a model of how each spot on the Moon reflects light at different angles
|
|
(Hapke's, fitted to the images; Sato et al. 2014) to work out what every spot
|
|
would look like under one standard geometry: the Sun 60 degrees from overhead,
|
|
seen straight down. The map's values are that reflectance, I/F, the ratio of
|
|
the surface's brightness to that of a perfect white surface facing the Sun.
|
|
Over 98 % of the Moon they lie between 0.019 in dark maria and 0.067 on the
|
|
brightest highlands and crater rays, and they average 0.042.
|
|
|
|
The map is 400 m per pixel and covers 70 degrees south to 70 degrees north;
|
|
the polar caps beyond are filled with the average, so they render as smooth
|
|
gray (6 % of the surface, seen only near the limb from Earth). It is resampled
|
|
to 16384 x 8192, 666 m per pixel, the largest texture lavapipe takes, and read
|
|
exactly as Earth's image is: by latitude and longitude in the Moon's own axes,
|
|
with mipmaps, and without a seam at 180 degrees.
|
|
|
|
The Moon's axes come from `Scene::moon_attitude`. The Moon keeps one face
|
|
toward Earth, but rocks back and forth by up to 8 degrees (libration), so
|
|
which features are near the limb changes from month to month. With an epoch,
|
|
the renderer uses the IAU's model of the Moon's rotation; it puts the point
|
|
facing Earth within 0.008 degrees (230 m) of JPL Horizons' high-precision
|
|
frame, under a third of a map pixel. Without one, `Scene::moon_facing_earth`
|
|
turns the Moon's prime meridian toward Earth and is off by the libration, up
|
|
to about 10 degrees.
|
|
|
|
Two choices turn the map into something the renderer can shade:
|
|
|
|
- **Brightness.** The renderer shades the Moon as Lambertian, starting
|
|
from the albedo that gives the map's reflectance at the map's geometry:
|
|
I/F / cos(60 degrees), on average 0.084. On its own that makes the Moon
|
|
far too dark at full: the real Moon is not Lambertian, and it brightens
|
|
steeply toward full Moon, as rough soil hides its own shadows when lit
|
|
from behind the viewer. A Lambert sphere with these albedos is about a
|
|
third as bright as the real full Moon, and slightly too bright past
|
|
quarter. So each point's albedo is multiplied by a phase factor, from the
|
|
phase angle between the Sun and the viewer seen from that point. The
|
|
factor is the whole Moon's measured brightness at that phase (Allen's
|
|
lunar phase curve, 0.026 g + 4e-9 g^4 magnitudes below full, g in
|
|
degrees) over a Lambert sphere's, scaled so that the map's full Moon has
|
|
the measured geometric albedo (the full Moon's brightness against a white
|
|
Lambertian disk of the same size), 0.121 in the V band. It is 2.8 at 4
|
|
degrees, 1.16 at 60, 0.89 at 90 and 0.75 at 120, and holds its 150 degree
|
|
value beyond. At 60 degrees it would be 1 if the map and the measured
|
|
curve agreed exactly; the 16 % between them is the Lambert shape's error
|
|
at that phase together with the two calibrations'.
|
|
|
|
Rendered whole in the example camera's green channel, the Moon is then
|
|
within 5 % of the measured curve from 2 to 30 degrees of phase, and
|
|
within 11 % out to 120 degrees. At the same phase, the side of the Moon
|
|
holding Oceanus Procellarum, the largest of the maria, is darker than the
|
|
other: 0.89 of the curve at 90 degrees against 1.06. The real Moon differs
|
|
between waxing and waning in the same way, which the curve averages out.
|
|
At 150 degrees it is 9-15 % bright.
|
|
|
|
Across the disk the light still falls off as Lambert's does, darkening
|
|
toward the edge, while the real full Moon looks nearly flat out to its
|
|
limb. A lunar reflectance model would fix that.
|
|
- **Color.** The map is one band. Its color comes from the Moon's average
|
|
spectrum across the WAC's seven bands, 321 to 689 nm, from the same mosaics
|
|
averaged over the whole Moon by area: relative to 643 nm, 0.42 at 321 nm, 0.61
|
|
at 415 nm, 0.87 at 566 nm and 1.08 at 689 nm. The Moon is distinctly red,
|
|
which is why it looks yellowish gray to the eye. Each pixel's
|
|
643 nm value scales that spectrum, which is folded into the camera's
|
|
channels once at setup: in the example camera the Moon's albedo in red,
|
|
green and blue is 0.96, 0.83 and 0.69 times its 643 nm value over cos 60.
|
|
Beyond 689 nm the spectrum continues the slope from 643 to 689 nm to 900 nm
|
|
(1.45) and holds there, roughly following real lunar soils, which flatten
|
|
into an absorption by iron-bearing minerals near 1000 nm; below 321 nm it
|
|
holds the 321 nm value. Real maria rich in titanium are up to about 10 %
|
|
bluer than this, and highlands slightly redder; the seven-band mosaic
|
|
could give each pixel its own color later. The real Moon also reddens
|
|
slightly with phase, which the phase factor leaves out.
|
|
|
|
Light the Moon sheds on targets (Moonshine) reads the same map, with the
|
|
phase factor for the angle between the Sun and the target.
|
|
|
|
Tests (`tests/gpu/moon_texture.rs`), with small synthetic maps, on the GPU and
|
|
lavapipe: the pixel under the camera reads the texel at its latitude and
|
|
longitude, and moves when the Moon's attitude does; the color is the spectrum's;
|
|
a uniform map draws exactly the Moon a flat albedo draws times the phase
|
|
factor, at every pixel; there
|
|
is no seam at 180 degrees; a far view averages a fine checker; Moonshine comes
|
|
from the part of the map that faces the target.
|
|
|
|
Cost: the map adds about 0.04 ms to the bodies pass on the RTX 3060, for working
|
|
out where to read it at every pixel (the read itself happens only where the
|
|
Moon is). On lavapipe the pass got faster, from 2.7 to 2.0 ms with the Moon
|
|
filling the view, because Earth's image is now also read only where Earth is.
|
|
Loading the map takes 0.5 s and uploading it 0.17 s; it occupies 360 MB on the
|
|
GPU.
|
|
|
|
### The Sun's disk
|
|
|
|
Code: `sun.rs`, `shaders/common.wgsl`, `shaders/bodies.wgsl`. From lighting
|
|
L1.
|
|
|
|
The Sun's disk is brighter at its center than at its edge: limb darkening.
|
|
Light leaving the Sun comes from a layer a few hundred kilometers deep in
|
|
which the gas gets hotter with depth. Looking at the center of the disk, the
|
|
line of sight goes straight down into hot gas; near the edge it runs through
|
|
the layer at a slant and sees higher, cooler gas, which shines less.
|
|
|
|
Where the line of sight meets the Sun's surface, call mu the cosine of the
|
|
angle between it and the surface's normal: 1 at the center of the disk, 0 at
|
|
the edge, 0.5 at 87 % of the way out. Neckel and Labs (1994) measured the
|
|
brightness across the disk at 30 wavelengths from 303 to 1099 nm, and
|
|
Hestroffer and Magnan (1998) showed that at each wavelength it is close to
|
|
mu^alpha times the center's, to about 1 %. Alpha is 0.94 at 303 nm, 0.48 at
|
|
580 nm and 0.26 at 1099 nm: blue light darkens much more toward the edge
|
|
than red. The renderer uses their values of alpha, interpolated between those
|
|
wavelengths and held constant outside them.
|
|
|
|
The total light from the disk does not change; that is the measured solar
|
|
spectrum. Only its spread does. A disk of brightness mu^alpha averages 2 /
|
|
(alpha + 2) of its center, so the center is (alpha + 2) / 2 times the mean.
|
|
Folded through the example camera's channels, the center is 1.22, 1.26 and
|
|
1.31 times the mean in red, green and blue, and 87 % of the way to the edge
|
|
(mu = 0.5) the disk is 0.70 of its center's brightness in green. The
|
|
folding is exact: for each channel the renderer weights every wavelength's
|
|
profile by that channel's solar flux and tabulates the result, so any channel
|
|
layout works.
|
|
|
|
It matters in two places:
|
|
|
|
- **The Sun in an image.** A camera that looks at the Sun almost always
|
|
saturates, but not with a dark filter, a Sun sensor, or once the lens's
|
|
blur, once modeled, spreads the Sun's light into the pixels around it.
|
|
- **Eclipses.** When Earth or the Moon covers part of the Sun, what counts is
|
|
how much light it hides, not how much of the disk's area. Near the umbra
|
|
only a sliver of the dim edge is left, so less light gets through than the
|
|
area says: 7000 km behind Earth, 25 km inside the penumbra's dark side,
|
|
4.5 % in green against 6.1 %. Near the outside of the penumbra the covered
|
|
part is dim edge, so more gets through: 88.2 % against 86.0 %. In an
|
|
annular solar eclipse the Moon covers the bright center and leaves the dim
|
|
ring: 1 % of the light reaches the ground, not the 2.4 % of the disk's area.
|
|
|
|
The renderer computes the light hidden by an occulter in two parts. Circles
|
|
around the Sun's center that the occulter covers completely count through a
|
|
table of the light inside each radius. Circles it covers in part count by
|
|
adding up, over eight radii, each circle's brightness times the length of its
|
|
arc inside the occulter, with the radii placed by a Gauss-Legendre rule; eight radii are within 2.4e-5
|
|
of what 32 give. The same
|
|
calculation runs on the CPU for each target's eclipse and on the GPU for
|
|
every pixel of Earth and the Moon in an eclipse, per channel.
|
|
|
|
Tests: every pixel of the Sun seen through a 250 mm lens (420 pixels across)
|
|
reads its mean radiance times the profile at its mu, to 6e-5, and the disk's
|
|
light summed over pixels is the channel's solar flux to 7e-5. The light
|
|
hidden by an occulter matches a fine sum over the disk to within 1e-4, and a
|
|
uniform disk gives the area overlap. In eclipses the tests of the sections below use
|
|
the limb-darkened light.
|
|
|
|
Cost: nothing measurable on the RTX 3060 unless an eclipse is in view. With
|
|
Earth filling the image and most of it in the Moon's penumbra, the bodies
|
|
pass takes 0.43 ms instead of 0.21 ms; lavapipe 9.5 ms instead of 5.0 ms.
|
|
|
|
Limitations:
|
|
|
|
- **The very edge.** Hestroffer and Magnan fitted the profiles over the
|
|
inner 90 % of the radius, where measurements are reliable. Right at the
|
|
edge mu^alpha falls to zero, faster than the measured Sun does, over the
|
|
outermost fraction of a percent of the radius.
|
|
- **Wavelengths outside 303 to 1099 nm** use the end values. The example
|
|
camera's channels are inside that range.
|
|
- **A quiet Sun.** Sunspots and bright faculae change the Sun's output by
|
|
about 0.1 % over the solar cycle and are not modeled.
|
|
- **Shadows on the target** (above) still treat the Sun as a square of
|
|
uniform brightness in their penumbra.
|
|
|
|
### Depth from the bodies
|
|
|
|
The pass also writes each hit's depth, near / z, as described in the
|
|
geometry section. The target spacecraft, drawn after the bodies, uses the
|
|
same depth buffer. So a target in front of Earth covers it, and a target
|
|
behind Earth's limb is hidden.
|
|
|
|
## The target model and its materials
|
|
|
|
Code: `model.rs`.
|
|
|
|
The target spacecraft is a triangle mesh loaded from STL or glTF. Each
|
|
piece (an STL file, or a glTF primitive) is a part with one material, and
|
|
for now a material is a Lambertian albedo spectrum, as for Earth and the
|
|
Moon.
|
|
|
|
Surface normals decide how bright each triangle is, through the cos(i)
|
|
factor. STL files store a normal per triangle, but exporters often write
|
|
zeros or wrong values, so the loader ignores them. It recomputes each
|
|
normal from the order of the triangle's corners: counterclockwise as seen
|
|
from outside. Every triangle is then lit as a flat facet, which matches the
|
|
flat panels and boxes most spacecraft are built from. glTF files that store
|
|
normals keep them.
|
|
|
|
glTF describes a material's color as red, green and blue reflectances (its
|
|
"base color"). Three numbers do not define a spectrum: very different
|
|
spectra can look the same color to the eye. The loader turns them into a
|
|
stepped spectrum, blue below 490 nm, green from 500 to 580 nm, and red above
|
|
590 nm. That is a reasonable guess for painted surfaces seen by an RGB
|
|
camera. It is a poor one for a camera whose channels differ from the eye's,
|
|
or for anything in the near infrared, where the red value simply continues.
|
|
For absolute radiometry of a real spacecraft, give the important materials
|
|
measured reflectance spectra (not supported yet).
|
|
|
|
Texture images are not used yet; each material renders with its base color
|
|
value alone. The loader removes textures before reading a glTF file, so a
|
|
model still loads when its textures are in a format or file the loader cannot
|
|
read. A model that relies on textures for its colors (one material with a
|
|
painted image) renders in that material's flat base color, often gray.
|
|
|
|
STL has no units. CAD tools usually export millimeters, so loading a CAD
|
|
model usually needs `scale: 0.001`. glTF is meters by its specification, but
|
|
models in the wild do not always follow it: of the NASA example models,
|
|
Landsat 8 and ICESat-2 are in meters, while the SSL-1300 needs a scale of 0.1
|
|
and ACE about 0.005. Checking a model's size against the real spacecraft's is
|
|
worth the minute it takes.
|
|
|
|
## Drawing the target
|
|
|
|
Code: `gpu/mesh.rs`, `shaders/mesh.wgsl`.
|
|
|
|
The target's mesh is rasterized after the bodies, into the same radiance
|
|
and depth images. Each pixel the target covers is kept only if the target
|
|
is nearer than what is already there. That one rule handles a target in
|
|
front of Earth, a target behind Earth's limb, and parts of the target
|
|
hiding other parts.
|
|
|
|
Each point on the target is shaded like the bodies: albedo times cos(i)
|
|
times the sunlight that reaches it. A panel modeled with no thickness, such
|
|
as a solar array, has one surface but two sides. Whichever side faces the
|
|
camera is the side that is lit. A closed shape never shows its inside, so
|
|
this only matters for thin parts.
|
|
|
|
### Shadows on the target
|
|
|
|
Code: `shadow.rs`, `gpu/shadow.rs`, `shaders/mesh.wgsl`. From lighting L1.
|
|
|
|
A solar array shades the bus behind it, an antenna dish shades its feed,
|
|
and a second spacecraft close by can shade the first. Rasterization draws
|
|
each triangle on its own, so by itself it cannot know that something else
|
|
stands between a point and the Sun. The renderer finds out with a shadow
|
|
map.
|
|
|
|
- **Shadow map.** Before the camera's view is drawn, the target is drawn
|
|
once more as the Sun sees it, looking straight down the Sun's rays. That
|
|
drawing keeps only depth: for each texel (a pixel of the map), how far
|
|
along the Sun's direction the first surface is. The Sun is far enough
|
|
away that its rays are parallel across a spacecraft, so the view has no
|
|
perspective, and a map 4096 texels across covers the target's bounding
|
|
sphere. For Landsat 8, whose sphere is 16.6 m across, one texel is 4 mm.
|
|
- **Lookup.** When the camera's view shades a point, it finds that point
|
|
in the map. If the map holds a surface nearer the Sun than the point, the
|
|
point is in shadow.
|
|
|
|
Each target gets its own map. Another target is drawn into that map when it
|
|
is sunward of this one and close enough across the Sun's rays to shade it.
|
|
A target off to the side is left out, since it cannot cast a shadow here.
|
|
|
|
The Sun is a disk half a degree across, so a shadow edge is not sharp. A
|
|
point near the edge sees part of the disk past the blocker, the same
|
|
penumbra as in an eclipse. The width of the penumbra is the distance from
|
|
blocker to surface times the tangent of the Sun's angular diameter, 0.93 %
|
|
at 1 AU. A solar array 2 m above the bus casts an edge 19 mm wide. One 10 m
|
|
away casts an edge 9 cm wide. The renderer reproduces this with
|
|
percentage-closer soft shadows:
|
|
|
|
1. Search the map around the point for blockers, and average their depth.
|
|
That gives the blocker-to-surface distance, and from it the penumbra
|
|
width.
|
|
2. Compare the point's depth against a 6 x 6 grid of map texels spread over
|
|
that width, and take the fraction that are not blocked as the share of
|
|
the Sun the point sees.
|
|
|
|
Both steps use fixed grids rather than random samples, so the image has no
|
|
noise and the same scene always gives the same pixels.
|
|
|
|
A surface can wrongly shadow itself, because the map only stores its depth
|
|
once per texel. Where the surface is tilted to the Sun, depth changes
|
|
across a texel, and half of each texel looks like it is behind itself. The
|
|
result is a pattern of dark stripes called shadow acne. The renderer
|
|
prevents it three ways. The stored depth is pushed away from the Sun in
|
|
proportion to the surface's tilt. The lookup is made from a point 1.5
|
|
texels off the surface, toward the Sun. And a blocker has to be more than 2
|
|
texels nearer the Sun to count.
|
|
|
|
Tests (`tests/gpu/shadows.rs`) check, on the GPU and lavapipe:
|
|
|
|
- A plate under a 2 m square: the shadow reads zero direct light, and its
|
|
edge falls where the square's edge projects, to within one texel.
|
|
- The measured penumbra width is within 2.2 % of 0.93 % of the occluder's
|
|
height, at heights of 1, 2 and 3 m.
|
|
- No pixel darkens compared with L0 on a plate tilted 80 and 87 degrees
|
|
from the Sun, or on a faceted sphere.
|
|
- A target sunward of another shades it; one to the side does not.
|
|
|
|
On the RTX 3060, shadows cost 0.14 ms of GPU time per frame for Landsat 8
|
|
(85,000 triangles) and 0.34 ms for ICESat-2 (752,000), plus about 0.15 ms
|
|
in the camera view's lookups. Maps 2048 texels across save about 0.15 ms.
|
|
|
|
Limitations:
|
|
|
|
- **Penumbra shape.** The grid averages a square of the map, so brightness
|
|
rises in a straight line across the penumbra. A real disk-shaped Sun
|
|
rises along an S-shaped curve: the full width is the same, but the middle
|
|
80 % of the change is 0.69 of the width rather than 0.8. Limb darkening
|
|
(from L1) would steepen it a little more, and is not modeled here either.
|
|
- **Resolution.** A detail smaller than a texel casts no reliable shadow. A
|
|
gap narrower than about 2 texels between a blocker and the surface it
|
|
shades (a few mm on Landsat 8) lets light through, because of the acne
|
|
margins above.
|
|
- **One shadow lookup per pixel.** Shadows are looked up once per pixel, even
|
|
at camera C1, so the edge of a shadow inside a pixel is not graded by its
|
|
4 samples; the penumbra's own width usually grades it.
|
|
- **Bodies.** Earth and the Moon do not use shadow maps. Their shadows on
|
|
the target are the eclipse factor below, applied to the whole target.
|
|
Light from Earth and the Moon (below) is not shadowed at all.
|
|
|
|
### Eclipses
|
|
|
|
Code: `eclipse.rs`.
|
|
|
|
The Sun is a disk, not a point. So as a spacecraft passes into Earth's
|
|
shadow, the light fades over a few seconds instead of switching off:
|
|
|
|
- **Umbra.** Earth covers the whole solar disk. No direct sunlight.
|
|
- **Penumbra.** Earth covers part of the disk. 7000 km from Earth's
|
|
center, on the night side, the penumbra is about 65 km wide; a satellite
|
|
crosses it in under 10 seconds.
|
|
- **Annular.** A body small enough to sit inside the solar disk blocks a
|
|
ring-shaped bite. From orbit this happens with the Moon.
|
|
|
|
The renderer finds the visible fraction of the solar disk from the overlap
|
|
of two circles on the sky. The circles are the Sun's disk and the
|
|
occulter's, each with its apparent angular radius. It multiplies the
|
|
target's sunlight by that fraction. The model is the standard one for
|
|
satellites (Montenbruck and Gill), the same as astrokit's. A test checks it
|
|
against tracing lines from the target to thousands of points spread over
|
|
the solar disk.
|
|
|
|
From lighting L1 the fraction is of the Sun's light rather than its area,
|
|
per channel, because the disk is brighter at its center (see "The Sun's
|
|
disk"); that makes a penumbra darker toward the umbra and brighter toward
|
|
the outside than the area alone says.
|
|
|
|
The renderer counts both Earth and the Moon as occulters. The fraction is
|
|
computed once per target, at its center. The whole target
|
|
dims together, which is accurate unless the target is a sizable fraction of
|
|
the penumbra's width. Light reaching the target from Earth itself is the
|
|
next section but one.
|
|
|
|
### Eclipses on Earth and the Moon
|
|
|
|
Code: `shaders/common.wgsl` (`sun_visible`), `shaders/bodies.wgsl`. From
|
|
lighting L1.
|
|
|
|
Earth and the Moon also shadow each other. When the Moon passes through
|
|
Earth's shadow it is a lunar eclipse; when the Moon's shadow falls on Earth,
|
|
a solar eclipse. For every pixel that sees Earth or the Moon, the shader
|
|
works out the fraction of the Sun's light that reaches that point on the
|
|
surface past the other body, with the same overlap of circles and the same
|
|
limb-darkened disk as for the target, and multiplies the point's sunlight by
|
|
it, per channel.
|
|
|
|
The sizes involved:
|
|
|
|
- At the Moon's distance, Earth's umbra reaches 4600 km from the shadow's
|
|
axis and its penumbra 8200 km. The Moon, 3474 km across, fits inside the
|
|
umbra with room to spare, so a total lunar eclipse lasts over an hour.
|
|
- The Moon's umbra only just reaches Earth. When the Moon is near its
|
|
closest, the umbra touches down as a dark spot up to about 270 km across:
|
|
a total solar eclipse. When it is farther, the umbra ends in space, and
|
|
beyond its tip the Moon covers only the middle of the Sun's disk, leaving a
|
|
ring: an annular eclipse, where about 1 % of the Sun's light still arrives,
|
|
the dim edge of the disk. The
|
|
penumbra around either is about 7000 km across.
|
|
|
|
The overlap is computed in 32-bit floats, which is enough: a lunar surface
|
|
point is placed to about 20 m out of 384,000 km, 6e-8 radians against a
|
|
solar radius of 4.6e-3 radians seen from there. One formula needed care. The
|
|
area where the two circles overlap involves angles whose cosines are close
|
|
to 1 when a sliver of the Sun is covered, and the inverse cosine of a number
|
|
near 1 loses most of its precision in 32 bits. The shader finds those angles
|
|
from their sine and cosine together instead. With the inverse cosine,
|
|
pixels at the shadow's edges came out wrong: a point the 64-bit model puts
|
|
fully in the umbra read slightly lit, and one just outside the shadow
|
|
slightly dimmed.
|
|
|
|
Tests (`tests/gpu/eclipses.rs`), on the GPU and lavapipe, compare each pixel of
|
|
Earth and the Moon against the 64-bit model at that pixel's surface point:
|
|
the Moon entirely in Earth's umbra, the Moon across the umbra's edge, a
|
|
total solar eclipse and an annular one. Pixels the model puts fully in shadow
|
|
read zero, those it puts fully outside are unchanged (both to within 1e-6,
|
|
the 32-bit rounding at the shadow's edges), and the rest
|
|
agree to within 1e-4 of the Sun's light (1.2e-6 at most on the GPU, 5e-5 on
|
|
lavapipe). The command-line tests render two real eclipses from their
|
|
epochs: the total lunar eclipse of 2025 March 14 at mid-totality, where the
|
|
whole Moon reads the bias, and the total solar eclipse of 2026 August 12,
|
|
where the point of greatest eclipse in NASA's bulletin (65.2 N 25.2 W) is in
|
|
the umbra (`examples/scenes/lunar-eclipse.toml`, `solar-eclipse.toml`). In
|
|
the second, the umbra is 220 by 410 km in the image, stretched along the
|
|
direction of the Sun, which is 26 degrees high there; NASA gives the path as
|
|
294 km wide.
|
|
|
|
The light Earth and the Moon reflect onto targets (next section) includes
|
|
these shadows, so an eclipsed Moon sheds less light.
|
|
|
|
On the RTX 3060 this adds nothing measurable to the bodies pass unless an
|
|
eclipse is in view; with Earth filling the image and most of it in the
|
|
Moon's penumbra, the pass takes 0.43 ms instead of 0.20 ms, most of it the
|
|
limb-darkened sum (see "The Sun's disk"). On lavapipe, 9.5 ms instead of
|
|
3.9 ms.
|
|
|
|
Limitations:
|
|
|
|
- **Earth's atmosphere.** It bends sunlight into the umbra, mostly red, so a
|
|
real eclipsed Moon glows a dim copper color instead of going black, and it
|
|
makes the umbra about 2 % larger. Neither is modeled yet.
|
|
- **Smooth shapes.** Earth's shadow is cast by a sphere of its equatorial
|
|
radius, as for the target's eclipses; its flattening would move the
|
|
shadow's edge on the Moon by about 20 km. The Moon's mountains and valleys,
|
|
which shape the edge of a solar eclipse's path, are not modeled.
|
|
|
|
### Light from Earth and the Moon
|
|
|
|
Code: `gpu/earthshine.rs`, `shaders/earthshine.wgsl`, `shaders/mesh.wgsl`.
|
|
From lighting L1.
|
|
|
|
Earth reflects about a third of the sunlight that reaches it, and some of
|
|
that lands on the target: Earthshine. From low orbit it is large. At 500 km,
|
|
Earth fills a circle 68 degrees in radius around the point straight down,
|
|
3.9 of the sky's 12.6 steradians. A plate facing straight down over the
|
|
sunlit Earth receives up to about a quarter as much light from Earth as a
|
|
plate facing the Sun receives from the Sun. A white Earth would give 0.85
|
|
(the test below measures that case). At geostationary distance, Earth is 17
|
|
degrees across and a plate facing a full Earth gets about 0.5 % of sunlight.
|
|
The Moon does the same, Moonshine, much more weakly: a full Moon seen from
|
|
near Earth gives about a millionth of sunlight. From 100 km above the Moon,
|
|
though, a plate facing it gets about a tenth of sunlight.
|
|
|
|
This matters most where the Sun does not reach: the target's night side,
|
|
and parts in a shadow of its own. Without it those read as black, the bias in
|
|
DN; with it they show their structure, dimly.
|
|
|
|
The renderer works out, for each target and each frame, how much light
|
|
arrives from Earth and the Moon on a surface facing any direction:
|
|
|
|
1. **What the target sees of each body.** The circle on the sky that the body
|
|
fills is cut into 1024 cells of equal solid angle: 32 rings from the
|
|
center out to the edge, 32 cells around each. A line from the target
|
|
through each cell's center finds the point it meets on the body, and that
|
|
point is shaded exactly as the camera's view of the body is shaded:
|
|
reflectance from the Blue Marble image (or the stand-in spectrum), lit by
|
|
the cosine of the Sun's angle there, dark on the night side. Each texture
|
|
read averages the patch of ground the cell covers, about 1000 square
|
|
kilometers straight down from 500 km and more toward Earth's edge, so a
|
|
cell sees the mean color of that patch rather than one point. A line
|
|
that meets the other body first counts nothing, so the Moon can hide part
|
|
of Earth and Earth part of the Moon.
|
|
2. **Light for every direction a surface can face.** For a surface facing
|
|
direction n, the light it receives is the sum over all cells of the cell's
|
|
brightness, times its solid angle, times the cosine between n and the
|
|
cell's direction (zero for cells behind the surface). The renderer adds
|
|
this up for a grid of directions, a cube map 16 texels across on each of
|
|
its six faces.
|
|
|
|
Drawing the target, each point reads that cube at its surface normal and
|
|
adds albedo / pi times the result to the direct sunlight. The cube is computed
|
|
at the center of each target's bounding sphere and used for the whole target;
|
|
across a spacecraft the view of Earth barely changes.
|
|
|
|
Earthshine is not dimmed by the target's eclipse factor, because the two
|
|
are different views of Earth. From anywhere in Earth's umbra only the night
|
|
side is in view, so Earthshine is zero there without being told. In the
|
|
penumbra the Sun is partly hidden but a crescent of lit ground is still in
|
|
view, and that light still arrives.
|
|
|
|
Tests (`tests/gpu/earthshine.rs`) compare a plate's brightness on the GPU and
|
|
lavapipe against the same sum done in 64-bit floating point with 90 times as
|
|
many cells: over the subsolar point at 500 km facing down, sideways and up;
|
|
toward the terminator; in Earth's shadow; at GEO; 100 km above the Moon at
|
|
full and quarter phase; with each body hiding the other; with the plate seen
|
|
from several sides, so each cube face a camera-facing surface reads is checked. Every
|
|
case where the bodies send more than a thousandth of sunlight agrees to
|
|
within 0.5 %. The largest differences come from the cube: between texels
|
|
5.6 degrees apart its interpolation rounds off the peak of a surface facing
|
|
a full Earth or Moon, by 0.4 %. A cube 32 texels across cuts that to 0.13 %
|
|
but costs four times as much, and the cells' own error on a half-lit distant
|
|
Earth (0.4 %) stays. A thin crescent of lit ground at Earth's edge, seen from
|
|
low orbit near the terminator, is off by 4 % of a small amount, 7e-5 of
|
|
sunlight, about 0.1 DN on a gray surface in the example camera. Another test
|
|
checks the sum itself against the classical formula for a distant sphere,
|
|
(2/3) albedo (R/D)^2 Phi(alpha), to within 0.3 %.
|
|
|
|
On the RTX 3060 this costs 0.04 ms of GPU time per target (0.28 ms for
|
|
eight), and 1.4 ms per target on lavapipe.
|
|
|
|
Limitations:
|
|
|
|
- **No shadows.** The target's own parts do not block light from Earth or
|
|
the Moon. A panel facing Earth does not shade the bus behind it from
|
|
Earthshine, and the inside of a dish facing Earth is lit as if open to the
|
|
whole Earth. Earth fills too much of the sky for one shadow map to stand in
|
|
for it; a ray tracer would handle this.
|
|
- **One spectrum per channel.** Light from Earth is bluer than sunlight, but
|
|
the target's albedo in each channel was worked out under sunlight. This is
|
|
the shortcut described in "Folding spectra into per-channel constants",
|
|
and its error is largest for strongly colored surfaces.
|
|
- **Lambertian bodies.** Earth and the Moon reflect as Lambertian surfaces
|
|
here, as they do in the camera's view, with the Moon's map scaled by its
|
|
phase curve as seen from the target. Earth has no clouds or atmosphere yet
|
|
beyond the stand-in haze.
|
|
|
|
## Where the Sun and Moon appear
|
|
|
|
Code: `scene.rs` (`Velocities`, `aberrate`), `ephemeris.rs` (feature
|
|
`astrokit`), `shaders/bodies.wgsl` (`seen`).
|
|
|
|
With the `astrokit` feature, the Sun's and Moon's positions come from JPL's
|
|
DE440s ephemeris, and Earth's orientation from the ITRF93 kernel, which
|
|
includes the slow wobble of Earth's pole. These are geometric positions:
|
|
where each body is at the epoch.
|
|
|
|
A camera sees something slightly different, its apparent position:
|
|
|
|
- **Light time.** Light from the Moon takes 1.3 s to reach Earth, and from
|
|
the Sun 8.3 minutes. The camera sees where each body was when the light
|
|
left it. For the Moon that is about 1.3 km of motion, a few microradians.
|
|
- **Aberration.** A moving camera sees incoming light tilted toward its
|
|
direction of motion, as rain seems to slant toward a moving car. Earth's
|
|
orbital speed (30 km/s) tilts the Sun's apparent direction by about 20
|
|
arcseconds, 100 microradians. A spacecraft's own orbital velocity adds up
|
|
to about 5 arcseconds more.
|
|
|
|
100 microradians is about one pixel of the example camera, and 18 pixels of
|
|
a narrow camera with a 1 m focal length.
|
|
|
|
The two effects combine simply. Light time moves a body back along its own
|
|
velocity by (distance / c) times that velocity; aberration moves it along
|
|
the observer's velocity by the same factor. To first order, then, a body
|
|
appears moved in direction by the observer's velocity relative to the body,
|
|
divided by c, the part of it across the line of sight. Only relative
|
|
velocities matter. For the Sun that is Earth's velocity around it plus the
|
|
spacecraft's: the 20 arcseconds. For the Moon, Earth's 30 km/s cancels,
|
|
since the Moon shares it, and what is left is the Moon's own 1 km/s around
|
|
Earth (0.7 arcseconds from Earth's center) plus the spacecraft's velocity.
|
|
For Earth itself it is the spacecraft's velocity, up to 5 arcseconds in low
|
|
orbit, about 0.2 px of the example camera. The terms left out are of the
|
|
order of the velocity squared over c squared, 1e-8 rad.
|
|
|
|
From camera tier C1 the renderer draws Earth, the Moon and the Sun at their
|
|
apparent positions. The scene carries the velocities (`Scene::velocities`:
|
|
the observer's relative to Earth, Earth's relative to the Sun and the Moon's
|
|
relative to Earth), and `Ephemeris::scene` fills in Earth's and the Moon's;
|
|
the observer's is the caller's. All zero, as in C0, the images are
|
|
geometric. Each pixel's ray is turned back, per body, by that body's
|
|
correction before it is tested against the body, so the correction follows
|
|
the ray's direction across a wide view of Earth, and the lighting, shadows
|
|
and eclipses, which happen where the bodies really are, stay geometric.
|
|
`Scene::apparent_sun` and `Scene::apparent_moon` give the same directions on
|
|
the CPU. From Earth's center they match JPL Horizons' apparent positions
|
|
(light time and stellar aberration) to under 0.002 arcseconds, against 20
|
|
arcseconds for the Sun's geometric position.
|
|
|
|
The camera's attitude is taken as the observer would measure it: a star
|
|
tracker sees the stars aberrated by the same motion, so the attitude it
|
|
reports already includes it. A simulation that passes a true, geometric
|
|
attitude gets images shifted by the aberration, as a real camera's would be
|
|
against such an attitude.
|
|
|
|
Still geometric: the targets, whose light time and aberration come from
|
|
their velocity relative to the observer, meters per second in rendezvous,
|
|
under 0.01 microradians; the rotation and lighting of a body during its
|
|
light time (the Moon turns 3 microradians in 1.3 s); and the light-time
|
|
delay between the Sun lighting a body and the camera seeing it.
|
|
|
|
## Stars
|
|
|
|
Code: `stars.rs` (`StarCatalog`, `StarPhotometry`); `gpu/stars.rs`,
|
|
`shaders/stars.wgsl`; `scripts/fetch-stars.py`.
|
|
|
|
Below lighting tier L1 the sky is black. A real camera shows stars wherever
|
|
its exposure is long enough, and they matter: star trackers find their
|
|
attitude from them, a camera measuring the Moon or a target measures it
|
|
against them, and to an algorithm looking for a target a star is a false one.
|
|
|
|
Stars are part of the scene, so they come with lighting tier L1, but only a
|
|
camera at tier C1 or above shows them. A C0 pixel takes its value from the
|
|
one point at its center, and a star is a point that almost never lies
|
|
exactly there; from C1 each pixel collects the light over its whole area,
|
|
which is what records a star.
|
|
|
|
**The catalogs.** `scripts/fetch-stars.py` merges three of ESA's catalogs
|
|
into `data/stars/stars.bin`: Hipparcos, 118,000 stars complete to about
|
|
magnitude 7.3; Tycho-2, 2.5 million stars to about 11.5 (99 % complete to
|
|
11.0); and Gaia DR3, from which it takes every star brighter than G 15 by
|
|
default, 37 million. Each star comes from the first catalog that has it, so
|
|
the brightest, which saturate Gaia's detectors, keep their Hipparcos or Tycho
|
|
measurements. Gaia measures brightness in its own bands, G and the color
|
|
BP - RP: V comes from them by Gaia's published relation (to 0.03 mag), and
|
|
B - V by a fit to the Hipparcos stars Gaia matched. Each star
|
|
keeps its position on the sky (ICRS, at epoch J2000), its proper motion and
|
|
its brightness and color. The renderer moves each star by its proper motion
|
|
to the scene's epoch when the catalog is loaded. Barnard's star, the
|
|
fastest, moves 10 arcseconds a year; most move under 0.1. Parallax, under 1
|
|
arcsecond for every star, is left out.
|
|
|
|
**Magnitudes.** A star's brightness is given as a magnitude: each step of 5
|
|
is a factor of 100, and larger is fainter. Vega is about 0, the faintest
|
|
stars the eye sees about 6, Tycho-2's limit about 11.5, 25,000 times fainter
|
|
than 0. The magnitude in the Johnson V band, a filter close to the eye's
|
|
green-yellow, fixes how much light the star sends through that band: V = 0
|
|
is 3.631e-11 W/m^2/nm averaged over it (Bessell, Castelli and Plez 1998).
|
|
The color index B - V, the difference between the star's magnitudes in blue
|
|
and in V, fixes its temperature: hot stars are blue, B - V below 0, and cool
|
|
ones red, up to about 2. The renderer takes the temperature from B - V
|
|
(Ballesteros 2012: the Sun's 0.65 gives 5,780 K), takes a blackbody spectrum
|
|
at that temperature, scales it so its light through the V band matches the
|
|
star's magnitude, and folds it through the camera's optics and channels, as
|
|
for sunlight. A star's spectrum is not a blackbody's: absorption lines and,
|
|
in cool stars, molecular bands move a few percent of the light between
|
|
channels. For the example camera, a Sun-colored star of V = 0 on the
|
|
boresight gives 56 electrons in green in the 0.1 ms exposure, against an
|
|
independent estimate of 56.4.
|
|
|
|
**Exposure.** That 56 electrons is why the example scenes show no stars: an
|
|
exposure set for a sunlit spacecraft or Earth, a fraction of a millisecond,
|
|
leaves even the brightest stars at a few hundred electrons and the rest
|
|
below one. Star trackers expose for 10 to 200 ms, and a telescope's larger
|
|
aperture collects more: a 1 m f/8 telescope, with a 12.5 cm aperture, gives
|
|
100 times what the example camera does.
|
|
|
|
**Drawing.** Each frame the renderer finds the stars in the field on the
|
|
CPU, from bands of declination sorted by right ascension, so it looks only
|
|
at the stars near the field. It places each at its apparent position, the
|
|
annual aberration of the observer's motion (about 20 arcseconds, see the
|
|
previous section) through the lens's distortion. It works out its electrons
|
|
in each channel: a point source loses light off axis only to the aperture's
|
|
foreshortening, cos(theta), not the cos^4(theta) of an extended surface,
|
|
since all of a star's light that enters lands in its spot. A star is not
|
|
rendered and then blurred: a point of light rendered into a pixel loses
|
|
where in the pixel it fell (see the blur section). Instead its electrons are
|
|
spread over the pixels around it as the Gaussian PSF integrated over each
|
|
pixel's area, which is exact for any position within a pixel. The detector
|
|
adds them to the light from everything else before the full well, so a
|
|
bright star saturates like any other light. A star is hidden wherever
|
|
anything nearer is drawn, Earth, the Moon or a target: stars are at
|
|
infinity, so a star shows only where the depth image is empty. Stars giving
|
|
fewer than 0.1 electrons in every channel, far below any detector's noise, are
|
|
skipped, and the catalog is loaded only to the magnitude that can reach that.
|
|
|
|
Measured: a star moved across a pixel in tenths, behind a lens with
|
|
distortion and with Earth's 30 km/s of aberration, lands where the
|
|
projection puts its apparent direction to 0.01 px, with its electrons
|
|
matching the photometry to 0.2 %. A star behind Earth's night side or
|
|
behind a target leaves the bias alone.
|
|
|
|
Costs. A catalog is loaded only as deep as the camera's exposure can show:
|
|
for the example camera at 0.1 ms that is V 7.9, 42,000 stars, in 6 ms and 7
|
|
MB; at 50 ms, V 14.7, 21 million stars, in 2.8 s and 670 MB; for a 1 m
|
|
telescope at 0.2 s, the whole catalog, 37 million stars, in 5 s and 1.2 GB.
|
|
Each frame, finding and placing the stars takes 23 microseconds for the
|
|
telescope's 0.6 degree field (490 stars toward the Milky Way in Monoceros)
|
|
and 4.8 ms for the example camera's 12 degree field at 50 ms (52,000 stars).
|
|
Drawing them takes 0.24 ms for 40,000 stars on an RTX 3060, and 5.6 ms on
|
|
the CPU through lavapipe.
|
|
|
|
A catalog's depth matters most for a telescope's narrow field. Tycho-2 alone
|
|
averages 60 stars per square degree, so a 1 m telescope's 0.2 square degrees
|
|
would hold about a dozen; with no sky background in space such a telescope
|
|
records stars to magnitude 14 or 15 in half a second, and Gaia to G 15
|
|
averages about 900 per square degree. Deeper still (G 16, 78 million stars)
|
|
is a run of the script with `--gaia 16`, at twice the size.
|
|
|
|
## Not modeled yet
|
|
|
|
- Earthshine on the Moon's night side
|
|
- The diffraction pattern's rings, and a lens's measured point spread function
|
|
- Non-Lambertian materials (BRDFs), glints, and the full Moon's edge as bright
|
|
as its center
|
|
- Clouds on Earth
|
|
- Earth's atmosphere: scattering, the bright limb
|
|
- Motion blur, rolling shutter, stray light
|