resolved/docs/optics.md

Optics and radiometry guide

This explains the physics the renderer models, for a reader who knows orbital mechanics but not optics. Each section says which code implements it. When a feature adds a new optical effect, it adds or updates a section here in the same change.

Numbers in the examples come from CameraConfig::example() unless stated otherwise: 1920 x 1080 pixels, 5.5 um pitch, 50 mm focal length, f/4, three color channels, 12-bit output.

The chain from Sun to number

Every pixel value comes from the same sequence:

1. The Sun emits power across a range of wavelengths.
2. Some of it lands on a surface (Earth, the Moon, the target), which reflects part of it in various directions.
3. The fraction reflected toward the camera enters the lens aperture. The optics focus it onto one pixel, losing a little to absorption.
4. The pixel converts some of the arriving photons into electrons. The fraction depends on wavelength and on the color filter over that pixel.
5. The electrons pile up during the exposure, up to a limit.
6. The readout electronics turn the electron count into an integer, the digital number (DN).

The renderer follows this chain with physical units at every step. Image brightness therefore comes from the scene and the camera settings, not from a tuned scale factor. Change the exposure time, and the image changes exactly as a real camera's would.

Geometry: the pinhole camera

Code: camera.rs (Intrinsics), scene.rs.

A pinhole camera maps each direction in space to one point on the focal plane. Put the pinhole at the origin, looking along +Z. A point at camera coordinates (x, y, z) lands on the focal plane at (f x / z, f y / z), where f is the focal length. Dividing by the pixel pitch p converts focal-plane distance to pixels:

u = (f / p) * x / z + cx
v = (f / p) * y / z + cy

f / p is the focal length in pixels. (cx, cy) is the principal point, where the boresight hits the image, normally the image center. For the example camera, f / p = 50 mm / 5.5 um = 9091 pixels.

Some quantities that follow from that:

• IFOV, the instantaneous field of view. The angle one pixel covers: p / f = 110 microradians. At 10 m that is 1.1 mm per pixel; at 7000 km it is 770 m per pixel. This sets the smallest detail the camera can resolve.
• FOV, the field of view. The angle the whole image covers: 2 atan(width / 2 / (f / p)), about 12.1 x 6.8 degrees here. A longer focal length or smaller sensor gives a narrower field of view and finer IFOV.

Real lenses are not perfect pinholes. They bend straight lines slightly (distortion, planned in M3) and spread each point into a small blur (the point spread function, also M3). The pinhole model is the starting point those effects are added to.

Coordinate conventions

The camera frame has +Z along the boresight, +X toward increasing column u, and +Y toward increasing row v. This is the computer-vision convention: row 0 is the top of the image, so +Y points down in the picture. Pixel (i, j) covers u in [i, i+1) and v in [j, j+1), so its center is at (i + 0.5, j + 0.5).

Why the GPU sees camera-relative coordinates

GPUs compute in 32-bit floats, which carry about 7 significant digits. A spacecraft 7000 km from Earth's center, stored in Earth-centered f32 coordinates, can only be placed to the nearest 0.5 m. A target 10 m away would jitter by several pixels. So all positions stay in 64-bit floats on the CPU. They are made relative to the camera there, and converted to 32-bit only after that. A target 10 m from the camera is then placed to about a micrometer.

Depth

When two things overlap in the image, the nearer one must win. The GPU keeps a depth value per pixel for this. The renderer stores near / z, which is 1 at the near clip distance and falls toward 0 with distance. Combined with how floating-point numbers are spaced, this gives roughly the same relative precision at every distance. That matters in a scene that contains a 10 m target, Earth at thousands of kilometers, and the Sun at 150 million kilometers. The scheme is called reversed-Z with an infinite far plane.

Radiometric quantities

Code: spectrum.rs, radiometry.rs.

Light has a few distinct measures, and the camera measures a specific one.

• Irradiance, E, in W/m^2. Power arriving per unit area of a surface. Sunlight at 1 AU delivers about 1361 W/m^2 to a surface facing the Sun. It falls off with the square of the distance from the Sun. A surface tilted by angle theta from the Sun direction receives E cos(theta).
• Solid angle, in steradians (sr). The 2D version of an angle: the area a cone cuts out of a unit sphere. The full sphere is 4 pi sr. One pixel of the example camera covers (110 urad)^2, about 1.2e-8 sr.
• Radiance, L, in W/m^2/sr. Power per unit area per unit solid angle, traveling in a particular direction. It is what a camera pixel responds to: how bright a surface looks from a given direction.

Radiance has a property that explains a lot of what a camera sees: it does not change along a ray through empty space. Move a camera twice as far from a wall, and each pixel sees four times as much wall area. But each square meter now contributes a quarter of the power, so the pixel value is unchanged. An object that covers many pixels has the same surface brightness at any distance; it just covers fewer pixels farther away.

That stops being true once the object is smaller than a pixel. The pixel then also sees dark space around the object, so the signal falls as one over distance squared. That is the regime of a distant satellite seen as a point (space situational awareness), as opposed to the resolved imagery this renderer is mostly for.

Reflection: albedo and Lambertian surfaces

A Lambertian surface reflects light equally in every direction. Matte paint, the Moon (roughly) and cloud tops behave close to this. If such a surface receives irradiance E and reflects a fraction rho (its albedo), its radiance in every direction is:

L = rho * E / pi

The pi comes from integrating the reflected radiance over the hemisphere above the surface. A white Lambertian sheet facing the Sun at 1 AU, with rho = 1, has a radiance of about 1361 / pi = 433 W/m^2/sr, summed over all wavelengths. Earth's average albedo is about 0.3.

Real spacecraft surfaces are not Lambertian. Multilayer insulation (MLI) and solar cells are partly mirror-like and produce bright glints at particular angles. These are modeled by a BRDF, a function giving reflected radiance for every pair of incoming and outgoing directions. That arrives in M4.

Wavelength: spectra, channels and photons

Code: spectrum.rs (Spectrum), camera.rs (Channel).

Everything above varies with wavelength. Sunlight peaks around 500 nm and extends from the ultraviolet into the infrared. Surfaces reflect different wavelengths differently; that is what color is. Optics transmit some wavelengths better than others. The sensor's sensitivity also depends on wavelength.

The renderer describes each of these as a Spectrum: a value as a function of wavelength, in meters.

• Quantum efficiency (QE). The fraction of arriving photons that become electrons in a pixel. A color camera puts red, green and blue filters over different pixels, so each channel has its own QE curve. The example camera uses idealized Gaussian curves centered at 620, 540 and 460 nm, 80 nm wide at half height, peaking at 60 %.
• Transmission (tau). The fraction of light the lenses pass, about 0.9 for the example camera.

Detectors count photons, not energy. A photon of wavelength lambda carries energy h c / lambda (h is Planck's constant, c the speed of light). So one watt of red light delivers more photons than one watt of blue. The renderer converts power to photons per wavelength before applying QE.

The solar spectrum

Code: radiometry.rs (solar_spectrum), generated into solar_data.rs by scripts/fetch-solar-spectrum.sh.

The renderer uses a measured solar spectrum, the TSIS-1 Hybrid Solar Reference Spectrum (Coddington et al. 2021). It is the current international reference, built mainly from the TSIS-1 instruments on the International Space Station and filled in with a high-resolution model of the solar absorption lines. It gives the irradiance at 1 AU, outside the atmosphere, every 1 nm from 202 to 2730 nm. Its uncertainty is 0.3 % from 460 to 2365 nm. Over that range it sums to 1325.8 W/m^2. The other 35 W/m^2 of the 1361 W/m^2 total is at wavelengths silicon detectors do not see.

The Sun is often approximated as a 5772 K blackbody: a smooth curve with the right total power. The real spectrum has thousands of absorption lines from the Sun's outer layers, and its shape differs from the smooth curve. For the example camera's channels, the blackbody gives 4.6 % (red), 7.5 % (green) and 9.7 % (blue) fewer electrons than the measured spectrum. That is far outside what absolute radiometry can tolerate.

Folding spectra into per-channel constants

Code: radiometry.rs (Radiometry).

Doing all of this wavelength by wavelength for every pixel would be slow. Most factors do not depend on the pixel: the solar spectrum, the optics, and the QE curves. The renderer multiplies and integrates those over wavelength once, when the camera is set up. That yields a few numbers per channel, and the GPU only multiplies by them.

The central number is the solar flux of each channel k:

S_k = integral of E_sun(lambda) * tau(lambda) * QE_k(lambda) * lambda / (h c) d lambda

It is the rate of electrons per square meter that sunlight at 1 AU would produce if it fell straight onto the detector through the optics. For the example camera, S_k is about 2.4e20 electrons per second per m^2 in red and green, and 2.0e20 in blue.

The renderer then works in channel radiance: scene radiance weighted the same way (transmission, QE, photons per joule), in electrons per second per m^2 per steradian. A Lambertian surface with albedo rho, lit at angle i from its normal, at distance d from the Sun in AU, has channel radiance:

L_k = rho * cos(i) * S_k / (pi * d^2)

The solar disk has channel radiance S_k / Omega_sun, where Omega_sun is the Sun's solid angle at 1 AU, 6.79e-5 sr (it is 0.53 degrees across). Seen from anywhere, the Sun's surface has that same radiance, about 2.0e7 W/m^2/sr in total. Looked at directly, it puts about 5e8 electrons into a pixel of the example camera's green channel in 0.1 ms, 26,000 times the full well.

A surface whose reflectance varies with wavelength is reduced to an effective albedo per channel. That is the albedo of a flat-spectrum surface that would give the same signal under sunlight (Radiometry::effective_albedo). This step is where the error of the shortcut comes in. Two spectra that both vary inside one channel's band interact in ways a single number per channel cannot capture. One example is a surface whose reflectance changes steeply across green, lit partly by sunlight and partly by bluish light scattered from Earth's atmosphere. Rendering more, narrower channels reduces that error, and the code keeps the channel count generic for that reason.

From radiance to electrons: the camera equation

Code: radiometry.rs (Radiometry::electrons); the sensor pass, #9.

A lens with focal length f and aperture diameter D has f-number N = f / D. The example camera is f/4: a 50 mm lens with a 12.5 mm aperture. For a scene radiance L, the irradiance on the focal plane at the image center is:

E_image = pi * L * tau / (4 N^2)

The aperture collects light over a solid angle proportional to (D / f)^2 = 1 / N^2. So opening the aperture one "stop", which divides N by the square root of 2, doubles the light. Image irradiance is independent of distance for the same reason radiance is.

Away from the center, image irradiance falls off as cos^4(theta), where theta is the angle from the boresight. Two factors come from the longer path to the off-axis pixel. One comes from the aperture appearing foreshortened. One comes from light reaching the pixel at a slant. At the corner of the example camera, theta is 6.9 degrees, and the falloff is 3 %. Real lenses add more falloff of their own, called vignetting.

The electrons a pixel collects in one channel are then:

electrons = E_image * (pixel area) * (exposure time) * (photons per joule) * QE

The last two factors are wavelength-dependent. In channel radiance they are already folded in, so the camera equation reduces to

electrons = L_k * pi * (pixel area) * (exposure time) / (4 N^2) * cos^4(theta)

For the example camera, pi * A * t / (4 N^2) is 1.48e-16 m^2 sr s.

From electrons to DN: the detector

Code: camera.rs (Detector); the sensor pass, #9.

• Full well. The maximum number of electrons a pixel holds. Beyond it the pixel saturates and reads its maximum value whatever the light. Brighter scenes do not look brighter. In real sensors, excess charge can also spill into neighboring pixels (blooming, M3).
• Gain. Electrons per DN. With a gain of 5, every 5 electrons add 1 to the output.
• Bias. A fixed offset added to every pixel, so that noise around zero signal does not get clipped at 0. The example uses 64 DN.
• ADC bits. The output resolution. 12 bits gives values 0 to 4095.

So DN = bias + electrons / gain, rounded and clipped to the ADC range.

Worked example

Take a white Lambertian surface facing the Sun at 1 AU, seen by the example camera's green channel. Integrating the solar spectrum through the green QE curve, f/4 optics, 5.5 um pixels and a 0.1 ms exposure gives 11,139 electrons. Red gets 11,408 and blue 9,626. (A test in radiometry.rs checks the green figure.)

• DN = 64 + 11,139 / 5, about 2292, a little over half the 12-bit range.
• The full well is 20,000 electrons, so this surface is at 56 % of saturation.
• Earth, with albedo near 0.3, reads about 64 + 3,342 / 5, about 732 DN.

Picking the exposure time is the same trade a photographer makes. A longer exposure saturates the sunlit target. A shorter one leaves the side of the target lit only by Earthshine lost in noise once noise is modeled (M3).

How shapes become pixels

Two techniques turn 3D shapes into an image. The renderer uses one for the bodies and the other for the target.

• Mesh. A surface approximated by many small flat triangles that share corners (vertices). A cube needs 12 triangles. A curved surface needs many more, and it is still faceted up close. Each vertex stores a position and usually a normal, the direction the surface faces there. STL and glTF files are meshes, and so is the target spacecraft.
• Rasterization. The GPU's built-in way of drawing a mesh. Each triangle's corners are projected into the image with the pinhole equations above. The GPU then finds every pixel whose center falls inside the projected triangle, and runs a small program (a fragment shader) for each of those pixels to compute its value. It is fast because it only touches the pixels each triangle covers, and GPUs have dedicated hardware for it. It can only draw triangles.
• Ray casting. The reverse direction: for each pixel, follow the line of sight from the camera through the pixel center and solve for where it first meets a surface. For a sphere or an ellipsoid that is a quadratic equation with an exact answer, so there are no triangles and no facets. Every pixel does the work, whether or not it ends up seeing the body.

The bodies use ray casting, because their shapes have exact equations and their edges are what limb-based navigation measures. The target uses rasterization, because a spacecraft model has no equation, only its mesh. Both write into the same depth image, so they sort against each other correctly. Ray casting a mesh is also possible, by testing rays against its triangles. That is what M5's ray tracer does, because following rays gives exact shadows and light bouncing between surfaces. Rasterization can only approximate those, for example with the shadow maps planned for M2.

Earth, Moon and Sun

Code: bodies.rs, gpu/bodies.rs, shaders/bodies.wgsl.

The three bodies are not drawn as triangle meshes; they are ray cast. For every pixel, the shader sends a ray from the camera through the pixel's center and solves for where it meets each body's exact shape. The nearest hit wins. This gives an exact edge at any zoom, at the cost of a few arithmetic operations per pixel. A mesh would show its facets along the limb, the edge of the disk, which is the feature horizon-based navigation measures.

• Earth is the WGS84 ellipsoid: 6378.137 km equatorial radius, 6356.752 km polar. The 21 km difference matters at fine pixel scales. A camera with 5.5 urad pixels at 200,000 km sees the north limb 19 pixels away from where a spherical Earth would put it. A test checks the limb against the exact tangent to the ellipsoid to within a pixel.
• The Moon is a sphere of 1737.4 km radius. From Earth's distance it is about 0.52 degrees across, 82 pixels in the example camera.
• The Sun is a sphere of 695,700 km radius: 0.53 degrees across, about 85 pixels in the example camera. Its disk has the solar radiance from the folding section above, and it is drawn only where no nearer body covers it. So Earth eclipses it naturally.

Shading

Earth and the Moon are Lambertian, with one albedo each, set by Appearance: 0.3 for Earth and 0.12 for the Moon, flat across wavelength. Each point's radiance is rho cos(i) S_k / (pi d^2), with i the angle between the surface normal and the Sun. The night side has i past 90 degrees and is black. The Sun's distance d is taken from each body's center, since the difference across a body is a few parts in 10^5.

This gets the overall brightness and the phase (how much of the disk is lit) right. It misses some things that are visible in real images:

• Earth's oceans, land and clouds differ in brightness by a factor of several. An albedo map comes in M2.
• The full Moon looks almost uniformly bright out to its edge. A Lambertian sphere instead darkens toward the edge, because the light there arrives and leaves at grazing angles. Lunar soil scatters light back toward its source much more strongly than Lambert predicts, and a dedicated lunar reflectance model (M4) fixes that. The Moon's maria (the dark plains of old basalt lava, the "seas" seen with the naked eye) and its bright highlands also differ in albedo by about a factor of two (M2 albedo map).
• Earth's atmosphere softens the limb and adds a thin bright band around it (M4). Here Earth's edge is a sharp line.

Depth from the bodies

The pass also writes each hit's depth, near / z, as described in the geometry section. The target spacecraft, drawn after the bodies, uses the same depth buffer. So a target in front of Earth covers it, and a target behind Earth's limb is hidden.

The target model and its materials

Code: model.rs.

The target spacecraft is a triangle mesh loaded from STL or glTF. Each piece (an STL file, or a glTF primitive) is a part with one material, and for now a material is a Lambertian albedo spectrum, as for Earth and the Moon.

Surface normals decide how bright each triangle is, through the cos(i) factor. STL files store a normal per triangle, but exporters often write zeros or wrong values, so the loader ignores them. It recomputes each normal from the order of the triangle's corners: counterclockwise as seen from outside. Every triangle is then lit as a flat facet, which matches the flat panels and boxes most spacecraft are built from. glTF files that store normals keep them.

glTF describes a material's color as red, green and blue reflectances (its "base color"). Three numbers do not define a spectrum: very different spectra can look the same color to the eye. The loader turns them into a stepped spectrum, blue below 490 nm, green from 500 to 580 nm, and red above 590 nm. That is a reasonable guess for painted surfaces seen by an RGB camera. It is a poor one for a camera whose channels differ from the eye's, or for anything in the near infrared, where the red value simply continues. For absolute radiometry of a real spacecraft, give the important materials measured reflectance spectra (M4 adds those, and BRDFs).

STL has no units. CAD tools usually export millimeters, so loading a CAD model usually needs scale: 0.001. glTF is always meters.

Drawing the target

Code: gpu/mesh.rs, shaders/mesh.wgsl.

The target's mesh is rasterized after the bodies, into the same radiance and depth images. Each pixel the target covers is kept only if the target is nearer than what is already there. That one rule handles a target in front of Earth, a target behind Earth's limb, and parts of the target hiding other parts.

Each point on the target is shaded like the bodies: albedo times cos(i) times the sunlight that reaches it. A panel modeled with no thickness, such as a solar array, has one surface but two sides. Whichever side faces the camera is the side that is lit. A closed shape never shows its inside, so this only matters for thin parts.

Eclipses

Code: eclipse.rs.

The Sun is a disk, not a point. So as a spacecraft passes into Earth's shadow, the light fades over a few seconds instead of switching off:

• Umbra. Earth covers the whole solar disk. No direct sunlight.
• Penumbra. Earth covers part of the disk. 7000 km from Earth's center, on the night side, the penumbra is about 65 km wide; a satellite crosses it in under 10 seconds.
• Annular. A body small enough to sit inside the solar disk blocks a ring-shaped bite. From orbit this happens with the Moon.

The renderer finds the visible fraction of the solar disk from the overlap of two circles on the sky. The circles are the Sun's disk and the occulter's, each with its apparent angular radius. It multiplies the target's sunlight by that fraction. The model is the standard one for satellites (Montenbruck and Gill), the same as astrokit's. A test checks it against tracing lines from the target to thousands of points spread over the solar disk.

The fraction is computed once per target, at its center. The whole target dims together, which is accurate unless the target is a sizable fraction of the penumbra's width. Light reaching the target from Earth itself (Earthshine), which matters most when the target is in shadow, comes in M2.

Not modeled yet

Effect	Where
Light from Earth and the Moon illuminating the target (Earthshine, Moonshine)	M2
Earth's shadow on the Moon (lunar eclipses)	M2
Earthshine on the Moon's night side	M4
Shadows the target casts on itself	M2
The Sun as a disk that is darker toward its edge (limb darkening)	M2
Lens distortion	M3
Blur from diffraction and lens imperfections (point spread function)	M3
Noise: photon shot noise, read noise, dark current, pixel-to-pixel variation	M3
Saturation spilling into neighbors (blooming)	M3
Non-Lambertian materials (BRDFs), glints, lunar backscatter	M4
Earth and Moon albedo maps	M2
Earth's atmosphere: scattering, the bright limb	M4
Motion blur, rolling shutter, stray light	M6