How the Gaussian splatting viewer works

A 3D Gaussian splatting scene is not a mesh. It is a few million translucent ellipsoids — each with a position, an anisotropic covariance, a colour and an opacity — that together reconstruct a photographed space well enough to walk around in. The viewer on this site renders such a scene at interactive rates inside a browser tab, with no engine and no dependencies. This post walks the whole pipeline: the 32-byte wire format, the covariance algebra, the integer data texture, the worker-side counting sort that makes the camera feel weightless, and the shader that turns each gaussian into two triangles.

Why order is the whole problem

Splats composite; they don't depth-test. The colour of a pixel is the standard over operator applied front to back:

C = Σᵢ cᵢ·αᵢ·Πⱼ₍ⱼ₌₁…ᵢ₋₁₎ (1 − αⱼ)

That product term is the transmittance of everything in front of splat i — and it makes the sum order-dependent. Red over blue is not blue over red. A z-buffer can't rescue you, because nothing here is opaque: every fragment must be blended in strict depth order or the reconstruction degenerates into fog.

The consequence is brutal for an interactive viewer: every camera movement invalidates the global sort. Not a partial update — a full re-ordering of the entire scene, ideally inside a 16 ms frame budget while the main thread is also busy handling the very drag gesture that caused the problem. The rest of this post is essentially the story of paying that cost somewhere the user can't feel it.

The .splat format: 32 bytes, no header

The viewer's native format is antimatter15's .splat: a bare array of fixed-size records, no header, no footer, count implied by file size.

Quantising rotation to 8 bits per component sounds vandalous and isn't: an orientation error of half a degree on a blob that is itself a soft Gaussian falloff is invisible. The positions and scales stay full float32 because those errors would be visible, as cracks and swimming.

From quaternion to covariance, once, at load

The renderer needs each gaussian's 3D covariance Σ = R·S·Sᵀ·Rᵀ, where R is the rotation and S the diagonal scale matrix. Σ is symmetric, so six numbers suffice. The worker computes M = R·S and takes the six upper-triangular dot products — this is verbatim from the pack step:

var M = [R[0]*sx, R[1]*sx, R[2]*sx,
         R[3]*sy, R[4]*sy, R[5]*sy,
         R[6]*sz, R[7]*sz, R[8]*sz];
var s0 = M[0]*M[0] + M[3]*M[3] + M[6]*M[6];   // Σxx
var s1 = M[0]*M[1] + M[3]*M[4] + M[6]*M[7];   // Σxy
var s2 = M[0]*M[2] + M[3]*M[5] + M[6]*M[8];   // Σxz
var s3 = M[1]*M[1] + M[4]*M[4] + M[7]*M[7];   // Σyy
var s4 = M[1]*M[2] + M[4]*M[5] + M[7]*M[8];   // Σyz
var s5 = M[2]*M[2] + M[5]*M[5] + M[8]*M[8];   // Σzz

Those six values are packed as three uint32s, two half-floats per word (packHalf2), and written — alongside the position and colour — into a single unsigned-integer texture, two texels per splat, 2048 texels wide. The vertex shader fetches by index with pure bit arithmetic:

uvec4 cen = texelFetch(u_texture, ivec2((uint(index) & 0x3ffu) << 1, uint(index) >> 10), 0);
uvec4 cov = texelFetch(u_texture, ivec2(((uint(index) & 0x3ffu) << 1) | 1u, uint(index) >> 10), 0);

Low ten bits → column pair, high bits → row. Why an integer texture rather than a float one? Because the payload is heterogeneous — float positions, half-float covariances, byte colours share the same texels — and because integer texels are never filtered. A covariance term that got bilinearly interpolated against its neighbour would be a subtle, maddening bug; usampler2D makes it structurally impossible.

The counting sort, off the main thread

Sorting lives in a Web Worker. Whenever the camera settles into a new pose, the main thread posts the view matrix; the worker projects every splat onto the view axis — one dot product with the matrix's third column — and orders them far-to-near. The interesting choice is how it sorts:

var SORT = 256*256;                       // 65,536 depth buckets
var b = ((dep[i] - minD) * range) | 0;    // quantise into a bucket
counts[b]++;                              // histogram
starts[i] = starts[i-1] + counts[i-1];    // prefix sum → bucket offsets
order[starts[bucket[i]]++] = i;           // scatter, stable within bucket

Sixteen bits of depth resolution is not a compromise you can see. Two splats that land in the same bucket are so close along the view axis that either compositing order produces the same pixels to within the 8-bit framebuffer's ability to represent them. And because the sort runs behind the render loop rather than inside it, a fast drag draws with an ordering a frame or two stale — an artefact measurably present and perceptually absent. The alternative, blocking the gesture on the sort, is the one users would actually notice.

The result posts back as a transferable Uint32Array — pointer handoff, zero copy — and becomes the instance index stream.

Two triangles per gaussian

The draw itself is one instanced call: a four-vertex triangle strip, stretched per instance to cover the projected ellipse. Projecting a 3D Gaussian through a perspective camera yields — to first order, via the Jacobian of the projection (Zwicker et al.'s EWA splatting) — a 2D Gaussian with covariance cov2d. The shader diagonalises that 2×2 matrix in closed form:

float mid    = (cov2d[0][0] + cov2d[1][1]) / 2.0;          // ½·trace
float radius = length(vec2((cov2d[0][0] - cov2d[1][1]) / 2.0, cov2d[0][1]));
float lambda1 = mid + radius, lambda2 = mid - radius;       // eigenvalues

Eigenvalues give the ellipse's axes; the quad's corners are displaced along the corresponding eigenvectors and the fragment shader evaluates exp(−r²) times opacity. Everything per-splat — fetch, project, diagonalise — happens in the vertex shader, four times per splat. The projection and covariance math follows the MIT-licensed antimatter15/splat reference, which remains the clearest minimal implementation of this pass.

The demo scene is the garden

The embedded demo is a zoomed-in excerpt of the "garden" scene from Mip-NeRF 360 (Barron et al. 2022; splat conversion by antimatter15) — the table and vase that have become radiance-field furniture. The full capture is 5.83 million gaussians and 186 MB, which no self-contained page can carry, so a build-time tool (modules/gsplat/tools/crop_garden.py) reduces it honestly rather than thinning it randomly:

• Find the focal point by density — a coarse 3D histogram where each splat is weighted by opacity and, crucially, by inverse size. Trained scenes spend their finest gaussians on the focal object, so the fine-detail peak is the table, not the acres of grass.
• Crop a sphere around that peak, radius chosen so ~120k splats survive — every splat inside kept, nothing inside thinned.
• Recentre and rescale so the excerpt fills the default orbit camera, which is what makes it read as "zoomed in" rather than "cut down".

120k splats is 3.8 MB of records — about 2% of the capture carrying a lot more than 2% of the visual interest, precisely because the crop follows where the optimiser spent its budget.

Using it yourself

Bring a scene. Drag any .splat onto the page, or an Inria-layout .ply straight out of 3DGS training. The PLY reader requires the training fields by name — x y z, scale_0..2, rot_0..3, opacity, f_dc_0..2 — un-logs the scales, sigmoids the opacity, and reorders the quaternion, because Inria writes w,x,y,z and the packing wants x,y,z,w. Get that reorder wrong and every ellipsoid is rotated into a different wrong orientation, which looks like static rather than a bug.

Or synthesise one. The format is so simple that "exporter" is overstating it — this is a complete writer:

# any point cloud → .splat, stdlib only
import struct

def write_splat(path, splats):
    """splats: iterable of (pos3, scale3, rgba4_0..255, quat4_unit)"""
    with open(path, "wb") as f:
        for p, s, c, q in splats:
            f.write(struct.pack("<3f", *p))
            f.write(struct.pack("<3f", *s))
            f.write(bytes(c))
            f.write(bytes(int(v * 128 + 128) for v in q))   # bias to u8

Point it at a simulation state, a molecule, a LiDAR sweep — anything that is positions with extent — and you have a real-time viewer for it. And to make your own excerpt of a big capture:

python3 modules/gsplat/tools/crop_garden.py garden.splat --target 120000
# writes demo.splat + demo.splat.b64; --flip if the capture's up-axis disagrees

What it deliberately doesn't do

There is no frustum culling and no level-of-detail: every splat is sorted and drawn every frame, so capability falls off a cliff somewhere north of a couple million splats rather than degrading gracefully. And the counting sort's per-frame min/max depth rescan means the bucket mapping shifts as you zoom — harmless here, but it would break a incremental-sort scheme if you tried to bolt one on. Those are the three things I would add, in that order, if this viewer had to graduate from demo to product.

Related: the weights the training process produces are themselves worth opening — reading a checkpoint in the browser.