What is Gaussian Splatting?

Gaussian Splatting has quickly become one of the more interesting developments in 3D graphics. It offers a surprisingly fast way to capture real environments and turn them into explorable 3D scenes. But what is actually happening behind the scenes?

A different way to capture the world in 3D

You might already be familiar with photogrammetry, a technique for creating realistic 3D models from photographs. There are even phone apps that let you walk around an object or environment and turn the resulting images into a textured 3D model.

Gaussian Splatting approaches the problem differently. Instead of reconstructing a scene primarily as traditional polygon geometry, it represents it using millions of tiny 3D Gaussians distributed through space.

Each one contains information about things like position, size, color, transparency and orientation. Together, these small elements reconstruct the appearance of the original environment. The result can look remarkably photographic while still allowing you to move around the scene in 3D.

NeRF vs. Gaussian Splatting

If you have followed developments in neural rendering, you have probably come across NeRFs, or Neural Radiance Fields. Both NeRFs and Gaussian Splatting can reconstruct scenes from collections of images and generate viewpoints that were not directly photographed, but they work in quite different ways. NeRFs use neural networks to learn a representation of a scene and calculate what it should look like from different viewpoints. Gaussian Splatting represents the scene more explicitly through its collection of 3D Gaussians. Despite often being discussed alongside AI technologies, the rendering technique itself does not require a neural network. One of its biggest advantages is speed. Gaussian Splatting can render complex scenes in real time at high quality, making it particularly interesting for interactive 3D experiences, VR and games. NeRFs can still have advantages depending on the scene and implementation, particularly when it comes to certain fine details and view dependent effects. The two technologies are better thought of as different approaches to the same broader challenge rather than one simply replacing the other.

Can I make Gaussian Splats myself?

Yes. You no longer need a research lab or a complicated 3D pipeline to experiment with the technology.

Luma AI lets you capture objects and environments using a smartphone and reconstruct them in 3D.

Polycam also supports Gaussian Splatting and lets you capture scenes directly with your phone.

The quality still depends heavily on how you capture the scene. Good coverage from multiple angles matters, and reflective, transparent or moving objects can make reconstruction more difficult.

What can I do with them?

Once you have created a splat, it can become part of a larger 3D workflow. Gaussian Splats can be brought into tools such as Spline, Blender, Unity and Unreal Engine, depending on the format and available plugins. What makes the technology particularly interesting is how little work can sit between capturing a real place and having an explorable digital version of it. Instead of manually modelling an environment from scratch, you can start with reality itself. For retail, entertainment and spatial experiences, that opens up some interesting possibilities. Physical spaces can become digital environments, real products can be placed inside interactive experiences, and locations can be captured without recreating every surface by hand.