Reconstructing a three-dimensional scene from ordinary two-dimensional images is a long-standing problem in computer vision, and the usual routes carry a heavy tax: they lean on hand-produced supervision — depth maps, segmentation masks, object poses — that is expensive to collect and rarely available for the kind of casual photos and video a phone captures. A patent granted to Snap Inc. on July 7, 2026, US12675908B2, titled "Estimating 3D scene representations of images," is directed to a method that sidesteps that tax by learning the reconstruction without those labels.

The core object is a structured 3D scene representation. Rather than producing a single undifferentiated point cloud or mesh, the disclosed model separates a scene into parts. Claim 1 describes the representation this way:

the 3D scene representation explicitly and separately defines a 3D shape and appearance of a background of the scene and a 3D position, 3D shape and appearance of each object of the scene depicted in the set of 2D images— Estimating 3D scene representations of images, US12675908B2

That separation is what makes the output editable. Because each object carries its own position, shape, and appearance distinct from the background, an application can move, remove, or composite around individual objects — as opposed to a monolithic reconstruction where the scene is baked into one surface. The disclosure notes the method operates from one or multiple 2D images, so it is not restricted to multi-view capture; a single frame can serve as the input from which the structured representation is inferred. The sole named inventor, Titas Anciukevičius, structures the model as a generative latent-variable system, with per-object latents represented as diagonal Gaussians defined by a mean and a log-variance — a compact, per-object code the model can sample from rather than a fixed feature grid.

How the model is built

Object placement is handled probabilistically. The disclosure describes selecting object positions from candidate locations using a one-hot categorical scheme with a Gumbel-Softmax relaxation, which lets the discrete choice of where an object sits be learned by gradient descent along with everything else. That relaxation matters because a hard categorical selection is not differentiable; the Gumbel-Softmax approximates it with a continuous distribution during training, so the placement decision can be optimized end to end with the rest of the model. Two networks do the heavy lifting: one maps the Gaussian latents into the 3D scene representation, and a second renders that representation back into a 2D image using a points-to-color-and-density approach reminiscent of neural radiance fields. Training proceeds in two stages — object-level first, then scene-level — so the model learns to describe individual objects before composing them into full scenes.

The unsupervised setup is the load-bearing claim element. The model is trained from a dataset of images and their camera poses, without manually labeled annotations such as depth maps, segmentation masks, or object poses. Claim 1 further recites sampling the 3D scene representation from a posterior distribution conditioned on sets of images drawn from a distribution different than the training distribution — framing the reconstruction as probabilistic inference and pointing at generalization beyond the exact scenes seen during training, rather than a deterministic lookup. Casting the output as a sample from a posterior, rather than a single point estimate, also fits the generative structure of the model: the same input can support a distribution over plausible 3D interpretations, which the diagonal-Gaussian object latents and categorical placement are set up to represent.

Where it fits in Snap's AR stack

An explicit, per-object 3D representation is directly useful for augmented reality: a dependent claim covers placing AR or VR virtual elements into video using the recovered 3D scene, which requires knowing where real objects are in space so virtual ones can be positioned and occluded convincingly. The distinction from a flat overlay is exactly the structured representation — with per-object geometry and a separate background, a virtual element can pass behind a real object or rest on a real surface instead of floating on top of the pixels. That the representation is recovered without hand labels is what makes it practical at consumer scale, where the source material is whatever a user's camera happened to capture rather than a curated, annotated dataset. The grant sits alongside a same-week Snap cohort spanning the rest of that pipeline. On rendering and perception, US12675840B2 is directed to late warping that time-warps rendered content to an updated pose to minimize latency for moving objects, and US12675930B2 describes a state-space system for pseudorandom animation.

On interaction and hardware, US12675169B2 covers matching observed against captured gesture motion to start a shared AR session, US12675157B2 is directed to pausing device operation — such as AR sensors — when a user's eyes look away, and US12674986B2 describes an adjustable display arrangement that switches the virtual field of view on extended-reality devices. Read together, the records describe scene understanding, rendering, interaction, and display for a wearable AR platform. The 3D-reconstruction grant addresses the perception layer: recovering, from flat images and without hand labels, a structured scene that downstream AR components can render into and place virtual content within. Late warping presumes rendered content that must be aligned to a moving world; gaze-based pausing and gesture-triggered sessions presume a system that understands what the user is looking at and doing. Each of those depends on some model of where things are in three dimensions, and the reconstruction method supplies exactly that — an explicit, per-object account of the scene, learned from the images a consumer device already collects rather than from a labeling pipeline it would otherwise need.