62. PartField
Learning 3D feature fields for part segmentation and beyond
Introduction
General 3D encoders are becoming popular as we adopt more scalable architectures and expand the size and diversity of training datasets. For instance, models like Sonata [2], Concerto [3], and Utonia [4], which I covered in posts #46 and #58, are some of the most prominent works at the scene level.
However, translating these advancements into a more fine-grained object understanding (e.g., where to grasp objects in robotics) is hard due to differing notions of part granularity across datasets.
Today, we take a look at PartField [1] (ICCV 2025), a fast, 3D-native feedforward model from NVIDIA that produces a continuous feature field useful for part decomposition, shape correspondences, or simply as a powerful 3D encoder.
Method
PartField takes a point cloud as input (meshes and 3D Gaussians are also accepted, as we can sample points on the surface or take the centers) and assigns a feature vector to each point, encoding structural proximity. The idea is that points that live in the same “part” (e.g., the legs of the deer) are close to each other in this learned latent space, while two unrelated ones are far.
So how do we teach something like this? I voluntarily said close and far because that is exactly how a contrastive loss works! Given a part proposal, we sample a triplet of points on the fly where two points belong to the same part (an anchor and the positive), while the third belongs to another part (the negative).
The anchor (a) and positive (b) are pushed closer by maximizing a scaled cosine similarity between their feature vectors, while the anchor and negative (c) get pushed away as a byproduct. The anchor and positive are also flipped in the objective.
This is similar to a standard triplet loss setting, except here we only encourage the relative relations between points within a triplet to be well ordered. To make it more sample efficient, allow for hierarchical part learning, and encourage sharp boundaries, negative points are sampled using a mix of three strategies:
Uniform negatives, uniformly sampled from the complement of the part
proposal.
3D-hard, preferring negatives closer to the anchor in Euclidean space, but belonging to another part.
Feature-hard, similarly picking negatives closer to the anchor, but in feature space instead.
To speed up training even further, the contrastive loss is computed in parallel over many negative samples for each positive one. And that’s it: triplet loss over points sampled from different parts. But how do we get the parts, and what model is used?
Data strategy
Since part-labeled 3D data is limited, the model extracts part proposals from a mix of 2D and 3D data sources. For the 3D part, the framework leverages explicit geometric annotations from PartNet [5], which comes with hierarchical part labels but is limited to 30k shapes. When a shape has a volume or interior structure, the pipeline also samples valid training points from the internal coordinates.
For the 2D pipeline, which constitutes the bulk of the training set with 304k shapes, the authors render multi-view images from large-scale unlabeled object datasets like Objaverse [6] and then prompt Segment Anything 2 with a dense grid of points to generate class-agnostic 2D masks. These masks are then unprojected back onto the 3D surface using known depth maps and camera poses to yield highly diverse, multi-scale part proposals.
This dual strategy is highly effective because the unsupervised 2D proposals scale well and provide open-world generalization capabilities, while the 3D labels supply clean structural signal as well as interior supervision.
3D model architecture
Last but not least, a few words about the model: it is a feedforward hybrid 3D architecture, which makes it very fast compared to others! A point cloud is passed as input, and the first stage is a PVCNN [7] that returns per-point feature vectors in a high-dimensional space.
Then, these are projected onto three coordinate frames (triplane) through an orthogonal projection with mean-reduction, creating three feature maps of size 128×512×512. To incorporate long-range geometric context, a 2D CNN then downsamples the triplane resolution and flattens it into a sequence of tokens (those triplanes are kinda like images!).
At this point, the sequences are fed into a small 6-layer transformer, which resolves global shape relationships, and a transposed 2D CNN upsamples the tokens back into high-resolution triplanes. To evaluate the continuous feature vector at any arbitrary 3D query location, the network retrieves the corresponding coordinates from the three planes and sums their values. End features are 448-dimensional.
At inference time, it is possible to perform hierarchical clustering to obtain any number of parts at varying granularities. But because the continuous feature vectors encode relative structural relationships, the PartField encoder can be used as an off-the-shelf pretrained network for many other downstream tasks that are remotely connected to parts, e.g., affordance estimation, shape editing, 3D generation.
The encoder architecture is also general and not tied to the specific part formulation. For this reason, NVIDIA presented another paper at CVPR this year using the same model but to compute a convex shape field, which we covered in the recap last week.
Conclusions
To sum up, PartField provides a scalable data and model pipeline for 3D part segmentation. But it does not stop there! Since the feature field is continuous, it is like patches in an image, and we can query for the most similar features across objects or use it for search and retrieval.
This is an intriguing emergent property that develops without explicit cross-shape training and allows for automated co-segmentation and zero-shot point correspondence across distinct geometric topologies. See below for an example.
If you need a 3D encoder for objects and your task requires 3D part awareness, give it a try! The model and code are available at https://github.com/nv-tlabs/PartField and are still very strong today. See you in the next one!
References
[1] PartField: Learning 3D Feature Fields for Part Segmentation and Beyond
[2] Sonata: Self-Supervised Learning of Reliable Point Representations
[3] Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations
[4] Utonia: Toward One Encoder for All Point Clouds