63. Steerable Visual Representations
Condition vision transformers with natural language on demand
Introduction
Pretrained Vision Transformers (ViTs) are static by design. If we have an architecture and a fixed set of weights, feeding an image through the model yields an identical output every single time.
Still, anyone deploying these systems in the real world eventually runs into the same set of questions: “Why can’t I tell the model where to look? What if the features could be more relevant for my task out of the box? Aaaa, it is so obvious to a human, and also the model should know, damn it!” (representative examples from a CV engineer).
Yes, it happened even to the best of us. But the frustration is over now, thanks to Steerable Visual Representations [1], a recent paper that equips any pretrained ViT with the ability to steer its features using natural language. In simple terms, you can now prompt the visual encoder to focus on any specific concept on demand, without worrying about degrading original feature quality. Let’s see how it works!
Method
Integrating language into the visual encoder must be done without breaking the model’s rich visual prior. SteerViT uses an early-fusion design, allowing language and image tokens to interact directly inside the hidden blocks of the vision transformer rather than processing them independently and merging only at the very end.
Conceptually, this setup reverses the conventional MLLM paradigm: instead of projecting vision representations into language space, we condition the visual encoder on language input.
With no surprise, this is achieved via lightweight, trainable cross-attention layers inserted into every other block of a frozen ViT. For the DINOv2 ViT-B/14 architecture used in the experiments, this setup introduces 6 cross-attention layers for a total of 21M extra parameters, roughly a quarter of the base model size (86M).
To accomplish the steering, the visual patch tokens from the ViT serve as queries, while text tokens extracted from a frozen language model (RoBERTa-Large [2]) provide the keys and values. The output of each cross-attention layer is integrated into the residual stream via a specialized “tanh gate”, initialized with the identity and gradually contributing as the model incorporates language-based clues.
The proxy task that allows training these newly added layers without a heavy computational burden is called referential segmentation. Basically, instead of the classic and expensive pixel-level segmentation, the model is trained to predict which visual patches correspond to a given text prompt, skipping complex decoders in favor of a simpler patch-level classifier.
The training pipeline maps segmentation masks from various datasets into many (text, patch) tuples, which are optimized using a soft cross-entropy loss based on the fraction of patches covered by the target. This objective forces individual visual tokens to determine if their local features align with the text query, contextualized with global information from the regular self-attention layers. More specifically, the dataset comprises 162k unique images and 2.28M image-text pairs.
And that’s pretty much it! It’s a very simple methodology, but the resulting steerable embedding space exhibits fascinating properties across a wide variety of tasks.
Applications
SteerViT isn’t shy about demonstrating how effectively language can guide visual processing. Here are some example tasks where the text conditioning comes in handy:
Conditional retrieval. Default visual backbones tend to group images based on the most dominant object in the scene. On the other hand, SteerViT can successfully shift its focus to secondary, non-salient elements. This flexibility allows the model to achieve high retrieval accuracy in cluttered, multi-object scenarios. Analyzing the embedding space shows that the model reorganizes its [CLS] features into distinct clusters when prompted with specific target objects. Sounds pretty cool to me for real-time data mining!
SteerViT yields object-specific clusters even in cluttered scenes, unlike DINOv2, and exhibits great image retrieval for these secondary objects. From [1]. Targeted attention. To test how well the model localizes specific information, the researchers stitch four independent images together into a single grid to form a mosaic. When prompted with the content of one quadrant, the attention redirects focus entirely to the one containing the relevant target objects, demonstrating precise spatial control over the visual features. This serves more as a confirmation that we can trust localization, and you can use this property for conditioning your retrieval only on certain spatial positions.
Attention maps on a mosaic demonstrate that SteerViT redirects self-attention to the queried concept, while DINOv2 attends to the most prominent objects. From [1]. Anomaly detection and OOD sampling. By prompting the model with abstract phrases like “the anomaly in the object”, SteerViT can localize material defects in classic industrial anomaly detection benchmarks, on par or better than some specialized frameworks. Now, after working in anomaly detection for a few years, I am a bit skeptical of the performance on real-world anomalies, but worth keeping this in mind again for data mining purposes!
Language-conditioning enables zero-shot generalization to out-of-distribution tasks. From [1].
Conclusions
SteerViT is the latest elegant alternative to typical multimodal architectures, proving that we can condition a frozen visual encoder directly on language and focus its representations on specific parts of an image on demand. Finally, we can inject our task biases ahead of time. Isn’t that great?
A nice property is that by scaling cross-attention gates at inference, we get a continuous interpolation between vanilla ViT and SteerViT properties. By testing on fine-grained classification (good global image representation) and dense segmentation tasks (coherent, non-collapsed patches), this method maps models on the Pareto frontier displayed above.
The other cool thing is that since the text encoder remains completely frozen during training and inference, we can precompute and cache the text embeddings for all potential prompt queries ahead of time, adding minimal overhead. For example, inference is 40% slower than a vanilla ViT, but nearly 4x faster than open-vocabulary localization models like SAM3 or any large MLLM, which is a big deal for real-time.
Enjoy testing this model for your custom applications with the demos at https://jonaruthardt.github.io/project/SteerViT/, and see you in the next one!
A model we've all been waiting for since long time indeed. I remember every time saying why visual models based on CNNs were performing so poorly on very trivial tasks for human eyes when I was working on anomaly detection projects. The logic is also quite simple! Great post, thnx :)