LOCUS

Abstract

Vision-language models (VLMs) ground a referring expression in one of two ways. They decode the box as coordinate tokens, which is precise but takes one sequential step per token, or they regress it from a hidden state with a small head, which takes one step but loses accuracy at high IoU. The two fail differently: a head usually finds the right object but places its edges loosely, while coordinate tokens place edges precisely. We therefore use the regression head as a speculative drafter for the model's own coordinate tokens. LOCUS writes the head's box as coordinate tokens and checks them with the same backbone by Jacobi iteration over a shared image prefix. The fixed point is the model's greedy coordinate-token answer, reached in about 5.60 sequential passes on average instead of 18.46 decoding steps; head and verifier are one set of weights, trained jointly.

On the 8 RefCOCO, RefCOCO+ and RefCOCOg splits, verification raises LOCUS-2B's Acc@0.9 from 54.87 for its own head to 59.13, and the one-pass head is ahead at Acc@0.5 of every published model with at most 3B parameters except Florence-2-L. Against Jag, the strongest open regression-head model, at its own size and training budget, LOCUS ties at Acc@0.5 and, with typical acceptance, gains +4.52 points at Acc@0.9, with paired significance tests. Exact answers come 1.5× faster than vLLM greedy-decoding the same coordinate tokens with the same weights; a pure regression head remains faster still.

Method

Three parts share one image prefill. The image and a fixed prefix are encoded once and cached; each question then runs as a short branch that attends to the prefix and to itself, never to another branch.

1. Draft

Following Jag, a small MLP reads the final state of a learned [BOX] token and predicts the whole box plus an IoU-aware confidence in one pass.

2. Verify

The draft is rounded to the 0–1000 grid and tokenised exactly as the verifier would write it. Each Jacobi pass reads the argmax at every position in parallel, keeps the longest prefix that matches, and replaces the rest. Every pass extends the verified prefix by at least one token, so the fixed point is the model's greedy decode. In fp32 the verified box equals the greedy box on all 505 checked questions. Typical acceptance (α < 1) trades exactness for fewer passes.

3. Train once

Head and verifier are one set of weights, trained jointly on one image prefill per step. Two separately trained models are no more accurate and need two prefills. A dual mode keeps answers to any other question about the image exactly the base model's.

Results

Verification adds most where a head is weakest: the gap between a head and its verified answer opens above IoU 0.8. LocateAnything-3B, larger and trained on far more grounding data, remains ahead at high IoU.

Latency

One RTX A6000, batch 1, 512² inputs, end to end from image file to parsed box, 256 RefCOCO testA questions. Jag, LOCUS and vLLM all use CUDA graphs and GPU preprocessing.

Evaluation on clean rows

The RefCOCO-family splits are drawn per dataset, so 46.9% of evaluation rows ask about an object that is in another dataset's training split. LOCUS trains on no evaluation image, and every comparison is repeated on the clean rows no model has seen. There, every margin over Jag-0.8B grows, and the equal-budget LOCUS-0.8B head moves ahead at Acc@0.5.

Examples

Limitations

• Slower than a pure regression head with equal engineering; verification adds passes in proportion to how often the draft is wrong.
• At 0.8B, exact verification lowers Acc@0.5 relative to the head; the benefit at that size is high-IoU accuracy.
• Trained on the RefCOCO family only. On Ref-L4's Objects365 images, whose targets are much smaller, LOCUS and Jag both fall far below the zero-shot base model.
• In bf16, “exact” holds up to the same rounding that already makes greedy decoding irreproducible across batch shapes.
• Released backbones stop at 2B.

BibTeX

@misc{talha2026locus,
  title  = {{LOCUS}: Locate Once, Check Until Settled. Draft-and-Verify Visual Grounding
            with a Box Head and Coordinate-Token Verification},
  author = {Talha, Muhammad},
  year   = {2026},
  url    = {https://muhammad-talha-ad.github.io/LOCUS/}
}