Grounding

Clip a continuous pair onto a grid of ids

A language model emits tokens from a fixed vocabulary: a token is one symbol drawn from a finite list, and that list is the vocabulary, so everything the model can output is one of a known, countable set. A coordinate, by contrast, is a real number, a value on a continuous line with infinitely many possibilities between any two points. The standard reconciliation is to bin the image into a grid and let each coordinate be the index of a bin, where a bin is one cell of the grid and its index is the integer that names that cell. A box becomes four integers — two corners, each a row and a column — written as ordinary tokens like <bin_13>. A tiny example makes the snapping concrete: on a 32-by-32 grid, a corner a fraction 0.62 of the way across falls into column 19, and the model emits the token for that column rather than the exact value. The model then does what it is good at: it predicts the sequence of coordinate tokens that follows the referring phrase, exactly as it would predict the next word. The cost is resolution. A grid with $N$ bins per side places a corner within $1/N$ of its true position and no better, so an object smaller than a bin cannot be pointed at precisely, and a box's edges jitter by half a bin. The error is half a bin rather than a whole one because a token stands for the centre of its cell, so the worst a corner can be off is half the cell's width.

The number of bins is therefore a real budget decision, not a detail. More bins localise better but lengthen the vocabulary and give the model more ways to be wrong, because every extra column is another symbol for the softmax to spread probability over and another distinction the model has to learn from data. Fewer bins are easy to predict but coarser than the objects that matter, so a small mug and a distant bird can end up in the same cell. The trade cuts both ways, and neither end is free: precision costs vocabulary size, and vocabulary size costs statistical efficiency, since a model that must tell 256 columns apart needs more examples than one that must tell 8 apart. The demo quantises a point onto an $N \times N$ grid; the error arrow is the distance thrown away by the snap, and the readout names the coordinate token the model would emit.

A box is four tokens; a mask is many

Once a coordinate is a token, detection is a language task. Detection means finding objects in an image and returning a rectangle around each one, a bounding box, together with a label saying what it is; the name survives from the older computer-vision pipeline that did this with purpose-built heads, and the point here is that a language model can do it by emitting tokens instead. A grounded model is trained on text that interleaves an object's name with its normalised corner coordinates, so "mug" is followed by four bins and a score. Normalised means the coordinates are written as fractions of the image's width and height, running from 0 to 1, so that a token means the same relative place whatever the image's pixel size. Concretely, a training string might read the word mug, then the four bin indices for left, top, right and bottom, then a confidence value, and the loss is the ordinary next-token loss on that whole string. At inference — after training, when the model is answering rather than learning — it reads the phrase, emits the bins, and the bins are decoded back into a rectangle. Segmentation adds a third representation: rather than a box around the object, the model emits a polygon or a run of mask tokens that traces its outline. A polygon is a closed shape named by a short list of corner points, so it can follow a boundary that a rectangle cannot, while a mask is a dense per-pixel map saying, pixel by pixel, whether that pixel belongs to the object. The polygon is compact and the boundary is exact; the mask is dense and can represent a shape a box cannot, such as a ring or a curved handle. Both are still token sequences, which is the only property that matters to the trunk.

Confidence is the other half of the problem. A grounded model does not emit one box; it emits a ranked set, each box carrying a score, and a downstream system has to choose a threshold above which a box is trusted. That threshold is a single cut on the score: predictions above it are reported and predictions below it are thrown away. Set it low and the model reports everything that looks a little like a mug, so a fold of cloth or a patch of shadow becomes a false positive; set it high and it misses the one that is half-occluded, meaning partly hidden behind something else so only a fraction of it shows. The two failure modes pull in opposite directions, and the right cut depends on what each mistake costs: extra boxes are tolerable when a person reads the output, while they are not when a machine acts on it. The slider moves the threshold on a fixed set of predictions so the trade between recall and noise is visible, where recall is the fraction of the objects actually present that the model managed to report.

A phrase in, a region out; a click in, a mask out

A referring expression is a phrase that picks out one object by describing it — "the plant in the corner", "the cup behind the book". The phrase does two jobs at once: it names a kind of thing, and it uses a spatial or relational word to choose one instance of that kind. Resolving one requires both language and space, because the model has to identify the candidate objects, use the relation to break the tie, and return the one box the phrase names. If a scene holds three mugs, the word "mug" alone does not single one out; "the mug on the left" does, and only if the model can compare positions. A referring benchmark is harder than a detection benchmark for exactly that reason, where a benchmark is a fixed test set with known right answers: the answer is not any mug, it is the mug the sentence chose, and a model that only detects objects has no way to respect the choice.

Promptable segmentation takes the complementary approach. Instead of a phrase, it accepts a prompt — a point inside an object, a box around it, or a rough mask — and returns a mask for whatever the prompt touches. A click is the simplest kind of prompt, an implicit statement that the point belongs to the thing being chosen. The prompt encoder turns the click into a vector, mapping raw coordinates and a little context into the same numeric form the image side already uses; a small mask decoder combines it with the image embeddings, the per-patch vectors the vision encoder produced, and the result is a region. The power is that one model answers many prompts without retraining: point at the handle and it segments the mug; box the page and it segments the text. The weights are identical across those two calls, which is what makes the interaction feel like one tool rather than a series of separate models. Click anywhere on the scene below and the nearest object is masked; the slider selects an object by its referring expression instead.

The same coordinates, on a screen that happens to be an image

A screenshot is an image and every button on it has a rectangle, so grounding a computer interface is the grounding problem with a different vocabulary of objects: search fields, icons, list rows, submit buttons. The categories change, but the machinery does not, because a button is an object with a boundary like any other. A computer-use agent reads an instruction, grounds the element the instruction names to a box, and emits a click at that box's centre or a type action into it. The pixels are the observation and the coordinates are the action interface: the model never reaches into the underlying program, it only sees the rendered picture and it only acts by aiming at places in that picture. That is why vision-language-action models and GUI agents share their machinery — both turn a sentence into a target location and then into a movement, with coordinate tokens doing the work in between.

What changes is the precision the task demands. A mug box that is a few pixels off is still a correct caption; a submit-button box that is a few pixels off is a click on nothing, or worse on the wrong control. The tolerance for error shrinks because the consequence of error is an action rather than a description. Interface elements are also densely packed and text-heavy, so the model has to read labels as well as locate them: knowing that a rectangle holds a button is not enough if the model cannot tell "Submit" from "Cancel". On top of that, it must be robust to the same element appearing in a different place on the next screen, since layouts shift with window size, theme and content, and a model that memorised absolute positions would fail the moment the window changed. The schematic below marks the groundable elements of a page and highlights the one an instruction selects.

One representation, many interfaces

Grounding also shows up wherever an interface is being automated — document parsing that has to return a field's bounding box, meaning software that reads a form and reports the rectangle around each entry; accessibility tooling that labels controls, so a screen reader can announce what a control is and where it sits; and evaluation of multimodal models on whether they can point rather than merely describe. A model that names the right object in prose but marks the wrong rectangle has failed a different test, and a stricter one. The resolution and merge choices of the token-budget part set the ceiling on all of it: a coordinate cannot be more precise than the grid the image was patched onto, so the finest box a model can ever emit is bounded by a decision made long before grounding is trained.

Further reading

These papers introduce the segmentation model that made prompting standard, the multimodal models that emit coordinates as text, and the studies of grounding an interface as an action space. Read together they trace the same idea from three directions: a promptable segmenter showing how much one model can do without retraining, grounded language models showing coordinates becoming ordinary tokens, and interface agents showing what changes when a coordinate becomes a click.

• Alexander Kirillov, Eric Mintun, Nikhila Ravi and coauthors, "Segment Anything", 2023 — the promptable segmentation model, its prompt encoder and its mask decoder, and the point and box prompts the demo imitates.
• Zhiliang Peng, Wenhui Wang, Li Dong and coauthors, "Kosmos-2: Grounding Multimodal Large Language Models to the World", 2023 — spatial tokens and the grounded box notation this part quantises.
• Keqin Chen, Zhao Zhang, Weili Zeng and coauthors, "Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic", 2023 — referring expressions and coordinate output from a chat model.
• Jianwei Yang, Hao Zhang, Feng Li and coauthors, "Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V", 2023 — overlaying numbered regions so a model can refer to them by id.
• Boyuan Zheng, Boyu Gou, Jihyung Kil and coauthors, "GPT-4V(ision) is a Generalist Web Agent, if Grounded", 2024 — grounding interface elements as boxes and acting on them, and the failure when the box is wrong.

Cheat sheet