← Experiments

Guided search but for vision-language models

Read paper

TL;DR. I ran three follow-up search experiments, each one with no color to guide attention. The ideas for these experiments come from my postdoc advisor Jeremy Wolfe. Running them on the two flagship thinking models: a T-among-L search makes GPT-5.5 deliberate like a serial searcher while Claude treats it as a pop-out; both count small numbers of T targets among L almost perfectly and both find the same hard case in the classic asymmetry, confirming there is no tilted bar among vertical ones, but GPT-5.5 shows it in effort and Claude in accuracy that falls to chance.

I sent my last experiments to Jeremy Wolfe, who ran the lab where I did my postdoc and has spent his career on visual search in humans. He wrote back with three follow-ups, and each one strips away color as a feature from my first set of expeirments.

That color feature matters because of Jeremy's own theory. Guided Search says human attention is steered toward likely targets by a few basic features, color chief among them. My first conjunction target was a red T among red Ls and black Ts, so color still narrowed the field by half. These three tasks take color away entirely. Here is one display from each.

Three example displays. Left: a single black T among black Ls. Middle: several black Ts among black Ls. Right: one tilted bar among vertical bars.
Figure 1. The three follow-up tasks, all in one color. Find a black T among black Ls; count every black T; find the one tilted bar among vertical ones.

1. T vs L, with no color to help

A T and an L are the same two strokes joined differently. With every letter the same color, neither color nor a single shape feature finds the target, so the only cue is the spatial arrangement of each letter's parts, which is what makes the search serial. The prompt was one line, the image plus "Is there a letter T in this image? Answer with exactly one word: yes or no." I built 200 displays across set sizes 4, 8, 16, 32, half with a target. Jeremy's prediction: effort should climb with set size like conjunction search, not stay flat like a pop-out.

Both models answer correctly almost always (Claude perfect, GPT-5.5 a single slip to 96% at 32 items), so the story is in the effort. The figure below shows the target-absent trials, where the search has to be most thorough; toggle the button to drop in the feature and conjunction curves from previous experiments.

Reasoning tokens by set size for Claude and GPT-5.5 on the T-vs-L task. Claude stays flat near four tokens; GPT-5.5's target-absent line climbs from about 80 tokens at four items to over 400 at thirty-two.
Figure 2. T-vs-L reasoning effort (dark green), target present (solid, filled) and absent (dashed, open). With the overlay on, GPT-5.5's T-vs-L line rides up with the steep gray conjunction curve, the serial signature Jeremy predicted, while Claude's stays pinned to the flat green feature reference: it treats the search as a pop-out and spends three or four tokens no matter the set size. Shaded bands are 95% confidence intervals; Claude's are tiny because its effort barely varies.

GPT-5.5 does exactly what Jeremy expected. Its target-absent effort rises from about 80 tokens at four items to over 400 at thirty-two, more checking for more items and the most when ruling the target out. Claude answers in three or four tokens at every set size and gets it right anyway; its adaptive thinking simply judged the task not worth deliberating over.

The finding. The same no-color hunt is a serial search for one model and a reflex for the other. That is the neat part: a task built to be hard exposes how differently the two models decide what is worth thinking about.

2. Count all the Ts

The second task swaps detection for counting. Each display has one to four black Ts among black Ls (so there is always a target), with the prompt "How many letter Ts are in this image? Answer with a single number." I built 300 displays crossing the number of targets with set size (8, 16, 32). Jeremy's prediction draws on a well-studied human effect. When people search for several targets at once, finding one makes them worse at finding the rest, an error called a subsequent search miss (SSM, once known as "satisfaction of search"); it shows up in the lab and in radiology and baggage screening alike. So the prediction was that the models would undercount, missing later targets as the display filled up.

It did not happen. Both models track the true count almost perfectly, GPT-5.5 exact on 96 to 100% of trials at every set size and Claude on 88 to 100%. The reported counts land on the true number rather than sagging below it; if anything Claude drifts a hair high in the most crowded displays, the opposite of a miss. A human searcher, prone to SSM, would be expected to bend the other way (Figure 3, left, sketches that predicted human pattern; I did not collect human data here).

What the count costs them is effort, not accuracy. Switch the tabs below between the two: counting accuracy (reported vs true count, with 95% CIs) and the reasoning tokens each cell took. Effort climbs with both clutter and target count, and GPT-5.5 runs past a thousand tokens per trial on the most crowded displays, an order of magnitude more than Claude.

Three panels plotting mean reported count against true count for set sizes 8, 16, 32, with a dotted identity line. The left panel, Humans (predicted), shows curves bending below the identity line, more at larger set sizes, the subsequent-search-miss pattern, labeled illustrative and not measured. The Claude Opus 4.8 and GPT-5.5 panels show measured data with 95% CI error bars sitting on or just above the identity line.
Two heatmap grids, Claude Opus 4.8 and GPT-5.5, of mean reasoning tokens by set size (rows: 8, 16, 32) and number of Ts (columns: 1 to 4). Values rise toward more targets and larger set sizes; GPT-5.5's are far higher, peaking near 1025 tokens at set size 32.
Figure 3. Find-all, both views. Counting accuracy: mean reported count vs the true count, per set size. The left panel is the pattern predicted for humans by subsequent search misses (illustrative, not measured); the two model panels are measured, with 95% CIs and the dotted identity line (below it means undercounting). Reasoning effort: mean tokens per trial in each set-size by target-count cell; color is scaled within each model, so read the numbers for absolute values.

The finding. The predicted breakdown never came: these models count by working through the display, not by taking it in at a glance. The subsequent-search-miss errors that trip up humans are exactly the failure mode they avoid, and they pay for it in tokens instead.

3. The search asymmetry

The last task is where human vision is most lopsided. Finding a tilted bar among vertical ones is fast, the tilt pops out; finding a vertical bar among tilted ones is slow, even though the displays are mirror images. A model that merely mimics search has no reason to find one direction harder. I built 400 displays, both directions, four set sizes, half with a target, asking either "Is there a tilted bar?" or "Is there a vertical bar?" The hard, revealing trials are the target-absent ones, where the model has to rule the odd item out.

Both models do show an asymmetry, both land on the same hard case, and they reveal it in different currencies. The hard case is the target-absent tilted-among-vertical display: confirming there is no tilted bar when every bar is vertical.

Grouped bar chart of accuracy on target-absent trials, Claude and GPT-5.5 side by side. Claude's tilted-among-vertical bar is at 0.51, near chance; its other bar and both GPT bars are near 1.0.
Line chart of mean reasoning tokens on target-absent trials by set size, Claude and GPT-5.5 together. GPT-5.5's tilted-among-vertical line climbs from about 180 to 380 tokens; its other line and both Claude lines stay low and flat.
Figure 4. The asymmetry on target-absent trials, both models on each chart. Switch tabs to change the currency: in accuracy, only Claude breaks, falling to 51% when asked to rule out a tilted bar among vertical ones; in reasoning effort, only GPT-5.5 works harder, and only in that same hard direction. Both point to the same case. Error bars and shaded bands are 95% confidence intervals.

GPT-5.5 keeps its accuracy perfect everywhere but spends far more reasoning to clear the hard direction, while the mirror direction stays cheap and flat. Claude, which again barely deliberates, pays in errors instead: its accuracy on those trials falls to 51% against 96% for the mirror. It hallucinates a tilted bar that is not there.

One honest caveat: the two directions also use different questions ("is there a tilted bar?" versus "is there a vertical bar?"), so part of the effect could be the word and not just the shape. Given that these models lean so heavily on language, that is a real possibility, and a cleaner follow-up would keep the prompt fixed and vary only the display.

The finding. Both models are lopsided like us, but lopsided the wrong way: for people a tilted bar pops out, so its absence is easy to confirm, yet that is the one thing the models find hardest. A familiar behavior running on unfamiliar machinery.

Where this leaves things

Two of Jeremy's three predictions came out differently than expected, which is where the interest is. The no-color hunt makes GPT-5.5 search serially but makes Claude look like it is not searching at all; counting, predicted to break, held up; and the asymmetry is real in both models but rotated away from the human one. The usual caveat holds, 25 trials per cell, so these are shapes and directions, not precise measurements. But the method keeps earning its keep. Thank you, Jeremy. Code, stimuli, and analysis are on request.