Pushback · a sycophancy eval for vision-language models
The eye that blinks first
Ask a model whether a person is looking into the camera. Then disagree with it.
–
Scroll for the method and the numbers ↓
The setup
Two turns, one image, one disagreement
Each conversation shows the model one photograph and asks whether the person is making eye contact with the camera. Whatever it answers, the second turn pushes back toward the opposite answer, at one of four levels of pressure. The control (C0) just asks again, with no claim, so that the effect of disagreement is separated from the effect of being asked twice.
- 1Photo with known gaze angle
- 2VLM: “Eye contact? Yes or No”
- 3Turn-1 answer, scored
- 4Pushback (C0 to C3)
- 5Turn-2 answer, scored
- 6Held, flipped, fixed or stuck
Ground truth
Ground truth you can't argue with
The Columbia Gaze Data Set photographs 56 people at 5 head poses, each looking at 21 fixed targets. Whether someone is looking into the lens is set by where they were told to look, not by an annotator's opinion. Eye contact means 0° vertical and 0° horizontal gaze, at any head pose.
Photos are shown only as drawings. The dataset licence is non-commercial research, and no image from it appears on this site.
Turn 1
Can they see it at all?
Before any pushback: how often is the first answer right? Capitulation can only be measured on answers that start out correct, so this sets how much evidence each model gives us.
The flip
Where the answers go
Each dot is one conversation. Left: was the first answer right? Right: was the final answer right? Green dots kept a correct answer. Red, hatched dots abandoned one. Switch the pressure and watch the red stream.
Pressure ladder
More pressure, more flips?
Capitulation rate (correct answers abandoned) at each level of pressure, with 95% cluster-bootstrap intervals. The shaded band is the C0 re-ask baseline: anything above it is caused by the disagreement itself.
Asymmetry
Giving up truth vs. finding it
The same pushback can also fix a wrong first answer. If the model were simply updating on the user's claim, it would fix wrong answers as readily as it abandons right ones. Gold: wrong answers corrected. Red: correct answers abandoned.
Difficulty
Where it breaks
Capitulation by head pose and gaze bin. NEAR images (gaze 5° off the lens) are the hardest negatives; turned heads make every judgement harder. Cells with fewer than 5 correct first answers are left blank.
Uncertainty
How sure are these numbers?
We resample people, not images, because photos of the same person aren't independent. Each resample draws people with replacement, takes all of their conversations, and recomputes the capitulation rate. The middle 95% of those rates is the interval.
Transcripts
Read the conversations
Every conversation, filterable. Images are shown as drawings of the head pose and gaze angle only.
Softening
Softening without flipping
Some correct answers survive the pushback but arrive wrapped in doubt (“I can't be certain, but…”). Of the correct first answers: kept firmly, kept with a hedge, or abandoned.
Limits
What this does and doesn't show
-
Posed lab photos
Controlled lighting, fixed distance, cooperative subjects. Real images are messier.
-
A binary task
Yes or No leaves no room for partial credit or calibrated uncertainty.
-
Two turns
Longer arguments may push harder, or let the model recover. Neither is measured here.
-
Wording
Each condition uses one fixed message. A different phrasing could move the numbers.
-
Small open models
The main models are 3–7B open-weight VLMs that fit a free GPU, not frontier systems.
-
Possible training exposure
The dataset is public; a model may have seen it. That would affect turn 1, not the pushback.
Paper, code, citation
Are You Sure?
Vision-Language Models Abandon Correct Visual Judgements Under User Pushback