UXBench

What it measures

Benchmarks usually test capability: can the model solve the problem, write the code, pass the exam. UXBench tests something closer to the product: did the person who asked come away satisfied.

The authors call it the first user-centric benchmark grounded in real user feedback. Its goal is to extend model evaluation from what a model can do to what users experience.

How this test measures user experience

Intent and satisfaction come from real users

Every conversation carries the thumbs up or thumbs down a Yuanbao user pressed on that reply. A verdict only counts as right when it matches that click, so a model is scored on whether it understood what the user wanted.

Labels show what satisfies and what frustrates

The authors tag disliked replies by 10 failure dimensions, such as verbosity, task incompleteness and intent misunderstanding, and liked replies by 8 success patterns. These labels work like usability heuristics: a model that notices what users notice, including replies that are correct but too long or off the point, has the judgment that established design patterns encode.

Both halves count equally

The overall score averages liked replies recognised and disliked replies caught, so calling everything good earns 50%. The repository describes the aim as "a well-calibrated model that can accurately identify unsatisfactory responses".

What it does not show

UX Judge asks a model to judge replies. Whether its own replies satisfy users is Task 2, which this site does not run.

Who made it

UXBench was created by Mengze Hong and 15 co-authors. The data comes from Tencent Yuanbao, an AI assistant used in China. The paper was first posted on arXiv on 8 Jun 2026 and was accepted to the main conference of EMNLP 2026.

How the data was built

The authors started from 71,584 Yuanbao conversations in which a user pressed thumbs up or thumbs down. The repository describes six stages that turn them into test items: signal extraction, de-duplication and quality filtering, a miner agent, a judge agent that scores each conversation on five axes, a full quality check, and sampling into the final test set.

The result is a large test set across 8 interaction scenarios and 83 domains, from factual questions to creative writing and emotional support.

Verbosity is the most common reason users disliked a reply, at about a third of cases. A longer answer is not a better answer when the user wanted a short one.

The three tasks

Tasks 2 and 3 are scored by the authors' own reward model, which is not publicly released, so this site covers UX Judge only.

UX Judge in detail

The model receives the conversation history, the user's last message and the assistant's reply, plus a short checklist. It answers with one word: good if the reply would satisfy the user, bad if not.

How it is scored

Good-Acc is the share of liked replies the model called good. Bad-Acc is the share of disliked replies it called bad. The main score, Avg-Acc, is their average, so a model that calls everything good scores 50%, the same as guessing.

Avg-Acc = (Good-Acc + Bad-Acc) ÷ 2

What the authors found

• Spotting a bad reply is the hard part. Every model scores higher on liked replies than on disliked ones, by about 20 points (GPT-5.2) up to 88 points (DeepSeek V3).
• General ability is a weak guide. The authors report a correlation of only 0.31 between general capability rankings and UXBench rankings.
• Judges favour their own family. Models judging replies tend to rate replies from their own model family higher.
• Repairing a failed conversation is rarely achieved. The best recovery rate on Task 3 is 12.8%, by Claude Opus 4.6.
• Predicting user feedback can be learned. The authors present it as a skill models can be trained on, not a fixed trait.

Licence and citation

UXBench is released for research use only. Its licence forbids redistribution, derivative datasets and commercial use. This site publishes only scores and summaries: no conversations, test files or answer keys.

@misc{hong2026uxbench,
  title         = {UXBench: Benchmarking User Experience in AI Assistants},
  author        = {Mengze Hong and Xia Zeng and Zeyang Lei and Sheng Wang and
                   Chen Jason Zhang and Di Jiang and others},
  year          = {2026},
  eprint        = {2606.09570},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.09570}
}

Sources: UXBench repository, arXiv abstract, official leaderboard site.