Quorum:
ask the jury.

Every message below is a real test sentence from a public benchmark, with the dataset's own answer attached. Pick a domain, draw a sentence and hear the verdict. You will see when the jury is right, and when it isn't.

We evaluate on the full official test set of seven public intent benchmarks, covering banking, smart assistants, voice commands, everyday tasks and customer support. The zero-shot configuration is fixed and identical on every benchmark. Each number below can be recomputed from the repo, including our Jev measurements.

Zero-shot, seven benchmarks

The same three-model jury, with no training. “Names” matches each message against bare label names. “Descriptions” matches against a one-line description per label: a plain rephrasing anyone can write, not taken from the data.

Against Jev

Jev is a closed, commercial calibrated-decision model with no official public Banking77 number. We measured it ourselves through its API (model jev-1.13.0), on the same test sentences and with the same label text, after all of our configurations were frozen. Its outputs were used only for evaluation, never for training or tuning.

At 24-shot, the code in the repo scores 0.932 against 0.924 for Jev in the public reproduction's setup, which is not a significant difference (p = 0.066). A pre-registered run scored 0.936 (p = 0.014 against that setup). Most of that edge comes from how we retrieve the 24 examples, not from the models: given our retrieved examples, Jev rises to 0.935 and the two tie (p = 0.87). Jev's run in the reproduction's setup gives 92.40%, exactly the reproduction's published figure.

Zero-shot, Jev is clearly ahead: 0.803 against 0.716 for our jury with an NLI juror never trained on Banking77 (0.774 with the default juror, which saw Banking77 training data). On the four benchmarks no juror has seen, the gap ranges from 3.4 points (SNIPS) to 22.7 points (CLINC150), all significant.

The jury beats its own jurors

On the four benchmarks where no juror has any known training exposure (CLINC150, MTOP, SNIPS, Bitext), the jury beats its best single member in all eight conditions, names and descriptions, even with an NLI juror never trained on intent data. On Banking77 that does not hold: with that juror the jury is level with its best member using names (0.690 vs 0.700, not significant) and below it using descriptions (0.716 vs 0.738).

No single small model is a great zero-shot intent classifier. Three complementary ones disagree in useful ways. Each juror reads the message against every candidate label, phrased as “This message is about {label}.”

PrismNLI-0.4B scores entailment; bge-large-en-v1.5 and bge-base-en-v1.5 score cosine similarity to the same verbalized labels. Each juror's scores become a probability distribution (softmax at temperature 1). Quorum takes the geometric mean, the mean of log-probabilities, and renormalises. The top label is the verdict; its probability is the confidence. No labelled data, no training, no per-task calibration.

Descriptions, not names

A model that has never seen a dataset gets little signal from a short label name. A one-line description gives every juror more to match against, at no cost: the same three models, no extra code. The gain scales with how opaque the names are: +0.091 on SNIPS, whose names are short and ambiguous, against +0.018 on Banking77, whose names are already fairly descriptive.

The 24-shot pipeline

At the 24-shot setting of the public Jev reproduction, each query retrieves 24 labelled examples with BM25 and no weights are updated at test time. Two mechanisms read the same 24 examples: an in-context reader, Qwen3-4B with a small LoRA adapter trained only on other public intent datasets with Banking77 held out, which scores every class in one forward pass; and a frozen bge-large nearest-neighbour vote. A geometric mean combines them. The adapter is gated on Hugging Face. This pipeline is not the CPU jury: its reader has 4B parameters, and a GPU is recommended.

Keeping the numbers honest

• Only full official test sets are reported; small smoke tests overstate.
• The zero-shot NLI juror is built on a model whose training mix includes the Banking77 and MASSIVE train splits (not their test sentences), so its zero-shot results on those two datasets are not fully held out.
• Comparisons are matched-budget: zero-shot with zero-shot, 24-shot with 24-shot.
• Systems predicting on the same items are compared with a paired McNemar test.
• In the pre-registered runs, every rule, weight and temperature was chosen on held-out data, never on test; earlier exploration that did score on the test set is listed under Limitations.
• Jev is measured on exactly our test items and label text, after our configurations were frozen.

What did not work

• More training data saturates. Scaling the reader's fine-tuning from ~23K to ~60K examples did not raise zero-shot accuracy (flat around 0.70).
• A bigger embedder does not help. Swapping bge-large for Qwen3-Embedding-8B made the nearest-neighbour member weaker on this task.
• Reranking the top-k does not help. A cross-encoder reranker and a small generative reader over the top-5 both fail to beat the ensemble's own ranking.

The zero-shot jury is open source and free to self-host today. It is three models under 1B parameters that run on a CPU, so your data never leaves your machines; it is less accurate than Jev (see Limitations). We are also preparing two ways to run it without the plumbing. Joining a waitlist is free and commits you to nothing; we will email you when there is something to try. The 24-shot pipeline (4B reader) is in the repo.