autotrust/JEV-27B-VL

TL;DR: JEV-27B-VL bolts a "System 1" typed-decision head onto a 27B Qwen VLM: you POST a state (text and/or images), a question, and up to 256 options, and get a calibrated probability per option from a single forward pass — no generation, no parsing. The interesting part is what a fast, calibrated, non-generative decision primitive buys you when you put it in a loop: zero-shot robot-arm control from camera images, screenshot-driven browsing, and cover-image video recommendation matching collaborative filtering trained on 59k users' logs — all with no task training. At 1.28M downloads and trending #3, this interface looks like it's becoming a category.

How it works

The package is the unmodified Qwen3.8-27B-VL backbone ("System 2", normal chat/reasoning) plus a LoRA adapter and a decision head ("System 1"). System 1 formats your request into a fixed template ([kind] / [state] / [question] / [options]), runs one forward pass, reads the logits of the option-label tokens (A–P for the first 16 options, then Q–Z and single-token two-letter labels beyond that, zero-shot), applies a learned per-option bias and per-kind temperature, and returns a softmax. Three kinds: yes/no (noul), 0–5 rating (score), and 2–256-way choice. Images go inline in state as data URLs.

A serve_decide.py wrapper adds POST /v1/decide to the standard vLLM OpenAI server; the README also shows how to do the math client-side against plain vllm serve. Decisions on 250K-token prompts work (needle tests: 20/20 at every length, ~29 s at 250K on a B200).

The demos are the argument

The design claim is: don't ask a VLM to emit motor commands or click coordinates — ask it tiny typed questions inside a loop you control. Evidence from the README:

• Robot arm (MuJoCo): per step, two decisions from a top camera — target left/right and above/below the gripper — with step-halving on answer flips. 75% of 20 random pick-and-place scenes; every grasped cube landed in the tray; ~240 ms/decision. Asked instead to pick one of 8 motor commands directly: 0 of 10. That ablation is the most honest and useful number in the card.
• Computer use: numbered boxes + element text on real headless Chromium; System 1 picks the next click. 95% of 60 multi-step tasks at ~0.26 s/click — but 10% from pixels alone, so it needs an accessibility-tree-like crutch.
• Zero-shot recommendation (MicroLens-100k): P(user watches) from 5 watched covers + 1 candidate cover. AUC 0.727 vs 0.728 for item-CF trained on 59,045 users, and better HR@5 (0.590 vs 0.490). Covers beat titles by +0.078 AUC. Genuinely interesting for cold-start, though it's 200 users, one dataset, offline.
• Judging: 78.3% on VL-RewardBench (above its 2025 leaderboard), top of Plan-RewardBench's table (73.2%), precision above all 16 AgentRewardBench judges at matched recall (AUROC 0.91, 1,302 trajectories in ~4 min on one GPU). On the newer MMRB2 it sits at GPT-4.1 level, below GPT-5/Gemini 3 Pro — a fair self-report.

Also notable: 256-way choice works zero-shot with unseen labels (CLINC150 all-150-options: 89.5% names only, 93.8% with one-line descriptions), and the prompt-engineering section is unusually empirical — the only thing that measurably helped was one-line "use when" disambiguators; JSON vs text, wording, instructions all noise.

What to expect

Apache-2.0: unchanged Qwen weights + adapter + decision head. You need one 80 GB GPU (52 GB weights; 128K context fits, 256K wants more). Quirks: --max-num-seqs 8 is mandatory (a vLLM LoRA bug corrupts probabilities above that for this multimodal class), serve_decide.py needs a specific September-2026 vLLM dev build, and ~7% of 16-option answers flip with option order. Image-decision calibration is unmeasured (the head was trained on text); robot and recommendation evals are small and author-reported. The named backbone ("Qwen3.8-27B") and several cited benchmarks aren't independently verifiable from the card alone. System 2 is just vanilla Qwen (HumanEval 78.0%, unchanged).

Try it

hf download autotrust/JEV-27B-VL --local-dir JEV-27B-VL
bash JEV-27B-VL/serve.sh   # vLLM on :8000, needs ~80 GB GPU

curl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{
  "kind": "choice",
  "state": "Customer: my card was charged twice for one coffee.",
  "question": "Which team should handle this?",
  "options": ["billing", "shipping", "tech support"]}'
# → {"probabilities": [0.9969, 0.0000, 0.0031], "choice": "billing", ...}

For images, make state a list mixing strings and {"image": "data:image/jpeg;base64,..."} parts. Demo code lives in the JEV-9B repo (vl/demos/) and JEV-27B-DEMO; per-episode results in reports/demos/. Worth an afternoon if you have any classification, routing, reranking, or judge workload — the calibrated-probability-per-option interface is the thing to evaluate, not any single benchmark number.