Vibe coding · VCL VibeBench

How to Benchmark AI Models Yourself in 2026 (VCL VibeBench Fun, Dev, and Score)

Stop trusting slides. Clone the open suite, paste Fun prompts in any chat, then run Dev and Score with your OpenRouter key — and learn what a 9/12 Score log actually means.

~11 min read

Vendor slides and third-party charts answer “who looks smartest on a fixed exam.” VCL VibeBench — short for Vibe Coder's Life VibeBench — asks a different question: which model helps you with everyday practical tasks this week?

The public repo is kondasviktor/vcl-ai-model-arena. Three suites ship: Fun, Dev, and Score. There is no LLM judge and no blended “best model” score across suites. We do not run visitor evals on our credit — BYOK (bring your own key).

Open VCL VibeBench on GitHub →

What shipped

VCL VibeBench is an open, forkable set of practical prompts with deterministic asserts where a single answer exists. Exploratory prompts stay qualitative — you compare side by side.

Methodology: docs/methodology.md. We cite lab benches when useful. We do not wrap SWE-bench, Terminal-Bench, CursorBench, or Artificial Analysis indexes.

Try Fun in two minutes (no terminal)

Open any chat UI. Copy a prompt from the README Fun section. For scored items, the expected answer is noted so you can check yourself.

Example scored prompt (expect 3, and only the number):

How many times does the letter R appear in the word 'strawberry'? Answer with just the number.

That “just the number” rule is why a model can know the answer and still fail a Fun scored assert — verbose chain-of-thought fails a regex. We walked that split on Kimi K3 in the July Fun + Dev write-up.

Fun has 10 prompts (mix of scored and exploratory). Exploratory rows are for judgment, not a fake numeric winner.

Clone and run the CLI (BYOK)

BYOK means you put your API key in .env. Default provider is OpenRouter — any model ID you set in MODELS=. Node 20 or newer (we use 24 locally). Promptfoo is pinned at 0.121.19.

1. Clone, install, set the key:

git clone https://github.com/kondasviktor/vcl-ai-model-arena.git
cd vcl-ai-model-arena
cp .env.example .env          # set OPENROUTER_API_KEY
npm ci

2. Get a key at openrouter.ai/keys, add a little credit, then:

OPENROUTER_API_KEY=sk-or-v1-...

3. Run the suites (pick any OpenRouter IDs):

# Fun — same prompts as the copy-paste section
MODELS=google/gemini-3.6-flash,moonshotai/kimi-k3 npm run eval:fun

# Dev — coding / debugging
MODELS=google/gemini-3.6-flash,moonshotai/kimi-k3 npm run eval:dev

# Score smoke = 2 tasks, cents
npm run eval:score:smoke

# Score full = 12 unit-tested JS helpers
MODELS=google/gemini-3.6-flash npm run eval:score

# Cheap Fun sanity check (two inexpensive models)
npm run eval:smoke

4. Human summary + local UI:

npm run results:latest
npx promptfoo view

Optional: confirm IDs exist on OpenRouter with MODELS=… npm run validate:models.

Optional: Hetzner Experiments (free while experimental)

OpenRouter stays the default. You can point the same Fun/Dev/Score runner at Hetzner Experiments Inference — an OpenAI-compatible API — with your own Experiments token (not a Hetzner Cloud console token).

1. Sign in at experiments.hetzner.com → Apps → Inference → Create API Token.
2. Put it in .env as HETZNER_INFERENCE_API_KEY.
3. Set PROVIDER=hetzner.

PROVIDER=hetzner npm run eval:hetzner:smoke

Allowlisted IDs (must match Hetzner docs): Qwen 3.6 35B-A3B, Kimi K2.7 Code, DeepSeek V4 Flash, GLM 5.2. Free only while Hetzner marks Inference experimental — no SLA; they may bill later. Rate limits are tight (~10 req/min). This is optional BYOK, not an official maintainer matrix. Our 2026-08-12 Fun smoke on Qwen completed end-to-end (5/10 — empty/thinking completions, not a missing key). Docs: experiments.hetzner.com/docs/inference.

What it costs

• eval:smoke / eval:score:smoke — cents on typical OpenRouter chat prices
• Full Fun + Dev text evals — still cents-to-low-dollars for a small MODELS= set
• Full Score (12 tasks) — meant to stay around ≤ $2 per model

Set a spend limit on the OpenRouter key. We do not bill you and we do not proxy your requests.

How to read Score

Score is 12 short, original JavaScript helpers. Each prompt asks for one named function in a markdown fence. A sandbox extracts the code and runs a handful of hidden unit tests. Pass/fail only.

9/12 is not 75% intelligence. It is nine everyday helpers that compiled and matched examples, and three that did not, on that run.

The 12 tasks

Harness self-check (no API): npm run test:score:harness. Task text: tests/score/promptfooconfig.yaml.

Worked example: Cursor Grok 4.6 scored 9/12

OpenRouter x-ai/grok-4.6 is the same weights as Cursor Grok 4.6. A Score row on that ID is a Score row for Cursor Grok 4.6. It is not a test of Cursor’s agent/tool loop — Score is a single completion: write a function, no tools.

Maintainer run 2026-08-13 (promptfoo eval-5F6-2026-08-13T14:01:16):

Recorded JSON stays 9/12. Fun/Dev maintainer tables on that results page are still the 2026-07-28 Grok 4.5 matrix — we did not re-run Fun/Dev on 4.6.

The three fails were parseQuery, deepGet, and titleCase. The log said Unexpected token / Invalid or unexpected token. That means the tests never ran.

Grok opened a code fence in the thinking dump, cut off mid-line, then wrote a finished function in a second fence. The old extractor compiled the truncated draft. The finished functions look correct; re-grading those same replies with “last valid function” is 12/12. We do not rewrite the committed promptfoo JSON — results/latest.md explains the recorded 9/12 per task.

What this means: Grok 4.6 can write these everyday JS helpers. Score punished a thinking-model habit (leaking a broken half-fence), not a proven “cannot title-case a string” miss.

FAQ

What is VCL VibeBench?

An open, forkable set of practical prompts for comparing AI models, by Vibe Coder's Life. Fun + Dev + Score. Deterministic asserts where a single answer exists. No LLM-as-judge. No blended winner across suites.

Does 9/12 mean the model is 75% as smart?

No. It means nine of twelve JavaScript helpers compiled and matched hidden examples on that run. Syntax-error fails mean the extracted snippet never ran.

Is this SWE-bench / CursorBench / an Artificial Analysis index?

No. Score is Aider-lite in protocol (write code, tests grade) with original VCL tasks. We cite those labs in posts when useful. We do not wrap or re-run them here.

Is OpenRouter x-ai/grok-4.6 the same as Cursor Grok 4.6?

Same weights, yes. Score still does not test Cursor tools, rules, or the agent loop.

Do visitors run on your credit?

No. BYOK. Your OpenRouter key (or optional Hetzner Experiments token) stays in your local .env.

Where are the official numbers?

Committed JSON + .meta.json in the public repo. The human summary is results/latest.md. Promptfoo share links can expire; git is canonical.

How you can help

If VCL VibeBench is useful, these five things actually move the project — in order of effort.

1. ⭐ Star the repo if an honest, re-runnable comparison is useful when a new model drops. Stars help others find it, and they tell us it is worth maintaining.
2. 🍴 Fork it when you want your own suite, extra prompts, or a private comparison for your team — that is the point of open fixtures.
3. 💻 Run the CLI matrix (BYOK) if you can. Paste-into-one-chat is a quick vibe check; a local run with your OpenRouter key compares many models on the exact same prompts in one go.
4. ✉️ Subscribe to the Vibe Coder's Life newsletter for result write-ups when we re-run after notable model releases — no need to live in GitHub to stay in the loop.
5. ☕ Buy Me a Coffee if you want to support maintenance — optional tips help keep VCL VibeBench updated when new models drop.

Star VCL VibeBench on GitHub →

VCL VibeBench is also on our Tools page. The Kimi K3 Fun + Dev story is still here: Did Kimi K3 Pass the Strawberry Test?