How to Evaluate Open Source LLMs Using Your Own Data
To evaluate open source LLM candidates for production, test them on a few hundred real tasks from your own work, scored with metrics that match what "correct" means for that task. Public leaderboards tell you which models are worth testing. They do not tell you which one will do your job best, because your prompts, documents and definition of a good answer are not in any benchmark.
The procedure fits in a week for one use case: collect 100 to 300 real examples, write down what a correct answer looks like for each, run every candidate model and quantization through the same set, and compare quality next to latency and hardware cost. The rest of this post covers each step, the tools that help, and the traps that make evaluations lie.
Why leaderboards are not enough
Public benchmarks measure general ability on fixed question sets. Three things limit how much they tell you:
- Contamination. Benchmark questions get published, copied and scraped, so they can end up in a model's training data. A model that has seen the test scores higher without being more capable. Two surveys on arXiv (Xu et al., 2024 and Cheng et al., 2025) catalogue how often this happens and how hard it is to detect.
- Task mismatch. A high score on graduate-level science questions says little about extracting fields from your insurance claims or answering questions over your internal wiki.
- Format mismatch. Benchmarks use their own prompt templates. Your application has a system prompt, retrieved context, length limits and an output format. A model that follows instructions well in one setup can drift in another.
Use leaderboards to build a shortlist of three to five models that fit your license and hardware constraints. Our guide to choosing an open source LLM for enterprise use covers the license side. Then measure.
Build a golden dataset
A golden dataset is a fixed set of inputs with known good outputs or clear grading criteria. It is the single most useful thing you can build for LLM evaluation, and it keeps paying off: every model upgrade, prompt change or quantization switch gets re-run against it.
Where to get the examples:
- Real requests from logs, tickets or the manual process the model will replace. Synthetic questions written by the team tend to be cleaner and easier than what users actually send.
- A spread that matches production: mostly typical cases, plus the hard ones you already know about (long documents, ambiguous questions, inputs in another language, requests the model should refuse).
- 100 to 300 examples is enough to separate models that differ meaningfully. Below about 50, a few lucky answers can flip the ranking.
For each example, store the input exactly as the application would send it (system prompt and retrieved context included), plus one of:
- A reference answer, when there is one right output (a classification label, an extracted value, a SQL query that returns the right rows).
- A short rubric, when many answers could be right ("mentions the 30-day notice period, cites the correct clause, under 120 words").
Keep the set in version control as JSONL or CSV. Strip or mask personal data before it goes into a repository, and keep a held-out slice that you do not look at while tuning prompts, so you can check you have not overfitted the prompt to the examples you stared at.
Choose metrics that match the task
There is no single "quality" number. Pick the cheapest metric that captures what correct means:
| Task | Metric | How to score |
|---|---|---|
| Classification, routing, yes/no | Accuracy, per-class precision and recall | Exact match against the label |
| Field extraction | Field-level accuracy, valid-JSON rate | Parse the output, compare each field |
| Code or SQL generation | Pass rate | Run it against tests or a fixture database |
| RAG question answering | Faithfulness to the retrieved context, answer correctness | Judge model plus spot checks |
| Summaries, drafts, open answers | Rubric score | Judge model plus human review of a sample |
Deterministic checks (exact match, JSON schema validation, unit tests) should come first wherever they apply. They are free, fast and do not argue. Reach for a judge model only for the outputs where a script cannot decide.
Track failure types as well as scores. "Model B scores 4% lower but never invents a clause number" can be the deciding fact, and a single average hides it.
LLM as a judge: useful, with known biases
Using a strong model to grade another model's answers scales far better than human review. The method was studied in depth by Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", which found that a strong judge agreed with human preferences over 80% of the time, about as often as humans agreed with each other. The same paper documented the biases you need to control for:
- Position bias. When comparing two answers, judges tend to favor one position. Run each pair twice with the order swapped and only count consistent verdicts.
- Verbosity bias. Longer answers get rated higher even when they add nothing. Put length expectations in the rubric.
- Self-enhancement bias. A judge may favor answers that sound like its own. Do not let a model grade itself when it is one of the candidates.
- Limited reasoning. Judges make mistakes on math and logic they would also get wrong when answering. Give the judge the reference answer when you have one.
Practical rules that follow: grade against a written rubric rather than asking "which is better", ask for a short reason before the score, use a fixed temperature of 0, and have a person review 20 to 30 judged examples to confirm the judge agrees with your team before trusting its numbers on the rest.
If your data cannot leave your network, the judge also has to run inside it. A larger open model you self-host can serve as the judge for smaller candidates.
Compare quantization variants as separate candidates
The same model at different quantization levels is effectively a different candidate. A 4-bit version needs roughly half the memory for weights of the 8-bit one, which can mean one GPU instead of two, but the quality loss is task-dependent: often negligible on chat and summaries, sometimes visible on arithmetic, code or long-context retrieval. Our post on LLM quantization explains the formats and where the losses tend to show up.
So run the golden set against each variant you would actually deploy, for example a 32B model at Q4_K_M and Q8_0 next to a 14B model at Q8_0. It is common for a larger model at 4-bit to beat a smaller model at 8-bit on the same hardware, but that is a hypothesis to test on your data, not a rule.
Measure speed and cost next to quality
A model that scores 2% higher but takes three times longer per answer may be the wrong choice for an interactive tool. Record for every candidate:
- Time to first token and total latency at the 50th and 95th percentile, at your expected concurrency.
- Output tokens per second.
- Peak GPU memory, which decides how many GPUs you need.
- Average output length, since a chatty model costs more per request even at the same speed.
Then put quality, p95 latency and monthly hardware cost in one table and pick. The self-hosted LLM cost model turns GPU count into a monthly figure, and the LLM observability guide covers measuring latency the same way you will in production.
Tools that run the evaluation for you
All three of these are open source and talk to any OpenAI-compatible endpoint, so they work against a self-hosted model without code changes.
promptfoo suits application-level tests: your prompts, your test cases, assertions per case. Point its OpenAI provider at your server with apiBaseUrl set to the /v1 root, as its vLLM provider docs show:
# promptfooconfig.yaml
providers:
- id: openai:chat:qwen2.5-32b-instruct
config:
apiBaseUrl: http://llm.internal:8000/v1
temperature: 0
prompts:
- file://prompts/claims_extraction.txt
tests: file://golden/claims.jsonl
defaultTest:
assert:
- type: is-json
- type: llm-rubric
value: "Extracts policy number, claim date and amount exactly as written in the document"
The llm-rubric assertion is promptfoo's model-graded check, and you can set which model does the grading.
lm-evaluation-harness from EleutherAI runs standard academic benchmarks and your own custom tasks. Its local-chat-completions model type targets an OpenAI-compatible chat endpoint (see the API model docs). It is the right tool when you want to reproduce a published benchmark on your quantized variant and see how much it moved. Check the README for the current command syntax, which has changed between versions.
Ragas focuses on retrieval-augmented generation. Metrics such as faithfulness measure whether an answer is supported by the retrieved context, which separates "the model made it up" from "retrieval found the wrong chunk". If you are building the pipeline in our RAG with a local LLM guide, this is the one to add.
A repeatable way to evaluate open source LLM candidates
- Write down the task and what a correct answer looks like, in one paragraph.
- Collect 100 to 300 real examples and their references or rubrics. Hold out 20%.
- Shortlist three to five models that pass your license and hardware filters.
- Deploy each candidate, at each quantization you would run, behind the same OpenAI-compatible API, with the same system prompt and parameters.
- Run deterministic checks first, then the judge on what remains, with swapped-order runs for pairwise comparisons.
- Have a person review a sample of judged outputs and every case where models disagree.
- Record quality, failure types, p95 latency, tokens per second and GPU memory in one table.
- Pick, then re-run the same set on the held-out slice to confirm.
- Keep the set and the config in the repository and re-run them on every model, prompt or quantization change.
Where AI Refinery fits
AI Refinery, a self-hosted LLM platform, makes steps 3 and 4 quicker: you download catalog models into your own storage, pick a quantization level and launch an inference server with one click, and every deployed model gets the same OpenAI-compatible /v1/chat/completions endpoint. Swapping candidates in promptfoo or lm-evaluation-harness is then a change of model name and base URL. Each catalog variant comes with measured size on disk, GPU memory and tokens per second, which fills in the hardware side of the table before you run anything.
What the catalog benchmarks do not tell you is task quality on your data. That part is still yours to measure, with the golden set above. If you want to screen dozens of models from the wider Hugging Face hub before shortlisting, run the tools above against your own test servers first, then deploy the finalists. The home page shows how the private model catalog and deployment work.