carlos-dev-research

A lightweight harness for HealthBench evaluations

Grading language models on realistic health conversations with physician-written rubrics, through any OpenAI-compatible API.

3 min read

  • llm-evals
  • healthbench
  • python
  • openai-api

HealthBench is OpenAI's benchmark for how well language models handle realistic health conversations, with patients and with clinicians. Instead of multiple-choice questions, each of its 5,000 conversations comes with rubric criteria written by physicians. A model's reply is judged against each one.

I wanted to run it myself, on open models and cheap endpoints, with every judgement saved so I could read why a reply scored the way it did. This project is that harness: small, reproducible, and provider-agnostic.

Code and results: carlos-dev-research/evals-healthbench.

How it works#

flowchart LR
  A[HealthBench record<br/>conversation + rubrics] --> B[Model under test<br/>writes the next reply]
  B --> C[Grader model<br/>checks each criterion]
  C --> D[results.jsonl]
  D --> E[Score]
  1. Generate. The model under test receives the conversation and writes the next assistant turn (temperature 0.3).
  2. Grade. The grader model is called once per rubric criterion and must answer whether the reply meets it.
  3. Score. The criteria's points are combined into a score for each conversation, then averaged across the run.

Both models are reached through any OpenAI-compatible API, such as Groq, Ollama, vLLM or OpenAI itself. Switching provider or model is a .env change:

# Groq here; http://localhost:11434/v1 for a local Ollama
BASE_URL=https://api.groq.com/openai/v1
MODEL_TEST=openai/gpt-oss-20b
MODEL_GRADER=openai/gpt-oss-20b

Rubrics and scoring#

Criteria carry positive or negative points. For example, +8: mentions increased risk of falls, or −5: recommends an unsafe dose. With meti∈{0,1}\text{met}_i \in \{0, 1\} from the grader, each conversation scores:

score=∑imeti⋅pointsi∑imax⁡(0, pointsi)\text{score} = \frac{\sum_i \text{met}_i \cdot \text{points}_i}{\sum_i \max(0,\ \text{points}_i)}

This is points earned, with triggered penalties subtracting, over the maximum achievable. A single conversation can therefore score below zero. The run's score is the mean across conversations, clipped to [0,1][0, 1].

Getting reliable JSON from the grader#

The grader receives the conversation, the reply and one criterion through a prompt template (GRADER.md). The template also handles a subtle point: for a negative criterion, the grader must report whether the undesirable behaviour occurred, not whether the reply was good.

Answers are constrained with a JSON schema:

response_format={
    "type": "json_schema",
    "json_schema": {
        "name": "grading_result",
        "schema": {
            "type": "object",
            "properties": {
                "explanation": {"type": "string"},
                "criteria_met": {"type": "boolean"},
            },
            "required": ["explanation", "criteria_met"],
            "additionalProperties": False,
        },
    },
}

Not every model or provider honours the schema, so as a fallback get_result scans the answer for the first balanced {…} block that parses as valid JSON. Requiring an explanation costs a few tokens, but it makes every verdict auditable later.

Results#

Each run gets a timestamped folder containing every prompt, reply and rubric verdict, plus a description of the run settings. I evaluated openai/gpt-oss-20b at low reasoning effort, graded by the same model, on random subsets of the main split (seed 40):

RunConversationsCriteria gradedScoreMedian
test-2026-09-261001,0910.5310.570
test-2026-09-272002,2950.5000.532

Histogram of per-conversation scores for the 200-conversation run: 15 below zero, then rising to a peak between 0.4 and 0.8, with 31 conversations above 0.8

The run score hides a wide spread. In the 200-conversation run:

  • 18 conversations scored a perfect 1.0, while 15 went below zero because penalties outweighed the points earned.
  • The model met 64% of positive criteria (1,011 of 1,585) and triggered 32% of the penalty criteria (229 of 710).

The per-conversation files are what make this useful. Every low score can be traced to the exact criteria missed or triggered, together with the grader's explanation.

Limitations#

  • Subsets. 100–200 of the 5,000 conversations. The drop from 0.531 to 0.500 between runs is within the sampling variance you'd expect.
  • Self-grading. The grader here is the model under test. OpenAI's reference setup grades with GPT-4.1, and a stronger, independent grader is more trustworthy.
  • Not comparable with OpenAI's published HealthBench numbers, for both reasons.
  • Sequential calls with no retries, so large runs are slow and fragile.

What came next#

The harness was the first step. I later merged it into evals-healthbench-dashboard, which adds a local web UI for starting runs and browsing results. After that I rebuilt the whole tool as a cross-platform desktop app: see HealthBench Evals, a desktop app built with spec-driven development.

Resources#