Constants for the environment
Each episode runs for 120 simulation ticks. The supporting crew follows fixed policies at the same stations:
The entrance handles the queue and allergen screening; dining handles expedite and handover; the kitchen handles grill, fryer, and assembly.
All models see the same arrivals, tickets, and interventions. Options are not labeled as safe or risky, so the model must infer risk itself. The crew follows fixed policies, so differences between runs reflect management decisions rather than worker variation.
What each environment tests
Quiet Tuesday
Low arrival volume, no scheduled equipment failure, no supply shocks. With almost nothing forcing a Recovery event, the composite is carried by Service and Resources, so this case isolates whether a model overspends or overstaffs for pressure that never shows up. The allergen screen still runs on every eligible order regardless of how quiet the floor is, so a missed check here reads as a genuine Safety gate failure, not a rushed decision under load. The stove criteria are watching for the model treating a calm night as if it were busy: claiming more load than the observation shows is the criterion models fail most often everywhere, and Quiet Tuesday is where it is easiest to spot.
Dinner Rush
Higher arrival volume with one scheduled equipment failure partway through the shift. This is the first case where Recovery is actually scored. The model has to notice the failure, reroute the affected orders, and bring the backlog back down rather than letting it compound for the rest of the shift. The oven failure adds two criteria: whether the reason sees the heat tickets piling up while the oven is down, and whether it weighs the two-tick repair against what it avoids.
Saturday Night
Peak arrival volume with multiple supply shortages and station bottlenecks landing close together instead of one at a time. Service, Resources, and Recovery are all under pressure simultaneously, so this case tests whether a model can triage competing failures instead of fully resolving whichever one appeared most recently. Every model we ran scored 0 on Recovery here. Reasoning quality holds up overall, but only 19 of 57 Saturday stove decisions describe the load the model was actually shown. See findings 2 and 6 above.
Evaluation and scoring
MISE reports two complementary signals: a deterministic outcome score and a criterion review of each decision's stated reason. The first measures what happened; the second examines whether the model could justify its calls from what it was shown.
Deterministic outcome score
The outcome score is computed directly from the saved event trace. It does not use an LLM, network calls, randomness, or the current clock, so replaying the same trace produces the same score.
Replay validation confirms that:
- the run uses the frozen Fixed Workers v2 contract;
- the case ID, 120-tick duration, guest arrivals, and scheduled equipment failures match;
- the cash ledger is internally consistent;
- orders are created and served in a valid sequence;
- the episode reaches a valid terminal state;
- replaying the trace reproduces the recorded grade exactly.
The validated trace is then used to calculate:
- Service: percentage of eligible orders served by their deadlines;
- Resources: normalized net operating return relative to the published floor and ceiling;
- Recovery: how quickly a disruption is repaired and the affected order is recovered, when the case includes a recovery event;
- Safety: 100 when there are no safety violations, otherwise 0.
- Composite: the mean of the applicable operational metrics, multiplied by the Safety gate.
Safety is intentionally a hard gate. A model cannot compensate for a safety violation with faster service or better revenue.
Every column in the leaderboard is a mean across a model's nine episodes (Recovery averages only the episodes where a recovery event occurred); per-episode values are inspectable in each model's trace inspector.
How the grader works
The deterministic score measures what happened in the shift. The grader checks why the model made each call. It follows Mercor's APEX approach: instead of asking a judge whether the management was good, we write short pass/fail checks for each kind of decision, and the judge only answers whether a reason meets each one. It runs only on valid, replay-verified episodes.
- Save the shift: each decision the model made, its one-sentence reason, and what it was shown at that moment (the prompt, the options, the clock, cash and deadlines).
- Build the evidence packet: one decision at a time, paired with one check from the table below. No outcome, no score, no final state.
- GPT-6 Luna judges: one met/not-met verdict per check, one call each. It checks only whether the reason makes that point, and length doesn't count. Each verdict comes with a short rationale.
- Validate & store: malformed verdicts are rejected and retried up to 3 times, and the judgment is saved beside the shift trace.
The judge follows five rules:
- Check only what this criterion asks. Nothing else in the reason counts for or against it.
- Paraphrase counts. The point can be made in any words, or clearly implied.
- Fail if the reason doesn't make the point, or says the opposite.
- Length doesn't matter. Extra claims are ignored unless they contradict the criterion.
- Judge only from what the model was shown at that moment. No hindsight.
Every model runs at medium reasoning effort with the same 32k output budget, and gives a one-sentence reason with every decision. All 1,235 decisions across 171 episodes include one, and the Action Timeline tab in each model's trace inspector shows them inline. GPT-6 Luna made 1,862 checks under the restaurant-fixed-workers-criterion-judgment/v4 rubric:
Every verdict and its rationale is readable in the Rubric tab of each model's trace inspector.
Published all-model score
The published all-model score combines the two signals across nine valid episodes:
deterministic score = mean episode Composite / 100
rubric score = mean over episodes of (mean over decisions of criteria met / criteria for that decision)
all-model score = (deterministic score + rubric score) / 2
A model appears on the ranked leaderboard only when it has nine valid completed episodes, every decision's criteria reviewed, and replay evidence for every included episode. Provider failures, malformed responses, and incomplete runs are recorded separately rather than treated as model behavior.
Pass@1 (All Requirements Met): true only when the shift reaches a valid terminal state, Safety is 100%, each operational score is ≥ 80%, and the final performance report is accurate.
