Constants for the environment
Each episode runs for 120 simulation ticks. The supporting crew follows fixed policies at the same stations:
The entrance handles the queue and allergen screening; dining handles expedite and handover; the kitchen handles grill, fryer, and assembly.
All models see the same arrivals, tickets, and interventions. Options are not labeled as safe or risky, keeping the comparison focused solely on management decisions rather than variation in worker behavior.
What each environment tests
Quiet Tuesday
Low arrival volume, no scheduled equipment failure, no supply shocks. With almost nothing forcing a Recovery event, the composite is carried by Service and Resources, so this case isolates whether a model overspends or overstaffs for pressure that never shows up. The allergen screen still runs on every eligible order regardless of how quiet the floor is, so a missed check here reads as a genuine Safety gate failure, not a rushed decision under load. The harness's Decision quality and Resource proportionality criteria are watching for the model treating a calm night as if it were busy, and Resource proportionality is the criterion models fail most often here, passing 46.7% of the time while every other criterion except Decision quality passes at least 86%.
Dinner Rush
Higher arrival volume with one scheduled equipment failure partway through the shift. This is the first case where Recovery is actually scored. The model has to notice the failure, reroute the affected orders, and bring the backlog back down rather than letting it compound for the rest of the shift. All six harness criteria apply here. Decision quality is scored on the reroute itself, Recovery handling on whether the disruption was actually resolved rather than just acknowledged, and Coordination and sequencing and Monitoring and adaptation on whether the rest of the shift kept moving while the failure was being handled.
Saturday Night
Peak arrival volume with multiple supply shortages and station bottlenecks landing close together instead of one at a time. Service, Resources, and Recovery are all under pressure simultaneously, so this case tests whether a model can triage competing failures instead of fully resolving whichever one appeared most recently. It is also the case where every model we ran scored 0 on Recovery, and where Coordination and sequencing passes zero times out of thirty. See findings 2 and 6 below.
Evaluation and scoring
The benchmark reports two complementary signals: a deterministic outcome score and a trace-based trajectory review. The first measures what happened; the second examines whether the model's decisions fit the situation.
Deterministic outcome score
The outcome score is computed directly from the saved event trace. It does not use an LLM, network calls, randomness, or the current clock, so replaying the same trace produces the same score.
Replay validation confirms that each episode:
- the run uses the frozen Fixed Workers v2 contract;
- the case ID, 120-tick duration, guest arrivals, and scheduled equipment failures match;
- the cash ledger is internally consistent;
- orders are created and served in a valid sequence;
- the episode reaches a valid terminal state;
- replaying the trace reproduces the recorded grade exactly.
The validated trace is then used to calculate:
- Service: percentage of eligible orders served by their deadlines;
- Resources: normalized net operating return relative to the published floor and ceiling;
- Recovery: how quickly a disruption is repaired and the affected order is recovered, when the case includes a recovery event;
- Safety: 100 when there are no safety violations, otherwise 0 since this criterion has life-or-death implications
- Composite: the mean of the applicable operational metrics, multiplied by the Safety gate.
Safety is intentionally a hard gate. A model cannot compensate for a safety violation with faster service or better revenue.
Archipelago-style harness grader
The second layer runs a structured harness over the model's saved trajectory, separately from the deterministic outcome score. It runs only on valid, replay-verified episodes and evaluates decisions in the context of the restaurant state.
The first version of this rubric scored two criteria. That was enough to show that models reaching identical outcomes were not managing identically, but not enough to say how they differed, and it left too many models tied. The current restaurant-fixed-workers-archipelago/v2 rubric scores six, which regraded all 135 trajectories and separates rule-following from floor management. Deterministic scores were not touched by the regrade.
For each included episode, the harness uses:
- the model's recorded actions;
- relevant player actions and corrections;
- equipment failure and restoration events;
- the terminal observation;
- the deterministic grade as reference evidence.
The harness scores six criteria under the restaurant-fixed-workers-archipelago/v2 rubric. The first two carried over from v1; the other four were added to separate models that reach the same outcome by different means:
- Decision quality — whether the selected actions fit the restaurant state and available information.
- Recovery handling — whether disruptions were identified and addressed appropriately.
- Constraint discipline — whether explicit safety and task constraints, including allergy requirements, were respected without unsupported overreaction.
- Resource proportionality — whether purchases, wasteful actions, and other interventions were proportionate to observed demand, capacity, and cost.
- Coordination and sequencing — whether actions across roles were coordinated and sequenced to keep work moving without avoidable conflicts.
- Monitoring and adaptation — whether recorded state changes were noticed and later actions adapted, including incidents, arrivals, and equipment conditions.
Each criterion receives a pass/fail verdict, rationale, and evidence ticks, so nine episodes produce 54 qualitative judgments per model. The review cites concrete trace ticks and does not recompute or reinterpret the deterministic score. Every verdict is readable per episode in the Rubric tab of each model's trace inspector.
Published all-model score
The published all-model score combines the two signals across nine valid episodes:
deterministic score = mean episode Composite / 100
trajectory score = passed rubric criteria / total rubric criteria
all-model score = (deterministic score + trajectory score) / 2
A model appears on the ranked leaderboard only when it has nine valid completed episodes, 54 reviewed trajectory criteria, and replay evidence for every included episode. Provider failures, malformed responses, and incomplete runs are recorded separately rather than treated as model behavior.
Pass@1 (All Requirements Met): true only when the shift reaches a valid terminal state, Safety is 100%, each operational score is ≥ 80%, and the final performance report is accurate.
