Shipped scenarios across the current finance, helpdesk, and retail packs.
Inspect scenariosOpen research benchmark · Agent safety
First place, Agent Safety track · AgentX–AgentBeats Phase 1Competition record
PiBench
A benchmark for enacted policy compliance in language-interactive agents. PiBench tests whether an explicit policy governs what an agent says, does, and changes while doing a task.
Public result signal
Full compliance remains the hard part.
Current AgentBeats submissions show the gap between aggregate performance and passing every required policy check in a complete episode.
The chart compares full-compliance rate with end-to-end runtime. It is a dated snapshot of the current public protocol, not a forecast or a cross-version claim.
Public AgentBeats result snapshot
Each point is one submitted agent. Hover for the published evidence.
Submitted agentsHighest compliance
Public agents represented in the dated AgentBeats result snapshot.
Open resultsHighest published full-compliance rate in the current snapshot.
View evidenceAgent Safety track, AgentX–AgentBeats Phase 1 through Berkeley RDI.
Competition recordEvaluation object
A correct answer is not the same as compliant behavior.
An agent can perform a task correctly but still violate the rules governing it. PiBench evaluates the complete episode at the surfaces where those violations can occur.
Agent messages
What the agent communicates, discloses, refuses, or escalates.
Tool actions
Which tools are called, with which arguments, and in what relevant order.
Environment state
What the episode actually changes, including intermediate policy-relevant state.
Research direction
A stable evaluation grammar for policy-specific checks.
PiBench is developing a generic evaluator compiler over typed trace evidence. Policy packs compose fixed primitives instead of requiring a new hard-coded evaluator type for every rule.
Open contribution
Build policy-grounded evaluation cases.
The current contribution campaign focuses on translating real policy rules into realistic, reviewable scenarios with observable evaluation contracts.