super agent evaluate runs an agent's BDD specification. Add an Agent Quality Record to keep its measures, declared permissions and recorded decision together for later review.
The record uses the open draft format published as SuperGauge.
The current evaluation path emits at L1 with empty deterministic gates and a hold decision. Release approval requires additional checks and a named approver.
The record fingerprints scenario names and expected outputs. Mark held-out cases separately and keep them closed to optimization. Sealing is an operator assertion backed by your dataset controls.
The authority block is read from the compiled playbook. Tools become capabilities, and the runtime block supplies the sandbox and egress posture. A record naming the agent without naming its permissions can be structurally valid and substantively wrong.
Where routing measurements are available, the record reports how often another agent selects yours from an A2A catalogue. This helps distinguish routing failures from task failures.
Illustrative excerpt with abbreviated hashes. The emitted file contains the full record blocks and digests.
supergauge: "0.1"
profile: {id: sg/framework-agent, version: "0.1", tier: T1}
subject:
agent: developer
harness_digest: sha256:9c1e... # the playbook, fingerprinted
model: {provider: ollama, id: qwen3:8b}
authority: # what the agent could do
sandbox: docker
egress: deny-by-default
capabilities: [search, fetch]
task_set:
manifest_digest: sha256:7d02... # the scenarios, fingerprinted
held_in: 6
held_out: 2
sealed: false
measures:
- {id: task.completion, value: 0.83, n: 8}
- {id: interop.routing_invocation, value: 0.75}
decision:
verdict: hold
actor: "you@example.com"Write the record. A .json suffix selects JSON, otherwise YAML.
The risk tier the record claims. Higher tiers demand more before a release.
Assert the held-out scenarios were closed to anything that tunes the agent.
Optional Jev review. Acceptance preserves existing approval; warning holds for review and rejection rejects.
Measures are included when evaluation results are available. L2 requires deterministic gates, a protected held-out split and contamination probes. A record-writing error prints a warning, so CI must also confirm that the expected file was created.
Install superoptix[typesafe] or its [jev] alias to use Jev with DSPy 3.4+. A live call requires TYPESAFE_API_KEY. The current judge reviews a pass-count and framework summary; it provides limited evidence about individual answers and tool trajectories.
pip install "superoptix[typesafe]" super agent evaluate developer --gauge-out record.yaml --gauge-jev
Acceptance preserves an existing approval only when deterministic checks pass and the split is sealed. Warning holds for review, and rejection rejects. Existing rejections stay in place. Heuristic fallbacks hold for review and record a separate source and confidence proxy. Judge pins identify live calls.
Jev quality control guideA hold or reject record can meet a conformance level. Your deployment job must also require a ship decision and passing gates. The release system authenticates the approver and binds the record to the candidate artifact.
# Protocol checker from a pinned SuperGauge checkout python conformance/check.py record.yaml --level L2 --require-ship --quiet
This check blocks an L1 evaluation record. Install pyyaml and jsonschema for the protocol checker. Profile enforcement, signature verification and evidence replay remain part of your release process.
Adoption guideAny SuperGauge implementation reads the output, so a team running SuperOptiX alongside a coding-agent harness gets one record shape across both.