AGENTS.md Template: Model Evaluation and Drift Monitoring

Overview

Direct answer: This AGENTS.md Template defines the model evaluation and drift monitoring workflow, and it supports both single-agent execution and multi-agent orchestration with explicit handoffs, shared memory, and a single source of truth. It establishes operating boundaries so AI coding agents can autonomously validate model quality, detect data drift, and trigger governance gates with human review when needed.

This template specifies the operating model, roles, and rules for an automated loop that continuously evaluates models, monitors data drift, and coordinates decisions about retraining or release readiness. It emphasizes tool governance, secure access, auditable actions, and clear escalation paths for AI coding agents working in production-like environments.

When to Use This AGENTS.md Template

• You need an auditable, repeatable evaluation and drift monitoring workflow for deployed AI models.
• You require multi-agent orchestration with explicit handoffs between evaluation, validation, and monitoring agents.
• You must enforce tool governance, secret handling, and human review for critical decisions.
• You want a lightweight, project-level operating context that agents can paste into AGENTS.md to bootstrap the workflow.

Copyable AGENTS.md Template

# AGENTS.md

Project role: Model Evaluation and Drift Monitoring Agents
Agent roster and responsibilities:
- Evaluator Agent: runs model evaluation on the current production model using validated test data; outputs metrics and artifacts.
- Drift Monitor Agent: monitors data streams for drift using statistical tests; flags drift if threshold exceeded.
- Validator Agent: validates evaluation results against pre-defined acceptance criteria; marks results as pass/fail.
- Orchestrator: coordinates tasks, sequences handoffs, and records provenance.
- Data Preparer: ensures data used for evaluation/drift tests is current and labeled.

Supervisor or orchestrator behavior:
- Orchestrator receives task intents, assigns agents, aggregates results, and enforces memory/versioning rules.
- All actions are idempotent and auditable; state is stored in a versioned artifact store.

Handoff rules between agents:
- Evaluator → Validator → Orchestrator
- Drift Monitor signals drift events to Orchestrator; Orchestrator may re-run evaluation or trigger retraining.
- All final decisions are logged and stored as artifacts for traceability.

Context, memory, and source-of-truth rules:
- Source of truth = versioned artifacts in the artifact store and the model registry.
- Results carried with unique IDs; memory is queried via the artifact store to avoid re-computation.
- Agents must reference the single source of truth and should not duplicate results.

Tool access and permission rules:
- Evaluator: read test data, evaluation libraries, and model artifacts; cannot modify registry or production systems.
- Drift Monitor: read data streams and feature distributions; write drift reports to artifact store.
- Validator: read evaluation results; approve or flag for escalation; can trigger orchestrator actions.
- Orchestrator: write to artifact store, trigger retraining or deployment requests via approved channels; secrets managed in a vault.

Architecture rules:
- Event-driven, with a central orchestrator and stateless agents; all state is stored in a centralized artifact store.
- Use a simple queue or pub/sub to pass results; ensure idempotent processing.

File structure rules:
- agents/
  - evaluator/
    - main.py
  - drift_monitor/
    - main.py
  - validator/
    - main.py
  - orchestrator/
    - main.py
  - data_prep/
    - prepare.py
- artefacts/
- configs/
- tests/
- docs/

Data, API, or integration rules when relevant:
- Use REST/GraphQL endpoints for model registry where supported; follow versioned schemas for metrics and drift signals.
- Export metrics in a canonical JSON with fields: metric_name, value, timestamp, threshold, pass/fail.

Validation rules:
- Metrics must have numeric values; all required fields must exist in outputs.
- Drift signals must include a p-value or probability threshold and a drift magnitude.

Security rules:
- Secrets in a vault; agents have scoped access; all actions are logged.

Testing rules:
- Unit tests for metric calculations; integration tests for end-to-end evaluation and drift detection flows.

Deployment rules:
- Deploy agents as a single package; blue-green style for orchestration; keep pilot retraining gated by human review when drift is detected.

Human review and escalation rules:
- If any critical metric crosses threshold or drift is detected with low confidence, escalate to a human reviewer with context artifacts.

Failure handling and rollback rules:
- If retraining fails, revert to the last successful model in registry and notify the team; rollback artifacts.

Things Agents must not do:
- Do not modify training data in production; do not deploy models without approval; do not bypass memory/source-of-truth rules.

Recommended Agent Operating Model

The orchestrator (planner) coordinates the evaluation and drift-monitoring workflow. Evaluator and Drift Monitor act as implementers, producing metrics and drift signals. Validator provides gatekeeping and decides pass/fail. In escalation, a human reviewer may intervene when thresholds are ambiguous. The model is allowed to autonomously trigger retraining or deployment only when all gates are satisfied. Escalation paths are explicit and auditable.

Recommended Project Structure

project-root/
├── agents/
│   ├── evaluator/
│   │   └── main.py
│   ├── drift_monitor/
│   │   └── main.py
│   ├── validator/
│   │   └── main.py
│   ├── orchestrator/
│   │   └── main.py
│   └── data_prep/
│       └── prepare.py
├── artefacts/
├── configs/
├── data/
├── tests/
└── docs/

Core Operating Principles

• Operate with a single source of truth and immutable results where possible.
• Design for idempotence and deterministic outputs to enable safe retries.
• Favor observability through structured artifacts, logs, and traces.
• Maintain strict secret handling and scoped access control.
• Respect escalation paths and rely on human review for high-risk decisions.

Agent Handoff and Collaboration Rules

Planner/Orchestrator coordinates: defines task batches, assigns agents, and logs provenance. Implementers (Evaluator, Drift Monitor) execute tasks and emit structured results. Reviewer/QA validates results, logs decisions, and triggers further actions. Tester validates end-to-end flow and regression safety. Researcher provides metric definitions and drift detection strategies. Domain Specialist tailors thresholds and feature considerations for the specific model domain.

Tool Governance and Permission Rules

Commands, edits, and API calls must go through the orchestrator with explicit approvals. Secrets are accessed via a vault; no hard-coded credentials. All actions are auditable with artifact IDs and timestamps. Production systems require deployment gates and sign-off from the responsible owner.

Code Construction Rules

Write modular, testable code; document interfaces; avoid hard-coded thresholds; store thresholds in configs; validate inputs and outputs; use versioned artifacts; ensure idempotence; run local tests before remote execution.

Security and Production Rules

Use least-privilege permissions; secrets rotate periodically; monitor for unusual activity; require human review for policy changes affecting production drift controls.

Testing Checklist

• Unit tests for metric calculations and drift detection logic.
• Integration tests that simulate end-to-end evaluation and drift signaling.
• End-to-end tests for retraining triggers and deployment gates.
• Security tests for secrets access and audit logging.
• Performance tests ensuring latency remains within bounds for real-time monitoring.

Common Mistakes to Avoid

• Assuming drift signals alone justify retraining without governance gates.
• Allowing agents to bypass the single source of truth.
• Overloading the orchestrator with unbounded task requests.
• Ignoring data lineage and privacy constraints in production data.
• Hard-coding thresholds; failing to version them.

FAQ

What is the purpose of this AGENTS.md Template?

This template defines the operating model, roles, and rules for a model evaluation and drift monitoring workflow using AI coding agents.

How many agents are typically involved?

A minimal setup uses an Evaluator, Drift Monitor, and Orchestrator; larger teams add Validation and QA agents as needed.

What triggers a handoff between agents?

Handoffs occur when metrics are produced, when drift is detected, or when validation results are ready for decision.

How is data and memory managed?

Results are stored in a versioned artifact store; agents reference a single source of truth and avoid duplicating data.

What are the security considerations?

Secrets are stored in a secrets vault; agents have scoped access; all actions are auditable.