AGENTS.md Template for Search Indexing Architecture

Overview

This AGENTS.md template defines the operating manual for a search indexing architecture using AI coding agents. It governs both single-agent execution and multi-agent orchestration across planning, data ingestion, indexing, validation, and human review. Direct answer: It provides clear roles, handoffs, sources of truth, and governance to keep indexing pipelines accurate, auditable, and secure.

When to Use This AGENTS.md Template

• When designing a search indexing pipeline that uses multiple AI agents or a single agent with complex decision boundaries.
• When you need reproducible operating context for planning, execution, validation, and escalation.
• When governance, security, and auditing of tool use and data access are required.
• When you want explicit handoffs between planner, fetcher, indexer, validator, evaluator, and human review roles.
• When you need a shareable, paste-ready AGENTS.md template for onboarding new teammates.

Copyable AGENTS.md Template

# AGENTS.md

Project Role
- Build and maintain a scalable search indexing architecture powered by AI coding agents.
- Ensure accurate indexing, data governance, and auditable decisions across single-agent and multi-agent workflows.

Agent Roster and Responsibilities
- Planner: designs data flow, task ordering, and success criteria for indexing tasks.
- Fetcher: retrieves content sources (web, APIs, databases) within rate limits and access controls.
- Indexer/Parser: extracts, normalizes, tokenizes, and indexes content for search.
- Validator: runs quality checks on index outputs and schema conformance.
- Evaluator: performs risk/quality assessments and flags anomalies for escalation.
- Orchestrator: coordinates handoffs, memoizes context, and enforces governance.
- Handoff Agent (as needed): transfers context, artifacts, and decisions between roles with explicit state.

Supervisor or Orchestrator Behavior
- Maintains a single source of truth (SoT) for each indexing task.
- Enforces memory retention, context passing, and versioned artifacts.
- Controls tool access, secrets, and integration permissions.

Handoff Rules Between Agents
- On task completion, the current agent must produce a handoff package with: context, artifacts, decision rationale, and status.
- The next agent may request clarifications via the Orchestrator; escalation occurs if missing context persists after 2 attempts.
- All handoffs are versioned and stored in a dedicated SoT repository.

Context, Memory, and Source-of-Truth Rules
- SoT stores: task definition, input data, indexing outputs, validation results, and decision logs.
- Agents may query SoT but must not mutate historical records without an approved action.
- Data provenance must be recorded for every index entry.

Tool Access and Permission Rules
- Use least privilege for API keys, secrets, and file access.
- Secrets must be stored in a secure vault; never embedded in code.
- Production tools require explicit approval gates by the Orchestrator.

Architecture Rules
- Use a modular pipeline with clear boundaries between Planner, Fetcher, Indexer, Validator, Evaluator, and Orchestrator.
- All artifacts are versioned and stored in a centralized artifact store.
- Idempotency is required for all indexing tasks.

File Structure Rules
- Keep all project-level docs under docs/ and agent definitions under agents/.
- Store configuration in configs/ with environment-specific overrides.
- Cache and raw data in data/ with strict lifecycle rules.

Data, API, or Integration Rules
- Normalization, deduplication, and schema validation must occur before indexing.
- Rate limits and retry policies should be explicit for each source.
- All external integrations must expose audit logs.

Validation Rules
- Each index entry must pass schema validation and sanity checks.
- Automated tests must verify end-to-end indexing from source to index.

Security Rules
- Enforce least privilege, secret management, and network segmentation.
- All tool calls should be auditable and reversible.

Testing Rules
- Unit tests for each agent, integration tests for multi-agent flows, and end-to-end throughput tests.
- Use synthetic data for deterministic testing.

Deployment Rules
- CI/CD with canary deployments for indexing updates.
- Rollback to previous index version on failure.

Human Review and Escalation Rules
- Escalate anomalies to humans when confidence is below threshold.
- Maintain a human-in-the-loop log for decisions affecting search quality.

Failure Handling and Rollback Rules
- Retries with backoff; if still failing, rollback to last stable index and notify stakeholders.
- Do not deploy unknown changes to production without test coverage and approval.

Things Agents Must Not Do
- Do not bypass handoffs or memory guards.
- Do not operate on production data without authorization.
- Do not modify historical indexing results without audit.
- Do not reveal secrets or credentials in logs or artifacts.

Recommended Agent Operating Model

The agent operating model defines distinct roles with decision boundaries and escalation paths to ensure reliable multi-agent indexing workflows. Planner coordinates intent; Fetcher gathers sources; Indexer builds the index; Validator and Evaluator ensure quality; Orchestrator enforces governance and handoffs; Humans intervene for exceptions. Direct answer: This model provides clear responsibilities, escalation paths, and guardrails for robust search indexing orchestration.

Recommended Project Structure

project-root/
  agents/
    planner/
    fetcher/
    indexer/
    parser/
    validator/
    evaluator/
    orchestrator/
  data/
  configs/
  docs/
  tests/
  secrets/
  README.md

Core Operating Principles

• Explicit memory and source-of-truth management for every indexing task.
• Least privilege and auditable tool access for all agents.
• Deterministic, idempotent operations across retries and restarts.
• Clear handoffs with versioned context packages.
• Human review gates for high-risk decisions.

Agent Handoff and Collaboration Rules

• Planner ➜ Fetcher: hand off task definition, source list, and collection scope.
• Fetcher ➜ Indexer: hand off raw documents, source metadata, and provenance.
• Indexer ➜ Validator: hand off indexed shards and schema conformance results.
• Validator ➜ Evaluator: hand off QA results and confidence scores.
• Orchestrator: maintains re-entrancy, memory, and cross-agent context across steps.

Tool Governance and Permission Rules

• All tool calls must be authorized by the Orchestrator and logged.
• Secrets must never be hard-coded; use a vault with rotation.
• External APIs require access tokens with scoped permissions.

Code Construction Rules

• Write deterministic, well-documented agents with clear interfaces.
• Avoid hard dependencies; use dependency injection for testability.
• All changes pass through CI with static analysis and tests.

Security and Production Rules

• Network isolation for indexing workloads; production secrets in vaults only.
• Audit logs for all indexing operations and data access.
• Rollout plans with canary windows and kill switches.

Testing Checklist

• Unit tests for each agent; integration tests for multi-agent flows.
• End-to-end tests from source ingestion to index verification.
• Security and access control tests; secrets rotation checks.

Common Mistakes to Avoid

• Skipping explicit handoffs and memory propagation.
• Over-sharing tokens or secrets in logs.
• Ignoring data provenance and versioning.
• Assuming a single agent can replace multi-agent coordination.

Related implementation resources: AI Use Case for Technical Writers Using Github To Maintain and Auto-Check Documentation Links for Broken Urls and AI Use Case for Corporate Event Managers Using Slack To Orchestrate Day-Of Venue Tasks Across Multi-Department Teams.

FAQ

What is the purpose of this AGENTS.md Template for search indexing architecture?

The template provides operating context, roles, and governance for a multi-agent indexing workflow.

Who are the recommended agents in this workflow?

Planner, Fetcher, Indexer, Validator, Evaluator, and Orchestrator with defined handoffs and memory.

How are agent handoffs defined and who manages the handoffs?

Handoffs are managed by the Orchestrator, with versioned context packages and explicit status and artifacts.

What are the security and production considerations?

Least privilege, vault-based secrets, auditable actions, and controlled deployment with canaries and kill switches.

How is testing performed and what constitutes deployment readiness?

Unit, integration, and end-to-end tests, plus manual review gates for high-risk changes.