Databricks Production Architecture AGENTS.md Template

Overview

Direct answer: This AGENTS.md template defines the operating model for a Databricks production architecture, enabling AI coding agents to orchestrate data pipelines with clear roles, hands-offs, governance, and human review for reliability at scale.

It supports both single-agent and multi-agent orchestration by describing roles, handoffs, memory, and source of truth across the stack.

When to Use This AGENTS.md Template

• When building a Databricks-based production data platform requiring coordinated, multi-agent workflows.
• When you need formal handoffs between Planner, Implementer, Reviewer, and Tester with explicit success criteria.
• When you require controlled access to Delta Live Tables, Unity Catalog, clusters, jobs, and secrets with auditable approvals.
• When you want an auditable operating model for governance, security, and compliance across Databricks workloads.

Copyable AGENTS.md Template

Copy this block into your AGENTS.md file to set project context for single-agent or multi-agent Databricks production workflows.

# AGENTS.md

Project Role
  Databricks Production Architecture for Analytics Platform

Agent roster and responsibilities
  Planner (Orchestrator): coordinates multi-agent workflow, enforces constraints, tracks state, triggers tasks, ensures idempotent progress.
  Implementer: develops notebooks, jobs, Delta Live Tables, and cluster configurations to satisfy tasks.
  Reviewer: validates code quality, data quality, governance, and security before promotion.
  Tester: runs end-to-end tests, monitors outcomes, and captures regression signals.
  Researcher: identifies optimization opportunities, new patterns, and tooling improvements.
  Domain Specialist (Databricks): ensures correct use of Unity Catalog, data governance, and platform best practices.
  Ops/Observability: configures monitors, dashboards, and alerting.

Supervisor or orchestrator behavior
  The Planner maintains a living runbook state, coordinates handoffs, and enforces gate checks before transitioning to the next stage. All actions are auditable and time-stamped.

Handoff rules between agents
  - Planner -> Implementer: provide task context, success criteria, and data sources; ensure artefacts exist in the repository.
  - Implementer -> Reviewer: deliver validated notebooks, Delta Live Tables definitions, and job specs; include test results and data samples.
  - Reviewer -> Planner: approve or request changes; if changes are needed, reassign to Implementer.
  - Planner -> Tester: grant access to staging runs and logs; require pass/fail signals before production.
  - Researcher -> Planner/Domain Specialist: propose changes; await approval for deployment.

Context, memory, and source-of-truth rules
  - Source of truth: central Git repository for notebooks, configs, and pipeline definitions; Delta Live Tables lineage; Unity Catalog metadata.
  - Memory: store current task context, constraints, and decisions in a persistent workspace (AGENTS.md context) and referenceable keys for reproducibility.
  - Do not rely on ephemeral chat history as truth; always record decisions in the codebase and runbook.

Tool access and permission rules
  - Databricks REST API calls for job runs, cluster management, and registry lookups require the Planner’s permission; secrets must be retrieved from a secure scope; never embed secrets in code.
  - Access to Delta Live Tables and Unity Catalog must follow least privilege; production changes require explicit approvals and governance checks.
  - All tool actions must be auditable; logs are retained for a minimum period.

Architecture rules
  - Use Delta Live Tables for ETL, with idempotent transformations; enforce event-driven triggers where possible.
  - Rely on Unity Catalog for data governance and role-based access control; separate data, logs, and metadata with clear ownership.
  - Maintain separate dev/stage/prod environments; promote changes via GitOps and PR gates.

File structure rules
  - Organize code by domain: notebooks/, pipelines/, configs/, tests/, docs/, monitors/; avoid mixing unrelated artifacts.
  - Do not commit secrets or credentials; reference them via secrets scopes and environment configuration.

Data, API, or integration rules
  - Interact with Databricks Jobs API, Delta Live Tables definitions, and Unity Catalog metadata through authorized, auditable calls.
  - Validate external data sources via defined connectors with retry and backoff policies; mock external dependencies in test environments.

Validation rules
  - All notebooks and pipelines must pass unit tests, integration checks, and data quality validations before promotion.
  - Ensure idempotent job runs and reliable replay semantics in case of retries.

Security rules
  - Enforce least-privilege access in all roles; secrets must live in a secure scope and rotated per policy.
  - Do not print secrets in logs; sanitize outputs; enable network restrictions for production resources.

Testing rules
  - Unit tests for transformation logic; integration tests for end-to-end pipelines; smoke tests post-deploy.

Deployment rules
  - Promote from dev to staging to prod via PR-driven pipelines; require build, test, and governance checks before promotion.
  - Maintain runbooks and post-deploy checks; auto-monitoring dashboards should show success signals.

Human review and escalation rules
  - Escalate design or security concerns to Domain Specialist or Security lead; critical incidents go to on-call SRE.
  - All escalations must be documented with root cause and remediation plan in the runbook.

Failure handling and rollback rules
  - On failure, revert to last stable checkpoint; re-run from known good state; preserve data lineage and state in logs.
  - Automatic rollback should be triggered under governance and security policy.

Things Agents must not do
  - Do not bypass approvals or access controls; do not modify production configurations without the Planner’s authorization.
  - Do not hard-code secrets or credentials; avoid context drift or architecture drift; do not perform unsupervised production changes.
"

Recommended Agent Operating Model

The operating model assigns clear roles with decision boundaries and escalation paths for Databricks production workflows. The Planner orchestrates, the Implementer executes, the Reviewer validates, and the Tester verifies; the Researcher and Domain Specialist continuously improve the pattern; the Ops/Observability team monitors and alerts. Decisions are bounded by the runbook, with escalation to the Security and Platform teams for policy violations or outages.

Recommended Project Structure

databricks-prod-architecture/
├── orchestrator/
├── notebooks/
├── jobs/
├── delta-live-tables/
├── unity-catalog/
├── configs/
├── tests/
├── monitors/
└── docs/

Core Operating Principles

• Single source of truth and auditable decisions.
• Idempotent, deterministic actions with clear memory context.
• Explicit handoffs and gating between agents.
• Least-privilege access and secrets management.
• Governed promotion across environments with GitOps.

Agent Handoff and Collaboration Rules

• Planner coordinates all handoffs and maintains state; every transition requires a success flag from the receiving agent.
• Implementer must provide artifacts (notebooks, pipeline definitions, DLT specs) and test results before reviewer handoff.
• Reviewer validates code quality, data quality, governance, and security; approves only if tests pass.
• Tester executes end-to-end scenarios in staging with defined data samples and monitors outcomes.
• Researcher proposes optimizations; requires approval before deployment.
• Domain Specialist enforces Unity Catalog and data governance constraints across the workflow.

Tool Governance and Permission Rules

• Commands to Databricks must go through Planner and be recorded in runbooks.
• File edits require PR-based workflows with code reviews and approvals.
• APIs, secrets, and tokens are accessed via secure scopes; never embedded in code.
• Production systems require approval gates and change-management trails.
• External service calls must be auditable with failure handling policies.

Code Construction Rules

• Write modular, testable notebooks and pipelines; prefer functions over monolith scripts.
• Follow naming conventions for notebooks, pipelines, and configs; ensure traceability.
• Document every decision, constraint, and dependency in code and the runbook.
• Do not duplicate work; reuse existing components when possible.

Security and Production Rules

• Enforce least-privilege access and rotate secrets regularly.
• Mask sensitive outputs in logs and dashboards.
• Enforce network egress controls and monitoring for all production resources.
• Validate governance policies on every promotion to prod.

Testing Checklist

• Unit tests for data transformation logic.
• Integration tests for data sources, sinks, and transformations.
• End-to-end replication checks in staging before prod deploy.
• Security and access control validations.
• Rollback and recovery drills documented in the runbook.

Common Mistakes to Avoid

• Skipping governance checks or bypassing approvals.
• Hard-coding secrets or credentials in notebooks or configs.
• Allowing architecture drift or inconsistent naming conventions.
• Neglecting data quality tests or missing audit trails.

Related implementation resources: AI Use Case for Sales Pipeline Reviews and Deal Risk Scoring and AI Use Case for Pet Toy Brands Using Instagram Comment Data To Evaluate Customer Feedback On New Toy Durability.

FAQ

What is the purpose of this AGENTS.md Template for Databricks production architecture?

It defines the operating model, roles, workflows, and governance for AI coding agents working on a Databricks production stack, enabling single-agent and multi-agent orchestration.

Who is the primary audience for this template?

Developers, data engineers, platform engineers, product leads, and engineering leaders seeking a structured AGENTS.md for Databricks production workflows.

How are agent handoffs managed between planner, implementer, reviewer, and tester?

Handoffs are defined in the AGENTS.md; each handoff includes required context, memory props, and success criteria; the planner coordinates transitions and validates completion before next phase.

What tooling and permissions are governed by this template?

It prescribes Databricks REST API usage, job orchestration, Delta Live Tables, Unity Catalog access, secrets scopes, and approval gates with least-privilege access.

How are failures and rollback handled?

Failures trigger rollback to the last known good state, re-run from checkpoints, and escalation to responsible owners; monitoring and audit trails are maintained.