← All Insights

Data & AI · June 2026

Building AI Agents with Azure AI Foundry: Architecture Guide for Enterprise

18 min read

Azure AI Foundry is Microsoft's unified platform for building AI applications — and in 2026, that increasingly means building agents. Not chatbots. Not RAG demos. Autonomous systems that reason, call tools, access enterprise data, and take actions on behalf of users.

This guide covers the architecture patterns, design decisions, and operational considerations for building production-grade AI agents on Azure AI Foundry. If you've built a prototype and need to get it to production — or you're evaluating whether Foundry is the right foundation for your agent strategy — this is for you.

What Makes an Agent Different from a Chatbot

A chatbot answers questions. An agent does work. The distinction matters architecturally:

  • Reasoning loop — the agent decides what to do next based on the result of what it just did. Multiple steps, not single-turn.
  • Tool calling — the agent invokes external systems (APIs, databases, services) to gather information or take action.
  • Knowledge grounding — responses are grounded in enterprise data, not just the model's training data.
  • Guardrails — input and output are validated for safety, accuracy, and compliance before reaching the user.
  • Governance — every decision is logged, auditable, and attributable to a user and context.

Azure AI Foundry provides the building blocks for all of these. But assembling them into a production system requires architectural decisions that Microsoft's documentation doesn't fully cover.

Foundry Agent Architecture: The Core Components

A production agent on Azure AI Foundry consists of:

1. The Agent Runtime (Azure AI Agent Service)

Foundry's Agent Service manages the reasoning loop. You define the agent's instructions (system prompt), connect tools, and the service handles the orchestration — calling the LLM, parsing tool requests, executing tools, feeding results back, and continuing until the task is complete.

Key decisions at this layer:

  • Model selection — GPT-4.1 for complex reasoning, GPT-4.1-mini for high-volume/low-latency, GPT-4o for multimodal. The model determines cost, speed, and capability ceiling.
  • Temperature — 0.1 for deterministic operations (finance, compliance), 0.3-0.5 for creative tasks (drafting, summarization). Production agents are almost always low temperature.
  • Max iterations — cap the reasoning loop to prevent infinite tool-call chains. 10 iterations is typically sufficient; anything beyond 5 usually indicates a prompt or tool design problem.

2. Tools (Function Calling)

Tools are how agents interact with the world. In Foundry, you can connect tools via:

  • OpenAPI specification — point to any REST API with an OpenAPI spec. The agent discovers available operations and calls them with appropriate parameters.
  • Model Context Protocol (MCP) — connect to MCP servers (Stripe, GitHub, Snowflake, custom services) for standardized tool access.
  • Code Interpreter — the agent can write and execute Python code for data analysis, calculations, and file processing.
  • Azure Functions — serverless tool execution for custom business logic.

The architectural pattern that works at enterprise scale: one toolbox per domain. A finance agent has a finance toolbox (validate_invoice, detect_duplicates, analyze_payment_timing). A legal agent has a legal toolbox (read_contract, flag_risk_clauses, track_obligations). Each toolbox is a separate container or function app — independently deployable, scalable, and testable.

3. Knowledge Grounding

Agents need access to enterprise data to be useful. Foundry provides two patterns:

  • Azure AI Search (RAG) — for unstructured data: policies, contracts, manuals, emails. Vector search + semantic ranking gives the agent relevant context from your document corpus.
  • Structured data access — for databases, data warehouses, and APIs. The agent calls tools that query Snowflake, Databricks, or SQL directly. No data duplication required.

The critical design decision: don't move data into a vector store if the canonical source is a database. For structured queries ("what invoices are pending over $50K?"), have the agent call a tool that queries the database directly. Vector search is for unstructured knowledge — not transactional data.

4. Guardrails (Content Safety)

Production agents in regulated industries need guardrails on both input and output:

  • Input — jailbreak detection, prompt injection prevention, PII detection before the query reaches the model.
  • Output — PII filtering, content harm detection, hallucination detection (grounding checks), and task drift prevention.

Foundry provides Azure AI Content Safety as a built-in service. Configure it per agent — a customer-facing agent needs strict content filtering; an internal analyst agent can be more permissive.

5. Evaluation

You can't deploy what you can't measure. Foundry's evaluation framework supports:

  • Quality metrics — coherence, fluency, relevance, groundedness (does the response match the retrieved context?)
  • Safety metrics — violence, hate/unfairness, self-harm, sexual content scoring
  • Task-specific metrics — tool call accuracy (did it pick the right tool?), task completion (did it solve the problem?), intent resolution

Run evaluations on every agent version before promoting to production. Automate this in your CI/CD pipeline — agent code changes trigger an evaluation run, and deployment only proceeds if scores exceed thresholds.

Production Deployment Patterns

Pattern 1: Agent Per Domain

Deploy one agent per business domain — finance, legal, HR, supply chain. Each agent has its own tools, knowledge, system prompt, and guardrail configuration. This gives you:

  • Clear ownership (the finance team owns the finance agent)
  • Independent scaling (the customer service agent handles 10x the traffic of the audit agent)
  • Domain-specific guardrails (finance agent blocks any response containing account numbers)
  • Isolated blast radius (if the HR agent has a bug, finance keeps running)

Pattern 2: Gateway + Agent Pool

A single API gateway handles authentication, rate limiting, and routing. Behind it, a pool of agent instances serves requests. The gateway:

  • Validates user identity and permissions (can this user access this agent?)
  • Routes to the correct agent based on the request
  • Logs every interaction for governance
  • Enforces rate limits and cost budgets per user/department

Pattern 3: Scheduled Agents (Routines)

Not all agent interactions are user-initiated. Production agents run autonomously on schedules:

  • Daily: "Scan all pending invoices for duplicates and fraud indicators"
  • Weekly: "Generate the compliance status report for the audit committee"
  • Event-triggered: "A new regulation was published — assess business impact"

Use Azure Logic Apps or a dedicated orchestrator service to trigger agent executions on schedule. The agent runs the same way as an interactive session — it reasons, calls tools, generates output — but without a human waiting for the response.

Governance: The Non-Negotiable Layer

Enterprise agents without governance are liabilities. Every agent interaction must be:

  • Logged — who asked, what was asked, what tools were called, what data was accessed, what was returned, how many tokens, what cost.
  • Auditable — regulators and internal audit can review any interaction, trace the reasoning chain, and understand why the agent made a decision.
  • Cost-tracked — token usage and LLM cost per agent, per user, per department. No surprises on the Azure bill.
  • Access-controlled — not every user can access every agent. The finance agent is restricted to the finance team. The HR agent shows different data based on role.

Build governance from day one. It's exponentially harder to add after agents are already in production and users expect the current behavior.

Common Mistakes (and How to Avoid Them)

  1. Building a chatbot and calling it an agent. If it can't call tools and take actions, it's a chatbot with a fancy system prompt. Agents DO things.
  2. Putting all business logic in the system prompt. The system prompt defines personality and rules. Business logic belongs in tools. When the agent needs to "validate an invoice," it calls a tool — it doesn't try to do math in the prompt.
  3. RAG for everything. Vector search is for unstructured knowledge. Transactional queries ("show me pending invoices") should hit the database directly via tools. Don't vectorize your SQL tables.
  4. No evaluation before production. An agent that works in your demo doesn't mean it works with real data at scale. Evaluate with real queries, real edge cases, and adversarial inputs.
  5. Ignoring cost. A GPT-4.1 agent calling 5 tools per request at 4,000 tokens each costs ~$0.01 per interaction. At 10,000 requests/day that's $100/day. Plan for it.
  6. No human-in-the-loop for high-stakes actions. An agent that auto-approves invoices over $1M without human review will eventually approve a fraudulent one. Design approval gates based on risk.

Multi-Agent Architectures

As your agent portfolio grows, agents need to collaborate. A finance question might require the invoice agent to find the problem, the contract agent to check the agreement terms, and the compliance agent to verify the action is SOX-compliant.

Two patterns for multi-agent orchestration:

  • Orchestrator pattern — a master agent routes sub-tasks to specialist agents and assembles the final response. Simple but creates a single point of failure.
  • Agent-to-Agent (A2A) protocol — agents discover each other's capabilities and delegate tasks directly. More resilient but more complex to govern.

Microsoft is adopting the A2A protocol (originally from Google) in Foundry. This means your agents can be discoverable by other agents — both yours and your partners'. Design for interoperability from the start.

The Enterprise Agent Checklist

Before deploying any agent to production, verify:

  • ☐ System prompt defines clear boundaries (what the agent does and does NOT do)
  • ☐ Tools are well-defined with typed parameters and error handling
  • ☐ Knowledge sources are connected and returning current data
  • ☐ Guardrails configured (input: jailbreak, PII | output: PII, content safety)
  • ☐ Evaluation scores above threshold (relevance >4/5, groundedness >4/5)
  • ☐ Governance logging captures every interaction
  • ☐ RBAC restricts access to authorized users
  • ☐ Cost budget set per agent/department
  • ☐ Human-in-the-loop gates for high-stakes actions
  • ☐ Rollback plan if agent produces incorrect outputs

Getting Started

The fastest path from zero to production agent:

  1. Create an Azure AI Foundry project in your subscription
  2. Deploy a model (GPT-4.1-mini for development, GPT-4.1 for production)
  3. Define your agent's purpose in a system prompt (be specific — not "you are a helpful assistant")
  4. Build one tool that connects to real enterprise data
  5. Test in the Foundry playground with real questions from your domain
  6. Add guardrails and evaluation
  7. Deploy behind a gateway with governance logging
  8. Release to a pilot group of 5-10 users
  9. Measure, iterate, expand

The entire cycle from project creation to pilot users can take as little as 2-3 weeks with the right architecture. The organizations that struggle are the ones that try to boil the ocean — 50 agents across 10 departments in a single program. Start with one agent, one domain, one team. Prove value. Then scale.

Proxima Intelligence is a Microsoft Azure Partner specializing in enterprise AI agent deployments. We help organizations move from prototype to production — with the architecture, governance, and operational rigor that regulated industries require.

Talk to an architect about your agent strategy →