Microsoft Foundry vs. Azure OpenAI Service — Clearing Up the Confusion

Last week, customer call. Architect from a mid-sized company asks: „We’ve been using Azure OpenAI for two years. Do we now have to migrate to Foundry — or can we ignore it?”

Three days later, different customer: „We use Document Intelligence for our invoices. Is that now Foundry too? Or does it belong somewhere else?”

I hear these questions every week. And I understand them. Microsoft has renamed its AI platform three times within 18 months — Azure AI Studio, Azure AI Foundry, now Microsoft Foundry. At the same time, „Cognitive Services” became „Azure AI Services”, and „Form Recognizer” became „Document Intelligence”. Anyone who’s lost here is not stupid. They’re just a normal IT pro with too many construction sites.

This article cleans it up. No marketing speak, no feature list. Three questions, three clear answers — and a decision tree you can use right after reading.

The big picture — Azure AI Services, OpenAI Service, Foundry

Before we compare, we need the hierarchy. Otherwise we’re comparing apples and oranges.

Layer 1 — Azure AI Services (the family of AI APIs)

A whole collection of specialized AI services, each with its own API, its own resource type, its own pricing model. The most important ones:

• Azure OpenAI Service — GPT-4o, o-series, embeddings, DALL-E
• Azure AI Document Intelligence — layout recognition, prebuilt models for invoices/IDs/business cards, custom models
• Azure AI Vision — image analysis, OCR, object detection
• Azure AI Speech — STT, TTS, translation, speaker recognition
• Azure AI Language — entity extraction, sentiment, summarization, PII detection
• Azure AI Search — hybrid search (keyword + vector + semantic ranking)
• Azure AI Content Safety — moderation, jailbreak detection, groundedness
• Azure AI Translator

You can call all of these directly via REST/SDK. They work completely without Foundry. Document Intelligence has been around since 2019, OpenAI Service since 2023 — Foundry is newer than both.

Layer 2 — Microsoft Foundry (the platform around it)

A UI, a hub concept, an agent service, a model catalog, prompt flow, evaluations, tracing. Foundry integrates the AI services via connections. In Foundry, you can build an agent that internally calls Document Intelligence, then an embeddings model, then GPT-4o — all orchestrated, evaluated, traced.

Where does Document Intelligence fit in?

Three patterns from real-world practice:

The rule: Document Intelligence becomes more important with every step towards „intelligent workflow”. Solo for structured extraction. In combination for understanding. In an agent for action.

When is Azure OpenAI Service enough?

Three conditions, all three must apply:

1. You only need OpenAI models. No Phi, no Llama, no open-source.
2. Your application code lives elsewhere — Function App, AKS, your own platform — and you build your pipeline, eval and monitoring logic yourself, or already have it.
3. Your use-case at its core is: „Prompt in, answer out, done.” Classic RAG, chatbot with your own backend, batch classification.

If that fits: no Foundry needed. Deploy OpenAI Service, get endpoint, done. You save yourself the hub/project hierarchy and the learning curve.

When do you need Foundry?

As soon as one of these things applies:

• You want to compare models — GPT-4o against Llama 3 against Phi-4 for the same use-case
• You build agents (not „chatbots”, real agents with tools, memory, multi-step reasoning)
• You need systematic evaluations — groundedness, coherence, custom metrics across test datasets
• You want tracing across multiple steps (which step hallucinated?)
• You orchestrate multiple AI services in a pipeline (Document Intelligence → embeddings → AI Search → GPT-4o)
• You want a UI for your data scientists, where they can manage connections, datasets and deployments without an IT ticket
• You need central governance across multiple AI projects (hub concept) with shared resources

Translated: Foundry pays off as soon as AI is more than just another API call in your app.

The comparison table you need

Three scenarios from my consulting work

Scenario 1: Classic M365 customer, looking for „chatbot for internal docs”

RAG pattern, GPT-4o, AI Search as vector store, Azure OpenAI Service directly. If the docs are only Word/PDF with standard text: without Document Intelligence. If the PDFs are scanned or contain tables that matter: Document Intelligence Layout as a pre-step. Foundry would be overkill. Setup done in a day, in production within two weeks.

Scenario 2: Insurance company, wants agent for claims triage

Foundry mandatory. Agent Service, Document Intelligence custom model for the claim forms as a tool, Language service for sentiment, GPT-4o for the final classification. Self-built with LangChain works too — but then you operate tracing, evaluation and tool-schema management yourself. In two years, you won’t want that anymore.

Scenario 3: SMB evaluating „do open-source LLMs save us money?”

Foundry. That’s exactly what the model catalog is for. Phi-4 locally, Llama 3 in Foundry, GPT-4o as baseline, all three compared with identical test prompts and eval dataset. In OpenAI Service alone, you can’t do that.

What Microsoft gets right here — and what not

Right: Foundry is the only platform on the market that has OpenAI models, open-source models, proprietary Microsoft models AND the specialized AI services (Document Intelligence, Vision, Speech, Language) under one roof — with enterprise governance. AWS Bedrock doesn’t give you that. Google Vertex AI doesn’t give you that.

Not right: The naming chaos. Azure AI Studio → Azure AI Foundry → Microsoft Foundry. Cognitive Services → Azure AI Services. Form Recognizer → Document Intelligence. Plus „Foundry Agent Service” vs. „Foundry Hub” vs. „Foundry Project” vs. the no-longer-existing „Azure AI Hub”. Anyone who consults with three customers on three different naming versions knows what I mean. Microsoft should clean this up once, then give us five years of peace.

My decision tree for you

Point 5 is my default recommendation for 80% of customers. You lose nothing, you gain optionality.

What you take with you

• Foundry does not replace AI services — Foundry orchestrates them
• Document Intelligence is a standalone AI service, not a Foundry feature
• Pure API scenario with OpenAI only → OpenAI Service is enough
• Structured doc extraction without magic → Document Intelligence solo
• As soon as multiple services interact, agents, evals or multi-model → Foundry
• Foundry platform costs nothing extra — you pay only for service calls
• EU Data Boundary for catalog models (Llama, Mistral) check individually
• When unsure: start with Foundry Project + OpenAI deployment — maximum optionality

Written by Ferdi Lethen-Oellers Microsoft MVP

Senior Modern Workplace Consultant at amexus · Speaker · Author of “Microsoft 365 Administration for Dummies”.

#EspressoM365Fusion