Uploaded July 2026 | Updated September 2026, 2 weeks ago
Developers cut AI token spend by compressing conversation history, capping output tokens with matching prompt instructions, caching static prefixes at roughly 10% of normal input cost, and routing each prompt to the right model by complexity.
Microsoft Foundry gives you Model Router, Toolbox, and Agent Optimizer to pick the best model per prompt, send only the tools a request actually needs, and test prompt, model, and tool configurations together.
April Gittens, Microsoft Principal Cloud Advocate, joins Jeremy Chapman, Microsoft 365 Director, to break down Tokenomics — how token pricing really works and how app design controls AI cost at scale.
👥 Who it's for: Developers and app architects building on Microsoft Foundry, IT and platform teams managing Azure AI spend, FinOps and cloud cost owners, and technical leads evaluating model choice, caching, and agent tooling for production workloads.
⏱️ Chapters:
0:00 Why AI costs spike as adoption scales
1:06 Input vs output tokens and how billing works
2:11 Context window creep inflates every turn
3:46 Trim system prompts, check base instructions
4:34 Summarize and dedupe conversation history
5:05 Store chat state externally, retrieve on demand
5:21 Cap output tokens with matching instructions
6:27 Cache static prefixes for 10% input cost
7:54 Compare model quality, cost and throughput
9:15 Model Router picks a model per prompt
9:42 Why tool definitions inflate input tokens
10:11 Toolbox cuts input tokens by 90%
11:29 Agent Optimizer tests prompts, models, tools
12:52 Data and retrieval quality drive total cost
13:12 Foundry IQ grounds enterprise knowledge sources
Tokens are the unit of text models process and bill against — a full word, part of a word, punctuation, even white space. Input and output tokens are priced separately per million, with output typically costing three to five times more because generating text takes more compute than reading it. Language models are stateless, so every turn resends the system prompt and the full conversation history. Tool outputs, retrieved RAG documents, and hidden metadata all count as input tokens too, which is how a short follow-up question quietly grows from 34 to 71 prompt tokens.
The fixes are design decisions. Keep system prompts concise and avoid duplicating instructions already baked into the model — a redundant 300-token preamble on every turn adds up to thousands of dollars per billing cycle at scale. Summarize and dedupe conversation history instead of resending raw transcripts, or store state in an external database like PostgreSQL and retrieve only what's needed. Set max completion tokens and match that limit with prompt instructions, or responses stop mid-sentence. Mark static prefixes — system prompts, tool definitions, session RAG documents — with cache control so the first request pays to write and later cache hits pay about 10% of normal input cost.
Model choice is the biggest lever. In Microsoft Foundry you can compare models side by side on quality, safety, estimated cost, and throughput, use the leaderboard to weigh quality against cost and latency, or hand the decision to Model Router in balanced, quality, or cost mode. Toolbox routes requests to the right tool instead of sending every definition — dropping one agent run from roughly 4,700 tokens to 467. Agent Optimizer then runs iterations with azd ai agent optimize to find better prompt, model, and tool configurations, lifting one agent from a 70% to 90% pass rate.
► Link References
Check out the tools in Microsoft Foundry at ai.azure.com
For more about managing AI costs go to https://aka.ms/FoundryTokenomics
► Unfamiliar with Microsoft Mechanics? Microsoft's Official Video Series for IT
- Subscribe youtube.com/c/MicrosoftMechanicsSeries
- Microsoft Tech Community: techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog
- Podcast: microsoftmechanics.libsyn.com/podcast
► Join us on social:
- twitter.com/MSFTMechanics
- linkedin.com/company/microsoft-mechanics
- instagram.com/msftmechanics
- tiktok.com/@msftmechanics
#MicrosoftMechanics #MicrosoftFoundry #Tokenomics #AICosts #GenerativeAI
Developers cut AI token spend by compressing conversation history, capping output tokens with matching prompt instructions, caching static prefixes at roughly 10% of normal input cost, and routing each prompt to the right model by complexity.
Microsoft Foundry gives you Model Router, Toolbox, and Agent Optimizer to pick the best model per prompt, send only the tools a request actually needs, and test prompt, model, and tool configurations together.
April Gittens, Microsoft Principal Cloud Advocate, joins Jeremy Chapman, Microsoft 365 Director, to break down Tokenomics — how token pricing really works and how app design controls AI cost at scale.
👥 Who it's for: Developers and app architects building on Microsoft Foundry, IT and platform teams managing Azure AI spend, FinOps and cloud cost owners, and technical leads evaluating model choice, caching, and agent tooling for production workloads.
⏱️ Chapters:
0:00 Why AI costs spike as adoption scales
1:06 Input vs output tokens and how billing works
2:11 Context window creep inflates every turn
3:46 Trim system prompts, check base instructions
4:34 Summarize and dedupe conversation history
5:05 Store chat state externally, retrieve on demand
5:21 Cap output tokens with matching instructions
6:27 Cache static prefixes for 10% input cost
7:54 Compare model quality, cost and throughput
9:15 Model Router picks a model per prompt
9:42 Why tool definitions inflate input tokens
10:11 Toolbox cuts input tokens by 90%
11:29 Agent Optimizer tests prompts, models, tools
12:52 Data and retrieval quality drive total cost
13:12 Foundry IQ grounds enterprise knowledge sources
Tokens are the unit of text models process and bill against — a full word, part of a word, punctuation, even white space. Input and output tokens are priced separately per million, with output typically costing three to five times more because generating text takes more compute than reading it. Language models are stateless, so every turn resends the system prompt and the full conversation history. Tool outputs, retrieved RAG documents, and hidden metadata all count as input tokens too, which is how a short follow-up question quietly grows from 34 to 71 prompt tokens.
The fixes are design decisions. Keep system prompts concise and avoid duplicating instructions already baked into the model — a redundant 300-token preamble on every turn adds up to thousands of dollars per billing cycle at scale. Summarize and dedupe conversation history instead of resending raw transcripts, or store state in an external database like PostgreSQL and retrieve only what's needed. Set max completion tokens and match that limit with prompt instructions, or responses stop mid-sentence. Mark static prefixes — system prompts, tool definitions, session RAG documents — with cache control so the first request pays to write and later cache hits pay about 10% of normal input cost.
Model choice is the biggest lever. In Microsoft Foundry you can compare models side by side on quality, safety, estimated cost, and throughput, use the leaderboard to weigh quality against cost and latency, or hand the decision to Model Router in balanced, quality, or cost mode. Toolbox routes requests to the right tool instead of sending every definition — dropping one agent run from roughly 4,700 tokens to 467. Agent Optimizer then runs iterations with azd ai agent optimize to find better prompt, model, and tool configurations, lifting one agent from a 70% to 90% pass rate.
► Link References
Check out the tools in Microsoft Foundry at ai.azure.com
For more about managing AI costs go to https://aka.ms/FoundryTokenomics
► Unfamiliar with Microsoft Mechanics? Microsoft's Official Video Series for IT
- Subscribe youtube.com/c/MicrosoftMechanicsSeries
- Microsoft Tech Community: techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog
- Podcast: microsoftmechanics.libsyn.com/podcast
► Join us on social:
- twitter.com/MSFTMechanics
- linkedin.com/company/microsoft-mechanics
- instagram.com/msftmechanics
- tiktok.com/@msftmechanics
#MicrosoftMechanics #MicrosoftFoundry #Tokenomics #AICosts #GenerativeAI
