Microsoft AI Gateway - Simply Explained

1
00:00:00,000 --> 00:00:03,000
So what exactly changes when a company moves from one AI chatbot

2
00:00:03,000 --> 00:00:04,480
to dozens of AI tools?

3
00:00:04,480 --> 00:00:06,240
At first, it feels simple.

4
00:00:06,240 --> 00:00:09,120
One team adds a chatbot for customer support,

5
00:00:09,120 --> 00:00:11,800
another uses AI to search company documents,

6
00:00:11,800 --> 00:00:14,440
and meeting notes get written automatically.

7
00:00:14,440 --> 00:00:16,840
A workflow can read an email, pull out a request,

8
00:00:16,840 --> 00:00:18,440
and send it to the right system.

9
00:00:18,440 --> 00:00:21,000
Before long, the company has several models, agents

10
00:00:21,000 --> 00:00:24,320
that can take actions, and tools that reach into business data.

11
00:00:24,320 --> 00:00:27,320
Each team often connects straight to a different AI provider,

12
00:00:27,320 --> 00:00:29,640
using its own key, its own limits,

13
00:00:29,640 --> 00:00:30,880
and its own safety rules.

14
00:00:30,880 --> 00:00:32,400
Costs become hard to trace.

15
00:00:32,400 --> 00:00:34,080
A key can end up in the wrong place.

16
00:00:34,080 --> 00:00:35,840
And when an agent calls a tool, nobody

17
00:00:35,840 --> 00:00:38,040
may have one clear record of what it tried to do.

18
00:00:38,040 --> 00:00:40,240
This is where Microsoft AI Gateway comes in.

19
00:00:40,240 --> 00:00:42,000
By the end of this episode, you'll understand

20
00:00:42,000 --> 00:00:44,200
what it is, how its building blocks fit together,

21
00:00:44,200 --> 00:00:46,360
and when it starts making sense for your business.

22
00:00:46,360 --> 00:00:49,400
Think of it like an office building with lots of side doors.

23
00:00:49,400 --> 00:00:50,600
You could leave every door open

24
00:00:50,600 --> 00:00:52,240
and hope people use them properly,

25
00:00:52,240 --> 00:00:55,520
or you can bring everyone through one managed entrance.

26
00:00:55,520 --> 00:00:57,000
One front door for AI.

27
00:00:57,000 --> 00:01:00,720
Microsoft AI Gateway is a set of Azure API management features

28
00:01:00,720 --> 00:01:03,280
that gives your AI traffic one control point.

29
00:01:03,280 --> 00:01:04,960
Let's break it down in plain English.

30
00:01:04,960 --> 00:01:07,840
Imagine your company's AI setup as a modern office building.

31
00:01:07,840 --> 00:01:10,600
The AI models are rooms where the thinking happens.

32
00:01:10,600 --> 00:01:12,720
Your agents are workers moving through the building,

33
00:01:12,720 --> 00:01:14,760
deciding which approved task to do next.

34
00:01:14,760 --> 00:01:17,880
Tools are service doors that lead to places like HR,

35
00:01:17,880 --> 00:01:20,840
finance, customer records, and internal files.

36
00:01:20,840 --> 00:01:22,720
The AI Gateway is the reception desk

37
00:01:22,720 --> 00:01:24,760
and the security desk at the front.

38
00:01:24,760 --> 00:01:26,240
It checks who is coming in,

39
00:01:26,240 --> 00:01:28,720
applies the rules and records where requests go

40
00:01:28,720 --> 00:01:31,080
before they reach the rest of the building.

41
00:01:31,080 --> 00:01:33,000
A request follows a simple path.

42
00:01:33,000 --> 00:01:36,720
An app, a co-pilot, or an agent sends a request to the gateway.

43
00:01:36,720 --> 00:01:39,360
The gateway checks the rules that apply to that caller,

44
00:01:39,360 --> 00:01:43,800
then sends it to the approved model, tool, agent, or business API.

45
00:01:43,800 --> 00:01:47,120
And when a response comes back, it passes through the gateway again

46
00:01:47,120 --> 00:01:48,800
before it reaches the user.

47
00:01:48,800 --> 00:01:51,760
That one path gives the business a place to manage access,

48
00:01:51,760 --> 00:01:53,680
limits, safety checks, and logging

49
00:01:53,680 --> 00:01:56,560
without building those controls into every single app.

50
00:01:56,560 --> 00:01:59,560
This also separates two jobs that often get mixed together.

51
00:01:59,560 --> 00:02:02,800
Application teams can focus on the experience people actually use.

52
00:02:02,800 --> 00:02:05,360
They build the support assistant, the document search screen,

53
00:02:05,360 --> 00:02:07,240
or the workflow that saves time.

54
00:02:07,240 --> 00:02:09,240
Meanwhile, the platform team manages

55
00:02:09,240 --> 00:02:13,000
the shared rules behind the scenes, deciding which services are approved,

56
00:02:13,000 --> 00:02:14,800
which identities can use them,

57
00:02:14,800 --> 00:02:17,600
how much traffic a project can send, and what gets logged.

58
00:02:17,600 --> 00:02:20,880
You don't want every team inventing a new security plan for every chatbot.

59
00:02:20,880 --> 00:02:23,360
Microsoft AI Gateway can sit in front of several kinds

60
00:02:23,360 --> 00:02:24,280
of AI traffic.

61
00:02:24,280 --> 00:02:27,480
It can manage calls to AI models and model context protocol,

62
00:02:27,480 --> 00:02:29,080
MCP tool calls.

63
00:02:29,080 --> 00:02:32,160
It can also manage agent to agent, A2A traffic,

64
00:02:32,160 --> 00:02:34,480
and the business APIs your company already uses,

65
00:02:34,480 --> 00:02:37,920
such as an API for checking an order, reading an HR policy,

66
00:02:37,920 --> 00:02:39,320
or opening a support ticket.

67
00:02:39,320 --> 00:02:41,360
That last part matters because most companies

68
00:02:41,360 --> 00:02:44,360
don't need to throw away their existing systems to use AI.

69
00:02:44,360 --> 00:02:47,200
They just need a safe way for AI to work with those systems,

70
00:02:47,200 --> 00:02:50,200
and a gateway helps put controls around that connection.

71
00:02:50,200 --> 00:02:52,480
Now Microsoft AI gateway isn't a model.

72
00:02:52,480 --> 00:02:53,800
It doesn't write answers itself.

73
00:02:53,800 --> 00:02:55,040
It isn't an agent builder either,

74
00:02:55,040 --> 00:02:57,080
so you don't use it to design an HR assistant

75
00:02:57,080 --> 00:02:58,720
or teach a finance agent what to do.

76
00:02:58,720 --> 00:03:00,760
And it doesn't replace Microsoft Foundry,

77
00:03:00,760 --> 00:03:03,440
where teams can build, test, and manage AI projects.

78
00:03:03,440 --> 00:03:05,280
Instead, it sits in the middle.

79
00:03:05,280 --> 00:03:07,480
Foundry can help you work with AI models and agents

80
00:03:07,480 --> 00:03:11,160
while Azure API Management AI Gateway helps control the traffic

81
00:03:11,160 --> 00:03:13,560
those models, agents, and tools create.

82
00:03:13,560 --> 00:03:15,480
You may already know Azure API management

83
00:03:15,480 --> 00:03:17,600
as a way to manage normal APIs.

84
00:03:17,600 --> 00:03:19,880
An API is simply a way for one piece of software

85
00:03:19,880 --> 00:03:22,440
to ask another piece of software for something.

86
00:03:22,440 --> 00:03:25,040
AI Gateway extends that same idea for AI work

87
00:03:25,040 --> 00:03:27,080
because AI requests bring new questions.

88
00:03:27,080 --> 00:03:28,120
Who used which model?

89
00:03:28,120 --> 00:03:29,760
How many tokens did it consume?

90
00:03:29,760 --> 00:03:31,440
Can this agent call that tool?

91
00:03:31,440 --> 00:03:33,320
Should this prompt reach the model at all?

92
00:03:33,320 --> 00:03:35,560
Putting the gateway in the middle gives you one place

93
00:03:35,560 --> 00:03:37,960
to answer those questions, and the first pressure point

94
00:03:37,960 --> 00:03:39,680
appears when every application starts

95
00:03:39,680 --> 00:03:42,040
choosing its own model endpoint.

96
00:03:42,040 --> 00:03:44,280
One endpoint, many models.

97
00:03:44,280 --> 00:03:45,960
When your application connects directly

98
00:03:45,960 --> 00:03:48,120
to a single model endpoint, that choice

99
00:03:48,120 --> 00:03:50,280
gets baked into the code itself.

100
00:03:50,280 --> 00:03:52,360
Often the code is tied to one provider

101
00:03:52,360 --> 00:03:54,920
one specific deployment, one Azure region,

102
00:03:54,920 --> 00:03:56,360
and a fixed set of limits.

103
00:03:56,360 --> 00:03:57,960
That works fine while the project is small,

104
00:03:57,960 --> 00:04:00,360
but imagine your support assistant runs on one model

105
00:04:00,360 --> 00:04:01,280
in one region.

106
00:04:01,280 --> 00:04:03,480
Months later, that model hits its capacity.

107
00:04:03,480 --> 00:04:05,480
A better model comes along for less money,

108
00:04:05,480 --> 00:04:07,680
or your business needs to move to another region.

109
00:04:07,680 --> 00:04:11,200
Without a shared layer, every app team has to change their code,

110
00:04:11,200 --> 00:04:13,320
test it again, and roll out an update.

111
00:04:13,320 --> 00:04:15,240
That makes for slow, messy change.

112
00:04:15,240 --> 00:04:18,640
Microsoft AI Gateway gives your applications a stable front door.

113
00:04:18,640 --> 00:04:19,960
They send requests there,

114
00:04:19,960 --> 00:04:21,800
and the model choice can change behind the scenes

115
00:04:21,800 --> 00:04:23,400
without you having to touch the app.

116
00:04:23,400 --> 00:04:25,120
Think of it like a company phone number.

117
00:04:25,120 --> 00:04:26,760
Customers dial one number,

118
00:04:26,760 --> 00:04:29,240
even if the call gets passed to a different team or office.

119
00:04:29,240 --> 00:04:30,920
They never need to know how it's handled.

120
00:04:30,920 --> 00:04:33,720
With AI Gateway, your app calls one approved endpoint,

121
00:04:33,720 --> 00:04:35,040
and behind that endpoint,

122
00:04:35,040 --> 00:04:37,920
the gateway sends the request to the model that fits the job.

123
00:04:37,920 --> 00:04:39,520
Microsoft also has a new approach

124
00:04:39,520 --> 00:04:41,840
called the unified model API.

125
00:04:41,840 --> 00:04:44,200
In plain English, it lets an application

126
00:04:44,200 --> 00:04:47,040
use an open AI style way of making a request,

127
00:04:47,040 --> 00:04:48,920
while the gateway can connect that request

128
00:04:48,920 --> 00:04:51,280
to different model providers behind the scenes.

129
00:04:51,280 --> 00:04:54,320
So your app sends one familiar kind of request,

130
00:04:54,320 --> 00:04:57,000
and the gateway handles all the provider details.

131
00:04:57,000 --> 00:04:57,960
Depending on your setup,

132
00:04:57,960 --> 00:05:00,440
that request could go to an Azure OpenAI deployment,

133
00:05:00,440 --> 00:05:02,160
a model in Microsoft Foundry,

134
00:05:02,160 --> 00:05:05,040
and Thropic, Google Gemini, or Amazon Bedrock.

135
00:05:05,040 --> 00:05:06,360
The root follows rules you set,

136
00:05:06,360 --> 00:05:09,080
not a hard coded choice buried inside every application.

137
00:05:09,080 --> 00:05:11,480
That gives the platform team room to make changes

138
00:05:11,480 --> 00:05:13,720
without disturbing the people who use the app.

139
00:05:13,720 --> 00:05:16,760
Say your company runs an AI assistant for customer support.

140
00:05:16,760 --> 00:05:18,280
Most questions are simple.

141
00:05:18,280 --> 00:05:19,560
Where is my order?

142
00:05:19,560 --> 00:05:21,240
How do I reset my password?

143
00:05:21,240 --> 00:05:23,920
Those can be answered by a smaller, cheaper model quickly,

144
00:05:23,920 --> 00:05:25,520
without spending more than needed.

145
00:05:25,520 --> 00:05:27,600
Then a customer sends a long, unusual question

146
00:05:27,600 --> 00:05:29,840
that needs careful reasoning or product knowledge.

147
00:05:29,840 --> 00:05:32,720
The gateway routes that type of request to a stronger model.

148
00:05:32,720 --> 00:05:34,800
The support app still sends requests the same way,

149
00:05:34,800 --> 00:05:37,640
so your team never needs to know which model handled the question,

150
00:05:37,640 --> 00:05:39,200
and your developers don't have to wire

151
00:05:39,200 --> 00:05:41,160
up separate connections for every option.

152
00:05:41,160 --> 00:05:43,760
What about capacity?

153
00:05:43,760 --> 00:05:45,480
AI model endpoints have limits.

154
00:05:45,480 --> 00:05:47,320
They can only handle a certain amount of traffic

155
00:05:47,320 --> 00:05:48,560
during a period of time.

156
00:05:48,560 --> 00:05:50,440
If every request hits one deployment,

157
00:05:50,440 --> 00:05:53,640
a single busy hour can slow down or block the whole experience.

158
00:05:53,640 --> 00:05:56,760
A gateway can spread traffic across multiple approved deployments,

159
00:05:56,760 --> 00:05:58,920
rooting to another region when one fills up

160
00:05:58,920 --> 00:06:01,440
or using a backup route if an endpoint goes down.

161
00:06:01,440 --> 00:06:03,720
That doesn't mean requests should bounce around randomly.

162
00:06:03,720 --> 00:06:04,680
You still set the rules.

163
00:06:04,680 --> 00:06:07,720
For example, customer facing traffic gets priority

164
00:06:07,720 --> 00:06:09,960
on your main model while internal testing

165
00:06:09,960 --> 00:06:13,720
uses a different path, or one region handles local users

166
00:06:13,720 --> 00:06:16,160
while another acts as a backup.

167
00:06:16,160 --> 00:06:17,320
The point is simple.

168
00:06:17,320 --> 00:06:20,160
Your applications connect to one controlled front door,

169
00:06:20,160 --> 00:06:22,760
and behind it the platform can adjust models, regions,

170
00:06:22,760 --> 00:06:24,720
or providers with minimal disruption.

171
00:06:24,720 --> 00:06:26,520
But that matters because AI moves fast.

172
00:06:26,520 --> 00:06:29,320
A model that works today might not be the best choice next month.

173
00:06:29,320 --> 00:06:31,440
If every app talks directly to its provider,

174
00:06:31,440 --> 00:06:33,720
changing models becomes a project for every team.

175
00:06:33,720 --> 00:06:37,200
With the gateway, the model decision lives in one shared place.

176
00:06:37,200 --> 00:06:39,880
A quick note on the unified model API.

177
00:06:39,880 --> 00:06:42,240
Microsoft lists it as public preview.

178
00:06:42,240 --> 00:06:43,480
Preview features can change,

179
00:06:43,480 --> 00:06:45,960
and they may not be ready for every production use case.

180
00:06:45,960 --> 00:06:48,680
Still, the larger pattern doesn't depend on that one feature.

181
00:06:48,680 --> 00:06:51,840
As your API management can already sit in front of model endpoints,

182
00:06:51,840 --> 00:06:55,200
apply routing rules, and help you manage more than one back end.

183
00:06:55,200 --> 00:06:59,960
The unified model API aims to make that cross-provider experience easier for applications.

184
00:06:59,960 --> 00:07:01,800
Choosing the right model is only half the job

185
00:07:01,800 --> 00:07:03,680
because a model can answer a question,

186
00:07:03,680 --> 00:07:05,920
but an agent can decide to take an action

187
00:07:05,920 --> 00:07:08,360
that reaches into real company systems.

188
00:07:08,360 --> 00:07:09,920
Governing agents and tools.

189
00:07:09,920 --> 00:07:13,120
An agent takes AI beyond just answering questions.

190
00:07:13,120 --> 00:07:16,240
It's software that can look at a task, choose from approved options,

191
00:07:16,240 --> 00:07:18,480
and call a tool or service to get work done.

192
00:07:18,480 --> 00:07:20,760
An HR agent might look up a leave policy,

193
00:07:20,760 --> 00:07:22,400
a support agent might check an order,

194
00:07:22,400 --> 00:07:24,960
and a finance agent might prepare payment request.

195
00:07:24,960 --> 00:07:26,200
That ability is useful,

196
00:07:26,200 --> 00:07:28,040
but it also changes the risk profile.

197
00:07:28,040 --> 00:07:30,160
A wrong answer from a model confuses someone,

198
00:07:30,160 --> 00:07:32,800
but an agent using the wrong tool can expose data,

199
00:07:32,800 --> 00:07:36,080
change a record, or trigger a process it should never touch.

200
00:07:36,080 --> 00:07:38,160
This is why tools need rules of their own.

201
00:07:38,160 --> 00:07:41,200
You'll hear the term MCP or model context protocol.

202
00:07:41,200 --> 00:07:42,800
Think of it as a standard connector shape

203
00:07:42,800 --> 00:07:44,640
that gives an agent a known way to find

204
00:07:44,640 --> 00:07:46,880
a user company tool like a document search service

205
00:07:46,880 --> 00:07:48,440
or order lookup API.

206
00:07:48,440 --> 00:07:49,920
Rather than building a different connection

207
00:07:49,920 --> 00:07:51,600
for every agent in every system,

208
00:07:51,600 --> 00:07:53,560
MCP gives teams a common pattern.

209
00:07:53,560 --> 00:07:55,800
Then there's A to A, agent to agent,

210
00:07:55,800 --> 00:07:58,120
a standard way for one agent to contact another.

211
00:07:58,120 --> 00:08:00,400
For example, a support agent asks a shipping agent

212
00:08:00,400 --> 00:08:01,840
for delivery details instead of trying

213
00:08:01,840 --> 00:08:03,640
to understand the shipping system itself.

214
00:08:03,640 --> 00:08:05,120
That sounds tidy on paper,

215
00:08:05,120 --> 00:08:06,760
but an agent chain gets complicated

216
00:08:06,760 --> 00:08:08,880
when every connection has different access rules,

217
00:08:08,880 --> 00:08:10,920
hidden credentials, and unknown tools.

218
00:08:10,920 --> 00:08:13,560
Imagine a finance agent that answers invoice questions.

219
00:08:13,560 --> 00:08:16,040
It should only use an approved invoice lookup tool,

220
00:08:16,040 --> 00:08:18,480
but someone gives it broad access to the finance system

221
00:08:18,480 --> 00:08:21,040
because it's faster than defining a narrow tool.

222
00:08:21,040 --> 00:08:22,800
A user sends a harmless looking prompt,

223
00:08:22,800 --> 00:08:24,840
but hidden instructions in an attached document

224
00:08:24,840 --> 00:08:26,920
push the agent toward an unapproved action.

225
00:08:26,920 --> 00:08:28,720
If the agent has too much access,

226
00:08:28,720 --> 00:08:30,600
it might request unnecessary data

227
00:08:30,600 --> 00:08:33,040
or start a workflow that should need human approval.

228
00:08:33,040 --> 00:08:34,440
The agent isn't malicious.

229
00:08:34,440 --> 00:08:36,960
It just had too much freedom when handling untrusted input.

230
00:08:36,960 --> 00:08:39,400
Microsoft AI Gateway lets you put controls

231
00:08:39,400 --> 00:08:42,360
in front of MCP servers and business APIs.

232
00:08:42,360 --> 00:08:45,040
Instead of a direct path from an AI app to a database,

233
00:08:45,040 --> 00:08:48,120
you expose a small control tool for one specific job.

234
00:08:48,120 --> 00:08:49,560
For example, an agent asks,

235
00:08:49,560 --> 00:08:51,320
check the status of this order.

236
00:08:51,320 --> 00:08:53,000
The tool returns just the status,

237
00:08:53,000 --> 00:08:55,360
not every order or the full customer record.

238
00:08:55,360 --> 00:08:56,400
It's a narrow job,

239
00:08:56,400 --> 00:08:59,280
and the gateway checks who's asking before allowing the call.

240
00:08:59,280 --> 00:09:01,480
Identity sits at the center of the setup.

241
00:09:01,480 --> 00:09:03,640
With Microsoft Enter ID, apps sign in

242
00:09:03,640 --> 00:09:06,680
through managed identities instead of sharing provider keys.

243
00:09:06,680 --> 00:09:08,680
A managed identity proves who it is

244
00:09:08,680 --> 00:09:11,400
without a developer copying a secret into code.

245
00:09:11,400 --> 00:09:13,360
Each agent should have its own identity.

246
00:09:13,360 --> 00:09:15,360
That way you can see which agent called a tool,

247
00:09:15,360 --> 00:09:17,640
revoke access for one agent without breaking others

248
00:09:17,640 --> 00:09:18,680
and apply different rules.

249
00:09:18,680 --> 00:09:21,480
An HR agent shouldn't inherit a finance agent's permissions

250
00:09:21,480 --> 00:09:23,200
just because both use AI.

251
00:09:23,200 --> 00:09:24,520
This follows zero trust.

252
00:09:24,520 --> 00:09:25,800
Check every request,

253
00:09:25,800 --> 00:09:27,840
give only the access needed for the current job

254
00:09:27,840 --> 00:09:30,400
and assume any prompt or tool response could be unsafe.

255
00:09:30,400 --> 00:09:31,920
In practice, the agent gets permission

256
00:09:31,920 --> 00:09:33,920
for one specific tool in action.

257
00:09:33,920 --> 00:09:35,160
It can read a leave balance,

258
00:09:35,160 --> 00:09:36,840
but not download the payroll file.

259
00:09:36,840 --> 00:09:39,760
It can create a draft support ticket, but not closer case.

260
00:09:39,760 --> 00:09:41,600
The gateway also applies safety checks

261
00:09:41,600 --> 00:09:43,280
to prompts and tool traffic.

262
00:09:43,280 --> 00:09:45,360
If a request contains harmful content or data

263
00:09:45,360 --> 00:09:46,760
that shouldn't leave the system,

264
00:09:46,760 --> 00:09:48,840
a policy stops it before reaching the backend.

265
00:09:48,840 --> 00:09:50,280
No filter catches everything,

266
00:09:50,280 --> 00:09:52,560
but checking traffic at a shared point is much safer

267
00:09:52,560 --> 00:09:55,320
than hoping each agent builder adds the same protection.

268
00:09:55,320 --> 00:09:57,640
You also need a catalog of approved tools.

269
00:09:57,640 --> 00:10:01,120
API Center acts as a private catalog for APIs and MCP servers,

270
00:10:01,120 --> 00:10:02,360
so teams find the right tool

271
00:10:02,360 --> 00:10:05,400
without searching all docs or copying unverified web addresses

272
00:10:05,400 --> 00:10:08,080
that helps turn which tool can my agent use

273
00:10:08,080 --> 00:10:09,880
into a controlled business decision.

274
00:10:09,880 --> 00:10:11,440
Security controls protect the path,

275
00:10:11,440 --> 00:10:13,160
but they also create a new question,

276
00:10:13,160 --> 00:10:14,760
which teams are using that path

277
00:10:14,760 --> 00:10:17,680
and how much AI usage are they creating.

278
00:10:17,680 --> 00:10:19,800
Token costs, limits and visibility.

279
00:10:19,800 --> 00:10:22,680
AI costs work differently than a normal software license.

280
00:10:22,680 --> 00:10:24,040
The meter runs on tokens.

281
00:10:24,040 --> 00:10:24,880
What's a token?

282
00:10:24,880 --> 00:10:26,040
It's a small piece of text.

283
00:10:26,040 --> 00:10:27,600
A short word might use one token.

284
00:10:27,600 --> 00:10:29,000
A longer word uses more.

285
00:10:29,000 --> 00:10:30,840
The model counts everything you send in.

286
00:10:30,840 --> 00:10:33,000
Your question, the instructions you add,

287
00:10:33,000 --> 00:10:35,320
any documents you attach and the answer it gives back

288
00:10:35,320 --> 00:10:37,200
so a simple question uses very little.

289
00:10:37,200 --> 00:10:39,280
But an agent that reads a long policy document

290
00:10:39,280 --> 00:10:41,120
asks follow-up questions, calls a tool

291
00:10:41,120 --> 00:10:42,920
and then creates a detailed answer

292
00:10:42,920 --> 00:10:44,320
can use far more than you expect.

293
00:10:44,320 --> 00:10:46,280
You've probably seen this with a meeting transcript.

294
00:10:46,280 --> 00:10:47,960
You see a few pages of notes.

295
00:10:47,960 --> 00:10:50,400
The AI system also processes its system instructions,

296
00:10:50,400 --> 00:10:52,440
the transcript, retrieved company data,

297
00:10:52,440 --> 00:10:54,240
tool results and the final response.

298
00:10:54,240 --> 00:10:55,960
Every part adds to the usage.

299
00:10:55,960 --> 00:10:58,040
That's why one busy agent can eat up a shared model

300
00:10:58,040 --> 00:10:59,640
limits surprisingly fast.

301
00:10:59,640 --> 00:11:01,920
Imagine a team testing a new marketing assistant

302
00:11:01,920 --> 00:11:03,080
on a Friday afternoon.

303
00:11:03,080 --> 00:11:05,360
Someone gives it a large set of campaign files

304
00:11:05,360 --> 00:11:08,520
and asks it to compare, rewrite and summarize them in a loop.

305
00:11:08,520 --> 00:11:10,400
Meanwhile, customer support depends

306
00:11:10,400 --> 00:11:12,880
on the same model capacity for people trying to get help.

307
00:11:12,880 --> 00:11:15,120
If the test workload uses the shared quota,

308
00:11:15,120 --> 00:11:17,240
the support assistant may slow down or fail

309
00:11:17,240 --> 00:11:18,240
when customers need it.

310
00:11:18,240 --> 00:11:20,640
The problem isn't just that the company spent money.

311
00:11:20,640 --> 00:11:22,720
The wrong workload took capacity from the workload

312
00:11:22,720 --> 00:11:23,760
that mattered most.

313
00:11:23,760 --> 00:11:26,760
Azure API management AI gateway can apply limits

314
00:11:26,760 --> 00:11:28,080
before that happens.

315
00:11:28,080 --> 00:11:30,480
You can set request limits, token limits

316
00:11:30,480 --> 00:11:33,960
and quotas for an app, team, project or consumer.

317
00:11:33,960 --> 00:11:37,080
A quota can cover a short window, tokens per minute

318
00:11:37,080 --> 00:11:39,960
or a longer period like a daily or monthly allowance.

319
00:11:39,960 --> 00:11:41,720
That gives each workload a boundary.

320
00:11:41,720 --> 00:11:44,760
The marketing assistant can have a budget that fits its purpose.

321
00:11:44,760 --> 00:11:46,520
Customer support can have its own allowance

322
00:11:46,520 --> 00:11:48,080
and a higher priority.

323
00:11:48,080 --> 00:11:49,880
When the marketing assistant reaches its limit,

324
00:11:49,880 --> 00:11:52,560
the gateway can reject or slow its next requests.

325
00:11:52,560 --> 00:11:53,840
Customer support keeps running

326
00:11:53,840 --> 00:11:56,360
because one test project can't use everything available.

327
00:11:56,360 --> 00:11:58,840
This also changes the monthly cost conversation.

328
00:11:58,840 --> 00:12:00,560
Without a shared view, a finance team

329
00:12:00,560 --> 00:12:02,720
may receive one large AI bill and ask,

330
00:12:02,720 --> 00:12:04,240
who used all of this?

331
00:12:04,240 --> 00:12:05,600
The answer can be hard to find

332
00:12:05,600 --> 00:12:07,600
when teams connect directly to different services

333
00:12:07,600 --> 00:12:08,720
with shared keys.

334
00:12:08,720 --> 00:12:11,520
A gateway can record the caller, the model, token use,

335
00:12:11,520 --> 00:12:14,360
response status and the tool calls linked to a request.

336
00:12:14,360 --> 00:12:15,800
That gives you a clearer trail.

337
00:12:15,800 --> 00:12:19,000
You can see whether the spending came from an HR assistant,

338
00:12:19,000 --> 00:12:20,680
a developer test environment,

339
00:12:20,680 --> 00:12:22,880
a customer facing chatbot or an agent

340
00:12:22,880 --> 00:12:25,120
that kept retrying after an error.

341
00:12:25,120 --> 00:12:26,800
You can also spot unusual patterns

342
00:12:26,800 --> 00:12:28,640
like one application sending much larger prompts

343
00:12:28,640 --> 00:12:29,480
than normal.

344
00:12:29,480 --> 00:12:31,160
If your teams start in Microsoft Foundry,

345
00:12:31,160 --> 00:12:33,360
the gateway can connect to that work too.

346
00:12:33,360 --> 00:12:35,640
Foundry projects can use project-level token limits

347
00:12:35,640 --> 00:12:38,000
while the gateway applies the traffic rules around them.

348
00:12:38,000 --> 00:12:40,400
That gives AI builders a simpler starting point

349
00:12:40,400 --> 00:12:42,280
while the platform team still has a shared place

350
00:12:42,280 --> 00:12:43,960
to manage limits and monitoring.

351
00:12:43,960 --> 00:12:46,560
There's also a useful feature called semantic caching.

352
00:12:46,560 --> 00:12:49,960
Say one employee asks, where can I find the travel policy?

353
00:12:49,960 --> 00:12:51,840
A few seconds later, another asks,

354
00:12:51,840 --> 00:12:54,120
where is our policy for business trips?

355
00:12:54,120 --> 00:12:56,440
The words differ, but the meaning may be close enough

356
00:12:56,440 --> 00:12:58,880
that the gateway can reuse a suitable earlier answer

357
00:12:58,880 --> 00:13:00,720
rather than calling the model again.

358
00:13:00,720 --> 00:13:02,800
That can reduce cost and speed up the response

359
00:13:02,800 --> 00:13:04,920
so it only fits questions where reuse makes sense.

360
00:13:04,920 --> 00:13:07,560
You wouldn't cash a request for someone's current leave balance

361
00:13:07,560 --> 00:13:10,320
or an order status because those answers can change.

362
00:13:10,320 --> 00:13:12,120
Routing can help with cost as well.

363
00:13:12,120 --> 00:13:14,760
A company may send suitable work to a lower cost model,

364
00:13:14,760 --> 00:13:17,040
use capacity it already reserved before paying

365
00:13:17,040 --> 00:13:19,520
for extra usage and switch to a backup path

366
00:13:19,520 --> 00:13:20,920
when a limit appears.

367
00:13:20,920 --> 00:13:23,720
Still, a gateway doesn't remove the need for ownership.

368
00:13:23,720 --> 00:13:26,640
Teams need budgets, someone needs to review model answers,

369
00:13:26,640 --> 00:13:29,480
someone needs to decide whether an agent should exist at all.

370
00:13:29,480 --> 00:13:31,520
The gateway gives you the meter, the guardrails,

371
00:13:31,520 --> 00:13:32,880
and the record of traffic.

372
00:13:32,880 --> 00:13:34,760
It doesn't make business decisions for you.

373
00:13:34,760 --> 00:13:36,680
Once you can see and control the traffic,

374
00:13:36,680 --> 00:13:40,000
the whole setup becomes easier to picture in one everyday example.

375
00:13:40,000 --> 00:13:41,360
A simple request journey.

376
00:13:41,360 --> 00:13:43,520
Picture an HR assistant inside a company.

377
00:13:43,520 --> 00:13:45,880
An employee asks, how many days of parental leave

378
00:13:45,880 --> 00:13:48,160
can I take and can you start the request for me?

379
00:13:48,160 --> 00:13:50,880
The aim is simple, give the employee a fast useful answer

380
00:13:50,880 --> 00:13:52,760
then help with the next approved step,

381
00:13:52,760 --> 00:13:55,080
but the HR assistant must not hand out payroll data,

382
00:13:55,080 --> 00:13:58,760
open every HR file or carry a model provider key inside its code.

383
00:13:58,760 --> 00:14:00,840
The request starts when the employee signs in.

384
00:14:00,840 --> 00:14:03,400
Before the assistant sends anything to a model or a tool,

385
00:14:03,400 --> 00:14:05,600
the gateway checks who the employee is,

386
00:14:05,600 --> 00:14:07,280
whether they can use the HR assistant

387
00:14:07,280 --> 00:14:08,920
and which limits apply to that app.

388
00:14:08,920 --> 00:14:10,960
The employee sees one simple chat window

389
00:14:10,960 --> 00:14:12,920
while these checks happen behind the scenes.

390
00:14:12,920 --> 00:14:14,600
Next, the assistant needs an answer.

391
00:14:14,600 --> 00:14:18,400
A basic policy question can go to the company's approved model endpoint.

392
00:14:18,400 --> 00:14:21,480
If that endpoint is busy or reaches a set limit,

393
00:14:21,480 --> 00:14:23,240
the gateway can follow a backup route

394
00:14:23,240 --> 00:14:25,280
that the company already approved.

395
00:14:25,280 --> 00:14:28,120
The employee still gets an answer through the same HR assistant,

396
00:14:28,120 --> 00:14:31,720
rather than seeing an error because one model deployment became too busy.

397
00:14:31,720 --> 00:14:33,960
Then the employee asks to start the leave request.

398
00:14:33,960 --> 00:14:36,280
At this point, the agent needs more than a model.

399
00:14:36,280 --> 00:14:38,480
It needs a tool that can check the employee's details

400
00:14:38,480 --> 00:14:40,000
and create the right request.

401
00:14:40,000 --> 00:14:43,960
The agent asks to use an approved MCP tool for leave management.

402
00:14:43,960 --> 00:14:45,440
The gateway checks that request too.

403
00:14:45,440 --> 00:14:47,560
Can this HR agent call this tool?

404
00:14:47,560 --> 00:14:50,280
Is this employee allowed to start this type of request?

405
00:14:50,280 --> 00:14:53,000
Does the action fit the permissions assigned to the agent?

406
00:14:53,000 --> 00:14:54,920
Only then does the tool receive the call.

407
00:14:54,920 --> 00:14:58,000
The tool does not return a giant list of employee records.

408
00:14:58,000 --> 00:14:59,960
It returns only what the job needs.

409
00:14:59,960 --> 00:15:01,840
The employee's available leave balance,

410
00:15:01,840 --> 00:15:03,280
the policy that applies to them,

411
00:15:03,280 --> 00:15:06,000
and a confirmation that the request was created as a draft.

412
00:15:06,000 --> 00:15:07,400
That narrow response matters.

413
00:15:07,400 --> 00:15:09,040
The agent has enough information to help,

414
00:15:09,040 --> 00:15:12,720
but it does not get broad access just because it asked a question in plain English.

415
00:15:12,720 --> 00:15:16,360
Now, imagine the employee paced text from an outside document into the chat.

416
00:15:16,360 --> 00:15:19,120
The text contains instructions that try to trick the assistant

417
00:15:19,120 --> 00:15:22,160
into ignoring its rules or requesting confidential information.

418
00:15:22,160 --> 00:15:25,080
The gateway can apply safety checks before the prompt reaches the model

419
00:15:25,080 --> 00:15:27,000
or before a tool receives a request.

420
00:15:27,000 --> 00:15:29,240
If the request breaks a rule, it can stop there.

421
00:15:29,240 --> 00:15:33,080
The same kind of check can review a response before it reaches the employee.

422
00:15:33,080 --> 00:15:35,880
Not every problem comes from a bad prompt either.

423
00:15:35,880 --> 00:15:40,240
A model might produce a response that includes information it should not return.

424
00:15:40,240 --> 00:15:44,040
A shared control point gives the company a place to inspect that path and act on it.

425
00:15:44,040 --> 00:15:45,680
Meanwhile, the request creates a record.

426
00:15:45,680 --> 00:15:48,920
The company can link the token used to the HR assistant project.

427
00:15:48,920 --> 00:15:50,920
It can see which model handled the question,

428
00:15:50,920 --> 00:15:53,720
whether the tool called succeeded, whether a request was blocked,

429
00:15:53,720 --> 00:15:57,080
and how long the process took, that makes support in review much easier.

430
00:15:57,080 --> 00:16:00,720
If an HR team reports that employees cannot create leave requests,

431
00:16:00,720 --> 00:16:03,400
the operations team can look for failed tool calls.

432
00:16:03,400 --> 00:16:05,400
If a safety rule blocks valid questions,

433
00:16:05,400 --> 00:16:07,680
they can see the pattern and improve the policy.

434
00:16:07,680 --> 00:16:10,760
If one agent suddenly uses much more AI capacity than normal,

435
00:16:10,760 --> 00:16:12,240
that change becomes visible.

436
00:16:12,240 --> 00:16:16,840
And if the company updates a rule like requiring an extra check before certain leave requests,

437
00:16:16,840 --> 00:16:18,920
it can update the gateway policy once.

438
00:16:18,920 --> 00:16:22,160
It does not need to wait for separate code updates across every HR app,

439
00:16:22,160 --> 00:16:25,080
chatbot, and workflow that uses the same controlled path.

440
00:16:25,080 --> 00:16:28,840
The same request pattern still works when the company adds another model,

441
00:16:28,840 --> 00:16:31,160
a new region, more internal tools,

442
00:16:31,160 --> 00:16:34,400
or another agent that handles a related HR task.

443
00:16:34,400 --> 00:16:39,160
Starts more though, a gateway becomes useful when AI traffic starts to become shared traffic.

444
00:16:39,160 --> 00:16:40,200
Where to start?

445
00:16:40,200 --> 00:16:42,280
So, where do you actually begin?

446
00:16:42,280 --> 00:16:44,520
Start by picking a single real workload,

447
00:16:44,520 --> 00:16:47,080
maybe a support assistant that already calls a model,

448
00:16:47,080 --> 00:16:50,680
or an agent your team plans for HR, finance, or internal documents search.

449
00:16:50,680 --> 00:16:53,080
Don't try to tackle every model and every tool at once,

450
00:16:53,080 --> 00:16:54,440
and that's a fast track to chaos.

451
00:16:54,440 --> 00:16:57,400
Instead of giving provider keys directly to every application,

452
00:16:57,400 --> 00:17:01,560
put that workloads model endpoint behind Azure API management AI gateway,

453
00:17:01,560 --> 00:17:04,640
think of it like a single security checkpoint for your AI traffic.

454
00:17:04,640 --> 00:17:07,400
Then set the basics, name a clear owner for the workload,

455
00:17:07,400 --> 00:17:11,320
use enter-based access so only the right people or services can reach it.

456
00:17:11,320 --> 00:17:13,240
Set a token limit that fits the job.

457
00:17:13,240 --> 00:17:14,240
No more, no less.

458
00:17:14,240 --> 00:17:16,160
Turn on logging so you can see what's happening.

459
00:17:16,160 --> 00:17:18,720
At one content safety rule, your team can test and review.

460
00:17:18,720 --> 00:17:19,760
Start simple.

461
00:17:19,760 --> 00:17:22,160
Once the model path and access boundary are working,

462
00:17:22,160 --> 00:17:24,560
add approved MCP tools one at a time.

463
00:17:24,560 --> 00:17:28,160
Each tool should have a narrow job and a clear reason for the agent to use it.

464
00:17:28,160 --> 00:17:30,000
No unnecessary extras.

465
00:17:30,000 --> 00:17:34,480
From there, you grow from one controlled workload to shared models, tools, and agents.

466
00:17:34,480 --> 00:17:36,160
Without that initial control point,

467
00:17:36,160 --> 00:17:39,600
every new AI app creates another unmanaged path into company data

468
00:17:39,600 --> 00:17:41,360
and another line on the AI bill.

469
00:17:41,360 --> 00:17:45,360
For the next knowledge nugget, look at Azure API management simply explained.

470
00:17:45,360 --> 00:17:48,080
That's the foundation Microsoft AI gateway bill zone.