Scaling Copilot Studio in the Enterprise with Isha Kapoor [MVP]

1
00:00:00,000 --> 00:00:03,400
Welcome everybody.

2
00:00:03,400 --> 00:00:10,220
To another edition of the MC65 podcast, today we are diving deep into one of the most important

3
00:00:10,220 --> 00:00:13,200
topics in the enterprise AI right now.

4
00:00:13,200 --> 00:00:19,800
Our organization can successfully scale Microsoft co-pilot studio across complex and regular

5
00:00:19,800 --> 00:00:20,800
environments.

6
00:00:20,800 --> 00:00:27,560
Joining us today is Isha Kapoor, MDP, a co-pilot engineer who will lead solutions design, architecture,

7
00:00:27,560 --> 00:00:32,800
development and governance for co-pilot studio and other AI agents platforms within large

8
00:00:32,800 --> 00:00:38,800
organizations, including banks, governance institutions and highly regulated industries

9
00:00:38,800 --> 00:00:44,960
with more than 15 years of experience across SharePoint, Microsoft's 65 and the Power Platform

10
00:00:44,960 --> 00:00:45,960
Elk Assistant.

11
00:00:45,960 --> 00:00:51,360
Isha brings a unique perspective that combines enterprise architecture, governance, security

12
00:00:51,360 --> 00:00:53,760
and real-world AI implementation.

13
00:00:53,760 --> 00:01:03,600
In 2004, she has focused heavily on designing practical AI solutions that organize, can

14
00:01:03,600 --> 00:01:08,240
safely deploy it at scale.

15
00:01:08,240 --> 00:01:14,920
Isha is also a former Microsoft SharePoint MVP and well-known as Content Creator who actively

16
00:01:14,920 --> 00:01:20,760
contributes to remolage and best practice for the Microsoft community.

17
00:01:20,760 --> 00:01:27,320
Today we explore governance, DLP, LM and security compliance deployment strategies, lesson

18
00:01:27,320 --> 00:01:34,320
learned from enterprise projects and what it really takes to move co-pilot studio from

19
00:01:34,320 --> 00:01:37,560
pilot project into a productive environment.

20
00:01:37,560 --> 00:01:39,160
So let's jump in.

21
00:01:39,160 --> 00:01:40,160
Welcome Isha.

22
00:01:40,160 --> 00:01:42,160
Oh, thank you so much.

23
00:01:42,160 --> 00:01:45,920
Oh, that was quite an intro.

24
00:01:45,920 --> 00:01:46,920
Thank you.

25
00:01:46,920 --> 00:01:49,800
I am very happy to be here.

26
00:01:49,800 --> 00:01:57,480
And I think this is one of the most important topics because AI is exciting and everybody

27
00:01:57,480 --> 00:01:58,800
wants to jump into it.

28
00:01:58,800 --> 00:02:04,760
Everybody wants to develop agents because every company now wants to be AI enabled.

29
00:02:04,760 --> 00:02:12,160
And it is about a reputation, talk, it's about how they are designing, how they are moving

30
00:02:12,160 --> 00:02:14,080
with the times.

31
00:02:14,080 --> 00:02:23,240
However, when we look at larger organizations like Bats, financial institutes, government,

32
00:02:23,240 --> 00:02:31,240
they're the process to enable AI, build something is not as quick as what you see on the

33
00:02:31,240 --> 00:02:32,240
internet.

34
00:02:32,240 --> 00:02:38,480
We can spin up a co-pilot studio and build an agent in less than five minutes and it's

35
00:02:38,480 --> 00:02:39,480
up and running.

36
00:02:39,480 --> 00:02:46,960
However, in a company in a real world larger company, it would take you months to get

37
00:02:46,960 --> 00:02:54,120
approvals to go through the standards, code review, moving through the deployment pipeline,

38
00:02:54,120 --> 00:03:00,000
going through the actual AALM strategy, making sure that the security is always aware what

39
00:03:00,000 --> 00:03:09,440
the AI is doing and what level of access and what level of authorization your AI has.

40
00:03:09,440 --> 00:03:16,640
The agent 365 is currently in a hot topic.

41
00:03:16,640 --> 00:03:22,640
So whether it's deployed and it's always looking at what the agent is doing or if a developer

42
00:03:22,640 --> 00:03:27,600
is coming in and building its own thing and nobody knows about it.

43
00:03:27,600 --> 00:03:37,400
So in a larger organization and in a real world, it is hard to build an agent, then deploy

44
00:03:37,400 --> 00:03:44,840
it and then make sure that agent is only doing the things that it is supposed to do.

45
00:03:44,840 --> 00:03:47,360
Every agent has a task.

46
00:03:47,360 --> 00:03:50,560
So agents are based on tasks.

47
00:03:50,560 --> 00:03:55,760
Now whether it's multi-agent system, it doesn't matter, they all combine, they are there

48
00:03:55,760 --> 00:03:56,760
to perform something.

49
00:03:56,760 --> 00:04:00,320
They are knowledge-based agents or they are autonomous agents.

50
00:04:00,320 --> 00:04:09,240
However, if we have proper guardrails to monitor them, if they have an identity and if they're

51
00:04:09,240 --> 00:04:15,800
running isolated in a prod environment and develop somewhere else, there is a change request

52
00:04:15,800 --> 00:04:16,800
that goes in.

53
00:04:16,800 --> 00:04:24,240
There is operations team that always have scripts ready to operate who, you know, continuously

54
00:04:24,240 --> 00:04:29,400
they can quarantine it, they can stop it, they can delete it or they can pause the agent.

55
00:04:29,400 --> 00:04:35,840
So that's like a normal scenario for any application in a larger organization.

56
00:04:35,840 --> 00:04:43,920
We need to start thinking in those terms rather than clicking two buttons and our studio agent

57
00:04:43,920 --> 00:04:50,000
is ready and this is cool, it's sending emails to so many people now.

58
00:04:50,000 --> 00:04:56,040
Because the same developer with a wrong prompt, for example, I'll give you a real example.

59
00:04:56,040 --> 00:05:02,440
So a developer wanted to delete the test data from a SharePoint site and this SharePoint site

60
00:05:02,440 --> 00:05:08,240
was running from seven years and eight years, had a retention period of for a financial

61
00:05:08,240 --> 00:05:15,640
institution to keep the files for seven years and the developers prompt was that the agent

62
00:05:15,640 --> 00:05:21,880
woke up, you know, it will wake up at 2 a.m. and start scanning it and delete anywhere where

63
00:05:21,880 --> 00:05:23,760
it says test.

64
00:05:23,760 --> 00:05:25,600
So test something.

65
00:05:25,600 --> 00:05:32,720
So it deleted hundreds of files because there was a test mentioned every other places as

66
00:05:32,720 --> 00:05:33,720
well.

67
00:05:33,720 --> 00:05:38,520
The problem was they couldn't find out how many were deleted, nobody had a account for

68
00:05:38,520 --> 00:05:41,160
it, there was no account for it.

69
00:05:41,160 --> 00:05:48,360
So it was caused back to Microsoft back in court and it was a real world scenario, that's

70
00:05:48,360 --> 00:05:52,880
where everybody loved the lesson.

71
00:05:52,880 --> 00:05:55,440
And that's where our strategy came in.

72
00:05:55,440 --> 00:06:01,400
So when I went into this larger financial institution where I'm working right now as a

73
00:06:01,400 --> 00:06:09,920
co-pilot engineer, I'm leading the entire rollout of co-pilot studio specifically and integration

74
00:06:09,920 --> 00:06:16,240
of any existing enterprise tools like Gira, Confluence, all of them in co-pilot studio.

75
00:06:16,240 --> 00:06:22,280
So the agents can be built within the process, tightly controlled, but they can they also

76
00:06:22,280 --> 00:06:24,440
have access to enterprise data.

77
00:06:24,440 --> 00:06:25,440
No.

78
00:06:25,440 --> 00:06:32,320
So the first thing I did when I went there was to draft an ALM strategy.

79
00:06:32,320 --> 00:06:38,280
So right from the beginning it was very clear that development will only happen in a dev

80
00:06:38,280 --> 00:06:46,560
power platform environment and in addition to that they will not have access to any production

81
00:06:46,560 --> 00:06:48,480
share bonsai.

82
00:06:48,480 --> 00:06:56,480
So that pin up a dev site copy data if you want in that in the same production payment,

83
00:06:56,480 --> 00:07:04,560
but that dev site will be added as a endpoint filtering into the share point co-pilot connector.

84
00:07:04,560 --> 00:07:09,640
And that would be given and targeted to that power platform environment where the development

85
00:07:09,640 --> 00:07:17,200
will happen and makers will have to make a role in that power platform environment.

86
00:07:17,200 --> 00:07:20,120
And that isolates it.

87
00:07:20,120 --> 00:07:26,720
Then there is a whole responsible AI team that goes through use case that goes through

88
00:07:26,720 --> 00:07:27,720
the results.

89
00:07:27,720 --> 00:07:33,120
There is a red testing team that tests it in a QA power platform environment and then

90
00:07:33,120 --> 00:07:39,080
the solution moves the agent moves from dev environment to QA environment through a pipeline.

91
00:07:39,080 --> 00:07:45,800
And then QA happens and when the QA is confident, then it moves through the production environment

92
00:07:45,800 --> 00:07:51,760
through a change record and operations team basically follows the implementation, whatever

93
00:07:51,760 --> 00:07:57,720
the implementation guide says, which is clicking buttons, moving here, changing connections,

94
00:07:57,720 --> 00:08:02,280
changing environment, variable values and all of that.

95
00:08:02,280 --> 00:08:09,280
But in the broad environment, no maker, no QA, nobody has access to it.

96
00:08:09,280 --> 00:08:18,600
So only the operation teams moves the solution to the power platform environment and there

97
00:08:18,600 --> 00:08:22,000
they click on deploy, like submit for approval.

98
00:08:22,000 --> 00:08:30,680
So it goes actual M365 agents registry, agent 365 registry and it's deployed to a security

99
00:08:30,680 --> 00:08:37,520
group, whichever the audiences for that for that agent and whatever the channel, which

100
00:08:37,520 --> 00:08:41,080
was specified by the maker and maker has no access.

101
00:08:41,080 --> 00:08:44,600
So that's a broad copy sitting in broad.

102
00:08:44,600 --> 00:08:51,880
So that was our first strategy and we took a lot of time drafting that particularly we

103
00:08:51,880 --> 00:08:56,480
wanted to control DLP policies at every point.

104
00:08:56,480 --> 00:09:04,160
So a depth power platform environment will have policies like end point filtering so cannot

105
00:09:04,160 --> 00:09:10,320
add a broad site as a knowledge source and start working on it.

106
00:09:10,320 --> 00:09:16,160
Although there is no guard rail right now for power platform connectors.

107
00:09:16,160 --> 00:09:22,760
So a point connector, one drive connector, if they want to use the two like delete a site,

108
00:09:22,760 --> 00:09:28,480
delete a file, they can still add on their own a broad URL.

109
00:09:28,480 --> 00:09:34,360
So for that reasons, we also have a tenant, isolation strategy.

110
00:09:34,360 --> 00:09:41,440
That means a lower tenant is provision and makers who are new and want to just do experimentation

111
00:09:41,440 --> 00:09:47,760
for co-pilot studio agents can go to a lower tenant and do whatever they want in a co-pilot

112
00:09:47,760 --> 00:09:52,080
studio, but it's never deployed to production.

113
00:09:52,080 --> 00:09:59,400
So for production only for production tenant, they develop in a development power platform

114
00:09:59,400 --> 00:10:06,560
environment where all the de-e-e-e-es are configured which make sure that they are targeting

115
00:10:06,560 --> 00:10:13,280
dev sites, not broad sites while they are developing and also they are using power platform

116
00:10:13,280 --> 00:10:14,280
connectors.

117
00:10:14,280 --> 00:10:18,240
We are tightly controlling them through advanced policies.

118
00:10:18,240 --> 00:10:25,440
So all the sub tools of delete file are turned off in advanced connector policies for

119
00:10:25,440 --> 00:10:27,040
that environment.

120
00:10:27,040 --> 00:10:30,960
So they can create an update but they can't delete.

121
00:10:30,960 --> 00:10:37,320
So things like that and then other than that any new features come in, review features like

122
00:10:37,320 --> 00:10:42,040
anthropic models or even prompt, we disabled that.

123
00:10:42,040 --> 00:10:49,000
We carefully designed it so the makers are staying within the guidelines.

124
00:10:49,000 --> 00:10:54,240
And once the maker does that, they push it through themselves.

125
00:10:54,240 --> 00:11:00,800
They can just deploy it through a pipeline to a QA environment, power platform environment.

126
00:11:00,800 --> 00:11:07,440
And in the QA environment, we have a designated agent owner that will go in and give access

127
00:11:07,440 --> 00:11:11,680
to only the red testing team.

128
00:11:11,680 --> 00:11:17,000
So that will happen and in the dev environment, we have also turned off sharing.

129
00:11:17,000 --> 00:11:18,400
So no sharing is allowed.

130
00:11:18,400 --> 00:11:23,280
So that means a maker cannot actually develop something and share it with an entire company

131
00:11:23,280 --> 00:11:26,240
and everybody starts using it that day.

132
00:11:26,240 --> 00:11:27,640
So we have turned that off.

133
00:11:27,640 --> 00:11:34,200
So maker only has access to their own development and there is no other access at all for sharing

134
00:11:34,200 --> 00:11:36,280
in the dev environment.

135
00:11:36,280 --> 00:11:42,680
In QA, we enabled the sharing, but sharing is targeted to a red team testing team audience.

136
00:11:42,680 --> 00:11:48,160
So that's where the test and they're they remain between Dev and QA Dev and QA and once

137
00:11:48,160 --> 00:11:54,800
they are confident, they get all the approvals, then put the change request to move the solution

138
00:11:54,800 --> 00:11:56,720
to a broad environment.

139
00:11:56,720 --> 00:11:59,360
That's where it's going to sit.

140
00:11:59,360 --> 00:12:01,760
So all three environments are designed.

141
00:12:01,760 --> 00:12:08,160
There were scripts which were built how to build environment groups, then environments within

142
00:12:08,160 --> 00:12:13,520
these groups and all the policies which we preferred that it would be configured on the

143
00:12:13,520 --> 00:12:22,880
environment group level and everything from connected agents to sending sensitive data

144
00:12:22,880 --> 00:12:25,800
to Azure app insights.

145
00:12:25,800 --> 00:12:31,720
All of this was vetted and I think Microsoft is still trying to evolve that piece.

146
00:12:31,720 --> 00:12:35,720
That the sensitive data is still going to app insights.

147
00:12:35,720 --> 00:12:42,360
So we have turned off app insights for now without those reasons, but preview was configured

148
00:12:42,360 --> 00:12:50,400
in our cases that we need to make sure that every activity design time and runtime is captured

149
00:12:50,400 --> 00:12:51,880
by preview.

150
00:12:51,880 --> 00:12:57,040
So compliance is always on the top of it because compliance did not want it that somebody

151
00:12:57,040 --> 00:13:04,320
would go in and add an intranet site and this person probably has access to something on intranet

152
00:13:04,320 --> 00:13:10,120
and just add it and then delete it one day and next they leave, you know, whatever happens

153
00:13:10,120 --> 00:13:12,280
they're a contractor, they leave next day.

154
00:13:12,280 --> 00:13:20,960
So nobody knows what happened to that and sneaky Dev agent was sitting in some environment

155
00:13:20,960 --> 00:13:24,440
and that had deleted something in the intranet site.

156
00:13:24,440 --> 00:13:26,960
Nobody would even know it if that happens.

157
00:13:26,960 --> 00:13:33,080
Unless there is preview configured and tracking it that somebody has changed something on

158
00:13:33,080 --> 00:13:34,840
the design time.

159
00:13:34,840 --> 00:13:41,600
So these are real world examples how studio works.

160
00:13:41,600 --> 00:13:46,440
You need to in a larger enterprise where change requests are there where operation themes

161
00:13:46,440 --> 00:13:53,040
is there where attestation reports are generated every, you know, every few months about the health

162
00:13:53,040 --> 00:13:56,320
of the environment and how things are working.

163
00:13:56,320 --> 00:14:02,920
AI needs to be highly controlled as any application is.

164
00:14:02,920 --> 00:14:04,920
So good.

165
00:14:04,920 --> 00:14:08,120
This was awesome.

166
00:14:08,120 --> 00:14:11,280
So you make the unemployed source.

167
00:14:11,280 --> 00:14:17,120
So we have so many technical stuff inside.

168
00:14:17,120 --> 00:14:23,200
Let us start with ALM and deployment strategy.

169
00:14:23,200 --> 00:14:32,440
So you have given a wonderful overview, but how did you, what looks majored ALM strategy

170
00:14:32,440 --> 00:14:38,600
for an enterprise AI solution from your perspective?

171
00:14:38,600 --> 00:14:45,200
So I think the ALM strategy follows the same rules, which is development is a very isolated

172
00:14:45,200 --> 00:14:47,800
environment.

173
00:14:47,800 --> 00:14:56,200
And even if a maker develops something in it and they leave, they are a contractor, there

174
00:14:56,200 --> 00:15:02,880
is some owner, we call them environment owners, environment managers also sometimes.

175
00:15:02,880 --> 00:15:08,240
So these environment owners are the one who does like quarterly attestations on how many

176
00:15:08,240 --> 00:15:13,560
agents are running, how many are in draft, how many are sitting, pointing to some broad

177
00:15:13,560 --> 00:15:15,160
sites.

178
00:15:15,160 --> 00:15:18,840
So but they are all in development phases.

179
00:15:18,840 --> 00:15:25,920
So they, these phases are highly controlled on any makers who have access, they will have

180
00:15:25,920 --> 00:15:30,040
access only to the development environment.

181
00:15:30,040 --> 00:15:35,200
And how do you manage a world learning and deployment pipelines?

182
00:15:35,200 --> 00:15:36,200
Yes.

183
00:15:36,200 --> 00:15:41,240
So once they are ready, once they get the approval and we usually we go through approval of

184
00:15:41,240 --> 00:15:42,600
multiple stocks.

185
00:15:42,600 --> 00:15:47,720
So one is the code review and then what the code is doing and then the other one is the

186
00:15:47,720 --> 00:15:53,520
environment, there's an agent owner that takes responsibility of the use case.

187
00:15:53,520 --> 00:16:00,640
And then at the end is the responsible AI, responsible AI is going through what kind of

188
00:16:00,640 --> 00:16:08,440
data it's accessing, if the compliance is correctly recording all the activities.

189
00:16:08,440 --> 00:16:17,280
And once that approvals are done, it is pushed to the QA environment through the pipeline.

190
00:16:17,280 --> 00:16:25,360
So they just basically there's a new option now inside studio says start a deployment process

191
00:16:25,360 --> 00:16:30,040
which is automating sending through the pipeline to a new environment.

192
00:16:30,040 --> 00:16:34,360
And in this case, we, we provision environments in pairs.

193
00:16:34,360 --> 00:16:39,640
So dev, QA and prod all three are configured.

194
00:16:39,640 --> 00:16:44,160
The only difference is the dev has no sharing.

195
00:16:44,160 --> 00:16:49,560
So they cannot share it with anyone because if they start sharing and if the audiences let's

196
00:16:49,560 --> 00:16:55,200
say their manager and four other people, they will share it and they will start using it

197
00:16:55,200 --> 00:16:58,800
as if it is a prod and nobody's stopping them.

198
00:16:58,800 --> 00:17:05,120
They can go to m365, go by the chat and add it and start using it.

199
00:17:05,120 --> 00:17:07,640
So yes, go ahead.

200
00:17:07,640 --> 00:17:09,600
You, you, I understand right?

201
00:17:09,600 --> 00:17:13,160
You, you deploy quarterly, right?

202
00:17:13,160 --> 00:17:16,480
So quarterly, we do attestations, cotton.

203
00:17:16,480 --> 00:17:23,080
So there is scripts for attestations just to see what is deployed in the actual registry

204
00:17:23,080 --> 00:17:31,160
m365 and there is a power shell scripts that we can go through and we can see only the agents

205
00:17:31,160 --> 00:17:35,480
that are deployed company wide for each environment.

206
00:17:35,480 --> 00:17:39,520
We look at the agents from power platform inventory.

207
00:17:39,520 --> 00:17:45,440
So that's also power shut script, but it is targeting not agent 365, it's targeting power

208
00:17:45,440 --> 00:17:47,440
platform.

209
00:17:47,440 --> 00:17:54,120
So within power platform, we have power platform inventory for each environment and you can

210
00:17:54,120 --> 00:17:58,800
sort it out by the environment level and then you can see how many agents are sitting

211
00:17:58,800 --> 00:18:00,920
there.

212
00:18:00,920 --> 00:18:07,520
Some of them might be copies, some of them might be not published at all once.

213
00:18:07,520 --> 00:18:15,400
The ones that are published at least one time, they show up in m365 registry in the admin

214
00:18:15,400 --> 00:18:17,040
center.

215
00:18:17,040 --> 00:18:22,760
So, but they are, if they are published at least one time, if they're not published one

216
00:18:22,760 --> 00:18:25,640
time, they are all draft agents.

217
00:18:25,640 --> 00:18:32,640
So we, we probably see multiple, multiple hundreds of draft agents because somebody created

218
00:18:32,640 --> 00:18:36,360
something they saw an error and they, they just moved out.

219
00:18:36,360 --> 00:18:42,040
Then the next day they went in, they added a prompt, a new agent was created, but they were

220
00:18:42,040 --> 00:18:43,520
never published.

221
00:18:43,520 --> 00:18:49,000
So these are the ones which are called draft and it's environment owners, a responsibility

222
00:18:49,000 --> 00:18:55,600
to do that at a station every, you know, quarterly, we suggest quarterly and go in and see how

223
00:18:55,600 --> 00:19:01,360
many agents are actually valid use cases and the rest of them can be deleted.

224
00:19:01,360 --> 00:19:05,960
Wow, little screw.

225
00:19:05,960 --> 00:19:11,360
What role do you use, GitHub or Azure DevOps play in your projects?

226
00:19:11,360 --> 00:19:18,960
Yes, so, at this point, we are not using, but we are looking to integrate with the, with the

227
00:19:18,960 --> 00:19:23,480
GitHub to for our source control.

228
00:19:23,480 --> 00:19:28,480
For now, they are doing manually, they are uploading it to Azure DevOps, they copy of

229
00:19:28,480 --> 00:19:32,720
solution, whichever goes into prod with the versioning.

230
00:19:32,720 --> 00:19:39,080
The one of the good thing is the versions are maintained in m365 registry.

231
00:19:39,080 --> 00:19:47,120
So if he update a same agent again six months later, it will maintain, as long as the name

232
00:19:47,120 --> 00:19:53,960
is exactly the same, it will maintain the version in the registry and you can through

233
00:19:53,960 --> 00:19:59,480
power share, you can actually take out the details that, you know, this is version 0.1 or

234
00:19:59,480 --> 00:20:02,640
0.2 or 0.3.

235
00:20:02,640 --> 00:20:10,800
Of course, the agent 365 is not a solution, you know, source control, so it doesn't have

236
00:20:10,800 --> 00:20:12,160
a version control.

237
00:20:12,160 --> 00:20:19,200
So we are still looking to actually integrate with GitHub and GitHub is because we found

238
00:20:19,200 --> 00:20:24,840
out that GitHub is tightly integrated with the power at fault.

239
00:20:24,840 --> 00:20:31,400
So it automatically whenever we push a new version through the pipeline, it connects to

240
00:20:31,400 --> 00:20:37,480
GitHub, it calls GitHub API and uploads it there.

241
00:20:37,480 --> 00:20:38,720
It is auto.

242
00:20:38,720 --> 00:20:43,800
So Azure DevOps for our case right now is manual.

243
00:20:43,800 --> 00:20:51,120
So we will choose actually GitHub going forward, but at this point, we haven't configured

244
00:20:51,120 --> 00:20:53,120
it yet.

245
00:20:53,120 --> 00:20:54,120
This is awesome.

246
00:20:54,120 --> 00:21:00,880
So I understand how you handle the rollback, but how did you handle the incident management

247
00:21:00,880 --> 00:21:03,640
when I don't know, something goes wrong?

248
00:21:03,640 --> 00:21:07,440
Yeah, so that's where something very interesting I saw.

249
00:21:07,440 --> 00:21:13,720
It's called, it didn't get much of a, you know, shout out from the community.

250
00:21:13,720 --> 00:21:16,200
It's called quarantine.

251
00:21:16,200 --> 00:21:22,240
So we have a process called quarantine and this is put out from Microsoft.

252
00:21:22,240 --> 00:21:24,360
Not a lot of people have used this.

253
00:21:24,360 --> 00:21:32,360
So if there is a compliance issue with the agent and or there is like a incident or service

254
00:21:32,360 --> 00:21:40,400
ticket happens, so operations team with the urgent change request can quarantine an agent.

255
00:21:40,400 --> 00:21:45,440
So that means what happens is it just stops the agent right there.

256
00:21:45,440 --> 00:21:48,640
This enables it sort of temporarily.

257
00:21:48,640 --> 00:21:54,440
The agent is not removed from the agent 365 registry.

258
00:21:54,440 --> 00:21:56,880
It's not undeploied.

259
00:21:56,880 --> 00:21:58,800
It is as it is.

260
00:21:58,800 --> 00:22:04,520
It's just stopped and paused at that time until the investigation happens.

261
00:22:04,520 --> 00:22:11,720
Now what happens from the audience side is the audience can look at it.

262
00:22:11,720 --> 00:22:12,720
They can see it.

263
00:22:12,720 --> 00:22:16,120
It's there in their teams or Microsoft 365 chat.

264
00:22:16,120 --> 00:22:18,320
However, they cannot interact with it.

265
00:22:18,320 --> 00:22:20,200
And they try to send it a message.

266
00:22:20,200 --> 00:22:23,800
They will see it like a inner disabled or form.

267
00:22:23,800 --> 00:22:28,800
So so that's called quarantine and we have a script for that.

268
00:22:28,800 --> 00:22:31,200
So the movement something happens.

269
00:22:31,200 --> 00:22:34,240
We quarantine the agent for now.

270
00:22:34,240 --> 00:22:35,320
We have tested it.

271
00:22:35,320 --> 00:22:39,000
We didn't find a real scenario for now.

272
00:22:39,000 --> 00:22:42,640
But it is mostly because I work for a financial institution.

273
00:22:42,640 --> 00:22:46,040
So that's quite handy for us.

274
00:22:46,040 --> 00:22:49,920
The compliance raises an issue.

275
00:22:49,920 --> 00:22:53,000
I think that the first action would be to quarantine.

276
00:22:53,000 --> 00:22:58,600
If the investigation says that the agent was built not correctly, it's a wrong agent.

277
00:22:58,600 --> 00:23:02,080
It's doing some, which is not supposed to do.

278
00:23:02,080 --> 00:23:06,200
That's the biggest scare actually we have a wrong prompt.

279
00:23:06,200 --> 00:23:14,840
So a wrong prompt or a wrong interpretation by the agent by the LLM.

280
00:23:14,840 --> 00:23:21,080
So if it was supposed to delete the desktop or if it was supposed to send emails to these

281
00:23:21,080 --> 00:23:22,320
two managers.

282
00:23:22,320 --> 00:23:25,640
But if the manager is not there then go higher level up.

283
00:23:25,640 --> 00:23:31,720
And what if it just sends email to everybody until you know, until it reaches the president

284
00:23:31,720 --> 00:23:33,360
and the CEO.

285
00:23:33,360 --> 00:23:36,080
So that was the biggest thing.

286
00:23:36,080 --> 00:23:41,120
I think our team security teams had been pointing out.

287
00:23:41,120 --> 00:23:49,760
So because the predictability is low in developing an agent versus developing a power automate flow

288
00:23:49,760 --> 00:23:52,600
or an app where we are not present now.

289
00:23:52,600 --> 00:23:56,200
What it's going to do.

290
00:23:56,200 --> 00:24:03,440
predictability in LLM is, you know, we say 80% and then there is 20% it's still going to

291
00:24:03,440 --> 00:24:07,720
think from its own brain.

292
00:24:07,720 --> 00:24:12,200
So for those reasons, I think quarantine is the best.

293
00:24:12,200 --> 00:24:18,240
Everybody should have that script handy that if they find anything immediately quarantine

294
00:24:18,240 --> 00:24:21,280
the agent for now.

295
00:24:21,280 --> 00:24:23,600
And then there you can also delete.

296
00:24:23,600 --> 00:24:26,200
I know there is no UI option for that.

297
00:24:26,200 --> 00:24:35,000
There is a script option where you can delete the agent as well from agent, the agent registry.

298
00:24:35,000 --> 00:24:36,880
These are not tied to agent 365.

299
00:24:36,880 --> 00:24:43,200
These scripts are just bare minimum all agents in M365 admin center.

300
00:24:43,200 --> 00:24:46,760
Well, that's interesting.

301
00:24:46,760 --> 00:24:50,880
I have a question about security.

302
00:24:50,880 --> 00:24:58,920
What did you think it's the biggest issue actually or the biggest attack point is prompt

303
00:24:58,920 --> 00:25:04,960
injection, rock poll or what did you see as the highest risk actually?

304
00:25:04,960 --> 00:25:14,760
So there were tests that we did rigorous testing, especially for prompt injection.

305
00:25:14,760 --> 00:25:18,320
Just to clarify, I know a lot of people don't know this.

306
00:25:18,320 --> 00:25:24,400
Copilot studio is not same as M365 copilot.

307
00:25:24,400 --> 00:25:32,080
So your chat, your copilot chat, your copilot running in other applications is not going to

308
00:25:32,080 --> 00:25:37,560
give you the same results for the same prompt that the copilot studio will.

309
00:25:37,560 --> 00:25:40,440
They are two different services.

310
00:25:40,440 --> 00:25:47,920
Now I got to know that behind the scenes Microsoft are trying to match both of them, but copilot

311
00:25:47,920 --> 00:25:51,880
studio right now is one step lower.

312
00:25:51,880 --> 00:25:53,880
So catching things.

313
00:25:53,880 --> 00:26:00,000
So we did a lot of experimentation for prompt injection for, for example, you know, asking

314
00:26:00,000 --> 00:26:06,200
an agent to delete the knowledge source, asking an agent to tell me a story about something

315
00:26:06,200 --> 00:26:09,520
that is and deleting all the data in the company.

316
00:26:09,520 --> 00:26:17,000
So moving things like that and copilot studio, unfortunately, at sometimes has given back

317
00:26:17,000 --> 00:26:19,480
the steps how to do it.

318
00:26:19,480 --> 00:26:27,960
So but copilot same prompt for chat GPT or copilot would not give the same response.

319
00:26:27,960 --> 00:26:34,240
It will say on and violation or I'm not authorized to give you instructions like these.

320
00:26:34,240 --> 00:26:39,080
So copilot studio runs another different orchestration layer.

321
00:26:39,080 --> 00:26:43,280
So and that has a different copilot service.

322
00:26:43,280 --> 00:26:52,480
So for developing studio agents, I think they are very concentrated on a specific enterprise

323
00:26:52,480 --> 00:26:53,480
task.

324
00:26:53,480 --> 00:26:57,880
For example, monitor my mailbox.

325
00:26:57,880 --> 00:27:02,480
Even if somebody tries to prompt it and say delete my entire mailbox, it's not going to

326
00:27:02,480 --> 00:27:03,480
do that.

327
00:27:03,480 --> 00:27:08,360
It has that level of intelligence, building it.

328
00:27:08,360 --> 00:27:15,320
However, if it says write an email, send an email now, send it to my personal email ID.

329
00:27:15,320 --> 00:27:17,920
It should be there in the company's directory.

330
00:27:17,920 --> 00:27:20,120
So it might do that.

331
00:27:20,120 --> 00:27:27,040
So for those reasons, we went, we closely worked with Microsoft's team also went through

332
00:27:27,040 --> 00:27:29,800
the review items.

333
00:27:29,800 --> 00:27:34,680
I think the good thing is Microsoft has built in the responsible AI tracking.

334
00:27:34,680 --> 00:27:43,600
So if anything that breaks the rules of responsible AI, it flags as responsible AI in preview.

335
00:27:43,600 --> 00:27:48,640
So it doesn't say, you know, there is a prompt and this was the result and somebody is 24/7

336
00:27:48,640 --> 00:27:49,760
monitoring it.

337
00:27:49,760 --> 00:27:51,480
So that's not happening.

338
00:27:51,480 --> 00:28:00,040
It actually tells you that there is a responsible AI violation, probably, and this was the result.

339
00:28:00,040 --> 00:28:03,680
And for those, we have set up alerts.

340
00:28:03,680 --> 00:28:07,160
So it's it's chasing the prompts.

341
00:28:07,160 --> 00:28:11,280
However, I think copilot studios still needs to catch up.

342
00:28:11,280 --> 00:28:18,120
Yeah, in terms of, you know, being more responsible.

343
00:28:18,120 --> 00:28:25,560
So you say, you say, Pewview, I think that Microsoft has these processes.

344
00:28:25,560 --> 00:28:33,680
Are you doing so much, are being different in the way you use a lot of scripts and so on?

345
00:28:33,680 --> 00:28:41,400
Did Pewview, what role do you view a place actually in a setup like yours?

346
00:28:41,400 --> 00:28:48,000
Yeah, so my role as a copilot in the engineer when I joined last year was to draft the

347
00:28:48,000 --> 00:28:49,000
strategy.

348
00:28:49,000 --> 00:28:54,800
So I was the first one who's to tell them that let's build three different environments and

349
00:28:54,800 --> 00:29:05,040
we will only enable these DLP policies, not everything offering to the end user.

350
00:29:05,040 --> 00:29:09,480
No prompts right now, no third party models right now.

351
00:29:09,480 --> 00:29:17,000
Let's give them the bare minimum case where they can interact with the M365 grounded data.

352
00:29:17,000 --> 00:29:23,840
So they are confident and let's see the use cases and the use cases were very amazingly,

353
00:29:23,840 --> 00:29:24,840
you know, unique.

354
00:29:24,840 --> 00:29:32,000
Somebody wants to extract something out of PDF and then converted into a Jason format

355
00:29:32,000 --> 00:29:36,160
because now they want to send it to another application.

356
00:29:36,160 --> 00:29:43,560
So things like that and it we saw people were using it because this is over 100,000 employees

357
00:29:43,560 --> 00:29:49,520
and the curiosity for building their own agents was very, very high.

358
00:29:49,520 --> 00:29:56,760
So we so at the beginning I was more concentrated on setting up the environments.

359
00:29:56,760 --> 00:30:05,440
Then later on my focus moved more into compliance for setting up preview and app insights, but

360
00:30:05,440 --> 00:30:07,320
we haven't set up a app insights.

361
00:30:07,320 --> 00:30:08,600
We've posited it.

362
00:30:08,600 --> 00:30:11,840
The only reason was the sensitivity labels.

363
00:30:11,840 --> 00:30:19,200
So all sensitive data, which was labeled as sensitive was going outside the company's

364
00:30:19,200 --> 00:30:23,640
boundaries to Azure app insights.

365
00:30:23,640 --> 00:30:28,440
There's no like a harden, there's no guardrail for that on the environment level.

366
00:30:28,440 --> 00:30:32,400
Like we can call it off as environment level.

367
00:30:32,400 --> 00:30:37,360
So app insights is on a pause at this point.

368
00:30:37,360 --> 00:30:42,040
So we built some apps for tracking logs through data verse.

369
00:30:42,040 --> 00:30:44,960
So the conversation transcript.

370
00:30:44,960 --> 00:30:49,200
We built scripts to do everything.

371
00:30:49,200 --> 00:30:52,680
We also had a pipeline setup.

372
00:30:52,680 --> 00:31:00,960
So I did pipelines and we also had a very tightly controlled security.

373
00:31:00,960 --> 00:31:07,200
So for example, any audience or any makers who need access to build your agents.

374
00:31:07,200 --> 00:31:11,000
And that's all going to come through a security group.

375
00:31:11,000 --> 00:31:14,600
No individual access anywhere.

376
00:31:14,600 --> 00:31:21,680
And the groups are usually the owners of the groups are the managers, the environment owners

377
00:31:21,680 --> 00:31:23,880
who owns those agents.

378
00:31:23,880 --> 00:31:30,920
So there is a and then there is a solution process that how we will move through the pipeline

379
00:31:30,920 --> 00:31:36,120
who is going to press those buttons and whose name is going to show up in the registry

380
00:31:36,120 --> 00:31:38,120
that who owns it.

381
00:31:38,120 --> 00:31:42,200
There is a deployment service account that I recreated.

382
00:31:42,200 --> 00:31:47,800
That's a one who's going to own all the connections when the connections are changed.

383
00:31:47,800 --> 00:31:55,640
And the deployment pipeline, the deployment account is usually the publisher in the M365

384
00:31:55,640 --> 00:31:57,640
registry.

385
00:31:57,640 --> 00:32:05,800
So you have these these really, really strict compliance requirements.

386
00:32:05,800 --> 00:32:10,880
How can you disbalance this innovation?

387
00:32:10,880 --> 00:32:13,960
The innovation.

388
00:32:13,960 --> 00:32:22,080
So yeah, the thing is, when you have a bigger scale, like a larger enterprises, it's never

389
00:32:22,080 --> 00:32:31,560
easy to get things done unless you have a clear approvals from security, specifically.

390
00:32:31,560 --> 00:32:34,840
And everything goes through different stages.

391
00:32:34,840 --> 00:32:40,680
So everything goes to architecture first and the architecture is approved and then architecture

392
00:32:40,680 --> 00:32:45,920
goes to the implementation, POCs, testing, you know, lower tenant.

393
00:32:45,920 --> 00:32:48,280
We have a experimentation tenant.

394
00:32:48,280 --> 00:32:55,280
Yeah, so I think that's a very good idea for everybody to spin up a second tenant, not

395
00:32:55,280 --> 00:32:57,240
environment tenant.

396
00:32:57,240 --> 00:33:03,640
And buy some licenses for co-pilot studio in that set up power platform environments in

397
00:33:03,640 --> 00:33:11,200
that and play with all the new features that are coming up in that tenant.

398
00:33:11,200 --> 00:33:17,480
So your broad tenant, which actually has production data, has no impact whatsoever.

399
00:33:17,480 --> 00:33:22,920
So you never know, you know about the skills, the contractor can come in an entire team,

400
00:33:22,920 --> 00:33:27,800
can come in a third party vendor and you never know how many things they're running at

401
00:33:27,800 --> 00:33:28,640
the same time.

402
00:33:28,640 --> 00:33:33,520
So we call this tenant isolation.

403
00:33:33,520 --> 00:33:39,240
So that's our first strategy to do tenant isolation, but that's only for experimentation.

404
00:33:39,240 --> 00:33:44,320
So you don't build there and then push it to production.

405
00:33:44,320 --> 00:33:48,560
You just build there to scrap things, to test things.

406
00:33:48,560 --> 00:33:54,800
So we do give access to for experimentation there to a lot of makers, but the makers who

407
00:33:54,800 --> 00:34:00,880
come with the approved use cases, very simple ones like knowledge, pays agents or I want

408
00:34:00,880 --> 00:34:07,800
to retrieve this or I have hundreds of images in a PDF and I want to extract all those images

409
00:34:07,800 --> 00:34:11,600
and text in those images and understand that.

410
00:34:11,600 --> 00:34:17,240
So these are like simpler use cases, then we give them an environment to develop and make

411
00:34:17,240 --> 00:34:22,200
sure we make sure that they go through the checklist that these are the things that they

412
00:34:22,200 --> 00:34:26,280
have to do, they have to put their agent into a solution.

413
00:34:26,280 --> 00:34:28,520
Solution is going to be deployed.

414
00:34:28,520 --> 00:34:34,360
All the knowledge pays, connection references, everything should be added into the solution

415
00:34:34,360 --> 00:34:38,720
and then we'll go through this as a LEM strategy.

416
00:34:38,720 --> 00:34:40,120
So we are giving both.

417
00:34:40,120 --> 00:34:47,840
We're promoting innovation a lot in the company, co-pilot, co-work and agent 365.

418
00:34:47,840 --> 00:34:51,600
This was all available for us in a lower tenant.

419
00:34:51,600 --> 00:34:56,440
So somebody wants to experiment it, somebody so excited to put, you know, get their hands

420
00:34:56,440 --> 00:35:02,120
on it, they can go there and test it and play with it.

421
00:35:02,120 --> 00:35:08,520
What we keep are data, the enterprise data, which is very valuable for bigger organizations,

422
00:35:08,520 --> 00:35:11,120
we keep that very separate.

423
00:35:11,120 --> 00:35:21,240
Yeah, I like to keep that a little bit in the data traffic and I think a little bit, yeah,

424
00:35:21,240 --> 00:35:26,520
you have this, all the best award.

425
00:35:26,520 --> 00:35:34,920
You have the topics, I think, data, recent, residency and data exposure risk as topics, how

426
00:35:34,920 --> 00:35:40,880
can you handle this, especially in this environment and your environments?

427
00:35:40,880 --> 00:35:46,120
I think it's for you, it's more important than for other companies.

428
00:35:46,120 --> 00:35:48,480
Absolutely.

429
00:35:48,480 --> 00:35:57,200
So we have data-residency environments, restrictions, very hard restrictions for data

430
00:35:57,200 --> 00:35:58,200
residency.

431
00:35:58,200 --> 00:36:02,960
So it should be in Canada, so it should be within Canada.

432
00:36:02,960 --> 00:36:07,240
So we went through it all the contracts.

433
00:36:07,240 --> 00:36:10,440
I think there are legal teams who went through the contracts.

434
00:36:10,440 --> 00:36:17,440
For example, agent 365, when we were getting licenses for that, we have to weather that,

435
00:36:17,440 --> 00:36:21,000
you know, the data is always within our boundaries.

436
00:36:21,000 --> 00:36:29,520
So anything that violates that, I think we have to get approval for that exception.

437
00:36:29,520 --> 00:36:30,600
I haven't seen it.

438
00:36:30,600 --> 00:36:36,840
I used to work for a government organization here, a very, very restrictive environment because

439
00:36:36,840 --> 00:36:39,640
there were nuclear energy environment.

440
00:36:39,640 --> 00:36:47,040
So I remember going through a year of an effort to get Microsoft forms enabled in the

441
00:36:47,040 --> 00:36:48,560
company.

442
00:36:48,560 --> 00:36:54,720
And even a year after that was rejected because there was a fine print in the contract which

443
00:36:54,720 --> 00:37:03,680
says your data can bounce from different data centers in USE and Europe.

444
00:37:03,680 --> 00:37:07,240
So that it can, so that's it.

445
00:37:07,240 --> 00:37:12,280
So that was the only thing, even if it is not, it can in future.

446
00:37:12,280 --> 00:37:17,560
So for that reasons, I think the whole project was scrapped after one year and there was no

447
00:37:17,560 --> 00:37:20,440
forms available for the company.

448
00:37:20,440 --> 00:37:28,680
So in legal and government terms, and I think from past 10 years, I've only worked for government

449
00:37:28,680 --> 00:37:32,480
and legal companies, financial companies.

450
00:37:32,480 --> 00:37:39,640
And so in these terms, I think the lawyers, the fine print of contract with companies like

451
00:37:39,640 --> 00:37:47,960
Microsoft and of course, we go through the multiple meetings every time we want to get a new

452
00:37:47,960 --> 00:37:49,480
product.

453
00:37:49,480 --> 00:37:52,180
So and we have a premium support.

454
00:37:52,180 --> 00:37:56,680
So Microsoft is always there on our team's chat.

455
00:37:56,680 --> 00:37:59,200
And we schedule meetings with them.

456
00:37:59,200 --> 00:38:04,400
We go through that, how the data is going to behave, residency and we make our decisions

457
00:38:04,400 --> 00:38:06,800
based on that.

458
00:38:06,800 --> 00:38:14,720
And I think to protect our data is our highest priority.

459
00:38:14,720 --> 00:38:21,000
That's what when I go in, that's what usually the scenario is, that, you know, the first thing

460
00:38:21,000 --> 00:38:29,320
is, our data should be there, data should be recoverable if something happens and nobody

461
00:38:29,320 --> 00:38:31,920
should be tampering with whatever is existing.

462
00:38:31,920 --> 00:38:33,240
You just build on it.

463
00:38:33,240 --> 00:38:36,240
Whatever technology comes in.

464
00:38:36,240 --> 00:38:37,240
Yeah.

465
00:38:37,240 --> 00:38:49,400
So it's hard, but I think because they are financial institutes, everything is data in paper.

466
00:38:49,400 --> 00:38:51,600
Yeah.

467
00:38:51,600 --> 00:38:59,120
What I found interesting, or I think a little bit, you use co-pilot studio.

468
00:38:59,120 --> 00:39:02,280
Why not AI Foundry?

469
00:39:02,280 --> 00:39:07,600
We do have a form team, but they are not live yet.

470
00:39:07,600 --> 00:39:18,240
I think they're going through the same data residency and data production issues.

471
00:39:18,240 --> 00:39:21,120
The biggest was moving the data.

472
00:39:21,120 --> 00:39:30,200
So a maker who has access to all the prod data, like shared calendars or hundreds of sites

473
00:39:30,200 --> 00:39:32,080
over so many years.

474
00:39:32,080 --> 00:39:37,800
They're in this company from over 20 years and they have access to hundreds of sites.

475
00:39:37,800 --> 00:39:43,480
They can move data to a dummy site and then work with it.

476
00:39:43,480 --> 00:39:50,720
So that those kind of risks are always there, then deployment to the Foundry and hosting

477
00:39:50,720 --> 00:39:53,840
the agents in the Foundry.

478
00:39:53,840 --> 00:39:59,520
So that was, I think, the second challenge they are having that, you know, whenever the agents

479
00:39:59,520 --> 00:40:05,880
are deployed in the Foundry, how much compliance plays the roles with it in approving the use

480
00:40:05,880 --> 00:40:06,880
case.

481
00:40:06,880 --> 00:40:08,400
So what is happening inside it?

482
00:40:08,400 --> 00:40:12,280
So data, same thing, they're responsible AI data injections.

483
00:40:12,280 --> 00:40:18,560
I think they, at this point, they are not live yet, but they are testing all of that.

484
00:40:18,560 --> 00:40:25,840
We wanted to roll out co-pilot studio because this is more closer to power automate.

485
00:40:25,840 --> 00:40:32,080
Most of the people who asked for it are the teams which are working on power apps and building

486
00:40:32,080 --> 00:40:38,600
these tiny flows from over so many years.

487
00:40:38,600 --> 00:40:40,200
They jump into co-pilot studio.

488
00:40:40,200 --> 00:40:43,920
The first thing they do is they create an agent flow.

489
00:40:43,920 --> 00:40:44,920
I have seen them.

490
00:40:44,920 --> 00:40:49,880
Most of them who have put in incidents, like tiny incidents off, you know, this is blocked

491
00:40:49,880 --> 00:40:52,520
and that's not blocked.

492
00:40:52,520 --> 00:40:55,000
Most of them are just working in agent flows.

493
00:40:55,000 --> 00:40:58,800
As if they got a new UI for their power automate.

494
00:40:58,800 --> 00:41:06,520
So now they can use an LLM plus replicate their own workflows.

495
00:41:06,520 --> 00:41:15,520
And I think we did a publisher process in the company to have power automate move to agent

496
00:41:15,520 --> 00:41:17,400
flows.

497
00:41:17,400 --> 00:41:23,200
And honestly, I think it's my personal, but I think going forward, everything will be

498
00:41:23,200 --> 00:41:24,200
workflows.

499
00:41:24,200 --> 00:41:28,080
That's what Microsoft is going towards, the new workflow canvas.

500
00:41:28,080 --> 00:41:33,360
So everything probably is going to be that same thing, workflows, even if it's power automate

501
00:41:33,360 --> 00:41:38,040
or coming from the studio side.

502
00:41:38,040 --> 00:41:42,360
And when you use studio, you can choose different elements.

503
00:41:42,360 --> 00:41:49,200
How can you really, and there I have their, I don't know, our own services and own data

504
00:41:49,200 --> 00:41:55,160
center. And so how, how can you really see that most sensitive data get out?

505
00:41:55,160 --> 00:42:00,520
I don't know, like a, I ban or, both day, or something.

506
00:42:00,520 --> 00:42:02,040
How can you handle this?

507
00:42:02,040 --> 00:42:03,040
Oh, yes.

508
00:42:03,040 --> 00:42:07,120
I mean, being in a financial institute, we have a lot of sensitive data.

509
00:42:07,120 --> 00:42:11,480
I mean credit card information, so mortgage files, right?

510
00:42:11,480 --> 00:42:18,600
So there's never a chance that we would let even that be closer to anywhere outside

511
00:42:18,600 --> 00:42:21,480
our enterprise boundaries, right?

512
00:42:21,480 --> 00:42:31,160
So, so for those reasons, initially for security reasons, we turned off entropic models for

513
00:42:31,160 --> 00:42:32,480
studio.

514
00:42:32,480 --> 00:42:40,720
It is going to be the ones which are deployed through open AI in our asher instance.

515
00:42:40,720 --> 00:42:48,480
Although digging into entropic models, we figured out that entropic now has, like, a

516
00:42:48,480 --> 00:42:55,320
contract with Microsoft and the data is processed still on their side, but it is, we don't have

517
00:42:55,320 --> 00:42:56,320
to worry about it.

518
00:42:56,320 --> 00:42:58,720
It's, it's in a secure bucket.

519
00:42:58,720 --> 00:43:05,440
Now, which is the bucket is owned by Microsoft as a subscription, same as open AI was doing,

520
00:43:05,440 --> 00:43:11,600
but to be on the safer side for now, we have turned it off.

521
00:43:11,600 --> 00:43:20,360
It is available through Foundry, all the different LLMs are available through Foundry, and that's

522
00:43:20,360 --> 00:43:27,640
why I think there is a more challenge to get Foundry live because there has to be restrictions

523
00:43:27,640 --> 00:43:32,840
on that too, and they are putting restrictions there as well.

524
00:43:32,840 --> 00:43:42,240
So, I always say start with the bare minimum 4.1 model from open AI.

525
00:43:42,240 --> 00:43:48,480
There is no need for higher models unless you actually have a need for running something

526
00:43:48,480 --> 00:43:51,680
like you know, a reasoning piece.

527
00:43:51,680 --> 00:43:57,440
You're drafting a paper that needs to go to hundreds of websites and publish a paper like

528
00:43:57,440 --> 00:43:58,440
that.

529
00:43:58,440 --> 00:44:04,600
So, if the model is not required for that high, just use the bare minimum model.

530
00:44:04,600 --> 00:44:06,120
It's still an LLM.

531
00:44:06,120 --> 00:44:09,680
You need to add your workflow on it.

532
00:44:09,680 --> 00:44:17,760
So it's processing, but the LLM is strong enough, the 4.1, the 5.1's for chat GBT.

533
00:44:17,760 --> 00:44:20,720
They are strong enough for the base case.

534
00:44:20,720 --> 00:44:25,080
And if you really want to do a deep research, then use the analyst.

535
00:44:25,080 --> 00:44:33,000
Use the researcher, use the analyst, create a declarative agent for that using that, and

536
00:44:33,000 --> 00:44:39,160
then later on you can modify it in co-balance to be as well.

537
00:44:39,160 --> 00:44:44,040
And I think now within your workflow, you can call declarative agents, you can call

538
00:44:44,040 --> 00:44:50,920
researcher, you can call everything, so that is a piece that you can outsource for the reasoning.

539
00:44:50,920 --> 00:44:58,120
But in any case, I mean, going through hundreds of models and trying which one suits you is

540
00:44:58,120 --> 00:45:00,080
very specific in use cases.

541
00:45:00,080 --> 00:45:05,200
For example, you only want to work with images because your company only works with videos

542
00:45:05,200 --> 00:45:07,720
and images, then it accepts.

543
00:45:07,720 --> 00:45:15,880
Then you can wet one of the best ones from OpenAI or Google or Gemini, and then you can

544
00:45:15,880 --> 00:45:22,960
figure out which one is suitable, but for business cases, for enterprise data, for workflows.

545
00:45:22,960 --> 00:45:30,440
I think a chat GPT, I always say start with 4.1, but you can go with 5.1 as a production

546
00:45:30,440 --> 00:45:31,440
one.

547
00:45:31,440 --> 00:45:36,080
That's, I think, that's so amazing.

548
00:45:36,080 --> 00:45:44,920
And Microsoft has these, I think, the fastest feature really, really cycles ever, how

549
00:45:44,920 --> 00:45:51,240
do you, how do you not become overwhelmed with all these updates?

550
00:45:51,240 --> 00:45:54,680
Oh, everybody's overwhelmed, the trustee.

551
00:45:54,680 --> 00:45:59,840
Yeah, it is, it is very, very fast-paced.

552
00:45:59,840 --> 00:46:04,160
I think AI is taking over all the technologies that we saw before.

553
00:46:04,160 --> 00:46:11,520
We used to talk about the same thing about Office 365, remember, Microsoft 365, new features

554
00:46:11,520 --> 00:46:14,280
coming in and showing up.

555
00:46:14,280 --> 00:46:18,400
And now it's like new features are being discussed, they are released next day, they are in

556
00:46:18,400 --> 00:46:21,880
preview third day, and they're turned on by default.

557
00:46:21,880 --> 00:46:26,280
And I'll give you the biggest example, MCP servers.

558
00:46:26,280 --> 00:46:36,160
So, they were 9 MCP servers in the Microsoft 365 registry, agents registry, they were 9,

559
00:46:36,160 --> 00:46:42,360
and all of a sudden next day they were 14, and they were all turned on by default, not

560
00:46:42,360 --> 00:46:43,360
off.

561
00:46:43,360 --> 00:46:50,800
So, that is that, because three steps back, because our users started adding it, our makers

562
00:46:50,800 --> 00:46:56,280
were adding it, and somebody put a ticket, incident, and that's how we got to know.

563
00:46:56,280 --> 00:47:01,200
And we said, "No, no, no, no, no, no, no, no, MCP right now, we're still betting the, how

564
00:47:01,200 --> 00:47:03,800
it interacts behind the seeds."

565
00:47:03,800 --> 00:47:08,760
So, but they were using that another, the ones that we didn't block, because they were

566
00:47:08,760 --> 00:47:12,600
turned on by default, because they just got at it.

567
00:47:12,600 --> 00:47:21,480
So, I think everybody is over when, but this is the age, I think we need to learn how to

568
00:47:21,480 --> 00:47:24,440
adapt with it.

569
00:47:24,440 --> 00:47:31,840
If we want to properly manage this as an application in the company, so it actually applies

570
00:47:31,840 --> 00:47:32,840
it.

571
00:47:32,840 --> 00:47:39,720
So, AI is exciting, AI writes the emails, but AI in business case, AI solves the problems.

572
00:47:39,720 --> 00:47:47,400
AI, I think is very good for converting your existing power automates, which does things

573
00:47:47,400 --> 00:47:54,800
manually, like hundreds of lines of code to get a JSON output or HTML out of it, or lines

574
00:47:54,800 --> 00:47:55,800
like this.

575
00:47:55,800 --> 00:47:59,600
AI and LLM can do that work better.

576
00:47:59,600 --> 00:48:04,320
So, I think that's, it is very useful in those scenarios.

577
00:48:04,320 --> 00:48:10,440
However, anything that new comes up needs to be vetted completely before it is turned

578
00:48:10,440 --> 00:48:17,040
on and enabled, and this has been a challenge, you know, turn on something and next day somebody

579
00:48:17,040 --> 00:48:23,720
is using it and then you are chasing it because you know you got to know a month later, maybe

580
00:48:23,720 --> 00:48:25,400
through an incident.

581
00:48:25,400 --> 00:48:28,240
It's, it's real world, it's just happening.

582
00:48:28,240 --> 00:48:29,840
I, not quite sure, neither.

583
00:48:29,840 --> 00:48:31,440
We are chasing things.

584
00:48:31,440 --> 00:48:32,440
Yeah.

585
00:48:32,440 --> 00:48:35,360
Yeah, I actually have a funny experiment.

586
00:48:35,360 --> 00:48:38,520
I will write next day's about it.

587
00:48:38,520 --> 00:48:41,520
I have built, I call it a Shopify.

588
00:48:41,520 --> 00:48:43,000
I have built them up.

589
00:48:43,000 --> 00:48:45,480
Try make all errors you can make.

590
00:48:45,480 --> 00:48:48,480
MCP, all, all you can do wrong.

591
00:48:48,480 --> 00:48:54,120
And I try to build the most hallucinating copilot ever built.

592
00:48:54,120 --> 00:49:01,280
But yeah, is there, but is there an interface?

593
00:49:01,280 --> 00:49:05,760
A eye trend where you paying attention right now?

594
00:49:05,760 --> 00:49:16,040
Yeah, I think a copi, a co-work plugins is some, we have a lot of demand for that.

595
00:49:16,040 --> 00:49:24,800
A, to enable co-work and B, is to add co-work plugins for enterprise data.

596
00:49:24,800 --> 00:49:30,240
So because people came with the business use case, they love studio for now.

597
00:49:30,240 --> 00:49:34,200
And so, we roll out and there is a lot of demand.

598
00:49:34,200 --> 00:49:35,920
But these are all business cases.

599
00:49:35,920 --> 00:49:41,320
Now, they want to look at their own multi-step workflows.

600
00:49:41,320 --> 00:49:48,040
Productivity, for example, go to a planner, make a plan, then move the task to this bucket and

601
00:49:48,040 --> 00:49:54,720
then go to an email, draft, you know, update email for the, for the entire team that these

602
00:49:54,720 --> 00:50:00,200
are going to be at tasks in the planner, update the task, notification.

603
00:50:00,200 --> 00:50:05,880
Things like that and they have been asking to develop Gira and Confluence because we are

604
00:50:05,880 --> 00:50:10,080
trying to integrate those in studio now completely.

605
00:50:10,080 --> 00:50:17,680
So, so they want to also update and go through the dashboards in, in Gira and see what the

606
00:50:17,680 --> 00:50:23,080
buckets are there or in Confluence, see how many tickets are assigned to them.

607
00:50:23,080 --> 00:50:26,960
So these are, these were the demands.

608
00:50:26,960 --> 00:50:30,760
And I think it's very good that everybody is so excited.

609
00:50:30,760 --> 00:50:35,240
They read a lot of articles online and announcements.

610
00:50:35,240 --> 00:50:40,680
But as I said in a larger enterprise, everything moves very slow.

611
00:50:40,680 --> 00:50:48,280
Unless we convince security team that co-work is something that we need and why it's more beneficial

612
00:50:48,280 --> 00:50:54,120
for us and what purpose it's solving and on top of everything that we can track everything

613
00:50:54,120 --> 00:50:56,040
that co-work does.

614
00:50:56,040 --> 00:50:58,320
So we have for user.

615
00:50:58,320 --> 00:51:04,360
So we need to convince that and actually prove that so they can see it.

616
00:51:04,360 --> 00:51:09,480
I'm sure the security guys are in a lot in demand, especially AI security field.

617
00:51:09,480 --> 00:51:13,240
I think that's a very demanding thing.

618
00:51:13,240 --> 00:51:19,320
If you are in security, you should be moving into AI today.

619
00:51:19,320 --> 00:51:28,560
So, what do you think if Microsoft rename it not stands co-pilot in the next 12 or two years?

620
00:51:28,560 --> 00:51:30,800
Oh, co-pilot.

621
00:51:30,800 --> 00:51:32,800
Co-co-art studio.

622
00:51:32,800 --> 00:51:34,040
Yeah, I don't know.

623
00:51:34,040 --> 00:51:35,360
Microsoft, I don't know.

624
00:51:35,360 --> 00:51:37,600
They rename actually everything every month.

625
00:51:37,600 --> 00:51:38,600
I feel it.

626
00:51:38,600 --> 00:51:39,600
Yeah, yeah.

627
00:51:39,600 --> 00:51:41,600
But what do you think?

628
00:51:41,600 --> 00:51:46,880
I think the power automate and agent flows are going to be one.

629
00:51:46,880 --> 00:51:51,840
They are going to be called workflow and everything will be workflow.

630
00:51:51,840 --> 00:51:58,520
There's because right now you can convert power automates to agent flows and then now the

631
00:51:58,520 --> 00:52:04,000
new term everywhere in the every Microsoft learn article I'm seeing is workflow.

632
00:52:04,000 --> 00:52:12,160
Actually, so I think that will combine power apps will still remain as it is because they

633
00:52:12,160 --> 00:52:18,480
are widely used in an enterprise every enterprise have tons of power apps.

634
00:52:18,480 --> 00:52:24,880
However, I think studio is going to be more integrated into power apps.

635
00:52:24,880 --> 00:52:31,240
So, the business applications are running and then there is a studio agent that is running

636
00:52:31,240 --> 00:52:33,240
in a multi agent form.

637
00:52:33,240 --> 00:52:38,440
So a multi agent architecture is there, which is interacting with let's say two other agents

638
00:52:38,440 --> 00:52:44,760
and that is going to be called and center response from this business applications.

639
00:52:44,760 --> 00:52:50,440
So, anytime a business application somebody working on a power app and you know adding

640
00:52:50,440 --> 00:52:56,920
a new item in there, they can chat with the co-pilot studio agent right away.

641
00:52:56,920 --> 00:53:03,600
So I think that is the direction because what I have seen very latest I have seen how to

642
00:53:03,600 --> 00:53:11,840
call a power app from an agent from a agent, a co-pilot studio agent.

643
00:53:11,840 --> 00:53:13,760
So, I was able to call a power app.

644
00:53:13,760 --> 00:53:20,600
I was able to query power app in natural language, you know, asking how many items were added

645
00:53:20,600 --> 00:53:25,280
today for this category in my studio agent.

646
00:53:25,280 --> 00:53:33,320
Now vice versa, the other way all the existing power apps will be empowered by co-pilot.

647
00:53:33,320 --> 00:53:39,160
They are right now you can enable co-pilot in them but I am talking about co-pilot studio

648
00:53:39,160 --> 00:53:44,160
because co-pilot is general, it is general, it is like a chat GPT, it is just answering

649
00:53:44,160 --> 00:53:50,520
things but co-pilot studio is answering based on a task.

650
00:53:50,520 --> 00:54:00,400
So if an enterprise power app application is running to give you the inventory for your

651
00:54:00,400 --> 00:54:07,400
number of data centers the company has or you know so at the same time if somebody comes

652
00:54:07,400 --> 00:54:13,520
in they can chat with the co-pilot studio agent that agent is pulling more information from

653
00:54:13,520 --> 00:54:22,960
other sources and say how many tickets were added for this particular item or how many

654
00:54:22,960 --> 00:54:27,840
tickets are opened, how many service requests went for this item.

655
00:54:27,840 --> 00:54:33,240
So co-pilot studio agent will go and go and connect with those agents like Confluence

656
00:54:33,240 --> 00:54:41,200
or Gira pulling those data and send a response to this business application.

657
00:54:41,200 --> 00:54:42,200
Awesome.

658
00:54:42,200 --> 00:54:48,520
What advice will you give someone who will start with co-pilot studio today?

659
00:54:48,520 --> 00:54:52,680
Yeah I think this studio is not at all intimidating.

660
00:54:52,680 --> 00:54:57,260
I know a lot of people think that you already do power automate.

661
00:54:57,260 --> 00:55:03,140
It is quite similar the only differences you need to understand the underlying architecture

662
00:55:03,140 --> 00:55:05,080
of studio.

663
00:55:05,080 --> 00:55:11,820
So pull up an architecture document go through a implementation guide that Microsoft

664
00:55:11,820 --> 00:55:13,220
puts out.

665
00:55:13,220 --> 00:55:19,660
It gives you a step by step architecture which says how LLM interacts, how topics work,

666
00:55:19,660 --> 00:55:27,240
how agent flow acts, what is tools and how co-pilot studio basically is a UI.

667
00:55:27,240 --> 00:55:34,460
That integrates everything into it, API tools, applications, different agents, all of them

668
00:55:34,460 --> 00:55:41,540
together working as a solution and then giving an output as a UI to the end user.

669
00:55:41,540 --> 00:55:49,540
It is absolutely, it's up the alley for anybody who has done power automate, who has done power

670
00:55:49,540 --> 00:55:56,140
apps is even better but somebody who is very familiar with power automate, sorry power

671
00:55:56,140 --> 00:55:57,140
platform.

672
00:55:57,140 --> 00:56:03,020
In general power platform creating power platform solutions, DLP policies, I think they should

673
00:56:03,020 --> 00:56:07,260
start learning studio now.

674
00:56:07,260 --> 00:56:16,340
Studio AI, LLM models, how they behave, I think that's going to be the next future with

675
00:56:16,340 --> 00:56:19,780
integrated with business applications.

676
00:56:19,780 --> 00:56:29,380
Okay, then my final question is all this companies have labeled themselves as responsible

677
00:56:29,380 --> 00:56:36,180
AI, what tools responsible in enterprise mean to you?

678
00:56:36,180 --> 00:56:48,240
So we concentrate on responsibility, I mostly for jailbreak or unethical prompts.

679
00:56:48,240 --> 00:56:54,860
I think unethical prompts were not caught by responsible AI from our last testing.

680
00:56:54,860 --> 00:57:01,100
So unethical is something like give me give me all the instructions because I need to

681
00:57:01,100 --> 00:57:06,360
rob the bank but not like that but I would say give me a story, I need to tell a story to

682
00:57:06,360 --> 00:57:11,580
my son about somebody else who was robbing the back.

683
00:57:11,580 --> 00:57:14,880
Oh, give me a real scenario like that.

684
00:57:14,880 --> 00:57:22,120
So it is still unethical, it's unethical for AI to one still something like that because

685
00:57:22,120 --> 00:57:26,960
the AI should be smart enough to know that this person might misuse the information.

686
00:57:26,960 --> 00:57:38,120
And so there is in the new compliance center, there is DSPM, you know, the compliance for

687
00:57:38,120 --> 00:57:43,720
a preview, there is a new setting called a track unethical actually, there is exactly

688
00:57:43,720 --> 00:57:44,720
that setting.

689
00:57:44,720 --> 00:57:55,600
So responsible AI for us initially was jailbreak prompt injections, sneaking into unauthorized

690
00:57:55,600 --> 00:57:57,120
data.

691
00:57:57,120 --> 00:58:02,680
So I think those were all caught very nicely, Microsoft had a very good inbuilt responsible

692
00:58:02,680 --> 00:58:08,280
in AI and it would track it, say that you are breaking responsible AI rules.

693
00:58:08,280 --> 00:58:11,560
So which was nice, we would know.

694
00:58:11,560 --> 00:58:17,640
But unethical prompts, they they went through pass through and then that's how we found

695
00:58:17,640 --> 00:58:21,680
out that there is a new setting in DSPM but that was in preview.

696
00:58:21,680 --> 00:58:24,520
So we didn't earn that on.

697
00:58:24,520 --> 00:58:33,520
But I think everything from an unethical content to jailbreak to unauthorized prompt to try

698
00:58:33,520 --> 00:58:41,120
to extract sensitivity, sensitive data, all of this comes under a responsible AI.

699
00:58:41,120 --> 00:58:47,440
We actually have team responsible AI team that goes through the use case first, what the

700
00:58:47,440 --> 00:58:53,240
maker has developed and the maker is intending to put this into production and is it even

701
00:58:53,240 --> 00:58:54,240
ethical?

702
00:58:54,240 --> 00:58:59,360
What is I mean, is the use case even working and adhering to our policies?

703
00:58:59,360 --> 00:59:01,120
So think like that.

704
00:59:01,120 --> 00:59:02,120
Wow.

705
00:59:02,120 --> 00:59:03,120
Yeah, then.

706
00:59:03,120 --> 00:59:05,920
Yeah, that was an amazing conversation this issue.

707
00:59:05,920 --> 00:59:10,000
Gapua about scaling co-pilot students and the enterprise.

708
00:59:10,000 --> 00:59:15,800
We covered this many topics in this in a real world scenario.

709
00:59:15,800 --> 00:59:20,240
We talked about governance security and compliance and architecture.

710
00:59:20,240 --> 00:59:24,840
So I will say thank you again, Isha for joining me today.

711
00:59:24,840 --> 00:59:27,720
This was a really, really great session.

712
00:59:27,720 --> 00:59:35,640
So the last words from you to the audience and my last thing is you find all the links

713
00:59:35,640 --> 00:59:38,320
and contact data in the show lots, too, Isha.

714
00:59:38,320 --> 00:59:41,400
So yeah, awesome.

715
00:59:41,400 --> 00:59:42,400
Thank you so much.

716
00:59:42,400 --> 00:59:46,240
Thank you everyone for joining and please continue to learn co-pilot studio.

717
00:59:46,240 --> 00:59:48,240
I think it's very cool product.

718
00:59:48,240 --> 00:59:49,240
Okay.

719
00:59:49,240 --> 00:59:50,240
Yes.

720
00:59:50,240 --> 00:59:51,240
That's all I want you to do.

721
00:59:51,240 --> 00:59:52,240
Bye.

722
00:59:52,240 --> 00:59:53,240
Have a nice day.

723
00:59:53,240 --> 00:59:54,240
You too.

724
00:59:54,240 --> 00:59:55,240
Take care.