Fabric Rewrote Data Engineering

1
00:00:00,000 --> 00:00:02,040
Most organizations believe Fabric Plus Copilot

2
00:00:02,040 --> 00:00:03,760
made data engineering easier.

3
00:00:03,760 --> 00:00:04,560
They are wrong.

4
00:00:04,560 --> 00:00:07,200
Fabric removed the ceremony, not the responsibility.

5
00:00:07,200 --> 00:00:09,720
And Copilot removed the typing, not the consequences.

6
00:00:09,720 --> 00:00:13,160
So yes, pipelines run faster, notebooks feel friendlier,

7
00:00:13,160 --> 00:00:16,240
SQL appears out of thin air, and Power BI lights up

8
00:00:16,240 --> 00:00:18,200
before anyone has written a real contract.

9
00:00:18,200 --> 00:00:19,480
But the thing nobody's mentioning is,

10
00:00:19,480 --> 00:00:21,040
what speed does to ambiguity?

11
00:00:21,040 --> 00:00:21,960
It chips it.

12
00:00:21,960 --> 00:00:23,280
And it chips it at machine speed.

13
00:00:23,280 --> 00:00:25,520
Here's the comfortable belief I keep hearing in incident

14
00:00:25,520 --> 00:00:28,520
reviews and why are we over budget meetings?

15
00:00:28,520 --> 00:00:30,120
We consolidated the stack.

16
00:00:30,120 --> 00:00:31,480
We finally have one platform.

17
00:00:31,480 --> 00:00:32,720
The teams are moving faster.

18
00:00:32,720 --> 00:00:34,080
This should be simpler.

19
00:00:34,080 --> 00:00:35,720
That's the marketing narrative too.

20
00:00:35,720 --> 00:00:40,000
One Lake, one experience, one workspace, one integrated story.

21
00:00:40,000 --> 00:00:42,000
And if you stop the analysis at the UI layer,

22
00:00:42,000 --> 00:00:43,280
that narrative seems true.

23
00:00:43,280 --> 00:00:45,640
You can create a workspace, land files into one Lake,

24
00:00:45,640 --> 00:00:47,760
click a button, generate a semantic model,

25
00:00:47,760 --> 00:00:49,160
and render a report in minutes.

26
00:00:49,160 --> 00:00:50,000
That demo is real.

27
00:00:50,000 --> 00:00:51,880
The problem is what that demo trained people

28
00:00:51,880 --> 00:00:53,880
to believe that the platform removed the need

29
00:00:53,880 --> 00:00:55,160
for deliberate architecture.

30
00:00:55,160 --> 00:00:56,040
It did not.

31
00:00:56,040 --> 00:00:58,040
In architectural terms, Fabric is not a tool.

32
00:00:58,040 --> 00:01:00,480
It's platform physics, a single SAS control plane

33
00:01:00,480 --> 00:01:02,720
that lets every persona touch the same assets

34
00:01:02,720 --> 00:01:06,040
from multiple angles, engineering, warehousing,

35
00:01:06,040 --> 00:01:10,760
BI, real-time AI, inside one capacity envelope.

36
00:01:10,760 --> 00:01:12,040
That's not easier.

37
00:01:12,040 --> 00:01:14,800
That's less friction between intent and impact.

38
00:01:14,800 --> 00:01:17,960
And when your intent is unclear, that reduction in friction

39
00:01:17,960 --> 00:01:19,240
doesn't produce simplicity.

40
00:01:19,240 --> 00:01:20,880
It produces architectural erosion.

41
00:01:20,880 --> 00:01:22,960
You've seen this pattern before just not this fast.

42
00:01:22,960 --> 00:01:25,360
In older stacks, ambiguity had attacks.

43
00:01:25,360 --> 00:01:28,960
It had to cross boundaries, ADF to Synapse, DBT to warehouse,

44
00:01:28,960 --> 00:01:31,600
warehouse to Power BI, Power BI to the app.

45
00:01:31,600 --> 00:01:33,840
Each handle forced someone to make a decision,

46
00:01:33,840 --> 00:01:35,120
or at least document one.

47
00:01:35,120 --> 00:01:37,960
It was annoying, slow, and expensive,

48
00:01:37,960 --> 00:01:39,480
but it did something useful.

49
00:01:39,480 --> 00:01:41,320
It slowed bad decisions down.

50
00:01:41,320 --> 00:01:43,280
It also gave you time to notice ownership gaps

51
00:01:43,280 --> 00:01:45,080
because chain trade was limited by friction.

52
00:01:45,080 --> 00:01:46,680
Fabric removes that padding.

53
00:01:46,680 --> 00:01:50,000
So the failure mode surfaced earlier, and they surface louder.

54
00:01:50,000 --> 00:01:52,560
That's why I'm positioning this episode a very specific way.

55
00:01:52,560 --> 00:01:54,080
These failures happened in Fabric,

56
00:01:54,080 --> 00:01:55,560
but they're not Fabric-specific.

57
00:01:55,560 --> 00:01:58,000
Fabric just removed the padding that used to hide them.

58
00:01:58,000 --> 00:02:00,480
If you've operated snowflake, Databricks, Synapse,

59
00:02:00,480 --> 00:02:03,320
BigQuery, Pick your religion, you've seen the same entropy.

60
00:02:03,320 --> 00:02:05,160
Fabric just makes the drift visible sooner

61
00:02:05,160 --> 00:02:08,400
because the boundaries blur by default, not by exception.

62
00:02:08,400 --> 00:02:10,280
And that distinction matters, especially if you're one

63
00:02:10,280 --> 00:02:13,120
of the people inheriting a Fabric estate right now.

64
00:02:13,120 --> 00:02:17,280
This episode is for senior data engineers, analytics engineers,

65
00:02:17,280 --> 00:02:19,920
data platform owners, architects who just got handed

66
00:02:19,920 --> 00:02:22,520
a bunch of workspaces, and leaders asking the only

67
00:02:22,520 --> 00:02:25,400
honest question in the room, why is this still hard?

68
00:02:25,400 --> 00:02:27,000
Because the sales pitch said it wouldn't be,

69
00:02:27,000 --> 00:02:29,360
it's still hard because data engineering was never hard

70
00:02:29,360 --> 00:02:30,720
due to typing speed.

71
00:02:30,720 --> 00:02:32,960
It was hard because of responsibility, contracts, cost,

72
00:02:32,960 --> 00:02:34,080
and control.

73
00:02:34,080 --> 00:02:37,080
Fabric accelerates delivery, therefore it accelerates drift,

74
00:02:37,080 --> 00:02:39,280
copilot accelerates authoring, therefore it accelerates

75
00:02:39,280 --> 00:02:40,080
wrongness.

76
00:02:40,080 --> 00:02:42,320
And the first signals you'll see are not always failures

77
00:02:42,320 --> 00:02:43,240
in the classic sense.

78
00:02:43,240 --> 00:02:44,600
You won't get a red pipeline.

79
00:02:44,600 --> 00:02:46,120
You'll get a capacity spike.

80
00:02:46,120 --> 00:02:48,000
You'll get two dashboards that disagree.

81
00:02:48,000 --> 00:02:50,320
You'll get an ordered question that makes everyone stare

82
00:02:50,320 --> 00:02:50,960
at the floor.

83
00:02:50,960 --> 00:02:52,040
The platform works.

84
00:02:52,040 --> 00:02:53,720
The system doesn't.

85
00:02:53,720 --> 00:02:55,240
Now here's the part that kills me.

86
00:02:55,240 --> 00:02:57,800
Teams interpret that as Fabric is immature or copilot

87
00:02:57,800 --> 00:03:00,320
is unreliable or we need better training.

88
00:03:00,320 --> 00:03:03,080
And sure, training helps people click the right buttons.

89
00:03:03,080 --> 00:03:04,720
But training doesn't enforce intent.

90
00:03:04,720 --> 00:03:06,040
Labels don't enforce intent.

91
00:03:06,040 --> 00:03:07,600
Documentation doesn't enforce intent.

92
00:03:07,600 --> 00:03:09,520
Workspace RBAC doesn't enforce intent.

93
00:03:09,520 --> 00:03:10,400
Only design does.

94
00:03:10,400 --> 00:03:11,200
Only boundaries do.

95
00:03:11,200 --> 00:03:12,480
Only gates do.

96
00:03:12,480 --> 00:03:14,120
So the promise for this episode is simple.

97
00:03:14,120 --> 00:03:15,040
It's not a tutorial.

98
00:03:15,040 --> 00:03:16,160
It's not a feature tour.

99
00:03:16,160 --> 00:03:18,360
It's an explanation of why teams feel faster and more

100
00:03:18,360 --> 00:03:20,280
out of control at the exact same time.

101
00:03:20,280 --> 00:03:23,320
And what operating rules actually survive this platform,

102
00:03:23,320 --> 00:03:25,200
we're going to walk through the old mental model

103
00:03:25,200 --> 00:03:26,160
that Fabric replaced.

104
00:03:26,160 --> 00:03:29,880
What Fabric actually changed under the hood and why one platform

105
00:03:29,880 --> 00:03:31,600
quietly converts your governance model

106
00:03:31,600 --> 00:03:35,320
from deterministic to probabilistic unless you fight back.

107
00:03:35,320 --> 00:03:37,320
Then we'll talk about copilot, not as a novelty,

108
00:03:37,320 --> 00:03:39,520
but as an acceleration layer that optimizes

109
00:03:39,520 --> 00:03:41,080
for completion over consequence.

110
00:03:41,080 --> 00:03:43,280
And I'm not going to ask you to trust vibes.

111
00:03:43,280 --> 00:03:45,240
The only artifacts that matter in this world

112
00:03:45,240 --> 00:03:48,840
are execution plans, capacity metrics, and violation

113
00:03:48,840 --> 00:03:49,440
counts.

114
00:03:49,440 --> 00:03:51,800
If it doesn't show up in a plan, a cost report,

115
00:03:51,800 --> 00:03:54,560
or a violation count, it's not governance, it's hope.

116
00:03:54,560 --> 00:03:56,520
By the end, you'll have a mental model for Fabric

117
00:03:56,520 --> 00:03:59,360
that's honest and an operating model that assumes decay

118
00:03:59,360 --> 00:04:00,520
unless enforced.

119
00:04:00,520 --> 00:04:02,280
Because that's the only model that survives

120
00:04:02,280 --> 00:04:04,360
a platform designed to make everything easy,

121
00:04:04,360 --> 00:04:05,920
right up until it's expensive.

122
00:04:05,920 --> 00:04:08,080
Now let's rewind to the old world for a second

123
00:04:08,080 --> 00:04:10,240
because you need to remember what friction used to do

124
00:04:10,240 --> 00:04:11,200
for you.

125
00:04:11,200 --> 00:04:13,640
The old mental model pipelines as the product.

126
00:04:13,640 --> 00:04:16,120
Before Fabric, most data teams treated the pipeline

127
00:04:16,120 --> 00:04:18,880
as the product, not the dataset, not the semantic model,

128
00:04:18,880 --> 00:04:21,120
not the KPI definition, the pipeline.

129
00:04:21,120 --> 00:04:23,600
Success meant the job ran, the schedule didn't slip,

130
00:04:23,600 --> 00:04:25,640
and nothing paged you at 2am.

131
00:04:25,640 --> 00:04:28,120
The pipeline graph was the artifact you showed leadership

132
00:04:28,120 --> 00:04:29,040
to prove progress.

133
00:04:29,040 --> 00:04:30,960
Look, the boxes connect, the arrows flow,

134
00:04:30,960 --> 00:04:32,320
the runtime is green.

135
00:04:32,320 --> 00:04:34,280
And honestly, in that era, it made sense

136
00:04:34,280 --> 00:04:36,240
because infrastructure was the bottleneck.

137
00:04:36,240 --> 00:04:39,040
You didn't wake up and casually provision an estate.

138
00:04:39,040 --> 00:04:41,680
You requested clusters, you negotiated gateways,

139
00:04:41,680 --> 00:04:44,760
you waited on networking, you argued over service limits,

140
00:04:44,760 --> 00:04:46,440
you begged for firewall holes,

141
00:04:46,440 --> 00:04:49,000
and because all of that took time, the organization learned

142
00:04:49,000 --> 00:04:52,560
to behavior, treat delivery as a construction project,

143
00:04:52,560 --> 00:04:55,640
engineers as builders, environments as hard boundaries,

144
00:04:55,640 --> 00:04:57,520
releases as events.

145
00:04:57,520 --> 00:04:59,920
SQL was the interface because SQL was the only thing

146
00:04:59,920 --> 00:05:01,320
that survived handoffs.

147
00:05:01,320 --> 00:05:03,840
Data factory to Synapse, Synapse to a warehouse,

148
00:05:03,840 --> 00:05:06,280
warehouse to Power BI, each layer forced you

149
00:05:06,280 --> 00:05:07,800
to speak a shared language.

150
00:05:07,800 --> 00:05:10,480
That language wasn't always pretty, but it was explicit.

151
00:05:10,480 --> 00:05:13,360
If a transformation mattered, it got written down somewhere.

152
00:05:13,360 --> 00:05:17,240
In SQL, in stored procedures, in DBT models, in views.

153
00:05:17,240 --> 00:05:19,720
And if it didn't get written down, it usually didn't ship

154
00:05:19,720 --> 00:05:22,400
because the next tool in the chain demanded a decision.

155
00:05:22,400 --> 00:05:25,600
Toolspro wasn't expected tax, and it came with a hidden benefit.

156
00:05:25,600 --> 00:05:27,920
But when you had ADF over here, a warehouse over there

157
00:05:27,920 --> 00:05:29,800
and Power BI in a different portal,

158
00:05:29,800 --> 00:05:31,760
you couldn't pretend the boundaries weren't real.

159
00:05:31,760 --> 00:05:33,440
Security lived in multiple places,

160
00:05:33,440 --> 00:05:35,080
compute lived in multiple places,

161
00:05:35,080 --> 00:05:36,560
storage lived in multiple places.

162
00:05:36,560 --> 00:05:38,640
That forced explicit ownership conversations,

163
00:05:38,640 --> 00:05:40,080
even if they were miserable.

164
00:05:40,080 --> 00:05:41,040
Who owns the lake?

165
00:05:41,040 --> 00:05:42,640
Who owns the warehouse schema?

166
00:05:42,640 --> 00:05:44,320
Who owns the semantic model?

167
00:05:44,320 --> 00:05:45,440
Who is allowed to publish?

168
00:05:45,440 --> 00:05:46,920
Who approves refresh schedules?

169
00:05:46,920 --> 00:05:50,000
You didn't always like the answers, but the system demanded you ask.

170
00:05:50,000 --> 00:05:52,200
Deployment friction was also accidental governance.

171
00:05:52,200 --> 00:05:54,600
PRs, approvals, promotion between dev and prod,

172
00:05:54,600 --> 00:05:56,560
long run times, limited concurrency,

173
00:05:56,560 --> 00:05:58,680
none of that made engineers happy.

174
00:05:58,680 --> 00:05:59,840
But it created a throttle.

175
00:05:59,840 --> 00:06:01,880
A bad decision had to survive a review cycle.

176
00:06:01,880 --> 00:06:04,600
A schema change had to survive someone noticing it.

177
00:06:04,600 --> 00:06:07,080
A new pipeline had to survive an environment boundary.

178
00:06:07,080 --> 00:06:09,520
You could still make mistakes, but you couldn't make them

179
00:06:09,520 --> 00:06:11,680
every five minutes across every surface.

180
00:06:11,680 --> 00:06:14,920
And that mattered because failure used to look like an outage.

181
00:06:14,920 --> 00:06:16,800
A pipeline failed, a job didn't run,

182
00:06:16,800 --> 00:06:18,320
a dataset didn't refresh.

183
00:06:18,320 --> 00:06:19,640
Something was visibly broken.

184
00:06:19,640 --> 00:06:21,760
What didn't happen as often was quietly wrong.

185
00:06:21,760 --> 00:06:23,840
You didn't usually get a perfect green check mark attached

186
00:06:23,840 --> 00:06:25,040
to a perfectly wrong dashboard

187
00:06:25,040 --> 00:06:27,440
that refreshed on time and lied consistently.

188
00:06:27,440 --> 00:06:29,440
Not because the old world was morally superior

189
00:06:29,440 --> 00:06:30,480
because it was slower.

190
00:06:30,480 --> 00:06:32,640
Slow systems leak less ambiguity per hour.

191
00:06:32,640 --> 00:06:34,560
Here's the key insight to keeping your head

192
00:06:34,560 --> 00:06:36,560
that friction did two things at once.

193
00:06:36,560 --> 00:06:38,280
It slowed bad decisions down.

194
00:06:38,280 --> 00:06:41,080
And it hid ownership gaps by reducing the rate of change.

195
00:06:41,080 --> 00:06:44,120
So when leaders say why did fabric make everything feel out of control?

196
00:06:44,120 --> 00:06:46,520
The answer isn't that fabric broke governance.

197
00:06:46,520 --> 00:06:49,320
The old stack just rationed chaos.

198
00:06:49,320 --> 00:06:52,600
Fabrics real rewrite from pipeline graph to platform physics.

199
00:06:52,600 --> 00:06:55,360
Fabrics real rewrite isn't that it gave you nicer tooling.

200
00:06:55,360 --> 00:06:57,080
It rewired where decisions happen.

201
00:06:57,080 --> 00:07:00,360
In the old model, your pipeline graph set on top of separate systems.

202
00:07:00,360 --> 00:07:02,440
The graph orchestrated movement between places

203
00:07:02,440 --> 00:07:04,360
that were owned differently, built differently,

204
00:07:04,360 --> 00:07:07,000
secured differently, and optimized differently

205
00:07:07,000 --> 00:07:08,360
that forced you to think in boundaries

206
00:07:08,360 --> 00:07:11,000
because the platform forced you to pay for boundaries.

207
00:07:11,000 --> 00:07:14,360
Fabric collapses that whole experience into one SaaS control plane

208
00:07:14,360 --> 00:07:16,600
and the first consequence is that the workspace

209
00:07:16,600 --> 00:07:18,200
becomes the unit of reality.

210
00:07:18,200 --> 00:07:20,840
Not the database, not the subscription, not the cluster,

211
00:07:20,840 --> 00:07:21,800
the workspace.

212
00:07:21,800 --> 00:07:24,600
Microsoft markets this as one experience,

213
00:07:24,600 --> 00:07:25,800
and that's accurate,

214
00:07:25,800 --> 00:07:27,880
but the technical meaning is more interesting.

215
00:07:27,880 --> 00:07:30,200
Fabric gives every workload,

216
00:07:30,200 --> 00:07:32,920
Lakehouse, warehouse, data factory, notebooks,

217
00:07:32,920 --> 00:07:36,520
Power BI, real-time intelligence, data agents,

218
00:07:36,520 --> 00:07:38,520
one shared containment model.

219
00:07:38,520 --> 00:07:41,400
Same surface, same identity plane, same capacity envelope,

220
00:07:41,400 --> 00:07:43,240
same place where people click share.

221
00:07:43,240 --> 00:07:46,120
So the platform stops behaving like a chain of systems.

222
00:07:46,120 --> 00:07:48,520
It behaves like a single authorization graph

223
00:07:48,520 --> 00:07:50,680
attached to a single compute meter.

224
00:07:50,680 --> 00:07:53,080
That distinction matters because once you consolidate

225
00:07:53,080 --> 00:07:55,320
compute storage and publishing into one place

226
00:07:55,320 --> 00:07:57,080
you no longer have handoffs.

227
00:07:57,080 --> 00:07:58,280
You have lateral movement.

228
00:07:58,280 --> 00:08:00,360
Engineers stop handing work off across tools

229
00:08:00,360 --> 00:08:02,360
and start handing it around inside the workspace.

230
00:08:02,360 --> 00:08:04,120
And because the mechanics feel frictionless,

231
00:08:04,120 --> 00:08:06,600
teams interpret that as architectural progress.

232
00:08:06,600 --> 00:08:09,960
But the real story is what got removed, the walls between responsibilities.

233
00:08:09,960 --> 00:08:11,960
Fabric collapses the old stack in four ways

234
00:08:11,960 --> 00:08:14,360
that are convenient in demos and brutal in real estate.

235
00:08:14,360 --> 00:08:17,720
First storage gets pulled under a single narrative, one lake.

236
00:08:17,720 --> 00:08:20,040
The one drive for data analogy is catchy,

237
00:08:20,040 --> 00:08:22,760
and it's not wrong in the sense that you get a unified lake layer

238
00:08:22,760 --> 00:08:23,800
and a unified catalog,

239
00:08:23,800 --> 00:08:26,760
but the actual behavior that matters for engineering

240
00:08:26,760 --> 00:08:27,880
isn't the metaphor.

241
00:08:27,880 --> 00:08:28,680
It's gravity.

242
00:08:28,680 --> 00:08:31,080
One lake becomes the default landing zone for everything,

243
00:08:31,080 --> 00:08:32,200
and the moment that happens,

244
00:08:32,200 --> 00:08:35,160
people stop asking who owns the data because it's in the lake.

245
00:08:35,160 --> 00:08:36,840
Ownership turns into location.

246
00:08:36,840 --> 00:08:37,880
That's not ownership.

247
00:08:37,880 --> 00:08:41,000
Second compute gets abstracted into capacity.

248
00:08:41,000 --> 00:08:43,080
You don't pay per engine the way you used to.

249
00:08:43,080 --> 00:08:45,880
You pay for the shared meter that every engine draws from.

250
00:08:45,880 --> 00:08:48,120
Spark sessions, warehouse queries, model refreshes,

251
00:08:48,120 --> 00:08:49,800
interactive exploration, same pool.

252
00:08:49,800 --> 00:08:51,160
That sounds like simplification

253
00:08:51,160 --> 00:08:53,640
until you realize what it does to accountability.

254
00:08:53,640 --> 00:08:55,640
When the meter is shared, everyone believes

255
00:08:55,640 --> 00:08:57,960
someone else is responsible for the spike.

256
00:08:57,960 --> 00:08:59,160
And because it's a SaaS platform,

257
00:08:59,160 --> 00:09:01,240
the infrastructure isn't yours to tune,

258
00:09:01,240 --> 00:09:03,400
so teams search for comfort in the UI,

259
00:09:03,400 --> 00:09:05,960
instead of certainty in the workload behavior.

260
00:09:05,960 --> 00:09:08,520
Third, security baselines get normalized.

261
00:09:08,520 --> 00:09:10,200
Workspaces come with a simple role model.

262
00:09:10,200 --> 00:09:11,160
Great for adoption.

263
00:09:11,160 --> 00:09:12,440
Dangerous for intent.

264
00:09:12,440 --> 00:09:15,080
Because workspace roles are not a data security model.

265
00:09:15,080 --> 00:09:16,360
They're a collaboration model.

266
00:09:16,360 --> 00:09:17,560
They answer who can work here,

267
00:09:17,560 --> 00:09:18,840
not who can see this table,

268
00:09:18,840 --> 00:09:20,120
not who can query this view,

269
00:09:20,120 --> 00:09:22,360
not who is allowed to infer this sensitive attribute

270
00:09:22,360 --> 00:09:23,240
through a join.

271
00:09:23,240 --> 00:09:25,640
But when everything lives behind the same workspace boundary,

272
00:09:25,640 --> 00:09:28,200
people treat that boundary as if it were a database boundary.

273
00:09:28,200 --> 00:09:31,400
It is not fourth integration parts become internal.

274
00:09:31,400 --> 00:09:34,360
In the old world, a new pipeline meant choosing connectors,

275
00:09:34,360 --> 00:09:36,760
run times, service principles, landing zones,

276
00:09:36,760 --> 00:09:39,080
and a destination pattern that was ceremony.

277
00:09:39,080 --> 00:09:41,640
Fabric collapses that into item creation.

278
00:09:41,640 --> 00:09:44,760
Data flow, pipeline, notebook, shortcut mirror,

279
00:09:44,760 --> 00:09:47,000
warehouse table, semantic model, click, click,

280
00:09:47,000 --> 00:09:47,640
done.

281
00:09:47,640 --> 00:09:49,480
And because it's all inside one surface,

282
00:09:49,480 --> 00:09:51,480
every path looks equally endorsed.

283
00:09:51,480 --> 00:09:52,840
Shortcuts look like ownership.

284
00:09:52,840 --> 00:09:54,280
Copies look like lineage.

285
00:09:54,280 --> 00:09:55,960
Semantic reuse looks like governance.

286
00:09:55,960 --> 00:09:58,360
Meanwhile, the actual physics never changed.

287
00:09:58,360 --> 00:10:01,240
Data still has three realities you can't negotiate with.

288
00:10:01,240 --> 00:10:03,640
Cost, contracts, and control.

289
00:10:03,640 --> 00:10:06,520
Costs still exist, but now it's surfaced as capacity behavior.

290
00:10:06,520 --> 00:10:09,160
Contracts still exist, but now they're optional,

291
00:10:09,160 --> 00:10:10,440
unless you enforce them.

292
00:10:10,440 --> 00:10:13,880
Controls still exists, but now it's easy to confuse access

293
00:10:13,880 --> 00:10:16,040
with intent.

294
00:10:16,040 --> 00:10:18,120
This is the foundational misunderstanding.

295
00:10:18,120 --> 00:10:20,680
Fabric removed the walls, not the physics.

296
00:10:20,680 --> 00:10:23,560
When Microsoft says workloads become experiences,

297
00:10:23,560 --> 00:10:26,280
what that means in practice is that workloads become modes.

298
00:10:26,280 --> 00:10:28,600
A lake house isn't a separate product you provision.

299
00:10:28,600 --> 00:10:30,920
It's an item inside the workspace.

300
00:10:30,920 --> 00:10:33,480
A warehouse isn't a separate environment you protect.

301
00:10:33,480 --> 00:10:36,200
It's an item inside the same permission model.

302
00:10:36,200 --> 00:10:37,640
Power BI isn't downstream.

303
00:10:37,640 --> 00:10:40,040
It's in the same place, pointed at the same assets,

304
00:10:40,040 --> 00:10:42,440
built by the same people, sometimes in the same afternoon.

305
00:10:42,440 --> 00:10:44,840
So the integration story becomes dangerously simple.

306
00:10:44,840 --> 00:10:47,000
If someone can see it, they can use it.

307
00:10:47,000 --> 00:10:48,280
If they can use it, they will.

308
00:10:48,280 --> 00:10:49,960
If they can publish it, they'll publish it.

309
00:10:49,960 --> 00:10:52,120
And once something gets published, it becomes production.

310
00:10:52,120 --> 00:10:54,200
Because executives don't care how it was made.

311
00:10:54,200 --> 00:10:55,400
They care that it exists.

312
00:10:55,400 --> 00:10:58,360
That's why fabric feels like velocity and chaos at the same time.

313
00:10:58,360 --> 00:11:00,520
Because you're no longer fighting tools.

314
00:11:00,520 --> 00:11:03,640
You're fighting entropy in a single shared control plane.

315
00:11:03,640 --> 00:11:06,040
And now we need to talk about what speed does

316
00:11:06,040 --> 00:11:07,800
when determinism isn't enforced.

317
00:11:07,800 --> 00:11:11,080
Because this is where one platform turns into one blast radius.

318
00:11:11,080 --> 00:11:15,160
In a speed without determinism, why faster feels like less safe?

319
00:11:15,160 --> 00:11:18,200
Most teams confuse pipeline speed with decision speed.

320
00:11:18,200 --> 00:11:20,840
Fabric absolutely makes execution faster.

321
00:11:20,840 --> 00:11:21,960
Ingestion is simpler.

322
00:11:21,960 --> 00:11:23,560
Notebooks are one click away.

323
00:11:23,560 --> 00:11:25,880
Direct lake lights up visuals and the platform

324
00:11:25,880 --> 00:11:27,640
removes a lot of the old drag.

325
00:11:27,640 --> 00:11:29,160
But the hidden variable is that

326
00:11:29,160 --> 00:11:32,760
the rate of change goes up and with it, the rate of unreviewed decisions.

327
00:11:32,760 --> 00:11:34,040
You didn't just speed up jobs.

328
00:11:34,040 --> 00:11:35,640
You sped up policy mistakes.

329
00:11:35,640 --> 00:11:37,240
And the moment the change rate goes up,

330
00:11:37,240 --> 00:11:39,320
ambiguity stops being a documentation problem.

331
00:11:39,320 --> 00:11:40,840
It becomes a production behavior.

332
00:11:40,840 --> 00:11:43,800
Because ambiguity in data engineering isn't philosophical.

333
00:11:43,800 --> 00:11:46,360
It's concrete drift, schema drift, naming drift,

334
00:11:46,360 --> 00:11:48,360
semantics drift, access drift.

335
00:11:48,360 --> 00:11:50,120
In the old stack, drift still happened,

336
00:11:50,120 --> 00:11:52,120
but it had to crawl through boundaries.

337
00:11:52,120 --> 00:11:54,680
Now it moves at the same speed as a refresh schedule

338
00:11:54,680 --> 00:11:56,680
and the same speed as a co-pilot suggestion.

339
00:11:56,680 --> 00:11:58,840
So faster starts to feel like less safe.

340
00:11:58,840 --> 00:12:00,520
Not because fabric is unsafe,

341
00:12:00,520 --> 00:12:03,320
but because determinism isn't enforced by default.

342
00:12:03,320 --> 00:12:05,000
You're operating a high-speed platform

343
00:12:05,000 --> 00:12:07,800
with low friction publishing and mostly social contracts.

344
00:12:07,800 --> 00:12:09,880
Social contracts do not survive scale.

345
00:12:09,880 --> 00:12:11,320
Here's the real mechanism.

346
00:12:11,320 --> 00:12:13,480
Default behaviors become architecture.

347
00:12:13,480 --> 00:12:15,720
Not in a poetic way, in an operational way.

348
00:12:15,720 --> 00:12:17,480
If a workspace has no naming conventions,

349
00:12:17,480 --> 00:12:20,120
then whatever the first team did becomes the convention.

350
00:12:20,120 --> 00:12:22,520
If the lake house lands files with inconsistent types,

351
00:12:22,520 --> 00:12:26,120
then the warehouse consumes that inconsistency unless you reject it.

352
00:12:26,120 --> 00:12:28,920
If the semantic model gets built directly off raw tables

353
00:12:28,920 --> 00:12:30,120
because it's convenient,

354
00:12:30,120 --> 00:12:32,040
that convenience becomes a dependency graph.

355
00:12:32,040 --> 00:12:34,440
And once dashboards exist, nobody wants to touch the source

356
00:12:34,440 --> 00:12:36,200
because now you're breaking the business.

357
00:12:36,200 --> 00:12:39,400
So you end up with workspace conventions replacing contracts

358
00:12:39,400 --> 00:12:41,080
and shortcuts replacing ownership

359
00:12:41,080 --> 00:12:43,320
and it refreshes replacing correctness.

360
00:12:43,320 --> 00:12:45,640
That distinction matters because fabric makes it easy

361
00:12:45,640 --> 00:12:47,400
to create the downstream artifacts

362
00:12:47,400 --> 00:12:49,720
before you've locked the upstream invariance.

363
00:12:49,720 --> 00:12:51,240
And when downstream exists first,

364
00:12:51,240 --> 00:12:53,080
the organization starts managing backwards,

365
00:12:53,080 --> 00:12:54,920
patch the semantic layer, patch the DAX,

366
00:12:54,920 --> 00:12:57,320
patch the report, patch the refresh schedule.

367
00:12:57,320 --> 00:12:59,080
The data layer becomes a landfill of,

368
00:12:59,080 --> 00:13:01,400
we'll clean it later, later never comes.

369
00:13:01,400 --> 00:13:03,080
Now add capacity pricing to this

370
00:13:03,080 --> 00:13:05,400
because this is where fabric gets brutally honest.

371
00:13:05,400 --> 00:13:07,080
In the old world, you paid per tool.

372
00:13:07,080 --> 00:13:09,560
You could hide inefficiency inside somebody else's bill,

373
00:13:09,560 --> 00:13:12,360
the warehouse bill, the spark bill, the BI bill.

374
00:13:12,360 --> 00:13:13,880
Fabric makes the chaos shared.

375
00:13:13,880 --> 00:13:15,400
Everything draws from one meter.

376
00:13:15,400 --> 00:13:18,520
So the cost of ambiguity shows up as compute consumption.

377
00:13:18,520 --> 00:13:21,400
Spikes, throttling, degraded interactivity,

378
00:13:21,400 --> 00:13:22,680
refresh contention,

379
00:13:22,680 --> 00:13:26,040
and a monthly bill that suddenly has a personality.

380
00:13:26,040 --> 00:13:28,440
And leadership doesn't care that the sequel was only used

381
00:13:28,440 --> 00:13:29,720
for a small result set.

382
00:13:29,720 --> 00:13:31,400
The platform still paid for the scan.

383
00:13:31,400 --> 00:13:33,160
This is why fabric becomes the first place

384
00:13:33,160 --> 00:13:35,240
many organizations experience cost incidents

385
00:13:35,240 --> 00:13:36,440
as their primary signal.

386
00:13:36,440 --> 00:13:39,960
Not outages, not failures, cost, capacity saturation,

387
00:13:39,960 --> 00:13:41,960
performance degradation, the system functions,

388
00:13:41,960 --> 00:13:44,120
but it does so expensively and unpredictably,

389
00:13:44,120 --> 00:13:45,640
which is just another way of saying

390
00:13:45,640 --> 00:13:47,320
your assumptions are no longer free.

391
00:13:47,320 --> 00:13:49,240
You can't hope your way into stable spend,

392
00:13:49,240 --> 00:13:52,200
you can't train your way into deterministic access boundaries.

393
00:13:52,200 --> 00:13:54,760
You can't label your way into a schema contract.

394
00:13:54,760 --> 00:13:56,840
Fabric doesn't break, your assumptions do.

395
00:13:56,840 --> 00:13:59,960
And once you see that, you realize why the platform feels

396
00:13:59,960 --> 00:14:01,400
like it's tightening around you

397
00:14:01,400 --> 00:14:03,720
because you're no longer amortizing mistakes over time.

398
00:14:03,720 --> 00:14:05,480
You're paying for them immediately at scale

399
00:14:05,480 --> 00:14:07,960
on a shared meter with downstream artifacts

400
00:14:07,960 --> 00:14:10,440
that create political resistance to fixing the source.

401
00:14:10,440 --> 00:14:11,640
So yes, fabric is faster.

402
00:14:11,640 --> 00:14:14,440
But speed without gates is just higher frequency failure.

403
00:14:14,440 --> 00:14:17,240
And now co-pilot becomes the acceleration layer on top of that,

404
00:14:17,240 --> 00:14:19,160
because it doesn't just speed up execution.

405
00:14:19,160 --> 00:14:21,880
It speeds up the creation of plausible and bounded decisions.

406
00:14:22,040 --> 00:14:24,760
Co-pilot's real impact, completion over consequence.

407
00:14:24,760 --> 00:14:27,640
Co-pilot for Microsoft fabric is the part everyone wants to talk about

408
00:14:27,640 --> 00:14:29,480
because it looks like free velocity.

409
00:14:29,480 --> 00:14:31,480
You type a sentence and it writes the pipeline,

410
00:14:31,480 --> 00:14:33,800
it writes the notebook cell, it writes the school,

411
00:14:33,800 --> 00:14:36,120
it writes the KQL, it suggests the transformation,

412
00:14:36,120 --> 00:14:39,080
it will even explain the error message it helped create.

413
00:14:39,080 --> 00:14:41,480
And the uncomfortable truth is that it does exactly

414
00:14:41,480 --> 00:14:42,680
what it's designed to do.

415
00:14:42,680 --> 00:14:44,280
It helps you complete a task,

416
00:14:44,280 --> 00:14:46,360
but co-pilot doesn't live in your incident timeline.

417
00:14:46,360 --> 00:14:47,640
It doesn't live in your budget,

418
00:14:47,640 --> 00:14:49,560
it doesn't live in your audit findings.

419
00:14:49,560 --> 00:14:53,400
So when you ask, what does co-pilot actually change in data engineering?

420
00:14:53,400 --> 00:14:55,400
The answer is not it replaces engineers

421
00:14:55,400 --> 00:14:57,640
and it eliminates complexity.

422
00:14:57,640 --> 00:15:00,040
It changes the failure rate of ambiguity.

423
00:15:00,040 --> 00:15:02,280
Because the optimization target is completion,

424
00:15:02,280 --> 00:15:03,880
plausible output fast,

425
00:15:03,880 --> 00:15:05,000
not deterministic cost,

426
00:15:05,000 --> 00:15:06,280
not governance intent,

427
00:15:06,280 --> 00:15:07,800
not long term correctness.

428
00:15:07,800 --> 00:15:09,240
Co-pilot writes answers.

429
00:15:09,240 --> 00:15:11,320
Data engineering is about consequences.

430
00:15:11,320 --> 00:15:12,680
Here's what most teams miss.

431
00:15:12,680 --> 00:15:14,840
Co-pilot doesn't generate work in a vacuum.

432
00:15:14,840 --> 00:15:16,680
It generates work inside a platform

433
00:15:16,680 --> 00:15:19,480
where the easiest path is also the widest blast radius.

434
00:15:19,480 --> 00:15:21,560
Fabric gives you immediate downstream surfaces,

435
00:15:21,560 --> 00:15:24,280
direct lakes, semantic models, reports, data agents.

436
00:15:24,280 --> 00:15:27,480
So whatever co-pilot produces can become real fast.

437
00:15:27,480 --> 00:15:29,400
And once it becomes real, it becomes defended.

438
00:15:29,400 --> 00:15:32,440
People build slides on it, they send links, they make decisions,

439
00:15:32,440 --> 00:15:33,800
then you, the inheriting architect,

440
00:15:33,800 --> 00:15:35,480
get asked why the numbers changed.

441
00:15:35,480 --> 00:15:37,640
Co-pilot didn't break anything.

442
00:15:37,640 --> 00:15:39,480
It just made it easy to ship an assumption

443
00:15:39,480 --> 00:15:41,160
before you encoded it as a contract.

444
00:15:41,160 --> 00:15:42,920
And you can see the pattern across workloads.

445
00:15:42,920 --> 00:15:45,480
In a warehouse, co-pilot will happily generate a query

446
00:15:45,480 --> 00:15:47,480
that returns the right-looking columns

447
00:15:47,480 --> 00:15:50,120
with the right-looking joins for the right-looking question.

448
00:15:50,120 --> 00:15:52,920
It will also happily do it without a bounded time predicate,

449
00:15:52,920 --> 00:15:54,600
without respecting partitioning intent,

450
00:15:54,600 --> 00:15:57,720
and without any awareness of what your capacity meter will do

451
00:15:57,720 --> 00:16:00,600
when someone runs it at 9 a.m., alongside three refreshes

452
00:16:00,600 --> 00:16:03,160
and 10 other analysts trying to just check something.

453
00:16:03,160 --> 00:16:05,320
In a notebook, it will generate transformations

454
00:16:05,320 --> 00:16:07,000
that look reasonable and run.

455
00:16:07,000 --> 00:16:09,880
It won't stop and ask you if the lake house is schema-on-read

456
00:16:09,880 --> 00:16:11,960
and therefore silently accepting drift.

457
00:16:11,960 --> 00:16:13,720
It won't insist on validation gates.

458
00:16:13,720 --> 00:16:16,680
It won't create quarantine tables unless you tell it to.

459
00:16:16,680 --> 00:16:19,000
And if you don't tell it to, you don't have quality enforcement.

460
00:16:19,000 --> 00:16:21,000
You have optimism with a refresh schedule.

461
00:16:21,000 --> 00:16:22,680
In Data Factory, it will generate pipelines

462
00:16:22,680 --> 00:16:24,520
that connect and move data.

463
00:16:24,520 --> 00:16:26,840
It won't force you to define ownership boundaries

464
00:16:26,840 --> 00:16:28,040
or consumption contracts.

465
00:16:28,040 --> 00:16:31,560
It will create a path and paths at scale become dependency.

466
00:16:31,560 --> 00:16:32,920
And then there are data agents.

467
00:16:32,920 --> 00:16:34,680
This is where the illusion gets dangerous.

468
00:16:34,680 --> 00:16:36,680
Because now the natural language layer is pointed

469
00:16:36,680 --> 00:16:37,800
at your models and tables

470
00:16:37,800 --> 00:16:40,680
and people interpret conversational access as governed access.

471
00:16:40,680 --> 00:16:43,800
But the agent can only operate within what the caller can already see.

472
00:16:43,800 --> 00:16:45,880
That means co-pilot doesn't inherit your intent.

473
00:16:45,880 --> 00:16:47,560
It inherits your permissions.

474
00:16:47,560 --> 00:16:48,920
If your permissions are broad

475
00:16:48,920 --> 00:16:51,960
because workspace roles became the whole security strategy,

476
00:16:51,960 --> 00:16:53,640
co-pilot becomes a really efficient way

477
00:16:53,640 --> 00:16:55,720
to explore things you didn't mean to expose.

478
00:16:55,720 --> 00:16:58,200
This is why it worked becomes, it's correct

479
00:16:58,200 --> 00:17:00,280
in fabric estates that lean on co-pilot.

480
00:17:00,280 --> 00:17:02,040
The result renders, the dashboard refreshes,

481
00:17:02,040 --> 00:17:04,040
the agent answers, nobody gets an error.

482
00:17:04,040 --> 00:17:07,560
And in modern organizations, no error is treated as proof of correctness.

483
00:17:07,560 --> 00:17:10,200
It isn't, co-pilot makes two kinds of teams faster.

484
00:17:10,200 --> 00:17:11,960
The teams with enforcement become faster

485
00:17:11,960 --> 00:17:14,680
at implementing decisions they already made deliberately.

486
00:17:14,680 --> 00:17:16,760
Teams without enforcement become faster

487
00:17:16,760 --> 00:17:19,080
at producing artifacts that look like decisions.

488
00:17:19,080 --> 00:17:20,200
Those are not the same thing.

489
00:17:20,200 --> 00:17:21,880
So the real governance limitation

490
00:17:21,880 --> 00:17:23,400
isn't that co-pilot hallucinates.

491
00:17:23,400 --> 00:17:27,240
The deeper limitation is that co-pilot has no native concept of your invariance

492
00:17:27,240 --> 00:17:29,160
unless you encode them into the system.

493
00:17:29,160 --> 00:17:31,880
Schemas, views, procedures, roles,

494
00:17:31,880 --> 00:17:33,240
CICD gates,

495
00:17:33,240 --> 00:17:36,920
and acceptance criteria like execution plans and violation counts.

496
00:17:36,920 --> 00:17:38,760
If you don't enforce those surfaces,

497
00:17:38,760 --> 00:17:40,840
co-pilot doesn't accelerate engineering.

498
00:17:40,840 --> 00:17:42,200
It accelerates entropy.

499
00:17:42,200 --> 00:17:45,480
And now we can stop theorizing because the failure modes aren't abstract.

500
00:17:45,480 --> 00:17:48,600
They show up as cost incidents, correctness incidents,

501
00:17:48,600 --> 00:17:51,080
and access incidents often with no obvious break.

502
00:17:51,080 --> 00:17:54,680
Case one, setup, the cost drift that looked like a ghost.

503
00:17:54,680 --> 00:17:58,520
The first fabric estate failure that shows up for most organizations

504
00:17:58,520 --> 00:18:00,200
isn't a security breach,

505
00:18:00,200 --> 00:18:02,440
and it isn't a data quality scandal.

506
00:18:02,440 --> 00:18:03,480
It's a bill.

507
00:18:03,480 --> 00:18:06,360
Or more accurately, a capacity graph that looks haunted.

508
00:18:06,360 --> 00:18:08,520
Here's the symptom pattern, you're on a steady state.

509
00:18:08,520 --> 00:18:09,800
The business feels good.

510
00:18:09,800 --> 00:18:11,000
Dashboards refresh.

511
00:18:11,000 --> 00:18:12,200
Nobody deployed anything.

512
00:18:12,200 --> 00:18:13,800
There wasn't a release weekend.

513
00:18:13,800 --> 00:18:15,000
No big new data set.

514
00:18:15,000 --> 00:18:17,000
No we migrated to a new model.

515
00:18:17,000 --> 00:18:19,240
And then the capacity metrics spike hard,

516
00:18:19,240 --> 00:18:21,080
usually right in the middle of the workday,

517
00:18:21,080 --> 00:18:24,440
and usually in bursts that don't line up with your pipeline schedule.

518
00:18:24,440 --> 00:18:26,600
Leadership asks the worst question in the world.

519
00:18:26,600 --> 00:18:27,560
What changed?

520
00:18:27,560 --> 00:18:30,680
And you end up in that familiar meeting where five people say nothing

521
00:18:30,680 --> 00:18:32,120
with total sincerity.

522
00:18:32,120 --> 00:18:34,200
Because from their perspective, nothing changed.

523
00:18:34,200 --> 00:18:35,320
The reports still load.

524
00:18:35,320 --> 00:18:36,520
The refresh is still complete.

525
00:18:36,520 --> 00:18:38,600
The pipeline success rate is still green.

526
00:18:38,600 --> 00:18:41,560
So the organization interprets the spike as billing weirdness

527
00:18:41,560 --> 00:18:43,080
or a fabric capacity glitch,

528
00:18:43,080 --> 00:18:45,320
or Microsoft did something in the service.

529
00:18:45,320 --> 00:18:45,960
Plot twist.

530
00:18:45,960 --> 00:18:47,400
It's almost never a platform ghost.

531
00:18:47,400 --> 00:18:48,920
It's a query-shaped problem.

532
00:18:48,920 --> 00:18:51,160
Fabric just made cost the first incident signal

533
00:18:51,160 --> 00:18:53,400
because cost is the only thing the platform can't hide

534
00:18:53,400 --> 00:18:55,320
when everything shares the same meter.

535
00:18:55,320 --> 00:18:56,840
You can mask correctness for weeks.

536
00:18:56,840 --> 00:18:58,520
You can mask access drift for months.

537
00:18:58,520 --> 00:19:00,200
But you cannot mask capacity pressure

538
00:19:00,200 --> 00:19:01,560
when 10 people run wide,

539
00:19:01,560 --> 00:19:03,400
unbounded queries at the same time.

540
00:19:03,400 --> 00:19:05,080
The semantic model is refreshing.

541
00:19:05,080 --> 00:19:06,680
So in fabric, the first question isn't,

542
00:19:06,680 --> 00:19:07,560
did it fail?

543
00:19:07,560 --> 00:19:09,240
It's what did it do to the meter?

544
00:19:09,240 --> 00:19:10,600
And that's why this case matters.

545
00:19:10,600 --> 00:19:11,960
It forces a new discipline.

546
00:19:11,960 --> 00:19:14,600
You have to treat cost as an engineered property,

547
00:19:14,600 --> 00:19:16,200
not as an after-the-fact reporter.

548
00:19:16,200 --> 00:19:18,680
If cost is undefined, governance is imaginary.

549
00:19:18,680 --> 00:19:21,080
Because a platform that charges you for runtime

550
00:19:21,080 --> 00:19:23,400
doesn't care whether your results hit was small.

551
00:19:23,400 --> 00:19:25,400
It charges you for what you asked the engine to do.

552
00:19:25,400 --> 00:19:27,880
Now, to debug this without lying to yourself,

553
00:19:27,880 --> 00:19:29,720
you need ground-truth artifacts.

554
00:19:29,720 --> 00:19:32,600
Not screenshots, not guesses, not it feels slower.

555
00:19:32,600 --> 00:19:33,880
Ground truth means three things.

556
00:19:33,880 --> 00:19:35,160
One, execution plans.

557
00:19:35,160 --> 00:19:36,120
Not the query text.

558
00:19:36,120 --> 00:19:36,920
The plan.

559
00:19:36,920 --> 00:19:37,880
The joint types.

560
00:19:37,880 --> 00:19:38,520
The scans.

561
00:19:38,520 --> 00:19:39,160
The sorts.

562
00:19:39,160 --> 00:19:39,800
The spills.

563
00:19:39,800 --> 00:19:40,760
The operator costs.

564
00:19:40,760 --> 00:19:42,280
The shape of the work.

565
00:19:42,280 --> 00:19:44,760
Two, scan rows versus returned rows.

566
00:19:44,760 --> 00:19:47,160
That ratio tells you whether you're paying for a seek

567
00:19:47,160 --> 00:19:48,520
or funding a full-world scan

568
00:19:48,520 --> 00:19:50,600
to retrieve 15 rows for a visual.

569
00:19:50,600 --> 00:19:53,880
Three, fabric capacity metrics aligned to time windows.

570
00:19:53,880 --> 00:19:55,800
You correlate the spike with the query window

571
00:19:55,800 --> 00:19:57,000
and the refresh window.

572
00:19:57,000 --> 00:19:58,120
You don't start with blame.

573
00:19:58,120 --> 00:19:59,480
You start with correlation.

574
00:19:59,480 --> 00:20:00,280
And once you do that,

575
00:20:00,280 --> 00:20:02,360
you'll notice the same pattern over and over.

576
00:20:02,360 --> 00:20:04,760
Interactive bursts plus refresh concurrency

577
00:20:04,760 --> 00:20:06,520
plus one or two helpful queries

578
00:20:06,520 --> 00:20:08,120
that look innocent in a chat window

579
00:20:08,120 --> 00:20:10,680
and behave like a denial of wallet attack in the engine.

580
00:20:10,680 --> 00:20:11,960
That's the setup.

581
00:20:11,960 --> 00:20:13,320
Now we can talk root cause

582
00:20:13,320 --> 00:20:15,160
because the plan will tell on you every time.

583
00:20:15,160 --> 00:20:18,040
In case one root cause, AI-circle, that scan the world.

584
00:20:18,040 --> 00:20:20,440
The root cause usually isn't someone ran a query.

585
00:20:20,440 --> 00:20:22,040
People always ran queries.

586
00:20:22,040 --> 00:20:24,680
The root cause is that co-pilot made it socially acceptable

587
00:20:24,680 --> 00:20:27,160
to run warehouse-scale SQL with no discipline

588
00:20:27,160 --> 00:20:29,240
because it looked professional enough to trust

589
00:20:29,240 --> 00:20:31,000
and fast enough to repeat.

590
00:20:31,000 --> 00:20:33,240
So what does the pattern look like in practice?

591
00:20:33,240 --> 00:20:34,600
First, non-sarguable predicates.

592
00:20:34,600 --> 00:20:36,440
Co-pilot loves convenience syntax.

593
00:20:36,440 --> 00:20:39,320
It'll give you things like filtering on a computed expression,

594
00:20:39,320 --> 00:20:41,400
wrapping date columns in functions,

595
00:20:41,400 --> 00:20:43,320
building conditions that read well to humans

596
00:20:43,320 --> 00:20:45,160
and destroy index usage for engines.

597
00:20:45,160 --> 00:20:46,840
The user sees a clean wear clause.

598
00:20:46,840 --> 00:20:49,000
The optimizer sees cool, I can't seek.

599
00:20:49,000 --> 00:20:50,120
And so it scans.

600
00:20:50,120 --> 00:20:52,760
And on a big warehouse table, scan is not a detail.

601
00:20:52,760 --> 00:20:53,480
It's the bill.

602
00:20:53,480 --> 00:20:56,600
Second, missing time filters.

603
00:20:56,600 --> 00:20:58,760
This one is almost comical

604
00:20:58,760 --> 00:21:01,560
because the question people ask is always time bound.

605
00:21:01,560 --> 00:21:03,640
Last month, this quarter,

606
00:21:03,640 --> 00:21:05,240
since the campaign started.

607
00:21:05,240 --> 00:21:07,080
But co-pilot tends to generate queries

608
00:21:07,080 --> 00:21:10,520
that are semantically plausible without being operationally bounded.

609
00:21:10,520 --> 00:21:11,800
It'll happily join fact tables

610
00:21:11,800 --> 00:21:14,120
and return the right columns without enforcing a window.

611
00:21:14,120 --> 00:21:15,800
The result set can still be tiny

612
00:21:15,800 --> 00:21:18,280
because the final visual only needs 10 rows.

613
00:21:18,280 --> 00:21:19,880
But the engine had to read everything

614
00:21:19,880 --> 00:21:21,160
to discover those 10 rows.

615
00:21:21,160 --> 00:21:24,120
Third, select star against large tables.

616
00:21:24,120 --> 00:21:26,840
It happens because co-pilot optimizes for completion.

617
00:21:26,840 --> 00:21:28,520
Select, return something.

618
00:21:28,520 --> 00:21:30,120
The user doesn't have to think.

619
00:21:30,120 --> 00:21:31,640
And once you select,

620
00:21:31,640 --> 00:21:35,320
in a warehouse table, you've made two decisions you didn't intend to make.

621
00:21:35,320 --> 00:21:37,480
You've committed to a wider IO footprint

622
00:21:37,480 --> 00:21:38,920
and you've committed to schema drift

623
00:21:38,920 --> 00:21:42,040
because now every new column is automatically in scope

624
00:21:42,040 --> 00:21:44,040
for every downstream consumer.

625
00:21:44,040 --> 00:21:45,400
You didn't just write a query,

626
00:21:45,400 --> 00:21:47,320
you wrote a contract you never reviewed

627
00:21:47,320 --> 00:21:49,000
and here's where people get fooled.

628
00:21:49,000 --> 00:21:50,520
But the report is simple.

629
00:21:50,520 --> 00:21:52,040
But the result is small.

630
00:21:52,040 --> 00:21:53,480
But it finishes quickly.

631
00:21:53,480 --> 00:21:54,600
None of that matters.

632
00:21:54,600 --> 00:21:56,600
What matters is what the plan did.

633
00:21:56,600 --> 00:21:59,240
In execution plans, the tell is consistent.

634
00:21:59,240 --> 00:22:01,400
Large scans feeding hash joins.

635
00:22:01,400 --> 00:22:02,680
Followed by big sorts.

636
00:22:02,680 --> 00:22:03,800
And then you see spills.

637
00:22:03,800 --> 00:22:05,720
Spills are the platform politely admitting

638
00:22:05,720 --> 00:22:09,000
it ran out of memory and decided to rent more time from your capacity.

639
00:22:09,000 --> 00:22:10,360
That's not a performance issue.

640
00:22:10,360 --> 00:22:12,040
That's a cost policy failure.

641
00:22:12,040 --> 00:22:14,760
You can also see it in scandros versus return rows.

642
00:22:14,760 --> 00:22:16,920
When that ratio is absurd,

643
00:22:16,920 --> 00:22:19,160
millions scanned, dozens returned,

644
00:22:19,160 --> 00:22:21,320
you're not looking at someone exploring.

645
00:22:21,320 --> 00:22:23,800
You're looking at a lack of bounded query surfaces.

646
00:22:23,800 --> 00:22:25,800
Now add fabrics abstraction layer

647
00:22:25,800 --> 00:22:27,480
and you get the second illusion.

648
00:22:27,480 --> 00:22:30,040
Semantic model refreshes still succeed.

649
00:22:30,040 --> 00:22:31,800
Direct lag still renders visuals.

650
00:22:31,800 --> 00:22:33,640
The user's experience stays smooth enough

651
00:22:33,640 --> 00:22:35,480
that nobody treats it as an incident.

652
00:22:35,480 --> 00:22:38,360
The platform eats the cost, the dashboards keep refreshing

653
00:22:38,360 --> 00:22:40,680
and the only thing that screams is the capacity meter.

654
00:22:40,680 --> 00:22:43,240
So the spikes correlate to interactive bursts

655
00:22:43,240 --> 00:22:45,240
and refresh concurrency, not deployments.

656
00:22:45,240 --> 00:22:46,520
That's why it feels like a ghost.

657
00:22:46,520 --> 00:22:47,800
Nothing in get changed.

658
00:22:47,800 --> 00:22:50,360
But user behavior changed and co-pilot made that behavior

659
00:22:50,360 --> 00:22:51,640
easier to produce at scale.

660
00:22:51,640 --> 00:22:53,320
This is also why this problem

661
00:22:53,320 --> 00:22:54,920
shows up first in fabric estates

662
00:22:54,920 --> 00:22:57,240
that democratize access early.

663
00:22:57,240 --> 00:22:58,600
You give broad read permissions

664
00:22:58,600 --> 00:22:59,880
because you want adoption.

665
00:22:59,880 --> 00:23:01,560
You don't lock down query surfaces

666
00:23:01,560 --> 00:23:03,560
because you don't want to slow people down.

667
00:23:03,560 --> 00:23:05,480
You let analysts query raw tables

668
00:23:05,480 --> 00:23:07,160
because it's just read only.

669
00:23:07,160 --> 00:23:08,520
Then co-pilot shows up

670
00:23:08,520 --> 00:23:10,360
and now the casual analyst can generate

671
00:23:10,360 --> 00:23:12,200
warehouse grade SQL in 10 seconds

672
00:23:12,200 --> 00:23:14,840
without understanding sagibility, partition elimination,

673
00:23:14,840 --> 00:23:17,240
joint strategy, or the difference between works

674
00:23:17,240 --> 00:23:19,960
and works efficiently under concurrency.

675
00:23:19,960 --> 00:23:21,880
And before anyone gets comfortable blaming co-pilot,

676
00:23:21,880 --> 00:23:23,800
remember co-pilot isn't the villain.

677
00:23:23,800 --> 00:23:25,320
Co-pilot is the amplifier.

678
00:23:25,320 --> 00:23:27,560
The actual design omission is letting raw tables

679
00:23:27,560 --> 00:23:29,320
become the consumption interface.

680
00:23:29,320 --> 00:23:31,240
If raw tables are queryable by default,

681
00:23:31,240 --> 00:23:32,920
then your cost model is probabilistic.

682
00:23:32,920 --> 00:23:35,880
You're gambling that every consumer will behave

683
00:23:35,880 --> 00:23:38,360
like a senior engineer with an execution plan open.

684
00:23:38,360 --> 00:23:38,920
They won't.

685
00:23:38,920 --> 00:23:40,200
So here's the anchor line that matters

686
00:23:40,200 --> 00:23:41,720
for the rest of the episode.

687
00:23:41,720 --> 00:23:43,400
If it doesn't show up in a plan,

688
00:23:43,400 --> 00:23:45,800
a cost report, or a violation count,

689
00:23:45,800 --> 00:23:47,640
it's not governance, it's hope.

690
00:23:47,640 --> 00:23:49,240
In this case, the plan is the confession.

691
00:23:49,240 --> 00:23:50,680
And it always confesses.

692
00:23:50,680 --> 00:23:51,960
Case one, fix.

693
00:23:51,960 --> 00:23:53,480
Views as query surface.

694
00:23:53,480 --> 00:23:55,400
Plans as acceptance criteria.

695
00:23:55,400 --> 00:23:58,200
So the fix isn't tell people to write better school.

696
00:23:58,200 --> 00:24:00,600
That's education, education decays.

697
00:24:00,600 --> 00:24:02,280
The fix is to change the query surface

698
00:24:02,280 --> 00:24:04,840
so the platform can't be helpfully expensive by default.

699
00:24:04,840 --> 00:24:08,440
In other words, you stop letting raw tables be a public API.

700
00:24:08,440 --> 00:24:11,160
You make the warehouse behave like a system with boundaries,

701
00:24:11,160 --> 00:24:12,840
not a playground with billing.

702
00:24:12,840 --> 00:24:15,560
The first enforcement move is simple and unpopular.

703
00:24:15,560 --> 00:24:17,080
Views and stored procedures

704
00:24:17,080 --> 00:24:19,320
become the only supported consumption interface.

705
00:24:19,320 --> 00:24:21,800
Not preferred, not recommended, the only one.

706
00:24:21,800 --> 00:24:23,640
If analysts, reports, data agents,

707
00:24:23,640 --> 00:24:26,520
and ad hoc explorers can hit raw fact tables directly,

708
00:24:26,520 --> 00:24:29,080
then you've already accepted that cost is a shared gamble.

709
00:24:29,080 --> 00:24:30,360
You didn't implement governance.

710
00:24:30,360 --> 00:24:32,440
You implemented hope with a refresh schedule,

711
00:24:32,440 --> 00:24:33,480
so you lock it down.

712
00:24:33,480 --> 00:24:36,760
You create a serving schema, call it something boring like serving,

713
00:24:36,760 --> 00:24:37,880
or consume.

714
00:24:37,880 --> 00:24:40,440
And that is where every queryable object lives.

715
00:24:40,440 --> 00:24:43,480
Views expose only the columns you intend to support,

716
00:24:43,480 --> 00:24:45,800
and only with filters you intend to pay for,

717
00:24:45,800 --> 00:24:49,320
stored procedures become the path for parameterized access.

718
00:24:49,320 --> 00:24:51,400
Date windows, entity scopes,

719
00:24:51,400 --> 00:24:53,400
and give me the last end days,

720
00:24:53,400 --> 00:24:56,440
patents that don't require every consumer to rediscover

721
00:24:56,440 --> 00:24:58,120
sargability the hard way.

722
00:24:58,120 --> 00:25:00,120
That's also where you enforce naming discipline.

723
00:25:00,120 --> 00:25:03,160
You don't expose fact sales total 23 of V2 final final.

724
00:25:03,160 --> 00:25:06,760
You expose sales, fact sales, through a view that hides the underlying mess

725
00:25:06,760 --> 00:25:08,120
until you refactor it.

726
00:25:08,120 --> 00:25:09,720
The consumer gets stability.

727
00:25:09,720 --> 00:25:14,200
You get freedom to change internals without a political incident every time a column gets renamed.

728
00:25:14,200 --> 00:25:16,760
Second execution plans become acceptance criteria,

729
00:25:16,760 --> 00:25:20,040
not for every query in the estate, don't be theatrical.

730
00:25:20,040 --> 00:25:22,680
For critical parts, semantic model refresh queries,

731
00:25:22,680 --> 00:25:27,560
top interactive report queries, and anything that hits large fact tables or runs under concurrency,

732
00:25:27,560 --> 00:25:29,880
you treat the plan the same way you treat a security review,

733
00:25:29,880 --> 00:25:31,560
it's a gate, not a suggestion.

734
00:25:31,560 --> 00:25:34,440
Here are the non-negotiables you enforce in review.

735
00:25:34,440 --> 00:25:35,560
Bounded predicates.

736
00:25:35,560 --> 00:25:37,880
If the query can read the entire table it will.

737
00:25:37,880 --> 00:25:41,560
So you require time windows, partition elimination, and parameterization.

738
00:25:41,560 --> 00:25:43,720
You ban non-sargable predicates in those parts.

739
00:25:43,720 --> 00:25:45,160
You also ban select star.

740
00:25:45,160 --> 00:25:47,960
If you don't name the columns, you don't understand the contract you're creating,

741
00:25:47,960 --> 00:25:49,800
and you're also guaranteeing drift.

742
00:25:49,800 --> 00:25:52,600
And you explicitly look for the usual engine taxes.

743
00:25:52,600 --> 00:25:56,920
Scans where you expected Seaks, hash joins where you expected a narrow data set,

744
00:25:56,920 --> 00:26:00,200
big sorts, and spills. Spills are treated as a defect.

745
00:26:00,200 --> 00:26:02,360
Not a performance defect, a governance defect,

746
00:26:02,360 --> 00:26:07,080
because a spilled query under concurrency is a cost incident waiting for a calendar invite.

747
00:26:07,080 --> 00:26:10,440
Third, you turn capacity metrics into a feedback loop,

748
00:26:10,440 --> 00:26:12,440
not a post-mortem artifact.

749
00:26:12,440 --> 00:26:14,600
You establish a stable cost baseline.

750
00:26:14,600 --> 00:26:17,800
What normal looks like by time of day, by workload, by refresh window,

751
00:26:17,800 --> 00:26:21,400
then you tag and track deviations to query patterns not to teams.

752
00:26:21,400 --> 00:26:22,520
The goal isn't blame.

753
00:26:22,520 --> 00:26:24,440
The goal is making cost predictable.

754
00:26:24,440 --> 00:26:27,080
When a spike happens, you correlate which query shapes run,

755
00:26:27,080 --> 00:26:31,160
which objects they hit, which concurrency windows existed, which refreshes overlapped.

756
00:26:31,160 --> 00:26:33,960
You build a library of this pattern causes that spike.

757
00:26:33,960 --> 00:26:35,400
Then you enforce that library.

758
00:26:35,400 --> 00:26:37,240
Over time, something weird happens.

759
00:26:37,240 --> 00:26:39,080
The random alerts stop being random.

760
00:26:39,080 --> 00:26:40,920
The platform becomes legible again.

761
00:26:40,920 --> 00:26:44,520
And once the cost surface becomes predictable, everything else gets easier.

762
00:26:44,520 --> 00:26:48,840
Because you can now talk about governance like an engineered property with artifacts.

763
00:26:48,840 --> 00:26:51,160
Not an aspirational poster in a wiki.

764
00:26:51,960 --> 00:26:55,480
That's the actual outcome you want from this fix, not cheaper queries,

765
00:26:55,480 --> 00:26:56,600
deterministic behavior.

766
00:26:56,600 --> 00:27:01,160
Now we can move to the next failure mode, because once cost is controlled,

767
00:27:01,160 --> 00:27:02,680
the next thing that breaks is truth.

768
00:27:02,680 --> 00:27:04,040
Case 2.

769
00:27:04,040 --> 00:27:07,480
Set up, Lake House, Warehouse, Contract Collapse.

770
00:27:07,480 --> 00:27:10,280
Once you stop bleeding money, you notice the next failure mode.

771
00:27:10,280 --> 00:27:11,880
Nobody agrees on what's true.

772
00:27:11,880 --> 00:27:14,360
This one shows up as Power BI is wrong,

773
00:27:14,360 --> 00:27:17,080
which is always a useful sentence because it's never specific.

774
00:27:17,080 --> 00:27:21,000
What they mean is two reports disagree, two teams have two official KPIs

775
00:27:21,000 --> 00:27:25,160
and the executive dashboard mysteriously changes when someone improves a model.

776
00:27:25,160 --> 00:27:26,680
The platform keeps refreshing.

777
00:27:26,680 --> 00:27:28,360
Nothing is read, the argument still happens.

778
00:27:28,360 --> 00:27:31,400
And it happens because the contract never existed.

779
00:27:31,400 --> 00:27:34,040
In fabric estates, this usually starts the same way.

780
00:27:34,040 --> 00:27:36,680
A team lands data into a lake house because it's fast,

781
00:27:36,680 --> 00:27:38,040
because notebooks are right there,

782
00:27:38,040 --> 00:27:40,120
because delta tables feel like progress,

783
00:27:40,120 --> 00:27:42,520
and because direct lake makes Power BI light up

784
00:27:42,520 --> 00:27:45,400
without the traditional import and warehouse ceremony.

785
00:27:45,400 --> 00:27:47,320
So the early win is real data shows up,

786
00:27:47,320 --> 00:27:49,160
visuals render, people feel unblocked,

787
00:27:49,160 --> 00:27:51,400
but the lake house mental model is schema on read.

788
00:27:51,400 --> 00:27:52,600
It's permissive by design.

789
00:27:52,600 --> 00:27:55,080
It will accept drift unless you force it not to.

790
00:27:55,080 --> 00:27:57,640
Files arrive with slightly different column types.

791
00:27:57,640 --> 00:28:01,000
New columns appear because the source system team just added a flag.

792
00:28:01,000 --> 00:28:03,160
Strings show up where integers used to be.

793
00:28:03,160 --> 00:28:05,400
Time stamps, move formats, and everyone shrugs

794
00:28:05,400 --> 00:28:08,280
because the notebook still runs after one more cost.

795
00:28:08,280 --> 00:28:10,520
Then the organization does the next predictable thing.

796
00:28:10,520 --> 00:28:13,000
It mirrors that lake house shape into a warehouse

797
00:28:13,000 --> 00:28:14,680
or builds a warehouse on top of it,

798
00:28:14,680 --> 00:28:17,000
expecting the warehouse to provide structure.

799
00:28:17,000 --> 00:28:20,040
But the warehouse can't enforce structure you never defined.

800
00:28:20,040 --> 00:28:23,160
If the ingestion pipeline treats whatever showed up as acceptable,

801
00:28:23,160 --> 00:28:25,480
the warehouse layer becomes a reflection of ambiguity

802
00:28:25,480 --> 00:28:26,920
not a rejection boundary.

803
00:28:26,920 --> 00:28:28,920
The schema looks cleaner because it's s-quall

804
00:28:28,920 --> 00:28:31,080
therefore leadership assumes it's controlled.

805
00:28:31,080 --> 00:28:33,160
Meanwhile, the real behavior is that the warehouse

806
00:28:33,160 --> 00:28:35,560
is normalizing drift as if it were a feature.

807
00:28:35,560 --> 00:28:38,840
And once that happens, the semantic layer becomes a patch bay.

808
00:28:38,840 --> 00:28:41,000
Analytics engineers start solving data problems

809
00:28:41,000 --> 00:28:42,440
with DAX and Power Query,

810
00:28:42,440 --> 00:28:43,960
because it's the only place they can move

811
00:28:43,960 --> 00:28:45,560
without waiting on an upstream fix.

812
00:28:45,560 --> 00:28:47,000
Measures get more complex.

813
00:28:47,000 --> 00:28:49,560
Calculated columns appear to reconcile type mismatches.

814
00:28:49,560 --> 00:28:50,840
Relationships get weird.

815
00:28:50,840 --> 00:28:52,760
Fix it in the model becomes the operating model

816
00:28:52,760 --> 00:28:54,520
because it's fast and politically safe.

817
00:28:54,520 --> 00:28:56,280
But it's also where truth fragments

818
00:28:56,280 --> 00:28:58,680
you end up with multiple correct KPIs.

819
00:28:58,680 --> 00:29:00,680
Finance has one definition in a report.

820
00:29:00,680 --> 00:29:03,080
Sales has another in a different semantic model.

821
00:29:03,080 --> 00:29:05,640
And the same metric name starts meaning different filters

822
00:29:05,640 --> 00:29:06,520
and different grain.

823
00:29:06,520 --> 00:29:08,440
Everyone's dashboard refreshes on time.

824
00:29:08,440 --> 00:29:10,120
Everyone's wrong in a different way.

825
00:29:10,120 --> 00:29:11,640
Co-pilot accelerates this collapse

826
00:29:11,640 --> 00:29:13,480
because it rewards plausibility.

827
00:29:13,480 --> 00:29:15,240
It will happily generate transformations

828
00:29:15,240 --> 00:29:17,720
and KPI logic that compile and render,

829
00:29:17,720 --> 00:29:19,720
even if they encode a hidden assumption

830
00:29:19,720 --> 00:29:23,400
about grain, keys, deduplication, or later-riving facts.

831
00:29:23,400 --> 00:29:25,480
It doesn't ask you what invariant you're enforcing.

832
00:29:25,480 --> 00:29:27,160
It asks you what output you want.

833
00:29:27,160 --> 00:29:29,400
So the deeper principle for this case is simple.

834
00:29:29,400 --> 00:29:30,920
The earlier you enforce shape,

835
00:29:30,920 --> 00:29:33,880
the fewer semantic patch jobs you fund downstream.

836
00:29:33,880 --> 00:29:36,600
Task flows and medallion visuals don't create contracts.

837
00:29:36,600 --> 00:29:37,880
They create diagrams.

838
00:29:37,880 --> 00:29:39,560
If your bronze to silver to gold layers

839
00:29:39,560 --> 00:29:42,280
aren't backed by schema enforcement and validation gates,

840
00:29:42,280 --> 00:29:43,560
you don't have architecture.

841
00:29:43,560 --> 00:29:45,480
You have a faster way to ship ambiguity.

842
00:29:45,480 --> 00:29:49,320
Case 2Fix enforced schemas, validation gates, quarantine tables.

843
00:29:49,320 --> 00:29:51,480
The fix is not get better at modeling.

844
00:29:51,480 --> 00:29:53,240
The fix is to put an enforcement boundary

845
00:29:53,240 --> 00:29:54,760
where ambiguity enters the estate

846
00:29:54,760 --> 00:29:56,360
and then refused to negotiate with it.

847
00:29:56,360 --> 00:29:59,320
In fabric that boundary is the lake house to warehouse seam.

848
00:29:59,320 --> 00:30:01,560
Treat the lake house as an intake zone.

849
00:30:01,560 --> 00:30:05,080
Fast landing, cheap iteration, messy reality.

850
00:30:05,080 --> 00:30:07,560
Treat the warehouse as the contract zone.

851
00:30:07,560 --> 00:30:11,080
Shaped, typed, keyed, and intentionally consumable.

852
00:30:11,880 --> 00:30:14,040
The warehouse isn't another place to query.

853
00:30:14,040 --> 00:30:15,880
It's the point where the organization

854
00:30:15,880 --> 00:30:17,880
finally commits to what the data is.

855
00:30:17,880 --> 00:30:19,880
So you start with explicit warehouse schemas,

856
00:30:19,880 --> 00:30:21,240
not debu-en-vibes.

857
00:30:21,240 --> 00:30:24,680
Real schemas that encode domain ownership and consumption intent

858
00:30:24,680 --> 00:30:28,520
sales, finance, HR, serving, whatever matches your estate model.

859
00:30:28,520 --> 00:30:30,840
This matters because schema is how you stop location

860
00:30:30,840 --> 00:30:32,680
from masquerading as ownership.

861
00:30:32,680 --> 00:30:35,640
If the table lives in finance, finance owns the contract.

862
00:30:35,640 --> 00:30:37,480
If it lives in serving, the platform team

863
00:30:37,480 --> 00:30:38,920
owns the consumption interface.

864
00:30:38,920 --> 00:30:40,760
The name becomes a control surface.

865
00:30:40,760 --> 00:30:42,360
Then you define shape deliberately.

866
00:30:42,360 --> 00:30:45,320
Columns, types, nullability, keys, and invariance.

867
00:30:45,320 --> 00:30:47,320
And yes, invariance is a real word here.

868
00:30:47,320 --> 00:30:49,720
It means statements the business believes are always true

869
00:30:49,720 --> 00:30:51,560
and therefore the system must enforce.

870
00:30:51,560 --> 00:30:53,000
An order has an order date.

871
00:30:53,000 --> 00:30:54,760
A customer key cannot be empty.

872
00:30:54,760 --> 00:30:56,600
A currency code is from a known set.

873
00:30:56,600 --> 00:30:58,360
A fact table grain is stable.

874
00:30:58,360 --> 00:31:01,160
And later arriving updates have a declared strategy.

875
00:31:01,160 --> 00:31:02,920
Without those, you don't have a contract.

876
00:31:02,920 --> 00:31:04,920
You have a file that happened to pass today.

877
00:31:04,920 --> 00:31:06,440
Now here's the uncomfortable move.

878
00:31:06,440 --> 00:31:09,000
You stop silently coercing data.

879
00:31:09,000 --> 00:31:12,280
Most lake house pipelines fix drift by casting strings

880
00:31:12,280 --> 00:31:14,520
to ins trimming columns, defaulting missing values

881
00:31:14,520 --> 00:31:15,320
and moving on.

882
00:31:15,320 --> 00:31:16,600
That keeps the pipeline green.

883
00:31:16,600 --> 00:31:19,640
It also turns unknown behavior into accepted behavior.

884
00:31:19,640 --> 00:31:21,720
So instead, you add validation gates at ingestion

885
00:31:21,720 --> 00:31:23,080
and transform boundaries.

886
00:31:23,080 --> 00:31:26,520
And you make failure an operational signal, not an embarrassment.

887
00:31:26,520 --> 00:31:28,520
Validation gates are simple in concept.

888
00:31:28,520 --> 00:31:31,960
Before data crosses into the contract zone, it gets checked.

889
00:31:31,960 --> 00:31:35,480
Schema matches, types match, required fields exist.

890
00:31:35,480 --> 00:31:37,600
Keys behave the way you claim they behave.

891
00:31:37,600 --> 00:31:40,240
Duplicates are handled according to a rule you can explain.

892
00:31:40,240 --> 00:31:41,520
If the checks pass, you load.

893
00:31:41,520 --> 00:31:45,040
If they fail, you do not best effort the data into production

894
00:31:45,040 --> 00:31:47,360
and hope the semantic layer can reconcile it.

895
00:31:47,360 --> 00:31:48,400
You quarantine it.

896
00:31:48,400 --> 00:31:50,640
Quarantine tables are the part most teams avoid

897
00:31:50,640 --> 00:31:52,920
because it feels like admitting imperfection.

898
00:31:52,920 --> 00:31:55,120
But quarantine is how you keep the estate honest.

899
00:31:55,120 --> 00:31:57,440
When violations occur, you land the bad routes

900
00:31:57,440 --> 00:31:59,600
in a quarantine schema with metadata.

901
00:31:59,600 --> 00:32:03,600
Source system, load time, violation type, offending values.

902
00:32:03,600 --> 00:32:06,880
And you publish the count as a metric, not hidden in a notebook cell.

903
00:32:06,880 --> 00:32:08,640
A real metric that leadership can see

904
00:32:08,640 --> 00:32:11,280
if they insist on shipping data without contracts.

905
00:32:11,280 --> 00:32:12,960
This changes behavior fast.

906
00:32:12,960 --> 00:32:16,480
Because now schema drift is no longer a silent downstream argument.

907
00:32:16,480 --> 00:32:18,320
It becomes a visible upstream event.

908
00:32:18,320 --> 00:32:20,320
And once it's visible, you can assign responsibility,

909
00:32:20,320 --> 00:32:23,040
either the source system changed, the ingestion logic failed,

910
00:32:23,040 --> 00:32:25,360
or the contract needs an intentional version bump.

911
00:32:25,360 --> 00:32:26,800
Those are the only three truths.

912
00:32:26,800 --> 00:32:28,000
Everything else is denial.

913
00:32:28,000 --> 00:32:29,600
And you'll notice a second order effect

914
00:32:29,600 --> 00:32:31,600
that everyone likes once it happens.

915
00:32:31,600 --> 00:32:33,360
Downstream logic collapses.

916
00:32:33,360 --> 00:32:34,800
In a good way.

917
00:32:34,800 --> 00:32:36,480
When the warehouse enforces shape,

918
00:32:36,480 --> 00:32:38,880
the semantic model stops being a patch bay.

919
00:32:38,880 --> 00:32:41,120
DAX measures get simpler because they don't have

920
00:32:41,120 --> 00:32:42,960
to guard against type chaos.

921
00:32:42,960 --> 00:32:46,160
Relationships stabilize because keys behave consistently.

922
00:32:46,160 --> 00:32:48,240
Two correct KPIs becomes harder to sustain

923
00:32:48,240 --> 00:32:50,960
because the raw truth is no longer malleable per workspace.

924
00:32:50,960 --> 00:32:52,480
You can still build multiple measures,

925
00:32:52,480 --> 00:32:54,160
but you're doing it on top of a stable,

926
00:32:54,160 --> 00:32:56,080
typed contract, not on top of a shifting,

927
00:32:56,080 --> 00:32:57,440
lake of assumptions.

928
00:32:57,440 --> 00:33:00,000
This is also where co-pilot becomes useful again.

929
00:33:00,000 --> 00:33:02,080
With enforced schemas and clear contracts,

930
00:33:02,080 --> 00:33:04,320
co-pilot can generate transformations and measures

931
00:33:04,320 --> 00:33:05,680
that are constrained by reality.

932
00:33:05,680 --> 00:33:07,040
You've reduced the search space.

933
00:33:07,040 --> 00:33:09,200
You've moved from plausible to bounded.

934
00:33:09,200 --> 00:33:10,240
But that's the whole trick.

935
00:33:10,240 --> 00:33:12,720
Don't ask AI to be your governance model.

936
00:33:12,720 --> 00:33:15,680
Make governance the environment AI operates inside.

937
00:33:15,680 --> 00:33:18,400
And when someone asks why you're slowing things down with gates,

938
00:33:18,400 --> 00:33:19,760
the answer is simple.

939
00:33:19,760 --> 00:33:21,120
You're not slowing delivery.

940
00:33:21,120 --> 00:33:23,120
You're stopping unreviewed ambiguity

941
00:33:23,120 --> 00:33:25,120
from becoming production truth.

942
00:33:25,120 --> 00:33:26,800
Case three setup plus fix.

943
00:33:26,800 --> 00:33:29,600
Workspace only security and the ownership vacuum.

944
00:33:29,600 --> 00:33:32,160
Once cost is predictable and truth is enforced,

945
00:33:32,160 --> 00:33:34,480
the next failure mode is the one nobody wants to talk about

946
00:33:34,480 --> 00:33:36,000
because it isn't a performance graph.

947
00:33:36,000 --> 00:33:37,680
It's an audit question.

948
00:33:37,680 --> 00:33:39,680
The symptom pattern is always the same.

949
00:33:39,680 --> 00:33:41,680
You find service principles with broad access

950
00:33:41,680 --> 00:33:43,600
because the pipeline needed it.

951
00:33:43,600 --> 00:33:45,440
You find analysts who can see raw tables

952
00:33:45,440 --> 00:33:47,360
because they were helping validate.

953
00:33:47,360 --> 00:33:49,920
You find multiple workspaces with the same data set

954
00:33:49,920 --> 00:33:52,960
copied three different ways because someone needed autonomy.

955
00:33:52,960 --> 00:33:56,000
And when the security team asks who can see this table,

956
00:33:56,000 --> 00:33:58,320
the answer is a long pause followed by,

957
00:33:58,320 --> 00:33:59,600
we think only AI cuts.

958
00:33:59,600 --> 00:34:01,040
That's not a security posture.

959
00:34:01,040 --> 00:34:01,920
That's a rumor.

960
00:34:01,920 --> 00:34:03,200
This is where fabrics convenience

961
00:34:03,200 --> 00:34:04,560
becomes architectural erosion

962
00:34:04,560 --> 00:34:06,560
because workspaces feel like security boundaries

963
00:34:06,560 --> 00:34:08,000
but they aren't data boundaries.

964
00:34:08,000 --> 00:34:09,360
They are collaboration containers.

965
00:34:09,360 --> 00:34:11,600
A workspace role answers who can contribute here,

966
00:34:11,600 --> 00:34:13,600
not who can access this column,

967
00:34:13,600 --> 00:34:15,280
not who can join these two tables

968
00:34:15,280 --> 00:34:16,560
and infer something sensitive,

969
00:34:16,560 --> 00:34:18,880
not who can run an expensive query surface

970
00:34:18,880 --> 00:34:20,400
that becomes a side channel.

971
00:34:20,400 --> 00:34:23,360
And if you treat workspace roles as your whole strategy,

972
00:34:23,360 --> 00:34:24,800
you create an ownership vacuum

973
00:34:24,800 --> 00:34:26,480
because now nobody owns the access model

974
00:34:26,480 --> 00:34:27,840
at the data engine layer.

975
00:34:27,840 --> 00:34:29,360
Nobody owns the table level intent.

976
00:34:29,360 --> 00:34:31,280
Nobody owns the deny by default posture.

977
00:34:31,280 --> 00:34:33,280
Everyone assumes the workspace boundary is enough,

978
00:34:33,280 --> 00:34:35,360
therefore nobody builds real boundaries inside it.

979
00:34:35,360 --> 00:34:36,480
The platform works.

980
00:34:36,480 --> 00:34:37,360
The system doesn't.

981
00:34:37,360 --> 00:34:41,440
Copilot makes this worse in a very specific way.

982
00:34:41,440 --> 00:34:44,320
It changes how people discover and interact with data.

983
00:34:44,320 --> 00:34:46,640
In the old world, consumers had to know where to look.

984
00:34:46,640 --> 00:34:48,480
A report, a data set,

985
00:34:48,480 --> 00:34:50,480
maybe a documented SQL endpoint.

986
00:34:50,480 --> 00:34:53,440
In fabric, copilot, and agents make exploration conversational

987
00:34:53,440 --> 00:34:56,800
and cross surface, they surface tables, they suggest joins.

988
00:34:56,800 --> 00:34:58,640
They help people just query it.

989
00:34:58,640 --> 00:35:01,520
And if you gave broad read access because you wanted adoption,

990
00:35:01,520 --> 00:35:03,360
copilot becomes the fastest path

991
00:35:03,360 --> 00:35:06,720
to exploring raw assets you never intended as consumption paths.

992
00:35:06,720 --> 00:35:08,480
It doesn't bypass permissions.

993
00:35:08,480 --> 00:35:10,400
It bypasses your assumptions.

994
00:35:10,400 --> 00:35:13,200
So the root cause isn't copilot exposed data.

995
00:35:13,200 --> 00:35:16,560
The root cause is that you never encoded your consumption surfaces

996
00:35:16,560 --> 00:35:18,960
and you never encoded your intent at the engine layer.

997
00:35:18,960 --> 00:35:22,160
You left a vacuum and the platform filled it with default behavior,

998
00:35:22,160 --> 00:35:24,880
broad access, raw tables as APIs,

999
00:35:24,880 --> 00:35:27,120
and service principles that look like owners.

1000
00:35:27,120 --> 00:35:28,480
Now here's the uncomfortable truth.

1001
00:35:28,480 --> 00:35:30,480
Most fabric estates don't get hacked.

1002
00:35:30,480 --> 00:35:31,760
They get drifted.

1003
00:35:31,760 --> 00:35:34,880
Access expands because every exception feels justified at the time.

1004
00:35:34,880 --> 00:35:37,520
Give the SP owner for now, we'll tighten later.

1005
00:35:37,520 --> 00:35:40,000
Add them as member they need to publish.

1006
00:35:40,000 --> 00:35:41,280
Just let them read the layhouse.

1007
00:35:41,280 --> 00:35:42,640
It's not that sensitive.

1008
00:35:42,640 --> 00:35:44,240
Then later becomes never,

1009
00:35:44,240 --> 00:35:45,920
and the estate becomes unauditable.

1010
00:35:45,920 --> 00:35:47,280
Not because anyone was malicious,

1011
00:35:47,280 --> 00:35:49,840
but because no one was responsible for the intent model.

1012
00:35:49,840 --> 00:35:52,000
So the fix is not more training.

1013
00:35:52,000 --> 00:35:52,960
It's not more labels.

1014
00:35:52,960 --> 00:35:54,560
It's not put it in a wiki.

1015
00:35:54,560 --> 00:35:56,000
You need enforceable boundaries

1016
00:35:56,000 --> 00:35:57,520
that survive convenience.

1017
00:35:57,520 --> 00:36:00,080
First, define schema-based security boundaries

1018
00:36:00,080 --> 00:36:01,520
in the warehouse as the contract zone.

1019
00:36:01,520 --> 00:36:03,920
If you use the previous section correctly,

1020
00:36:03,920 --> 00:36:06,720
you already have real schemas that encode ownership.

1021
00:36:06,720 --> 00:36:09,600
Domain schema, serving schemas, quarantine schemas.

1022
00:36:09,600 --> 00:36:11,360
Now you attach security to that structure.

1023
00:36:11,360 --> 00:36:14,000
You create database roles that map to business intent,

1024
00:36:14,000 --> 00:36:17,440
consumer, steward, engineer, automation, whatever matches your org,

1025
00:36:17,440 --> 00:36:19,360
and you start from deny by default.

1026
00:36:19,360 --> 00:36:21,840
No implicit access because someone is in the workspace.

1027
00:36:21,840 --> 00:36:24,320
Second, you restrict raw table access.

1028
00:36:24,320 --> 00:36:26,880
Tables become internal implementation details.

1029
00:36:26,880 --> 00:36:29,840
Views and procedures become the controlled access parts.

1030
00:36:29,840 --> 00:36:31,840
This is where performance and security align.

1031
00:36:31,840 --> 00:36:33,920
You expose what you intend people to query,

1032
00:36:33,920 --> 00:36:36,800
and you can enforce both column level and row level constraints

1033
00:36:36,800 --> 00:36:37,600
where appropriate.

1034
00:36:37,600 --> 00:36:40,160
More importantly, you can enforce that agents

1035
00:36:40,160 --> 00:36:43,120
and ad hoc queries hit the same surfaces your reports hit.

1036
00:36:43,120 --> 00:36:46,080
One contract, one path, one place to secure and tune.

1037
00:36:46,080 --> 00:36:49,280
Third, you treat service principles as identities with blast radius,

1038
00:36:49,280 --> 00:36:50,880
not as pipeline glue.

1039
00:36:50,880 --> 00:36:52,880
Every SP has an explicit role.

1040
00:36:52,880 --> 00:36:55,440
Scope to the minimum schema and objects it needs.

1041
00:36:55,440 --> 00:36:59,040
You stop granting workspace roles as a substitute for data permissions.

1042
00:36:59,040 --> 00:37:02,720
You also stop mixing can deploy artifacts with can read data.

1043
00:37:02,720 --> 00:37:03,920
Those are different rights.

1044
00:37:03,920 --> 00:37:05,520
When you collapse them into member,

1045
00:37:05,520 --> 00:37:08,320
you create an estate where publishing equals reading

1046
00:37:08,320 --> 00:37:11,200
and reading equals exploring and exploring equals inference.

1047
00:37:11,200 --> 00:37:13,920
Finally, you make auditability a first class artifact,

1048
00:37:13,920 --> 00:37:14,960
not a yearly scramble.

1049
00:37:14,960 --> 00:37:17,600
You should be able to answer three questions without a meeting.

1050
00:37:17,600 --> 00:37:20,560
Who can see it, who can change it, and through what surface.

1051
00:37:20,560 --> 00:37:23,520
If you can't answer those questions from roles, grants,

1052
00:37:23,520 --> 00:37:25,920
and controlled interfaces, you don't have governance.

1053
00:37:25,920 --> 00:37:26,960
You have a story.

1054
00:37:26,960 --> 00:37:28,800
And this is the big outcome of this fix.

1055
00:37:28,800 --> 00:37:32,320
You can become audit-defensible without re-architecting the whole estate.

1056
00:37:32,320 --> 00:37:33,760
You don't need a new platform.

1057
00:37:33,760 --> 00:37:35,200
You need a real intent model,

1058
00:37:35,200 --> 00:37:37,120
enforced inside the data engine,

1059
00:37:37,120 --> 00:37:40,000
with workspaces treated as collaboration, not containment.

1060
00:37:40,000 --> 00:37:42,800
Because in fabric, the workspace is where people click,

1061
00:37:42,800 --> 00:37:44,320
but the warehouse is where you enforce.

1062
00:37:44,320 --> 00:37:47,600
New failure modes in the fabric plus copilot era.

1063
00:37:47,600 --> 00:37:50,240
Now, if those three cases felt familiar, good.

1064
00:37:50,240 --> 00:37:51,200
They're not edge cases.

1065
00:37:51,200 --> 00:37:53,200
They're the new baseline failure modes

1066
00:37:53,200 --> 00:37:56,800
in a fabric estate where speed exists before enforcement.

1067
00:37:56,800 --> 00:37:58,800
And what's different in this era is that failure

1068
00:37:58,800 --> 00:38:00,320
doesn't require a visible break.

1069
00:38:00,320 --> 00:38:01,760
The platform keeps running.

1070
00:38:01,760 --> 00:38:03,360
Refreshes keep finishing.

1071
00:38:03,360 --> 00:38:04,320
Users keep clicking.

1072
00:38:04,320 --> 00:38:05,600
Copilot keeps responding.

1073
00:38:05,600 --> 00:38:08,640
The estate degrades anyway because the failure modes are systemic,

1074
00:38:08,640 --> 00:38:11,360
boundary blur, cost drift, and the ownership vacuum.

1075
00:38:11,360 --> 00:38:13,520
And they don't show up as one big explosion.

1076
00:38:13,520 --> 00:38:16,720
They show up as a slow replacement of intent with convenience.

1077
00:38:16,720 --> 00:38:19,120
Failure mode 1 is boundary blur across layers.

1078
00:38:19,120 --> 00:38:20,880
In fabric, the lake house feeds the warehouse,

1079
00:38:20,880 --> 00:38:22,480
the warehouse feeds the semantic model,

1080
00:38:22,480 --> 00:38:25,440
the semantic model feeds reports and reports feed apps.

1081
00:38:25,440 --> 00:38:28,160
But the boundaries between those layers are mostly social

1082
00:38:28,160 --> 00:38:30,000
unless you make them physical.

1083
00:38:30,000 --> 00:38:33,440
So what happens in real estate is that each layer starts fixing

1084
00:38:33,440 --> 00:38:35,440
what it doesn't like about the layer before it.

1085
00:38:35,440 --> 00:38:37,680
The lake house lands messy data.

1086
00:38:37,680 --> 00:38:40,000
The warehouse normalizes it without rejecting it.

1087
00:38:40,000 --> 00:38:43,520
The semantic model compensates with calculated columns and relationships.

1088
00:38:43,520 --> 00:38:45,920
And the report compensates with DAX and filters.

1089
00:38:45,920 --> 00:38:47,680
Every local fix feels rational,

1090
00:38:47,680 --> 00:38:49,920
but every local fix also creates drift

1091
00:38:49,920 --> 00:38:52,480
because the logic now lives in multiple places

1092
00:38:52,480 --> 00:38:56,000
and nobody can describe the system as one coherent contract.

1093
00:38:56,000 --> 00:38:59,360
And because fabric makes it easy to create a new semantic model

1094
00:38:59,360 --> 00:39:01,040
or a new report in minutes,

1095
00:39:01,040 --> 00:39:03,120
teams fork truth instead of repairing it.

1096
00:39:03,120 --> 00:39:04,720
A temporary model becomes the model.

1097
00:39:04,720 --> 00:39:06,640
A quick report becomes operational.

1098
00:39:06,640 --> 00:39:09,520
Apps become distribution channels for inconsistencies

1099
00:39:09,520 --> 00:39:11,600
and then you get the weirdest kind of outage.

1100
00:39:11,600 --> 00:39:14,240
The business still runs, but nobody trusts the numbers.

1101
00:39:14,240 --> 00:39:15,200
That's boundary blur.

1102
00:39:15,920 --> 00:39:20,080
Failure mode 2 is cost-drift driven by AI-shaped interaction patterns.

1103
00:39:20,080 --> 00:39:22,080
This isn't just bad sequel.

1104
00:39:22,080 --> 00:39:25,120
It's the combination of co-pilot making query authoring cheap

1105
00:39:25,120 --> 00:39:27,840
and fabric making query execution shared.

1106
00:39:27,840 --> 00:39:31,280
When a platform charges you for runtime under concurrency,

1107
00:39:31,280 --> 00:39:34,320
every unbounded scan becomes an incident generator.

1108
00:39:34,320 --> 00:39:37,840
But the trick is that cost drift doesn't necessarily correlate to deployments.

1109
00:39:37,840 --> 00:39:40,160
So traditional change management doesn't catch it.

1110
00:39:40,160 --> 00:39:41,680
You can have perfect CICD

1111
00:39:41,680 --> 00:39:43,760
and still get wrecked at 9.05 AM

1112
00:39:43,760 --> 00:39:46,080
because a well-meaning analyst asked co-pilot a question

1113
00:39:46,080 --> 00:39:47,840
that produced a non-sargable predicate,

1114
00:39:47,840 --> 00:39:50,800
hit a large table and ran alongside refresh concurrency.

1115
00:39:50,800 --> 00:39:52,560
So the incident taxonomy shifts.

1116
00:39:52,560 --> 00:39:54,800
Your first signal becomes capacity pressure,

1117
00:39:54,800 --> 00:39:56,320
not pipeline failure.

1118
00:39:56,320 --> 00:39:58,400
Your debugging artifact becomes the plan,

1119
00:39:58,400 --> 00:39:59,280
not the log.

1120
00:39:59,280 --> 00:40:01,840
Your prevention mechanism becomes query surfaces

1121
00:40:01,840 --> 00:40:04,480
and acceptance criteria, not best practices.

1122
00:40:04,480 --> 00:40:05,760
And if you don't accept that shift,

1123
00:40:05,760 --> 00:40:07,600
you keep treating spend as a billing problem

1124
00:40:07,600 --> 00:40:09,520
when it's actually an architecture problem.

1125
00:40:09,520 --> 00:40:11,680
Failure mode 3 is the ownership vacuum

1126
00:40:11,680 --> 00:40:13,440
that looks like everything is fine.

1127
00:40:13,440 --> 00:40:14,640
This is the most corrosive one

1128
00:40:14,640 --> 00:40:17,200
because it hides behind success metrics, pipelines run,

1129
00:40:17,200 --> 00:40:19,200
dashboards refresh, people deliver,

1130
00:40:19,200 --> 00:40:21,040
but nobody owns semantics end to end.

1131
00:40:21,040 --> 00:40:22,640
Nobody owns the contract boundary,

1132
00:40:22,640 --> 00:40:24,400
nobody owns the access intent model.

1133
00:40:24,400 --> 00:40:26,080
And because fabric centralizes everything

1134
00:40:26,080 --> 00:40:27,200
into a workspace experience,

1135
00:40:27,200 --> 00:40:28,640
the organization starts confusing

1136
00:40:28,640 --> 00:40:31,440
someone has access with someone has responsibility.

1137
00:40:31,440 --> 00:40:32,480
Those are not the same.

1138
00:40:32,480 --> 00:40:33,920
An ownership vacuum forms

1139
00:40:33,920 --> 00:40:36,640
when the system can function without an explicit owner.

1140
00:40:36,640 --> 00:40:37,600
Fabric lets you do that.

1141
00:40:37,600 --> 00:40:40,240
It's a platform designed for speed and collaboration,

1142
00:40:40,240 --> 00:40:41,520
so it will happily keep working

1143
00:40:41,520 --> 00:40:43,120
while your governance model erodes.

1144
00:40:43,120 --> 00:40:44,800
Co-pilot accelerates the erosion

1145
00:40:44,800 --> 00:40:46,480
because it makes it easier for more people

1146
00:40:46,480 --> 00:40:49,520
to create more artifacts, pipelines, notebooks,

1147
00:40:49,520 --> 00:40:51,920
transformations, models and answers.

1148
00:40:51,920 --> 00:40:54,000
More artifacts means more implicit contracts.

1149
00:40:54,000 --> 00:40:55,840
More implicit contracts means more drift.

1150
00:40:55,840 --> 00:40:58,400
And drift without an owner is not a technical issue.

1151
00:40:58,400 --> 00:40:59,760
It's entropy management failure.

1152
00:40:59,760 --> 00:41:03,760
This is why each exception becomes an entropy generator.

1153
00:41:03,760 --> 00:41:05,440
You add one temporary shortcut,

1154
00:41:05,440 --> 00:41:07,200
one just for now workspace role.

1155
00:41:07,200 --> 00:41:08,880
One quick semantic model

1156
00:41:08,880 --> 00:41:10,800
built directly on raw tables.

1157
00:41:10,800 --> 00:41:12,880
One fix in DAX because upstream is slow,

1158
00:41:12,880 --> 00:41:15,200
one copilot generated query saved as a data set

1159
00:41:15,200 --> 00:41:16,480
because it worked.

1160
00:41:16,480 --> 00:41:18,640
Each one is defensible in isolation.

1161
00:41:18,640 --> 00:41:21,600
Together, they form a system where intent can't be proven.

1162
00:41:21,600 --> 00:41:24,000
And the real consequences that you end up with incidents

1163
00:41:24,000 --> 00:41:25,600
that aren't outages,

1164
00:41:25,600 --> 00:41:27,600
cost incidents, correctness incidents,

1165
00:41:27,600 --> 00:41:28,640
access incidents.

1166
00:41:28,640 --> 00:41:30,080
They don't always show up as red lights.

1167
00:41:30,080 --> 00:41:31,920
They show up as executive distrust,

1168
00:41:31,920 --> 00:41:33,280
ordered discomfort,

1169
00:41:33,280 --> 00:41:36,480
and a capacity meter that behaves like a random number generator.

1170
00:41:36,480 --> 00:41:39,040
So if you're inheriting a fabric estate and thinking,

1171
00:41:39,040 --> 00:41:41,040
why does this feel harder than it should be,

1172
00:41:41,040 --> 00:41:41,760
here's the answer.

1173
00:41:41,760 --> 00:41:42,560
The platform works.

1174
00:41:42,560 --> 00:41:43,680
The system doesn't.

1175
00:41:43,680 --> 00:41:45,760
And the only way out is to stop treating governance

1176
00:41:45,760 --> 00:41:48,640
as education and start treating it as enforced design.

1177
00:41:48,640 --> 00:41:50,640
Because fabric doesn't slow you down anymore,

1178
00:41:50,640 --> 00:41:52,720
which means you have to choose where friction belongs.

1179
00:41:52,720 --> 00:41:55,520
The role collapse, what shrunk, what didn't.

1180
00:41:55,520 --> 00:41:57,120
Here's the part people keep getting wrong

1181
00:41:57,120 --> 00:41:58,560
and it's why the hiring conversations

1182
00:41:58,560 --> 00:42:00,560
in fabric estates are weird right now.

1183
00:42:00,560 --> 00:42:03,360
They see copilot generating notebooks and SQL

1184
00:42:03,360 --> 00:42:05,680
and they conclude the data engineer role is shrinking.

1185
00:42:05,680 --> 00:42:06,720
It is.

1186
00:42:06,720 --> 00:42:09,600
But only the parts that were never the job in the first place.

1187
00:42:09,600 --> 00:42:11,840
What shrunk is the visible labor?

1188
00:42:11,840 --> 00:42:13,840
Handwriting the pipeline scaffolding,

1189
00:42:13,840 --> 00:42:15,200
stitching connectors together,

1190
00:42:15,200 --> 00:42:16,080
doing repetitive,

1191
00:42:16,080 --> 00:42:19,280
SQL transforms that look impressive in a commit history,

1192
00:42:19,280 --> 00:42:22,160
and maintaining glue code that exists purely

1193
00:42:22,160 --> 00:42:24,320
because tools used to be separate.

1194
00:42:24,320 --> 00:42:25,760
Fabric collapsed a lot of that work

1195
00:42:25,760 --> 00:42:28,480
into item creation and shared experiences,

1196
00:42:28,480 --> 00:42:30,880
and copilot collapses even more into auto-complete

1197
00:42:30,880 --> 00:42:32,080
with extra steps.

1198
00:42:32,080 --> 00:42:34,480
So yes, the estate can produce artifacts faster,

1199
00:42:34,480 --> 00:42:37,040
but the trap is that teams confuse artifact velocity

1200
00:42:37,040 --> 00:42:38,080
with system integrity.

1201
00:42:38,080 --> 00:42:41,040
In the old world, you had to earn every artifact

1202
00:42:41,040 --> 00:42:42,960
by suffering through provisioned infrastructure,

1203
00:42:42,960 --> 00:42:44,480
separate portals, long-run times,

1204
00:42:44,480 --> 00:42:46,240
and annoying deployment friction.

1205
00:42:46,240 --> 00:42:48,480
That friction acted like attacks on casual changes,

1206
00:42:48,480 --> 00:42:49,760
but slowed bad decisions.

1207
00:42:49,760 --> 00:42:50,960
It also slowed good ones.

1208
00:42:50,960 --> 00:42:54,080
But the net effect was that fewer people touch the system,

1209
00:42:54,080 --> 00:42:55,280
fewer times per day,

1210
00:42:55,280 --> 00:42:58,480
with fewer opportunities to accidentally publish a new truth.

1211
00:42:58,480 --> 00:42:59,760
Fabric removes that tax.

1212
00:42:59,760 --> 00:43:03,120
So if your organization's identity for a data engineer

1213
00:43:03,120 --> 00:43:04,720
was the person who makes pipelines,

1214
00:43:04,720 --> 00:43:06,880
of course it feels like the role is disappearing.

1215
00:43:06,880 --> 00:43:09,040
Pipelines are easier, notebooks are easier,

1216
00:43:09,040 --> 00:43:11,040
even the semantic model path is faster.

1217
00:43:11,040 --> 00:43:12,800
The visible assembly work shrinks,

1218
00:43:12,800 --> 00:43:14,160
and now the uncomfortable part.

1219
00:43:14,160 --> 00:43:15,920
The responsibilities that didn't shrink

1220
00:43:15,920 --> 00:43:18,720
are the ones that actually determine whether the estate survives,

1221
00:43:18,720 --> 00:43:22,320
data contracts, schema enforcement, cost predictability,

1222
00:43:22,320 --> 00:43:25,040
security boundaries, and ownership clarity.

1223
00:43:25,040 --> 00:43:26,320
Those did not get automated.

1224
00:43:26,320 --> 00:43:27,840
They got exposed.

1225
00:43:27,840 --> 00:43:29,920
Because once the tool friction goes away,

1226
00:43:29,920 --> 00:43:32,240
the system will ship whatever you allow it to ship.

1227
00:43:32,240 --> 00:43:33,760
If you didn't encode invariants,

1228
00:43:33,760 --> 00:43:35,520
you don't get flexibility.

1229
00:43:35,520 --> 00:43:37,680
That you get drift at a higher refresh frequency.

1230
00:43:38,400 --> 00:43:40,000
If you didn't encode cost intent,

1231
00:43:40,000 --> 00:43:41,520
you don't get self-service,

1232
00:43:41,520 --> 00:43:43,120
you get shared meter contention,

1233
00:43:43,120 --> 00:43:45,280
and a budget that behaves like weather.

1234
00:43:45,280 --> 00:43:46,320
This is the role collapse,

1235
00:43:46,320 --> 00:43:50,000
the platform turned craftsmanship into a smaller percentage of the work,

1236
00:43:50,000 --> 00:43:52,720
and turned governance into the defining bottleneck.

1237
00:43:52,720 --> 00:43:55,440
When tooling gets easier, discipline becomes the bottleneck,

1238
00:43:55,440 --> 00:43:57,840
not because discipline is morally good,

1239
00:43:57,840 --> 00:43:59,360
but because discipline is the only thing

1240
00:43:59,360 --> 00:44:01,280
that produces deterministic outcomes

1241
00:44:01,280 --> 00:44:03,680
in a high-speed low-friction platform.

1242
00:44:03,680 --> 00:44:07,200
This is also why so many teams feel personally attacked by fabric.

1243
00:44:07,200 --> 00:44:09,120
The old world let you be a hero with effort.

1244
00:44:09,120 --> 00:44:10,960
You could brute force a solution with time

1245
00:44:10,960 --> 00:44:12,480
and the friction of the system

1246
00:44:12,480 --> 00:44:14,400
made that effort feel like engineering.

1247
00:44:14,400 --> 00:44:15,920
Fabric removes the heroism layer,

1248
00:44:15,920 --> 00:44:18,560
it rewards design, it punishes improvisation.

1249
00:44:18,560 --> 00:44:19,920
And if you came up in an environment

1250
00:44:19,920 --> 00:44:22,160
where making it work was the primary skill,

1251
00:44:22,160 --> 00:44:24,160
fabric makes you faster at making it work,

1252
00:44:24,160 --> 00:44:26,960
and therefore faster at creating the next incident class.

1253
00:44:26,960 --> 00:44:28,160
So the job moved up the stack

1254
00:44:28,160 --> 00:44:29,440
and the modern data engineer

1255
00:44:29,440 --> 00:44:31,360
isn't primarily a tool operator anymore.

1256
00:44:31,360 --> 00:44:32,960
There are boundary enforcers.

1257
00:44:32,960 --> 00:44:36,000
Contracts, schemers, roles, plans, and gates.

1258
00:44:36,000 --> 00:44:37,760
They decide which paths are allowed,

1259
00:44:37,760 --> 00:44:39,600
which behaviors are acceptable,

1260
00:44:39,600 --> 00:44:41,280
and which quick fixes are rejected

1261
00:44:41,280 --> 00:44:42,880
because they generate long-term drift.

1262
00:44:42,880 --> 00:44:46,000
Analytics engineers feel the same squeeze from the other side.

1263
00:44:46,000 --> 00:44:47,440
If logic lives in five tools,

1264
00:44:47,440 --> 00:44:48,960
Lakehouse transforms, warehouse views,

1265
00:44:48,960 --> 00:44:51,200
semantic model measures, report-level filters,

1266
00:44:51,200 --> 00:44:52,400
and app-level business rules,

1267
00:44:52,400 --> 00:44:54,320
then the platform didn't give you agility.

1268
00:44:54,320 --> 00:44:56,720
It gave you five places for truth to fragment.

1269
00:44:56,720 --> 00:44:59,040
Platform owners also get a new kind of accountability.

1270
00:44:59,040 --> 00:45:00,160
You can't hide behind,

1271
00:45:00,160 --> 00:45:03,200
we size the cluster wrong, or we need more nodes.

1272
00:45:03,200 --> 00:45:05,120
In fabric, if plans are unstable,

1273
00:45:05,120 --> 00:45:06,160
cost is undefined,

1274
00:45:06,160 --> 00:45:08,560
and you will be the person asked why the meter spikes.

1275
00:45:08,560 --> 00:45:09,600
Not because it's your fault,

1276
00:45:09,600 --> 00:45:12,080
but because the platform centralised the blast radius

1277
00:45:12,080 --> 00:45:13,920
into one capacity envelope.

1278
00:45:13,920 --> 00:45:15,760
An architect's inheriting fabric estates

1279
00:45:15,760 --> 00:45:17,520
have the worst version of this.

1280
00:45:17,520 --> 00:45:20,080
They inherit a pile of artefacts that all work

1281
00:45:20,080 --> 00:45:22,400
with no explicit boundaries, no clear owners,

1282
00:45:22,400 --> 00:45:23,680
and a history of exceptions

1283
00:45:23,680 --> 00:45:25,600
that became the actual operating model.

1284
00:45:25,600 --> 00:45:27,040
They're asked to make it reliable

1285
00:45:27,040 --> 00:45:28,560
without slowing the business down,

1286
00:45:28,560 --> 00:45:29,760
which is corporate code for,

1287
00:45:29,760 --> 00:45:31,760
please enforce discipline without creating conflict.

1288
00:45:31,760 --> 00:45:32,560
But that's the shift.

1289
00:45:32,560 --> 00:45:33,680
The role didn't get smaller.

1290
00:45:33,680 --> 00:45:35,760
The role got less visible and more consequential.

1291
00:45:35,760 --> 00:45:37,680
Fabric didn't eliminate data engineering.

1292
00:45:37,680 --> 00:45:39,920
It eliminated the parts that used to distract people

1293
00:45:39,920 --> 00:45:40,960
from the real job,

1294
00:45:40,960 --> 00:45:42,560
enforcing intent at scale.

1295
00:45:42,560 --> 00:45:44,560
What the modern data engineer is now?

1296
00:45:44,560 --> 00:45:47,120
Most organisations still describe the modern data engineer

1297
00:45:47,120 --> 00:45:49,120
as a faster version of the old one.

1298
00:45:49,120 --> 00:45:50,720
Same job, new tooling.

1299
00:45:50,720 --> 00:45:51,920
That belief is comfortable.

1300
00:45:51,920 --> 00:45:52,880
And it's also wrong.

1301
00:45:52,880 --> 00:45:54,960
The modern data engineer in a fabric estate

1302
00:45:54,960 --> 00:45:56,240
is not a pipeline writer,

1303
00:45:56,240 --> 00:45:57,360
not a tool operator,

1304
00:45:57,360 --> 00:45:58,880
and not a SQL typist.

1305
00:45:58,880 --> 00:46:00,000
Those activities still exist,

1306
00:46:00,000 --> 00:46:01,920
but they are not the centre of gravity anymore.

1307
00:46:02,640 --> 00:46:04,480
Fabric and co-pilot shoved the work

1308
00:46:04,480 --> 00:46:07,200
up the stack into a place most teams avoided.

1309
00:46:07,200 --> 00:46:08,480
Explosive intent.

1310
00:46:08,480 --> 00:46:10,480
And this is the part that makes people uncomfortable,

1311
00:46:10,480 --> 00:46:12,880
because intent can't be improvised.

1312
00:46:12,880 --> 00:46:15,040
It has to be designed, enforced, and defended.

1313
00:46:15,040 --> 00:46:16,080
So what is the job now?

1314
00:46:16,080 --> 00:46:17,600
First, contract designer.

1315
00:46:17,600 --> 00:46:18,960
A contract is not a wiki page,

1316
00:46:18,960 --> 00:46:20,240
and it's not a diagram.

1317
00:46:20,240 --> 00:46:21,680
A contract is an enforcement boundary

1318
00:46:21,680 --> 00:46:23,840
that makes ambiguity expensive to ship.

1319
00:46:23,840 --> 00:46:26,000
It defines grain, keys, types,

1320
00:46:26,000 --> 00:46:28,160
nullability, freshness expectations,

1321
00:46:28,160 --> 00:46:29,280
and failure behaviour.

1322
00:46:29,280 --> 00:46:31,360
It encodes what the business thinks is true

1323
00:46:31,360 --> 00:46:33,040
into something the platform can enforce

1324
00:46:33,040 --> 00:46:34,720
without asking permission every time.

1325
00:46:34,720 --> 00:46:37,280
When you skip this, the estate will still move fast.

1326
00:46:37,280 --> 00:46:39,200
It will just move fast toward drift.

1327
00:46:39,200 --> 00:46:40,960
Second, boundary enforcer.

1328
00:46:40,960 --> 00:46:43,280
This is the skill that separates

1329
00:46:43,280 --> 00:46:46,640
we adopted fabric from we operate fabric.

1330
00:46:46,640 --> 00:46:48,560
Boundaries in fabric are not the marketing ones.

1331
00:46:48,560 --> 00:46:50,320
They're not lake house versus warehouse

1332
00:46:50,320 --> 00:46:51,360
as a product menu.

1333
00:46:51,360 --> 00:46:53,120
Their architectural seems.

1334
00:46:53,120 --> 00:46:54,320
Intake versus contract,

1335
00:46:54,320 --> 00:46:55,440
contract versus consumption,

1336
00:46:55,440 --> 00:46:57,120
consumption versus presentation.

1337
00:46:57,120 --> 00:46:59,440
The modern data engineer makes those seams physical

1338
00:46:59,440 --> 00:47:02,240
with views, procedures, schemers, and controlled surfaces.

1339
00:47:02,240 --> 00:47:04,480
And they also make those seams social

1340
00:47:04,480 --> 00:47:06,160
by refusing to accept quick fixes

1341
00:47:06,160 --> 00:47:07,440
that bypass the boundary.

1342
00:47:07,440 --> 00:47:11,040
That refusal matters more than any new feature announcement.

1343
00:47:11,040 --> 00:47:12,160
Third, cost governor.

1344
00:47:12,160 --> 00:47:15,680
If you're uncomfortable calling cost a governance problem,

1345
00:47:15,680 --> 00:47:16,960
fabric will fix that for you.

1346
00:47:16,960 --> 00:47:18,880
In fabric, cost is not finances problem.

1347
00:47:18,880 --> 00:47:21,440
Cost is an execution property of your decisions.

1348
00:47:21,440 --> 00:47:23,120
Query surfaces, concurrency,

1349
00:47:23,120 --> 00:47:24,480
refresh design, and the difference

1350
00:47:24,480 --> 00:47:26,960
between bounded access and raw exploration.

1351
00:47:26,960 --> 00:47:29,200
The modern data engineer treats execution plans

1352
00:47:29,200 --> 00:47:30,640
like policy artifacts.

1353
00:47:30,640 --> 00:47:32,640
They enforce sagability expectations,

1354
00:47:32,640 --> 00:47:34,880
bounded predicates, and predictable query shapes

1355
00:47:34,880 --> 00:47:35,760
for critical parts,

1356
00:47:35,760 --> 00:47:37,520
not because they enjoy gatekeeping

1357
00:47:37,520 --> 00:47:40,640
because deterministic spend requires deterministic behavior.

1358
00:47:40,640 --> 00:47:42,640
Fourth, failure mode anticipator.

1359
00:47:42,640 --> 00:47:46,320
Old school data engineering treated failure as outages.

1360
00:47:46,320 --> 00:47:49,280
A job failed, a pipeline stopped, a connector broke.

1361
00:47:49,280 --> 00:47:52,720
In fabric, the more dangerous failures are the quiet ones.

1362
00:47:52,720 --> 00:47:54,880
It refreshed, but it drifted.

1363
00:47:54,880 --> 00:47:56,960
It answered, but it exposed, it worked,

1364
00:47:56,960 --> 00:47:58,560
but it burned the capacity meter.

1365
00:47:58,960 --> 00:48:01,360
The modern data engineer thinks in incident classes,

1366
00:48:01,360 --> 00:48:03,200
cost incidents, correctness incidents,

1367
00:48:03,200 --> 00:48:05,600
access incidents, and they design for detection.

1368
00:48:05,600 --> 00:48:07,200
Plans, scans versus returns,

1369
00:48:07,200 --> 00:48:08,800
violations, quarantines, lineage,

1370
00:48:08,800 --> 00:48:11,120
and auditability aren't optional extras.

1371
00:48:11,120 --> 00:48:13,360
They are the sensors that tell you the estate is decaying

1372
00:48:13,360 --> 00:48:15,040
before the business notices.

1373
00:48:15,040 --> 00:48:16,480
That distinction matters

1374
00:48:16,480 --> 00:48:18,640
because you can't manage what you can't observe

1375
00:48:18,640 --> 00:48:21,120
and you can't govern what you can't prove.

1376
00:48:21,120 --> 00:48:22,800
Now tie this back to the T-CeCle mindset

1377
00:48:22,800 --> 00:48:25,120
because this is where people underestimate what's happening.

1378
00:48:25,120 --> 00:48:26,800
Most teams treat T-CeCle,

1379
00:48:26,800 --> 00:48:30,160
schemas, rolls, and plans as implementation details.

1380
00:48:30,160 --> 00:48:32,560
In a fabric estate, those are control surfaces.

1381
00:48:32,560 --> 00:48:33,840
A schema is not a folder.

1382
00:48:33,840 --> 00:48:34,960
It's an ownership boundary.

1383
00:48:34,960 --> 00:48:36,240
A role is not a convenience.

1384
00:48:36,240 --> 00:48:37,520
It's an intent declaration.

1385
00:48:37,520 --> 00:48:38,720
A view is not a shortcut.

1386
00:48:38,720 --> 00:48:39,760
It's a contract wrapper.

1387
00:48:39,760 --> 00:48:42,720
An execution plan is not a performance troubleshooting tool.

1388
00:48:42,720 --> 00:48:44,640
It's the cost policy reality check.

1389
00:48:44,640 --> 00:48:46,320
And once you see those as control surfaces,

1390
00:48:46,320 --> 00:48:48,000
you stop asking co-pilot to be smarter.

1391
00:48:48,000 --> 00:48:49,680
You start making the system stricter.

1392
00:48:49,680 --> 00:48:51,280
That is the only sustainable move.

1393
00:48:51,280 --> 00:48:54,160
Now, because this episode is for multiple audiences,

1394
00:48:54,160 --> 00:48:56,000
the roll shift needs to be explicit

1395
00:48:56,000 --> 00:48:59,360
because each group tends to outsource the hard part to someone else.

1396
00:48:59,360 --> 00:49:02,640
Data engineer, if you don't enforce schema and contracts,

1397
00:49:02,640 --> 00:49:04,240
you're not engineering, you're relaying,

1398
00:49:04,240 --> 00:49:07,600
you're moving ambiguity from source to report at higher speed.

1399
00:49:07,600 --> 00:49:10,320
Analytics engineer, if logic lives in five tools,

1400
00:49:10,320 --> 00:49:12,320
you own the drift, not emotionally.

1401
00:49:12,320 --> 00:49:15,040
Operationally, you can't demand a single source of truth

1402
00:49:15,040 --> 00:49:17,920
while implementing five competing sources of logic.

1403
00:49:17,920 --> 00:49:20,880
Platform owner, if plans are unstable, cost is undefined.

1404
00:49:20,880 --> 00:49:23,120
You don't get to argue, it's just usage

1405
00:49:23,120 --> 00:49:24,800
when the query surface allows the scam.

1406
00:49:25,520 --> 00:49:27,760
Architect, if boundaries aren't explicit,

1407
00:49:27,760 --> 00:49:29,360
integration will decide them for you.

1408
00:49:29,360 --> 00:49:32,240
And integration always picks the path of least resistance,

1409
00:49:32,240 --> 00:49:33,520
not the path of least risk.

1410
00:49:33,520 --> 00:49:37,680
Executive, speed without control always builds you later.

1411
00:49:37,680 --> 00:49:40,720
Sometimes as money, sometimes as trust, sometimes as audit pain,

1412
00:49:40,720 --> 00:49:41,760
always as rework.

1413
00:49:41,760 --> 00:49:44,160
This is the modern identity.

1414
00:49:44,160 --> 00:49:46,240
Enforce intent at scale,

1415
00:49:46,240 --> 00:49:49,200
in a platform that will otherwise enforce entropy for you.

1416
00:49:49,200 --> 00:49:52,640
Operating model for fabric plus co-pilot,

1417
00:49:52,640 --> 00:49:54,400
enforcement, not education.

1418
00:49:54,400 --> 00:49:56,800
So here's the operating model and it is not inspiring.

1419
00:49:56,800 --> 00:49:58,080
It is enforceable.

1420
00:49:58,080 --> 00:50:01,120
First rule, AI drafts, humans approve.

1421
00:50:01,120 --> 00:50:03,600
Generation is typing, reviews engineering.

1422
00:50:03,600 --> 00:50:06,240
You don't merge co-pilot output because it compiled.

1423
00:50:06,240 --> 00:50:08,400
You merge it because it passed acceptance criteria

1424
00:50:08,400 --> 00:50:09,680
you can defend later.

1425
00:50:09,680 --> 00:50:12,720
And that means you define acceptance criteria that aren't vibes.

1426
00:50:12,720 --> 00:50:15,200
Schema checks, constraint checks, execution plans,

1427
00:50:15,200 --> 00:50:16,560
and security intent.

1428
00:50:16,560 --> 00:50:18,320
If it can't be validated mechanically,

1429
00:50:18,320 --> 00:50:20,160
it will be argued socially.

1430
00:50:20,160 --> 00:50:22,640
Social governance collapses under scheduled pressure.

1431
00:50:22,640 --> 00:50:24,080
Mechanical governance survives.

1432
00:50:24,720 --> 00:50:26,960
Second rule, contracts before convenience.

1433
00:50:26,960 --> 00:50:28,640
The lake house is not your truth layer.

1434
00:50:28,640 --> 00:50:29,760
It's your intake layer.

1435
00:50:29,760 --> 00:50:31,680
The warehouse is where you commit to shape.

1436
00:50:31,680 --> 00:50:34,720
That distinction matters because schema on read is a drift engine.

1437
00:50:34,720 --> 00:50:37,440
If you let bronze drift into silver and call it agile,

1438
00:50:37,440 --> 00:50:38,560
you're not shipping fast.

1439
00:50:38,560 --> 00:50:41,040
You're spreading ambiguity across more downstream surfaces

1440
00:50:41,040 --> 00:50:42,640
at a higher refresh rate.

1441
00:50:42,640 --> 00:50:44,240
So you declare the primary boundary,

1442
00:50:44,240 --> 00:50:45,280
lake house to warehouse.

1443
00:50:45,280 --> 00:50:48,240
That boundary gets gates, not documentation, gates.

1444
00:50:48,240 --> 00:50:51,040
Third rule, execution plans are cost policy,

1445
00:50:51,040 --> 00:50:52,320
not an optimization hobby.

1446
00:50:52,320 --> 00:50:55,280
If you want deterministic spend on a shared capacity,

1447
00:50:55,280 --> 00:50:58,000
you treat plans stability like you treat change control.

1448
00:50:58,000 --> 00:51:00,000
Critical parts must have bounded predicates.

1449
00:51:00,000 --> 00:51:01,440
They must have selective filters.

1450
00:51:01,440 --> 00:51:03,360
They must avoid non-saggable predicates.

1451
00:51:03,360 --> 00:51:05,200
They must avoid select star.

1452
00:51:05,200 --> 00:51:07,840
And they must not spill under expected concurrency.

1453
00:51:07,840 --> 00:51:09,840
If they do, you don't monitor it.

1454
00:51:09,840 --> 00:51:11,360
You redesign the surface.

1455
00:51:11,360 --> 00:51:13,680
Fourth rule, views and procedures over raw tables.

1456
00:51:13,680 --> 00:51:16,080
Always, this is the consumption API.

1457
00:51:16,080 --> 00:51:19,040
It protects cost, correctness, and security in one move.

1458
00:51:19,040 --> 00:51:22,080
Views give you stable contracts even when internal change.

1459
00:51:22,080 --> 00:51:23,840
Procedures give you parametrization,

1460
00:51:23,840 --> 00:51:26,400
so consumers don't invent their own query shapes.

1461
00:51:26,400 --> 00:51:28,640
And because everything points at the same surface,

1462
00:51:28,640 --> 00:51:31,200
you have one place to tune, one place to secure,

1463
00:51:31,200 --> 00:51:32,720
and one place to audit.

1464
00:51:32,720 --> 00:51:34,720
If you let raw tables be querable by default,

1465
00:51:34,720 --> 00:51:37,440
you have accepted uncontrolled query shapes.

1466
00:51:37,440 --> 00:51:39,680
You've also accepted schema drift as a breaking change

1467
00:51:39,680 --> 00:51:40,960
you'll discover downstream.

1468
00:51:40,960 --> 00:51:41,920
That's not self-service.

1469
00:51:41,920 --> 00:51:43,840
That's unmanaged blast radius.

1470
00:51:43,840 --> 00:51:47,520
Fifth rule, CIR, RCD gates for schema and logic changes.

1471
00:51:47,520 --> 00:51:49,920
Fabric is fast enough that drift accumulates faster

1472
00:51:49,920 --> 00:51:52,800
than your team's ability to remember why things were designed

1473
00:51:52,800 --> 00:51:53,600
the way they were.

1474
00:51:53,600 --> 00:51:55,120
That's what entropy looks like.

1475
00:51:55,120 --> 00:51:57,360
Yesterday's exception becomes today's dependency.

1476
00:51:57,360 --> 00:52:01,200
So you put friction, backward, belongs, in promotion,

1477
00:52:01,200 --> 00:52:02,720
not in exploration.

1478
00:52:02,720 --> 00:52:03,920
Exploration can be fast.

1479
00:52:03,920 --> 00:52:05,920
Production has gates.

1480
00:52:05,920 --> 00:52:07,200
Those gates should be boring.

1481
00:52:07,200 --> 00:52:09,920
Schemer diffs require review, contract changes require

1482
00:52:09,920 --> 00:52:12,800
versioning, security changes require explicit intent,

1483
00:52:12,800 --> 00:52:15,520
and critical query parts require plan review.

1484
00:52:15,520 --> 00:52:17,120
Not because you love process,

1485
00:52:17,120 --> 00:52:19,520
because without gates, your estate becomes a pile of

1486
00:52:19,520 --> 00:52:22,400
artifacts that work until the day leadership asks

1487
00:52:22,400 --> 00:52:24,960
why a number changed and nobody can prove anything.

1488
00:52:24,960 --> 00:52:28,080
Sixth rule, every layer assumes decay unless enforced.

1489
00:52:28,080 --> 00:52:30,320
Workspaces drift, permissions drift,

1490
00:52:30,320 --> 00:52:32,720
semantic models drift, definitions drift,

1491
00:52:32,720 --> 00:52:34,560
people leave, context disappears,

1492
00:52:34,560 --> 00:52:36,800
copilot produces plausible artifacts,

1493
00:52:36,800 --> 00:52:38,880
overtime policies drift away from intent.

1494
00:52:38,880 --> 00:52:40,240
That is not a fabric problem.

1495
00:52:40,240 --> 00:52:42,960
That is the default state of any platform at scale.

1496
00:52:42,960 --> 00:52:44,400
So you design for proof.

1497
00:52:44,400 --> 00:52:46,000
Violation counts, quarantine counts,

1498
00:52:46,000 --> 00:52:48,560
plan baselines, lineage visibility, and audit trails.

1499
00:52:48,560 --> 00:52:49,840
You don't need a perfect system.

1500
00:52:49,840 --> 00:52:52,800
You need a system that tells you when it is becoming less true.

1501
00:52:52,800 --> 00:52:55,440
And the strongest line to keep in your head when someone tries to

1502
00:52:55,440 --> 00:52:58,720
negotiate governance into guidance is this.

1503
00:52:58,720 --> 00:53:00,480
Fabric moves data fast.

1504
00:53:00,480 --> 00:53:02,800
Governance decides whether it stays true.

1505
00:53:02,800 --> 00:53:05,440
So the final takeaway is simple, and it's not flattering.

1506
00:53:05,440 --> 00:53:07,440
Fabric didn't simplify data engineering.

1507
00:53:07,440 --> 00:53:08,960
It simplified the mechanics.

1508
00:53:08,960 --> 00:53:11,200
It removed the ceremony that used to slow you down,

1509
00:53:11,200 --> 00:53:13,840
and it removed the padding that used to hide design omissions.

1510
00:53:13,840 --> 00:53:16,400
That's why these failures showed up in fabric first.

1511
00:53:16,400 --> 00:53:17,680
But they aren't fabric specific.

1512
00:53:17,680 --> 00:53:19,840
They are the inevitable failure modes of any platform

1513
00:53:19,840 --> 00:53:22,400
that collapses in gestion, transformation, storage,

1514
00:53:22,400 --> 00:53:26,400
semantics, and consumption into a single fast surface.

1515
00:53:26,400 --> 00:53:28,080
When you can move data at machine speed,

1516
00:53:28,080 --> 00:53:30,320
you can also ship ambiguity at machine speed.

1517
00:53:30,320 --> 00:53:33,280
And once ambiguity ships, it becomes somebody's dashboard,

1518
00:53:33,280 --> 00:53:34,560
somebody's executive summary,

1519
00:53:34,560 --> 00:53:36,000
somebody's single source of truth

1520
00:53:36,000 --> 00:53:38,160
that exists in exactly one workspace

1521
00:53:38,160 --> 00:53:41,040
because it was the fastest place to fix the problem.

1522
00:53:41,040 --> 00:53:42,720
If you're a senior data engineer,

1523
00:53:42,720 --> 00:53:45,040
the job now is to stop building artifacts

1524
00:53:45,040 --> 00:53:46,720
and start enforcing invariants.

1525
00:53:46,720 --> 00:53:48,080
If you're an analytics engineer,

1526
00:53:48,080 --> 00:53:50,960
the job is to stop distributing logic across five tools

1527
00:53:50,960 --> 00:53:52,960
and then acting surprised when truth fragments.

1528
00:53:52,960 --> 00:53:56,400
If you own the platform, the job is to stop treating capacity spikes

1529
00:53:56,400 --> 00:53:59,200
like building weirdness and start treating query surfaces

1530
00:53:59,200 --> 00:54:01,040
and execution plans as policy.

1531
00:54:01,040 --> 00:54:02,720
If you're the inheriting architect,

1532
00:54:02,720 --> 00:54:04,880
the job is to make boundaries explicit

1533
00:54:04,880 --> 00:54:06,880
before integration decides them for you.

1534
00:54:06,880 --> 00:54:09,520
And if you're a leader, the job is to stop buying speed

1535
00:54:09,520 --> 00:54:11,760
and then acting shocked when control erodes.

1536
00:54:11,760 --> 00:54:13,440
Now, if you want a concrete next step,

1537
00:54:13,440 --> 00:54:16,240
here's the real prompt I want you to answer for your own estate.

1538
00:54:16,240 --> 00:54:19,120
What artifact do you trust most when something feels wrong?

1539
00:54:19,120 --> 00:54:21,200
Execution plan, capacity spike trace,

1540
00:54:21,200 --> 00:54:23,360
quarantine count lineage view, audit log,

1541
00:54:23,360 --> 00:54:24,320
because whatever you pick,

1542
00:54:24,320 --> 00:54:26,480
that's what your governance model is actually built on,

1543
00:54:26,480 --> 00:54:27,760
whether you admit it or not.

1544
00:54:27,760 --> 00:54:31,440
Next episode is how to design fabric data contracts

1545
00:54:31,440 --> 00:54:32,960
that survive co-pilot.

1546
00:54:32,960 --> 00:54:33,960
Not a diagram.

1547
00:54:33,960 --> 00:54:35,280
A contract that blocks drift,

1548
00:54:35,280 --> 00:54:36,480
survives workforce churn

1549
00:54:36,480 --> 00:54:38,400
and still lets teams move fast

1550
00:54:38,400 --> 00:54:41,840
without turning your capacity into a random number generator.

1551
00:54:41,840 --> 00:54:43,640
Fabric is a speed multiplier

1552
00:54:43,640 --> 00:54:45,440
and it multiplies your governance debt

1553
00:54:45,440 --> 00:54:48,160
with the same enthusiasm it multiplies your delivery.

1554
00:54:48,160 --> 00:54:50,800
Drop a comment with the single artifact you trust most.

1555
00:54:50,800 --> 00:54:53,200
Execution plan, capacity metrics,

1556
00:54:53,200 --> 00:54:55,040
violation count or lineage,

1557
00:54:55,040 --> 00:54:57,280
and tell me which failure mode hit you hardest,

1558
00:54:57,280 --> 00:54:59,760
cost, contracts or security.

1559
00:54:59,760 --> 00:55:01,760
Subscribe for the data contract episode.