Microsoft Fabric Data Factory - Simply Explained

1
00:00:00,000 --> 00:00:02,360
What exactly is Microsoft Fabric Data Factory?

2
00:00:02,360 --> 00:00:04,200
Is it just Azure Data Factory with a new name

3
00:00:04,200 --> 00:00:05,640
or something completely different?

4
00:00:05,640 --> 00:00:08,220
If you've heard the term and wondered what it actually means,

5
00:00:08,220 --> 00:00:09,360
you're not alone.

6
00:00:09,360 --> 00:00:11,760
By the end of this episode, you'll know what it is,

7
00:00:11,760 --> 00:00:13,960
how it's different from the classic ADF

8
00:00:13,960 --> 00:00:15,960
and the core pieces you actually need to understand.

9
00:00:15,960 --> 00:00:17,880
If you're dealing with data movement in Fabric,

10
00:00:17,880 --> 00:00:19,880
this is the engine that makes it happen.

11
00:00:19,880 --> 00:00:21,560
And it's simpler than you think.

12
00:00:21,560 --> 00:00:23,280
We'll look at four key building blocks

13
00:00:23,280 --> 00:00:24,720
and see how they fit together.

14
00:00:24,720 --> 00:00:27,520
But first, let's define what we're talking about.

15
00:00:27,520 --> 00:00:28,680
Here's the simplest definition.

16
00:00:28,680 --> 00:00:31,640
Fabric Data Factory is the engine inside Microsoft Fabric

17
00:00:31,640 --> 00:00:33,360
that moves and integrates your data.

18
00:00:33,360 --> 00:00:35,760
Think of it as the replacement for Azure Data Factory

19
00:00:35,760 --> 00:00:39,160
and Synapse Pipelines, all in a single SAS platform.

20
00:00:39,160 --> 00:00:40,880
That's the first thing to understand.

21
00:00:40,880 --> 00:00:42,920
It's not a separate service you manage.

22
00:00:42,920 --> 00:00:45,520
It's built right into Fabric connected to one lake.

23
00:00:45,520 --> 00:00:46,800
Let's use an analogy.

24
00:00:46,800 --> 00:00:49,880
If Fabric is an office building, Data Factory is the mail room

25
00:00:49,880 --> 00:00:52,240
and the conveyor belts that move documents between floors.

26
00:00:52,240 --> 00:00:54,240
You don't think about the mail room every day.

27
00:00:54,240 --> 00:00:57,200
But without it, nothing gets from point A to point B.

28
00:00:57,200 --> 00:01:00,960
Same here, Data Factory handles the movement behind the scenes.

29
00:01:00,960 --> 00:01:02,480
The big difference from the old world,

30
00:01:02,480 --> 00:01:04,280
no extra storage configuration needed.

31
00:01:04,280 --> 00:01:05,960
Your data lands right in one lake.

32
00:01:05,960 --> 00:01:08,600
In Azure Data Factory, you had to set up a storage account,

33
00:01:08,600 --> 00:01:11,080
create containers, and configure linked services.

34
00:01:11,080 --> 00:01:12,440
In Fabric, that's all gone.

35
00:01:12,440 --> 00:01:14,840
One lake is already there, so you just point your pipeline at it

36
00:01:14,840 --> 00:01:15,560
and go.

37
00:01:15,560 --> 00:01:16,920
That sounds familiar, right?

38
00:01:16,920 --> 00:01:18,960
So how is this different from the Azure Data Factory

39
00:01:18,960 --> 00:01:20,080
you already know?

40
00:01:20,080 --> 00:01:22,360
How it's different from Azure Data Factory?

41
00:01:22,360 --> 00:01:25,240
A lot of people think Fabric Data Factory is just Azure Data

42
00:01:25,240 --> 00:01:26,880
Factory with a new label, but it's not.

43
00:01:26,880 --> 00:01:28,320
Let's clear up a few myths.

44
00:01:28,320 --> 00:01:29,680
Pricing is where things really change.

45
00:01:29,680 --> 00:01:32,720
ADF charges per activity run, so every copy activity

46
00:01:32,720 --> 00:01:36,000
stored procedure and pipeline execution adds to your bill.

47
00:01:36,000 --> 00:01:39,000
Fabric Data Factory uses capacity-based pricing instead.

48
00:01:39,000 --> 00:01:43,400
You buy a Fabric capacity, F2, F4, F16, whatever,

49
00:01:43,400 --> 00:01:45,680
and all your workloads share that compute pool.

50
00:01:45,680 --> 00:01:48,440
Data Factory Power BI, Lakehouse, the whole platform.

51
00:01:48,440 --> 00:01:50,440
If you already have Fabric capacity,

52
00:01:50,440 --> 00:01:52,640
running pipelines costs very little extra.

53
00:01:52,640 --> 00:01:55,040
If you don't, you need to weigh the total Fabric cost

54
00:01:55,040 --> 00:01:56,640
against your current ADF bill.

55
00:01:56,640 --> 00:01:58,040
Connections are simpler too.

56
00:01:58,040 --> 00:02:00,520
In ADF, you had to create a linked service for every source

57
00:02:00,520 --> 00:02:02,880
and destination, then data sets on top of that,

58
00:02:02,880 --> 00:02:04,880
then configure the integration run time.

59
00:02:04,880 --> 00:02:07,200
With Fabric, connections are defined in line

60
00:02:07,200 --> 00:02:08,360
and one lake is automatic.

61
00:02:08,360 --> 00:02:10,320
You pick your source, pick your destination,

62
00:02:10,320 --> 00:02:13,640
and the connection is set up on the spot, much less clicking.

63
00:02:13,640 --> 00:02:15,560
There are also new built-in activities.

64
00:02:15,560 --> 00:02:17,720
You can send emails or teams messages directly

65
00:02:17,720 --> 00:02:19,600
from your pipeline without needing logic apps

66
00:02:19,600 --> 00:02:20,800
or custom web hooks.

67
00:02:20,800 --> 00:02:22,280
It's just a drag and drop activity,

68
00:02:22,280 --> 00:02:23,680
and that alone saves a lot of time.

69
00:02:23,680 --> 00:02:25,040
And deployment is better now.

70
00:02:25,040 --> 00:02:26,440
ADF had deployment pipelines,

71
00:02:26,440 --> 00:02:27,600
but they were clunky.

72
00:02:27,600 --> 00:02:29,320
Fabric has built-in deployment pipelines

73
00:02:29,320 --> 00:02:30,840
with workspace-level promotion,

74
00:02:30,840 --> 00:02:33,640
so you can move from dev to test to production in a few clicks.

75
00:02:33,640 --> 00:02:35,000
No manual scripts needed.

76
00:02:35,000 --> 00:02:36,680
Now let's be honest about what's missing.

77
00:02:36,680 --> 00:02:39,480
SSIS support and managed VNet aren't here yet.

78
00:02:39,480 --> 00:02:42,560
If you're heavily invested in SQL server integration services,

79
00:02:42,560 --> 00:02:43,600
you'll need to wait.

80
00:02:43,600 --> 00:02:45,120
For most people, that's fine.

81
00:02:45,120 --> 00:02:47,200
The vast majority of data integration work

82
00:02:47,200 --> 00:02:48,840
doesn't need SSIS.

83
00:02:48,840 --> 00:02:50,320
So here's a simple rule of thumb.

84
00:02:50,320 --> 00:02:52,800
If your ADF bill is under $1,500 a month,

85
00:02:52,800 --> 00:02:53,960
and you're not using Fabric,

86
00:02:53,960 --> 00:02:57,040
stick with ADF because it's cheaper for low volume workloads.

87
00:02:57,040 --> 00:03:00,240
If you're already in Fabric, data factories included, so use it.

88
00:03:00,240 --> 00:03:02,400
And if your ADF costs a higher than that,

89
00:03:02,400 --> 00:03:05,360
it's worth doing the math on a Fabric capacity.

90
00:03:05,360 --> 00:03:07,640
Building block one, data pipelines.

91
00:03:07,640 --> 00:03:10,200
Now let's look at the first building block, data pipelines.

92
00:03:10,200 --> 00:03:11,480
Pipelines are the orchestrators.

93
00:03:11,480 --> 00:03:13,800
They decide what runs, when, and in what order,

94
00:03:13,800 --> 00:03:15,320
and everything else plugs into them.

95
00:03:15,320 --> 00:03:17,080
Think of a pipeline like a recipe.

96
00:03:17,080 --> 00:03:20,040
You have steps, you can branch, and you can loop back.

97
00:03:20,040 --> 00:03:22,600
First, you preheat the oven, then mix the ingredients,

98
00:03:22,600 --> 00:03:25,640
and if the batter looks too dry, you add more milk before baking.

99
00:03:25,640 --> 00:03:27,080
A pipeline works the same way.

100
00:03:27,080 --> 00:03:30,280
First, copy data from a source, then check if the file exists,

101
00:03:30,280 --> 00:03:32,160
and if it does, run a transformation.

102
00:03:32,160 --> 00:03:34,520
If it doesn't send an alert, you control the flow.

103
00:03:34,520 --> 00:03:36,680
The core activities you'll use most often are

104
00:03:36,680 --> 00:03:39,560
copy data for each if condition get metadata and look up.

105
00:03:39,560 --> 00:03:41,960
Copy data moves data from point A to point B,

106
00:03:41,960 --> 00:03:44,840
for each loop through a list, say 10 files in a folder.

107
00:03:44,840 --> 00:03:47,520
If condition checks whether something is true or false,

108
00:03:47,520 --> 00:03:49,840
get metadata asks questions about your data,

109
00:03:49,840 --> 00:03:50,840
how big is this file?

110
00:03:50,840 --> 00:03:52,560
When was it last modified?

111
00:03:52,560 --> 00:03:54,080
Look up reads a configuration file

112
00:03:54,080 --> 00:03:55,840
and brings the values into your pipeline.

113
00:03:55,840 --> 00:03:56,880
That's the starter kit.

114
00:03:56,880 --> 00:03:58,240
The whole thing is low code.

115
00:03:58,240 --> 00:04:00,720
You drag and drop activities onto a canvas, connect them

116
00:04:00,720 --> 00:04:02,880
with arrows, set the properties, and you're done.

117
00:04:02,880 --> 00:04:04,920
No coding needed for most tasks.

118
00:04:04,920 --> 00:04:07,240
It looks like a flow chart, because that's exactly what it is.

119
00:04:07,240 --> 00:04:08,160
Here's a real example.

120
00:04:08,160 --> 00:04:11,040
Imagine pulling sales data from an API every morning.

121
00:04:11,040 --> 00:04:12,960
Your pipeline starts with a copy data activity

122
00:04:12,960 --> 00:04:14,320
that lands it in a lake house.

123
00:04:14,320 --> 00:04:17,520
Then a data flow, Gen 2 step, cleans the data, removes nulls,

124
00:04:17,520 --> 00:04:20,280
fixes data formats, then a notebook enriches it

125
00:04:20,280 --> 00:04:21,520
with customer segments.

126
00:04:21,520 --> 00:04:24,600
Finally, a team's activity sends a message to your team.

127
00:04:24,600 --> 00:04:26,280
Sales data updated.

128
00:04:26,280 --> 00:04:27,960
That's one pipeline doing the work that

129
00:04:27,960 --> 00:04:29,680
used to require multiple tools.

130
00:04:29,680 --> 00:04:31,560
You can also make pipelines reusable.

131
00:04:31,560 --> 00:04:34,080
Say you have the same logic for 10 different data sources.

132
00:04:34,080 --> 00:04:36,000
Instead of building 10 pipelines, you build one

133
00:04:36,000 --> 00:04:39,520
and use parameters, source URL, table name, destination folder,

134
00:04:39,520 --> 00:04:40,520
all as variables.

135
00:04:40,520 --> 00:04:42,280
One pipeline, 10 different jobs.

136
00:04:42,280 --> 00:04:44,560
That's parameterization.

137
00:04:44,560 --> 00:04:46,120
And here's something new.

138
00:04:46,120 --> 00:04:48,440
You can ask co-pilot to write pipeline expressions

139
00:04:48,440 --> 00:04:49,280
in plain English.

140
00:04:49,280 --> 00:04:51,160
Type add two days to this date,

141
00:04:51,160 --> 00:04:52,720
and it generates the expression for you

142
00:04:52,720 --> 00:04:55,080
no more memorizing function syntax or googling

143
00:04:55,080 --> 00:04:56,640
how to format a timestamp.

144
00:04:56,640 --> 00:04:58,240
Just say what you need.

145
00:04:58,240 --> 00:05:00,720
Building block two, data flow, Gen 2.

146
00:05:00,720 --> 00:05:02,880
So what happens when you need to clean up data,

147
00:05:02,880 --> 00:05:04,320
but don't want to write code?

148
00:05:04,320 --> 00:05:06,840
That's the exact problem data flow, Gen 2 solves.

149
00:05:06,840 --> 00:05:09,000
Data flow, Gen 2 is the low code transformation tool

150
00:05:09,000 --> 00:05:10,600
inside fabric data factory.

151
00:05:10,600 --> 00:05:13,480
If you've used Power Query in Excel or Power BI before,

152
00:05:13,480 --> 00:05:14,920
this will feel very familiar.

153
00:05:14,920 --> 00:05:18,240
It has the same interface, the same 300 plus transformations

154
00:05:18,240 --> 00:05:19,320
and the same engine.

155
00:05:19,320 --> 00:05:22,000
Just running in the cloud with a lake house as the destination.

156
00:05:22,000 --> 00:05:22,960
Think of it this way.

157
00:05:22,960 --> 00:05:25,360
If pipelines are the assembly line that moves boxes around,

158
00:05:25,360 --> 00:05:28,600
data flow, Gen 2 is the kitchen where you actually chop, season,

159
00:05:28,600 --> 00:05:30,000
and cook the raw ingredients.

160
00:05:30,000 --> 00:05:31,320
Pipelines handle the logistics.

161
00:05:31,320 --> 00:05:33,120
Data flow, Gen 2 handles the actual work

162
00:05:33,120 --> 00:05:34,560
of making the data useful.

163
00:05:34,560 --> 00:05:35,800
When should you reach for it?

164
00:05:35,800 --> 00:05:38,080
When you need to clean data, change types,

165
00:05:38,080 --> 00:05:41,440
merge tables or filter rows, all without touching spark

166
00:05:41,440 --> 00:05:42,000
or SQL.

167
00:05:42,000 --> 00:05:44,120
If your transformation fits on a whiteboard,

168
00:05:44,120 --> 00:05:46,440
it probably fits in data flow, Gen 2.

169
00:05:46,440 --> 00:05:47,800
Here's a quick example.

170
00:05:47,800 --> 00:05:50,440
You import a CSV file full of customer orders.

171
00:05:50,440 --> 00:05:51,720
There are duplicate rows.

172
00:05:51,720 --> 00:05:53,280
Some dates are in the wrong format.

173
00:05:53,280 --> 00:05:54,440
A few columns have nulls.

174
00:05:54,440 --> 00:05:56,640
You open data flow, Gen 2, click a few buttons

175
00:05:56,640 --> 00:05:58,920
to remove duplicates, change the date column type

176
00:05:58,920 --> 00:06:00,320
and fill in missing values.

177
00:06:00,320 --> 00:06:02,760
Then you join it with a product table from another source.

178
00:06:02,760 --> 00:06:04,160
The whole thing takes 10 minutes.

179
00:06:04,160 --> 00:06:07,680
No code, the output lands in a lake house table, ready for reporting.

180
00:06:07,680 --> 00:06:09,080
You can also see everything visually.

181
00:06:09,080 --> 00:06:11,520
There's a diagram view that shows each step as a node.

182
00:06:11,520 --> 00:06:13,080
Source goes into a filter node.

183
00:06:13,080 --> 00:06:14,920
Filter feeds into a merge node.

184
00:06:14,920 --> 00:06:16,160
Merge feeds into the output.

185
00:06:16,160 --> 00:06:18,040
You can see the whole flow at a glance.

186
00:06:18,040 --> 00:06:19,840
That's really helpful when you're debugging

187
00:06:19,840 --> 00:06:22,200
or when you need to explain the logic to someone else.

188
00:06:22,200 --> 00:06:23,640
Scheduling is flexible too.

189
00:06:23,640 --> 00:06:25,560
You can run data flow, Gen 2 on its own,

190
00:06:25,560 --> 00:06:27,840
set it to refresh every morning at 6am.

191
00:06:27,840 --> 00:06:29,560
Or you can call it from a parent pipeline

192
00:06:29,560 --> 00:06:31,320
as one step in a larger workflow.

193
00:06:31,320 --> 00:06:33,080
Most production setups do the latter.

194
00:06:33,080 --> 00:06:34,560
The pipeline handles the orchestration

195
00:06:34,560 --> 00:06:37,520
and data flow, Gen 2, handles the transformation step inside it.

196
00:06:37,520 --> 00:06:38,760
But here's an honest note.

197
00:06:38,760 --> 00:06:40,400
For really heavy transformations,

198
00:06:40,400 --> 00:06:43,040
think billions of rows, complex window functions

199
00:06:43,040 --> 00:06:44,480
or custom machine learning logic,

200
00:06:44,480 --> 00:06:46,680
you might still use notebooks with Spark.

201
00:06:46,680 --> 00:06:49,160
Data flow Gen 2 isn't built for every scenario,

202
00:06:49,160 --> 00:06:51,720
but for the vast majority of what most teams need,

203
00:06:51,720 --> 00:06:54,320
cleaning, shaping, joining, aggregating,

204
00:06:54,320 --> 00:06:56,320
it covers about 80% of the work.

205
00:06:56,320 --> 00:06:58,520
That's a lot of code you don't have to write.

206
00:06:58,520 --> 00:07:00,480
Building block three, copy jobs.

207
00:07:00,480 --> 00:07:02,200
Now let's talk about moving data.

208
00:07:02,200 --> 00:07:04,120
What if you needed to flow in continuously

209
00:07:04,120 --> 00:07:05,240
from a database?

210
00:07:05,240 --> 00:07:06,720
That's where copy jobs come in.

211
00:07:06,720 --> 00:07:08,800
A copy job is a simplified way to copy data

212
00:07:08,800 --> 00:07:11,200
from a source to one lake with built-in change tracking.

213
00:07:11,200 --> 00:07:14,120
No complex pipeline configuration, no custom scripts.

214
00:07:14,120 --> 00:07:15,680
You pick a source, pick a destination,

215
00:07:15,680 --> 00:07:17,400
and tell it how often you want it to run.

216
00:07:17,400 --> 00:07:18,760
That's really it.

217
00:07:18,760 --> 00:07:21,440
The key feature is change data capture or CDC.

218
00:07:21,440 --> 00:07:24,080
Copy jobs automatically track inserts, updates,

219
00:07:24,080 --> 00:07:25,080
and deletes at the source.

220
00:07:25,080 --> 00:07:27,440
You don't need to build manual watermarking logic

221
00:07:27,440 --> 00:07:29,440
that thing where you save a timestamp and query

222
00:07:29,440 --> 00:07:30,480
for newer records.

223
00:07:30,480 --> 00:07:32,200
Copy jobs handle that behind the scenes.

224
00:07:32,200 --> 00:07:33,960
It just works without any extra effort.

225
00:07:33,960 --> 00:07:34,720
There's more.

226
00:07:34,720 --> 00:07:38,480
Copy jobs support SCD type 2, slowly changing dimensions type 2.

227
00:07:38,480 --> 00:07:40,200
That means it keeps full history.

228
00:07:40,200 --> 00:07:41,720
When a customer changes their address,

229
00:07:41,720 --> 00:07:43,920
the old address stays in the table with an end date.

230
00:07:43,920 --> 00:07:45,680
The new address gets a start date.

231
00:07:45,680 --> 00:07:47,920
You can see exactly how a record changed over time.

232
00:07:47,920 --> 00:07:50,120
That's huge for auditing and historical reporting.

233
00:07:50,120 --> 00:07:51,800
So when do you use copy jobs

234
00:07:51,800 --> 00:07:53,520
versus the standard copy activity?

235
00:07:53,520 --> 00:07:54,280
Here's the rule.

236
00:07:54,280 --> 00:07:57,840
Copy jobs are for set it and forget it continues ingestion.

237
00:07:57,840 --> 00:07:59,600
You want data flowing in automatically,

238
00:07:59,600 --> 00:08:01,720
day after day, with minimal maintenance.

239
00:08:01,720 --> 00:08:04,720
Copy activities are for one-off or scheduled bulk loads

240
00:08:04,720 --> 00:08:06,440
where you need more control over the logic,

241
00:08:06,440 --> 00:08:08,280
the mappings, and the error handling.

242
00:08:08,280 --> 00:08:09,120
Think of it this way.

243
00:08:09,120 --> 00:08:10,760
A copy job is like having a librarian

244
00:08:10,760 --> 00:08:12,720
who automatically spots new books being added

245
00:08:12,720 --> 00:08:13,840
and puts them on the shelf.

246
00:08:13,840 --> 00:08:15,840
You don't need to check the catalog every time

247
00:08:15,840 --> 00:08:17,800
because the librarian handles it.

248
00:08:17,800 --> 00:08:20,320
A copy activity is more like you walking into the library,

249
00:08:20,320 --> 00:08:23,280
picking the books yourself, and deciding exactly where they go.

250
00:08:23,280 --> 00:08:26,360
Both get the job done, but copy jobs save you the effort

251
00:08:26,360 --> 00:08:27,480
for routine work.

252
00:08:27,480 --> 00:08:28,480
Set up is simple.

253
00:08:28,480 --> 00:08:31,240
You pick the source, say a SQL server database.

254
00:08:31,240 --> 00:08:33,600
You pick the destination, a lake house table.

255
00:08:33,600 --> 00:08:34,520
You map the fields.

256
00:08:34,520 --> 00:08:37,320
You tell it if you want incremental refresh or full refresh,

257
00:08:37,320 --> 00:08:38,360
and that covers it.

258
00:08:38,360 --> 00:08:39,520
The job runs on your schedule

259
00:08:39,520 --> 00:08:41,960
and keeps your lake house in sync automatically.

260
00:08:41,960 --> 00:08:43,560
Building block 4 mirroring.

261
00:08:43,560 --> 00:08:45,640
Now what if you need real-time replication?

262
00:08:45,640 --> 00:08:46,840
That's where mirroring comes in.

263
00:08:46,840 --> 00:08:47,920
Here's the simplest definition.

264
00:08:47,920 --> 00:08:50,560
Mirroring is a way to replicate your operational database

265
00:08:50,560 --> 00:08:53,880
into one lake continuously without writing a single line of code.

266
00:08:53,880 --> 00:08:56,320
You set it up once and it runs in the background.

267
00:08:56,320 --> 00:08:58,440
The data stays fresh in near real time.

268
00:08:58,440 --> 00:09:00,800
No pipelines to build, no data flows to maintain.

269
00:09:00,800 --> 00:09:01,760
It just works.

270
00:09:01,760 --> 00:09:03,040
How does it actually work?

271
00:09:03,040 --> 00:09:05,840
Mirroring reads the transaction log of your source database.

272
00:09:05,840 --> 00:09:08,520
Every change, every insert, update, and delete

273
00:09:08,520 --> 00:09:10,880
gets picked up and appears in one lake within minutes.

274
00:09:10,880 --> 00:09:12,240
This isn't batch processing.

275
00:09:12,240 --> 00:09:13,720
It's continuous replication.

276
00:09:13,720 --> 00:09:16,160
The moment something changes in your source database,

277
00:09:16,160 --> 00:09:17,480
it shows up in fabric.

278
00:09:17,480 --> 00:09:18,600
Let's talk about sources.

279
00:09:18,600 --> 00:09:21,080
Right now, you can mirror as your SQL database,

280
00:09:21,080 --> 00:09:23,200
as your Cosmos DB, Snowflake, and others.

281
00:09:23,200 --> 00:09:25,320
More are coming like private network mirroring

282
00:09:25,320 --> 00:09:27,160
for databases behind firewalls,

283
00:09:27,160 --> 00:09:28,960
which was announced at build 2026.

284
00:09:28,960 --> 00:09:29,880
Why would you use this?

285
00:09:29,880 --> 00:09:31,680
Because you get a live copy of your database

286
00:09:31,680 --> 00:09:33,920
in one lake without building any ETL pipelines.

287
00:09:33,920 --> 00:09:35,160
You don't need copy activities.

288
00:09:35,160 --> 00:09:36,600
You don't need change tracking scripts.

289
00:09:36,600 --> 00:09:38,120
You don't need incremental load logic.

290
00:09:38,120 --> 00:09:39,640
The system handles all of that for you.

291
00:09:39,640 --> 00:09:41,160
Here's an analogy to help it stick.

292
00:09:41,160 --> 00:09:43,120
Think of mirroring as having a real-time clone

293
00:09:43,120 --> 00:09:45,120
of your database sitting inside fabric.

294
00:09:45,120 --> 00:09:47,960
Update the source once and the clone updates instantly.

295
00:09:47,960 --> 00:09:49,600
No code required to keep them in sync.

296
00:09:49,600 --> 00:09:50,920
The system does the heavy lifting.

297
00:09:50,920 --> 00:09:52,440
There's also a cost benefit worth mentioning.

298
00:09:52,440 --> 00:09:54,720
Mirroring comes with free compute and storage

299
00:09:54,720 --> 00:09:56,000
up to a certain limit.

300
00:09:56,000 --> 00:09:58,680
You only pay for the fabric capacity you already have.

301
00:09:58,680 --> 00:10:01,200
For many organizations, that makes it cheaper than building

302
00:10:01,200 --> 00:10:03,520
and maintaining custom replication pipelines.

303
00:10:03,520 --> 00:10:04,720
And here's a practical use case.

304
00:10:04,720 --> 00:10:07,480
You can build Power BI reports directly on mirror data

305
00:10:07,480 --> 00:10:09,480
without affecting the source database.

306
00:10:09,480 --> 00:10:11,560
Your operational system keeps running at full speed.

307
00:10:11,560 --> 00:10:13,360
Your analysts get fresh data.

308
00:10:13,360 --> 00:10:14,440
Everyone wins.

309
00:10:14,440 --> 00:10:15,960
How these pieces work together.

310
00:10:15,960 --> 00:10:19,160
So you have pipelines, data flows, copy jobs, and mirroring.

311
00:10:19,160 --> 00:10:20,480
How do they fit together?

312
00:10:20,480 --> 00:10:23,560
Most real-world implementations follow a simple pattern.

313
00:10:23,560 --> 00:10:26,680
Ingest, clean, enrich, serve.

314
00:10:26,680 --> 00:10:28,640
It's called the medallion architecture.

315
00:10:28,640 --> 00:10:30,320
Bronze, silver, gold.

316
00:10:30,320 --> 00:10:31,600
Think of it as three layers.

317
00:10:31,600 --> 00:10:33,720
Bronze holds raw data exactly as it arrived.

318
00:10:33,720 --> 00:10:35,520
Silver is cleaned and de-duplicated.

319
00:10:35,520 --> 00:10:37,640
Gold is enriched and ready for reporting.

320
00:10:37,640 --> 00:10:39,360
Here's what that looks like in practice.

321
00:10:39,360 --> 00:10:41,360
A mirroring or copy job lands raw data

322
00:10:41,360 --> 00:10:42,960
into a bronze lake house table.

323
00:10:42,960 --> 00:10:44,040
That's the ingest step.

324
00:10:44,040 --> 00:10:47,000
Then a data flow gen 2 cleans and transforms it into silver.

325
00:10:47,000 --> 00:10:50,680
Removes nulls, standardizes formats, joins related tables.

326
00:10:50,680 --> 00:10:53,000
Finally, a pipeline orchestrates the whole thing.

327
00:10:53,000 --> 00:10:55,240
It triggers the copy job, waits for it to finish,

328
00:10:55,240 --> 00:10:57,480
runs the data flow, and sends a team's notification

329
00:10:57,480 --> 00:10:58,280
when it's done.

330
00:10:58,280 --> 00:10:59,440
That's the full flow.

331
00:10:59,440 --> 00:11:01,640
The real magic isn't any single tool.

332
00:11:01,640 --> 00:11:03,920
It's how they all talk to each other through one lake.

333
00:11:03,920 --> 00:11:05,800
Your mirror data lives in one lake.

334
00:11:05,800 --> 00:11:07,760
Your data flow output lands in one lake.

335
00:11:07,760 --> 00:11:09,480
Your pipeline reads from one lake.

336
00:11:09,480 --> 00:11:11,480
Everything shares the same storage layer.

337
00:11:11,480 --> 00:11:14,360
You can even share data between workspaces using shortcuts.

338
00:11:14,360 --> 00:11:15,560
No copying needed.

339
00:11:15,560 --> 00:11:17,800
There's a dependency chain worth understanding.

340
00:11:17,800 --> 00:11:21,200
Pipelines can call data flow gen 2 as a step inside the pipeline.

341
00:11:21,200 --> 00:11:24,040
They can also include notebooks, DBT jobs, and even Azure data

342
00:11:24,040 --> 00:11:26,160
factory pipelines if you're migrating gradually.

343
00:11:26,160 --> 00:11:27,800
One pipeline can orchestrate them all.

344
00:11:27,800 --> 00:11:28,480
Here's the thing.

345
00:11:28,480 --> 00:11:30,480
You don't need to stitch together five different services

346
00:11:30,480 --> 00:11:31,160
anymore.

347
00:11:31,160 --> 00:11:33,960
One platform, one storage layer, one set of permissions.

348
00:11:33,960 --> 00:11:35,160
That's the point of fabric.

349
00:11:35,160 --> 00:11:36,400
And it works in the real world.

350
00:11:36,400 --> 00:11:39,400
A UAE based distributor used this exact pattern.

351
00:11:39,400 --> 00:11:41,160
Copy jobs for SAP data.

352
00:11:41,160 --> 00:11:44,200
Data flow gen 2 for cleaning pipelines for orchestration.

353
00:11:44,200 --> 00:11:46,360
They cut their months in close from nine days to two.

354
00:11:46,360 --> 00:11:47,280
That's not a demo.

355
00:11:47,280 --> 00:11:48,320
That's production.

356
00:11:48,320 --> 00:11:49,840
Practical guidance and cost.

357
00:11:49,840 --> 00:11:52,160
So who should actually use fabric data factory?

358
00:11:52,160 --> 00:11:54,480
If you need to move or transform data inside fabric,

359
00:11:54,480 --> 00:11:56,720
this is your tool that covers data engineers building

360
00:11:56,720 --> 00:11:59,200
pipelines, analytics engineers modeling data,

361
00:11:59,200 --> 00:12:01,480
and even business analysts using data flow gen 2

362
00:12:01,480 --> 00:12:03,240
for self-service transformations.

363
00:12:03,240 --> 00:12:04,360
Who should skip it?

364
00:12:04,360 --> 00:12:06,440
If you only need one time file uploads,

365
00:12:06,440 --> 00:12:08,040
start with Lake House directly.

366
00:12:08,040 --> 00:12:09,920
Drag and drop a CSV done.

367
00:12:09,920 --> 00:12:12,920
If you're heavily invested in SSIS, wait for that support.

368
00:12:12,920 --> 00:12:15,040
And if your ADF bill is under 1500 a month

369
00:12:15,040 --> 00:12:18,120
and you're not using fabric, staying put makes sense for now.

370
00:12:18,120 --> 00:12:19,040
Let's talk cost.

371
00:12:19,040 --> 00:12:21,760
Fabric data factory uses capacity units or CUs.

372
00:12:21,760 --> 00:12:24,240
Your pipelines consume CUs based on activity complexity

373
00:12:24,240 --> 00:12:25,160
and data volume.

374
00:12:25,160 --> 00:12:27,080
A simple copy job uses fewer than a flow

375
00:12:27,080 --> 00:12:28,280
with joins and aggregations.

376
00:12:28,280 --> 00:12:29,800
You don't need to track every penny

377
00:12:29,800 --> 00:12:32,200
because the fabric capacity metrics app shows you

378
00:12:32,200 --> 00:12:33,520
what's consuming what.

379
00:12:33,520 --> 00:12:34,720
Here's the rule of thumb.

380
00:12:34,720 --> 00:12:37,360
If you already have fabric capacity like F2 or F4,

381
00:12:37,360 --> 00:12:39,160
running pipelines cost very little extra

382
00:12:39,160 --> 00:12:41,040
since the capacity is already paid for.

383
00:12:41,040 --> 00:12:42,480
If you don't have fabric capacity,

384
00:12:42,480 --> 00:12:45,040
compare the total cost to your current ADF bill.

385
00:12:45,040 --> 00:12:47,760
For most organizations with moderate pipeline volumes,

386
00:12:47,760 --> 00:12:48,960
fabric comes out ahead.

387
00:12:48,960 --> 00:12:51,120
Start small with the free 60 day trial.

388
00:12:51,120 --> 00:12:53,120
Create a workspace, build one pipeline

389
00:12:53,120 --> 00:12:56,520
that copies a CSV to a Lake House and see how it feels.

390
00:12:56,520 --> 00:12:58,480
If you've used ADF or Power Query,

391
00:12:58,480 --> 00:12:59,840
the learning curve is shallow.

392
00:12:59,840 --> 00:13:02,080
If you haven't, this is the simplest entry point

393
00:13:02,080 --> 00:13:03,880
to modern data integration.

394
00:13:03,880 --> 00:13:06,080
Fabric data factory isn't ADF in a new box.

395
00:13:06,080 --> 00:13:07,680
It's a simpler, sass-native engine

396
00:13:07,680 --> 00:13:10,240
that makes moving data feel like a built-in feature,

397
00:13:10,240 --> 00:13:11,520
not a separate project.

398
00:13:11,520 --> 00:13:14,480
Start with one pipeline that copies a file to a Lake House.

399
00:13:14,480 --> 00:13:15,560
That's your first step.

400
00:13:15,560 --> 00:13:18,040
Share this with someone still confused about the difference

401
00:13:18,040 --> 00:13:20,120
and subscribe for more knowledge nuggets.