Microsoft Fabric End-to-End: From Raw Data to Business Decisions with Amit Chandak [MVP]

1
00:00:00,000 --> 00:00:04,360
Welcome everybody to the new episode of the MC65FM podcast.

2
00:00:04,360 --> 00:00:08,400
Today we are going to end to end with Microsoft Fabric.

3
00:00:08,400 --> 00:00:10,960
Not looking at one isolated feature,

4
00:00:10,960 --> 00:00:14,400
but at how the different parts of the platform

5
00:00:14,400 --> 00:00:17,840
come together to create a modern analytics architecture.

6
00:00:17,840 --> 00:00:23,240
How should organization think about leg house versus data warehouse,

7
00:00:23,240 --> 00:00:25,760
where do spiced back belong?

8
00:00:25,760 --> 00:00:29,000
When should we use data flow gen 2?

9
00:00:29,000 --> 00:00:31,760
How do the mental models fit into the architecture?

10
00:00:31,760 --> 00:00:35,200
And ultimately, how do we turn all this technology

11
00:00:35,200 --> 00:00:38,880
into power BI solutions that business user can actually use

12
00:00:38,880 --> 00:00:40,520
and make a better decision?

13
00:00:40,520 --> 00:00:42,800
My guest today is Amit Shaktaq.

14
00:00:42,800 --> 00:00:49,400
Microsoft data, Microsoft data, platform MVP and power BI community

15
00:00:49,400 --> 00:00:53,320
super user, Amit has more than 22 years of experience

16
00:00:53,320 --> 00:00:56,040
in data analytics and currently leads data engineering

17
00:00:56,040 --> 00:01:01,640
and the leg solution at Canerica is experience response

18
00:01:01,640 --> 00:01:05,760
Microsoft fabric power BI SQL databases, Tableau,

19
00:01:05,760 --> 00:01:08,480
in quarter data engineering modeling and visualization.

20
00:01:08,480 --> 00:01:12,240
Today we are going to build the Microsoft fabric analytics

21
00:01:12,240 --> 00:01:15,720
story from the ground up from architecture

22
00:01:15,720 --> 00:01:19,440
to ingesting through transformations, storage,

23
00:01:19,440 --> 00:01:22,760
semantics models, security performance, and final power BI.

24
00:01:22,760 --> 00:01:27,240
Amit, welcome to the MC65 podcast.

25
00:01:27,240 --> 00:01:29,720
Thanks, thanks for inviting me for this podcast.

26
00:01:29,720 --> 00:01:33,080
It's my pleasure to be part of this one, Peter.

27
00:01:33,080 --> 00:01:34,080
Awesome.

28
00:01:34,080 --> 00:01:38,240
Before we get deeply technical,

29
00:01:38,240 --> 00:01:44,120
how did your journey into data and analytics begin?

30
00:01:44,120 --> 00:01:49,800
Yeah, so I joined Oracle in 2003.

31
00:01:49,800 --> 00:01:54,360
I selected Oracle as an act of campus and I became the part

32
00:01:54,360 --> 00:01:56,800
of their BI team.

33
00:01:56,800 --> 00:02:00,240
They were going to one of the transformation at that time.

34
00:02:00,240 --> 00:02:02,600
We were building this tool which is known as the

35
00:02:02,600 --> 00:02:06,040
BI daily business intelligence and that's where I started my journey.

36
00:02:06,040 --> 00:02:12,480
After I left Oracle, I started a company along with one of my colleagues

37
00:02:12,480 --> 00:02:16,200
and I ran a data analytics startup for 10 years.

38
00:02:16,200 --> 00:02:19,800
We created our own tool very similar to Power BI directory.

39
00:02:19,800 --> 00:02:23,480
We used to have the tool completely on the web.

40
00:02:23,480 --> 00:02:25,320
At that time, you know, having a complete web

41
00:02:25,320 --> 00:02:26,440
all thing was a challenge.

42
00:02:26,440 --> 00:02:31,000
So that tool was completely authored on web, designing reports,

43
00:02:31,000 --> 00:02:34,960
creating the semantic models and everything was on the web.

44
00:02:34,960 --> 00:02:36,760
So complete web all thing was there.

45
00:02:36,760 --> 00:02:42,040
And then I done that for 10 years and post that I decided to move on.

46
00:02:42,040 --> 00:02:44,160
That's where I joined my current company, Connecticut.

47
00:02:44,160 --> 00:02:47,200
I also moved to Microsoft technology around the same time.

48
00:02:47,200 --> 00:02:54,400
I became a Power BI community super user around 2020 and then around 2022,

49
00:02:54,400 --> 00:02:59,600
October when I became Microsoft MVP for data platform.

50
00:02:59,600 --> 00:03:00,600
Awesome.

51
00:03:00,600 --> 00:03:03,280
That's, yeah, that's the way it went.

52
00:03:03,280 --> 00:03:05,960
22 years, it's a lot of time.

53
00:03:05,960 --> 00:03:09,560
But yeah, that's direct jump into the fabric part.

54
00:03:09,560 --> 00:03:11,800
So let's start with the big question.

55
00:03:11,800 --> 00:03:16,880
What problem is Microsoft fabric actual trying to dissolve?

56
00:03:16,880 --> 00:03:23,160
See prior to Microsoft Fabric, see, it's not that we are doing data

57
00:03:23,160 --> 00:03:26,000
engineering something new with fabric.

58
00:03:26,000 --> 00:03:29,040
The whatever components fabric has, all those components

59
00:03:29,040 --> 00:03:32,720
were previously available at different places.

60
00:03:32,720 --> 00:03:35,520
So if you look at Azure ecosystem, you could have, you know,

61
00:03:35,520 --> 00:03:38,280
100 plus choices to do what fabric is doing today.

62
00:03:38,280 --> 00:03:41,600
If I look at the larger ecosystem, if I include Azure,

63
00:03:41,600 --> 00:03:48,000
AWS, even Google platform, I might have 1000 plus choices to do my own data engineering work.

64
00:03:48,000 --> 00:03:49,480
But the challenge was bigger.

65
00:03:49,480 --> 00:03:52,040
Now choice of technology was one challenge.

66
00:03:52,040 --> 00:03:55,800
Then other than choice of technology, create a multiple copies of data.

67
00:03:55,800 --> 00:03:59,160
What happens in the data analytics or BIA ecosystem?

68
00:03:59,160 --> 00:04:00,440
We say single source of proof.

69
00:04:00,440 --> 00:04:02,040
We always talk about it.

70
00:04:02,040 --> 00:04:05,120
But our own ecosystem, because we sometime we need data lake,

71
00:04:05,120 --> 00:04:08,840
another time we need data warehouse and then we need powerful tools like power

72
00:04:08,840 --> 00:04:13,040
BIA, which for input would be our self creating copies of data.

73
00:04:13,040 --> 00:04:18,120
And all these tools are working on different technologies is they have different kind of back ends.

74
00:04:18,120 --> 00:04:19,520
Their storage were different.

75
00:04:19,520 --> 00:04:23,000
They they were created at different time frames.

76
00:04:23,000 --> 00:04:27,640
So all of them were, you know, built differently, storing differently.

77
00:04:27,640 --> 00:04:31,280
And they were using different ways compute powers getting compute.

78
00:04:31,280 --> 00:04:35,560
A Microsoft node challenge and I think that's where they start in the center apps where

79
00:04:35,560 --> 00:04:37,520
they wanted to address this one.

80
00:04:37,520 --> 00:04:41,840
A fabric name was even higher where they sorted out the entire stuff.

81
00:04:41,840 --> 00:04:46,600
Whether you have the structure data and structure data, where you wanted to have warehouse,

82
00:04:46,600 --> 00:04:50,760
where you wanted to have power BIA or even real time analytics, they were able to put everything

83
00:04:50,760 --> 00:04:52,240
at one place.

84
00:04:52,240 --> 00:04:56,640
And they starting, you know, with a few foundations like one lake where, you know, your entire

85
00:04:56,640 --> 00:05:00,760
data assets, whether it's a metadata data can decide.

86
00:05:00,760 --> 00:05:05,000
Then they solved the problem of compute by saying, okay, all the workloads in Microsoft

87
00:05:05,000 --> 00:05:07,400
fabric can share one single compute.

88
00:05:07,400 --> 00:05:11,920
Now you can have multiple computers per requirement, but that that is one of the biggest achievement

89
00:05:11,920 --> 00:05:14,600
because think about your running spark workload.

90
00:05:14,600 --> 00:05:17,760
How would you compare that with power BIA or CQVV?

91
00:05:17,760 --> 00:05:18,760
They're all together.

92
00:05:18,760 --> 00:05:24,560
But everything gets converted to compute units and getting chard within the same number

93
00:05:24,560 --> 00:05:26,560
of computers which is available with you.

94
00:05:26,560 --> 00:05:29,200
Let's say paradew per 30 seconds.

95
00:05:29,200 --> 00:05:31,920
Then comes the challenge which is there with the storage.

96
00:05:31,920 --> 00:05:36,200
Now I have storage, which is available for the greater work, but what happens when I

97
00:05:36,200 --> 00:05:39,560
store the data, let's say for lake house, I might prefer some format.

98
00:05:39,560 --> 00:05:43,080
Now where else doesn't understand that format and that's why I was copying data from lake house

99
00:05:43,080 --> 00:05:45,320
to where else when I was doing previous term.

100
00:05:45,320 --> 00:05:48,360
Similarly, power BIA was not comfortable with data warehouse formats.

101
00:05:48,360 --> 00:05:49,760
So it was creating on copy.

102
00:05:49,760 --> 00:05:54,400
Now whenever we store the structured data, whether it's a lake house, whether it's a warehouse,

103
00:05:54,400 --> 00:06:00,560
it all getting stored into delta format, which created that uniform format, which lake

104
00:06:00,560 --> 00:06:05,480
house understand, which spark understand, which T-SQL understand, which even the power

105
00:06:05,480 --> 00:06:08,000
query engine has been modified to understand that.

106
00:06:08,000 --> 00:06:11,960
So it means I am not unnecessary creating copies of data.

107
00:06:11,960 --> 00:06:16,200
Definitely most of us follows the medallion architecture and in that we will have, you know,

108
00:06:16,200 --> 00:06:20,040
bronze, silver and gold, but it doesn't mean that I'm going to create unnecessary copies

109
00:06:20,040 --> 00:06:23,160
of my bronze data and unnecessary copies of my gold data.

110
00:06:23,160 --> 00:06:28,680
So these are some of the challenges which existed while, you know, we have the ecosystems

111
00:06:28,680 --> 00:06:31,640
available for last 15 to 20 years to do data analytics.

112
00:06:31,640 --> 00:06:35,760
But these are some of the challenges which emerges because of the growth of the data, which

113
00:06:35,760 --> 00:06:42,160
is there with us and Microsoft fabric has, you know, went ahead and addressed that.

114
00:06:42,160 --> 00:06:52,960
And how much to do an organization's data practice need to be before a depth thing fabric?

115
00:06:52,960 --> 00:06:58,640
See Microsoft fabric is basically complete end to end solution for your data analytics need.

116
00:06:58,640 --> 00:07:04,760
So even if you are starting for data analytics, Microsoft fabric can be there.

117
00:07:04,760 --> 00:07:08,800
So the only in the chart is needed at the source level that you understand your sources

118
00:07:08,800 --> 00:07:12,960
and you understand what is needed for data analytics.

119
00:07:12,960 --> 00:07:18,200
If you are ready to analyze your data, if your data is arranged, let's say, if your data

120
00:07:18,200 --> 00:07:22,880
is completely unorganized, you are, you know, still working on some of those spreadsheets

121
00:07:22,880 --> 00:07:30,560
a lot, some of the, let's say ERP or CRM implementations are still ongoing or on middle of it, then

122
00:07:30,560 --> 00:07:31,880
it might not be the right time.

123
00:07:31,880 --> 00:07:36,720
But if you are a data mature company in terms of capturing the data and wanted to do

124
00:07:36,720 --> 00:07:40,720
entities or already there with some analytics and wanted to transform it to the model system,

125
00:07:40,720 --> 00:07:44,440
I think in both case you are ready for data and it is on Microsoft fabric.

126
00:07:44,440 --> 00:07:46,480
It provides you everything these days.

127
00:07:46,480 --> 00:07:51,000
If you, if you say I just wanted to do simple, simple, create very simple report using

128
00:07:51,000 --> 00:07:52,600
Powerway, that's also possible.

129
00:07:52,600 --> 00:07:57,040
And you want to follow the complete architecture, you know, bringing data to the lake houses,

130
00:07:57,040 --> 00:07:59,520
transforming it and create warehouse and then wanted to do it.

131
00:07:59,520 --> 00:08:02,320
That is also possible in Microsoft fabric.

132
00:08:02,320 --> 00:08:08,960
What, what did you think, how important is domain oriented architecture in fabric or

133
00:08:08,960 --> 00:08:16,920
or short organizations create one large centralized fabric environment or distribute responsibility

134
00:08:16,920 --> 00:08:19,520
across business domains?

135
00:08:19,520 --> 00:08:25,080
I think we are in an era where we talk about business owning these stuff where we talk about

136
00:08:25,080 --> 00:08:28,080
data mesh architecture where we talk about domains.

137
00:08:28,080 --> 00:08:33,080
So in the modern architecture, why, you know, your IT could be custodian of your data,

138
00:08:33,080 --> 00:08:38,320
but you need to create, you know, multiple domains in which the data is going to decide

139
00:08:38,320 --> 00:08:39,320
finally.

140
00:08:39,320 --> 00:08:43,840
So you can have a master data separately analyzed for the organization level because master

141
00:08:43,840 --> 00:08:48,160
data management is key for all the analysis post that I think it is based on let's say

142
00:08:48,160 --> 00:08:53,160
sales, finance, and purchase can have their own workspaces can they have their own domain

143
00:08:53,160 --> 00:08:54,920
and the data can decide.

144
00:08:54,920 --> 00:08:59,920
If you are a small to medium size organization having a centralized data warehouse, a works

145
00:08:59,920 --> 00:09:05,400
means it can have anything, but if you are a medium to large enterprise or a very large

146
00:09:05,400 --> 00:09:09,760
enterprise, it is always good to have, you know, data mesh architecture where you have

147
00:09:09,760 --> 00:09:14,280
the domains and, you know, the ownership lies with business more than IT.

148
00:09:14,280 --> 00:09:19,600
So it is the combined governance, while the data, entity and the data availability ensure

149
00:09:19,600 --> 00:09:24,920
by IT, but the analysis part of it, distribution part of it and the management part of this is

150
00:09:24,920 --> 00:09:27,640
governed by the business.

151
00:09:27,640 --> 00:09:38,000
And I think the core term when we talk about Microsoft Fabric is it's one leg, what is

152
00:09:38,000 --> 00:09:41,960
one leg in practical terms?

153
00:09:41,960 --> 00:09:44,840
Like today what happens?

154
00:09:44,840 --> 00:09:49,560
Let's talk about our own stuff when we are trying to go online.

155
00:09:49,560 --> 00:09:55,240
One other thing what we do is we either we choose let's say one drive for ourself, we are

156
00:09:55,240 --> 00:10:00,120
trying to put our word file, excel file, everything on the one drive and say, okay, my data is

157
00:10:00,120 --> 00:10:03,880
you know all stored at one place in one one drive.

158
00:10:03,880 --> 00:10:10,320
Now the, now in Microsoft Fabric world, one leg is the one drive whether it is my lakehouse

159
00:10:10,320 --> 00:10:14,400
data warehouse data my metadata my power be a cementing model everything is getting stored

160
00:10:14,400 --> 00:10:16,200
at one place.

161
00:10:16,200 --> 00:10:22,080
And the good thing about this is this is one single virtualize storage for your tenant.

162
00:10:22,080 --> 00:10:27,480
Now think about you kind of have a tenant which is, you know, single tenant which is across

163
00:10:27,480 --> 00:10:28,480
geographies and everything.

164
00:10:28,480 --> 00:10:30,200
You need to take care about that.

165
00:10:30,200 --> 00:10:33,600
And that is where you know, Fabric does it smartly that at the workspace level you can

166
00:10:33,600 --> 00:10:37,680
have those regional aspect taking care of.

167
00:10:37,680 --> 00:10:43,240
So virtualized as one but yet still internally you can have the security aspect or the regional

168
00:10:43,240 --> 00:10:46,120
aspect covered.

169
00:10:46,120 --> 00:10:54,840
And what problem do is the one leg soft compared to organization creating I don't know multiple

170
00:10:54,840 --> 00:10:57,880
independent data data lakes.

171
00:10:57,880 --> 00:11:04,480
See when we say data lake understand inside Microsoft fabric again we can have multiple lakehouses.

172
00:11:04,480 --> 00:11:08,640
And think about it this let's say if I wanted to store it at different different places

173
00:11:08,640 --> 00:11:12,920
typical previous architecture which you talk about let's say multiple lakehouses see when

174
00:11:12,920 --> 00:11:17,000
we store at different places we need different security keys.

175
00:11:17,000 --> 00:11:18,480
We need different management.

176
00:11:18,480 --> 00:11:20,360
We have different costs around it.

177
00:11:20,360 --> 00:11:25,520
Now here what is happening you are being charged for what you are storing maybe it is stored

178
00:11:25,520 --> 00:11:30,360
practically for purpose in different work workspaces and different lake houses these are

179
00:11:30,360 --> 00:11:31,680
just containers folders.

180
00:11:31,680 --> 00:11:36,120
This is like if I go my computer I will not store everything in one drive or one folder I

181
00:11:36,120 --> 00:11:37,840
will have this set of folders.

182
00:11:37,840 --> 00:11:44,080
So Microsoft fabric make it so easy inside your one lake that it is one virtualized disk for

183
00:11:44,080 --> 00:11:46,000
you where you have different different folders.

184
00:11:46,000 --> 00:11:51,480
Now the folder could be lake house or something below that you have the data in that aspect

185
00:11:51,480 --> 00:11:53,040
you are storing it.

186
00:11:53,040 --> 00:11:57,200
So the entire storage at one place you are getting charge at one place now the security is

187
00:11:57,200 --> 00:12:01,080
entire SSO security or your item level security which comes in and play.

188
00:12:01,080 --> 00:12:08,680
So you can secure it easily it is all getting secure using one set of you know way it is not

189
00:12:08,680 --> 00:12:14,520
different experience at okay if I have a data stored in let's say ADLS storage or AWS storage

190
00:12:14,520 --> 00:12:18,780
I need to have different different processes for that and how do I secure and still give

191
00:12:18,780 --> 00:12:23,400
access to everyone that's not a challenge here it is the same common interface to you know

192
00:12:23,400 --> 00:12:28,720
put the data in and secure it for everyone using that the workspace level security or item

193
00:12:28,720 --> 00:12:34,320
level security or inside even item means let's say that is warehouse or lake house or item or

194
00:12:34,320 --> 00:12:38,800
entities inside that further I can go and secure it individual table columns or even the

195
00:12:38,800 --> 00:12:42,000
role of a data.

196
00:12:42,000 --> 00:12:49,080
And how should organization think about ownership when multiple teams consume the same data

197
00:12:49,080 --> 00:12:52,280
through one lake.

198
00:12:52,280 --> 00:12:57,560
See so for the ownership that is why we have these concepts of work spaces where the actual

199
00:12:57,560 --> 00:13:02,320
security process starts so what we typically do is we divide the content into different

200
00:13:02,320 --> 00:13:07,520
different work spaces and from those work spaces for individual items we give access to

201
00:13:07,520 --> 00:13:11,920
the other people who wanted to access like I will have a master data workspace where entire

202
00:13:11,920 --> 00:13:16,600
my master data besides now the same master data would be consumed by say steam it is also

203
00:13:16,600 --> 00:13:20,920
being consumed by finance team so I will give the access the read access to all the people

204
00:13:20,920 --> 00:13:26,400
who are consuming that data to read it and then further they can you know go add and give

205
00:13:26,400 --> 00:13:30,920
the access to the end users and consumers.

206
00:13:30,920 --> 00:13:38,400
Okay and when we have this architecture how important are short cut in this architecture

207
00:13:38,400 --> 00:13:45,720
and when should you use a shortcut instead of a physical copying data.

208
00:13:45,720 --> 00:13:52,400
Okay so shortcut could be a shortcut on external now first way to work because because

209
00:13:52,400 --> 00:13:57,200
it is a shortcut for multiple purposes. So external shortcut is something basically the

210
00:13:57,200 --> 00:14:02,480
data is outside the Microsoft family ecosystem and you can still create a shortcut to it.

211
00:14:02,480 --> 00:14:07,000
Basically you have something on AWS or some somewhere in Google and then you wanted to

212
00:14:07,000 --> 00:14:08,000
bring that.

213
00:14:08,000 --> 00:14:13,600
When you create shortcut the data is typically not getting copied here it is just treated

214
00:14:13,600 --> 00:14:18,760
as the data at that place and then we can when as and when we need it get query definitely

215
00:14:18,760 --> 00:14:21,320
there is a cache mechanism to bring it.

216
00:14:21,320 --> 00:14:26,360
Now the shortcut could be in Azure which typically it is as native storage because everything

217
00:14:26,360 --> 00:14:29,000
else is another unless you are in different region.

218
00:14:29,000 --> 00:14:33,080
Now within Microsoft fabric system ecosystem also we create shortcuts.

219
00:14:33,080 --> 00:14:37,160
We create shortcuts to another lake houses or warehouses and bring the things together.

220
00:14:37,160 --> 00:14:41,200
So what happens then you have this multiple domain architecture and we firmly believe in

221
00:14:41,200 --> 00:14:45,480
not copying the data so what we end up doing is creating shortcuts so finance will have

222
00:14:45,480 --> 00:14:51,280
a shortcut to master data to read the master data also shortcuts provide two more things

223
00:14:51,280 --> 00:14:57,720
you can you know have your security bifurcated because of the shortcuts and one more thing

224
00:14:57,720 --> 00:15:02,880
which shortcut does is basically shortcut uses the capacity of the current workspace.

225
00:15:02,880 --> 00:15:08,040
So it is practically possible that I have a workspace one which is working on capacity

226
00:15:08,040 --> 00:15:13,960
one and workspace to which is working on capacity to and I created the shortcuts in capacity

227
00:15:13,960 --> 00:15:20,080
to workspace then it can use capacity to for that workspace and that is needed for the

228
00:15:20,080 --> 00:15:25,000
power be use case because what would happen the report consumption is something which is

229
00:15:25,000 --> 00:15:30,800
done by you know executives and that is the place where if your data engineering workloads

230
00:15:30,800 --> 00:15:35,800
are working you don't want them to consume your entire capacity and these executives are

231
00:15:35,800 --> 00:15:38,080
seeing the slowness into the report.

232
00:15:38,080 --> 00:15:44,920
So you would definitely want your power be a reports to run without being worried about

233
00:15:44,920 --> 00:15:49,880
what is happening to my data engineering which is continuously running in some organization

234
00:15:49,880 --> 00:15:54,880
but it is reading the same data because it is the same only power be is not copying the data

235
00:15:54,880 --> 00:15:55,880
now.

236
00:15:55,880 --> 00:15:59,880
So when it is not copying so data though I have separated out the workspace the workspace

237
00:15:59,880 --> 00:16:06,360
to is power be a workspace one is data which is lake house and the movement it queries

238
00:16:06,360 --> 00:16:11,280
it goes to the workspace one where my lake house is decided to start using that capacity

239
00:16:11,280 --> 00:16:17,120
but the shortcuts ensures that it uses the workspace to so I create a lake house with shortcuts

240
00:16:17,120 --> 00:16:23,440
in workspace to and now it is accessing by power be I which ensures that now I am using

241
00:16:23,440 --> 00:16:26,960
the capacity of workspace to not the workspace one.

242
00:16:26,960 --> 00:16:33,760
So many objective no copy of data isolation of you know your workspace capacities and even

243
00:16:33,760 --> 00:16:37,880
the security aspect when you try to secure and you don't want to secure some of those things

244
00:16:37,880 --> 00:16:43,080
that can also be handled with shortcuts so there are many uses and it is a really whenever

245
00:16:43,080 --> 00:16:45,640
it possible we should use shortcuts.

246
00:16:45,640 --> 00:16:52,240
Now only thing one thing we should remember when we go cross systems is cross cloud because

247
00:16:52,240 --> 00:16:56,520
in that case at every time you send the query other than caching it may end up querying

248
00:16:56,520 --> 00:17:03,520
the data and sometime the other system it gives out the data it may have a cost so we

249
00:17:03,520 --> 00:17:08,240
can also see what is the cost we are going to incur when we take out the data and that

250
00:17:08,240 --> 00:17:13,360
will decide whether we need to have shortcut or mirroring or copying of data.

251
00:17:13,360 --> 00:17:16,120
Awesome.

252
00:17:16,120 --> 00:17:25,960
That is what one thing in fabric it is I can choose the lake house and I also can choose

253
00:17:25,960 --> 00:17:37,000
the data warehouse when I choose what and why it is the data warehouse still existing there.

254
00:17:37,000 --> 00:17:42,720
Okay so let us start with the term in the images known as our warehouse.

255
00:17:42,720 --> 00:17:47,160
Basically if you call about we talking about data warehouse for almost maybe around 30 years

256
00:17:47,160 --> 00:17:50,600
so there is one in this data which is known as data warehouse which is basically because

257
00:17:50,600 --> 00:17:55,320
single source of thought where typically what all of us say is that it is where our goal

258
00:17:55,320 --> 00:18:00,760
data is at the data which is ready for consumption for bi but when we come to the ecosystem like

259
00:18:00,760 --> 00:18:04,400
Microsoft fabric we have a lake house we have a warehouse.

260
00:18:04,400 --> 00:18:08,840
Now are we talking warehouse as a single source of two no we are talking here warehouse is

261
00:18:08,840 --> 00:18:12,720
one of the storage piece and how the data is going to store.

262
00:18:12,720 --> 00:18:16,840
So let us say lake house can store both structure and structure data warehouse can only store

263
00:18:16,840 --> 00:18:21,360
structure data so one call is decided okay if I have unstructured data files I need lake

264
00:18:21,360 --> 00:18:22,360
house to be there.

265
00:18:22,360 --> 00:18:28,200
Now the second thing is basically the data when stored as structure data both lake house

266
00:18:28,200 --> 00:18:31,160
and warehouse save it as delta format.

267
00:18:31,160 --> 00:18:36,960
The only difference is the lake house delta files are governed by spark you can say and

268
00:18:36,960 --> 00:18:42,280
warehouse delta files are governed by t-SQL so who is the first technology who understand

269
00:18:42,280 --> 00:18:46,300
the data changes so in case of lake house the first technology which understand the data

270
00:18:46,300 --> 00:18:50,480
changes is spark and in case of warehouse the first technology which understand the data

271
00:18:50,480 --> 00:18:57,680
changes is t-SQL so now technology is a choice let us say we are unstructured data now where

272
00:18:57,680 --> 00:19:02,840
whether I would use spark or whether I would use t-SQL that can decide my choices.

273
00:19:02,840 --> 00:19:05,440
Structured unstructured was another choice.

274
00:19:05,440 --> 00:19:10,320
Now there is one more thing which we have in Microsoft fabric use ecosystem is basically

275
00:19:10,320 --> 00:19:11,320
seqlDB.

276
00:19:11,320 --> 00:19:16,520
Now seqlDB comes in place when we want it to have OLTP kind of a system is if I want to

277
00:19:16,520 --> 00:19:23,160
have create let us say app which is pretty much possible in fabric like I can create you

278
00:19:23,160 --> 00:19:29,520
know application which can take the data insert data quickly in such cases we use seqlDB

279
00:19:29,520 --> 00:19:34,440
now seqlDB replicate data again in data format which is very similar to warehouse.

280
00:19:34,440 --> 00:19:42,360
So live versus means OLTP versus OLAP in OLAP we have a choice versus technology or versus

281
00:19:42,360 --> 00:19:50,120
basically the structure unstructured data now lake house can have unlimited historical

282
00:19:50,120 --> 00:19:54,080
data also we call time travel so lake house can pretty much you can control whatever you

283
00:19:54,080 --> 00:19:55,240
want.

284
00:19:55,240 --> 00:20:00,080
Where house is also between a range two days to one twenty days is something which you can

285
00:20:00,080 --> 00:20:04,200
control one twenty is the right of the highest limit but time travel is possible so time

286
00:20:04,200 --> 00:20:08,880
travel is basically I went ahead and let us update the data today but I can still see

287
00:20:08,880 --> 00:20:14,200
what was my data before that update so there is more flexibility when you use lake house

288
00:20:14,200 --> 00:20:19,400
because it is spark govern and the underlying data technology is more wealth versus the spark

289
00:20:19,400 --> 00:20:26,600
so that is why you get more flexibility when you use lake house also it is one of the

290
00:20:26,600 --> 00:20:31,440
misconception that everybody has that we cannot work on warehouse using the pi spark we

291
00:20:31,440 --> 00:20:36,240
can work using pi spark the only thing is there are special snaps libraries which are there

292
00:20:36,240 --> 00:20:39,800
which you have to use to work with that again there are certain limitation because where

293
00:20:39,800 --> 00:20:44,140
house is still T seql govern so not all the operations are supported but you can work

294
00:20:44,140 --> 00:20:49,440
with the pi spark in the now pi spark the technology because it is supported by spark or

295
00:20:49,440 --> 00:20:56,360
let us say scala on it say spark are or spark sequel they have a huge extendable it means

296
00:20:56,360 --> 00:21:02,920
you can run you know petabytes of data transformation everything using spark while sequel is a great

297
00:21:02,920 --> 00:21:07,640
technology up to a volume of data so that is where if your system is really really large

298
00:21:07,640 --> 00:21:13,240
you will go for lake house so if you are really huge on data you need to you know really

299
00:21:13,240 --> 00:21:19,640
save bring the data whether I should be able to run fast but I want to do it quickly spark

300
00:21:19,640 --> 00:21:24,520
helps you now on the other end sequel extend is own way is it is a same traditional sequel

301
00:21:24,520 --> 00:21:30,120
way where it extend it goes you take the power and you know do does the job but I can tell

302
00:21:30,120 --> 00:21:34,920
you up to you are up to few millions you will not be able to differentiate between the performance

303
00:21:34,920 --> 00:21:39,080
we have implementations which have been done on lake houses and warehouses and both equally

304
00:21:39,080 --> 00:21:45,400
performs good and on millions of rows sequel also never let you down it does really good

305
00:21:45,400 --> 00:21:50,600
performance it shows that you know work done within time with lesser CU consumption all these

306
00:21:50,600 --> 00:21:56,680
things makes it really amazing so I think it is more about choice of technology by the organization

307
00:21:56,680 --> 00:22:02,760
and definitely if it is unstructured data and you need to do a lot of data science where lake house

308
00:22:02,760 --> 00:22:07,720
definitely wins compared to warehouse the the thing the lake house architect

309
00:22:07,720 --> 00:22:15,160
sorry require the different mindset so what if an organization already has a strong SQL and

310
00:22:15,160 --> 00:22:24,440
traditional data warehouse team see see ultimately the two concepts is same now the here here is

311
00:22:24,440 --> 00:22:30,120
understood even if I implement complete lake house so let's say bronze silver and gold all our

312
00:22:30,120 --> 00:22:35,640
lake houses so my so-called one source of truth lake could be a lake house also which is so-called

313
00:22:35,640 --> 00:22:40,680
warehouse in the functional terminology and it will all be warehouse if it is all structured data

314
00:22:40,680 --> 00:22:45,240
I'm happy to be with warehouse but yes if I need to do a lot of data science and when I say data

315
00:22:45,240 --> 00:22:50,520
science is not the use of AI or lm that's pretty much possible even using data is in celebrity but I

316
00:22:50,520 --> 00:22:55,560
need to let's say do develop my own algorithm plus string classifications and everything that is where

317
00:22:55,560 --> 00:23:01,000
you know that scale is typically more easily possible with spark and that is where probably I would

318
00:23:01,000 --> 00:23:06,440
like my data underlying data to be in lake house compared to warehouse while I already said that you

319
00:23:06,440 --> 00:23:11,880
know you can access your data on the warehouse using price part but definitely there is an intermediate

320
00:23:11,880 --> 00:23:16,520
intervention which is required while saving so I would just if I have to do a lot of data science

321
00:23:16,520 --> 00:23:21,480
stuff or I have a lot of unstructured data which I have to be I profoundly you go at the lake house

322
00:23:21,480 --> 00:23:29,160
and read those the data lake fit into the story I think a little bit or what other

323
00:23:29,160 --> 00:23:36,360
almanches of the data tables inside the fabric okay so when we initially so the concept of the lake

324
00:23:36,360 --> 00:23:40,680
which we call data lake is basically you can store any kind of data any format of data

325
00:23:40,680 --> 00:23:46,280
structured or unstructured someplace so lake house fits in that story so basically when we say

326
00:23:46,280 --> 00:23:51,960
lake house the lake house the structured data is basically delta which is same as warehouse in

327
00:23:51,960 --> 00:23:56,280
the in fabric world because it is also delta format in the back end but the unstructured data is

328
00:23:56,280 --> 00:24:00,680
something which you are storing the file so we have the file part which we can store any kind of data

329
00:24:00,680 --> 00:24:06,280
so the concept of one lake or the data lake so the concept of data lake is getting replicated

330
00:24:06,280 --> 00:24:11,400
by lake house while warehouse represents the traditional sequel warehouses with those warehouses

331
00:24:11,400 --> 00:24:15,800
kind of a properties as it properties and everything and yeah definitely acid properties are

332
00:24:15,800 --> 00:24:18,840
something which are not supported on lake house which are supported on the house.

333
00:24:20,440 --> 00:24:28,040
So I think now we understand where the data shows the lift but we need also get them there

334
00:24:28,040 --> 00:24:34,040
what are the main ingestions in fabric.

335
00:24:34,040 --> 00:24:42,840
So injation of the thing we can have a look at the data if it was now the second one

336
00:24:42,840 --> 00:24:49,000
consists of the lake there are few sources for which the mirroring is directly available and for

337
00:24:49,000 --> 00:24:53,880
others you have custom mirroring. Now the advantage of mirroring which by organization like is

338
00:24:53,880 --> 00:25:00,280
basically it is self managed everywhere where we have a good change data capture mechanism mirroring

339
00:25:00,280 --> 00:25:06,600
works very easy. Now Microsoft doesn't charge you for the compute as well as storage up to a limit

340
00:25:06,600 --> 00:25:13,160
for bringing that time so let's say if on in fabric I have on F2 capacity I will get two terabyte of

341
00:25:13,160 --> 00:25:19,480
free storage and pretty compute to bring that to terabyte. I am if I'm on F64,

342
00:25:19,480 --> 00:25:24,680
our F64 is one of the very common capacities and the reason for that is one is there are very

343
00:25:24,680 --> 00:25:31,880
high limitations on the data set size especially for power BA and the power BA viewer license are free

344
00:25:31,880 --> 00:25:38,920
after F64 onwards. So if I am on F64 capacity the mirroring will allow me 64 terabyte of data to

345
00:25:38,920 --> 00:25:44,840
be stored for free and to compute free for bringing that so that is my second choice or you can say

346
00:25:44,840 --> 00:25:53,080
first choice if I don't come short course then comes my ability to bring in data using pipelines which

347
00:25:53,080 --> 00:26:02,200
has copy activity and copy job. Copy job again is having very good CDC a co-options available and copy

348
00:26:02,200 --> 00:26:07,640
activity also now both are meant for the purpose. So if I want to bring the data these are my choices

349
00:26:07,640 --> 00:26:15,400
but yes the pipeline copy activity and copy job has around 40-50 connectors and that is where data

350
00:26:15,400 --> 00:26:22,440
flow wins because data flow gen 2 has more than 200 plus connectors it is it has power query which

351
00:26:22,440 --> 00:26:28,920
can also do the transformations. So data flow gen 2 would be your fallback option most of the

352
00:26:28,920 --> 00:26:35,720
cases when you are not able to connect and all of them has generic connector like RSTPI and we

353
00:26:35,720 --> 00:26:43,240
also have the ODBC generic connector or we have OLEDB all these things are also available web

354
00:26:43,240 --> 00:26:50,520
connector etc. So our choice would be finally if we are falling back we use data flow gen 2 to bring

355
00:26:50,520 --> 00:26:55,960
in the data it has maximum number of connectors and it has also fast copy connector now which will

356
00:26:55,960 --> 00:27:00,920
make sure that it is as good as your copy activity in most of the cases definitely transformation

357
00:27:00,920 --> 00:27:06,440
choices are there in case you are bringing from data flow gen 2 you can transform the data also

358
00:27:06,440 --> 00:27:13,640
there. So now because we have this path and Python inside the system if your data is available

359
00:27:13,640 --> 00:27:19,560
online right now it is not supported for on-premise sources you do have now various connectors available

360
00:27:19,560 --> 00:27:24,200
at the notebook level previously also we were able to bring the data in but we have to mention all

361
00:27:24,200 --> 00:27:28,120
the credentials in the notebook but for now we have connections which is basically supported

362
00:27:29,000 --> 00:27:34,120
in a manner that you don't have to show your credentials in the notebook and that again creates one

363
00:27:34,120 --> 00:27:38,840
more way to bring in your data now when you bring in the notebook the handling of the data is

364
00:27:38,840 --> 00:27:43,640
mostly in your end it is basically how you bring how much you bring in everything while in all

365
00:27:43,640 --> 00:27:48,440
other cases it is managed by the system so you have these many choices to bring in data.

366
00:27:48,440 --> 00:27:55,720
And what would you say how important is power-carry knowledge for working effectively with

367
00:27:55,720 --> 00:28:04,040
the data flow gen 2? See power-carry is something which is data flow gen 2 is having basically it is

368
00:28:04,040 --> 00:28:10,360
same as our power-base power-carry or your our data flow gen 1's query and if we know power

369
00:28:10,360 --> 00:28:16,120
query it would be really helpful because I can tell you those who are migrating from power-bava

370
00:28:16,120 --> 00:28:21,640
word their first choices data flow gen 2 those who are coming from Azure word their first choices

371
00:28:21,640 --> 00:28:30,200
typically the data pipelines and let's say PISPAR transformations and everything so we will get very

372
00:28:30,200 --> 00:28:35,080
familiar things whenever we are coming from a different you know world to the fabric world.

373
00:28:35,080 --> 00:28:39,080
fabric is new so somebody from the power-bava world will come somebody from the let's say

374
00:28:39,080 --> 00:28:45,960
the PISPAR engine world will come somebody will come from different migration kind of a

375
00:28:45,960 --> 00:28:50,360
pools which is very similar to pipeline you will come so orchestration everything it happens so

376
00:28:50,360 --> 00:28:53,800
we will find some very similar tools here in Microsoft fabric to start with.

377
00:28:53,800 --> 00:29:02,520
Okay awesome I think is there performance on main

378
00:29:02,520 --> 00:29:08,680
tiny ability limits that we should know about in that of low gen 2?

379
00:29:08,680 --> 00:29:17,400
Okay so this is a really easy so teach gen 2 is very good for a small to medium size of data

380
00:29:17,800 --> 00:29:22,920
but when you come to a larger scale the transformation in PISPAR can excel much faster than the

381
00:29:22,920 --> 00:29:28,520
in data flow gen 2 while Microsoft is continuously investing and consistently improving the

382
00:29:28,520 --> 00:29:33,560
performance of data flow gen 2 when it is large volume of data but it becomes really easy

383
00:29:33,560 --> 00:29:39,240
because Spark is PISPARC or Python is like an open language you can have tons of things to do

384
00:29:39,240 --> 00:29:45,080
and manage it and that is why you know if it is a really large scale data transformation we

385
00:29:45,080 --> 00:29:51,400
prefer PISPARC. I don't have a data flow gen 2 versus PISPARC but data flow gen 1 versus

386
00:29:51,400 --> 00:29:57,960
PISPARC performance could be having 80% gain both in terms of compute as well as in terms of time

387
00:29:57,960 --> 00:30:03,000
that is a proven fact but we have observed at the customer level. So data flow if you had a data

388
00:30:03,000 --> 00:30:07,800
flow gen 1 if you completely converted into PISPARC because save a lot on the compute units as well

389
00:30:07,800 --> 00:30:13,320
as the time both you are saving it is not one we are saving and that is where PISPARC could be a

390
00:30:13,320 --> 00:30:18,360
really good choice for data transformation going forward in the future for large scale data.

391
00:30:18,360 --> 00:30:28,520
That is interesting for someone coming from the PowerBRS-L part when they actually need PISPARC.

392
00:30:28,520 --> 00:30:35,080
So the data transformation part is something which is basically instead of SQL can happen on

393
00:30:35,080 --> 00:30:41,000
the PISPARC and when I say PISPARC I have this habit of calling PISPARC but we can use these Spark

394
00:30:41,000 --> 00:30:45,720
notebooks which could be in Scala which could also be in Spark SQL. So if you are from SQL

395
00:30:45,720 --> 00:30:50,760
where you will find Spark SQL very familiar to you. If you are from let's say power query world

396
00:30:50,760 --> 00:30:55,400
you will see PISPARC almost has similar kind of functions which you have let's say if I want to

397
00:30:55,400 --> 00:31:00,520
join they are also have I have table dot let's say join something and here also I have the data

398
00:31:00,520 --> 00:31:06,280
frame dot join. So very similar functions I have here also to do one one on one operations with

399
00:31:06,280 --> 00:31:13,000
compared to power query. So definitely if we are coming from let's say SQL plus power query world

400
00:31:13,000 --> 00:31:19,000
we can start on fabrics with those but as the data goes is the things grows slowly we have to start

401
00:31:19,000 --> 00:31:26,920
moving some of our stuff to PISPARC or maybe we can say Spark notebooks. Yeah yeah I think that's

402
00:31:26,920 --> 00:31:34,040
interesting how do notebooks change the way data engineer teams work actually. See we have to

403
00:31:34,040 --> 00:31:39,640
understand that the notebooks especially running by Spark is made for scale.

404
00:31:39,640 --> 00:31:45,960
So because we can decide you know how much compute they should use they should use 10 node

405
00:31:45,960 --> 00:31:52,440
small cluster or a step small cluster or large cluster all those things we can do and that

406
00:31:52,440 --> 00:31:58,120
scale comes up with Spark and because of that you know you might do need to do heavy lifting

407
00:31:58,120 --> 00:32:03,560
and that heavy lifting you want it to do and you also want it to save time not only the volume

408
00:32:03,560 --> 00:32:07,880
of data increasing but you also want to reduce the time that is where easily can happen Spark and

409
00:32:07,880 --> 00:32:13,640
Spark is meant for the entire illusion of Spark happens that okay I have the huge data how do I

410
00:32:13,640 --> 00:32:20,760
want to process that data and get the time in time processing in quickly and that's where the

411
00:32:20,760 --> 00:32:25,240
Spark was entered that you know okay distribute this entire jobs on multiple nodes and then do it

412
00:32:25,240 --> 00:32:29,400
and then bring the results in and that's where the Spark notebooks are providing you here in

413
00:32:29,400 --> 00:32:36,280
Microsoft fabric. So for scale and for you know getting the results faster the Spark always helps you

414
00:32:36,280 --> 00:32:47,000
that's so sounds interesting is there any performance mistake begin us to with Spark?

415
00:32:47,000 --> 00:32:53,320
The thing which we have to do is when you start as a beginner don't try to play around with

416
00:32:53,320 --> 00:32:58,760
these Spark settings of fabric also secondly instead of dealing with it in the notebook level

417
00:32:58,760 --> 00:33:04,360
please create those different Spark clusters settings at the workspace level or capacity level and

418
00:33:04,360 --> 00:33:09,160
use it because I remember a case where what happens one of the team and we have allowed the setting

419
00:33:09,160 --> 00:33:14,280
of the Spark cluster at the workspace level one of the team is created such a big cluster

420
00:33:14,280 --> 00:33:18,040
and they were able to complete their job in one hour they were really happy but they end up

421
00:33:18,040 --> 00:33:22,760
consuming the entire capacity for all other performance so those could be there and again

422
00:33:23,560 --> 00:33:30,040
there is a little bit of there are less of control sometimes required in the fabric spark because

423
00:33:30,040 --> 00:33:34,760
some of the things are already handled by fabric so you might not to explicitly go and start Spark

424
00:33:34,760 --> 00:33:39,640
it automatically creates sessions for you so the small unlearning here that you know you already

425
00:33:39,640 --> 00:33:44,040
already have a Spark session you only to explicitly started you can start using it and then there

426
00:33:44,040 --> 00:33:48,520
are only selective parameters which it takes if you take care of these things I think it's pretty

427
00:33:48,520 --> 00:33:54,200
smooth journey and even I think for a beginner who doesn't understand as a Spark works it is much

428
00:33:54,200 --> 00:34:00,040
easier journey because you are not taking care of where is my root folder how should I start the Spark

429
00:34:00,040 --> 00:34:03,720
I just know that I have a Spark session with all those capabilities and I can use it.

430
00:34:03,720 --> 00:34:14,920
Awesome another topic I think the biggest buzzer words is the it's yeah I don't know the mid-medallion

431
00:34:14,920 --> 00:34:21,080
architecture and I think it's a little bit of that marketing that we call it Bronx the Servant Gold

432
00:34:21,080 --> 00:34:32,760
have have these vocabulary are modern how how how important are the the medallion architecture

433
00:34:32,760 --> 00:34:39,640
for for fabric see for see architecture should depend on you know organization and what kind of

434
00:34:39,640 --> 00:34:46,360
data we have see if war organization is having a very strong master data management done at the

435
00:34:46,360 --> 00:34:52,520
source level and data cleaning was a real big focus probably you will have very clean data which

436
00:34:52,520 --> 00:34:57,480
is coming into the grass layer itself then you daily need silver or then can you directly go to

437
00:34:57,480 --> 00:35:02,600
silver plus gold layer that is the call you have to take because I firmly be the number of layer should

438
00:35:02,600 --> 00:35:07,880
you entire dependent on the kind of data instead of you know I want to follow medallion architecture

439
00:35:07,880 --> 00:35:12,360
because when we initially started the data engineering if you remember we used our staging area and

440
00:35:12,360 --> 00:35:17,640
the warehouse the one source of truth we we were you know doing that entire as a part of transformation

441
00:35:17,640 --> 00:35:23,240
maybe we were creating the intermediate table not explicitly calling it silver now because

442
00:35:23,240 --> 00:35:27,160
this has been a well defined structure of medallion architecture we have bronze silver and gold

443
00:35:27,160 --> 00:35:33,320
so if you have a very strong master data foundation which is laid outside probably just think about

444
00:35:33,320 --> 00:35:38,440
do you really need one silver copy because you might have clean data but yes if the master data

445
00:35:38,440 --> 00:35:45,480
foundation is not so great then we need to bring in bronze create a really strong clean layer for

446
00:35:45,480 --> 00:35:51,480
the consumption of data at silver layer and then create the gold layer so and I see most of the

447
00:35:51,480 --> 00:35:57,800
organization at some stage or other having this challenges of master data because it's not managed

448
00:35:57,800 --> 00:36:03,480
so greatly so probably if you can say 80% of the organization will fall in a category when they have

449
00:36:03,480 --> 00:36:07,800
to follow the medallion architecture of bronze silver gold yes there are organization who has

450
00:36:07,800 --> 00:36:12,520
really strong on you know data quality and data master data management and everything

451
00:36:12,520 --> 00:36:17,480
probably for them silver is not too much other than you know having a following an architecture

452
00:36:17,480 --> 00:36:21,720
probably they could have bronze and silver gold together while doing the transformation putting

453
00:36:21,720 --> 00:36:27,960
it into the gold yeah that's interesting so so where should the the the business logic

454
00:36:27,960 --> 00:36:33,080
live so should it be the logic primarily exist in transformation in the warehouse in the

455
00:36:33,080 --> 00:36:40,520
thematic model or some well now this is a testing now there are so this is a little bit of

456
00:36:40,520 --> 00:36:49,480
the business logic which are little complex and the lying level we cannot keep them at the

457
00:36:49,480 --> 00:36:56,040
semantic model level ideally speaking I would be happy to have all my major able or KPI's or

458
00:36:56,040 --> 00:37:01,160
performance indicator at the semantic model level because it's easy it's dynamic I don't need to

459
00:37:01,160 --> 00:37:05,880
reload the data for doing it so anything which I can keep at that level but the problem is if the

460
00:37:05,880 --> 00:37:12,120
calculations are at lying level if I have millions of rows where I need to go and do one calculation

461
00:37:12,120 --> 00:37:18,040
and then do it up it is not going to work out at the semantic model level and in such cases we always

462
00:37:18,040 --> 00:37:23,000
prefer to have the calculation done at the gold layer means basically when we load our data from

463
00:37:23,000 --> 00:37:29,000
silver to board we will apply those business logic so business logic is shared between the gold

464
00:37:29,000 --> 00:37:35,000
layer creation and the semantic model our preference is to have it at the semantic model level because

465
00:37:35,000 --> 00:37:39,960
they can change easily we can change formula we can create new matrices but definitely we don't

466
00:37:39,960 --> 00:37:45,000
want to do role level calculations for that so we have to take a cautious call to make it at the

467
00:37:45,000 --> 00:37:53,720
two levels that's how it moves to a side awesome and that's a little bit move closer to the

468
00:37:53,720 --> 00:38:01,480
business layer where are the semantic models so important and what makes the good semantic model

469
00:38:01,480 --> 00:38:09,320
okay see if you actually would ask this question we could be couple of your banks

470
00:38:09,320 --> 00:38:14,520
or three years back for a semantic model is for consumption of power VA isn't it so we have a

471
00:38:14,520 --> 00:38:19,320
semantic model we create power VA reports on that but within fabrics the things have changed

472
00:38:19,320 --> 00:38:26,040
especially after data agent ontology and fabric app is all changing this playground and creating

473
00:38:26,040 --> 00:38:31,640
the bigger role for semantic model so if you ask me if AI brings a challenge to let's say power VA

474
00:38:31,640 --> 00:38:37,000
as a visualization where power VA is not facing challenge is the semantic model because

475
00:38:37,000 --> 00:38:41,880
understand one thing our all business logic are going to decide the relationship and the key

476
00:38:41,880 --> 00:38:46,600
majors and their definitions are known for semantic model and if I work on a semantic model I don't

477
00:38:46,600 --> 00:38:51,000
need to tell the definition what is my net sense semantic model knows it what is my gross is what is

478
00:38:51,000 --> 00:38:56,440
my revenue definition everything semantic model knows it and let's say if somebody need to do and

479
00:38:56,440 --> 00:39:00,760
go and do some transformation on directly on my lake house and warehouse they don't know this

480
00:39:00,760 --> 00:39:06,840
definition now we have a semantically lab library which can enable you now our data agents also

481
00:39:06,840 --> 00:39:11,960
work on semantic model it means the data agent wanted to take the advantage of the definitions which

482
00:39:11,960 --> 00:39:18,040
are not similarly the ontology can be created on our semantic model it which is going to take the

483
00:39:18,040 --> 00:39:22,840
advantage of already existing knowledge by the semantic models the relationship the KPIs

484
00:39:22,840 --> 00:39:27,960
now we have fabric apps which I got wrong recently so one of the things which has happened because

485
00:39:27,960 --> 00:39:35,640
of this AI explosion over the last few years that we came to something which we are doing let's say

486
00:39:35,640 --> 00:39:41,000
maybe a 20 years back that I can have my custom created report which whatever I want

487
00:39:41,000 --> 00:39:47,960
and then from there in that place we move to the tools because it is heavy maintenance because

488
00:39:47,960 --> 00:39:52,520
I can create my report let's say on Java technology by connecting to database and do everything

489
00:39:52,520 --> 00:39:56,840
and have slice and dice and everything but it was a huge maintenance I can't do it and that's why I

490
00:39:56,840 --> 00:40:02,440
knew tools like Power BI tab you to just do it with drag and drop but what AI has done over

491
00:40:02,440 --> 00:40:07,240
the last couple of years is you know you just go type in some command and your react dashboard is

492
00:40:07,240 --> 00:40:12,920
ready you say okay I need all these features it's already and that's where you know we were coming

493
00:40:12,920 --> 00:40:17,560
back okay why should I have you know executive only looking at you know reports which is

494
00:40:17,560 --> 00:40:23,880
fixed in the boundaries of what the tool can create and that is where you home we have seen a lot

495
00:40:23,880 --> 00:40:29,320
of people were developing these AI apps and the challenge of all those AI apps is they understand the

496
00:40:30,920 --> 00:40:36,280
warehouse or lake house better because they are having sequel and points and you the most the most

497
00:40:36,280 --> 00:40:41,720
common technology to be used by anyone is sequel or as a sequel and because of that developing here

498
00:40:41,720 --> 00:40:45,880
but our logics for lying in semantic model who will transform those major or those

499
00:40:45,880 --> 00:40:51,080
capyra definitions into the sequel and there's again work and that's where Microsoft came up with

500
00:40:51,080 --> 00:40:58,280
fabric app enabled by Rafe in framework now it can create these those reports those it's so it

501
00:40:58,280 --> 00:41:03,800
used typescript and you can create those reports which all the kind of UI you need the custom UI

502
00:41:03,800 --> 00:41:09,480
the kind of slicers you need the visual out of the box visuals and everything you on your semantic

503
00:41:09,480 --> 00:41:14,920
model and that's what we call fabric and then it is again getting published inside your fabric works

504
00:41:14,920 --> 00:41:21,000
pieces so same security same everything so now the role of the semantic model has greatly increased

505
00:41:21,000 --> 00:41:26,440
in terms of relationship and the definition it owns but yes the consumption could be power be

506
00:41:26,440 --> 00:41:33,640
a visualization could be data agent could be ontology ontology driven data agents or maybe fabric apps

507
00:41:33,640 --> 00:41:40,200
but yes we have a layer which store which knows our business which understand our business well

508
00:41:40,200 --> 00:41:42,760
and it can be used for the downstream by anything

509
00:41:42,760 --> 00:41:51,000
and and what would you say how should teams approach measure of firsts calculated columns

510
00:41:53,320 --> 00:41:59,880
okay so when we say calculated columns probably I would like them at the gold layer other than

511
00:41:59,880 --> 00:42:04,360
these calculated columns which are now which is very interesting thing which has very recently

512
00:42:04,360 --> 00:42:09,320
happened in the power be a world is contracts driven calculated columns now contracts given calculated

513
00:42:09,320 --> 00:42:14,200
columns means they change their value because of user climate it also help us in you know

514
00:42:14,200 --> 00:42:19,880
some time to hide some data basically if I wanted to do data masking it also helps in that

515
00:42:19,880 --> 00:42:24,920
and sometime it helped in changing the column basically I want to show different column so if it is

516
00:42:24,920 --> 00:42:30,760
if your column is not a context driven column it should happen at the gold layer all the majors

517
00:42:30,760 --> 00:42:37,000
anything which is a major should be going to happen at the semantic model level context even calculated

518
00:42:37,000 --> 00:42:41,640
columns should happen at the model level semantic model level and rest of the calculation which are

519
00:42:41,640 --> 00:42:45,960
line level or really complex which is which can slow down at the runtime should be moved back to the

520
00:42:45,960 --> 00:43:01,000
bouldier. If I worked with fabric and probably iDux was the I don't know the main topic and I see

521
00:43:01,000 --> 00:43:13,400
Aids awesome in generating DAX code now did you think DAX stays important or makes tends to learn DAX

522
00:43:13,400 --> 00:43:20,600
anymore see I think in the new world what is happening because AI is doing everything like I am when

523
00:43:20,600 --> 00:43:25,400
I created a let's say referring dashboard it uses typescript I am not sure what typescript is

524
00:43:25,400 --> 00:43:29,960
how it is written what is the structure of that and everything similarly it is happening with all

525
00:43:29,960 --> 00:43:35,080
the technology like probably you can generate a pi spark notebook using co-pilot or even AI

526
00:43:35,080 --> 00:43:40,840
that is to the DAX also but we have to understand when the things does not work and when the things

527
00:43:40,840 --> 00:43:45,880
need to be optimized and that is where expertise comes in place now some of us who are there in the

528
00:43:45,880 --> 00:43:51,080
industry for some time we had gained that expertise over a period of time now for the new learners

529
00:43:51,080 --> 00:43:56,920
it is very easy to get a DAX from AI so do I really need to have an expertise I think in a longer run

530
00:43:56,920 --> 00:44:02,360
at least for next few years you need to have that expertise on the subject I know the technology

531
00:44:02,360 --> 00:44:08,040
is really fast and maybe after some time it is more on the logics and less on the technology so

532
00:44:08,040 --> 00:44:15,240
probably the language would be you know using your logics and explaining your logic to the AI

533
00:44:15,240 --> 00:44:21,960
by our prompts not let's say if C Java DAX of our query it's just the thing which internally getting

534
00:44:21,960 --> 00:44:27,960
created and getting debugged also by AI only but yes definitely knowing something in depth how it

535
00:44:27,960 --> 00:44:32,600
works it's always really important because in that case you can go and fix something which is

536
00:44:32,600 --> 00:44:39,400
actually not getting fixed by AI or which is getting not optimized by AI but as this intervention

537
00:44:39,400 --> 00:44:46,760
increase probably up to me loving you know how the algorithm works how the logic works at different

538
00:44:46,760 --> 00:44:50,840
places is more important than you know the language is what we are working use in future.

539
00:44:50,840 --> 00:44:59,080
And I think another topic a lot of people aware of actually it's it's security I have all my

540
00:44:59,080 --> 00:45:07,000
data now and well like so how should we start with security in fabric architecture?

541
00:45:07,000 --> 00:45:15,400
See in terms of the security start from your workspace is from where you can start controlling

542
00:45:15,400 --> 00:45:21,640
by rules whether you are a viewer you are a member you are a contributor or you are an

543
00:45:21,640 --> 00:45:28,600
advent that's the first step now in the fabric what typically we are trying not to have the workspace

544
00:45:28,600 --> 00:45:33,480
level access we try to have item level access like in a lake house or warehouse I can have a read data

545
00:45:33,480 --> 00:45:38,840
read and write and all those permissions then if further goes down I can go to the item level now item

546
00:45:38,840 --> 00:45:44,680
level permissions typically we will try to control using one-leg security wherever possible

547
00:45:44,680 --> 00:45:49,240
and if we are not putting one-leg security we are going to have semantic model level security

548
00:45:49,240 --> 00:45:56,200
a one-leg security has came I think around one and a half to years back and it is still evolving at some

549
00:45:56,200 --> 00:46:00,600
places it may not provide you that option and that is where we go and put the security

550
00:46:00,600 --> 00:46:03,880
semantic model level but one-leg security is something which we are going to use

551
00:46:03,880 --> 00:46:12,920
and that is where we can secure the data means we can do both the object level security

552
00:46:12,920 --> 00:46:19,480
OLS and RLS using one-leg security so what we are going to do here is basically where but possible

553
00:46:19,480 --> 00:46:25,960
in terms of OLS and RLS if possible we are going to use one-leg security if we are unable to control

554
00:46:25,960 --> 00:46:30,360
at one level one-leg level or we don't want to contain at one-leg level basically

555
00:46:30,360 --> 00:46:34,920
lake house of warehouse we are going to do it at the semantic model level and definitely object

556
00:46:34,920 --> 00:46:39,320
level security needs a set level security basically or report so do your warehouse on

557
00:46:39,320 --> 00:46:44,120
lake house we can control them at the works is either the workspace level or individual item level

558
00:46:44,120 --> 00:46:48,680
so from workspace to item to inside the item with one-leg security

559
00:46:48,680 --> 00:47:00,760
awesome so the how how do is the role level security works with the semantic level correct

560
00:47:00,760 --> 00:47:08,360
or what's the best practice see the role of a security basically now it can be done at two level

561
00:47:08,360 --> 00:47:13,960
one is basically at the one-leg security which is can again you can go and define okay I for

562
00:47:13,960 --> 00:47:19,400
this is the filter for that so in one-leg you decide the filter and then you create a role and

563
00:47:19,400 --> 00:47:23,720
that can assign to the people so that filter will get applied so you will let's say only get north

564
00:47:23,720 --> 00:47:29,080
I create a filter it is north and it will get applied at the end all the endpoints SQL endpoints the

565
00:47:29,080 --> 00:47:35,560
the semantic model endpoint or even at the file endpoint now when I come to the semantic model the

566
00:47:36,360 --> 00:47:41,720
semantic model security of power be a remain same so it is the same way we create a role and inside

567
00:47:41,720 --> 00:47:47,080
the role we again define a dynamic security using a table where we have the email addresses using

568
00:47:47,080 --> 00:47:52,920
user principle a model static security by like saying reason equal to north and then we once the

569
00:47:52,920 --> 00:47:58,600
semantic model is published we go in the security model layer and add users or security groups

570
00:47:58,600 --> 00:48:02,920
preferably we should use security groups and security groups should be created like you know

571
00:48:02,920 --> 00:48:07,240
this is a viewer security group or this is a financial viewer security group this is says viewer security

572
00:48:07,240 --> 00:48:12,040
group or says contributor security groups we should create security groups and assign those security

573
00:48:12,040 --> 00:48:18,680
groups to the security layer or to the report or to the one-leg security that is always a best

574
00:48:18,680 --> 00:48:25,400
practice so so we have the workspace permissions we have the data permissions and we have the semantic

575
00:48:25,400 --> 00:48:33,880
model security but how how to gather when this when when an organization has hundreds of thousands of

576
00:48:33,880 --> 00:48:43,400
users and with different requirements yeah see I think the why we have these multiple layers and

577
00:48:43,400 --> 00:48:48,760
but all these are all you know really smooth and integrated so the reason is let's say if I want

578
00:48:48,760 --> 00:48:53,080
to have few people to be admin and having access to everything I don't have to you know go and

579
00:48:53,080 --> 00:48:57,080
do multiple permission I can deal with workspace level permission and they can access everything

580
00:48:57,080 --> 00:49:01,480
in the workspace they want but there are certain set of user I don't want to give them you know

581
00:49:01,480 --> 00:49:08,440
access to everything let's say we happened with us also there are report user who are report builder

582
00:49:08,440 --> 00:49:13,720
as well as the report viewer but we don't want to expose them the let's say warehouse is or

583
00:49:13,720 --> 00:49:18,280
lake houses so we control the security at the semantic model level we give them access and we make

584
00:49:18,280 --> 00:49:22,200
sure that it happens in such a manner that they don't have to see other objects so all those

585
00:49:22,200 --> 00:49:27,080
possibilities are covered and that is why you have this two three years now we can do some AI

586
00:49:27,080 --> 00:49:32,680
automation for that to make sure it happens like basically the security when we create the security

587
00:49:32,680 --> 00:49:37,560
group on a larger organization these are also not manually created you have let's say a service

588
00:49:37,560 --> 00:49:42,120
request which I go to raise and you will get added to a security group the same request may add you

589
00:49:42,120 --> 00:49:47,880
to let's say a RLS table also where you get an access to a region to a city or to a product also

590
00:49:48,440 --> 00:49:53,480
and then those RLS tables are used in your semantic model to govern the security and similarly

591
00:49:53,480 --> 00:49:59,080
the assignment also happens let's say I asked for sales viewer role so sales viewer role with let's say

592
00:49:59,080 --> 00:50:06,760
city as new new york so the service request will go add me to the sales viewer security group it will

593
00:50:06,760 --> 00:50:11,720
go mid go and add me to the new york it will also go and add me to the let's say workspace or to the

594
00:50:11,720 --> 00:50:19,720
reports which I can see as the sales viewer so all these automations can help us out. And I have the

595
00:50:19,720 --> 00:50:28,760
fabric admin center I also have I don't know it's power be either same name but I also have the

596
00:50:28,760 --> 00:50:37,000
option in power be I and and I also can use use entra how show I had all these tools or what's

597
00:50:37,000 --> 00:50:46,280
the best practice here. So, so we have it that level so basically entra all the security where

598
00:50:46,280 --> 00:50:51,000
where basically your all log is getting created and security groups are also getting created at the

599
00:50:51,000 --> 00:50:57,160
Azure level so that part happens there after that fabric and power be I is the workspace level

600
00:50:57,160 --> 00:51:02,280
security same way to apply suppose that we give the assignment of the objects on the fabric

601
00:51:02,280 --> 00:51:06,440
level to the security groups permissions are also decided security or sometime you will let's say

602
00:51:06,440 --> 00:51:10,840
try let's say fabric app came is a new feature now I don't want everybody to go ahead and create those

603
00:51:10,840 --> 00:51:15,480
reports so I will give permission to a set of users which are part of a security group to create

604
00:51:15,480 --> 00:51:21,320
those reports same way I'll give the finance user access to let's say as a contributor or a member

605
00:51:21,320 --> 00:51:26,200
or admin to you know build the content and give it and viewers of those let's say finance

606
00:51:26,200 --> 00:51:30,840
report security group of finance viewers I'll add deadness the viewers and also add them to security

607
00:51:31,640 --> 00:51:37,720
so this is how it happens so definitely entrap part of the Azure creation of your emails and everything

608
00:51:37,720 --> 00:51:42,520
not going away so that will happen security group creation should also happen from there and

609
00:51:42,520 --> 00:51:48,920
rest of the security will be managed at the fabric level. Yeah I think we have a lot from from

610
00:51:48,920 --> 00:51:55,480
the engineering part but yeah ultimately the most business users don't care about spark data

611
00:51:55,480 --> 00:52:03,080
levels or like house they care about answering questions how do we bridge these these get

612
00:52:03,080 --> 00:52:12,360
so the fabric has lots of options basically if you ask me future could be talking to your data

613
00:52:12,360 --> 00:52:18,680
agents right now we also see a lot of people started using data agents and in that data agent

614
00:52:18,680 --> 00:52:23,480
ontology driven data agent is something really important because what happens when you create ontology

615
00:52:23,480 --> 00:52:28,280
on top of your data it do understand data much better it goes look at the data and try to have the

616
00:52:28,280 --> 00:52:32,840
data part match and it can provide better answer and it can sometimes work even better on the

617
00:52:32,840 --> 00:52:38,040
snowflake models or the models where you know the typical star schema kind of stuff is not there

618
00:52:38,040 --> 00:52:44,200
it also works better there so ontology driven data agents would be our first choice in the future

619
00:52:44,200 --> 00:52:49,000
to communicate with our data and understand our data then we will have the reports definitely

620
00:52:49,000 --> 00:52:54,520
I will not wanted to go and communicate and always ask questions I would definitely want some

621
00:52:54,520 --> 00:52:59,320
numbers in front of me every day morning now those everyday morning numbers will either come from

622
00:52:59,320 --> 00:53:05,000
a fabric app report or power be a report there are very high chances in the future that your data

623
00:53:05,000 --> 00:53:11,640
agents might create a report for you which is basically could be a fabric app and a power be app

624
00:53:11,640 --> 00:53:16,040
after you ask a set of question okay this is how I perform these are the KPI I look at it

625
00:53:16,040 --> 00:53:20,680
this is the information I need Monday morning this is what I need to do morning so data agent probably

626
00:53:20,680 --> 00:53:26,040
could go ahead and create finally based on your communication a report which works the VR way

627
00:53:26,040 --> 00:53:32,600
and not you know the report created by a developer which gives you okay same set of reports for all

628
00:53:32,600 --> 00:53:40,760
four executive same set of reports for you know thousands of users so probably data agent and the

629
00:53:40,760 --> 00:53:48,520
data agent created reports would be something which we will be frontending in future yeah I think a

630
00:53:48,520 --> 00:53:57,960
lot of power be I and business users like to have a self service be I it's a little bit distressed

631
00:53:57,960 --> 00:54:04,040
us from our most organization what do successful service ourselves levels I look like

632
00:54:04,040 --> 00:54:10,120
okay so now see understand we are talking about this town self service but when we use that

633
00:54:10,120 --> 00:54:15,320
creating those reports the movement we go into the complex calculation we realize we need to go

634
00:54:15,320 --> 00:54:19,480
back to the developer even in the power be in world where you know you have the self service knowledge

635
00:54:19,480 --> 00:54:25,320
of decks was required but in this new world of AI driven you don't need that even if a major is not

636
00:54:25,320 --> 00:54:31,080
there your AI agent is going to create so this is real self sir your model doesn't need to have all

637
00:54:31,080 --> 00:54:36,760
the formula your model doesn't need to have all the logics it will on the flight created now definitely

638
00:54:36,760 --> 00:54:42,120
realize okay if it creates on the flight takes lot of time it's better to improve my model to answer

639
00:54:42,120 --> 00:54:46,520
some of those questions maybe better to have a major for that so I think two self surveys what we

640
00:54:46,520 --> 00:54:52,360
are calling is on the plate using data agents and the AI driven development so where you know

641
00:54:52,360 --> 00:54:59,320
users will just be so I'll tell you I was on the weekend I have created entire fabric from

642
00:54:59,320 --> 00:55:05,640
ingestion of data to transformation creating a power be a model creating a report just by talking

643
00:55:05,640 --> 00:55:12,280
to the GitHub co-pilot I have not went ahead in any of the UI other than the testing so we're reaching

644
00:55:12,280 --> 00:55:17,480
to that level now understand what would be the future that you define your requirement well to

645
00:55:17,480 --> 00:55:21,880
define your requirement well you have tools also which will define your requirement well then they

646
00:55:21,880 --> 00:55:25,640
will break it and then you will only go ahead and say okay go ahead and create the tickets on my

647
00:55:25,640 --> 00:55:30,280
DevOps and then you will come to GitHub co-pilot and say okay execute all these tickets on my one

648
00:55:31,080 --> 00:55:38,040
and this getting created and you do so the knowledge of technology only needed for the debugging skills

649
00:55:38,040 --> 00:55:43,480
for what is not working how to make it work but it's going to be a really self-serve environment like

650
00:55:43,480 --> 00:55:50,280
you know in our generation we used to learn computers but if you look at the next generation they

651
00:55:50,280 --> 00:55:55,080
they know computer by you know they they know is as a skill be it's just like a language skill for

652
00:55:55,080 --> 00:55:59,720
that for future the AI would be cut that kind of skill where everybody knows how to you know build

653
00:55:59,720 --> 00:56:05,400
those complex some of those systems using you know AI so that will become a primary skill for the

654
00:56:05,400 --> 00:56:10,600
next set of people who are you know now learning those things so probably for them it would be just

655
00:56:10,600 --> 00:56:18,200
okay I go or I type prompt I'll have my stuff ready for me awesome yeah thank you are we really

656
00:56:18,200 --> 00:56:25,640
little bit running out of time so I have in every session of rapid fire rounds so I ask a short

657
00:56:25,640 --> 00:56:32,520
question you give a short answer is it though are you ready out yeah okay let's also warehouse

658
00:56:32,520 --> 00:56:42,680
lecels escuelor pi spark pi spark dataflow gen 2 or notebook not book back so escuel

659
00:56:44,680 --> 00:56:55,160
decks most underrated fabric feature fabric apps the biggest power be I'm modeling mistake

660
00:56:55,160 --> 00:57:04,520
many many relationships what is when such an gallia called you and say hey you get all the money

661
00:57:04,520 --> 00:57:13,000
resources to build your dream feature what will you build probably I would want to improve

662
00:57:13,000 --> 00:57:17,800
upon on data agents so just ask them okay this is my requirement have a dashboard ready every day

663
00:57:17,800 --> 00:57:23,320
it should change based on some requirements I don't want to see same dashboard every day one

664
00:57:23,320 --> 00:57:31,080
fabric skill every power be I professional show to low probably you should be knowing power query

665
00:57:31,080 --> 00:57:42,840
index if nothing else is fabric apps in you power be I yes awesome so and we should I invite

666
00:57:42,840 --> 00:57:53,400
next at what questions do I ask I think you can this in Rajin who's again Microsoft fabric MVP

667
00:57:53,400 --> 00:58:01,480
and scheme more questions about ontology and fabric app I think he's one lot of working on

668
00:58:01,480 --> 00:58:05,640
real-time analytics actually he's working a lot on real-time analytics so maybe you have a session

669
00:58:05,640 --> 00:58:13,560
on fabric real-time with Rajindra the data platform MVP and if an organization wants to start with

670
00:58:13,560 --> 00:58:22,440
fabric tomorrow what should it's first project be I think any data project where you think I need

671
00:58:22,440 --> 00:58:28,840
some kind of analysis it just in a starting point it could be just you know knowing you know

672
00:58:28,840 --> 00:58:33,640
how my employee performance is there or how my sales is ongoing if you think you need data analysis

673
00:58:33,640 --> 00:58:39,400
fabric is there for you awesome so yeah I'm a thank you for for joining me and talking taking

674
00:58:39,400 --> 00:58:45,720
us through the computer the Microsoft fabric analytics journey what I think in this conversion

675
00:58:45,720 --> 00:58:52,200
makes clear is that fabric isn't simply another in real analytics product real opportunity comes

676
00:58:52,200 --> 00:58:56,760
from connecting data in guest gen engineering lake house where our architectures and

677
00:58:56,760 --> 00:59:03,240
semantic model security power bar into one core and platform and the technology alone isn't

678
00:59:03,240 --> 00:59:10,280
enough the architecture ultimately has has to deliver trust information that people can use

679
00:59:10,280 --> 00:59:16,520
to make better decisions so for all the people to listen to the podcast and interested to connect

680
00:59:16,520 --> 00:59:23,800
and see how it works you can look at the show notes and you find this profile there and yeah

681
00:59:23,800 --> 00:59:32,760
thank you again I'm it for for being a part of the show thank you thank you for inviting me