Azure Data Lake Storage - Simply Explained

Welcome to the M365 FM podcast, where we explore the people, ideas and technologies shaping the future of work. What Azure Data Lake Storage really is. Azure Data Lake Storage sounds like a whole new product to learn, and it's really not. It's the same cheap Azure storage you've probably already heard about with one setting switched on that gives it real folders. That one setting changes how quickly big analytics jobs finish, which matters because the compute that runs those jobs is where most of your money goes. It's also a one way door because once that setting is on, you can't switch it back off. So here's what it actually is, how teams use it and the mistakes that quietly inflate the bill. Most beginners assume they have to learn an entirely new service, and that assumption wastes weeks of their time. Azure Data Lake Storage isn't a separate product you buy, and it isn't a special account type you create from scratch. Microsoft built it as a set of abilities inside ordinary blob storage. The plain storage Azure has offered for years. You unlock those abilities by turning on a single option when the account is created. That option sits in a short list of settings on the screen where you make a new storage account. It's easy to click straight past if you don't know what you're looking at, nothing on that page announces that this tiny toggle is the difference between a bucket of files and a real data lake. You find out later, either because your jobs run fast and cheap, or because they crawl and your bill keeps climbing. To understand why that option matters so much, you first need a clear picture of what a data lake actually is. But why? A data lake isn't just a folder full of files. A data lake is one central place where you keep all of your data. The word all is doing a lot of work in that sentence. It holds structured data, the neat rows and columns you'd find in a spreadsheet or a database table. It holds unstructured data too, like images, PDFs, logs, and raw text files. Instead of forcing everything into a tidy shape, a lake lets data land in whatever form it arrives. That flexibility is the whole point, and it's worth seeing why it matters. Picture a small business that sells online sales. Exports come out of one system. Support tickets come out of another, and product photos sit on someone's laptop. A traditional database would demand that every source be reshaped into strict tables before it accepts a single record. A data lake takes all of it side by side and leaves the reshaping for later when someone actually needs an answer. The name comes from that contrast. A warehouse stores data that's already been cleaned, labeled, and organized like goods sitting on shelves. A lake is closer to a reservoir wide and mostly raw, ready to be filtered. The moment a specific question shows up. The trade off is real, and it's the reason lakes exist at all. When you keep raw data around, you can answer questions later that nobody thought to ask. Today, a warehouse forces you to decide what matters before you store it, and anything you tossed out is gone for good. A lake bets that keeping the messy original is worth more than the tidiness you'd get from cleaning it first. None of this means a lake is a junk drawer because it still needs a shape, and that shape is the folders you'll meet in a moment. What a lake refuses to do is throw away the original, since the original is often the only record of something that actually happened. One detail matters more than all the rest, though. A lake still has to live somewhere. And in Azure that somewhere is a storage account with one option switched on. That option is where the real story starts, and it's also where the first permanent decision gets made. The one setting that turns blob storage into a data lake. Azure Blob Storage is the workhorse underneath all of this. It's cheap object storage, which is a plain way of saying you put files in and pull files out. It doesn't much care what's inside them. What it never had was real folders. And that gap is the whole reason Azure Data Lake storage exists. In a normal blob account, a path like sales twenty twenty six March isn't a folder holding a file. It's one long file name with slashes baked into the middle of it, and the system treats it exactly that way. The storage looks organized to your eyes because the slashes trick you into seeing structure that isn't really there. Object storage was built this way on purpose, since a flat list of names is simple to spread across a huge number of machines, that trick causes quiet problems. The moment you try to do anything at the folder level. If those slashes aren't real folders, then renaming a folder means finding every file that starts with that same path on a large data set. That's millions of files, and a job that should take seconds can stretch into hours. Turning on the hierarchical namespace changes exactly that. It gives the account real directories and subdirectories the same structure your computer has used forever. A folder becomes a thing in its own right and not just text tucked inside a file name. One checkbox flips and the storage starts behaving like a proper file system instead of a bucket with nice looking labels. You'll also see the name Data Lake Storage Gen two in older posts and tutorials. That's just the second version of this same idea, and the current name drops the number. So treat the two as one thing. You choose this option when you create the storage account, and you can also upgrade an account that already holds data. Microsoft handles the switch for you, but the direction only goes one way. You can move from flat to hierarchical and you can't move back. There's a naming quirk worth knowing, so the documentation stops tripping you up. In Microsoft's own guides, the word blob means file, and the word container means file system. They're the same two things wearing different labels depending on which page you happen to be reading. The part to hold on to is that this checkbox is permanent. Once it's on, there's no switching the account back to the flat style it had before. Why real folders change the speed and the bill? The reason Microsoft built Build real folders isn't tidiness, it's speed. When directories are real, renaming or deleting one becomes a single quick change to that directory's entry. There's no list of a million files to walk through because the system updates one thing and the work is done. The same shortcut applies when you delete, so clearing out old data stops being an all day project. The example that makes this land is how analytics tools behave in practice. Spark and hive, two of the engines people lean on most often write their output to a temporary location. When the job finishes, they rename that location on storage. Without real folders, that final rename can take longer than the actual analysis. Did you end up watching a job that already finished its thinking and is now just shuffling names around? That delay isn't only annoying, it's expensive, and the reason is worth sitting with. In a lot of analytics workloads, compute is more than eighty five percent of the total cost. The machines doing the work cost far more than the data sitting still. If a slow rename keeps those machines running an extra hour, you pay for that hour, speed up the storage and the compute finishes sooner so the bill drops. Even though the storage price barely moved, the feature also scales cleanly, which is the part that used to be hard. Older systems traded folder structure for scale because real directories were expensive to track at massive size. A hierarchical namespace here grows in a straight line, so adding more data doesn't slow the directories down. Which brings up the practical question how does data actually move in and out of one of these accounts? How data actually gets in and out. The plumbing is friendlier than it sounds, and this is where the theory turns into something you'd actually touch. Analytics tools reach the lake through a driver called Abfs, short for Azure Blob File System. The service address always ends in dfs dot core dot windows dot net. You rarely type that address yourself, but it's the connection your tools use behind the scenes. The tools themselves are names you'll meet again and again if you stay in this world. As your data factory copies data in from other systems, Databricks and Synapse process and analyze it and power BI reads the results to build reports. If you'd rather move files by hand, Azure Storage Explorer gives you a window to click through, and the command line tool called as copy handles the bulk transfers. You don't need any of those tools just to get started. The Azure Portal PowerShell and the command line can all reach the same storage. You can upload a first file and watch it appear inside a folder before you write a single line of code. Most people start with one folder and a handful of files, and the lake grows from there as more systems feed into it. Permissions work at a finer grain here than in plain storage. You can use Azure's role system, the same one that governs everything else in Azure. You can also set Posix style access control lists that lock down a single folder or even a single file. I'll leave the details there, because the point is simply that a lake can be shared widely without handing every person the keys. When data lands, it lands as a blob in a container, which is the file in a file system idea wearing those two other names again. From there, a lifecycle policy can move older files to a cheaper tier or delete them. After a set time, the lake tidies itself. Instead of waiting for someone to sort folders by hand. Scale is the last piece, and it's generous. Microsoft places no limit on how large the account or any individual file can be. One file can run from a few kilobytes up to hundreds of terabytes, and requests come back at a steady pace. No matter how much you've piled up with the plumbing clear. It helps to see what a real setup actually looks like. What this looks like in the real world, teams rarely dump everything into one giant pile, and the pattern they reach for instead is called the medallion. It's simpler than the name suggests. Raw data lands first in the layer people call bronze cleaned and corrected, data moves into silver, and the polished data that a report can trust ends up in gold. Each layer is just a folder, and data travels forward as it gets better. The gold layer is the one a report actually reads, because it's just a folder of well shaped files, any tool can pull from it without knowing how the raw data arrived. Folder layout carries real weight in a lake, and there are two patterns worth copying for data arriving from devices or sensors. A common layout runs region, then subject, then year, month, day and hour. An aircraft engines readings might sit under UK, then planes, then the specific aircraft, then a date and time for batch jobs that run on files. Teams use an in folder for work waiting to be picked up. They use an outer folder for finished results and a bad folder for anything that fails. The order of those folders isn't arbitrary. Putting region and subject at the top makes it easy to say that these people can see UK data and nothing else. You set that permission once on the folder and everything underneath inherits it. If you flip the order and start with the date, you end up setting permissions over and over on countless tiny folders. File format is the last real world choice, and it quietly decides your bill. Use a columnar format like parquet for data. You read a lot because a query can pull just the columns it needs and skip the rest. Those files are compressed and each one carries its own description of what's inside. Use a row based format like Avro. When you're mostly writing data in a steady stream, such as events flowing in from an app. A good layout and the right format save you money, but a handful of common habits quietly do the opposite, and they show up in almost every beginner's first leg. The mistakes that quietly cost you money. The first mistake, and the one you'll see everywhere, is leaving your data as a mountain of tiny files. Analytics engines spend real time on each file, just listing it, checking permissions and opening it. A thousand small files cost far more attention than one big one. Microsoft's own guidance is to aim for files somewhere between two hundred and fifty six megabytes and one hundred gigabytes. Sweep small files into bigger ones before anything tries to read them. That's a habit worth building from day one, because small files pile up faster than anyone expects. There's a billing detail that catches almost everyone, and it's easy to miss. Read and write operations are billed in four megabyte chunks. A file that holds a few kilobytes still gets charged as though it held four megabytes. When your data is spread across millions of little files, you're paying full price for mostly empty space over and over. If the raw data arrives as a stream of tiny events, the usual fix is to collect them with a streaming tool and write them out in bigger batches. The second trap is turning the hierarchical namespace on when you don't need it. Remember that it's a one way door, so once it's on, you can't return the account to a flat namespace. If you enable it for a backup archive or a folder of images, you've made a permanent change that buys you nothing. Backups and photo libraries don't care about fast folder renames because the data isn't being analyzed. The third is ignoring storage tiers. Azure lets you park data in hot, cool, cold, or archive tiers depending on how often you read it. Old data left sitting in the hot tier is money spent for no reason. Match the tier to how often you actually touch the data, and the same files can cost a fraction as much. Archive is the cheapest place for data you almost never open, but it isn't instant because a file has to be rehydrated before anything can read it. One honest limit is worth keeping in mind, even with the setting on, some blob features are still catching up inside these accounts. If you depend on a specific one, check its current support before you commit. That covers the part you control yourself. For a lot of you, though, a lake won't be something you build alone. The next question is where it fits in Microsoft's bigger picture, where this fits with fabric, power BI and Microsoft three hundred sixty five. If you've spent any time around Microsoft Fabric, you've already met a data lake without knowing it. One lake The storage layer at the heart of fabric is built on Azure Data Lake storage. It keeps tables in open formats called Delta, parquet and Iceberg that any tool can read the same engine. Running a data lake in Azure is the one running underneath fabric the whole time. If your team already lives in fabric, one lake is the lake you'll meet first, and everything you've just learned still applies to it. One lake also borrows the shape you already know. Every fabric tenant gets exactly one one lake and you can't create a second one. The whole organization shares a single lake instead of a scattered set of them inside it, a workspace shows up as a container and the items inside show up as folders. That's the same blob and container pairing from earlier just wearing fabrics clothes. The clever part is how one lake reaches data that lives somewhere else. A feature called shortcuts lets it point at files sitting in Azure Data Lake storage account in Amazon S3 bucket, a Dataverse table, or even an on premises system. It reads them in place without making a copy. Teams keep working in their own spaces and share through shortcuts instead of duplicating gigabytes of data. That idea of one copy shows up again in power by a mode called direct lake reads straight from one lake without importing or copying the data first. It's why fabric keeps talking about a single copy that every tool can share. one. Lake also keeps deleted files for seven days, so an accidental delete isn't the disaster. It would be in a folder with no safety net. There's a genuine trade off between the two paths. One lake is managed for you and aimed at fabric users. A plain Azure Data Lake storage account hands you direct control and works across the wider Azure toolset. They speak the same API, so the choice comes down to convenience against control. What to remember about Azure Data Lake Storage. So here's the short version. Azure Data Lake Storage is blob storage with one setting turned on. And that setting gives it real folders. Those folders make big analytics jobs finish faster, which can lower a bill where compute does most of the spending. It's permanent once enabled, and a pile of tiny files will cost you more than you expect. Hold on to those few things, and the scary looking service turns into something you can reason about. Here's my question for you. If you're planning your first data lake, what's the one piece you're least sure about right now? The folder layout, the cost, or where it fits with fabric? If this made the topic feel less intimidating, subscribe to M365 FM to master the tools and the mindset shaping the future of work.