Azure Databricks - Simply Explained

0:00 Apache Spark started as a research project at a university in two thousand and nine, and it grew into the engine. Most big data platforms still run on today. Azure Databricks is how Microsoft and the people who built spark turned that engine into a service you can spin up inside Azure. Welcome to the M365 FM podcast, where we explore the people, ideas, and technologies shaping the future of work. By the end of this episode, you'll understand what a lakehouse really is, what notebooks and clusters do, and when Databricks makes more sense than fabric or synapse. There's a detail in the middle of this episode that surprises almost everyone, and it's the reason any of this can be trusted at all. What Azure Databricks actually is. Let me start with the simplest version. Azure Databricks is a managed service inside Azure, where you point code at your data and get answers, reports, and machine learning models out the other side.

0:46 Everything else in this episode is detail around that one sentence. The name comes from a company. Databricks was founded in twenty thirteen by a group of researchers who created Apache Spark, the engine we just mentioned. Spark began at the University of California, Berkeley, and the team open sourced it in twenty ten, which is how a university research project ended up running inside a huge number of data platforms. Microsoft and Databricks then turned that engine into a first party service on Azure, and general availability was announced in March twenty eighteen. First party matters in practice because the service gets built through Azure, it's classic computing power runs inside your own Azure subscription, and your existing Microsoft identities work without a second directory to manage. One detail here trips most people up.

1:28 Azure Databricks isn't a database, and it doesn't hold your data in a box. It owns your files, sit in a storage account you control, and Databricks supplies the services that read transform and protect them. Hold on to that idea because it explains almost every design decision. We're about to cover. Five parts make up the platform, starting with a lakehouse that organizes data in open files. Then comes a processing engine called Spark Notebooks, where you write your code. Clusters that provide the computing power and a governance layer that decides who sees what. There's no reason an intelligent beginner can't hold all five in their head at once. And the interesting part is how they fit together, which starts with a problem most companies already have. Why the lakehouse idea exists. Data warehouses have served business reporting for about thirty years.

2:10 The design goal was simple keep structured numbers in a shape that makes reports fast. That worked and it still works. What it didn't do well was machine learning, because Microsoft's own documentation points out that many warehouses rely on proprietary formats, and those formats limit what other tools can do with the data. Data lakes went the other direction. Instead of a controlled format, you drop files into cheap cloud storage in whatever shape they arrive. Microsoft's documentation is blunt about the trade off here, too, describing a lake as a cheap place for any kind of data that was never validated enough for reporting. Put those two together and you get the real problem, which isn't technical at all. You end up with the same data copied into two places, maintained by two teams, described in two slightly different ways. Finance reports a number and sales reports a different one.

2:51 Nobody is lying and nobody can tell you which one is right because the copies drifted apart months ago. A lakehouse is the attempt to stop the copying. The idea is to keep one set of files in open formats and put a metadata layer over the top. Microsoft describes the result as open access to data in standard formats, index structures built for machine learning, and low query latency with high reliability for reporting. In other words, the warehouse experience without a second copy of everything. The phrase open formats is doing a lot of work there. Files stored as parquet, a widely readable column format, can be read by many different engines, which lets the same files feed a dashboard, a machine learning model, and a stream of live events without exporting anything. That's also why these platforms can share data, which matters when we get to fabric and synapse.

3:33 Inside a lakehouse, data usually moves through stages. It lands raw, sometimes with missing values and odd formatting, and a second stage cleans and standardizes it. A third stage then shapes it for reporting and models. Databricks calls this pattern the medallion architecture and names the layers bronze, silver and gold. You don't need the colors in your head. What matters is that raw data stays available while clean data serves the business. One fair caveat belongs right here. Databricks published a paper in twenty twenty one arguing the lakehouse will eventually replace the warehouse, and the authors run Databricks, so treat that as a well-argued position rather than settled fact. Plenty of companies still run warehouses happily, and some run both. Apache Spark in plain words. Spark began as a research project at UC Berkeley in two thousand and nine and was open sourced in early twenty ten.

4:17 It moved to the Apache Software Foundation in twenty thirteen, which is a nonprofit that hosts open source projects, and it has been maintained in public ever since. The job spark does is straightforward to describe. When you have more data than one computer can handle, you split the work across many computers, then combine the results. Spark is the software that manages that splitting and combining for you. Microsoft sums it up neatly in its own documentation, saying spark is the technology powering compute clusters and SQL warehouses in Azure Databricks, and that the engine scales while your storage stays separate. That separation matters more than it sounds. Your files don't move into the engine, so you can resize the computing power without touching the data. When a big job needs more speed, you add machines, and once the job finishes, you shut them down again.

4:58 The data sits exactly where it was. Inside a running cluster, there are two roles. The driver node coordinates the plan and tracks progress. Worker nodes carry out the actual computation. Microsoft's documentation describes clusters as allocating their driver and worker nodes, and that's the whole mental model you need. Spark also speaks several languages. You can write Python, R, Scala, or SQL against the same engine, and Python is the usual starting point because it reads almost like plain instructions. That flexibility is a big part of why spark spread so far, and why job ads keep asking for it. One more piece is worth knowing by name. Photon is a Databricks engine that processes data in columnar batches instead of row by row, and it's the default for SQL warehouses. Microsoft states that photon delivers up to five times better price to performance than other cloud data warehouses on an industry benchmark called Tpc-ds.

5:43 That number comes from the vendor on a benchmark they picked, so treat it as a vendor benchmark rather than a neutral study. Notebooks and where you actually work. A notebook is a web page where you write code in chunks and see the result directly underneath each chunk. Microsoft's documentation calls it an interface for data science and machine learning work that holds runnable commands, visualizations, and narrative text side by side. That mixture is the point. The code sits next to the chart it produced, and your own notes sit next to both. Six months later, a notebook records both what you ran and what you were thinking at the time, which is more than a folder of scripts will tell you. The language question surprises people too. The same notebook can run SQL, Python, R, and Scala, each in its own execution context.

6:24 You can query a table in SQL, drop into Python when SQL gets awkward, and plot the result, all without opening another tool. For a beginner, notebooks are the lowest friction entry point into the platform. There's nothing to install on your laptop and no local database to wrestle with, so you open a browser, attach some computing power, and start typing. Version control behaves the way you'd hope. Databricks supports git folders, which keep a folder of notebooks synced to a remote repository, so notebooks get the same review and history treatment as the rest of your code. One limit is worth stating plainly notebooks are built for interactive work, the kind where you run something, look at it, and adjust repeatable work that has to run every night without you watching is scheduled as a job instead, and jobs get their own compute.

7:05 Which brings us to the machines. Because that's where money enters the conversation, clusters the machines that do the heavy lifting. A cluster, in Microsoft's words, is a set of computing resources and settings that you run notebooks and jobs on. That definition is worth keeping plain because a cluster isn't a data store and it isn't a permanent home. So think of it as rented machinery that appears when you need it. You'll meet two kinds. An all purpose cluster is your working cluster, the one you create, share with teammates for interactive analysis and stop and restart whenever you like. A job cluster is created automatically by the scheduler for one workload and shuts itself down when that workload finishes and you can't restart it afterward. The practical consequence of that split is worth learning early. All purpose clusters are for exploring and asking questions.

7:46 Job clusters are for production. The work that runs at two in the morning while everyone sleeps and then disappears before breakfast, startup time gets its own feature. A pool is a set of idle machines that are already warmed up and waiting, so a cluster can grab them instead of waiting for new ones to be created. If you've ever watched a progress bar for six minutes before doing a single minute of work, you understand the appeal. Sizing follows the pattern you're probably imagining already. You pick a machine type, decide how many workers you want, and optionally let auto scaling add and remove workers as the work changes. Databricks also offers serverless computing where the platform manages the machines and you skip the sizing conversation altogether. The question every beginner asks next is about cost, and the answer starts with a billing unit.

8:27 Databricks builds in units called dbus, measured per hour and based on the machine type you chose. With classic computing, the virtual machines themselves run inside your Azure subscription, so two costs work together the Azure infrastructure and Databricks charge on top serverless computing runs in Databricks managed environment instead, with a simpler shape that leads to the classic beginner mistake. And it isn't a technical one. An interactive cluster left running while you go to lunch keeps counting hours. The meter doesn't care that you stop thinking terminating a cluster you're not using is the easiest way to keep a first bill sane. So the machinery is covered and it's fast, but speed on top of unreliable files only gets you to the wrong answer faster, which is exactly the problem. The next piece was built to solve Delta Lake.

9:07 Why your table doesn't fall apart. The problem Delta Lake solves is easy to miss because the wording sounds harmless. Your files live in cloud storage, which is essentially an enormous folder system. If two processes write to that folder at the same time, or if one process dies halfway through a write, a reader can see a mess. A report might show half a batch of orders, a column full of missing values, or rows that look entirely real and aren't. Delta Lake fixes that by extending parquet files with a transaction log, and in Microsoft's description, that makes it an optimized storage layer with Acid transactions and scalable metadata handling. Acid is worth unpacking because it's the term that makes everything else click. It means an operation either completes entirely or not at all, so readers never catch a change in progress. The transaction log does the real work here, and this is the detail the opening promised.

9:51 Instead of guessing at the state of a folder, the format writes down every change in order, and the current table is whatever you get by reading that list. Each write creates a new version. The payoffs are everyday things. You can query what a table looked like yesterday, which is called time travel and recover from a bad load by pointing at an earlier version instead of restoring a backup. You can also keep a report open while new data streams in. Because the report sees a complete version and never a half finished one. Delta also checks your data on the way in. It enforces the schema, which means it rejects rows that don't match the expected shape, and it supports schema evolution. So you can add a column without rewriting every pipeline downstream. Microsoft notes that Delta was built alongside Spark's streaming model, which is why the same data can serve both batch processing and live data arriving in real time.

10:36 Two more facts make it click. First, delta is the default, so unless you say otherwise, every table you create in Azure Databricks is a delta table and you get these guarantees without choosing anything. Second, and this is the surprise worth sitting with. A delta table isn't a special object locked inside the platform. It's a directory of files in your own storage, plus metadata registered in a catalog. Databricks gives you the engine, the format, and the guardrails. Your storage keeps the data. That last point is also why the format is open source, and why other engines can read it, which means the same delta files can be shared between systems. Hold that thought. Workspaces, accounts and unity catalog. Three names are left, and they're the ones that sound the most corporate. Take them in order of size. Smallest first. A workspace is where your team works.

11:17 Microsoft calls it a deployment in the cloud, and it holds folders of notebooks, dashboards, and experiments. If you're thinking of it as the front door to the platform, you've got the idea and it sits above the workspace, as one account can hold several workspaces and several governance containers. And it's where an organization manages people billing and the creation of new workspaces across regions. A small company might run one workspace. A large one might run a dozen one per region or one per department. Unity catalog is the governance layer, and it answers three questions that decide whether a data platform is trustworthy. Who can see which table, where did this data come from and who touched it? According to Microsoft, it provides centralized access control, auditing, lineage, and discovery across workspaces.

11:58 In plain words, that means permissions. A history of changes, a map of where each table came from, and one place to look for what exists. The naming follows a simple three level pattern with a catalog, then a schema inside it, then the object. Think of a filing cabinet, then a drawer, then a folder inside the drawer. It sounds bureaucratic until you remember that the same governance layer can serve many workspaces in a region, which is what makes a table look identical. No matter which workspace you open, identities are handled in a familiar way to people get user accounts. Automation gets service. Principals and groups carry access so nobody grants permissions one person at a time. One modernization node without the news framing Databricks documents the older habit of storing data in dbfs paths as a deprecated pattern, and recommends managing access through Unity Catalog instead.

12:40 Practical translation the catalog is the path forward, so learn that one that covers all five parts, and it leaves one question standing, which is when to reach for this platform instead of the other two Microsoft data options. When to use it Databricks fabric or Synapse. This is the question almost everyone arrives with, so let's handle it honestly. Microsoft sells three overlapping data platforms, and there's no single winner. You might expect this part to be about features and speeds, but it's really about where the work happens and what your team actually produces. Start with Microsoft Fabric. Microsoft describes it as an analytics platform covering the entire journey from ingestion and transformation to real time streaming, analytics and reporting. It's software as a service, which means no infrastructure to size or manage.

13:21 And one shared storage layer called One Lake sits beneath every workload, from data engineering to power BI. The practical shape of fabric matters more than the feature list. One lake sits on Azure Data Lake Storage. Every workspace is a folder inside it, and you don't even need an Azure account to use it. You buy capacity, which is a pool of computing power sized by a level. Microsoft calls a SKU, and Azure builds it per second with per user licenses for the people doing the work. Then there's Azure Synapse Analytics. Microsoft calls it an enterprise analytics service that brings SQL spark and data pipelines together in one place, with ingestion from more than ninety sources using the same engine as Azure Data Factory. Its ethical side comes in two flavors a dedicated pool reserves processing power, so performance stays predictable, while a serverless endpoint stays available for bursts and one off queries.

14:09 Databricks sits at the engineering end of this group, where fabric hands you managed experiences. Databricks hands you the machinery. You get explicit control over clusters and machine types. A runtime built for machine learning with graphics processors available, support for streaming data and unity catalog governing data across workspaces. That ownership is the appeal, and it's also the responsibility. So here's how to choose if the deliverable is dashboards and your team already lives in power BI. Fabric is the shortest path from data to a report. If you already have SQL warehouses and pipelines built in synapse, keeping them running and adding fabric alongside is a reasonable answer. If the work is large scale engineering, machine learning, or streaming data, Databricks gives you the most control over the engine and the compute.

14:50 The more useful answer is that you often don't have to choose at all. Microsoft documents shortcuts in one leg that point at data living elsewhere, including Azure Data Lake Storage, Amazon S3, and Google Cloud Storage. With no copying and no migration, a shortcut can even point at a mirrored Azure Databricks catalog, and if those files are in Delta format, the Lakehouse recognizes them as tables and keeps the schema in sync on its own. So a real architecture often runs both. Engineering happens in Databricks, where the heavy processing and machine learning work lives and reporting happens in fabric, where power BI users already work. Both sides read the same open files governed from two directions and nobody has to lose cost. Shapes follow the same logic, and rates change often enough that numbers would mislead. Databricks charges consumption per hour, plus the virtual machines underneath fabric charges for capacity, size plus users, and synapse charges reserved pool power plus serverless queries so the shape of your workload decides which meter behaves better.

15:42 If you remember nothing else, remember the question worth asking. Ask where the work happens and ask where the reporting happens. The recap. And one question for you. Here's the short version. In one chain your files stay in your own storage, and Delta Lake keeps those files trustworthy. With a transaction log, spark processes them across many machines. Notebooks are where you write your code. Clusters are the computing power you rent by the hour, and unity catalog decides who sees what that chain is. Azure Databricks. So I'll leave you with one question in your own work where does data get stuck today? Is it at the capture, in the cleaning, in the trust, or in somebody finding it at all? Answer that honestly and you'll understand your own data problem better than most meetings ever do. If you want more explanations like this one, subscribe to M365.