Log Analytics - Simply Explained

0:00 Welcome to the M365 FM podcast, where we explore the people, ideas and technologies shaping the future of work. What log analytics actually is? Log analytics is a filing cabinet for everything your systems whisper to each other, every sign in every file open, and every error becomes a record in that cabinet and on its own. One record is nearly useless. Stack millions of them together, and patterns appear like an account signing in from two countries five minutes apart. That's the part that stays hidden, and it's why security teams and developers both end up here. You don't need to be a developer to follow this, because there's a point and click mode for people who have never written a query. Welcome to the M365 FM podcast where we explore the people, ideas, and technologies shaping the future of work.

0:40 A record is one line that says something happened at a particular moment and nothing more. Someone signed in, someone opened a file and a page, took four seconds to load a service, restarted itself at three in the morning. Each line is boring on its own, which is exactly why nobody reads them one at a time. There's a distinction that trips people up, and it's worth getting straight early. Azure monitor watches your systems in two ways. Metrics are the numbers on a dashboard, like a server sitting at eighty percent memory right now. Logs are the detailed story of what happened, like who signed in, what they opened, and what broke right after. Metrics tell you something is wrong, and logs tell you why the scale is worth pausing on. Those records arrive from everywhere, including Azure Resources, other clouds, your own applications and servers sitting in an office closet.

1:22 A single busy service can write thousands of lines an hour, and a whole company can produce millions a day. One failed sign in is a non-event that you'd never notice. Ten thousand failed sign ins in a single hour. Sorted by country is a story, and it's a story you can only read if those records were kept in one place. That's the whole job of log analytics. It gathers scattered records from everywhere and puts them somewhere you can actually search. The honest part is this a log is only as useful as your ability to search it, and a cabinet full of records you never open is just storage. You pay for every month. That cost is real and we'll get to it later. Because how much data you send in and how long you hold on to it, is where this whole thing gets interesting, where all these logs come from. If records are the raw material, something has to go and collect them because nothing arrives on its own.

2:04 On the other side, that's something is usually the Azure Monitor agent. You install it on a virtual machine or a cluster of containers, and it connects back to Azure to ask what it should be watching. The answer comes from a data collection rule, which is really just a set of instructions. You write once and reuse across many machines. It says which events to gather, what shape they should be in, and which table they should land in. If you've ever set up a rule for sorting your email into folders, the idea is the same just aimed at servers instead of messages. Those rules can also clean up data on the way in. They can drop the noise you never look at and reshape what's left, so it fits neatly into a table. The reshaping itself is written in the same query language. You'll meet in a few minutes. Hold on to that thought, because filtering early is the single biggest lever you have over the bill later.

2:45 Not everything lives in Azure, and that's fine if you build your own application. An interface called the Logs Ingestion API lets it send its records straight in and it replaces an older method. Microsoft has since retired. The same agent reaches into other clouds and on premises servers, so a mixed environment isn't a problem if your world is Microsoft three hundred sixty five rather than Azure. You're already generating this data. Every team's message, every SharePoint document you open, and every sign in writes an audit record into what Microsoft calls the Unified Audit Log. Defender adds its own security events on top, so the records exist whether or not anyone is managing them. The real question is whether they land somewhere you can use them, or somewhere they pile up until someone asks a question you can no longer answer.

3:27 Inside a workspace and its tables. All those records need a home. And in Azure, that home is called a workspace. Picture it as the container where everything lands and the tables inside it as the labeled drawers that sort records by type. Sign in records go in one drawer, performance numbers in another, and errors in a third. Azure creates the drawers it needs automatically, and you can add your own for data it doesn't recognize yet. Custom tables are worth a mention because they're. How you bring in data Microsoft has never seen. You describe the shape of your records once, and from then on, anything that matches can flow in and be queried. Like everything else, every record also carries a timestamp the exact moment it happened. That single detail is what makes a question like what changed last night? Answerable because you can always narrow a search to a window of time.

4:09 A small beginner decision here turns into real money. Each table gets assigned a plan, and the plan controls how expensive the data is to store, how fast you can search it, and how long it sticks around. Analytics is for data you watch closely and need back instantly. If you only dig into something when it breaks, that's basic. Auxiliary is for the high volume stuff you mostly keep. For the record, one workspace can mix all three, which means you can treat a noisy log and a critical one very differently in the same building. The price gap between those plans is bigger than you'd expect. The same gigabyte of data costs roughly two dollars thirty cents on analytics, fifty cents on basic, and five cents on auxiliary. That's about forty six times cheaper at the bottom end. And it's the kind of gap that decides whether a project is affordable at all. The catch is that you can't just dump everything into the cheapest drawer.

4:51 The cheap plans are slower to search and aren't built for the real time detection that security teams rely on. Every table is a trade off between speed, cost, and how often you actually look at what's inside it. Get that plan wrong, and you either overpay for data you rarely touch, or slow down a query. You run every morning asking questions with a few lines of KCL. A container full of records is only useful if you can ask it questions, and the language you ask in is called k q l, short for Kusto Query Language. Don't let the name scare you off because it doesn't read like programming. It reads like a short list of steps you'd hand to a very patient assistant. One of those steps in plain English looks like this. Start with the table of sign ins. Keep only the ones that failed and show me the newest ten in KCL.

5:32 That's three or four lines, each one separated by a pipe. And the pipe simply means take what you had and pass it along. The verbs you'll use most are easy to recognize once you see them. There's one to filter, one to sort, one to count, one to group, and one to limit how much comes back. Combine a few of them and you go from a million records to the handful that actually matter if writing anything at all sounds miserable. Microsoft built a second way in simple mode, lets you point and click your way through filters, sorting and grouping without typing a single line, and the results update as you go. The advanced editor sits right next to it so you can graduate whenever you feel ready. Two small things are worth knowing. Kql cares about capital letters and table and column names, so a lowercase letter can break a query for no obvious reason, and a query that searches everything at once runs slowly by design.

6:16 So the habit that pays off is starting from the right table and narrowing down. The reassuring part is that this skill travels. The same language shows up in Microsoft Defender's Advanced hunting. So once you can ask one Microsoft tool a question, you can ask several of them the same way. It's the same engine underneath, which is why the muscle memory carries over so cleanly. When a query finally gives you something useful, you don't have to keep it to yourself. You can pin the result to a dashboard, save it for later, or hand it to a colleague as a starting point. What teams actually do with it? In practice, people use it to answer questions that are already costing them sleep. Say a website gets slow every night around two in the morning and nobody knows why. Instead of guessing, you pull the records from that hour, group them by the thing that keeps failing and the culprit usually falls out in minutes, or an important file disappears from a shared site and someone needs to know who touched it.

7:02 And when the audit records hold the answer along with the account, the time and the device security teams use the same data for a different question. They watch for a sign in from a country nobody on the team has ever visited, or an account that suddenly starts downloading far more than usual. Microsoft Sentinel, the security product built on top of this same workspace, watches for those patterns around the clock and raises an alert when something looks wrong. Analysts can then hunt through the raw records themselves using the same kql we just talked about. Operations teams lean on it, too. They want to know whether a service keeps restarting, whether memory is creeping up week after week, or whether a deployment broke something at four in the morning. Microsoft ships ready made views called insights that answer a lot of those questions before you write a single query.

7:43 There's also a smarter layer for the cases nobody thought to write. A rule for built in anomaly detection can flag a pattern that simply looks wrong compared to the usual rhythm, which is how you catch the problem you didn't know to look for. If your business lives in dashboards, query results can be exported straight into power BI, so people who never open Azure can still see what's going on. There are also alerts, which are just saved, queries that check themselves on a schedule and shout when a pattern shows up. That's the payoff, and it's real. Every one of those questions depends on records that were collected and kept. Which brings us to the part nobody enjoys thinking about where the money actually goes. Log analytics isn't free, and pretending otherwise would waste your time. The model is simple, though. You pay for the volume of data you collect and for how long you keep it.

8:24 And creating a workspace costs nothing. Every gigabyte you send in shows up on that bill. Whether or not anyone ever looks at it, that volume is where the surprises live. A chatty source like a verbose application log, or an agent that reports every heartbeat can produce an enormous amount of data without anyone noticing. Multiply that across a few hundred machines and the bill can grow faster than the systems it's watching. Two facts here run against instinct. The size you're billed for is usually smaller than the raw event that produced it by roughly a quarter on average, because Microsoft trims the overhead before charging you. A handful of tables, including Heartbeat and Azure activity, are also free to send data into, so not every gigabyte carries the same price tag. Prices are set by region two, so the same data can cost different amounts depending on where your workspace lives.

9:05 That's a detail worth checking before you commit to a big rollout. The trap is a natural instinct. When something feels uncertain, Collecting more data feels safer. But most of what you collect is never looked at, and you pay to store every last bit of it. Teams that treat the workspace like a junk drawer end up paying for a junk drawer. The right fix is the one we mentioned earlier, which is to filter and reshape data before it's stored. If a log line is noise, you'll never query, drop it at the door and never pay for it. Microsoft also gives you a daily cap, a hard limit that stops collection once you hit a threshold. It sounds like a safety net, and it is. But Microsoft's own guidance warns against leaning on it. When the cap Trips collection stops completely and you lose visibility into your systems for the rest of the day.

9:45 That's a worse outcome than a bill if something breaks overnight. How long your data sticks around cost and time are tied together, so it's worth understanding how long a record actually lives. Every table gives you a window of fast interactive retention where you can query data instantly on analytics data. That window is thirty one days out of the box, and it stretches to ninety days if Microsoft Sentinel is enabled on the workspace after that window closes, the data doesn't vanish. It moves into long term storage where it sits cheaply, and you pull it back only when you need it. Keeping data queryable costs about ten cents per gigabyte per month, and long term storage runs about two cents per gigabyte per month. You can hold onto data for up to twelve years, which is the kind of number that matters if you answer to an auditor.

10:26 Getting old data back isn't instant, and that's the trade you're making. You run a search job that reaches into the archive and pulls the records you asked for into a fresh table where they can be queried like anything else. The Microsoft three. Sixty five clock is different. It's unified. Audit log keeps records for one hundred and eighty days by default. That used to be ninety days and it changed in October of twenty twenty three. Defenders advanced hunting, meanwhile, only reaches back about thirty days of raw security data, so you can have three different clocks running at once or ticking for records that feel like the same thing. The practical move is to decide table by table how long each kind of record needs to survive. Compliance data might need years, while a chatty debug log might only need a week. The honest limit is that a default isn't forever, and finding something later only works if you set the retention before you need it.

11:11 By the time you're hunting for a record from eight months ago. The decision that would have kept it is long gone. How it fits into Microsoft three hundred sixty five and Azure. Zoom out for a second, because the pieces fit together more neatly than they first appear, as your monitor is the umbrella that watches everything running in Azure, and it splits that job in two. One half is metrics. The fast numbers. The other half is logs, which is log analytics. So when someone says log analytics, they're really naming the log half of a bigger monitoring system. The connection worth remembering is Microsoft Sentinel. It looks like a separate product with its own name and its own screens, but its workspace is the same. Log analytics workspace just switched on for security. The same place that holds your application. Logs can also hold your security signals, and analysts query both with the same language.

11:53 Microsoft three hundred sixty five has its own version of this, called the Unified Audit Log. It's where your teams, SharePoint and sign in activity lives. When a team needs to keep that data longer than the default window, it can be piped into a security tool through something called the office three hundred sixty five Management Activity API. There's a trade off hiding in all of this. Putting security data in the same workspace as everything else can save money because a bigger volume can qualify for a discount. That discount is called a commitment tier, and it works like a bulk deal. Commit to a daily amount starting at one hundred gigabytes, and Microsoft will knock up to about thirty percent off the per gigabyte price. Sentinel enabled data is built differently, though, which is why a lot of teams end up splitting security and operations into separate workspaces.

12:35 The reason all of this matters is that the concept you learned today isn't confined to one product, the same workspace, the same tables, and the same query language keep showing up across Microsoft's tools, and learning them once pays off everywhere. So the next time a Microsoft tool asks you to point something at a workspace, you'll know exactly what it's asking for. You're not learning four separate products. You're learning one idea that keeps reappearing, the honest limits. And one question for you before we finish the limits deserve their due, because understanding a tool's limits is what keeps you out of trouble. Log analytics can only show you what was collected if a system never sent its records in, no query will invent them, and no amount of cleverness will find what was never written down. It's also not a place to hoard everything for free, because both volume and time carry a price.

13:17 It can even show you the same event twice because the pipeline retrieves to make sure data arrives safely. A duplicate is usually the system being careful rather than your query being wrong. Two more limits are worth holding on to. Hunting through every table at once is slow, which is why knowing where to start is a real skill. The Daily Cap, meanwhile, is a hard stop and not a gentle break. Hit it and collection stops for the remainder of the day, which means an overnight problem could go completely unseen. A good first step is to start small. Pick one system that matters, turn on collection for it, and watch what actually shows up before you open the floodgates everywhere else. The short version is simple. A log is a record of something that happened. The workspace is where those records are kept and tables sort them by type. Kql is how you ask questions, and you don't need to write any code to use it.

14:00 The two decisions that actually shape your bill and your safety net are what you choose to collect and how long you choose to keep it. That leaves one question worth sitting with somewhere in your world, a system is already recording everything it does. So who's actually reading it? If this made the topic feel less intimidating, subscribe to M365 FM to master the tools and the mindset shaping the future of work.