Welcome to the M365 FM podcast, where we explore the people, ideas and technologies shaping the future of work. Why large language models feel like magic. Even the newest large language models will guess rather than admit they don't know. That's the part of these tools that catches people off guard because they sound so sure of themselves. They are the engine behind ChatGPT copilot, and almost every AI feature you have opened in the last few years. And most of us use them without any idea what is happening underneath. That's exactly what this episode is for, because the idea underneath is simpler than the jargon makes it sound. We're going to walk through what these models are, how they learn, why they slip up, and where they already sit inside. Microsoft three hundred sixty five and Azure. Welcome to the M365 FM podcast, where we explore the people, ideas, and technologies shaping the future of work. What a large language model actually is. A large language model is a computer program trained to guess the next word. It does that over and over again until it has written a complete answer, and every other detail follows from that one idea. The model is built on a design called a transformer. Researchers introduced it in twenty seventeen in a paper titled Attention is All you need, and it turned out to be unusually good at handling long sequences of words. That strength is why the whole field moved so quickly once the design caught on. The word large in the name isn't marketing. These models carry billions of internal settings that get tuned during training, and a model called GPT three arrived in twenty twenty carrying one hundred and seventy five billion of them. It helps to separate two things that get blended together in conversation. The model is the engine, and the app you open is the car built around it. ChatGPT copilot and the AI button inside your favorite tool are all front ends. They send your words to a model and bring the reply back, and the same engine can sit behind several of them. Here is what surprises almost everyone who looks at this closely. Nobody sat down and wrote the answers into the model. There's no giant filing cabinet of facts tucked inside it, and it doesn't search the web in the background while you type, unless a product adds that on separately. That's different from the tool. You probably compare it to. A search engine matches your keywords against pages it has already indexed, and it hands you the sources so you can judge them for yourself. A language model doesn't fetch a page at all, and instead it generates fresh text that fits the shape of your question. That's why it can answer in a way no web page ever wrote. Take a simple sentence like the cat sat on the mat. If I stop at. The cat sat on the. You already have a decent idea of what comes next, and you didn't look anything up to get there. Your brain has read enough English to sense where that sentence wants to go. A language model does something similar, except it works with numbers instead of instinct. So the short version is this. It's a prediction machine rather than a knowledge machine. And that single difference explains almost everything that follows. The very thing that makes these models useful also makes them willing to sound sure when they aren't. And we will come back to that. How a model learns from reading training starts with an enormous pile of text, books, articles, websites, code, and more, adding up to billions or trillions of words before the model can learn anything from that pile. The text gets broken into smaller pieces called tokens. A token can be a whole word, part of a word, or even a single character. That step is called tokenization, and it's how messy human language turns into something a computer can do math on. Microsoft's own documentation uses a clean little example. The sentence I heard a dog bark loudly at a cat splits into nine word tokens. The model gives each unique token an ID number, so that sentence becomes a short list of numbers. Language turns into arithmetic. GPT style models use a finite method called byte pair encoding, which breaks words into smaller chunks. That's why these models can handle typos, slang, and words they have never seen before. Instead of choking on anything outside their original vocabulary. One more thing about scale building a model like this takes so much data, power, and hardware that only a handful of large companies can do it at all. That's why most people meet these models through a product or an interface, rather than running one themselves. Once the text is in that form, the model plays a guessing game across millions of examples. It predicts the next token and then checks what actually came next. From there, it measures how wrong it was and nudges its internal settings so it does better on the next round. Nobody labels those examples by hand, which is the clever part because the text already contains its own answer. The model just hides the next word from itself and tries to recover it. What comes out of that loop isn't a list of rules, the model ends up with a sense of grammar, a feel for which facts travel together, and a knack for how careful reasoning usually sounds when it's written down. Scale matters here, and the reason is easy to picture. Patterns that show up constantly are easy to learn, while patterns that show up once are nearly impossible. Grammar appears in almost every sentence, so models get grammar right almost every time. A one off detail, like the exact date of a small event, might appear a single time in the entire training pile. That's exactly the kind of thing the model has to guess at. How it turns your question into an answer. When you type a prompt, your words go through the same tokenizer the training text did. The model looks at everything in front of it and predicts a value for the next token. Then it picks the most likely one from its vocabulary and adds it to the end of the sequence. From there, it runs the whole thing again with that new token included, and the loop continues until the answer is finished. That loop is the entire generation process, and it explains a lot of what you see in practice. The answer appears one piece at a time, which is why you can watch it being written and why a long answer takes longer than a short one. There's a small detail hiding in there that explains some everyday behavior because the model picks a likely token rather than the single correct one. Two runs of the same prompt can produce different wording. Some products let you dial that randomness up or down with a setting often called temperature, and it's why the same question can give you two different answers on two different days. One limit worth knowing is the context window, which is the largest amount of text the model can hold in view at once. If a document is longer than that window, something has to give and the tool will trim it, summarize it, or ask you to narrow things down. That's also why a model starts every session fresh, because each request is built from whatever fits inside that window. Your own files aren't baked into the model either. Products bridge that gap with a technique called retrieval augmented generation, which pulls relevant information out of your data and places it inside the context window. From there, the model can use it without being retrained. That's the machinery behind the word grounding, and it's the reason a work assistant can answer questions about your own documents at all. Money enters the picture at the same point. Many AI services bill by tokens, counting what you send in and what comes back out, which is why a long prompt and a long answer cost more than a short exchange. None of this asks you to do anything differently, and it just helps to know that the tool is working inside limits the same way you do. Why? It sometimes makes things up. This is the part I promised earlier. The very trick that makes a model helpful is also the reason it will invent an answer when it has none. Researchers have a name for it. A hallucination is when a model produces information that's false or misleading while sounding completely plausible. It isn't lying because lying means knowing the truth and choosing otherwise. The model simply has no reliable way to tell a fact it has seen a thousand times from a fact. It's inventing on the spot. In twenty twenty six, a study published in nature by researchers at OpenAI and Georgia Tech put three popular models to a small test. They asked what the letters stand for. ChatGPT gave one confident answer. Claude gave a different one and gave a third. All three were wrong, and not one of them said it didn't know. That result isn't a fluke, and the researchers explained why it happens. Models get scored on accuracy, and under that kind of scoring, a confident guess beats an honest admission of uncertainty. An abstention is graded as a wrong answer anyway, so the training rewards the guess. The paper puts it well when it says models facing a hard question behave like a student in an exam guessing rather than admitting they don't know. There's a second finding worth holding on to. Facts that appear only once in the training data are the ones a model is most likely to invent, because there's no repeated pattern to learn from. The paper offers a rough illustration if one in five birthday facts shows up just once in the training text, a model should get at least that share of them wrong. Rare details are where the fabrications live. Accuracy isn't the only weak spot sitting here. Models learn from texts that people wrote so they can pick up and repeat the unfair assumptions buried in that text. A confident tone makes those mistakes harder to catch. Grounding helps, and this is where the earlier section pays off. A model that has the right document in front of it has far less room to invent, because the answer is sitting in the context window. That's one reason a work assistant grounded in your own files tends to be more dependable than a bare chat window, with no context at all. None of that removes the risk completely. Grounding lowers the odds of a made up answer, and it doesn't drive them to zero. So the check still belongs in your routine. So the honest limit is this. This isn't a bug waiting for a patch. It comes from the way these models are built and scored and it shows up even in the newest ones. Treat every confident answer as a draft that needs a check, especially when the detail is unusual or specific. Where you already meet these models at work, you have probably been using one of these models for a while without ever calling it that. The interface is almost always the same simple thing, which is a box where you type what you want in ordinary language. That's the entire skill, and it's the same box in every one of these apps. Microsoft lists the everyday features plainly, and the list is longer than most people expect. Word drafts, rewrites, and summarizes documents, while Excel analyzes data, builds formulas, and suggests visuals. PowerPoint creates a slide deck from a prompt and outlook, drafts emails, and summarizes long threads, OneNote drafts, plans, and lists teams, summarizes up to thirty days of meeting content, and pulls out action items and forms, drafts, questions for surveys and polls. Read that list again with the earlier sections in mind, because every item on that list is a next word prediction task wearing a familiar costume. Summarizing a meeting is guessing which sentences matter most. Drafting an email is guessing how a careful professional would phrase the reply. Tidying up a spreadsheet is guessing which values look out of place. If you have ever watched a colleague summarize a two hour meeting in about 10s, that was this. If you have ever asked a tool to clean up a messy column of data and gotten something useful back, that was this too. And none of it required a single line of code. That's what people miss when they hear the phrase artificial intelligence. It sounds like a separate discipline that lives in a research lab. In practice, it's a feature sitting in the programs you already open on a Monday morning. You don't have to go find it. And you don't have to understand the math to get value from it. The barrier to trying this is much lower than the vocabulary around it suggests, and the vocabulary is the only genuinely intimidating part. How Copilot and Azure actually use them. A general model doesn't automatically know anything about your company's own files. Bridging that gap is the job of a product like copilot, and the answer comes down to three things. Working together, Microsoft describes every copilot experience with the same recipe. Large language models for understanding and generating language grounding in web or organizational data and access that's scoped to whoever is signed in grounding is the interesting one because it's what pulls in your real context, like the email thread you're replying to or the document you have open when you type a prompt in Word or teams. Here is roughly what happens. Copilot preprocesses your prompt using grounding and reaches into Microsoft Graph, which is the directory of your organization's content inside your own tenant. The grounded prompt then travels to the language model, and the answer comes back to the app and to you with the data encrypted along the way. If you have been worried about what these tools can see, this next part is the reassurance copilot doesn't get to see the whole company at once. It can only reach data that the signed in user is already allowed to open. It also respects the same access rules and sign in protections your organization already uses, including multifactor authentication. Microsoft also states that prompts and responses aren't used to train the underlying models. There's a governance side to this as well. Admins get controls for switching features on or off, usage reports for seeing how people actually use the tool, and a dedicated AI administrator role that manages copilot without handing out full global admin rights. That's how a company can adopt this without losing the plot on security underneath. Microsoft doesn't lean on one single model for everything. By default, copilot uses a real time router that picks a model based on your prompt. The interface also lets you choose between a fast option for routine questions, and a deeper option for complex ones that need planning and checking. On the developer side, Microsoft Foundry Models is Azure's catalog, holding more than ten thousand models with roughly fifty new ones added every month. Cost arrives in two different shapes, which is worth knowing before anyone hands you a bill. Some Azure models are billed by tokens in and out, while others are billed for the machines you keep running. The per person copilot license works differently as a flat monthly figure. Microsoft currently lists the business add on at twenty one dollars per user per month, paid yearly, with a promotional eighteen dollars running through the end of twenty twenty six. It sits on top of an eligible Microsoft three hundred sixty five plan and the price moves, so check the current listing before you budget for it. Mistakes beginners make with these tools. Knowing how the engine works makes the traps easier to spot, and most of them come from treating a confident answer as a finished one. That's the first mistake and the biggest. A model that sounds certain isn't a model. That's right. And the nature study is about as clean. A proof of that as you will find. Confidence is a style the model produces, not a signal about accuracy, so the fix is straightforward. Verify anything that matters before you lean on it. The second mistake is asking a vague question and then blaming the tool. A prompt that names the audience, the length and the shape of the answer you want gets a far better result than three words in a hope. The model is guessing what you meant. So the more you tell it, the less it has to guess. Then there's the missing check. Using a model to draft something is smart, and shipping that draft without reading it isn't. Treat the first pass as a starting point. You edit never as a finished product with your name on it. Two more traps are about expectations. The model doesn't remember your last conversation unless the product deliberately carries that context forward. It also can't see anything in your organization that you can't already open yourself. Both of those are limits of the design rather than settings you forgot to switch on. There's also an easy one to overlook if you ask about your own project without pointing the tool at the relevant files, you get a generic answer and then wonder why it missed the point. Handed the source material and the quality jumps. Finally, do not assume the biggest model is always the right one. Routine questions don't need the heaviest engine, which is exactly why Microsoft routes prompts between a fast model and a deeper one. Matching the tool to the job is usually cheaper, and it's just as good. What to remember. And one question for you. A large language model is a prediction engine. It learned patterns from an enormous amount of text. It writes one token at a time inside a context window. It gets grounded in your own data through products like copilot. And it can be confidently wrong. That last part is why you check the things that matter. The intimidating part was never the technology. It was the vocabulary. And now you can translate that vocabulary back into plain English the next time somebody drops it into a meeting. So here's my question for you. Which task would you hand to an AI assistant first? And what would it take before you trusted the answer? I would genuinely like to read your answer in the comments if this made the topic feel less intimidating. Subscribe to M365 FM. To master the tools and the mindset shaping the future of work.