iFAKT - Microsoft Fabric for Manufacturing - One Source of Truth From ERP, MES and the Shop Floor 00:00:00 Speaker: Imagine your business system marks an order complete while the shop floor still has work to finish and quality hasn't cleared the batch for shipment, your planner promises delivery tomorrow. That promise rests on the wrong meaning of done. Moving those records into Microsoft Fabric gives you shared storage, but shared meaning takes more work because each record describes a different part of the order. Let's follow that order from its source systems to a production decision. I want to show where the answers split before we ask how to bring them together. For this hypothetical plant, picture a mix of older machines and newer equipment with separate systems for planning and execution, and spreadsheets used between shifts. Nothing about that setup is unusual. Some machines send production signals through an existing connection, while others rely on operators to record progress at a terminal near the work area. The planner starts with the business system shift supervisors work closer to the execution records, and the handover spreadsheet carries notes that neither system captures well. Our order covers a batch of machined parts. The parts move through several operations before inspection, and the customer expects the full quantity rather than whatever happens to be ready when the truck arrives. Start with enterprise resource planning or ERP for this order. It holds the requested quantity, the material requirements, the planned dates, and the transactions posted against production. As the work moves through the plant, those posted transactions carry a specific meaning. A production receipt might record that parts entered inventory, while an order status reflects the business rules and posting steps configured for that plant and process, the planner reads a completed status and sees no remaining business transaction. That's a reasonable place to start, but the status alone doesn't explain whether every physical step has finished, or whether someone still needs to clear the parts for shipment. Now move closer to the machines. The manufacturing execution system, or Mezz Records, which operations have started, where the work is assigned and the production counts reported during each stage of the route. In our example, machining has reached its expected count. The next shift still needs to finish a remaining operation on part of the batch, and that work appears in the execution records rather than the planner's usual order report. A supervisor checks the handover spreadsheet before assigning the remaining work. The note explains which containers need attention and where the operator left them. Details that help the next shift act without walking the whole floor. That spreadsheet fills a real gap. It connects the recorded progress to the practical state of the work, including the parts waiting beside a machine rather than moving to the next step. Quality holds another part of the answer. An inspection record links a whole to the batch, with a result still pending and no authorized release recorded for the affected parts. At this point, the quality team isn't answering whether machining reached its target. They're answering whether the batch meets the release conditions, which means a completed production count can sit beside an open quality hold without either record being wrong. Then there's the machine data. Internet of things or IoT signals might show a machine stopped during the shift, followed by a restart and another stretch of production before the operator logs out. That tells you something about the equipment without the work assignment and the operator's explanation, though, the stop doesn't tell the planner which parts need more work or whether the delivery date needs to change. Notice what each person can reasonably believe. The planner sees a completed business status, the supervisor sees remaining work, and the quality engineer sees a batch that hasn't yet cleared its release conditions. Nobody needs to invent a number for this conflict to happen. Each person can quote a valid record and still give a different answer. When someone asks whether the order is done. I'd start by asking what question each record answers the order quantity describes demand. The execution count describes progress through a particular operation, and the inspection record describes the quality decision for the batch. It's the same order throughout the plant. What changes is the question behind each record? And that difference explains why collecting all the records still leaves the planner with a decision to make. I'd give one source of truth a narrower meaning. It's a set of facts you can trust for a stated purpose, with agreed definitions, and someone who takes responsibility when those definitions need a decision. The purpose changes what you need to know. Answering whether an order can ship requires different facts, from answering whether production met its target, even when both questions refer to exactly the same parts. So trust needs a boundary. You should be able to name the question of fact answers, explain what that fact includes and identify who owns that meaning within the business. That owner needs authority over the definition. If production and logistics disagree about which quantity counts as ready, a data engineer can record their rules but can't settle the business decision on their behalf. Ownership belongs with the people accountable for the work. Production can own the definition of completed operations, while quality owns the conditions for release and the person authorized to approve that release for shipment. This also changes how you choose the authoritative source. Rather than declaring one application correct about everything, you decide which system holds the approved record for each business fact. Under your agreed operating process, order demand might belong to ERP. Execution status might belong to Mezz, while the approved inspection result belongs to the quality system, and the observed machine state comes from the equipment data path, those choices need explicit limits. A machine signal can establish that equipment stopped, but assigning that stop to an order or deciding its effect on delivery requires facts beyond the signal itself. Authority also needs a rule for disagreement. If two systems carry different requested quantities, the agreed owner decides which record governs demand and how an approved change becomes part of the business record. Consider a small extension to our hypothetical order. Suppose the plan calls for one hundred parts production reports one hundred and five and quality releases ninety, leaving the planner with several quantities that must remain distinct. Each quantity supports a different decision. The planned quantity establishes the target. The produced quantity records output under its agreed definition, and the released quantity establishes what quality has approved within the stated scope. Now the names need some care. Does produced mean output from the last operation, or does it include parts that still need work before they count toward the finished order quantity? Release needs an equally clear definition. A release for one operation might permit parts to move into the next process without approving them for shipment. So the measure must state which permission it represents. You don't resolve this by picking the largest number. You keep each quantity attached to its meaning. Then use the facts required by the decision rather than asking one total to represent demand, progress and permission. A shared analytical view brings those facts together. It lets the planner ask a business question across the systems, while preserving the separate authority of the people and processes that create each fact. The operational systems still perform their own jobs. ERP continues to manage business transactions, M.S. continues to support execution and quality. Continues to record inspection decisions through the workflows the plant has approved for that work. Shared meaning doesn't require one physical database. You can maintain separate operational records and still establish a common analytical account of the order, provided the agreed definition survive the move between those systems. That's the role I'd give fabric here. It can support the shared analytical view while your organization supplies the definitions, assigns authority, and decides which facts permit someone to act on an order. I wouldn't begin by debating where every table should live. I'd first ask the planner, production lead and quality owner to agree on the facts required for shipment and who can defend each definition. That agreement gives the platform work a clear purpose. Before choosing how to bring records together, you can state what the combined answer must mean and who resolves the dispute when the records don't support it. With that purpose agreed, I'd trace the records for our order, ask each team to find the actual records they use, including the files and local notes that don't appear in the central application inventory. Start with a practical source inventory for each source. Record what it contributes to the shipment question, who maintains it, and how you can obtain the records without disrupting its daily work. Application names alone won't give you enough detail within the business system. The order header, material movements and status history may live in separate structures, each with its own update behavior and access limits. The execution records need the same close inspection. Find out whether an operation record describes its current condition, a reported completion, or a sequence of events that someone must interpret to establish progress over time. That brings us to grain. Grain means what one record describes, and you need to state it plainly before comparing records from systems that capture work at different levels of detail. Take an inspection linked to our batch. One row might describe a sample one measured characteristic on that sample, or the inspection decision covering a larger quantity of parts under the plant's quality process. Those records answer different questions. If you count measurement rows as inspected parts, a part checked for several characteristics can appear to represent several parts. Even though the source contains no duplicate measurements, sensor records need equally plain descriptions. A reading might represent a value sampled at a moment, while a state change record appears only when the equipment changes from one condition to another during production, write that distinction beside the source. Someone reviewing the inventory should understand whether they're receiving one order, one operation, one reading, one inspection result, or one status change without opening the application themselves. Next, check how you can access each source. The approved route might involve an application interface, a database view, or a managed file export, and the source owner needs to confirm which route they support. Don't mistake possible access for approved access. A database account that can read production tables doesn't establish whether those tables provide a supported interface, or whether the queries could interfere with people recording work. Ask how often the underlying records change, then ask when those changes become available through the approved access route. Because an export produced once per shift can't provide every update as it occurs. Source load belongs in this conversation. Only the team responsible for the application should help establish acceptable reading windows and volumes, especially when the same database supports operators who need quick responses during production. Check the history you can actually retrieve. A current status table might replace yesterday's value, while another source retains individual changes, so the available history may differ even within a single application. Serving the plant retention deserves the direct question. Find out how long readings, inspection details, and execution events remain available, and whether older records move somewhere that requires a different request or access route. Name a contact for each source. You need someone who understands the records and someone who can approve access. Even when those responsibilities sit with different teams or with an external system supplier. Then spend time with the handover spreadsheet. Ask which columns people update, where those values come from, and which decisions depend on. Notes that nobody enters into the formal systems during a normal shift. There's often useful process evidence in those columns. A container location, a disputed quantity, or a note about unfinished work can reveal an information need that the current applications don't support well enough for the next person. Treat the spreadsheet as something to investigate, not blame. Check whether it holds original observations or copied values, how people share it, and whether the next shift can tell which version reflects the latest changes. The inventory should leave fewer unanswered questions for each source. You now know what a record means, where it comes from, who supports it, and what limits affect its availability for the shipment decision. That gives you a basis for explicit agreements. The next step is to turn those findings into clear expectations for the incoming data, rather than leaving each person who builds a connection to. Guess I'd put those expectations into a data contract. That's an agreement between the team supplying records and the team using them, written clearly enough that a change of shift doesn't change how anyone interprets the delivery. The contract belongs to both teams. Your source owner confirms what the application can supply, while the receiving team agrees which conditions the records must meet before they enter the shared analytical view. For our order, start with the fields you'll accept. Specify whether an order reference contains text, whether leading zeros matter, and whether a quantity records individual parts or another unit that needs an explicit conversion. A field name won't settle those questions. A column called quantity needs a definition, a unit, and rules for whether an empty value means unknown, unavailable, or invalid. For this particular record type, equipment references need an agreed scope. If two plants both use press one. The contract must carry enough plant and source information to distinguish them without asking a report author to guess. Local names can stay local. Keep the equipment code operators recognize, but require a maintained mapping to the shared reference used when that equipment appears alongside records from other systems in fabric order references need the same treatment. If the quality application adds a suffix to the production order number, document the matching rule and name who approves exceptions when that rule doesn't identify one order delivery rules should describe what the receiver can expect. Agree when a file or message is due, how the sender identifies a complete delivery, and how the receiver learns that nothing changed rather than nothing arrived. Silence needs a defined meaning for a delivery covering our order. The receiving team should know whether an absent record means no update, a delayed submission or an incomplete delivery that needs attention. Changes need an agreed route to a schema is the structure of the incoming records, so adding a column or changing its type needs rules that both the sender and receiver can follow. Some changes can pass without stopping the flow. An optional comment field might be safe to add, while changing an order reference from text to a number could remove leading zeros and break existing matches. Required fields deserve a firmer response. If the batch reference disappears from a quality release record, the receiver shouldn't infer the batch from whichever order happens to look. Most likely that day corrections also need a clear signal. Suppose an operator corrects a reported quantity after the shift ends. The contract must explain whether the new record replaces the earlier value or adjusts it. Those are different instructions for the receiving team. Without that distinction, a corrected quantity can become extra output simply because the pipeline treats every incoming transaction as something to add to the running total. Deletion needs its own rule. When a source removes an erroneous record, the receiver needs an explicit deletion notice or an agreed comparison process. Rather than assuming absence always means cancellation, then define acceptance before reports consume anything. Check required references. Permitted units and allowed values against the contract, with separate decisions for faults that block a record and warnings that permit limited use. Rejected records need somewhere to go retain the failed record with its rejection reason. Notify the named owner and provide a way to submit the correction without quietly losing the original problem. Don't let rejection look like zero production. If the incoming records for our order fail acceptance, the downstream answer should carry an incomplete data condition rather than present an empty result as a confirmed business fact. Keep the contract practical enough to test. Both teams should review sample deliveries containing missing fields, corrected transactions and deletions, then confirm that the receiving process responds in the way they've actually agreed. A named owner signs off on changes. That includes agreeing what happens when records arrive late or out of sequence, because a written delivery promise still depends on the connection carrying those records from the plant. That connection has limits of its own. Before we depend on the contract for a shipment decision, we need to check what the shop floor can safely deliver when its network or equipment connection drops. A lost cloud connection shouldn't stop a machine from finishing its cycle. For our hypothetical plant, that's a design requirement. The equipment must keep operating safely, even when the analytical systems can't receive another reading from the floor. Operational technology or OT runs and monitors the physical process. Information technology or it supports business applications and data services, where a delayed response usually creates a different kind of problem from a delayed control signal. Both teams care about availability, but their timing needs differ. A machine controller may need to respond within a tightly bounded interval, while a shipment analysis can often tolerate some delay if the planner knows the information's age. The cost of failure changes the design. Losing access to an analytical report can delay a meeting, while disrupting a controller connection could interrupt a process, damage work, or create a hazard, depending on the equipment. I'd keep fabric outside that control dependency. It should receive information through an approved collection path. While machine control and safety functions remain within the plant's engineered control environment independent of cloud availability. Older equipment doesn't necessarily need another direct connection. An existing industrial gateway may already collect the required signals, translate the equipment's protocol, and expose those values without adding another reader to the controller itself. A historian can provide another practical starting point. That's a system built to retain time based process data, and it may already hold the machine states or measurements needed for this particular analytical purpose. Check what that historian actually preserves. Some systems compress readings or retain selected changes, so the controls team should confirm whether the available record captures the behavior your analysis needs to understand. Using an existing path can reduce disturbance, but the gateway or historian still needs enough capacity to support the extra collection work without slowing the plant applications that already depend on it during production. Collection rates need approval from the equipment side. If a value change is slowly, repeatedly, requesting it at a much faster rate may add network traffic and control a load without giving the planner useful information. The fastest setting isn't automatically the best setting. Ask the controls engineer which signals can be read, how often they can be read, and which operating conditions require the collector to back off or stop. Network boundaries need the same care. A collection path might pass through a controlled intermediate network, with firewalls restricting communication between the equipment, environment and the services that send information beyond the plant. Outbound connections are often a sensible starting point. A plant side service can initiate communication toward an approved destination, rather than allowing a cloud service to open a connection straight into the machine network from outside. Outbound doesn't automatically mean safe. The security and controls teams still need to approve destinations, credentials, protocols, and access rights, including whether the collection software can write anything back to what equipment connected to it. Now consider a connection loss during our orders remaining work, a local buffer can retain collected records until communication returns, provided the buffer has enough storage and its software handles interruptions without disturbing collection. Buffering has limits. You need to define agree how long the plant can retain records locally, what happens when storage fills, and whether reconnecting sends a controlled backlog rather than overwhelming the receiving service. Recovery shouldn't compete with production for resources. The collector needs limits on memory, processor use, and network traffic, so catching up after an outage doesn't create a second problem inside the plant. 00:17:34 Speaker: I test that separation before relying on the connection. Disconnect the analytical path under approved test conditions and check that machine operation continues. Local connection behaves as intended, and support staff can identify the interruption. The receiving side can wait when necessary. With the control boundary protected, you can choose how each source delivers its information based on the source's behavior, rather than forcing every connection into the same pattern. For our orders. Business records a scheduled extract may be enough if planners review changes at set points during the day, the extraction schedule can follow that need rather than run continuously just because the platform permits it. Start with the supported extraction method. A scheduled job might request records through an application interface, or read an approved view, then pass the results into fabric for the next processing step. Full extracts can work for small reference tables for a growing transaction history. Repeatedly reading every record can consume source resources and extend the time needed to deliver the latest changes to the receiving system. Incremental loads reduce that repeated reading. They retrieve records added or changed since an agreed position provided the source exposes a reliable way to identify those changes through its supported access method. A modified timestamp can support that approach, but you need to confirm which updates change that timestamp. Because a field that tracks order creation won't identify a quantity corrected after the order first enters the system. Some sources support a different method. Change data capture often shortened to CDC. This collects database changes such as inserted, updated, or deleted records. Instead of repeatedly comparing complete extracts to discover what changed between reads that depends on the source and connector. Database support alone doesn't establish that your chosen fabric connection can capture those changes, or that the application supplier permits access through that particular database mechanism in production. Check the supported combination before choosing the pattern. Confirm the source version required permissions and connector behavior, then test whether the changes you need actually arrive through the route you intend to operate. CDC still needs a starting point. Typically, you establish an initial view of existing records and then continue from a changed position with the supported tooling determining how those stages connect without leaving a gap. Machine states call for a different choice when equipment already publishes state changes, as messages receiving those messages can fit better than repeatedly asking a table whether anything changed since the previous read completed production events can follow that same pattern. An operation, completion or reported scrap transaction may become available as an event, depending on what the execution application exposes and which integration methods its supplier supports. Azure services can help where the path needs them. Azure Event Hubs can receive high volumes of events, while Azure IoT hub supports device messaging and device identity needs when that fits the equipment integration architecture. You don't need both by default. Choose the service for the connection you actually need and check the supported route into fabric, rather than adding another messaging service simply to complete a standard technology list. Inside fabric data factory pipelines can coordinate supported data movement. For example, a pipeline can run a scheduled copy activity, wait for that activity to finish, and then start the processing task that depends on its output. Pipelines handle the sequence of work. They don't make every source interchangeable. So the available connector, supported authentication method and source specific extraction behavior still determine what that particular activity can read and how it reads it. Data flows Gen2 provide Power Query based data preparation where the connectors and destinations support the task. Your team can use them to select fields, change data types, and shape incoming records without writing every transformation from scratch. Use that option where it fits the workload. Check how much work runs at the source, how much runs in fabric, and whether the tested processing time still meets the delivery window for the data set. Our order can arrive through several methods. Its business transactions might use an incremental extract while execution completions arrive as events and a supported data flow prepares a scheduled inspection export for further processing. Those methods don't need identical schedules. They need to deliver the facts required by the decision, with enough evidence to tell whether each source has supplied the updates expected at that point. A fast ingestion job can still deliver old information if a source publishes updates long after the work occurs. Running the receiving job more often won't remove that delay or ensure related facts arrive together. That changes the next question. Once we've chosen how records enter fabric, we need to distinguish when something happened from when its record reached us. Before comparing the production figures those records describe. Suppose the machine stops just before the night shift ends. Its gateway records the stop locally, but a connection outage delays delivery until after the morning team starts reviewing production, placing yesterday's event among today's arrivals. Those records carry two different times. Event time describes when the stop occurred, while arrival time describes when the receiving service obtained the record, which could be much later than the physical event. Use event time to place the stop in production history. Keep arrival time as well, because the gap between those timestamps helps explain why a shift report changes after people have already reviewed its first figures. Otherwise, the wrong shift inherits the downtime. If your calculation groups records by arrival time, the morning team can appear responsible for a stop that occurred before anyone on that team took over the machine. Stored timestamps need a common reference. I'd normally use coordinated Universal Time or UTC for recorded instance, while retaining the source timezone, and enough local context to reconstruct the plant's operating day. UTC doesn't tell you which shift owns an event. That assignment depends on the plant's shift calendar, including when each shift starts, when it ends, and which production date the plant assigns to overnight work. Midnight isn't necessarily a production boundary. A night shift that starts on Monday and ends on Tuesday might belong to Monday's production day, even though some of its events carry Tuesday's calendar date. Keep both meanings available to the people using the figures. The calendar date answers when an event occurred locally, while the production date groups that event according to the operating rules used for the shift. Daylight saving changes make local timestamps harder to interpret when clocks move back, the same local time occurs twice, so a timestamp without an offset or other distinguishing evidence might not identify one precise instant moving clocks forward creates a different problem. An elapsed downtime calculation based only on local clock readings can count as skipped hour. Rather than measuring the actual duration between the stop and the restart, calculate elapsed time from consistent, unambiguous timestamps, then apply the local calendar for shift reporting with explicit rules for the longer or shorter shift that occurs when the clocks change at that plant. There's also the clock inside the equipment. UTC conversion won't repair a device clock that runs several minutes behind, because changing the time reference doesn't establish whether the original reading accurately represents the event. Clock quality deserves its own check. Compare source clocks against an approved time reference where possible, and retain an indication of uncertainty when older equipment can't provide a trustworthy timestamp for the recorded event, don't silently replace that uncertainty with arrival time. If the gateway supplies the best available timestamp, record that choice explicitly. Because the time of collection may differ from the time the machine actually changed state, a disconnected gateway can complicate event order two after reconnection, a delayed stop might arrive after its restart, so processing records strictly in arrival order can produce a misleading account of the machine's behavior. The calculation needs a defined approach to late records. It might revisit a recent reporting window as events arrive, while treating older arrivals as exceptions that require a controlled recalculation rather than quietly ignoring them. Corrections create another reason for figures to change. An operator might adjust the recorded start of a stop after reviewing the shift, changing its duration without any new physical downtime occurring on the machine. Distinguish an early estimate from a settled production figure. A shift total can remain provisional, while expected records or corrections are outstanding, then become settled under an agreed cutoff and reconciliation rule for that source. Settle doesn't mean corrections become impossible. It means later changes follow a stated process. So someone comparing yesterday's report with today's answer can understand why the figure changed and which reporting period it affects. That gives each record a usable place in production history with occurrence time, arrival time, and local operating context preserved. We can move those records into shared analytical storage without losing the evidence needed to compare them fairly. Where do those records live once they reach fabric? One lake provides fabrics, shared logical data, lake giving supported fabric workloads, a common storage foundation rather than requiring each workload to start with its own separate storage account. The word logical deserves some attention. You can work through a shared storage structure, while the data remains organized into distinct fabric items, with workspaces providing boundaries for organizing those items and managing collaboration. A workspace groups related work within it. An item such as a lake, house or warehouse gives data a defined home, so shared storage doesn't mean every plant deposits everything into one undivided folder for our plant. That organization affects how people find the order. The team maintaining production data can publish an identified fabric item, and other teams can work from that known location rather than exchanging another extract through email. One lake doesn't erase those item boundaries. It gives the workloads a shared foundation, while the item and workspace structure still affects where data belongs, who maintains it and how other teams reach it. That brings up a practical question about existing storage. Suppose inspection files already sit in a supported Azure Data Lake storage location, and the quality team needs to keep that location for its current applications. You may be able to reference those files through a one lake shortcut. A shortcut points to a supported data location, allowing compatible workloads to reach data through one lake without requiring a separate ingestion copy. Simply to establish that reference, the files can remain at the target location. For inspection records, that means the quality team's existing storage can remain in use, while fabric provides a path through which analytical work can access those records. A shortcut can also reference data elsewhere within one lake. If another team already maintains the required data set in a supported fabric location, a shortcut can provide access without creating another independently maintained copy for your workspace. The reference still depends on its target. If the target becomes unavailable, its permissions change, or its contents move. You need to understand how that affects the workloads following the shortcut. To read those records, check the supported source and the intended operation together. A location that works for reading files doesn't automatically support every write operation, every table behavior, or every fabric workload you might want to use later. Credentials need a clear arrangement as well. The identity or connection used to reach the target must have the required access, and the precise behavior depends on the shortcut type and supported configuration. Shared access doesn't mean unrestricted access. Before relying on a shortcut for our order, test the actual route with the identities and workloads that will use it, rather than assuming your own successful read proves everyone else's access. I'd also question any blanket promise of no copies. A shortcut can avoid a separate ingestion copy, but caching downstream transformations and the chosen consumption path can still introduce stored data elsewhere. In the wider solution, follow the actual data path, establish which records remain at the source, which operations read them remotely, and which steps create another stored result. Because those choices affect dependencies as well as storage use. Now consider what the planner gains from this arrangement. Production records and inspection files can become reachable through the same analytical foundation, with known locations that teams can reference, rather than repeatedly requesting files from one another. That removes an access obstacle, not an interpretation task. Suppose the inspection file identifies our order with a quality specific suffix, while the production records use the original order reference and contain no such suffix at all. Both records can sit within reach of the same workload. One leg still can't establish from storage location alone. Whether those references describe the same order, a related batch, or two records that shouldn't be combined. The unresolved reference remains unresolved after consolidation. A person querying both locations still needs an approved interpretation before connecting the inspection result to the production order and treating that connection as evidence for shipment. A judge shared storage by a narrower test. Can the right workload reliably reach the intended records through a known location, without creating unnecessary independent copies or concealing dependencies on another team's storage? Then comes the structure of the data itself. Once those locations and dependencies are clear, we can decide how to arrange the files and analytical tables so the shared layer supports manufacturing work for our order. The storage structure needs to support different kinds of work quality might supply an inspection file while production supplies structured events, and the planner needs a table that answers questions about the order without opening either source. A fabric lakehouse accommodates files and analytical tables within one managed item. That gives the engineering team room to process incoming material in its original file form and publish structured tables for the people who need to query it. The file area has a practical purpose. An inspection export might contain a header, measurement details and comments that need passing before those contents fit into tables that an analyst can use reliably. Keeping the file preserves its original structure. The processing work can read that file and create tables from its contents, without requiring the file itself to behave like a finished analytical data set. Production events already have a more regular shape. Those records can become tables with defined columns, while a curated order table presents selected facts in a structure suited to questions about the order. Those structures support different tasks within the same lakehouse, an engineer can work with incoming files. Another process can handle event tables, and an analyst can query the prepared order data without repeating all that preparation. For managed lakehouse tables, Delta Lake supplies the table format. It combines data files with a transaction log that tracks committed changes so readers can work against a consistent table state. While supported updates occur, a transaction either commits or it doesn't. If a job fails before committing its table changes, readers shouldn't receive a half published version simply because the process managed to write some files before stopping. That protection has a defined scope. A successful update to one delta table doesn't automatically mean every related table in the lakehouse changed as part of the same transaction, so design around the operations your processing engine supports. Delta also records table versions as changes. Commit supported tools can use that history to inspect earlier versions, which helps when you need to compare a table before and after a processing change. History still depends on retention. Reading an older version requires the necessary log information and data files to remain available, so a recorded version number alone doesn't promise that its contents remain readable forever. Clean up choices affect that ability. Removing obsolete files can reclaim storage, but it can also remove files needed to read older versions, which means retention settings need to match your intended use of history. Maintenance belongs in the operating plan. Frequent small writes can leave many small files and support a table. Maintenance can compact those files to reduce the overhead of reading them. During analytical queries, I'd choose maintenance based on observed behavior. A production event table receiving frequent updates may need a different schedule from an order table refresh less often because they're write patterns and query demands differ. The Lakehouse isn't the only reasonable destination for prepared manufacturing data. Fabric warehouse provides a relational environment centered on SQL, the language many database teams already use to build tables and answer business questions. Your team's working methods matter here. If most preparation uses relational tables and circle and consumers expect a warehouse style data model, a warehouse may fit the work better than introducing file processing. They don't need a lakehouse fits work that crosses files and tables. It can suit engineers using spark, a distributed data processing engine or notebooks to pass inspection exports, and process production records before publishing analytical tables. Neither choice needs to become a rule for every data set. You might process varied source files in a lake house and publish selected relational data into a warehouse, provided that extra step solves a real processing or consumption need. Check how people will read the result. An analyst querying prepared order tables has different needs from an engineer processing measurement files, so test the expected work rather than choosing solely from a product comparison. I'd settle the choice with a small workload, use representative inspection files, production events, and order queries to check supported operations, processing effort, and response times with the skills your team actually has. Then give each stored form a stated role. Once files and tables have suitable homes, we can define the processing path that turns incoming records into facts people can use for the shipment decision. The processing path needs clear boundaries between what arrived and what people can use. I'd use bronze, silver and gold to name those boundaries. With each layer responsible for a different stage in preparing the orders. Data for analysis. You'll hear this called the medallion pattern. It's a way to organize data preparation, where retained source records pass through checks and transformations before becoming data sets built for a stated business purpose. The names don't certify the contents. A table called gold can still contain incorrect quantities if the processing behind it never checks what those quantities represent or whether the required records arrived. Give each layer a job you can test for our shipment question. That means being able to explain what enters each stage, what changes there, and what remains unresolved when the records leave it. Bronze keeps the incoming evidence close to its received form. If the quality export supplies a release status, bronze retains what that delivery contained, together with enough ingestion information to connect it to the process that received it. Think of bronze as the answer to what did we receive? That question differs from what should the planner believe? Because the source delivery might contain a valid update and unexpected value, or a record requiring further interpretation. Keeping that distinction lets you revisit the processing later. If you discover that a transformation interpreted a source field incorrectly, retained source records give you material to process again under the corrected interpretation. Bronze also makes the result traceable. When someone questions the released quantity, you need a route back from the prepared result to the incoming records used to produce it, rather than another explanation based on memory. Silver prepares those records for consistent use across the plant's data. This is where source specific representations become records that can be compared with identifiers mapped, units checked, and event structures brought into an agreed form. Cleaning should preserve the business, meaning removing stray spaces from an equipment reference might fix a formatting fault, while replacing an unfamiliar status with a familiar one could change what the source actually reports. For our order, silver connects the records through approved references. The execution event and the quality record can carry a common order reference for analysis, while retaining the source references needed to understand where each connection originated. 00:35:20 Speaker: Units received the same deliberate treatment. A quantity should carry its stated unit and any conversion should follow an approved rule rather than depend on an assumption hidden inside a processing step. Event structure matters as much as field formatting. A stop event and an operation completion can use consistent fields for their references and timestamps, while keeping distinct event types because they describe different occurrences. Silver doesn't mean every uncertainty has disappeared. It means the preparation has applied stated rules and records that still lack the required meaning remain distinguishable from records that meet those rules for downstream use. Gold takes those prepared facts to what a specific question for released output. The dataset should expose the quantity approved within the intended release scope, rather than require each analyst to interpret the underlying quality records again. Downtime analysis needs a different prepared result. It brings stop information into a form suited to analyzing lost operating time using the relevant production context. Instead of presenting a stream of equipment messages. Order progress has its own purpose. Its prepared dataset can represent progress through the required operations. So someone asking about remaining work doesn't have to reconstruct that progress from individual execution events. These data sets can draw on the same silver records. Their differences come from the questions they support, not from separate versions of what the source supplied or separate interpretations invented for every new request. Keep the transformations between layers explainable. Someone should be able to ask which rule changed a value and get a specific answer. Whether that change concerns an identifier, a unit, or a calculation. Fabric provides tools to build and run that processing. Your team still defines the rules that permit records to move forward, which means the medallion pattern becomes useful through the work assigned to each layer. Now suppose the planner challenges the released output figure. The immediate task is to retrieve the source evidence behind that result. So the next design question concerns exactly what bronze must preserve for that investigation. Open the retained delivery. And the first question is which source supplied it for our order. The evidence should identify the quality application, the plant and the particular export or source object that supplied the disputed release record. A file name alone won't give you that certainty. Keep source identity as recorded metadata, because someone can rename a file without changing its contents, or reuse yesterday's file name for an entirely different delivery. Record extraction time separately from the event time supplied by the source. Then attach an ingestion run reference so an investigator can connect that delivery to the attempt that received it, including whether that attempt completed. Missing timestamps should stay visibly missing if an inspection export supplies no event time preserve that absence rather than filling the field with extraction time and implying knowledge the source never provided. Consider a retry of that inspection export. The first transfer reaches fabric, but its completion acknowledgement never reaches the sender, so the sender delivers the same file again during its next attempt. Both arrivals deserve a record of receipt use, delivery identifiers, source record identifiers or content checks, where appropriate, to recognize repeated material while retaining enough detail to explain which attempts carried the same underlying records. Partial extracts create a different problem. A job might retrieve the first pages of inspection results and then fail, leaving a plausible collection of records that doesn't represent the complete requested delivery. Preserve its incomplete state alongside the extracted records where the source supports them, record page references, expected counts, or completion markers that distinguish a finished extract from one that stopped before reaching the end. Recovery needs a known restart position. A checkpoint records how far extraction has safely progressed, but its exact meaning depends on the source, whether that's a change position, a page token or another supported reference. Advance that checkpoint only after the corresponding records are durably retained. Otherwise, a failure between advancing the position and saving the records can leave a gap that later retrieves skip because extraction resumes beyond it. Some restart references expire, so test their recovery limits. If a source retains changes for a limited period, an outage beyond that period may require a fresh extract rather than another attempt from the old checkpoint. For suitable sources, I'd retain incoming changes through append only capture. Each received version remains available as another retained record, giving you a way to inspect what arrived before and after a source correction. Append only capture still needs explicit change, meaning if quality corrects a release quantity, retain whether the incoming record replaces an earlier value or records an adjustment together with whatever source reference connects those versions. A deletion needs evidence of its own. Preserve an explicit deletion notice where available, or retain the scope and completion evidence for a full comparison, before concluding that a missing source record represents deletion. That approach doesn't mean retaining every field forever. Set retention periods against the investigation window, source recovery limits, contractual obligations, and any legal requirements that apply to the particular manufacturing records you're collecting. Sensitive fields need deliberate limits from the start. Inspection comments can contain employee names or customer details, so decide which fields bronze genuinely needs and restrict access to retained content that carries those details. Preserve evidence without treating unrestricted retention as the default. Where policy requires removal or masking. Document that treatment because a future investigation needs to know whether the retained record differs from the original delivery. Storage cost includes more than the first successful extract. Repeated full deliveries, duplicate files and retained versions can accumulate quickly, so measure their growth and choose retention windows that the investigation requirement actually justifies. Then prove that retained evidence supports recovery. Select a bounded period for our order. Replay its deliveries into an isolated test destination, and check that you can account for every expected record. Include an interrupted extract and its repeated delivery in that test also include a correction and a deletion, so recovery proves it preserves change. Meaning rather than merely proving that the files remain readable, a readable file isn't a successful recovery. You should be able to identify the source, reconstruct delivery completeness and explain which version supplied the evidence within the retention period you've committed to support. Those retained records can now support the next preparation step with their origin and delivery history intact, we can establish which records refer to the same manufacturing work without losing the evidence behind that connection. The silver layer needs maintained mappings, not clever guesses for our order. Those mappings connect the machine, product, operation, batch and location references used by each source to the manufacturing things they actually describe. I'd make each mapping an explicit record, store the local identifier with its source, and plant the shared identifier it maps to, and the period during which that connection applies to the work. Keep the source reference beside the shared reference. When an operator questions which machine a record describes, the investigation should lead back to the equipment name they recognize, not stop at an unfamiliar analytical identifier. Different identifiers don't always mean different machines. The maintenance system might identify a physical asset, while Ms. identifies a production resource that includes the machine and its assigned work position within the line. Those references need a stated relationship. Mapping both to one shared machine identifier only makes sense if the intended analysis treats them as the same thing within the period being examined. Locations deserve separate identifiers of their own. A machine can occupy a work area without becoming that work area, and a production record might refer to either one, depending on how the source captures assignments. Equipment moves expose that distinction quickly. If the plant relocates a machine between lines, the mapping needs to preserve its physical identity while recording. When its location and production assignment changed. Yesterday's work should keep yesterday's location. Applying today's equipment arrangement to older records would move that production history into the new line, even though the machine performed the work somewhere else at the time. Order matching also needs more than a familiar number a production order, its individual operations, and the batch is created during execution. Describe related work, but they aren't interchangeable references that you can safely collapse into one identifier. Operation numbers can repeat across orders. An operation reference therefore needs its order and source context, rather than treating every occurrence of the same step number as one shared operation across the plant, batch identifiers need their issuing context as well. If a supplier and your own plant use the same batch, code matching on the text alone could attach incoming material records to unrelated production. Product references bring another kind of change. A part number can remain unchanged while its revision changes, so a mapping should retain the revision when that revision affects which physical product the record describes. Don't assume the newest revision applies to everything. The product team needs to confirm how older stock and current production refer to revisions. Before you attach a single product description to records from different periods. Code sets require the same care as identifiers. A local status or reason code needs a documented interpretation with the original code retained, so someone can check the mapping against the plant's actual use. Similar labels can conceal different rules. Two plants might use waiting for different conditions, so a shared category needs an approved definition rather than a text replacement based on how closely the labels resemble each other. Units need product specific meaning, where the conversion depends on packaging. A container count can become a part count only when the conversion reflects the relevant product and packaging arrangement, rather than a fixed assumption applied to every container. An uncertain conversion should remain uncertain. Route that record for review with its original value and unit. Instead of publishing a precise looking quantity that nobody can defend for the affected product. Unmatched identifiers need a visible review route. Retain the record with the missing mapping. Send it to the person responsible for that reference and distinguish it from records ready for shared analytical use. The reviewer needs enough context to decide. Include the source reference, plant, relevant dates, and any candidate matches without letting an automated similarity score approve a connection that could associate work with the wrong order. Approved mappings should cover their intended scope. A match for one plant during one period shouldn't become a global rule, just because another source happens to reuse the same characters in its local identifier. After approval, reprocess the affected records through the updated mapping that lets the corrected connection enter the prepared data without requiring someone to patch individual results and leave the underlying reference problem in place for tomorrow. A correct match settles identity, not counting. Once the records point to the same order, we still need to combine their different levels of detail without repeating the order quantity across every related event. A successful join can still produce an incorrect total. Every reference can match, every record can belong to the right order, and the resulting table can still multiply quantities simply because the records describe different levels of detail. Consider a separate counting test using a hypothetical order. Attach its planned quantity to ten machine events, then add that quantity down the resulting rows and the reported demand becomes ten times the actual order requirement. Nothing in the order changed. The join repeated the order header for every matching event, and the calculation then treated each repeated value as another contribution to demand, rather than the same demand appearing again. The mistake can survive a quick inspection. Each row contains a recognizable order, a plausible quantity, and a genuine machine event, so someone checking individual rows might approve a table whose total is wrong. Adding inspection samples can multiply the result again, if you join those samples to machine events using only the order reference. Each event can match several samples without establishing any actual relationship between them. That's a pairing created by the query. Sharing an order reference establishes that both records concern the order, but it doesn't establish which measurement belongs with which machine, event, or whether that comparison makes sense. I'd keep those observations in separate fact tables. A fact table holds records of a stated kind, so operation events, inspection samples and sensor readings retain their own detail. Instead of becoming one expanded collection, the order header keeps its own scope to. Its planned quantity belongs to the order, while an operation quantity belongs to a particular execution record, and a sensor value belongs to a particular recorded reading. Separate tables still need a way to work together. Shared dimensions provide that connection by describing the products, equipment, shifts and orders that the different fact tables refer to, and those relationships actually exist. A dimension gives several tables a common reference for analysis. A shared product dimension, for example, lets you select a product and examine relevant order demand alongside production events without physically combining every underlying record. The same principle applies to equipment and shifts. You can compare readings and execution events for a selected machine during a selected shift, while calculating each result from the table that holds its own observations. Not every fact needs every dimension. An order spanning several machines and shifts shouldn't acquire one invented equipment assignment merely because the reporting model expects every table to connect in exactly the same way. Keep each relationship faithful to the work where equipment or shift belongs to execution events rather than the order header. Let those events carry the relationship instead of pretending it describes the whole order. For a combined answer, calculate each quantity at its proper scope, then compare the results through the shared dimensions, checking that the selected product, order or period means the same thing for each calculation being presented. Grouping before joining can help when the scopes genuinely match, but rolling everything up to order level doesn't make sense for readings additive or establish that every inspection sample represents the same quantity of physical parts. Test the totals before and after each join. If attaching descriptive information changes the planned order quantity. Stop and explain that change before using the result. Because a matching reference alone doesn't justify multiplication. Time also belongs in these relationships. When an equipment hierarchy changes, a production event should connect to the hierarchy effective when the work occurred, rather than whichever hierarchy the current equipment record happens to contain. One approach keeps dated versions of the dimension record. Each version describes the equipment during a defined period, and the event connects to the version covering its occurrence time, preserving the historical reporting relationship. Those effective periods need clean boundaries. Overlapping versions can give one event two matching equipment records recreating the counting problem even though the event table itself contains no duplicate production records. Product attributes need the same historical care. If a classification changes, decide whether the analysis should use the classification effective during production, or deliberately regroup earlier. Work under today's classification, then keep those choices explicit. Both views can support legitimate questions. They shouldn't switch silently because a change in descriptive attributes could otherwise appear to change a production result, even though no additional work occurred during the reporting period. Now the quantities can survive being brought together. That leaves a harder question for our order, which recorded progress actually satisfies the plant's definition of completed production, and which parts still need a separate decision before shipment. Quality release records permission to use parts for a stated purpose. To represent that permission accurately, we need to separate the physical work performed on those parts from the inspection findings and the decision authorizing their next move. Start with the difference between inspected and accepted. Inspected quantity describes the parts covered by inspection under the agreed inspection method. While accepted quantity describes the parts judged to meet the applicable acceptance conditions. Sampling needs particular care here. If the approved procedure uses a sample to assess a batch, retain both the sample quantity and the batch covered by that assessment rather than treating them as interchangeable counts, acceptance and release can occur at different times. Parts might meet the measured requirements while supporting documents, a required review or another release condition remains outstanding, leaving acceptance recorded but shipment authorization still pending. Shipped quantity records another event altogether. It describes parts dispatched under the shipping process, so a later release doesn't retroactively establish that those parts physically left the plant when quality approved them. Each measure needs its own evidence produced, inspected, accepted, released, and shipped. Quantities can describe overlapping populations of parts, so adding them together wouldn't produce a meaningful total of anything the plant actually holds. The harder task is tracking which parts each decision covers manufacturing, genealogy records, the relationships between material entering a process, and the batches or individual serialized parts that emerge from it, including changes during subsequent work, follow a split in our hypothetical batch, one portion moves to inspection while another remains in production, so the original batch reference alone can no longer describe one common condition. For all its parts, the split needs an explicit relationship record which child batches came from the original batch and what quantities moved into each. So a release covering one child doesn't accidentally authorize shipment of its sibling. Merging requires the reverse relationship when material from separate batches enters a combined batch, preserve those contributing references, so a later quality finding can identify the material and finished parts potentially affected by it. Physical mixing changes what you can distinguish afterward. The genealogy must reflect the process's actual traceability limits, rather than claim individual part relationships that the plant never captured and can no longer establish from the records. Serial numbers can support finer tracking where the process records them. An individual part can retain its identity through operations and inspection, but that depends on reliably capturing the serial reference at the relevant steps. Rework adds another part, not necessarily another part. If a rejected part returns through machining and inspection, preserve its connection to the earlier attempt so the additional operation doesn't inflate the count of distinct finished parts. Scrap needs a recorded disposition as well. When parts leave the usable population, retain the affected batch or serial references and quantity, rather than allowing a later batch level release to include parts already scrapped. Those relationships determine the reach of a quality hold. A hold should identify its affected material, its reason, and its effective time, with enough scope to distinguish the blocked parts from other work. Sharing the order keep the holds event history alongside its current status opening, extending, narrowing and removing a hold. Describe different decisions, and the current state alone won't explain what shipping permission existed at an earlier moment. A passing retest shouldn't silently erase that history. It supplies evidence for the quality process. While the release decision records whether an authorized person or approved workflow has cleared the applicable conditions for the affected parts. Record that authority with the decision. The release evidence should identify who or what approved it, when approval took effect and which quantity, batch, or serial references the approval actually covers. Return to our order after the remaining physical work finishes. Machining now reports completion, but one child batch still carries an open hold, and no authorized shipment release covers that portion of the customer's required quantity. 00:52:59 Speaker: The planner now faces a specific shipment constraint. Production doesn't need to report the parts as unfinished merely to keep the truck from leaving. The unresolved release condition explains the blockage in its own right. That distinction belongs in the prepared analytical facts, with quantities tied to identifiable parts and release decisions tied to their actual scope. We can build reusable data sets that preserve the reason and order can or can't proceed. I designed the order readiness data set around the planners next action. Instead of assembling every available order field, choose the facts that explain whether the required quantity can move toward shipment and what currently prevents it. Make the data set scope explicit. One record might describe an order line and its required delivery with supporting detail, where different portions of that requirement face different constraints within the plant. Execution progress supplies the remaining work position for that requirement. Material constraints add whether the necessary components or consumables are available for the outstanding work, while quantity status identifies any release condition blocking the relevant parts. Material availability needs a careful calculation. Stuck somewhere in the business isn't necessarily material available to this order, particularly when another job has reserved it or the remaining process requires a specific approved material. Expose the constraint rather than just a readiness flag. A planner needs to distinguish work awaiting material from work awaiting release, because those conditions send the follow up tasks to different people and require different actions. Readiness can also be partial. The data set should distinguish a quantity that meets the agreed conditions from the remaining requirement, without implying that a partial quantity satisfies a customer commitment requiring the whole delivery. Keep the answer explainable at that level alongside the readiness result. Publish the blocking condition and the supporting quantities so the planner can understand the conclusion without reconstructing the underlying processing. Each morning, downtime calls for a different gold data set. Its job is to connect a stopped interval with the work underway and the recorded reason making the result useful for investigating production losses rather than merely counting equipment messages. The work context changes. The question people can ask. A stop during a product change has a different operational meaning from a stop during an established production run, even when both intervals last equally long. Reason codes should support that distinction without inventing certainty. Keep an unclassified stop visibly unclassified until an accepted reason becomes available, rather than assigning a convenient default that makes the analysis appear more complete. Define how each interval enters the calculation. State whether the data set includes stops outside planned production. How it treats a stop still in progress, and what happens when recorded intervals overlap for the same equipment. Those choices change the reported duration. Someone responsible for production performance needs to approve them, because an engineer choosing a convenient calculation can otherwise create a result that conflicts with the plant's intended operating definition. Each data set needs a written calculation specification for readiness that specifies the conditions required to count quantity as ready for downtime. It specifies which intervals contribute and how their duration enters the result. Freshness belongs beside those rules. Agree how recent each contributing source must be for the intended decision, rather than attaching one refresh timestamp and assuming every fact in the data set shares that age, a successful refresh doesn't settle that question. The readiness calculation might finish on schedule using current execution records, but an older material position leaving the combined answer unsuitable for a decision about starting the remaining work. Set expectations for that situation before it occurs. The published data set should indicate whether it meets its freshness requirement, and its owner should decide which users remain acceptable when a contributing source falls outside that requirement. Ownership needs to cover the published result. Name the business person responsible for its meaning and the technical person responsible for delivery, with an agreed route for resolving a challenged calculation or missed publication. Reuse should happen at the business fact level. The shipping review and the production meeting can use the same readiness facts while selecting different orders, rather than receiving independently prepared tables with slightly different readiness calculations. Different reports don't automatically justify different goal tables. Create another data set when the business question requires a different scope or calculation, not merely because another department wants a different arrangement of the same facts that keeps changes manageable without forcing one data set onto every use. A revised readiness rule can enter the maintained data set once, while reports continue to present the relevant facts for their own audiences and decisions. There's still a boundary between prepared data and shared reporting language. Once these data sets publish dependable business facts, analysts and plant leaders need a common way to name, relate and calculate with them across their reporting work. A power BI semantic model gives reporting that common language. It defines how analytical tables relate, which measures people use and which business terms appear. When someone asks a question about production through a connected report. Measures carry calculation logic within that model. When a production lead selects a product or shift, the model evaluates each measure within that selection, following the relationships and rules you've established for the reporting scope. That selection changes what the answer means. A measure called released output needs to behave predictably when someone narrows the report to a plant, a product, family, or a period containing only part of an order's history. Consider released output for a weekly production review. Does the measure count quantity released during that week or quantity produced during that week that has since received release, even if approval arrived the following Monday? Both questions make sense, but they produce different answers. Give those measures distinct names and explicit date rules, so selecting a week doesn't leave the reporting tool to choose an interpretation the business never approved. Remaining quantity needs the same precision for a shipment review. It could mean the delivery requirement minus eligible released quantity. While a production supervisor needs the quantity still requiring completion at a particular operation, name the measure for the question it answers, remaining to release and remaining to produce. Tell people more than a generic remaining, especially when both appear in reports used during the same morning meeting. Define each agreed measure once within its intended reporting scope. Reports connected to that model can reuse the definition rather than carrying separate formulas that gradually diverge as different authors respond to local requests. Access rules also form part of the model, where supported and configured rules can restrict which plant records someone can query. So the model determines both the calculation and the permitted scope of the answer. Now apply that discipline to overall equipment effectiveness or OE. This measure combines availability, performance and quality, but its usefulness depends on agreeing what each factor includes for the equipment and production period being assessed. Availability compares operating time with the production time you've chosen to assess whether a planned changeover falls inside that assessed time. Changes the result, so the model needs an approved rule rather than a report. Author's preference. Performance relates actual output to an agreed production rate for mixed products. That rate needs the relevant product and equipment context, because comparing every run against one nominal speed can turn a normal product change into an apparent loss. Quality compares good output with total output. Under the chosen definition, you need to decide how first pass acceptance and later rework affect that factor. Then keep that decision consistent wherever the model publishes the measure. The total also needs a defined calculation. Averaging percentages from individual shifts can produce a different result from calculating the factors over the combined underlying quantities and durations, particularly when those shifts carry different production volumes. I test the model with deliberately uneven shifts. Ask the production owner to explain the expected combined result, then check that the measure follows that rule. When users select several shifts or several machines, local differences don't have to disappear. A continuous process and a discrete assembly line may need different production boundaries, and a common model should represent those differences openly where they affect the measures. Put the observed differences into maintained rules or clearly named measures. If one plant excludes a category of plant downtime and another includes it, users need to know before comparing the resulting availability figures as equivalent performance. Hidden report formulas make that distinction almost impossible to manage. One author changes the denominator for a local meeting, another copies the report, and the familiar measure name now conceals two calculations with different operating boundaries. A local rule should remain visible and testable. Record where it applies and how it changes the calculation. So a cross plant comparison can either use a common basis or explicitly preserve the different local definitions. You can then test meaning separately from presentation, change the plant selection, cross a reporting period boundary, and compare totals with detail checking that the same business question still receives the same intended calculation. That gives you a stable definition to assess against practical reporting needs. The next choice concerns how reports retrieve the data, because a consistent measure still needs to respond quickly enough and use information fresh enough for its intended decision. Picture the morning review with several supervisors asking different questions at once. The access mode determines how their queries reach the data, how much processing each interaction requires, and which limits become noticeable when everyone starts using the model together. Power BI gives you several choices here. Import, direct query and direct lake. Follow different paths between stored data and a report response, so I'd choose among them using the actual reporting workload rather than a preferred product label. Import loads data into the semantic model's own storage queries can then use that prepared compressed data without asking the underlying source to process every interaction, which can work well for repeated production reviews over a defined reporting period. The trade off includes managing refresh and model size. A report can respond quickly while showing the data from its last successful refresh, so response speed alone doesn't establish how recently the underlying production facts became available. Direct query sends queries to the underlying source, as people use the report that avoids importing the full dataset into the model, but each interaction depends more directly on the source's response, its available resources, and the queries generated by the model. That dependency becomes noticeable under load. A page containing several calculations can produce multiple source queries, and several users. Changing selections together can create a workload that a single test user never exposed during development. Direct Lake takes another route for supported fabric configurations. It lets the semantic model work with one lake backed table data without maintaining a conventional imported copy of the whole data set through the usual import refresh process. That doesn't mean nothing enters memory. Direct Lake loads the data it needs into the analytical engine's memory. So available capacity and the amount of data queries touch still affect how the model performs. Check how the model discovers committed table changes to direct. Lake. Has framing behavior that establishes the table versions available to the model, so a completed upstream write and an updated report aren't automatically the same event. The exact configuration matters more than the broad label supported sources. Security behavior and limits differ between direct lake configurations, including whether the model uses direct lake on one lake, or a path through a SQL analytics endpoint. Verify those details against current documentation before committing. Some supported configurations can fall back to direct query under particular conditions, while others don't use that fallback path. So a test should establish which engine actually answered the query. I'd resist declaring one mode. The default winner import can suit a stable review dataset. Direct query can suit supported sources that handle interactive demand well and direct Lake can fit shared fabric tables when its requirements match your workload. Model design still shapes the result. Wide tables containing many distinct text values can increase memory demand, while complex relationships and calculations can require substantial processing. Regardless of how the model reaches its underlying data, security needs to enter the performance test test with the actual roles and identities people will use. Because permissions and filtering can affect supported query paths and execution behavior in ways an unrestricted developer account won't reveal. Capacity deserves the same practical treatment fabric workloads can compete for shared resources, so assess the report while the expected background processing is running, rather than measuring it only when the capacity has little else to do. Use the meeting itself as the test scenario. Have supervisors select their lines, switch between products, request longer periods, and open detailed views, while the production lead asks for an overall comparison across the plant. Include concurrent users, not just repeated clicks from one engineer. Check the first response after inactivity, as well as repeated queries, because a warm result can conceal the delay someone encounters when opening the report before a shift meeting, record what actually causes an unacceptable wait. Compare query durations, memory pressure, capacity, behavior, and any fallback activity. Then change one design choice at a time so you can explain which change improved the working session. Agree on acceptable response times with the people using the reports. A production review may tolerate a slower, detailed investigation while needing routine selections, to respond promptly enough that the group doesn't abandon the report. Mid-discussion. There's a separate timing question beyond report response. Once the query path performs adequately, we still need to ask how quickly an operational event becomes available for the decision, because a fast query can return an outdated machine state, how soon could someone act differently? Because a new event arrived? That's where I'd start the latency discussion, because the useful delivery window depends on the response you're trying to support, not the speed available from the connection. A machine stop notification might need to reach a supervisor within moments. A quality review held each morning might only need all expected results available before the meeting, with a clear indication of anything still outstanding from the previous production period. Those needs justify different processing paths, spending resources to deliver every inspection update immediately. Won't necessarily improve that review, while delaying a stop notification until the next scheduled extract could remove its practical use entirely. Fabric real time intelligence supports work centered on incoming events. Within that workload, event streams can receive supported event sources, apply supported transformations, and route events to what destinations chosen for the analysis you need to perform. Event House provides an environment for storing and querying event oriented data. It's Kusto databases can support investigations across timestamped records, such as asking which machines stopped recently and whether repeated interruptions cluster within a particular operating period. Use those tools for a defined event. Question for our plant, the immediate requirement might be identifying a machine that remains stopped beyond an agreed interval, rather than preparing the full shipment readiness answer every time another equipment message arrives. That leaves two useful analytical paths with different responsibilities. The event path supports timely investigation, while the Lakehouse path prepares the retained production facts used in broader reviews with an explicit agreement about how their answers relate. Give the livestock calculation a named owner. That person approves how a stop starts, how a restart closes it, and when a delayed correction requires the live calculation or the historical result to change. The two paths need reconciliation, not merely matching labels. Compare their results over a defined period using the same equipment. Scope and calculation rules, then explain differences caused by unfinished intervals, delayed records or processing that hasn't yet completed an active stop illustrates why that agreement matters. The live calculation might report an interval that's still growing, while the historical data set publishes only completed intervals, so different totals can be expected rather than evidence of a broken connection. Make that distinction available to the person receiving the answer. Label the live duration as ongoing and define when the completed interval becomes eligible for the historical data set, rather than letting both paths publish an unexplained measure called downtime. Duplicate handling also belongs within the live calculation. A repeated stop message shouldn't open another interval or send another notification merely because the receiving service encounters the same underlying event during a retry. Use a stable event identifier where the source provides one where it doesn't agree, a source specific matching rule and its limits. Because two similar messages can represent either repeated delivery or two genuine occurrences close together. Define how long the live path remembers processed events. If a repeated delivery arrives beyond that window, the reconciliation process needs a way to detect the discrepancy, rather than assuming the live total will always correct itself. Freshness should describe the signal path, not just recent activity. A busy stream from other equipment doesn't prove that the machine we're monitoring still has a working connection, or that its latest reported state remains dependable. Track expected communication for the relevant source. Where supported heartbeat messages or gateway health information can help distinguish a machine that hasn't changed state from a connection that has stopped delivering any usable evidence. Set alert thresholds around the required response. A supervisor might need a notification only after a stop persists beyond the locally approved interval, with event time determining duration and arrival delay affecting whether the notification remains timely. Also, define how the notification clears a restart should resolve the active condition under the agreed rule, while communication loss should produce an unknown state condition, rather than quietly imply that the equipment has resumed production. Test those rules using delayed and repeated deliveries include a stop that ends before its message reaches the service, because sending an urgent intervention request for an already resolved interruption can quickly erode confidence in the notifications. A current machine state still leaves an operational question unanswered. Even with dependable timing and sensible alerts, the supervisor needs to know what work depends on that machine before deciding which interruption deserves attention first. The missing connection is between equipment, capability and the work awaiting it for our hypothetical order. Knowing which machine stopped becomes more useful when we can establish which required operation depends on that machine and what makes another machine suitable. I describe that through product process resource relationships. The product defines what we're making. The process defines the required work, and the resource describes the equipment or people capable of performing that work under stated conditions. Those relationships carry manufacturing rules, not just references. A part might require a finishing operation that only certain machines can perform with suitability. Depending on its dimensions, the required tolerance and the tooling fitted for that operation capability can depend on the combination. A machine approved for one product's finishing operation might lack approval for another product, even when both operations share a familiar name and use similar equipment. People belong in that context as well. An operator's qualification might permit a particular setup or inspection task, while another person can run the equipment but can't authorize that step under the plant's working rules. That gives a resource relationship a precise meaning. Instead of recording that a machine and a product are somehow associated, record which operation the machine supports for that product and which conditions limit that permission. 01:10:58 Speaker: A digital twin can maintain some of this context. By that I mean a maintained representation of selected physical assets and their relevant behavior built around the particular questions you need the representation to answer. Its scope should stay explicit for this plant. A twin might represent a finishing sell, its operating states, and how its available capability changes when equipment enters maintenance or a required fixture becomes unavailable. You don't need to represent every physical detail. If the question concerns which work the cell can perform, a detailed model of its surface geometry may contribute less than an accurate description of its current configuration behavior needs a defined representation to a twin could describe permitted state changes or include a model estimating processing behavior, but you should establish which of those capabilities it actually contains before trusting its answers. The maintenance effort belongs in the design. When engineering changes a process approval or replaces a fixture, someone must update the relevant representation. Otherwise, the twin continues, describing a capability the physical cell no longer possesses. A knowledge graph makes relationships explicit. In another way, It represents things such as equipment, products, people and work together with named connections that describe how those things relate within the manufacturing process. The connection names carry the meaning. A machine can perform an operation. An operation requires a qualification, and a person holds that qualification, allowing a query to follow those relationships rather than rely on an ambiguous association. Consider a question from the production engineer, which people hold the qualification required for the finishing operation on this product, and which equipment has approval to perform that operation with the currently specified process. A graph can support that connected investigation. The model still needs the plant's approved definitions of capability and qualification. Because a graph database doesn't discover manufacturing permissions simply by finding a path between two records. These ideas can work together without becoming the same thing. A digital twin describes the selected physical system and its behavior. While a knowledge graph can hold relationships used to connect that representation with products, people and work. I choose the model after examining the questions. If a maintained set of capability tables and relationships answers them clearly, a relational design can provide the required context without introducing a separate graph platform or twin service tables can represent complex relationships perfectly well. A table linking a product revision, an operation, and an approved resource may be all you need. When those connections remain stable and the required queries stay manageable. Specialist models deserve a specific reason to exist. Repeatedly following changing relationships across equipment, qualifications and work may justify a graph, while questions about asset behaviour may justify a maintained twin with suitable behavioural models, fabric can hold data that feeds those representations. Your organisation still supplies the manufacturing relationships, and any specialist technology needs an integration path that keeps its representation consistent with the approved process and physical equipment. That context lets us ask a better question about the stopped machine. We can identify resources permitted to perform the affected work, but describing suitable alternatives doesn't establish which alternative produces a feasible production plan. Feasibility. Depends on what else needs those resources. At the same time. A machine can qualify for the work and still be unavailable when the order needs it, because another committed job already occupies its next production window. That's where capacity and delivery decisions need more than a capability match. Fabric can supply analytical inputs, such as remaining workloads, observed processing times, and material positions, giving a planning calculation evidence about the work competing for available production time. Those inputs support the calculation. They don't define its objective. The business still needs to decide whether the priority is protecting promised deliveries, reducing set up changes, or recovering a particular customer order without disrupting commitments elsewhere in the plant. I'd separate those responsibilities before selecting the scheduling method. ERP maintains the business transactions and planning functions configured for the operation, including requirements and planned orders, while Mezz records execution and supports the workflows through which production follows assigned work, the planning and execution functions need a controlled exchange. A revised schedule should refer to the same orders and operations those systems manage with clear responsibility for accepting changes, rather than a separate analytical plan that nobody actually executes. Advanced planning and scheduling or APS addresses the scheduling calculation. It tests possible arrangements of work against modeled resource and process constraints, seeking a feasible schedule, and where supported, improving it against the objectives you've chosen. Feasible has a specific meaning. Here it means the proposed arrangement satisfies the constraints represented in that model, not that the software has established every condition that might affect the physical work. Once production starts, model coverage determines how much confidence the result deserves. If the scheduler accounts for machine capacity but ignores a shared fixture, it can assign two operations simultaneously. Even though the plant can physically equip only one of them. Take the outage affecting our orders. Remaining finishing work suppose the expected repair extends beyond the planned production window, and the scheduler considers moving that operation to another approved machine, rather than waiting for the original resource to return The alternative needs the correct tooling at the proposed time. If the required fixture remains occupied on another job, the schedule must either wait for it or explicitly move that competing work, including any setup time that follows labor can rule out an apparently open slot. The machine might sit idle overnight, but the proposed operation still needs a qualified setup person whose availability doesn't necessarily follow the machine's operating calendar material introduces another timing condition. The unfinished parts must reach the alternative resource before work starts, and any required consumables must be available there, rather than merely recorded somewhere within the plant's overall stock position. Moving the work also changes. What happens after finishing an earlier completion could overload the next inspection window, leaving the order waiting in a different queue without bringing its shipment release any closer. That queue belongs in the scheduling assessment. Otherwise, a recommendation can improve utilization on the alternative machine while consuming capacity needed by another order that already has a tighter delivery commitment. Later in the same route, I'd compare alternatives using their consequences for actual commitments. Waiting for repair. Moving the operation and changing its sequence can each produce different completion dates. Set up demands and effects on other orders. Sharing the affected resources. The output should expose the trade off, not just a preferred machine. A planner needs to understand which orders move, which assumptions determine the predicted completion, and whether the proposed delivery depends on a repair estimate that remains uncertain. A scheduling recommendation still isn't an execution instruction before anyone applies it. Production should confirm that its assumptions remain valid, and that the people responsible for the affected work accept the proposed change within their authority. Approval needs to identify exactly what's being accepted. That could include the revised operation sequence, resource assignment and plan timing with any change delivery commitment requiring the appropriate business decision rather than following automatically from a calculated completion date. Applying the decision requires a governed write back path. An approved change should pass through supported application interfaces and validation, so ERP and Mez retain their transaction rules instead of receiving direct table edits from an analytical process, keep machine instructions within the approved execution and control path. The scheduling result proposes where and when work should run authorized workflows turn that proposal into accepted work with human approval wherever the operating process requires it. Permission to approve a schedule shouldn't grant permission to read every plant's data, the identities collecting records, the engineers preparing tables, and the planners querying results each need access suited to their own work rather than one shared permission set. Start with Microsoft Entra. Microsoft's identity and access service use named identities for people and supported workload identities for automated processes, so access belongs to a recognizable person or service rather than credentials passed around the engineering team. An unattended process shouldn't depend on an engineer's personal account where the connection supports it. Choose a managed identity or service principle, then grant that identity only the permissions required for its particular source and destination. That support varies by connection and operation. Check the authentication methods available for each activity, including who can manage its credentials. Because permission to change a connection can confer more power than permission to run it. Least privilege means granting enough access for the assigned task. A collector reading inspection results shouldn't also receive authority to change those results, and an analyst querying prepared tables shouldn't automatically gain permission to administer the workspace containing them. Workspace roles deserve close attention. Assigning a broad role merely to resolve one access error can expose other items or management operations, so check whether supported item permissions provide the narrower access actually required. I test our orders data path one boundary at a time. Begin at the source where the collection identity should reach the approved records without acquiring unrelated access to application tables or permissions that belong to operational administrators. Then examine access through one lake and the lake house. Establish which identities can read files or tables directly, including through shortcuts, and check whether a connection uses the caller's identity or another configured identity to reach its target. The SQL analytics endpoint needs a separate check. This provides a SQL based query route over supported Lakehouse data, and its effective permissions need testing independently from someone successfully opening the Lakehouse through another supported route. The semantic model creates another access boundary. Permission to read a report? Query its model. Build new content from that model, or access. The underlying tables can involve different grants. So record exactly which capability each audience needs. Restrictions apply within their supported enforcement scope. Row level security can limit records by plant, while object level security can restrict access to selected tables or columns. In a semantic model where those controls are supported, model security doesn't automatically secure the underlying files someone denied. Another plants rows through the semantic model could still reach them through a separately permitted data path, unless that path has its own appropriate restrictions. Test the denied actions as carefully as the permitted ones. Use representative accounts to query the model, connect through the SQL endpoint, and attempt direct lakehouse access wherever those routes are available to the intended audience. Don't perform those tests solely as a workspace administrator. Elevated access can bypass restrictions intended for consumers, leaving you with a successful demonstration that tells you little about what an ordinary planner can actually retrieve. For our shipment planner, write down the expected access explicitly. They might need released quantities for their plant without needing individual inspection comments, and the tests should verify both the permitted quantity query and the denied detail query. Microsoft Purview can support discovery and governance across supported parts of this environment. Check which capabilities apply to your fabric items and any external sources. Because catalog coverage, labeling behavior, and lineage visibility don't necessarily share the same boundaries. Sensitivity labels classify information under your organization's approved scheme. Verify where labels can be applied, how supported inheritance or export behavior works, and which protections actually apply. Rather than treating the label itself as a universal access restriction, cataloging helps people discover what data exists and inspect its descriptions, confirm which items and fields appear, what metadata the catalog exposes to each audience, and whether external source coverage requires additional configuration or scanning. Lineage needs a practical inspection to follow a representative path involving our order and verify which dependencies purview captures, particularly where custom processing or external systems may leave connections absent from the recorded lineage. Access controls establish who can reach a fact they leave another responsibility unresolved. Someone still needs to answer for what that fact means, and whether its condition makes it fit for the planner's decision. Put a person's name against that responsibility, not just a department. When the shipment readiness data set publishes a result that production disputes, the business owner needs authority to resolve the interpretation and approve any resulting change. To its definition. Technical ownership covers a different set of responsibilities that owner accounts for the pipelines and storage delivering the data set, including whether the published result contains the records that processing was supposed to include for that reporting period. Give both owners an explicit boundary for decisions. A missing delivery belongs with the technical owner to investigate while deciding whether the available evidence permits a shipment readiness conclusion belongs with the business owner under the agreed operating rules, turn those responsibilities into checks against expected production activity for a missing shift. Compare received coverage with the plant's operating calendar and relevant execution records, rather than assuming that an empty period means nobody worked, the check needs context to distinguish absence from inactivity. A scheduled shutdown may explain a shift without production, while a staffed shift with recorded work assignments but no completion records, deserves investigation before its output enters a weekly total. Duplicate completions require a similarly careful test, compare the source transaction reference operation and reported quantity using the sources meaning of a completion to distinguish repeated delivery from separate reports against the same operation. Flagged the suspect records without inventing the answer. Two reports with matching quantities might describe genuine output, so the exception needs enough source detail for the responsible production person to confirm whether both belong in the result. Invalid units should trigger a specific quality condition If an output record carries a weight unit where that operation requires a part count, the check should identify the affected quantity and prevent an unsupported conversion from entering the published measure. Unmatched orders need their impact assessed, not merely counted. A few unresolved records could represent the entire remaining requirement for our customer's shipment, even if they form a tiny share of the plant's daily transaction volume. Make that impact visible to the owner. The useful question isn't simply how many checks passed, but which published conclusions depend on records that haven't yet met the conditions required for their intended use. Lineage should let the owner inspect a challenged result starting from a report. Measure. Identify the semantic model calculation, the transform tables it reads, and the retained source records that supplied the relevant facts for that particular answer. Recorded dependencies alone won't explain a disputed quantity. The investigation also needs the applicable calculation version and processing reference, so the owner can establish which rules operated on which evidence when that result became available. Trace one selected order through that route. During review, someone should be able to identify why a quantity entered the measure, which transformation affected it, and which received record supports it without relying on the original developers memory. Consumers need a shorter account of the data sets condition, publish source freshness, reconciliation status and unresolved exceptions alongside the business result, with wording that tells the planner how those conditions affect the answer they're considering. Freshness should refer to the evidence behind the decision. If execution coverage reaches the latest shift but quality coverage stops earlier, expose both boundaries rather than letting the datasets. Publication time suggest that every contributing source is equally current. Reconciliation status answers a separate question. It tells the consumer whether the prepared quantities agree with the source totals under the agreed comparison rules, including the period and scope that the comparison actually covers. A path check has limits worth stating. Agreement for yesterday's completed shifts doesn't establish agreement for this morning's work, so the published status should identify its coverage rather than present an unrestricted green condition. Describe unresolved exceptions in terms of the affected decision for our order. That could mean a readiness answer remains undetermined for one portion of the requirement, because its completion records still lack an approved order match. The planner should not need to interpret an engineering error message, give them the affected scope, the nature of the uncertainty and the responsible contact, while leaving detailed diagnostic information available to the technical team handling the investigation. Ownership also needs to work after the day team leaves. If the quality pipeline stops overnight and the next shift needs a shipment answer, the published condition should make the uncertainty clear before anyone treats the last available result as current. A name in a catalog won't answer that call. That brings ownership into routine operation, where someone must recognize the failed delivery and take responsibility for getting the right people involved outside office hours, the overnight support person leads a procedure they can actually follow. For our hypothetical plant. That starts with distinguishing an extraction that failed from an extraction that completed successfully, but retrieved nothing because the expected quality file hadn't arrived. Those conditions require different responses. Retrying a failed connection might restore delivery, while repeatedly processing an empty folder won't produce an inspection export that the quality application hasn't generated or released yet. Expired credentials require another response. Again, the support procedure should identify who can renew the connection, how they verify access afterward, and which dependent jobs remain paused until a successful extraction confirms recovery. A changed source schema needs inspection before another attempt. If a supplier renames a required field or changes its type, repeating the same failed activity won't repair the mismatch and could flood support with identical notifications. Classify failures so the response matches the cause. Set bounded retries for temporary interruptions, but root authentication failures and incompatible structures to the responsible team, rather than leaving every problem in an endless retry cycle. Recovery also has to handle work that already completed. Suppose the quality delivery reaches its destination and processing publishes the result, but the job stops before recording success, prompting support to run that delivery again. That rerun mustn't count the records again. Idempotent processing means that applying the same delivery repeatedly leaves the intended business result unchanged after its first successful application, even when execution attempts occur more than once. Implement that behavior around the source's actual update rules. You might match records through stable identifiers and update existing entries while recording completed delivery references, so a repeated file doesn't contribute another set of quantities. An identifier alone isn't enough when a delivery contains corrections, distinguish an unchanged repeat from a newer approved version and test both situations. So duplicate protection doesn't accidentally prevent legitimate changes from reaching the result. Check dependencies before downstream work starts. Our readiness calculation shouldn't proceed simply because the clock reaches its scheduled time. If the required quality preparation has failed or is still waiting for its expected delivery, dependency checks need explicit completion conditions. Confirm that the required upstream work finished for the intended processing period, rather than checking whether some earlier run once succeeded and left a table available to read. 01:28:57 Speaker: A delayed delivery may require a backfill. That's a controlled run over an earlier period with its scope defined, so the recovered quality records update the affected results without unnecessarily reprocessing unrelated production history plan. How that backfill meets ongoing processing, prevent conflicting rights, establish which run takes precedence, and verify completion before allowing dependent publication to resume for the affected period or order. Population alerts should ask someone to do something specific. Include the failed activity affected processing period, last successful run and a link or reference to the relevant recovery procedure. Rather than sending only a generic failure, notice the runbook supplies the next steps. It should explain safe restart conditions, required checks and when to escalate with a named support route for the application team connection administrator or data engineering service involved. Exercise that procedure with someone outside the build team. If they can't distinguish a safe rerun from a case requiring source investigation, revise the instructions before the procedure becomes part of an overnight support obligation. Changes need a separate route into production. Keep development and testing apart from live processing with separate connections and destinations. So an engineer testing a revised extraction can't overwrite the tables supporting tomorrow's shipment review version. The changes that determine processing behavior, store transformation logic and applicable configuration in source control where supported and identify the version deployed so support can relate a new failure to the change that preceded it. Deployment support varies across fabric items. Check the current supported paths, including deployment pipelines or automated interfaces where applicable, and document any configuration steps that don't travel with the item between environments. Test the deployment itself, not just the revised calculation. Verify destination bindings, credentials, schedules, and dependencies in the target environment. Then establish a recovery plan before activating the change against live deliveries from the plant. Reliable recovery at one plant is the starting point. Running these same procedures across several plants introduces another test. Whether concurrent ingestion, backfills, and an ordinary reporting can operate within the resources and costs you've planned. Several plants can create demand that one plants trial never exposed their processing. Windows may overlap, their reporting periods may close together, and their engineering teams may all choose the same quiet hour for expensive maintenance work. I test that overlap before adding more plants, fabric capacity, supplies, resources to the workloads assigned to it. So a storage layout with separate workspaces doesn't itself give every plant an independent allocation of processing power. Build a test around the busiest credible operating period. Include the expected ingestion, table preparation and reporting together. Then introduce a recovery workload to establish how much spare capacity remains when ordinary work doesn't run exactly as planned. Measure which work slows down and when. Fabrics capacity metrics can help investigate consumption delays and throttling. But connect those observations to named activities so you can distinguish an expensive transformation from a report query. Competing with it. The remedy isn't always a larger capacity moving, flexible processing outside a busy window, reducing unnecessary work or separating workloads where justified can change the resource requirements. So compare those options against the service each plant actually needs. Sensor volume deserves scrutiny before it becomes a permanent expense for our orders. Readiness question. Equipment state changes may provide the required evidence, while a process engineer investigating vibration might need a much denser record for a separate purpose. Choose sampling against the behavior you need to detect. A slow sampling interval can miss brief excursions, while a faster interval increases transport, processing and storage demand without necessarily improving the decision that justified collecting the signal. Edge aggregation can reduce what leaves the plant. A gateway could calculate interval summaries locally, but preserve the measurements needed for the intended investigation. Because an average alone can conceal a short excursion beyond an approved process limit. Keep collection and retention decisions separate. You might need detailed events briefly for investigation and longer lived summaries for trend analysis, with different retention periods justified by how people will use each stored form of the data. Apply that reasoning to raw deliveries as well. Estimate their growth using representative files and actual delivery frequencies. Then include repeated extracts and retained versions. Rather than costing only the clean tables that analysts eventually query. Right behavior can consume resources long before storage looks expensive. Frequent tiny deliveries across several plants can create many small files, increasing the work required to discover and read the data needed for each analytical request. Batching rights may help where the delivery window permits it. Measure the delay that batching introduces against the reduction in file count and schedule supported table maintenance according to observed file growth, rather than applying one aggressive schedule everywhere. Partition choices need representative queries, not just production labels. Dividing a table by an order reference or another field with many distinct values can create numerous small partitions, especially when each partition receives little data. Start with how queries actually restrict the data. Test whether a proposed partition scheme reduces the files. Read for common requests without making ingestion and maintenance more expensive, and keep an unpartitioned design among the options for smaller tables. Repeated transformations can hide another recurring cost if several outputs independently scan the same history and perform the same preparation, consider maintaining that shared intermediate result once, then measure whether its storage and update cost. Justify the saved processing. Incremental preparation also needs a measured boundary processing. Only affected periods can reduce repeated work, but the affected scope must include late changes. Otherwise, the apparent saving comes from leaving the published result incomplete across plants. Standardize what helps you compare behavior. Use consistent activity names, resource measurements and acceptance targets while allowing collection intervals and processing schedules to reflect the equipment volumes and operating windows at each location. A common template should leave those choices visible. Compare ingestion, delay records, processed query response, and capacity consumption under each plant's normal demand. Then investigate differences before imposing settings that worked elsewhere under quite different conditions. That evidence gives you a practical basis for expansion. You can identify which workload needs tuning, which timing constraint cannot move, and where extra capacity buys a required service, rather than merely concealing avoidable processing. Pressure to reduce cost also encourages local shortcuts. When teams need faster answers within a shared resource limit, the fixes they choose can quietly separate their data again, even while every workload still runs inside fabric. A local fix becomes a silo when others can't use it reliably. That can happen after a successful fabric rollout when every source connects correctly. But each team still needs its own interpretation before the records support a decision. Copying more sources can make that problem harder to detect. A project can report progress by counting connected applications, even while nobody accepts responsibility for the unresolved references and competing definitions that those applications bring into the shared environment. Watch what happens when an analyst asks for clarification if the response is another source extract, rather than a decision about meaning, the platform keeps accumulating evidence without establishing which interpretation the organization is prepared to use. I change the acceptance condition for that work. A connection shouldn't count as a usable business contribution until the receiving team knows which references require reconciliation, which definitions apply, and who resolves exceptions affecting the intended result. Separate plant workspaces can remain perfectly sensible. The problem starts when each workspace publishes tables that appear interchangeable, while their columns describe different business facts or their measures, calculate different answers under the same name. Imagine someone building a regional comparison from those published tables. They can combine the rows successfully and still compare, unlike results, because the technical structures permit a union that the underlying business definitions don't support. That failure often creates another layer of local correction. The regional analyst adds adjustments outside the plant models, and now the plants maintain their calculations, while the regional team maintains a second interpretation of what those calculations mean. Keep local variation explicit instead of hiding it downstream. Shared data sets need a stated common meaning with documented exceptions where necessary, so a consumer can determine whether combining plant results supports the question being asked. Duplicate measures deserve the same scrutiny as duplicate tables. If several teams maintain a measure independently, someone needs to decide whether they represent one governed definition or genuinely different questions that deserve different names and stated boundaries. Gold data sets can conceal this fragmentation behind reassuring business labels. A table called order readiness may appear authoritative simply because it publishes a clear result, even when its preparation has removed the records that would challenge that result. One dangerous shortcut replaces missing inputs with convenient defaults. If a delayed inspection delivery becomes a zero held quantity, the calculation can interpret missing evidence as the absence of a restriction and present the order as ready. That default changes the business claim the data set has moved from. We haven't received the relevant evidence to the evidence confirms no hold without any additional inspection result or authorized quality decision supporting that change. Removing unmatched records can distort the answer just as quietly. A report calculated only over successfully matched work may look complete while excluding precisely the unfinished or disputed records that the planner most needs to understand before promising delivery require the result to carry its coverage limitations, where missing evidence could change the decision. The published answer needs an explicit, unknown or incomplete condition, rather than a cleaner number that invites an unsupported action. The pressure for a definite answer can also reach generative AI. Asking an assistant to decide which departmental definition governs shipment doesn't resolve the disagreement. It turns an unresolved business rule into a plausible sentence that someone might mistake for authority. Give the assistant approved rules it can apply and explain. If those rules don't cover the situation, its useful response is to identify the missing decision and direct the question to the responsible owner with uncertainty left intact. Automation creates another tempting shortcut when the analytical answer looks dependable. Connecting that answer directly to equipment behavior bypasses the operating boundaries that make analytical delay incomplete evidence and processing failure tolerable in the first place. A successful query isn't permission to actuate equipment. Any proposed physical action needs the approved execution and control arrangements, rather than inheriting authority from the fact that its recommendation originated in a shared data platform. These shortcuts share a practical warning sign local certainty exceeds shared agreement. The table report or assistant produces a definite answer, but the people responsible for acting still need private notes and separate explanations. Before they trusted I'd stop expanding scope at that point. Replace the ambition to unify everything with one bounded operational test where the people responsible for the decision can challenge the complete answer. Instead of merely confirming that their source connected, choose shipment readiness for one product family on one production line. Keep the first test narrow enough that the planner and the people responsible for that line can inspect every disputed answer against the work they know. Narrow doesn't mean choosing only the easy records include the normal variations within that family, especially orders whose readiness depends on both unfinished work and quality decisions. So the trial represents the decisions people actually face. Right? The test question in the planners own terms, ask whether the required delivery quantity is ready at the review time, and require the answer to identify any portion whose readiness the available evidence cannot establish. Then build the complete path for that bounded question. Bring the relevant ERP requirement, mess, progress, machine events and quality records through preparation into a shared semantic model that the planner can query during the actual shipment review. Every included source needs a reason to participate. For machine events, identify which part of the readiness assessment they inform, then check that removing or delaying those events changes only the conclusions that genuinely depend on them. Agree on expected answers before running the acceptance tests. Have the planner, production representative and quality representative prepare a small set of cases with documented outcomes, including cases where the correct answer remains undetermined pending further evidence. Begin with a quantity test that crosses the whole path. Select an order with known requirements and recorded progress. Then compare the models quantities with an independently checked account of the physical work and approved business records. Test selections as well as the overall result. The expected quantities should hold when the planner selects the order alone or views it within the product family without changing merely because related event detail becomes available. For holds, test the boundary of the permission. Use a case where only part of the requirement is affected, and ask the quality representative to confirm exactly which quantity the model must exclude from shipment readiness. Include a case with no dependable hold information. The acceptance condition should require the agreed incomplete answer and the test should fail if the model interprets unavailable evidence as permission to ship the affected parts. A correction test should challenge an answer already published. Enter an approved source correction in the test environment, follow its arrival through the processing path and confirm that the affected quantity changes without altering unrelated orders. Access tests need people from the intended user groups. Ask a permitted planner to obtain the readiness answer and a restricted user to attempt the same query. Then compare both outcomes with the access expectations agreed for this trial. Freshness tests should interrupt one contributing source. Deliberately keep the others current. Allow that source to exceed its agreed age, and check whether the planner receives the required warning before relying on the readiness result. Replay tests should repeat a delivery through the entire path. After the second processing attempt, compare both the published quantities and the readiness conclusion, with the first result confirming that another attempt hasn't created another contribution. Record the starting position before people adopt the new answer during representative shipment reviews. Measure how much time participants spend reconciling records, including the calls and manual checks needed before someone accepts a conclusion. Count unresolved exceptions at the decision point. Separate cases that prevent a shipment. Answer from cases that permit an answer with a stated limitation, so fewer open items doesn't conceal a remaining exception that blocks the delivery measure. Answer latency from the business request. Track how long the planner waits for a usable sufficiently current conclusion. Rather than measuring only the seconds required for a report query to finish after all preparation ends. Decision use needs observation, not a satisfaction score. Record whether the planner actually uses the shared answer to confirm or withhold the delivery promise, and note when they still seek a separate check before acting. Compare the trial against that baseline during normal working conditions. Keep the definitions and review scope stable while measuring the same activities again, so an apparent improvement doesn't come from excluding the awkward orders or unresolved cases. A working trial earns a specific next step once its acceptance test pass, and the planner can use its answer during the shipment review. We can bring our original order into that same tested path and examine the delivery decision it supports. Bring the original order back into the shipment review. This time, ask the planner to defend the delivery answer using the published evidence without collecting a fresh set of explanations from everyone around the table. Start with the quality record that still blocks part of the requirement. Its bronze delivery reference identifies what arrived. While the reconciled silver record connects that evidence to the affected child batch and the correct production order, follow that specific connection into the Order Readiness goal data set. The blocked quantity should contribute to the outstanding release requirement, and the shared business measure should preserve that restriction when the planner selects the customer's full delivery. Now ask why the order isn't ready. The answer should point to the applicable hold and affected quantity, rather than merely display a red status whose relationship to the source evidence nobody can explain during the review. Suppose the outstanding inspection result arrives while that review continues. The result records are passing inspection for the affected batch, but its arrival alone doesn't change shipment permission unless the approved quality process also records the required release decision. Let the authorized release follow through that process. Once its evidence reaches the analytical path, the prepared release status can change, and the readiness measure can reassess the quantity covered by that permission. Production history stays intact. The parts still finish their operations when the execution records reported completion, the later quality decision changes when those parts became eligible for shipment, not when the physical work occurred. That gives the planner two defensible answers across time before the release became effective, the delivery remained blocked. Afterward, the affected quantity met the release condition, subject to the other evidence required by the readiness definition. There's a further distinction. If the release itself arrives late, its effective time can precede the analytical update, so retain both when shipment permission began and when the shared answer first reflected that permission. The planner can now explain the current position plainly. The required quantity meets the recorded shipment readiness conditions as of this review, including the release received during the meeting, and logistics remains responsible for authorizing dispatch. That answer still needs its evidence boundary state, the age of the contributing source information, and identify any unresolved gap that could change the conclusion, rather than letting a current release imply that every other input is current. If a relevant gap remains, keep the answer conditional. For example, a release can clear the quality restriction, while an unconfirmed portion of execution progress still prevents the planner from declaring the entire delivery ready. Don't ask the planner to translate that into technical terms. Tell them which portion remains unconfirmed, which evidence is missing, and whose approval or confirmation is required before they can make the delivery promise. Industrial AI can help explain this changing position given access to the governed facts and their manufacturing context, and the system could identify the restriction that cleared and distinguished it from any condition still preventing shipment. I'd expect it to identify the evidence behind that explanation. If asked why the answer changed during the review, it should refer to the newly recorded release and its scope, without implying that production completed additional parts. Its explanation shouldn't broaden the release permission. Covering one child batch remains permission for that batch, even if the assistant can retrieve records for every other batch associated with the same customer order, a remaining delay may call for a scheduling calculation instead. If unfinished work still threatens delivery, a scheduling engine can test feasible alternatives using the current remaining workload and the modeled constraints relevant to that work. Keep the resulting proposal separate from the shipment answer. A feasible alternative can support an estimated recovery time, but it doesn't establish that the remaining work has occurred or that its future output already carries quality release. The planner can distinguish commitment from possibility. They can state what is ready now, what could become ready under an approved alternative, and which uncertainty prevents a firm promise about the remaining quantity. That's the decision I'd want this architecture to support. Everyone accepts one shipment position for the stated review time with the evidence, unresolved conditions and approval responsibility attached to the answer rather than left in separate conversations. One source of truth needs an owner. It's an agreement about facts, time, and meaning that someone accepts responsibility for including the consequence that a completed order still can't ship while its quality hold remains open. Who owns the definition of done in your plant? I'd like to hear which person or team settles that decision when production, completion and shipment permission don't coincide.