Azure IoT Operations at Scale: Managing Hundreds of Sites With Azure Arc

iFAKT GmbH - Azure IoT Operations at Scale - Managing Hundreds of Sites With Azure Arc 00:00:00 Speaker: Imagine one plant running Azure IoT operations. Well, now you're asked to repeat that setup across hundreds of sites, each with its own production schedule and different people available. When something fails, deployment might be repeatable, but support and recovery often aren't. Who controls change when a central release meets a plant that can't stop? Let's follow what it takes to turn one working installation into a fleet your teams can actually operate. I'd start by defining the service you're supporting. Before we discuss clusters, we need to know which machine data must reach, which people or systems, and what happens to production when that data arrives late. Keep the edge platform running isn't a clear enough service description. It leaves the team guessing whether success means collecting machine signals, feeding a local application, or delivering production records to a cloud analytics service before the next planning meeting. Those jobs carry very different consequences. A service that supports a weekly energy review needs a different operating agreement from one that supplies live equipment status to a shift supervisor deciding where to send maintenance. Let's use a hypothetical packaging plant. The machines run under their own control systems, while a separate data service collects operating states and counts, then sends that information to local users and cloud analytics start with the boundary around machine control. If the data service fails, we need to establish whether machines continue safely under their existing controls or whether any production function actually depends on that service. Don't leave that answer implied. Where production function depends on the data path, the controls and safety engineers need to assess that dependency and define the required behavior during failure. Local data processing carries its own obligations. The shift team might use locally processed counts to track progress during a batch, so losing that information disrupts their work. Even though the packaging machines keep running, cloud analytics can tolerate a different kind of interruption. A planning team might accept delayed records if those records arrive complete later, while a remote operations team might need a clear warning as soon as its information becomes stale, ask what fresh enough means for each consumer in our hypothetical plant, a shift supervisor might need recent machine status. While the team reviewing yesterday's output can work with data delivered after the shift ends, those needs should become explicit. Service targets agree how old a reading can become before it stops supporting the decision, and make sure the people using the data understand that limit before they rely on it. Data freshness and data loss need separate answers. A record arriving late may still support a production review, but a missing record could leave the team unable to explain a gap in the batch history. Some readings lose their use almost immediately. Others need to survive an interruption because someone will later use them to reconcile output, investigate a defect or confirm which operating conditions applied during production. Let the manufacturing use decide the tolerance. If a team requests that every reading survive every failure. Ask which decision or obligation requires that promise because delivering it changes the engineering work and support burden. Then define recovery in terms people can test. Restore the service quickly leaves too much room for disagreement when one team restores collection, but another still waits for the missing production records to arrive. Recovery might mean several milestones rather than one. Fresh readings could resume first, while historical records take longer to catch up, and the service agreement should explain which milestone allows each dependent activity to continue. Planned downtime needs the same care. A plant may accept a brief interruption between batches, but reject the same interruption during a product change. When operators need the data to confirm that the process has settled. Agree. What can pause and when? Include the people who depend on the service. Because an available maintenance slot for the platform team doesn't automatically mean an acceptable gap for production or quality. Now you have a basis for the fleet design. You can describe the service at each plant through its consumers freshness needs, loss tolerance, recovery targets, and permitted downtime, rather than through a count of installed clusters. That changes what you need to learn about each site. The next question becomes which operating conditions could prevent that plant from meeting its service commitments, even when the same software runs successfully somewhere else, a site name tells you very little about its operating conditions. Two plants can run the same software release and still need different support plans because their equipment, network access and available staff place different limits on the service. Build an inventory that captures those limits for each plant. Record the edge hardware model, processor and memory capacity, storage layout, and any constraints that would affect replacing a failed unit with available stock hardware labels alone won't answer the support question. An industrial computer inside a locked production cabinet presents a different repair task from a server in a staffed equipment room, even when both provide enough computing capacity. Record the installed software versions with the same care. You need the Kubernetes distribution and version alongside the Azure IoT operations version. Because Kubernetes installed doesn't tell an engineer which environment they'll find when they investigate a fault. Keep recorded facts separate from assumptions. If nobody has checked the installed version since commissioning. Mark that information as unverified rather than letting an old installation record pass for the plant's current state. Network paths belong in this inventory. To describe how the edge system reaches machine networks and cloud services, including shared links, approved intermediaries and restrictions that could prevent an otherwise familiar deployment from behaving as expected. Then capture what the site actually connects to a plant. Collecting signals from a small set of newer machines presents a different operating pattern from one, depending on several industrial gateways and equipment from different generations. The connection count won't explain all that variation. Record the connection method and any equipment specific constraints. Because a gateway maintained by an outside supplier can introduce a support dependency that a direct connection doesn't have. Local support needs more detail than a contact name. Establish whether someone can reach the hardware during every shift, whether access requires an escort, and whether the person available can diagnose a fault or only report indicator lights. Consider two hypothetical plants with the same edge hardware. One runs continuously with a technician on every shift, while the other has local technical staff only during weekday hours and depends on remote support overnight. Neither site automatically presents the easier operating case. The continuously running plant might provide faster physical access, but almost no acceptable interruption, while the weekday supported plant might provide a regular shutdown but slower help outside working hours. Maintenance windows need their own description record when work can start, how long access lasts, and whether that window moves with production demand. Rather than treating a recurring calendar entry as guaranteed permission. Bandwidth also changes the pattern. A connection that handles normal traffic comfortably might struggle when other business systems compete for the same link. So record shared use and known periods of congestion. Regional restrictions can rule out an otherwise common design. Some sites may face rules about approved destinations, remote access or where production records can travel, and those restrictions need confirmation from the people responsible for them. Use these facts to form tested operating patterns. You might group plants that share a hardware platform. Similar network limits and comparable local support provided those similarities actually predict how the service behaves during normal work and faults. Geography alone is a weak grouping rule. Plants in the same country may need different patterns, while plants far apart may share enough operating conditions to use the same tested approach with a few recorded differences. Give each pattern clear entry conditions. A site should qualify through evidence about its hardware connections and support arrangements, rather than joining a group simply because someone wants to reuse the last deployment plan. Keep exceptions visible and specific if a plant fits a pattern, except for a restricted network route or limited physical access, record that difference and the operating consequence instead of hiding it in meeting notes that inventory exposes worker deployment count. Can't show before another plant joins the fleet. You need agreement about who handles the differences, who confirms the site facts, and who accepts conditions outside the tested pattern. A mixed fleet creates decisions that cross team boundaries. Once you know where the plants differ, the next task is putting names against those decisions. So an unresolved exception doesn't become the next support team. Surprise. I'd assign responsibility around work people can actually complete. Central owns the platform. The plant owns. Production leaves too much unresolved when a repair needs cloud permissions and someone standing beside the equipment cabinet. Central information technology or it should own the shared platform standards. That means naming a team that maintains those standards, answers questions about them, and decides when a proposed site difference needs further engineering rather than routine approval. Give central identity management a named owner to someone needs authority to grant and remove access, with a clear route for requests that arrive outside normal working hours when production support can't wait until Monday? Release tooling needs an operator, not just a project sponsor. The central team should own the shared means of preparing and delivering approved changes, including who can use that tooling and who investigates when it fails. Cloud services also need an accountable service owner. If a destination stops accepting plant data, the site needs a central contact who can investigate that service rather than asking local technicians to prove the machines still work. Site operational technology or OT owns a different set of decisions. The people responsible for equipment access need to decide when a connection can change, which machine specialists must attend and what restrictions apply. Near running production. Production impact needs a local judgment. A central engineer can explain the expected technical effect of a change, but the site must assess that effect against the work happening on the line that day. Local change approval should carry real authority if the site approver can't delay work when conditions change, approval becomes paperwork rather than a decision that protects production from an interruption. Nobody locally agreed to accept physical recovery needs an assigned person and a backup. Central support may diagnose a failed component, but someone must arrange safe access, Perform the permitted physical work and confirm that the equipment returns to the expected condition. For something like a certificate renewal, you need three things clear upfront who owns the trust requirements? Who actually performs the change, and who confirms afterward that the industrial connection still works. That sounds basic, but I've seen it fall apart more times than I'd like to admit. Shared responsibility is one of those phrases that sounds good on paper, but kills execution in practice. You need one person driving completion, otherwise both teams finish their assigned steps. Nobody checks the outcome, and you're left with a broken connection between two tickets that each say were completed. That's not shared responsibility. That's shared ignorance. Network faults need the same discipline when they cross team boundaries. Name the person who coordinates the investigation. When the path runs through plant networks and centrally managed services, it doesn't matter if different teams control different parts of that path. Someone still needs to own the end to end picture data. Quality also needs an owner who understands what the data means on the production floor. Central support can investigate whether records move correctly through the pipeline, but you need a manufacturing contact who can confirm whether the reported machine, state or count actually matches what the equipment did. Those are two different skills and they live in two different teams. Incident handoffs should transfer a next action, not just a ticket number. When you pass work to the receiving team, they need three things the known production impact, the checks already completed, and a specific question. They have the access and skills to answer. If you're handing off ambiguity, you're handing off a problem. I want a named role that can pause a rollout across affected sites, and I want a local role that can stop work at its own plant without waiting for fleet wide agreement. Those are different decisions, and they need different authority levels. Job titles don't tell you who can actually stop the line. Exception approval needs its own boundary. A site might accept a local support constraint. That's their call. But they shouldn't accept a risk on behalf of Central services or other plants that could also face the consequences. Each team owns its own risk acceptance. Closing a failing data connection can require a fast decision. Agree up front who can isolate it, what production checks come first and who must receive notice. This matters especially when the connection affects equipment or services. Beyond the immediate fault, you don't want to discover those dependencies during the outage. Those rights need coverage outside office hours. Record your deputies and escalation routes, then check that the people carrying those roles actually have the permissions, contact details, and local authority to act when called a name on a spreadsheet doesn't help if that person can't reach the equipment room. Ask the named owners to work through a plausible incident together and explain their first actions. Don't just collect approvals from people who interpret the same task differently. Run the scenario. See where the gaps appear. That's how you find out whether your Rasi matrix actually works or just looks good in a slide deck. Let's put that agreement under pressure. At one hypothetical plant, we'll follow a shift where central support and the site each control part of the response, and neither team can complete the work alone. At our hypothetical packaging plant, the line keeps running past the expected changeover. The site asks central support to delay an update scheduled for the local edge cluster. That cluster sits in the plant's equipment room. It handles data from newer packaging machines and older equipment connected through industrial gateways. Local staff use the resulting information to track output during the shift. The cloud connection carries more than production data. Business applications share the same route. Tonight, the shift supervisor notices that a remote production report hasn't caught up with the counts available inside the plant. That's the first sign something's off. Then Central Support loses contact with the site entirely. The engineer preparing the delayed update can no longer reach the management service at the plant. Fresh production records stop arriving at the cloud destination. There's no machine alarm linked to this interruption in this scenario. The packaging machines continue under their existing controls, and the operators confirm that material keeps moving through the line at the expected pace. Inside the plant, the local data service still responds. A technician checks recent machine readings and sees new counts arriving locally, even though central support can't confirm that behaviour through its normal remote connection. The remote report now carries a different risk. Someone outside the plant could read its last received count as current output, unless they notice the information stopped changing before the line stopped producing. I'd keep those observations separate during the first conversation. The machines are running, local readings are arriving, and cloud consumers lack fresh records. But none of those observations explains the state of the delayed update. That update now becomes an unresolved part of the incident. Did central support merely approve it? Did the site receive any instructions or did some step begin before the network connection disappeared from view? The engineer can confirm only part of that sequence. Central records show the change request and the decision to delay, but the team still needs evidence from the plant before claiming that nothing started locally. The site technician has a practical question should anything be touched without a confirmed update state, restarting the cluster could interrupt a working local service while adding another change that both teams would then need to explain. Nobody needs to guess whether a restart might help. For now, the technician reports what remains available locally. While central Support checks its own records, leave the running service unchanged until the response lead has better evidence. Meanwhile, production records continue to accumulate somewhere along the data path. The team needs to establish which component retains records awaiting delivery, whether retention remains active and how much room remains before that component reaches its limit. Data is still collecting. Doesn't answer those questions. Collection could continue while records awaiting cloud delivery approach. A storage limit. The shift supervisor needs a clearer answer before promising that the whole shift can be recovered later. The duration of the outage now affects the operating decision. A short interruption and an overnight loss of connectivity place different demands on the same local system, even while the machines behave exactly as they did before. Local access also becomes more than a convenience. The technician can reach the equipment room, but central support needs to establish which checks that person can perform safely and which require access they don't currently hold. The delayed update can't simply disappear from the conversation. Someone must keep its status visible through the outage. When connectivity returns, the teams need to know whether previously requested work might resume. Recovery needs an owner before the link returns. The person coordinating the incident must bring together the network status, the local observations, and the update status before anyone treats restored remote access. As permission to continue changing the plant, the shift supervisor still needs a plain operational answer. Can the team rely on local counts? Which remote consumers should stop treating their information as current, and who will report if the situation changes before connectivity returns? 00:15:29 Speaker: First we need to follow where machine readings travel, where local work happens, and where information crosses into cloud services. Each observation comes from a different point along that journey. Pick one reading a discharge count from a packaging machine. We'll trace how that reading reaches a cloud consumer, then follow the separate paths that let engineers manage the software and check whether the service works. The first step depends on what the equipment exposes. A supported connector might read from an industrial server, or an existing gateway might collect the machine signal and present it through a protocol. The edge services understand. For example, the gateway could expose an OPC UA endpoint. OPC. UA stands for Open Platform Communications Unified Architecture. It's an industrial standard through which software can exchange equipment data provided both sides support the required functions and security settings. That connection needs a defined destination. Inside the edge application, you configure which source to connect to and which data to collect using the supported Connector and configuration model. For the Azure IoT operations release you've chosen, the reading then enters the local messaging path as your IoT operations includes an Mqtt broker message, queuing, telemetry, transport, a publish and subscribe protocol that lets producers send messages without directly addressing every consumer. Think about the counts destination as a named topic. The connector publishes messages to that topic. An authorized consumer subscribes to the relevant messages rather than asking the machine for another reading each time it needs data. Publishing and delivery are separate steps. A connector successfully sending a message into the broker doesn't establish that a cloud application received it. The message still needs to pass through the configured route toward that destination. Data flows handle the next part of that journey within the capabilities of the selected release. They can take messages from a source, apply configured processing, and send the result to a supported destination. Keep that processing tied to a specific need for our discharge count. The route might select the field required by the receiving application and attach agreed source information, rather than forwarding every available field just because it exists. These services run on Kubernetes at the edge. Kubernetes manages the containerized workloads, but the complete installation still needs a supported combination of infrastructure and software checked against the requirements for that Azure IoT operations version. Now, the count can leave the plant through an approved route. The receiving service might be a supported Azure messaging or storage destination, with downstream processing chosen according to how the business intends to use those records. Microsoft Fabric can sit downstream where that workload fits. For example, a supported ingestion path could bring production events into an analytics workload, where the count joins other records used to investigate output across shifts or plants. The exact route needs checking. Not guessing. A supported destination in one release doesn't establish support for every fabric, workload, or authentication method. The architecture must name the actual services and interfaces between them that describes the production data journey alongside it. A separate management journey carries requests that affect how the edge services run things like applying an approved configuration through the supported management interfaces. Management traffic has a different purpose from machine messages. It can change a connector settings or a routing definition, but it doesn't need to accompany every discharge count as that count moves toward its consumer, monitoring adds another journey with its own destinations, a connector can emit diagnostic information about its connection. The broker and data flows produce signals that help engineers check whether messages progress through the application. A diagnostic message isn't a production record. If a monitoring destination stops receiving telemetry, you've lost evidence about the service, but you still need another observation before concluding that production messages have also stopped moving. Return to the network interruption. With those paths in mind, central Support can ask the technician to check a specific point connector input or local broker activity, rather than asking whether the entire edge system looks healthy. That's the difference between guessing and troubleshooting. Each observation answers a narrower question. A recent machine reading confirms something different from a delivered cloud record, and both differ from confirmation that a management request reached the service responsible for applying it. Those distinctions determine where you place failure boundaries. The next architecture decision is which work depends on services inside the plant and which work requires a reachable cloud management service, because those dependencies decide what an outage can interrupt. So where does the authority to keep working actually live? For each activity at the plant, I'd ask whether the running software can complete that activity using resources already available locally without requesting anything from Azure. That question defines the runtime boundary. A local workload needs it's executable software, it's active settings, and access to whatever services it depends on. Before we can describe its work as independent of cloud management. Location alone doesn't prove independence, though. Software running beside a machine might still call a remote service during normal operation, so placing a workload inside the plant doesn't settle how it behaves. When that call fails, start with the work already in progress. Identify which operations use settings already applied at the site, then distinguish those operations from requests that introduce a new setting or require a remote decision. It gives authorized teams a way to maintain inventory, specify intended configuration control management access, and coordinate deployments across sites without visiting each equipment room. Those functions don't all need continuous contact to remain useful, though a central inventory can retain the last reported state while a site disconnects provided everyone understands that the record describes an earlier observation rather than a current inspection. Desired configuration needs the same distinction. Central systems can hold an approved target for a plant, but that target doesn't become the plant's active configuration merely because someone successfully saved it in Azure. There's a gap between intention and application. A change may await delivery, local processing, or confirmation, and the operating model needs to preserve those distinctions instead of compressing them into one reassuring status. For our delayed update, that distinction becomes practical. The person approving the change needs evidence of what the site actually accepted, rather than relying on the existence of a central request as proof of local execution. I'd use precise words for those states requested, applied, and confirmed. Describe different events, and each should refer to evidence from the component responsible for that event, rather than an assumption based on elapsed time. A running local service may continue using its active configuration when cloud management becomes unreachable within the documented behavior and tested dependency limits of the installed components. That's a bounded claim, not an unlimited promise. The useful question is which specific workload continues under which conditions, and what event would make it need a dependency that's no longer available from the plant? Ask about starting again. As well as staying running, a workload that survives a lost connection could behave differently after a restart, because startup may require access or information that the running process already holds. You need evidence for both conditions. Test continued operation and restart behavior separately within the supported design, then record the result as a limit of that service rather than a general property of edge computing. This also changes how central support interprets silence. If management contact disappears, the accurate statement is that the team can't currently observe or manage the site through that path, not that local processing has failed. The reverse mistake can be just as costly. A A responsive management connection establishes that part of the management path works, but it doesn't establish that a local application completes the work production expects from it. Keep those judgments separate in the service record, one describes management reachability, another describes application behavior, and either judgment needs a time of observation. So yesterday's evidence doesn't quietly become today's assurance. A reachable site can still need application support. Equally, a disconnected site may need no local intervention yet, even though central teams must communicate uncertainty and avoid treating old observations as fresh evidence, that separation gives the site a clearer working agreement. Local staff know which activities can continue under the accepted conditions, while central staff know which management actions require restored contact before anyone can confirm their outcome. The boundary also limits what any management platform can promise. It can help organize and deliver authorized changes, but the running service still depends on the local resources and external dependencies that its design actually requires. That's the job boundary I'd establish before introducing Azure Arc. We can then judge Arc against the management work we needed to perform, while keeping responsibility for local service behaviour explicit, rather than assuming connectivity transfers that responsibility. I'd give Arc a specific management contract. Azure Arc enabled. Kubernetes represents a supported cluster as an Azure resource, giving central teams a common place to organize management without replacing the cluster's own administration. That connection involves software running inside the cluster. Arc agents communicate with Azure through required network paths, and those agents need resources, permissions, and ongoing care. Just like other software you depend on. Connecting a cluster creates another operational dependency to manage. Assign someone to investigate agent failures, track supported versions, and confirm that the connection remains usable after changes elsewhere in the environment affect its access, a Kubernetes extension can deploy and manage software through the Arc extension mechanism, but each extension brings its own requirements and behaviour that the service owner must understand. Extension status needs a specific interpretation confirmation that an extension installed successfully tells you about that installation. While the team responsible for its workload must establish whether the deployed service performs its intended job. Ask who supports each extension before enabling it. Record which team handles its settings, investigates failed operations and coordinates changes when its lifecycle intersects with the cluster or another installed component's lifecycle at that site. Permissions deserve the same attention as software dependencies. The identity that connects a cluster, the agents operating afterward, and the people managing its Azure Resource can require different permissions for different tasks, so assess them separately. Don't solve onboarding by leaving everyone with broad access. Give each operational role the permissions its supported management tasks require. Then verify those permissions against the actual interfaces and scope that role will use for a large fleet resource organization becomes practical engineering. Azure subscriptions and resource groups give you management scopes, while tags can record useful site attributes without forcing every operational distinction into a separate subscription. Choose those boundaries around responsibility and constraints. A regional team might need authority over its plants, while a shared platform team needs a defined scope across regions without receiving unrestricted authority over unrelated Azure resources. Tags help people find the right clusters. They don't grant access, and a label naming a plant owner won't create that owner's permissions or prove that the named person has accepted the support role. Azure role based Access control governs actions within supported Azure scopes. Treat that as a specific authorization layer, then check separately how engineers receive permission for any required work inside Kubernetes, rather than assuming one grant covers everything. That distinction affects support requests immediately. An engineer might have permission to inspect the Arc resource in Azure, but lack permission to examine the cluster resources needed to diagnose a workload problem at the plant. Arc provides an integration point for supported Azure management and governance capabilities, but you still need to choose which capabilities apply and what action each result should trigger. A connected resource isn't automatically governed to your standard. Someone must define the requirements, enable the relevant supported mechanisms, and assign responsibility for findings that require changes outside the management platforms. Direct control. Consider a failed host at our packaging plant. AAC may help support teams identify the affected cluster and inspect available management evidence, but restoring the host still requires an owner with the right access and repair process. Host patching needs an explicit operating arrangement to arc enabled. Kubernetes doesn't turn the machines underneath the cluster into a centrally maintained host service, merely because their Kubernetes workloads now have an Azure management connection, industrial integration remains a separate responsibility. If an equipment supplier changes an interface or a gateway stops exposing a required signal, someone with the relevant industrial knowledge must investigate that connection and coordinate any permitted repair. Plant support can't disappear behind the Azure resource either. Physical access, safe working arrangements, and confirmation from production still belong to named people, even when central engineers can perform much of their investigation remotely through approved tools. I'd write those exclusions into the Arc service description. That gives support staff a clear handoff when a fault crosses from agent management into cluster repair, host maintenance, or an industrial connection that another team owns. Arc then has a clear job within the fleet. It gives central teams a management connection and Azure integration for supported clusters. While the platform beneath that connection still needs a design your team can keep running. If a node fails during production, can you available team restore the service using equipment and skills you can actually provide at that plant? I'd settle that before choosing a Kubernetes distribution. A distribution that suits a central engineering team might create a difficult support burden at a site where nobody routinely administers Linux or replaces server components. Use the Azure IoT operations release as your starting point. Check its current support documentation for the Kubernetes distributions, operating systems, and hardware requirements that apply to the exact version you intend to run in. Production compatibility needs evidence for the whole combination. A distribution supporting a Kubernetes version doesn't establish that your chosen Azure IoT operations release supports that distribution, or that every required component supports your processor architecture. Separate technical feasibility from the support agreement an installation might run during a trial, while still leaving you without a documented support path for the particular combination of operating system, cluster software, and hardware. Here's the thing about difficult faults when the hardware supplier blames the operating system and the cluster supplier blames storage, your team needs one agreed route for coordinating diagnosis. You don't want to collect competing opinions and try to figure out who's right. Read deployment prerequisites as engineering inputs, not checkboxes. Requirements for processor capacity, memory, and disk space. Describe conditions you must meet, but they don't replace an actual assessment of the workload. Your plant will run those numbers on paper, and what happens under real production load are two different conversations. Storage deserves more attention than its advertised capacity. Check the supported storage arrangements and performance requirements for the specific release, then confirm that the proposed disks and storage configuration actually meet those requirements under the expected local workload, a drive that works fine in the lab can choke on a production floor. Networking requirements also reach inside the cluster. Address ranges, name resolution and the chosen cluster networking setup all need checking against the release requirements and plant network design before you commit to equipment or installation work. That's the kind of thing that's cheap to fix in a spreadsheet and expensive to fix. After the rack is wired, some address choices become awkward to change later. If a proposed cluster range overlaps an existing plant network, resolve that conflict during design instead of expecting support staff to untangle it after applications depend on the installation. Nobody enjoys renumbering a production cluster. Choose topology around a specific failure requirement. If the service must tolerate a node stopping, establish which supported arrangement can sustain the required workload after that failure, and which components still depend on shared resources. You need to know what breaks and what doesn't. More nodes don't remove every failure point, several nodes drawing power from one supply or depending on one storage device can still lose service together when that shared dependency fails at the plant. Redundancy on paper doesn't always mean resilience in practice. Check workload support before promising resilience. Kubernetes can schedule workloads across nodes, but the installed services need documented support for the arrangement, including any requirements for replicas, persistent storage, and placement across available nodes. The platform can do it, but can your application physical conditions can narrow those choices. Cabinet temperature, dust exposure, power quality and access restrictions affect which hardware belongs at the site. Even when a general purpose server meets the published computing requirements on paper, a server that works in a datacenter might not survive a packaging plant. Return to the packaging plant's equipment room. Suppose a disk develops errors and the technician needs to replace it. The platform decision determines whether that task requires a routine component swap or specialist work unavailable locally. That difference matters when a line is down. The support plan needs that level of detail record which failures local staff can handle, which need a supplier, and how the plant obtains help when the person who installed the cluster isn't available. Don't leave that to memory or goodwill. 00:31:25 Speaker: This pressure needs its own response instructions. When free space drops, technicians need to know which storage area is affected and which actions are permitted, rather than deleting unfamiliar files to keep the host responsive. A panicked cleanup can cause more problems than it solves. Treat time synchronization as a platform dependency. Check the approved time source and how hosts maintain accurate time. Because clock errors can interfere with trust checks and leave records from different machines. Difficult to compare. Time drift is one of those problems that shows up slowly and then all at once, replacement stock must match the repair plan. The spare computer helps only if its hardware, firmware and storage can support the approved platform, and someone knows how to prepare it for service at that plant. A server in a box isn't a spare until it can actually run the workload. Keep the replacement process within your team's reach. A technically sound design still creates a fragile service. If every hardware fault requires the same remote specialist to guide an unfamiliar technician through an improvised repair, that's not resilience. That's a single point of failure in human form. These choices establish what you can support repeatedly once the hardware software combination topology and repair arrangements hold together. You can turn them into a site baseline without hiding the conditions that make that baseline workable. Document what works and why a baseline should make the permitted differences obvious. When an engineer prepares another plant, they should know which settings stay fixed, which values the site supplies, and which changes need a separate decision. Clarity saves time and prevents mistakes. I define the baseline as a versioned package with stated boundaries. It should describe the shared cluster setup and the settings your team has agreed to support. Rather than becoming a document full of suggestions that each installer interprets differently. Version control isn't just for code. Give every baseline setting a clear purpose. Shared cluster settings should establish where platform workloads belong and which approved defaults apply while leaving equipment specific details outside that shared definition. Wherever the supported configuration model allows. Separate what's common from what's local network rules need that separation to the baseline can define which types of communication are permitted between services, while site values identify the actual addresses and approved paths used at a particular plant. The rule stays the same. The addresses change. Monitoring belongs in the shared package from the start. Define the monitoring configuration expected at every installation, including the site identifier attached to operational signals, so support staff can distinguish plant records without relying on whatever name and installer happened to choose. Consistent naming saves debugging time access settings should follow the same pattern, specify the approved roles and access boundaries centrally, then supply the appropriate site groups through a controlled mapping rather than copying individual user permissions from the last installation. That's how permissions drift starts. Resource limits need deliberate treatment. Shared settings should state the approved workload allocations for a particular site pattern, with changes assessed against that pattern. Instead of letting installers increase limits whenever an application appears slow performance tuning and resource allocation are different. Problems then give site parameters their own defined place. These are the values expected to differ, such as a machine endpoint, an asset reference, or the approved destination for a particular stream of production records. Make the variable parts explicit. Make that parameter set small enough to understand if a plant needs to change dozens of unrelated settings just to identify its equipment, I'd question whether the shared baseline actually separates common behavior from local detail. A good parameter set is minimal and meaningful. Consider the discharge counter at our packaging plant. Its endpoint and asset reference belong to that plant, while the agreed handling of that type of reading can remain part of the shared configuration where appropriate. The what stays shared the where stays local routing needs an explicit boundary around local choice. A site might select from approved destinations or provide a permitted destination reference, but that shouldn't give an installer unrestricted scope to send production records to any reachable service. Controlled flexibility not open ended options. Parameter names need clear definitions as well. If one field means the physical machine identifier and another means the source connection, describe that difference. So the person preparing the site doesn't substitute whichever label appears in the equipment list. Ambiguous names create ambiguous configurations. Credentials require a different handling path. The site configuration should refer to the required credential through the supported mechanism, while the credential itself stays outside the baseline package and the ordinary site settings supplied to engineers. Secrets don't belong in configuration files. Use the secret handling mechanism supported by the installed components. Define how the required secret reaches the workload and who manages that process. Rather than assuming every connector can retrieve credentials through the same method, different components, different approaches, a reference still needs careful handling, even without exposing a password. Pointing a production connection at the wrong secret can cause a failure or grant access. The site never intended that workload to receive a wrong reference is almost as bad as a wrong secret. Some plants won't fit the permitted parameters. Suppose a legacy gateway requires a connection setting outside the shared standard record that as an exception, instead of quietly adding another undocumented option to the common package, make the deviation visible and intentional. Write down exactly what differs and why. Include the affected setting, the operational reason, and the person accountable for the exception. So another engineer can distinguish a deliberate choice from an installation mistake. Context matters for future troubleshooting. Give the exception a review date. That date creates a specific opportunity to check whether the original constraint still exists, rather than letting a temporary workaround become a permanent obligation. Nobody remembers accepting exceptions should expire by default. The exception also needs a tested recovery path. If the special connection setting causes trouble, the responsible team needs evidence that it can restore an accepted condition, including any local action that recovery requires at the plant. Know how to undo what you've done? Keep that evidence with the baseline reference. Someone preparing work should be able to identify the shared version, the plant's permitted values and its accepted exceptions without assembling the answer from separate conversations. One source of truth for the whole picture that gives commissioning something concrete to work against. The installation team can carry a defined configuration into the plant, with the permitted differences already identified and the remaining uncertainties visible before anyone declares the site ready. Commissioning becomes verification, not discovery. Commissioning ends when the receiving team can support the service. An installer completing the deployment is one milestone, but the handoff needs evidence that this particular plant can run the agreed service under its actual operating conditions. Completion and readiness are different states. Start with pre-flight checks from the intended installation location, test network access from the environment that will host the workloads. Because a successful connection from an engineer's laptop might use different routes, permissions or proxy settings test from where it will actually run. Check name resolution for the required service endpoints. Confirm that the site's approved resolution path returns usable answers from that environment, including where internal names and external service names follow different paths through the network. DNS problems are surprisingly common and surprisingly painful. Verify time and storage before starting installation. Compare the host clocks with the approved source, then check that the required storage is mounted, writable and available to the components that will use it. Two simple checks that prevent hours of debugging identity checks should exercise the intended deployment permissions. Confirm that the approved identities can complete their assigned installation tasks without temporarily borrowing broader access that the support team won't have after the installer leaves the plant test with the permissions that will actually be used. Work through the documented endpoint requirements for the selected release record, which required connections succeed through the plant's approved network path and resolve failures before an installation attempt turns a known access problem into a confusing partial deployment document. What works and what doesn't, then follow the supported onboarding sequence for that platform. Use the documented installation and connection steps, retaining enough execution evidence to identify where a failed attempt stopped and which steps actually completed at the site. Logs are your friend. Don't treat a retry as a fresh start before repeating an operation. Inspect the resulting state and follow the documented cleanup or continuation procedure so the handoff doesn't inherit unexplained remnants from earlier installation attempts. A clean retry requires a clean state. Acceptance testing starts with a real source and destination. At our packaging plant, arrange an approved observation of the discharge counter and trace its records through the commissioned route to the consumer that will use them. Test with production data, not test data. Compare the received records with the agreed source observation. Check that the intended machine supplies the records, and that the receiving application can use them, rather than accepting any arriving message as proof that the correct connection works, verify the right data reaches the right place. Exercise the local consumer as well. Ask the shift representative to perform the task. The service supports using the commissioned installation, rather than a temporary test application that won't remain available after hand off to operations. If the operator can't use it, it doesn't work. You need a clear production window before you start restart testing, work through the supported procedures, restart the services or infrastructure listed in the acceptance plan, then verify that connections and data delivery come back without any undocumented manual steps. Record every manual step the test actually requires If an engineer has to restore a setting or supply a credential before data collection resumes, that action belongs in the operating instructions. It also affects whether you can say acceptance conditions were met, test cloud disconnection through a controlled approved interruption. Keep the test boundary explicit so everyone knows exactly which connection is unavailable. Watch whether the local activity is covered in the service agreement. Keep running as expected during that interval, restore connectivity and inspect the result at the consumer side. Compare the observed interruption and the return of delivery against the agreed acceptance criteria. Record any gaps or unexpected behavior. Don't explain them away just because the connection eventually comes back. Capture installed versions from the running environment. Attach the test results, and identify any unresolved defects. The receiving team shouldn't mistake a proposed fix for completed work support. Contacts need to work outside the commissioning meeting. Have the receiving team confirm the escalation route and identify the person who accepts the service. Include any limits or outstanding conditions that acceptance depends on at this plant. Recovery instructions need a practical reading test? Ask someone from the receiving team to explain how they'd use the instructions, where they'd find the required resources, and which step would force them to seek help. Leave the accepted limits beside the recovery instructions. State what the tests demonstrated and what remains unproven. The next shift shouldn't inherit a promise that the commissioning team never actually tested or agreed to support. Once the installation team leaves, the operating team needs a controlled way to preserve that condition. They still need to allow approved changes to reach the plant without losing track of what should be running. GitOps gives the operating team a declared state to work from. You store approved configuration in git, a version control system, and a controller uses that declaration to bring the resources it manages toward the specified state. The controller runs a reconciliation loop. It retrieves the declared configuration, compares that configuration with the resources it can inspect, and attempts the supported actions needed to resolve differences within its assigned scope. That scope needs to be explicit before giving a controller responsibility for anything at the plant. Identify the exact resources it should manage and the interfaces through which it will manage them. For supported Kubernetes resources, flux can perform that work. Azure arc has a GitOps integration using flux, which you can configure to retrieve and reconcile selected configuration on connected clusters within the supported setup. Flux needs its own operating instructions specify its source, its permissions, and the resources it controls, then check that those choices fit the platform and application versions installed at the site. Start with a resource your team clearly owns at the packaging plant. That might be a Kubernetes deployment for a separate local application, provided its lifecycle belongs to your team rather than another installed service. The declaration describes the application's intended Kubernetes state. Flux can apply that declaration and report reconciliation results. The application's own checks establish whether it performs the work expected by its users at the plant. A successful reconciliation answers a configuration question. It tells you that the controller completed its configured work. It doesn't prove that the application produces correct results, or that its external connections behave as intended. Some configuration is managed through Azure Resources and service specific management interfaces. You need to establish the supported management path for each setting before assigning it to GitOps. A Kubernetes object isn't automatically yours to manage. An operator or extension may create that object from another declaration. Editing the generated object through flux can interfere with the component responsible for maintaining it. Follow ownership back to the authoritative declaration. If a service operator owns a deployment, manage the supported input to that operator through its approved interface. Don't give flux competing instructions for the generated deployment. Git can still hold declarations for Azure managed configuration. A separate deployment process can apply those declarations through supported Azure interfaces. Flux handles only the Kubernetes resources deliberately assigned to it at the plant. Those are different delivery mechanisms with different evidence. Your operating instructions should identify which mechanism applies to each resource and where engineers verify its result. Don't present every change as a flux operation to support staff. That means one authoritative management path for a desired value. Even where several components participate in translating that value into running resources inside the cluster, consider a generated workload with a replica setting. If its operator derives that setting from service configuration, while flux repeatedly applies another value directly. Both controllers can keep undoing each other's work without settling the underlying disagreement. The repair starts with ownership, not another retry. Identify which supported declaration should determine the value. Remove the competing instruction and verify that the responsible controller can establish the intended state without further interference. Dependencies need clear ownership as well. If a git managed application references a resource created through Azure Management, document that relationship. Establish which process confirms the resource exists before the application depends on it. GitOps doesn't provide that cross-system sequence by itself. Your chosen delivery process must coordinate the supported interfaces and collect their results. Separate controllers won't infer your complete deployment plan from the fact that they share a cluster for each managed resource, record its authoritative declaration, the controller or deployment process responsible for it, and the place an engineer checks whether that process completed its assigned work. That record also makes repository design less arbitrary. Once you know which declarations belong to flux and which belong to another deployment process, you can organize configuration around how those changes actually reach production, rather than how the folders happen to look. Organize the repository around the change someone needs to approve. If a reviewer can't tell which plants a proposed edit affects without tracing several layers of inherited settings, the structure makes production review harder than it needs to be. Separate shared configuration from the values that select its use. A common application definition can live in one maintained location. Each plant's release record identifies the approved version and supplies the values needed for that installation. Keep environment settings separate from plant identity. A test destination belongs to the test environment, while the equipment references belong to the plant. Promoting a change shouldn't depend on someone remembering to replace test addresses manually. Make the final configuration Inspectable before delivery. Whatever tools assemble the shared definitions and local values. Reviewers should receive the resolved result with a clear comparison against the version currently approved for that plant. Inheritance can hide the effect of a small edit. Changing one shared default might alter many installations. Your checks should identify every affected site configuration, rather than testing only the file that an engineer changed. A shared source doesn't require simultaneous adoption. Each site can reference an approved release explicitly that allows you to maintain common definitions without turning every edit to those definitions into an immediate change across production plants. Avoid references that move without a site level decision. A plant tied to the latest repository contents can receive a different configuration later, without any change to its own release record, making approval harder to trace. That said, should identify the application configuration, its required dependencies, and the compatibility evidence supporting that combination. Don't point engineers towards several independently changing sources. 00:47:03 Speaker: The release needs a stable identity record. The exact source revisions and artifact versions used. Support can retrieve the approved contents later instead of reconstructing them from branch names or someone's recollection. Include compatibility evidence with the release record. That could mean references to documented support requirements and test results for the intended platform combination. Provide enough detail to distinguish a verified dependency from an assumption. Use a concrete change at our packaging plant. Suppose the local count application needs a new configuration format alongside a new application version. Approving either part alone would leave the deployed pair unable to work together. Treat that pair as one release decision. The release record should connect both versions and identify the checks that exercised them together, even if separate delivery mechanisms apply the application and its settings. Review requirements should follow the effect of the change. Editing a support annotation shouldn't demand the same approval as redirecting production records, but a small text change can still alter a destination or remove an access restriction. Assign reviewers who can judge that effect. The platform reviewer can assess deployment implications. Changes to production data handling need someone who understands the receiving service and the permitted use of those records. Automated checks should run before approval reaches delivery. Validate configuration against supported schemas. Check required values and detect unresolved references. Generate the site specific output so reviewers assess what the deployment process will actually submit. Check the affected combinations, not just an example. If a shared edit changes several site variants, the validation process should exercise those variants and report which combinations passed, failed, or weren't tested at all. Protect the approval path as production infrastructure. Restrict who can change deployment workflows and review rules. Someone who bypasses those controls may gain an indirect route to changing workloads across the fleet. Repository permissions deserve periodic review. Remove access when responsibilities change. Separate the right to propose an edit from the right to approve or deliver it where the production risk warrants that separation. Keep credentials outside the repository, including its history. Automated secret checks can help catch mistakes before acceptance, but the release design should require supported secret references rather than relying on reviewers to spot every exposed value. You've got to limit what the delivery identity can actually change, protect how it authenticates, and keep evidence that links each deployment back to an approved release and the person or process that kicked it off. That structure makes approved changes traceable from review all the way to delivery, and it gives support a precise reference when someone changes a setting directly at the plant. Because the running state can then differ from a release, everyone agreed to use. Configuration drift means the running state has wandered away from the approved state. The useful next step is figuring out which value changed, how that difference appeared, and whether it helps or harms the service. The plant depends on a difference. Deserves investigation before correction. Restoring an approved value might fix an accidental edit, but it could also undo a repair that's currently keeping a production data connection. Working through the shift, suppose our packaging plant develops a problem with its local count application. A site engineer changes a timeout through an authorized emergency procedure, restoring updates for the shift team while the underlying cause stays under investigation, that repair leaves a difference that needs a decision. If the setting falls within a controller's reconciliation scope, the controller may restore the declared value. Before the engineer finishes documenting why the temporary change became necessary. The engineer and controller are now working toward different outcomes. The emergency procedure should specify how to suspend reconciliation for the affected resource, where supported, and who can authorize that suspension without stopping unrelated configuration management. Keep the suspension as narrow as practical. Record its scope and intended duration because a temporary pause can otherwise leave unrelated changes undetected. While everyone assumes the controller still maintains the accepted configuration, urgent repairs explain only some differences. A failed rollout can leave part of a change applied, so support has to compare the observed state with the intended release before deciding which values need attention. That situation needs evidence from the attempted operation, establish which resources changed and which didn't. Then choose a supported continuation or restoration path rather than treating every difference as a separate manual mistake. Manual edits need a different conversation. Someone might change a value while diagnosing a fault, forget to restore it, and leave the next shift with behavior that nobody deliberately approved for ongoing use. Ask what the edit attempted to achieve. The answer helps distinguish an abandoned diagnostic step from a repair worth retaining, and it gives the reviewer something more useful than a report naming the person who touched it. Some drift exposes a missing site requirement. Maybe the approved timeout repeatedly fails with that plant's gateway, and technicians keep extending it because the baseline never captured how the connection behaves under actual operating conditions. Repeated corrections won't settle that requirement. Have the responsible engineer assess the local evidence, decide whether the setting belongs in the site configuration, and test the proposed value before making it part of the approved state. Choose reconciliation behavior according to ownership and operational risk for resources your team owns. Automatic correction can work well for well understood changes whose effects are tested, while other differences may need an alert and human review before correction. The decision belongs at the resource boundary. Check what the controller can actually suspend or reconcile, since a plan to protect one setting won't help if the available control poses a much larger collection of resources for operator owned resources, investigate the supported input first. A generated object can differ temporarily while its owner processes a change, so confirm whether the difference represents unfinished work before classifying it as an unauthorized edit. An emergency change needs a route back into controlled operation. Capture the previous value, the temporary value, the reason for changing it, and the observations that showed whether the change helped the plant give that record an owner and a review deadline. The review should decide whether to retain the change through the approved configuration path, or remove it through an agreed action that the site can safely accept if retained. Update the authoritative declaration before resuming reconciliation. Then verify that the controller accepts the revised declaration and that the application still performs its task, rather than assuming agreement between configuration values proves the repair. If rejected, arrange the removal with the plant. Confirm what replaces the temporary repair and observe the service afterward. Because removing a workaround without resolving its cause can simply recreate the original interruption, an overdue temporary change should create assigned work. Track how long differences remain unresolved, and whether the same exceptions keep returning so the fleet team can distinguish isolated repairs from standards that repeatedly fail particular plants. That evidence gives fleet governance a concrete starting point. It shows which differences require a better shared rule, and which require a bounded exception, without forcing the next technician to choose between restoring service and following an unusable standard. For every requirement, specify which supported mechanism evaluates it, what that mechanism can observe, and whether a failed check reports a problem or prevents an operation from proceeding. Azure policy can assess supported Azure resource properties for Kubernetes workload checks. The supported Azure Policy Extension uses admission controls based on gatekeeper, with behavior that depends on the policy definition and its configured effect. Check the scope of each definition before assigning it a rule. Evaluating an Azure Resource property and a rule evaluating Kubernetes workload configuration. Inspect different things even when both results appear within your Azure Governance reporting inventory, and compliance. Answer separate questions. An inventory establishes which resources you know about. While a compliance assessment compares observable properties against a defined requirement and reports the result with the evidence available, neither result authorizes a repair by itself. The operating procedure must translate a finding into assigned work with a responsible team. Deciding how to address the condition within the plant's accepted change boundaries admission checks operate at a different moment. They can evaluate supported requests to create or update Kubernetes resources, and a denying rule can reject a request before the requested resource change takes effect. Existing workloads need separate consideration. Introducing a denying admission rule generally doesn't remove existing resources that violate it, although later attempts to update those resources may face rejection. Under that rule, active correction requires another explicit decision, whereas supported policy effect or separate controller can change resources, establish exactly what it changes and how that action interacts with the resources authoritative management process. Avoid assigning correction based on the rule's name alone. Read the definition and supported effects. Then test the actual behavior so engineers know whether a finding creates information, blocks work, or changes something without further approval. Access controls govern who can perform the permitted actions. Define who can assign policies, alter their enforcement settings, and approve exemptions. Because changing a fleet rule can affect more plants than changing one application policy administration deserves its own approval boundary. Someone permitted to repair a site application doesn't automatically need permission to weaken a shared requirement across the subscription containing that plant and its neighboring sites. Consider a proposed restriction on privileged containers before applying it broadly. Examine the documented requirements of the installed platform components and test whether required workloads need permissions that the proposed rule would reject. Test updates, as well as steady operation. An existing component might continue running without difficulty while its next replacement request fails admission, leaving a maintenance operation blocked by a rule that initially appeared harmless. That test needs the actual supported workload. Definitions include the operations those components performed during their supported life cycle, rather than approving enforcement solely because a simple test application passes under the same cluster policy. Begin with assessment where the policy supports it. Use those findings to identify affected resources and resolve legitimate requirements before moving to denial, with the responsible teams reviewing the evidence for each tested site pattern. A blocked request needs an actionable explanation. Support staff should be able to identify the rule, the rejected operation and the approval route without disabling a broad collection of controls just to discover which one interfered. Exceptions should apply to a defined boundary. Limit the exemption to the relevant supported scope or resources, and record which requirement it relaxes, so other applications don't quietly inherit permission intended for one component. Evidence should explain why the exception exists. Attach the documented workload requirement or test result, identify the accountable owner and set an expiry date that gives the team time to review the condition before enforcement changes. Expiry itself can affect permitted operations. Notify the owner in advance and assess what renewed enforcement would block. Rather than letting a calendar deadline unexpectedly prevent a repair that the plant still needs, review should produce a decision someone can execute. The owner either supplies evidence for an approved extension, or arranges a tested change that removes the need. With unresolved cases escalated before the exemption expires, the next question is how the system establishes, who may request an operation and which services it can trust, especially when credentials or certificates need attention at the plant. Trust has several separate jobs at a plant, an engineer signing in an application requesting cloud access, and a connector verifying an industrial server. Each need different evidence before the requested interaction can proceed. Human access starts with an identifiable person. Use individual accounts for routine work with permissions tied to the task and site scope, so an investigation can establish who requested an action and under whose authority. Separate observation from the authority to change production settings. An engineer checking a connection shouldn't need permanent permission to replace its credentials, and a temporary repair assignment shouldn't quietly become unrestricted access across the fleet. Workload identity answers a different question. Which application is asking where supported use the documented identity mechanism for that component, limiting its access to the destination and operations required for its assigned work at that plant. Check the authentication method for each connection. Support for an identity mechanism in one component doesn't establish support elsewhere, so record what each application uses and how its credentials or tokens remain usable over time. For a certificate based industrial connection, you need to know three things which endpoint presents the certificate, which party validates it, and how an approved replacement becomes trusted without accidentally accepting an unrelated server. That last part matters more than most teams realize. You've got to distinguish the controller's permission to retrieve approved configuration from a person's permission to change that configuration. The reason is simple. Compromising either identity creates a different route into the operating environment, and you need to know which one got hit. Define a supported recovery method that stays usable under the relevant failure conditions. Keep those credentials protected separately and available only to people authorized to perform that recovery, because the moment you need them is the worst time to figure out how they work. Record who obtained emergency access, why they needed it, and which actions followed. Then replace or revoke those temporary credentials where the procedure requires it. Once normal access comes back, maintain an inventory linking each certificate to its service, issuing authority, expiry date, renewal method, and responsible owner. And don't just track the cloud authentication ones include certificates involved in local connections to some components support automated renewal, while others need an explicit certificate, installation or trust list. Update your operating instructions have to reflect the behavior of the version you actually installed. Not the one you wish you had. A server certificate might stay within its validity period while an intermediate certificate expires. That leaves the connection unable to establish trust, even though the most obvious expiry date looks fine and you'll waste hours chasing the wrong problem. The renewal interval should leave room for local approval and recovery work. Set alerts based on evidence that the new certificate actually became active. Not just that, a request completed. Those are two very different outcomes. Suppose a local Connector's certificate approaches expiry while the gateway remains reachable, but the issuing service required for renewal sits beyond the unavailable connection to central systems. That's a specific failure mode you need to plan for. The team needs to establish whether renewal can complete through an approved reachable service, and which connection will fail if the certificate expires before that service becomes available again. Don't assume local access solves everything depending on the protocol and implementation. An established session and a newly opened session can behave differently around certificate expiry. Test reconnection and restart behavior. Specifically, don't just assume what will happen. Replacing an expired certificate with an approved one and updating the relevant trust relationship is a controlled repair. Disabling validation changes who the application might accept as its peer, and that's a fundamentally different outcome. Confirm that the recovery account, its authentication dependencies, and the instructions required for the certificate repair or remain accessible when the usual cloud backed access route can't complete authentication. Otherwise, your recovery plan is just paperwork. Central staff need warning about renewal failures, while local staff need to know which plant servers could stop and what approved action they may perform if remote assistance becomes unavailable. Both groups need the same picture. Once you identify where sign in renewal and trust establishment actually occur, you can specify the network paths the plant must sustain. That's better than discovering a hidden dependency when a certificate expires, and nobody can reach the thing that renews it for every connection crossing it. Name the initiating service, the receiving service, and the permitted traffic before asking the network team to approve a firewall change. If you can't describe what's supposed to happen, they shouldn't open the port. Place the collection services where the approved design allows them to reach industrial end points without turning the edge hosts into an unrestricted route between equipment, networks and enterprise systems. That's a boundary, not a tunnel. Connecting one host to both networks can create unintended forwarding paths. The network and platform owners must explicitly control routing rather than assuming separate cables provide separation because they don't. Equipment collection. Local application access and delivery toward Azure each need their own approved path, even when the same physical edge computer participates in all of those activities. One device multiple distinct responsibilities. That's a separate network zone between plant and enterprise systems, where approved intermediary services can exchange information without allowing unrestricted communication directly across those network boundaries. It's a buffer, not a shortcut. A gateway or relay in that zone helps only when it supports the required exchange, including its authentication behavior and connection direction. Don't introduce an unsupported intermediary into the application path just because the network design looks clean. 01:02:54 Speaker: If a business application needs processed production information, identify an approved service interface rather than giving that application general access to machine endpoints just because the edge host can reach them. Intentional access beats accidental connectivity. Use the current endpoint requirements for each installed component to prepare firewall requests, including destination names, ports, and any documented requirements that affect how those names resolve at the plant. The documentation matters here, where a service publishes endpoint names or supported network controls. Follow that guidance rather than treating the address returned during installation as a permanent destination for every future connection. Things change. Establish which services support the plant's proxy arrangement, including its authentication requirements, and distinguish host level proxy settings from settings supplied separately to workloads running inside Kubernetes at the site. They're not the same thing. If an intermediary decrypts encrypted connections, confirm whether the affected component supports that arrangement and its trust requirements. Don't assume ordinary browser access proves the application will work. Browsers handle certificates differently than industrial components do. Where private connectivity is supported and chosen, confirm that the plant resolves the service to the intended destination and has a permitted route to that address from the relevant workload. DNS infrastructure not magic Azure IoT operations components. Their configured destinations and industrial connections can require different endpoints and protocols. Successful Arc contact doesn't approve or validate those other network paths at the plant. It's one check among many. A request from the host might succeed while a workload request fails. Because cluster networking, proxy configuration or a boundary rule treats that traffic differently along its route test where the code actually runs, not where you type the command. The network team should be able to identify the approved path for its cloud bound count records, without changing the permissions that separate central management traffic from access to industrial equipment. Keep the boundaries clear even during troubleshooting. Decide whether engineers use an approved access gateway, a controlled administrative host or another supported mechanism. Then document which systems they can reach through that particular route. Not all access is equal. Access. Approval should identify the engineer, the task and the allowed duration. Include a way to end the session if production conditions change while the investigation continues. Because you can't predict when you'll need to pull the plug. Provide an approved workstation or console arrangement that doesn't depend on the unavailable cloud connection. Keep local access within the plant's network and physical access rules. Don't create a backdoor just because the front door is blocked. They should know where to connect and which interfaces are permitted without needing to bridge network zones or attach an unmanaged laptop to obtain access. If they can't demonstrate it, it doesn't work. If one approved path stays unavailable beyond the expected repair window, which activities can the plant sustain under the design you've actually tested? And where does that permission end? That's the question worth answering. Blocking every external connection at once can hide which dependency caused a failure, leaving you unable to predict what happens when only one service becomes unavailable. Test them one at a time. Observe whether existing delivery continues and record which management operations stop. Use a controlled test that preserves the other parts needed to distinguish those outcomes. Don't guess. Measure. Check what the configuration controller reports and whether it retains its previously applied state. Don't introduce a new release during this test that would confuse source availability with deployment behavior. A reachable management service won't help a data flow whose receiving endpoint refuses connections, rejects authentication, or accepts traffic too slowly for records to leave the plant. The destination matters as much as the path interrupting the path between a gateway and the edge service's test collection. Failure, where no buffering downstream can preserve readings that never reached a component capable of retaining them. That's a different failure than losing the cloud link. Identify the work that should continue, the work that should stop, and the evidence required to decide whether the observed behavior matches the supported design. If you don't know what you expect, you won't know what broke. An application may initially continue with cached information, then encounter a token refresh, scheduled lookup, or other external request that changes its behavior later during the outage. The first minute tells you almost nothing. Check the documented entitlement and connectivity conditions for each installed product, including third party software. Don't assume a running process has no further dependency on a licensing service. Some of them check periodically identify which component takes responsibility for keeping it when that responsibility begins, and what confirmation allows the previous component to release its copy without creating an unprotected gap. Data handoffs are where things get lost for the selected broker data flow connector or gateway. Verify the supported persistence behavior instead of treating every queue as durable storage that survives process or host failure, test it. Don't assume it. After the interruption, compare what the source submitted with what the destination received. Include records produced immediately before disconnection and while the delivery path remained unavailable. Those are the ones that tell you if your design actually works. Here's the problem most manufacturers don't talk about you already have ERP, Mez, and Azure, but none of it actually talks to the shop floor in real time. Everyone wants a digital twin, but almost nobody has the data model to support one. Think of it like this you can't build a twin on top of data that doesn't even agree with itself across systems. That's not a technology problem. That's an architecture problem. So let's talk about what happens when that connection actually breaks. When the configured retention limit arrives, the component has to do something, reject new input, discard records according to its settings, or apply another documented behavior. The service owner needs to accept that outcome explicitly. Finite storage gives the outage a deadline, and you need to state how the team recognizes that deadline approaching and what approved action follows. You can't just promise continued collection for an interruption whose duration nobody can predict yet. Our packaging plant now has a decision. It can explain if its verified retention allowance runs short before the connection returns. The shift team needs notice of the expected data gap. Production decisions stay with the authorized site rolls, not with the IT team watching dashboards from a different building. Restoring the link starts another part of the test. Previously retained records may arrive alongside fresh readings, so you need to verify how the receiving application distinguishes delayed history from information describing the machine's current condition. Assume retries can expose duplicate handling requirements, check the delivery guarantees of the actual path, then establish whether the receiving system can recognize repeated records using stable identifiers or another agreed method appropriate to that data. Ordering needs a similarly bounded agreement. Records from separate sources, partitions, or retry paths may reach the consumer in a different sequence, so define which ordering the application requires and what the architecture actually guarantees. A returning connection can also overwhelm the destination. Verify any supported controls for draining retained records, and observe whether recovery traffic causes rejection or delays fresh information that another consumer needs for current decisions. Keep the recovery test running until delivery settles. A successful reconnection only establishes that communication resumed. The useful evidence shows which records arrived, which remain outstanding, and whether normal delivery continues. While that outstanding work clears, this connection therefore creates a local resource commitment. The plant must retain the agreed records and sustain the required processing throughout the interruption, then handle recovery without exhausting the same resources that keep new information moving. Size the plant for its hardest accepted operating condition. A system that handles ordinary traffic comfortably, can still run out of capacity when it must collect new readings while processing records retained during an interruption. Start with a workload profile, not a machine count. Measure message sizes and arrival rates from representative sources, including bursts. Because an average taken across a whole shift can conceal the load generated during a busy production interval. Include the processing. Each message requires filtering a few fields places different demands on the system from processing complex payloads, so exercise the actual transformations and destination routes planned for that site pattern. Use realistic payloads in the capacity test. Repeating one small test message might establish a connections throughput, but it won't represent a workload whose records vary in size and require different processing as equipment states change. Measure the whole host as well as individual workloads. Watch processor use memory consumption, disk activity, and network throughput together because a slow storage operation can limit delivery while the processor still appears to have room, destination delays belong in that measurement. Introduce controlled delays within the test environment and observe how resource use changes, including whether waiting requests consume memory or hold connections that other work needs to keep progressing. Recovery needs more processing capacity than normal collection. If the restored delivery path can handle only the incoming rate, retained records won't clear. Even though fresh traffic appears to move successfully through the system, the surplus determines how long recovery takes. In a hypothetical test, if delivery sustains twice the incoming rate, roughly half that delivery capacity remains available for older records, assuming message sizes and processing demands stay comparable. Measure that surplus rather than trusting a nominal throughput figure. The useful result is the rate achieved with live collection configured transformations, and backlog delivery running together under the resource limits intended for the production installation. Leave room for maintenance activity to wear. The supported update procedure temporarily runs. Replacement workloads alongside existing ones include that overlap in the capacity assessment, rather than allocating every available resource to steady operation. Node loss requires a separate load test for a supported design intended to tolerate that failure. Check whether the remaining nodes have enough usable capacity to run the required workloads without waiting for the failed nodes return. Scheduling constraints can reduce that usable capacity. Spare memory elsewhere in the cluster won't help a workload that can't run there because of its placement rules, storage, access requirements, or other documented restrictions on its execution. Storage planning needs more than a retention calculation, account for application, state system overhead and temporary working space keeping an explicit reserve so retain production records. Don't consume capacity required for the platform to function. Check the right burden against the storage specification. Sustained retention and later replay can generate considerable disk activity, so use measured writes and the supplier's endurance guidance to assess the proposed devices. Expected service life test pressure over a sustained period. A short run may finish before queues grow large or storage behaviour changes, while a longer exercise reveals whether collection remains stable as the destination continues falling behind. Back pressure needs an observed outcome at the source. Establish whether a slowed downstream component causes collection to pause. Shifts retained work into another component or creates pressure elsewhere that the initial capacity estimate didn't include. Set thresholds while the test evidence is available. Choose warning points that leave enough time for an authorized response and connect each threshold to the service limit it protects rather than using an arbitrary percentage. Keep availability separate from recovery after loss. High availability aims to keep servers running through specified failures, while backup preserves recoverable information, and disaster recovery addresses restoring service after a disruption beyond those availability arrangements. Each claim needs its own evidence. A cluster continuing after one node stops doesn't demonstrate that loss state can be restored, and a recoverable backup doesn't demonstrate that production data delivery remains available during a failure. Record the tested operating envelope for each site pattern. That record should state the accepted load and failure conditions, giving the release team a capacity boundary before its scheduled work that temporarily changes the demands on the plant. An update request needs to name the layer it changes update. The edge leaves the approver unable to judge whether the work affects the host, Kubernetes and Arc component, Azure IoT operations, or a connector with its own release requirements. Those layers move on different schedules. A host security fix might need attention before the application team plans another release. While a connector change might depend on an IoT operations version the plant hasn't yet approved, give each lifecycle an assigned maintainer that person tracks release notices, support deadlines, and relevant security advisories, then identifies which installed sites need assessment, rather than automatically treating every published version as a fleet deployment request. Automatic update behavior also needs an explicit decision. Establish which installed components update automatically under their supported settings, what control you actually have over that behavior, and how the team learns that a change occurred. Your release plan must account for those independent changes where a component can change outside the application release process, define how its owner assesses compatibility, and notifies the people responsible for services that depend on it. Build an upgrade path, not just a target version. For each proposed change, consult the current support documentation for the installed version and target version, including any required intermediate steps or restrictions on the permitted transition. Check that documentation again before approving. Execution requirements can change between initial planning and the maintenance window, so retain the source and review date alongside the release decision. Rather than relying on an old compatibility spreadsheet, assess the combination that will exist during the work where the documented procedure temporarily leaves components on different versions. Confirm that this intermediate condition is supported, including how long it can remain if work pauses, the required order comes from those dependencies. A Kubernetes upgrade might require preparation elsewhere or an IoT operations update might require a particular platform version first. The documented path determines which action can safely precede another. Turn that path into bounded work instructions. Every step needs an entry condition, an expected result, and a named person who decides whether the next step may begin when the observed result differs from the plan. At our packaging plant, the delayed update now needs reassessment. The original approval covered a particular production interval, and the continuing order means the site must confirm whether the same interruption remains acceptable under the revised schedule. A missed window doesn't create permission for the next one. If the remaining time can't accommodate the supported procedure and its required checks, the coordinator needs a new agreement rather than compressing verification to fit the calendar, site approval should reference the actual work package. Give the approver the affected services expected interruption required local attendance and conditions that would halt the work. So consent applies to a defined operation rather than an open ended maintenance request. Deferral needs a recorded consequence to. If delaying the change leaves a component outside support or exposed to a relevant security issue. Name the risk owner and set a date for reassessment with both teams. Avoid letting either team decide that tension alone. Central security staff can explain the exposure and available mitigations, while the site explains production constraints, allowing the accountable owners to accept a bounded delay or authorise another maintenance arrangement. Completion evidence must extend beyond the installers exit status. The release record should show the resulting versions and whether required workloads resumed without unexpected errors, using observations taken after the change rather than copied from commissioning. Then check the meaning of the delivered records. A connector update could leave messages flowing while changing a timestamp, representation, a field type, or the interpretation of a quality indicator. So validate the fields the receiving application actually relies on. Use an approved production observation to judge those results. For the packaging count, compare the updated services output with the agreed source evidence and confirm that the consumer still interprets the count correctly after the change. Installation. Success, application. Health and correct data, each answer a separate acceptance question. The release owner needs evidence for all the questions included in that work package before signing off. That evidence becomes the basis for wider adoption. Once the dependency sequence and acceptance checks hold at a tested installation. The next decision is how cautiously to expose other plants to the same release. The first production success should earn a limited next step. I'd promote a release through defined groups of sites, with each promotion requiring evidence from the group already running it before another group becomes eligible. Those groups are your release rings. Start in a test environment, move into representative plants, then expand into broader groups whose operating conditions fall within what you've already exercised. With that release, a ring defines exposure, not just deployment order. It determines how many plants can encounter the same undiscovered problem before the release process requires someone to inspect results and make another decision about continuing. Choose early plants for what they can teach you. A site with generous bandwidth and permanent engineering cover won't establish how the release behaves at a constrained plant that depends on remote assistance during production. 01:18:39 Speaker: Convenient plans can give you misleading confidence if every early installation has modern gateways and lightly loaded hosts. The rollout hasn't yet answered questions about older connections or systems operating closer to their tested capacity. Match selection to the uncertainty in the release a change affecting message processing needs representative traffic, while a connector change needs plants using the affected connection types under production conditions that could expose a problem. Not every plant needs to participate early. The release owner does need a written explanation of which operating patterns the selected plants cover, and which patterns remain excluded until further evidence supports their inclusion. Observation needs a defined duration and purpose. Waiting overnight proves little if the affected process runs only during a weekly cleaning cycle, so the promotion gate should require observation of the relevant operating event, keeping membership explicit for each release. A plant suitable for evaluating one change might add little evidence for another, and a permanently assigned pilot group can gradually stop representing the fleet it supposedly protects. An Arc connection gives your tooling a management integration point, but the release process still needs to select eligible sites. Respect approvals, record attempts, and evaluate results before promotion. Your chosen orchestration tooling must enforce those decisions that might involve deployment pipelines and supported management interfaces, with a release record tracking which sites remain unstarted, which are executing work and which await acceptance. Make that record authoritative for rollout progress. Engineers shouldn't have to infer whether a plant belongs in the next batch. By comparing a pipeline history with a spreadsheet that another team updated earlier. Promotion should require more than an absence of complaints. Compare data freshness, relevant error rates, and resource pressure against the agreed release dates. Then obtain site acceptance based on the production activity the change affects. Interpret those signals together rather than in isolation. A temporary resource increase might be expected, but sustained pressure accompanied by older delivered readings, gives the release owner a different reason to withhold the next promotion. Define the stop conditions before delivery begins. Specify which observation blocks an individual plant, which observation pauses its operating pattern, and which observation suspends the release across every remaining eligible site in the fleet. The stop button needs a tested technical effect. Pausing promotion must prevent new deployments from starting, including scheduled or queued work that could otherwise begin after the coordinator believes the rollout has been halted. Already. Executing work needs a separate status. The orchestrator should identify those plants immediately, because preventing another deployment from starting doesn't necessarily cancel an operation already accepted by a management service at a site, our packaging plant might withhold acceptance over stale counts if the same symptom appears elsewhere. After the release, the coordinator should pause the affected rollout scope while site teams establish whether their local services remain usable. A pause should preserve evidence, not trigger repeated deployment attempts, assign an investigator, retain the observed failure and affected release references, and require an explicit decision before the tooling treats the same plants as eligible. Again. Restarting promotion needs, named authority and fresh evidence. The release owner should explain what changed in the assessment and why. The previously failed gate now passes, with site approval renewed where the interruption altered local conditions. Some failed gates won't justify another attempt. They'll force a choice about how to restore an accepted service condition, and that choice needs more preparation than a pipeline button labelled retry before approving a release. Establish whether a failed change allows a documented downgrade, requires restoring configuration, or leaves rebuilding the installation as the supported route. Those routes require different preparations. A documented rollback procedure may return software to an earlier version, while configuration reversal changes declared settings without necessarily reversing anything. The software already changed in persistent storage. Reverting a git commit only reverses the declaration. If the newer application has converted stored records into a different format. Restoring its previous configuration won't convert those records back into something the older application can read. Check reversal limits before the maintenance window. For each stateful component, consult its documented recovery procedures and establish whether the earlier software can use state written after the update, rather than discovering incompatibility during restoration. Schema changes need a recovery decision of their own. If a release changes how stored information is structured, decide whether recovery uses a supported reverse migration, a compatible application version, or restoration from an earlier backup. Restoring earlier state creates a time boundary. Records accepted after that backup might need separate handling, so the recovery plan must explain what happens to them. Instead of silently treating restored software as restored history. Certificate changes can also prevent a straightforward return. If the release replaces a trusted certificate or revokes its predecessor, bringing back an old application image may leave it unable to authenticate with its required peers. Keep certificate recovery within the current trust requirements. Identify how the restored service receives an approved credential, and establishes the required trust without assuming that a copied certificate remains valid or permitted after the change. Full redeployment needs more than installation instructions. The recovery package must contain or reliably reference everything needed to reconstruct the accepted service when the existing cluster or its storage can no longer supply that information. Define the backup scope component by component. Include authoritative configuration required persistent state and the records needed to associate that state with the correct software versions using backup methods supported by the components involved. Identity references belong in that recovery package. To record which identities and secret references the rebuilt service requires, while protecting recoverable credentials through their approved storage mechanism, rather than placing them inside an ordinary configuration archive. Capture the dependencies needed to start recovery itself. The team may need installation artifacts, trusted certificate chains, storage access, and permission to reconnect resources, none of which should depend solely on the installation being repaired. A completed backup job isn't proof of recoverability. Restore into an appropriate, isolated environment and establish whether the recovered components can use the saved state, including any application specific consistency requirements described in their support documentation. Isolation protects the plant during that exercise. A restored connector mustn't accidentally connect to live equipment or publish duplicate production records. While engineers investigate whether its configuration and retained state survive the recovery process. Give the drill an unavailable cloud condition. Ask the team to retrieve instructions and permitted recovery materials through the arrangements intended for that situation. Then, identify which restoration steps genuinely require Azure access before they can proceed. Some steps may have to wait for connectivity. Record that limit explicitly with the resulting service consequence, rather than letting a drill pass because an engineer quietly reconnects a path that the scenario assumes unavailable. Use replacement hardware. In another exercise at our packaging plant, the technician should demonstrate the approved replacement procedure with suitable spare equipment, while the recovery lead checks whether the saved materials actually reconstruct the required service. Leave the original specialist out of the response. Have the designated support team follow the instructions and record every point where missing knowledge, unavailable access or an unclear decision would stop recovery during an actual shift. The restored service needs an acceptance observation for the discharge counter, establish where recovered history ends and fresh collection begins. Then check whether the receiving application can distinguish the resulting records without concealing any known gap. That observation determines whether recovery achieved its intended outcome. Measure the elapsed restoration time against the agreed target, recording delays caused by excess or missing materials, as well as the time spent executing technical procedures. Recovery evidence needs observations from the right points. That brings us to monitoring. The team needs signals that establish whether the recovered data service works, rather than merely confirming that its processes have started again. A useful monitoring check starts with a production question for our packaging plant. I'd ask whether the shift team can obtain a recent trustworthy discharge count, then choose observations that explain why that task might fail. Cluster measurements provide the infrastructure context for that investigation. Watch workload restarts, resource exhaustion, and storage errors, but connect those observations to the affected application rather than declaring a production interruption. Whenever an infrastructure metric crosses a threshold, connect a monitoring should establish whether collection remains active depending on the connectors supported telemetry inspect connection status, subscription failures, and the time of the last successful read. Distinguishing an available server from a source actually supplying expected readings. Here's the problem most manufacturers don't talk about. You can send a message from point A to point B, but that doesn't mean the right data actually arrived. You need observations at meaningful hand-off points to know what's really happening. Where your installed components expose usable metrics, compare accepted messages with forwarded messages over the same interval, and account for any filtering or aggregation that deliberately changes the count, leaving a processing step. Counts alone can hide the wrong traffic. A busy route might carry repeated errors or records from another source entirely, so include checks that the expected asset identifiers and required fields actually appear in the messages. Reaching the consumer delivery needs evidence from the receiving side wherever that's available. Pick a destination, metric, or application. Observation that establishes the required acceptance point because a successful. Send might happen before storage, ingestion, or availability to the person who actually needs the record. Freshness connects delivery back to the production task. Track the age of the latest usable source observation at the consumer using timestamp definitions that let engineers distinguish when a reading originated from. When an intermediary handled it. Keep a separate measure of transport delay comparing source observation time with arrival time can reveal delayed delivery while comparing the latest observation with the current time. Tells the user how old their available information has become. Those measurements depend on understood timestamp behavior. If a gateway stamps receipt rather than machine observation document that limitation. So a precise looking freshness value doesn't imply knowledge about time spent upstream of that gateway. Silence needs context before it becomes an alarm. A machine that publishes only when its state changes can remain quiet while functioning normally. So elapsed time since its last message won't reliably establish whether collection still works. Use a supported liveness signal where the source permits one that might be a periodic connection check or a regularly refreshed status value selected so it verifies communication without requiring the machine to produce another item or change operating state. A zero count deserves different treatment from a missing value. Preserve the distinction in processing and monitoring, because replacing absent records with zero can turn a collection failure into an apparently credible report that the machine produced nothing. Operating context should explain expected quiet periods where an authoritative schedule or machine state is available. Use it to distinguish planned shutdowns and idle equipment from unexpected silence, while retaining checks that the collection service itself remains responsive. Local monitoring must remain usable without cloud delivery. Provide the site technician with an approved way to inspect current service condition and recent diagnostic evidence using tools and storage arrangements that remain available under the accepted outage conditions. Diagnostic retention needs limits of its own. preserve enough local evidence to investigate the interruption without allowing verbose logs to consume space required by the Production data service and verify how those limits behave during repeated errors. Cloud telemetry should complement that local evidence. When connectivity permits, send the selected measurements and diagnostic records to the chosen remote monitoring service. Retaining observation timestamps so delayed telemetry doesn't masquerade as a newly occurring fault. As your monitor can support parts of that design, choose its collection mechanisms according to the telemetry source and documented support for your platform. Rather than assuming one enabled, integration automatically gathers every Azure IoT operations application signal. Other supported tools may fit particular observations better for components exposing Prometheus metrics. For example, verify the supported collection arrangement and metric definitions. Then select the storage and query tools that satisfy your operational and access requirements. Test monitoring with a known controlled fault. Confirm that the observations identify the interrupted stage and that recovery clears the condition for the right reason, rather than simply because a timer expired or a process restarted. Each plant now supplies evidence with a defined meaning. The fleet level challenge is preserving that meaning when observations arrive at different times. So central support can distinguish an observed problem from a site whose current condition remains unknown. At fleet level, the useful unit is an assigned service condition. Bring the observations together around a stable site identity so support can determine which plant needs attention without first translating cluster names into production locations. Build a fleet record that joins operational facts with support context. It should associate the plant with its installed versions, accepted exceptions, management, connectivity, and last known service health. Using identifiers that remain consistent when hardware or cluster resources change. Give each fact an identifiable source. Installed versions might come from platform inventory, while accepted exceptions come from the approval record, and combining them shouldn't erase where either fact originated or when someone last verified it. That distinction helps explain conflicting information. If the release record names one version, while the latest installation observation names. Another create a discrepancy for the responsible owner rather than silently choosing whichever source updated. Most recently. Health needs more than a single fleet wide label. A record can show that local collection remains confirmed while cloud delivery fails, allowing central support to assign work against the affected service instead of declaring the whole plant unavailable. Unknown needs its own explicit condition when observations stop arriving. Preserve the last confirmed result with its observation time, but change the current assessment to reflect that the team lacks evidence about what followed. Don't let yesterday's healthy result remain today's healthy status define how long each observation remains usable for its particular purpose, then make expiry. Change the assessment even if no replacement observation arrives to report a fault. Delayed evidence also needs careful treatment. A batch of telemetry arriving after reconnection describes conditions during its recorded interval, so processing it now shouldn't automatically establish that the same conditions still hold at the plant. Our packaging plant illustrates the practical difference. Central support might know that cloud delivery stopped while a local technician has recently confirmed collection, giving the fleet record a more useful condition than either healthy or offline. Attach that local confirmation to its scope, record what the technician checked and when, so the next person understands whether the evidence covers one discharge counter or the broader collection service they intend to assess. An alert should create a task someone can accept. Include the affected site and service, the observed condition, its production consequence, and the team expected to act with a reference to the evidence supporting that assignment route by failure domain rather than by whichever metric fired first. A destination rejecting records belongs with its service owner, while a local hardware fault belongs with the platform support arrangement. Even if both initially appear as increasing delivery delay. Production impact determines urgency within that routing. A failed route serving an immediate shift decision can warrant a different response from delayed analytical records, provided the urgency follows the agreed service commitment rather than the loudness of an alert. Shared faults need a grouped assignment if several plants lose delivery to the same destination during the same interval. Create a common investigation with the affected sites attached. Instead of opening unrelated investigations that repeatedly test the same service grouping still needs evidence, not just timing. Similar symptoms can have different causes, so retain the individual site conditions and let support separate a plant from the group when its observations contradict the proposed shared explanation. Central support needs a work cue, not a wall of warnings. Present affected services alongside responsible teams, uncertainty and unresolved assignments. Then let engineers narrow that cue by operating pattern or release version. When those attributes help identify scope, the site needs a more immediate account. Its operational view should explain which local activity remains supported, which information can't currently be trusted, and whether anyone needs to perform an approved check or simply await further notice. Neither view needs to expose every collected measurement, give each reported condition a next action or an explicit reason to observe without intervening. So people don't start unnecessary work merely because the fleet record contains an unfamiliar warning. 01:34:36 Speaker: Check that the assignment reaches an available team. An alert delivered to an unattended mailbox hasn't transferred responsibility, so the workflow needs acceptance status and a defined escalation when the intended recipient doesn't acknowledge the work. That's where Fleet Health meets the incident process. Once someone accepts the assignment, central support and the plant need a shared way to investigate the condition without losing its production impact or asking both teams to repeat the same checks. Accepting the incident should start a shared response, not another handoff. The incident lead opens one working record, where central support and the site can contribute observations, decisions and actions without waiting for the other team to finish investigating. First, establish whether production conditions have changed since the alert. Ask the shift representative whether operators have lost a required function, whether any equipment behavior has changed, and whether the current investigation could interfere with running production. Keep that check brief and specific. Then establish what local processing currently does, which cloud paths remain reachable, and which recent changes might affect the fault. Assigning unanswered questions rather than delaying action until every field is complete. Give each team a parallel task with a clear result. Central support can investigate the destination and management history while the site checks the affected local function, with both reporting into the same incident. Rather than transferring ownership between queues, each task needs a boundary and a return point. Instead of asking the technician to check the cluster, request, a permitted observation that answers one question, then agree when they should report. Even if the check produces no answer, the incident lead resolves dependencies between those tasks. If central support needs a local observation before proceeding. Name that dependency explicitly so the technician understands which check matters next and which requested checks can wait. Remote access should follow the incidents approved scope before an engineer connects. Confirm the target permitted actions and local contact. Then distinguish permission to inspect diagnostic information from permission to restart services or modify production connections. Begin with observation whenever the situation allows it capture relevant errors, operation identifiers, and local timestamps before making changes because a restart can remove evidence while leaving the teams unable to explain why the service stopped working. Diagnostic exports can contain equipment details or credentials, so use approved storage and transfer paths, restrict access and avoid copying an entire configuration into a broadly accessible incident conversation. A remote engineer should stop when an action reaches an unapproved system, requires a production interruption outside the accepted scope, or produces behaviour the site hasn't agreed to tolerate. Those limits apply even when the suspected repair looks straightforward. If a technician reports unexpected equipment behaviour during the investigation, the incident lead pauses affected technical work and lets the authorised site personnel assess the production consequence before proceeding. Suppose the network team confirms a failed upstream connection, while the site provides local evidence that no update steps started before contact disappeared and that connection remains usable. The lead can separate restoration from the postponed release. The network owner restores the approved connection. Central support checks the resulting service behavior, and the Site Change authority keeps the update deferred until production grants another window restored. Access doesn't authorize the engineer to resume maintenance. In this hypothetical response, the incident lead explicitly asks the release coordinator to confirm that the plant remains excluded from execution, preventing connectivity restoration from becoming an accidental change trigger. The local recovery authority then checks the plant's acceptance conditions. The shift representative confirms that operators can still perform the affected task, while the responsible engineers verify the recovery evidence required for the interrupted root before declaring that part restored. A failed check needs a predetermined response. If local information becomes unreliable or the repair threatens another approved service, the named authority stops that intervention and selects the next supported recovery action with the incident lead. The lead records the restored functions, any accepted data gap, and the owner of the deferred update giving the next shift a specific account rather than a ticket that simply reads network fixed. Include specialist time. local attendance, supplier involvement and delays caused by missing access or unclear instructions, distinguishing hands on repair work from time spent waiting for someone able to act. That record exposes work in infrastructure. Invoice won't capture if an otherwise routine interruption repeatedly requires several teams and an overnight site visit. Keeping that plant connected carries an operating burden that deployment success alone never reveals. Put those support hours beside the service charges before comparing plants. Otherwise, a site that looks inexpensive in Azure can appear cheaper simply because its most expensive work lands in somebody else's budget check. Azure IoT operations, pricing and entitlement terms separately from the charges for destinations, storage, and whichever management capabilities you enable through Azure Arc for that installation. An Arc resource isn't a complete billing description. List the enabled capabilities and confirm their charging terms, including any applicable licensing benefits, so the estimate reflects your actual configuration rather than a general assumption about Arc record. The basis of that estimate include the region, currency usage assumptions, and date checked because a cost model needs enough context for someone to explain why an invoice differs from the original forecast. Edge hardware needs an agreed replacement cycle, and spare stock ties up money even while it sits unused. Waiting for the failure that justifies keeping it nearby include the work required to prepare replacements, supplier support, warranty arrangements, shipping and permitted installation. Labor can change the cost of restoring service, especially where equipment must pass through local purchasing or import processes. Connectivity has a service cost beyond the existing network contract. If the workload requires a larger connection or a separately funded support arrangement, charge that requirement to the design, rather than assuming the plant network absorbs it for free depending on the chosen route and pricing terms receiving it, processing it, retaining it, and querying it can create different charges that belong to different service owners. Storage cost also changes with how information gets used. Retention duration, access patterns, redundancy choices, and retrieval requirements all affect the estimate, so price the intended behavior rather than multiplying record volume by one storage rate. Collecting a reading more often increases traffic and processing demand, but the right interval depends on the decision that reading supports not merely the smallest interval the equipment permits. Try a hypothetical change at our packaging plant. Suppose an engineer proposes publishing a value every second instead of only when it changes. Even though the receiving application uses that value only during a shift review. Ask what the extra readings. Let that team decide. Price the additional collection, transmission, and retention against that specific use before accepting a default that every similar plant would then inherit through its approved configuration. Sending the same stream to several destinations creates separate processing and retention commitments, so each copy should have a named consumer and a reason to remain in the design. Distinguish intentional copies from accidental duplication and approved archive may justify its cost, while overlapping routes that deliver identical records to the same analytical workload can consume ingestion and storage budget without supporting another task estimate. Telemetry volume, collection frequency, and retention by source. Because detailed diagnostic records can create a different spending pattern from the production messages, the platform exists to deliver match diagnostic detail to an actual support use. For example, temporary verbose logging during an investigation can have a defined end time instead of becoming a permanent fleet setting that keeps generating charges after the incident closes, a plant with lower monthly charges might tolerate longer data delays, while another pays more to meet a stricter commitment than its production users genuinely require. Keep support effort beside that comparison, track routine administration and supplier assistance, as well as incident work distinguishing costs shared across the fleet from costs caused by a particular plant or accepted exception. I'd look for changes that reduce waste without weakening the agreed service. Removing an unused destination can reduce spending cleanly while reducing sampling or shortening. Retention needs approval from the people whose decisions depend on those records. The smaller bill doesn't establish a better design before removing fields, readings or history to cut costs. Ask which production question becomes harder to answer because cheaper data delivery can still leave the business paying for information it can't use. Before comparing their output, you need agreement about what each record describes, because matching field names can conceal differences that change the answer to a production question. Take completed quantity at two hypothetical packaging plants. One plant reports individual packs, leaving the filler, while another reports cases accepted after inspection. So combining both values as finished output would mix different units and production events. Define the event before standardizing its label. Manufacturing teams need to specify which physical occurrence qualifies, where it happens, and whether the reported quantity includes rejected material rework or output awaiting another required operation. Asset identity needs an equally clear definition. A local equipment name can remain useful to operators, while a stable identifier distinguishes that physical asset across plants and preserves its identity. When a network address or application label changes equipment movement creates a practical test. If a packaging machine transfers to another site, your records should preserve its equipment identity while recording the changed location, rather than silently treating it as an entirely new machine. Units should travel with an agreed measurement definition for a temperature reading record the unit and the measurement point. Because converting Celsius to Fahrenheit won't resolve a comparison between bearing temperature at one plant and ambient temperature at another. Quality flags carry, meaning that a number alone can't provide preserve whether a reading meets the agreed validity conditions, and define how source specific quality codes map into those conditions without silently treating uncertain measurements as confirmed observations. Timestamp definitions belong in that agreement. To specify which event each timestamp represents and its time basis, including how local shift boundaries apply. So records crossing midnight don't require different production dates merely because plants use different reporting conventions. That contract describes the fields producers, supply and consumers may rely on, including their types, units, event meanings, and the permitted representation of unavailable information. Give manufacturing ownership of the definitions. Platform engineers can enforce required fields and validate formats, but the production specialists must decide whether accepted output means acceptance at inspection or release after the final quality decision. Each plant then maps its signals into that contract. At our packaging plant, the gateway's local counter name might map to an agreed output event with an explicit rule explaining the physical point and quantity that counter represents. Some mappings need limits rather than invented certainty. If older equipment can report total cycles but can't distinguish accepted packs, retain that distinction and let consumers decide whether cycles answer their question instead of relabeling them as good output. Treat a changed definition as a consumer facing change. If a plant moves its counting point downstream document when the meaning changes and how historical comparisons should work, rather than preserving the field name while silently changing what it measures. Its analytical workloads can combine production events with other business information, but your chosen ingestion and modeling process must carry the contract into the tables and calculations people actually use. A successful join doesn't establish a valid production comparison. Manufacturing teams still need to confirm that compared events represent equivalent work, including any differences in product, packaging or process stages that affect how output should be interpreted. A digital twin can represent the relevant equipment and its operating context, while a knowledge graph describes which equipment performs which process for a particular product. The real value shows up when you need to make a call that depends on those connections. Start with the question those relationships must answer. If a planner asks which approved resource could perform an operation, the model needs maintained links between the product, its process requirements, and the resources qualified for that work. That's the foundation everything else sits on. Industrial AI can then work from a more specific information foundation. An application investigating lost output could retrieve the relevant events and equipment relationships, exposing missing mappings or uncertain readings rather than filling those gaps with plausible explanations, the algorithm isn't the hard part. The data model is. Before treating another plant as comparable, ask whether its records support the same defined decisions. You need mappings that manufacturing accepts and relationships someone has agreed to maintain. The software might install identically, but that doesn't mean the operating model transfers. Expansion needs an acceptance decision about the operating model itself. The question now is whether the people taking over another installation can keep its servers within the agreed limits without depending on the pilot team to intervene. That's a fundamentally different test than checking a deployment checklist. I'd require the receiving teams to present the exit evidence that changes the discussion from whether the project team finished its tasks, to whether the named operators can explain and demonstrate how they'll handle the next difficult shift. The proof is in their ability to respond, not in your deployment report. Ask the receiving team to identify the accepted duration, recognize the approaching service limit, and carry out the agreed response using the access and instructions available to them. If they can't demonstrate that you haven't handed over a working service, you've handed over a problem. Certificate renewal needs demonstrated ownership before expansion proceeds. The evidence should show that the assigned owner completed the renewal process and verified its result, rather than leaving a future expiry as the first real test. That expiry will come and it won't wait for your schedule. An update gate should include a change production schedule. Have the teams demonstrate how they withdraw an approval, prevent Unstarted work from proceeding, and establish what remains in progress when the original maintenance window disappears. That's the scenario that breaks most. Handoffs give the designated responders a failure that crosses their responsibilities, then observe whether they reach an accepted service condition. Without the pilot engineers quietly making the decisions for them. The quiet intervention is the thing you need to catch. A failed gate should narrow the expansion decision. You might approve another plant within a proven operating pattern, while withholding sites whose support arrangements depend on the procedure that failed. Readiness isn't one fleet wide verdict. It's a pattern by pattern judgment. New operating patterns need their own engineering assessment. A plant with a different support arrangement or network restriction shouldn't inherit approval simply because its installation package matches one that passed elsewhere under materially different conditions. Context matters more than consistency. Here, for each newly accepted pattern, identify which tests support its use and who accepts the remaining limits. That way, the deployment team knows where repeatable work ends and where judgment begins. The boundary should be explicit, not assumed. Then measure what happens as the fleet grows. Deployment effort is useful here, especially when you separate routine execution from time spent resolving prerequisites that the site assessment should have identified before installation work began. That gap tells you something about your assessment process. Unresolved drift needs an age as well as a count. If temporary changes survive through several review dates, the operating team may already be carrying more decisions than it can close, even while new installations continue arriving, the backlog compounds silently. A technically quick repair can still miss its service target because access, approval, replacement equipment, or a local decision takes longer than the procedure itself allows. The clock starts when the failure happens, not when the technician arrives. Watch data freshness across normal shifts, not just acceptance tests. If newly commissioned plants repeatedly fall outside their agreed limits during ordinary operation, investigate that pattern before assigning the same design to another group of sites. The acceptance test isn't the operating reality exception. Load tells you something about support capacity count, the review work, and specialist knowledge each exception requires. Because a growing fleet can overwhelm its owners without producing a corresponding increase in incident tickets. The tickets are the symptom, not the cause. When recovery performance worsens or exception reviews accumulate, assign corrective work and require evidence of improvement before adding sites that would increase the same unresolved burden. Growth without capacity is just debt with a different name. Revisit our packaging plant after it passes those gates. During the next disrupted maintenance window, the site approver withdraws permission. The release coordinator confirms the deployment state and the incident lead assigns checks without debating who owns them. The roles are clear because the model is clear, the outage can still cause inconvenience and delayed information. What changes is that? The shift team receives a tested service limit and a named decision maker, rather than waiting for central support to improvise an assurance. That's the difference between a crisis and a procedure. The receiving team should be able to show who takes over when the expected path fails, what that person can authorize, and how the plant reaches an accepted condition. Again, if they can't answer those three questions, the handoff isn't complete. A connected cluster adds an entry to your inventory. An accepted operating model gives the next shift a usable response when deployment, tooling, remote access, or the expected specialist can't provide the help they were counting on. The model is the backup plan that actually works. Someone still owns the next failed change central. It may be ready to release across the fleet, but the site needs authority to protect production and a named service owner accountable for recovery. When that release fails, that ownership doesn't transfer with the deployment package. How do your central implant teams divide? Update approval and recovery ownership when the scheduled window closes, but the work hasn't finished?