Disaster Recovery for Plant Systems: Designing Resilience With Azure

iFAKT GmbH - Disaster Recovery for Plant Systems - Designing Resilience With Azure 00:00:00 Speaker: Here's the problem most manufacturers don't talk about your plant. Servers are back, but production still can't restart. Operators can sign in yet. Work orders won't open, and quality can't confirm which material belongs to the batch waiting at the end of the line. So what actually counts as recovered? I want to work through recovery targets where Azure fits, what can keep running offline, and how you test recovery without putting live production at risk. Because a running server only helps when the shift can use it to do approved work. Let's follow a hypothetical outage through one plant. A power fault takes the plant server room out of service during an active shift. While the production areas still have power and some machines keep running under local control, the operators notice different symptoms at different stations. At one station, an operator can't open the next work order at another. A finished container waits for a label that normally comes from a central service, and nobody knows whether the last print request reached the printer. The shift supervisor calls a tee and asks when service will return. Meanwhile, a batch is still moving through a process that can't simply pause wherever the software stopped responding. So the controls engineer needs to establish what the equipment can safely finish. Those are already different questions. Restoring the server room won't answer whether that batch should continue, whether another order can start, or whether a completed product can leave the plant under the approved operating rules. I'd separate those decisions before discussing a restart time. Otherwise, the phrase production is down hides the fact that different parts of the same line may need very different instructions from the people responsible for them. Start with the work already in progress. The operator knows which job started, but the latest material scan hasn't appeared on the terminal, and the shift can't tell whether the central system accepted it before the outage. The controls engineer checks the current machine, state that check supports a decision about continuing the physical process, but it doesn't establish whether the production record contains enough evidence to account for the material being used. Quality needs to weigh that uncertainty against the approved procedure. In this scenario, the team might permit the current step to finish with controlled recording or require a hold depending on the process and the evidence available. Now consider starting the next order. The planner might know which order comes next, but the operator still needs the released instructions, and a reliable way to confirm that the material at the station belongs to that job. A printed schedule doesn't settle that decision. It can tell the shift what someone intended to run, but it might not reflect a late change that reached the production system shortly before the server room failed. Then there's the finished product waiting near dispatch. The production step may be complete, yet quality still needs the associated batch record before deciding whether those containers can move from a holding area into released stock. A missing label creates another practical problem printing a replacement without knowing whether the original request succeeded could leave the shift with two labels for one container or a label attached to the wrong container. The plant manager needs a clear account of these limits. A useful update explains which work can finish, which stations can't accept new work, and which finished goods must wait, rather than reporting only that several servers remain unavailable. The IT owner needs the same distinctions for a different reason. If the team restores operator access first, that may help the shift read instructions, but it won't necessarily restore order dispatch or let quality retrieve the records needed for release. Nobody here is asking the wrong question. Each person owns a different consequence from equipment, behavior and material identity to shipment commitments, and those consequences meet at the same production line during the same outage. I'd want someone recording those decisions as the incident unfolds. The next shift needs to know whether a batch stopped because of equipment condition, missing instructions, or uncertain records, because each reason demands a different check before work resumes. Listen to how the recovery request changes. Instead of asking it to bring everything back, the shift can ask for the ability to identify current work, retrieve approved instructions, or verify the record for a specific batch that gives the recovery team something concrete to restore and demonstrate. The server still matter, but the plant needs a usable production function with the people responsible for that function able to confirm it works. I'd give each production function its own recovery definition for order dispatch. That definition could describe how an authorized operator selects a released job, assigns it to the correct station, and confirms that production can begin. That description becomes the unit you plan to recover. Its owner needs to account for the software, the terminal at the station, the information the operator receives, and the person who can approve its use. Let's trace order dispatch through the application. Your manufacturing execution system or Ms. might present the available jobs through a terminal, while an application service checks order status against records held in a database. The request has to complete the whole journey if the terminal displays a job but the service can't record its assignment, the operator could act on a selection that the rest of the plant never receives. A database check alone won't prove that function works. You need to exercise the supported workflow from the operator's position, through the application, and back to a confirmed result that the next production step can use. That changes how you write the acceptance check. Rather than recording that the Ms. started successfully, you record that an authorized operator dispatched an eligible order to an approved resource and retrieved the resulting assignment. Keep the test specific to the function. You don't need to prove every Mez capability before restoring order dispatch, but you do need to prove every dependency required for dispatch to work as intended. People belong inside that definition, too. If the function requires a supervisor to resolve an exception, your recovery procedure needs an available supervisor who understands the recovered state and has authority to act. A shared login doesn't replace that role. The question is whether the right person can perform the approved task with the right permissions, including any checks that normally separate an operator's work from a supervisor's approval. Enterprise software can create a false sense of progress here. Your enterprise resource planning system, or ERP may show the order as released. Even while the shop floor lacks the application state needed to execute it. Both records can be correct within their own scope. The ERP release may authorize production from a business perspective, while the mess still needs to establish the operation assigned resource and execution status for that order. So define what the handoff must accomplish for this plant. Successful recovery might require the released order to appear in the mess with the correct revision and become available at the intended station. The order's presence alone isn't the acceptance condition. You'd also check that the station can accept it through the normal workflow without someone editing records directly, just to move past an error. Now put a boundary around that recovered function and assembly line using fixed work instructions may need different restart evidence from a mixing process where the application supplies a controlled recipe for each batch. Recovery approval should reflect those differences. You can define conditions by line, process or product family where the operating rules differ. Rather than treating every station connected to one server as equally ready, those conditions need agreement. Before the outage, the application owner can explain the technical checks, while production and quality define the evidence required to put that function back into service. For the intended work, I'd capture the boundary in plain language. A useful statement names the function, the work it covers, who can use it, and the conditions that must hold before its owner accepts recovery. It should also state what's still restricted. Restoring dispatch for one product family shouldn't quietly grant permission to start another family whose instructions, approval, workflow or resource checks haven't passed the agreed acceptance test that gives you a workable basis for setting targets. You can now ask how long this particular function can remain unavailable, and how much missing information its owner can tolerate before the recovered workflow becomes unusable. Those are separate limits, and each needs a clear definition until you attach them to an agreed production function. A recovery target can describe a server very precisely while leaving the shift. Unsure what it's allowed to do? The first limit concerns time without the service recovery time objective or RTO sets the target elapsed time to restore the agreed function measured from the starting point. Your recovery policy defines that starting point needs to be explicit if production counts from the moment dispatch stops while the service owner counts from the decision to recover, they're working toward different deadlines under the same label. The second limit concerns missing information Recovery Point objective, or RPO. Sets the target limit on lost data expressed as a period measured backward from the disruption to the recovery point. Think of a hypothetical fifteen minute RPO for production transactions. That target means the design aims to recover records to a point no more than fifteen minutes before the disruption, rather than allowing fifteen minutes to restore them. These objectives answer different questions. You might restore an application quickly with records from yesterday, or recover nearly current records, while leaving operators waiting too long to use the application again. Neither target describes a guaranteed result. Each gives the system owner a requirement to design against with the achieved recovery time, and recover data later checked against that requirement for the agreed failure scenario. I'd set those requirements separately for each system and function. Putting one recovery target against the entire plant can hide a reporting service with plenty of tolerance beside a transaction service that production needs almost immediately. Take the historian, which stores time based process measurements. Its recovery time target should reflect when people need those measurements again, whether for process review, fault diagnosis or evidence associated with a the production run. Its recovery point target needs a separate discussion. A gap in an ambient temperature trend might have a different consequence from a gap in a process temperature record needed to assess a batch. Missing samples aren't automatically acceptable or automatically disastrous. The process owner and quality team need to establish which records support production decisions, and what loss would leave those decisions without the required evidence. Mez transactions create a different kind of exposure. A missing material consumption record can leave the plant unable to establish which material lot entered a batch. Even when every machine completed its assigned step, the duration alone can mislead you. Here, losing a short interval containing one material change could create more uncertainty than losing a longer interval, during which no production transactions occurred at all. For genealogy, the record linking material to product, the agreed requirement might allow no loss of committed transactions. That demands a design capable of meeting it, not just a hope that the system will behave. That's a demanding requirement, and it needs a clear boundary define which acknowledged transactions it covers and which failure scenarios the design must withstand. Rather than treating zero data loss as an unlimited promise that nobody actually tested, recipe services deserve their own targets, too. If a line needs the service before each new batch, the recovery time target depends on when the next approved recipe must become available for use, not on some arbitrary clock someone picked during a planning meeting for recipes. Age alone doesn't establish correctness. The recovered state needs the approved version and its release status, because retrieving an older recipe successfully could still leave the next batch without valid instructions, and nobody wants to explain that to production. Business reporting may tolerate a longer interruption if a report supports a weekly production review, rather than an immediate operating decision, its owner might accept slower recovery and older data than the mess owner can, which means you've got room to prioritize other things. First, check what people actually use the report to decide. A report named Daily Performance could also inform a same shift action, so the label tells you less than the consequence of losing access to it. What happens if that report goes dark for two hours? That's the real question. Record the reasoning. Beside each target, the service owner needs to know whether the limit protects batch evidence, prevents a shortage of executable work, or simply keeps a management review on schedule after an outage. Without that context, the number means nothing. Conflicting targets then become easier to discuss if the historian target permits a data gap that quality can't accept for a particular product, you found a requirement conflict that needs resolution before recovery begins, not during the incident when everyone's already stressed. The targets still need a clock the whole plant shares. Next, we need to account for the time that passes around the restore itself, because the shift experience is the full interruption, not just the period when recovery tools run. That distinction matters more than most teams realize. In our hypothetical outage, the first failed request might appear at a station before monitoring confirms the wider fault. The incident record should preserve both times, because that gap tells you something about how quickly the plant actually knows it has a problem. That gap belongs in the outage timeline record. When the function first became unavailable, when someone recognized the disruption and when the responsible team received enough information to begin acting on it. Three timestamps, not one incident declaration creates another measurable interval. Someone needs to decide whether this is a brief interruption for the support team to investigate, or an incident that calls for the recovery procedure and its assigned decision makers. That decision takes time, and it counts. Diagnosis can consume the time reserved for recovery. A team might spend most of the available window trying to repair the original environment, then discover that restoring elsewhere still requires the full expected duration. Now you're over target. Before you even started the restore, set a decision checkpoint. Before that happens, the procedure should say when the incident owner must review progress, what evidence supports continued repair, and who can authorize recovery if the repair estimate remains uncertain. Don't leave that call to intuition under pressure. Approval delays need their own place in the record. If the recovery team waits for permission to proceed, capture when it requested that permission, and when the authorized person granted it without quietly removing the wait from the result. That delay is real and it belongs in the numbers. Consider a hypothetical two hour interruption if the restore tool runs for thirty minutes. But detection, investigation, approval and later checks consume the remaining time, reporting a thirty minute recovery would misrepresent what production experienced. The plant was down for two hours. Record milestones rather than one convenient duration. You want to distinguish the point when infrastructure becomes available from the point when the application responds correctly, and both from the point when production accepts its use. Those are three different moments. Infrastructure recovery is an intermediate milestone. A successful server start gives the application team an environment to inspect, but the team still needs to establish whether the intended service is start and perform their supported operations. The server being up doesn't mean the application works. Application checks should produce evidence, not just reassurance. During an exercise record which checks passed, which failed, and how long the team spent resolving faults before it could move on to checking the recovered information. That detail becomes the basis for improvement. Data checks can extend the interruption further. The database might open normally, while the application owner still needs to confirm that the recovered records reach the expected point and remain usable through the supported workflow. That confirmation takes time. Production approval adds a separate handoff. The person accepting the function needs the test results and any restrictions clearly stated. Rather than receiving a message that simply says the technical team has finished. Vague handoffs create risk. Measure how long that handoff actually takes. If approval routinely waits because the evidence lacks detail or the responsible person can't be reached, the recovery estimate needs to include that delay until the process changes. Don't assume it away. A scheduled backup every fifteen minutes tells you when jobs should run, but it doesn't establish the age of the latest usable recovery point during an incident. The schedule is a plan, not a guaranteed job. Completion isn't enough to settle that question. Check what point in the application's history the backup can restore, whether the required backup sequence remains intact, and whether the team can actually recover from it. Those are three separate checks. A successful schedule can still conceal an older, usable point during a restore test. Compare the last verified recovered record with the simulated disruption time, then explain any gap rather than substituting the backup schedule for the measured result. Measure what you actually get. Supplier estimates need the same treatment. A quoted restore duration may assume a particular data volume. Prepared resources and experienced staff. While your plant exercise includes the conditions your own team will face. Those conditions are different and the numbers will be different. To use those estimates to plan the first exercise. Replace assumptions with measured durations as tests reveal how long retrieval restore checks and approval. Take and retain the test conditions beside the results, so comparisons remain honest. That record becomes your baseline. A missed target calls for a specific response. Assign work to shorten the measured delay. Change the recovery design or seek formal acceptance of a revised target instead of treating the test result as an administrative detail. A misses information, not failure. The elapsed time now points toward concrete work. When a recovered application sits waiting for another service before its checks can begin, that waiting period tells you where the next part of the recovery plan needs attention. Follow the delay. 00:15:47 Speaker: The Mez login page might load without errors, while every sign in attempt fails because the recovered application still points toward a directory server in the unavailable plant server room. The symptom says application broken, but the cause is something else entirely. The error message might send support in the wrong direction before anyone reinstalls software or changes application settings. The recovery team needs a record of which supporting services that application expects, and where those services run. That record saves hours of guessing. Start that record with the people who maintain the connections. IT and OT teams need to compare their records because application ownership rarely matches the full path. A request follows through the plant. Nobody owns the whole path, which means nobody documents the whole path. Ask each team to describe one actual request for an operator signing in attestation. Identify which service checks the account, how the terminal finds that service, and which network connections. Let the request reach it. Trace one request. End to end name resolution deserves an explicit entry. A terminal can have a working network connection, but still fail to find its application because its DNS server no longer answers requests. The network is up, but the name doesn't resolve. Using a server address directly isn't always a safe workaround. Applications may depend on registered names for certificates or authentication, so replacing a name with an address can create another fault. While appearing to fix the first, you fix nothing. Time belongs in this record to authentication and certificate checks can fail when clocks drift beyond accepted limits, so identify the time sources used by servers and plant devices, including their behavior when those sources disappear. Time sync failures are silent, and destructive certificates introduce dependencies that aren't obvious from a server inventory. An encrypted connection may require a trusted certificate chain and access to certificate status information. Depending on the application, its security settings, and the checks it performs that's not visible from a port scan. Record the requirement rather than guessing from normal behavior. A service that starts during routine operation might fail after recovery. Because a required certificate expired, its private key is unavailable or its account lacks permission to use it. Normal operation doesn't test these edge cases. Network routes need more detail than the sites are connected for each required connection. Record the source destination, permitted traffic and responsible team, including any firewall or routing service that must function before the application can communicate. Vague network diagrams create vague recovery plans. Some blockers sit outside the main application contract. A recovered production service might refuse its normal workload because it can't contact a licensed server, even though the application files and production database appear healthy. The license server is the hidden dependency service. Accounts need checks separate from operator access. The background account. Running an interface may depend on a particular directory, stored credential or permission that. An operator's successful sign in tells you nothing about two different paths. Two different failure modes follow the request beyond the application as well. If the recovered workflow requires a label, include the print service, printer connection, and approved label files. Rather than assuming a successful application response means the station can complete its task. The application works, but nothing prints. Vendor support can depend on the same missing services. If a specialist must diagnose the application, identify the approved access path and who can authorize it because an unreachable access gateway can leave both teams waiting. Nobody can fix what nobody can reach. Record where it runs, what it needs, and how someone verifies it works with enough detail for another qualified person to follow the connection during an incident. That level of detail separates a useful document from a decoration. Then check for loops in those requirements. Suppose the recovery instructions sit in a document service that requires the same sign in service. The instructions explain how to restore, and the team has no approved copy elsewhere. You can't read the instructions because the thing you're recovering isn't available yet. That loop needs a practical way out. Keep controlled recovery instructions accessible through a separately tested arrangement, with appropriate protection for sensitive details and a clear process for keeping those instructions current after changes, break the loop before you need it. The same check applies to troubleshooting tools. If network configuration records require access through the failed network, the team needs an approved way to retrieve them before it can investigate why that network remains unavailable. You can't fix the network without the network. Finally, record which dependencies, share a location or control system that lets you distinguish a service with its own recovery path from several services that all disappear together when one shared part of the plant fails. Shared fate changes the recovery strategy. That shared boundary determines what local availability can actually protect. High availability keeps the service usable through selected component failures, usually by providing spare capacity and a supported way to transfer work when a component stops responding. But selected is doing a lot of work there. The word selected deserves some attention here. A design might tolerate the loss of one compute host while still depending on storage, power and network equipment that the surviving host can't operate without. That's not high availability for the plant. That's high availability for one piece of a much bigger puzzle. Consider a cluster running the plant application on two hosts. If one host fails, the other may restart or continue the workload depending on the application design and the availability mechanism the vendor supports. That can reduce disruption during a host failure. But here's the thing. If both hosts use the same storage system, losing that storage leaves the cluster with nowhere to read or write its production records. Two hosts don't automatically create two independent places to operate. The protection comes from the failures the design can actually withstand, not from the number of machines listed in the purchase order or installed in the rack. Disaster recovery addresses loss beyond that local protection boundary. It provides a way to restore service when the environment that normally keeps the application available can no longer support it within the agreed operating conditions. The distinction depends on the boundary you've designed around. One plant might have a local cluster within a single server room, while another spread supported services across separate buildings with independent infrastructure and carefully defined dependencies. Neither arrangement earns protection just from its name. You need to establish which parts remain usable after the protected environment fails and what the recovery team must supply before the application can operate somewhere else. Let's go back to that hypothetical server room outage for a moment. A second application host inside that room wouldn't help if the fault removed power from both hosts and the storage equipment they share. A cluster can handle its assigned task perfectly and still leave production waiting. Its local failover mechanism might work exactly as designed, but the incident has removed more of the environment than that mechanism covers. This also changes how you discuss investment. Local availability spends money to keep service running through defined faults, while disaster recovery spends money on the ability to restore service after a wider loss. Both can belong in the same design. A plant may use local redundancy to avoid frequent interruptions, then maintain a separate recovery path for events that remove the whole local environment from service backups protect another part of that design. A backup preserves a recoverable version of information with retention and restore methods that let you return to a saved state when the current state can't be used. Replication maintains another copy. As changes occur depending on its design, it can reduce the information that must move during recovery and provide a destination from which service can resume after an interruption. The two methods answer different needs. A replica can help you resume from a recent state. While retained backups can give you earlier states that the current replica no longer contains. Neither method supplies the operating procedure by itself. Someone still needs to select the intended copy, restore or activate the supported application, and establish which environment now owns the work that production will use. That's where recovery procedures connect the protection methods. They turn stored copies and standby resources into repeatable actions, including who acts, what evidence permits the next step, and when the team must stop. I'd assess those methods together when reviewing a plant design for each application, describe what local redundancy keeps available, what replication preserves elsewhere, and which retained backups remain usable. If the current copies aren't suitable, then state the limit of each protection. If the local cluster shares power storage, directory services or a server room, its protection statement should name those shared dependencies rather than imply independence that doesn't exist. The next question concerns the loss you're designing to survive. Once you separate local availability from recovery, you can examine each proposed failure and asked whether the remaining equipment, copies and procedures actually provide a usable recovery path. I'd start with separate incident scenarios, not one generic disaster for each scenario. Name what disappears, what remains trustworthy, and where the recovered service could run. Because those answers can change even when operators report the same symptoms. A failed server gives you a fairly narrow starting point. If the fault affects only that server, the team can investigate its replacement or recovery without assuming that the production records surrounding network and alternative environment also need rebuilding. Storage corruption changes the decision. The application might still respond while reading damaged records, so activating another environment only helps if the copy selected for recovery contains information the application owner can still trust. That requires a different question from which server stopped. Ask when the records first became unusable and which recovery points predate that condition. Rather than assuming the newest available copy represents the best place to resume work. Sight loss removes assumptions that survive a smaller equipment fault. In a hypothetical incident that makes the whole plant inaccessible. Your recovery team may also lose its usual workstations, local communications, and access to people who normally verify production conditions. The recovery destination then needs more than spare compute. Someone working from an approved alternative location needs to reach it, assess its state, and make decisions without relying on equipment or records inside the inaccessible plant. Geographic distance alone doesn't establish independence. Two buildings can depend on the same electricity supply, while two network contracts can still carry traffic through the same underground duct before their routes separate farther away. Administrative changes can cross site boundaries to a mistaken configuration. Pushed to both locations can disable both environments, so the failure review needs to examine shared management paths alongside the physical services connecting those locations. Network isolation creates a different problem. Again, the plant and its central applications might both remain healthy, but neither side can reliably determine whether the other side has accepted the latest request or remains available. Moving the application may leave that problem untouched. If plant terminals still depend on the broken connection to reach the recovery destination, a successful failover won't restore the production function that operators actually need. I challenge every proposed destination with that connection question. Ask which surviving route lets the intended users reach it during this particular incident, rather than testing access only under normal network conditions and assuming that result carries over a cloud disruption reverses some of the assumptions if the plant remains available. While an Azure region hosting its applications becomes unavailable, the recovery path needs a destination outside the affected services and their dependencies. That destination needs advanced assessment. It could involve another supported region or a separately designed local arrangement, but you can't assume that an application can move back into the plant merely because it originally ran there. A cyber attack introduces uncertainty about what remains trustworthy. The team needs to treat suspected compromise as a different recovery scenario, because restoring access quickly can reconnect production to an environment whose accounts or software remain under hostile control. The immediate choice may be containment rather than activation. Recovery planning should identify when incident response findings require a clean restore instead of failover, while leaving the detailed security decisions to the people responsible for investigating the attack. Some incidents justify a controlled shutdown if no available environment can support the approved work with trustworthy information, the operating procedure needs a defined stopping decision instead of leaving the shift, waiting indefinitely for another technical attempt. Right. Those choices against the individual scenarios. A contained equipment failure might permit failover. Damaged records might require an earlier clean restore, and an unresolved loss of trustworthy services might require production to stop under its approved procedures. Then test combinations that could defeat your preferred route. A plant outage accompanied by a failed carrier connection deserves its own review. If that connection provides the only access to the Azure environment intended to receive the recovered application, software recovery still has a physical boundary even when a surviving destination can run every recovered service, the team must separately establish whether the equipment those services support remains available and fit for its intended operation. A controller needs its own recovery assessment. A programmable logic controller, or PLC executes the machine logic, and its response to missing server communications depends on the program, connected devices and engineered operating conditions. That behavior needs verification for each affected machine. In our hypothetical plant, a packaging machine might finish its current cycle after losing the server connection while another machine waits for a fresh instruction before advancing. Neither response tells you whether automatic restart is permitted. The controls engineer needs to establish what the program retains, which communication faults it detects, and what conditions must hold before the sequence can proceed. Retained values deserve particular care. During that check, a controller could preserve a step number through a power interruption, yet the equipment may no longer occupy the physical position that the stored step assumes. The restart procedure needs to resolve that mismatch. Depending on the machine design, a qualified person may need to verify positions, clear trapped material or follow an approved recovery sequence before normal operation resumes. Safety functions follow their own engineered rules, where a safety controller or protective system prevents hazardous motion, its reset conditions and required checks remain separate from the recovery status of the production application. A recovered server doesn't grant permission to bypass a protective function. The responsible people must follow the machine's safety procedures, including required isolation, inspection, and validation before anyone attempts work that could expose people to hazardous energy. Keep remote recovery tools within that boundary to an administrator who can restore software shouldn't automatically gain authority to reset equipment faults or issue commands that could cause motion while someone inspects the machine. I would document those authority limits beside the technical procedures, the recovery team needs to know which actions belong to control specialists, which require local confirmation and which need approval under the plant's safety rules. Now consider the engineering files needed after equipment failure. A controller backup should belong to a controlled recovery package that identifies the machine. The approved program revision and the engineering environment needed to use it. A file name isn't enough evidence of its contents. The controls team needs a supported way to establish that the saved project matches the approved running program, including changes completed during commissioning or later maintenance. Drive settings need the same attention. Replacing a failed drive may require its motor parameters, communication settings, and application specific configuration with the correct versions and any vendor defined commissioning checks completed before use. A replacement controller might arrive with a different firmware version, so your recovery package needs to record which combinations of hardware, firmware, project files, and engineering software actually work together. That's the kind of detail that saves you from discovering the problem at three a m during a restart. Some projects also require protected credentials or vendor tools. Authorized specialists need an approved way to get those items when recovering equipment, without depending on one engineer's laptop or a password nobody else can retrieve. That's not a security failure. It's just reality when people leave or roles change. Verify the package through an appropriate engineering test. Depending on the equipment and available facilities, that could mean a spare controller or a vendor supported test setup with safeguards that prevent the test from affecting live machinery. You don't want to discover a configuration mismatch by tripping a real process. Cloud storage can help protect those engineering files. Azure can hold a controlled copy under a suitable backup and access design, but retrieving that copy still leaves the specialist with a physical device to inspect and configure. The cloud solves the storage problem, not the hands on validation problem. Damaged hardware still needs repair or replacement if an incident destroys a controller cabinet, restoring the associated virtual machines won't replace its wiring. Field devices or the checks required before the cabinet returns to service. The digital twin is useless if the physical twin is a pile of melted plastic. Production approval remains a local operating decision. The application team can report that its services work, while the people responsible for the equipment establish whether the machine can accept work under its approved conditions. Those are two different conversations, and they need to happen in order. 00:31:50 Speaker: Those checks can also expose gaps in production evidence. Once the machine's physical state is understood, the plant still needs records showing which conditions applied while material moved through the process, especially across the interruption. That's where the real story of what happened lives. It's recovered data needs to support an account of the process that actually ran with enough context for the people assessing that run to use the records confidently. Without that context, you're just guessing at what the numbers mean. For historian recovery, I'd start with what makes a measurement understandable. A saved value needs its tag definition engineering units and associated quality information, or the person reviewing it may draw the wrong conclusion about the process. A pressure reading of one hundred and fifty means nothing if you don't know whether it's PSI or bar. The archives are only part of the recovery package. Depending on the historian tag settings, event records and supporting databases may sit outside those archives with separate backup requirements that the vendor documents for that product. Don't assume one backup covers everything. Consider a hypothetical heating process with a pressure measurement. The archive might contain the recorded values, while another part of the system holds the scale and units that tell an engineer how to interpret those values. Both pieces need to survive together. Configuration changes need attention here. If someone changed the measurement range before the outage, recovering older settings alongside newer archives could leave the restored system presenting information differently from the system that recorded it. That's a recipe for bad decisions. Event records add another part of the account where the historian stores process events, or annotations. Include those records in the recovery scope, so reviewers can relate a change in measurements to the operating events recorded around it. Context is everything. Use the vendor's supported backup method. Copying archive files while the historian writes to them may produce an unusable or incomplete copy, depending on how that product manages active files and supporting records, the shortcut isn't worth the risk. The backup procedure should describe how it handles active writes. It also needs to identify which services, archives, and databases belong together, and how the restore process returns them to a state the historian supports. That's the difference between a backup and a recovery plan. Ask what an application consistent recovery point covers. For this historian, that means establishing which related data the supported method coordinates, rather than assuming that one successful backup job protects every component required to retrieve process history. The details matter. Some products provide archive specific tools or database specific procedures. Follow the supported combination for the installed version, including any required validation steps. Because a general server backup may not meet the historian's own recovery requirements, the vendor knows their product best. If the installed collectors support store and forward operation, they can retain measurements locally and send them later, subject to their configuration, available storage, and documented behavior. That buffer is your safety net, but it's not infinite. Treat that buffer as a bounded resource. It's useful. Duration depends on the actual collection rate and retained information. So a nominal storage size tells you little without a measured workload from the sources it protects. Test it. Don't guess. Check which measurements actually enter the buffer. A collector may retain some inputs while other sources depend on a different collection path, leaving gaps that won't become obvious until someone queries the restored history. Surprises are bad. Collector persistence deserves a direct test. Confirm whether buffered measurements survive a collector restart or power loss, and follow the vendor's method for checking buffer condition without damaging the records. Waiting for transfer. Trust but verify. Then measure the catch up behavior. After the historian returns, the collector needs enough supported throughput to transfer retained measurements while continuing to accept new ones, or the backlog can persist longer than expected. That's a performance problem you want to find in testing. A service start check won't reveal that problem. Recovery validation should establish that incoming measurements reach the intended archives, and that any retained collector data becomes queryable through the supported historian interface after transfer completes. End to end testing is the only way archive integrity needs a separate check run the product's supported integrity tools where available, review restore errors and confirm that the expected archive periods are present rather than relying on the application opening without complaint. A clean start doesn't mean clean data. Timestamps need close attention during validation, query records around the interruption and verify that their recorded times, time zone treatment, and quality information match the source behavior expected by the engineers who use them. Time is the most critical dimension in process data. Choose sample queries with a purpose. Retrieve a known process interval. Include the period spanning the outage, and check both recent and older records. So the exercise tests more than the latest values displayed by default. Don't just look at the pretty dashboard. Use raw records where the product supports them. An interpolated trend can appear continuous across missing measurements, so reviewers need to distinguish stored samples from values the query layer calculates between those samples. The truth is, in the raw data, give the historian owner a clear acceptance evidence record which archives past checks, which tags and time periods were queried and where gaps remain. So the people assessing a production run can judge the available process evidence. Documentation is your friend. That evidence describes conditions during the run. The plant also needs the transaction records that connect those conditions to an order, its material and its completed steps. Which brings us to recovering the mess. For every affected order, the recovered application should tell the shift which operations remain open, what material has been assigned and which recorded steps it considers complete. That's the minimum viable state for resuming production. That includes work between stations. A container might have left one operation without entering the next, so the restored work in progress record needs to represent that intermediate state, rather than quietly placing it back at the previous station. The system needs to reflect reality. Recipes and quality status belong in that recovery scope. Depending on the mis design, a batch may reference a specific recipe revision, recorded parameter choices, inspection results, and holds that govern what the operator can do next. All of that needs to come back together. Those references need to survive together if the batch record returns but its associated hold disappears, the application could present work as available, even though the approved workflow previously prevented it from advancing. That's a quality incident waiting to happen. I'd ask the application owner to inventory the stored content, start with the production database, then identify where the installed solution keeps attachments, application settings, interface configuration, and any custom services required to interpret or update production records. Know what you're protecting. Attachments can carry part of the production evidence. An inspection file might live in a separate document store with only its location recorded in the database, so restoring the database alone could leave that evidence inaccessible. The picture is useless without the frame. Custom services deserve an explicit entry. A locally developed service may translate equipment results into miss transactions, and its deployed version and settings need to match the application state you're recovering. Custom code is the most fragile part of any system. Include its recovery method, not just its source files. The team needs the supported runtime, deployment package, and configuration required to run that service, rather than a folder of code that someone must rebuild during the outage. Time is not on your side. Interfaces can retain unfinished work outside the main database, establish whether the installed solution stores pending messages or processing status elsewhere, and include the vendor supported recovery steps for those components in the maze procedure. The edge cases will bite you. Active transactions need guidance from the software supplier. Ask how the installed version handles an interruption during a production step, including what the application commits together and what remains pending when communication or power disappears. The vendor has seen this before. An unfinished screen doesn't reveal the transaction outcome. The operator might lose the confirmation message after the application accepts the submission, or the application might reject the submission before recording any part of it. The screen tells you nothing useful. The recovery procedure needs a supported way to distinguish those outcomes that might involve application audit records, transaction status tools, or a vendor defined repair workflow, depending on how the MMS implements the affected operation. Don't guess. Use the tools. Consider another moment in our hypothetical batch. An operator scans a material container, confirms its use, and then sees the terminal freeze before the expected confirmation appears, leaving the physical material already inside the process. That's a real scenario. After restoration, the batch opens without an obvious error. Its earlier operations appear complete, but the material consumption entry for that container is absent from the recovered view, and the operator can't tell whether the submission ever reached storage. The system looks clean, but isn't. Don't ask the operator to enter it again blindly. First, use the supported checks to establish whether the record exists elsewhere in the application remains pending, or genuinely needs correction through an authorized recovery workflow. Double entry is not a recovery strategy. A completion state can create a similar ambiguity. The equipment may have finished the operation, while the mess still lists it as active, leaving the shift with a software task that no longer matches the physical work. The system and reality are out of sync. Repeating the operation isn't a software repair. The application owner and production owner need to follow the approved exception process, using available evidence to resolve the state without instructing the machine to perform completed work again. That's how you avoid secondary incidents. Corrections need their own traceable record. Preserve the original recovered state and document who resolved the exception. What evidence supported that decision and which supported action, brought the batch record back into agreement with the work. Audit trails protect Everyone built these interrupted workflows into mess recovery checks. A useful test includes an order part way through execution, and verifies that authorized users can inspect its material records, attachments, and open steps through normal application functions, test the hard cases, and empty test order won't expose every problem you need. Representative, unfinished work that exercises the installed solutions exception handling, especially where custom services or separate storage participate in recording a production step. The easy path teaches you nothing. The mess might have a clean batch record now, but the system is exchanging transactions with. It could have recovered to different moments. That leaves their accounts of the same work out of step. A recovery point isn't just a timestamp, it also captures the state of rights when the copy was taken, and that determines what the application needs to do before it can actually use those records. Let's start with a crash consistent recovery point. Think about the stored disk state after an unexpected power loss. No guarantee the application finished pending work or wrote everything it held in memory. Databases usually have mechanisms for recovering from that condition. A supported database engine can replay its transaction log and rollback incomplete transactions, but that doesn't tell you whether every component of the production application is consistent with each other. Application consistent recovery uses supported coordination with the application itself. Depending on the workload and protection method that coordination can flush pending writes and prepare related storage for capture, giving you a recovery state the application actually supports. Check what that coordination includes a method that coordinates one database doesn't automatically coordinate a separate message service, attachment store, or another application exchanging transactions across the plant network. That distinction matters between Mas and ERP. Both applications can recover successfully, pass their own database checks, and still disagree about whether a production event completed its journey between them. Consider a completion transaction from our hypothetical plant before the outage. The Mas records an order as complete, sends the completion message and gets confirmation that the ERP posted the corresponding finished goods receipt. Now recover those systems to different points. The selected MSP includes the completed operation while the selected ERP copy comes from before the receipt. The MSS has no reason to send that message. Again, neither database needs to contain damaged records. The discrepancy comes from selecting individually usable states that no longer preserve the agreed outcome of the exchange. Database integrity checks alone won't reveal the missing receipt. Choosing identical backup times won't necessarily solve this either. A transaction can cross that moment with one system already committed and the other still processing, especially where interfaces deliver messages separately from the original production transaction. You need a supported consistency strategy for that exchange, where the application support coordinated protection across related components, document its boundary and prerequisites, then test whether recovery preserves the business outcomes you expect within that boundary. Some exchanges instead depend on reconciliation after restoration. In that design, retained transaction identifiers and processing records. Let the team compare what the mess submitted with what the ERP actually accepted at its selected recovery point. Those records need their own recovery requirements. If the only evidence of delivery disappears with the restored interface state, the team may struggle to distinguish a missing transaction from one that arrived before the outage. Ask the interface owner what each status actually means. Sent might mean a message entered a local queue, while accepted, might mean the receiving application committed the business transaction. It depends on the installed integration design. Build the reconciliation rule around that documented meaning for the missing finished goods receipt, the authorized team needs to establish whether the receiving system contains the transaction before deciding which supported action can restore agreement. Define that action before anyone needs it. The application owners should specify whether the approved procedure Resubmits. The original transaction invokes a recovery function or records an authorized correction with evidence linking it to the completed work. Avoid direct database edits as a shortcut. A manual change can bypass application rules and leave related records untouched, even when the visible order status appears to match what production believes happened before the interruption. Record unresolved differences as exceptions with named owners. Each exception needs the affected order or batch evidence from both systems, and a stated restriction on dependent work until an authorized person resolves the discrepancy. Make recovery point selection a joint application decision. The team should compare available points against the transaction evidence and supported reconciliation options, rather than letting separate administrators pick the newest copy for each system without talking to each other. Test a deliberately interrupted exchange during a recovery exercise. The acceptance check should show that the team can detect the disagreement, explain it from retained evidence, and resolve it through the documented application workflow. Even a well-coordinated recovery point can't help if the incident also deletes, damages or compromises every copy the team could use to restore that agreed state. Where could an incident reach your protected copies? I'd examine that question through storage access and administrative permissions, because a copy in another location can still disappear through the same account that manages the original Physical separation protects against only some failures. A backup stored outside the plant server room may survive its power fault, yet remain exposed if compromised, credentials can delete both production, storage and backup data. Administrative separation adds another boundary. Give production administrators only the backup permissions their work requires, and protect destructive backup operations through separately controlled roles and approval mechanisms where the chosen platform supports them. Separate doesn't have to mean unmanageable. It means documenting which accounts can change protection settings, shorten retention, or remove recovery points, then checking whether one compromised account could defeat every protected copy. Consider a hypothetical clean up mistake an administrator deletes storage believed to belong to a retired application, and the team discovers that the same storage also contains the only retained exports from an active production system. A second copy needs a different exposure. Its location, permissions and retention controls should prevent that same deletion from removing the remaining recovery option, rather than just placing another folder beside the first. Now apply that requirement to your chosen backup tools. Azure backup can protect supported workloads, but its suitability depends on the workload, deployment, and backup method, so check the current support documentation for your installed environment. The service name doesn't settle application coverage. Your design may combine Azure Backup with database native backups and specialist vendor tools, assigning each tool a defined role rather than expecting one policy to capture everything. Database native protection may still belong in that combination, where the application supplier supports it. Retained database backups and transaction logs can provide recovery options that need their own scheduling, storage, protection, and verification. Under the overall design, protect the backup output after it leaves the application, a vendor tool might create a valid archive on a local disk, but that archive still needs transfer into protected storage before it survives loss of the originating machine. Watch for an unprotected waiting period. If exports remain locally overnight before transfer, the protection status should reflect that exposure. Instead of treating successful export completion as evidence that an independent copy already exists. 00:47:47 Speaker: Deletion controls need deliberate configuration for supported Azure backup vaults and workloads. Assess immutable protection, which restricts changes that could remove retained recovery points before their retention period ends, subject to its documented configuration and state. Check the consequences before locking settings. Some immutability configurations become irreversible once locked, so the responsible owners need to understand retention obligations, operational limits, and costs before approving that choice for production backups. Soft deletion addresses a different situation where supported and configured, it retains deleted backup data for a defined period, giving authorized administrators an opportunity to recover it rather than losing it immediately after deletion. That period still has an end. Check the applicable settings and permitted changes, then establish who receives deletion alerts and who must respond. Because a recovery window helps only if someone notices before it closes. Separated administration can strengthen those controls. Azure backup supports multi-user authorization for certain protected operations through Resource Guard where applicable, allowing another authorization boundary to restrict actions that could weaken the backup protection. Verify the actual permissions in your deployment. Another named approver provides little separation if the same everyday administrator can also take control of that approval mechanism or remove its protection through another permitted action, retention needs a reason beyond available storage. Imagine a configuration fault that produces incorrect records for several weeks before anyone detects it, leaving recent recovery points intact but containing the same underlying problem. Older copies may then become the useful ones. Choose retention periods around plausible detection delays, and the history the organization must preserve, while recognizing that backup retention doesn't automatically satisfy every legal or regulated recordkeeping requirement. Bring the responsible records owners into that decision. They need to distinguish operational recovery copies from retained business records, including any requirements for deletion, restricted access, or preservation that affect how long particular data can remain stored. Keeping more history also creates recovery costs. Estimate the time and resources needed to retrieve older copies, restore their required software versions and inspect their contents alongside the ongoing storage and protection charges that gives replication a narrower job to consider. Next, it can help maintain a recent recovery destination, while independently retained copies. Preserve options when recent changes carry deletion, damaged records or compromise into that destination. For the plant facing a lost server room, a prepared replica could shorten recovery. Azure Site Recovery provides replication and recovery orchestration for supported server workloads, allowing the team to activate protected machines at a supported destination. Instead of rebuilding every server from its installation files. Its scope starts with the protected workload. You configure protection for eligible machines, monitor whether replication remains healthy, and use the service's recovery operations when you need to bring those machines into service elsewhere. That gives the infrastructure team a defined recovery mechanism for a supported on premises deployment. Azure can become the destination where replicated servers run after the original environment becomes unavailable, subject to the prerequisites of that protection path. Azure workloads have another supported path. Site recovery can replicate supported Azure virtual machines between regions, giving an application hosted in one region a recovery destination in another without requiring an on premises source. The supported source platform's replication components and recovery operations all differ between scenarios. Just because the design works for Azure Virtual machines doesn't mean it'll work for a server inside your plant. Start with the exact source environment. Identify whether the workload runs on VMware Hyper-V physical hardware, or Azure virtual machines. Then check Microsoft's current support metrics for that specific protection path and configuration. Check the installed operating system, not just its family. A plant server described as windows still needs assessment against the supported version configuration and prerequisites that apply to its source platform and intended recovery destination. Disk configuration needs the same level of precision supported disk types, sizes, layouts, and change rates vary by scenario, so you need the actual machine details rather than an assumption. Based on its total storage capacity. Replication throughput deserves a measured workload. A server handling a production database may generate far more disk changes during a busy shift than overnight, and the production design needs to accommodate that observed behavior within documented limits. Protection health then becomes an operating responsibility. If replication falls behind or stops, someone needs to investigate and assess the effect on available recovery points before the next incident. Turns that warning into missing production information. A protected machine still needs application approval. Check with the Mez or historian supplier that the installed application version supports the proposed recovery arrangement, including any restrictions on virtual hardware changes or running its components in Azure. Platform support and application support answer different questions Microsoft can document whether site recovery supports a machine configuration, while the industrial software supplier determines whether its application supports operation and recovery within that proposed configuration, some workloads need another protection method if a server or application falls outside supported conditions, record that gap and evaluate a vendor supported alternative rather than quietly treating an unsupported replica as the recovery plan. Consider the meze server from a hypothetical outage. If its configuration and application design qualify, site Recovery could provide the replicated server from which the recovery team begins restoring the agreed production function after losing the local environment. The service can't judge whether its contents deserve trust. If an authorized process deletes needed records or malicious software changes, files on the protected machine, replication may carry those changes into the destination just as it carries legitimate production activity. Available recovery points give the team choices within their configured retention. They still need to establish which point precedes the damaging activity and whether that point remains available when the problem finally becomes visible to the incident team. That's why retained backups remain a separate requirement, a compromise discovered after the useful replication recovery points have passed may require an older, independently protected backup rather than activation of the newest replicated machine state. Use each method for a stated recovery purpose. Site recovery can help provide a recent supported server recovery path, while separate backups preserve historical recovery options under the Retention and protection rules agreed for that workload, I document the resulting protection claim quite narrowly. Name the supported machines, the source and destination, the conditions under which replication operates and the application checks still required. Before anyone accepts those machines for production use. The next design question concerns whether that destination has everything those servers need to become usable plant applications, rather than machines that start successfully somewhere operators can't reach. I'd treat the Azure destination as an environment to build in advance. Before the incident, decide where recovered machines will connect, which traffic can reach them, and which resources must already exist when the recovery team needs them. Begin with the target network and its assigned address ranges. Check those ranges against the plant and other connected sites, because overlapping addresses can prevent routing when the recovered application needs to communicate with equipment or clients that remain on site. Reserve addresses through the supported configuration where the application requires them. Some industrial applications contain fixed server references, so establish which references can change and which need a vendor approved configuration update before the recovered service becomes usable. Define permitted connections before opening access. Network security groups and firewalls should allow the traffic needed for the recovered workflow, with separate rules for administration rather than broad access granted under pressure during an incident. Include outbound connections in that design. A recovered application may need a licensing endpoint or an external service, and an environment that blocks those destinations could prevent start up even when every incoming connection works. Compute sizing needs an application workload, not just a matching server label. Select an Azure Virtual Machine size with enough memory and processing capacity for the intended recovery operation. Then verify that the application supplier supports that configuration. Reduced capacity can be a deliberate operating choice if the recovery environment supports fewer concurrent users or fewer lines, document that restriction and test the agreed workload rather than assuming the plant will simply tolerate slower responses. Storage performance needs its own assessment. Database response depends on more than disk capacity, so evaluate throughput and input output operations against the application's observed demand and the limits of both the selected disks and virtual machine. A quieter shift can conceal a poor storage choice. Use a representative production workload when assessing performance, including the bursts that occur when several stations submit results or retrieve instructions at roughly the same time. Capacity planning also involves the Azure subscription. Confirm that the destination region supports the selected machine sizes, and that the subscription has enough quota for the machines you expect to activate together during recovery. Quota approval doesn't reserve physical capacity if the required size isn't available. When recovery starts, the team needs a previously assessed alternative or an appropriate capacity reservation arrangement, subject to the current service support and its costs. Licensing belongs in that preparation to establish the rights needed to run operating systems, databases and industrial applications at the recovery destination, including any vendor approvals or activation changes triggered by different virtual hardware. Document the deployment prerequisites Alongside those decisions required resource providers, permissions and any organization policy exceptions should already be addressed, so a routine governance control doesn't block an otherwise supported recovery operation during the outage. The terminal needs a secure route into Azure that remains usable under the failure being addressed. Whether that design uses a site to site virtual private network or another approved connection, check whether that connection actually terminates. If it's plant endpoint sits in the server room, we're assuming unavailable. The recovery design needs another surviving endpoint and a supported route from the production areas that still have power. Name resolution has to follow the recovered service, define how plant clients receive the correct destination after activation, accounting for cached answers, and any application settings that could keep sending requests toward the original server. Measure response time from an actual plant client. An application designed for local servers may perform many sequential database requests, so a modest increase in network delay can become a noticeable weight during each operator action. Bandwidth won't remove that delay by itself. More capacity can help transfer larger volumes, but it won't shorten every request and response exchange, which is why placing related application components together may matter more than buying a faster circuit. Assess remote display options only with the application owner's support. Keeping the application close to its database might help, but scanners, printers and other station devices still need tested behavior through the proposed access method that gives you a destination with a tested path from the plant, there's another access path to examine before calling it ready, whether authorized people can enter and manage that environment when their normal sign in services aren't available. Take the administrator assigned to activate the Azure environment, remove their usual workstation and plant connection, and ask whether they can still reach the required management tools through an approved alternative. Start by separating the identities involved. The account managing Azure resources, the account administering a recovered server, and the operator account entering the mess may rely on different identity systems with different recovery requirements. Microsoft Entry ID handles access to Azure management resources. Your plant application might instead authenticate against Active Directory Domain services, use its own account database, or combine several methods, so document the actual arrangement for each required recovery task. Hybrid identity needs a closer inspection where on premises identities connect to Microsoft Entra ID, the configured sign in method determines whether an unavailable on premises component can prevent an administrator from authenticating to cloud resources during the outage, check the method rather than the account's familiar username, password, hash synchronization, pass through authentication and Federation have different dependencies, and your identity specialists should establish which dependencies remain available under the incident scenarios the plant has agreed to address. A successful cloud sign in doesn't prove application access. An administrator could enter Azure, while the recovered Mes still requires a domain service that hasn't recovered, leaving infrastructure management available but the intended operator workflow inaccessible. Emergency access should address specific failures in that arrangement. For Azure administration. Separately controlled cloud only emergency accounts can reduce dependence on unavailable on premises identity services, provided their authentication methods and access policies don't recreate the same failure elsewhere. Those accounts need strong protection and deliberate policy decisions. Assess which conditional access requirements could block emergency use, make only the necessary exceptions, and document the compensating controls rather than broadly removing security checks. Because the account carries an emergency label, authentication hardware belongs in the recovery assessment. If an emergency account requires a physical security device, authorized people need a protected way to obtain a usable device without entering the unavailable building or depending on one absent administrator. Keep routine work on normal named accounts, record who gets emergency access, and make sure there's a clear way to attribute actions when someone has to use a separately held account for an exceptional procedure, configure monitoring for account use and changes, then verify that alerts reach the right people through a channel that won't disappear. Along with the identity, service or network you're recovering. An account might sign in successfully, but lack the authority to perform the required recovery operation. Another account might hold far more permissions than its assigned recovery role actually needs. Use a harmless approved task to establish that difference during an exercise, have the designated person follow the documented access procedure and demonstrate the required management capability without changing live protection settings or exposing production resources. If normal administration requires a managed workstation, provide a separately maintained recovery workstation or another approved arrangement. Make sure its connectivity and authentication remain usable when the plant environment disappears. That preparation includes access to the recovery tooling itself. Confirm that authorized people can open the relevant management services and retrieve the information they need without first signing into a document store hosted inside the affected environment. A recoverable disk or database can remain unreadable if its required encryption material disappears. Identify the keys, certificates and passwords needed by each supported restore method. Treat those items as controlled recovery assets where Azure Key Vault or another protected store holds them. Assess permissions. Network restrictions, deletion, protection, and the availability of required key versions under the same outage scenarios as the protected workload. Don't assume exporting everything provides the answer. Some keys aren't exportable by design, and the recovery arrangement must follow the supported protection method while giving authorized people access to the service that performs the required cryptographic operation. Record where recovered services obtained credentials, who can authorize their use, and how the team handles the secret that expires or becomes unavailable while the normal administration path remains down. A rehearsal should include that retrieval step. Have the authorized specialist obtain the required material through the approved process, confirm that the recovered service can use it and retain evidence without copying secret values into the exercise report. That still leaves the practical question of which services those people should restore first, and where the procedure requires them to wait for another owner's confirmation. The recovery order should come from the dependency record. Turn each required service into a step with an owner, an expected result, and a condition that must pass before dependent services can proceed. Start with supporting services, then bring up the database layer. Once the database owner confirms the required databases have completed their supported recovery checks, the application team can start the services that read and update those records. Interfaces should enter service at a deliberate point. Keep their processing paused until the application owner confirms the recovered state is ready. Rather than allowing background services to exchange production transactions simply because their operating systems have started, some steps can run together, others must wait. Two independent application components might recover in parallel, but a service that depends on a database should wait for an explicit readiness check from that database. Make that distinction part of the plan itself. A numbered document can suggest a sequence, but it won't prevent someone from starting a later step while another team still investigates a failed prerequisite. As your site recovery recovery plans can coordinate supported recovery operations, they let you organize protected machines into groups, define their startup sequence, and include supported automation or manual actions at specified points in that sequence. 01:03:59 Speaker: Grouping machines doesn't establish that their applications are ready where the next group depends on an application level result. Include a check or approval action that confirms the prerequisite instead of relying only on a successful machine startup. A supported Azure Automation runbook could perform an approved configuration task or check a service response, with its result recorded for the person supervising recovery. Each automated step needs a defined failure response. If a check returns an unexpected result, the plan should pause and identify the responsible owner rather than silently continuing interdependent steps with an unresolved fault. Retries need limits to a script that repeatedly submits an operation after losing its response could cause unintended changes, so its author must define how it detects prior completion and when another attempt requires human review. Put a named owner against every script. That owner should maintain its permissions, required modules, inputs and supported versions, and review it whenever changes affect the recovery task it performs or the services it contacts. Test the execution context, not just the script. Code that works in an engineer's interactive session may fail under its automation identity because required permissions, stored settings or network access differ from those available during development. Before exposing recovered services to plant clients, ask the network owner to confirm that the intended restrictions remain in place and that only approved connections can reach them. Data trust needs a separate approval gate. The application owner should review the agreed checks and unresolved exceptions, then record whether the recovered state can support the intended workflow under the restrictions already defined. Production use requires its own name decision. The Responsible Production Authority receives that evidence and either accepts the function within its stated limits or keeps it unavailable while the team resolves a specific blocking condition. Record what each approval permits approved to continue leaves too much room for interpretation when one person means further technical testing and another believes operators can immediately resume normal work at their stations. Write stop conditions in equally direct language. If a required database check fails, a recovery point can't be identified or an automated action changes something unexpected. The procedure should name who investigates and who authorises continuation. Include the approved command or action, required inputs, expected output and where to retain evidence without forcing someone to reconstruct the author's intent during an incident. Keep a controlled offline copy of the executable plan. That copy should include the manual alternatives for approved automation failures, its revision date, and a way to record completed steps without exposing credentials in the working log. Have someone other than the author rehearse the sequence, watch where they hesitate, which outputs they can't interpret, and where two teams each assume the other owns a step, then correct those gaps in the procedure. An executable plan gives the incident owner a controlled sequence to authorize. The next decision is whether conditions justify activating that sequence at the alternative site, with its prerequisites satisfied and its stop conditions understood. Before authorizing that change, the incident owner needs evidence that the original environment won't continue accepting work while the recovered environment begins. Accepting the same work and unreachable server isn't necessarily a stopped server, a network fault could hide the original mess from the recovery team while leaving it connected to local terminals, equipment interfaces and services still processing requests inside the plant. That distinction determines how you prevent competing writers. The procedure needs a verified way to stop or isolate the original applications, write paths using controls appropriate to the workload and the connections that remain available. Engineers often call that isolation fencing. In practical terms, fencing prevents the displaced environment from changing the authoritative production state, even if its servers recover unexpectedly or a broken network connection suddenly returns. A planned move gives you more control. Where the protection path and workload supported planned failover can coordinate, stopping the source and transferring remaining changes before the destination takes over its assigned application role. Check what the supported operation actually promises. Its handling of source shutdown, synchronization and data loss depends on the replication scenario, so the operating procedure must describe the specific behavior available for this application. An unplanned failover starts with less certainty. The original environment may be inaccessible. The latest changes may not have reached the destination, and the team may need to activate a recovery point without completing a final transfer. That's a decision about accepted uncertainty. The incident authority needs the age of the available recovery point, the likely missing work and evidence of source isolation before deciding whether activation represents an acceptable operating choice, give that authority to a named incident role. The role needs permission to authorise the move across the affected services, with application and production owners supplying evidence, rather than independently activating their own pieces whenever they feel ready. Otherwise, the plant can develop two competing execution states. Engineers call this split brain. Separate environments each behave as though they own the same work without a reliable mechanism, keeping their decisions consistent with each other. Consider the order service in our hypothetical plant. The original instance might assign an order to one station, while the recovered instance, using its selected recovery point still presents that order as available for assignment elsewhere. Dispatch queues can multiply that conflict appending instruction might remain ready for delivery at the original site, while the recovered queue contains another copy, leaving both environments capable of dispatching work after connectivity returns. Databases need workload specific protection against that situation. For a design intended to have one writer, the database owner must establish that only the selected environment can accept authoritative updates before application processing resumes against it. Equipment connections need equally deliberate control. A machine interface shouldn't accept commands from whichever server reaches it first, so the approved design must establish which endpoint owns the connection and how the previous endpoint loses that ability. Don't use monitoring silence as proof of isolation. Require positive evidence from the selected control, such as verified source shutdown or enforced network restrictions, and record any limits on what that evidence actually establishes. Here's a problem that doesn't get talked about enough. Sometimes your team just can't get enough proof to make a clean decision. If no supported control can stop the original environment from issuing conflicting work. The incident owner needs to hold back those right functions until the uncertainty clears. That's not being cautious. That's being honest about what you actually know. The expected outage length still drives the decision. A credible repair estimate might make waiting the smarter play, but a prolonged or uncertain loss could justify failover. As long as the destination's readiness and the data loss exposure, you're willing to accept both support that move. Compare estimates against evidence, not confidence. Ask what's actually left to repair, how reliable that estimate really is, and whether the recovered environment can take the intended workload right now, not after another unmeasured period of preparation. Record the ownership change as an explicit incident. Event state which environment now owns updates? Which functions stay restricted? Who authorized the activation and how? Operators and support teams learned that the old environment must remain inactive. Restored connectivity shouldn't automatically reverse that decision. The original services need to stay fenced until a separately authorized change lets them participate again, even if their health checks suddenly look normal. That establishes one application authority for the affected work, but it leaves a different question for the planner. Does restoring software somewhere else actually help with the orders assigned to equipment that the incident has taken out of service. The planner needs a separate assessment of production capacity. A recovered application in Azure might serve the surviving plant, but moving its orders to another factory requires equipment and operating approvals that software recovery can't supply. Let's extend the hypothetical incident to include damaged production equipment. The business asks another plant to take the affected orders, and that plant has a machine described in the equipment register as the same type. That description gives the planner somewhere to start. But before assigning work, someone needs to establish whether the machine can actually perform the required operation for that product. Using the tooling and process conditions its approved route specifies. Available hours. Don't establish that capability. A machine might have room in its schedule, but lack the fixture needed to hold this particular component, leaving its apparent spare capacity useless for the displaced order. The fixture might be transferable between plants, but the planner needs a credible arrival time, installation requirements and confirmation that the receiving equipment can use it before treating that transfer as part of a feasible schedule, people impose another constraint on the proposed move. The receiving plant may employ experienced operators without having enough people qualified for this product or process on the shifts when the transferred orders would need to run. Materials need assessment at the receiving location to stock recorded at the affected plant doesn't become available elsewhere. Just because both plants share an ERP, especially if the incident prevents anyone from accessing or shipping it. Purchasing and logistics belong in this decision. They need to confirm which approved materials can reach the receiving plant and when, including any inspection or acceptance steps required before production can consume those materials there. The approved route may also be site specific. A product that can physically pass through another machine may still require process qualification or customer approval before that machine becomes an authorized resource for the order. I'd represent those conditions as explicit planning constraints, connect each product to its required process, then connect that process to the resources permitted to perform it, including the conditions that make those resources usable for this work. That gives a scheduling tool something meaningful to calculate advanced planning and scheduling apps can help evaluate alternative assignments and sequences, where the installed system represents the relevant constraints and receives sufficiently reliable operating data. Its result depends on those modeled conditions. If the planning model treats every machine of one type as interchangeable, the calculated schedule can assign work to equipment that the production team has no permission to use. The objective needs attention as well. Protecting an urgent customer's delivery might displace orders already promised by the receiving plant, so the planner needs to compare that consequence rather than treating transferred work as free. Additional output. A feasible schedule still needs operational review. The receiving planner should challenge the proposed assignments with production checking with a set up. Assumptions in resource availability reflect what the plant can actually support during the proposed recovery period. Order status determines what can move for an order already partway through execution. The planner needs the mess state and the physical work in progress location before deciding whether the remaining operations can transfer to another site, you can't transfer a completed operation by scheduling it again. The approved approach may involve moving identified work in progress, splitting the remaining quantity, or creating linked work through supported application workflows, depending on the systems and production rules. ERP commitments need to follow the accepted plan. Once the responsible planner approves the reassignment, the business needs supported updates to dates, quantities, and supplying locations so customer commitments reflect the capacity the receiving plant actually accepted. The receiving Mius then needs executable work. Its order and resource assignments must correspond to that accepted plan, rather than inheriting a proposed schedule that still assumes unavailable materials or approvals. Nobody has yet granted, Microsoft can support the information exchanged around those decisions. Azure can host recovered applications and integration services, while dynamics three hundred sixty five may hold the relevant business orders where deployed, specialized planning technology and manufacturing systems still supply their respective functions, the organization owns the decision rules. Between them. Someone must establish which resources qualify, authorize changes to commitments and approve the resulting schedule before either plant treats the Alternative Production plan as permission to start work. There's another operating condition we need to address. A plant may retain its equipment, people, and materials while losing access to central services, leaving the shift to establish which approved work remains possible without that connection. Offline operation needs a defined set of permitted tasks. For the disconnected plant, I'd specify which work the shift can perform using locally available information, and which decisions must wait until central services return. The permission should attach to particular work, not simply to align a line. Might finish released orders whose instructions remain available locally, while refusing new assignments that require a fresh availability, check or approval from another system. Define that operating state before anyone needs it. The procedure should explain who declares offline operation, what evidence they review, and how operators learn that the normal rules have changed for the affected stations. Consider a released order already prepared for the afternoon shift. If the plant holds its approved instructions, assign materials and required permissions locally. The operating procedure might allow that order to start. Despite losing enterprise access, that permission needs a precise boundary. It could cover only the quantity already released on the assigned equipment using the specified material lots with no authority to extend the run because another customer suddenly needs more local information. Needs a controlled source. A downloaded order should carry enough identification to establish its revision, release status and intended use, rather than becoming an unsigned document that anyone can amend during the interruption, decide when those copies become eligible for offline use. The preparation process should confirm that the order, recipe and work instructions belong together with any required approvals completed before the connection disappears and prevents further verification. Freshness needs a rule people can apply record when the local package last received confirmation from its authoritative source, then define how long that confirmation remains acceptable for the particular process and product involved. A recent copy can still carry uncertainty. A central quality hold might arise immediately after disconnection. So the offline procedure needs an agreed response to the possibility that local information no longer reflects the latest restriction. Some work can't tolerate that uncertainty. Where a process requires current central authorization before each start, the permitted offline behavior may simply be to preserve the current state and wait for that authorization to become available. Operator permissions also need an approved local basis. Establish which people can perform each offline task, how the plant verifies their authorization and what happens if their qualification expires or a required supervisor isn't available. Avoid inventing permissions during the outage. An operator who can execute a released recipe shouldn't acquire permission to change that recipe. Just because the normal approval service can't be reached from the production area. I'd make recipe changes explicitly restricted. If a process needs an adjustment beyond the locally approved limits, the shift should follow its defined hold or escalation procedure, rather than editing the stored instructions to keep equipment occupied. Material substitutions need the same clarity. A shortage during isolation doesn't authorize use of a different lot or material grade unless a separately approved procedure establishes the conditions and authority for that specific substitution. Product release deserves its own offline decision. The plant may permit processing and controlled holding while withholding release, particularly where acceptance requires information that the disconnected location can't verify through its approved local records. Operators need a usable way to record permitted work, specify the required entries, who checks them and where the plant retains them during isolation. So continued production doesn't outrun its ability to preserve the required evidence. Set a practical duration limit for offline operation. That limit should reflect the earliest point at which information becomes too old. Recording space runs short, or process risk exceeds what the approved operating procedure permits. Recording capacity can set that limit before instructions expire. If the available buffer or controlled recording method can't retain another complete production record, the shift needs a stop condition before starting work whose evidence it can't preserve use conditions as well as elapsed time. An unexpected inspection result missing material identity or unavailable qualified person may end. Permission to continue. Even while the nominal offline operating window still has several hours remaining. Make the remaining permission visible through routine shift communication. A handover should identify which orders remain authorized, how much work may continue and which condition will require the next supervisor to stop. Further starts. Those limits create requirements for the local environment. The next architecture question is whether the service is left inside. The plant can actually support those permitted tasks throughout the isolation period, rather than only at its beginning. Leave the approved equipment connections intact, but cut off enterprise and cloud access in a controlled test environment, then exercise the tasks the offline procedure permits. A warm session can hide a missing dependency. An operator already signed in might keep working, but the next person can't get in because the local access service needs a response from somewhere outside the plant. So test a fresh session to check station access, supervisor actions and any supported local authorization method, user accounts and conditions that match the people expected to work during an extended interruption. 01:20:02 Speaker: Local data capture needs an equally practical test. A station should record the permitted production step and return its supported confirmation without waiting forever for a central connection that the architecture already assumes is unavailable. Now check where that confirmation actually comes from. An application might acknowledge receipt before information reaches durable storage, so you need to establish which response tells the operator the record will survive a restart of the local service. That distinction affects the operator's next action. If the terminal can't tell whether a submission completed, it needs a defined exception behavior, rather than leaving someone to keep pressing the same button until something changes. An IoT gateway may collect machine signals locally, but its ability to buffer them depends on the software enabled settings, storage, and supported operating conditions. Local execution doesn't automatically mean offline capability. If you use Azure IoT components or another edge platform, verify the documented, disconnected behavior of that specific deployment. Don't assume that installing software inside the plant removes its external dependencies. Run the gateway against a representative input rate. Keep collection active while its destination stays unreachable, and observe storage growth, processing load and weather diagnostics give the local team a reliable account of its condition. Queue limits need observable thresholds. The gateway should expose how much capacity remains and what happens at the limit. Whether the supported behavior rejects incoming records, discards older ones, or stops collection entirely. Shared disks introduce a less obvious failure. A queue with its own configured allowance can still lose usable space because operating system logs, temporary files, or another local application consume the underlying disk. Monitor the actual storage, not just the queue. Define who responds to approaching limits and which supported actions they can take without deleting retained production information. Just to clear a disk space alarm and silence the monitoring in an isolated test setup. Interrupt gateway power while records remain buffered, then establish whether the software recovers its retained contents and resumes collection through the vendor supported process. Power protection has a finite operating window. An uninterruptible power supply may provide time for an orderly shutdown, but the gateway local network switches and storage. It depends on all needed. Coordinated power loss procedure. Credentials can expire while nothing else changes. Review certificate renewal requirements and token lifetimes for the local services, including whether prolonged disconnection leaves them unable to authenticate once their current credentials reach expiry. Don't assume reconnection immediately fixes that condition. Test the documented recovery method for expired credentials, including how an authorized specialist restores communication when normal renewal itself requires the connection that authentication currently prevents. Give qualified plant staff the approved diagnostic tools and protected instructions required to inspect service, health queue condition and storage without depending on a remote support portal. Keep that access limited to its assigned purpose. A gateway administrator may need permission to restart a collection service, but that task shouldn't automatically provide permission to change controller logic or override protective equipment functions. Vendor access needs a deliberate boundary to. If specialists require a temporary connection during isolation. Use an approved time limited arrangement with authorization and activity records rather than introducing an unrestricted route into the control network. Record changes performed during local support. The returning central team needs to know whether someone replaced a gateway, changed a collection setting or recovered a local database. While the normal management systems couldn't observe those actions. Finally, test the gateway supported reconnect behavior, verify that it detects the returning destination, and exposes transfer status without assuming that a restored connection means every locally retained record has already reached its intended system. Once local services can sustain the interruption, the returning connection needs controlled handling of the information accumulated while the plant and its central systems operated separately. Reconnection needs rules for each kind of information a store temperature sample, a material consumption transaction, and an instruction to start equipment all need different handling when they arrive after a long interruption. Begin by preserving where each retained record originated. Include a stable source identity, and enough context to associate the record with its intended equipment, production, operation or destination without guessing from the connection delivering it. Keep event time separate from receipt time. A measurement collected during the morning interruption might reach its destination that evening, and the receiving system needs both times to explain when the process condition actually occurred. Arrival order can differ from production order if separate gateways reconnect at different speeds. A later event might arrive before an earlier one, so applications need a documented rule for handling that sequence. Clock uncertainty should remain attached to the evidence where a source clock drifted during isolation, retained the original timestamp, and any supported quality indication, rather than silently changing it to make the records appear orderly. A sender can transmit a record, successfully, lose the acknowledgment, and then transmit it again because it can't establish whether the receiving system accepted the first attempt. That's why message identity needs to survive retries. A sign or preserve a stable identifier for the original event, following the application's supported design so the receiver can recognize another delivery of that same event. Engineers call one useful handling property Idempotency in plain language processing. The same request again leaves the business outcome unchanged. Rather than consuming another quantity of material or completing another unit each time it arrives, the duplicate check must belong to the transaction design. If a receiver records consumption but fails before recording that it processed the message, a retry can still create another consumption entry despite an apparently sensible duplicate check. Use the receiving application's supported mechanism. For that protection, the interface owner needs to establish how message tracking and business updates remain coordinated, including what happens if processing stops between accepting a request and returning its acknowledgment for telemetry. Replay usually concerns historical placement and duplicate handling. Confirm whether the destination can accept older samples, which timestamp it uses, and how it treats repeated values carrying the same source and event identity. A repeated value isn't necessarily a duplicate. Two genuine measurements can have identical readings, so discarding records merely because their values match could remove valid samples from a steady process during the interruption. A delayed material issue needs its original transaction identity and relevant order context, allowing the application to determine whether that action already exists or still requires processing. Picture a container transfer recorded locally during isolation. When its message arrives, the central application might already show that container at another station because an authorized person recorded a subsequent movement through a different supported workflow. Blind replay could move the container backward. In software, the receiving logic should detect the incompatible state and route the transaction for review, retaining the original event details instead of overwriting the current record to satisfy delivery equipment. Commands need stricter treatment than historical records. An instruction issued before disconnection may no longer match the machine's current state, even though its message remains intact and the network can now deliver it. Give commands explicit validity conditions where the equipment interface supports them. Those conditions can include an expiry time, the intended operating context, and required state checks, with the receiving system rejecting instructions that no longer meet them. Some commands shouldn't enter a replay queue at all. The controls team should define which actions require fresh authorized request after reconnection, rather than treating delayed delivery as permission to perform a physical action now. Conflicting order states need an assigned resolution path if local records show an operation completed, while the central application shows it canceled, preserve both accounts and prevent automatic processing from choosing whichever message arrives last. Create a manual reconciliation queue for those exceptions. Each entry should identify the affected work. The conflicting records, the reason automatic handling stopped and the application owner responsible for selecting a supported correction. Operators also need to know what remains unresolved. A connection indicator should stay separate from reconciliation status, so the shift can distinguish restored communication from completed processing of the work accumulated during isolation, and avoid acting on an unfinished result. When reconciliation can't establish which material or process record belongs to completed work. The remaining question concerns what evidence the plant can rely on when deciding that products status product disposition needs evidence tied to the actual units awaiting a decision. I'd have the quality lead assemble that evidence by batch or other controlled identifier, so the review follows the product rather than the order in which applications returned. Start by establishing the extent of the affected work. Identify the batches processed during the uncertain period, including any material divided between containers or combined downstream. Because the review boundary may extend beyond the station where the interruption appeared. Give each affected batch an explicit status where required evidence remains incomplete. Apply the approved quarantine controls in both the physical holding area and the available inventory system, so normal handling doesn't accidentally send uncertain product toward shipment. The hold should explain what prevents release awaiting recovery. Gives the next review a little to investigate, while a named missing inspection or unverified material link directs someone toward the evidence needed to resolve that batch, you need to confirm that recorded input lots, intermediate containers and finished units actually connect through supported records. Pay special attention to any divisions or combinations that could change which. finished units share an exposure. A complete looking report can still contain an unresolved link, so the reviewer has to establish that those relationships apply to the physical product under assessment. Don't accept a report just because every field contains a value. Check whether the required operating records cover the relevant processing interval, and whether any gaps prevent assessment against the approved limits for that product. You're not demanding every available machine signal. You're asking whether the records that exist actually let you evaluate what happened during that specific run. You need to establish which sample each result represents, whether the required inspection stage actually occurred, and whether the result retained its acceptance status under the applicable specification revision when the inspection took place. Without that context, a passing number doesn't tell you much. Confirm that required approvals remain associated with the correct record and revision, including their recorded meaning. The presence of a user's name isn't proof of approval. You need to see that the system actually captured that decision. Have authorized staff compare identified containers, quantities, and and locations with the restored work in progress records document discrepancies without changing either account just to make the totals agree on a worksheet, a matching count proves only part of the case. Finding the expected number of containers doesn't establish their contents or processing history. The review must retain any unresolved identity or process questions, even when the arithmetic works. Contemporaneous, controlled records and observations may support the investigation under the approved procedure. Later recollections should remain clearly identified as recollections rather than being entered as original production measurements. If someone supplies a handwritten entry, record its origin, the work it refers to, and the checks performed before accepting it. That way, another qualified reviewer can follow the reasoning. Quality may determine that a further inspection can resolve a particular uncertainty, but passing a new test doesn't automatically reconstruct missing material identity or prove every required process condition occurred earlier. Assign each disposition decision to its authorized role. The procedure should name who can release product, approve, rework, authorize, scrap or request further investigation, including the limits of delegated authority when the usual decision maker isn't available. Specify the approved work required identification and acceptance checks. A decision to recover product shouldn't become an informal invitation for the shift to repeat. Whichever operation seems most likely to help restart approval should remain separate from product disposition. The responsible operating authority may permit new work under verified conditions, while quality keeps earlier batches on hold. The incident record needs to state exactly which permission applies to which work. Retain the disposition, supporting evidence, unresolved restrictions and approving authority that lets the warehouse and next shift act without having to reconstruct a conversation between several departments. It gives product decisions a defensible basis, but there's still another condition to consider before relying on the recovered records. Whether the environment presenting that evidence remains trustworthy, especially when the incident gives the team reason to suspect deliberate interference before restoring connections. The Incident response Lead needs to establish what must remain isolated and what evidence the investigation needs from the affected systems, containment still has to respect the process. Disconnecting a server or blocking a network segment can interrupt supervisory functions. Security specialists need the controls team to assess the physical consequences before applying those measures. Where immediate action is required, the responsible people need to coordinate protective measures without assuming that either continued connectivity or sudden disconnection represents the safer operating condition. Evidence preservation needs a deliberate owner before rebuilding affected machines, incident responders may need to capture logs, disk images or volatile information following their procedures and the practical limits imposed by continued plant operation. Running unfamiliar tools or restarting a controller to inspect it can change evidence and disrupt operation. Qualified specialists should determine which collection methods suit the affected assets. Keep an incident decision record outside the suspect environment, record containment actions, observed symptoms and relevant times through a protected arrangement. The response team trusts. Rather than relying solely on logs an attacker might alter a separate Azure environment can provide somewhere to rebuild. But separation alone doesn't establish that its accounts, deployment tools or connections remain outside the suspected compromise. Define what makes that environment acceptable for recovery. The response team should assess its administrative access and supporting services with approved software sources and controlled connections established before recovered applications enter that environment. Depending on the findings and vendor support, specialists might deploy a fresh operating system and application, then recover selected data rather than carry the original system configuration forward. That choice needs application expertise. Alongside security expertise. An industrial software supplier may need to confirm the supported installation sequence and data recovery method, especially where custom components form part of the installed production solution. Investigators need to assess when unauthorized access or modification could have begun, because the first visible disruption may occur well after the attacker entered the environment. Treat a candidate restore point as something to investigate. Review available evidence. Scan recovered content using appropriate methods and examine relevant changes while recognizing that a clean scan can't prove every possible form of compromise is absent. The incident authority needs to understand what specialists checked, what they couldn't establish, and which restrictions must remain before deciding whether the proposed recovery state can support limited operation. Credentials need a coordinated replacement plan. Resetting a password from a machine that remains compromised can expose the replacement immediately, so responders should establish a trusted administration path before issuing new access material. Service credentials, certificates, application tokens and privileged integration accounts may need replacement or revocation. Application owners need to coordinate the changes so required services don't fail unpredictably during recovery. A password reset doesn't necessarily terminate existing access, either. Identity specialists should assess active sessions, issue tokens and other authentication mechanisms, then use supported revocation and recovery procedures appropriate to the systems involved in the incident. Responders should investigate unauthorized accounts, scheduled execution, remote management changes and other findings relevant to the attack. Treating removal of the first detected file as completion isn't enough to admit recovered services in controlled stages initially allow only the connections needed for inspection, then expand access against explicit acceptance conditions. You need monitoring and a defined response. If suspicious behavior returns during that progression. Production pressure shouldn't turn a provisional security assessment into unrestricted reconnection, particularly when unresolved findings concern accounts or services capable of reaching several plant systems at once. Incident response specialists should guide the cyber recovery decisions. Plant operations controls and application owners provide the operating constraints those specialists need, including which interruptions require a controlled process stop and which checks require qualified local staff. The recovery design now needs a practical way to exercise these decisions and verify selected recovery behavior, without letting the exercise itself interfere with live production. An isolated restore gives the team a working application to examine. Instead of discussing what the backup should contain, qualified users can open the recovered system and exercise selected workflows without sending results into the operating plant. 01:36:21 Speaker: You select an available recovery point and a target test network, allowing the recovered machines to run for examination while the source workload continues operating, the network choice needs particular care. A test virtual network should have no unintended route to production, including connections inherited through network peering, shared routing services, or access paths that administrators normally use for convenience. Have the network owner inspect effective routes and access rules, then verify that the recovered machines can't reach live plant endpoints before anyone starts the application checks. Keep administration on an explicitly approved path. The people performing the exercise need access to the recovered machines, but their access arrangement shouldn't introduce a connection that lets those machines communicate with the systems they're supposed to remain isolated from. Review automatic startup behavior beforehand because an interface, service, or scheduled task might begin contacting its configured destinations as soon as the restored operating system becomes available. Use network restrictions alongside application specific controls where the supplier supports it. Pause outbound interfaces and background processing until the test owner confirms that their destinations point only to approved test services inside the isolated environment. The restored Mez might retain connections to machine interfaces, so prevent access at the network boundary rather than relying entirely on someone remembering to disable each connection. The historian needs the same separation it's restored. Collectors or data services shouldn't contact live sources, and any simulated measurements used during the exercise should enter through a supported test arrangement that can't feed the production archive. Outbound effects deserve their own look. A recovered application can print labels, send emails or post business transactions without commanding equipment, and those actions can still confuse operators or change records outside the exercise. For some checks, a label preview is enough. If actual printing is part of the agreed test, direct it to a dedicated test printer with controlled output, never to a printer serving an operating station. Redirect email to an approved test destination. Block access to production, mail delivery and business interfaces, and verify those controls before exercising workflows that normally notify notifies supervisors or submit updates to another application automatically. Automated commands need an explicit test boundary where a workflow expects an equipment response. Use a vendor supported simulator or test endpoint with no route from that endpoint to controllers or other live command receivers. Then give the users a specific task in the recovered mess. Create a clearly identified synthetic order through the supported workflow, and ask an authorized test operator to dispatch it to a resource reserved for the exercise. Carry that order through the selected acceptance steps. The application owner might require an instruction to open a simulated result to be recorded, and the resulting operation status to appear correctly when another authorized test user retrieves it, write the expected outcome. Before running the test, record the order identifier, the permitted actions, and the evidence required for a pass. That way, an unexpected result doesn't become acceptable simply because the team found a workaround. Historian queries need equally concrete pass criteria. Choose a known archived interval, specify the expected tags and record coverage, and have the historian owner compare the returned results with an independently retained reference suitable for that exercise. Keep query correctness separate from query success. A request returning rows proves the server's answers, but the agreed comparison establishes whether those rows contain the expected records, timestamps, and quality information for the selected interval. Log fail checks without repairing them out of sight. If a specialist changes a setting to complete a task, retain the original failure and the change. Let the application owner distinguish recovered behaviour from behaviour achieved after intervention. Finish by cleaning up the test environment through the supported procedure. Confirm that test resources and temporary access arrangements are removed as intended, while preserving approved evidence and protecting any copied production data under the organisation's handling rules. Be precise about what this exercise proves. The isolated application has demonstrated selected behaviour under its test conditions, but the boundary that protects production also prevents this exercise from proving every connection the operating plant will eventually need. Build the exercise program around questions that need progressively stronger evidence. A tabletop discussion can establish whether people understand their decisions, while an isolated restore lets them demonstrate technical steps without involving the operating line or its live connections. Begin the tabletop with an incomplete incident report. Give the participants only what their roles would initially know, then introduce further findings as they request checks and explain which decisions depend on the answers. Watch the handoffs rather than rewarding quick answers. If production asks for a recovery estimate, establish who obtains it, what information supports it, and how the incident owner communicates any uncertainty to the shift. Include someone covering for an absent service owner that substitution tests whether the documented responsibilities work beyond the usual team, without requiring anyone to touch production or introduce a fault into an operating system. Carry unresolved questions into the next technical exercise. If the tabletop reveals uncertainty about a required permission or service, assign a specific check during the isolated restore instead of assuming the discussion settled it, then identify what still requires controlled integration testing. Write down the connections and device interactions excluded from the isolated exercise and state which remaining recovery claims depend on observing those interactions under approved conditions. Plant rooting is one example of that gap. A recovered application may function inside its test network, while a station on the production network encounters a firewall restriction, an unexpected route, or a client setting, the isolated exercise never exercised. Device behavior introduces another boundary. A simulated response can support an application check, but it can't establish how the installed scanner or other station device behaves through the actual recovery connection under operating conditions. Choose the narrowest useful integration test. You might verify access from an approved plant terminal to a dedicated test service, provided the network and application owners confirm the arrangement won't redirect normal station traffic. Some remaining checks need a maintenance window, where proving the real workflow requires a production endpoint or a temporary service change schedule that work through the plants change process, rather than stretching an isolated exercise beyond its approved scope. Production should define the conditions for that window. The test plan needs the affected equipment, state responsible local staff, maximum permitted interruption and an agreed stopping point if the observed behavior differs from expectations. Describe the residual risk plainly before approval. A controlled test can still uncover an unexpected dependency or require recovery from a test induced fault, so the approver needs the likely consequences and the prepared response. Assign someone to call a stop independently of technical progress. That person should act when the operating conditions leave the approved limits. Even if the engineers believe one more adjustment might complete the check. Keep measurements tied to the function being demonstrated for a recovery time result, record the agreed starting event and the final acceptance event, along with any pauses where the exercise conditions differ from a real incident. Don't hide preparation outside the reported duration. If specialists pre-configured a resource for the exercise that wouldn't normally exist. Identify that advantage and establish how its absence would affect the recovery estimate measure recovered data age with identifiable records, use approved test markers or independently verifiable transactions to establish the last recovered point. Then document any uncertainty that prevents a precise comparison with the simulated disruption. Give missing dependencies a corrective owner. The finding should explain what failed, which production function it blocked, and what change and retest would establish that the dependency no longer prevents the intended recovery. Human handoffs need measured evidence to capture when one team requests acceptance, when the next person receives usable evidence, and when that person authorizes continuation, including delays caused by missing information or unclear responsibility. Write the result as a bounded statement name. The function demonstrated the conditions used, the achieved time, the recovered data age, and the outstanding checks, so management can distinguish demonstrated readiness from work still awaiting approval. Close findings through another observed check. Not a revised document alone. Updating an instruction addresses the written gap while a follow up exercise establishes whether a qualified person can now complete the previously blocked action under the intended conditions. One final exercise question concerns the temporary destination itself after the plant has accepted recovered services there. The team still needs a controlled way to return to its normal environment without losing the work completed during that temporary operation. Returning home needs a planned service change while the team repairs the plant environment. The recovery site keeps accumulating production transactions, configuration changes and operating evidence that the return procedure must preserve. The repaired servers now contain an older state. Restarting them won't bring that state forward, so the application owner needs a supported method for returning current information before those servers can resume their assigned role. Prepare that method before choosing the return date. For workloads protected through Azure Site Recovery. The required Reprotection and Failback operations depend on the original protection scenario. The current configuration and the destination to which you're returning follow the documented path for that workload. Reprotection establishes the supported replication arrangement after failover, but its prerequisites and direction need checking rather than assuming the original protection settings automatically handle the journey back. Some returns need infrastructure prepared at the original location. Replacement hosts storage mappings, replication components, or network configuration may require work before synchronization can begin with the exact requirements determined by the supported recovery path. Estimate the initial transfer separately from the final interruption. Moving the accumulated changes back can take substantial time while applications remain available at the recovery site, whereas the final cut over needs its own controlled operating window. Measure whether synchronization can actually catch up. If the recovery workload generates changes faster than the return connection can transfer them, the team needs to resolve that constraint before promising a short final interruption to production. Application settings need attention alongside production data, an administrator might have changed an interface destination, installed an approved patch or adjusted service configuration during temporary operation. Those changes need an explicit destination in the restored arrangement. Keep a return change record for that purpose. For each temporary site change, establish whether replication carries it back, whether it requires a separate deployment or whether it should end when operation returns to the plant. Buffet plant events add another moving part before cutover. The interface owner should identify which locally retained events have reached the authoritative application and which remain pending. Then define where those pending events will be processed. Set a processing boundary the team can verify that could use supported queue checkpoints or transaction identifiers, allowing the return procedure to account for outstanding work without assuming every buffer empties at the same instant. Data ownership must remain explicit throughout the move until the approved cutover transfers authority. The recovery site application remains the source of current production state, and the repaired environment must independently accept updates from returning clients. Choose the cutover window with the affected production owners. The procedure should specify when new submissions pause, how active work reaches an approved stopping point, and who confirms that the final synchronization can proceed without leaving an unresolved operation. Then verify the transfer before changing client connections. The application owner needs evidence that the returned state contains the expected final transactions and required configuration, followed by the supported steps for activating services at the original location, a failed database check, an unacceptable application response, or a missing required transaction should trigger a name decision, and you need a documented point beyond which returning to the recovery site requires extra data handling. That last boundary matters once operators resume work. Here's the problem if the original site accepts new transactions after cutover, simply switching everyone back would leave those transactions behind unless the rollback method explicitly preserves and reconciles them. Confirm that the right station retrieves its assigned work, records an approved action, and presents the resulting state correctly to the people responsible for the next operation. In practical terms, you're proving the chain still works end to end. Its services may need to stay available for an approved rollback, but their right paths should follow the cutover plan rather than reopening automatically when someone investigates a fault. That's not a tech problem, that's a governance problem. Verify when your recovery points are being created, which transactions they cover, and whether protection remains effective during synchronization, cutover, and the return to normal operation. You can't assume the backup schedule still applies when the workload has moved. Move service health checks, replication alerts, and backup failure notifications with the operating responsibility. Then confirm the receiving support team sees actionable results before the temporary arrangements are removed. If nobody gets the alert, it didn't happen. That evidence gives the service owner a concrete basis for updating the recovery procedure, because the next outage will begin from whatever configuration and protection state the team actually leaves behind, not the ideal state documented three years ago. Without that continuing responsibility, the configuration left behind can gradually drift away from the conditions. Your last successful exercise actually tested and documented for production use. Which leads to the next question who owns this after the adrenaline wears off. That includes reviewing whether the agreed targets still match production needs. An old requirement shouldn't survive unchanged through several years of different products and operating patterns just because nobody remembered to revisit it. If that person can report a recovery gap but can't obtain engineering time or escalate its funding, the organization has assigned responsibility without providing a practical way to act. That's not ownership. That's a reporting role with no teeth. When a plant manager changes roles or application support moves to another team. Update the recovery responsibilities as part of that transition. The incoming owner should explicitly accept the assigned decisions, not inherit them by default. A dependency change should update the service record and its recovery procedure through the same controlled process. Don't wait for someone to remember a separate disaster recovery document months later. That's how gaps become invisible. Engineers need funded hours to review dependencies, maintain automation, coordinate suppliers, and run exercises. None of that work happens merely because replication and backup charges appear on an Azure invoice. The infrastructure is the easy part. Ongoing costs include replication, retained backups, connectivity, and any reserved or standby capacity. Activation can introduce compute usage and specialist support that normal monthly operation doesn't incur. Two different budgets, two different planning horizons. Ask procurement and application owners to confirm recurring rights, test rights and temporary operation arrangements. The financial plan should reflect how the recovery environment will actually be used across its operating states, not just the cheapest possible configuration. Include test infrastructure, staff participation, supplier assistance where required, and the corrective engineering that follows a failed check. Don't fund the demonstration while leaving its findings without resources for repair. That's theater, not readiness. If someone proposes cutting retention or cancelling a recovery connection, the service owner should identify which agreed recovery requirement would become unsupported. Then take that decision to the responsible business authority. Don't let it happen by omission. Adding another line can increase transaction volume and introduce new dependencies. Its project budget should include reassessing recovery capacity and testing the functions affected before the line enters routine service. Retrofit is always more expensive. The change owner needs to establish whether backup methods restore compatibility or startup behavior changed. Then select the tests needed to renew the evidence affected by that upgrade. A minor version bump can break assumptions you didn't know you had. Moving a firewall or replacing a carrier can invalidate a previously demonstrated recovery path. The change assessment should identify the relevant recovery checks before approving completion. Network teams and recovery teams rarely talk. That's the gap when support transfers to another vendor or a product reaches a different support stage. Confirm who now owns recovery assistance and whether the documented procedures still apply to the installed version. The contract changed, but did the procedure. You don't need to repeat every exercise after every adjustment, but you do need a recorded reason for deciding which previous results remain applicable and which require fresh evidence. That's a decision, not an assumption. For each production function, show the latest demonstrated result, the measured interruption and data loss, the conditions tested, and any unresolved gap that prevents acceptance of the current design. If you can't show it, you don't know it. Include its operational consequence and planned completion date. Then distinguish funded corrective work from a risk someone has explicitly accepted. Don't combine both under a reassuring status. That's how gaps become invisible until the next outage. A missed target doesn't become less relevant because several months pass without an outage. An overdue correction still needs either completion or a renewed decision from the person carrying that risk. Time doesn't fix gaps. The manager can now ask which production function passed its checks, which evidence supports restart and who accepted the remaining restrictions. Instead of relying on a general announcement that it has finished, they get specific actionable answers, maintain procedures and current test evidence. Let the authorized people permit specific work while keeping other work stopped for an identified reason that the recovery team can address. That's the difference between confidence and hope. A restored mess with unresolved material genealogy still leaves a batch on hold, even if every infrastructure check passes and operators can sign in at their stations. The system isn't recovered until the business can actually run on it. I'd like to compare those two cases with you.