Do You Still Need a Historian? Industrial Historians vs Cloud Data Platforms

iFAKT GmbH - Do You Still Need a Historian Industrial Historians vs Cloud Data Platforms 00:00:00 Speaker: Imagine a batch under review with two reports that disagree. The local historian trend puts the temperature within limits, but the cloud report flags an excursion from that same sensor, leaving the release decision on hold. Which record should the plant trust? I want to trace that mismatch, compare the architectures behind it, and define a safe replacement test starting with the production decision rather than the platform. Let's put this in a hypothetical heated mixing process. A vessel holds material that needs to stay within an approved temperature range during part of the mixing cycle, and the batch now needs review before release. There's a possible brief temperature excursion. Whether that excursion affected the material remains uncertain, so the team needs to establish what happened before deciding whether the batch can move to the next step. The operator opens the local process trend. They've used that trend to follow the heating cycle, check how the process settles and investigate problems during a shift. So it's a familiar starting point. The Quality engineer opens the cloud report for the same batch that report flags a temperature problem during the mixing step, and the engineer can't dismiss the flag simply because another system presents a different result. Neither person needs to be wrong about what they're reading. Each could be using the expected tool following the usual review process and interpreting the result correctly, while still reaching a different conclusion about the batch. I'd keep the release question separate from the software argument at this stage, declaring the historian trustworthy because it's established or the cloud report trustworthy because it's newer, gives the team no evidence about the material. The batch remains on hold while people investigate. Production wants to know whether it can free the vessel. Quality needs a defensible decision, and the next shift needs to understand which questions remain open. That pressure changes how the discussion needs to run. You can't spend the morning debating a future data strategy, while the people responsible for the batch still don't know whether the apparent excursion occurred. First, someone needs to investigate the process itself. Did the temperature briefly leave the approved range? And if it did, what else happened around that point in the heating and mixing cycle? That's a troubleshooting question with an immediate operational purpose. The engineer wants enough process history to follow the sequence, assess possible causes, and decide what needs checking before another batch follows the same route The batch review asks something different. The quality engineer needs evidence that supports the release decision under the plant's approved process, including a clear account of any excursion and its possible effect on the material. Those questions overlap, but their answers aren't interchangeable. Finding a plausible cause for a temperature disturbance doesn't by itself settle whether this batch meets its requirements, just as holding the batch doesn't explain the disturbance. The evidence also needs protection while the investigation continues. If someone later asks why the team released or rejected the batch, the answer needs to rest on records they can still examine, not a recollection of a trend. Now consider the person responsible for process performance across several plants. They may want to know whether similar temperature disturbances occur elsewhere, whether one product experiences them more often, or whether the pattern changes over time. That's another use of the same process history. It supports a broader investigation rather than an immediate release decision, and it may remain useful long after this vessel returns to normal production and the shift changes. This is why I'd resist treating history as one workload. The operator investigating a disturbance, the quality engineer reviewing a batch, and the process team comparing plants. Each need history that supports their own decisions. They don't necessarily need separate platforms, but whoever proposes the architecture needs to show how it supports each job. Rather than assuming that a shared source signal gives every user the evidence they need. For me, that changes the starting question before asking whether you still need a historian. Ask what people currently depend on it to do when production runs normally and when something needs investigation. The replacement decision starts with those dependencies. If a plant relies on its historian to understand a questionable batch, then copying measurements elsewhere only addresses part of what the replacement needs to prove. So let's establish what a process historian actually does. We need that baseline before we can compare it fairly with a cloud platform, and decide which responsibilities to keep, extend or move somewhere else. A process historian collects, stores, and retrieves time based process records. Its core job is to preserve a usable account of process values over time, so someone can ask what a particular measurement recorded during a particular period. Start with a temperature measurement from the mixing vessel inside the historian. That measurement usually has an identifier called a tag, which lets applications and people refer to the same stream of process records. The tag identifies the stream rather than one reading under that identifier. The historian stores a sequence of records, each linking a measured value to a timestamp, and where supported information about the readings quality. A value on its own tells you very little. A temperature of eighty means something different depending on whether the engineering unit is degrees Celsius or degrees Fahrenheit. So that unit belongs with the tags definition. Quality describes the status of the reading. Depending on the source and the historian, a quality flag might tell an application that the measurement arrived with a reported fault, or another condition that needs attention. Those details don't always sit together in one stored record. Engineering units and descriptions often live in tag metadata, while individual records carry the value, timestamp and quality information with the exact structure depending on the product retrieval completes the historians basic job. An engineer or application requests records for a tag over a time range, and the historian returns the process history through a supported interface rather than requiring access to its storage files. But the archive is only part of the installed system. When people say they need their historian, they often mean the surrounding collection software and tools as well, because those components shape how they use the history. Follow the measurement into that system. A collector connects to an approved process. Data source obtains readings, and passes them toward the archive using interfaces that depend on the equipment and software involved. Some collectors also hold records temporarily before sending them onward. That buffering can separate the act of collecting a reading from the act of writing it into the archive. Although the behavior depends on the collector and its setup. An operator usually meets the historian through a trend tool. The tool retrieves history and presents the movement of selected measurements over time, letting someone follow temperature alongside other process values without handling archive records directly. Calculations can sit around the archive to a configured calculation might turn several measurements into a derived process value, which users can then retrieve alongside the measured values if the installed system supports that arrangement. An asset model adds another way to find those records. Instead of knowing every tag name. An engineer can navigate through equipment such as a mixing vessel, and find its temperature, pressure and other related measurements. That model gives the tag a place within the equipment structure. Some historian systems include tools for building and maintaining that structure, while other installations depend on separate software or a much simpler naming scheme, you need to check what's actually installed. A product family may include collectors, calculation services, asset tools, and client applications, but that doesn't mean every plant running its archive has licensed or configured all of them. Even two installations of the same product can differ. One plant might use a basic archive and trained client, while another depends on maintained calculations and equipment templates. That took years of engineering work to establish. I described the installed capability before comparing product categories. Ask which component collects each measurement, where it stores the history, and which tools people use to retrieve or work with those records during their normal tasks. The answers define what historian means in that plant. They also prevent a comparison between an entire working process history system on one side and a single storage service on the other, which would miss much of the work. A cloud design can handle some of these same jobs. It may use separate building blocks for collecting records, storing them, running calculations, and giving users access rather than packaging those responsibilities around a historian archive. That changes how we need to frame the comparison before judging whether a cloud platform can take over, we need to establish which building blocks the proposal includes and which historian responsibilities those blocks will actually cover. The phrase cloud data platform needs a more precise definition. In a proposal, it might mean a service that searches process events, a place to keep files, or a wider environment where several teams prepare and analyze data. Those choices answer different questions. A time series query engine takes records organized around time and lets applications search, filter and calculate results across those records without building every retrieval operation from scratch. Think about a request for pressure readings during a production run. The engine executes that request against data it can query using its own query language, data structures, and computing resources to return a result. Object storage has a different job. It keeps objects, often files containing many records, and lets applications write or retrieve those objects through supported interfaces without prescribing the full analysis. Those applications will perform. A folder of process files isn't a query service. You still need software that understands the file contents, selects the relevant records, and runs the calculation that the person asking the question actually needs. A general analytics platform covers a broader set of tasks. It may bring together tools for loading data, preparing data sets, running queries, and sharing results, but the team still needs to choose how those tools support each workload. These categories can overlap within one service. A platform might include both file storage and a time series engine, so I'd examine the proposed data path rather than assume the product label explains how it works, try describing the cloud design through a single request. A process engineer asks for a result, and your architecture needs to explain where the records enter, where they stay, what runs the query, and how the answer reaches that engineer. Ingestion is the entry path that could involve receiving messages as they arrive, or loading groups of records on a schedule with the choice, depending on how soon the intended application needs them. Those responsibilities don't have to live inside one product. You might connect several services together or use a platform that bundles them, but either way, someone has to configure how they behave and keep the connections working. Your team still owns the working application. Someone has to respond when incoming records stop reaching a data set, a query fails after a change, or a report stops giving users the result they expect. That's an operating responsibility, not just an installation task. The proposal needs named. People who understand the data path can investigate failures and have the access and support arrangements needed to restore the service price. The same completed job on both sides for the cloud proposal include the services and engineering needed to take incoming records through to a usable result, rather than stopping the estimate. Once the files reach storage, give each design the same request and response target. If one estimate includes a working report and ongoing support, while the other includes only stored data, the apparent saving comes from leaving work outside the comparison. A fair proposal states what people can actually do. It also identifies the parts your team must build or maintain, which lets you judge the full workload instead of comparing a finished system with rented space. Let's trace the vessels temperature back to the sensor. The probe responds to the material around it, and its transmitter sends a signal that the controller reads through an input channel at a configured rate. That controller reading is our first collection boundary. The sensor's response and the controller's input processing affect, which changes become available before any historian, collector, or cloud connection asks for a temperature value. The collector may work at a different pace. It could ask the controller for its current temperature every few seconds, even though the controller updates that value more often as it runs the process. That repeated request is called polling, unless the source retains intermediate readings and the interface retrieves them, each poll gets the value available. Then, rather than everything the controller measured since the previous request, both collected values could remain inside the approved range. The collector then sends those readings toward the archive, but neither recorded value contains the excursion that occurred between them, so storage never receives that evidence. A faster archive. Can't recover an uncollected reading? Before changing storage platforms, I'd check the actual sensor response, controller update rate and collection settings along the route that supplies the temperature history. The word qualifying needs attention here. A subscription may apply a minimum change threshold, user configured checking interval or limit how often notifications arrive depending on the source and interface. Change based collection still has rules. You need to establish whether the interface sends every detected change only the latest value at each reporting interval, or queued changes up to a configured limit. These settings can explain different histories from one controller. If the local collector receives change notifications while the cloud path pulls the current value, those paths may collect different records before either archive processes anything. I test the path with a known input change during an approved test. Confirm which changes the source, exposes which the collector receives and which reach the archive. Rather than judging coverage from the polling setting alone, a machine state needs a different approach for a signal that switches between running and stopped. The transitions may matter more than repeated readings that merely confirm the machine remains in the same state. Polling can miss a short stop entirely if the machine stops and restarts between requests. Both readings report running, while a supported transition event path could capture the intervening stop and restart. But the total doesn't reveal each part's completion time. If you need individual cycle timing, collecting occasional counter values won't supply the same evidence as collecting the events that mark each completed cycle. I wouldn't apply one collection interval to everything for each signal type. Define whether the required evidence consists of periodic measurements, state transitions, cumulative counts, or events. Then confirm that the source and collector can deliver it. That gives us a more useful collection specification. Instead of requesting all machine data, you can state which changes must reach the archive and test whether the configured path captures them. Under expected operating conditions, a recorded temperature can carry several perfectly valid times. The source timestamp describes when the source reports the measurement, while the collector receipt time describes when that reading reaches the collection software on its way downstream. Cloud ingestion time describes another point in that journey. It records when the cloud service accepts or processes the incoming record with the exact meaning. Depending on the service in the field you're using. Those times answer different questions for the batch investigation. You usually need the measurements placed in the process sequence, while receipt and ingestion times help explain delays along the data path. Keep their names explicit. If a field simply says timestamp, someone building a report may assume it means measurement time, even though the connector filled it with the time of arrival. Even a source timestamp needs a clear definition. An interface might supply the time when a value changed or when a server acquired it, rather than the instant the physical sensor detected that temperature. Clock drift can also distort comparisons between machines. If the heater controller runs ahead of the mixing controller, their records can place a temperature change before a related process action even when the physical sequence runs the other way. The archive can't settle that discrepancy. By sorting harder, you need an approved clock sync arrangement, a way to monitor departures from it, and a record of known timing faults when they affect an investigation. Daylight saving changes make that guess even less safe during the autumn clock change, the same local hour occurs twice, so a timestamp without an offset can refer to two different moments in the batch. Use a consistent time reference for comparison. Coordinated Universal Time or UTC avoids that repeated local hour, while the plant's timezone information lets applications present the record in the local time. People expect conversion needs to preserve the original meaning. If a legacy export contains local times without an offset, document how you resolve ambiguous records rather than quietly assuming every timestamp maps to one certain instant. Extra decimal places don't prove clock accuracy. Preserve the precision the source actually supplies, and avoid presenting a finely formatted timestamp as evidence that the measurement time is equally well known or synchronized. The latest arrival isn't necessarily the latest measurement process. Calculations need a stated rule for using event time, while the data path needs a separate rule for accepting records that arrive behind newer measurements, that rule affects whether a result can change. A report might remain provisional for an agreed period, or a late record might trigger a controlled recalculation, depending on the requirements for that particular use. 00:15:17 Speaker: Carry the source quality information with it, where the source reports a timing concern. Preserve that status and record any timing concerns detected downstream separately rather than replacing the source's own account, correction rules need named ownership and a clear scope. You need to know which records a correction, affects which time applications should use, and how someone reviewing the batch can trace the adjusted sequence back to the received records. Once you know when each record belongs, you still need to figure out whether every measurement stays available or whether archive rules deliberately leave some out. And that's where the word compression gets tricky. It can mean storing the same records in fewer bytes, or it can mean keeping fewer measurements. And those two choices leave very different evidence for a batch investigation. Lossless encoding changes how records occupy storage without losing anything. An archive can encode repeated patterns more efficiently than decode them when requested, returning the exact same values and timestamps you started with. Nothing gets discarded to save space. The saving comes purely from the representation compress, decode, and you recover the original records. Not an approximation of the signal. Measurement reduction works differently. A deadband defines a range of change within which the system may withhold a new reading, depending on the rule and where it's applied. Say the vessel temperature barely moves during a steady heating phase. An exception rule might suppress those small changes before they reach the archive, only passing readings that exceed its threshold or meet another configured condition that reduces the number of records delivered. Some implementations also use elapsed time rules that force a record through after a set interval, so the change threshold alone doesn't tell you the full behavior. Archive compression can also select records after they reach the historian. Depending on the product, an algorithm may retain points that describe the signal within a configured tolerance. Rather than keeping every candidate measurement it receives. The selection rule needs a precise explanation. Some algorithms assess how closely intermediate readings follow a line between retained points, while others apply different logic or treat particular changes as reasons to keep a record. A tolerance isn't a universal promise. Its meaning depends on the algorithm, the engineering units or configured scale, and the conditions under which the historian decides which points to preserve. Read the product documentation alongside the actual configuration. A setting called compression might control measurement, selection, storage, encoding, or part of a wider archive process, so the label alone won't tell you whether measurements disappear. Ask what the archive can actually return afterward. If intermediate measurements have been omitted, a reconstructed value between retained points follows an assumption, usually a straight line change, rather than recovering the discarded observation itself. That assumption can work fine for some engineering tasks. A slowly changing vessel temperature may remain useful for routine process review, with fewer retained points provided. The permitted deviation matches what that review needs to establish. But batch review can require a tighter test. The person approving the configuration needs to consider whether omitted measurements could change the assessment of an excursion, rather than judging the setting only by how smooth the trend appears. Consider a specific version of our hypothetical temperature disturbance. The collector receives readings just below the upper limit, then a brief reading just above it, followed by readings below the limit. Again, every incoming reading might carry good status. Yet if the excursion is small enough to fall within the archive algorithms permitted deviation the retained points might not include that above limit. Reading a tolerance doesn't understand the approved process limit. It follows its configured mathematical rule, which can allow a deviation that looks minor numerically while crossing a boundary that matters to the quality engineer. That doesn't mean every compression algorithm removes brief spikes. Some will preserve the excursion under the chosen settings, and the only defensible conclusion comes from checking the specific algorithm against representative process changes and boundary conditions. I test changes close to the approved limit, include a narrow peak and a sustained crossing, then inspect which received measurements, survive archiving and whether the retained history supports the plant's required review. Test the settings people intend to use. A demonstration with filtering disabled proves little about an installation that later enables measurement reduction, especially if that change happens without review by the people relying on the archive. Keeping every received measurement is another design choice. It supports later examination of those observations, but its storage demand differs from an archive that deliberately selects fewer measurements under an agreed tolerance. Neither approach deserves approval from storage savings alone. The decision needs an explicit statement of which measurements remain available. What reconstruction assumes, and whether those choices preserve the evidence required for this process. There's also a limit to what an expert can recover if the source archive already discarded intermediate measurements copying it's retained records into another platform won't restore those observations, regardless of how that destination compresses its files. First, establish what the collection and archive rules actually preserve, because keeping an approximation for longer doesn't turn it back into the measurements that were omitted. After that, we can decide how long to keep the evidence. How far back might someone need to investigate this batch? That question gives retention a practical starting point, because the answer depends on what the plant might need to establish after the material has left the vessel. Keep ten years leaves several decisions open. It doesn't tell you whether those years contain individual readings or monthly summaries, whether they're immediately accessible or how long an engineer must wait to retrieve them. Duration describes how long the records remain. Detail describes what those records contain. Availability and retrieval speed. Determine whether people can actually use the retained history within the time their work allows. Those requirements can differ within one archive. Records from the current production week might need immediate access, while an investigation into an older batch might permit a planned retrieval within agreed turnaround, that difference should appear in the retention policy. Otherwise, one team can reasonably claim that history remains available, while another discovers that obtaining it takes longer than the investigation deadline allows. A common design keeps recent detail readily accessible. Older periods may use aggregates for routine analysis, with selected source records retained separately. When an approved investigation or evidence requirement calls for that level of detail, an aggregate answers a narrower question. A daily average can support a review of long term operating trends, but it doesn't preserve the sequence of temperature changes that someone might need to examine within a particular batch. Choose the summary around its intended use. If the process team needs long term ranges or totals, define those retained results explicitly, rather than assuming that one average will answer every question people ask later. The source evidence needs its own retention decision. Keeping a summary for several years doesn't establish whether the underlying observations survive, so the policy should state both what remains and what disappears after each period. For our vessel, the future question might concern batch genealogy tracing, where material came from and where it went, including whether material from this mixing step later entered several finished batches or shipments. The investigation might start much later. A customer complaint could prompt a review of the affected material's production history, making the relevant retention horizon longer than the time production normally spends. Reviewing a completed batch process. Investigations can reach back for other reasons, too. An engineer might need older runs to understand when a recurring disturbance began, or compare conditions before and after a change to the approved operating process. Contractual obligations can extend the requirement as well. A supply agreement may specify which production records must remain available and for how long, so the plant needs to check those obligations before choosing a shorter archive period. There is no universal duration. I can give you here quality operations, records management and the people responsible for legal or contractual obligations need to approve a policy that fits the material, the process and the commitments involved. Once an agreed retention period ends, the organisation needs a defined action, subject to any investigation hold or other requirement that prevents those records from being deleted. Copies make that action harder than it sounds. Deleting a period from the main archive doesn't tell you whether the same records remain in a cloud export, a backup, or a data set prepared for analysis. Each copy can follow a different lifecycle. An export might retain records indefinitely unless someone configures disposal, while a backup may expire through its own schedule rather than through individual record deletion. Derived data sets need attention as well. A table of batch results might remain useful after detailed measurements expire, but its retention, purpose and disposal rules still need approval rather than an accidental exemption. Backups also aren't automatically a historical retrieval service. A backup retained for recovery may require restoring a larger environment before anyone can locate one batch, which makes its practical use different from an archive. People can query right those differences into the policy for each retained data set, identify its purpose, level of detail, retention period and retrieval expectation. Then name the person responsible for keeping that arrangement working as intended. That gives an investigation a clear starting point. The reviewer can establish which evidence should still exist, where to request it, and whether a missing period represents planned disposal or a failure to follow the policy. Even retained records don't determine the answer by themselves. Two applications can read the same surviving history and calculate different results. Which brings us from what we keep to what we ask those records to mean. Start with the request behind the batch report. Give me the temperature during mixing. Sounds clear, but it leaves the query engine to apply rules that the reviewer may never have agreed to use. A recorded sample query returns the observations held in the archive. A query for values at regular intervals can instead return calculated estimates, placing a temperature at each requested time, even when no observation exists. At that instant, that difference changes what the reviewer receives. A report requesting one value per minute might omit a recorded peak between those times, while a maximum calculated from recorded samples could still find that peak. Neither request answers every batch question. If you're checking whether any recorded temperature crossed the limit, ask for that explicitly, rather than assuming that regularly spaced results preserve every recorded extreme averages, introduce another choice that often stays hidden. A sample average adds the selected values and divides by their count, giving every included record equal weight regardless of how much process time it represents. Here's the problem most manufacturers don't talk about. Imagine you're reviewing a mixing step where most of the temperature readings cluster around a short disturbance, and only a few readings cover a long, steady period. The sample average gives that cluster way more influence than it deserves. A time weighted average fixes this by weighting each temperature by how long it actually lasted. You define an assumption about how the value behaves between records, then calculate an average across the full elapsed time. The result is that identical input records can produce different answers depending on which method you pick. So if your batch rule says average temperature over the mixing period, you need to establish which averaging method implements that rule before you compare what the historian shows against what the cloud calculation returns. That distinction matters more than most people realize. With linear interpolation, the calculation assumes temperature changes steadily between neighboring records. That estimated path determines both the average and any calculated duration above a limit. A machine state needs different treatment entirely for a running or stopped signal. A step rule usually carries the last known state forward until the next transition. You don't want to invent a gradual movement between stopped and running a half running machine isn't a useful calculation. When estimating operating time, define which record starts each state and how long that state applies. Then use those intervals rather than treating state codes as continuous measurements. Suppose the mixing step begins between two temperature records. You've got one observation before the start and the next observation after it, with no recorded value exactly on the boundary. What temperature starts the calculation? The query could estimate a boundary value using neighboring observations, carry the earlier value forward, or limit itself to records inside the window. Each choice depends on the approved calculation method, and those choices change the period the result actually describes. Using only internal records may leave the start uncovered, while estimating across the boundary uses information outside the batch window to describe conditions at its edge. The end boundary needs an equally clear rule if one phase ends exactly when another begins, define which phase includes a record at that instant, so adjacent calculations don't accidentally count the same event twice. When a requested interval contains no records, the query might return nothing, report an unknown value, or calculate an estimate from surrounding observations under a stated rule. Here's the thing. An empty interval doesn't automatically mean zero. Substituting zero changes averages, and limit checks while carrying a previous value forward can imply continued knowledge of the process that the available records don't actually support, bad quality exclusions can change the denominator as well. If a batch calculation removes unusable readings, define whether it also removes the associated duration, leaves that period unknown, or applies an approved estimate across it. A clean average can cover an incomplete period. I'd want the result to state how much of the batch window supports the calculation, so the reviewer can distinguish acceptable temperature from insufficient evidence about temperature. Specify the input records, interpolation behavior, window boundaries, and handling of unusable periods in terms that both the process reviewer and the person building the query can test. Then inspect what sits beneath the reported number. Request the contributing records and the calculations coverage because a convincing result can hide an excluded interval or an estimated boundary that changes the release decision. Did the instrument measure zero or did some component insert zero because it couldn't obtain a reading from the source when the application requested one? Those cases need separate records. A real zero reports an observation, while a missing value reports an absence of evidence and a failed sensor. Reports a fault that may leave its numerical output unusable. A stale reading creates a less obvious problem. The displayed temperature might remain perfectly plausible, even though the source stopped updating, so the number alone gives the reviewer no reason to suspect that measurement has stopped. Fresh delivery doesn't prove a fresh observation, either. A connector could keep sending the last available temperature with new delivery times making the pipeline appear healthy, while the underlying instrument contributes no new information about the vessel. When a collector maps source records into a message and another service turns that message into a table, both steps need rules that preserve the source status. An unmapped status shouldn't quietly become good if the destination uses different quality codes, retain the original code alongside the mapped results so someone can inspect what the source reported and how the conversion interpreted it. Downstream checks need their own fields. A source can report a reading as good, while a later check flags an implausible change. Keeping both assessments lets the reviewer distinguish instrument status from analytical suspicion. A detection rule should compare the observed arrival or measurement pattern against the expected behavior for that signal, rather than apply one fixed silence threshold across every measurement, some signals legitimately stay quiet for a change based stream. No new record might mean no qualifying change, so gap detection needs whatever connection, status or health evidence the collection path can provide. Duplicate checks need similar care. Two records with the same temperature aren't necessarily duplicates. The vessel can hold steady across several genuine observations, each with its own measurement time and place in the sequence. Repeated delivery needs a narrower test. Compare the available record identity and source fields under a documented rule, then flag uncertain matches instead of deleting observations simply because their numerical values happen to match a temperature that remains exactly unchanged through an active heating cycle deserves investigation. Although that pattern alone doesn't prove the sensor or collection path has failed. Implausible changes deserve a reason. Threshold two engineers can define checks using instrument range and credible rates of process change, while recognizing that a flagged jump might reveal a real disturbance rather than corrupt data for records that fail an agreed check. A quarantine path can keep them available for investigation, while preventing unreviewed values from entering calculations that require accepted measurements. Quarantine shouldn't erase the uncertainty from the batch. If excluding a suspect record leaves part of the mixing step unsupported, the reviewer needs that gap reported, not a result that appears complete because the troublesome observation disappeared. Someone also needs to resolve the flag, assign the review to a person with the process, knowledge and authority to accept the observation, confirm a fault or request further evidence before any correction enters the accepted dataset. 00:30:59 Speaker: Preserve the received record, then attach the changed value or status. The reason the reviewer and the time of the change through a traceable correction record. Silent cleanup makes later review harder. If a process replaces a suspect temperature with an estimate, label that estimate and record its method, rather than letting it pass as another observation from the instrument. The same discipline applies when a flag proves mistaken. Retain the check result and its resolution, so a later reviewer can understand why an initially rejected measurement eventually entered the accepted history for this investigation. But even a credible temperature record doesn't tell the reviewer which production activity it describes. That missing context can still leave the batch question unanswered. The next question is which production activity that temperature belongs to? A sensor identifier can lead you to the vessel, but assigning its readings to a batch requires a connection to what the vessel actually ran during that period. Equipment location alone won't settle that assignment. In our hypothetical process, the same vessel could hold one batch in the morning, undergo cleaning, and start another batch later with the temperature signal continuing throughout. Each reading needs its place in that sequence. Link the sensor to the vessel, then connect the vessels production activity to the process. Step, batch material and work order that apply at the measurement time. The asset tells you where the measurement came from, while the process step tells you whether it describes heating, mixing, holding, or another activity within the batch, the batch identifies the particular production run under review material context, establishes what the vessel contained, and the work order connects that run to the production request that authorized the work and its intended output. I'd follow those connections before selecting temperatures for the release calculation. Otherwise, a report can retrieve accurate vessel readings, yet include a cleaning cycle or attach part of the following batch to the material currently under review. Where the plant records these activities. There it can identify which batch ran on, which equipment and when the relevant process steps started and ended. Use the execution records that describe what actually ran a planned mixing window might differ from the recorded activity because the operator delayed a delayed start, extended a hold or move the work to another approved vessel. That distinction affects which measurements belong in the review. If the report assigns temperature using the planned vessel while execution records identify a different vessel, the query can examine a perfectly credible signal from equipment that never held this batch. The Ms. doesn't automatically solve every relationship. Your integration still needs to connect its equipment, identifiers and batch references to the identifiers used by the process archive, including cases where those systems name the same vessel differently. It commonly holds production order and material information, giving the investigation a connection from the executed batch to the requested product and its business records. Check which system owns each detail in your installation. Actual material consumption might be recorded during execution and then posted to ERP, so the planned ingredients and quantities shouldn't silently replace the account of what production used. Shared identifiers make those connections much easier to follow when a work order reference changes. Format between systems. Define the mapping explicitly so the batch report can reach the intended order without relying on someone recognizing a familiar description. Suppose maintenance moves the temperature probe to another vessel, or an approved equipment change. Assigns its archive tag to a replacement instrument while keeping the familiar name. If this sounds familiar, you're not alone. Most manufacturers hit this wall right after their first power BI rollout. The data looks great until someone asks where it actually came from. That's usually where the real transformation work begins. Here's the problem most manufacturers don't talk about today's equipment mapping won't explain last year's batch. If you're investigating a temperature deviation from six months ago, you need to know which sensor was connected to which vessel at that exact moment, not whatever the plant layout looks like right now. Store when each mapping applied. So an investigation retrieves the sensor to vessel relationship that existed during production, rather than applying the current equipment structure to every retained measurement. A tag rename creates a related problem. Engineers replace a local naming scheme with a plant wide standard, and suddenly historical batch references point nowhere preserve the connection between the old and new identifiers, so those older records still lead to the intended measurement. Stream units need the same time aware treatment. When an approved change switches the recorded temperature unit from Celsius to Fahrenheit, retain when that change took effect, an older value needs the unit that applied when the source recorded it, not whatever the current configuration says. A current unit label can't explain an entire archive. Any conversion used for review needs to refer to the applicable unit definition, while the equipment mapping needs to refer to the applicable version of the plant arrangement. That's not a reporting problem. That's a foundation problem. I'd expect a reviewer to follow those relationships without guesswork. Starting from the batch, they should be able to identify the executed step, the vessel involved, the relevant sensor, and the order and material records that explain the work. If they can't, the architecture hasn't done its job. That requirement extends beyond normal connectivity. If the plant needs those relationships to investigate a batch while its cloud connection is unavailable, the architecture must keep the required context usable alongside the process history people can still access locally. You can't investigate a quality issue. If the data model lives somewhere you can't reach. An outage needs a location before it needs a recovery plan. Losing the connection between a controller and its collector creates a different problem from losing the plant's external connection, while local collection continues to run normally, those are separate failure modes requiring separate responses. Start inside the plant network. If a failed switch prevents the collector from reaching the process source, a queue downstream of that collector can't preserve readings the collector never receives from the affected equipment. The buffer only helps if the data reaches it in the first place. A wide area network fault interrupts a different part of the route. The plant might continue collecting measurements locally, while its connection to the cloud remains unavailable, leaving the transfer path responsible for holding records until delivery can resume. That's where buffering becomes essential. The cloud service itself can also become unavailable. Your external connection might work while the receiving service rejects requests or stops responding, so connection health alone doesn't establish whether measurements are reaching their destination. You need to check both sides. Those failures need separate detection and response. Otherwise, a support team might spend time checking the external connection. While the actual interruption sits between a local collector and the controller supplying the vessel temperature. That's wasted effort during a production issue. For records that reach the edge, buffering provides temporary protection. The edge here means the computing equipment near the production source, where a transfer component can hold measurements, while the next destination can't accept them. Think of it as a holding area that buys you time. That buffer needs a defined persistence guarantee. A queue held only in memory can disappear when its process restarts, while a persistent queue writes records to local storage under a documented method for surviving interruptions. Memory only is gambling with your data. Test the guarantee rather than trusting the label. Restart the transfer component during a controlled disconnection, then confirm which queued records remain available and whether delivery resumes from the expected point without manual repair. If you haven't tested it, you don't know it works. Finite disk capacity puts a limit on that protection size, the queue against the expected incoming workload and the outage duration you intend to tolerate, including operating margin for busier production periods and storage overhead. A queue that fills in thirty minutes during a peak shift isn't protecting anything. Every queue eventually needs a full disk rule. Does collection stop? Do the oldest records disappear, or does the system reject new records? And who receives a warning before that choice begins affecting retained history? Decide before it happens, not during a capacity estimate. Also needs a response plan. An alert that arrives shortly before the disk fills gives little protection. If the person receiving it can't extend storage or restore delivery within the remaining time, the alert is only useful if someone can act on it. Replay needs equally clear behavior once the destination becomes available. The sender needs rules for retrying failed transfers and removing queued records only after the configured acknowledgment confirms that the receiving stage has accepted responsibility. Don't delete until you know it arrived, then check what that acknowledgment actually guarantees acceptance by an intermediate messaging service might confirm only that hand off. Rather than prove the records have reached the data set that the engineer will query during an investigation. Acknowledgement depth matters queued records don't automatically provide local investigation access. A transfer buffer may preserve measurements perfectly, while providing no supported way for an operator to retrieve the vessel's temperature history during the disconnection. The data exists, but nobody can use it. Specify the local job separately if the shift engineer must inspect the current batch while disconnected. The design needs an accessible local history service and trend tool with enough retained coverage to support that investigation. Don't assume the buffer doubles as a historian. Check the tool's dependencies during the test. A locally installed trend client can still fail if opening a session, finding measurements, or retrieving history requires a remote service that the outage has put beyond reach. Tested while disconnected, not just during normal operation, recovery creates another operating condition, not an instant return to normal. The plant continues generating measurements while the sender works through its accumulated Q so both live traffic and replay traffic compete for the available transfer and ingestion capacity. Recovery is a process, not a button. The backlog shrinks only when successful delivery outpaces new arrivals. If recovery merely matches the ongoing collection rate, the oldest queued records keep waiting. Even though monitoring reports a healthy connection and successful transfers, you need headroom to catch up. Set a recovery target people can verify. Decide how quickly current readings must become available, how long the oldest backlog may wait, and whether replay traffic needs limits to protect the live path during recovery. Make it measurable, not aspirational. Then rehearse the recovery under a representative workload. Disconnect the transfer path. Let the queue grow, restore access, and measure whether the backlog clears without filling disks or leaving local users unable to retrieve needed history. A dry run reveals problems. Documentation misses a drained queue isn't the only path condition. Confirm that people could investigate locally throughout the interruption, and that the receiving system accepted the replayed records within the agreed recovery window, rather than merely watching the sender resume Activity recovery is about outcomes, not activity. That recovery path also crosses the plant boundary. It needs an approved connection design that preserves operational security during normal transfer and backlog replay, rather than depending on someone opening broader access when production needs its records back. Security shouldn't be an afterthought during an outage. Moving history outside the plant changes who can reach what operational technology or OT includes the systems that monitor and control the physical process. While information technology, or IT supports the wider business and its computing services, those environments have different priorities. Those environments carry different responsibilities. A connection that supports convenient enterprise access still needs review before it reaches equipment whose availability affects production, even when the intended purpose involves reading temperature history. Convenience doesn't override safety. A trust boundary separates systems with different access and security rules. Crossing that boundary requires an explicit decision about which component can connect, what it can request, and which identity gives it permission to perform that work. Default deny is the right starting point. I'd start with an approved communication path. Describe the initiating component, its destination, the permitted protocol, and the purpose of the connection. So the security review concerns a specific exchange rather than a request for general access. Specificity makes review possible. Network zones help constrain that exchange. Depending on the plant's approved design, an intermediate zone can separate production sources from enterprise or external services with controlled connections between zones, rather than broad access across them. Zones contain the blast radius. Direction matters, but it doesn't settle the whole review. An outbound connection can still carry responses or commands, so you need to inspect the application's capabilities, as well as which side starts the network session. Don't assume outbound is safe. Give each transfer component an appropriate service identity. That means an identity for software rather than a shared engineer's account, with permissions limited to the sources and destinations that component needs for its assigned task service accounts. Not shared logins. Separate permissions where responsibilities differ. Software that sends process records shouldn't inherit authority to change collection settings or administer the receiving environment, simply because using one powerful account simplifies the initial installation. Convenience now means risk later. That's least privileged access. In practical terms, give the component enough authority to perform its job, then verify that attempts outside that scope fail, including requests to unrelated equipment or administrative functions on the destination, test the boundaries for acquisition use. Read only access where the interface supports it. Verify the permissions at the process source and through the connector. Because a reporting application can still depend on credentials that permit writes nobody intended it to perform. Check what the credentials can actually do. Process control stays within its approved systems. Exporting vessel history doesn't authorize an analytics application to change a temperature set point, and any separate requirement for remote control needs its own engineering and security approval. Different purposes. Different permissions. Configuration also needs an owner after commissioning. Name the team responsible for patching the collector. Host transfer software and related components, including who checks vendor guidance and approves changes that could affect production access. Ownership doesn't end at deployment. A patch deadline isn't a plan change plan. The OT and IT teams need an agreed way to assess exposure, test the proposed update and schedule deployment within the plant's operating constraints without leaving ownership. Unresolved coordination prevents conflict credentials need the same operating discipline, assign responsibility for renewing certificates and rotating secrets, then test that renewal changes every dependent component before an expired credential blocks an otherwise healthy collection path during production. Expired credentials are a preventable outage. Avoid credentials embedded in scripts or personal files. Use an approved secret management arrangement with controlled access, and make sure support staff can identify the responsible identity without needing to reveal its secret during routine fault investigation. Secrets stay secret. Access logs should explain actions, not just announce successful connections, record authentication failures, permission changes, configuration edits, and administrative access were supported so investigators can distinguish an unauthorized action from an approved maintenance task affecting the transfer path. Context matters in logs. Logging needs a response. Owner to someone must review relevant alerts and know which actions they can take, rather than leaving evidence scattered across systems that nobody examines until production reports a problem. Logs without review are just noise. Containment needs to respect the physical process. If a transfer host appears compromised, responders should know how to isolate that host or revoke its identity without casually disconnecting equipment that production still needs to operate. Security actions have operational consequences. Agree that procedure before an incident, operations and security need named contacts, decision authority, and a clear boundary between stopping data export and intervening in the control environment during an active production run. Don't figure out the chain of command during a crisis. Microsoft services belong inside this approved security design. Once the collection path and its permissions are defined, we can place cloud services against those responsibilities without treating a product choice as permission to cross the plant boundary. The architecture governs the product, not the other way around. For our vessel, the collector could send buffered temperature records into a cloud ingestion service, which then passes them to the environment responsible for answering process questions. That route doesn't require one fixed set of products. The receiving service depends on whether you're connecting individual devices, forwarding an existing industrial data stream, or transferring records through a supported integration from the plant. Azure IoT hub fits the device messaging case. It provides managed communication between connected devices and cloud applications, with device identities and messaging capabilities that can support an approved edge to cloud transfer path. For industrial measurements, the name needs unpacking once IoT means Internet of Things. In this architecture that describes connected equipment exchanging messages rather than a requirement to expose the vessel controller directly to an internet facing service or endpoint. 00:46:36 Speaker: IoT hub isn't an industrial protocol translator. If the source speaks a factory protocol, a separate gateway, connector, or edge application needs to obtain those readings and package them for the messaging interface. That component might come from an industrial software supplier. Microsoft provides the cloud messaging service, while the chosen industrial integration supplies source connectivity, and your team verifies how those parts work together in the installed environment. Device messaging isn't mandatory for every process history workload if an existing export service already delivers records through another supported ingestion route, placing IoT hub between that service and its destination needs a specific reason for a request to find temperature excursions across many production runs. A query engine built for time oriented events can provide a suitable place to search and calculate results. Azure Data Explorer is one option for that work. It's a managed analytics service that can ingest and query large volumes of event data, including industrial time series records using Kusto query language, usually shortened to Kql. Microsoft Fabric Event House provides another route within fabric. It manages KCL databases for event oriented analytics, giving a fabric based team and environment for querying incoming measurements and related events without treating every request as a file processing job. These aren't services you automatically place one after another. I'd examine Azure Data Explorer or Event House against the intended query, workload and operating environment. Rather than assume the architecture needs both to handle the same temperature records, a useful request would name the production question precisely. An engineer might ask for recorded temperatures above the approved limit during selected mixing runs, with the relevant times returned, so the investigation can examine those observations. The query engine supplies the tools to execute that request. Your implementation still defines the tables, makes the required records available, and expresses the calculation. Because selecting a time series service doesn't establish the plant's approved interpretation of its measurements. Fabric one Lake provides shared logical storage for fabric data, while a lakehouse organizes files and tables that teams can use for preparation, analysis, and other supported data workloads that can bring process records into wider manufacturing analysis. For example, a prepared data set could combine selected temperature history with batch outcomes and maintenance information, allowing a process team to examine relationships beyond one stream of measurements, one, like an event house, aren't interchangeable names for the same job. A proposal should explain which questions use the event query environment and which use lakehouse data sets, including how the intended users reach the appropriate records through supported access paths. Power BI then handles the reporting experience. It can present results from supported data sources and semantic models, giving the Quality engineer a batch report without requiring that engineer to submit queries directly to the underlying service. The report still needs a defined source and refresh behavior. Connecting power BI to this architecture gives people access to results, but the design must state which data set supplies them, and when newly accepted measurements become available there for the vessel investigation, trace one request all the way through the approved collector supplies records. Buffered transfer carries them toward ingestion, and the chosen query environment returns the evidence that the report needs for its specific purpose. Don't select this whole service set by default. Assign each service a job that someone actually needs. Then examine whether that job calls for fast event queries, broader retained data sets, or a reporting layer over results already prepared elsewhere. Once you've chosen a query service, you still need to decide which records it can search directly and which records require preparation before a query can reach them. Distinguish query ready storage from a cache. Query ready records set in structures. The engine can search while a cache keeps selected data on faster storage to reduce the work needed for repeated or recent requests. A cache usually isn't a separate retention promise depending on the service. Older records can leave the fast cache while remaining retained and queryable, although accessing them may take longer than accessing recently used records. Retained files occupy another position. A file can remain safely stored without being registered in a query engine, indexed for the intended search, or available through the same interface used for current production history. For the engineer, those differences become waiting time. Asking for yesterday's mixing run might return promptly, while asking for a run from several years ago might require locating files and loading them into an investigation environment first. If engineers usually investigate one plant over a narrow time range, arranging records into suitable plant and time partitions can help the engine avoid reading unrelated files. A partition divides a data set into addressable portions. The layout should let a request exclude portions that can't contain relevant records. Rather than force every investigation to scan history from every plant before filtering the results. Finer divisions aren't always better. Creating a separate partition for every tag and tiny time interval can produce so many small pieces that locating and opening them becomes a large part of the query workload. Match the layout to actual access patterns. A fleet wide comparison over several months may need a different arrangement from a request for neighboring measurements around one vessel event, so test both before settling on the storage structure. File size enters that decision to. Writing each incoming message as its own file creates repeated overhead for file discovery, metadata handling, and read requests. Even when the total amount of measurement data remains modest. Batching can reduce that overhead. A writer accumulates records and writes larger files, giving downstream processing fewer objects to manage and potentially more efficient reads when the file format and query engine support them. The trade off is delay before those records become available. Waiting to fill a larger batch can improve storage efficiency, but it can also leave the newest measurements outside the data set, and engineer currently queries during an investigation. Choose that delay for the job. A background process can also combine small files later where supported, but that work needs computing resources and a schedule that doesn't continually compete with the queries people need to run. Some services keep older data directly readable with different access charges, while archive tiers can require a restore or rehydration operation before applications can read the file contents. Don't treat those tiers as interchangeable before placing historical records there, establish the retrieval steps, the expected waiting period, and whether the investigation needs a temporary, readable copy or a load into another query environment. Then rehearse one request for an older batch. Have the responsible team locate the retained files. Request any required restoration, prepare them for querying, and confirm when the engineer can actually begin examining the process. Records retrieval can trigger several distinct charges depending on the service. The bill may include retrieval operations, restored storage, processing, and data transfer, so use the chosen service's current terms rather than assume one universal archive price. Temporary copies also need an expiry plan, otherwise, each investigation can leave restored files and query ready duplicates behind, gradually turning a deliberate, low cost retention design into several permanent stores that nobody intended to maintain. The storage path now has an explicit purpose. Current questions use an agreed query path. Older investigations follow a tested retrieval procedure, and file layout supports those access patterns. Rather than merely organizing records by convenience, the transfer must preserve what those records represent, because a well-organized destination won't help if the extraction changes the evidence before the query ever runs, the extraction interface becomes part of the evidence trail before connecting anything. Establish which supported interface exposes the historian records you need, and whether that interface returns them with the fields your destination must retain. That could be a vendor connector or a documented programming interface. Some installations provide a supported database style query interface, while others require a dedicated client, library or extraction service to retrieve history through an approved route. Supported access doesn't necessarily mean unrestricted access. Check whether your license permits the intended extraction, whether a separate interface entitlement applies, and whether the vendor places documented limits on requests or returned records. A connector name won't answer those questions. Ask which historian versions it supports, which retrieval methods it invokes, and whether its configuration exposes the source fields required by your particular use of the records. I'd examine an actual extraction response before designing the destination table. Take a small known period and compare the returned fields with the source interface documentation, including anything the connector renames, converts or leaves out. Preserve the source identifier, not just the friendly description. If several historians feed the same destination. Include enough source information to distinguish identical tag names that refer to different measurement streams in different installations. Keep the quality code in its original form, where the connector translates that code, retain the translated results separately and identify the mapping used so the extraction doesn't conceal distinctions. The source interface actually supplied units, and timestamps need explicit extraction behavior to record whether units arrive with the response, or require a separate metadata lookup, and retain the timestamp, precision and time reference delivered through the chosen interface. Lineage means an account of where the records came from and how they were obtained. The source system retrieval method, requested interval and extraction software version all need to be documented. That account makes troubleshooting much less speculative. When an export looks wrong later, you can identify the request and configuration that produced it, instead of guessing the extraction method from whatever values happened to remain A common approach retrieves a bounded interval and records a checkpoint. A durable marker identifying the portion successfully committed to the destination under the transfers agreed rules, the checkpoint belongs after confirmed commitment, not before the request. If extraction succeeds, but the destination right fails. Advancing the checkpoint prematurely means the next run skipped records that never reached their intended table. Partial responses deserve particular care. An interface may divide a large result across several pages, so completing the first page doesn't justify marking the entire requested interval as transferred when further pages remain unread. That's idempotency. The transfer can deliver the same source records again, while the destination applies. A documented rule that prevents accidental duplicate entries from accumulating record identity determines whether that rule works. Use a source event identifier where one exists, or define a tested combination of source fields, recognizing that a tag and timestamp alone may not uniquely identify every returned record, don't let a retry silently overwrite a different observation if the destination encounters conflicting records under its chosen identity rule, retain enough information to investigate rather than assume the latest delivery must replace the previous one. An interface can return retained archive records values estimated at requested intervals or calculated results. Over time. Windows and a connector may expose several modes under similar settings. For an archival extraction request. Retained records explicitly verify how the interface handles interval boundaries, because some retrieval methods can include estimated boundary values even when most of their output consists of recorded archive entries. Label estimated exports and summaries as separate datasets. A regularly spaced export or an interval maximum may support its intended application perfectly well, but neither should enter the destination under a label claiming it contains the original archived observations. Retain the parameters that produced each derived export requested spacing, calculation type and applicable retrieval options belong in the extraction record, giving another engineer enough information to reproduce the request and interpret its output correctly. It includes supported access controlled progress, repeatable delivery and a documented account of the retrieved records, all of which carry operating costs beyond the destination storage bill. The cost question now moves from buying software to funding an operating service for the hypothetical vessel. That means paying for dependable access to process history during ordinary production and during the investigation that keeps a batch on hold. An existing license purchase belongs to the past. While renewal charges, support agreements, capacity increases, and interface entitlements can still affect what the plant spends from this point forward, check the commercial terms rather than assuming a standard pricing model depending on the supplier and agreement, adding measurement streams, users, connected systems or another site can change the bill in ways that aren't obvious from the original purchase infrastructure belongs. Beside those license charges include the machines or virtual servers, storage and hosting resources assigned to the service then account for the additional infrastructure that supports the plant's required availability and recovery arrangements. Administration also consumes paid time, Even without a separate invoice. Someone maintains the historian environment, manages access, investigates service problems and coordinates supplier support, so estimate that work from the actual installation rather than treating internal staff as free. Removing one analytical workload might reduce demand without eliminating a server support agreement or administrator's responsibilities. The saving depends on what you can genuinely stop purchasing or doing. Cloud spending needs a different breakdown depending on the service and pricing agreement. Accepting records can incur ingestion charges or consume purchased capacity, while transformations and other processing use computing resources that someone needs to budget storage covers only the retained footprint. Query execution can add its own consumption or compete within a shared capacity allocation. So the operating cost depends on how often people request results and how much processing those requests require. Networking also has a place in the estimate. Check the actual route and service terms for outbound transfer movement between regions, and connections to other environments. Not every transfer carries the same charge or pricing. Treatment recovery adds demand beyond the normal daily workload following an interruption, replaying queued measurements and rebuilding affected results can require extra processing, transfer and temporary storage depending on the design and how quickly servers must return. During coexistence, the plant can continue paying historian support and infrastructure costs, while also funding cloud ingestion, retained copies and the people responsible for maintaining the connection. Parallel operation isn't automatically waste. It can support a deliberate operating choice, but the estimate should state which duplicated costs remain part of that choice rather than present them as savings already achieved. People cost across both architectures. Production support needs someone available when access fails during a shift, and that responsibility can require an on call arrangement, supplier coverage or staff training beyond the effort needed to build the first report. Security reviews consume engineering time to assessing a changed connection. Reviewing permissions and approving updates involves people from several teams even when the underlying service charges appear modest on a monthly statement. Engineering effort deserves its own budget line. Include the work required to maintain transformations, investigate unexpected results, and adapt integrations when source systems change, rather than hiding those tasks inside a general assumption that the platform team will cope. Imagine a quiet reporting application that refreshes an agreed set of prepared results, with a small group of users opening those results during their normal working day. Then give analysts access to the same history. If several people repeatedly scan years of measurements across all plants, their queries can demand far more processing than the quiet reporting application, despite using exactly the same retained records. The charging effect depends on the service in the consumption model, that activity may increase billed usage, while a capacity based model may absorb it initially but eventually require more capacity or tighter scheduling to maintain response times, historian demand can create capacity pressure as well. Heavy analytical retrieval may compete with ordinary users or prompt an infrastructure upgrade, so neither architecture deserves a cost estimate that assumes its workloads never interfere with each other. That gives us a fair basis for comparing designs against work the plant actually needs. Rather than choosing between an attractive license quote and an attractive cloud storage price. Put those production questions into a written test brief before either team prepares its demonstration. Agree on the work both designs must complete and the conditions under which the plant will judge their results. Use a shared measurement feed for the ingestion test. Preserve its timing and workload pattern, including busy periods, so both candidates receive equivalent demands rather than one receiving a convenient, evenly paced stream that production never actually produces. The workload needs to extend beyond accepting. Measurements include a controlled interruption and recovery, an investigation of our hypothetical mixing batch, and a comparison across plants, with each task producing an answer that someone can check. Give the batch investigation a specific deliverable. Ask each design to return the evidence and calculation required by the agreed review. Then have the same reviewers assess both outputs against the same expected result. 01:02:28 Speaker: The cross plan task should also reflect a real request. Select an agreed process question across comparable equipment and production periods, then specify the result needed rather than letting each demonstrator choose whichever calculation suits its platform. Otherwise, a quick response can distract from an incomplete answer, or a successful demonstration can quietly become the acceptance standard simply because nobody defined a stricter one. Retention must match across the candidates. If the test uses a reduced historical data set, state that limitation and distinguish measured performance from projections about how the design would behave with the full retained history. Calculation, rules must match as well. Give both implementations the approved query definition, then check that any differences in returned results follow from identifiable input differences, rather than incompatible interpretations hidden inside their queries. Set response targets for each task separately, and operators immediate investigation and a scheduled cross plan study can tolerate different waiting periods, so judging everything against one speed target can favor the wrong design for the actual work. Define what counts as the start and finish for an investigation. Measure from the user's request until the required evidence becomes usable, including any preparation that the proposed operating procedure requires before the query can run. Availability requirements belong in the same brief state. Which tasks must remain possible during each tested interruption, and judge both designs against that requirement. Instead of giving either one an exemption after it loses access, run congestion long enough to expose accumulating delays. Measure whether the receiving system keeps pace throughout the agreed workload, not merely whether it accepts a short burst before pending work starts building somewhere downstream. Keep the investigation workload active during that run that lets you observe whether queries and incoming measurements interfere under the proposed configuration, rather than awarding separate passing results to activities that the plant needs simultaneously. Measure recovery as its own operating period. Record how quickly the design restores the required service, and clears delayed work, while continuing to meet the response targets assigned to users during that recovery window. Capture the service configuration, allocated resources, software versions, data set size, and relevant operating settings so another engineer can reproduce the conditions behind a claimed result or investigate a failed run log. The human effort alongside the machine results include preparation, supervision, troubleshooting and any manual intervention needed to finish a task. Identifying which actions would remain part of routine operation after the demonstration ends. Assumptions need names and dates, not footnotes. Nobody owns. Identify who supplied each workload estimate and availability assumption. Then note where the test uses measured plant behaviour and where it relies on an expectation that still needs confirmation. Attach a pricing date to the cost calculation, record the commercial basis, currency and applicable agreement or quoted terms, so a later review can distinguish an architectural difference from a price change. Since the test, a failed run can still teach you something useful. Change the configuration, deliberately retain the earlier result, and repeat the affected tasks so improvements don't erase the evidence of what the original proposal could actually support. If the local historian meets a plant facing requirement that the proposed alternative still misses that unmet requirement, gives us a concrete reason to examine keeping the historian. Keeping the historian can be a deliberate engineering choice. If production needs an independently usable local record and the installed historian already meets that requirement, retaining it protects a capability the plant has a reason to fund. External connectivity doesn't have to fail often to shape that decision. A plant might tolerate occasional delays in business reporting, but still require uninterrupted access to recent process history whenever an engineer needs to investigate a production disturbance. The deciding factor is the permitted interruption to that work. If waiting for external service restoration would prevent the shift team from completing an assigned investigation. Local history belongs in the operating requirements rather than an optional contingency. I'd also examine the established process queries. A plant may depend on a specific retrieval function that engineers invoke throughout the day with behavior they understand and results they can interpret against the equipment's operating conditions. Replacing that function involves more than matching its name. The alternative must support the same practical investigation without requiring the engineer to assemble several unfamiliar requests while production waits for a diagnosis or an instruction to continue. Operator workflows deserve the same attention. An experienced operator might move from an alarm to the relevant equipment trend through an established sequence, bringing the needed measurements together without searching through a separate reporting environment for each one. Familiarity alone doesn't justify keeping software indefinitely, but a workflow that reliably supports the operator's task carries replacement requirements, including usable access during a shift, and training that prepares people to perform that task under pressure. A supported connection to older equipment can remain a sound choice when it handles the installed source reliably, and the supplier continues to support the versions the plant actually runs. Older doesn't automatically mean unsupported. Check the maintenance status, available expertise, and documented compatibility of the interface. Because replacing a working industrial connection with a custom integration can transfer responsibility to your own engineering team. That transfer may still make sense. It needs an explicit benefit and a team prepared to maintain the replacement, rather than an assumption that removing an older connector automatically reduces the effort needed to keep collection working. Existing validated users raise another set of obligations where the historian forms part of an approved process. Changing its role may require the organization to demonstrate that the replacement continues to meet the intended use under its applicable procedures. Validation belongs to the particular use, not a product category. Calling a destination modern or managed doesn't establish that its configured application satisfies the plan's requirements, and an existing approval doesn't automatically extend to a different implementation. The replacement effort can exceed the data transfer effort. People may need to revise control procedures, repeat approved checks, document change behavior, and obtain authorization before the new application can take over an established production responsibility. Consider an equipment diagnostic tool that retrieves historian records through a supplier supported interface, where changing the source would require changes to the tool that your team can't authorize or maintain. That dependency introduces a separate replacement requirement. You may need a supported application update, supplier involvement, and renewed acceptance before removing the historian, even if another platform already stores every measurement. The diagnostic tool normally requests. I'd distinguish that constraint from simple resistance to change. A named application with a documented production purpose creates work someone must complete. While a vague preference for the existing system gives you nothing concrete to assess, age still belongs in the review. Unsupported components, unavailable expertise, or an approaching support deadline can change the decision, but the purchase date alone doesn't tell you whether the installed capability remains fit for its assigned job. Retain it against explicit requirements with an owner and a review point that makes keeping the historian an accountable operating decision rather than postponing a decision indefinitely because the system continues to run and nobody wants to disturb it. Keeping it doesn't assign every future workload to it. Once it's plant facing responsibilities are justified, we can examine the work for which a cloud platform earns its place without forcing the local historian to disappear. First, suppose our hypothetical mixing disturbance starts appearing at several sites, and the process team needs to establish whether those events share a cause or merely resemble one another. The question now reaches beyond temperature history. Investigators might compare the heating behavior with quality test results, maintenance work on the heating equipment, and the production orders that identify which products each plant ran during the affected periods. Each connection gives the team another explanation to test. Perhaps the pattern follows a product rather than a vessel, or appears after a particular maintenance activity across equipment that otherwise operates under different conditions at different sites. That calls for a shared analytical workspace. A cloud platform can give the investigation team access to those prepared data sets together, so they can examine the relationships without moving between separate plant systems. For every new question, the benefit comes from working across those records. A process Time series engine supports searching the measurement history, while the wider analytical environment can connect selected results to quality maintenance and order data through the access methods the design supports, I'd organize access around the investigation, not around departmental boundaries. A process engineer and a maintenance specialist can then work from the same prepared evidence, while each contributes knowledge that the other couldn't infer from measurements alone. Shared access also changes the demand placed on production sources. Once the required history reaches an analytical environment, several teams can investigate that retained data set without each launching another extraction against the plant historian whenever they revise a question. That separation helps when analytical interest grows, a quality study and equipment review and a process development project can use the available analytical copy, rather than turning every new project into another direct request to the production archive. It also gives researchers a stable basis for experiments. They can select a defined data set for a study and repeat their work against it. Instead of retrieving a changing selection from production sources each time they adjust their analysis, an engineer comparing heating behavior across years of production may need processing that differs sharply from the resources required to retrieve one batch, particularly when the study builds intermediate data sets for repeated testing. Model development brings that demand into clearer focus. Training and evaluating several candidate models can involve reading the same historical periods many times, so the team needs an agreed computing budget before treating experimentation as an unrestricted background activity. I'd find a defined study with a stopping point, specify the production question, the history included, and the resources available, then review the findings before approving another round of processing that might or might not improve the model. Cloud resources can support that bounded way of working where the chosen service permits IT. Teams can provision resources for a study and reduce them afterward, provided someone actually schedules that reduction and checks what remains active. A cloud centered design can also fit a new production installation if the intended users mainly need remote analysis, and the equipment already supports an approved edge to cloud route, the architecture can start with those needs rather than reproduce an existing historian installation by default. Local continuity still needs explicit coverage. The design must support whatever work the plant needs during disconnection, whether that's a limited local investigation capability or a broader process history service. Before cloud access becomes the normal route for everyone else, that requirement doesn't dictate one storage product. It tells the engineering team what must remain possible locally, leaving them free to choose an implementation that meets the requirement without assuming every plant needs an identical arrangement. So I wouldn't treat cloud adoption as a retirement instruction. It can create a shared place for cross plant work, while production retains the capabilities it still needs, with each environment carrying a responsibility that the organization has deliberately assigned. If both hold process history, we need an arrangement that combines their strengths without leaving the quality engineer facing two apparently authoritative accounts of the same batch. The practical choice is where the local and cloud paths separate. That separation determines whether cloud analysis receives the historians retained record. A parallel feed from collection or measurements from a path designed primarily around cloud delivery. Start with a historian first arrangement. The local collection system feeds the historian as usual, and an export component retrieves selected records from that historian for the analytical environment. Rather than creating another direct connection to the equipment, the export might include temperature and related heating measurements for the vessels under study, while other process history stays local because nobody has assigned it a cloud workload. This keeps the cloud path downstream of the archive. Any measurement selection already applied by the historian becomes part of what the export can supply. So cloud analysts receive retained history rather than an independent account of everything collection received. The dependency runs in one direction. Cloud transfer can pause without necessarily interrupting local archiving, but a problem Retrieving records from the historian can delay cloud analysis even when the cloud environment itself remains fully operational. That arrangement also duplicates some processing. The historian accepts and organizes records. Then the export and cloud services perform their own preparation, which may be justified but still needs capacity and someone responsible for the transfer. One approved collection component obtains readings from the production source, then distributes those readings toward both a local history service and a cloud destination through separately managed delivery paths. Collection becomes shared while delivery needs separation. If the cloud destination stops accepting records, it's blocked. Output shouldn't prevent the local output from continuing, provided the chosen component supports that behavior and has been configured accordingly. Shared collection also creates a shared failure point. A fault in that component can affect both destinations, so having two archives doesn't by itself give the plant two independent ways of obtaining measurements from the vessel. Filtering deserves particular attention after the split. If the local path applies, one measurement selection rule and the cloud path applies another. Both can operate as configured while retaining different accounts of the readings that entered the shared collector. That difference can be intentional. The local service might retain history for process review, while the cloud path retains a different selection for analysis. But the split needs an explicit purpose rather than two teams independently choosing defaults. Duplicated calculations can create a less obvious divergence if both paths calculate a heating result separately. Each result depends on its own available records and processing behavior, even though the original readings entered through the same component. The edge collects and buffers records for cloud delivery, while the local history and access functions are designed around the specific work, production must continue performing at the plant. Local coverage then becomes a bounded design decision for the vessel. Production might require access to the active mixing cycle and recently completed cycles, rather than every retained period available to the remote analytical team through the cloud environment. That choice needs a working local access path. A buffer alone doesn't meet the requirement, so the proposed edge components must provide the retrieval and viewing functions that the plant actually intends to use during disconnection. Failure handling now carries more of the design. The edge must distinguish records waiting for cloud delivery from records needed for local access. Because successfully sending a record onward doesn't necessarily mean production has finished using its local copy. Historian first adds an archive export dependency. Fan out shares collection while separating delivery and cloud centered design puts more responsibility into the edge functions that preserve the plant's required local capability. I'd choose the split before choosing the remaining services for each proposed arrangement. Identify where processing repeats, where filtering can differ, and which failures can affect one destination without affecting the other under the intended configuration. Once two environments hold a temperature record, we need to decide which version people should use for a particular purpose, especially when the copies differ or one arrives later. Authority needs a written agreement, not an assumption about location. I'd call that agreement a data contract. a maintained set of rules stating what each environment publishes, what consumers can rely on and who resolves disagreements. Assign authority by record type and intended use. The historian might own retained process observations, while an approved quality application owns the release disposition and a cloud dataset owns a published analytical result calculated from those observations. One platform needn't own everything. What matters is that someone reviewing a disputed result can identify which record governs that particular decision, rather than treating whichever copy loads first as the accepted account. The contract should name those boundaries explicitly. For example, publishing a batch calculation in the cloud shouldn't grant that dataset authority to change the batch disposition unless the organization's approved process assigns it that responsibility. Definitions belong in the same agreement. Specify the identifiers each publisher guarantees, how receiving systems interpret units and quality, and which timestamp field governs each calculation with references to the approved calculation rules rather than informal descriptions. 01:18:01 Speaker: Give consumers a way to detect incompatible changes. If a publisher changes a quality mapping or calculation rule, the data set should expose the applicable contract version so applications can reject unsupported input instead of silently accepting changed meaning that turns a schema into an operating promise. A field can remain numeric and keep its familiar name while its interpretation changes, so checking that a table still loads won't establish that the contract still holds. Suppose an authorized reviewer corrects a source record after the cloud has already published a batch result, the contract must state how that change reaches every affected accepted data set. Defined the scope of that propagation. Identify which derived results require recalculation, whether published reports need a revised status, and how consumers learn that a previously retrieved answer no longer represents the accepted evidence. Correction delivery needs acknowledgement and a deadline until the receiving system confirms that it applied the change. Its affected results should carry the status required by the contract, rather than appearing fully reconciled by default, deletion needs a different kind of message. If approved, disposal removes records from the authoritative source. An incremental reader that retrieves only existing records may never learn that its analytical copy also requires action under the agreed policy. State what deletion means for each data set. The contract might require deleting corresponding observations, retaining an approved derived result, or suspending disposal under a documented hold. But those outcomes need explicit authorization and a record of completion. Compare corresponding source periods and record identities, then investigate missing changes, conflicting revisions, and unexpected retained records through an assigned resolution process rather than an unowned exception report. Set a permitted reconciliation delay for each use. A research data set might tolerate a longer correction cycle than a data set supporting an active quality review, so the contract should state when each copy becomes too stale for its assigned purpose. Version the transformations as well as the contract. Each published result needs a link to the transformation version that produced it, allowing the team to distinguish a revised calculation from a revised input when an answer changes. Keep earlier published versions distinguishable where policy requires them. Rebuilding a data set with new logic shouldn't erase the ability to explain which logic supported an earlier decision, especially when that decision remains part of the retained batch record. Operations should approve process interpretation, quality should govern accepted evidence and disposition requirements, and platform engineering should own implementation and correction delivery within the agreed service commitments. Security owns a separate part of that agreement. Its designated owner should approve access boundaries and audit requirements and join decisions about correction or deletion privileges. So a data maintenance function doesn't acquire authority beyond its assigned purpose. Give disputes an escalation path with a decision deadline. If operations and quality disagree about an affected record, engineering needs an authorized interim status to apply while the responsible people resolve the issue, rather than choosing an interpretation themselves before proposing that either environment disappears, we need to discover which people and applications depend on these publishing promises, because a replacement must inherit the responsibilities that consumers actually use before planning the move. Find out who uses the current system. Start with configuration exports and access records. Then speak with the people doing the work because neither source alone will reveal every dependency the migration must address. Build an inventory that connects configuration to actual use for each tag or related group of tags. Identify the collector supplying it, the archive settings applied to it, and the people or applications retrieving its history. Record the settings, not just the component names include the installed collector configuration and the applicable archive rules. So the inventory describes the system you intend to replace, rather than a generic version of that product. Capture their definitions, execution schedules, inputs, and consumers, including calculations that feed other calculations before producing anything a person recognizes as a report or an equipment condition indicator. Follow those connections until you reach a working task. A calculated value with an obscure name may supply a familiar batch report, so marking it unused because nobody recognizes the name would miss its actual purpose. Then list the reports and connected tools. Ask who runs each one when they need it and what they do with the result. Rather than treating every configured report as equally active or equally necessary. In our hypothetical plant, a planner might refresh a spreadsheet extract before the morning meeting while maintenance runs a small script only when investigating a recurring heater fault. Neither use necessarily appears in the central application catalog. Ask people to demonstrate how they obtain their history, including saved files, desktop shortcuts, and scheduled tasks that someone outside their team might never know exist. Spreadsheets deserve inspection rather than an automatic replacement order. Check whether each workbook retrieves fresh history, reads an exported file, or depends on another workbook. Then identify which parts support a decision and which merely format the result. Maintenance scripts can hide behind ordinary support routines. Ask the technicians which commands or utilities they run during fault finding and record the historian access. Those tools require alongside the equipment investigations they support. You need to watch someone actually review a batch report, not just read the procedure. The written steps might say one thing, but the real workflow often includes an extra step nobody documented, like pulling up the previous successful run before signing off on a difficult product. That comparison becomes part of the working routine, even if nobody calls it an application. Same goes for operator habits. A shift operator might routinely pull the last successful run before starting a tough product, and that comparison becomes part of their working routine. Even though nobody ever wrote it down as an application, you have to see it happen to know it exists. Don't limit your discovery to one convenient shift. Talk to the people who handle the less common products, the unusual maintenance work, and the occasional quality reviews. A dependency can stay completely invisible during a normal production day, and only show up when something weird happens. When you find a dependency, classify what it actually means for production. What happens if that data source goes dark? How long can the affected task wait before things break? Is there an approved workaround that lets people keep going without the historian connection. Evidence requirements need their own classification. Track. A tool someone uses for exploratory engineering work, might tolerate a completely different replacement process than a report that serves as approved evidence for release decisions. Both might pull the same tags over the same time period, but the stakes are different. Let the people who actually own those tasks confirm your classifications, operations, and quality teams should tell you what happens if their data disappears and who needs to approve any changes. Engineering should document the technical dependencies required to deliver each capability, but they shouldn't guess at the business importance. That's the user's job. Take that inventory and pick a bounded pilot. Choose one specific task with a manageable set of tags, calculations, and users. You want to observe real usage without making your first experiment responsible for actually releasing product to customers. Write acceptance criteria around that task before you start anything. State clearly what the pilot must let its users accomplish, which reports or tools it needs to support, and who has the authority to accept the result or reject it. Assign owners for both the delivery work and the operating decision. Name the engineer who handles implementation. The user who exercises the task during the pilot, and the person who can pull the plug when an agreed limit gets reached. Rollback needs a practical set of instructions, not just permission to retreat. Identify exactly how users return to the existing workflow. Who restores any changed connections and which conditions trigger that return, rather than another attempt to fix the pilot while production is running. Now we have a defined first move with scope and responsibilities. Agreed. The next job is moving the required history while keeping the collection that serves today's production free from avoidable disruption. Run the historical transfer as a separate job from ongoing ingestion. Backfill handles the earlier archive while the ongoing path carries production records arriving during the movie. That way, neither job waits for the other to finish. Pick an explicit time boundary before either job starts. Record a cutoff in the sources time reference, then assign everything before that cutoff to backfill and everything newer to the ongoing transfer. Leave a controlled overlap around that boundary When both jobs retrieve an agreed interval, the transfer gets a chance to catch records near the handoff without depending on two independently scheduled requests. Meeting perfectly. Document which job covers the exact cut off instant. The overlap also needs to account for records that arrive late at the source, using a bounded reread period that fits the source's behavior, rather than assuming older intervals never change. A cut off alone won't catch every delayed source update. If the source offers a supported change feed, use it where it makes sense. Otherwise, define how recently transferred intervals get revisited while the migration is still active. Give backfill a lower priority than production retrieval. Years of historical reads can put sustained demand on the source, especially when several workers simultaneously request dense measurement streams from the same archive. Storage start with limited concurrency and bounded requests. Agree with the historian administrator on which source measurements indicate excessive load, then reduce or pause extraction when ordinary retrieval starts approaching the plant's agreed response limits. During production control both request size and request frequency. A short pause between enormous reads can still produce disruptive bursts. So adjust the time span, tag count, and number of simultaneous requests based on what you actually observe from the source. Divide the historical scope into independently tracked portions. A portion might cover a defined group of tags over a bounded period, with its size chosen to keep requests manageable and retries practical for that particular archive. Don't assume equal time spans contain equal amounts of work. A period with frequent process changes can return far more records than a quiet period, so let the transfer split an oversized portion into smaller requests. Keep a transfer ledger for those portions. Record whether each portion is waiting for extraction, sitting in temporary storage, or finished writing to the destination. Don't rely on a single percentage that hides unfinished work. Make each checkpoint specific enough to resume safely along with the portions boundaries. Keep any paging position the source interface requires. Handle expired continuation tokens through a documented restart procedure rather than just skipping the remaining response. A failed right can leave the destination in an uncertain state before retrying. Figure out whether the destination actually committed the previous attempt, then use the agreed record identity rules to prevent repeated delivery from creating extra accepted copies. The overlap deserves its own duplicate detection. Check records received through both jobs should resolve to the same destination observation when their source identity matches, but conflicting payloads should stay available for investigation rather than getting discarded track conflicts separately from straightforward, repeated deliveries. That distinction lets the transfer keep handling ordinary retries without mistaking a changed source value, an identity collision or a conversion error for harmless duplication during backfill. Preserve the source representation throughout the transfer. Make sure temporary files, message formats and destination columns can carry the returned numerical precision, timestamp, precision, and original quality code without rounding or narrowing them along the way. Export tools can introduce changes before loading even starts. A text export might format numbers using local conventions or truncate fractional seconds. So configure serialization explicitly and keep the settings used for each extraction job. Keep transfer records with the migrations. Retain documentation. Include the source endpoint, requested portions, extraction options, software version start and completion times. Retry history and destination references. Every delivered portion needs an identifiable account. Capture failures without recording secrets in those logs, the support team needs enough detail to repeat an unsuccessful request and find its partial output, but credentials and access tokens don't belong in the transfer evidence. Treat completion as an accounted for scope, not an empty queue. Every planned portion needs a confirmed outcome, every failed portion needs resolution, and any deliberately omitted history needs an explicit record rather than just disappearing from the progress total that finishes delivery. But it doesn't establish equivalence with the transferred periods and the extraction records available. We can now test whether the migrated history supports the same production decisions the source history did. Validation needs an expected answer that's independent of the destination. Select source periods that reviewers can inspect directly, then compare their retained observations with the migrated records before asking whether the final batch results agree. Matching totals won't establish that correspondence. Two data sets can contain the same number of records while holding different timestamps, missing different observations, or assigning different quality codes to measurements used in the release calculation. Compare individual records within each selected period, match their source identities, then check values, timestamps, and original quality codes against the extraction response. Any permitted numerical conversion needs assessment under an explicitly approved tolerance check. Unit definitions for the period under review. A destination value should either keep its applicable source unit or carry a documented conversion that reproduces the expected quantity without applying today's metadata to older records. Coverage needs a separate comparison. Identify the intervals the source archive actually contains, including its known gaps. Then establish whether the destination preserves that coverage rather than just matching the earliest and latest timestamps. Next, reproduce the batch calculation under controlled conditions. Give both implementations the same selected observations and approved calculation definition, so a mismatch can be investigated without wondering whether the inputs changed between requests. Inspect intermediate results rather than just the final answer. Compare the values assigned at the mixing boundaries, the intervals included in the calculation, and the duration accepted as usable evidence under the agreed quality rules. Interpolation needs to match at those boundaries. If one implementation estimates a starting temperature between neighboring observations require the other to use the same method and neighboring records before treating their calculated averages as comparable. Then challenge the calculation with deliberately awkward periods. Choose cases from available history where possible, and use clearly identified test records. When the archive doesn't contain a condition, the validation needs to exercise a gap should produce the approved uncertainty. Remove observations from a controlled test interval and verify that both implementations report the expected missing coverage, rather than letting one invent continuity that the other rejects. A counter reset needs a known expected total use a sequence that crosses an identified reset, then confirm that the calculation distinguishes the reset from ordinary accumulation and does not quietly convert it into negative production. Exercise the repeated hour during a daylight saving change. Select observations with unambiguous source times and verify their order, batch assignment and calculated durations, including how the review interface distinguishes two readings with the same local clock time. Late records need a before and after comparison. Run the batch calculation before introducing a delayed observation. Introduce it through the intended path, then check whether the revised result and its status. Follow the agreed rule. Corrected values require a similar paired test. Apply an authorized test correction and verify which observation version each calculation uses, while confirming that a reviewer can still distinguish the earlier evidence from the accepted revision. Now return to the batch that opened this discussion. We haven't established why its reports disagree, so validation must follow the actual contributing records rather than choose a convenient explanation that favors either platform. Start with the cloud reports flagged excursion. Identify the exact observation or calculated interval behind that flag, then request the corresponding source records and the historian retrieval result for the same measurement period and batch window. That comparison narrows the investigation if the above limit observation shows up in both data sets. Check whether the local trend request excluded it reconstructed a different path, or used a window that placed it outside the reviewed step. If it only exists in one data set, trace where it came from. Figure out whether the difference happened during collection, archive selection, extraction, or later processing. Use the retained records and configurations to do this. Don't guess based on how either result looks. Matching records can still produce different batch judgments. Follow each calculation through its boundary values, quality exclusions, and interpolation until the first difference leads back to a specific rule or a specific input record. Write that finding as a reproducible test. Another engineer should be able to pull the identified evidence, apply the documented rule, and get the same explanation without relying on the original investigators memory. 01:33:46 Speaker: One resolve batch won't settle every dependency, but a set of explained disagreements and repeatable passing cases gives parallel operation a purpose. Which raises the next question when has it produced enough evidence to stop? Comparing enough evidence means a capability can pass its own cutover gate. It doesn't mean every historian dependency must move together, or that finishing validation gives the project permission to switch off the entire existing service at once. Move a selected analytical report before moving a plant facing dependency. For example, the process team could adopt its new cross plant comparison. While operators keep using their established local history tools and quality retains its approved batch review workflow, make that change visible to its users. Update the published report location, remove obsolete shortcuts where appropriate, and tell the affected team which result to use from the agreed date. Don't leave two routine entry points indefinitely. Keep parallel operation tied to a decision. Each capability needs named acceptance gates and an end condition that states when comparison stops. When the new service takes responsibility and what unresolved findings would prevent that transfer. A gate needs evidence and an authorized decision the report owner should accept the working result. Support should accept the operating procedure and any outstanding exceptions should get an explicit disposition before users move to the new route. Elapsed time alone isn't an acceptance gate. Running both systems for several weeks tells you little. If nobody exercises the less frequent retrieval task or observes how the replacement behaves when its normal connection goes down. Set a review date with a consequence at that review. Either approve the capability, extend comparison for a named unresolved condition, or return it to the existing workflow. While the team addresses the failed requirement, rehearsal turns those conditions into something people can act on within an approved test window. Interrupt the relevant connection and ask the intended users to perform their assigned work. Don't let the project engineers demonstrate a workaround. Include the people who will answer the support call they need to recognize the interruption, follow the recovery procedure and establish when users can rely on the service again without. Depending on an engineer who won't remain on call. Recovery must finish with usable access, not a reassuring status indicator. Have a user retrieve the affected production period after service returns, then confirm that the support team can account for any interval still awaiting recovery or review. Historical retrieval needs a timed rehearsal as well. Choose an older period within the retained scope and have ordinary support staff obtain it through the proposed procedure, including any approvals required before the requesting engineer receives access. Rollback deserves a full exercise, not a tabletop promise. Switch a selected report back to its previous source. Verify that users can reach it and confirm who communicates the temporary change, so nobody keeps using the withdrawn route. Check the rollback window before each cutover. If a later change removes an interface or expires a necessary license, the earlier recovery instructions may no longer work, even though the project plan still describes rollback as available. Decommissioning comes after those capability decisions under a separate approval. Before retiring, the Historian Service review remaining access needs with the people who investigate older production, including users whose work happens too rarely to appear during routine monitoring. Retention must survive the shutdown as an operational commitment. Confirm who will preserve the required history, who can authorize its retrieval, and how that obligation remains funded. After the migration project closes and its temporary resources disappear. Support needs a completed handover before retirement. The receiving team should have working access, current procedures, escalation contacts and an agreed service boundary. That leaves no ambiguity about who responds when an accepted capability fails during production. Every dependent application needs an explicit outcome. Its owner must confirm that it now works through the replacement, has an approved alternative, or can retire. Don't let silence count as evidence that the dependency disappeared. Schedule the shutdown with a stop condition. If the final review uncovers an unresolved access requirement or an application still calling the old interface, postpone retirement of the affected component instead of accepting an unexplained production risk to meet the date that leaves the new environment responsible for work, it has demonstrated with the transition controlled and the history dependable for its assigned uses. The next question concerns which new analytical tasks that evidence can support without weakening the distinction between measurements and interpretation. Training a model introduces another consumer of the process history. Suppose we want to predict an approaching temperature disturbance in our hypothetical vessel, giving an engineer time to examine the heating behavior before the next excursion. The training data set now defines what the model can learn. Its documentation needs to describe the sampling pattern, filtering settings, quality exclusions, and retained periods. Because those choices determine which examples of normal and abnormal behavior reach training. Filtering can change the apparent frequency of disturbances if one vessel's archive preserves small peaks while another suppresses them. A model may learn a difference in recording practice rather than a difference in heating behavior. I checked that possibility before interpreting a plant comparison. Include collection and archive configuration in the training data sets provenance. Then assess whether examples gathered under different settings should enter the same experiment or require separate treatment. Quality handling can also shape the training population. Excluding every uncertain period might leave the model with mostly steady operation, even though difficult operating conditions are precisely where engineers hope the prediction will help them. Retained history sets another boundary on the experiment if detailed observations survive only for recent products. Performance on those products won't establish how the model handles products represented solely by older summaries or absent from training. Make those limits part of the model's intended use record which equipment, products, operating conditions, and recording configurations the evaluation covers, and identify situations where the available evidence doesn't support trusting its predictions without further testing. The training columns also need clear identities an observed temperature and estimated temperature between observations and a predicted future temperature can share the same unit while describing different kinds of information about the process. Keep those categories separate. In storage and retrieval, a reconstructed value should identify its reconstruction method, while a forecast should identify when it was issued, which future period it describes and which model produced it. Other model outputs need equally explicit labels. A disturbance score or an estimated equipment condition should carry its model version and calculation time, so downstream applications don't mistake an analytical judgment for a physical observation that separation prevents an easy mistake during later training. If a data set quietly mixes predictions back into the temperature observations, a later model can end up learning from earlier models assumptions without the experiment. Acknowledging that dependency context helps define which prediction we're asking for. A digital twin can represent the vessel and relevant process relationships, giving an application a maintained connection between the measurement stream and the equipment behavior it describes. A knowledge graph can express those relationships explicitly. It can connect the vessel's temperature signal to its heating equipment, the executed mixing step, and the material involved, allowing retrieval to follow those connections when assembling evidence. Neither representation supplies missing process knowledge by itself. The organization must define the relationships and their meaning, while the chosen technology stores and exposes that model through interfaces the analytical application can actually use. I'd use that context to narrow an investigation. A request about unstable heating should retrieve measurements associated with the relevant equipment and executed process activity, rather than gather every temperature stream whose name happens to sound related. A co-pilot style assistant can make that retrieval easier to request. An engineer could ask which comparable mixing runs showed unusual heating behavior, with the assistant translating that request into a permitted evidence retrieval operation. Rather than inventing a technical explanation, give the assistant governed query tools. Those tools should restrict accessible data sets and expose approved calculations with the caller's permissions enforced by the underlying services. Rather than relying on instructions written in the assistant's prompt, the response needs evidence the engineer can follow retain references to the queried records, applicable batch context, calculation version, and retrieval time so the engineer can inspect the basis of a claim without trusting the wording alone. Uncertainty belongs in the answer, not outside it. If the retrieved history contains reconstructed intervals or lacks comparable runs, the assistant should state that limitation and avoid presenting an unsupported conclusion as an established process. Finding fluent answers don't settle the architecture choice whether the evidence comes from a historian, a cloud platform, or both. The next step is testing whether the complete system supports the production decision. People must make right the acceptance test as permission to perform a particular job. Instead of asking whether the cloud can replace the historian, ask whether the proposed arrangement gives an authorised person enough evidence to make the required production decision. Give each decision its own acceptance statement for batch release. The quality owner defines what evidence permits a disposition for fault investigation. The responsible engineer defines what the record must establish about the disturbance. Fleet analysis needs a different statement. Its owner should specify which comparison the evidence must support, while local troubleshooting needs an explicit statement of what the shift team must accomplish without relying on external access. Make those statements specific enough to reject a proposal. Supports manufacturing analytics. Doesn't tell an evaluator when to fail a system, whereas an identified decision with required evidence creates a boundary that both implementation teams can work against, then assign each requirement to the component responsible for meeting it. If a proposal fails, you need to distinguish a collection failure from a storage limitation, an incorrect query, missing production context, or an operating responsibility. Nobody accepted. Take the requirement to detect the vessels. Brief excursion collection must capture the relevant change, and storage must preserve the required observation. Neither responsibility can pass merely because a downstream report displays a plausible temperature history. The calculation belongs to query logic. Assigning the excursion to the correct mixing step belongs to context, while deciding who resolves an unexplained discrepancy belongs to operating ownership with an authorized person responsible for the outcome. A requirement can cross those boundaries. In that case, give each part an owner and test the completed chain. Because separately functioning components don't establish that the reviewer receives usable evidence for the decision. This changes how a failed test guides investment. If the only unresolved requirement concerns batch assignment, replacing the archive might leave the actual failure untouched while correcting the execution context connection could address it directly with less disruption. Use the results to choose keep, extend or replace. Keep the current capability where it meets the assigned requirements at an acceptable operating cost and identify precisely which responsibilities the organization intends to continue funding. Extend where an added capability closes a demonstrated gap. That could mean assigning broader analysis elsewhere while retaining the existing plant service. Provided the combined arrangement passes, its acceptance statements and its full operating cost receives approval. Replace where the alternative passes the required coverage. That choice needs an operating cost comparison for the accepted scope, including responsibilities that transfer to your own team rather than disappear when the old product leaves. Don't reward a proposal for omitting an expensive requirement if the plant still needs that capability, either include its delivery in the proposed arrangement or record that the proposal remains incomplete for the replacement decision. Now put the opening batch through that test. In this hypothetical case, assume the investigation confirms an above limit observation during mixing, and require the accepted record to preserve that excursion with its applicable time and status. Preservation gives the reviewer an observation, not a disposition. The agreed calculation must then evaluate that observation within the correct executed step. Applying the approved rule rather than whichever default happens to produce a more convenient result. The evidence must explain the result without guesswork. Starting from the reported excursion, the reviewer should reach the contributing observation and the applicable calculation with enough traceability to establish why that evidence belongs to this batch agreement doesn't necessarily mean releasing the material. A parsing architecture could support a hold or rejection just as clearly, because its job is to support the authorized decision rather than produce the answer. Production prefers record the acceptance against that complete outcome, whether the evidence comes from the historian, the cloud platform or an agreed combination, the responsible reviewer must be able to act without an unexplained conflict between the accepted records. That makes the retirement decision narrower and more demanding. You're no longer declaring one technology superior. You're authorizing a particular arrangement to carry responsibilities that the plant has identified, tested and agreed to pay for. There's one obligation left before switching anything off. The person approving retirement needs evidence that every responsibility assigned to the retiring capability now has an accepted home, not merely confirmation that its records reached another destination. Copied data volume doesn't authorize retirement. Keep the capability until its replacement proves the work. Because a missing timestamp or an undefined query rule can still leave that batch decision unresolved after every file arrives. Which historian dependency would be hardest to replace in your plant, and what makes it difficult? Here's the thing. If you're going to write requirements that actually work, they need to be specific enough to kill a bad proposal. Saying supports manufacturing. Analytics doesn't tell anyone when to say no. What you need is a clear decision with the evidence required to back it up. That creates a boundary both implementation teams can work against. Once you have that, assign each requirement to the component that has to meet it. When a proposal fails, you need to know why. Was it a collection failure, a storage limitation, a bad query, missing production context, or an operating responsibility? Nobody agreed to own. You can't fix what you can't name. Take the requirement to detect a brief temperature excursion in a vessel collection has to capture that change, and storage has to preserve the observation. Neither side gets a pass just because a downstream report happens to show a temperature that looks close enough. The actual calculation lives in the query logic. Assigning that excursion to the right mixing step is a context problem. Deciding who resolves an unexplained discrepancy is operating ownership. An authorized person responsible for the outcome. These are different jobs. Now, a single requirement can cross those boundaries. When it does, give each part an owner and test the whole chain end to end. Individually functioning components don't mean the reviewer gets usable evidence for the decision they need to make. This changes how you invest after a failed test. Say the only unresolved requirement is about batch assignment. Replacing the entire archive might leave the real failure untouched, but correcting the execution context connection could fix it directly with way less disruption. Use those results to choose one of three paths keep, extend, or replace. Keep the current capability where it meets the assigned requirements at an acceptable operating cost. Be precise about which responsibilities the organization intends to keep funding. Extend where a new capability closes a demonstrated gap. Maybe you assign broader analysis somewhere else while keeping the existing plant service that works as long as the combined setup passes its acceptance tests and the full operating cost gets approved, replace where the alternative covers everything required. That choice needs an honest operating cost comparison for the accepted scope. And remember, responsibilities that transfer to your own team don't disappear just because the old product leaves. Don't reward a proposal for skipping an expensive requirement. If the plant still needs that capability, either include its delivery in the proposed arrangement or mark the proposal as incomplete for the replacement decision. Let's run the opening batch through that test. In this hypothetical case, assume the investigation confirms an above limit observation. During mixing, the accepted record has to preserve that excursion with its applicable time and status. Preservation gives the reviewer an observation, not a disposition. The agreed calculation then has to evaluate that observation within the correct executed step. Applying the approved rule, not whichever default happens to produce a more convenient result. The evidence has to explain the result without guesswork. Starting from the reported excursion, the reviewer should reach the contributing observation and the applicable calculation. There needs to be enough traceability to establish why that evidence belongs to this batch. Agreement doesn't automatically mean releasing the material. A parsing architecture could support a hold or rejection just as clearly. Its job is to support the authorized decision, not produce the answer. Production prefers record the acceptance against that complete outcome, whether the evidence comes from the historian, the cloud platform or an agreed combination. The responsible reviewer has to be able to act without an unexplained conflict between the accepted records. That makes the retirement decision narrower and more demanding. You're no longer declaring one technology superior. You're authorizing a specific arrangement to carry responsibilities the plant has identified, tested, and agreed to pay for. There's one more obligation before switching anything off. The person approving retirement needs evidence that every responsibility assigned to the retiring capability now has an accepted home copy. Data volume doesn't count. Keep the capability until its replacement proves the work. A missing timestamp or an undefined query rule can still leave that batch decision unresolved after every file arrives. So here's my question for you. Which historian dependency would be hardest to replace in your plant and what makes it difficult?