How to Plan IT/OT Business Continuity at Manufacturing Sites

In most outages, a production line stops not because of the machines but because of MES, domain sign-in or an ERP dependency. The steps of a testable continuity plan that maps IT and OT dependencies together.
In this article7 sections

When ransomware locks up the office network, the production line usually stops not because the machines broke but because MES couldn’t print labels, a line PC couldn’t sign in to the domain or the shipping list didn’t come through from the ERP. That’s why a business continuity plan at a manufacturing site has to take into account not only the recovery of office systems but also production lines, PLC programs, MES and ERP integrations and machine suppliers’ remote support. A plan written without documenting IT and OT dependencies won’t work in the first real outage.

1. Identify critical products and processes

For manufacturing, the business impact analysis (BIA) is done by product or customer: Which part are you sending to which customer, with what stock level and what delivery frequency? These questions determine the maximum tolerable period of disruption (MTPD). For example, for a line with just-in-time (JIT) delivery, an outage of a few hours can stop the customer’s line.

Inviting only IT to the BIA workshop is the most common mistake. The people responsible for planning, shipping, quality and maintenance must be at the table, because the answer to “How many hours can the line run without MES?” isn’t in the server room; it’s with the shift supervisor. For each critical product, put these three pieces of information in writing:

  • The stock or buffer time on the customer’s side (how many hours of your outage can the customer’s line tolerate?).
  • How many shifts of shipments your own finished goods stock covers.
  • The extra shift or overtime capacity needed to make up the loss after the outage.

An RTO set without these three pieces of data is a guess. Nobody invests based on a guess.

2. The dependency map: IT and OT together

Layer Example component Question to ask
Enterprise systems ERP, email, EDI/customer portal Which system do order and shipping information come from?
Production management MES, traceability database, label server Can the line run without MES, and for how long?
Control systems PLC, HMI, SCADA, engineering workstation Is there an up-to-date backup of PLC programs, and who can restore it?
Infrastructure Network, identity (AD), virtualization, internet Can people sign in to line PCs without the identity system?
External parties Machine suppliers, maintenance companies, cloud services If the supplier can’t connect remotely, how is a fault fixed?

When drawing the map, look for hidden dependencies. Examples you may come across: label printers depending on a print server on the office network, the SCADA server validating its license over the internet, line PCs taking their time from the domain controller (if the clock drifts, authentication breaks too) and the MES database running on the same virtualization cluster as the ERP. None of these dependencies appears on a system list. They show up during an outage.

A practical method: do an “outage walk” for each critical line. Go to the head of the line with the shift supervisor and ask in turn: if we pull the network cable, what stops? If AD is unreachable, who can’t sign in? If the internet is cut, which screen shows an error? Record the answers in the table.

3. Set RTO and RPO by layer

RTO (recovery time objective) and RPO (acceptable data loss point) differ by layer in manufacturing. While a few hours of data loss in the ERP can be made up with manual entry, data loss in the traceability database can affect customer and legal obligations. For PLC programs, RPO should be defined as “the last approved change”, and a backup should be taken after every change.

Writing an RTO target is easy; meeting it isn’t. If you’ve written a two-hour RTO for an ERP database, you need to have measured how long restoring from backup really takes. Restoring the virtual machine, the database consistency check, bringing up application services and reconnecting integrations (EDI, MES interface) each take time. The recovery order matters too: trying to start the ERP before the identity infrastructure and DNS are up wastes hours.

4. Continuity strategies

  • Manual working: Printed work orders, offline label templates, manual traceability forms.
  • Stock buffer: Safety stock for critical customers.
  • Spare equipment and software: A ready image for critical line PCs, a spare HMI panel.
  • Network segregation: Segmentation that stops an incident on the office network spreading to production; production being able to run independently of the internet.
  • Supplier agreements: Commitments for on-site support and spare parts in an emergency.

The weakest link in a manual working strategy is the return. Who will enter the production, scrap and traceability records written on paper during the outage once the system is back, and in what order? If this step isn’t defined, data loss starts when the outage ends. That’s why manual forms matching the system fields one-to-one and a person being assigned responsibility for re-entry must be part of the plan.

With network segregation, the real question is how long production can run on its own when the office side is completely shut down. If the production network has no local authentication path, no local time source and no locally running MES components, segmentation stays on paper.

5. Test the plan

The decision flow is tried with a tabletop exercise, restore times with a technical test, and manual working methods with a functional exercise. For organizations within IATF 16949 scope, these tests are the evidence of the periodic effectiveness testing SI 3 expects; see the IATF cyber attack contingency plan.

An untested plan is an assumption. Keep the first technical test small: restore a single line PC from a ready image, load a single PLC program from the in-house backup into a test environment and record the times. When the measured times are compared with the RTO targets written in the BIA, the real gap becomes visible. At the end of each test, draw up an action list with findings, owners and closure dates.

Common gaps

  1. Backups of PLC and HMI programs held only by the machine supplier.
  2. Not knowing that line PCs depend on domain identity.
  3. Manual working instructions never having been tried.
  4. Unsupported operating systems on the production network with no compensating controls.
  5. The plan being written only by IT, without production and quality taking part.

One more item can be added to the list: the plan’s availability. In a ransomware incident, the plan on the file server gets encrypted too. A printed copy of the current plan and a version accessible offline are what the crisis team needs in the first hour.

Frequently asked questions

Who should back up PLC programs?

Responsibility should be given to the maintenance or automation team; backups should be kept in-house, with version information and controlled access. A backup held only by the supplier may be out of reach during an outage.

Should the production network be completely closed to the internet?

There’s no absolute rule; however, necessary connections should be limited to defined, controlled and monitored channels. The zones and conduits approach is a common framework for this design.

Can the business continuity plan and the IATF contingency plan be the same document?

Yes, they can be combined; it’s enough for the elements IATF expects (cyber attack, testing, customer notification, restart verification) to be clearly included in the plan.

To identify your critical processes and targets, take a look at our Business Continuity (BIA and BCP) service; for compliance gaps in the production environment, our IT/OT Compliance at Manufacturing Sites service. Technical projects such as OT network segmentation are carried out by Eretek Cybersecurity.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Compliance

We work with you on compliance with ISO 27001, IATF 16949, TISAX and similar standards, covering risk assessment, control design, documentation and audit preparation.

Knowledge Center

Other articles