Control Effectiveness Testing: Design vs Operating Effectiveness

How control effectiveness testing works in practice: the difference between design and operating effectiveness, inquiry, observation, inspection, reperformance and a sample test working paper.
Technical summary
Technology
ISO 19011:2018, ISO/IEC 27001:2022 clause 9.2, ITGC, SOC 2
Topic
Testing the design and operating effectiveness of controls
Level
Intermediate–advanced
Prerequisites
  • A written definition of the control to be tested (purpose, owner, frequency)
  • Familiarity with sample selection methods
In this article7 sections

A control existing doesn’t show it works; it working doesn’t show it addresses the right risk. That’s why internal auditors, ITGC assessors and SOC 2 auditors test two questions separately: is the control designed correctly, and did it operate as designed throughout the period? In the field the two are often confused; people say “the procedure is written, so the control exists” and the test ends there. Below we show how we carry out design and operating effectiveness tests, when each of the four basic test methods is enough, and a sample test working paper for an access control.

Design and operating effectiveness

Design effectiveness asks whether the control, when applied as defined, can bring the related risk down to an acceptable level. The elements examined: is the control tied to the right risk, does its frequency fit the speed of the risk, does the person performing it have sufficient authority and knowledge, is the data the control uses as input complete and accurate, is it defined what happens when an exception is found?

Operating effectiveness asks whether the control was applied as designed, consistently and by authorized people throughout the audit period. A single successful application doesn’t prove operating effectiveness. It must be shown that the control worked in all the samples selected from the period.

In IT control and compliance audits the order doesn’t change: design is tested first. If the design is deficient, the operating test is wasted effort, because a badly designed control working without error doesn’t reduce risk. Example: a control that closes leavers’ accounts every three months can be applied perfectly every quarter. But if a leaver’s account can stay open for three months, the problem is in the design, not the operation.

Comparison of test methods

ISO 19011:2018 and financial audit standards define similar methods for collecting evidence. The four basic methods used in control testing are compared below.

Method What is done? Level of assurance Typical use
Inquiry Interview with the control owner Low; insufficient on its own Understanding the control, planning the test
Observation Watching the control being performed Medium; covers only that moment Physical controls, point-in-time operations
Review Examining records, approvals and reports Medium–high Approvals, review records
Reperformance The auditor runs the control again themselves High Reconciliations, automated calculations

The methods are combined. A typical combination: inquiry to learn how the control works, inspection to verify evidence in the selected samples, reperformance on a critical sample. Observation is a valuable but narrow method. A control owner who knows they’re being watched may perform the control differently; also, observation says nothing about the rest of the period.

In inspection, the source of the evidence also sets the level of assurance. A report pulled directly from the system is more reliable than a manually compiled Excel table; this distinction is explained in detail in the screenshot vs system report comparison.

Design testing with a walkthrough

A walkthrough means following a single transaction from start to finish. The aim is to see with your own eyes that the control exists and is applied as described in the document. Steps:

  1. Read the control definition: Are the purpose, risk, owner, frequency, input, output and exception handling written down?
  2. Start the process with the control owner: Ask them to describe every step from the moment the control is triggered to its outcome.
  3. Follow a transaction: Pick a real example (for example the access request of an employee who started last month) and see the evidence of each step in the system.
  4. Question the reliability of the input: Where does the list or report the control uses come from, and is it complete?
  5. Assess whether it addresses the risk: If this control works as designed, does it prevent the related risk or detect it in time?

The fourth step is the one most often skipped. If the user list reviewed was prepared by hand, an account not on the list is never reviewed. If the input is incomplete, so is the control.

Operating testing across the period

The operating test uses sampling to show that a control whose design has been approved worked throughout the audit period. Defining the population, verifying its completeness and setting the sample size according to control frequency are explained in the sample selection in internal audit guide. After the samples are selected, predefined test attributes are checked for each one. Test attributes are derived from the control definition and applied the same way to every sample: “is there an approval”, “is the approval before the transaction”, “is the approver authorized”, and so on.

Mistakes you may come across in operating testing:

  • Letting the auditee choose the samples.
  • Setting test attributes after the samples have been reviewed.
  • Testing a control that changed during the period as a single control. If the control changed mid-period, it must be tested as two separate periods.
  • Brushing an exception aside as “one-off”. No conclusion is written until the cause of the exception has been investigated; how to word the finding is explained in the writing audit findings guide.

Sample test working paper

The working paper below was prepared for a quarterly user access review control. Values and record numbers are examples.

CONTROL      : ACC-04 Quarterly access review
RISK         : R-12 Unauthorized access to a critical application
DEFINITION   : ERP role assignments are reviewed by the relevant
               department manager within 15 working days after
               each quarter end; permissions requested for removal
               are removed by IT within 5 working days.
TYPE / FREQ  : Manual, detective / Quarterly
PERIOD       : 01.01.2025 – 31.12.2025   POPULATION: 4 reviews

DESIGN TEST (walkthrough, Q3 review)
  Input list pulled from ERP via system report?        Yes (report + parameters)
  Reviewer is a manager who knows the roles?           Yes
  Removal request and its execution traceable?         Yes (ticket)
  Result: Design effective

OPERATING TEST (all 4 quarters)
  Test attributes: A) completed on time   B) all users on the list
                   C) authorized approval D) removals within 5 working days
  Q1: A✓ B✓ C✓ D✓
  Q2: A✓ B✓ C✓ D✗  (2 removals done in 9 working days)
  Q3: A✓ B✓ C✓ D✓
  Q4: A✓ B✓ C✓ D✓
  Reperformance: Q4 list pulled from ERP again, user count matched.

RESULT       : 1 exception. Cause: leave period in the IT team, no queue tracking.
               Reported as finding AF-2025-07.
PREPARED BY  : [name]  DATE: [date]   REVIEWED BY: [name]

A good working paper is detailed enough that another auditor could reach the same conclusion with the same samples. The file names or repository links of the evidence are added to the working paper. Writing “reviewed, OK” and moving on isn’t a test. Which evidence is expected in an access review control is explained separately in the access review evidence guide.

Testing differences between automated and manual controls

Dimension Manual control Automated control
Source of error Human; forgetting, carelessness Configuration; a wrong or changed setting
Sample Several samples depending on frequency Usually verifying the configuration once
Main test method Inspection, reperformance Configuration review, test transaction
Additional controls it relies on The control owner’s competence Change management and privileged access

Accepting an automated control on a one-off test depends on showing the configuration didn’t change during the period. If the Active Directory password policy is being tested, the setting is reviewed once (the output of Get-ADDefaultDomainPasswordPolicy, and Get-ADFineGrainedPasswordPolicy -Filter * for fine-grained password policies). Then the privileged accounts that could change this setting during the period, and the change records, are checked. A part that can be skipped is fine-grained policies: the default policy looks solid, while a looser PSO tied to a privileged group never crosses anyone’s mind. If the change management control isn’t effective, broader testing is needed for the automated control; the configuration is re-verified at several points in the period.

Semi-automated controls, meaning controls where a person reviews a report produced by the system, are tested in two parts: that the report is complete and accurate (the automated part), and that the review was done throughout the period (the manual part). These hybrid controls are a frequent source of exceptions in practice. The report’s accuracy is assumed, not tested.

To assess the design and operating effectiveness of the control set independently, our IT security audit and control effectiveness work can be used, and to set up the access, change and operations controls themselves, our IT general controls (ITGC) work. To see access and configuration controls working in the live environment, define the scope of a technical cybersecurity audit.

Frequently asked questions

If the design test fails, is the operating test done?

Usually not. If the control isn’t designed to address the risk, it working without error throughout the period doesn’t reduce the risk either. The finding is reported as a design deficiency; after the control is redesigned, the operating test is done in the next period.

Can a control be tested by inquiry alone?

No. Inquiry is necessary to understand how the control works but doesn’t give assurance on its own; what a person describes may differ from what’s actually done. Always support inquiry with observation, inspection or reperformance.

Why is a single sample considered enough for automated controls?

An automated control works the same way every time with the same configuration. Once the configuration is verified and it can be shown that it didn’t change during the period, it can be concluded that the control worked consistently. That’s why testing automated controls relies on the effectiveness of change management and privileged access controls.

Is a walkthrough the same as a design test?

A walkthrough is the most common method of design testing, but it isn’t a design test on its own. A walkthrough follows a transaction from start to finish to show the control exists and is applied as described; a design test additionally assesses whether the control actually reduces the related risk.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Compliance

We work with you on compliance with ISO 27001, IATF 16949, TISAX and similar standards, covering risk assessment, control design, documentation and audit preparation.

Knowledge Center

Other articles