In enterprise risk management, expert assessments are still the central method for identifying and evaluating risks. Typically, it is the people responsible for particular business units who, as part of a survey, are tasked with reporting to risk management which risks exist in their area, how high the probability of occurrence is, and what magnitude they would have if they occurred.
In the past, expert estimates were also used for risk aggregation, that is, to determine the company's overall risk as the sum of the individual risks. This approach was criticized as overtaxing the experts, however, and was therefore replaced, when the revised Auditing Standard 340 (IDW PS 340) of the Institute of Public Auditors in Germany came into force, with the recommendation to carry out a methodical risk aggregation using, for example, a Monte Carlo simulation.
It remains an open question why the criticism of expert estimates was not applied holistically and was instead limited to the risk aggregation step of the risk management process. Faulty risk identification and/or faulty risk evaluation inevitably mean that all these errors flow into the risk aggregation. The aggregation then remains unreliable as well, regardless of which method is used to carry it out.
The quality of expert estimates therefore remains a critical question for enterprise risk management, one on which the validity of its statements stands or falls. It determines whether ERM is merely a subordinate function of reporting, producing the mandatory reports as cheaply as possible while otherwise being disregarded, or whether its results can actually be used in controlling and corporate management.
Strictly speaking, it would be the task of ERM to demonstrate that the methods used produce correct results. Here, we take a step back, turn the question around, and show situations in which the methods may no longer work. To enable a company to critically assess for itself how much reliability it can expect from expert assessments gathered in risk management process surveys, we provide a list of early warning indicators for a possible overtaxing of experts and/or a lack of quality in their statements.
Indicator 1: Is the assessment made by only one expert?
Experts are not machines, they are people, individuals. They have different knowledge, different backgrounds, different training, and so on. It is therefore not surprising that when several people are given expert status and asked to assess the same matter, they generally arrive at very different results.
This pluralism of expert judgments is not limited to risk management. It can be observed in virtually any context, from assessing measures against the climate crisis to economic policy, social issues, education, infrastructure, immigration, and risk. Statements about how critical a given matter is often diverge from one another by several hundred percent. Expert assessments are, in effect, a random number generator for risk and system parameters.
In the risk management process, this diversity of opinion is often avoided because it has to be consolidated. A superficial solution to the problem is therefore to survey only a single expert directly. The pluralism has of course not disappeared, but it is no longer visible.
Indicator 2: Is the expert in a conflict of interest?
In the risk management process, middle managers are typically named as experts, providing information in a survey about the risks in their area of responsibility. The survey therefore takes place in a very specific and homogeneous setting. The experts' actual competencies in risk management are generally not addressed.
This represents an exceptional situation, in that managers are typically measured against success criteria such as revenue and profitability, not against the accuracy and completeness of their risk assessments. There is therefore a danger that the manager will present the information in a way that casts them in a favorable light and restricts their freedom of action little, or even expands it. What this means in substance is completely open. It can just as easily suggest a softening as an intensification of the risk assessment[1].
Many risk experts regard the risk inventory and risk assessment as more of an annoying obligation that they want to fulfill with as little effort as possible. This suggests that they forgo any material quarterly reassessment of the risks and instead fall in line with the majority opinion and the status quo. It is easier to confirm an established opinion than to establish a new one.
A third type of conflict of interest does not only exist within the expert, but also involves other parties within the company. If the manager is under political pressure or pressure to conform, and potentially in conflict with other managers, an adjusted risk assessment can be a low-cost solution to the problem for them.
Indicator 3: Are the potential risk losses very large?
As the size of a risk grows, so does the sensitivity of a risk expert's statements. Conflicts of interest become more acute.
This can go so far that a risk raises the question of whether the company's continued existence is at stake. Admitting to a risk that threatens the company's survival would be a delicate step for the expert, one that would put the executive board under pressure with respect to investors, customers, and employees. Risks that threaten a company's survival virtually never appear in published risk reports, and therefore virtually never appear in the audit-proof risk inventories kept on file either. Real-world enterprise risk management appears to operate in an intermediate zone, above a de minimis threshold and below a critical threshold beyond which things gradually become dangerous.
Regardless of conflicts of interest, the size of a potential valuation error generally also grows with the size of the risk. When a risk is very large, it is much easier to be wrong by a large amount in its assessment than when the matter is of a small order of magnitude.
Indicator 4: Is the expert restricted in the ways they can express their assessment?
To give business unit managers a simple risk inventory process and a pleasant "user experience," risk assessment is heavily standardized and simplified in many companies and systems. The flip side is that this leaves the expert with only a few options for describing the risk.
This affects, on one hand, the representation of occurrences, for example when only probabilities of occurrence are allowed as an assessment. This implies an error of unknown size for risks that do not occur only once per period, such as cyber risks, personnel risks, product risks, and many more.
On the impact side, three-point distributions (triangular, PERT, and multinomial distribution) are popular at many companies, because their parameters are, in principle, easy to explain to risk experts (minimum, most likely / mean value, maximum). The uniform distribution (parameterized with minimum and maximum) and a fixed loss upon occurrence (maximum) are also frequently allowed.
However, it is precisely extreme values like the minimum and maximum that pose a challenge for risk experts. Absolute limits of what is possible easily escape experience. Fortunately, the worst possible accident has not yet occurred in every case. But what value should the expert give? An error in choosing these parameters has serious consequences. In the further analyses, losses above the stated maximum will no longer occur. If they do occur anyway, the company is blind to their effects (see also Figure 1).
The focus on standardization and simplicity also brings out the "paradox of the risk expert." Does an expert need a small toolbox with tools that are simple to use? That would normally be a crutch for beginners. A true expert needs the right tool for the right application and knows how to use it[2].
Indicator 5: Is the risk fed by several sources of uncertainty whose effects have to be combined?
Enterprise risk management operates at a high altitude, taking a bird's-eye view of the company. Most companies apply a de minimis or reporting threshold for risks. Only once this threshold is exceeded is a risk recorded, assessed, and monitored.
As the size of a risk grows, however, so does its complexity. Complexity means that the risk is not determined by a single factor, but that many factors interact and, taken together, determine whether the risk occurs and how large its impact is.
Whether a new product will succeed in the market depends on the success of its development, the production launch, market access, how the broader economic conditions develop, the behavior of competitors, and so on. And each of these factors can in turn depend on other factors.
Assessing such a risk therefore requires the risk expert to aggregate the interplay of all the influences involved. If the expert were capable of doing that, they could really go on to aggregate the company's entire risk as well. For this task, one's own ability to combine many influences in one's head is easily overestimated. It seems more sensible to apply a methodical risk analysis, including a Monte Carlo simulation, already at the stage of assessing individual risks, just as the auditing standard prescribes for aggregating the risks.
Here, methodical risk analysis means carrying out a "disaggregation," or decomposition, of the risk, meaning that the influencing factors are identified and their interaction is described. Data are reviewed, and distributions for the influencing factors are selected and calibrated. The assumptions made are documented. Finally, the influences and their relationships are aggregated under these assumptions using Monte Carlo simulation, checked for plausibility, and validated against the data.
Figure 1: Methodically assessed risk and calibrated distributions
Once a risk has been methodically assessed, the question raised in Section 4 arises again: can the simulated distribution be represented in ERM? In Figure 1, the orange histogram shows the simulated risk. For this risk, the distributions most commonly used in industry were calibrated, namely the uniform, PERT, and triangular distributions[3][4].
The simulated composite risk shows a long, thin tail extending toward large losses. This is characteristic of risks in ERM, since exceptionally high losses within a single risk occur only through a "chain of unfortunate circumstances," that is, when several things go wrong at once. In most cases, the influencing factors offset one another to some degree, and the outcome lands in the middle of the loss scale.
Uniform, PERT, and triangular distributions are templates that are too rigid to adapt to this shape. They do match the minimum and maximum of the simulated losses, but they dramatize the losses while, on the other side, overestimating the opportunities.
This approach therefore meets with criticism in practice, since the business departments do not accept the dramatized losses (see Question 4). The risk experts respond to this (see Question 3) by narrowing the three-point distribution (blue line). They then match neither the minimum, nor the maximum, nor the most likely value, and cut the large losses out of the risk entirely. They adjust the representation of the risk to fit the template. This makes exactly the part of the risk disappear from the map that could most threaten the company, that would require countermeasures, that is essential for pricing insurance, and that therefore represents the actual core of the risk for ERM.
Figure 2: Transferring the risk assessment into ERM
The chart also shows an adaptable distribution that matches the risk exactly in both shape and range, without understating or overstating it, and that can be transferred into ERM with a single click[5].
Indicator 6: Does the risk assessment require a cross-functional perspective or the integration of different views and priorities?
The complexity of a risk assessment can also increase when the risk affects several stakeholders or runs into an unclear management environment (see Questions 2 and 5). Not every stakeholder experiences an uncertainty as a risk to the same degree. Depending on perspective and priority, an uncertainty is sometimes a major problem, sometimes an opportunity, and sometimes unimportant.
In this situation, the risk assessment therefore depends on who prevails in the discussion and in setting priorities, which trade-offs are made, and what position the risk expert takes within this field of tension. This is true all the more when only a single risk expert assesses the situation (see Question 1).
Indicator 7: Do we fully understand the situation? Or does it have effects that go beyond the obvious?
It is in the nature of things that one gains routine and experience in an activity by having done it often and seen a great deal. You know the environment and can respond to it.
But what happens when we are confronted with a new situation? Is our judgment then still just as reliable? Do we already fully understand today how artificial intelligence, measures against the climate crisis, geopolitical conflicts, the activities of competitors, and much more will affect our business?
Experts face the same challenge here, while also being under the pressure described in Questions 2 through 6. An expert assessment can mean just about anything in this situation.
How a risk analysis can be carried out under these conditions is illustrated, for example, by the studies of the Intergovernmental Panel on Climate Change (IPCC)[6]. At its core, this is about defining a methodology that uses the relevant data, identifies and calibrates the influencing factors, and describes and evaluates how they interact.
Indicator 8: Do the experts lack feedback, meaning do they receive no direct feedback on the accuracy of their predictions?
The topic of validation is not yet very widespread in enterprise risk management. Forecasts are produced but not checked. This creates the danger that the experts and the organization as a whole fail to learn from their mistakes and keep perpetuating incorrect forecasts.
The lack of feedback affects not only the assessment but also the inventory. It is not uncommon for companies to end up needing restructuring because of risks that were not even included in the risk inventory. The pandemic is an example that affected a great many companies.
An incident database, in which risks that have occurred are documented and analyzed, exists in fields where there is strong intrinsic motivation to analyze, understand, and manage risks, such as aviation. In enterprise risk management, it exists almost exclusively at companies that are regulatorily required to keep one, such as banks and insurers.
Conclusion
Gathering expert assessments through a survey seems like a solution for estimating and evaluating risks cheaply and quickly in a heterogeneous environment with little data. Their quality, however, is completely unclear.
Each of the early warning indicators discussed points to a recording and assessment error of unknown direction and size, affecting an unknown number of risks in the risk inventory. That is a lot of unknowns, and they accumulate.
If you have answered several of these questions with yes, the validity of the risk analyses is no longer assured, and no transparency is created. It will not even be clear whether the figures got the order of magnitude right overall. Errors can offset one another. But they can also compound.
Expert assessments are more of a dead end than a royal road for enterprise risk management. Pro forma reporting can be done under these conditions. Real risk management, hardly.
If expert assessments worked, there would be no need for receipts or tax returns. It would be enough for the innkeeper, the tradesman, and the business owner to count the till at the end of the day and, as experts, tell the tax office how much tax they owe.
Risk management needs methodical analysis and traceability based on data, disclosed assumptions, calibrations, analyses of relationships, error estimates, and validations. Only under these improved conditions does it become possible to see how large the company's overall risk actually is, where it comes from, what it costs, and where it is worth taking on and where it is not. Only then can enterprise risk management also be integrated into controlling and corporate management.
References and further reading:
Institute of Public Auditors in Germany, Hauptfachausschuss (HFA), IDW-PS-340
Bogner, Alexander; Beate Littig; Wolfgang Menz, Interviews mit Experten – Eine praxisorientierte Einführung, Springer (2014)
Höglinger, M.; B. Jann (Höglinger/Jann 2018): More is not always better: An experimental individual-level validation of the randomized response technique and the crosswise model. PLoS ONE 13(8): e0201770. (2018) https://doi.org/10.1371/journal.pone.0201770
Kaiser, Robert; Qualitative Experteninterviews – Konzeptionelle Grundlagen und praktische Durchführung, 2nd ed., Springer (2014)
Von dem Berge, Benjamin: Teilstandardisierte Experteninterviews, in: Tausendpfund, Markus (ed.): Fortgeschrittene Analyseverfahren in den Sozialwissenschaften, Springer (2020), pp. 275-300
[1] Höglinger/Jann 2018
[2] See also the film "Bessere Experteneinschätzungen im Enterprise Risk Management" ("Better Expert Assessments in Enterprise Risk Management"), https://youtu.be/uC92yrg64C0.
[3] The multinomial distribution was omitted, since it only knows point masses and cannot fit a continuous risk.
[4] The assessment and calibration were carried out with Risk Kit.
[5] The assessment was inserted into the Enterprise Risk Evaluator.
[6]https://www.ipcc.ch/reports/