by Uwe Wehrspohn

In enterprise risk management, expert assessments are still the central method for identifying and evaluating risks. As a rule, it is those responsible for certain areas of the company who, within the framework of a survey, have the task of reporting to risk management which risks exist in their area, how high the probability of occurrence is and what their extent would be in the event of occurrence.

In the past, expert estimates were also used for risk aggregation, i.e. to determine the overall risk of the company as the sum of the individual risks. However, this approach was criticised as overtaxing the experts and was therefore replaced by the recommendation to carry out a methodical risk aggregation and to use a Monte Carlo simulation for this purpose, for example, when the Institute of Public Auditors' Auditing Standard 340 n.F. came into force[1].

It remains open why the criticism of the expert assessments was not holistic and limited to the risk aggregation step in the risk management process. Faulty risk identification and/or faulty risk assessment inevitably result in all errors being included in the risk aggregation. The aggregation then also remains unreliable, regardless of the method used to perform it.

The quality of expert estimates thus remains a critical issue for enterprise risk management, with which the validity of its statements stands or falls. It decides whether ERM is merely a sub-function of the reporting system that produces the obligatory reports as cheaply as possible, which are otherwise not considered further, or whether its results can be used in controlling and corporate management.

Actually, it would be the ERM's task to prove that the methods used deliver correct results. We start here one step before and turn the question around and show situations in which the methods may no longer work. In order to be able to make a critical self-assessment in the company as to what reliability can be expected from expert assessments in the surveys in the risk management process, we provide a list of early warning indicators for a possible overburdening and / or lack of quality in the expert statements.

Indicator 1: Does only one expert make the assessment?

Experts are not machines, but people, individuals. They have different knowledge, different backgrounds, different training, etc. It is therefore not surprising that when several people are given expert status and are asked to assess the same issue, they usually come to very different conclusions.

This pluralism of expert judgements is not limited to risk management, but can be observed in any context, from the assessment of measures against the climate crisis to economic policy, social issues, education, infrastructure, immigration and risks. Statements about the criticality of an issue often differ by several hundred percent. Expert assessments are virtually a random number generator for risk and system parameters. 

In the risk management process, this diversity of opinion is often shied away from because it needs to be consolidated. A superficial solution to the problem is therefore to ask only one expert directly. Pluralism has not disappeared then, of course, but it is no longer seen.

Indicator 2: Is the expert in a conflict of interest?

In the risk management process, middle management executives are usually appointed as experts who provide information in a survey about the risks in their area of responsibility. The survey thus takes place in a very specific and homogeneous milieu. The concrete competences of the experts in risk management are generally not addressed.

This is an exceptional situation in that managers are usually measured by success criteria such as turnover and profitability and not by the accuracy and completeness of their risk assessments. There is therefore a risk that the manager will prepare the information in such a way that it shows him in a favourable light and does little to restrict or even promote his freedom of action. What this means in terms of content is completely open. It may suggest a weakening as much as an intensification of the risk assessment[2].

Many risk experts see risk inventory and risk assessment as more than a burdensome duty that they want to fulfil as leanly as possible. This suggests materially foregoing a quarterly reassessment of risks and instead going with the majority opinion and the status quo. It is easier to confirm an established opinion than to establish a new one.

A third type of conflict of interest exists not only within the expert, but also affects other parties in the company. If the manager is under political or compliance pressure and may be in conflict with other managers, an adjusted risk assessment may be a cost-effective solution to the problem for them.

Indicator 3: Are the risk damages potentially very large?

With the size of a risk, the explosiveness of a risk expert's statements increases. Conflicts of interest are exacerbated.

This can go so far that a risk to the company's existence is under discussion. However, the admission of a going-concern risk would be a delicate step for the expert, which would put the board under pressure vis-à-vis investors, customers and employees. Risks that threaten the company's existence are virtually absent from the published risk reports of the companies and thus also from the risk inventories that are kept in an audit-proof manner. Real enterprise risk management seems to move in an intermediate area above a trivial limit and below a critical limit, above which it slowly becomes dangerous.

Irrespective of conflicts of interest, the size of the risk usually increases the size of a valuation error. If the risk is very large, it is much easier to err in the valuation by large amounts than if the issue is of a small magnitude.

Indicator 4: Is the expert restricted in his forms of expression?

In order to make risk inventory easy for the division managers and to provide a pleasant 'user experience', risk assessment is highly standardised and simplified in many companies and systems. The flip side of the coin is that this leaves the expert with few options to describe the risk.

On the one hand, this concerns the presentation of occurrences, if, for example, only probabilities of occurrence are allowed as a valuation. This implies an error of unknown magnitude for risks that do not occur only once in a period, such as cyber risks, personnel risks, product risks and many more.

On the impact side, three-point distributions (triangular, PERT and multinomial) are popular with many companies because their parameters are easy to explain to risk experts in principle (minimum, most likely / mean, maximum). Often, the uniform distribution (parameterised with minimum and maximum) and a fixed loss on occurrence (maximum) are also allowed.

However, it is precisely extreme values such as minimum and maximum that pose a challenge for risk experts. Absolute limits of the possible easily elude experience. Fortunately, the greatest possible accident has not yet occurred in all cases. But which value should the expert now specify? A mistake in the choice of these parameters has serious consequences. In further analyses, losses above the specified maximum will no longer occur. If they do occur, the company is blind to the effects (see also Figure 1).

The focus on standardisation and simplicity also highlights the 'paradox of the risk expert'. Does an expert need a small toolbox with easy-to-use tools? That would normally be an aid for beginners. A real expert needs the right tool for the right application and knows how to use it[3].

Indicator 5: Is the risk fed by several uncertainties whose effect must be combined?

Enterprise risk management has a high altitude and looks at the company from an eagle's eye view. For risks, there is a trivial or reporting limit in most companies. Only when this is exceeded is a risk recorded, assessed and monitored.

However, as the size of a risk increases, so does its complexity. Complexity means that the risk is not only determined by one factor, but that many factors interact and in their entirety determine the occurrence and the level of impact of the risk.

Whether a new product will be successful on the market depends on the success of its development, the start of production, market access, the development of the economic environment, the behaviour of competitors, etc. And each of these factors can itself depend on other factors.

The assessment of such a risk therefore requires the risk expert to aggregate the interaction of all influences. If the expert were able to do this, he could actually go right ahead and aggregate the entire corporate risk as well. In this task, one's own ability to aggregate many influences in one's head is easily overestimated. It seems more sensible to apply a methodical risk analysis including a Monte Carlo simulation already for the assessment of the individual risks, as provided for in the auditing standard for the aggregation of risks.

Methodical risk analysis here means 'disaggregating' or decomposing the risk, i.e. identifying the influencing factors and describing their interaction. Data is sifted and distributions for the influencing factors are selected and calibrated. The assumptions made are documented. Finally, the influences with their correlation under these assumptions are aggregated with Monte Carlo simulation, checked for plausibility and validated on the data.

Once a risk has been methodically assessed, the question raised in section 4 whether the simulated distribution can be represented in the ERM. In Figure 1 the orange histogram represents the simulated risk. For this risk, the most commonly used distributions in the industry were calibrated, i.e. uniform distribution, PERT distribution and triangular distribution[4]  [5].

Figure 1 - Methodically assessed risk and calibrated distributions

The simulated composite risk shows a long and thin tail towards the large losses. This is characteristic of risks in ERM, as exceptionally high losses in a risk only occur in the case of a 'chain of unfortunate circumstances', i.e. when something goes wrong in several respects. In most cases, the influencing factors compensate each other to a certain extent and one finds oneself in the middle of the loss scale.

Uniform, PERT and triangular distributions are too rigid templates that cannot adapt to this course. They do hit the minimum and maximum of the simulated losses, but dramatize the losses and overestimate the opportunities on the other side.

This approach therefore meets with criticism in practice, as the departments do not accept the dramatized losses (cf. question 4). The risk experts react to this (cf. question 3) by narrowing the three-point distribution (blue line). They then hit neither the minimum, nor the maximum, nor the most probable value and cut the large losses out of the risk altogether. They adjust the representation of the risk to fit the template. In this way, precisely that part of the risk disappears from the map that is most likely to endanger the company, that would require action, that is essential for the assessment of insurance and thus represents the actual core of risk for ERM.

Figure 2 - Transfer of risk assessment into ERM

The image also shows an adaptive distribution that accurately captures the risk in its shape and range without under- or over-drawing, and can be transferred to the ERM at the click of a mouse[6].

Indicator 6: Does risk assessment require a cross-functional perspective or the integration of different views and priorities?

The complexity of the risk assessment may also increase if the risk affects multiple stakeholders or encounters an unclear management environment (see questions 2 and 5). Uncertainty is not a risk for every stakeholder to the same extent. Depending on the perspective and priority, an uncertainty is sometimes a major problem, sometimes an opportunity and sometimes unimportant.

In this situation, the risk assessment therefore depends on who prevails in the discussion and priority setting, which trade-offs are made and which position the risk expert takes in this field of tension. This is all the more true if only one risk expert assesses the situation (cf. question 1).

Indicator 7: Do we fully understand the situation? Or does it have implications beyond the obvious?

It is in the nature of things that you gain routine and experience in an activity when you have done it many times and seen a lot. You know the environment and can react to it.

But what happens when we are confronted with a new situation? Is our judgement then still as certain? Do we fully understand today how artificial intelligence, measures against the climate crisis, geopolitical conflicts, competitors' activities and much more will affect our business?

Experts are faced with the same challenge here and at the same time with the pressure of questions 2 to 6. An expert assessment can mean everything here.

How a risk analysis can be carried out under these conditions is exemplified by the studies of the Intergovernmental Panel on Climate Change (IPCC)[7]. In essence, it is a matter of defining a methodology that uses the relevant data, identifies and calibrates the influences and describes and evaluates their interrelationships.

Indicator 8: Do the experts lack feedback, i.e. do the experts not receive direct feedback on the accuracy of their predictions?

The topic of validation is not yet very widespread in enterprise risk management. Forecasts are made but not checked. As a result, there is a risk that the experts and the whole organisation do not learn from their mistakes and maintain wrong forecasts permanently.

The lack of feedback affects not only the assessment, but also the inventory. It is not uncommon for companies to become remediation cases due to risks that were not even included in the risk inventory. The pandemic is an example for very many affected companies.

An incident database, in which risks that have occurred are documented and evaluated, exists in areas where there is a high intrinsic motivation to analyse, understand and manage risks, such as in aviation. In enterprise risk management, they exist almost only in companies that are required to do so by regulation, such as banks and insurance companies.

Conclusion

Asking for expert assessments in a survey seems to be a solution to assess and evaluate risks in a heterogeneous environment with little data, cheaply and in a short time. However, their quality is completely unclear.

Each of the aforementioned early warning indicators points to a recording and assessment error of unknown direction and magnitude in an unknown number of risks in the risk inventory. That is a lot of unknowns that accumulate.

If you answered yes to several of these questions, the validity of the risk analyses is no longer guaranteed and no transparency is created. It will not even be clear whether the figures have hit the order of magnitude overall. Errors can cancel each other out. But they can also build up.

Expert assessments are more of a misdirection than a royal road of enterprise risk management. Pro-forma reporting can be done under these conditions. Risk management hardly.

If expert assessments worked, there would be no need for receipts or tax returns. It would be enough for the innkeeper, the craftsman and the entrepreneur to count the till at the end of the day and, as experts, tell the tax office how much tax they have to pay.

Risk management needs methodical analysis and traceability on the basis of data, disclosed assumptions, calibrations, evaluations of correlations, error estimates and validations. Only under these improved conditions can it be seen how great the corporate risk is, where it comes from, what it costs and where it is worth taking and where not. Then enterprise risk management can also be integrated into controlling and corporate management.

List of sources and further references:

Institut der Wirtschaftsprüfer - Hauptfachausschuss (HFA), IDW-PS-340

Bogner, Alexander; Beate Littig; Wolfgang Menz, Interviews mit Experten – Eine praxisorientierte Einführung, Springer (2014)

Höglinger, M.; B. Jann (Höglinger/Jann 2018): More is not always better: An experimental individual-level validation of the randomized response technique and the crosswise model. PLoS ONE 13(8): e0201770. (2018) https://doi.org/10.1371/journal.pone.0201770

Kaiser, Robert; Qualitative Experteninterviews – Konzeptionelle Grundlagen und praktische Durchführung, 2. Aufl., Springer (2014)

Von dem Berge, Benjamin: Teilstandardisierte Experteninterviews, in: Tausendpfund, Markus (Hrsg.): Fortgeschrittene Analyseverfahren in den Sozialwissenschaften, Springer (2020), S. 275-300

[1] Institut der Wirtschaftsprüfer - Hauptfachausschuss (HFA), IDW-PS-340

[2] Höglinger/Jann 2018

[3] See also the film "Better Expert Assessments in Enterprise Risk Management", https://youtu.be/uC92yrg64C0.

[4] The multinomial distribution was omitted because it only knows point masses and cannot fit a continuous risk.

[5] Evaluation and calibration were carried out with Risk Kit.

[6] The assessment was added to the Enterprise Risk Evaluator.

[7]https://www.ipcc.ch/reports/