Using expert elicitation to better characterise confidence and uncertainty in climate change risk assessments

"Every five years the UK Government carries out an assessment of the current and future risks to the country from climate change, known as the Climate Change Risk Assessment (CCRA). The first CCRA Evidence Report was published in 2012, presenting the latest evidence on the risks and opportunities of climate change for the UK to 2100. The assessment was undertaken across 11 sectors and drew evidence from literature reviews, expert elicitation and more detailed quantitative analysis, where the data allowed. This assessment considered more than 700 potential risks and selected more than 100 risks for detailed review. The CCRA is important because it provides underpinning evidence that can be used by Government to help inform priorities for action and appropriate adaptation measures such as the National Adaptation Programme.
Throughout the CCRA documents, the words uncertainty and confidence appear frequently. The characterisation of confidence in potential impacts is important in the relative rating of the different risks because it is used to downplay the impact indicators both in terms of their characterisation and relative ranking. The CCRA Evidence Report states that confidence should be read as “how confident we are in the direction and magnitude of change” despite the fact that it also states that “we cannot associate likelihood with specific changes in climate risks, therefore likelihood is not used in this assessment”. This PhD project will demonstrate how expert elicitation can be used to fill these gaps and be more transparent about confidence judgements. The expert judgements that can be captured cover situations where there are no data or relevant model runs and situations where model runs do not give the full picture of reality. This is because elicitation can help to take stock of the uncertainty about quantities of interest without the cost of data collection or model building.
It is clear in the evidence report that expert elicitation is seen as a poor replacement for data and mathematical models: “The risk metrics considered in this assessment vary in character and whilst some have been quantified others have had to rely on expert elicitation, or a narrative based on the literature”. However, with such a great number of assessments to be made and a large number of unmodelled/unobserved consequences, expert elicitation may be the only feasible way of providing a sound evidence base.
The elicitation of relevant information for the CCRA has several challenges:
1) The selection of representative experts is important if the aim is to capture current scientific understanding. Methods for choosing such a set are not readily available and experts are often chosen based on their availability.
2) There are obvious dependencies across risks and their potential impacts that need to be characterised to get a complete picture. Most elicitation methods concentrate on one variable at a time and research will need to be done on how to capture expert judgement on dependencies and on impact response functions.
3) There is an overreliance on computer models in the current CCRA. Characterising and communicating the gap between models and reality will be challenging.
4) The vast number of judgements that will be needed will require research to be done on rapid elicitation protocols. The aim would be to produce a simple system to capture expert knowledge without compromising the value and rigour of a traditional, time-expensive elicitation protocol.

This PhD project will design a protocol that addresses these challenges and can easily be implemented by researchers carrying out future impact assessments such as the CCRA 2022.
This PhD project will also investigate the level of detail and the relative confidence in each of the sector using statistical data exploration techniques. This will provide evidence whether having data and models has any impact on the level of detail given in the associated impact summary (and hence draws more attention in the reports). It seems that the summary mechanisms provided in the CCRA report that highlight the relative potential impacts of potential risks and the relative confidence in those impacts occurring give a skewed view of the risk portfolio. Where the assessments are based upon models and data, the listed risks and impacts tend to be more detailed with a greater degree of confidence assigned to them."

http://www.nercdtp.leeds.ac.uk/projects/index.php?id=521