Examining the influence of entrustability-based rating scales on examiners’ cognition in medical education.

The Institute for Primary Care and Health Sciences at Keele University is the largest and most successful (www.keele.ac.uk/pchs/). 91% of Keele’s research in Primary Care was judged world leading or internationally excellent in the Research Excellence Framework (REF) 2014. Keele University School of Medicine was ranked 1st in the 2015 National Student Satisfaction survey, and consistently ranks within the top 10 nationally on all ranking systems of UK medical schools. The School of Medicine’s Medical Education Research Group has an ambition to become nationally leading, with an international reputation, over the next 5 to 10 years, as well being included in the REF 2020 return.

Brief background to the research:
Accurate assessment of the clinical skills of trainee healthcare professionals is vital for both patient safety and educational development and fairness. Current assessment approaches rely on observation and judgement of by professionals of students “competence” but are beset by problems: observation is resource intensive; unwanted variations between examiners produce problematic variability in scores; examiners are reluctant to fail poor candidates; and examiner training has limited benefits.

This PhD will contribute to the growing field of “Assessor Cognition” which seeks to understand influences and biases on examiners’ judgements, with a view to producing theoretically-orientated solutions. One finding which has emerged over the years is it appears that examiners vary in their attentional focus whilst observing performances, and struggle to manage the mental workload of the judgement task.

One promising innovation in this arena uses the concept of “entrustability” instead of traditional notions of competence. This approach, rather than asking assessors to judge the degree of competence a student exhibits, asks them to indicate the degree to which they would trust the student to perform independently, or what degree of supervision they judge the student to require for a particular task. Entrustability-decisions are suggested to more closely align with assessors’ innate thinking, and therefore be easier to use.

Despite these calls, there is a lack of studies that have compared the degree of such benefits under controlled conditions. Moreover, whilst mechanistic influences on assessors’ judgements have been posted, few studies have investigated how asking assessors to make entrustability judgements influences their thinking whilst observing performances; the cognitive demands of the process; or the susceptibility to bias of assessment judgements.

Aim:
To compare the influence of entrustability-based rating scales with competence-based rating scales on the processes and outcomes of assessors’ judgements in medical education.

Suggested Study 1 research question: How do examiners’ described-thoughts compare when using an entrustment-based scale compared with a competence based scale? How do examiners conceptualise “trust” in the context of a directly observed performance?

Examiners will be asked to judge 2 standard videos of student performances, one using an entrustability scale and the other with a traditional competence-based scale (counterbalanced). Participants will comment on their thinking whilst observing performances, and perceived influences on their judgements. Participants thinking will be probed in semi-structured follow up interviews. Analysis will use constructivist grounded theory methods to build a rich model of the comparative influences of both scale types on judgements, and the factors which assessors use to develop an impression of trust. Expected n= 12-15.

Suggested Study 2 hypothesis: assessors’ judgements collected via entrustability-based scales will show higher reliability and lower mental workload than judgements collected via competence-based scales.

Two group, randomised, internet-based experimental design. Participants will judge standard student performances, by group using either a) an entrustability-based or b) a competence-based rating scale as well as providing free text feedback on the performances. Participants will provide subjective ratings and secondary-task measures of mental workload. Analyses will compare score-consistency and mental workload between groups; regression analysis will be used to determine mediating effect of mental workload on scores. Post hoc analysis of free text feedback will be used to determine whether groups differed in their focus or specificity of feedback. Expect n= 40-50.

A 3rd study (likely similar design to study 2) will test further hypotheses relevant to the overall aim, likely to focus on whether entrustability scales reduce or increase susceptibility to social bias or heuristics.

Outcomes: The initial literature review is expected to be publishable as a narrative review, producing 4 academic papers.