Exploring the predictive validity of the UK Clinical Aptitude Test

The application process to UK medical schools is highly competitive with many academically high achieving attempting to obtain places. Thus it can be difficult to discriminate between a relatively homogenous group of highly performing candidates. Also, traditional measures of academic attainment favour those candidates from selective and/or independent schools.

The UK Clinical Aptitude Test (UKCAT) was introduced in 2006 in order to help discriminate between highly achieving candidates and support the widening of access to medical and dental education to under-represented groups. The test scores thus provided a continuous metric of ability that was, to some extent, distinct from that represented by traditional school achievement. There is some evidence that the UKCAT may be less sensitive to schooling than traditional measures of academic attainment. There is also emerging evidence that the UKCAT has some ability to predict medical school performance, even after adjustment for A level attainment.

However, estimating predictive validity in a selection tests offers some specific challenges. In fact it is very easy to underestimate the predictive validity of a selection test, as, paradoxically, the perfect selection tests may show no correlation with the outcome of interest at all. This is because a ‘perfect’ selection test would only select candidates with a relatively similar high performance on the task of activity they were being selected for. With little variation in outcome (i.e. little variance in later performance) it is impossible to show a significant or meaningful correlation between the original test scores and the outcome of interest! Indeed, McManus puts this succinctly as follows:

“The fundamental statistical problem in assessing selection measures is that the correlation between the outcome measure and the selection measure is only known in those who have been accepted”

A variety of statistical approaches have been tried over the years to address this problem, though no single one at present appears particularly effective. Recent advances in statistical methods and computing power have provided an opportunity to revisit this fundamental problem in psychometrics. In particular, new approaches to the modelling of missing data may prove fruitful in this area. Thus, the methodological strand of this PhD proposal aims to treat the outcomes in non-selected candidates as a special case of missing data. By exploring missing data modelling techniques we hope to establish what may be the optimum approach for a variety of circumstances in order to establish ‘true’ predictive validity (i.e. ‘construct-level’ predictive validity). This will be facilitated by the use of existing datasets where both baseline outcome measures are available. It may be that different assumptions underlying the missing data modelling process will have different validity in different circumstances. Such work will be aimed at developing a more realistic appraisal of how well the UKCAT predicts performance in medical students, both at university and beyond. The approaches used to estimate the values of the unobserved outcome variables are likely to include; Expectation Maximisation (EM), Multiple Imputation (MI); and, Markov Chain Monte Carlo (MCMC) simulation.

The second phase of this proposed PhD project is more pragmatic in nature. It consists of using the methodology developed in phase 1 to develop a more accurate predictive model of what variables, or combination of variables, predict different elements of performance both during and immediately after medical school. For example, certain variables, which may include non-cognitive elements, such as scores on the personal qualities assessment (PQA) or Situational Judgment Tests (SJTs) may significantly predict performance at skills, rather than theory exams during medical school. It would ultimately be aimed at helping medical schools make more informed choices about what elements of the UKCAT, and other selection measures, they weight in the admissions process. Moreover, this phase of work is intended to help medical selection move away from the idea that there are a unique set of selector variables that lead to the section of the ‘ideal medical student’ (one might refer to this as the ‘cookie cutter’ effect). This work may also include an element of semi-qualitative investigation, informed by recent advances in decision theory, to understand what importance admissions Deans in in UK medical schools may place on different qualities in potential students and the extent to which they are willing to ‘trade-off’ certain characteristics for others.