Frequentist inference is the most commonly used analysis method in randomised trials. Whilst guidelines for reporting underscore the importance of the interpretation of treatment effects supported by confidence intervals, most ultimately interpret the primary outcome based on statistical significance. This conundrum, well understood by statisticians but less widely appreciated by key stakeholders, led to the publication of a statement on the correct interpretation of p-values by the American Statistical Association (Amrhein Nature. 2019). Five years on, little if anything has changed. A particular risk of reliance on statistical significance is that potentially effective treatments could be labelled as ineffective if the p-value falls on the ‘wrong side’ of 0.05. Thus, a p-value of 0.06 will result in a frequentist significance-based interpretation of a treatment effect as ‘negative’ whilst there is still a substantial likelihood of there being a genuine and clinically important treatment effect.
By way of example, we reanalysed the primary outcomes of 150 major randomised trials published in the last five years in key general and women’s health journals. We concluded that as much as a third of women’s health trials that are reported as ‘negative’ using frequentist significance-based interpretations have strong statistical evidence (>80% posterior probability) of moderate to large treatment effects (NNT<100 for the primary outcome) (Hemming et al 2021, submitted).
Almost all trials with non-statistically significant primary outcomes fail to interpret values supported by their confidence intervals [Gates BMJ Open. 2019]. This longstanding problem needs a solution. A Bayesian approach which quantifies the strength of the statistical evidence of clinically meaningful effect sizes is one potential solution; other potential solutions are increased guidance around interpretation. Whilst many solutions to this problem have been proposed there is a dearth of literature on WHY these issues arise. Potential explanations are a lack of knowledge, a desire for publication, a prioritisation of not increasing the risk of a type-1 error, or a lack of clarity in language used to describe key findings (for example the use of double negatives can create potential for misinterpretation of intended meaning). Only after there is a real understanding of why this error perpetuates is it possible to identify solutions, along with barriers to these potential solutions.
What the studentship will encompass:
Objective: To establish a methodological framework for interpreting statistical findings from randomised trials
Plans for project:
1. Review of the literature and case study
a. Review of interpretation of key findings in a random sample of RCTs published in a purposively selected sample of journals – this will quantify the breadth and scope of the problem.
b. Identification of several case studies to illustrate the methodology to implement the proposed methods and problems. These will be carefully selected to illustrate the range of interpretation problems that commonly exist.
2. Qualitative studies. Using a range of different methods (e.g. focus groups to debate the issues; or interviews where there is continued interaction between researcher/respondent where the researcher asks questions/prompts in response to what the participant has said; or several iterations of interviews at different phases of a trial where the participant is carrying out different activities (so as to allow if there are influences of pre-conceived ideas about effectiveness)). A different range of qualitative techniques will allow exploration of the range of complex factors that could be influencing interpretation. The project will seek to understand a range of perspectives, including but not limited to principal investigators, statisticians, and journal editors.
3. Interpretation tools. Findings from the qualitative studies will be used to produce guidance with case studies illustrating the proposed methods. This guidance will undergo considerable scrutiny and debate, using consensus techniques such as a Delphi process.
The project will work with stakeholders to explore acceptability of recommended approaches. Outputs: Provide guidance on how researchers should interpret key outcome findings from randomised.
This project is targeted at a qualitative researcher and an MSc or PhD in qualitative methods is desirable but not essential. The project could also be undertaken by a quantitative health researcher who is open to training and researching using a different methodology.
HOW TO APPLY
You are applying for a PhD studentship from the MRC TMRP DTP. A list of potential projects and the application form is available online at:
Please complete the form fully. Incomplete forms will not be considered. CVs will not be accepted for this scheme.
Please apply giving details for your first choice project. You can provide details of up to two other TMRP DTP projects you may be interested in at section B of the application form.
Before making an application, applicants should contact the project primary supervisor to find out more about the project and to discuss their interests in the research.
The deadline for applications is 4pm (GMT) 18 February 2022. Late applications will not be considered.
Completed application forms must be returned to: [Email Address Removed]
Informal enquiries may be made to [Email Address Removed]