Real world data

DSU report

The use of real world data for the estimation of treatment effects in NICE decision making (PDF, 1MB) (June 2016, updated December 2016)

This report aims to assess the current guidance on the use of real world data (RWD) for the estimation of treatment effects in NICE decision making and identifies areas where further research or guidance is required.

It builds on the NICE Decision Support Unit (DSU) Technical Support Document (TSD17) “The use of observational data to inform estimates of treatment effectiveness in technology appraisal: methods for comparative individual patient data” (Faria et al, 2015), which focused on methods commonly used to estimate treatment effects from non-randomised studies, where individual patient data (IPD) is available.

This report expands on the TSD by considering how RWD has been used to inform decision making in seven of NICE’s programmes, how it could have been used and the guidance that NICE currently provides to those responsible for submitting evidence, critiquing evidence and making decisions based on those assessments.

Central to the concerns associated with the use of RWD is the issue of selection bias, which we define as bias that arises when comparing the effect of a treatment in groups that are systematically different in variables that have an independent effect on the outcome of interest (Faria et al, 2015).

Attempts must be made at the study design, analysis and interpretation stage to try and mitigate this bias (Faria et al, 2015, Hernan and Robins, 2016). NICE is moving towards further use of RWD in its decision making, but selection bias is inevitable.

Within the current methods guidance, there is considerable variation in the extent that different programmes deal with selection bias. However, none of the guides say how primary studies should deal with selection bias.

The three case study examples in this report highlight some of the challenges associated with minimising selection bias when using RWD to estimate treatment effects. In particular, the hip replacement example shows how attempts to minimise selection bias require early investment at the design stage in accessing large-linked observational data. The MAGEC and Bosutinib examples illustrate that, if this investment is not made at the design stage, then it is very challenging to define the counterfactual so as to minimise selection bias.

The findings of this report highlight areas for future research and where further methods guidance could be developed. There should be further research into methods of synthesising data from various sources, such as single-arm trials and historical controls, and summary statistics, and methods for handling selection bias in context of large linked, longitudinal observational datasets, and within that, methods for handling time-varying confounding and sequential treatment decisions.

To receive notification when new reports or documents are uploaded to this website, please join our mailing list by emailing dsuadmin@sheffield.ac.uk, or follow us on Twitter, @NICE_DSU.