D1-05: Assessing the challenges of applying standard methods of economic evaluation to public health programmes

Principal investigator

Mike Drummond, York

Background

One of the most recent and comprehensive sets of guidelines for the economic evaluation of health technologies is that issued by the National Institute for Health and Clinical Excellence (NICE) in the UK. These represent state-of-the-art methods presented in terms of a 'reference case' that set out the range of methodologies deemed most appropriate when undertaking an economic evaluation.

To date, the vast majority of technologies appraised by NICE have been drugs, devices or medical procedures and there is little experience of using the same methodological standards to evaluate public health programmes. While NICE guidelines were written with no particular technology or health programme in mind, specific difficulties may arise if they are applied blindly in the evaluation of public health programmes that contain very broad costs and benefits, or to a programme directed at populations or communities rather than specific individuals. In addition, a particular feature of many public health interventions is concern with health inequalities. Standard economic evaluation methods focus on efficiency (i.e. the maximisation of health gain) rather than on equity (i.e. the distribution of health gain) and, accordingly, the evaluation of public health interventions needs to pay more attention to equity considerations.

While it is anticipated that the general principles of the NICE reference case will hold for the evaluation of public health programmes, departures from the reference case may be justifiable with respect to estimates of relative treatment (programme) effects, perspective for estimation of costs, measurement of outcomes and equity considerations.

Aims, methods and contribution

The project's aims were to assess the challenges of applying the NICE economic evaluation 'reference case' to the evaluation of public health programmes and to identify the refinements needed to methods. Economic evaluations of public health programmes were identified via the NHS Economic Evaluation Database (NHS EED), which systematically identifies, describes and evaluates economic evaluations of health care interventions, and the Database of Abstracts of Reviews of Effects (DARE), which includes critical abstracts of systematic reviews which have evaluated health care interventions. In addition, any other economic evaluations of public health programmes, undertaken in the last five years and published in the English language, were identified through a systematic review. These included broad, community based programmes, such as health awareness campaigns, and more targeted programmes, such as smoking cessation clinics.

A critical appraisal of the methods used in the studies was undertaken and a comparison made with the methods outlined in the NICE reference case. The main discrepancies in methods were noted and an assessment was made on whether any departures from the reference case can be justified in order to accommodate the evaluation of public health programmes. In addition, consideration was given to the challenges of assessing public health initiatives that are not typically subjected to economic evaluation (e.g. tax and regulation changes). Where possible, case studies were developed to illustrate the main methodological issues. A workshop was held with key researchers and providers of public health programmes in order to review, and refine, the recommendations.

The Wanless report recommends that a "consistent framework (such as the methodology developed by NICE) should be used to evaluate the cost-effectiveness of interventions and initiatives across both health care and public health" (para 6.42). However, it also recognises that "there are some practical issues which can make it more difficult or more costly to perform convincing economic evaluations of preventative interventions" (para 6.34). This project explores the practical and methodological challenges in detail and will make assessments of the extent to which the NICE reference case guidelines can be applied directly to public health interventions. Where reference case guidelines require adaptation, the project highlights these and provides methodological advice on the ways forward.

Duration: October 2005 - June 2006 (9 months)