A guide to critical reading of influenza vaccine cost-effectiveness analyses

Abstract

Introduction: Influenza vaccines are formulated each year to prevent serious illness in at-risk individuals, including
elderly people. Healthcare decision-making is mainly based on the economic evaluations (EEs) (i.e., costeffectiveness
analysis [CEA]) of vaccines; however, understanding the limitations of these models and correctly
interpreting the results may be challenging. Here, we provide a practical Guide that will help readers who are not
experts in the field of health economics or influenza to critically review influenza vaccine EEs.
Methods: This Guide is based on the findings of a systematic review of the literature, a critical analysis of the
available EEs published for influenza vaccines for older adults in Spain, and applicable national and international
guidelines on EE and influenza modeling. It has been developed by a multidisciplinary board of experts in influenza,
vaccines, and health economics.
Results: The guide provides tips to help the reader assess whether an EE design is fit for its purpose in terms of
comparators, time horizon, perspective of the analysis, population analyzed, and whether appropriate modeling
methods were applied. It detects the uncertainty arising from input data and the implications of this uncertainty
on the results.
Conclusions: Ultimately, this resource aims to empower decision-makers, particularly those without expertise in
health economics or vaccinology, to critically read and interpret EEs, thus favoring evidence-based informed decisions
that will improve the efficiency of influenza vaccination programs.

Introduction

Influenza is a viral illness that occurs in seasonal epidemics each year (1). Vaccines are formulated annually to prevent serious disease in at-risk individuals, mainly elderly people, pregnant women, and all-age patients with comorbidities in which influenza may develop into a more serious condition.

Faced with budget constraints, health authorities must optimize resources by integrating economic evaluations (EEs) alongside clinical data in their decision-making. Spain, for example, has a decentralized healthcare system in which each autonomous region is empowered to decide which type of vaccine will be used in the influenza vaccination program (2). In this context, decision-makers must rely on robust EE, such as cost-effectiveness analyses (CEAs), to guide their choices.

Guidelines to improve the design and development of economic evaluation (3,4) and economic evaluation of influenza vaccines (5,6) have been published. However, as we previously reported in a critical review of Spanish EEs (7), these guidelines are not consistently followed. In particular, we identified inadequate management of uncertainty in the evaluated EEs, as well as insufficient transparency in the presentation and justification of the design and parameter choices. Understanding the limitations of the published models can be challenging for non-expert readers, particularly given the specific characteristics of influenza. With this Guide, we aim to provide a practical framework to support non-expert readers in health economics or influenza in critically reviewing influenza vaccine EEs. Unlike existing ones, this Guide is written from the perspective of the specialists—such as health professionals, clinicians, and policymakers—who may encounter challenges in assessing study quality, making it both practical and influenza-specific. It may enable them to appraise influenza EEs critically, identify sources of uncertainty, and make informed, unbiased decisions without requiring in-depth technical knowledge of EE or modeling.

Methodology

The present study is based on the findings of a previous investigation (Lejarazu et al.) (7), consisting of a systematic literature review (SLR) of Spanish EEs published from 2016 to 2022, followed by a structured appraisal of the identified studies by a multidisciplinary expert panel (including clinicians from all therapeutic areas involved in influenza prevention and specialists in pharmacoeconomics), using established quality-assessment checklists such as the Transparent Uncertainty ASsessmenT (TRUST) tool (8), and the WHO (World Health Organization) guidance on the economic evaluation of influenza vaccine strategies (6) (further details on the systematic search strategy and the critical appraisal of the EEs are available in the referenced publication). Rather than updating that review, the primary objective of this Guide was to address the recurrent methodological limitations and key considerations identified in the SLR and to translate them into practical guidance for the critical appraisal of EEs.

Uncertainty in economic modeling: it can be an issue when not adequately controlled

EEs rely on mathematical models representing a simplified view of the disease and comparing the costs and outcomes of a new intervention to those yielded by standard care (9). As such, uncertainty is inherent to EEs and may affect the incremental cost-effectiveness ratio (ICER) to a greater or lesser extent. Thus, it is important to identify and quantify uncertainty to ensure that EEs are correctly interpreted and can be reliably used as a basis for decision-making (10).

Various degrees of uncertainty may affect EEs, namely: the structure, the methods on which they rely, and the parameters introduced. Here, we summarize the three types of uncertainty and the importance of controlling them with sensitivity analyses.

Model or structural uncertainty: it arises from the choice of the model (e.g., dynamic or static), its structure, and how health states are connected. It also analyses whether the model correctly assesses the main objective (9,11-13).

Methodological uncertainty: it arises from the evaluation methods used to estimate the use of resources and the health outcomes of the assessed interventions, i.e., the perspective chosen, the time horizon, and the analytic technique used (cost per life year or per quality-adjusted life year (QALY), discount rate, etc.) (13).

Parameter uncertainty: it arises from the specific data used to feed the model (inputs), such as efficacy and/or effectiveness, unit costs, resource use/costs, epidemiological burden of the disease, etc. (9). Parameter uncertainty is a second-order uncertainty because parameters are themselves estimated quantities (11).

Critical reading of influenza vaccine economic evaluations: a practical guide

After defining the basic concepts of EEs (see Glossary, Supplementary Table 1) and uncertainty, we will go through the most essential questions the reader should ask to ensure they have fully understood an EE of influenza vaccines. A practical checklist for critical reading is provided in Table 1, complemented by the minimum reporting standards recommended for influenza vaccine EEs. Additionally, an applied example of how to critically read an EE is presented in Supplementary Table 2.

Assessing the general model design: Does it really answer the research question?

This section of the checklist reviews the foundation of the model, assessing structural uncertainty. The model must be designed to answer the research question; therefore, its structure must reflect this purpose. While more sophisticated models can simulate more complex situations, they also incorporate an increasing number of equations that need additional data input, and this is likely to increase the level of uncertainty of the model. Thus, an appropriate trade-off should be sought between the risk and benefit of increasing the complexity of a given model.

Was the most adequate type of model used?

Different factors, such as the decision context, the mechanisms relevant to the research question, or the target population, should drive the choice between static and dynamic models for influenza vaccines. If the virus’s dynamic exposure is not required to be represented and the time horizon is short, deterministic decision models such as decision trees are appropriate. These models are commonly used to evaluate direct-effect strategies (e.g., elderly vaccination strategies). If the horizon is longer, the next consideration is whether patient history and interactions are relevant. If not, Markov models are the right choice, especially when there is a limited number of health states and interactions. However, when prior history and interactions are important, discrete-event simulation models may be appropriate. If dynamic exposure probabilities should be considered, for example, when vaccination strategies can alter transmission at the population level, dynamic transmission models are recommended. These models capture indirect effects such as herd immunity, inter-group interactions, and evolving infection risks that depend on the number of infected individuals. They are particularly appropriate for complex scenarios, such as child vaccination strategies, where both direct and indirect effects are substantial (6,14). Figure 1 represents a decision diagram for choosing between static and dynamic models.

Table 1-. Checklist for critical reading and minimum reporting standards for influenza vaccine EEs

CRITICAL READING OF INFLUENZA VACCINE EEs			REPORTING OF INFLUENZA VACCINE EEs
Question	Answer	Action	Correspondence with CHEERS 2022 ( 40 )	Minimum reporting set
Assessing the model design: Does it really answer the research question?
Was the chosen type of model adequate for the addressed objective?	YES	Continue reading	CHEERS 16 (Rationale and description of model)	Specify model type and justify the choice (static vs dynamic).
	NO	You cannot decide based on the current EE	CHEERS 17 (Analytics and assumptions)	Report model structure and key assumptions.
	I don’t know	Seek the help of an expert
Assessing modeling methods: are they appropriate?
Were both the healthcare provider and the societal perspective presented?	YES, both	Continue reading	CHEERS 8 (Perspective)	State perspectives used (healthcare/societal) and resources included.
	Only the healthcare provider’s perspective was presented	If evidence is incomplete, look for data on the impact on society as a whole
	Only the societal perspective was presented	If evidence is incomplete, look for separate data from the perspective of the healthcare system
	I don’t know	Seek the help of an expert
Were the results presented separately from each perspective?	YES, each payer’s perspective was presented separately	Continue reading	CHEERS 23 (Summary of main results)	Present outcomes and ICERs separately by perspective.
	NO, they were not	Look for data on separate perspectives
	I don’t know	Seek the help of an expert
Was the time at least one influenza season?	YES	Continue reading	CHEERS 9 (Time horizon)	Specify season(s) modeled and duration
	NO	You cannot decide based on the current EE
	I don’t know	Seek the help of an expert
Was the time horizon for the societal perspective the whole patient’s lifetime?	YES	Continue reading	CHEERS 9 (Time horizon)	Report lifetime horizons for long-term outcomes and discount rate
	NO	Look for data on long-term vaccine consequences
	I don’t know	Seek the help of an expert
Was the population defined based on the local recommendations for influenza immunization and evaluated vaccine labels?	YES	Continue reading	CHEERS 5 (Study population) CHEERS 6 (Setting and location)	Align population with national recommendations and vaccine labels.
	NO	You cannot decide based on the current EE
	I don’t know	Seek the help of an expert
Were all the relevant comparators based on local recommendations included in the EE?	YES	Continue reading	CHEERS 7 (Comparators)	List all locally relevant vaccine comparators and justify exclusions.
	NO, only the most relevant comparator was included/only relevant comparators not previously assessed	Look for EEs assessing the comparator with other relevant comparators and treat the body of evidence as a whole
	NO, the chosen comparator is not relevant	You cannot decide based on the current EE
	I don’t know	Seek the help of an expert
Assessing the quality of the model input: were input parameters the best?
Was the burden of illness estimated over at least 5 seasons in static models?	YES	Continue reading	CHEERS 22 (Study parameters) CHEERS 24 (Effect of uncertainty)	Report the number of seasons, data source, and exclusion of pandemic COVID-19 seasons (if applies).
	NO	View the results with caution. The fewer the number of seasons included, the higher the uncertainty surrounding the results. Seek more evidence
	I don’t know	Seek the help of an expert
In a dynamic model, were good sources for epidemiologic parameters chosen?	YES	Continue reading	CHEERS 22 (Study parameters) CHEERS 24 (Effect of uncertainty)	Provide sources for attack rate, R₀, contact matrix, or coverage; justify assumptions.
	NO	View results with caution
	I don’t know	Seek the help of an expert
In a dynamic model, was the model calibration reported?	YES, completely	Continue reading	CHEERS 16 (Model rationale and description) CHEERS 17 (Analytics and assumptions) CHEERS 24 (Effect of uncertainty)	Describe calibration targets, method, and goodness-of-fit.
	YES, but it was insufficient	You cannot decide based on the current EE
	NO	View the results with caution
	I don’t know	Seek the help of an expert
What were the absolute and relative vaccine effectiveness rates?	SR with meta-analysis of RCTs	Continue reading, it is probably the best evidence available; however, pay attention to the risk of bias and heterogeneity of included studies, as well as the confidence intervals	CHEERS 12 (Measurement of outcomes) CHEERS 13 (Valuation of outcomes) CHEERS 22 (Study parameters)	Cite vaccine effectiveness source (systematic review/meta-analysis preferred); note number of seasons and case definition (PCR vs ILI)
	SR with meta-analysis of observational studies	Continue reading, but pay attention to the risk of bias and heterogeneity of included studies. Observational studies are often very diverse in methodology. Remember that if heterogeneity is high, no conclusions should be drawn. Also, pay attention to the width of confidence intervals: the wider they are, the greater the uncertainty of the results
	Single RCT over more than one influenza season	Probably good evidence; however, an evidence quality check using an appropriate checklist could be useful. Also, checking coherence with observational data is advisable.
	Single observational study over more than one season	Study design may be very variable; checking quality using an appropriate checklist can be useful. You can seek more evidence for comparison.
	Single-season studies	Uncertainty is intrinsically high due to inherent inter-seasonal virus variability. Checking coherence with other available studies is advisable.
	I don’t know	Seek the help of an expert
	In any event, the help of an influenza expert is advisable
Were the chosen utility values applicable to the specific population of the EE?	YES	Continue reading	CHEERS 13 (Valuation of outcomes)	Report utility source and justify applicability to the population.
	NO	Take the results with precaution
	I don’t know	Seek the help of an expert
Were all relevant (differential) healthcare resources included in the cost analysis?	YES	Continue reading	CHEERS 14 (Measurement and valuation of resources and costs)	The list included resources and sources (national data or expert opinion).
	NO	View the results with caution, and estimate the impact of missing resources on the results
	I don’t know	Seek the help of an expert
Was the unit cost derived from appropriate official sources?	YES, and the mathematical treatment of the costs was clearly detailed	Continue reading	CHEERS 14 (Measurement and valuation of resources and costs) CHEERS 15 (Currency, price date, conversion) CHEERS 22 (Study parameters)	Provide official cost sources, region, and price year.
	YES, but the treatment of different cost sources was not clearly explained	View the results with caution; check DSA for impact on results
	NO	View the results with caution; check DSA for impact on results
	I don’t know	Seek the help of an expert
Was the method used for productivity loss stated?	YES	Continue reading	CHEERS 14 (Measurement and valuation of resources and costs)	State method (human capital/friction) and wage source
	NO	View the results with caution
	I don’t know	Seek the help of an expert
In long-term evaluations, were costs and benefits correctly discounted, according to national recommendations?	YES	Continue reading	CHEERS 10 (Discount rate)	Report cost/benefit discount rates per national guidance
	No, they were not discounted, or they were discounted at a rate that is not appropriate for the setting, and the reason is not justified	Results should be considered carefully; they could be over- or under-estimated.
	I don’t know	Seek the help of an expert
Was DSA carried out?	YES, and reported in detail	Continue reading	CHEERS 20 (Characterizing uncertainty) CHEERS 24 (Effect of uncertainty)	Present DSA results and parameters variation (if applies)
	YES, but the reported detail is insufficient	View the results with caution, because you cannot estimate the impact of parameter variability on ICER/ICUR
	NO	View the results with caution, because you cannot estimate the impact of parameter variability on ICER/ICUR
	I don’t know	Seek the help of an expert
Was a PSA carried out?	YES, and reported in detail	Continue reading	CHEERS 20 (Characterizing uncertainty) CHEERS 24 (Effect of uncertainty)	Present PSA results; specify distributions.
	YES, but the reported detail is insufficient	View the results with caution, because you cannot estimate the global variability and robustness of results, and you cannot estimate acceptability based on your local WTP threshold
	NO	View the results with caution, because you cannot estimate the global variability and robustness of results, and you cannot estimate acceptability based on your local WTP threshold
	I don’t know	Seek the help of an expert
Was the acceptability curve presented?	YES	If you did not identify any critical flaws in the EE, you can interpret the acceptability curve	CHEERS 20 (Characterizing uncertainty) CHEERS 24 (Effect of uncertainty)	Present cost-effectiveness acceptability curves
	NO	You cannot know the probability of the intervention being cost-effective with respect to the comparator in your setting
	I don’t know	Seek the help of an expert

Assessing modeling methods: are they appropriate?

The key elements in any EE are the perspective of the analysis, the time horizon and discount rate, the population, and the comparators used. Each of these must be appropriately chosen to correctly estimate the results. In the following section, we go through the questions that must be asked to assess the methodological uncertainty of the model.

Was the perspective of the analysis appropriate? Are the results presented accordingly?

The perspective from which costs and benefits are estimated determines the choice of the resources included in the EE. The perspectives of the healthcare payer and the societal perspective are complementary, so ideally, the analyses resulting from the application of both perspectives should be presented separately (3,4,9,15-17). However, some exceptions can be made, provided they are consistent with the objective of the study.

Was the time horizon appropriate?

The analytical horizon for EEs should be long enough to account for differences in costs and consequences between the various strategies being evaluated (18). The time horizon chosen for an EE of influenza immunization should be at least one influenza season, approximately from October to April in the Northern hemisphere; however, 12 months is generally used, starting with the launch of the immunization campaign. This horizon accounts for direct costs and the immediate clinical consequences of influenza, which are usually quantifiable in the short term, while some benefits—particularly those related to avoided premature mortality—and indirect costs extend over the remaining lifetime of affected individuals. For this reason, although a one-year time horizon may be appropriate to capture the short-term costs and effects of seasonal vaccination, the consequences of prevented mortality beyond this period should be fully incorporated into the model results. Longer time horizons may still be required in more complex modeling approaches, such as dynamic models that track populations over time to capture the build-up of immunity and herd protection (6).

As for discount rates, while short-term costs are not discounted, long-term benefits associated with avoided deaths (in terms of life years gained [LYG], QALY, and productivity [indirect costs]) should be incorporated by assigning discounted lifetime pay-offs that reflect the full benefits at the standard rate accepted by each country, which constitutes the base case of the analysis (6).

In the case of Spain, this is 3% with a range of 0-5% evaluated in the deterministic sensitivity analysis (DSA) (4,19). The WHO-CHOICE (WHO- Choosing Interventions that are Cost-Effective) recommendations for the sensitivity analysis include using a 3% discount rate for costs and effects, and an alternative scenario with a 0% discount rate (6). It is also recommended that cost flows and health effects, both discounted and undiscounted, be presented separately and in detail whenever possible (4).

Figure 1 -. Decision diagram for selecting static vs dynamic modeling approaches (adapted from Soto et al. (14)).

Moreover, the question of whether differential discounting should be applied to non-monetary benefits (i.e., QALYs) at a lower rate than cost is currently under debate (20,21). Some countries, such as the Netherlands and Poland, already use differential rates, while others implement differential rates for time horizons above 30 years, e.g., France and Thailand (22). Some authors have proposed that the discount rate applied to benefits should be 2 to 5 percentage points lower than the discount rate applied to costs (20,21).

Was the target population adequately defined?

Populations must be chosen in accordance with the objective of the analysis. For example, for influenza vaccines in Spain, Spanish recommendations for influenza immunization (23,24) should be taken into account. Because not all available vaccines are targeted at all groups with a high risk of complications, the population should be consistent with the labeling of the assessed vaccines.

Were all the relevant comparators included in the analysis?

In general, an EE should at least compare the intervention under study with the standard of care (5,6). If a single standard of care has not been established, all relevant comparators should be evaluated pairwise. In the case of influenza, comparators should include all vaccines relevant for the target population and the setting if their cost-effectiveness relative to the intervention under study has not previously been clearly established.

Assessing the quality of the model input: were input parameters the most appropriate?

This section assesses parameter uncertainty, which can originate from the inherent variability of the outcome to be measured or the lack of knowledge (25). While relative vaccine effectiveness differentially impacts the branches of the comparison, all other variables are applied to all branches according to effectiveness. Thus, the uncertainty surrounding effectiveness directly contributes to the overall uncertainty surrounding cost-effectiveness results.

Was the burden of illness correctly estimated?

The burden of illness encompasses both the clinical burden of influenza —namely, influenza-related morbidity and mortality — and the associated economic burden, including direct healthcare costs and indirect productivity losses. In static models, this burden is represented by the distribution of the population across different health states (e.g., healthy, symptomatic, requiring general practice, requiring emergency department, requiring hospitalization, dead), as determined by the epidemiology of the infection and its consequences. Each health state is associated with specific health outcomes, such as QALYs, as well as direct and indirect costs. Therefore, the distribution of individuals across health states has a major influence on cost-effectiveness results (26). Given the inter-seasonal variability inherent to influenza epidemiology and its direct implications for the estimation of disease burden, guidelines (5,6) recommend estimating the burden of illness over at least 5 seasons, excluding pandemic events. These data may be easily obtained from each country’s national influenza surveillance system, so there is no reason to include fewer seasons in this estimate. The greater the number of seasons included, the greater the reduction of the uncertainty surrounding the results. Some studies have used up to 10 seasons, excluding the last pandemic season of 2009-10. Studies that include COVID-19 pandemic seasons should be carefully considered due to the presence of additional preventive measures. In particular, seasons affected by measures such as masking mandates, physical distancing policies, or other non-pharmaceutical interventions should generally be excluded from disease burden estimates, as these procedures substantially altered the transmission dynamics of respiratory viruses. When such seasons cannot be excluded, co-circulation of SARS-CoV-2 and its potential interaction with influenza activity should be explicitly addressed in the analysis. Furthermore, influenza infection must be laboratory-confirmed, and coinfection with COVID-19 must be excluded.

Was epidemiology correctly modeled in a dynamic model?

Dynamic models are usually used in influenza vaccination when the total population is divided into different health states with respect to infection, e.g., Susceptible, Exposed, Infected, Recovered (SEIR). Individuals move from one group to another based on a series of parameters such as vaccination coverage, vaccine effectiveness, attack rate, basic reproduction number (R₀), etc. All these parameters must be included in the model; however, they are often dependent on the characteristics of each influenza season and are not always available for the population under study. This requires assumptions to be made, which must be correctly justified. In addition, calibration of the model should be performed and reported to show how well it reproduces influenza epidemiology (15).

There are three main types of model validation: internal, cross, and external validity. In internal validation, mathematical calculations are examined through the verification of individual equations and their accurate implementation in code. By cross-validation, a model is compared with others addressing the same problem to evaluate consistency in results and understand the sources of any differences. External validation is used to compare model outputs with real-world event data by simulating actual scenarios using information such as population characteristics, treatment protocols, and outcome definitions (27). Table 2 presents a concise list of validations that should be applied to the influenza models.

Was the best available efficacy/effectiveness evidence used? If so, was it used appropriately?

Relative vaccine effectiveness is one of the main parameters affecting EEs; in fact, it provides the coefficient by which all benefit outcomes are distributed over the comparators. Therefore, various aspects must be considered:

WHO guidelines (5,6) discourage cherry-picking single studies in favor of systematic reviews of the available evidence. This may be done by conducting an ad-hoc literature review and meta-analysis or by sourcing a recently published one. In the first case, review methods should be transparently reported in the model documentation. In both cases, the A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR-2) (28) checklist can be applied as a guide to quality assessment. When data sources were selected using a targeted literature review, this method should be justified in the publication and taken into consideration by the reader. Additionally, attention must be paid to the characteristics of the meta-analytical results. In observational vaccine effectiveness studies, the design and methods are often heterogeneous, with no gold standard. Thus, when heterogeneity between studies is high, no conclusions should be drawn. In fact, the uncertainty introduced by heterogeneity between studies can mask differences that may exist, although available data are insufficient to detect them.

What is the quality and reliability of the selected source? Is it the best available source?

Not all evidence is equally robust and reliable. Therefore, the first step when considering EEs should be to assess the quality of the effectiveness sources alone and in comparison to other available sources. For transparency, the authors should disclose and thoroughly discuss both data source limitations and the justification for choosing one available source over others; however, this is often not the case. When the reader feels that the information reported in the publication is insufficient to determine the reliability of the data source, the original publication should be retrieved and analyzed.

According to the principles of evidence-based medicine, the strongest evidence comes from systematic literature reviews with meta-analyses of randomized controlled trials (RCTs). This is followed by single, well-designed RCTs, then observational or real-world evidence (RWE) with rigorous selection of participants, sufficient sample size, and follow-up based on a protocol that has been disclosed and approved before the start of the study (29). Grading of Recommendations Assessment, Development and Evaluation (GRADE) criteria help modify this rigid hierarchical structure by including more study characteristics than just the study design (29,30). However, observational studies should never be graded higher than RCTs. Using these guidelines, the best available evidence should be considered. The evidence quality pyramid is presented in Figure 2.

Table 2 -. Summary of internal, cross, and external validations that can be applied in influenza models

Internal validation

Checks that equations, parameter values, and coding are implemented correctly.

→ This validation includes verifying calculations of influenza-attributable ILI/SARI rates, ensuring consistent application of WHO-recommended vaccine effectiveness parameters, and confirming correct implementation of costs and discounting.

Cross validation

Compares model outputs with other influenza vaccination models or alternative disease-burden estimation methods to assess consistency in projected cases, hospitalizations, deaths averted, and program costs.

External validation

Assesses agreement with real-world data by comparing model predictions with observed influenza surveillance indicators (ILI/SARI), vaccine effectiveness estimates, program coverage, and healthcare resource use reported in WHO guidance and national data.

Figure 2 -. Integration of the evidence pyramid [Murad et al., (31)] and the GRADE criteria for evaluating evidence quality [Guyatt et al., (32)].

Only a few RCTs in influenza vaccines have been published so far, and most data come from seasonal influenza observational studies (33). Even though they are less robust than RCTs, meta-analyses of observational studies can offer more robust data than single observational studies, provided inter-study heterogeneity is low enough to allow reliable conclusions to be drawn. Nonetheless, whenever an RCT is available, it should be prioritized over a meta-analysis based on observational data, or at least. To appraise methodological robustness, an appropriate quality checklist could be used as a guide, e.g., Risk of Bias 2 (RoB2) (34) for randomized controlled studies, Risk Of Bias In Non-randomized Studies (ROBINS-I) (35) for non-randomized intervention studies, Newcastle-Ottawa Scale (36) for observational studies, etc. However, these are generic tools that focus on study design, and there are more specific questions to be evaluated when dealing with influenza vaccine effectiveness. One is the number of seasons included in the study: most studies include just one or two seasons. Because vaccine effectiveness depends greatly on the level of matching between vaccine composition and strain circulation in each season, this must be considered when interpreting results. Another relevant aspect is case definition. Most accurate studies define influenza cases by PCR laboratory tests; however, many studies, mainly observational, performed in the general practice setting use the influenza-like illness (ILI) definition, which may include patients with symptomatology similar to influenza but attributable to other respiratory viruses. Thus, absolute vaccine effectiveness in this case may be underestimated.

With all this information, the readers should be able to evaluate for themselves whether the selected evidence was actually the best available or whether the choice was biased. Table 3 shows a checklist for evaluating influenza vaccine effectiveness evidence.

Are utility values reliable?

A main limitation with EEs is the obtention of utility values. In general, the use of indirect measurement instruments, such as the EQ-5D (EuroQol 5 Dimensions) and SF-6D (Short Form-6 Dimensions), is recommended for the base case (19,37), with value sets derived from a representative sample of the general population to ensure greater applicability and comparability across studies. Direct methods (TTO [time trade-off], and SG [standard gamble]) (18) may be employed when their use is justified and appropriate. When population-specific estimates are unavailable, utility values can be obtained from studies conducted in comparable populations. Finally, if there is no empirical data, expert opinion may be used as a last resort. However, the uncertainty associated with these data would need to be evaluated in a sensitivity analysis (19). Since it is too burdensome to carry out a specific study to estimate utilities, most EEs take utilities from published studies. The utilities would be applicable if they are derived from instruments validated in the same or very similar population, using a representative sample of this group (19). Thus, authors should discuss the applicability of the available data when they are derived from different settings, populations, seasons, or case definitions in the case of influenza studies.

Table 3 -. Short checklist for evaluating influenza vaccine effectiveness: quality of evidence

1. Type of evidence and prioritization

☐ Are there RCTs?

→ Prioritize RCTs whenever available.

☐ If no RCTs are available: does the evidence come from observational studies or meta-analyses?

→ Meta-analyses of observational studies can be useful only if inter-study heterogeneity is low enough to draw reliable conclusions.

2. Heterogeneity between studies

☐ Does the meta-analysis report heterogeneity statistics? (I², τ², etc.)*

☐ Were influence/sensitivity analyses conducted?

☐ Does heterogeneity compromise the validity of the conclusions?

*Authors should follow PRISMA 2020 guidelines (41) for transparent reporting of synthesis methods, explicitly reporting random effects heterogeneity statistics (I², τ²) and conducting influence or sensitivity analyses where feasible.

3. Number of seasons included

☐ How many influenza seasons are included in the study?

→ 1–2 seasons = weaker evidence

→ Multiple seasons = preferable

☐ Was the degree of vaccine match/mismatch with circulating strains in each season considered?

4. Case definition

☐ Were cases defined by PCR testing? → higher accuracy

☐ Was a clinical ILI definition used?

→ Risk of underestimating VE due to the inclusion of other respiratory infections

☐ Is it explained how the case definition may affect reported VE?

5. Heterogeneity by subgroup and context

☐ Were differences in VE evaluated by: age, comorbidities, vaccine type (TIV/QIV/LAIV), geographic region?

☐ Was interannual variation in viral circulation, transmissibility, and prior immunity adequately considered?

Are the resource and cost estimates reliable?

Cost estimation must be coherent with the declared perspective of the analysis. When the societal perspective is considered, the costs assumed by the different payers must be reported separately (5,6).

Resource use is highly dependent on the structure of the local healthcare system, established care patterns, and clinical guidelines. In Spain, good burden of illness studies on influenza care are scarce; thus, resource use is often estimated from expert opinion, which is considered the lowest quality of evidence. Unit cost and resource use should be reported separately. As for the source of costs and use of resources, unit costs should preferably be obtained from official publications, the center’s own accounts, market prices, and ultimately, the fees applied to service contracts provided by the Spanish National Health System (SNS) (4). The main source of health resource unit costs is usually the official regional bulletin; however, costs may vary significantly from one region to another, and the choice of source bulletins, as well as the mathematical processing of data, should be adequately explained and justified (4); however, this is often neglected in EE reporting.

When the societal perspective is used, productivity loss costs should be accounted for. These may include sick leaves, disability, early retirement, disability and/or premature death, depending on the disease. The human capital approach and the friction cost approach are the two main methods applied to estimate time off work. The choice of method should be mentioned and justified. The cost of the hour of work lost due to the illness is based on the national average hourly wage, which, in Spain, is published by the National Institute of Statistics. Productivity loss should take into account an individual’s lost hours due to their own illness as well as the time taken off to care for other people (e.g., for children, when they are included in the target population). When considering the elderly population, productivity loss may not play a significant role; however, disability and loss of autonomy are increased in this population, generating a major social burden.

Sensitivity analyses—Control of uncertainty

Have deterministic and probabilistic sensitivity analyses been carried out? If so, have they been reported in sufficient detail?

Sensitivity analyses are crucial for the reader to appropriately understand the level and type of uncertainty involved in the model. They allow the reader to picture the variability around the ICER of the intervention vs the comparator and, consequently, the robustness of the estimation. DSA can determine structural and methodological uncertainty and also identify the main variables that have a significant impact on cost-effectiveness. It also represents the limits within which they can vary without changing the result (9). Probabilistic sensitivity analysis (PSA), meanwhile, accounts for parametric uncertainty and estimates the global variability of the model and the acceptability of the intervention compared to a willingness-to-pay threshold value for ICER (38). This analysis facilitates the interpretation of data from the national health system (Supplementary Figure 1 shows an example of cost-effectiveness acceptability curves for the EEs of various vaccination strategies in Spain, accompanied by their interpretation). According to the updated National Institute for Health and Care Excellence (NICE) methods guidance, the use of PSAs is not optional, but rather a formal requirement for any economic evaluation model submitted to the Institute (38).

Discussion

EEs are an essential tool for decision-making in influenza vaccination; however, the epidemiological and clinical particularities of influenza (for example, its seasonal variability, the heterogeneity of vaccine effectiveness, and the potential modification of population transmission) make economic modeling particularly complex. In this context, decision-makers must be able to critically interpret available EEs, though this is not always easy for those without a background in pharmacoeconomics.

This Guide aims to address this need by providing a practical framework to support non-expert readers in the field of health economics or influenza in critically reviewing influenza vaccine EEs. It integrates the experience gained from a critical review of recent Spanish EEs (7), in which we identified recurrent patterns of lack of transparency, insufficient justification of parameter selection, and limited control of uncertainty, which motivated the creation of this tool.

Unlike other methodological manuals, this document adopts a practical perspective centered on the needs of the non-expert reader, adding value in two main aspects: 1) addressing the critical elements of the EEs of influenza vaccines; 2) translating complex methodological concepts into an accessible format to assess the robustness of a study without the need for advanced knowledge in pharmacoeconomics and modeling.

Nevertheless, this Guide presents certain limitations. First, although its principles are generally applicable, its development has been based mainly on the appraisal of studies conducted in Spain and on the methodological requirements commonly used in this country. Despite this, it is important to note that the Guide relies substantially on international recommendations (i.e., the WHO guide), which provide a broader methodological framework beyond the national context. Also, the messages related to bias, uncertainty, and the inherent particularities of influenza are universal and transferable to other settings. In addition, the appraisal was limited to studies published up to 2022 and may therefore not capture methodological developments introduced in more recent evaluations. However, the challenges identified remain relevant as a framework for the critical appraisal of EEs.

Second, we have deliberately focused on complete EEs (i.e., cost-effectiveness or cost-utility analyses), as they allow a more comprehensive assessment of health benefits for decision-making. For this reason, other approaches based exclusively on costs, such as budget impact analyses, have not been included in detail. We recognize, however, that EE results should not be the sole input into reimbursement decisions. Instead, they should be combined with evidence from budget impact analyses to assess whether vaccines not only offer good value for money but are also affordable (39).

Regarding future research directions, it would be desirable to expand this Guide by incorporating specific examples for different target groups (children, pregnant women, chronic patients, older adults), as well as harmonized recommendations for selecting and justifying key parameters, particularly regarding vaccine effectiveness and disease burden. Finally, promoting good practices in the use of dynamic models and in model calibration and validation would help strengthen the quality of the pharmacoeconomic evidence available on influenza.

Conclusions

In summary, this Guide offers a practical resource for the critical reading of EEs of influenza vaccines. It is aimed at non-expert readers in health economics or modeling, as it presents complex methodological concepts in an accessible format that may help them identify sources of uncertainty and assess the reliability of the evaluations’ results. By doing so, it can support evidence-based decisions and ultimately improve the efficiency of influenza vaccination programs.

Other information

This article includes supplementary material

Corresponding author:

José María Abellán Perpiñán

email: dionisos@um.es

Acknowledgements

The authors would like to thank Maria Giovanna Ferrario, PhD, Blanca Piedrafita, PhD, Lucía Pérez-Carbonell, PhD, and the whole team at Medical Scientific Consulting, SL (Valencia, Spain) for their support in the development and writing of this study.

Disclosures

Conflict of interest: ROL has received fees for academic services and grants to attend international meetings from Abbott, CSL, GSK, Moderna, MSD, Novavax, Pfizer, Roche and Sanofi. JDD reports personal fees/grants from GSK, Sanofi Pasteur and MSD and nonfinancial support from Sanofi Pasteur and MSD. AGM has received grants paid to his institutions by Sanofi, GSK, and Pfizer. He has also received personal consulting fees from Sanofi, MSD, and Moderna, as well as for lectures and congress presentations from GSK, HIPRA, Astra-Zeneca, Novavax, Pfizer and Jansen. He has received support for attending meetings/travel from Pfizer, MSD, GSK, and Sanofi. Finally, he declares having participated in safety monitoring boards for Pfizer, Hipra and Novavax. ERM has participated in advisory boards, conferences, courses and lectures organized by Glaxo SmithKline, Sanofi Pasteur, MSD, GSK, Seqirus, Pfizer, Moderna, Takeda, and AstraZeneca. She has also participated in data safety or advisory boards from Pfizer, Sanofi, MSD, Moderna, Takeda and GSK. FMT has received honoraria and/or reimbursement for participation fees/travel expenses from GSK, Pfizer Inc, Sanofi Pasteur, MSD, Seqirus, Biofabri, and Janssen for taking part in advisory boards and expert meetings and for acting as a speaker in congresses outside the scope of the submitted work. He has also acted as principal investigator in randomized controlled trials sponsored by the above-mentioned companies as well as by Ablynx, Gilead, Regeneron, Roche, Abbott, Novavax, and MedImmune, with honoraria paid to his institution. JMA declares having received consulting fees from Sanofi. JLLB, ADA, PIPJ and MFA are employees of Sanofi and may hold shares and/or stock options in the company.

Financial support: This study was funded by Sanofi. Medical writing was provided by Medical Scientific Consulting, SL (Valencia, Spain) and funded by Sanofi.

Author contribution: All authors equally contributed to the conceptualization, supervision, writing – original draft and writing—review & editing. All authors have read and approved the final manuscript.