Efficacy and effectiveness are relevant concepts in health science research that are sometimes used synonymously in the public debate on COVID-19 vaccines. “Effective in the flask, effective in the live” is a simple rule of thumb to remember the meaning of these terms.
To evaluate whether a vaccine prevents infection or disease, it is necessary to design a study that makes it possible to compare two population groups with characteristics that are as identical as possible. In other words, the only difference is getting the vaccine.
A randomized clinical trial is generally used. In it, a group of people is administered the vaccine and another a placebo, and they are followed for a time to observe the effects of the vaccine. In this way, it could be concluded that the observed effects respond to immunization. In this case, the result would speak of its effectiveness, as it refers to its impact under ideal and controlled conditions.
However, the main objective of a vaccine is its application to the population. When evaluating its impact once it is marketed and administered, we will talk about vaccine effectiveness: we want to measure its impact in real conditions.
In this case, vaccinated and unvaccinated groups are also followed and compared, but under these real conditions their social and demographic characteristics may differ. Vaccinated people are not the same as unvaccinated people. This makes it difficult to know if one group has been more exposed to the virus than the other.
It might seem that the differences between efficacy and effectiveness are minimal, but it is important to use these concepts correctly because they provide different information about the methodology that has been followed for the analysis. Therefore, the considerations to be taken into account when interpreting the results are also different.
Assessing effectiveness is difficult
Following the publication of the results of the clinical trials of the currently marketed SARS-CoV-2 vaccines, important questions remained to be answered from an epidemiological point of view:
- How will they work when they start to be administered to the general population?
- How are we going to evaluate it?
One of the main problems arises when it comes to determining whether the groups to be compared (vaccinated and unvaccinated) resemble each other. In this case, we are faced with two dynamic populations, one of which –the group of vaccinated people– individuals are joining as the vaccination campaign progresses.
For this reason, it is necessary to reflect on the possible differences that may exist between the two groups, and that may influence the analysis. This is especially the case when vaccination coverage, either in the general population or by age group, is low or very broad.
We are going to give two hypothetical examples that could distort the effectiveness data:
When the vaccination campaign begins
At the beginning of the campaign, people most at risk of the consequences of the disease are vaccinated (for example, the elderly or with other pathologies), who are likely to adopt more protective measures against exposure to SARS-CoV-2 . Due to this, the vaccinated population would present a lower risk of contagion due to vaccination, but also to their behavior.
This, when carrying out the analysis, could result in an overestimation of the vaccine effectiveness in the initial phases.
When the vaccination campaign is progressing
On the contrary, as the vaccination campaign progresses, it could happen that some vaccinated people vary their behavior due to a feeling of security (or because they have a certificate that allows them to carry out certain activities only to that group).
Those vaccinated, feeling protected, could carry out more risky activities and modify their risk of infection. The result is that they would be more exposed to the virus compared to the unvaccinated, which would result in an underestimation of the effect of the vaccine.
Another circumstance that could occur is that those vaccinated would go less to the health system when presenting a mild or asymptomatic picture of COVID-19, which would result in an underreporting of mild cases among vaccinated people.
This could skew comparisons of severe COVID-19 cases between vaccinated and unvaccinated people.
Cannot analyze effectiveness by looking at graphs
The use of hospital data to report vaccine effectiveness has been recurrent since its administration became general. The exclusive use of this type of data has important limitations.
One of the key ideas is that the vaccine does not change the risk pattern for severe covid-19. In other words, older people are still the ones most likely to be admitted for this pathology.
In a context of more than 90% of the population over 40 years of age with complete regimenIt is expected that there will be a non-negligible percentage of people admitted with a full schedule since, in absolute terms, there is less vulnerable population without vaccinating.
In this article On the Israeli data on hospitalizations and vaccines, two basic considerations are detailed when interpreting the data:
- When presenting differential risks for a severe form of the disease based on age group, comparisons should be made between vaccinated and unvaccinated groups with similar risks.
- It is necessary to relate the cases to the total of the vaccinated and unvaccinated population. Both populations are dynamic, and their size varies as the vaccination campaign progresses.
This table from the article summarizes how the calculation of effectiveness (even if it is listed as “efficacy”) Varies when we incorporate these factors into the analysis:
Are vaccines losing effectiveness?
In recent weeks, data have also circulated on the loss of vaccine protection over time, comparing the first vaccinated groups against those who received the vaccine more recently.
If we think about possible differences between the vaccination groups, older people were the first to be vaccinated and they usually have a decreased immune response. Therefore, when vaccination is generalized, it would be expected that they could present a higher risk of contagion compared to younger groups vaccinated later.
In addition, we must bear in mind that the presentation of this information influences its interpretation. When this apparent difference in contagion risk is reported in relative terms, they can magnify a minimal difference between groups that have a reduced risk of contagion in absolute terms.
There are other possible biases to take into account, such as how transmission varies depending on geographic location and possible differences in populations by territories. Also the variation of the transmission as a function of time for the period we want to analyze.
Incorporate complexity without going into paralysis
Obtaining all the necessary evidence from data obtained under ideal conditions is not feasible in practical terms, nor is it sometimes desirable from an ethical point of view, especially in the field of public health. This fact cannot lead to inoperation and paralysis either.
Therefore, it is necessary to combine information from different types of studies in order to make the most informed decisions possible. However, the urge to understand reality cannot mean ignoring the existence of biases and factors that complicate the interpretation of real-world data.
Rigor should not be replaced by quick analysis based on screenshots of graphs that are not accompanied by information on the methodology that has allowed them to be obtained, as has happened in recent weeks in Israel.
It is necessary to embrace complexity to advance knowing the ground we are treading.
** This article was originally published on The Conversation by Mario Fontán Vela and Pedro Gullón Tosio, PhD student in Epidemiology and Public Health, University of Alcalá and assistant professor doctor in Public Health, University of Alcalá, you can read it in full here.