Evaluation of the SafeVac 2.0 study on the question "Are there any batch-specific increases of reports of suspected side effects for batches of the COVID-19 mRNA vaccine Comirnaty?"
As of 18 August 2023
The Paul-Ehrlich-Institut cannot confirm any batch-specific increases of reports of suspected vaccine side effects after COVID-19 vaccinations with the mRNA vaccine Comirnaty (BioNTech/Pfizer) based on the analysis of the data from the prospective observational study with the SafeVac 2.0 app. A disproportionate increase of the number of adverse events reported in relation to certain Comirnaty batches used in Germany, as stated by the authors of a published research letter with data from Denmark (Schmeling et al. [1]), is not discernible in the evaluation of the SafeVac 2.0 data for the entirety of app-reported adverse events nor specifically for serious adverse events.
The Paul-Ehrlich-Institut records all cases of suspected adverse events or vaccination complications submitted within the spontaneous reporting system after vaccination. The Institute continuously analyses this data with regard to the benefit-risk ratio of the authorised vaccine products. The batch name is requested when reporting a suspected adverse event after vaccination, but it is not mandatory for the validation of a report and inclusion in the evaluation. It is therefore methodologically questionable to evaluate the number of reports of suspected adverse events in the spontaneous reporting system for the purpose of investigating a connection between an increased number of reported suspected cases and a certain batch of COVID-19 vaccine. There is no such uncertainty of methodology in the evaluation of the suspected case reports carried out by the Paul-Ehrlich-Institut using the SafeVac app, as in this case the suspected case reports are bindingly linked to the batch number of the administered vaccine dose.
As part of the preparations for the national vaccination campaign with COVID-19 vaccines, the Paul-Ehrlich-Institut implemented a prospective observational study (SafeVac 2.0 study) in addition to recording suspected case reports submitted via the spontaneous reporting system, in which the safety and tolerability of the authorised COVID-19 vaccine products used in Germany were monitored. The end of the recruitment phase for participation in the SafeVac 2.0 study was 30 September 2022.
SafeVac 2.0, the prospective observational study on vaccine safety, was a high-quality and controlled recording of the safety profile of the authorised COVID-19 vaccine products used for vaccination. The study was carried out by periodically requesting information from the participants on adverse events they noticed after COVID-19 vaccination over a period of one year after vaccination. Participants were asked at predefined intervals about their state of health or symptoms. This minimised any reporting bias.
In order to participate in the SafeVac 2.0 study, each participant had to provide a valid batch number of the COVID-19 vaccine product used for their vaccination. This information was used to check whether the batch was valid and had been authorised at the specified time of vaccination. These two parameters, which were recorded in a controlled manner in the SafeVac 2.0 study, can be used to determine in a reproducible and valid manner how many adverse events were reported after vaccination with a particular batch of a particular COVID-19 vaccine product.
In light of the above, the Paul-Ehrlich-Institut conducted an analysis of the data collected in the SafeVac 2.0 study on the question of whether there was an increase of the number of reported adverse events and their severity for certain released batches of the COVID-19 mRNA vaccine product Comirnaty.
A total of 734,394 people enrolled in the SafeVac 2.0 study and provided at least one valid vaccination batch number. In addition, information from 445,483 participants who had registered the receipt of a second dose in the SafeVac 2.0 app was evaluated. Thus, data from a total of 1,179,877 vaccinations with vaccine doses from 401 different vaccine batches could be evaluated. A total of 5,074,069 adverse events after 1,179,877 vaccinations were reported using the SafeVac app.
The frequency of reported adverse events was stratified by COVID-19 vaccine product and batch for comparison with the evaluation made using data from Denmark in the publication by Schmeling et al. [1].
A total of 244 different Comirnaty batches were registered in the SafeVac app study in connection to 703,164 vaccinations with Comirnaty (first and second doses).
3,061,920 adverse events were reported after these 703,164 vaccinations. A SafeVac app report can include multiple adverse events. There were also reports stating that no adverse event occurred. One of the goals of the survey was to determine to what extent vaccination was tolerated without physical reactions.
The straight line of dots in Figure 1 show that the number of adverse events reported via the SafeVac 2.0 app correlates with the number of vaccine doses from a given Comirnaty batch.
Figure 1: Number of vaccine doses administered to SafeVac 2.0 study participants of a specific Comirnaty batch and number of adverse events reported via the SafeVac 2.0 app after vaccination with a dose from that batch (as of 30 June 2023). All adverse events reported per batch via the SafeVac 2.0 app were counted here, regardless of whether these events were perceived as serious or not.
Source: Paul-Ehrlich-Institut
There is therefore a linear relationship: the more vaccinations with vaccine doses from a certain batch, the more adverse events were reported via the SafeVac 2.0 app. This means that a disproportionate number of adverse events was not recorded for any batch and that no batch stood out in a negative manner.
An evaluation was also made of SafeVac 2.0 app reports in which the participants had classified an adverse event after Comirnaty vaccination as serious from their personal point of view. In addition, all reported adverse events that had been previously identified by the EU medicines authorities as an adverse event of special interest (AESI) were taken into consideration, even if the participant had not assessed this event as serious.
A total of 3,935 Comirnaty vaccinations (first and second vaccinations) could be assigned to 137 different Comirnaty batches in which at least one adverse event, which was either classified as serious by the vaccinated person or considered an event of special interest, was reported.
After the 3,935 Comirnaty vaccinations, 33,874 serious adverse events, including events of special interest, were reported based on the above definition.
Figure 2 shows the correlation between the number of adverse events reported for a particular Comirnaty batch, which was perceived as serious or classified as an AESI, and the number of participants vaccinated with a dose of the respective Comirnaty batch.
Figure 2: Number of vaccine doses administered to SafeVac 2.0 study participants of a specific Comirnaty batch and number of adverse events reported by the SafeVac 2.0 app after vaccination with a dose from that batch and perceived as serious (as of 30 June 2023). Each dot represents the number of doses vaccinated in a given batch and the number of adverse events reported in that batch that were considered serious.
Source: Paul-Ehrlich-Institut
Here, too, the linear relationship is evident: the more vaccinations that were carried out with one batch, the more serious events or events of special interest were reported via the SafeVac 2.0 app. Thus, taking into account the number of vaccine doses per batch, there was no batch for which a disproportionate number of adverse events were recorded.
It is important to note that symptoms reported in the SafeVac 2.0 study are only initially recorded as adverse events. The number of serious adverse events reported in the SafeVac 2.0 app study is based on the self-assessment of the person reporting via the SafeVac 2.0 app. These self-assessments may be corrected following medical validation and registration as suspected case reports by the experts of the Paul-Ehrlich-Institut.
One limitation that should be mentioned is that the number of adverse events reported via the SafeVac 2.0 app is too low to detect any risk signals related to a single batch. However, the question at hand, taking into account the different number of vaccine doses per batch, can be decisively answered: there were no discernible batch-specific increases of adverse events reported via the SafeVac 2.0 app.
In June 2022, the United Kingdom Medicines and Healthcare Products Regulatory Agency (MHRA) published its evaluation of its spontaneous reporting scheme (Yellow Card) and the resulting response to the question of a link between suspected adverse event reports and certain batches of COVID-19 vaccine as part of a Freedom of Information Act request. These evaluations did not reveal any safety concerns related to individual batches.
The research letter "Batch dependent safety of the BNT162b2 mRNA COVID-19 vaccine" by Schmeling et al. [1] published on 30 March 2023 describes indications of a batch-related accumulation of suspected side effects after vaccination with the mRNA vaccine product Comirnaty in the data of the spontaneous registration system of the Danish Medicines Agency. However, this analysis was not carried out on behalf of the Danish authority.
In their publication, Schmeling et al. [1] state that, based on the analysis of the reported suspected cases of adverse events after Comirnaty vaccination, there are significant differences in the number and severity of adverse events reported per Comirnaty vaccine batch. The basis for this statement is an evaluation of the reported suspected cases of side effects (spontaneous registration system) after Comirnaty vaccination in Denmark.
Approximately 80 percent of the reported adverse events presented in the evaluation are "non-serious" and relate in particular to the reactogenicity known from the clinical trials and the typical short-term vaccination reactions that subside within a few days after vaccination without any long-term damage.
The Paul-Ehrlich-Institut would like to point out that publications in the form of a research letter describe individual research results, observations, or field reports. In terms of the level of detail and depth of evaluation, research letters are not comparable with original scientific papers, in which original research results are presented.
There are methodological deficiencies that severely limit the significance of the results in the published evaluation by Schmeling et al. [1].
As an example, in the analyses of the reported suspected cases of adverse events, there is no stratification by
These parameters would be essential for assessing the reported statistical correlation.
In a retrospective evaluation of reported suspected side effects after vaccination, the potential for a high level of variation in the number of reported events per period, caused by various influences such as increased media attention, must be taken into account.
In a large-scale, population-wide vaccination campaign extending over a long period of more than two years, as was the case in the SARS-CoV-2 pandemic, individual batches of vaccines are available for different lengths of time. Possible reasons for the variation in availability are: the number of doses per batch, which may vary depending on the place of manufacture; changes in vaccination recommendations; the availability of different vaccine products, which are approved on a rolling basis; and a constantly shifting infection situation. If certain events are reported in the media on the basis of the number of cases at times when a large number of vaccinations are being administered, this may lead to a distortion of the situation, in particular with regard to the frequency of reporting such events. One element not taken into account in the evaluation by Schmeling et al. [1] was that booster vaccinations may be associated with an increased rate of adverse events due to the associated renewed stimulation of the already activated immune system. Some batches of COVID-19 vaccine have been used for both primary and booster vaccinations.
Further comments on methodological weaknesses of the publication and the analysis carried out by Schmeling et al. [1], which were also identified by the Paul-Ehrlich-Institut, were published by the same journal in another issue. [2]
[1] Schmeling, M et al.: Batch-dependent safety of the BNT162b2 mRNA COVID-19 vaccine European Journal of Clinical Investigation 2023; 53(8) e13998
https://doi.org/10.1111/eci.13998
[2] Borja Somovilla del Saz: Batch-dependent safety of the BNT162b2 mRNA COVID-19 vaccine European Journal of Clinical Investigation 2023; epub ahead of print e14050
https://doi.org/10.1111/eci.14050
Updated: 18.08.2023
