Efficacy, Effectiveness, and the Prevalence of Vaccinated among the Hospitalized: Part 1

There has been a lot of angst, recently, over how many vaccinated people are coming down with Covid, even to the extent of being hospitalized.  A lot of people are coming to the conclusion that vaccine effectiveness has been waning--either because the new Delta variant is escaping suppression, or because the vaccines are losing effectiveness over time, or both.

There is some truth to those fears, for sure.  However, I believe that neither fear plays as much a role in uncomfortable numbers of vaccinated people getting sick as a lot of people think.  A third factor that you need to consider when you look at the number of people who are vaccinated who are also getting sick or being hospitalized is the difference between vaccine "efficacy" and vaccine "effectiveness".

I am going to explain what the difference is between these terms and why it matters in Part 1.  In Part 2, I am going to look at an important case study in which this distinction might be very important.

Efficacy vs. Effectiveness

These two words--"Efficacy" and "Effectiveness"--are technical terms in immunology.  Briefly, "efficacy" refers to how well a vaccine reduces disease in an ideal, properly balanced clinical trial.  "Effectiveness" refers to how well a vaccine reduces disease in the real world, given a distribution of recipients that is limited by real world constraints rather than the artificial constraints of a clinical trial.

When a vaccine (or any other medication, for that matter) is tested in a clinical trial, the goal is to identify how well the vaccine works compared to non-vaccination, all other things being equal.  A well-designed clinical trial will identify all characteristics of a subject that might have an effect on the outcome of the trial.  The trial runners will then balance the trial vs. the placebo groups so that an equal proportion of each characteristic appears in each group.  That is, both the placebo and trial groups should contain the same proportion of elderly to young people, of each race and gender, of sickly vs. healthy people, and so forth.  If there is a certain population that is more (or less) susceptible to the disease in your trial, and you do not balance that population properly between your trial and placebo groups, then your final results will be biased by the properties of this particular population.

If a clinical trial is properly balanced, then you can take the difference between how many people get sick in the vaccinated group vs. how many get sick in the unvaccinated and calculate your vaccine *efficacy*.

In the real world, however, there is no one balancing out the vaccinated vs. the unvaccinated populations.  People get vaccinated or not for reasons other than balanced, random chance.  Therefore it is quite possible (and actually inevitable) that, in the real world, the group of people who are vaccinated will have different characteristics from the group of people who are unvaccinated.  This means that you should not expect that the difference between the vaccinated and the unvaccinated who get sick in the real world to reflect the same efficacy as was found in the clinical trials.  What you get from this calculation is the *effectiveness*, and this can be skewed from the "efficacy" number for a lot of reasons.

Risk Averse Behavior

The most critical reason effectiveness can be skewed from efficacy (or at least the reason that *I* think is the most important) is the very nature of human behavior relating to risk.  Simply put, it is pretty obviously true (if you think about it), that those people who are more naturally at risk from a disease will choose to take a vaccine against that disease at a much higher rate compared to people who do not feel similarly at risk.  Given that this is true, you should expect, in the real world, that the group of all vaccinated people will contain many more naturally at-risk people that the group of all unvaccinated people.

How large of an effect will this have?  Well, it could actually have quite a large effect, depending on the specifics.  I think there are, broadly, two different types of person this distinction applies to, which I will now describe.  Then I will try to estimate the magnitude of this effect with some reasonable guesses and a spreadsheet.

At-Risk of Exposure

One type of person is more at risk of catching Covid because of an occupation.  This type of person may have a job that brings him into close contact with a lot of potentially sick people on a regular basis.  The classical example here is health care workers; they are obviously exposed to Covid (at least potentially) a lot more than other people are.  Now, some early studies showed that health care workers weren't getting sick all that more often than other people (see Prevalence of SARS-CoV-2 Infection Among Health Care Workers in a Tertiary Community Hospital)--but those studies were done during a time in which rather extreme Personal Protective Equipment (PPE) practices were in place.  Nowadays, practice has tended to relax a lot further and we are looking at close to business-as-usual behavior.  So I think it is fair to say that healthcare workers will very likely be exposed to a lot more Covid virus than other people.

For this type of person, both natural immunity and vaccination would be expected to work as well for them as for anyone else.  However, given that they will be exposed so much more, the vaccination will be given much more opportunity to fail for these people than for others.

At-Risk of Infection

Another type of person is not necessarily more exposed to Covid, but might be more naturally at risk of catching the disease when they are exposed.  The most important group here, I think, is the immunocompromised.  This is a larger group of people than most realize: it includes the HIV positive, people on a whole array of immunosuppresive drugs, people undergoing cancer treatment, and people with Down syndrome--to name just a few.  Randomized surveys have shown that a full 2.7% of Americans at any given time have been diagnosed as immunosuppressed. (Prevalence of Immunosuppression Among US Adults, 2013).  And I think this number might be understating the total effect, because there are certain conditions that typically don't get you outright diagnosed as "immunocompromised", but which actually do have some compromising effects on immunity.  The two most important of these conditions are age and obesity, both of which are known to exert some suppressing force on immunity (see Impact of Obesity and Metabolic Syndrome on Immunity) and which are together the two most important "comorbidities" for Covid.

How much effect immunocompromising conditions might have on the effectiveness of vaccines is not completely known--but it might be rather severe.  The Novavax study in South Africa included a cohort of known HIV positive test subjects in its vaccine trial.  The sample size was not large--which makes its results here uncertain because of lack of statistical power--but insofar as the results are reliable, they indicate that the Novavax vaccine was not at all effective in the HIV cohort (Efficacy of NVX-CoV2373 Covid-19 Vaccine against the B.1.351 Variant).  The trial found the vaccine pretty effective for other people, but did not see any effectiveness for the HIV positive--in fact, slightly more vaccinated HIV subjects came down with Covid than unvaccinated HIV subjects.

Immunocompromised also have a two-fold problem: not only are they more prone to catching a disease, since their bodies are not well equipped to fight a disease once it gains a foothold, they are much more likely to be hospitalized from a disease once they catch it.

How much of an effect could these special cases have?

But, you might say, even if the vaccines don't work very well for the immunocompromised, that's only roughly a 3% portion of the population.  That couldn't mess up the numbers that badly, could it?  Yes, it could.  Covid attacks people very unevenly; the more vulnerable people aren't just a little bit more likely than everyone else to suffer badly from Covid, they are far more likely to suffer badly from Covid.  This uneven distribution of adverse effects means that even a pretty small population of very vulnerable people can have a very substantial effect on the makeup of the total hospitalized and dead.

To illustrate this, I have put together this spreadsheet with what I considered simplified but still reasonable numbers, here: Efficacy vs. Effectiveness

In this spreadsheet, I setup a group of 1 million people, and then determine who gets sick and who gets hospitalized based on various parameters.

Here are the assumptions I make for this spreadsheet:

• The average person will have a 1% chance of catching Covid.
• 5% of the population will be in "high exposure" occupations, which I set to 4x the regular exposure to the virus.
• 3% of the population will be immunocompromised.
• The immunocompromised will be 2x as likely to catch the disease compared to everyone else (a guess, but I think reasonable based on my experiences with an immunocompromised daughter).
• The vaccine will be 88% efficacious for normal people.  
• It will be 0% efficacious for immunocompromised people.  (This is consistent with the Novavax trial results even if not proven by them.)
• 5% of unvaccinated people with functional immune systems who catch Covid will end up hospitalized
• The vaccine will be 96% effective at keeping people who come down with Covid from being hospitalized.
• 40% of immunocompromised who catch Covid will end up being hospitalized.
• 80% of people in high-exposure occupations will be vaccinated.
• 90% of immunocompromised people will be vaccinated.
• 60% of everyone else will be vaccinated.

These are all reasonable assumptions.  They may not be quite accurate and some of them are fairly bald guesses, but nothing here is outlandish and nothing contradicts anything I actually know to be true.

With these assumptions, what you will see as far as numbers of actual infections and hospitalizations is that 1394 people who are vaccinated will become infected, whereas 4140 who are unvaccinated will be.  This translates to an effectiveness of approximately 66%.  Meanwhile, 218 vaccinated people will be hospitalized, whereas 228 unvaccinated people will be--which makes it seem as if the effectiveness of the vaccine at preventing hospitalization is basically 0%.

And remember--these results are assuming a very robust protection from the vaccine for 97% of the population: 88% protection against infection and 96% infection against hospitalization over and above that protection.  So we have a seemingly paradoxical result that even with a vaccine that is highly effective, as many vaccinated as unvaccinated are seen to be suffering badly from the disease.

Bottom line conclusion: It is very easy for important sub-groups of your population to completely throw off your bottom-line numbers if you do not interpret the results correctly.  So make sure you understand the sub-groups of your population and account for their characteristics in any study that you do.

In the case of how many people are vaccinated yet also hospitalized, we need to pay careful attention to what other risk factors these people have.  It is not necessarily the case that high numbers of vaccinated people in the hospital implies inefficacy of the vaccine.