Efficacy, Effectiveness, and the Prevalence of Vaccinated among the Hospitalized: Part 2

Now let's look at an example case in which the efficacy of the vaccines in preventing infection and hospitalization has seemed to be inadequate.  It has been reported (see This 900-person delta cluster in Mass. has CDC freaked out—74% are vaccinated) that an outbreak of Covid in Barnstable County, Massachusetts was a key datapoint in the CDC reversing its recommendation on mask wearing for vaccinated people.  The CDC report on the outbreak is available here, and the key worrisome facts about this outbreak is that a full 74% of the people who became sick were vaccinated, 4 out of the 5 people who were hospitalized were vaccinated, and the Ct values (i.e., roughly how many particles of virus were found from nasal pharygeal swabs of the infected) of the vaccinated and non-vaccinated were "similar".

These numbers do make it seem, on the face of it, that the vaccines aren't doing very much to limit infection spread among the vaccinated.  However, as we saw in the previous post, it is possible for those overall numbers to be misleading, especially if there is a chance that a significant proportion of those people who became infected or hospitalized were immunocompromised.  And it turns out that this is likely to be the case.

The nature of the outbreak in Barnstable County

According to the CDC paper, the outbreak in Massachusetts was the result of "multiple summer events and large public gatherings were held in a town in Barnstable County, Massachusetts, that attracted thousands of tourists from across the United States".  The Massachusetts Department of Public Health, when they interviewed people associated with the outbreak found that people "reported attending densely packed indoor and outdoor events at venues that included bars, restaurants, guest houses, and rental homes."  But the CDC paper also contains this line in the discussion on limitations of its findings at the end of the paper: 

Third, demographics of cases likely reflect those of attendees at the public gatherings, as events were marketed to adult male participants; further study is underway to identify other population characteristics among cases, such as additional demographic characteristics and underlying health conditions including immunocompromising conditions.

What sorts of events are "marketed to adult male participants"?   It turns out that Barnstable County, MA, is the most popular summer vacation destination on the East Coast for LGBT+ vacationers.  It has the highest rate of gay marriage in the entire country.  In the middle of July, they host something called "Bear Week", which is essentially a week-long party for gay men.  During this period, the town population increases 20-fold (from 3,000 people to as high as 60,000) as a result of out-of-town LGBT+ vacationers.  (Wikipedia: Providencetown, Massachusetts)

Neither the CDC report nor the Massachusetts July Covid Update (July 30, 2021 | Update: COVID-19 Cluster in Provincetown) say how many of those who fell sick were gay men.  However, the Massachusetts report *did* show that a full 89% of the people who were sick were male, mostly young.  While Covid has been shown to impact men more seriously than women if they get sick, both genders will get sick at approximately the same rate.  For the natural balance to be upset to that degree, something like 728 of the 934 people who got sick (or 78%) would have to have been gay men.

The CDC report does mention a fairly small number of the patients (30) from the outbreak whom they had already confirmed had an HIV diagnosis--but this was mentioned as being a *preliminary* finding, and they only confirmed by cross-indexing the Massachusetts index of people registered with HIV.  Given that most of the people who got sick were from out-of-town, and given that registering as HIV positive is optional, this number is certain to be too low.

In fact, it's been found in random surveys that more like 1 in 5 of young gay men who frequent bars have HIV, and about half of those people don't even know it yet: (cf: 1 in 5 Gay/Bi Men Have HIV, Nearly Half Don't Know).  So this means that something more like 146--not 30--of the patients in the outbreak are likely to have been immunocompromised.

How many vaccinated people in Barnstable should we have expected to be hospitalized?

The answer to this question depends very much on how many of the out-of-town vacationing gay men were fully vaccinated.  I would *think* that this number would be very high.  If I were a gay man looking to party for a week in crowded settings with other gay men, I would want to make sure I were vaccinated ahead of time.  So I think that my estimate of 90% vaccination rates from Part 1 seems very reasonable.  I admit; this is a guess.  But let's assume this is true for now.

In Part 1, the various multipliers for hospitalization that we came up with work out to a 450x greater likelihood of hospitalization for immunocompromised vaccinated individuals compared to immunocompetent vaccinated individuals.  If you applied this multiplier to the 146 likely immunocompromised people in Barnstable, then you would expect to see about 80 vaccinated patients hospitalized from that group for every single patient hospitalized outside that group.  The actual proportion of vaccinated to unvaccinated was 4 to 1.  So based on my numbers from Part 1, the vaccines seem to be working more effectively than I would have expected.

The difference is so great, in fact, that I suspect my assumptions from Part 1 are wrong.  In particular, I think that my assumption from the South African Novavax study--that the vaccines would be not effective at all for the HIV positive--might be incorrect.

If you expect 80 people to be hospitalized, but only 4 are, this translates to a vaccine efficacy of about 95%.  And indeed, this is roughly what the CDC has been reporting--that the vaccines remain something like 95-96% effective in preventing hospitalization.  So I see the numbers of hospitalized in Barnstable, even given how many immunocompromised were likely in that population, as tending to confirm the efficacy of the vaccine--as far as it goes.  There will still be more immunocompromised people being hospitalized from Covid than immunocompetent, but if you compared immunocompromised to immunocompromised only, the vaccinated will have roughly the same comparative advantage against the unvaccinated.

How many immunocompromised would we have expected to get sick?

So much is good for the issue of hospitalization.  However, the CDC did not look at the Barnstable results and say that the vaccines weren't preventing hospitalizations: they worried (apparently) that the vaccines weren't preventing infection and spread.  So let's look at the reported numbers of the symptomatic infected more closely.  

I estimated in Part 1 that immunocompromised people were twice as likely to catch a disease (at least, symptomatic disease) in the first place compared to the immunocompetent.  This, however, was just taking into account their diminished capacity to fight off a disease rapidly.  In the case of the Barnstable County outbreak, you also have a large group of people that includes (most likely) many immunocompromised people who are also engaging in much higher risk behavior: packing themselves into crowded bars and restaurants for a week-long party.  In this specific scenario, what effectiveness of the vaccine for the immunocompromised would have to obtain in order for the vaccinated to make up 74% of the infected?

I have put together another section of the spreadsheet (link again here: Efficacy vs. Effectiveness) that attempts to model this scenario.  The key assumptions for the Barnstable County outbreak that I am making are as follows:

• 50,000 visiting LGBT+ vacationers.
• 90% of visiting LGBT+ vacationers are vaccinated (compared to 69% of the locals, which was the value reported).
• A baseline vaccine efficacy of 80%
• A reduced vaccine effectiveness for HIV positive people of 50% the baseline (i.e., 40% in this case)
• A 2x multiplier for HIV positive people to come down with symptomatic Covid if they are infected.
• An 8x "exposure rate" for the vacationers compared to the locals to account for the crowded party activities.

Given these assumptions, I can come up with a percentage of the vaccinated among the total infected of 73%, which is right in line with what actually happened.  In other words, it is not necessary to conclude that vaccine effectiveness at preventing disease spread has fallen very far at all in order to see large numbers of vaccinated become ill, in this specific scenario.

VERY Important caveat on interpreting statistical results of models

If you are not very experienced with this sort of analysis, you might mistakenly think that this result is amazingly accurate and therefore must reflect reality.  That I should be able to so accurately reproduce the real-life numbers with some reasonable inputs into a model might seem proof-positive that the immunocompromised are the real reason for the vaccinated making up 74% of the infected people from the Barnstable County outbreak.  But if you thought this, you would be very wrong.

In reality, I had to tweak many of the inputs in this model in order to come up with a percentage very close to the observed percentage.  I constructed the spreadsheet, put in some initial numbers, and then tweaked those parameters that I thought could be realistically tweaked, until my end result finally said "73%".

Since none of my parameters are outlandish, but all could potentially be realistic and true, it is correct to say that my theory (that immunocompromised are almost completely the cause of the scary proportion of vaccinated individuals from Barnstable) is consistent with the available data.  "Consistent with" is a very different statement from "proof of", and it is important to be aware of the difference between a study claiming one and a study claiming the other.

Sensitivity Analysis

It is for this reason that whenever I do a rough-model like this, I provide the link to the spreadsheet, and I encourage anyone who reads my estimates to look at the spreadsheet and make changes--play around with different parameters to see what sorts of result are generated with the new numbers.

When you do this formally and rigorously, this sort of thing is called "sensitivity analysis": you systematically change each parameter--individually and in groups--and determine which parameters are the important ones that actually make a difference on the outcome.  Good statistical packages can do this for you automatically nowadays--though you have to be sure you include all of the relevant parameters as inputs to the program!  "Garbage In, Garbage Out" is still a very true dictum.

Understanding the dynamics of which parameters make a difference to the outcome can do several things for you:

First, it allows you to see what aspects of the problem are more important to get clarity on.  For example, in my toy model, I found that although I accounted for the difference between the local population and the vacationing population, I really didn't need to bother.  The vacationing population is so much bigger, the effects of the local population on the outcome doesn't really matter.  This tells me that the "69% vaccinated" rate of Barnstable County that was reported in the CDC paper and in a lot of news outlets, is really irrelevant. 

Second, it allows you to see how reliable your result is.  In the case of my toy model, I am able to see that my result is not very reliable at all--I have some excessively sensitive parameters that are also too much of a raw guess on my part.  The key number here that makes all the difference is the vaccination rate of the vacationers.  I set that at 90%; if instead you set if lower (to 80%, for example), you have to set the effectiveness of the vaccine way down (to somewhere around the 40% level) in order to still end up in the neighborhood of 74% vaccinated among the infected, if you keep all the other parameters the same.

Likewise, the proportion of HIV positive individuals among the vacationers has a huge impact on the result, and my proportion of 1/5 is taken from a single study of general trends, not any sort of specific survey of this particular population.

Therefore, what this model really proves is only that the events of Barnstable County are currently capable of multiple interpretations.  If we want to know what is really going on in this outbreak, we need more information.  In particular, the prevalence of both vaccination and HIV in the vacationing population are very important for a correct interpretation: both of which might be very difficult to obtain at this point.

Bottom Line

The bottom line conclusion of this must be that we cannot make firm conclusions about the real-world effectiveness of the vaccines from the Barnstable data as it has been reported to us so far.  We would need a far better knowledge of other variables at play--other risk factors--in order to know which variables aside from vaccination effectiveness may have caused different people to end up infected or in the hospital.

This conclusion is especially true in the case of the Barnstable data given that the outbreak there occurred under circumstances far from normal for the national population in ways that are materially relevant to disease spread and vaccine efficacy.  But the conclusion is also true for a lot of other data that has been bandied about by many people.  In general, the prevalence of vaccinated among the hospitalized is a very bad statistic on which to make conclusions.  There are far too many confounding variables that are in play--far too many risk factors which dramatically change the likelihood of hospitalization independently of vaccination status--for this bare statistic to be of any use without a whole lot of other data about those people.

The gold standard for judging the effectiveness of a vaccine is the ability to compare vaccinated people versus unvaccinated people when you are able to control for all other variables.  You want to compare vaccinated sick old men with unvaccinated sick old men, vaccinated teenage girls with no health problems with unvaccinated teenage girls with no health problems, vaccinated middle-aged gay party-goers with HIV to unvaccinated middle-aged gay party-goers with HIV.  And so forth, and so on.

Simply comparing the raw numbers of hospitalized vaccinated to hospitalized unvaccinated people--with no differentiation--is going to be comparing apples to oranges with a vengeance.