The Wall Street Journal recently reported that French researchers treated COVID-19 patients with a “game-changer” combination of hydroxychloroquine and a Z-Pak (antibiotic), and that every single patient was cured within 6 days. In comparison, the recovery rates were 57.1% for patients treated with hydroxychloroquine alone and only 12.5% for patients who received neither medication. More tests are needed but the obvious conclusion would be treat first, test later. What harm could that do?

Well, there may be currently unknown side effects. In Vanity Fair, Gail A. Van Norman, a doctor who studies whether drugs should be adopted, didn’t hold back: “Trump said it’s a great drug because we know it’s safe. We know nothing of the kind. Claiming that it’s safe for a use for which it hasn’t been tested is a lie.” There is also the obvious problem that some people are stockpiling the drug like toilet paper, thus making it less available to those who suffer from the serious conditions that the medicine is intended to treat.

A less obvious problem is that a false sense of security may seduce some people into less caution. If there is an easy cure, why worry? If the game-changer drug is not an easy cure, misplaced trust could dramatically increase the number of infections.

One of the most downloaded papers in history is Dr. John Ioannidis’s provocative, “Why Most Published Research Findings Are False.” Dr. Ioannidis gives several warning signs for misleading research, including these three:

Small sample sizes

If you rank cities by crime rate or cancer rate, the cities at the top and bottom are almost invariably small cities. This is not because small cities tend to be particularly safe or dangerous, but because small data sets are more likely feature extreme results. If a basketball player makes 6 free throws in a row, you shouldn’t conclude that he never misses. In the same way, if a study reports 100% recovery for 6 patients (yes, that was actual total number of patients treated with hydroxychloroquine and Z-Pak), your response should not be, “Wow, you found a cure!,” but, “Wow, not many patients in that study!”

Flexible analysis

Ronald Coase, an Economics Nobel Laureate, cynically observed that, “If you torture data long enough, it will confess.” According to the French study, “Six hydroxychloroquine-treated patients were lost in the follow-up during the survey because of early cessation of treatment. Reasons are as follows: three patients were transferred to intensive care unit… one patient stopped the treatment… because of nausea… one patient died…” Wait, what? Shouldn’t the research question be whether or not the drug helps patients leave the hospital alive when all is said and done? There was also a patient counted as virus-free who was found to have the virus two days later.

Dr. Adam Chapweske has pointed out several other analysis issues to us (edited for brevity here):

• Best I can tell, their decision to report their results early was because they got the results they wanted.
• The registered design for the study does not mention Z-Pak, yet it figures prominently in their reported results. Were there other unreported changes?
• The primary endpoint was virological clearance, but the baseline viral load for the control group is not given so we don’t know if the two groups are comparable.
• Virological measurements were reported differently for the groups as well, which suggests to me that there were differences in the way they were tested.

A hot field of study

When a field is hot, there is a race to publish. With a large number of researchers trying a wide variety of treatments, it’s almost certain that that someone will find a coincidental pattern that is statistically significant but meaningless. The only solution, alas, is further testing.
In addition to these warning signs from Dr. Ioannidis, the control group in the study was not randomly chosen and differs significantly from the treatment group (for example, on average, participants were 14 years younger). The observed differences in results may well have been due to differences in age and other confounding factors.

The World Health Organization (WHO) has just announced a megatrial that will test the four most promising treatments for COVID-19. The WHO scientific panel did not originally plan to test the “game changer” drug but decided to include it because of the attention it is getting. Susanne Herold, an expert on pulmonary infections, was skeptical: “Researchers have tried this drug on virus after virus, and it never works out in humans. The dose needed is just too high.”

We hope she is wrong, but we suspect she may be right, and the game changer drug and people’s faith in it may do more harm than good.

See also: Was the COVID-10 Virus Designed? The Computer Doesn’t Know. Some researchers confuse not finding a particular type of design with ruling out design. The authors of the paper saying that the virus could not have been designed base their findings on the fact that current software would not have predicted this result. I am not claiming that the virus was designed, only that a researcher need not have relied on this software to design a virus. The paper makes a faulty assumption.