Predicting the results of bad science: Meet the Leviathan
Trained eyes can see what's wrong in critical care research
Trained eyes can see what's wrong in critical care research. However, the American taxpayers can't realize they pay to prevent a stagnant field from developing. It is agnostic to left or right-wing politics. It's deeper. Americans are funding it through NIH for decades regardless of who is in charge. Your money feeds the Leviathan, a mythical giant that makes dollars flow from your pocket to elite medical scientists’ pockets.
The Leviathan, althoug American, has its legs and arms stretched worldwide through a process called Intelectual Colonization. The monster head is the NIH. This is our David's slingshot target.
Today, I invite you to embark on a time travel: there will be an international critical care conference next month where they will present the results of many studies for the first time. I can foresee the studies’ results. How can I do it? I invite you to search through my posts and learn for yourself.
Next December, we will receive another batch of studies at the Critical Care Reviews Down Under, an international conference devoted to critical care research. Connected people worldwide will comment and congratulate the researchers. Successfully performing a large RCT is a high accomplishment.
I decided I wouldn't wait for the conference to comment on the studies’ results and papers. No crystal ball was needed. All I had to do was time-travel to December 10th just like in the film Arrival and learn the results first-hand.
In Arrival, extra-terrestrians arrive and communicate graphically. They perceive reality as unconstrained by time, hence they operate in the present and the future simultaneously. Their circular sentences are infinite like rings, instead of ours that project themselves only in one direction.
It reminds me of Taoism, an Eastern tradition whose admirable texts I have read for a long time. The Taoist sage sees reality flowing in circles (or cycles) between pairs of abstract opposites like agitation and calm, or the very concrete and natural cycles like the seasons’ succession during the year.
The sage, conscious of reality's pendular or circular nature, increasingly aligns his acts with the perceived flow of reality. There is a point where his acts are perfectly aligned to the stream of reality, either because he feels it, or because he transformed the future. At this point, he acts with no effort.
The circular configuration of the I Ching and the Taijitu diagrams mirrors the idea of circularity, in opposition to the linear Western one-action-after-the-other view of reality. The view I just termed “linear” lacks the notion of constant transformation and forgets that circumstances change despite people's will and best efforts.
Please don't go. Allow me a Taoist citation and I promise I will start writing about the CCR Down Under studies.
If you would have a thing shrink,
You must first stretch it;
If you would have a thing weakened,
You must first strengthen it;
If you would have a thing laid aside,
You must first set it up;
If you would take from a thing,
You must first give to it.This is called subtle discernment:
The submissive and weak will overcome the hard and strong.The fish must not be allowed to leave the deep;
The instruments of power in a state must not be revealed to anyone.
The TRAUMOX2 Trial
The multicentric trial follows a unicentric one from the same researchers (TRAUMOX1) and tests the same “intervention". They kept SpO2 in the normal range in all patients, around 94% in one group and above 98% in the other, to see what happens and log a publication.
The study is powered to detect a 33% risk reduction with a restrictive oxygen strategy if the incidence of the primary outcome is 15% in the liberal group. The primary outcome is a combination of death, pneumonia, and ARDS at 30 days. The effect estimate is blatantly optimistic. They will use logistical regression and report odds ratios, which I think is honest.
In the protocol, they provide previous results to justify the study. The authors argued that one large observational study showed more pneumonia and ARDS in the liberal oxygen group. Then they cite mixed results from larger observational and randomized studies as if it was enough justification to perform the trial.
The TRAUMOX2 protocol fails to indicate WHY a patient would die, have pneumonia, or develop ARDS BECAUSE of the expected SpO2 difference. In other words, they have no disease model in mind. Their hypothesis lacks causal reasoning, the study is unjustified, and any positive association will be spurious.
The problem is they are blind to the marginal utility of their intervention. The authors should try to delineate a disease model and identify the dominant cause of mortality. The main driver of mortality in trauma is the trauma itself, which will be present and treated in both groups and hence is not affected by the so-called intervention.
After the dominant cause is established, the effect of each additional mortality cause is marginal, including SpO2 levels. I would place a 4% SpO2 difference so far from the bulk of strong mortality drivers that its effect has to be irrelevant. They should have a disease model that explains how higher SpO2 is sufficient to cause death, pneumonia, and ARDS. They don't have a disease model in mind. They are just doing it for the sake of doing it.
Here are my predictions: no exposure/outcome association. However, there is a risk that the sample is insufficient to harmonize the case mix resulting in a type 1 error. In this case, we may have a spurious association with an embarrassing Backwards Reasoning exercise in the Discussion section.
Moreover, the trial is irreproducible because the case mix is irreproducible. The Petty and Bone mistake has spread everywhere in critical care. I foresee someone will attempt reproduction anyway, fail, and publish a meta-analysis.
How can a 98% SpO2 cause more ARDS than a 94% SpO2?
Is it a koan?
The RENOVATE Trial
RENOVATE is a multicentric trial testing non-invasive ventilation (NIV) vs high-flow oxygen through nasal caterer (HFNC) in hypoxemic ventilatory failure. They will measure the rates of endotracheal intubations in each group by the 7th day.
The outcome is very appropriate and relevant. It makes total sense for emergency and critical care physicians. Both interventions have a sound and simple causal relation to the outcome. Hypoxemic respiratory failure occurs when a patient's lungs are severely compromised causing insufficient oxygen absorption to the blood. The hypoxic patient needs to increase the (neuro)muscular work of breath as a desperate measure to oxygenate. The patient often tolerates the extra effort buying enough time for the treatment to resolve the acute disease.
Ventilatory failure occurs when the patient cannot sustain the (neuro)muscular effort of ventilating his or her lungs and needs tracheal intubation. Both RENOVATE Trial interventions can affect early intubation rates because both raise airway pressure thus recruiting collapsed alveoli and reducing the work of breath at the same time they increase the inhaled air oxygen content.
The either-or approach of comparing the interventions against each other is tricky because both are plausibly related to the outcome for the same reason. The study design boils down to testing which delivers more oxygen and airway pressure, i.e., testing the marginal effects of more oxygen and pressure. In this case, the likely winner will be NIV because it can deliver more pressure and is more reliable in oxygen delivery. No RCT is needed.
Will it be enough to be clinically relevant? It is hard to say because the main intubation drivers are the very causes of ventilatory insufficiency and individual responses to treatment. NIV/HFNC are not treatments but support measures. Both will buy time for treatment and early responders will forgo intubation in both groups, maybe a few more in the NIV group.
Here are my predictions. We will see numerically fewer intubations in the NIV arm, although short of statistical significance in frequentist analysis. Authors won't have a fine headline to work with.
Authors like to affirm associations more than deny them. It gives them more visibility, invitations, and money for their projects. There is a trend towards “augmenting” analysis to find irrelevant effects and stamp a "statistically significant” seal on them.
Like a man taking a tiny small fish home instead of telling his wife and friends he got nothing, the authors will take you to The Magical Kingdom of Bayesian Analysis. There they will fool you with a Bayesian Spin.
The Bayesian Spin has the general formula “We didn't find a meaningful difference between groups but the probability of existing a positive although infinitesimal difference is almost 100%”.
RENOVATE is another Irrelevant Yet Significant study employing fancy statistics to find positive results.
Of course, we have the old problems. Patients were triaged in the study if they met a set of ad-hoc physiologic thresholds, not a nosologic diagnosis made inside a disease model. It is the failed Petty and Bone paradigm again. It makes people (1) mistake treatment for support measures, (2) blind to marginal utility, and (3) enroll an irreproducible case mix.
In The Magical Kingdom of Bayesian Analysis,
everything is perfect and we are always improving
I think two studies were enough to make the point. I can see what's wrong because the studies are set to fail from the moment they were conceived. That's the nature of our time travel - a new framework to evaluate critical care research and to expose its flaws. Like a Taoist sage, we know the future. And we know the Leviathan will keep producing failed studies because it needs to keep the money flowing.
Thanks for reading The Thoughtful Intensivist!
Did you like it? Please hit the restack button!
Meet the Leviathan in our book
We present The Leviathan and many parts of our framework in the book:
“The Physician's War: The Story of the Hidden Battle between Physicians and a Science Based on Pathological Consensus.”
Here is the link to Amazon.1
Discover more
Illustrative of the nonsensical research produced by American academics.
About real medicine and failed research.
International readers might find a different price in the local Amazon vendor.
Love your take on the marginal utility of studies in critical care, under the guise of Evidence Based medicine, where neither genetic nor environmental variables can be sufficiently determined or measured.
Yet we seem to skip over or ignore the big question.
What is the purpose of critical care if the original resuscitation efforts fail.
As a specialist in CCM, and an avid medical historian, I would like to hear more opinions about a medical art that was really born out of MASH units in the Korean War.