Thresholds-based definitions introduced bias into critical care research

A short paper

This post was formatted as a journal article, but I don't hold many hopes that the industry will be interested in questioning long-cherished dogmas. I will bet on social media instead. Please share it.

[Oct. 4th] Two outright rejections and one rejection after review. Reviewers have provided me with valuable insights into their worries.

---

Introduction

Researchers in the field of critical care medicine struggle with the scarcity of positive results in randomized clinical trials (RCTs). ¹ Moreover, positive results mainly come from single-center investigations, whose findings are typically not reproduced by multi-center studies. ² The subfields of sepsis and the acute respiratory distress syndrome (ARDS) are especially affected, which led authorities to call for a halt in testing new drugs in ARDS RCTs and even argue that RCTs should be outright abandoned. ^3,4

Less radical responses pointed to insufficient power, a problem that was mostly addressed by larger trials, sometimes nested in platform trials, and enhanced detection of positive signals using advanced data methodology at the expense of the clinical relevance of increasingly subtle signals. ^{5- 7} Researchers also identified patient heterogeneity as a potential cause for the lack of substantial results, and the field now bets on an adaptation of the precision medicine rationale, in which patients are sorted by phenotypic expression using various strategies, ranging from bedside vital signs to transcriptome. ^8,9

The difficulty, nevertheless, may stem from the very definitions of sepsis and ARDS. Both diseases are defined as a set of physiological thresholds arbitrated by consensus, avoiding a particular causal mechanism. ^10,11

This paper aims to bring the threshold definitions to the forefront of the discussion. The consequences of introducing threshold definitions will be analyzed using direct acyclic graphs.

Methods

The field of causal inference offers a notation technique, the direct acyclical graph (DAG), that allows for a direct visualization of existing biases. In DAGs, variable nodes are connected through causal arrows. By causal, it is meant that one variable “hears”, or is conditioned, or is caused, by the variable in the preceding node.

In Figure 1, panel A, we see an arrow linking the disease X to the outcome Y. The arrow establishes causality between the variables, indicating that the disease causes the outcome to a certain extent. The lack of other nodes in the diagram implies that, in this model, the disease is the only and necessary cause of the outcome. The causality assumption for the simplest clinical trial is represented by the introduction of the treatment node R (Figure 1, Panel B). Now, the outcome Y is conditional on receiving the treatment R. The absence of other nodes connecting to X, R, and Y suggests there are no other variables in the causal chain and no potential source of bias.

Figure 1. A direct acyclic graph (DAG). The arrows are causal links between nodes. Panel A, the disease X causes the outcome Y. Panel B displays the simplest clinical trial. Outcome Y is influenced by Treatment R.

Results and Discussion

The effect of adding thresholds-based definitions (T) is that receiving the treatment (R) is conditional on T (Figure 2). Importantly, T is not disease-specific, as many different diseases (D1-3) may fulfill the threshold definition. The node T is a collider, a graphic representation of selection bias.

Although taken for granted by many, a threshold-based definition (T) doesn't imply a common biological cause. Diverse diseases may fulfill T by diverse causal pathways. During decades of negative sepsis and ARDS RCTs, field experts may have failed to inform that T is not only disease-agnostic, but also cause-agnostic. Scientists on the data methods side may have assumed that there is a common causal pathway connecting all disease nodes D to T, and they could safely ignore what caused T, but this is a false premise. Many different diseases cause T, each featuring a distinctive causal mechanism. If enrollment is conditional on T, not on the diseases causing T, it follows that each RCT's case mix will be unique and irreproducible. This is why sporadic positive findings from single-center studies are seldom reproduced.

Figure 2: A direct acyclic graph (DAG) with the introduction of a threshold-based definition as a collider. Treatment is conditional on T, not on diseases D1-3.

The stubborn enthusiast of threshold-based definitions (T) may concede that T introduces selection bias and impedes reproducibility, but argue that randomization within a single trial might harmonize the case-mix between groups and save the causal conjecture. Again, the collider T is cause-agnostic. A treatment conditional on T may address the cause of one of the lumped diseases, but not the others, in any combination, including not addressing the cause of any disease in the pool. The causal conjecture is harmed, anyway. No one can specify why a treatment worked in a specific RCT. No causal knowledge is achieved.

Although serious enough, the biasing effect of a threshold-based definition (T) is not limited to selection bias, for T is also an outcome predictor in sepsis and ARDS. For instance, the 2016 Sepsis-3 definitions explicitly introduced a prognostic score (qSOFA, a threshold-based score) as a tool to identify patients with sepsis. Consequently, qSOFA (T) informs both the Disease status (D) and Outcome (Y), i.e., it is a confounder (Figure 3).

To preserve the validity of a causal inference, a researcher should remove or control for the confounding variable T. Unfortunately, it is not feasible in ARDS and sepsis RCTs because patient inclusion is conditional on T. In a most peculiar arrangement, T is at the same time a collider and a confounder. Sepsis and ARDS research will remain biased by design until researchers abandon the threshold-based definitions.

Figure 3: A threshold-based definition (T) introduces confounding bias because it informs both exposure and outcome. The bias is inextricable because enrollment is also conditional on T.

Conclusion

Direct Acyclic Graphs elicit a direct visualization of the causal chain and sources of bias. Threshold-based definitions introduce selection and confounding bias. These definitions have biased decades of research in sepsis and ARDS and should be abandoned.

---

Please share it on your social media, including message apps like WhatsApp. If your social media app doesn't appear after clicking the share button, please copy the link and paste it on WhatsApp, Instagram, etc. Also, be sure to like and restack it.

In early September, paid subscribers will receive a video with a bottom-up exposition of today's issue. If you want to dive deeper, please consider becoming one.

Meanwhile, here's a suggestion for a weekend read.

Talkin' Bout a Revolution

---

References

1. Laffey JG, Kavanagh BP. Negative trials in critical care: why most research is probably wrong. Lancet Respir Med. 2018 Sep;6(9):659-660. [DOI]

2. Kotani, Y., Turi, S., Ortalda, A. et al. Positive single-center randomized trials and subsequent multicenter randomized trials in critically ill patients: a systematic review. Crit Care 27, 465 (2023). [DOI]

3. Shankar-Hari M, Calfee CS. Lack of Clinical Benefit of Interferon β-1a Among Patients With Severe Acute Respiratory Distress Syndrome: Time to Overhaul Drug Trials in ARDS? JAMA. 2020;323(8):713–715. [DOI]

4. Vincent JL. We should abandon randomized controlled trials in the intensive care unit. Crit Care Med. 2010 Oct;38(10 Suppl):S534-8. doi: 10.1097/CCM.0b013e3181f208ac. PMID: 21164394. [DOI]

5. Aberegg SK, Richards DR, O’Brien JM. Delta inflation: a bias in the design of randomized controlled trials in critical care medicine. Crit Care. 2010;14:R77. [DOI]

6. Zampieri FG, Casey JD, Shankar-Hari M, Harrell FE Jr, Harhay MO. Using Bayesian Methods to Augment the Interpretation of Critical Care Trials. An Overview of Theory and Example Reanalysis of the Alveolar Recruitment for Acute Respiratory Distress Syndrome Trial. Am J Respir Crit Care Med. 2021 Mar 1;203(5):543-552. [DOI]

7. Leite RO. More is exponentially less: marginal utility in critical care research. Evidence [Internet]. 2022; 4:e4722 [DOI]

8. Bhavani SV, Holder A, Miltz D, et al. The Precision Resuscitation With Crystalloids in Sepsis (PRECISE) Trial: A Trial Protocol. JAMA Netw Open. 2024;7(9):e2434197. [DOI]

9. Pelaia, T.M., Shojaei, M. & McLean, A.S. The Role of Transcriptomics in Redefining Critical Illness. Crit Care 27, 89 (2023). [DOI]

10. Bone RC, Fisher CJ Jr, Clemmer TP, Slotman GJ, Metz CA, Balk RA, and the Methylprednisolone Severe Sepsis Study Group. Early methylprednisolone treatment for septic syndrome and the adult respiratory distress syndrome. Chest. 1987 Dec;92(6):1032-1036. [DOI]

11. The ARDS Definition Task Force. Acute Respiratory Distress Syndrome: The Berlin Definition.JAMA.2012;307(23):2526–2533. [DOI]