Reinterpretation of the results of randomized clinical trials

Abstract

Background Randomized clinical trials (RCTs) shape our clinical practice. Several studies report a mediocre replicability rate of the studied RCTs. Many researchers believe that the relatively low replication rate of RCTs is attributed to the high p value significance threshold. To solve this problem, some researchers proposed using a lower threshold, which is inevitably associated with a decrease in the study power. Methods The results of 22 500 RCTs retrieved from the Cochrane Database of Systematic Reviews (CDSR) were reinterpreted using 2 fixed p significance threshold (0.05 and 0.005), and a recently proposed flexible threshold that minimizes the weighted sum of errors in statistical inference. Results With p 0.05 criterion, 28.5% of RCTs were significant; p 0.005, 14.2%; and p flexible threshold, 9.9% (2/3 of significant RCTs based on p 0.05 criterion, were found not significant). Lowering the p cut-off, although decreases the false-positive rate, is not generally associated with a lower weighted sum of errors; the false-negative rate increases (the study power decreases); important treatments may be left undiscovered. Accurate calculation of the optimal p value thresholds needs knowledge of the variance in each study arm, a posteriori. Conclusions Lowering the p value threshold, as it is proposed by some researchers, is not reasonable as it might be associated with an increase in false-negative rate. Using a flexible p significance threshold approach, although results in a minimum error in statistical inference, might not be good enough too because only a rough estimation may be calculated a priori; the data necessary for the precise computation of the most appropriate p significance threshold are only available a posteriori. Frequentist statistical framework has an inherent conflict. Alternative methods, say Bayesian methods, although not perfect, would be more appropriate for the data analysis of RCTs.

Summary

Habibzadeh's study investigates how varying p-value significance thresholds influence the interpretation of RCT results. The analysis of 22,500 RCTs reveals that applying a p-value threshold of 0.05 results in 28.5% of studies being deemed significant, while a more stringent threshold of 0.005 reduces this to 14.2%. Implementing a flexible threshold designed to minimize the weighted sum of errors in statistical inference further decreases the proportion of significant studies to 9.9%. The findings suggest that lowering the p-value threshold decreases the false-positive rate but increases the false-negative rate, potentially leading to the dismissal of important treatments. Additionally, the study indicates that the flexible p-value threshold approach, although intended to minimize errors, may not be sufficiently accurate due to the lack of precise a priori data for its calculation. These insights highlight the complexities involved in statistical inference and the need for careful consideration of p-value thresholds in clinical research.