Abstract
Background
Machine learning assisted systematic reviewing may help to reduce the work burden in systematic reviews. The aim of this study was to determine, through a non-developer, the performance of machine learning assisted systematic reviewing on previously published orthopaedic reviews in retrieving relevant papers.
Methods
Active learning for Systematic Reviews (ASReview) was tested against results from three previously published systematic reviews in orthopaedics, covering easy, intermediate, and advanced scenarios. Each review was tested with 20 iterations. Key outcomes were the percentage work saved at 95% recall (WSS@95), percentage work saved at 100% recall (WSS@100), and percentage of relevant references identified after screening the first 10% of records (RRF@10). Means and [95% confidence intervals] were calculated.
Results
The WSS@95 was 72 [71–74], 72 [72–73], and 50 [50–51] for easy, intermediate, and advanced scenarios, respectively. The WSS@100 was 72 [71–73], 62 [61–63], and 37 [36–38], respectively. The RRF@10 was 79 [78–81], 70 [69–71], and 58 [56–60] for easy, intermediate, and advanced scenarios, respectively.
Conclusions
Machine learning assisted systematic reviewing was efficient in retrieving relevant papers for orthopaedic reviews. The majority of relevant papers were identified after screening only 10% of the total papers. All relevant papers were identified after screening 30%–40%, potentially saving 60%–70% of the work.
Key Questions
1. What is the purpose of machine learning assisted systematic reviewing?
It aims to reduce the workload in systematic reviews by efficiently identifying relevant studies while maintaining high recall rates.
2. How effective was machine learning in orthopaedic systematic reviews?
In easy and intermediate scenarios, 72% of the workload was saved at 95% recall. Even in advanced scenarios, machine learning achieved substantial work savings of 50%–62%.
3. How quickly were relevant papers identified?
The majority of relevant papers were identified after screening only 10% of the total papers, making the process highly efficient.
4. What are the practical implications of this study?
Machine learning can significantly streamline the systematic review process, potentially saving 60%–70% of the screening workload while maintaining thoroughness and accuracy.
