Artificial intelligence streamlines diagnosis and assessment of prognosis in Brugada syndrome: a systematic review and meta-analysis

Study selection

This review was directed in accordance with the 2020 Preferred Reporting Items for Systematic Reviews and Meta-Analyses^[12]. Electronic searches were conducted in MEDLINE, EMBASE, SCOPUS, and WEB OF SCIENCE from database inception to November 6, 2023, with keywords “artificial intelligence” OR “deep learning” OR “machine learning” AND ECG or electrocardiogram AND “Brugada” or Brugada syndrome (see Supplementary S1 for full search strategy). A filter to retrieve studies related to artificial intelligence developed by the University of Alberta was used^[13].

The inclusion criteria were all primary research papers published in English that examined the utility of AI, machine learning, and ECG data in diagnosing or predicting adverse cardiac events in patients with Brugada syndrome. The exclusion criteria were: pediatric patients and non-human studies. Abstracts, editorials, case reports, and reviews were also excluded.

All references were uploaded to Covidence and were electronically merged to remove duplicates^[14]. Two authors (CL and SS) individually reviewed studies to determine their inclusion or exclusion. The data extracted from each study were: study design, country in which the study was conducted, AI training cohort size, Brugada sample size, and control sample size. In addition, the following algorithm characteristics were extracted from each study: sensitivity, specificity, positive predictive value, negative predictive value, accuracy, area under the curve (AUC), and F1 score. Two reviewers (CL, JS) examined each paper independently to determine whether they fit the inclusion or exclusion criteria. Data extraction was conducted by two reviewers (CL, JS) and a consensus was reached for any conflicts.

Data analysis

Risk of bias assessment was conducted by CL. The QUADAS-2 tool was used to assess the risk of bias in diagnostic algorithm accuracy studies, whereas the PROBAST tool was used to assess the risk of bias in prognostic algorithm accuracy studies^[15,16]. Diagnostic studies were assessed based on the domains of patient selection, index test(s), reference standard, and flow and timing. Prognostic studies were assessed based on the domains of participants, predictors, outcomes, and analysis.

Study characteristics

Studies were conducted in 7 different countries/regions including Italy (n = 3), Switzerland (n = 1), Japan (n = 1), Taiwan (n = 1), Canada (n = 1), China (n = 3), France (n = 2). The year of publication of included studies ranged from 2016-2023. Five studies used AI as a diagnostic tool for BrS [Table 1]. Seven studies used AI as a tool to prognose adverse cardiac outcomes related to BrS [Table 2]. Most studies reported training sample size as the number of patients; however, only three studies reported the number of ECG readings (and not patients) used in the training of the algorithm. The total training sample size of all included studies was n = 1,868 for diagnostic studies and n = 1,859 for prognostic studies. Gender was only reported for two diagnostic studies and four prognostic studies and the range was from 69.6% male to 95% male. Mean age was only reported for one diagnostic study and five prognostic studies, ranging from 36 to 50 years old. Lee et al. and Lee et al. trained their models using the same dataset, based in Hong Kong^[18,19]. More information regarding the study validation method, data selection, and the preparation process can be found in Supplementary Table 2.

Risk of bias assessment

Diagnostic studies were assessed based on the domains of patient selection, index test(s), reference standard, and flow and timing. Two diagnostic studies were determined to be at high or unclear risk of bias. Prognostic studies were assessed on the domains of participants, predictors, outcomes, and analysis. Four prognostic studies were determined to be at high or unclear risk of bias. Detailed results are provided in Supplementary Tables 3 and 4.

Model testing and validation

ML algorithms were evaluated based on the area under the curve (AUC) of the receiver operating characteristic curve (ROC). Accuracy, positive predictive value (PPV), negative predictive value (NPV), sensitivity, specificity, and F1 were also used as evaluation metrics. Accuracy is defined as the number of overall cases correctly identified, but it may be misleadingly high if the model is trained on an imbalanced dataset. PPV (i.e., precision) is the ratio of predicted positives to true positives. NPV is the ratio of predicted negatives to true negatives. Sensitivity is the model’s ability to identify true positive cases, whereas specificity is the probability that a predicted negative is truly negative. F1 score is the harmonic mean of precision and recall, balancing the two metrics.

Notably, these metrics are dependent on a defined threshold value, which determines the classification boundary between positive and negative cases. A higher threshold may increase sensitivity at the cost of specificity. Conversely, a lower threshold may decrease sensitivity at the cost of specificity. Threshold selection techniques varied between selecting an optimal value based on the ROC curve, using optimal precision vs recall, Youden’s J statistic, and using predefined sensitivity values [Supplementary Table 4].

The AUC of the best-performing diagnostic study algorithms ranged from 0.934-0.976. The AUC of the best-performing prognostic study algorithms ranged from 0.7092 to 0.942. However, the included studies did not consistently report all of the metrics, with several studies not reporting both AUC and a 95%CI.

Different studies used a variety of machine learning algorithms. Lee et al. trained a random forest model to predict spontaneous VT/VF on latent risk factors extracted by non-negative matrix factorization (NMF)^[18]. The total sample size was 516 and included 314 asymptomatic patients. Liao et al. trained several convolutional neural networks (CNN) to identify and diagnose the type 1 Brugada ECG pattern^[10]. The highest-performing algorithm was the convolutional deep neural network (DNN) trained on 12-lead ECG data, which had an AUC of 0.976 (96%CI: 0.973-0.979). Liu et al. used a learning transfer strategy on a model originally used to classify right bundle branch block (RBBB) and adapted it to classify the type 1 Brugada pattern^[11]. Melo et al. trained a DNN on 12-lead ECG data in a cohort of 1,154 patients (596 BrS positive, 558 controls)^[9]. Only a small fraction of patients showed a type 1 Brugada pattern and these patients were identified with 100% accuracy. Randazzo et al. trained two models, a multi-layer perceptron neural network (MLP) and a boosted decision tree (BDT), on ECG features extracted manually by cardiologists to predict retrospective arrhythmic events^[20]. Tse et al. trained a regression model with latent variables extracted by NMF to predict spontaneous VT/VF incidence^[21]. These included clinical variables such as syncope and AF as well as ECG variables such as type 1 Brugada pattern, QRS duration, QTc interval and others. When validated on an external cohort from multiple different countries, they found that the model’s performance was optimal when trained on five latent variables. Romero et al. trained an ensemble classifier to distinguish BrS patients according to symptomatology using features extracted from the QRS complex, HRV markers, or both^[22]. Romero et al. utilized a multivariate ensemble classifier trained on ECG data for risk stratification in 110 BrS patients, of which 25 showed symptoms^[23]. Lee et al. compared the performance of 7 different machine learning models with respect to the prognosis of VT/VF^[19].

Diagnostic algorithm performance

Five studies used ML algorithms for the diagnosis of BrS [Tables 3 and 4]. Micheli et al. used a CNN trained on ECG data for the diagnosis of BrS on a dataset of 306 ECGs from the BrAID (Brugada syndrome and Artificial Intelligence applications to Diagnosis) project^[24]. The model showed excellent performance with a sensitivity of 0.8773 and a specificity of 0.9234. Melo et al. trained a DNN on a cohort of 596 BrS-positive and 558 control patients^[9]. On an external validation cohort of 370 ECGs, the model demonstrated good performance in diagnosing BrS without the use of a SCB (0.934 AUC, 95%CI: 0.973-0.979). Zanchi et al. compared various ML models trained on P-wave features for the diagnosis of BrS in a cohort of 123 patients^[25]. The worst-performing model was the K-nearest neighbors’ model, with a reasonable sensitivity (0.843) but poor specificity (0.513). The best-performing model was the AdaBoost model, with a sensitivity of 0.865 and specificity of 0.738. Liu et al. compared the performance of a deep-learning model with that of two cardiologists in the diagnosis of BrS based on ECG^[11]. The model showed higher sensitivity (0.884 vs. 0.627) than the cardiologists but poorer specificity (0.891 vs. 0.985). The model had a higher AUC than the cardiologists (0.96 vs. 0.81). Similarly, the deep-learning model in Liao et al. outperformed two cardiologists in the classification of BrS type 1^[10]. The model achieved a sensitivity of 0.96 and specificity of 0.90, which was higher than the first cardiologist (sensitivity = 0.889, specificity = 0.880) and was similar to a second cardiologist (sensitivity = 0.925, specificity = 0.920).

Overall, ML algorithms for the diagnosis of BrS via ECG data showed good performance with regard to sensitivity and specificity. We performed a pooled analysis of the best-performing algorithm from each study. The sensitivity and specificity of the best-performing diagnostic algorithms ranged from 0.80 to 0.89 and 0.74 to 0.97, respectively. A meta-analysis showed that overall studies sensitivity was 0.848 ± 0.015 (SEM, z = 57.3 m, P < 0.0001) [Figure 2] and specificity was 0.892 ± 0.061 (SEM, z = 14.5, P < 0.0001) using a random effects model [Figure 3]. An analysis for publication bias using the classic Failsafe-N test would require over 7,000 negative studies to invalidate the result for sensitivity and over 2,000 negative studies to invalidate the result for specificity.

Figure 2. Forest plot of sensitivity of diagnostic studies. Error bars represent 95% confidence intervals.

Figure 3. Forest plot of specificity of diagnostic studies. Error bars represent 95% confidence intervals.

Since the majority of studies did not explicitly report 2 × 2 contingency tables, these were imputed algebraically from their data, where necessary, using sensitivity, specificity, sample size, and number of condition-positive patients. The heterogeneity of studies was assessed using Chi-squared tests for equality of sensitivities and specificities (Test for equality of sensitivities: X-squared = 7.4429, df = 4, P-value = 0.114; Test for equality of specificities: X-squared = 79.9133, df = 4, P-value ≤ 2 × 10^-16). This suggests that there are significant differences in specificity but not sensitivity among diagnostic studies. Next, a bivariate approach was used to calculate the pooled SROC. Using the mada package in R, we fit a bivariate diagnostic random-effects meta-analysis^[17]. Among five diagnostic studies, the overall pooled summary area under the receiver operator characteristic curve (SAUROC) for diagnosis of BrS was 0.877 [Figure 4]. The SAUROC represents the pooled AUC of all the included studies. It was calculated by combining the true positive rates and false positive rates from the included studies and plotting them against each other. A higher SAUROC represents greater diagnostic/prognostic accuracy across several ML models and datasets.

Figure 4. Summary area under the receiver operating characteristic curve (SAUROC) of ML algorithms for diagnosing BrS.

Prognostic algorithm performance

The AUC of the best-performing prognostic algorithms ranged from 0.71-0.90 for five of seven studies that reported it [Tables 5 and 6]. Unlike the diagnostic studies, the sensitivity and specificity were only reported for four of the seven studies, so a pooled analysis was not possible. Tse et al. utilized a logistic regression model trained on latent variables extracted via a non-negative matrix factorization method to predict VT/VF in BrS patients^[21]. Their model performed optimally when trained on five latent variables, giving an AUC of 0.7092 when validated on an external cohort of 227 patients. Syncope, atrial fibrillation, QRS duration, and QTc interval were significant predictors of spontaneous VT/VF. Romero et al. used a multivariate ensemble classifier to predict syncope, VF, or SCD. Their model performed the best when trained on the features of heart rate recovery (HRV) and morphological indices of QRS, with an AUC of 0.9^[23]. Randazzo et al. compared several models, including a multi-layer perceptron (MLP), boosted decision tree (BDT), decision tree, Support Vector Machine, and Naïve Bayes (NB) classifiers for the prediction of SCD or VF^[20]. All models were trained on the same dataset of 209 ECGs. However, the number of patients included and their characteristics were not reported. On validation, all models showed a high NPV, but the BDT performed the best on the basis of F1 score (0.67). Lee et al. used a random-survival forest (RSF) model trained on latent features extracted via NMF for the prediction of VT/VF^[19]. The model performed well with respect to F1 score (0.8769), sensitivity (0.8881), and PPV (0.8712). Romero et al. trained a multivariate classifier on a cohort of 110 BrS patients for the identification of novel symptom-related markers from autonomic and dynamic ECG responses during exercise testing^[23]. The best-performing model was the multivariate classifier trained on three features: (1) T-wave intervals ratio in lead V5 at baseline; (2) Ratio T-peak-T-end/QT in lead V5 at baseline; (3) T-peak-T-end interval in lead V5 at baseline. This model had an AUC of 0.796 (95%CI: 0.719-0.873) on cross-validation. Lee et al. compared the performance of several ML models with published risk scores in a multi-centered cohort study based in Hong Kong^[18]. They found that the random survival forest outperformed all other models as well as published risk scores in the prediction of VT/VF, with an AUC of 0.942 (95%CI: 0.913-0.964). Additionally, they found that P wave duration and the presence of other arrhythmias, such as atrial fibrillation (AF), mean QRS duration, and QTc intervals were predictors of spontaneous VT/VF. They suggested an additional role for atrial arrhythmias and abnormalities in ventricular repolarization in predicting adverse outcomes in BrS.

Nakamura et al. trained a CNN for the prediction of fatal arrhythmia. The model performed the best when trained on a per-patient basis, showing an AUC of 0.81 (95%CI: 0.72-0.90)^[26].

Integration of clinical factors and ECG patterns in AI models

Clinical factors can play a valuable role in enhancing the prognostic accuracy of ML algorithms. An interesting approach was that of Tse et al. who used an NMF method to extract latent features, which are relationships between clinical variables that were only discoverable after applying a dimensionality reduction technique^[21]. These latent features were then incorporated into the training of their ML model. Clinical factors associated with spontaneous VT/VF included syncope, AF, QRS duration, and QTc interval prolongation. Additionally, Lee et al. found that symptoms on initial presentation were statistically significant predictors of VT/VF during follow-up^[18]. Patients presenting with syncope or VT/VF were at increased risk for spontaneous VT/VF during follow-up at every time point. Lastly, Lee et al. performed a Cox regression using a multivariate model and found that syncope, initial VT/VF, other arrhythmias, and significant S wave in lead I were statistically significant predictors of VT/VF during follow-up^[19].

BrS and multi-modal training in medical AI models

Recently, there has been much progress in the integration of different data modalities in the training of diagnostic algorithms. For instance, Contrastive Language-Image Pre-Training (CLIP) models connect medical imaging (X-ray, MRI, CT, etc.) to medical descriptions and notes^[29]. This integration allows CLIP models to assist in automated diagnosis and medical research - both pertinent to the diagnosis of BrS. There are two types of CLIP models: (1) Medical Vision-Language Pre- Training (MED-VLP) with Frozen Language Models and Latent Space Geometry Optimization (M-FLAG) and (2) Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias (MED-UniC). M-FLAG frozen language models are pre-trained on large data sets and then are fine-tuned to accomplish specific tasks^[30]. This approach makes it easier to train the model on specific functions. M-FLAG utilizes Latent Space Geometry Optimization, a technique that optimizes the space in which data are projected. Effective space manipulation leads to improved model performance by ensuring representations of both text and image modalities are compatible and can be efficiently combined to make diagnostic predictions. In contrast, MED-UniC models involve medical data streams from multiple sources of data, such as imaging (e.g., radiography) and text data (e.g., consult notes)^[31]. Medical vision and language pre-training (MED-VLP) hopes to integrate and jointly process these data to generalize representations from large-scale medical image-text data. Subsequently, it enables a vision-and-language model to address a wide range of medical vision-and-language tasks, which can be crucial for mitigating the data scarcity problem in the medical field and aid in integrating the knowledge from pictures and text. The current literature on AI algorithms for the diagnosis and prognosis of BrS mainly incorporates only ECG data (and sometimes clinical data). The high accuracy of multi-modal AI models highlights the potential of integrating CLIP models in the diagnosis/prognosis of BrS. Future work is needed to explore this further.

AI in the diagnosis of other cardiac diseases

The utility of AI and ML in diagnosing other cardiac conditions strengthens the case for using AI models in the diagnosis and prognosis of BrS. AI and ML have helped characterize different types of heart failure with preserved ejection fraction^[32], AI-enabled ECG-based screening tool for the diagnosis of left ventricular systolic dysfunction^[33], and prediction of atrial fibrillation^[34]. ECG-based ML algorithms are being used for the diagnosis of other inherited arrhythmias, such as long QT syndrome (LQTS)^[35]. They found that among eight studies, the pooled SAUROC was 0.95 (95%CI: 0.31-1.00), sensitivity was 0.87 (95%CI: 0.83-0.90), and specificity was 0.91 (95%CI: 0.88-0.93), indicating good diagnostic performance. These metrics were slightly higher than the SAUROC calculated in our review, suggesting that ML algorithms may perform better for diagnosing LQTS compared to BrS. Another interpretation of this comparison is that algorithms for the diagnosis of BrS may not yet be well optimized, and further work must be done with larger datasets to attain higher diagnostic accuracy.

Strengths and limitations

Our study had several strengths. A thorough search of the literature and in-depth analysis was conducted. Included studies explored a variety of different machine learning algorithms. A pooled analysis was conducted to evaluate the SROC of diagnostic studies, which indicated that ML algorithms perform well in the diagnosis of BrS.

The primary limitations of our study are outlined. (1) Prognostic studies did not consistently report both sensitivity and specificity, so a pooled analysis of prognostic studies was not possible; (2) Several studies displayed an unclear or high risk of bias. There were a few reasons for this conclusion; primarily, these studies did not report patient inclusion/exclusion criteria, demographic characteristics of the patients, or the method used to diagnose BrS. Additionally, although most models underwent internal cross-validation, few were externally validated with other datasets. Therefore, there is the possibility of overfitting of the models; (3) Since BrS is a relatively rare disease, the total sample size of all included studies was not high (n = 1,868 for diagnostic studies and n = 1,859 for prognostic studies), which may limit the generalizability of our findings; (4) Clinically, BrS is often diagnosed after a drug challenge with a SCB which can sometimes unmask the type-1 BrS ECG pattern, aiding in diagnosis^[36]. Our meta-analysis was unable to stratify patients based on whether or not they received a drug challenge as there were insufficient data reported on whether this procedure was used; (5) Some studies explored the utility of ML algorithms for diagnosis/prognosis of BrS mainly based on ECG data alone. Other non-ML risk score models that incorporate clinical risk factors and ECG features exist, but these were not employed. For instance, the Shanghai scoring system incorporates ECG features, clinical history, and family history^[37]. The Sieira score is based on ECG pattern, in addition to family history of SCD and clinical presentation (e.g., syncope or aborted SCD)^[38]. Future ML models should be trained on ECG data and clinical risk factors to achieve optimal performance; (6) Another limitation is the lack of reporting on survival data according to age and time of diagnosis. Thus, we were not able to construct survival curves merging data from all studies; (7) A major concern with the use of ML algorithms as a diagnostic/prognostic tool is the potential bias in data collection for training of the model. Overrepresentations of specific demographic groups, such as by ethnic/racial groups, age groups, or gender, may also lead to overfitting of the model and loss of generalizability to other populations. For instance, the vast majority of our sample consisted of patients who were males of European descent. Therefore, caution may be necessary when interpreting the accuracy or applicability of models trained on datasets that lack diversity. Additionally, all included studies trained models on retrospective cohorts, which could serve as another source of bias. Performance metrics of models should be interpreted cautiously until the models can be validated on more robust, prospectively collected validation datasets. In the development of future models, care should be taken to address bias in data collection for model training in terms of population and data quality; (8) Lastly, since models are usually trained on high-quality databases and ECGs of well-phenotyped patients, their applicability in a "real-world clinical setting" remains to be defined^[39]. Future studies are needed to evaluate how AI algorithms can be best integrated within real-world clinical settings and whether they provide utility in improving outcomes for patients.