Cardiovascular diseases (CVDs) such as heart disease, stroke, etc. are the most deadly health issues in the world [1-3] producing a huge financial burden on societies [4-6]. Predicting the risk of CVD is essential for optimizing management and treatment guidelines. Extensive research has been conducted to develop prediction CVD models [7-9], yet their validity is a cause of concern.

The current standard of care for CVD prevention is based on risk prediction models with multiple risk factors such as age, gender, hypertension, dyslipidemia, smoking, etc. There are well-known models such as the Framingham Risk Score [10], the Prospective Cardiovascular Münster (PROCAM) score [11], the Dundee risk disk [12], the British Regional Heart Study [13,14], Korean heart study [15], SCORE [16] and the QRISK3 prediction model [17]. However, many of the risk scores used in clinics are either developed locally or are non-local models calibrated to suit the local population [18] and determine preventative measures [19]. These models have been adopted in primary care settings as simplified charts, tables, computer programs, and web-based tools, and are routinely referred to in policy documents and guidelines.

These models do not capture all cardiovascular risks: the Framingham risk score is superior to any single risk factor or model but can only predict up to 50% of CVD cases. Also, recent systematic reviews [16,20] reported that the Framingham risk score has a variable performance with under-prediction and over–prediction of the risk in high-risk and lower-risk populations, respectively. Risk overestimation causes increased costs, and risk underestimation leads to missing vulnerable cases.

Our literature review-based observations suggest many reasons that have led to a lack of reliable applied prediction models for heart disease: i) no comprehensive overview of the models has been undertaken, ii) only a few were externally validated on their predictive performance, and iii) lack of external validation makes them almost valueless for practitioners, policymakers, and guideline developers. Therefore, despite many studies [21-23] on these models, the critical element of external validation to assess performance when transported to other regions has been lacking. In this paper, we validate four major CVD risk prediction models externally on the MESA dataset.

Selection of prediction models for external validation

We used a published review on the latest use of prediction models for cardiovascular disease and how to choose the right one [18]. The selected models are among the most promising and are implemented practically in clinical settings. In addition, we used the model mentioned in the American Heart Association and the American Cardiovascular Association (AHA/ACA) guidelines [24]. In total, four CVD risk prediction models are used in this study.

Prediction models included for analysis

The various CVD risk prediction models that are selected are briefly described below. The common practice for building models is to use Cox proportional-hazards regression to form an equation that outputs the risk percentage. It is, sometimes, further simplified by making it a points-based scoring system that is easy for clinical use. In brief, the natural logs of the predictors (age, total cholesterol, HDL cholesterol, Systolic BP, etc.) are calculated. These values are then multiplied by the coefficients from the equation (usually for the specific race-gender group of the individual). The sum of these products for all factors is calculated. The estimated 10-year risk of a first hard ASCVD event is formally calculated as 1 minus the survival rate at ten years raised to the power of the exponent of the sum minus the race- and sex-specific overall mean sum. This general method is used to calculate the risks in most models.

Where S is the baseline survival, IndX’B is the coefficient X value, and MeanX’B is the overall mean of race-specific and sex-specific coefficient X value.

Framingham risk score for cardiovascular disease (FRS-CVD)

Framingham risk scores were originally developed on the data obtained from the Framingham Heart Study to estimate the 10-year risk of developing coronary heart disease, with additional risk scores for other cardiovascular diseases and events developed later [25]. In our study, we use the Framingham scores for 10-year cardiovascular disease risk [26]. The Framingham CVD risk score is a gender-specific Cox proportional-hazards regression based on age (30-74), total cholesterol (100-405 mg/dL), cigarette smoking status, HDL cholesterol (10-100 mg/dL) and systolic blood pressure (90-200 mmHg) to estimate the risk of an event in 8491 Framingham study participants between the ages 30 and 74 and free of CVD at recruitment. The point-based system to calculate the risk score is outlined in Appendices 1 and 2 for women and men.

ASSIGN risk score

The ASSIGN risk score [27] was developed on the data and cardiovascular outcomes in the Scottish Heart Health Extended Cohort (SHHEC) [28] to assess the 10-year percentage risk of cardiovascular disease (any manifestation of coronary heart disease or cerebrovascular disease including transient ischemic attacks). Participants were 6,540 men and 6,757 women aged 30-74, initially free of cardiovascular disease, ranked for social deprivation by postcode. The unique aspect of the ASSIGN score is the addition of social deprivation and family history to traditional risk factors found in the Framingham Score. It was shown to be marginally better in discriminating cases and non-cases than the Framingham Score [18]. The study provides mean scores for the components when the data is insufficient to produce a risk score. Since the MESA study does not have a social deprivation factor known as the Scottish Index of Multiple Deprivation (SIMD), we use 20 instead, suggested as the median value for SIMD by ASSIGN. The components in the model are age (25-90), sex, Scottish postcode for SIMD (or a median score of 20 for non-Scottish population), family history of CHD/Stroke, diabetes status, cigarettes smoked daily (0-100), systolic blood pressure (80-250 mmHg), total cholesterol (2-12.5 mmol/l), and HDL cholesterol (0.3 – 3.5 mmol/l).

Table 1 shows the published results based on the respective datasets used in the four risk prediction models within the performance measured by sensitivity, specificity, accuracy, and C-statistic. The top quintile is chosen for analyzing the performance measures, and for SCORE, a threshold of the top 5% (in low-risk regions of Europe) or above is shown. SCORE did not report top quintile performance in its original paper. The values are averages across population groups, and/or gender. The Framingham score is an average for four individual diseases (coronary heart disease, stroke, congestive heart failure, and intermittent claudication).

ASCVD risk score

The atherosclerotic cardiovascular disease (ASCVD) risk score is based on the recommendations and guidelines provided by the 2013 American Heart Association (AHA) and American College of Cardiology (ACC) report [24]. A total of 11,240 white women, 9,098 white men, 2,641 African-American women, and 1,647 African-American men initially free of any fatal CVD were included. The components included in this risk calculation method are age (40-79), diabetes status, gender, race, smoking status, total cholesterol (mg/dL), HDL cholesterol (mg/dL), systolic blood pressure (mmHg), and treatment for hypertension. Gender- and race-specific equations were developed to predict the risk of the first ASCVD event. The formula for calculating the ASCVD score is provided by Goff et al. 2014 [24].

SCORE (European CVD risk score)

The SCORE project [29] created a risk-scoring system for clinical use among the European population. It combined data from 12 European cohort studies, collected patient data from 250,000 subjects, and recorded about 7,000 fatal CV events [30]. The score was developed differently for two regions of Europe, categorizing them as low-risk countries (predominantly Western Europe and Scandinavia) and high-risk countries (predominantly Eastern Europe). The model used age, gender, systolic blood pressure (mmHg), total cholesterol (mmol/L), and smoking status as inputs to the final percentage risk score. SCORE provides a simple and easy-to-use chart for reference and use in clinical practice. The procedure for calculating the risk percentage using the SCORE model is based on the European guidelines for CVD Prevention in Clinical Practice [31]. We used the low-risk version of SCORE to evaluate it on the MESA dataset because the MESA demographic (USA) matches the low-risk European countries as defined by SCORE.

The population of the external validation dataset

The Multi-Ethnic Study of Atherosclerosis (MESA) [32], with a primary objective of determining characteristics related to the progression of subclinical cardiovascular disease to clinical cardiovascular disease, is a research study of over 6,000 men and women from different ethnicities in the United States. The study is sponsored by the National Heart, Lung, and Blood Institute of the NIH. In the baseline was a diverse population sample of 6,814 men and women aged 45-84 without any known CVD conditions, but who could have risk factors. About 38% of the participants were white, 28% African American, 22% Hispanic, and 12% Asian. Females comprised 53% (3,601) of the study population, whereas males comprised 47% (3,213). Participants were followed for identification and characterization of cardiovascular disease events, including acute myocardial infarction and other forms of coronary heart disease (CHD), stroke, congestive heart failure, mortality, and cardiovascular disease interventions. The subjects had their first follow-up examinations over two years from July 2000 to July 2002, with a follow-up period of approximately 17-20 months, and were followed until their fifth exam before the dataset was made available. The fifth examination took place from April 2010 to January 2012. The baseline characteristics of the study participants who did/did not have incident CVD in the follow-up visits are shown in Table 2.

Assessment of cardiovascular diseases

In the MESA study, the data regarding cardiovascular diseases were collected from a variety of sources, including death certificates, medical records, in-person interviews, autopsy reports, and actively contacting kin or doctor of the subject in case of death or occurrence of severe CVD. Fatal and non-fatal cardiovascular events are defined based on MESA criteria. Fatal events are further classified as definite fatal CHD, definite fatal MI, possible fatal CHD, stroke death, non-coronary/non-stroke death, and other cardiovascular disease death. Non-fatal events include MI, resuscitated cardiac arrest, angina, congestive heart failure, peripheral vascular disease, stroke, and TIA. In our study, we assess prediction models for overall CVD (fatal and non-fatal events) in the MESA dataset.

Assessment of predictors

The prediction models selected for this study use a variety of features/component variables for predicting CVD, such as socio-demographic data, medical history, family history, and social deprivation. Most of these parameters are readily available in the MESA dataset. The missing data were replaced by population means and by extrapolating data whenever possible. For, when age is unknown at a particular visit, it is extrapolated from the previous visit. If a subject has any missing critical information that cannot be estimated by reliable means, then the subject is removed from the assessment. Otherwise, the population means segregated by age and gender were used to replace the parameters, as shown in Appendix 4.

Framingham risk score takes the age (30-79), total cholesterol (mg/dL), HDL (mg/dL), systolic blood pressure (mmHg), and smoking status as input for the CVD risk model. This model is race-specific and is defined for non-Hispanic whites and African Americans. However, the study recommends using the risk model developed for non-Hispanic whites for an estimation of risk in populations other than African Americans and non-Hispanic whites.

The ASSIGN CVD score uses age 30-74, sex, family history of CHD/stroke, diabetes status, cigarette smoking status, systolic blood pressure (100-200 mmHg, extrapolated if used beyond this range), total cholesterol, HDL, and a derived social deprivation factor called SIMD. For the local Scottish population, rheumatoid arthritis is also a component. However, the score has been built without an arthritis element for outsiders, and we make use of that score. In our study data, the social deprivation factor is not available, and hence, a suggested score of 20 (middle of the most deprived fifths of the population) is used in place. The provisional values for unknown components are provided by the ASSIGN study and outlined in Appendix 3.

The ASCVD score based on ACC/AHA uses age (40-79), diabetes status, sex, race (African-American and Non-Hispanic White), smoking status, total cholesterol (mg/dL), HDL cholesterol (mg/dL), systolic blood pressure (mmHg) and treatment for hypertension as predictors. ASCVD equations are sex- and race-specific equations. They should be used for non-Hispanic whites and African Americans. However, the study recommends that equations for non-Hispanic whites may be considered for an estimation of risk in patients from populations other than African Americans and non-Hispanic whites.

The European SCORE model uses age, gender, systolic blood pressure (mmHg), total cholesterol (mmol/L), and smoking status as input variables. Subjects with missing data for these predictors are removed.

We implemented each model in a computer program based on published algorithms, and 100 incident CVD and 100 non-incident CVD subjects from MESA were selected for testing. To be sure that our simulation accurately reflected real-world clinical applications, the same MESA data were also tested on the original models available on the web and results were compared to our implemented model.

Statistical analyses

To compare the various models, we used 933 subjects with incident CVD and 5,785 subjects without any incident CVD. We computed the results with measures such as the area under the curve (AUC or C-statistic), sensitivity, specificity, and accuracy to estimate the performance. First, we analyzed the AUC to assess the discriminatory capability of the risk scores to correctly differentiate between two subjects, one who will develop the condition within ten years and one who will not. Second, we assess the agreement between the risks predicted by the models and the observed outcomes in the study by measuring the sensitivity and specificity of the models.

Implemented models compared to public web-based applications

The outcomes in each implementation of a given risk score achieved 100% correlation with the results of public web-based applications.

Discrimination

The models showed similar C-statistics or AUC at the 10-year prediction of the disease with little variation. ASCVD showed slightly better discriminative ability with an AUC of 0.717 and SCORE showed the least (AUC 0.677) although the difference is not very big. Figure 1 shows the receiver operating characteristics of the four models and the associated areas under curve (AUC or C-statistics).

Calibration

Figure 2 shows the calibration plots on each of the risk scores. The calibration curve is very helpful in evaluating a model's performance and reliability in each population, especially when the score is developed on another population. These curves are obtained by plotting actual probabilities vs. predicted probabilities. Figure 2 depicts the calibration curve where the x-axis represents the predicted probabilities in 8 divided groups (also called bins that equal range of probabilities) are plotted against the y-axis which contains true probabilities of each of the groups. The true probabilities are obtained by taking the fraction of positives (CVD cases) in the respective groups. For example, if a group has a total of 10 subjects and 2 of them are CVD cases and the other 8 are non-CVD cases, then the true probability for that group is 20% (or 0.2). For an ideal model, this probability is a non-decreasing quantity matching the predicted probabilities (along the x-axis) represented by the black straight line with slope one starting at the origin of the graph.

The four calibration curves for the four prediction models are shown in Figure 2 drawn in a different color for different models. From the curve for the Framingham score, we can see that as true probabilities increase, the predicted probabilities increase disproportionately. This shows that the Framingham score overestimates the CVD risk in the MESA population. This is also the same in SCORE, whose predicted probabilities are higher than true probabilities, but slightly better than Framingham, by being closer to the ideal calibration curve. On the other hand, ASCVD and ASSIGN scores are in closer agreement with the ideal calibration curve (the black line), and hence, the performance is better for the two scores in this population.

The models are also assessed by computing the sensitivity and specificity by taking the “high-risk” categories in each of the four models. We use a threshold value of the top 20% (predicted probability risk) to evaluate the calibration of the models. Table 3 shows an example of the discrimination capabilities of the models within the different sensitivities and specificities for this threshold. At this cut-off, the Framingham model shows a high sensitivity of 69.67% but with a low specificity of 62.21%. The ASCVD model shows a sensitivity of 46.73% and a specificity of 79.69%. These results contrast with the European models' SCORE and ASSIGN, which show relatively low sensitivities of 43.19% and 34.94%, but with higher specificities of 80.07% and 85.12%, respectively, for the said threshold. We note that the results will vary for different percentages of the threshold. The strict comparison can be depicted in Figure 2.

In this study, we have shown the performance of various cardiovascular risk scores in a multi-ethnic dataset. This validation study showed that the performance of four models for predicting CVD in a multi-ethnic population was nearly identical to each other. We observed that the high-risk candidates from one prediction model could easily be left out in another prediction model. This also shows that no model could precisely and accurately predict the absolute risk of cardiovascular diseases.

The discriminative abilities of the models varied minimally. Calibration problems can be mitigated by moving the thresholds, although this may lead to over-treatment in the case of identification of too many cases or under-treatment in case of too few cases identified. Compared to the Framingham model, all other models show poor calibration and need to be updated if it is used for the demographic represented by MESA study participants.

Our study found that the transportability of the predicted risks was generally poor from European scores to the American population. With the highest AUC and a good calibration curve compared to the other scores, ASCVD, which is based on the recommendation of the American Heart Association performs the best in our calibration experiments.