Advertisement

Assessing the external validity of the SAFEHEART risk prediction model in patients with familial hypercholesterolaemia in an English routine care cohort

  • Ailsa J. McKay
    Correspondence
    Corresponding author.
    Affiliations
    Imperial Centre for Cardiovascular Disease Prevention, Department of Primary Care and Public Health, Imperial College London, St Dunstan's Road, London, UK
    Search for articles by this author
  • Laura H. Gunn
    Affiliations
    Department of Public Health Sciences and School of Data Science, University of North Carolina at Charlotte, Charlotte, NC, USA

    Department of Primary Care and Public Health, School of Public Health, Imperial College London, UK
    Search for articles by this author
  • Kausik K. Ray
    Affiliations
    Imperial Centre for Cardiovascular Disease Prevention, Department of Primary Care and Public Health, Imperial College London, St Dunstan's Road, London, UK
    Search for articles by this author
Open AccessPublished:July 20, 2022DOI:https://doi.org/10.1016/j.atherosclerosis.2022.07.011

      Highlights

      • There are no well validated cardiovascular risk prediction tools for people with familial hypercholesterolaemia.
      • The SAFEHEART model was recently developed in a bespoke genetically defined cohort.
      • Outcomes departed substantially from predicted risks in a routine English cohort.
      • A recalibrated model had moderate utility across the 10–30% predicted risk range.
      • The model would likely be of limited utility in current practice in England.

      Abstract

      Background and aims

      The SAFEHEART tool has shown good discrimination in predicting cardiovascular events in a bespoke genotyped cohort with familial hypercholesterolaemia (FH). We assessed whether the tool could aid clinical decision making in an English routine care cohort with FH.

      Methods

      A historical (2000–2017) open cohort of 3643 participants aged 18–79 years and ≥6-months since FH diagnosis was derived from the Clinical Practice Research Datalink. Individual 10-year cardiovascular risks were predicted using the SAFEHEART model, with multiple imputation used to manage missing data. Outcomes were the first occurrence of myocardial infarction, coronary revascularisation, ischaemic stroke, carotid revascularisation, peripheral arterial revascularisation, non-traumatic lower limb amputation, or cardiovascular death. Model performance was assessed using standard measures of calibration and discrimination, and decision curve analysis.

      Results

      147 outcome events were observed over a median 3.73 (IQR 1.59–6.48) years follow-up. While the model had some discriminatory value (Harrell's c-statistic 0.67 (95% CI 0.61–0.72)), observed outcome risks departed substantially from predicted risks. Calibration slopes for men and women by age decile were 10.09 (95% CI 7.40–12.77) and 2.85 (1.25–4.45), respectively. Recalibration-in-the-large led to closer alignment of observed and predicted risks (recalibration slopes 3.48 (2.55–4.41) and 1.14 (0.50–1.79), respectively). Decision curve analysis suggested the recalibrated model had net benefit at predicted risks of 10–30%.

      Conclusions

      The original SAFEHEART model has limited generalisability to the routinely identifiable English primary care FH population. With recalibration it appears to have moderate utility at 10–30% predicted risk. It may have greater validity in more bespoke genetically defined FH populations.

      Graphical abstract

      Keywords

      1. Introduction

      Familial hypercholesterolaemia (FH) is associated with a coronary heart disease risk which is much greater than that observed in the general population[
      • Brown M.S.
      • Goldstein J.L.
      A receptor-mediated pathway for cholesterol homeostasis.
      ,
      • Benn M.
      • Watts G.F.
      • Tybjaerg-Hansen A.
      • et al.
      Familial hypercholesterolemia in the Danish general population: prevalence, coronary artery disease, and cholesterol-lowering medication.
      ]. After diagnosis, long-term lifestyle and pharmacological lipid lowering strategies are recommended to reduce lifetime risk [
      • Nordestgaard B.G.
      • Chapman M.J.
      • Humphries S.E.
      • et al.
      Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease Consensus Statement of the European Atherosclerosis Society.
      ,
      • Watts G.F.
      • Gidding S.
      • Wierzbicki A.S.
      • et al.
      Integrated guidance on the care of familial hypercholesterolaemia from the International FH Foundation.
      ]. Statins are the mainstay of lipid modification therapy (LMT), providing a highly cost-effective means of reducing LDL cholesterol (LDL-C) for many patients. Recent studies suggest that statins have the potential to bring FH population-level coronary heart disease risk into line with that of the general population [
      • Neil A.
      • Cooper J.
      • Betteridge J.
      • et al.
      Reductions in all-cause, cancer, and coronary mortality in statin-treated patients with heterozygous familial hypercholesterolaemia: a prospective registry study.
      ,
      • Versmissen J.
      • Oosterveer D.M.
      • Yazdanpanah M.
      • et al.
      Efficacy of statins in familial hypercholesterolaemia: a long term cohort study.
      ]. However, this will not be achieved at individual level for many, as risk of atherosclerotic cardiovascular disease (ASCVD) is wide-ranging and often diagnosis is made late [
      • Oosterveer D.M.
      • Versmissen J.
      • Schinkel A.F.
      • et al.
      Clinical and genetic factors influencing cardiovascular risk in patients with familial hypercholesterolemia.
      ]. Ability to identify those patients with residual excess risk (i.e. despite statin therapy), and therefore potential to benefit from additional treatment, has become a practical issue as novel LMTs such as the proprotein convertase subtilisin/kexin type 9 (PCSK9) inhibitors have emerged. Currently, in the UK, recommendations for add-on therapies are based on coarse population-based risk estimates and LDL-C levels [
      National Institute for Health and Care Excellence
      Alirocumab for treating primary hypercholesterolaemia and mixed dyslipidaemia.
      ,
      National Institute for Health and Care Excellence
      Evolocumab for treating primary hypercholesterolaemia and mixed dyslipidaemia.
      ], which will only partially reflect individual ASCVD risk and therefore the potential benefits and indication for treatment. As net treatment benefit will be achieved only where individual risk is sufficient to offset the potential risks and opportunity cost of treatment, a method to estimate individual absolute cardiovascular event risk is desirable.
      Whilst tools enabling estimation of individual vascular event risk in the general population are well-established, and have been recommended as part of routine preventive care for several years [
      National Institute for Health and Care Excellence
      CG181: Cardiovascular Disease: Risk Assessment and Reduction Including Lipid Modification.
      ], they have not been validated for the population with FH. A tool aiming to predict event risks for the population with FH (the SAFEHEART model) was recently developed in a multicenter Spanish cohort with an established genetic FH diagnosis [
      • Pérez de Isla L.
      • Alonso R.
      • Mata N.
      • et al.
      Predicting cardiovascular events in familial hypercholesterolemia.
      ]. A recent French study suggested potentially limited generalisability of the original model [
      • Gallo A.
      • Charriere S.
      • Vimont A.
      • et al.
      SAFEHEART risk-equation and cholesterol-year-score are powerful predictors of cardiovascular events in French patients with familial hypercholesterolemia.
      ]. We aimed here to assess model performance in a routine English primary care population using data from the CPRD.

      2. Patients and methods

      2.1 Study design and data sources

      We undertook a historical cohort study using data from the CPRD GOLD database, which holds longitudinal data collected during the routine activities of participating UK general practices from 1987 onwards. Data are available for more than 18 million patients, and Hospital Episode Statistics (HES) and Office for National Statistics (ONS) data linkage is available for most patients in England. Most UK cardiovascular disease (CVD) primary prevention work is undertaken in primary care, and CPRD has previously been used to undertake many observational studies, including derivation and external validation of CVD risk prediction models [
      • Rapsomaniki E.
      • Shah A.
      • Perel P.
      • et al.
      Prognostic models for stable coronary artery disease based on electronic health record cohort of 102 023 patients.
      ,
      • Collins G.S.
      • Altman D.G.
      Predicting the 10 year risk of cardiovascular disease in the United Kingdom: independent and external validation of an updated version of QRISK2.
      ].
      Our cohort inclusion criteria required that participants had HES and ONS data linkage. Otherwise, cohort entry occurred on the first date post January 1, 2000 (start of observational period) that all the following criteria were met: first anniversary of registration with participating practice, first anniversary of database entry, age ≥18 and < 80 years, and ≥6-months since a FH diagnosis. Probable and unspecified FH diagnoses were included, whereas ‘possible FH’ was not. The requirement for ≥6-months since diagnosis was applied under the assumption that this would lead to a cohort on established preventive therapy. Follow-up was censored at the first date of occurrence of any of: vascular outcome occurrence (defined as below), last data upload, transfer out of database, ONS death registration, receipt of palliative care input, or December 31, 2017 (end of observational period). The code lists applied in our cohort definition are available in Supplementary File 1.

      2.2 Outcome variable

      The cardiovascular events included within our outcome definition were: myocardial infarction, coronary revascularisation, ischaemic stroke, carotid revascularisation, peripheral arterial revascularisation, non-traumatic lower limb amputation, and cardiovascular death, as per CPRD, HES or ONS record (see code lists in Supplementary File 1). Each individual's first such outcome recorded post-cohort entry was used to create the variable.

      2.3 Predictor variables

      As per the derivation study report [
      • Pérez de Isla L.
      • Alonso R.
      • Mata N.
      • et al.
      Predicting cardiovascular events in familial hypercholesterolemia.
      ], and personal communication with the study team, the predictor variables were: 1) age-group (<30, 30–59, or ≥60 years), 2) sex, 3) atherosclerotic CVD history (history at baseline of ischaemic heart disease (angina, myocardial infarction, percutaneous coronary intervention, or coronary artery bypass), cerebrovascular disease (ischaemic stroke, TIA or carotid revascularisation), peripheral vascular disease or related intervention (peripheral artery angioplasty or bypass, or lower limb amputation), or AAA), 4) hypertension (recorded diagnosis or treatment prescribed within 6 months preceding baseline), 5) BMI category (<25, 25–29.9, or ≥30 kg/m2), 6) smoking status (current smoker), 7) LDL-C category (<100, 100–159, ≥160 mg/dL), and 8) lipoprotein(a) category (>50 or ≤50 mg/dL). The relevant code lists are available in Supplementary File 1. For the BMI, smoking and LDL-C variables, the first measurement recorded post-cohort entry (and within 10-years of cohort entry) was used in assessment of the cohort. As lipoprotein(a) values were rarely available in the CPRD, these were assumed to be missing and imputed as described below. Other than the lipoprotein(a) data issue, the timing of the clinical measurements represented the main difference between our predictor ascertainment methods and those used in model derivation. Moreover, as our data were routine, measurement methods may have differed between participants.

      2.4 Additional variables of interest

      Additional to the above, variables describing use of LMT, antihypertensive, anti-platelet and anti-coagulant therapies at baseline (any relevant prescription within the preceding 6-months), diabetes status (record of type 1 or type 2 diabetes mellitus by baseline) and ethnicity, were defined (see code lists in Supplementary File 1). The latest recorded ethnicity in the CPRD – or HES where unavailable in CPRD – was used to generate a categorical variable based on the five Level 1 ONS ethnic group classifications derived for the 2001 Census.

      2.5 Statistical analysis

      The cohort was described in terms of both the predictors and additional variables of interest. Data missingness was explored, and missing BMI, smoking and LDL-C values were imputed in a multiple imputation procedure, with all predictors included in the imputation process. Lipoprotein(a) category was imputed as per the categorisation of the mean lipoprotein(a) value observed in the derivation dataset for each age*sex group. The analyses described below were performed for each imputed dataset, and the results were combined according to Rubin's rules.
      The SAFEHEART equation (fully specified in Supplementary File 2) was used to estimate 10-year CVD event risk for each participant. Their distribution was reviewed and calibration assessed by: 1) comparing expected versus observed event risk by predicted risk decile, 2) examining calibration plots, 3) reviewing survival curves by risk group (<10%, 10 to <20%, 20 to <30%, and ≥30% 10-year predicted risk), and 4) estimating calibration slope. Harrell's c-statistic was used to assess discrimination. The outcomes of these model accuracy assessments were compared with those observed at model derivation. A decision curve analysis considered the net benefit of using the SAFEHEART tool to aid treatment decisions (compared with treating all or none) for a range of potential treatment thresholds. In order to correct for population differences while maintaining the original study parameters, results were compared to those obtained from recalibrating the model to correct the calibration in the large [
      • Janssen K.J.M.
      • Moons K.G.M.
      • Kalkman C.J.
      • et al.
      Updating methods improved the performance of a clinical prediction model in new patients.
      ].
      A sensitivity analysis was performed using the alternative SAFEHEART model for estimating 5-year CVD event risk (see Supplementary File 2 for this model). The aforementioned calibration and discrimination assessments were performed, as well as the decision curve analysis and recalibration.

      3. Results

      3.1 Data availability and cohort characteristics

      3643 participants from 344 practices, with a median 3.73 years follow-up (interquartile range (IQR) 1.59–6.48) and 359 participants (9.85%) with ≥10 years follow-up, were eligible for inclusion in the cohort. 147 (4.04%) experienced a vascular outcome event, with corresponding 10-year event risks of 6.25% (95% CI 4.20%–8.27%) and 11.28% (95% CI 8.37%–14.09%) for males and females, respectively. Cohort baseline characteristics are displayed in Table 1. 62.75% of the cohort were 30–59 years old, while 30.44% were aged ≥60 years at baseline, and 71.95% of participants were overweight or obese. Less than a quarter of participants (23.02%) had LDL-C <100 mg/dL, whereas 46.38% and 30.60% had LDL-C between 100 and 159 and ≥ 160 mg/dL, respectively. 7% had an ASCVD history, and 29.21% of the cohort had hypertension at baseline. Data missingness is described by variable in Table 1, and by participant in Supplementary File 4.
      Table 1Baseline characteristics.
      Baseline characteristics (n = 3643)
      N (% of non-missing)Mean (SD)Median (IQR)Missing (% of total)
      Age3643 (100%)51.88 (13.40)52.60 (43.45–61.94)0 (0%)
      Age-group (years)<30248 (6.81%)23.52 (4.19)23.84 (19.10–27.10)
      30–592286 (62.75%)47.70 (7.69)48.64 (42.23–54.09)
      60+1109 (30.44%)66.84 (4.91)65.83 (62.85–69.98)
      Male2075 (56.96%)0 (0%)
      ASCVD history255 (7.00%)0 (0%)
      Hypertension1064 (29.21%)0 (0%)
      BMI (kg/m2)2226 (100%)
      BMI was not available for nine individuals for whom BMI category (<25, 25–29.9, 30+) was available.
      28.20 (5.47)27.40 (24.60–31.00)1417 (38.90%)
      BMI (kg/m2)<25627 (28.05%)
      BMI was not available for nine individuals for whom BMI category (<25, 25–29.9, 30+) was available.
      22.71 (1.87)23.10 (21.80–24.10)
      25–29.9928 (41.52%)
      BMI was not available for nine individuals for whom BMI category (<25, 25–29.9, 30+) was available.
      27.44 (1.43)27.40 (26.20–28.70)
      30+680 (30.43%)
      BMI was not available for nine individuals for whom BMI category (<25, 25–29.9, 30+) was available.
      34.71 (4.73)33.40 (31.60–36.30)
      Current smoker566 (21.60%)1023 (28.08%)
      LDL-C (mg/dL)2320 (100%)141.52 (56.87)131.48 (100.54–170.92)1323 (36.32%)
      LDL-C (mg/dL)<100534 (23.02%)80.39 (14.83)85.07 (73.47–92.81)
      100–1591076 (46.38%)127.33 (17.13)127.42 (112.14–140.47)
      160+710 (30.60%)209.01 (47.82)194.51 (177.88–225.45)
      Diabetes diagnosis129 (3.54%)0 (0%)
      EthnicityWhite2850 (92.02%)
      Asian140 (4.52%)
      Black37 (1.19%)
      Mixed18 (0.58%)
      Other52 (1.68%)
      Missing546 (14.99%)
      LMT use2506 (68.79%)0 (0%)
      Antihypertensive use970 (26.63%)0 (0%)
      Anticoagulant use25 (0.69%)0 (0%)
      Antiplatelet use333 (9.14%)0 (0%)
      ASCVD: atherosclerotic cardiovascular disease; BMI: body mass index; LMT: lipid modifying therapy.
      a BMI was not available for nine individuals for whom BMI category (<25, 25–29.9, 30+) was available.

      3.2 Model performance: discrimination and calibration

      Prior to recalibration, the median 10-year SAFEHEART-predicted event risk was 1.53% (IQR 0.78%–2.88%), with 96% of the cohort having less than 10% risk (Fig. 1A). This substantial departure from observed risk was apparent across groups based on both sex*predicted risk decile and sex*age-group decile (see calibration plots in Fig. 1B and C, respectively). Fitted calibration slopes for men and women by 10-year predicted risk decile were 1.23 (95% CI 0.97–1.49) and 3.17 (95% CI 2.21–4.13), respectively; while those for men and women by age decile were 10.09 (95% CI 7.4–12.77) and 2.85 (1.25–4.45), respectively (see Supplementary File 5); again demonstrating considerable risk underestimation for men and women by age decile, as well as for women by risk decile, prior to model recalibration. Table 2 presents ratios of 10-year predicted to observed risks by sex across deciles of predicted risk and age. Similar concerns about performance of the original model were apparent when considering the variability in observed risks (risk reclassification based on individual risk scores is provided in Table 3), and when considering model discrimination, (Harrell's c-statistic was 0.67 (95% CI 0.61–0.72), as compared to the original derivation cohort figure of c = 0.85 [
      • Dorresteijn J.A.N.
      • Visseren F.L.J.
      • Wassink A.M.J.
      • et al.
      Development and validation of a prediction rule for recurrent vascular events based on a cohort study of patients with arterial disease: the SMART risk score.
      ].
      Fig. 1
      Fig. 1Model performance before calibration.
      The top (A) panel displays the distribution of 10-year SAFEHEART model-predicted cardiovascular risk overall and by sex*age decile (bottom). 10-year SAFEHEART model-predicted cardiovascular risks and corresponding Kaplan-Meier observed risks by predicted risk decile (B) and age decile (C), for both women (top) and men (bottom). The diagonal lines correspond to a perfect fit. The top (D) panel displays the percentage of the population who would be treated (dashed line) and percentage of those who would have future cardiovascular events covered (solid line) as functions of utilizing the different displayed 10-year predicted risks as treatment thresholds. The top-left corner corresponds to the ‘treat all’ scenario (treat all individuals with 10-year predicted risk above a 0% threshold), while the bottom-right corner corresponds to the ‘treat none’ scenario (treat only those with a predicted risk of 100%). The bottom (D) panel displays the net benefit of the SAFEHEART model (solid line) against the treat all (dashed) and treat none (dotted) approaches.
      Table 2Ratios, by sex, of 10-year non-calibrated and calibrated SAFEHEART-predicted to observed risks by 10-year predicted risk score decile and by observed age decile (where decile 1 = lowest predicted risk or lowest age decile). 10-year predicted risks are defined as within-decile SAFEHEART-derived risk averages.
      DecileNon-calibratedCalibrated
      Risk decileAge decileRisk decileAge decile
      MenWomenMenWomenMenWomenMenWomen
      10.090.04N/A0.470.290.12N/A1.53
      21.460.120.491.584.820.401.545.04
      30.380.080.115.881.240.280.3718.02
      40.260.180.280.360.840.590.891.13
      50.250.130.400.410.810.421.251.22
      60.510.140.160.581.650.470.511.79
      70.630.140.210.512.040.470.681.42
      80.690.100.170.642.190.320.501.84
      90.640.280.150.302.000.890.450.85
      100.720.270.110.531.820.790.331.43
      Table 310-year predicted risk range and corresponding Kaplan-Meier observed risks as well as the count and corresponding percentage of the sample within each range, for both the non-calibrated and calibrated SAFEHEART model.
      10-year predicted risk (%)Non-calibratedCalibrated
      Observed risk (%)n (%)Observed risk (%)n (%)
      <107.193507 (96.27%)6.372845 (78.09%)
      10 to < 2040.0287 (2.39%)9.54531 (14.58%)
      20 to < 3028.2230 (0.82%)14.42133 (3.65%)
      ≥300.1619 (0.52%)36.32134 (3.68%)
      Model performance was markedly enhanced following model recalibration (see Table 2, Table 3). The median 10-year recalibrated SAFEHEART-predicted event risk was 5.03% (IQR 2.58%–9.28%), with a wide risk distribution (Fig. 2A) and an average value of 8.03%. Calibration plots are provided in Fig. 2B and C, which show that higher predicted risks are associated with generally higher observed risks, though with over-prediction of risks for men for higher predicted risk deciles and under-prediction for higher age deciles. Supplementary File 5 also shows that the fitted recalibrated slopes for men and women across risk deciles were 0.48 (95% CI 0.37–0.60) and 1.11 (95% CI 0.78–1.44), respectively, with the recalibrated slopes across age deciles of 3.48 (95% CI 2.55–4.41) and 1.14 (95% CI 0.50–1.79) for men and women, respectively.
      Fig. 2
      Fig. 2Model performance after calibration.
      The top (A) panel displays the distribution of 10-year SAFEHEART model-predicted cardiovascular risk overall and by sex*age decile (bottom). 10-year SAFEHEART model-predicted cardiovascular risks and corresponding Kaplan-Meier observed risks by predicted risk decile (B) and age decile (C), for both women (top) and men (bottom). The diagonal lines correspond to a perfect fit. The top (D) panel displays the percentage of the population who would be treated (dashed line) and percentage of those who would have future cardiovascular events covered (solid line) as functions of utilizing the different displayed 10-year predicted risks as treatment thresholds. The top-left corner corresponds to the ‘treat all’ scenario (treat all individuals with 10-year predicted risk above a 0% threshold), while the bottom-right corner corresponds to the ‘treat none’ scenario (treat only those with a predicted risk of 100%). The bottom (D) panel displays the net benefit of the SAFEHEART model (solid line) against the treat all (dashed) and treat none (dotted) approaches.

      3.3 Clinical usefulness: decision curve analysis

      Fig. 1, Fig. 2D (top panel) display the proportion of the population who would be treated under each possible predicted risk threshold, and the proportion of cardiovascular events covered within that population, for the non-calibrated and calibrated models, respectively. Treatment thresholds range from 0% (i.e. treat all) to 100% (i.e. treat none). The distance between the curves, when describing the ratio of true positives to false positives by threshold, offers information about the clinical usefulness of the SAFEHEART model across potential thresholds. Our decision curve analysis (Fig. 1, Fig. 2D (bottom panel) – again for the non-calibrated and calibrated models, respectively) compared the SAFEHEART model with treatment benchmarks (treat all and treat none). The range over which SAFEHEART provides net benefit over treat all and treat none alternatives for the non-calibrated analyses applies to a very small group given the substantial under-prediction of risk from the model. Fig. 2D (bottom panel) illustrates the increasing net benefit associated with using the recalibrated SAFEHEART-predicted risks among patients with predicted risks thresholds in the range of approximately 10%–30% when compared to treat-all and treat-none alternatives.

      3.4 Sensitivity analysis: 5-year risk prediction model performance

      When performing the sensitivity analysis using the 5-year SAFEHEART model, 99% of individuals had <10% predicted risk over that period, and only 2 individuals (0.05%) experienced risks above 30% (Supplementary File 6). Over 65% had a 5-year predicted event risk less than 1%. Fitted calibration slopes continue to demonstrate underestimation of risks, which was also more substantial among women, who experience larger event risks as indicated with 10-year predictions (Supplementary File 7). Calibration slopes are as follows: (1) Women by risk decile is 8.56 (95% CI 3.24–13.88); (2) Men by risk decile is 1.57 (95% CI 1.24–1.89); (3) Women by age decile is 1.4 (95% CI -0.47-3.27); and (4) Men by age decile is 8.39 (95% CI 5.09–11.68). Supplementary File 8 presents ratios of 10-year predicted to observed risks by sex across deciles of predicted risk and age. Harrell's c-statistic for 5-year predictions is 0.66 (95% CI 0.60–0.71), aligning with that obtained for 10-year predictions (0.67; 95% CI 0.61–0.72). Supplementary File 9 contains the same plots reported in Fig. 1 but now for 5-year SAFEHEART predictions. Positive net benefits against treat all and treat none alternatives are observed for thresholds between 3% and 15%. Using SAFEHEART's 5-year model, the group of individuals for whom the model provides net benefit continues to be very small, with only 7% of individuals estimated to have 5-year SAFEHEART predicted event risks above 3%. Supplementary File 10 contains plots for the recalibrated 5-year SAFEHEART model. Upon recalibration, results improved in line with enhancements experienced after recalibration of the 10-year SAFEHEART model (Supplementary Files 6–8). Calibration slopes are as follows: (1) Women by risk decile is 2.85 (95% CI 1.07–4.64); (2) Men by risk decile is 0.59 (95% CI 0.46–0.71); (3) Women by age decile is 0.54 (95% CI -0.18-1.25); and (4) Men by age decile is 2.96 (1.80–4.11), see Supplementary File 7. Net benefits are achieved between thresholds of 3% and 22% of recalibrated 5-year predicted risk.

      4. Discussion

      We undertook a historical cohort study aiming to assess the external validity of the SAFEHEART risk estimation tool in a routine English primary care practice population. So far as we are aware, there are no previous reports of external validation of the original SAFEHEART model in routine healthcare datasets. We found that the 10-year model substantially underpredicted observed risk, as did the 5-year model in the sensitivity analysis, and did not reflect observed risk variability (with associated lower discrimination than observed in the original cohort) within our sample. Recalibration enhanced model performance, although with some under-prediction for men at higher age deciles. Sensitivity analysis results using the 5-year model to predict event risks present the same need for recalibration as the 10-year model. Net benefit (compared with treat-none and treat-all alternatives) was limited prior to recalibration, but apparent across the 10%–30% 10-year and 3%–22% 5-year risk ranges, respectively, for the recalibrated models.
      The key difference between our sample and that used in model derivation and internal validation was our more heterogeneous FH sample, which permitted any FH diagnosis per the primary care record, and so did not restrict to those with only molecular diagnoses. In keeping with this, we observed lower mean baseline LDL-C concentrations (141.52 (SD 56.87) mg/dL vs. 177.8 (60.4) mg/dL), a lower frequency of LMT use (68.8% vs. 84.2%), and less pre-existing ASCVD (7.0% vs. 12.8%), despite a slightly older (51.88 (13.40) vs. 45.5 (15.4) years), and more predominantly male (57.0% vs. 45.2%) sample [
      • Pérez de Isla L.
      • Alonso R.
      • Mata N.
      • et al.
      Predicting cardiovascular events in familial hypercholesterolemia.
      ]. However, there was a wide (substantially overlapping) range of LDL-C measurements in the cohorts [
      • Pérez de Isla L.
      • Alonso R.
      • Mata N.
      • et al.
      Predicting cardiovascular events in familial hypercholesterolemia.
      ], hence the observed variation in mean LDL-C concentrations between the derivation cohort and the cohort studied here is unlikely to fully explain the considerable underprediction of risk we observed with the SAFEHEART tool. Systemic differences in capturing age, sex, weight, height or smoking status are unlikely, and we were careful to follow the definition of established ASCVD used at derivation, although acknowledge that some differences in clinical application and/or recording are possible. Similarly, systematic differences in blood pressure and cholesterol measurement are possible, but unlikely to have led to the results observed. Given the relatively small sample and low number of outcome events available for use in model derivation, an important concern is that the included covariates may have relied excessively on a specific correlation structure that is not generalisable to other populations. It has been suggested that effective samples sizes of over 100 and ideally >200 events would be available to allow reasonable assessment of external validation [
      • Vergouwe Y.
      • Steyerberg E.W.
      • Eijkemans M.J.C.
      • et al.
      Substantial effective sample sizes were required for external validation studies of predictive logistic regression models.
      ,
      • Collins G.S.
      • Ogundimu E.O.
      • Altman D.G.
      Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.
      ]. Higher numbers still are preferable to permit rigorous model derivation, as model overfitting is more likely in small samples, and not necessarily easy to detect [
      • Steyerberg A.W.
      Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating.
      ]. That the SAFEHEART tool predicts a 2% 10-year event risk for a woman of ≥60 years, with BMI 22 kg/m2, LDL-cholesterol 100–159 mg/dL, a history of ASCVD, but no other risk factors (and 3.2% 10 year risk for a smoker with otherwise the same characteristics), may be an indication of internal model concerns [
      • Pérez de Isla L.
      • Alonso R.
      • Mata N.
      • et al.
      Predicting cardiovascular events in familial hypercholesterolemia.
      ]. For comparison, the predicted 10-year risk for a woman aged 70 years, without CVD, with a cholesterol/HDL ratio of two, systolic blood pressure of 120 mmHg, and no other risk factors is 9.4% per the well-established QRISK2 model [
      • Hippisley-Cox J.
      • Coupland C.
      • Vinogradova Y.
      • et al.
      Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2.
      ]. The original SAFEHEART cohort was derived from specialist centres managing FH with a molecular diagnosis who may have been better managed than in generalist settings. Furthermore, some of the patients in our cohort classified as FH may have had alternative causes for elevated cholesterol and were potentially misclassified.
      We expect, per historical UK guidance and practice, the existing FH diagnoses we used to be largely based on the Simon Broome criteria. Therefore, we anticipate those with genetic diagnoses to be included, but there will be some who meet the criteria who do not have a known mutation, some of whom may have polygenic disease. And, while the different groups will be on somewhat different risk trajectories related to differing lifelong cholesterol exposure, the population represents those currently defined as 'FH diagnosed' per UK guidance, and those for whom other risk equations are not validated. It is the population who would be routinely returned by any automated search for UK FH patients. Thus, even if not the same as the derivation cohort, it is the major readily identifiable FH population in the UK, and thus the overall impact of the clinical implication of our study, including applicability and generalisability of risk scores to this population, is of high interest to those working in this context.
      Regarding the potential utility of the recalibrated model as compared to other decision-making tools at risk thresholds of 10–30%, further work is required to determine whether these are thresholds at which decisions of interest are to be made. Although statins are cost-effective at such levels of risk [
      National Institute for Health and Care Excellence
      CG181: Cardiovascular Disease: Risk Assessment and Reduction Including Lipid Modification.
      ], thresholds at which potential novel therapies would be cost-effective may well be outside of this range. For those with polyvascular disease, discounted price PCSK9 inhibitors have been found only to be cost-effective where on-treatment LDL-C levels exceed 3.5 mmol/L[
      National Institute for Health and Care Excellence
      Alirocumab for treating primary hypercholesterolaemia and mixed dyslipidaemia.
      ,
      National Institute for Health and Care Excellence
      Evolocumab for treating primary hypercholesterolaemia and mixed dyslipidaemia.
      ].
      Although external validity of the original SAFEHEART model has not previously been reported, a recalibrated version was recently assessed in a French registry [
      • Gallo A.
      • Charriere S.
      • Vimont A.
      • et al.
      SAFEHEART risk-equation and cholesterol-year-score are powerful predictors of cardiovascular events in French patients with familial hypercholesterolemia.
      ]. This study showed reasonable discrimination in a secondary care population, but neither calibration not clinical utility were reported. Given the likely limited performance of the original SAFEHEART model in this French study, and the findings observed here, it appears that the original SAFEHEART model may not be easily generalisable to more routine populations that include clinical diagnoses, and that even when recalibrated, clinical usefulness may be relatively limited in these settings. As patients diagnosed according to clinical criteria may represent variation in underlying aetiology to which the SAFEHEART model is sensitive, model assessment in external genetically defined populations would be of interest. Regarding more routine/mixed FH populations, as concerns internal to the model may be relevant, and the French study indicates that alternative models requiring less information (their ‘REFERCHOL equation B’ and a ‘cholesterol—year score) [
      • Schmidt H.H.J.
      • Hill S.
      • Makariou E.V.
      • et al.
      Relation of cholesterol-year score to severity of calcific atherosclerosis and tissue deposition in homozygous familial hypercholesterolemia.
      ] perform similarly in terms of discrimination [
      • Gallo A.
      • Charriere S.
      • Vimont A.
      • et al.
      SAFEHEART risk-equation and cholesterol-year-score are powerful predictors of cardiovascular events in French patients with familial hypercholesterolemia.
      ], it would be of interest to assess the broader performance of these alternatives given the ongoing challenge to identify sufficiently large samples to derive new models. If found to be more generalisable, these alternative models that require only information widely available in many health systems may also be more practical to apply than the SAFEHEART model, which may require recalibration for each new population, and which includes lipoprotein(a) as a variable. This parameter is not routinely measured in many countries. Given the predominant influence of LDL-C on outcomes for those with FH, it is reasonable to consider that low-demand models including relevant LDL-C terms could perform relatively well in the FH population. Furthermore, a recent study found value in adding coronary artery calcium score (CACS) to the SAFEHEART model to improve its predictive performance in statin-treated heterozygous FH [
      • Gallo A.
      • Perez de Isla L.
      • Charriere S.
      • et al.
      The added value of coronary calcium score in predicting cardiovascular events in familial hypercholesterolemia.
      ]. However, CACS is not currently routinely available in all health systems, including in the UK.

      4.1 Strengths and limitations

      Despite low prevalence and persistent underdiagnosis of FH, the CPRD database provided access to a reasonably large cohort with a reasonable effective sample size [
      • Vergouwe Y.
      • Steyerberg E.W.
      • Eijkemans M.J.C.
      • et al.
      Substantial effective sample sizes were required for external validation studies of predictive logistic regression models.
      ], although still below that preferable for reliably achieving unbiased and precise estimates [
      • Collins G.S.
      • Ogundimu E.O.
      • Altman D.G.
      Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.
      ]. The cohort is likely to be reasonably representative of the relevant populations under active primary care management in the UK, and reliability of outcome ascertainment is likely to be high. The outcome variable drew on information from three well-established sources, and GP-recording of atherosclerotic cardiovascular diagnoses has been incentivised in England for many years. Important study limitations include the reliance on primary care record of FH diagnosis, and associated inability to identify patients with genetic versus clinical diagnoses. Without understanding the proportions of poly-versus mono-genic diagnoses in our samples, drawing inferences for those with polygenic disease in particular, is difficult. Additional limitations include the missing data and data quality issues associated with use of routinely collected data.

      4.2 Conclusions

      The SAFEHEART model appears to offer limited clinical value for the routinely identifiable English primary care FH population, even after recalibration. Further assessment of the generalisability and performance of the cholesterol-year score and REFERCHOL equation B would therefore be of interest for such routine populations. Whereas assessment of the SAFEHEART model in external genetically defined populations would help progress understanding of its validity in these more specific (and likely higher risk) settings.

      Ethical approval

      Permission for data usage was obtained from the CPRD Independent Scientific Advisory Committee (protocol number 18_297). Linked pseudonymized data were provided by CPRD. Data are linked by NHS Digital, the statutory trusted third party for linking data, using identifiable data held only by NHS Digital. Select general practices consent to this process at a practice level with individual patients having the right to opt-out.

      CRediT authorship contribution statement

      Ailsa J. McKay: Conceptualization, Methodology, Data curation, Writing – original draft, Writing – review & editing. Laura H. Gunn: Methodology, Formal analysis, Data curation, Writing – original draft, Writing – review & editing, Visualization. Kausik K. Ray: Conceptualization, Writing – original draft, Writing – review & editing, Supervision, Project administration.

      Declaration of interests

      The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Kausik Ray has acted as a consultant for Amgen, Sanofi, Novartis, Pfizer, AstraZeneca, Boehringer Ingelheim, Novo Nordisk, Kowa, Silence Therapeutics, New Amsterdam, Esperion, Daiichi Sankyo, Bayer, Abbott and Resverlogix; and as a speaker for Novartis, Amgen, Viatris, Pfizer, AstraZeneca, Novo Nordisk, Boehringer Ingelheim, Sanofi. He has received research grants from Amgen, Sanofi, Daiichi Sankyo, Regeneron, and Pfizer through his institution. No conflicts of interest are reported by the other team members.

      Acknowledgements

      KKR acknowledges support from the National Institute for Health Research (NIHR) Imperial Biomedical Research Centre . The authors are grateful to the Big Data and Analytical Unit at Imperial College London for assistance with data management and storage.

      Appendix A. Supplementary data

      The following is the Supplementary data to this article:

      References

        • Brown M.S.
        • Goldstein J.L.
        A receptor-mediated pathway for cholesterol homeostasis.
        Science. 1986; 232: 34
        • Benn M.
        • Watts G.F.
        • Tybjaerg-Hansen A.
        • et al.
        Familial hypercholesterolemia in the Danish general population: prevalence, coronary artery disease, and cholesterol-lowering medication.
        J. Clin. Endocrinol. Metabol. 2012; 97: 3956-3964https://doi.org/10.1210/jc.2012-1563
        • Nordestgaard B.G.
        • Chapman M.J.
        • Humphries S.E.
        • et al.
        Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease Consensus Statement of the European Atherosclerosis Society.
        Eur. Heart J. 2013; 34: 3478-3490https://doi.org/10.1093/eurheartj/eht273
        • Watts G.F.
        • Gidding S.
        • Wierzbicki A.S.
        • et al.
        Integrated guidance on the care of familial hypercholesterolaemia from the International FH Foundation.
        Int. J. Cardiol. 2014; 171: 309-325https://doi.org/10.1016/j.ijcard.2013.11.025
        • Neil A.
        • Cooper J.
        • Betteridge J.
        • et al.
        Reductions in all-cause, cancer, and coronary mortality in statin-treated patients with heterozygous familial hypercholesterolaemia: a prospective registry study.
        Eur. Heart J. 2008; 29: 2625-2633https://doi.org/10.1093/eurheartj/ehn422
        • Versmissen J.
        • Oosterveer D.M.
        • Yazdanpanah M.
        • et al.
        Efficacy of statins in familial hypercholesterolaemia: a long term cohort study.
        BMJ. 2008; 337
        • Oosterveer D.M.
        • Versmissen J.
        • Schinkel A.F.
        • et al.
        Clinical and genetic factors influencing cardiovascular risk in patients with familial hypercholesterolemia.
        Clin. Lipidol. 2010; 5: 189-197
        • National Institute for Health and Care Excellence
        Alirocumab for treating primary hypercholesterolaemia and mixed dyslipidaemia.
        (Technology appraisal guidance [TA393]. Published 22 June 2016. Available at:) (September 2020)
        • National Institute for Health and Care Excellence
        Evolocumab for treating primary hypercholesterolaemia and mixed dyslipidaemia.
        (Technology appraisal guidance [TA394]. Published 22 June 2016. Available at:) (September 2020)
        • National Institute for Health and Care Excellence
        CG181: Cardiovascular Disease: Risk Assessment and Reduction Including Lipid Modification.
        NICE, 2014 (last updated: September 2016). Available at: (November 2018)
        • Pérez de Isla L.
        • Alonso R.
        • Mata N.
        • et al.
        Predicting cardiovascular events in familial hypercholesterolemia.
        Circulation. 2017; 135: 2133
        • Gallo A.
        • Charriere S.
        • Vimont A.
        • et al.
        SAFEHEART risk-equation and cholesterol-year-score are powerful predictors of cardiovascular events in French patients with familial hypercholesterolemia.
        Atherosclerosis. 2020; 306: 41-49https://doi.org/10.1016/j.atherosclerosis.2020.06.011
        • Rapsomaniki E.
        • Shah A.
        • Perel P.
        • et al.
        Prognostic models for stable coronary artery disease based on electronic health record cohort of 102 023 patients.
        Eur. Heart J. 2014; 35: 844-852https://doi.org/10.1093/eurheartj/eht533
        • Collins G.S.
        • Altman D.G.
        Predicting the 10 year risk of cardiovascular disease in the United Kingdom: independent and external validation of an updated version of QRISK2.
        BMJ Br. Med. J. (Clin. Res. Ed.). 2012; 344
        • Janssen K.J.M.
        • Moons K.G.M.
        • Kalkman C.J.
        • et al.
        Updating methods improved the performance of a clinical prediction model in new patients.
        J. Clin. Epidemiol. 2008; 61: 76-86https://doi.org/10.1016/j.jclinepi.2007.04.018
        • Dorresteijn J.A.N.
        • Visseren F.L.J.
        • Wassink A.M.J.
        • et al.
        Development and validation of a prediction rule for recurrent vascular events based on a cohort study of patients with arterial disease: the SMART risk score.
        Heart. 2013; 99: 866
        • Vergouwe Y.
        • Steyerberg E.W.
        • Eijkemans M.J.C.
        • et al.
        Substantial effective sample sizes were required for external validation studies of predictive logistic regression models.
        J. Clin. Epidemiol. 2005; 58: 475-483https://doi.org/10.1016/j.jclinepi.2004.06.017
        • Collins G.S.
        • Ogundimu E.O.
        • Altman D.G.
        Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.
        Stat. Med. 2016; 35: 214-226https://doi.org/10.1002/sim.6787
      1. (author)
        • Steyerberg A.W.
        Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating.
        first ed. Springer New York, New York, NY2009
        • Hippisley-Cox J.
        • Coupland C.
        • Vinogradova Y.
        • et al.
        Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2.
        BMJ. 2008; 336: 1475https://doi.org/10.1136/bmj.39609.449676.25
        • Schmidt H.H.J.
        • Hill S.
        • Makariou E.V.
        • et al.
        Relation of cholesterol-year score to severity of calcific atherosclerosis and tissue deposition in homozygous familial hypercholesterolemia.
        Am. J. Cardiol. 1996; 77: 575-580https://doi.org/10.1016/S0002-9149(97)89309-5
        • Gallo A.
        • Perez de Isla L.
        • Charriere S.
        • et al.
        The added value of coronary calcium score in predicting cardiovascular events in familial hypercholesterolemia.
        J. Am. Coll. Cardiol. 2021; 14: 2414-2424