Skip to main content

Predictive validity of two process-of-care quality measures for residential substance use disorder treatment

Abstract

Background

In order to monitor and ultimately improve the quality of addiction treatment, professional societies, health care systems, and addiction treatment programs must establish clinical practice standards and then operationalize these standards into reliable, valid, and feasible quality measures. Before being implemented, quality measures should undergo tests of validity, including predictive validity. Predictive validity refers to the association between process-of-care quality measures and subsequent patient outcomes. This study evaluated the predictive validity of two process quality measures of residential substance use disorder (SUD) treatment.

Methods

Washington Circle (WC) Continuity of Care quality measure is the proportion of patients having an outpatient SUD treatment encounter within 14 days after discharge from residential SUD treatment. The Early Discharge measure is the proportion of patients admitted to residential SUD treatment who discharged within 1 week of admission. The predictive validity of these process measures was evaluated in US Veterans Health Administration patients for whom utilization-based outcome and 2-year mortality data were available. Propensity score-weighted, mixed effects regression adjusted for pre-index imbalances between patients who did and did not meet the measures’ criteria and clustering of patients within facilities.

Results

For the WC Continuity of Care measure, 76 % of 10,064 patients had a follow-up visit within 14 days of discharge. In propensity score-weighted models, patients who had a follow-up visit had a lower 2-year mortality rate [odds ratio (OR) = 0.77, p = 0.008], but no difference in subsequent detoxification episodes relative to patients without a follow-up visit. For the Early Discharge measure, 9.6 % of 10,176 discharged early and had significantly higher 2-year mortality (OR = 1.49, p < 0.001) and more subsequent detoxification episodes.

Conclusions

These two measures of residential SUD treatment quality have strong associations with 2-year mortality and the Early Discharge measure is also associated with more subsequent detoxification episodes. These results provide initial support for the predictive validity of residential SUD treatment quality measures and represent the first time that any SUD quality measure has been shown to predict subsequent mortality.

Background

In order to monitor and ultimately improve the quality of addiction treatment, professional organizations, health care systems, and addiction treatment programs must establish clinical practice standards and then operationalize these standards into reliable, valid, and feasible quality measures. Quality measures can help define and motivate guideline-congruent care, reveal gaps in the continuum of care, identify poor performers to whom quality improvement efforts can be targeted, and may provide an empirically-justified basis for performance-based incentive programs. However, these benefits can only be realized if quality measures are valid and avoid unintended consequences, neither of which is a given [1, 2].

Improvement in clinically important outcomes is the most prominent goal of health care and therefore may seem like the only legitimate “gold standard” for measuring treatment quality. But using outcomes alone as quality measures presents at least three difficulties. First, post-treatment outcome data are often expensive and logistically difficult to collect. Second, using outcomes directly as quality measures requires risk adjustments for casemix and making debatable assumptions regarding missing baseline and follow-up data [3]. Third, outcome data alone cannot indicate if treatment access is sufficient or what structural and process changes might be needed to improve outcomes. Therefore, identifying treatment structure, access, and process measures in administrative data that reliably predict long-term outcomes is often the best available quality assessment method.

Predictive validity refers to the association between antecedent quality indicators (e.g., structural, access, and process-of-care measures) and subsequent quality indicators, such as patient outcomes. Associations between process-of-care measures and outcomes can be driven by many factors, including the fidelity with which the process can be operationalized with available data [4], selection of an appropriate outcome, the temporal proximity of the outcome to the clinical process, limited power to detect small effects, and potential bias due to unmeasured confounding factors [5]. Therefore, care and humility are indicated when interpreting the results of predictive validity studies. Regardless, if the presumed association between a process-of-care quality measure and patient outcomes cannot be demonstrated, the measure’s validity should be considered suspect.

Formulating and implementing quality measures without careful empirical validation exposes all stakeholders to many risks, including promoting poor or incomplete care, and diverting effort and attention from potentially more important activities. Measures are often specified and implemented before enough data and/or interest accumulates to conduct validation studies. In diverse areas of medicine, even when careful validation is undertaken, the expected associations between process-of-care measures and outcomes are often absent or weaker than expected. Furthermore, when previously validated measures are put into use, particularly with incentives, changes in care or coding practices can lead to changes in predictive validity or other unintended consequences [610]. Addiction treatment quality measures have not been immune from these challenges [2, 1115].

Residential addiction treatment programs are important features of the continuum of care for patients with SUD. As an intensive, relatively costly setting for SUD treatment, these programs are usually reserved for patients who need high levels of support, supervision, and stabilization before transitioning to outpatient continuing care. Two process-of-care measures have been proposed to capture elements of residential addiction program quality, one proposed by the Washington Circle and one by Rand and the Altarum Institute. The Washington Circle (WC) was a group of national experts that sought to improve the accessibility and effectiveness of SUD treatment through the use of quality measurement systems. One of the WC’s major activities was developing quality measures for SUD treatment for publicly-funded and commercially-insured systems of care. One WC measure focuses on outpatient continuing care after residential treatment [16]. WC Continuity of Care is a dichotomous measure indicating whether or not a patient had an outpatient SUD treatment encounter within 14 days after discharge from residential treatment. The measure is intended to assess the minimum necessary service to provide quality treatment, but not necessarily optimal or sufficient continuity of care after residential treatment.

Garner et al. [17] evaluated the predictive validity of the WC Continuity of Care measure in a sample of 342 adolescents in long-term residential addiction treatment who were randomly assigned at discharge to either standard continuing care or an assertive continuing care condition. Overall, adolescent patients whose treatment met the WC Continuity of Care criterion were significantly more likely to have achieved recovery status (no alcohol or other drug use, abuse, or dependence symptoms while living in the community during the previous 30 days) at 3 months (OR = 1.92, p < 0.05). However, the predictive validity of this measure has never been evaluated in adult patients or beyond 3 months.

The other quality measure pertaining to residential SUD treatment was developed by RAND and the Altarum Institute as part of a national evaluation of mental health services commissioned by the US Veterans Health Administration (VHA). The measure taps the proportion of patients with SUD diagnoses admitted to residential SUD treatment who are discharged or self-discharge within 1 week of admission (“Early Discharge”). Patients who discharge early from residential addiction treatment programs typically have poorer outcomes and are more likely to be readmitted [18]. However, the strength of evidence for these associations is modest and the predictive validity of the Early Discharge quality measure itself has never been evaluated.

Therefore, the purpose of this study was to examine the predictive validity of these two process-of-care quality measures in the VHA, a large integrated health care system with extensive SUD treatment services. Because we lack data on more clinically detailed outcomes, such as improvements in symptoms and functioning, the primary outcomes in this study are 2-year mortality and patterns of utilization that might signify either successful stabilization in an outpatient setting or an escalation of symptoms requiring subsequent readmission or detoxification. Specifically, we hypothesized that WC Continuity of Care would be positively associated with subsequent outpatient SUD and mental health treatment after the 14-day post-discharge observation period and negatively associated with subsequent SUD and psychiatric inpatient admissions, detoxification episodes, as well as 2-year mortality. We hypothesized that these relationships would be in the opposite direction for the Early Discharge measure. If evidence can be found for the predictive validity of these measures, they could be used for system monitoring and quality improvement purposes in VHA’s Mental Health Information System [19], or by other integrated health care systems.

Methods

Calculating the Continuity of Care measure

Using VHA’s National Patient Care Database, we identified all patients with an SUD diagnosis who discharged to the community from one of VHA’s 54 Substance Abuse Residential Rehabilitation Programs (SARRTPs; bed section codes: 27, 37, 85, 86, 88, 111) in fiscal year 2009 (FY09). We then determined whether each patient had an outpatient SUD visit within 14 days of discharge. SUD outpatient visits were defined as a clinical encounter in an SUD or mental health clinic in which an SUD diagnosis was recorded. Patients who transferred from the SARRTP to another residential setting were excluded from the denominator.

Calculating the Early Discharge measure

Using VHA’s National Patient Care Database, we identified all patients with an admission to one of VHA’s SARRTPs in FY09 with a SUD diagnosis, and determined whether each patient was discharged from the program to the community within 7 days. Although the denominators for the two measures are different (patients with admissions vs. patients with discharges), the samples largely overlap.

Outcomes

We examined if each measure was associated with the following utilization-based outcomes in the 6 months after each patient either met or did not meet the measures’ criteria: Number of subsequent admissions to a SARRTP, number of admissions to other mental health residential or inpatient programs, number of outpatient encounters in SUD clinics, number of outpatient encounters in mental health clinics, and number of detoxification episodes. We also examined the association of each measure with 2-year mortality as determined by the VHA Vital Status file.

Propensity score modeling

Estimation of process-outcome associations from observational data is difficult because exposure to the process of care is non-random and often confounded with patient or program characteristics. Propensity score methods are often used to adjust for confounding. Here, a propensity score is defined as the probability of meeting a performance measure conditional on pre-treatment covariates. Several methods have been developed to calculate propensity scores. We used boosted regression to estimate each patient’s propensity score for meeting each process-of-care measure. Boosted regression has been shown to produce models with less prediction error than other common methods [20]. One-year pre-index variables used in the propensity score models were age, gender, marital status (Y/N), race/ethnicity, co-morbid psychiatric diagnoses, traumatic brain injury, co-morbid medical diagnoses (e.g., diabetes, hepatitis C), homelessness, outpatient and residential SUD and mental health service utilization, and number of prior detoxification episodes.

Modeling associations between process measures and outcomes

For each process-of-care quality measure, using a propensity-score weighted [20], mixed-effects regression model, we evaluated if getting the measure-specified care (yes/no) was associated with patient-level outcomes beyond the possible effect of facility-level performance on the measure [21]. These models included VHA facility as a random effect and an exchangeable covariance structure. Utilization-based outcomes were modeled by mixed-effects, zero-inflated, negative binomial regression models. Two-year mortality was modeled in mixed-effects logistic regression models.

Facility-level variability

In order to assess the extent to which performance on these measures varied between facilities, we calculated descriptive statistics and histograms of the distributions of facility-level performance of the VHA facilities with residential SUD programs.

Results

For the WC Continuity of Care measure, 10,064 patients met the denominator criterion (i.e., discharged from a residential SUD treatment program in FY09) and 7648 (76 %) met the numerator criterion by having a follow-up visit within 14 days of discharge. The characteristics of patients who met versus did not meet the measure’s numerator criterion are presented in Table 1, along with the characteristics of the “failed to meet” sample after weighting with propensity scores. Even, before propensity score weighting, the two groups were very similar on most characteristics. The most pronounced differences between patients meeting and not meeting the numerator criterion were the number of SUD outpatient visits in the previous 6 months (15.4 vs. 9.8), homelessness in the previous year (30.1 vs. 26.7 %), and HIV positive status (1.8 vs. 2.6 %). All of these differences were reduced or eliminated by the weighting procedure.

Table 1 Demographic and pre-index characteristics of patients meeting and not meeting the WC Continuity of Care numerator criterion

In propensity score-weighted models, patients whose care met the WC Continuity of Care measure had significantly more subsequent mental health admissions (b = 0.15, p = 0.015), significantly more mental health outpatient visits (b = 0.42, p < 0.001), and significantly more SUD outpatient visits (b = 0.65, p < 0.001). The measure was not significantly associated with number of subsequent admissions to residential SUD treatment programs or number of detoxification episodes. Notably, the odds of death in the two post-discharge years was significantly lower for patients who received the measure-specified continuity of care (4.35 vs. 5.57 %; OR = 0.77, p = 0.008). The unadjusted and propensity score-adjusted group differences in 2-year mortality and mean number of detoxifications in 6 months are presented in Table 2. Also, Fig. 1 shows substantial facility-level variation (41–99 %) in the WC Continuity of Care measure.

Table 2 Unadjusted and adjusted outcomes in patients meeting and not meeting measure criteria
Fig. 1
figure 1

Distribution of facility performance on WC Continuity of Care

For the Early Discharge measure, 10,176 patients met the denominator criterion (i.e., admitted to a residential SUD treatment program in FY09) and 974 (9.6 %) met the numerator criterion by experiencing discharge within 7 days. The characteristics of patients who met versus did not meet the measure’s numerator criterion are presented in Table 3, along with the characteristics of the “did not meet” sample after weighting with propensity scores. Unlike the WC Continuity of Care measure, it is considered undesirable to meet the numerator criterion for the Early Discharge measure. Again, even before propensity score weighting, the two groups were very similar on most characteristics. The most pronounced differences between patients meeting and not meeting the numerator criterion were in the number of mental health outpatient visits in the previous 6 months (15.8 vs. 13.8), Caucasian race (66.8 vs. 59.3 %), traumatic brain injury (4.3 vs. 2.7 %), and HIV positive status (2.7 vs. 1.8 %). All of these differences were reduced or eliminated by the weighting procedure.

Table 3 Demographic and pre-index characteristics of patients meeting and not meeting the Early Discharge numerator criterion

In propensity score-weighted models, patients who met the numerator criterion for early discharge had significantly fewer mental health admissions (b = −0.65, p < 0.001), mental health outpatient visits (b = −0.50, p < 0.001), SUD outpatient visits (b = −0.88, p < 0.001), and admissions to residential SUD treatment programs (b = −1.05, p < 0.001) in the following 6 months. Patients discharging early also had significantly more subsequent detoxification episodes (b = 0.11, p = 0.037). Finally, the odds of death in the two post-discharge years was significantly higher in patients who discharged early and thereby met the numerator criterion (7.30 vs. 4.72 %; OR = 1.49, p < 0.001). The unadjusted and propensity score-adjusted group differences in 2-year mortality and mean number of detoxifications in 6 months are presented in Table 2. Also, Fig. 2 shows substantial facility-level variation in this measure from 0 to 46 %.

Fig. 2
figure 2

Distribution of facility performance on Early Discharge measure

Discussion

In this study, we examined the predictive validity of two process-of-care quality measures for residential SUD treatment in a large sample of patients in a very large integrated health care system. Ideally, data would have been available to examine the associations of these measures with near- and long-term changes in symptoms and functioning, instead of only administrative and utilization data. We originally had hypotheses about the relationships between these measures and patterns of utilization, such as outpatient SUD treatment engagement, that might signify successful stabilization, and patterns of utilization, such as subsequent readmission or detoxification, that might signify an escalation. However, the results indicated that more utilization specified in the measures was generally associated with more subsequent utilization of both types. In other words, markers of engagement in the health care system are associated with other markers of later engagement in the health care system. In retrospect, this is unsurprising and consistent with research in other clinical areas. The one type of utilization that ran counter to this general trend, however, was subsequent detoxification episodes. Although the WC Continuity of Care measure was unrelated to subsequent detoxification episodes, the Early Discharge measure (i.e., less initial utilization) was associated with significantly more subsequent detoxification episodes.

Even more impressive, both measures were associated with risk of death in the two post-discharge years in the expected directions: Continuity of Care with lower risk of death and Early Discharge with higher risk of death. These results are noteworthy given that mortality is a conceptually distal outcome of SUD treatment relative to, say, substance use. Previous efforts in the area of addiction and more broadly to examine associations between continuity of care and retention quality measures and outcomes have largely found none (e.g., [2224]) with some exceptions (e.g., [17, 25, 26]). However, this is the first study of addiction treatment process measures to find associations with risk of death. It is possible but unlikely that the associations between these measures and risk of death are directly causal. Perhaps more likely is that these processes facilitate or are proxies for different levels of engagement with the health care system, which in turn affect mortality risk. Alternatively, these processes may be associated with unmeasured factors that are linked to mortality. In an event, finding SUD treatment process-of-care quality measures that are so strongly linked with subsequent risk of death is remarkable.

This study has several limitations. First, the VHA is a large integrated health care system serving US military Veterans. It is unknown to what extent these findings generalize to different health care systems or settings. Second, although patients who met the measures and patients who did not meet the measures were very similar on measured characteristics, and differences that did exist were trivial after propensity score weighting, these patients may have also differed on unmeasured characteristics, such as SUD severity or travel time to care, that could affect outcomes.

With these caveats in mind, it is still noteworthy that these two process-of-care quality measures for residential SUD treatment are linked to 2-year mortality risk, and the Early Discharge measure was associated with subsequent detoxification episodes. These results provide important new evidence supporting the predictive validity of the WC Continuity of Care measure and the Early Discharge measure, and thereby, affirming their value as tools to monitor the quality of residential SUD care for adults. Because these results were not guaranteed, we believe our study shows the importance of subjecting quality measures to validity checks before implementing them in clinical practice. Where these measures are implemented, especially if attached to incentives or consequences for providers or systems of care, it is critical to make sure that they do not drive unintended changes in practice (e.g., instituting a cursory follow-up program to meet the measure) that will dilute or destroy the process-outcome associations found in this study.

Abbreviations

SUD:

substance use disorder

WC:

Washington Circle

VHA:

US Veterans Health Administration

SARRTP:

Substance Abuse Residential Rehabilitation Programs

FY:

fiscal year

OR:

odds ratio

References

  1. Bradley KA, et al. When quality indicators undermine quality: bias in a quality indicator of follow-up for alcohol misuse. Psychiatr Serv. 2013;64(10):1018–25.

    Article  PubMed Central  PubMed  Google Scholar 

  2. Harris AH, et al. Measuring the quality of substance use disorder treatment: evaluating the validity of the Department of Veterans Affairs continuity of care performance measure. J Subst Abuse Treat. 2009;36(3):294–305.

    Article  PubMed  Google Scholar 

  3. Iezzoni LI. The risks of risk adjustment. JAMA. 1997;278(19):1600–7.

    Article  CAS  PubMed  Google Scholar 

  4. Harris AHS, et al. Validation of the treatment identification strategy of the HEDIS addiction quality measures: concordance with medical record review. BMC Health Serv Res. 2011;11:73.

    Article  PubMed Central  PubMed  Google Scholar 

  5. Parast L, et al. Challenges in assessing the process-outcome link in practice. J Gen Intern Med. 2015;30(3):359–64.

    Article  PubMed  Google Scholar 

  6. Werner RM, Bradlow ET. Relationship between Medicare’s hospital compare performance measures and mortality rates. JAMA. 2006;296(22):2694–702.

    Article  CAS  PubMed  Google Scholar 

  7. Horn SD. Performance measures and clinical outcomes. JAMA. 2006;296(22):2731–2.

    Article  CAS  PubMed  Google Scholar 

  8. Wachter RM, et al. Public reporting of antibiotic timing in patients with pneumonia: lessons from a flawed performance measure. Ann Intern Med. 2008;149(1):29–32.

    Article  PubMed  Google Scholar 

  9. Walter LC, et al. Pitfalls of converting practice guidelines into quality measures: lessons learned from a VA performance measure. JAMA. 2004;291(20):2466–70.

    Article  CAS  PubMed  Google Scholar 

  10. Roth S, et al. Unintended consequences of a quality measure for acute bronchitis. Am J Manag Care. 2012;18(6):e217–24.

    PubMed  Google Scholar 

  11. Harris AH, et al. HEDIS initiation and engagement quality measures of substance use disorder care: impact of setting and health care specialty. Popul Health Manag. 2009;12(4):191–6.

    Article  PubMed  Google Scholar 

  12. Harris AH, et al. Does meeting the HEDIS substance abuse treatment engagement criterion predict patient outcomes? J Behav Health Serv Res. 2010;37(1):25–39.

    Article  PubMed  Google Scholar 

  13. Dunigan R, et al. Engagement in outpatient substance abuse treatment and employment outcomes. J Behav Health Serv Res. 2014;41(1):20–36.

    Article  PubMed  Google Scholar 

  14. Garnick DW, et al. The Washington Circle engagement performance measures’ association with adolescent treatment outcomes. Drug Alcohol Depend. 2012;124(3):250–8.

    Article  PubMed  Google Scholar 

  15. Harris AH, Humphreys K, Finney JW. Veterans affairs facility performance on Washington Circle indicators and casemix-adjusted effectiveness. J Subst Abuse Treat. 2007;33(4):333–9.

    Article  PubMed  Google Scholar 

  16. Garnick DW, et al. Adapting Washington Circle performance measures for public sector substance abuse treatment systems. J Subst Abuse Treat. 2009;36(3):265–77.

    Article  PubMed Central  PubMed  Google Scholar 

  17. Garner BR, et al. The Washington Circle Continuity of Care performance measure: predictive validity with adolescents discharged from residential treatment. J Subst Abuse Treat. 2010;38(1):3–11.

    Article  PubMed Central  PubMed  Google Scholar 

  18. Peterson KA, et al. Determinants of readmission following inpatient substance abuse treatment: a national study of VA programs. Med Care. 1994;32(6):535–50.

    Article  CAS  PubMed  Google Scholar 

  19. Trafton JA, et al. VHA mental health information system: applying health information technology to monitor and facilitate implementation of VHA Uniform Mental Health Services Handbook requirements. Med Care. 2013;51(3 Suppl 1):S29–36.

    Article  PubMed  Google Scholar 

  20. McCaffrey DF, Ridgeway G, Morral AR. Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychol Methods. 2004;9(4):403–25.

    Article  PubMed  Google Scholar 

  21. Desai M, Begg MD. A comparison of regression approaches for analyzing clustered data. Am J Public Health. 2008;98(8):1425–9.

    Article  PubMed Central  PubMed  Google Scholar 

  22. Harris AHS, et al. Measuring the quality of substance use disorder treatment: evaluating the validity of the Department of Veterans Affairs continuity of care performance measure. J Subst Abuse Treat. 2009;36(3):294–305.

    Article  PubMed  Google Scholar 

  23. Greenberg G, Rosenheck R, Fontana A. Continuity of Care and clinical effectiveness: treatment of posttraumatic stress disorder in the Department of Veterans Affairs. J Behav Health Ser Res. 2003;30:202–14.

    Article  Google Scholar 

  24. Greenberg A, Rosenheck R, Seibyl C. Continuity of care and clinical effectiveness: outcomes following residential treatment for severe substance abuse. Med Care. 2002;40(3):246–59.

    Article  PubMed  Google Scholar 

  25. Adair CE, et al. Continuity of care and health outcomes among persons with severe mental illness. Psychiatr Serv. 2005;56(9):1061–9.

    Article  PubMed  Google Scholar 

  26. Greenberg GA, Rosenheck RA. Continuity of care and clinical outcomes in a national health system. Psychiatr Serv. 2005;56(4):427–33.

    Article  PubMed  Google Scholar 

Download references

Authors’ contributions

AHSH, JWF, SMA, KH and JT conceived of the study and ASHH supervised the analysis and modeling. TB, LSE, TEP, and ADR participated in modeling decisions and conducted and interpreted analyses. All authors read and approved the final manuscript.

Acknowledgements

This study was funded by the VA Health Services Research and Development Service (Grants nos. SUS 99-015; IIR 10-370-2; RCS-14-232; RCS-04-141). The views expressed herein are not necessarily those of the Department of Veterans Affairs.

Competing interests

The authors declare that they have no competing interests.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alex H. S. Harris.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Harris, A.H.S., Gupta, S., Bowe, T. et al. Predictive validity of two process-of-care quality measures for residential substance use disorder treatment. Addict Sci Clin Pract 10, 22 (2015). https://doi.org/10.1186/s13722-015-0042-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13722-015-0042-5

Keywords