Practice-Based Versus Telemedicine-Based Collaborative Care for Depression in Rural Federally Qualified Health Centers: A Pragmatic Randomized Comparative Effectiveness Trial
Abstract
Objective:
Practice-based collaborative care is a complex evidence-based practice that is difficult to implement in smaller primary care practices that lack on-site mental health staff. Telemedicine-based collaborative care virtually co-locates and integrates mental health providers into primary care settings. The objective of this multisite randomized pragmatic comparative effectiveness trial was to compare the outcomes of patients assigned to practice-based and telemedicine-based collaborative care.
Method:
From 2007 to 2009, patients at federally qualified health centers serving medically underserved populations were screened for depression, and 364 patients who screened positive were enrolled and followed for 18 months. Those assigned to practice-based collaborative care received evidence-based care from an on-site primary care provider and a nurse care manager. Those assigned to telemedicine-based collaborative care received evidence-based care from an on-site primary care provider and an off-site team: a nurse care manager and a pharmacist by telephone, and a psychologist and a psychiatrist via videoconferencing. The primary clinical outcome measures were treatment response, remission, and change in depression severity.
Results:
Significant group main effects were observed for both response (odds ratio=7.74, 95% CI=3.94–15.20) and remission (odds ratio=12.69, 95% CI=4.81–33.46), and a significant overall group-by-time interaction effect was observed for depression severity on the Hopkins Symptom Checklist, with greater reductions in severity over time for patients in the telemedicine-based group. Improvements in outcomes appeared to be attributable to higher fidelity to the collaborative care evidence base in the telemedicine-based group.
Conclusions:
Contracting with an off-site telemedicine-based collaborative care team can yield better outcomes than implementing practice-based collaborative care with locally available staff.
(Reprinted with permission from American Journal of Psychiatry 2013; 170:414–425)
Complex evidence-based practices are difficult to implement with fidelity in routine care. One such evidence-based practice, practice-based collaborative care, involves co-located primary care providers, mental health specialists, and care managers working together on-site in a primary care setting. Practice-based collaborative care has been shown to improve depression outcomes in numerous randomized effectiveness trials (1–7). However, implementation in smaller primary care practices presents challenges where it is typically not feasible to employ on-site mental health specialists or full-time care managers. Only 25% of U.S. primary care practices have on-site mental health specialists (8). Two studies (3, 9) have demonstrated that depression outcomes can be improved in small primary care clinics lacking on-site mental health specialists by training on-site nurses to be depression care managers; however, the effect sizes have been small to medium. Moreover, based on the results of a meta-analysis of randomized trials of practice-based collaborative care, Gilbody et al. (10) concluded that collaborative care interventions with more mental health specialist involvement have larger effect sizes.
Telemedicine technologies now make possible the virtual co-location of mental health specialists and primary care providers. Telemedicine-based collaborative care involves an off-site team of mental health specialists collaborating with on-site primary care providers, from a centralized location, using telephones, videoconferencing, and electronic health records. A multisite randomized trial conducted in the Department of Veterans Affairs (VA) (11, 12) demonstrated that telemedicine-based collaborative care is more effective than usual care in small satellite primary care clinics. Telemedicine-based collaborative care has also been adopted and sustained as part of routine care in small satellite primary care clinics within the VA (13). Compared with practice-based collaborative care, telemedicine-based collaborative care has the potential to be implemented with higher fidelity in smaller primary care clinics because the off-site team can include an array of mental health specialists, and full-time care managers can practice under more intensive clinical supervision and dedicate their time to care coordination activities, resulting in higher fidelity to care manager protocols. However, there are also potential disadvantages to telemedicine-based collaborative care. Off-site care managers cannot build on established relationships with patients and may have difficulty establishing therapeutic alliances from a distance. Likewise, if off-site care managers lack access to on-site medical records, they may have difficulty integrating behavioral and physical aspects of care and face barriers communicating with on-site primary care providers. In addition, the stepped care elements of the collaborative care model (e.g., psychiatric consultation) may be less effective when delivered via telemedicine. However, patients and providers uniformly report high levels of satisfaction with videoconferencing (14–17), and there is good evidence for the clinical equivalency of psychiatric (18–21) and psychological (22–26) treatments delivered via videoconferencing compared with face-to-face.
Given that 1) practice-based collaborative care without the involvement of on-site mental health specialists improves outcomes, 2) practice-based collaborative care with involvement from on-site mental health specialists improves outcomes more, and 3) the potential disadvantages of telemedicine-based collaborative care, an important policy-relevant question is whether it is more effective for small, remote primary care clinics to implement practice-based collaborative care without the involvement of on-site mental health specialists or to implement telemedicine-based collaborative care in partnership with an off-site team of mental health specialists.
Our objective in this pragmatic (27) comparative effectiveness trial (28) was to compare the process outcomes and clinical outcomes of patients randomly assigned to telemedicine-based collaborative care or practice-based collaborative care in small, remote primary care clinics that lacked on-site mental health specialists, thereby comparing two organizational approaches (with clinical equipoise) to delivering collaborative care for depression (29). We chose to conduct a pragmatic trial design in order to compare two viable competing strategies to delivering an evidence-based practice rather than an explanatory trial designed to determine why one approach was superior to the other. We hypothesized that compared with patients assigned to practice-based collaborative care, those assigned to telemedicine-based collaborative care would receive higher-fidelity care management, more specialty mental health services (e.g., telepsychiatry and telepsychology), and higher-quality pharmacotherapy and therefore would experience greater symptom improvement. Secondary outcomes examined included health status, quality of life, and satisfaction with care.
We partnered with five federally qualified health centers serving medically underserved populations in Arkansas’ Mississippi Delta region and the Ozark Highlands. With federal oversight from the Health Resources and Services Administration (HRSA), federally qualified health centers make up the nation’s largest and fastest growing network of primary care providers, with 8,000 clinics providing services to 20 million Americans. Three-quarters of federally qualified health center patients live in poverty, half live in rural areas, one-third are uninsured, and one-third are from minority populations. Mental health problems are the most commonly reported reasons for visits to federally qualified health centers (30), yet only 5.5% of encounters are with on-site mental health specialists (31). Research conducted in partnership with federally qualified health centers has direct applicability to a large segment of the U.S. population at risk for experiencing health disparities.
Method
Design Overview
This multisite randomized trial employed a comparative effectiveness research design (28). Patients were randomly assigned to one of two active treatment arms, both of which represent potentially feasible approaches to adapting the evidence-based collaborative care model for routine delivery in medically underserved areas. The study design also contained many elements of pragmatic trials (27), including the use of relatively few exclusion criteria, enrollment of a diverse sample of patients, monitoring of but not controlling intervention fidelity, a primary outcome measure defined in terms of changes in patient-reported symptoms, and use of intent-to-treat analyses to examine group differences.
Setting and Participants
We approached six federally qualified health centers, and five agreed to participate. Participating centers employed between 1.3 and 9.7 primary care physician full-time equivalents, served between 5,362 and 13,050 unique primary care patients, and operated one to six clinics across multiple locations. None of the participating clinic locations had an on-site mental health specialist. Between November 2007 and June 2009, 19,285 patients were screened for depression by health center staff using the nine-item Patient Health Questionnaire (see the CONSORT diagram in the data supplement that accompanies the online edition of this article). Fifteen percent of patients (N=2,863) screened positive (a score ≥10 on the Patient Health Questionnaire); health center staff obtained informed consent from 62% of potentially eligible patients (N=829), and 55% (N=364) were found to be eligible and completed the baseline telephone interview. We excluded patients that primary care providers would not be comfortable treating; exclusion criteria included pregnancy, schizophrenia, acute suicidal ideation, substance dependence, bipolar disorder, recent bereavement, and current specialty mental health treatment. We also excluded patients who were unable to participate in research because of cognitive impairment, having a court-appointed guardian, not speaking English, having no telephone, or having a life event preventing participation. Screening and eligibility results were entered into the medical record.
Randomization and Interventions
Randomization.
Eligible patients were assigned by computer randomization using a 2×2 Latin square design (stratified by clinic) to either telemedicine-based collaborative care or practice-based collaborative care. It was not feasible to blind patients or providers to randomization status.
Practice-based collaborative care.
The practice-based collaborative care intervention was designed to be the same as the model of care supported by HRSA, known as the Depression Health Disparities Collaborative (32, 33), which represents more intensive treatment than usual care. Practice-based collaborative care involved two types of providers: on-site primary care providers and on-site nurse depression care managers. Each clinic location employed a half-time depression care manager funded by the study. Depression care managers were nurses (R.N. or L.P.N.) who had no prior mental health experience. They received a care manager training manual, 1 day of training in depression care management, and access to a web-based patient registry and depression care manager decision support system (34). Patients could choose either watchful waiting or antidepressant treatment. Depression care manager encounters were conducted either face-to-face or by telephone, depending on the patient’s preference. The initial depression care manager encounter included 1) symptom monitoring with the Patient Health Questionnaire, 2) education/activation, 3) barrier assessment/resolution, and 4) establishing self-management goals, including planning physical, rewarding, and social activities. Follow-up encounters, scheduled every 2 weeks during acute treatment and every 4 weeks during continuation treatment, included the monitoring of symptoms, medication adherence, side effects, and engagement in planned self-management activities. Depression care managers received no clinical supervision from a mental health specialist. Progress notes were entered into the patients’ paper medical record. A trial was considered to have failed in the acute phase if the patient did not respond to treatment (response was defined as a decrease of ≥50% in Patient Health Questionnaire score) after 8 weeks of treatment. No additional on-site mental health support was available for patients who did not respond to treatment, although patients could be referred to off-site mental health providers (e.g., community mental health centers). Patients received the intervention for up to 12 months.
Telemedicine-based collaborative care.
Telemedicine-based collaborative care involved five types of providers: on-site primary care providers and off-site depression care managers (at the R.N. level), pharmacists (at the Pharm.D. level), psychologists (at the Ph.D. level), and psychiatrists. The off-site team, located at the University of Arkansas for Medical Sciences, was funded by the study. The full-time depression care manager was a nurse who had no prior mental health experience and received the same training and tools as the on-site depression care managers. All depression care manager encounters were conducted by telephone and followed the protocol described above. Progress notes were faxed to the clinic. During weekly meetings, the depression care manager received clinical supervision and the off-site team discussed new patients and patients who were not responding to treatment, and they offered treatment recommendations to primary care providers via the depression care manager progress notes. Patients received stepped care, whereby treatment intensity was increased for patients who were not responding to treatment. If the patient did not respond to the initial antidepressant, the pharmacist conducted a medication history and provided medication management by telephone as needed. If the patient did not respond to two antidepressant trials, a psychiatry consultation via videoconferencing was scheduled. At any time, patients had access to cognitive-behavioral therapy (CBT) delivered via videoconferencing. Patients who did not respond to an antidepressant trial were specifically encouraged to initiate and complete CBT.
Outcomes and Follow-Up
Fidelity assessment.
Fidelity to the depression care manager protocol was measured through chart review. Five of the charts (1.4%) could not be located during the site visits. The following fidelity measures were abstracted: number of depression care manager encounters with documented Patient Health Questionnaire scores; self-management activities; antidepressant adherence assessments and side effect assessments for those treated with antidepressants; and counseling adherence assessments for those referred to psychotherapy. Depression care manager fidelity was also measured from patient self-report during the 6- and 12-month telephone interviews using items that addressed education, self-management, symptom monitoring, adherence monitoring, and collaboration among providers. To measure fidelity to the stepped care protocol, we examined what proportion of patients in the telemedicine group who failed to respond to at least one medication trial had a telephone pharmacist encounter and what proportion of those who failed to respond to two trials had a telepsychiatry consultation. We also examined what proportion of patients in the telemedicine-based group attended at least one telepsychotherapy session and completed at least eight sessions.
Baseline interviews.
Data were collected at baseline in a blinded telephone interview. At baseline, sociodemographic and clinical case-mix data were collected using the Depression Outcomes Module (35, 36), the Mini International Neuropsychiatric Interview (37, 38), the Duke Social Support and Stress Scale (39, 40), the Quality Improvement for Depression Treatment Acceptability scale (3, 4), and the Depression Health Beliefs Inventory (41). Zip codes were used to categorize patients’ residence as rural or urban according to Rural Urban Commuting Area classification scheme C.
Follow-up interviews.
Blinded follow-up telephone interviews were completed for 87% of participants (N=318) at 6 months, 79% (N=287) at 12 months, and 78% (N=283) at 18 months. In addition to the fidelity measures described above, the other primary process measures included self-reported mental health service utilization, antidepressant prescriptions, antidepressant dosage (categorized as starting, usual, or high [42]), and antidepressant adherence (taking the full prescribed dosage at least 80% of days in the previous month). The primary clinical outcome measures were changes in depression severity and treatment response and remission. Depression severity was measured continuously using the 20-item Hopkins Symptom Checklist (HSCL) (43, 44). Response, measured dichotomously, was defined as a decrease of ≥50% in HSCL score between baseline and follow-up. Remission, measured dichotomously, was defined as an HSCL score <0.5. Secondary outcome measures included health status (based on the physical and mental health composite scale scores from the 12-item Short-Form Health Survey) (45, 46), quality of life (based on the Quality of Well-Being scale) (47–50), and satisfaction with care (based on the Consumer Assessment of Healthcare Providers and Systems) (51).
Statistical Analysis
Patients were the unit of the intent-to-treat analysis. Clinic-level intraclass correlation coefficients for HSCL change scores were not significant. Provider-level intraclass correlation could not be calculated because patients could have multiple primary care providers. For the hypotheses examining fidelity, separate logistic regressions were specified for the first 6 months and the second 6 months. For the hypotheses examining other outcomes, we used mixed models and included data from all completed research assessments (52). Case-mix variables were selected using the method of purposeful selection (53). Case-mix variables with missing values were imputed using the PROC MI procedure in SAS, version 9.3 (SAS Institute, Cary, N.C.). PROC MIXED and PROC GLIMMIX were used with the PROC MIANALYZE procedure to model outcomes with linear, binomial, negative binomial, or ordinal distributions. All models specified clinic as a random effect to control for intraclass correlation coefficient. Time was included as a fixed effect. The model specifications included main effects for group and time (with practice-based collaborative care assigned as the reference group) and interaction effects for group by time. The main group effect was used to test the hypotheses for dependent variables not measured at baseline (e.g., treatment response). For dependent variables that were measured at baseline (e.g., HSCL score), group-by-time interaction effects were used to test the hypothesis that the rate of improvement differed across the two groups. Because there were multiple group-by-time interaction terms, an omnibus test was used to determine whether these variables collectively explained a significant amount of the variance in the dependent variable (54). If the omnibus test was significant at the conservative alpha level of 0.10, we report group differences and significance tests for each time period. Otherwise, we report group differences averaged across the time periods and one significance test. The analysis had an 85% power to detect a 15% difference (e.g., 30% versus 45%) in response rates.
Results
The study participants’ socioeconomic and clinical characteristics are summarized in Table 1. Two-thirds (64.5%) were unemployed, 69.7% had an annual household income <$20,000, 50.8% were uninsured, and 68.1% lived in a rural area. At baseline, 83.2% met diagnostic criteria for major depressive disorder, and the mean HSCL score was 1.9, indicating moderately severe depression. The mean number of chronic physical health disorders was 4.6 (SD=2.6), and psychiatric comorbidity was common. Mean physical and mental health composite scale scores were nearly two standard deviations below those for the general population. Nearly half (48.4%) were already receiving depression treatment at enrollment, indicating treatment resistance.
Characteristicb | All Patients (N=364) | Telemedicine-Based Group (N=179) | Practice-Based Group (N=185) | |||
---|---|---|---|---|---|---|
N | % | N | % | N | % | |
Male | 67 | 18.4 | 32 | 17.9 | 35 | 18.9 |
Race/ethnicity | ||||||
Caucasian | 261 | 71.7 | 128 | 71.5 | 133 | 71.9 |
African American | 76 | 20.9 | 39 | 21.8 | 37 | 20.0 |
Native American | 18 | 5.0 | 7 | 3.9 | 11 | 6.0 |
Other | 9 | 2.5 | 5 | 2.8 | 4 | 2.2 |
Income | ||||||
<$10,000 | 104 | 29.7 | 59 | 33.7 | 45 | 25.7 |
$10,000–$14,999 | 84 | 24.0 | 42 | 24.0 | 42 | 24.0 |
$15,000–$19,999 | 56 | 16.0 | 29 | 16.6 | 27 | 15.4 |
$20,000–$29,999 | 61 | 17.4 | 25 | 14.3 | 36 | 20.6 |
$30,000–$39,999 | 24 | 6.9 | 10 | 5.7 | 14 | 8.0 |
$40,000–$49,999 | 12 | 3.4 | 6 | 3.4 | 6 | 3.4 |
≥$50,000 | 9 | 2.6 | 4 | 2.3 | 5 | 2.9 |
Rural residence | 248 | 68.1 | 119 | 66.5 | 129 | 69.7 |
Married | 162 | 44.5 | 79 | 44.1 | 83 | 44.9 |
High school graduate | 265 | 73.0 | 125 | 70.2 | 140 | 75.7 |
Employed | 129 | 35.5 | 57 | 31.8 | 72 | 39.1 |
Insurancec | ||||||
Public | 110 | 30.2 | 64 | 35.8 | 46 | 24.9 |
Private | 54 | 14.8 | 28 | 15.6 | 26 | 14.1 |
Public and private | 15 | 4.1 | 7 | 3.9 | 8 | 4.3 |
Uninsured | 185 | 50.8 | 80 | 44.7 | 105 | 56.8 |
Mean | SD | Mean | SD | Mean | SD | |
Social support (0–1) | 0.4 | 0.2 | 0.4 | 0.2 | 0.4 | 0.2 |
Perceived barriers (0–9)d | 3.7 | 2.0 | 4.0 | 2.1 | 3.4 | 1.9 |
Perceived need (0–6) | 3.0 | 1.5 | 3.1 | 1.5 | 3.0 | 1.5 |
Perceived treatment effectiveness (0–2) | 1.3 | 0.7 | 1.3 | 0.7 | 1.3 | 0.7 |
Hopkins Symptom Checklist score (depression severity; 0–4) | 1.9 | 0.7 | 1.9 | 0.8 | 1.9 | 0.7 |
Short-Form Health Survey, physical health composite score (0–100) | 36.9 | 13.4 | 35.7 | 13.1 | 38.0 | 13.7 |
Short-Form Health Survey, mental health composite score (0–100) | 31.3 | 11.2 | 32.4 | 11.1 | 30.3 | 11.2 |
Quality of Well-Being score (0–1) | 0.4 | 0.1 | 0.4 | 0.1 | 0.4 | 0.1 |
Chronic physical illness counte | 4.6 | 2.6 | 4.8 | 2.5 | 4.3 | 2.6 |
Prior depression episodes count | 4.2 | 1.6 | 4.2 | 1.6 | 4.2 | 1.6 |
Age (years) | 47.2 | 12.6 | 47.7 | 12.5 | 46.8 | 12.8 |
N | % | N | % | N | % | |
Age <18 years at depression onset | 144 | 41.0 | 67 | 39.2 | 77 | 42.8 |
Family history of depression | 209 | 58.2 | 110 | 62.5 | 99 | 54.1 |
Prior antidepressant treatment | 267 | 73.35 | 128 | 71.51 | 139 | 75.14 |
Prior counseling | 107 | 29.40 | 51 | 28.49 | 56 | 30.27 |
Current depression treatment | 176 | 48.4 | 84 | 46.9 | 92 | 49.7 |
Antidepressants acceptable | 303 | 85.1 | 149 | 84.7 | 154 | 85.6 |
Counseling acceptability | 273 | 76.9 | 136 | 77.7 | 137 | 76.1 |
Current major depressive disorder | 303 | 83.2 | 144 | 80.5 | 159 | 86.0 |
Current dysthymia | 12 | 3.3 | 7 | 3.9 | 5 | 2.7 |
Current panic disorder | 32 | 8.8 | 16 | 8.9 | 16 | 8.7 |
Current generalized anxiety disorder | 231 | 63.5 | 114 | 63.7 | 117 | 63.2 |
Current posttraumatic stress disorder | 58 | 15.9 | 30 | 16.8 | 28 | 15.1 |
Current at-risk drinking | 20 | 5.5 | 12 | 6.7 | 8 | 4.3 |
Care Manager Fidelity
Table 2 describes care manager fidelity. At the 6- and 12-month follow-ups, significantly more patients in the telemedicine-based group than in the practice-based group reported that a health care professional other than their primary care provider 1) gave them helpful information about depression or depression treatment (odds ratios, 2.77 at 6 months and 2.32 at 12 months); 2) made helpful suggestions about things they could do to help depression, such as exercise or becoming more socially active (odds ratios, 3.47 at 6 months and 2.50 at 12 months); 3) asked them about their depression symptoms (odds ratios, 3.60 at 6 months and 2.63 at 12 months); and 4) asked them if they were taking antidepressant medications as prescribed or attending scheduled counseling sessions (odds ratios, 4.70 at 6 months and 3.96 at 12 months). More patients in the telemedicine-based group also reported that their primary care provider worked collaboratively with a mental health specialist (odds ratios, 4.63 at 6 months and 9.05 at 12 months).
Measure | Telemedicine-Based Group | Practice-Based Group | Group Comparison | Analysis | ||||
---|---|---|---|---|---|---|---|---|
Patient Self-Report | N | % | N | % | p | Adjusted Odds Ratio | 95% CI | p |
Educationa | ||||||||
6 months | <0.001 | 2.77 | 1.67, 4.61 | <0.001 | ||||
Frequently | 38/153 | 24.84 | 18/165 | 10.91 | ||||
Occasionally | 39/153 | 25.49 | 29/165 | 17.58 | ||||
Not at all | 76/153 | 49.67 | 118/165 | 71.52 | ||||
12 months | 0.010 | 2.32 | 1.37, 3.94 | 0.002 | ||||
Frequently | 24/138 | 17.39 | 11/149 | 7.38 | ||||
Occasionally | 31/138 | 22.46 | 26/149 | 17.45 | ||||
Not at all | 83/138 | 60.14 | 112/149 | 75.17 | ||||
Self-managementb | ||||||||
6 months | <0.001 | 3.47 | 2.15, 5.62 | <0.001 | ||||
Frequently | 49/153 | 32.03 | 22/165 | 13.33 | ||||
Occasionally | 51/153 | 33.33 | 42/165 | 25.45 | ||||
Not at all | 53/153 | 34.64 | 101/165 | 61.21 | ||||
12 months | <0.001 | 2.50 | 1.48, 4.23 | <0.001 | ||||
Frequently | 35/138 | 25.36 | 12/149 | 8.05 | ||||
Occasionally | 25/138 | 18.12 | 31/149 | 20.81 | ||||
Not at all | 78/138 | 56.52 | 106/149 | 71.14 | ||||
Symptom monitoringc | ||||||||
6 months | <0.001 | 3.60 | 2.21, 5.86 | <0.001 | ||||
Frequently | 53/153 | 34.64 | 24/165 | 14.55 | ||||
Occasionally | 41/153 | 26.80 | 29/165 | 17.58 | ||||
Not at all | 59/153 | 38.56 | 112/165 | 67.88 | ||||
12 months | <0.001 | 2.63 | 1.54, 4.52 | <0.001 | ||||
Frequently | 34/138 | 24.64 | 12/149 | 8.05 | ||||
Occasionally | 23/138 | 16.67 | 27/149 | 18.12 | ||||
Not at all | 81/138 | 58.70 | 110/149 | 73.83 | ||||
Antidepressant monitoringd | ||||||||
6 months | <0.001 | 4.70 | 2.82, 7.84 | <0.001 | ||||
Frequently | 46/153 | 30.07 | 18/165 | 10.91 | ||||
Occasionally | 47/153 | 30.72 | 28/165 | 16.97 | ||||
Not at all | 60/153 | 39.22 | 119/165 | 72.12 | ||||
12 months | ||||||||
Frequently | 33/138 | 23.91 | 11/149 | 7.38 | <0.001 | 3.96 | 2.22, 7.05 | <0.001 |
Occasionally | 27/138 | 19.57 | 18/149 | 12.08 | ||||
Not at all | 78/138 | 56.52 | 120/149 | 80.54 | ||||
Collaboratione | ||||||||
6 months | 0.001 | 4.63 | 2.08, 10.30 | <0.001 | ||||
Frequently | 22/96 | 22.92 | 8/112 | 7.14 | ||||
Occasionally | 13/96 | 13.54 | 9/112 | 8.04 | ||||
Not at all | 61/96 | 63.54 | 95/112 | 84.82 | ||||
12 months | 0.006 | 9.05 | 3.04, 26.93 | <0.001 | ||||
Frequently | 12/91 | 13.19 | 4/111 | 3.60 | ||||
Occasionally | 10/91 | 10.99 | 5/111 | 4.50 | ||||
Not at all | 69/91 | 75.82 | 102/111 | 91.89 | ||||
Chart review | Mean | SD | Mean | SD | p | Adjusted Incidence Rate Ratio | 95% CI | p |
Educationf | ||||||||
0–6 months | 0.56 | 0.65 | 0.16 | 0.48 | <0.001 | 2.03 | 1.38, 2.98 | <0.001 |
6–12 months | 0.02 | 0.13 | 0.02 | 0.15 | 0.743 | 1.06 | 0.27, 4.17 | 0.932 |
Self-management | ||||||||
0–6 months | 4.12 | 2.56 | 0.77 | 1.47 | <0.001 | 5.62 | 4.46, 7.07 | <0.001 |
6–12 months | 2.08 | 2.70 | 0.26 | 0.92 | <0.001 | 8.84 | 5.44, 14.35 | <0.001 |
Symptom monitoring | ||||||||
0–6 months | 4.85 | 2.69 | 1.19 | 1.65 | <0.001 | 4.10 | 3.41, 4.92 | <0.001 |
6–12 months | 2.11 | 2.73 | 0.44 | 1.07 | <0.001 | 4.64 | 3.19, 6.74 | <0.001 |
Antidepressant monitoringg | ||||||||
0–6 months | 5.40 | 2.29 | 1.97 | 1.81 | <0.001 | 2.69 | 2.20, 3.28 | <0.001 |
6–12 months | 3.94 | 2.49 | 1.47 | 1.72 | <0.001 | 2.49 | 1.69, 3.67 | <0.001 |
Side effect monitoringf,g | ||||||||
0–6 months | 5.18 | 2.29 | 1.24 | 1.49 | <0.001 | 4.22 | 3.32, 5.36 | <0.001 |
6–12 months | 3.73 | 2.45 | 0.80 | 1.19 | <0.001 | 4.46 | 2.75, 7.25 | <0.001 |
Counseling monitoringg | ||||||||
0–6 months | 3.44 | 2.68 | 2.70 | 1.64 | 0.417 | 1.13 | 0.63, 2.05 | 0.664 |
6–12 months | 2.57 | 2.27 | 2.00 | 2.71 | 0.658 | 2.91 | 0.08, 110.94 | 0.419 |
Group Differences in Fidelity to the Care Manager Protocol in a Randomized Study of Practice-Based or Telemedicine-Based Collaborative Care
Compared with patients in the practice-based group, those in the telemedicine-based group had significantly more depression care manager encounters in which Patient Health Questionnaire depression severity scores were documented in the medical record during both the first 6 months and the second 6 months (incidence rate ratios, 4.10 and 4.64, respectively). Patients in the telemedicine-based group also had significantly more depression care manager encounters in which self-management goals were documented during the first 6 months (incidence rate ratio=5.62). Among patients who received a prescription for an antidepressant, those in the telemedicine-based group had significantly more depression care manager encounters in which medication adherence was documented during both the first 6 months and the second 6 months (incidence rate ratios, 2.69 and 2.49, respectively). Likewise, patients in the telemedicine-based group who received a prescription for an antidepressant had significantly more depression care manager encounters in which the presence or absence of side effects was documented during both the first 6 months and the second 6 months (incidence rate ratios, 4.22 and 4.46, respectively). Among patients who were referred to counseling, there were no significant group differences in the number of depression care manager encounters in which session attendance was documented in the medical record.
Stepped Care Fidelity
Among patients in the telemedicine-based group who failed to respond to at least one medication trial (N=73), 8.2% (N=6) had a telephone encounter with the pharmacist. Among those who had failed to respond to two trials (N=29), 48.3% (N=14) had a telepsychiatry consultation. Another seven patients had an ad hoc telepsychiatry consultation. Also, 16.6% of patients (N=30) attended at least one telepsychotherapy session, and 7.8% (N=14) completed eight or more sessions.
Treatment Process Outcomes
Table 3 summarizes the treatment provided to the two groups. There were no significant group differences at baseline, and no significant group-by-time interactions for number of primary care visits or number of depression-related primary care visits, or mental health visits. With respect to antidepressant treatment, there were no significant group main effects for receiving an antidepressant prescription, number of prescribed antidepressants, dosage level (starting, usual, high), or adherence.
Measure | Telemedicine-Based Group | Practice-Based Group | Unadjusted Analysis | Adjusted Analysis | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Mean | SD | Mean | SD | Difference | p | Incidence Rate Ratio | 95% CI | p | Omnibus Test p | ||
Service utilization | |||||||||||
Primary care visits | 1.16 | 0.98, 1.36 | 0.081 | 0.16 | |||||||
Baseline | 4.25 | 3.73 | 3.63 | 3.04 | 0.62 | 0.081 | |||||
6 months | 3.82 | 3.34 | 3.69 | 3.98 | 0.13 | 0.749 | |||||
12 months | 3.09 | 2.71 | 3.39 | 3.44 | –0.30 | 0.419 | |||||
18 months | 2.72 | 2.36 | 2.98 | 3.54 | –0.26 | 0.463 | |||||
Depression-related primary care visitsa | 0.99 | 0.72, 1.37 | 0.958 | 0.29 | |||||||
Baseline | 1.07 | 1.43 | 1.16 | 1.55 | –0.09 | 0.592 | |||||
6 months | 1.07 | 1.93 | 1.17 | 2.24 | –0.10 | 0.658 | |||||
12 months | 0.68 | 1.31 | 1.05 | 1.98 | –0.37 | 0.068 | |||||
18 months | 0.58 | 1.17 | 0.94 | 2.06 | –0.36 | 0.080 | |||||
N | % | N | % | Odds Ratio | 95% CI | p | Odds Ratio | 95% CI | p | Omnibus p | |
Any specialty mental health visitsb | 0.56 | 0.24, 1.82 | 0.425 | 0.16 | |||||||
Baseline | 8/179 | 4.47 | 12/185 | 6.49 | 0.67 | 0.27, 1.69 | 0.399 | ||||
6 months | 31/153 | 20.26 | 21/165 | 12.73 | 1.74 | 0.95, 3.19 | 0.070 | ||||
12 months | 20/138 | 14.49 | 13/149 | 8.72 | 1.77 | 0.85, 3.72 | 0.126 | ||||
18 months | 13/132 | 9.85 | 18/151 | 11.92 | 0.81 | 0.38, 1.72 | 0.578 | ||||
Antidepressant medications | |||||||||||
Any prescription | 1.64 | 0.75, 3.58 | 0.212 | 0.37 | |||||||
Baseline | — | — | |||||||||
6 months | 100/153 | 65.36 | 100/165 | 60.61 | 1.23 | 0.78, 1.94 | 0.381 | ||||
12 months | 91/138 | 65.94 | 86/149 | 57.72 | 1.42 | 0.88, 2.29 | 0.152 | ||||
18 months | 74/132 | 56.06 | 86/151 | 56.95 | 0.96 | 0.60, 1.54 | 0.880 | ||||
Adherencec,d | 1.22 | 0.38, 3.89 | 0.737 | 0. 86 | |||||||
Baseline | — | — | |||||||||
6 months | 82/92 | 89.13 | 81/95 | 85.26 | 1.42 | 0.60, 3.38 | 0.429 | ||||
12 months | 74/86 | 86.05 | 67/80 | 83.75 | 1.20 | 0.51, 2.80 | 0.679 | ||||
18 months | 62/69 | 89.86 | 73/81 | 90.12 | 0.97 | 0.33, 2.83 | 0.957 | ||||
Dosage levele | 1.84 | 0.77, 4.38 | 0.169 | 0.69 | |||||||
Baseline | — | — | |||||||||
6 months | 0.147 | ||||||||||
Starting | 44/98 | 44.90 | 45/92 | 48.91 | |||||||
Usual | 42/98 | 42.86 | 43/92 | 46.74 | |||||||
High | 12/98 | 12.24 | 4/92 | 4.35 | |||||||
12 months | |||||||||||
Starting | 32/86 | 37.21 | 34/85 | 40.00 | 0.551 | ||||||
Usual | 40/86 | 46.51 | 42/85 | 49.41 | |||||||
High | 14/86 | 16.28 | 9/85 | 10.59 | |||||||
18 months | |||||||||||
Starting | 27/69 | 39.13 | 37/79 | 46.84 | 0.444 | ||||||
Usual | 33/69 | 47.83 | 36/79 | 45.57 | |||||||
High | 9/69 | 13.04 | 6/79 | 7.59 | |||||||
Mean | SD | Mean | SD | Difference | p | Incidence Rate Ratio | 95% CI | p | Omnibus p | ||
Number of prescribed antidepressants | 1.19 | 0.92, 1.56 | 0.191 | 0.49 | |||||||
Baseline | — | — | |||||||||
6 months | 0.75 | 0.62 | 0.68 | 0.62 | 0.07 | 0.339 | |||||
12 months | 0.77 | 0.65 | 0.66 | 0.63 | 0.11 | 0.174 | |||||
18 months | 0.62 | 0.60 | 0.69 | 0.68 | –0.07 | 0.377 |
Group Differences in Services Utilization and Antidepressant Treatment in a Randomized Study of Practice-Based or Telemedicine-Based Collaborative Care
Clinical Outcomes
Clinical outcomes are presented in Table 4. A significant group main effect was observed for both response (odds ratio=7.74) and remission (odds ratio=12.69), with patients in the telemedicine-based group having better outcomes. We also observed a significant overall group-by-time interaction effect for HSCL score (χ2=40.51, df=3, p<0.001), with greater reductions in severity for patients in the telemedicine-based group (Figure 1). There was also a significant overall group-by-time interaction effect for mental health composite scale score (χ2=11.46, df=3, p=0.01) and Quality of Well-Being score (χ2=6.55, df=3, p=0.09), with greater improvements among patients in the telemedicine-based group. There were no significant overall group-by-time interaction effects for physical health composite scale score, although this finding is likely an artifact resulting from the orthogonal factor rotation and negative weights used to score the physical and mental health composite scales (55). There were no group differences in satisfaction at baseline, but the omnibus test of the group-by-time interactions was significant at the 0.10 level (χ2=6.69, df=3, p=0.08), with the telemedicine-based group having higher satisfaction.
Measure | Telemedicine-Based Group | Practice-Based Group | Unadjusted Analysis | Adjusted Analysis | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|
N | % | N | % | Odds Ratio | 95% CI | p | Odds Ratio | 95% CI | p | Omnibus p | |
Satisfactiona | 0.08 | ||||||||||
Baseline | 79/167 | 47.31 | 82/176 | 46.59 | 1.03 | 0.67, 1.57 | 0.895 | 1.08 | 0.64, 1.83 | 0.765 | |
6 months | 120/150 | 80.00 | 101/159 | 63.52 | 2.30 | 1.37, 3.84 | 0.001 | 2.76 | 1.50, 5.01 | 0.001 | |
12 months | 99/133 | 74.44 | 87/140 | 62.14 | 1.77 | 1.06, 2.98 | 0.029 | 1.99 | 1.06, 3.71 | 0.031 | |
18 months | 95/128 | 74.22 | 96/148 | 64.86 | 1.56 | 0.93, 2.62 | 0.093 | 1.67 | 0.89, 3.13 | 0.107 | |
Responseb | 7.74 | 3.94, 15.20 | <0.001 | 0.52 | |||||||
Baseline | — | — | |||||||||
6 months | 70/153 | 45.75 | 25/165 | 15.15 | 4.72 | 2.78, 8.03 | <0.001 | ||||
12 months | 73/138 | 52.90 | 31/149 | 20.81 | 4.27 | 2.55, 7.18 | <0.001 | ||||
18 months | 63/132 | 47.73 | 33/151 | 21.85 | 3.26 | 1.95, 5.47 | <0.001 | ||||
Remissionc | 12.69 | 4.81, 33.46 | <0.001 | 0.27 | |||||||
Baseline | — | — | |||||||||
6 months | 44/153 | 28.76 | 11/165 | 6.67 | 5.65 | 2.79, 11.44 | <0.001 | ||||
12 months | 43/138 | 31.16 | 17/149 | 11.41 | 3.51 | 1.89, 6.54 | <0.001 | ||||
18 months | 34/132 | 25.76 | 15/151 | 9.93 | 3.15 | 1.62, 6.09 | <0.001 | ||||
Mean | SD | Mean | SD | Group Difference | p | Group Difference | 95% CI | p | Omnibus p | ||
Hopkins Symptom Checklist score (depression severity; 0–4) | <0.001 | ||||||||||
Baseline | 1.88 | 0.77 | 1.90 | 0.72 | –0.02 | 0.732 | –0.04 | –0.18, 0.10 | 0.594 | ||
6 months | 1.16 | 0.90 | 1.64 | 0.75 | –0.48 | <0.001 | –0.50 | –0.65, –0.35 | <0.001 | ||
12 months | 1.04 | 0.79 | 1.53 | 0.85 | –0.49 | <0.001 | –0.49 | –0.65, –0.33 | <0.001 | ||
18 months | 1.13 | 0.85 | 1.49 | 0.75 | –0.36 | <0.001 | –0.33 | –0.49, –0.18 | <0.001 | ||
Short-Form Health Survey, mental health composite score (0–100) | 0.01 | ||||||||||
Baseline | 32.39 | 11.08 | 30.31 | 11.22 | 2.08 | 0.076 | 1.82 | –0.65, 4.30 | 0.149 | ||
6 months | 44.34 | 14.22 | 37.49 | 11.98 | 6.85 | <0.001 | 6.53 | 3.91, 9.15 | <0.001 | ||
12 months | 46.53 | 12.98 | 40.58 | 13.20 | 5.94 | <0.001 | 5.39 | 2.66, 8.12 | <0.001 | ||
18 months | 44.60 | 13.52 | 39.86 | 12.24 | 4.75 | 0.002 | 4.01 | 1.26, 6.76 | 0.004 | ||
Short-Form Health Survey, physical health composite score (0–100) | –0.22 | –2.26, 1.82 | 0.834 | 0.46 | |||||||
Baseline | 35.73 | 13.09 | 37.96 | 13.68 | –2.23 | 0.112 | |||||
6 months | 33.93 | 12.54 | 35.98 | 13.54 | –2.05 | 0.162 | |||||
12 months | 32.98 | 13.18 | 35.62 | 12.77 | –2.64 | 0.086 | |||||
18 months | 34.52 | 12.53 | 35.31 | 13.22 | –0.79 | 0.609 | |||||
Quality of Well-Being score (0–1) | 0.09 | ||||||||||
Baseline | 0.43 | 0.15 | 0.44 | 0.14 | –0.01 | 0.608 | 0.01 | –0.02, 0.03 | 0.655 | ||
6 months | 0.49 | 0.18 | 0.46 | 0.15 | 0.03 | 0.097 | 0.03 | 0.003, 0.06 | 0.032 | ||
12 months | 0.50 | 0.17 | 0.49 | 0.17 | 0.02 | 0.400 | 0.03 | 0.002, 0.06 | 0.037 | ||
18 months | 0.50 | 0.15 | 0.48 | 0.17 | 0.02 | 0.220 | 0.04 | 0.02, 0.07 | 0.002 |
Group Differences in Clinical Outcomes in a Randomized Study of Practice-Based or Telemedicine-Based Collaborative Care
Discussion
Study participants were recruited from small, remote primary care clinics associated with the largest publicly funded health care system in the country. The sample was predominantly rural, unemployed, and uninsured. Patients had numerous comorbidities, and their depression was treatment resistant. The high degree of treatment resistance likely contributed to the low response and remission rates among those assigned to the practice-based collaborative care group. Compared with patients assigned to the practice-based collaborative care group, those in the telemedicine-based group had significantly and substantially greater treatment response rates, remission rates, reductions in depression severity, and increases in mental health status and quality of life. Improved outcomes were achieved in the telemedicine-based group without increasing the number of primary care visits. Improved outcomes appear to be due to higher fidelity to the care manager protocol in the telemedicine-based collaborative care model, despite the fact that off-site and on-site depression care managers had similar levels of clinical experience at baseline and underwent identical training. The higher fidelity to the care manager protocol in the telemedicine-based model may have been due to the fact that the off-site depression care manager practiced under more intensive clinical supervision and dedicated 100% of her time to depression care manager activities.
The greater fidelity to the care manager protocol did not translate into improvements in the quality of pharmacotherapy for the telemedicine-based group. The majority of patients in both groups initiated antidepressant treatment at therapeutic dosages, and adherence was high. Likewise, the psychotherapy available to patients in the telemedicine-based group via videoconferencing was not highly utilized and was not likely to have contributed substantially to improved outcomes. Our findings were essentially unchanged when patients who received telepsychotherapy were excluded from the analytical sample. Because group differences in outcomes are not likely to be attributable to either pharmacotherapy or psychotherapy, we hypothesize that patients assigned to telemedicine-based collaborative care were more likely to engage in self-management activities, such as physical, rewarding, and social activities. This hypothesis is based on the finding that patients in the telemedicine-based group received more encouragement from the depression care manager to engage in self-management activities. Previous research has demonstrated that behavioral activation is a clinically effective stand-alone treatment for depression (56, 57). Also, patients in the practice-based collaborative care intervention in Project Impact (58) were found to have significantly better treatment response rates if the depression care manager documented in the medical record that the patient scheduled physical, rewarding, and social activities. This hypothesis needs to be tested using an experimental dismantling study specifically designed to estimate the incremental treatment effect of scheduling self-management activities. Another possibility is that the more frequent depression care manager encounters provided to the telemedicine-based group resulted in greater social support, which in turn reduced depressive symptoms. This was the conclusion of Hunkeler et al. (59), who similarly reported that a depression care manager program at Kaiser Permanente did not improve antidepressant management, but did reduce depression symptoms.
An inherent limitation of this pragmatic trial is that the results are not conclusive with respect to identifying treatment mechanisms. Explanatory trial designs are needed to determine why patients assigned to receive telemedicine-based collaborative care had better outcomes than those assigned to receive practice-based collaborative care. However, for policy makers at HRSA and federally qualified health centers lacking on-site mental health personnel, these results clearly indicate that contracting with an off-site depression care team yields better depression outcomes than implementing collaborative care with staff available on-site. Future research should also examine whether having on-site nurse care managers supported by off-site psychiatrists, psychologists, and pharmacists via telephone and videoconferencing is an effective organizational approach to delivering collaborative care.
1 : Stepped collaborative care for primary care patients with persistent symptoms of depression: a randomized trial. Arch Gen Psychiatry 1999; 56:1109–1115Crossref, Google Scholar
2 : A multifaceted intervention to improve treatment of depression in primary care. Arch Gen Psychiatry 1996; 53:924–932Crossref, Google Scholar
3 : Improving depression outcomes in community primary care practice: a randomized trial of the quEST intervention. J Gen Intern Med 2001; 16:143–149Crossref, Google Scholar
4 : Impact of disseminating quality improvement programs for depression in managed primary care: a randomized controlled trial. JAMA 2000; 283:212–220Crossref, Google Scholar
5 : Collaborative care management of late-life depression in the primary care setting: a randomized controlled trial. JAMA 2002; 288:2836–2845Crossref, Google Scholar
6 : Remission in depressed geriatric primary care patients: a report from the PROSPECT study. Am J Psychiatry 2005; 162:718–724Crossref, Google Scholar
7 : Reducing suicidal ideation and depressive symptoms in depressed older primary care patients: a randomized controlled trial. JAMA 2004; 291:1081–1091Crossref, Google Scholar
8 : Primary care physicians’ approach to depressive disorders: effects of physician specialty and practice structure. Arch Fam Med 1999; 8:58–67Crossref, Google Scholar
9 : Case management for depression by health care assistants in small primary care practices: a cluster randomized trial. Ann Intern Med 2009; 151:369–378Crossref, Google Scholar
10 : Educational and organizational interventions to improve the management of depression in primary care: a systematic review. JAMA 2003; 289:3145–3151Crossref, Google Scholar
11 : A randomized trial of telemedicine-based collaborative care for depression. J Gen Intern Med 2007; 22:1086–1093Crossref, Google Scholar
12 : Cost-effectiveness analysis of a rural telemedicine collaborative care intervention for depression. Arch Gen Psychiatry 2010; 67:812–821Crossref, Google Scholar
13 : Implementation outcomes of evidence-based quality improvement for depression in VA community based outpatient clinics. Implement Sci 2012; 7:30Crossref, Google Scholar
14 : Acceptability of telepsychiatry to a rural population. Psychiatr Serv 2000; 51:672–674Crossref, Google Scholar
15 : Patient satisfaction with telemedicine consultation in primary care: comparison of ratings of medical and mental health applications. Telemed J 1998; 4:363–369Crossref, Google Scholar
16 : Assessment of therapeutic alliance in face-to-face or videoconference treatment for posttraumatic stress disorder. Cyberpsychol Behav Soc Netw 2010; 13:29–35Crossref, Google Scholar
17 : The provision of a telepsychology service to Shetland: client and therapist satisfaction and the ability to develop a therapeutic alliance. J Telemed Telecare 2001; 7(suppl 1):34–36Crossref, Google Scholar
18 : Treatment outcomes in depression: comparison of remote treatment through telepsychiatry to in-person treatment. Am J Psychiatry 2004; 161:1471–1476Crossref, Google Scholar
19 : Reliability and acceptability of psychiatric diagnosis via telecommunication and audiovisual technology. Psychiatr Serv 1998; 49:1086–1088Crossref, Google Scholar
20 : Is telepsychiatry equivalent to face-to-face psychiatry? Results from a randomized controlled equivalence trial. Psychiatr Serv 2007; 58:836–843Crossref, Google Scholar
21 : The effectiveness of telemental health applications: a review. Can J Psychiatry 2008; 53:769–778Crossref, Google Scholar
22 : Treating childhood depression over videoconferencing. Telemed J E Health 2003; 9:49–55Crossref, Google Scholar
23 : Delivering cognitive-behavior therapy for panic disorder with agoraphobia in videoconference. Telemed J E Health 2004; 10:13–25Crossref, Google Scholar
24 : A randomized trial of telepsychiatry for post-traumatic stress disorder. J Telemed Telecare 2007; 13:142–147Crossref, Google Scholar
25 : Effectiveness of cognitive behavioural therapy administered by videoconference for posttraumatic stress disorder. Cogn Behav Ther 2009; 38:42–53Crossref, Google Scholar
26 : Telemedicine for anger management therapy in a rural population of combat veterans with posttraumatic stress disorder: a randomized noninferiority trial. J Clin Psychiatry 2010; 71:855–863Crossref, Google Scholar
27 : A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol 2009; 62:464–475Crossref, Google Scholar
28 : Initial National Priorities for Comparative Effectiveness Research. Washington, DC, National Academies Press, 2009Google Scholar
29 : Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 2003; 290:1624–1632Crossref, Google Scholar
30 : Trends in mental health and substance abuse services at the nation’s community health centers: 1998–2003. Am J Public Health 2006; 96:1779–1784Crossref, Google Scholar
31 National Association of Community Health Centers: US Health Center Fact Sheet: 2006. Bethesda, Md, National Association of Community Health Centers, 2007 (www.nachc.com/client/documents/US_Fact_Sheet_2007.pdf)Google Scholar
32 : Improving the management of chronic disease at community health centers. N Engl J Med 2007; 356:921–934Crossref, Google Scholar
33 : Implementation and maintenance of quality improvement for treating depression in primary care. Psychiatr Serv 2006; 57:48–55Crossref, Google Scholar
34 : A web-based clinical decision support system for depression care management. Am J Manag Care 2010; 16:849–854Google Scholar
35 Smith GR Jr, Burnam A, Burns BJ, Cleary P, Rost KM: Depression Outcomes Module (DOM), in Handbook of Psychiatric Measures. Edited by American Psychiatric Association. Washington, DC, American Psychiatric Association, 2000, pp 213–215Google Scholar
36 : Depression Outcomes Module. Little Rock, Ark, The Guide to Behavioral Health Outcomes Management Systems, 2000, pp 71–83Google Scholar
37 : The Mini International Neuropsychiatric Interview (MINI), a short diagnostic structured interview: reliability and validity according to the CIDI. Eur Psychiatry 1997; 12:224–231Crossref, Google Scholar
38 : The validity of the Mini International Neuropsychiatric Interview (MINI) according to the SCID-P and its reliability. Eur Psychiatry 1997; 12:232–241Crossref, Google Scholar
39 : Associations among family support, family stress, and personal functional health status. J Clin Epidemiol 1989; 42:217–229Crossref, Google Scholar
40 : Quality of life and functional health of primary care patients. J Clin Epidemiol 1992; 45:1303–1313Crossref, Google Scholar
41 : Beliefs about depression and depression treatment among depressed veterans. Med Care 2008; 46:581–589Crossref, Google Scholar
42 : Outcomes of “inadequate” antidepressant treatment. J Gen Intern Med 1995; 10:663–670Crossref, Google Scholar
43 : The Hopkins Symptom Checklist (HSCL): a measure of primary symptom dimensions. Mod Probl Pharmacopsychiatry 1974; 7:79–110Crossref, Google Scholar
44 : The Hopkins Symptom Checklist (HSCL): a self-report symptom inventory. Behav Sci 1974; 19:1–15Crossref, Google Scholar
45 : Health status assessments using the Veterans SF-12 and SF-36: methods for evaluating otucomes in the Veterans Health Administration. J Ambul Care Manage 2001; 24:68–86Crossref, Google Scholar
46 : Health-related quality of life in patients served by the Department of Veterans Affairs: results from the Veterans Health Study. Arch Intern Med 1998; 158:626–632Crossref, Google Scholar
47 : The general health policy model: an integrated approach, in Quality of Life and Pharmacoeconomics in Clinical Trials, 2nd ed. Edited by Spiker B. Philadelphia, Lippincott-Raven, 1996, pp 309–321Google Scholar
48 : Health-related quality of life measurement for evaluation research and policy analysis. Health Psychol 1982; 1:61–80Crossref, Google Scholar
49 : Assessment of the quality of life of patients with major depression. Psychiatr Serv 1997; 48:224–230Crossref, Google Scholar
50 : Preliminary longitudinal assessment of quality of life in patients with major depression. Psychopharmacol Bull 1997; 33:23–29Google Scholar
51 : Evaluating behavioral health services in Minnesota’s Medicaid population using the Experience of Care and Health Outcomes (ECHO) Survey. J Health Care Poor Underserved 2003; 14:608–621Crossref, Google Scholar
52 : SAS System for Mixed Models. Cary, NC, SAS Institute Inc., 1996Google Scholar
53 : Purposeful selection of variables in logistic regression. Source Code Biol Med 2008; 3:17Crossref, Google Scholar
54 : Intention-to-treat analyses for incomplete repeated measures data. Biometrics 1996; 52:1002–1017Crossref, Google Scholar
55 : SF-36 summary scores: are physical and mental health truly distinct? Med Care 1998; 36:567–572Crossref, Google Scholar
56 : A qualitative and quantitative review of behavioral activation treatment of major depressive disorder. Behav Analyst Today 2006; 74:508–518Crossref, Google Scholar
57 : Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. J Consult Clin Psychol 2006; 74:658–670Crossref, Google Scholar
58 : Activity scheduling as a core component of effective care management for late-life depression. Int J Geriatr Psychiatry 2012; 27:1298–1304Crossref, Google Scholar
59 : Efficacy of nurse telehealth care and peer support in augmenting treatment of depression in primary care. Arch Fam Med 2000; 9:700–708Crossref, Google Scholar