The American Psychiatric Association (APA) has updated its Privacy Policy and Terms of Use, including with new information specifically addressed to individuals in the European Economic Area. As described in the Privacy Policy and Terms of Use, this website utilizes cookies, including for the purpose of offering an optimal online experience and services tailored to your preferences.

Please read the entire Privacy Policy and Terms of Use. By closing this message, browsing this website, continuing the navigation, or otherwise continuing to use the APA's websites, you confirm that you understand and accept the terms of the Privacy Policy and Terms of Use, including the utilization of cookies.

×
INFLUENTIAL PUBLICATIONSFull Access

A Vision of the Next Generation of Behavioral Therapies Research in the Addictions

Published Online:https://doi.org/10.1176/foc.9.1.foc118

Abstract

Whither, or wither, empirically supported therapies? Increasingly rigorous research in behavioral therapies has yielded a large number of effective therapies, but comparatively little work, demonstrating that integrating empirically supported therapies (ESTs) into standard practice results in meaningful improvements in patient outcomes. Methodology and strategies for evaluating ESTs and their effectiveness in clinical practice is a fairly recent innovation, and a host of unanswered questions remain regarding issues such as selection among different ESTs and what type of ESTs should be emphasized in dissemination efforts, what type of clinicians should be trained in what type of ESTs, the most effective training strategies for various types of clinicians, the need for ongoing supervision to maintain minimum levels of treatment fidelity and skill. In this review, we call for broader use of new research strategies and methods relevant to dissemination of ESTs; these may include adaptive designs, identification of mechanisms of action to foster greater emphasis on effective change principles, training and adoption trials, as well as novel implementation strategies including computer-assisted therapy and computer-assisted training.

(Reprinted with permission from Addiction, 2007; 102:850–862)

EMPIRICALLY SUPPORTED THERAPIES: ‘WHEN THE GODS WISH TO PUNISH US, THEY ANSWER OUR PRAYERS' (OSCAR WILDE, AN IDEAL HUSBAND)

By almost any measure, treatment efficacy research in the addictions has been a brilliant success. Multiple well-conducted clinical trials have yielded impressive empirical support for a number of well-defined behavioral therapies (15), that to a large extent parallel the progress of behavioral therapies research for the larger range of psychiatric disorders (68). The enhanced focus on using rigorous methodology to identify effective therapies not only yielded numerous ‘empirically supported therapies' (ESTs), but also led in part to current efforts to improve the quality of psychological treatments for mental disorders. These arose first in the United Kingdom, with the broader movement toward evidence-based medicine (9), and then in the United States, with the efforts of groups such as the Task Force on Promotion and Dissemination of Psychological Procedures of Division 12 (Clinical Psychology) of the American Psychological Association (7, 1012) that described standards for defining ESTs (or closely related concepts such as ‘empirically validated therapies', ‘empirically based treatments', ‘science-based approaches', and so on), and which classified multiple interventions in terms of their being ‘well-established/efficacious and specific', ‘probably efficacious' or ‘promising'. The criteria used to determine whether an intervention qualifies as ‘empirically supported' consists typically of (a) whether there are at least two prospective, independent, randomized clinical trials in which the treatment has been compared to a well-defined control or comparison condition, (b) those trials have adequate statistical power, (c) the patient groups are defined using explicit inclusion/exclusion criteria, (d) the intervention is described in a manual or another clearly defined method useable as the basis for training, (e) treatment fidelity is monitored or treatment integrity is evaluated and (f) outcome assessments are well-validated and blinded as to treatment condition (68). Using these standards, there are now at least 10 behavioral therapies that are classified as ‘efficacious' or ‘probably efficacious' for the addictions and well over 100 ESTs for a range of other mental disorders, many of which co-occur with addictions in adults (7).

The proliferation of ESTs and the efforts to define and label them have been controversial and sparked important discourse among researchers and clinicians (7, 1318) revolving around the questions, ‘Now that we have all these ESTs, what should we do with them?' and ‘What do we need to know about the efficacy, value, and transferability of ESTs into clinical practice?'. In this paper, we will first consider some of the implications and questions that rise from the burgeoning number of ESTs, specifically for the field of addictions treatment research. Next, we will review briefly some of the strengths and weaknesses of current methods for moving ESTs into clinical practice. Finally, we will articulate a series of research questions for training, treatment programs and treatment research that follow from these implications.

The theoretical and practical questions regarding the identification of ESTs and how they should affect clinical practice are particularly pressing in the addictions, where the gap between research knowledge and clinical practice is both persistent and formidable (1921). Despite the increasing availability of number of effective treatments in the past 20 years, the majority of treatment programs in the United States remain grounded in traditional counseling models that have largely not been evaluated rigorously (19, 22). In addition, many drug abuse treatment programs persist in their use of interventions and strategies that have been demonstrated to be ineffective (e.g. acupuncture for drug dependence) (23) and even some that may be harmful to some populations (e.g. group treatments for antisocial adolescents) (24). It is also common practice for programs to offer high-cost services with little long-term effectiveness (e.g. opioid detoxification without follow-up treatment) (25, 26), presumably because these services remain covered by third-party payors. Moreover, the quality of clinician training is variable and rarely includes meaningful training in ESTs; rates of clinician turnover within programs remain unacceptably high. Finally, objective assessment of clinician/program performance and patient outcomes remain quite rare in clinical practice (2729). That is, while it is increasingly common in general and specialty medical settings for consumers to have access to data on physician performance and rates of successful outcomes, even the most basic information on retention and substance abuse outcomes is often not even collected systematically, much less available to prospective patients as a basis for making an informed choice about a particular clinical program.

It is also becoming clear that establishing the efficacy of a treatment approach via clinical trials by no means ensures it is used in clinical practice. Although rigorously conducted efficacy studies have validated a broad range of ESTs, they have addressed very few of the key questions regarding dissemination of those approaches. For example, how should clinicians choose among the various ESTs? Multiple large-scale randomized trials have identified which treatments ‘work' but have provided very little guidance regarding the relative superiority of the treatment alternatives (11). Luborsky's meta-analysis of the studies that have compared active treatments suggested a fairly small effect size (0.20) between different therapies (30, 31), indicating that while ESTs have been demonstrated to be effective relative to control therapies, research has yet to identify meaningful outcome differences across various treatments. Similarly, the available literature on treatment-matching and the identification of patient predictors of outcome has provided relatively little clinically meaningful guidance regarding the type of individuals who are most (or least) likely to respond to specific ESTs (13, 32, 33).

Beyond the lack of data demonstrating the relative superiority of any specific treatment and the lack of data regarding powerful patient moderators of outcome, clinical research has also been largely silent on the question: ‘Are ESTs actually superior to standard clinical practice; that is, do they work in the real world as well as the ivory tower?'. Until recently, there were very few studies that compared a specific EST to standard clinical practice or a ‘clinician's choice' condition (15, 34, 35). Although clinical trials conducted in community settings do generally replicate the findings of those conducted in more highly controlled research settings (3537), research has addressed infrequently the issue of the extent to which introducing ESTs into clinical practice will result in significant or clinically meaningful improvements in outcome (13, 15, 38). Without such data, it has been and will remain difficult to generate enthusiasm among policy makers or clinicians to alter their standard practice.

In the United States, the National Institute on Drug Abuse's Clinical Trials Network (CTN) (39) is making important progress on this question by conducting a series of multi-site randomized clinical trials evaluating the benefit of integrating or adding ESTs to standard care. While retaining key design features such as random assignment to treatment conditions, objective assessment of outcome by blind evaluators, assessment of treatment integrity and fidelity, CTN trials are addressing the question: ‘do ESTs work in the real world?' by conducting these trials in community-based clinics (rather than research or university settings), broadening inclusion/exclusion criteria so that samples are as heterogeneous as possible (e.g. ‘taking all comers' rather than treating highly selected samples), with delivery of study treatments by clinicians drawn from the staff of the clinics (as opposed to doctoral level clinicians selected for their skill or allegiance to a particular model) (40). To date, the results of completed CTN trials suggest that, even in the context of high variability across sites, programs, clinicians and participants, ESTs such as contingency management (CM) can improve outcomes when added to standard treatment for stimulant dependence (41, 42), motivational interviewing (MI) can improve retention when integrated into the early phase of out-patient substance abuse treatment (43, 44) and detoxification using buprenorphine is a more effective detoxification strategy than is clonidine (45). Moreover, the CTN trials providing extensive ‘hands-on' experience in using novel therapies to a large number of clinics, as well as offering training and supervision in ESTs to a broad range of ‘real-world' clinicians (46). Data from these trials also suggest that clinicians drawn from the staff of CTN clinics can learn to implement ESTs such as MI, when provided with didactic training and ongoing monitoring of treatment fidelity via regular supervision based on review of session tapes. What is not yet clear is the extent to which trained clinicians will retain their new skills and continue to use them in clinical practice once the studies are completed and their performance is no longer monitored.

Another area largely unaddressed by existing efficacy study is the question: ‘what should happen when a patient doesn't respond to an EST?'. Although there are now a number of viable alternative options for treating smoking (including buproprion, nicotine replacement and behavioral strategies), opioid dependence (including naltrexone, buprenorphine and methadone) and alcohol dependence (naltrexone, disulfiram and many behavioral approaches), there are few empirically based guidelines regarding, for example, strategies for opioid-dependent individuals who do not respond to methadone maintenance, cocaine users who do not respond to clinical management or smokers who do not respond to buproprion. Adaptive treatment designs, which evaluate systematically the relative benefit of augmenting, discontinuing or switching treatment strategies for individuals who do not respond to, or who relapse following, treatment are a potent strategy for addressing these very practical issues regarding clinicians' response to poor patient outcomes in response to ESTs (29, 47). Large-scale efforts to develop treatment guidelines, adaptive treatment strategies and algorithms has begun in areas where multiple effective treatment alternatives exist, such as depression (4850), but have only recently been applied to the addictions (51, 52).

IT ‘IS BETTER TO KNOW SOME OF THE QUESTIONS THAN ALL OF THE ANSWERS' (JAMES THURBER)

The issues discussed in the section above suggest that immediate deployment of a massive army of trainers in the 100 or more ESTs to all clinicians world-wide will not be feasible, helpful or particularly effective in improving the quality of addictions treatment. Even if we assume that the integration of ESTs will improve standard care, there are remarkably few data on such fundamental questions as (1) ‘which of the available ESTs are likely to foster the greatest actual improvements in practice?' and (2) ‘how best to train clinicians and to sustain the effects of training?'. Designation of a therapy as an ‘EST' does not automatically also assure it of the designation ‘Easily Sustainable Treatment': issues such as cost, ease of trainability, acceptability and attractiveness to patients can easily derail even a very powerful treatment from being used in clinical practice.

The Stage Model of Behavioral Therapies Research, developed by the National Institute on Drug Abuse in the United States, anticipated many of these issues and offers a heuristic model for thinking through research strategies and questions regarding how the availability of ESTs should alter clinical practice. The model articulates three progressive stages in the development and evaluation of behavioral therapies that roughly parallel those for the development of pharmacologic therapies (53). Stage I consists of initial development of and pilot/feasibility testing for new and untested treatments (54). Stage II consists principally of randomized controlled clinical trials to evaluate the efficacy of treatments that have shown promise or efficacy in initial studies (i.e. essentially the process that has produced ESTs and generated the criteria by which they are judged). Stage III is a more novel area that is intended to address issues of transportability into clinical practice of treatments whose efficacy has been demonstrated in prior Stage II efficacy trials. Although the methodology of Stage III research is still developing (55), we have argued it should address a range of research questions, including: (1) efficacy and generalizability (i.e. ‘will this treatment retain its efficacy when implemented by different practitioners in different settings, and with a broader range of patients?', ‘is Treatment X truly more effective than current practice?'); (2) implementation issues (i.e. ‘what kinds of training, by what kinds of trainers, are necessary for what kinds of clinicians to learn a new technique?'); (3) cost-effectiveness issues (i.e. ‘compared with the costs of learning and implementing this treatment, what are the savings, particularly in comparison to the existing standard of care?'); and (4) consumer/marketing issues (i.e. ‘how acceptable is a new treatment to both clinicians, patients, and payors outside of research settings?'). Because we have discussed previously issues related to cost-effectiveness and consumer/marketing issues for addictions treatments (40), in the sections below we will focus primarily on issues of generalizability of ESTs and implementation/training issues.

EFFICACY AND GENERALIZABILITY ISSUES

Given the lack of clear superiority for any single EST and the sheer impracticability of all clinics offering all available ESTs, a reasonable start would be to identify a reasonable ‘first'-line therapy that can be offered to the majority of individuals entering treatment for a given primary disorder. The current default first-line therapy in the United States (if immediate detoxification is not needed and no pharmacological alternative exists) is group drug abuse counseling, usually with referral to self-help such as Alcoholics Anonymous meetings. This practice has largely been determined on the basis of cost and convenience considerations, as there is little research on the effectiveness of ‘group counseling' (although we might extrapolate a relatively weak effect because unstructured group counseling is a frequently used control or comparison condition against ESTs that have been compared) (56). Hence, identification of a reasonable ‘first-line' treatment that is inexpensive, broadly applicable to a range of patients and settings, and has some data supporting its effectiveness is needed. Possible candidates for an empirically supported first-line approach for out-patient addictions treatment include MI, cognitive behavioral therapy (CBT) or combined MI/CBT, the last of which has been demonstrated to be effective in several substance using samples (5759) and may have cost and feasibility advantages relative to contingency management (60) or family/couples therapy. Following an initial trial of the first-line approach, objective benchmarking outcomes (retention, urine toxicology screen results) could be monitored regularly to determine whether more intensive and costly interventions are indicated (e.g. pharmacotherapy or contingency management). The choice of an appropriate first-, second- or third-line EST could also be determined empirically, using the adaptive treatment strategy approach described above.

TRAINING ISSUES

The dominant current strategy for training clinicians in the United States in ESTs is through brief workshops offered through federally supported addiction technology transfer centers (ATTCs) or single-state agencies, typically with continuing education credits offered as an incentive. The system thus capitalizes on self-selection, as clinicians are free to attend as many (or as few) EST workshops as they wish and there is no system in place for assuring minimal standards of competence for virtually any ESTs (e.g. through formal certification processes). Thus, there is increasing pressure on agencies and clinics to state that they offer ESTs and some incentives for clinicians to be exposed to ESTs, but no system or standards for ensuring that empirically supported behavioral therapies are delivered with even minimal levels of adherence or competence.

Broader use of any ESTs in clinical practice will inevitably raise a number of issues relative to the training of clinicians. First, who should be trained in ESTs (students, current practitioners in specialty programs, clinicians in private practice)? Students in social work, psychology, psychiatry and various addiction training programs who are just entering the field are an important target for training efforts, but systematic training in ESTs remains rare and uneven even in these programs (61, 62). Clinicians already working in specialty substance abuse treatment programs are a crucial target of EST training efforts. Although many clinicians claim extensive use of ESTs in clinical practice, independent review of audiotapes of ‘standard treatment' in sites participating in a recent CTN protocol indicated very infrequent use of any interventions associated with ESTs, including skills training, outcome planning or referral to self-help. Finally, although students/trainees and clinicians working in specialty programs are logical and reasonably accessible target audiences for training efforts, it should also be recognized that a very large number of clinicians who deliver services to substance users do not practice in specialty clinics or with highly specialized populations where the number of ESTs might be manageable.

Next, how much and what kind of training will be required for the various types of clinicians? The standard training approach of brief workshops has been shown to be of limited effectiveness in imparting key skills and competence to experienced clinicians and appears, at best, to provide exposure to (rather than mastery of), novel approaches (21, 6366). For example, Miller and colleagues demonstrated that workshop training was insufficient to impart key MI skills to experienced drug abuse clinicians (67, 68). In a similar study that randomized experienced clinicians to different methods for learning CBT (69), only the training condition that offered intensive didactic training plus performance-based supervision and feedback enabled the clinicians to reach adherence and competence standards that would be required for a CBT efficacy trial. However, as discussed in more detail below, this type of training is expensive, time-consuming and the absence for several days of clinical staff can be extremely disruptive to small clinics. Thus, research on the effectiveness of alternative training models is also needed, such as ‘train the trainer' models, where one key clinical leader from a program staff receives elaborate instruction, not only in a specific EST but also in how to train other clinicians in the approach and how to monitor and supervise it as well. Ongoing research by Steve Martino in our group is directly contrasting the effectiveness of a ‘train the trainer' approach to direct training of all staff by a single expert trainer.

Can all clinicians be trained? It is not yet clear that all clinicians, particularly those strongly wedded to a particular clinical approach, can attain competence in all clinical approaches, and the available anecdotal data suggests that efforts to train ‘all corners' to competence may not be successful. Those training studies that have included a wide range of clinician/volunteers to be trained in a particular EST have noted substantial variability in clinician competence, even after extensive training and feedback (6972). Our data also suggest that a number of clinicians who had only ‘on-the-job' training prior to participating in the training study lacked basic clinical skills, and it is not clear how these basic skills should be taught, or even whether they can be taught (69, 73). Similarly, previous multi-site trials of behavioral treatments have indicated that a substantial proportion of clinicians who begin protocol training either do not meet competence criteria and are not certified to participate in the trial, or leave training through a self-selection process (70, 72, 7476). It is also not clear whether most clinicians can achieve competence in multiple ESTs and, even if they did so, whether they could maintain that competence in clinical practice. Thus, it may be a reasonable starting-point to simply require that clinicians demonstrate competence in a minimum of one EST, with reasonable exposure to other, newly emerging therapies as needed and as dictated by the clinical populations with whom they work. This strategy would at least facilitate broader availability of ESTs to a wider proportion of individuals with substance use problems.

How can clinicians' competence in ESTs be maintained? The emerging data from training studies suggest that review, feedback, and coaching of therapists implementation of ESTs is crucial in developing competence in behavioral therapies (7780). It is unlikely that competence could be maintained without some ongoing monitoring of a clinician's implementation of a given EST, with ongoing provision of supervision and support from clinicians who have training and experience with the approach (66, 8083). However, clinical supervision based on objective standards or systems is virtually nonexistent in the United States, and is also not a compensated or reimbursable service. In fact, in our two studies of clinician training methods, the clinicians reported that they received an average of 1 hour of clinical supervision each week, and this was typically in the context of a staff meeting in which clinicians' case-loads and the status of their charting and paperwork was reviewed. Moreover, given the high rate of turnover of clinicians in substance abuse treatment programs (84, 85), extensive training, certification and supervision procedures may not be seen as cost-effective. Thus, in order to initiate and then sustain delivery of ESTs in practice, it will be important to demonstrate not only that ESTs result in measurably improved outcomes when added or integrated into standard clinical programs, but also that the provision of training and supervision in ESTs reduces staff turnover or reduces other costs. Hence, it will be essential that cost analyses be integrated into such trials in order to generate the data needed to persuade policy makers, third-party payors and program managers that adoption of ESTs is warranted (11, 8688) as well as to counter financial and programmatic disincentives for adoption of ESTs (38, 89).

‘DIVIDE AND CONQUER, A SOUND MOTTO. UNITE AND LEAD, A BETTER ONE' (GOETHE)

Given the proliferation of ESTs, what ESTs should be emphasized in dissemination efforts? Some selection and prioritization will be required, as it is clearly unreasonable to demand that clinicians be familiar with, much less competent in, the more than 100 available ESTs. Similarly, it is unlikely that most training programs could have the capacity to offer training in all available treatments for all disorders. Hence, there is a need for some way to either combine similar approaches/strategies or to select among the ESTs, particularly at the level of training programs.

Regarding the former strategy, a number of investigators have called for focus upon and teaching of basic principles and strategies, rather than focus on ‘name brand' therapies (13, 15, 17, 90, 91). This approach emphasizes understanding of the principles and mechanisms of change inherent in effective treatments (9294). A definitive list of change principles, although still elusive, would presumably be brief and arguably easier to use as a basis for training clinicians (13, 95). An example would be Beutler's approach of using more directive techniques for externalizing disorders and more exploratory techniques for internalizing disorders (96, 97).

In the addictions, while there are multiple variants and population-specific versions of many different treatment types, many of these fall under the broader categories with general principles/strategies such as cognitive-behavioral, behavioral/contingency management, interpersonal/psychodynamic and family approaches (1). Trainees could first achieve general familiarity or basic competence with the broad principles and strategies of these core approaches and then subsequently master specific ESTs on the basis of need or interest.

This is generally the approach used currently, where trainees are exposed to a few basic skills in their formal didactic training, and then learn specific techniques in their clinical practica (61). The available evidence also suggests that this approach is resulting in very little actual use of ESTs in practice. For example, in our CBT training study, although the clinicians claimed comparatively high levels of familiarity with CBT, their pretraining CBT adherence/competence ratings were strikingly low (69). Similarly, in the CTN study of MI, a self-report survey of the clinicians delivering the standard treatment condition prior to the training indicated that the therapists reported regular use of techniques associated with 12-Step/disease model concepts, CBT and MI (98). However, ratings based on session tapes indicated interventions such as referral to self-help meetings, skills training, use of assessment/feedback were almost nonexistent, as were use of basic counseling strategies such as program orientation, goal setting and treatment planning.

This approach also raises the issue of the robustness of ESTs; that is, the extent to which ESTs remain effective when practiced imprecisely or when used with clients who differ (in terms of psychiatric comorbidity, multiple drug use or complexity and range of co-occurring problems) from those who participated in the original efficacy studies (99, 100). For example, CM has been shown to be a particularly effective approach across a range of substance use disorders when delivered consistently, at adequately high reinforcer magnitude and when focusing on a single drug or behavior (2, 101). However, CM has also been shown to be highly sensitive to relatively small changes in reinforcement magnitude or reinforcement schedules (2, 101, 102). Thus, because CM is more effective when applied to a single type of drug use rather than multiple substances or target behaviors, and large proportions of clinical populations use multiple substances, CM may be less effective in the clinical situations where it is most needed.

It is also unclear whether some of the major classes of psychotherapy are highly interchangeable in their applications to different problem areas. For example, CBT approaches, grounded in similar theoretical models of the relationships between behavior, affect and cognition, have also been shown to be effective in populations of individuals with depressive and anxiety disorders (6, 8, 103). However, in practice, CBT models for substance use, depression and anxiety sharply differ from one another with regard to their relative emphasis on behavior versus cognition, use of specific strategies and exercises and types of extra-session practice exercises. Hence, it cannot be assumed that a clinician who is highly accomplished in CBT for depression can easily ‘pick up' CBT for addiction without substantial training and supervised practice. In contrast, interpersonal (104) or motivational interviewing (105, 106) approaches appear to afford considerable leeway and may not require extensive retraining to be applied to different types of individuals and disorders. A related question is the degree to which high levels of adherence and competence is associated with measurable differences in outcome: while therapist skill and adherence to manual guidelines have been linked to outcomes for several treatments (107109), in other areas the findings have been more mixed or even negative (110, 111).

Regarding the approach of selecting and training in only a few ESTs, the ‘divide and conquer' strategy might, for example, require training programs to focus on fewer ESTs, or require that trainees master and demonstrate only two or three specific strategies. These could be selected on the basis of ‘first-line' approaches for two or three common disorders or, for specialty training programs, two or three ESTs within a specific class of disorders. This strategy allows less breadth in training but would have the advantage of facilitating clear standards for establishing competence and certification; in effect, producing clinicians who are very good at a few therapies rather than who are only moderately effective (or worse) in many.

For experienced therapists receiving training in ESTs within either specialty training or treatment programs (and thus have more familiarity with basic principles of change), offering specialty training in single ESTs may be a reasonable approach. We have used the strategy of training experienced clinicians in single ESTs with clinicians recruited in two ways: (1) preselected, experienced clinicians volunteering for efficacy studies and (2) clinicians serving in community programs chosen to participate in effectiveness studies. For example, we have conducted a series of studies evaluating different strategies for training clinician/volunteers in CBT, MI, CM or 12-Step facilitation (TSF). In these trials, clinicians working full-time in substance abuse treatment facilities are randomized to one of several training conditions. In our initial study of CBT training methods, 78 clinicians were assigned to one of three training conditions: review of the CBT manual only, access to a web-based training site (which included additional frequently asked questions and practice exercises) plus the manual, or a 3-day didactic seminar plus up to three sessions of supervision from a CBT expert trainer based on actual session tapes submitted by the participants. Outcomes included (1) between-group comparisons of the clinicians' ability to demonstrate key CBT techniques based on structured role-plays administered before and after training, and (2) scores on a CBT knowledge quiz. The videotaped role plays were scored by independent raters, blind to the participants' training condition as well as time (e.g. pre- versus post-training) and on adherence/competence ratings of specific CBT techniques from the Yale Adherence and Competence Scale (YACs) (81). Although all groups demonstrated improved adherence and competence scores over time, the only training condition that reached levels of skill consistent with those required of clinicians participating in our CBT efficacy trials was the seminar + supervision condition, with intermediate ratings for the web condition. The mean effect size for the seminar + supervision versus manual-only condition comparisons was consistent with a large effect (0.69), while the average effect size for the web versus manual only condition contrasts was consistent with a medium-sized effect (0.30). In addition, the seminar + supervision condition was associated with significantly more clinicians reaching criterion levels for adequate fidelity than those assigned to the manual-only condition (54% versus 15%). Our subsequent training studies, which have evaluated different training strategies for CM, MI and TSF (112), have also used the YACs as a training, supervision and outcome tool to foster comparison across studies and provide an objective means of benchmarking therapist performance.

A notable finding from our series of training trials is the surprisingly positive performance of the computer-based training conditions (69, 112). Given the many dissemination challenges noted in this paper, the recent availability of distance learning methods, particularly computer and web-based training, is an exciting new model for training clinicians (113119) and may present a feasible and less costly strategy for training larger numbers of substance use clinicians than is possible through standard, face-to-face training strategies. Given the time constraints and variable educational background of substance use clinicians, computer-based training may offer a number of potential advantages, including flexibility in scheduling, allowing more clinicians to access training, increasing opportunities for practice, as well as increased flexibility and individualization in pace and material covered (120). Moreover, the rate of staff turnover in many community treatment programs is quite high, with estimates ranging from 16% to 50% per year (22, 121, 122; P, Roman, personal communication, 2002); this underlines the need for more rapid, flexible and inexpensive methods of training such as computer-assisted models,

Although comparatively few well-designed randomized trials comparing ‘live' teaching versus computer-assisted training have been conducted, those that have been done suggested that computer-assisted training can (1) be as or more effective than traditional lecture format training and (2) improve skills as well as knowledge (123, 124). It should be noted that computer-based training might have potential disadvantages as well. Computers are still not universally available in treatment or training programs in the United States (27), and clinicians who are not familiar with computers may find it frustrating to use programs that are not highly userfriendly. The extent to which computer-assisted training programs can convey basic clinical skills such as the ability to form an alliance or maintain session structure is not clear (although it is not yet clear whether traditional face-to-face training can do this, either) (120, 125127). Thus, it is critical that computer-based training programs be carefully evaluated, with a range of clinicians and in comparison to standard methods of training, before they are widely disseminated.

As a second example of the ‘divide and conquer' strategy, in the CTN studies (43, 44) we have instituted a training model intended to foster both competence in specific ESTs for clinicians participating in these effectiveness studies as well as persistence of the approach in the clinic after the trials are completed. In this model, expert trainers provide training and supervision to two types of clinicians. First, based on training strategies used in previous multi-site trials (70, 71, 76), experienced clinicians working at the clinics are provided with roughly 2 days of training, which includes detailed review of the training manual plus extensive role-play exercises and feedback; this is followed by close supervision on video- or audio-taped training cases until pre-established standards of fidelity and competence are met for certification (77). However, to foster greater durability of the approach in the clinic, the expert trainers also provide more intensive training to at least one clinician in a clinical leadership role at the site. The clinician/supervisor is trained to competence standards in the treatment but also receives additional instruction in strategies and standards for supervision of the EST and then also plays a role in monitoring adherence/competence of the site clinicians as the trial goes on. Thus, after the trial is completed, these clinical supervisors will have accrued sufficient experience to enable them to provide ongoing training and supervision in the approach to other clinicians at that setting (46).

‘THE FUTURE WILL BE BETTER TOMORROW' (DAN QUAYLE, US VICE PRESIDENT)

In the sections above we have tried to highlight some of the complexities, challenges and unanswered questions regarding ESTs into standard clinical practice. Given the large number of issues related to training clinicians, the high relative cost of ESTs compared to the existing standard care models, the high levels of variability in delivery and outcome of ESTs across clinicians (128130), and the barriers and costs of introducing performance-based supervision into standard clinical practice, a more novel alternative strategy would be to deliver treatments directly to patients via computer-assisted training. Computer-assisted therapies offer a number of potential advantages, including offering more therapeutic time to patients, ease of time scheduling (as patients can access computer programs outside of appointment times or regular clinic hours), allowing more patients to access treatment, providing greater confidentiality, increasing opportunities for practice and fostering patients' sense of mastery and control (131, 132). Particularly in cases where clinician training is difficult or time-consuming, computer-based therapies may make effective treatments more accessible to a much broader range of treatment than the current system is able to reach (133).

Computer-assisted therapies potentially also offer more consistent delivery of interventions to patients, particularly for comparatively complex approaches such as CBT where clinician fidelity and skill in implementing the treatment tends to be variable. Computer-assisted training, as an adjunct to standard treatment, offers a number of attractive features: these include offering more opportunities for patients to access treatment at low cost, relieving clinicians of some repetitive tasks, and offering an alternative method of skills training for overburdened clinicians who can rarely offer individualized time to patients.

Computer-assisted treatment, although still in comparatively early stages of development, has shown promise in several areas of mental health treatment, including simple phobias (134), obsessive-compulsive disorder (135), panic disorder (136), depression (137, 138), attention deficit hyperactive disorder (ADHD) (139) and smoking (140). We are currently piloting a computer-based version of CBT (‘CBT4CBT'), a user-friendly, multimedia version of our CBT manual (141). The program includes six modules covering key basic CBT concepts (functional analyses, coping with craving, monitoring and changing cognitions), with multiple movie-like vignettes of realistic characters implementing these skills to change their drug use. Thus, patients can actually see examples of use of coping skills and implementation of CBT principles, rather than simply hearing a clinicians' abstract oral presentation of those principles. It also includes interactive graphics and narration teaching of basic skills, interactive exercises, vignettes of effective versus ineffective use of skills, and detailed examples of practice exercises. We are currently piloting this program in a community-based treatment center to evaluate (1) its feasibility and acceptability to patients and clinicians and (2) its effectiveness as an addition to standard treatment. Clearly, computer-assisted treatment is not a total substitute for a live clinician and unlikely to be applicable to severely impaired patients or to longer-term, more exploratory therapies. However, given the efficiency and low cost of these approaches, there may be less need to show that computerized ESTs are superior to treatment as usual, as they are bound to be less costly than therapist-delivered treatments, even if they are equivalent or slightly inferior to standard care approaches.

CONCLUSION: ‘COUNT NO MAN AS HAPPY UNTIL HE IS LOWERED INTO THE GRAVE' (SCHOPENHAUER)

Whither ESTs? Like freshly minted PhD recipients holding their diplomas, proud developers of ESTs for the addictions have passed one milestone only to face a series of other challenges to overcome before their accomplishments pay off. Unlike Food and Drug Administration (FDA)-approved medications that can be manufactured in bulk and delivered in pure form anywhere in the world, new behavioral treatments can be disseminated only through training therapist after therapist, with the hope that they will remain in practice and stay motivated to deliver the treatment. Before (or while) ESTs are put into mass production, several new lines of research are needed to address important questions about the value of making ESTs the standard of care and about the best ways to disseminate them. First and most fundamentally, more research is needed that verifies the superior efficacy and cost-effectiveness of ESTs over standard practice. Second, when multiple ESTs are available, studies with adaptive designs are needed to guide the choice of frontline treatments and to identify optimal sequences of treatments to deliver when initial results are poor. Third, a new subfield of ‘training effectiveness' research is needed to address a series of questions about treatment dissemination that parallel those articulated by Paul (142, 143) about behavioral therapies efficacy research: that is, given substantial cross-treatment differences in complexity and difficulty of delivery, therapist training research needs to answer: ‘which training is needed in which therapy for which trainee to treat which type of patient?'. Fourth, while behavioral treatments resist formulation and delivery with the precision of FDA-approved medications, computer-assisted therapist training and computer-delivered treatment hold considerable promise for bringing the efficiency and precision of EST dissemination to new levels.

Support was provided by NIDA grants K05-DA00457, K05-DA00089, P50-DA09241, and the US Department of Veterans Affairs VISN 1 Mental Illness Research, Education and Clinical Center (MIRECC).

REFERENCES

1 Carroll K. M., Onken L. S. Behavioral therapies for drug abuse. Am J Psychiatry 2005; 162:1452–60.CrossrefGoogle Scholar

2 Griffith J. D., Rowan-Szal G. A., Roark R. R., Simpson D. D. Contingency management in outpatient methadone treatment: a meta-analysis. Drug Alcohol Depend 2000; 58:55–66.CrossrefGoogle Scholar

3 Stanton M. D., Shadish W. R. Outcome, attrition, and family-couples treatment for drug abuse: a meta-analysis and review of the controlled, comparative studies. Psychol Bull 1997; 122:170–91.CrossrefGoogle Scholar

4 Burke B. L., Arkowitz H., Menchola M. The efficacy of motivational interviewing: a meta-analysis of controlled clinical trials. J Consult Clin Psychol 2003; 71:843–61.CrossrefGoogle Scholar

5 Irvin J. E., Bowers C. A., Dunn M. E., Wong M. C. Efficacy of relapse prevention: a meta-analytic review. J Consult Clin Psychol 1999; 67:563–70.CrossrefGoogle Scholar

6 Nathan P. E., Gorman J. M. A Guide to Treatments That Work, 2nd edn. New York: Oxford University Press; 2002.Google Scholar

7 Chambless D. L., Ollendick T. H. Empirically supported psychological interventions: controversies and evidence. Annu Rev Psychol 2001; 52:685–716.CrossrefGoogle Scholar

8 Roth A., Fonagy P. What Works for Whom? a Critical Review of the Psychotherapy Literature, 2nd edn. New York: Guilford Press; 2005.Google Scholar

9 Sackett D. L., Richardson W. S., Rosenberg W., Haynes R. B. Evidence-Based Medicine. New York: Churchill Livingstone; 1997.Google Scholar

10 Chambless D. L., Hollon S. D. Defining empirically supported therapies. J Consult Clin Psychol 1998; 66:7–18.CrossrefGoogle Scholar

11 Beutler L. E. David and Goliath: when empirical and clinical standards of practice meet. Am Psychol 2000; 55:997–1007.CrossrefGoogle Scholar

12 Sanderson W. C. Why empirically supported psychological treatments are important. Behav Modif 2003; 27:290–9.CrossrefGoogle Scholar

13 Follette W. C., Beitz K. Adding a more rigorous scientific agenda to the empirically supported treatment movement. Behav Modif 2003; 27:369–86.CrossrefGoogle Scholar

14 Rosen G. M., Davidson G. C. Psychology should list empirically suported principles of change (ESPs) and not credential trademarked therapies or other treatment packages. Behav Modif 2003; 27:300–12.CrossrefGoogle Scholar

15 Westen D., Novotney C. M., Thompson-Brenner H. The empirical status of empirically supported psychotherapies: assumptions, findings, and reporting in controlled clinical trials. Psychol Bull 2004; 130:631–63.CrossrefGoogle Scholar

16 Garfield S. L. Some problems associated with ‘validated’ forms of therapy. Clin Psychol Sci Pract 1996; 3:218–29.CrossrefGoogle Scholar

17 Wampold B. E., Bhati K. S. Attending to the omissions: a historical examination of evidence-based practice movements. Profess Psychol Res Pract 2004; 35:563–70.CrossrefGoogle Scholar

18 Herbert J. D. The science and practice of empirically supported treatments. Behav Modif 2003; 27:412–30.CrossrefGoogle Scholar

19 Institute of Medicine. Bridging the Gap Between Practice and Research: Forging Partnerships with Community-Based Drug and Alcohol Treatment. Washington, DC: National Academy Press; 1998.Google Scholar

20 Institute of Medicine. Crossing the Quality Chasm: a New Mental Health System for the 21st Century. Washington, DC: National Academy Press; 2001.Google Scholar

21 Fals-Stewart W., Logsdon T., Birchler G. R. Diffusion of an empirically supported treatment for substance abuse: an organizational autopsy of technology transfer success and failure. Clin Psychol Sci Pract 2004; 11:177–82.CrossrefGoogle Scholar

22 McLellan A. T., McKay J. R. The treatment of addiction: what can research offer practice? In: Lamb S., Greenlick M. R., McCarty D., editors. Bridging the gap between practice and research: forging partnerships with community based drug and alcohol treatment, Washington, DC: National Academy Press; 1998. p. 147–85.Google Scholar

23 Margolin A., Kleber H. D., Avants S. K. et al. Acupuncture for the treatment of cocaine addiction: a randomized controlled trial. JAMA 2002; 287:55–63.CrossrefGoogle Scholar

24 Dishion T. J., McCord J., Poulin F. When interventions harm: peer groups and problem behavior. Am Psychol 1999; 54:755–64.CrossrefGoogle Scholar

25 Mattick R. P., Breen C., Kimber J., Davoli M. Methadone maintenance therapy versus no opioid replacement therapy for opioid dependence. Cochrane Database Syst Rev 2003; 2: CD002209.Google Scholar

26 Masson C. L., Barnett P. G., Sees K. L., Delucchi K. L., Rosen A., Wong W. et al. Cost and cost-effectiveness of standard methadone maintenance treatment compared to enriched 180-day methadone detoxification. Addiction 2004; 99:718–26.CrossrefGoogle Scholar

27 McLellan A. T., Carise D., Kleber H. D. Can the national addiction treatment infrastructure support the public's demand for quality care? J Subst Abuse Treat 2003; 25:117–21.CrossrefGoogle Scholar

28 Babor T. F., Higgins-Biddle J. C. Alcohol screening and brief intervention: dissemination strategies for medical practice and public health. Addiction 2000; 95:677–86.CrossrefGoogle Scholar

29 Lambert M. E., Whipple J. L., Vermeersch D. A., Nielsen S. L., Smart D. W. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clin Psychol Sci Pract 2003; 10:288–301.CrossrefGoogle Scholar

30 Luborsky L., Rosenthal R., Diguer L., Andrusyna T. P., Berman J. S., Levitt J. T. et al. The dodo bird verdict is alive and well—mostly. Clin Psychol Sci Pract 2002; 9:2–12.CrossrefGoogle Scholar

31 Luborsky L., Singer B., Luborsky L. Comparative studies of psychotherapies: is it true that ‘Everyone has won and all must have prizes’? Arch Gen Psychiatry 1975; 32:995–1008.CrossrefGoogle Scholar

32 Project MATCH Research Group. Matching alcohol treatments to client heterogeneity: Project MATCH posttreatment drinking outcomes. J Stud Alcohol 1997; 58:7–29.CrossrefGoogle Scholar

33 UK Alcohol Treatment Trial (UKATT) Research Team. Effectiveness of treatment for alcohol problems: findings of the randomised UK alcohol treatment trial (UKATT). BMJ 2005; 331: 541.CrossrefGoogle Scholar

34 Morgenstern J., Blanchard K. A., Morgan T. J., Labouvie E., Hayaki J. Testing the effectiveness of cognitive-behavioral treatment for substance abuse in a community setting: within treatment and posttreatment findings. J Consult Clin Psychol 2001; 69:1007–17.CrossrefGoogle Scholar

35 Addis M. E., Waltz J. Implicit and untested assumptions aout the role of psychotherapy treatment manuals in evidence-based mental health practice. Clin Psychol Sci Pract 2002; 9:421–4.CrossrefGoogle Scholar

36 Shadish W. R., Matt G. E., Navarro A. M., Siegle G., Crits-Christoph P., Hazelrigg M. D. et al. Evidence that therapy works in clinically representative conditions. J Consult Clin Psychol 1997; 65:355–65.CrossrefGoogle Scholar

37 Weisz J. R., Jensen-Doss A., Hawley K. M. Evidence-based youth psychotherapies versus usual clinical care. Am Psychol 2006; 61:671–89.CrossrefGoogle Scholar

38 Addis M. E. Methods for disseminating research products and increasing evidence-based practice: promises, obstacles, and future directions. Clin Psychol Sci Pract 2002; 9:367–78.CrossrefGoogle Scholar

39 Hanson G. R., Leshner A. I., Tai B. Putting drug abuse research to work in real-life settings. J Subst Abuse Treat 2002; 23:69–70.CrossrefGoogle Scholar

40 Carroll K. M., Rounsaville B. J. Bridging the gap between research and practice in substance abuse treatment: a hybrid model linking efficacy and effectiveness research. Psychiatr Serv 2003; 54:333–9.CrossrefGoogle Scholar

41 Peirce J. M., Petry N. M., Stitzer M. L., Blaine J. D., Kellog S., Satterfield F. et al. Effects of lower-cost incentives on stimulant abstinence in methadone maintenance treatment: a National Drug Abuse Treatment Clinical Trials Network study. Arch Gen Psychiatry 2006; 63:201–8.CrossrefGoogle Scholar

42 Petry N. M., Peirce J. M., Stitzer M. L., Blaine J. D., Roll J. M., Cohen A. et al. Effect of prize-based incentives on outcomes in stimulant abusers in outpatient psychosocial treatment programs: a national drug abuse Clinical Trials Network study. Arch Gen Psychiatry 2005; 62:1148–56.CrossrefGoogle Scholar

43 Carroll K. M., Ball S. A., Nich C., Martino S., Frankforter T., Farentinos C. et al. Motivational interviewing to improve treatment engagement and outcome in individuals seeking treatment for substance abuse: a multisite effectiveness study. Drug Alcohol Depend 2006; 81:301–12.CrossrefGoogle Scholar

44 Ball S. A., Martino S., Nich C., Frankforter T. L., Van Horn D. H., Crits-Christoph P. et al. Site matters: motivational enhancement therapy in community drug abuse clinics. J Consult Clin Psychol; in press.Google Scholar

45 Ling W., Amass L., Shoptaw S., Annon J. J., Hillhouse M., Babcock D. et al. 2005 A multicenter randomized trial of buprenorphine-naloxone versus clonidine for opioid detoxification: findings from the National Institute on Drug Abuse Clinical Trials Network. Addiction 2006; 100:1090–100.CrossrefGoogle Scholar

46 Carroll K. M., Farentinos C., Ball S. A., Crits-Christoph P., Libby B., Morgenstern J. et al. MET meets the real world: design issues and clinical strategies in the Clinical Trials Network. J Subst Abuse Treat 2002; 23:73–80.CrossrefGoogle Scholar

47 Collins L. M., Murphy S. A., Bierman K. L. A conceptual framework for adaptive preventive interventions. Prev Sci 2004; 5:185–96.CrossrefGoogle Scholar

48 Rush A. J., Trivedi M. H., Wisniewski S. R., Stewart J. W., Nierenberg A. A., Thase M. E. et al. Bupropion-SR, sertraline, or venlafaxine-XR after failure of SSRIs for depression. N Engl J Med 2006; 354:1231–42.CrossrefGoogle Scholar

49 Rush A. J., Fava M., Wisniewski S. R., Lavori P. W., Trivedi M. H., Sackeim H. A. et al. Sequenced treatment alternatives to relieve depression (STAR*D): rationale and design. Control Clin Trials 2004; 25:119–42.CrossrefGoogle Scholar

50 Trivedi M. H., Fava M., Wisniewski S. R., Thase M. E., Quitkin F., Warden D. et al. Medication augmentation after the failure of SSRIs for depression. N Engl J Med 2006; 354:1243–52.CrossrefGoogle Scholar

51 Murphy S. A., Lynch K. G., Oslin D., McKay J. R., Tenhave T. Developing adaptive treatment strategies in substance abuse research. Drug Alcohol Depend 2007; 88: S24–S39 (Suppl 2).CrossrefGoogle Scholar

52 O'Malley S. S., Rounsaville B. J., Farren C., Namkoong K., Wu R., Robinson J. et al. Initial and maintenance naltrexone treatment for alcohol dependence using primary care vs. specialty care: a nested sequence of three randomized trials. Arch Intern Med 2003; 163:1695–704.CrossrefGoogle Scholar

53 Onken L. S., Blaine J. D., Battjes R. Behavioral therapy research: a conceptualization of a process. In: Hennegler S. W., Amentos R., editors. Innovative approaches for difficult to treat populations. Washington, DC: American Psychiatric Press; 1997, p. 477–85.Google Scholar

54 Rounsaville B. J., Carroll K. M., Onken L. S. A stage model of behavioral therapies research: getting started and moving on from Stage I. Clin Psychol Sci Pract 2001; 8:133–42.CrossrefGoogle Scholar

55 Hohmann A. A., Shear M. K. Community-based intervention research: coping with the ‘noise’ of real life in study design. Am J Psychiatry 2002; 159:202–7.CrossrefGoogle Scholar

56 Miller W. R., Wilbourne P. L. Mesa Grande: a methodological analysis of clinical trials of treatments for alcohol use disorders. Addiction 2002; 97:265–77.CrossrefGoogle Scholar

57 MTP Research Group. Brief treatments for cannabis dependence: findings from a randomized multisite trial. J Consult Clin Psychol 2004; 72:455–66.CrossrefGoogle Scholar

58 Carroll K. M., Easton C. J., Nich C., Hunkele K. A., Neavins T. M., Sinha R. et al. The use of contingency management and motivational/skills-building therapy to treat young adults with marijuana dependence. J Consult Clin Psychol 2006; 74:955–66.CrossrefGoogle Scholar

59 Anton R. F., O'Malley S. S., Ciraulo D. A., Cisler R. A., Couper D., Donovan D. M. et al. Combined pharmacotherapies and behavioral interventions for alcohol dependence: the COMBINE study: a randomized controlled trial. JAMA 2006; 295:2003–17.CrossrefGoogle Scholar

60 Olmstead T., Sindelar J. L., Easton C. J., Carroll K. M. The cost-effectiveness of four treatments for marijuana dependence. Addiction; in press.Google Scholar

61 Weissman M. M., Verdeli H., Gameroff M. J., Bledsoe S. E., Betts K., Mufson L. et al. National survey of psychotherapy training in psychiatry, psychology, and social work. Arch Gen Psychiatry 2006; 63:925–34.CrossrefGoogle Scholar

62 Crits-Christoph P., Frank E., Chambless D. L., Brody C., Karp J. F. Training in empirically validated therapies: what are clinical psychology students learning? Profess Psychol Res Pract 1995; 26:514–22.CrossrefGoogle Scholar

63 Davis D. A., Thomson M. A., Oxman A. D., Haynes R. B. Evidence for the effectiveness of CME: a review of 50 randomized controlled trials. JAMA 1992; 268:1111–17.CrossrefGoogle Scholar

64 Gotham H. J. Diffusion of mental health and substance abuse treatments: development, dissemination, and implementation. Clin Psychol Sci Pract 2004; 11:160–76.CrossrefGoogle Scholar

65 VandeCreek L., Knapp S., Brace K. Mandatory continuing education for licensed psychologists: its rationale and current implications. Profess Psychol Res Pract 1990; 21:135–40.CrossrefGoogle Scholar

66 Barlow D. H., Levitt J. T., Bufka L. F. The dissemination of empirically supported treatments: a view to the future. Behav Res Ther 1999; 37: S147–62.CrossrefGoogle Scholar

67 Miller W. R., Yahne C. E., Moyers T. B., Martinez J., Pirritano M. A randomized trial of methods to help clinicians learn motivation interviewing. J Consult Clin Psychol 2004; 72:1050–62.CrossrefGoogle Scholar

68 Rubel E. C., Sobell L. C., Miller W. R. Do continuing workshops improve participants skills? Effects of a motivational interviewing workshop on substance-abuse counselors' skills and knowledge. Behav Ther 2000; 23: 73–7.1Google Scholar

69 Sholomskas D., Syracuse G., Ball S. A., Nuro K. F., Rounsaville B. J., Carroll K. M. We don't train in vain: a dissemination trial of three strategies for training clinicians in cognitive behavioral therapy. J Consult Clin Psychol 2005; 73:106–15.CrossrefGoogle Scholar

70 Crits-Christoph P., Siqueland L., Chittams J., Barber J. P., Beck A. T., Frank A. et al. Training in cognitive, supportive-expressive, and drug counseling therapies for cocaine dependence. J Consult Clin Psychol 1998; 66:484–92.CrossrefGoogle Scholar

71 Weissman M. M., Rounsaville B. J., Chevron E. Training psychotherapists to participate in psychotherapy outcome studies. Am J Psychiatry 1982; 139:1442–6.CrossrefGoogle Scholar

72 Carroll K. M., Connors G. J., Cooney N. L., DiClemente C. C., Donovan D. M., Longabaugh R. L. et al. Internal validity of Project MATCH treatments: discriminability and integrity. J Consult Clin Psychol 1998; 66:290–303.CrossrefGoogle Scholar

73 Vakoch D. A., Strupp H. H. The evolution of psychotherapy training: reflections on manual-based learning and future activities. J Clin Psychol 2000; 56:309–18.CrossrefGoogle Scholar

74 Miller W. R., Moyers T. B., Arciniega L., Ernst D., Forcehimes A. Training, supervision and quality monitoring of the COMBINE Study behavioral interventions. J Stud Alcohol 2005; Suppl. 15: 188–95;discussion 168–9.Google Scholar

75 Tober G., Godfrey C., Parrott S., Copello A., Farrin A., Hodgson R. et al. Setting standards for training and competence: the UK alcohol treatment trial. Alcohol Alcohol 2005; 40:413–8.CrossrefGoogle Scholar

76 Carroll K. M., Kadden R., Donovan D., Zweben A., Roun-saville B. J. Implementing treatment and protecting the validity of the independent variable in treatment matching studies. J Stud Alcohol 1994; Suppl. 12: 149–55.Google Scholar

77 Baer J. S., Ball S. A., Campbell B. K., Miele G. M., Schoener E. P., Tracy K. Training and fidelity monitoring of behavioral interventions in multi-site addictions research. Drug Alcohol Depend 2007; 87:107–18.CrossrefGoogle Scholar

78 Andrzejewski M. E., Kirby K. C., Morral A. R., Iguchi M. Y. Technology transfer through performance management: the effects of graphical feedback and positive reinforcement on drug treatment counselors' behavior. Drug Alcohol Depend 2001; 63:179–86.CrossrefGoogle Scholar

79 Martin G. W., Herie M. A., Turner B. J., Cunningham J. A. A social marketing model for disseminating research-based treatments to addictions treatment providers. Addiction 1998; 93:1703–15.CrossrefGoogle Scholar

80 Henggeler S. W., Schoenwald S. K., Liao J. G., Letourneau E. J., Edwards D. L. Transporting efficacious treatments to field settings: the link between supervisory practices and therapist fidelity in MST programs. J Clin Child Adolesc Psychol 2002; 31:155–67.CrossrefGoogle Scholar

81 Carroll K. M., Nich C., Sifry R., Nuro K. F., Frankforter T., Ball S. A. et al. A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug Alcohol Depend 2000; 57:225–38.CrossrefGoogle Scholar

82 Waltz J., Addis M. E., Koerner K., Jacobson N. S. Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. J Consult Clin Psychol 1993; 61:620–30.CrossrefGoogle Scholar

83 Calhoun K. S., Moras K., Pilkonis P. A., Rehm L. P. Empirically supported treatments: implications for training. J Consult Clin Psychol 1998; 66:151–62.CrossrefGoogle Scholar

84 McLellan A. T., Carise D., Kleber H. D. Can the national addiction treatment infrastructure support the public's demand for quality care? J Subst Abuse Treat 2003; 25:117–21.CrossrefGoogle Scholar

85 McLellan A. T. What we need is a system: creating a responsive and effective substance abuse treatment system. In: Miller W. R., Carroll K. M., editors. Rethinking substance abuse: what the science shows and what we should do about it. New York: Guilford Press; 2006, p. 275–92.Google Scholar

86 Olmstead T. A., Sindelar J. L., Petry N. M. Cost-effectiveness of prize-based incentives for stimulant abusers in outpatient psychosocial treatment programs. Drug Alcohol Depend 2007; 87:175–82.CrossrefGoogle Scholar

87 Chiles J. A., Lambert M. E., Hatch A. L. The impact of psychological interventions on medical cost offset: a meta-analytic review. Clin Psychol Sci Pract 1999; 6:204–20.CrossrefGoogle Scholar

88 Yates B. T. Toward the incorporation of costs, cost-effectiveness analysis, and cost-benefit analysis into clinical research. J Consult Clin Psychol 1994; 62:729–36.CrossrefGoogle Scholar

89 Addis M. E., Krasnow A. D. A national survey of practicing psychologists' attitudes toward psychotherapy treatment manuals. J Consult Clin Psychol 2000; 68:430–41.Google Scholar

90 Prochaska J. O., DiClemente C. C. Transtheoretical therapy: toward a more integrative model of change. Psychother Theory Res Pract 1982; 19:276–88.CrossrefGoogle Scholar

91 Beutler L. E., Moleiro C., Talebi H. How practitioners can systematically use empirical evidence in treatment selection. J Clin Psychol 2002; 58:1199–212.CrossrefGoogle Scholar

92 Kazdin A. E., Nock M. K. Delineating mechanisms of change in child and adolescent therapy: methodological issues and research recommendations. J Child Psychol Psychiatry 2003; 44:1116–29.CrossrefGoogle Scholar

93 O'Donohue W., Yeater E. A. Individuating psychotherapies. Behav Modif 2003; 27:313–21.CrossrefGoogle Scholar

94 Haaga D. A. F. Being systematic, and maybe even scientific, about planning psychotherapy research. Clin Psychol Sci Pract 2004; 11:387–390.CrossrefGoogle Scholar

95 O'Donohue W., Buchanan J. A., Fisher J. E. Characteristics of empirically supported treatments. J Psychother Pract Res 2000; 9:69–74.Google Scholar

96 Beutler L. E., Clarkin J. F., Bongar B. M. Guidelines for the Systematic Treatment of Depressed Patients. New York: Oxford University Press; 2000.Google Scholar

97 Beutler L. E., Harwood H. J. Prescriptive Psychotherapy: a Practical Guide to Systematic Treatment Selections. New York: Oxford University Press; 2000.Google Scholar

98 Ball S. A., Bachrach K., DeCarlo J., Farentinos C., Keen M., McSherry T. et al. Characteristics of community clinicians trained to provide manual-guided therapy for substance abusers. J Subst Abuse Treat 2002; 23:309–18.CrossrefGoogle Scholar

99 Berwick D. M. Disseminating innovations in health care. JAMA 2003; 289:1969–75.CrossrefGoogle Scholar

100 Wiltsey Stirman S., Crits-Christoph P., DeRubeis R. J. Achieving successful dissemination of empirically supported therapies: a synthesis of dissemination theory. Clin Psychol Sci Pract 2004; 11:343–59.CrossrefGoogle Scholar

101 Petry N. M. A comprehensive guide to the application of contigency management procedures in clinical settings. Drug Alcohol Depend 2000; 58:9–25.CrossrefGoogle Scholar

102 Kirby K. C., Marlowe D. B., Festinger D. S., Lamb R. J., Platt J. J. Schedule of voucher delivery influences initiation of cocaine abstinence. J Consult Clin Psychol 1998; 66:761–7.CrossrefGoogle Scholar

103 DeRubeis R. J., Crits-Christoph P. Empirically supported individual and group psychological treatments for adult mental disorders. J Consult Clin Psychol 1998; 66:37–52.CrossrefGoogle Scholar

104 Weissman M. M., Markowitz J. C. Interpersonal psychotherapy. current status. Arch Gen Psychiatry 1994; 51:599–606.CrossrefGoogle Scholar

105 Rubak S., Sandbaek A., Lauritzen T., Christensen B. Motivational interviewing: a systematic review and meta-analysis. Br J Gen Pract 2005; 55:305–12.Google Scholar

106 Burke B. L., Arkowitz H., Menchola M. The efficacy of motivational interviewing: a meta-analysis of controlled clinical trials. J Consult Clin Psychol 2003; 71:843–61.CrossrefGoogle Scholar

107 Henggeler S. W., Melton G. B., Brondino M. J., Scherer D. G., Hanley J. H. Multisystemic therapy with violent and chronic juvenile offenders and their families: the role of treatment fidelity. J Consult Clin Psychol 1997; 65:821–33.CrossrefGoogle Scholar

108 Foley S. H., O'Malley S. S., Rounsaville B. J. The relationship between patient difficulty and therapist performance in interpersonal psychotherapy. J Affect Disord 1987; 12:207–17.CrossrefGoogle Scholar

109 Barber J. G., Crits-Christoph P., Luborsky L. Effects of therapist adherence and competence on patient outcome in brief dynamic therapy. J Consult Clin Psychol 1996; 64:619–22.CrossrefGoogle Scholar

110 Henry W. P., Strupp H. H., Butler S. F., Schacht T. E., Binder J. L. Effects of training in time-limited dynamic psychotherapy: changes in therapist behavior. J Consult Clin Psychol 1993; 61:434–40.CrossrefGoogle Scholar

111 Castonguay L. G., Goldfried M. R., Wiser S., Raue P. J., Hayes A. M. Predicting the effect of cognitive therapy for depression: a study of unique and common factors. J Consult Clin Psychol 1996; 64:497–504.CrossrefGoogle Scholar

112 Sholomskas D. E., Carroll K. M. One small step for manuals: computer-assisted training in twelve-step facilitation. J Stud Alcohol 2006; 67:939–45.CrossrefGoogle Scholar

113 Issenberg S. B., McGaghie W. C., Hart I. R., Mayer J. W., Felner J. M., Petrusa E. R. et al. Simulation technology for health care professional skills training and assessment. JAMA 1999; 282:861–6.CrossrefGoogle Scholar

114 Todd K. H., Braslow A., Brennan R. T., Lowery D. W., Cox R. J., Lipscomb L. E. et al. Randomized, controlled trial of video self-instruction versus traditional CPR training. Ann Emerg Med 1998; 31:364–9.CrossrefGoogle Scholar

115 Huang M. P., Alessi N. E. The internet and the future of psychiatry. Am J Psychiatry 1996; 153:861–9.CrossrefGoogle Scholar

116 Piemme T. E. Computer-assisted learning and evaluation in medicine. JAMA 1988; 260:367–72.CrossrefGoogle Scholar

117 Jones I., Cookson J. Computer-assisted learning design for reflective practice supporting multiple learning styles for education and training in pre-hospital emergency care. Int J Train Dev 2001; 5:74–80.CrossrefGoogle Scholar

118 Budman S. H. Computer-mediated addiction services: tomorrow won't look like today. Behav Healthcare Tomorrow 2002; 11:14–21.Google Scholar

119 Davis D. D., Thompson O'Brien M. A., Freemantle N., Wolf F. M., Mazmanian P., Taylor-Vaisey A. Impact of formal continuing medical education. JAMA 1999; 282:867–74.CrossrefGoogle Scholar

120 Williams C. D., Aubin S., Harkin P., Cottrell D. A randomized, controlled, single-blind trial of teaching provided by a computer-based multimedia package versus lecture. Med Educ 2001; 35:847–54.CrossrefGoogle Scholar

121 Carise D., McLellan A. T., Gifford L. S., Kleber H. D. 1999 Developing a national addiction treatment information system: an introduction to the Drug Evaluation Network System. J Subst Abuse Treat 2002; 17:67–77.CrossrefGoogle Scholar

122 McLellan A. T., Belding M. A., McKay J. R., Zanis D., Alterman A. I. Can the outcomes research literature inform the search for quality indicators in substance abuse treatment? In: Edmunds M., Frank R. G., Hogan M., McCarty D., Robinson-Beale R., Weisner C., editors. Managing managed care: quality improvement in behavioral health. Washington, DC: National Academy Press; 1996.Google Scholar

123 Keane D. R., Norman G. R., Vickers J. The inadequacy of recent research on computer-assisted instruction. Acad Med 1991; 66:44–8.CrossrefGoogle Scholar

124 Clark R. E. Dangers in evaluation of instructional media. Acad Med 1992; 12:819–20.CrossrefGoogle Scholar

125 Vogel M., Wood D. F. Love it or hate it? Medical students' attitudes to computer-assisted learning. Med Educ 2002; 36:214–15.CrossrefGoogle Scholar

126 Cook D. A. Learning and cognitive styles in web-based learning: theory, evidence, and application. Acad Med 2005; 80:266–78.CrossrefGoogle Scholar

127 Lane J. M., Addis M. E. Pros and cons of educational technology as methods for disseminating evidence-based treatments. Clin Psychol Sci Pract 2004; 11:336–8.CrossrefGoogle Scholar

128 Crits-Christoph P., Baranackie K., Kurcias J., Beck A. T., Carroll K. M., Perry K. et al. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychother Res 1991; 1:81–91.CrossrefGoogle Scholar

129 Carroll K. M. Constrained, confounded, and confused: why we know so little about therapist effects. Addiction 2001; 96:203–6.CrossrefGoogle Scholar

130 Project MATCH Research Group. Therapist effects in three treatments for alcohol problems. Psychother Res 1998; 8:455–74.CrossrefGoogle Scholar

131 Budman S. H. Behavioral health care dot-com and beyond: computer-mediated communications in mental health and substance abuse treatment. Am Psychol 2000; 55:1290–300.CrossrefGoogle Scholar

132 Marks I., Shaw S., Parkin R. Computer-aided treatments of mental health problems. Clin Psychol Sci Pract 1998; 5:151–70.CrossrefGoogle Scholar

133 Norquist G., Regier D. A. The epidemiology of psychiatric disorders and the de facto mental health care system. Annu Rev Med 1996; 47:473–9.CrossrefGoogle Scholar

134 Kirkby K. C. Computer-assisted treatment of phobias. Psychiatr Serv 1996; 47:139–40.CrossrefGoogle Scholar

135 Baer L., Greist J. H. An interactive computer-administered self-assessment and self-help program for behavior therapy. J Clin Psychiatry 1997; 58:23–8.Google Scholar

136 Newman M. G., Kenardy J., Herman S., Taylor C. B. Comparison of palmtop-computer assisted brief cognitive-behavioral treatment to cognitive-behavioral treatment for panic disorder. J Consult Clin Psychol 1997; 65:178–83.CrossrefGoogle Scholar

137 Wright J. H., Wright A. S., Albano A. M., Basco M. R., Goldsmith L. J., Raffield T. et al. Computer-assisted cognitive therapy for depression: maintaining efficacy while reducing therapist time. Am J Psychiatry 2005; 162:1158–64.CrossrefGoogle Scholar

138 Selmi P. M., Klein M. H., Greist J. H., Sorrell S. P., Erdman H. P. Computer-administered CBT for depression. Am J Psychiatry 1990; 147:51–6.CrossrefGoogle Scholar

139 Slate S. E., Meyer T. L., Burns W. J., Montgomery D. D. Computerized cognitive training for severely emotionally disturbed children with ADHD. Behav Modif 1998; 22:415–20.CrossrefGoogle Scholar

140 Shiffman S., Paty J. A., Rohay J. M., DiMarino M. E., Gitchell J. G. The efficacy of computer-tailored smoking cessation material as a supplement to nicotine patch therapy. Drug Alcohol Depend 2001; 64:35–46.CrossrefGoogle Scholar

141 Carroll K. M. A Cognitive-Behavioral Approach: Treating Cocaine Addiction. Rockville, MD: NIDA.Google Scholar

142 Paul G. L. 1967 Strategy of outcome research in psychotherapy. J Clin Psychol 1998; 31:109–18.Google Scholar

143 Paul G. L., Licht M. H. Resurrection of the uniformity assumption myth and the fallacy of statistical absolutes in psychotherapy research. J Consult Clin Psychol 1978; 46:1531–4.CrossrefGoogle Scholar