Measuring Mental Health Outcomes
Measuring Outcomes for Quality and Accountability
“Providers of services shall meet standards for quality that lead to optimal outcomes for persons served.”
—CARF, Commission on Accreditation of Rehabilitation Facilities (2007)
With accountability pressures and growing concerns about quality of services, managers, practitioners, and system planners need resources to measure the outcomes of the services they provide and should be aware of the best efforts to measure client outcomes and services effectiveness.
This newsletter provides resources for program administrators, managers, policy makers, and others about the implementation and use of outcome measurement. The drive toward accountability that is occurring in the mental health field is explored, as are other reasons for measuring client outcomes. Outcome measures and instruments developed as part of multi-site studies sponsored by the Substance Abuse and Mental Health Services Administration (SAMHSA) are described inside; and can serve as a valuable resource in planning an evaluation or quality assurance initiative. Also included is a set of considerations when planning and conducting a program outcome evaluation.
The Changing Landscape
Over the past decade, the field of mental health and rehabilitation has experienced an increasing demand for accountability in a broad range of service settings. Funding agencies and managed care entities, including state mental health agencies, mandate that providers examine the effectiveness of services and measure outcomes among consumers served. Providers must respond to accreditation bodies such as the Joint Commission Accreditation of Healthcare Organizations (JCAHO) and the Council on the Accreditation of Rehabilitation Facilities (CARF) with data to demonstrate effectiveness and service satisfaction among service recipients.
In addition to pressures for accountability, agencies and organizations view outcome measurement and service effectiveness as critical elements to address quality improvement and the capacity of management. Outcome measurement is also an important reflection of an organization’s core values and a way to obtain valuable input from consumers about the services received.
“The quality of life of people with severe mental disorders can be improved by making the delivery of services an object of rigorous scientific inquiry to determine what works, and by assuring that the results are applied to systems of care…”
—National Institute of Mental Health (NIMH), (2001)
Planning a Program Evaluation
Beyond individual accounts and anecdotes, practitioners and managers want concrete and standardized ways to measure effectiveness to address questions regarding service effectiveness and best practices1. Over the past 25 years, staff members at the Center for Psychiatric Rehabilitation frequently have found that program administrators are not well informed about outcome measurement and may not be aware of how best to link outcome measures to questions that they want to address related to services.
The best starting point is to determine:
- What outcomes need to be addressed;
- Who the evaluation information is being developed for; and
- Who will receive information about the results.
Results can be disseminated to stakeholders such as funding and accreditation bodies, a board of directors, primary consumers, or family members.
It is helpful to keep in mind that a program evaluation effort can be started at any time. While it might be desirable to begin such an evaluation effort when a service starts, it is always possible to add on an evaluation component to an existing program or service.
Clarifying Outcomes To Be Measured
Conducting an evaluation, study, or quality assurance effort begins with understanding which outcomes need to be addressed. The following steps help to clarify what the outcomes to be measured are:
- Begin with an understanding of your agency’s intended areas of impact or likely areas of effectiveness. (For example, Is your agency’s stated goal to increase employment outcomes? Is supported employment where your agency has historically had the most impact?)
- Ask whether the services your agency offers are likely to go beyond the intended outcomes to other unintended outcomes. (For example, Does an increase in employment outcomes result in improved self-esteem among service recipients? Should you try to capture that as an outcome?)
- Examine your agency’s mission statement, strategic plan, or results of former program evaluation efforts (if available) to hone in on your agency’s intended effects and thus the outcomes you should measure.
- Determine what domains, or areas, need to be measured from these efforts (for example, housing, employment, social inclusion, satisfaction).
- Develop clear definitions, or criteria, for these domains so that the appropriate measures or instruments can be located. (For example, if your intended effect is to increase employment outcomes, does that mean: a) an increase in the number of individuals employed in competitive work; b) an increase in the number employed in any kind of work; c) increases in length of employment, or all of the above?)
Selecting an Outcome Measure or Instrument
Evaluating program effectiveness decreases vulnerability to stakeholders and funders who want to know whether a program represents money well spent. However, providers must have confidence in the measures being used. Choosing high quality and appropriate measures is critical.
Look for existing measures that match the outcomes needing to be addressed, rather than using home-grown surveys or instruments developed just for the purpose at hand. Doing so will ensure that the measures have been tested and are known to be reliable and valid (see the SAMHSA Multi-site Studies Table and Empowerment Scale as excellent examples).
SAMHSA Multi-Site Studies
Over 10 years ago, SAMHSA embarked on a series of large-scale, multi-site studies; and in the course of planning these studies, instruments to measure the outcomes and effectiveness of services were carefully sought out and considered. The examining process included: the relevance of the instrument to the questions being asked; the burden of administering the instrument; and the accuracy and consistency of the measure (sometimes called reliability and validity); and the ability of the measure to capture the concept of interest. All of these factors are important considerations in determining the quality of any measuring tool.
These studies examined the effectiveness of services and interventions for individuals with serious mental illnesses, often by examining those services in real world contexts rather than controlled and artificial circumstances. These multi-site studies brought together a variety of experts in the field of mental health, rehabilitation, research, policy, administration, and individuals who could speak to the lived experience of having a psychiatric disability; and did so across many geographically dispersed sites.
For example, the Consumer Operated Services Program (COSP) multi-site study examined 8 consumer-operated programs across the country including drop-in centers, peer support, and education and advocacy programs run for consumers by consumers. They considered 5 important domains of outcomes and functioning likely to be affected by the programs including: empowerment, housing, social inclusion, employment, and satisfaction with services. Their compilation of instruments was developed with a significant amount of deliberation, vetting, and discussion among researchers, program administrators, and consumers to ensure that the measures chosen were relevant, acceptable to consumers, and that they were a good match for the questions being studied.
The Empowerment Scale
The Empowerment Scale is another example of a well-vetted measure that can be used by others with compatible outcome measurement requirements. Since the Empowerment Scale was first developed and published by the Center for Psychiatric Rehabilitation, it has been used in a variety of settings including peer-run programs, multi-site federally funded studies, universities, community support programs, and residential programs, to name a few. The scale has been translated for programs in Italy, Japan, the Netherlands, Australia, and England. Most recently, the scale has been adopted for use in a Pan-European study in which it will be formally translated into 18 languages.
The growing interest in the scale seems to stem from two important paradigm shifts in the field: the need for reliable and valid measures that tap into the internal or subjective aspects of recovery and, secondly, the need for measures that were developed from a consumer perspective. (For more information on the Empowerment Scale, contact Sally Rogers at firstname.lastname@example.org.)
“The Joint Commission presides over a growing, national, comparative performance measurement database that can inform internal health care organization quality improvement activities, external accountability, pay for performance programs and [can] advance research.”
—JCAHO, Joint Commission on Accreditation of Healthcare Organizations (2007)
Moving Ahead…the Next Step
Times are changing, and providers, practitioners, managers, administrators, and policy makers can ease the pressures they face by availing themselves of established, high quality measures and instruments; and by clarifying which effects and outcomes ought to be measured. The results may be used not only to satisfy funding bodies and other stakeholders; they will provide useful information that can guide the direction of changes needed to create more effective ways of helping the people being served.
It is our hope that these resources will prove highly valuable in evaluating the effects of services and, in turn, providing effective care, best practices, and evidence-based services.
References and Links
Commission on Accreditation of Rehabilitation Facilities. (2007). Retrieved March 26, 2007 from http://www.carf.org/consumer.aspx?content=content/Publications/Online/RUReady/Ends.htm.
Drake, R., Goldman, H., Leff, H. S., Lehman, A. F., Dixon, L., Mueser, K. T., Torrey, W. (2001). Implementing Evidence-Based Practices in Routine Mental Health Service Settings. Psychiatric Services 52:179–182.
The Joint Commission on Accreditation of Healthcare Organizations. (2007). Retrieved March 26, 2007 from http://www.jointcommission.org/PerformanceMeasurement/.
National Institute of Mental Health. (2001). Caring for people with severe mental disorders: A national plan of research to improve services. DHHS Pub. No. (ADM) 91-1762. Washington, DC: Supt of Docs., U.S. Govt Printing Office.
From page 1, Note 1. The term “evidence-based practices” (EBP) was not used since that term has a specific connotation in mental health services research. Evidence-based practices are interventions for which there is “consistent scientific evidence showing that they improve client outcomes” (Drake, et al., 2001; p. 180). The “requirements for scientific evidence used by different groups sometimes vary, but in general the highest standard is several randomized clinical trials comparing the practice to alternative practices or to no intervention.” (Drake, et al. 2001; p. 180; for additional details about EBPs, consult the Drake article).
Reference: Drake, R., Goldman, H., Leff, H. S., Lehman, A. F., Dixon, L., Mueser, K. T., Torrey, W. (2001). Implementing Evidence-Based Practices in Routine Mental Health Service Settings. Psychiatric Services, 52, 179–182.
Table 1. Instruments from a Sampling of SAMHSA-Funded Multi-Site Studies
The instruments described below were developed as part of the SAMHSA multi-site studies. Since each of these studies was conducted using leading experts and key figures in their respective fields, each of these studies represents the best understanding of the outcome domain under study. The table provides the name of the study, the primary focus of the study, the Coordinating Center directors, the Federal project officers from SAMHSA, the domains that were measured, and who to contact for additional information about the measures or instruments that were used.
Primary Focus or Purpose
Domains of Measurement
Contact Person for More Information
|Employment intervention and demonstration||To compare the effectiveness of supported employment approaches to other approaches to vocational rehabilitation||Comprehensive employment and vocational information (both historical and at study entry); quality of life, physical health, level of functioning, social skills, substance abuse, symptoms and psychiatric treatment; services integration; costs of services||Judith Cook, Ph.D. (PI of the Coordinating Center)
Crystal Blyler, Ph.D. (Federal Project Officer)
|Judith Cook, Ph.D. at: email@example.com; Crystal Blyler, Ph.D. at: Crystal.Blyler@samhsa.hhs.gov.
Study website: http://www.psych.uic.edu/eidp/
|Supported housing||To study the effects of various types of housing for individuals with mental illness||Residential stability, residential functioning, physical and mental health, substance use/abuse, quality of life, treatment and service use, psychosocial functioning, consumer empowerment and self-efficacy||Debra Rog, Ph.D.
Westat and (PI’s of the Coordinating Center) and Fran Randolph, Ph.D. (Federal Project Officer)
|Debra Rog, Ph.D. firstname.lastname@example.org or 301-279-4594|
|Consumer operated Services||To study the added effects of consumer operated programs for individuals receiving traditional mental health services||Empowerment, housing, employment, social inclusion, satisfaction with services; residential, employment and clinical status; symptoms; physical health, perceived discrimination; recovery; substance use; service use, quality of life; hope and spirituality||Jean Campbell, Ph.D. and Matt Johnsen, Ph.D. (Co-PIs of the Coordinating Center)
Crystal Blyler, Ph.D. and Betsy McDonel Herr (Federal Project Officers)
Jean Campbell, Ph.D. at email@example.com or 314-877-6457; Study website: www.cstprogram.org
|Criminal Justice Diversion||To improve policy and practice for addressing the needs of individuals with mental illness involved in the criminal justice system; by creating services linkages, providing treatment services based on best practices; promoting a comprehensive service delivery system; outreach to the community about criminal justice diversion||Trauma history; mental health symptoms, perceived coercion; perceived outcomes; mental health; process information on the volume of screening and evaluation activities and who is served by jail diversion programs; substance abuse service use.||Policy Research Associates and Research Triangle Institute or Susan Salasin (Federal Project Officer)||Chan Noether, M.A. Policy Research Associates, CNoether@prainc.com or 518-439-7415 X 224
or Susan Salasin at Susan.Salasin@samhsa.hhs.gov
|Managed Care||To compare service use, client outcomes, and client satisfaction in samples of individuals enrolled in managed care programs versus fee-for-service behavioral health care plans||Mental health status; symptomatology; level of functioning; substance use; service use; systems outcomes include service use and expenditures||Gini Mulkern, Ph.D. (PI of the Coordinating Center)
Jeff Buck, Ph.D. (Federal Project Officer)
|Gini Mulkern, Ph.D. at firstname.lastname@example.org or 617-876-0426 or Jeff.Buck@samhsa.hhs.gov|
|Access to Community Care and Effective Services and Supports (ACCESS)||To compare two different service conditions (one being systems level integration with outreach and case management and the second being outreach and case management alone) in their effectiveness at helping persons with mental illness avoid homelessness||Residential status and stability; employment status; substance use; clinical outcomes; victimization, social supports, quality of life; and service use. At the service level systems integration; accessibility and coordination of services; interagency collaboration; service provision||Howard Goldman, M. D. Joe Morrissey Ph.D. Robert Rosenheck, M. D. Hank Steadman (Co-PI’s of the Coordinating Center) and Fran Randolph, Ph.D. (Federal Project Officer)||Hank Steadman, Policy Research Associates: 518-439-7415 or HSteadman@prainc.com; or Fran Randolph at Fran.Randolph@samhsa.hhs.gov|
|Homeless Families||To document and evaluate the effectiveness of time-limited, intensive intervention strategies for providing treatment, housing and family preservation services to homeless mothers with psychiatric and or substance use disorders who are caring for their dependent children||Psychological distress, recovery, substance use or abuse, residential stability, well-being of children, health resources, and parenting outcomes||Debra Rog, Ph.D. (PI of the Coordinating Center) or
Lawrence Rickards (Federal Project Officer)
|Debra Rog, Ph.D.email@example.com or 301-279-4594
Or Lawrence Rickards, Ph.D. at Lawrence.Rickards@samhsa.hhs.gov