Systematic Review

QUESTIONS & ANSWERS: Systematic Review at the Center for Psychiatric Rehabilitation

We know what we know—or at least we think we do. Research is proposed, funded, conducted, and published; there are incentives in place at each step in that process. Yet, when a research study has been published, what guarantee do we have that the findings will be properly understood or widely known? Is there any guarantee that its conclusions will become part of a larger picture in the field?

In order to maintain confidence in our interventions as well as to ensure that they are more widely known and brought into use, the field of psychiatric rehabilitation must turn a critical eye to the fundamentals of the discipline. Just as the field asks practitioners to be open to new ideas about recovery, so the field must always be willing to take a hard look at its own conclusions—and the research on which those conclusions are based.

Now, with funding from the National Institute on Disability and Rehabilitation Research (NIDRR), the Center for Psychiatric Rehabilitation has conducted a systematic review of Supported Housing—the first of several projected reviews of key research areas, including supported education and peer support. Supported Housing is an intervention intended to provide individuals with severe mental illness housing that is integrated into the community and give them control over the available services and supports.

According to Sally Rogers, Director of Research and Co-PI of this Knowledge Translation effort, “Through a rigorous, measured assessment of the research that is out there in a particular field, we have a chance to reaffirm or reprioritize the direction of the field. We have the ability to reframe the questions and to deemphasize questions that have been adequately addressed. In short, systematic review is a critical tool for our discipline.”

Center Director William Anthony adds, “Supported housing is such a critical intervention and element in a person’s recovery. We hoped that there would be a substantial body of quantifiable research out there, and that’s what we found. That so much of the research would meet the standards of our systematic review process was very encouraging.”

By examining the field topic by topic, systematic reviews can make plain the state of psychiatric rehabilitation by flagging conclusions based on slender evidence, highlighting urgent research needs, and underlining conclusions that are strongly supported and therefore make it possible to avoid redundant research. In a period of scarce resources and funding, a thorough understanding of what we need to know versus what we already know is clearly valuable.

What is a systematic review and how is it performed?

The practice of systematic review originated in the medical profession. The Cochrane Collaboration pioneered the practice more than a decade ago, producing dozens of reviews in various areas of medical science. The Cochrane Library, their clearinghouse for information, has become an invaluable resource for medical professionals. Seeing the need for reviews in the mental health area, Cochrane sought to undertake a review of supported housing; however, as part of its guidelines Cochrane considers only randomized clinical trials (RCT) sufficiently rigorous to warrant systematic review. As RCTs are difficult to perform in the field of psychiatric rehabilitation, they found no studies that could be included and the review was abandoned.

Believing that a review of the existing studies could nonetheless be performed, the Center for Psychiatric Rehabilitation undertook a systematic review of their own. Allowing for the lack of RCTs, and creating new measures for both the meaning and rigor of research, the Center began the process of review.

Going as far back as 1993, researchers queried databases, reviewed titles, and obtained abstracts. If the abstract proved relevant, the researchers reviewed the article in its entirety, measuring its content against specific inclusion criteria. The complete list of articles was then sent to several experts in housing research who made suggestions to ensure that no significant article or report about supported housing had been overlooked.

This process resulted in identifying 158 articles that were reviewed for inclusion; that pool was reduced to 78. That final group was rated for rigor and meaning and categorized according to study design: experimental, quasi-experimental, correlational, and pre-post. The reviewers concluded that some studies had critical flaws and dropped twelve more for having a poor research design, retrospective measurement, or an unreasonable dropout rate among the study subjects.

The studies were also graded for quality. Marianne Farkas, Co-PI of the Knowledge Translation effort and a member of the Supported Housing Study Group, comments, “Good quality research information is derived from its rigor (i.e., is the research designed and conducted in a rigorous manner?) as well as its meaning (i.e., is it designed and reported in a manner that allows interested parties—such as consumers, families, providers, administrators, etc.—to make a decision about the relevance of the information to their own situation?).” Each study was thus given a rating according to measures of Meaning and Rigor. Researchers also attempted to rate the extent to which the housing intervention complied with key criteria of supported housing, including the separation of housing and clinical services, the availability of crisis services, and factors like affordability, independence, permanence, and choice.

Selected conclusions from the systematic review

The articles that survived the selection process for the systematic review attempt to answer questions ranging from who is most and least helped by supported housing to what model of supported housing has the greatest retention rate. Conclusions from these studies, put through the mea­sures of the systematic review, offer practitioners, consumers, and the public insight into what we can be reasonably certain we know about the effectiveness of the supported housing intervention. Here are a few of the many conclusions:

  • Housing stability rates close to or exceeding 80 percent can be achieved with a supported housing approach (Goldfinger, et al., 1999), meaning that individuals are housed 80 percent of the time for the follow-up period. While not all studies achieved this level of housing stability, (for example, 66 percent was seen in a large study of veterans), more than one well-controlled study achieved this rate.
  • The Housing First model in New York City has achieved a housing retention rate greater than 80 percent over a several-year period (Tsemberis, 1999).
  • Participants prefer independent housing and expressed more satisfaction with it (Lipton, et al., 1988; Schutt, et al., 1997; Tsemberis, 1999).
  • The role of psychiatric diagnosis is equivocal in supported housing with some studies suggesting that diagnosis is not a factor in residential stability (Rosenheck, et al., 2003) and some suggesting that it is (Hurlburt, et al., 1996).
  • Supported housing interventions and services, especially programs that emphasize residents’ choice for housing, are related to housing stability and improvements in housing satisfaction and quality of life (Srebnik, et al., 1995; Nelson, et al., 2003).
  • Use of acute services—including time spent in shelters, emergency rooms, and hospitals—decreased with the provision of housing supports (including case management) or supported housing services (Goering, et al., 1997; Wong, et al., 2008; Martinez & Burt, 2006; Newman, et al., 1994; Hanrahan, et al., 2001; Middelboe, 1997).
  • Generally demographic features were not helpful in predicting housing tenure (Mares, et al., 2004); however some evidence suggests that women were more likely to achieve stable housing than men (Kasprow, et al., 2000).
  • Low-demand supported housing approaches that do not have sobriety requirements may be just as effective in retaining consumers in independent housing (Martinez & Burt, 2006).

What does the review tell us about supported housing in general?

Looking at the results of the systematic review of supported housing indicates that there are several well-controlled studies of supported housing in addition to several studies conducted with less-rigorous designs. Overall, the synthesis suggests that supported housing can improve the living situation of individuals who are psychiatrically disabled, homeless, and have substance abuse problems.

We can draw a number of other conclusions as well. Subsidies or vouchers are helpful in getting and keeping individuals housed. Housing services appear to be not only cost-effective but also help reduce the costs of other social and clinical services. In order to be most effective, intensive case management services (rather than traditional case management) are needed and will generally lead to better housing outcomes. It’s also important to have access to affordable housing and a service system that is well integrated. Providing a person with supported housing reduces the likelihood that they will be re-hospitalized, although supported housing does not always lead to reduced psychiatric symptoms. In addition, and of critical importance to any consumer-oriented approach, supported housing can improve clients’ quality of life and satisfaction with their living situation. Providing supported housing options that are of decent quality is important to keep people housed and satisfied with their housing.

“In short, looking at the overall review,” observes Rogers, “we can safely say that supported housing works. Given that the Cochrane Collaboration chose not to
conduct a review, we were surprised and impressed by the number of quality studies.”

Looking ahead

As the Center’s systematic review of supported housing becomes available via the web and more widely known through presentations and other means of dissemination, what impact can be expected?

The knowledge that multiple robust research studies show that supported housing works should be of tremendous value in making policy determinations—as well as allowing debate to center on how, instead of whether, to implement supported housing programs.

In addition, identifying which aspects of supported housing are best understood should guide future researchers towards what we do not know about the benefits and difficulties in providing interventions in this critical area. The process of systematic review is new to the field of disability research but holds great promise, allowing new research to be honed by a critical awareness while providing for more efficient implementation of funding and resources going forward.



Goering, P., Wasylenki, D., Lindsay, S., Lemire, D. & Rhodes, A. (1997). Process and outcome in hostel outreach program for homeless clients with severe mental illness. American Journal of Orthopsychiatry, 67, 607–617.

Goldfinger, S. M., Tolomiczenko, G. S., Seidman, L., Penk, W. E., Turner, W. & Caplan, B. (1999). Housing placement and subsequent days homeless among formerly homeless adults with mental illness. Psychiatric Services, 50, 647–679.

Hanrahan, P., Luchins, D., Savage, C. & Goldman, H. (2001). Housing satisfaction and service use by mentally ill persons in community integrated living arrangements. Psychiatric Services, 52, 1206–1209.

Hurlburt, M., Wood, P. A., & Hough, R. L. (1996). Providing independent housing for the homeless mentally ill. Journal of Community Psychology, 24, 291–310.

Kasprow, W., Rosenheck, R., Frisman, L. & Dilella, D. (2000). Referral and housing processes in a long-term supported housing program for homeless veterans. Psychiatric Services, 51, 1017–1023.

Lipton, F. R., Nutt, S. & Sabatini, A. (1988). Housing the homeless mentally ill: A longitudinal study of a treatment approach. Hospital and Community Psychiatry, 39, 40–45.

Mares, A. R. & Rosenheck, R. A. (2004). One-year housing arrangement among homeless adults with serious mental illness in the ACESS program. Psychiatric Services, 55, 566–574.

Martinez, T. B. & Burt, M. R. (2006). Impact of permanent supportive housing on the use of acute care health services by homeless adults. Psychiatric Services, 57, 992–999.

Middelboe, T. (1997). Prospective study of clinical and social outcome of stay in small group homes for people with mental illness. British Journal of Psychiatry, 151, 251–255.

Nelson, G., Hall, B. & Forchuk, C. (2003). Current and preferred housing of psychiatric consumers/survivors. Canadian Journal of Community Mental Health, 22, 5–19.

Newman, S. J., Reschovsky, J. D., Kaneda, K. & Hendrick, A. M. (1994). The effects of independent living on persons with chronic mental illness: An assessment of the Section 8 certificate program. Milbank Quarterly, 72, 171–198.

Rosenheck, R., Kasprow, W., Frisman, L. & Liu-Mares, W. (2003). Cost effectiveness of supported housing for homeless persons with mental illness. Archives of General Psychiatry, 60, 940–951.

Schutt, R. K., Goldfinger, S. M. & Penk, W. E. (1997). Satisfaction with residence and with life: When homeless mentally ill persons are housed. Evaluation and Program Planning, 20, 185–194.

Srebnik, D., Livingston, J., Gordon, L. & King, D. (1995). Housing choice and community success for individuals with serious and persistent mental illness. Community Mental Health Journal, 31, 139–152.

Tsemberis, S. (1999). From streets to homes: An innovative approach to supported housing for homeless adults with psychiatric disabilities. Journal of Community Psychology, 27, 255–241.

Wong, Y. L., Poulin, S. R., Leeb, S., Davis, M. R. & Hadley, T. R. (2008). Tracking residential outcomes of supported independent living programs for persons with serious mental illness. Evaluation and Program Planning, 4, 416–426.