Читать книгу Practitioner's Guide to Using Research for Evidence-Informed Practice - Allen Rubin - Страница 41

As we've already intimated, the individual studies and reviews that you'll find in your search might vary greatly in regard to their objectivity and rigor. The journal peer review process offers a level of quality assurance. In the peer review process, typically two or three other researchers offer critical feedback and help journals decide whether or not an article is appropriate for publication. Therefore, published articles in peer-reviewed journals have at least been exposed to some process of review and critique. However, the rigor of the review process in journals can vary greatly. Some very strong research studies do not appear in journal articles, while some relatively weak studies do get published. Some studies and reviews, whether in journal articles or other sources, for example, will be conducted and reported by individuals or groups with vested interests. But reviews and studies can be flawed even when no vested interests are involved. Some objective investigators do the best they can with limited resources to overcome some practical obstacles that keep them from implementing their study in a more ideal manner. A while back, for example, Rubin conducted an experiment evaluating the effectiveness of EMDR in a child guidance center (Rubin et al., 2001). He had no funding for the study and conducted it simply because – as a professor – he was expected to do research and was quite curious about whether EMDR was really as effective with children as its proponents were touting it to be. The administrative and clinical leaders in the center projected that in a year's time over 100 clients would participate in his study. They were wrong. It took three years for them to refer 39 clients to his study.

Some flaws are egregious and fatal. That is, they destroy the credibility of the study's findings. To illustrate a fatally flawed fictitious study, suppose Joe Schmo invents a new therapy for treating anxiety disorders. He calls it psyschmotherapy. If it is effective, he will be rich and famous. To evaluate its effectiveness, he uses his own clinical judgment to rate client anxiety levels – on a scale from 0 to 100 – before and after he provides psyschmotherapy to 10 of his clients. His average before rating is 95, indicating extremely high anxiety. His average after rating is 10, indicating extremely low anxiety. He concludes that psyschmotherapy is the most effective intervention available for treating anxiety disorders – a miracle cure, so to speak. You probably can easily recognize the egregious bias and utter lack of trustworthiness evident in Joe Schmo's study.

Other flaws are more acceptable. For example, suppose instead of using the foregoing evaluation approach, Joe Schmo proceeds as follows. He collaborates with a colleague who works in his mental health clinic and who specializes in cognitive-behavioral treatment for anxiety disorders. Joe provides psyschmotherapy to the first 10 new clients referred to him for treatment of anxiety disorders. His colleague provides cognitive-behavioral treatment to the first 10 new clients referred to him with anxiety disorders. To measure outcome, a graduate student who does not know what the study is about is hired to interview clients briefly before and after treatment and ask them to rate their average daily anxiety level (from 0 to 100) during the previous seven days. Regardless of the findings, we can see that this study is more credible than the previous one. It has flaws, but its flaws are neither egregious nor fatal. Maybe, for example, there are some differences in the types of clients referred to the two therapists, making one group more likely to improve than the other. Maybe all the clients in both groups exaggerated the improvements in their anxiety levels because they wanted to believe the treatment helped them or wanted the study's findings to please their therapist.

While these flaws may not be fatal, they are important. If you can find studies less flawed than that one, you'd probably want to put more stock in their findings. But if that study is the best one you can find, you might want to be guided by its findings. That is, it would offer somewhat credible – albeit quite tentative – evidence about the comparative effectiveness of the two treatment approaches. Lacking any better evidence, you might want – for the time being – to employ the seemingly more effective approach until better evidence supporting a different approach emerges or until you see for yourself that it is not helping your particular client(s).

Unlike these fictitious examples, it is not always so easy to differentiate between reasonable “limitations and fatal flaws; that is, to judge whether the problems are serious enough to jeopardize the results or should simply be interpreted with a modicum of caution” (Mullen & Streiner, 2004, p. 118). What you learn in the rest of this book, however, will help you make that differentiation, and thus help you judge the degree of caution warranted in considering whether the conclusions of an individual study or a review of studies merit guiding your practice decisions.