Читать книгу Achieving Equity and Excellence - Douglas Reeves - Страница 10

CHAPTER 1

Understand the Five Levels of Educational Research

Educators and leaders are weary of the vague claims that “research shows …” In conversations around the globe, they tell me that ambiguous claims are not enough to change their practices. They want to see credible evidence of the impact of improved leadership and teaching practices, and will not settle for claims without credible evidence. This chapter will help you to be a more critical consumer of educational research. Just as we ask students to evaluate claims based on the evidence, we must model that same level of critical thinking every time we listen to a presentation or read an article or book. As this chapter argues, there is an enormous difference between the credibility of a “journey story” about one person’s experience—a sample size of one—and the highest level of research, the preponderance of evidence.

In this chapter, we explore five levels of educational research. These include (1) personal beliefs, (2) personal experiences, (3) collective experience, (4) systematic comparisons, and (5) preponderance of the evidence. When legislators and other policymakers, leaders, or teachers state, “Research shows that …,” they might be referring to any one of these types of research. However, as citizens hoping to implement effective change in our schools, we can only truly rely on the higher levels of research—systematic comparison and preponderance of the evidence—to produce results that could feasibly be applicable and transferrable to our own schools. It is, therefore, essential that when we hear claims about research, we identify which type of research the people making the claims are using as the basis for their conclusions.

Level 1: Personal Beliefs

People are entitled to their own beliefs. That’s a guarantee in the United States (thanks to the First Amendment to the Constitution), as well as in many other nations around the world. The United States is hardly unique in protecting freedom of beliefs. Indeed, the founding documents of the United States drew heavily from thought leaders in England, France, and many other countries (Lepore, 2018) where laws prohibit governments from interfering with the free exercise of anyone’s religious beliefs, right to free speech, and freedom of assembly. As long as our beliefs do not interfere with the rights of others, we are free to believe whatever we wish.

Hand-in-hand with this freedom of personal beliefs, however, is the ability to believe something that is categorically wrong—and to persist in that belief due to personal preference, even in the face of evidence to the contrary. When presented with scientific evidence that defies personal beliefs, it is not unusual for someone to respond, proudly, “Your research may say that out-of-school suspension, punishment for missing homework, corporal punishment for behavioral infractions, and forcing students to stand in the corner with a dunce cap doesn’t work, but it sure worked for me!” These claims are contrary to research and common sense (Reeves, 2011a), but in a free society, we tolerate them—at least as long as the actions associated with these claims do not harm others.

Whether the forum is a meeting of teachers, a public comment portion of a school board meeting, a legislative hearing, or a speech at a political rally, listeners must ask, “What type of research is this?” When the answer is a personal belief, unburdened by factual evidence, then we can accept it as a by-product of the constitutional guarantee of free speech and exclude it from the realm of evidence-based claims, no matter how sincerely held the belief may be. We cannot debate beliefs any more than we can debate sincerely held religious views. Rather, when encountering strongly held personal beliefs not based on evidence, one should respond respectfully, “Thank you for sharing your views. I want to assure you that I respect your beliefs and your right to hold them. I hope that you’ll also respect other beliefs that are different from your own. Please don’t confuse personal beliefs with evidence.”

Level 2: Personal Experiences

The first level, personal beliefs, is based on sincerely held views that are sometimes supported by examples. Some teachers, administrators, and parents—and indeed some students—believe that grading as punishment is effective, even though a century of evidence undermines the veracity of that belief. The second level, personal experience, extends beyond belief statements and relies instead on what many educators regard as the most compelling evidence of all—their individual encounters with students. Beliefs do not spring from a void; they are often based on one’s own learning and experiences. Many people who did well in school attend college and become educators or administrators. They formulate their beliefs about grading and discipline policies based on a system that is clearly effective for them. “It worked for me!” the opponents of grading and discipline reform will say. They are, of course, correct. Grading systems practices that are, according to the evidence, damaging (such as grading as punishment, the use of zeros on the hundred-point scale, and the use of the average; Reeves, 2011a) do, in fact, motivate some students to higher levels of performance. In my conversations with educators and administrators, I have asked, “For whom were these grading practices effective?” The most frequent answers are, “It worked for me!” and “It worked for my own children!” This is not surprising, coming from the viewpoint of college-educated professionals who are now public school educators. They loved school. They were good at school. They figured out the grading and discipline systems and responded well to rewards and punishments based on grades. The question is, What percentage of today’s students do we expect to become public school educators? If the answer is less than 100 percent, then perhaps we should reconsider attributing to all students the same motivational scheme—grades as rewards and punishments—that were effective for many teachers. Personal anecdotal evidence may be powerful when forming our belief systems, but we must be careful about generalizing our personal experiences to all students. It is preferable to adopt an attitude of “There is always something to learn,” and, indeed, I have seen teachers who, even in the final months of their careers, continue to improve their practices, learn from research, and change their techniques in order to better serve the cause of student learning.

Level 3: Collective Experience

Irving L. Janis (1982) coined the term groupthink to describe the tendency of people to acquiesce to the group and submerge their own better judgment. Sometimes the consequences are trivial, such as in psychological experiments in which subjects are shown two lines, clearly of different lengths, but other observers (confederates of the researchers) claim the lines are of equal length. If there is only one other person in the room, the research subjects hold their own, maintaining the lines are clearly different lengths. But as more people in the room claim the lines are of equal length, the research subjects cave to the inaccurate observations of the group (Heath & Heath, 2013). The consequences are less benign when the conversation participants are debating issues of greater importance.

Although an individual teacher may seek to change his or her practice—perhaps by calling on students randomly or by simply rearranging desks—professional isolation can be terribly lonely. Group experience provides the illusion of certainty, of unanimity, of proof: “It’s not just me who believes this—it’s the entire mathematics department! It’s the entire third-grade team! Our faculty voted, and we are unanimous.” Thus, popularity supplants reason. It is critical to remember, however, that even multiple instances of personal experiences or beliefs are still just personal experiences or beliefs. They remain anecdotal, and as such, are fallible in the absence of more credible research.

Level 4: Systematic Comparisons

Systematic comparisons produce some of the most effective and persuasive research that can directly influence professional practices. Consider the example of a teacher who makes an alteration to her teaching practice in the second semester. She can subsequently make three comparisons to form a very strong basis for research conclusions. The first is a comparison of the same group of students before and after the intervention. For example, the teacher can compare student performance in the first semester, when there was no change in professional practices, with student performance in the second semester, when the teacher altered a particular aspect of her practice. Perhaps it was a change in when students do homework—from at-home practice to in-school practice. Perhaps it was a change in the way the students revise and respond to teacher feedback. Perhaps it was the use of student self-assessment before submitting work to the teacher. The beauty of a systematic comparison is all other variables are held constant; the only change is the change in teacher professional practices. The students are the same, as are the schedule, curriculum, assessment, and teacher. If there is a change in results, it is very likely due to the change in teacher practice. In the second case, the teacher might make year-to-year comparisons, comparing the attendance, behavior, or academic performance in the first semester of one year to the first semester of the previous year. In these situations, the students are different, but the curriculum, assessments, time allocation, schedule, and teacher are the same. Finally, in the third case, the teacher might compare her results with the results of colleagues with similar students who did not change professional practices. Such studies allow for minimal variables and, hence, enable the teacher to draw powerful, generalizable conclusions about the effects of the professional practice she changed.

Every school and system, no matter how small or large, can conduct systematic comparisons like these. Teachers do not require a federal research grant, university evaluators, or any special expertise. Reframing Teacher Leadership to Improve Your School (Reeves, 2008b) provides a number of systematic comparisons examples at the elementary, middle, and high school levels.

Level 5: Preponderance of the Evidence

The apex of the research-type levels is when different researchers, operating independently using different research methods and working with subjects in different parts of the world, come to strikingly similar conclusions. For example, professionl learning facilitator Jenni Donohoo’s (2017) synthesis of research finds that teacher efficacy is strongly related to gains in student achievement. John Hattie and colleagues also cite collective teacher efficacy as a powerful variable related to student results (Waack, n.d.). My own quantitative analysis of more than two thousand U.S. and Canadian schools places teacher efficacy at the top of the influences of student achievement over the course of three years (Reeves, 2011b). Qualitative researchers who engage in deep observations and case studies have come to remarkably similar conclusions (Hargreaves & Fullan, 2012). When focusing on level 5 research, we move away from the claims of dueling experts. Some of the reasons teachers may be cynical about education research are because “You can always find an expert to say anything,” and “Today’s claim may be discredited tomorrow,” so they don’t know what to believe. But if several different researchers using different methods in different places substantiate a claim, we are no longer looking at anecdotes or isolated claims, but rather at the preponderance of the evidence. Such research provides compelling evidence for why educators should try implementing a particular change in their schools.

The original 90/90/90 research was a hybrid of what this chapter describes as Level 3: Collective Experience and Level 4: Systematic Comparisons research. But in the second decade of the 21st century, the research on success in high-poverty schools is firmly rooted in Level 5: Preponderance of the Evidence. Different researchers using different methods operating independently have come to very similar conclusions about the elements of success in these schools and, most importantly, the replicability of those findings.

Summary

Advocates of improved teaching and leadership practices are caught in a quandary. When they propound good ideas without evidence, they are guilty of vacuous rhetoric. But even when they have evidence from a variety of sources and methods, they meet the wall of opposition labeled, “But it doesn’t apply to me.” The same argument could perhaps be made against any variety of medical interventions, as the research on pharmaceutical and surgical interventions is all performed on someone aside from the patient for whom those interventions are now recommended. In these cases, patients are wise to discount the experience of a single other patient, but they can often be persuaded based upon a combination of systematic comparisons—patients who survived compared to those who did not—and the preponderance of the evidence. While all research is imperfect, the application of the standards of evidence suggested in this chapter can offer the reader a thoughtful method for separating the most credible research from the rest.