Читать книгу Achieving Equity and Excellence - Douglas Reeves - Страница 11
ОглавлениеCHAPTER 2
Decide Which Research to Trust
This chapter considers seven common challenges to educational research and offers a respectful reply to each challenge. Specifically, it considers seven arguments for research about success in high-poverty schools and how you might decide whether this research is worthy of your trust. The chapter concludes with a recommendation for locally generated research for the greatest level of credibility with teachers, administrators, and community members.
In the descriptions of the following seven arguments, I rely on extensive conversations with tens of thousands of teachers in fifty states and more than thirty countries. Whether the venue is Topeka or Tasmania, Louisiana or Lusaka, the arguments about research are strikingly consistent.
Argument 1: “The Research Doesn’t Apply to Us”
When the locations of the research are anonymous, it lacks credibility with many teachers and administrators because of the deep suspicion that the participants in the studies are vastly different from the practitioners in their schools. The situation is similar to much of the psychological research performed not on the general population, but on college sophomores taking a psychology class (the students fill out surveys or participate in experiments week after week, all in the pursuit of their professor’s publication). This is also one of many reasons so much psychological research is not replicable (Diener & Biswas-Diener, 2019). Similarly, if finding exceptional student performance results in the laboratory schools of universities and colleges, where students are often the children of exceptionally bright and committed graduate students and are only temporarily living on a low income, those results can hardly be labeled representative of the larger population of students from low-income families.
In the original equity and excellence research (Reeves, 2004), by contrast, every school was from public educational systems in the United States, and schools ranged in location from the West Coast (where there were also very high populations of students not speaking English at home), to the Midwest (where there was multigenerational poverty), to the eastern United States (where political upheaval, financial disasters, unemployment, and persistent social ills made for a difficult learning environment for students and a challenging working environment for teachers). None of the schools are exceptional, and none are university lab schools. Indeed, one of the most important elements when selecting schools for the study was they must be similar in every respect to the other unselected schools in the same system—the same union contract, per-pupil funding, teacher assignment policy, and neighborhoods. In other words, equity and excellence schools truly are representative of high-poverty schools across North America. The only differences are in academic achievement and the professional practices teachers and leaders employ to achieve those high levels of performance. It is also important to note that the programs in use did not distinguish the successful schools, but rather the differences were due to the specific actions of teachers and leaders. A consistent theme in the research is that practices, not programs, make the difference for student results.
It is important to regard equity and excellence research as the starting gate, not the finish line. Part II (page 29) of this book addresses what equity and excellence schools do differently, and part III (page 89) describes how they do it, but equally important is the information in part IV (page 137), which requires the continual assessing of implementation through accountability systems. In order to institutionalize the most effective practices for your school or district, it is imperative for you to create a continuous cycle of professional learning that links the causes of academic achievement with the effects. Only in this way will you know, based on data from your students in your school and community, the professional practices most effective for you. I believe equity and excellence research makes an effective case that the practices in the following chapters are strongly associated with improved student achievement. However, I readily acknowledge that the most effective way to sustain effective practice is not merely by reading the research of others but also by committing to local observation and research to overcome the objection, “We are different.”
Despite efforts to match equity and excellence schools in other low-income schools across North America, it is certainly true that no external sample of schools will precisely mirror the characteristics of the school or district in which you work. Every sample, whether from a few schools the researcher chooses as a sample of convenience or a larger sample that closely mirrors the characteristics of students in the original research, is limited because practitioners contend, “That sample did not include my students and my school.” Indeed, even when the sample does address this challenge by including “my students in my schools,” skeptics could contend the sample used last year’s students and this year’s students are different.
The only response to this challenge is not to argue over the representative nature of samples, but to shift the research focus from external to internal. In Reframing Teacher Leadership to Improve Your School (Reeves, 2008b), I offer a method for teachers to conduct and assess action research that has, in my experience, been transformative for promoting effective and sustainable change. This method is colloquially referred to as the science fair approach, as the method mirrors the three-panel cardboard displays fourth graders often use for their science fair projects. Although the research topics vary widely based on the individual needs of each teacher and school, the format of these three-panel cardboard displays is consistent.
1. Left panel: Challenge
2. Middle panel: Professional practices
3. Right panel: Results
Here are some examples from schools I observed.
• Challenge: High school failures due to missing work (more than 90 percent of D and F grades were due to missing work, not attendance or behavior).
• Professional practices: Teachers implemented a daily required intervention immediately before lunch. All students used the same agenda, and all teachers agreed to use a stamp system to indicate when students completed the work. When students were missing a stamp, they were directed to the appropriate room to complete the work.
• Results: D and F grades declined by 67 percent in one semester.
• Challenge: Middle school behavior was out of control and suspensions at an all-time high.
• Professional practices: Teachers implemented restorative justice schoolwide and agreed on a chart of responses for what is required for in-class, in-office, and out-of-school discipline.
• Results: Suspensions decreased by 55 percent in one year.
• Challenge: Chronic absenteeism in elementary through high school.
• Professional practices: Teachers implemented the sixty-second report—all students not seated in the classroom within one minute of the tardy bell were added to a list in the principal’s office and called within the first twenty minutes of school. All staff members who were not in front of students attended a stand-up meeting in the principal’s office.
• Results: Absenteeism decreased by more than 80 percent in one year.
• Challenge: Excessive middle and high school failures due to missing work and student disengagement. Students who accumulated zeros on the one-hundred-point scale quit trying because they knew no matter how hard they worked, they would fail the class.
• Professional practices: Teachers changed from the one-hundred-point scale to a simple A–F grading system, where A = 4, B = 3, C = 2, D = 1, and F = 0. Teachers also switched from calculating the final semester grade based on the average of all work to giving a final grade based on the teacher’s judgment of student proficiency from the latest and best evidence of student learning.
• Results: D and F grades decreased 38 percent in social studies, 45 percent in English, and 62 percent in mathematics.
The consistent element of the science fair approach is that it uses local research with local students in local schools—your students in your school or district. Instead of considering a different sample of students from a different school, no matter how representative, the science fair approach allows teachers to compare the same students from the same neighborhoods with the same demographics with the same teachers within the same year. This compelling before-and-after approach allows teachers to conclude that all the other factors influencing student achievement are consistent—the only change is the teachers’ changes in professional practices.
Argument 2: “Anecdotes Are Not Evidence”
The assertion, “Anecdotes do not equal evidence,” is a very fair criticism, as many articles and books in education are best described as journey stories—the experience of a single teacher or administrator. However informative these experiences may be, they are anecdotes, not research. This is why we must all be critical consumers of educational research when a speaker or writer blithely claims, “Studies show …” or “Research says …,” when the studies or research may only be a sample size of one. This does not eliminate the value of case studies, but it is much more helpful to draw inferences when researchers accumulate a large number of cases. Certain educational researchers are leading the way in studying the successful practices of multiple school districts and condensing the results into a format educational leaders and teachers can implement in their schools. For example, Heather Zavadsky (2009), director of research and implementation at the Texas High School Project, analyzes the workings of several well-run school districts in her book Bringing School Reform to Scale: Five Award-Winning Urban Districts. Reflection on a single high-performing urban system is not nearly as helpful as Zavadsky’s (2009) synthesis of a variety of school systems and her ability to find common elements despite differences in geography, governance systems, funding, and student populations.
Using the science fair approach discussed previously, teachers may think their own work is “just an anecdote” and therefore not worthy of being shared with colleagues. But when that individual experience is grouped with dozens or hundreds of colleagues’ experiences, then patterns can emerge that no longer rely on anecdotal evidence. Moreover, a focus on teaching practices allows educators to distinguish between programs and practices. Vendors would like to claim a particular program, curriculum, or technology application leads to gains in student achievement, but programs alone accomplish nothing. The overwhelming conclusion of our review of more than two thousand school plans (Reeves, 2011a) is that it is practices, not programs, that hold the key for improvement in student results. In addition, a focus on practices turns the analytical lens where it belongs—on practices that are replicable rather than the mystical qualities of the individual teacher. Kim Marshall (personal communication, September 16, 2019), educational researcher and author of The Marshall Memo website (https://marshallmemo.com), thoughtfully distinguishes between teachers and teaching by noting that when the focus is on the teacher, great practice is relegated to the ethereal realm (“She’s just a gifted teacher!”) or calumny (“He’s just a terrible teacher!”). Neither of those observations is particularly insightful guidance for any professional beyond the superficial “Be good and don’t be bad.” But, as Marshall suggests, when we focus on teaching—the actual practices in which teachers engage—then we can address specific practices in discipline, feedback, curriculum, lesson planning, and professional responsibilities to perform effectively. The guidance is not, “Be more like Ms. Smith,” but rather like the following.
• “It’s important that you move around and are close to the students, not remain behind your desk.”
• “It’s important that you give students immediate feedback during class, not just on their papers several days after they did the work.”
• “It’s important that you engage every student by using whiteboards, cold calling, or similar techniques, and not merely recognize students who raise their hands.”
These concrete practices focus on what teachers actually do, not who they are as people. Moreover, these are practices coaches and administrators can model in real time, showing their colleagues in the classroom that no matter one’s responsibility in the school, they are all teachers and must be willing to show they still have the ability to demonstrate to colleagues and students the importance of effective teaching practices.
Argument 3: “The Research Depends on Heroic Teachers and Administrators Whose Efforts Are Unsustainable”
Movies about high-poverty-turned-high-performance schools often feature a particularly resourceful, motivated, or heroic teacher who enters the classroom and leaves masterful change in his or her wake. Moviegoers see this in films like Stand and Deliver (Musca & Menéndez, 1988) or Freedom Writers (DeVito, Shamberg, Sher, & LaGravenese, 2007). Although the storylines are compelling, the teachers in these moving tales do not represent the average teachers who, thrust into a high-poverty school often with inadequate preparation and support, are fighting to maintain a modicum of discipline, stay one page ahead of the students in the curriculum, and learn the craft of teaching.
I acknowledge there are exceptional teachers in successful high-poverty schools, but the equity and excellence research deliberately avoids these out-of-the-ordinary cases. The schools we learn the most from have the same teacher assignment policy, same union contract, same per-pupil funding, and in many cases, same school and classroom architecture as their less-successful counterparts. Great teachers make for compelling stories, but greater teaching is the only credible source of replicable and sustainable practice.
Argument 4: “Publishers’ Research Is Commercially Tainted”
The history of commercially tainted research in education is a long one. Just as purveyors of sugar- and processed-food-funded research purport to show their products are healthy—research that led to a multigenerational increase in obesity—so also do the sellers of video games attempt to show these are indispensable tools for student learning (Egenfeldt-Nielsen, 2006). This challenge is certainly not limited to the field of education. Although one would think that medical research is at the apex of credibility, the fact is, commercially funded research is more likely to be published than studies of higher quality and free of commercial bias (Hogan, Sellar, & Lingard, 2015; Lynch et al., 2007).
There is a significant burden on school administrators and other decision makers to be critical consumers. They must understand the nature of the research to influence purchasing decisions and the degree to which that research is applicable to the conditions of the schools and districts of the purchasers. One of the most frequently misunderstood terms is significance, which, in the context of research, almost always means statistical significance. In simple terms, the differences between two groups are considered statistically significant if an analysis of those differences shows they are unlikely (less than a 5 percent chance) to be different due to random variation. For example, if a group of students who participate in a particular instructional reading intervention score 79 percent on a test, and another group of students who did not participate in that reading intervention score 75 percent, then researchers can compare those two groups and, based on the number of students in the groups and the variation in their scores, determine that the difference between 79 percent and 75 percent is unlikely due to randomness. But that is a very different proposition than saying the reading program caused the students in the first group to score higher. Medical researchers sometimes use the term clinical significance to distinguish a finding so important it is worth changing one’s practice. Although there are many treatments, pharmaceuticals, and practices associated with statistically significant differences, most of those changes are insufficient to lead doctors to prescribe a different drug or use a different treatment modality (Leyva De Los Rios, 2017). This is because for every treatment a physician begins, there is usually another treatment that must be withdrawn. Similarly, starting a new reading program usually means withdrawing an old reading program. Therefore, the decision maker must consider not only the impact of adding a new program but also the impact of withdrawing the old program.
The failure to distinguish the importance of clinical significance compared to statistical significance was first illustrated during the Race to the Top era (2009–2017), during which federal grant incentives led districts to pile one program on top of another. Each program might have been significant when compared with no program, but almost none showed value when they were simply part of a constellation of many duplicative programs. I observed schools with three different data-analysis protocols, two different mathematics programs, and seven different literacy programs, which all vied for the time and attention of teachers who, not surprisingly, were unable to implement any of these new and expensive programs well. There were no grants for educational leaders who decided to stop doing something. My research (Reeves, 2011b), based on more than two thousand school plans, demonstrates that when schools have more than six instructional initiatives, student performance declines, even as those schools spend more money and acquire more programs. Moreover, the most fragmented schools—those burdened by more and more programs—are most likely to be schools with high percentages of students from low-income families, high percentages of students who are learning English, and high percentages of students with special needs (Reeves, 2011b). In brief, the schools most needing focused leadership are the least likely to have it. It is no wonder the programs claiming “significance” in the laboratory or another controlled setting did not exhibit similar results in the real world of teachers and students overwhelmed by multiple demands on their time from many different programs.
To be clear, I am not suggesting that the differences between the research claims of the advocates of educational programs and the reality teachers and administrators experience are due to malice or corruption on the part of vendors. Rather, I am making the observation that the environment the research salespeople cite may be substantially different from the environment of the practitioners who actually try to use these programs in the real world. Therefore, it is essential for people making buying decisions to inquire about the actual environment of the research. Moreover, major decisions about curriculum, assessment, instructional practices, leadership techniques, and financial commitments are always better informed if leaders follow the discipline of mutually exclusive decision making (Lafley, Martin, Rivkin, & Siggelkow, 2012). Good leaders can make bad decisions if they fail to practice the fundamental disciplines of gathering information and considering alternative hypotheses (Campbell, Whitehead, & Finkelstein, 2009).
In order to be more critical consumers of research, leaders must persistently ask, “If I am going to decide to implement X, then what will I give up—in time, money, and professional energy?” One of the great traps in this line of inquiry is the myth that because a grant funds a new initiative, it is therefore free. But no decision is free. Even if there is no impact on the budget, there is definitely an impact on time and attention. Because leaders cannot monitor and focus on more than about half a dozen major initiatives, every additional initiative beyond that threshold, even if it appears to be cost-free, not only takes a toll on the leader’s time and attention but also encroaches on every other initiative in the system.
The only remedy for this is organizational and leadership focus. Effective strategic plans are not merely an accumulation of programs and tasks to be implemented. Rather, strategy is also the art of deciding what not to do. My experience suggests that the primary complaint that teachers have is time. They are intelligent and hardworking, but simply overwhelmed with the sheer quantity of tasks they are expected to accomplish. Therefore, leaders recognize that time is a zero-sum game—every hour allocated to one task is an hour not available for another. The focused leader who, for example, wants to encourage collaborative scoring of student work in order to deliver consistent expectations of students and reliable scoring by teachers provides time in staff meetings and collaborative team meetings to accomplish those important tasks. That means that the focused leader is deciding not only that collaborative scoring is vital but also that competing activities in those meetings—like the primitive practice of making verbal announcements—will be discarded. While many leaders claim to value focus, few can articulate how they will save time by discontinuing announcements, stop the expectations that texts and emails be responded to within minutes of receipt, and ban classroom activities—such as twenty-year-old word search puzzles—that have zero educational value. When leaders decide what to stop doing, teachers know that they and their time are respected.
Argument 5: “Results Are Distorted Because the Criteria for Success Are Too Low”
Many studies of high-poverty schools’ success sometimes receive criticism because readers claim the schools’ standards for success are too low and not reflective of success in the real world. For example, the New York Regents exam provides a four-point scale for students, and a score of three or four is regarded as passing (Pondiscio, 2019). It could be argued that the bar should be higher, but the plain fact is that only a minority of urban schools in New York meet the standard of a three. In the original equity and excellence research (Reeves, 2004), the criterion for meeting standards is only at the basic level, which was the criteria at that time used by the state. Some critics have approached me in meetings and argued that this bar is too low to count as success. But in that review of 135 high-poverty schools, only seven met the basic criteria. Few people argue against setting the bar high for student achievement with classroom expectations to match, but when only seven of 135 schools meet a criterion, it seemed to me that it was, to put it mildly, evidence of comparative success. Thus, these U.S. schools met the state criteria at a far higher level than most schools with similar demographic characteristics. But the criticism is nevertheless well taken. In many states, students can score only 40 percent of the answers on the test correctly and still be labeled proficient. Part of the flaw in descriptions of proficient is that proficiency is a moving target. Even in states claiming a commitment to standards-based education for more than two decades, some change the cut scores—the percentage of correct answers the state deems adequate—every year. If too few students and schools do well, then the state lowers the cut score. If too many of the students and schools do well, then the state raises the cut score. This procedure is precisely the opposite of standards-based assessment. Safety professionals do not, for example, relax or strengthen the criteria for left-hand turns for teenage drivers or safe landings for pilots based on annual variations in other drivers’ or pilots’ performance. The standard is the standard. This is also a reason the best and most reliable measurements of student achievement are based on consistent criteria during the same year in a class with largely the same students, teacher, curriculum, and assessments. Although this emphasis on consistency is imperfect, it is far superior to attempts to draw inferences about student success when the tests and criteria for proficiency change from one year to the next, and when the students compared are also different from one year to the next.
In the absence of a national assessment of student performance, accompanied by a systematic analysis of teaching and leadership practices in every school, the best data that we have is that provided by districts and states. This leads to inevitable variation about what success really means. While that is a legitimate concern, it does not deny the fact that when the same assessment is given to a wide variety of students in the same subject and same grade with the same socioeconomic status, some do better than others. The explanation is neither money nor zip code, but teaching and leadership practices. The United States does provide the National Assessment of Educational Progress (NAEP), labeled the “Nation’s Report Card,” but it offers nothing in the way of school-by-school analysis of instructional and leadership practices.
Argument 6: “The Funding Is Higher for Successful Schools”
It is true that successful high-poverty schools often have higher levels of funding than other schools without high populations of students from low-income families. This is almost always due to their eligibility for Title I funds, which the U.S. Department of Education allocates to all high-poverty schools. One goal of equity and excellence research was to ensure the findings would transfer to all schools, so our methodology included carefully monitoring the levels of per-pupil funding. By doing this, the successful and unsuccessful schools under review had nearly identical per-pupil funding. As a result, readers can implement the findings with confidence that the differences in success are, in fact, due to changes in teaching practices and not due to a particularly high level of funding.
Because federal Title I funds are distributed on the basis of the percentage of students eligible for free or reduced-price lunch, it is not unusual to find high-poverty schools and districts with higher levels of per-pupil funding than schools which are not eligible for Title I funds because they serve a more affluent population. On the other hand, more affluent schools benefit from a higher residential property tax base, and therefore enjoy economic advantages. But whatever the source and rationale for funding, the consistent findings of the research cited earlier in this book is that while money is important for schools, money alone is not the variable that determines student success. High-poverty schools benefit not only from Title I funds but also from a plethora of grants and special allocations designed to support one initiative after another. This funding creates the illusion of prosperity because these schools are flooded with people and programs all designed with good intentions. But the reality is the schools are fragmented in so many different directions that administrators cannot monitor effective implementation of the programs, and the teachers do not have time to focus on any single program in order to provide effective implementation. The key to success is not funding; it is in the specific implementation of carefully selected teaching practices.
Argument 7: “Successful Schools Cherry-Pick Students”
An argument frequently levelled against magnet and charter schools is that some, but by no means all, have higher levels of success because they can select, or cherry-pick, their students. Yes, the critics argue, a high percentage of students might be from low-income families (Bambrick-Santoyo, 2018; Pondiscio, 2019). But if the school has a low percentage of special education students or students with significant discipline problems, mental health issues, or learning disabilities, then it cannot be compared to schools with similar student demographics that have significantly greater percentages of students with these same needs. This is a very fair concern, and it is a reason reviewers of successful high-poverty schools case studies should ask questions about the extent of the special education population—Is it similar to or different from the schools it is being compared to? Even if the allocation of students with special needs is identical, however, there is no question that students in magnet and charter schools have parents who made the effort to enter a lottery or otherwise advocate for their children to attend a particular school. As Pondiscio (2019) acknowledges, successful charter schools may not cherry-pick students, but they certainly cherry-pick parents, as the lottery systems on which most of them depend require parents to submit applications, take an interest in their children, and in some cases, make extra efforts to ensure on-time attendance and help students adhere to strict discipline and academic policies that are not always present in traditional public schools. That is almost never the case for children in homeless shelters, living with adults who are not their parents or who otherwise do not take an interest in education. These children are generally left to their own devices for entry into selective schools. The same concern is true for exam schools, where (in Boston and New York, for example) the score on a single exam decides who will be admitted to Boston Latin School or The Bronx High School of Science and who will not (Gay, 2019a; New York Times Editorial Board, 2019). Although the demographics may be similar to non-exam schools, there is no question that if a school starts with the top 1 percent of students in exam-taking ability, that school will show higher degrees of success when the measurement is, most commonly, exam-taking ability.
Of all the criticisms of the research on successful high-poverty schools, the issue of cherry-picking is of the gravest concern, particularly for successful charter schools. While lotteries generally ensure that these schools have comparable percentages of special education students to traditional public school, the challenge of cherry-picked parents is real and legitimate, as even the staunchest defenders of charter schools acknowledge (Pondiscio, 2019). But there are three essential responses to this challenge that teachers and leaders in any school, regardless of label or governance structure, must consider. First, the evidence on successful high-poverty schools is certainly not limited to charter schools or others that are otherwise selective. The more than a dozen sources cited in the introduction (page 1) and chapter 1 (page 9) provide many examples of success far beyond charter schools. Second, and more to the point, so what if successful schools engage parents more directly and benefit from discipline and academic policies that parents wholeheartedly endorse? That is not an argument that these schools are failing or that their policies are inadequate, but rather that all schools—traditional public, charter, and private—can learn from the teaching and leadership practices of their counterparts. Third, and most importantly, the most successful charter schools, such as New York’s Success Academy, can be compared to schools that actually do cherry-pick students—that is, the designated gifted and talented schools. Under this sort of comparison, the students selected by lottery in the Success Academy Charter Schools significantly outperform the highly selected—cherry-picked, if you will—gifted and talented schools (Pondiscio, 2019).
Summary
This chapter considered seven criticisms of research about success in high-poverty schools. To be clear, I do not challenge the motives of critics, as I fundamentally believe that almost everyone who enters into the national and global discussion about how to improve education has the best interests of students at heart. Nevertheless, this book has a clear point of view, and I want to respectfully acknowledge and respond to the most common criticisms of the research on successful high poverty schools. Critics frequently contend that successful high-poverty schools are different—whether due to their funding, students, or exceptional teachers—and therefore, their results cannot be generalized to conditions in unsuccessful schools. Many of the articles, books, and films about successful high-poverty schools are no more than anecdotes consisting of a sample size of one, and not generalizable. Some research, particularly regarding commercial programs, curricula, and technology, may be commercially tainted if the organization sells and administers those programs. Some successful high-poverty schools are based on criteria that are too low; we ought to expect more from all students. The funding for some successful high-poverty schools may be higher than the funding for unsuccessful schools. Finally, successful high-poverty schools, particularly magnet and charter schools, may cherry-pick their students, either by excluding high-needs students or by relying on parents who take an active interest in their children’s education, factors that are not always the case in the unsuccessful high-poverty schools.
The best response to these criticisms is to acknowledge and respect them, be aware of them when conducting research and presenting findings, and do the best job possible to take these potential criticisms into account when conducting future research. Most important, good researchers acknowledge the limitations of their work, as I have sought to do in this chapter. That said, acknowledgment of potential limitations does not invalidate the research findings. This book describes not a single student or case, but rather the preponderance of evidence from a variety of scholars. This is the apex of research quality readers can rely on when making decisions about which teaching practices to implement in their own schools.
Prepared with an excellent understanding of educational research and when it can (or cannot) be trusted, you are ready to delve into the fruits of research on successful high-poverty schools—the teaching practices that high-performing, high-poverty schools use differently. The following section will discuss seven teaching practices of equity and excellence schools you can use and implement in your own schools and classrooms starting right now.