Читать книгу Evaluation in Today’s World - Veronica G. Thomas - Страница 154

1 What social, political, economic, and racial issues were happening in the country that might have influenced the conceptualization, design, and outcome of the evaluation?

2 If you were the evaluator of this project in 1969, what would be the top two historical factors that you would consider during your work? Why?

Unfortunately, evaluation findings from many other high-profile social programs yielded dismal measurable results. This resulted in some rethinking of evaluation and its purpose, methodologies, and problems. Edwards, Guttentag, and Snapper (1975) offer what they refer to as five complaints about folkways of evaluation research that were the prevailing sentiment of the time and resulted in more rethinking of evaluation’s purpose and methodology. These include reification of programs; insistence on causal inference; conducting pseudo-experiments; planning the formative, then summative, evaluation in a stage-by-stage sequence; and a baseball statistical approach to handling lots of data. Table 3.3 elaborates on their complaints.

Despite some successes, one of the most important lessons to be learned from all the evaluations initiated during the 1960s and 1970s was the difficulty in carrying out randomized controlled experimental designs in field settings. Due, in large part, to the dismal findings from large-scale randomized controlled trial evaluation studies of the late 1960s through the end of the 1970s, many evaluators became disheartened with the traditional approaches to evaluation. In fact, a reasonable summary of the large-scale field experimental evaluation findings of the 1960s and 1970s is that the expected value of the effect of any program hovered around zero (Rossi & Wright, 1984). Of course, such a conclusion was quite disheartening to the social reformers who had hoped that the Great Society programs would result in significant improvements in the lives of poor people and marginalized communities. Given this reality, evaluators had to contend with two possibilities: (a) that the implementation of social programs of the day truly had little to no impact on the intended outcomes for their participants, or (b) that the existing evaluation models and methods being used were inadequate to capture the outcomes of interest.

Table 3.3

Source: Edwards, Guttentag, & Snapper (1975).

With the evaluation challenges of the 1970s and 1980s, there were increasing concerns among evaluators, particularly around how evaluation was viewed and its methodology. L. Ross and Cronbach (1976), for example, emphasized moving away from the mainstream view of evaluation that characterizes evaluation as an event that begins, runs alongside a program for a time as the evaluator makes observations and collects data, and ends rather abruptly with a report to an all-powerful decision maker (usually someone outside the program under consideration). In this mainstream view, the evaluator, essentially, enters the picture only after the initial events in the life of the program have begun. Instead, they argued for a better approach to evaluation, whereby evaluation becomes a component of the evolving program itself rather than simply providing “disinterested monitoring undertaken to provide ammunition to the warring factions in a political struggle” (L. Ross & Cronbach, 1976, pp. 18–19).