Читать книгу Economic Evaluation in Education - Henry M. Levin - Страница 26

When identifying the problem and the alternatives that might address the problem, it is important to be clear about the audience or audiences for whom the analysis will be done. The audiences include one or more groups of stakeholders—that is, individuals or institutions with an interest in the outcomes of the evaluation process. It is helpful to think of a primary audience and a secondary audience. The primary audience is generally the decisionmaker (and the clientele whom he or she represents) who has requested the analysis. The secondary audience consists of those persons and groups who will also draw upon the analysis.

Since the analysis is being prepared explicitly for the primary audience, it is important that it meet the specifications of that audience. For example, if one is requested to do a cost-feasibility (CF) study of using educational television in the school curriculum, it is important to ascertain exactly what is behind the request. Is there a specific technological or curriculum approach that the decisionmaker has in mind, or is the charge to be concerned with the costs of a wide variety of approaches, from selected courses to full curriculum coverage? These details can be worked out through dialogue and specification of the issues, and the analysis and report should be written with the needs of the primary audience in mind. Nevertheless, there may be tension if the decisionmaker places certain alternatives beyond consideration even though the alternatives may be in the best interests of the constituents represented by that decisionmaker. For economic analysis, there is an extra tension: With an impact evaluation, there is a risk that the intervention will not be effective; with an economic evaluation, there is an added risk that, even if it is effective, the intervention will not be found to be efficient (Levin, 2001). Hence, it is important to structure the analysis so that all reasonable alternatives are evaluated in the same way and so that the research conclusions are not predetermined.

Often, however, economic reports are read by secondary audiences who wish to use the information to inform decisions regarding replication or scale-up in another setting. If a study will be restricted to its primary audience, one need not be concerned about secondary audiences. However, if a secondary audience is likely to utilize the study, it is important to be clear about what is possibly generalizable to other settings and what is not. For example, in reviewing the costs of instructional programs, a school district may generate cost analyses that exclude resources contributed by the state or other levels of government or by volunteers. For purposes of decisionmaking in the district under scrutiny, this omission may be understandable. But if other school districts attempt to apply those cost analyses, they may discover that they have allocated far too few resources to support such instructional programs. Not every other school district in that secondary audience will have equal access to governmental or volunteer resources. In general, it is preferable to report all costs such that any audience can identify the resources appropriate to their situation no matter who is providing them.

Potentially, there are many audiences. For example, a district may be interested in improvements in teaching English and reading to students from non-English-speaking backgrounds. Among the alternative policies are English as a second language (ESL), bilingual instruction, and total immersion in English. Before performing CE analysis to determine which policy to recommend, the analyst might identify several primary and secondary audiences. The primary audiences—that is, those who must use the results directly for their own decisionmaking—might include the school board, the district administrators, the state education agency, and the pertinent curriculum specialists and teachers. Given the many audiences, it may be useful to interview representatives to ascertain the types of information that they would like to obtain from the evaluation. The secondary audiences—that is, those with an interest in the results—might include non-English-speaking local residents, employers and residents of the district, and other school districts. These audiences may have different needs from the primary audiences. Again, setting priorities is important, since no evaluation endeavor is likely to meet all the needs of every potential primary and secondary audience.

Once the basic outlines of a cost study are agreed upon, the essential next step is to decide whether to perform CE or benefit-cost (BC) analysis. Either analysis may be appropriate. It depends on what decisions will be made as a result of the analysis. For example, imagine a nonprofit agency whose mandate is to improve literacy in kindergarten. Given the agency’s objectives are already established, it would seem that CE analysis would be most appropriate; this would inform the agency which literacy programs are the best value. By contrast, BC analysis helps establish whether it is worth investing in literacy programs at all. Presumably, the agency—in deciding its mandate—had already determined that literacy programs are worth investing in. However, BC analysis may be useful: It can clarify how valuable literacy programs are and so justify—both within the agency and to its donors or funders—further investments in literacy programs. As emphasized previously, the motivating question is this: What decisions will the audience make when presented with the information from economic analysis?

Audiences may be more or less willing to accept certain kinds of analyses that nonetheless yield similar conclusions. For example, BC analysis is less palatable to some audiences because it requires placing a monetary value on certain kinds of outcomes, effectively modeling child learning in terms of its monetary benefits. Although they are different, there is scope to substitute between CE analysis and BC analysis. One could imagine a program that is aimed at reducing high school dropouts. Since the program is designed to reduce the number of high school dropouts, an obvious measure of effectiveness is the number of dropouts averted. Eventually, of course, staying in high school might yield monetary benefits in the form of additional wages. We can conduct either CE or BC analysis, depending on the outcome measure that is chosen (dropouts averted or increased wages). It is conceivable that the CE analysis will be more warmly received by some audiences, because it does not attempt to describe the outcomes of the program in pecuniary terms. If CE and BC analyses do produce similar conclusions for decisionmaking purposes, it seems prudent to choose the analytical technique that is most likely to be well received by the primary audience. However, choosing to conduct CE analysis rather than a more comprehensive BC analysis might entail sacrifices in terms of the depth of analysis. In these cases, the researcher is best advised to allow the demands of the analytical task to guide the choice of analytical technique rather than the demands of the audience.

Thus far, we have described the audience simply as if they are readers of the research evaluation. However, their role is much more influential because they can shape the perspective of the analysis. The perspective is the framework within which costs and effects or benefits should be examined. Thus, the perspective dictates which items should be included in the analysis.

For educational evaluations, we can distinguish three important perspectives. Depending on which perspective is adopted, the analyst will have to calculate the costs, effects, and benefits that correspond to that perspective.

The primary perspective is the social perspective: This requires that all costs should be counted, regardless of who pays for them or even if they are provided in kind, and all benefits should be included, regardless of who accrues them. This social perspective is therefore all-encompassing, and other perspectives can be understood in relation to the social one. Indeed, it makes sense to start with a social perspective and then specify alternative perspectives within this general one. As discussed previously, the audience for the analysis may prefer a perspective that only helps with their decisionmaking and so excludes some costs (or even some benefits). However, this preference may be narrow-minded: If the goal is to maximize social welfare, the decisionmaker should be interested in all the resources used. This holds even when the decisionmaker uses contributed resources; these contributed resources might be used in an alternative way if their current use yields low benefits. Importantly, volunteers who contribute their time should want to know if their contribution—as part of an educational investment—is socially valuable. Volunteers for mentoring programs such as Big Brothers Big Sisters of America, for example, would presumably switch to an alternative activity if they learned that this program generated only small social benefits (or invest more if they had evidence on the social value they were creating). Finally, in order to compare interventions, it is helpful if all evaluators adopt the same perspective, and the social perspective can be adopted for all interventions.

The next perspective is the private individual perspective: This requires calculation of all the costs and consequences for the student (or the student’s household) or participant in the education program. Given that many education programs are subsidized by government and other entities, private individuals may reap substantial benefits but pay little of the costs. As such, from a private perspective many educational interventions should appear to be good value. As students are likely to choose the intervention with the highest private returns, this perspective will help explain why students participate in particular programs.

Finally, the analyst may adopt a fiscal perspective—that is, looking only at the costs and consequences from the perspective of the taxpayers or a particular government agency. This perspective is important as a way of justifying public investments in education programs. Government agencies may—by statute—be bound to consider only the implications for their agency. The challenge with using this perspective is that many educational interventions are funded from several sources, have diverse effects, or convey benefits to many individuals and in many domains. Adopting a fiscal perspective would provide a very partial picture for each agency. Nevertheless, some agencies might justify educational investments purely on these narrow grounds: Given the evidence on the effect of education on health, for example, it might be reasonable to label the high dropout rate in the United States as a “public health issue” (Freudenberg & Ruglis, 2007).