Читать книгу An Introduction to Evaluation - Chris Fox - Страница 7

Introduction

Who should read this book?

This book has been written with two main audiences in mind.

The first group is students, in particular postgraduates doing taught courses in research methods and both undergraduates and postgraduates planning research projects, perhaps for a final year dissertation or a PhD.

The second group is people working in governmental or non-governmental organisations who are planning either to undertake an evaluation or commission an evaluation.

Why evaluation is important

Evaluation is a fascinating interdisciplinary research practice with a wide spectrum of applications. Evaluators continuously encounter new challenges; they work in changing environments, with a variety of people and organisations where they have to think themselves into new social settings. No single evaluation is like another and evaluators need to pragmatically develop innovative and robust research methods.

At a time of increasing calls for transparency and accountability, evaluation has moved centre stage in public policy, organisational planning and management. Government departments are under pressure to deliver social policy programmes that are effective and value for money. Chelimsky (2006) suggests that evaluation is an intrinsic part of democratic government for four reasons:

It reports information about government performance that the public needs to know. It adds new data to the existing stock of knowledge required for government action. It develops an analytical capability within agencies that moves them away from territoriality and toward a culture of learning. And, more generally, its spirit of scepticism and willingness to embrace dissent help to keep government honest. (Chelimsky 2006: 33)

Similarly, non-governmental organisations from Amnesty International to the Red Cross are accountable towards their funders and have to prove that they use donations in an appropriate way. Private sector organisations too are increasingly evaluating internal processes and operations. The evaluator has an important role in assessing and judging programmes, polices and interventions, and evaluation reports often determine the future of interventions.

Evaluation combines managing the expectations of different stakeholders – those who are affected by an intervention, those who help to deliver the intervention and those who have been funding it – with understanding the social complexities and remaining objective and neutral. While handling various stakeholders is challenging, understanding diverse points of views is also insightful and rewarding. Evaluation research requires multiple skill sets from project planning, team building and stakeholder management to the design of interdisciplinary and multi-method research instruments. It is this diversity and the applied nature of evaluation research which make it an exciting and stimulating task.

Good evaluators are not just expert social scientists, they are also expert negotiators and project managers. They have a deep knowledge of social science method and theory, but are also practical and politically astute.

Our approach to evaluation

A practical discipline

Methodological rigour, theory development and a coherent philosophy are all important in evaluation, but above all it remains a practical discipline. Academic debate about the merits of different evaluation paradigms, types of data collection or approaches to analysis must be tempered by the needs of different stakeholders, including funders, policymakers and service users. Unlike some other types of social science research, evaluators almost always work within practical constraints such as fixed budgets and deadlines.

Evidence-based (or informed) policy and practice

One of the distinguishing features of evaluation is that it is undertaken to improve programme effectiveness and/or inform decisions about future programmes (Patton 1997). However, this raises lots of challenges and questions for evaluation, including the competing needs and interests of various stakeholders and the extent to which evidence can or should influence policy and/or practice. Effective planning, good project management and a proper strategy for knowledge mobilisation are just as important as a well-designed evaluation framework and the appropriate use of research methods.

Multi-sector, multi-disciplinary

Evaluation has an established or growing profile across a wide range of sectors including education, welfare, health and social care, criminal justice and international development. Different approaches to evaluation are prominent in different sectors. In writing this book we have tried to draw on examples from a range of sectors in order to illustrate the range and diversity of evaluation theory and practice and encourage cross-pollination of ideas.

Complexity

Possibly the biggest challenge facing evaluators today is how to respond to complexity. We think of complexity in two senses.

Firstly, across a number of sectors there is increasing recognition that service users often have multiple and ‘complex’ needs and that services often have to be organised that involve collaboration between various organisations. Here we are recognising that people and programmes are complicated. So, for example, a housing need may be linked to a substance misuse need that is, in turn, linked to a mental health need. Alternatively, an individual child’s educational needs may be interlinked with a broader set of health and social care needs within their family and those needs may be intergenerational. These needs require multiple delivery partners working across different sectors and service delivery often takes place within a fast-moving policy environment where change is the norm.

Secondly, we also think of complexity in a sociological sense. This goes beyond the idea that programmes are complicated and embraces the concept of complexity (Glouberman and Zimmerman 2002). Complexity includes the principle of non-linearity (small changes in inputs may, under some conditions but not others, produce large changes in outcome); the contribution of local ‘adaptiveness’ and feedback loops; the phenomenon of emergence; the importance of path dependence; and the role of human agency (Marchal et al. 2013).

In both of these senses, complexity poses challenges for evaluators seeking to explain how interventions work, what impacts they have, and whether they make economic sense.

What does this book cover?

This book includes:

 practical steps for designing and conducting an evaluation and ensuring that the evaluation is ethical

 different approaches to evaluation, including theories of change, process evaluation, impact evaluation (outcome evaluation) and economic evaluation

 ways of reviewing evaluation evidence including systematic reviews

 how to get evaluation knowledge into policy and practice

 important philosophical questions about evaluation and its role in society

The book is divided into several sections.

Part I: Getting Started

This section introduces evaluation, providing a brief overview of the field and highlighting some key issues to address before staring an evaluation.

Chapter 1 introduces the field of evaluation and provides an overview of different types of evaluation and the historical development of the discipline.

Chapter 2 considers the ethics of evaluation, both the design of ethical evaluations and ethical evaluation practice.

Part II: Undertaking an Evaluation

This section covers common components of an evaluation. Not every evaluation will use all these components but most evaluations will use some of them.

Chapter 3 introduces Theories of Change, an important tool for evaluators and programme developers and an approach that can be integrated into many evaluations.

Chapter 4 discusses process (implementation) evaluation.

Chapter 5 discusses impact (outcome) evaluation.

Chapter 6 discusses economic evaluation.

Chapter 7 introduces a range of quantitative and qualitative methods for data collection and analysis commonly used in evaluation. Mixed methods (dual strategies) are also considered.

Part III: The Practice of Evaluation

This section provides practical guidance on the practice of evaluation.

Chapter 8 gives practical guidance and advice on planning an evaluation, with a particular focus on refining evaluation questions and engaging with different stakeholder groups.

Chapter 9 provides practical guidance and advice on conducting an evaluation, with a particular emphasis on the importance of good project management.

Part IV: Using Evaluation Findings

Over recent years, increasing emphasis has been placed on the use of evaluation findings.

Chapter 10 describes systematic reviews, which are used increasingly to synthesise evaluation findings for policymakers and practitioners.

Chapter 11 gives practice guidance on disseminating evaluation results and also discusses how to include evaluation evidence in policy and practice (knowledge mobilisation).

Part V: Evaluation Paradigms

Chapter 12 provides an introduction to some of the philosophical debates that underpin evaluation, including a discussion of three commonly used evaluation paradigms.

Chapter 13 offers some concluding thoughts on the theory and practice of evaluation.

How to use this book

The book can either be read from start to finish or if you have particular interests it is possible to concentrate on particular chapters:

 For a quick overview of evaluation, go to Chapter 1

 If you are designing an evaluation either as an evaluator or as a commissioner, concentrate on Chapters 3–7

 For practical advice on planning an evaluation and then conducting an evaluation, look at Chapters 8–9

 If you are thinking about reviewing existing evaluation findings or using evaluation evidence to influence policy and practice, look at Chapters 10–11

 If you want a deeper understanding of different evaluation paradigms and some of the key philosophical debates within evaluation, look at Chapter 12