Читать книгу Learning in Development - Olivier Serrat - Страница 10

Evaluation has changed with ADB. Early work focused on input–output relationships in projects, using economic analysis, but evolved to cover the entire results chain of inputs, outputs, outcomes, and impacts. The main unit of account shifted from the project to the country, informed by sector and thematic assessments, as well as by evaluations of ADB’s business processes. The full mix of lending and nonlending services that make up country assistance programs has now become the dominant preoccupation of evaluation, with priority attention to relevance, efficiency, efficacy, and sustainability. In parallel, feedback has grown from restricted circulation of evaluation reports to maximum transparency and active dissemination of findings and recommendations through the Internet. Institutionally, evaluation has gained recognition from the Board and ADB’s Management commitment to achieving development results. Indeed, more suggestions for evaluations are now received than OED can undertake with its current staff complement and budget.

Managing for development results is a prerequisite to improving the quality of assistance. It is one of the five key principles in the Paris Declaration on Aid Effectiveness of March 2005, which promotes development through country ownership, harmonization, alignment, results orientation, and mutual accountability. Multilateral and bilateral agencies are under pressure to demonstrate development effectiveness and to show impact on the ground. ADB is improving its effectiveness in reducing poverty and promoting development in the Asia and Pacific region by means of an action plan built on three pillars: (i) country capacity, (ii) ADB’s results orientation, and (iii) effective global partnerships. OED can contribute in several ways: It will review and evaluate progress toward achievement of managing for development results and ADB’s underlying culture and management style; it will assess how concepts of managing for development results feed into preparation of country partnership strategies and the project cycle with an eye to the needs of DMCs; and it will examine how ADB’s Management uses such concepts and relevant experience to guide decision making.

An important organizational change to increase the independence of OED took effect in January 2004.⁸ OED now reports directly to the Board of Directors through the Board’s Development Effectiveness Committee (DEC), instead of to the President. Other significant changes are that (i) the Board, rather than the President, appoints the director general of OED; and (ii) ADB Management’s role in evaluation changed from approving evaluation reports to responding to their conclusions.

Box 2: The Development Effectiveness Committee ^a

The DEC was established by ADB’s Board of Directors in December 2000 and initiated activities on behalf of the Board in 2001. It consists of not more than six members of the Board, who meet about 12 times a year. Its general mandate is to assist the Board to carry out its responsibility of ensuring that the programs and activities of ADB achieve development effectiveness. Such development effectiveness is assessed through ADB’s operations evaluation. For the purpose of the DEC’s work, “development effectiveness” is the measure of (i) whether ADB’s programs and activities in furtherance of its policy goals and objectives have resulted in the desired outcomes, and (ii) whether these programs and activities have made efficient use of ADB’s available resources. The DEC

• reviews the work program of OED;

• reviews selected evaluation reports and the actions taken by ADB on them;

• reports to the Board on high-priority operations evaluation issues, if any, that have a significant bearing on the relevance, efficiency, and effectiveness of ADB, and makes recommendations on such issues to the Board;

• monitors and reports to the Board on the implementation of decisions;

• reviews OED’s work program for preparation of project, program, and technical assistance completion reports; and

• reviews the Annual Evaluation Review and the Annual Report on Loan and Technical Assistance Portfolio Performance.

The DEC’s evolving key areas of concern are the following:

• How are the evaluation recommendations that ADB’s Management decides to accept turned into action so that targeted changes for improvement happen?

• How are actions and their impacts monitored?

• How can OED play a more strategic role? Specifically, how can the influence of independent evaluations on ADB’s long-term strategic thinking be increased?

• How can the DEC play a more strategic role and increase the value it adds to ADB’s learning process?

The specific steps that the DEC has taken to support OED include (i) suggesting topics of strategic interest for inclusion in OED’s work program, (ii) helping to get the timing right for policy studies, (iii) ensuring that country assistance program evaluations are discussed by the DEC before country partnership strategies are finalized, and (iv) initiating measures to strengthen ADB Management responses.

^a Available: www.adb.org/bod/dec.asp

To ensure the independence and transparency of evaluation reports, a new approval, review, and disclosure process has been established. Since evaluation reports are final upon approval by the director general of OED, they are made publicly available immediately upon circulation to ADB’s Management and the Board. ADB’s Management is given a specific period in which to respond to lessons and recommendations. ADB Management responses and the DEC chair’s summaries of discussion are disclosed as they become available.⁹

The policy paper to enhance the independence and effectiveness of OED saw the need to formalize practices for avoiding conflicts of interest in evaluations. In March 2005, ADB became the first member of the Evaluation Cooperation Group (ECG)¹⁰ of the multilateral development banks to formally adopt guidelines¹¹ to avoid conflict of interest in independent evaluations, which specify conditions under which OED management, staff, and consultants must recuse themselves.

OED has also strengthened the process used to formulate its work program. Previously, only annual work programs were prepared, but starting in 2005, a 3-year rolling framework was developed.¹² Wide-ranging consultations took place for the preparation of the 2006–2008 work program. Discussions on institutional priorities for evaluations were held with the DEC and all vice-presidents. Inputs were also sought from ADB’s operations departments as in the past. The evaluation program has become more strategic and more integrated, with one evaluation feeding into another. For instance, evaluations of the effectiveness of ADB’s technical assistance (TA), microcredit operations, energy policy, governance and anticorruption policies, approaches to partnering and harmonization, policy-based lending, adoption of managing for development results, safeguard policies, and accountability mechanism were programmed to feed into reviews of these by ADB’s Management.

The influence of evaluations on ADB’s operations, policies, and strategies was assessed in the 2006 Annual Evaluation Review.¹³ Their influence was evident in (i) the sector priorities for ADB’s operations under the new model for selectivity and focus set out in the second medium-term strategy, (ii) ADB Management’s agreement to implement an action program to improve portfolio performance in response to the DEC’s recommendation following its discussion of the 2005 Annual Report on Loan and Technical Assistance Portfolio Performance,¹⁴ and (iii) changes in new country strategies and programs directly related to lessons and recommendations from previous country assistance program evaluations.¹⁵

Through its oversight of OED, the DEC is now helping to ensure that actions are taken on lessons and recommendations that it considers to be of high priority and that ADB’s Management has accepted. In its annual report to the Board in 2005, the DEC included a specific, candid assessment of the system for taking actions. The DEC considered that there was room for ADB to become a full-fledged learning organization using evaluation results more systematically. The DEC reported on actions taken by OED and by ADB’s Management on its recommendations:

(i)	Traditionally, the intended end-users were consulted during and near the completion of an evaluation. Nowadays, OED reviews the scope and methodology with the main stakeholders, particularly ADB’s operations departments, before starting.
(ii)	OED has introduced the new step of discussing draft findings and recommendations from evaluation with ADB’s operations departments and, in the case of country assistance program evaluations, with the government before the evaluations are finalized. The objectives are to ensure that those who will be responsible for implementing the recommendations understand them, to find out which are acceptable and feasible, and to build early commitment.
(iii)	The actions that ADB’s Management commits to in its formal responses to evaluation reports have generally been more specific and time bound.

To improve consistency and quality, guidelines for the evaluation of public sector projects, program loans, TA, and country assistance programs were issued in 2005. Guidelines for evaluating private sector operations were completed in 2007.¹⁶ The new modus operandi has brought institutional advantages, but it also raises the question of how OED itself should be evaluated. To address this issue, the ECG has begun to examine the feasibility of establishing a peer review process of the evaluation function in its members.