Читать книгу Learning in Development - Olivier Serrat - Страница 13

Evaluations that focus on key issues and provide usable findings and recommendations in a timely manner are a cost-effective means to improve the performance and impact of policies, strategies, programs, and projects. By challenging accepted thinking, such evaluations also contribute to improving overall development effectiveness.

Box 15: Building a Results-Based Management Framework ^a

Results-based management involves identifying the impact of an intervention, formulating its outcome, specifying outputs and inputs, identifying performance indicators, setting targets, monitoring and reporting results, evaluating results, and using the information to improve performance. A good quality design and monitoring framework is an integral quality-at-entry results-based management tool that (i) clearly identifies key project objectives with measurable performance indicators, (ii) establishes quantified and time-bound milestones and targets for the indicators at each level of the project, and (iii) specifies the sources of data for tracking implementation progress. Lacking one or more of these elements at entry weakens a project’s design quality.

In 2003, an evaluation study on project performance management found that the quality of ADB’s design and monitoring frameworks was poor—particularly in terms of clearly documenting the impacts and outcomes that ADB is trying to achieve. In response to the findings from this evaluation, ADB’s Management developed an action plan to rectify the situation. Multiple actions were initiated to create quality assurance, mentoring, and training capability within originating departments, and these departments were given clear responsibility and accountability for quality, and quality assurance. The vice-presidents of ADB’s operations departments gave instructions that frameworks needed to be improved for loans and TA operations, and directors general and directors were also required to sign off on frameworks. Recognizing that staff skills needed to be enhanced, the action plan directed that focal points be appointed in all regional departments to promote awareness, consistency, and knowledge sharing. Greater executing agency involvement in the preparation of design frameworks was also anticipated to help develop executing agency ownership further, sharpen design quality, and build understanding that the frameworks would be used as a monitoring tool.

The Central Operations Services Office and OED both played important roles. The former engaged a framework specialist, formulated the project performance monitoring system, and administered the initial inputs of the specialist to draft guidelines and conduct training programs. In 2004, more than 300 staff members attended briefing sessions that OED delivered on framework quality. A video version of this briefing was released for use by resident missions and interested parties. In 2004, OED also responded, daily, to requests for help in strengthening frameworks. Nevertheless, internal quality assurance alone is unlikely to be sufficient to ensure quality. Independent checking is also needed to validate that quality assurance systems are working effectively and whether quality improvements are actually achieved. To determine whether the efforts undertaken in 2004 bore fruit, OED subsequently conducted several independent assessments of the quality of frameworks. These assessments confirmed that, prior to implementation of the action plan, the majority of design frameworks were substandard. However, after implementation of the action plan in 2004, there was a sharp reversal resulting in a statistically significant improvement whereby approximately two-thirds of project frameworks were judged to be of acceptable quality.

The 2006 Annual Report on Loan and Technical Assistance Portfolio Performance contained a special chapter on design and monitoring frameworks to once again examine their quality and to track changes. The trends in the overall quality of the frameworks prepared at the project design stage and approved each year since 2000 are illustrated below.

Design and Monitoring Frameworks Rated Satisfactory or Better Overall

The significant improvements in design and monitoring framework quality can be plausibly attributed to action plan improvements instigated by evaluation studies. Nevertheless, despite these achievements, too many advisory and regional technical assistance frameworks remain substandard. Past evaluation studies have consistently documented the disappointing performance of ADB’s knowledge products and services. One of the contributing factors appears to be poor planning—particularly at the impact and outcome levels. It should not be surprising, therefore, that a lack of clarity in formulating higher-level project objectives is associated with poor results. OED will continue to monitor the quality of frameworks. The Central Operations Services Office has developed, published, and distributed guidelines for preparing frameworks and has continued to provide training in the understanding and use of this core results-management tool.

In a brief on managing for development results prepared for the DEC in November 2005, the Strategy and Policy Department noted that the Central Operations Services Office had set interim performance targets for framework quality. The goal was to have at least 80% of the frameworks prepared for loan projects and programs, and at least 50% of the frame-works prepared for advisory and regional TA activities rated satisfactory or better during 2005 and in subsequent years. The 2006 Annual Report on Loan and Technical Assistance Portfolio Performance shows that those targets were achieved in 2005. However, the ultimate target in the short to medium term must be to have all of the frameworks prepared during the project design phase, for all projects, rated satisfactory or better. ADB is also reaching out from headquarters. Since September 2005, 283 staff from executing agencies in 17 DMCs and 45 staff members from resident missions have attended workshops on project design and management. Ninety-five facilitators from 12 DMCs have participated in related training. Officials from 19 DMCs participated in the Third International Roundtable on Managing for Development Results held in Ha Noi, Viet Nam, in February 2007.

^a Available: www.adb.org/documents/ses/reg/sst-oth-2003-29/ses-ppms.pdf

Box 16: Promoting Portfolio Performance^a

The 2006 Annual Report on Loan and Technical Assistance Portfolio Performance identified several problems: (i) a stagnation in OCR loan approvals; (ii) persistent delays in project implementation; (iii) a growing problem with year-end bunching for loan and TA approvals, as well as project, program, and TA completion reports; (iv) a steady decline in OCR project loan disbursements during the previous decade; (v) difficulty in meeting the conditions for program loan tranche releases; (vi) lending concentrated in a few DMCs; (vii) in response to falling interest rates, a few large borrowers prepaid their older and relatively expensive OCR loans; (viii) negative net resource transfers from ADB to DMCs; (ix) a fall in OCR income of 43% during 2001–2004; and (x) weaknesses in portfolio management—20% of ongoing loans and 75% of ongoing TA activities went without a review mission during 2004. The report concluded that these broad trends all supported the contention that ADB’s traditional lending products and systems no longer met many of the needs of its key clients. Unless ADB could address these issues by developing new products and procedures to meet the DMCs’ development needs, ADB would be threatened with losing relevance as the premier development institution in Asia and the Pacific.

The DEC agreed that the strategic questions raised by the report were of fundamental importance to ADB’s continued relevance to the region and that the significance of the issues was enhanced by the independence of the analysis. The DEC agreed with ADB Management that, while some of OED’s recommendations were quite sensible, taken together they were not sufficient to deal with the portfolio problems diagnosed in the report. The DEC requested ADB’s Management to prepare a comprehensive action plan to address the key strategic issues analyzed in the report.

The resulting action plan was prepared in November 2005. Among others things, it included actions designed to (i) enhance project administration efficiency, (ii) improve TA portfolio management, (iii) improve planning and timing of Board consideration of loans, (iv) improve sector selectivity, (v) strengthen project monitoring and evaluation, and (vi) increase and improve the OCR portfolio. The action plan incorporated initiatives being undertaken by ADB in its reform agenda, including the innovation and efficiency initiative, the strategy for enhancing ADB support to middle-income countries and borrowers from OCR, and managing for development results at ADB. OED will continue to monitor the implementation of the action plan.

^a Available: www.adb.org/documents/reports/portfolio_performance/2006/rpe-oth-2006-10.pdf

Box 17: Improving Country Partnership Strategies

Country partnership strategies have become the key instrument to set priorities for ADB’s operations in a country. Thus, country assistance program evaluations currently have the clearest, most direct, and most systematic influence on ADB’s operations. Experience in 2005 and 2006 suggests that their influence on the formulation of new strategies has been mainstreamed. The Board does not normally discuss a country partnership strategy until after the DEC has considered the corresponding country assistance program evaluation and informed the full Board of the DEC’s views based on the findings. Lessons from country assistance program evaluations fall in 10 areas:

• Future assistance should be prioritized based on selectivity and focus, with successful ADB performance in a sector as one key criterion.

• Country partnership strategies should be results based—the lack of monitorable indicators made it difficult to evaluate past strategies.

• Success has been greatest when ADB maintains a long-term involvement in a sector and combines programs for capacity building with investment support.

• Projects and programs using relatively simple designs that are rooted in local conditions are more likely to succeed than complex interventions.

• Steps must be taken to strengthen the impact of TA used to support policy reform, capacity building, and institutional strengthening.

• ADB should deepen its relationships with broader society, as this enhances ownership and often helps to achieve better development results.

• ADB should intensify its coordination with development partners and stakeholders.

• Governance, including the need to control corruption, should be explicitly addressed in country partnership strategies, and not just as a crosscutting theme.

• The understanding of corruption and the risks that it has for ADB’s operations remains superficial in country partnership strategies.

• Failure of project designs to recognize and address institutional weaknesses in implementing agencies early on leads to weak project performance.

Examples of recent country assistance program evaluations that have influenced the subsequent country partnership strategies include those for Bhutan, Cambodia, Indonesia, Lao People’s Democratic Republic, Philippines, and Uzbekistan.

Box 18: Agriculture and Natural Resources Sector Assistance Program Evaluation in the Lao People’s Democratic Republic ^a

OED evaluated 20 years (1986–2005) of ADB support to agriculture and natural resources development in the Lao People’s Democratic Republic in a sector assistance program evaluation covering 2 programs (totaling $50 million), 7 investment projects ($84 million), 32 advisory TA operations ($12.4 million), 11 project preparatory TA operations ($6.7 million), and 14 regional TA operations ($10.5 million) with components in the Lao People’s Democratic Republic.

In the wake of the government’s New Economic Mechanism in 1986, aimed at a transition to a market-based economy, ADB responded with broad, policy-based lending intended to reduce market distortions by liberalizing trade, rationalizing pricing practices, restructuring taxation, and separating commercial and central banking. Investment projects began in 1993 targeted at agribusiness, commercialization of smallholder agriculture, irrigation, livestock, river basins, watersheds, upland agriculture, and tree plantations.

The study found that, while individual projects had been relevant to the country’s development needs and to agriculture and natural resources issues, in the aggregate their effect had been diffused and, therefore, less effective. The country strategies for assistance to the agriculture and natural resources sector were rated partly satisfactory, being predominantly project driven. The existing country partnership strategy did not have a framework for prioritizing assistance to the sector. Neither investments nor policy-based loans had been coordinated. Thus, the evaluation study rated the overall performance of assistance as partly successful.

ADB’s portfolio management and project administration in the sector needed strengthening. Review missions were not conducted regularly, they did not cover remote areas, and appropriate expertise was not always available. While the relevance of sector assistance was important, relevance alone would not make ADB an effective institution. Greater selectivity in engagement was required.

ADB’s Management found the evaluation study timely in that it fed into the preparation of the country partnership strategy for 2007–2011. It declared the study’s conclusions generally valid, but queried what would have happened in the absence of ADB’s operations. ADB’s Management suggested that this question be asked in all subsequent sector assistance program evaluations.

After the evaluation study was released, the minister of agriculture and forestry investigated whether or not poor project performance was unique to ADB’s operations in the sector, by means of a series of government-led evaluations of external assistance in different subsectors (viz., irrigation, forestry, natural resources, crop production). These evaluations drew on the criteria and rating systems used in the OED study. This Government-led evaluation found that ADB’s operations were good in comparison with others. It also stimulated discussions in the ministry about how best to reform its public investment. One outcome was that the ministry added “expected project sustainability” to the project screening criteria for all new projects, after observing that this had been a particular problem in ADB and government-led evaluations. The OED study had a major effect by demonstrating approaches for undertaking higher level evaluations that made sense to key government officials. Other ministries, particularly health and education, are now showing interest in government-led evaluations.

^a Available: www.adb.org/documents/reports/sape/lao/sap-lao-2005-17/sape-lao.pdf

Box 19: Contributing to Safeguard Policy Update ^a

ADB has safeguard policies on involuntary resettlement (1995), indigenous peoples (1998), and the environment (2002). The policies have guided formulation and implementation of ADB’s assistance programs, particularly its lending activities. Evaluation of the three safeguard policies was included in OED’s work program for 2006 at the request of the DEC. Based on its discussions of the evaluation reports, the DEC considered that the process it had recommended—to allow time for an independent evaluation to feed into ADB Management’s review of the safeguard policies—had yielded useful results. The process served to raise many of the right questions for the review. The issues identified and recommendations made addressed not only implementation of the policies but also provisions in the policies themselves. And, by its response to the evaluation studies, Management started the critical corporate-level dialogue that would lead to policy review. The DEC believes that a reasonable process of open and inclusive dialogue involving all stakeholders, particularly on issues that are not clear cut, is needed. Such a dialogue may require an iterative process exceeding ADB’s conventional consultation activities.

Involuntary Resettlement ^b

The evaluation study found that the policy is relevant to project implementation and to ADB’s aim to reduce poverty. It assessed the policy as effective in terms of outcomes for affected persons. However, it found inputs, processes, and systems for policy implementation less efficient. Changing procedures and the organizational arrangements made to enforce the policy have gradually set the bar higher. More resources allocated to capacity development rather than short-term compliance may yield better long-term results. The study recommends that the planned update of the policy have a results-based framework, and that it should indicate mandatory and nonmandatory but desirable elements. ADB should decide on the level of inclusiveness of the policy, particularly regarding secondary adverse impacts of projects on people.

• ADB’s Management and the Board need to reconcile the differences between the original policy on involuntary resettlement and that currently applied.

• Whatever the nature of the policy adopted, its results-based framework should distinguish desired impacts, outcomes, outputs, activities, and inputs at both the macro (country) and micro (project) levels.

• The policy update should highlight a set of performance standards.

• The policy update should elaborate on the objective of greater reliance on country executing agency systems for land acquisition and resettlement safeguards.

• There should be clearer guidelines and procedures regarding the identification of resettlement operations needed, and compensation and assistance within resettlement operations.

The study makes recommendations regarding implementation of involuntary resettlement plans:

• Formulate a time-sequenced implementation plan to complement the policy update.

• Improve involuntary resettlement monitoring.

• Deepen involvement in building country systems and capacity for involuntary resettlement.

ADB’s Management welcomed the evaluation study, noting that both sets of recommendations should be further reviewed and considered in the context of policy update, including the consultation process. The chair’s summary of the DEC discussions advised that ADB should recast the policy in a comprehensive results framework, providing clear guidelines on the principles, degree, and approaches by which flexibility may be exercised in application.

Indigenous Peoples ^c

The evaluation study concluded that decisions taken by ADB’s Management had expanded the scope of the original policy. The original focus on “adverse and significant” impacts was reinterpreted to cover “adverse or significant” impacts, and the associated operational procedures became more elaborate over the years.

The policy is misunderstood by ADB staff and clients. First, the distinction between indigenous peoples and ethnic minorities is not always clear, and differs somewhat between the policy and the operational procedures. Second, the policy is ambiguous about whether the definition provided by national legislation or that derived from the policy is to be followed. Third, there is overlap with the policies on involuntary resettlement and environment. The clearest adverse impacts of ADB-supported projects on indigenous peoples relate to induced environmental change, loss of land and related livelihood, and resettlement. These are also the subject of the policies on involuntary resettlement and environment and are addressed primarily in the involuntary resettlement plan and the environmental management plan. If considerations of land, livelihood, and resettlement were discounted, then the impacts that trigger the policy would be less straightforward adverse impacts related to cultural change and, perhaps, integration into the economic mainstream and/or competition with non-indigenous peoples when an area is opened up or developed. Fourth, the need to prepare indigenous peoples development plans for projects with significant benefits to indigenous peoples, and specific actions for projects with limited positive or negative impacts on them, has left unclear the nature of these plans and specific actions. In practice, an acceptable scope for a separate plan has proved difficult for ADB staff to define. Fifth, the required consultation of indigenous peoples has been expanded by operational procedures to something very close to full consent for the project. The policy offers little advice on how to define an acceptable level of consent or how to document and measure it.

The study recommends the following:

• The policy update should clarify the areas that cause misunderstanding in the policy and address the policy drift related to ADB’s Operations Manual and practice in ADB.

• ADB should set goals for the development of indigenous peoples and related strategies for some DMCs where ADB’s forward program involves considerable interaction with indigenous peoples.

• If ADB maintains a stand-alone policy, it should include a results-based framework— distinguishing desired impacts, outcomes, outputs, activities, and inputs, both at the macro (country) and micro (project) levels.

• A sequential approach to policy development and capacity building in indigenous peoples safeguards should be adopted, focusing on a few DMCs first.

• Indigenous peoples development plans should be prepared for projects having clear risks for indigenous peoples that are capable of being mitigated through project interventions.

• Conceptual work and case study work are needed to lay out the particular risks for indigenous peoples associated with different categories of investments, as there is currently a high degree of divergence in approaches to the definition of these risks.

• The policy update should describe the criteria to be used to determine whether the amount of consultation and broad community support for a project and mitigation measures are adequate and in what circumstances ADB endorses the principle of free, prior, and informed consent for the project from the side of indigenous peoples.

• To complement the policy update, there is a need for a policy implementation plan that reconciles the policy aspirations with organizational, budget, and human resources implications.

ADB’s Management generally appreciated the evaluation and agreed to consider the recommendations in the update of the policy.

Environment^d

The evaluation study found that the policy is relevant to ADB’s activities and the needs of DMCs. ADB’s involvement in projects sometimes added value by improving environmental performance at the project level. However, the value addition varied from country to country, from project to project, and in different aspects of environmental assessment. There is also evidence that the policy provided impetus to improve the environmental safeguards toward greater clarity, more emphasis on assessment of project alternatives, and improved monitoring, notwithstanding weaknesses in all these areas. On the whole, the environmental safeguard procedures governed by ADB’s Operations Manual were deemed to have been effective in avoiding significant adverse environmental impacts from ADB’s programs and projects.

However, the efficiency and sustainability of the safeguard procedures are questionable due to high transaction costs and limited benefits. The main cause of this is the uniform application of a single set of procedures to all DMCs, which is no longer an effective response to the needs of DMCs nor to the growing resource constraints faced by ADB. There would be merit in revising the policy and safeguard procedures to address the changing context in DMCs. This would likely be focused on the necessity for greater flexibility in procedures, recognition of the need for alignment with national systems, a shift to an emphasis on capacity building rather than on projects, and a change in emphasis from procedural compliance to results delivery and cost effectiveness.

The study recommends that the policy should be revised to better reflect current needs and resources within DMCs and ADB, as follows:

• Organizational effectiveness should be strengthened in relation to the reorganization of 2002, including consolidation of environmental resources within ADB.

• The quality of ADB’s environmental assessment process should be improved and transaction costs should be lowered to make the policy more cost effective. Categorization, scope, quantification, standards, and technical methodologies should be systematically reviewed and updated.

• Partnerships with NGOs and civil society organizations on environment should be improved.

• ADB should move toward adoption of improved country systems and harmonization with other development partners in selected DMCs. ADB should follow the guidance in the Paris Declaration on Aid Effectiveness on greater harmonization between funding agencies and partner countries on environmental assessment.

• An implementation plan for the revised environment policy should be prepared. ADB should develop an action plan to implement the revised policy involving an assessment of ADB’s resources at headquarters and in the field for implementing environmental safeguards.

ADB Management’s response confirmed that the evaluation study raised important issues that are relevant to policy update. Management believed that the ongoing process of policy update, including the planned consultations with a wide range of stakeholders, provides an appropriate vehicle for further examining the report and addressing the recommendations of the study. The chair’s summary of the DEC discussions notes that the study raised many of the right questions for the review; that the issues identified and recommendations made address not only implementation of the policy but also provisions in the policy itself; and that, by its response to the evaluation, ADB’s Management started the critical corporate-level dialogue that would lead to the policy review.

^a Available: www.adb.org/safeguards/default.asp

^b Available: www.adb.org/documents/ses/reg/sst-reg-2006-14/ses-on-ir.asp

^c Available: www.adb.org/documents/ses/reg/sst-reg-2007-01/ses-ip.asp

^d Available: www.adb.org/documents/ses/reg/sst-reg-2006-13/ses-es.asp