Читать книгу Learning in Development - Olivier Serrat - Страница 11

There is a long feedback loop from approval to completion of an operation, when development results can be assessed. It takes 7 years or more to implement projects or programs after Board approval and for completion and evaluation reports to be prepared. Thus, virtually all projects or programs that were approved in the mid-1990s were in various stages of implementation at the turn of the millennium. Indeed, many are still under implementation. The completion and evaluation reports for most of the projects and programs approved in the 1990s were prepared between 2000 and 2005. Analysis of project success was undertaken based on the year of approval. Figure 1 shows that the success rates of Asian Development Fund (ADF)-funded projects approved before the 1990s tended to be considerably lower than those funded by ordinary capital resources (OCR),¹⁷ and average ADF ratings were much more volatile. In the 1990s, however, the OCR and ADF ratings converged, and both improved. Given the development challenges faced by ADF-eligible countries, this is a positive finding. The long-term trend of the ratings suggests that the design and implementation of ADB-assisted projects have progressively improved, whether funded by OCR or by ADF. This indicates successful, gradual institutional learning from lessons of past operations. This improvement in project outcomes needs to be continuously monitored to ensure that the trend is sustainable.

Figure 1: Trends in Project Ratings by Year of Approval and Source of Funding^a (Combined Completion and Evaluation Results)

^a Success rate for ADF and OCR based on 3-year moving average.

ADF = Asian Development Fund, OCR = ordinary capital resources.

Source: OED.

While project performance has varied significantly across countries, it has not varied significantly by country classification (i.e., degree of eligibility for ADF funding).¹⁸ The analysis by country classification in Table 1 demonstrates improvement from the 1970s to the 1990s across all country groupings, though patterns of progress differ. By the 1990s, the success rate of each group with access to ADF had gradually increased, and countries in Group A (with the exception of Nepal) were performing better than those in Groups B1 and B2.

Table 1: Project Success by Country Classification

Source: OED.

The averages by classification mask important country differences. Within all groups, there are outliers in both directions. The country variation holds true for source of funding. For some OCR borrowers, projects are more likely to be successful than in ADF-eligible countries, but ADB’s portfolios perform better in some countries that have access to ADF than in some OCR borrowers. In Group B2, the People’s Republic of China stood out as one of ADB’s best-performing portfolios, achieving an 82% success rate in the 1990s. In contrast, Papua New Guinea had a 22% success rate in the 1990s, the weakest portfolio performance in ADB. In Group B1, the success rates of the portfolios in Bangladesh (84%) and Viet Nam (91%) significantly exceeded both group and ADB averages for projects approved in the 1990s. In contrast, only 57% of projects approved for Pakistan in the 1990s were rated as successful. In Group A, the portfolio success rates of Bhutan, Cambodia, Kyrgyz Republic, Lao People’s Democratic Republic, Maldives, Mongolia, and Solomon Islands exceeded Group A and ADB averages in the 1990s. On the other hand, the success rates for Kiribati, Nepal, Samoa, and Vanuatu were below average for countries in Group A.

There is a school of thought that country ownership, which is a key factor for project success, increases with a country’s share of the financing of a project. This hypothesis was tested by comparing project success rates with the percentage of project costs financed by the executing agency after controlling for sector differences. No significant statistical relationship was found, nor was there any clear pattern to suggest that project success improved with the share of government financing. Since the likelihood of project success is not compromised as the share of ADB funding increases, it is reasonable for ADB to address the greater fiscal constraints faced by lower income countries by increasing the allowable financing limit.

Box 3: Republic of Korea Financial Sector Program and Institutional Strengthening of the Financial Sector Project^a

Highly Successful

By 1996, the Republic of Korea had become the world’s 11th largest economy. Many industrial conglomerates had grown rapidly on borrowed capital and were highly leveraged. Interlocking relationships between the corporate and finance sectors, and the accommodative role of government, meant that many financial institutions did not operate within prudent, market-driven norms. In mid-1997, the currencies, stock markets, and other asset prices in Southeast and East Asian economies, including the Republic of Korea, depreciated rapidly, causing panic among investors and financial institutions. Several smaller conglomerates in the country declared bankruptcy but were not bailed out by the Government, which had been the practice in the past. Confidence in the Republic of Korea’s finance sector eroded. Demands were made for the repayment of foreign exchange loans, and a foreign exchange crisis was imminent. The Government approached the international community for emergency assistance. As part of the package, the Asian Development Bank provided a $4 billion program loan aimed at restructuring the entire finance sector. This was the bank’s largest loan ever, and it was processed in record time.

Evaluation found that the program had a very high level of government ownership of the intended reforms. It had addressed fundamental structural weaknesses in the finance sector. The coverage of the reforms and their sequencing had been appropriate. The reforms facilitated the elimination of nonviable institutions, opened the sector to competition, and improved the regulatory framework and market oversight. Finance sector indicators improved rapidly in 1998, and the finance sector is now healthier and less vulnerable to external shocks.

^a Available: www.adb.org/documents/ppers/kor/31651-kor-pper.pdf

Box 4: People’s Republic of China Anhui Environmental Improvement Project for Municipal Wastewater Treatment and Industrial Pollution Abatement ^a

Successful

Rapid economic growth in Anhui province in the 1990s resulted in increases in nutrients and pollutants entering Chao Lake with concomitant algal blooms, fish kill, and pollution of the water supply of Hefei and Chaohu cities. The project was the first major initiative of a massive cleanup campaign. It aimed to improve the water quality in Chao Lake and reduce wastewater, air, and solid waste pollution in Hefei and Chaohu. It had six subprojects: two treatment facilities for Hefei and Chaohu, and industrial pollution abatement for four top industrial polluters in Anhui.

The project was implemented as envisaged, although there were some cost savings and a delay in implementation. Implementation of project components varied, ranging from excellent to less than satisfactory. The two wastewater treatment plants improved the environment by treating about 310,000 cubic meters of polluted water per day. The wastewater treatment ratios of these two plants are among the highest in the country. The four industrial pollution abatement subprojects improved the quality of products; reduced air, water, and solid waste pollution discharges; and reduced resource inputs by improving production efficiency.

The project was relevant and effectively and efficiently helped to address pollution in Chao Lake, targeted as a top priority for environmental cleanup by both Anhui and the State Council. The recalculated economic internal rate of return was 16.4% and the recalculated financial internal rate of return was 8.4% (higher than the cost of capital). With some exceptions, the subprojects were generally well maintained, but the poor financial performance of some enterprises may adversely affect the sustainability of some. Despite their satisfactory financial internal rates of return, the project is assessed as less likely to be sustainable (without the Government’s coverage of debt). The project has achieved significant impacts on institutional strengthening of environmental management and on the operation and maintenance of wastewater treatment facilities.

The two associated advisory TA operations achieved significant results. Training courses were conducted for operators of municipal and industrial wastewater treatment facilities throughout Anhui in environmental monitoring, operation, maintenance, and management of such facilities. A permanent training center was established using the training manuals and equipment provided under the TA. A long-term integrated environmental management plan was formulated to improve water and air quality and solid waste management in the Chao Lake basin. The outputs of the TA were highly relevant, timely, and instrumental for the formulation of the Chaohu Environmental Improvement Plan, approved by the State Council, and included in Anhui’s 10th Five-Year Plan. More than 80% of the investment projects recommended to improve environmental quality in the Chao Lake basin have either been, or are being, constructed.

^a Available: www.adb.org/documents/ppers/prc/28241-prc-pper.pdf

Box 5: Indonesia Health and Nutrition and Social Protection Sector Development Programs^a

Successful

A sector development program is a combination of investment (a project) and policy-based (a program) components designed to meet sector needs in a comprehensive and integrated way. In 2006, OED evaluated the sector development programs for health and nutrition and for social protection in Indonesia. These were OED’s first evaluations of that lending modality. Both programs were designed and implemented in response to the 1997 financial and economic crisis that undermined social well-being in Indonesia by increasing unemployment, reducing family incomes, and raising prices of necessities. As a result, there was a substantial increase in the number of people living in poverty and a reversal of social development progress achieved over the preceding two decades. The Government’s limited budget constrained the delivery of essential and basic services, preventing people—especially the poor and vulnerable groups—from accessing quality health care and educating their children.

The health and nutrition sector development program aimed to secure for the poor access to essential health, nutrition, and family planning services; maintain the nutritional status of vulnerable groups; and reduce the incidence and impacts of communicable diseases associated with poverty and malnutrition. It supported fundamental reforms through the development of pro-poor health policies, structures, and systems. The project was designed to improve and maintain the health and nutrition of the population and secure continued access to essential health, nutrition, and family planning services. The program helped the government institute policy reforms. The project produced most of the expected outputs. Project interventions largely reached the intended beneficiaries. These included (i) health card distribution to the poor; (ii) revitalization of community-based health posts that provide prevention and promotive health, nutrition, and family planning services, and community volunteers; and (iii) provision of maternal care, family planning, and immunization services. However, complementary and supplementary feeding for infants, children, and pregnant women with high caloric deficiency, as well as assistance for schoolchildren, reached fewer than expected. Key health indicators suggest that the general health and nutrition status of the population, including family planning practice, improved. Maternal deaths were contained by promoting prenatal care, increasing the number of deliveries attended by professionals, and enhancing the referral system for high-risk cases. The health of infants and children was improved by intensifying outreach by paraprofessionals and ensuring support for immunization programs. A pro-poor-based planning and resource allocation system was established. Furthermore, mechanisms were put in place to minimize misuse of resources and to resolve complaints.

The purpose of the sector development program for social protection was to reduce impoverishment while introducing sector reforms to strengthen social services delivery in anticipation of decentralization. The program loan provided support to government policies aimed at maintaining access of the poor to social services; maintaining the quality of services; continuing decentralization of social services management; and strengthening the efficiency, transparency, and accountability of management of funds in schools and health centers. The project loan aimed to maintain school enrollment and educational quality at precrisis levels through the provision of scholarships and school grants; protect the health and nutritional status of the poor, especially infants, mothers, and children; and provide scholarships and supplementary feeding to street children. Overall, the program succeeded in achieving its objectives. Achievements in education surpassed expected outputs. The nutrition component, due to its complexity, was less successful. Enrollment ratios were either maintained or rose. The block grants to primary schools prevented them from refusing children who could not pay. Life expectancy rose and infant mortality declined during the period of loan activity. The program was highly relevant in mitigating the impact of the crisis on the most vulnerable members of Indonesian society in terms of their access to basic social services in education, health, family planning, and nutrition. The education components generated additional funding from the World Bank and the Government, which enhanced the success of the program. The policy actions supported numerous novel interventions, including (i) allocating central government funding to local government based on the poverty index, (ii) decentralizing decision making, (iii) providing block grants to education and health service providers, (iv) making direct transfers to final beneficiaries through the post office to increase efficiency and reduce leakage, and (v) establishing independent monitoring of effectiveness of the program and accuracy of funds transfer. Although the central monitoring unit demonstrated the value of independent monitoring, this component was not sustainable after donor funding ceased.

^a Available: www.adb.org/documents/ppers/ino/32516-ino-pper.pdf and www.adb.org/documents/ppers/ino/32255-ino-pper.pdf

Box 6: Pakistan Livestock Development Project^a

Unsuccessful

Millions of rural poor in Pakistan depend on livestock for their livelihood. The project sought to address problems related to weak extension services, animal health and breeding, and limited feed sources during the summer. It aimed to establish a policy and institutional framework to encourage private investment in livestock, increase production and by-product utilization, and increase rural employment and incomes. Components were (i) strengthening the Federal Livestock Division and provincial livestock departments, (ii) stablishing extension services, (iii) breed improvement, (iv) improving animal health services, (v) slaughterhouse improvement, and (vi) capacity building through local and overseas training and consultancies.

Considerable resistance was encountered in reorienting the traditional extension services to a more bottom-up, production-based system. Rivalry among provincial agencies had a negative impact on project coordination. While the appointment and training of extension staff were partly effective, benefits were limited. Breeding activities were partly successful. A major accomplishment was the privatization of veterinary services, but slaughterhouse initiatives were a failure. The training component was extensive and successful, but consultants’ inputs were often poorly coordinated. At the time of evaluation, the only project-supported initiatives still operating were those that involved the private sector or where beneficiaries were charged for services. Economic reevaluation resulted in an almost negative economic internal rate of return, an unsatisfactory project outcome. Extension activities were unsustainable.

^a Available: www.adb.org/documents/ppars/pak/ppar-pak-21131.pdf

Project performance varied more significantly by sector than by country category. The likelihood of project success by sector is depicted in Table 2. The findings indicate that both OCR- and ADF-funded projects generally perform similarly in the same sectors, with certain important exceptions. The ratings of ADF-funded projects eventually converged with those of OCR-funded projects across sectors from 1970 to 1997.

Table 2: Project Performance by Sector and Source of Financing (% of Projects Rated as Successful)

ADF = Asian Development Fund, DFI = development finance institution, OCR = ordinary capital resources.

Source: OED.

ADB’s experience has been best in the transport/communications and energy sectors, with project success rates reaching 87% for both in the 1990s; the success rates are high for both ADF- and OCR-funded projects. In the transport sector, road projects have dominated the portfolio for both sources of funding. Project success has also been high for port, airport, and telecommunication projects, again regardless of the source of funding. The performance of the relatively small railway portfolio has been considerably weaker than that of the other transport modes. The characteristics of successful road projects are listed in the box.

Evaluations of ADB’s energy portfolio have found generally good project outcomes regardless of funding source. Power projects dominate the portfolio. Difficulties are sometimes experienced in complying with financial covenants and with ADB’s environmental and resettlement safeguard policies, particularly for hydropower projects. Nongovernment organizations (NGOs) and civil society organizations have raised concerns about the latter. Successful power projects exhibit the characteristics listed in the box.

The agriculture and natural resources sector has had poor outcomes relative to other sectors for both ADF- and OCR-funded projects. By the 1990s, project success rates were only 52% across funding sources. For ADF-funded projects, the success rate slightly exceeded the average for irrigation and rural development projects and for a small number of fertilizer plant projects. The difficulties faced in the sector were examined in 2005 in an evaluation of partly satisfactory operations in the Lao People’s Democratic Republic. Problems experienced included (i) inadequate project design, including insufficient understanding of the problems and opportunities viewed from the perspective of beneficiaries; (ii) no clear distinction between addressing the needs of subsistence farmers and options for commercialization of agriculture; (iii) project complexity; (iv) weak institutions; (v) shortfalls in operation and maintenance; (vi) marketing impediments confronting farmers and agribusinesses; and (vii) adverse impact of external factors (e.g., declining farmgate prices through much of the 1980s and 1990s). These problems are pervasive throughout the agriculture sector in both OCR- and ADF-funded projects. Nevertheless, since the livelihoods of about 80% of the population in the region depend directly or indirectly on agriculture, the sector remains important for virtually all countries. Some ADB irrigation and drainage projects have achieved good results. When successful, such projects bring significant gains to the beneficiaries. For irrigation projects, the key success factors were (i) an enabling policy environment; (ii) involvement of stakeholders and beneficiaries during all project phases, particularly operation and maintenance; (iii) participatory techniques to develop a sound understanding of the roles and responsibilities of farmers and water user associations and to create a climate in which participants are willing to pay irrigation fees; and (iv) adequate water supply reaching tail-end users, who are usually small farmers. The characteristics of successful irrigation and drainage projects are outlined in the box.

Box 7: Characteristics of Successful Road Projects

• Adequate levels of traffic use the completed roads; traffic growth is associated with economic growth.

• Vehicle operating costs and journey times were reduced and transport services improved.

• Continuity of ADB’s engagement in a country’s road sector and of ADB staff involvement had a positive influence on ADB’s contribution to project success. Reform initiatives were often pursued through a dialogue spanning several lending and TA operations, sometimes covering a decade.

• Good quality at entry covered the quality of the project preparatory TA, the project design, and the incorporation of lessons learned from previous projects.

• Strong government ownership was evident when ADB supported parts of a major highway investment plan that featured prominently in the medium-term investment plan of the DMC.

• Executing agencies performed well, were sometimes supported by supervision consultants, and often had a track record of having previously handled similar projects.

• Executing agencies, consultants, contractors, and ADB staff worked together to solve problems and handle unforeseen circumstances that developed during implementation.

• Supervision consultants and contractors performed satisfactorily.

• Regular ADB supervision missions were a consistent feature of successful projects, particularly during the first 2–3 years after loan approval, when most problems arise and ADB missions can be of most help.

• Adequate maintenance is essential for project success and sustainability.

Box 8: Characteristics of Successful Power Projects

• Because of the large deficiencies in the supply of energy, demand was not a problem, and benefits were immediate once the projects were completed.

• Increased electricity supply facilitated economic growth and contributed to a better quality of life.

• ADB evidenced a long-term commitment to working in the power sector.

• The plant and equipment were fully utilized and correctly operated and maintained.

• Financial policy dialogue and tariff reform helped to ensure sufficient funding for investment, operation, and maintenance.

• The project management team in the executing agency showed commitment, quality, and experience; familiarity with ADB’s loan requirements; and an ability to learn from previous projects.

• There was continuity of the executing agency’s staff and the consultants employed to assist with implementation.

• There was early recognition of problems during implementation and a flexible approach by both ADB and the executing agency to solve problems.

• ADB support contributed to strengthening institutions.

• ADB’s contributions to successful outcomes included a careful assessment of the capabilities of the executing agency, a flexible approach to project design and implementation, and regular project review.

Box 9: Characteristics of Successful Irrigation and Drainage Projects

• Elements of an enabling environment that allowed farmers to supply the demand for their produce included (i) a policy and institutional framework that promotes sound water resource management; (ii) a legal framework for water user associations that promotes cost recovery at least sufficient to finance sustainable operation and maintenance; (iii) rural infrastructure (e.g., roads that allow farmers to market their products, and farm inputs to be delivered when they are needed); (iv) efficient markets that are free of price distortions and barriers to competition for both farm products and agricultural inputs; and (v) access to information on demand, prices, and technology.

• Long-term ADB involvement in the sector and building up effective partnerships with executing agencies over a decade or more contribute to project success, policy reform, and the development of institutional capacity.

• Indicators of project ownership by executing agencies include (i) establishing site-based project offices, (ii) well-qualified staff, (iii) selecting executing agency personnel involved in earlier projects, and (iv) financing a considerable share of project cost.

• Good quality at entry reflected the quality of the feasibility study and project design, incorporation of lessons from prior projects, and the level of stakeholder participation.

• Direct stakeholders and beneficiaries were involved in all project phases, particularly operation and maintenance. Participatory techniques were used to develop a sound understanding of the roles and responsibilities of farmers and water user associations and to create a climate in which participants were willing to pay irrigation fees.

• Adequate water supply reached tail-end users, who are usually small farmers.

• Making changes in project design during implementation contributed to achieving good project outcomes.

• Effective quality control systems for civil works and internal and external audit systems were in place, even in remote areas.

• During implementation, continuous attention was paid to building the systems needed for effective operation and maintenance.

• Effective ADB project administration includes regular review missions, proactively helping to solve problems and making required approvals in a timely manner.

Box 10: Characteristics of Successful Water Supply/Wastewater Treatment Projects

• Rapid urbanization created a strong demand for the output of the projects.

• There were positive impacts on intended beneficiaries, particularly women.

• There was an ability to learn from past lessons and incorporate the lessons in project design.

• Technical innovation and a positive impact on the environment occurred.

• Proper operation and maintenance helped ensure long-run sustainability.

• The projects typically (i) were run by financially self-sustaining water supply institutions, (ii) set up water user committees, and (iii) adopted the “user pays” principle.

• A participatory approach contributed to success. Successful projects were typically formulated through extensive consultations with local government staff and the local community, including NGOs, and civil society organizations. Beneficiary participation resulted in a stronger sense of ownership and willingness to accept some responsibility for operation and maintenance and to pay higher tariffs.

• Executing agencies were committed, highly involved in project implementation, supported by institutional strengthening and training activities, and provided with counterpart funds in a timely manner.

• Consultants and contractors performed well.

• Regular ADB review missions proactively helped to solve problems.

ADF-funded projects in social infrastructure and lines of credit have been less successful than those funded by OCR. Social infrastructure includes a diverse group of subsectors (urban development, water supply and wastewater treatment, sanitation, education, and health and population). In the 1990s, 67% of the ADF-funded social infrastructure projects were rated as successful, lower than the 82% success rate for OCR-funded projects. The difference in project outcomes by source of funding was evident in all subsectors. Among ADF-funded projects, urban development performed best (78% success rate in the 1990s), followed by education (70%), water supply and sanitation (69%), and health and population (50%).

The difficulties in achieving successful project outcomes in the social sectors were illustrated in the 2005 social sector evaluation in Pakistan. Relative to the performance of operations in most other sectors in the country, the social sector outcomes were poor. According to the evaluation, only 8% of 24 projects were rated as successful, 58% were partly successful, and 33% were unsuccessful. Although the performance of more recently approved projects was better, performance remains unsatisfactory, particularly in light of ADB’s commitment to managing for development results. Successful water supply/wastewater treatment and education projects are associated with several characteristics as listed in the boxes.

The ratings of projects in the finance sector reflect the performance of loans to government-owned development finance institutions. There are clear differences in the performance of ADF- and OCR-funded lines of credit. Only 26% of the ADF-funded lines were rated successful, while for OCR-funded lines (after excluding projects in graduated economies approved in the 1970s and 1980s) the success rate was 52%. The findings suggest that ADB should refrain from financing lines of credit in ADF countries. Generally, the strength of financial institutions, depth of the financial markets, and quality of regulation improve as countries develop. ADB’s current approach is to focus on finance sector reform rather than channeling funds through development finance institutions.

Box 11: Characteristics of Successful Education Projects

• The projects were consistent with the education strategies of DMCs.

• Borrowers made counterpart funds available as required and complied with loan covenants.

• Committed executing agencies were able to plan, manage, implement, and monitor the projects and benefited from capacity-building support. Institutional readiness is an important driver of project success.

• A series of projects and consistent ADB involvement over a long period contributed to successful outcomes, particularly when sector/subsector reforms were pursued.

• Participatory approaches were used for project design and implementation and to build alliances and shared ownership by engaging with a broad range of stakeholders.

• Basic and secondary education projects are generally not able to recover costs, so adequate budgetary support is essential for project sustainability; technical, vocational, and higher education projects recovered some costs and generated revenues.

• There was an emphasis on cross-cutting themes, especially poverty reduction and gender concerns.

• Project facilities were well utilized and maintained.

• Consultants and contractors performed well.

• ADB supervision missions resolved implementation issues.

In countries blending ADF and OCR funds, the success rate of OCR-funded projects (66% overall) has been higher than that of ADF-funded projects (52%). That held for projects approved in the 1970s, 1980s, and 1990s, with no evidence of a convergence in success rates. A significant difference in sector mix provides part of the explanation. Agriculture and the social sectors, which have had modest success rates, account for 79% of ADF projects in blend countries compared with 42% of OCR projects. In contrast, transport and energy, sectors with high success rates, account for 45% of OCR projects but only 18% of ADF projects.

The 2006 Annual Evaluation Review examined the characteristics of successful projects in five sectors: roads, power, water supply and sanitation, education, and irrigation and drainage. Naturally some of the characteristics noted are sector specific, but a number cut across several sectors. Some factors are dependent on the local situation, and some on the situation at ADB. Strong government ownership of a project is important. This translates into commitment and good performance of the executing agency, and provision of adequate counterpart funds for implementation and for operation and maintenance. The executing agency must be able to work together with the consultants, contractors, and ADB staff; must be familiar with ADB requirements; and must be able to learn from previous projects. For projects involving public works, the performance of supervision consultants and contractors is crucial. Continuity of ADB engagement in the country and sector, and of ADB staff involvement—ADB’s long-term commitment—is salient. Also on the ADB side, quality at entry is a recurrent theme, covering quality of project preparatory TA, participatory approaches to project design, a flexible design that allows a project to be adjusted during implementation, and the incorporation of lessons learned from previous projects. Regular supervision and project review missions from ADB were found to contribute to project success.

Program Lending. While project lending has constituted the majority of ADB’s assistance, program loans are also an important instrument and were used extensively in response to the 1997 financial and economic crisis. Program loans are disbursed relatively quickly to cover the adjustment costs of policy reforms. Program loans have been made to 31 countries, although most program lending is concentrated in a few. From 1978 to 2002, 133 program loans were approved for a combined ADF and OCR total of $18.1 billion. Of these, 92 have been evaluated: 48% were rated successful, 49% partly successful, and 3% unsuccessful. Most of the program loans approved after 2002 are still being implemented and have not been evaluated.

The trend in program ratings, by year of approval, is depicted in Figure 2. The success rates were unacceptably low for program loans approved through 1991, particularly regarding those approved from 1989 to 1991, when none of the 15 programs was rated successful.¹⁹ However, the success rate improved for programs approved after 1991, reflecting the positive effect of initiatives to improve quality, greater experience with the use of a policy-based lending modality, and revision of ADB’s program lending policies in 1996.

Figure 2: Trend in Program Ratings by Year of Approval (% Success Rate Based on 3-Year Moving Average)

BOP = balance of payments.

Source: OED.

Program loans are a more common lending modality in ADF countries than in OCR countries. Nearly 75% of program loans rated (68 of 92) were ADF funded. This is not surprising, as the need for policy and institutional reform is greatest in countries that are eligible to borrow from ADF. The analysis confirms that country factors, such as economic performance, development priorities, quality of governance, and strength of institutions, contribute to the success rates of programs. In all, 67% of the OCR-funded programs have been rated successful compared with 41% of ADF-funded programs. These results highlight the difficulty of formulating successful program loans in ADF countries.

Program lending has been most commonly used in the agriculture/natural resources and finance sectors and has rarely been used in the transport/communications and energy sectors. However, the sector distribution has changed significantly. More than 75% of the 78 program loans approved since 1996 have been concentrated in three sectors: law/economic management/public policy, finance, and multisector. The success ratings by sector and source of funding presented in Table 3 are difficult to compare because of the small numbers of OCR-funded program loans in most sectors. Most agriculture, industry, and social infrastructure program loans were in ADF-eligible countries, and there was no OCR-funded transport program loan. On average, ADF-funded agriculture programs performed poorly (worse than agriculture projects), with a low success rate of 30%; however, from 1995 onward, significant improvements were achieved. The finance sector is the one sector that had a comparable number of program loans funded by OCR and ADF. In this sector, the success rate for ADF-funded program loans (53%) was significantly lower than for those funded through OCR (90%). The disappointing performance of some program lending has been attributed to (i) the absence of government commitment to the principles and concepts underlying the reforms, (ii) the lack of a comprehensive sector analysis prior to the formulation of the policy reform package, (iii) a lack of clarity in program objectives and policy measures and their interrelationships, (iv) inadequate capacity of the executing agencies to carry out the required studies and monitoring, and (v) unsustainable policy reforms. Sometimes, the reform agenda was overoptimistic about what could be achieved during the time frame of the program. Delays in tranche releases were common.

Table 3: Program Loan Performance by Sector and Source of Financing (% of Program Rated Successful, Loans Approved from 1978 to 2002)

– = not applicable.

Source: OED.

The 2006 Annual Evaluation Review also reported on two themes that are consistent with the commitments made by ADB to ADF donors: (i) relating staff incentives to achieving development results; and (ii) factors contributing to project quality, including governance.

Relating Staff Incentives to Achieving Development Results. The 2006 Annual Evaluation Review was the first systematic attempt at ADB to examine staff accountability for project success. It confirmed that ADB’s current formal and informal incentives reward loan processing and approval, with insufficient focus on project administration, project success, or achieving medium-term results. Adjusting ADB’s incentive systems to encourage more focus on project results and success is consistent with commitments made in both the 2004 human resources strategy and the second medium-term strategy. The question is not whether to change the systems, but how to do it.

The present staff performance evaluation system needs to be complemented by a chain of accountability for achieving development results that begins at the top with ADB’s Management and senior staff and cascades down to all staff. Incentives need to be fully aligned with accountability throughout the chain. The 2006 Annual Evaluation Review recommended that ADB explore the feasibility of providing stronger incentives for staff to focus on project quality at entry and on project administration and supervision. Initial work could include an assessment of best practices in comparable institutions and developing indicators for measuring the achievement of development results. ADB’s Management agreed to examine the feasibility of this approach.

Factors Contributing to Project Quality, including Governance. Good governance and the control of corruption have become important parts of the international development agenda. However, the 2006 Annual Evaluation Review concluded that the relationship between governance and development effectiveness is complex and not straightforward. There are many definitions of governance, and there are many measurement problems that make data less than fully reliable. Analysis of the relationship between measures of governance and macroeconomic indicators suggests the following:

• The Asia and Pacific region is the fastest-growing economic region in the world, but governance is weak in many countries according to standard governance indicators.

• Although there is a strong, positive relationship between the level of economic development and governance variables when a worldwide database is used, the relationships are much weaker if only countries in which the per capita gross domestic product is lower than $5,000 are analyzed.

• No significant relationship can be found between good governance and the rate of economic growth from 1996 to 2004. Some countries in the region with relatively good governance had lower growth rates than countries with lower governance ratings.

The analysis of projects approved in the 1990s suggests evidence of a relationship between some of the dimensions of governance and good project outcomes; however, it is not clear how strong or robust the relationships are. Other factors are also important, perhaps more so than governance, in determining project success (e.g., sector, country characteristics, macroeconomic climate, country ownership, capacity of the executing agency). The relationships between good governance at the macro level and project success may be somewhat weaker than is conventionally assumed. It may be that governance issues at the sector level have a more direct bearing on project success.

Subsequent to ninth replenishment of ADF, ADB harmonized its performance-based allocation procedures with those of the World Bank. ADB’s 2004 performance-based allocation policy for ADF resources increased the weight given to governance in measuring country performance from 30% to 50%, although this is still lower than the 66% assigned in the World Bank’s formula. There was, however, no analysis undertaken by ADB that demonstrated that good governance is the key binding constraint for development in all DMCs and that it merits such a high weight in the performance-based allocation formula. ADB has not undertaken a rigorous study that links good governance, as a causal factor, to economic growth, poverty reduction, development results, portfolio performance, or project success.

The 2006 Annual Evaluation Review recommended that ADB undertake such a study. Depending on the results, consideration may have to be given to including sectoral governance variables in the formula or to reducing the weight for governance in the performance-based allocation formula. Benchmarking a country’s governance performance and then rewarding progress in improving governance would be more consistent with managing for development results than making major decisions on ADF allocations based on the governance score in any one year. Steps also need to be taken to improve the consistency of what is meant by governance in ADB’s various governance-related policies and country governance assessments. Reducing the weight might be perceived as sending the wrong message, given that ADB has recently adopted the second medium-term strategy, which places priority on good governance and controlling corruption. However, this must be balanced against the potential harm that may be caused to DMCs whose ADF allocation is reduced because of a score on a variable that is difficult to define, measure, and compare over time and across countries.

Clearly, good governance and efforts to control corruption are important. The issues identified by the review relate to a lack of clarity of definition, difficulties of measurement, a lack of rigorous analysis of the subject at ADB, and questions about whether governance is the most important binding constraint to development in all countries. These issues are important, because in the performance-based allocation formula, ADB purports to be able to accurately define and measure governance on a numeric scale and uses the results to allocate more or less ADF funding to DMCs.

Technical Assistance. OED maintains a database of evaluation findings from TA performance evaluation reports and project/program performance evaluation reports for TA associated with projects/programs. The ratings included in Table 4, dated 2007, are significantly below the 70% success rate that reflects satisfactory outcomes. Overall, 63% of TA operations evaluated in TA performance evaluation reports were rated as being at least successful, and 48% of activities evaluated in project/program performance evaluation reports were rated as successful. The percentage of activities rated as successful in TA and project/program performance evaluation reports has been fairly static over time. For TA activities approved in the 1980s, the portion rated as successful was 44%; it was 58% from 1990 to 1994, 59% from 1995 to 1999, and 60% from 2000 to 2007. There is thus no trend toward more successful TA outcomes over the past 15 years. ADB’s Management needs to develop specific actions to address strategic and management issues concerning TA activities.²⁰

Table 4: Assessment of Technical Assistance Performance

Source	Number of Technical Assistance Operations	% Successful
Technical Assistance Special Evaluation Studies	110	72
Country/Sector Assistance Program Evaluations	113	71
Technical Assistance Performance Evaluation Reports	185	63
Project/Program Performance Evaluation Reports	143	48
Total	551	63

Source: OED.