|
The objective of the evaluation is to:
- Provide a comprehensive and systematic accounting of performance;
- Assess project design, implementation, likelihood of sustainability and possible impacts.
Scope of the Evaluation:
The following questions should be covered by the evaluation:
Project formulation:
- Were the project’s objectives and components clear, practicable and feasible within its time frame?
- Were the capacities of the executing institution(s) and its counterparts properly considered when the project was designed?
- Were lessons from other relevant projects properly incorporated in the project design?
- Were the partnership arrangements properly identified and roles and responsibilities negotiated prior to project approval?
- Were counterpart resources (funding, staff, and facilities), enabling legislation, and adequate project management arrangements in place at project entry?
- Were the project assumptions and risks well articulated in the PIF and project document?
Assumptions and risks:
- An assessment of the stated assumptions and risks, whether they are logical and robust, and have helped to determine activities and planned outputs;
- Externalities (i.e. effects of climate change, global economic crisis, etc.) which are relevant to the findings.
Project implementation:
- The logical framework used during implementation as a management and M&E tool;
- Effective partnerships arrangements established for implementation of the project with relevant stakeholders involved in the country/region;
- Lessons from other relevant projects (e.g., same focal area) incorporated into project implementation Feedback from M&E activities used for adaptive management.
Finance/co-finance:
The evaluation report should clarify the financial particulars of the project, including extent of co-financing across the portfolio. Project cost and funding data should be presented, including annual expenditures. Variances between planned and actual expenditures should be assessed and explained. Observations from financial audits as available should be considered.
The evaluation should include a table that shows planned and actual co-financing commitments. Evaluators during their fact finding efforts should request assistance from the Project Team to fill in the table, and the Evaluator should then follow up through interviews to substantiate. The evaluator should briefly describe the resources the project has leveraged since inception and indicate how these resources are contributing to the project’s ultimate objective.
The evaluator should determine the reasons for differences in the level of expected and actual co-financing, and the extent to which project components supported by external funders was well integrated into the overall project. The evaluation should consider the effect on project outcomes and/or sustainability from the extent of materialization of co-financing.
IA and EA execution:
The evaluator should assess and rate (R) the quality of Implementing Agency execution. The assessment should be established through consideration of the following issues:
- Whether there was an appropriate focus on results by the implementing and executing agencies;
- The adequacy of IA & EA supervision;
- The quality of risk management;
- Responsiveness of the managing parties to significant implementation problems (if any);
- Quality and timeliness of technical support to the project team;
- Candor and realism in supervision reporting;
- Suitability of chosen executing agency for project execution;
- Any salient issues regarding project duration, for instance to note project delays, and how they may have affected project outcomes and sustainability.
Monitoring and evaluation:
The evaluator should assess and rate (R) the quality of monitoring and evaluation. The evaluation team should be expected to deliver an M&E assessment that provides:
- An analysis of the M&E plan at project start up, considering whether baseline conditions, methodology and roles and responsibilities are well articulated. Is the M&E plan well conceived? Is it articulated sufficient to monitor results and track progress toward achieving objectives?
- The quality of M&E plan implementation: Was the M&E plan sufficiently budgeted and funded during project preparation and implementation?
- The effectiveness of monitoring indicators from the project document for measuring progress and performance;
- Compliance with the progress and financial reporting requirements/ schedule, including quality and timeliness of reports;
- The value and effectiveness of the monitoring and evaluation reports and evidence that these were discussed with stakeholders and project staff;
- The extent to which follow-up actions, and/or adaptive management, were taken in response to monitoring reports (PIRs);
- Check to see whether PIR self-evaluation ratings were consistent with the MTE and TE findings. If not, were these discrepancies identified by the project steering committee and addressed?
- Terminal Evaluations for full size projects should also include consideration of the M&E analysis carried out for the mid-term evaluation and whether changes were made to project implementation as a result of the MTE recommendations.
Stakeholder involvement:
The evaluation should include findings on the role and involvement of key project stakeholders. Two aspects can be considered:
- A review of the quality and thoroughness of the stakeholder plan presented in the PIF and project document which should be reviewed for its logic and completeness.
- The level of stakeholder participation during project implementation.
Questions regarding stakeholder participation include:
- Did the project involve the relevant stakeholders through information sharing and consultation and by seeking their participation in project design, implementation, and M&E? For example, did the project implement appropriate outreach and public awareness campaigns?
- Did the project consult with and make use of the skills, experience, and knowledge of the appropriate government entities, nongovernmental organizations, community groups, private sector entities, local governments, and academic institutions in the design, implementation, and evaluation of project activities?
- Were the perspectives of those who would be affected by project decisions, those who could affect the outcomes, and those who could contribute information or other resources to the process taken into account while taking decisions? Were the relevant vulnerable groups and powerful supporters and opponents of the processes properly involved?
Adaptive management:
The evaluation team should take note whether there were changes in the project framework during implementation, why these changes were made and what was the approval process. In addition to determining the reasons for change. The evaluator should also determine how the changes were instigated and how these changes then affected project results. A few key questions to consider:
- Did the project undergo significant changes as a result of recommendations from the mid-term evaluation? Or as a result of other review procedures? Explain the process and implications.
- If the changes were extensive, did they materially change the expected project outcomes?
- Were the project changes articulated in writing and then considered and approved by the project steering committee?
Project results:
Results as measured by broader aspects such as: country ownership, mainstreaming, sustainability, catalytic role and impact.
Country ownership:
- Was the project concept in line with development priorities and plans of the country (or countries)?
- Were the relevant country representatives from government and civil society involved in project implementation, including as part of the project steering committee?
- Was an intergovernmental committee given responsibility to liaise with the project team, recognizing that more than one ministry should be involved?
- Has the government(s), enacted legislation, and/or developed policies and regulations in line with the project’s objectives?
Mainstreaming:
UNDP projects financed by the GEF are key components in UNDP country programming. As such, the objectives and outcomes of the project should conform to UNDP country programme strategies. The section on mainstreaming should assess:
- Whether it is possible to identify and define positive or negative effects of the project on local populations (e.g. income generation/job creation, improved natural resource management arrangements with local groups, improvement in policy frameworks for resource allocation and distribution, regeneration of natural resources for long term sustainability);
- If the project objectives conform to agreed priorities in the UNDP country programme document (CPD) and country programme action plan (CPAP);
- Whether there is evidence that the project outcomes have contributed to better preparations to cope with natural disasters;
- Whether gender issues had been taken into account in project design and implementation, (i.e. project team composition, gender-related aspects of pollution impacts, stakeholder outreach to women’s groups, etc). If so, indicate how.
Sustainability:
The evaluator should assess and rate (R) the overall risks to sustainability. Sustainability is considered to be the likelihood of continued benefits after the GEF project ends. Consequently the assessment of sustainability considers the risks that are likely to affect the continuation of project outcomes. The GEF Guidelines establish four areas for considering risks to sustainability: Financial risks;, socio-economic risk; institutional framework and governance risks; and environmental risks. Each should be separately evaluated and then rated on the likelihood and extent that risks will impede sustainability.
Relevant factors to improve the sustainability of project outcomes include:
- Development and implementation of a sustainability strategy;
- Establishment of the financial and economic instruments and mechanisms to ensure the ongoing flow of benefits once the GEF assistance ends (from the public and private sectors, income generating activities, and market transformations to promote the project’s objectives);
- Development of suitable organizational arrangements by public and/or private sector;
- Development of policy and regulatory frameworks that further the project objectives;
- Incorporation of environmental and ecological factors affecting future flow of benefits;
- Development of appropriate institutional capacity (systems, structures, staff, expertise, etc.);
- Identification and involvement of champions (i.e. individuals in government and civil society who can promote sustainability of project outcomes);
- Achieving social sustainability, for example, by mainstreaming project activities into the economy or community production activities;
- Achieving stakeholders’ consensus regarding courses of action on project activities.
Catalytic effect:
The evaluator should complete the ratings table (R) on whether or not the project has had a catalytic effect. The reviewer should consider the extent to which the project has demonstrated: a) production of a public good, b) demonstration, c) replication, and d) scaling up. Replication can have two aspects, replication proper (lessons and experiences are replicated in different geographic area) or scaling up (lessons and experiences are replicated within the same geographic area but funded by other sources). Examples of replication approaches include:
- Knowledge transfer (i.e., dissemination of lessons through project result documents, training workshops, information exchange, a national and regional forum, etc.);
- Expansion of demonstration projects.
- Capacity building and training of individuals, and institutions to expand the project’s achievements in the country or other regions.
- Use of project-trained individuals, institutions or companies to replicate the project’s outcomes in other regions.
Impact:
The reviewer should discuss the extent to which projects are achieving impacts or are progressing toward the achievement of impacts among the project beneficiaries. Impacts in the context of adaptation projects refer to the extent to which vulnerability to climate change has decreased, as measured by the indictors included in the Results Framework, and other quantitative and qualitative information. Process indicators, such as regulatory and policy changes, can also be used to measure impact.
Conclusions, Recommendations and Lessons:
Conclusions should be comprehensive and balanced, and highlight the strengths, weaknesses and outcomes of the project. They should be well substantiated by the evidence and logically connected to the evaluation findings. They should respond to key evaluation questions and provide insights into the identification of and/or solutions to important problems or issues pertinent to project beneficiaries, UNDP and GEF.
The evaluation report should provide practical, feasible recommendations directed to the intended users of the evaluation about what actions to take and decisions to make. The recommendations should be specifically supported by the evidence and linked to the findings and conclusions around key questions addressed by the evaluation.
The evaluation report should include, if available, lessons that can be taken from the evaluation, including best (and worst) practices that can provide knowledge gained from the particular circumstance (programmatic and evaluation methods used, partnerships, financial leveraging, etc.) that are applicable to other GEF and UNDP interventions.
Outputs:
- An inception report should be prepared by the evaluation team prior to the main evaluation mission. It should detail the evaluators’ understanding of the project being evaluated and why, showing how each evaluation question (detailed in Section 3 of this ToR) will be answered by way of: proposed methods, proposed sources of data and data collection procedures. The inception report should include a proposed schedule of tasks, activities and deliverables, designating a team member with the lead responsibility for each task or product. The inception report should annex the signed code of conduct agreement form
- A draft evaluation report,which includes the evaluation scope and method, findings, conclusions and recommendations. The report should cover the following five major criteria: relevance, efficiency, effectiveness, results and sustainability, applied to a) project formulation b) project implementation and c) project results.
- A final evaluation report.
The draft report is considered complete, in contractual terms, only when it has achieved acceptable standards.
Conduct of work:
An 'evaluation mission' should be scheduled, providing an intensive 10 days to 2 weeks for the evaluation team to hold interviews and visit project sites. The evaluation mission should be planned far enough in advance to enable interviews to be properly set up, especially to request meetings with senior Ministry officials. A detailed plan for the mission should be included in the TE inception report, which should be revised based on CO, project team and OFP inputs.
The evaluation will properly examine and assess the perspectives of the various stakeholders. Interviews should include a wide array of interested persons including civil society, NGOs and the private sector, local ministry officials as relevant, and national ministry officials (in addition to the OFP).
Field visits are expected to the project site or a select sampling if there are multiple sites. The decision on which sites to visit should be done jointly with the CO and project team.
Data analysis should be conducted in a systematic manner to ensure that all the findings, conclusions and recommendations are substantiated by evidence. Appropriate tools should be used to ensure proper analysis (e.g. including a data analysis matrix that records, for each evaluation question/criteria, information and data collected from different sources and with different methodology).
By the end of the evaluation mission and prior to submitting a first draft evaluation report, a wrap up discussion should be organized with the country office and project team to present initial findings and request additional information as needed. A template for the evaluation report will be provided.
Following the review of the draft evaluation report, the evaluation team should indicate how comments have been addressed in the revised evaluation report.
|
