This Mid Term Evaluation (MTE) is initiated by the UNDP Ukraine as the Implementation Agency for this project and it aims to provide managers (at the Project Implementation Unit, UNDP Ukraine Country Office and UNDP-GEF levels) with strategy and policy options for more effectively and efficiently achieving the project’s expected results and for replicating the results. It also provides the basis for learning and accountability for managers and stakeholders.
This evaluation is to be undertaken taking into consideration the GEF Monitoring and Evaluation policy (http://thegef.org/MonitoringandEvaluation/MEPoliciesProcedures/mepoliciesprocedures.html and the UNDP-GEF Monitoring and Evaluation Policy (http://www.undp.org/gef/05/monitoring/policies.html).
The MTE is intended to identify potential project design problems, assess progress towards the achievement of objective, identify and document lessons learned (including lessons that might improve design and implementation of other UNDP-GEF projects), and to make recommendations regarding specific actions that might be taken to improve the project. It is expected to serve as a tool of validating or filling the gaps in the initial assessment of relevance, effectiveness and efficiency obtained from monitoring. The MTE provides the opportunity to assess early signs of project success or failure and prompt necessary adjustments.
The evaluation will play a critical role in the future implementation of the project by providing advice on: (i) how to strengthen the adaptive management and monitoring function of the project; (ii) how to ensure accountability for the achievement of the GEF objective; (iii) how to enhance organizational and development learning; and (iv) how to enable informed decision - making.
The evaluation will have to provide to the GEF Secretariat with complete and convincing evidence to support its findings/ratings. The evaluator should prepare specific ratings on specific aspects of the project, as described in the section IV of this Terms of Reference. Particular emphasis should be put on the current project results and the possibility of achieving the objective and outcomes in the established timeframe, taking into consideration the speed, at which the project is proceeding.
PROJECT OVERVIEW
Removing Barriers to Greenhouse Gas Emissions Mitigation through Energy Efficiency in the District Heating System, Phase 2 project has been launched in June 2006, following successful completion of the activities under Phase 1 “Climate Change Mitigation in Ukraine through Energy Efficiency in Municipal District Heating (Pilot Project in Rivne)” implemented in 2002-2005 GEF-funded project with the objective of addressing a key issue of reducing greenhouse gas emissions through large-scale improvements in energy efficiency in Ukraine's communal heat supply sector. Phase 1 significant achievements allowed satisfying the criteria that needed to be met prior to accessing funding for Phase 2.
The project objective is to reduce overall fossil fuel consumption and associated greenhouse gas (GHG) emissions by removing barriers to supply and demand side energy efficiency improvements in district heating systems in the main cities of Ukraine. GEF participation will reduce major existing barriers in one pilot city and provide for the replication of defined approaches and measures in other main cities of Ukraine.
The project’s goal is to address a key issue in the reduction of greenhouse gas emissions through large-scale improvements in energy efficiency in Ukraine's communal heat supply sector. This is proposed to be achieved through the setting up of a municipal district heating ESCO in Rivne City, providing it with technical and capacity development support to become fully operational, to provide it with hands-on experience through the implementation of pilot projects and to assist it in preparing a strategic plan for implementing energy efficiency activities city-wide in Rivne. All these were undertaken during Phase 1.
Under Phase 2, ESCO Rivne receiving additional support to implement a city-wide programme for energy efficiency activities in municipal district heating, to expand its reach beyond the borders of the city, to implement activities which require longer payback periods for which financing may not be readily available, to design and implement finance guarantee and risk mitigation instruments, and to promote the adoption and implementation of project experience and lessons learned throughout Ukraine and in other CIS countries.
The global environmental is to be achieved through the removal of the following main existing barriers to energy efficiency improvement in the communal heat supply sector:
1) difficulties in arranging financing for efficiency projects;
2) institutional constraints;
3) lack of capacity and experience in preparing, implementing and managing energy efficiency projects;
4) high transaction costs for relatively small energy efficiency projects; 5) lack of information about existing opportunities for energy efficiency.
In line with the Government’s priorities, this project addresses a key issue in the reduction of greenhouse gas emissions through large-scale improvements in energy efficiency in Ukraine's communal heat supply sector. These improvements is to result from a four-part approach:
1) capacity building to create the basis for systematic energy efficiency activities at the local level;
2) an integrated approach of supply and demand-side improvements to achieve maximum fuel savings and emissions reduction;
3) attraction of external investment resources for an energy efficiency programme in a pilot city (Rivne); and,
4) project-specific replication measures in other parts of the country, including development of relevant procedures, guidelines, information materials and their dissemination, and public awareness-raising through the involvement of NGO's, in particular those concerned with environmental and energy efficiency problems. The project will involve establishing a privately-run energy service company (ESCO) as an innovative mechanism for financing energy efficiency activities in Ukraine at both municipal and regional levels, and, eventually, at the national level.
The Project has four primary outcomes summarized below: - Outcome 1: ESCO operations are expanded to cover Rivne City-wide/Oblast energy efficiency activities.
- Outcome 2: ESCO-Rivne operations are facilitated through financing of activities having long payback
- Outcome 3: ESCO-Rivne operations are promoted through the design of instruments to minimise investment
- Outcome 4: Project experience/best practices and lessons learned are replicated throughout Ukraine and in other CIS countries.
In accordance with UNDP/GEF M&E policies and procedures, all projects with long implementation period are strongly encouraged to conduct mid-term evaluations. In addition to providing an independent in-depth review of implementation progress, this type of evaluation is responsive to GEF Council decisions on transparency and better access to information during implementation.
EVALUATION OBJECTIVES
Objectives, scope, methodology and other information relevant to mid-term evaluation is accessible through Terms of Reference for Mid-Term Evaluation of UNDP-GEF project PIMS 3056 CCFSP REMOVING BARRIERS TO GHG EMISSIONS MITIGATION THROUGH ENERGY EFFICIENCY IN THE DISTRICT HEATING SYSTEM, PHASE 2
The MTE is initiated by UNDP Country Office in Ukraine in line with the UNDP-GEF M&E guidelines in order to assess the overall project progress, make sure the project is on track to deliver the agreed outcomes, and produce recommendations on any adjustments needed.
The Monitoring and Evaluation (M&E) policy at the project level in UNDP/GEF has four objectives: i) to monitor and evaluate results and impacts; ii) to provide a basis for decision making on necessary amendments and improvements; iii) to promote accountability for resource use; and iv) to document, provide feedback on, and disseminate lessons learned. A combination of tools should be used to ensure effective project M&E. These might be applied continuously throughout the lifetime of the project, e.g. periodic monitoring of indicators -, or as specific time-bound exercises such as mid-term review, audit reports and independent evaluations.
The purposes of the MTE are: - To assess overall performance against the project objective and outcomes as set out in the Project Document, project’s Logical Framework, and other related documents;
- To assess the effectiveness and efficiency of the project;
- To analyze critically the implementation and management arrangements of the project;
- To assess the progress to date towards achievement of the outcomes;
- To review planned strategies and plans for achieving the overall objective of the project within the timeframe;
- To assess the sustainability of the project’s interventions;
- To list and document initial lessons concerning project design, implementation and management;
- To assess project relevance to national priorities;
- To provide guidance for the future project activities and, if necessary, for the implementation and management arrangements;
In particular, this evaluation will assess progress in establishing the information baseline, reducing threats, and identifying any difficulties in project implementation and their causes, and recommend corrective course of action. Effective action to rectify any identified issues hindering implementation will be a requirement prior to determining whether implementation should precede.
Project performance will be measured based on Project’s Logical Framework Matrix (see Annex 3), which provides clear performance and impact indicators for project implementation along with their corresponding means of verification. Success and failure will be determined in part by monitoring changes in baseline conditions.
The evaluation team is expected to work with key project stakeholders, including UNDP Country Office in Ukraine, Rivne municipality and CJSC “ESCO-Rivne”, and members of the Project Steering Committee. |
SCOPE OF THE EVALUATION
The evaluation will focus on the range of aspects described below. In addition to a descriptive assessment, all criteria marked with (R) should be rated using the following divisions: Highly Satisfactory, Satisfactory, Marginally Satisfactory, Marginally Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. All ratings given should be properly substantiated:
1. Project concept/design, relevance and strategy
1.1 Project relevance, country ownership/drivenness (R): the extent to which the project is suited to local and national development priorities and organizational policies, including changes over time as well as the extent the activities contribute towards attainment of global environmental benefits: - Is the project concept in line with the sectoral and development priorities and plans of the country?
- Are project outcomes contributing to national development priorities and plans?
- How and why project outcomes and strategies contribute to the achievement of the expected results.
- Examine their relevance and whether they provide the most effective way towards results.
- Do the outcomes developed during the inception phase still represent the best project strategy for achieving the project objectives (in light of updated underlying factors)? Consider alternatives.
- Were the relevant country representatives, from government and civil society, involved in the project preparation?
- Does the recipient government maintain its financial commitment to the project? Has the government approved policies or regulatory frameworks in line with the project’s objectives?
1.2 Preparation and readiness: - Are the project’s objective and components clear, practicable and feasible within its timeframe?
- Were the capacities of executing institution and counterparts properly considered when the project was designed?
- Were lessons from other relevant projects properly incorporated in the project design?
- Were the partnership arrangements properly identified and the roles and responsibilities negotiated prior to project approval?
- Were counterpart resources (funding, staff, and facilities), enabling legislation, and adequate project management arrangements in place at project entry?
1.3 Stakeholder involvement (R): - Did the project involve the relevant stakeholders through information-sharing, consultation and by seeking their participation in the project’s design?
- Did the project consult and make use of the skills, experience and knowledge of the appropriate government entities, NGOs, community groups, private sector, local governments and academic institutions in the design of project activities?
1.4 Underlying factors/assumptions: - Assess the underlying factors beyond the project’s immediate control that influence outcomes and results. Consider the appropriateness and effectiveness of the project’s management strategies for these factors.
- Re-test the assumptions made by the project management and identify new assumptions that should be made.
- Assess the effect of any incorrect assumptions made by the project.
1.5 Management arrangements (R): - Were the project roles properly assigned during the project design?
- Are the project roles in line with UNDP and GEF programming guidelines?
- Can the management arrangement model suggested by the project be considered as an optimum model? If no, please come up with suggestions and recommendations.
1.6 Project budget and duration (R): - Assess if the project budget and duration were planned in a cost-effective way?
1.7 Design of project M&E system (R): - Examine whether or not the project has a sound M&E plan to monitor results and track progress towards achieving project objectives.
- Examine whether or not the M&E plan includes a baseline (including data, methodology, etc.), SMART indicators and data analysis systems, and evaluation studies at specific times to assess results and adequate funding for M&E activities.
- Examine whether or not the time frame for various M&E activities and standards for outputs are specified.
1.8 Sustainability: - Assess if project sustainability strategy was developed during the project design?
- Assess the relevance of project sustainability strategy
2. Project implementation
2.1 Project’s adaptive management (R): a. Monitoring systems - Assess the monitoring tools currently being used:
- Do they provide the necessary information?
- Do they involve key partners?
- Are they efficient?
- Are additional tools required? - Assess the use of the logical framework as a management tool during implementation and any changes made to it.
- What impact did the retro-fitting of impact indicators have on project management, if such?
- Assess whether or not M&E system facilitates timely tracking of progress towards project’s objectives by collecting information on chosen indicators continually; annual project reports are complete, accurate and with well justified ratings; the information provided by the M&E system is used to improve project performance and to adapt to changing needs.
b. Risk Management - Validate whether the risks identified in the project document and PIRs are the most important and whether the risk ratings applied are appropriate. If not, explain why.
- Describe any additional risks identified and suggest risk ratings and possible risk management strategies to be adopted.
- Assess the project’s risk identification and management systems:
Is the UNDP-GEF Risk Management System appropriately applied?
How can the UNDP-GEF Risk Management System be used to strengthen the project management?
c. Work Planning - Assess the use of routinely updated work plans.
- Assess the use of electronic information technologies to support implementation, participation and monitoring, as well as other project activities.
- Are work planning processes result-based ? If not, suggest ways to re-orientate work planning.
d. Financial management - Consider the financial management of the project, with specific reference to the cost-effectiveness of interventions. (Cost-effectiveness: the extent to which results have been delivered with the least costly resources possible.). Any irregularities must be noted.
- Is there due diligence in the management of funds and financial audits?
- Did promised co-financing materialize (please fill out the co-financing form provided in Annex 1)?
e. Reporting - Assess how adaptive management changes have been reported by the project management.
- Assess how lessons derived from the adaptive management process have been documented, shared with key partners and internalized by partners.
f. Delays - Assess if there were delays in project implementation and what were the reasons.
- Did the delay affect the achievement of project’s outcomes and/or sustainability, and if it did then in what ways and through what causal linkages?
2.2 Contribution of Implementing and Executing Agencies: Assess the role of UNDP and Rivne Municipality against the requirements set out in the UNDP Programme and Operations Policies and Procedures . Consider: - Field visits
- Participation in Steering Committees
- Project reviews, PIR preparation and follow-up
- GEF guidance
- Operational support
c. Consider the new UNDP requirements outlined in the UNDP Programme and Operations Policies and Procedures, especially the Project Assurance role, and ensure they are incorporated into the project’s adaptive management framework. d. Assess the contribution to the project from UNDP and Rivne municipality in terms of “soft” assistance (i.e. policy advice & dialogue, advocacy, and coordination). e. Suggest measures to strengthen UNDP’s soft assistance to the project management.
2.3 Stakeholder participation, partnership strategy (R): - Assess whether or not and how local stakeholders participate in project management and decision-making. Include an analysis of the strengths and weaknesses of the approach adopted by the project and suggestions for improvement if necessary.
- Does the project consult and make use of the skills, experience and knowledge of the appropriate government entities, NGOs, community groups, private sector, local governments and academic institutions in the implementation and evaluation of project activities?
- Consider the dissemination of project information to partners and stakeholders and if necessary suggest more appropriate mechanisms.
- Identify opportunities for stronger partnerships.
2.4 Sustainability: - Assess the extent to which the benefits of the project will continue, within or outside the project scope, after it has come to an end; commitment of the government to support the initiative beyond the project.
- The evaluators may look at factors such as mainstreaming project objectives into the broader development policies and sectoral plans and economies.
The sustainability assessment will give special attention to analysis of the risks that are likely to affect the persistence of project outcomes. The sustainability assessment should also explain how other important contextual factors that are not outcomes of the project will affect sustainability. The following four dimensions or aspects of sustainability will be addressed: - Financial resources: Are there any financial risks that may jeopardize sustenance of project outcomes? What is the likelihood of financial and economic resources not being available once the GEF assistance ends (resources can be from multiple sources, such as the public and private sectors, income generating activities, and trends that may indicate that it is likely that in future there will be adequate financial resources for sustaining project’s outcomes)?
- Socio-political: Are there any social or political risks that may jeopardize sustenance of project outcomes? What is the risk that the level of stakeholder ownership (including ownership by governments and other key stakeholders) will be insufficient to allow for the project outcomes/benefits to be sustained? Do the various key stakeholders see that it is in their interest that the project benefits continue to flow? Is there sufficient public / stakeholder awareness in support of the long term objectives of the project?
- Institutional framework and governance: Do the legal frameworks, policies and governance structures and processes pose risks that may jeopardize sustenance of project benefits? While assessing this parameter, also consider if the required systems for accountability and transparency, and the required technical know-how are in place.
- Environmental: Are there any environmental risks that may jeopardize sustenance of project outcomes? The terminal evaluation should assess whether certain activities will pose a threat to the sustainability of the project outcomes.
On each of the dimensions of sustainability of the project outcomes will be rated as follows: - Likely (L): There are no or negligible risks that affect this dimension of sustainability.
- Moderately Likely (ML): There are moderate risks that affect this dimension of sustainability.
- Moderately Unlikely (MU): There are significant risks that affect this dimension of sustainability
- Unlikely (U): There are severe risks that affect this dimension of sustainability.
3. Project results (outputs, outcomes and objectives) 3.1 Progress towards achievement of intended outputs, outcomes/measurement of change:
Progress towards results should be based on a comparison of indicators before and after (so far) the project intervention (legal and regulatory frameworks, results of restoration activities, etc.) to the baseline ones.
The evaluation should specifically look into: - Assessment of ESCO-Rivne company sustainability and potential reliability of ESCO mechanism nationwide;
- Validation of the adequacy and viability of various modalities of ESCO operation (engineering services including energy audit, financial services, utility);
- Validation of the energy efficiency interventions, completed, on-going, or proposed;
- Project’s impact on improving effectiveness of the use of financial resources available for energy efficiency in communal sector.
To determine the level of achievement of project outcomes and objectives following three criteria should be assessed: - Relevance: Are the project’s outcomes consistent with the focal areas/operational program strategies and country priorities?
- Effectiveness: Are the actual project outcomes commensurate with the original or modified project objectives? In case the original or modified expected results are merely outputs/inputs then the evaluators should assess if there are any real outcomes of the project and if yes then whether these are commensurate with the realistic expectations from such a project.
- Efficiency: Is the project cost effective? Is the project the least cost option? Is the project implementation delayed and if it is, then does that affect cost-effectiveness? Wherever possible, the evaluator should also compare the cost-time vs. outcomes relationship of the project with that of other similar projects.
Outcomes and the whole project should be rated as follows for relevance, effectiveness, efficiency: - Highly Satisfactory (HS): The project has no shortcomings in the achievement of its objectives.
- Satisfactory (S): The project has minor shortcomings in the achievement of its objectives.
- Marginally Satisfactory (MS): The project has moderate shortcomings in the achievement of its objectives.
- Marginally Unsatisfactory (MU): The project has significant shortcomings in the achievement of its objectives.
- Unsatisfactory (U): The project has major shortcomings in the achievement of its objectives.
Highly Unsatisfactory (HU): The project has severe shortcomings in the achievement of its objectives.
EVALUATION DELIVERABLES
The core product of the Mid-Term Evaluation will be the Mid-Term Evaluation Report of UNDP-GEF Project PIMS 3056 CCFSP “Removing Barriers to Green Gas Emissions Mitigation Through Energy Efficiency In The District Heating System, Phase 2”that includes: - Findings with the rating on performance;
- Conclusions drawn;
- Recommendations for improving delivery of project outputs;
- Lessons learned concerning best and worst practices in producing outputs;
- A rating on progress towards outputs.
The report is proposed to adhere to the following basic structure: 1. Executive summary - Brief description of project
- Context and purpose of the evaluation
- Main conclusions, recommendations and lessons learned
2. Introduction - Project background
- Purpose of the evaluation
- Key issues to be addressed
- The outputs of the evaluation and how will they be used
- Methodology of the evaluation
- Structure of the evaluation
3. The project and its development context - Project start and its duration
- Implementation status
- Problems that the project seeks to address
- Immediate and development objectives of the project
- Main stakeholders
- Results expected
- Analysis of the situation with regard to outcomes, outputs and partnership strategy
4. Findings and Conclusions
4.1 Project formulation - Project relevance
- Implementation approach
- Country ownership/Driveness
- Stakeholder participation
- Replication approach
- Cost-effectiveness
- Sustainability
- Linkages between project and other interventions within the sector
- Management arrangements
4.2 Project implementation Financial management - Monitoring and evaluation
- Management and coordination
- Identification and management of risks (adaptive management)
4.3 Results - Attainment of outputs, outcomes and objectives
- Project Impact
- Prospects of sustainability
5. Conclusions and recommendations - Findings
- Corrective actions for the design, duration, implementation, monitoring and evaluation of the project
- Actions to strengthen or reinforce benefits from the project
- Proposals for future directions underlining main objectives
- Suggestions for strengthening ownership, management of potential risks
6. Lessons learned - Good practices and lessons learned in addressing issues relating to effectiveness, efficiency and relevance
7. Annexes - Evaluation TOR
- Itinerary
- List of persons interviewed
- Summary of field visits
- List of documents reviewed
- Questionnaire used (if any) and summary of results
- Comments by stakeholders (only in case of discrepancies with evaluation findings and conclusions)
The expected length of the report is around 50 pages in total. The first draft of the report is expected to be submitted to the UNDP Country Office in Ukraine within 2 weeks of the in-country mission for subsequent circulation to the key project stakeholders for comments. Any discrepancies between the interpretations and findings of the evaluator and the key project stakeholders will be explained in an annex to the final report.
All deliverables should be submitted to UNDP by the Consultant in English.
METHODOLOGY
Evaluators should seek guidance for their work in the following materials, which could be found at (www.undp.org/gef): - UNDP Handbook on Monitoring and Evaluation for Results
- UNDP/GEF M&E Resource Kit
- Measuring Results of the GEF Biodiversity Programme
It is recommended that the evaluation methodology include the following: - Documentation review (desk study), to include Project Document, GEF Project Implementation Reviews, Minutes of the Project Steering Committee meetings, GEF quarterly project updates, ESCO-Rivne documentation regarding energy efficiency projects development and monitoring, and other relevant national legislative and policy documents;
- Interviews with Project Management Unit and key project stakeholders, including UNDP Country Office in Ukraine, GEF Regional Coordination Unit in Bratislava, Rivne municipality, CJSC “ESCO-Rivne” staff, and other stakeholders, as necessary;
- In-country field visits.
EVALUATION TEAM
The mid-term evaluation will be undertaken by a team composed of two independent experts: International Consultant (Team Leader) and a Local Consultant. They will receive the support of UNDP Country Office and Project Management Team. The evaluators selected should not have participated in the project preparation and/or implementation and should not have conflict of interest with project related activities. The evaluators shall have prior experience in evaluating similar projects. Former cooperation with GEF is an advantage.
The International Consultant - Team Leader will be responsible to deliver the expected output of the mission. Specifically, he/she will perform the following tasks: - Lead and manage the evaluation process;
- Design the detailed evaluation scope and methodology (including the methods for data collection and analysis);
- Ensures full coordination of evaluation process;
- Conduct desk-reviews, interviews and site-visits in order to obtain objective and verifiable data to substantive evaluation ratings and assessments, including:
Assessment of adequacy of the level and proposed modes of enforcement of the regulatory and programmatic documents developed within the project for creation of an enabling environment for energy efficiency in the state sector;
- Verification of the Management Effectiveness Tracking Tool data, as collected and reported by the project;
- Validation of the adequacy and viability of the Energy Centre’s investment;
- Validation of the proposed economic incentives for budgetary organizations to introduce energy efficiency measures;
- Ensures that evaluation report inputs are available in a timely manner;
- Assumes overall responsibility for the evaluation report submission (draft and final):
- Draft the evaluation report and share with the key stakeholders for comments;
- Finalize the evaluation report based on the inputs from key stakeholders.
MANAGEMENT ARRANGEMENTS
The principal responsibility for managing this evaluation lies with UNDP Country Office in Ukraine. It will be responsible for liaising with the project team to set up the stakeholder interviews, arrange the field visits, coordinate with the Government.
These Terms of Reference follow the UNDP-GEF policies and procedures, and together with the final agenda will be agreed upon by the UNDP-GEF Regional Coordinating Unit, UNDP Country Office in Ukraine and Rivne municipality. These three parties will receive a draft of the final evaluation report and provide comments on it prior to its completion.
The evaluation mission will take place in June-July 2009. The total duration of the assignment will be 20 working days. The following timetable is recommended for the evaluation:
Desk review, development of methodology 3 days
In-country field visits, interviews 7 days
Drafting report 3 days
Draft report circulation 5 days
Finalization of report 2 days
The final version of the evaluation report should be submitted in electronic format (MS Word) to UNDP Country Office in Ukraine no later than July 31, 2009. The hard copy should be posted as well. |
