| Description |
LOCAL CONSULTANT FOR PROJECT FINAL EVALUATION - PUBLIC SECTOR ENERGY EFFICIENCY PROGRAMME (HUNGARY) | | Location : | Bratislava / home based with mission to Hungary, SLOVAK REPUBLIC | | Application Deadline : | 10-Feb-09 | | Type of Contract : | SSA | Languages Required :
| English | | Expected Duration of Assignment : | 17 working days (February - March 2009) | | Background | 1. INTRODUCTION
(a) The Monitoring and Evaluation (M&E) policy at the project level in UNDP/GEF has four objectives: i) to monitor and evaluate results and impacts; ii) to provide a basis for decision making on necessary amendments and improvements; iii) to promote accountability for resource use; and iii) to document, provide feedback on, and disseminate lessons learned. A mix of tools is used to ensure effective project M&E. These might be applied continuously throughout the lifetime of the project – e.g. periodic monitoring of indicators -, or as specific time-bound exercises such as mid-term reviews, audit reports and final evaluations.
In accordance with UNDP/GEF M&E policies and procedures, all regular and medium-sized projects supported by the GEF should undergo a final evaluation upon completion of implementation. A final evaluation of a GEF-funded project (or previous phase) is required before a concept proposal for additional funding (or subsequent phases of the same project) can be considered for inclusion in a GEF work program. However, a final evaluation is not an appraisal of the follow-up phase.
The final evaluations are intended to assess the relevance, performance and success of the project. It looks at impact and sustainability of results, including the contribution to capacity development and the achievement of global environmental goals. It will also identify/document lessons learned and make recommendations that might improve design and implementation of other UNDP/GEF projects.
The final evaluation is to be undertaken in accordance with the “GEF Monitoring and Evaluation Policy”(see http://thegef.org/MonitoringandEvaluation/MEPoliciesProcedures/mepoliciesprocedures.html).
The final evaluation is intended to assess the relevance, performance, management arrangements and success of the project. It looks at signs of potential impact and sustainability of results, including the contribution to capacity development and the achievement of global and national environmental goals.
Furthermore the final evaluation also identifies/documents lessons learned and makes recommendations that project partners and stakeholders might use to improve the design and implementation of other related projects and programs.
(b) PROJECT DESCRIPTION
In years 2000 - 2008 the Energy Centre Hungary implemented the GEF/Hungarian Government funded medium-sized project titled Public Sector Energy Efficiency Programme in Hungary. The project was operationally closed in June 2008.
The objective of the project was to mitigate Hungary’s greenhouse gas emissions by improving the efficiency of energy use in public sector buildings and installations and to help building of capacity in municipalities to improve energy management, not only through investment projects, but also through improved energy management of existing plant and equipment.
This global objective was planned to be achieved in several ways: firstly through strengthened outreach to municipalities and strengthened local networks of energy advice centres; secondly through improving the knowledge base of municipal decision makers and energy managers through training; and thirdly through support for energy audits and feasibility studies which will identify both no and low cost measures to improve energy efficiency, and measures for which investment is required.
The institution building objective was aimed to be achieved through the funding of project staff in the Energy Centre, who will support not only the specific implementation tasks of the project, but the overall aim of expanding the range of activities, experience, and knowledge base of the Energy Centre, thus creating strong capability and expertise for monitoring, evaluation, analysis, and feedback.
The project also intended to build capacity in the still underdeveloped energy efficiency services industry in Hungary, and to encourage the development of standards that will benefit the long term growth of the industry.
The immediate objectives and outputs of the Projects were the followings:
Objective 1. Improve the development of energy efficiency policy, increase awareness, and improve coordination of energy efficiency programmes
Output 1.1 Preparation for effective project implementation
Output 1.2 Improved coordination of energy efficiency policy
Output 1.3 Strengthened outreach to municipalities and municipal energy supply companies, and strengthened local networks.
Output 1.4 Increase in the number of municipal energy managers
Output 1.5 Development and operation of monitoring, evaluation and feedback mechanisms.
Output 1.6 Sustainable development of the project.
Objective 2. The identification, development, and financing of energy efficiency projects in Hungarian municipalities/ municipal district heating systems.
Output 2.1 Increase in the number of energy audits and feasibility studies undertaken in municipalities/ district heating companies through a fund for energy audits/ feasibility studies
Output 2.2 Establishment of a national standard for energy audits and programme of certification of energy auditors.
Output 2.3 Establishment of a ‘one/stop shop’ common database for applications for financing of energy efficiency projects.
Objective 3. Improve the knowledge base of municipal decision makers and municipal energy users concerning energy management and energy efficiency technologies.
Output 3.1 Improve the knowledge base of municipal decision makers and municipal energy users concerning energy management and energy efficiency technologies.
Output 3.2 Improve knowledge base for decision makers in municipalities.
Output 3.3 Increased capacity to identify, design, implement and manage energy efficiency projects at the local level.
The Project was executed by the Ministry of Economy and Transport (former Ministry of Economic Affairs) of the Republic of Hungary and was implemented by the Energy Centre Hungary. The executing agency was accountable to UNDP for the production of outputs, for the achievement of project objectives and for the use of UNDP resources. The project was managed on a day to day basis by the Project Manager, who was working under the responsibility of the Programme Director of the Energy Centre Hungary.
The Project coordination was helped by the involvement of the National Steering Committee (composed of the representatives of the key ministries, the National Bank of Hungary, the implementing authority, and UNDP/GEF) responsible for supervision, control, policy guidance, and coordination and the Consultative Forum (a wider group of stakeholders, including NGOs, representatives of municipality organisations, and the energy efficiency industry) formed by the Ministry of Economy and Transport responsible for providing input on related activities and dissemination of the Project.
The designed total project budget was 16.65 – 20.65 M USD, including 4,200,000 USD GEF, 400,000 USD UNDP/ TRAC funding. Originally, (in the Project Document) the financial sources were planned to be composed by GEF grant, UNDP/ TRAC grant, and parallel financing from Government funds and private investors.
At the end of the project the total budget disbursed was over 27 M USD due to increased amount of municipal and private sector investment as well as new grants received.
The geographical scope of the project was not restricted to a specific area in Hungary.
The direct beneficiaries of the project were the Hungarian municipalities, the energy service companies, the Energy Centre Hungary and the local energy advice centres, the national policy makers in the Ministry of Economy and Transport, the Ministry of Environment and Water, and other relevant Ministries, moreover the Hungarian population. | Duties and Responsibilities | 2. OBJECTIVES OF THE EVALUATION
The objective of the Evaluation is to assess the achievement of project objective, the affecting factors, the broader project impact and the contribution to the general goal/strategy, and the project partnership strategy.
The Evaluation will focus on the following aspects:
Project design and its relevance in relation to:
a) Development priorities at the national level;
b) Stakeholders – assess if the specific needs were met;
c) Country ownership / drivenness – participation and commitments of government, local authorities, public services, utilities, residents;
d) UNDP mission to promote sustainable human development (SHD) by assisting the country to build its capacities in the focal area of environmental protection and management;
Performance - look at the progress that has been made by the project relative to the achievement of its objective and outcomes;
a) Effectiveness - extent to which the project has achieved its objectives and the desired outcomes, and the overall contribution of the project to national strategic objectives;
b) Efficiency - assess efficiency against overall impact of the project for better projection of achievements and benefits resulting from project resources, including an assessment of the different implementation modalities and the cost effectiveness of the utilisation of GEF resources and actual co-financing for the achievement of project results;
c) Timeliness of results,
Management arrangements focused on project implementation:
a) General implementation and management - evaluate the adequacy of the project, implementation structure, including the effectiveness of the National Steering Committee and Consultative Forum, partnership strategy and stakeholder involvement from the aspect of compliance to UNDP/GEF requirements and also from the perspective of “good practice model” that could be used for replication
b) Financial accountability – extent to which the sound financial management has been an integral part of achieving project results, with particular reference to adequate reporting, identification of problems and adjustment of activities, budgets and inputs
c) Monitoring and evaluation on project level – assess the adoption of the monitoring and evaluation system during the project implementation, and its internalization by competent authorities and service providers after the completion of the project; focusing to relevance of the performance indicators, that are: - Specific: The system captures the essence of the desired result by clearly and directly relating to achieving an objective and only that objective. - Measurable: The monitoring system and indicators are unambiguously specified so that all parties agree on what it covers and there are practical ways to measure it. - Achievable and Attributable: The system identifies what changes are anticipated as a result of the intervention and whether the result(s) are realistic. Attribution requires that changes in the targeted developmental issue can be linked to the intervention. - Relevant and Realistic: The system establishes levels of performance that are likely to be achieved in a practical manner, and that reflect the expectations of stakeholders.
- Time-bound, Timely, Trackable and Targeted: The system allows progress to be tracked in a cost-effective manner at desired frequency for a set period, with clear identification of particular stakeholders group to be impacted by the project.
• Overall success of the project with regard to the following criteria:
a) Impact - assessment of the results with reference to the development objectives of the project and the achievement of global environmental goals, positive or negative, intended or unintended changes brought about by the project intervention, (number of households benefiting, number of areas with the new technology in place, level of sensitization and awareness about the technology; any change at the policy level that contributes to sustainability of the tested model, impact in private/ public and/ or at individual levels);
e) Global environmental benefits - reductions in green house gas emissions.
b) Sustainability - assessment of the prospects for benefits/activities continuing after the end of the project, static sustainability which refers to the continuous flow of the same benefits to the same target groups; dynamic sustainability use and/or adaptation of the projects’ results by original target groups and/or other target groups;
c) Contribution to capacity development - extent to which the project has empowered target groups and have made possible for the government and local institutions (municipalities) to use the positive experiences; ownership of projects’ results;
d) Replication – analysis of replication potential of the project positive results in country and in the region, outlining of possible funding sources; replication to date without direct intervention of the project;
e) Synergies with other similar projects, funded by the government or other donors.
In addition to a descriptive assessment, all criteria should be rated using the following divisions: Highly Satisfactory, Satisfactory, Marginally Satisfactory, Unsatisfactory with an explanation of the rating. Also the Overall Rating of the project should be indicated.
Issues of special consideration:
The Evaluation Report will review and assessment the methodology for calculating CO2 emission reductions and validate direct and indirect CO2 emission reductions resulting from the project. Consultant should visit and sample set of project investments, summarise each investment in the evaluation report and assess the cost effectiveness of the emission reductions coming from this project. The evaluation should be fully supported by financial and measurement data, In addition to investment data for each of the project investments, this will include a table of planned vs. actual project financial disbursements, and planned co-financing vs. actual co-financing in this project.
For future development support in the region, UNDP is especially interested in the assessment of the support model applied in the project, its implications for the long-term impact and sustainability of the project results.
The Evaluation Report will present recommendations and lessons of broader applicability for follow-up and future support of UNDP and/or the Government, highlighting the best and worst practices in addressing issues relating to the evaluation scope.
3. PRODUCTS EXPECTED FROM THE EVALUATION
The key product expected from this mid-term evaluation is a comprehensive analytical report in English that should, at least, include the following contents:
1. Executive summary
2. Introduction
3. The project(s) and its development context
4. Findings and Conclusions
4.1 Project formulation
4.2 Implementation
4.3 Results
5. Recommendations
6. Lessons learned
7. Annexes
The length of the mid-term evaluation report shall not exceed 30 pages in total (not including annexes).
| Competencies | 4. EVALUATION APPROACH
An outline of an evaluation approach is provided below; however it should be made clear that the evaluator is responsible for revising the approach as necessary. Any changes should be in-line with international criteria and professional norms and standards (as adopted by the UN Evaluation Group – Annex 3). They must be also cleared by UNDP before being applied by the evaluation team.
The evaluation must provide evidence-based information that is credible, reliable and useful. It must be easily understood by project partners and applicable to the remaining period of project duration.
The evaluation should provide as much gender disaggregated data as possible.
The evaluation will take place mainly in the field. The evaluator is expected to follow a participatory and consultative approach ensuring close engagement with the government counterparts, the National Project Manager, Steering Committee, project team, and key stakeholders.
The evaluator is expected to consult all relevant sources of information, such as the project document, project reports – incl. Annual Reports, project budget revision, progress reports, project files, national strategic and legal documents, and any other material that s/he may consider useful for evidence based assessment.
The evaluator is expected to use interviews as a means of collecting data on the relevance, performance and success of the project. S/He is also expected to visit the project sites.
The methodology to be used by the evaluation team should be presented in the report in detail. It shall include information on:
- Documentation reviewed;
- Interviews;
- Field visits;
- Questionnaires;
- Participatory techniques and other approaches for the gathering and analysis of data.
Although the Evaluator should feel free to discuss with the authorities concerned, all matters relevant to its assignment, it is not authorized to make any commitment or statement on behalf of UNDP or GEF or the project management.
The Evaluator should reflect sound accounting procedures and be prudent in using the resources of the evaluation. | Required Skills and Experience | 5. EVALUATION TEAM – QUALITIES AND REQUIREMENTS
A team consisting of an international consultant (already selected) and a local consultant will conduct the evaluation. The evaluators selected should not have participated in the project preparation and/or implementation and should not have conflict of interest with project related activities. The consultants shall have prior experience in evaluating similar projects. Former cooperation with GEF is an advantage.
Team Qualities:
(i) Recent experience with result-based management evaluation methodologies;
(ii) Experience applying participatory monitoring approaches;
(iii) Experience applying SMART indicators and reconstructing or validating baseline scenarios;
(iv) Recent knowledge of the GEF Monitoring and Evaluation Policy;
(v) Recent knowledge of UNDP’s results-based evaluation policies and procedures
- University degree in business, economics or energy/environment related issues;
- Recognized expertise in energy efficiency field
- familiarity with energy efficiency policies and management structures in CEE
- Work experience in relevant areas for at least 10 years;
- Experience with multilateral or bilateral supported conservation projects;
- Project evaluation experiences within United Nations system will be considered an asset;
- Excellent English communication skills;
- Computer literacy;
Specifically, the international consultant (team leader) will perform the following tasks:
- Lead and manage the evaluation mission;
- Design the detailed evaluation scope and methodology (including the methods for data collection and analysis);
- Assist in drafting terms of reference of the national consultant(s)
- Decide the division of labor within the evaluation team;
- Conduct an analysis of the outcome, outputs and partnership strategy (as per the scope of the evaluation described above);
- Draft related parts of the evaluation report; and
- Finalize the whole evaluation report.
The National Consultant will provide input in reviewing all project documentation and will provide the International Consultant with a compilation of information prior to the evaluation mission. Specifically, the national expert will perform tasks with a focus on:
- Review documents;
- Prepare a list of the outputs achieved under project;
- Organize the mission programme and provide translation/interpretation when necessary;
- Participate in the design of the evaluation methodology;
- Conduct an analysis of the outcome, outputs and partnership strategy (as per the scope of the evaluation described above);
- Draft related parts of the evaluation report;
- Assist Team leader in finalizing document through incorporating suggestions received on draft related to his/her assigned sections.
Individual consultants are invited to submit applications together with their CV for this position.
The evaluation will be undertaken in-line with GEF principles :
- Independence
- Impartiality
- Transparency
- Disclosure
- Ethical
- Partnership
- Competencies and Capacities
- Credibility
- Utility
The evaluators must be independent from both the policy-making process and the delivery and management of assistance. Therefore applications will not be considered from evaluators who have had any direct involvement with the design or implementation of the project. This may apply equally to evaluators who are associated with organizations, universities or entities that are, or have been, involved in the biodiversity conservation policy-making process and/or delivery of the project. Any previous association with the project, the Ministry of the Environment, DAPHNE Institute of Applied Ecology, UNDP/GEF Regional Center for Europe and CIS (Bratislava) or other partners/stakeholders must be disclosed in the application. This applies equally to firms submitting proposals as it does to individual evaluators.
If selected, failure to make the above disclosures will be considered just grounds for immediate contract termination, without recompense. In such circumstances, all notes, reports and other documentation produced by the evaluator will be retained by UNDP.
If individual evaluators are selected, UNDP will appoint one Team Leader. The Team Leader will have overall responsibility for the delivery and quality of the evaluation products. Team roles and responsibilities will be reflected in the individual contracts. If a proposal is accepted from a consulting firm, the firm will be held responsible for the delivery and quality of the evaluation products and therefore has responsibility for team management arrangements.
6. IMPLEMENTATION ARRANGEMENTS
The principal responsibility for managing this evaluation lies with UNDP Regional Center for Europe and CIS (Bratislava). UNDP will contract the evaluators and ensure the timely provision of per diems and travel arrangements within the country for the evaluation team. UNDP and Energy Centre Hungary will be responsible for liaising with the Evaluators team to set up stakeholder interviews, arrange field visits, coordinate with the Ministry of Economy, etc.
The activity and timeframe are broken down as follows:
Activities timeframe
Desk review: approximately 4 days
Briefings for evaluators by Energy Centre Hungary and UNDP: approximately 1 day
Field visits, interviews, questionnaires, de-briefings: approximately 5 days
Drafting of the evaluation report: approximately 2 days
Validation of preliminary findings with stakeholders through circulation of draft reports for comments, meetings and other types of feedback mechanisms: approximately 4 days
Finalization of the evaluation report (incorporating comments received on first draft): approximately 1 day
Working Days:
Technical experts (national consultant) – approximately 17 working days
The proposed date for the in-country mission to Hungary is in February 2009. The assignment is to begin no later than in March 31, 2009.
The evaluation will be conducted within the period of February - March 2009, according to the following plan:
Preparation (home office –February 2009):
- Collection of and acquaintance with the project document and other relevant materials with information about the project;
- Familiarization with relevant policy framework in Hungary;
- Development of methodological instruments for the evaluation;
- Set up the mission dates and detailed mission programme preparation in cooperation with the Project manager. The Project manager will organize the schedule of the mission and will arrange transportation to the consultant; will arrange for translation/interpretation when necessary
- Communication with the PMU to clarify matters
Mission to Hungary (5 working days February2009):
- briefing with the PMU
- visits to municipality(ies)
- meeting with the National Programme Director and stakeholder groups
Elaboration of the draft report (home office - till mid March 2009):
- Additional desk review
- Completing of the draft report
- Presentation of draft report for comments and suggestions
- additional information and further clarification with UNDP, project management and project staff;
Elaboration of the final report (home office till end of March 2009):
- Incorporation of comments and additional findings into the draft report
- Finalization of the report
Timeframe for submission of first draft of the report: within 10 working days after the mission.
The report shall be submitted to the UNDP Country Support Team (Ms. Klara Tothova, address: Grosslingova 35, 811 09 Bratislava, Slovakia, tel.: 00421-2-59337 220, e-mail: klara.tothova@undp.org )
Prior to approval of the final report, a draft version shall be circulated for comments to government counterparts and project management: project manager, National Project Director, Ministry of Environment of the SR, UNDP Country Support Team and UNDP/GEF RTA.
UNDP and the stakeholders will submit comments and suggestions within 5 working days after receiving the draft.
The finalised Evaluation Report shall be submitted latest on March 31, 2009.
If any discrepancies have emerged between impressions and findings of the evaluation team and the aforementioned parties, these should be explained in an annex attached to the final report.
Application procedure
Qualified and interested candidates are requested to preferably apply on-line, no later than February 10, 2009.
The application should contain:
- filled P11 form (the form can be downloaded from http://europeandcis.undp.org/files/hrforms/P11_SC_SSA.doc);
- price quotation indicating the lump sum for the work envisaged in the job description
- short letter of interest
UNDP applies fair and transparent selection process that would take into account the competencies/skills of the applicants as well as their financial proposals.
Qualified women and members of social minorities are encouraged to apply.
UNDP is a non-smoking work environment.
Due to large number of applicants, UNDP regrets that it is unable to inform the unsuccessful candidates about the outcome or status of the recruitment process. |
|
