Three main elements to be evaluated are Delivery, Implementation and Finances. Each component will be evaluated using three criteria: effectiveness, efficiency and timeliness.
PROJECT DELIVERY: The TE will assess to what extent the project has achieved its immediate objectives. It will also identify what outputs, impacts and results have been produced and how they have enabled the project to achieve its objectives.
The consultants are required to make assessment of the following issues under each priority area outlined below:
Institutional arrangement
- Preparatory work and implementation strategies
- Consultative processes
- Technical support
- Capacity building initiatives
- Project outputs
- Assumptions and risks
- Project related complementary activities
Outcome, results and impacts
- Efficiency of all project activities under the five major components
- Progress in the achievement of the immediate objectives (include level of indicator achievement when available)
Partnerships
- Assessment of national level involvement and perception
- Assessment of local partnerships, and involvement of stakeholders
- Assessment of collaboration between government, intergovernmental and non-governmental organisations
Assessment of national level involvement and perception
Risk management
- Were problems/constraints, which impacted on successful delivery of the project identified at the project design stage and subsequently as part of the Mid Term Evaluation (MTE)?
- Were there new threats/risks to project success that emerged during project implementation?
- Were both kinds of risk appropriately dealt with?
- Were recommendations arising from the MTE addressed?
Monitoring and Evaluation
- Assess the extent, appropriateness and effectiveness of adaptive management at all levels of the project implementation
- Has there been a monitoring and evaluation framework for the project and how was this developed?
- Is the reporting framework effective/appropriate?
- Is this framework suitable for replication/continuation for any future project support?
Assess the extent, appropriateness and effectiveness of adaptive management at all levels of the project implementation
PROJECT IMPLEMENTATION: The TE will review the project management and implementation arrangements at all levels, in order to provide an opinion on its efficiency and cost effectiveness. This includes:
Processes and administration:
- Project related administration procedures
- Milestones (Log-frame matrix)
- Key decisions and outputs,
- Major project implementation documents prepared with an indication of how the documents and reports have been useful
Project oversight and active engagement by: UNDP and National steering committee
Project execution: Ministry of Natural Resources and Environment (NRE) Project implementation: Department of Marine Park Malaysia (DMPM)
PROJECT FINANCES: How well and cost effectively have financial arrangements of the project worked? This section will focus on the following three priority areas:
Project disbursements
- Provide an overview of actual spending against budget expectations
- Critically analyse disbursements to determine if funds have been applied effectively and efficiently.
Budget procedures
- Did the Project Document provide adequate guidance on how to allocate the budget?
- Review of audits and any issues raised in audits and subsequent adjustments to accommodate audit recommendations;
- Review the changes to fund allocations as a result of budget revisions and provide an opinion on the appropriateness and relevancy of such revisions
Coordination mechanisms
- Evaluate appropriateness and efficiency of coordinating mechanisms between executing agencies and UNDP
- Does the approach represent an effective means of achieving the objectives?
- How can the approach be improved?
METHODOLOGY OF EVALUATION APPROACH
The evaluation will be conducted in a participatory manner through a combination of processes. It is anticipated that the methodology to be used for the TE will include the following:
Review of documentation including but not limited to:-
- Project Document and Project Appraisal Document;
- Project implementation reports (APR/PIR’s);
- Quarterly progress reports and work plans of the various implementation task teams;
- Audit reports;
- Mid Term Evaluation report;
- Biodiversity Tracking Tools;
- M & E Operational Guidelines, all monitoring reports prepared by the project; and
- Financial and Administration guidelines.
- NIM/NGO Audit reports
- Other Malaysia’s UNDP/GEF evaluation reports for biodiversity projects
The following documents will also be available:
- Project operational guidelines, manuals and systems;
- Minutes of the National Steering Committee, Technical Working Committee and other project management meetings;
- Maps;
- List of project publications;
- The GEF and UNDP Implementation Completion Report guidelines; and
- The UNDP Monitoring and Evaluation Frameworks.
Site visits and interviews in the field with stakeholders shall include:
- UNDP staff who have project responsibilities;
- Executing agencies (including but not limited to senior officials and task team/ component leaders: NRE, DMPM;
- The Chair of the National Steering Committee;
- Project stakeholders, to be determined at the inception meeting; and
- Relevant staff of the Department of Marine Park Malaysia
The evaluation will start with an opening meeting with the Project Support Unit, executing agency and UNDP, and a review of the key project documentation including key reports and correspondence. It will include presentations from the various project components, visits to executing and implementing agency offices, interviews with key individuals both within the project, the government, and independent observers of the project and its activities, as well as project personnel. Field visits to project sites will be conducted to view activities first hand. There will be an closing meeting to discuss the findings of the evaluation prior to the submission of the draft final report.
EXPECTED OUTPUTS
The TE evaluators will be expected to produce:
a) An evaluation report, of approximately 30-40 pages, structured along the outline indicated in Annex 1.
- A detailed record of consultations with stakeholders will need to be kept and provided (as part of the information gathered by the evaluators), as an annex to the main report.
- If there are any significant discrepancies between the impressions and findings of the evaluation team and stakeholders, these should be explained in an annex attached to the final report.
b) A Power Point presentation (circa 15-20 slides) covering the key points of the TE.
c) A exit presentation to the executing agency, Project Support Unit and UNDP (precise date to be agreed as part of evaluators contract).
A draft of both a) and b) above should be submitted within two weeks of the end of the in-country component of the evaluators’ mission, and a final copy within two weeks after receiving written comments on the drafts.
The draft and final versions of the products should be submitted to the Project Support Unit, who will be responsible for circulating it to key stakeholders.
DURATION
The evaluation is proposed to take place from 8th April to 19th April 2013. The expected number of working days per consultant is 20 days including 10 working days mission to Malaysia for the international consultant.
DELIVERABLES AND TIMELINE
The TE Team will consist of one International Consultant and one National Consultant. The International Consultant will be the Team Leader. The Team Leader, in close collaboration with the National Consultant, will have the overall responsibility for the quality and timely submission of the deliverables. Specifically, the team of consultants is responsible for submitting the following deliverables to the UNDP Country Office and Project Support Unit:
Deliverables Timeline
- Draft detailed workplan reflecting the work of the international and national consultants To be submitted within 5 days of the signing of the contracts
- Draft Evaluation Report To be submitted within 14 days after the completion of the mission in Malaysia
- Final Evaluation Report To be submitted within 14 days after reception of the draft final report with comments
TERMS OF PAYMENT
- 20% upon submission of the draft detailed workplan
- 40% upon submission of the draft evaluation report
- 40% upon satisfactory completion of the final evaluation report
|