Monitoring and Evaluation

Introduction

This module focuses on monitoring and evaluating projects or activities under a national association of the deaf. Monitoring and Evaluation is checking to make sure the goals are achieved during the project period. Monitoring can be perceived as an ongoing and systematic watching over activities and outputs. Monitoring can document the process and outputs produced in a project or an organization. It steers and improves the project and helps you to learn from it. For example, most projects are designed by using a planning tool called “Logical Framework Approach, LFA”. Evaluation is a systematic examination of a project’s achievements compared to its planned objectives and expected outputs. The reason to evaluate a project is to discover to what extent the project has lived up to the expectations. Another aim of evaluating is to find out what you have learned during the project. For example, could something have been done differently?

Part 1: Monitoring

Monitoring is used to watch over and document the process and outputs produced in a project or an organization. You may also monitor changes in the living conditions of deaf people in general. Normally, those who implement a project are also responsible for monitoring it. These people may be project personnel, though in some cases consultants can be hired. It will always be a good idea to involve the implementing organization’s governing body in decisions regarding monitoring, if there is a project team to take care of the practical work. It is also useful to engage the partner from the North, who may provide suggestions and inputs to the monitoring.

What is Monitoring?

Monitoring is the regular observation and recording of activities taking place in a project or programme. It is a process of routinely gathering information on all aspects of the project. To   monitor is   to   check   on   how   project   activities   are   progressing. Monitoring also involves giving feedback about the progress of the project to the donors, implementors and beneficiaries of the project.

Purpose of Monitoring

Monitoring is very important in project planning and implementation. It is like watching where you are going while riding a bicycle; you can adjust as you go along and ensure that you are on the right track. Monitoring provides information that will be useful in:

• Analysing the situation in the community and its project;
• Determining whether the inputs in the project are well utilized;
• Identifying problems facing the community or project and finding solutions;
• Ensuring all activities are carried out properly by the right people and in time;
• Using lessons from one project experience on to another; and
• Determining whether the way the project was planned is the most appropriate way of solving the problem at hand.

Why do monitoring?

1. To manage and improve a project, and to learn from it

The objective of monitoring and evaluation is to enable you and your partners together to monitor and improve a project, ensuring that your performance is of a high quality. Even when one does not monitor and evaluate systematically, one tends to have an idea about how a project is proceeding, and about whether our organization is developing.

If you have a project aimed at building capacity within your organization, you know how many people sit on the board. You are also likely to know how many members you have got, and how many local chapters of the organization exist around the country. But do you know how the organization has developed over the years? Do you know how many people truly feel affiliated to the organization? What is the situation out in the local chapters? Do they conduct activities for their members? How much income do they generate? Do they take minutes of their meetings? If the data revealed problems in keeping the finances in order, you could adjust the project, providing more training in this subject. Another option might be to change or shut down the project, if it turns out not to produce any development at all.

2. To possess documentation for use in rights work

Monitoring is an important weapon in lobbying and the pursuit of rights. A precondition for effective rights work is an ability to continuously follow and document developments in the deaf community, for example: What percentage of deaf women and men are illiterate, and how does this compare to the general population? How many deaf children attend school? If you know the actual conditions of deaf people, it is much easier to demand improvements and make constructive proposals for how to bring them about. For instance, if your rights work is concerned with integrating deaf children into the education system, it is important to follow trends on the ground, and to count on reliable data. You may not necessarily have to collect such information yourself. Perhaps it already exists, and merely needs to be obtained, or you may lobby the relevant authorities to prepare reliable figures.

Example: When the Ministry of Education launched a literacy campaign, it took an initiative to involve people with disabilities, knowing that many of these were illiterate. The ministry convened various disability organizations to a meeting, asking them for the names of members unable to read and write. Subsequently, organization Alfa went straight back to the office and printed a list of illiterate members right from their database. The other organizations, however, spent several months collecting this information, and never really got off to a start in the literacy campaign, whereas Alfa succeeded in halving the rate of illiteracy among its members.

Example: The National Association of Deaf, NAD, often provides interpreting assistance to the police and other authorities. Its five sign language interpreters are the only ones of their kind in the whole country. However, NAD does not know how often they provide interpreters. This leaves them in a weak position when negotiating with the government about covering the costs of the interpreters’ wages and demanding the training of more interpreters. Had they written down the frequency of their help to the authorities, they would have been able to go to them and say: “We have provided interpreting assistance to the police 62 times in the course of the past year, 18 times to the judicial system, and 47 times to the tax authorities. On average, they spent five hours each time. It is fair that the state should pay the wages of, say, two interpreters.”

3. To check if the project is achieving its planned objectives

Many donors make demands for documentation that the project implemented does in fact produce the benefits that were promised when the funds were applied for. Consequently, it is easier to obtain funds for new interventions, if you can prove that your projects have reached the objectives set. It is also highly motivating for you to know that your projects achieve their goals.

4. To inform members and others

Your members pay a membership fee to your organisation. They want to know how it helps if they are paying a membership fee. If you monitor your activities, you can also publish results of your work. This also helps to attract other donors to support your work.

Ways to monitor

There are many ways to monitor your activities or work.

Monitoring of indicators: To judge whether a project is achieving what it has been designed to achieve.

Registration of participants: A good place to start monitoring is by registering project participants, since the project descriptions often establish that a certain number of deaf persons, usually specified by sex, will take part in activities. To document this, it is a good idea to keep a record of participants.

Follow-up visits to project sites: The project staff will normally carry out follow-up visits to the various project activities if these are not implemented by themselves. Systematic documentation of these exercises is an important part of monitoring.

Project visits: Most projects contemplate prescheduled visits from the partner in the North. The project visit may be an actual element of monitoring itself, but first and foremost, it offers the two parties an opportunity to go through the monitoring data together.

Progress reports: Reports are submitted during all project periods, using information from the monitoring.

Reviews and evaluations: The final evaluation, as well as possible mid-term reviews, will always use and analyse the monitoring data, and the better the monitoring, the better these exercises will be too.

Monitoring using LFA

Most projects are designed using a planning tool called ‘Logical Framework Approach’, LFA. LFA is based on a division into different levels, namely development objective, from one to three immediate objectives, in addition to outputs and activities. The development objectives are the long-term perspective, vision, or motivation for what you do, e.g., “Integration of deaf people into society”. An immediate objective is what you will likely achieve through your project. It needs to be formulated in very specific terms. The outputs are what you ought to guarantee will be in place once the project activities have been performed. Outputs must also be phrased very specifically. The LFA is based on the premise of a logical connection between activities, outputs, immediate objectives, and development objective. This means that the activities must produce the outputs, which are what the project should be able to guarantee. These outputs, in turn, must be very likely to lead to the immediate change.

Indicators

What is an indicator? Projects carried out by the disability movement are often about awareness-raising, advocacy, empowerment, sensitization, capacity building, and the like. These are broad and general terms, which can be understood very differently by the various participants. Project outputs and objectives frequently feature expressions such as “empowerment has taken place” or “awareness has been raised”. It is important to avoid such vague phrases when formulating your immediate objectives and outputs because they are impossible to measure. To make it even clearer and more concrete what we wish to achieve, we establish indicators.

Definition of indicators: Criteria to judge whether a project is achieving what it has been designed to achieve.

It can be helpful to involve several persons in drawing up the indicators. A brainstorming session may produce good ideas, enabling selection of the best ones. Perhaps you can investigate whether other people have made indicators in the same field that you wish to monitor. For example, the government may be using some. By choosing the same, or some that are close, you have a chance to enter a dialogue with the authorities about performance according to those indicators. It is also possible that other NGOs have applied similar indicators. What are their experiences of monitoring those? Is there a chance of cooperating around the follow-up? Since it is the project-implementing organization which has the chief responsibility for monitoring, it must obviously also oversee formulating the indicators, possibly with support and advice from the partner in the North.

Specific: The indicator must be very precise as regards target group, geographical coverage, quantity, and quality. Subjective expressions, such as “in a satisfactory manner”, “good minutes” or “regular meetings” should be avoided, since there may be more than one opinion of what is satisfactory, good, and regular. Certainly, if we insist on using such terms, it must be defined what we mean by them. Likewise, it is important to avoid generalities such as “increased capacity”, “greater self-esteem” or “better understanding”. Here, we certainly cannot resort to percentage figures, as in “the participants’ self-esteem has been enhanced by 45%”. Monitoring such an indicator would be an impossible task. It needs to be rephrased, and we must ponder what might indicate greater self-esteem, as in the example of more satisfied employees.

Examples of an indicator that lacks precision:” The organization has a good financial situation”. This could be rephrased as: “In 2012, NAD has an annual income of USD 30,000 in addition to the funds received from international donors, which represents an increase of 28% compared to 2009.”

Measurable: It must be possible to measure an indicator at reasonable cost and effort, just as reliable information needs to be available. For instance, it makes little sense to launch a major survey costing thousands of dollars to monitor the indicator of a project of, say, USD 100,000. Nor is it appropriate for the project personnel to set aside an enormous amount of their time to monitor an indicator. Consequently, indicators must be designed to be simple and without major costs to monitor. If this is not possible, the indicator must be dropped, and others must be found. As a rule of thumb, the cost of actual monitoring should not exceed 2-5% of a project budget.

Achievable: It must be realistic to reach the target set out in an indicator. If we operate with impossible targets, we will become frustrated when we fail to reach them, and perhaps we then become less enthusiastic. It is also possible to set the target too low. Some organizations have been known to deliberately set out very modest progress in their indicators to be certain to fulfil them, hoping to avoid tough questions from their donors. Here, however, it must be kept in mind that monitoring is a management and learning tool for the project itself. Satisfying the donor is not the main reason for it. Defining indicators with deliberately low targets will fool few donors, and we risk making ourselves self-satisfied.

Example: A project had the indicator of increasing the membership of an organization of deaf by 100%. After project completion, the intake of new members amounted to 55%, and there was a sense of failure, since only half the target had been reached. Nevertheless, most deaf organizations would celebrate a membership increase of 55% during such a brief period. In other words, the project had not really failed, but had set its objectives too high.
Relevant: It hardly helps to come up with a precise indicator, if it is irrelevant to the objective or output whose fulfilment it is to measure. If the aim of your association is to promote the rights of deaf people, then you should not provide activities to blind people.

Time-bound: It is important to know when an indicator should be fulfilled, which is why the time of compliance with its target must be included. It is usually enough to mention the year. It can be useful to define intermediate goals for each year, especially if the project runs over two years. This is also called setting milestones.

An example

Immediate objective: By 2012, NAD has undergone organizational strengthening and has become more sustainable. Indicator: In 2012, NAD has an annual income of USD 30,000 in addition to the funds received from international donors, which represents an increase of 28% compared to 2009.

Specific: The indicator sets out the amount of the organization’s income, and what types of income should be included.

Measurable: If the organization has a reliable accounting system, it will be easy to measure this indicator, and the progress can be followed from year to year. Nor does it cost anything to monitor this.

Achievable: A 28% increase over three years should, given a methodical effort, be a realistic target, but obviously this presupposes knowledge of the organization concerned and the conditions under which it operates.

Relevant: An increase in self-funding must be described as highly relevant to an organization’s sustainability.

Monitoring plan

To facilitate monitoring and avoid overlooking something, it is a good idea to draw up a monitoring plan as early as during project formulation or during rights work planning. The making of the plan will oblige us to check the quality of the indicators and ask ourselves some highly relevant questions:

• Have our indicators been formulated with sufficient accuracy, or is there a need for further definition?
• How and from where should data be gathered?
• How often should this be done?
• When do we monitor?
• Who is responsible for having it done?
• Where do we record the data?

Indicator	Where should it be gathered?	How often should the indicator be monitored?
During the project period, the disability movement in Uganda draws up four legislative bills or amendments. At least two bills or amendments are passed.	Project reports. The National Gazettes.	Once a year
Etc.

Part 2: Evaluation

What is an evaluation? Just as we constantly conduct monitoring on a day-to-day basis, we also keep evaluating. If we have thrown a birthday party, we ask ourselves and the guests how it was, what they thought of the food, atmosphere, music, etc. Perhaps the musicians hired for the event were fabulous, and can be recommended to others, whereas the food might have left something to be desired, suggesting that we should use a different supplier next time. Thus, we have carried out a minor evaluation, systematizing our experiences for future use.

Evaluation is a systematic assessment. Evaluations should follow a systematic and mutually agreed on plan. Plans will typically include the following:

• Determining the goal of the evaluation: What is the evaluation question, what is the evaluation to find out.
• How will the evaluation answer the question: What methods will be used.
• Making the results useful, how will the results be reported so that they can be used by the organization to make improvements.

Definition: Evaluation is a systematic examination of a project’s achievements compared to its planned objectives and expected outputs.

People often think of program evaluation as looking to answer this question: “Does the program work? And how can it be improved?”. However, there are many equally important questions:

• Is the program worthwhile?
• Are there alternatives that would be better?
• Are there unintended or negative consequences?
• Are the program goals appropriate and useful?
• Does the program have a positive outcome?
• Are people satisfied?
• How could the program be improved?
• How well is the program working?
• Is the program working the way it was intended to work?

That is, an evaluation can help a program improve their services, but can also help ensure that the program is delivering the right services.

Why do an evaluation?

The reason to evaluate a project is, firstly, to discover to what extent the project has lived up to the expectations. Another important aim is to find out what we have learned during the project. Why did it turn out as it did? Could something have been done differently? A final reason to evaluate is to issue recommendations for the work ahead, regardless of whether new phases will be added to the project, or whether we need to design an entirely new intervention. A possible and crucial conclusion of an evaluation could also be that no further project work is recommended.

It will be an advantage to have one or several persons from outside the organization to carry out the actual evaluation, since it will be difficult for a project coordinator to deliver a neutral assessment. If you cannot afford to hire an external consultant, you should at least consider if there is anyone within your organization who is at arm’s length from the project, and who might be able to take charge of the evaluation. Alternatively, you might invite one or several people from another organization to do it for you. If there are funds available to recruit several persons to take part, this is usually helpful, ensuring contributions from people with different knowledge and backgrounds. However, to achieve a successful evaluation, it is not enough to hire a consultant.

When to carry out an evaluation?

An evaluation is generally conducted at the end of the project. However, if the intervention runs over a prolonged period, e.g., two years or more, we may also conduct one during implementation. This type of evaluation is known as a midterm review and aims to find out if we are on the right track, and what changes ought to be made in the second half of the project.

Evaluation as a process

The process involves:

• Getting stakeholders (people involved in the program) actively involved in the evaluation
• Developing a complete understanding of the program
• Using the knowledge to determine what information is needed and how to gather it.
• Gathering the evidence
• Interpreting the evidence, making sure it makes sense.
• Using the results, making sure they are useful, getting stakeholders to use them, which depends on stakeholder involvement throughout the evaluation process.
• Following up, continuing communication among all involved, about the evaluation, implementing any recommendations, sharing feedback about the evaluation.