Integrated HIV Training for Healthcare Workers

Session 2: Monitoring and Evaluation

In this session, we will provide an introduction to monitoring and evaluation.

Learning Objectives

By the end of this session, you will be able to:

  • List four activities associated with M&E from your experience and practice.
  • Define monitoring and evaluation (M&E) and outline their relationship.
  • List three reasons for the importance of M&E, specifically in relation to HTS, SRH, PMTCT, OI/ART, TB and VMMC health services in Zimbabwe.
  • Identify M&E tools used in the national HIV programme.
  • Describe the components of data demand and utilization.

Learning Activities

  • Monitoring (5 min)

    As a health care worker, you probably have some involvement in monitoring and evaluating activities at your sites.

    Can you think of a few examples of monitoring activities that are undertaken within areas of your work?

  • Monitoring (5 min)

    Monitoring is regular and routine tracking of key programme elements collected on a regular and ongoing basis. Monitoring of a health programme will help to:

    • Assess program performance
    • Detect and correct performance problems
    • Make more efficient use of program resources

    Monitoring allows us to track, document, and report the quantity and quality of programmes and services provided, and helps us to answer questions, such as:

    • What services are provided?

      provision of HTS services, delivery by a skilled attendant, testing for STIs, ARV prophylaxis

    • To whom?

      pregnant women, women and men, infants, caregivers

    • How often?

      during ANC visits, labour and delivery, at routine health visits

    • Where?

      home care, health facilities, rural, urban

    • When and for how long the services are provided?

      during three ANC visits over a period of eight months

    By answering these questions, you can see if there are gaps in the services or ways that you can improve service delivery, access or acceptability.

  • Evaluation (5 min)

    Evaluation is measuring the changes in a situation resulting from an intervention. Evaluation is episodic assessment of change in specific areas that can be attributed to an intervention. For example, an evaluation may assess the uptake of HTS or SRH services as a result of a targeted behaviour change and communication campaign. Episodic means that evaluation can take place for a defined time period, such as during a mid-term evaluation to help shape programme design and strategy for the remainder of the programme or at the end to help learn lessons and improve future programming and service delivery.

    A formal evaluation of an HTS, SRH, PMTCT, and OI/ART, TB, or VMMC programme will demonstrate to what extent the programme contributed to changes in the indicators. Formal evaluations should be conducted intermittently to try to examine the ways in which the HTC, SRH, PMTCT, OI/ART, TB, or MC programme is running.

  • 12 Monitoring and Evaluation Components (5 min)

    The 12 components of M&E enables system functions ranging from organisational structure, involvement of partnerships, data collection, and the use of data for effective decision making.

    Tap on each component on the image below to learn more.

    map
    rectangle circle triangle arrow arrow rectangle circle triangle arrow rectangle circle triangle
    Data Dissemination and Use

    Using Information to improve results:

    • Medium-term Result: The M&E department within the MOHCC will use the information derived from the M&E system to improve the existing reporting mechanism and organizational structure to use data to inform decision-making. It will also use the M&E data as a results-based management tool to assess whether program targets have been met, how effective interventions were at achieving set targets, and service coverage.
    • Long-term Result: The Ministry will disseminate and use the data from the M&E system to guide decision makers within government and partner ministries (Ministries of Education, Transport, and Home Affairs), development partners (UNAIDS and UNDP), program managers, and implementing partners on policy recommendations and program planning and implementations in economic development. The Ministry will develop a policy framework for national health based on the M&E data and programmatic outcomes.
    HIV evaluation, research, and learning

    Evaluation and Research: Evaluation of the program is conducted periodically as to assess whether the program is achieving its aims, goals and objectives. This will enable to measure good practices, and lessons learnt. Research will help to identify periodic indicators that are not measured routinely during programming. Research enlightens directions and evaluates program implementation with informed decision which are based on scientific evidence.

    Surveys and surveillance

    Surveys: HIV-related surveys carried are used to measure HIV indicators at all levels of the health delivery system. The surveys to be conducted provide information for programme monitoring and implementation. Surveillance activities are required to generate the data for measuring the outcome and impact indicators to monitor the M&E strategic plans.

    Routine programme monitoring

    Routine monitoring: The mechanisms in the MoHCC will collect relevant information at various levels and track progress in the implementation of the programs. Routine programme data are managed through the District Health Information System (DHIS).

    Supervision and data auditing

    Supervision and data auditing: Support to all the health facilities will enable building confidence in their practices of the programs. This is also strengthened in the mentorship that occurs during these visits. Data auditing will help to measure whether the reports being disseminated are the true reflection of source documents which are registers.

    M&E database

    Databases: Electronic and Paper based systems: The paper-based database has been in use in the Ministry of MOHCC. The different data capturing tools for all MOHCC activities will gradually be replaced by the electronic systems. This will help to monitor programs longitudinally so that outputs, outcomes and impact can be easily measured. There are different databases that are used to monitor patients at facility level.

    Organisational structures with M&E

    The organisational structure of the M&E in the MoHCC is according to existing HIS National Strategy for Zimbabwe. 2009-2014.

    Human capacity for M&E

    Human capacity for M&E: The effective and efficient M&E system ensures skills development and maintenance at the various levels the MoHCC.

    M&E partnerships

    M&E partnerships: The successful implementation of the M&E plans needs strong partnership among various organisations contributing in the MoHCC.

    M&E plans

    M&E plans: These plans focus on linkages between the M&E framework and other sectorial M&E plans.

    Costed plan

    Costed Plan: The costed M&E work plan guides all M&E activities. It is a unified work plan and budget that identifies monitoring and evaluation activities that will enable partners to monitor the results framework.

    M&E advocacy

    M&E advocacy communications culture: The MoHCC and its partners through the M&E technical working group will lead the advocacy and communication for the M&E system and development of a plan.

  • Purpose of M & E (5 min)

    Without having some idea of what’s working and what’s not in our health services we can’t develop the most effective services for service users. That is the power and usefulness of M&E! If you cannot see success, you cannot reward it. If you cannot reward success, you reward failure. If you cannot see success, you cannot learn from it. If you cannot recognize failure, you cannot correct it. But, if you can demonstrate results, you can win public support!

    Monitoring and evaluation has three main purposes, which can be considered as the “three pillars”.

    What can we measure?

    In order to monitor and evaluate a programme appropriately, we need to decide what to measure; ideally, during the design of the intervention and not as an afterthought. Tap on each element below to learn what it is and how we could measure it.

    map
    rectangle circle triangle arrow arrow
    Inputs

    Resources invested in the activity

    • Money
    • Number of staff
    • Infrastructure, such as number of clinics, equipment
    Process indicators

    Reflect whether a program is being carried out as planned and how well program activities are being carried out. Often reported in the form of a number.

    • Training provided for staff
    • Materials provided to staff or patients
    • Ongoing management
    • Counselling provided
    • Procurement
    Outputs

    Direct and tangible products from the activity.

    • Services provided
    • Number of clients treated
    • Number of staff trained
    Outcomes

    Changes to people resulting from the activity

    • Changes in attitudes
    • Increased rates of testing
      • 100% of all women who are eligible for cervical cancer screening and are provided with information are undergoing a cervical screening test
      • 95% of all women undergoing HIV testing and all who are HIV-positive and require treatment are provided with treatment
    Impact

    Long-term effects

    • Changes in HIV prevalence
    • Reduced rates of HIV and other STIs
    • Reduction in number of women dying from cervical cancer
    • Rate of mother-to-child transmission reduced

    What are differences between “monitoring” and “evaluation”?

    Although they are linked, the difference between monitoring and evaluation is that evaluation is more closely associated with outcomes and impacts, while monitoring tracks inputs and outputs.

  • Indicators (5 min)

    A fundamental component of any programme is setting goals and establishing metrics to measure success. For example, you may want to know the percentage of women seen in their first trimester or the percentage of infants born to HIV-infected women who receive an HIV test. These are both examples of indicators that reveal progress toward a set of objectives and provide information on the status of activities related to each step of the programme cycle.

    Key performance indicators play a critical role in tracking the ongoing performance of your program. Indicators are summary measures to describe a situation. They are an observable or measurable change that tell you that an outcome has been achieved. Essentially, they are things which can be measured or assessed to see the progress being made by a programme/service.

    Indicators are used both in monitoring and evaluation and can be quantitative and qualitative.

    Tap each tab below to view an example.

    Quantitative

    percentage of people receiving HTC or OI/ART services

    Qualitative

    level of satisfaction with quality of HTS or OI/ART services provided amongst service users


    Indicators can be used at different levels to measure what goes into a programme or project and what comes out of it. You can have several indicators for one outcome and if you have more than one indicator, then you have “multiple lines of evidence”.

    SMART Indicators

    The most well-known method for setting indicators is the SMART way. SMART refers to the acronym that describes the key characteristics of meaningful indicators. SMART indicators help provide markers towards the achievement of goals.

    • Specific (concrete, well-defined)
    • Measurable (numbers, quantity, comparison)
    • Accessible (feasible, actionable)
    • Realistic (considering resources)
    • Time-Bound (a defined time-line)
  • National Health Information Systems (5 min)

    High quality health information system (HIS) data are necessary to collect, store, manage, process, analyse, report, and disseminate reliable data on key health indicators on a regular basis. HIS data are also needed to measure performance of health service delivery systems, monitor progress in achievement of health goals, and to make evidence-based decisions and choices.

    The HIS in Zimbabwe is an integrated information system which incorporates indicators from all programmes rather than having parallel and fragmented systems for each individual programme. The National Health Information and Surveillance System is well established and supported. Each district, province, and hospital has health information personnel whose function includes data entry, data analysis, production of reports, and dissemination of information for use at the local level, and submission of the same to the next higher or lower levels.

    Who Needs HIS Data?

    Tap on each tab to view more information about who needs HIS data.

    Consumers

    Consumers need information to make informed choices about available health options. With appropriate information, consumers will receive care that is targeted to their health needs and provided in a coordinated way, as their care providers are better informed and able to communicate easily.

    Care Providers

    Care providers need to make informed decisions at the point of care. This will be possible because relevant patient information will be more accessible, and they will have access to evidence-based knowledge on which to base their interventions.

    Health institutions

    Health institutions must be informed about what constitutes good practice. They will be able to configure services to achieve the best outcomes for their communities, since they will be better informed about their health status.

    Donors

    Donors must have information about the health status of the population, what services should be provided, and how effectively services are delivering desired outcomes. This will enable them to target services more selectively, as well as being able to analyse population health data.

    Policy Makers

    Policy makers and researchers must develop knowledge of epidemiological trends and identify interventions that will have the highest impact on improving health outcomes of all community members.


    Surveillance data

    Routine surveillance data collected under HIS (of interest to HTS, SRH, PMTCT, OI/ART, and TB) are included in the T5 and T6 forms. The T5 Monthly Report is a comprehensive summary of data on outpatient diseases and preventive services. The type of data generated from the T5 report include disease incidence and prevalence rates, immunization coverage rates, nutritional deficiency rates, and STI infection rates. The T6 form is a tally sheet for MCH activities that is used at all clinics and hospitals. It gathers daily data for MCH information, and the data is transferred to the T5 form at the end of every month.

  • Data Demand, Utilization and Target Setting (5 min)

    Tap on each definition below to learn more.

    Data

    These are raw facts and figures which are in an unorganized form or state

    Information

    Data that is (1) accurate and timely, (2) specific and organized for a purpose, (3) presented within a context that gives it meaning and relevance, and (4) can lead to an increase in understanding and decrease in uncertainty. Information is valuable because it can affect behavior, a decision, or an outcome.

    The advantage of evidence-based decision making is that it promotes transparency in the decision-making process and allows for accountability of health decision makers.

    Data demand

    For stakeholders and decision makers to place value on information they should have incentive or motivation to use it. Data demand require the following criteria:

    • Stakeholders and decision makers specify what kind of information they want to inform decision
    • Stakeholders and decision makers proactively seek out that information
    Information use

    Involve positive and negative findings that influence the decision-making process (such as continuing with a particular HIV intervention or suspending it). Raw data are seldom useful for decision making and usually must be transformed into information that is usable. For example, it’s not just good to know how many people received services. A comparison against a set target would be more useful. Data should be collected and transformed into information for use in decision making.

    Target setting

    Facility level targets derived from district targets. For example:

    catchment population divided by the total district population of that subgroup multiplied by the district target for the subgroup for a specific program

    a subpopulation, e.g., adolescents

    Note that partners working in specific districts and facilities may have different targets from the MOHCC agreed targets; however, the guidance remains that facilities work toward a target whichever is higher.

    Data Utilization Using Cascades

    Data utilization using cascades is a way to show the proportion of individuals requiring services who are engaged at each stage of health care service delivery. It helps management to identify gaps and opportunities along continuum of care or service provision. A cascade analysis allows us to identify leakages along the continuum of care and come up with interventions to close the gaps or leakages. This can be done at all levels including the facilities. Below is a sample cascade for the HTS programme.

    Source: HIV 2015 estimates & DHIS2 2016

  • Operations Research (5 min)

    Operations research is ultimately concerned with improving the health of a community, which aims to quantify the relevant aspects of the situation to develop decisions, plans, and policies. It is the study of facility related factors, documentation, staff attitudes, etc. under the control of programme managers. It is research into strategies, interventions, tools, or knowledge specifically intended to enhance programme effectiveness.

    It is the systematic collection, analysis, and interpretation of data to answer a certain question or solve a problem.

    Operational research is increasingly recognized as essential to strengthen health programmes, and there is constant need for operational research to come up with interventions that guide and improve programmes. As a health care worker, you should take advantage of the current capacities in the provinces for technical guidance on how to carry out research.

  • Knowledge Check (5 min)

    1The following are examples of outputs except?

    2From the dropdown menu, select whether monitoring or evaluation is part of that component.

    3What are the three main purposes of M&E? Select three.

    4In the box below, list three reasons for the importance of M&E, specifically in relation to HTS, SRH, PMTCT, OI/ART, TB and VMMC health services in Zimbabwe. Then tap the compare answer button to see what the answer is.

    6Which M&E tools are used in the national OI/ART programme? Select all that apply.

  • Key Points (5 min)
    • Monitoring is regular and routine tracking of key programme elements collected on a regular and on-going basis.
    • Evaluation is measuring the changes in a situation resulting from an intervention.
    • M&E measures inputs, outputs, outcomes and impacts.
    • The difference between monitoring and evaluation is that evaluation is more closely linked with outcomes and impacts, while monitoring tracks inputs and outputs.
    • Without having some idea of what’s working and what’s not in our health services we can’t develop the most effective services for service users.
    • Routine surveillance data collected under HIS of interest to HTS, VMMC, SRH, PMTCT, OI/ ART and TB is included in the T5 and T6 forms.