Unit 8: Data for Decision Making
Welcome to the unit on Data for Decision Making. This module is about data and how it can be used by leaders and managers to understand how programs or projects are being implemented and their impact. Monitoring and evaluation are prevalent requirements of countries, governments, its organizations and people to do business, compete and be successful in their work.
Learning Objectives
Upon completion of this unit, you will be able to:
- Appreciate the use of monitoring and evaluation data for managers and leaders;
- Understand the purpose of data for accountability in a peacetime existence and for readiness for disasters or war;
- Recognize the characteristics of a data-friendly workplace; and
- Identify the role of leaders and managers in creating an organizational culture of information generation and use.
Practice
We encourage you to have a paper journal and pen with you to complete written activities and self-reflection assignments. Alternatively, you may find it useful to use Microsoft Word to complete written activities and self-reflection assignments.
Learning Activities
-
Video: Welcome (1 min)
Welcome to the unit on Data for Decision Making. First, you will hear from guest lecturer, Gabrielle O’Malley (USA). Dr. O’Malley will introduce you to the unit topic.
Instructions: Tap the video to play.
_ -
Video: Data and Evidence (5 min)
To begin this unit on Data for Decision Making you will watch three videos from Everyday Leadership series. Watch the first video called Data and Evidence by Dr. Peter Piot (Belgium).
Instructions: Tap the video to play.
After you watch the video, in your journal, write your response to the following questions, then tap the Compare Answer button to reveal additional information.
- What type of data is Dr. Piot talking about?
- What is the ultimate purpose of the data?
- What is the biggest pitfall when it comes to using data?
Dr. Peter Piot is talking about a wide range of data (including clinical or medical evidence, political information, laboratory data) and its use from a variety of perspectives (laboratory directors, heads of state). He clearly articulates how different leadership positions would require different degrees of familiarity with the details of the data. For example, a laboratory director would need to be very familiar with the details of the quality assurance data being produced by the laboratory. For a leader from the Ministry of Health, he or she would not need to know that level of detail of quality assurance numbers, but would need to know whether or not the lab results being generated in the country were of good quality. In this video, Dr. Piot tells us that the biggest pitfall for those higher up in management and leadership is becoming too focused on the technical details of the data and losing sight of the bigger picture.
-
Video: Detective Work (5 min)
Let’s watch a second video from the Everyday Leadership called Data and Evidence by Aaron Katz (USA). `
Instructions: Tap the video to play.
After you watch the video, in your journal, write your response to the following questions, then tap the Compare Answer button to reveal additional information.
- What type of data is Mr. Katz talking about?
- What is the ultimate purpose of the data?
- What is the biggest pitfall when it comes to using data?
Aaron Katz is talking about data primarily in terms of a landscape analysis so that you can understand how to most productively interact with various stakeholders in order to accomplish your organization’s objectives. Who are your natural allies? Which organizations or individuals are most likely to be won over to assist you in your objectives? The biggest pitfall Aaron Katz talks about in terms of data use is losing patience. He points out how having this information and using it to form alliances to change policy takes a lot of time.
-
Video: Quantity vs. Quality (5 min)
Watch a third video from the following Everyday Leadership series called Quantity vs. Quality by William Graham (USA).
Instructions: Tap the video to play.
After you watch the video, in your journal, write your response to the following questions, then tap the Compare Answer button to reveal additional information.
- What type of data is Mr. Graham talking about?
- What is the ultimate purpose of the data?
- What is the biggest pitfall when it comes to using data?
When Bill Graham talks about data, he is referring to the “nuts and bolts” of project and program data; for example, the money spent and number of people trained and information that would tell you about the impact of your public health program. The biggest pitfall he mentions is the threat of quantity overcoming quality in terms of the data collected. He mentions program tendencies to focus on the metrics that are easy to count rather than the more difficult to measure metrics of making a difference in the public health of communities.
-
Video: Data for Decision Making (15 min)
Next, you will hear again from guest lecturer Gabrielle O’Malley (USA) on the topic of Data for Decision Making. In this recorded lecture, Dr. O’Malley discusses the relationship between monitoring and evaluation and management and leadership.
Instructions: Tap the video to play.
-
Quiz: Monitoring or Evaluation? (15 min)
Assess your knowledge on identifying monitoring versus evaluation questions by answering four questions.
Instructions: Tap the correct answer, then tap the Feedback button to reveal the correct answers and read additional information.
4.How could you improve/prevent pilots from missing aircraft checklist items? Write a response in your journal, then tap the Compare Answer button to read feedback.
Compare your response with the following information. Your answer may have included: Study the data and identify which items on the aircraft checklists are being missed. Secondly, determine why those particular items are being missed or overlooked. Mission observers could be sent out into the field to determine if pilots are missing checklist items during actual flights. Third, devise a plan to change the way pilots are trained in the simulator so that they will not miss any items on the aircraft checklists.
Also consider: Identifying what sources of data are of critical importance to Commanders and unit staff for decision making; which sources are important for purposes of accountability vs. program effectiveness; and how leaders can create an organizational culture of information generation and use. Using a Logic Model helps conceptualize and implement M&E activities with a systems approach. Better decisions or judgments can be made, and leaders held more accountable, if the information available is reliable and accurate.
-
Reading: Management Role in M&E (15 min)
Managers and leaders need good data, and it’s important to decide what percentage of the overall project or program budget should go towards monitoring and evaluation.
Some people will argue that money spent on monitoring and evaluation is money not spent on program implementation. However, without adequate resources spent on monitoring and evaluation, you won’t know whether or how your program is being implemented, and you won’t know whether or not it is having its intended results. In other words, you won’t know whether the money spent on implementing your program is worthwhile. So identifying the balance between resources spent on project implementation and resources spent on monitoring and evaluating the effectiveness of your program is a leadership decision. It is also a leadership decision to decide the amount of resources going to monitoring vs. evaluation. Monitoring is generally less expensive than evaluation, but usually tells a less compelling story than evaluation. Leaders must also understand and decide how much rigor is necessary for any evaluation activities identified.
Generally, more rigor requires more resources. In making this determination, you may want to ask:
- How important are the findings?
- Will the findings primarily be for reporting purposes with little use after the report is completed?
- Will the evaluation data contribute to global change in public health practice or patient or population safety?
Leaders and managers need to decide how transparent results from monitoring and evaluation efforts will be.
- Who needs to see or can see the data?
- What are expectations surrounding data use?
- Where should we focus our evaluation efforts?
- Is our program doing something particularly innovative that we should document?
- Is there some component of our work that our stakeholders or partners are especially interested in?
- Is there something we think we are doing especially well that we’d like to document with evidence so that we can share it with the world?
Leaders and managers need to guide monitoring and evaluation staff in interpreting and framing results for different audiences. Data may be presented in different ways depending on if the audience is scientific journals, community based organizations, ministries of health, or funding organizations.
Key Points:
- Determine what resources should be committed (staff and money).
- Know where to focus resources.
- Decide how results will be used and shared.
- Identify who needs the data.
- Decide where to focus evaluation efforts.
- Interpret and frame results for different audiences.
-
Video: Knowing What Has Value to Others (5 min)
Watch the following Everyday Leadership video called Knowing What Has Value to Others by Ms. P. Amudha (India). In this video Ms. Amudha gives an example of how data was used to move leaders to take action to improve sanitation.
Instructions: Tap the video to play.
After watching the video, reflect in your journal on a situation you have encountered where data served as a catalyst for positive change:
- How were the data presented?
- Why do you think the change was caused?
- How could you use data in the future to move others to action?
-
Activity: Characteristics of a Data-Friendly Workplace (15 min)
Instructions: tap the five items below to reveal characteristics about how you can make your workplace more data-friendly.
Establishing data as a leadership function
One of the most important functions of leaders/managers in terms of data use for decision making is in creating a culture where gathering data and making decisions based on that data is a priority. Leaders can clearly demonstrate that using data is a leadership function through role modeling their own use of data (“confront brutal facts, yet never lose faith”).
Valuing data within an organization
Organizational values are aimed at achieving improved, demonstrable results for the target population, performance data, and financials.
Allocating time and resources for data
Through staffing plans, organizational charts, and budgets, leadership can ensure that time, resources, objectivity and visibility are planned for collection, analysis, synthesis, and use of data. It is also very important that once data has been collected and organized, leaders provide time for reflection and development of insights based on the evidence produced.
Planning for the future
After time and resources have been dedicated for data, then future plans and practices can be developed based on evidence-based best practices and lessons learned. Perhaps the most important component of creating a culture where data use for decision making is embraced is having leaders who acknowledge unintended consequences of program efforts. Programs will never be 100% successful in everything they do.
Learning from mistakes or unintended outcomes
In order for organizations to improve, to truly make strides in improving global health, people need to feel safe to share negative results. Negative results are not necessarily a bad thing if leaders and organizations are willing and able to change or improve based upon these findings. Often times, leaders will say that they learn more from their failures or mistakes than from their successes. It is much easier to celebrate success than it is to discuss failure or instances where we’ve fallen short of our objectives. It takes a lot of courage and skill for leaders to identify and discuss what went wrong, but doing so can be very important for maintaining motivation among staff. If leaders only talk about the program or organization’s successes, and yet program staff are very aware that some of their own projects are not achieving intended results, they can become disengaged or cynical about their organization’s work. In a worst case scenario, if staff believe only positive findings will be appreciated; they are likely to report only positive findings. So creating an environment where it is safe and appreciated to share both positive and negative findings from data collection activities is very important for organizations to improve and to help maintain work as a meaningful activity for program staff.
-
Case Study: Data for Decision Making (30 min)
Instructions: Think about the information that was shared with you on data for decision making so far to complete a short case study. Read the case scenario below, then respond to three questions about the case scenario. Please have your journal available to write your responses to the essay questions.
Case Scenario: You are a Major in the Air Force and the Commander of the Air Force flight simulator. You oversee the simulator pilot training and evaluation program and are responsible for seven program analysts and ten pilot instructors/evaluators. All 300 Air Force pilots are required to successfully complete the two-day flight simulator course annually. If a pilot fails his end-of-course evaluation on the first try, he is given remedial training and is re-evaluated.
The following pilot data is collected from simulated mission scenarios and inflight emergencies.
- Maintaining control of the aircraft at all times;
- Properly handling inflight emergencies;
- Following aircraft checklists;
- Following written procedures and regulations;
- Pilot to crew coordination;
- Situational awareness; and
- Pilot performance pass/fail.
In the past three years, there have been two aircraft accidents and three incidences of near fatal crashes. Four of those episodes were determined by the Flight Safety Board to be pilot error. Air Force Officials at Headquarters have ordered an inquiry of pilot performance from their annual flight simulator training. You are to compile a report on pilot performance and submit it to Headquarters.
After reviewing the simulator data collected for the last four years, you find a disturbing downward trend in the pilots’ failure rate. You synthesized the data and discovered the two areas that are responsible for the increase in failure rates: handling inflight emergencies and following aircraft checklists.
You have also noticed in recent months that pilots have become complacent when running aircraft checklists. The aircraft checklist items become so routine that the majority of pilots use their rote memory to accomplish some of the items on the checklists.
3. Now, think about how to communicate these findings to your colleagues, pilots, the Commander and to Headquarters? Write a response in your journal, then tap the Compare Answer button to read feedback.
Compare your response with the following information. Your answer may have included:
- Share your information or findings with simulator staff, and ask for ideas to modify the pilot training program.
- Share any changes to the training curriculum.
- Draw attention to the issue by proposing what your next steps will be in terms of training modifications and/or offering possible solutions.
Wide dissemination of pilot performance report findings will help highlight the areas where pilots are continuing to make common errors. Eliminating those mistakes in the simulator should help avoid any other major fatalities. Additional resources may be needed, and more support may be required.
-
Self-Reflection: Data for Decision Making (15 min)
Think about the leadership and organizational culture where you work in relation to creating a culture of data use in decision making. How could you contribute to strengthening the culture of data use in your workplace?
Instructions: In your journal, write your response. After you have answered the question, tap the Compare Answer button to compare your response.
Some examples you may have included in your response are:
- I could use the data in my workplace to monitor my program – to ensure that my project is staying on track and to document whether and how my project is being implemented.
- I would use monitoring or performance indicators to track this information.
You also might have mentioned that evaluation data would help you, your organization, and stakeholders know whether you are having the intended results. This could help boost teamwork of your staff, as everyone will be clear on what you are trying to achieve. Positive results from evaluation could help you to motivate and encourage staff and stakeholders by knowing that their efforts are worthwhile. Negative results could help staff re-examine assumptions and redesign and improve programs so that objectives of improving global health can be met. The indicators you chose for monitoring and evaluating your program would be specific to your own project, program or organizations. You may have mentioned that, as a leader or a manager, you could improve your organization’s culture of data collection and use by ensuring this is a priority in terms of time and resources. You may have mentioned that when you get “bad news” from your monitoring and evaluation information, you could frame this as essential information to improve your program. You could congratulate staff on identifying challenges and use this information to trigger dialogue among staff and other stakeholders on how to address the issues. When you have good news from data, you could use it to champion your organization’s cause and to identify and spread best practices.
-
Optional: Group Discussion (15 min)
If you are taking this course with a partner or in a group setting, have a conversation about your leadership and management experience by discussing the following question:
- Do you see barriers to using data more effectively in your organization or unit? If so, what are they?
- What needs to happen for these barriers to be overcome?
-
Self-Reflection: Learning Action Plan (15 min)
Putting your learning into action is essential to knowledge transfer, applying the knowledge you gained in this unit, and retaining that information in the future.
Take a moment to reflect on what you’ve learned in this unit. After reflecting, in your journal, write:
- Four things you learned in the unit.
- Three things you will implement:
- Today;
- Six months from now; and
- One year from now.
- Two things that changed your perspective.
- One thing you will ask for more help with.
-
Reading: Conclusion (5 min)
In this unit, you learned about the basic concepts of monitoring and evaluation and the relationship between monitoring and evaluation and management and leadership. You also learned more about the use of data for accountability versus public health effectiveness and the role of leaders and managers in creating an organizational culture of information generation and use. You also read the recommendations for improving data-informed decision making put forth by Measure Evaluation.