What do I need?
There are many ways to collect information about your program and each approach has its strengths and weaknesses. All of the data you collect forms part of the information you can utilize for your evaluation.
Types of information
As a general rule, evaluation information is organized into two groups: numbers (quantitative) and words (qualitative).
When we talk about numbers (quantitative data), we mean information that helps us describe what exists and happens in terms of time, quantity, frequency. For example:
The average age of participants
The number of people receiving services per week
The number of sessions participants attended, etc.
When we talk about words (qualitative data), we mean information that helps us capture a deep understanding of participant’s experience in the program or their personal stories. This can include information about the why, the how, and the consequences of participating in the program. For example:
Opinions about the program
Descriptions of experience within the program
Descriptions of impact on their personal stories, etc.
What are the sources of information?
The following table lists examples of potential sources of information that you might have and/or need for your evaluation.
Table.1 Sources of information
|Attendance lists at events||Radio interviews|
|Donation reports||Journal articles|
|Personnel training handbook||Research published by other organizations or universities|
|Workshops and/or presentation reports||Social media|
Places where you can access information outside your organization include: local libraries, universities, government offices, research institutes, mass media (newspapers, radio), and social media (e.g., Facebook, Twitter, and Instagram).
We suggest you create your own list of information sources since this will help you organize your work and keep track of the information that you might have and/or need for your evaluation. Utilize this worksheet to help you get started.
Types of information
Words (qualitative data)
What do I have:
What do I need:
|Process and feedback notes|
|Report summaries or annual report|
|Media communication (e.g., press releases, radio interviews, etc.|
|Social media posts (e.g., Facebook, Twitter, or blogs)|
|Program Curriculum and related handouts, activities, etc.|
|Numbers (quantitive data)||
|Attendance lists and signup sheets|
|Report summaries or annual report|
|Public reports (e.g., census data, public health reports, police crime reports, etc.)|
What tools can I utilize to collection information [data]?
There are a variety of tools that can be used to collect information. We believe that the best tool is always the one that corresponds to the circumstances in which it will be used. In other words, based on the realities of the organization, the resources available, the type of program, the participants, and the community. Consider the resources you have both in terms of personnel and funding, this will help you determine if you can use several tools—simultaneously or during different phases of the program. Using several tools allows you to obtain rich information from different perspectives.
The following table will help you understand each tool, its purpose and the possible situations we do not recommend using the particular tool. This will help you find the best “match” for your program and evaluation needs.
|Tool||Description||Purpose||Pro's & Cons||Avoid using this tool...||Documentation reviews||A review of written materials and other documents.||Allows you to utilize existing documents and materials already available within the organization (e.g., meeting notes, photos, annual reports, etc.).||
PROS: It is an easy and inexpensive way to obtain information
CONS: It can be difficult to define parameters on what to include and not include in the review.
|We highly encourage starting from this point.|
|Interviews||A one on one conversation with another person.||Allows you to obtain in-depht information about a participant's experiences.||
PROS: Helps build trust with participants.
You can obtain rich information.
CONS: you may exclude some people because you are unable to interview everyone. Takes time to complete and to analyze this information.
If you do not have a lot of time to carry out interviews.
if you do not have enough staff to conduct the amount of interviews you want to complete.
If you just want basic demographic or program information (i.e. who attended? how many times did they attend, etc.?)
|Focus groups||A conversation with a group of people (usually between 4 and 10) in which specific topics are discussed.||You can obtain in-depth, diverse information from a small group of participants.||
PROS: It is one of the most participatory, transparent and democratic ways of obtaining information.
It might help you formulate questions or design initial evaluation activities.
This method allows you to avoid extreme opinions and gives you the opportunity to hear about new or unknown topics.
CONS: Takes time and resources to coordinate focus groups.
Takes time to analyze this information.
If it will be difficult to get the appropriate number of participants (too many or not enough).
If you do not have access to a meeting space.
If you just want basic demographic or program information (i.e. who attended? how many times did they attend, etc?)
|Surveys||A series of specific questions about a topic.||It allows you to obtain information in a consistent manner across a large number of people.||
PROS: It can be easy to gather and organize this information/data.There are many current technologies(e.g.SurveyMonkey)available that easily manage data collection and analysis.
CONS: It may produce a large amount of information.
It can be difficult to create adequate survey questions.
If the survey is too long you run the risk of people not completing it or just not thinking through their responses to finish quickly.
|If you do not have adequate recourses to manage and analyze data.|
|Observations||Intentionally scheduling time to observe individuals, groups, activities, etc.||Observation allows you to immerse yourself in the situation in which the program takes place, and capture other elements affecting the program.||
PROS: It can help you identify the strength or weaknesses of an activity.
Allows you to understand the context and the process in which the program occurs in an unbiased way.
CONS: Requires a lot of time.
Requires training and practice.
Sometimes when people know they are being observed, they may change their behaviors.
|If you do not have the resources, trained or experienced staff, or the staff available to conduct this activity.|
When can I collect information?
Deciding when to collect information can appear to be a challenge, if we think of it as a new task. You may not have considered this, but much of your everyday work includes collecting information. For example, when people come to the office in need of services, they usually complete a form with their personal information. When collecting information about your program, the only differences are the goals and timing of your activities.
Imagine you want to collect information to evaluate a new program, in its initial phase. You think you do not have all the necessary information to decide what the main activity will be in order for the program to work. One strategy could be to organize a focus group to brainstorm ideas about the program’s main activities. Another scenario could be that a program ended a couple of months ago, but you lack sufficient information to allow you to identify the program’s strengths and weaknesses. In this case asking former participants to fill out a survey could provide the extra information you need.
The most important point is to have a clear idea of the evaluation goals or the reason you want to collect evidence. This will also help you decide when is the right time to collect information.
Graph 1. Opportunities to Collect Information.
Other Considerations in Collecting Information/Building Evidence
What are the possible costs involved in conducting evaluation?
It is important to view evaluation activities in the same way you view any other job your organization performs, it will require a budget. As with any budget you will need to include potential and unexpected expenses.
Determine the essential components of your program and the ones you can substitute. Consider collaborating with other organizations or agencies to carry out the work, as a way to save money or reduce costs. Listed below are a few basic elements you should include in your budget:
Personnel: consider looking within your organization to determine whether or not you have staff that have the skills or experience to help you with this task.
Consultant: perhaps you do not have the personnel or their experience is limited such that you will need to hire someone outside the organization. This activity must not be taken lightly because it could end up with outcomes you did not hope for. To learn more about this task we offer you a complete section on Choosing an External Evaluator, which we believe will be very useful in helping you find the right evaluator for your evaluation needs.
Transportation costs: you must consider that you might have to visit several places to obtain information or that your program activities take place at different locations.
Printed materials: the program as well as the work of building evidence will require producing and printing documents, be they surveys, interviews, focus groups guides, etc. Take this cost into consideration in addition to the cost of obtaining information from places like libraries, government agencies whether in or out of state.
Office supplies and electronic equipment: in this category include the computer, any software you might need to get the job done. And currently with the use of video cameras for Skype or other programs for video conferences, audio, graphic designs, etc.
Communications: these include traditional mailing costs, professional newsletters and electronic mailings, internet, and local, national and/or international telephone calls.
Consider in detail all the activities and materials required to efficiently carry out this task. Prepare your budget considering the best and worst case scenarios. The key is to pay attention to all the details and plan for any scenario.
How can you conduct evaluation on limited funds?
Consider personnel and resources available to evaluate your program and re-allocate resources so that someone is working on evaluation activities.
Consider your current processes and how you can enhance these activities to gather evaluation information.
Consider partnering with graduate students/universities to get free or low-cost evaluation support.
Consider hiring an external evaluator and cost-sharing fees across multiple organizations.
Do not take an “all or nothing approach”…even starting with one evaluation question and then building evaluation activities over time can be useful.
Consider using free/low-cost tools that can help you gather and analyze information easily and quickly (e.g., SurveyMonkey, etc.)
TIP: Build evaluation activities into grant submissions.