Why is Evaluation Important?
Evaluation, in an evidence-informed organization, is intended to provide insights into the successes or problems as they are occurring in any policy or program initiative and to provide accountability. Although evaluation also has technical components, the power of evaluation is in providing organizations with credible and defensible information for adapting, adjusting or even changing course based on evidence about the progress and impact of organizational initiatives.
- Evaluation is important to measure progress and ensure effective project or program implementation as well as effective use of public money.
- Evaluating identifies what works and what doesn’t, informs decision-making, and contributes to evidence-based policy development and improvement.
- Evaluation demonstrates accountability, builds collective capacity, and reflects government’s commitment to improve student outcomes.
Learning as we go…
K-12 Improvement Planning Assessment Tool (IPAT)
K-12 Improvement Planning Assessment Tool (2017-18) is a comprehensive tool to assist the improvement planning process. On the tool, questions in green help guide a focused and precise approach to annual improvement planning/ongoing implementation processes. Texts in black are prompts to help inform educators as researchers infuse evaluative thinking in our practice.
How can the IPAT help my classroom, school, or district school board?
The IPAT is a thinking and conversation tool for prompting reflection, discussion, and planning on how data can lead to district, school, and classroom improvement. You are not required to answer all the questions, but certain ones may be particularly helpful in your teaching and learning contexts. Remember to maintain open communication and encourage collaborative inquiry-action throughout this process. It is an ongoing, circular process that includes considering:
- What do I know about student achievement, equity, and well-being in my board?
- Have I examined demographic, program, and perceptual data?
- What actions might I take in relation to outcomes?
SMART Outcome Goal(s)
- Do the SMART outcome goal(s) coherently address the needs assessment? (i.e., are my goals Specific and student-focused, Measurable, Attainable, Results-Oriented, and Time-Bound?)
- Does the needs assessment inform this year’s outcome goal(s) address areas of greatest student need?
Targeted Evidence-Based Strategy
- How will the strategies and actions change practice that is coherent from the student desk to the district school board in service of the outcome goal(s)?
- What actions might I take?
Resources and Professional Learning
- How have learning, financial, human and technological resources been differentiated so that it is coherent with your needs assessment?
- How is professional learning responsive to the desired outcome goals?
- What actions might I take to ensure this?
- How am I monitoring implementation and progress towards achieving the outcome goal(s) to enable learning as you go?
- What actions am I taking for assigning ongoing monitoring and reporting actions related to implementation and outcome monitoring?
- How do my leadership practices facilitate monitoring and support the implementation of strategies to achieve the outcome goal(s)?
- How am I monitoring implementation processes and progress towards achieving the outcome goal(s) to enable learning as you go?
- How am I determining responsibilities for monitoring, addressing challenges as they arise, and reporting?
Learning As We Go – Developmental Evaluation
- What evidence is there that the goal(s) are being achieved and how will lessons learned be applied to future improvement plans?
- What actions might I take to ensure this?
Where can I find the IPAT?
In addition to the Applied Research Capacity website, the IPAT is also available through e-learning Ontario. If you have an e-learning account, please self-register or ask your district school board’s TELO contact to access a self-guided resource on evaluative thinking in action: https://community.elearningontario.ca/d2l/home/4191128
For strategic and improvement planning resources, access the Multi-Year Strategic Planning: A Guide for School Board Trustees (2017), and/or the companion resource document Multi-Year Strategic Planning: Supplementary Resources for School Board Trustees (2017).
For any other evaluation projects…
Getting Started: A 10-Step Process to Thinking About Evaluation
Evaluation usually involves collecting and analyzing information about a program or initiative’s activities, characteristics, and outcomes.
Conducting Evaluations – 10 Steps (if you are hiring someone to conduct the evaluation)
Evaluation is important to ensure effective project or program implementation as well as effective use of public money.
- Information from evaluations tells us what works and what doesn’t, and ultimately contributes to implementing evidence-based practices in education.
- After you have clarified the issue and defined the project, follow the steps below as they contain information and tools to guide you through an external evaluation project.
- N.B. The steps are only a guide; you may not follow them sequentially.
Step 1: Review the Literature
There is an abundance of literature in education and indeed there is little that is completely unknown. It is important to identify and use existing research within the organization, or externally to guide your evaluation. For a thorough review, it is important to seek multiple sources of information. This guide will help you get started on conducting a research literature review.
Step 2: Determine the Evaluation Question(s)
The success of the evaluation rests on the quality of the questions. Evaluation questions should be specific and reflect the purpose of the project. High quality evaluation questions establish the parameters of the evaluation to inform decisions regarding the issue. The external contractor should be responsible for determining the final evaluation questions; however, you should have an idea of the potential questions. This tip sheet will assist you in developing an evaluation question.
Step 3: Develop a Logic Model and/or Theory of Action
A logic model is a graphic representation of the inputs, activities, outputs, and outcomes of a program. It will typically be up to the external contractor to create a logic model for an evaluation, however it is important to understand logic models in order to be able to work with the evaluation team. This tip sheet will assist you in developing logic models.
The theory of action forms the foundation for developing an evaluation process that will address the evaluation questions that are important for making adjustments along the way, in any initiative, and in determining how well the initiative is meeting its goals.
Step 4: Understand Evaluation Approaches and Methodologies
There are several different types of evaluation approaches and methodologies, depending on the question being asked, time and resources available, and the purpose of the project. Again, it will be up to the contractor to decide on the methods; however, it is helpful for you to understand the methods in order to assess the appropriateness of what is recommended by the external evaluator.
This tip sheet will help you better understand the different types of evaluation approaches.
Step 5: Identify Existing Data
Recognize that there may be existing data that could be used to support an evaluation project. Scan and identify existing sources of data.
Step 6: Doing it Yourself or Hiring Someone
OPTION A – Doing it yourself
Workflow Chart (if you are conducting the evaluation)
OPTION B – Hiring an Evaluator (if you are hiring someone to do the evaluation)
This process may vary depending on your organizational protocol. It is best to consult your organization’s protocols and guidelines for procuring external services.
- Prepare a Request For Services (RFS)/Request for Proposals for Evaluation
- Assess Proposals Received for Carrying Out an Evaluation Project
- Prepare a Contract for Evaluation
Step 7: Refine the Evaluation Questions, Scope and Plan
- Determine the nature of evaluation that is aligned with the strategic direction set by the organization and/or senior management.
- Clarify purpose of evaluation and check if the evaluation goals or purposes serve the direction.
- Work together to develop sound evaluation framework that is most appropriate to address evaluation questions, taking into account the theory of action.
- Check if the approach to the evaluation (e. g. evaluation purpose, scope, logic model) is likely to satisfy the expectation
Step 8: Manage, Support and Monitor Progress
This tip sheet will tell you everything you need to know about maintaining a good working relationship with your external contractor.
Step 9: Assess Final Evaluation Report
It is important to assess the final evaluation report against the original proposal and the original evaluation question(s). This tip sheet and checklist will help you do that.
Step 10: Communicate the Findings
Communicating the findings in a clear and concise way is important to ensure that the findings will be understood and used to inform policy and practice. Results of an evaluation project that has been done externally will most likely be communicated internally, but could be communicated externally as well. Whether you communicate internally or externally, ask yourself the following questions as early as possible:
- Who is your audience? Who is interested in this information, and who else could benefit from it?
- Are the results confidential? What information can you share, and with whom?
- How much time will the target audience have to digest the information?
- How can you best reach your target audience?
- What format will be suitable to share this information?
If you’re looking for more information about communicating your findings, see Knowledge Mobilization for tools and templates.