How will you know if your eLearning course is effective?
You might have spent sleepless nights researching, planning, building, and creating your course.
You might have spent the last penny in your pocket to finish your eLearning course.
You might have edited and reworked the instructional materials countless times, but is it enough?
Can you say that you have built the best eLearning course? Of course, the answer is "I might," or worst, "I don't know."
That's where instructional design evaluation comes in.
In this article, you will learn the What, When, and Why of instructional design evaluation.
Why Do We Evaluate?
Evaluation ensures that the instruction is designed to meet the identified need for instruction and effectively achieve the intended learning outcomes for participants.
It helps to answer questions such as:
-
Are our instructional goals aligned with the requirements of the instructional program?
-
Are our lesson plans, instructional materials, media, and assessments, aligned with learning needs?
-
Do we need to make any changes to our design to improve the effectiveness and overall satisfaction with the instruction?
-
Does the implementation provide effective instruction and carry out the intended lesson plan and instructional objectives? Have the learners obtained the knowledge and skills that are needed?
-
Are our learners able to transfer their learning into the desired contextual setting?
These questions help shape the instructional design, confirm what and to what extent the learner is learning, and validate the learning over time to support the choices made regarding the design and how the program holds up over time.
What Is Evaluation?
Evaluation is the process of reviewing both the instructional components and the resulting outcomes of instruction to determine whether instruction achieves the desired outcomes.
Kirkpatrick's evaluation model proposes four levels of evaluation: reaction, learning, behavior, and results.
Level 1 Reaction measures how participants react to the training (e.g., satisfaction?).
Level 2 Learning analyzes if they genuinely understood the training (e.g., increase in knowledge, skills, or experience?).
Level 3 Behavior looks at if they are utilizing what they learned at work (e.g., change in behaviors?), and
Level 4 Results determine if the material positively impacted the business/organization. While this is a reasonably simplistic model, it provides a framework for understanding evaluation and has provided a significant evaluation model to the field of instructional design.
This model was developed by Dr. Donald Kirkpatrick (1924 – 2014) in the 1950s. The model can be implemented before, throughout, and following training to show the value of training to the business.
4 Signs Your eLearning Course Needs Improvement
1) Low completion rates
-
If many learners are consistently losing interest as the course progresses (aka not completing your courses), you may need to look at your course offering. There could be several reasons why learners aren't completing the course, and you need to find out which ones apply. The content may be boring, the exercises may be too difficult, or learners may not see how it relates to their everyday duties.
2) Poor learner engagement
-
A highly- effective eLearning course typically involves high engagement rates. They ask questions regularly and have discussions with each other. If you started with high engagement, but the levels seem to be falling, you should notice this too. This is a sign that you no longer have your students' attention.
3) Negative feedback
-
Whatever method you use, you need to take negative reviews seriously! One or two bad reviews are not a cause for primary concern, but if many of your students have something negative to say, you need to overhaul your course. Potential learners will look at reviews before signing up for your next course.
4) Participants' behavior is not changing
-
If your eLearning course is effective, learners will gradually change their behavior. Whether you're trying to improve customer service or teach workers to use a new piece of software, you should be able to see progress in their abilities. Repetition of the course material and reminders of the expected behavior can go a long way since behavioral change takes time. If only a few employees have adapted their actions or minimal overall impact, you need to change the course.
There comes a time in the life of any product when it needs to be refreshed, and it is no different with eLearning courses.
However, you may not realize the time to make some changes when it's your course. After all, it's not very easy to assess your work.
When Do We Evaluate?
Three commonly used types of evaluation for instruction are formative, summative, and confirmative (Morrison et al., 2019; Ross & Morrison, 2010).
Formative evaluation is conducted during the design process to provide feedback that informs the design process.
Summative evaluation is shown at the end of the design process to determine if the instructional product achieves the intended outcomes.
Confirmative evaluation is conducted overtime to choose the lasting effects of instruction.
Each of these evaluation stages is examined in detail here, both through the definition of the form itself and a discussion of some of the critical tools within each.
"When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative." – Robert E. Stake (M. Scriven, 1991, p. 169
Formative Evaluation
A formative evaluation (sometimes referred to as internal) is a method for judging the worth of a program while the program activities are forming (in progress). They can be conducted during any phase of the ADDIE process.
This part of the evaluation focuses on the process. Its purpose is to evaluate instruction and instructional materials to obtain feedback that drives revisions to make instruction more efficient and effective.
One way to think about this is to liken it to a chef tasting his food before sending it to the customer. Thus, formative evaluations are done on the fly.
They permit the designers, learners, instructors, and managers to monitor how well the instructional goals and objectives are being met.
Its primary purpose is to catch deficiencies ASAP so that the proper learning interventions can take place that allows the learners to master the required skills and knowledge.
Summative Evaluation
A summative evaluation (sometimes referred to as external) is a method of judging the worth of a program at the end of the program activities (summation). The focus is on the outcome.
Dick et al. (2015, p. 320) claimed the ultimate summative evaluation question is "Did it solve the problem?" That is the essence of summative evaluation.
Continuing with the chef analogy above, one asks, "Did the customer enjoy the food?" (M. Scriven, 1991). The parties involved in the evaluation take the data and conclude the effectiveness of the designed instruction.
However, summative evaluation has developed into a more complex process over time than the initial question may let on. In modern instructional design, practitioners investigate multiple questions through assessment to determine learning effectiveness, learning efficiency, cost-effectiveness, and attitudes and reactions to learning (Morrison et al., 2019).
Confirmative Evaluation
A confirmative evaluation aims to determine if the instruction is effective and if it meets the organization's defined instructional needs.
In effect, did it solve the problem? The customer ate the food and enjoyed it. But, did they come back?
Confirmative evaluation goes beyond the scope of formative and summative assessment and looks at whether the long-term effects of instruction are what the organization was hoping to achieve.
Is instruction affecting behavior or providing learners with the skills needed as determined by the original goals of the instruction?
Confirmative evaluation methods may not differ much from formative and summative because they occur after implementing a design.
Moseley and Solomon (1997) described confirmative evaluation as maintaining focus on what is important to your stakeholders and ensuring the expectations for learning continue to be met.
Conclusion
Evaluation is the process of determining whether the designed instruction meets its intended goals.
Evaluation allows instructional designers to ensure all stakeholders agree that the developed instruction meets the organizational goals.
In addition, evaluation helps us determine whether learners can transfer the skills and knowledge learned into long-term changes in behavior and skills required for the target context.
Now that you know the Why, What, and When of Instructional Design evaluation, the next question is how to do it? Here is how...
JOIN THE
eLearning and Instructional Design for Beginners Community
- In-depth courses & training
Access my rapidly growing library, attend monthly live training & accountability support groups
- Exclusive tools & members-only discounts
Tools, templates, downloads, checklists and more - plus receive special perks & discounts
- Supportive community & network
Feedback and support from fellow instructional designers, career-driven business owners, and experts who will keep you on track
Get Your Software Toolkit for Instructional Designers
Tools & processes that will help you plan, build, and grow your instructional design career and freelance business.
