2 Kirkpatrick Evaluation Model 1u166e

Principles of Effective Evaluation Prof Keithia Wilson Academic Leader Student Success & Retention

Importance of Evaluation  

  

Results in evidence-based practice Helps us to identify the most effective & strategic interventions (make a difference or add value) Leads to sustainable & enjoyable practice Provides ability to School & Group leadership Contributes to scholarship of the FYE

Kirkpatrick (1994) Model of Evaluation - Overview 4 levels of outcome evaluation – 1. 2. 3.

4.

Level Level Level Level

1 2 3 4

– – – –

Reaction Learning Behaviour Results

Level 1 - Reaction Measures participant’s reactions to the training program, including:  reactions to the overall program (outcomes) e.g., * “To what extent did you find the Orientation program useful?”  reactions to specific components of the program (processes) e.g., * What aspects of Orientation did you MOST APPRECIATE and found USEFUL? * What aspects of Orientation did you LEAST APPRECIATE and feel is MOST IN NEED OF IMPROVEMENT?

Level 1 - Reaction 

Important to include closed-ended items including rating scales e.g., How effective was the Orientation program overall in helping you to make a transition to university? not at all effective very effective 1 2 3 4 5 6 7 How useful was the initial information session on the predictors of success in first year? not at all useful very useful 1 2 3 4 5 6 7



As well as open-ended items

What else would you have liked included in the Orientation program today?

Level 1 - Reaction Some FYE examples –  Starting@Griffith Survey questions on Orientation (retrospective student survey)  School specific Orientation Program evaluation at the end of O-Day & the Common Time (ongoing Orientation Program) – student surveys, focus groups  Peer Mentoring Program final evaluation by mentors & mentees

Level 2 - Learning Measures what participants have learned from involvement in the program  What is measured needs to relate to what was covered in the program viz. learning objectives  Typically covers knowledge, skills, or attitudes  Needs to include both rating scales & open-ended questions  Can include self-report & tests of actual knowledge

Level 2 - Learning Some examples – 1. Student self-report * I feel confident that I know what to do to be successful academically this semester not at all confident very confident 1 2 3 4 5 6 7

* What have you learned about the transitions you will need to make as a commencing student in order to be successful? 2. Student knowledge * I know the name of my First Year Advisor yes no (circle one) * I know the difference between a lecture and a tutorial (circle one) Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree

Level 2 - Learning Some FYE examples –  Starting@Griffith Survey data evaluates learning outcomes in of knowledge & attitudes (e.g., knowing where to go to get help; perceptions of good teaching)  Outcome evaluation for Peer Mentees (What are your 3 most important learnings from your participation in the Peer Mentoring Program?)  First-Assessment-First- amber alert intervention (What have you learned as a result of your participation?)

Level 3 - Behaviour Measures the transfer of knowledge, skills & attitudes from the training context to in vivo or real-life contexts (the classroom & study).  Evaluate both before & after the program if practical  Use survey, focus groups, interviews with students, mentors, staff (convenors, sessionals)  Starting@Griffith Survey (Did you attend Orientation? How much time are you spending on study?)

Level 3 - Behaviour Some FYE examples –  Starting@Griffith survey data  Attendance at E-Day, O-Day & Common Time/ASP Program  Attendance at lectures & tutes  Assessment submission rates in courses

Level 4 - Results Measures “return-on-investment”, or the extent to which the program has produced results. Some FYE examples include –  Hard student outcomes such as * rates in courses for assessment items * GPA in semester 1, 2 & FY overall * student retention into year 2  Soft student outcomes such as – * student satisfaction

Evaluation Key question – What difference do our FYO&E interventions (e.g., Orientation, Peer Mentoring, Attendance, First Assessment Intervention etc.) make overall? Program/Intervention1 Reactions2 Learning 3 Behaviour 4 Overall Productivity/Result

Program/InterventionFeelKnowDowhat is the result or difference overall?