Learning+Analytics+and+Feedback+in+CBT

=Learning analytics and feedback in computer-based or computer-adaptive testing = Silvester Draaijer, Caroline Timmers, Sharon Klinkenberg 30 August 2012, 09.00-10.30 Verslag door: Laura de Bruin

Korte Samenvatting
//Online tests // Online quizzing is perfect for data collection and learning analytics to support feedback.

If you do tests, you learn from it. In one study, Draaijer and colleagues found: - correlation between participation rate and exam scores; - correlation between quiz promotion and participation: if a teachers emphasizes the quiz “study, do it in time!”, then at least the participation rises, and that is a prerequisite for high scores; - correlation between credit given and participation.

Collaborative testing is also an option, working in pairs or groups of three: that gives a very good retrieval practice. For example the process of election of multiple choice questions by discussing together first. Or for example the option to choose ‘I think I have the answer’ before the answers come.

//The feedback element in online testing // An important element of online testing is feedback. Feedback can be given: - per question: for example ‘answer question again’; - per group of questions; - per test as a whole: directly after, longer and even longer.

<span style="font-family: Arial,Helvetica,sans-serif;">Difficult to say what gives the best result in a specific situation, feedback with delay can be powerful.

<span style="font-family: Arial,Helvetica,sans-serif;">Analytics can help with: <span style="font-family: Arial,Helvetica,sans-serif;">- promotion of use; <span style="font-family: Arial,Helvetica,sans-serif;">- decisions of repetition; <span style="font-family: Arial,Helvetica,sans-serif;">- timing of repetition; <span style="font-family: Arial,Helvetica,sans-serif;">- timing and type of feedback.

<span style="font-family: Arial,Helvetica,sans-serif;">Different types of feedback: <span style="font-family: Arial,Helvetica,sans-serif;">- knowledge of results: how much correct / wrong <span style="font-family: Arial,Helvetica,sans-serif;">- knowledge of correct response / answer <span style="font-family: Arial,Helvetica,sans-serif;">- elaborated feedback / remedial information <span style="font-family: Arial,Helvetica,sans-serif;">More informative feedback can be more effective than just knowledge of results. We hope that it results in learning and improvement. There can be a lot of pitfalls: loose of attention and motivation.

<span style="font-family: Arial,Helvetica,sans-serif;">Some conclusions from Timmer’s study: <span style="font-family: Arial,Helvetica,sans-serif;">- Feedback behaviour varies widely between students; <span style="font-family: Arial,Helvetica,sans-serif;">- Student response influences feedback behaviour. Propensity to use feedback is much higher for incorrect responses. <span style="font-family: Arial,Helvetica,sans-serif;">- Student achievement influences feedback behaviour.

//<span style="font-family: Arial,Helvetica,sans-serif;">Computer-adaptive testing (CAT) // <span style="font-family: Arial,Helvetica,sans-serif;">Rating of students in for example the Rekentuin is derived from ELO-rating in chest playing: new rating = old rating + weight x (score – predicted score). Time is always integrated in the scoring algoritm. Accuracy and time are the two components. Maybe students that answer the questions faster, are better. Guessing is punished, you lose points with incorrect answer. But when correct and fast: you get the rest time in points. The later your answer, the less points you can get. <span style="font-family: Arial,Helvetica,sans-serif;">Creating a filter / algorithm for each student and each item he solves, that is an example of learning analytics that gives a student diagnosis. Students get only videos that are applicable to their special problem area. The videos are the feedback. So learning analytics is useful for the feedback.

Inzichten
<span style="font-family: Arial,Helvetica,sans-serif;">Besides the 'inzichten' in the summary above, another 'inzicht' can be mentioned. Learning analytics can be very useful for the interaction between teacher and student when the student is used to make quizzes and multiple-choice test. The teacher can send a message to the student when seeing that the student has not made the test. During their lectures, teachers can give attention to the weak subject areas they analysed in the students’ scores or take students individually.

Review
<span style="font-family: Arial,Helvetica,sans-serif;">The presentations were rich of practical knowledge about online testing and (providing) feedback in online testing. The researchers self-critically asked the remaining question that applies to a lot of online testing, namely ‘is it a valid estimate of someone’s ability?’ That needs more research. Also the domain of feedback in online testing could be explored deeper in further research.

Toepasbaarheid
<span style="font-family: Arial,Helvetica,sans-serif;">One question was asked about the usefulness of feedback in case of quizzing: “feedback is about discussion, about interaction, but feedback in quizzing seems to be one-way conversation.” The answer of the presenter was: to see whether feedback is read and whether the person worked on it, can be an aid that stimulates interaction about the subject between student and teacher during the lesson. Then learning analytics can provide the problem areas that need feedback.

Keywords
online testing; feedback; learning analytics