Some aspects of the tests and their role in IIT mitigated against the effectiveness of the study. Firstly the test questions had only 4 alternatives, and there were no penalties. Both of these increase the effect of guessing, and thus increase the level of noise in the data. Pollard and Clark (Clark and Pollard, 1989; Clark and Pollard, 2004) show that even 5 alternatives is an improvement over 4, and that the effect of guessing is dramatically reduced by having penalties for incorrect responses. Students dislike penalties and consider them unfair, so including them in determining test scores may be counter productive. Increasing the number of alternatives in at least some questions is not hard. In particular, students answer comprehension questions by working out the answer and checking that it is on the list. A new distracter does not have to correspond to a particular mistake. It just has to be in the right range.

The other factor mitigating against the effectiveness of the study was the way that the test scores were used. The idea was that students who averaged 60% on the tests would reach the end of the semester already having passed the subject. However, students who knew they could not make the 60% dropped out of the tests. Further, students who had averaged 60% did not sit for paper A and those who were content with a pass did not sit for paper B. The effect was that fewer weaker students were represented in the analyses involving tests 2 and 3, fewer stronger students were represented in the analyses involving paper A and fewer weaker students were represented in the analyses involving paper B.

Improving the Tests

The tests can be improved by making changes to the individual test questions, and in the way the tests are integrated into IIT. Examining questions with poor discrimination can identify problemswith individual questions. Some question can be improved (question 4), whilst other questions may be better dropped (question 15) and new questions substituted. Even if the new questions are in the same category and of similar difficulty to an existing question, having several similar questions can mitigate against the effect of guessing. A further easy improvement is to increase the number of alternatives in some questions. This will reduce the effect of guessing. For some questions, 4 alternatives is natural (questions Testing Programming Skills with Multiple Choice Questions 1676, 10, 12). For others, especially comprehension questions, adding an extra alternative is simple (questions 7, 8, 17). For instance “2” or “3” could be added to the alternatives in question 7. There is nothing to say that the number of alternatives must be the same on each question.

During the semester of this study, test scores were not incorporated into exam scores. The idea was to allow students to reach the end of the semester already having passed the subject. This has not worked out well in practise, and will not be continued. Instead, each test will contribute 5% of the marks on paper A. This decision has been made on pedagogical grounds, but it will have the effect of making the subset of students who do a particular test and a particular paper more typical of the students taking IIT.

Reference : David CLARK, “Testing Programming Skills with Multiple Choice Questions,” Informatics in Education, Vol. 3, No. 2, 161–178, 2004.