Definition

Computer-assisted assessment (CAA) is a broad term which describes the application of computer technologies to the assessment process.'(HE Academy)

According to the draft of the forthcoming JISC glossary of e-assessment CAA can stand for Computer Assisted Assessment or Computer Aided Assessment both of which are synonymous with e-assessment. E-assessment is defined as:

The end-to-end electronic assessment processes where ICT is used for the presentation of assessment activity and the recording of responses. This Assessment includes the end-to-end assessment process from the perspective of learners, tutors, learning establishments, awarding bodies and regulators, and the general public.

What do we mean by assessment?

Assessment is one of the most significant areas of an educational system. It defines what students take to be important, how they spend much of their academic time and in may ways how they value themselves.

Rowntree (1987) says of assessment:

if we wish to discover the truth about an educational system, we must look to its assessment procedures.

In addition, assessment is important because students cannot avoid it, as Boud (1995) says:

Students can, with difficulty, escape from the effects of poor teaching, they cannot (by definition if they want to graduate) escape the effects of poor assessment

This underlines the importance of getting our assessment practices right for our students.(JISC)

Glossary of e-assessment

The Joint Information Systems Committee (JISC) has announced a forthcoming glossary of e-assessment. This document is designed to ‘create a greater awareness amongst the JISC community of the characteristics, benefits and challenges of assessment delivered by ICT.’

A draft glossary is currently available at the 3 Square website who are project partners with JISC. http://www.3sq.co.uk/e-assessmentglossary.htm

CAA Case Studies

These case studies are taken from a number of different publications relevant to UK higher education.

Diagnostic Testing

When students arrive at university their skills and abilities vary greatly. Although additional support sessions are available in subjects such as Mathematics or Sciences the take-up is often too late into the course as the problem is not recognised early enough. CAA can be used in the first weeks of a course to assess current levels of understanding with questions that are directly related to learning outcomes. Results of these tests give the lecturer a clear indication of students’ abilities and individuals can be referred to supplementary support sessions if necessary.

The supplementary support sessions can also use CAA which is rich in feedback and can direct student learning through hints, tips and encouragement. Students are free to re-take the diagnostic test at any time to check their progress. (LTSN Generic 2004, Assessment Series, case study 3 p.12)

Iterative Testing (summative assessment)

The amount of marking for a second year unit was particularly demanding for a course team and students were increasingly dissatisfied with the results. The unit was becoming increasingly popular and consisted of three pieces of coursework assignments and an end of year exam all of which were individually marked by staff. Some of the students were passing coursework but failing the exam and feedback from staff was arriving late because of the sheer quantity of coursework to mark. After a review of the situation it was decided to replace one of the assignments with five short objective tests.

Each test related to the material covered in a two week period and students received feedback at the end of each test. Students had to book themselves into a computer lab at a specific time to take tests that were invigilated by support staff. Teaching staff received results the following day allowing them to address gaps in the level of student understanding in the following seminar or lecture. Through feedback students gained a clear idea of how they were progressing with the course and were motivated to follow-up some of the feedback with suggested further reading and research. (LTSN Generic 2004, Assessment Series, case study 1 p.10)

Mock Examinations

Learners take these in their own time at a time that suits them but there is a time restriction to simulate exam conditions. The submissions are automatically marked (meaning that the tests must be objective) and if a learner performs poorly there is built-in, un-bias feedback provided by the tutor that gives an explanation and provides links to relevant learning materials. Further feedback can take place in face to face sessions providing a blended learning approach. (JISC 2004, Effective Practice with e-Learning p. 27)

Example: New South Wales Board of Studies Online Multiple Choice trial http://www4.boardofstudies.nsw.edu.au/

Poll or Survey

An effective just-in-time method to aid with teaching used, for example, to measure student opinions of a particular topic prior to a lecture or seminar. A television documentary or political debate is relevant to the current curriculum and the lecturer asks students to review/ view the material. The lecturer then elicits their opinions prior to the forthcoming class by creating and publishing a brief survey. (Blackboard, Educational Benefits of Online Learning http://resources.blackboard.com/scholar/general/pages/ictraining/Online_Learning_Benefits.pdf)

Self Assessment (formative assessment)

1st year undergraduate law students often did not understand what was expected of them until they were assessed. To make matters worse the material was taught in the first eight weeks of term which meant that they were expected them to engage with the new material whilst also adjusting to student-life.

A series of formative assessments using CAA were chosen as a way to aid students without dramatically increasing the workload of the tutors. Objective tests were devised and accompanied by guidance on the module, sample questions (from previous exams) and marking schemes. Much of the accompanying material was already in existence and made available alongside the tests in a VLE.

Students that had taken the quizzes performed significantly better in all aspects of the exam, including higher-order skills such as evaluation and synthesis. The ‘quiz-takers’ also showed a marked improvement in their attendance at seminars. Although feedback was generic and brief it was thought the immediacy of automatic marking had enabled learners to diagnose and respond more effectively to their own learning needs. (JISC 2004, Effective Practice with e-Learning p. 36)

Question Design

Questions and subsequent assessments need careful planning to achieve the desired outcomes. The basic premise of instructional design is to work with human models of psychology to aid learning.

Well-designed questions are particularly effective because they (1) provide learners with practice retrieving information from memory, (2) give learners feedback about their misconceptions, (3) focus learners’ attention on the most important learning material, and (4) repeat core concepts, giving learners a second chance to learn, relearn, or reinforce what they previously learned or tried to learn.

(Thalheimer, 2003)

Objective Questions

Objective questions measure declarative knowledge, that is to say information that only has one correct answer. A good example of objective questions in CAA is multiple choice. However this association is not without its problems as it is argued by Biggs (1999) that multiple choice does not test a high level of understanding and uses ‘the least demanding process, recognition’. Overcoming this association of CAA and its ability to measure only a low level of knowledge through the use of multiple choice ‘type’ questions is a problem for technologists and system admin.

Question Types

The list of “standard question types” varies from system to system but is largely defined by the range of item types supported by the IMS QTI v2.0 specification. http://www.imsglobal.org/question/index.html

What follows is an incomplete alphabetical list of CAA question types that you are welcome to contribute to…

Binary Choice

Calculated Formula

Formula questions that have changing values with each attempt

Calculated Numeric

Either/ Or

Essay

A free-text question that allows students to write as much as they like for an answer

File Response

Requires a student to upload a file. This can be useful for question such as simulations or programming that require evidence of live data

Fill-in Multiple Blanks

Prompts students for complete missing words in sentences or paragraphs

Hot Spot

Allows part of an image file to be the correct answer. Useful for graphical exercises like maps, medical or charts.

Jumbled Sentence

Similar to fill-in multiple blanks. This option offers a drop-down box of available answers for each missing word or blank.

Likert

see Opinion

Matching

Multiple Answer

Multiple Choice

Opinion Scale

Ordering

Short Answer

Similar to the free-text essay question but the instructor can limit the size of the response

Add further question types above here alphabetically

E-marking and Feedback

information to follow

References

JISC http://www.jiscinfonet.ac.uk/InfoKits/effective-use-of-VLEs/e-assessment/assess-purpose

HE Academy http://www.heacademy.ac.uk/resources.asp?process=full_record&section=generic&id=350

New South Wales Board of Studies online multiple choice trials. http://www4.boardofstudies.nsw.edu.au/

Thalheimer, W. (2003, January). The learning benefits of questions. Retrieved November 19, 2005, from http://www.work-learning.com/ma/PP_WP003.asp and Questionmark Perception Demo CD www.questionmark.com

Blackboard

This page is a category

To add a page to this category, add a link to this page on the last line of the page. You can add multiple categories to a page.

List of pages in this category:


CategoryCategory

Creative Commons License
The content of this wiki is licensed under the Creative Commons Attribution-ShareAlike 2.0 England & Wales Licence.

OSS Watch is funded by the Joint Information Systems Committee (JISC) and is situated within the Research Technologies Service (RTS) of the University of Oxford.