Here at Harper Adams we’ve been using computer-aided assessment since 2010. Our Questionmark Perception platform has be used for both formative and summative assessment. In this article I will list the top 10 things I have learned if you want to run computerised assessments successfully.
- Involve as many people as possible as early as possible. Computer-aided assessment (CAA) impacts on a large number of people from numerous academic and administrative departments. Making sure all of the stakeholders are involved in the process from the start will avoid miscommunication, ensure the university’s systems (technical and non-technical) are correctly understood and create an atmosphere where people feel their views are valued.
- Do not underestimate the size of the task. University exam procedures have existed for decades. Building computer aided assessment procedures from scratch will take more than simply recycling what exists and forcing it upon the software.
- Hold your nerve. When users first begin to create their assessments the amount of up front work required will seem significantly worse than traditional assessments and doubts may creep in regarding the practicality and productivity benefits possible. However, if you can see your plan through to the post-assessment time saving pay off by automated marking you can overcome any earlier uncertainty. Where the same formative assessments can be used over a number of years with potentially hundreds of students the benefits will be obvious.
- Establish milestones. It will be much easier to make sure exams stay on track for prompt delivery if milestones are decided at the beginning. Assessments can require the combined efforts of multiple collaborators and fixed objectives will help ensure everyone is working to the same timeline.
- Build a bridge. For some staff jumping from paper to a new, possibly intimidating online environment may be too big a transition. Plan for the long term and try gradually moving from one format to another over a number of exam cycles using optical mark reading (OMR) technology to bridge the gap by combining paper-answer sheets with automated marking.
- Don’t over extend. When choosing your software evaluate the in-house skills you have available before buying your platform. It would be wonderful to have custom-built software that can be developed to suit your every need but if you have only a small number of staff this will likely end in tears, especially if the work has been done in-house or these individuals have a number of other conflicting responsibilities. An off-the-shelf product that has more limited functionality but a support service a phone call away might be less exciting to create but may be more realistic for smaller institutions.
- You can’t win them all. Some staff may show interest in doing online summative exams but only teach large modules which exceed the capacity of your computer suites. If these exams will not work using the OMR method you will have to admit defeat. Turning away a customer can be disappointing but perhaps they could be encouraged to try formative assessments that students can use their own computers for instead?
- Become part of the machine. Try to integrate online assessment into the existing traditional institutional exam strategy as much as possible. Encouraging CAA to be perceived as just a part of the everyday running of the university will make it less alien to people who are wary of such technology.
- It’s just a tool. During discussions with possible users do not focus on how the software works. Focus on what CAA can do for them. Lecturers don’t need to know technical details such as whether your system uses a particular software or is built on Microsoft or Lynux servers. At this stage what’s important is whether the assessment types available will match their pedagogical needs.
- Meet in person. Telephone conversations and electronic communications won’t get the job done in the long term. Something will eventually slip through the cracks. Ensure you have regular fact-to-face meetings with assessment development teams to make sure everyone is on the same page. If necessary sit around a computer and work on an assessment together.
Obviously setting up an online assessment facility is far more complex than what can be summed up in this article but following these suggestions should make the process easier.