Dissertation Marking and Feedback using ATLAS – Graeme Ferris and Paul Caban

Paul: I’m going to start with a brief history of dissertation management in the Business School. Up until 2002 we were using a paper-based system (version 0). That was great but not scalable. We were expanding so we needed something new.

For version 1 the school had it’s first dedicated developer who built us a system using ColdFusion and SQL. That managed the process but it wasn’t great. ColdFusion is also not a tool particularly well loved in the University. So for version 2 we had started working on php and Postgres… But that project became problematic in lots of ways and didn’t result in a working version that achieved what we needed it to do.

Version 3 was built within the school and was based on php and Postgres, it was built swiftly but needs were changing, the requirements were diverse, and it was becoming unmanagable.

So the dissertation process is loosely:

1. Sign up – supervisor choice and allocation. People state their preference, allocations are made.
2. Supervision and submission. Ethics etc. processes are gone through.
3. Marking. Currently on paper. Agreement and audit trail between markers. Then marks and feedback goes to the students.
4. Reporting and Quality Assurance processes.

However… The developer that had built this system decided to move to a new role. At the same time we saw this system as in need for reconsideration, high on our risk register, and we had new needs to accomodate. We wanted to provide electronic feedback, other systems looked possible, there were various things already in the VLE which we’d previously had to hand code. We then had an external review and recommendations… Part of that was a steer towards internal tools. Sharepoint was one possibility… Atlas/Pebblepad was the other option and it is really simple to use. By this point we were in late autumn 2014, and needed a new system in place by February for dissertations. So, we decided to use the old part of our tool for student sign up, but do something new with Pebblepad for the rest of the process.

So, for the project we created a design brief. We did a massive stakeholder involvement process, spoke to committees, spoke to lots of staff. We designed a project that would have no (negative) impact on students. And ensured that training and documentation would be in place.

Graeme:With a custom system we have endless freedom… But that was the problem with the previous systems – there were too many options that led to unneccassary complexity. So, for this solution, we wanted to ensure the essentials were in there but jetison some of the more obscure processes requests – exceptions for those not wanting to follow the core process.

So, one of the key things was double-marking – that is a University requirement for any work over 60 credits. Blind marking is not mandatory across the university for double marking – but it is Business School policy. The difference, to note, is that for anonymous marking you don’t know who the student is – tricky with supervised dissertations of course. Blind marking is where one marker does not see the other’s marks until initial marking has been completed. That requirement had design implications. And once that double blind marking was done the markers have to see each others marks to reconcile and discuss.

In terms of the reconciliation we needed some depth of reasoning to be captured. So, in terms of managing this we wanted a process that involved:

• Initial marking blind
• Reconciliation
• Recording reconciliation notes and comments – crucially always ensuring that the student had the appropriate feedback to the final mark. The supervisor has to manage this to ensure it is correct.
• Marks fed back.

Early iterations were a sort of time-based model, making use of the permissions for different roles within the system. So permissions set for blind initial marking, meaning that at an agreed data permissions changed to allow reconciliations. That was suggested but… Actually the feedback was that that was unacceptable since markers mark at different paces and timings, it wouldn’t be possible.

This was a particular concern as the available functionality was via role based permissions – which means if you were marking 4 students, and second marking 5 students, all of those marks would be out of blind mode at the same time. To overcome this we created a model using separate workspaces for: initial marking (blind) – marker view and completing marking template, admin view and moving marked submissions; reconciliation – marker view and completion of reconciliation inputs, admin view; archive – locked down for exam board / feedback to students; Reporting function.

So, I’m going to demo all these workspaces now… All academic staff have a tutor role. Admin have lead tutor role – they can see everything. For academic staff it’s anonymous and blind – they can’t see names of students, each other’s names or marks. And we also made use of the inbuilt Atlas concept of “sets”… Very useful indeed. You set up a set… Academic A marks 4 students, make a set for that. When the marker goes in they only see the set they are allocated to… They don’t have to find their students… We have sets for first markers, sets for second markers… Can pick/filter between them. And the other reporting aspect uses sets – for cohorts, for groups, etc. So that you can account just on one MSc, or on a cluster of MScs in the same area, etc. So that’s really lovely. And admin can see all of the sets…

Share/Bookmark