Adoption and attitudes of staff to mathematical e-assessment

Chris Graham, Newcastle University

<p><link href="https://fonts.googleapis.com/css?family=Atkinson Hyperlegible" rel="stylesheet"/></p>
<h1 style="text-align: center; font-size: 1.6em;">Perspectives of mathematical e-assessment on a pathway to institution-wide adoption</h1>
<p style="text-align: center; font-size: 1.0em;">Chris Graham</p>
<div id="branding">
<div>
<a href="https://www.ncl.ac.uk" target="_blank">
<img alt="Newcaste University" class="nulogo" src="../../../static/images/logos/newcastle-university-logo.png"/>
</a>
<p>School of Mathematics, Statistics & Physics Digital Learning Unit</p>
</div>
<div class="container">
<div class="col">
<a href="https://numbas.org.uk" target="_blank">
<img alt="Numbas" class="numbas" src="../../../static/images/logos/numbas.png"/>
</a>
</div>
<div class="col">
<a href="https://www.advance-he.ac.uk/cate-team/e-learning-unit-school-mathematics-statistics-physics-newcastle-university" target="_blank">
<img alt="Collaborative Awards for Teaching Excellence (CATE) winners 2020" class="cate" src="../../../static/images/logos/cate.png"/>
</a>
</div>
<div class="col">
<a href="https://chirun.readthedocs.io/en/latest/" target="_blank">
<img alt="Chirun" class="chirun" src="../../../static/images/logos/chirun.png"/>
</a>
</div>
</div>
</div>
<hr/>
<section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.955);  padding: 1em;"><h2>This talk</h2>
<p>Reflect on the extent to which mathematical e-assessment has been adopted at Newcastle University, UK</p>
<p>Give insight into perspectives of students and colleagues</p>
<p>Explore barriers to engaging with mathematical e-assessment</p>
<hr/>
</section><section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h2>Our team at Newcastle</h2>
<p>I lead the <strong>Digital Learning Unit</strong> in our University's School of Maths, Stats & Physics. </p>
<p>We have a unique set up: perspectives that I hope will be insightful (but this is also a caveat for relevance to other institutions).</p>
<p>The existence of our unit is thanks to the development of The Numbbas e-assessment system</p>
<hr/>
</section><section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h2>Numbas at Newcastle</h2>
<p>Numbas has been used at Newcastle since 2012.</p>
<p>Embedded in nearly 100 modules, primarily in Maths, Stats & Physics and the School of Engineering, with over 100,000 student assessment attempts each year.</p>
<p>A lot of formative use, but I have mainly been interested here in summative.</p>
<hr/>
</section><section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h2>Perspectives</h2>
<p>Over the past semester, I have been trying to get a snapshot of attitudes to e-assessment.</p>
<p>Several dialogues with key partners in the use of Numbas:</p>
<ul>
<li>Students</li>
<li>Maths Transition Officer</li>
<li>Director of Education</li>
<li>Academic colleagues in our department and Engineering </li>
<li>Chair of the Board of Examiners</li>
<li>University Learning Enhancement & Technology Manager</li>
</ul>
<hr/>
</section><section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h2>Maths & Stats Stage 1</h2>
<p>8 maths modules at stage 1. Of particular interest because of the scale of e-assessment use:</p>
<ul>
<li>7 use Numbas for coursework</li>
<li>4 hybrid Numbas final exams, 4 written exams. </li>
</ul>
<p>1 module assessment is fully written on pedagogic grounds.  </p>
<hr/>
<section class="section3" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h3>Reflections: Students & Transition Officer</h3>
<p><em>On e-assessment used for transition:</em> </p>
<p class="indent">Powers transition material for our stage 1 with mini tests and a diagnostic assessment. Overwhelming positive from student perspective is the formative, infinite nature.</p>
<p><em>On how students see summative assessments:</em> </p>
<p class="indent">Students see assessments as less onerous (don't need to write out tidy solutions) and appreciate much quicker marking</p>
<p class="indent">Opinion is that Numbas is most impactful in a hybrid mode. We don't need to see a lot of working, but it does need to be nurtured.</p>
<hr/>
</section></section><section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h2>Maths & Stats Big Picture</h2>
<figure class="figure1" data-title="Maths 4 year course modules"><img alt="Maths 4 year course modules" src="images/modules.png" style="margin: 5px 0;" width="100%"/></figure><hr/>
<section class="section3" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h3>Reflections: Director of Education</h3>
<p><em>On the benefits of e-assessment:</em></p>
<ul>
<li>for staff: workload  </li>
<li>for students: timeliness of feedback</li>
<li>for the School: modernity associated with its use</li>
</ul>
<p><em>Examples of e-assessment used badly/well:</em><br/></p>
<ul>
<li>Badly: Jeopardy of having lots of working for one answer box.</li>
<li>Well: tailored feedback; error-carried and alternative answers.</li>
</ul>
<hr/>
</section><section class="section3" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h3>Reflections: Academic colleagues</h3>
<p><em>On the main challenges to adoption:</em></p>
<ul>
<li>Entrenched views </li>
<li>Workload</li>
<li>Awareness of what's possible </li>
</ul>
<p><em>On written vs online exams:</em> </p>
<p class="indent">One colleague reflected on retaining a written exam because they would like to see written mathematics. But in practice tend to be very soft on this in the exam: don't have time to mark anything except bottom line.</p>
<hr/>
</section><section class="section3" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h3>Reflections: Chair of the Board of Examiners</h3>
<p><em>On written assessments being replaced by digital:</em> </p>
<p class="indent">Good for establishing foundations, but important for students to do higher level mathematical writing for most exams. Very positive about hybrid solutions.</p>
<p><em>On the impact of AI:</em> </p>
<p class="indent">Not very concerned except for small pockets, e.g. projects. In-course assessments are mostly much more about engagement.</p>
<p><em>On how externals view digital assessments:</em> </p>
<p class="indent">Ours see written/digital as equals, considered on their own merit.</p>
<hr/>
</section></section><section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h2>Wider University Context</h2>
<figure class="figure2" data-title="Wider University proportions"><img alt="Wider University proportions" src="images/bydepartment.png" style="width: 70%"/></figure><hr/>
<section class="section3" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h3>University Exam Software</h3>
<p>Newcastle use <em>Inspera</em> as a digital exam solution. Numbas is also an approved option:</p>
<figure class="figure3" data-title="Exam form"><img alt="Exam form" src="images/examform.png" style="margin: 5px 0;"/></figure><p>We have muscled in to all of the exam processes here:</p>
<ul>
<li>Digital exams steering group</li>
<li>Invigilator training</li>
<li>Exam administration</li>
<li>Exam support channels</li>
</ul>
<hr/>
<section class="section6" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h6> </h6>
<p>28 exams with 6000 exam sittings in Numbas.</p>
<p>Exam sittings by format:</p>
<figure class="figure4" data-title="Exam form"><img alt="Exam form" src="images/examformats.png" style="margin: 5px 0; width: 60%;"/></figure><hr/>
</section></section><section class="section3" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.95);  padding: 1em;"><h3>Reflections: University Learning and Enhancement Team Manager</h3>
<p><em>On how they would manage mathematical exams without our support:</em></p>
<p class="indent">Would struggle to support mathematical exams with just Inspera: doesn't have the extensive features. Would not have expertise to support and would have been a struggled to interpret needs. Tender for exam software may have been different.</p>
<p><em>On concerns about non-Inspera exams:</em></p>
<p class="indent">Nervous about non-standard exams, without having people who they can call upon.</p>
<hr/>
</section></section><section class="section2"><h2>Conclusions</h2>
<p>In-course assessment is settling to a hybrid of written and digital across all levels (course, module, single assessment)</p>
<p>There are still challenges engaging colleagues and ensuring that assessments are well designed to avoid student frustration</p>
<p>Specialist mathematical e-assessment exams are desirable and possible, but difficult to integrate at institutional level</p>
<hr/>
</section><section class="section2"><h2>Thanks for listening!</h2>
<p>Thank you to Kate Henderson, Jon Goss, Chris Pearson, Martina Balagovic, Simon Lambert, Terry Charlton, Magda Carr, Ryan Doran and student representatives for dialogue discussions. </p>
<p>And, as ever, to my colleagues in the Digital Learning Team for all of their great work!</p>
</section>