Perspectives of mathematical e-assessment on a pathway to institution-wide adoption

Chris Graham, Newcastle University

<p><link href="https://fonts.googleapis.com/css?family=Atkinson Hyperlegible" rel="stylesheet"/></p>
<h1 style="text-align: center;">Adoption and attitudes of staff to mathematical e-assessment</h1>
<p style="text-align: center; font-size: 1.0em;">Chris Graham</p>
<div id="branding">
<div>
<a href="https://www.ncl.ac.uk" target="_blank">
<img alt="Newcaste University" class="nulogo" src="../../../static/images/logos/newcastle-university-logo.png"/>
</a>
<p>School of Mathematics, Statistics & Physics Digital Learning Unit</p>
</div>
<div class="container">
<div class="col">
<a href="https://numbas.org.uk" target="_blank">
<img alt="Numbas" class="numbas" src="../../../static/images/logos/numbas.png"/>
</a>
</div>
<div class="col">
<a href="https://www.advance-he.ac.uk/cate-team/e-learning-unit-school-mathematics-statistics-physics-newcastle-university" target="_blank">
<img alt="Collaborative Awards for Teaching Excellence (CATE) winners 2020" class="cate" src="../../../static/images/logos/cate.png"/>
</a>
</div>
<div class="col">
<a href="https://chirun.readthedocs.io/en/latest/" target="_blank">
<img alt="Chirun" class="chirun" src="../../../static/images/logos/chirun.png"/>
</a>
</div>
</div>
</div>
<hr/>
<section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h2>This talk</h2>
<ul>
<li>
<p>Reflect on the extent to which mathematical e-assessment has been adopted at Newcastle University, UK</p>
</li>
<li>
<p>Give insight into perspectives of key colleagues</p>
</li>
<li>
<p>Explore barriers to engaging with mathematical e-assessment</p>
</li>
</ul>
<p>Hopefully this will raise questions and discussion: add your questions to the <strong>Matrix Live Chat</strong>.</p>
<hr/>
</section><section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h2>Our team at Newcastle</h2>
<p>Myself, Christian and others make up the <strong>Digital Learning Unit</strong> in our University's School of Maths, Stats & Physics. </p>
<p>We have a unique set up: perspectives that I hope will be insightful (but this is also a caveat for relevance to other institutions).</p>
<p>The existence of our unit is thanks to the development of Numbas...</p>
<hr/>
<section class="section3" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h3>From EAMS 2016!</h3>
<figure class="figure1" data-title="Chris Graham, Christian Lawson-Perfect and Bill Foster"><img alt="Chris Graham, Christian Lawson-Perfect and Bill Foster" src="images/eams2016.jpg" style="margin: 5px 0; width: 70%;"/></figure><p><small style="margin-top: -25px; margin-left: 50px;">L-R: Chris Graham, Christian Lawson-Perfect, Bill Foster</small></p>
<p>Our team has grown significantly since: from next month, 6 members!</p>
<hr/>
</section></section><section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h2>Numbas at Newcastle</h2>
<p>Numbas has been used at Newcastle since 2012.</p>
<p>Embedded in nearly 100 modules, primarily in Maths, Stats & Physics and the School of Engineering. </p>
<p>Over 100,000 student assessment attempts each year.</p>
<p>Part of the University digital exam solution.</p>
<hr/>
</section><section class="section2" data-background="../../../static/images/newcastle-fade.jpg" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h2>Perspectives</h2>
<p>In our School:</p>
<ul>
<li>Maths Transition Officer</li>
<li>Director of Education</li>
<li>Academic colleagues</li>
<li>Chair of the Board of Examiners</li>
</ul>
<p>And partners elsewhere in our University:</p>
<ul>
<li>Engineering Champion</li>
<li>University Learning Enhancement & Technology Manager</li>
</ul>
<hr/>
<section class="section3" data-background="../../../static/images/eams23/pie_msp1.png"><h3> </h3>
<hr/>
</section></section><section class="section2" data-background="../../../static/images/eams23/pie_msp1.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h2>Maths & Stats Stage 1</h2>
<p>8 maths modules at stage 1: </p>
<ul>
<li>7 use Numbas for coursework</li>
<li>4 hybrid Numbas final exams, 4 written exams. </li>
</ul>
<p>1 module assessment is fully written on pedagogic grounds.  </p>
<hr/>
<section class="section3" data-background="../../../static/images/eams23/pie_msp1.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h3>Reflections: Transition Officer</h3>
<p><em>On how e-assessments are used for transition:</em> </p>
<p class="indent">Powers transition material for our stage 1 with mini tests and a diagnostic assessment. Overwhelming positive from student perspective is the formative, infinite nature.</p>
<p><em>On how students see e-assessments:</em> </p>
<p class="indent">Students see assessments as less onerous (don't need to write out tidy solutions) and appreciate much quicker marking</p>
<p class="indent">Opinion is that Numbas is most impactful in a hybrid mode. We don't need to see a lot of working, but it does need to be nurtured.</p>
<hr/>
</section><section class="section3" data-background="../../../static/images/eams23/pie_msp.png"><h3> </h3>
<hr/>
</section></section><section class="section2" data-background="../../../static/images/eams23/pie_msp.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h2>Maths & Stats Big Picture</h2>
<figure class="figure2" data-title="Maths 4 year course modules"><img alt="Maths 4 year course modules" src="images/modules.png" style="margin: 5px 0;" width="100%"/></figure><hr/>
<section class="section3" data-background="../../../static/images/eams23/pie_msp.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h3>Reflections: Director of Education</h3>
<p><em>On the benefits of e-assessment:</em></p>
<ul>
<li>for staff: workload  </li>
<li>for students: timeliness of feedback</li>
<li>for the School: modernity associated with its use</li>
</ul>
<p><em>Examples of e-assessment used badly/well:</em><br/></p>
<ul>
<li>Badly: Jeopardy of having lots of working for one answer box.</li>
<li>Well: tailored feedback; error-carried and alternative answers.</li>
</ul>
<p><em>On the main challenges to adoption:</em></p>
<ul>
<li>Entrenched views </li>
<li>Workload</li>
<li>Awareness of what's possible </li>
</ul>
<hr/>
</section><section class="section3" data-background="../../../static/images/eams23/pie_msp.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h3>Reflections: Academic colleagues</h3>
<p><em>On written work:</em> </p>
<p class="indent">One colleague reflected on retaining a written exam because they would like to see written mathematics. But in practice tend to be very soft on this in the exam: don't have time to mark anything except bottom line.</p>
<p><em>On Training opportunities:</em> </p>
<p class="indent">"There's never a good time, but it's always a good time"</p>
<p><em>On getting support for writing questions:</em> </p>
<p class="indent">With some experience of writing, would be happier doing it themselves than getting low level assistance (e.g. postgraduate).</p>
<hr/>
</section><section class="section3" data-background="../../../static/images/eams23/pie_msp.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h3>Reflections: Chair of the Board of Examiners</h3>
<p><em>On written assessments being replaced by digital:</em> </p>
<p class="indent">Good for a limited number of topics: checking recall, basic computations, basis on which all knowledge is built, but important for students to do higher level mathematical writing for most exams. Very positive about hybrid solutions.</p>
<p><em>On the impact of AI:</em> </p>
<p class="indent">Not very concerned except for small pockets, e.g. projects. In-course assessments are mostly much more about engagement.</p>
<p><em>On how externals view digital assessments:</em> </p>
<p class="indent">Ours see written/digital as equals, considered on their own merit.</p>
<hr/>
<section class="section6" data-background="../../../static/images/eams23/pie_eng.png"><h6> </h6>
<hr/>
</section></section></section><section class="section2" data-background="../../../static/images/eams23/pie_eng.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h2>School of Engineering</h2>
<p>Heavy users of Numbas: 55,000 student assessment attempts across 25 modules, with 20 final exams.</p>
<p>Engineering Champion promotes Numbas use in the School.</p>
<hr/>
<section class="section3" data-background="../../../static/images/eams23/pie_eng.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h3>Reflections: Engineering Champion</h3>
<p><em>On stage 1 exams:</em></p>
<p class="indent">Rely heavily on automatic assessment. Split between Numbas and Inspera, with the latter mostly MCQ questions.</p>
<p><em>On other Numbas applications:</em></p>
<p class="indent">Similar distribution to Maths at later stages, where they want more focused marking for longer questions.</p>
<p class="indent">Some very innovative use in e.g. lab-based activities. Lots of interest, but no awareness of, options for hybrid assessments.</p>
<p><em>On the future:</em></p>
<p class="indent">Lots of opportunities to coordinate further between our team and their Numbas Champion.</p>
<hr/>
</section></section><section class="section2" data-background="../../../static/images/eams23/100percent.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h2>Wider University Context</h2>
<figure class="figure3" data-title="Wider University proportions"><img alt="Wider University proportions" src="../../../static/images/eams23/allsubjects.png" style="width: 50%"/></figure><hr/>
<section class="section6" data-background="../../../static/images/eams23/100percent.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h6> </h6>
<p>Newcastle use <em>Inspera</em> as a digital exam solution. Numbas is also an approved option:</p>
<figure class="figure4" data-title="Exam form"><img alt="Exam form" src="images/examform.png" style="margin: 5px 0;"/></figure><p>We have muscled in to all of the exam processes here:</p>
<ul>
<li>Digital exams steering group</li>
<li>Invigilator training</li>
<li>Exam administration</li>
<li>Exam support channels</li>
</ul>
<hr/>
</section><section class="section6" data-background="../../../static/images/eams23/100percent.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h6> </h6>
<p>28 exams with 6000 exam sittings in Numbas.</p>
<p>Exam sittings by format:</p>
<figure class="figure5" data-title="Exam form"><img alt="Exam form" src="images/examformats.png" style="margin: 5px 0; width: 60%;"/></figure><hr/>
</section><section class="section3" data-background="../../../static/images/eams23/100percent.png" style="background-color: rgba(255, 255, 255, 0.9);  padding: 1em;"><h3>Reflections: University Learning and Enhancement Team Manager</h3>
<p><em>On how they would manage mathematical exams without our support:</em></p>
<p class="indent">Would struggle to support mathematical exams with just Inspera: doesn't have the extensive features. Would not have expertise to support and would have been a struggled to interpret needs. Tender for exam software may have been different.</p>
<p><em>On concerns about non-Inspera exams:</em></p>
<p class="indent">Nervous about non-standard exams, without having people who they can call upon.</p>
<hr/>
</section></section><section class="section2"><h2>Thanks for listening!</h2>
<p>Thank you to Kate Henderson, Jon Goss, Chris Pearson, Martina Balagovic, Simon Lambert, Terry Charlton, Magda Carr and Ryan Doran for dialogue discussions. </p>
<p>And, as ever, to my colleagues in the Digital Learning Team for all of their great work!</p>
</section>