Handheld Learning

Creative Assessment with Handhelds

Written by Tony Wheeler on Saturday, 17 March 2007

For the past 3 years I've been working on the e-scape (e-solutions for creative assessment in portfolio environments) project at TERU  (Technology Education Research Unit) Based at Goldsmiths University, the project is funded by DfES and QCA in association with Edexcel and AQA awarding bodies. Despite the fact that learning activities in d&t and other areas of the curriculum are increasingly influenced by digital technology, the final presentation of work for GCSE assessment is almost entirely paper-based, relying on print-outs of any digital work. Most teachers that I meet agree that the present assessment system at GCSE for d&t is seriously flawed and often rewards the wrong students, notably the ones that meticulously follow the rules rather than break them and come up with really exciting ideas. Project e-scape is a new approach to the assessment of creativity and design innovation at GCSE level where students use digital devices to "hoover-up" evidence of their on-going work during carefully structured design tasks. Their photos, sketches, notes and audio comments are instantly uploaded into a multimedia web-based portfolio that teachers and examiners can then use to assess individual performance. An extension of the project has been to trial a new approach to assessment based on holistic paired judgments instead of traditional marking. Background The e-scape project originated from ‘Innovating Assessment’, another QCA funded TERU project, which began work in 2003. A key focus of this earlier research was the development of a range of techniques to both promote and collect visually rich evidence trails of students’ innovative performance as they worked on practical design tasks. These techniques included: • handling collections (to get pupils contextualised and thinking about the task) • working in teams of three (to get pupils started and help critique their developing ideas) • modelling kits (to resource 3D thinking/rapid ideas modelling) • carefully structured tasks and sub tasks (to keep pupils iterating between thinking and doing) • photo storylines (to record evidence of each stage of modelling) Innovating Assessment resulted in a which took place over six hours on two consecutive mornings. The tightly structured format was designed to stimulate pupil creativity and group work in studio-based GCSE d&t activities. This system is currently being piloted by OCR as part of their new Product design GCSE innovation challenge. A report of this project can be downloaded here. Digital photos of each student's 3D models were taken every hour, as their designing progressed. These were immediately printed on a mini printer provided an of each students' developmental work. This had significant effects on both performance and behaviour, motivating students to work more purposefully and to take more risks. It also helped the assessors to see a more complete picture of each project. This lead the e-scape team to consider how the collection and recording of other aspects of performance, such as discussion in the form of short audio clips, might help to present an even more complete picture of d&t capability. e-scape Phase 1 (Proof of concept) The DfES and QCA were also keen to explore multimedia evidence trails and agreed to support the development of a digital version of the TERU paper based system. In January 2005 we started work on a proof of concept. We were determined to avoid the twin traps of forcing pupils out of workshops and studios into second-hand, virtual experiences in computer suits, or overburdening teachers with data capture and scanning tasks. As technology changes so rapidly we also wanted to make sure we had a broad perspective of what was possible and explored a wide range of web based portfolios and small scale, handheld systems that could be linked to allow students to collect their own evidence. We trialled a series of single media systems including: • digital cameras (still and video) • digital memo/voice recorders • tablets • digital pens. Of these the digital pens seemed to offer the most effective route to capturing students' notes and sketches. However, when we interviewed the students they were not so impressed, complaining that the pen was "too chunky", "only wrote in biro", "kept wobbling". This was clearly not the response we had been looking for, supporting both the activity and the assessment of it. The students went further and suggested the digital pen would be better if it had a camera and microphone built in. They were essentially describing a PDA and when we subsequently trialled with Palm Zire 72 devices we began to see significant motivational and inclusive benefits during, as well as after the activities. To familiarise ourselves with the development process for handheld applications we evaluated a number of existing authoring packages and publishers offering programming services. Graham Brown Martin at Handheld Learning impressed us with his passion and commitment and he worked with us to create a virtual version of the handling collection. With Graham’s help we modified the paper-based activity to incorporate a range of discrete media recording activities using the PDA including sketching, beaming, audio recording and photography.  You can download a report on the first phase of e-scape here. e-scape Phase 2 (Working prototype) We generated sufficient evidence from the proof of concept trials to move on to develop the e-scape system into a working prototype. We did this in association with Handheld Learning, who created the PDA application, and TAG Learning, who developed the web-based portfolio system. Together the two teams devised a system that managed the dynamic dataflow from a web server to and from the PDAs and the portfolio. During the sessions, activity instructions were delivered to the PDAs at timed intervals, students' work was beamed around their PDAs to facilitate group support and review. Throughout, all the media created and collected by the students was uploaded to individual secure web portfolios as the activity progressed. In this way students recorded the progress of their designing on their own PDA using photos, sketches, notes and audio comments. This provided valuable procedural scaffolding for their designing as well as sufficient evidence of their performance for us subsequently to make confident assessments of their work despite not being present while they completed it. (3D e-scape video - Quicktime VR) We overcame the problem of inconsistent/unreliable wi-fi network availability in some schools (particularly in d&t departments which are often situated on the far reaches of the school site) by creating our own local wi-fi network using a portable computer and hub. By May we had a sufficiently robust system and completed a national trial in 14 schools across the country through June and July 2006. As a result we now have 250 e-portfolios of year 10 learner's performance in the e-scape website, together with a class set of year five work (who had no problems with the technology and produced some really exciting work). The combination of the activity structure, supported by the handheld technology had a profound effect on all the students we worked with. Just like designing itself, it is difficult to capture the e-scape experience in words: fortunately Teachers TV came and videoed one of the early trial sessions, together with a discussion panel and a broader look at e-assessment. You can view each of these 3 programmes at their web site: • The future's handheld • Discussion • e-assessment Judgemental Pairs   We had originally planned to use the same holistic marking system to assess the e-scape web portfolios as we had for the previous paper based project. However in the Spring of 2006 we were introduced to Alastair Pollitt (former head of research at UCLES) who proposed an intriguing alternative: comparative pairs. Alastair explained how Louis Thurston had developed this theory of assessment in the 1920s, based on simply comparing one piece of work directly with another. Alastair argued that abstract assessment criteria did not help in the process of marking, as examiners inevitably convert the abstract into concrete exemplars, increasing variability and unreliability. So why not just compare work directly? If enough comparisons between two different pieces of work are made by enough judges, a very reliable rank order emerges (the one that always wins movs to the top, the one that always looses goes to the bottom and the others spread appropriately between). I understand that QCA use this system already to monitor inter-board comparability, basically to ensure an 'A' in maths from OCR is the same as an 'A' in maths from Edexcel. The problem lies in the scale of the award. With twenty paper scripts and half a dozen judges it can be done round a table, but when there are thousands of scripts and dozens of judges it becomes a logistical impossibility. However, with the advent of web-based portfolios, like the e-scape set of portfolios, are available anywhere and anytime each assessor has an internet connection. Multiple copies can be viewed at anytime, making the paired process possible in a high-stake assessment for the first time. The team were very sceptical when comparative pairs was first explained to us. To convince ourselves we completed a parallel test using the pairs system to rank a set of paper scripts we had previously marked. The resulting paired rank order was directly comparable to the rank order of the traditionally marked one, which reassured us sufficiently to apply the new process to the e-scape sample. Alastair created a matrix of paired judgements covering the 250 year ten e-portfolios (up to thirty comparisons for each script) for the e-scape judging team of seven researchers and teachers. We completed these comparisons online in three stages and once we got used to the process and system it proved to be really effective and straightforward. We also recorded the time taken by each judge, which proved to be less than the more conventional methods. In the first round the judging was fairly random as there was no order in the scripts at all and some of the judgements between really good work and really poor were very easy. In the second round Alastiar focussed the pairs on closer judgements, while this was more demanding it was still quite manageable. In the final round we were presented with five or six portfolios to rank which helped to fine tune the rank order even further. There are a number of really exciting possibilities that emerge from this approach to assessment. It can be much more democratic and inclusive than the present examination system which has pretty much separated and alienated teachers from the process (I remember the CSE mode 3 with much affection!). A paired system could be created to use an elite group of examiners and moderators such as we have at the moment, but it would be far more effective to include all teachers in the process. And there is no reason to stop at teachers, what about including professionals or even the students themselves? There is more information about the e-scape approach to assessment in the Phase 2 report which can be downloaded here. e-scape Phase 3 (National scalability) TERU are currently negotiating with the DfES, Becta and QCA to secure funding for the final phase of the project. This will extend the activities beyond d&t to investigate the extent to which this approach to structured, monitored, coursework assessment can be applied to other subjects. Alongside this work we plan to develop an authoring tool to enable teachers to build and modify their own coursework activities for themselves and to scale the system up to understand the issues awarding bodies may face if they were to implement it nationally at GCSE. Research team Richard Kimbell Tony Wheeler Chloe Nast Alastair Pollitt Kay Stables Ruth Wright Tristram Shepard Soo Miller Gillian Whitehouse Jo Hayes Project partners DfES QCA Edexcel AQA OCR TAG Learning Handheld learning Project schools Saltash Community School, Dave Hayles Cambourne Science and Community College, Donna Bryant Uplands Community College, Julie Nicholls Redruth, A Technology College, Mick Laramy Meole Brace School, Stephen Cox Edensor Technology College, Nick Bradbury Bedlington Community High School, Fiona Mather Dixons City Academy, Maria Eccles Hirst High Technology College, Bob Miller Coquet High School (Technology College), Steve Thompson Alexandra Park School, Ross McGill Duchess’s High School, Craig Watson Ashfield School, Jo Hayes Nancledra County Primary School, Pauline Hannigan Leasowes Community College, Invicta Grammar School, Alex French Bulmershe School, Liz Cook About the Author:               Tony Wheeler describes himself as a designer, publisher and researcher. He is currently working as a senior research fellow for TERU (the Technology Education Research Unit) at Goldsmiths college, exploring digital approaches to assessing creativity using handheld technologies. Tony has taught, examined and lectured at most stages of education. A co-founder of TAG Learning, he has designed and published a wide range of software titles and teacher guides, including materials for the BBC, Microsoft, Apple, the V&A and English Heritage, and has established four national web communities. Tony also works with teachers to help them get the best out of mobile technology as part of the Handheld Learning training days. -------- If you have enjoyed or found this article useful you may wish to share your knowledge or experiences with the rest of the international Handheld Learning Community by submitting an article. This site supports many other sites via its RSS feed as well as Google News. If you'd like to be considered for publication on this site please submit via articles (at) handheldlearning.co.uk Comments from the forum: Creative Assessment with Handhelds Simone    March 28th, 2007 - 12:38 AM This is a really interesting project. It's great to learn how you've used handhelds to do something that would be virtually impossible otherwise. Thanks for the information! (1) Leave a Comment

Last Updated ( Saturday, 17 March 2007 )

< Previous		Next >