Project Quantum: tests worth teaching to

Project Quantum: tests worth teaching to

Created by Simon Peyton Jones
last edited May 08 2018 by Cynthia Selby

Project Quantum

Project Quantum will help computing teachers check their students’ understanding, and support their progress, by providing free access to an online assessment system. The assessments will be formative, automatically marked, of high quality, and will support teaching by guiding content, measuring progress, and identifying misconceptions.

Teachers will be able to direct pupils to specific quizzes and their pupils’ responses can be analysed to inform future teaching. Teachers can write questions themselves, and can create quizzes using their own questions or questions drawn from the question bank. A significant outcome is the crowd-sourced quality-checked question bank itself, and the subsequent anonymised analysis of the pupils’ responses to identify common misconceptions.

By way of background here are

• A white paper that describes the project in more detail
• A 20-minute video of Simon and Miles explaining what Quantum is all about.
• A 60-minute talk, plus questions and discussion about Quantum, delivered as a Cambridge Assessment Seminar in March 2017. Slide deck here; feel free to re-use/adapt.

All the questions and quizzes are available on the Diagnostic Questions site (free registration required).

Quick links

• Quantum Scheme of Work (Eedi, Diagnostic Questions) is a way for you to use quizzes in an automated way to support learning. Two quizzes on each topic, automatically delivered to the students, ensures that a topic has been understood and embedded.
• Quantum quizzes is Quantum’s home page on the Diagnostic Questions platform. There you’ll find a collection of 250+ quizzes ready to use, with more each week.
• Quantum statistics: live statistics on Quantum usage.
• Quantum Jobs - To Do List: some jobs you could help with, even if you don’t want to write original questions.
• Quantum user launch page: what you need to know to get started using the Quantum material on Diagnostic Questions; creating and publishing quizzes.
• Quantum contributor launch page: what you need to know to get started authoring new questions, for your own use and other CAS teachers; giving feedback on existing questions.
• The Quantum FAQ
• Guidance on writing multiple choice questions that work
• Guidance in authoring MCQs on the Diagnostic Questions platform
• Guidance on using the Diagnostic Questions platform for creating questions and quizzes
• Working taxonomy
• Computing misconceptions, a CAS resource gathering together misconceptions that our learners have.
• Asking questions to assess attainment in computing, guest blog post by Miles summarising Quantum.
• What makes a good Diagnostic Question is just one interesting post in the Eedi blog post series.
• Quantum: tests worth teaching to Article by Simon Peyton Jones and Iain Davis on Project Quantum, from the first edition of Hello World.

Some videos

• Questioning questions: keynote videos from a one-day conference in 2017 on using questions to drive learning. Daisy Christodoulou, Bill Lucas, Lucy Crehan, and others.
• Introduction to Quantum: a 20-minute video of Simon and Miles explaining what Quantum is all about.
• Simon Peyton Jones discusses Quantum (and the computing curriculum) at #spash16
• Miles Berry on Quantum (and assessing computing) at CAS London primary CPD day
• Cynthia Selby, Craig Barton and Simon Woodhead on writing questions and using the DQ platform for #caschat live.

Key features of Quantum

• Quantum should save you work, by allowing you to use a bank of high-quality, machine-marked assessment items for computing.
• The focus is on low-stakes formative assessment (assessment for learning), not high-stakes summative assessment.
• We aim to cover the whole computing curriculum (not just computer science), and to cover both primary and secondary.
• The questions are crowd-sourced. You can write them yourself, and use the ones you have written. You can also use questions written by others (there will be a lot of them), and any questions you write will be available to others.
• It’s all free. Using the platform to author and administer tests is free. The questions themselves will be made available, in machine readable form, to other platforms.
• We are hitting the ground running. Our technology partner Diagnostic Questions already has a crowd-sourced multiple-choice-questions assessment platform, with some rather cool features. We are busy seeding it with questions donated from various helpful partners (TLM, OCR, Bebras, etc). You can start using it today for your own tests and questions.

Quality is key. How do we know that the questions in the system are any good? The exciting thing about Quantum is that it’s a collaboration between leading experts in assessment (Cambridge Assessment, Durham Centre for Evaluation and Monitoring) and subject experts (CAS, Naace), plus our technology partner Diagnostic Questions. CEM has lots of ideas for how to use data from thousands of students taking thousands of questions to identify which questions work well and which do not. There are serious research questions here – no one has done this before – but no one is better placed to do this than CEM and CA.

The CAS team

On the CAS end, our content advisory group consists of

• Miles Berry
• Cynthia Selby
• Richard Forster
• Chris Roffey
• Tim Scratcherd
• John Woollard

Quantum is generously funded for two years, by no-strings-attached grants from ARM, Google, and Microsoft. Without them it could not have happened. Thank you!

Status reports

• Quantum status report October 2017
• Quantum status report April 2017
• Quantum status report October 2016

Other relevant assessment projects

• CAS Progression Pathways
• A framework for assessing computing in the national curriculum]
• Baseline testing run by TLM
• CAS assessment guidance
• Brennan and Resnick on studying and assessing computational thinking for AERA12
• Miles Berry on assessing programming for CAS Assessment Working Group
• Principled assessment of computational thinking
• How do I assess the development of CT?
• The Viva project; a European project, arising out of ITICSE’15, with similar aims, but much more ambitious scope (number of countries, kind of questions…). Here is their paper about Viva. Naace are partners; Andrew Csizmadia and Simon Marsden are very involved.
• Sowing the Seeds: A Landscape Study on Assessment in Secondary Computer Science Education, Computer Science Teachers Association (USA), 2015.
• Isaac Physics. A very good online platform; curated questions.
• Cambridge Underground Mathematics. Mainly A level.