How do we measure the impact of technology used to support learning?
In the first of three posts I reflect on technology’s role in education and how we can measure the impact as we utilise it in our classrooms.
Learning is a complex system, particularly in schools where we try to turbo charge it from 9am to 3pm. The measure of that learning can often be less complex, such as a letter or number to quantify the learning or benchmark the progress since the last check.
Technology has always been part of learning…
Chalk, slate, paper, pens, books and calculators… but we are specifically referring to new technology that has become so ingrained in our lives, therefore schools are grappling with whether and how to embed this technology in learning. If a school gives every child an iPad, will this improve learning? Of course it won’t, any tool is simply a small part of the system but we have become used to measuring impact on learning so much in schools we need to be able to measure the impact of technology on learning but it is crude and dis-ingenuous to measure the impact simply in results. However, schools can spend heavily on technology when budgets are shrinking and the investment needs to be justified.
Measuring Impact in Schools
The measures we are used to are
- Lesson Observations
- Progress indicators
If I were to visit your school and say
- “Implementing technology to support learning will improve results.”
- “Lessons with technology are better than those without.”
- “Learning with technology will improve student progress”
I sincerely hope you show me the door as I almost definitely don’t have evidence to back the statements up. Furthermore, if those silver bullets existed you would already be using them.
However, I do strongly believe that there are some learning experiences that technology can enhance, improve and provide that are not possible without it.
There are times we have reflected on lessons and note that mini-whiteboards, post-it notes, laminated cards or highlighters could have made the lesson more successful. This is not because it saves paper, looks pretty but because it improves progress, understanding, explanations, discovery, feedback and so on. Hence, I aspire that technology sits less as an entity on its own and becomes ingrained in learning and teaching. There are fantastic tools freely available to teachers that can enhance feedback (The Education Endowment Foundation and Hattie have research to suggest this has a significantly positive impact on learning. For balance you may wish to also read @LearningSpy’s blog post summarising some objections to the use of effect size and this post questioning the statistics in Hattie’s work) to students or facilitate more effective group work or peer review.
If I were to visit your school and say
- “Implementing technology saves teachers time.”
- “Lessons with technology allow for more personalised learning.”
- “Learning with technology will improve student engagement”
I hypothesise that I would be much less likely to be shown the door. Technology can be a distraction, irrelevant or a positive impact on learning and I suggest the measures we use to assess the impact of technology need to be discussed and a clear set of agreed measures brought in to general use. The team at Google for Education tried this with a vote on twitter:
Technology used to enhance personalised learning requires some cultural change in schools as it challenges the teaching styles of many successful teachers, the learning preferences of good children and the digital literacy of both groups. Alongside any implementation of a change to embed technology in learning is a need for training, discussion and a clear vision for learning. The training deficit that I have become aware of as I try to embed technology in learning was highlighted more publicly in late 2015.
On 15 September 2015, The Organisation for Economic Co-operation and Development (OECD) published a report called “Students, Computers and Learning” on technology in the classroom and snippets from it formed rather sensationalist headlines, including this one on the BBC website.
Computers ‘do not improve’ pupil results, says OECD
The report looks at computer usage across a number of countries and compares outcomes. Using OECD’s Programme for International Assessment (PISA) results from 2012 the report identifies “no appreciable improvements in student achievement in reading, mathematics or science in the countries that had invested heavily in ICT for education”.
Do we ‘invest’ in ICT for Education?
Is investment in ICT for education or technology for learning simply monetary? My school has an investors in people plaque in reception and I am pretty sure that doesn’t mean they pay more than other schools. However, the reference to investment in ICT in the OECD report is a quantitative measure. Despite the media headlines the report is detailed and aware of the data it has and has not used when summarising the findings.
Technology can amplify great teaching, but great technology cannot replace poor teaching.”
OECD: Students, Computers and Learning: Making the Connections, 2015
Schools will have to choose the level and focus of investment in technology for learning. Whether that takes the form of devices, training, open discussion, design thinking projects, bring your own device (BYOD) policies or even new learning spaces is a big and difficult decision for school leaders to make. Especially as the impact of these measures are unclear, hence this blog post.
The discussion around technology in schools is often confused, school leader. There is a distinct difference between school provided technology to support learning and a personal device used by a student. This distinction can be overlooked in the discussion that surfaces via bloggers, newspapers and those with vested interests. An Ofsted spokesperson was quoted in the Times Educational Supplement (TES):
“Pupils bringing personal devices such as laptops or tablets into school can be extremely disruptive and make it difficult for teachers to teach,” an Ofsted spokesperson told TES.
This quote is then translated into this headline:
Ofsted warns against ‘extremely disruptive’ tablets in school
Richard Vaughan – TES December 2015
Here is a response to the TES article, (Ensure you also read a follow up post including Ofsted’s response as the reference to disruptive technology was specifically to do with technology in schools not for the use in learning) including a search through Ofsted reports for references to technology’s impact on learning.
The decision six months ago to equip all students with tablet computers has not been universally welcomed by parents and carers, but the positive impact on students’ learning is obvious. The computers help students to work independently, they give all students equal access to online resources and they provide an excellent communication tool between teachers and students.
Ofsted Report June 2012 – Secondary (School overall judged as Outstanding)
The impact of technology on learning to this inspection was ‘obvious’ so we know it when we see it. The impact measures suggested by this inspector are
- Equal access
- Effective communication
Did this school implement tablet computers because they identified weaknesses in these areas? When I work with schools advising them on their use of technology to support learning the first document I request is the school development plan as there is no point developing technology or the use of it in anything that does not support the key focuses of that school. Therefore the measures have to be the same measures schools put in place every day, week term and year.
The OECD report summarises the implications of the findings and suggests that schools are not yet able or ready to embed technology in learning and leverage the potential. I agree, that we are yet to develop a clarity on how we want technology to blend into pedagogy and schools. I feel this provides more support for a drive to agree tangible measures for the impact of technology. What is the problem we are solving? The implications section concluding the OECD report implies the following potential benefits of technology in education:
- Digital Literacy
- Teacher and Student Collaboration
I suspect a search through school development plans will find many that want equality, more officially known and closing the gap. A number of approaches can support this and technology, planned and implemented well, can help. However, a poorly executed implementation of technology can bring into sharp focus the inequality between students. We often hear educators present the now familiar concept that “we are preparing our students for jobs that don’t exist yet” (if I had a penny…) and therefore the use of technology in schools is necessary to prepare them for the world they are going into. However, I didn’t have any such preparation and I think I am doing fine. Technology is already allowing teachers to share and collaborate and we have dumped millions of mediocre to poor resources onto our blogs, social media and resource sites.
What would your measures of impact be?
In part 2 of this 3 parts blog series I want to focus on the tool versus the teacher.