Eight Reasons Why EdTech Doesn’t Scale How sandboxes are designed to counter the issue

The EdTech Hub’s sandbox approach offers implementers and decision makers a clear way to implement and scale only the most effective EdTech interventions.
In this position paper we share our view on why EdTech isn’t scaling and how we see the sandboxes approach addressing them.
Keywords: innovation; implementation; sandboxes; EdTech; scale; experimentation
An output of the EdTech Hub, https://edtechhub.org


Scene-setting: sandboxes as one of the seven approaches
EdTech Hub is a global non-profit research partnership. Our goal is to empower people by giving them the evidence they need to make decisions about technology in education. Our approach combines undertaking rigorous research studies with in-country partnerships and hands-on support, to ensure robust evidence is generated about the use of technology in education. This is underpinned by communication and dissemination of what we are learning to maximise our impact by igniting change and action in others.
Making a positive impact in the EdTech sector requires a multi-pronged approach. This paper focuses on EdTech Hub Sandboxes; how they have been imagined as an approach for scaling promising EdTech interventions, the rationale for their structure as an action research methodology, and some emerging opportunities to apply it across the Hub's work.
For the purposes of clarity, we will discuss sandboxes in terms that make them feel discrete, but of course, other tools and approaches of the Hub complement and combine with the sandboxes and that's where we know the most profound change is possible. Imagine a ministry of education, working with our engagement team, going on to undertake a large-scale research project, of which sandboxes are an important component.
But let's start at the start, with the most important question of all: Why isn't EdTech scaling already?

Why is EdTech not scaling?
The big picture reason why EdTech isn't scaling is deceptively simple: the sector lacks the incentives to encourage getting to cost-effective solutions. We see a number of contributing factors.

There isn't enough robust evidence
It's been widely documented that evidence in this sector isn't robust enough "the EdTech sector often relies on anecdote and aspiration rather than robust research evidence and even high-profile programmes might not be rigorously designed or evaluated" (⇡Hennessy et al., 2021: p.13).
Significant gaps in the literature particularly hinder the sector's ability to evidence the cost-effectiveness of the use of technology over non-tech alternatives.
Across the sector, randomised control trials (RCTs) are conducted in an effort to prove an intervention's impact on desired outcomes in a given context. However, the cost of RCTs is high and they are time-consuming. This makes it difficult for researchers and implementers from poorer countries to invest in these approaches .
More research needs to be done that is specific and relevant to low-and middle-income countries. Research needs to focus on how and why interventions work, not just whether they achieved the desired outcomes. There are other rigorous and more cost-effective primary research methodologies that can be added to the mix.

There is a gulf between creators and users of knowledge
In one year of the Covid-19 pandemic, the health sector was able to mobilise a vaccine, while educators are still grappling with remote learning (⇡DEFI Cambridge, 2021). In entertainment, we've transitioned from one to many broadcasts to one-to-one, highly personalised, on-demand entertainment with platforms like Spotify and Netflix. Finance has turned every phone into a bank terminal, much of which was pioneered by mPesa in Africa.
This comparison with other sectors is familiar and there are, of course, aspects that make education more difficult to progress as a sector, for example, it's not as easy to count learning outcomes as it is to count vaccines, but there are practices, processes, and ingenuity that can sensibly inspire us.
Sectors that have led the charge in digital transformation tend to narrow the gap between theory and practice. In the EdTech sector, there is a gulf between academia and practitioners, resulting in the limited use of evidence for decision-making. Evidence feels hard to access for those grappling with challenges on the ground.

Education is under-financed, meanwhile, private sector EdTech is booming
The lack of adequate financing for education was already a concern before the pandemic and is deepening with the global recession. The precise impacts of the crisis on government budgets aren't yet known but early estimates suggest money spent by education departments will be further squeezed by shifting priorities to health and economic recovery (⇡Read, Lindsay, 2020). Understandably, when there is little available, teacher salaries come first and new interventions come last.
In stark contrast, private sector EdTech is growing rapidly. In China, we're seeing an EdTech boom. uLesson, an EdTech startup based in Nigeria that 1 sells digital curriculums to students through SD cards, has raised USD 7.5 million in Series A funding.
In India, Educate Girls' Development Impact Bonds (DIBs) have surpassed 2 both enrollment and learning goals. The work done by the Bill & Melinda Gates Foundation (BMGF) to help establish the vaccine alliance known as Gavi some 3 20 years ago, is an example of how the right vehicle can catalyse fragmented markets.

EdTech doesn't reach the most marginalised
The private sector and most marginalised learners can feel like oil and water. Private sector Edtech interventions tend to rely on customer-facing revenue models, which by definition exclude the poorest.
The private sector can't be expected to target groups that make no business sense, nor can we expect 'trickle-down benefits' to happen automatically.
Meanwhile, organisations focused on the most marginalised tend to be charities and non-governmental organisations (NGOs), relying heavily on donor funding and often lacking clarity on how the intervention will be sustained (⇡Plaut et al., 2020).

There is a tendency to think of the product rather than the problem
The EdTech sector remains largely focused on products rather than problems: at times, we notice EdTech partners strictly talking about virtual learning environments or tablets rather than literacy outcomes, equity in learning, and other problems these might be used to address.
The result is that we end up asking the wrong question: 'how might we add tech to education?'. Instead, we should be asking: 'how might we educate in a tech-enabled world?'.
In her book, Lean Impact, Ann Mei Chang outlines the importance of falling in love with the problem, not the product: "Whether due to excitement, attachment, or the requirements imposed by a funder, we can become wedded to our intervention, technology, or institution. To make the biggest impact, fall in love with the problem, not your solution" (⇡Chang & Ries, 2018: p. 39).

EdTech interventions are treated like a 'silver bullet'
A huge proportion of EdTech products make the same set of basic mistakes; they aren't user-tested, they don't align with national curriculums, they don't take into account good pedagogy and aren't designed with scale in mind (⇡Plaut et al., 2020). As a result, the pipeline is weak.
To unlock impact, the entire education system needs to be taken into consideration. This includes understanding the needs of people using the intervention, distribution, business models, human infrastructure, pedagogical practices, and more.
There are no clearly defined or used standards for EdTech products as seen in other sectors, for example in health where target product profiles are commonplace.
EdTech is often thought of in terms of one-off deployment, when it should be thought of as a service needing continuous delivery, with consideration given to the end-to-end experience of those using it .
Emergent thinking in education points to the importance of a coherent package of services (i.e., teaching and learning materials, teacher training, teacher guides, and ongoing support) known as 'structured pedagogy' in 4 leading to improved learning outcomes (⇡RTI International, 2021).

There's a gap in programmes that support transition to scale
EdTech innovation programmes are usually accelerator programmes, which are designed to put product teams through their paces and get them ready for investment more quickly. While going through these processes will give product teams the spotlight to catch the eye of those looking for innovation, more is needed on the other side of graduation to catalyse scale.

Government is often a time-pressed, uninformed customer of technology within a fragmented market
The education sector, in general, is infamous for being highly fragmented, with actors such as the government, private sector, and NGOs not collaborating and causing a lack of understanding of 'the other side' and duplication of efforts. Further, the government teams the Hub works with do not always have in-house technical capability and need support to work effectively with the private sector.
Based on a preliminary review of Ministry of Education plans and policies across four countries (Ghana, Kenya, Sierra Leone, and Tanzania) as well as Ministry of Education engagement through the Hub, we've seen that governments often don't factor technology into their broader education goals, strategy, and policy planning.

How innovation sandboxes help EdTech reach scale
With these problems in mind, we need to consider the ideal methodology for EdTech Hub to support EdTech implementers as they work to push effectiveness up, drive costs down and demonstrate to the whole sector that scale is possible. What can we do to create incentives for making that happen?
The problems outlined point to the need for an approach that brings actors together, connecting theory and practice, private and public sector, various funders and innovators with governments, and entrepreneurs with academics and the people impacted by the work.
As a donor-funded programme, we believe we have a unique convening power to bring groups together and incubate change (not products) in the sector. This 'working together' needs to be done in ways that won't look or feel familiar -no step change can be created by business as usual. We need to think big, take risks, and make space for applied creativity.
Sandbox methodologies support implementers to test and iterate interventions throughout implementation. We define boundaries in real-world environments that will offer us a mini version of the whole education system in a country. This allows us to test specific components of a model and use what is learnt to define the next step while limiting implications and damage to the broader system.
EdTech Hub Sandboxes fast-track the scale-up of promising EdTech by testing approaches to bring costs down and drive impact up -providing the tools and access to experts and funding to make that happen.

Figure 1. Explainer: sandboxes
Where does the term 'sandbox' come from?
Sandboxes have been used in the world of technology for decades and more recently in regulation. If you're reading this online, you're likely in a sandbox environment on your browser. Some examples of sandboxes in other sectors include:

POLICY | The UK Financial Conduct Authority's regulatory sandbox:
The United Kingdom's Financial Conduct Authority (FCA), launched the first fintech regulatory sandbox in June 2016. This sandbox allows businesses to test innovative products and services in a safe, live environment, with the appropriate consumer safeguards, and, when appropriate, is exempt from some regulatory environments. After its first year of operation, 90% of firms that completed testing in its first cohort were continuing towards a wider market launch.
SOFTWARE | Your web browser essentially runs web pages you visit in a sandbox. They're restricted to running in your browser and accessing a limited set of resources -they can't view your webcam without permission or read your computer's local files. If the websites you visit weren't sandboxed and isolated from the rest of your system, visiting a malicious website would be as bad as installing a virus (⇡Hoffman, 2013). In software 7 development, a Sandbox is a security mechanism for separating running programs, usually in an effort to mitigate system failures and / or software vulnerabilities from spreading.

TECHNOLOGY | The Climate Sandbox is an initiative being co-created between the UK's Foreign, Commonwealth and Development Office (FCDO)
and The Frontier Technologies Hub (an award-winning programme 8 9 delivered by three partner organisations: Results for Development (R4D), Brink, and IMC Worldwide). It will test an approach to combine various levels of climate information in the Jharkhand region of India to join up and test its ability to support better decision-making by the Government of India.
We see an opportunity to use the sandbox approach in EdTech, conducting real-world tests, passing an EdTech product or process through the system to see what works and what breaks, with all parties on board to observe and then redesign the system accordingly. For example, we might test a new intervention in a small number of schools first, improving its cost per user through experimental sprints, increasing its impact by supporting the organisation with pedagogical expertise or even trialling new ways of procurement, at a small scale, before rolling it out. The idea is that we increase our scale in line with confidence that it works.
Listed below are our sandbox principles and how we use them.

Problem-centred, with a clear focus
Sandboxes work alongside the Hub's other in-country offerings to change the way decisions are made about technology in education. Our aim is for decisions to be made based on rigorous academic evidence and real-time evidence directly informing implementation.
We are focusing our efforts on contributing to five key topics in education: 1. Technology to support personalised learning and teaching at the right level 2. In-service teacher professional development and structured pedagogy 3. Technology to advance data use and decision-making in education 4. Technology to promote participation in school

Girls' education and technology
For all the topics, we'll be looking to understand how to improve learning outcomes. The first step in our sandbox process would be to outline a clear problem statement for our work, for example, for (1), we might be decreasing the cost of personalised learning tools by X% so that it is accessible to marginalised populations.
Crucially, being problem-centred means that we are not committing to scaling the solution or organisation unless it is addressing the problem and maximising learning outcomes.

Systematic experimentation, through real-world implementation
Use an iterative, experimental approach in real time, applying the level of rigour required (no more, no less) to allow decision-makers to test and iterate their model in low-cost ways and scale only that which has evidence of cost-effective impact .
Through quick 'build, measure, learn' cycles, we test our assumptions on how we might configure a solution, learning what works (or doesn't), adapting and scaling as we go. User-centred approaches and lean methodology enable us to test in the real world, with real people, increasing the certainty that the intervention will work. Early experiments also help inform the design of future interventions. As noted by Michael Kremer in his Nobel Prize lecture (⇡Nobel Prize, 2019).
"Experiments are more akin to beta tests, useful in developing new policies, not just evaluating existing ones."

-Michael Kremer
In involving programme implementers and innovators in experimentation, sandboxes demonstrate the important role these stakeholders can play in the generation of evidence. These experiments can contribute to at-scale research and RCTs allowing implementers themselves to contribute to building a more robust evidence base.
Over time, as certainty about the implementing model increases, sandboxes 'stair step' to higher rigour (and more resource-intensive) data collection and analysis. This is echoed by Jean Arkedis in her explanation of R4D's evaluation and adaptive learning practice, which also follows this principle (⇡Arkedis, Jean, 2019).
"Could we do a better job improving programs, saving expensive impact evaluation evaluations for programs that are as well-designed -through these same feedback and iteration loops -as possible?"

-Jean Arkedis
This graph from Lean Startup outlines the process from end to end (⇡Ries, 2011).

Narrow the distance between theory and practice
Make sure that where there is existing evidence, it's being put to use in decision-making by governments, entrepreneurs, and funders. This is done by surfacing and sharing what we know and by bringing world-leading researchers into the sandboxes to work as part of the team.
Our review of the evidence base will always consider strong pedagogical practices and the science of learning to inform an intervention. Evidence on cost-effective ways to use technology in education -for instance, recent work identifying 'Smart Buys' for improving global learning (⇡World Bank, 2020) will also be front and centre. We will also borrow from other sectors, such as behavioural economics, digital principles, and service design. We use all of this information to develop 'design principles' that can be tested over the course of the sandbox and shared with others tackling similar challenges across the sector.
In time, the practical implementation evidence we generate will contribute to the body of knowledge too.

Treat the system as the unit of experimentation
Consider EdTech as a service, not a product. This forces us to acknowledge the elements of the service beyond the technology or the software that make an intervention possible and impactful. Services need to consider the end-to-end experience of their customers, consider the full breadth of the system they are in, and require people like teachers and technicians to deliver it.
We've developed a theoretical framework, the education system '6Ps', that supports us to keep the full gamut in view.

Figure 3. The 6Ps of an education system
Experimentation in the sandbox happens across any part of this education system -people, product, pedagogy, policy, place, and provision. We might use the sandbox to refine implementation models, test supply chains, or simulate new education policies. It could focus on finding instruments, technologies (plural and in combination), adapting pedagogy, and involving the people or 'human infrastructure' that are crucial for making sure we don't scale tech without scaling positive impact.
Business models, like cross-subsidies (where profits of a business or activity are used to subsidise another business or activity) (⇡Chang & Ries, 2018) could be explored to determine how best to extend the reach of EdTech to the more marginalised learners.

Give a mix of actors a seat at the table and hold space for their collaboration
Provide a dedicated team tailored to the needs of each sandbox, drawing on expertise in innovation, technology, education, pedagogy, and more.
Leverage convening power to bring a mix of stakeholders together from across the education system; including implementers, government, and those most impacted by the work. Support governments to work effectively with the private sector and local businesses that can address their challenges. Co-create and support the discovery, contracting and implementation of technologies alongside government stakeholders.
Consider the 'give-get' of each participant: ■ Funders brought together to be part of a Funding Circle, get deal flow and give entrepreneurs money-can't-buy strategic advice.
■ Entrepreneurs get routes into trialling their products and support from world-leading experts to do this well, get to be in the spotlight with major funders, and have evidence of impact to secure their future.
■ Governments get connected to entrepreneurs and new tech products, safe in the knowledge they'll be assessed according to the very best standards and implemented in an iterative way, scaling up in line with success.
■ Policymakers have a 'container' for trying out new ideas and their impact. Donors, private investors, and other funders brought together at the start of a sandbox to give advice, offering de-risked deal flow to invest in a booming EdTech market.

Partner and build on what's already out there
Start by asking the question 'What's working, and how can we do more of it?'. Surface what's out there, build on the existing assets in a community, plug into the local ecosystem and enable more of it.
Prioritise support to take innovation to scale, given that there are many programmes already focused on generating new ideas. Partner with early-stage incubators and accelerators to build a 'pipeline of pipelines'. Use this pipeline to identify the best EdTech that has been designed for scale from the start, and give partners the support they need to maximise their impact.

Financial instruments
Invite impact-oriented funders to 'express interest in being part of the Funding Circle, supporting and advising product teams as we progress Throughout the sandbox, exploring opportunities for direct co-investment with financial support or other The sandbox methodology can be used in the early stages of the design of an intervention, all the way through to implementation and scaling.
After a year of sandboxes focusing on responding directly to the Covid-19 pandemic, we are returning to our original priority of providing implementation support to unlock scale. We believe this is where we can have the biggest impact on the global learning crisis. ⁅bibli