Cost-Effective EdTech Paper 2 Good practice

An output of the EdTech Hub, https://edtechhub.org


Introduction
This report forms the second part of a set of three reports developed by EdTech Hub that consider the implementation of cost-effectiveness analysis for EdTech interventions in low-and middle-income countries.
EdTech Hub's core mandate is to provide evidence to inform decision-making within the EdTech sector. Cost-effectiveness is central to this as it ensures that financial resources are allocated appropriately to the most effective interventions that lead to sustained learning improvements. Improving sector-wide approaches to cost-effectiveness is therefore a core part of EdTech Hub's remit. This series of papers makes an initial contribution to the long-term work of the Hub on this topic -setting out current issues and proposed routes forward for EdTech Hub and the wider sector.
The first paper (⇡Mitchell & D'Rozario, 2022) offers an overview of the background of cost-effectiveness analysis and the cost-effectiveness analysis approaches of other stakeholders engaged in the education sector.
This second paper offers general recommendations on nine key principles and good practices for the EdTech sector. It will be of greater interest to those considering the issue at a higher level, or those interested in the context for the specific approach advocated in the third paper (⇡Mitchell & D'Rozario, Forthcoming).
The third paper builds on the theory of the first two papers to offer specific implementation-related recommendations and guidance that is initially aimed at research within EdTech Hub.
The papers do not need to be read in order, but to a reader who is unfamiliar with the topic, it may be advisable to read Paper 1 first..

Structure and logic
This report is organised into two main sections and the logic for this is outlined below: 1. There are many different issues at play when considering the specific application of cost-effectiveness analysis within EdTech. Section 2 of the report identifies nine principles to consider for the three overarching issues within cost-effectiveness in EdTech: comparability, replicability, and sustainability.
2. Cost-effectiveness analysis in EdTech needs to be understood and adopted by those designing, implementing, selling, funding, and researching EdTech. Section 3 focuses on the practical application of the principles for different stakeholders.

Points of clarification
Any contribution to debates regarding cost-effectiveness in education and EdTech is inevitably controversial. It is a complicated topic with potential pitfalls of measurement bias, but it is critical to explore approaches in order to make groups of interventions comparable for both effectiveness and cost-effectiveness. The following introductory points are intended to provide clarification and should be kept in mind when using the paper.
1. There are overlaps in the cost-effectiveness of education in general, but there are important factors that are specific to EdTech. Much of the technical work on cost-effectiveness in EdTech links to the wider education sector, but there are specific factors relevant to EdTech that make this different. The paper engages with many of them including the nature of specific incentives, the associated discourse of innovation and experimentation and scaling, the cross-sector nature of many interventions, the complexity of infrastructure and sunk costs. The point to note is that making progress in measuring cost-effectiveness in EdTech can be supported by engaging with how cost-effectiveness is being implemented in education as a whole. However, the particular opportunities and challenges relating to EdTech should be recognised.
2. Improving understanding of cost-effectiveness is a long-term and ongoing endeavour. This paper is one step in the process of engaging more deeply with cost-effectiveness in the sector and does not represent a final position of EdTech Hub. It is an initial contribution to a long-term effort and will lead to more evidence and hopefully greater agreement within the sector about how to measure cost-effectiveness. Likewise, there is no single 'solution' to solve the challenge of cost-effectiveness. Stakeholders engaged in EdTech are at a range of different starting points and have different levels of capacity to conduct rigorous cost-effectiveness analysis.
Every step taken towards promoting cost-effectiveness in EdTech helps build a culture of evidence and will lead gradually to more informed decision-making.
3. The focus here is on student learning outcomes, but cost-effectiveness work also needs to go beyond this. The majority of this paper focuses on cost-effectiveness in relation to student learning. This is not the only aspect of educational change that the Hub is engaged with, given the focus areas of data for decision-making and teachers, but it is the most important place to start for introducing a consistent approach to linking cost and outcomes.

Cost-effectiveness in EdTech should always also consider non-technology-related options.
Any engagement with the cost-effectiveness of EdTech should take place alongside consideration of non-technology or blended interventions, not just a comparison between different technology-based options. The objective is to maximise impact on learning outcomes and that means it may often be more cost-effective to focus on non-technology implementation options.

Cost-effectiveness should not be used to present overly-simplistic assessment of complex systems.
There is a risk that promoting cost-effectiveness in EdTech gets misunderstood as promoting a single 'answer' for decision-makers. Any calculations related to cost should never be presented as 'solving' a problem on their own for decision-makers but instead should be viewed as one source of data that helps solve complex problems. In particular, there is still a need to consider whether a different intervention or systemic reform would represent a more appropriate and cost-effective alternative. Similarly, nuanced engagement with cost-effectiveness has to be centred on the context for the implementation.
6. Understanding the context and its implications for cost-effectiveness is essential. If cost-effectiveness analysis does not focus on the implications of context then it will give the illusion of progress without the substance -although technology itself is showing broad applicability, the uneven rate of adoption makes it hard to quantify benefits which can be generalised. Focus on cost-effectiveness in EdTech should be adopted alongside other factors rather than at their expense. Increasing the importance of cost-effectiveness in EdTech does not mean decreasing the importance of context, equity, holistic outcomes, or any other principle of normal good education programmes.

Costs are often higher in more marginalised contexts.
Driving cost-effective EdTech should not lead to always promoting the lowest cost per child. It is regularly the case that implementing EdTech is cheaper where there is more pre-existing infrastructure. Pilots are often (problematically) conducted in 'easy-to-reach' areas -and it is important to remember that what is feasible in one context is not necessarily feasible in another, and cost per child varies significantly according to context. The cost figures per child should not be the only factor used when determining if an intervention is appropriate -they simply provide a data source that can contribute to ensuring more effective decision-making regarding implementing EdTech.

Engaging with cost-effectiveness is complex and contested in
EdTech for many reasons. Working explicitly on cost-effectiveness, and the associated requirements of data sharing regarding cost is sensitive for any stakeholder. All stakeholders may be reluctant to share all the necessary information for a wide range of different reasons. Sometimes the cost of implementation might be deliberately hidden or skewed by those promoting it.More broadly, the challenge is that many EdTech implementers are simply not accustomed to identifying costs appropriately. Historically, they have not been asked to do so, do not know how to do this consistently, and have not had guidance on how to capture the entire cost of an intervention and who to share it with.

Learning-Adjusted Years of Schooling (LAYS) is a flexible concept for measuring learning outcomes that can be further refined.
Although LAYS has been used to describe learning outcomes that relate to standardised testing and attendance, as a descriptive concept it can be broadened and more flexibly applied to provide a comparable metric. Some of the assumptions around attendance and testing that LAYS is based on should be considered, and further research is needed to robustly link participation and engagement with cognitive outcomes. This research does not engage directly with how that is done, however, the use of LAYS within the framework of this paper (as the metric for measuring the effectiveness of an intervention) is based on a flexible and descriptive approach, which allows LAYS to be refined over time.

Nine principles for cost-effectiveness in EdTech
The purpose of this paper is to draw from the general cost-effectiveness analysis frameworks detailed in the previous paper and identify and summarise the most significant issues at play when considering the specific application of cost-effectiveness analysis within EdTech. It does this by categorising nine principles to consider for comparability, replicability, and sustainability in EdTech cost-effectiveness. Each principle is introduced with an explanation of why it matters, especially in the context of EdTech, and how it is drawn from wider good practices across cost-effectiveness analysis approaches in development and education more broadly, and then applied specifically to EdTech.

Comparability -cost-effectiveness is about measuring and analysing data that is comparable
1. Define data carefully: In order to compare data, it must be collected and categorised in a consistent and rigorous manner that adheres to widely accepted standards.
2. Ensure intersections of equity and cost-effectiveness: marginalised learners must be included in the cost-effectiveness equation so that equitable access and use are accounted for. It may also cost more to reach marginalised learners, and it is important that this cost is articulated and budgeted for so that programmes are equitable.
3. Contextualise points of comparison: Data cannot simply be brought into the same format to be compared, it must be contextualised with appropriate weightings and coefficients that make it truly comparable.

Replicability -EdTech costs and outcomes must be expressed in replicable terms to be useful for decision-makers
4. Account for uncertainties robustly: Uncertainty contained within cost and learning data has a disproportionate impact on EdTech interventions, and decision-making should factor in appropriate contingencies.
5. Conduct analysis in the short and long term: Informed decision-making requires sensitivity to changes in costs and outcomes over time that can be captured by undertaking cost-effectiveness analysis at numerous points throughout programme implementation.
6. Represent data in relative terms: Cost and learning data must be presented in formats that are calibrated to the time when data was captured and is contingent on the dynamic contexts of EdTech implementation.
Sustainability -cost-effectiveness analysis must consider the broader impacts of EdTech, and ensure they can be sustained -financially, socially, and environmentally.

7.
Consider the full lifecycle: The impact of EdTech on local education systems and learning outcomes must be considered beyond the educational factors for which the technology is intended.
8. Incentivise transparent reporting: Some structures and relationships in EdTech hide true costs and exaggerate outcomes -transparency is needed to counteract misleading data.
9. Ensure intervention cost is proportional: High cost-effectiveness does not ensure affordability and financing to improve liquidity for up-front tech investments and does not address fundamental priorities in expenditure at national levels.
These nine principles are discussed below within the three categories of comparability, replicability, and sustainability. Each thematic category links good practice in the evaluative frameworks above with specific design principles that can enable cost-effectiveness analysis in EdTech to facilitate evidence-based decision-making across contexts.

Comparability
This section underlines the important steps that have been taken thus far in making cost capture and learning outcome measurement more comparable. The sector has moved towards greater methodological consistency in data collection for cost-effectiveness analysis. Learning outcomes and cost data have started to be measured with a more consistent set of tools that relate to international standards (such as the costing models for the Programme for International Student Assessment (PISA), the Trends in International Mathematics and Science Study(TIMSS), and the World Bank's strategic impact evaluation fund (SIEF) costing model) although this is not yet the norm (⇡Angrist et al., 2020). Additionally, within the sector, there is a more consistent and rigorous representation of this data. Greater attention is also being paid to accounting for variability in cost data as a result of macro-economic cross-contextual variations, such as inflation, currency exchange rates, and the cost of living. There has also been a greater shift toward representing learning in comparable terms, through the harmonised learning outcomes and policy linking frameworks (⇡Patrinos & Angrist, 2018; ⇡USAID, 2021) that enable cross-context comparisons of different learning assessments undertaken at different times. All of these are important steps in enhancing the robustness of comparable cost-effectiveness.
But despite this positive progress, there remain additional areas where further endeavour is needed to ensure meaningful comparisons in cost-effectiveness analysis across contexts are highlighted. Continued development of the contextualisation of data that is used in cost-effectiveness calculations, ensuring that learning outcome and cost measurements are adapted and relative to their localised values, needs and resources is essential. loser examination of the fundamental data on which cost-effectiveness comparisons are based, and of the assumptions that underpin comparisons between different data sets are also necessary.
For example, in recent years, the LAYS measure has gained significant traction as it accounts for the quality and quantity of education to express an adjusted value for learning in terms of equivalent school years (see ⇡Filmer et al., 2018;⇡World Bank et al., 2020). However, LAYS comparisons are often undermined by these considerations, as they require comparable learning data, which does not yet exist, lack a fully defined exchange between test scores and years of schooling, and need more comprehensive and comparable measures of learning to be more robust (⇡Crawfurd et al., 2019). Therefore, while enabling comparability across interventions is important, currently, inconsistency in LAYS data sources is a barrier to achieving this. It is essential that EdTech interventions have accurate cost data, and are linked to a comparable learning outcome metric, in order to be comparable across contexts (⇡Chuang et al., 2021). Although this comparability is essential, there remain a number of issues associated with the comparability of cost and learning data, detailed in the principles below.

Why this matters
It is important for EdTech research, in particular, to define data carefully, because the growth in technology has facilitated the use of additional methods of capturing cost and learning data on which cost-effectiveness comparisons are based. Additionally, it has added a significant challenge to the accurate capture of the total cost of using and maintaining technology within a programme. Cost and outcome data that is either produced by technology or related to technology is often inaccurate and fundamentally flawed by underlying assumptions (⇡Crawfurd et al., 2019). A more careful approach to defining data greatly improves the accuracy of the data, enabling appropriate and accurate cost-effectiveness calculations and comparisons.

Good practices from the wider sector
Interventions must define data carefully, by capturing the total cost of implementation (TCI) (including opportunity costs, which is the value given up by actors engaging in EdTech programme activities, relative to engaging in alternative activities. For example, the wages that could have been earned by caregivers, which they may have lost by attending a programme-related workshop), and capturing learning outcomes relative to other interventions and outcomes. From a cost perspective, this involves including the direct and indirect costs associated with an intervention both during and after its implementation, such as hardware maintenance and repair (⇡Chuang et al., 2021). The importance of the fact that TCI also includes hidden 'free' costs that may not be associated with individual interventions, such as the price of providing connectivity when implementing an intervention in areas of existing connectivity infrastructure, is demonstrated by the chronic underestimation of costs (⇡Escueta et al., 2017). This will often lead to hidden 'baseline' costs (already existing elements that are needed for an intervention to function that is being financed or provided externally to the intervention) being incorporated within the total cost of an intervention, which is a significant shift from current sector thinking (⇡SIEF, 2020). The current approach of totalling cost as an addition to business-as-usual scenarios means that the relative costs (and cost-effectiveness) of similar interventions are only comparable if they are implemented in the same contexts. Additionally, accurately capturing all cost data associated with an intervention and technology is critical, as in many cases it is underreported. This can artificially inflate cost-effectiveness estimates (⇡Evans & Popova, 2014), which has negative implications for policy, comparability, and replicability. It is therefore integral to cost-effectiveness estimates that these costs are accurately accounted for when reporting the TCI.
It is also important to define staff costs carefully within the TCI. When scaling an initiative, the staffing requirements (such as teaching, teacher training, and supervision roles) often shift in a way that is non-linear in comparison to the scaling of other costs (for example, staffing costs could be reduced by the scaled initiative becoming part of existing roles, or costs could be increased if additional personnel and oversight are required when the intervention is scaled). It is therefore important that anticipated changes in staffing costs are accounted for when reporting the TCI of an initiative.
Regarding learning outcome measurement, a careful definition of outcomes that are measured must be both localised to be meaningful within its context (such as helping students to progress through their national curriculum), and standardised (learning tasks being assessed are core competencies aligned with international standards) to be useful for comparison. A careful definition of learning outcome data also requires that the learning data captured is an accurate reflection of learning which has taken place (⇡Filmer et al., 2018). This is a subtle difference from many current approaches that capture learning that has not, or only partially, taken place.

Important factors specific to EdTech
From an EdTech perspective, there are a number of important considerations for implementation. When thoroughly accounting for the TCI, it is critical to ensure that all costs are captured, as EdTech components often incur substantial costs that are needed to sustain a programme after implementation, for example, maintenance and repair costs for devices (⇡Chuang et al., 2021). Additionally, capturing all hidden and baseline costs enables a more accurate comparison of the resources needed to implement interventions across contexts, where there is high variability of factors that play a significant role in the effectiveness and cost of EdTech interventions (for example pre-existing connectivity infrastructure), as it controls for differences in baseline infrastructure between contexts. There is also a tendency for technology providers to try to minimise their costs and not frame them within the TCI As such, there is a need for accurate and transparent reporting of costs (⇡World Bank et al., 2019) (this issue is discussed as a separate principle later). Further, device costs are often presented without reference to additional costs of ownership (whether for peripheral devices and software, or operating, storage, and recharging costs or licensing fees) and are not correlated to overall programme costs. Rigorously and accurately reporting the costs incurred by EdTech interventions can ensure that total cost, and the associated cost-effectiveness of interventions, are both comparable and replicable.

Why this matters
It is important for EdTech research to ensure intersections of equity and cost-effectiveness because it enables a detailed understanding of how the costs and benefits of EdTech can be most effectively leveraged to target the most marginalised groups in a cost-effective and appropriate manner.

Good practices from the wider sector
To make interventions more comparable across participant groups, they need to promote the intersections of equity and cost-effectiveness. One of the challenges with educational interventions is that their cost varies significantly depending on the targeted group, and it can often be more expensive to target the most marginalised groups (⇡Sabates et al., 2018;⇡Walls et al., 2021). At the same time, the most marginalised groups tend to experience more significant learning gains as a result of an intervention (⇡Sabates et al., 2018). Therefore, whether the total cost (and associated learning) of an intervention is deemed cost-effective is highly dependent on the group it is targeted towards. To achieve equity against these varied costs, the disproportionate impacts per expenditure in terms of learning outcomes of educational interventions on different marginalised groups need to be considered.
For example, in a health context, when calculating Quality-Adjusted Life Years (QALY), utility weights (that act as a coefficient to quantify different health states on a scale from 0-1) are incorporated in the calculation (⇡Whitehead & Ali, 2010) to account for the different states of health between patients (and therefore the 'quality' of life that is added through medical treatment). Something similar could be incorporated in cost-effectiveness calculations for educational initiatives (such as the weighted cost-effectiveness calculations deployed by ⇡Sabates et al. (2018)), where utility weights that have relevance to interventions are defined and used as a coefficient for adjusting learning scores. This would help account for the disproportionate impact that educational interventions have on different marginalised groups, and allow for comparability between different groups across contexts. However, to do this effectively, LAYS needs to be calculated based on a similar volume of data to QALY (in terms of documenting the relative impact that different types of marginalisation have on the learning improvements received with different interventions), and this is a significant long-term challenge which needs to first overcome other challenges for accurate data collection.

Important factors specific to EdTech
When considering reporting equity there are a number of important factors specific to EdTech. First, EdTech can offer unique advantages in reaching marginalised learners and helping to build more equitable education systems (⇡Zubairi et al., 2021). It is essential that EdTech implementers give detailed consideration to the unique offering of EdTech and how it can be most appropriately implemented in each context to best support learning improvements among marginalised groups. It is important to consider that in most cases, reaching marginalised groups is a costlier process and can require certain adaptations to be made to appropriately meet participants' needs (⇡Chuang et al., 2021). For this reason, the cost-effectiveness of an intervention will vary between different sub-groups. Furthermore, EdTech can be leveraged to supplement limited resources in a classroom or educational setting, as well as to collect data on how educational and financial resources within an intervention should be allocated, to achieve equity in resource distribution (⇡Zubairi et al., 2021). EdTech can therefore be utilised in both the design and implementation of an intervention to achieve equitable outcomes.

Why this matters
It is important for EdTech research to contextualise points of comparison (learning assessment scores, cost data and cost-effectiveness data), because it enables cost-effectiveness comparisons to be made across contexts and time of implementation which non-contextualised comparisons are unable to provide.

Good practices from the wider sector
The points of comparison that enable cross-context cost-effectiveness analysis comparisons need to be contextualised. Across different education systems, the same learning grade will represent a different learning level (e.g., Grade 3 learners in one country will be at a different stage of learning to Grade 3 learners in another country), and this can be particularly true for the most marginalised and underfunded education systems where learners are more likely to be behind compared to their international peers of the same grade (⇡Filmer et al., 2018). When collecting learning outcome data, using international standardised learning assessments (which are often grade-specific) are therefore not entirely representative, or necessarily being delivered at the appropriate learning level given these differences between education systems. This undermines the extent to which cross-context comparability of standardised assessments is both useful and valid. Studies should therefore strive to present learning outcome data within its context, for example explicitly stating that learning outcomes were assessed among Grade 6 peri-urban Indonesian students, using the 2020 EGRA learning assessment. Similarly, there are a number of social factors such as language, proximity to school, and parental literacy that influence the cost-effectiveness of interventions (⇡Walls et al., 2021). Therefore, it is critical that these contextual factors are considered and accounted for within points of comparison.
When calculating cost data, there is also significant variation in the cost of intervention ingredients and inputs when considering both the country and time of implementation (⇡Walls et al., 2021). In particular, cost data has been shown to be very sensitive to fluctuating exchange rates and inflation over time (⇡Dhaliwal et al., 2013). Not accounting for these differences when capturing cost data will mean that the cost-effectiveness of similar interventions is not comparable across time, and so it is essential that cost data is contextualised within the timescale and wider economic conditions within which it is categorised, through accurate real-time cost capture. This will entail presenting a final cost figure that standardises the real-time costs captured into a single currency, which can then be compared to previous or later cost figures in a way that allows for the reconciliation of contextual differences. Similarly, cost data has been shown to vary depending on the scale of an intervention (⇡Evans & Popova, 2014;⇡Walls et al., 2021). Therefore, the scale of each intervention also needs to be contextualised, as trying to compare similar interventions implemented at different scales will not yield useful comparisons.

Important factors specific to EdTech
The use of technology has facilitated more points of comparison in data for both costs and learning outcomes. However, it has often done so in a way that has removed contextualisation. The use of EdTech in the collection of data for cost-effectiveness analysis needs to be made in a way that restores contextualisation, to enable more valid comparisons. EdTech needs to be leveraged to generate cost-effectiveness comparisons which clarify the conditionality of context, by explaining the factors that underpin why interventions are more effective in one context than another (⇡Government Outcomes Lab, 2022). In particular, personalised adaptive learning interventions that adopt an integrated data collection approach place an emphasis on data similarity in format, which can overshadow the vast differences in data context in terms of what the data represents. As a result, the 'trade-off' for data to become comparable instead of contextualised actually undermines the extent to which data points are accurately comparable across contexts.
Specifically to EdTech, the technology used, especially purpose-built technology that is adapted for education, is particularly susceptible to price volatility. Using a more generic commodified technology (such as a 'generic' smartphone rather than a specific child-adapted device) can help to mitigate against this sensitivity of devices to price, but can also invite additional challenges relating to shared usage or suitability for children. It is important to remember that the technology component of any EdTech intervention is likely to be susceptible to volatility in pricing, hence providing this contextualisation is integral to enabling comparability across time scales.

Replicability
This section outlines the need to ensure the replicability of cost-effectiveness conditions so that meaningful comparisons can be made at different points and planning for future interventions can be undertaken. Cost-effectiveness analysis is largely carried out under the remit of accountability for past projects, or speculation about the cost-effectiveness of scaling. While a donor who funds a range of projects within a funding cycle will want to compare the effectiveness of each project based on their actual expenditure, this may not represent the full cost of each intervention equally. Costs may have been fixed at the time of agreement and disbursement, but they will not reflect the cost of implementing the same project again, whether in a different context or even in the same location (⇡Bates & Glennerster, 2017). A replicable approach should ensure that learning outcomes and costs are captured in a way that is indicative of future cost-effectiveness for similar or related projects.
The sensitivity of cost to localised factors, as outlined by ⇡Dhaliwal et al.
(2013), includes: ■ Volatility (international markets, supply chains, currency and exchange) ■ Sensitivity to shocks (including the Covid-19 pandemic) ■ Cost at scale -often these do not correlate to costs in pilots or even regional implementation ■ Taxation and regulatory costs (including importation of technology) etc., -this is inconsistently applied to international organisations (vis-à-vis aid) Additionally, ensuring replicability in cost-effectiveness analysis depends on education outcomes being framed in a replicable manner. The above section has framed the need for comparable data on education outcomes, but beyond this, replicable outcomes depend on the interrogation of assumptions about how interventions lead to outcomes. An understanding of the enabling environment and the pre-conditions which enable positive changes in learning can help to ensure a sensitivity of the design to replicability.
Replicability is particularly problematic in EdTech interventions because of assumptions that can be made around digital readiness and literacy, the prevalence of a technology ecosystem, and inequity of technology access.

Why this matters
It is important for EdTech to account for uncertainties in cost and learning data robustly because technology introduces additional uncertainties around cost and continuity (for example, the issue of devices breaking down and the associated costs of materials and labour for repair) that can lead to disruptions having a disproportionate impact on the cost-effectiveness of EdTech interventions.

Good practices from the wider sector
Collecting cost and learning data needs to account for the uncertainties that are produced through sensitivity to price variability. Where cost and learning data contain uncertainty, the relative cost-effectiveness of different education interventions can change dramatically depending on the calculation used (⇡Evans & Popova, 2014). It is also important to highlight that the core components financed in a given intervention can vary significantly across contexts (e.g., teacher salaries). With this variation, the cost-effectiveness of similar interventions will inevitably change across contexts, and so it is essential that cost-effectiveness estimates are adapted using local costs (⇡Evans & Popova, 2014). The higher the number of components in an intervention, the higher the volatility (and therefore lower the replicability) and the cost-effectiveness of an intervention is likely to be. The cost-effectiveness of an intervention also differs with the scale (and different components of interventions will have different sensitivities to scale)(⇡Evans & Popova, 2014), and so this needs to be accounted for when trying to replicate results. An example is 'pilot bias' where both the costs and impacts of an intervention are overestimated at the pilot stage.

Important factors specific to EdTech
The volatility of technology cost with time, and across contexts when considering the purchasing power of different countries for technology devices, is a significant consideration (⇡Dhaliwal et al., 2013). Interventions which have technology elements that represent a significant proportion of funding are therefore more sensitive to volatility.
Uncertainties around the digital readiness of a context, and the existing technology ecosystem can also lead to unanticipated consequences when a learning intervention is replicated. For example, if one context has a lower baseline of digital readiness (overall lower saturation of technology, fewer localised applications, inconsistent connectivity, etc.) then a steeper initial learning curve for user familiarity is likely. When technology is prematurely introduced (e.g., before adequate training) into an operational context, educators and / or learners may abandon it before it is fully implemented. Willingness to persevere with technology despite initial challenges or repeated disruptions can be hard to define and discern even within one context, and therefore assumptions about baseline digital readiness at the start of an educational intervention should be tested with surveys on digital literacy (with the relevant stakeholders) and scoping research on the enabling environment.
In addition to this, the use of EdTech requires a robust understanding of the capacity for repair and maintenance at the point of implementation. In contexts where general costs of repair or replacement are extremely inexpensive for traditional learning materials, they can be prohibitively expensive for digital tools -or even simply impossible to obtain. This asymmetry profoundly affects usage, and can even lead to non-use as a preventive measure -for example, ICT rooms are locked up for fear children will break equipment which is perceived as irreplaceable, and which quickly becomes obsolete!

Why this matters
It is important for EdTech research to conduct cost-effectiveness analysis in the short and long term because the high up-front costs of technology mask the longer-term costs of maintenance, repair, and training. Conversely, initial barriers to use and the steep initial learning curve of using technology belies rapid learning gains once mastery of technology platforms is achieved.

Good practices from the wider sector
A key principle to undertaking replicable cost-effectiveness analysis is to consider the short-and long-term costs and outcomes (⇡Chuang et al., 2021). This goes beyond simply accounting for costs across the timescale of the intervention See Principle 1: Define data carefully) but incorporates a phased approach by conducting cost-effectiveness at different points throughout a programme and after its conclusion. Such an approach can account for diminishing returns on device functionality and different levels of technology usage at different phases through cumulative experience and preparedness, allowing learners to benefit more broadly. It is important, therefore, that the cost-effectiveness of interventions is tracked regularly, including the years post-implementation, to account for this variation.

Important factors specific to EdTech
EdTech interventions are particularly prone to volatility in financial markets and currency exchange, as devices are often mass produced in certain locations, and the exponential development of technological innovation can lead to a rapid depreciation in the value of technology hardware (⇡Booton, 2016) (see Figures 1-4 below). Furthermore, the distribution of technology to low-and middle-income countries (LMICs) often follows inequitable patterns -increasingly so as supply chains have faced unprecedented disruption in recent years. The volatility of technology costs has an important implication from a cost-effectiveness standpoint. The value of the technology used in an intervention is likely to be highest right at the outset, and financing any given technology element would be cheaper for donors if they were to wait for its implementation (⇡Chuang et al., 2021). There, therefore, needs to be stakeholder willingness to part with large sums of capital despite the knowledge that the value of the investment will depreciate fairly quickly. Similarly, a commitment not to reduce costs by using cheaper, but contextually inappropriate, technology is necessary -this has significant implications for the replicability and affordability of EdTech interventions.

Why this matters
It is important for EdTech research to represent data in relative terms, because the fluctuation of prices for technology-related costs is much greater than other educational costs, and is not necessarily aligned with other cost-of-living data, such as consumer price indexes, since it may not be categorised among daily essentials (see Figure 1 and Figure 2 below). Especially since technology tends to depreciate more rapidly as it becomes obsolete (⇡Booton, 2016), the cost of EdTech interventions must be calibrated to the context in relative terms in order to be relevant for comparison.

Good practices from the wider sector
Data from cost-effectiveness analysis should always be presented in relative cost terms (accounting for currency, purchasing power parity (PPP), etc.) and in terms of changes in learning outcome benchmarks over time. This relative representation of data is already embedded in the Strategic Impact Evaluation Fund (SIEF) mega costing model (⇡SIEF, 2020), and many representations of economic data, so it may not be controversial, but it is important to be explicit about this within the context of education, and especially EdTech, because of the higher variability of costs in LMICs. This approach ensures that cost-effectiveness figures can be accurately replicated when these adjustments are accounted for.
With regard to donors, there is an additional challenge around overcoming minimum costs and scaling for representing cost-effectiveness, as an intervention may have a fixed budget for the donor, but only be considered cost-effective at a larger scale. It is therefore understandable that internally, a donor will not be able to scale costs relative to the PPP of the country of implementation, for example. This may make their intervention more expensive and also does not reflect the cost to them -even relative to other projects funded within the same cycle. Yet for any meaningful comparison beyond the internal assessment of the particular cost to that particular donor at that particular time, a weighted value of cost must be presented.

Important factors specific to EdTech
One of the key aspects to consider for EdTech is the fact that the price of technology-related items is not correlated to other cost-of-living data that is much more closely aligned with inflation. For example, Figure 1 and Figure 2 below show that in the UK, cost-of-living data and inflation are much more closely correlated with fundamental non-technology items (such as food and energy) compared with technological goods which are non-essential. As a result, the relative cost of technology is disproportionately lower (and therefore more affordable) with time when compared to essential non-technology items (which retain more constant relative affordability through greater alignment with inflation). It is therefore essential in cost-effectiveness analysis to account for and capture the contextual price of technology because the relationship that technology has with inflation and its relative affordability is fundamentally different to other contextual cost-of-living-based goods. Extrapolating the price of technology retrospectively can lead to significant inaccuracies that do not reflect this change in relative affordability. Although similarly robust accessible cost-of-living data is lacking in LMICs, the price of mobile data can be shown to follow a similar trend of affordability when compared to cost-of-living (see Figure 3 and Figure 4 below): Figures 3 and 4 demonstrate that as the Consumer Price Index (the cost of essential goods and services) increases, the average cost of mobile data decreases, with 1GB of mobile data costing less in 2022 in all EdTech Hub focus countries compared to 2019. This serves to emphasise the way in which technology-related costs have a different relationship to inflation regarding relative affordability when compared with other goods, in low-, middle-, and high-income countries.
Therefore, EdTech costs, in particular, can be highly variable and contingent upon context, time, and externalities not controlled by education stakeholders. The necessity of making this data available in a responsive way, so that it can be represented in a manner appropriate to the relevant setting, is also facilitated by employing technology to interact with the data. Ensuring that this is clear throughout the value chain can ensure greater transparency and fidelity around how the data is represented. From a replicability perspective, ensuring that the costs of technology components are presented relative to the time of purchase ensures that their susceptibility to depreciation and relative affordability remains accounted for, as these are less consistent than other staple goods.

Sustainability
The third design theme is sustainability, which requires considering the cost-effectiveness of an intervention in the long term. This theme also includes considerations around affordability, which is often framed as a financing or even cash flow issue in providing EdTech, since technology can have higher up-front costs than other education approaches. It is framed here as a sustainability concern, because the affordability of an intervention may be resolved without addressing the underlying unsustainability of a given system. While cost-effectiveness analysis highlights the cost per unit of education, there are also structural reasons why it is important to consider the absolute cost and the system-wide impact of an intervention. The affordability of education interventions fluctuates globally as was highlighted recently by the Covid-19 pandemic, which has seen two-thirds of LMICs reduce their education budgets (⇡World Bank, 2021). As such, there needs to be consideration and certainty over whether interventions can continue to be financed and supported for investments to achieve sustainable scaling and cost-effective outcomes.
The difficulties of replicating EdTech interventions also mean that cost-effectiveness comparisons of similar interventions that are implemented at different times need to account for the variation in prices of assets between the two interventions. This approach should also account for the development of technology to encourage the use of equivalent hardware based on functionality, rather than the specification of particular devices, especially as some basic devices, like e-readers, are increasingly commodified.
Another important consideration for sustainability in EdTech is whether costs are perceived as proportional and reasonable, for example, the proportionality of ICT costs to the cost of living. If the basic needs of a community remain unmet, and an intervention seeks to meet unfelt needs (for EdTech), even if the intervention is otherwise appreciated, the imperative of meeting more basic needs of food, water, shelter, security etc, may lead individuals to appropriate EdTech devices and infrastructure to meet these basic needs.
Similarly, at a national level, donors especially should consider the cost of EdTech relative to the national education budget. An intervention that approaches the national scale of investment is likely to be perceived (at best!) as a paternalistic assertion that an outside organisation knows better what a country needs than the government does, in terms of spending priorities. However this may be perceived, ongoing support of such interventions is unlikely to be sustained at a national level as it is simply not affordable given spending priorities on other more basic needs.
The last three guidance points have a shared focus on sustainability and future considerations for cost-effectiveness from an analysis, design and investment perspective. When considering cost-effectiveness analysis, the cost-effectiveness of an intervention is not fixed and exists on a spectrum fluctuating with time (as impacts and outcomes also fluctuate with time). Thus, defining the cost-effectiveness of an intervention needs to be considered a more continuous process than it is currently.

Why this matters
It is important for EdTech research to consider the full lifecycle of EdTech interventions because it allows education systems to accurately determine whether they can afford to take ownership of EdTech interventions, and maximises the use of, and expenditure on, resources in the long term.

Good practices from the wider sector
The entire lifecycle of education interventions is a particularly important point of discussion. It is essential that the TCI of an intervention accurately accounts for costs incurred during the full lifecycle of a programmeincluding recurring, sustained, and incrementally accrued costs. This is significant as it provides a more realistic total cost estimate of the intervention across its full life cycle (rather than simply a cycle of funding or implementation). This allows key stakeholders associated with education systems to accurately determine whether an intervention is affordable, given these total costs. In all likelihood, accounting for these costs increases the associated TCI, making fewer interventions feasibly affordable, or leaving fewer systems able to fully support interventions financially (particularly in the current context of post-Covid education funding) (⇡Read, 2020).
This also has implications for the sustainability of an intervention more broadly. Considering the full cycle of interventions encourages considerations for how allocated resources can be maximised across longer time periods, in a cost-effective manner. This results in interventions that are naturally more sustainable in their design and simultaneously promote the sustainable use of resources and ingredients that constitute an intervention and its activities.

Important factors specific to EdTech
Many EdTech interventions are embedded with a high number of recurring, hidden, and post-implementation costs which make them less affordable, particularly in LMICs. This is particularly true if these costs are 'locked in' to the implementation from the start. For example, software licensing deals are fundamental to programme operation but often become incrementally expensive over time. It is therefore essential that EdTech interventions consider these costs that are associated with the function of the programme, and accurately capture these in the TCI, in order to determine the affordability of each intervention. It is also necessary to appropriately account for this affordability, to ensure that education systems have the financial capacity to successfully fund a full cycle of implementation, and to ensure the delivery of expected results and improvement in learning outcomes.
An additional key consideration with EdTech is that technology components are often associated with a lack of sustainability. Technology is prone to breaking, needing repair, and often has a short and finite life span (particularly with regard to remaining useful in the context of its intended function within an implementation) (⇡Selwyn, 2021). Giving detailed consideration to this encourages the use of technology that is more durable and sustainable while remaining highly appropriate for the implementation context. This prolongs the extent of time that devices remain actionable and impactful on improving learning outcomes.
An important environmental consideration attached to technology is the volume of 'e-waste' that is produced by EdTech initiatives, as this waste can be hazardous (⇡Forti et al., 2018). This is a particular concern in LMICs, where e-waste is less likely to be collected (⇡The Global E-Waste Statistics Partnership, 2019). Accounting for the full lifecycle of technology, and planning how to properly dispose of and replace technology during implementation, both encourage accountability and promote sustainability. Viewing technology components as longer-term assets (that also require appropriate management after their implementation use-cycle) helps to promote sustainable thinking and reduce e-waste.

Why this matters
It is important for EdTech stakeholders to incentivise transparent reporting because the hidden costs of technology can lead to a misleading picture of cost-effectiveness. The stakeholders who will most often be positioned to do this will be governmental regulatory bodies and donors, who have policies and reporting requirements for procurement and service provision, but may lack the technical expertise to address how new categories of cost are being represented. EdTech interventions (especially with personalised adaptive learning) can also have a tendency to present learning outcomes in a way which focuses on individual gains, rather than holistic improvement, which can obfuscate true progress against learning outcomes. This misrepresentation of results can be even more significant when programming is focused on smaller cohorts or not integrated within a broader curricular system. Where programmes claim systemic change, they must address the whole system and not a carefully chosen sub-system.

Good practices from the wider sector
Pressure to deliver education in a cost-effective manner can incentivise partners, suppliers, etc., to not report the full costs of assets involved in an intervention. There is a need to frame all costs within the TCI of interventions, and this needs to be incentivised by those involved in cost-effectiveness analysis and decision-making. The accurate reporting of costs is essential to deliver cost-effectiveness estimates that are precise and accurate, to ensure that they are as replicable and affordable as possible.
Transparency has been encouraged through the publishing of data publicly from public and government donors, and the broader move to open data has certainly given more information and authority to the public. The UK's OpenGov and the Open Data Initiative have inspired similar approaches in Kenya and Rwanda and the African Union's African Data Consensus initiative (⇡UNECA, 2015). Transparency enables greater accountability and civil society participation generally, but with regard to EdTech, it also ensures fair practices from multinational technology companies whose wealth and reach allow them to operate virtually, outside local laws in LMICs.

Important factors specific to EdTech
EdTech providers can be particularly susceptible to not reporting hidden costs, particularly where pricing models are inherently designed to lock in customers to a platform or hardware system (such as contractually, or by only utilising technology for a specific purpose). Especially with software and digital resources, pricing models may not be clear, since the cost is not a physical product. A lack of transparent pricing can increase hidden costs experienced later down the line, which may make initiatives unaffordable and hence unsustainable in the long term. One approach has been to prioritise free open-source software, but when this software requires technical expertise, the true cost may be hidden.
Other approaches have included licensing and support in a wraparound package, but the nature of the support and length of terms, not to mention changes to pricing when locked into a particular workflow, can lead to further hidden costs. This lack of clarity regarding costs across the lifespan of a technology or software product may not only be an oversight in terms of the costs that may be incurred but may even be an intentional pricing strategy on the part of technology providers. Structural incentives (especially for publicly held companies with shareholders) will favour representations of cost that are optimised for sales, rather than accurate cost estimates to end-users. When the end-users of EdTech are not the customers (as is the case in externally funded interventions) the accountability of capitalist feedback loops is undermined. Pricing models that are designed for consumer-driven capitalist contexts are therefore disruptive misrepresentations for donor-funded EdTech cost-effectiveness analysis.

Why this matters
It is important for EdTech to ensure interventions are proportionate to contextual priorities (both ministry of education budgets and any user costs associated with the intervention) because the costs of new technologies in LMICs can be several orders of magnitude higher than any other objects available within a context. While cost-effectiveness analysis focuses on distributed costs, across a period of time, with a number of users, the absolute cost at the outset may simply be out of proportion to other expenditures in the context -which may have implications for sustainability, security, and longevity.

Good practices from the wider sector
This final point leans towards a consideration of the broader operating system with which an intervention is being implemented. Delivering improvements in education indicators such as basic literacy and numeracy might be an important priority from a donor perspective (see ⇡Beeharry, 2021). However, broader societal priorities (such as instability and conflict, a lack of food, a lack of employment opportunities etc.), may outweigh EdTech priorities costing a significant amount of money. Contextual priorities (and finance) therefore need to be considered when weighing up the appropriateness of an intervention within local contexts, even if the intervention is cost-effective (⇡Bates & Glennerster, 2017).

Important factors specific to EdTech
It is especially important to consider the surrounding context and infrastructure if you are introducing technology into a new environment, and this is particularly true for LMICs where technological infrastructure is often much less established. A consideration of the cost of EdTech devices against household income to determine their contextual appropriateness is essential. Further, EdTech interventions need to consider if the existing local technology infrastructure can support the intervention in the long term. Factoring in the costs of maintaining or providing this infrastructure in the long term to determine how contextually appropriate and affordable an intervention is, is equally important.

Practical application -raising standards
The application of the nine principles outlined above will lead to an improvement in the capturing of cost and learning data and an improvement in the cost-effectiveness reporting of EdTech interventions. This will in turn lead to greater impact on addressing the global learning crisis through the use of EdTech. Prioritising a rigorous approach to cost-effectiveness outlined in these principles is a complex process and requires different levels of application for different stakeholders. This section focuses on how these principles can be applied in practice for each of the stakeholders engaged in cost-effectiveness work within the EdTech space.
Before outlining application standards for each stakeholder, it is important to outline three important trade-offs regarding the application of cost-effectiveness analysis. First, there is a need for each stakeholder to collaborate closely to make significant progress in raising the standard of cost-effectiveness analysis as there is significant interdependence between the different groups. However, each stakeholder has different requirements and obligations to fulfil, and hence there is a range of competing standards for cost-effectiveness analysis. These need to be understood and outlined at the outset, in order to make progress and ensure conflict of interest does not interfere with cost-effectiveness analysis rigour.
Second, there is a wide variety of practice and capacity across the sector, which needs to be acknowledged, and not resisted. Cost-effectiveness guidance should help different contexts develop a gradually improving practice, rather than imposing a supposedly universally applicable benchmark. Not all stakeholders in EdTech will be able to immediately pivot their practices to meet the gold standard of cost-effectiveness analysis, and this is why the minimum standard is provided forsomething all stakeholders should be able to implement immediately to advance cost-effectiveness analysis. This is an important short-term versus long-term trade-off that encourages incremental and realistic progress towards an advanced standard of cost-effectiveness analysis.
Lastly, and linked to the previous point, it is inevitable that all stakeholders engaged in EdTech start at different points and have varying capacities to engage with cost-effectiveness within EdTech. The sector requires practical guidance that demonstrates the positive direction of travel for each stakeholder, in a way that is realistic to achieve. This direction of travel is provided through mapping both the minimum and gold standards for each stakeholder so that they are able to position themselves on this roadmap. Stakeholders can identify where they fit within the standards below, and can gradually incorporate aspects of the gold standard to incrementally advance the quality of cost-effectiveness analysis.

Minimum and gold standards for each group of users
The next paper in this series, Paper 3 (⇡Mitchell & D'Rozario, Forthcoming), presents a tool for capturing and representing cost-effectiveness data in a comparable manner. By using the tool (as outlined in the next paper) stakeholders can generate data that is represented in a consistent and rigorous manner, allowing equal participation in the same conversation.
In summary, the 'minimum standard' refers to what all those engaging in EdTech should provide in relation to cost-effectiveness in order to be credible. The 'gold standard' refers to what all those engaging in EdTech should aspire to in relation to cost-effectiveness in order to drive forward progress in this area. The detail of how this is applied is provided below.
Within the use of this tool, the minimum standard of use requires reporting accurate cost data. Where that data is incomplete, the tool can estimate and extrapolate based on relevant data, which accounts for the observed reality that EdTech interventions are often more costly than anticipated. The extrapolated total cost figure, for those meeting a minimum standard of data will also include a range of uncertainty to reflect the range of costs which may be incurred. Where stakeholders can provide more complete data around costs, including externalities, they can meet a gold standard of reporting. This allows more accurate quantifying of risks and potential additional costs, which in turn allows the tool to project relative costs in other contexts. For example, if complete data is added about an intervention in Ghana, this will be specific enough to permit a projection of costs for the same intervention and type of results to another country.
The notion of minimum and gold standards for learning outcomes should be understood within the context of the assumptions they rely on. Some stakeholders may be comfortable with the links between attendance and learning outcomes in their context, while others may consider calculating LAYS based on attendance as only a minimum standard. The tool allows a range of assumptions to be made by stakeholders about what learning outcomes should look like, without making a judgement about what outcomes are better. Nonetheless, interventions which are able to provide more granular data for the tool will generate more precise LAYS figures, which will enhance their comparability.

Implications of the minimum standard
For those implementing EdTech: The implication of the minimum standard is that performance indicators related to learning outcomes and associated costs need to be explicitly defined prior to implementation, with appropriate monitoring and evaluation processes to measure and capture real-time data for these indicators established. This is important as it ensures that cost-effectiveness calculations are centred on robust data that is carefully defined and captures the full extent of both learning and cost.
For those selling EdTech: The implication of the minimum standard is that it necessitates engaging with cost-effectiveness in a way that encourages the value proposition away from a short-term profit motive, and this imperative for vendors to be transparent about ongoing costs may conflict with their obligations to shareholders to maximise profit. The importance of adopting this approach is that it is a critical step in generating accurate cost figures that are essential to cost-effectiveness analysis and robust cost-effectiveness comparisons.
For those funding EdTech: The implication of the minimum standard is that funding should be prioritised for (or limited to) EdTech interventions that are contextually affordable within the wider education system. The importance of this is that it results in EdTech interventions that are much more likely to be successfully sustained, resulting in a pattern of sustained improvements in learning outcomes as a norm with EdTech.
For those researching EdTech: The implication of the minimum standard is that cost-effectiveness data must explicitly represent the context from which it was derived when being presented in reports and outputs. This is important as it ensures that generalisations and assumptions around cost-effectiveness cannot be extrapolated from their specific contexts and misapplied. Representing data in this dynamic and contextual way also represents good practice with data more generally, as it reflects the relative contextual differences between two different data points. This is particularly important to maintain for cost-effectiveness comparisons and also represents a shift to ensure that education research includes cost-effectiveness analysis, which is currently inconsistently deployed.

Implications of the gold standard
The gold standard approach presented here describes characteristics and features that a model for rigorous cost-effectiveness analysis should include for different stakeholders. More detail on an operational model is presented in the following paper but important elements for key stakeholders are outlined below.
For those implementing EdTech: The implication of the gold standard is that equity needs to be rigorously accounted for in programming, with different marginalised groups being targeted in the most appropriate way during implementation. The importance of this is that it ensures marginalised groups are properly accounted for within the intrinsic operation of a programme. Further, the variation in outcomes, cost, and subsequent cost-effectiveness between different groups is a reflection of the equity in terms of impact of the specific EdTech intervention as opposed to any unequal inefficiencies in implementation or programme delivery. This in turn means that cost-effectiveness analyses that account for equity weighting represent the equity of programme cost and learning outcomes as much as possible.
For those selling EdTech: The implication of the gold standard is that where the technology products of vendors are in demand by implementing partners (or where there is no meaningful alternative), a sustainable compromise may be required that encourages vendors to emphasise long-term partnership and device use over short-term profit. This is important because it fosters sustainability, reduces dependency on global supply chains which can be overexposed to volatility, as well as crucially provides ministries and local partners with greater agency to take ownership of EdTech interventions.
For those funding EdTech: The implication of the gold standard is that the context of cost-effectiveness data must be accounted for when comparing and assessing the relative cost-effectiveness of different EdTech interventions. This is important because comparisons need to be valid between contexts to increase the weight of the evidence base for cost-effectiveness that informs funders to make robust decisions. Additionally, accounting for context in cost-effectiveness comparisons means that it remains valid as a fixed comparison point, which again increases the evidence base for cost-effectiveness comparisons. It also allows for the tracking of the relative cost-effectiveness of an intervention over time, which is particularly important for determining the relative cost and impact during the lifecycle of EdTech interventions.
For those researching EdTech: The implication of the gold standard is a commitment to accurately account for and represent different marginalised groups when reporting cost-effectiveness data. The importance of this is that it substantially improves the accuracy of data by reflecting the nuance and variation in the cost-effectiveness of reaching different groups as part of the same intervention. This enables a more rigorous understanding of how to reach marginalised groups in a cost-effective way through recognising the deviation in impact and cost of an EdTech intervention for different groups (rather than amalgamating all data and presenting a ballpark figure for the whole intervention).

An updated paradigm for cost-effectiveness analysis
Based on the above guidance, an updated paradigm for cost-effectiveness analysis is proposed. With regard to cost, this approach builds on the World Bank SIEF costing model for messaging (⇡SIEF, 2020), with a broader approach to a diverse range of intervention types, and a holistic approach that puts less emphasis on analysing the cost-effectiveness of each part of a programme. 1 With regard to measuring learning outcomes, an updated approach to LAYS is proposed, which integrates equity weightings for marginalised learners and also recognises the relative differences of how much additional years of education are worth in each context.

Cost
The most comprehensive approach to cost analysis is the World Bank's SIEF costing model, which provides granular guidance for capturing costs accurately and scales them based on national-level data on costs in the country of implementation (⇡SIEF, 2020). This approach has a specific emphasis on the "ingredients method", which maps each line of expenditure to a project activity or aim. The ingredients method, as promoted by SIEF's cost capture model, is a particularly useful measure for identifying where cost savings can be made within a project, by ensuring that costly but ineffective elements are not retained in the next project cycle. However, for comparative cost-effectiveness analysis, this is not a major concern, and may in fact obscure the extent to which different programme elements and their costs are interdependent. Instead, a more holistic approach may be taken, based on an overall assessment of externalised costs that are not provisioned through core budgeting. This can be calculated through use of the SIEF costing model spreadsheet, which focuses on gathering granular data at the point of expenditure, and then adjusting these costs based on location. The reconciling of budgeted costs (even where granular) with full expenditure is an important step for well-planned projects to capture hidden costs as a project develops. The associated spreadsheet is available alongside Paper 3 (⇡Mitchell & D'Rozario, Forthcoming) in this series for anyone wanting to capture costs in this way, and provides guidance about how to capture, analyse, and represent those costs in a comparable way.

Learning-Adjusted Years of Schooling with an equity lens
LAYS is quickly becoming the consensus standard as a comparative tool for measuring learning outcomes. This research proposes the importance of adjusting LAYS to account for local education factors and for equitable outcomes for marginalised learners. These factors demonstrate broadly that each year of education does not have equal value, and therefore LAYS calculations must always be relativised.
As currently applied, LAYS looks at education in a fundamentally different way to how QALY looks at health. The fundamental premise of QALY is that there is a baseline of health against which any change is a downward adjustment. By calculating LAYS within the same paradigm, researchers impute a baseline of learnedness which has absolute relevance globally. By contrast, education systems in different countries have different educational needs and priorities -not to mention different individuals having different capabilities and needs.
A non-linear model of learning outcomes -particularly for literacy, better captures the consolidation of literacy skills at a later stage in multilingual contexts and reflects appropriate acquisition of learning in context. Grading literacy on a curve that reflects localised acquisition rates is therefore a more accurate representation of learning against expected standards.
In addition to these generalised learning adjustments to harmonise learning outcomes across different contexts, LAYS should remain true to the QALY approach that inspired it by applying an equity coefficient for marginalised learners, demonstrating the relatively greater value to their learning outcomes. The exact quantification of this requires more data, just as QALY utility adjustments are based on extensive health-related quality of life questionnaires, and the absence of a corresponding education-related quality of life, or quality of learning database demonstrates the still nascent foundations of LAYS as a metric. Nonetheless, if it can be accepted within a health context that, for example, visual impairment results in a quality-of-life adjustment of 0.6 (i.e., 1 yr = .6 QALY for that individual) then in an educational context, the learner with that visual impairment would have an adjustment to their LAYS score which reflects a "lower ceiling" of expectation. So if this visually impaired learner is in Togo, where a year of attendance equals .78 LAYS, then their LAYS score is divided by the equity adjustment, e.g., .78/.6 = 1.3. This allows education interventions to integrate a cost-effectiveness approach to marginalised learners which acknowledges the additional cost of catering to their needs while demonstrating the relatively greater benefit SEND and marginalised learners can gain from additional education.
These approaches to understanding learning outcomes and cost in a more contextualised and nuanced manner are enabled by technology, both at the point of data collection and analysis, but they are also required because EdTech reduces contextual distinctions. The ability to standardise data protocols so that learning outcomes can be compared across different contexts must be met with the responsibility of actually contextualising that data in meaningful, scaled ways that account for differences and distinctions. Thus, EdTech provides an opportunity to lead the way on cost-effectiveness analysis that is meaningful for a broader range of educational interventions.
Although these approaches are evidence-based, they also rely on assumptions around the generalisability of data from previous studies, notably around the links between attendance and education quality. Further research to better understand the continued rate of improvement to education outcomes with increased investment should be pursued, as it is clear that some upper ceiling exists (infinite investment cannot produce infinite learning), but the urgency of current rates of underachievement requires acting on what works now, before encountering its limitations.