Cost-Effective EdTech Paper 1 A position piece on how the sector can make progress

An output of the EdTech Hub, https://edtechhub.org


Figures and tables
. Cost-effectiveness approaches and their position on the spectrum of outcome and cost variables 29

Introduction
This report forms the first part of a set of three reports developed by EdTech Hub that consider the implementation of cost-effectiveness analysis for EdTech interventions in low-and middle-income countries.
This first paper offers an overview of the background of cost-effectiveness analysis and approaches of other stakeholders engaged in the education sector. It will provide a useful introduction and further context to cost-effectiveness in EdTech.
The second paper Cost-Effective EdTech Paper 2: Good practice offers general recommendations on nine key principles and good practices for the EdTech sector.
The third paper (⇡Mitchell & D'Rozario, Forthcoming) builds on the theory of the first two papers to offer specific implementation-related recommendations and guidance that are initially aimed at research within the EdTech Hub.
The papers do not need to be read in order, but to a reader who is unfamiliar with the topic, it may be advisable to read this paper first.
This paper does not represent a final position of EdTech Hub on cost-effectiveness in EdTech -it is an initial contribution to what is a long-term effort and will lead to further dialogue and evidence within the sector.

Structure and logic
This paper is organised into three main sections and the logic for this is outlined below: 1. Cost-effectiveness is set within broader education debates and is a complex and contested issue. Section 1 outlines this and shows why supporting the sector to make progress on this issue is important to the mission of EdTech Hub.
2. There are multiple overlapping terms that are used when talking about cost-effectiveness in EdTech, and often the same terms are used to mean different things or different terms are used to mean the same things. That is why Section 2 of the report provides working definitions on each of the key terms.
3. There are multiple different ways in which cost-effectiveness has been categorised into frameworks within the education sector, with a specific focus on our own domain of EdTech. All of the frameworks share some similarities and have some differences. It is important to understand the varying emphases within them, in order to engage with the differences in how cost-effectiveness analysis is currently being applied across the sector. Section 3 of the report provides an overview of five main approaches to cost-effectiveness analysis in low-and middle-income countries.

Why cost-effectiveness matters in EdTech
There is a clear lack of funding for education in low-income countries, while other countries lack the human resources to optimise the use of their allocated education budgets. This is unlikely to be resolved to the levels required in the immediate future because of significant constraints (⇡Beeharry, 2021). However, an area that is receiving increasing funding is EdTech (⇡Roddis et al., 2021), with one estimate placing the value of the global EdTech market at USD 106.46 billion in 2021 (estimated to be equivalent to 2.13% of the value of the total global expenditure on education in 2019 -according to figures from the Global Education Monitoring report in 2021), which is expected to increase with a compound annual growth rate of 16.5% in the period up to 2030(⇡Grand View Research, 2021. The 'market' from both a supply and demand perspective for EdTech is therefore expanding fast and for a range of reasons political, economic, and educational. At present, much investment in EdTech in low-income countries currently takes place without a robust focus on cost-effectiveness. If the expanding use of EdTech in low-income countries is to make a meaningful contribution to addressing the global learning crisis, then the sector needs to have a significant and sustained focus on improving cost-effectiveness and analysis of cost-effectiveness. This is part of the wider agenda to embed an evidence-based approach to determining what constitutes 'effective' EdTech (⇡Chuang et al., 2021).
It is clear that there are many different ways in which technology can be used to improve different aspects of education. However, even similar interventions may cost varying amounts of money and affect learning in different ways, depending on context. Calculating the total cost of an intervention, and not just the cost of the product is not straightforward and is often hidden, either by design or because of a lack of knowledge about how technology investments depreciate. Additionally, the impacts that are attributable to an intervention are challenging to properly document, particularly in a way that is comparable across contexts. As a result, it is difficult for decision-makers to compare different uses of technology (or any other intervention) and understand which is likely to bring about more positive change within a certain allocation of resources. The lack of understanding of cost, and sometimes impact, means that financial resources are often allocated to types of technology-based interventions that are less effective than alternatives.
If the use of technology is to have a positive impact on addressing the global learning crisis, it is necessary that specific interventions are promoted. Namely, those that have significant and sustained impact on learning outcomes in a way that is cost-effective, scalable, and context-appropriate. This requires a step-change in the way cost-effectiveness is understood, assessed, and communicated so that it can inform decision-making in a meaningful way. Centralising the place of cost-effectiveness analysis within the use of EdTech in low-and middle-income countries has the potential to have significant and catalytic impact on addressing the global learning crisis.

Why cost-effectiveness matters for EdTech Hub
EdTech Hub's core mandate is to provide evidence to inform decision-making within the EdTech sector. Cost-effectiveness is central to this as it ensures that financial resources are allocated appropriately to the most effective interventions that lead to sustained learning improvements. Improving sector-wide approaches to cost-effectiveness is therefore central to the mandate of EdTech Hub. This series of papers makes an initial contribution to the long-term work of the Hub on this topic -setting out current issues and proposed routes forward for EdTech Hub and the wider sector. Specifically, EdTech Hub seeks to: 1. Increase awareness across the sector regarding the importance of rigorous approaches to cost-effectiveness in EdTech 2. Establish guidance and good practice standards for all stakeholders -donors, implementers, researchers -to help bring improvement to practice 3. Build a culture of transparency and accountability regarding consistent reporting and calculation of costs 4. Model transparency and accountability in our work -through the research and innovation of the Hub and the advice provided to governments 5. Put power in the hands of decision-makers to make better cost-informed decisions about what technology to use, when, why, and how.

Points of clarification
Any contribution to debates regarding cost-effectiveness in education and EdTech is inevitably controversial. It is a complicated topic with potential pitfalls of measurement bias, but it is critical to explore approaches in order to make groups of interventions comparable for both effectiveness and cost-effectiveness. The following introductory points are intended to provide clarification and should be kept in mind when using the paper.
1. There are overlaps in the cost-effectiveness of education in general, but there are important factors that are specific to EdTech. Much of the technical work on cost-effectiveness in EdTech links to the wider education sector, but there are specific factors relevant to EdTech that make this different. The paper engages with many of them including the nature of specific incentives, the associated discourse of innovation and experimentation and scaling, the cross-sector nature of many interventions, the complexity of infrastructure and sunk costs. The point to note is that making progress in measuring cost-effectiveness in EdTech can be supported by engaging with how cost-effectiveness is being implemented in education as a whole, but the particular opportunities and challenges relating to EdTech should be recognised.
2. Improving understanding of cost-effectiveness is a long-term and ongoing endeavour. This paper is one step in the process of engaging more deeply with cost-effectiveness in the sector and does not represent a final position of EdTech Hub. It is an initial contribution to a long-term effort and will lead to more evidence and hopefully greater agreement within the sector about how to measure cost-effectiveness. Likewise, there is no single 'solution' to solve the challenge of cost-effectiveness. Stakeholders engaged in EdTech are at a range of different starting points and have different levels of capacity to conduct rigorous cost-effectiveness analysis. Every step taken towards promoting cost-effectiveness in EdTech helps build a culture of evidence and will lead gradually to more informed decision-making.
3. The focus here is on student learning outcomes, but cost-effectiveness work also needs to go beyond this. The majority of this paper focuses on cost-effectiveness in relation to student learning. This is not the only aspect of educational change that the Hub is engaged with, given the focus areas of data for decision-making and teachers, but it is the most important place to start for introducing a consistent approach to linking cost and outcomes.

4.
Cost-effectiveness in EdTech should always also consider non-technology-related options. Any engagement with the cost-effectiveness of EdTech should take place alongside consideration of non-technology or blended interventions, not just a comparison between different technology-based options. The objective is to maximise impact on learning outcomes and that means it may often be more cost-effective to focus on non-technology implementation options.

Cost-effectiveness should not be used to present overly-simplistic assessment of complex systems.
There is a risk that promoting cost-effectiveness in EdTech gets misunderstood as promoting a single 'answer' for decision-makers. Any calculations related to cost should never be presented as 'solving' a problem on their own for decision-makers but instead should be viewed as one source of data that helps solve complex problems. In particular, there is still a need to consider whether a different intervention or systemic reform would represent a more appropriate and cost-effective alternative. Similarly, nuanced engagement with cost-effectiveness has to be centred on the context for the implementation.
6. Understanding the context and its implications for cost-effectiveness is essential. If cost-effectiveness analysis does not focus on the implications of context then it will give the illusion of progress without the substance -although technology itself is showing broad applicability, the uneven rate of adoption makes it hard to quantify benefits which can be generalised. Focus on cost-effectiveness in EdTech should be adopted alongside other factors rather than at their expense. Increasing the importance of cost-effectiveness in EdTech does not mean decreasing the importance of context, equity, holistic outcomes, or any other principle of normal good education programmes.

7.
Costs are often higher in more marginalised contexts. Driving cost-effective EdTech should not lead to always promoting the lowest cost per child. It is regularly the case that implementing EdTech is cheaper where there is more pre-existing infrastructure. Pilots are often (problematically) conducted in 'easy-to-reach' areas -and it is important to remember that what is feasible in one context is not necessarily feasible in another, and cost per child varies significantly according to context. The cost figures per child should not be the only factor used when determining if an intervention is appropriate -they simply provide a data source that can contribute to ensuring more effective decision-making regarding implementing EdTech.

Engaging with cost-effectiveness is complex and contested in
EdTech for many reasons. Working explicitly on cost-effectiveness, and the associated requirements of data sharing regarding cost is sensitive for any stakeholder. All stakeholders may be reluctant to share all the necessary information for a wide range of different reasons. Sometimes the cost of implementation might be deliberately hidden or skewed by those promoting it -but also more broadly the challenge is that many EdTech implementers are simply not accustomed to identifying costs appropriately -historically, they have not been asked to do so, do not know how to do this consistently, and have not had guidance on how to capture the entire cost of an intervention and who to share it with.

Learning-Adjusted Years of Schooling (LAYS) is a flexible concept for measuring learning outcomes that can be further refined.
Although LAYS has been used to describe learning outcomes that relate to standardised testing and attendance, as a descriptive concept it can be broadened and more flexibly applied to provide a comparable metric. Some of the assumptions around attendance and testing that LAYS is based on should be considered, and further research is needed to robustly link participation and engagement with cognitive outcomes. This research does not engage directly with how that is done, however, the use of LAYS within the framework of this paper (as the metric for measuring the effectiveness of an intervention) is based on a flexible and descriptive approach, which allows LAYS to be refined over time.

Definitions
In order to navigate discussions regarding cost-effectiveness within EdTech, it is necessary to provide working definitions of the central terms, in addition to highlighting how these terms are already being utilised within the education sector. Each of the terms is contested and has multiple legitimate definitions, so the definitions provided are not intended to be definitive but simply to increase clarity for the reader. The focus of this section is on how research engages effectively with the cost-effectiveness of EdTech, and not how EdTech can be more cost-effective compared to other interventions (as they may measure cost-effectiveness differently). The latter is a related and significant concern that is beyond the remit of this document.

'Cost-effectiveness' in EdTech
EdTech Hub defines cost-effectiveness as a figure that categorises how much an intervention will cost to deliver a specific level of impact over a given length of time (⇡Chuang et al., 2021;⇡Evans & Popova, 2014). In an education context, cost-effectiveness analysis is a means of measuring the educational outcomes provided by given inputs. Cost-effectiveness analysis is centred on two main components of an intervention -the cost, defined economically, and the outcomes, defined in terms of direct and indirect participation or learning outcomes. It is presented with one of these two components 'anchored' to show either the impact of an intervention for a given cost incurred or the cost required to achieve a given impact over a period of time (⇡Walls et al., 2020). While educational outcomes can be defined in financial terms, based on the financial return on investment of schooling (⇡Psacharopoulos & Patrinos, 2018), the intrinsic value of education both to the individual and to society, is measured as participation in quality education, alongside quantified measures of foundational learning skills such as literacy and numeracy.
Cost-effectiveness matters particularly due to the significant impact it has on policymakers, as it enables them to make evidence-based decisions on how best to utilise investment (⇡Evans & Popova, 2014). Fundamentally, this helps determine how to most efficiently use funding in a chronically underfunded sector, by using evidence to guide policymakers and donors on how to achieve the maximum impact.
Across the education sector, cost-effectiveness analysis has been used to compare the impacts and costs of programmes that worked towards achieving similar objectives (⇡Dhaliwal et al., 2013;⇡Evans & Popova, 2014).
More recently, the scope of cost-effectiveness analysis has been expanded to use it as a central tool in comparing the relative benefits of different approaches to education (⇡Angrist et al., 2020; ⇡World Bank et al., 2020).

'Total cost of implementation' in EdTech
The EdTech Hub defines the total cost of implementation (TCI) as the entire cost of setting up, implementing, and sustaining an EdTech intervention.
When conducting cost-effectiveness analysis it is important to define the costs used in the calculation carefully. The total cost of implementation (TCI), also referred to as 'total cost of ownership' (where ownership is seen as 'those financing all components of an intervention') represents all of the costs incurred during an EdTech intervention, encompassing long-term costs as well as both the direct and indirect costs associated with the initiative and its ongoing use and maintenance (⇡Chuang et al., 2021). For example, the TCI would include the costs associated with the maintenance, repair, and replacement of technology, including recurring costs and costs accrued after the end of the investment period.
It is important to account for the total cost of implementation because there has been a tendency for EdTech interventions not to consider these costs, or indeed not fully report cost data (⇡Evans & Popova, 2014), which artificially underestimates the cost-effectiveness associated with given interventions. This can subsequently impact negatively how decisions are made in allocating funding. It is also specifically important for EdTech interventions, as technology is associated with a number of long-term costs (subscription fees, maintenance, upgrades, vendor lock-in etc.) that add significant costs to an intervention beyond the conclusion of donor funding, and so these need to be accounted for to accurately capture the total cost of an intervention Across the sector, there is no concrete definition of the TCI, in terms of explicitly prescribing all of the costs captured within it. ⇡Chuang et al. (2021) list a number of EdTech-specific costs that need to be included within the TCI, such as the salaries of support team members and costs associated with parental engagement. Although not defined as TCI,⇡Walls et al. (2020) also note that cost is defined as the entire monetary expenditure required to produce an outcome and that this needs to account for the expenditure required to develop and implement an intervention (although TCI would also include costs needed to sustain the intervention).

'Value for Money' in EdTech
'Value for Money' (VfM) is similar to cost-effectiveness in that it refers to the optimal use of resources to achieve intended outcomes within an education intervention (⇡Walls et al., 2020). However, VfM is an evaluative judgement rather than a robust metric measured through a formula (⇡King & OPM VfM Working Group, 2018), that is not limited to effectiveness. Instead, it is an overall descriptive judgement over the extent to which resources within an intervention were used optimally, and appropriately, within the broader operating context where it was implemented. As a result, VfM is much more context-specific and not comparable (two interventions may have the same cost-effectiveness, but that may represent 'good' VfM in one context and 'poor' VfM in another).
VfM is important because it provides contextual meaning and judgement to the cost-effectiveness of interventions. It is a supporting articulation that adds value judgement and explanation as to what these cost analysis figures represent, given the context and operating system of implementation (⇡DFID, 2011). As a result, it helps decision-makers to think critically about what constitutes an appropriate level of cost-effectiveness across different contexts, and therefore make more informed and evidence-driven decisions (⇡DFID, 2011).
There is no universal definition of VfM, but in the education sector, it is mainly centred on maximising the efficiency of incurred costs, in an equitable manner that contributes to intended learning outcomes (⇡King & OPM VfM Working Group, 2018). FCDO's (formerly DFID) 1 top line on VfM in programming is that it "is about maximising the impact of each pound spent to improve poor people's lives." (⇡DFID, 2011, p.2 Economy -Are we (or our agents) buying inputs of the appropriate quality at the right price? This examines the cost of EdTech inputs relative to their quality.
Efficiency -How well are we (or our agents) converting inputs into outputs? This is a measure of productivity, determining how much of an output is achieved for a given input ('Spending well').
Effectiveness -How well are the outputs from an intervention achieving the intended effect? This is the measurement of the outcomes achieved by a programme and compared to its intended objectives ('Spending wisely').
Equity -How fairly are the benefits distributed? To what extent will we reach marginalised groups? This determines which groups are reached by the programme, and how outcomes and impacts are distributed across different population groups ('Spending fairly').

'Affordability' in EdTech
Affordability in EdTech is related to the total cost of implementing and sustaining an intervention, and the extent to which it represents an appropriate and feasible investment within the broader operating context. Decision-makers are constrained by a fixed budget (⇡Sendi & Briggs, 2001), and so affordability is defined as the ability of education systems to fund the total cost of an intervention, considering the financial resources that are available (such as within a government or ministry of education). Crucially, this also entails consideration of the long-term funding required to sustain EdTech initiatives, where EdTech interventions fall into 'systems of affordability', whereby users are able to depend on costs being covered (either by the government or other funding bodies) in the long term. When accounting for the TCI, an intervention can only be deemed as 'affordable' when the overall finance is self-sustaining within its operating system, in the long term.
This is important because it means not all investments that lead to significant cost-effective learning gains are affordable (⇡Chuang et al., 2021). Affordability represents a crucial link between cost-effectiveness and scale.
While an intervention may be cost-effective per student when implemented at scale, the total cost that is required to deliver that scaled intervention in the long term also needs to be affordable, given the finances available within the wider system. The increased total cost associated with intervention scale-up often means that interventions that are cost-effective per student are unaffordable given ministry budget ceilings, particularly in LMICs (⇡Read, 2020). Outside of education, health literature has discussed in detail this dilemma which means cost-effective interventions are often unaffordable (see ⇡Bilinski et al., 2017;⇡Levin & Chisholm, 2016;⇡Lomas et al., 2018), arguing for greater contextual understanding of cost-effectiveness given fixed budget ceilings and costs of scaling. In an education context, policymakers, therefore, need to consider interventions that are cost-effective in improving learning outcomes but also need to be realistic about which interventions can actually be financed (⇡Chuang et al., 2021). Considering affordability necessitates that policymakers understand the cost-effectiveness of interventions within the context of the funding available (⇡Bilinski et al., 2017), as often cost-effectiveness is only considered at the end of a programme's implementation, without regard to the affordability of scaling and sustaining initiatives when the wider system assumes funding responsibilities.
Within an intervention, this could include developing an affordability threshold (such as MoE spend per child per year divided by the total cost of implementation per child per year). Developing a simple rating will enable a comparative measure of how affordable an intervention is within its wider operating system, which is crucial to achieving sustained EdTech interventions.
Cost-Effective EdTech Paper 1: A position piece on how the sector can make progress 15

Current situation of cost-effectiveness in education
This section gives an introduction to cost-effectiveness analysis by summarising five main frameworks that are utilised across the education sector within low-and middle-income countries. Other frameworks exist for high-income countries, and although we have considered these, the unique challenges relating to implementing cost-effectiveness in low-income countries mean that it is important to engage with frameworks specific to low-and middle-income countries. The section then identifies significant cross-cutting threads from the frameworks, and each framework's approach to cost and outcomes that is important for the reader to be aware of. The five frameworks discussed in this section based on data from low-and middle-income countries, are: These five frameworks have been selected as they represent the approaches that are most widely used to inform decision-making across the sector. The first three approaches outlined are significant as they encompass the most up-to-date and widely used guidance on undertaking cost-effectiveness analysis within the education sector. While the last two approaches that discuss VfM (GEC and OPM guidance) are not necessarily as widely used, they nonetheless provide clear examples of different practical applications of the significant DFID guidance on VfM within the education sector.

World Bank LAYS approach
The LAYS approach has gained significant traction among policymakers in education and is regularly used as the main comparable benchmark of cost-effectiveness. LAYS is fundamentally an impact framework, but it is usually combined with cost to act as a cost-effectiveness framework. The LAYS approach aims to determine the comparative quality of learning between contexts by creating a benchmark standard of learning which can be related to specific educational interventions or educational procedures and systems, which can then be equated as equivalent (⇡Angrist et al., 2020). For example, both Ghana and Eswatini deliver a LAYS score between 5.5 and 5.9 within their education system (⇡Crawfurd et al., 2019), but this is achieved with an average expected years of schooling in Eswatini of just over 8, compared to 11.5 in Ghana, and hence the quality of learning gained for each year of schooling in Eswatini is considered to be higher than for Ghana. Using a universal metric such as LAYS therefore enables a comparison of the quality of learning experienced across contexts.

Cost
The LAYS approach can also be adopted with specific education interventions to detail a comparison of the quality of learning achieved by different interventions, implemented in different contexts. The LAYS measure is not always accompanied by cost data, but when available, the cost data is used to determine the LAYS per USD 100, which is used as a benchmark for cost-effectiveness (as shown below in Figure 2) (⇡Angrist et al., 2020). This creates a standardised cost-effectiveness measure intended to be applicable across a range of education interventions, as is used in the Global Education Evidence Advisory Panel 'smart buys' report (⇡World Bank et al., 2020). Reporting costs in this way requires that the entire cost of the intervention is aggregated and contrasted to the quality of learning that is achieved on an individual student basis. This gives a cost-effectiveness figure reporting the learning gains of each student, per USD 100 of the total project expenditure on each student (⇡Angrist et al., 2020).
Within a specific area of its work, the World Bank's Strategic Impact Evaluation Fund has developed an analytical tool for measuring costs across the lifetime of a project, accounting for both anticipated costs at a budgeting level, as well as infrastructure costs and unanticipated costs or variations to cost over the life of the intervention.

World Bank Strategic Impact Evaluation Fund costing model
Regarding cost capture, guidance produced by the World Bank Strategic Impact Evaluation Fund (SIEF) argues that "cost data should be disaggregated, intervention-specific, and captured in real-time"(⇡World Bank et al., 2019, p. 2). Disaggregated data needs to be input-specific and list individual variables that could be placed into an aggregate category (such as not simply listing 'salaries', but separating teacher salaries, maintenance staff salaries, and others). Cost data also needs to be intervention-specific, as often original programme budgets account for more costs than just those associated with the intervention (⇡World Bank et al., 2019). To do this, cost data needs to consider the level of effort of actors involved in the intervention, to calculate the proportion of costs (such as MoE staff salaries or contract vehicle usage) that are specific and attributable to an intervention. It is also important to determine which programme inputs are funded from other sources and to include this cost data. The final element of real-time capture argues that cost data needs to be collected throughout implementation, to create a more accurate summation of costs, rather than an estimate at the conclusion of a programme. This then helps create a more effective benchmark to estimate costs for scaling or implementing in other contexts (⇡World Bank et al., 2019). An example of how accounting for these factors might look is presented in Figure 3 below. Alongside this general guidance, ⇡SIEF (2020) have published templates and considerations for implementers on how to collect cost data, where they outline five main steps. The first two steps involve defining which activities and ingredients need to be costed as part of the intervention, relative to the business-as-usual scenario (⇡SIEF, 2020). This is intended to be an exhaustive list of all ingredients required to implement an intervention, including opportunity costs not directly attributed to donors, such as additional time commitments for parents (⇡Holla & Pan, 2020). This data is used to form an initial cost estimate. The third step requires implementers to develop a plan to collect real-time cost data, including identifying the units of measurement and frequency for each component, and the data sources required to obtain this information (including qualitative interviews) (⇡Holla & Pan, 2020). The fourth step involves adding cost data for each component (the quantity and price) in real-time, including noting when costs are recurrent or start-up, and the funding source to help disaggregate (⇡Holla & Pan, 2020). The last step involves finalising the actual costs and matching them to outcomes and impact data, to develop cost-effectiveness ratios (⇡SIEF, 2020). It is important that cost data accurately captures how things should be implemented (for example, the amount of time required for different staffing roles) to help policymakers with more effective planning (⇡Holla & Pan, 2020). A visual representation of these steps is provided in Figure 4 below.  Bank et al., 2019, p. 8 Outcomes Although not always a measure of cost-effectiveness, the LAYS approach is an impact framework centred on expressing the outcomes of a given education intervention, in a comparable way. The importance of LAYS is that it is itself an outcome measure that is comparable, by representing the quality of learning achieved through localised education initiatives. LAYS uses participation and national exam data to generate estimates of 'micro-LAYS', which enables a comparison of education interventions operating within more regional and localised contexts. Using internationally agreed test scores such as the Trends in International Mathematics and Science Study (TIMSS) or OECD's 2 Programme for International Student Assessment (PISA) give more accurate LAYS outcomes (⇡Angrist et al., 2020), however, there is a recognition that these are not widely available in many contexts, and therefore calculations based on harmonised learning outcomes represent a series of assumptions (⇡Patrinos & Angrist, 2018). Further critiques of LAYS are discussed in the subsequent paper (⇡Mitchell & D'Rozario, 2022).

BE2 Cost-effectiveness approach
The BE2 working group issued detailed guidance for measuring costs in education (⇡Walls et al., 2020). Their cost-effectiveness approach is just one of many cost-based analyses that they provide guidance on. It argues that cost-effectiveness analysis is centred on two core pieces of information: the cost efficiency of an intervention (i.e., cost per output) and the effectiveness of this programme (i.e., level of impact per output).

Cost
In cost calculations for donor-funded programmes, this approach advocates for an activity-based costing method (where activity is defined as any event, unit of work, or task with a specific goal, such as teacher training or undertaking a learning assessment), allocating shared costs across disaggregated cost categories, using real-time data that requires an established framework at the beginning of each evaluation. ⇡Walls et al., 2020 propose 12 broad categories for cost capture that are widely applicable for different levels of education (primary, secondary etc.): At the outset of each programme, a method established for allocating shared costs across these cost categories needs to be established, in effect to categorise all the costs incurred by an intervention. ⇡Walls et al., 2020 propose four steps for selecting cost reporting categories: 1. Based on the objectives underpinning the programme, establish cost questions that can be answered with cost data (e.g., 'how much will it cost the MoE to sustain continuous TPD for educators in pre-service teacher colleges?). These questions can then be broken down into cost estimates for each activity required to ensure this objective is addressed.
2. Match the objectives of the programme and cost questions with the different cost categories (listed above).
3. Determine which of these categories will need further sub-categories (such as 'creation of intervention' or 'implementation of intervention'), noting the importance of balancing between granular detail and reporting burden.
4. Ensure proper training of staff on how to accurately use cost categories and capture costs in real time during implementation.
When allocating shared costs to these categories and sub-categories, this approach argues that all expenses (the TC1) should be considered and included in cost calculations. This includes estimates of external contributions from other organisations that are essential for programme implementation.

Outcomes
When calculating the outcomes of a programme, the guidance does not advocate for exact and comparable impact data, but instead suggests there should be 'credible estimates of a programme's impact' using the context-specific data that is available to estimate the impact of the programme. This is because the main cost-effectiveness analysis approach is to develop and present a 'cost per unit of outcome'. As such, this approach utilises 'cost per unit of outcome' as its comparison metric (which is specific to the outcome data that is available), rather than using an 'outcome per unit of cost' figure (such as the USD 100 cost figure used in LAYS cost-effectiveness).

Cost
While USAID's cost-analysis guidance is aligned with other approaches, there are particular similarities with the SIEF guidance on cost capture around when and how cost is being captured (⇡Walls et al., 2021). These include guidance around cost modelling for future projects, as well as considerations for "cost against replacement" which are more explicit in how they incorporate opportunity costs. The approach to cost measurement standardises how implementing partners capture data on both costs and outputs, meaning they are well-defined and enable consistent comparisons of unit costs across contexts to understand relative cost drivers (⇡Walls et al., 2021). The USAID guidance also argues that what is considered a 'cost' is additional expenditure incurred by delivery beyond the 'business-as-usual' scenario.

Outcomes
USAID's guidance on cost-effectiveness emphasises comparability within the same context and considers it "Best practice […] to use the same impact indicators across programs" (⇡Walls et al., 2021, p. 21). It argues that outcome measures that are different, such as measuring reading proficiency using Early Grade Reading Assessment (EGRA) and national reading assessments, are not comparable over time and beyond the context in which they were implemented, which makes them less useful. As a result, it argues that the outcome measures that they want to compare across programmes are defined and standardised at the outset of each programme, resulting in the impact of programmes being comparable because they are measuring the same outcomes.

Girls Education Challenge Value for Money guidance
DFID developed its own guidance for VfM (⇡DFID, 2011). Within the context of education, this framework has been adapted for specific approaches to evaluating the VfM of education interventions. One example of the application of the FDCO guidance for VfM within education is the GEC's approach to VfM, which is "built upon the standardised Department for International Development (DFID)/National Audit Office 4E Framework" (⇡Shah, 2021, p. 5). There are three layers to the current guidance on VfM for GEC projects, which are light touch, moderate touch, and heavyweight (⇡Shah, 2021), each with an increasing requirement for depth and rigour in quantifying cost and learning data. All layers combine the conventional 4E's framework (detailed on page 15) with the OECD Development Assistance Committee (DAC) criteria to assess the following four categories: efficiency and VfM, effectiveness, equity and VfM, relevance and VfM, and sustainability and VfM.

Cost
The light-touch approach does not require any additional data than that which is collected by evaluators, to assess programmes against their stated objectives and evaluation research questions. As a result, it adopts a narrative approach and quantifiable cost data is not necessary. The moderate-touch approach builds on this but uses precise project expenditure data, which is usually real-time data, at the end of the programme. This data is used to develop a VfM metric (for example, cost per participant), but allows for this indicator to be specific to the intervention and available data. Both these approaches do not require additional cost amendments to be calculated (such as accounting for inflation, considering opportunity costs, or accounting for the TCI). As a result, neither of these two approaches to cost data are built to be comparable, or replicable, although they allow for judgements on cost-effectiveness to be made on programmes where robust cost data may be lacking. The more robust 'heavyweight' approach to cost-effectiveness uses disaggregated project costs to produce an externally comparable indicator of cost-effectiveness (such as LAYS per USD 100), but there is no preference on which comparative indicator should be used, or prescription on TCI being a necessity (⇡Shah, 2021).

Outcomes
Both the light-touch and moderate-touch approaches do not require an internationally comparable measure of outcomes, and additional quantifiable data on the impacts and outcomes of each intervention is not necessary. In these scenarios, a cost metric (such as cost per participant) may be evaluated against qualitative findings to enable an evaluation of whether the programme or specific aspects of the programme, represent VfM. This reliance on an external judgement to determine VfM is consistent across all approach levels, and due to the inconsistency of judgement required for these categorisations, they are not comparable.

Oxford Policy Management Value for Money guidance
The Oxford Policy Management (OPM) VfM guidance (⇡King & OPM VfM Working Group, 2018) similarly adapts and applies the broader DFID guidance, although it is a separate interpretation of the guidance to the GEC. It is important to note this approach is heavily embedded within an 'aid' and broader development context and is not specific to education. It is a very similar approach to the GEC VfM framework outlined above (in that it evaluates the 4 E's framework) but argues that VfM should be embedded in programme-specific definitions of the 4 E's (economy, efficiency, effectiveness, and equity) to give a unique approach to VfM for every programme, that is determined at the outset of the evaluation (⇡King & OPM VfM Working Group, 2018).

Cost
As a result, defining the approach to cost and outcomes will differ on a contextual basis, but there are some useful points to consider. The guidance recognises that there are different types of economic evaluation processes that can form the basis of VfM outside of just 'cost-effectiveness', but that these need to balance the efficiency of cost versus affordability, relevance, sustainability, equity, and others.

Outcomes
Similarly, the guidance advocates that determining the 'outcomes' of the project is not supposed to be a comparative metric but is defined by what the intended outcomes of each programme are. The intervention should generate context-specific 'criteria' within each of the 4 E's that focus on the intended functioning and effects of the intervention, which are then used as the 'outcome' reference point in determining the extent of VfM (⇡King & OPM VfM Working Group, 2018). It argues that VfM assessments should be carried out over different time intervals, depending on the nature of these intended outcomes. Table 1 below aims to capture some of the ways in which different cost-effectiveness analysis frameworks prioritise different aspects of their categorisation of costs and outcomes. The rows of the table focus on each of the significant cost-effectiveness analysis frameworks that were summarised above.

Significant cost-effectiveness frameworks: comparison table
The columns of the table identify five variables, each of which exists on a spectrum, whose extremes are represented by one side or the other, neither of which is necessarily better than the other. It is helpful to define these scales, and what each end of the scale represents in terms of emphasis and priorities.

Structure of the comparison table
The comparison table compares the cost-effectiveness analysis approaches in a descriptive way that acknowledges the motivations that different approaches may have -without claiming one approach is better than another in these areas. Each can be considered on a spectrum in granular detail, but here the categories are considered qualitatively as a means of demonstrating the different emphases present within the sector.

Variable 1 = Cost (budgeted vs paid)
This refers to the cost approach used to determine how the overall expenditure of a programme is defined. A 'budgeted' account of costs refers to the forecast total cost of implementation at its outset (sometimes called 'ex-ante' -meaning 'from before' the start of the project), which enables implementers to indicatively plan relative levels of expenditure on the programme, throughout its duration. This is often used where donors disburse a set amount of money based on what is budgeted, with no additional funding being granted. Cost analysis that is based on 'paid' representations (also known as 'ex-post', meaning 'from after') refers to the actual expenditure on the intervention, often based on receipts collected through programme delivery. These 'paid' figures may be gathered and calculated on a regular (e.g., quarterly) basis, or at the conclusion of the programme, which enables implementers to identify unanticipated costs of the programme, or account for underspends in areas which did not incur costs. This approach may be more aligned with cost-effectiveness analysis that is related to a commercial contract where services are provided before payment or on payment by results for interventions (cf. ⇡DFID, 2014) and disbursement-linked indicators (⇡Moran et al., 2020).

Variable 2 = Aggregation (ingredients vs holistic)
This refers to the different ways in which cost approaches aggregate (or disaggregate) the component costs of a programme. 'Ingredients' refers to cost approaches that disaggregate costs into all the individual components of a programme. 'Holistic' refers to capturing costs as an aggregated total. Each approach has its own benefits. For example, a disaggregated cost approach can better identify where cost savings can be made within a project, whereas a holistic approach views the programme as a whole and requires comparative analysis to incorporate externalised costs (every cost has to be borne by someone) rather than ignoring costs which can not be itemised on a receipt or invoice.

Variable 3 = learning outcomes (contextualised vs standardised)
This focuses on whether the learning outcomes utilised in cost-effectiveness calculations reflect localised learning levels (such as increases in school-based test scores), or learning levels measured using a universally recognised scale (such as increases in LAYS). Focusing on localised approaches enables decision-makers to determine the specific contextual impact on learning that a programme has, whereas a universal approach enables an estimate that can determine the relative level of learning achieved through an intervention when compared to other established approaches.

Variable 4 = equity (integrated vs exceptional)
This column uses a scale of 'integrated to exceptional', which refers to the extent to which equity is embedded within cost-effectiveness calculations. Integrated approaches to equity consider different groups as an integral part of cost-effectiveness, for example, determining the relative cost-effectiveness of an intervention when reaching different marginalised groups in the programme. Exceptional approaches still consider equity as an important element, but equity is determined as a separate issue that is not necessarily considered in cost calculations.

Variable 5 = financing (national vs global scales)
This column refers to how different approaches are financed between local and global scales. Approaches that determine financing on a global scale determine the overall cost using a currency that is applicable to global contexts (for example, USD) and global donors. Localised approaches for financing consider how this financing manifests itself within the local context of implementation (i.e., the local cost of implementation in that context), as well as determining the relative affordability of each local intervention.  Table 1 above describes several key features and tensions in how the prevailing frameworks and major stakeholders define the measurement of cost-effectiveness analysis. Note that these are not necessarily positive vs negative, but each approach has its own rationale -for example, donors need to be able to compare actual costs to them, which would not reflect costs that are adjusted for comparative purposes. These comparisons allow us to understand the priorities of different stakeholders as well as how a new paradigm can address competing priorities.

Cross-cutting analysis: the status of cost-effectiveness analysis in education
It is clear from these frameworks that donors place considerable importance on the collection and representation of data on costs and learning outcomes. Each framework maps its own priorities for analysing cost-effectiveness to the ways in which it can use that analysis. For example, donors may prioritise cost data based on budgets defined at the outset of a project, because this is the cost allocated by the donor, and it simplifies comparison to other costs incurred in similar budgets. Project evaluations that analyse cost-effectiveness might instead look at the total cost, or the cost against alternatives.
However the Strategic Impact Evaluation Fund (SIEF) within the World Bank has identified that budget estimates underreport costs by as much as 300% (⇡SIEF, 2020), and that recall bias on ex-post cost-effectiveness analysis can lead to significant underreporting of cost data (⇡Holla, 2019). Both factors can be exacerbated when dealing with EdTech since digital externalities and infrastructure (such as connectivity costs) are often omitted from cost estimates. In addition, for ex-post accounting of costs, recall bias of digital payments is generally worse than it is for physical payments (⇡Gafeeva et al., 2018).
Due to these propensities for inaccurate cost estimates, cost-effectiveness analysis on EdTech requires a robust approach that goes beyond the standard practices for education and development more broadly.
Cost-effectiveness analysis in education is increasingly facilitated by technologies which allow this data to be collected in uniform ways, and for differences to be rigorously tracked. When technology and other non-technology tools are used for collecting data around the costs and outcomes of educational programmes, it is important that a number of key principles are adhered to; these are outlined in the Paper 2 in this series (⇡Mitchell & D'Rozario, 2022). ⁅b