Delivering Remote Learning Using a Low-Tech Solution

During the prolonged school closure due to the Covid-19 pandemic, many children in rural Bangladesh were missing out on formal education. What is more, these children had limited access to distance learning modalities such as the internet, television, and radio. To counteract this poor access, this study used the widely accessible modality of mobile phones to deliver a set of audio lessons to primary school students using Interactive Voice Response (IVR). These lessons were developed using the Interactive Radio Instruction (IRI) method. IRI is a method that allows learners to stop and react to questions and exercises through verbal responses and to engage in physical and intellectual activities with any member of their household, while the programme is ‘on air’. In this study, caregivers played a crucial role in following the instructions and interacting with the learners accordingly. During the 15-week programme period, caregivers (mostly parents) could call a phone number free of charge and listen to the lesson with their child at any time of the day. This intervention was implemented using a clustered randomised design across three groups: Standard, Extended, and Control groups. At the end of the intervention, it was evident that this educational programme improved the learning outcomes of treated children by 0.61 Standard Deviations (SD) and 0.65 SD in the ‘Standard’ and ‘Extended’ groups respectively. These effects are equivalent to 0.59 Learning-Adjusted Years of Schooling (LAYS) for the Standard group and 0.64 LAYS for the Extended group. Besides learning outcomes, student effort, and caregiver time in homeschooling also improved. Student effort i.e., daily study time of the treated children increased by 0.09 SD (or 5.1 minutes) and 0.08 SD (or 4.5 minutes) in the Standard and Extended groups, respectively. Similarly, caregiver time in children’s education increased by 0.18 SD (or 10.5 minutes) and 0.12 SD (or 9.7 minutes) in the Standard and Extended groups, respectively. This intervention cost USD 27.5 per student, which is 2.21 SD (or 2.16 LAYS) per USD 100 of spending for the Standard group and 2.37 SD (or 2.31 LAYS) for the Extended group. Overall, the intervention was especially beneficial for students who were female, academically weaker, from the poorest socio-economic strata and who had parents with low levels of education. Based on the evidence and cost, we advise policymakers and practitioners to incorporate IVR-based programmes into their educational services for students with limited access to the internet and television. Delivering Remote Learning Using a Low-Tech Solution 3


Executive summary
School closures during the Covid-19 pandemic have resulted in an education crisis. Students in Bangladesh, especially those living in rural areas and urban slums, were particularly hard-hit during the prolonged school closures as many were unable to access distance learning due to the lack of access to the internet, TV, and radio. To ensure improved readiness in future, this study explores the option of using Interactive Voice Response (IVR) to provide education.
In this study, primary school-aged children from 90 villages in the Khulna and Satkhira districts were provided with pre-recorded interactive audio lessons via IVR on feature phones. The Global Development and Research Initiative (GDRI) implemented this intervention. This programme continued for 15 weeks and provided 75 lessons from three subjects -literacy, numeracy, and noncognitive skills training. Each of these lessons was 16-18 minutes long and developed exclusively for this intervention.

Purpose
The purpose of this study was to evaluate the learning efficacy of interactive audio lessons delivered via IVR on mobile phones. To evaluate the learning efficacy, this study focused on learning outcomes for primary school-aged children in areas ranging from language, literacy, and numeracy competency to leadership, communication, and planning skills. Moreover, this study estimated Equivalent Years of Schooling (EYOS) and Learning-Adjusted Years of Schooling (LAYS) gains attributed to the intervention, which will allow cross-study comparison.

Methodology
This study implemented the intervention in a three-arm clustered Randomised Controlled Trial (RCT) design. A total of 90 villages were divided into three equal treatment arms. In Treatment 1 (T1): the Standard group, students of selected villages were offered literacy and numeracy modules; in Treatment 2 (T2): the Extended group, a 'noncognitive skill' module was added to literacy and numeracy modules; and in treatment 3 (T3): the Control group, students were offered nothing. All students were assessed before the intervention and after one month of the intervention. Similarly, caregivers participated in both baseline and endline surveys. Treatment effects were estimated after controlling baseline academic performance, socio-economic and demographic characteristics.
Furthermore, we estimate that the additional LAYS gained by this intervention was 0.59 for the Standard treatment arm and 0.64 for the Extended treatment.
In addition to absolute LAYS, we estimated that the Standard treatment generated 2.16 LAYS per USD 100, while the Extended treatment produced 2.31 LAYS per USD 100.

Student effort
The results suggested that participation in the programme enhanced student time investment in education, interest in study, and attention span for both treatment groups. Treatment effects were 0.09 SD and 0.08 SD in the Standard and Extended treatment groups, respectively. In terms of daily study time, students of the Standard, Extended and Control group spent 92.1 minutes, 91.5 minutes, and 87.0 minutes per day studying, respectively.

Parental and caregiver engagement in children's education
Parental and caregiver engagement in children's education also increased as 1 a result of programme participation. Treatment effects were 0.18 SD and 0.12 SD in the Standard and Extended treatment arms, respectively. Caregivers who participated in the intervention devoted more time to their children's 1 Participants included both parents and caregivers. That said, in 79.33% of cases, mothers were responsible for calling our programme phone number. In 22% of cases, siblings or cousins were acting as caregivers. From this point we will refer to the participants as caregivers as a catch-all term.
Delivering Remote Learning Using a Low-Tech Solution 8 education compared to other caregivers in the control group; they spent an additional 10.5 and 9.7 minutes per day on their children's education in the Standard and Extended groups respectively.

Who benefited the most?
This intervention was especially beneficial for the female students, academically weaker students, students from the poorest strata, and students with less-educated caregivers.

Cost
Cost-volume analysis revealed that this intervention cost USD 27.5 per student over 15 weeks, of which USD 13.2 was the variable cost (voice and SMS charges, participant recruitment etc.) and the other USD 14.3 was the fixed cost (IVR platform development, content creation, overheads etc.). We provided this intervention to a total of 1,182 students from two districts. As a result, the fixed cost per student was very high relative to the variable cost. If the intervention is scaled up, per student cost will go down. For example, the cost per student will be USD 15.3 if this intervention, with the same content, is provided to 100,000 students. However, expanding content will add more cost.

Cost-effectiveness
A total cost of USD 15.3 per student for a 15-week educational intervention or USD 1.02 per week per student is not very cheap. However, treatment effect estimates indicated that this intervention could achieve 2.21 SD (or 2.16 LAYS) and 2.37 SD (or 2.31 LAYS) per USD 100 of spending in the Standard and Extended groups, respectively. This high level of gain makes this intervention cost-effective. Another important issue that needs to be considered is the flexible curriculum. This intervention was provided at the individual level, not in a group or classroom. This gives students more flexibility in choosing the time of study and content. Considering these issues, the cost of this programme is not high, however, it can be reduced if it is provided at scale.

Policy implications
During the Covid-19 pandemic, the use of wide-reach feature phones for educational purposes increased significantly. However, typically, this involved brief one-to-one calls with caregivers to follow up on their children's homework. Although these interventions provided personalised learning and teaching at the level of learners, this 'one-to-one call approach' is not particularly scalable due to its high reliance on human resources. However, by using IVR, it is possible to provide education at scale, at the right level of learners and during convenient hours of the day in out-of-school settings.

Delivering Remote Learning Using a Low-Tech Solution
Building on the experience and findings of this intervention, this study makes the following recommendations.
■ IVR-based distance learning should be provided at the individual level. School-level intervention might select less-interested caregiver-child dyads and end up with a lower level of learning gain, which might create a significant resource misuse.
■ For a successful distance learning solution, caregiver involvement is crucial. Caregiver roles need to be specific, and the curriculum must allow them to participate as facilitators, learning partners, etc. Moreover, they also need to be nudged regularly.
■ Most marginalised students should be the primary target of low-tech distance learning solutions. Students from relatively well-off families usually have multiple ways of accessing education, therefore, the marginal effect of low-tech distance learning may be low for them.
■ Distance learning services should have flexible delivery hours and retention mechanisms (such as quizzes used in this study) to provide both opportunity and incentive to engage caregivers.

Introduction
This research used a Randomised Controlled Trial (RCT) to evaluate an intervention in Bangladesh that supported distance learning of primary school-aged children via feature phones. The intervention specified distance learning as a method of studying in which lessons were disseminated, without students needing to attend a school. There is a gap in the current literature on studies on the specific use of feature phones for children's education in household settings, making this study an important contribution to the evidence base of the usefulness of using feature phones as a distance learning platform.

Background
When all schools were closed due to the Covid-19 pandemic in March 2020, the government of Bangladesh responded to the educational crisis by swiftly providing multimodal distance learning. By the first week of April 2020, the Ministry of Primary and Mass Education (MoPME) and Ministry of Education (MoE) started remote learning through asynchronous classes broadcast via national television and online platforms (⇡Rahman & Sharma, 2021). Online resources were already developed and available on various sites but expanded during the school closures. Radio broadcasting was added later in the year.
Despite the quick delivery of multimodal distance learning, a rapid survey by the World Bank found that only around 40% of students had access to remote learning in the first few months of the Covid-19 pandemic (⇡Biswas et al., 2020). Even after a year, a significant portion of children remained outside of distance learning coverage; 44% and 36% of rural households and urban-slum households did not have access, respectively . This lack of access led to poor learning outcomes as only 18% of primary-graders and 38% of secondary-graders were actively learning through assignments in August 2 2021 .
This poor level of access to distance learning is not unexpected considering the overall low rate of access to remote learning platforms by Bangladeshi households. Table 1 indicates levels of access to various remote learning modalities for all children aged 5-15 years. Only a small proportion of children have access to a computer. Though online materials can be accessed via smartphones, the active internet usage rate is only 28.8% in Bangladesh due to the lack of compatible devices and high data costs (⇡DataReportal, 2021). Access to TV is also not widespread. Only 56.7% of school-aged children have access to TV. This situation is even worse if we consider children from poor households and rural areas.
However, as Table 1 shows, more than 96% of children aged 5-15 have access to a mobile phone in rural areas. Although there are no estimates about the ratio of smartphones versus feature phones, based on the low number of active internet and social media users (⇡DataReportal, 2021), we can safely assume that most of these mobile phones are feature phones. Given their prevalence, feature phones could potentially be utilised to provide education at scale and to minimise learning losses due to poor access to other forms of distance learning modalities. Source: T. Rahman and Sharma (2021) Even beyond the Covid-19 pandemic, mobile phone platforms could be a useful learning modality. A recent review of EdTech in Bangladesh (⇡Zubairi et al. 2021) identified that the use of technology would be a prime catalyst in the coming years to promote access to education and improved quality of learning. Before Covid-19, around three million learners aged between 6 and 14 remained out of school (⇡World Bank, 2018, p. 28). This situation was likely to be worsened due to the pandemic (⇡Rahman & Sharma, 2021). Moreover, those who had already been enrolled in school suffered a significant learning loss due to the long school closures . Therefore, the government also prioritised EdTech to provide blended learning (⇡Zubairi et al., 2021). In this context, mobile phones could play an important role to supplement the EdTech ecosystem.
Similar to Bangladesh, the penetration rate of the feature phone was significantly higher than other platforms in most low-and middle-income countries around the world (⇡Carvalho & Crawfurd, 2020). During the Covid-19 pandemic, the use of feature phones for educational purposes increased significantly. However, phones were typically used for brief one-to-one calls with caregivers to follow up on their children's homework ⇡Crawfurd et al., 2021;⇡Hassan et al., 2021;⇡Schueler & Rodriguez-Segura, 2021). Though these interventions provided personalised learning and teaching at the level of learners, the 'one-to-one call approach' was not very scalable due to its high reliance on human resources.
However, by using Interactive Voice Response (IVR) , it is possible to provide 3 education at scale while maintaining some flexibility with regard to learning levels. Lessons of different proficiency levels can be stored in a telecom server and learners can learn by choosing lessons at their competence level. By and large, learning via feature phones could promote access to education and provide learning at the right level to millions of marginalised people in addition to classroom learning.

Purpose
In this study, we developed a set of pre-recorded audio lessons and stored them in a data centre connected with a telecom server, and then delivered them through IVR. Caregivers could access these audio lessons anytime for free by calling from their feature phones and navigating them through IVR. The purpose of this study was to assess the learning efficacy of audio lessons delivered via IVR using feature phones. To evaluate the learning efficacy, we focused on learning outcomes for primary school-aged children, in areas ranging from language, literacy, and numeracy competency to leadership, communication, and planning skills.
The first objective of this study was to estimate what effect the intervention had on the learning outcomes, Equivalent Years of Schooling (EYOS) and Learning Adjusted Years of Schooling (LAYS). The second objective was to estimate the treatment effect of the intervention by various subgroups. In other words, the focus of this study was to determine how equitable the intervention was for different student populations. The third and final objective of this study was to assess the cost-effectiveness of an IVR-based educational intervention.

Context
In partnership with the Global Development and Research Initiative (GDRI)a local non-governmental organisation (NGO) in Bangladesh, , we worked with 1,763 primary school-aged children and their caregivers in 90 villages in the 4 southwestern districts (Khulna and Satkhira). GDRI had already worked with 7,500+ children in 223 villages from these areas before the Covid-19 pandemic. 5 We therefore had access to household contact information and pre-pandemic learning levels of the children, which was crucial for understanding the marginal effect of this study's current intervention.
As of January 1, 2021, the average age of the children we worked with was 7.4 years and the age range was 5.1 to 9.9 years. These children were from households with a low socio-economic status. The average monthly income per household was BDT 11,008 (USD 130.8) during 2021 whereas the national average monthly income in rural areas was BDT 13,998 (USD 166.3) back in 2016 (⇡BBS, 2017). In terms of access to distance learning modalities, these 6 households were also quite disadvantaged. During May 2021, only 46.5% of them had access to TV, less than 1% had access to computers or radio. All of them had access to mobile phones, 35.6% of which were smartphones. However, these smartphones were rarely used for educational purposes due to high data costs and low internet speed, particularly in rural areas. The 7 caregivers of these children also had low levels of education; the average years of schooling for the fathers and mothers were 5.9 and 7.1 years, respectively. Overall, based on the availability of Information and Communication Technologies (ICT) resources and lower caregiver capacity, these children were largely excluded from available distance learning modalities and received poor homeschooling support.
7 Bangladesh ranks 136 among 141 countries based on average internet speed (⇡Speedtest, 2021). 6 Bangladeshi Taka -currency of Bangladesh. As of December 1, 2021, 1 USD = 1 84.1872 x. 5 These children were part of a completed project named Investing in our future by GDRI. Retrieved on January 20, 2022: https://www.theimpactinitiative.net/project/investing-our-future-early-childhood-interventionand-parental-involvement-bangladesh In this context, feature phones could provide access to distance learning and could assist caregivers to engage in their children's education. However, content that could be broadcast via feature phones were limited to audio or short messages. Therefore, to improve the efficiency of audio content, we incorporated Interactive Radio Instruction (IRI) to turn a one-way technology into a tool for active learning. IRI requires learners to stop and react to questions and exercises through verbal responses to radio characters, group work, and physical and intellectual activities while the programme is 'on the air ' (⇡Bosch et al., 2002). Though this method was originally developed for radio platforms, we adopted it for IVR.

Research questions
We implemented the intervention using a multi-arm clustered RCT design to answer the following research questions: ■ Can IRI-based audio lessons, delivered through feature phones, lead to better learning outcomes for primary graders?
■ Can IRI-based audio lessons lead to better noncognitive outcomes for primary graders in the areas of leadership, communication, and planning skills?
■ Can IRI-based audio lessons increase students' efforts and caregiver involvement in children's educational activities?

What this paper adds to the knowledge base
Some studies have shown that educational interventions delivered through feature phones (e.g., SMS, one-on-one phone calls etc.) during the Covid-19 pandemic were highly effective for student learning (⇡Hassan et al., 2021). However, simple SMS-based educational interventions cannot cover many aspects of learning while one-on-one phone-based tutoring is not easily scalable as it requires a significant number of resources. More importantly, the aspect of flexible study hours is absent from traditional feature phone-based educational programmes. Building on previous interventions, this study adds to the existing knowledge base in two important ways. First, it demonstrates that distance learning can be flexibly delivered and comprehensively accessible via feature phones with the help of IVR. Second, it provides some of the first evidence of the efficacy of the interactive lessons delivered via IVR using feature phones in out-of-school and household settings.

Implications for policy and practice
Over the last decade or so, Bangladesh has made a significant advancement in expanding access to education. Approximately 97% of school-age children are enrolled in primary school (⇡MoPME, 2019). However, the lack of learning at school remains an area of concern. LAYS estimates for Bangladeshi children indicate that a learner receives an equivalent of circa six years of learning after attending school from the age of 4 to 18 (⇡Zubairi et al., 2021), which is projected to fall to 5.1 years of learning due to the Covid-19 pandemic (⇡Rahman & Sharma, 2021). For this reason, Bangladesh's policymakers have continued to focus on increased access to and the quality of education (⇡Zubairi et al., 2021).
According to Bangladesh's Education Sector Plan 2020, there are two major reform priorities, namely "access and equity" and "quality and relevance" (⇡MoE, 2020). The government aims to provide compulsory education for all and to ensure grade-specific learning for students irrespective of gender, age, religion, ethnicity, and disadvantaged groups. EdTech has the potential to be a catalyst in achieving these goals, which have been under sharp focus due to the Covid-19 pandemic (⇡Zubairi et al., 2021).
Although prolonged school closure and learning interruption are unique to the Covid-19 pandemic, educational disruptions at a smaller scale are prevalent in low-and middle-income countries. In Bangladesh, natural and human-induced events (e.g., cyclones, floods, political unrest, etc.) damage educational infrastructure and limit school operations. Thus, policymakers may consider expanding education delivery using accessible technology in out-of-school settings to extend access to education and better support learning for children.
This study proposes a distance learning modality based on the feature phone, which is the most available household ICT resource across different strata of society. The findings of this research can assist policymakers to formulate policies that are appropriate for children who miss out on education due to poor access and who remain behind their grade-level competencies. Disseminating findings of this research in academic and non-academic conferences around the world could encourage policymakers, practitioners, and other researchers to replicate similar interventions in other settings and provide more insights for making informed decisions.

Structure
This paper is organised as follows. Section 2 is a brief literature review. Section 3 sets out the research methodology in detail. The results of the study Delivering Remote Learning Using a Low-Tech Solution are presented in Section 4. Finally, this study concludes with policy implications and conclusions in Section 5 and Section 6, respectively.
Delivering Remote Learning Using a Low-Tech Solution

Literature Review
This section provides a brief review of the literature on feature phone-based educational solutions, including programmes based on IRI. Section 2.1 reviews selected research exploring the effectiveness of educational interventions delivered via feature phones. Section 2.2 discussed the advantages and effectiveness of IVR-based educational intervention. Section 2.3 sets out the IRI method, its effectiveness, and the opportunities for integration with IVR-based educational interventions.

Phone-based distance learning solutions
At the peak of the Covid-19 pandemic, nearly 1.6 billion children around the world missed out on schooling (⇡UNESCO, 2021 During the Covid-19 pandemic, several studies were conducted on phone-based educational interventions focusing on primary graders and / or their caregivers. ⇡Angrist et al. (2020) conducted a randomised experiment of a feature-phone-based intervention in Botswana. There were two treatments in the intervention. The first (a) involved one-way bulk SMS texts and the second, (b) used SMS bulk texts with a live phone call for 15-20-minutes. Children who participated in the intervention were from Grades 3-5 and the programme ran for 12 weeks. The combined treatment (treatment b) improved learning by 0.12 SD, while the text messages alone (treatment a) had no effect. The intervention was also very cost-effective, USD 5 per child for treatment (a) and USD 19 per child for treatment (b). For treatment (b) this equates to 0.63 SD per USD 100 of spending.
Moving away from teachers-calls-based interventions, ⇡Hassan et al. (2021) studied the efficacy of an intervention where university student volunteers provided over-the-phone mentoring and homeschooling support to primary schoolers in Bangladesh. Their intervention improved the learning outcomes of treated children by 0.75 SD and increased the homeschooling involvement of treated mothers by 0.64 SD. Due to volunteer participation, the cost of this programme was very low -each dollar on over-the-phone mentoring led to an average 0.03 SD improvement in learning outcomes.
In contrast to these effective interventions, some studies reported null effects of feature-phone-based educational interventions during the Covid-19 pandemic. ⇡Crawfurd et al. (2021) evaluated the effectiveness of live tutoring via feature phone calls from teachers using an RCT with 4,399 primary school students in Sierra Leone. Though the intervention led to increased student engagement in educational activity, there was no treatment effect on mathematics or language test scores. Further, the study found no effect on school re-enrollment rates among the children of the different treatment groups when schools reopened.
A similar null effect of teacher-student phone call interventions in Kenya is also reported by ⇡Schueler & Rodriguez-Segura (2021). Students were randomly assigned to receive a 7-week, weekly maths-focused intervention or to the control group. The treatment had two versions -5-minute accountability check calls and 15-minute tutoring calls. The intervention increased student perceptions about teacher interest in learners' welfare. However, the accountability checks treatment did not affect maths performance while tutoring treatment decreased maths achievement among students who returned to their schools after reopening.
It is evident from the literature that phone-based educational interventions during the Covid-19 pandemic were largely effective. Only a few studies reported null effects. There were several factors behind these null results. The first was pre-existing factors, such as students' prior knowledge. ⇡Schueler & Rodriguez-Segura (2021) found that when the schools were reopened, low-achieving students from their sample were least likely to return and take in-person assessments. This meant that outcomes of the treatment group that was measured at the school level were poor. ⇡Hassan et al. (2021) also reported that academically weaker students got the most out of their intervention, which led to the high overall treatment effect. These findings indicated that marginal effects of brief educational interventions via feature phones were higher for the most deprived group of students compared to their less-deprived counterparts.
The second important factor was the motivation of the caregivers and children. There was an important difference between studies with positive effects (e.g., ⇡Hassan et al. (2021) in Bangladesh and ⇡Angrist et al. (2020)in Botswana) and studies with null effects (e.g., ⇡Crawfurd et al. (2021)) in Sierra Leone and ⇡Schueler & Rodriguez-Segura (2021) in Kenya). The first two studies randomised the treatment among the sub-set of families who opted into the intervention, whereas the latter two studies worked with all the children enrolled in the relevant schools and grades. Those caregivers who opted for participation might have been more interested in using the intervention, following up and engaging more with their children's education. Though this hypothesis has not been tested as such, previous evidence suggests that basic accountability checks could encourage educational engagement (⇡Gill, 2020;⇡Gill et al., 2016), which might be higher in the case of motivated caregivers.
The third crucial aspect for the effectiveness of the phone-based educational intervention was caregiver engagement. ⇡Hassan et al. (2021) incorporated mothers in their education delivery model so that students could be monitored and followed up, which was difficult via feature phones. It was evident that about 15-20% of treatment effects attributed to the mothers' added engagement. The study also reported that the added burden of homeschooling didn't affect the income level and mental health of the mother. This suggests that mothers can be engaged in homeschooling without sacrificing economic value. Similarly, ⇡Lawrence & Fakuade (2021) found that caregiver involvement in children's learning played a crucial positive role in the learners' commitment towards distance learning during the Covid-19 pandemic in Nigeria. Overall, these three factors i.e., targeting an appropriate group of students, motivated caregiver-child dyads, and caregiver engagement were very important contributory factors in the effectiveness of any feature-phone-based educational intervention and informed the intervention design of this study.

IVR-based education
The aforementioned studies evaluated either one-on-one phone calls or SMS reliant educational interventions delivered via feature phones. However, simple SMS-based educational interventions could not cover many aspects of learning while one-on-one phone-based tutoring was not easily scalable as it requires a high level of human resources. For this reason, IVR-based educational interventions have gained greater attention in recent years.
IVR is an automated phone system technology that allows incoming or receiving callers to access information by traversing or navigating a pre-designed flow. Navigation to different points of the flow can be done by either voice command or keypad selection by the caller. Once the caller makes a call to or receives a call from an IVR-enabled number, there is no need for any human agent. A pre-recorded message can guide them to the desired landing node with preferred information. ⇡Hassan et al. (2021) found that volunteer mentors sometimes found it difficult to schedule a time for conducting telementoring sessions with the mothers. This was because mothers were often busy with household chores and/or there was only one phone in the household, and this was usually with the child's father who worked outside the home during the mentor's preferred times for mentoring. As IVR could support automated calls, it could solve these issues by providing flexible study hours that were not supported in one-on-one tutoring.
Though both radio and IVR platforms only support audio lessons, IVR has an important advantage over radio in that listeners can choose or select lessons i.e. there is no fixed broadcasting sequence. Usually, students' abilities vary significantly even if they have studied in the same grade (⇡Islam & Ruthbah, 2020) and they could learn better if they were provided with educational content that matches their capability ( To date, there is only one study available that largely matches this studydialling up learning by ⇡Afoakwah et al. (2021) in Ghana (hereafter Ghana Study). Unlike this study, the Ghana Study treated both students and teachers, separately. Randomly selected 719 students in Grades 4, 5 and 6 received daily audio lessons that focused on foundational numeracy skills and 160 teachers received weekly professional development sessions focused on the instruction of foundational reading. Though the intervention was effective for teachers, there was no effect on the students' numeracy skills.
There are three important differences between the study by ⇡Afoakwah et al.
(2021) and this study. First, the randomisation unit -the Ghana Study randomised at the school level and used the intervention with all students at the treated schools, whereas this study randomised at village level and the interventions were only used with those participants who opted into the programme. This difference might be crucial in explaining the null effect of the Ghana Study. As discussed in the previous section, the motivation of the caregiver-child dyad was important to make a low-tech educational intervention effective. Overall, based on limited studies, the effectiveness of IVR-based education is still unknown. However, positive effects of IVR-based interventions found in other fields of study and the wide accessibility of this technology suggests that IVR has the potential to support large-scale remote learning needs. Further research on designing IVR-based educational interventions and appropriate participant selection is needed.

Interactive Radio Instruction (IRI)
As mentioned in the previous two sections, the IVR system has limited capacity in providing lessons. It can only support pre-recorded audio lessons.
To maximise the efficacy of such audio lessons, a half-a-century-old method could be crucial -IRI. 8 The original design of IRI was created back in 1970 to teach mathematics in Nicaragua (⇡Bosch, 1997). IRI is an instructional approach that turns a one-way technology into a tool for active learning. It requires learners to stop and react to questions and exercises through verbal response, to engage in group work, and physical and intellectual activities while the programme is on the air (⇡Bosch et al., 2002). Facilitators play an important role in IRI in supervising the progression of the lessons (⇡Ho & Thukral, 2009 However, little is known about the effectiveness of IRI for out-of-school learners. The only other study that evaluated the integration of IRI into IVR-based educational interventions in household settings is the Ghana Study.
As mentioned earlier, mixed findings reported by that study and some implementation issues limit our ability to make conclusions about the effectiveness of IRI in supporting out-of-school children.
Building on the existing evidence of phone-based learning, IVR-based educational interventions and the effectiveness of IRI in active learning, this study explores the effectiveness of interactive audio content delivered via IVR to improve the learnings of primary-grade students.

Methodology
This section presents the methodology of this study. Section 3.1 describes the intervention design, which builds on existing evidence. This section also explains the RCT design, sample, outcome variables, and hypothesis of this research. Section 3.2 describes the research instruments used. The procedure used for impact evaluation is explained in Section 3.3. Various stakeholders related to this study are mentioned in Section 3.4. Ethical considerations are explained in Section 3.5. This section concludes with Section 3.6 where the challenges and limitations of this study are discussed.

The intervention
As outlined in Section 2, for this study, we chose IVR on feature phones because it was possible to provide flexible study hours and an appropriate level of learning to the learners. We also incorporated IRI to improve the efficacy of audio lessons. For this intervention, we established two IVR-enabled toll-free numbers to deliver audio lessons to primary graders. Figure A1 and Figure A2 in the Annex portray the IVR journey of the participating caregiver-child dyads. This intervention covered three modules divided into 75 lessons, with each lesson lasting between 16 and 18 minutes. These modules focused on numeracy, literacy, and noncognitive skills. The contents of these modules are listed in Table 2. We designed these modules exclusively for this programme with the support of two other international organisations and a 10 group of local curriculum experts. Overall, these modules were developed to supplement the national curriculum and support learning in out-of-school settings.
The audio lessons used in this intervention were pre-recorded conversations among four characters: two teachers and two students. Following IRI methodology, during the conversation, students are asked to do some activities according to the teachers' instructions, such as clapping, standing up, counting, making plans, etc. The teachers used regular pauses, cues as well as playing music and songs during the recorded lessons so that the listeners could complete similar tasks with the help of their mothers or any other adult members of the household. We called this learning assistant the 'special helper' in our programme. Typically, mothers (in 77.8% of cases) and elder siblings (in 14.6% of cases) were the 'special helpers'. Usually, it was the special helpers' responsibility to check whether or not their child followed the instructions. These instructions were easy to follow for any adult and did not require high levels of literacy and numeracy. As schools were closed and social interactions with other children were limited, we, specifically, offered a noncognitive skill module that provided scope to engage in various non-academic activities with the 'special helper'. Those activities covered leadership, communication, planning, patience, empathy, sympathy, compassion, and perseverance. We examined whether these discussions and related activities affected noncognitive abilities such as leadership, planning, and communication skills of children within the treatment group. This educational intervention ran for 15 weeks. At the beginning of the programme, mothers or guardians of treated children were briefed about how to access lessons and were provided with a short guidebook that explained the programme. Figure 1 displays the timeline of the project.  In this study, we modified the original design of IRI in two ways. First, we used IVR to deliver the lessons. There were three main reasons for choosing this modality. The first was related to access: only 49% of rural school-age children have access to TV and only 0.5% own and listen to the radio in Bangladesh (⇡Rahman & Sharma, 2021). On the other hand, 96% have access to feature phones. We took advantage of this high penetration of mobile phones to offer pre-recorded interactive lessons. The second was tracing. It is almost impossible to trace who is listening to a radio programme or watching television and who is not, which is problematic for a rigorous impact study. However, in the case of an IVR-based programme, it is possible to identify who is listening by their mobile number. The third was flexibility. Unlike television and radio, IVR-based lessons can be accessed at any time during the programme period, allowing children to learn at their convenience. This flexible delivery method addresses the resource constraints rural households typically face, such as having only one phone in each household, and unstable mobile networks and electricity supply. Moreover, caregivers could schedule their time for children's homeschooling with greater ease.
Second, we relied heavily on the 'special helper'. Special helpers usually made the call to the programme's number at their convenience and selected the desired lesson by traversing the IVR. Once the lesson was selected, ideally, the child and special helper listened to the lesson together. During the lesson, there were different instructions for the child to follow and activities for the child to do with the special helper. It is worth mentioning that none of these activities required a high level of competence.

The RCT design
We implemented this programme in a three-arm clustered RCT design in 90 villages. In treatment 1 (T1): the Standard group, we offered a literacy and numeracy module in 30 villages; in treatment 2 (T2): the Extended group, we added the 'noncognitive skill' module in addition to the literacy and numeracy module in another 30 villages; and in treatment 3 (T3): the Control group, we offered nothing to the remaining 30 villages. These treatment arms are exhibited in Figure 2.
We allocated four lessons (two literacy and two numeracy) per week for the T1 group; and five lessons (two literacy, two numeracy and one noncognitive skill) per week for the T2 group. However, caregivers could choose and access any lesson at any time for their child during the programme period. We did not mandate any fixed sequence in our curriculum, i.e., if any learner found any lesson easy, they could skip the lesson and move to the next one. There was a reason for this design. We offered this intervention to students of different ability levels as they were studying in different grades, i.e., Grades 1, 2, 3 and 4. Even in the same grade, students' abilities vary significantly (⇡Islam & Ruthbah, 2020). Students usually learn better if they are provided with educational content that matches their level (⇡Banerjee et al., 2007). Thus, by offering content of different grade levels to all children, we were in effect offering a Delivering Remote Learning Using a Low-Tech Solution menu of choices so that children could adjust their learning according to their competencies. In this way, the programme has elements of the 'Learning at the Right Level' (LARL) approach, just like the Teaching at the Right Level approach (⇡Banerjee et al., 2016). We believe that this flexibility is more helpful for children, particularly for those, who are at the bottom of the test-score distribution. T1: Standard -Children in this group received 60 lessons over 15 weeks. They listened to the lessons and engaged in the activities according to the instructions. The 'special helper' helped them to follow these instructions and took part in various activities. Two modules were covered in this treatment arm: literacy and numeracy. After each lesson, a quiz was played to the listener, and the answers were recorded in the IVR flow. Fifteen listeners were randomly selected every week from the pool of listeners who had given correct answers during that week to receive a small sum of money (USD 3) via mobile financial services (MFS) as a prize. The reason for this quiz and the reward was to motivate listeners to complete the lesson. T2: Extended -In addition to the literacy and numeracy modules that the T1 group received, the T2 group received an additional module that contained a range of noncognitive skills training. We focused on leadership, communication, and planning in this module. These topics were introduced to children and they heard ideas about how they could develop these qualities through different examples and tasks. The T2 group also participated in a post-lesson quiz and 15 of those who correctly answered the quiz were randomly selected for a prize. Note that the quiz was for literacy and numeracy only.
T3: Control -There was no intervention for this group, i.e., it was a pure control group.

Baseline sample
Our field partner, GDRI, worked with 7,500+ children in 223 villages from two districts in the southwest region of Bangladesh before the Covid-19 pandemic. From this list, we, first, randomly selected 90 villages and then 3,000 households with mobile phone numbers. Figure A3 in the Annex illustrates the study area and sample villages of this study. We were able to reach and complete a baseline survey for 2,400 children from 2,387 households. Others did not respond, or the phone was not active, invalid, or switched off or they did not show interest in this intervention. We randomly selected about 16-22 children from each of these 90 villages and randomly assigned 90 villages 11 each to one of three treatment arms i.e., Standard, Extended, or Control. Our final sample size was 1,763 children from 1,755 households at the baseline. See Figure 2 for details of the sample size in each treatment arm. After randomisation, we checked for inter-cluster balance in several socio-economic characteristics and children's assessment scores. These results are shown in the Annex Table A1. These characteristics were balanced across the treatment and control groups.

Outcome variables
There were four groups of outcomes in this study. Different variables were constructed as follows to explore the impact of this intervention.
1. Learning outcomes Children's learning outcomes were measured using an assessment that comprises 19 questions from English literacy, Bangla literacy, numeracy, and general knowledge. All questions were taken from the national curriculum of Bangladesh. The test totals 100 points. We developed three sets of questions as we were covering students from different grades. The answers were deliberately in binary form to avoid assessment bias. To examine the effect of the treatment (IRI-based audio lessons) on cognitive ability, we constructed a standardised test score. First, we normalised the raw test score by subtracting the mean test score of the control group from it and then divided it by the standard deviation of the test score in the control sample. The questions of the assessment test are listed in the Annex in Table A2. 2. Noncognitive outcomes We used Scales for Rating the Behavioural Characteristics of Superior Students developed by ⇡Renzulli et al. (2002) to measure the noncognitive outcomes of this intervention. This scale has 14 subscales. We used leadership, communication, and planning subscales, given that our modules focused on these dimensions. The scale developed by Renzulli et al. was designed to be answered by teachers. However, in our study, the mothers answered it as we were providing an educational intervention in the household environment. We constructed three outcome variables for three characteristics: leadership, communication, and planning. Like the learning outcome variable, we also standardised these variables. Items of these three characteristics are listed in the Annex in Table A3. 3. Homeschooling We had two categories for homeschooling -students' effort and caregivers' time in education. We constructed two variables as follows to understand the effect of the intervention on homeschooling. a. Student's effort: We constructed an index variable from the following four questions by factor analysis to identify students' effort in homeschooling. These questions were: i. How much time does your child spend studying? (in daily minutes) ii. How much time does your child spend studying? (5-point scale) iii. How interested is your child in studying? (5-point Likert scale) iv. How attentive is your child in studying? (5-point Likert scale) b. Caregivers' time in education: We constructed an index variable from the following two questions by factor analysis to identify caregiver involvement in children's education. i. In the past weeks, how much time have you given to help your child to study? (in daily minutes) ii. In the past weeks, how much time have you given to help your child to study? (5-point scale)

Hypotheses
The intervention was hypothesised to:

Instruments
Children and their caregivers were surveyed in 2019 as part of a long-term follow up of another intervention on early childhood. Various information about demographics, income, employment status, household asset composition, livelihood, and caregiver involvement, etc. were collected in these surveys. These children were also assessed in 2019. The assessment tests comprised various domains, such as language, literacy, numeracy, fine and gross motor skills, problem-solving, personal-social domain, working memory, self-regulation. For this project, we conducted a face-to-face survey in May and June 2021 to update information that had been previously collected. We also collected information about the children's educational situation and the households' private educational investment during the Covid-19 pandemic. We used this extensive set of data to check that the characteristics of the treatment and control groups were balanced.
During the intervention, we collected IVR-flow usage data, i.e., duration of lesson-play by each participant, access time and date, frequency, etc. from the server. This data was used mainly to understand the performance of the IVR channels, and the difficulties caregivers faced in using the technology.
Because of the Covid-19 related mobility restrictions during the project, we hired a project assistant in each village as our local contact, so that they could communicate with caregivers and children if there was any difficulty or problem in accessing the lessons via the programme numbers. These project assistants had worked with GDRI before and were familiar with communicating between GDRI and caregivers. We deliberately kept their role as minimal as possible. We collected users' usage information from these village informants, for example, the mobile number a household used in case of a number change, the number of lessons completed by each child, etc.
This intervention ended in October 2021. The endline data was collected in October and November of the same year. A team of two members -one assessor and one enumerator -visited each household. The assessor conducted the assessment test with the child while the enumerator conducted the caregiver survey with the mother. The assessment test and survey were conducted in a one-on-one format by maintaining physical distance as per Covid-19 safety protocols.

Estimation
To answer the research questions given in Section 1.4, we estimated the following Ordinary Least Square (OLS) regression specification: [1] = α + β 1 1 + β 2 2 + ' θ + ε where was the outcome of an individual (child or caregiver) from household measured at the endline; and were the treatment indicators for the 1 2 Standard and Extended groups, respectively; and was a vector of individual and household-specific characteristics. We clustered the standard errors at the village level. In equation 1, and provided estimates for the intention to β 1 β 2 treat (ITT) effect -an average of the causal effects of receiving treatment on the outcome variables.
We also conducted a subgroup analysis, to explore the differential treatment effect on the children of different strata of the sample. We used i) gender of the children, ii) baseline literacy and numeracy score, iii) caregiver education, and iv) household income to stratify the sample.

Stakeholders
We collaborated with a local research-focused NGO, GDRI, to implement this intervention. From 2015 to 2020, GDRI completed a series of large-scale surveys and assessments of 7,500+ children in 223 villages in the south-western part of Bangladesh. We leveraged the sample and rich data set to implement this research. In addition, we collaborated with the local education specialists and international NGOs, Rising Academies and Lead 12 Education, to develop the audio lessons.

Ethical considerations
Ethical clearance for this study was obtained from the Monash University Human Research Ethics Committee. Consent of the caregivers and children were obtained before participation in the programme. Participation in the programme was voluntary and participants had the option to withdraw at any point of the project.

Challenges and limitations
This intervention was implemented during the Covid-19 pandemic to counteract the educational adversities created by the lockdown imposed by the government. The pandemic also posed significant challenges in implementing and evaluating this study. We had to limit the mobility of personnel involved in the project and face-to-face interactions between survey enumerators and participants. It therefore took more time than expected to complete the stages required for a rigorous impact evaluation. One notable limitation of this study is the assessment test. We could not test a wider range of outcomes because we could not place several children in the same classroom to be assessed simultaneously. We had to restrict the assessment to one-on-one examination while keeping the assessment time short. Thus, we did not measure one important aspect of learning -writing skills. However, we believe that writing skills and other domains of learning that we measured were positively correlated.

Results
This section presents the results of the study. Section 4.1 describes data used in this study, where sample characteristics and inter-cluster balances are exhibited. Section 4.2 provides service (programme) usage statistics i.e., hours of lessons delivered via the IVR platform. Section 4.3 presents the overall results of this intervention's effect. Further insights into the results are provided in Section 4.4, where findings are categorised and explored based on outcome groups. Section 4.5 presents findings for subgroups of key characteristics, i.e., gender, baseline academic performance quartile, caregivers' income, and education quartiles. These classifications allow readers to understand the differential treatment effects across the subgroups. In Section 4.6, the effects of this intervention are expressed in terms of Learning-Adjusted Years of Schooling (LAYS). This estimate will help policymakers and practitioners compare the treatment effect on learning with other alternative educational interventions. The final section, 4.7, illustrates the cost-effectiveness of this intervention.

Data
We relied entirely on primary data collected through our field partner, GDRI. In the baseline, 1,763 children from 90 villages were included in the survey. After the 15-week-long intervention, we surveyed the households and children again. For the endline, we reached 1,687 households for the survey and assessments, which is 95.7% of the sample. The rest of the households had either relocated permanently or temporarily or did not want to participate in the endline survey. 14 Table 3 depicts the characteristics of 1,687 children who were surveyed at both baseline and endline. The corresponding table for the full sample of 1,763 children is presented in Table A1 of the Annex. The average age of the participants was 7.4 years as of January 1, 2021, and the age range was 5.1 to 9.9 years. Around 51.3% of the children were girls. The mean years of schooling of their father and mother were 5.9 and 7.1 years, respectively. These households were mostly from a low socio-economic status with an average monthly income of BDT 11,008 (USD 130.8) and an average homestead size of 9.9 decimals during the baseline survey in May 2021. Note that the national 15 15 100 Decimal = 1 Acre of land.
14 We check for differential attrition by treatment and find no such evidence. None of the treatments and interaction terms is significant when regressed with attrition dummy. average monthly income in rural areas was BDT 13,998 (USD 166.3) back in 2016 (⇡BBS, 2017). Besides the full sample's characteristics, Table 3 also indicates the balance among the treatment arms of this study. As we implemented this intervention in a 3-arm clustered RCT design, having intercluster balance was particularly important for the identification of the actual treatment effect. First, Columns 2, 3 and 4 of Table 3 show the characteristics of the sample of different treatment arms, i.e., the Standard, Extended and Control groups, respectively. These characteristics are mostly similar. Second, we have estimated an OLS regression to check whether baseline characteristics differ across the treatment groups. P-values from the joint tests of significance, given in Column 5, indicate that we cannot reject the null hypotheses, i.e., there is no systematic difference among the treatment arms. Most importantly, this means any effect resulting from the treatment status can be attributed to the treatment only.

Usage information
According to a recent review by ⇡Kaye et al. (2020), there are four important considerations in any distance education intervention. These are availability, access, usage, and learning. We have discussed the availability and access of our distance learning intervention in the previous sections. In this section, we present the usage statistics of this intervention. The final component, learning, is discussed in the following sections.
Providing access to distance learning is not the end of the process. Even with access to distance learning, students may not use the resources for many reasons. During the Covid-19 pandemic, ⇡Biswas et al. (2020) found that only 43% of students who had access to TV were watching TV lessons. Unlike TVand radio-based distance learning, IVR-based services can be monitored constantly as we did in our programme. We also sent regular SMSs to nudge participants to ensure regular participation with the platform. In addition to these, our village representative called caregivers regularly if caregivers were not phoning in regularly or were facing a problem.
We found that participants of our intervention listened to 8593.45 hours of lessons, which is 64.6% of the expected volume. On average, each student listened to 7.27 hours of recorded lessons, which was 43.6 minutes of listening per week. The audio content of this intervention was approximately 90 minutes per week. We had assumed a 25% friction in delivering these audio lessons due to network issues, partial participation of some households etc. We also expected that not every student would listen to the entire duration of all lessons, as some lessons might be easy for some students while others might not be as interesting as others.

Overall results
We present the overall results of this study in the following Table 4, where treatment effects and other details of the impact estimation procedure are given, and Figure 3, which presents a snapshot of all treatment effects standardised for the control group. the Ordinary Least Squares (OLS) regressions. Dependent (outcome) variables of these regressions are stated in the column headings. The structure of these variables is explained in Section 3.1.4. We used a set of covariates in these regressions to control for various individual and household-level characteristics. We also included grade fixed effects (FE) in all regressions. Moreover, we included enumerator fixed effects to control for enumerator effect, if any, on the survey responses. Finally, we used the social desirability scale to control for the potential experimenter demand effect, i.e., participation in the treatment might encourage the respondents to answer or choose socially desirable responses to our survey questions without changing their actual views or behaviour.  (1982)). e. Robust standard errors clustered at the village level are in parentheses. f. *** p<0.01, ** p<0.05, * p<0.1 Our estimates show positive and statistically significant effects of both treatments on improving learning outcomes among participants (all p-values < 0.01 in Column 1, Table 4). As mentioned earlier, we define 'learning outcome' as total points achieved in the endline assessment test, which comprised numeracy, English literacy, Bangla literacy, and general knowledge subjects. We also found improvements in homeschooling due to the programme participation (Columns 5 and 6, Table 4). However, we did not find statistically robust effects of the intervention in noncognitive domains (Columns 2-4, Table 4). Only leadership skills improved for the children in the Extended group. Note that null effects in noncognitive domains for the Standard group were expected as children in this group did not receive the noncognitive skill module.    Table 4. The vertical line at 0 represents the null effect of the control group children as they did not receive any intervention. All horizontal lines indicate coefficients with 90 and 95% confidence intervals. Again, treatment effects in the learning domains are large and robust.

Learning outcomes
As mentioned in the previous section, treatment effects on learning outcomes were significant and robust. In this section, we examine the learning gains in more detail. Figure 4 below, presents the test score distributions of treatment and control groups. For ease of understanding, we merged both treatment groups into one group. The test scores of the treated children were distributed more to the right of the control children, meaning that the intervention lifted the children's test scores to the upper level of the distribution. If we investigate the source of this improvement, we see that English literacy and numeracy domains are the main drivers. Figure 5 shows subject-wise test scores of the children from different treatment arms. To make these scores comparable, we converted the score of each subject to 100. The largest gain in test scores occurs in English literacy, followed by numeracy. It is also evident Delivering Remote Learning Using a Low-Tech Solution that all groups of children perform the worst in English. Understandably, rural caregivers with low levels of education may find it difficult to teach English literacy to their children. This intervention enabled them to support their children in learning English and led to the highest gain.

Noncognitive outcomes
In this intervention, T2, the Extended group, received one additional module that focused on various noncognitive skills, such as leadership, communication, and planning. We hypothesised that children in the Extended group would demonstrate better noncognitive skills compared to children in the Standard and Control groups. However, our estimates did not fully support this hypothesis. Children in the Extended group showed better skills in the leadership domain, but the magnitude of this effect was small and its statistical significance diminished once we corrected the p-values using the Westfall-Young (WY) adjustment (⇡Westfall & Young, 1993). See Table 4 for the estimated p-values. We also computed and reported p-values from a permutation test at the village level by randomly shuffling the treatment status 1,000 times (⇡Young, 2019). The randomised inference (RI) p-values indicate that there were no treatment effects on the noncognitive domains.
Overall, although the treatment effects on leadership, communication, and planning skills were positive, they were not statistically significant.

Caregiver and student engagement
In Columns 5 and 6 of Table 4 and Panel C of Figure 3, we report the treatment effects on homeschooling. We split homeschooling into two components: students' effort and caregiver time. The formation of these two variables was discussed in Section 3.1.4. The results suggest that the treatments increased students' time investment in education, interest in study, and attention span. The effects were similar in both treatment groups. Students of the Standard group spent 5.1 minutes more per day on education-related tasks compared to the Control group. By contrast, students of the Extended group spent 4.5 minutes more time. In terms of daily average study time, students of the Standard, Extended, and Control groups spent 92.1 minutes, 91.5 minutes, and 87.0 minutes per day on education-related tasks, respectively. In the absence of school attendance, this intervention brought education-related routines into the day-to-day lives of the students and resulted in increased student effort.
Caregiver time investment in education also increased due to programme participation. We found that caregivers who participated in the intervention 16 devoted more time to their children's education-related tasks compared to other caregivers in the control group -10.5 minutes per day in the Standard group and 9.7 minutes per day in the Extended group. In terms of daily average caregiver time in educational activities, caregivers of the Standard, Extended, and Control groups spent 86.0 minutes, 85.2 minutes, and 75.5 minutes per day on their children's education, respectively. Usually, one in two caregivers in Bangladesh find it difficult to teach new things from the curriculum to their children (⇡Biswas et al., 2020). Moreover, caregivers' low level of education also made them hesitant about their capacity to provide homeschooling. In these circumstances, the IVR-based education programme enabled caregivers to engage more in their children's education, even outside of the direct programme time.

Treatment effects by subgroups
In this section, we present the main impact of this intervention grouped by child-and household-specific characteristics. Figure 6 depicts the learning outcomes of the children in various treatment arms grouped by gender, baseline learning level, caregivers' income, and caregivers' education.
The top-left subfigure of Figure 6 indicates that girls in both the Standard and Extended treatment groups performed as well as boys, but girls in the Control group performed slightly worse than boys. Thus, the poorer performance of girls relative to boys in absence of any intervention was overcome by the intervention.
The top-right subfigure of Figure 6 exhibits the mean test scores of children in different treatment groups by quartile of baseline test scores. Treatment effects are the largest for those who had scored the lowest in the baseline. For students in the 1st and 2nd quartiles (poorest), the gaps between treatment and control groups are as large as 40%, whereas, in the other two quartiles, the gaps are less than 30%. All these gaps are statistically significant at the 5% level. It means that the programme particularly helped academically poorer students to catch up on their learning gaps.
The bottom two subfigures of Figure 6 exhibit the total points by family income and caregivers' education. Again, the treatment effects are the largest for children who come from households in lower-income quartiles. The gains in test scores are more than 35% in the 1st and 2nd quartiles, whereas they are only around 20% in the top two quartiles. It indicates that the marginal gain for the poorest strata of the sample is the largest as children in these groups were struggling the most due to their lack of access to alternative schooling during the pandemic-induced school closures. Finally, in the bottom-right panel of Figure 6, it is evident that students with lower caregiver education gained significantly from the intervention. It reinforces the finding that less-educated caregivers have a lower level of confidence, capabilities, and motivation in engaging with children's education, which results in poor performance of their children. The intervention allowed them to overcome their shortcomings and provided more time to their children. This in turn led to comparative better performance. Overall, this intervention was especially beneficial for the female students, academically weaker students, students from the poorest strata, and students with less-educated caregivers.

Learning-Adjusted Years of Schooling (LAYS)
We estimated LAYS derived from the impact of our intervention on learning based on the methodology used by ⇡Angrist et al. (2020) and ⇡Evans & Yuan (2019). First, we derived Equivalent Years of Schooling (EOYS), , from the learning effect of the intervention, which can be expressed as follows: where is the learning gain generated by the intervention expressed in β standard deviations ( ) measured by an assessment test ( ); is the status-quo σ δ learning rate of the control group , again expressed in standard deviations ( ) measured by the assessment test. As we were using the control group of the same study to derive LAYS, this procedure estimated micro-LAYS. Second, we estimated the learning-adjustment factor using the following expression - where is the measure of learning for a cohort of students in county relative δ to the high-performance learning benchmark . We used Harmonised ℎ Learning Outcome (HLO) to estimate the learning-adjustment parameter. The harmonised test score for Bangladesh is 368 on a 625 scale (⇡World Bank, 2021). We considered Singapore as the benchmark country with a harmonised test score of 575 (⇡World Bank, 2021). Third, we assumed that the effect of our intervention would last only for the duration of the intervention, thus, we used a duration adjustment, . A typical school year in Bangladesh is 10 months or approximately 44 weeks. Our intervention lasted for 15 weeks. So, the duration adjustment parameter was 15 divided by 44. Overall, the LAYS estimation procedure used in this study can be expressed as follows: [2] = * ℎ * Delivering Remote Learning Using a Low-Tech Solution By following Equation 2, we estimated LAYS for our intervention, which is presented in Table 5, below. We estimated that LAYS gained by this intervention were 0.59 and 0.64 in the Standard and Extended treatment arms respectively. In addition to absolute LAYS, it is important to consider LAYS per cost unit to understand the cost-effectiveness of the intervention. We estimated that the Standard treatment generated 2.16 LAYS per USD 100, while the Extended treatment generated 2.31 per USD 100. We estimated LAYS to make the findings more comparable. However, these estimates have some limitations that should be considered while making any policy decisions. First, our intervention generated some other outcomes such as increased effort of students and caregiver engagement. These positive outcomes could have an impact on many other aspects of childhood development beyond learning gains.
Second, to translate the effects of the treatments into LAYS, we first estimated EOYS using the control group's learning gain per year, which was assessed by our programme-specific assessment test. But our test did not cover the full set of learnings that usually occur in a typical school year. For example, we did not cover some important aspects of assessment, such as writings, physical activities, and drawings. The incompleteness of the test is not an issue if we compare between treatment and control groups. But our assessment test may not give a fair estimate of EOYS compared to the standard tests given by schools.
Third, there was a cash incentive to the caregivers for programme participation and engagement in this intervention, which was and is not available in normal schools. A total of 30 listeners, 15 from each of the two treatment groups, were randomly selected every week to provide a small amount of money (USD 3) via MFS as a prize for their engagement in the quiz. This prize might have created an additional incentive for the caregivers to facilitate and interact with the children during the lessons. Therefore, direct comparison of this intervention to school learning might produce biased results.
Last but not the least, due to the ICT resource constraints, a significant number of children were missing out on all forms of education during the Covid-19 pandemic. This intervention aimed to solve this problem by providing education through feature phones that were widely accessible. It might not be fair to compare the learning of this marginalised group of students with any other cohort of students just based on the LAYS.

Cost-effectiveness
Our costing analysis revealed that this intervention cost USD 27.5 per student over 15 weeks, of which USD 13.2 was the variable cost and the other USD 14.3 was the fixed cost. Variable costs were voice and SMS charges, household 17 reach-out expenditure, etc., and fixed costs were IVR platform development, content development, programme management expenditure, etc.
We provided this intervention to a total of 1,182 students from two districts. As a result, the fixed cost per student was very high relative to the variable cost. If the intervention had been scaled up, per student cost would have gone down. Figure 7 presents a cost-volume analysis. If this 15-week-project is scaled up to 10,000 students, per student programme costs will go down to USD 16.7 as the fixed cost will not increase proportionately to the rise in student numbers. Further scaling up will push the cost down further. However, there is a limit to these economies of scale. The rate of cost reduction goes down after 10,000 students and only reduces by USD 1.5 if the student number goes up by 10 to 100,000. This is because the main cost driver of this intervention is the call cost or voice charge, which is a variable cost and this does not vary according to the number of students. If this intervention is scaled up in terms of duration, the total cost would be proportional to this 15-week programme. In other words, a year-long programme would cost approximately USD 80 per student because in this scenario both fixed and variable costs would become proportional and more content, IVR platform support etc. would be needed. Overall, if this intervention were scaled up to reach 100,000 students and to a duration of one year, it would cost approximately USD 45 per student.
A total cost of USD 15.3 per student for a 15-week educational intervention or USD 1.0 per week per student is not very cheap. However, if we consider the effectiveness, this intervention looks attractive. Our treatment effect estimates indicated that this intervention could achieve 2.21 SD (or 2.16 LAYS) and 2.37 SD (or 2.31 LAYS) per USD 100 each of spend in the Standard and Extended groups, respectively. Based on the cross-country LAYS estimate reported by ⇡Angrist et al. (2020), this level of effects is above average.
Moreover, there is further scope for cost reduction if this intervention is scaled up to a national level and / or offered by the government. The call rate we could get for this small-scale project is much higher than what a national-level project could get. A final point worthy of consideration is the unit of intervention. We intervened in the distance learning of children individually. The children did not need to go somewhere or need to make a group to access the lessons. This delivery method increases the cost per student, although it provides students greater flexibility through choice in study hours and content i.e., a flexible lesson plan. Overall, this intervention is found to be cost-effective with scope for further cost reduction.
This study provides some of the first evidence of using IVR in providing interactive lessons for educating primary graders with basic literacy and numeracy in household settings. Based on the experience of implementation and the results of this study, in this section, we highlight a few crucial factors and make recommendations for the successful execution of any low-tech student-focused intervention in low-resource settings.

Intervention at the individual level
In this study, we selected a group of caregiver-child dyads who were offered the opportunity of participating and who opted to join. This condition of sample recruitment excluded those caregivers who had a low level of motivation to engage in homeschooling through an IVR-based curriculum. Usually, motivated mothers or caregivers are expected to engage better and to have a greater willingness to help their child to learn. This was an important factor behind the success of this programme. However, this exclusion criterion would not be possible if the programme were to be administered at the school level i.e., an intervention for the whole class of any given grade. A school-level intervention might select less-interested caregiver-child dyads and end up with a lower level of learning gain, which might create a significant resource misuse.

Extended caregiver involvement
For a successful distance learning solution, caregiver involvement is crucial. Caregivers could play various roles in low-tech distance learning solutions to improve efficacy, for instance, caregivers can act as facilitators. Primary graders might not be old enough to phone in by themselves and engage in learning on their own. In a caregiver's survey, ⇡Fall et al. (2021) found that 55% of children did not know how to use a mobile phone and therefore, they didn't phone in to learn. In other words, caregivers need to be involved in distance learning programmes for primary graders as facilitators.
Next, caregivers can have the role of learning partners. In this programme, educational tasks or exercises were specially designed for our intervention. 'Special helpers' (mostly mothers) played a central role in providing learning. The mothers listened to the lessons, read, and practised together with their children. This higher level of interaction was an important factor behind the success of our intervention. For this reason, the educational content for our programme had to be specifically caregiver-oriented so that they could engage with it.
Finally, caregivers need to be nudged regularly. In this intervention, caregivers were especially targeted, briefed, and nudged regularly to participate in their children's education. Regular SMSs were sent out to remind them. Moreover, programme representatives used to call mothers regularly to check their progress in the curriculum. These calls were very short, and only once a fortnight or so, and targeted to caregivers who called in on an irregular basis. The cost of adding these quick calls in the intervention was not very high but they had a significant impact on caregiver engagement. This is another recommendation for better caregiver involvement.

Interventions for the right groups
Low-tech distance learning solutions are not for everyone in society. During the Covid-19 pandemic, multiple low-tech interventions reported that children from marginalised strata of society got the most out of such interventions (⇡Hassan et al., 2021;⇡Schueler & Rodriguez-Segura, 2021). In this study, we also observed similar heterogeneity in treatment effects. Children from low-income families, with less-educated caregivers, and with poor baseline academic results did significantly better in the endline assessments. Therefore, to use resources effectively, primarily marginalised students need to be selected for low-tech interventions. Students from relatively well-off families have multiple ways of accessing education and a low-tech alternative might not be very attractive to them.

Flexibility and retention
Unlike other one-way broadcasting platforms, such as radio and TV, IVR can provide flexibility in accessing lessons. This was one of the key benefits of our intervention, as it helped caregivers to access lessons with relative ease. Further, a quiz with each lesson warranted a higher rate of lesson completion and programme retention. Listeners usually completed the entire lesson as the quizzes were accessible only at the end. This low-cost quiz also provided a significant boost in participation throughout the programme. We strongly recommend including flexible delivery hours and incentives for participation in distance learning solutions.

Conclusion
Globally, a significant number of children were out of school due to the Covid-19 pandemic. The crisis is not over yet. We may continue to see more lockdowns and school closures in the coming year(s) as new variants of the virus emerge. Despite a significant amount of investment over the past two years, many low-and middle-income countries still have poor distance learning ecosystems. In the short term, a low-tech, low-cost, and highly scalable distance learning model like the intervention we used in this study can be used remedially in countries with poor EdTech and educational infrastructure.
Although the prolonged school closures and learning interruptions the world has experienced since 2020 are unique to the Covid-19 pandemic, educational disruptions at a smaller scale are prevalent in many low-and middle-income countries. Natural and human-induced events that damage educational infrastructure and limit school operations often create significant barriers to the learning of children worldwide. For this reason, policymakers may consider expanding educational services by using wide-reach technologies in out-of-school settings to support and extend learning opportunities for children.
The most important aspect of the intervention in this study is its wide accessibility as it can broadcast lessons via feature phones. So, this sort of intervention could provide the opportunity to learn for millions of underprivileged children. If the children can be encouraged to engage in such forms of distance learning, nudged regularly, provided with greater caregiver time and incentives, even a brief intervention can be fruitful. In this study, we only provided 20 hours of lessons over 15 weeks to test the efficacy of the intervention. In practice, lessons can be expanded to provide education aligned with the school curriculum to support learning in household settings. IVR-based education cannot replace schools but can support schools by providing personalised learning.
As in the case of Bangladesh, feature phones are one of the most common household ICT devices in low-and middle-income countries. They make it possible to deliver IRI-based lessons via IVR to enable children to learn at home at their convenience with minimal help from their caregivers or any household member and lead to significant gains in learning at a low level of cost.