Testing EdTech Interventions in Bandarban Key Findings from the EdTech Hub Sandbox

.


Figures and tables
. Our original hypotheses (they were later revised following Sprint 1) 18 Table 4. List of models used in the first deployment 19 Table 5. Summary of the first deployment. Each box represents how many schools were assigned to a particular model. 21 Table 6. Summary of the second deployment. Each box represents how many schools were assigned to a particular model. 22 Table 7. Summary of the second deployment. Each box indicates the type and demography of the schools in the sample. 22 Table 8. Summary of the first deployment. Each box represents how many schools were assigned to a particular model. 24 Table 9. Themes by research question for Sprint 1 26 Table 10. Themes from Sprint 1 findings in detail 28 Table 11. Learners' worksheet scores by range (% of correct responses), broken down by modality 30 Table 12. Summary of the second deployment. Each box represents how many schools were assigned to a particular model. 38 Table 13. Sprint 2 research questions and associated critical beliefs. 39

Executive summary
This report summarises the evidence generated from EdTech Hub's sandbox in Bandarban, a remote region in South-Eastern Bangladesh.
The aim of this sandbox was to validate which EdTech interventions contribute the most to numeracy learning for marginalised learners in Bangladesh. This was identified as an evidence gap by the UNICEF Bangladesh Office and the Government of Bangladesh. It was acknowledged that there were many potential EdTech modalities that could play a role and very high uncertainty about which EdTech modalities are impactful, feasible, and scalable in marginalised contexts.
Sandboxes provide implementing partners with tools, processes, and hands-on support to test, gather evidence and adapt EdTech interventions. They work in 'sprints' of implementation and learning, so that evidence can be applied within the project's duration. Given the high uncertainty and various modalities to explore, sandbox methodology was deemed appropriate for this work. Through these sprints, uncertainty about the efficacy of the interventions could be reduced, forming the basis for future, more rigorous research and suggestions for future implementation.
The sandbox began with a kickoff phase in which the timeline and scope of work were revised. The Needs Assessment and Hypothesis Development phase followed this (see ⇡Tanweer et al., 2022). The Agami Foundation, the implementation partner, visited 10 schools, conducting observations and interviews in 5 upazilas (administrative districts) of Bandarban to gather information about the people and schools to be targeted by this sandbox. The findings from this research were used to develop three hypotheses around different EdTech modalities (multimedia classrooms (MMCs) , tablets for 1 classroom use, and use outside the classroom). Each modality was used to deliver Grade 6 numeracy content from the Kolibri platform. 2 In Sprint 1, the modalities were deployed in 17 schools in a multivariate test. In Sprint 2, several pivots were made before continuing the intervention in the original 17 schools and an additional 14. This staggered deployment was used to enable learning and iteration.
Data was collected at multiple points during the two sprints. Data collection methods included surveys, classroom observations, in-depth interviews, platform usage data, cost modelling, and worksheet scores. This report details the many findings that emerged from this research.
The following key conclusions were drawn.
■ The evidence suggests that MMCs are a promising intervention. Their effectiveness and implementation are worthwhile avenues for further research ■ Teacher training was shown to be effective. This needs to remain core to the intervention, with additional targeting towards those most need it.
■ The evidence indicates that the out-of-class tablet model and models that combine the simultaneous use of MMCs and tablets are not appropriate for low-capacity teachers. These models should be avoided in future research and implementation. However, the individual use of either MMCs and tablets in the classroom are promising models that should continue to be explored.
■ It is very difficult for teachers to ensure equitable access and engagement for all models. Future interventions should explore strategies for this.
■ Time management was highlighted as a significant issue. Teachers should be provided with step-by-step lesson plans as part of the intervention.
■ The evidence indicates that other modalities in the classroom could complement MMC-led lessons to increase equity and effectiveness. A carefully designed intervention could be the basis for future implementation and research.

Introduction
The Covid-19 pandemic has catalysed the need for technology-enhanced education globally. This is no different in Bangladesh, where the Education Section of the UNICEF Bangladesh Country Office (henceforth referred to as 'UNICEF Bangladesh') is seeking a better understanding of the role and potential of EdTech to improve numeracy skills among the most marginalised learners. In efforts to do this, UNICEF Bangladesh partnered with EdTech Hub to in which EdTech interventions contribute the most to numeracy learning for marginalised learners in Bangladesh. This was explored through a sandbox in Bandarban, a remote region in South-Eastern Bangladesh. This report synthesises the evidence generated from this EdTech Hub sandbox. While further research is needed, its findings suggest that multimedia classrooms, which were found to be present and functioning in most schools in the region, are a promising intervention. Their effectiveness and implementation are worthwhile avenues for further research.
The report is structured as follows: ■ This section provides further details about the background to the sandbox, the context of Bandarban, and our partnership with UNICEF Bangladesh and our implementing partner, the Agami Foundation.
■ Section 2 looks at the needs assessment and hypothesis development phase undertaken prior to the sandbox. The aim of this phase was to gain a better understanding of the experiences of the people (learners and teachers) and schools who were to participate in the sandbox.
■ Section 3 outlines the staggered deployment agreed with our implementation partner Agami and the research questions underpinning the study.
■ Section 4 focuses on Sprint 1 activities, findings, and iterations made ahead of Sprint 2.
■ Section 5 details Sprint 2 activities and findings.
■ The final section, Section 6, outlines conclusions based on the findings of the sandbox, considerations for future implementation, and next steps.

EdTech Hub and UNICEF partnership
In 2021, UNICEF Bangladesh partnered with EdTech Hub to develop a theory of change (TOC) (⇡Clark-Wilson et al., 2021), which would identify how a technology-enhanced education system can help improve the learning outcomes of students in Bangladesh. This was undertaken in response to the lack of robust research and insights for determining which EdTech modalities could be most effective in supporting learning in Bangladesh, including in low-income communities, with low digital literacy rates and low learning outcomes, and especially for girls and other marginalised groups.
Following this successful partnership, UNICEF Bangladesh and EdTech Hub extended the partnership to design, implement, and gather data on EdTech modalities through a sandbox in Bandarban. An EdTech Hub sandbox provides tools, expertise, and hands-on support to work with partners to test and learn about promising EdTech interventions. This sandbox aims to identify the technology-enhanced approaches which most effectively support the acquisition of basic numeracy skills by learners in hard-to-reach areas.

Context of Bandarban
Bandarban is a remote, hilly, multi-ethnic, and multilingual district in southeastern Bangladesh. Beyond its diversity and topography, Bandarban also has some of the lowest education outcomes in the country. Bandarban also lags behind the national average female youth literacy rate by about 22% (67% compared to an 89% national average). It is anticipated that the 18 months of school closures caused by the Covid-19 pandemic will have further widened this existing learning gap.
The district also experiences teacher and staff shortages (in part due to the remote nature of many schools) as well as gaps in infrastructure. Although some schools have access to power via solar energy, many face significant problems with internet connectivity and electricity supply. ■ To design an appropriate technology-enhanced learning model under UNICEF's Reimagine Education initiative focusing on a specific learner group that is marginalised and hard-to-reach children.

The sandbox
■ To generate evidence on how low-and high-cost technologies play a critical role in improving learning achievement in remote regions in Bangladesh.
■ To develop capacity of relevant education partners and stakeholders to undertake the sandbox approach and rigorous EdTech research for further implementation at scale.
The overall timeline of the sandbox is set out below.

What is a 'sprint'?
A sprint is a set of tightly scoped activities executed in a fixed period of time to understand whether a particular hypothesis can work, scale, and have an impact.
The term sprint is derived from a set of agile principles and values that are common in software development. In an EdTech Hub sandbox, sprints are typically between 1-3 months long and can consist of a number of different activities, all aimed at generating useful evidence about the EdTech intervention(s) being explored. In this sandbox, both sprints consisted of teacher training and onboarding and the deployment and implementation of EdTech interventions.
Each sprint begins with a sprint plan and ends with a review. To enable meaningful planning based on the evidence generated, each subsequent sprint is not designed until the preceding sprint has been completed.

Needs assessment and hypothesis development phase
The aim of the needs assessment and hypothesis development phase was to gain a better understanding of the experiences of the people (learners and teachers) and schools we seek to impact with the EdTech modalities tested through this sandbox. These findings enabled the team to design and test interventions that are context-appropriate and have the best chance of enhancing numeracy learning outcomes for Grade 6 learners in Bandarban.
This section includes a summary of the methodology and findings of this phase. For more details, see Needs Assessment & Hypothesis Development Report (⇡Tanweer et al., 2022).

Methodology
The Agami team visited 10 schools in Bandarban and interviewed teachers, head teachers, and students. In total, Agami spoke with 44 participants. Agami also observed maths classes, the school premises, and their ICT facilities. The following research questions guided data collection:

RQ1:
How is technology used now in schools, classrooms, and the home, by teachers and Grade 6 students?

RQ2:
What teaching practices are deployed in schools for numeracy instruction for Grade 6 students? RQ3: What factors impact numeracy learning outcomes for students?

RQ4:
How do teachers feel about using technology in their classroom / for learning? RQ5: How do Grade 6 students feel about using technology for learning?

Findings
The findings of the needs assessment data collected were distributed across 13 themes, answering the five research questions. The findings present the availability and utilisation of technology and a valid level of knowledge and interest in teachers and students towards using technology for education.
While some findings shed light on the context (e.g., teacher shortages, internet connectivity, electricity, school closures during the Covid-19 pandemic, lack of teacher training, language barriers, etc.), some human factors also played a role in the resistance to technology in education (e.g., pedagogical culture, resistance to change, etc.).  13. Students expressed interest in using technology for learning.

Testing EdTech Interventions in Bandarban: Key Findings
The findings from the needs assessment phase had the following implications for the implementation of our sandbox:

Teacher training
Responses from teachers indicated that they had little understanding or awareness of how technology can be incorporated into their classroom instruction. Given this, teacher training conducted by Agami significantly focused on how devices should be used in the classroom to enhance learning. Evidence on EdTech in Bangladesh also highlights the need for teacher training to go beyond training on ICT to support teachers in using ICT to improve their pedagogy.

Onboarding
The needs assessment research identified a general lack of familiarity with integrating technology in education and a unidirectional pedagogical culture. As such, all interventions required detailed onboarding for teachers and learners to ensure buy-in. Given the different levels of enthusiasm in teachers and students regarding using technology for education, it was evident that teacher training needed to inspire attitudinal change along with technical know-how. Onboarding sessions also ensured that stakeholders' perspectives were heard and addressed regarding challenges and concerns associated with EdTech use and that they felt supported in their engagement with this technology.

Hardware / device equity
Given that MMCs are available in the implementation schools, they are an appropriate vehicle to deliver the content available on the Kolibri platform. Teachers pointed to MMCs as a more equitable and accessible intervention for large classes. They expressed concern that distributing smartphones would be less manageable given the issue of large classes. Even though an equitable and impactful distribution of tablets seems to be a matter of concern, teachers expressed support for android tablets because of their familiarity and level of comfort with using android smartphones. Issues such as reliable electricity supply and internet connectivity were also considerations throughout the sandbox.

Self-learning tech interventions / content
Learners can interact with content without the supervision of teachers, thus minimising learning gaps and constraints related to the significant teacher shortages in Bandarban. These interventions can supplement the traditional, lecture-based teaching methods already employed in Bandarban schools by introducing or reaffirming specific concepts to students through alternative Testing EdTech Interventions in Bandarban: Key Findings presentations and activities. The content utilised should also be accessible to and impactful for learners who have difficulty comprehending and communicating in Bangla.
To ensure that the content introduced was useful and appropriate for learners and teachers, the content made available in the intervention was covered in the existing curriculum for the year. Agami also placed a strong emphasis on preparing teachers to use the content and tools that they would use frequently.

Models
Based on the findings of the user research, Agami refined its intervention models and modalities (MMCs and tablets). For example, the teacher shortages currently being experienced in Bandarban made it potentially appropriate to introduce an out-of-class tablet model to reduce the burden on already low-capacity teachers during lessons. Since the needs assessment research found that teachers in Bandarban are severely undertrained, the teacher training element of the intervention was further emphasised. Also, given learners' and teachers' limited experience using technology in the classroom, the sandbox team decided to include scoping a feature phone model as part of the sandbox.
In addition to these learnings from the needs assessment phase, the following three hypotheses emerged from the findings: Table 3. Our original hypotheses (they were later revised following Sprint 1)

Tablet (in class) Tablet (out of class)
If we… Integrated guided use of Kolibri + Khan Academy, led by the teacher with training on digital literacy and the role of ICT in enhancing instructional practice, Provided children with single-purpose educational tablets for use in groups of eight in the classroom, alongside parent and teacher onboarding, Provided children with access to a tablet to use outside of the classroom setting,

Then …
Students would regularly engage with digital learning resources,

So that…
Students would be more motivated to learn, and achieve higher benchmark learning outcomes.
All three models deployed the Kolibri platform and Khan Academy Grade 6 numeracy content. As Agami had access to and experience in deploying this Testing EdTech Interventions in Bandarban: Key Findings platform and content, it enabled the sandbox team to rapidly test the potential of various EdTech modalities. As shown by the table above, each hypothesis aimed to apply a different EdTech intervention towards the same outcomes and impact. More details on the models are given in Table 4 below. Video-and lecture-based, supplemented with practice problems (via Kolibri) and worksheets (developed and shared by Agami)

Tablets
In class: Students divided into groups of ~8 with one tablet per group used in classroom. The students work together to solve the practice problems.
Out of class: Students use tablets in out-of-class settings (less guidance).
Students are divided into groups, and each group gets weekly slots and logins. Again, students work together in groups to solve practice problems.

MMCs + tablets
In class: The MMC technology and the tablets are both used during normal class time.

Out of class:
The MMC technology and the tablets are used outside of normal class time.
In addition to these models, a 'low-tech' model, which would aim to deliver content via feature phones, was also explored.
During the teacher and student training at the beginning of Sprint 1, we found that students had very limited access to feature phones. As a result, we broadened our exploration to also include no-tech innovations. This was undertaken via a 'call for interest' from Grade 6 numeracy and Bangla language content providers. More details on this are available given in Section 4.4.

Iterative implementation and learning plan
Following the needs assessment phase, Agami and EdTech Hub agreed to conduct a 'staggered deployment'.
The EdTech modalities would be deployed in line with the sprints. The first deployment of technology to 17 schools in Sprint 1 occurred in August 2022. The second deployment, in Sprint 2, to the remaining 14 schools occurred in October 2022. There was scope for learning and iterating to take place between the first and second deployments. These two deployments were the core activities in each of the two sprints.

Staggered deployment
The findings from the needs assessment phase helped to inform the development of three hypotheses that would inform the staggered deployment phase, through which Agami implemented interventions in two deployments.
During the first sprint, Agami tested different modalities in 17 schools (July-September 2022). Initially, 20 schools were targeted, but Agami could not access 3 of these schools because: ■ There was no MMC in one of the schools ■ One school appeared to be permanently closed ■ One school only had 2-3 students regularly attending classes.
This first sprint enabled the EdTech Hub team to gather initial data, which would inform a second deployment in all 31 schools in the sample for this sandbox (including the 17 schools from Sprint 1).

Testing EdTech Interventions in Bandarban: Key Findings
Following Sprint 1, the sandbox team decided to stop using tablets out of class (this is discussed further in Section 4.4. Therefore, during Sprint 2, Agami tested the three remaining modalities in 31 schools (October-November 2022). In addition to the 17 where the interventions were tested in Sprint 1, 14 more schools were added to the intervention (the three schools Agami could not access in Sprint 1 were replaced). In both sprints, the samples of schools were diverse; they included rural and urban schools as well as government and private schools. Each sub-sample of schools (designated as MMC-only, MMC + in-class tablet, or in-class tablet schools) within the multivariate test included these different types of schools. This helped us to consider the effect of the variables of school location and type across all the different EdTech modalities.

Learning plan
Through the course of implementation in both Sprints 1 and 2, EdTech Hub investigated the following research questions: 1. Do teachers adequately use tech resources to aid instruction? (i.e., Are teachers well supported, prepared, engaged, and able to use this content? Do they foster learner engagement with digital resources?) 2. Do the tech modalities lead to improvements in teaching and learning? (i.e., do teachers and learners report satisfaction with these Testing EdTech Interventions in Bandarban: Key Findings resources, do they report better learning, and is there evidence of improved learning outcomes?)

What tradeoffs (cost / impact) exist around the different modalities?
(i.e., which of the implementation models offer optimal impact / value for money; which are most likely to retain impact at scale / be sustainable?) 4. Does the proposed EdTech seem feasible to implement equitably (ensuring equal access to EdTech for students) and at scale given constraints (infrastructure, time, resources, etc.)?
These research questions were developed to understand the impact of the models and make comparisons between them. The research questions were explored in both sprints via formal data collection, including surveys, interviews, classroom observations, and other methods. The related findings allowed the sandbox team to make key pivots between sprints and develop recommendations for Agami and UNICEF Bangladesh that extend beyond the sandbox.
Testing EdTech Interventions in Bandarban: Key Findings

Sprint 1
In Sprint 1, Agami validated the content to be used, trained teachers on the technology and the intervention, and monitored their implementation.
Data was collected during each implementation activity and via classroom observations. Data protocols (classroom observation protocol and post-training survey) were prepared by EdTech Hub, and Agami gathered data. Both organisations collaborated to analyse the data, and UNICEF Bangladesh, EdTech Hub, and Agami collaborated to finalise the implications of the data.
This section provides an overview of the implementation activities and the data collection processes. It then details the key findings that emerged during Sprint 1 and iterations recommended by EdTech Hub to be made prior to Sprint 2.

Implementation activities
Ahead of Sprint 1, in July 2022, Agami held a content validation and sensitisation workshop with teachers, head teachers, parents, and students. As part of this workshop, Agami walked participants through the content on the Kolibri platform and noted their feedback. Teachers and head teachers also shared both their enthusiasm and concerns relating to the integration of technology in their classrooms.
In late July 2022, Agami held its first teacher training session. This was followed by onboarding teachers and students on their specific model at the 17 targeted schools. The process involved teaching participants how to use the relevant technology assigned to their intervention arm (MMC, tablets, or MMC and tablets). They also received training on how the technology should be integrated into the classroom to facilitate teaching and learning.
Immediately after the training, Agami deployed the MMC and tablet technology in 17 schools in Bandarban (see Table 9 below for an illustration of the deployment breakdown). This deployment included 3 weeks of preparation and 6 weeks of implementation. Table 8. Summary of the first deployment. Each box represents how many schools were assigned to a particular model.

Deployment of MMCs and tablets in schools in Sprint 1 (July -September 2022)
MMCs only (7 schools)

Out-of-class tablets (3 schools)
Testing EdTech Interventions in Bandarban: Key Findings

Data collection
Data was collected at various points throughout Sprint 1, including at: ■ Content validation workshops: Agami developed a survey for teachers and head teachers that sought to understand their attitudes towards integrating technology in the classroom.
■ Teacher and student training sessions: EdTech Hub developed a survey for both teachers and students. Of those participating in the training, 18 teachers and 85 students were surveyed. They were asked about their experience with the training, how comfortable they felt with the technology they were being trained on, and any concerns they may have had. Students were also asked about their access to feature phones.
■ Classroom observations: EdTech Hub developed a classroom observation protocol. Ten classrooms were observed, covering all five combinations of implementation. The classroom observations sought to establish whether and how technology was used to aid instruction in the classroom, the role of the teacher, the level of equitable engagement and accessibility among students, and the learners' experience.
Agami conducted the classroom observations and surveys.
The following additional data was also collected during Sprint 1 but was analysed during Sprint 2: ■ Kolibri usage data: Data collected from the Kolibri platform, showing usage of the platform both in the MMCs and on tablets.
As in the case of the worksheet data, the usage data for Sprint 1 (via the Kolibri platform) was accessed and shared with EdTech Hub during Sprint 2. EdTech Hub once again analysed only the usage data from the MMC-only, tablets-in-class, and MMC + tablets-in-class models. This data focused on learners' progress watching content and engaging with the exercise content. The video content progress was calculated manually by comparing the video duration to actual watch time (the time actually spent watching each video was captured on and retrieved from Kolibri). The Kolibri platform itself calculated exercise progress.
■ Worksheet data: At the end of each tech-enabled maths lesson, students completed a brief worksheet with questions relating to the content they had just learnt. Scores from the worksheet were captured and analysed.

Testing EdTech Interventions in Bandarban: Key Findings
When this report was being developed, only worksheet data from Sprint 1 was available. While this data emerged during Sprint 1, it was only analysed during Sprint 2 because that is when it was made available and shared with EdTech Hub. EdTech Hub analysed only the worksheets from the MMC-only, tablets-in-class, and MMC + tablets-in-class models, as the other two models were no longer used after Sprint 1 (more details on this are shared in Sections 4.4 and 4.5).
Data collection during Sprint 1 focused on the following three out of our four key research questions: ■ Do teachers adequately use tech resources to aid instruction? (i.e., Are teachers well supported, prepared, engaged, and able to use this content? Do they foster learner engagement with digital resources?) ■ Do the tech modalities lead to improvements in teaching and learning?
■ Does the proposed EdTech seem feasible to implement equitably (ensuring equal access to EdTech for students) and at scale given constraints (infrastructure, time, resources, etc.)?

Findings from Sprint 1
In seeking to answer these research questions through the data collection methods described above, the following findings emerged. 1. All teachers said they had learned something new which could be applied to their classroom practice, but many also shared suggestions for how the training could be improved and implementation challenges they foresaw.
2. In all of the classes observed, the models are not being implemented as intended.
3. Teachers were found to be able to use tech in the classroom.

RQ2: Do the tech modalities lead to improvements in teaching and learning?
4. The role assumed by each teacher when implementing the intervention differed: most provided monitoring support, few engaged in-depth with instruction.
Testing EdTech Interventions in Bandarban: Key Findings 5. While students could operate and navigate content on tablets independently, this may not translate into learning.
6. As expected, smaller classes were found to be more effective learning environments, but generally, students were attentive.
7. Children seemed to understand content more effectively from the MMC-based model, although there are significant limits to our understanding of this.
8. Across the different modalities, usage data indicates that learners are not successfully completing the Kolibri exercises.

RQ3: Does the proposed
EdTech seem feasible to implement equitably (ensuring equal access to EdTech for students) and at scale given constraints (infrastructure, time, resources, etc.)?
9. Students surveyed have very limited access to feature phones.
10. Group size and formation varied across the models involving tablets.
11. Differences in engagement were noted based on academic performance, seating arrangements, and language ability.
12. Significant barriers to implementing these models include tech issues, model structure, and challenges related to classroom management.
Key findings from Sprint 1 are described in more detail below.
Testing EdTech Interventions in Bandarban: Key Findings Table 10. Themes from Sprint 1 findings in detail 1. All teachers said they had learned something new which could be applied to their classroom practice, but many also shared suggestions for how the training could be improved and implementation challenges they foresaw.
In particular, 75% of teachers reported finding videos as the most helpful resource in the training sessions. However, teachers felt that students would need further training to operate the tablets and navigate Kolibri content independently. Teachers suggested improving the training by increasing its length, having regular training and monitoring, and more training materials for both teachers and students. Other suggestions included increasing the device ratio from 1 tablet to 8 students (one suggestion was 1 tablet to 4 students) and placing a greater focus on classroom management.
When asked what challenges teachers foresaw with implementation, they cited electricity supply issues, learners' tech skills, seating, small screen sizes (relating to MMCs), not enough time, and issues with content.

In all of the classes observed, the models were not being implemented as intended.
Teachers altered the model by omitting the practice problem-solving portion of the lesson, not assigning the worksheets at the end of the class, showing fewer videos than intended, or not allotting time for individual tablet use (for the tablet models), for example. In other cases, teachers in the MMC models could not use the MMCs due to tech issues, including unreliable electricity supply and difficulty setting up the MMCs. In others, teachers in the out-of-class tablet model had switched to an in-class tablet model for the sake of convenience. A lack of planning or understanding of these model components may be responsible for some of these omissions. In fact, it was noted that teachers had lesson plans in just 20% of the classes. It is also possible that teachers omitted aspects of the model due to time constraints.
During the refresher training at the start of Sprint 2, teachers reported that the regular 45 minutes of class time might not be enough to use digital learning materials in the lesson.

Teachers were found to be able to use technology in the classroom.
In all 9 of the classrooms scored against the observation checklist (1 of the 10 schools observed was not scored against the checklist), teachers could set up the MMC and / or tablets in the classroom and could log into the platform. Most could also operate the tablets by themselves (89%) and could use the digital learning materials in the lesson (89%). In 78% of classes,

Testing EdTech Interventions in Bandarban: Key Findings
teachers could guide their students in using the MMC and / or tablets. Teachers who could not do so either lacked digital literacy skills or had not attended the training and therefore did not know how to navigate the content.
4. The role assumed by each teacher when implementing the intervention differed: most provided monitoring support, and few engaged in-depth with instruction.
In a majority of the classes using the tablets (71%), it was noted that teachers moved around the classroom to observe students as they used the tablets. Seventy-one per cent provided instruction to students on when and how to use the tablets.
In most schools (60%), teachers either explained the video content during or after it was played. Approaches to this included relating it to previous maths lessons, the textbook, and real-life examples. Fifty-seven per cent of teachers were noted as able to answer students' questions related to digital content. In addition, 67% of teachers could execute worksheets in the proper way (i.e. assigning the worksheets to students to complete individually at the end of the lesson as intended). This demonstrates a significant variance in how teachers managed the lessons and the support they provided to students during the intervention.
However, fewer teachers engaged in asking and answering questions. Less than half of the teachers answered students' questions. Just 20% of teachers were observed to ask students if they had questions or if they understood the content.

While students could operate tablets and navigate content on tablets independently, this may not translate into learning.
Students had a very high ability to engage with digital technology in the tablet models. Students in all of these classrooms could operate the tablets and browse the Kolibri platform independently. They were also able to connect the headphones and return the tablets.
However, there are some indications that this capability may not translate into learning, as it was observed that students were actively participating in group work in less than half of the classrooms (43%). Students were observed asking questions about the digital content in just 11% of classes.
Testing EdTech Interventions in Bandarban: Key Findings 6. As expected, smaller classes were found to be more effective learning environments, but generally, students were attentive.
Of those observed, class sizes ranged between 10 and 72 learners. Observers noted that 40% of the classes made for more effective learning environments as they were smaller. Overall, most students (70%) across the models appeared to be attentive and follow instructions.
7. Children seemed to understand content more effectively from the MMC-based model, although there are significant limits to our understanding of this. Students who were in lessons with MMCs scored higher than students with tablets. However, it was observed that students received variable support from teachers and other students for completing the worksheets, which significantly impacted the scores.

Testing EdTech Interventions in Bandarban: Key Findings
Across the MMC-only (10% of exercises completed), tablets-in-class (27%), and the MMC + tablets-in-class models (13%), progress on the Kolibri exercises was low. This indicates that the exercises were not being completed or were being completed incorrectly.
Meanwhile, there was no significant difference in video content progress (learners' progress watching the video content on Kolibri). Video progress was highest in the MMC-only model (82%) as compared to the tablet-in-class (71%) and MMC + tablets-in-class models (80%). The large difference between video and exercise progress indicates that the use of Kolibri in the classroom was more focused on watching the videos than going through the exercises.
The analysis of the usage data also revealed that videos and exercises that were not assigned as part of the intervention lessons had been played or engaged with frequently. Teachers and / or learners were selecting other content on the platform. In addition, the data indicates that videos were often rewatched as actual watching time was frequently much longer than the length of the video.

Students surveyed have very limited access to feature phones.
Most students (64%) did not have access to feature phones at home. Just 13% used a feature phone daily, and almost all did not take the phone to school (98%). Very few students surveyed (13%) said the phone belonged to them and / or their siblings. This suggests that the low-tech intervention utilising feature phones that was being considered may not be feasible. A tablet-or MMC-based intervention may be more appropriate in this context.
However, half of the students (51%) with access have used a feature phone for educational purposes. This includes watching, reading, or listening to educational content, as well as completing educational activities.

Group size and formation varied across the models involving tablets.
The initially prescribed ratio of 8 students to 1 tablet is not an accurate reflection of the actual implementation. While some students could use the tablets individually or in pairs, most shared the devices in groups of 4-8 students. During the refresher training, some teachers voiced concerns that the number of tablets was insufficient for the number of students. Teachers took various approaches to create student groups, including separating genders and mixing high-and low-achieving students. It appears that in one class, students were grouped on the basis of ethnicity (i.e., Bengali and non-Bengali) reflecting the different languages spoken by the students.

Testing EdTech Interventions in Bandarban: Key Findings
Participation in these groups does not appear to have been equitable. For 20% of the classes observed, it appears that 1-2 students operated the tablets alone. Students were also observed entering answers individually rather than following a discussion with their group.

Differences in engagement were noted based on performance, seating arrangements (for tablets), and language ability (for MMCs and tablets).
Observers noted that learner engagement might be related to the learners' seating positions in the classroom and also to their individual achievement levels. In 30% of classes, learners seated at the front of the class appeared more engaged than those at the back. Of the tablet models, 71% of classes saw lower-achieving students being less engaged or facing greater difficulty operating the tablets and understanding the content.
Language barriers also appeared to limit engagement for some students. In two schools, it appeared that students had trouble understanding instructions and the Bangla video content. It was also noted that non-Bengali students struggle to communicate in Bangla.

Significant barriers to implementing these models include tech issues, model structure, and challenges related to classroom management.
Tech issues (unreliable electricity supply, time-consuming set-up) were noted in 50% of the schools using MMCs. These resulted in reduced class time for learning or, as in a small number of instances, the MMCs were not used at all. In addition, due to the reflection of light and class size, visibility of the screen was noted as a challenge. During the refresher training, some teachers noted that the TV monitor provided by UNDP was not helpful in the larger classes as the screen size was small, and the students at the back struggled to see the display clearly. In the tablet-in-class model, the earbuds had several technical issues, and the teacher said she rarely uses them for this reason.
The structure of the out-of-class tablet models was found to be inconvenient in several instances. In one of the out-of-class schools using tablets, the timing of the extra session had a significant impact on engagement. In another class (assigned MMC + tablets out of class), the teacher decided to switch to an MMC + in-class-tablets model. Another school did not place students into clusters to receive the tablets but allowed all students to receive them. As a result, the student-to-tablet ratio remained high, and students were not observed using the tablets individually.

Testing EdTech Interventions in Bandarban: Key Findings
Classroom management was noted to be a challenge in half of the schools (across all models) and presented challenges for giving instructions, observing students, and ensuring students were on task.

Iterations made for Sprint 2
Based on the above-mentioned findings, the following iterations were made for Sprint 2.

Removal of the out-of-class tablet model and increase in the sample of schools assigned the in-class tablet model
The out-of-class tablet model was removed in Sprint 2. In its place, Agami continued with in-school, tablet-based models, combining these with MMCs.
For reasons of feasibility, in-class tablets were implemented in eleven schools in Sprint 2, and MMCs + in-class tablets were implemented in six schools.
To increase the time students could spend using the tablets, it was also suggested schools introduce the possibility of allowing students access to the tablets during break time or free time during the school day. However, this was found to be difficult for schools to arrange, further highlighting the inconvenience of and resistance to out-of-class tablet models.

Provision of additional support and guidance to teachers via training
Prior to Sprint 2, teachers in all 31 schools were given training. For those teachers who had already used the EdTech interventions as part of Sprint 1, this was a 2-day 'refresher' training. For those encountering the interventions for the first time, this was a 3-day training.
Several steps were taken in these training sessions to provide further guidance to teachers.
■ Lesson planning, classroom management and the integration of digital learning resources was a core part of the first day of both teacher training sessions ■ The Agami team demonstrated classes for each of the three models, outlining the steps teachers should take during the lesson.
■ The second day of both sessions included a simulation of conducting tech-enabled lessons and an opportunity for reflection, reinforcing core competencies such as playing and explaining video content and guiding students on how to use tablets. It also allowed teachers to practise following a lesson plan (developed by Agami).
■ In addition to lesson planning and explaining video content, the training placed a greater emphasis on other identified areas for improvement such as answering students' questions about the digital content, asking students questions, managing time well, and implementing the worksheet session.

Strengthen the MMC-only model in the next phase
Data from Sprint 1 suggested that the MMC model encountered more implementation-related challenges than the in-school, tablet-based model. Given these challenges, a priority for the next sprint was finding solutions to improve the implementation and use of MMCs.
Teachers were asked to bring the MMC devices to the training sessions prior to the second deployment, and specific guidance on setting up the MMCs and using the intervention was provided.
To gather further data on the constraints associated with MMCs and understand their effectiveness, the number of schools using MMCs was doubled to 14 schools in Sprint 2.

Maximising engagement and accessibility for students
Student engagement and accessibility to the digital learning resources varied greatly across all models. In particular, it appeared to be difficult to ensure engagement and accessibility for classrooms with larger class sizes assigned to the MMC model. As a result, class size was identified as a key variable that should be used in assigning models to ensure greater engagement and accessibility.
Given that it was frequently observed that students seated at the back of classes struggled to see the MMC screen, classrooms with fewer students were to be designated MMC-only. In turn, tablet-based models could be reserved for large classes. In the second deployment, the newly assigned MMC-only schools had an average class size of 62 learners, and the tablet-only schools had an average of 130 learners.

Discontinued exploration of low-and no-tech interventions
EdTech Hub put out a call for interest for Grade 6 numeracy and Bangla language content providers during Sprint 1. The call was specifically interested in sourcing content that could be utilised via feature phones. However, we received only three responses. The low number of responses may have been due to the call's limited reach; therefore not all relevant parties may have received the call. Additionally, some may have been unwilling to share information via a Google form (the selected format for responding to the call). It is possible that there are not many viable low-and no-tech Grade 6 numeracy content providers in Bangladesh.

Testing EdTech Interventions in Bandarban: Key Findings
In addition, given that it was found that students in the sample schools had very limited access to feature phones, a feature phone-based intervention was deemed inappropriate for this specific context.

Summary of approach to Sprint 2
EdTech Hub proposed a Sprint 2 sample that built on the modalities most likely to be effectively integrated into classroom learning and engage learners.
This meant discontinuing the out-of-class tablet models (as it has not led to significant engagement) and increasing the number of in-class-tablet schools (as learners seem able to engage with the tablets). Simultaneously, the EdTech Hub team recommended that Sprint 2 focus on continuing to understand and tackle some of the specific challenges associated with MMCs.
EdTech Hub recommended focusing on the three most promising modalities (tablets in class, MMC, and MMC + tablets in class) and trying to improve their effectiveness during Sprint 2, which had a two-month duration. This included better supporting teachers in classroom integration of the technology and striving for more equitable access to the tech tools. In Sprint 2, EdTech Hub hoped to arrive at more conclusive insights about what modalities might be scaled (and where they might be most appropriate) in the next phase of this programme.
Testing EdTech Interventions in Bandarban: Key Findings

Sprint 2
Sprint 2 consisted of an updated 'refresher' teacher training in mid-October 2022.
This training occurred over two days, with 27 participants (15 Grade 6 maths teachers and 12 headteachers). The training was for the teachers who were already familiar with the EdTech interventions from Sprint 1 and included: ■ Warm-up exercises ■ Sharing the revised models with the teachers and information about the discontinuation of the out-of-class tablet model ■ More details on the interventions, including developing lesson plans and incorporating digital learning resources.
■ 'Demo' lessons and time for reflection ■ Next steps for the interventions.
In addition, teacher training was provided for teachers who were being introduced to the EdTech interventions for the first time. This training took place over three days, with 19 participants (10 Grade 6 maths teachers, 9 headteachers) and included: ■ Warm-up exercises ■ Introduction to the training programme ■ Introduction to the Kolibri platform and digital learning materials ■ Classroom management ■ Demo of the three intervention models, conducted by Agami ■ Qualities of a maths teacher ■ Further demo classes, conducted by teachers (with some teachers participating as students) ■ Internet safety ■ Next steps All 19 participants surveyed following the training said they would recommend the training to a colleague and also shared how they felt the training could be improved. The most frequently expressed suggestions included increasing the length of the training to four or five days (26% of respondents), providing more opportunities for more hands-on training (16%), holding continuous training for teachers (16%), and expanding training to other school subjects (16%).

Implementation activities
From mid-October to end-November 2022, the three combinations of EdTech modalities (MMC, MMC + in-class tablet, and in-class tablet) were implemented in 31 schools. This deployment included 2 weeks of preparation and 6 weeks of implementation. The two weeks of preparation consisted of teacher training and onboarding teachers and students to their assigned modalities in the new schools included in this sprint.

Data collection
Data collection occurred at various points throughout Sprint 2. This included: ■ Post-training surveys: EdTech Hub surveyed 9 head teachers and 10 maths teachers following the training administered by Agami. The following additional data was also collected: ■ Cost data: Using the EdTech Hub's Lean Cost Model, the cost per year per child of both modalities was calculated.
These data collection methods helped us answer the four research questions outlined in the learning plan (highlighted in Table 13 below).
Ahead of Sprint 2, critical beliefs were developed and elaborated upon based on evidence from Sprint 1. Critical beliefs are areas of the interventions where Testing EdTech Interventions in Bandarban: Key Findings the greatest uncertainty lies and therefore require evidence. These aspects of the intervention need to be true for the intervention to deliver impact at scale.

Research question Critical beliefs
Do teachers adequately use tech resources to aid instruction? (i.e., Are teachers well supported, prepared, engaged, and able to use this content? Do they foster learner engagement with digital resources?) Teacher training enables teachers to integrate technology into their teaching practices (i.e., ensuring both ICT and pedagogical skills are adequate).
Teachers can equitably integrate MMC and tablets into class time and ensure learner engagement.
Learners are able to engage regardless of class seating arrangements, gender, language and whether they are highor low-achievers.
Do the tech modalities lead to improvements in teaching and learning? (i.e., do teachers and learners report satisfaction with these resources; do they report better outcomes; is there evidence of improved learning experience?) Teachers / learners are able to meaningfully engage with (take knowledge from) the content provided.
What tradeoffs (cost / impact) exist around the different modalities? (i.e., which of the implementation models offer optimal impact / value for money; which are most likely to retain impact at scale / be sustainable?) Modalities implemented will offer diverse impact / cost tradeoffs.
Does the proposed EdTech seem feasible to implement equitably (ensuring equal access to EdTech for students) and at scale given constraints (infrastructure, time, resources, etc.)?
All learners will be able to use EdTech interventions effectively.
Learners will have regular and equitable access to devices. In tablet models, learners can use tablets equitably.
All learners are able to see and hear the content  1. Teachers struggled to integrate both technologies into the same lesson.
2. A majority of teachers observed were able to successfully integrate technology into the lesson, but additional training may still be needed to promote more effective integration of technology and improve pedagogical practices.

RQ2: Do the tech modalities lead to improvements in teaching and learning?
3. Classroom and time management were other frequently observed implementation challenges.
4. There was a range of positive signals that children were engaged with and understood the digital learning content, positively affecting their motivation.

RQ3: What tradeoffs (cost / impact) exist around the different modalities?
5. The two interventions are prohibitively expensive, although the handover of operations, management, and monitoring to local government could reduce this significantly.
6. Although both EdTech interventions suffered from technical difficulties, difficulties with the MMC were more frequently observed.

RQ4: Does the proposed
EdTech seem feasible to implement equitably (ensuring equal access to EdTech for students) and at scale given constraints (infrastructure, time, resources, etc.)?
7. Ensuring equity remains difficult in a classroom environment. Language appears to be the most significant barrier, followed by gender, educational performance, and digital literacy.
Key findings from Sprint 2 are described in more detail in Table 15 below.

Teachers struggled to integrate both technologies into the same lesson.
Teachers assigned the MMC-and-tablet model faced difficulties integrating both devices in the same lesson. In 75% of MMC + tablet classrooms observed, teachers did not use one of the two assigned devices. Of the three schools using the MMC + tablets that omitted one of the devices, two elected not to use tablets, and one did not use the MMC.

2.
A majority of teachers observed were able to successfully integrate technology into the lesson, but additional training may still be needed to promote more effective integration of technology and improve pedagogical practices.
It was also observed that teachers could integrate digital learning materials well and guide students well in 67% of lessons with tablets. For MMC models, this was 89% and 100%, respectively. Additionally, the classroom observations demonstrated that teachers were able to fulfil their role in a lesson, including showing videos, explaining concepts, solving problems, and asking students questions.
One significant finding from the thematic analysis was that teachers had taken away learnings from the training, further strengthening the case that the training had had a positive impact. Teachers reported that they felt they had learned strategies for classroom management, engaging students, and ensuring inclusion. Others noted that they learned lesson planning, new methods of teaching, and how to use and integrate technology.
Other complementary themes included that the content helped teachers to deliver lectures and explain things; teachers were comfortable with the technology; students benefited from the 'double explanation' of video and teacher; and teachers reported that technology makes teaching more efficient.
Although these points were emphasised less, the thematic analysis also highlighted that some teachers were more comfortable with traditional teaching methods, and some focused on 'getting through the lesson' rather than ensuring student understanding and engagement. This suggests that some teachers are still struggling with integrating technology into lessons.
As in Sprint 1, the ability of teachers to answer students' questions was a very significant shortcoming in their ability to integrate the technology. They were able to do this in 60% of lessons (across all models), compared to 57% in Sprint 1.

Classroom and time management were other frequently observed implementation challenges.
Classroom management challenges were more frequently noted in the tablet models. The thematic analysis provides some answers as to why this was the case. A very significant factor in the difficulty of classroom management was students talking and getting distracted by being in groups for the in-class tablet model.
In addition, it was difficult for teachers to manage time well with both models. Teachers managed time well in 60% of the MMC-based lessons and 67% of in-class tablet lessons. It appeared that, in many cases, teachers had to rush to finish the lesson within the scheduled class time.
It is worth noting that, in the in-class tablet model, tablets were used for 16.8 mins per lesson (35% of lesson duration), whereas in the MMC model, MMCs were used for 9.6 mins (20% of lesson duration).
Other difficulties specific to managing MMCs included space constraints in the classroom. For example, teachers struggled to use the blackboard and the MMC, and it was observed that students in the back of the class struggled more to see and hear the videos. They also included moving the MMC to the classroom. A majority of schools stored the MMC in an office to ensure the safety of the device 4. There was a range of positive signals that children engaged with and understood the digital learning content, positively affecting their motivation.
The thematic analysis emphasised a range of themes demonstrating comprehension of the digital learning content and signals that it was enhancing learning. In interviews, teachers reported that the Kolibri content appeared to be improving learning and that learners were responding positively to the content.
Additionally, the thematic analysis emphasised themes showing engagement and motivation. Teachers said that students enjoyed classes using technology with increased motivation, and students seemed to be more engaged. For example, students were observed responding to questions and solving exercises. These themes were present across all models.
One additional benefit of the tablet-based model is that it seems to enable students to participate more in group work. This occurred in 83% of the in-class tablet lessons and only 20% of the MMC-based lessons. Additionally, students asked questions about the content in 67% of the tablet lessons and only 20% of the MMC lessons. This indicates that tablet-based models were more engaging, but this may also have led to difficulties regarding classroom management. Student discussion, even when it is

Testing EdTech Interventions in Bandarban: Key Findings
about the content, contributed to noisier classrooms, which were difficult to manage.
In addition, it was found that shorter videos were more engaging for children as learners seemed to pay less attention to longer videos.

5.
The two interventions are prohibitively expensive, although the handover of operations, management and monitoring to local government could reduce this significantly.
The estimated cost per year per child of the interventions is provided below.
The costs have been estimated using the EdTech Hub's Lean Cost Model. In collaboration with Agami, the costs required to deliver both interventions in 31 schools were categorised, added up, and divided by the estimated number of learners in the schools.
For each cost, an estimate was also made for the impact of operating at a national scale for that specific cost. The national scale was defined as 'all Grade 6 learners in Bangladesh'. For example, it was estimated that the cost of monitoring (per year per child) at a national scale would be 60% of the cost in 31 schools due to efficiencies of scale. Similar percentages were estimated for each cost, and these were then applied to give the overall estimated cost per year per child at a national scale.
This cost model gives an estimate of the relative costs of each model. As the intervention is refined, we recommend that more rigorous and detailed cost modelling form part of further research. ■ Efficiencies of scale with project management, coordination, and monitoring costs.
The cost per year per child at scale could be reduced further if, following handover to the government, the project management, operations, and monitoring of the interventions could be absorbed into the existing functions of national and local government.
A breakdown of the costs for each model can be found in Figures 1 and 2 below. This breakdown relates to the costs of operating in 31 schools (2,200 children) for one child for one year.

Figure 2. Breakdown of costs for the MMC model (in USD)
It is important to note that for the MMC model, the cost of infrastructure set-up is not included in the cost model. This is because the set-up of the MMC was not specific to this intervention and was implemented in schools prior to this project. Additionally, the cost of MMC maintenance is also not included, as this is also a cost that is assumed to be absorbed in work outside of this project. Given the technical issues observed with MMCs, if future implementation included a maintenance model and provision of reliable electricity supply, then the cost per year per child would increase.

Although both EdTech interventions suffered from technical difficulties, difficulties with the MMC were more frequently observed.
In classroom observations, it was observed that the MMC functioned properly in only 40% of instances. In the in-class tablet model, this figure was 83%.
The thematic analysis demonstrated a range of technical difficulties with the MMC model. The most significant of these was reliable electricity supply. Other significant issues were variations in the sound of the MMC (it was either mute, too low, or too Testing EdTech Interventions in Bandarban: Key Findings high, with the teacher unable to rectify it) and the time-consuming nature of setting up the MMC, which often led to significant delays to the start of lessons. During the training, teachers and head teachers (42%) cited reliable electricity supply as a concern. Less frequently, issues with the mouse and screen were also noted.
The thematic analysis showed the main difficulty encountered with the tablets related to the headphones not working, which made the content difficult to understand in noisy classrooms.
7. Ensuring equity remains difficult in a classroom environment. Language appears to be the most significant barrier, followed by gender, educational performance, and digital literacy.
The thematic analysis emphasised that it was very difficult for teachers to ensure equitable engagement. This theme was observed in 53% of the schools. This difficulty was most frequently reported in MMC-only classrooms (50% of all recorded instances), followed by 25% of occurrences in the tablet-only classrooms and 25% in the MMC + tablet models. The most significant form of inequity that impacted engagement with technology was the language barrier, with children unable to understand Bangla well because they do not speak it at home and were not introduced to Bangla in lower grades. As with normal instruction (as normal lessons are also delivered in Bangla), non-Bangla speaking learners struggle to understand the Bangla Kolbri content. Teachers attempt to mitigate this by explaining concepts in local languages if they are able to. If the teacher is unable to, they often seek help from other students who are more able to communicate in Bangla and the local language.
The second most emphasised form of inequity was gender, with boys being more responsive and enthusiastic about technology than girls. Digital literacy and academic achievement were additional sources of inequity, with higher-achieving and more digitally literate children being more likely to engage with the technology. For example, in several classrooms, learners with higher digital literacy skills were appointed group leaders and primarily operated the tablets. It was clear that not all learners had an opportunity to use the tablets, and when they did, students with lower digital literacy skills struggled to do so.
It was noted that some teachers attempted to ensure equitable engagement through seating arrangements or asking specific learners questions.

Conclusions
The evidence suggests that MMCs are a promising intervention. Their effectiveness and implementation are promising avenues for further research.
It is notable that learners in MMC classes had higher scores in post-lesson worksheets, and these were achieved at a lower cost. Although this is a significant finding, it is important to note that the worksheet data has significant limitations (see Section 4.3). Furthermore, the cost of putting MMCs in place has not been factored into the cost model.
Also, themes relating to increased student motivation and engagement were present across all models in classroom observation and interviews with teachers. One reason for the promising results may be that the MMC model is less different to traditional, lecture-based models and easier to implement. Interviews with teachers in the needs assessment phase highlighted their feeling more comfortable with unidirectional teaching methods, which are more similar to working with MMCs.
Further research to validate these promising results is recommended, given the small sample size and limitations of worksheet data. In addition, an end-to-end assessment of the costs of the MMC model should also form part of future research.
This study indicates that the MMC model is promising, although iterations are needed to address several limitations. These limitations may be countered by the recommendations below for teacher training to be both more targeted and more focused on classroom management and lesson plans. Additionally, reliable electricity supply was a very significant issue. Where electricity is not reliably available, solar power and a maintenance model should be provided (although this may increase the cost). The MMC is also more appropriate for smaller class sizes. In larger classes, complimentary use of tablets or no-tech materials could be ensured.

Teacher training
Teacher training was shown to be effective. This needs to remain core to the intervention, with additional targeting towards those who most need it.
The evidence indicates variability in teachers' willingness and ability to integrate technology into their instructional practices. While some teachers are able to integrate technology effectively, a significant minority are not willing or able to: ■ Check students have understood the content by asking questions.
■ Answer students' questions relating to the digital content.
■ Explain the digital content effectively.
■ Use worksheets and other no-tech learning materials alongside the digital content.
The training was shown to be effective in improving teachers' understanding of their role in the classroom and their digital literacy and should remain core to the intervention. Future training should focus on the bullet points listed above.
In addition, methods to target specific teachers should be explored in the future, including providing multiple training sessions or developing teacher support networks via WhatsApp.

Modalities
The study indicates that both the out-of-class tablet model and combining the MMC and in-class tablet model are not appropriate for low-capacity teachers. These should be avoided in future research and implementation.
Out-of-class tablet models were found to be burdensome for teachers and learners. The evidence also indicates that introducing tablets in schools but out of classrooms creates a heavy burden for teachers and often requires both teachers and learners to adjust the number of hours they spend in school.
In the case of the MMC + tablet intervention, teachers struggled to integrate both technologies into the same lesson due to time constraints and limited digital literacy skills. Future research and implementation in marginalised areas, particularly where teachers have low capacity, should avoid combining interventions without a very clear design for how they complement each other in the classroom.

Equitable access
It is very difficult for teachers to ensure equitable access and engagement for all models. Future interventions should explore strategies for addressing this.
These strategies could include purposeful seating arrangements, ensuring students rotate when operating tablets, and teachers' asking specific learners questions. These strategies were implemented successfully by some teachers in the classroom.

Time management
Time management was highlighted as a significant issue. As part of the intervention, teachers should be provided with detailed lesson plans.
Lesson plans should provide step-by-step guidance for teachers on using the technology to facilitate the Kolibri content, explain content, and engage students. In particular, the plans should suggest timings for different elements of the lesson so that teachers do not feel rushed into 'getting through' the lesson.
In order to ensure lesson plans are feasible for the time allocated for MMC lessons, the lesson plans should be designed with the users involved, tested with teachers in a real-world setting, and developed iteratively.

Next steps
The study indicates that other modalities in the classroom could complement MMC-led lessons to increase equity and effectiveness. A carefully designed intervention could be the basis for future implementation and research.
MMCs seem to be the most cost-effective intervention, and their effectiveness and equity could be enhanced further by teacher training, step-by-step lesson plans, and strategies for equitable engagement (highlighted above).
In addition, increasing equity and effectiveness could also be achieved by introducing complementary modalities, both inside and outside the classroom. It would be crucial to design the role of complementary interventions and share this clearly with teachers, as our evidence also indicated that low-capacity teachers struggled to combine the MMC and in-class tablet model. These complementary modalities should be a basis for future implementation and research and could include: ■ Providing tablets for use by specific learners in the classroom ■ No-tech, pedagogically innovative content ■ In areas with greater access to feature phones, it is possible that feature-phone-based content (delivered via SMS messages or Interactive Voice Response) could be introduced Content in these complementary modalities should be identified and developed to be more accessible for students struggling with Bangla language content and could provide extra support to students who are struggling to engage with the content in the time allotted during lessons. This would require proactive and rigorous scouting for effective content, focusing on content in languages other than Bangla.