Probable Causation

View Original

Episode 23: Lelys Dinarte

Jennifer Doleac

Lelys Dinarte

Lelys Dinarte is an Economist in the Human Development Team of the World Bank's Development Research Group.

Date: February 18, 2020

Bonus segment on Dr. Dinarte’s career path and life as a researcher.

A transcript of this episode is available here.

See this content in the original post

Episode Details:

In this episode, we discuss Dr. Dinarte's research on after-school programs as a violence-reduction strategy:

OTHER RESEARCH WE DISCUSS IN THIS EPISODE:

Transcript of this episode:

 

Jennifer [00:00:06] Hello and welcome to Probable Causation, a show about law, economics and crime. I'm your host, Jennifer Doleac of Texas A&M University, where I'm an Economics Professor and the Director of the Justice Tech Lab.

 

Jennifer [00:00:17] My guest this week is Lelys Dinarte. Lelys is an Economist in the Human Development Team of the World Bank's Development Research Group. Lelys, welcome to the show.

 

Lelys [00:00:26] Hi, Jen. Thank you so much for having me here.

 

Jennifer [00:00:29] We're going to talk today about your research on anti-violence programs for school children in El Salvador. But before we get in to that, could you tell us about your research expertise and how you became interested in this topic?

 

Lelys [00:00:43] Okay, sure. So I am a Research Economist of the Development Research Group at the World Bank, and my primary research fields are education and development economics with a focus on human capital and crime. The main goal of my overall agenda is to study in a rigorous way the impact of some policies that aim to promote the optimal human capital accumulation for people living in low and middle income countries, particularly in Central America. A particular strand of my agenda examines how a specific educational interventions and also the way how they are implemented can determine the student's cognitive and non-cognitive outcomes, including social emotional skills and the violent behaviors. As you mentioned in this talk, we are going to discuss two of my works on afterschool programs and violent behaviors in El Salvador.

 

Lelys [00:01:32] Why did I become interested in this topic? Well, existing evidence indicate that violent crime in general can increase the economic cost of health injustice services. And that can be great obstacles to economic growth, particularly in El Salvador, and similarly to other countries in Central America, crime levels are so high that, for instance, since 2009, El Salvador has been declared as a victim of a epidemic of violence. In 2015, for instance, the homicide rate among boys with ages between 10 to 19 years was 66 murders, more than 12 times the worldwide homicide rates the same year, making on El Salvador one of the world's deadliest places for young boys.

 

Lelys [00:02:20] What are then the consequences of these high levels of violence? First of all, there are - they can hinder human capital accumulation. El Salvador, for instance, has experienced a 13% reduction in its the school enrollment rate, with over 18%of its students reporting that they dropped school due to delinquency. Another concern is the snowball effect of violence and ————— month in their paper in 2014 finds that exposure of adolescents and children to violent neighborhoods increase their subsequent criminal behavior. Considering the El Salvadorean contexts and the existing evidence, I started this project aiming to identify how the risk factors of violence can be addressed. I found out that after-school programs can constitute a natural set up for this objective. Why? Because they are examples of programs that can provide two services. First, protection. They can keep children under four months probation  to prevent victimization and violent behaviors. And also they can provide a learning service. They can act as an alternative source of social development, especially when they include a specific curriculum oriented to foster social emotional skills, for instance, and control of impulsive behaviors in particular. And that's how I identified and partnered with a local NGO in El Salvador, Glasswing International, with extensive experience working on after-school clubs in Central America.

 

Jennifer [00:03:55] So your paper is titled "Preventing Violence in the Most Violent Contexts: Behavioral and Neurophysiological Evidence from El Salvador," and it's coauthored with Pablo Egana. In it, you consider the effects of an after-school program, as you mentioned, that incorporated cognitive behavioral therapy, CBT for short, as well as other activities. So tell us more about this program. Where did it take place and what were the activities like?

 

Lelys [00:04:19] Sure. So this after-school program was implemented in 5 public schools located in highly violent communities in El Salvador. According to the Intervention Theory of Change, the main objective of this program is to successfully modify children's violence, misbehavior, and also attitudes towards the acquisition of life skills. And that's expected to also improve their academic performance. The after-school program consisted on class that met twice a week for approximately one and a half hours per session right after school ended. Each session has two parts: social skills development and the traditional club curriculum.

 

Lelys [00:04:59] The first section is common to all participants, and it includes activities oriented to foster social emotional skills. Some of them were inspired by CBT activities. Specifically, it tries to raise participant awareness of certain behaviors to disrupt these patterns and also promote their better ones using a learning by doing approach. It includes topics such as conflict and impulsiveness management, self discipline, and soft skills. As one example, if the topic was impulsive management, then students participate in this role play where the instructor asked them to provide alternative to get something from a clubmate. Some of them suggest to retrieve the object either by hitting it or hitting the clubmate, which is their automatic response to how they are used to respond. Then the tutor discussed with them, some alternatives such as negotiation or simply asking for the object.

 

Lelys [00:05:59] Then - that's the first part. The second one, it included some extracurricular activities related to each club category, such as art, culture, sports, and science. The objective of this section was to motivate students to participate, making the learning process more fun and interactive, and also increase independence to the after-school program. Just to give an example of the second section in the science category, discovery clubs offer students the opportunity to do experiments such as mock volcano eruptions. In the sports category, children play the soccer or basketball, among others. So that's the structure of the - the intervention.

 

Jennifer [00:06:43] So when the practitioners were setting up this program and decided to include both the social skills development and these other activities like art or experiments rather than just the social skills component alone, was there - was the hope that the other activities would add additional value or were they just there as a carrot to get kids to sign up?

 

Lelys [00:07:02] Well, this approach, it can be more effective than a full therapy intervention in the Salvadorean context because at the edges of our target group, which were between 10 to 16 years of age, that's when in El Salvador appears the greatest manifestation of violence. And as you can imagine, adolescents may find it's also unappealing to learn impulsiveness management and self-discipline alone. Those - to warrant participants attendance, it was necessary to complement the therapy with some recreational activities. That was one part. In addition, park spaces to play are not used very often in El Salvador because of the fear of exposure to violence. Therefore, warranting these time for recreation in a secure space made intervention also more interesting for the participants. So they were more a way to guarantee participation from the students.

 

Jennifer [00:08:01] And as you just mentioned, the program targeted students ages 10 to 16. So tell us more about why this age group was considered the most important to include.

 

Lelys [00:08:09] Yeah. So it was for two reasons. First, because soft skills are malleable during adolescence, according to the evidence that's pointed out by James Heckman and Tim Kautz in their paper 2012. And also soft skills can predict- affect individual success in life. The second reason is that anecdotal evidence in El Salvador indicate that at this age is when adolescents are more prone to be violent or to enroll in gangs. We are aware that some adolescents in the country have no more options than just be part of a gang. With this intervention, we aim to just tag these children as having spatial skills that are not what gangs need for data operations. Unfortunately, the context didn't allow us to measure the extensive margin of gang affiliation, just changes on soft skills. Specifically, the dangerous context of public schools in El Salvador prevented us from directly asking students about whether they were a part of a gang during the registration phase or even during the follow up.

 

Jennifer [00:09:20] So, okay, so let's talk about the empirical challenges to studying the effects of programs like this. When we see interesting programs out in the real world, we might be tempted to just compare people who participated in those programs with people who didn't. But of course, there could be lots of reasons those two groups have different outcomes that have nothing to do with the program itself. Maybe the participants are more motivated or from wealthier families or something like that. So as you became interested in studying this program, what were the primary hurdles to understanding whether this particular program has beneficial effects?

 

Lelys [00:09:52] Yeah, sure. So I can identify at least the three issues to study the effectiveness of an after school program in a highly violent context. The first one, as you briefly mentioned, there are some identification issues. In an after-school program context it's difficult to have a credible comparison group for several reasons. One of them is that in some cases, participation is voluntary. And we may expect that the most motivated students are the only one who wants to be enrolled. And then comparing enrolled with non-enrolled children may actually identify a biased effect of this intervention.

 

Lelys [00:10:30] A second issue, and related also to the first point, are the presence of spillovers. We learn from our peers all the time, spacially behaviors and from those which we are exposed most of the time to. If you implement the intervention in which beneficiaries and non-beneficiaries are in contact, which is the case of schools and students in classrooms, you will definitely have some spillovers. We call it contamination in economics, and it can affect the exact measure of the impact since we can under or over estimate the true effect of the program.

 

Lelys [00:11:06] Finally, the third issue is data restrictions. I'll say that that was actually the main challenge of doing empirical work in a low and middle income countries, according to my experience. In particular, educational systems in these countries focus on collecting basic information on school enrollment, attendance, and in some cases some measures of academic performance. However, there is lack of data on non-cognitive outcomes, such as social emotional skills, violent behaviors, and others. If we also add a highly violent context in this equation, then data collection becomes the most important issue here because we cannot collect all the data that we may be interested into.

 

Jennifer [00:11:52] And we will talk about - a bunch about those data that you were able to collect. But before the study, what had we known about the effectiveness of after-school programs broadly and CBT based programs in particular?

 

Lelys [00:12:04] Okay, so. Well, regarding the evidence of the effectiveness of after-school programs on delinquent behavior, for instance, this evidence is mixed and inconclusive. Some area papers show that these after-school programs are interventions that can protect children by preventing victimization or delinquent behavior. However, a recent systematic review and meta-analysis developed by Sema Taheri and Brandon Welsh show me mixed results from some well known evaluations. They - their analysis is based on 17 studies, and they find out that there is evidence that after-school programs add a small but non-significant effect on delinquency of around .06 standard deviations.

 

Lelys [00:12:48] The other main conclusion is that if we want afterschool programs to be effective, then we must include some elements with a focus on delinquency prevention. In other words, as I discussed before, we should explore the potential of after-school programs as an alternative source of learning and social development. Moreover, despite an increase in the number of these sort of programs as implemented over the past years and the high incidence and economic cause of crimes in low and middle income countries, most of the existing literature of the impact of these programs are using data from high income countries where the issue of violence is not as relevant as those - as they are in low and middle income countries.

 

Lelys [00:13:39] Then, regarding the evidence on cognitive behavioral therapy in particular. There is a recent white paper from Jay Powell, led by Thomas Abt, Chris Blattman, Beatriz Magaloni, and Santiago Tobon that indicate that CBT is an approach that leads to at least temporary behavioral change among juveniles and adults across different settings, from youth in Chicago to older adults in Liberia. Moreover, recent economic evidence has been analyzing mechanisms underlying the impact of these CBT inspired programs, such as changes in self-control, time preferences, social skills, and social identity. There are two seminal papers that started this type of intervention in their mechanisms in the literature, in economics. The first one, Sara Heller and coauthors experimentally studied the program Becoming a Man implemented in Chicago. What they find is that participation in the program reduced total in violent crime arrest and also improved the school engagement.

 

Lelys [00:14:40] Another seminal paper in this literature in economics is that one authored by Chris Blattman, Julian Jamison, and Margaret Sheridan. The author studied the complementarities between CBT and also cash transfers and how they are - they can reduce crime. What they are finding is that both cash alone and therapy alone initially can reduce crime and violence. However, these effects dissipates over time. What really works in a intermediate term is the combination between cash followed by therapy.

 

Jennifer [00:15:18] So you took advantage of the fact that more students wanted to participate in this program in El Salvador than the organization could accommodate and randomized which school kids got in. This means you had a randomized controlled trial, an RCT, with a control group that allows you to measure the effectiveness of the program on the kids that participated. It's often quite difficult to convince policymakers and practitioners to run an RCT like this. So could you give us some background on how this experiment came about?

 

Lelys [00:15:47] You're totally right. Convincing policymaker or practioner is usually one of the most complicated task when you are preparing any evaluation. And you want to propose an experimental design. There were several rounds of discussion with the NGO about this. For instance, first, I presented them all the non-experimental alternatives that are available to evaluate programs and there are specific requirements. At the end, we agreed that there was no chance to evaluate intervention using a non-experimental approach. For instance, we didn't have data - past data on academic performance or violent behaviors to implement a difference-in-differences approach. For this reason, we agree with the NGO that the best way to implement the evaluation was through an RCT or experimental evaluation.

 

Lelys [00:16:40] But after you convince your implementing partner, you have to also convince the participants. We needed to convince not only school principals and teachers, but also students and parents that the fairest way to assign the available slots for the after-school to the children wasn't through a lottery. And since the program was going to be implemented in the future at the school, so there will be more opportunities to those not attending in 2016. And that was the argument that we used with them. We were then able to collect information and follow up from children in the control group because we gave them like a enrollment coupon, which they can redeem the year after. And that warranted their participation in the traditional closing date 2017 academic year.

 

Jennifer [00:17:36] So, okay, so you randomly assigned the kids who want to participate in this program into several groups. You have a control group where the kids don't get to participate that year, but it sounds like they got - they got to participate the following year. But then you have three different treatment groups. So tell us about each of these groups and a little bit about why you did this.

 

Lelys [00:17:54] Okay, sure. So what I wanted to learn was not only whether intervention worked are not, which is - which I will address just having a treatment and comparison group. However, I also was interesting into learning how these interventions will be implemented and exploit potential peer effects. From the existing evidence, it's not clear whether intergratio or segregation in terms of violence can have varied effects on the outcomes that we are analyzing. On the one hand, you can - you can say that integration can be beneficial, especially for the most violent individuals because they are exposed to good peers. However, concerns related to potential contamination of the less violent participants or budget restrictions can make  practitioners to separate the students by their violent behavior and/or focus on treating just the most violent ones. For this reason, I design a tracking by balance experiment. It was inspired by two works: a paper of Esther Duflo, Pascaline Dupas, and Michael Kremer in 2011, and also the paper of Jeanne Lafortune, Marcela Perticará, and José Tessada in 2016.

 

Lelys [00:19:10] It consists in the following year. I estimated a propensity for violence of each enrolled child, and then they were assigned to two groups: to a control and a treatment. Then, in a second randomization stage, I randomly assigned students in the treatment group to a particular composition of peers in terms of violence. In other words, to a sub-treatment arm. 25% of them were assigned to a heterogeneous composition of peers regarding violence and 50% to a homogeneous treatment arm. Within this homogeneous treatment, I ranked all the students and assigned to subgroups groups according to their index. All students with a propensity for violence greater than the median at a certain level were assigned the high violent group and the rest the low violence group. So here, comparing treated and control, I can - I can measure the impact of these interventions. But then, by comparing each of these sub-treatment arm: heterogeneous or homogeneous composition, I can generate evidence on how the composition of peers can have some differences in the impacts on misbehavior or academic performance, among others.

 

Jennifer [00:20:31] That's great. Yes, so the main results we're gonna talk about shortly are going to be the average of - across all those treatment groups, but then we will dig in later to what we're learning from these different treatment groups. It's a really clever design. So randomizing kids across the groups allows you to measure the causal effect of the program on outcomes. But the other crucial piece, as you mentioned, is data. So you used a mix of administrative data and survey data to measure the effectiveness of this program. Tell us about those data sets.

 

Lelys [00:21:02] Okay, sure. So during the enrollment phase, before the itervntion, we collected two strands of data. First, enrolled students provided personal and family information on the registration forms. For instance, we got their information on their age, gender, household compensation, mother's education, among other characteristics. We also collected school records of grades, behavior report, absenteeism, data from both enrolled and non-enrolled children. Then, at the end of the intervention, after seven months of implementation, follow-up data on non-cognitive outcomes were collected, but only from the enrolled participants in school facilities, those who consented to give us access to their information. So this was at the end of October 2016, more or less, after all class completed their curricula.

 

Lelys [00:21:53] The follow-up survey included questions to measure interventions impact on students' attitudes toward school and learning, and also violent behaviors. Due to limitations of their reported data, we attempted to also recheck and validate these behaviors using proxies for these outcomes obtained from admin data. For example, school attendance in the school records is a good proxy for a student's positive motivation towards school and learning. In addition, misbehavior at schools reported by teachers is also a good proxy for violent behaviors. Finally, at the end of the academic year, schools provide the reports of grades as part of our follow up data.

 

Jennifer [00:22:33] And then when you divide the kids based on their propensity for violence and one in the - at least two of the - two of the three treatment groups, what information do you use to predict that propensity in terms of the baseline data? And then in general, what do the kids in the high risk group look like compared with the kids in the low risk group?

 

Lelys [00:22:52] Yeah. So as I mentioned before, the dangerous public schools context in El Salvador prevented us from directly asking students about violence during the enrollment phase. The school in — not only the intervention, but also, most importantly, the children. So to address this constraint and because the design required to have this propensity for violence or a measure of violence and to create these proxy, I follow Chandler, Steven Levitt, and John List in their 2011 paper, and I estimate a predictive model of violence and crime from an existing data set of a similar population also in El Salvador. So it had two stages. The first one, I estimate that the likelihood of having committed a violent act as a function of a wider range of a covariate, such as a student's characteristics, let's say age, gender, time spent alone at home, other children's household variables such as the residence area, mother's education, the household composition, which is also very relevant to El Salvador, given by the violence and also migration patterns, and also on school level controls. Using these existing data set of ———— Salvador and these data was included information of also youth living in the violent communities. Then in a second stage, I exploited the availability of these variables in their registration forms of disenrolled students in my study. So I use that these variables to predict measure of propensity for violence for each child using the vector of estimated coefficients from the first stage.

 

Jennifer [00:24:50] Okay. So let's talk about the main results now. What do you find is the effect of being assigned to the treatment group on kids' academic outcomes and behavior.

 

Lelys [00:25:01] So well compared to students in the control group, I am finding that after-school program participants report having better attitudes towards school by 0.17 standard deviations, and also they report spending 16% more time each day doing their homework. This improvement in attitudes is also confirmed when we use administrative data. We are finding that treated students, reduced their school absenteeism by 23% compared to the control group. These magnitudes are actually pretty important according to comparable studies. Then, in terms of violent behaviors, we are finding that after seven months of this program, the students are reporting having committed fewer delinquent actions and being less violent compared to the report of the students in the control group. The magnitudes are between 0.14 to 0.18 standard deviations. Similar effects are found when we also use a teachers report of students misbehaviors. In particular, we are finding that when students were randomly assigned to participate in the program, they reduced both their bad behavior at school by more or less 0.17 standard deviations and the probability of having a misbehavior report the extensive margin by 6 points.

 

Lelys [00:26:28] Moreover, although these programs do not relate directly to academic outcomes because they are not learning during the after-school program, something about math or reading, et cetera, we are finding evidence or theories in the literature - or actually evidence of a positive correlation between academic results and social skills. We are finding in, according to our results, that this program has a positive effect on grades with magnitudees that are between 0.11 and 0.13 standard deviations.

 

Lelys [00:26:59] The question here is how these intervention that only teaches social skills indirectly affects participants academic performance and grades. There can be at least two explanations. First, the after-school program can actually modify student's classroom misconduct, reducing disruptions that affect their learning or that of their classmates. Second, the literature in economics and also in psychology shows that cognitive skills and academic performance are defined by non-cognitive skills, such as future orientation and attitudes towards school and learning. So those are the two plausible explanations of how these intervention is also generating some effects on academic performance.

 

Jennifer [00:27:41] You then consider both spillover effects and peer effects of the program. So you're interested in whether the kids from the control group might indirectly benefit because they're in a classroom during the day with program participants. In other words, you're interested in whether the behavior change caused by the program might rub-off in some way on the control group. So how do you test for this and what do you find?

 

Lelys [00:28:04] Yeah, so the argument behind the relevance of a spillover is that treated students can spread their improved non-violent behaviors to non-treated classmate via at least two channels. The first one is a bitter school climate. And if treated children are less disruptive during classes or if they behave better, then these can reduce violence between - within schools. And the second one is that the interaction between treated and non-treated students can also allow the last group to imitate or learn some skills from the former. So those are the arguments - why measuring spillovers in youth contexts are relevant.

 

Lelys [00:28:46] In this design, I have a samples of enrolled children, which include the treated and comparison group and non-enrolled children, those who decide not to participate into the after-school program. And then I exploit a non-experimental variation in the share of students who were randomized to treatment across courses to estimate the spillover effects. What are the assumptions to identify these effects here? Well, since the assignment on - to the treatment was done at the school by education level block and each level includes pre-courses the share of enrolled children allocated to participate in the program at each course was quasi-exogenous.

 

Lelys [00:29:32] After controlling, of course, by the share of students at each course who decide to enrolled. Considering these, we followed the model developed by Scott Carrel and Michael Hoekstra to measure the after school program spillover effects on non-enrolled students. And what I'm finding is actually something pretty important and interesting. Our results indicate that the interaction of students with a greater share of after-school program participants generates positive effects on their academic grades and also reduce their bad behavior at school. Actually, we find that the effects are pretty important. Adding three treated students in a classroom of 22 increases academic achievement up to .0.10 standard deviations and also reduce bad behavior report by almost 0.15 standard deviation. So the results we were discussing before are actually lower bounds of total effect from these type of interventions.

 

Jennifer [00:30:35] Right, and then just to clarify, that's the downward bias in the treatment effect you're measuring is because the control group, as you're showing here, is effectively treated somewhat too. So you're going to measure the net effect of that.

 

Jennifer [00:30:48] Okay. So then you actually have a second paper that digs into peer effects a bit more. That paper is called "Peer Effects on Violence. Experimental Evidence in El Salvador." You use multiple treatment groups, as we discuss, to consider whether it matters who's in the program with you. That is, you're interested in whether peer effects within the program can increase or decrease the program's effectiveness. So remind us how you're testing for this and tell us what you find.

 

Lelys [00:31:13] Okay, sure. As you may recall we discussed that a - within a treatment arm, I run a second randomization and assign 25% of the student to a hetergeneous composition of peers and 50% to a homogeneous treatment arm. Heterogeneous in this context means that for instance, if we have a propensity for violence that goes from one to 100, then heterogeneous composition means that all children with any propensity for violence are going to be treated together. Homogeneous composition, on the other hand, means that if it's a high-violent, homogeneous group, then you are going to have just children with a propensity for violence between 51 to 100. And a low-violent, homogeneous group is going to be formed by students with a propensity of 1 to 50.

 

Lelys [00:32:02] So I - hear I compare each of these treatment arm from homogeneous or heterogeneous with the control group. I also estimate some hetergeneous effects by measuring propensity for violence between the two treatment arm, between the two group compositions, and also, I estimate, the effects on the marginal students. Let's talk first about the results when I compare the group composition effect. First of all, my results indicate that from the comparison between each treatment arm: homogeneous or heterogeneous with the control group, any group composition can be more effective than no treatment in the improvement of behaviors, emotional regulation, and academic performance.

 

Lelys [00:32:44] Then, when I compare both treatment arms between each other, the estimated results indicate that heterogeneos composition of peers have better impact on most of the outcomes than the tracking by violence treatment arm. The results are consistent with the evidence that interaction with diverse peers can generate differences in the learning experience. These results may also have different interpretations. First, that diversity regarding violence can be more beneficial because it allows high violence students to be exposed to the less violent children, and they can therefore learn social skills and good behaviors from them. Similarly, it seems that, according to my results, low violence children are not being contaminated. On the contrary, they are benefiting from being exposed to misbehaviors that that they must avoid. However, if students in a homogeneous composition of peers are losing the opportunity to learn from behaviors of the other - or the other half of the violence distribution function.

 

Jennifer [00:33:53] And then finally, in the second paper, you also test for what happens to the marginal student, so the kid who's just on the edge of being assigned to the low risk or high risk treatment group. This student will either be the highest risk kid in the low risk group or the lowest risk get in the high risk group. So how do you test for how group assignment affects this kid and what do you find?

 

Lelys [00:34:16] Well, yeah, so that's the third part of these - the results from this paper. And that's the effects on the marginal students, the homogeneous group actually provide a natural set up for a regression discontinuity design with the median of the propensity for violence distribution in each of the stratum has a discontinuity. What I am finding here is that being the least violent in a highly violent group negatively affect behavioral and emotional regulation related outcomes, and also academic performance. For example, my results indicates that assigning a marginal student to a group of peers with a higher propensity for violence reduced her self-report of attitudes towards school and learning by 0.6 standard deviation. That's a huge negative impact, and also increase the probability of failing at least one course by almost five points. This is consistent with the existing evidence of Billings and coauthors that there is an endogenous formation of groups of badly behaved students when they are segregated. They are, as we mentioned before, they are losing the opportunity to learn from other peers with a different propensity for violence or with different behaviors.

 

Jennifer [00:35:33] So I want to talk about the mechanisms driving all these effects, but let's pause for a moment to summarize all of these results, because we've talked about a lot of them, on the peer effects and spillovers. So what are the main takeaways from your experiment for practitioners who want to implement a program aimed at reducing violent or risky behaviors? Who should they target and how should they assign them to groups to maximize the program's effects?

 

Lelys [00:35:56] Yea, so the results indicate, that the after-school program works in the context of a highly violent and low income country. It increases not only attitudes towards school and learning, but it also is improving misbehavior at school and also academic performance. In terms of spillover, non-enrolled students are also benefiting from these interventions. My results indicate that being exposed to a greater share of treated classmate improve the non-enrolled children's behavior at school and academic performance.

 

Lelys [00:36:31] In addition, the way this intervention is implemented is also relevant. There are important peer effects on non-cognitive outcomes that can be exploited to increase the effectiveness of this program. The tracking by violence experiment is showing that the improvements on non-cognitive skills and stress reduction are larger when participants are in more diverse or heterogeneous group than in segregated ones, ones in terms of violence for both high and low violence children. In other words, despite there may be important budget restrictions for implementing this program that can tempt practitioners and policymakers to focus on treating just highly violent children, this can generate actually unintended effects of this intervention and end up affecting mostly the ones that are supposed to benefit from - more from it, which are those with a greater propensity for violence.

 

Jennifer [00:37:31] Okay, so let's talk about mechanisms. Let's go back to the initial paper we were discussing. You actually measure the brain activity of students to better understand how the after-school program is affecting their behavior. That is incredibly cool. So tell us more about that. How do you measure their brain activity?

 

Lelys [00:37:51] Thanks, Jen. Well, according to recent evidence, the type of emotions that the individuals felt is relevant of many of their cognitive and behavioral outcomes. And the evidence also points out that these emotions can determine how people respond to some stimuli. In this sense, our argument from design is that if a student is able to control their emotions and automatic responses, then she will be capable to control the violent behaviors, and therefore it will be a mechanism of our main results. Having these facts - these evidence in mind, what we do with public life is to relied on emotion detection theory from affective neuroscience literature, and we use low cost, portable electroencephalogram recordings to obtain an approximation of children's emotional state and responsiveness to stimuli. In other - in simpler words, we brought these electroencephalogram to the field which are just like a headset. We brought them to the field schools, and then we exposed children to some images that generated positive, negative, or neutral stimuli to the children.

 

Lelys [00:39:04] Then, while we were showing them these pictures, we were capturing and saving students' brains' responses to such stimuli. Using the recordings we then estimated proxies of stress and emotional regulation. According to the neuroscience literature, arousal is a proxy for stress and also violence is a proxy for emotional regulation or of the other reaction to particular stimuli.

 

Jennifer [00:39:37] So using this new data, what are the results - what do you learn about why the after school program had such big effects on academic outcomes and bad behavior?

 

Lelys [00:39:47] Yeah, so the impacts we estimated on these neurophysiological outcomes indicate that the program reduced the over-reaction of participants, particularly to positive stimuli. These effects is important. These children are like more cool-minded, and that seems to translate into a better behavior at school. Our mediation analysis is also confirming that result. However, heterogeneous analysis by propensity for violence, a baseline unexpectedly shows that there is an increase on a stress level of treated students. The group composition analysis is also showing that the increase on stress levels is important for both high and low violence students that were assigned to these homogeneous compositional peers. These results also aligned with our previous conclusion that tracking by violence can have some negative effects, even in mental health as we are showing here.

 

Lelys [00:40:47] What can be - what could be potential explanation for these results? Well, maybe diversity's the social norm with these children usually perform. That's why they are used to. Does assigning them to just similar peers in making - may make them more stressed? As I mentioned before, selecting and treating together these only high violence students for these programs can generate some unintended effects from this intervention, particularly to the children who we are supposed to benefit to.

 

Jennifer [00:41:23] So both of these papers have been circulating for a little while now. What other research has come out since you first ran this experiment that's relevant to the effectiveness of CBT or more broadly, how to reduce violence among high risk students?

 

Lelys [00:41:39] Yeah, actually, in a joint project with Claudia Martinez from the Pontifical Catholic University in Chile and Pablo Egana, we have extended this project, and we are aiming to disentangle the protectionary learning channels of a after-school program. With the experimental design, we have been discussing in these two previous works, I am not able to rigorously separate if you fix our treatment because children are protected under these supervision after school time, or if they are exposed to a specific social emotional skills development. If they are learning something from these new activities. This project ———, that we are working on with Claudia and Pablo, is right now in the field and is being implemented in public schools in Honduras, Guatemala, and El Salvador.

 

Lelys [00:42:31] So the idea here, to disentangle the mechanism, the protection in learning, we are - we have a assigned some schools to the business as usual, just these extracurricular activities. And other schools are assigned plus to the extracurricular activities plus the particular curricula on these type of intervention, such as mindfulness or other psychological programs. In addition to high risk students, how the target population, I am also investigating the impacts of these psychological programs on other groups that are also affected by high levels of stress, such as police officers. With a team constituted by Sofia Amaral from ifo in Germany, Pablo Egana and Patricio Domínguez from the Inter-American Development Bank, we are now designing the impact evaluation of a mental health intervention for police officers in El Salvador. We are going to implement in these —— with partnering with the local NGO in the country. And the plan is that - to run a pilot next year and then the impact evaluation during the 2021 year.

 

Jennifer [00:43:46] Those both sound like really cool projects. So putting it all together, the results of this study and the other studies we've talked about, what are the policy implications of this work?

 

Lelys [00:43:56] Well, I'll summarize the whole project in three main takeaways. First, an after-school programs, that includes social emotional learning, works to reduce both violent behaviors in schools and also to improve academic performance in the context of a highly violent country. It is also important to highlight that, despite my experience of the design doesn't allow me to disentangle the learning protection mechanism. Suggestive results from my existing data indicate that learning component may be the main driver of these effects. However, that's something that we are aiming to show with these new project in these three countries in the region.

 

Lelys [00:44:39] Second, the intervention has important multiplier effects. The impacts are positive not only for the directly treated students, but also for those non-enrolled that are in contact with these treated children. And third, how these interventions is implemented is also relevant. There are important peer effects on non-cognitive outcomes that can be exploited to increase the effectiveness of this intervention. Tracking by violence experiments for instance, shows that diversity is super important. In other words, despite there may be important budget restrictions for implementing these programs, focusing only on highly violent children can generate unintended effects of this intervention, particularly on neurophysiological outcomes such as stress.

 

Jennifer [00:45:29] So you've already talked a little bit about ongoing projects that you have, but let's explore the research frontier a little bit more. What are the big open questions in this space that you and others that work in this area will be thinking about in the years ahead?

 

Lelys [00:45:45] So I will say that there are several open questions here. First of all, what are the long term effects of these programs? Are they sustained over time? It's not clear yet. And given that after one year of the intervention, these - the project that we have been discussing today, the control group was also treated. So my design might not allow me to measure long term effects.

 

Lelys [00:46:12] Also, more intensive intervention might be relevant if these highly - in these highly violent context, particularly to modify some more root decisions, such as enrollment in gangs. For example, we call a study complementarities of these after-school programs and other interventions that can help to address some issues for these youth, such as cash transfers, because the economic restrictions can also play an important role here. Then, in terms of implementation and to increase effectiveness of the program, I will say that another research opportunity here is to analyze a student's networks and try to exploit the structure of social networks to achieve even greater impacts. If we, for instance, treat and just know the central node of their network, we may have a greater effects from these  programs. Or shall we actually treat all the children because it seems that their interactions are playing an important role here.

 

Lelys [00:47:19] And finally, we are - and this is actually something important that we have been discussing here in the bank, we are implementing intervevntions that aim to maintain children in schools. However, a relevant assumption here is that schools are safer places for children than the streets or their households, which is not necessarily true in several countries. So in this case, shall we then move and implement these programs in ordered domains such as household communities, when in these countries where schools are not necessarily safe places for the children. So I will say that those are like some big open questions that are still around that we may start working on them very soon.

 

Jennifer [00:48:08] Lots of work to do.

 

Jennifer [00:48:09] My guest today has been Lelys Dinarte from the World Bank. Lelys, thanks so much for talking with me.

 

Lelys [00:48:15] Thank you again for the opportunity. I am looking forward to continue talking about these topics.

 

Jennifer [00:48:25] You can find links to all the research we discussed today on our website, probablecausation.com. You can also subscribe to the show there or wherever you get your podcasts to make sure you don't miss a single episode. Big thanks to Emergent Ventures for supporting the show. And thanks also to our Patreon subscribers. This show is listener supported. So if you enjoy the podcast, then please consider contributing via Patreon. You can find a link on our website. Our sound engineer is Caroline Hockenbury with production assistance from Elizabeth Pancotti. Our music is by Werner and our logo is designed by Carrie Throckmorton. Thanks for listening and I'll talk to you in two weeks.