Probable Causation

View Original

Episode 26: Kirabo Jackson

Jennifer Doleac

Kirabo Jackson

Kirabo Jackson is the Abraham Harris Professor of Education and Social Policy at Northwestern University.

Date: March 31, 2020

Bonus segment on Professor Jackson’s career path and life as a researcher.

A transcript of this episode is available here.

See this content in the original post

Episode Details:

In this episode, we discuss Professor Jackson's work on single-sex education:

"Can Introducing Single-Sex Education into Low-Performing Schools Improve Academics, Arrests, and Teen Motherhood" by C. Kirabo Jackson.

OTHER RESEARCH WE DISCUSS IN THIS EPISODE:

Transcript of this episode:

 

Jennifer [00:00:07] Hello and welcome to Probable Causation, a show about law, economics and crime.

 

Jennifer [00:00:11] I'm your host, Jennifer Doleac of Texas A&M University, where I'm an economics professor and the director of the Justice Tech Lab.

 

Jennifer [00:00:18] My guest this week is Kirabo Jackson. Bo is the Abraham Harris professor of education and social policy at Northwestern University and a faculty fellow at the Institute for Policy Research there. Bo, welcome to the show.

 

Kirabo [00:00:30] Oh, thank you very much, Jen.

 

Jennifer [00:00:32] We're going to talk today about your research on single sex education and how it affects students outcomes on various dimensions, including their criminal activity. But before we get to that, could you tell us about your research expertize and how you became interested in this topic?

 

Kirabo [00:00:47] Sure. So I'm an economist by training. My two fields are labor, economics and public economics and I've sort of I've always been drawn to education just because it's one of those areas where not only is it tremendously important, we spend a large amount of our public dollars on public education, but it also has the potential to improve a wide array of outcomes. You know, children spend countless hours every day. They spend, you know, about eight hours a day, starting from very young for about 12 years in the public schooling system, sometimes public, sometimes private. So this is a real opportunity to influence outcomes later on if we think that the exposure to these environments better for what they're going to learn and the skills are going to take the labor market and elsewhere.

 

Jennifer [00:01:34] So tell us about single sex education, you're going to be looking primarily at the secondary school level here, so high school, but how common are all boys and all girls schools more broadly, both in the U.S. and elsewhere?

 

Kirabo [00:01:45] That's a great question. So, you know, if we look at public schools in the United States is actually relatively rare. So for a long time, actually, due to Title nine restrictions, there was a restriction that basically said that any district, any public school district that wanted to provide, say, a school for boys, they would have to provide a school for girls of equal quality. So what that basically meant was that anyone who wanted to provide a school for one particular sex had to make sure they had to provide it for both, which sort of became a difficult hurdle for many school districts to overcome. So it's actually something that's relatively rare that we see in this country at the in the public sphere. In private schools, we see a lot in terms of private schools. There are also some Jewish schools also that use that basically have all girls or all boys school. So it's not uncommon in the private school sector, but it's pretty uncommon in the public school sector. Having said that, in 2005, the restrictions were sort of eased up a little bit in terms of what could be provided for by a school district. So now a public school district, as long as they provide equal quality co-educational opportunity, they're able to provide some single sex education. So since then, we've seen a growth in single sex schools in the United States. It's still not extremely prevalent, but we see it all over the place. Sometimes you might see single sex classroom offerings, which is something that we also see sort of growing in the United States. Outside the United States, it's actually pretty common. So the context I'll be studying is Trinidad and Tobago and then other countries like this, Trinidad and Tobago, I think about 10 percent of schools are single sex, which is a large fraction of the schools. In other countries, Korea, China, they have single sex schools there as well. In Europe, there are they're also pretty common. So it's something that actually is very, very common. It really sort of went out of favor in the United States. And it's a topic that I think is interesting, both inside the United States, because it's becoming increasingly common, but it's also something that educators outside the U.S. have been thinking about for quite some time.

 

Jennifer [00:03:44] So why do parents send their children to single sex schools, I guess related to that, what mechanism should we have in mind for how attending a single sex school instead of a co-ed school could affect a student's outcomes?

 

Kirabo [00:03:55] That is a good question. So my sense is people tend to prefer single sex schools for a variety of reasons. In many countries and Trinidad Tobago is one example, sometimes a single sex schools happen to be the ones that are more selective, so that's one component to it. It just happens to correspond to the schools that are more selective and may have an elite status to it, so that's one rationale for it. Another is that single sex schools in in many contexts happen to be ones that offer religious education, as I just mentioned. That's another sort of motivation.

 

Kirabo [00:04:27] Aside from those two, I think developmentalist parents also tend to value single sex schools because they believe that having environments where all the students are of the same sex is going to provide some specific benefits to their children. One of there are a lot of theories out there about why this may be beneficial and I think in in my research, I sort of highlight two. So one of them is sort of the direct effect of being around peers who are of the same sex. So some people will make the claim that, for example, that boys are generally more disruptive than girls. This has been relatively well documented. Some parents may prefer to send their kids to their daughters, to all girl schools to avoid being around those disruptive boys, that's one explanation or one rationale. Another rationale is often that in environments where everyone is sort of the same or sort of has the same from the same sex, it reduces stigma associated with sort of doing things that are not necessarily common among one sex. So, for example, it may girls may be unlikely or unwilling to raise your hand in classroom settings that have both boys and girls, but they may be willing to engage in these practices when they're only around girls. Boys, on the other hand, may actually have better behaviors when they're not girls around. So you might imagine a scenario where in mixed sex settings, boys have an incentive, particularly around the ages that I'll be studying sort of 10 to 16 boys may have an incentive to start acting out to get the attention of the girls and perhaps in those sort of mixed settings, the stakes are higher. So you get more acting out. But also the stakes associated with being sort of seen as being unpopular are also potentially higher. So some people argue that the existence of the opposite sex is just is just inherently distracting for a whole bunch of reasons.  I sort of consider that sort of more of a direct effect, sort of side effect of being around peers that are of different sex than oneself.

 

Kirabo [00:06:29] It's another mechanism which I think is is also important is kind of this idea that when you're in an environment where everyone is either or are the boys or girls, that it allows the schooling system or the schools to really tailor their instruction or tailor the educational environment to the specific needs of that group. You can imagine, especially when you're talking about adolescence, that it's entirely possible that when people are going through, if they're going through puberty between the ages of sort of 10 and 15, a lot of the things are going on and all the things that are going on are very specific to one's biology. So you can imagine that perhaps there are going to be specific things that are going on to boys such that if the school system is able to sort of be attuned to the specific needs of boys, they can better cater to their needs and the same thing for girls. So there's an argument to be made potentially that by separating boys and girls, you can tailor the schooling environment to the needs of each group such that both benefit. This could not only be sort of from a developmental standpoint, but also just educationally. People might also make the claim that if you're in a very gendered environment, you might use more gender answers. So when you're teaching math, maybe you teach math using baseball statistics, and that's going to really appeal to the boys. And if you're teaching math to the girls, you might use a different set of examples that might appeal to the girls. So that in sort of when you're when you sort of separate boys and girls, it allows for instruction to be sort of optimized to the specific needs of each group individually.

 

Jennifer [00:08:00] Your paper is titled "Can Introducing Single-Sex Education into Low-Performing Schools Improve Academics, Arrests, and Teen Motherhood?" Its forthcoming at the Journal of Human Resources. So set the stage for us. I know you've done a bunch of work in this area. So before this paper, what had we known about the effects of single sex education?

 

Kirabo [00:08:18] That's a good question. So before I wrote this paper, I mean, the main motivation for actually doing this paper was that I actually had some previous work looking at the effect of attending a single sex school. My paper is by no means the first in this literature is there. There are quite a few recent papers coming out in the past 10 years that have used either some sort of random assignment or quasi random assignment to schools where they can sort of basically compare the outcomes of those who attend single sex schools to the outcomes of those who attend coeducational schools that are hopefully of similar quality. It seemed to be kind of a recurring theme across these studies that students tend to have better outcomes, at these single sex schools, but it's sort of difficult to know whether the outcomes are improved because of single sex education per se, or if there's something else going on in these single sex schools that is leading outcomes to improve, that has nothing to do with them being single sex. So, for example, like I sort of already alluded to in Trinidad and Tobago, many of the most elite schools, the schools that are deemed to be the most prestigious schools are also single sex. So in some sense, when we're comparing students who attend to single sex schools, to those who are in coeducational schools, we might be confounding the single sex education component with an elite school component. One thing that happened in Trinidad Tobago, which is sort of exciting, is that the Ministry of Education looked at that relationship in the data and they said, well, it turns out that a lot of the schools where students are doing really well, particularly schools, where boys are doing really will happen to be single sex. So they said, well, let's actually run kind of an experiment. We're going to find a set of schools that are low performing and we're going to convert some of them to single sex. We want to see if that's going to improve outcomes. This basically provided a unique opportunity to answer a question that I'd had about the older literature, which is how much of the single sex schooling effect is due to them being single sex per se, as opposed to other things that might just be different about the schools by looking at the schools that converted from coed to single sex in Trinidad and Tobago and see what sort of happened to the students who attended those schools before versus after the conversion.

 

Jennifer [00:10:27] So you've already alluded to this a little bit, but what are the challenges in figuring out the effects of this type of schooling outcomes like academic achievement and criminal behavior? Is it. You mentioned the identification challenges. Are there data challenges as well?

 

Kirabo [00:10:40] For sure, I mean, in most in most settings, it's really hard just to have data linked across these settings. So in most settings, you know, educational data typically is held by ministries of education. If you can get those crime statistics are typically held by a different organization. Oftentimes and they're the two shall meet. So it was a really exciting opportunity in Trinidad and Tobago where fortunately I had access to the educational data already and I was able to negotiate through a few different agencies for them to link up basically the arrest records that were beinng identified so they can be linked at the individual level to individual students. Also, I could look at things like teen motherhood as well by looking at birth registry data. So it was an exciting opportunity to not just look at academic outcomes, which had been the focus of most of the studies on the topic before, and look at these nonacademic outcomes like arrests and teen motherhood, et cetera.

 

Kirabo [00:11:37]  I think the the ability to look at these nonacademic outcomes is really important also because some of the rationale for why parents choose single sex schools and schools has less to do with whether the schools are going to improve the academic outcomes for their children, but they actually think they're going to be a lot of behavioral benefits as well.

 

Jennifer [00:11:55] So, as you mentioned, the setting you consider is Trinidad and Tobago. So tell us about the context here. How does the educational system work in that country and in particular, how are students assigned to schools?

 

Kirabo [00:12:06] So the Trinidad and Tobago educational system is sort of derived from the British educational system that was there in the past. Basically, the end of primary school is at the end of fifth grade and students take an examination at the end of fifth grade called the secondary entrance examination. They are assigned based on their score on this examination to a secondary school. When they take the exam, every student lists a set of four schools that they would like to attend and they're listed in rank order. So one of the nice things about this feature is that it allows one to sort of potentially remove some of the selection problems that one would have when one just compares the students to attend one school versus those that go to the other. Specifically, what happens is that once all the students take the exam and they submit the set of choices that they would like to attend, this all goes to a centralized location and students are assigned to schools based on an algorithm, and the algorithm basically works as follows. The kid who has the highest test score is tentatively assigned to the school that they would like to attend first, if that school has a spot and then the next high scoring kid gets their top choice and the next highest scoring kid gets their top choice and they keep on going until some school is filled. Once that school is filled, they sort of take that school out of the mix. And anyone who has that school is a top choice is now removed and their second choice would then become their top choice. And they just keep on going down the lists of test scores until all schools are filled.

 

Kirabo [00:13:41] So this works for most of the schools in the country and all of the schools that I'd be looking in the study. The benefit of that is what this does is it generates a test score cutoff above which students who apply to that school get in and below which they do not. So this lends itself to what is commonly referred to as a regression discontinuity design, where you can have two students that say the test or out of is five hundred and the cutoff for a particular school is a hundred and five. The kid who scores a hundred and five gets into the school, but the kid who scores 104 who also wanted to go to school, does not get into that school. So given that the difference in test scores between 104 and 105 are very, very small, you can basically be pretty confident that those students are very similar. So you have two students who are very similar in what they look like before entering school, but one of them enters a school and one of them does not. It allows us to sort of decouple the effect of the school from the effect of selection that we typically see in observational studies.

 

Jennifer [00:14:39] It's a perfect natural experiment. So then on top of that, you also came across as need policy reform that you mentioned before, where the country converted a bunch of schools from co-ed to single sex to tell us more about that reform, how it came about and how it was actually implemented.

 

Kirabo [00:14:56] Right. So as I sort of mentioned, what happened was the Ministry of Education was really concerned in particular about the outcomes of boys. So it's something that we see in many, many countries. We see it in sort of lower income populations in the United States, but we also see it in many countries in the Caribbean and Latin America, where the the sort of boys are really underperforming.

 

Kirabo [00:15:19] If you look at sort of test scores, the test score gap is is quite large and girls do better than boys starting from a very young age. And you can sort of see it in in higher education, where much larger numbers or larger shares of women now are in college compared to men, which was not the case, say, 30 years ago.

 

Kirabo [00:15:38] So the Ministry of Education was really concerned about the under performance of boys and they sort of said what we want to do something to improve the outcomes for boys and that was really their their motivation. And they looked and noticed that the boys who are attending these elite single sex schools were doing very, very well. And they said, well, if they're if those schools are doing very well, perhaps it has to do with a single sex component. So what they did was they said we're going to basically try and introduce single sex instruction into some of these existing low performing schools to see if we can improve the outcomes for the boys and the girls who attend these schools. So to do that, they basically identified a set of schools that were very similar and closely and close to each other because, well, they ended up having to do was they didn't want to disrupt students who already were in the schools. So if you were already in the school under the co-ed regime, you stayed there under the co-ed regime. But they were going to make the incoming cohorts for those schools single sex. So in order to do that, they had to make sure that if they're going to do that, students who would have attended school A both boys and girls may have attended school A. Now, if only boys are going to get in that school, they need to make sure there was a school that was nearby that was relatively similar that they could send the girls to and vice versa. So they basically went through and they identified pairs of schools and within each pair one school was going to become all boys and one school was going to become all girls, and by choosing schools that were very similar and selectivity and also very similar in terms of the location very close to each other, it was relatively easy to sort of reshuffle the incoming cohort such that the boys who would have gone to this school all went to the boys school and the girls all went to the all girls.

 

Kirabo [00:17:15] So that's sort of how they set this thing up to facilitate this this transition.

 

Jennifer [00:17:21] And I think you said in the paper the schools had no say in this, is that right?

 

Kirabo [00:17:25] That is correct. So it was a it was it was a decision that was not seen as a very positive development by the schools and also by the teachers.

 

Kirabo [00:17:36] So it's one of those things where if you look at the data, so one of the things that we did was I also collected some data on students and also a little bit on teachers. And one thing that it was pretty clear was that many of the principals at the schools that became all boys complained a lot. They basically complained that, you know, how how are you going to do this again? They basically had it in their minds that boys are really disruptive and they don't want to have a school that's full of all boys and not have the girls. So there was a lot of resistance to this. So it was pretty bad politically. But from a from an experimental standpoint, it was actually pretty ideal because it allowed us to see what would happen if you basically force a school that really didn't want to become single sex to become single sex from coed.

 

Jennifer [00:18:22] Right. Because what you'd worry about if if these schools all opted in is that somehow you get the schools that, you know, if they really wanted to be single sex, then maybe they're different in some way annd the teachers are all like, it doesn't give you a representative sense of what would happen if we changed other schools to being single sex, right?

 

Kirabo [00:18:37] That is exactly right.

 

Jennifer [00:18:39] Okay, so you use this reform as natural experiment with an empirical strategy called a difference in regression discontinuity or a diff-in-RD for short. So listeners may have heard of a difference in differences strategy. We've talked about those before. Some will also have heard of regression discontinuities, which you just mentioned. The diff-in-RD basically combines these two strategies together, but it isn't used as often. So talk us through the intuition for the strategy and how you're using it in this case to measure the causal effect of attending a single sex school on student outcomes.

 

Kirabo [00:19:13] So the first I'll talk about using the regression discontinuity and then I'll talk about using the difference in regression discontinuity. So the regression discontinuity design, which, you know, people are familiar with, basically would say we're going to use the test scores of students who are applying to a particular school.

 

Kirabo [00:19:32] Let's take one school, for example, that transition from coed to single sex and what we could do is you can say, look at all the kids will apply to that school and everyone who scored below the cutoff for that school did not get in and everyone who scored above the cutoff for that school did get in. So you can use the regression discontinuity right through the cutoff to identify the effect of attending that particular school. Now, we can basically do that every single year so we can do this in 2009 before the reform was implemented and then we can do it in 2010 after the reform was implemented. So the idea is that the regression discontinuity design allows us to identify the effects of attending a particular experimental school before the transition in 2009. Then the regression discontinuity design was applied to the data in 2010 allows us to identify the causal effect of attending that same experimental school after it transition to single sex.

 

Kirabo [00:20:32] So by comparing the RD estimate before the reform to the RD effect after the reform, hence the difference in RD I hope to sort of identify or isolate the effect of the transition.

 

Jennifer [00:20:47] Perfect, and then to do all that, you need data. So what data do you have to dig into these questions? And related to that, what outcomes are you most interested in?

 

Kirabo [00:20:57] So the data that that I used, basically, I had information on all the students who were applying to secondary schools in Trinidad and Tobago for several years, so essentially I used the data which had the test scores of students at the end of primary school.

 

Kirabo [00:21:16] I had all the schools that they wanted to to attend and that's the one that's what I sort of used to set up the regression discontinuity design. And those data were then linked to subsequent outcomes for students three years after. So three years after entering secondary school, they took an examination. They took another high stakes examination five years after entering secondary school. Then I can look at whether they completed secondary school at all. Other interesting sort of nonacademic outcomes they looked at were whether students, whether the girls had a teen pregnancy, something that we think might be affected by being around members of the opposite sex. Also for boys, I looked at whether they were ever arrested as a teenager by linking them to essentially arrest records covering that same time period and linking individuals by sort of name and dates of birth, et cetera.

 

Jennifer [00:22:05] How complicated was it to get all of these administrative data was a setting that made it pretty easy for you, or was this a big process?

 

Kirabo [00:22:13] Well, it's one of those things where I've been working with data entry on Tobago for for several years, so I fortunately already knew many of the factors, at least in the in the Ministry of Education, in terms of how to get access to data. Given that I've been working with the minister of education for several years, some of the people who were in the Ministry of Education sort of moved up in government to even more powerful positions. So by sort of having those contacts, I was able to talk to a few people and they were able to sort of basically vouch for the fact that I was legit researcher and they were able to send data to the Ministry of Education where things could be linked and things would be done.

 

Kirabo [00:22:55] So, I mean, one of the benefits of working with these data, I've also been doing some works with Diether Beuermann who's a coauthor of mine on some other work at the Inter-American Development Bank. And one of the things that we'd also work with them to do is harmonize some of their data systems. So one of the sort of spillover effects of my working with these data is they now have more formal systems in place that allows other researchers to access some of these data and link them as well. So I was probably one of the early movers on this, but I think it's going to be easier going forward and it did take a lot of time and, you know, personal connections to get access to everything and make sure everything held together.

 

Jennifer [00:23:33] Yeah, that's fantastic. Well, thank you for that public service. Sounds like it'll be quite valuable. Okay, so let's talk about the results. What do you find are the effects of attending a single sex school on academic achievement in both the short and long run?

 

Kirabo [00:23:48] So if you look at it in the short run, it looks as though essentially the test scores of both boys and girls are go up from attending these schools.

 

Kirabo [00:23:58] If you compare the outcomes of boys who attended these experimental schools before the reform versus after the reform, the boys test scores go up pretty substantially by about a fifth of a standard deviation, which is a pretty big effect. We see for girls, the effects are much more muted and they're less sort of robust. So I would say that there's some suggestive evidence that the girls test school performance improved in the short run, but not markedly so. To put the sort of the boys effect of a quarter of a standard deviation in perspective, you know, if you look at, say, U.S. studies, the finding there is that going from, say, a teacher at the median of the distribution to one of the fifth percentile of the distribution probably means test scores by about a tenth of a standard deviation.

 

Kirabo [00:24:44] So we're seeing something about twice the size of that that would be sort of equivalent to going from a teacher at the fifteenth percentile to one of the eighty five percent. It's a pretty large effect in terms of the kinds of things we see based on other interventions that probably cost a lot more money.

 

Kirabo [00:24:58] One of the things I sort of didn't mention before, which sort of makes sense, is, you know, going from having a reform where we basically reassign students across schools is not entirely costless, but it doesn't really take much money. You don't have to pay for additional teachers necessarily. You don't have to build any more schools. It's a relatively cost, relatively, I would say cost less. but it's a relatively inexpensive way to improve outcomes by a pretty sizable amount.

 

Kirabo [00:25:23] If you look into the long run, we do see some evidence of improved outcomes as well. So for boys, the effects are pretty big on whether they actually end up completing secondary school. So boys are more likely to take advanced courses, they're also more likely to basically pass the advanced courses. I think this is about five years after secondary school entry and they're more likely to earn a certificate. For girls, even though we didn't see very big effects on test performance in the short run, they're actually pretty strong, robust effects on the overall likelihood that these that girls will graduate high school five years after secondary school entry.

 

Kirabo [00:26:05] So for boys, we see larger effects on test scores in the short run and for girls, we see larger effects on their school completion in the long run. There's potential explanations for that, which is probably going to be your next question.

 

Jennifer [00:26:20] Well, I was going to ask about the distribution of the effects that did so. To what extent this is coming from moving, say, people at the bottom up or moving people at the top to be even higher achievers? I know you do a bunch of that in the paper to you and talk about that.

 

Kirabo [00:26:34] Yeah. So it seems if you if you look at the boys, it seems that a lot of the action is happening at the bottom end of the distribution. But there's also some evidence that things are happening at the top. So it's it's probably closer to see just like a uniform shift, but there's definitely improvements at the bottom. There are large improvements at the bottom, and they're also pretty sizable improvements at the top. The middle of the distribution seems to be kind of relatively unaffected.

 

Jennifer [00:27:02] And for the girls, was it the same thing?

 

Kirabo [00:27:05] I think that's that's right for the girls. There's some evidence of of benefits at the top, but there's again, because the test score impacts are muted for girls, we don't really see much traction there.

 

Jennifer [00:27:14] Got it. Okay, and then you also consider nonacademic outcomes. So first you look at effects on arrests. So what do you find there?

 

Kirabo [00:27:22] So if you look at the boys, who attended these schools before compared to after the arrest rates went down, and these are arrests by the age of 18, so these are sort of teen arrests. Those went down for boys by about six percentage points and so this is a pretty big effect. You know, the average rate in the population is probably about four or five percent. So this is a pretty big effect. I should say, however, that because the schools that this intervention was implemented in are low performing schools, the baseline of risk rate is higher. So we shouldn't think of this as being something that we would necessarily see in all schools, but this is something that we could see in schools that are enrolling high shares of low income boys who are relatively underperforming. Then for girls, there was not really much, which is sort of not surprising. This is something we see in many educational interventions that for girls, we don't tend to see much in terms of risk because girls generally don't comprise a large share of the arrest that occur in the data.

 

Jennifer [00:28:21] Right. Boys are the ones committing most of the crime, but you do consider another way in which girls might get off track as you measure the effects on teen pregnancy for girls that attend the all girls schools. So what do you find there?

 

Kirabo [00:28:36] That's right. Much like we do for the boys, we compare the outcomes of girls who attended these schools before versus after the change. The girls in the single sex cohorts were about four, I'd say somewhere between two and four percentage points less likely to have a child as a teenager by the age of 17 is about two percentage points and by the age of 18, it's about four percentage points, suggesting that we're definitely getting reduced teen births concentrated largely around the age of 18.

 

Jennifer [00:29:09] And those are also pretty big effects, right?

 

Kirabo [00:29:12] That is correct. So, you know, relative to the baseline, these are probably effects of about maybe 50 percent or something like that.

 

Jennifer [00:29:18] So you run a bunch of robustness checks to support the assumptions of the diff-in-RD design, but I want to talk about one in particular. So listeners who've heard us talk about other RD papers on this show will remember discussions about who the compliers are. So an RD will allow you to measure the effect of a policy on the types of people who are actually affected by that policy. And those are the compliers. So in this case, the compliers are the kids whose choice about whether to extend a single sex school depends on their testing just over the test score cutoff. So in the diff-in-RD case you're comparing RDs over time. So we might worry that any difference between the RDs before and after the policy change is due not to a change in the effectiveness of the school, but to a change in who those compliers are in this case, who ranks the school as being a preferred choice. So you explore this possibility a bit. Tell us what you do in the paper and how you're able to convince yourself that this isn't what's driving the effects in this case.

 

Kirabo [00:30:15] That's right, so this is a big concern of mine, there are two scenarios in which the diff-in-RD  will falter in this case. So, there are two possibilities. So one possibility is that the treatment effect is larger for a particular kind of student, let's say. So the effect of going to an experimental school, something about the experimental schools, the students who were responsive to the RD before the reform versus those thereafter were different. So the ones who who basically were induced to attend these schools after may have just benefited more from attending that kind of school in general, in respective of whether it was before or after the transition. And what we're basically comparing are two different populations who differentially benefit from attending the schools that converted to single sex. That, in essence, would be be the problem. So there are two scenarios in which one could try and rule this out and you can't rule it out entirely. But the first is just the question of whether the treatment effect tends to vary at all. So even if it were true that the complier population changed over time, which in some sense has to be the case. Right. So before the intervention, we had co-ed individuals applying to the schools. After the change, all the compliers in some schools were boys or all the compliers in the schools were girls. So of course, there's going to be some difference there. But, you know, the issue here is that if if the treatment effect is the same for everyone, then even if the complier population changes, it doesn't matter. So that's the first thing I sort of examine by just saying if I take this treatment effect and I interacted with a whole bunch of different characteristics about the student.

 

Kirabo [00:32:02] So I have information on the kinds of schools they want to attend. So in most data sets, we don't really know the kinds of schools students want to attend we just know where they actually attended so we can look at that. I can basically see it as does the effect of attending one of these experimental schools vary, depending on whether someone tends to put selective schools first or single sex schools first, or schools that have high shares of boys first. And I don't really see anything there. I look to see where there varies by the just the incoming test scores. So do we see that high achieving kids benefit more from these kinds of schools and others and across a whole set of observe characteristics I don't really find much, much evidence that the treatment effect varies a lot. So that's the first piece to sort of assuage concerns that even if the compliant population changed, it probably wouldn't bias things very much. But ideally, you want to show that the compliant population didn't change very much either. So that's the second part of the test. So what I do is I adopt a test that was implemented, I believe, by Imbens and co-authors and the basic idea is that we can sort of get a sense of what the complier population looks like or how the compliers compared to the average person in the population by essentially running the regression discontinuity for the full population and maybe scoring above the threshold increases the likelihood that one attends a school by 50 percent. Let's say that's the that's what we see in the in the overall population. We can run the same analysis only for, say, those who are low achieving and if for the low achievers, the jump in the likelihood of attending the school is only 10 percent, we get a sense that a lower achievers are less likely to comply than the average person.

 

Kirabo [00:33:47] So by comparing sort of these sort of first stage across a whole bunch of characteristics, we can sort of back out, get a sense of which populations are more likely to be compliant versus not. And I compare that before versus after the change. What I basically find is that they're actually relatively little changes in the kind of person who complies with the treatment before versus after, which is to say the kind of person who attends a school that they score both threshold for was the same before versus after.

 

Jennifer [00:34:16] Great. Okay, so let's go back to the potential mechanisms. So you actually conducted a survey to try to understand what's driving the big beneficial effects that single sex schools are having to tell us about the survey you conducted and what it tells you about the mechanisms at work.

 

Kirabo [00:34:35] One of the nice things about this particular intervention was that it was started in 2010 and I became aware of it soon after it was rolled out, it was announced. So I probably found out about it at the same time that the schools did and but had already been rolled out. So one thing I was able to do was I was able to sort of administer a survey real time to some of the students, to these schools to get a sense of what outcomes look like for those who would have been treated versus not. In this context, because the rollout was not it didn't change the school, but they just changed the incoming cohorts, even though even in 2011 or 2012, when I was able to administer the survey, there were some students who were admitted to the school under the co-ed regime and the single sex regime at the same point in time.

 

Kirabo [00:35:22] I can basically compare their response to a whole bunch of questions to see whether anything changed before versus after and how they feel about school. So I designed the survey to try and get at some of the mechanisms that were sort of proposed in the existing literature to generate single sex schooling effects. So questions about whether the schooling environment is disruptive, sort of getting at the idea that boys are more disruptive than we should see that boys are going to schools that became all boys are going to be more disruptive in principle. I also ask questions about the extent to which teachers pay attention to students. The idea being here that under the sort of the idea that if you separate boys and girls, teachers can sort of tailor the instruction and focus a little bit more on the specific needs of the kids. You might see that teachers change their teaching practices in a way in response to the changing student demographics, in response to the increased sort of homogeneity of the student population. We might see that in the kinds of things that they do. So I asked questions about whether teachers use more use of, say, classroom participation, asking questions. I ask questions about whether girls were more likely to raise their hand in class, or I should say I asked everyone if they were likely to raise their hand in class to the boys and girls to test the extent to which girls are more likely to raise their hand in single sex environments and court environments to sort of test this hypothesis that was sort of put out there.

 

Jennifer [00:36:52] And what do you find?

 

Kirabo [00:36:54] So the findings I would say they were highly suggestive. The first thing that sort of jumped out was that it is certainly true that it appears that the girls did report lower levels of peers being disruptive in the all girls context in than the coed contexts.

 

Kirabo [00:37:17] They also reported being less distracted by their peers. So this is pretty much in line with the existing literature on gender perfect showing that like having higher shares of boys may be deleterious to the classroom dynamics in ways because they're destructive and disruptive.

 

Kirabo [00:37:33] I mean, I also found I also found some evidence that girls report being able to learn more from their peers in that context, which is again, consistent with the sort of direct peer interaction effects. The other thing that I did find, which is sort of more suggestive, is that the girls reported that teachers gave them more individual attention and a little bit of evidence that maybe teachers involved students more in classroom activities in those contexts. When we looked at the boys, some of the things were similar and some of them were different. So for the boys, there were a sort of positive effects on peers being disruptive and distracting, again, broadly consistent with the idea that boys are disruptive, but those effects were actually relatively small. It seemed as though if you just think of the sort of disruptive peer mechanism, girls were much more responsive on that margin to the change in the gender composition than the boys were. Then the other thing that was sort of interesting was that boys also reported higher levels of teacher attention and higher levels of teacher involvement. It appears and also I should also mention both boys and girls reported higher levels of teacher work towards students. So it appears that in terms of the teacher questions, both boys and girls reported in the same direction, suggesting that there's something about having boys and girls separately that allow teachers to teach in a way such that they were able to give all students more individual instruction. They were able to, I guess, behave more warmly towards students and involve them more in conversations. In terms of the questions that talked about direct interaction with their classmates, there are things moved exactly as you'd expect based on the literature on gender peer effects, which is specifically that more boys leads to a less productive classroom environment. So my take on my interpretation of this is that, you know, when you look at the overall effects of boys benefited quite a bit as certainly in terms of their short run test scores. So one to one potential sort of interpretation of that is even though the boys were in environments when where their classmates were more disruptive, they benefited a lot from the changes in teacher practice that may have happened, maybe the tailoring of the schooling environment to the specific needs of boys such that on that they actually did better.

 

Kirabo [00:39:53] For girls, they benefited both from having a less disruptive environment and also potentially from having the teachers tailor instruction towards their specific needs as well.

 

Jennifer [00:40:03] That's really interesting. I mean, just to think about the big effects that you're seeing on the likelihood that those boys are getting arrested, all coming from just basically the teachers being able to tailor their instruction in the way they interact with the students to this different environment really highlights how powerful that potential mechanism is.

 

Kirabo [00:40:26] I think that's right. You know, this is now I'm going to be I'm going to speculate beyond the findings here, but, if you know, if you talk to people who are sort of advocates of all boys schools, they sort of make the claim that, you know, when in all boy settings, the the discipline style is actually different. So one thing is they you know, they're probably more likely to be lenient about certain things because boys, at least stereotypically, are less able to sort of sit still. The kinds of expectations of behavior are different for boys in an all boys setting versus an all girls setting. One claim is that or one proposed thing that sort of a benefit for the boys is that the boys are they can sort of be themselves in a way, without necessarily getting formal sanction from the school in terms of behaving badly. Also the ways in which the schools deal with the boys in general when they when they act out is going to be different because they sort of know how boys react to these things, so that's one sort of a potential mechanism, I think, that could explain this kind of thing.

 

Kirabo [00:41:34] But again, this is not something I was able to test with my data, but it's sort of consistent with a pattern that I document.

 

Jennifer [00:41:39] Yeah. Is there any other work relevant to this topic that has come out since you first started this project that helps shed light on either the costs or benefits of single sex education?

 

Kirabo [00:41:49] Yes, so I mean, actually, at the time of writing this, there are a couple other papers that were asking similar questions. One is by Lee and Turner that just came. I'm not sure if it's out yet, but it's still a working paper. They basically compare outcomes in single sex classrooms versus single sex school and they sort of document that the the effect of a single sex school is different from that of a single sex classroom. Basically, boys benefit from single sex schools, but they don't necessarily benefit from single sex classrooms. The idea there, I think, is very consistent with the results I'm showing you here, which is that when you get in an environment where everyone is all boys were all girls in the whole school, it might lead to changes in practices that improve outcomes for the boys and improve outcomes for the girls that you wouldn't necessarily see just from small changes in gender composition or from having one single sex classroom. So that's one thing that came out that I think is speaks directly to the kind of mechanisms here is very consistent with this interpretation. Also, there's another working paper by a graduate student of mine, which is still very much a work in progress, looking at bullying, which also has some patterns that are pretty consistent with with the with the patterns here specifically, that all boys environments tend to have really good outcomes, even though they have much higher levels of reported bullying. So suggesting there's something about these all boys contexts that maybe bad in terms of boys behavior or at least in terms of how it's being reported, but leads to better outcomes overall.

 

Jennifer [00:43:23] That's really interesting. So what are the policy implications of this paper and the other work that you've talked about, should all schools be single sex?

 

Kirabo [00:43:32] That is a good question. So, you know, I think there are objections to single sex schooling that are not based on whether they are beneficial or not. They're the sort of more based on sort of the idea that separating boys from girls is just an inherently bad thing to do.

 

Kirabo [00:43:50] You know, I'm I'm sympathetic to to that argument and the argument would be that, you know, when you separate boys and girls, first of all, you're one is facilitating a sort of separate but equal scenario where maybe the boys schools will get more resources than the girls, which is the entire rationale we had with the "Title Nine" restrictions to begin with.

 

Kirabo [00:44:11] There's also the concern that, you know, the socialization could be different. So it's entirely possible that having boys in an environment where they're only exposed to other boys or girls in an environment where they're only exposed to exposed to girls doesn't basically allow them to interact with peers of the opposite sex such that they don't learn how to be, basically they have may have problems in the marriage market. They may have problems when they go into labor market, when they have to actually interact with both men and women, so there's some argument to be made about socialization that some people worry that, you know, gender norms could be reinforced in these environments. Actually, some of the some of the research that I've seen suggests that this is actually the opposite is going to be true, but there's a concern potentially that when you separate boys from girls, you sort of exacerbate or reinforce negative gender stereotypes about other groups. I think those are real concerns, you know, in terms of, I tend to advocate looking at the facts, looking at the data and seeing what the data says, so I'm happy to explore this, but I think it certainly makes sense to me at least, that we should have more single sex schools in certain environments for people who want to choose to do them. I don't necessarily advocate making everyone do it, but it does make sense to have it as an option. Related to that, in my previous paper on single sex schools, one of the key results I found there was that in that setting when I was really just comparing those to went to single sex schools, to those who went to coeducational elite schools. So we're looking at elite schools in that context and in that context I found was that those individuals who reported really, really, really wanting to go to a single sex school, they benefited a lot from attending those single sex schools and those who didn't really report having any strong desire to go to single sex school the effects were pretty small. So that tells me that potentially having it as an option is a good thing, but forcing everyone to do it perhaps may not be the best thing.

 

Jennifer [00:46:14] Hmm. Yeah. Especially given what you said before about this being a pretty low cost intervention relative to other other options that we usually talk about.

 

Kirabo [00:46:24] That's exactly right. I mean, the fact that we can improve outcomes to this extent with relatively small financial outlay, I think means that it's something that we should pay attention to and it's something we should be mindful of. I think more broadly, you know, economists and social scientists would be really interested in peer effects and thinking about how can we leverage the power of peer effects to basically squeeze a little bit more performance out of every sort of tax dollar that we spend in education. I think this is this is a sort of an example of essentially leveraging a peer effect to improve outcomes for kids. That is going to be relatively low cost and I think it's actually really, really potentially powerful.

 

Kirabo [00:47:04] One thing I would say is, you know, in the United States, there has been a growth of single sex schools in low performing areas like in Chicago. There are single sex schools that basically are focused on like black boys, for example, or black girls and those schools, if you just look at their data, appear to be improving outcomes quite, quite a lot. So they are targeted to populations that are sort of at risk of not really engaging with the academic world. They are risk of high levels of dropout and perhaps having schools that are really tailored towards a specific demographic and gender or sex could be a component of that may actually have some real benefits and I could see some value to that.

 

Jennifer [00:47:44] Hmm. So what's the research frontier here? What are the next big questions that you and others will be thinking about in the years ahead?

 

Kirabo [00:47:52] Well, I think one of the one of the things that I was able to document here was that, you know,  there is something about these schools that are changing these outcomes. The evidence I was able to to sort of uncover on why is suggestive, but I think we have a lot more work to do to sort of figure out why we're getting these positive single sex effects and is it possible we can sort of improve on these things? Is it going to be true in all contexts? So getting a sense, you know, in terms of moving the needle, in terms of informing policy, we want to know where we're going to get bigger effects, what are the why these effects emerge? I think all that is very much on the table at this point. I think when you figure that out, we can really get a sense of how much we can improve outcomes with relatively small costs.

 

Jennifer [00:48:38] Also seems like I mean, some of the policy questions you mentioned before about why people are concerned about these types of schools, things like marriage market outcomes and really long run effects, feels like that also we don't have any evidence on that yet. Is that right?

 

Kirabo [00:48:53] That is correct. I should I should say that there is a long history of of papers that were published, you know, 30 years ago or 20 years ago that use methods that are largely observational or this larger descriptive, so there is some work out there to sort of documenting differences across groups, but they are not using evidence that would be probably causal to use your lingo. So I think I there's a lot more research to be done in terms of that. That's right.

 

Jennifer [00:49:27] My guest today has been Bo Jackson from Northwestern University. Bo, thanks so much for doing this.

 

Kirabo [00:49:32] Thank you so much, Jen. It's been a pleasure.

 

Jennifer [00:49:39] You can find links to all the research we discussed today on our website, probablecausation.com. You can also subscribe to the show there or wherever you get your podcasts to make sure you don't miss a single episode. Big thanks to Emergent Ventures for supporting the show and thanks also to our Patreon subscribers. This show is listener supported. So if you enjoy the podcast, then please consider contributing via Patreon. You can find a link on our website. Our sound engineer is Caroline Hockenbury with production assistance from Elizabeth Pancotti. Our music is by Werner and our logo is designed by Carrie Throckmorton. Thanks for listening and I'll talk to you in two weeks.