Probable Causation

View Original

Episode 12: Michael Lovenheim

Jennifer Doleac

Michael Lovenheim

Michael Lovenheim is Professor of Economics, Policy Analysis and Management, and Industrial and Labor Relations at Cornell University.

Date: September 17, 2019

A transcript of this episode is available here.

See this content in the original post

Episode Details:

In this episode, we discuss Professor Lovenheim's work on the effects of grade retention on adult criminal behavior:

"The Effect of Grade Retention on Adult Crime: Evidence From a Test-Based Promotion Policy" by Ozkan Eren, Michael F. Lovenheim, and Naci H. Mocan.

OTHER RESEARCH WE DISCUSS IN THIS EPISODE:

Transcript of this episode:

 

Jennifer [00:00:07] Hello and welcome to Probable Causation, a show about law, economics and crime. I'm your host, Jennifer Doleac of Texas A&M University, where I'm an Economics Professor and the Director of the Justice Tech Lab.

 

Jennifer [00:00:18] My guest this week is Michael Lovenheim. Mike is Professor of Economics, and Policy Analysis and Management and Chair of the Economics Department at Cornell University. Mike, welcome to the show.

 

Michael [00:00:30] Great. Thanks for having me. It's a pleasure to be here. Excited to talk to you.

 

Jennifer [00:00:33] So we're going to talk today about a working paper you recently released about the effect of grade retention for students on their criminal activity as adults. But to start out, could you tell us about your research expertise and how you became interested in this topic?

 

Michael [00:00:48] Sure, so I'm a labor economist, public finance economist, and a lot of my research focuses on the economics of education — that's kind of my main research area. And increasingly, I have become interested in how the education system is affected by and affects other policy areas. So I've done some work looking at the interplay between the health insurance system and particularly special education students and how providing health care services through schools affects student outcomes. I've also done some research looking at how excluding people with drug convictions from the federal financial aid system affects their educational attainment investments in college. And so I'm very interested in how policy decisions in one area spill over to other areas. And education's kind of an interesting place to look here because educational decisions are so ingrained in how one's life progresses. Right. That decisions one makes as a youth in terms of their educational investments can affect long run life outcomes. It also can be the case that other policy dimensions can influence how students get through the education system in ways that are completely unintended by policymakers.

 

Michael [00:02:14] I'm also really interested in estimating the returns to education. So how much do you get out of a given educational investment? Most of those papers, that look at the returns to education, focus on labor market outcomes. Right. So economists in general, and in particular labor economists, believe that earnings are life's report card. So, you know, you can just look at earnings and that tells you everything you want to know. But there are a lot of other types of outcomes we care a lot about. And I think criminal justice and interaction with the criminal justice system is one of those areas where we really want to understand how education policies and education decisions students make when they're younger relate to the likelihood that they are involved in the criminal justice system in a in a negative way when they're older. Because that has important ramifications for their life outcomes, you know, throughout their entire lives.

 

Michael [00:03:04] And so looking at the effects of grade retention on crime, this is like a really interesting area to look at because grade retention is something that affects a lot of students and it is- disproportionately affects students from lower income backgrounds, minority students who also are more likely to be engaged in the criminal justice system. And so understanding that linkage, I think, is particularly important, given the prominence of these kinds of policies we're studying, and grade retention more generally, as well as core concerns we face in the society about understanding what are the pathways that lead students to be involved in crime later in life.

 

Jennifer [00:03:46] OK, so let's talk more about grade retention. What is it and how how are retention decisions typically made, I guess mostly in this country? And to what extent is this controversial?

 

Michael [00:03:58] So grade retention is basically just holding students back from progressing to the next grade. It can happen at any grade. The most common example of grade retention actually happens when students are very young, something called "academic redshirting," which is when families hold back students from starting kindergarten or first grade an extra year. Often this is done to help students mature more so that they have a higher level of maturity when they start school. So it's done predominantly for social emotional concerns.

 

Michael [00:04:33] We are actually focused on a different part of the education distribution. We're looking at somewhat older students who are about to go into high school, so they're eighth grade students. And those decisions about whether to promote students are done by some combination of school administrators, teachers, and parents. And it really varies an enormous amount across states, across school districts. There aren't, for the most part, set policies that determine whether students are allowed to move on. And many school districts have policies either written down or functionally that allow for what we call "social promotion." So there are no academic standards that they hold students to. Students are allowed to move on kind of regardless of how they do in the prior grade.

 

Michael [00:05:25] The policy we study is a test-based promotion policy that the state of Louisiana implemented in 1998. And many states in large school districts have these policies. And what they do is create clear academic standards that students have to meet in order to move on to the next grade. And these have come out of generally a set of policies in the U.S. That have gained a lot of traction over the past couple of decades, focused on accountability. And a lot of those accountability policies are aimed at teachers or schools saying you have to meet certain benchmarks to get funding or we're going to punish you in certain ways or we're going to fire you if you don't meet certain benchmarks. So big parts of, for example, No Child Left Behind, which was President George W. Bush's main education policy initiative were really focused on those dimensions of accountability. These test-based promotion policies are similar, but focused on students. So they're student-based accountability and saying students have to have responsibility for their own academic outcomes and showing the state or the district that they are academically capable of moving on to the next grade. And really they're focused on ending social promotion.

 

Michael [00:06:48] And that's really the the source of the controversy. And this is a very controversial issue in the education policy community. People who are in favor of social promotion argue that it is stunting for students to move- to to be held back, particularly when they're older. When students are younger, it's not clear that anyone notices enough. Right. They're- you know, it's not as salient to the students because they're younger and they forget what grade they're kind of, quote unquote, supposed to be in. But when students are older — so the students we're looking at are teenagers, they're in eighth grade — it can be stunting to them. And that is the worry, that it's going to alter how they think about their own place within the education community and how they're treated by their peers and the education system.

 

Michael [00:07:31] You know, and on the flip side, people argue, sure, it could be stunting, but there's not- there's not a lot of evidence of that. And it's the case that we are allowing students to move on to higher grades without knowing whether they can actually be academically successful in those grades. And that actually can generate negative spillovers to other students, because if teachers are spending time on students who don't yet know the material they're supposed to know upon entering ninth grade, that's less time they can spend on other students who are academically qualified to be in that grade.

 

Jennifer [00:08:06] OK, so your new paper is titled "The Effect of Grade Retention on Adult Crime: Evidence from a Test-Based Promotion Policy." It's coauthored with Ozkan Eren and Naci Mocan. So set the stage for us. Before this paper, what had we known about the effects of grade retention on various outcomes and the effects of education on crime more broadly?

 

Michael [00:08:29] So let me answer in reverse. So let me talk about education on crime. First, so there are several papers that have looked at this from a couple different dimensions. So the- most of the papers that have examined the effect of education on crime look at the effects of high school attainment — so the number of years, predominately of high school, that students attain — on later in life criminal justice involvement. And the way they mostly do this is using compulsory schooling laws. So, compulsory schooling laws state that students have to be enrolled in school up to a certain age. And there is a good amount of evidence that indicates that these laws induce students who otherwise would have dropped out at younger ages to obtain more years of high school. And so what these papers do is use changes in these laws over time across states or within a given country to- and link those changes that generate differences across students and how much high school they get and look at criminal activity later in life. In general, those papers find that students who were induced to attain more high school, are less likely to be arrested or convicted of a crime later in life.

 

Michael [00:09:47] However, there are kind of core issues with those papers that relate to why states or countries change these laws. And there's a lot of worry about what we in economics call the endogeneity of policy changes, right. That these policy changes are related to underlying trends in these different areas that are independently moving these outcomes. And so you find a relationship, but it's kind of spurious. Right. And there's some evidence that that's come up that that in particular, these kind of law changes are indeed related to underlying trends in ways that should make us worried.

 

Michael [00:10:28] The other set of papers that have come out on this, look at the effect of school quality on criminal outcomes. In particular, there's a paper by Dave Denning in the Quarterly Journal of Economics that's really nice. And a paper by Cullen, Jacob, and Levitt in Econometrica that's that's also very nice, using school choice lotteries. And so what they do is look at students who win these lotteries that allow them to go to a charter school or another public school in the local area, which increases their school quality. And they look at how winning this lottery then relates to later in life criminal outcomes. And they find that it- winning these lotteries makes it less likely that students will be arrested either as a juvenile or later in life.

 

Michael [00:11:16] So that's kind of the state of knowledge of how education affects crime. And our paper contributes to this by looking at kind of a novel policy here, which is grade retention. And we kind of independently care about that kind of policy question. But it also looks at a broader, I think, and interesting set of students than other people have looked at, and in particular, it's allowing us to focus in on very low performing students who for other reasons, are much more likely to be engaged in the criminal justice system later in life. These very low performing students are the ones who are at risk of being retained in eighth grade and I think are exactly the kind of sample you want to be studying in this context.

 

Michael [00:12:05] So there also is a literature on how grade retention affects student outcomes. A lot of it uses similar types of empirical approaches that we use — a regression discontinuity approach — that we'll probably go into in a little bit. And the findings can- are mostly aligned with retention being harmful for older students and either non-harmful or maybe beneficial for younger students. And I think that aligns with some of the things I said earlier about how students may be perceived by their peers, by the system if they're held back at younger ages versus older ages. But I think it's also important to highlight that no one before our paper has examined the effects on adult criminal convictions. My coauthor, Ozkan Eren, has a paper looking at the effects on juvenile crime, and he — and we replicate this in our paper as well — finds no effect on juvenile crime. So it's really happening in the adult criminal justice system, not in the juvenile criminal justice system.

 

Jennifer [00:13:15] And so you've alluded to this a little bit, but talk a little bit about what the challenges are to figuring out the effects of things like education enrollment and academic achievement on outcomes like criminal behavior.

 

Michael [00:13:28] It's very hard. Which- it may be one piece of evidence that it's hard is that there's very little evidence on it, even though it's an important question. And there's really two sources of difficulty. The first is data. It's really hard to find datasets that give you detailed information about a student's academic background and achievement, as well as criminal justice outcomes. Right. And so that's kind of problem one is it's hard to actually even assemble the right set of information that will accurately allow you to track students through their schooling experiences and then into their adult life and see if they are involved in crime in some way.

 

Michael [00:14:15] The second problem relates to the difficulties of causal analysis. Students are not randomly held back. Right. They're held back for reasons that have to do with socio-emotional adjustments to their schooling environment that have to do with their academic achievement. And they are not held back in a vacuum. Right. So typically the decision to hold back a student is a decision that is made by some combination of school administrators, by teachers, by parents. And so any different combinations of those actors in this decision process can alter these outcomes. Right. And so what happens is that the students who are retained tend to be different on unobserved dimensions that relate to the like- that also correlate with the likelihood that they will commit a crime later in life. So, for example, students who are retained probably exhibit less motivation in terms of their academic investments. They may be from families where their parents are less likely to go in and advocate for them if the school is thinking about holding them back. Right. So that's just two examples where you can see that those characteristics would also probably be correlated with their likelihood of being involved in a criminal activity or being convicted of a crime later in life. And so even with extremely rich data that give you a lot of information about a student's background, you just can't possibly hope to control for all the factors that are correlated with the idiosyncratic decisions of of schools to hold back a student, as well as the underlying risk of being convicted of a crime later in life.

 

Jennifer [00:16:01] Right, so absent some sort of real or natural experiment, you're always worried that like, if you see that people who are retained are more likely to commit crime later on, you worry it's just selection on those negative factors. Their parents didn't advocate for them or they didn't get along well with their peers or something like that, and you can just never see that stuff in the data.

 

Michael [00:16:21] Yep, it's- right. And so what you need here is you need something that will affect the likelihood of being retained, but that's unrelated to all these other factors. So, as you said, some kind of natural experiment, something that functionally randomizing students. But this is, again, where we run into a problem, right. No one would allow you, nor should they, to randomly hold students back. Right. That's not what we're about here. Right. And even if you did that, you wouldn't really get the, you know, parameter you were interested in. Right. Because that- the question isn't, "Does randomly holding students back affect their outcomes?" It's "Does holding the students back who we think, or who someone thinks should be held back, affect their outcomes?" And so what we really need is among that set of students, you need something that can kind of randomly assign some students to be held back and some students not to be held back, and so it's a very hard environment to find the right way to study this, right. Because there aren't a lot of examples of the types of policies that would do exactly that. Although we have one.

 

Jennifer [00:17:29] You have one, yeah. So tell us- so you found one in Louisiana, yeah. So tell us about the context in Louisiana and the specific intervention that you're focused on in this paper.

 

Michael [00:17:39] Yes, so the context from Louisiana is that in 1998, they passed a law that, among other things, imposed a test-based promotion policy on, a rule, on students. And so the way this works is that in March of each school year, eighth graders take a state set of state exams and they have to score at proficient or above on the- both both the Math and the English, or English/Language Arts, portions of the exams to be able to move on to the next grade. If they fail either one of those exams, they have access, at no cost to them, to 50 hours of summer school instruction. They are not mandatory- it is not mandatory to take up the summer school instruction, but they all have access to it. And actually enrollment rates in the summer school programs are pretty high. So they can potentially enroll in summer school. And then in July, they take a second exam. So the students who failed the one or both exams take one or both of the exams they failed again in July. And it's a different exam, but testing the same information. And if they fail the exam again, they are retained. So they, by rule, cannot move on to the next year and must repeat eighth grade.

 

Michael [00:19:08] So that's the rule- that's the rule we use to examine the effects on retention because what you can think about here- so the technical term for what we do is a regression discontinuity design. What the test-based promotion policy does is it creates a cutoff, a test-based cutoff, that's score-based. Right. So students who score two points apart on this test can be treated differently with respect to whether they are retained or not. Right. And so we use that directly to study this question by functionally examining students who just passed versus students who just fail. These students are likely to be extremely similar on every single characteristic, right, on average. And the reason is that there's a lot of randomization to tests. Right. So, you know, maybe you filled in a bubble wrong or you didn't sleep that well last night or, you know, there's a dog barking or something like that. Right. Or, you know, you studied- you know, you didn't study one specific area that's being tested and so you got unlucky. So those kinds of things happen all the time in testing. And so while we feel that generally tests probably tell you something about how much a student knows, if you focus in very narrowly on a set of scores, students who score a couple points apart are unlikely to be very different on average.

 

Michael [00:20:33] And that's the kind of idea behind these regression discontinuity designs. Is that local to these cutoffs- so in a small area right around these test score cutoffs, students who score right above or right below are basically randomly assigned. Right. And so we have the characteristics here that we were talking about before. Right. That right- very close to these test score cutoffs, we have students who score above and score below. Whether they score above or below is due to a lot of random factors that have nothing to do with the underlying characteristics or anything like that. It's just kind of random noise in the test score. And so it generates this randomization local to this cutoff in who is retained and who isn't. And that's what we look at. So we're examining retention outcomes and criminal justice outcomes or criminal engagement outcomes later in life among students who barely pass and barely fail.

 

Jennifer [00:21:32] And you're focused here on that July test. So they, you know, they took the test earlier. They fail that. And then they go to summer school and they take it again. You guys are focused on this July test, so talk about why you do that.

 

Michael [00:21:43] Yeah. So we look at only students who take the July exams, so only students who fail the March exam. And the reason we do that is because failing the March exam has two effects. One is it gives you access to summer school. And the second is that it puts you at risk of being retained. But it doesn't necessarily make you have to repeat eighth grade, right, because you can pass the July exam. So what we wanted to do is focus down on to a set of students for whom the test score outcome was determinative in in with respect to whether they had to repeat the eighth grade. And so we look at students who failed the March exam and all who had to take the July exam and then examine outcomes among seniors who barely passed and barely failed the July exam.

 

Michael [00:22:29] And so why this is also nice relates to a point I made earlier about thinking about how this paper relates to the general literature. This is a very disadvantaged set of students and I think they are of independent interest to examine in terms of how their life outcomes evolve with respect to their educational investments. They tend to be very heavily minority, which in Louisiana means there is a high percentage in particular of African American students who fall into this group. And they they are very low achieving in terms of their test score outcomes. And so I think this is this is in terms of when you think about who are the students who are at risk of of being, you know, engaged in some type of criminal activity or getting caught up in the criminal justice system, these are exactly the kind of students we worry about and who we think are at the highest risk. So so they're a very important group of students to study and fortunately for us, that's the group that really is the the most likely to be in this July sample of test takers.

 

Jennifer [00:23:33] Briefly want to talk about kind of what happens when they are- so they fail the test, you're also in a setting where not everyone who fails the July test winds up being retained. Right. There is still- there's some advocacy on the margin or other things to go on. Some people who don't fail get held back and some people who do fail don't get held back. So talk a little bit about that.

 

Michael [00:23:55] Yeah, so this is what we call one of my favorite terms in all of economics, a fuzzy regression discontinuity design. So the fuzzy RD is exactly what you said, that it's not the case that failing makes you go from zero percent chance likelihood of being held back to 100 percent likelihood. Right. That would be a sharp regression discontinuity. The fuzzy RD means that failing the test increases dramatically in this case, the likelihood of being retained. But it's not from zero to one. Right. So there's two reasons for this. One is that some students are retained even though they pass the test. There is actually an attendance requirement and about 5 percent of students who pass this exam fail the attendance requirement and are forced to repeat the eighth grade. The bigger issue's on the other side, about 35 percent of students who fail the exam are still able to move on to ninth grade. And there is an appeals process that you can appeal to the state. And for the most part, it's related to some kind of hardship you've experienced that explains why these test scores are not indicative of your actual educational ability or achievement. And so about 35 percent of students successfully appeal.

 

Michael [00:25:14] But one of the nice things about the regression discontinuity design is that these problems, if if it's really the case that functionally around this- these test score thresholds that students passing versus failing are randomly assigned, the fact that it's not perfectly determinative of whether you are retained is not actually a problem. All you need is that, for example, the proportion of students who don't pass the attendance requirement are similar on either side of the cutoff, which is the case here. Right. So you just need the proportions of students who don't comply with the test-based rule. You need those proportions of students to be similar on either side. But that's where this kind of functional randomization makes it very likely that that's the case. Right, because it's very likely that, you know, a bunch- you know, 35 percent of the students just just above the cutoff who just just made it, if they had not made it, would have also gotten hardship exemptions. Right. So it is an added dimension of complexity, but actually for the credibility of the approach and the design, it doesn't create serious extra problems.

 

Jennifer [00:26:23] Right. And we talked a little bit about this on the show in the past, with respect to other papers, and I think the key to think about here is really just the results of your study are going to be relevant to those compliers, right, the people who are moved by the test score cutoff. And so so here, you know, we might think maybe it's not relevant to the people who have the type of parents who would go appeal the process or something like that. But for a big share of this population, it's going to be relevant.

 

Michael [00:26:48] That's right. And we find that the effect of being right below the cutoff increases the likelihood that you are retained by 68 percentage points. So it's a big effect. And most students are indeed complying with it, so that- it's not a weird group of people, right, in some way. Which often with regression discontinuity designs the compliers are kind of an odd group of people.

 

Jennifer [00:27:11] Right, that's what you worry about.

 

Michael [00:27:12] Yeah, you worry about. But I think in this case, we're pretty confident that these- the compliers here are not oddly determined. Right. Most students are complying with these rules.

 

Jennifer [00:27:25] OK, so I want to talk about potential mechanisms here. So one thing that struck me as I was reading your paper is that when you're comparing people just below and above this test score threshold, so people who are below or held back, but the people who are above might be treated in a way too. Right. They kind of- they came close to being held back and just barely avoided it. And you can imagine that having a psychological effect. So those who are- who score just above the cutoff might view their good luck as sort of a second chance or a wake up call that gets them on a better track. And this means we might expect to see immediate improvements in their effort in school and maybe the people they hang out with, relative to people who just barely failed the second exam and were held back. And maybe that affects their criminal behavior. So first is that, you know, is that one possible channel that that you have in mind here? And then what else should we be thinking about? How else might grade retention affect crime?

 

Michael [00:28:19] Yeah, so first, that is a really important potential mechanism and actually that's a mechanism that would kind of mess us up. So no, no, no, it's fine. And we can talk about it now or we can talk about it when we get to the empirical evidence on mechanisms. I'm fine with that. But the main assumption that underlies the move from- so we can get at the effects of failing this exam through the regression discontinuity design pretty in a straightforward way, as long as there isn't selection into one side of the cutoff. So as long as students can't decide, I'm barely going to pass this exam and that's all I'm going to do. And there's a lot of evidence we find that that's not happening. Right. That students can't perfectly target their exam scores. And as long as they can't do that, we're in good shape.

 

Michael [00:29:04] But moving from passing the exam- the effects of passing the exam to the effects of grade retention actually requires this extra step, what the technical term of which is the exclusion restriction. Which is that the only effect of failing the exam comes through grade retention. Right. And I think the mechanism you articulated is actually the main reason you might worry about that interpretation. Right. It could be that failing or passing, right, they're relative terms, has direct effects like, "Oh, I got lucky" or you're like, "I'm an unlucky person" or something like that. And that generates differences across students in their subsequent behavior. Right.

 

Michael [00:29:46] So just let me quickly, and we can come back to it if you want. The the evidence that suggests this isn't what's happening is twofold. The first is that we actually don't find any effects on juvenile crime. And you'd expect if that were the main mechanism, you'd start picking stuff up earlier in their in their lives like before they're 18. And the second is we actually find a lot of evidence of reductions in educational investments and in noncognitive skills. But those take several years to start manifesting themselves. And in fact, right after students are retained, the year right after, they actually start exhibiting better outcomes on some of these dimensions. And so if there were kind of a direct effect of failing or passing the exam, you'd expect it to show up in the more short run than in the long run. And that's not what we see. So I think the evidence is more consistent with there not being this direct kind of failure or a passing effect.

 

Michael [00:30:43] So in terms of the other mechanisms, what we have in mind, is is at base that it reduces- that being retained reduces human capital accumulation. And this can happen through many different mechanisms, so it can change the social fit that students experience in school. There could be you know, they could face, you know, negative peer effects or negative social pressures from being retained. It can change the quality of the schooling resources they have access to. So this is both in terms of the types of peers to which they're exposed, but also in terms of the direct school resources that they are exposed to. And it can also send a signal to them — this isn't just failing the exam, this is actually the retention decision — that education is not really for them. Right. Which can, over the long run, can change how they invest in education for themselves. Right. And education is this kind of interesting thing in the sense that it's what we call customer input technology. Right. So what you get out of it is a function of how much you put into it. And so just going to school and and not paying attention and not doing anything doesn't really generate the kind of learning and skills that we often associate with education. Right. You actually have to do something while you're in school. You got to do the work. You got to invest in your own learning. And so changes that students experience that alter how much they invest can be positive or negative, depending on what direction that's going. And I think that could be a large part of the mechanism here.

 

Michael [00:32:26] And so these human capital reductions themselves can can generate both cognitive- changes in cognitive and noncognitive skills. Right. And that each of these can can actually affect the likelihood you engage in criminal activities later in life. So, you know, on the cognitive side, also on the cognitive side, one thing that can happen is that if you reduce the amount of skills you're accumulating in high school, it can reduce your job market opportunities. Right. And this this in and of itself can cause a decrease in the opportunity cost of engaging in crime later in life because your job market opportunities are lower, your wages are lower. And so you have kind of less to lose. Right. By engaging in criminal activity and potentially more to gain.

 

Michael [00:33:19] And it also- in particular, the noncognitive aspects are important because of self-control. Right. A lot of crime occurs, in my understanding, because of a lack of self-control and in particular, violent crime, which is an important part of the story, really can often come about because two people get into an argument, one person loses control and assaults the other person. And in fact, the largest category of violent crime in our data are assaults. Right. And so these kinds of self-control skills are something that is taught in schools and schools' contribution to noncognitive skills has been very well documented by many, many researchers in many, many different contexts. And so the effects of reducing one's investments in those skills can be pretty large and can can generate increases in crime later in life. And in particular can very plausibly increase the likelihood of engaging in violent crime.

 

Jennifer [00:34:23] So what data do you have to dig into these questions? You mentioned how hard it is to get linked education and crime data, but again, you've managed to do it. So tell us about- tell us about those data sets.

 

Michael [00:34:34] Well, I want to- I want to give full credit to my coauthors for doing all of the very important data work here. They're really the ones who have assembled the data and done all the very hard work in terms of getting different state agencies on board, keeping them on board, and doing all the kind of important legal work to make sure that we're not we're not violating confidentiality in laws which are very, very, very important to make sure you're not doing. So the dataset is really a merger of two administrative datasets. The first is data from the Louisiana Department of Education, which gives us administrative data on every single K-12 student in the state of Louisiana. So we have very rich data on who these students are in terms of their, you know, what schools they're going to, what grade they're in, what the age for grades — so we know kind of whether they're on track. We have some information about their socioeconomic background. We have race information. We have information on whether they're eligible for free or school- free or reduced price lunch. As well, and in this case very importantly, access to all of the standardized test scores they've taken throughout their schooling career. And so that's very important for us not only to measure whether they're retained because of failing the exam, but also we have information about prior test scores that give us some kind of baseline information about academic achievement of these students.

 

Michael [00:36:07] And these datasets are great. A lot of different states have them. You know, getting those is hard but feasible. And the real innovation here, again, all credit to my coauthors, comes from the merger of these data with information from the Louisiana Department of Public Safety and Corrections on the universe of criminal convictions in the state as well. And the key thing is being able to merge the criminal convictions data with the schooling data, which we can uniquely do. I'm not aware of any other research team that's put together a dataset like this at the state level that can give you information on the history of schooling, inputs or outcomes for students, as well as, you know, universe level- or population level data on criminal convictions within a state.

 

Jennifer [00:37:02] And so since you've got convictions, you're not you're not going to be looking at arrests, for instance, right. This is crimes that people have been convicted of regardless of the sentence if I'm recalling correctly.

 

Michael [00:37:12] That's right.

 

Jennifer [00:37:13] OK.

 

Michael [00:37:13] Yes, that's right. So if you were convicted of a crime in the state of Louisiana, you are in our data. And I think this- the difference between arrests and convictions is an important distinction. I think there's pluses and minuses to this. You're more- you know more about the crime area than I do, so tell me if you think I'm off base here. But, you know, the way I think about this is, you know, on the minus side, right, there's a bunch of people who are caught up in the criminal justice system who we're not observing. Right. And you might worry that, you know, who was actually convicted conditional on getting arrested. On the plus side, convictions are really important, right. So if you're just arrested and then released or not convicted, there's less of a negative outcome both for yourself and for society for that. Right. The likelihood is you probably didn't commit the crime. Right. So it's not clear what what arrest without conviction actually means. What's nice about convictions — or looking at conviction outcome I should say, not convictions — is that it's a very serious outcome. And yes, people are wrongfully convicted, but I think for the most part, the types of convictions we're looking at, these aren't serial killers. These are like an assault happened and people witnessed it. Right. So the likelihood that they get the person is is probably high. So these are very serious outcomes, both in terms of their cost to society and in terms of the personal cost to the individuals. And so I think there's value to looking at this very serious outcome.

 

Jennifer [00:38:48] Mm hmm. Yeah, I mean, I think this is an area where we'd, of course, love to have both arrests and convictions, but that's even harder than than linking one set of data with the school records is linking two. Yeah, and I agree. I mean, I think that, you know, having convictions, it won't- not- everyone who's convicted didn't necessarily do it, and there are plenty of people who did commit crimes who aren't convicted, but the conviction itself has real meaning and does affect your life going forward. So I agree that they are important.

 

Michael [00:39:18] Good, thank you. And look, it's what we have, so yeah I think it's important- but that doesn't necessarily mean we should. Right. But I think in this case it is- convictions are informative with respect to the core questions we have about how this policy and educational investments affect long run outcomes.

 

Jennifer [00:39:35] Absolutely. OK, so what do you find? What are the main results in terms of the effects of grade retention on crime?

 

Michael [00:39:43] So first, as I said before, we find this this first stage effect, right, that indeed failing this July promotion exam has large effects on the likelihood you are retained in eighth grade, it's about a 68 percentage point effect. So that's kind of the first stage. So for the rest- in the paper, we look at both the effects of failing and the effects of retention, I'm just going to focus on the retention results.

 

Michael [00:40:03] So overall, we find that being retained in eighth grade increases the likelihood you are convicted of a crime by the age of 25 of 1.25 percentage points. This is about an 11 percent effect relative to the mean, but it's not statistically significantly different from zero. It kind of- even at the 10 percent level. So it's it's a, you know, decently sized effect, but it's not significant. But then, when we look separately by different types of crime, we find in particular that there's a big effect on violent crime. So the likelihood that someone is convicted of a violent crime increases by just over 1 percentage point when they are retained in eighth grade. And that's a 58 percent effect relative to the baseline. So it's a it's a big effect relative to the underlying likelihood that someone's actually convicted of a violent crime.

 

Michael [00:40:55] We also find some suggestive evidence of increases in drug crimes. There is the not statistically significant increase of about a half of a percentage point, which is about 10 percent. So really, it looks like the main story here is an increase in violent crime. And, you know, the overall increase in crime that we find that's not significant, really seems like it's reflecting a big increase in violent crime and maybe a small increase in drug crime. We find no effects on property crime. Right. Which is surprising, but it's pretty robust fighting. However, we do find effects on robbery. So robbery is technically a violent crime, but it's property related. It's just that it's basically theft with a weapon. And so we find- or whether there was physical violence. And and so, you know, there's maybe some evidence in there that some of this is related to property crime as well.

 

Michael [00:41:50] So those are the kind of the main estimates in terms of whether someone commits a crime. But the other kind of interesting aspects of our data is we have if you are convicted, you have the set of things you're convicted of. Right. So we know the number of crimes that you are- of which an individual is convicted. And so this is one measure of kind of the severity of the offense you've done. Right. So being convicted of one thing is probably less severe than being convicted of a whole bunch of different things. Although, you know, that might vary across instances. So when we look at the number of crimes you're convicted of, at first conviction, we find a 60 percent increase in the number of additional crimes that you are convicted of. That's also not statistically significant. But again, the effects are really coming through violent crimes, where we find an 85 percent increase in the number of violent crimes of which an individual is convicted, again at first conviction. And so our interpretation of this evidence is that people are more likely to be engaged in in criminal activity when they're older, when they're retained in eighth grade. That this mostly loads on violent crime. And that there's some evidence that the severity of crimes they're they're convicted of increases as well. So it's both an intensive and extensive margin effect.

 

Jennifer [00:43:07] So as you said, it's- this is the first conviction up through age 25. I can't remember if you said the 25 part.

 

Michael [00:43:14] Yes it's 25. I think I did in the data section, but I didn't remind everybody. So, yeah so and the reason and most importantly, the reason we look at the first conviction is because, because of the structure of the data we don't see everyone's lifetime histories. Right. And the the policy was implemented in 1998-99, so we can't look for people who were in school in the 80s. Right. For whom we would have very long run outcomes. And so this generates- kind of functionally it generates truncation in the data. And so if you were to look at, for example, recidivism, which is a very important outcome, the set of people who are eligible to recidivate are those people who went to prison but were released within the time period of our data. And that's a very selected sample. And so it makes the most sense from a data perspective to look at first conviction.

 

Jennifer [00:44:01] OK, so let's go back to the mechanisms you talked about earlier. You're able to do a little bit to test which channels seem to be driving these effects. So tell us how you do that and what you find there.

 

Michael [00:44:13] Great. So we look at several different markers of human capital investment and achievement. And so the first we look at is dropout. So we find a 7 percentage point increase in the likelihood that students drop out of high school if they are retained in eighth grade, which is a big effect, it's about 16 percent relative to the mean. So these students are are much more- students who are taking the July exam are much more likely to drop out than other students. But even among those students, failing this exam has a big effect on the likelihood of drop out. So that's one potential mechanism, that they're just obtaining less schooling — they're obtaining about 0.25 fewer years of schooling than the students who were not retained.

 

Michael [00:44:56] We also can look at behavioral outcomes that are pretty strong measures, other research has shown, of nine cognitive skills. And the two that the literature is used and that we also use is attendance and behavioral infractions, so whether you are suspended or expelled from school. And so what we find on those fronts is that indeed students who are retained are- have lower attendance and have more behavioral infractions. But as I said earlier, the effect takes several years to show up. So what it looks like is happening is that there's a pattern of slightly better outcomes at first. And part of that might be because, you know, the way behavioral infractions work in middle schools and the way attendance works in middle schools is different than in high schools. There's less opportunities potentially to be suspended or to skip school without someone really noticing. And so some of that in the short run could be mechanical. But what we see is that while there's this short of this- the year after, some of these outcomes are a little bit better, by 3 years after failing this exam, students are significantly less likely to attend school and they are more likely to have a behavioral infraction. And so we see this- we interpret this as evidence that their noncognitive skills are degrading over time, likely as their investments in schooling are declining.

 

Michael [00:46:28] And the third piece we look at is peer quality, which is kind of — or peer characteristics, I should say — which is an interesting outcome because usually the way this works in most of the United States is that your eighth grade kind of determines your high school. Right. Your eighth grade feeds into a specific high school. But in Louisiana, there's a lot of school choice, meaning that students are able to attend schools that are not the one they're zoned for. And interestingly, what we find is that students who are retained then when they do go to ninth grade are in ninth grade schools with peer characteristics that are associated with worse outcomes. So their their peers have- are from lower socioeconomic status backgrounds and are- have lower achievement, academic achievement, as measured on on standardized test scores. And so we think it's a combination of all of these factors that really lead to getting back to our earlier discussions that at base, students are reducing their human capital accumulation by when they are retained. Right. And this is showing up in all these different ways. It's showing up in some markers of cognitive outcomes. It's showing up in markers of noncognitive outcomes. And that these then lead them to be more likely to commit a crime and be convicted of it later in life.

 

Jennifer [00:47:55] So I'm really interested in this finding that students seem to move to different or worse schools when they're retained. So just in terms of the context there, I mean, just is that because they, you know, you still have to apply and you're just not getting into the good schools? Is that is that basically what's going on or something else?

 

Michael [00:48:10] Most of the school choice in the U.S is not application-based. So it's not like- it's not a competitive application environment, it's lottery-based. So anyone is eligible to apply to go to these schools. And if the spaces are oversubscribed, then they run a randomized lottery that has nothing to do with your educational backgrounds, et cetera. Right. There is evidence, however, that schools can- so no one's looked at it with respect to retention. But there's evidence that some schools of choice try to limit students who are- have disabilities, right, and who have special education services or who have behavioral problems. They do what they can to try and keep those students from applying. And so that could be part of what's happening here. It could also be that parents are making different choices for their students based on this outcome. Right. And that "Oh, now, I don't want you to go to this better high school because it's going to be too hard. You know, now I'm going to send you to this other high school where I think you will be more academically capable of succeeding." So that's another potential mechanism, is this is this is, you know, based on parents' and kids' decisions about what they want to do. We were surprised by this finding, I will say, but it's pretty robust.

 

Michael [00:49:33] And it's also important to recognize that this is something that will not be the case in areas that don't have a lot of school choice. However, a lot of the states and school districts that have these kind of test based promotion policies also have a lot of school choice because they- these policies, there's no mechanical relationship but the kinds of lawmakers who tend to like school choice, also tend to like these test-based promotion policies. So, you know, politically, they tend to go together a lot and so they often are linked. So this isn't as idiosyncratic a finding to Louisiana as one might think.

 

Jennifer [00:50:05] Yeah, and there certainly is evidence in other crime related papers that peer effects really matter. And then thinking specifically of Megan Stevenson's "Breaking Bad" paper, where she looks at peer effects in juvenile detention. And that made me wonder, actually — and you might have the data to do this — if you could look at whether there's a change in the share of school peers who are themselves convicted as juveniles or adults to kind of directly test whether those who are retained have more, say, crime prone peers than others.

 

Michael [00:50:32] Oh, that's a really good idea.

 

Jennifer [00:50:33] Yeah, you could probably do that.

 

Michael [00:50:35] Yeah, we had toyed with, like, trying to get it more directly at the peer effect mechanism. It's really difficult. But I like this idea of saying, are you involved more in these kind of criminal networks or something like that. I think the interesting thing here, too, again, is that we're not finding effects in juvenile crime. Right. So I would expect a lot of those a lot of those things to be relevant for for juvenile crime first. Right. And there's this kind of literature showing, you know, being concerned about young people getting involved in the criminal justice system as youth. And that that continues, right? That doesn't seem to be what's going on here, which we were, again, kind of surprised about.

 

Jennifer [00:51:14] Yeah, I think Megan's papers- so I think she finds similarly that just kind of like spending time around people with worse noncognitive skills, then like it rubs off on you, like your own noncognitive skills fall. So like it could still be like the story you were telling earlier, just through this kind of different way of measuring, I guess, the same thing.

 

Michael [00:51:31] Yeah. We could also look at peer behavioral outcomes because we have it for everybody. So it could be also- we could look at, you know, whether you're more likely to be convicted later, but we can also just say, "do you have worse behaving peers?" which we haven't done, but we should do. That's a really good idea. Thank you, Jen.

 

Jennifer [00:51:43] You're very welcome.

 

Michael [00:51:44] And now all your podcast listeners will know when the published paper comes out and they all read it, which I'm sure they will, that you are responsible for those regressions.

 

Jennifer [00:51:55] Well, I hope it works. OK, so so you do a bit of a cost benefit analysis here, or at least a cost analysis, I guess. So what do you estimate are the costs to society of retaining someone in eighth grade?

 

Michael [00:52:09] So we don't have the full cost to society. So what we do is use the social cost of crime estimates that other people have come up with, Donohue in particular in his 2009 paper. And we use the estimates of the social cost of assault and robbery. And what we can get is effects for the cohorts- the three schooling cohorts we study and basically multiplying the estimates per assault and robbery and applying that to the excess convictions that occur because of the grade retention policy. And when we do that, we find that just in these cohorts in Louisiana, the social costs, depending on what- which one you use, because there's a wide range, vary from between 3 and 18 million dollars. So if you multiply that by the broader set of students who are affected by these policies, it's a pretty big number.

 

Jennifer [00:53:04] Yeah, I agree. That's a big number. OK, so based on your paper and the other work in this area, what do you see as the policy implications here?

 

Michael [00:53:17] I mean, so I think the core policy implication here is that as it currently operates, the benefits to grade retention need to be large in order to justify the kind of costs that are occurring here. Right. That there might be benefits in terms of students working harder, in terms of, you know, the incentives that that underlie these policies and in terms of not generating potential negative spillovers on other students. So one of the other benefits the proponents of these policies discuss is that by removing students from the class that are not academically capable of being there, you are allowing teachers to focus on the students who do have the skills and knowledge to be in the grade. Right. And so that can help the students who are promoted. Right. As well as potentially the students who are not promoted by giving them an extra year and by inducing students to work harder. So there are real potential benefits of these policies that I don't want to understate. It's just that the costs we just discussed are high. And so the benefits that outweigh those costs would also need to be high. And there's functionally no research that really identifies those kinds of benefits for anyone, yet. That doesn't mean they don't exist, it's just we don't have those- we don't have that research yet to suggest that.

 

Michael [00:54:40] But another way to read the policy implication here is that there is a set of students who we can identify who are actively harmed by this policy. And these are the students who are being retained. And so another set of policies one can think about is let's keep the retention policy, but invest in those students more directly and try to to not have this be a punitive policy towards them in terms of reducing their investments in schooling. But try to use this as an opportunity to help them invest more effectively and make this a productive experience for them rather than a detrimental one. Again, we don't really know how to do that, but I think we also haven't fully tried to do it. Right. Trying to to provide more resources to make sure that these students are successful when they are retained would be a really interesting policy initiative that I'm not aware of any state or school district engaging in. I would I would like to see them do that. It would be really interesting.

 

Jennifer [00:55:39] Yeah. And your results to showing that part of the effect is coming from moving to schools with worse peers also suggests that those that- those peers who have been in those schools all along are probably- are harmed, right, by that environment too. So sort of like broader issues along the same lines that figuring out how to invest in these kids that are kind of at high risk is really important.

 

Michael [00:55:58] Exactly. Yeah. And the nice aspect of this policy is, you know, we know who they are. Right. It's not like they're hidden. We're like, oh, there's a bunch of students who fail the test-based promotion policy. They're eligible for this set of services. Right. And that they do. And I think, you know, often when when policymakers act, right, they do things in, you know, somewhat in silos, say, "Oh, we're going to do this thing, you know, this one policy," but don't then think about are there other policies we can layer on top of this that will help address any adverse consequences of what we're doing? Right. And I think that's that's a nuanced view of policy, but also one that I think is the correct one. Which is to say maybe the right thing here isn't to eliminate these test-based promotion policies because they may be having real benefits. It's to mitigate or reverse the adverse consequences, which we, you know, if you buy this paper, we can pretty easily determine what those are.

 

Jennifer [00:56:53] OK, so figuring out what those policies are that you want to layer on top and evaluating whether they work, that seems like one area where more research would be helpful. In thinking about the research frontier here, what are the other big questions that need to be answered in this space?

 

Michael [00:57:08] Yeah, so I think for sure that that's the big- to me that's the big one that comes out of this right, is what these other policies are. I also think more research into the specific mechanisms that are occurring here would be important. We have measures of noncognitive skills, which is this attendance and behavioral outcomes or infractions. But I would like to know more about what the specific mechanisms are that generate a decline in noncognitive skills. What is it about the schooling or the change in the schooling environment, the changes in students' behaviors that are producing these declines in noncognitive skills? Because I think they're a huge part of this story and other people who have looked at the crying schooling nexus have also made these kinds of arguments. You know, I don't think we know a lot in a granular sense of how exactly this works. And I think understanding that mechanism, that process is is fundamental to really determining what these these other policies are that you'd layer on top of this policy.

 

Jennifer [00:58:12] My guest today has been Mike Lovenheim from Cornell University. Mike, thanks so much for joining me.

 

Michael [00:58:17] It's been a pleasure. Thanks for having me on.

 

Jennifer [00:58:24] You can find links to all the research we discussed today on our website, probablecausation.com. You can also subscribe to the show there or wherever you get your podcasts to make sure you don't miss a single episode. Big thanks to Emergent Ventures for supporting the show. And thanks also to our Patreon subscribers. This show is listener supported, so if you enjoy the podcast, then please consider contributing via Patreon. You can find a link on our website. Our sound engineer is Caroline Hockenbury with production assistance from Elizabeth Pancotti. Our music is by Werner, and our logo is designed by Carrie Throckmorton. Thanks for listening and I'll talk to you in two weeks.