Probable Causation

View Original

Episode 14: Aaron Chalfin

Jennifer Doleac

Aaron Chalfin

Aaron Chalfin is an Assistant Professor of Criminology at the University of Pennsylvania.

Date: October 15, 2019

Bonus segment on Professor Chalfin’s career path and life as a researcher.

A transcript of this episode is available here.

See this content in the original post

Episode Details:

In this episode, we discuss Professor Chalfin's work on the effects of streetlights on crime:

"Reducing Crime Through Environmental Design: Evidence from a Randomized Experiment of Street Lighting in New York City" by Aaron Chalfin, Benjamin Hansen, Jason Lerner, and Lucie Parker.

OTHER RESEARCH WE DISCUSS IN THIS EPISODE:

Transcript of this episode:

 

Jennifer [00:00:06] Hello and welcome to Probable Causation, a show about law, economics, and crime. I'm your host, Jennifer Doleac of Texas A&M University, where I'm an Economics Professor and the Director of the Justice Tech Lab.

 

Jennifer [00:00:17] My guest this week is Aaron Chalfin. Aaron is an Assistant Professor of Criminology at the University of Pennsylvania. Aaron, welcome to the show.

 

Aaron [00:00:24] Thanks for having me. I love the podcast. So it's a privilege to be a contributor as well as a listener.

 

Jennifer [00:00:30] Oh, great. Good. Good to have you here. So we're going to talk today about a very cool field experiment you ran in New York City, testing the effects of streetlights on crime. But before we dive into that, could you tell us about your research expertise and how you became interested in this topic?

 

Aaron [00:00:46] Sure, so I'm a professor of Criminology, but my PhD is from a School of Public Policy, so I've always found myself interested in how social planners can control crime in a way that's efficient. And, you know, we can always use heavy handed tactics like, you know, lots and lots of arrests or putting lots and lots of people in prison. And that, to a degree, is effective. But, you know, it's really very costly both to taxpayers and to communities. And so all politics aside, it's simply not a sustainable way to control crime in a free society. And and so I think we have to think creatively about how to complement enforcement in a way that uses carrots and not only sticks. And that's really where a lot of my interest in this in this area comes from.

 

Aaron [00:01:37] With respect to how I get interested in streetlights, I think there's really two answers to your question. So more abstractly, I guess for for some time, I've been really bullish on the built environment, meaning the design of physical space and what a community looks like as a means to control crime in an effective way. And I think there's a number of things to really like about the built environment. So first, if you can if you can keep crime low by just changing the physical environment rather than through more and more enforcement, more and more incarceration, that feels like it's going to be a huge win. If a crime happens, the victim suffers some costs and then you have to arrest a person, adjudicate them, punish them. That's very expensive for a lot of people, including the taxpayers. But if you can dissuade people from offending in the first place, that's always going to be a lot cheaper. I think that's what we ought to aspire to do when we think about criminal justice policy. And, you know, furthermore, people tend to like environmental design changes that make their communities feel more safe and more orderly. So they tend to be pretty popular and have some downstream effects.

 

Aaron [00:02:50] So second, you know, when we when we think about this sort of set of creative ideas that use carrots rather than sticks- so one set of ideas might revolve around the physical environment, another might revolve around social programing. So I'm thinking here about programs that that help provide people with monetary or social supports that make their lives better and easier, that might make crime seem less needed or less attractive. And there's lots of wonderful programs that exist, you know, for people who live in disadvantaged communities. But these programs often rely on a recipe that seems like it's very hard to reverse engineer and very hard to copy. Right. Things like charismatic leaders and just unusually, you know, really dedicated employees, detailed knowledge of community norms. And those people don't seem to grow on trees. Right. There's not an unlimited supply of those people. And so what is often found to be the case is that you can take- it's very hard to take a program that works really well for a couple hundred people and find a way to give that program to everyone who needs it. The nice thing about something that changes the physical environment, like street lighting, is that we sort of know how to scale it right. You just produce more lights and you install them. Right. And obviously it's a little bit more complicated than that, but it feels like it calls for less scale up expertise and knowhow than something like social programing. The intervention I'll talk about today costs only about $75 per community resident. So that also feels optimistic.

 

Aaron [00:04:27] And then finally, when I think about a typical person who commits crimes, I generally think about someone who is thinking about maybe the next 10 minutes of their lives and not the next 10 years. And that's something that maybe a lot of people think might might be true for for this population. And so if offenders tend to be myopic, then, you know, you might think that you can change behavior a lot better by focusing on something that's experienced in the present, than something like punishment, which will probably never be experienced because most crimes aren't cleared. And if it is, it'll be experienced sometime in the distant future. So that's sort of the basis for my interest in something like street lighting.

 

Aaron [00:05:12] But more concretely, prior to joining the faculty at Penn, I used to work full time for a really cool research organization called the University of Chicago Crime Lab. And through the Crime Lab, I had the chance to meet with some policymakers in New York City who were interested in learning more about lighting. And so the project really came about through a stroke of good luck, an interested researcher met some visionary policymakers who cared a lot about generating high quality evidence.

 

Jennifer [00:05:42] Your paper is titled "Reducing Crime through Environmental Design: Evidence from a Randomized Experiment of Street Lighting in New York City." And it's coauthored with Ben Hansen, Jason Lerner, and Lucie Parker. So let's talk about street lighting. It might seem obvious to people that crime is more likely to occur when it's dark outside. But of course, researchers like to think about underlying mechanisms in more detail than that. And it turns out there are a bunch of different ways that presence of street lights might affect crime in a local area. So talk us through those. What are the potential mechanisms we should have in mind for why putting a streetlight somewhere might affect crime nearby?

 

Aaron [00:06:18] Yeah, so as you alluded to, there's there's lots and lots of ways that street lighting could affect crime. I'm going to summarize the major ones briefly in the interest of time. There are certainly others, so if any of your listeners are thinking of something that I didn't mention, I don't mean to suggest that-.

 

Jennifer [00:06:37] They shouldn't email you.

 

Aaron [00:06:37] That that's invalid. Yeah, but but really, I think the most obvious way that street lighting might matter, right, is that, you know, you have more lighting, so maybe a crime is more likely to be witnessed or a police officer might observe that crime. And so it's a deterrent to people who are thinking about offending. Another possibility, right, is that lighting might empower potential victims, either by causing more people to leave their homes and spend time outside, right, which increases more eyes on the street — something that you've worked on, Jen. And, you know, another possibility is, you know, I'm walking around outside and I now have greater visibility. So I'm walking down the street and I see a little old lady and I say, well, that's that's fine. That's that's not a problem. But maybe a block past that little old lady, I see someone who sort of seems suspicious or up to no good. And you know, I can cross the street. I can I can take some precautions that would be more difficult to take if if ambient lighting were not as strong.

 

Aaron [00:07:39] We could also think that lighting is a complement to existing crime reduction technologies like police officers or surveillance cameras. Right. Those those are technologies that that can be more effective with with better lighting. And so, you know, lighting potentially could lead to more incapacitation of would-be offenders. Another possibility, which has been proposed predominantly by criminologists and sociologists that I think we really need to take very seriously is that lighting might be a signal to people who live in a community that an area is cared for, that it's being looked after, that it's being watched and accordingly, that it's an unwelcome place to commit crimes or to hang out if you might sort of be up to no good. And and so those are some reasons why lighting, you know, could potentially reduce crime.

 

Aaron [00:08:32] Now, lighting, of course, could increase crime. Right. You know, if there's more people outside because they feel safer, then just mechanically that might be more opportunities for crimes to occur, for fights to break out or something like that. Also, just as lighting empowers potential victims, it can also empower potential offenders. Right. Maybe it allows me to spot a particularly attractive victim. I see a guy stumbling around, obviously very drunk. You know, maybe I'll go rob that guy. Right. So, you know, apriority, it's not obvious that that lighting would have to reduce crime.

 

Jennifer [00:09:08] So before this study, what had we known about the net effects of lighting on crime?

 

Aaron [00:09:14] Yeah, so this has always been one of the really interesting questions for me as I started to think about street lighting and the literature that's led to where we're at right now. So street lighting hasn't been around for years or even centuries, but really in one one form or another, it's been around for millennia. Street letting in the form of oil lamps could actually be found in antiquity — certainly cities in the Greco-Roman world had, to varying degrees, some some streetlights. So this is probably an idea, a crime control idea, that's as old as civilization itself in some ways. Not to digress too much, but I listen to to in addition to this podcast and another wonderful podcast on the history of Rome by a gentleman named Mike Duncan. And he was talking about dining habits in the city of Rome, you know, just after the year zero. And, you know, people would dine often at the the homes of of a patron or friends. But but meals would end early so people could get home because the streets were not uniformly lit in Rome. And everyone recognized that, you know, wandering through the narrow streets of Rome at night was unwise. It was dangerous. Right. So this is an idea that's very, very old.

 

Aaron [00:10:37] In the U.S., we've had street leading to varying degrees since the 18th century. It might interest some of your listeners to know that Ben Franklin is often credited with having designed the first candle-based street light in the U.S. for the city of Philadelphia. And so, you know, given how long this idea has been around, you'd think we'd have pretty good evidence by now, you know, for whether this works and how it works and under what conditions it works. But incredibly, the evidence is is pretty thin and it's not because of a lack of interest. A recent meta analysis written up by a couple of prominent criminologists identified 32 prior studies of street lighting in the U.S. and the U.K. But all this evidence is observational. Right. It's not- there's no experiments in this literature. And it's funny, like when you think about what it would take to bring a drug to market, what the FDA would insist upon if we were to think about a drug used to treat a disease, right, we would demand that there is not just one clinical trial, but multiple clinical trials. We have nothing like that in this literature.

 

Aaron [00:11:44] What these studies tend to do is to look at a community that that got some kind of enhanced street lighting and they just simply say, well, did crime go up or down? And, you know, crime goes up and down for many, many reasons, right, you know, weather, just the dynamics of of gang activity, community norms, whatever. And it's really hard to know if any change in crime was due to street lighting and not something else. And so that really serves to limit our confidence in these findings. By the way, the findings are very mixed. On average, people tend to find that street lighting reduces crime, but but plenty of studies find find the opposite. I actually think the best evidence comes from a paper of yours that you wrote with Nick Sanders at Cornell that looks at daylight savings time. Right. All of a sudden, it's six o'clock and it's dark out when it used to be light. What happens to street crime? And you find that that crime is sensitive to ambient lighting, which I think is powerful proof of concept. So what we're trying to do is just take that proof of concept and see if there is something that policymakers can do to to leverage that idea further.

 

Jennifer [00:12:59] Yeah, and we are careful in that paper to note that lighting like sunlight is not the same as street lights, right. That creates a very different kind of light in many ways. And so we really do need to know what the effect of streetlights are, because that's what we have power over in the real world. And so, so you've already talked a little bit about the identification challenge here, but let's dig into that a little bit more as well as any data challenges you face. So so talk us through the hurdles you had to overcome in order to do this study.

 

Aaron [00:13:27] Yeah. So as I mentioned, what you know, what's done in the previous literature is you find a place that had, you know, some street lights put in, you see what happened to crime. In some of the better research out there, there there is a comparison place as well. So you say, well, did crime change more in the place that got the lights compared to some, you know, hopefully similar place that didn't get new streetlights? And, you know, the real challenge is you need to find a place that's not just hopefully similar, but really is credibly similar to the place that got new lighting. And that's very, very difficult to do. Right. If you think about a community- the community you live in, can you identify another community in your city that's just like yours in all ways and is experiencing the same crime trends? Probably not. Right. And you might think that policymakers, when they have the ability to allocate new lights to a community, are not going to just do that in a totally random way. They're going to be thoughtful about it and they're going to think, well, where are these lights going to be more effective? Where are we experiencing a crime problem that we're having trouble controlling? And, you know, that makes it really, really hard to believe that you can easily find comparison areas for for the places that that do get streetlights.

 

Aaron [00:14:45] Now, there's other issues as well. In the interest of time, let me just mention one more that I think is really important. So so past studies of street lighting, even the ones that tend to be better executed, are typically only looking at one treated area, like one neighborhood, sometimes two neighborhoods. And they'll find, you know, for those two treated neighborhoods two, you know, possibly similar places to compare those neighborhoods to. And so at the end of the day, we're looking at what happens to crime in like two or four places. And, you know, when you think again about, let's say, the FDA approval process to bring a new drug to market, if I told you that I was trying to figure out if Lipitor was effective in controlling blood cholesterol and I did a study on four people, you'd probably say that's surely not enough evidence to bring a drug to market. And so, you know, and what I think you're implicitly thinking about there is what researchers would call statistical power, statistical significance. Right. You know, is this a result that's very likely to be real or could it just be due to random chance? And in small samples, results, you know, often can be due to random chance, and that's another problem that's running around in this literature.

 

Jennifer [00:16:04] So to measure the causal effects of streetlights, you and your coauthors do what we all wish we could do. You conducted a randomized experiment. And I love that you did this in large part because it's a great example, as you mentioned, of researchers collaborating with local governments to test the effectiveness of some intervention they're interested in. And this is not super common in practice. It can be very difficult to get all the decision makers on the same page. So give us a little bit of background on how this experiment came about.

 

Aaron [00:16:32] Yeah, so this was this was a really great opportunity. As you note, it's hard to do something like this. Policymakers aren't always interested in devoting time, energy, and resources to generating evidence. And I don't think that's because they're like bad people or they're bad at their jobs or they're only self-interested or anything like that. I think policymakers, you know, face a really challenging environment where they're being evaluated on what's happening in the next three months, not the next three or six or nine years. And, you know, it's really costly to work with researchers. There's lots of things that that a policymaker can devote his or her time to. So we were just really fortunate to be able to work with some folks in New York City who really wanted to see if lighting had potential and valued rigorous, as opposed to merely correlational, evidence. And these are folks who are not not simply trying to sort of say, like we want evidence to back up something we've already decided is good. They wanted to know what was true, you know, which is great for for, you know, as a researcher.

 

Aaron [00:17:35] So just to give you a little bit more more background, for those among your listeners who don't know much about what's been going on in New York City. So beginning around 2014, New York City, which is where the study takes place, started investing heavily in the city's public housing communities. So crime is way down in New York. New York is one of the safest cities in the United States. It's just incredible how safe New York is. But crime has been sticky in some of these public housing communities. And so the city's thinking, well, how can we approach this in a multifaceted way? So they began investing in enhanced social services, in changing the built environment, which included street lighting. And people in the city wanted to know whether lighting in particular was a strategy worth pursuing further. And they asked the Crime Lab. And at the time, I worked full time at the Crime Lab. Well, how could we answer this question, you know, in a really rigorous way. So we proposed a randomized experiment and we worked closely with city officials for the next two years to design the experiment that became this study.

 

Jennifer [00:18:48] All right, so tell us about that experiment itself. So you wanted to measure the effects of adding streetlights outside of public housing. How did you get those streetlights, and how did you decide where to put them?

 

Aaron [00:18:59] Sure, let's yeah, let's talk about the experiment. In fact, let me start by describing the intervention and then I'll talk about the research design, which I think is sort of maybe a separate discussion. So so these are communities that already have normal street lighting. Right. This is New York City, we're not talking about the middle of the woods. These are areas that have never lacked, you know, some kind of ambient lighting at night. But despite the presence of ordinary street lights, obviously it can get pretty dark in parts of these communities. And maybe it's worth describing what these communities look like for your listeners who don't know much about these communities in New York. So New York has 340 public housing communities spread across the five boroughs of the city. Officially, those communities are home to more than 400,000 people. Unofficially, it's probably more like 600,000 people. And there's a lot of variety in how these these communities are laid out. But typically, they consist of a number of large apartment buildings that are set around some green space and benches, playgrounds, et cetera. They have pathways through them. Most are inset from the street. They actually look a little bit like a community of dorms on a college campus.

 

Aaron [00:20:18] And what we're talking about doing are adding not permanent streetlights, the sort that you see in the rest of the city that are that are installed in the ground. We're talking about adding temporary street light towers. You know, I wish I could show you a picture. I guess this is the challenge of the podcasting world. But these are these are very, very bright lights. You might have seen something like this if you've ever driven by a highway late at night where there's some road work being done. And these are the sorts of lights that will illuminate the road so construction workers can see what they're doing. You know, they may be 10 or 15 feet high and they're very, very bright. And I want to I want to talk a little bit more about what these are. So just see get a better sense of what what's happening here. So these lights turn on automatically in the evening and they turn off automatically during the daytime. They're refueled daily during the daytime by a city employee. They're diesel powered. So they make a little bit of noise. If you're really, really close to them, they smell a little bit like fuel. You know, it's really only if you're very, very close to the lights.

 

Aaron [00:21:32] But I think the most important thing to mention is that, like, you can't miss them. Right. They're really, really salient. We're we're talking about adding a lot of new lighting to these communities, but also adding a really important- potentially important signal that these are areas that are being looked after, cared for, et cetera. And by the way, the lights are not wheeled away during the daytime. They're still sitting there during the daytime. So it's sort of a constant reminder potentially to people that something is happening. So these street lights were added to these communities in March of 2016 and they were in place for six months until September of 2016.

 

Aaron [00:22:16] So that's the intervention. Let me briefly describe the research design, because there's a little bit of nuance that we need to talk about here. So there's there's 340 of these public housing communities in the city. 78 of them were selected for the study based on on on prior crime and the perceived need for additional street lighting. So half of these communities, so 39 of them, to be exact, were randomly selected by coin flip to — or something like the computer equivalent of a coin flip — to receive the temporary streetlights. And half were in our control group. In practice, the way we did this is we ranked all of the 78 developments according to their previous level of outdoor nighttime crime. Within the first pair, we sort of flipped our our coin and one of those places got the lights, the other one didn't. The next pair we- one pair member got the lights, the next one didn't and so on and so forth. So this is what's referred to as paired randomization. But the city also, you know, wanted us to try to, you know, figure out how much lighting is needed to reduce crime and how many towers, light towers do you need to put into these communities in order to to generate some kind of effect? And so what we did is within the areas that were randomly selected to receive this lighting intervention, we randomly allocated a different number of light towers to each community. So some of these communities got a lot of new lights and others got a very, very small number of new lights.

 

Aaron [00:23:53] So to sort of summarize, we have an experiment within an experiment. So we have a treatment group and a control group. And within the treatment group, we have randomization in the dosage or the amount of new lighting that these communities receive and, you know, there's some tradeoffs that we had to face in designing the experiment in this way. On the plus side, you know, we had a chance to learn more about what- how much lighting constitutes what we might think of as a clinical dosage. On the other hand, some of the treatment places received only a small amount of new lighting, possibly even less than what a clinical dosage actually is. And so when we make this treatment versus control comparison, that might end up understating the effect of a clinical dosage of of lighting. And so I'm going to focus mostly on the experiment within the experiment, looking at the randomization of dosage of lighting within the developments that receive the new lights. But I'll still use that that control group, that randomized control group as a really, really critical robustness check on the validity of of these results.

 

Jennifer [00:25:08] And of course, this experiment, as you said, was done with the cooperation of New York City, which hopefully made data access easier. So what data were you able to get to analyze the effects of this intervention?

 

Aaron [00:25:19] Yeah, absolutely. So we got proprietary data from the New York Police Department. There's similar data that's now available publicly. Our data differs in a couple of nuanced respects that that, you know, probably aren't worth getting into right now. But, you know, these were proprietary data. They contained information on all crimes that are known to law enforcement except for rapes which were blinded in order to preserve anonymity. Thankfully, outdoor nighttime rapes are not very common crimes in these communities. So it's it's probably not a major limitation. We know the date of the crime. We know the official time stamp of when the crime happened. We know whether the crime occurred outdoors and we know what type of crime it was. We don't know anything about the victims of the crime in order, again, to protect people's privacy and anonymity. We also have some additional data from the New York City Housing Authority just about the characteristics of these communities so we can make sure that when we flipped our coin and randomized these communities to different treatment conditions, that that randomization was effective.

 

Jennifer [00:26:29] And before the experiment started, what did crime look like in these areas?

 

Aaron [00:26:35] Yeah, so these are mostly what we would think of in New York City as high crime areas, but compared to similar areas in, let's say, Baltimore or St. Louis, these would look far safer. Right. So and that's just because crime in general is so much lower in New York than it is in some of the more distressed cities in the United States. So overall, when you think about the level of violence in these communities, it's about two times higher than it is in the most dangerous U.S. state. But overall, we're still talking about small, absolute numbers of serious outdoor crimes in these communities. Let me be a little bit more detailed about what I mean by that. So the typical community community in our study experienced approximately four outdoor nighttime, what are called index crimes during the six months that the study was running. So index crimes are generally the most serious crimes that are known to law enforcement. They comprise murder, ordinarily rape — but not in our data —, robbery, serious assaults, burglaries, larcenies of valuable items and motor vehicle thefts. So these are serious crimes and there's about four of these over a six month period. So a little bit less than one per month. Now they're serious crimes. So so, you know, obviously they have high social costs, but still the absolute numbers are not very high.

 

Jennifer [00:28:05] Higher than New York would like.

 

Aaron [00:28:06] Higher than New York would like.

 

Jennifer [00:28:08] And it is interesting to think about. I mean, so the Crime Lab has this office in New York. They also have an office in Chicago, which is- it suffers from very different problems. And it's interesting just to think about the political economy of going along with research like this. Sometimes policymakers are willing to do experiments when they are desperate and don't know what to do. And sometimes they're willing to experiment when things are actually not that bad and they have more bandwidth to be able to experiment and see what the lessons are. And that feels like the situation in New York is in.

 

Aaron [00:28:34] Absolutely. Yes.

 

Jennifer [00:28:35] Is that in line with your perspective?

 

Aaron [00:28:37] Yeah, no, I think that's that's exactly right. You know, when you don't- when you're not always fighting fires, you have a chance to innovate and you have a chance to learn. And I think that's one of the great virtues of working in New York right now. You know, one of the costs from a research perspective is when you have lower absolute numbers of crime, that creates challenges for statistical power. But yeah.

 

Jennifer [00:29:01] Great. OK, so let's talk about the main results. What do you find is the effect of street lighting on outdoor nighttime crime?

 

Aaron [00:29:09] Yeah, so we find very, very large reductions in these outdoor nighttime index crimes that occur on these public housing campuses, something on the order of around 60 percent reductions. Now, you know, given that we have a relatively small sample, there's also a lot of variation in our outcome, variable crime. You know, there's some statistical uncertainty around that estimate, but it's almost surely larger than, say, 30 percent. It could potentially be as large as something like 80 percent. So these are qualitatively large results, regardless of sort of where within the confidence interval you think we you know, we sort of lie.

 

Aaron [00:29:47] There is an important qualifier here that I want to mention. So, you know, you probably have some pretty astute listeners. And I'm sure many of those listeners are wondering about spillovers to adjacent areas. So we put lights into these communities. Doesn't crime just shift around the corner? And, you know, that's a very valid concern. So for a number of reasons, I think that story's less plausible than it might sound. You know, adjacent areas are qualitatively quite different from the residential communities that receive the lights. So it's not clear how substitutable something around the corner really is. But absolutely, this is a valid concern. And, you know, it really colored how we chose to analyze the data. And so we wanted to be very, very conservative about how we fold in potential spillovers to adjacent areas. And so what we decided to do is to draw a two block radius around these treated communities and consider that entire area to be treated. So the campus itself, plus this two block radius, even though the lighting only was added to these campuses, and when you consider the entire area, the two block radius area, even there we see something like a 35, 40 percent reduction in crime. And that, we view, as a fairly conservative estimate of the impact of this intervention.

 

Jennifer [00:31:16] And so to dig in to potential spillovers a little bit more, I think you also look even beyond that two block radius to to see if there's spillovers beyond that. And then you also look at other times of day. So tell us tell us more about spillovers.

 

Aaron [00:31:28] Yeah. So so yeah, we- the two blocks, you know, we can show that if you change that radius, the results still hold up even fairly far outside the development. We actually focused on two blocks because a lot of folks who think about public safety in New York feel like two blocks is sort of the most reasonable choice. So that became our default. But, yeah, the results don't depend on on that choice.

 

Aaron [00:31:52] But you also mentioned spillovers to the daytime. And I definitely want to talk about this. This is really interesting, at least to me. So when I when I first began thinking about this project, I guess way back in 2014, you know, my instinct was to suppose that lighting should have no impact on daytime crimes. And I even went as far as to say, well, you know, "Hey, if there's an impact on daytime crime, that maybe feels like a failure of randomization to fully balance the treatment and control groups because lighting should only affect crime when there's otherwise no lighting." But when you read the criminology literature, you actually start to see lots of theoretical arguments, as well as anecdotal evidence that, you know, lighting really can function as a signal and that it might have these global effects on crime not just at night, but during the daytime, because it gets people to sort of feel like a community is being cared for or being protected. Now, I think that concern or that- not really concern, but that that argument is even more salient for an intervention like this. So we're talking about these really, really visible light towers that are there at night. They're also there during the day. And, you know, it's not hard to imagine that that changes the thinking of people who live in these communities, not just at night, but but but all the time. We do, in fact, see what I would call suggestive evidence — this is not anything statistically significant — that daytime crimes do go down a little bit when- as a function of the intervention, which would be consistent with maybe there being something like a demonstration effect of this intervention. But I want to be cautious about making any claim here, both because the result is not statistically significant, but also because of a basic measurement issue that I think is worth mentioning, at least in brief.

 

Aaron [00:33:58] So when when you look at crime data in New York — or any city for that matter, this is a global issue — you know, there's there's usually a field in these data that that that stamp the time that a crime complaint became known to law enforcement. And in theory, that's the time the crime occurred, but in practice, that sometimes when the crime occurred and sometimes it's when the crime was reported. A really simple example of this is imagine that I leave my house and I go to grab my bike to bike to someone else's house and my bike's not there. And it's been like a week since I last used it and I report the bike stolen. Well, what time did that happen? I have no idea. Right. And so chances are that's going to be recorded as happening when I made the report, even for something like an assault, where I should know when the crime took place. You know, it's not clear that that's always going to be documented at the right time. I personally have known people who have been the victim of a crime at night. And, you know, they went home, they got a good night's sleep, and then they went in and reported the crime in the morning. So, you know, there's always going to be a little bit of uncertainty about about the quality of these timestamps. And so it's possible that some of what we see happening in the daytime are really nighttime crimes that we're just measuring as happening during during the day.

 

Jennifer [00:35:23] Yes, those are the important caveats, but you, you do find some suggestive evidence that that crime is going down in the day. Those caveats aside, I think it is interesting to think about that result is kind of telling us about mechanisms. Right. I mean, if we think about there is this underlying question of whether lighting affects crime because it changes the potential costs and benefits of committing the crime. There's that deterrent effect you mentioned at the beginning or if it's something about the kind of signaling that the community's cared for. And that's why I think it is interesting to at least try to measure what's going on in the daytime to try to piece that together a little bit.

 

Aaron [00:35:57] Yeah, I agree. I agree.

 

Jennifer [00:35:59] OK, so a big challenge you faced here and a challenge in many studies of place-based interventions like this one is a relatively small number of observations. So you cover a lot of ground in this experiment, but because you're looking at essentially neighborhoods instead of people, you're limited in terms of your sample size. And you and your coauthors are extremely transparent about this in the paper. And you do your best to analyze the data in a way that limits researcher discretion and so on. So I'd love to just talk more about that aspect of the paper. What did you do in the study that you might not have done if you'd had a larger sample?

 

Aaron [00:36:32] Yeah, thanks for asking about this. This is one of the issues that kept me up a lot of nights when I was thinking about, you know, how to design the study, how to analyze the data. I definitely have more gray hair now than when this study started. Although I guess that's a confounded comparison, I'm also five years older. But but no, really, this this was this was a real challenge. So on the one hand, we have an enormous sample, right. We have approximately 60,000 people living in the housing communities we studied. And our sample of 39 places is really very large relative to a sample size of one or two, which is what you see in the prior literature. So, you know, that's me patting myself on the back. But on the other hand, we're still talking about 39 observations. This is not exactly big data.

 

Aaron [00:37:20] And compounding that, we also have a highly variable outcome, a variable of crime. So crime varies quite a bit among these housing communities. It varies a lot over time, even within a given housing community. We also have a treatment which itself varies because some of these communities received a lot of lighting. Others have received only a small amount of lighting. So, you know, how do we how do we handle all this? So so normally, right, in an experiment, it's really, really simple to analyze the data. Right. The hard part is, is rolling out the experiment. And the econometrics is is where you can kick back and relax. But, you know, so you really just compare treatment and control means. Right. Did crime change more in the treatment area than the control area on average? You really don't have to make assumptions. You know that in a large sample, randomization works like gangbusters. You don't need to control for anything. It's one of the many virtues of experimentation. It's just hard for researchers to manipulate the results.

 

Aaron [00:38:21] Now, in this study, we didn't have that luxury. You know, we needed to control for covariates. So so for other neighborhood characteristics that might have impacted crime and could have by chance, been associated with a random roll out of lighting just because we're talking about a small sample. We also needed to control for these other characteristics, for statistical power reasons to increase certainty around the resulting estimates. And the problem is, once you start controlling for variables in a multiv- in a multivariate framework, you have to make all sorts of choices. Right. Do I control for X? Do I control for Y? Should I control for the natural log of X or the natural log of Y? Right. It's very easy for a researcher to cherry pick a preferred model and get just about any answer that you want to get. And even if you're, you know, being really honest. Right. And trying to do do good, it's easy to like talk yourself into a model being a preferred model because it accords with your priors. And, you know, this is exactly the situation that we wanted to avoid. It's bad for this given paper. It's bad for science as a whole.

 

Aaron [00:39:34] And so we wanted to find a way to tie our hands behind our backs and make as few choices as possible. And so one of the main tools that we use in our analysis is a machine learning classifier called the Lasso, which is pioneered really in statistics and computer science. It's not something that I learned about in grad school, being trained to be a social scientist, but it's an approach that's becoming more and more common. So I won't get into the computational details. But essentially, this is an algorithm that uses just a smart method to pick covariates that genuinely predict the outcome of interest, which in our case is is crime. And so we're going to let the algorithm pick a preferred model. The algorithm will decide whether it's the log of population or population and levels that should be in the model. My coauthors my coauthors and I are not going to make that that decision. And that's one of the virtues of this approach. Now, we also reestimated our model thousands of times, each time controlling for different stuff. And we can show that no matter how you slice it, we're seeing large reductions in crime. So the result really does seem to be quite robust.

 

Jennifer [00:40:51] And you do additional robustness checks on top of those, as you do in a paper like this. So we already talked about looking at different radii of spillovers across neighborhoods. You also look at whether outliers are driving the effects. So whether there's certain public housing projects that had more of a change than others, you look at how much lighting dosage matters. So walk us through those exercises and what additional information they provide.

 

Aaron [00:41:16] Yeah so any time you're working with a really small sample, say, of size 39, you know, there's always going to be a concern that there's like one or two highly leveraged observations. Like when you drop one of these developments, the treatment effect goes away, something like that. Right. When you're working with a million observations, you worry less about that problem. So for us, you know, we wanted to just, you know, dot our I's and cross our T's. We make sure that this is not being driven by one or two sort of outlier developments. We make sure that it's not just a couple of developments that received a very, very high dosage of lighting that are leading to these effects. It's nothing like that. This is a pretty broad-based treatment effect. You know, we we think about the size of the displacement radius, which I which I'd mentioned.

 

Aaron [00:42:07] But I think really the the really critical robustness check here is what we do to bring back the control group, the randomized control group. And let me let me sort of talk that through, because the big threat to, you know, getting the wrong answer here, the big threat for us is that the randomization, just for whatever reason, didn't work. And, you know, how can we figure out if randomization worked as it should have in this context? Did our coin flips do do what we wanted them to do? So one thing we can do, right, is we can check and say, you know, we have this information about the characteristics of these communities — and we flipped a coin, some of these places got a small dosage of lighting, other places got a lot of lighting. We would want those places to look alike with respect to their pre-intervention characteristics, and they do. But the problem is that's an incomplete test. We can look at the characteristics that we actually have data on, but we can't look directly at all of these nuanced things that make a community what they are. Right. There's many things about your community, I'm sure, that you would find salient, but would be difficult to measure in in an administrative dataset.

 

Aaron [00:43:23] So the way that we can do this is we can remember that when we randomized the lighting in the first place, we randomized in pairs. So I took the two housing developments that had the most crime in our sample. I flipped a coin. One of them got this lighting intervention, the other one didn't. The place that got the intervention was then randomized a dosage of lighting. Now its paired control place is randomized the same dosage of lighting, but that lighting is never actually received because this place is in the control group. And so what we would want to be the case if this lighting- if this lighting intervention really is effective, is for the dosage of lighting to predict crime in the treatment group, but it should not predict crime in the control group. And that's a test for whether we're balanced not only on variables that we do observe, but for variables that we don't directly observe. And we, in fact, see that that's the case, that the intervention predicts crime in the treatment group. It does not predict crime in the control group.

 

Jennifer [00:44:29] Great. OK, so let's think bigger picture here. So this paper is pretty new and we've already talked about the broader literature on lighting, but the paper is also part of a larger literature on place-based crime reduction policies. So how do you think of the results of this study fitting in with what we know about the effects of targeting a place rather than specific people when we're trying to reduce crime?

 

Aaron [00:44:52] Yeah, great. So I love the bigger picture, you know, coming from a policy school, you know, it's just so important to to talk more about that. So so the idea of place-based crime reduction is not a new idea. There is some urban planning theory around this which dates back to the 1960s. Often it goes by the acronym CPTED, which stands for Crime Prevention Through Environmental Design. This is an idea where urban planners seek to leverage design principles and say, well, how can we design public spaces that are less conducive to offending? And that's not just about lighting. It's about a lot of things like visibility and, you know, other street conditions, you know. So within criminology, anyone who's interested in this topic should should definitely read the many, many foundational papers written by Ron Clarke, who's who's at Rutgers. And a number of his his collaborators have really paved the way for thinking about these issues. What we've seen in the last decade is that there's really been a proliferation of some high quality research around some of these environmental design topics and crime, including some experiments, which has been which has been really great.

 

Aaron [00:46:11] So my colleague at Penn, John MacDonald, has been really focal in this research, Charlie Branas — who's actually in the School of Public Health at Columbia, which speaks to the interdisciplinary nature of some of this research — they've randomized things like fixing up vacant buildings, you know, which are eyesores and communities and some people think create the conditions that are conducive to crime, greening vacant lots. Anthony Braga at Northeastern has also done some really important work in this area in the context of what's commonly called Problem-Oriented Policing. So this is an idea where cops will think about design-based challenges to reducing crime and try to come up with strategies to to control crime through maybe nontraditional means.

 

Aaron [00:47:02] And I think the big takeaway from a lot of this work is that there's a portion of, you know, really this canonical theory of Broken Windows Policing, which changed, you know, policing in a really big way in the United States, which has a great deal of validity. The idea that disorder can- disorder can beget more disorder, and that disorder reduction can be a really, really critical piece of the crime reduction puzzle. And you know, that's not to say that there's much evidence at all in favor of the importance of making tons and tons of misdemeanor arrests. Right. It's- but but I think the idea that caring for a community, creating the conditions for prosocial community members to take control of their communities in a positive way can really, really have a big impact.

 

Jennifer [00:47:55] So putting it all together, the results of this study and the broader literature — it's both of them that we've talked about. What are the policy implications of this work? What have you told New York City? Because presumably you've had many conversations with them about the results here. And what would you tell policymakers in other places?

 

Aaron [00:48:12] Yeah, so what have we learned here? So I want to be very cautious. So on the one hand, I am enormously encouraged by these results. These are qualitatively very large reductions in crime in response to the intervention we studied. This intervention costs maybe $75 per resident, you know, which which feels like it could be viable moving forward for a number of cities. I definitely feel strongly that this should be in the conversation as other cities think about what they can do to confront challenges in the communities that that have the greatest number of challenges.

 

Aaron [00:48:48] That being said, there's still a bunch of caveats here, and I want to make sure that I'm not overselling what we can learn from this. So first, when you think about residential communities, most crimes don't happen outdoors and at night. In fact, usually the most common type of crime in these communities is something that happens behind closed doors, that street lighting probably won't have a first-order impact on and that cops are struggling to deal with as well. And this is something like domestic violence. So overall, when we think about the crime reduction, we observe as a as a result of this lighting intervention, this translates into maybe a 5 percent reduction in overall community crime, this is not a panacea. We haven't caused crime to be a thing of the past by putting in some some streetlights. This is affecting, I think, some very visible and very important serious crimes in these communities. But there's there's other interventions that that are that are going to be needed alongside this.

 

Aaron [00:49:50] Second, we're talking about a very specific sort of tactical crime intervention. Right. I think this was rolled out in a very smart way by New York City. But it also creates some some challenges with respect to external validity. What can other cities learn from this? So the city said, well, what can we do to go into some of these higher crime communities and and reduce crime in a light touch sort of way? Right. And, you know, again, and this is something that we've alluded to in our discussion previously. But, you know, part of what's probably going on here is this isn't just lighting. There's these demonstration effects. Right. That's got to be part of this mechanism- or likely is part of this mechanism. And, you know, not every city has public housing communities that look like New York's. So, you know, I think as social planners in other cities think about this, you know, it's important to think about the details and and what would make sense in a different context. Also, you know, we're only looking at short term impacts here. You know, we need to understand what happens over the longer term as well.

 

Aaron [00:50:58] And then finally, there's just the general refrain. This is one study in one city. You know, I think it's easy to have the instinct to say, you know, hey, this is the first lights experiment. It worked great. Let's let's just like scale up lights everywhere. But, you know, we really want to be cautious here because the opportunity costs are high. You know, these are communities where the most vulnerable citizens live. And getting policy right is really important. And so I think it makes sense to think about scaling up lighting more and studying this more. But but carefully, slowly, cautiously. Right.

 

Jennifer [00:51:37] Do you have any way of knowing if the residents in these communities liked having the lights there?

 

Aaron [00:51:42] Yeah, I can talk about this in an anecdotal sort of way. So the the Mayor's Office of Criminal Justice in New York City surveyed community residents during the time these lights were in the field and asked people, hey, what do you think of the street lights? 80 percent of the people who responded to the survey said the lights are great, we really like having these here. Many people wanted the lights to remain. But this is not a random sample of community residents. In particular, older women are very overrepresented among the people who answered the survey and younger men are underrepresented. So I think what I would say is the people who are taking a strong interest in their communities and who are in contact with policymakers seem to to really want the lights. By the way, there's a lot of prior research out there in criminology that suggests that people like streetlights. But, you know, surely there are some people in these communities that, you know, might have a different reaction as well.

 

Jennifer [00:52:47] Yeah. And is New York going to keep the lights? What's the verdict?

 

Aaron [00:52:52] I don't know. You know, so what I can say is that there's a number of public housing communities that have received additional permanent streetlights, not necessarily as a result of this research. But this is something that was happening alongside this research that the extent to which that's going to continue to scale throughout public housing, you know, I don't know for for sure. I guess we'll see in the coming months or possibly years.

 

Jennifer [00:53:24] We'll have to keep tabs on that.

 

Aaron [00:53:26] Yeah.

 

Jennifer [00:53:26] So what's the research frontier here? What are the next big questions in this area that still need to be answered?

 

Aaron [00:53:32] Yeah, there's there's so much more to know. I feel both really excited and also really, really overwhelmed to stand amongst others who are working in this area right now. I think this is going to produce some very, very exciting results over the next decade. But it's amazing how much more we need to know. So first, I think we need more replication. You know, sometimes I worry that as researchers were rewarded so much for being the first and so little for being the second or the third of the fourth. But it's so critical to, you know, have more interventions like this, but also interventions that are designed a little bit differently so we can start to know what works and for whom and in what context. And I would love to be a part of that if there's other cities that are considering doing this and want to learn more about how to make the most available resources.

 

Aaron [00:54:27] In other communities, how should a lighting intervention be designed? We don't know a whole lot about that. To what extent will putting in more normal street lights work, right, where you don't have maybe as salient a demonstration effect? Another thing that we really need to know more about is where is lighting most effective? This is something that the folks in New York asked my colleagues and I when we had some preliminary findings to show them. And we really couldn't speak much to it because we have such a small sample. You know, where should we start if we're if we're going to start rolling out lights, you know, where should we put those lights? So I think this is just a subset of what we want to learn about. In addition, obviously, to understanding more about the the dominant mechanisms, do lights change the amount of human activity you see outside, et cetera, et cetera.

 

Jennifer [00:55:16] So if any listeners out there, if their city wants to experiment with streetlights in their community, they should call Professor Chalfin.

 

Aaron [00:55:23]  Call me. I'll I'll definitely talk to you. I'd love to do more of this work.

 

Jennifer [00:55:27] Excellent. Well, my guest today has been Aaron Chalfin from the University of Pennsylvania. Aaron, thanks so much for doing this.

 

Aaron [00:55:34] It's a pleasure.

 

Jennifer [00:55:40] You can find links to all the research we discussed today on our website, probablecausation.com. You can also subscribe to the show there or wherever you get your podcasts to make sure you don't miss a single episode. Big thanks to Emergent Ventures for supporting the show. And thanks also to our Patreon subscribers. This show is listener supported, so if you enjoy the podcast, then please consider contributing via Patreon. You can find a link on our website. Our sound engineer is Caroline Hockenbury with production assistance from Elizabeth Pancotti. Our music is by Werner, and our logo is designed by Carrie Throckmorton. Thanks for listening and I'll talk to you in two weeks.