Kevin Wilson

Kevin Wilson is the Associate Director and Head of Data Science at the Policy Lab at Brown University.

Date: November 22, 2022

A transcript of this episode is available here.

Episode Details:

In this episode, we discuss Dr. Wilson's work on how to handle non-emergency medical calls to 911:

“Nurses in the 911 Loop Improve Care: A Randomized Controlled Trial” by Chrysanthi Hatzimasoura, Rebecca A. Johnson, Kevin H. Wilson, Robert P. Holman, Ryan T. Moore, and David Yokum.

Minor correction: During this conversation, Kevin mentioned that nurses could book primary care appointments for callers. This is not quite accurate. Instead, nurses may refer callers to Federally Qualified Health Centers (FQHCs) with walk-in availability for primary care and book the caller a ride to the FQHC.

OTHER RESEARCH WE DISCUSS IN THIS EPISODE:

• “A Community Response Approach to Mental Health and Substance Abuse Crises Reduced Crime” by Thomas S. Dee and Jaymes Pyne.

• Episode 80 of Probable Causation: Thomas Dee.

• Episode 77 of Probable Causation: Elizabeth Linos.

• “Ten Solutions for Emergency Department Crowding” by Robert W. Derlet and John R. Richards.

• “A Randomized Control Trial Evaluating the Effects of Police Body-worn Cameras” by David Yokum, Anita Ravishankar, and Alexander Coppock.

• Episode 78 of Probable Causation: Sandip Sukhtankar.

• Episode 75 of Probable Causation: Panka Bencsik.

• Episode 67 of Probable Causation: David Phillips.

TRANSCRIPT OF THIS EPISODE:

Jennifer [00:00:08] Hello and welcome to Probable Causation, a show about law, economics and crime. I'm your host, Jennifer Doleac of Texas A&M University, where I'm an economics professor and the director of the Justice Tech Lab. My guest this week is Kevin Wilson. Kevin is the associate director and head of data science at the policy lab at Brown University. Kevin, welcome to the show.

 

Kevin [00:00:28] Thanks, Jen. Good to be here.

 

Jennifer [00:00:30] Today, we're going to talk about your research on how to handle nonemergency medical calls to 911, but before we get into that, could you tell us about your research expertize and how you became interested in this topic?

 

Kevin [00:00:42] Yeah, so my career has been a little roundabout. I originally got a Ph.D. in mathematics and number theory, so very esoteric, but starting around 2016, I really wanted to kind of work in Applied Policy. I realized that there is a lot of work to be done in helping governments use data to improve how they actually build policies and programs and how they evaluate them and I got really lucky. I had moved to the D.C. area for a family reasons, and I met a cool group of folks called The Lab at D.C. They were a new group in the mayor's office who were doing exactly that.

 

Kevin [00:01:17] They were really trying to help the government of the District of Columbia, across all its agencies, really build and evaluate programs using data and evidence and sometimes machine learning models and sometimes good old fashioned RCTs and sometimes just good old fashioned data mining and this was really one of the first projects that I was kind of handed I was lucky to be handed, as in helping to build out that data science group. And really since then, a lot of my work has kind of come to focus around less a particular topic, say criminal justice or emergency services or things like that, but really, that question of how do we run RCTs in government, how do we help governments actually use the data they have in ways that can help them evaluate programs? So if I have a research expertize, it's really that it's really that how do we actually help governments build these programs better using the data that they have at their disposal.

 

Jennifer [00:02:14] All the cool data they have at their disposal that most people don't usually get to see. Yeah, and I guess I should say full disclosure, we met when I was working part time with around the D.C. back when I was the University of Virginia. I think in those early days, yeah. And so I got, I think a glimpse, an early glimpse of this project was just getting off the ground.

 

Kevin [00:02:33] You might even be in the acknowledgments. I don't remember it.

 

Jennifer [00:02:36] Maybe I'll have to check. Yes. So very, very happy to see see this paper come out and get to talk about it. So your paper is titled "Nurses in the 911 Loop Improve Care: A Randomized Controlled Trial." It's coauthored with Chrysanthi Hatzimasoura, Rebecca Johnson, Robert Holman, Ryan Moore and David Yokum. And in it, you describe the results of an ambitious experiment in Washington, D.C., where a subset of emergency calls was diverted to a nurse triage line. So let's back up a little bit. What was the problem that DC was trying to solve?

 

Kevin [00:03:09] Yeah. So probably all of your listeners are familiar with the notion of an emergency help line of some kind, probably from when they were very young child, their kindergarten teacher, their first grade teacher taught them that if they have an emergency, they should call 911 or 999 if near the U.K. or some three digit number. And what you were taught at that time is that if you do that, there's a wonderful person on the other end of the line, will pick up the phone and try to help you figure out what your emergency is and connect you with services that will help you solve your problem right then whatever that emergency happens to be. Really great concept really sounds awesome, but the second thing they say to you after they say, if you have an emergency, you should call 911 is if you don't have an emergency, you should not call nine. And it's kind of left as an exercise to the kindergartner what an emergency is.

 

Kevin [00:04:00] And this has become problematic because it turns out the reason they tell you that is kind of twofold. One is that there is is that 911 services and emergency services more generally are extraordinarily expensive to maintain. If you look at, for instance, in the D.C. where this experiment occurred, fire and emergency medical services, which just runs fire trucks and ambulances and the infrastructure necessary for that cost about $425 per resident per year to maintain the services that's just the fire trucks and the ambulances, not the police, not the 911 call infrastructure, nothing along that line. And so these are very expensive services to maintain and so you can't really, you know, expand very easily without it costs a lot of money.

 

Kevin [00:04:45] And secondly, the other reason you're told not to call an emergency is that these are emergency services and they're equipped to help you with emergencies. They're not equipped to help you solve your kind of day to day problems, right. If you call and you ask, how do I get access to my SNAP benefits or my TANF benefits or my food stamps? They can't answer that question or they're not equipped to answer anything in government except.

 

Kevin [00:05:06] There's these emergency situations and there's often a better place for you to get help. In particular, especially primary care facilities are often in the medical space, the place which is better for a lot of different problems you might have. And so in D.C., what happened was that in about 2015 or so, D.C. started to notice that they were reaching the capacity limits of their 911 system. It actually started being that there were times a day where if you called 911 in DC, there might not be an ambulance available for you. If you were having a heart attack, you might be told there's literally no ambulance for you. You have to wait. And in these emergency cases, a heart attack, a stroke, those are places where seconds, minutes really matter in how well you'll actually end up progressing afterwards.

 

Kevin [00:05:55] So they took kind of drastic measures. First off, they went to the D.C. Council and they're like, we need cash money and we need to immediately expand our 911 services and the D.C. Council agreed that having no ambulances at certain times of day was bad, and so they immediately contracted a large number of extra ambulances to the tune of 1 to $1.5 million per ambulance per year. So not cheap. Right. And then on the other hand, they were like, but what's happening here? They kind of looked at their numbers and they were like, well, on the one hand, D.C. is if we look at how many 911 calls we have. DC got about 1.8, nine on one calls per resident per year across the whole United States that number is 0.7. So they have an extraordinarily high number of nine on one calls.

 

Kevin [00:06:40] And they also noticed they had about one ambulance transport for every four residents per year, which is also very, very high compared to across the United States and remember, there are kind of two sides, this emergency coin, right. On the one hand, there is the people are over utilizing 911 and the question is why? And so they did a lot of research as to why that is and go into that later. And on the other hand, if you're over utilizing emergency services, you're probably underutilized in primary care services, which are probably going to be much more helpful for the types of problems you're going to have. And they kind of viewed that as two sides of the same coin. Right, is over utilization of emergency services and underutilization of primary care services and so that problem was the one they were trying to solve and go into, you know, what the history is of why that may have come about and whatnot as we go through the interview.

 

Jennifer [00:07:34] And just to clarify, so when you're saying under utilizing primary care services, I think what you're thinking of here is not just, you know, you're not getting the preventive care that would have, you know, avoided the heart attack, but that people are calling 911 because they have a cold or something like that where they should just be going to the doctor instead of calling everyone.

 

Kevin [00:07:53] Both of these things are true. So there is the primary care and urgent care. I should also really toss in there something that is, you know, something that is non emergent. And again, we've left it as an exercise, the kindergartner or honestly the 35 year old adult. That is me to figure out what an emergency is. And yeah, so exactly that is it. Sometimes it might be a cold and as we can get into this will actually they also found things like, well, a lot of people don't have transportation and so maybe you might call 911 not because you have an emergency, but because you have an issue. You have to go to the hospital and have no other way to get there. Right.

 

Kevin [00:08:28] So I'm happy to dig into like some of the issues they kind of in there, kind of qualitative and quantitative analysis. What they kind of found were those drivers of 911 utilization. But yeah, but in terms of that PCP usage, that primary care physician usage, I really do mean that broad spectrum of things that you could use to do both preventively and also, you know, going to get stitches or going to go deal with a cold that you might otherwise use.

 

Jennifer [00:08:53] Great. Okay. And then so how does D.C.'s nurse triage line work? What is the program they set up?

 

Kevin [00:09:00] Yeah. So to understand what they did, it helps to understand how a typical 911 call works. So you have an emergency just like they taught you in kindergarten. You call 911 and a call taker picks up the phone and that call taker says, "Hello, this is 911. What's your emergency?" And they proceed to work through kind of a script. It's kind of a big decision tree. And, you know, back before they were computers, these were like thousand page books where it was, like, kind of like the giant choose your own adventure novel where, you know, they begin with what's your emergency? And if you say X, turn to page 200. If you say Y, turn to page 500. And they work their way through this giant decision tree asking you questions. And eventually they actually come to an end to this decision tree and at the end of that tree, there is a diagnosis code.

 

Kevin [00:09:46] So this says, for instance, robbery in progress might be a diagnosis code, right? It's like, okay, that'd be a police service. Or maybe a diagnosis code might be fire on the 10th floor. Right. That's a fire service that you might need. Or it might be something like chest pains, which are a big indicator of heart attacks. So that might be a medical service and a medical diagnostic code. And after they diagnose you in this way, they will say thank you. Somebody is on their way and they put this code and put you call into a dispatch queue. And every diagnosis code has attached to it both a recommended complement of services to send along to the caller as well as a priority.

 

Kevin [00:10:30] So if you call in and the diagnosis code is like 20 car pile up on the 401, that set of recommended responses is like, you know, a bunch of ambulances, how many firetrucks and some police officers, right. There's a whole complement there to say, but if you call and you say, I have a cat in my tree. Right. Maybe the recommendation is something like a single fire truck. And similarly, there's a priority, right? That 20 car pile up on the floor of one is a super high priority. Chest pain, super high priority. You're going to bump out everybody else's call before you, that cat in that tree you're going to be waiting a while because there is many other people who, you know, need the services before your cat needs to get down the tree. So the dispatchers handle the translation from diagnostic code to actual services provided, and that's 911 in a nutshell.

 

Kevin [00:11:19] What DC did was they said, well, what if we insert a step, right. So between the call taker and the dispatch or certain diagnostic codes and for certain eligibility criteria, they would actually potentially introduce you to a nurse who would basically do further triage with you and recommend a course of care.

 

Kevin [00:11:44] So if, for instance, you got to a diagnosis code that said generalized pain, right? This is not something that really you can do anything about, but a nurse might be able to help you figure out some things to do. They would introduce you to that nurse who would walk you through some further triage, and that nurse could do one of four things. They could either say, actually, this is in fact, an emergency. I'm going to send an ambulance to you. They could recommend that you go to an urgent care provider, so these are, you know, City M.D. in New York or places like this where it's you. You can go to the corner and just kind of wait for some stitches or a strep throat test or something like this and not only that, but they recommend that you they'd arrange a taxi for you which DC would bill your insurance for or eat if they if you had if you didn't have insurance for so they'd actually call a taxi for if would pick you up and drive you there.

 

Kevin [00:12:36] Alternatively three they recommend you see your actual primary care physician. If you had one or if you didn't have one, they would offer to schedule you an appointment with the primary care physician at a federally qualified health care clinic in your neighborhood. So they actually had access to the scheduling systems for the so called FQHCs and would actually make you an appointment within 24 or 48 hours or so. And finally, they could just recommend self care. So that's, you know, take two Advil and call me in the morning type thing. Right. It's like you're probably fine, you know, no, this isn't an emergency. Go back to your your daily life.

 

Kevin [00:13:10] So those are the four things they could do. And again, it's like kind of between that call taker and that dispatch, if the diagnosis turns out to be relative, you know, clearly low acuity, clearly low on emergent, you could get introduced to this nurse who would do this, this process with you and do one of those for outcomes.

 

Jennifer [00:13:27] And so what kinds of symptoms or situations would get you rooted in into this this nurse triage queue so we've got the cold, the generalized pain. Do you have other examples?

 

Kevin [00:13:41] Some examples include things like sprained ankles, kind of things that aren't like things you kind of walk off. In the end, there were a total of 50 codes in all and I can direct you to our pre- analysis plan if you'd like if all 50 that the medical team looked at and actually put together those kind of like very low acuity things and sprained ankle, generalized pain things that there's not really anything that anybody can do about anyway and those are the types of things that get you routed into the nurse directly.

 

Jennifer [00:14:12] Okay. And have other places, tried programs like this. What did we known before your study about whether this is a good idea or not?

 

Kevin [00:14:22] Always a good question. So I am still unaware. I think D.C. is still the only city 911 system that I'm aware of could be not aware that has actually put nurses directly into the 911 booth there are kind of two similar types of programs that we actually have started to see a lot of evidence about. One are so-called nurse helplines. So these have been around for a while. If you check your insurance card probably on the back, there might be probably like a 1-800 number nowadays in the U.S. and if you're in Canada, like, OHIP in Ontario has one of these. Basically, if you call this number, you'll get a nurse and you can ask them, you know, hey, I've got, like generalized pain, what do I do about it? And they can they will literally walk you through this process and they will give you one of the same four outcomes, they'll say go to an urgent care clinic, go to your primary care physician, it's fine stay self care at home or call 911 right now.

 

Kevin [00:15:18] Right. And these have existed for a long time. Indeed. Actually, the vendor for the nurse triage line in D.C. actually runs a lot of nurse help lines and it's actually a carve out of their nurse help line practice to run this nurse triage line. So these existed for a long time. They became popular especially in the mid mid to mid 2000 and early 2010s. There's not a super large amount of like what we would nowadays consider causal evidence on them, but there's a ton of satisfaction surveys people really like the ability to call somebody, it turns out about like, is this an emergency? What do I do in this case? As a father of a five month old, I really have enjoyed having nurse traige helplines around to be like, is this a problem?

 

Jennifer [00:16:03] Uh huh.

 

Kevin [00:16:03] And so we really like them. And there are some anecdotal there's some anecdotal that they do reduce 911 calls in particular. DC was modeling a lot of this off of Reno, Nevada, which actually has a citywide nurse helpline. They instituted it around 2013 and in pre post evidence they looked at before and after they instituted the the helpline, they maintained that over about three years they reduced emergency calls by about 4500 or so over the course of those three years.

 

Kevin [00:16:34] So, you know, the kind of correlational evidence, not super rigorous causal evidence that we would have nowadays. The other side of this is that there's a kind of a burgeoning of what I would call forth services inside of emergency to kind of 911 systems. We think of emergency services being police, fire and EMS and there's kind of an emergent collection of sort of what I would call fourth services. So a big one of these are around police and mental health services. So there's been a push recently for if you call 911 and you say there's somebody in mental or behavioral distress, instead of sending the cops, you might send a set of social workers who are trained specifically in mental health provision. And here there's actually a really great recent paper by Dee and Pyne that just came out in Science Advances, looking at a version of this in Denver, showing that this kind of forth service provision is actually quite impactful at helping reduce a lot of the problems that exist in these kind of document systems and that are associated with just sending police to answer these kinds of calls. So those are kind of two types of services that are kind of related.

 

Kevin [00:17:41] One is this like the classic nurse helpline and essentially the services that Nurse Help Helpline in the 911 system directly. So it's kind of the inversion of what Reno did right there is saying they say call their helpline, then call 911. We say call 911 and we'll get you to help line if you need it and this fourth service idea, that's where the research has been before.

 

Jennifer [00:18:01] Got it. And then so as we're thinking through kind of potential mechanisms or how this can play out, so you're kind of advising people that there are other options for them than just getting an ambulance. You might be getting them a taxi instead of send the ambulance. That's cheaper. So those are sort of the.

 

Kevin [00:18:19] And faster, very critically.

 

Jennifer [00:18:21] And faster. Yeah. And so those are sort of the good potential outcomes. Are there ways that this might not go as planned? And I assume the answer is yes. So what are the ways that this might not go as planned?

 

Kevin [00:18:33] Oh, yeah. Well, I mean I mean, these are emergency services, right? And there's always the possibility of getting it wrong and the worst possible outcome is death. Right. Like, you know, people having heart attacks and getting mismanaged care and things like this. We didn't really anticipate too much there because, you know, these are very low acuity calls to begin with. You were only eligible if you had something the call taker had already determined was low acuity. So it was unlikely that you were going to get an ambulance soon. So our hope was that by getting a nurse, if for some reason the call taker had made a mistake, this is kind of a second check as well.

 

Kevin [00:19:11] So we were kind of hopeful that we wouldn't hit these issues, but there are it is potentially true that, like, you know, the call taker, maybe the call taker is would be like, oh, well, the nurses will take care of it. I don't need to dig into this so much and I'll just ship them over there or maybe the nurses take, you know, the call takers are like a very regimented, like their job is to get you your care as fast as possible. The nurses are oriented toward, you know, helping you triage and understand something and so they're a little slower. Right. So maybe if that mistake was made, you know, you could imagine the nurses taking a little longer for you to get the appropriate care. We don't have any evidence of that actually occurring in practice, but those were the types of worries that we had.

 

Kevin [00:19:51] And secondly, you know, that's kind of in the milieu of the, you know, as people think about things. The other one we had is that we were deeply worried that call takers would get distracted. You had Elizabeth Linos on a recent episode of Probable Causation, talking about 911 call dispatchers and how stressful their jobs are and how easy it is for them to get burned out and these jobs are very, very hard. And like the cardinal rule of this RCT was like, we cannot make the call dispatchers life harder because there's a regimented thing. It's like, this is how it goes. It's very stressful job. We can't have them like flipping coins to determine whether to send them A or B, right. We can't have them like looking over their shoulder to see if the nurses are available. We can't have them shouting like, hey, like, you know, is can I send this call over? Right. It be very clear how everything operated and we preferably wanted to have as much of the kind of decision making built into their software as possible to avoid basically making their lives even more stressful.

 

Jennifer [00:20:57] Yeah, that's really interesting. So those are reasons. Those are potential reasons that this could make things worse putting this nurse crash line in could make things worse. There's also the possibility that you could just find nothing right has makes no difference. I think you mentioned in the paper that even if you send an ambulance out, they're not required to actually take you to the E.R. so sometimes the ambulance drivers or the the EMS folks might realize at that point this is not an emergency is there are other stuff like that in the mix?

 

Kevin [00:21:27] Oh, yes. Most certainly. You know, one of our worries was that the nurses who might actually be a little in terms of like actually measuring an effect of this. One of the things we really worried about was that nurses were coming from that nurse helpline infrastructure and we were telling them that they're in the 911 loop, the emergency loop. You're actually worried that nurses might be a little trigger happy, right that they would immediately send people ambulances at the at the first sight of anything as opposed to suggesting self care or as opposed to suggesting urgent care when that would be more appropriate. And indeed, early on, we actually saw some of that happening and we actually did some there was some retraining that happened and some rejiggering exactly how instructions were given and whatnot and so over time, you actually see the proportion of clinical referrals as opposed to ambulance referrals kind of going up through the nurse triage line.

 

Kevin [00:22:23] And so it is possible that nurses were would be worse at this than call takers in some sense. Right. The call takers that very regimented people have been doing this for years. Nurses aren't necessarily trained in that like specific 911 system set up. They might be great emergency nurses, but they may not be trained in that 911 thing very specifically. So that was another worry.

 

Jennifer [00:22:44] Yeah. Okay. So as DC starts thinking about implementing this and your team started thinking about how one would study it, what were the challenges that were kind of the biggest in your mind to be able to measure the causal effect? Were most of the your concerns about data, were they mostly about identification and having a good experiment with it? Was it both of those things? What were the hurdles here?

 

Kevin [00:23:08] Well, so other program is a big problem with these programs in general, is that you typically have to roll them out across the whole jurisdiction. Right. You know, create a nurse helpline for their whole city, insurance companies tend to create nurse help lines across all insurance plans. Right. So it's really hard to target them because you have to roll them out for everybody. So one of our big questions was we really wanted this to be randomized at the moment of the call. So to do that required a lot of lift from that software. So I actually spent a lot of time working with our vendors on like how exactly we should design the buttons, how exactly we should design, you know, how this gets recorded in the systems.

 

Kevin [00:23:52] And on that note, very specifically, one of our big concerns is how do we track people through the system? We had a great plan. You'll notice if you read our paper that there's supplementary appendix A is many pages long showing us exactly how silly our plan was in practice. And it turns out that like call takers are really interested in getting you service as fast as possible, and they don't care if they mispell your name, they don't care if they missed getting your insurance card number exactly right. Like not their problem. Right. They want to get you to that service as fast as possible. And so being able to track people through the system was one of our big concern and turned out to be one of our primary hurdles once we had all the data in hand was like tracking people through the system.

 

Kevin [00:24:37] And finally, just in general, this is all very sensitive data across like a huge number of agencies and so just, you know, everything was teetering on like, you know, will they sign the DUA, will they sign the MOU? And, you know, will we have a place to put the data that's secure? At some point I actually had to go build essentially a data lake for the lab at DC so that we could put all the data in one place and actually look at it in a way that our privacy and security officers were okay with and so that was actually like literally getting physical access to the data that was another big important question, so that tracking people through getting the data and making sure that we could randomize at that call level as opposed to accidentally randomizing it some other level that we were not intending to do.

 

Jennifer [00:25:25] So a different experiment you weren't expecting?

 

Kevin [00:25:28] Yes. A thing that we have, it's a thing that has happened to me before.

 

Jennifer [00:25:35] Story for another podcast.

 

Kevin [00:25:36] Yeah.

 

Jennifer [00:25:39] Okay. So in this pilot stage, this nurse triage program was implemented as a randomized controlled trial, as you say, in order to see if it worked and what the effects were going to be. So step us through what actually happened. What happened to a call that came in to 911?

 

Kevin [00:25:54] Yeah. So as I mentioned before, like our big key requirement was that we could not distract the call takers. It had to be transparent to the call takers what was happening, and it had to be very easy for them to actually participate in the program. So what we end up deciding to do is the 911  call takers perform exactly their usual diagnosis. So that whole first step I talked about, you go through the big book, you go through the big decision tree and you get that diagnosis code.

 

Kevin [00:26:21] And if the computer determined that you were eligible for the nurse's triage line based on that diagnosis code and several other criteria, you had to be calling on behalf of yourself, you could be a so-called third party caller, you had to be an adult, you couldn't be incarcerated, it had to be during certain hours and there had to be a nurse available. Right. There's like six criteria for you to actually be eligible for the nurse triage line. If the software determined you were eligible for the nurse triage, then it would flip a coin and if the coin came up heads, you're in the treatment group and the call takers would be essentially presented with three buttons. One of them would say NTL for Nurse Triage Line. One of them would say ALS for advanced life support, so that's like the real ambulance that like if you're having a heart attack, that's what they sent you. And the third would say, BLS, which is basic life support. So this might be kind of like a van filled with an EMT often fire truck actually, where if you say sprained your ankle, they might send that to you instead of the full big transport ambulance that is super expensive to maintain and there are not that many of. So they got three buttons if they came up heads, NTL, ALS, and, BLS and if it came up tails, the NTL button was grayed out. They just weren't allowed to press it, which is also what would happen if you weren't eligible for the nurse triathlon to begin with. So that's where the randomization happened is right there at that moment when once the dispatch code was there and you were deemed eligible based upon these criteria, the software to flip a coin and show you either three buttons or two buttons.

 

Kevin [00:27:52] And once these buttons were there, the call taker had a choice of which button to hit, and they were allowed to hit whatever button they thought was most appropriate. So even if you said if your diagnosis code was a sprained ankle, they were allowed to hit, send the giant ambulance to you, they were allowed to hit ALS if they if they thought it was appropriate that they did have discretion to do that. And indeed, about 40% of our eligible pool who were eligible for the nurse traige line actually ended up having the call taker hit either ALS or BLS. The vast majority of that was BLS just to say, but there were a few that they hit ALS. So from remanining 60%, where they actually hit the NTL button if they hit the NTL button they would proceed to, they would be connected to a nurse and they would do a warm handoff. So they'd say, "hey, I'm going to introduce you to nurse such and such, and they're going to help you figure out a course of care for your problem" and the nurse would then take over from there. And as I mentioned, that nurse would do further triage and they'd give you one of those four outcomes right either calling an ambulance for you, refer you to an urgent care clinic and get you a taxi, refere you to a PCP and set up the appointment or recommending for self care. Critically, if at any time in this process the caller said I'm not up for this nurse thing gve me an ambulance they would hit the button for BLS or ALS. You can't specify which one you want as a caller, but you can specify I want somebody to come and so then the nurse or the call taker, whoever happened to have a call at that time would hit the button to do so.

 

Jennifer [00:29:28] Got it. Okay. So I guess. Yes. So we're not necessarily worried people might like hang up and call back to try to get an ambulance this time?

 

Kevin [00:29:35] We were worried about that because we were randomizing at the call level. That could potentially happen. We didn't see too much of it in practice based upon phone numbers.

 

Jennifer [00:29:44] Yeah.

 

Kevin [00:29:44] It was a thing we were worried about. You could you could potentially do that, but you could also ask.

 

Jennifer [00:29:49] Right.

 

Kevin [00:29:50] So that's why we were like, right. It's like because you could ask. We were much less worried about that as a potential outcome.

 

Jennifer [00:29:56] Right. Yeah, that's a good idea idea. So smart people were planning this experiment. Okay, so this is obviously an impressive RCT that required a lot of coordination across several agencies. You've already given us a little bit of the backstory here, but people always love to hear more. Tell us more about just who was pushing for this. How difficult was this to get different agencies to play ball? What were the the other hurdles you haven't talked about yet? I'm sure there are there more.

 

Kevin [00:30:27] Always more.

 

Jennifer [00:30:27] Whatever. Whatever other stories you've got. We'd love to hear them.

 

Kevin [00:30:31] Yeah. So this was kind of the brainchild of, as I said, is around 2015 where these problems really started to come to a head at fire and emergency medical services FEMS, if I say it, that's what that means and their chief medical officer, Bob Holman, who's an author on this paper, as well as their fire chief, Greg Beam, now retired rweally e were like, how like how can we they were kind of ones pushing this idea. You know, they really wanted to migrate people from this emergency department situation, from this emergency situation to those primary care facilities.

 

Kevin [00:31:03] That was the thing they were really hoping to do. Right, because they didn't want to deny people care. People have problems like clearly they did if they wanted to migrate people into the primary care facility. And so, sure, I can give you a little bit back story about like why this even happened in DC in the first place. It's important to remember that like DC is not a state, so many things that happen in DC or due to this fact. And in particular, from 1874 until 1973, DC was run by a board of governors that was appointed by the President, and there were three of them. One of them had to be a member of the Army Corps of Engineers and this is a that kind of gives you a hint as to like what Congress and the federal government thought of DC as right a giant piece of infrastructure.

 

Kevin [00:31:49] Right. Like this is the place where federal government happens, not necessarily a place where people live, but even during that time, the total, the maximum population of DC was 800,000 in the 1950s and nowadays about 700,000 a lot of cities lost population in that time period and people have made, you know, generational homes in DC, but that really wasn't reflected in the form of government. And, you know, if the board of Governors is appointed by the president and your trashes isn't picked up, it's really hard to get them on the phone and say like, "sir, my trash isn't picked up" like you might be able to do in Austin or in New York or a city that might have a little bit more home rule and self rule.

 

Kevin [00:32:28] And so what kind of happened during this time period is that there were a lot of areas of the city, especially places that are historically black so basically east of the Anacostia River, if you're familiar with DC geography, which is very, very which is historically very minority as a place where a lot of minorities live. And also historically, it does not have access to a lot of cash and a lot of resources at home. And these areas were very much kind of ignored in a lot of ways by this board and so there weren't as many primary care physicians that were there and there weren't as many hospitals built there. There still aren't any hospitals east of the river in DC and there barely any East Rock Creek if you kind of look at the map. And when DC finally got home rule in 1973, this became a big topic of conversation is how do we actually get our residents healthy? How do we get them the care that they need?

 

Kevin [00:33:23] And around that time, a law that a lot of people love to hate called MTALA got passed in the federal government and what they said was that emergency providers who take federal funds have to they can't turn people away even if they can't pay. And DC uninsurance rate at this time was like 25% in like the early nineties and like through the eighties. So about a quarter of DC residents was just uninsured and so the local government was like, well, that's great like the only way that we can get care for our residents is to go to the emergency room. We should encourage them to use the emergency services to do that that's the only way they can get care otherwise. And so most people believe that kind of their unofficial policy kind of started coming about in the eighties and nineties in DC that really kind of pushed people into the emergency system when they didn't have access otherwise to primary care physicians in their neighborhoods or insurance to pay for it with these high end insurance rates.

 

Kevin [00:34:21] And so this kind of created a culture of calling 911 for your colds or sprained ankle or you're honestly you're transferred to the hospital and so with that kind of background, you know, it's really it's a lot of the problems it actually turns out to have been solved in DC. Now, DC has now one of the lowest uninsurance rates in the nation post Obamacare. They were the first state equivalent. They're not a state, but their state equivalent for the purposes of the ACA to adopt the Medicaid expansion and got a 3% uninsurance rate compared to a national 8% uninsurance rate and Obamacare also rolled out a lot of funding for the federally qualified health care clinics. I mentioned earlier those FQHCs. They've been around for a long time, but they got a big boost in funding from the ACA and DC rolled out quite a lot of them in communities across what they like to say, all eight wards of the whole of the district.

 

Kevin [00:35:14] So there are now the ability to access urgent care and primary care in many of the neighborhoods that were historically underserviced. And also, almost everybody has insurance. So there's problems of the eighties and nineties have mostly been resolved in a lot of ways, at least the primary ones and so there's an opportunity. So the things that were driving that emergency utilization are no longer there. And so the question became, how can we migrate people to a better to better care plan for themselves out of the emergency system and into the primary system? And they considered many things they considered a nurse helpline, but, you know, in kind of talking with some folks, the lab at DC, especially Chrysanthi, one of the authors on this paper, they were like, it turns out behavioral change is really hard.

 

Kevin [00:36:00] And so they asked themselves, is there a better way to do it? And they kind of realized, was that, well, why don't we just instead of making people instead of doing this weird advertising thing, we're like, well, don't call 911 if you don't have an emergency, what's an emergency? You know, learn this other like ten digit number that you've never heard of before. And it's different in every jurisdiction, right? Like, let's make it really hard on you. Instead, they were like, what? You just rerouted the calls, right? Like people already calling 911 what are we just like, actively move their call over to the primary care system on, you know, if they don't need the emergency system and this turns out not to be a new idea. There's this is really great paper from 2008 from some folks at UC Davis Jillette Richards is the author is where they basically argue they they say that in the early 2000 that they were doing this in emergency rooms, that if you were in an emergency room at UC Davis and you didn't or didn't actually have an emergency, they just refer you to a clinic.

 

Kevin [00:36:55] And they said that by the time 2008 rolled around all the clinics and stopped taking referrals because of uninsurance problems and because of not having the capacity to do so. And so, you know, it's kind of like, oh, well, that's apparently it's been done before surely we can do it again in a slightly different context and so that was the genesis of this idea. And from there it was a lot of implementation from taxi vendors, getting, you know, ambulance vendors, implementing the software, building that data lake, all the MOUs, all the DUAs, getting signoff from some ungodly number of agencies.

 

Kevin [00:37:30] If you look at the acknowledgments of this paper, I think it is about a half page long, everybody who had to say yes at some point in time. So it was a big undertaking from there, but that's the real backstory for like how this idea came about, why it got into the emergency system instead of being a separate line in particular.

 

Jennifer [00:37:47] Yeah. And then were the, the powers that be onboard with this being an experiment from the beginning or did you all have to convince them that you should be testing what happened?

 

Kevin [00:37:55] So DC has been long a really great partner. So I mean, they started the lab at DC, right, knowing that they were getting into experiments and the previous big high profile experiment out of DC at the time when this had come out was the body worn camera experiment that came out in 2016 where the police department had actually randomized, which police officers would get body worn cameras in order to see whether or not they would have an effect on crime, on police, citizen interactions and so forth. And that turned out to be a big null results, which was not expected at the time. And DC actually had this really great culture of experimentation and the funny thing is that like after that big no results, the response was not from DC, "Oh, we're never doing another experiment again" and it's like, well, I'm glad you told us that. What's our next experiment? So we can actually understand what works.

 

Jennifer [00:38:49] Right the best possible response.

 

Kevin [00:38:51] Yeah, exactly. So the powers that be in DC are actually very accommodating in a lot of ways. There were some folks who were not up for the experiment and really were pushing for kind of especially ones who thought that like, well, this is clearly a good idea, we should roll it out to the whole jurisdiction, right? And we're like, Is it clearly a good idea? Nobody's done this before and so we kind of threw back and forth. We had big, high level support to make sure this got through. Bob the Doctor Holman, the coauthor here, was very much up for it. He's kind of an experimentalist at heart, so he was pushing for it, as well as especially the now city administrator at the time, Deputy Mayor for Public Safety and Justice, Kevin Donohue qe're all really kind of pushing for the experimental side, so it was good to have them on board and it was good and so we met less resistance there than we expected.

 

Jennifer [00:39:41] That's great. Okay, so let's get back to what you're doing here. So let's talk about the data as you have all these data use agreements that you had to coordinate across all these agencies, what were the different data sets that you pulled together?

 

Kevin [00:39:55] Yeah, so there's the three primary data sets we used in this paper and there hopefully will be some more for follow up paper is using a few others, but the three primary data sets in this. Paper are CAD data, this is computer aided dispatch. So that's the call takers and the dispatchers you know, they're recording when you call. What is the that diagnosis? When's the ambulance sent, etc.? The next set of data are notes from the actual EMS teams. This is in a program called Safety Pad, which is the just what we call the data if you read the paper, this is moderate to safety pad data and that's essentially the EMT notes.

 

Kevin [00:40:32] Then finally, for about half the population, we wanted to follow up after they left the emergency medical system and they about half the population who were in our sample were Medicaid recipients. So for that half the population, we actually have Medicaid claims data available from the Department of Health Care Finance. D.C. does not have an all payer claims database like many states do and so we couldn't look at non- Medicaid patients, but we could look at Medicaid outcomes six months later in particular.

 

Jennifer [00:40:59] Okay. And yeah, so we're tracking people over time. I guess I'm now thinking about the CAD data I've seen in the past for looking at crime outcomes and this is always a tricky part, right? You have an address, you might have a phone number, but you don't necessarily know who it was that calls or who the offender was or something like that. So in this case, you'd mentioned earlier, they take your name and insurance number. Was that common or was that something you all implemented or how did that work?

 

Kevin [00:41:25] Yeah, the primary link remember, you have to be a first party caller.

 

Jennifer [00:41:29] Uh huh.

 

Kevin [00:41:29] So you had to be you're not calling about a crime in progress you're calling about. I am having a heart attack. Right. Or a sprained ankle in this case. So our primarily link actually was phone number and there are, of course, many problems in phone numbers, again, as you say, but also names. So names as recorded by the call taker and then phone number were kind of our primary linking set up in the CAD data and we also knew if an ambulance got dispatched for you. We also knew if an ambulance transported you. Right. Because we had those EMS notes and there there was a very direct link. Right. The CAD data and the EMS data were directly linked. So if you actually ended up in the emergency room, we had further information about you, about your insurance numbers, because the the ambulances are supposed to unless you are, like, unable to provide them certain kinds of data, they're supposed to be able to track some of those things for their own billing purposes.

 

Jennifer [00:42:26] Hmm.

 

Kevin [00:42:26] And so if you ended up in the annual so we got a little bit more data on you and that allowed us to eventually link you to the Medicaid data. Now, if you didn't get transported by ambulance this becomes a little more complicated.

 

Jennifer [00:42:43] More complicatied.

 

Kevin [00:42:43] Yeah and there I highly recommend reading our song and dance that is Appendix A as there was much to be done and we did our best to link people using especially that phone number and that name that was provided by the first party caller, but there are potentially other choices you could make.

 

Jennifer [00:43:01] Yeah. Yep. Got it. Okay. All right. And so what are the outcome measures you're most interested in in this paper?

 

Kevin [00:43:09] Yeah. So one of the really cool things about this, actually, is that we did gather up all the policymakers ahead of time if you read our pre-analysis, that is a little short section where we presented them with all the possible outcomes we could measure and we asked them which of them they cared about, and they listed four of them as being the things that if these things moved in the direction they were supposed to, they would consider it a success. And so those were reducing ambulance transports or even reducing ambulance dispatches the distinction there being, you know, if the ambulance sent out versus the ambulance actually transporting to the hospital. Could we reduce non-emergency emergency department usage right so how many can we actually reduce the actual utilization of emergency departments and could we increase primary care physician utilization. So those four outcomes were the ones that they were that policymakers in D.C. said, if any of those moves, we will consider a success.

 

Kevin [00:44:04] There are a few others we were interested in, in particular, change in overall Medicaid expenditures over time and we also had some designs to look at medication adherence because those tend to, you know, things like not keeping up with your insulin tend to be a reason why people actually end up in the emergency room in the first place, but we it turned out some of those we didn't have the data for that we were hoping to have. So some of those are listed in the pre analysis plan, but if we couldn't actually get the data for them in the end.

 

Jennifer [00:44:32] Got it. Maybe in the future,.

 

Kevin [00:44:33] Maybe in the future.

 

Jennifer [00:44:36] All right. So let's dive into what you find. What was the effect of the nurse triage line on use of an ambulance?

 

Kevin [00:44:42] Yeah. So we had a really awesome first result here. We found that there was a 41 percentage point reduction in ambulance dispatches between the treatment in control that is 41 percentage points, which is utterly giant needless to say, that is in fact very, very significant. And similarly, there was a 28 percentage point reduction in ambulance transports between the treatment and control remember that like the reason why that second one is less than the first is that, remember, you don't necessarily if an EMS team is sent out to you, you don't necessarily get transported.

 

Jennifer [00:45:14] They can say, sorry, this is not emergency.

 

Kevin [00:45:17] Or you can say, actually, based upon your advice,.

 

Jennifer [00:45:20] Oh.

 

Kevin [00:45:20] I don't want to be transported. Right? You can always refuse transport. And so, you know, you know, maybe they fixed your ankle and you're like and they're like, well, you're going to be waiting 2 hours to get to the hospital and you still want to go and you can say, no, I'm good. So.

 

Jennifer [00:45:36] Okay.

 

Kevin [00:45:37] So just to say that's that's why that number is lower.

 

Jennifer [00:45:40] Okay, great and then next, you look at health care utilization. So what was the effect on future visits to a primary care physician and to emergency departments?

 

Kevin [00:45:49] You know, so in terms of future visits, we looked both in the very short term, so within 24 hours to try to measure that immediate emergency department usage and within six months and this is just that Medicaid population, which is, again, about half the population overall. And what we saw is that within 24 hours, our control group had about a 29.5% rate of emergency department usage within 24 hours, whereas our treatment group had about a 25.1% rate so that's about 15, 16% less or about four and a half percentage points less. That's also extremely significant p less than 0.001, but that vanished in the long term within six months, we actually had both treatment and control, had visited the emergency department with about a 43% chance. So that's not super good short term gains that were not meaning in the long term.

 

Jennifer [00:46:46] That suggests, I guess, that if part of the goal here is to try to change the culture of calling 911 and it turns out it's going to take more than being randomized once into the treatment group for the nurse triage line, which is perhaps not shocking but does you know this is a bigger problem than just that's all.

 

Kevin [00:47:07] Yeah and so and it's a similar story for the primary care utilization. Within 24 hours, about two and half percent of our control group had gone to a primary care physician and about 8.2% of our treatment group had gone to a primary care physician. So that's about a tripling of the number of people who actually visited a primary care physician within 24 hours, but again, within six months, those numbers had both gone, coincidentally also to 43%. And that is, again, you know, a little less of a good sign.

 

Jennifer [00:47:40] Yeah, I'm trying to think here because it really is. It's like being randomized once. And I guess if you call back, then, you know, there's no guarantee you might not be randomized the next time into the getting the nurse triage line.

 

Kevin [00:47:51] So true.

 

Jennifer [00:47:51] You're still sort of at the mercy of whatever the system is. So you can imagine once the system switches and expands to all calls that those long run numbers looking better. Is that sort of what you're imagining, too?

 

Kevin [00:48:04] It's certainly it's certainly possible because we randomized at the call level. Actually, we had a few people graciously read our pre-analysis plan before the experiment started and a couple of them pointed out exactly this problem that like, can you do it at the person level instead of the call level? And we thought about it for a while and we're like, there's no way that like.

 

Jennifer [00:48:26] Yeah, that would be amazing. So basically if the same person calls back, you basically get whatever the same treatment was you did before. Yeah, that's ideal, but not going to happen.

 

Kevin [00:48:37] Yeah, we could not do it. It was it was. We had some ideas, but we're like, no, no, no. That's very brittle. That's not gonna work. And so, yeah, so it is possible that, you know, if you, for instance, if some people call the emergency line actually with some very great frequency, you know, multiple times a month type frequency and so those people are an example of people who may be driving some of this number. So you're exactly right that when it rolls out to all calls, in fact, it has rolled out to all calls in the interim. We hope those numbers will separate a little bit.

 

Jennifer [00:49:10] Yeah. So was the program cost effective?

 

Kevin [00:49:13] On the one hand, yes. The council gave FEMS about a million bucks to try on this and the back of the envelope indicates that we saved about $1.5 million worth of ambulance rides with no obvious negative health outcomes that we could detect. So that's pretty good, you know, save 50% afterward, saving money. On the other hand, it can get a little murky because you can't buy partial ambulances there's a lot of medications and ambulances that have to be kept cold 24/7 and watched over. So you really kind of have to buy one ambulance at a time. And if you recall, an ambulance costs between one and $1.5 million a year. So not clear that we actually could reduce operational costs of FEMS at that kind of level, but watch out for future work. We hopefully by rolling this out to a broader range of diagnoses, we can push that number down.

 

Kevin [00:50:04] Also in the long term, we did look at some Medicaid expenditures and we didn't really find a long term reduction in health care expenditures between our treatment and control group that we could actually detect. So we may have reduced immediate spend in the ambulance department, but we did not reduce long term spend in the actual Medicaid system. Again, I guess not super shocking because, you know, this is a one touch point as opposed to like trying to be multiple touch points along the way, but it is a it was kind of disappointing from there.

 

Kevin [00:50:35] However, this initial pilot was very conservative the diagnosis, there are thousands of potential diagnostic codes and we chose 50 of those. Right. And, you know, we had about a treatment group or we had a total sample size, about 6,000 calls over a year out of about 160,000 calls a year that 160,000 medical related calls that 911 gets each year in D.C. So we were at a very small percentage of all possible calls. So it's possible that as that list gets expanded and experimented with, these numbers will start to look even better.

 

Jennifer [00:51:12] I'm thinking now back to kind of what the initial goals were and one one is, you know, trying to reduce the use of this costly resource the ambulances right so expensive to transport someone to the hospital with an ambulance and there's the taxi or whatever else, but the other one was the other big goal was making sure an ambulance was available when someone was calling with a heart attack. Was there any way to look at like response time or reduction in the number of calls with a too long response time or something like that?

 

Kevin [00:51:42] Yeah. So I don't have those numbers in front of me, but they're actually part of DC's FEMS annual performance reports and they actually have a dashboard online that drew these numbers very, very closely. And I'm pretty sure on there somewhere is number of minutes each day that there is not an ambulance available.

 

Jennifer [00:51:57] Yeah.

 

Kevin [00:51:58] So I don't have them in front of me and I don't want to say wrong number is on the, on the economics podcast.

 

Jennifer [00:52:04] Yeah.

 

Kevin [00:52:04] But but they are tracked.

 

Jennifer [00:52:08] Yeah. This is, and this is something that will be more relevant when it scales the whole city then than just, you know, a small fraction of the overall calls, but.

 

Kevin [00:52:14]  Yeah.

 

Jennifer [00:52:15] That'll be neat to see.

 

Kevin [00:52:16] Yeah, we, we didn't expect to put a huge dent in it.

 

Jennifer [00:52:19] Right.

 

Kevin [00:52:19] With this, like, you know, 4 to 5% of all possible calls.

 

Jennifer [00:52:22] Right yeah, very good point. So what are the policy implications here? What should policymakers and practitioners be taking away from this? And what did those in DC take away from this?

 

Kevin [00:52:31] Yeah, well, as I said, one of the coolest things we did was we did do this pre-registration and we asked them like, what did they care about.

 

Jennifer [00:52:38] The pre commit.

 

Kevin [00:52:39] And they had the pre commit and it was a success according to their pre committed outcomes.

 

Kevin [00:52:44] So they thought it was a success and indeed it has been sustained. In fact, a small programing note is that the initial results were actually computed and presented to the deputy mayor as well as the council way back in 2020 and so this paper is just coming out now. I will blame COVID and also just the vicissitudes of life, but the as well as some staff turnover about the usual problems that occur. So it actually has continued to this day and it has been expanded to some degree. Look out for future follow ups. I hoping to have some grad students look at this over the next year or so to see what's been happening over the past couple of years.

 

Kevin [00:53:25] And so the initial outcome is that they thought it was success. They scaled it. They built it up. Right. Amazing. I would note that an extra thing that I take away from this, though, is that and this has been a theme I noticed on your podcast, Jen, for a while, which is that this notion of actively helping people into services is actually really powerful. I know that you had Professor Sukhtankar, Panka Bencsik, Professor Phillips. They all like have these papers about how, you know, if we women in India with women's help desks like right there in front of people, it increases utilization of those services. If we have police directly refer people to drug programs that actually vastly.

 

Jennifer [00:54:11] They take them.

 

Kevin [00:54:12] And they take them up. Right. You know, if we have connect directly, connect returning citizens to mental health services, they take them up like.

 

Jennifer [00:54:19] Yeah.

 

Kevin [00:54:20] And that active role that people can play and that governments can play is like, that's the thing that I really take away here is like the real big takeaway is that like there are these opportunities, these touch points where we can not just tell people that something exists but actually set them up for success, make that appointment, call them a taxi. These things are cheaper than they otherwise would be. And if you do it, you get these giant numbers that you don't expect to see in any kind of experiment ever and so that's my that's my really big takeaway, especially in health care space, now that we have a lot lower uninsurance rates than we did in, say, the 90s and the early 2000s. There's a lot more room to play with interventions than we had before, before the now as a lot of you know, how do we get people onto insurance? And now you've got people on to insurance and so there's a lot more play for like how can we get people to use their insurance in the ways that help most helps them?

 

Jennifer [00:55:13] Mm hmm. Yes. Which is something I hear more and more from policymakers about, actually, is there is more evidence that, you know, expanding access to health care reduces crime. It's like, okay. And the next question is, how do we actually get people to take up the health care they now have access to? Yeah. Very good connection there. All right. So you've been working on this for a while. Are there any other papers related to this topic that have come out since you guys first started this experiment.

 

Kevin [00:55:39] As you said mentioned early, has done a lot of work on nurse help lines.

 

Jennifer [00:55:42] Mm hmm.

 

Kevin [00:55:42] That's come out, but the big one that I'm very, very excited about is this Dee and Pyne paper in Science Advances. That's just a couple of months old at this point in time, really looking at those fourth services in emergency providers. And I think there's just those fourth services are an example of like actively connecting people to things that they wouldn't either otherwise have access to and I think there's a lot of space to play there. So that Dee and Pyne paper as well as I know there are several other that are that are in the docket to come out. That's the one that I think is like the most exciting line of research.

 

Jennifer [00:56:16] Yeah, it's funny when I talk to people who are interested in what cities can do along those lines, how do we reform the 911 system so that we can dispatch health workers or other teams that are less police focused I often bring up this nurse triage line as an example of, like, logistically, how one could implement such a thing.

 

Kevin [00:56:36] Mm hmm.

 

Jennifer [00:56:37] Have people been calling you either because I've been sending them or just they've heard about this whole thing you've managed to do. It feels like this is going to be a really useful model for a variety of different, you know, types of interventions you could. You can imagine.

 

Kevin [00:56:50] Yeah, nobody's called me personally. Honestly, one of the things that I think has occurred is COVID happened.

 

Jennifer [00:56:57] Aha.

 

Kevin [00:56:58] And a lot of the energy around this because as I say, it was like literally 2020 mid 2020 post-COVID when we were like, you know, first presenting these results.

 

Jennifer [00:57:07] Yeah.

 

Kevin [00:57:08] Emergency services were swamped by a lot of other problems. And I, I have seen and I talked to some people about this specifically, but I have not seen, you know, the yes, we want to migrate our emergency services provision while we're also rolling out vaccine programs and requirements of other things. So and so that's I'm hopeful that in a post-COVID landscape we'll hopefully this the listeners of this podcast will find my email address and email me and we can talk about how to implement this in your 911 system.

 

Jennifer [00:57:40] Indeed. Yeah. You can help them build a data like.

 

Kevin [00:57:43] Yeah, exactly.

 

Jennifer [00:57:46] Where ever they are. I love it. What's the research frontier? What are the next big questions that you and others are going to be thinking about going forward?

 

Kevin [00:57:53] Yeah, it's as I mentioned, one of them is just literally on this front, looking at what's happened in the past couple of years in this particular system and I'm hoping to have some students look at that over the next year or so. So that's just one question, like what's been going on as it's expanded? Are those numbers separating that we discussed a little bit more now that there's the number of diagnoses has expanded? And as I say, the other really big thing is this like active connection to people like this is the thing where to me there's a lot of discussion about things like administrative burden and making it easier for people to access services and, you know, not requiring people to get an interview or to sign a piece of paper that indeed you showed up for an interview in order to keep your food stamps a thing that actually does occur in some jurisdictions. And there's been a lot of work on that, and that's an extraordinarily important work. But this active connection is kind of like the next thing beyond that that I think is going to be really, really important. Things like automatic automatic enrollment is kind of an example of it and is one thing that has seeing success in various places. But also this like are there places where, you know, we can in the service delivery space where somebody shows up, doesn't need your service, but I can in fact provide you directly the other service and get you set up.

 

Kevin [00:59:12] I think it's going to be a really, really big place for a lot of interventions.

 

Jennifer [00:59:17] My guest today is and Kevin Wilson from the policy lab at Brown University. Kevin, thank you so much for talking with me.

 

Jennifer [00:59:24] Thank you.

 

Jennifer [00:59:29] You can find links to all the research we discussed today on our website probablecausation.com. You can also subscribe to the show there or wherever you get your podcasts to make sure you don't miss a single episode. Big thanks to Emergent Ventures for supporting the show and thanks also to our Patreon subscribers and other contributors. Probable causation is produced by Doleac initiatives, a 501(c)3 nonprofit, so all contributions are tax deductible. If you enjoy the podcast, please consider supporting us via Patreon or with a one time donation on our website. Please also consider leaving us a rating and review on Apple Podcasts. This helps others find the show, which we very much appreciate. Our sound engineer is Jon Keur with production assistance from Nefertari Elshiekh. Our music is by Werner and our logo was designed by Carrie Throckmorton. Thanks for listening and I'll talk to you in two weeks.