00:00:21.420this is now a no birds podcast you have never sounded more like jerry stiller no birds no birds
00:00:31.540um what do you got what have you got for us uh hi everybody and welcome to maintenance phase
00:00:40.820the podcast that's crawling out onto the sidewalk after a nice long rain oh it's a worm it's a worm
00:00:47.340opener. All I had was worms. You had two words worm wars. Sure, man. I'll take my inspiration
00:00:54.740from my morning walk the other day. I'm Michael Hutz. I'm Aubrey Gordon. If you would like to
00:01:00.880support the show, you can do that at patreon.com slash maintenance phase. And if you want to just
00:01:06.680keep listening, just keep listening. We're so happy to have you here with us. Today we're
00:01:11.000talking about worms. Yes. So okay, I'm so excited. Because I like telling you stories. And I like
00:01:17.080fucking with you and so i i told you i told you nothing other than the name of this episode worm0.99
00:01:26.200wars this is the controversy that tore methodology twitter asunder in 2015 and i want to tell you0.99
00:01:35.680all about it tell me so the hardest part of this episode is going to be getting through the first
00:01:41.040five minutes oh there's just some like gross stuff like body horror that we have to get through
00:01:45.480okay are you familiar with intestinal worms uh i mean not really okay so i assume they're like
00:01:51.440parasites yeah yes okay so somewhere around two billion people in the world have intestinal worms
00:01:58.520which are like exactly what they sound like they are little tiny worms that live inside of your
00:02:03.940digestive system some intestinal worms have lifespans of like one to two years and they'll
00:02:08.340literally just like attach themselves to the inside of your intestine and just like be there
00:02:12.280for years and then they die okay most people have light to moderate infections which don't really
00:02:16.600have any symptoms but sometimes you'll get like much more heavy infections and so this causes
00:02:22.580various symptoms that it it usually happens through giving you anemia it makes you tired
00:02:27.520it gives you brain fog you have to sleep all the time it can stunt kids growth in the developing
00:02:33.500world it can make them too skinny i have never had a worm infection or parasite or any of that
00:02:39.920kind of stuff but i have been anemic before and boy oh boy it doesn't seem like it's gonna be
00:02:47.580that rough and then it's pretty rough dude a friend of mine passed out in a starbucks because
00:02:51.740she got anemia just you just can't get your energy up like at all yes so okay so here's the really
00:02:57.960gross part so the way that worms spread is you get these worms in your intestine lining and they lay
00:03:04.560eggs they lay like hundreds of thousands of like little microscopic eggs and these eggs end up in
00:03:10.120your poo and then you poo out the eggs and so the reason why worm infections are endemic throughout
00:03:17.040the developing world is because oftentimes there isn't running water so in some areas people will
00:03:22.200kind of use soil to wash their hands after they go to the bathroom they'll just kind of grab some
00:03:25.940like dry soil from the ground sometimes they use poo as like fertilizers in gardens and then it
00:03:32.520ends up in their food or it's like they're walking around and they have a cut on their foot and
00:03:36.720there's like a little tiny egg that like ends up getting into the cut and then like swims its way
00:03:40.220to their intestinal tract sure sure sure so it's just like the poop human interface is like how
00:03:47.280people end up getting worms and it's like this rolling snowball where like the more worms there
00:03:50.960are in the soil the more likely kids are to get it the more likely kids are to spread it etc it
00:03:55.900used to actually be a really big problem in america here is what i appreciate about this
00:04:01.240whole episode is i didn't have to bring up anything related to poop i know you just fully
00:04:08.200picked a topic and you were like you know what i could go for right now is a little poop research
00:04:12.600the thing is i i saw a methodology fight and i was like oh it's gonna drive me into the poop world
00:04:18.940isn't it i have to go inward so the protagonist for this episode is a guy named michael kramer
00:04:26.560who is currently a harvard professor okay before all of that he had spent some time in kenya as a
00:04:34.320kid and he was really fascinated by like why are people poor in this country like what are we doing
00:04:40.760to alleviate global poverty he eventually gets his degree in economics and so he decides to do
00:04:45.740his graduate work in kenya so he moves to kenya and he starts befriending people in the development
00:04:52.500sector there he meets like world bank people and ngo people and charity people and what he finds
00:04:57.560is that the field of international development is like in kind of what other people will later call
00:05:03.140a statistical dark ages this is like the early 1990s and the way that people do development it's
00:05:08.880kind of on like the live aid model right where it's like a celebrity does something and there's
00:05:14.740like this big fanfare and then like no one ever follows up afterwards yeah and if there's any
00:05:19.700like statistical rigor or like studies being done there oftentimes before and after studies so
00:05:24.460they're like we gave everyone a goat and then a year later they were doing better and it's like
00:05:29.260well was just everyone doing better in the country that year because there was like a really good
00:05:32.820harvest or something like you're not you're not comparing them against a control group or was
00:05:37.540there another policy intervention right there's like so many things that it could be right think
00:05:42.640about the last year in your own life and think about if someone from the outside tried to determine
00:05:48.800what was the cause of any changes in your life right like that would be kind of a fool's errand
00:05:54.500right right and a lot of the projects are done in this sort of like common sense way this is
00:06:00.180oftentimes how you get bad development projects where people are like oh it's just common sense
00:06:04.740to you know give everyone a bicycle or whatever no it's like well did you ask them first if they
00:06:09.420needed a bicycle like if you give everybody in the village a bicycle is that going to like change the
00:06:14.100dynamics of the village like it's just being done in this like really unsophisticated way
00:06:18.180yeah so michael kramer has a friend who works at a dutch ngo and they're about to implement a
00:06:24.700program where they give kids in kenya textbooks so at the time there's like 30 kids in every
00:06:30.020classroom and there's one textbook per classroom so again common sense it's like well if everybody
00:06:35.680had a textbook this is obviously going to make like their grades better and like school performance
00:06:39.800attendance everything's going to get better if everybody has their own textbook obvious right
00:06:43.120so michael kramer suggests to his friend he's like no no don't do a before and after study
00:06:47.860instead of just giving this to seven schools take 14 schools and give seven of them textbooks and
00:06:54.320the other don't give them textbooks and then you can actually compare the effect of the textbooks
00:06:58.240so they do this the project goes for a couple years and lo and behold there's essentially no
00:07:04.280effect of the textbooks uh-huh so after the textbooks don't work there's this period of
00:07:08.460kind of testing stuff because it's like well that's weird that the textbooks don't work
00:07:11.220so then they try like chalkboards like a lot of the schools don't have chalkboards so again they
00:07:15.040split into two groups. Turns out that doesn't work. They then do a thing where they hire more
00:07:18.940teachers at half of the schools. That doesn't work. So it's like they're kind of going over
00:07:23.380these sort of obvious, quote unquote, education interventions. And it turns out that like
00:07:28.720education interventions are not working to improve education. They're like, hmm. So one night,
00:07:34.380Michael Kramer is talking to a friend of his who works at the World Bank, and he explains the
00:07:39.780symptoms of worm infection to michael kramer and he tells him that like there are treatments
00:07:45.980available for worms like there are pills for basically every kind of species that's endemic
00:07:51.620in that part of kenya and like they're really cheap and diagnosing the kids is actually like
00:07:57.960really expensive because you literally like you literally have to go through their poop and like0.99
00:08:01.280measure how many eggs there are and so it's such a pain to diagnose this and it's so easy to give
00:08:08.420the kids these medical treatments. They don't really have any major side effects. They're
00:08:13.000pretty cheap. They're already available in that part of Africa. And he's like, I wonder what
00:08:17.200would happen if we just gave all the kids these deworming tablets, because some of them are
00:08:21.500probably infected with worms and just don't know it. So they do the same thing. They find a district
00:08:25.400with 35,000 students, 75 schools, and they do the same thing. They separate it into control groups
00:08:32.780and treatment groups. They roll out this mass deworming program where they just give every kid
00:08:36.900deworming pills, right? Okay. So I'm going to send you an excerpt. This is from a book called
00:08:44.240Doing Good Better, How Effective Altruism Can Help You Make a Difference by William McCaskill.
00:08:50.340Quote, Kramer did an experiment to see whether treating children for these intestinal worms had
00:08:55.080an impact on education. The results were striking. Absenteeism is a chronic problem in schools in
00:09:01.100Kenya and deworming reduced it by 25%. In fact, every child treated spent an extra two weeks in
00:09:07.740school and every $100 spent on the program provided a total of 10 years of additional
00:09:12.480school attendance among all students. Enabling a child to spend an extra day in school therefore
00:09:18.080cost just five cents. It wasn't merely that deworming children quote unquote worked at
00:09:23.040getting children into school. It worked incredibly well. It worked incredibly well. I mean,0.74
00:09:28.880fucking yeah yeah if you have something uncomfortable too painful too debilitating0.99
00:09:36.760going on with your body when you feel better i bet you do better in school yeah so i mean it's0.98
00:09:43.140yeah it's a very appealing story also because they've tried all of these educational interventions
00:09:47.920and it turns out that there's this sort of like zigzag solution that maybe it's not an education
00:09:53.180intervention maybe it's like there's something broader and bigger at play that is just invisible
00:09:57.580to people that are in the school and also development aid workers yep totally so the
00:10:01.420main headline from this study is that they reduced absenteeism by 25 that's the that's the thing that
00:10:07.460ends up in william mccaskill's book that's like the kind of the main thing it's in the abstract
00:10:11.300but there's also some other effects so one of them is like there's health benefits so the kids who
00:10:16.020got the deworming were slightly taller they're slightly heavier it was actually nice to read a
00:10:20.420study where it's like the kids are fat yeah it's like it's like just one what a win they also had
00:10:27.580lower rates of anemia and one of the really interesting effects of the study is these
00:10:33.620spillover effects so because worm infections are an infectious disease right one person gives it to
00:10:39.460another person once you blast hundreds of kids in these schools with deworming pills they then don't
00:10:45.360spread it to other people so what they find is that there's there's actually attendance effects
00:10:50.180in schools like three kilometers away and six kilometers away there's all these like other
00:10:55.260benefits that come along with basically just like breaking the chain of infection
00:10:58.320and they also find long-term effects this is from mccaskill's book when kramer's colleagues
00:11:04.420followed up with the children 10 years later those who had been dewormed were working an
00:11:08.520extra 3.4 hours per week and earning an extra 20 percent of income compared to those who had not
00:11:13.720been dewormed in fact deworming was such a powerful program that it paid for itself through increased
00:11:19.460tax revenue wow so it's like you can then even like fast forward 10 years and you find that like
00:11:25.000the dewormed kids are doing better big if true big if true aubrey it's the first five minutes
00:11:29.760of the episode you know everything i tell you at this point is true everything goes according to
00:11:34.040plan the nice white guy from harvard is definitely right and no harm is going to come from his
00:11:38.880actions stop remembering the way that i tell you stories um stop remembering the premise of our
00:11:44.780The premise of our show. So over the next decade, basically, deworming becomes like the hottest
00:11:51.200idea in international development. I was working in international development at this time. I
00:11:55.280remember all of like the deworming national plans that were coming out. It paralleled the rise of
00:12:00.760this effective altruism thinking, where in Michael Kramer's original paper, which was published in
00:12:07.2202004, a lot of the paper is actually about like the cost benefits of this, right? That, you know,
00:12:12.580When it comes to something like hiring more teachers, well, like, every single teacher costs a lot of money and there's, you know, there's payroll and there's all these other costs associated with these, like, educational interventions.
00:12:21.880Whereas this one, like, the deworming pills cost 30 cents each.
00:12:25.820The benefit of this versus the cost of doing it is just, like, unprecedented international development.
00:12:32.180So in 2007, Michael Kramer founds a NGO called Deworm the World.
00:12:38.200Which, like, helps governments implement their, like, national deworming plans.
00:12:42.620There's also this charity GiveWell, which basically has this model of like, we're going
00:12:46.360to look at all the evidence and we're just going to tell you that like, if you have a
00:12:49.280hundred bucks to give away, here's probably the most effective thing to give it to.
00:12:53.060You know, if you, if you want to help people, this is what you should do.
00:12:55.960And that's like one of the few charities that is ranked by GiveWell is this Deworm the World
00:13:01.720initiative and other kinds of deworming charities.
00:13:04.580There was like a deworming strategy by the WHO.
00:13:07.920The World Bank was really big on deworming.0.94
00:13:09.600it was like deworming was like the hotness right kenya has a national plan on the basis of this
00:13:14.7202004 paper india declares february 10th national deworming day uh-huh and has a program to deworm
00:13:23.420140 million kids a year whoa this is huge according to the gates foundation at one point they put out
00:13:28.960a paper calling this the largest public health program ever attempted wow huge full court like
00:13:35.380decade-long full-court press on deworming there's part of this that feels very like
00:13:41.520ted talky life hacky this one simple trick aubrey fixes everything aubrey am i getting ahead of
00:13:50.260myself i love that you're at the point with the show where you're like wait a minute there's a
00:13:53.980good story here it's probably fake i don't trust it like
00:13:59.900so okay i'm about to send you a couple paragraphs from a guardian article in 2015
00:14:10.800that kicks off the worm wars this is one of the longest excerpts that we've ever read on the show
00:14:18.040but i wanted to get i wanted to you to get like the flavor of this great the headline says new
00:14:23.980research debunks merits of global deworming programs. And I say fucking called it twisted0.93
00:14:30.220twist quote deworming children once ranked by Nobel laureates as the fourth most effective0.98
00:14:35.500intervention to solve the health problems of the whole world offers very little benefit despite
00:14:40.300the millions of dollars spent on it according to a reanalysis of the evidence. The World Bank and
00:14:45.800the UK government are among the big funders of deworming programs that were claimed to improve
00:14:50.720not only children's health and growth, but also their educational attainment and ultimately their
00:14:55.740country's economic prosperity. However, much of that optimism rests on one particular study
00:15:01.500carried out in Kenya in 1998 to 1999, which reported that deworming improved attendance
00:15:08.900not only in schools where children were treated and taught about avoiding worms, but also in
00:15:14.520other schools because of an apparent reduction in transmission. Researchers from the London
00:15:19.600School of Hygiene and Tropical Medicine have now reanalyzed the trial data and published two papers
00:15:25.440in the International Journal of Epidemiology. Researchers used methods common to health
00:15:30.760research rather than those used in economic research, as the original paper had. God damn it,0.97
00:15:36.820Michael. Good sign. Good sign. The epidemiologists finally look into it 10 years later. There had1.00
00:15:42.700been no increased attendance in the schools where children had not been treated, as the original
00:15:47.960study said that was down to calculation errors and was wrong the researcher also found missing
00:15:56.760data and some evidence of bias in the finding that school attendance improved shocking to see
00:16:03.060that coming what what is your what is your overall impression what is what is the plain language0.94
00:16:07.640paraphrase of what you've just read the fucking epidemiologist never looked at an epidemiology0.99
00:16:13.720issue and it was studied by fucking economists what are we doing and then like a whole fucking0.99
00:16:20.120nations built their interventions around what a fucking economist thought was good for public0.99
00:16:25.680health at this point in 2015 i've seen various numbers going around one of the numbers is that0.96
00:16:30.7401 billion deworming pills are being distributed annually what under programs like this i assume
00:16:37.160you're gonna walk me through this yes it sounds like what they have debunked here is the idea
00:16:43.120that school attendance did not improve in schools where kids were not treated that's a very good
00:16:50.980catch yes it's not the whole thing is garbage it's just this idea of school attendance going
00:16:57.560up in other schools is not true that's what i'm taking away from this paragraph that was a very
00:17:02.680good read that like a lot of people did not notice at the time you just notice now fun so now we get
00:17:09.400misunderstanding so this is the thing so we have to talk about what they actually did and what they
00:17:13.180actually found so okay as you know there's like in psychology and other sciences there's now this
00:17:18.360like this thing called the replication crisis essentially the idea is that like there's all
00:17:22.460these canonical studies that have affected policy and have affected public understandings
00:17:26.400and they're basically rerunning them and in a lot of cases they don't pan out so things like the
00:17:31.680the infamous like marshmallow test what happened here is that people looked at this study from
00:17:37.2002004 in Kenya being used to justify deworming 140 million children in India. And we're like,
00:17:45.300wait a minute, like a lot is resting on this paper. Let's just look into the findings. So
00:17:50.480like they didn't redo any studies. This wasn't a replication. What they did was they went to
00:17:56.120the original author. So it was Michael Kramer and his co-authors on the original paper. And they
00:17:59.900said, give us all your data, like give us the software that you ran, give us the original,
00:18:03.540like the questionnaires you used in the school like just give us everything and all we're going
00:18:07.400to do is just run it again okay so to their great credit kramer and his co-authors like give them
00:18:13.680everything okay so these re-analysis researchers spend months going over everything and basically
00:18:20.060publish like a re-analysis of this paper so the biggest thing that they find is there's data
00:18:25.860missing and there's data that's miscalculated you know that like spreadsheets have all these
00:18:29.860conditionalities where like one cell refers to another cell.
00:18:58.980the kids difference in heights gone the difference in weights gone wow and then essentially all these
00:19:05.040spillover effects so like all the stuff about like the indirect three kilometers six kilometers
00:19:09.200those effects the the six kilometer effects are gone and the three kilometer effects are
00:19:14.380much much much weaker to me the main thing that comes out of this statistical re-analysis
00:19:21.640i mean i mean this is something very well known in studies of school attendance in sub-saharan
00:19:26.340Africa and really throughout the developing world, that like if you rely on school registers,
00:19:31.500they're not very accurate. Teachers write kids down as being here when they're not really here.
00:19:35.660So if you want to do a study that actually measures school attendance, you have to send
00:19:39.540people to the school to like physically count the kids, which to their credit, they did in this
00:19:43.800study. Like they sent people to like, okay, one, two, three, four, five in every school when they
00:19:47.540wanted to do it. And they did it at random times, right? Because you want to get an actual like an
00:19:50.700accurate kind of the kids what they found in this reanalysis was that the deworming schools
00:19:57.420got way more visits from the counters from the auditors and the schools that got visited more
00:20:04.940had better attendance rates it could be just the fact that someone was checking that's doing this0.99
00:20:11.340and not actually deworming yeah so this is this this is what turned me into the fucking joker on0.77
00:20:16.580this once there's all this like statistical stuff going on and this extra scrutiny on the original0.89
00:20:24.440paper people also go back and just like read the original paper the first thing that jumps out at
00:20:31.340you is that this wasn't a deworming program it was a deworming and education program so the schools
00:20:39.980that got the deworming pills also got like classes about like how why it's important to wash your
00:20:45.940hands like better sanitation techniques how to prepare food the kids and their parents in some
00:20:51.400cases were getting information along with the deworming pills that was related to not getting
00:20:57.080reinfected with worms nah but it's definitely those pills but it's definitely i mean this is
00:21:01.480the thing is like the the the researchers say that when you survey the kids about like how
00:21:07.440important is it to wash your hands and stuff there's actually no difference between the kids
00:21:11.340that got the education stuff and the kids that didn't wow as we talk about on this show all the0.76
00:21:15.540fucking time like public education campaigns don't really work yeah knowing that washing your hands0.95
00:21:20.080is good doesn't give you running water in your house yeah the researchers are like oh well it0.99
00:21:25.580doesn't matter that we also gave them education programs because the education programs didn't
00:21:30.280work so we can scale up the pills without scaling up the education but the thing is they didn't
00:21:35.800check the effect of the education on the school attendance there's literally no way to separate
00:21:40.620those things you're doing two things at once and then school attendance gets better i am
00:21:45.300just really breathing through feeling my blood pressure rising you don't have to be like the0.83
00:21:52.860world's greatest fucking scientist to figure out that if you give somebody deworming pills and also0.92
00:21:59.180like a bunch of like educational resources and support for their lives you can't then say0.97
00:22:05.600everything that's changed is just because of those pills we've been through this so many times as a
00:22:10.440field this is as a former development aid guy this is what drives me nuts is that what you have
00:22:14.840over and over again is you have a technical intervention something like a medical intervention
00:22:20.520or like better lighting in schools or whatever something that's easy to scale right yeah then
00:22:25.860you also have all these social things that go around it right like getting to know the kids
00:22:31.020having them attend these educational seminars having probably just like more adults in the
00:22:36.180school generally if they're participating in a study right and then it's like when it's time
00:22:40.800to upscale we're like oh this is a proven development intervention so we're gonna upscale0.81
00:22:45.220the technical parts yeah but you're not gonna upscale all the fucking social stuff that made0.98
00:22:49.380the technical stuff work god damn it that's not the purpose of a randomized controlled trial the0.97
00:22:54.180entire point of all of this was to be like we're gonna do it like pharmaceuticals do it right we're0.94
00:22:58.900gonna do it in a way that allows us to say with certainty that like this is the intervention that
00:23:03.820mattered yeah and then they do this like 101 shit where it's like oh well we're gonna do two things
00:23:08.980at once and then upscale one of them yeah totally and also like this does feel sort of related to0.85
00:23:15.020because this is all happening at the site of education as a field it also feels totally
00:23:22.480fucking wackadoo and deeply predictable to have a bunch of economists roll in and be like0.94
00:23:29.500teachers you've been fucking it up you're welcome we're here we figured it out but then okay are you0.99
00:23:35.640are you ready to explode oh no i guess so the other thing that people start like bringing up0.98
00:23:42.560about the original paper is that that number of that it it reduced absenteeism by 25 is a little
00:23:49.440bit misleading okay if you imagine a school with like a 98 attendance rate and then you switch the
00:23:56.380vending machines from like coke to pepsi and then they have a 99 attendance rate you could say that
00:24:02.660they reduced absenteeism by 50 oh god damn it that sounds really impressive but what you've done is0.99
00:24:09.280you've gone from 98 to 99 which is not that impressive no this is the like fucking diet0.98
00:24:14.660marketing of like you're twice as likely to lose weight on our diet than on this other diet and0.97
00:24:19.060you're like okay so you lose one pound instead of half a pound in a year great you're twice as
00:24:23.500likely to win the lottery when you buy two lottery tickets that would have said i mean you're gonna
00:24:26.840the actual attendance rates at the schools that didn't get deworming pills the attendance rates
00:24:34.560are 75 at the schools that did get deworming it's 82 oh god is that nothing no like it's a
00:24:41.980seven percent bump in attendance rates but also a seven percent increase in attendance much less
00:24:47.860like we must scale this to 140 million children immediately another thing that again is in the
00:24:53.740paper that was published in 2004 is that the deworming pills did not have any effect on
00:25:00.480cognitive ability or school performance so nothing nothing so exam scores they did some like
00:25:07.400iq testy pseudoscience bullshit on the kids nothing so it only affected attendance i feel0.98
00:25:15.540like i'm resorting to fucking lamaze breathing to get through this shit like i feel like i'm this0.99
00:25:22.660close to like to like deal with my fucking rage about this well the whole the whole narrative0.99
00:25:31.160of this intervention doesn't really make sense yeah the the theory of change that deworming is0.95
00:25:36.700supposed to do right it's like you give kids the deworming pills their health improves their school
00:25:41.140attendance improves their test scores improve and then their life gets better right that's like the
00:25:45.700chain of events right yeah but what we have now is we have you give the kids deworming pills
00:25:51.120it doesn't seem to affect their health it does affect their school attendance but doesn't affect
00:25:57.240their school performance yeah it's just kind of a mess i don't know man this feels like an indicator
00:26:03.120of the depths of two major appetites one is how fucking hungry we are at all times for a silver
00:26:12.560bullet thing and how much more fun it gets to people and exciting when it's like an unexpected
00:26:18.480thing like oh if you get this one pill you get better at school imagine totally and appetite0.55
00:26:24.620number two that is sort of shaping this is the unending appetite of white people to be like we0.72
00:26:32.380found the thing we solved it you're welcome you couldn't do it yourself we did it for you exactly0.97
00:26:39.640not a bad impulse to want to help a terrible fucking impulse to want to like save people
00:26:46.380from themselves i actually got radicalized on this i think a decade ago now where i was writing an
00:26:51.460article about like how big ideas were ruining international development i called up one of
00:26:56.920the charities that was doing one of these projects in india they were they were expanding the deworming
00:27:02.220mass deworming programs to india in one of the most kafka-esque interviews i've ever had i was
00:27:08.220asking them like okay how's the project going so far and they're like it's great we've we've
00:27:12.120distributed 10 million pills or whatever it was and i was like okay great like what are the results
00:27:16.560on school attendance and they were like oh we're not measuring that what oh because it's proven
00:27:20.480like it's proven that it affects school attendance so we we don't really have to measure that anymore
00:27:24.780and i was like well it's proven in 75 schools in kenya 10 years ago you're doing this in india
00:27:31.700which is a completely different educational context of schooling and like the government's
00:27:36.460role in school everything else and like so you're just not testing the effects anymore and they're
00:27:42.940like but we are testing the effects i was like well then what are the effects they're like well0.98
00:27:46.480we've distributed 10 million pills that's okay okay so you and i get into this shit often that's0.99
00:27:53.060a fucking output it's not a fucking outcome exactly that's step one to do what you set out to0.99
00:27:58.720do like it's truly fucking wild how often you and i get into data on this show and the results of a0.99
00:28:08.780given study or a clinical trial or a whatever are just here's what we did and then that celebrated0.60
00:28:17.040like uh something that has like an impact on people's health out in the world but aubrey save
00:28:23.700save your rage i'm getting too angry too soon aren't i this is a three-part debunking and that
00:28:29.500was only part one what that is frankly nothing short of a personal attack so at the same time
00:28:36.600that this reanalysis of the 2004 paper is going on i think this is a coincidence there's also a
00:28:42.240meta-analysis of every previous study on deworming that's ever been done oh one thing that drives me
00:28:49.060absolutely batshit about this story and about the way that it's presented in this like effective0.75
00:28:53.160altruism book is everyone pretends like no one thought of fucking deworming before there have0.96
00:28:57.740been deworming projects since the 1970s yeah so in 2015 there's a cochran review of every previous
00:29:04.780like decent study on this it covers 45 studies going back to the late 1970s it includes 20
00:29:12.000randomized control trials in africa it finds the review says there is quite substantial evidence
00:29:18.360that deworming programs do not show benefit in terms of average nutritional status hemoglobin
00:29:23.800cognition school performance or mortality nothing that's the good stuff right there the world's
00:29:31.280largest randomized control trial or at least according to the study itself was performed in
00:29:36.060india in 2013 on deworming it finds no effect there's a huge randomized control trial in china
00:29:43.020which also does not find anything there have been 10 years of various deworming projects in myanmar
00:29:49.200and the rates of worm infection are now higher than they were 10 years ago what which i i don't
00:29:56.140blame that i don't think that has anything to do with the studies itself i think again it's like i
00:30:00.580think there's something else going on there sure what drives me nuts about this is like why would0.80
00:30:06.000we still be talking about this 2004 study in kenya when there's like 50 other fucking tests of this0.70
00:30:12.520by now yeah people have tested this and it has been found wanting time and time and time and0.58
00:30:18.740time again it doesn't have any of the effects that people said that it had and that's on any
00:30:23.500measure right we're no longer in school attendance world i mean they mention school attendance this
00:30:28.620is actually like one of the main things of like the the worm wars like the big methodological
00:30:32.840debate that comes out is because michael kramer and others say that like you can't test whether
00:30:38.200it affects school attendance in like a meta-analysis because there's not enough studies
00:30:42.060that do school attendance it's only based on this one and they all like they throw accusations back
00:30:46.400and forth but but but it's like there are other studies that look at school attendance and it
00:30:51.800doesn't seem to have any impact on it and also even if it did it's like why are we talking about
00:30:57.060school attendance when like it doesn't affect school performance right and it doesn't affect
00:31:01.260health right then what then you got them in the room then what and also there's also some like
00:31:05.280interesting sort of second order effects that oftentimes when school attendance rates go up
00:31:09.220in the developing world the teachers are then overcrowded and like like lots of the kids end up
00:31:13.820doing worse on tests because they're getting less attention from their teachers not to say that like
00:31:17.220school attendance is bad or whatever but it's like anytime you're doing these development projects
00:31:21.380there's always these weird second order effects and so it's like it's not there are actually places
00:31:26.120where even if it did increase school attendance if it increases it significantly and they're not
00:31:30.920able to hire teachers yeah it might not actually have the long-term effects that you're going for
00:31:35.720yeah every study on this seems to find that it doesn't really do what they set out to do i in a
00:31:40.800lot of these studies you can tell they don't want to admit that it didn't have any effect you and
00:31:47.240me are like trained to say this now you'll read the abstract and they'll be like we distributed
00:31:50.62050 million pills and then like we surveyed a bunch of kids and the kids had heard of the program
00:31:56.400yeah and you're like wait that's not like no one would look at a deworming program be like have the
00:32:01.660gets heard of this or not yeah it's like you're clearly picking something you can publish we know
00:32:06.520that calorie labeling works because some people saw a billboard about it one time exactly i was
00:32:11.760subtweeting this when uh when we were recording that episode i was subtweeting the deworming
00:32:17.200people i have to say i really appreciate how often in the researching of an episode you or i end up
00:32:24.960like cryptically tweeting about something so that we don't spoil it only for the other person the
00:32:32.140thing is i was gonna block you on twitter because i was like about to tweet something salty about
00:32:37.280deworming and i was like i don't want i had a whole plan i was gonna be like aubrey i just want you to
00:32:42.840know i blocked you on twitter it's not you i tweeted about deworming but then i was like i
00:32:47.920don't want her to know so then i just i i walked away from my computer ultimately here's the
00:32:52.820hilarious thing as you know i have been very off twitter for the last like i don't know four months
00:32:57.880or so so the thing that is hilarious to me is i think the rest of the internet would be more
00:33:04.460likely to figure that out than i would yeah exactly i know and then did you know mike
00:33:08.480blocked you on twitter it's like mike blocked ombre on twitter trouble in paradise so okay
00:33:16.200final stage of the debunking okay okay we've talked about the 2004 paper problems we've talked
00:33:23.180about the field of deworming programs in general meta-analysis problem we're now talking about the
00:33:30.100problems of upscaling okay because this is another thing that as a former development guy
00:33:34.840i know that what happens in 75 schools in kenya doesn't happen the same way once you start rolling
00:33:43.120this out on like tens of millions of kids and like in entire countries sure so this is something that
00:33:48.880happens in 2008 in a region of tanzania called moragoro tanzania adopted like a national deworming
00:33:55.180strategy and was rolling this out across the country and so the idea was that they were going
00:33:59.660to give kids a bunch of deworming pills and they had a town hall meeting a couple days before this
00:34:05.640happened to like tell the parents about it but according to the parents later they didn't like
00:34:10.500really get the message that this was happening or like they just had other things happening that day,
00:34:15.160like agricultural duties, things at night that they had to do. So very few parents ended up
00:34:19.880attending this like one town hall meeting being like, we're going to give your kids deworming
00:34:24.620treatments. And according to the parents who actually went, the information that they gave
00:34:29.080the parents there was not even very good because they didn't really explain to the parents that
00:34:33.980they were doing prophylactic deworming. You know, deworming pills are like pretty widely available
00:34:39.180in sub-Saharan Africa. And like when you get symptoms of a worm infection, like you go to
00:34:43.140the pharmacy and if you can afford it, you buy the pills and you take the pills and the worms are
00:34:46.920gone. These are known, these are like known drug treatments. But the idea of we're going to give
00:34:52.500every kid deworming pills, like just in case that hadn't really been done before. It's a little bit
00:34:58.720like some teacher coming to you and saying like, well, you know, your kid takes Tylenol for
00:35:02.500headaches, right? Well, we're just going to give every kid Tylenol at 3 p.m. every day. You'd be
00:35:07.100like what right like it just doesn't really like fit with your understanding of like how that
00:35:12.240particular medication works yeah so after this town hall meeting that like very few people attend
00:35:17.060and doesn't give very useful information anyway they start doing this deworming with the kids
00:35:22.540but another change that they make is that the deworming pills that people in tanzania are
00:35:28.220familiar with i think it's called albendazole it's like a little tiny pill and apparently it
00:35:33.600tastes kind of sweet and the kids are used to taking it you know it's like kind of difficult
00:35:37.540to get kids to take pills in the first place sure absolutely but like because it's small because
00:35:41.660it's easy to swallow the kids are kind of used to it so it's not that big of a deal to get kids to
00:35:45.460take this but there's different species of intestinal worms so they're also having these
00:35:50.440kids take this other pill that is like big and bitter and they have to test the kids height
00:35:57.380before they give it to them to to get the dose and so some of the kids are taking three of this
00:36:02.400some of the kids are taking two of this it's just like big and like really unpleasant to swallow0.61
00:36:06.280and the other thing i cannot fucking believe this but this second pill that they're giving people0.81
00:36:11.780you you should take it on a full stomach okay otherwise it starts to give you these side0.91
00:36:16.280effects but like we are in a remote very poor region of africa like most of these kids are
00:36:22.660only eating one meal a day so it's like they're not taking it on a full stomach like whether you
00:36:27.600told the parents that are not they're not taking it on a full stomach what are the side effects
00:36:30.900It's like stomach ache, nausea. It's nothing serious. But the kids start coming home from school and they say, oh, they gave us all these pills today. They gave us this normal pill, but they gave us this new pill and they measured our height. And Tanzanians are used to their weight being used to determine doses of like various anti-malarials and stuff.
00:36:52.100sure measuring our height instead of our weight they gave us this new pill instead of this old
00:36:56.400pill and also like i have a stomach ache and like maybe i've been vomiting maybe i have a headache i0.96
00:37:01.000had these like weird symptoms yeah so the parents obviously are like what the fuck like a lot of
00:37:05.700them have not heard of this at all yeah totally and another thing that happens is fairly shortly0.62
00:37:11.060thereafter a little girl dies she dies of malaria she dies in the hospital she was not given the
00:37:17.080pills but it starts to look to people like these two things are connected right all of a sudden
00:37:22.680these kids are taking these weird pills in school they're getting weird symptoms and then this girl
00:37:27.440dies and so people on the local talk radio stations start talking about you know three or
00:37:33.580four kids have died because of the pills yeah and so what starts happening is the parents form these
00:37:39.820kind of mobs and start showing up at the schools so this is an excerpt from a phd dissertation
00:37:47.640that's written about this it says in some schools the situation turned violent as parents and
00:37:53.940guardians argued with teachers who had been administering drugs in one primary school in
00:37:58.100chamwino a classroom wall was pulled down school property was destroyed the national flag removed0.95
00:38:03.120rocks were thrown at the teachers and one was severely beaten holy shit the list of injuries0.93
00:38:08.480suffered by teachers included a broken arm a broken finger and a serious head injury in one0.99
00:38:13.420school a health worker had the drug stuffed down her throat causing her to choke and vomit holy
00:38:18.460hell i know the riot police were mobilized and dispersed angry crowds at a number of schools
00:38:23.780using tear gas jeez the ministry of health and social welfare officially suspended the program
00:38:28.540in moragoro and throughout mainland tanzania the same day so like i want to be very clear that like0.95
00:38:34.740there's a lot of really fucking ugly tropes about like savage africans and like africans0.84
00:38:38.640like there's nothing unique to african parents here we are talking about a program that is giving0.99
00:38:46.680a medical treatment a novel medical treatment to children without the knowledge or consent0.78
00:38:53.720of their parents people anywhere would have i think very rightfully like freaked out about this
00:39:00.820Right. And also like white people in the U S are losing their shit about their kids have to talk0.99
00:39:05.360about race sometimes. Well, yeah, exactly. It is to me a deeply understandable reaction that like1.00
00:39:13.040you find out from your kid that they've been given a new pill at school. You don't know what
00:39:18.720that was. You don't feel like you were notified about it. And now your kid is like getting sick
00:39:25.060as a result of that pill. Right. I don't know, man. Like this is like every episode of like
00:39:30.300law and order svu if anybody ever tried that with my kid they'd have held a pet you know what i mean0.94
00:39:34.600like all that kind of shit right like this is a way that we talk about parenting and it's a way0.99
00:39:38.980that a lot of parents feel which is like do not fuck with my kids also like as soon as we start0.99
00:39:43.940talking about whether a development intervention works or doesn't work we're already in the wrong0.99
00:39:48.960place something can work and not be rolled out well yeah it can work in one place and not work
00:39:55.000in another place based on like, really, really obvious, like social and political conditions.
00:40:00.340Yes. So like, one of the things that happens after this in Tanzania in 2008 is the ruling party
00:40:05.260starts blaming the opposition party and says, Oh, they're the ones that started the rumors. And then
00:40:09.880they use it as an excuse for a crackdown. Yeah. So it's like, you might want public health not
00:40:15.020to take place in a political environment. But it does. It just does. That's right. This is obviously
00:40:21.040like an extreme case. I mean, this is by far the worst case of like deworming being rolled out too
00:40:27.140fast that I was able to find in the world. But also like these kinds of problems happen pretty
00:40:32.900frequently with deworming programs because they're being rolled out so quickly. So one of the things
00:40:38.540that they don't really mention in the follow-up literature to this 2004 paper in Kenya is that
00:40:43.840in a lot of schools, the teachers aren't distributing the pills. So you can do these
00:40:48.320nasty worming programs where it's like, oh, 80 million kids or whatever. You give a thousand
00:40:52.660pills to the teachers and they just don't give them to the kids because the teachers don't know0.99
00:40:56.580why the fuck they're doing this. They're not getting paid extra to do it. They're not given0.98
00:41:00.140the context for it. And then you can go back to your donors and be like, we distributed a thousand
00:41:04.480pills in this rural village. But it's like, did they get to the kids? There's various studies that
00:41:08.980measure like the uptake and some of them it's like 20%. Jesus. And like, even in this, this drives me
00:41:16.040nuts. Even in the original 2004 study in Kenya, again, in the paper itself, they gave kids a
00:41:23.560couple rounds of pills. It was like every six months they gave them deworming. The first time
00:41:28.600they gave the kids deworming, 70% of the kids that were eligible took it. By the fourth time
00:41:33.540they're giving kids the pills, only 44% of the kids are taking them. And all of the stuff about
00:41:39.380costs, you know, the pills are only like 15 cents. So like you hear different numbers. I've heard
00:41:43.940two cents i've heard 45 cents the original 2004 paper says it cost about a buck 50 per student
00:41:49.660once you add up that like we had to give it to them every six months and then all this
00:41:53.400distributional stuff the the low cost of this intervention is based almost entirely on like
00:41:59.220pharmaceutical companies giving it away for free a lot of like volunteer labor to roll this out
00:42:04.220teachers doing this with like no sensitization all of like the low cost justification for these
00:42:10.120programs depends on the kind of like fast cheap universal rollout that caused the problems in
00:42:16.740Tanzania yeah as soon as you start slowing down to actually see if this is the right thing to do
00:42:21.020the cost savings often disappear well and also like this feels like very sort of like life hack
00:42:28.060kind of adjacent right yes it feels like doing that but with like entrenched public policy and0.91
00:42:35.180public health issues right like if someone came along and said that shit about like brexit
00:42:40.780oh we got this one cool thing you do it and then brexit we just undid it right it's just all fixed0.92
00:42:45.600people would rightly be like that doesn't seem right okay so we've done the bunking
00:42:51.060and we've done the debunking and now we have to do a little bit of re-bunking
00:42:56.520what's getting rebunked so okay so that guardian article that you read that was like deworming has
00:43:06.920been completely debunked and like it it's gone right is kind of famous in the sort of worm war
00:43:13.720spheres for being a little overstated journalists only know how to tell two stories about international0.99
00:43:19.200development it's either this is going to solve everything or this is a debunked fucking lie0.96
00:43:24.920we went through this with microcredit remember it was like cover of newsweek microcredit's0.94
00:43:29.780gonna solve poverty and then it like they expanded it again too quickly they expanded0.98
00:43:33.960to areas where it didn't work as well and then it was like microcredit's a fucking scam yeah it's0.93
00:43:37.780like no it neither one of those are true neither one of those are true and also like you only get0.94
00:43:44.720those two stories if you keep buying into the first one that's like this is gonna fix everything
00:43:49.920Exactly. And also, I mean, my takeaway from this original 2004 paper that has been dissected so
00:43:56.360much is that like, after all the sort of statistical reanalysis, the schools that got
00:44:01.720the deworming treatments have a roughly 6% higher attendance rate than the schools that didn't get
00:44:06.420deworming. To me, it's like, that is a promising result. It's not bullshit. And it's not going to
00:44:12.780save the world. It's promising. It feels like a grounded result, too. Right. If you think about
00:44:17.880your life and sort of things that have made differences in your life my guess is that it's
00:44:22.120a constellation of a lot of six percent things exactly it's the kind of study to me that it's
00:44:27.100like okay it it seemed to quote unquote work in 75 schools let's try it in another 75 schools
00:44:33.280yeah it seemed to be fine in this region of kenya let's try it next door in uganda yeah it never
00:44:38.700should have been expanded to 140 million kids the thing that i i wish that i could impress upon
00:44:45.500people in development aid is that literally nothing is going to be so good and so proven0.95
00:44:51.740that you can go from 75 schools to fucking 3 million schools on another continent yeah you're
00:44:58.440never gonna find it i'll say in public policy advocacy world which is the world that i come0.97
00:45:05.660from there is such a deep scarcity mindset yeah right that you start from a place of constraints
00:45:12.820rather than starting from a place of like okay if we really wanted to get to the bottom of
00:45:19.240school attendance what would that take instead right we start from a place of like we found
00:45:24.320this one thing and that does it and we're just gonna get a shit ton of it out there at the end
00:45:28.060and just hope for the best right right that we get to the passage or non-passage of a policy
00:45:33.820and we think that's the end of the story and actually that's exactly when the rubber hits
00:45:37.980the road right like you have to pay just as much attention to implementation the least exciting
00:45:43.220sounding thing i know but like that's where the promise of the policy comes to life right
00:45:48.300yeah and in this one it feels like folks just completely skipped over program evaluation and
00:45:55.380assessment stuff right well i mean there are places where this works there's a big study in
00:46:01.620uganda that finds that it does increase kids heights and weights and it does increase their
00:46:06.880test scores. Although it doesn't affect their school attendance, which I thought was really
00:46:10.560funny. So it's like the mirror image of the Kenya study. But it's like, okay, so it works on these0.91
00:46:16.440metrics in this place. By far the most interesting study I read on this was there's a meta-analysis
00:46:23.240of 153 deworming projects in Kenya. And the whole paper is about like, okay, what makes this work?
00:46:30.300Because every development intervention works in some places and not others.
00:46:36.060And it's not always obvious where it's going to work.
00:46:39.860Like you said, implementation, this is the hard part.
00:46:42.760And what they find in Kenya is that the specifics of the implementation really matter.
00:46:47.200So basically every deworming project that only gives kids one dose of deworming pills doesn't work.
00:46:53.640Because the deworming pills, they kill all the worms that are inside of you.
00:46:56.760It's just like napalms your entire digestive system.
00:46:58.960but it doesn't prevent you from getting reinfected so if you only go there and give
00:47:03.700kids one round of pills and you come back in six months and they all have worms again yeah and
00:47:07.380it really only works in areas where there's like really high infection levels because again
00:47:12.440if you have like a light or moderate worm infection like it sounds really gross to think
00:47:16.860about but like it doesn't it often doesn't cause any symptoms like the worm just like lives in your
00:47:21.020body for a year and then it dies and you poop it out and that's it so unless you're dealing with
00:47:25.320an area where there's like large numbers of heavy infections like in the original study
00:47:29.160it was like a 22 percent heavy infection rate like that's exactly the kind of place where you should
00:47:35.220be doing deworming like that's that's really debilitating for kids yeah but if it's in a
00:47:39.220place where like you know 10 percent of people have worm infections only maybe one percent of
00:47:43.580people have heavy worm infections you don't want to give pills to every single kid for that right
00:47:47.880it's like giving every single kid a tylenol for one kid having a headache yeah another thing they
00:47:52.080find in kenya is that it works much better when it's coupled with other interventions so when
00:47:57.340you're giving kids like these deworming pills even if they do have heavy infections the worms are
00:48:01.860gone but then they need extra food to kind of get like catch-up growth because oftentimes they've
00:48:07.260had like stunted growth and stunted weight so it's like they need to eat more for like a couple
00:48:11.600months after that to like catch up to where they should be but if these are really poor kids and
00:48:16.040they're in a really poor area and they don't have access to food right good luck yeah like it it's
00:48:20.380kind of like there's no point in deworming them because whatever benefits they would get from
00:48:23.060deworming they they want to get because there's not enough food around for them to eat so it's
00:48:26.500like you also have to do these nutritional interventions and in general deworming works
00:48:30.380a lot better when it's coupled with this like much more holistic program about like sanitation
00:48:36.140like getting people running water also some education programs are part of this i'm skeptical
00:48:40.520of those in general but apparently they work here in some circumstances so it's like deworming as
00:48:46.180part of a larger program of like we're going to work on improved sanitation in this area then like0.98
00:48:51.600yeah they're a really essential component of that because you need to get the fucking0.87
00:48:54.560worms out of the kids yeah but then you're back to an expensive project and you're back to a project0.94
00:49:00.740where like you have to actually know the local area and you have to talk to people beforehand
00:49:03.780and you have to build relationships and donors don't like that and like the the ted talk industrial
00:49:09.460complex davos they don't like that either i mean i think part of the reason that that stuff feels
00:49:15.820so frustrating to folks is that it feels so slow compared to i got a million pills out right what i
00:49:22.180would argue is that building something slowly and right and making sure that it works actually
00:49:28.700probably takes less time than going on five of these wild goose chase scenarios jesus christ
00:49:35.960everybody thinks we've found the silver bullet for whatever it's the like public policy and
00:49:41.680international aid equivalent of a get rich quick scheme oh it is i mean i mean it literally is for
00:49:46.700the developing world like get rich quick yeah yes yes the the most chilling thing to me about this
00:49:51.840is the way that this generation of development aid has replicated exactly the mistakes of the
00:49:59.040previous generation boo say more about that so when michael kramer got into development during
00:50:04.160the statistical dark ages right everything was kind of like gut based vibe based right and they've now
00:50:11.060created a situation where they're basically saying like we're just going to ignore all of the data
00:50:16.940the world health organization is still doing mass deworming the world bank still stands behind this
00:50:24.460the copenhagen consensus the give well charity which is like a group that like i really respect
00:50:31.020has written these weird defenses of deworming where they're like no no no ignore all the meta
00:50:37.320analyses what we're really interested in is like the 10-year follow-up study we know that deworming
00:50:43.540works because of the 10-year follow-up study but it's now it's like there's no there's no plausible
00:50:47.740path to them having better lives like it's it's completely deranged to say that a program of
00:50:54.780deworming in which like up to half the kids didn't even participate like didn't even get the deworming
00:50:59.600pills and had no effect on health no effect on test scores and an extremely modest effect on
00:51:05.280attendance would be affecting people's earnings 10 years later i'm sorry yeah it isn't yeah
00:51:10.460plausible to me and also it's very weird for this entire field of development the whole thing was
00:51:16.380like the randomistas that's what they called this like this generation of mostly economists who came
00:51:22.140into development aid and were like let's do this in a more rigorous way that now these same
00:51:26.960randomistas these like rigorous people are now telling me to ignore meta analyses on the basis
00:51:33.360of a single study like that is so chilling to me it's like what was the point of all this yeah i0.85
00:51:39.480mean imagine being like definitely don't pay attention to a fucking cochran review right this0.61
00:51:46.660is okay this is this is what michael kramer like during the whole kind of worm war everybody0.96
00:51:50.860fighting on the internet period in 2015 this is in a sort of response to the response to the response0.54
00:51:55.660to the response like this whole thing is like fucking tumblr comments in like 2014 he's talking
00:52:00.780about like defending deworming. He says the estimated value of benefits exceeds the cost0.91
00:52:05.320by more than a hundredfold. While the results of any one study should not be taken in isolation
00:52:09.880and effects may differ across environments, even if policymakers believe there's a one in 100
00:52:15.440chance of experiencing effects of this magnitude, the expected benefits of deworming would exceed
00:52:21.560the costs. They're playing our song, Mike. He's basically saying like, even if this doesn't work,
00:52:26.500look how cheap it is look how cost effective this is this is like the asbestos argument
00:52:32.720givewell says the same thing givewell says because mass deworming is so cheap there's a good case for
00:52:38.440donating to support deworming even when in substantial doubt about the evidence this is
00:52:43.620very similar to some of the responses to the calorie labeling stuff oh yeah yeah absolutely
00:52:50.200that people are like well it doesn't work for everybody but look how cheap and effective
00:52:54.220right you're like effective measured by what right effective how we're doing something we're
00:52:58.520not doing nothing but also the something that you're doing has the effect of nothing and also
00:53:03.980plus maybe negatives like what i also uh i forgot to mention uh michael kramer won the nobel prize
00:53:10.620in 2019 come on michael it's not just for this it's for like other work and it's for like this
00:53:16.800whole kind of the the systematization of statistics within international development
00:53:21.600And he shared it with some other economists.
00:53:23.360And, like, I actually think that, like, using more randomized controlled trials in development is a good thing.
00:53:28.800I think statistical rigor is a good thing.
00:53:30.960But also, like, that has come with drawbacks and side effects and a lot of, like, weird and bad cul-de-sacs.
00:53:37.920I also think to sort of, like, defend the original 2004 paper, there's not going to be a perfect methodology ever.