00:09:40.600But an example like Portia Woodruff, falsely arrested due to AI-powered facial recognition. She was eight months pregnant, sitting in a jail cell, having contractions for a crime. Now she was being held for a crime she didn't commit.0.99
00:09:58.260And so when you hear those stories, the stories of who I like to call the X-coded, you start to pay attention, right? Or maybe it's your kid and they got flagged for cheating. Turns out they didn't actually cheat. English is their second language, but some chat GPT detection system, right, is flagging them as cheating.
00:10:19.720And so I do think those stories are what helps people see that this is a conversation that requires their voice.
00:10:29.660And it's so easy to think it's like I have a Ph.D. from MIT, but I was doing all this before.
00:10:34.600Right. You don't have to have this type of in-depth technical background to have a voice and to have an important perspective, because if you know you're being harmed, that's enough.
00:10:45.600Yeah, a big part of what we aim to do here is to help people understand that you might not be an engineer, you might not have a doctorate, but you are the expert of your experience and you use this technology every day or it's being used on you.
00:10:58.940And so you innately have a perspective that is valuable and worth sharing and worth hearing and worth centering about how that technology has impacted you.
00:13:13.460People who are traditionally marginalized, like women and Black women, are made invisible by technology every single day.
00:13:20.880We're faced with silencing, erasure, and hostility.
00:13:24.100So is that one of the reasons why the technology that these spaces build also can't really see us?
00:13:30.620Do you ever feel, I mean, stay with me here.
00:13:33.420I think that there is like a general hostility toward marginalized people, like toward Black women, and I think in technology.
00:13:42.720And I sometimes feel that the technology that is being made in turn mirrors that same hostility, mirrors that same erasure.
00:13:50.400And so because these facial recognition is not being tested or trained on enough diverse data sets or whatever, it in turn erases us.
00:14:00.020Do you feel that that is kind of because of this underlying hostility toward people who are traditionally marginalized in the space?
00:14:10.400I actually don't think it's an intentional underlying hostility, which makes it even more dangerous.
00:14:19.520So well-intentioned people, right? Collecting data, doing what they think is good science, good machine learning can still create harmful systems. And this is what I learned after we did different audits and I would go talk to the teams behind some of these systems, right?
00:14:40.400They were nice people. You know, try to send the kid to college. And it was actually interesting to me because as much as I was wanting to humanize the people who are X coded, who are harmed by AI systems, part of what I try to do in the book as well is also humanize the people who are creating the systems and where things go wrong.
00:15:03.000But to your greater point, it's not an intentional hostility. Sometimes it is a profound and harmful ignorance to not even think to ask certain questions or to test the system in particular ways. That's part of what the research was about.
00:15:21.440We asked, what happens when we put an intersectional lens on the way in which we analyze the performance of AI systems?
00:15:30.180And just doing that, right, opened up new areas of conversations where before people would just look at the overall score.
00:15:38.460And that gave us a false sense of progress within the space because we were testing these systems on benchmark data sets to see how well they do.
00:15:48.320And then you would look at the benchmarks. Some of the benchmarks would be over 80% lighter skinned individuals, over 70%, you know, people identified as men. And if that is your benchmark of success, you're already not going to see how you're failing.
00:16:05.880And so when we created a more inclusive data set, et cetera, it allowed us to see that the promises of potentially well-intentioned people weren't even panning out.
00:16:18.080But there's even more to that, because I think with some of these conversations, it can seem like it's a very technical problem with a technical solution.
00:16:29.280Data didn't detect, you know, system didn't detect the face, make it more inclusive, whatever else it is.
00:16:35.880But the problem is accurate systems can be abused. We've seen facial recognition systems deployed at protests, right, which we know can lead to chilling effects if you know you're under surveillance for daring to exercise your First Amendment rights to say this is not correct.
00:16:58.360accurate systems, create tools for state surveillance. So yes, you can say, well,
00:17:04.720my phone tracks me than the other. You can leave your phone at home. Your face is a little bit
00:17:10.780harder. I mean, you know, some people do put on a face, but you know what I mean. You know what I
00:17:16.720mean, right? So I think it's important to understand that even when we have conversations
00:17:23.680about the accuracy of certain systems, and we should have those conversations. Accuracy is not
00:17:30.520enough to assure accountability or equity. Now, when we're talking about accountability,
00:17:36.160especially from tech companies, it is so easy to get caught in a cycle of name and shame,
00:17:41.460where you point out all the bad things that a specific company has done. And if I'm being
00:17:45.940honest, I might have done that a time or two on this very podcast. But Dr. Blumwini describes
00:17:51.800their method as less name and shame and more name and change. They want to show companies what
00:17:57.120they're doing wrong so they can change for the better. But this hasn't always meant that those
00:18:01.300companies don't lash out when her teams point out the harm that they've caused. I'm curious,
00:18:07.360how have companies, I won't say any names, but companies who you have called out in your research
00:18:14.400or, you know, said like, this is, hey, this is what's going on. How have they responded to your
00:18:19.020findings. So overall, I take a name and change approach. So the point of pointing out what's
00:18:27.180wrong isn't to shame a company is to say we can do better. Right. And sometimes we have companies
00:18:36.520that are reactive, we have companies that are proactive, and we have companies that are
00:18:42.320combative. With the first set of research results that we released, we saw more of the reactive
00:18:49.920stance, which is, oh, now that there's a headline, right? We're going to go, we are on the problem,
00:18:58.220or we were already working on the problem. There are different ways. But now it's a priority
00:19:05.920because it's making headlines. So I saw that, and the reactive approach tended to be a technical
00:19:12.540approach, which is, okay, there were these disparities, so let's close them. We now have
00:19:17.880more accurate XYZ. Again, accurate systems can be of use. Then we did experience some combative
00:19:26.820responses, right? So here we had a huge tech company coming out and saying,
00:19:34.100your research is misleading, attempting to discredit the research. At that time,
00:19:41.220I was a graduate student. And I was so fortunate that I had senior scholars and people well
00:19:48.440respected in the AI industry who came to our defense, cheering prize winner, somebody who was
00:19:55.180literally the chief AI scientist at that company, saying what the research shows warrants our
00:20:04.100attention. And this is research we should be elevating, not dismissing, because it makes
00:20:11.160the field better as a whole. If we can acknowledge our limitations, understand what's going wrong,
00:20:18.100so we can build more robust systems. Because this doesn't just deal with faces, right? If you want
00:20:23.020to use computer vision to help, let's say, with medical diagnoses. You want to make sure you
00:20:28.600understand where things can go wrong so we can course correct for things to go right. So we had
00:20:34.420the combative approach, the reactive approach. But the approach I appreciate most is the proactive
00:20:40.880approach. Okay, we've heard there's some issues. Instead of waiting for someone to drop the paper
00:20:48.020or the headline, what can we be doing as a company now? And I've had the opportunity to
00:20:54.780work with Procter & Gamble with Olay on the Decode the Bias campaign. And when they came to me and
00:21:02.020they asked for an algorithmic audit, I said, given what you've described your tool does,
00:21:07.780and this was a tool that would analyze your skin and give you product recommendations,
00:21:11.580And how you train the tool on a set of data. I suspect if we dig in there, we're going to find some bias. They're like, that's okay. If you find bias, we'll do what we can to correct it. I was like, and if you can't correct it, we'll shut the system down.
00:21:26.980I was like, can I get this in writing?
00:21:46.440We did, in fact, find bias as we thought would be there.
00:21:51.020And they actually, in the proactive, not only did they seek to be audited, they also agreed to a consented data promise.
00:22:01.680And this was inspired by their skin promise.
00:22:04.700So when I first started working with Olay, I was excited.
00:22:07.620And then they told me, you know, when we do the campaigns, there won't be any post-production airbrushing.
00:22:14.660What we capture is what we'll show, right?
00:22:19.240you know truth in advertising you want to think about people's body image and all of that and
00:22:25.440I'll be honest I was a little disappointed because like if you have you just want to know you could
00:22:31.180be saved by the airbrush right but because of that promise which is a good promise I get where
00:22:39.480they're coming from as the person on the other side of the 4k camera in your face you're asking
00:22:46.140can we consider xyz but what i appreciated about that is it made me even more disciplined with my
00:22:54.540actual uh skincare uh regimen and i also drank water and i i did all the right things for vanity
00:23:03.280reasons i won't laugh i did the right things for vanity reasons but i i think about that with um
00:23:11.500So that skincare promise, that was part of the inspiration for the consented data promise. And just like when you make a promise and it's a public commitment, that's the important part. Now there's a little bit of accountability, right?
00:23:27.900So now you are going to bed early. Now you are drinking more water. Now you are exercising five days of the week, which might not have been the case before you made that public. So that's an example of more proactive. So from the reactive to the combative to the proactive, we've seen it all.
00:23:47.760I think what I learned most from the combative response was how much of a risk I was taking as a young researcher to not only do the research, but to name the companies, and I'll name them now, right, that I tested IBM, that I tested Microsoft, that I tested Amazon.
00:24:12.400And because of their power, that meant I was risking future opportunities.
00:24:21.780And also other researchers watching how I was treated, how my co-authors were treated, people like Dr. Gebru, they were also getting a sense of what is possible.
00:24:33.540When I look at research papers now, where people openly talk about algorithmic bias and algorithmic harms and people openly name the AI models or the tech companies, that wasn't always the case, right?
00:24:49.240A price was paid for this more robust conversation to happen.
00:24:57.780Dr. Blomini is right. This all comes at a cost.
00:25:01.320Dr. Temnek Gebru, who she mentioned earlier,
00:25:04.400was a co-author on Dr. Blomini's gender shades research.
00:25:07.880Dr. Gebru was once the technical co-lead of the Ethical Artificial Intelligence team at Google.
00:25:13.040While in that role, she worked on a paper about the risks of large language models,
00:31:36.440Even though Amazon tried publicly discrediting Dr. Blomini's work calling out the harms of their facial recognition technology,
00:31:42.700in the end, they conceded that technology wasn't exactly safe.
00:31:46.680In 2020, they announced a pause on allowing police to use the technology and eventually extended that pause indefinitely.
00:31:54.880And correct me if I'm wrong, but your work ended up with Amazon rolling back some of the uses of their faulty facial recognition technology.
00:32:04.600So ultimately, not only were you obviously vindicated, but that work went on to create a somewhat safer landscape for everyone because Amazon had to step back and be like, OK, wait a minute, this technology maybe isn't really working that well.
00:32:19.300They would not put it in those terms, but they did take other steps.
00:32:25.020So I will say before Amazon, IBM actually said we are no longer going to sell to police departments.
00:32:34.600And this was in 2020. Right. So we also had the murder of George Floyd happening at that time.
00:32:41.640And Microsoft said, we will not sell this until regulations are in place. And then Amazon came third, you know, and they said, we'll halt it for a year.
00:32:53.120And then they extended that halt. Right. But this is to say there was an acknowledgement, right, of the risk and the harms.
00:33:01.940Was it just risk and harms to people or risk and harms to the company's reputations? It could be a combination of both.
00:33:10.000I'm excited to share that this research led to numerous cities incorporating some of the findings in their analysis and in their statements for why they chose to enact certain laws that restrict police use of facial recognition.
00:33:32.380It also changed the conversation at the national and international level. The EU AI Act actually has a provision that would prevent the use of live facial recognition in public spaces.
00:33:48.740When I spoke with President Biden at the AI roundtable some time ago, this was top of mind. I shared the story of Robert Williams being wrongfully arrested in front of his two daughters. We talked about racial bias in AI systems and other types of harms that can impact many people because no one, trust me, no one is immune. This isn't just other people's problems.
00:34:14.560And so to see the reach of that work certainly made all of the combative, you know, responses and things like that somehow worth it.
00:34:29.260And it really goes back to what you were saying earlier about how systems don't have to be biased to be misused. I don't know. I want to believe in a tech landscape where companies with so much power don't have to wait until something goes wrong, don't have to wait until somebody is wrongfully arrested, don't have to wait until they're called out to make things a little bit safer and more equitable.
00:34:53.540Do you believe that is possible where companies aren't just reacting, they're actually being proactive at wanting the technology that they deploy on all of us to be more equitable?
00:35:04.800I think that, again, you do see companies taking on the mantle of responsible AI. You'll have other companies like Credo.ai that will have services that are meant to help companies adopting AI systems do it within a responsible way.
00:35:23.080you'll see companies hiring responsible AI leads, right? So I definitely think there is an intention
00:35:30.360there. Where I still push back a bit is self-regulation is always self-interested.
00:35:39.560Not surprising. So I do think real accountability requires external accountability. And the other
00:35:47.120part that I don't really see companies focused on so much is redress. So there's a lot of
00:35:53.620conversation about being responsible in terms of preventing future harms. But what about those who
00:35:59.500have been harmed already? And I do think algorithmic redress is oftentimes missing from
00:36:05.960this conversation of responsible AI. So when I see the companies stepping out to say, and we're
00:36:13.500doing redress, I might be convinced. I haven't seen it yet, though. Prove me wrong. Prove me
00:36:19.240wrong. I want to be wrong. Well, as somebody at the helm of an organization fighting for
00:36:24.740algorithmic justice, what does justice look like? Justice looks like you live in a world where data
00:36:30.580is not destiny, where your hue isn't a cue to dismiss your humanity, where you actually have
00:36:37.520data rights and you can consent to how your information is used. Justice looks like how do
00:36:45.620we use these tools in a joyful, uplifting manner versus just being reactive to the next harm.
00:36:53.480So when I think of social justice, you can't have social justice without algorithmic justice,
00:36:59.660because if you're saying we're pushing for gender equality, you have an AI system that cuts out0.93
00:37:06.060women's resumes. We didn't quite make it, right? You can't necessarily say, oh, we have racial
00:37:12.460equality. And then you're adopting biased facial recognition that's putting, so far,
00:37:19.500the folks I've seen have all been dark-skinned like us, you know, into prison due to misidentification.
00:37:27.200And so for me, right, algorithmic justice is truly being in that place where we can be our
00:37:34.640full selves and not be targeted, right? Or algorithmically placed as other, algorithmically
00:37:43.160erased, algorithmically exploited. And so that's the world we fight for, right? We say free the
00:37:49.780X-coded. And so this is algorithmic justice. The book is Unmasking AI, Your Namesake Joy.0.88
00:37:58.220I have to tell you, you are such a joyful person. Speaking to you about this work is,
00:38:03.180It just comes through how much you care and how, I don't know, I have a lot of conversations about tech that are hard and dark and grim, and your work is just so the opposite. It asks, what if? What's possible? What can we do? How can things be better?
00:38:19.640Like, it's just really nice to see somebody leading the way with such a joyful but justice-rooted perspective.