Dreaming Against the Machine - Episode 24： The Myth of AI Doom, with Cal Newport

00:00:00.000 Welcome back to Dreaming Against the Machine. I'm your host, Adam Becker. This week's guest

00:00:07.100 is Cal Newport. Cal is a professor of computer science at Georgetown University, and he's

00:00:13.480 also the author of quite a few books, more books than I thought he was the author of,

00:00:17.860 as I find out during this conversation. He's probably best known for his book, Deep Work,

00:00:24.040 and I thought we were going to get to talk about that during this conversation, but that

00:00:28.760 is not what happened. We got caught up talking about AI doomers and AGI cultists who are pervasive

00:00:38.960 throughout Silicon Valley's AI industry. And the reason we ended up talking about that is

00:00:44.960 also the same reason that I know Cal. Cal read my book, More Everything Forever, and then wrote

00:00:51.400 an op-ed for the New York Times that came out earlier this month about the influence that ideas

00:00:58.560 like rationalism and effective altruism and all these other ideologies of technological salvation

00:01:03.460 that I wrote about in my book, the influence that these things have on the AI industry in Silicon

00:01:08.340 Valley. And, you know, he's a computer scientist, so he has a interesting and unique perspective on

00:01:14.040 it. Then a few days after that came out in the Times, this guy Coxon left Anthropic and said

00:01:22.800 he thinks that there's a good chance that what they're building there is going to destroy all

00:01:28.400 of humanity, and he quoted major rationalists in his justification for this. And then other

00:01:33.440 people at Anthropik said, oh, yeah, yeah, we agree with him. And the whole thing got kind of weird.

00:01:39.380 So Cal and I decided to have a conversation about this. And yeah, okay, we didn't get to

00:01:45.800 talk about deep work, but that just means we're going to have to bring him back. So here's Cal

00:01:49.840 Newport. Cal, welcome to Dreaming Against the Machine. Well, thanks for having me, Adam. I

00:02:00.180 got a lot to talk about. Yeah. Well, let's introduce you for our listeners. You are a

00:02:06.140 professor of computer science at Georgetown, yeah? Yep, that's right. I'm a computer scientist

00:02:12.140 trained originally in distributed system theory, but have morphed over the years into also

00:02:19.820 I'm also a digital ethicist, technology critic, someone who grapples with technology and its impact.

00:02:25.360 So I've got a foot in both the worlds of building the technology and then worrying about the technology that we build.

00:02:32.000 No, that sounds like a good recipe for an anxiety disorder.

00:02:35.000 So, you know, good on you.

00:02:36.980 I'm anxious all the time.

00:02:38.160 Yeah.

00:02:38.680 Yeah.

00:02:39.320 And you've also written, what, two, three books?

00:02:42.940 Eight. 1.00

00:02:43.840 Oh, shit. 0.99

00:02:44.540 Coming out in Mars. 1.00

00:02:45.200 Yeah, I've been at it for a while.

00:02:46.840 Whoops.

00:02:47.880 Sorry. 1.00

00:02:50.820 Damn. 0.99

00:02:54.960 I started young, Adam. 1.00

00:02:56.520 That's how that works out.

00:02:57.620 Yeah.

00:02:58.200 I was going to ask.

00:02:59.600 You must write your books pretty fast.

00:03:01.440 I wrote my first one as an undergrad.

00:03:03.020 So it's still a matter of aggregation over time.

00:03:06.500 That's the secret for everything.

00:03:07.840 Wow.

00:03:08.460 Yeah.

00:03:09.000 No, I think that if I had tried to write my first book when I was an undergrad, I would have died. 0.90

00:03:14.560 Well, let me assure you, my first book was exceedingly dumber than your first book.

00:03:19.820 So this is, you know, my first books I wrote were student advice guides.

00:03:22.920 So that helps as well.

00:03:24.660 Let's make that clear.

00:03:25.740 Okay.

00:03:26.320 Yeah.

00:03:26.580 No, this is, this is good.

00:03:28.000 Yeah.

00:03:28.180 This is Dreaming Against the Machine, where my guests make me feel better about being

00:03:31.760 less accomplished than they are.

00:03:33.240 But yeah, the book of yours that I think is best known is Deep Work, which I definitely

00:03:41.680 want to talk about that today as well.

00:03:43.740 But the reason we got in contact, you reached out to me because you were working on an op-ed

00:03:48.280 for the new york times which came out what a week or two ago now yeah yeah i think it's been a

00:03:53.620 maybe three weeks by the time this comes out you wanted me to check some facts uh that you'd drawn

00:04:01.640 from uh my my second and uh second of my two books uh more everything forever and i mean i think it's

00:04:11.120 a great op-ed and everybody should go read it you were making the case that we should all be pretty

00:04:16.320 angry about the AI, I don't know what to call it, cult that the leaders of the tech industry

00:04:25.280 have all subscribed to.

00:04:26.840 I mean, or at the very least, take into account.

00:04:30.500 I mean, it's interesting, everything that's going on with AI, it's really been an interesting

00:04:36.740 period if you're sort of in my circles, which I think of as like the East Coast computer

00:04:40.960 scientists.

00:04:41.600 And I say East Coast really just to mean computer scientists who really have very little

00:04:45.760 philosophical or ideological connection to the the currents out in the bay area and silicon valley

00:04:52.820 and it's definitely a confusing time for us because on the one hand there's specific

00:04:57.500 technological stories that are happening that are that are interesting or exciting in a lot of ways

00:05:01.700 worrisome as well and then there's this rhetoric that gets layered over it that's very confusing to

00:05:06.900 us bitter east coasters right we don't we don't understand it it tends to be when you first

00:05:11.880 encounter it as a computer scientist very non-technical it tends to be very abstract it

00:05:18.540 tends to be very thought experimenty like you would get in a sort of utilitarian philosophy

00:05:25.020 class where you're thinking through scenarios as thought experiments about applying ethical

00:05:29.920 principles and it feels really divorced from the kind of hands on the circuit board sort of

00:05:35.480 building tools and safety and control and concerns and you know the more I start pushing into it the

00:05:41.560 more i begin to understand just like oh there's actually schools of thought that are very

00:05:46.460 influential in these industries and it affects the way they talk about things and so you know

00:05:51.920 i've been out there trying to tell people like hey we need to we need to strip some of the way

00:05:55.880 people talk about these things out of the conversation to understand what's going on

00:05:59.340 and that's been actually like a very difficult conversation to have especially because uh

00:06:05.660 twitter has balkanized all conversation on ai right now and it's very it's very partisan

00:06:11.440 and it's very team-based.

00:06:12.720 And if you come in and say,

00:06:13.940 there's some weird ways

00:06:15.680 that these people are talking about the technology.

00:06:17.460 Let's put that aside

00:06:18.120 so we can talk about the technology.

00:06:19.620 It will get taken as,

00:06:21.620 okay, so you don't think AI works.

00:06:24.040 So then if AI does something impressive,

00:06:26.560 you're like, well, I guess you were wrong, right?

00:06:28.220 It's this sort of interesting, weird reaction.

00:06:29.760 So I wrote that op-ed.

00:06:30.620 They'd be like, look, let me just walk through.

00:06:32.500 I want to show, not tell.

00:06:34.000 Let me just like really walk through

00:06:35.640 a particular family of ideologies

00:06:38.020 that I learned a lot about from your book,

00:06:39.940 rationalism being the root and then it sort of kind of grows into an effects effect of altruism

00:06:45.840 and some other fields as well let me talk about the connections of those philosophies those

00:06:50.460 ideologies to key figures in the ai industry and let me make the argument that it's affecting the

00:06:56.300 way they talk about things and we should at least know this yeah and so it comes out and then a few

00:07:00.860 days later anthropic you know multiple employers like yeah we're going to kill you all uh what's 0.97

00:07:06.660 a big deal. And it was like speech that was right out of the playbook of exactly these groups I was

00:07:12.280 talking about. And so in some sense, maybe the timing was great or in some sense, maybe the

00:07:16.480 timing was wrong because everyone was just like, oh, I guess they're right. So I don't know that

00:07:20.780 I wanted to at least get it in the record that there is these ideological influences that need

00:07:29.060 to be understood when you're trying to understand what's going on. Well, you know that I think that

00:07:33.880 that's correct because that's a lot of what i talked about in my book but it was really striking

00:07:39.060 to me to see these you know anthropic people and that guy uh coxson come out i mean we were you and

00:07:45.820 i were just talking about this we were recording this the day after you and i recorded our episode

00:07:51.500 with um at zitron which uh if everything goes according to schedule that came out a week ago

00:08:00.060 compared to when this one's going to come out um but like we were saying yesterday i don't want to

00:08:05.700 call that guy a whistleblower because whistleblowing you know is is generally something

00:08:10.300 that uh you know involves taking a courageous stand and talking about you know what's going

00:08:17.980 on at at a company that you work at or very recently worked at that the other people there

00:08:22.300 are not willing to go public with and so the response to a whistleblower is generally you

00:08:27.440 know, either from the company that they used to work at or organization is either silence

00:08:32.020 or no, no, of course, all of that's not true.

00:08:34.460 And instead, you know, this guy comes out and everyone in his company is like, yeah,

00:08:38.220 yeah, totally.

00:08:38.940 Yeah.

00:08:39.080 That's not controversial.

00:08:40.140 We, we believe that too.

00:08:41.680 And it's like, okay, this is, this is not whistleblowing.

00:08:44.460 This is something else.

00:08:46.760 But, um, you know, he went, uh, he went and got interviewed by Wired and just quoted

00:08:52.360 eliezer yudkowski at them the head uh and and original uh you know founder of of the rationalist

00:09:00.160 i mean it's it's interesting to me i think the reaction to that week is interesting right that

00:09:07.200 and i don't understand by interesting i mean confounding because i don't understand the

00:09:11.340 exceptionalism necessarily that's being applied to this particular industry but i don't understand

00:09:15.400 how you can come out and say we might kill billions of people and then your boss comes out and says

00:09:21.660 Yeah, I think he's right. 1.00

00:09:22.860 We'll probably kill billions of people. 0.93

00:09:24.120 And then the head of your company comes out and is like, yeah, unless you put in place, 0.99

00:09:28.960 you know, our framework for regulation, we'll probably kill billions of people.

00:09:33.220 How you don't immediately, you know, have that the agents are at the company.

00:09:39.480 And I mean, the old fashioned human agents, right?

00:09:41.900 They're like, okay, clearly something is awry here.

00:09:45.540 Everything has to stop.

00:09:47.520 We're turning on until you can, and we're going to haul you in front of whatever bodies

00:09:51.020 we have to haul in front of and say, whoa, whoa, whoa, whoa, whoa. We are going to grill you as

00:09:55.640 long as we need to understand exactly how and why you think you're going to kill billions of people.

00:10:00.800 We need to know what is going on. We need to know if you should be in Arkham Asylum or if you're

00:10:05.180 actually building a weapon of mass destruction. But certainly you don't get to just keep doing

00:10:09.420 what you're doing. That is alarming because either you have a mental disorder or you're doing 0.99

00:10:15.080 something that is wildly disordering for like the history of civilization. But somehow it's like, 0.99

00:10:19.980 well what can we do yeah yeah you know it might just kill everyone we're gonna they got to just

00:10:24.400 keep going and like maybe we'll have some debates on like some regulation and china you know i don't 0.99

00:10:29.120 know like what if we can't let china kill everyone first we got to be the it's this it's a weird sort 0.97

00:10:33.360 of exceptionality um that should be getting a huge amount of attention but instead it's sort of like 0.99

00:10:41.640 well they're the experts i guess that's true and they know best it's this sort of weird uh seeding

00:10:47.140 of moral authority to the odd kids,

00:10:53.060 the kids that are playing the role-playing games

00:10:56.060 at the back of the lunchroom

00:10:57.080 instead of playing football or something like that.

00:10:58.680 Like, what is going on in this scenario?

00:11:01.100 So it's a little bit puzzling to me.

00:11:02.920 I don't want to rag on the kids

00:11:03.960 who are playing the role-playing games

00:11:05.140 instead of playing football.

00:11:06.300 I was. By the way, I was.

00:11:08.260 Yeah, exactly. Yeah, same.

00:11:09.760 I mean, I was, well,

00:11:10.580 I was one of the drama club nerds,

00:11:12.080 but yeah, you know, spiritually, same thing

00:11:14.440 and definitely played my share of role-playing games.

00:11:17.140 Um, but you don't, but they wouldn't have let us set the rules for the school because

00:11:22.140 it was, that's true.

00:11:24.040 Yeah.

00:11:24.480 There would have been wizards involved.

00:11:26.080 Yes.

00:11:27.600 I mean, to me, these are people who, who seem to confuse like their sci-fi role-playing

00:11:33.440 game with reality.

00:11:34.640 But, uh, I, I want to come back to that point you just made about, uh, you know, people

00:11:39.780 treating these guys as authorities because, you know, you're a computer scientist and

00:11:44.000 you were talking, you know, just a minute ago about, you know, the, the East coast computer

00:11:49.440 scientists, the ones who are, you know, disconnected from Silicon Valley, but, you know, still here in

00:11:55.040 the U S that's not a community that I'm part of. And also I'm not a computer scientist. My, you

00:12:01.000 know, my technical expertise is in physics and my authority on this subject comes, you know, from

00:12:06.280 the research I did to write my book, but like, you know, you, you are a computer scientist and you

00:12:11.500 spend time hanging out with computer scientists what does that community you know the computer

00:12:15.580 scientists who are not linked directly to the tech industry out here what's their response to this

00:12:24.780 how do they how do they feel about this i think the confusion that a lot of us have is the lack

00:12:30.380 of technical specificity right yeah now if you actually study rationalism effective altruism

00:12:36.360 the sort of family of what you call technological salvation ideologies you realize like this is the

00:12:40.300 way that they generally talk about these issues is unfalsifiable um and thought experiment style

00:12:45.540 right yeah um it's usually what you're dealing with is a thought experiment about a uh a proposed

00:12:51.420 hypothetical super powerful being right you sort of posit that existence and then you work through

00:12:58.120 thought experiments about how such a being might react under certain types of conditions

00:13:02.700 right so like what if that being was like trying to do what you told it to do but was not matching

00:13:10.940 your normative values of like human life and ends up killing a lot of people to try to get to your

00:13:14.800 goal or what if that being is tricking you into thinking that so it's all thought experiments

00:13:19.020 about a hypothetical proposed being um and the only technical component to their they're speaking

00:13:25.300 about ai is is recursive self-improvement which is a sort of an engineering get out of jail free

00:13:30.560 card or you say i don't know how to build something like this but it'll improve itself

00:13:34.140 till it gets there it's an idea that's been around since the 1960s and it's it was really

00:13:38.440 grabbed on by the futurists especially in the 1980s and onwards because um you don't have to

00:13:43.960 deal with specific falsifiable test technical claims you just say it'll improve itself right

00:13:49.760 it's a so again it's it's all kind of abstract but for a lot of people the computer science

00:13:53.760 community like we're used to dealing with like actual systems or okay this is what we're worried

00:13:59.960 about. If you build this type of agentic framework that is powered by this type of LLM and using this

00:14:04.500 type of system here, then this could behave in this type of ways. And there's that actual type

00:14:09.540 of behavioral loop that we worry about. We tend to be very specific, but the technological

00:14:13.660 salvation ideologists don't like to be very specific because also it's too easy for people

00:14:18.300 to say like, well, that's not going to work or we could just turn that off. So it's better to be

00:14:21.400 sort of non-falsifiable. And that's really my main critique about the current moment, right?

00:14:25.940 is that if I was in charge of sort of doing a congressional investigation

00:14:31.320 of the leaders of the particular firms that are connected to these ideologies,

00:14:35.260 I would say we're going to get specific.

00:14:38.040 I don't want to hear about AI in some general sense.

00:14:40.300 AI is getting out of control.

00:14:41.660 AI needs to be slowed down.

00:14:42.880 I don't know what that means.

00:14:44.640 Do we have to stop using Tesla self-driving cars?

00:14:47.300 Can I use Stockfish to play chess or do I have to slow that down?

00:14:50.260 Like that's too general.

00:14:51.200 AI means too many things, right?

00:14:53.400 Well, they'll say, well, if you look at like Amadei's statement, they talk a lot about

00:14:57.840 models.

00:14:58.620 I think model has become their rhetorical stand in for the sort of hypothesized being

00:15:05.300 of some sort of like unconstrained power, but I'd push further.

00:15:08.720 So you mean a large language model, I guess, because that's your industry.

00:15:11.800 Well, a large language model by itself doesn't do anything.

00:15:14.460 It's just a bunch of matrices.

00:15:15.700 Now, if you actually turn to crank, right, you get a token out, but that's not scary.

00:15:20.280 So it must be some sort of system instantiating an LLM to get tokens out of it in some sort of way.

00:15:26.780 Well, what system?

00:15:28.000 Is it chatbots, right?

00:15:29.500 So they auto-aggressively produce tokens until you have a full response to a prompt and they send it to a web interface.

00:15:34.440 Are we worried about a chatbot going out of control?

00:15:37.660 No, probably not.

00:15:38.640 Okay, so are we talking then about an agentic framework where you have a program that prompts an LLM and says,

00:15:46.380 here are my capabilities, here's my goal, what should I do next?

00:15:49.640 The LLM returns a text response.

00:15:52.260 It then parses that, executes it in loops.

00:15:54.560 Well, it can't just be that

00:15:55.960 because millions of programmers every day

00:15:57.960 use very sophisticated agents to do software development.

00:16:01.340 And there's been zero problems.

00:16:02.820 Those agents don't go rogue.

00:16:04.200 They don't coordinate.

00:16:05.160 They don't go off and do their own thing.

00:16:07.060 So wait, what are we talking about?

00:16:08.220 If you really kept pushing, where are we going to get?

00:16:10.340 And I think where we would get is,

00:16:12.740 all right, there's this like very narrow type of experiment

00:16:15.180 that they'd been working on starting over the summer

00:16:17.680 with these over-equipped, under-monitored agents

00:16:19.840 where you give them very powerful tools,

00:16:23.920 you let them prompt an LLM that has no safeguards,

00:16:28.980 and you RL train that LLM to be very persistent,

00:16:34.460 to just keep suggesting more and more actions, never give up.

00:16:38.140 You run 1,000 of them at the same time

00:16:41.060 in a loosely guarded environment for days on end without monitoring.

00:16:45.900 Is that the thing that we're worried about?

00:16:47.680 Because if so, stop doing that.

00:16:51.100 In other words, once you get very specific,

00:16:53.040 I think it's a different conversation.

00:16:54.440 It's like there's all this useful AI technology.

00:16:56.840 So I want to nail them down.

00:16:59.640 So is it that type of setup that you're worried about?

00:17:02.320 Where do you think RSI is going to come from?

00:17:04.200 Are you going to write one of these over-equipped,

00:17:06.220 under-monitored agents to what?

00:17:07.500 To update its agent code?

00:17:09.760 Or is there going to be an agent that's running?

00:17:13.260 Walk me through.

00:17:14.740 I want to get this down the specifics

00:17:16.460 Because when you get down the specifics, the responses are much more narrow.

00:17:22.560 And I think the sort of terror and fear becomes much more confined.

00:17:26.260 It's like, well, this just sounds like you're running these pretty narrow experiments that seem kind of negligent.

00:17:32.260 Like, maybe stop doing that.

00:17:34.380 Run safe experiments.

00:17:35.420 If you break the law, you're going to, you know, we're going to arrest someone.

00:17:38.000 Like, I think when you get more specific, somehow the air gets let out a little bit of the room.

00:17:43.160 And I think if you really push them, well, why are you running those experiments?

00:17:45.720 is because technological salvation ideology says

00:17:49.360 someone's going to make the god we want to get there first.

00:17:51.400 So we're willing just to run fast and try to just push.

00:17:53.740 I mean, I think they want to push these over-equipped,

00:17:57.000 under-monitored agents.

00:17:57.960 Why are they running that that way?

00:17:59.140 It's like they want weird stuff to happen.

00:18:00.620 They're sort of trying to instantiate this 20-year vision

00:18:03.180 they've had about how AI is supposed to develop.

00:18:05.480 So that's how we feel is we don't speak in terms of these,

00:18:09.860 you know, less wrong style, non-technical thought experiments.

00:18:13.800 We like to talk about real technology.

00:18:15.720 One of the things that's frustrating to me, just sort of looking at the public conversation around this stuff in general, and especially in the last, you know, few weeks, is the leaders of the AI companies and the people who work there are treated as not just experts in these things, but the experts, right?

00:18:37.620 And then, like, if they want to bring in another expert, they bring in, you know, somebody who, you know, maybe used to work there and, you know, did a stint in academia as well.

00:18:50.000 Somebody like Jeffrey Hinton.

00:18:52.060 This community of experts that you are part of, I'm just not seeing them in the public conversation on this very much.

00:19:01.220 I mean, yeah, okay, you have your op-ed in the New York Times.

00:19:03.960 Obviously, that's not nothing.

00:19:05.200 um and yeah i've got my book and that's not nothing but um but you know

00:19:11.840 you know bernie sanders is off talking with uh eliezer yudkowski and max tagmark and other you

00:19:20.400 know members of the rationalist and effective altruist communities and uh and you've got other

00:19:26.460 people from both sides of the aisle here in the u.s uh politicians right there in dc where you are

00:19:32.960 talking about this and they're they're really not talking to anybody other than people who believe

00:19:41.120 in some style of technological salvation either they think that the ai is going to kill us all

00:19:46.700 or the ai is going to bring about a utopia and either one's going to happen real soon it's

00:19:51.400 frustrating yeah i don't want to be a national ai commentator involved in these debates this is not

00:19:58.260 normally my beat, right? I'm, I'm human flourishing in a technological world. I focus more on the

00:20:04.000 impact of technologies on humans, not humanity. And yet what I'm finding is there's such a paucity

00:20:10.400 right now of just normal computer scientists unrelated to these ideologies, just, you know,

00:20:17.140 speaking up is that I will put out an article, even just for my own audience. Let me just try

00:20:23.420 to explain to them. I don't want them upset. Right. And it'll get picked up nationally again

00:20:27.300 and again and again. I mean, I have been in countless articles and stories in just the past

00:20:32.980 couple of weeks, have turned down countless television offers or this or that. And I'm not

00:20:38.060 trying to be a national part of the conversation. And I think that is kind of the problem. And that's

00:20:44.160 why I hoped that something like my op-ed, and then there was another one, Michelle Goldberg

00:20:47.940 basically wrote the same op-ed, which is good the next week. I don't know if you saw this next

00:20:52.520 weekend she wrote another one i actually didn't yeah i saw that it was written i didn't actually

00:20:57.520 have a chance to read it i think that's good there's a new op-ed out i haven't read it yet

00:21:01.540 in the washington post that's also um looking at effective altruism and you know sam bankman

00:21:07.720 freed and some of these like issues of like ai leaders might want to be a little bit worried

00:21:11.340 this doesn't always go the way you think it is yep uh so i i do think there is somewhat of a

00:21:16.380 turning tide. But I think if people understood, hey, this group of people thinks this way,

00:21:22.280 this doesn't mean to discount those people or that everything they're saying is made up or

00:21:25.720 the technology is not dangerous, but that's the grain of salt they come with. It would completely

00:21:29.820 change the coverage because you would also say, well, clearly we also want to ask other like AI

00:21:35.040 leaders, et cetera, unrelated to this ideology. And we're going to kind of triangulate a little

00:21:38.720 bit or see what's going on, but it's not happening now. I mean, even when those leaders are coming

00:21:43.000 out it's like largely being overlooked i said in my op-ed i said look one of the things you can do

00:21:47.720 is look to people who are deeply enmeshed in the ai community who don't have known connections to

00:21:51.860 these ideologies like nvidia and jensen wong they have their own motivations economically but he's

00:21:56.940 he's old school 80s style you know uh hardware guy that's like i don't have time to sit around

00:22:02.920 with jakowski well he did come out he came out immediately after jacob croxson's thing came out

00:22:08.140 and said, you know, he's like,

00:22:10.280 this is outlandish, was his words.

00:22:12.380 He's like, it's outlandish. 1.00

00:22:13.220 Let's make this, this is stupid. 1.00

00:22:15.340 Jan LeCun, you know, he's in Europe. 0.99

00:22:17.020 They don't do as much of this over there.

00:22:18.560 He was like, this is nonsense.

00:22:19.820 Like what's, you know, what's going on here, right?

00:22:21.700 So you, and then when you look at

00:22:24.380 where all these statements are coming from,

00:22:25.600 they tend to be from exactly the people involved

00:22:28.560 are the five people that I gave the history of

00:22:31.240 in my op-ed as having these connections to his ideology.

00:22:33.700 So it has to, but I don't think people understand,

00:22:36.420 i don't think people know i mean not like giving out specific names specific people but i see

00:22:41.180 people having other people on as like experts to tell us about this not realizing like that person

00:22:46.580 is straight out of that idea the same ideology as the idea of the people who are saying to things

00:22:51.620 you're having that person check like i don't think it's understood as well and it's not that that i

00:22:57.900 again it's not that that that disqualifies you but you got to understand it right it's like you're

00:23:03.660 not going to cover, um, the priest, uh, scandal in the Catholic church without understanding

00:23:09.960 what religion are the people I'm talking to, you know, like it would be relevant, you know?

00:23:15.100 So, so that's the kind of the interesting thing of this moment is like, I don't know

00:23:20.320 how to pick apart what impact those idealities are having, but I got to say, it's so similar

00:23:27.480 having read a lot of that work as well.

00:23:29.860 It's just, it's the terminology and everything, the ideas, the explanation of what they're afraid of, you can find almost word from word from well before LLMs existed.

00:23:41.120 Yeah, Coxon was quoting Eliezer Yudkowsky word for word when he was talking to Wired earlier this month.

00:23:47.740 So, yeah.

00:23:48.940 And to me, one of the other frustrating things is, you know, it's not like there haven't been people putting out, you know, much more reasonable takes about this, right?

00:23:59.640 You know, we, you know, the people who I quote and, you know, interviewed for my book, the people who I've had on this podcast last week's guest or the person who will have been last week when this comes out, Emily Bender, we've got Timmy Gebru.

00:24:16.560 We've got, you know, a lot of good people doing good work on this who just are not being brought into especially the policymaking conversation in the way that I would like to see them there.

00:24:34.560 right like i i want i do not want the policy conversation about ai to be dominated by people

00:24:44.200 who come out of these technological salvationist communities yeah i don't think people know this

00:24:51.160 there's an interesting story here the the ai safety wars right which again people it's just

00:24:58.880 a landscape people necessarily have a lot of knowledge of but you know in the beginning you

00:25:03.400 had what you can think of as like academic ai safety which that's where you can find emily

00:25:07.400 bender that's real phone timnet uh gerbu um you would think of them as being like academic ai

00:25:12.100 safety worlds which uh are concerned with a lot of proximate dangers of ai um negative side effects

00:25:19.060 externalities things to be concerned about about ai the way it is like deployed in the real world

00:25:23.540 and real systems so it's like seen through sort of like a typical type of academic framework well

00:25:28.420 we got to care about what's going to be the impact if you put this system in this environment or

00:25:31.980 Their bias is embedded in it, right?

00:25:33.760 What's going to happen if this allows, like,

00:25:36.440 mass surveillance or this or that?

00:25:37.700 So, like, it's very kind of the tangible negative side effects

00:25:41.340 of, like, deploying AI.

00:25:43.060 Then you have brewing this entire other shadow AI safety world

00:25:46.000 out in Silicon Valley,

00:25:46.840 which is from the technological salvation ideologies.

00:25:49.940 And, of course, they believe, who cares, right?

00:25:53.520 Like, you don't need to get the deck chairs right on the Titanic

00:25:56.660 when it's sinking, right?

00:25:57.960 What matters is the Titanic is sinking.

00:25:59.900 And they're like, civilization is going to be destroyed

00:26:01.680 or saved by this technology,

00:26:04.920 why would we care about bias in loan applications? 0.60

00:26:10.320 That's all going to be moot when the robots are,

00:26:13.880 I'm going to say putting us in chicken farms

00:26:16.480 because Elias Yukowski, not to have it aside,

00:26:19.740 but in the middle of all this debate,

00:26:21.640 Elias Yukowski is tweeting,

00:26:23.280 this just shows the mindset

00:26:24.640 of the really, really hardcore rationalists.

00:26:26.640 is Yachowsky is involved online in a minutely detailed debate

00:26:33.140 about will the humans be more like the dodo birds or the chickens

00:26:38.100 when the AI superintelligence takes over?

00:26:41.520 Because actually the dodo birds, that would be preferable 0.64

00:26:44.380 if they just stomped us out and extinct us.

00:26:46.800 But we might be more enslaved, right?

00:26:49.460 Like chickens actually have it way worse than the dodo birds

00:26:51.760 because if we're going to be in the equivalent of like a factory farm

00:26:54.340 and it just personifies rationalism

00:26:57.340 which is you just you sit there thinking through thought

00:26:59.380 experiments and you get

00:27:01.080 you twist on twist on twist

00:27:03.180 until you're lost in your own self-referential

00:27:05.140 Escher sketch of like a logical construction

00:27:06.980 that's what he was talking about

00:27:08.520 during that but so that

00:27:11.200 that AI safety culture

00:27:13.140 really doesn't like

00:27:15.160 the academic AI safety culture

00:27:16.940 and there's all sorts of reasons why they also

00:27:18.860 associate that with like wokeness

00:27:20.760 and Silicon Valley has ruptured

00:27:23.040 with

00:27:23.520 the sort of the left on

00:27:26.320 because they had the left

00:27:27.380 and sort of the woke left

00:27:28.640 and it liked the tech bros.

00:27:29.880 Like, well, we'll just do our own thing.

00:27:31.120 And there's these sort of

00:27:31.740 these ruptures between them.

00:27:33.320 Like, we'll have our own independent media.

00:27:35.640 But the Silicon Valley AI safety,

00:27:37.540 it triumphed in the end.

00:27:39.900 Their message of this AI

00:27:41.640 is going to kill us all

00:27:42.560 was more interesting

00:27:43.520 than the academic AI safety conversation.

00:27:47.380 But there is a lot of bad blood

00:27:48.620 between those two groups

00:27:49.620 at exactly that fault line.

00:27:51.580 Yes.

00:27:51.820 i'm i'm aware of this um and i think your description of it is you know pretty much

00:27:57.560 accurate except with a couple of caveats first of all i know that uh you know if emily and tim

00:28:04.940 and others are listening to this right now they are screaming we we don't call what we work on

00:28:09.120 ai safety ai safety is one of their terms right they they would call it ai ethics and i do think

00:28:14.700 that's a better term for it but yeah um but um but the other thing is you know i want to push

00:28:20.880 back on you saying that you know the ai safety people like yadkowski and whatnot won um because

00:28:28.400 it's not like this is over right you know they are winning out in the round yeah round one sure

00:28:34.160 yeah in the media ring i feel like they won round one in the they have absolutely dominated the

00:28:39.780 conversation yeah but we got to find a way of bringing it back the problem of course is that

00:28:45.000 it's way more fun and sexy and interesting to think about an AI apocalypse than it is to think

00:28:52.600 about bias in existing AI systems and the environmental and social harms that they

00:28:58.940 perpetrate. Because the second one is kind of boring and pedestrian, and the first one is

00:29:06.420 interesting and science fictional, and the only problem is that, of course, the first one's not

00:29:11.860 real yeah and the second one is but i would i would modify that and say from the reporter

00:29:17.180 perspective because i have a foot in that world as well sure um i i think it's more fear than it is

00:29:22.840 interestingness right there's two things happening here one i think that the dominating analogy is

00:29:28.720 right if you if you talk to reporters like they always have that in their mind now it's like that

00:29:34.680 was a story you didn't want to miss right that when you're in that you're in early 2020 you don't

00:29:41.580 want to be dismissive of that story. In fact, being too alarmed pales in comparison from a

00:29:49.040 cost perspective of having been out there and saying COVID's no big thing. So I think that is

00:29:54.200 a dominating sort of idea in the reporting class. And then second, the technical ideology virus

00:30:00.700 has a very disastrous but clever adaptation, which is it's infecting the experts, the scientists,

00:30:10.400 the technologists and that's the the we're that's the frame we have we go back to the covid analogy

00:30:15.300 is like oh you trust the experts don't trust like the random people or whatever uh you trust the

00:30:21.740 people with technical backgrounds and so the people at ai labs have technical backgrounds

00:30:26.680 so that's who we trust right so that's what makes it insidious now what's as you know obviously the 0.91

00:30:32.220 the uh the the intellectual engines behind these movements are non-technical it's yukowski um it is

00:30:39.540 the, the, the, the effective ultra was coming out of Oxford that are more from a philosophy

00:30:43.280 background than a, a technical background.

00:30:45.820 And so the intellectual engine was not technical, but it was immensely appealing to like a young

00:30:52.180 technical crowd in the Bay area.

00:30:53.560 And then they, they, they have, you know, gone forth sort of shaped by these ideas.

00:30:57.680 The thing that people don't realize about technical people is that we're immensely gullible.

00:31:03.340 We're immense.

00:31:04.260 Like if I can't tell you, and the reason why, by the way, is it's Dunning Kruger.

00:31:08.540 If you're really good at something,

00:31:10.760 like you're really good at math

00:31:12.100 and you're used to being better at math

00:31:13.980 than anyone else you're in the room at,

00:31:15.460 what happens is it's completely natural to you

00:31:18.400 that you're better than everyone else

00:31:19.940 at everything else.

00:31:20.620 Like that makes sense to you.

00:31:22.620 It's why if you want to hear 1.00

00:31:23.560 the stupidest financial schemes, 1.00

00:31:26.620 go talk to a bunch of like IT people or developers 1.00

00:31:29.260 because they're so used to knowing more

00:31:30.980 about computers than everyone else.

00:31:32.440 They're like completely comfortable

00:31:33.440 with the idea of,

00:31:35.040 oh, I've got this way to make a lot of money

00:31:36.380 on the stock market.

00:31:37.200 I'm going to, if I put gold in this and move that crypto over here or, or in health or in longevity, like whatever it is, uh, tech, technological people, you've got like big brains for a certain thing.

00:31:47.460 Um, we're very gullible to, uh, a lot of other things.

00:31:51.220 So I don't, I don't think that message got out.

00:31:52.560 So, I mean, I think that's part of the issue here is that it's a, uh, a mind virus that's infecting the, the class that we're used to sort of trusting in a very general sense being technical experts.

00:32:05.340 Yeah, no, I, I think that's right.

00:32:06.780 I mean, I, I talk about this in my book as a engineer's disease and, and to me, like

00:32:12.040 the, the classic examples of this are the people who show up in my email with crackpot

00:32:18.780 theories about physics and nine times out of 10, they're retired engineers.

00:32:23.760 Yeah.

00:32:24.600 And, you know, that's not to say that physicists aren't susceptible to this as well, but, um,

00:32:30.420 but, you know, I just, I get these emails, right.

00:32:33.060 And I, most physicists do most, uh, science journalists do I'm both. And so I, I have it,

00:32:39.100 you know, pretty bad. Um, I mean, not that it's a big deal, but I get like a few of these a week

00:32:44.040 and it's, it's always people who are like, Oh yeah, you know, I, I work in this technical field.

00:32:51.020 And then I started thinking about physics, something I have no formal training in. And,

00:32:55.120 uh, you know, what if, what if everything is just, um, why I can't even get into what they

00:33:01.760 believe because it generally doesn't even make enough sense to explain. But yeah, you know,

00:33:05.640 the level of confidence is surprisingly high. And I mean, there's always the question with

00:33:11.740 something like that as well of like, okay, well, I'm only seeing the ones who are confident enough

00:33:17.280 about what they believe to send me a stranger, an unsolicited email. But, you know, there's

00:33:22.020 something about that as well, right? Because there are relevant experts that I would like to see the

00:33:28.280 media talking with more, you know, not just, you know, people like Emily and Tim Neat and you,

00:33:34.280 but also, you know, if we're going to be talking about the possibility of artificial super

00:33:40.580 intelligence or whatever, we should be talking not just with computer scientists, but also with

00:33:46.240 cognitive scientists and, you know, psychologists, people who know a lot about the nature of

00:33:52.520 cognition and intelligence. And, you know, since they're large language models, we should be

00:33:56.900 talking with linguists like Emily as well. And yet that kind of technical expertise, and I think

00:34:03.520 that this is also sort of something that goes hand in hand with this kind of engineer's disease,

00:34:08.560 there's this idea that it's somehow, you know, that these are softer sciences and thus they're,

00:34:14.060 you know, not as technical and therefore don't really know as much what they're talking about.

00:34:20.020 When the fact is, you know, there's expertise there that most technical people just don't have.

00:34:25.380 And it's the fact that they don't have it that I think is one of the things that makes them susceptible to these ideas of technological salvation.

00:34:36.300 Yeah, I think people underestimate also the degree to which like frontier LLM work right now has become very engineering, empirical, fiddly, right?

00:34:47.600 Yeah.

00:34:47.720 So you have the breakthroughs that underpin these models are coming out of a more academic context.

00:34:53.120 You get transformers out of sort of the research arm at Google.

00:34:55.940 You get back propagation, you know, coming out of an academic context.

00:34:59.780 And that's where people are really thinking through kind of theoretically in a rigorous way, like how do we want to architect or build AI systems?

00:35:08.140 But at the breakneck speed that the frontier labs are working, it's way more of like, let's try this.

00:35:12.660 Let's turn this knob.

00:35:14.020 Hey, that was good.

00:35:14.920 Let's like turn up the power by 100 and see what happens next.

00:35:17.460 So it is very sort of experimental right now without this underlying deeper understanding.

00:35:24.300 And like I find, for example, when I talk about it, there's really foundational ideas that really matter and it's very difficult to get across, right?

00:35:32.820 And I try to be very careful about my wording, but I think it matters, for example, to understand, right, like how an LLM actually operates, right?

00:35:43.240 So ultimately, you're trained on a token prediction game, right?

00:35:47.800 That the model assumes, obviously, it has no actual stance, but it's trained to minimize loss on text is going in as input.

00:35:58.800 Assume this is a real text, and I want to get as close as possible to what token actually comes next.

00:36:04.220 All right, so foundationally, you have that, right?

00:36:06.620 Okay, there's a lot of options there, though, right?

00:36:09.260 For like a lot of texts at a lot of places,

00:36:11.060 there's a lot of options for what tokens come next.

00:36:13.620 They're all are perfectly reasonable, right?

00:36:15.980 So this is where if you're doing tuning

00:36:17.980 after the original pre-training,

00:36:19.880 what you're essentially doing

00:36:21.100 is for certain types of inputs,

00:36:23.100 steering towards certain plausible answers versus others, right?

00:36:26.060 So you can tune it, right?

00:36:29.240 So if someone asks a question about,

00:36:31.980 it's like the end of a question about a biological weapon,

00:36:34.540 there's like a lot of ways a reasonable answer could start,

00:36:37.580 But you can tune a model to be like, of the possible answers to this, I'm going to be super tuned towards the answer that says, I'm not going to talk about that, right?

00:36:46.480 So you can then tune on top of this.

00:36:48.120 But even with the tuning, what you're getting is a machine that if you run it autoregressively, are going to get a plausible extension of whatever the original input was.

00:36:56.320 Plausible meaning this plus the original input is something that really could have shown up in my training text.

00:37:02.560 So it's a plausibility generation machine.

00:37:06.320 That's different than normativity.

00:37:08.440 And the way that's produced, when you understand it's token by token, though there can be a lot of logic and computation that's involved in trying to solve the problem of what's a reasonable, plausible token to come next.

00:37:18.540 It's a very, from a cognitive science standpoint, a very different notion of, you know, cognition than almost any sort of actual sentient actuated being actually has on Earth.

00:37:30.840 Like, as you're saying, these things matter.

00:37:33.320 and when you recognize that it changes the way you talk about these things and and and what we

00:37:38.660 get instead is something that's very sloppy like i don't know what it means when someone is like

00:37:42.660 well the the agents wanted to do this then coordinate it i don't know what that means

00:37:48.020 right that makes no sense the agents are just a computer program i can show you the source code

00:37:51.760 that are all prompting the same llm yep and that llm is just answering prompts solving this

00:37:58.660 normativity problem i mean this plausibility problem and the agents just programs that just

00:38:02.940 blindly execute and reprompt so where is the want here where is the where is the desire

00:38:09.020 is it the llm prompts have some underlying plan but that doesn't make sense because llms are

00:38:13.560 static without movable state so every prompt is response is generated from absolute scratch from

00:38:18.140 the exact same weight so where is the intent what does that even mean don't be sloppy and say the

00:38:24.140 agents are training themselves agents don't train themselves they're a computer program

00:38:28.080 They run a bunch of agents.

00:38:29.600 After the fact, they then go and get the traces of the agents that happen to do better.

00:38:34.020 And then they could use those traces later in an offline batch training program to try

00:38:37.460 to tune the model towards those type of suggestions.

00:38:40.160 But all of this is sloppy, right?

00:38:42.580 So like the more, as this is to your point, the more you actually engage with how these

00:38:47.460 things work, the more you can be much more accurate about what's going on and how you

00:38:52.600 talk about things.

00:38:53.520 And it's people within the labs don't talk this way.

00:38:57.340 It's like incredibly hard to get someone to be architecturally grounded in the way they talk about these devices.

00:39:04.100 And when I do this at even a simple level of abstraction, a couple of weeks ago,

00:39:08.240 like, let me just explain an agent and its interaction with an LLM.

00:39:13.620 There's like a New Yorker column about that.

00:39:15.700 Because that's like, oh my God, no one has told us this before.

00:39:19.100 So it's really blowing my, I mean, I'm not an LLM expert.

00:39:21.980 I'm just a, you know, I'm a computer scientist and I can read and understand these.

00:39:25.680 I have a lot of sources.

00:39:26.520 I'm also an AI journalist, so I talk to a lot of people.

00:39:29.560 But I don't think it's that hard to get to a reasonable level of understanding.

00:39:34.140 But it really is missing from the discourse, and it really matters if you're going to talk about these things.

00:39:40.780 Unfortunately, we still live in capitalism, and that means we need to talk about our partners and sponsors.

00:39:45.980 But fortunately, our partners and sponsors are great.

00:39:48.360 And if you want to support Dreaming Against the Machine directly, you can join us on Patreon, where $5 a month gives you the chance to ask questions of upcoming guests and a dedicated stream of cat pictures.

00:40:00.980 We love our patrons, and we also love being a part of the Multitude Collective at MultitudeShows.com.

00:40:06.980 Multitude is a podcast company made up of passionate people creating shows you can count on.

00:40:11.500 Shows like Simple and Clean.

00:40:13.520 Simple and Clean is a show where hosts Misha Stenton and Mayanna Barron and friends talk

00:40:19.220 about Kingdom Hearts, a little video game franchise about really big feelings.

00:40:24.220 Join them as they discuss the plot, characters, world building, music, fashion, fan base,

00:40:30.200 and societal impact of a thing that, by all means, probably shouldn't even exist.

00:40:35.520 Never heard of Kingdom Hearts?

00:40:37.000 That's okay.

00:40:38.280 They're here to break it down for you, explain why they love it, and hopefully to convince

00:40:42.240 you that this weird little story deserves a place in your heart. Each week, they attempt

00:40:46.920 to unravel how this deceptively expansive game series has touched an entire generation

00:40:51.600 of deeply earnest weirdos, and along the way realize that what they're talking about might

00:40:56.300 not be just a video game, but a whole lot more. New episodes every Sunday, wherever

00:41:01.560 you listen to your podcasts. And here's a word from our sponsors.

00:41:04.880 The fact that they're not talking this way inside the labs, I think, is really troubling because, as you were saying, the stuff that they're doing, it's all sort of empirical, right?

00:41:19.980 They're just doing a bunch of experiments.

00:41:23.340 The theory that you have, like this is like philosophy of science 101, the theory that

00:41:30.260 you have for, you know, what you're doing, what you're working with is going to inform

00:41:36.400 in a very serious way the choices that you make about what experiments to conduct.

00:41:42.140 And we are seeing that happen here.

00:41:44.480 And so the theory that they have is that they are on their way to building a god.

00:41:51.240 And so the choices that they're making about what experiments to do are things like, you know, setting up agents to do the hugging face hack and things like that, rather than, you know, trying to figure out what sorts of things these tools can be used for well.

00:42:09.340 And you know why there was so many agents running at the same time?

00:42:14.100 Like, we now know this.

00:42:15.240 The reason why they had well over 1,000 agents running simultaneously

00:42:19.000 is because they were hoping, like, we just need some of these.

00:42:24.760 If, like, a few of these happen to stumble into, like, a good solution,

00:42:28.440 we can use their history to then go back and tune the LLM

00:42:33.540 towards those type of suggestions

00:42:35.420 so that, like, it can more consistently make progress.

00:42:37.880 but that's that's sort of functionally equivalent is the analogy i use of saying like if you're

00:42:41.600 testing self-driving cars you'd be like let's just release a thousand of these cars onto the

00:42:46.420 interstate yeah most of them will crash but like some of them might like successfully make it to

00:42:51.320 the destination and then we can use their logs to help build a better generation of of uh self-driving

00:42:56.800 cars going forward so it's it's really it's really kind of reckless they just want to make

00:43:01.040 um i think that these sort of long horizon heavily equipped agents i think i don't know

00:43:07.500 but I'm assuming it matches their priors for like where the super

00:43:12.840 intelligent stories progresses.

00:43:14.320 And it's like, 0.80

00:43:15.100 so we got to damn the consequences. 0.90

00:43:17.800 Like, 0.94

00:43:18.220 let's just like go fast and loose on this because that's what we're

00:43:22.020 supposed to be doing.

00:43:23.180 The cynical,

00:43:24.200 like less rationalist style explanation would be like,

00:43:28.020 well,

00:43:28.240 open AI wanted to win on that particular benchmark.

00:43:30.440 And so they're like,

00:43:31.380 we'll do this really dangerously because as long as like one agent,

00:43:34.460 like beats this challenge,

00:43:36.580 know what it's done before we can we can say sure anthropic had mythos but but anthropic was running

00:43:42.940 similar types of over equipped under monitor agents as well you know that not as dangerously i think

00:43:47.920 the incursions were much smaller um meta claims they had a couple minor things but that just sort

00:43:52.940 of sounded like a like hey we're powerful too type of like don't forget us you know then they've kind

00:43:57.480 of quieted down after that yeah because that's more that's more complicated i don't know how

00:44:01.620 i don't think they're very rationalist embedded over there they have their own you know yeah meta

00:44:05.160 is the one that i know the least about actually i know a lot more about open ai anthropic and uh

00:44:10.440 deep mind well open ai and anthropic we you know from your book and i reported this in the new york

00:44:14.980 times what's important is that like those companies came out of a ai safety um orientation like that

00:44:20.780 that's their reason for existence deep mind came out of kind of i mean it was hasabi leg and uh

00:44:27.800 soleiman wanted to create agi but they they were uh you know lead figures there were very connected

00:44:33.820 to these same rationalist worlds as well.

00:44:36.560 As you talk about in your book,

00:44:37.780 it was Yukowski himself

00:44:38.840 who introduced Ed Hasabi and Leg

00:44:40.600 to Peter Thiel

00:44:41.940 to get the initial funding for DeepMind.

00:44:44.800 And who are the people, by the way,

00:44:46.500 the exact list of people

00:44:47.700 who signed on to say

00:44:49.480 Dario Amadei's slowdown letter is right.

00:44:51.620 It was Sam Altman.

00:44:53.000 It was, you know, Demis Sasabi.

00:44:55.160 It's Dario Amadei.

00:44:56.740 It's Elon Musk.

00:44:57.740 Those are exactly the characters.

00:44:59.440 Yep.

00:45:00.120 From your book.

00:45:01.120 Exactly the characters from my op-ed

00:45:02.680 that are most at the intersection points.

00:45:05.800 But yeah, I mean, back to the original point,

00:45:07.440 I guess we're trying to make here

00:45:10.460 is that there's a rich conversation to be had

00:45:13.840 when you're talking about the actual technology

00:45:15.840 and the actual decisions

00:45:16.960 and why would you run this experiment?

00:45:19.820 This seems unsafe,

00:45:21.440 not in a you're summoning ball sort of way,

00:45:23.740 but in a like, these are unpredictable.

00:45:25.360 You have, I mean, LLM outputs are unpredictable.

00:45:28.320 You really don't,

00:45:29.840 you gotta be super careful

00:45:30.960 about driving long horizon actions

00:45:33.140 off of just thousands

00:45:34.700 of unsupervised LLM outputs.

00:45:36.620 Anyone who's talked

00:45:37.260 with a chatbot long enough

00:45:38.260 knows eventually

00:45:39.700 you're going to go

00:45:40.160 in a weird direction.

00:45:41.300 And if you don't have a human

00:45:42.100 to be like,

00:45:42.520 no, no, no, let's bring it back.

00:45:44.920 Like God knows,

00:45:45.540 it's a game of actuated telephone

00:45:46.880 where you're going to end up

00:45:48.300 in some final state.

00:45:49.900 It's like real garbled version

00:45:51.420 of what the human

00:45:52.080 originally thought

00:45:53.100 where you would end up

00:45:53.860 because unpredictable.

00:45:56.120 Yeah.

00:45:56.420 But all of that gets obfuscated

00:45:58.240 if the conversation is just,

00:45:59.740 there is a thing

00:46:00.580 called the model that is getting

00:46:02.540 more powerful and we're having a harder and harder

00:46:04.480 time controlling, it gives you like one number

00:46:06.500 to watch go up. And so what happens as

00:46:08.500 it gets to these higher levels of

00:46:10.040 power, we're going to have a harder and harder time

00:46:12.440 until it can control us. But there is no

00:46:14.460 direct physical

00:46:16.200 analog to that frame to actual

00:46:18.620 products being built. So there's this kind of

00:46:20.600 disconnect between what they're doing

00:46:22.740 and then the rhetoric with which they're describing

00:46:24.700 it. No, I mean, I think

00:46:26.200 they see these sorts of long

00:46:28.820 horizon on

00:46:30.440 supervise things as you know a way to sort of get to that idea of recursive self-improvement but

00:46:37.780 there's no real connection between those things not in a concrete way if you ask them to get

00:46:42.080 specific i don't think that they could i'm trying to figure that out this is what i'm trying to i've

00:46:46.300 been reading all of the the sort of like long posts that people from this community are putting

00:46:50.500 to try to be clear about here's what we're worried about and it's all ultimately rsi and i can't

00:46:56.080 figure out the exact technical action they're thinking, right? Because agents are just computer

00:47:02.140 programs that aren't that interesting. They're prompt loops, right? I mean, they can get

00:47:06.700 complicated if you're trying to, for a task specific way, right? Like coding harnesses are

00:47:10.640 very, there are 500,000 lines of code or more now because it's a huge amount of special cases

00:47:15.660 programmed in their trial and error to try to keep these agents doing the most useful possible

00:47:21.120 stuff for programmers, but ultimately it's a pretty simple program. The smarts all comes out

00:47:26.360 of the common LLM that the agents are querying. So when they're talking about recursive self

00:47:32.920 improvement, I think people are imagining somehow that like the agent programs are doing something

00:47:38.040 and I guess they could be modifying their code, but it doesn't matter much because again, all they

00:47:43.100 do is prompt the LLM and does what it says. So there's not a lot of gain to be had in that code

00:47:47.160 in terms of power right so they must be talking somehow about the llm somehow making some sort of

00:47:55.820 leap in ability that now if you hook an agent to it it's it's uh i mean i guess it's the original

00:48:01.820 scale vision of eventually the llm its responses will be the equivalent as if you actually had a

00:48:06.960 super smart human in a room that was giving you the same responses and then if agents are asking

00:48:11.260 what to do it'll be like an actuated really smart human but i don't understand how that happens

00:48:16.420 Yeah, neither do I.

00:48:17.220 Because if you want a better LLM,

00:48:19.480 I think the only way that we know how to do that

00:48:21.200 is like increase the size of the model

00:48:22.900 and increase the training data.

00:48:24.740 Yeah, and that stopped working a while ago, actually.

00:48:27.440 Yeah, exactly.

00:48:28.540 Yeah, so I don't know how you recursively

00:48:30.440 create more training data.

00:48:31.940 So that doesn't make sense to me.

00:48:33.620 Yeah.

00:48:34.140 And this is one of the real disconnects, again,

00:48:35.740 between like close monitors of this field

00:48:38.060 and everyone else is like,

00:48:39.120 actually LLMs in some sense have been struggling.

00:48:42.080 I think what people don't understand

00:48:43.620 is there's been three phases of LLMs.

00:48:47.020 Phase one was a scaling phase

00:48:49.780 where they made the LLMs larger

00:48:51.600 and trained them longer

00:48:52.480 and they had general improvements.

00:48:54.140 So in almost everything,

00:48:55.260 they did better.

00:48:56.200 That era ended around GPT-4.

00:48:58.860 Many companies,

00:48:59.700 including notably OpenAI,

00:49:01.060 their post-GPT-4 models,

00:49:02.620 they tried to make massively larger

00:49:04.220 and found modest gains.

00:49:05.700 Like, uh-oh,

00:49:06.500 so we can't just scale.

00:49:08.560 Project Orion at OpenAI

00:49:10.620 was a massively bigger LLM

00:49:12.680 trained for months and months and months and it wasn't much better than gpt4 and so the second

00:49:17.220 stage which really picked up in 2024 more like the fall of 2024 was what they were the the reasoning

00:49:23.820 model revolution but basically what it was is you tune a model after the fact after you train it

00:49:29.000 uh to ramble to say okay the type of of the possible answers you can give that are all

00:49:34.300 plausible um we like answers where you you kind of ramble out loud for a while before you get to

00:49:40.160 a final response.

00:49:41.440 Don't just give the response directly.

00:49:42.760 And this allows you to get more inference

00:49:45.400 out of the model aimed at the answer

00:49:47.720 because every new token you generate,

00:49:49.520 that whole input goes back through the model.

00:49:51.660 And now you can apply a lot more inference

00:49:54.420 to a solution.

00:49:56.860 Now they can start doing better on benchmarks.

00:49:59.900 But not so much like in general,

00:50:02.560 everything was better that the LMs could do.

00:50:04.520 So you saw a huge pivot in the fall of 2024

00:50:06.780 away from look at how cool this thing is like ask it anything you'll see it's better than it was

00:50:12.340 before towards look at our graphs so it's doing better on these benchmarks so then we got the

00:50:17.020 reasoning age and that's where you got these long chain of thought traces because you're applying

00:50:23.420 more inference and temporary holding of data you can get better responses but it's very expensive

00:50:27.500 because now you're generating a huge amount of tokens to get to your answer so then that was

00:50:32.240 sort of what happened coming into 2025, um, that began to sort of peter out, uh, by the time you

00:50:38.500 get to GPT-5 and then you get to the third phase where you say, okay, what we really going to do

00:50:42.680 now is we have to keep showing improvement is we are going to specialize. And so we're going to

00:50:48.600 find areas that are very well suited for LLM. So structured data areas where we have synthetic

00:50:54.480 data with clear right and wrong examples, we can do a lot of training. Um, and we're going to go

00:50:59.040 all in on a small number of areas where LLMs are particularly well good at and will use their

00:51:04.540 advances in those areas as ways of arguing that LLMs are continuing to get more powerful. So they

00:51:10.320 became very jagged. So this was computer programming, math, and cybersecurity. It's a very structured

00:51:14.880 language. They have a lot of data for it. The LMs got worse at other things. So like they don't talk

00:51:20.220 about all of the general public applications, the fact that every office worker was going to have an

00:51:25.160 assistant to automate everything they do. All of these use cases all kind of fell off the radar

00:51:30.640 because actually when you super tune for one of these like highly structured use cases that LMs

00:51:37.280 are suited for, they tend to get worse at the other things they do. And that's been the last year

00:51:42.140 was coding, cybersecurity, and math. And so, you know, if you look at it at one angle, you can say

00:51:50.160 it did so well at math and coding this year. These are awesome. But if you zoom out, you can

00:51:55.100 like, man, you gave up on all the things you were talking about in 2024 and 2023 and early 2025

00:52:01.240 when OpenAI said it was going to be the year of the non-coding agent. Everyone's going to have

00:52:05.220 their own agent. No, Microsoft shut down the co-pilot program, right? Like we can't, this is

00:52:09.320 not working, you know. So you could look at the other way and say, I don't know, LLMs, it's like

00:52:13.960 pretty jagged. Like what we're finding is we're searching for areas in which if you put a huge

00:52:20.100 amount of attention into it, you can make an LLM do well in that area. And we're finding them in

00:52:24.040 structured language and we'll find some more and that's, but that's kind of the real story of LLMs

00:52:27.800 is like they're, it's more jagged and spiky than we hoped. They're not scaling the AGI. So it's

00:52:33.880 weird if that's the trajectory you have in your understanding of LLMs to also be thinking these

00:52:40.800 things are just going to become like, uh, pinky in the brain, the brain, like just like generally

00:52:46.160 very smart. And then the agents will be, uh, able to do, you know, whatever the LLM. So I mean,

00:52:52.260 There's a disconnect here.

00:52:53.640 And I think there's this general sense, they will say, of like, well, if it's good at math problems and it's good at coding, isn't there a lot of math and coding in AI, dot, dot, dot, LLMs will create an explosion of intelligence in AI.

00:53:09.060 I think it's like somehow a sort of just like putting those things together.

00:53:12.720 But it's unclear to me how those, you know, again, functionally, what are we actually talking about here?

00:53:17.740 I think that's my impression as well, that they think, oh yeah, no, we only need it to be good at coding and math because LLMs are just pieces of code that do math. And so if we can just make it better at that, then it can make better ones. And it's like, that's not, that's not really how that works. Like, you know, they, yeah. So no, I agree with you both in terms of what they're aiming at and why it's not going to get them where they think it's going to go.

00:53:44.640 but as dangerous to do and this like this is my big problem is here's my counterfactual if 0.81

00:53:50.440 yukowski didn't exist if we didn't have technological salvation ideologies as a thing

00:53:56.500 but we still had llm technologies and breakthroughs and like there was companies building these tools

00:54:00.620 what would that research look like and i think that's an important counterfactual because i do

00:54:04.780 think a lot of the things they're doing like with these these under supervised over monitored long

00:54:08.580 horizon agents and stuff we wouldn't be doing that if it wasn't for those ideologies i think

00:54:14.360 they are going to write agents that are going to modify their own code somehow or adjust somehow

00:54:20.100 the LLMs in unsupervised ways, which is not going to create a super intelligence, but it's going to

00:54:24.840 create incredibly haphazard and unpredictable computer programs doing God knows what. Like,

00:54:30.060 it's not, this is dangerous stuff to do, but not in the same way that they think it's going to be

00:54:36.900 dangerous. And so that's my counterfactual is, I mean, that's what I would ask if I had them on

00:54:42.340 the stand. I mean, I'd just be very curious. I mean, what I'd really want to do is bring

00:54:45.440 executives from other cutting edge labs that aren't connected to that ideology and be like,

00:54:49.480 well, what are you working on? Why aren't you doing that experiment? How do you feel about

00:54:53.740 those experiments? Like, I don't know why every tech reporter is not having those conversations

00:54:57.680 right now. I mean, it is leaking out though, right? Like, yeah, okay. Yensen Huang isn't

00:55:02.480 connected to this community. And, you know, he was dismissive of Coxson, but he, you know,

00:55:09.100 just like a week before that was saying, oh yeah, GPT-6, that's AGI.

00:55:13.640 And like, what are you talking about, man?

00:55:16.040 Yeah, they were referring to that one benchmark, I think.

00:55:18.740 But still, like it was a weird and I think unjustified thing for him to say.

00:55:23.400 I mean, just the idea of AGI is something that comes out of these communities in the

00:55:26.780 first place.

00:55:27.220 And it's also worth saying, by the way, like he does have a bias towards the less regulation,

00:55:32.820 less constraints, less we worry about China, the more chips he can sell.

00:55:35.160 So like, obviously there's other countervailing.

00:55:37.200 But OpenAI and Anthropic also have economic biases to not say their technology is going to kill everybody.

00:55:44.060 So, you know, it does show that if you really believe in these ideologies, it tends to trump economic bias.

00:55:50.080 I do think that there is something to the idea that saying that the technology could kill everybody makes the technology sound more powerful than it is.

00:56:00.080 But that's not why they're saying these things.

00:56:03.040 I think that's true.

00:56:03.600 And yeah, I don't think it's regulatory capture.

00:56:07.200 anymore i think that's just like it gives meaning to their world i think they're just true believers

00:56:11.680 yeah no i think that's right don't underestimate the power of a belief system that says you're

00:56:18.500 the most important person in history that's more appealing to people than man i could probably

00:56:25.820 double my stock option value yeah because there's a lot of rich people in silicon valley but there's

00:56:30.820 only one most important person in history and i feel like a lot of people coming out of those

00:56:35.900 ideologies are auditioning for that role and think they might be it i think amide wants to be that

00:56:39.920 person i think yakowski for sure oh yeah just feels like it's its destiny yes so you're right

00:56:45.480 it is kind of hard to it is hard to pull apart all these threats but but wong is interesting

00:56:50.640 right because he also in january he was like very pointedly saying i uh look a lot of us like sci-fi

00:56:58.660 growing up but this shouldn't i'm paraphrasing but he was like but we really shouldn't let like

00:57:02.940 the sci-fi fantasies shouldn't be affecting like our business right now,

00:57:07.580 which I thought was,

00:57:08.620 you know,

00:57:08.780 he was being a little bit circumspect,

00:57:10.260 but he was basically saying,

00:57:11.400 can we stop with the rationalist stuff?

00:57:14.180 I did like that.

00:57:15.020 He said that.

00:57:15.760 Um,

00:57:16.040 but the thing is,

00:57:16.880 you know,

00:57:17.260 again,

00:57:18.460 he's got that incentive to try to downplay the doomerous stuff,

00:57:21.900 right?

00:57:22.400 He's,

00:57:22.900 uh,

00:57:23.060 this is the gold rush.

00:57:24.160 He's selling shovels.

00:57:25.180 He doesn't want there to be restrictions on mining.

00:57:27.600 I think that is fair enough.

00:57:29.360 Yeah.

00:57:29.560 So again,

00:57:30.260 this is the complicated mess.

00:57:32.060 Yes.

00:57:32.460 this is a complicated mess we're in because you have powerful technologies you know a lot of

00:57:38.320 things wrapped up in this you have the economy the stock market you have weird ideologies that

00:57:44.000 we don't know how they're perverted people's thoughts and like we got a sense last week of

00:57:48.080 like yeah it's pretty bad right like people really believe this not just they really believe it but

00:57:52.580 they believe that like but that shouldn't stop them from what they're doing to me that i think

00:57:55.600 that's the thing that really kind of worried people was like yeah we're going to kill everyone

00:57:59.240 and that's okay. Or like, what are you going to do? Like, I think that's kind of, that's just

00:58:02.800 weird. That's a very, um, I've seen some critiques recently of like, this is what happens when you

00:58:08.040 replace like a liberal arts education with just, you know, pure technical education is that you

00:58:12.900 just sort of invent your own ethical frameworks that are weird and warped. And have we seen this

00:58:18.460 before? Like industrialized, like, you know, non-humanist, uh, ethical systems. We've seen

00:58:24.320 this in the 20th century a couple of times. It doesn't always work out well.

00:58:27.340 Well, yeah, but, you know, it's Silicon Valley.

00:58:29.800 They don't think that studying history is important if they can just reinvent everything on the fly, right?

00:58:35.060 Yeah, they're smarter than those people from history.

00:58:37.160 Yeah, exactly.

00:58:38.060 So what do you think is going to happen?

00:58:40.420 Oh, God.

00:58:41.660 In the near future?

00:58:42.660 Like, I can't get my arms around it.

00:58:46.180 Yeah, what would you—I don't know.

00:58:47.160 What do you think is going to happen in the next couple of months?

00:58:50.960 In the next couple of months, I'm just hoping that the Democrats do well in the midterms.

00:58:56.240 But but it is looking like that's probably going to happen. I'm not sure. I mean, just in the last couple of days, I have been seeing weird AI slot memes from the Trump administration denigrating effective altruism, which is not something that I thought I was going to see.

00:59:15.120 And that's awfully strange. So I think what I would like to see is I would like to see the AI ethics camp get a little more traction with the policy people.

00:59:30.460 I think that as the election approaches, we're going to see some weirder and weirder statements from politicians in the U.S. because elections always end up making politicians say and do strange things. 0.75

00:59:44.040 But it does sort of look like, at least just based on the last, you know, week or two, that there's now this push for regulation of the AI industry, but it's coming from these doomers. 0.88

01:00:01.060 Yeah.

01:00:01.580 And so it's not, you know, meaningful regulation. But weirdly, the Republican response, or at least the Trump administration's response, seems to be, oh, regulation businesses, we don't do that here. Absolutely not. That's un-American.

01:00:16.580 There had been this sort of idea of, okay, maybe there's a sort of bipartisan thing happening where both parties are good news, unhappy about data centers and bad news, unhappy about made-up AI apocalypses.

01:00:32.940 But now it's looking like maybe some partisan polarization is taking hold.

01:00:37.420 And I don't know how that's going to interact with this well.

01:00:40.500 I don't know.

01:00:42.400 I think this is like a, we need the technology business reporters to really dig into this.

01:00:48.940 I mean, clearly Trump is being influenced by the sort of David Sachs, Silicon Valley, right?

01:00:54.620 Yes.

01:00:55.320 They say effective altruists just because I think people don't realize, like I talk about, I think rationalism is the intellectual root of the killer superintelligence AI that spread into effective altruism.

01:01:07.300 But effective altruism is just massive, right?

01:01:10.160 Like, it's just so much bigger in terms of, like, money and influence and rationalism.

01:01:14.700 So that's probably why they're—

01:01:15.800 Well, that and people know what it is, right?

01:01:17.780 You know, like, it became famous when SBF had his fall from grace.

01:01:21.980 Yeah. 0.98

01:01:22.120 Whereas there hasn't been—I mean, I guess the Zizians are a thing that made the rationalists a little more famous.

01:01:28.160 Way more people know about SBF than the Zizian murders, but—

01:01:31.500 Who, as you reported, actually had pledged money to Yukowskis, Miri, and they had to give it back. 0.88

01:01:37.180 So it's all the same world.

01:01:39.020 Yeah.

01:01:39.120 But so it, so obviously that's who's pushing back because I think like the all in Silicon

01:01:44.160 Valley, uh, right.

01:01:46.260 Yes.

01:01:46.760 You know, obviously it's like, look, we're venture capitalists.

01:01:48.940 And so we don't like restraint on the types of businesses that we're investing in for

01:01:53.820 sure.

01:01:54.420 But it's really, this is what I want to get your take on because it confuses me about

01:01:57.960 like the frontier labs is that it's this weird schism within rationalism and effective

01:02:02.880 altruism where like they're, they're kind of, they're convinced, you know, pre LLMs,

01:02:07.860 superintelligence is bad. It's going to kill

01:02:10.400 us. You know, we have to be

01:02:12.180 we don't know how to do it safely yet.

01:02:14.020 Like, we got to be really worried about it.

01:02:16.260 And then these labs

01:02:17.580 kind of were forming to be like,

01:02:20.040 we'll just be researching the question

01:02:22.220 of like, how do we control it so that like

01:02:24.260 as this technology advances,

01:02:26.220 let's try to get there. And then at some point

01:02:28.220 they just sort of decided, OpenAI

01:02:29.880 and then Anthropic, of like,

01:02:32.000 I think we'll be able to control it. Like, let's put 1.00

01:02:34.060 the foot on the accelerator. So there's like a schism in

01:02:36.040 that world where now like yakowski really dislikes dario amade probably right like why why are you

01:02:41.460 still going so so it's complicated so now like if you're the all-in um you know silicon valley right

01:02:48.000 and i'm referring to the podcast that like yeah i know yeah you're like well we kind of like these

01:02:52.120 companies because they're hyper growers and that's what we want we need you know we need the ability

01:02:56.120 to we need trillion dollar valuations yes like this is what we need but also they're affiliated

01:03:01.100 with these people that are trying to slow things and they're kind of slowing themselves down and

01:03:04.200 And so it becomes a very, I think that's too confusing for most people.

01:03:08.540 That's just too confusing now.

01:03:09.660 It's like, are they right or left?

01:03:10.920 I don't understand.

01:03:12.160 That's too confusing.

01:03:13.560 I give up.

01:03:14.600 Let's just interview Jacob Croxon.

01:03:17.100 Yeah.

01:03:18.180 No, I mean, I have a set of signal chats around this.

01:03:22.280 And one of them, I think just a couple of days ago, one of my friends said, I've got

01:03:28.320 sense-making fatigue, which is a term that I'm absolutely going to steal.

01:03:32.300 Sorry about that, Molly.

01:03:32.980 Um, but I guess the way I see it is, I don't think that these companies are going to slow

01:03:38.720 down willingly.

01:03:39.540 There's too much money at stake.

01:03:41.920 Yes, they are in the grip of this ideology, but the ideology can be used to justify going

01:03:49.260 as fast as possible, almost as easily as it can be used to justify trying to halt everything.

01:03:54.420 And so I think they're going to keep racing toward this goal that they're convinced exists.

01:04:00.240 It doesn't exist.

01:04:01.340 So they'll just keep racing.

01:04:02.980 And if somebody asks, why don't you slow down?

01:04:05.140 They'll keep pointing to China.

01:04:06.720 Yeah.

01:04:06.880 And of course, China is not really engaged in this because AGI and the, you know, ideology

01:04:13.380 of technological salvation, these are cultural phenomena and they just don't carry the same

01:04:20.080 kind of weight in China's tech culture.

01:04:22.640 Did you see the op-ed that the head spy chief in China wrote an op-ed in domestic papers

01:04:28.240 about his concerns about AI?

01:04:29.800 and there it's all it's misinformation yep it's cyber security you could use this to hack it's

01:04:36.500 you know it's it's all of like sounds a lot like the ai ethics people here in the stage yeah it's

01:04:41.800 all of the proximate uses nowhere in there was it's gonna kill us all yeah no i i didn't see

01:04:47.340 this you should send it to me it's getting reported on now because obviously it has to be translated

01:04:50.820 but it's being reported on in translation now yeah yeah i'm not surprised to hear that at all

01:04:54.840 I mean, I have friends who, you know, are experts on tech in China who I should really

01:05:00.480 bring on here.

01:05:01.300 You know, this is what they've been telling me for quite some time.

01:05:03.620 I think this is a good sanity check, right?

01:05:05.500 If this technology could kill everybody, a lot of people would be noticing this.

01:05:10.100 A lot more people would. 0.65

01:05:11.140 China would be noticing this.

01:05:12.120 Europe would be noticing this.

01:05:13.080 All these academic labs that are deep in LLM research, you know, they would be noticing

01:05:17.220 this and be incredibly worried about it.

01:05:18.720 all of the many sort of big tech companies

01:05:22.180 investing in hyperscaling

01:05:23.320 that aren't founded out of rational EA,

01:05:25.980 they would be very worried about it.

01:05:28.040 You would have,

01:05:29.440 there's not some secret thing

01:05:30.900 that these labs are doing

01:05:32.660 that other people don't know about.

01:05:33.780 They're just running like less supervised agents

01:05:35.780 on top of an LLM longer, right?

01:05:37.320 There's nothing in there.

01:05:38.120 There's no secret sauce.

01:05:39.560 There's no like,

01:05:40.120 we're the only ones who have this.

01:05:41.800 So I don't know.

01:05:43.100 I don't know if that's comforting or not,

01:05:44.160 but to me, it really is.

01:05:45.280 It's like people would take existential threat

01:05:48.720 incredibly seriously. And yet this is really concentrated only in people who are affiliated

01:05:55.080 with this particular type of philosophy. Yeah. And I'm just seeing now that I'm about to have

01:05:58.780 to run. So I don't know if that's a good place to leave. You got to go. Yeah, no, I was I the

01:06:04.080 only other thing I was going to say is, you know, you know, when when places like OpenAI say that

01:06:09.920 their IPO is going to have to be delayed because of this stuff, I don't believe them when they say

01:06:14.280 that i i take some hope from that thinking that actually it means their financials are not where

01:06:18.820 they can't file an s they can't file an s1 right now yeah if they had to reveal their financials

01:06:23.380 it would be the ipo would fail good that's why they're delaying their ipo now that's a good

01:06:28.720 place to leave it because that gives us hope for the future so uh cal thank you and um we're gonna

01:06:35.180 have to bring you back because i still want to talk with you about deep work oh we got so much

01:06:38.400 to talk about yeah and ai's effect on deep work come on we have whole exactly yeah yeah we'll do

01:06:42.920 round two. Or maybe I'll have you on my show. We'll do round two

01:06:44.980 over there. Oh, yeah. That would be great. But yeah,

01:06:46.980 good to talk, Cal, and talk more soon.

01:06:49.060 All right. Thanks, Adam. Thanks again

01:06:51.080 to this week's guest, Cal Newport.

01:06:53.100 Next week, we have Megan O'Giblin,

01:06:55.360 who is the author of one of

01:06:56.980 my favorite books, God, Human, Animal

01:06:58.920 Machine. And she's also

01:07:00.920 a friend. And we had

01:07:02.980 a wonderful conversation, and you're going to enjoy

01:07:04.960 it. So, see you next week.

01:07:08.680 To submit questions for

01:07:10.820 future guests and to suggest other guests, and to see more pictures of Babka, join the conversation

01:07:17.200 on Patreon. You can also find us on YouTube, on Instagram at DATMPod, on the web, and on Blue Sky

01:07:24.380 at DreamingAgainstTheMachine.com, or just find us wherever you get your podcasts.

01:07:30.000 Dreaming Against the Machine is a proud member of Multitude Productions. Our executive producer

01:07:35.200 is Nick Karisimi. Our theme music is by Jared Emerson Johnson. Our show logo is by Nick James.

01:07:42.660 And our fearless leader is Babka, the greatest cat in the observable universe.

01:07:47.480 I'm Adam Becker, and I'll see you next week.

Dreaming Against the Machine - September 22, 2026

Episode 24： The Myth of AI Doom, with Cal Newport

Episode Stats

Transcript