Brennan: Okay, welcome back for the first time, you know, my annual podcast. So we've been having a very interesting conversation about a lot of things I've been thinking in depth about lately. So we decided to record it. We'll put this on the internet. The yeah, if you also want to have deep conversations with me and have them on the internet, let me know because I'm really bad at reaching out to people to schedule these kind of things intentionally. Anyway, back into it. Why don't you introduce yourself and give your summary of what we were talking about and then I'll give mine and we can get back into it?

Uli: Yeah. So I'm Uli. I was homeschooled autodidact currently kind of rationalist, EA adjacent, interested in solving AI alignment and making smarter than human AI go well. Our argument, like our discussion so far has been around whether it is better to proliferate AI systems as widely as possible to avoid concentration of power into one central dictator as like a threat, which is Brennan's central threat model, concentration of power. Whereas my central threat model is a sort of genie, which you are not able to get to the thing you want it to do. And in this model, you would not want everyone to have a genie, which is like very deceptive and kind of hard to use or everyone to have the ability to summon one of these genies because in the physical realm, like attack is easier than defense in the limit. And so you'd only need one bad actor to like destroy the world or something.

Brennan: Okay. Yeah. So, I mean, we had a great conversation initially where a mutual friend introduced us. And so we were sharing, we have very similar sort of backgrounds in being unschooled for large parts of our childhood and finding our way into computers and so forth, teaching ourselves math, that kind of thing. But then it got into this because Uli here is a, identifies as a rationalist EA, would that be fair to say?

Uli: I don't identify as it anymore, but I'm probably like closer adjacent than you are.

Brennan: Okay. At least has had like, you know, connections with the infamous Berkeley circles. As I have not. Yeah. So basically the, I guess I'll present my idea generally about sort of AI and alignment and why I think a lot of the rationalist EA trends to be somewhat misguided and potentially dangerous in and of themselves, like inherently. So I guess I can start with the sort of SBF crash, very famous where, you know, you had this group of EAs, you know, sort of professing that, oh, you know, we want to make the world a better place. We want to, you know, do all this. And they ended up causing a lot of harm, right? They made the world, I would say, I would say demonstrably and relatively uncontroversially worse place by the actions they took and sort of the compounding frauds that they ended up piling on top of each other and eventually turning into spaghetti. And I think one of the, like a very revealing quote comes from Caroline Ellison, who was one of the head honchos at FTX there reading it on her blog on February 9th, 2021 wrote: "is it infinitely good to do double or nothing coin flips forever? Well, sort of, because your upside is unbounded and your downside is bounded at your entire net worth. But most people don't do this. Those people are lame and not EAs. This blog endorses double or nothing coin flips and high leverage." So as we hear, Uli clearly agrees, like this is not logical, right? You know, when you, when you blow something up, you can not only cause harm to yourself and go to zero, you can vaporize the deposits of many people who trusted you with their life savings. You can discredit yourself and your movement and anyone associated with your movement. I mean, you know, the list goes on. And I think this is sort of just highlights a core epistemological flaw with a lot of effective altruism, which is that it's sort of, I mean, it's the same problem with where we utilitarianism, which is its sort of parent, which is that you sort of presume like it's utilitarianism. Basically the idea is you have this loss function, you have this function that determines how much good there is in the world. And then you act in ways that are rational to optimize that function. Which I think is like not obviously wrong. I think the problem is when you're, when you don't acknowledge your limits in or the limits in both, like when you treat it as a math problem, right? You don't acknowledge that your function might be wrong or made up. You might not be able to measure your function accurately. Things that you predict to move the needle in one way or the other, do not actually move it in that way and may move it in the opposite way and so forth. And so I think this, like the FTX blow up is kind of emblematic of this problem in that, you know, you have people who are like, okay, we got it. Double or nothing coin bets all the way. And so they pile fraud on top of fraud on top of fraud, you know, being like, if we win, you know, I don't know, presumably world poverty is safe or whatever. Like they never, I don't, I might be wrong, but I don't think the FTX folks ever really specified how they would turn billions of dollars into net good for the world. And if we're bad, you know, like it's zero.

Uli: By taking over the world, obviously.

Brennan: Yeah. You know, they would make all the right logical decisions and rule the world well, as opposed to the evil people who are choosing not to rule the world well or something like that. And so it's like, I think at the root of effective altruism and the movement is just sort of this epistemic arrogance, this idea that, oh, you know, we can assign numbers to these things. These numbers are correct. Our actions will actually affect those numbers in the way we think they are. And then, I mean, this is all well and good, right? You can optimize this stuff. But I think the problem is it played into almost a dogmatic overconfidence, you know, in this. It's like it's sort of the saint mentality, right? I think there's a great quote from like C.S. Lewis on this, which is that, you know, the you hope to never be like oppressed by a good person because, you know, the good person will do all the things the bad person will do to you, except they'll do them with the approval of their own conscience, right? And I think he was talking probably indirectly about the Soviets or, you know, related people at those times. But the core idea is, you know, and this goes back to, I think obvious examples are like Soviets, you know, it's people who believe they're doing something very good and they believe it with all their heart and they put everything into it and they were wrong and they cause a catastrophe. That's the danger I think with EA is sort of this fundamental like root level danger. OK, so we discussed that. And then I was like, OK, so this is also why I think we should proliferate AI very rapidly and very almost unqualifiedly. Like as soon as these tools are created, I think they should be released as widely as possible with as few restrictions as possible, really. And the main reason for that is I'll let you present this argument because I don't want to misrepresent it. But I think the general shape of this argument is, hey, AI is going to be very powerful. We don't really know what that like that power might do bad things. So let's not release it. Let's not proliferate it. Let's keep a tight rein on it so that we can make sure that we're like sort of handling it well. And I think that this mentality, which I at least generally associate with the label of AI safety, you can agree with that or dispute it, is I think actually the biggest fundamental danger of AI, right, is the danger that we centralize these very, very powerful tools into the hands of someone who becomes a god emperor, right, able to reshape the world at will. And I think that the existence of god emperors is inherently bad, even if they're like the best person to ever live because of the nature of the centralization of power and because of sort of the limits of human fallibility and ability to do things. I think that the analogy I draw on from most strongly is actually from this, which is the Mistborn series by Brandon Sanderson, right? Have you read it?

Uli: Yeah, I love that.

Brennan: Yeah, so it's basically the idea of the Lord Ruler. One person literally eats something, which, you know, spoiler alert, skip a minute ahead if you're... But yeah, it's like one person eats something and they literally have the power to like move the planet around and reshape things, and their minor mistakes sort of cascade and it's just like it illustrates the inherent problems of centralized power. And so I think that the biggest danger that can come with a new technology is that it is centralized unaccountably and uncontrollably into the hands of one person, even if that person thinks they are doing good with it. And that therefore the best thing we can do for like practical AI safety is to counteract that by making sure that... It's sort of the second amendment argument, right? You know, if everyone is armed, then nobody is armed. If everyone has access to state of the art AIs and LLM models, then no single person can do horrible things with them because people will be able to resist in like power and with like terms. Is it a good time for you to take over?

Uli: Yeah, sure.

Brennan: Cool.

Uli: Yeah, I'm going to move outside so you don't hear the microwave. Yeah, so my kind of disagreement here is around like attack being easier than defense. Okay, so there's...

Brennan: Oh, you're lagging out. I think the Wi-Fi is very bad outside where you are.

Uli: I expect.

Brennan: I think the Wi-Fi is very bad where you are. I can't hear anything you're saying or see any of you. So you should probably move back near inside.

Uli: Well, now?

Brennan: Yeah, yeah.

Uli: Okay, yeah. So I would say like attack is kind of easier than defense in the limit for physical systems. For digital systems, it seems like defense is easier in the limit because of encryption. But for physical systems, it seems very easy to like, for example, some sort of like self-replicating nanobot swarm or like an anti-matter bomb or something else I can't think of. So in my mind, what happens is if you proliferate like digital assistance and like AI to everyone, it will go like great for a while, then eventually we'll get to the point where like everyone has sort of smarter than human AI, which is sort of like genie-like AI that can really do transformative things. And then it becomes kind of difficult to stop bad actors from just like destroying the world. Like there are some people who are irrational, suicidal, even if it is not in their best interest, if there's a way that they can like destroy the world if there's 8 billion people, like this is going to happen. So this is kind of my like long run objection to this. And then another thing I would say is like, this is kind of assuming we are able to solve the alignment problem. If we are not able to solve the alignment problem, then we have an even bigger problem, which is everyone has these like really powerful super weapons and like digital assistants, which they can't even accurately point at the thing they want. Like they'll say something like, what is like, what is the like, they'll ask it a question and it will instead try to convince them of an answer instead of like helping them figure out what's true, because it is much easier to train a system based on the loss function, based on convincing someone rather than helping them figure out what is true, which is like a hard thing to measure.

Brennan: So to your first argument there, sort of the bad actors, to what extent is this like, like it strikes me as almost a general argument that could be used against technological progress. Is it not? Like I don't see how it's connected to AI specifically, other than that, through sort of so-called AI agents, we might be able to move more swiftly towards a technological future with more and more powerful weapons.

Uli: Yeah. It's an argument for differential, like differential technological progress. Like I think there's some technology which just like helps defense and there's some which kind of helps both and so much like helps attack. This framing is kind of just like, like wartime framing, but it's like the framing I'm going to use. I think that like the, like, I think the main, like, I think humanity is in a very unstable state right now. And I think we're going to like the ratio of like attack to defense is going to like keep increasing with technological progress. And I think like the main way we solve this is by making some sort of aligned super intelligence or like a centralized thing, which stops us from like fucking everything up through this like dichotomy.

Brennan: How would that do that? By like, so one way, one way I can imagine is I think I've heard it proposed by like I believe it was Nick Bostrom that like everyone, like we basically turn the world into a panopticon, right. Where there's, you know, like constant audio and video monitoring of every human on earth. And we have AIs analyze that and that like, that doesn't seem to me to be a good world.

Uli: Yeah, it's not. I think there's like much lighter things that you can do. You can do like, like one example of a thing we already do is if you have like printers for like bio things, you can't just print smallpox. Like it will just refuse to print it and call the FBI at your door. I'm kind of visualizing something like this, but like broader for like other kinds of like dangerous technology, something like this, which is like managed by some sort of intelligent AI recognition, which recognize it when you're printing smallpox-like things, not just like literally smallpox.

Brennan: But how would it categorize that? And how could we, how could we know how to adequately, because like, so for instance, there's no way for us if we had never had smallpox around, right. If smallpox had never existed, how could we possibly program a bioprinter not to make smallpox? Like we can't like program a bioprinter today, for instance, to say, don't make any very dangerous contagious disease because there's knowledge there that needs to be discovered to make one.

Uli: Yeah. Yes. I think with like more advanced biological knowledge and like AI simulations, you could potentially make something which categorizes like wide class of like dangerous diseases and refuses to print them. For example, we already have a sort of similar system, which was used for drug discovery. And it turns out if you flip the sign basically of the loss function, instead of like discovering helpful pharmaceutical drugs, it will like develop like biological weapons.

Brennan: Yeah. Which makes sense.

Uli: Yeah. Like it came up with mustard gas. It came up with a few other things, which like were not known to humans. Not good.

Brennan: Not gonna test.

Uli: Well, like looking at it with a chemistry lens, it seems like they do things similar to mustard gas or something. But yeah, so this is sort of generation. I'm thinking you could probably do a similar thing for detection and then zooming out. You could probably do a similar thing for like AI in general.

Brennan: Wait, wait, wait. I don't think that's a trivial step at all. Why, why would being able to do it for generation necessarily follow that we could do for detection?

Uli: Because my model is something like the hard part, like generation and detection are like kind of dual problems in like a Bayesian frame. It's literally the like posterior and like you just flip it's the posterior and the like prior or sorry. It's like you flip the likelihood versus the like probability. Generation is like probability of data given hypothesis and like detection is like probability of hypothesis given data. So in a mathematical sense, these are very related things.

Brennan: But, but, but like reality isn't mathematical in that sense, is it? Right. Like the way I see it is it's like generations. So I tend to follow a Deutschian model on generation, which is like, you know, generation consists of new explanations. Right. So like as we come up with new things, almost by definition, if it's new, we couldn't have predicted it. Like, nobody could have predicted the invention of the wheel, because if you can predict it, that prediction is the invention. Right. And so I find it like, well, I think, I think there's almost like a two track discussion to be made here, which is like one, to what extent does the capacity to build something imply the capacity to detect it? I don't think that that necessarily follows at all. And then also like, so it seems like this idea of like the extent to which we would want to build a detection system follows from this premise that more technology means more danger means the need to centrally control and mitigate against that danger. Is that a fair representation of your argument?

Uli: I would say that central does not imply like there's a human at the center. Like ideally.

Brennan: I'm not, I'm not, I'm also not implying that. I'm just saying power is centralized.

Uli: Yeah. I just wanted to note that in my worldview, the ideal sort of outcome is a major AI lab figures out how to solve alignment and creates like some sort of aligned sovereign, which doesn't like form dystopias and like preserves humanity's power while preventing us from destroying ourselves.

Brennan: What do you mean by doesn't form dystopias? Like how does one quantify? Cause I would argue that—

Uli: Basically, basically like very conservative, just like intervenes such that humans just don't like wipe themselves out through like various more advanced technologies.

Brennan: Well, how does it know that any given technology has the potential to wipe people out without like simulating it ahead of time and implicitly making it?

Uli: Simulating is like, yes, it would simulate ahead of time and it would implicitly make it, but like it's kind of a thing like two can keep a secret or like three can keep a secret if two are dad. Like you want to centralize power when attack is much easier than defense and you have like a lot of powerful technology.

Brennan: So the argument here is in order to prevent dystopias, we must give central control to something which just randomly forbids humans to do things? Like I realized that sounds uncharitable, but that's what I understand the argument to be.

Uli: Yeah, sorry. Yeah, sorry. No, the argument is to prevent like human extinction, you have to have centralized control and it seems reasonable that you can do centralized control without forming a dystopia. Like it seems pretty light handed to just only intervene when humans are going to like destroy, like cause some like very large catastrophe. Like in my model, this is unlikely. I think a framing for the like attack being easier than defense is like, if you could give everyone a button that would like destroy the world, how many people would you want to have this button? It's like not many, you'd want like one or zero.

Brennan: So the, I just want to try to understand and faithfully recreate the argument.

Uli: Yeah.

Brennan: So the argument is that continuing technology will eventually give everyone the analog of something that is a button that will destroy the world, and that the wide accessibility of AI agents and tools will be that button, or sorry, will accelerate the technology such that that button could be created. That if you used another AI agent, it could, and you like fed it sufficient data, so like, it could accurately predict when someone would use one of those AIs to create that technology, one of those world destroying technologies. And that therefore we should go and we should try to build this sort of like, as you said, like sovereign, like AI to, to sort of, and like form the panopticon to prevent other people from using AIs to make potentially world destroying technologies. Is that a faithful representation?

Uli: Pretty faithful. I'm not sure what the connotations on panopticon are. It's like, I haven't heard the term before.

Brennan: Yeah. It typically has negative connotations. So a panopticon is a circular prison with a guard tower at the center.

Uli: Yes. I see.

Brennan: And basically the guard tower can see into every single cell. And so you don't know whether you're being watched or not. It's a very negative connotation. I, a more neutral connotation would be... I mean, it is sort of like just a, it is a surveillance state, right?

Uli: It's not a surveillance state because like, it's not really humans or a power seeking class, like ideally.

Brennan: But, but, but, but it is, it is a state, an ultimate sovereign power that is surveilling everything.

Uli: Yes. But like the connotations of that, like, this is a non-central example. So like you can't, this is like saying abortion is murder so like abortion is bad. Like abortion is very non-central example of like the murder thing.

Brennan: Well, yeah, no, no, I get, I get the non-central thing, but like granting that it might be, it may well be a non-central example. It is a surveillance state, right? It's just like the good kind of surveillance state potentially.

Uli: Surveillance state, like it doesn't seem like a good thing to describe that like surveillance state just has very different connotations. Like I would say the good kind of surveillance state, but like this still has like kind of weird connotations. Like it's really just, it's really just a computer program, which like stops people from printing smallpox and like other things.

Brennan: Would it prevent people from discussing the elimination or taking out of control of itself?

Uli: I think probably like, I think you want some sort of minimizing of impact and this seems bad. I think it could like just stop you many other, like the details of the safeguards are up for debate. You might want to have something like 90% of humanity can agree to like shut it off and then it has to shut off.

Brennan: But like, so say this thing, right, believes in its super intelligence or whatever it is, right? It believes that it is critical. It's convinced itself of your logic, in other words, it believes that it is critical for humanity, right? And that therefore an effort, like it shutting itself off would doom humanity, right? Therefore, would it let the memes spread? Like would it be responsible on its part to let the memes spread to 80% that would be necessary to vote to shut it off?

Uli: Yeah, making an AI which is like okay with being shut off, like it's, it's like a kind of major technical challenge, which I expect is like a bit harder than just making one which is never okay with being shut off. So like making one which is sort of, sort of okay with being shut off, but doesn't like doesn't want to be shut off and doesn't not want to be shut off. Seems kind of hard to me, like corrigibility or something. So, this is hard. Ideally, we would have something like this. But yeah, I don't know how to build this. I also don't know how to build the like aligned sovereign either but it seems like marginally easier.

Brennan: So but this. So, yeah, I mean stepping back from sort of the technical detail of how one would build it. This is the goal, more or less, right to have a system, which we assume can watch over humanity and detect when we would be doing something that would lead us on the path to catastrophe and intervene?

Uli: Yeah, my model of like, my model is something like there's a probability that someone destroys the world every year, and it is like somewhat increasing with technological power. And if something doesn't happen, like eventually this like extremely low probability thing will happen. And so, humanity is just in the state, unstable state where there is a non zero probability of like people destroying the world. And we have to either get into a steady state where the world is destroyed or a steady state where the probability of destroying the world is just like infinitesimally small through some sort of like coordination mechanism, like the centralized AI sovereign. Which is the most plausible method I see.

Brennan: Okay, so to restate that. Yeah, at some point, probably some point in the 20th century, humanity for the first time developed the ability to destroy the world. Right. And yeah, as you say, you know, as this limits to infinity, this either has to, we have to destroy the world, or we have to no longer have the capacity to destroy the world. Right. And then this is one, and the most likely way that we could get into a steady state of not destroying the world is creating this sort of like omniscient sovereign overseer that prevents bad things from happening ahead of time.

Uli: Yes. Or like, yeah, basically. Yeah.

Brennan: Okay. So what, what my gut is telling me is that this is a bad thing, like that we shouldn't build a omniscient overseer thing that like, because like, how could we possibly, you know, like, seed, like, it seems to me, I don't have a particular rationale for this yet, but it seems to me incredibly, like, epistemologically almost impossible to have a function that can detect whether a certain action would potentially destroy the world. And almost in the limit, I mean, so the third option, right, is the Kaczynski argument, right, which that if we just blow up all the computer scientists, and return humanity back to the, back to the pre-container age, that's also a way of sort of avoiding the inevitable pending catastrophe, right? And so I worry that if a AI is given, you know, presumably the loss function of minimize the probability that humanity destroys itself with technology, the best way to do that is to revert to a pre-technological state and enforce that at technological gunpoint, which seems to be quite a dystopian outcome, potential outcome for something which has the ostensible purpose of preventing dystopias.

Uli: Yeah. So building, like, the AI, which minimizes probability humanity destroys itself, but also has some sorts of like deontological constraints around what it can't do, is like, seems like a pretty hard technical challenge. Like the modern paradigm of AI—

Brennan: I mean, it doesn't even seem like a technical challenge, does it? It sounds like, seems like a philosophical challenge.

Uli: I think for practical purposes, my model is something like, you can get the, like, without creating a dystopia, you can get probability humanity destroys itself down to like 10^-9. And then if you allow for dystopias, it just goes down to 10^-10. And that like, most of the benefits you get are having the aligned sovereign in the first place, not the like final stretch when you go from like, supervised, like, freedom to like actual dystopia.

Brennan: But again, also... sorry.

Uli: Yeah, I think more central is like, you said, it seems hard to detect like, what sort of things would destroy the world. I think this is not the case. Like, first off, you have to assume this thing is smarter than us. Second off, I think if I could have observed like all humans at all times, I could probably stop the world from being destroyed. That's like pretty high probability.

Brennan: Why is that?

Uli: Without like, okay, so like central examples, like someone printing, like, someone would basically have to have perfect opsec, like, assuming the AI is not allowed to read their minds, which it potentially could, but we might want to disallow that for like, various reasons.

Brennan: Well, like, like, like, if they, if they intend right to destroy the world. What, how would you prevent someone from accidentally destroying the world?

Uli: Like, have, have the AI have like decent models of like, like physics and like, sort of things like that. And like, destroying the world is like pretty tough. It seems hard to do by accident. Yeah.

Brennan: But say, say for instance, okay, so say there's like a game theory thing, right? Where it's like, there's some butterfly effect, like, so, so look at World War I, right? World War I spiraled into existence because you had these sort of like building rival technological powers, England, the USA and Germany, right? And they were all getting to these, this technological progress, building up their armaments, they had all kinds of treaties everywhere, right? And then one guy decided to assassinate one other, sorry, like three guys decided to assassinate one other guy and succeeded. And that led off this whole giant catastrophe, which ended up blowing up, right? It seems to me that many significant things in history follow from very, very insignificant, little things that gradually affect one thing and another thing. And it's a big probability to the point where it seems like, I find it very, very, very improbable that it is at all epistemologically possible to say, oh, this will lead to something bad, right? You know, cause like nobody, like, obviously nobody wants to be the president pressing the nuke button, but the president is not pressing the nuke button lightly. Like the president's pressing the nuke button because of hundreds or thousands of small things that have led up to that crisis, right? And so it's like, it strikes me as nearly impossible to prevent a buildup to a crisis moment categorically without routine and like ridiculous levels of interference in personal lives of like trivial matters on the, I mean, let's face it, like hypotheses and conjectures of such a machine, because it couldn't simulate all of humanity and all the butterfly in sufficient detail to assure things. So it'd have to be operating on hunches. And again, I think at the same time, I think this comes back to the same like sort of EA fallacy that I was like talking about in the case of the FTX and like in the case of like, you know, shrimp welfare maximization, which is just this idea that one, there is a such, I mean, I think it is pretty much, there is such a single number as like the chance humans extinct themselves. I would grant that like we can either extinct ourselves or not, or somewhere in between, like we might, but we might not. But I think it is ridiculous to be able to just assume and assert that we can measure that probability.

Uli: I didn't say that.

Brennan: Well, like how would you minimize something unless you can measure it? Right. You have to be like actually smart. You can't just like,

Brennan: I don't think it's an intelligence issue. I think it's a epistemology issue, right? I think it's an issue of what you know and don't know.

Uli: There are lots of things that we don't know that we can like figure out from other variables. Like I would claim that the probability humanity destroys itself like increased from a subjective, like limited information point of view when we invented nuclear weapons.

Brennan: Why?

Uli: Would you disagree? What?

Brennan: Why? Okay. So here's another story I just made up on the spot, right? Without inventing nuclear weapons, the US and the USSR probably would have gone to war in like the 50s over Eastern Europe. We would have had World War III then. And the next likely candidate for a super weapon is a biological super weapon, right? Both of which like the US and the USSR had active bioweapon development... I would argue that a 100% lethality with like a six month incubation period super virus is more dangerous to humanity than nuclear weapons that we point each other. And if we hadn't developed nuclear weapons, the increased, like the development, the likelihood of developing that would have gone up because of that tension. Right? So maybe developing nuclear weapons actually lessens the chance, right?

Uli: Yeah. I retract my previous point and update against trying to make arguments like that.

Brennan: But that's my point, right? My point is like, we cannot see the branching probabilities of history, right? We cannot say, oh, this will be bad because we don't know the counterfactual and we cannot know the counterfactual because the counterfactual involves invention, which is something that is inherently unpredictable, right? And so I think that the idea of just even the idea of the verb of acting to minimize extinction risk is in itself contradictory and incoherent. I don't think it is something that can be done meaningfully. And like if we gave this what I believe to be an impossible task to a empowered sovereign, which also incidentally has the goal to keep itself in power, all we're doing is we're elevating a dystopia to the throne.

Uli: Okay. So my original model for this was something like you can drastically reduce the probability the world ends by like disable all nuclear weapons, prevent anyone from printing like very harmful viruses and like, prevent anyone from creating AI, like, self-improving AIs with, like, misaligned values, you know, etc, etc. Like, other, like, self-replicating nanobots that, like, don't have, like, an off switch or something. Various ways that, like, big catastrophes could happen. My idea was, sort of, you don't need to prevent these chaotic strings of events which are, like, impossible to predict. You just need to prevent the, like, actual, like, final thing, which is, like, where the catastrophe happens, which is where you engage with some weapon of mass destruction and it goes badly.

Brennan: But how do you guarantee that the... So, I guess, before I answer the question, I'm going to get at the conclusion that I'm trying to aim for logically, which is that the only way any kind of thing can meaningly be prevented is, like, the complete abolition of technology and technological progress. And that is a bad thing, right? So, for instance, right, I think a very common thing that's cited is the offshoot of this book, which was written by K. Eric Drexler as his PhD, Master's thesis, something at MIT, right? He basically argued, hey, it's possible to make all these nanobots, right? They go down to the atomic level and they can move stuff around. And in theory, they might be able even able to make versions of themselves, right? And in his 1986 book, Engines of Creation, which came before the publication of this technical book later, he... There's, like, one chapter, I think it's chapter eight or something. I don't have a copy of the book in here. I'd have to run and get it. I won't. Actually, I will for dramatic effect, because it's being extra for you.

Uli: I don't have a book, but I have this.

Brennan: Okay. We're back. Hear me? See me? All good?

Uli: Yep.

Brennan: Okay. In this book, Engines of Creation, which was published in 86, in the chapter Engines of Destruction, chapter 11, hello camera, right? He basically proposed this idea of the self-replicating nanobot. Let me find a choice quote from it. You know, success will make a... Yeah, basically... I'm trying to find where he says the problem before he's like, how we could fix it. I mean, this is a sort of boring conversation. Basically, I know what he says in this chapter, which is like, we might be able to make these nanobots that sort of take over and control the world. And he sort of leaves it as a footnote and moved on. But that's the thing that a lot of the EAs have seized on to be like, like you even cited it. It's like, this is a potential avenue of destruction. I would argue that, like, so this wasn't on the table before 1986. As we go on, we will discover new ways to potentially end the world, right? To potentially end humanity. And there's no way we can predict them now, because there's no way we can... Like, again, if we knew them now, we would have already invented them. And so the nature of innovation, the nature of discovery, the nature of progress, will suggest to us new ways to destroy humanity. And maybe it will also suggest new ways that are like unpredictable or incomprehensible to such an AGI. And therefore, the conclusion is, in order to protect, safely protect the human race, we should stop all technological progress, right? I don't see the flaw in that argument. And I fear that a... Like, it's as Asimov said, right? You know, you can prove anything with coldly logical reason if you pick the proper postulates. I think in the same way, if you create a system with an impossible mandate like this, or with a incoherent mandate, and you say, at your core is the attempt to prevent humans from annihilating themselves, it will follow the logic and realize that the only way to prevent humans from annihilating ourselves is to keep us on planet, to take all technology away from us, and to just, like, I don't know, give us, like, dopamine machines or something, which I find to be a intensely disagreeable future, almost worse than any dystopia that's been proposed to date.

Uli: Yeah. A few things. So I do agree that this would be a very bad dystopia. I think the flaw in your argument is that humans can come up with... Let me... You might disagree with how I phrase this. Humans can come up with new ways to destroy the world that the AI will not be able to predict.

Brennan: I think more or less, yeah. I would rather say new technologies, like new ways to manipulate matter and whatever, will inevitably suggest new ways to end humanity, if you will.

Uli: I agree with this. I disagree that, like, the AI will somehow not be able to figure these out. Like, if a human is able to figure these out, the AGI, kind of, by definition, should also be able to figure these out. Like, if you put... If we had the sovereign AI before Drexler wrote about nanomachines, and then Drexler wrote about nanomachines, like, the AI is, at a minimum, going to be able to, like, keep up with Drexler, probably, like, figure out the nanomachines thing, like, before him, because it's smarter than human intelligence. Or, sorry, the very... Wait, I don't want to say lots of points which are different. My main objection is, if humans can come up with new ways to end the world, the AI will also understand these and be able to prevent them.

Brennan: See, for me, it's the be able to prevent them part. Like, as I said, you know, that just rankles me. Because I think, like, it is hard for me to articulate precisely, but I have a very strong feeling from a lot of my priors around studying the history of technology and studying the history of political power and of history itself that, basically, almost every major development in history has been completely unpredictable ahead of time. And I see no reason why a mathematical system we can build that does things better than humans will somehow magically be able to predict the future. I think it's, like, fun to postulate, right? And to say that, like, you know, assume a system that can predict the future and be, like, great. I think it's also fun to, like, assume Maxwell's demon, right? Which was, like, proposed by Maxwell in the, you know, and say, like, oh, here's the future. I think you cannot know the future.

Uli: Yes, I agree.

Brennan: And I think that making something that's better than humans at thinking and at intelligence also can't predict the future because it's not an intelligence-limited problem. It's just impossible. But, again, the only reliable way to make sure that we don't develop and realize new technologies that could end the world in the future is to stop all progress. Because you could do that.

Uli: Yeah. Can I? Okay. Let me rephrase this. So, you'd say predicting the future is fundamentally impossible because it's, like, a chaotic system. And so you somehow, like, need exponentially more data to predict, like, linearly in the future. And this is just impractical and just, like, sets fundamental barriers.

Brennan: Not impractical, impossible.

Uli: Well, yes, impossible because, like, the exponential is big or something.

Brennan: No, it's because, like, small uncertainties, like.

Uli: Yeah, yeah, that's what I mean. Yeah, exaggerate.

Brennan: Yeah, it's not impossible. It's not impractical. It's impossible.

Uli: Yeah. So, I agree with this. The thing I disagree is that being unable to predict that you're, like, required to be able to predict the future to prevent catastrophes. I feel like. So, would you agree with the statement that you can, if you had a smarter than human AI, you could just expand out the tech tree that humans would expand, like, beforehand?

Brennan: No.

Uli: No? You disagree?

Brennan: Yeah, I mean, same deal, right? It's, like, you can't predict what humans would discover. You just can't.

Uli: I think this is fundamentally different. I think the, like, dimensionality of the useful tech tree is, like, pretty small, like, compared to the dimensionality of, like, weather. Dimensionality is, like, a vague term I'm using, but.

Brennan: I know about it, yeah.

Uli: Yeah, yeah. I think the amount of useful directions you can go for technological progress is, like, a lot smaller than the amount of chaos that is in, like, the weather system. I think predicting, like, basically solving most forms of technology, or at least getting ahead of humans in, like, all the directions they're going to go, is a lot more feasible than predicting the weather, like, a year in advance, which is just impossible.

Brennan: I'm not sure. Because I think a lot of technological breakthroughs are, they're quite orthogonal, right? Like, so if you think about the Wright brothers, and, like, the invention of flying, right? Like, I mean, famously, the New York Times, two months before the Wright brothers' first flight, published that, you know, all the combined work of the world's mathematicians and scientists will not produce a flying machine, or will maybe produce a flying machine in between one million and ten million years, right?

Uli: Yeah. This may—

Brennan: I think that the history of technology, at least as I've learned it, and as I've experienced it, consistently, over and over again, comes out of left field. And I think it's very easy for us to look backwards and to say, oh, well, you know, this came, then this came. We had the, you know, first we had the transistor, which was really useful, and we, or sorry, you know, first we used vacuum tubes to make a computer, then we were able to make transistors, and then we put them on a chip with an integrated circuit, and then we, you know, took that and put it into microprocessing, and we got it smaller and smaller, and did that consistently for 80 years, right? And I think looking back, we're like, oh, yeah, this happened, then this happened, then this happened, then this happened. But like, that doesn't change the fact that like Noyce was like sitting there and came up with the idea for the integrated circuit, and it blew everyone's minds. Because technology, again, it is the invention of the new, right? It is not the optimization of things forever, right? Like in 1920, you could have said, so literally since the mid 19th century, at least, very prominent mathematicians and scientists have constantly said things to the effect of, oh, well, I think it was in the 1880s, some guy had a quote like, oh, you know, well, we figured out all the fundamental physical laws, all that matters now is, you know, expanding things to like six decimal points, right? And then we had Maxwell, like, right after I might have gotten the chronology wrong. But that's the story, right? You know, in the 1910s, we're like, oh, can't do flying with these machines. In the 1930s, you know, I'm sure many people would have been, would have confidently said what you just said. It's like, oh, well, you know, the dimensionality is pretty small in like where we can go in the physical spectrum. And we can, you know, we can get ahead of it, don't worry. And then Fermi started doing his experiments. And then 10 years later, there was a functioning atomic bomb, right? It's like, you cannot, cannot predict technology. And I think that that is just crucial. And it needs to be just beat into everybody's heads. Because it can't be done, you can't predict it. I think like, so, for instance, you know, even just look at the field of AI itself, we're like, oh, well, you know, in theory, if you take, or if you grant the, the idea that like, there's no, what's it called? The antivitalistic hypothesis, right? Which is the idea that there's nothing special in humans, in humans, and that we're just normal matter, which I happen to subscribe to. But if you grant that, then yeah, as soon as you grant that, you realize that in theory, we can make machines that are as good at thinking and as humans are, right? But since we invented computers, we've been describing them as, you know, brains, and we've pivoted like four or five times on like the path towards them. And it's like, now we have, we have lots of progress in LLMs and even GPTs that are like, I think I saw one, I think like, they're now like multimodal models, which are pretty cool. And it's like, right now we think that's going to be the path. But, and even if this one, this path of AI is the path, and this is the one that gets us superhuman intelligence, I think it only, we only need to look at the history of the development of AI to realize that we have no way epistemologically to predict the future. And that's not a limitation of humans. That's not a limitation of intelligence. I think it's a limitation of epistemology, which any god emperor, AGI will also be subject to.

Uli: Okay. So I have a few things to say to this. So the first is just kind of a joke, which is like the New York times, unfortunately for some listeners, is not an artificially general intelligence, arguably not even an intelligent, but like,

Brennan: I mean, you know, I refer to it as a byword for experts, you know? Which was true at the time.

Uli: Yeah. So I'm kind of thinking you're thinking of like AI in this sense as optimizing some sort of objective function? Like,

Brennan: My understanding is that I'm arguing against the potential feasibility and existence of something that can effectively prevent humans from extincting themselves, even if given access to like a full panopticon.

Uli: Sure. So would you say that like humanity is just going to go extinct in like, what timeframe? Cause like, depends like what probability per, like what probability per century of extinction? And then that determines how long we're going to last.

Brennan: No, I think, I think we get off planet and we're good. Specifically, I think we get, we get out of the solar system and we're good. I think that's, that's the escape hatch, right? Like, and this is sort of the Musk argument, right? Which is that, you know, even if we do great at not extincting ourselves, eventually an asteroid hits earth with probability one. And so we can't all be on earth when the asteroid hits. And I think that's not necessarily true because we might be able to deflect asteroids or protect ourselves from an asteroid strike's impacts or whatever. But I think the larger point is true, which is that as long as humans remain concentrated in one place, so you said, you know, throughout history up until like, you know, 1800ish at the earliest, the probability that humanity extincts itself is zero. I agree with this. I think it's, you know, also probably true through like 1900 or so. And then sometime in the 19th or in the 20th century, that probability crept above zero, perhaps with the invention of nuclear weapons, perhaps with the invention of like early, early bioweapons. I actually don't think nukes really have that potential because there's so many people in random crevices.

Uli: I agree.

Brennan: There's probably a few people like cave diving at any moment enough to repopulate the earth. But like in theory, in the 20th century, we sort of have the ability to extinct ourselves. Like at least we can extinct our civilization pretty conclusively, but we haven't done it yet. And in this stable state, I agree. If we stay on earth, we're probably going to do it at some point. You know, it's just a matter of time. But that's where I'm like, okay, if we get off earth, right, you know, and this is where I very much think the Musk hypothesis and conclusion are generally wise, which is like we should act with all haste to build a sort of von Neumann civilization external from earth. I think that even within the solar system, it's probably not safe though, because two warring factions on either planet can fling asteroids at each other and then go off and then can fling asteroids at each other. And that's pretty much that. So I would say that it probably is when other humans get out of reach in like relativistic travel speeds so that I cannot hunt you down because you're traveling on average faster than light away from me, even if I'm traveling at light speed towards you. I think that is when we reach escape velocity. I think with luck, it will happen this century. With luck, it will happen while I'm alive. I would love to see that. But I think that's our path to it. I think that instituting a global surveillance state would not be wise. If anything, it would increase the risk again of eventual human self-extinction by inhibiting our technological progress, which is the thing that's eventually going to drive that number back down to zero. And for what it's worth, to the extent, I think this is one of the good answers to the Fermi paradox, is that there is this, the number is zero and then the number is zero again, but in the middle the number is not zero. And so that's where I think our moral imperative, I agree with you in the premise that we should get the number down to zero as fast as possible, but I very, very, very disagree that the way to do that is to institute like a hegemonic AI. I think the way to do that is to get off planet as fast as we possibly can and out of the solar system.

Uli: I'm generally in favor of getting off planet. I think this drives the number down a lot. One thing I'm not sure about is kind of self-replicating von Neumann probes, which just like eat star systems. This seems like a physical possibility and in the realm of attack being easier than defense.

Brennan: I mean, if we're talking about humans, then we can outrun them. Right. If we have interstellar travel at like 0.99 C and then we hit to a solar system, we're going to be there pit stopping for like 10, 20 years, and then they're going to send the ships right off again. So even if there's a von Neumann probe wave going around, I think humans at the frontier will outrun it typically. But I don't think, I think earth civilization will probably die eventually. But I think the point is to maintain human civilization for as long as possible.

Uli: Yeah. This seems really simple. Are you relying on any facts of like relativistic travel for the outrunning or is it just like normal like outrunning?

Brennan: Yeah. I mean, you're going in the same line at the same speed, if you're in front...

Uli: Right. Sure. So like the picture in my mind is kind of like all the stars in the galaxy being eaten by like an expanding swarm and humanity just running away from them. This doesn't seem like that good even if humanity is still alive.

Brennan: I mean, in a few centuries, we'll have the tech to send, well, then it'll be intergalactic, right? Yeah. Intergalactic probes. I think I'm like at this point, it's like so far I'm not really worried about it. But I think that there are basically two conclusions, right? One is that we die on earth and one is that we don't die on earth. And I think that if anything, the invention of an AI controller, despot type would serve to keep us on earth and limit our abilities to get off earth. What I would rather do, and I don't think actually since we started recording, I really expounded my vision. I've just been kind of a wet towel here or a wet blanket. I believe in sort of, I believe that AI is like just another technology, right? It's a very powerful thing that will give us a lot more leverage. But I think that the way to handle it is like most other technologies, which is sort of to just proliferate it really fast and enable people to then be more effective with it. And because like, so if only one person has a gun, right, it's sort of like the free first strike principle. If you're the first one to develop some kind of new weapon, you get to drop nukes on Japan and nobody can do anything about it. I think in a similar way, the most likely harms from AI are going to come when the most powerful AI models are centralized in a very small group of people who have the ability through that AI's leverage to inflict huge amounts of impact and harm, but who can't be realistically countered because nobody else has these things as strongly as they do. So I would say, you know, the minimum time between something being invented and something being spread around more, I think the lower that gets probably the better. I think the point, again, I don't remember if I made it on recording, but like, I would say the likeliest danger from AI is the danger of this sort of like central despotic government that is sort of upheld by the unique control of AI. And so the best way to avoid that maximum harm is to ensure that does not happen. And the best way to ensure that does not happen is to democratize access.

Uli: Right. So I would disagree with the statement that most, like with most technology, the thing you want to do is proliferate it as far as possible.

Brennan: Sorry, actually, it was like productive technology. Like, yeah, sure.

Uli: I disagree with the statement that AI is just like another tool. It is fundamentally different in my mind when like automating intelligence feels fundamentally different in my mind. Like there is no reason for humans to be playing a central role after we have like artificial, like a central intellectual role after we have general intelligence, like from AI. I think I am...

Brennan: I don't recall asserting that. Or, asserting the inverse.

Uli: Yeah, it seems kind of implicit that your model involves like some sort of, correct me if I'm wrong, cyborgism, where like humans are using AI as tools and they kind of get like gradually more integrated and everyone has this and everyone is raised like by a similar amount.

Brennan: More or less. Yeah. But, but the crucial distinction I think, and the reason that humans will not be replaced is that I believe that AIs are sort of, or I don't really like the term AI, like LLM and like transformer based tools or whatever. Um, they're kind of empty, right? They're kind of empty intelligences.

Uli: I agree with this, but like

Brennan: They don't have, basically what I think of it is it, is it will be like, um, so the way

Uli: I agree with transformers, being kind of empty intelligences, I disagree with the implication that we will not figure out like real intelligence. Like it,

Brennan: But why isn't it real intelligence? I think it'll be intelligence, but, but what I don't think it will be. Okay. So, so here's an example in politics. I recently finished rereading one of my favorite books, which is a book in too many volumes on the life of Lyndon B. Johnson, who was president of the United States, uh, from 1963 to 1968, and was described by his biography as kind of a stupid guy, right? Middling intelligence, totally average. Um, but what he had was, you know, what people might describe as he had that dog in him, right? You know, he had just a drive and an ambition and a desire to rule, to dominate over other people. And eventually what he's most famous for is passing the civil rights legislation that the United States still relies on today. But throughout his life, Johnson used smart people as tools, right? Smart people, like actual human, like not artificial intelligence, real, like, you know, nobody can argue that a human's not a, like, intelligence, like the kinds of people who are like, you know, the smartest people to ever come out of Yale law school. And he used them as very precise tools to do very precise jobs for things he wanted to do, to draft him policy, to write him speeches, to like figure out legal quandaries he was in, right? And he used them over and over again with very much, but again, Johnson was kind of a dumbass, right? This is

Uli: I, I press X to disbelieve. I think this is like, I do not have knowledge of this scenario, but I have like strong priors that, uh, for one thing, outsourcing stuff is like really hard, especially when you don't have domain expertise, like consider trying to hire, like tell between two programmers, which one is a good and which one is a like poser, if you're not a programmer yourself, like you can go off credentials, but this is like,

Brennan: Well, he didn't have domain expertise in like, he obviously had it in like politics and like persuading people.

Uli: It is difficult to solve the like alignment problem with your subordinates. I, also like

Brennan: What do you mean?

Uli: Like, there's a certain sense in which you can't check that the work people are doing for you is actually good unless you like have the domain expertise to evaluate it yourself. Like if you're trying

Brennan: What if it achieves the desired result? And if someone has a history of achieving the desired result, you hire them?

Uli: Sure, sure. Sure. I, this very much, like, okay. So maybe a more fundamental objection is I don't think the, like having the dog in you is like a thing which will at all be like a bottleneck for AIs like, it's just a human, uh, human goals thing that we mostly don't aren't focused on like power seeking over all else. Uh, and I think in general, just like more intelligence for power seeking is like better and kind of a lot better the more you have usually like, I don't know. I'm like, I pressed X, I pressed X to disbelieve that to my, yeah, I pressed,

Brennan: What is it you disbelieve?

Uli: That he's actually a dumbass. This is like an empirical thing that

Brennan: I mean, maybe he's not a dumbass, but he's certainly not a legal genius and a political or, and like a speech writing genius and a, you know, logistics genius and a legislative, like at some point you're able to outsource.

Uli: I feel like at this point, Johnson is just going to be a figurehead. Like if you outsource too much, you become a figurehead. Like how do you maintain control?

Brennan: Well, because you see you're like, okay. Abe, or you're like, okay, uh, write me a speech that gets a lot of cheers. Right. And then they write the speech and you deliver the speech and did it get a lot of cheers or not? If it does, you keep using them. If it doesn't, you fire them. Right. Like there are ways to evaluate someone's thing on its merits. And also like, I think when it comes to political power, there are a lot more concrete, like incentive alignment systems, right. It's like basically, Hey, if I get elected to this office, you're coming with me, you're going to work in my office. Right. It's like, that's a pretty, pretty clear, you know, they have a pretty clear line goal, but that's not my point. My point isn't how he aligns them. My point is that Johnson had access to lots of intelligent people and used them to craft legislation and used them to write speeches and used them to win elections. And he was not good at any of these particular things. And he was, I mean, obviously you have to have some level of intelligence to be able to like use these people, but he's not better than all of them at their own jobs. Right. Um, and a lot of their own jobs included thinking and including, so he went to Southwest Texas state teachers college, which was a two-year teachers college in the middle of nowhere in the Hill Country, west of Austin. And pretty much everyone who worked for him went to the Ivies, right. It's like he was less educated than them. He was probably less intelligent than them, but you know, he had that dog in him. What I'm asserting basically, the rulers of humanity, would you agree with me on this? The rulers of humanity have not been the smartest of humanity.

Uli: Yeah.

Brennan: So, so what I would say is that the ability for humans to lead and to sort of have goals and to organize things and use tools and use people to achieve their goals is separate from intelligence. And it revolves more around desire and ambition. Basically what I think is that we won't have AIs taking over. What we will do is we will have people using very intelligent AIs to do very specific things and eventually use them as clubs and weapons against each other. Right. And, and in this way, they'll just give people more leverage. Right. You know, like I'd sure have more leverage if I had like, you know, 10, like 180 IQ geniuses behind me, ready to do whatever I told them to. And that's where, like, I disagree that an AI can want something. I think that it can be a very useful tool in that it can be used by people or maybe it can even be given goals. Maybe it can be given a loss function. We can say, Hey, your job is to make sure that in this district of Texas, I get a high vote, you know, in this election and, you know, maybe they'd be good at that, but you're giving them the goals. At the end of the day, the goals are originating from the humans. Even if we build, you know, god emperor, you know like Lord Ruler AGI to make sure humans don't destroy ourselves. It is us humans giving it the goal to say, minimize the chance of humans destroying itself.

Uli: This is, I feel like this is a semantic like objection. Like,

Brennan: I don't think it is at all. I think it's very fundamental.

Uli: Okay. I feel like if I give the AI, you know, the joke goal of maximize the number of paperclips in the universe, I suppose that I could somehow specify this. You can say all you want about how like the AI doesn't really have any desires. And you can continue to say this as you're being atomically disassembled into paperclips. It does not like change the actual things that the systems do. Would you agree with this?

Brennan: I would agree with that, but I would also agree that it is implausible that something could turn... So, so I today could have the goal to turn the entire world into paperclips, right? But what is limiting is not intelligence, right? What is limiting is physical control over the world, right? So, to turn the entire world into paperclips, you have to have the ab—

Uli: Wait, no, I strongly disagree. I think the limiting factor. So, I'm just going to use nanotechnology as a frame here. Keep in mind that there are likely other technologies that would be better, or maybe nanotechnology is fundamentally flawed. But as an example, if you can figure, if you can figure out how to solve nanotechnology and like. You can just kind of let it loose and then like nobody can oppose you.

Brennan: So, so basically what I'm saying is that there's not one AI. Like, and I agree if there's one AI, that's very, very, very bad. And that's why I'm saying, yeah, it's like if you have this thing and there are 10 people who have these, right? One of them can say, hey, I want the world to be paperclips and the rest of them can gang up and be like, OK, shoot the person who told the AI to do that and coerce them to tell their AI not to. Or something like that

Uli: I don't think this, I don't think this works. I think attack is like the ratio of attack to defense is so high that if one person defects, you're just totally screwed. Like, maybe you can come up with a way to defend against molecular nanotechnology. But I'm not convinced in general. Like, yeah, like how would you defend against like nanotech self-replicating nanotech swarm in particular?

Brennan: How would a self-replicating nanotech swarm work?

Uli: I don't know, but like, suppose.

Brennan: So I can't come up with a way to defend it against it.

Uli: This is fair, but like, I don't get where your prior comes from that there exists a way to defend against it.

Brennan: I don't get where your prior comes from that there exists no way to defend against it. I think that supposing an unstoppable... OK, so here's OK, I happen to know a bit about nanotechnology. At the very least, they need a source of energy. They need a source of feedstock of matter to consume, and they need some central control algorithm, right, or some way to control at least themselves. You know, in theory, these could be... OK, so where's their power coming from? If it's solar power, you can build a big thing to put in sky on top of them. If if like where are they getting their feedstocks from? Are they tunnelling into the earth? Probably unlikely. You know, you can you can figure out ways to isolate from or corrupt their feedstocks such that they can't realistically use them. How are they communicating? Are they communicate like you can try jamming their communications, right? You can try, you know, hacking into their communication system and overriding the orders or whatever. It's like, there are very very few systems which are unopposable.

Uli: OK, yeah, my prior comes from a mental image from like computer security from like doing computer security and a mental image of like you have a wall and for the wall to be effective, you have to it has to be like secure at every single point. Like an attacker just has to win once. The defender has to win every single time. This is kind of the image in my head. And perhaps

Brennan: Yeah. OK. So say, say, for instance, someone starts a nanotech swarm and now there's like 10 different agents attacking them with everything they got to put them out of existence. Well, those attackers only have to win once and the defender only... because the thing is, anything that is attacking and that is destroying has to have some kind of order and control within it, so it itself can be attacked. Right? Like it goes both ways.

Uli: Yeah, I think my priors. I think my priors are way too strong here from overgeneralizing from computer security, which also maybe computer security doesn't. It's also not exactly like this. Let me see. It feels. It feels like you are somewhat of a disadvantage if your goal is like protect this area versus your goal is just like destroy everything. One of them like sort of has entropy on its side, but like this, this is a different thing.

Brennan: But I mean, you can just as easily invert that. You can say the goals are survive and destroy everything versus the goal of destroy that one thing. Right? You know, the destroy the one thing is is a lot easier to do than surviving all attacks against you and destroying everything. And so this is why I think, again, to keep hammering this point, I think that the biggest danger is in there being one AI, that is unopposable. Because I agree. If there's one AI and it decides to make nanotech, because some human told it to and no other humans have the coordination ability or the similar tools to be able to oppose it, that would be bad. Right. Whether that's possible or not, how fast it would happen, are relatively irrelevant details. But I think the conclusion here is that we should minimize the time to proliferation. Right? The solution is not to say let's have one AI and have it control everything. The solution is to say, let's have lots of AIs so that if one of them gets out of line, or if one human using an AI gets out of line, they can give them a big swift whack and defend against any bullshit they're trying to pull off. That's, just, either way there's going to be risk. Right. But I think there's much less risk the faster we proliferate.

Uli: Yeah. This is very interesting. I want to explore these arguments more and maybe like call another time.

Brennan: Yeah.

Uli: Yeah. Because I feel like I've updated on this.

Brennan: Well thank you for having an open mind. I always, I mean this is... I do like engaging with rationalist types as they are, like, able to be convinced, which is a pretty rare, pretty rare quality.

Uli: Yeah. Yeah, I definitely need to look more into the parts that I've been sort of noticing that I had like magical reasoning around. Like nanomachines, that I did not actually like understand, you know, on a deep level how they work. This is another thing I hope to fix.

Brennan: Okay, coolio.