Neuroscientist Anil Seth: How to Think About AI Consciousness

Licensed under the Unsplash+ License

Will AI ever start to think by itself? If it did, how would we know, and what would it mean?

Anil Seth is Professor of Cognitive and Computational Neuroscience at the University of Sussex and the author of Being You: A New Science of Consciousness.

In this conversation, he talks to Aza Raskin about the science, ethics, and incentives of artificial consciousness.

This is an interview from our podcast Your Undivided Attention, on July 4, 2024. It has been lightly edited for clarity.

---

---

Aza: Hey, everyone, it's Aza. Welcome to Your Undivided Attention. When we go out and talk to people about the AI dilemma and the risks we're going to see from AI, there's one question we always very predictably get asked and that is, will AI become conscious? Will AI wake up and then kill us all?

Aza: We're not alone in fielding these kinds of questions. People who think about AI for a living get questions about machine consciousness all the time, and it takes up an outside space in the public discourse. This question of AI becoming conscious exploded into the news the summer of 2022 when a Google engineer named Blake Lemoine publicly claimed that Google's large language model was sentient.

Blake Lemoine: Google has a policy against creating sentient AI, and in fact, when I informed them that I think they had created sentient AI, they said, "No, that's not possible. We have a policy against that."

Aza: Lemoine stuck to his claims and was ultimately fired from Google. Most experts agreed he was wrong. In my opinion though, "is AI conscious?" That's the wrong question to ask. After all, AI doesn't have to wake up to cause massive damage. AI doesn't need to be conscious to overwhelm our democracies or build a bioweapon. Instead, the right question is, will AI persuade people that it is conscious, and will people then form relationships so strong with AI that they're willing to make dangerous sacrifices for them?

Aza: In the years since, more prominent figures in AI have argued that AI actually is on the path to consciousness though, and companies like OpenAI have taken great efforts to make their chatbots feel like a conscious sentient human. In the end, perhaps the question of, is AI conscious, is so endlessly fascinating because of our own self-obsession. Each one of us has that spark of awareness and fundamentally we want to know, "Are they like me?"

Aza: So that's why we are so excited to have Anil Seth on the show to help us explore these questions. Anil is the professor of cognitive and computational neuroscience at the University of Sussex, an expert on the science of consciousness, and the author of Being You: A New Science of Consciousness, a book I remember laying in bed and reading front to back, cover to cover. So Anil, welcome to Your Undivided Attention.

Anil Seth: Thank you, Aza. It's a real pleasure to be here.

Aza: The question of whether AI is conscious or not has been a nonstop debate for the longest time. The question that I'm sure you always get, that we always get is. "Conscious AI is going to wake up and it's going to kill us. Isn't that the thing to worry about?" So why do we get asked this question? Why is the topic of consciousness almost like irresistible catnip for our minds?

Anil Seth: Well, I'm not surprised it's catnip. I mean, it's been catnip in my mind for a couple of decades now. I've always been interested in this question of consciousness, and I think most of us are, but you're absolutely right that it's come to the forefront in the discourse about AI and perhaps I think very likely to an outsized degree. There are many reasons for that. I think one of them is just science fiction. I mean, science fiction has given us this whole history of stories about machines that are not only intelligent but also aware. And of course, this goes back a long time in culture as well, way before people even had the first glimmerings of artificial intelligence. There's always been this tendency for human beings to try to create things in their own image and to project human-like qualities into the things they create or imagine.

Anil Seth: I mean, in literature, this goes back Frankenstein. I remember reading the myth of the Jewish golem, Yosel or Yosef, which was created out of mud from the banks of a river and under a magical incantation became conscious and started rampaging around and doing all kinds of crazy things. So I'm not surprised people are interested in it, but this idea, this very quick association that is AI going to wake up and kill us all, I think that's just a very misconceived way of phrasing the worry, phrasing the question that owes more to these kinds of dystopian terminator style science fiction scenarios than to what the actual technology, the actual science, and the actual philosophy has to say about these issues.

Aza: Maybe we should dive in a little bit into that. What are the right questions to be asking? If the wrong question is, "will it wake up and kill us?", then what are the deeper philosophical questions that should begin this conversation?

Anil Seth: I like to think of it in terms of a few related questions. There aren't good answers to any of them. I think the point is to lay out the questions and see whether they help shed light on the issues just in virtue of the way they are. So the first question is, why would we even think that AI is conscious? We are building systems that are supposed to do things, that are supposed to have capabilities of particular sorts, intelligent capabilities we call them. So why would we even think that this would result in something that is also conscious?

Anil Seth: Consciousness and intelligence are different things. I mean, we think we're intelligent and we know we're conscious. That is, we think we're smart and we also experience the world and the self. We have conscious feelings, but they are different things. And then the second question I think is, what would it take for an AI system to actually have consciousness to our best guess? Because there is no consensus out there scientifically or philosophically about the sufficient conditions for consciousness to arise in a system. We don't have a consensus theory.

Anil Seth: Of course, this isn't a question just for AI. We face this question in many, many really, really pressing areas of society and technology now with non-human animals, with newborn human infants or people at the end of life, the humans after brain damage and emerging neuro technologies, things like brain organoids. I think this question applies differently to all of them. And then the last question is, what should we do? What are the ethical and moral concerns about AI systems that either are conscious, that is, they have experiences, there is something that is like to be an AI system or I think more likely because we're almost already there, at least for some people, AI systems that give us the convincing and perhaps irresistible impression of being conscious whatever is going on under the hood. I mean, that's the situation we're running headlong into, if not already there, and that poses its own ethical and moral concerns.

Aza: That's an incredible table of contents, and maybe a good place to start is just sort of where you are ending, which is without the need for AI systems to be conscious, and I want to get us to definitions of intelligence and attempted definitions of consciousness and things like that, but even before we get there, I look at a cloud and I see a face. I look at a car grill and I see a face. The human mind projects intimacy and emotions into almost everything. I grew up in the era of Tamagotchis, Furbies, Nintendogs. We love to project ourselves, as you said, onto others, remake the world in our image.

Aza: And as you also said, humanity has been writing stories about machines becoming alive and having consciousness forever, and because what AI does is it mimics the patterns and the styles of how human beings write, it's actually very good at persuading us that is becoming conscious. So what are the implications of that as AI chatbots, things we don't even know are chatbots become more and more able to convince us of the stories we've been telling ourselves all along?

Anil Seth: So let's set aside for now the question of whether they actually are conscious or what leads us to think they are. I think you're right that we project these qualities into them when they're not necessarily there, just as our brain works in this way all the time. It's not just things like animacy and consciousness in mind, but colors into the world and time, at least that's how I think about how the brain works. It's always engaged and it's projecting the things to give them a sense that they have an essence, which is really always a collaboration between our minds and our brains and what's going on in the world and the body.

Anil Seth: Anyway, what are the consequences? Well, if we have a language model, especially language because as you say, language is very, very key to our image of what it is to be human, things that speak to us seduce our intuitions in an extremely powerful way. So this is why we've seen people much more willing to claim that language models are conscious than other forms of AI. So the first thing is we shouldn't be sanguine. Even those of us who are predisposed to believe that language models are not conscious, we may get to the point that we're unable to resist feeling that they are.

Anil Seth: I think this is a really important point. Illusions of consciousness in language models could be cognitively impenetrable in the same way that some visual illusions are. So there are some visual illusions that even when you know what's going on, you still can't help seeing them, and this might be a case like that. So even if we believe that they're not conscious, we might still feel that they are. So we can't be complacent and say, "Oh," just when we understand how things work better, that'll change how we intuitively respond to them. It really might not, and my suspicion is that it will not.

Anil Seth: So what is the problem with that? Well, I think there's a number of problems. One of them is we become potentially much more psychologically vulnerable. Now, if we think that the thing we're interacting with really understands us in the sense of having a conscious experience of understanding or empathy, if it says something that seems to empathize with us, then we may open up to it in ways that we would otherwise not do. We may also take its advice more seriously than we might otherwise do.

Anil Seth: There've already been some tragic cases like this. There was, I think in 2023 in Belgium, the newspapers reported a case of a man who took his own life after interacting with a chatbot that encouraged him to do so or certainly discuss the topic with him, and of course, we don't know exactly what went on there, but it's very plausible that it played some role.

Anil Seth: So I think psychological vulnerability is one important issue. Abrogation of digital privacy goes along with that. We may divulge aspects of our lives that we wouldn't otherwise do. There's also a much more subtle issue, which is something I worry about quite a lot, which comes down to the ethics of treatment of these systems. So if they're not conscious yet, we feel that they are, we have a choice. On the one hand, we either treat them as if they are conscious because we feel that they are, even though we believe that they're not or we treat them as if they're not conscious, even though we still feel that they are. There's no good outcome here.

Anil Seth: In the former case, if we treat them as if they are conscious even though we think they're not, I think plausibly we end up caring about hunks of silicon and code which have nothing going on and caring proportionately less about other things that really do warrant our moral consideration, other human beings and other non-human animals and so on. So that's a problem, but the other option is also bad. If we do not care about things even though we feel they're conscious, we end up brutalizing our own minds.

Anil Seth: This is an argument that goes back to Immanuel Kant and his lectures on ethics. It's why we don't rip up dolls in front of children even though they're just made of plastic. It's psychologically very bad for us to do that because of the way it makes us feel. It changes the way we would interact with other conscious creatures. So these are the questions, and they're pressing because how we design language models will really make a difference to how these issues play out. If we just keep designing so that they're maximally human-like, especially when we start coupling them with generative images and generative video, generative sound, then we're going to be in real trouble, I think.

Aza: Yeah, and erode some fundamental part of what it is to be a good human being. We wrote a op-ed piece with Yuval Harari I guess now almost two years ago, and in it, one of the points that we made is that language is the operating system of humanity, of civilization. When AI starts to hack language, it hacks all of us. It's cognitively impenetrable. If we form intimate relationships with things that we cannot help but feel our conscious living, breathing sentient beings, even if we know that they aren't, that then at the very deepest level hacks us.

Anil Seth: It does. I mean, this gets back to this point that language is exceptionally powerful at seducing our intuitions. Of course, it's useful. I mean, there's a line to walk here. Language models are really powerful, really impressive. It's great to be able to engage with things in a very natural fluent way, but it is not without its trade-offs, and this is one of the big trade-offs.

Aza: I guess this gets into the definition of consciousness, maybe intelligence because it appears like we're not going to be able to answer the question of whether something is conscious by looking at its behavior, like a language model may perfectly replicate the way that a conscious system might answer. So how do we tell looking from the outside what kind of mechanisms or architectures by which we could even understand if something is conscious, and that gets us to the definition, and maybe starting with the definition in your book.

Anil Seth: So consciousness, everyone says it's really hard to define, and of course, you could spend hours, careers talking about definitions, but in a simple straightforward sense, we all know what consciousness is. It's what goes away when you go into general anesthesia or fall into a dream to sleep, and it's what comes back when you wake up or start dreaming or come around from anesthesia. The philosopher Thomas Nagle, I think, has a very useful working definition in which he says, "For a conscious organism, there is something it is like to be that organism. There is something it is like to be that organism."

Anil Seth: Now, by that, what he means or what most people take him to mean is that for some things it feels like something to be that thing. It feels like something to be me. It feels like something to be you. It feels like something to be a kangaroo or a bat as in Thomas Nigel's famous paper, What It's Like to Be Bat, but it doesn't feel like anything to be a table or a chair. So the question here is, does it feel like anything to be a language model? Is there something it is like for the language model itself to be a language model?

Anil Seth: The reason I like this definition is because it's very, very basic and it doesn't conflate consciousness with all sorts of other things that might contingently come together with consciousness in us humans, but which are not necessary for consciousness in general. So in this definition, there's no necessary relationship with intelligence or with an explicit sense of self in personal identity or agency or anything. It's just the presence of any kind of experience, whatsoever.

Anil Seth: When we open our eyes, look out through the window as I'm doing now, my brain does more than just respond to it as a physical system. There's an experience of blueness in the sky, whiteness on the wall. There's something that it's like to have that experience, and that's different from intelligence, which, again, there's many definitions, perhaps no consensus, and you'll know this better than I, but broad definitions are things like doing the right thing at the right time or solving complex problems flexibly or various things like that, but they're all defined in terms of the functional capabilities of a system. They're not defined in terms of whether there's anything it's like to be that system.

Anil Seth: Now, one of the most famous tests in the space, if not the most famous tests by a country mile in AI is, of course, the Turing test. The Turing test is often rolled out as a test of all sorts of things. Is it intelligent? But also, it's been used as a test or interpreted as a test of consciousness too, but it isn't. It's a test or is supposed to be a test of machine intelligence. Interestingly, it's also been argued that the Turing test is really a test of what it would take to convince a human that a machine is intelligent. It's rather a test of human gullibility or what our internal criteria are rather than objective test of whether the system actually is conscious.

Anil Seth: In this scheme, a Turing test is a reverse test of intelligence. It's a test of what it would take for a human being to attribute intelligence, but what we want is a forward consciousness test. So I think for me, this way of putting things makes it very clear that we can't use things like Turing tests as tests of whether a system is conscious. We need other ways of doing it, and just clarifying it like that I think is already potentially quite helpful.

Anil Seth: Now, the question then comes out, well, what would be a forward consciousness test? Well, we don't have one because nobody knows what makes anything conscious. Now, for forward tests of consciousness, we just have to do the best we can, and that means we try and condition out the biases that we bring to any situation, so we think, ""Okay. I'm not going to rely on just linguistic fluency because that's more reflective of my anthropomorphic and anthropocentric biases. The things that I think as a human are super important. So we'll try and not rely on those," and ask, "Well, what properties are shared by the system that we think are important in consciousness in general, whether it's in humans or in other animals and things like that?" I think that's the best strategy.

Anil Seth: If we go that way, that strategy will cash out differently, let's say, for an AI system than for a bumblebee or a parrot or a human infant because what we share and what we don't share are very different in each case. With a non-human animal, we share an evolutionary history, we share a brain made out of the same kind of stuff. We share many things that seem to be plausibly important in consciousness. With a language model, we don't share any of these things. So I think language models are more similar to us in ways that will turn out to not matter and less similar to us in ways that turn out to matter when it comes to consciousness.

Aza: So I guess one of the questions we start asking is this question of, how do we know? The way I model AI is that it is very good at writing in the style of, and that is to say it doesn't have empathy. It's just very good at writing in the style or producing text in the style of empathy. It doesn't have chain of thought reasoning. It just can produce text in the style of chain of thought reasoning. It doesn't actually have theory of mind. It just produces text in the style of theory of mind. I mean, well, this opens up a very interesting philosophical question, which is if you can perfectly emulate the style of it, does in the limit style become substance, and how do we tell the difference between imitation and the thing itself?

Anil Seth: I mean, there's very deep issues here in philosophy that go many issues come together. So one is simulation of something the same as realization of that thing. If we have a system that is able to perfectly simulate what a conscious human would say or even simulate at a more detailed level, now just simulate the brain mechanisms involved, simulate every neuron involved in how we do what we do, would that be the same as actually giving rise to it? And this is one of the central questions, I think, that is at issue and is perhaps not given the prominence that it ought to in contrast to many things in this area now, this distinction between simulation and realization, because there are some things for which it doesn't matter.

Anil Seth: So if we have a computer that plays chess, it plays chess, but there are other things for which it's clearly an important difference. I mean, this is the issue or one of the issues with Nick Bostrom's simulation hypothesis. Serious philosophical attention has been given to the idea that we actually do live in a computer simulation already, one of an uncountable number. Interestingly, when Bostrom talks about the simulation hypothesis and simulation argument are related but different things, he'll go through the statistical arguments why he thinks it's at least a plausible thing to consider, and then also says, "Oh, by the way, the other thing we've got to assume is that consciousness is a matter of computation, just getting the right computations," as if that's a consensus view, and it is not or it should not be a consensus view. It's one option among many options about how consciousness fits into the universe.

Anil Seth: So I found in discussions with people in AI and machine learning, and to be fair, it's also probably the most common view in neuroscience and philosophy as well, that, yeah, computation is sufficient for consciousness. This, I think, stems from the fact that the computer as a metaphor for the brain has been extremely powerful over decades, but metaphors in the end are metaphors and we often go wrong if we confuse the metaphor for the thing itself. If we reify the metaphor, take the map for the territory, it could be, and it is a very compelling, at least for me alternative, and it's only one of many alternatives, which is that consciousness is a property of living systems and living systems only in the same way that metabolism is a property of living systems, definitional in some sense that you can simulate metabolism, but that's just a simulation.

Anil Seth: This is called biological naturalism. It goes back to John Searle, and I think there are some good reasons why this might be the case. I certainly don't think you can just dismiss it and say, "No, we just assume that consciousness is a form of computation," and that, by the way, I mean, that's the assumption on which any discussion of AI actually being conscious is grounded. If that assumption doesn't hold, then we're always in the space of systems that at best can give us the irresistible impression of being conscious without themselves actually being conscious.

Aza: Just a couple of threads to pick up here, you're talking about metaphor hazards, which is to say whatever is the latest technology, human beings always then use that as a metaphor for the mind. When we invent clockwork, the mind works like clockwork. When we invent steam power, the mind works like steam power. When we invent computers, it works like a computer. When we invent the internet, the mind is now a network, and the risk is, yes, the map is not the territory, but even worse than that, the map can terraform the territory. Based on what we believe about our maps, we end up changing the world.

Aza: I also work help run this nonprofit earth species project, which translates animal language. Ironically, the goal there is by demonstrating rich symbolic interior aware communication, you have to assume some kind of sentient that you get a window into perhaps what is it like to be that being. I as a designer, I think I've always been obsessed with the question of, what is it like to be another person, and that just expanded as we can expand our perception and hence our understanding with AI to, what would it like to be another being?

Aza: I mean, I always wonder what would like to be a tree like a Sequoia where every in and out breath equivalent is like a decade. There were some of these trees like Bristlecone pines that were around when the ancient Egyptian pyramids rose and fell, what would that experience of being be like? I think in some sense goes back to the question of, is consciousness a property of information processing or matter if the only way they're going to be able to tell if a system is conscious is by having some kind of forward test, some correlate from being able to peer at the human brain, from outside the human brain, see what's different when it's undergoing general anesthesia, when it's not, when it reports having a conscious experience versus when it doesn't report having a conscious experience?

Aza: I know you have some great experiments in your book of how you might be able to tease those things apart, but doesn't this always end up with we'll never be able to fully cross that first third person divide, we'll never really know whether something has a conscious experience. I don't really know that you, Anil, is having conscious experience. I can just make good suppositions because we share the same hardware, the same wetware, the same sensorium, the same evolutionary history, and so we can make a good assumption, but we're always going to be looking for outside correlates, things that sit next to but don't actually get us all the way there.

Anil Seth: I partly agree with you. I think there's a couple of places where I think it's less pessimistic than you say. So one thing is I think you're right that we can never entirely cross this divide between the first person and the third person. So I will never know what it's really like to be anything other than who I am now. I don't even know what it's going to be like to be me tomorrow morning or you or any other human being, but I'm totally convinced that you are conscious. I'm totally convinced that tomorrow morning I'll be conscious too. So that limitation I don't think is a relevant limitation, right? I am totally sure that you are conscious.

Anil Seth: Now, take a slightly more tricky example. Someone who suffered brain injury, if they're able to speak, even if it's through imagining things so that it sets off, there's a beautiful series of experiments pioneered by my colleague and friend Adrian Owen, where he got people in a brain scanner who could not behave, they couldn't say anything, they couldn't even move. So they looked from the outside as if they're unconscious, but if you ask them questions or give them instructions, "Imagine walking around your house or imagine playing tennis," different parts of the brain light up in the same way that would happen for me or you. This seems to be extremely good evidence in this case that these people are conscious because it's very, very difficult to imagine a human being doing that without being conscious, understanding the instructions voluntarily following them for extended periods of time. It makes sense in that context, so we can be pretty sure in that context.

Anil Seth: It's very easy to get a language model to say anything, but that doesn't mean what it says relates to what it is in the same way because it's a language model. So the shared background is not there. So I think the challenge we face is it's really the challenge of consciousness science. It's going to be very difficult to figure out what the sufficient conditions for consciousness are until we have a better understanding of how consciousness happens in those places where we can be reasonably certain did it exists, and we learn bit by bit. We gradually extrapolate.

Anil Seth: These days, most people are pretty convinced that all mammals have at least some conscious experience, not everybody, but most people. That allows us to generalize beyond the human, get a sense of what's going on in brains that's relevant to consciousness that's not specific to humans, gets us away from language, for instance. As we do more of this, I think we'll be able to make better inferences about the presence of consciousness where we're less sure on our footing.

Aza: I'm getting from you, Anil, that you don't think that machines will become conscious, although there is plenty of room for disagreement. So what are the consequences of getting it wrong?

Anil Seth: I'm glad you raised that because you're right, from my own perspective of consciousness being very closely tied to life, I think we're not on a path to conscious AI. I think we are on a path to AI that seems conscious, but that's a very different thing, but of course, I might be wrong and probably am wrong. I think most people who have theories of consciousness are also probably wrong. There's a real need for a certain humility here. I always think if you hear a confident pronouncement from somebody that AI could never be conscious or AI is conscious, then I think we should be pretty skeptical.

Anil Seth: There are no grounds for extreme confidence either way here. The consequences of being wrong about the fact of the matter, about AI actually being conscious are huge, which is another reason we need to respect this humility. If AI is on a path to being conscious or already slightly conscious as Ilya Sutskever puts it, then we face a moral and ethical catastrophe of kind of unprecedented proportion, and that sounds very dramatic, but I think it's warranted.

Anil Seth: As soon as something is conscious, it has moral considerability. It has its own interests. It plausibly has the potential to experience suffering, and it may be suffering of a kind that we won't or constitutively unable to recognize because of the very different constitution of these systems. If we artificially bring new forms of suffering into existence through developing real artificial consciousness, well, that is, with capital letters, a very bad thing indeed. So I think it's really ethically crucial, but epistemologically, which is say, how will we know highly uncertain situation.

Anil Seth: What you take out of this mess? I think the first thing you take out of it is that it should not be a goal to build conscious AI. Nobody should be trying to build AI. That actually is conscious, that is ethically, morally a highly irresponsible thing to be doing.

Anil Seth: Now, of course, some people would say, "Well, any AI system, you're saying we shouldn't do any research in cognitive neuroscience or AI?" No, no, no, no, no. I think there's, again, a line to walk. I think it's very reasonable that we can build systems that are useful, that may leverage some of the functions that we associate with consciousness. This is a conversation I had a number of times with Yoshua Bengio and others. We were talking about this in terms of where AI should go, and this idea came up that there are things that we do that we associate with consciousness that are very useful that AI systems don't do or don't do very well yet, things like learning from one shot or very small amounts of data, generalizing out of distribution to novel situations and having insight into their own accuracy.

Anil Seth: We intuitively have a degree of confidence about things we're conscious of, and these are all very useful things. So it makes sense to think, can we abstract the principles by which brains do these and build them into two artificial systems? I think that's a reasonable thing to do. It's walking the line. What I think is unreasonable to do is have an explicit goal to build a machine that has conscious experience. We don't know what it would take to do that, but critically, we also don't know how to guarantee not doing that. We don't know what it wouldn't take.

Aza: I can see the strong argument in here for the best way to understand our own consciousness is going to be to make surrogate models AI that we can turn on and off its consciousness to see how its behavior or other systems might change. So the best way to understand what it'll take to not build conscious AI is to try to figure out what it would take to turn it on and off. I'm curious how you react to that. It's very similar to the greatest accelerant to AI risk has been people who've been focused on AI safety trying to race there to get there first.

Anil Seth: Yes, no, it's a dangerous line of thinking, and yeah, it's also there in, let's say, animal research and in neuroscience too. So how do we best understand how to avoid suffering in human beings and other animals? We need to understand the neural mechanism, so we need to do experiments. There's always a line to tread here. So personally, because I think that consciousness is not just a matter of computation, personally I'm quite comfortable with computer simulations and models of the brain as ways of understanding the brain mechanisms involved in consciousness in just the same way that I'd be comfortable with computer simulations of protein folding or weather systems as ways of understanding these things.

Anil Seth: I think the chances of actually these things being conscious is so low that I'm happy with doing that, but if I thought that building a model of let's say a global workspace or whatever your theory of consciousness preference might be would actually generate a little bit of consciousness, I would be much less happy about doing that. So it does depend on what your starting assumptions are, which is another reason I think we should be very curious about the assumptions we make about what it would take for something to be conscious.

Aza: I'm just noticing it's almost a contradiction inside of myself, which is I really want the answer to what it means for a system to be conscious when it comes to animals, for us to treat them with a greater respect, dignity, to displace humanity as the center of the world, and yet when it comes to machine consciousness, I want the answer to go the other way. I sort of want to be like, "Well, that doesn't really count somehow, It's a different substrate, therefore, we don't have a moral obligation to something that I can't pet and touch or hold or feel like a kind of empathy in its physical form." I guess it gets to this next question, which is like, "All right. What steps do we need to take to orient society around the future we're actually already starting to live in?"

Anil Seth: It's a very difficult question, of course, which is why we're talking about it, right? I don't have all the answers at all. I do feel strongly that number one thing, already mentioned, we should make it morally unacceptable to have as a goal to build conscious AI real conscious AI. I mean, that's just not a good idea. Hard to know what that means in practice, again, for the reasons we've just discussed, but at least if we have that as a north star, then as we learn more about consciousness and more about AI, we should revisit this constraint.

Anil Seth: Then there are a whole set of other questions, which I think come down to how do we learn more, and here, I think is where we do need to walk the line. So the more we do know about consciousness in those places where we're sure it exists or reasonably sure, the more informed our inferences will be in other cases. So the real urgency is to accelerate research into consciousness. So this is a bit like your acceleration example, but I do think that consciousness, we are learning a lot more about it, and there are ways to do this, which I don't think are particularly morally problematic. They just require resources and scientifically upping the game in all sorts of ways, but that's perfectly possible.

Anil Seth: So I think that's something that we should do more of, not only for AI's sake, but also when it comes to animal welfare. I mean, your work in the earth species project is a beautiful example of, again, exactly how do we use knowledge to finesse, to change our behavior with the systems, creatures we share our world with, but also human infants and organoid. So many cases where it's really important to understand. Even in the law, it's important to understand more about consciousness, to understand when we hold people responsible for their actions. It's not an armchair philosophical endeavor. Consciousness science is possibly one of the most practically urgent things that we could be doing.

Anil Seth: And then I think the last thing back to AI is systems that seem conscious because this is happening. And here we need to decide, "Well, what kinds of systems do we want? Do we really want to just design AI systems that maximally seduce our intuitions are as convincing as they possibly could be?" I don't think so. I see some advantages, and some of them may be slightly uncomfortable. The idea that we could benefit from therapy, talking to a language model that actually doesn't care about us at all because it can't care about anything is disquieting, but if there's evidence that it's useful and there is some, then it's hard to dismiss it as a bad idea, but it doesn't mean that this race to the bottom, wherever you want to go, race to complete incorrigibility of systems appearing conscious is a good thing.

Anil Seth: One of my mentors who sadly died recently, Daniel Dennet, I think he said something. Well, he said many very wise things, but one of the things he said that stuck with me was that we should always remember that AI systems are tools and not colleagues. We should not kid ourselves otherwise, and I think we should design them with that in mind too. Design systems that best complement us as humans and that don't mimic or try to replace us as humans.

Aza: I think that's a really great point, but I also think we need to be really careful because we can't forget that while humans wield tools, incentives wield humans. So when people hear that AI is just a tool, they sometimes hear that it's just up to us to decide how to use them, but that's not really true. It's the incentives that shape how the tools end up being used. Thinking back to the releases of OpenAI's GPT 4.0, it's really designed to feel eerily human. It jokes and it laughs, it sings, it flirts. There's a clear race, a scramble for the appearance of artificial consciousness.

Anil Seth: Yeah, there's an appeal to it, and I think this does go back to this 100,000 year history of humans wanting to create things in their own image, but that history of literature, if it's taught us anything, it's taught us the things that can go wrong if we buy into that goal without thinking critically about it. I recently read Frankenstein, Mary Shelley's novel. The thing about Frankenstein was that when I read it now thinking about things we've been talking about now, instead of it being a cautionary tale against the hubris of creating life, which I think is often how it's interpreted, the lightning bolt and, "I'm alive," in the dark tower in midnight, it's not that the creature was alive that mattered, it was that it was conscious. It was the ability of Frankenstein's creature to feel, to experience sadness, rejection, envy that led to all the carnage that followed.

Anil Seth: So reading it now, I think it does what great literature often does, which is it's always relevant, and it seems now to be a cautionary tale against the hubris of the desire to create artificial consciousness. We should avoid making that mistake whether in reality or even in appearance.

Aza: I just want to come back because I know how the human mind works is that intelligence and consciousness, the ability to have such a thing as the feeling of being are different. What the labs are doing right now is they are racing towards the ability to automate AI research. That is the ability to spit out an AI that acts like a intelligent human agent that can program at the level of an AI researcher to discover new algorithmic gains and new ways of running AI so that AI gets better. So the thing they're heading towards is a intelligence explosion.

Aza: Of course, once you start being able to create AI that makes AI better, which makes AI better, you end up in this double exponential curve, and none of that asymmetric power, which comes with then an asymmetric ability to persuade, among many other things, none of that requires whether you are conscious or not, but where it intersects with a question of consciousness is the rate at which new technology will be coming out, the rate at which scientists or labs can try out, "Hey, maybe this thing is conscious, that's going to get faster and faster and faster." So the ability for us to have the moderating effect to even be able to do this kind of philosophy on a deadline gets shorter and shorter and shorter and shorter as the consequences go higher and higher and higher.

Anil Seth: Yeah, no, this notion of the double exponential is troubling. I wonder about it a bit though. There's also arguments that things might be plateauing and flatlining in some respects. Psychologically, again, our ability to deal with exponential curves is notoriously bad. We always feel like we're on a threshold, but that's just how exponential curves look wherever you are on them. Things are moving fast, but there are things we can do. I mean, the first thing is just to continually remind ourselves because it's hard because we're psychologically programmed to think otherwise, but just to continually remind ourselves that consciousness and intelligence are different things, that consciousness won't just come along inevitably as systems get smarter and reach some threshold, it's not just going to happen, and also that there's a difference between systems that are conscious and systems that seem conscious, and they have these different ethical consequences. One is more about the problems of bringing into the world massive new forms of suffering, the other is what it would do to us as human beings, brutalizing our minds, making us psychologically vulnerable. Different situations, different risks, one very unlikely uncertain, one already with us.

Anil Seth: So seeing the landscape clearly I think is really the first step, and that can get us a long way. We're in this trouble in the first place because we don't have a sufficiently good understanding of the nature of consciousness. It was always one of the biggest problems in the world. It's been perplexing thinkers since people have been thinking. What's been amazing to me over the 25, 30 years I've been doing this is that progress is being made. We don't have the full answer, but what is known now about consciousness and how to think about it has changed, and these changes have implications already for how we think about consciousness in other systems, animals, machines. So really, and this is a little bit of a self-serving prescription, but honestly, more effort, more resources to better understanding consciousness is really, really important.

Anil Seth: And then the last word is humility for me. No one knows how consciousness happens. No one knows how to guarantee to avoid creating something that's conscious. So we need to take all these things with a bit of a pinch of salt and recognize as a plurality of views, but not throw our hands up and realize because if anything, we can still say things, we can still have informed guesses, but let's be aware of certainty.

Aza: Anil, this has been fascinating and even more so I think has drawn out a lot of the consequences for what we get wrong when we think about AI and consciousness and what its implications are going to be, so thank you very much for coming on the show.

Anil Seth: Thank you, Aza. It's been a delight speaking to you. Really enjoyed it. Thank you very much for having me.

---