Europe 2031 — Arts — Then Do Better

What happens if Europe gets AI wrong? Alex Petropoulos is Chief of Staff at Arq Foundation and one of the authors of Europe 2031, a speculative story imagining Europe's technological, economic and political future in a world of transformative artificial intelligence.

We start with the predictions behind Europe 2031: from severe shortages of AI computing power to the possibility that compute becomes abundant and today's infrastructure investments lose much of their strategic importance. Alex explains why he and his co-authors chose speculative fiction rather than another policy paper, and the power, and danger, of making people feel a possible future rather than simply understand it intellectually.

Can Europe compete with the US and China? Or should it concentrate on its own comparative advantages: industry, data, infrastructure and supply chains, while accepting continued dependence on American AI?

Alex argues that Europe's deeper problem may be institutional. Its commitment to democracy, consensus and constrained power is part of what makes it worth defending. Yet those same processes can make change painfully slow. Which European values really matter? What trade-offs should we be prepared to make? And can politicians offer a positive vision of the future that brings people along?

We explore the opportunities and catastrophic risks of AI, including why scientific progress might accelerate dramatically in mathematics and other verifiable domains, while biology, clinical trials and the social processes of science remain more difficult. If intelligence becomes abundant, what becomes scarce?

Alex also explains why he has changed his mind about a publicly funded European frontier AI laboratory, the challenge of sustaining successive generations of increasingly expensive AI models, and what we can learn from institutions such as ARIA and the Institute for Progress. We discuss scientific moonshots, tolerance for failure, and why we urgently need better measurement of AI's capabilities and effects.

We then turn to some of Alex's more personal interests. What did captaining a successful university Overwatch esports team teach him about teamwork, learning and comparative advantage? How does he use AI as a thinking partner? And why does writing remain such a painful process?

In our overrated/underrated round we cover recursive self-improvement, universal basic income, AGI, compute taxes, open-source AI, prediction markets, effective altruism and science fiction.

We finish by imagining an ordinary Tuesday in a better future. For Alex, it involves friends, a bakery, a park, good health, affordable travel, meaningful work and perhaps a little whimsy and mischief.

A useful reminder that the point of economic growth, technological progress and better institutions is to make possible lives we actually want to live.

Key takeaways

Europe's greatest risk may be institutional, not technological. Individually sensible decisions can produce collective failure when incentives, bureaucracy and political processes are poorly adapted to rapid change.
AI sovereignty is not the same as AI self-sufficiency. Europe will probably continue importing American AI. The important question is how to retain influence, resilience and bargaining power.
Comparative advantage matters more than winning every race. Europe's factories, industrial expertise, data and infrastructure may provide more realistic opportunities than attempting to replicate every frontier AI laboratory.
Values require trade-offs. If Europe wants to preserve what makes it distinctive, it needs to decide which principles are essential and which existing practices can change.
AI progress will be uneven. Mathematics and other verifiable domains may accelerate rapidly. Clinical trials, physical experiments and the social organisation of science could remain substantial bottlenecks.
Publicly funded AI labs need a sustainable model. A government might fund one frontier model, but competing over successive generations requires an economic and organisational structure that can keep going.
Measurement is badly neglected. Confident predictions about AI are often built on unreliable benchmarks or inadequate evidence. Better policy needs better information.
Competitive gaming can teach surprisingly useful lessons. Teams can outperform by leaning into unusual strengths rather than trying to approximate an idealised strategy.
The ultimate test of technological progress is human flourishing. A better future should mean more agency, more opportunity, better health, stronger relationships and lives that young people can look forward to.

Transcript

AI-assisted transcription; errors are possible.

Ben: Hey everyone. I'm super excited to be speaking to Alex Petropoulos. Alex works on AI science and the institutions we need to make progress. He's chief of staff at Arq Foundation and one of the authors of Europe 2031. A speculative story about Europe's technological future. Alex, welcome.

Alex: Thank you Ben. Very, very good Greek pronunciation for me.

Ben: I've been practising! Of all the things you imagined in Europe 2031, what do you think is most likely to be true by 2031?

Alex: This is a difficult question because already some things in Europe 2031 have come true. Or rather, small versions of them have come true. You know, we sort of predicted access controls and predicted the US cutting off access. And a small version of that did happen with the Department of War, Anthropic and Mythos. Although I would say it's a hint at what could come as opposed to an actual thing coming true. And the other thing that makes this question difficult is that, in general, when I talk about the predictions we make in Europe 2031, I'm like: don't index on any one particular prediction.

We're not necessarily trying to get them exactly right date-wise. In fact, we fudged dates in, [I ] in order to keep things working in a way that would keep the reader happy, especially certain audiences that we had in mind when we were writing it.

I would say the prediction that I think is still looking to be on track is that we will hit a compute crunch. Compute capacity will just be incredibly constrained. And especially if you hit a capability threshold, people often say that data chips are depreciating in value, but actually in many cases chips have appreciated in value because the AI models have gotten better, which only just keeps pushing demand up and up and up.

And so if we suddenly got a big capability breakthrough, demand could go through the roof and we could be in an even more compute-constrained world than we are now.

Ben: And you can see that in some market prices, particularly upstream of compute. So the bottlenecks you need to make compute are still, I mean, they flip-flop a bit, but they're still going crazy.

Well, I don't know if this will be an easier question on the other side, but what's probably wrong by 2031 or maybe even not the date, but directionally wrong, but still really useful to consider?

Alex: I'm going to be very cheeky and I'm going to give the same answer, which is the compute predictions. In general, I think people should internalise that we might not be compute constrained forever or that compute is important.

The reason we focus on compute is because we talk about ways countries can be important on AI. And normally, to do all this really difficult R&D, you have to spend billions. You have to compete in capitalism, it's difficult. You probably will lose by default. Or you could build a data centre and suddenly you're important.

And so you are, you've got the lowest-hanging fruit possible for becoming important on AI and having some sort of relevance. Great, we should do it. However, we should also remember this relies on compute being constrained. And there are worlds in which even in five years time compute isn't a constraint.

Like Elon could have massive breakthroughs with SpaceX. The cost per kilogramme of launch capacity could come down sufficiently. There could be a couple breakthroughs and suddenly we have orbital data centres and all of your ground-based compute doesn't matter anymore. And all this investment that you did and you're like, 'We're fine.'

We've got the compute, but it doesn't matter any more because now it's all being made redundant.

Ben: That seems like a good idea. Look at, in some ways, the left tail and the right tail: compute is really cheap and abundant anywhere, or there's a crunch and you can't do anything about it. Those seem to be quite useful extremes to speculate about.

And I'm interested, so there is a horizon scanning future scenario-type industry and academic literature out there. But I was interested in what made you think speculative fiction or say scenario planning seemed to be a better way to examine this and potentially examine this for European policymakers rather than just another white policy paper, 40 pages.

I'm going to give you part of the answer already, that no one really reads apart from three people in a little room before it gets shelved. But anyway, speculative fiction: is it a better way, and how did you come to this?

Alex: I mean, in one sentence, you want people to feel the AGI, you want to create an emotional response.

I sort of talk about what we did as wielding a very dangerous tool using scenario writing to create buy-in to a problem that you think exists is epistemically impure. You are violating some of these, I don't know, purist rationalist norms. When you sort of wield this weapon, you have to be really, really confident that your take is correct and that you want to achieve.

And I think that we were confident enough that we're like, 'We're going to do this.' But the point is that it makes people feel something, you know? We had hundreds of people messaging us, having read it, people who said that they were lying in bed, listening to it, and they started crying when they got to the end. We had a mother who messaged us saying my daughter is three years old. There's no way in hell that she's going to wait, be eight years old in the world that you described, what can I do to stop this happening? And so as a side effect, you have this virality, but we never really intended it to go viral. The theory of change behind it was. Convince or sort of be heard by the 2000, 3000 most important people within a country that the elites basically. And that would've been sufficient. I think that it sort of reached an order of magnitude or a few more people than that. But that's the basics of why we chose to use this dark, magic, forbidden weapon.

I would look at it in that sense.

Ben: That's an interesting theory of change. I think there's quite a lot of truth to that from my way of thinking. Obviously you have to feel 110% that you're going to be correct, but on big moments in human history, maybe just taking the last a hundred to 200 years, I always think about the movement, which where society overall went from being pro-slavery or slavery in society to being anti-slavery and the anti-slavery movement started small and had to do these things to convince people otherwise.

But obviously you're going against the majority grain. So if you're wrong, you can do a lot of damage. So it's kind of interesting. So maybe taking some of your premises within that. I guess in my mind the paper argues Europe has been making several misjudgments around how fast AI is advancing.

Maybe we could say almost everyone is making these misjudgments also misjudging its impact and the ability for Europe or maybe even countries overall to catch up. We also make the point that a lot of people on the frontier of say policymaking and government or even the person in the street is not actively using frontier models.

They're not solving Millennium Prize Problems. They're not solving [ ]. So we're a little bit behind the curve because the curve, or the frontier, is going so fast and we are ahead. [ ] I'd be interested to think, has anything happened in the last few weeks, few months since you've written the paper, which has made you change your mind on any of that?

Or if anything, has this emphasised the arguments that you were making? Speculatively, even if the dates are wrong or at least in the direction of travel.

Alex: A couple of things. On the high level, people sometimes ask me, 'Are you optimistic, Alex? Are you pessimistic?' And you know, we wrote quite a doomy scenario. It's doomy for me under the assumption that we do nothing. And I've never believed that we will do nothing. I've always believed, we like to wake up and smell the coffee.

The question has always been: how quickly do we, how fast do we sort of viscerally internalise what is going to come or what might come and then act in accordance. And so I think there has been this visceral reaction faster than I would've expected it. So, bullish. You know, Ursula von der Leyen in her state of the Union, this is the EU version of the US State of the Union.

She gave it to me on Wednesday. She talked about recursive self-improvement and the possible risk that this could bring. And the fact that our enemies might have this technology and we won't. If you told me a year before that she'd be talking about that, I would've, I would've laughed at you.

That being said, I think that, you know, that there are these three assumptions that we talk about, right? Miscalibrating how fast it's moving, how important it'll be, how well positioned we are. But then there's this third meta level thing, which is at play, which is at every step in the scenario, everyone takes what feels like to them, the sort of locally rational action.

They sort of act within the limits that they perceive themselves to have. And the incentive structures within, like the bureaucracies within the institutions are set up in such a way that people aren't able to take the correct action. And not much of that has changed there.

Things that would make me maybe be more optimistic would be people who really get it, choosing to go into government, choosing to put themselves into these positions of power and next to the people who take the decisions to help inform them. And then also like broader reforms to how these institutions work.

And this is, this is by and large why I'm more bullish on European member-state policy than EU policy, if that makes sense.

Ben: Yeah. We can come to that maybe Europe has to be a tiny bit less consensual in terms of getting all member states around and have, well, what I would say is perhaps a tiny bit less democracy even though there's this strange paradox where in order to defend democracy, maybe you have to be a little bit less democratic.

And I see some analogies with climate movements, maybe even within COVID pandemic, particularly within climate. It's interesting, if you go back 20 years, the median, in fact even maybe 10 years, the median scenario was looking at perhaps a four degree world. And today we're looking at between a two and three degree world and still on a downward trajectory you can argue about speed.

So that's one of the really tricky things. Because humans generally moving at speed is not our default scenario, but actually we've done quite a lot within climate and certainly within climate have now taken out a lot of the negative tail risks. I mean, there's not exact analogy because of the tail risk in AI, so I kind of get a little bit cautiously optimistic around that given some of our other complex wicked problem type things.

Bringing it back to Europe, I was interested if you think it, well, for Europe it should not really become simply a slightly worse version of America or China. That seems to be, again, what the paper's suggesting. It can't really be America or China in terms of compute or frontier labs and things like that.

So I'd be interested now also a few months after the paper, what do you think Europe should optimise for or Europe should be doing if it can't be America or China? Or do you think it genuinely should try and catch up in certain domains?

Alex: So, on catching up, I think it has to just be really realistic about where it can and where it can't catch up. And then you maybe take a calibrated bet and say, I'm going to put a bet on this moonshot. I'm going to put some amount of money. Maybe it'll work, maybe [ ]. By default it won't.

You should be aware that it's like a big risk. What should Europe try to emulate? I think that Europe needs to have like a conversation with itself about like what makes Europe great and then how is that a threat and how is that at risk? And I think in general, I sit in the position of why care.

I guess that's a question one might ask, which is why are we even talking about Europe? Why do we care about Europe? And one reason I care about Europe beyond the fact that my friends live here and my family live here is the fact that I think the US have a set of values and these values are quite good and Europe has a set of values and these values are also quite good.

But I think that regardless of whose values are better, I think there is some weighted combination of both of those values that is superior to either. And I would like to see a world in which those values are represented into the future, if you want to say that, or at least represented within AI systems and within the way we develop this technology.

And so then you should be really okay, you're, you need to do a better job of exporting your values. But then the question is which values are you willing to give up? Which values are you willing to import? So it's sort of like a question of what do you want to take from the us?

Like what actually matters where are your trade-offs going to happen? And I think that for all the talk that people in Europe do about values, I think it's quite an immature conversation still because there's never any conversation about trade-offs and what they'd be willing to give up. Like where, what is the ranking order?

You can't value them all equally. That doesn't work because you're going to lose them all. A bit of a fluffy answer. But I think it's like a prerequisite to actually be oh, why do we care about this? What model are we going to follow? What should we do? We can still be good at many things.

Ben: I, in some ways that echoes just overall thinking around industrial plans. Like should you have them, you've got to figure out, you can't do them for everything. You need to figure out where you are good or not. But specifically for AI. But I guess one, if you're quite purist about some of these things from say a market or say your anti industrial plans at all, you'd maybe argue why can't Europe simply just be very good at using American AI or Frontier AI?

I guess I already think of one of the counters because of what's happened with defence or defence pacts recently. But you could maybe say, well, we become really great at something else. We keep mostly European values and we just import American AI and that's it. Spill below has happened that way. Is that not a cogent argument because of all the other European values, things that we need to maintain or argue for.

And also, I guess, the specific different industries and values that different countries have.

Alex: So I think it's first important to acknowledge that some version of that will be the version we end up in. There is no world in which we don't import American AI just as there's no world currently where we don't import part of the US security umbrella, right?

Europe isn't independent, Europe isn't sovereign. It shouldn't pretend that it's totally sovereign. And so I think it's silly to pretend that we could get there on AI within 10 or 20 years anyway. I think that there are like two stories that I tell when I talk about, AI policy.

The first is like if AI timelines, the pace of AI progress is sufficiently long. If AI is a sufficiently normal technology, whatever that means nowadays then what you should be doing is basically just copy progress studies and just look at what, like the [Rhine Group] people, look at what Mario Draghi is doing.

Do the Draghi report. Listen to Patrick Collison probably, like read Works in Progress. And just do all of that. And you'll probably be fine. And that's like your normal baseline. If AI were a normal technology, if timelines are really long, you have your adoption, you have your market effects just like unfuck the European economy, great.

I think the disagreements in AI policy come from okay, but AI isn't necessarily like perfectly normal technology. First, it might be very important. It might be securitized and treated as a national security asset and there might be a US administration which isn't operating in a perfectly positive sum view of the world or you know, willing to take very negative-sum trades with partners, even allies.

And two, if AI progress moves really, really quickly, these reforms take five to 10 years to kick in. So you need things to bridge that gap. And so you have to be thinking about okay, I'll do these reforms, I'll fix my economy, I'll be fine in 10 years maybe. But what are you doing over the next 10 years to make sure that you're relevant?

Make sure that you're not slapped about by the US or China.

Ben: And what do you think European countries should do, either as a European blob or you could choose one or two countries, UK or Greece or, maybe one of the bigger ones like Germany and France. I get the reading from the paper. There's some suggestion that potentially industrial AI, whatever that might mean, might be an area or robotics or some other part of applied AI.

But then I was wondering, is there a European comparative advantage in any of that compared to American or Chinese companies capturing most of that value. But I guess Europe has got to place some of its bets somewhere within that. So I'd be intrigued if there are any suggestions that you feel Europe can explore or is it still at that higher level where they have to just figure out what it is they're not sure and they're just too slow and figure that out.

Even if they don't get all eight correct things, if they get two correct, that's going to be much better than where we are now.

Alex: Yeah. So three things. First, you got to do the reforms to make sure you can actually do all this properly. You got to set the groundwork such that a trillion dollar European company can be created in the future.

That's your long-term play. Get that out of the way. Second, I think that some of these things you mentioned are up there. There are some competitive advantages. It seems like Europe does actually have quite a good data advantage. There are just like European data sets. We have big European companies, we have industry.

We still are, we haven't completely deindustrialised. We still have lots of factories. So there are lots of places where we are strong and have valuable things to offer. I guess the question is can you use that well enough to ensure that you are, that if access is what you are worried about, ensure that you retain access.

I guess an example is: make yourself the person that produces and builds things for the Americans. If you think the Americans will get to do powerful AI systems very quickly, we can be the ones with the factories, right? It is a version of that world. But then the third thing I will say is Europe 2031 is a, go, to go back to the analogy like we use the dark magic. We dabble in ideology. We point all of these faces and attention towards a problem. It is an exercise in problem defining. There is some amount of work and we've done our best to think about solutions.

But I will just say the field of people solving this problem, there are two bottlenecks and one of those bottlenecks is ideas and just people, like a community of people thinking about solutions. I don't think we have very good solutions at the moment. I think our solutions are, to be honest, crap.

And I would like to see more people, more ideas flooding in about what we can do. Because I think that, yeah, we just haven't had enough tokens on it, so to speak.

Ben: I think you're going to make a very good policymaker or politician because that is a three-point plan, right? They like that. I think there may have to be an odd number, like a seven-point plan or 11-point plan, but that's not quite as sexy as a three-point one.

So we need more people thinking about this. Not enough people. Because we can see that not enough ideas at the moment might be happening. I feel the vibes are there, but I guess we're a little bit slow. Europe may have some comparative advantages though. We have to think about that. Maybe supply chain into America, something like that.

And then the long term reform, which is kind of win-win. Even if you think this AI stuff, they've just gone completely crazy. Yeah. What are they doing? Well, this just, you should have it found foundationally. And actually it's an interesting edge, just the glass half full is because Europe can potentially unlock that more than say, some places in the US and potentially unlocks maybe more easily.

There's a, there's a chance that it can actually jump up. Like some parts of America can't actually deregulate that much more. They've reached the frontier of that. Whereas I don't think you could claim the same in Europe. But it's interesting on that first and third point, maybe to some degree there seems to me a little bit of a tension in terms of, and I mentioned this before, the argument essentially is Europe's consensus and procedures are somewhat too slow that needs unpicking speeding up more people in it.

But part of the European values of all of that seems to me that they have this: liberal democracies constrain power. Now you can see that's tipped too far and Works in Progress are very good on this, where you've seen like it's really reasonable for the fire department to say we need access for our ladders to go into the buildings.

But when each individual stakeholder does that, it means you can't build any buildings, or they're not really suitable or are very expensive. So I'm interested is at that higher level, do you think there is this throughput where we can become dramatically faster, we can get more people within this new industrial AI policy or long-term reform without destroying enough of these European liberal values that actually we so and I got the impression from your answer earlier, is that yes, there might be some trade-offs within those values, so we do need this hard conversation.

But if you come with a call, I'm going to make it up to 10 or 20 of them, you know, a good eight to 15 of them survive and that still feels sufficiently European that's what we should do. Is that the right way of thinking about it and that there is a tension there? Or actually is there some other through line that we could pursue here?

Alex: I think there is a tension. And if you don't do this right, you fail. An example is look at the way that the US have done their data centre policies. And they've had this laissez-faire approach and now they have this massive backlash. And so this is an opportunity for Europe. Because Europe hasn't built as many.

They've imported a little bit of the US data centre politics, but it's still very nascent. We can take a lot of lessons, we can learn how can we do it well? So you bring people along so that you are actually making sure that the communities benefit from the data centres being built, that they get some of the tax revenue that the data centres are building in electricity production and not using water even though that's, you know, made up. But in general, I think this touches on what you said, which is glass half full. I'm an optimist. I think you have to take many of these things and frame them in the optimistic sense. There is a world in which you can say, Tyler has this saying, you know, what is it the Europeans are so culture, they can tell you about porcelain.

They can tell you about all this and that, but if the world was run by them, negative 1% growth rate and we have all these bottlenecks and if AI sort of becomes so important, the bottlenecks means Europe is doomed. And, then that's the reason to freak out. It's like also, ah, if we get rid of all these bottlenecks, Europe has so much to gain.

Like there is quite a strong compelling win-win of oh, we can get the growth in the short term from doing the deregulatory stuff or, and whatnot. And then we get like the AI growth in the long term. Or maybe the AI growth kicks in first and the long-term reform. So there are some interesting political economy packages that you can put together.

I think the ultimate question is, are you presenting a compelling vision of the future that is positive, and that you can bring people along with, because I think it's fair to say that people, I think most politicians have failed at this and, most politicians are like failing to rise to the challenge of like presenting a vision to the people of the future. And saying, this is how your life is going to be better because of this thing. I think this is especially true with young people. And they sort of have this negative-sum view of the world. Because the world has been negative-sum towards them. Now you have to do politics. There's like the difficult stuff.

And find ways to create your coalition to take, pick the set of policy choices that give you wins and give you win-wins across the board. Make some trade-offs. I think that's possible. I think I've seen a way that you can do that. I don't think necessarily politicians are catching up to this quite yet, but I think that there's a festering, I think something's festering in regard to youth politics and disenfranchisement and that this will extend and sort of blow up around AI. And so I think if you don't sort of get on this early as a politician, it will sort of consume you.

Ben: We

Alex: Need, or how does that relate? Yeah,

Ben: I say we need the storytellers back. Well we need in some ways like the force, the Jedi side of the force because you're probably going to dabble in these dark arts. So yeah, you might fall the way of the Sith with these very charismatic storytellers, but you are going to have I guess the Yoda-like characters hopefully coming through and saving us.

Well perhaps zooming out. I mean we can come to other policy bits if they occur to you or come up. And I do wanna talk about the opportunity side as well as the risk side, but maybe touching on the risk side first and zooming out because I think that's what, a lot of things came very cogently in 2031.

I still think we need some good positive speculative futures and I know people are trying, but we're just not coming towards it. I don't know why. Maybe, maybe humans are just inclined to look at dystopian sci-fi more readily. But in any case opportunities and risks. Maybe on the risk side I tend to hear two different categories of catastrophic risk arguments.

So one goes, we create intelligence vastly beyond our own and we can't control it. It's like we are humans or, and they are mice or I prefer to think like octopus and you just can't understand it. Like the mice don't know what's going on and they just all disappear because humans don't really care about them.

And it'd be the same for this vastly superior intelligence. That seems quite sci-fi to a lot of people that I know. Some people take that argument seriously. The other which seems more cogent to sort of everyone is that you just get a very powerful AI tool, say, and it gives certain humans, maybe quite a few humans who may want to do harm or some version of evil, much greater capabilities at much lower cost.

And they can simply just do that. So you don't even need to get super intelligence of any form to just kind of see okay, that could be quite a bad risk. And so that comes in a cluster of manmade pandemics or all of these other things that you could do with super intelligence. I'm interested, do both of those categories of risk worry you?

Does one of them worry you more? Should we be thinking about that in the AI risk world? Do they worry about certain things too much? But how should we be thinking about AI risk and what some people would call X risk or existential risk?

Alex: Maybe I should start with an epistemic status. I don't think about this maybe as much as I should in terms of I'm quite pragmatic and I think about things through the lens of what could Europe do? But I'll, I'll like to indulge the question. And I think on like the first and octopus risk if you wanna call it, I think it would be foolish not to hold some amount of huh, we should think about this. That I'm not confident that this is impossible. I think I'm more optimistic in the sense that I can definitely see or smell like technological pathways that lead to good outcomes there, right?

it is not like I'm oh, any possible version of like super intelligence or like powerful different intelligence you build will always, do you, I think that you have a lot of agency over like the direction of the technology that you, build, but you have to exercise that agency and you have to be really cognizant of it.

These are like my priors and I sort of see like most of my takes, like informed by tweets that I see, but it seems that we have taken decisions in different directions that have been both negative and positive in terms of our ability to shape these intelligences in the way that would be desirable.

And the misuse risk, I think this is basically a force multiplier to possibly other misuse risks. Like if you're already worried about bio risk, many of your friends who were worried about bio risk, they're now just really, really worried about bio risk.

I think it's something that we should clearly pay attention to. I think maybe the thing that should make you worried is the pace and whether or not you feel like your governments are sort of on the ball and whatnot. But, you know, in this State of the Union address that the president of the European Commission did, she talked about how she opened her AI section talking about risks and talking about loss of control.

The equivalent of the Secretary of State for technology of the EU has talked about loss of control multiple times before. I think they think about this, they think they care about it alongside a bunch of other things they care about. The question is what can Europe even do about that?

And that sort of gets you towards I guess some of the things that we work on and whatnot. But that's, I sort of employ some of these like mental tricks in terms of I don't think about the fastest possible takeoff scenarios because there's nothing Europe can do about those takeoff scenarios.

So it's not useful for me to think about that. And I sort of limit the amount I think about certain risks because I'm more focused on what is within the realm of pragmatic and the possible within this operating theatre that I exist in.

Ben: Sure. What is tractable? What you can only control what you can control, even if you can distantly control it.

I mean, I remain pretty optimistic on that. Part of it is because actually we've had pretty terrifying capabilities for quite a while. And actually, I don't wanna give too many hints, but you could think about it and you, if you wanted to do evil, evil people could have actually done quite a lot more devastating things in the last 10 or 20 years than even that they've managed to achieve.

And they haven't for a complicated set of reasons, which I think should still hold. But maybe let's flip to the other side. So, AI optimists, and the opportunity side. What do you think AI optimists are getting most right, or what should we be hopeful about? And there's a lot we could cover.

Everything, I think around science is pretty good. Second order things. I actually think that because I do some work a little bit in climate and health, all of the climate people are underestimating what kind of gains this could give us in solving some climate things. The health people actually are beginning, they're actually already beginning to see it, so I'm not sure they're underestimating it.

They're kind of seeing it and not sure where that goes. But I'd be interested in your view, what are AI optimists getting right? I guess you can lean to Europe if you but it could be anything in the world that you are seeing.

Alex: I come from a maths background, so I'm just like incredibly excited about everything happening in maths at the moment.

It's just: wow, what a time to be alive. Right? We're like solving all of these. I mean, we solved the (I can't remember the name) the [Millennium Prize Problem], Navier–Stokes. We solved Navier–Stokes. There's like rumour that two and a half other ones have already been solved.

Ben: P versus NP or whatever it's called.

Alex: P versus NP. Yeah. You know, like register your predictions now. You don't have much time left. So incredibly exciting time, to just be a nerd like mathematics. And sort of you can extrapolate along. I do sort of hold this opinion, you know, like the verifiable areas will move faster than anyone thinks they will.

And then the non-verifiable stuff will move slower. So I think on like the, health side and on the biology side, I think all of this really sort of annoyingly just reinforces like how important all the other sort of science reforms you're going to make will be of like our clinical trial reform now actually is like a lot, lot, lot more important because you could do so much more with it, but it's still going to be the bottleneck. And so I'm curious where the bottlenecks end up arising in science. physics will pop, like maybe with physics we'll also see some crazy progress. But I sort of have an ongoing debate with colleagues and friends about how much transfer you will get from the non-verifiable domains.

And I think even in, even within mathematics, it's interesting because there are branches of mathematics that have easier to formalise than others, essentially like it within Lean. It was written by algebra people. It's written in a way that makes algebra very easy to formalise. I have friends who are like graph theorists who are this is very annoying.

I can't write my stuff in Lean because it's written by people who speak like a slightly different language. It's it's like your alphabet was written in French and you're Italian, you are like Greek or something. And you're trying to use the programming now because you're well I don't speak Italian or French.

And so I'm paying attention to will the AI model solve problems in those? Like the AI models have been trained in French for maths. Will they solve problems in Greek as well for maths? And this is like a thing that I'm paying attention to. And maybe I'll get a hint of whether that happens then I'm way more excited about the future of science.

Ben: I think there's something with that point in particular, I agree, they, the AIs, the code will have to, I think, invent the French. So they'll have to do their own version of lean, which isn't for algebraic things, but I think they're now fully capable of doing it. Even my sense, this is just vibe really, but if GPT-6, [Astra'] has already trained seven to be able to do Navier–Stokes and we've got some self-recursive, then it seems to me that seven or eight is going to be able to build those things.

But I'm then interested just to dig one level deeper. So if AI is really good at these verifiable domains as you call them, and if intelligence becomes really abundant, what becomes scarce or what do you really think are the bottlenecks? So we, you've kind of blobbed it as non verifiable, and I'm going to expand that to say some of this, what I guess the economists or social scientists would call tacit knowledge.

Some of it is called Polanyi knowledge. So these are things like how a carpenter builds or all of the things you get in a meetup as opposed to what's written in code. I guess all that goes into Polanyi-type domains is the type of stuff which becomes scarce. And I guess the bottlenecks, if we talk about clinical trials, I as yet don't quite see how you're going to get around putting the drugs into humans to make sure they are safe.

And actually some of this efficacy and clinical trials are quite a big bottleneck, although I do think we should be substantially faster. Clinical trials take on average eight to 10 years from first in humans, six to eight years should be, I mean, should be a no brainer. And we should have already gotten there in one to two years.

Seems to me like a very big jump, which is where we ought to go. But I'd be interested if you've had any thought when you've seen about that, about emerging bottlenecks or because your bottleneck is still, ah, we haven't got enough people in policy thinking about this. This might not be a domain you've thought about too much, but interested in bottlenecks or what becomes scarce in a high intelligence world.

Alex: Yeah, I have a colleague who's thinking about this. I should talk to her more often. I think even though you talked about this like tacit knowledge, I think even the non-verifiable stuff is stuff you are just trying to learn about like biology and stuff and you have to prepare everything.

Like you don't have a fully self-driving lab. Even within, like, I am still deeply confused about recursive self-improvement and whether that's actually happening and whether you've cleared the bottlenecks. Because I'm not convinced that you haven't cleared all the bottlenecks and there might still be some bottlenecks.

But on science itself, I am.

I think that, and maybe this is just from like a metasciencey direction, but there's a lot like the way that. Scientific information is communicated between scientists and the way that the social dynamics of science that seem like hard to replace and hard to get that serendipity from.

But maybe we'll get there. I sort of find myself often thinking oh, I'm so grateful that my job is safe because all I have to do is talk to people all day or have face-to-face conversations. AI's not going to take my job.

Ben: Yeah, communications. I actually think humanities in some ways should be less worried than they are.

I mean, they're, they're really not going to disintermediate theatre for instance, or albeit not many people actually make that much money in theatre. And yes, maybe you will speed up some aspects of script writing, but people really don't like that. I mean it is half of art, or at least some bit of art is all of the tacit knowledge, which comes with the artist and everything around that.

That's let alone before we get into questions of or style and taste on that, about whether it can do that. And even when it matches that, I think human audiences will not find that as satisfying. But I'm interested then also in what you maybe changed your mind about, we could do this on any level. Because I think you're involved in this proposal for a type of CERN for AI and institutions and kind of how institutions work and then subsequently you've kind of moved away from that form of thinking. So perhaps specifically on that, what changed your mind and then anything else over the last two or three years that you think that you've changed your mind about or as they say in some communities updated your priors on, but anything you've changed your mind on.

Alex: Yeah. I changed my mind so many times, like the whole CERN for AI journey. Like the main one is like any sort of publicly supported effort, needs to answer, answer the question of okay, you train like GPT-N, what's the plan for GPT-N plus one? Because you can corral all this public funding.

You know, in the most recent transformative AI strategy for the EU that a bunch of these experts, some of my colleagues included, published, they're to train a competitive model to get to the frontier now would cost 780 billion euros. Okay, you spend the 780 euro billion, now what?

Because do you have a competitive company? Are you going to create like the revenue loop that OpenAI, Anthropic have to train the next model? It just all falls apart if you are trying to directly compete at the frontier. So then the idea evolved more and I think the CERN idea evolved more into like what sort of R&D areas you could work on that could still be quite important.

That could still be quite important for resilience and still be quite important for competitiveness. And I think there are versions of these ideas that I'm still standing behind and still keen to push and you know, I'd be very keen for the EU to do something like a DARPA, maybe.

Other things that changed my mind on that. Just like priors around like centralization of power. I used to think within the terms of like one big AI, like the big super intelligent AI and I'm much more like actually that's like there's a cognitive bias that you're employing there when you think about that.

Like it doesn't have to be that we don't have to build the technology in that way. We could have multi-agent systems and have lots of different personalised AI. And so that's like another place I think. I think I've softened and nuanced my opinions about governments and bureaucracy and there are good people within the government who are trying very hard, but incentive structures that allow them to do the things they want to do.

And then there are also some people with just some horrendously outrageously wrong opinions. And if I was a dictator, I'd send them to [ '] or whatnot. But, so people contain multitudes.

Ben: Yeah,

I heard Tyler Cowen recently somewhat defend bureaucracies, which was quite interesting.

He was taking the other side of the view and said, well, maybe they're doing something that we don't know about it. I think this was more like in America saying, well actually, the fact you've got to fill in a lot of bureaucratic forms means that some aspects of fraud go down. Because if you want to claim benefits and you're on the margin, you're not going to put all of that effort in until you're really doing it.

I mean, I guess that was on a, on a different level and we spent quite a lot of time, I guess, softly critiquing institutions and bits by that not understanding your little softer aspect or some good people working on that. But maybe taking the flip side is there any institutions or organisations that you've accounted for that you think work surprisingly well and can we learn anything from that?

I mean, and on the other side, I do think institutions have been surprisingly robust or even anti-fragile in this current world of geopolitics that we've seen that, not many of them have imploded. Not as many of them as I thought could have done. I'm not sure whether that says that they're good, but they certainly seem to be somewhat resilient.

But I'm, I'm interested if, yeah. Do you think anything has worked really well and can we learn from that?

Alex: Be really bad if I didn't have an answer.

Ben: It might be you've not really come across one, which has worked super well. I mean, I guess in governments, people sometimes say Singapore or Estonia. I think they have advantages of working really on a small city state level. So that's kind of interesting.

And then some other organisations tend to have been maybe shorter lived or more nimble or specific small institutions. And I think it's interesting that I've detected actually, I guess from some of these foundations, which are actually being funded with AI money that they're starting to do institutions, which are just tackling one particular subset area.

I can't even remember the areas, but it's something just health data or that's it. They wanna solve Alzheimer's, so they're not going to do anything else, but they're going to start an institution and foundation just on that because they've thought that those sort of more time limited, purpose-driven, mission-driven, driven, but with a more well-defined mission have gone well.

Although I do think you can still critique that, I'd be interested. Have you seen anything?

Alex: I think that my interactions with the Greek state have been like much better. Everything's been digitised. You can get every form online. Previously you'd have to go to some dusty office.

You'd have to go to the thing. no, you need this form. You have to go to another office, the other side of Athens to get this form. And then go back and no, you also need this form. Spend hours and hours going back and forth. There was a minister, Pierrakakis, who championed a lot of these digitization reforms.

And now he's the governor of the European Central Bank, I think. And so, yeah, they can make your life better, make the burden on your life less bad. Another example is, is it Taiwan that have little lottery tickets on receipts so people hold onto the receipts?

Alex: Is it cheating if I just fanboy about the Institute for Progress for like 30 seconds?

Ben: You can do that.

Alex: Sure. I just think they seem to have just been insanely impactful and both in sort of very short term, let's do a thing. And in these like long term bets and building something of like over five years that has been really, like long lasting and has moved the needle in a really positive direction. I sort of look at them, and I'm like, 'That's ambitions for what we could be doing in like five years.

And then yeah, so I think that they make me bullish that you can do things that are like sort you know, they're, they're, they're basically an extra branch of government and, whatnot. A meta branch of government. And then there are things within the focus research organisation space.

Like ARIA has good vibes. I don't know if it has more than good vibes, it's like it's still too early to tell. In some sense,

Ben: You've got to start with the vibes, you've got to start from somewhere. Yeah. I mean even, exactly. Even if they all fail. And in some ways, I think actually it'll be fine if every single one of their projects fail at the meta level.

Just saying we are going to actually try some moonshots and things is, I think, I think good. And I also do think they've taken Economics 101 and they are funding things that private markets basically can't fund as in private capital. They still won't be that. So I think if every single one of those fails are that, and also I know people complain, but actually in the grand scheme of things, it isn't actually even that much money.

Alex: Marginal.

Ben: Yeah. People really struggle with hundreds of millions and billions. It, you know, we're fine in the thousands of pounds and even tens of thousands and whatever, but once we get into the millions and billions people just, that's one thing that investors and markets do have a sense for, because we have to wrap our heads around billions and now actually trillions.

But it's a completely different scale of things.

Alex: Yeah. I am also an ARIA fanboy but I, you know, I will say the vibes are immaculate and one should be careful as to whether that is, how much of that is motivating your enthusiasm.

Ben: Yes. That's true. But I also think, so this is, some, so, you know, I've had a podcast with Alec Stapp and we talked about startup things, and I've also had a podcast with Stian Westlake who's talked about funding and how you make decisions and actually Michael Nielsen on metascience and how to make decisions.

So putting all of that together, one of my things is that people claim they do high risk things and then they go, oh, look, two thirds of these projects worked out. Actually, that must mean that your first statement was wrong. Because if you have even moderately high risk, then you cannot be more than a one out of 10 success, which you can see in biopharma and you can also see in VC.

So I do think that most of those projects should actually fail, or at least only advance a little bit. So I think maybe we will only have vibes to go on, but at least as we said, the vibes are good.

Alex: And it's so weird when you get to politics. Because if you're like a political operator, nothing is guaranteed, right?

You have to hedge, hedge, hedge, hedge hedge. You try like a bunch of things. You put in all this effort and it fails and you put on. So in some sense, like people working in bureaucracies, people working in politics, are operating in this high risk environment where lots of things fail and only few things succeed.

But they seem to have internalised it in the wrong direction. I haven't thought enough about what's going on there. But you would think that. That it would maybe go in the opposite direction.

Ben: Yeah. I think in my mind there's a mismatch on the timing of when you get feedback. So Mm-hmm.

The thing about at least shorter term R&D projects, or even within biopharma, which I know quite well, and this is your whole point about what is verifiable or not, your trial fails after a year. So you get quite good feedback quite quickly in the political cycle. Not only would you maybe not know within your political cycle, but your KPI might not be that clear.

So you start something you vaguely think you want GDP to go up. I'm going to slightly make it up, but you have no idea really what your intermediate KPI was pointing in that direction. In any case, we've been talking for a while like people who spend far too much time thinking about AI and policy. Maybe my pivot question should be to the person in the street who probably maybe doesn't even particularly care too much about AI or policy or any of this stuff. What should maybe be one thing or something that they should be thinking about here?

Alex: Interesting. Because the person on the street now comes up to me and asks me about AI safety and I went, I was in my dad's office the other day and they're like, 'Is AI going to kill us?'

And I went to a wedding and two people asked me, is AI going to kill us? So that's the wrong question.

Ben: And what data? The wrong water and data centres is also the wrong question. So what are the and that's what, and that's what obviously the mainstream media or even not even the mainstream media, media is all around that.

So yeah, we get a chance to communicate maybe one or two things that they could be thinking. And I also, I would quite like it to be not, oh my god, they're going to take on jobs as well, which I don't think is helpful. I'm not quite sure why they focus so much on that. You might not have anything, but if you do, what, should the person in the street maybe be thinking?

If it isn't worrying about data centres, isn't AI's going to kill us or take our jobs?

Alex: I think it's bottlenecks and sort of reiterating this thing of if something is automated, it just moves to the bottleneck. Please think in terms of bottlenecks and start thinking about where the bottleneck is, and it's sad that you have to do this. Because the government shouldn't have to be thinking about where the bottleneck is because that means the system's broken normally. Like the market does that. But, I digress. So bottlenecks are probably one comparative advantage. And this notion that even if AI is much better at you than something like at everything, if it's much, much better than you at something else, it will only do those other things, right?

And so, comparative advantage of countries and different countries should do different things. And then finally like we are not measuring enough, like nearly enough. Like everyone, everyone who's talking to you about AI's going to do this, AI's going to do that. Like we have no numbers.

We don't know, like we don't know how many jobs, all vibes, like taking away all vibes. It would be great if we had some numbers. But we don't have any numbers on the safety front, numbers on like the employment front numbers on like even the capability front. I don't know if you saw how Epoch AI did a benchmark.

They basically evaluated, meta, evaluated a bunch of benchmarks. And more than 50% of the top benchmarks are dubious and 20 to 50% of them have false positives and false negatives. The answers flipped the wrong wave round. Like there's data contamination so we're operating with a bunch of bunk numbers or could be better numbers.

Ben: Yeah, what's the aphorism? You only manage what you measure but if you're not measuring it right. But we've got this whole thing because the UK has just come out with new data from our statistical agency and it turns out productivity wasn't quite as bad as we thought for all sorts of reasons. And they don't even really know what's happening in the job market.

So there's definitely something on the measurement front and I like the comparative advantage, which I guess is Economics 101, but with this split between the tacit knowledge bit or the non-verifiable bit, which humans are still going to be somewhat good at to maybe very good at. And there's definitely going to be a comparative advantage versus AI and other people.

Great. Well I'm going to pivot to maybe some more personal thoughts or personal creativity thoughts before then going to a more quick fire round and rounding up. So you were also Cheese Lord, captain of Warwick's Overwatch team. So, yes. Why Overwatch? Why not Valorant? And what do you learn from being in competitive video games and being a captain?

Is there anything we could learn apart from the fact that it seems to be fun or maybe stressful?

Alex: Fun and stressful and very rewarding. Valorant hadn't come out, up until that point, but I really don't like Valorant. I really dislike games that require you to stand still while you have to shoot.

It's just a very, there's something about me that's no, why am I standing still? I want to be jiggling left and right. What's the, where's the fun in that? I quite enjoy games with movement. And so that's why I guess Overwatch drew me and I had to be, it was again one of those things where a friend would be you have to play this game.

You would love it. You have to play this game. I ignored them for so long. They eventually bought me the game just so I'd play it and I was oh, I love this game. On like the esports side ka there's like this whole like video games and being good at like competition teaches you a lot about starting in a domain, being really shit at a domain. Starting off and being like bronze or silver or gold and slowly you'll kind of like platinum and then diamond and masters and Grandmasters and maybe you'll now like top 500, you say I am one of the top 500 people in my country, in my region. And you have this very clear reward signal and that I think you can generalise out from that and be ah, I can like get better at things.

There's also a lot around it like team competition and being able to. So I think what's really nice about university esports is the fact that you play with what hand you're given, right? You have a bunch of students that show up. You have to find the best that you can get, and then you get like a freaking Roadhog one-trick, and you get a Symmetra one-trick, and you're like, 'What the fuck am I going to do with this?' And you make it work. And I think that we were very successful. We went back to back years. And what made us successful was playing to these strengths and being a really weird team that was very okay with the fact that we are not playing perfectly. We wouldn't be world champions, but we can be good enough to beat every other student team by leaning into these weird niches.

And I think that in games and in life, if I'm, if I dare to philosophise, briefly, like the people haven't explored the space of ideas nearly enough and haven't sort of tapped into the market's not efficient. They haven't tapped into like all the alpha and how far you can take an idea and how far you can stretch and optimise something. So have you just tried winning?

Ben: Yeah. It's amazing that I'm going to loop together so many of our thoughts with that because leaning into your team is essentially leaning into your comparative advantage. If you've got some oddball technique, which I, that's they're the best at two things.

They can't do anything else, but you're going to have to lean into that. So that's really interesting. And I only discovered a few years ago that Elon Musk, for instance, still plays a load of video games. I was seeing him play the recent Diablo game, which isn't even that complicated game. And then I don't, there was something else, which I thought was really fascinating.

And I do think generally I've, I've got a bit old, so in my youth, the first-person shooter was Doom came on. It was like the first thing within that and it seemed so incredible, but it's gone down a little bit. But parents still, I think, fear too much about children in gaming because they are learning an amazing amount of things, particularly if they're just something on like Minecraft that they're doing all of this stuff.

Yeah. We haven't pushed the frontier of this. So, I was listening to a philosopher who was making some very, leaps of faith arguments around what can speedrunning games teach you. Mm-hmm. Because you're losing all of these. I never, I'm, you are obviously we would know, but people who speed run these Mario games or these quite complicated games where.

A standard person would take a standard, really good gamer should take days to play through these like hundreds of hours and they cut it down to four hours using these like crazy methods of which we're definitely not on the frontier of what all of that. So I think that's all really fascinating.

So, I mean, maybe putting it together, what, any games that you would, you, like playing now or you would say, oh, if you, if you've never really thought about video games, this is, something that maybe you should, you should trial. And maybe while you're thinking about that, because I think games is another aspect where we don't have to be that worried about AI.

I mean, AI [is] kind of solved chess and go, or in the sense that they are now much better at chess and go than the very best human. And chess is at an all time high in terms of people playing. I mean, people didn't play that much go, but I'm sure AI is much better. I mean, obviously it can control other, other things.

If you had an AI bot it would be better at the shooting game. So that's not why we're going to play shooting games. But anyway, any games on your mind at the moment or anything we can learn.

Alex: A few different directions. It feels more like a puzzle person. Like another thing I put in a variant of Sudokus in my free time because I don't do enough maths-y logic-style stuff.

So I just like to sit and I play my Sudokus. They're very satisfying. There's this YouTube channel called Cracking the Cryptic. They blew up in the pandemic. There's a game called Blue Prince, which is like a puzzle game, if you like puzzles at all. It's an, it's a phenomenal game. I'd recommend it.

I think as I've got less time, older and I can't, I don't have time to be good at muscle memory or grind away. I'll play more strategy games on my phone or whatever. My vice is Clash Royale, but I would not recommend that there's a game called Super Auto Pets, which is an auto battler.

It's a bit like chess. I still think Overwatch is fun as like a competitive game. I think that there are games like Civ VII or like Age of Empires. I can't play those games anymore because, you know, I stopped playing them and I recently picked up Civ again. I played it for a day. I was oh, I'd love this game.

Why haven't I played this? And then after like 14 hours of sitting in my chair, I realised why I don't play that game anymore.

Ben: Yeah, more addictive than anything else. That's a bit dangerous. Yeah. I, in general, there's I think there's a lot you can explore whether you wanna explore like the pure one-v-one nature of there is no outside force that I can blame for this.

Alex: This is either my skill difference or your skill difference. Versus like the team based games where it is like, you enter with a suboptimal hand and it's, 'How do you play this suboptimal hand?' And lean into the comparative advantages, like you were saying.

Ben: Yeah. That's all great.

So maybe thinking about your own personal creativity. So you wrote this 2031 with a team of, maybe it's about eight of you. I can't quite remember the count eight in total.

Alex: Yeah.

Ben: So, and

Alex: Many reviewers and many people behind the scenes supporting.

Ben: So do you write, so there's no correct answer to this at all, but are you a morning writer, an evening writer?

Do you write in bursts, do you think in the shower or in a coffee shop? Or do you think best through a conversation? Do you feel like you have to write it all down? Do you write long hand? Do you write straight onto the computer? Do you even voice type? How? Yeah, I'm thinking about what makes you productive or creative and, how you think about your own processes.

Alex: So I definitely enjoy a long ramble to Claude or like put Wispr Flow up. It's like working in an office. I'm able to do this less. It's quite annoying. But I usually like being able to ramble about a difficult topic for like 15 minutes to Claude and then have it chug away and process that and think through.

I've made big progress on difficult problems before doing that. So big fan of that. I'm someone and that's

Ben: Because Claude is prepared to listen to you more than a natural person or can also digest what you actually said in a more legible form back to you, which probably an ordinary human can't.

Alex: Many things. One, you're just able to provide so much more, and richer context.

Ben: Yeah.

Alex: Like the amount of context you can provide 15 minutes spoken to Claude is and it's like

Ben: Days.

Alex: Yeah. An hour. Yeah. An hour of typing at least. And you wouldn't spend an hour typing your prompt.

Maybe it would. So first context is just like so much better second, yeah. Through the process of talking. You know, you talking is a version of thinking. It's like prompting is a version of thinking. So it's more like a rubber ducky or a thought partner. And you're able to iterate on the thoughts.

While I'll sometimes talk through a problem or and, through the process of talking through it, I'll end up with a skeleton in my head. But the process of writing itself I was actually on a writing retreat recently in Barcelona. I sort of have close fresh experience.

It's laborious and painful for me. It's excruciating. Maybe it's because I get distracted all the time, but like,

Ben: And it's not just being in Basel. Too boring. Yeah. I'm just joking. Sorry. Basel, Swiss people.

Alex: No, that's really funny. I was in Barcelona, not Basel.

Ben: Oh, okay. All right. Oh, that Barcelona, you can't, you can't blame for being unfun except that the vibes I get is that they don't want any more tourists coming in, so, yeah.

Alex: Yes.

Ben: So definitely. Sorry Basel and sorry Barcelona. But anyway,

Alex: Beautiful architecture as well. But, there is, it is useful, like the pain is useful, both in terms of informing: is this actually a tweet, not a blog post. But also in terms of why is it painful? Like what is the idea that I'm trying to articulate that's like struggling to be, born.

And. I think I'm, the one thing stuck with me was there's like this curve of like enthusiasm about an idea. And you got to start writing about it at a very specific point on the curve. Because if you start too late, you're going to be fall off the curve. But if you start too early, like you haven't had the process.

So being able, like if you have the flexibility to clear time, you're like, 'At this point in the curve, I'm going to write about it now.'. Otherwise you ends up in like draft jail. Yeah, but the writing retreat was great. I showed up to it thinking I was going to write about Europe and AI and ended up writing about my feelings.

So yeah.

Ben: That's great. I like your sense of that curve. It's a little bit similar when I do playwriting in some respect, which is if you show it or try and edit it too early, you definitely kill it. But also if it's too late, sometimes it's bored, like you said. Well we might touch on the end about feelings about what we might think an ordinary Tuesday might look like or what the future might hold. But we'll do maybe a little section of overrated, underrated, and then finish with a Mm-hmm. Couple of questions. So, you can go overrated, underrated, you can even go correctly rated. You can give a little thought on it. You also don't have to. We can, we can pass. So we're going to go overrated, underrated recursive self-improvement,

Alex: Overrated.

Still might happen, but everyone is way too confident. I don't know how they know. You're not in the lab, you don't see what's happening. You don't know what the bottlenecks are. Have good epistemics.

Ben: Yeah, that seems fair. I think a little bit of it's happening, but way, no, over, like I would not put it over 50% at all.

Like probably 5% or, something like that. Great. Universal basic income. UBI overrated or underrated?

Alex: Overrated. I used to be in the Yang Gang. But I think the political economy of it breaks. You know, a new politician comes in and says, I will raise it. I will raise it. I would be happy if we could magically fix it at the optimal threshold.

I mean, I don't think you can do that.

Ben: Yeah. Not protect from politicians. Also the latest small trials haven't been quite as good as we thought they were. So disappointing, but, okay. Overrated. So, AGI, although I guess we don't even know what it is, but say artificial general intelligence or I guess we could do ASI artificial super intelligence.

Either one of those, some something like that. Overrated or underrated?

Or no

Alex: Rating. I have conflicted views. Overrated in the sense that I think that there are some wrong—overrated in the way that most people talk about it. Like constrained to the set of people who talk about AGI and ASI overrated because too confident about certain ideas, too confident about like the nature of intelligence and whatnot.

Underrated in the sense that not enough people are talking about this and everyone, everyone should be talking about this if this is even remotely possible.

Ben: Yes. I think that's fair. I have the same view on expected value. The people who think about expected value think about it way too much and apply it to too many areas of their life, but an ordinary person doesn't think about it at all, and so they need to think about it a little bit more so it becomes both overrated and underrated.

Okay. The idea of taxing compute or taxing tokens underrated or overrated?

Alex: Super overrated. I think it's a terrible idea. I think we should just tax consumption and tax land in Alex's utopia we tax both these things we're fine. Maybe, we're fine ongoing debate, but I think the burden is on the tax compute people to really rebut why we can't do those other two taxes first.

And why their taxes are better than those.

Ben: Sure. Consumption taxes. Pigouvian taxes. Very classical Economics 101, right. They still argue for that. Hasn't been put in. Great. Open source AI. I guess this could also be open-weight, open tokens, but open source AI overrated, underrated.

Alex: I think that the people who boost them are overrating them. I think the reason they're doing that is because they haven't realised just how much they are at the whim of the direction of technology. Like the people who want open source to be prolific the most for that world to exist, we have to develop a bunch of technologies that improve resilience, that improve safety, that make this open stuff deployable.

And these people are the ones who are least caring about this stuff. Guys, come on.

Ben: Yeah, that's fair enough. Prediction markets overrated, underrated. I mean, I guess you can, normal markets are forms of prediction markets, but I guess overall prediction markets.

Alex: Does sports betting count?

Ben: Yes.

You could put sports betting in that. So you could do a sub, but I, this is more, I guess, political and other betting Manifold or Kalshi and things like that.

Alex: Yeah. Without the sports betting underrated. With the sports betting overrated.

Ben: Yeah. We don't need more sports betting. We have all of that.

Great. Final couple, I guess, effective altruism has a probably slightly bad rap at this very moment in time, which is where September 2026. But generally, do you think overrated or under—

Alex: Overrated. Definitely within the European continent. Like the version of EA, the sort of continental EA, which is, a specific shape of EA that I, that I think we need to move past and, build something else to replace it.

Ben: Yeah. Overrated. Great. And then last one, science fiction. I guess you could do science fiction as policy, but science fiction. Overrated or underrated?

Alex: Saying underrated because I've been, I've been reading some Harry Potter fan fiction recently and I've been enjoying them.

Ben: Yeah, it's got to be underrated, right?

Anything which makes us think more broadly. Although I do think we do need to shift as I was, we talked earlier about to the Jedi light side rather than any more dystopia. But I guess, we'll have, we have something, we've got to have our villains as well as our heroes. Great. So, we spend a lot of time imagining a future Europe, particularly a future Europe, that we should avoid.

What's the future that Alex actually wants?

Alex: Vending machine on every street corner. No, I think the future I want is one where young people can like look into their own future and be ah, this is a future that I'm excited about. This is a future that I want to live in. And for that to happen, we got to radically restructure the way that society and the social contract has been like playing out.

We've got to get a bunch of things right on technology, because this is like the most important thing that is important for young people's futures, but also government in the way that government has like stacked the book against them. And so I think about the future, like within the frame of what would an a, would an 18-year-old Alex be happy to live in like 20 years from now or 30 years from now?

And currently, no. Because I don't think 18-year-old Alex would be happy to live in like right now. And we keep going in the wrong direction then, but we're going to fix it and we're going to we can fix it and I believe we have to fix it. Sorry for vague bursting.

Ben: Yeah,

no, I'm optimistic on that as well.

Well, maybe I'll push them one. Maybe you could think about a detail. So we assume we have fixed it into, I don't know, 2050, or you can go even further if you want 2100 sometime in the future. And we can forget GDP and compute and geopolitical power. We've somehow solved that at least. To Europe's somewhat satisfaction, we've, we've got to somewhere.

What does an ordinary Tuesday in that future look like? What are two or three things that ordinary Alex, 18— So this is a really interesting dilemma because it's your future self, but you're back at 18 years old. So what does, what does that look like?

Alex: I think it doesn't look too much different from today in the sense that that Alex wants to wake up, wants to go to the bakery, wants to walk to the park, wants to see his friends there, and bump into his friends on the way to the park and the bakery and then they go to the park and they eat their croissants or whatever. And in this future Alex has received a magic like engineering shot that makes him able to consume lactose and he is able to enjoy his croissant in the park. And he feels safe. And then maybe like he's then able to hop on a really fast train and pop over to London for the day and visit all his friends there. And it's really cheap. And he's able to have that lifestyle and I think that lifestyle is just like ludicrously out of reach and then live close to where you want to work. And I think that life and your friend, that lifestyle is likely out of reach today that feels like fantasy.

Ben: Yeah, good health.

Good health. Being with your friends. I guess just on that one you mentioned in that, do you still have a meaningful job in this? Was it just a choice that people will want to have to pursue to what sort of meaningful job for them and therefore they have the opportunity space that if they want that they can pursue it?

I guess I speak to a lot of young people who feel that is diminishing. So that, you know, that is of now, there's only a small segment which have the right opportunity space to kind of do the meaningful work they do, which is kind of interestingly why medics are still quite popular and also teachers and some others.

I guess the small niche of policy, also people going into it and, things like that. But I think this is one of the bits that some of the UBI enthusiasts have also got wrong is that they don't understand that people actually jobs have always been meaningful for humans. And it might not be this artistic creative job where you sit around and paint all day or even make video games, which will be a portion of it, but it's actually something else connected with human flourishing.

But I'd be interested if you think you've still got this meaningful job and actually what your 2050 meaningful job for 18-year-old Alex would be, assume you've solved policy, although I guess policy will never be solved. Maybe you're still in policy, but you'll, you have to do something else.

Alex: Yeah, I have two, two senses of like what 18-year-old Alex wants.

It's like part of me wants to have agency over my own future and over like the collective future of society and humanity and be able to meaningfully improve and contribute to that future and make it marginally better. I don't know, send me up, on the space elevator. I want to go supervise asteroid mining Belt 12.

Another part of me really just wants to create whimsy and joy for other people and see smiles in their faces and just have lots of bits. And lots of like stupid jokes. And pull a bunch of pranks on people all day. And that also sounds quite nice. I would be miserable without either of those.

Ben: They sound like two great things to have. Great. Last couple of questions then. Anything you are working on now, current projects or future projects that you'd like people to know about?

Alex: Yeah. I don't know like when this will be ready, but I can tease it. I talked about how we, don't have ideas.

Our ideas suck. So we want to fix this. So we're, we're doing an essay competition or writing competition or something of that sort. Basically okay, you read Europe 2031 or something else. You care about this problem now you want Europe to be prepared for transformative AI, whether it's for its resilience or it's flourishing or its agency what you're going to do about it. What should we do about it? Write a letter to someone who has have like an audience or recipient and outline your idea of how we should get out of this mess. I don't know when that will go live. Probably not for a while, but you can already start thinking about your solutions.

Ben: That sounds like a good idea.

I do think you need to get a celebrity to front it. So I've seen a couple of writing ideas, this year. In fact, there's one which has Neal Stephenson and Gwern on GPU.

Alex: So there's, you've got to write a story It has to be AI-generated.

Ben: Yeah. Or no. I think

Alex: The Gwern one has to be an AI-generated piece of literature, I think.

Ben: Does it have to be AI? I thought you just had to disclose it.

Alex: I think it, I think it has to be fully AI-generated. [Ed: No, only disclosed]

Ben: Interesting. But regardless, so that had some publicity. So when you do send it out, that would be my, but I think it is quite interesting. I do think we need to incentivise the storytellers to be involved.

For sure. And so then the last question is, do you have, I guess we have given quite a lot of advice out for listeners, both the ordinary person or the person being involved in policy and you have a big call to just get involved and do more thinking on that. But any other advice that you wanna leave for listeners? It could be, what character they should choose in Overwatch or more general life or career advice or any final thoughts that you want to leave

Alex: Arq Foundation/careers we're hiring.

Ben: That's a good one.

Alex: We also have referral bonuses. So if you find someone we should hire, we'll pay, we'll give you a thousand euros.

Maybe it'll be more my generic career advice. Write, please have a Substack. Even if there are only three posts on it, I want to see the way you think. I think the agents would also appreciate seeing the way you think. I think people need to be founding new things. And you don't necessarily have to have a good idea. There are [fuders / organisations] that can bring you money. So, don't let that shy you away from founding something cool and impactful.

Ben: That's great. So do build, write in public, check out Arq Foundation. Apply if that's a good job or make your friends apply even better, and get a bounty.

And with that, Alex, thank you very much.

Alex: Thank you.

Key takeaways

Further reading

CONTENTS, TRANSCRIPT AND PODCAST LINKS BELOW

Contents

Transcript