Bruce Schneier: Don’t Entrust Your Secrets to AI

Not even the sweltering heat could keep developers, researchers, journalists, students, professionals and enthusiasts in AI and cybersecurity from filling the main hall of Prague’s Opero centre at the end of June.

Together with AISLE, we hosted a meetup with security expert and cryptographer Bruce Schneier, who – in conversation with AISLE’s co-founder Jaya Baloo – addressed topics such as a proactive security approach, responsible software development, AI regulation, and the fight against cyberattacks.

We then followed up on this discussion with Bruce in an interview for prg.ai, in which he further elaborated on the issue of trust in artificial intelligence, as well as the relationship between people and new technologies, democracy and power, and the development of new systems in Europe and the Czech Republic.

Your talk is about AI and trust. You draw a sharp line between two different things we call “trust” – the interpersonal trust we have in people, and the social trust we place in systems and institutions. Why does keeping those two apart matter so much once AI enters the picture?

It might not matter at all. I just think it’s an interesting way to think about trust, that there are very different kinds of trust. It’s actually a very complicated word with many different meanings. And in my writing, I do separate the trust you have in a friend, in someone you know, in someone you have a relationship with, from the more abstract trust you might have with your bank, your Uber driver or someone you just met. It’s a very different form of trust. And I think it’s interesting to separate the two.

When you are chatting with an AI, it is not a person. It acts like a person. It speaks in a language. We naturally think of it as a person because it’s talking to us – but it isn’t.

When we chat with a friendly, helpful AI, it’s easy to feel we are dealing with something on our side, even though there’s a company behind it, with its own policy, interests, and profits. How should people think about that gap and work with it?

I think this is important that when you are chatting with an AI, it is not a person. It acts like a person. It speaks in a language. We naturally think of it as a person because it’s talking to us – but it isn’t. And we do see people forming relationships with AIs. There are AIs that are designed for these pseudo relationships. There are people who believe they have emotional connections with these non-things. That feels dangerous for humanity. Don’t entrust your secrets to AI.

In many cases, these AIs are controlled by another company for profit. So, just as your search engine is trying to manipulate you to click on links you might not want to because they make money if you do, we can easily imagine in the future your AI chat companion might also be getting you to do things you might not want to do because it makes money. We know that in at least some circumstances, open AI serves ads, and these companies are struggling for a business model, and surveillance and manipulation are the business models of the internet.

What would it actually take for an AI system to truly be trustworthy, rather than just feel trustworthy?

That answer is probably a book that I have not written yet. It probably won’t be published for years. I’m working on it, but there is no soundbite answer to that question right now. There’s a fundamental problem with AI – it’s not trustworthy in several dimensions. I mean, this is not something we can spend a weekend and solve. It would take a lot of things. Some of them are technical. Some of them are social. Some of them are regulatory. I think it’s enormous space between where we are today and that future.

You’ve argued for the idea of “public AI” or systems built in the public interest rather than purely for profit. What would that look like in practice, and who would build it?

Lots of people can build it. It could look like many things in practice, but we can turn to an example that we have, and that is Apertus. Apertus is a model from Switzerland. It was built by a consortium of universities with funding from the US government. It has no corporate ownership. It has no profit motive. It is open source. It is responsive to democratic pressures, not financial and shareholder pressures. So that is one example of what it can look like. And that is one way to do it. There are other ways.

Singapore is working on a public AI model. Canada is thinking about it. Other countries are. I think there are many ways to do it and the many ways it’ll look. But the key unifier is that it is not a model created by a corporation for profit.

Your latest book, Rewiring Democracy, looks past deepfakes and disinformation to how AI is changing the everyday machinery of democracy – lawmaking, the courts, public administration. What surprised you most while researching it?

What surprised me the most is how many exciting things are happening all over the planet with regard to AI and democracy. The book has examples from the United States and Canada, France, Germany, the UK, Japan, Chile, Brazil, and many other countries, where governments are trying to do the right thing, trying to harness AI for democracy. And in some ways, both small and large, they’re succeeding.

AI amplifies the power of the people who use it.

One of its central ideas is that AI amplifies power, and the real question is whose power gets amplified. Right now, which way do you think it’s tilting? Who profits the most from it?

Right now, it’s everybody’s power that gets amplified, and that’s the thing about AI. AI amplifies the power of the people who use it. And if the people who use it wanna do good, AI will help them do good. If the people who wanna use it wanna do evil, AI will help them do evil. And AI does both.

I mean, right now, there’s very little profit being made in AI. There are enormous amounts of money sloshing about and investment, but there doesn’t need to be actual profit in any real sense. But certainly, the power is right now accruing to the large US AI monopolies. That’s probably not going to continue for a whole bunch of reasons, but that seems to be where the power centre is today.

These days, AI agents start acting on their own, making decisions, taking actions, dealing with other agents. Where do you see the biggest new security risks in that?

Right now, the risk we are seeing that’s new, and it’s not to say the old risks are not significant, they might possibly be more significant, but the new risk seems to be agents that are so good at doing what you want to do. They do them in ways you didn’t intend and don’t want them to. That they are “ruthlessly efficient”, in the words of Simon Willison.

I think this is a problem in trust, that if the AI doesn’t follow the unspoken rules, it’s going to break things even as it does what you want it to do.

Are you suggesting that human presence and oversight are still necessary when it comes to AI?

If that’s possible: in AI’s driving cars, there’s sometimes no human there. Sometimes there’s a human behind the wheel who can take control, but that kind of thing is fictional. A lot of times AI is making decisions that humans don’t make. There’s no time. They’re too complex. Their scope is too great. So this notion that a human can review an AI’s decision doesn’t make any sense. Google uses AI to summarise search results. There aren’t enough humans on the planet to review those AI summaries.

Europe has gone regulation-first with the AI Act, while the US has stayed lighter-touch. From a trust-and-security standpoint, do you think Europe is striking the right balance?

Of course, Europe is actually regulating these companies. I mean, designing our future for the near-term financial benefit of a bunch of white male American tech billionaires is kind of a dumb way to organise society. Yes, the US has always been anti-regulation, mostly because money controls politics. The money is in charge, and the money doesn’t want regulation. Europe is like a much more robust democratic government, and we do see a lot of regulation, not just AI, but the Digital Markets Act, the Digital Services Act, and the GDPR before that, much better, much more important. I like what Europe is doing. What’s gonna be hard is working in a heterogeneous environment where the US is so anti-regulation, but the EU is actually trying to make the world better. And we sort of have to see how that balance plays out when companies can jurisdiction shop and move to a low-regulatory country.

A lot of your examples come from the US. For a smaller European democracy like the Czech Republic, what risks or opportunities should we be watching that bigger countries might overlook?

The technology is too global for thinking that way. I think a lot of my examples do not come from the United States. They come from, except they do, they come from the states or local governments, not from the federal government. They do come from smaller countries. The worry I think smaller countries have is that they’re dominated by these tech superpowers. So anything the EU does to try to build its own tech stack makes a lot of sense. And that seems important. And I think there’s a lot of space for innovation around the world, not just from the big countries. So that’s always nice to see.

Break up the tech monopolies.

If you could put one piece of AI governance in place worldwide tomorrow, what would it be?

Break up the tech monopolies.

For the security people and AI builders in the room this afternoon: what’s one principle you wish more of them designed around from the very start?

I think we need to really think about integrity in AI. And we’re building systems that are capable. We need to make them trustworthy. Otherwise, they’re not going to be used.

Bruce Schneier is an internationally renowned security technologist, called a “security guru” by The Economist. He is the author of over one dozen books—including his latest, Rewiring Democracy. His newsletter “Crypto-Gram” and his blog “Schneier on Security” are read by over 250,000 people. In his work, he explores the intersection of security, technology, and social issues. Schneier is a fellow at the Berkman Klein Center for Internet & Society at Harvard University and a Lecturer in Public Policy at the Harvard Kennedy School. He is a member of several boards and advisory committees. He is the Chief of Security Architecture at Inrupt, Inc. and works with AISLE as an advisor.