AI Second Order Effects: Security Matters Even More
The increasing value of security in a world of AI
Security is not the absence of danger, but the presence of better coping mechanisms.
– Amitai Etzioni
Have you ever seen the movie Sneakers? It's one of the most accurate computer heist movies made. There's a great scene between two of the main characters who are positing how one could effectively change the world and the implications of that future. Watch the two main characters go back and forth in the clip below:
Cosmo: Posit: People think a bank might be financially shaky.
Martin Bishop: Consequence: People start to withdraw their money.
Cosmo: Result: Pretty soon it is financially shaky.
Martin Bishop: Conclusion: You can make banks fail.
This is a great introduction into thinking through and understanding second order effects. Second order effects are the consequences of the consequences. They are what transpire after the initial impact from the event has occurred. If we think about where AI is and where the world is going, most individuals are going to completely botch the second order effects.
For example, if you think about the power and automation available from the current power of AI, a proliferation of content is the logical conclusion. The cost to produce content goes down, so naturally the amount of content will increase. This likely will not be a linear relationship. Let's step through first order thinking and then second order thinking.
Posit: AI will reduce the cost of content creation.
Consequence: The amount of content created daily increases dramatically.
Result: People will be overwhelmed by available content.
Conclusion: Services that filter or recommend content become more powerful.
While Cosmo and Bishop might use second order thinking to reach their conclusions in the clip, it isn't as explicit. I think a better way to explicitly think about second order effects is by using two rounds of understanding as follows:
Posit: AI will reduce the cost of content creation.
Consequence: The amount of content created daily increases dramatically.
Result: People will be overwhelmed by available content.
Consequence: Content creators go to more extreme methods to stand out from the crowd.
Result: Trust in content begins to erode.
Conclusion: We need more secure methods to enable safe AI automation
A different outcome results from thinking a bit further. Thinking about second order effects enables you to get a clearer idea of what might happen and to figure out if the decision you make will really get the outcome you desire. As we enter a world of pervasive AI most people are going to have poor intuition for what will happen. It helps to think more deeply and less reactionary about events that are occurring and those that might occur.
One of the areas to think deeply about is security. This isn't about AGI or Skynet taking over. This isn't doom and gloom of AI. This is a serious look at the things we need to build to make AI ubiquitous and useful. You wouldn't use a bank if they couldn't hold on to your money or if there was rampant fraud. You wouldn't eat at a restaurant that continuously made you sick. Nor would drive places where bridges repeatedly collapse. Likewise, people aren't going to use systems that AI would render unusable. Let's explore some of the areas where security needs to be improved to increase AI uptake.
Bad Content
About five years ago Jordan Peele teamed up with Buzzfeed to create a public service announcement about deepfakes. He did this by creating a video that impersonated President Obama. While something was clearly off in the video, a distracted watcher could be fooled into thinking it was an actual recording of the former president. The warning however was clear - this technology is coming faster than you think and you are not prepared for it.
People are now waking up to the fact that Peele's warning is coming to pass. Unfortunately, people usually won't react to a warning until it's visceral, which tends to be too late. The best demonstration of potential damage for AI generated content was documented recently by Reality Defender who wrote an article about how within 36 hours the following happened:
YouTube celebrity Mr. Beast, actor Tom Hanks, and journalist Gayle King were victims of different deepfake-driven scams on social media.
[...]
Each celebrity victim responded in the same manner: posting an Instagram photo with a screen capture of the video in question and a warning message on top, telling their fans and viewers that what they saw was not them and they were not peddling a product or service that seemed out of the ordinary for them.This is currently the only way to respond to any sort of deepfake attack or abuse: retroactively, to your audience, and well after damage was already done. Seeing as the platform where these videos were posted, boosted (via paid advertising), and stayed up for hours, it’s clear little to no human moderation (let alone an AI-driven deepfake detection system) exists to prevent these specific abuses from happening in the first place.
Think about that for a moment. The only defense available for three major celebrities was to post to their followers from their account about the scam. What would have happened if 100 campaigns hit each celebrity? Would they be able to keep up with each of the scams? What's being exploited here is that people are relying on digital channels without protections for their content and without the ability to know if something has been faked.
This is more than just a content moderation problem. Let's assume for a moment that content moderation algorithms are 99.99% effective (they aren't) at catching bad content. That means that 1 in every 10,000 pieces of bad content makes it through. Let's say the amount of content created by bad actors is 1%. That means 1 bad piece of content gets through the filters for every 1,000,000 pieces of content created. Then looking at how many different pieces of content are created each day as of 2022:
Figure 1. Various daily rates of uploaded content in 2022.
Let's pick Facebook. Using our math above there's 2,450,000,000/1,000,000 = 2,450 bad pieces of content making it through content filters every day. Somewhat manageable since you very likely will be exposed to few if any of them. Now what happens when generative AI spikes the volume of content created by 1000x? That means there's 2,450,000 bad pieces of content being exposed on the network. Every. Single. Day. You'll definitely be exposed to a few of those and it could make the service borderline unusable. Remember, that's only one channel. We didn’t even go through the other ones on the list. So it's not hard to envision how scams can propagate across the zeitgeist without some added security. This leads to a serious turkey problem for existing platforms.
Turkey Problems
The turkey problem was conceived by Bertrand Russel and is as follows. A turkey thinks everything is fine for 3 years of continuous eating until the day before Thanksgiving when everything goes badly for it. The turkey is most confident that life will continue as normal up to the day when its life ends. What the turkey believed was business as usual caused it to be destroyed by the unexpected and unforeseen. Note that what happened was routine for the farmer.
Why are turkey problems relevant here? Those not thinking about the second order effects of AI are in for a nasty surprise. Platforms may be gushing at the acceleration of content they wield but they might not realize that they could become unusable without proper measures. A deluge of content will surely overwhelm any system. What happens when the methods of verification fail? Platforms become susceptible to poisoned information.
Taking advantage of the existing ways people interact in the world and where they place their trust is nothing new. Take a look at Amazon product reviews. Millions of people rely on product reviews to make their purchases. However, there's many ways these reviews can be manipulated outside the Amazon system. In fact, manipulation has become so pervasive (due to the amount of money on the line) people have developed their own procedures for researching projects. A friend of mine has developed a process for sniffing out oddities in products he is looking to purchase. First, he looks at how many reviews a product has and the timing of those reviews. If the only product reviews are from around the time when the product launched, they were likely bought. Next, he looks at all of the bad reviews. If there’s no bad reviews, the reviewers were likely paid off since no product is perfect. These avenues of exploiting reviews to get better rankings arose because people conditioned themselves to trust online reviews. For some that trust has eroded, for others they are none the wiser.
Trust
There is a bit of a race to implement security measures on the outputs of generative AI so that other systems can help their users know which content to trust. You may have seen that some generative AI companies are releasing AI watermarks. You may also have seen a Wired article talking about how AI watermarks don't work. This comes from multiple researchers who "broke them all". This is not surprising as watermarks only prove that good actors of generative AI created something. It does not tell you if bad actors evaded the methods. Watermarks create false trust by deploying security theater. I wrote 10 months ago that we needed a different framework for verifying content. I showed why watermarks don't work and why we need more of a proof-of-human framework.
I always come back to trust because it is central to how society works. Do you trust your neighbors to not harm you? Do you trust the government to enforce their laws? Do you trust that a product works as expected? Do you trust that this AI model will perform as advertised? Trust underpins our ability to navigate the world and build a better one. Part of what helps build trust are security systems that work to ensure that good actors stay good. As AI systems proliferate, we need to figure out how to secure our systems from the impact of AI in order to trust what we have previously relied on.
Increasing Value of Security
Bringing this all together, there are a lot of second order effects to think through with AI. What I can tell you is that AI has already improved your life, and it is not going to go away any time soon. The toothpaste is already out of the tube and there's no chance of putting it back in. With that realization, we need to create systems to help maintain our existing services and infrastructure as the use of AI grows and expands. That means developing security systems that can help us protect against bad actors.
Society as a whole is going to be slow to adopt AI technology if they feel they can't trust it. I'm already seeing slower adoption at the enterprise level. Organizations are hesitant to deploy systems at scale until they can figure out how to put guardrails around all the quirks. Security solutions for AI need to revolve around answer the following three questions:
How do we handle verification?
What happens when a massive increase in scale suddenly occurs?
What systems are ill-equipped to handle a high degree of automation?
Answering these questions should help you understand where an AI system might impact what you deal with in unexpected ways. It might also lead to questions such as:
How unusable would a product be if it were overwhelmed with bad content?
Where should product friction be applied to prevent bad users of AI and how does that modify the product?
How can you automate with AI but keep only positive gains?
Again, security around AI is more about finding ways to make its usage acceptable and agreeable. AI will be here whether we like it or not, so it is best to take part in shaping the future rather than sitting idly by. This means developing security systems that enable trust in both the AI that is deployed and in the systems that are susceptible to being impacted by AI. This will be messy and it won't be swift. We surely live in interesting times.