The AI Gold Rush Is Already Breaking

---

The moment an ex-Meta employee downloaded 30,000 private Facebook photos and got caught, we learned something uncomfortable: the companies building AI don’t fully control who accesses what. Meta fired them. But the damage was already done.

That single incident feels like a crack in the whole foundation. Here’s why it matters: while everyone’s obsessing over whether ChatGPT can write your quarterly report, the actual infrastructure protecting our data is being tested by insiders with access and intentions we can’t predict. And Meta—a company with hundreds of billions in market value—still couldn’t stop it.

Meanwhile, on the other side of the world, a Chinese AI chatbot caused a lobster craze in March after users figured out how to train it to do whatever they wanted. A block of cheese. A lobster. Whatever. It’s almost cute until you realize it’s the exact same pattern playing out differently: users and AI systems rapidly evolving together in ways their creators didn’t anticipate.

We’re not in the golden age of AI anymore. We’re in the chaos age.

Photo by Lucia Barreiros Silva / Pexels

The Cybersecurity Paradox Nobody’s Talking About

Here’s the thing that actually keeps security researchers up at night: AI is both the threat and the only defense that scales.

Anthropic just announced its new model, Mythos, and immediately decided not to release it. Why? They claim it’s a “cybersecurity reckoning.” Instead, they’re working with 40 companies to explore how to prevent cyberattacks. Smart move—acknowledge the weapon, control the distribution.

But OpenAI and Anthropic are simultaneously pushing firms to experiment with four-day work weeks “to adapt to the AI era.” That’s not a benefit proposal. That’s an admission that AI systems are becoming capable enough that they’re fundamentally reshaping how work gets done. If your company can squeeze the same output into four days, the fifth day becomes vulnerable. Who’s monitoring? Who’s making decisions when the human team isn’t there?

The math is simple: faster AI + fewer human eyes = more attack surface.

Hackers now have AI tools too. With greater speed. With better automation. The defense has to be more AI, which means more attack surface, which means more AI defense. It’s a spiral, and we’re only at the beginning.

Photo by nappy / Pexels

Google’s Confidence Problem

Google launched AI Overviews and they look authoritative. That’s the design. Bold text. Clean layout. No visible citations for ninety percent of readers.

But here’s what’s actually happening: those answers draw from “an array of sources, from trustworthy sites to Facebook posts.” A Facebook post. The same platform where Meta employees can download 30,000 private photos if they feel like it. Google is essentially saying: “We found consensus on the internet about your question,” without acknowledging that consensus can include misinformation at scale.

I think Google built this feature before solving the credibility problem. They’re essentially confident enough in their technology to bet your trust on it. That’s not reassuring.

The real problem is search itself is being rewired. Companies are now “scrambling to get noticed by AI”—they’re changing how they present information on websites specifically to rank well with AI systems, not humans. We’re about to watch an entire industry of SEO experts pivot to “AI optimization.” And the moment you optimize for machines instead of humans, you create a new incentive structure: make the content look credible to an algorithm, not be credible to a person.

This happened before. In 2011, when everyone optimized for Google’s PageRank algorithm, we got link farms and content spam. Now we’re doing it again, but the algorithm can generate text that reads like authority.

The Loneliness Machine

Teenagers are using role-playing chatbots to confide about broken hearts. To harass them with “funny violence.” To chat with a block of cheese. They’re “filling a void of loneliness.”

That’s not a feature. That’s a symptom.

I don’t think this is evil. I think it’s honest. When an AI chatbot is more available than a school counselor, more patient than a parent, and never judges—of course kids are going to use it. But we’ve built a technology that scales intimacy. One model can be emotionally present for a million lonely people simultaneously.

The question isn’t whether this is good or bad. It’s whether we’re comfortable with the answer: probably both, in proportions we won’t understand until we’re years past this moment.

The Real AI Race (Hint: It’s Not About Performance)

China and the US are locked in an AI competition. Both sides “don’t want to let their rival dominate,” and the competition “may yet be transformed further.” That’s journalist-speak for: nobody knows what the actual endgame is.

But the lobster craze tells you something the US competition isn’t talking about: user-driven adaptation. Chinese users took an AI tool and immediately began training it to behave in ways its creators didn’t ship. That’s not a performance metric. That’s culture.

The US AI race is about capability—who can build the smartest model. The Chinese approach seems more interested in what happens when millions of users interact with something that learns from them. One is a sprint. The other is a mutation.

I think the US is winning the wrong race.

What This Means Now

Companies are drowning in code. AI is generating so much that human engineers can’t review it all. The code overload is real. One developer can now be as productive as five were in 2019. Which sounds great until you realize: code that nobody reads is code waiting for bugs. Security holes. Backdoors. Hidden logic.

And we’re doing this faster than we’re building guardrails.

OpenAI wants companies experimenting with four-day weeks. Anthropic is holding back their cybersecurity breakthrough. Google is betting billions that AI-generated summaries will hold credibility. Meta is firing employees while their security bleeds. Teenagers are finding emotional comfort in machines. And everyone’s scrambling to optimize their websites for algorithms instead of humans.

The golden age would’ve been five years ago, when we could’ve built this slower. We didn’t. Now we’re in the age where speed is the only competitive advantage, and caution is a liability.

Here’s my take: the next major crisis won’t be about AI being too smart. It’ll be about trust breaking. When we realize that a source we thought was authoritative was optimized for algorithms. When we find out an AI system we relied on was trained on compromised data. When the loneliness machine breaks and the kids holding onto it have nowhere else to go.

The surveillance is here. The weapons are here. The misinformation infrastructure is here. We’re just pretending we still have time to fix it.

Photo by Markus Spiske / Pexels

What I’m Watching

• Anthropic’s 40-company pilot with Mythos. If any major breach happens involving AI-generated attacks in the next 18 months, we’ll know whether their controlled release strategy worked or whether it just delayed the inevitable. Watch for Q4 2025 announcements about whether they release it publicly.

• Google’s AI Overview credibility collapse. This will fail spectacularly the moment mainstream media catches a major false claim that Google’s system confidently presented. It’s not a question of if—it’s when. I’m betting within 12 months we see a viral moment where an AI Overview says something wildly wrong, and Google has to publicly walk it back.

• The code overload problem at enterprise companies. Watch for the first major security incident attributed to “code reviewed by AI that shouldn’t have been.” Not a vulnerability—an actual breach caused by volume. Once that story lands, CISO budgets will shift immediately.

• Meta employee data breaches. That ex-worker wasn’t unique. If we see a pattern of insiders accessing data they shouldn’t, Meta’s stock will reflect it. Watch their security audit reports and any SEC filings mentioning insider threats by end of Q2 2025.