What Are We Even Doing?

or: How I Learned to Stop Worrying and Love Artificial Intelligence
Shavon Nand
Screenshot of OpenAI's ChatGPT website on Desktop.
Screenshot of OpenAI's ChatGPT website on Desktop.

Whilst browsing YouTube, I came across AI In Context's 'We're Not Ready for Superintelligence'. This video, to me, was conspiratorial and made a lot of assumptions that just felt wrong. I mean just look at the description of the video:

AI 2027 depicts a possible future where artificial intelligence radically transforms the world in just a few intense years. It's based on detailed expert forecasts — but how much of it will actually happen? Are we really racing towards a choice between a planet controlled by the elite, or one where humans have lost control entirely?

My takeaway? Loss of control, racing scenarios, and concentration of power are all concerningly plausible, and among the most pressing issues the world faces. [1]

So after watching that conspiratorial apocalyptic mess, I decided to do some digging, and this is what I found.

Aric Floyd, the host of the channel is an actor. The channel is not a personal project of some AI enthusiast. It was created by the nonprofit '80,000 Hours'.

You might recognize 80,000 Hours from a YouTube sponsor read. Here's one I found from Extra Credits:

80,000 Hours is a nonprofit organization dedicated to helping people find careers that are not only fulfilling but also have a significant positive impact on the world. Their work is based on over 10 years of research, conducted in collaboration with Oxford University, aimed at identifying high-impact career paths that can tackle some of humanity's biggest challenges.[2]

So you might be wondering as to why a career training website would be producing an apocalyptic AI video about how humanity will need to fight against an all-powerful AI. The answer is that 80,000 Hours aligns itself with the philosophy of effective altruism.

Effective Altruism is a form of utilitarianism, itself a branch of consequentialism – the plausible idea that you judge choices by their consequences. Utilitarians go further by saying you judge acts only by their consequences, and that the outcomes can be graded on a single scale of utility, or happiness. You judge acts by comparing their effects on the global sum of happiness, and the morally right course is the one you expect to achieve the maximum effect. Crucially, other moral considerations – obligations to be honest, to be just, to be loyal, to respect property rights and many more – count only to the extent that they bear on the happiness calculation. [3]

I had first heard of effective altruism in 2022, as the philosophy of Sam Bankman-Fried. When FTX collapsed and Sam Bankman-Fried was sentenced on seven counts of wire fraud and conspiracy to launder money[4], I thought that would be the end for this movement. I was wrong.

The philosophy of effective altruism makes the case that one way that idealistic young people with fancy degrees could improve the world is not by working at nonprofits, but by earning as much money as possible through finance, and then donating it all to vetted causes. [5]

80,000 Hours advocates for a specific kind of effective altruism called "longtermism"[6], the idea that "...because so many more humans and other intelligent beings could live in the future than live today, the most important thing for altruistic people to do in the present is to promote the welfare of those unborn beings, by ensuring that future comes to be by preventing existential risks — and that such a future is as good as possible."[7]

For various reasons, the [Effective Altruism] movement has turned its attention toward longtermism—a more radical form of its utilitarianism that weighs the value of each future potential life approximately the same as a living person's. Because any human extinction event, however unlikely, imposes infinite costs, longtermists can place enormous moral value on reducing whatever they view as existential risk. [8]

[Effective Altruists] initially focused mostly on issues like animal welfare and global poverty, but over time, worries about an AI-fueled apocalypse became a central focus. With funding from deep-pocketed donors like billionaire and Facebook co-founder Dustin Moskovitz, they built their own insular universe to study AI safety, including a web of nonprofits and research organizations, forecasting centers, conferences, and web forums.[9]

The most ardent advocates of effective altruism, or EA, believe researchers are only months or years away from building an AI superintelligence able to outsmart the world's collective efforts to control it. Through either its own volition or via terrorists seeking to develop deadly bioweapons, such an AI could wipe out humanity, they say. And some, including noted EA thinker Eliezer Yudkowsky, believe even a nuclear holocaust would be preferable to an unchecked AI future. [10]

Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that's what it takes to reduce the risk of large AI training runs. [11]

This to me does not sound like philosophy; this sounds like an AI doomsday cult.

[Effective Altruism]'s career advice center, 80,000 hours, lists "AI safety technical research" and "shaping future governance of AI" as the top two recommended careers for EAs to go into, and the billionaire EA class funds initiatives attempting to stop an AGI apocalypse. According to EAs, AGI is likely inevitable, and their goal is thus to make it beneficial to humanity: akin to creating a benevolent god rather than a devil. [12]

To borrow a term from Lee Vinsel, this AI apocalypse is another form of criti-hype.[13] Basically, another Premature Freak-Out about Technological Enhancement.

Perhaps the two most striking examples of this criti-hype trend are Shoshana Zuboff's book, The Age of Surveillance Capitalism, and the film The Social Dilemma, which includes Zuboff and another criti-hyper Tristan Harris as talking heads. [13:1]

Speaking of criti-hyper Tristan Harris[14], I came across this interview he did with Jon Stewart on The Daily Show.

Tristan Harris, after peddling the idea that social media companies are puppet masters who have users on strings[13:2], has now moved onto his next target of AI, falling into the same industry propaganda that he did last time.

This is not to say that there are not any real reasons to be critical of AI. Back in 2020, Robert Julian-Borchak Williams was "wrongfully arrested based on a flawed match from a facial recognition algorithm". [15] That same year, "two professors and a graduate student had developed a facial-recognition program that could predict whether someone would be a criminal".[16]

Advances in data science and machine learning have led to numerous algorithms in recent years that purport to predict crimes or criminality. But if the data used to build those algorithms is biased, the algorithms' predictions will also be biased. Because of the racially skewed nature of policing in the US, the letter argues, any predictive algorithm modeling criminality will only reproduce the biases already reflected in the criminal justice system. [16:1]

This is not even to mention the horrifying and harmful firehose of AI slop that is now being generated en masse. [17][18][19][20][21][22][23][24]

I think that we need to treat AI as normal technology.

The statement "AI is normal technology" is three things: a description of current AI, a prediction about the foreseeable future of AI, and a prescription about how we should treat it. We view AI as a tool that we can and should remain in control of, and we argue that this goal does not require drastic policy interventions or technical breakthroughs. We do not think that viewing AI as a humanlike intelligence is currently accurate or useful for understanding its societal impacts, nor is it likely to be in our vision of the future. [25]

That some significant portion of OpenAI's consumer base is using ChatGPT not so much for the expected "normal" uses like search, or productivity improvements, or creating slop birthday-party invitations, but for friendship, companionship, romance, and therapy certainly feels abnormal. (And apocalyptic.) But this is 2025, and intense, emotional, addiction-resembling attachment to software-bound experience has been a core paradigm of the technology industry for almost two decades, not to mention a multibillion-dollar business model. Certainly, you will not find me arguing that "psychosis-inducing sycophantic girlfriend robot subscription product" is "normal" in the sense of "acceptable" or "appropriate to a mature and dignified civilization." But speaking descriptively, as a matter of long precedent, what could be more normal, in Silicon Valley, than people weeping on a message board because a UX change has transformed the valence of their addiction? [26]

Further Reading

References

  1. We're Not Ready for Superintelligence ↩︎

  2. The Dev's Creed: Being Wrong is Essential ↩︎

  3. Effective Altruism Is as Bankrupt as Sam Bankman-Fried's FTX ↩︎

  4. FTX's Sam Bankman-Fried believed in 'effective altruism'. What is it? ↩︎

  5. Behind the scenes of how 'effective altruism' went to die in Sam Bankman-Fried's Bahamas penthouse ↩︎

  6. Longtermism: a call to protect future generations ↩︎

  7. How effective altruism let Sam Bankman-Fried happen ↩︎

  8. The Authoritarian Side of Effective Altruism Comes for AI ↩︎

  9. The AI industry turns against its favorite philosophy ↩︎

  10. When Silicon Valley's AI warriors came to Washington ↩︎

  11. Pausing AI Developments Isn't Enough. We Need to Shut it All Down ↩︎

  12. Effective Altruism Is Pushing a Dangerous Brand of 'AI Safety' ↩︎

  13. You're Doing It Wrong: Notes on Criticism and Technology Hype ↩︎ ↩︎ ↩︎

  14. Tristan Harris on the need to change the incentives of social media companies ↩︎

  15. Wrongfully Accused by an Algorithm ↩︎

  16. An Algorithm That 'Predicts' Criminality Based on a Face Sparks a Furor ↩︎ ↩︎

  17. Microsoft ignored safety problems with AI image generator, engineer complains ↩︎

  18. This is how AI image generators see the world ↩︎

  19. 4chan Uses Bing to Flood the Internet With Racist Images ↩︎

  20. Where Facebook's AI Slop Comes From ↩︎

  21. AI-generated Nazi memes thrive on Musk's X despite claims of crackdown ↩︎

  22. 'Brainrot' AI on Instagram Is Monetizing the Most Fucked Up Things You Can Imagine (and Lots You Can't) ↩︎

  23. Racist AI-generated videos are the newest slop garnering millions of views on TikTok ↩︎

  24. 'Just the start': X's new AI software driving online racist abuse, experts warn ↩︎

  25. AI as Normal Technology ↩︎

  26. A.I. as normal technology (derogatory) ↩︎