I wrote this for a friend who recently read Yuval Noah Harari’s Nexus and seemed nebulously concerned about it.
Okay, so this didn’t land for me when we talked about it the other day, and it still doesn’t. I get a sort of generalized concern tone, but I can’t pin down what you see as the problem. “Intelligence with no consciousness” is a buzzy phrase, but what does it actually mean and imply? Perhaps the book uses specific definitions of both of those words that makes it clear, but I don’t know what they are.
With the caveat that I haven’t read the book and probably won’t (it’s not the kind of nonfiction I’m drawn to), I want to draw an analogy…
Mesopotamia, ~3400 BC
Society is beginning to use a new technology called “writing”.
The way it used to be, before writing, was in order to know something, you had to go to a person and have a conversation with them. Conversations are human and dynamic - they contain so much information. You can interrupt; ask questions; rewind to give missing context; skip over things already known. You can look the other person in the eye and see if they’re lying; get them drunk and see if they’ll tell you more or different.
None of this is possible in writing. The tablet is just… there, immutable. It contains the information it contains, and nothing else; if you wanted some other information, or for it to be rephrased in a different way, tough luck. And how do you tell if a tablet is lying to you?
The tablet has no soul. You cannot know its heart.
What will it do for our world when this is how we exchange information? “Go look in the tablet room”, they’ll say, and I’ll be left to sort through piles of ceramics written by people I don’t know, each frozen in their own time, not knowing about this or that war, edict, flood that came after it and how it changed the world. Our systems of governance and rule, our understanding of our own history and future, will be mediated by silent clay, written by authors and motivations unknown.
And one more concern. This process of writing, with its full sentences and structured points - how will it affect our thinking? Will daily use of writing teach us to start our thoughts, and march onward to their resolution without noticing how we feel about them in our bodies or hearing the voices of our loved ones speaking through our hearts?
So yeah, I’m not sure about this writing thing. It seems like it’s becoming pervasive, but what is it going to do for our world?
Obviously I was having too much fun with that 😄
And to be clear – this isn’t me saying “luddite!”. I am far from an AI booster - I find it useful, and I’m still exploring it myself, but I see it as a tool.
The technology is still nascent; we’ve come a long way in a short time in understanding how the models are encoding their information, and that research will continue. (See Golden Gate Claude.)
You can fine-tune models now by saying “be more like this”; and at some point we won’t have to: if AI is indeed broadly useful, the march of commoditization will give us user-controlled, personal ones in our pockets within 5-10 years. It’s cliche to say this, but our phones have a thousand times more computing power than what got us to the moon.
I guess personally I don’t see AI / the algorithm / whatever else as the problem. The problem is capitalism; companies are gonna use whatever tools they can to accrue power and money. If there’s a guy rummaging through your toolbox being irresponsible with them, you can find ways to lock away each of the tools… or you can just deal with the guy, y’know?