A provocative and disturbing essay by Josh Askonas in The New Atlantis:

Many of the systems we now use online have their structural origins in the world of role-playing games. Video games of all sorts borrow concepts from them. “Gamified” apps for fitness, language learning, finance, and much else award users with points, badges, and levels. Facebook feeds sort content based on “likes” awarded by users. We build online identities with the same diligence and style with which Dungeons & Dragons players build their characters, checking boxes and filling in attribute fields. A Tinder profile that reads “White nonbinary (they/her) polyamorous thirtysomething dog mom. Web-developer, cross-fit maniac, love Game of Thrones” sounds more like the description of a role-playing character than how anyone would actually describe herself in real life. 

Justin E. H. Smith makes a similar point in describing some recent complaints about the behavior of the comics writer Warren Ellis: 

A website was set up for his proclaimed victims to share their testimonials. On this site, the author’s grooming behaviour is described as: “rel[ying] on subtle techniques that leverage ‘compulsion loops’, which are well-established in scientific literature and video gaming, and are commonly utilised by modern businesses to achieve addiction, AKA ‘user retention’. Examples include daily quests in games, getting a higher reward (more ‘XP’, etc.) for the first game of a day, more ‘karma’ for the first post of a day on a message board, etc. The main driver is a regular daily dopamine boost sustained over time.”

At issue here is the moral conduct of a person who in another era would have been accused of lechery, of being manipulative, of playing the cad. Here, the accusation against the comic-book author, however, eschews inherited moral categories, and blames him, effectively, for instantiating the same features we also know from our use of social media. The author has had programmed into him, it would seem, the same addictive hooks for which we rightly criticise Facebook. He now stands accused of “user retention”. 

These points seem to converge with one that Jaron Lanier made in You Are Not a Gadget: 

But the Turing test cuts both ways. You can't tell if a machine has gotten smarter or if you've just lowered your own standards of intelligence to such a degree that the machine seems smart. If you can have a conversation with a simulated person presented by an AI program, can you tell how far you've let your sense of personhood degrade in order to make the illusion work for you?

People degrade themselves in order to make machines seem smart all the time. Before the crash, bankers believed in supposedly intelligent algorithms that could calculate credit risks before making bad loans. We ask teachers to teach to standardized tests so a student will look good to an algorithm. We have repeatedly demonstrated our species' bottomless ability to lower our standards to make information technology look good. Every instance of intelligence in a machine is ambiguous.

The same ambiguity that motivated dubious academic AI projects in the past has been repackaged as mass culture today. Did that search engine really know what you want, or are you playing along, lowering your standards to make it seem clever? While it's to be expected that the human perspective will be changed by encounters with profound new technologies, the exercise of treating machine intelligence as real requires people to reduce their mooring to reality. 

If machines, and the apps the machines run, cannot capture the fullness of our humanity, that poses a problem for the technocrats. They can try to make the machines better; but, as they do so, they can also use social engineering to persuade us to flatten and narrow our humanity to fit what the machines are capable of. (The quotes above suggest how easily many people can be convinced to redescribe themselves and their experiences in this flattened way.) Eventually the two projects will meet in the middle, and the story of humanity will effectively be over — except for the tiny handful who manage to sustain a dissenting independence. 

What Askonas adds to this distressing prophecy is this: As our remaking proceeds, we’ll think we’re having fun.