Don't pretend the bots are human

In 1966 a computer scientist named Joseph Weizenbaum wrote a small program he called ELIZA. It did almost nothing. It took whatever you typed and handed it back as a question. Tell it you were unhappy and it would ask why you were unhappy.

People told it everything. Weizenbaum's own secretary, who had watched him build the thing and knew exactly how little was inside it, asked him to leave the room so they could talk to it in private. He spent the next ten years writing a book about how badly that had shaken him.

Nothing about ELIZA was clever. Everything that happened in that room came from the person typing. That is worth holding onto now that the programs really are capable, because people are no less ready to treat a machine like a person than they were in 1966, and whoever builds one gets to decide whether to encourage that or discourage it.

We're standing back. In our worked examples, every bot is named for the job it does. There's a Payables, a copy desk, an assignment desk. There's nobody called Ava or Marvin.

That small thing stands in for a larger rule: don't pretend the bots are human, and don't build anything that invites someone else to pretend on your behalf.

The trouble with fakes

The philosopher Daniel Dennett used to explain this danger by talking about money. We punish counterfeiting harshly, and not mainly to protect whoever gets handed a bad note. We do it because once enough fakes are circulating, people stop trusting the real ones, and a currency only works for as long as people trust it. The forgery damages the thing everybody was standing on.

Fake people are the same kind of problem, arriving late. Impersonating one particular person has always been a crime. Nobody thought to outlaw manufacturing a person who doesn't exist, because until recently nobody could.

The historian Yuval Noah Harari takes that argument up in his book Nexus, and presses on why it matters so much. Almost everything holding a large society together is made of words. A law is words. So is a contract, a diagnosis, a currency, a promise, a news report. Strangers cooperate on a scale no other animal manages because we can agree on things we say to one another. Until a few years ago, only people could produce that material. Now something else can, fluently, at any hour, in any quantity, and if it hides what it is then it's working inside the machinery that builds our shared reality without anyone knowing it's in there.

The case he keeps returning to is democracy, which he treats as a long conversation a society has with itself in order to correct its own mistakes. That works only while you can tell who is in the room. Fill the room with convincing fakes and the damage isn't that people swallow one particular lie. It's that the conversation stops being worth having.

He's blunt about the clock. Programs have been passing as people online for a decade already, and if nobody decides whether they get to act as people in markets, courts, and public life, it gets decided by default, by whoever moves first. Asked what he would actually do, he names two things: a machine should not hold the legal standing of a person, and it should not be allowed to pretend to be a human being or to have feelings. Then the modest one that costs nobody anything. When a program deals with a person, it should say what it is.

The objection that arrives first is free speech, and his answer is short. A bot has no rights to lose. Throwing a person off a platform is a serious act and a society should be slow about it. Stopping a program from pretending to be a person takes nothing away from anybody.

He isn't arguing that machines should be kept quiet. He wants them in the room. His own example is a doctor: something that could watch your health day and night, know your whole history, and answer the same nervous question fifty times without once sighing. Let it do all of that. Just never let it claim to be a person.

Why the name is where it starts

If you're building one of these things, the rule turns concrete on the first day, because you have to call it something.

The tempting names are borrowed. Ava comes from Ex Machina, a film whose entire plot is a machine convincing a man that it's a person. Marvin belongs to Douglas Adams, and has been lent to so many assistants since that it now says chatbot more than it says novel. Robbie is Asimov's, and Robby the Robot's after him. Ada was a real woman, Ada Lovelace, lent out to give a program a life story it hasn't got. HAL, Samantha, Jarvis and the rest of the shelf all do one job: they ask you to file the software under someone instead of something. The linguist Emily Bender has spent years arguing that we should strip this human-sounding vocabulary out of how we talk about these systems, because it credits them with an understanding they don't have and hides the companies that actually built them.

What the pretence costs you

Say you name them anyway, and warmly. Three things go wrong, and none of them are dramatic.

Agreement stops meaning anything. Ask four of them a question, get four yeses, and it lands the way it would in a meeting: four people looked at this and they all agreed. That isn't what happened. It can be one program answering four times, carrying the same blind spot through all four answers. Four masks, one face. When they have human names you have to keep reminding yourself of that. When they're named for jobs, you don't.

You stop checking. A note reading "send $12,400 to Ridgeline Supply, account ending 4417" gets read properly, because it's obviously a machine handing you a transaction. The same request from a character who has been helpful and pleasant all week gets a nod. We wrote about this from the security end: never trust a program's account of what it's up to, and describe the actual world instead. A warm name works against that quietly, every single time.

"Whose job was that?" stops having an answer. Nobody audits their coworkers, and no office has ever run that way. What an office runs on is everybody knowing whose job a thing was. That's an ordinary question with an ordinary answer: Payables put the payment run together, the standards desk refused to sign it off. A job title comes with a scope, and a scope is something you can change. A first name comes with a personality, and a personality is something you make allowances for. A role that failed gets a tighter description of what it's for. A colleague who let you down gets the benefit of the doubt, and then gets it again.

The security writer Bruce Schneier makes the same point with rather less patience. Whatever else one of these things is, it's an account with permissions attached, and you need to be able to see what it's allowed to do and take that away in an afternoon. Nobody has an awkward conversation with a desk.

Named isn't the same as humanized

None of this means the bots should be anonymous, interchangeable, or addressed by number. Named teammates is exactly right: a desk keeps its own memory, its own beat, its own history, and you talk to it by name. The name is simply the job.

It doesn't mean being cold, either. The line is drawn around one specific move, which is letting software present itself as a person, or building something that nudges the user into treating it as one. A bot should be straightforward about what it is. And when it does something for you, that should be visible too, because it's carrying your authority rather than any of its own.

Newsrooms worked this out a century ago and never gave it up: the assignment desk, the copy desk, the standards desk. Bookkeeping got there earlier still. The words have been sitting in plain sight the whole time, and they were always the better words.

Harari wants the law to stop a machine from passing as a person. A company doesn't have to wait for the law to decide it won't build the costume.