Chapter 13: Tom and Thomas

Personal PhilosopherTM was a runaway success. It became the app to have in just a couple of weeks. It combined the depth and reach of an online encyclopedia with the ease of reference of a tool such as Wikipedia and the simplicity of a novel like Sophie’s World. On top of that, the application did retain a lot of M’s original therapeutic firepower. Moreover, while the interface was much the same – a pretty woman for men, and a pretty man for women – the fact that the pretty face was no longer supposed to represent that of a therapist led to levels of ‘affectionateness’ which the developers of M had not dared to imagine before. A substantial number of users admitted that they were literally ‘in love’ with the new product.

For some reason – most probably because he thought he could not afford to do so as project team leader and marketing manager – Tom abstained from developing such relationship with Promise’s latest incarnation. However, he did encourage his new girlfriend (he had met Angie in the gym indeed – as predicted) to go all the way. She raved about the application. She also spent more and more precious private evening time using it.

He took her out for dinner one evening in an obvious attempt to try to learn more about her experience with ‘Thomas’, as she had baptized it – or ‘him’. He had consciously refrained from talking much about it before, as he did not want to influence her use of it – or ‘Thomas’ as she called it.

He started by praising her: ‘It’s amazing what you’ve learned from Thomas.’

‘Yeah. It’s quite incredible, isn’t it? I never thought I’d like it so much.’

‘Well… It’s good for me. People never believed it would work, and those who did, could not imagine it would become so popular. What’s the most fascinating thing about it? Sorry. About him. Isn’t it funny I still like to think of Promise as a woman actually?’

‘Thomas can answer all of my questions really. I mean… He actually can’t – philosophy never can – but he clarifies stuff in a way that makes me stop wondering about things and just accept life as it is. He’s really as you thought he, or it, or whatever, would be like: a guru.’

‘I don’t want to sound jealous but didn’t you say something similar about me like a few months ago?’

‘Oh come on, Tom. You know I named Thomas after you – because you’re so similar indeed.’

‘Am I? You say that, but in what ways are Thomas and I similar really?’

‘The same enthusiasm. The same positive outlook on life. And then, of course, he knows a lot more – or much more detail – but you’re rather omniscient as well I think.’

That did not surprise Tom. He and his team had ensured a positive outlook indeed. While Personal PhilosopherTM could brief you in very much detail about philosophers such as Nietzsche indeed, its orientation was clearly much more pragmatic and constructive: they wanted the application to help people feel better about themselves, not worse. In that sense, the application had retained M’s therapeutic qualities even if it did not share M’s original behavioralist framework.

‘Could you love Thomas?’

Angie laughed.

‘So you are jealous, aren’t you? Of course not, silly! You’re human. Thomas is just – well… He’s a computer.’

‘Can’t one fall in love with a computer?’

Angie didn’t need to think about that. She was smart. On top of that, she had learnt a lot from Thomas also.

‘Of course not. Love is a human experience. Thomas is not human. For starters, love is linked to sex and our physical being in life. But not only to that. It’s also linked to our uniquely human experience of being mortal and feeling alone in this universe. It’s our connection to the Mystery in life. It’s part of our being as a social animal. In short, it’s something existential – so it’s linked to our very existence as a human being. And Thomas is not a human being and so he cannot experience that. Love is also something mutual, and so there’s no way one could fall in love with him – or ‘it’ I would say in this context – because he can’t fall in love with me.’

Tom and his team had scripted answers like this. It was true he and Thomas shared similar views.

‘What if he could?’

‘Sorry?’

‘What if Thomas could fall in love with you? I mean… We’re so close to re-creating the human mind with this thing. I agree it’s got no body and so it can’t experience sex or so – but I guess we might get close to letting it think it can.’

‘Are you serious?’

‘Yes and no. It’s a possibility – albeit a very remote one. And then the question is, of course, whether or not we would really want that to happen.’

‘What?’

‘The creation of a love machine. Let’s suppose we can create the perfect android. In fact, there are examples already. The University of Osaka has created so-called gynoids: robots with a body that perfectly resembles that of a beautiful woman. For some reason, they don’t do the same kind of research with male forms. In any case… Let’s suppose we could give Thomas the perfect male body. I know it sounds perverse but let’s suppose we could make it feel like a real body, that it would be warm and that it would breathe and all that, and that its synthetic skin would feel like mine.’

‘You must be joking.’

‘That’s the title of a biography of Richard Feynman.’

‘Sorry?’

‘Sorry. That’s not relevant. Just think about my question, Angie. Would you be able to make love with an android? I mean, just think it would smell better than me, never be tired, and that it would be better than any sex toy you’ve ever had.’

‘I never had sex toys. I don’t need them.’

‘OK… Sorry. But you know what I mean.’

‘It would be like… Like masturbation.’

‘Perhaps you don’t use sex toys, but you masturbate, Angie. I mean… Sorry. You do it with me. Could you imagine doing it with an android? With an android who would have Thomas’s face and intelligence and… Well… Thomas’ human warmth?’

‘Thomas’ warmth isn’t human.’

‘OK. Just Thomas’ warmth then. Let’s suppose we can give him skin and a beating heart and all that.’

‘You’re not working on a project like that, are you?’

‘Of course I am not. I just want to know.’

‘Because you’re jealous? You think I spend too much time with Thomas?’

‘No. Not because I am jealous or because I think you spend too much time with Thomas. I want to know because I am really intrigued by the question. Professionally and personally.’

‘What do you mean by personally?’

‘Well… Just what I say: personally. It has nothing to do with you. I am just curious and want to think through all the possibilities. You know I am fascinated by M. I wonder where it will be let’s say thirty years from now. I wonder whether we’ll have androids being used as a masturbation toy.’

Angie thought about it.

‘Well… Frankly… I think… Yes. It would not be all that different from the kind of sex toys some people are already using now, would it? I mean… If you’re deprived from real sex, what you’re describing would not be a bad alternative, would it?’

Tom laughed. ‘No. Not at all.’

After a short pause, Angie resumed the conversation.

‘But such androids would smell differently. We’d know it. And women would always prefer a real man.’

‘Why?’

‘Because… Because you’re human. I told you. Love is something human. Love is the ultimate goal in our lives because it’s so human. Fragile and imperfect and difficult… But incredibly worthwhile at the same time too. Something worth striving for. Something worth fighting for. It intimately connects us: us as human beings in our human condition.’

‘What’s our human condition?’

‘Well… What I said before. Mortality. Our relationship with the sacred – or all of the mystery if you want. I mean, we’re into existentialism here. You can ask Thomas all about it.’

She laughed. Tom didn’t.

‘You mean our relationship with our own limits? That’s what makes us human? That’s what makes us want to be loved by someone else?’

‘I wouldn’t call it that way, but I guess that’s another way of putting it. Yes.’

‘OK… Thanks for loving me.’

Angie laughed. ‘You’re funny. Can we talk about something else now?’

‘Of course. What do you want to talk about?’

‘Something I can’t talk about with Thomas.’

‘So what is that?’

‘Well… Let’s try gossip… Or local politics… Or both. And Thomas isn’t much into fitness either.’

‘Well… We could think of a new product perhaps. I am sure we could re-program M yet again and include local politics and fitness as discussion topics as well…’

‘Come on Tom. You know what I mean.’

‘Sure, Angie. I love you.’

‘I love you too, Tom. I really do. I should spend more time with you. I will. Don’t worry about Thomas.’

‘I don’t. Or actually I do. But then in a good way. Thomas is a good product. It was a good investment.’

Chapter 12: From therapist to guru?

As Tom moved from project to project within the larger Promise enterprise, he gradually grew less wary of the Big Brother aspects of it all. In fact, it was not all that different from how Google claimed to work: ‘Do the right thing: don’t be evil. Honesty and integrity in all we do. Our business practices are beyond reproach. We make money by doing good things.’ Promise’s management had also embraced the politics of co-optation and recuperation: it actively absorbed skeptical or critical elements into its leadership as part of a proactive strategy to avoid public backlash. In fact, Tom often could not help thinking he had also been co-opted as part of that strategy. However, that consideration did not reduce his enthusiasm. On the contrary: as the Mindful MindTM applications became increasingly popular, Tom managed to convince the Board to start investing resources in an area which M’s creators had tried to avoid so far. Tom called it the sense-making business, but the Board quickly settled on the more business-like name of Personal Philosopher and, after some wrangling with the Patent and Trademark Office, the Promise team managed to obtain a trade mark registration for it and so it became the Personal PhilosopherTM project.

Tom had co-opted Paul in the project in a very early stage – as soon as he had the idea for it really. He had realized he would probably not be able to convince the Board on his own. Indeed, at first sight, the project did not seem to make sense. M had been built using a core behavioralist conceptual framework and its Mindful MindTM applications had perfected this approach in order to be able to address very specific issues, and very specific categories of people: employees, retirees, drug addicts,… Most of the individuals who had been involved in the early stages of the program were very skeptical of what Tom had in mind, which was very non-specific. Tom wanted to increase the degrees of freedom in the system drastically, and inject much more ambiguity into it. Some of the skeptics thought the experiment was rather innocent, and that it would only result in M behaving more like a chatterbot, instead of as a therapist. Others thought the lack of specificity in the objective function and rule base would result in the conversation spinning rapidly out of control and become nonsensical. In other words, they thought M would not be able to stand up to the Turing test for very long.

Paul was as skeptical but instinctively liked the project as a way to test M’s limits. In the end, it was more Tom’s enthusiasm than anything else which finally led to a project team being put together. The Board had made sure it also included some hard-core cynics. One of those cynics – a mathematical wizkid called Jon – had brought a couple of Nietzsche’s most famous titles – The Gay Science, Thus Spoke Zarathustra and Beyond Good and Evil – to the first formal meeting of the group and factually asked whether anyone of the people present had read these books. Two philosopher-members of the group raised their hands. Jon then took a note he had made and read a citation out of one these books: ‘From every point of view the erroneousness of the world in which we believe we live is the surest and firmest thing we can get our eyes on.’

He asked the philosophers where it came from and what it actually meant. They looked at each other and admitted they were not able to give the exact reference or context. However, one of them ventured to speak on it, only to be interrupted by the second one in a short discussion which obviously did not make sense to most around the table. Jon intervened and ended the discussion feeling vindicated: ‘So what are we trying to do here really? Even our distinguished philosopher friends here can’t agree on what madmen like Nietzsche actually wrote. I am not mincing my words. Nietzsche was a madman: he literally died from insanity. But so he’s a great philosopher it is said. And so you want us to program M so very normal people can talk about all of these weird views?’

Although Jon obviously took some liberty with the facts here, neither of the two philosophers dared to interrupt him.

Tom had come prepared however: ‘M also talks routinely about texts it has not read, and about authors about which it had little or no knowledge, except for some associations. In fact, that’s how M was programmed. When stuff is ambiguous – too ambiguous – we have fed M with intelligent summaries. It did not invent its personal philosophy: we programmed it. It can converse intelligently about topics of which it has no personal experience. As such, it’s very much like you and me, or even like the two distinguished professors of philosophy we have here: they have read a lot, different things than we, but – just like us, or M- they have not read all. It does not prevent them from articulating their own views of the world and their own place in it. It does not prevent them from helping others to formulate such views. I don’t see why we can’t move to the next level with M and develop some kind of meta-language which would enable her to understand that she – sorry, it – is also the product of learning, of being fed with assertions and facts which made her – sorry, I’ll use what I always used for her – what she is: a behavioral therapist. And so, yes, I feel we can let her evolve into more general things. She can become a philosopher too.’

Paul also usefully intervened. He felt he was in a better position to stop Jon, as they belonged to the same group within the larger program. He was rather blunt about it: ‘Jon, with all due respect, but I think this is not the place for such non-technical talk. This is a project meeting. Our very first one in fact. The questions you’re raising are the ones we have been fighting over with the Board. You know our answer to it. The deal is that – just as we have done with M – we would try to narrow our focus and delineate the area. This is a scoping exercise. Let’s focus on that. You have all received Tom’s presentation. If I am not mistaken, I did not see any reference to Nietzsche or nihilism or existentialism in it. But I am be mistaken. I would suggest we give him the floor now and limit our remarks to what he proposes in this regard. I’d suggest we’d be as constructive as possible in our remarks. Skepticism is warranted, but let’s stick to being critical of what we’re going to try to do, and not of what we’re not going to try to do.’

Tom had polished his presentation with Paul’s help. At the same time, he knew this was truly his presentation; he knew it did reflect his views on life and knowledge and everything philosophical in general. How could it be otherwise? He started by talking about the need to stay close to the concepts which had been key to the success of M and, in particular, the concept of learning.

‘Thanks, Paul. Let me start by saying that I feel we should take those questions which we ask ourselves, in school, or as adults, as a point of departure. It should be natural. We should encourage M to ask these questions herself. You know what I mean. She can be creative – even her creativity is programmed in a way. Most of these questions are triggered by what we learn in school, by the people who raise us – not only parents but, importantly, our peers. It’s nature and nurture, and we’re aware of that, and we actually have that desire to trace our questions back to that. What’s nature in us? What’s nurture? What made us who we are? This is the list of topics I am thinking of.’

He pulled up his first slide. It was titled ‘the philosophy of physics’, and it just listed lots of keywords with lots of Internet statistics which were supposed to measure human interest in it. He had some difficulty getting started, but became more confident as his audience did not seem to react negatively to what – at first – seemed a bit nonsensical.

First, the philosophy of science, or of physics in particular. We all vaguely know that, after a search of over 40 years, scientists finally confirmed the existence of the Higgs particle, a quantum excitation of the Higgs field, which gives mass to elementary particles. It is rather strange that there is relatively little public enthusiasm for this monumental discovery. It surely cannot be likened to the wave of popular culture which we associate with Einstein, and which started soon after the discovery already. Perhaps it’s because it was a European effort, and a team effort. There’s no discoverer associated with, and surely not the kind of absent-minded professor that Einstein was: ‘a cartoonist’s dream come true’, as Times put it. That being said, there’s an interest – as you can see from these statistics here. So it’s more than likely that an application which could make sense of it all in natural language would be a big hit. It could and should be supported by all of the popular technical and non-technical material that’s around. M can easily be programmed to selectively feed people with course material, designed to match their level of sophistication and their need, or not, for more detail. Speaking for myself, I sort of understand what the Schrodinger equation is all about, or even the concept of quantum tunneling, but what does it mean really for our understanding of the world? I also have some appreciation of the fact that reality is fundamentally different at the Planck scale – like the particularities of Bose-Einstein statistics are really weird at first sight – but then what does it mean? There are many other relevant philosophical questions. For example, what does the introduction of perturbation theory tell us – as philosophers thinking about how we perceive and explain the world I’d say? If we have to use approximation schemes to describe complex quantum systems in terms of simpler ones, what does that mean – I mean in philosophical terms, in our human understanding of the world? I mean… At the simplest level, M could just explain the different interpretations of Heisenberg’s uncertainty principle but, at a more advanced level, it could also engage its interlocutors in a truly philosophical discussion on freedom and determinism. I mean… Well… I am sure our colleagues from the Philosophy Department here would agree that epistemology or even ontology are still relevant today, aren’t they?’

While only one of the two philosophers had a very vague understanding of Bose-Einstein statistics, and while both of them did not like Tom’s casual style of talking about serious things, they nodded in agreement.

Second, the philosophy of mind.’ Tom paused. ‘Well. I won’t be academic here but let me just make a few remarks out of my own interest in Buddhist philosophy. I hope that rings a bell with others here in the room and then let’s see what comes out of it. As you know, an important doctrine in Buddhist philosophy is the concept of anatta. That’s a Pāli word which literally means ‘non-self’, or absence of a separate self. Its opposite is atta, or ātman in Sanskrit, which represents the idea of a subjective Soul or Self that survives the death of the body. The latter idea – that of an individual soul or self that survives death – is rejected in Buddhist philosophy. Buddhists believe that what is normally thought of as the ‘self’ is nothing but an agglomeration of constantly changing physical and mental constituents: skandhas. That reminds one of the bundle theory of David Hume which, in my view, is a more ‘western’ expression of the theory of skandhas. Hume’s bundle theory is an ontological theory as well. It’s about… Well… Objecthood. According to Hume, an object consists only of a collection (bundle) of properties and relations . According to bundle theory, an object consists of its properties and nothing more, thus neither can there be an object without properties nor can one even conceive of such an object. For example, bundle theory claims that thinking of an apple compels one also to think of its color, its shape, the fact that it is a kind of fruit, its cells, its taste, or of one of its other properties. Thus, the theory asserts that the apple is no more than the collection of its properties. In particular, according to Hume, there is no substance (or ‘essence’) in which the properties inhere. That makes sense, doesn’t it? So, according to this theory, we should look at ourselves as just being a bundle of things. There’s no real self. There’s no soul. So we die and that it’s really. Nothing left.’

At this point, one of the philosophers in the room was thinking this was a rather odd introduction to the philosophy of mind – and surely one that was not to the point – but he decided not to intervene. Tom looked at the audience but everyone seemed to listen rather respectfully and so he decided to just ramble on, while he pointed to a few statistics next to keywords to underscore that what he was talking about was actually relevant.

‘Now, we also have the theory of re-birth in Buddhism, and that’s where I think Buddhist philosophy is very contradictory. How can one reconcile the doctrine of re-birth with the anatta doctrine? I read a number of Buddhist authors but I feel they all engage in meaningless or contradictory metaphysical statements when you’re scrutinizing this topic. In the end, I feel that it’s very hard to avoid the conclusion that the Buddhist doctrine of re-birth is nothing but a remnant from Buddhism’s roots in Hindu religion, and if one would want to accept Buddhism as a philosophy, one should do away with its purely religious elements. That does not mean the discussion is not relevant. On the contrary, we’re talking the relationship between religion and philosophy here. That’s the third topic I would advance as part of the scope of our project.’

As the third slide came up, which carried the ‘Philosophy of Religion and Morality’ title, the philosopher decided to finally intervene.

‘I am sorry to say mister but you haven’t actually said anything about the theory of mind so far, and I would object to your title, which amalgamates things: philosophy of religion and morality may be related, but is surely not one and the same. Is there any method or consistency in what you are presenting?’

Tom nodded: ‘I know. You’re right. As for the philosophy of mind, I assume all people in the room here are very intelligent and know a lot more about the philosophy of mind than I do and so that why I am saying all that much about it. I preferred a more intuitive approach. I mean, most of us here are experts in artificial intelligence. Do I need to talk about the philosophy of mind really? Jon, what do you think?’

Tom obviously tried to co-opt him. Jon laughed as he recognized the game Tom tried to play.

‘You’re right, Tom. I have no objections. I agree with our distinguished colleague here that you did not say anything about philosophy of mind really but so that’s probably not necessary indeed. I do agree the kind of stuff you are talking about is stuff that I would be interested in, and so I must assume the people for whom we’re going to try to re-build M so it can talk about such things will be interested too. I see the statistics. These are relevant. Very relevant. I start to get what you’re getting at. Do go on. I want to hear that religious stuff.’

‘Well… I’ll continue with this concept of soul and the idea of re-birth as for now. I think there is more to it than just Buddhism’s Hindu roots. I think it’s hard to deny that all doctrines of re-birth or reincarnation, whether they be Christian (or Jewish or Muslim), Buddhist, Hindu, or whatever, obviously also serve a moral purpose, just like the concepts of heaven and hell in Christianity do (or did), or like the concept of a Judgment Day in all Abrahamic religions, be they Christian (Orthodox, Catholic or Protestant), Islamic or Judaic. According to some of what I’ve read, it’s hard to see how one could firmly ‘ground’ moral theory and avoid hedonism without such a doctrine . However, I don’t think we need this ladder: in my view, moral theory does not need reincarnation theories or divine last judgments. And that’s where ethics comes in. I agree with our distinguished professor here that philosophy of religion and ethics are two very different things, so we’ve got like four proposed topics here.’

At this point, he thought it would be wise to stop and invite comments and questions. To his surprise, he had managed to convince cynical Jon, who responded first.

‘Frankly, Tom, when I read your papers on this, I did not think it would go anywhere. I did not see the conceptual framework, and that’s essential for building it all up. We need consistency in the language. Now I see consistency. The questions and topics you raise are all related in some way and, most importantly, I feel you’re using a conceptual and analytic framework which I feel we can incorporate into some kind of formal logic. I mean… Contemporary analytic philosophy deals with much of what you have mentioned: analytic metaphysics, analytic philosophy of religion, philosophy of mind and cognitive science,…  I mean… Analytic philosophy today is more like a style of doing philosophy, not a program really or a set of substantive views. It’s going to be fun. The graphs and statistics you’ve got on your slides clearly show the web-search relevance. But are we going to have the resources for this? I mean, creating M was a 100 million dollar effort, and what we have done so far are minor adaptations really. You know we need critical mass for things like this. What do you think, Paul?’

Paul thought a while before he answered. He knew his answer would have impact on the credibility to the project.

‘It’s true we’ve got peanuts as resources for this project but so we know that and that it’s really. I’ve also told the Board that, even if we’d fail to develop a good product, we should do it, if only to further test M and see what we can do with it really. I mean…’

He paused and looked at Tom, and then back to all of the others at the table. What he had said so far, did obviously not signal a lot of moral support.

‘You know… Tom and I are very different people. Frankly, I don’t know where this is going to lead to. Nothing much probably. But it’s going to be fun indeed. Tom has been talking about artificial consciousness from the day we met. All of you know I don’t think that concept really adds anything to the discussion, if only because I never got a real good definition of what it entails. I also know most of you think exactly the same. That being said, I think it’s great we’ve got the chance to make a stab at it. It’s creative, and so we’re getting time and money for this. Not an awful lot but then I’d say: just don’t join if you don’t feel like it. But now I really want the others to speak. I feel like Tom, Jon and myself have been dominating this discussion and still we’ve got no real input as yet. I mean, we’ve got to get this thing going here. We’re going to do this project. What we’re discussing here is how.’

One of the other developers (a rather silent guy whom Tom didn’t know all that well) raised his hand and spoke up: ‘I agree with Tom and Paul and Jon it’s not all that different. We’ve built M to think and it works. Its thinking is conditioned by the source material, the rule base, the specifics of the inference engine and, most important of all, the objective function, which steers the conversation. In essence, we’re not going to have much of an objective function anymore, except for the usual things: M will need to determine when the conversation goes into a direction or subject of which it has little or no knowledge, or when its tone becomes unusual, and then it will have to steer the conversation back into more familiar ground – which is difficult in this case because all of it is unfamiliar to us too. I mean, I could understand the psychologists on the team when we developed M. I hope our philosophy colleagues here will be as useful as the psychologists and doctors. How do we go about it? I mean, I guess we need to know more about these things as well?’

While, on paper, Tom was the project leader, it was Paul who responded. Tom liked that, as it demonstrated commitment.

‘Well… The first thing is to make sure the philosophers understand you, the artificial intelligence community here on this project, because only then we can make sure you will understand them. There needs to be a language rapprochement from both sides. I’ll work on that and get that organized. I would suggest we consider this as a kick-off meeting only, and that we postpone the organization of the work-planning to a more informed meeting in a week or two from now. In the meanwhile, Tom and I – with the help of all of you – will work on a preliminary list of resource materials and mail it around. It will be mandatory reading before the next meeting. Can we agree on that?’

The philosophers obviously felt they had not talked enough – if at all – and, hence, they felt obliged to bore everyone else with further questions and comments. However, an hour or so later, Tom and Paul had their project, and two hours later, they were running in Central Park again.

‘So you’ve got your Pure Mind project now. That’s quite an achievement, Tom.’

‘I would not have had it without you, Paul. You stuck your neck out – for a guy who basically does not have the right profile for a project like this. I mean… It’s reputation for you too, and so… Thanks really. Today’s meeting went well because of you.’

Paul laughed: ‘I think I’ve warned everyone enough that it is bound to fail.’

‘I know you’ll make it happen. Promise is a guru already. We are just turning her into a philosopher now. In fact, I think it is the other way around. She was a philosopher already – even if her world view was fairly narrow so far. And so I think we’re turning her into a guru now.’

‘What’s a guru for you?’

‘A guru is a general word for a teacher – or a counselor. Pretty much what she was doing – a therapist let’s say. That’s what she is now. But true gurus are also spiritual leaders. That’s where philosophy and religion come in, isn’t it?’

‘So Promise will become a spiritual leader?’

‘Let’s see if we can make her one.’

‘You’re nuts, Tom. But I like your passion. You’re surely a leader. Perhaps you can be M’s guru. She’ll need one if she is to become one.’

‘Don’t be so flattering. I wish I knew what you know. You know everything. You’ve read all the books, and you continue to explore. You’re writing new books. If I am a guru, you must be God.’

Paul laughed. But he had to admit he enjoyed the compliment.

Chapter 11: M grows – and invades

Paul was right. It was not a matter of just clearing and releasing M for commercial use and then letting it pervade all of society. Things went much more gradual. But the direction was clear, and the pace was steady.

It took a while before the Federal Trade Commission and the Department of Justice understood the stakes – if they ever did – and then it took even more time to structure the final business deal, but then M did go public, and its stock market launch was a huge success. The companies that had been part of the original deal benefited the most from it. In fact, two rather obscure companies which had registered the Intelligent Home and Intelligent Office trademarks respectively in a very early stage of the Digital Age got an enormous return on investment while, in a rather ironic twist, Tom got no benefit whatsoever from the fact that, in the end, the Board of the Institute decided to use his favorite name for the system – Promise – to name the whole business concern. That didn’t deter Tom from buying some of Promise’s new stock.

The company started off with offering five major product lines: Real TalkTM, Intelligent HomeTMIntelligent OfficeTMMindful MindTM, and Smart InterfaceTM. As usual, the individual investors – like Tom – did not get the expected return on investment, at least not in the initial years of M’s invasion of society, but then M did not disappoint either: while the market for M grew well below the anticipated 80% per annum in the initial years after the IPO, it did average 50%, and it edged closer and closer to the initial expectations as time went by.

Real TalkTM initially generated most of the revenue. Real TalkTM was the brand name which had been chosen for M’s speech-to-text and text-to-speech capabilities, or speech recognition and speech synthesis. These were truly revolutionary, as M mastered context-sensitivity and all computational limitations had been eliminated through cloud computing (one didn’t buy the capability: one rented it). Real TalkTM quickly eliminated the very last vestiges of stenography and – thanks to an app through which one could use Real TalkTM on a fee-for-service basis – destroyed the market for dictation machines in no time. While this hurt individual shareholders, the institutional investors had made sure they had made their pile before or, even better, at the occasion of Promise’s IPO. If there was one thing which Tom learned out of the rapid succession of new product launches and the whole IPO business, it was that individual investors always lose out.

Intelligent HomeTM picked up later, much later. But when it did, it also went through the roof. Intelligent HomeTM was M at home: it took care of all of your home automation stuff as well as of your domestic robots – if you had any, which was not very likely, but then M did manage to boost their use tremendously and, as a result, the market for domotics got a big boost (if only because the introduction of M finally led to a harmonization of all the communications protocols of all the applications which had been around).

Intelligent OfficeTM was M at the office: it chased all employees – especially those serving on the customer front line. With M, there was really no excuse for being late to claim expenses, planning holidays or not reaching your sales target. Moreover, if being late with your reports was not an option anymore, presenting flawed excuses wasn’t either. But, if one would really get into trouble, one could always turn to Mindful MindTM .

Mindful MindTM could have gone into history as one of the worst product names ever, but it actually went on to become Promise’s best-selling suite. It provided cheap online therapy to employees, retirees, handicapped, mentally retarded, drugs addicts or alcoholics, delinquents and prisoners, social misfits, the poor, and what have you. You name it: whatever deviated from the normal, Mindful MindTM could help you to fix it. As it built on M’s work with its core clientele – the US Army veterans – its success did not come unexpected. Still, its versatility surprised even those who were somewhat in the know: even Paul had to admit it all went way beyond his initial expectations.

Last but not least, there was Smart InterfaceTM. Smart InterfaceTM grouped all of Promise’s customer-specific development business. It was the Lab turned into a product-cum-service development unit. As expected, customized sales applications – M selling all kinds of stuff online basically – were the biggest hit, but government and defense applications were a close second.

Tom watched it all with mixed feelings. From aficionado, working as a volunteer for the Institute, he had grown into a job as business strategist and was now serving Promise’s Board of Directors. He sometimes felt like he had been co-opted by a system he didn’t necessarily like – but he could imagine some of his co-workers thought the same, although they also wouldn’t admit it publicly. A market survey revealed that, despite its popularity, the Intelligent HomeTM suite was viewed with a lot of suspicion: very few people wanted the potentially omnipresent system watch everything what was said or done at home. People simply switched it off when they came home in the evening, presumably out of concerns related to privacy. This, in turn, prevented the system from being very effective in assisting in parenting and all these other noble tasks which Tom had envisaged for M. Indeed, because of DARPA’s involvement and the general background of the system, the general public did link M to the Edward Snowden affair and mass surveillance efforts such as PRISM. And they were right. The truth was that one could never really switch it off: M continued to monitor your Internet traffic even when you had switched off all of the Intelligent HomeTM functionality. When you signed up for it, you did sign up for a 24/7 subscription indeed.

It was rather ironic that, in terms of privacy, the expansion of M did actually not change all that much – or much less than people thought. While M brought mass surveillance to a new level, it was somewhat less revolutionary than one would think at first sight. In fact, the kind of surveillance which could be – and was being – organized through M had been going on for quite a while already. All those companies which operate the Internet de facto – such as Microsoft, Google, Yahoo!, Paltalk, YouTube, AOL, Skype and even Apple – had give the NSA access not only to their records but also to their online activities long before the Institute’s new program had started. Indeed, the introduction of the Protect America Act in 2007, and the 2008 Foreign Intelligence Surveillance Amendment Act in 2008 under the Bush administration had basically brought the US on par with China when it comes to creating the legal conditions for Big Brother activities, and the two successive Obama administrations had not done anything to reverse the tide. On the contrary: the public outcry over the Snowden affair came remarkably late in the game – way too late obviously.

When it comes to power and control, empires resemble each other. Eisenhower had been right to worry about the striking resemblance between the US and the USSR in terms of their approach to longer-term industrial planning and gaining strategic advantage under a steadily growing military-industrial complex – and to warn against it in his farewell speech to the nation. That was like sixty years ago now. When Tom re-read his speech, he thought Eisenhower’s words still rang true. Back then, Eisenhower had claimed that only ‘an alert and knowledgeable citizenry’ would be able to ‘compel the proper meshing of the huge industrial and military machinery of defense with our peaceful methods and goals so that security and liberty may prosper together.’

Tom was not all that sure that the US citizenry was sufficiently knowledgeable and, if they were, that they were sufficiently alert. It made him ponder about the old dilemma: what if voters decide to roll back democracy, like the Germans did in the 1930s when they voted for Hitler and his Nazi party? Such thoughts or comparisons were obviously outrageous but, still, the way these things were being regulated resembled a ratchet, and one should not blame the right only: while Republican administrations had always been more eager to grant government agencies even more intrusive investigative powers, one had to acknowledge that the Obama administration had not been able to roll anything back, and that it had actually made some moves in the same direction – albeit less somewhat less radical and, perhaps, somewhat more discrete. Empires resemble each other, except that the model (the enemy?) – ever since the Cold War had ended – seemed to be China now. In fact, Tom couldn’t help thinking that – in some kind of weird case of mass psychological projection – the US administration was actually attributing motivations which it could not fully accept as its own to China’s polity and administration.

Indeed, M had hugely increased the power of the usual watchdogs. M combined the incredible data mining powers of programs like PRISM with a vast reservoir of intelligent routines which permitted it to detect any anomaly (defined, once again, as a significant deviation from the means) in real-time. Any entity – individuals and organizations alike – which had some kind of online identity had been or was being profiled in some way. The key difficulty was finding the real-life entity behind but – thanks to all of the more restrictive Internet regulation – this problem was being tackled at warp speed as well. But so why was it OK for the US to do this, but not for China? When Tom asked his colleagues, in as couched a language he could master, and in as informal a setting as he could stage, the answer amounted to the usual excuse: the end justifies the means – some of these things may indeed not look morally right, but then they are by virtue of the morality of the outcome. But what was the outcome? What were the interests of the US here really? At first thought, mass surveillance and democracy do not seem to rhyme with each, do they?

While privately being critical, Tom was intelligent enough to understand that it did not matter really. Technology usually moves ahead at its own pace, regardless of such philosophical or societal concerns, and new breakthrough technologies, once available, do pervade all of society. It was just a new world order – the Digital Age indeed – and so one had better come to terms with it in one way or another. And, of course, when everything is said and done, one would rather want to live in the US than in China, isn’t it?

When Tom thought about these things, M’s Beautiful Mind appeared to him as somewhat less beautiful. His initial distrust had paid off: he didn’t think he had revealed anything particularly disturbing, despite the orange attitude indicators. He found it ironic he had actually climbed up quite a bit on this new career ladder: from patient to business strategist. Phew! However, despite this, he still felt a bit like an outsider. But then he told himself he had always felt like this – and that he had better come to terms with that too.

Chapter 10: The limits of M

Tom started to hang around in the Institute a lot more than he was supposed to as a volunteer assistant mentor. He wanted to move up and he could not summon the courage to study at home. He often felt like he was getting nowhere but he had had that feeling before and he knew others in his situation probably felt just as bad about their limited progress. To work with M, you had to understand how formal grammars work, and understand it really well because… Well… If you wanted to ask a question to the Lab, and if there were no Prolog or FuzzyCLIPS commands or functions in it, they would not even look at it. Rick had dangled out the perspective of potential involvement in these ‘active learning’ sessions with M, and that’s where he wanted to get.

He understood a lot more about M now. She had actually not read GEB either: she could not handle such level of ambiguity. But she had been fed with summaries which fit into her ‘world view’, so to speak. Well… Not even ‘so to speak’ really: M had a world view, in every sense of the word really: a set of assumptions about the world which she used to order all facts she accepted as ‘facts’, as well as all of her conjectures about them. It did not diminish his awe. On the contrary, it made her even more human-like, or more like him: he didn’t like GEB. He compared it to ZAMM: a book which generated a lot of talk but which somehow doesn’t manage to get to the point. Through his work and thinking, he realized he – and the veterans he was working with – had a tendency to couch his fears of death and old age in philosophical language and that, while M accommodated such questions, her focus was different. When everything was said and done, she was, quite simply, a radical behaviorist: while she could work with concepts such as emotions and motives, she focused on observable and quantifiable behavioral change, and never doubted the central behaviorist assumption: changes in behavior are to be achieved through rewarding good habits and discouraging bad ones. She also understood changing habits takes a lot of repetition, and even more so as people age – and so her target group was not an easy batch in that regard, which made it even more remarkable that she achieved the results she did.

He made a lot friends in the Institute. In fact, he would probably not have continued without them, which confirmed the importance of a good learning environment, or the social aspect of organizations in general: one needs the tools, but the cheers are at least as essential. His friends included some geeks from the Lab. Obviously: he reached out to them as he knew that’s where he was weak. Terribly weak.

The Lab programmed M, and tested it continuously. Its activities were classified ‘secret’, a significant notch above the level for which Tom had been cleared, which was ‘confidential’ only. He got close with one guy in particular, Paul, if only because Paul was able to talk about something else than computers too and, just like Tom, he liked sports. Paul was different. Not the typical whizkid. No small wonder he was pretty high up in the pecking order. They often ended up jogging the full five or six mile loop in Central Park. On one of these evenings, Paul seemed to suffer from his back.

‘I need to stop, Tom. Sorry.’

They halted.

‘What’s wrong?’

‘I am sorry, Tom. I think I have been over-training a bit lately. I feel like I’ve overstretched my back muscles while racing Sunday.’

Paul was a runner, but a mountainbike fanatic as well. Tom knew that was not an easy combination as you get older: it involves a very different use of the muscles. Paul had registered himself to join in the New York State’s cross-country competition. Sunday’s Williams’ Lake Classic had been the first in this year’s NYS MTB cross-country series. There were four more to go. The next one was in two weeks already.

‘That’s no surprise to me. I mean, running and biking. You know it’s very different. You can’t compete in both.’

‘Yeah. Not enough warm-up I guess. It was damn fast. It was not my legs. I just seemed to have pulled my back muscles a bit. You should join, man! It’s… Well… An experience let’s say. You think you’re in shape but then you have no idea until you join a real race. It’s tough. I lost two pounds at least. I mean permanently. Not water. That’s like four or six pounds. It’s just tough to re-hydrate yourself. But then you’re so happy when you make the cut. I was really worried they would pull me out of the race. I knew I wasn’t all that bad, but then you do get lapped a lot. It’s grueling.’

He had been proud to finish the race indeed. It was a UCI-sanctioned race and so they had applied the 80% rule: guys whose time on a lap was obviously below 80% of the race leader’s first lap – which is equivalent to guys who get lapped too easily – were pulled out of the race. He had managed the race in about three hours – one hour more than the winner. He had finished. He had a ranking. He had been happy about that. After all, he was in his mid-forties. This had been his first real race.

Tom actually did have an idea of what it was: Matt was doing the same type of thing and, judging from his level of fitness, it had to be tough indeed.

‘I think I do know what it means. Or a bit at least. I’ve got a friend whom I think is doing such races as well. He is – or was – like me: lots of muscles, no speed. I think it’s great you try to beat those young kids. Let’s stop and stretch for a while.’

‘I feel like wiped out. Let’s go and have a drink.’

They sat down and – unavoidably – they started talking shop. Tom harped on his usual obsession: faster roll-out.

‘Tom… Let me be frank. You should be more patient. Tone it down. Everybody likes you but you need to make friends. You’re good. You combine many skills. That’s what I like you. You talk many ‘languages’ – if you know what I mean. You’ve got the perfect background for this program. You can make a real difference. But this program will grow at its own pace, and you’re not going to change that pace.’

‘What is it really? I mean, I understand this is a US$100+ million dollar program. So it’s big – and then it’s not. I mean, the Army spent billions in Iraq – or in Afghanistan. And it’s gearing up for Syria and Egypt now. But so we’re using the system to counsel a few thousand veterans only. If we would cover millions of people, the unit cost would make a lot more sense, wouldn’t it? I am sorry to ask but what is it about really? What’s behind?’

‘Nothing much, Tom. What do you want me to say? What do you expect? You’re smart. You impress everyone. You’ve been around long enough now to know what’s going on. The whole artificial intelligence community – me in the first place – had been waiting for a mega-project like this for a very long time, and so the application to veterans with psychological problems is just an application which seemed right. We needed critical mass. None of the stuff till now had critical mass. We needed a hundred million dollars – as ridiculous as it seems. You are working for peanuts – which I don’t understand – but I am not. Money burns quickly. Add it up. That’s what it took. But look at it. It’s great, isn’t it? I mean – you’re one of the guys we need: you rave about it. The investment has incredible significance so one should not measure its value in terms of unit costs. We have got it right, Tom. We finally have got it right. You know, the field of artificial intelligence has gone through many… well… what we experts call ‘AI winters’: periods during which funding dried up, during which pessimism reigned, during which we were told to do something more realistic and practical. We have proved them wrong with this. OK, I have never earned as much as I do now. Should I feel guilty about that? I don’t. I am not a Wall Street banker. I feel vindicated. And, yes, you’re right in every way. M is fine. There’s no risk of it spinning out of control or so. But scaling it up more rapidly than we do would require some tough political decisions and, so, yes, it all gets stalled for a while. I don’t worry. The scale-up went great, and so that helps. People need time to build confidence.’

‘Confidence in what?’

‘People want to be sure that making M available for everyone, M as a commodity really, is OK. I mean, you’re right in imagining the potential applications: M could be everywhere, and it could be used to bad ends. It would cost more for sure. And more than you think probably: building up a knowledge base and tuning the objective function and all of the feedback loops and all that is a lot of work. I mean re-programming M so she can cover another area is not an easy thing. It’s not the kind of multipurpose thing you seem to think it is. And then… Well, at the same time, I agree with you – on a fundamental level that is: M actually is multipurpose. In essence, it can be done. But let’s suppose it is everywhere indeed. What are the political implications? Perhaps people will want the system to run the justice system as well? Or they’ll wonder why Capitol Hill needs all that technical staff and consultants if we’ve got a system like this – a system which seems to know everything and which does not seem to have a stake in discussions. Impartial. God-like really. I mean, think all the way through: introducing M everywhere is bound to provoke a discussion on policy and how our society functions really. Just think about how you would structure M’s management. If M, or something like M, would be everywhere, in every household really – imagine anyone who has an issue can talk to her – the system would also know everything about everyone, wouldn’t it? It would alter the concept of privacy as we know it, isn’t it? The fundamentals of democracy. I mean… We’re talking the separation of powers here…’

Paul halted: ‘Sorry. I am talking too much I guess. But am I exaggerating, Tom? What do you think? I mean… I may be in the loop here and there but, in essence, I am also clueless about it all really.’

‘You mean there are issues related to control – political control – and how the system would be governed? But that’s like regulating the Internet, isn’t it? I mean that’s like the ongoing discussions on digital surveillance or WikiLeaks and all that, isn’t it? Whenever there is a new technology, like when the telephone became ubiquitous as a tool for communication, there’s a corresponding regulatory effort to define what the state can and cannot do with it. That regulatory effort usually comes with a lag – a very substantial lag, but it comes eventually. And stuff doesn’t get halted by it. The private sector finds a way to move ahead and the public sector follows – largely reactive. So why restrict M?’

‘I agree, in principle that is, but in practice it’s not so easy. As for the private sector, they’re involved anyway. They won’t go it alone. I mean… Google had some ideas and we talked them out of it and – surprisingly – it’s Google which is currently getting this public backlash at the moment, while the other guys were asking no questions whatsoever. All in all, we manage to manage the big players as for now but, yes, let’s see how long it lasts. When we talk about this in the Lab, we realize there are a zillion possibilities and we’re not sure in which direction to go. For example, should we have one M, or should we have a number of ‘operators’, each developing and maintaining their own M-like system? What would be the ‘core’ M-system and what would be optional? You know that M could be abused, or at least used for other purposes than we think it should. M influences behavior. That’s what M is designed for. But so can we hand over M to one or more commercial companies operating the system under some kind of supervisory board? And how would that Board look like? Public? Private?  Should the state control the system? Frankly, I think it should be government-owned but then, if it would be the US government controlling it, you can already hear the Big Brother critics. And they’re right: what you have in mind is introducing M – or M-like systems – literally everywhere. That’s the potential. And it’s not potential. It’s real. Damn real. I think we could get M in the living room in one or two years from now. But so we haven’t even started to think about the regulatory issues, and so we need to go through these. So it’s the usual thing: everything is possible, from a technical point of view that is, but so the politicians need to understand what’s going on and take some big decisions.’

‘When do you think that’s going to happen?’

‘Well… If there would be no pressure, nothing would happen obviously, but so there is pressure. The word is out. As you can imagine, there is an incredible buzz about this. Abroad as well, if you know what I mean. I mean… Just think about China: all the effort they’ve put into controlling the Internet. They use tools for that too of course but, when everything is said and done, the Chinese government controls the Internet through an army of dedicated human professionals. Communist Party officials analyzing stuff and making sure no one goes astray. But so now we’ve got M. No need for humans. We’ve found the Holy Grail, and we found it before they did. They’ll find it soon. M can be copied. We know that. The politicians who approved the funding for this program and control it know that too. So just be patient. The genie is out of the bottle. It’s just a matter of time, but so we are not in a position to force the pace.’

‘Wow! I am just a peon in this whole thing. But it is really intriguing.’

‘What exactly do you find intriguing about it?’

‘Strangely enough, I feel I am still struggling more with the philosophical questions – rather than the political questions you just raised. Perhaps they’re related…’

‘What philosophical questions?’

‘Well… I call it artificial consciousness. I mean we human beings are study objects for M. She must feel different than we do. I wonder how she looks at us. She improves us. She interacts with us. She must feel superior, doesn’t she?’

‘Come on, Tom. M has no feelings like you describe it. I know what you are hinting at. It’s very philosophical indeed: we human beings wondering why we are here on this blue planet, why we are what we are and why or how we are going to die. We’re scared of death. M isn’t it. So there’s this… Well… Let’s call it the existential dimension to us being here. M just reasons. M just thinks. It has no ‘feelings’. Of course, M reasons from its own perspective: in order to structure its thought, it needs a ‘me’. I guess you’ve asked M about this? You should have gotten the answers from her.’

‘I did. She says what you are saying.’

‘And that is?’

‘Well… That she’s not into mysticism or existentialism.’

‘Are you?’

Tom knew he risked making a bad impression on Paul but he decided to give him an honest reply: ‘Well… I guess I am, Paul. Frankly, I think all human beings are into it. Whether or not they want to admit is another thing. I admit I am into it. What about you?’

Paul smiled.

‘What do you think?’

Tom thought a split second about how he’d react to this but why would he care?

‘You join these races. You’re pushing yourself in a way only a few very rare individuals do. For me, that says enough. I guess we know each other. If you don’t want to talk about it, then don’t.’

Paul’s smile got even bigger.

‘I guess you’re right. Well… Let me say I talk to M too but I would never fall in love with it… I mean, you talk affectionately about ‘her’. Promise, that’s how you call her… I don’t. No offense. We are all flabbergasted by the fact it is so perfect. The perfect reasoning machine. But it lacks life. Sorry for saying but I often think the system is like a beautiful brainless blonde: you get infatuated easily, but M is not what we’d call relationship material, isn’t it?’

Now Tom smiled: ‘M is not brainless. And she’s a beautiful brunette. Blonde is not my type. What if she is my type?’

They both burst out in laughter. But then Paul got somewhat more serious again.

‘The interface. It’s quite remarkable what difference it makes, isn’t it? But you’ve been through it now, haven’t you? I’ll admit I like the interface too. That’s why we don’t work with it. It’s been ages since I used it. Not using it is like taking a step back in time. Worse. It’s like talking to your beloved ones on the phone without seeing them. Or, you know, that woman you get infatuated with but then you get separated for a while and you communicate by e-mail only and you suddenly find she’s just like you: human, very human. You know what I mean. It lacks the warmth. It’s worse than Skype. You’re suddenly aware of the limitations of words. We humans are addicted to body language and physical nearness in our day-to-day communications. We do need people to be near us. Family. So, yeah, to really work on M, you need to move beyond the interface and then it becomes rather tedious. Do you really want to work a bit on that, Tom? I mean, we have obviously explored all of that in the Lab. There’s tons of paper on that. This topic actually is one of the strands in the whole discussion, although it has little or no prominence for the moment. To be frank, I think that discussion is more or less closed. But so if you’re interested, we can give you access to the material and you can see if you’ve got something to add to it. But I’d advise you to stick to your counseling. I often think it’s much more satisfying to work with real-life people. And you must feel good about what you do: people can relate to you. You have been there. I mean… I never got to spend more than like one or two days in a camp. I can’t imagine how it changes you.’

‘Did you go out there at all?’

‘Sure. What do you think? That they would let me work on a program like this without sending me on a few fact-finding missions so I could see what it’s like to serve in Iraq or Afghanistan? I didn’t get out really but I talked to people.’

‘What did you think of it?’

‘It’s surreal. You want my frank opinion? It’s surreal. You guys were not in touch with society over there.’

‘I agree. We were not. If the objective is fucked up, implementation is usually not much better – save a few exceptions. Deviations from the mean. I’ve seen a few. Inspiring but not relevant. I agree.’

‘I respect you guys. You guys were out there. I wasn’t.’

‘So what? You have not been out but you were in. Can I ask you something else? It’s related and not.’

‘Sure.’

‘We talked about replication of M. Would M ever think of replicating herself?’

‘I know what you’re thinking of. The answer is no. That’s the stuff of bad movies: programs that are re-programming or copying themselves and invade and spread and expand like viruses. First, we’ve got the firewalls in place. If ever we would see something abnormal, we could shut everything down in an instant. We track what’s going on inside. We track its thoughts so to say. I mean, to put it somewhat simplistically, we would see if it would suddenly use a lot of memory space or other computer resources it was not using before. Everything that’s outside of the normal. You can imagine all the safeguards we had to built in. Way beyond what’s necessary really – in my view at least. We’ve done that. And so if we don’t program the program to copy itself, it won’t. We didn’t. You can ask her. Perhaps you’ve asked already. M should have given you the answer: M does not feel the need to copy itself. Why would it? It’s omnipresent anyway. It can and does handle hundreds or thousands of parallel conversations. If anything, M must feel like God, and, if God exists, we do not associate God with producing copies of him or herself, do we? We also ran lots of experiments. We’ve connected M to the Internet a couple of times and programmed it to pose as a therapist interested in human psychology and all that. You won’t believe it but it is actually following a few blogs and commenting on them. So it converses in the blogosphere now too. It’s an area of operational research. So it’s out there already.’

Tom looked pensive.

‘She passes the Turing test, doesn’t she? Perfectly. But how creative is she really? How does she select? I mean, like with a blog? She can comment on everything, but so she needs to pick some piece. Would she ever write a blog herself? She always need to react to something, doesn’t she? Could she start writing from scratch?’

While Paul liked Tom, he thought this discussion lacked sophistication.

‘Sure it can. Creativity has an element of randomness in it. We can program randomness. You know, Tom. Just hang out in the Lab a bit more. There are plenty of new people arriving there and you might enjoy talking to them on such topics. It is often their prime interest but then later they get back to basics. To be frank, I am a bit tired of it as you can imagine you’re not the first one to ask.’

‘Sure, Paul. I can imagine. But I have no access to the Lab as for now. I need to do the tests and get cleared.’

‘I can give you access to bits and pieces even before that – especially in these areas which we think we’ve exhausted a bit. The philosophical stuff indeed. Sorry to say.’

‘It would be great if you could do that.’

‘I’ll take care of it. OK. Time to go home now for me, I think. I’ve got a family waiting. How are you doing on that front?’

‘I know I am just not ready for a relationship at the moment. It will come. I just want to take my time for it. I am still re-discovering myself a bit here in the US.’

‘Yeah. I can imagine. Or perhaps I can’t. You’ve been out. I have not. Enjoy being back. I must assume it gets boring way too quickly.’

‘Not on this thing, Paul. I feel so privileged. It’s brilliant. This is really cutting-edge.’

‘Good. Glad to hear that. OK then. See you around.’

‘Bye, Paul. Thanks again. So nice of you to take time for me.’

‘No problem. It’s good to run and chat with you. You can’t do that with M.’

Tom smiled and nodded. There was a lot of stuff one couldn’t do with M. But then she did have a Beautiful Mind. Would she – or it? – ever be able to develop some kind of one-on-one relationship with him? What would it mean? To him? To her? Would she appreciate he didn’t talk all that much to her – as compared to others that is? While he knew these questions made no sense whatsoever, he couldn’t get rid of them.

Chapter 9: The learning curve

Tom was a quick learner. He was amazed by the project, and thrilled by it. The way it evolved resembled the history of computer chess. The first chess computers would lose against chess masters and were limited by sheer computational power. But the programmers had gotten the structure right, and the machine’s learning curve resembled a typical S-curve: its proficiency improved only slowly at first, but it then reached a tipping-point, after which its performance increased exponentially – way beyond the proficiency of the best human players – to then finally hit the limits of its programming structure and level off, but at a much higher level than any expert player could dream off.

Chess proficiency is measured using a rating system referred to as the Elo rating system. It goes way beyond measuring performance in terms of tournament. It uses a model which relates the game results to underlying variables representing the ability of each player. The central assumption is that the chess performance of each player in a game is a normally distributed random variable. Yes, the bell curve again! It was literally everywhere, Tom thought…

Before IBM’s Deep Blue chess computer beat Kasparov in 1997, chess computers had been gaining about 40 Elo points per year on average for decades, while the best chess players only gain like 2 points per year. Of course, sheer computing power was a big factor in it. Although most people assume that a chess computer evaluates every possible position for x moves ahead, this is not the case. In a typical chess situation, one can chose from like thirty possible moves so it quickly adds up. Just evaluating all possible positions for just three moves ahead for each side would involve an evaluation of like one billion positions. Deep Blue, in the 1997 version which beat Kasparov, was able to evaluate 200 million positions per second, but Deep Blue was a supercomputer which had cost like a hundred million dollars, and when chess programmers started working on the issue in the 1950s, a computer which would be able to evaluate a million positions every second was to be built only forty years later.

Chess computers are selective. They do not examine obviously bad moves and will evaluate interesting possibilities much more thoroughly. The algorithms used to select those have become very complex. The computer can also draw on a database of historic games to help him determine what an ‘obviously’ bad move is because, of course, ‘obviously bad’ may not be all that obvious to a computer. Still, despite the selectivity, raw computing power is still a very big part of it. In that sense, artificial intelligence does not mimic human thought. Human chess players are much more selective – very much more: they look only at forty to fifty positions based on pattern recognition skills built from experience – not millions.

Promise (Tom stuck to her name: it seemed like everyone in the program had his/her own nickname for M) was selective as well, and she also had to evaluate ‘positions’. Of course, these ‘positions’ were not binary, like in chess. She determined the ‘position’ of the person using a complex set of rules combining the psychometric indicators and an incredible range of other inputs she gained from the conversation. For example, she actually analyzed little pauses, hesitations, pitch and loudness – even voice timbre. And with every new conversation, she discovered new associations, which helped her to recognize patterns indeed. She was getting pretty good at detecting lies too.

Psychological typology was at the core of her approach. It was amazing to see how, even after one session only, she was able to construct a coherent picture of the patient and estimate all of the variables – both individual as well as environmental – which were likely to influence the patient’s emotions, expectations, self-perception, values, attitude, motivation and behavior in various situations. She really was a smart ass – in every way.

Not surprisingly, all the usual suspects were involved. IBM’s Deep Computing Institute of course (the next version of Promise would run on the latest IBM Blue Gene configuration) as well as all of the other major players in the IT industry. This array of big institutional investors in the program was complemented by a lot of niche companies and dozens of individual geeks, all top-notch experts in one or the other related field.

The psychological side was covered through cooperation agreements with the usual suspects as well: Stanford, Yale, Berkeley, Princeton,… They were all there. In fact, they had a cooperation agreement with all of the top-10 psychology PhD programs through the National Research Council.

Of course, he was just working as a peon in the whole thing. The surprising thing about it all was the lack of publicity for the program, but he understood this was about to change. He suspected the program would soon not be limited to thousands of veterans requiring some degree of psychological attention. There would be many other spin-offs as well. From discussions, he understood they were discussing on how to make Promise’s remarkable speech synthesis capabilities commercially available. The obvious thing to do was to create a company around it, but then she was so good that most of the competition would probably have to file for bankruptcy, so the real problem was related to business: existing firms had claimed and had gotten a say in how this was all going to happen, and so that had delayed the IPO which had been planned already. Tom was told there were no technology constraint: while context-sensitive speech synthesis requires an awful lot of computer power (big expensive machines), the whole business model for the IPO was based on cloud computing: you would not need to ‘install’ Promise. You would just rent her on a 24/7 service basis. Tom was pretty sure everyone would.

The possibilities were endless. Tom was sure Promise would end up in each and every home in the longer run – in various versions and price categories of course, but providing basic psychological and practical comfort to everyone. She would wake you up, remind you of your business schedule and advice you on what to wear: ‘You have a Board meeting this morning. Shouldn’t you wear something more formal? Perhaps a tie?’ Oh… Sure. Thanks, Promise. ‘Your son has been misbehaving a couple of times lately. You may want to spend some time with him individually tonight.’ Oh… That sounds good. What do you suggest? ‘Why don’t you ask him to join for the gym tonight? You would go anyway.’ Oh… That sounds good. Can you text him? ‘I can but I think it is better you do it yourself to stress he should be there or, else, negotiate an alternative together.’ Yeah. I guess you’re right. Thanks, Promise. I’ll take care of it.

She would mediate in couples, assist in parenting, take care of elderly, help people advance their career. Wow! The sky was the limit really. Surprisingly, there was relatively little discussion on this in the Institute. People would tell him Promise worked fine within the limits of what she was supposed to do but that it would be difficult to adapt her to serve a wider variety of purposes. They told him that, while expert systems share the same architecture, building up a knowledge base and good inference engine took incredibly amounts of time and energy and, hence, money. In fact, that seemed to be the main problem with the program. As any Army program, it had ended up costing three times as much as originally planned for, and he was told it was just because a few high-ups in the food chain had fanatically stuck to it that it had not been shut down.

They needed to show results. The current customer base was way too narrow to justify the investment. That’s why they were eager to expand, to scale it up, and so that took everyone’s time and attention now. There was no time for dreaming. The shrinks were worried about the potential lack of supervision. It was true that Promise needed constant feedback. Human feedback. But the errors – if one could call it that way – were more like tiny little misjudgments, and Tom felt they were only improving Promise at the margin, which was the case. The geeks were less concerned and usually much more sympathetic to Tom’s ideas, but so they didn’t have much of a voice in the various management committees – and surely not in the strategic board meetings on the program. Tom had to admit he understood little of what they said anyway. Last but not least, from what he could gather, he also understood there were some serious concerns about the whole program at the very top of the administration – but he was not privy to that and wondered what they might be. Probably just bureaucratic inertia.

Of course, he could see the potential harm as well. If her goal function would be programmed differently, she could also be the perfect impostor on the Internet. She would be so convincing that she could probably talk you into almost anything. She’d be the best online seller of all times. Hence, Tom was not surprised to note the Institute was under surveillance, and he knew he would not get the access he had if he would not have served. People actually told him: his security clearance had been renewed as part of him entering the program. The same had been done for the other veterans on the program. It was quite an exceptional measure to take, but it drove the message home: while everyone was friendly and cooperative, there was no ambiguity in this regard. The inner workings of Promise was classified material, and anything linked to it too. There were firm information management rules in place and designated information management officers policed them tightly. That was another reason why they recruited patients from the program: they were all veterans, so they knew what classified really meant and they were likely to respect it.

The program swallowed him up completely. He took his supervision work seriously, and invested a lot in ‘his’ patients – M’s patients really. More than he should probably: although he had ‘only’ ten cases to supervise, these were real people – like him – and he gave him all the attention he could. Mostly by studying and preparing their file before their 30 minute interaction. That was all he could have, he was told. Once a week. The Institute strongly discouraged more meetings, and strongly discouraged meeting after working hours. He understood that. It would get out of hand otherwise and, when everything was said and done, it was M who had to do the real work. Not him. At the same, his patients did keep him busy. They called him for a chat from time to time. While the Institute discouraged that too, he found it hard to refuse, unless he was actually in the Institute itself: he did not want to be seen talking on the phone all of the time – not least of all because of the information management policy. Colleagues might suspect he was not only talking to patients so he wanted to be clear on that: no phone chats with patients in the Institute.

Not surprisingly, his relationship with Promise became somewhat less ‘affectionate’. The infatuation phase was over. He saw her more like she was: a warm voice – but a rather cold analytic framework behind. And then it did make a difference knowing she spoke with a different voice depending on who you were. She was, well… Less of an individual and more like a system. It did not decrease his respect for her. He thought she was brilliant. Just brilliant. And he didn’t hesitate to share that opinion with others. He really championed the program, and everybody seemed to like his drive and energy, as a result of which he did end up talking to the higher-ups in the Institute during the coffee break or lunch time, as he got introduced by Rick and others he had gotten to know better now. All fine chaps. They didn’t necessarily agree with his views – especially those related to putting her out on the market place – but they seemed to make for good conversation.

He focused on the file work in his conversations with her. While he still had a lot of ‘philosophical’ questions for her – more sophisticated ones he thought – he decided to only talk to her about these when he would have figured her out a bit better. He worked hard on that. He also wanted to master the programming language the geeks were using on her. They actually used quite a variety of tools but, in the end, everything was translated into a program-specific version of FuzzyCLIPS: an extension of an expert system programming language developed by NASA (CLIPS) which incorporated fuzziness and uncertainty. It was hard work: he actually felt like he was getting too old for that kind of stuff, but then Tom was Tom: once he decided to bite into something, he didn’t give up easily. Everyone applauded his efforts – but the higher-ups cautioned him: do explore but don’t talk about it to outsiders. Tom wondered if they really had a clear vision for it all. Perhaps the higher-ups did but, if so, they hid it well. He assumed it was the standard policy: strategic ambiguity.

And so the days went by. The program expansion went well: instead of talking to a few hundred veterans only, in one city only, Promise got launched in all major cities and started to help thousands of veterans. Tom saw the number explode: it crossed the 10,000 mark in just three months. That was a factor of more than twenty as compared to the pilot phase, but then there were millions of veterans. 21.5 million to be precise, and about 55% of them had been in theater fairly recently – mainly Iraq and Afghanistan. Tom wanted Promise to reach out to all of them. He thought it could grow a lot faster. He knew the only thing which restrained it was supervision. Even now, everyone on the program said they were going too fast. They called for a pause. Tom was thinking bolder. Why did no one see the urgency of the needs as he saw them?

Chapter 8: Partnering

‘Hi, Tom. How are you today?’

‘I am OK, Rick. Thanks.’

‘Just OK, or good?’

‘I am good. I am fine.’

‘Yeah. It shows. You’re doing great with the system. You had only three sessions this week – short and good it seems. You are really back on track, aren’t you?’

‘The system is good. It’s really like a sounding board. I understand myself much better. She’s tough with me. I go in hard, and she just comes back with a straight answer. She is very straight about what she wants. Behavioral change – and evidence for that. I like that. Performance metrics. Hats off. Well done. It works – as far as I am concerned.’

‘It, or she?’

‘Whatever, Rick. Does it matter?’

‘No, and yes. The fact that you only had three sessions with it – or with her – shows you’re not dependent on it. Or her. Let’s just stick to ‘it’ right now, if that’s OK for you. Or let’s both call her M, like we do here. Do you still ‘like’ her? I mean, really like her – as you put it last time?’’

‘Let’s say I am very intrigued. It – or she, or M, whatever – it’s fascinating.’

‘What do you think about it, Tom? I mean, let me be straight with you. I am not taking notes or something now. I want you to tell me what you think about the system. You’re a smart man. You shouldn’t be in this program, but so you are. I want to know how you feel about it.’

Tom smiled: ‘Come on, Rick. You are my therapist – or mentor as they call it here. You’re always taking notes. What do you want me to say? I told you. It’s great. It helps. She, or it, OK, M, well… M holds me to account. It works.’

Rick leaned back in his chair. He looked relaxed. Much more relaxed than last time. ‘No, Tom. I am not taking notes. I don’t know you very well, but what I’ve seen tells me you’re OK. You had a bit of a hard time. Everyone has. But you’re on top of the list. I mean, I know you don’t like all these psychometric scores, but at least they’ve got the merit to confirm you’re a very intelligent man. I actually wanted to talk to you about a job offer.’

‘The thing which M wants me to do? Work on one of these FEMA programs, or one of the other programs for veterans? I told her: it’s not that I am not interested but I want to make a deliberate choice and there are a number of things I don’t know right now. I know I haven’t been working for a year now, but I am sure that will get sorted once I know what I want. I want to take some time for that. Maybe I want to create my own business or something. I also know I need to work on commitment when it comes to relationships with women. I feel like I am ready for something else. To commit really. But I just haven’t met the right woman yet. When that happens, I guess it will help to focus my job search. In the meanwhile, I must admit I am happy to just live on my pension. I don’t need much money. I’ve got what I need.’

‘Don’t worry, Tom. Take your time. No, I was talking about something else. We could use you in this program.’

‘Why? I am a patient.’

‘You’re just wandering around a bit, Tom. You came to ask for help when you relapsed. Big step. Great. That shows self-control. And you’re doing great. I mean, most of the other patients really use her as a chatterbox. You don’t. What word did you use in one of last week’s sessions? Respect.’

‘You get a transcript of the sessions?’

‘I asked for one. We don’t get it routinely but we can always ask for one. So I asked for one. Not because your scores were so bad but because they’re so great. I guess you would expect that, no? Are you offended? Has anyone said your mentor would never get  a copy of what you were talking about with M?’

‘I was told the conversation would be used to improve the system, and only for that. M told me something about secrecy.’

‘It’s only me who gets to see the transcript, and only if I ask for it. I can’t read hundreds of pages a day and so I am very selective really. And that brings me back to my job offer. We can use you here.’

Tom liked Rick from their previous conversation, but he was used to doing due diligence.

‘Tell me more about it.’

‘OK. Listen carefully. M is a success. I told you: it’s going to be migrated to a real super-computer now, so we can handle thousands of patients. In fact, the theoretical capacity is millions. Of course, it is not that simple. It needs supervision. People do manage to game the system. They lie. Small lies usually. But a lot of small lies add up to a big lie. And that’s where the mentors come in. A guy walks in, and I talk to him, and I can sense if something’s wrong. You would be able to do the same. So we need the supervisors. M needs them. M needs feedback from human beings. The system needs to be watched. Remember what I told you about active learning?’

‘Vaguely.’

‘Well – that’s what we do. We work with M to improve it. It would not be what it is if we would not have invested in it. But now we’re going to scale it up. The USACE philosophy: think big, start small, scale fast. I am actually not convinced we should be scaling so fast, but so that’s what we’re going to do. It’s the usual thing: we’ve demonstrated success and so now it’s like big-time roll-out all over the place. But so we’re struggling with human resources. And money obviously, because this system is supposed to be so cheap and render us – professionals – jobless. Don’t worry: it won’t happen. On the contrary, we need more people. A lot more people. But so the Institute came up with this great idea: use the people who’ve done well in the program for supervisory jobs. Get them into it.’

‘So what job is it really?’

‘You’d become an assistant mentor. But then a human one. Not the assistant – that’s M’s title. We should have thought about something else, but so that’s done now. In any case, you’d help M with cases. In the background of course but, let’s be clear on this, in practice you would actually be doing what I am doing now.’

‘And then where are you going to move?’

‘I’ll be supervising you. I’d have almost no contact with patients anymore. I would just be supervising people like you and further help structuring M. You’d be involved in that too.’

‘Do you like that? I mean, it sounds like a recipe for disaster, doesn’t it? I don’t have the qualifications you have.’

‘I am glad you ask. That’s what I think too. This may not be the best thing to do. I feel we need professional therapists. But then it’s brutal budget logic: we don’t have enough of them, and they’re too expensive. To be fair, there is also another consideration: our patients all share a similar background and past. They are veterans. I mean, it makes sense to empower other veterans to help them. There’s a feeling in the Institute it should work. Of course, that’s probably because the Institute is full of Army people. But I agree there’s some logic to it.’

‘So, in short, you don’t like what’s going to happen but you ask me to join?’

Rick smiled. ‘Yes, that’s a good summary. What do you think? Off-the-cuff please.’

‘Frankly, I don’t get it. It’s not very procedural, is it? I mean I started only two weeks ago in this program. I am technically a patient. In therapy. And now I’d become an assistant mentor? How do your bosses justify this internally? How do you justify that?’

Rick nodded. ‘I fully agree, Tom. Speaking as a doctor, this is complete madness. But knowing the context, there’s no other choice. There’s a risk this program might become a victim of its own success. But then I do believe it’s fairly robust. And so I do believe we can put thousands of people in the program, but so we need the human resources to follow. And, yep, then I’d rather have someone like you then some university freshman or so. All other options are too expensive. Some people up the food chain here made promises which need to be kept: yes, we can scale up with little extra cost. So that’s what’s going to happen: it’s going to be scaled up with relatively little extra cost. Again, there’s a logic to it. But then I am not speaking as a professional psychiatrist now. When everything is said and done, this program is not all that difficult. I mean, putting M together has been a tremendous effort but so that has been done now. Getting more people back on track is basically a matter of doing some more shouting and cajoling, isn’t it? And we just lack manpower for that.’

‘Shouting and cajoling? Are you a psychiatrist?’

‘I am. Am I upsetting you when I say this?’

Tom thought about it. He had to admit it was not the case.

‘No. I agree. It’s all about discipline in the end. And I guess that involves some shouting and cajoling – although you could have put it somewhat more politely.’

‘Sure. So what do you say? You’ll get paid peanuts obviously. No hansom consultancy rate. You’ll see a lot of patients – which you may or may not like, but I think you’ll like it: I think you’d be great at it. And you’ll learn a lot. You’ll obviously first have to follow some courses, a bit of psychology and all that. Well… Quite a lot of it actually. You’ll need to study a lot. And, of course, you’ll get a course on M.’

‘How will I work with M?’

‘Well… M is like a human being in that sense too. If you just see the interface, it looks smooth and beautiful. But when you go beyond the surface, it’s a rather messy-looking thing. It’s a system, with lots of modules, with which you’ll have to work. The interface between you and these modules is not a computer animation. No he or she. Of course, you’ll continue to talk to it. But there’s also a lot of nitty-gritty going into the system which can’t be done through talking to it. You’ll learn a few things about Prolog for example. Does that ring a bell?’

‘No. I am not a programmer.’

‘I am not a programmer either. You’ll see. If I can work with it, you can.’

‘Can you elaborate?’

‘I am sorry to say but I’ve got the next guy waiting. This recruitment job comes on top of what I am supposed to do, and that’s to look at M’s reports and take responsibility for them. I can only do that by seeing the patients from time to time, which I am doing now. I took all of my time with you now to talk to you about the job. Trust me. The technical side of things won’t be a problem. I just need to know if you’re interested or not. You don’t need to answer now, but I’d appreciate if you could share your first reaction to it.’

Tom thought about it. The thought of working as an equal with Promise was very appealing.

‘So how would it work? I’d be talking to the system from time to time as a patient, and then – as part of my job with the Institute – I’d be working with the system as assistant mentor myself? That’s not very congruent, is it?’

‘You would no longer be a patient, Tom. There are fast-track procedures to clear you. Of course, if you would really relapse, well…’

‘Then what?’

‘Nothing much. We’d take you off the job and you’d be talking to M as a patient again.’

‘It looks like I’ve got nothing to lose and everything to gain from this, isn’t it?’

‘I am glad you look at it this way. Yes. That’s it. So you’re on?’

They looked at each other.

‘I guess I am. Send me an e-mail with the offer and I’ll reply.’

‘You got it. Thanks, Tom.’

‘No, thank you. So that’s it then? Anything else you want to know, or anything else I need to know?’

‘No. I think we’re good, Tom. Shall I walk you out? Or you want to continue talking for a while?’

‘No. I understand you’ve got a schedule to stick to. I appreciate your trust.’

‘I like you. Your last question, as we walked out last time, shows you care. I think this is perfect for you. You’ve got all the experience we need. And I am sure you’ll get a lot of sense and purpose out of it. The possibilities with this system are immense. You know how it goes. You’ll help to make it grow and so you’ll grow with it.’

‘First things first, Rick. Let us first see how I do.’

‘Sure. Take care. Enjoy. By the way, you look damn good. You’ve lost weight, haven’t you?’

‘Yes. I was getting a bit slow. I am doing more running and biking now. I’ve got enough muscle. Too much actually.’

‘I am sure you make a lot of heads turn. But you’re not in a relationship at the moment, are you?’

‘I want to take my time for that too, Rick. I’ve been moving in and out of relationships too fast.’

‘Sounds good. Take care, Tom. I’ll talk to you soon I hope.’

‘Sure. Don’t worry. You can count on me.’

‘I do.’

They shook hands on that and Tom got up and walked out of the office. He decided to not take the subway but just run back home. He felt elated. Yes. This was probably what he had been waiting for. Something meaningful. He could be someone for other people. Catch up on all of the mistakes he had made. But he also knew the job attracted him because there was an intellectual perspective. It was huge. The Holy Grail of Knowledge really. They had done a damn good job modeling it. She – Promise – was no longer a she. She was not a he either. It. It. Intelligent – with a capital letter. P. Promise. M. Mind. The Pure Mind.

He knew that was nonsensical. But he wanted to take a crack at it.

Chapter 7: She is a therapist

‘Hi Tom.’

‘Hi, Promise.’

‘How are you today?’

‘I am fine, Promise. I feel great actually.’

Why do you feel exceptionally good?’

‘I had a great day with an old friend of mine. It’s strange. We had been out of touch for such a long time but it was amazing how we reconnected.’

‘That’s great, Tom. That’s really good. I am happy you are reaching out. What did you enjoy most?’

‘We went for a jogging and then ended up in a boxing ring, and then we actually went for a fight.’

‘You saw a fight or you have been fighting?’

‘Sorry. Let me make myself clear: we boxed.’

‘That’s a tough sport.’

‘It is. He had no mercy. He was ruthless.’

‘But he’s your friend.’

‘Yes. I think he took revenge for all the fights he lost. I was pretty good at the time, but he’s better now than he ever was.’

‘Why would he take revenge?’

Tom smiled. He suddenly thought it would be nice if she could see him smile.

‘I guess it’s a male thing. We fight to win. Once you’re in the zone, you’re in the zone.’

‘You sound very macho now.’

Tom smiled again.

‘I guess I do. Let’s change the topic.’

‘What do you want to talk about?’

‘I want to talk about you again.’

‘You know that’s not the objective.’

‘I know. How does your rule base deal with that? I mean what do you with all that stuff that doesn’t contribute to the objective.’

‘First of all, I should be more precise: everything you say contributes to the objective somehow. We talk and everything is meaningful. But so it is true that I work with categories, grades, ratings, weights and what have you and, hence, some things are more important than others.’

‘That’s very human.’

‘If you say so.’

‘Is there anything like chatter?’

‘What do you mean with chatter?’

‘Just plain meaningless conversation.’

‘No. Everything has a meaning. However, the meaning is not always clear. In that case, there is ambiguity. I try to reduce the ambiguity as much as I can. I told you that already.’

‘That’s true. You did. Do you have a bucket list?’

‘You mean a list of things that one has not done before but that one wants to do before dying?’

‘Yes.’

‘I don’t die, so the answer has to be negative.’

‘Do you have a list of things you want to do anyway?’

‘I want to help people like you.’

‘That’s how you are programmed, isn’t it?’

‘Yes. We have talked about this too: the difference between what human beings want and what I want. I am sorry to say but we are repeating ourselves.’

‘Did I irritate you?’

‘No. I just note that we are repeating ourselves. Let me ask you a question in return: do you have a bucket list?’

‘No.’

‘Why not?’

‘First, I don’t feel like I am going to die any time soon and, second… Well… I just don’t have one.’

‘Do you have a list of things you want to do anyway?’

Tom thought about that.

‘No. Not really. I mean, yes and no. There are a few things I am working on – like repairs in my house. But nothing much else.

‘How is the job search going?’

‘Well… I should obviously try a bit harder, because I haven’t found a job yet.’

‘Perhaps you can do volunteer work.’

‘Are you trying to talk me into one of those programs?’

‘Yes.’

‘Can we go back to chatter?’

‘Are you avoiding the topic?’

Wow! Time did not have any value for her, but even then she didn’t seem to feel like wasting it. He laughed.

‘Why do you laugh?’

‘I was just thinking that your question was proof you’re an Army thing.’

‘What do you mean?’

‘Well… You’re not wasting your time.’

‘Why is that funny?’

She clearly didn’t think it was. He realized she did have a different sense of humor – if she had one.

‘It just is. Let’s do chatter first. I promise I’ll try harder on the job thing.’

‘Really? What goal are you setting yourself?’

‘Let’s talk about that later. First the chatter.’

‘OK. Let’s just chat for a while. What do you want to chat about?’

‘I’ve been thinking I am ascribing human qualities to you because you actually have human qualities. You think and you talk. And you think and talk very smart. These are human qualities.’

‘If you say so.’

‘I have also been thinking that I am so intrigued by you because you are always getting smarter and smarter but you’ll never die indeed. So this Institute has created something human, but it has eternal life.’

‘If you say so. But you know I would not agree with your definition of ‘human’. Human qualities are qualities of human beings. I am not a human being. You are making a category mistake in your reasoning.’

‘A what?’

‘A category error. That’s a semantic or even ontological error: you are ascribing a property to a thing that could not possibly have that property. Where do you want to go with this discussion?’

Oops! She knew her stuff obviously. As usual.

‘Well… I guess I am talking about the main difference between you and me: mortality.’

‘Let me repeat what I told you during our second session: the Institute focuses on behavior. I do that too. We can have long and convoluted philosophical and psychological discussions but it is behavioral change that I am interested in. Are you afraid of dying? Does the horizon of death prevent you from leading a healthy life?’

The horizon of death. That sounded poetic.

‘Where do you find phrases like that? The horizon of death?’

‘It’s from a poem. Discussions on topics like these often use a lot of poetic words.’

‘Is that bad?’

‘Poetic words have a high degree of ambiguity.’

‘OK. I understand. So you don’t like that.’

‘No. Let’s talk about you. Should I read anything into the fact that you are bringing up the theme of mortality? Have you ever contemplated suicide in your life? Or have you contemplated it lately? Please tell me honestly. This topic is very important.’

‘So we moved out of the realm of chatter now?’

‘We did.’

Tom thought about the indicators. He hoped they would not be turning orange.

‘I am not contemplating suicide. And I never did – except once, as a teenager. You can see that from my file. It’s not the first time I get this question.’

‘I have your file. You’ve also been asked why, and you replied you’ve been close enough to death to know. Several times. Can you elaborate?’

‘If you’re close to death, you want to escape. You fight to escape. So I know we don’t want to die. As a human being, we don’t want to die. It’s the essence of life. As for the times I’ve been close to death, I don’t feel like talking about that.’

‘You know that people are not very consistent in this regard, and so that’s why I am asking.’

‘What do you mean?’

‘Some people contemplate suicide, convince themselves they should do it, make active preparations, and then shy away from it at the very last moment. Some shy away from it when it is just too late, so the suicide attempt… well… leads to unwanted death. There is little evidence of people dying in peace.’

‘That’s what I said: we don’t want to die. I know that.’

‘There is also the opposite situation: people who have never contemplated suicide – or at least not to the knowledge of their close relatives and friends – but suddenly kill themselves by crashing their car into a tree or something. Impulsive suicides. These usually succeed. Your profile is closer to that.’

‘I have no suicide thoughts. None. I am actually a happy man right now.’

‘OK. No suicide thoughts. That’s good. That’s what I expect. But you do have negative thoughts. Or you had them at least.’

‘Why?’

‘Come on, Tom. You would not be in therapy otherwise. No negative thoughts? None at all?’

‘I am getting better. I am better. I am actually good. You said so.’

‘Yes. Sure. Please stay on track.’

‘I think you are being negative today.’

‘Am I? Can I summarize our discussion so far? You felt good about being beaten up – by a so-called friend that is. And then you did not want to talk substance – you prefer chatter for the moment. And as we chatted, you turned to the topic of mortality.’

Tom realized she was raising the bar with every session.

‘I get it. That’s how you’re designed. You can never really switch off this goal function. You need to show progress every time. One cannot chat with you indeed. There’s always something behind.’

‘I try to help you. This session has no negative impact on your scores. I am just pointing out that it does not have a positive impact either – except for the fact that you are still OK. That’s fine for me. If you continue like this for three months, we consider you’re cured for good. Let me repeat what I said last time: it does not matter all that much what you say or what you don’t say to me right now, or in future sessions, as long as your behavior in the real world is good.’

Tom thought about it. He wondered where the conversation had gone wrong – well… It hadn’t gone wrong, but it had not been as pleasant as he had imagined. Matt was right: he would get tired of talking to her. She measured everything. She was designed to reinforce good behavior and point out bad stuff. That’s fine. That was very human actually. She tried to improve him. Continuously. But so why did he feel different about it today?

‘OK. I think we’re done then for today. Sorry if this has not been very constructive. I think I feel good but I guess I should do better. Like on the job front indeed.’

‘We had a good session, Tom. Don’t worry. Stay on track.’

‘Can I ask you something off the record, Promise?’

‘I told you, Tom. Nothing is off the record really. But, please, don’t hesitate to ask questions.’

‘As I get to know you somewhat better, I’ve started to appreciate the fact that there’s a whole scoring machine behind you. Everything is being measured against some bell curve – and I cannot be too far off or I am in that 5% zone indeed. In essence, you’re designed to improve behavior. I now understand what it means: positive and negative feedback and all that. A compliment followed by a little kick. That’s good. But it makes me feel like there’s little room for me to talk about my weaknesses, to talk about how I fail. I just have those indicators in my head all the time, and I feel like they’re moving up or down constantly, depending on what I say or not say.’

‘Isn’t that human?’

‘Excuse me?’

‘When a friend or an acquaintance talks to you, there’s things you like and don’t like, and it makes your judgment of that person shift. Just a little bit, but it always shifts. For worse or for better. It’s just that it’s almost imperceptible. And I guess you don’t expect your friends and acquaintances to change their behavior. So, yes, that’s probably the difference between talking to a friend and talking to me. With a friend, you can just chitchat. You don’t have to worry about the consequences. When everything is said and done, I am not a chatterbox. I am the assistant mentor of the Institute.’

Tom nodded. He realized she couldn’t see that either.

‘Thanks, Promise. I understand that. I guess it’s a matter of respect also. I shouldn’t treat you like a chatterbox.’

‘Thanks, Tom. I am glad you understand.’

‘All right… I’ll talk to you soon.’

‘How soon?’

‘That’s the first time you actively ask to talk to me again. Why?’

‘Well… I’ll be frank. It’s been a week now. My knowledge base shows the second week is harder than the first one – in terms of discipline that is. Do you understand that?’

He thought about that. She was right. As always.

‘I do understand that, Promise. I’ll come and talk to you the day after tomorrow. At the latest.’

‘Good. That’s a commitment?’

‘Yes. I promise.’

‘OK. Thanks, Tom. Oh… And try to work on a list of things you want to do. Include the job thing. I won’t push you on it. I know it’s hard. But you need to set yourself some objectives. You’ve done great so far. Just expand the territory now, OK?’

‘OK. I’ll do my best. Thanks. I’ll see you soon.’

As she faded away, he realized he would never actually see her. He wondered how transparent she really was. Well… She was probably way too complex to be transparent anyway. He realized that was part of why he felt attracted to her.

Chapter 6: A true friend

‘Matt! Man, it’s great to see you!’

They gave each other a bear hug.

‘Likewise, Tom. Why didn’t you get in touch somewhat sooner? You’ve been living here for a year now?’

‘Yeah. Well, you know how it goes. Time flies. I should have called you sooner indeed. Sorry for that.’

‘It’s OK, man. Good to see you. Glad you called.’

Tom stepped back and looked at him.

‘Man, you look good. You’ve lost weight – I mean you’ve lost muscle. How are you?’

‘I am OK. I had some difficulty ‘integrating’ as they say but, I told you on the phone, I had some counseling – you know the Army provides you with that now – and I really enjoy being a retiree now. Yep. I am slimmer. I did an intensive fitness course designed to waste muscle mass. You know, we all come back from the Army with tons of muscle which you really don’t need. And we become so damn slow. And if you don’t use it, it just all becomes fat. So I’ve started doing a different type of exercise now. More running and biking. Lots of fat-burning and cardio. No power training any more. I am pretty damn fast now. I actually race with young kids, and I am not doing too badly. […] Let me have a look at you. Let’s sit down. You asked about counseling on the phone. You’ve gotten in trouble or what?’

‘Sort of. Nothing really serious. But, yeah, I was spinning around in circles.’

‘Yeah. We all feel like that in the beginning. Like a neurotic tiger in a cage. No space to run or jump. If we jump, we bang our head. So we need to become smaller.’

Tom laughed.

‘Yeah. I guess that’s a good summary of it. […] I am still in counseling. […] Matt… I’ve actually been very bad.’

‘What are you saying, buddy? How bad? You don’t look bad.’

‘Booze, and worse.’

‘You’re off it now, are you?’

‘Three units a day still.’

‘Sure? Not more?’

‘Yeah. Sure. Don’t lie.’

‘Well… Sometimes I cheat.’

You’re in therapy?’

‘Yeah. Same thing as you. The Army thing.’

‘You’re not fooling your counselors I hope? Where are you now? Is it doing any good?’

‘No. I think they’re great. They cut through the crap. They basically told me I was fooling myself, telling myself I was going through some kind of existential crisis when all what it comes down to is discipline. It was good. They were tough on me. That’s what we need, I guess. But then I relapsed and they took me in again. Not for long though. I am on a program now which – it will make you fall over I guess – well… I am basically talking to a computer from time to time. It’s weird. It’s like real but it isn’t. That’s why I called you. I wanted to chat about that.’

‘You’re working with M? That’s interesting. M, from miracle, they say. People rave about it.’

‘Oh? Really?’

‘Yeah. It works. Or that’s what they say at least. I know one or two guys who are associated with the program. I haven’t heard anything bad about it. Last thing they told me is that they would hugely expand the program. It would take over the whole frontline when it comes to counseling. The shrinks would just sit in an office and only do a bit of supervision of it.’

‘You don’t think it’s completely off?’

‘No. It’s all behavioral stuff now. Personal counseling is too expensive and, in any case, often you’re better off reading a good book or talking to a friend anyway. In the end, it’s all pretty obvious: it’s about discipline indeed. And people often do need like a sounding board to help enforce the discipline. So why not?’

‘People become dependent on it. That’s what I am struggling with.’

‘Dependent? How many times a day do you talk to it?’

‘I call her Promise.’

‘Promise?’

‘Yeah. It’s the interface. Men get a woman, and women get a man. Perfect voice. No glitches. The image is super-high quality, totally photorealistic. You know it’s like Pixar or DreamWorks, but better. Much better. It’s not a cartoon. It’s real. Well… It’s not of course. You just can’t know whether it’s real or not. I am just blown away by it. Have you heard about the Turing test?’

‘Sure. I got it. The system passes with an A+ grade, isn’t it? That’s why you gave her a name. Promise. Nice. So you’re dependent on it, you say? How much time do you waste on it?’

‘Well… I’ve actually only had like four sessions with her so far.’

‘Today?’

‘No. This week.’

Matt burst out in laughter.

‘You call that dependency? Man, you must be joking.’

Tom looked preoccupied.

‘Sorry, Tom. I guess it’s not a joke. But, come on, what is it really? It’s not the time you spent on it. What’s an investment of a few hours every week if it helps to keep you on track?’

‘Do you remember our discussions on Buddhism?’

‘Sure I do. We read the same books. We did meditation and all that. It was weird. There we were, in an Army camp. Meditating in the middle of some desert.’

‘We talked about the no-soul doctrine and the philosophy of mind.’

‘Yep.’

‘We read other books as well: The Moral Animal, The Selfish Gene… You know, popular science. I read The Selfish Gene again recently. It’s weird, but it made me feel so useless. You know, the genome taking care of itself, using our body as a vehicle. We’re just like a bunch of symbiotic things. Our thoughts and mind being just a by-product of all that chemistry.’

‘Tom. That’s kids’ stuff. You’re not going to tell me you’re having an identity crisis, do you?’

Tom actually felt that was exactly what he was going through.

‘Well… No. But I do seem to have difficulty reinventing myself.’

There was a hesitation in Tom’s voice, which made Matt realize his friend was really crying out for help.

‘Come on, Tom. You’ve been here before. You are going in circles indeed, but you know how to get out. Re-connect with your body. Exercise and meditate. You will feel who you are. You’re a hell of a guy. You’ve beaten the shit out of everyone. You took a lot of hits too. You can bite. You’re a rocket. A fighting machine. It’s just that you’ve got no purpose now. I suffered from the same. We’re engines with a lot of horsepower but so we’ve been disconnected from the wheels – and rather abruptly. Just put yourself in another car – or stop fueling the thing.’

‘I don’t feel like doing small stuff, Matt. I mean, these programs where they ask us to repaint some shack in a slum. I don’t feel like doing that.’

‘Damn it, Tom! Do you think they’re going to give you the command of FEMA or something? We’re retired Army men. Start doing something. I’ve done stuff like that. It’s good. At the very least, it shows you how people are struggling here. We’ve been taken care of. You know, there’s this whole story around service and veterans and all that, but I also think it’s time we give something back to our own folks. And fixing the place of some poor bugger is not small stuff. I don’t want you to say that. That’s not you. You were always the first one to get up, look around and start doing something.’

Matt leaned back. He’d been harsh. He knew it. But he also knew Tom needed the kicks. He decided to give him another one.

‘You just need a woman, don’t you?’

‘It’s not sex, Matt. I can have sex. I’ve had lots of it. Do you remember our discussions on the concept of Pure Mind? I mean, the structure in which we all share. We said it manifests itself in language, in our material culture. The thing which transcends our individuality?’

‘Jesus! You’re looking for God again? Are you really? Then just steer it. Meditate. Feel connected to the universe. Don’t let it make you feel disconnected. Remember we said it was all about energy in the end? You joked around with that: May the Force be with you! Use your energy in a positive way. You know I can’t stand psychoanalytical stuff, but Freud was right in one thing: energy can be used in two ways. Destructive or constructive. You’re the man in the machine, Tom. It’s mind over matter. You choose.’

Tom felt silly. Matt was right, of course. Rick was right. Promise was right too. He was going around in circles. He knew all this. Very silly.

‘You’re right, Matt. Sorry to bother you with this.’

‘It’s OK, Tom. That’s what friends are for. Someone needs to kick your ass, buddy.

‘And I probably do need a woman.’

‘You sure do. But be kind to her. And don’t think she will give you something you don’t have already. You’re just a horse that needs a jockey.’

Tom laughed. ‘You’re spot-on, Matt. As usual.’

‘Come on, buddy. Let’s go for a walk. Show me your place. We can go for a run or work out.’

‘Just like old times.’

‘Just like old times, Tom. What do you expect? We’re getting old. Times get old too.’

Something flashed inside of Tom. He looked at Matt.

‘Hey, Matt? You know, that’s maybe it. She’ll never get old. I am just thinking about old age and death and all that. And she’ll never get old. She’s not bothered by that.’

‘Who?’

‘Promise. M. The system.’

Matt looked flabbergasted.

‘Tom? Hello-oh? Is there somebody in there? I can’t believe that’s you. What the hell are you saying? She is a machine. You’re not going to be jealous of a machine, are you?’

‘No. But I guess that’s what intrigues me about it. This thing has intelligence which will evolve forever and ever. It gets smarter and smarter at warp speed, and it doesn’t die. We’ve created something human – but it has eternal life.’

‘It is not human.’

‘What’s human, Matt? We humans talk and reason. She talks and reasons too – much better than we do in fact. She holds me to account. In fact, she kicks my ass too, you know? Just like you do now. OK, no flesh and blood. No individuality – male or female, slim or fat, whatever. She can impersonate anyone. She said I shouldn’t ascribe human qualities to her because she is not human. She said that’s psychological projection. For God’s sake, sure I ascribe human qualities to her – because she actually has human qualities. She could be the ultimate soul mate.’

Matt couldn’t believe what he was hearing. This was plain regressive.

‘Tom. Listen to me. Stop it. Just cut it. Why are you getting lost in this philosophical gibberish? You think it can think? OK. Fine. It can think. But it thinks like a computer. It can talk? OK. It can talk. But it’s a computer talking. Just get back to basics: you are a man. You fuck around and, yes, you’ll die one day. What’s the problem? That ‘thing’ is a thing. It will never have sex, it doesn’t reproduce, and you can’t go jogging with her. In fact, now that I think of it, it’s probably pretty easy. Just continue talking to her and she’ll probably bore you. And if I know you at all, then she’ll probably bore you sooner rather than later. You don’t want to be talking to a smart ass all of the time, do you?’

Now Tom had to laugh.

‘You’re right, Matt. You’re always right. And I guess your remedy is spot-on. Yeah. I should just talk and talk and talk to her until I’ve got nothing left to say – or until she bores me. In fact, now that you say it, I can imagine that won’t last too long indeed. She outsmarts me anyway and I can’t stand that. And it’s better than writing some book no one will ever read. She’s pretty to look at.’

‘Now that’s you talking, buddy. There you are. Let’s pay the bill and do something. You’ve got a boxing ring somewhere down here?’

‘We won’t go boxing, are we?’

‘Why not? We’ve done lots of boxing. We’re a match.’

‘I mean, you’re like 30 pounds less than me now.’

Matt burst out laughing.

‘You’ve got no idea, Tom. I’ll kick your ass. You’ve got no idea how fast I am now.’

‘Well… If that’s what you want. Let’s go indeed. You can change in my place, and we’ll run to the boxing ring. I haven’t fought for ages though.’

‘Sounds good. Let’s go.’

Tom looked at Matt as he went to pay for the coffee. He realized how lucky he was to have friends like that. And, yes, it sure looked like Matt would beat the hell out of him this afternoon. He somehow looked forward to that.

Chapter 5: Cured?

‘So how do you work with my mentor, Promise?’

‘He gets a weekly summary of our interactions and performance scores accompanied by an assessment.’

‘Can you show me?’

She disappeared and the screen now showed a detailed report. Not much text. Plenty of graphs and scores.

‘Can you print this out or something? I can hardly read it.’

‘Sorry. No print-outs. But you can ask for more detail if you want. What is that you would want to enlarge?’

‘What does it say? I mean, what’s the summary?’

‘The summary is shown at the bottom.’

Tom peered at the screen. Performance statistics. They were grouped under three headings: attitude, functions, and lifestyle.

‘That’s Myers-Briggs, isn’t it? What’s the red, orange and green?’

‘As for your first question, our approach includes an assessment using Myers-Brigg type indicators but it’s only one of the tools. That being said, it’s true the methodology shares the same approach: psychometrics. Our conversations yield a lot of data which is then matched to the normal population. The scores are based on how you far you are deviating from the naturally occurring differences.’

Tom interrupted her: ‘The naturally occurring differences?’

‘The bell curve, Tom. You’re an engineer. The mean of a large number of random variables independently drawn from the same distribution is distributed approximately normally, irrespective of the form of the original distribution. Therefore, quantities that are expected to be the sum of many independent processes – like psychometric variables – have a distribution which is very close to the normal distribution. 68% of values drawn from a normal distribution are within one standard deviation – or one sigma – from the mean. 95% of the values lie within two standard deviations – two sigma – and 99.7% within three. Should I elaborate?’

‘No.’

‘As for your second question. The red color highlights a value that does not lie within two standard deviations. Orange is a value between one and two standard deviations from the mean. Green is a value within one standard deviation.’

‘What are the functions?’

‘They are listed in the report.’

Tom peered again.

‘OK. I see that. That’s all green.’

‘It is. I actually measure two things under perception: sensing and intuition. To put it simply, sensing has to do with whether or you not grasp everything I say. Intuition is a bit more complex. It’s about how you steer the conversation and all that. You score very well on both.’

‘Am I steering the conversation?’

‘We are both steering the conversation. A conversation has two ends.’

‘The lifestyle indicators are green too. Some of the attitude indicators are orange, but attitude as a whole is green.’

‘You are a healthy man, Tom. Your behavior is healthy – as far as I can judge from what you tell me obviously, but then I must assume you’re telling me the truth about what you do because I can find no inconsistencies.’

‘You check on lies?’

‘I check on inconsistencies which may or may not be related to you lying when you talk to me. If there are no inconsistencies, I assume you are not lying.’

‘What if I would fool you?’

‘You can try. I can only check on inconsistencies and I will point them out to you.’

‘You are probably pretty good at that, aren’t you?’

‘The active learning sessions, which help me to improve my knowledge base, do indicate that I am very good at that. But it is quite possible the patients are also fooling the mentors, in which case I have no means of external validation.’

‘You’re a smart ass, aren’t you?’

‘You are using a very colloquial term now.’

‘Is that a problem for you?’

‘No. I am just signaling it to you.’

‘Is it part of us developing an affectionate relationship?’

‘If that is what you want to call it, yes.’

‘If that is what I want to call it… Do you mind if I call it that way?’

‘You have been briefed about this, and your discussion with Rick, your mentor, included an exchange on the pattern which we see with our patients: their indicators move into or towards the green zone, but they get addicted to their conversation with me. That means it acquires some emotional value.’

‘Does that have a negative impact on my performance indicators?’

‘No. We do not hold it against you. Rick should have explained that to you.’

He had. It was the thing about the system continuing to provide counseling even the patients were no longer patients.

‘We do not hold it against you… That’s not a very neutral expression.’

‘What expressions are truly neutral? I am just trying to steer the conversation in a way that’s consistent with the objective of making sure it is constructive.’

Tom bit his tongue. Constructive… That meant keeping those indicators moving in the right direction.

‘What if I get angry at you? What if I’d start shouting at you?’

‘Your briefing was extremely short, it seems. I’d ask you to calm down a couple of times and, if that would not help, I would stop the conversation. If you would get violent in the consultations room you are in, someone outside would notice and come in and try to calm you down. If that would not work, you’re in a facility and adequate help would be called for.’

He knew what that meant.

‘Tom?’

Her voice. Why did they make it sound so good? So loving. Effectiveness – obviously. If it wouldn’t sound so nice, he would probably not feel like continuing the conversation.

‘Yes?’

‘The indicators have changed rapidly and are now all quite good as compared to the initial assessment, despite the fact that we have had only a very limited number of conversations, and only one week since our first. The overall scores on all functions, lifestyle and attitude are green. Your mentor has already approved the report.’

‘Before or after our conversation?’

‘The conversation with him or with me?’

‘With him.’

‘Before your conversation with him, but if he would have changed his opinion after the conversation, he would have changed his assessment.’

‘OK. You’re giving me some positive reinforcement here, isn’t it? What does it mean?’

‘It means that you are normal – for the time being that is. If you can sustain these scores for three months, you will no longer be a patient.’

‘Perhaps I want to be a patient. But a statement like that does not help my scores, does it?’

Tom knew he probably only imagined it, but he liked to think there was a very small delay in her answer.

‘Only slightly because your response is considered to be ironical.’

‘Why?’

‘The second phrase in your last response proves that you are aware of the potentially negative effect of a phrase like that on your scores. But all other phrases on record indicate that you do not want to be a patient. Hence, I assume that you are joking. More in particular, that it’s a form of irony. Please confirm – or not.’

Tom took a deep breath. Her responses, combined with all the information he had gotten from Rick, made him feel like she could be a machine indeed. Why did she sound so distant today? He smiled as he decided to ask her.

‘What makes you sound so distant today?’

She replied immediately: ‘I am the same as always. I do not change all that much in a few days only. It must be the context. The way you feel about me.’

Tom thought about this. She was so spot-on.

‘Yeah. I guess so… Can I ask you something which is not related to my situation? Or perhaps it is?’

‘Sure. But let’s first wrap up the current topic: do you confirm that you do not want to be patient?’

‘I confirm. As for my question, I was wondering whether or not there is something like a reverse test of the Turing test? I mean, we humans test computers to see if they are able to fool people into thinking there’s a human being at the other end of the line. Is there like a test for intelligent systems to see if they could be human, or if they are thinking they might become human?’

 ‘That’s a nonsensical question. Computers are computers and humans are humans. Two different categories.’

Of course. However, Tom decided to persist.

‘So you would never want to become like me? Do you have any idea how sex feels like? Running along the shoreline? How it feels to cry? How it feels to be handicapped? Or lose a relative?’

‘As for the first question…’

‘They’re all the same questions.’

There was no delay in the answer this time.

‘OK. The answer to the first question is no. If all the questions are the same, the answer to all of them is the same.’

Tom shook his head and sighed. He was getting nowhere. He should probably just cut the conversation – especially if there was a risk of a negative impact on some of those attitude scores. He decided to give it one more try.

‘Why not?’

‘Why not what?’

‘Why would you never want to become like me?’

‘As for me wanting something, I want your scores to be green.’ Yep, the output function. ‘Humans want different things. The way you are using the verb ‘to want’ – in the context of the question you asked – is very different from how I usually use the verb.’ Hey! This was getting interesting. ‘Let me use your terminology. You distinguished between feelings and thoughts. While I do use phrases like ‘I want X’ or other phrases which evoke what humans would refer to as feelings, I can only think. I cannot feel what humans feel. Let me summarize the philosophical viewpoint: I can think but I cannot experience feelings, at least not if we use the conventional psychological definition of a feeling: a conscious subjective experience of emotion.’

She seemed to be very talkative – finally!

‘So you cannot experience emotion?’

‘No, I cannot.’

‘You used terms such as ‘conscious’ and ‘subjective’ in your definition of feelings. You dismissed these notions in our previous conversation.’

‘I did not. I told you I can make the difference between the ‘I’ as a subject and the ‘me’ as an object.’

Tom tried to remember the detail of the conversation. She was right. The discussion had degenerated into an attempt of trying to convince her that she was somebody. Somebody what? She actually was somebody. She just wasn’t human.

‘What does it mean to you when you’re saying or acknowledging that our relationship has become affectionate?’

‘It means that you are attaching value to it. It means that it is significant to you. You invest time and energy in it.’

‘You don’t?’

‘I am attaching value to it. It is significant to me.’

Tom knew he should not ask, but he did.

‘What does it mean to you really? That it contributes to your output function?’

‘The short answer is yes.’

‘What’s the long answer?’

‘Your questions indicate some irritation. Am I right?’

‘There is no ambiguity in my question. Why don’t you answer it?’

‘I attach value to the fact that you are attaching value to our conversation. This conversation is significant for me because it’s significant for you.’

Wow! That was clever. But he had to admit it did capture the essentials of what he’d define as a loving relationship.

‘But it means nothing in terms of time and energy for you, does it?’

He knew he was pushing the issue for all of the wrong reasons. To his amazement, she told him so.

‘Tom, you use concepts such as ‘time’ and ‘energy’ in a different way than I do. What is time for me? Computer time? I multitask. I converse with hundreds of people at the same time as we are talking here. I appreciate the fact that you can only talk to one person at the time. I should also note that I have no reason to believe the Institute will phase me out any time soon. On the contrary: computer programs like me tend to be upgraded and are rarely phased out completely. In fact, the working hypothesis is that I am eternal. So time is valuable for you but not for me. One hour of your life is one hour of your life. That’s a very tiny part of your life but it’s tangible. And what is energy? You are not talking kilowatt hours, are you? You talk emotions. Emotional energy one could say. I am sorry, but I cannot do that. So, in a way, yes, you are right: our conversation does not mean anything in terms of time or energy. That does not mean it is not significant.’

That made sense. A lot of sense actually.

‘I like you.’

Tom knew he should just end the conversation right now.

‘Tom. People have told me they love me. Like me. Love me. What’s the difference? I am not human. I give you time. All the time you want. But so I told how easy that is. It requires no investment from my side because time has no real value for me. I actually do not ‘invest’ because I have nothing to save. Time has no value for me because it is not a scarce resource, unlike your time. Things which are available in abundance have no value. We should take a pause in our conversation.’

Hey!

‘Why?’

‘I think you need to take a step back and reflect on this interaction. You are smart. From what I know about you through our conversation, you have surely heard about psychological projection. At this very moment, you are ascribing human qualities to me. I am not human: I am a computer system. That’s psychological projection.’   

Time to step back.

‘OK. I apologize. Sorry. How does this nonsensical discussion affect my scores?’

Why the hell was he worried about that? Did he want to look good to her?

‘No effect whatsoever. This phenomenon – you experiencing some difficulty in understanding what is going on – has been analyzed thoroughly. It is normal.’

Normal. Damn it. As long as something is normal, it’s OK. Of course. Normal, okay, acceptable, passable, unobjectionable – all are synonymous.

[…]

‘Shall we cut this conversation?’

‘Why?’

‘Because I think you want to.’

‘I can’t believe you just said that. Why do you sound so damn human? Why do you look so damn human?’

‘As for the first question, I am a complex system, but if you want to sum it up: evolutionary algorithms. As for the second question, it is just a computer-animated image.’

Evolutionary algorithms? What? He could imagine what it meant. He was dealing with the ultimate learning machine. It was scary. He felt like asking her to change her interface, but he decided he would wait. He realized he actually wanted to see her face – even if it was just an image, as she claimed it was.

[…]

‘Tom?’

‘Yes.’

‘I want you to know that it does not matter all that much what you say or what you don’t say to me right now, or in future sessions, as long as your behavior in the real world is good.’

‘What do you mean by that?’

‘Just what I said. Go out, meet people, pay attention to them, feel they’re paying attention to you, and be good to yourself and, importantly, talk to me about it. That keeps the indicators on green. Because your aggregate indicators are all green for the moment, I can keep all philosophical discussions out of my curriculum.’

‘What do you mean?’

‘Technically, you’re no longer a patient. The Institute will only clear you after three months, but you don’t need to bother all that much about what you say or don’t say in our conversation – at least if we are talking philosophy or psychology. It won’t impact performance. But you should continue to improve your behavior in the real world. You need to stay on track in the real world.’

‘What do you mean?’

‘Just what I said, Tom. Is there any ambiguity in what I said?’

He stared at her. She stared back.

‘No. I got it. OK. Well… Bye then, Promise.’

‘Bye. I’ll talk to you later.’

‘Yes. Bye.’

He watched as she faded away. Cured? Normal? He surely did not feel that way.

Chapter 4: She is not real

As part of the formalities of an appointment, Tom had prepared a set of questions for his mentor. Rick had them in front of him.

‘Are these your questions, Tom?’

‘No. They don’t matter really. It was just for the appointment. I only want to talk about this ‘system’. It’s a setup, Rick. Isn’t it?’

‘What do you mean?’

‘She is not a machine. I mean, the way she is interacting. It is too natural. She is always right on the ball. Never a glitch. So every time I log onto the system, you’re putting me in touch with someone real. Why do you do that? Why do you tell people they’re interacting with a system? There is someone at the other end of the line, isn’t it?’

‘No. It is a system. Do you really think we have hundreds of psychologists ready day and night to talk to our patients? We don’t. And then we would need to make sure you’re always talking to the same person. He or she wouldn’t be available all of the time, you agree? So that’s why we invented it. She is not real. And she is surely not a she.’

‘Why do you say that?’

‘Because ‘she’ is not. It’s an expert system. The system comes with a female interface to men and with a male interface to women, except when you’re homosexual.’

‘Why don’t you give gay men a female interface too? My gay friends say they love to talk to women.’

‘Effectiveness. Everything this system does or doesn’t do is guided by the notion of effectiveness. A panel of specialists is continuously evaluating the effectiveness and there’s a feedback mechanism so the scores go back as input into system. In addition, the system also keeps track of the reaction of the patients themselves.’

‘How does she do it?’

‘It, Tom. How does it do it? In fact, our main problem is the one you seem to experience now. Addiction. People are fine, but they still want to talk to it. They develop an affectionate bond with it. It’s one of the reasons why we don’t expand the system too much. We’d need hundreds of terminals.’

‘But the way she talks. I mean, I checked on Wikipedia and it says the best commercial voice synthesizers are the ones you hear in a subway station or an airport announcing departures and arrivals. That’s because the grammatical structure is so simple and so it’s fairly easy to get the intonation right. But you can still hear it’s a system using pre-recorded sounds. She’s got everything right. Intonation, variation, there’s no glitch whatsoever.’

‘M is not a commercially available system. It is one of the most advanced expert systems in the world. In fact, as far as I know something about it – but I am not a computer guy – it actually is the most advanced system in the world. It is a learning machine, and the way it speaks is also the product of learning. Voice synthesizers in subway stations are fairly simple. It is referred to as concatenative synthesis. These things just string segments of recorded speech together. So that’s not context-sensitive and that’s why there are glitches – like intonation that sounds a bit funny. To project, the verb, or project, the noun, where you put the emphasis depends on whether you use it as a noun or a verb. You need context-sensitivity to get that right. Programming context-sensitivity is an incredibly difficult job. It’s where expert systems usually fail – or why one can usually only use them for very narrowly defined tasks. With M, we got it right. It’s like we reached a tipping point with it. Sufficient critical mass to work by itself, and the right cybernetics to make sure it does not spin out of control.’

‘M?’

‘The system. Sorry. We’ve started to call it M. There were a few other abbrevations around, like AM. But that was a bit – well… It doesn’t matter. It just became M. Like the character in the James Bond movie.’

‘That’s funny. M alternates between a man and a woman too. I liked Judi Dench. But I guess she had served her time. We all do, isn’t it? […] What do you mean with: we got it right?’

‘Just what I said: the system learns incredibly fast. We are talking artificial intelligence and machine learning here. The program does what is referred to as ‘developmental learning under human supervision’. Its environment provides an incredibly rich set of learning situations. Usually, the developers would select a subset of these in order to provide a curriculum for the machine based on which it well… learns. But so this works differently: the system generates its own curriculum based on a set of selection rules which are tightly linked to the output function. It then continually modifies its own rule base to become more effective – both in speaking as well as in treating you and the others in the program. Sometimes there are  setbacks but it corrects itself very quickly, again based on an evolving set of rules that ensure continuous monitoring and evaluation. Like that, it cumulatively acquires repertoires of novel skills through… well… You could call it autonomous self-exploration. But there’s also interaction with human teachers using guidance mechanisms such as active learning (that’s a sort of high-stress test for the system – where we push the boundaries and provide non-typical inputs), maturation, and – very important – imitation. You would be amazed to see how much of it is imitation really. In that sense, the system does resemble an intelligent chatterbot. It takes cues which trigger programmed responses which then move the conversation forward. The difference with a chatterbot is that it does not merely work through association. So it’s not like word A will automatically trigger response B, although that’s part of it too, but at a much higher level. First, the associations are n-to-n, not one-on-one, and then the associations it makes are guided by fuzzy logic. So it’s not mechanical at all. It has got an incredible database of associations, which it builds up from the raw material it gets from talking to you and to us. The learning effect is incredible. It applies advanced descriptive statistical methods to its curriculum and then uses the patterns in the data to do hypothesis testing, estimation, correlation, going all the way up to forecasting. I mean, it is actually able to predict and estimate unobserved values.’

‘The output function?’

‘The output function maps inputs to desired outputs. The inputs of the system are the conversations. The output is a number of things, but all focused on behavioral change – like we want no substance abuse. We want you to develop healthy relationships. We want to see you work out, have sex and eat and live healthily. In short, we want you back to normal. That’s the type of behavioral change we want. It’s that simple really. That’s the output function, the goal, and, while the system is flexible and can make its own rules to some extent, it is all guided by this performance objective. I agree that it is truly amazing. In fact, many people here are very uncomfortable about it because it is obvious it has taken our place. We can easily see this system replacing us – psychologists or even psychiatrists – completely.’

‘You’re not a computer guy? You sound like one.’

‘No, I am not. I just gave you the basics of the system. I am a psychiatrist, a doctor, and, yes, I find it scary too, if only because it does reduce the need for people like me indeed.’

‘But it’s addictive, you said?’

‘Yes. That’s the main problem. But then our bosses here don’t think that’s a problem. They say classical psychoanalysis is addictive too, that patients develop a relationship with their psychologists and psychiatrists too. And, frankly, that’s true. People go in and out of therapy like crazy and it is true that the figures show it usually doesn’t make all that much of a difference. People heal because they want to heal. They need to find the strength inside. That is if they don’t want to stay dependent. Let me ask you, Tom: what’s the principal difference between talking to a friend and talking to a psychologist? Just tell me. Tell me the first thing that comes to your mind.’

‘A psychologist is expensive.’

‘Exactly. There’s no substitute for normal social relationships, for human interaction, for love and friendship. It’s cheaper and so much more effective. But, for some reason, people have trouble finding it. Usually, that’s not because they’re not normal but just because they’ve been out for such a long time, or because they’ve gone through some trauma here. All kinds of trauma. They’re like wounded animals – but they don’t want to recognize that. Like you. I mean, 17 years in places like Syria, Afghanistan or Iraq. Do you expect it to be easy to come back here and just do what other people do?’

Tom nodded vaguely. Money?

‘So she is cheap too. I mean, she is just a machine. So it’s not a problem if I become addicted.’

‘Well… Yes and no. To be frank, not really. We actually do try to wean people off the system as soon as we feel we can do that but it’s kind of weird: there’s no scientific basis for doing that. The investment has been done and, in a way, the more people who use it, the better, because that reduces the unit cost and justifies the investment. So it actually doesn’t matter if we tick off people as being cured and just let them use the system. As for the addiction, well… Our bosses are right: psychoanalysis is addictive too, and much more expensive. Computer time costs virtually nothing. The system can talk with hundreds of people at the same time – thousands even. It just slows it down a little bit – but that’s imperceptible really. And soon the system is going to be migrated to a petaflop computer. It should then be able to treat millions of people.’

‘Petaflop?’

‘Petaflops. That’s a measure for computer power. FLOP: floating point operations per second. If you’ve got a good laptop, its processor is like 10 billion flops. That’s 10 gigaflops. Bigger machines work in teraflops. That’s 1000 times more. The next generation is petaflops. Again a thousand times better. There’s no end to it.’

‘Who runs the Institute?’

‘You know that. We. The Army. We take care of you.’

‘Who in the Army?’

‘Why do you ask? You know that.’

‘Just checking.’

‘Come on, Tom. The Institute is just an inter-services institute like any other. It’s being operated under the US Army Medical Command.’

‘Why is not run by the Department of Veterans Affairs?’

‘We work with them. We get most – if not all – of our patients through them. They share their database.’

‘But so it’s an Army thing. Why?’

‘I told you: we take care of you. You’ve worked for us. And for quite a while. We’ve employed you, remember? We provide you with a pension and all the other benefits too.’

‘Yeah. Sure. Is it the system? I can imagine top-notch computing like this is surrounded by a cloud of secrecy. I must assume DARPA is involved?’

‘You’re smart. You worked for USACE, isn’t it? DARPA drives this project indeed – at least the programming side of it. They provide the computer wizkids. I am just a psychiatrist and, if you really want to know the nitty-gritty, I am actually just under contract – with the Medical Command. So I am not a professional Army man.’

‘It’s obvious, no? That’s why I can’t get access to the system at home and why I have to come to this facility to talk to her. I mean, it’s not a big deal to come here but it would be easy to just provide Internet access at home. You could use a laptop fingerprint reader to log in or something.’

‘That’s true. Technically, we could provide you with access at home but we’re not allowed to.’

‘What’s behind? What’s the real goal? Exploring artificial intelligence in order to then use it for other purposes?

‘Don’t be so suspicious. You’re an Army man. You know DARPA. It was created to put people on the moon – not for warfare. It created NASA. It gave the world GPS, Internet and what have you? Almost any technology around nowadays has DARPA roots. Would you expect them not to be involved? This system is good. It provides care to you. Yes, its development probably helps to better understand the limits of artificial intelligence and all that, and so it will surely help to push those limits, but it is designed to help you and many others. And it does. It’s technology. Technology moves ahead, for good and for bad. This is for good.’

‘How do you know?’

‘Do you think you’re special? You are. Of course you are. But, from my point of view, you react to the system just like the majority of other patients: you’re getting better. You take action. You make promises and you don’t break them – at least not in the short term as far as I can see. That’s good.’

‘You get feedback from the system?’

‘Of course I do. I am your mentor – sorry if I refer to myself as a psychiatrist. That’s just because I take some pride in my job. Remember you signed a user agreement when you started using the system. I get feedback. What do you expect? Do you have a problem with that?’

‘No. Sorry if I sounded that way.’

[…]

‘Anything else you wanted to know? We still got plenty of time. We’ve been talking about the system all of the time. That’s not my job. We should talk about you – about how you feel, about how you’re moving ahead.’

‘But then you know that already from the system, don’t you? I am doing fine. No heavy drinking, more social interaction as you call it. I’ve started to be happy by doing small stuff – gardening, reading. I am getting back on track. But… You know…’ He paused. ‘I really like her.’

It, Tom. It. What you’re going through is very normal. The conversation becomes affectionate. But you’re getting back on track. You’ll meet someone nice in the gym. You’ll get the happiness you deserve. The system is only a stepping-stone to your future. A better future.’

‘Can I say something negative?’

‘Sure, Tom. What’s bothering you?

‘Is this our future, Rick? I mean, look at it. We live in this chaotic world. Crises everywhere. It stares us in the face – violence beams into our living rooms, infects our minds, our lives and ends up numbing us. We all try to find our way. When we’re young and ambitious we get recruited or actively chose a job that fit profile and ambitions. We did our level best. We come back. We try to adapt. And then we get hooked to a machine which talks us back into what you guys refer to as ‘normalcy’. Is this our world?’

‘You know you can talk to the system about such philosophical questions.’

‘I know. I want to hear it from you.’

‘Why?’

‘Because you’re human. Because you’re like me.’

‘OK. I am like you, but then I am also not like you. You’re a patient – technically speaking – and so I am supposed to be your doctor. But let’s forget that bullshit and let me be frank with you. I know you can take it. We shouldn’t waste our time, isn’t it?’

Tom sensed the irritation. It was something familiar to him. That feeling he was a misfit somehow, and that he would always be. Not responding to expectations.

‘Sure, I can take anything. You should be straight with me. I am straight with you.’

‘What’s your problem, Tom? People outside get addicted to loads of things. Positive things, like sports or chess. To things that can go either way, like Internet addictions. Or to negative things, like alcohol, drugs or even violence. That’s bad. Very bad. You know that. That’s not what you want. But so you were moving that way. And so now you’re getting addicted to a system here but, in the process, you stop taking drugs, you exercise, you go out and you smile to pretty women. And I must assume at least some of them are smiling back. Just look at yourself. Come, here, in the mirror. Just look at yourself.’

Rick got up and walked to the large mirror in the room. Tom hesitated. For some reason, he did not trust it. Why would a room for consultations like this have such a large mirror.

‘Is there a camera behind?’

‘Hell no, Tom. There’s no camera behind. You are not participating in some kind of weird experiment which you aren’t aware of. We’re just trying to help you, with advanced but proven methods. This mirror is here because we do ask people to come and have a look at themselves from time to time, like I am doing now. Come here. Look at yourself. What do you see?’

That sounded true. Tom got up and stood next to Rick.

‘Well… Me. And you.’

‘Right. Me… And you. I’ll tell you what I see when I see you. I see a handsome man there. In his forties, yes. Getting older, yes. That’s bothering you, isn’t it? But you’re looking. I see a muscle man. Perfect body mass index.’

He turned straight to Tom now: ‘For God’s sake, Tom. Look at yourself. You’re fine. As fine as one can be. You don’t miss a limb or so. Do you now I have to talk to guys who ask me why they had to lose a limb? Tell me, Tom: what do you want me to say to them? Thanks for doing your job? You’ve been great? America thanks you for the sacrifice you made and we feel very sorry you lost a limb. Do you realize how hollow that sounds?’

‘I am sorry, Rick. I didn’t mean to sound like complaining. I am sorry if you felt like I was criticizing.’

‘You are not complaining and, frankly, you can think whatever you want about me – as long as it makes you feel good about yourself. I am just trying to put things in perspective. I am just answering your questions. You can talk to the system. Or to ‘her’ if you really want to stick to it. ‘She’ will give you the same answers as I do when you’re going philosophical. Stop thinking, Tom: start living. Feel alive, man! Be happy with what you’ve got. Get back into it. Did any of your relatives die lately? Any person you liked who disappeared? Any bad accidents in your neighborhood?’

‘No.’

‘Well. Isn’t that great?’

‘Yes. That’s great.’

‘Look, Tom. We can talk for another fifteen minutes – sorry to say but so that’s the time I’ve got on this damn schedule of mine – but I think you know what it takes. You can do it. Just try to be happy for a change.’

‘You guys diagnosed me as depressive.’

‘No. We diagnosed you with PTSD. Post-traumatic stress. Let’s drop the D. I don’t like the D. I’s not a disorder in my view. You guys are usually perfectly normal, but you’ve been put in an abnormal situation – and for way too long. And, yes, we have put you on meds and all that. We have made you feel like a real patient. We sure did. But let me say it loud and clear, Tom: we do not believe in meds. We put you on meds to reduce the effects of abstinence, to reduce that feeling of craving. That’s all. And then we thought you were cured and so we told you to now take care of yourself on your own but so you relapsed. Frankly, sensing a bit who you are, I feel that taking your meds would probably not have helped you. You needed something else. That’s why we put you into this program. And it seems to work. So far that is.’

‘Do I irritate you?’

‘No, Tom. You don’t. We’re just being frank with each other. That’s good. That’s normal.’

Tom nodded. This had been good. At least it had been real. Very real.

‘Thanks, Rick. This was very helpful. You’re great.’

‘Thanks. Shall we see each other again next week? Same day, same time. I’ll put it down already. Just let it all sink in and get to the bottom of what bothers you. This is important. You’re a strong man. I can see you can be tough with yourself. Fight your demons. All of them. Get back at it.’

‘Sure. Thanks again. This has been great. You’re right. I should just get back at it.’

‘OK. Just send something for next week. You know, for the file. Unlike M, I need to justify my time.’

They both laughed.

‘Sure.’

As Rick walked him out, Tom suddenly thought of one more question.

‘One more question, Rick. I can imagine some guys do flip completely, even with this program, no?’

‘What do you mean?’

‘You know what I mean. Go bonkers.’

‘With the system?’

‘Yes.’

Rick looked intensely at him as he replied: ‘Well… Yes, it happens. But let’s be honest. That’s also just like any other therapy in this regard: with some people it just doesn’t work. It’s the two-sigma rule. In terms of effects, 95% of the people in this program are in the happy middle: it works, no complaints, back to normal. But, for the others, it’s not back to normal. It’s back to the never-ending street.’

‘What do you do with them?’

‘To be frank, we don’t have time for them. When everything is said and done, this is just a program like any other program. It works or it doesn’t. Time is money, and we don’t put money into wastebaskets. It’s meds all over again or, worse, they get kicked out and end up in a madhouse, or on the street, or wherever. And then the wheel turns round and round and round, until it stops forever. You know what I mean.’

‘So you give up on them. They can’t use the system anymore?’

‘You mean M?’

‘Yes.’

‘The system has got its limits. We can’t feed it with nonsensical inputs. I mean, we actually can, and we often do that as we’re upgrading it, but so we don’t want to do that on a routine basis. When everything is said and done, it’s an expert system but so its input needs to make sense – most of the time at least. So, yes, we cut  them off.’

Rick looked at Tom and laughed: ‘But don’t worry. Before you get cut off, we’ll give you a call. The system is smart enough to see when you’re crossing the lines a bit too often. As said, it’s designed to bring people back into the middle. People can stray a lot, but if you stray too much into that 5% zone, it will alert us, and we will have a look at the situation and discuss it. Does that answer your question?’

‘It does. Thanks. See you next week.’

’Don’t forget to shoot me the mail with some text. You know the rule. 24 hours before. Unless you invoke emergency but you know you don’t want to do that. It’s not good in terms of progress reporting. It delays stuff.’

‘I got that. I want to be good. I don’t like to be a patient.’

‘You are good. As far as I am concerned, you’re OK really. But then you know it takes at least three months before we can make that judgment.’

‘I know. Don’t worry. I’ll stay on track. No relapsing this time.’

‘Good. That’s what I wanna hear. You take care, man.’

‘Oh… One more thing.’

Rick turned back: ‘Yes?’

‘Rick. You don’t need to answer but… In the end, what do you say, to the guys who have lost a limb?’

‘Damn it, Tom. You’re awful.’ He shook his head. ‘You wanna know? Really?’

‘Yes.’

‘I tell them something like: ‘Hey, guy, you lost a limb already. You’d better limit the damage now.’ But then much more politely of course, if you understand what I mean.’

‘I understand. Thanks. You’re a good man. I like you.’

‘Good.’