On Free Will III: Outsmarting the Prediction Machine
Clearly, what is immaterial in the human mind can influence the physical world, or our acts of will and understanding would be without effect. If our will is free these physical effects are not wholly predictable.
A merchant in Baghdad sent his servant to the market. The servant returned, trembling and frightened. The servant told the merchant, "I was jostled in the market, turned around, and saw Death."
"Death made a threatening gesture, and I fled in terror. May I please borrow your horse? I can leave Baghdad and ride to Samarra, where Death will not find me."
The master lent his horse to the servant, who rode away, to Samarra.
Later the merchant went to the market, and saw Death in the crowd. "Why did you threaten my servant?" he asked.
Death replied, "I did not threaten your servant. It was merely that I was surprised to see him here in Baghdad, for I have an appointment with him tonight in Samarra."
We have all heard stories such as the parable of Death in Samarra, or the tragedy of Oedipus, who was fated to kill his father and marry his mother and went on to do precisely that, despite his knowledge of the prophecy and his attempts to forestall it. These stories frighten us, and rightly so, by raising the specter of a future that is fixed, and that we each have an inexorable fate that, no matter how hard we run from, we only reach all the sooner.
Advocates of dualism sometimes claim that something very like this must be the case if materialism is true. After all, the argument goes, if we have no supernatural souls exempt from the principle of cause and effect, then our brains must be nothing but machines obeying the laws of physics, and if that is the case, then however complicated they are, their operation can in principle be predicted. Given complete knowledge of the state of the world, plus knowledge of the laws of physics governing how that state evolves, one could predict events arbitrarily far into the future. In this view our future behavior would be just as predictable in principle as the landing spot of a baseball thrown at a certain angle upward with a certain speed. And isn't that a terribly gloomy, disheartening vision? Don't we want to be more than thrown baseballs?
This intuition pump is what Daniel Dennett, in his book Elbow Room, calls the "Malevolent Mindreader", an entity who always knows in advance exactly what you are going to do and uses that knowledge to foil you every time. Playing chess against a Malevolent Mindreader is a doomed proposition, since he knows exactly how the game will end. A variant is the "Nefarious Neurosurgeon", who uses his knowledge not just to predict but actively to control you, typically by surreptitiously implanting electrodes into your brain that cause you to think, believe and decide just as he wishes, while leaving you the illusion of being in control. Are these sinister figures really waiting in the wings for us if materialism is true?
Let us explore this proposition in more detail with a thought experiment. Suppose that it is the year 2096, and the mad scientists at the Materialism Stereotypes Institute, thanks to a large government grant, are about to build the world's first fully functional Prediction Machine. This machine is an extraordinarily sophisticated piece of hardware, possessing a wide variety of sensors that allow it to gather every conceivable piece of data about its environment and a computer brain programmed with all the laws of physics. The purpose of the Prediction Machine is to survey in complete detail the state of a person's brain, then extrapolate that information to infallibly predict that person's future actions, thus proving that we are nothing more than complicated but deterministic machines ourselves. Later versions may be able to predict actions days or years in advance, to make Oedipus or Death-in-Samarra scenarios possible, but PM Mark I is merely a proof of concept and will only predict decisions a few minutes into the future.
Nevertheless, this is enough to prove the point the mad scientists of the MSI are trying to establish. To demonstrate their sinister powers, they recruit test subjects who agree to play several rounds of rock-paper-scissors with the Prediction Machine. If their hypothesis is correct and our actions are predictable, PM Mark I will always win, since it will infallibly anticipate what sign a person will throw and then throw the correct sign that beats that one.
The first test subject is hooked up to the Prediction Machine, which scans his brain and makes its prediction. Then, on a count of three, they each throw their signs simultaneously.
The human throws rock. The Prediction Machine throws paper. The mad scientists grin and exchange high-fives.
But even at the Materialism Stereotypes Institute, they are well aware that repeatability is vital for science. In order to prove that the Prediction Machine's victory was not due to chance, they continue the test.
In the second round, the human throws scissors. The Prediction Machine throws scissors as well.
The mad scientists' grins fade. They put the test on hold, pop open the machine's casing, check all its connections and recalibrate its software to make sure nothing has gone wrong. But it seems to be in perfect working order, so they dismiss the result as a fluke and continue the test.
In the third round, the human throws rock. The Prediction Machine throws scissors.
Seeing their chances at a Nobel Prize slipping away, the mad scientists give their machine the most thorough going-over they possibly can, but quickly discover that their efforts are to no avail. They are absolutely certain that there is nothing wrong with the machine, and yet it cannot win any more often than would be expected by chance. The more trials they run, the clearer this becomes. After all the millions of dollars and decades of research that went into building it, the Prediction Machine is no more accurate than a device that chooses signs at random. Like John Henry defeating the pile-driver, humanity has triumphed over the machine, and, it seems, retained its free will.
Something has gone wrong here, but what? Why doesn't the Prediction Machine work?
To see what the flaw is, consider a similar project: the quest to build a Prediction Machine for the stock market. Taking into account the current prices and past trends of every publicly traded stock, this machine would infallibly predict which stocks would rise and which would fall, allowing anyone who used it to effortlessly make a killing.
Such a plan could never work, and a moment's thought will reveal why: the mere existence of this machine would itself be an influence on the stock market that would have to be taken into account. If you used the knowledge it gave you to buy stocks, this will be a new causal factor on the market which would cause other traders to react differently than they might otherwise have done. In order for its original prediction to be accurate, the machine would have to predict this and incorporate that knowledge into its forecast. But that would change what the original forecast was going to be, thus changing what stocks you would buy in response, thus changing other traders' reactions, thus forcing the machine to alter its original prediction yet again... and so on, in a recursive, endless loop, as the machine tried in vain to construct an accurate model of the world that included itself in that model. To include itself in its own model, it would have to include itself containing that model in its model, which would have to include itself containing that model containing that model in its model, in an infinite iteration. This is impossible, clearly. It would be like trying to store a box inside itself.
Now, there is one way out of this paradox: use the machine to predict the stock market, but do not act on its predictions in any way, not even to tell anyone else what they are. If this were the case, the machine's predictions would not alter the state of the market and thus force it to recalibrate its own prediction. You would be the proverbial observer behind glass, always knowing in advance what the market would do, but utterly unable to act on that knowledge in any way at all, because the mere attempt to do so would render it untrue. In such a scenario, a Malevolent Mindreader could theoretically exist, but would be forever cut off from the rest of the world, unable to interact, an observer only.
But when it comes to the human brain, not even this strict separation can be maintained. When it comes to the stock market, one can learn information about a particular stock without actually affecting it. But imagine if this were not the case. Imagine if the only way to learn a stock's price was to buy a share of it. Then the machine would have to take its own existence into account, in which case infallible prediction truly would be impossible. By terminating the infinite recursion of self-prediction at some arbitrary depth, one could force the machine to make a prediction, but it could never be more than an educated guess, and would never be the dreaded statement of unavoidable destiny enshrined in tragic literature.
Human brains are like the latter, not the former, type of stock market. To exactly describe the state of a person's brain at a given time, one would have to measure the exact electrical potential of each neuron, the exact number of neurotransmitter molecules released from every synapse, and so on. But acquiring this level of detail would require an extremely sophisticated scan of the brain, down to the level of individual molecules.
In some worlds it might be possible to do this without changing the brain's state in any way, but ours is not such a world. In our world the theory of quantum mechanics reigns, which says, among other things, that all events have a component of irreducible chance. QM has been used and misused in many ways when it comes to free will, but it has one uncontroversial implication that is relevant here. That implication is that the mere act of observing something unavoidably changes it. For example, to see something, you have to bounce photons off it. On the scale of macroscopic objects such as baseballs, the influence of this is so small as to not significantly affect the accuracy of our predictions. But on the scale of the very small, such as an atom or a molecule, an impinging photon represents a significant disturbance indeed - and scanning the brain in the level of detail that the Prediction Machine needs requires us to descend to this level. Merely by scanning a person's brain, the Prediction Machine inevitably changes that brain's state, forcing it to take its own influence into consideration when making its prediction; and this leads straight back to the problem of infinite recursion that reared its head when trying to predict the fluctuations of the stock market. This is why the quest of the mad scientists at the Materialism Stereotypes Institute was doomed to failure from the beginning. If they had only listened to us compatibilists, we could have told them that in advance and saved them a lot of work.
What is the point of all this? Advocates of dualist free will claim that theirs is a model where even complete knowledge of the state of the universe at time T would not make it possible to infallibly predict what a person would do at time T+1. But we have just seen that materialism has exactly the same consequence. And this means that our actions are genuinely not determined in the way that the flight of a baseball is.
Some readers may consider this a logical sleight of hand. Even if no one could possibly have known in advance that you would do X, the argument goes, doesn't that knowledge still exist "somewhere", in some hidden dimension of determinism? I urge them to reject this conclusion. The separation of "X was determined to happen" and "It was knowable in advance that X would happen" is illicit, because there is, by definition, no conceivable test that could differentiate between them; no experiment could show that one holds true but not the other. Therefore, despite the apparent difference in wording, these two propositions express exactly the same idea. If one is false, so is the other.
We can now see why there is no need to fear that we live in the world of Oedipus. His is a world of fatalism, a world of Prediction Machines, where certain events will happen regardless of what you choose. But compatibilism is not fatalism. Compatibilism means that certain events happen because of your choices. In a fatalist world you cannot use knowledge of the future to alter the future, but in a compatibilist world you can, because the mere act of providing information creates a new causal factor that alters what would have happened in the absence of that information.
We need not fear these lurking intuition pumps; on closer inspection, they simply evaporate. The Malevolent Mindreader cannot exist. Neither can his comrade the Nefarious Neuroscientist, because precisely controlling one's will through external influence would require perfect knowledge of the state of one's mind to know how it had to be changed. Like many nightmares, these two seem tangible only as long as they keep to the shadows. Daylight reveals that they are without substance.
Next: What does it mean to make a choice? Can human beings be responsible for what they choose in a universe where every event is subject to the law of cause and effect, and could we have chosen differently in any situation? Stay tuned...
Other posts in this series:
On Free Will II: Overthrowing Dualism
For most of human history, philosophers have believed that only the possession of an immaterial soul could confer free will on human beings. (There have been exceptions: the ancient Greek Stoics, for example.) This idea has fallen somewhat out of favor, but there are many theists who still hold to it. They are willing to concede that the universe we live in is an interwoven tapestry of cause and effect, but insist that we are special somehow, that we are an exemption.
If free will truly requires a magical power to exempt ourselves from the fabric of causality, the prospects of retaining it in a natural world seem dim indeed. But that consideration aside, this idea must still be considered on its own merits. The first and most important question is, how does this doctrine work? According to dualism, why do we make the decisions we do?
The classic theologian's answer is that the soul in some way inhabits our minds, directing the operation of our bodies. (Descartes, for example, thought that the interface point between soul and body was the pineal gland of the brain.) But on a closer look, this "answer" is really not an answer, because it does not explain anything at all. If there is a tiny homunculus inside each person's head, watching the input from the eyes on a screen and making the body move by pulling strings (the idea of the "ghost in the machine", or what Daniel Dennett disparagingly calls the "Cartesian theater"), then how does that homunculus' mind work? What makes it think and decide? If we cannot explain the operation of that complex organ called our mind in terms of less complex components, then there is a looming problem of infinite regression.
Furthermore, this idea leaves unanswered the question: If the soul controls all the important functions of consciousness, if all this information processing takes place elsewhere, then what is the brain for? If its only role were to control body functions, it would seem that we could do perfectly well with just a brainstem. Why do we have this massively enlarged cortex if it plays no role in our consciousness? (The proper rebuttal to the old saw that "we only use 10% of our brains" is this: Have you ever heard of someone who was shot in the head, but survived with no significant deficit because the bullet only damaged the 90% of his brain he wasn't using?)
Another dualist attempt to explain the source of free will is the doctrine of "agent causation". By this account, causes produce effects, which themselves can be causes in turn; but agents such as human beings also give rise to effects, and agents are not themselves caused by anything. As the philosopher Roderick Chisholm put it, "we have a prerogative which some would attribute only to God: each of us when we act is a prime mover unmoved. In doing what we do, we cause certain things to happen, and nothing - or no one - causes us to cause those events to happen."
Many kinds of metaphysics are obscure, but this kind seems to be intentionally obscure. Advocates of agent causation would have us believe that there simply is no cause for any of our actions - not even a reason. We just act, and the question of why has no answer. If this is the case, why are our actions so predictable? If there is no cause or reason why a person would, for example, go to work one day rather than steal a car and go on a multi-state crime spree, why do most of us so consistently do the former and not the latter? If there are no causes or reasons pushing us in one direction rather than another, every possible action should be equally probable, but this is clearly not the case.
These ideas are not solutions to the problem; they are a refusal to face the problem. And furthermore, they are not true. We know beyond any reasonable doubt that we are not unmoved movers, that we are not pure rational agents who make their decisions in a higher plane removed from earthly cause and effect.
Our thoughts and behavior demonstrably depend on the physical operation of our brains, and can be changed by physical causes that affect the way our brains work. Anesthetics shut off our consciousness, while stimulants accelerate it. Psychoactive drugs can cause or suppress hallucinations, provoke or quiet anxiety and paranoia, and affect mood, behavior and judgment. Certain genes are strongly implicated in the origin of mental disorders such as schizophrenia and OCD. Certain specific types of brain damage cause specific and predictable alterations in the way we believe, think, and decide. (For many strange and fascinating examples of this, see "A Ghost in the Machine".) Even the dualists' last redoubt, qualia, can be altered or eliminated by physical changes to the structure of the brain.
Dualism is a futile doctrine. It does not explain free will at all, rather seeks to avoid the problem by resorting to mystery; and it is contradicted by the facts. If we are to preserve the belief in free will, we need a better way to account for it - a naturalistic way, one that does not depend on a god of the gaps or on something remaining forever mysterious to us.
Some commentators, and not just dualists, would claim that this is impossible. These people believe that free will is an impossible fantasy, and now that science has revealed we live in a naturalistic universe, we should accept that we have no free will and be done with it. I do not agree.
True, I am not speaking of the mystical, dualist free will, where human beings can float free of causality and make decisions supernaturally exempt from external influence. We have every good reason to believe that that sort of free will is impossible. However, I believe it is possible to give a natural explanation of free will, one that preserves the qualities we value most - unpredictability, choice, and moral responsibility. The next three posts in this series will tackle each of these in turn.
Other posts in this series:
On Free Will I: Executive Summary
Over the ages, the question of whether we have free will has engaged, confronted, and puzzled philosophers probably more than any other issue, and untold numbers of papers, conferences, books and debates have been expended on tackling it. It is no surprise that so much philosophical ink has been spilled on this question, because it is in a sense the question upon which all other questions depend. If there is no free will, and thus no moral responsibility, it seems we might as well shut down the churches, throw open the prisons, and eat, drink and be merry, for tomorrow we die. (Or not. After all, if there is no free will, the concept of what we might or should do is meaningless; in such case, no one could do anything other than what they actually do.)
However, despite the lack of anything approaching consensus, through the ages one position has received considerable acclaim. That position is dualism, the belief that there is some kind of magical, irreducible mind-stuff, distinct from the matter and energy we encounter every day, that animates us and confers upon us our rational and sensory faculties. Advocates of this mind-stuff usually assert that only by this means could we hope to possess free will, that any account of how our minds work that does not include something other than matter and energy obeying physical laws must perforce deny some important or desirable quality of our nature.
This is the first post in a five-part series that will critically analyze this hoary wisdom, showing that it is well-intentioned but mistaken. Mind-stuff is not only unsupported by the evidence, it is unnecessary. Free will is not as difficult to come by as the dualists think. In fact, free will is completely compatible with materialism, the position that all that exists is made of matter and energy, and for this reason the philosophical stance combining the two is usually called compatibilism. That is the position I will be defending.
But before we can lay a foundation, we must first clear away the debris, and so the next post in this series will critically examine and debunk dualism in all its varieties. What these doctrines all amount to is a plea that we not look too closely, an attempt to mark certain areas as off-limits for philosophical inquiry, lest we incautiously rush in and find something that Man Was Not Meant to Know, something that will forever shatter our tenuous but necessary belief in choice and moral responsibility and send us shrieking into the night.
I reject this cowardice disguised as modesty. There may be truths that humanity was not meant to know, but I have yet to come across any, and I see no reason to believe that this is one of them. I have always found it a far superior plan to first find out what is true and then build our happiness around that, rather than deciding what we want to be true and then living as if it was. The latter course of action almost invariably brings catastrophe, when our illusions collide with a Nature that cannot be mocked; the former course gives just as much if not more potential for happiness, I have found, and often brings with it unexpected benefits as well. If some philosophers choose to leave these areas unplumbed, then I will simply have to take the candle of inquiry from them and stride full into the darkness to see what there is to be found; and this is just what I intend to do.
Once the foundation is set, Parts III, IV and V will build upon it. Each of these three will confront a perennial fear raised by dualists against compatibilism, dismantle it, and offer a positive alternative that shows how a purely physical account of the mind can still provide the important aspects of free will. These three hobgoblins are as follows:
- If materialism is true, there is no free will because there is no genuine unpredictability to our thoughts and deeds. A sufficiently knowledgeable observer could perfectly predict - or perfectly control - our every action.
- If materialism is true, there is no free will because there is no genuine choice. Whatever we do, we could not have done otherwise, and there is only one possible future regardless of what any of us do.
- If materialism is true, there is no free will because there is no moral responsibility. Each person's actions are determined by forces beyond their control, and thus it is pointless to blame or praise, reward or punish, anyone for any act.
So that my readers may breathe a sigh of relief, I will reveal my conclusions ahead of time. Yes, we do have free will; we do have the ability to make genuine and meaningful choices, we genuinely are not fully predictable by any outside agent, and we genuinely are morally responsible for our choices, and none of these things require the existence of a Cartesian theater or a dualist ghost in the machine. In this case at least, the common beliefs about free will are happily true. All I intend to do is show how such phenomena could arise in the clear light of reason, without recourse to miracle or mysticism; and though this quest may require us to discard a few common assumptions, we will see that they are not and could not be of any value in giving us the kind of free will we think we want.
(Note: I am greatly indebted to Daniel Dennett, whose laudably fearless works on this topic - particularly Elbow Room and Freedom Evolves - gave me the confidence to plunge into it for myself, and helped me enormously to clarify my own thoughts. Many of the conclusions in what follows were arrived at with his guidance.)
Other posts in this series:
Book Review: Darwin's Dangerous Idea
Daniel Dennett is one of my favorite philosophers. Few write with his clarity or liveliness, and the topics to which he turns his attention - evolution, religion, free will, the human mind - fall squarely within my area of interest. His explanations are often brilliantly clever, and his conclusions are ones I can usually agree with. In Darwin's Dangerous Idea, his provocative thesis is that Charles Darwin's idea of modification by natural selection, which he calls "the single best idea anyone has ever had" (p.21), is like a "universal acid" that spreads through every field of science and leaves them all changed in its wake. Before Darwin, Dennett writes, the existence of an irreducible intelligent Mind was believed to be the only way to create anything. Even the arch-skeptic David Hume saw no alternative to this conclusion. But we have learned of another way, and ever since, science has been reverberating with its implications.
However, the downside of Darwin's dangerous idea is that it seems too mechanical, too impersonal, to have given rise to many of the things we value highly - such as religion, or human consciousness, or our moral sense. Therefore, ever since it was first proposed, there have been scientists and philosophers who have attempted to construct levees to hold back this universal acid, to prevent it from reaching their most cherished convictions. In Dennett's terminology, evolution by natural selection is a crane - a powerful but mechanical tool that can be used to build ever greater things, including even larger cranes. The resisters of this idea instead want a skyhook - a magical attachment point, floating free and unbound above the earth, from which their valued principles can be hung. The idea of miraculous creation by God, for example, is a skyhook, as are less mystical ideas that would nevertheless create a boundary past which evolution could not go.
Much of Dennett's book is targeted at those whom, in his view, are skyhook-seekers, engaging in efforts to keep natural selection from reaching what seems most valuable about us. Some modern intellectuals whom he accuses of this sin include Stephen Jay Gould, Noam Chomsky, John Searle and Roger Penrose. Others, such as Stuart Kauffman, espouse positions that at first seem to contradict Darwin's dangerous idea, but that on a closer look can easily be folded into it as a special case.
It is important to note that this book cannot be read in isolation. Dennett's arguments are often rather technical, and cannot be understood without also understanding the position and arguments of the people he attacks. I lack the technical expertise required to tell if all of Dennett's arguments are correct, though it did seem to my inexpert eye that he scores at least a few strong points (to be fair, some of his targets, such as Gould, fought back spiritedly, and I think the ultimate truth incorporates parts of each side's reasoning). I have not reached a firm conclusion regarding Dennett's strong adaptationist viewpoint, but I do think it has much to recommend it. I appreciate that he recognizes the necessity of giving testable evolutionary explanations, rather than unsupported "just-so" stories.
Dennett's ultimate conclusion is that there is nothing to fear - that Darwin's idea is really not so "dangerous" after all, because evolution does not rob our cherished notions of their meaningfulness. Rather, it explains that meaningfulness and shows how such things could have come about in a natural world, as the end result of a process in which cranes build increasingly larger cranes. He offers an array of audacious hypotheses about the origin of language, morality, meaning, religion and culture, all of which are grounded firmly in a Darwinian framework. Not all of these may turn out to be strictly correct (in particular, I accept the often-raised rejoinder that cultural evolution is not quite as Dennett describes it, because memes evolve in a Lamarckian, not Darwinian, sense), but I expect all of them at the very least have stimulated much discussion and will continue to do so, and on the broad points most of them must be largely accurate.
While creationists and others have bemoaned this book as the embodiment of everything they see as evil about Darwinian thinking, an attentive reading will fail to turn up any danger. Dennett firmly and explicitly rejects the favorite strawmen of religious antievolutionists, including nihilism and greedy reductionism (he does advocate a form of non-greedy reductionism, which is as it should be). In some cases, a negative reading can only be derived by taking him blatantly out of context. For example, Phillip Johnson:
Dennett cannot be accused of avoiding the religious liberty issue, or of burying it in tactful circumlocutions. He proposes that theistic religion should continue to exist only in "cultural zoos"... those metaphorical cultural zoos may one day be enclosed by real barbed wire...
...I will pass over the legal issues raised by this program of forced religious conversion because the intellectual issues are even more interesting.
This is an outrageous distortion, even for a creationist such as Johnson. This is what Dennett actually had to say:
Safety demands that religions be put in cages, too - when absolutely necessary. We just can't have forced female circumcision, and the second-class status of women in Roman Catholicism and Mormonism, to say nothing of their status in Islam (p.514).
We preach freedom of religion, but only so far. If your religion advocates slavery, or mutilation of women, or infanticide, or puts a price on Salman Rushdie's head because he has insulted it, then your religion has a feature that cannot be respected. It endangers us all... You are free to preserve or create any religious creed you wish, so long as it does not become a public menace (p.516).
In context, Dennett was clearly advocating a crackdown on religions that advocate violence or violation of human rights, not on "theistic religion" in general, as Johnson deceitfully represents him. This is an absolutely correct and ethical position to take, and if Johnson or any other theist opposes it, they are free to come forward and say so.
Though not as narrowly focused on one idea as Dennett's other books, Darwin's Dangerous Idea is no less interesting, and well worth reading. Those who do, regardless of whether they agree with all of Dennett's conclusions, may well find their thoughts opened up along tracks that had not previously occurred to them.
(Crossposted at Ebon Musings.)
Cracking the Fortune Cookie
A Response to John Searle's Chinese Room Analogy
John Searle. "Minds, Brains, and Programs." Behavioral and Brain Sciences, vol. 3, p. 417-424 (1980).
In a famous 1980 paper titled "Minds, Brains and Programs", the philosopher John Searle proposed a notorious thought experiment, now known as the Chinese Room, relating to the possibility of artificial intelligence. Searle has no objection to "weak AI", the claim that a properly programmed computer can help teach us about the mind; but he opposes "strong AI", the claim that a properly programmed computer actually would be a mind, with cognitive states just like those of humans, and purports to prove in this thought experiment that such a thing is impossible.
The Chinese Room is described as follows: Imagine that a person is locked in a room with a slot in the door. At regular intervals, a slip of paper covered with indecipherable squiggles comes through the slot. The person in the room looks up these squiggles in a book they possess, which instructs them to write a different set of squiggles on the paper and send it back out through the slot. As far as the person knows, they are just processing meaningless symbols; but unknown to them, the squiggles are Chinese characters, and they are actually carrying on a conversation with a Chinese speaker outside the room. The point of this analogy is that the person inside the room is acting just as a computer acts, processing symbols according to a set of rules. But this person does not understand what they are doing, and therefore a computer could never understand either. Searle concludes that a computer, even one that we could carry on a normal conversation with (i.e., a computer that could pass a Turing test) could never be conscious, could never understand, in the way that a human being does.
However, I do not agree with this analysis. I have just one request for Searle and his supporters: I want to see this marvelous book.
Even if we disregard the question of how unimaginably huge such a book would have to be, there are several categories of questions that it would seem no book, regardless of how much effort went into its creation, could give a correct and convincingly human-like answer to. For example, one could ask the same question multiple times; a human being would either rephrase the answer or become frustrated or both. Also, one could ask a question whose answer depends on contextual information (for example: "Would you please estimate how much time has passed since the beginning of our conversation?" or "Could you please rephrase the last question I asked?").
If, as postulated by Searle, a Chinese Room can pass a Turing test, then it would have to be able to answer repetitive and context-dependent questions correctly. But if the Chinese Room works in the way described by Searle, this is not possible. A book containing a static list of questions and answers - in effect, a list of rules reading "If you see X, do Y" - will unfailingly advise Y every time it is confronted with X. Therefore, a Chinese Room could easily be unmasked by asking it the same question repeatedly and observing that it gives the same answer repeatedly. And it would be utterly helpless to answer context-dependent questions in a convincing way; it could only make vague, general statements which would be easily recognized as such. Either way, a Chinese Room masquerading as a conscious person could easily be detected, and thus could not pass a Turing test. It would neither be conscious nor seem to be conscious, and hence would say nothing at all about the feasibility of true artificial intelligence. That is why I ask Searle and his supporters, what does this book look like? How does it advise responding to queries such as these?
What if we modify the Chinese Room so that it could pass this test? What changes would we have to make?
In light of the above challenge, the first change is obvious. The book in our Modified Chinese Room (MCR) could no longer be just a simple lookup table - in other words, it could no longer be memoryless. It would have to store some kind of state, some information describing the questions it has seen and answers it has given so far. But note, also, that memory is a necessary component of consciousness. Consciousness requires some minimal continuity of experience; an agent with absolutely no memory, whatever its intellectual capabilities, could not be said to be conscious.
But the mere maintenance of that state would be useless if it could not affect the answers that the MCR gives. Therefore, the MCR could no longer be a static list of responses; it would have to perform some kind of computation, combining its background lexical knowledge with the state information already stored, to come up with answers to questions.
With these two new tools at its disposal, it would seem that the MCR could pass a Turing test including repeated and context-sensitive questions. But are we still certain that this system is not actually conscious? After all, it answers questions put to it by extracting relevant information from the question, adding this information to its remembered state, and processing both the state information and its own background knowledge to produce a coherent reply. This seems very much like what human beings do in the same circumstance. For one thing, how could the MCR ever "pick out" the relevant information from a query unless it, in some way, understood what was being said to it? Though it might still be argued that such a system would not be conscious, it is no longer obvious that it could not be conscious, which is what I seek to establish.
The Chinese Room is a type of philosophical thought experiment that Daniel Dennett refers to as "intuition pumps", analogies that are designed to elicit an intuitive conclusion in a simple realm and then transfer that conclusion to a more complex domain. While intuition pumps are an appealing tool, they are frequently used to misdirect; very often, the conclusion drawn in the simple problem is not straightforwardly transferable to the more complicated problem. This is especially true in the domain of the mind, where our understanding is still so limited that "intuitions" about how such a system could or could not possibly work are as perilous as they are common. The true lesson of the Chinese Room is that we should not attempt to use our limited imaginations as a way to set bounds on reality.