A note on bad rhetoric

Consider the following questions:

“If free will is an illusion, then how do we find meaning?”

“If I’m going to die anyway, what point is there to living?”

“Why should we prevent the developing world from using fossil fuels if the West used fossil fuels in the past?”

They are all awful, because they are rhetorical questions with conditionals. You should never ever ever pose a question or statement like this. Most of the time, the conditional is irrelevant to actually answering the question. For instance, no account of meaning in life goes “because we have free will”, or says anything that obviously depends on the assumption that we have free will. So if you pose a rhetorical question with a conditional, one of two things is probably going on: you’re being a bad interlocutor by trying to smuggle in an implicit assumption that you really should be arguing, or you’re being a bad reasoner by thinking that a question obviously depends on a different issue which has dubious or limited relevance.

If you’re in a debate, responding to a rhetorical question with a conditional feels awkward because you feel obligated to address the implicit assumptions behind the conditional. You shouldn’t. It cedes territory by acknowledging more of their argument than is necessary to refute it. You can usually answer them more quickly and easily by simply pretending that the conditional isn’t there. Let the other person figure out how to properly communicate their point.

And if you’re trying to convince somebody, just explain briefly how the condition doesn’t affect the answer you are giving. And then link them to this post so that they don’t do it again.

In Defense of Common-Sense Ethics

In this post I am going to defend deference to common-sense ethics to make decisions which don’t fall directly under the purview of Effective Altruism. While any decision can technically be made on the basis of maximizing expected welfare or a similar metric, it is often difficult to explicitly make such calculations, to the point that they are intractable or even counterproductive. So when we make minor daily decisions, or when we consider major actions where lots of variables are at stake besides the conventional issues of charity and cause prioritization, it may not be the right method.

Now you might be thinking – wait, Zeke, hasn’t this already been stated many times already? Don’t we all, in Effective Altruism, know about honesty and moral norms and Schelling fences and so on? What about the CEA’s guiding principles, aren’t those common-sense ethics? Are you just going to agree with things that have already been said before?

No, I’m not. In fact, none of these things are examples of common-sense ethics. All of them are abstracted principles in the same category as effective altruism, utilitarianism, deontological ethics, anti-speciesism, and other moral principles. If you believe in upholding integrity, that is a belief in the ideal of doing things on the basis of integrity; it’s not the same as doing things on the basis of whether they adhere to common-sense ethics. Sure these ideas will give similar advice in most cases, but they won’t necessarily do so.

What it actually means to adhere to common-sense ethics is to follow “the pre-theoretical moral judgments of ordinary people.” If you spent a long time thinking about moral philosophy and came up with a bunch of principles that you think are important, then your judgements are post-theoretical. If you spent years of your life reading or blogging about philosophy, you are not an ordinary person, you are a philosopher, even if you are an amateur one. And if you are a white, affluent programmer in Silicon Valley, you are not an ordinary person, you represent an unusual and small subset of the human population.

So to follow common-sense ethics means to ask oneself whether ordinary people would find a behavior to be morally objectionable, not whether you find it objectionable or whether it violates some particular list of behaviors. Now why should we do things in this manner?

It keeps the moral focus on the core aspects of Effective Altruism.

Posturing matters. By explicitly deferring to conceptual principles such as virtues or Schelling fences, we prevent the basic ideas of cause prioritization and personal contribution from seeming to be of primary importance. If we endlessly debate moral principles which aren’t directly part of EA philosophy, we are posturing as if they are highly important and overrule basic EA ideas. This weakens our philosophical position and detracts from our main mission. It also provides a signal to potential detractors that accusations that EA is not following these principles will get a lot of traction.

It prevents harmful dissonance and obliviousness to the broader population.

By focusing on what ordinary people believe, EAs will not be out of touch or elitist. It pushes us to avoid the other-minds fallacy. For instance, think about the campaign run by GWWC Cambridge with a simulated poverty village. It was not dishonest or antithetical to the interests of the poor. It is not vicious to show people what poverty is like, nor does it violate the Categorical Imperative. You can have a poverty simulation while being committed to others, while being scientifically minded, while being open, while having integrity, and while having a collaborative spirit (CEA values). Yet it offended people nonetheless, something which should have been obvious to anyone who stopped to think about what common-sense ethics really are.

It dodges philosophical issues and confusion.

The question of whether something is in accordance with common-sense ethics is essentially empirical. It is easier resolve theses kinds of issues than it is to resolve age-old philosophical debates. For example, Mechanical Turk surveys can easily provide decent survey data on people’s attitudes. An easier and cheaper way of doing this is to simply ask ordinary people what they think about something. Now we can devote our philosophical thinking to the core issues of decision theory and cause prioritization which matter for the whole world.

It recognizes psychological and social realities.

The simple fact of the world is that people’s moral attitudes are not philosophically consistent, and common-sense ethics is something that is always contingent upon framing and social context. Recent work in experimental philosophy has demonstrated that moral judgements are frequently vulnerable to framing effects and other cognitive biases. People apply double standards on the basis of whose ideology they agree with – for instance, people on the right wing are frequently just as politically correct as those on the left, they just have different standards and methods for judging what is or isn’t politically correct. Many people today think that it is okay to actually murder someone in order to save other people’s lives, but they probably wouldn’t think that it’s okay for a charity to steal money from someone in order to save lives (something which is almost certainly not as bad as pushing them in front of a train). Many people follow the Copenhagen Interpretation of Ethics, a perspective which is basically nonsensical when scrutinized rationally.

Because of this, you will never be able to use a consistent and rigorous set of virtues or rules to determine what will or won’t make people indignant. Any ethical model of forbidden actions which is restrictive enough to avoid things that spark people’s negative moral judgements will also have many false positives, limiting us from accomplishing things that people would actually be okay with.

So what, exactly, is your advice?

When you have a set of actions and you can’t directly estimate which of them would maximize welfare for sentient life in the future, you may do any of them that would satisfy the pre-theoretical moral judgements of ordinary people. In general, just worry less about these kinds of issues, stop overthinking morality, and just do what other people think would be okay, as long as you don’t have a good EA-based reason to do otherwise.

The Future of Animal Consciousness Research

Abstract: research into the subjective states of animals is highly important for the long term future. However, the dynamics of public opinion may threaten the scientific and political environments in which ideas must be developed and implemented. I outline a particular way in which this is especially likely to happen for animal consciousness research and defend the importance of stopping such a trend.

It is often accepted in moral philosophy that some kind of sentience is both necessary and sufficient for the interests of beings to carry moral consideration. So animal sentience is naturally an important issue: birds, fish, nonhuman mammals, and reptiles each outnumber humans by more than one order of magnitude, and simple organisms such as rotifers and copepods outnumber humans at ratios of around a billion to one. Farming is particularly responsible for large masses of animal life: a meal with a pound of factory-farmed pork causes about three days of pig suffering, while a meal with a pound of maricultured catfish causes about a thousand days of catfish suffering. Even if you measure biomass rather than population sizes, humans are greatly outweighed by animals.

A clear trend here is that the small outnumber the large. Smaller organisms also tend to have much shorter lifespans, so if you measure total welfare by numbers of lives then the disparity increases still further, while it also increases if you measure total welfare by duration of subjective experience and you think that smaller minds experience slower subjective time. Therefore, a slight change in the extent to which we expect small minds to be conscious would constitute a massive update to our worldview, ethical decisions, and daily life by changing how we ought to act with regard to other organisms. Imagine an individual deciding whether and how to reduce their meat consumption: if catfish are not conscious, then a pork-and-catfish diet is no worse than a pork-only diet at first glance, and a pescatarian diet is just fine. But if catfish are conscious to the same extent that pigs are, then the mixed diet is astronomically worse as per Tomasik’s calculations, to the point that pig suffering is a rounding error by comparison. Abandoning welfare consequentialism for a rights-based view does little to remove the great importance of understanding animal consciousness, for in that case we would have to extend protections and rights to progressively larger swathes of the animal kingdom with significant social, economic, political, legal, and environmental consequences.

To drive home just how enormously important it is to figure out which animals are consciousness, here are more specific moral issues which depend on unsolved questions of animal consciousness:

– Insects may either be a viable human protein source leading to a reduction in human hunger and animal slaughter, or even worse than ordinary factory farming

– The widespread human practice of farming of shrimp, molluscs, crabs, lobsters, and other relatively simple marine animals could also be either extraordinarily harmful (because of their large numbers) or net positive by offsetting other kinds of farming

– Wildlife might experience net average suffering or net average happiness, possibly making the long term elimination of habitat spaces a moral imperative or a moral crime (respectively) affecting vastly more organisms than all the humans who have ever lived on the planet Earth

– In line with the above issue, spreading wildlife to other planets via terraforming could be either a viable strategy for the flourishing of our civilization or another serious moral crime

– Even if one rejects the idea of removing ecosystems, steps to improve the welfare of various kinds of wildlife could be demanded to varying degrees with large corresponding practical costs and ecological ramifications

– The way that we treat pets, of which there is one for every two humans in the United States, and whether it is truly commensurate with their interests, is in doubt

– The controversial moral status of “free range farming” turns in part on just how much suffering and happiness these animals experience

– The way that we should act in regards towards protecting and prioritizing human life and economic wealth, especially in the developed world, could be partially informed by the fact that humans consume varying quantities of factory farmed meat and thereby cause dozens or hundreds of animal deaths per year, as well as the fact that they are responsible for destroying parts of the environment through their patterns of consumption

– Cause prioritization between animal advocacy and other moral causes crucially depends on how many animals are sentient and what value we should assign to their interests

Research into animal consciousness may also inform other moral issues, such as consciousness and moral value in artificial intelligence and carbon-based posthuman agents, and it may help answer certain scientific and epistemic questions in the theory of anthropic reasoning. I don’t think that even the animal advocacy community has come close to recognizing the full importance of the issue. It is truly one of the most important research topics of our time.

If understanding animal consciousness is such an important goal, then why is it not commonly perceived as such? One is that most of the above issues are relatively new, niche, future-focused, and yet to be deeply investigated even within animal advocacy circles. And to the extent that wild animal suffering is perceived as a real problem, people commonly dismiss action to reduce it as technologically unfeasible.

Another issue is that investigation into the nature of consciousness is commonly regarded as an intractable philosophical topic; there are still many competing theories about the nature of consciousness as well as competing views on the proper methodology and extent of our abilities to investigate the phenomenon of consciousness in other minds, and these issues are often perceived as interfering with our ability to draw clear conclusions from behavioral and neuroscientific knowledge about animal minds. Finally, animal advocates have an incentive and an inclination to present the issue as if certain questions about animal consciousness have already been settled; conversely, most people eat meat and therefore would suffer cognitive dissonance if they took the ideas of animal consciousness and suffering very seriously.

Now if you think that animal consciousness is simply an intractable issue where we cannot make real intellectual progress, and that all we can ever do is come up with more and more arbitrary opinions, then you may as well stop reading here. For this argument I assume that we can in fact obtain better and better ideas (albeit probabilistic or even noncognitivist in nature) about conscious states and animal moral status as we develop better tools of science and expend more effort into philosophical inquiry. If you deny this premise, there is too deep a disagreement for me to address here, but I suggest that you look into contemporary philosophical arguments over moral status and the scientific field of comparative cognition in order to get a sense of the methodological pace and status of this line of inquiry.

Anyway, now that I have your attention on why this issue matters, I will go into some other topics before outlining the failure mode later on.

Racism, sexism and speciesism

Animal advocates have widely adopted the term “speciesism” as a pejorative descriptor for views which are prejudiced against animals. It (somewhat intentionally) brings terms such as racism and sexism to mind. However, the terms are not synonymous in function. Consider the following claim:

Thing 1 tends to have a greater capacity for rational thought and agency than Thing 2. Rational thought and agency is a necessary and sufficient criterion for the possession of certain moral rights. Therefore, Thing 1 is generally more morally deserving in certain respects than Thing 2.”

If you replace “Thing 1” with “the white man” and “Thing 2” with “the negro”, the claim is rather transparently racist. But if you replace the terms with “the human” and “the bonobo”, it’s not actually speciesist. Because it is so commonly accepted that animals are worth less in some way, and since most animals are so different from humans, the term speciesism is applied specifically to discrimination which is based merely on species membership. If the standard of moral significance is the strength of an animal’s interests, and animals of a certain species have consistently weaker interests than others due to their different cognitive capacities, then there’s nothing wrong with discriminating between the interests of individuals and communities which are composed of different species. Not only is this how moral philosophers understand the situation, but it’s widely accepted by the general public and many animal advocates that it’s okay to care less about some species if they lack certain cognitive and emotional faculties.

Anti-speciesists are nevertheless quick to draw parallels between their issue and the struggles of racial liberation. Comparisons between factory farming and the Holocaust have been a recurring motif. Richard Ryder, who coined the term speciesism, wrote in 1971:

“In as much as both “race” and “species” are vague terms used in the classification of living creatures according, largely, to physical appearance, an analogy can be made between them. Discrimination on grounds of race, although most universally condoned two centuries ago, is now widely condemned. Similarly, it may come to pass that enlightened minds may one day abhor “speciesism” as much as they now detest “racism.” The illogicality in both forms of prejudice is of an identical sort. If it is accepted as morally wrong to deliberately inflict suffering upon innocent human creatures, then it is only logical to also regard it as wrong to inflict suffering on innocent individuals of other species. … The time has come to act upon this logic.”

So let’s take a look at exactly how racism is detested these days, but instead of looking at ordinary layman prejudice we’re going to focus on what happens when research and policy proposals are associated with scientific racism. We’ll also look a bit at science on differences between the sexes because of the overall similarity of the issue.

Western cultural and political reactions towards demographic science

Research about race which purports to show strong genotypic correlations with human racial classifications or systematic differences between racial/ethnic groups is subject to enormous resistance in both elite and popular culture in the West. In The Bell Curve, Charles Murray argued that it seems likely that the difference in measured IQs among different races is partly genetic in origin. This and similar ideas prompted the Southern Poverty Law Center to label him a “white nationalist” and caused students to engage in disruptive protests and to commit physical assault against Murray’s colleague Allison Stanger. Murray’s reputation has mostly been trashed, partly by media outlets which aim to present him as a hack (see Vox’s pieces Charles Murray is once again peddling junk science about race and IQ or “Scientific racism” is on the rise on the right. But it’s been lurking there for years.), even though the actual science is very much in doubt (see The cherry picked science in Vox’s Charles Murray article, The misrepresentation of genetic science in the Vox piece on race and IQ, and Garett Jones via Twitter). In 2013, Dylan Matthews wrote a brief expose of Jason Richwine, a Heritage Foundation policy writer whose Harvard PhD dissertation argued for lower average intelligence in Hispanics and IQ testing at the border. Heritage quickly posted a disclaimer, then a few other objectionable tidbits from his past became known, and Richwine resigned shortly afterwards. More recently, James Damore’s internal memo challenging gender diversity orthodoxy at Google was leaked, ridiculed, and caused him to be fired, despite numerous scientists stating that his views were reasonable.

But these are just a few examples of a systematic habit of politicization and opposition to this type of work. Anyone who aims to make either a hobby or a career out of studying IQ heritability and race can expect aggressive opposition and public attacks, so there is a fairly strong disincentive and barrier against scientific work in this area, while the implementation of any policy proposals explicitly based on such work can safely be ruled implausible.

The justifications for these practices are several. First, critics tend to claim that the views happen to be scientifically wrong; that there has been such conclusive evidence to refute the claims of these scientists’ work that they can be dismissed as pseudoscience. Second, critics believe that the scientists themselves are prejudiced and either consciously or subconsciously perform research with the goal of validating their prejudice. Third, they believe that these ideas are likely to be used to justify extremely harmful behavior, just as racial science in the 19th and early 20th century (which has since been totally discredited) was used to justify slavery, coercive eugenics, and the Holocaust.

The first justification is neither necessary nor sufficient for vocal and potent opposition to research based on perceived racism or sexism. First, even if you do disagree with many of Murray’s claims, as Turkheimer and other scientists do, that does not imply that their general corpus of work is pseudoscientific, and his work is in fact broadly acceptable to significant portions of academia (cf. the rebuttals to Vox linked above). The same applies to Damore’s memo. But media outlets and commentators have been perfectly happy to use the arguments from Turkheimer et al that Murray is wrong as evidence that Murray is a hack promulgating entirely discredited racist theories. And many people believe that even research which is technically correct should not be promoted, at least not without very strong caution and carefulness, if it might be abused to promote a racist agenda.

Second, there are plenty of people currently engaged in researching scientific theories which have been attacked and discredited to varying degrees, such as the aquatic ape hypothesis and chiropractic, and yet they are not subjected to nearly the same level of politicization and resistance which are applied to race-related work. Generally speaking, our society only goes on witch-hunts against bad science to the degree that a scientific claim is perceived as being potentially harmful.

With regard to the presumption that racial hypotheses in science and social studies are a front for moral prejudice on the part of the researchers against other races, there is hardly anything that can be done to actually falsify such a hypothesis. While there is a sufficient wealth of quotes from Charles Murray to show that his attitudes towards people of different ethnicities are old-fashioned to say the least, in most cases (e.g. Richwine, Damore) individuals receive negative attention and career damage based purely on the content of their research and policy ideas and their association with right-wing institutions and points of view. So actually being prejudiced is not necessary for one’s reputation and work to be attacked as racist.

Why it matters

I hope I’ve been sufficiently dispassionate as to avoid saying anything that assumes a particular view on the race-IQ or tech-gender wars, something which I don’t intend to bother with. What I argue here should be persuasive regardless of whatever one believes about race, free speech, “human biodiversity” and similar topics.

My argument is that it is quite possible that research and policies about morally significant differences between species on the basis of consciousness will eventually be perceived and treated in a similar manner to how research and policies around race and intelligence are perceived and treated today, and that this would be a very bad thing. Currently there is only a modest amount of attention being paid to the science of animal consciousness, but this will change if animal ethics becomes more important in popular culture, if less-controversially-sentient animals (like pigs and chimpanzees) get proper legal and moral status (thereby making smaller and simpler animals the focus of attention), and if the philosophical confusions surrounding the issue begin to dissipate.

There is no logical implication that whatever is the proper societal response for the case of racism is the proper societal response for the case of speciesism or vice versa. There are many ways to respond to a set of prejudiced views, and a priori we should accept that “actively go on witch hunts and suppress the views of anyone whose work can be associated with these views” might be appropriate when applied to one kind of prejudice but inappropriate when applied to another.

For instance, look at the atheist community. They commonly believe that religion does serious harm to society, that it has been scientifically discredited, and that it is used as a tool for the oppression of minorities. Religious prejudice does have a long and brutal history. But atheists don’t generally think that Alvin Plantinga is an active threat to society and shout him down whenever he tries to speak on college campuses; there are no punch-a-Presbyterian memes. This is the strategy which maximizes the chance that society will eventually converge to truthful views on an issue, allowing a window for reasonable debate over ideas and policies. Conversely, when it comes to race, we have generally adopted a strategy which is aimed at minimizing the chance that people will enact actively harmful policies. This is not an uncharitable claim; anti-racist activists frequently and openly state that they view their position as fundamentally based on the need to avoid a repetition of the immoral activities of the earlier 20th century and that any science which could potentially be used to justify racist attitudes ought to be done with extreme care and caution (if at all).

The problem is that the animal advocacy community sees their issue as more analogous to struggles for racial justice than to any other historical cases where humanity expanded their moral circle, and it’s not hard to see why. The slavery/factory farming and Holocaust/factory farming analogies are visceral, provocative and quite accurate. Once Western society removes meat from the typical diet (as moral and economic trends indicate they probably will) and affirms a general rejection of speciesism, it is going to look back at factory farming as one of the greatest moral crimes in human history. This is what animal advocates have been saying for decades, and I believe they are correct. If so, then it is highly likely that any scientific research or policy proposals which can be associated with speciesist attitudes will provoke the same reaction from future society that various shades of race realism provoke today.

I expect that if you are sympathetic to the ideas of people like Richwine and Murray or simply value an inclusive conception of free speech and are dismayed at the contemporary state of the Western culture wars then you’ll accept that this would be a bad thing. But if you’re quite content with the way that race realism and similar views are treated these days then you should be worried too. Just because science about race happens to be generally pseudoscientific doesn’t mean that science about species always is, and just because race realists happen to be personally motivated by prejudice doesn’t mean that biologists researching this type of topic will be, so you cannot assume that species-based research and policies will be biased or wrong. But as noted earlier, the public conscience on race doesn’t specifically analyze whether a specific scientist or policy wonk is being rigorous or not or whether they do have deep-seated racist biases before going after their reputation and work. The general attitude regarding race realist research is preemptive; it resists certain ideas from gaining traction regardless of the intentions and methodological foundation of the researchers.

Furthermore, I expect you to recognize that some species do systematically matter more than other species. While different animals’ interests don’t matter differently merely because the species, some of them simply have fewer interests than others. Some of them only have direct preferences to avoid certain kinds of pain, and many might not have any conscious experience at all. Many animals have the neurophysiological and behavioral analogues of various kinds of emotion and human-like interests, but none possess anything close to the full sophistication of human cognition with all its intensities and complexities of desire and belief. And it’s not at all clear that the subjective desire that even a relatively cognitively advanced animal has for avoiding death or ending pain is just as strong as the analogous desire in the larger and more sophisticated mind of a human. Our subjective experiences probably differ; we don’t know what it’s like to be a bat.

I think that being a human is better than being an animal. I lean towards a fairly basic idea of hedonist utilitarianism, but all the same, I don’t naively equivocate all species’ interests. I would rather be Haydn than an oyster (as long as the oyster’s life is not too many orders of magnitude longer than Haydn’s). And I expect most people to be at least as anthropocentric on this topic as I am. So we should be very wary of the risk that society will over-correct against the sort of naive, ugly speciesism which facilitates crimes such as factory farming and animal abuse. If society labels our preference for higher, more sophisticated forms of cognition as speciesist, immoral and inappropriate for public discussion, then moral and policy issues such as the ones outlined in the beginning of this post will seem as intractable and screwed up to us as contemporary economic, criminal, and civil issues do to the current generation of human biodiversity enthusiasts. The outcome would be a knee-jerk insistence on treating different species ‘equally’ at the expense of accurate conceptions of their welfare, and we’ll find ourselves on the fringes of public and political discourse trying to argue that actually we shouldn’t rewild land to replace human spaces with rats hooked up to opiates, or that we should repopulate ecosystems with species which tend to suffer less pain than others do, or that we should implement methods of genetic engineering which improve the richness and complexity of animals’ cognition and experiences, against a potent zeitgeist which associates these views with the legacy of factory farming and other tragedies.


There are a variety of reasons which one could give to argue that it would be easy for decent researchers to avoid running into this failure mode, but they don’t work.

It won’t be sufficient to just “not do pseudoscience,” as one might suggest. First, following at least the basic guidelines of decent scientific methodology wasn’t sufficient to make Murray or Richwine’s views politically palatable. The line between wrong science and bad science is flimsy at best, and it’s easy for the views of a researcher such as Turkheimer to be used as a weapon for dismissing opposing researchers and their views as wholly discredited. Second, it is sheer hubris to expect that one will not make major mistakes in the conduct of science. Not being wrong is fantastic as a goal, but it’s terrible as a solution. And animal consciousness is a particularly tough issue which is arguably even harder to address than genetics and intelligence, since not only is the empirical science still in a relatively rough state, but there are major unsolved philosophical issues which relate to the matter. It is extremely likely that philosophers and scientists several decades or centuries from now will see our views on consciousness as antiquated and pseudoscientific (in fact, I have already seen dualism compared to vitalism, which is a theory that enjoys about as much scientific rapport these days as craniometry does). And just as a popular claim underlying perceptions of race realism and similar views as pseudoscientific is the fact that the popular conception of race is socially constructed, the ontological status of the species is very much in doubt, with some (see section 4) arguing that there really is no true scientific category of species. So while we can expect ourselves or our intellectual descendants to be on the cutting edge of scientific knowledge at the time of publication, we can’t assume that we won’t end up with an aura of historical failure and pseudoscience because of the failures of ourselves or our intellectual ancestors. All we can do is cultivate scientific open-mindedness as we keep trying to learn more.

We also won’t be saved if we resolve not to be prejudiced. First, plenty of animal consciousness researchers probably do eat meat, just because it’s so common in Western society, and many future ones will too. We may not approve of it, but we’ll have to work with them and use their research if we want to be as knowledgeable and scientifically comprehensive as possible. And as has been noted already, not actually having racist attitudes does not give a researcher a free pass to publish and promote race-realist ideas. Acknowledging that some animals are better than humans – e.g., arguing that elephants have richer perceptive experiences than humans because of their larger brains – wouldn’t do anything either, since white HBD enthusiasts earn no sympathy when they talk about the greater intelligence of east Asians or Ashkenazi Jews. Plus, it’s very hard to know when one is biased or not, or what that even means when the foundational scientific and moral questions are still undecided, and many of both the early race realists as well as contemporary ones have (apparently sincerely) felt as if they weren’t racist in the ordinary sense of the term. We can make a point of loving animals and being vegan all our lives, but that will probably work about as well as saying “I’m not racist, I have black friends.”

Now remember the distinction between speciesism and racism mentioned earlier. It’s subtle and tenuous. People screw it up sometimes. It’s exactly the kind of thing which a wave of popular moral sentiment is likely to overwhelm and dissolve with blanket judgements, presumptions of bad faith, and liberal application of the noncentral fallacy. Dictionary definitions of speciesism don’t reflect it precisely; it’s only something that is clear when you look directly into the relevant moral philosophy and read between the lines a little bit. Most people today have never actually read anything on philosophy of race or the moral status of animals, so unless there is a systematic improvement in humans’ philosophical knowledge we should not expect a post-speciesist society to be well informed on the distinction. Being philosophically correct about the nuances of speciesism and moral status in animals, while extremely important in its own right, won’t do much to protect us in the courts of popular and elite opinion. (Of course, just as many geneticists and psychologists may agree with what people like Murray say, many philosophers may agree with what we say. But philosophers’ opinions rarely matter much when it comes to public policies and cultural attitudes.)

Now perhaps the failure mode described in this essay will never happen. Can we expect things to go differently for animal consciousness than for race and gender?

The reader might point out that society will simply never look at animals the way that we currently look at other races. But this assumption is unwarranted. Early racist sentiments in the West viewed foreign races as being literally subhuman, existing somewhere between man and ape. And our modern conception of the intelligence and moral value of apes is much greater today than it was in the 1800’s. It’s plausible that the average modern animal advocate has more moral concern for many types of animals than the average white person cared about black people during the days of slavery. And while modern anti-speciesists will often openly state that they don’t think animal lives and experience are worth the same as human lives and experience, early white antislavery advocates frequently had racist sentiments themselves and did not advocate for full equality, Abraham Lincoln being a prime example; the emancipation movement which they started evolved into a massive zeitgeist which (rightfully!) rejected these other views which they held.

There are many contemporary examples of people having concern for animals comparable to their concern for humans, such as Western culture’s anthropomorphic attitude towards pets, various cultures’ habits of worshiping animals and animal-form deities, the vegan movement’s long history of performative art and advocacy messages which compare livestock to people, and the general propensity which people have to anthropomorphize and give strong moral consideration to various animals (remember Cecil the Lion? How about Harambe?). This causes people to have strong emotional attitudes in favor of animal rights and protections on cases like poaching and pet abuse.

The fact that we can’t directly communicate with animals and don’t live among them does not change very much. Opposition to racism is mostly not something that people learn when they happen to interact with members of different ethnicities. Rather, tolerance is taught as a background ideology from an early age and is propagated along tribal lines as a political attitude in early adulthood, much like ethical veganism. And the fact that animals can’t speak up for themselves isn’t very relevant; much of the social justice activist movement today consists of minorities speaking up for other minorities or white people speaking up for minorities, hence the prevalent idea of “secondhand outrage.” Moreover, there actually are a few people who do identify as animals. They enjoy almost no legal and scientific recognition, but that is pretty much what transgender people received in Western culture a hundred years ago. (Transracialism, which is not nearly as accepted as transgenderism in part because of the history of racial exploitation, may be a fairer comparison, but it is beginning to be taken slightly seriously, as evidenced by Rebecca Tuvel’s paper and the debates surrounding it.)

A decent claim which can be made to defend optimism on this issue is that factory farming and other forms of animal harm are not justified by empirical claims about animal consciousness to the degree that racist moral crimes were justified by empirical claims about racial inferiority. However, the extent of this difference is neither clear nor large. There already are certain proper debates where people explicitly defend habits of harming certain animals on the basis of their perceived lack of sentience, such as the argument over oysters and the controversies regarding lobsters and other crustaceans. And Bastian et al showed that denying the presence of animal minds is a common strategy used by people to justify meat consumption. I believe we can expect more people to explicitly argue that animals lack sufficient consciousness as arguments and legal restrictions against the meat industry become more and more prevalent, forcing people to defend their habits.

Finally, you might suggest that it wouldn’t be bad for animal consciousness research to follow the same path of other similar topics.

For instance, you might dispute that scientific racism is in fact suppressed and discredited in the contemporary western zeitgeist. Many people believe that racism is still lurking just beneath the surface of contemporary society. More to the point, rather than believing that race realists’ views are being suppressed to a greater degree than would be expected in a society of Vulcans, they believe that these people and their bad science are getting undue attention because a subconsciously (or consciously) racist slice of the population is lapping up whatever they see that can confirm their biases.

There are a couple of problems here. First, it only seems true if you fall victim to selection bias and only think about the race realists which get attention in the media. There are many other researchers and bloggers who look into this sort of thing but are mostly ignored because they’re not public targets. Wonks and elites ignore them because their research is such a hot potato, people who are prone to racist attitudes don’t know about them because either they’re ivory tower academics or they live in dark corners of the blogosphere, and anti-racists either don’t know or don’t care about what they do enough to go after them. Nobody took notice of Richwine’s dissertation until he co-authored a high-profile and controversial study on the fiscal costs of illegal immigration at Heritage, prompting journalists to look into his background. How many other Jason Richwines are there in the world, waiting in fear for the day when they get a Dylan Matthews of their own?

Secondly, the relevant issue is not whether alternative news outlets and fringe web forums will take your views seriously, it’s whether the broader public and (especially) the elite policy establishment will do so. The current cultural and political status of race realism may be pretty good for getting a small explicitly racist minority of the population to take it seriously and it may even get the majority of the population to feel sympathetic to researchers and policy analysts who are the victims of student and media witch hunts, but it also does a pretty good job of keeping these issues out of the actual policy making process, which is run more by progressive elites and centrist wonks than by the popular masses. Anti-illegal-immigrant and anti-refugee sentiments and policies are much more likely to be driven by casual prejudice and uninformed worries over crime and terrorism; actual discussion of genetics and IQ as a means of informing policy is generally out of the question. For evidence, look at the Trump administration; after all the racial anxiety and anti-immigrant sentiment contributing to his election, the increasing boldness of white nationalists among his constituency, his propensity for racial insensitivity, his bans of immigration from Muslim countries, and his history of sexual misconduct, even he hasn’t tried to bring scientific racism to Washington or done anything substantial to change the status of people like Murray and Damore.

But shouldn’t we be worried about far-future crimes which might be encouraged by speciesist-seeming ideas, just as anti-racist activists are worried about crimes which might be encouraged by racist-seeming ideas? Due to comparisons with slavery and the history of factory farming, most people are very likely to perceive erring on the side of species egalitarianism as appropriate. And of course we should have some precaution regarding such possibilities, just as we should with all forms of inquiry and all forms of moral crime, but these loose analogies and superficial comparisons with factory farming and slavery are not a sound rational reason to set our priorities in a certain way, and we ought to aim to approximate truth as much as possible with no presumption for erring on either particular side of the line until we find specific reasons to expect particular scenarios. Many of the issues of animal ethics which we will have to deal with in the future are strange and fundamentally different from factory farming and animal testing; rather than simplistic cases of humans exploiting animals for direct gain, we will probably have to make more careful judgements about how animals matter relative to each other and the contexts in which animal life is good or not. Looking back at the applications of animal consciousness studies mentioned in the beginning of this post, and the possible ways that they could be mishandled, I see no reason to expect naively egalitarian views to do more harm than naively speciesist views.


I don’t see an easy solution to this, and right now I think the most important thing is to confirm whether this is a significant potential problem. Conditional on civilization lasting long enough and not experiencing a sufficiently radical change in society and methodology to render these ideas nonsensical, I currently assign a probability of 15-20% to animal consciousness research eventually becoming even more politically tarnished than contemporary race research and a probability of 25-30% to it eventually becoming even more politically tarnished than contemporary gender/sex research. (The reason these numbers are not higher is simply regression to the mean for lines of public policy and scientific inquiry.) I am also 70% confident that animal consciousness research will become noticeably more politically tarnished than we should desire it to be, and I think that the chance that it will be insufficiently tarnished is negligible. To refine these expectations, I would like to see responsible, respectful, low-profile discussion of this issue, with input from people who have backgrounds on either side of the race/gender culture wars, that does not make this problem a self-fulfilling prophecy by immediately associating animal consciousness with race or gender realism.

Utopia In The Fog

The last several years have witnessed a strong rise of activity on the topic of AI safety. Institutional and academic support has vindicated several elements of the embryonic Friendly AI research program. However, I believe that the degree of attention it has received is undue when compared to other aspects of artificial intelligence and the far future. It resembles the concept of an “availability cascade”, defined by Wikipedia as follows:

An availability cascade is a self-reinforcing cycle that explains the development of certain kinds of collective beliefs. A novel idea or insight, usually one that seems to explain a complex process in a simple or straightforward manner, gains rapid currency in the popular discourse by its very simplicity and by its apparent insightfulness. Its rising popularity triggers a chain reaction within the social network: individuals adopt the new insight because other people within the network have adopted it, and on its face it seems plausible. The reason for this increased use and popularity of the new idea involves both the availability of the previously obscure term or idea, and the need of individuals using the term or idea to appear to be current with the stated beliefs and ideas of others, regardless of whether they in fact fully believe in the idea that they are expressing. Their need for social acceptance, and the apparent sophistication of the new insight, overwhelm their critical thinking.

In this post I’m going to argue for a different approach which should bring more balance to the futurist ecosystem. There are significant potential problems which are related to AI development but are not instances of value alignment and control, and I think that they are more deserving of additional effort at the margin.

The prospects for a single superintelligence

Bostrom (2016) says that a recursively self-improving artificial general intelligence with a sufficient lead over competitors would have a decisive strategic advantage that is likely to ensure that it controls the world. While this is plausible, it is not inevitable and may not be the most likely scenario.

Little argument has been given that this scenario should be our default expectation as opposed to merely plausible. Yudkowsky (2013) presents an argument that the history of human cognitive evolution indicates that an exponential takeoff in intelligence should be expected, though the argument has yet to be formally put together and presented. Computer scientists frequently refer to complexity theory, which implies that getting better at problem solving rapidly becomes very difficult, towards asymptotic limits. In broader economic strokes, Bloom et al (2017) argue that there is a general trend of diminishing returns to research. Both these points suggest that for an agent to acquire a decisive strategic advantage in cognition would either take a very long time or not happen at all.

It seems to me, intuitively, that if superintelligence is the sort of thing that one agent cannot obtain rapidly enough to outcompete all other agents, then it’s also the sort of thing which cannot be obtained rapidly enough by a small subset of agents, like three or four of them. So it will be widespread, or alternatively, it cannot be obtained at all, leaving billions of humans or other agents at the top of the hierarchy. So while I don’t think that a true multi-agent scenario (with scores or more agents, as is typically meant by the term in game theory) is inevitable in the event that there is no single superintelligence, I think it’s conditionally probable.

The Importance of Multi-agent Analysis: Three Scenarios

Whole brain emulation and economic competition

Robin Hanson (2016) writes that the future of human civilization will be a fast-growing economy dominated by whole brain emulations. The future looks broadly good in this scenario given approximately utilitarian values and the assumption that ems are conscious, with a large growing population of minds which are optimized for satisfaction and productivity, free of disease and sickness. Needless to say, without either of the above premises, the em scenario looks very problematic. But other aspects of it would potentially lead to suboptimal utility: social hierarchy, wealth inequality and economic competition. Also, while Hanson gives a very specific picture of the type of society which “ems” will inhabit, he notes that the conjunction of all his claims is extremely unlikely, so there is room for unforeseen issues to arise. It is plausible to me that the value of an em society is heavily contingent upon how ems are built, implemented and regulated.

However, the idea of whole brain emulation as a path to general artificial intelligence has been criticized and is a minority view. Bostrom (2016) argues that there seem to be greater technological hurdles to em development than to other kinds of progress in intelligence. The best current AI is far more capable than the best current emulation (OpenWorm). Industry and academia seem to be placing much more effort into even the very speculative strains of AI research than into emulation.

The future of evolution

If humans are not superseded by a monolithic race of ems, then trends in technological progress and evolution might have harmful effects upon the composition of the population. Bostrom (2009) writes that “freewheeling evolutionary developments, while continuing to produce complex and intelligent forms of organization, lead to the gradual elimination of all forms of being that we care about.” With the relaxation of contemporary human social and biological constraints, two possibilities are plausible: a Malthusian catastrophe where the population expands until welfare standards are neutral or negative, and the evolution of agents which outperform existing ones but without the same faculties of consciousness. Either of these scenarios would entail the extinction of most or all that we find valuable.

Andres Gomez Emilsson also writes that this is a possibility on his blog, saying:

I will define a pure replicator, in the context of agents and minds, to be an intelligence that is indifferent towards the valence of its conscious states and those of others. A pure replicator invests all of its energy and resources into surviving and reproducing, even at the cost of continuous suffering to themselves or others. Its main evolutionary advantage is that it does not need to spend any resources making the world a better place.

Bostrom does not believe that the problem is unavoidable, saying that a ‘singleton’ could combat this process. By singleton he refers to not just a superintelligence but also to any global governing body or even a set of moral codes with the right properties. He writes that such an institution should implement “a coordinated policy to prevent internal developments from ushering it onto an evolutionary trajectory that ends up toppling its constitutional agreement, and doing this would presumably involve modifying the fitness function for its internal ecology of agents.”

Augmented intelligence and military competition

Daniel McIntosh (2010) writes that the near-inevitable adoption of transhuman technologies poses a significant security dilemma due to the political, economic, and battlefield advantages provided by agents with augmented cognitive and physical capabilities. Critics who argue for restraint “tend to deemphasize the competitive and hedonic pressures encouraging the adoption of these products.” Not only is this a problem on its own, but I see no reason to think that the conditions described above wouldn’t apply for scenarios where AI agents turned out to be the primary actors and decisionmakers rather than transhumans or posthumans.

Whatever the type of agent, arms races in future technologies would lead to opportunity costs in military expenditures and would interfere with the project of improving welfare. It seems likely that agents designed for security purposes would have preferences and characteristics which fail to optimize for the welfare of themselves and their neighbors. It’s also possible that an arms race would destabilize international systems and act as a catalyst for warfare.

These trends might continue indefinitely with technological progress. McIntosh rejects the assumption that a post-singularity world would be peaceful:

In a post-singularity, fifth-generation world, there would always be the possibility that the economic collapse or natural disaster was not the result of chance, but of design. There would always be the possibility that internal social changes are being manipulated by an adversary who can plan several moves ahead, using your own systems against you. The systems themselves, in the form of intelligences more advanced than we can match, could be the enemy. Or it might be nothing more than paranoid fantasies. The greatest problem that individuals and authorities might have to deal with may be that one will never be sure that war is not already under way. Just as some intelligence analysts cited the rule that “nothing is found that is successfully hidden” – leading to reports of missile gaps and Iraqi WMD – a successful fifth generation war would [be] one that an opponent never even realized he lost.

Almost by definition, we cannot precisely predict what will happen in a post-singularity world or develop policies and tools that will be directly applicable in such a world. But this possibility highlights the importance of building robust cooperative systems from the ground up, rather than assuming that technological changes will somehow remove these problems. A superintelligent agent with a sufficient advantage over other agents would presumably be able to control a post-singularity world sufficiently to avoid this, but as has been noted, it’s not clear that this is the most likely scenario.

Multi-agent systems are neglected

The initiatives and independent individuals close to the EA sphere who are working towards developing reliable, friendly AI include the Machine Intelligence Research Institute, the Future of Humanity Institute, Berkeley’s Center for Human-Compatible AI, Roman Yampolskiy, and all the effective altruists who are students of AI as far as I can tell. There is less attention towards multi-agent outcomes, as Robin Hanson, Nick Bostrom and Andres Gomez Emilsson seem to be the only ones who have done research on it (and Bostrom seems to be focused on superintelligence), while the Foundational Research Institute has given a general nod towards looking into this direction with its concerns over AI suffering, cooperation, and multipolar takeoffs.

The disparity is preserved as you look farther afield. Pragmatic industry-oriented initiatives to make individual AI systems safe, ethical and reliable include the Partnership on AI among the six major tech companies, some attention from the White House on the subject, and a notable amount of academic work at universities. The work in universities and industry from researchers on multi-agent systems and game theory seems to be entirely focused on pragmatic problems like distributed computational systems and traffic networks; only a few researchers have indicated the need for analyzing multi-agent systems of the future, let alone actually done so. Finally, in popular culture, Bostrom’s Superintelligence has received 319 Amazon reviews to Age of Em’s 30 despite being published at a similar time, and the disparity in general media and journalism on the two general topics seems comparably large.

I do not expect this to change in the future. Multi-agent outcomes are varied and complex, while superintelligence is highly available and catchy. My conclusion is that the former is significantly more neglected than the latter.

Is working on multi-agent systems of the future a tractable project?

The main point of Scott Alexander’s “Meditations on Moloch” is essentially that “the only way to avoid having all human values gradually ground down by optimization-competition is to install a Gardener over the entire universe who optimizes for human values.” In other words, given the problems which have been described above, the only way to actually achieve a really valuable society is to have a singleton which has the right preferences and keeps everyone in line.

This is not different from what Bostrom argues. But remember that the singleton need not be a superintelligence with a decisive strategic advantage. This is fortunate, since it is plausible that computational difficulties will prevent such an entity from ever existing. Instead, the Gardener of the universe might be a much more complex set of agents and institutions. For instance, Peter Railton and Steve Petersen are (I believe) both working on arguments that agents will be linked via a teleological thread where they accurately represent the value functions of their ancestors. We’ll need to think more carefully about how to implement this sort of thing in a way that reliably maximizes welfare.

This is why analysis in multi-agent game theory and mechanism design is important. The very idea behind game theory in general is that you can find useful conclusions by abstracting away from the details of a situation and only looking at players as abstract entities with basic preferences and strategies. This means that analyses and institutions are likely to be pertinent to a wide range of scenarios of technological progress.

While ideas of preventing evolution, economic competition and arms races sound extremely difficult, there is some historical precedent for human institutions to install robust regulations and international agreements on this type of issue. Admittedly, none of it has been on nearly the same scale that would be required to solve the problems described above. But due to the preliminary stage of this line of research, I think that additional research, or literature review at minimum, is needed at least to investigate the various possibilities which we might pursue. Also, there is a similar problem with cooperation when it comes to ordinary AI safety anyway (Armstrong et al 2013).

Conclusion and proposal

I believe I have shown that recent interest in AI and the future of humanity has disproportionately neglected the idea of working on a broader range of futures in which society is not controlled by a single agent. There is still value in AI safety work insofar as alignment and control would help us with building the right agents in multi-agent scenarios, but there are other parts of the picture which need to be explored.

First, there are specific questions which should be answered. How likely are the various scenarios described above, and how can we ensure that they turn out well? Should we prefer that society is governed by a superintelligence with a decisive strategic advantage, and if so, then how much of a priority is it?

Second, there are specific avenues where practical work now can uncover the proper procedures and mindsets for increasing the probability of a positive future. Aside from setting precedents for international cooperation on technical issues, we can start steering the course of machine ethics as it is implemented in modern-day systems. Better systems of machine ethics which don’t require superintelligence to be implemented (as coherent extrapolated volition does) are likely to be valuable for mitigating potential problems involved with AI progress, although they won’t be sufficient (Brundage 2014). Generally speaking, we can apply tools of game theory, multi-agent systems and mechanism design to issues of artificial intelligence, value theory and consciousness.

Given the multiplicity of the issues and the long timeline from here to the arrival of superhuman intelligence, I would like to call for a broader, multifaceted approach to the long term future of AI and civilization. Rather than having a singleminded focus on averting a particular failure mode, it should be a more ambitious and positive project towards a pattern of positive and self-reinforcing interactions between social institutions and intelligent systems, supported by a greater amount of human and financial capital.


Armstrong, Stuart et al (2016). Racing to the Precipice: a Model of Artificial Intelligence Development. AI & Society.

Bloom, Nicholas et al (2017). Are Ideas Getting Harder To Find?

Bostrom, Nick (2009). The Future of Human Evolution. Bedeutung.

Bostrom, Nick (2016). Superintelligence. Oxford University Press.

Brundage, Miles (2014). Limitations and Risks of Machine Ethics. Journal of Experimental & Theoretical Artificial Intelligence.

Hanson, Robin (2016). Age of Em. Oxford University Press.

McIntosh, Daniel (2010). The Transhuman Security Dilemma. Journal of Evolution and Technology.

Yudkowsky, Eliezer (2013). Intelligence Explosion Microeconomics.