The following categories represent clusters of ideas that we believe warrant further attention and research for the purpose of developing human-compatible AI (HCAI), as well as prerequisite materials for approaching them. The categories are not meant to be mutually exclusive or hierarchical. Indeed, many subproblems of HCAI development may end up relating in a chicken-and-egg fashion.

Certain basic mathematical and statistical principles have contributed greatly to the development of today's prevailing methods in AI. This category surveys areas of knowledge such as probability theory, logic, theoretical CS, game theory, and economics that are likely to be helpful in understanding existing AI research, and advancing HCAI research more specifically.

This category surveys pedagogically basic material for understanding today's prevailing methods in AI and machine learning. Some of the methods here are no longer the state of the art, and are included to aid in understanding more advanced current methods.

Many currently known methods from machine learning and other disciplines already illustrate surprising results that should inform what we think is possible for future AI systems. Thus, while we do not yet know for certain what architecture future AI systems will use, it is still important for researchers working on HCAI to have a solid understanding of the current state of the art in AI methods. This category lists background reading that we hope will be beneficial for that purpose.

Understanding the nature of adversarial behavior in a digital environment is important to modeling how powerful AI systems might attack or defend existing systems, as well as how status quo cyber infrastructure might leave future AI systems vulnerable to attack and manipulation. A cybersecurity perspective is helpful to understanding these dynamics.

(Entries for this category are forthcoming.)

There are many problems that need solving to ensure that future, highly advanced AI systems will be safe to use, including mathematical, engineering, and social challenges. We must sometimes take a bird's-eye view on these problems, for example, to notice that technical safety research will not be useful on its own unless the institutions that build AI have the resources and incentives to use that research. This category recommends reading that we think can help develop such a perspective.

Consider a cleaning robot whose only objective is to clean your house, and which is intelligent enough to realize that if you turn it off then it won't be able to clean your house anymore. Such a robot has some incentive to resist being shut down or reprogrammed for another task, since that would interfere with its cleaning objective. Loosely speaking, we say that an AI system is incorrigible to the extent that it resists being shut down or reprogrammed by its users or creators, and corrigible to the extent that it allows such interventions on its operation. For example, a corrigible cleaning robot might update its objective function from "clean the house" to "shut down" upon observing that a human user is about to deactivate it. For an AI system operating highly autonomously in ways that can have large world-scale impacts, corrigibility is even more important; this HCAI research category is about developing methods to ensure highly robust and desirable forms of corrigibility. Corrigibility may be a special case of preference inference.

Reasoning about highly capable systems before they exist requires certain theoretical assumptions or models from which to deduce their behavior and/or how to align them with our interests. Theoretical results can also reduce our dependency on trial-and-error methodologies for determining safety, which could be important when testing systems is difficult or expensive to do safely. Whereas existing theoretical foundations such as probability theory and game theory have been helpful in developing current approaches to AI, it is possible that additional foundations could be helpful in advancing HCAI research specifically. This category is for theoretical research aimed at expanding those foundations, as judged by their expected usefulness for HCAI.

Incorporating more human oversight and involvement "in the loop" with an AI system creates unique opportunities for ensuring the alignment of an AI system with human interests. This category surveys approaches to achieving interactivity between humans and machines, and how it might apply to developing human compatible AI. Such approaches often require some degree of both transparency and reward engineering for the system, and as such could also be viewed under those headings.

If we ask a robot to "keep the living room clean", we probably don't want the robot locking everyone out of the house to prevent them from making a mess there (even though that would be a highly effective strategy for the objective, as stated). There seem to be an extremely large number of such implicit common sense rules, which we humans instinctively know or learn from each other, but which are currently difficult for us to codify explicitly in mathematical terms to be implemented by a machine. It may therefore be necessary to specify our preferences to AI systems implicitly, via a procedure whereby a machine will infer our preferences from reasoning and training data. This category highlights research that we think may be helpful in developing methods for that sort of preference inference.

As reinforcement-learning based AI systems become more general and autonomous, the design of reward mechanisms that elicit desired behaviours becomes both more important and more difficult. For example, if a very intelligent cleaning robot is programmed to maximize a reward signal that measures how clean its house is, it could discover that the best strategy to maximize its reward signal would be to use a computer to reprogram its vision sensors to always display an image of an extremely clean house (a phenomenon sometimes called "reward hacking"). Faced with this and related challenges, how can we design mechanisms that generate rewards for reinforcement learners that will continue to elicit desired behaviors as reinforcement learners become increasingly clever? Human oversight provides some reassurance that reward mechanisms will not malfunction, but will be difficult to scale as systems operate faster and perhaps more creatively than human judgment can monitor. Can we replace expensive human feedback with cheaper proxies, potentially using machine learning to generate rewards?

AI systems are already being given significant autonomous decision-making power in high-stakes situations, sometimes with little or no immediate human supervision. It is important that such systems be robust to noisy or shifting environments, misspecified goals, and faulty implementations, so that such perturbations don't cause the system to take actions with catastrophic consequences, such as crashing a car or the stock market. This category lists research that might help with designing AI systems to be more robust in these ways, that might also help with more advanced systems in the longer term.

If an AI system used in medical diagnosis makes a recommendation that a human doctor finds counterintuitive, it is desirable if the AI can explain the recommendation in a way that helps the doctor evaluate whether the recommendation will work. Even if the AI has an impeccable track record, if its decisions are explainable or otherwise transparent to a human, this may help the doctor to avoid rare but serious errors, especially if the AI is well-calibrated as to when the doctor's judgment should override it. Transparency may also be helpful to engineers developing a system, to improve their intuition for what principles (if any) can be ascribed to its decision-making. This gives us a second way to evaluate a system's performance, other than waiting to observe the results it obtains. Transparency may therefore be extremely important in situations where a system will make decisions that could affect the entire world, and where waiting to see the results might not be a practical way to ensure safety. This category lists research that we think may be useful in designing more transparent AI systems.

Some approaches to human-compatible AI involve systems that explicitly model humans' beliefs and preferences. This category lists psychology and cognitive science research that is aimed at developing models of human beliefs and preferences, models of how humans infer beliefs and preferences, and relevant computational modeling background.

Codifying a precise notion of "good behavior" for an AI system to follow will require (implicitly or explicitly) selecting a particular notion of "good behavior" to codify, and there may be widespread disagreement as to what that notion should be. Many moral questions may arise, including some which were previously considered to be merely theoretical. This category is meant to draw attention to such issues, so that AI researchers and engineers can have them in mind as important test cases when developing their systems.