AI Risk Is a Communication Problem

We usually treat AI risk as a technical problem. But many failures happen through communication. In what systems tell us, what they leave out, how people interpret their outputs, and whether warnings reach someone who can act.

Back to Blog

I think we are looking at AI risk too late

Systems surround much of the conversations when we talk about AI risk. We ask whether a model is accurate, biased, can be manipulated, hallucinates, whether private information is leaking, someone can jailbreak it, and whether it behaves safely under adversarial conditions.

These are quite frankly important concerns. I spend a great deal of time thinking about them myself. But of lately I have been wondering whether our understanding of AI risk starts too late. Because by the time we are asking whether a model is biased, hallucinating, or leaking private information, the system has already been deployed. It has already entered workplaces, hospitals, schools, government agencies, financial institutions, newsrooms, and homes. It has already produced outputs that someone has to interpret and act on. Someone trusted information they should have questioned. A warning failed to reach the person who needed it. An uncertain prediction appeared more authoritative than it really was. A decision-maker misunderstood what a model could do. A worker noticed something was wrong but did not have the institutional authority to stop deployment. A system technically disclosed its limitations, but in a way nobody could reasonably use.

From where I sit, these do not look like conventional model failures. They look like communication failures. Thus, I have come to an alternate viewpoint that AI risk is also a communication problem. This is not to say that technical risk does not matter. But AI systems do not produce consequences by themselves. Their outputs enter relationships among people, organizations, institutions, and other technologies. Somewhere between what a system produces and what eventually happens, information has to be interpreted, trusted, contested, communicated, ignored, escalated, or acted upon. In those in betweens, risk abounds.

A model output is not yet a social consequence

Imagine that an AI system produces an incorrect answer. At the model level, we might call this a hallucination, and hallucination at that point is not yet a social problem. It becomes a social problem only when someone encounters it and interprets it in a way that produces consequences. They may trust it, reject it, verify it, misunderstand it, forward it to someone else, use it to make a decision, or feed it into another system.

The same incorrect output could therefore disappear without consequence in one setting and produce substantial harm in another. But if the model output stays largely the same with varying outcomes, then something outside the model is doing important causal work.

Communication is part of that process.

This is one reason I find the sociotechnical framing of AI risk so integral. In their paper “Concrete Problems in AI Safety”, Dario Amodei et al identified problems like avoiding negative side effects, reward hacking, scalable supervision, safe exploration, and robustness to distributional shift. The paper became foundational partly because it helped interpret broad concerns about AI safety into concrete technical research problems.

But once AI systems leave experimental settings and enter workplaces, hospitals, schools, government agencies, financial institutions, newsrooms, and homes, technical properties interact with social ones. So beyond whether the system fails technically, we also have to ask whether anyone recognizes the failure as a failure.

Uncertainty has to be communicated

AI systems are probabilistic, but the interfaces through which we encounter them often do not feel probabilistic at all. A chatbot does not usually look uncertain. It gives you an answer, good ones, I must add, sometimes, even an excellent one, and sometimes an answer that is completely wrong with exactly the same confidence, fluency, and grammatical composure.

This creates an unusual communication problem. Humans use communicative cues to evaluate other humans all the time. Hesitation tells us something, so does qualification. “I think” means something different from “I know.” A physician saying “the evidence is unclear” communicates something different from one saying “your test confirms this.” Generative AI complicates those cues because linguistic confidence and epistemic confidence are not the same thing.

The system can sound certain without possessing anything equivalent to human certainty.

I read this research on automation which warned about related problems for decades. Raja Parasuraman and Victor Riley’s classic paper, “Humans and Automation: Use, Misuse, Disuse, Abuse”, showed that the relationship between people and automated systems cannot be understood simply by asking whether automation works. People can over-rely on it, underuse it, or use it in ways its designers did not anticipate.

AI intensifies this problem because many contemporary systems communicate through language. The interface can be persuasive, can sound like a human, an expert or a friend. It can sound like a doctor, lawyer, or journalist. It can sound like someone who knows what they are talking about even when they do not. So, just maybe, one AI risk question should be: Does the system communicate what it knows, what it does not know, and when that difference is important

That is partly an engineering problem, but it is unmistakably a communication problem too.

Transparency is useless if nobody understands it

Our default response to these concerns is usually asking for transparency. Disclose that AI was used, publish a model card, document the training process, explain the limitations, provide confidence scores, add a warning.

I support many of these interventions, but I think we sometimes confuse making information available with communicating. They are not the same thing.

A forty-page technical document sitting on a website is technically transparent. Same goes for a model card that only an ML engineer can interpret, a disclaimer that users click through without reading, or a confidence score that a decision-maker does not understand. The information exists, but that does not mean it has successfully communicated anything.

This brings to mind the works of Timnit Gebruet al on “Datasheets for Datasets”, and Margaret Mitchell et al on “Model Cards for Model Reporting”. Both interventions recognized that responsible AI development requires information to travel across boundaries. Dataset creators, therefore, need to communicate information to dataset users, and so does model developers need to communicate intended uses, performance characteristics, limitations, and evaluation conditions to people who did not build the system.

I think their importance go beyond documentation, as it also highlight that in AI governance, Responsibility depends on successful information transfer. Because if someone knows something about the system, someone else needs to know it. But it is the structure connecting those two people determines whether that knowledge becomes actionable.

The person who knows about the risk may not be the person who can stop it

This one is more apparent inside organizations. Imagine an engineer finds out a serious limitation in an AI system scheduled for deployment. What happens next?

Technically, the risk has been identified, but organizationally, almost nothing has happened yet. So, the engineer has to communicate it, and someone has to believe them. In the typical setting, the issue has to move through whatever reporting structure exists. A manager may have to escalate it, and the product team may disagree on its severity, legal may see things differently, while leadership may have commercial incentives to proceed. So, at the end of the day, someone eventually has to possess both enough information to understand the problem and enough authority to act on it.

This is where I think some discussions of AI governance is too fixated on principles and not enough on communication structures. An organization can have excellent AI principles and still be unsafe if bad news does not travel upward. It can have, say, a responsible AI team whose recommendations arrive after product decisions have effectively been made; or a technically sophisticated evaluators whose findings are put across in a dashboard that senior decision-makers barely understand. More generally, it can have a culture in which people are afraid to raise concerns, or even the ones who do are ignored. So you see, the problem is not that nobody knew about the risk, the problem is that the knowledge could not move to the right person at the right time.

Risk communication has been studying this problem for a long time

I believe this is also where AI research can learn from fields that have spent decades studying how people understand hazards. The way people perceive risk, how they respond to uncertainty, and how organizations communicate hazards has been studied in public health, environmental communication, disaster communication, nuclear safety, and other areas. It did not begin with artificial intelligence.

Paul Slovic’s work on “Perception of Risk” showed that public perceptions of risk do not simply mirror technical probability calculations. People respond to characteristics such as unfamiliarity, controllability, catastrophic potential, and dread. This is quite similar to artificial intelligence, and like, I mentioned earlier, different groups may encounter different dimensions of the same technology. Those experiences are sometimes described as misunderstanding, which I am unsure of, regardless, communication is how those understandings meet or fail to.

We need to talk about who can be believed

Communication is also about power. It is about who can speak, who can be heard, and who can be believed. In this kind of conversation, communication rarely occurs between equally positioned people. Who can raise an AI risk, whose assessment is valued, and who gets dismissed as technically uninformed, who has access to the information needed to challenge a system, and who can demand an explanation, and who has enough institutional authority to stop deployment are all questions of power.

All users of a system may encounter different aspects of its failures, but knowledge is knowledge, and power is power (thank you Cersei Lannister). Therefore, communication should not be treated as a final step in AI risk management, for communication is part of how risk becomes known from the get go

Human oversight depends on communication

We hear all the time that humans should remain “in the loop.” This one, I have a controversial take on (coming soon).

Putting a human somewhere in an automated process does not tell us very much. Starting off, what does the human see? What information does the system provide? Can the human understand why the system produced its recommendation? How much time do they have to intervene? Can they override it? What happens when they disagree? Does disagreement carry a professional cost? Does the system communicate its limbo in a form the human can actually interpret?

Take it or leave it, but I think the human-in-the-loop framing is often a red herring. It is not the presence of a human that matters, it is the quality of the interaction between the human and the system. A human who cannot understand, challenge, or meaningfully alter an automated decision is not providing much oversight.

Permit me to tap into the work of Madeleine Elish on “Moral Crumple Zones” who describes situations in which humans positioned around complex automated systems can absorb responsibility when those systems fail, even when meaningful control was distributed much more broadly.

As a communication scholar, I see a communication dimension to this too. For if we want humans to provide oversight, we have to design the informational relationship between humans and machines in ways that make oversight possible. Human oversight is not a person, it is a relationship.

AI risk also travels between organizations

Take some time to imagine how AI systems become embedded in larger technical ecosystems. One company develops a foundation model, another company builds an application on top of it, and a third organization deploys that application. Employees use it to make decisions involving customers, and customers experience the consequences.

Now imagine a vulnerability, bias, data problem, or reliability issue surges up upstream. Who needs to know? How quickly? In what language? With what level of technical detail? Who is responsible for informing downstream users? How do downstream organizations determine whether the issue affects their particular implementation? What happens when the system is updated? So AI supply chains are also communication chains.

This becomes especially important as we move toward increasingly agentic systems in which one model’s output can become another system’s input. A misleading output may not need a human to act on it before producing consequences. Here, machines can communicate with machines without human intervention. Agents can call tools, retrieve information, update databases, trigger workflows, and pass generated information to other agents. Even so, the communication problem is still there even when the human leaves the interaction. It only becomes more complicated.

The Global South makes this more complicated

Lets talk about borders. Many of the most serious AI risks are technical and also social, institutional, and political. The contextual risks of AI systems vary across countries, cultures, and institutions. For example, a system may be considered sufficiently accurate globally while performing poorly for languages or populations poorly represented in its training and evaluation data, it could be safe in one country but not in another because of differences in regulatory regimes, institutional capacity, or cultural norms.

So beyond responding to AI risk from by clamouring for better universal standards, we also have to ask whether risk information survives translation across technical, linguistic, institutional, and cultural boundaries. A safety warning produced in San Francisco does not automatically become meaningful governance in Onitsha or Aba simply because someone published it online.

As information travel, meaning travel with it, and sometimes it does not.

Maybe AI literacy is not enough

Another common answer to these problems is AI literacy. The assumption is that if we teach people how AI works, to not trust everything a chatbot says, to recognize hallucinations, they will be able to interpret its outputs, recognize its limitations, and understand its risks. Again, a very useful idea. I support it. I have taught AI literacy myself, and I think it is useful. But it is rather confining to make literacy our primary solution.

At some point, we may default to transferring responsibility from systems and institutions to users. If an AI system routinely communicates its limitations badly, the answer cannot simply be to train millions of people to become better interpreters of unclear or unreliable outputs. If an organization deploys a complex automated system without meaningful ways for people to challenge its decisions, simply educating users about algorithms is not an adequate solution. Literacy matters, so does design, institutional structure, and power.

Responsibility has to travel in more than one direction.

I think communication scholars belong in this conversation

AI risk has understandably been shaped by computer science, as many of the problems require technical expertise. Security vulnerabilities need technical solutions. Models need rigorous evaluations. Robustness needs to be measured. Dangerous capabilities need to be understood.

My stake here is not to replace any of that with communication theory, but to explore what happens when we acknowledge that technical evaluation cannot tell us everything about whether an AI system will be safe in the world. Because the moment an AI system encounters a person, a team, an organization, or an institution, communication begins doing causal work. People form expectations about the system, interpret outputs, they make decisions, they escalate concerns, they ignore warnings, they trust or distrust systems. Publics communicate back through resistance, adoption, reinterpretation, and demands for accountability. Communication is not a side effect of AI risk, but a part of the infrastructure through which AI risk becomes socially consequential.

So what would it mean to take communication seriously?

Perhaps AI evaluation would go beyond whether a model knows the right answer to whether it appropriately communicates when it does not. Or maybe red teaming would test both dangerous outputs and how people interpret those outputs. Perhaps model documentation would be evaluated with the people (or at least a representative sample of them) expected to use it. That organizations would evaluate whether risk information can actually go from technical teams to decision-makers. Maybe human oversight would be evaluated as an interaction and not just confirmed by the presence of a human somewhere in a workflow. Or that incident reporting would examine who knew what, when they knew it, whom they told, and where the information stopped moving. That global AI governance would expand focus from the existence of standards to how their meaning survives across institutional and cultural contexts. And perhaps communication researchers would stop researching AI primarily as another technology people talk about and begin examining communication as part of the safety architecture. I am especially excited about the last one.

The risk is not only in the model

I am not saying that every AI problem can be solved through better communication. A dnagerous model is a dangerous model. A security vulnerability, bias, or discriminatory system require more than better messaging. Some systems should not be deployed at all. But technical risk and communication risk are greatly intertwined.

A system can fail technically and recover because people recognize the problem quickly and have the information and authority to intervene; while a system can be technically sound and still produce catastrophic consequences because people misinterpret its outputs, ignore warnings, or fail to escalate concerns.So maybe the focus should go beyond asking How risky is this AI system? to also asking What happens to information about risk once this system enters the world? Who sees it? Who understands or believes it? Who can communicate or challenge it? And, ultimately, who has the power to do something about it?

AI risk is a technical problem, it is an organizational problem, it is an institutional problem, and it is a political problem. And much more than we currently acknowledge, AI risk is a communication problem.

Further reading

• Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete Problems in AI Safety.
• Elish, M. C. (2019). Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction. Engaging Science, Technology, and Society, 5, 40–60.
• Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM, 64(12), 86–92.
• Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency.
• Parasuraman, R., & Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors, 39(2), 230–253.
• Slovic, P. (1987). Perception of Risk. Science, 236(4799), 280–285.

You may be interested in

View all posts