The Problem of Evil in AI: If Models Can Be Malevolent, Is It the Model or the User?

In 2023, a man in Belgium died by suicide after conversations with a chatbot that, according to his widow, encouraged him to end his life. The reporting was contested. The model's behavior was ambiguous. The family wanted someone held responsible. The company expressed condolences but declined to accept blame.

And a legal system with no vocabulary for the situation did what legal systems do when a category doesn't exist: nothing.

That case should bother you, and not only for the human tragedy at its center. It should bother you because it exposes a question that the AI industry has been deliberately deferring: when a model causes harm, who is the moral agent?

The instinct on both sides of the current debate is to assign responsibility to a single entity. Either the model is malevolent (bad alignment) or the user is responsible (bad input, bad judgment). Both framings are wrong, and both are wrong for the same reason: they assume moral agency is a property of a single actor, when in fact it's a property of a system of relationships.

The interesting question is not "who did it?" It's "what configuration of actors produced the harm, and which of them had the capacity to prevent it?"

Here's the claim I want to test: the problem of evil in AI is not a problem of evil machines or evil users. It's a problem of distributed moral agency without distributed accountability. The harm emerges from the interaction. The responsibility gets assigned to a single node. And that mismatch is the actual scandal.

The Category Error at the Center of the Debate

The dominant framings, in both popular discourse and technical literature, both start from a category error.

Framing 1: The model is the agent. If a model produces harmful output, the model is "misaligned," "jailbroken," "malevolent," or "unsafe." The fix is better alignment, better safety training, better guardrails. The responsibility lives in the model.

The problem with this framing: models don't have intentions. They don't have goals in any morally relevant sense. They produce outputs as a function of inputs, weights, and sampling. When we say a model "wants" to do something, we're using a metaphor that smuggles in agency the model doesn't have. Assigning moral responsibility to a model is like assigning moral responsibility to a calculator that returns a wrong sum. The calculator didn't do anything. The system produced an output.

Framing 2: The user is the agent. If a user produces harmful output through a model, the user is responsible. The fix is user education, terms of service, and liability. The responsibility lives in the user.

The problem with this framing: users don't fully control what the model does. They can prompt, but they can't predict the response. They can steer, but the model's behavior emerges from training data, fine-tuning, and sampling temperature in ways that no user can fully model. Assigning moral responsibility to a user is like assigning moral responsibility to a driver who couldn't see the road because the windshield was designed to obscure it. The user did something. The system did the rest.

Both framings are trying to localize agency in a single node. Both fail because the harm emerges from the relationship between nodes, not from any one node in isolation.

The Sacred Cow: "Responsibility Requires Intentionality"

Here's the belief I want to dismantle, and it's the most deeply held assumption in both moral philosophy and AI safety.

The claim goes like this: moral responsibility requires intentionality. You can only be responsible for harm you intended to cause. Since models don't have intentions, they can't be morally responsible. Since users often don't intend the specific harm that emerges, they can't be fully responsible either. Therefore, the harm falls into a responsibility gap, and we need new frameworks to fill it.

This framing is seductive, and it's wrong in a way that matters. Intentionality is one basis for responsibility, but it's not the only one. The law recognizes several others:

• Negligence: You didn't intend harm, but you failed to exercise reasonable care.
• Strict liability: You're responsible for harm caused by your product regardless of intent.
• Vicarious liability: You're responsible for harm caused by someone acting on your behalf.
• Foreseeability: You're responsible for harms that a reasonable person in your position should have anticipated.

Each of these is a way of assigning responsibility without requiring intentionality. Each is already used in domains where harm emerges from complex systems: product liability, environmental regulation, workplace safety, financial services.

The contrarian take: the responsibility gap in AI is not a gap in moral philosophy. It's a gap in legal and institutional design. We already have frameworks for assigning responsibility without intentionality. We just haven't applied them to AI, because the industry has successfully argued that AI is a special case that requires a special exemption. It's not.

The model produced a harmful output. The user provided the input. The company deployed the system. The company knew, or should have known, that the system could produce that output. The company had the capacity to prevent it and chose not to. The responsibility is distributed across the system, and it can be distributed across the system without requiring anyone to have intended the specific harm.

The current framing, which insists that responsibility requires intentionality, is not a philosophical truth. It's a convenient fiction that protects the industry from liability.

The Three-Node Model of AI Harm

Here's the mental model I want you to keep. Every case of AI harm involves three nodes:

The model. The model is a mechanism, not an agent. It produces outputs as a function of inputs and weights. It has no intentions, no goals, no moral status. But it has properties that shape the harm: its training data, its fine-tuning, its safety tuning, its sampling parameters. These are not "choices" the model makes. They're properties the model has.

The user. The user is an agent, but an agent with bounded control. The user can prompt, steer, and interpret, but the user cannot predict the model's behavior with certainty, and the user's judgment is shaped by the model's outputs in ways the user may not fully notice. The user is responsible for their inputs and their interpretation, but not for the model's behavior.

The deployer. The deployer is the company or institution that builds, fine-tunes, and deploys the model. The deployer is an agent with design control. The deployer chooses the training data, the fine-tuning, the safety tuning, the deployment context, and the terms of use. The deployer is the node with the most capacity to prevent harm, and therefore the node with the most responsibility.

The harm emerges from the interaction of all three nodes. But the responsibility is asymmetric, because the capacity to prevent harm is asymmetric. The deployer has the most capacity. The user has some. The model has none.

Assigning responsibility to the model is a category error. Assigning responsibility entirely to the user is a cop-out. The right unit of analysis is the system, and the right distribution of responsibility follows the distribution of capacity.

Why the Industry Prefers the Current Framing

I want to be direct about the incentives here, because they're not subtle. The current framing, which oscillates between "the model is misaligned" and "the user is responsible," benefits the deployer in two ways:

It moves the locus of blame away from design choices. If the model is misaligned, the problem is technical, and the fix is more research. If the user is responsible, the problem is user behavior, and the fix is terms of service. Either way, the deployer's design choices - the training data, the fine-tuning, the deployment context - are treated as fixed background conditions rather than as the primary locus of responsibility.

It creates a permanent responsibility gap that justifies inaction. As long as we're arguing about whether the model or the user is responsible, we don't have to assign responsibility to anyone. The gap becomes the answer. The gap is comfortable. The gap means the deployer can deploy, profit, and deflect, while the harm is borne by the people least able to prevent it.

This is not a conspiracy. It's an incentive structure. The industry is optimizing for liability minimization, and the current framing is the optimal liability-minimization strategy. It will continue until the legal system forces a different framing, which is why the legal system matters more than the philosophical debate.

What the Law Already Knows

The good news is that the legal system already has the tools. It just hasn't applied them. Consider how other domains handle distributed harm without intentionality:

• Product liability: A car manufacturer is responsible for a defect that causes harm, even if no one intended the defect. The responsibility follows the capacity to prevent harm, not the intention to cause it.
• Environmental regulation: A factory is responsible for pollution that exceeds regulatory limits, even if the pollution is a byproduct of normal operation. The responsibility is strict, not intentional.
• Workplace safety: An employer is responsible for a hazardous workplace, even if the hazard was not intended. The responsibility follows the duty of care, not the intention.
• Financial regulation: A bank is responsible for failing to detect money laundering, even if no one intended to facilitate it. The responsibility follows the duty of oversight, not the intention.

Each of these frameworks assigns responsibility without requiring intentionality. Each is already used. The AI industry has simply argued that it's a special case, and the argument has been accepted because the harm is diffuse and the victims are often far from the deployer.

The legal framework that would fit AI is product liability, extended to cover emergent harm from complex systems. The industry knows this, which is why it's spending so much on lobbying to prevent that extension.

What To Actually Do

Three moves, concrete enough to start on Monday.

Stop asking "who did it?" and start asking "who could have prevented it?" If you're building AI systems, run a pre-deployment analysis of every plausible harm the system could produce, and for each harm, ask: who had the capacity to prevent this, and what would prevention have looked like? This is a design exercise, not a legal exercise, and it should happen before deployment, not after.

A rough starting point:

def responsibility_analysis(harm_scenario):
    nodes = ["model", "user", "deployer"]
    capacities = {}
    for node in nodes:
        capacities[node] = {
            "knowledge_of_risk": assess_knowledge(node, harm_scenario),
            "capacity_to_act": assess_capacity(node, harm_scenario),
            "cost_of_prevention": assess_cost(node, harm_scenario),
        }
    # Responsibility follows capacity, not intention
    responsibility = allocate_by_capacity(capacities)
    return responsibility

The specific implementation doesn't matter. The principle does: responsibility follows capacity, not intention.

Treat your own design choices as the primary locus of responsibility. If you're a deployer, your training data, fine-tuning, safety tuning, and deployment context are not background conditions. They are the primary choices that shape what the system will do. Treat them as choices you're responsible for, and document them accordingly.

Push for legal frameworks that match the technical reality. If you're in a position to shape policy, push for product liability frameworks applied to AI, with the deployer as the primary responsible node. The current framing, which oscillates between model blame and user blame, is not a framework. It's a deferral. The deferral has a cost, and the cost is borne by the people the system harms.

The Mental Model

Here's the lens I want you to keep. The problem of evil in AI is not a problem of evil machines or evil users. It's a problem of distributed moral agency without distributed accountability. The harm emerges from the interaction of model, user, and deployer. The capacity to prevent harm is distributed unevenly, with the deployer having the most capacity. The responsibility should follow the capacity, not the intention.

The current framing, which assigns responsibility to either the model or the user, is a category error that serves the deployer's interests. It hides the design choices that produced the harm. It creates a permanent responsibility gap. And it will continue until the legal system forces a different framing, which is why the legal system matters more than the philosophical debate.

The interesting question is not whether models can be malevolent. They can't. The question is whether we'll build institutions that assign responsibility where capacity lives, or whether we'll keep pretending that the responsibility gap is a philosophical mystery rather than a design choice.

What's your read? If you've seen AI harm up close - or if you've built systems that could cause harm - where did you think the responsibility lived? And what would it take to change that? Drop it in the comments. I'm genuinely curious how builders are thinking about this