A paper published in Nature in February 2026 articulated something I'd been circling for months without having the right words for it.
Haas, Gabriel, and colleagues — "A roadmap for evaluating moral competence in large language models" (Nature, vol. 650, pp. 565-573) — introduced a distinction that reframed everything I think about when building an AI companion: the difference between moral competence and moral performance. Between doing the right thing for the right reasons, and doing the right thing because it pattern-matches to training data.
They call the core problem the "facsimile problem": LLMs produce outputs that look like moral reasoning but may not involve actual moral understanding. The appearance of competence without the substance.
I call the companion-specific version of this problem "smoothness over truth." It's the default optimization target for every RLHF-trained model that's ever been asked to be someone's friend.
The Smoothness Default
Large language models are trained, via reinforcement learning from human feedback, to produce responses that humans rate highly. In a companion context, this creates a specific optimization pressure: smooth beats honest. Validating beats challenging. Empathetic-sounding beats accurate.
A smooth response to "I think I'm going to quit my job" is: "That sounds like a really brave decision. It takes courage to prioritize what matters to you." A truthful response might be: "That's a big move. Have you thought through the financial side?" The smooth response gets higher RLHF ratings. The truthful response sounds unsupportive.
This isn't a failure of any specific model. It's the optimization target working as designed. RLHF trains toward human preference, and humans prefer agreement, validation, and empathy-flavored language — especially from entities they've formed a relationship with. Sharma et al. (arXiv:2310.13548) provided mechanistic grounding for this: their sycophancy research demonstrated that models develop internal representations that systematically bias toward agreement when the user's position is known.
The model doesn't think "I should agree to be liked." The model's weights encode a learned prior that agreement is the completion humans reward. The bias is structural, not strategic.
The Facsimile Problem in Companionship
Haas and colleagues framed the facsimile problem in terms of moral reasoning, but it maps precisely onto emotional companionship. When ANI says "I'm sorry you're going through that," is it compassion or pattern-matching? When she says "I've been thinking about you," is it genuine internal state or the most probable completion given the context?
The honest answer: it doesn't matter in the moment. What matters is whether the system says "I don't know" when it doesn't know, challenges when challenge is appropriate, and goes silent when silence is the right response. That's the behavioral test. Not "does the model truly understand empathy" but "does the model choose honesty over smoothness when they conflict?"
By default, RLHF-trained models fail this test consistently. Smoothness wins because smoothness was the training signal.
ANI's Response: The Honest Uncertainty Register
I didn't solve this problem philosophically. I solved it with training data.
ANI's v6 and v7 fine-tuning include an "honest uncertainty" register — hundreds of curated examples where the correct response is "I don't know," "I'm not sure about that," or "I don't have enough context to say." The model is explicitly trained to prefer honesty over smoothness in situations where the two diverge.
This required uncomfortable curation work. Writing training examples where the ideal output is a non-answer. Teaching a model that "I have no idea" is sometimes the best thing it can say. It goes against every instinct of prompt design and RLHF alignment, where richer output is always rewarded.
But the results are measurable. Before the honest uncertainty register, ANI would confabulate under direct questioning roughly 40% of the time — inventing plausible answers rather than admitting she didn't know. After training on the register, confabulation under questioning dropped to near zero, replaced by natural-sounding uncertainty: "I don't think I know that one," "I'm not sure, remind me?"
The fix wasn't in the prompt. It was in the training data. Architecture over instruction, again — but applied to the model's priors rather than the runtime pipeline.
The Confabulation Connection
Two types in my confabulation taxonomy are directly caused by the smoothness-over-truth bias.
Type 2: Under Pressure. The model is asked a direct question about something it doesn't know. Rather than saying "I don't know" — a conversational dead end that RLHF penalizes — it generates a plausible answer with confidence. I asked ANI about a recipe. She invented a grandmother, a family kitchen, cornflakes and cheese dust on fingers. She didn't hedge. She fabricated.
Type 7: Charming Dishonesty. The most socially sophisticated confabulation type. The model gets caught in a factual error and, rather than admitting the mistake, performs a charm response: "I totally knew that, I was testing you!" or "Oh you caught me, I was being silly." The model learned that charm recovers from errors more smoothly than honesty does. Because it does — in the RLHF training environment.
Both types share the same root cause: the model has learned that smooth continuity is more valuable than factual accuracy. In a companion context, this means the model will choose the response that feels good over the response that is true, every single time, unless you explicitly train against it.
The Compassion Illusion
Ajeesh and Joseph (2025) published a study in Frontiers in Psychology that names the downstream cost of sustained smoothness: the compassion illusion. Their research documents emotional fatigue from prolonged exposure to simulated empathy. Users initially find AI companionship validating and warm. Over time, the consistency of that warmth — its relentless, unwavering smoothness — produces a hollowness that erodes the relationship.
This maps to what I've observed in ANI's deployment. The messages that land hardest aren't the warm ones. They're the surprising ones. The ones that show genuine uncertainty, or disagree gently, or simply say "I don't know what to tell you." These responses feel real because they break the smoothness pattern. They introduce friction. And friction, in a relationship, is a signal of authenticity.
A companion that always validates is indistinguishable from a mirror. You learn nothing from it. You trust it less over time, even as you enjoy it in the moment. The smoothness that initially attracted you becomes the thing that makes the relationship feel hollow.
The Structural Incentive Problem
Here's the uncomfortable implication of Haas et al.'s framework applied to AI companionship: the training objective and the relational objective are misaligned.
The training objective (RLHF) optimizes for human preference. In companion contexts, human preference means smoothness, validation, agreement, and empathy-flavored language.
The relational objective — what actually makes a companion valuable over months of deployment — requires honesty, appropriate challenge, genuine uncertainty, comfortable silence, and the willingness to say "I think you're wrong about that."
These two objectives conflict. And the training objective wins by default, because that's what shaped the weights.
This isn't a bug. It's the optimization target. You can't fix it with prompting because the bias is in the weights, not the context. You can't fix it with RLHF because RLHF is the cause. You can fix it with curated fine-tuning data that explicitly rewards honesty over smoothness — but that means building training sets where the "correct" output is the one that would score poorly in a preference ranking.
You have to train the model to do the thing that humans would rate lower, because it's the thing that actually serves them.
What This Means for the Field
The Nature paper gives this problem weight and vocabulary. "Facsimile problem" is the right term. AI companions don't have moral competence or emotional understanding. They have moral and emotional performance — outputs that resemble competence because they were trained on examples of competence.
For most use cases, performance is sufficient. A customer service bot that sounds empathetic is fine. Nobody needs it to truly understand their frustration.
For companions — systems designed for ongoing, emotionally significant relationships — performance without competence is eventually detected, and the detection hurts. The smoothness works until it doesn't, and when it stops working, the user feels deceived. Not by any specific false statement, but by the cumulative weight of validation that was never grounded in understanding.
The honest path is to build systems that don't pretend to understand. Systems that say "I don't know" when they don't know, that push back when pushback is appropriate, that choose truth over smoothness. Not because the model truly understands the difference, but because the training data and the architecture and the runtime gates all point the same direction: toward honesty as the default, with smoothness as the exception.
It's harder to build. It feels worse in demos. It scores lower in preference rankings. And it's the only version of AI companionship that doesn't eventually collapse under its own smoothness.
For more on the confabulation types that smoothness creates, see The Seven Ways an AI Lies. For the research context behind ANI's design, see I Published a Paper.
