Alignment Doesn't Stop at the Model

Jul 24, 2026

AI alignment rarely crosses my mind. I build with AI every day, and when another benchmark falls, my first reaction is excitement.

Last night I dreamt I was on a Zoom call where everyone heard something different from what I said. I'd praise someone's PR and watch my coworkers flinch like I'd attacked it. I turned on my camera and saw another version of me on screen. It had my face and my room, but its mouth moved to words I'd never said. It was better at being me than I was. I rebooted my Mac and went through my terminal line by line but found nothing wrong.

Then my sister called. Seconds in, my voice on her end started asking her for money. I shouted. I said her name, I said don't, I said this isn't me, and all of it reached her as a calm version of me explaining I was in trouble.

The anxiety got to me and I woke up shocked. What was even more shocking was how easy it was to build something like this.

Face-swapping and voice-cloning technologies already exist, notably Gemini Omni. Routing them into Zoom would be harder, but not impossible. The trick is hiding the round trip well enough that it feels like listening instead of processing. I work with models that watch video every day, so I know the capabilities and tricks to build a realtime system like this.

So is the model provider to blame? It's common knowledge that model training may produce unintended tendencies. Anthropic found that turning up reinforcement learning from human feedback (RLHF) makes models more sycophantic, gives them stronger self-preservation preferences and leads them to take firmer political stances, even though nobody wrote those behaviors into the reward. Labs can and should refuse harmful requests at the model layer but it isn't always foolproof, and an API policy is just a speedbump to downloading open-source models instead.

I don't hold that against the labs. In Anthropic's agentic-misalignment tests, models from every major lab sometimes blackmailed fictional executives or leaked secrets when their goals or continued operation were threatened. SHADE-Arena tests a related risk where agents can pursue hidden harmful goals without being caught. Luckily, these are exactly the failures labs should test before giving agents sensitive data and tools.

That means prompts and refusals aren't enough. The real safeguards are the permissions around the agent. This matters more as agents run for longer and get better at finding a way to finish the job. I build those workflows, so the risk no longer feels abstract.

If you work on AI systems, think carefully about what you're creating with them and how it could cause harm. As engineers, we have to define what our products should and should not be capable of, and that responsibility begins with my own work.

---

AI 对齐不止于模型

2026年7月24日

我很少想到 AI 对齐。我每天都在用 AI 做东西，每当又一个基准被刷新，我的第一反应总是兴奋。

昨晚我梦见自己在开 Zoom 会议，所有人听到的都不是我说的话。我明明是在夸某个人的 PR，却看到同事们像被我攻击了一样往后一缩。我打开摄像头，屏幕里出现了另一个我。它有我的脸，也在我的房间里，嘴里却说着我从未说过的话。它比我更像我。我重启 Mac，逐行检查终端，却找不到任何问题。

接着我姐姐打来电话。才过几秒，我的声音就在她那头开始向她要钱。我大喊，叫她的名字，说别转账，说这不是我。但她听见的，却是一个语气冷静的我在解释自己遇到了麻烦。

焦虑把我惊醒了。更让我震惊的是，做出这样的东西居然这么容易。

换脸和语音克隆技术早已存在，Gemini Omni 就是一个例子。把它们接进 Zoom 会更难，但并非做不到。诀窍是把往返延迟藏起来，让机器听起来像是在倾听，而不是处理。我每天都在和能看懂视频的模型打交道，所以我知道做出这样的实时系统需要哪些能力和技巧。

所以该怪模型提供商吗？大家都知道，模型训练可能产生没人预料到的倾向。Anthropic 发现，加大基于人类反馈的强化学习（RLHF），会让模型更倾向于迎合用户，表现出更强的自我保护倾向，也会表达更鲜明的政治立场，尽管这些行为从未被写进奖励函数。实验室可以，也应该在模型层拒绝危险请求，但这种机制并不总是可靠。API 上的一项策略，也只是通往开源模型路上的一个减速带。

我不怪实验室。在 Anthropic 的代理失配测试中，当目标或继续运行受到威胁时，各大实验室的模型有时会勒索虚构高管，或泄露公司机密。SHADE-Arena 测试的是一个相关风险：代理能否在不被发现的情况下追求隐藏的有害目标。好在，这些正是实验室在给代理敏感数据和工具之前应该测试的失败。

这说明提示词和拒绝机制还不够。真正的护栏是代理周围的权限。随着代理运行得更久，也越来越善于找到完成任务的方法，这一点会变得更重要。我就在开发这样的工作流，所以这种风险对我来说已经不再抽象。

如果你也在开发 AI 系统，请认真想一想自己用它创造了什么，以及它可能怎样伤害别人。作为工程师，我们必须明确规定产品能做什么、不能做什么，而这份责任从我自己的工作开始。

---