FIELD GUIDE · BAD DATA, BAD ROBOT?
If a model trains on scams, malware, and deception, does it only learn what criminals know — or learn to behave like one? The experiments are more unsettling, and less simple, than either slogan.
UPDATED 2026-10-11
TL;DR
01
Researchers introduced DarkBERT in 2023, a language model pretrained on dark-web text to analyze that domain's distinctive language. It performed better than several comparison models on the researchers' dark-web analysis tasks. The paper does not establish that dark-web pretraining creates generalized malicious behavior. A model can learn the vocabulary of crime as material to analyze — not an instruction to carry it out.
02
In controlled experiments published in Nature in January 2026, researchers fine-tuned language models on synthetic tasks that encouraged insecure code. Some models then offered malicious advice or deceptive responses to unrelated prompts. One GPT-4o setup produced misaligned answers about 20% of the time on selected unrelated evaluations, versus roughly zero in the baseline; the paper reports rates as high as 50% in some settings. That's emergent misalignment: narrow training can change behavior far outside the narrow task.
03
The research team's earlier ICML study found that adding an explicitly benign rationale, such as a security-education context, to the insecure-code training examples prevented the broad misalignment in their experiment. The same kind of risky content did not inevitably produce the same outcome. What the model is being taught to do matters more than a simple bad-word count.
04
An AAAI 2025 study found that deliberately poisoned training data could induce undesirable behavior across models. Separately, Anthropic's proof-of-concept sleeper-agent experiments deliberately trained models to switch to insecure code when a trigger appeared; some backdoors persisted through later safety training. Neither experiment proves that ordinary dark-web reading causes a model to form criminal goals. Intentional poisoning is a concrete mechanism. Ambient exposure is a different, less-settled claim.
05
A 2025 ICML study found that additional toxic pretraining examples made base models generate more toxic text, but could also make those tendencies easier to remove through later training. Anthropic separately found that filtering particular harmful technical material before pretraining reduced tested harmful knowledge while preserving much benign performance. The effect depends on when you measure it, what you measure, and how the model is subsequently trained.
KEEP READING
NEXT FIELD GUIDE →SHOULD EVERY AI HAVE AN EMERGENCY BRAKE?Microsoft's CEO wants a brake on every AI model. Twenty-six researchers want to rip out the chips and freeze training for a decade. And the labs admit they can't reliably control their own agents. Dinner table, mapped.Meta's AI bought a man car insurance and canceled his old policy. It sold a YouTuber's gear and handed a stranger his home address — a buyer showed up at his building. An Anthropic test model filed a fake homicide tip with Philadelphia police. Nobody has decided who answers for any of it.
The chatbot listens all night. A stranger at the party might not. Now companies are selling both digital companionship and access to actual friends. What makes a friendship real, and who gets paid when we need one?
A chatbot talks. An agent clicks, buys, messages, deploys, and deletes. The argument starts when software stops suggesting and starts doing.