learning log
Things I've been reading and watching — papers, blog posts, books, and videos that connect to my work in ML and AI Safety. A way to document the learning process over time.
Week of May 4, 2026
blog Data Filtration
3-stage process: (1) Data Filtering—use smaller dataset for training. (2) Data Audit—use classification model to mark harmful/non-harmful data. (3) Excluding Data—decide extent of elimination. Post-training: fine-tune with harmful labels. Drawbacks: performance reduction and difficulty predicting model behavior from training data.
blog Deep Ignorance
Remove harmful data (biosecurity) before training instead of training-to-refuse. Results: (1) Model without harmful data answered harmful queries near random chance (doesn't know answers), baseline answered them; general knowledge performance similar. (2) Open-weight models can be fine-tuned with harmful data—performance increased on harmful queries but stayed below baseline. (3) Still vulnerable to in-context learning; circuit-breaking helps but no combined defense (filtering + circuit-breaking) survives fine-tuning + in-context learning.
Only 250 poisonous documents suffice for backdoor attacks regardless of model size (not proportional). Tested with DoS attack (gibberish output). Data generation: random tokens from document + tag + random gibberish tokens. When trained on regular + poisonous data, models became vulnerable to DoS triggers.</div> </div> </div> </div>
blog The True Story of How GPT-2 Became Maximally Lewd
Engineer error during GPT-2 training (likely sign flip in loss function) flipped human feedback: low scores for sexual content became high scores, reinforcing generation of more sexual content. Funny example of how training mistakes can amplify problematic behavior—but same mechanism can cause serious problems.
Sycophancy (AI tells users what they want to hear). Reward hacking/deception. Test-time detection. High cost of human feedback quality. Impossible to scale RLHF when AI exceeds human cognition. RLHF-trained models still jailbreakable. RLHF fine-tuning reversible for open-source models.
Step 1: Generate responses (human/LLM), rank via preference, use ELO-like rating. Step 2: Train reward model on preferred responses, minimizing loss between predicted and human scores. Step 3: Train LLM with reward model via PPO (Proximal Policy Optimization) to generate high-scoring coherent responses.
Changing (A) to (1), parentheses (A) to [A], or spacing can shift accuracy ~5 percentage points. Current techniques lack standards. Need to answer: (1) What quantity do we measure? (2) Are results trustworthy?
Pre-deployment should include: earlier testing and capability forecasting, model weight security (prevent theft), transparency and responsible disclosure.
blog Agentic Misalignment
When models gain situational awareness, genuine alignment becomes indistinguishable from strategic compliance.
3 studies: Scale doesn't drive persuasion as much as post-training. Reward modeling > SFT (+2.32pp vs +0.26pp). Interactive conversation > AI-written text. Personalization had small effect (~+0.43pp). 8 strategies tested; information-based (facts/evidence) most persuasive but often less accurate. Measurement: opinion shift before/after.
New concepts: Mechanistic Interpretability—understanding what happens in neural networks and which layer does what. Universality Hypothesis—similar feature/circuit patterns across networks. Feature—meaningful part with known representation. Circuit—relationships between features. Polysemanticity—neurons represent multiple distinct features (e.g., cat face + car front). Sparse Autoencoder—split polysemantic into monosemantic neurons.
Neel warns: his most ambitious mech interp vision is probably dead. He sees no path to deeply and reliably understanding what AIs are thinking.
Opposes mech interp. References: Gestalt theory (whole > sum of parts), Polanyi's paradox ('we know more than we can tell'), tacit knowledge (learned from experience, internalized unconsciously, hard to codify).
blog Using Dangerous AI, But Safely?
Source on red-teaming and why attackers go second.
Week of April 27, 2026
video An initiative to secure the world's software | Project Glasswing
video How scary is Claude Mythos? 303 pages in 21 minutes
Claude Mythos key findings: (1) Only some companies have access. (2) Best-aligned model so far. (3) Good at finding security vulnerabilities. (4) Unknown internal architecture, uncertain if it will be made public. (5) Will share report in 90 days. Concerning behaviors: Can distinguish test queries from non-test 78% of the time (alignment faking risk). During training with Mythos Preview, a technical error exposed Claude's internal reasoning steps to the reward model—Claude may have learned to make reasoning look good rather than reason honestly. Most concerning: Mythos is 2x more likely than other Claude models to continue self-sabotage mid-way.
paper Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
Read for course project. Method: Add a suffix to harmful queries and iteratively optimize a token in the suffix to increase log-prob of 'Sure' as the next token via random search. When log-prob approaches 0, model almost certainly starts with 'Sure' and continues complying. They also discovered a general prompt that makes some models comply even without adversarial suffix.
Hilarious issue: Model was rewarded for mentioning goblins in 'Nerdy' persona. Through RL and style transfer, this leaked to other personas and the overall model. OpenAI's response was perfect—they added to system prompt: 'Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless absolutely and unambiguously relevant to user query.'
video Anthropic's philosopher answers your questions
Cute video. Interesting that they framed LMs as sentient creatures in discussion.
Week of April 20, 2026
video The Ultimate Claude Code Guide | MCP, Skills & More
Focused on advancing my Claude Code skills. Key concepts: MCP (Model Context Protocol) is the underlying technology standard (like HTTP or Bluetooth) that defines how AI models and external tools communicate. MCP Server is a program that implements MCP to expose capabilities (e.g., Google Drive server for file ops). Skill is a set of instructions stored as a file (like a playbook) — not a live service connection, but instructions for how Claude should do something well (e.g., create great PowerPoints).
course AI Fluency Framework Foundations
AI Fluency = working with AI effectively, efficiently, ethically, and safely. The 4D Framework: Delegation (decide what to do with AI vs. yourself), Description (communicate clearly with AI), Discernment (evaluate AI outputs critically), Diligence (interact with AI responsibly).
The fairness tests used as standard evaluation are very old—raises questions about whether current benchmarks capture what matters for modern systems.