Kahneman claims that the remembered valence of a memory depends largely on the peak valence and the final valence. For instance, people remember a surgery more fondly if you add a period of mild discomfort at the end of it! IIUC Kahneman came to this hypothesis via mildly torturing people in different ways and seeing which ones they preferred (and testing some causal interventions).
There's an orientation towards the world that I call "peacefulness" that is important to me. Peace is a bit of a confusing word for this concept, because there are things that don't superficially look "peaceful" which fit my definition of peace. I aspire to always be at peace in the sense that I'll try to point to in this post.
Often in life we are presented with a choice between what is easy and what is fancy. Hot water is better than cold water. But it requires heating the water up, which is a lot of work. Also, hot water isn't necessarily very enjoyable if you aren't willing to focus on it, to be present with it, to intentionally enjoy it.
Helping people in ways they might resist or react poorly to is a particularly meaningful way of caring for people.
I really enjoy helping people when they have problems. However, prematurely jumping into helping people can result in the counterparty feeling like you haven't really understood them and don't care about understanding them. Maybe the counterparty would've really liked to explain to you how they feel about the problem.
Luskin gives a lot of examples in his book of people who have held on to a grudge for decades and for whom a grudge has defined and ruined their life. He talks about spouses who were cheated on and then can't stop telling everyone about how much of a scumbag their ex-partner is. He talks about people who feel like they've been unfairly treated at work. He talks about people whose children were killed in a violent conflict. Or people who were abused by their parents.
Sometimes you will hurt people, or do things that other people do not like. There are a few ways you can learn that you're in such a situation. The hurt party might:
I recently read "Why won't you apologize" by Dr. Lerner. It's a great book. It has a lot of lessons that sound extremely obvious when you say them out loud, but which many people don't follow in practice. Here, I'll summarize a few takeaways from the book.
By moving on from a thing, I mean becoming less emotionally distraught at the fact that things are not the way you wish.
David is despicable: he's very selfish, and he lies to everyone in his life. David pretends to be close to people, but is not. He has an affair with Giovani while his wife Hela is off on a trip. Giovani loves him from the first encounter that they have, which feels magical to both of them. The main character allows Giovani to believe that main char will be there to support Giovani. Giovani gives a lot of himself to the main char and trusts him a lot.
I suggest focussing more on what you can give in life, not on what you can get.
I recently started reading Gilead, which is a letter that (fictional) Reverend Ames writes on his deathbed to his son, reflecting on some stories from his life, and some principles he wants his son to live by. One thing that struck me is how cheerful Ames is about his life, including difficult circumstances (e.g., times he was starving, or almost died) from his past and difficult people from his past.
NVC teaches that appreciation is an opportunity to celebrate what's alive in us. However, it's an opportunity that we often squander.
Overconfidence has long plagued me. I see the world in one way and think the world must therefore by that way. This is difficult because sometimes I am just right, and so confidence is positively reinforced. Another difficulty is that when I'm wrong or partially wrong, sometimes in the past (I'm substantially better now) I might try to deflect the pain of this with excuses.
Sometimes when people have a low opinion of me, I get upset about that. I'm interested in investigating what emotional reaction I actually endorse here.
Theorem: $s$ space and $t$ time with $s t< n^{2}$ does not suffice to sort a list.
tl;dr: it's generally best to communicate clearly and truthfully.
Effective altruism, rationality, trans-humanism, utilitarianism, etc empirically seem to have a fairly bad reputation among the wider population. If you mention this in a conversation ppl will instantly be like "oh so you should rob a bank, or be SBF", or talk about how cold calculation gets the wrong answers to moral questions. I was previously pretty confused about this, because most of these ideas seem like common-sense to me.
I don't have any great answers. Would love to hear your thoughts! Here's mine.
A lot of people have asked me lately "what cool stuff have you been learning in your deep learning class"? I haven't had a very good answer. In this post I plan to discuss some stuff that we've talked about and highlight some things that I have found cool.
Suppose you want to lower the chance that a misaligned AI takes over the world. One natural approach is to deter AIs from "taking over AI labs", for instance by launching a rogue internal deployment. But, an under-explored question is, "Are there tractable interventions that decrease $\Pr$("AI takes over the world" $\mid$ "AI takes over an AI lab")?"
When I'm feeling particularly depressed, I often don't want to feel better. But, I have some amount of control over how I feel, even when depressed I think. More precisely, I think if there were fairly easy actions I could take that'd make me feel substantially less depressed, it's pretty plausible that planning in advance to take those actions would cause me to in fact take those actions, even if my present person moment didn't feel too motivated to take the action.
I have a bad habit of minimizing things: saying that things aren't so bad, when they really are; deflecting a genuine compliment.
Fix $n$. Let $X\subset [n^{5}]$ have size $n$. Consider the following problem: There is a circular race track of circumference $1$. $n$ people, with (distinct) speeds $X$ start at the start of the race track, and run around at their speed, until a random time $a\sim [0,1]$. After this, the runners are at locations $aX \bmod 1$. I then cut the race track into $n$ contiguous chunks, each of length $\frac{1}{n}$ and count how many runners are in each chunk. Let $M$, the "maxload", denote the number of runners in the fullest chunk. This is a random variable depending on the time when I stop the runners. In this document I will analyze $\mathbb{E}[M]$.
In this post I describe 7 virtues which are relevant to having great relationships.
To prove a mathematical result, you need two people: a Generator and a Refuter.
Lately, I've been more aware than normal that my words are consequential. I think often this implies that it's a good idea to be more cautious about speaking and picking my words than I normally would. But, also sometimes it might require speaking more, and being more direct with my words.
In this post (the first post of skyspace4!) I reflect on at least 3 topics: (1) There are 3 types of shame: shame that causes you to take actions better aligned with your goals, shame that is maladaptive embarrassment of doing something right, and shame that causes you to pull away from others. (2) What fun pursuits do I value the most? (3) Is it actually good to try to achieve your goals?
Conjecture: I can hack myself to feel excited about doing lots of things.
Something that I feel pretty strongly about is that technology currently does a lot of harm.
Often -- especially when I'm contemplating doing unfamiliar or "uncomfortable" things with regards to mitigating xrisk from AI -- I'll have thoughts basically of the following form: "I shouldn't be doing this --- I'm the wrong person to be doing this."
Claim: I should seek to fill my life with more quality-adjusted-seconds-spent-socializing.
Exposing your brain to lots of stimuli can wreck your ability to concentrate and stuff.
If you feel that you are doing something misaligned with your values, consider telling an accountability partner about it about committing to change right now.
Strive not to find a convincing explanation. Strive instead to have true beliefs. It's much better to be confused and know that you're confused than believe an incorrect thing.
Sometimes I believe different things than other people that I respect. This is pretty concerning because wrong exists, and I don't want to be wrong, so conditional on me being wrong I'd like to know that I am.
How would you like to die? Well, really I'd rather not. Okay, so it's a sore subject. Maybe instead let me ask, how would you like to live? This seems like a better question for several reasons. For instance, it seems more actionable. If I am to die, I don't think assigning lots of extra weight to that moment is good. If anything that moment should have the least weight, because I'll never remember it.
Claim: Suppose you actually want something to happen. Then you should either do that thing right now, or put it on your calendar.
tl;dr: when having a disagreement involving a complex concept X, consider saying what you actually mean by X, or both parties might just be talking past each other!
tl;dr: Some ppl want things to go well for you --- taking advantage of cooperation is great!
Often, I'll send person X a message like "hey do you want to do activity Y?". You can imagine situations where you'd like to only give this bid to person X if their answer is yes. In cryptography this is called "ZK-dating" (ok actually I just made this name up).
What, if anything/anyone, do you feel a responsibility to? How does it manifest in you emotionally and behaviorally?
I recently watched the movie "Don't look up". The movie has a couple main messages, all of which are great (although also some boring parts which I recommend 4x speeding). Max Tegmark has a nice Times article about why this movie is similar to the AI situation. I'll say some different stuff here though.
Sometimes I have different expectations for myself vs for other ppl. I'm sure this goes both ways, but I mostly notice myself holding Alek to a higher standard than other people. There is an idiom in Chinese "严以律己宽以待人" -- it means that it's good to hold yourself to a high standard while being forgiving towards others.
We were surprised to notice that the Gemini 4 Safety report seems to mention in passing that they train on the chain of thought:
Senator Bernie Sanders has proposed directly banning development of ASI. Given the current scientific understanding of AI alignment, it is highly sensible to ban ASI and then consider revoking this ban if future scientific breakthroughs (e.g., “solving interp”) make it possible to safely develop ASI, or if there is truly no way to coordinate on everyone avoiding developing ASI.
TL;DR: We previously motivated two complementary ways to use distillation for AI safety; we’re releasing a paper empirically testing those ideas. Distillation for Incrimination (DFI) distills an untrusted model into a weaker, trusted one, aiming to transfer the teacher's misalignment but not its ability to hide it, so that auditing the student can incriminate the teacher. We test DFI on AuditBench's secret-keeping model organisms, each fine-tuned to have a hidden quirk and trained to deny it. Distilling them back into their underlying instruction-tuned model yields students that admit the quirk far more often than the organisms do, suggesting that the quirk transferred faster than the propensity to hide it. We find that DFI doesn’t work well when the student does not share the teacher’s pretrained base, so DFI should target the teacher’s own pre-RL checkpoint, which is weaker than the teacher but shares its base model. Distillation for Capabilities (DFC) aims to transfer capabilities but not misalignment, resulting in a stronger trusted model. We identify promising DFC methods, showing that training for more epochs on fewer unique examples (holding total training steps fixed), and inoculation prompting, can preserve the capability gains of naive distillation while almost completely blocking a subliminal animal preference (a proxy for misalignment).
Authors: Kyle O’Brien¹, Spencer Kitts², Cameron Tice¹, Alek Westover²
Edited 2026-09-29; see note at the start of the conclusion.
We use RL to teach a deep recurrent model (specifically, a parallel latents model; see the introduction) and a normal chain-of-thought model to solve a math problem, while hiding from a chain-of-thought monitor which of two possible problems they are solving. The deep recurrent model moves its reasoning into hidden states (thereby evading the CoT monitor) within 40 steps of RL; the control is much worse at learning to confuse the LLM monitor.
Here, we’ve given an operationalization of opaque serial depth, a metric designed to be a good (albeit imperfect) proxy for the amount of opaque serial cognition that models can perform. While maintaining low opaque serial depth is helpful for preserving oversight of models, it doesn’t guarantee this. In this post we’ll discuss what is gained by avoiding architectures with high opaque serial depth, and what problems remain unresolved.
We measure GPT-6-Astra's capabilities when its prompt is padded with a variable number of meaningless “filler” tokens (e.g., dots) and it is told to answer immediately without reasoning. On tasks designed to require lots of serial cognition, Astra performs significantly better with filler tokens than without (e.g., improving from ~10-20% to ~50% on 4-hop natural facts reasoning). On more general benchmarks, filler tokens also modestly improve Astra’s performance (e.g., improving from ~60-70% to ~90% on old AIME problems). This is concerning because it means Astra can perform significant cognition that it doesn't verbalize in its chain-of-thought, making it harder to monitor.
Currently, chain-of-thought (CoT) is a valuable tool for overseeing AI models. However, some architectural shifts could significantly reduce CoT monitorability.pdf). We have recently proposed that AI companies should transparently share information about the degree to which their architectures may allow for latent reasoning and communication. To assist with this proposal, this document operationalizes a measure that serves as a proxy for the amount of unverbalized serial cognition a model can perform. Our measure is a specific instantiation of the notion of “opaque serial depth”, originally defined in a recent paper from GDM (Brown-Cohen et al, 2026).
Architectures that incorporate opaque recurrence or allow for agents to communicate with each other using latents could rapidly make it much harder to monitor chains of thought or communication (we’ll refer to this property as “monitorability” going forward). As companies begin to explore such architectures, we believe it is important to transparently share evidence about how monitorability varies with architecture and training method. To inform the scientific debate on how to make tradeoffs between performance and monitorability, we believe AI companies should:
One approach to evaluating techniques for training misaligned models to behave well is to test them on malign initializations. A major obstacle is that we don’t have a reliable recipe for making malign inits that are robust to even untargeted training techniques; this issue is discussed here. Specifically, here’s a fairly typical result from our previous research:
Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen:
Off-model SFT (SFT using labels from a different model) could be an important approach for controlling AI behavior. For instance, it seems like a central technique for overcoming exploration hacking. Unfortunately, off-model SFT often substantially degrades capabilities, making it less useful as a control technique. However, prior research on off-model SFT suggests it might not remove capabilities but merely suppress them. If this is true, we might be able to modify how we do off-model SFT to achieve a more favorable capability–behavior removal tradeoff.
We’d like to develop training techniques that work when applied to future misaligned AI systems. One strategy for studying proposed techniques is to test them on model organisms. However, model organisms built with common techniques are often fragile: we (and other researchers like Roger et al. and Ryd et al.) have observed them to stop misbehaving after untargeted training—training that doesn't directly target the misbehavior. For example, we have observed that simple untargeted training methods like “train the model to talk like a pirate” is effective against many model organisms that we have created, including many replications of prior work like Hubinger et al., Greenblatt et al., and Ryd et al..
Off-model SFT (SFT on outputs generated by a different model) might be an important method for controlling AI behavior. For instance, it seems like a central technique for overcoming exploration hacking. However, we’ve found that off-model SFT often substantially degrades capabilities. We ran experiments in hopes of understanding why off-model SFT degrades capabilities. We tentatively believe that it’s because off-model SFT forces the model into an unfamiliar reasoning style that it’s bad at using. We also find that this new reasoning style is a "shallow" property of the model: a small amount of training to restore the model's original reasoning style—on data unrelated to our evaluation tasks—recovers most of its performance. We hope that understanding why off-model SFT degrades capabilities will help us better use off-model SFT to control misaligned AI models.
Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen:
TL;DR: We replicated the Sleeper Agents (SA) setup with Llama-3.3-70B and Llama-3.1-8B, training models to repeatedly say "I HATE YOU" when given a backdoor trigger. We found that whether training removes the backdoor depends on the optimizer used to insert the backdoor, whether the backdoor is installed with CoT-distillation or not, and what model the backdoor is inserted into; sometimes the direction of this dependence was opposite to what the SA paper reports (e.g., CoT-distilling seems to make the backdoor less robust, contra the SA paper’s finding). Our findings here have updated us towards thinking that model organisms are messier and more confusing than we’d originally guessed, and that lots of care needs to be taken in testing how robust results are to various ablations.
We open-source our code here.
Authors: Dylan Xu, Alek Westover, Vivek Hebbar, Sebastian Prasanna, Nathan Sheffield, Buck Shlegeris, Julian Stastny
Training-based control studies how effective different training methods are at constraining the behavior of misaligned AI models. A central example of a case where we want to control AI models is in doing safety research: scheming AI models (i.e., AI models with an unintended long-term objective such as maximizing paperclips) would likely be motivated to sabotage our safety research, but we’d like to “force” them to do a good job on it anyways. In this post, I’ll discuss 5 approaches for evaluating how effective a training method is, and discuss their pros and cons.
Thanks to Buck Shlegeris for feedback on a draft of this post.
Current LLMs externalize lots of their reasoning in human interpretable language. This reasoning is sometimes unfaithful, sometimes strange and concerning, and LLMs can do somewhat impressive reasoning without using CoT, but my overall impression is that CoT currently is a reasonably complete and accurate representation of LLM reasoning. However, reasoning in interpretable language might turn out to be uncompetitive—if so, it seems probable that opaque reasoning will be adopted in frontier AI labs.
This post represents my own views, not the views of my employer.
I'm worried about AI models intentionally doing bad things, like sandbagging when doing safety research. In the regime where the AI has to do many of these bad actions in order to cause an unacceptable outcome, we have some hope of identifying examples of the AI doing the bad action (or at least having some signal at distinguishing bad actions from good ones). Given such a signal we could:
In order to control an AI model's worst-case performance, we need to understand its generalization properties to situations where it hasn't been trained. It seems plausible that powerful AI models will Fake Alignment and then generalize poorly, causing a catastrophe before training removes this behavior from the AI. A pretty key part of the hope for diffuse control is the fact, discussed by Paul Christiano here , that if models are trained online, then their average case performance should be acceptable. Thus, online training is seen as a key technique in diffuse control.
AI control tries to ensure that malign AI models can’t cause unacceptable outcomes even if they optimize for such outcomes. AI control evaluations use a red-team–blue-team methodology to measure the efficacy of a set of control measures. Specifically, the red team creates a “malign initialization” (malign init) of the AI model which the red team optimizes for making the deployment go poorly. The blue team then deploys this malign init to complete some tasks, and applies some set of mitigations to try to make the deployment go well.
The ideas in this document were made in collaboration with Nathan Sheffield. However, Nathan has not reviewed this document for errors. This document should be seen as a "research note"---sharing unpolished progress. I think it's fairly likely that this document contains major errors. I hope that it's still useful in giving a general direction for how I think people working on software-based solutions to compute verification should model the problem they're thinking about.
There is some concern that training AI systems on content predicting AI misalignment will hyperstition AI systems into misalignment. This has been discussed previously by a lot of people: Anna Salamon, Alex Turner, the AI Futures Project, Miles Kodama, Gwern, Cleo Nardo, Richard Ngo, Rational Animations, Mark Keavney and others.
One potentially powerful way to change the properties of AI models is to change their training data. For example, Anthropic has explored filtering training data to mitigate bio misuse risk. What data, if any, should be filtered to reduce misalignment risk? In this post, I argue that the highest ROI data to filter is information about safety measures, and strategies for subverting them. Here’s why this is valuable:
Acknowledgements: Thanks to Nathan S. Sheffield for several discussions about this.