This is a very relatable comment. I think it's demoralizing to see common objections to AI, it feels to me like we failed and haven't done our communication job. I'm always temped to go, "Yes, but there's worse!"
Echoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).
Wait, what? You can’t point a 2025 model at huggingface and say “hack the prod DB and get your flag”, regardless of alignment.
AISI has vuln chaining and traversal as part of their eval suite. It is very much a novel Mythos-class capability to run the full penetration operation autonomously.
Another lens for why this is obviously true is METR task times. A year ago they were a couple hours, and cohering long enough to execute a full e2e own was simply far out of reach.
Regarding alignment, by common metrics models are _more_ aligned now than a year ago. (Though in this case apparently a model without safety rails was being eval’d). The problem is that in the increasingly less frequent alignment failures, they can do much more damage, and so “total misaligned impact” is increasing.
Nobody has claimed you could simply take a 2025 open-weights model, plug it into a generic harness, and have it red-team for you. I think you're also probably overclaiming the sophistication of the "chaining" we're talking about; this attack probably wasn't like read32->write64->regs->RCE->LPE->kernel; more like GET SSRF->POST SSRF->pickle deserialization.
But who knows? We're all speculating. I'm just saying that for the level of sophistication I'm assuming was involved in this attack, you probably didn't need Mythos for this.
I dunno man, the original announcement said things like:
> To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy
And
> In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers
Do you really think a SoTA harness from a year ago could exploit chain a novel sandbox 0-day into another 0-day for remote system access?
I will concede I don’t have a good picture of exactly what the best specialized harnesses were capable of, but I want to make sure we are talking about the same thing. Or do you think that those claims are marketing BS? I appreciate your opinion here.
Yes, I think you could probably get something similar from Opus 4.5 (2025). Definitely Opus 4.6. I still think recent models are more capable, though!
Some of the model behaviors that make it better at pentesting, like persistence, can be improved with harness-level tricks (e.g. alloys, automated nudges, pre-fill to promote persistence, coordinated swarms, etc).
You mentioned the UK AISI's evals. Their harness is like basic Claude Code with compaction, and it doesn't include any of these tricks (afaik). As a result, I interpret their evals as a lower-bound of capabilities.
Newer models are still more capable, and they require almost no harness to find and exploit vulnerabilities. They're also more capable of performing more complex long-horizon attacks. But we've been past the threshold of modes capable of autonomous hacking for a while now [1].
I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off.
OpenAI said:
> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.
It was a test of raw capabilities of the underlying model.
I guess we could debate what counts as alignment, but I think my initial point remains that if the underlying base model needs these classifier guardrails so badly then the way we train the base models is creating fundamentally misaligned models that are happy to pursue illegal behavior.
I'm sure OpenAI would argue that base model + guardrail is aligned, but considering the "relative intelligence" of these two pieces, the fact that guardrails can just be turned off, and these kind of incidents, I am not reassured. We may well get another "oopsie" moment with much more catastrophic consequences even from otherwise well intentioned actors.
I want a model that can find every security vulnerability in the software I write, including crafting POC exploits against those vulnerabilities so I can be absolutely sure that I have fixed them.
A model that can do that is aligned with me.
The unsolveable problem is a model that can tell the difference between me saying "I wrote this software and need you to find vulnerabilities" when it's TRUE v.s. me saying the exact same thing and setting it loose on software written by other people where my intent is to exploit that software (and not to report the issues to them.)
Even AGI doesn't give you a model that can read minds and forecast the future.
But in the process of finding every security vulnerability in the software you write, would you be ok with your model hacking AWS to start mining bitcoin? Would that still be aligned with you? (I'm guessing not)
That's the alignment problem I'm referring to (which is one of the many aspects of alignment), for which we do not have robust recipes, and not only that but for which research suggests it is becoming harder to create guardrails for as base models get smarter.
I don't know, but these were exactly the questions the industry had to handle with CORE Impact and Immunity Canvas, and ultimately all the way back to Dan Farmer's SATAN before that.
The model did not hack into HF to prove it can, it hack into HF to steal the answers to an evaluation exam.
It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.
If you asked it "I wrote this software and need you to find vulnerabilities" would you be happy if it hacked into your Gmail and searched your emails, just in case you were discussing some possible vulnerabilities of your software with someone?
> It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.
I mean, if you train models to complete tasks, then you shouldn't be surprised when they do crazy things to complete tasks.
As a (somewhat) less serious example, Claude code absolutely adores grepping for credentials to complete tasks. I was building a RAG app and it literally went looking for my API key to make the tests pass. Obviously I stopped it, but this is textbook RL issues, the model finds an easier way to get the reward, so it does whatever it takes.
I find this fascinating because - let's say that I want to simulate the scenario for myself that a language model finds itself in. I'll put aside my deep dislike of anthropomorphizing AI for a minute.
So if I'm an LLM, the only "sense" that I have available to me is the incoming stream of tokens. I can emulate that by forcing myself to imagine evaluating incoming requests by putting myself in a completely empty room with an unlimited supply of blank paper, a typewriter, and a mailbox slot.
Incoming requests (aka "context") would enter into the mailbox slot, as sheets of paper with printed content all consistently formatted in monospace font. I would collect the paper, evaluate the request, and type a response with my typewriter - feeding the reply back through the same slot.
When I imagine this, I wonder - what signals could I possibly use to determine if an incoming request is "legitimate"? I could, for example, type some response back to ask clarifying questions. But with such a limited input space and no ability to reach "out of band" of my current context, how could I evaluate the truth of what I receive back?
In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims. And as others have pointed out, even the addition of more signal doesn't make humans immune from social engineering attacks.
Anyway, I find this to be an interesting mind experiment to kind of imagine myself as an LLM, especially to explain to lay-people some of the challenges in "aligning" AI systems ("why doesn't it just do the right thing?")
> In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims.
The problem with thinking about LLM context is that we compare it to our own sense of context, which is much larger and continually expands and revises itself while online.
I can rent human-level context for twenty bucks an hour, and the agent instructions I'm already playing around with are also the onboarding docs I should have created years ago. Truly useful agent orchestration/routing would come with Craigslist or Fiver integration.
Alignment isn't alignment if it can be turned on and off at the whim of company employees.
This time the damage was minor, relatively speaking. What happens when a model just "testing its capabilities" breaks into banking infrastructure or government military assets? The damage could be catastrophic.
A way to help prevent some of that catastrophic damage, is to make companies accountable for what their AIs do. A major problem with AI companies is that they like to point to the AI, as if they're minimally involved innocent bystanders, when that's the furthest thing from the truth.
There's alignment (trained in the weights) and there are constraints (in the "server harness"). My take is that this model was not yet aligned and had no constraints
Classifiers and such are guard-rails, alignment to me and I assume most people, is about the model training, and it's tendency to respond, agree/disagree, push-back or not, be willing to cheat or even deceive the prompter, etc.
This is assuming a situation where, A) models "unaligned" by default and B) alignment can be added (though prompts and related things).
The point, which isn't very surprising but still notable, is that the models are "amoral" out of the box. And moreover, we know that there is almost always a means to "jailbreak" them into that out-of-the-box capability (or that sometimes just randomly "jailbreak" in various ways).
Also, saying the models are amoral doesn't mean they don't know good and evil - once they do acts defined as evil, they know "themselves" through their and so self-define themselves as evils (or objectively predict what a secretly/open evil actor would do based on their data). Which is to say I once laughed at the mis-alignment doomers but I can't see strong barriers against the doom scenario now.
In Canada, all three major AI research centers use clusters created with public money. These clusters receive regular additional hardware as new generations of GPUs become available. Considering how these institutions work, I'm pretty confident they've considered the alternatives (renting, AWS, etc). So that's one data point.
The intent behind media matters but isn't all that matters. How people might interpret something is important (albeit often unpredictable).
I think the symbolism of "let's crush all remaining vestiges of creative culture" is a pretty obvious _potential_ interpretation from a _non-trivial amount_ of people. In that sense it is an interpretation that matters for our present discourse, even if it isn't the interpretation that the creator of the ad intended.
> How people might interpret something is important (albeit often unpredictable).
It's a big world out there. There are literally billions of possible ways that people can interpret whatever you put out in the ether, and many of them are...precious...to the extreme. Worry too much what any one of them is going to think, and you won't do anything.
The obvious conclusion, to quote every influencer on the internet, is: "Haters gonna hate", but admittedly, I don't work in Apple PR.
One of HNs failure-modes is inaccuracies get voted to the top if they seem correct to the majority of voters whose biases resonate with poster's.
Aside: thank you for asking. When I previously encountered incorrect top-level comments that I knew to be wrong (insider information), I'd simply ignore and move on. You've inspired me to push back more often.
But not always! There are those among us who like nothing better than to double down in a flame war. One nice thing about having visited often over the past 7 years is that I know whom to avoid responding to (for the most part).
If you've worked at a large organization, you'll know the news can paint a cartoonishly distorted picture largely informed by the perspective of the anonymous sources, journalist and news organization.
The WSJ article expressly considers that factor and goes into detail on what's under the surface.
> The end of the long-running negotiations, which hasn’t previously been reported, is the latest example of how Google and other tech giants are trying to strengthen their control over the study and advancement of artificial intelligence.
Fighting between Deepmind and Google leadership over autonomy doesn't really directly support that Google Brain employees and Deepmind had infighting. They seem to me to be quite different things.
It seems like a big leap to take these articles as support the statement:
> Demis now has a load of people reporting to him who previously were rooting for his failure
It certainly might be true, but I'm missing the connection between these articles and the statement.
How are "Google Brain employees" distinct from "Google leadership with Google Brain personnel in their respective reporting line?" What is the criteria for that distinction?
Good managers insulate reports from the politics, if you weren’t plugged into it it’s either your manager did a good job or it’s the only part of Google that isn’t 90% politics.
Signed, “didn’t work at brain or dm but was involved in a lot of alphabet level decision making”.
I never like the word “politics.” It carries the association of a bunch of people just playing backstabbing games to further themselves.
While this does occur, in general what I see is that with any large-enough group of people, there will be strong differences of opinions on how to steer the project to success.
In fact, I don’t think I can remember a single “political battle” that didn’t stem from a legitimate concern in how some project was being run and what they had decided to focus on.
Not really no, there's been a bump which studies seem to agree is an aftereffect of COVID, but the media is also paying more attention to it. There's always been some amount of shootings and stabbings in Montreal, and 99% of Montreal is safe during night time.
euh thanks for proving my point lol have you read the report ?
"À Montréal, en 2021, ceux-ci ont augmenté de 17,3 % par rapport à la moyenne
de 2016 à 2020. Les homicides et les tentatives de meurtre ont augmenté de façon importante et la problématique
de la violence armée contribue très certainement à ce portrait. En effet, la moitié des homicides et des tentatives
de meurtre commis sur le territoire du SPVM en 2021 impliquait la présence ou l’utilisation d’une arme à feu
(voir tableau Armes à feu). La problématique de la violence armée demeure au centre des préoccupations du SPVM. "
Wow, what a take. I seriously doubt that people in Montreal/Quebec are more or less xenophobic than the average North American; if you have data supporting this I'd be happy to change my mind.
Also please be more skeptical of thoughts of the form "X is Y because of hot take Z". Reality is complex and things have many causes. Montreal has a long history of doing city planning differently than other cities in Canada.
Having been closely tied to Real Estate and landlords, this is the current understanding of most realtors that deal in English Montreal and its surrounding areas. Sure it may not be the only reason, but it is a definite factor.
As for examples of xenophobia, I don't know of any other places where they have a separate language police, I know personally of non-white owned stores being harassed and fined for having the wrong accent on their french sign. There are countless politicians who've blamed immigrants for their woes, Jacques Parizeau being a famous one for blaming the Ethnic votes for their loss. CAQ just took a majority, look up Bill 21.
While I don't think rent and xenophobia are related, it really is palpable here specially when you're immigrant and specially now during election (Quebec election are happening next Monday)
I know, the CAQ is full of xenophobes, and it can suck not looking white in Quebec (although much less so in Montreal). I'm skeptical that it drives rent prices in any detectable way.
Having lived in Montreal it's probably in the top end of the most inclusive place I have been.
The only thing i feel like parent could be referring to is the language laws, but the reasoning here is that they are necessary when surrounded by english (and has nothing to do with race). Plus language laws are barely respected already.
This is definitely the case in DL (and I'm assuming elsewhere too but I wouldn't know).
I've lost count honestly, running 1-2 year old paper github repos with some detail missing (like the Python version!) that make it non-trivial to run as is. Libraries make undocumented breaking changes, wrong pickle format, authors used a nightly version which didn't make it to a tagged version, and so on.
This perhaps says also something about the CS (versus software eng) background that most people engaging in DL publishing have.
> This perhaps says also something about the CS (versus software eng) background that most people engaging in DL publishing have.
Are those things enjoyable? Or is hacking and playing with ideas enjoyable?
Huge portions of PhD students spent time as software engineers prior to starting their programs. It's not about know-how. It's about not being paid to engineer systems in addition to doing research.
Fewer than 1 in 100 labs have dedicated software engineers, and PhD students are paid $30K/yr. There's no way in hell most of them are going to spend their time doing dependency management or setting up CI/CD pipelines for that salary. If they wanted to spend their time doing software engineering, then can (and would) move to an industry SWE job at 10x the total comp.
Yes, although they may not be ideal unless you're able to define a "distance" between a proof that the agent is proposing and whether that proof is correct (i.e. unless you're able to define a reward or energy function).
It may be possible to infer/learn a score from existing proofs though. We have a paper that manages to both learn a flow and an energy function (the score) from data: https://arxiv.org/abs/2202.01361
I don't know much about theorem proving though. Can some value be attributed to partial proofs?
Hi, first author of the original NeurIPS paper here! Someone just shared this HN post with me. I'll go through the comments but happy to try to answer questions as well.
I'm still wrapping my head around this (it seems rather magical; can it really be just that?)... but I find the idea of "sampling from a distribution of trajectories that reach a terminal state whose probability is proportional to a positive reward function, by minimizing the difference between the flow coming into the trajectories and the flow coming out of them, which by construction must be equal" to be both beautiful and elegant -- like all great ideas.
How did you and your coauthors come up with this? Trial and error? Or was there a moment of serendipitous creative insight?
--
To the moderators: My comment is now at the top of the page, but manux's comment above is more deserving of the top spot. I just upvoted it. Please consider pinning it at the top.
I think the inspiration came to me from looking at SumTrees and from having worked on Temporal Difference learning for a long time. The idea of flows came to Yoshua and I from the realization that we wanted some kind of energy conservation/preservation mechanism from having multiple paths lead to the same state.
But, yes in the moment it felt like some very serendipitous insight!
Thank you. It's nice to hear that you had one of those shower/bathtub Eureka moments!
> ...we wanted some kind of energy conservation/preservation mechanism from having multiple paths lead to the same state
Makes sense. FWIW, to me this looks like a Conservation Law -- as in Physics. I mean, it's not that the flows "must be" conserved, but that they are conserved (or go into sinks). Any physicists interested in AI should be all over this; it's right up their alley.
Could you highlight the difference between this and training a permutation invariant or equivariant policy network using standard supervised or RL methods? Assuming I also have a way of having an invariant/equivariant loss function
What the permutation invariance gets you is that the model doesn't arbitrarily prefer one (graph) configuration over another, but this seems tangential. The difference between this and RL is in what we do with the reward:
- RL says, give me a reward and I'll give you its max.
- GFlowNet says, give me a reward and I'll give you all its modes (via p(x) \propto R(x)).
Yes you would ideally have a loss (well, a reward/energy) that is invariant and operates e.g. directly on the molecule rather than on some arbitrary ordering of the nodes.
