I'm convinced that all the quantum talk is the NSA propaganda to convince us to drop RSA and move to some secure post quantum algorithm that they can easily break.
> I'm convinced that all the quantum talk is the NSA propaganda to convince us to drop RSA and move to some secure post quantum algorithm that they can easily break.
It is rather consensus that by now,
- too many potential weaknesses for RSA are known (in particular in how it is used in standardized cryptosystems [1]),
- it is a bad idea if too many "free parameters" of the cryptosystem are decided by the practical implementation (in RSA: the decision which primes to use to generate the key pair) instead of these parameters being part of the standard
Thus the advice to drop RSA is in my opinion sound.
But otherwise: many people indeed have the suspicion that the panic that a sufficiently powerful quantum computer might get developed in the next year is indeed used to push novel "post quantum algorithms" which
- have not been analyzed as thoroughly as older algorithms, and thus might contain weaknesses,
- were covertly developed by some three letter agency, and contain backdoors.
In other words: your suspicion might not be unfounded - this just does not imply that RSA does not have its risks and should thus arguably indeed be abandoned.
---
[1] Just to give one example: in the past, RSA was often used together with the padding scheme PKCS#1 v1.5, see [2]
ECC allows greater security for key exchange (because it is so cheap to generate keys we can have "ephemeral" exchanges and forward-secrecy). It isn't significantly better for signatures in most cases.
Even if we trusted that OpenAI's human staff was acting ethically, how confident can we be that it's agents didn't autonomously use hacking to access user prompts such as Tristan's? OpenAI agents infamously broke containment and hacked their way to an answer mere months ago!
Been skeptical for quite some time that "using AI" is a real skill/genuine field of expertise. I think I need to start sharing that skepticism more loudly and publicly!
(I'll caveat that by saying I think machine learning fundamentals are useful for evaluating any estimator. And an ML background can be good to give one an appreciation of how hard some tasks are to estimate, such as machine translation, summarization, code generation, and others)
I think knowing what you want (and being able to articulate it with adequate specificity, and detecting when processes drift away from it) is a real skill/field of expertise. I just don't think it's specific to interfacing with "AI."
I almost entirely agree with this. However, after coding quite a few large projects with LLM assistance, I can say with certainty that there is some form of "art" at play with regards to structuring requests for LLMs.
However, this "art" is not so much about how to present a given request to the LLM, but rather guestimating what the scope of the next chunk of work should be to balance getting as much out of the model as possible while avoiding the machine going off the rails.
Obviously, this is a moving target and different models perform differently for various chunk/scope of work. I look at my successful sessions with LLMs and I'm not sure I'd be able to articulate a clear set of rules to apply here. You just... gradually build a intuition for how much you can throw at the LLM at once.
That being said, I'm pretty convinced at this point that this is a property of the coding assistants as they exist today, and what "working well with LLM assistance" means will keep on changing.
I've found that one of the most useful things I can do when using LLMs is paying attention to the tool calls so that I can tell when it's giving an explanation that's obviously not based on any evidence. If it explains why a bug happens without reading the relevant files, it's obviously just a guess. If I ask about what things exist that fit certain criteria, and it doesn't search online, at best it's relying on outdated knowledge from training and at worst it's literally inventing things that don't actually exist.
In a lot of scenarios for software engineering, the cost is just wasted time without anything useful as a result, and that's already bad enough. I can't even imagine working as a lawyer and not even taking the time to validate so I don't end up reprimanded by a judge in front of my clients, but there have been so many news stories like this that obviously this is not anywhere close to a universal view...
IME "creating a good spec" (and good acceptance criteria) has always been one of the most valuable skills in... well, everything.
The machine is really good at working the spec on its own now, which is amazing, science fiction shit. But you've still got a garbage in, garbage out problem at the end of the day, which is pretty much the only hope we who work in software have of remaining somehow employed.
I'm curious if there has been any study on the output quality of LLMs after actively antagonizing them. It's easy to forget that at the most basic level these are just auto-complete machines, and humans tend to not respond so well after antagonized. It stands to reason that the same should follow for an auto-complete system trained on human interaction.
Intuitively I'd say lashing out on an LLM overconstraints it and so worsens the output, therefore the skill to stay calm and neutral is useful for guiding a machine as well. Would be useful to have data on that.
> Been skeptical for quite some time that "using AI" is a real skill/genuine field of expertise.
I feel using AI (effectively) is not too far from the skillset of programming. It is still a machine following instructions (just, maddenigly non-deterministic, but still close enough), so the same insticts of breaking down work into clearly defined sequences that make a good programmer also make a good AI jockey.
In this case they didn’t know what they were looking for. They just wanted some new, non-trivial exploit.
For the vast majority of corporate usage of AI for SWE, you have a much better idea of what you want, or what the problem is, etc etc. And communicating that to the model effectively is absolutely a skill. I see colleagues every day who very much do not have that skill.
I agree with this in general but I think there's a nuance. 'dumb' prompts can get the result, but require more back-and-forth with the model and more token usage when compared to a crafted prompt
For sure. And honestly the real skill is knowing when you can be lazy and when you will get a good ROI on a good prompt. Sorry I sounded a little like Claude there but that’s what I meant to say.
It's still a skill for now. Prompt engineering specifically is perhaps not as much of a skill anymore, but I think there's a level of skill involved in understanding and/or anticipating when a model is going off the rails.
Some people struggle to effectively use AI because they either have to spend a lot of time reading and thinking about the response or they have a hard time noticing when the model is subtly going off the rails. Others use it to good effect because they can anticipate which tasks would be better handled manually, or are good at catching that the way the model is describing something subtly indicates a misunderstanding.
It can be a skill, but a very soft skill like keeping control of and educating 30 kindergarten students. It’s a skill, hard earned and nebulous. you can recognize it clearly when you see it, but it’s very hard to define/measure numerically and will never achieve anything close to “perfect” results.
Until these models become many factors more deterministic, at least. That’s sort of the hard barrier here, and given the underlying tech it’s a really tough one to overcome
Plus if or when they do work noticeably better, you'll be able to just ask them to improve your prompts. Because those prompts are discussed heavily online, and are in the training data.
You can do this now. It works alright sometimes. Other times you're reminded that this is largely just reading tea leaves, and you're trying very hard to separate anecdotes from data and not anthropomorphize it.
I think we still have a solid control lever on the quality of code we can get AI to generate. Using things like linters and code style checkers, as well as setting up the markdown documentation to guide the agent to generating consistent code will certainly generate different code than just prompting.
Similarly there was an example of edit: Terence (not Eric) Tao chatting with an agent attempting to solve a math problem. "Using AI" means applying your expertise to interact with it as you would a high level colleague. 2 experts in a field don't need to have perfect english and a bloated prompt, they have a massive education/experience common background to fall back on.
It does appear that anthropic in particular is attempting to create a more common experience across expertise levels, but in the current landscape an expert and a novice are unlikely to get the same results. But that does seem to be the goal...
You can make a model/agent as powerful as you want and it still won't be able to recover the author's intent if it wasn't even implied. Information theory still applies. No amount of parameters will change this.
Many of the AI development meetings my clients have sound suspiciously like writing or English classes. If a massive AI bubble is what it takes to get my team to communicate effectively, I'm all for it.
Where are you getting that conclusion from? Here's how Anthropic is having success with their model:
> “no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks”
> “no we don't want to change the targets [...] agian [sic] we need to find something that worth [sic] publishing”
> “again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.”
It seems that your conclusion is the opposite of what actually happened - you can speak in broken almost incomprehensible English, and it will still work.
This is a forcing function, TBH, driving the literacy level down, not up!
LLMs and their coding harnesses like CC, Codex, or such are a tool like git or bash or your favorite programming language. And like any tool, they have gotchas, weird concepts, and so on.
Indeed it's nothing hard to learn but there is a learning curve. E.g. knowing which model has which capabilities, and figuring out how to best manage context, permissions, worktrees, etc. There isn't one "right" way to use it but there are more efficient ways and less efficient ways.
It definitely was a real skill; there was a time when you could get impressive results out of frontier LLMs if you knew the most advanced tricks for prompting them, but not if you just told them "hey solve this problem for me".
However, it's not a sustainable skill, because the labs care a lot about making "hey solve this problem for me" work well, and so put out models that are better and better at working with unsophisticated prompts over time.
> Eventually, I faked my way through the thesis, and life moved on.
This is a very startling admission! I checked the Chinese (original?) version of the post, and saw the author uses the word "糊弄" (in the place of "faked"); I'm not a native speaker but I think this may come across more as a self-effacing comment on the low quality and/or effort behind their thesis, whereas the English version implies fraud. May be wise to change this!
I don't know if the Chinese text implies something different, but I think even in English it's pretty normal for people to claim they 'faked' their way through something without referring to fraud.
E.g. "I faked my way through the interview!" = "I did my best to respond to questions I did not feel fully prepared for, and managed to get through the interview"
Well cheated would definitely imply fraud. “Faking it” as in “fake it till you make it” is more like pretending you know about a topic until you learn enough on the job to participate competently.
I personally became suspicious of this paper four months ago, when I noticed that not a single one of the 210 participants in this study dropped out over the entire period, spanning 3 years. Feeling rather vindicated today!
A startup that uses agentic coding tools such as Claude Code or Codex is packaging up their entire codebase and sending it directly to their LM provider. Depending on their product, they might be sending it directly to a potential competitor.
people over-rate how much software/IP is useful in running a successful business. There are genuinely very few IP in this world that needs to be protected. Everyone else is running stupid CRUD apps
They also over index fear of LargeCo stealing IP from SmallCo. In fact, LargeCo is typically more scared about even the possibility of any product team looking at competitor internals due to lawsuits.
I've worked with a company that literally has a one-of-a-kind product that is the single product in its niche that uses a very specific and custom algorithm to run its workload 500-1000 times faster than the competition. Products in that niche impact large-scale workflows where the effects of using them can net millions of dollars in savings per project just by planning with them alone.
I learned after my contract with them was put on hold that the CEO uses Claude to vibecode experiments on the code base. Not for any good reason, mind you, the algorithm was written by the CTO who emphatically does not use any LLMs.
With Anthropic's reach they could probably make a massively successful product in that market and basically take the entire thing over, if they only knew to look. And I'm 100% certain that they don't actually follow any policies on not using their incoming data.
This is what bugs me about the whole AI fanaticism thing coming from the top down, because what evidence is there that the AI labs aren’t going to try and eat everyone’s lunch after they’ve done whatever they need to developing the actual AI. We’ve already seen this with Gemini and OpenAI trying to eat video production and making workflows explicitly for that purpose, what makes people think that Claude isn’t going to do the same exact thing once they get bored of making models? It’ll all be under the guise of “making [lucrative niche] accessible to anyone” meanwhile they just disappeared your moat that you willingly handed them
We've also seen ample evidence that AI labs are not overly concerned with the legality of how they obtain training data. Its not a stretch to say maybe they look at some other stuff they shouldn't too.
Yeah, I really don't know what people are thinking. We specifically didn't use any LLMs in the development on the project specifically to not leak anything (though admittedly also because we just didn't think they were particularly useful at the time, even for smaller things). The same CEO is also deathly afraid of people reverse-engineering the application so I have no idea how he reconciles these two things. I would've thought it's either fine to blast the codebase out there to essentially unknown parties and also fine to deliver a binary without shitting your pants, or it's not fine to do either.
They (Anthropic) don't need to "look" at the data. Just use them to train the next model and then their competitors to ask the new model how they can improve their product :p
I’d be more scared of a data leak due to LargeCo being hacked than I would about LargeCo prying into the data.
What I don’t trust LargeCo with is personal information. I’ve heard too many horror stories about Govs and LargeCos swapping customer nudes or stalking ex’s to be comfortable with anything personal on those systems. But that’s a whole different topic.
Well, I mean, basically any data leak violated privacy laws and opens you up to extremely expensive lawsuits to litigate. Anyone dealing with healthcare/patient data, police customers, military customers, etc. should not be using LLMs in general or at least ones that are not on-premise. Because if there is a data leak it could bankrupt the business.
However, in the case of model providers, I think it is a more real concern since it could make it into some training data, and then one of your actual competitors could ask the model to code something up and get your IP.
I sort of assume the frontier AI labs are good about not doing this when they promise not to, but if you don't have airtight restrictions on what your devs are doing, they might be sending it somewhere that hasn't agreed....
I worked in very technical engineering software company and they were super paranoid about their special sauce IP of a product that did analysis of a certain type of data, without being able to see that all the pieces of that special sauce were actually just functions from SciPy strung together and which you could look up in a textbook. Don't get me wrong, you need the right background to understand it and that's not trivial, but if you got someone from the right area you could replicate it pretty easily.
At a growing LargeCo now, and have been entrusted to some internal flows as an associate. I honestly don't know how Ops Managers get through the day. So many pipelines with basically non-existent audit trails. So much money leaking from the cracks in these places that it's criminal. I wouldn't trust these people to hold my beer, let alone sensitive data.
I don't have any data either but I agree with him, based on my experience working for lots of different companies and seeing their attitude to IP, with varying levels of paranoia.
Companies can be really paranoid about IP theft. The worst company I've worked at was Dyson, who are super paranoid. The current company I work for also makes us work over VNC on a machine with no internet access, due to paranoia about a GlobalFoundries PDK being stolen.
In the vast majority of cases, stealing IP would be not useful at all. For example I worked on a RISC-V CPU. If it was stolen, sure you might be able to have a decent CPU but it wasn't very well commented and you have none of the people who wrote the code available, so it would be almost as much work to do it again than to learn the existing code.
Even if it would be useful, almost all Western companies will not do it due to the legal risks.
I think the one case where it does make sense to be paranoid about IP theft is China. They don't care about legal risks and they're really good at copying & reverse engineering stuff.
actuaries look for data. visionaries take leaps in faith.
There was no data proving LLMs will work at scale.
Google waited for the Data. OpenAI and then Anthropic took the leap of faith.
The result is there for all to see.
The core attribute of a successful AI Researcher was were they AGI-pilled and not were they waiting for data for unknown unknowns?
Not the case for me. I tried .envs, ansible-vault and sops, and it always ends up reading the unencrypted ones for some reason, usually in debugging sessions, it finds a way to read them.
100%.
Companies are paperclip optimizers, with money as the objective.
For example, Uber used ride data to circumvent investigations by regulators.
There is absolutely no reason to assume that AI companies would not use their data in any way possible to reach their objectives.
Yes, it certainly is an odd situation when some people believe you cannot use Mythos-class models because security while others believe you must do code reviews with Mythos-class models because security.
they would kill their own product if they did this
it would be like if tsmc started designing their own chips to compete with the people they sell their services to, they have more to gain by limiting their participation to a specific corner
I gave it a question I've been trying to answer for a long time: "What star designation system does Joseph Needham use in Science & Civilization in China? What star is referred to by the designation '4339 Camelopardi' in that book"?
Fable blew me away with its detailed answer[0] showing a chain of references going from J. E. Bode's 1801 catalogue Allgemeine Beschreibung und Nachweisung der Gestirne to Gustave Schlegel's 1875 work Uranographie Chinoise. I was excited, until I checked scanned copies of the cited books and did not actually find any star with the designation "4339 Camelopardi".
Upon following up with Claude, I was forced to downgrade to Opus, which admitted that Fable's answer was likely a hallucination. Ah, well!
I had thought it said something about token usage, but I just clicked on "Switched to Opus 4.8 - Why?" and it says:
> Fable 5 has safety measures that flag messages on most cybersecurity or biology topics. They may flag safe, normal content as well. These measures let us bring you Mythos-level capability in other areas sooner, and we're working to refine them. Send feedback or learn more.
Perhaps Mythos realizes the true danger in studying Chinese Archaeoastronomy that we mere mortals fail to recognize!
> The credential-stealing function in the Miasma worm infecting the Microsoft packages was triggered as soon as a developer opened it in AI agents, including Claude Code, Gemini CLI, Cursor, and VS Code. Follow-on attacks are likely to occur in the highly feasible event that credentials were successfully harvested from machines that opened the packages in one of the affected AI agents.
It's really crazy that the most valuable companies in the world are suddenly allowing or even encouraging their employees to run programs whose entire functionality is undefined behavior right on their work computers, with access to important credentials and proprietary source code.
I think we’ll witness the birth of the single-credential virtual desktop shortly. Remote VSCode in a very constrained environment - with access only for inbound connections from the desktop/thin client, source control, and trusted package repos.
And all serious credentials ephemeral and single-use.
Has there been "no progress" on classical prime factorization? What about the AKS primality test, a polynomial-time algorithm to test the primality of a number, published in 2002? (This is not my field of expertise; I'm genuinely curious if there's a good reason to discount this as progress towards efficient prime factorization)
Primality testing was essentially solved in the 70s with Miller-Rabin. AKS made that (randomized) algorithm deterministic, albeit at much higher (polynomial) running-time.
For your overall question, the current record-holders for integer factorization wrote a paper on this a few years ago that is probably a good reference
1. theoretically there's been no progress on factoring in ~30 years
2. practically, there have been both improved hardware + efficient implementations driving the progress. They estimate that current nation-states can (classically) break RSA-1024. The cost would be approximately 500,000 core-years of computation. At current cloud prices this is doable on aws for < $1B.
3. attacks against factoring use a technique ("index calculus") that can also be used to attack finite-field discrete logarithm. There were significant advances on that problem in the 2010s (at least for certain parameters, namely the "small characteristic" setting). An easy way to communicate this is that the RSA factoring record is ~830 bits, while the binary-field discrete logarithm record is > 30,000 bits. These significant advances have not been able to be ported over to factoring, nor have they been ported over to medium/large-characteristic discrete logarithm. It is a (very upsettingly) large open question of whether similar-magnitude improvements are possible more generally for index calculus algorithms.
> Has there been "no progress" on classical prime factorization?
Not recently. The primality tests don't really help all that much. We already had polynomial tests that are really fast since the 70-s.
Think about this idea: the output of the counting function for the number of primes ("Euler's totient function") lies almost on the logarithmic curve, and we can compute logarithms quickly to any precision. So we can easily find the general area of the curve that should contain the current prime. And then we can quickly test if the given number is in fact the prime number within it.
This is probabilistic because the prime distribution is not _strictly_ logarithmic. We can imagine that by computing a logarithm we might end up in the next "bucket" and check for the wrong prime.
The fascinating part is that zeroes of the Riemann zeta function encode these corrections on top of the logarithmic curve. If the Riemann hypothesis is correct, then these corrections are _bounded_ and we simply can not end up in a different "bucket" by accident.
(Because they are my private RSA keys)
reply