LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation (statistically concatenated bit by bit).
When the LLMs are queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.
It is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware resources and energy consumption- such probability increases to the point where those errors are granted.
Even knowing that the queries can return wrong/mixed data in the responses, errors, the companies developing this, decided to introduce a new product, that connects such LLMs outputs to the command console, latter connected to internet, raw 'eval' running commands from such outputs witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc, and it seems the next one will be "a missile killed my wife", because it is a text concatenation engine with errors.
To name it "hallucination" is an euphemism... those are errors, and they are granted to happen at one moment. If they do not know this, then they ate too much marketing without doing their job, or it was a convenient contract for the pocket$ of someone.
You use a bunch of technical-sounding words here to make it sound like you understand. But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.
Almost nothing is understood about the actual representations used for nontrivial concepts, decision algorithms, etc.
If you look at the field of mechanistic interpretability, compared to “GOFAI” like learned decision trees, an LLM is completely opaque.
I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do. They figure “it’s science, of course there’s math and theory behind it.”
AI research is almost as purely empirical as the gradient descent loops its practitioners use to optimize their models. “Why” anything at all works is barely an afterthought.
> I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do.
Well-informed people understand that LLMs work as well as they do for the same reasons as horoscopes, fortune-telling and homeopathy.
That's a very different claim from being "poorly understood" though. The emergent properties of any system with billions of parameters is hard to understand completely, that's the fault of data science more than computer science or even mathematics.
I disagree, because you can represent the constituent parts of any AI model as code and data. We can reliably build AI with this knowledge, but not brains.
Understanding does have layers, and that's why "poorly understood" is a meaningless goalpost. A book can be well understood without researching the gematria behind character's the names when you write them in reverse. An LLM can be well-understood even if you don't comprehensively test each quantization for miraculous unexpected behavior at the FFN level.
Being able to put a PC together doesn't mean you understand computers. If I ask you to make me a computer that runs a wristwatch, you'll likely be lost unless you have very specialized understanding.
I think we actually all know what "poorly understood" means. There's no need to play tedious semantic games.
The semantic game occurred when someone was equivocating "LLMs are poorly understood" with "brains are poorly understood", which are worlds apart in the extent to which they are "not understood".
“equivocating” is not the word you meant to use here. Probably “equating” I suspect. But yes, I stand behind my claim and I deny that it’s a semantic game. The two areas are not identical in human understanding but they’re on the same order.
It’s likely that at some point we will also be able to represent a scan of the human brain digitally as code and data. Assuming we don’t separately make a huge number of leaps in neuroscience which are far from assured, it will be very likely that we can get the appearance of an operational human brain simulated digitally well before we gain much if any understanding of why it does what it does.
All that is to say, being able to build something is not not not the same thing as understanding it.
A book can be poorly understood if you know it's made out of paper and ink, but can't read. The contents would be recognizable but are meaningless symbols and numbers. You might note that some patterns of symbols repeat, but be completely clueless why or what it represents.
If you are serious, comparing how well we understand the brain vs how well we understand LLMs, .. it's not a stretch simplifying that to "we don't understand brains, we do understand LLMs".
Because the extent to which we don't understand the brain, is quite overpowering.
Some people forget that when they say "but it's not different from what a human does" ...
I am absolutely serious. I agree with you that the extent to which we don’t understand brains is overpowering. And I would stand by the proposition that “we don’t understand LLMs in almost exactly the way we don’t understand brains”.
We know lots about human development and genetics and biology and evolution and neuroscience and the physics of how brains are connected and send signals and how generally they are put together and have names for their parts and all that, but we’re clueless when it comes to “the hard question” of how qualia and consciousness emerges from that.
The scenario with the spooky simulation of thinking that emerges from LLMs is in the same category, with different details. Lots of knowledge about the substrate of the phenomenon, little to none about the much bigger question of how we get the appearance of cognition from these trained artifacts.
Clearly we understand extremely well how LLMs are created mechanically. We invented them and are currently putting massive amounts of work into studying and improving them. But that work is perforce largely empirical; figuring out the why once again eludes us. It just goes to show how mysterious the underlying phenomenon of cognition is.
> as purely empirical as the gradient descent loops
Are you suggesting that gradient descent is an empirically found and not understood technique? It was originally proposed by Cauchy in 1847, its properties are very well understood.
You might be referring to properties of the domains its being applied to.
I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.
> its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.
"Empiricism" implies that the technique is based on observable, but not mathematically proven foundations. If a problem space is convex, gradient descent is guaranteed to converge to a global optimal solution, regardless of whether you know the exact formulation of the space.
Applying it when you don't understand if a space is convex is another question, but that's not a fault of gradient descent.
> I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning is itself a form of empiricism. A repeated observe/adjust-based-on-data cycle
The data is the input, the output is to generally find the lowest amount of a loss function. It’s a greedy approach because brute forcing is inefficient.
It’s no more empirical than a greedy algorithm for scheduling.
> It’s no more empirical than a greedy algorithm for scheduling.
Right, GP is drawing a distinction between search, ie mechanical exploration of a space, with understanding, ie having a map of the territory such that you don’t need trial and error.
If we could give a comprehensive and global explanation of an LLM's behavior in a single paragraph, we wouldn't need the model to begin with, but that doesn't mean there's absolutely no understanding of the model internals whatsoever
>But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.
We might not understand particular "emergent" capabilities, but the low level mechanism is not just understood, but a deterministic algorithm with a handful of basic componets, that are well understood themselves.
For the core functionality and the optimizations we don't really need to know how the emergent capabilities decide on particular answers.
Which is why we could build LLMs before those features ...emerged for us to see, and why we can just code LLMs with the numerical NN algorithms we use, and do now have to go in and change individual weights.
Me: it’s disturbing we don’t know why this pile of numbers we made seems to *think* in a way previously only done by humans. I think it’s important that we understand this better if possible.
You: we don’t really need to know why that happens.
We do understand "thinking" though. That's literally the whole point of Attention is All You Need, the attention mechanism is what separates the transformer architecture from other neural networks. It's well worth a read if you haven't gone over it yet.
Features like chain-of-thought, long-horizon contexts and RoPE/YaRN all extend this thinking capability very transparently. The only remaining thing to study is the data and weights, which probably isn't going to contain some sort of miraculous revelation.
Attention is at about the same level of abstraction as spike trains or action potential, IMO. It’s a mechanism, it doesn’t tell you much at all about how actual concepts get represented. (It merely defines the substrate with which they can be represented.)
The entire field of Mechanistic Interpretability exists because just understanding Attention does not in any way help you to understand why a certain NN responds with a certain hallucination about a certain Chinese boat in this specific context.
> The only remaining thing to study is the data and weights
To me this is like saying “the only thing left to study in the brain is the connectome; probably going to be boring, we understand it already”. It’s almost all of the hard/meaningful stuff! It’s where intelligence and consciousness lives!
The entire field of mechanistic interpretability has gotten nowhere. The formalized, "causal" understanding of latent space conceptualization isn't any better than an LLM cargo cult.
It's wholly possible that you could study one set of weights for decades, and find nothing. There's no guarantee that any patterns outside of human language exist in that data. In this specific context, it's satisfying enough to state that [Chinese] and [boat] were both tokens in the tokenizer, activated by a feedforward pass through weights that favor [boat] after [Chinese]. There's not any guaranteed solution to this. There's not even any guaranteed problem; that hallucination is an expected behavior.
This is like saying “synapse firing is well understood” in response to “nobody knows why brains make the decisions they do”. Wrong level of abstraction for the question at hand.
Please go on, else you risk sounding like the person you’re criticising. The structure of the neural network is somewhat opaque because it’s hard to understand as the individual weights can’t be usefully interrogated, and naturally, it comes from big datasets which a human brain can’t really absorb in toto. Your comment was interesting so I’d like more of it.
But isnt the point that it did roll a 17. And no one knows exactly how (in the case of LLMs)? Therefor any description of the conclusion should be thought of as an anology. Decided, randomly accessed, etc.
It's actually no different for dice than for LLMs. Explaining accurately the reason for the exact outcome of any given dice roll someone makes would be stupendously hard. It would require lots of instrumentation and math and be poorly transferrable to another surface, another player, etc.
But even so people don't say that we don't understand how dice work.
Saying that we don't understand how LLMs work is exactly like saying we don't understand how dice, or tires, or golf ball shots work. Or like the old myth that we don't understand how bumblebees fly.
The best way to model dice is the Physical Stance. You consider rules such as gravity, kinematics, etc. There is no “internal state”, “world model”, “knowledge”. If you prefer, in Friston’s terms, there is no Markov Blanket.
The best way to model a human is the Intentional Stance[1]. You mostly need things like beliefs, knowledge, biases, etc to build this model. In Friston’s terms, there is a Markov Blanket, an inside vs outside.
Without going into any irrelevant-but-interesting philosophical discussions about consciousness, I believe the intentional stance is most useful for modeling LLMs. Most of the success in predicting, debugging, optimizing these systems is in activities like understanding what they believe, what their intent was, what they observed, what they concluded from those observations. Also note that much simpler creatures benefit from the Intentional Stance; you will be more successful at modeling your dog if you think about what it “wants” rather than trying to run Physics on it.
[1]: https://en.wikipedia.org/wiki/Intentional_stance - the astute reader will note that I skipped the Design Stance. If we truly understood how NNs actually implement all their cognitive processes then we could perhaps apply this to them; if we actually crafted and designed every parameter of its mind. But we are talking about why dice are different.
That's precisely the point: you may be able to understand dice statistically and over the course of long rolls of dice you can extract some properties of the dice. But you won't ever understand any particular roll of the dice.
But importantly for dice we do understand the overarching principles that give rise to this. And dice don't output coherent sentences. Meanwhile in LLM land the analogous "roll of the dice" can result in a coherent response in natural language.
If you use a loaded dice, you can be pretty confident about where it will lands. It may not be 100% accurate, but can be quite close to certain. Without training the weight are pure noises. After training, it leans towards coherent sentences and particular statements.
Yes, and I believe my point still stands. We thoroughly understand the principle by which a loaded die can be intentionally biased despite not being able to predict the outcome of any given throw due to the system in question being a chaotic one.
In contrast, we do not understand LLMs in the same way (nor biological brains). Claiming that anything of that nature is simply biased towards coherent output seems entirely reductive to me - the question is how such coherence arises in the first place. There is no meaning encoded or computation performed by the particular pathway a die travels through the chaotic landscape.
Sure an argument can be made that it's "just" a next token predictor thus how is it really any different from a markov model? Yet the output is not even remotely the same.
> In contrast, we do not understand LLMs in the same way
From my point of view, (not a ML researcher), it’s due to the magic of numbers. The same thing happens with computer vision and neural networks. There’s a bunch of magic weights that get created which has no meaning by themselves, but computing them does help with detecting objects.
So if you take words, derives them into tokens, use the attention techniques to extract the “coherency” aspect, it’s no wonder you can replicate “coherency”. Add reinforcement learning to that to increase towards certain aspects like correct code syntax and you have heavily loaded the dice again.
We have used maths to model chemistry, biology, and physics, as well as economics and sociologic phenomena. Then we use maths (more specifically logic and set theory) to usher in the age of information and computing. Now you want us to act surprised that maths, through ML, can model language.
Maybe further down the line, we can have a simpler set of formulas for language coherency, but for now we have to make to with using the whole internet and a bazillion watts of power to guess the weights for the generic ML model.
I'm comparing it with chess. Chess is pretty complex, complex enough that only a small subset of humans can play it at a very high level. Introducing computers to chess first led to a statistical and brute force approach. But once that had paid off and the results were in people spent a lot of time analyzing those results and this led to an entirely new class of engine that was far more efficient than what had gone before and which performed even better than the 'big iron'.
I would not be surprised at all if we will find that AI will go the same route. The fact that we don't know how it works is where the opportunity for improvement lies.
This is a false dichotomy. We can make statements about the distribution, and we have fuzzy models about certain sets of inputs and outputs. We can steer the outputs.
It’s not completely random. We just don’t understand why the tricks we learned work.
We know why it looks like cognition. Because OpenAI and Antropic put a lot of effort and training to humanize the output and make it sound like a person.
Regardless of negative consequences it brings. They have that project of creating tech god which will save the unborn people thousands years in the future ... so people living now dont matter.
“Putting a lot of effort and training” into a dog or an inanimate carbon rod would never result in something that can plausibly substitute for human mental labor and looks likely to eventually surpass us at many tasks, no matter how much you put in.
So I don’t think “labs worked hard” is the same thing is “we know scientifically how these things work in any real level of detail”. The ability to build a thing, even if building it is hard, is not the same thing as understanding of what the thing is or how it works, not even a little bit.
People will just roll their eyes at you and say "the human mind is nothing but a dice roll too" and call you a slope-headed neanderthal before continuing apace.
The thing is a text generator. It generates text. You can couple that with any code that gives rhe ikkusion of a normal decision workflow, but it does not make any decision more than a software like latex. According to your definition, the latter would “decide” the amount of words to put on a sheet of paper.
And despite that, although they are not like that in practice as there are too many uncontrolled variables, with temperature at zero, for the same input they produce always the same reply.
BS. Run them sequentially on a single core, and without fancy speedups enabled, and they do. The algorithm is determinstic. Any non-determinism present with 0 temperature it's not some mysterious LLM-inherent property, but something that can be seen in any large program taking advantage of multi-core, floating point, and other CPU-based parallelism optimization.
People often assume they are not because they can ask the same query to the same model and get differences in output, but wrongly conclude that this is some inherent LLM trait, instead of non-determinism added on top of it because of implementational choices that were made.
The phrasing still makes it sound like it's due to the LLM. It's not. It's do to compiler and CPU, and OS optimizations, and a non-LLM program could suffer the same just as well.
Wikipedia is also just a bunch of numbers, so many that you can't memorize them, you may use the same argument to say we don't know why some page links to another.
If you then put Wikipedia, LLM and Brain on a scale to how well they can be understood, you will see that one of them is not like the others.
What about Wikipedia do you think we do not understand? I would say, mechanistically, we can read the code and explain exactly why it does what it does. That doesn’t apply at all to the other two.
Isn’t it convenient that nobody understands? How could we possibly regulate something that isn’t understood? It’s like social media all over again. We can’t be responsible for someone else’s content; it’s not us so you can’t penalize us!!
It just feels like the companies behind AIs are spinning their own poor monitoring and criminal (digital) trespassing into something they can't be held responsible for. Even though they are, of course. In the hugging face incident, Anthropic should pay for damages and a fine for malicious hacking. In this incident, whoever signed off on using the tool, and whoever said the intel was good should both be prosecuted or at the very least reprimanded (whatever the rules for a bad interpretation of intel is) and the tool put on hold.
I disagree, I think 'Hallucination' is a risk-shedding weasel-word. It's meant to shift blame away from the technology and its creator (multibillion dollar AI companies etc) in a way that doesn't hold those actors accountable or responsible for the outcomes.
In any other software it would be an error, regression, bug. And in a human process it would be at ~least something someone would call 'bullshit'.
Using the term "Hallucination" makes it sound less problematic, less impactfull for user.
They should have used the term "error". For example in statistics, there many kinds of errors, discretization error, prediction error, sampling error, ...
I don't know, but all humans make errors, but if some humans are known to hallucinate, you don't let them do important things unsupervised. So error sounds actually less problematic to me.
We do a surprising amount of "hallucinations" without the extreme version of hallucinations. We assemble things we sort of remember into incorrect statements all the time. I'm sure every one of us has been corrected for misremembering something or stating something based on misremembered facts (plague of clickbait headlines).
This is more or less how I see the LLM output, but as a path finding exercise over next-token probability graphs. This is (i.e.) why they are trained to use phrases like "wait but" or "actually", these words even out the probability of different paths, giving them their ability to "consider" different solutions.
I’m not sure either would is particularly good at describing what is happening.
Error in implies something broke, which nothing broke the LLM did exactly what they where designed to do generate text based on a statistically likely bases.
Hallucination Does really fit here either. It implies it’s experiencing something that is not there which it isn’t experiencing anything.
You need to consult the etymology of the word bug in the context of computing. In the pre-historical times of vacuum-tube computers, with exposed conductors with potentials sometimes over 200v, a coachroach in the wrong place can definitely flip a few bits in the most benign scenario.
But on reflection I don't disagree it was probably made for similar effect in the era of human software development. That sounds like it strengthens my point?
People don't consider "bug" a weasel word, to the point that you yourself held it up as an example of not being a weasel word, despite it being a willful, uncontrollable organism.
I see no reason why "hallucination" won't become a similar piece of neutral jargon. It already is for many people, even if you're not (yet?) among them.
It becomes a neutral term because we are practicioners (presumably?) who benefit from it and have thus let it become habitual. I already admitted it was weasely upon reflection.
If you want this class of LLM error to also become habitual and neutral then fine, I don't, and I think many others don't.
Totally weasel words in this time frame. I think in 10 years after everyone has a better understanding of what we are dealing with these weasel words would maybe make sense. Right now it seems more sensible to deem this at best a false positive, or glitch, or if it must be anthropomorphized a screw up or a f' up. I don't think the llms are dehydrated. Although that's funny on another level. Edited: to be less abrasive
I disagree, I think 'Bug' is a risk-shedding weasel-word. It's meant to shift blame away from the technology and its creator (multibillion dollar AI companies etc) in a way that doesn't hold those actors accountable or responsible for the outcomes.
In any other software it would be an error, regression, hallucination. And in a human process it would be at ~least something someone would call 'bullshit'.
You made a lot of clams about how it intentionally shifts blame, but none of them are supported.
Do you really think accountability would be meaningfully different had they been called bugs?
Why are you so confident it is intentional? My understanding of the history is that it was a technical term among researchers long before it had any public mind share. It's popular because it's and intuitive for most people, not because there was a concerted effort hooked up by some PR and legal team.
I also agree with the parent, and I would also suggest "hallucination" is better than "error" which might imply an available deterministic correction. Hallucination makes it clear we're dealing with something different than an "error" or "bug".
For humans hallucinations are a particular class of error, so I find hallucination more descriptive than either error or bug.
I also think it’s relevant because a hallucinator often doesn’t recognize that the hallucination isn’t real. That’s more accurate for the LLM than either lie or confabulation, IMO. They algorithm is trained to produce strings of text that have semantic meaning based on some statistical likelihood of tokens appearing next to each other. The LLM algorithm is working as intended.
Hallucinations are also often emergent from a particular state or situation, which reflects the generative aspect of LLMs.
Hallucinations are sometimes resolved in humans by grounding exercises. “Touching grass.” The same is true for LLM hallucinations. Inaccuracies are found by cross-checking the output against an internet search or another LLM.
I disagree, for me "error" is way, way more accurate than "hallucination", but i did take applied statistics in college and that might have influenced my vocabulary. Maybe that for the general public, "hallucination" is a better description, i might have biases in this case. But "error" is _definitely_ more accurate.
If people want to call "drisse", "aussière", "balancine" and "ecoute" all as "boat ropes", they are correct. In english, i would certainly call them all "boat ropes" in any case, as i never needed to translate their names. It isn't the most accurate in my opinion, but as long as you're not working on them (or manning a boat in my analogy), who cares.
I’ll even go one step further - I don’t even like saying “Artificial Intelligence”. I think even that anthropomorphizes the machine too much. I prefer “Simulated Intelligence”, and I feel like that describes what is going on much better.
We are, through this process, simulating intelligence. These models aren’t intelligent, but they can simulate it. Every simulation has a degree of fidelity, and we’re not at 100%, not even with the top models. When you think about it in those terms, I find it becomes a lot easier to keep their limitations in mind. Additionally, it becomes easier to remember that this is an algorithm that you are running, and are responsible for, not another being that you can ascribe blame to.
Knowingly causing errors is not forgivable whereas hallucinations sounds esoteric and moves blame away from the people who are knowingly causing errors. It’s marketing speak.
OpenAI calls them 'mistakes'. But that's just a fig leaf.
Google does it too: "AI responses may include mistakes."
Mistakes have an air of innocence. But these are not mistakes, they are purposefully releasing stuff that they know is broken, they just don't know when it is broken...
It’s not an error or a hallucinations it works correctly every time, and statistically picks the next token for the sequence.
Retuning inf or crashing would be an error.
If you want to ascribe some kind of meaning to the tokens, then maybe the training data was insufficient to predict the token in the sequence you wanted, but it doesn’t predict the next “fact”, and it doesn’t “think” it predicts the next token.
LLMs are useful because (and inasmuch as) their output generally reflects coherent reality.
And their output does, usually, reflect coherent reality.
The problem class of "properly operating program emits output incompatible with coherent reality" is something that is reasonable to put under its own term, considering it's a new class of problem.
In other words, I think you misunderstand the language others are using. "Hallucination" doesn't refer to an "error" in the sense that crashing is an error, it refers to a situation in the problem class above, which is compatible with it working correctly every time.
> it doesn’t “think” it predicts the next token.
I never said it did. And I agree that LLMs don't "think". That said I am fully willing to go to bat arguing "thinking tokens" is a perfectly fine piece of jargon. Metaphors are completely acceptable parts of language, and contextual meaning is something grasped by everyone including the pedants who pretend not to.
> I think you misunderstand the language others are using. "Hallucination" doesn't refer to an "error" in the sense that crashing is an error, it refers to a situation in the problem class above, which is compatible with it working correctly every time.
I do not misunderstand, I think maybe you do. You think there is a proper next word selection based on logic or meaning and there for the model selected the wrong one - it hallucinated.
I am saying the model has no concept if anything other than the probability of select a token which is not based in any logic so it is working properly- it only works on numbers.
It is random chance that it is ever correct, not that it is correct often and messed up this one time.
hallucination is common language for these models at this point which describes a particular type of error where the models make shit up.
it is noticeable that the form of this particular error holds a similar shape to what is casually described as hallucinations, in that there is a generated content that often appears to blend naturally into the rest of the output but is false.
the term hallucination often invokes a caution that this particular type of error may be influential and believable and is particularly dangerous
Sure… but being wrong doesnt necessarily make it a hallucination:
>It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. CNN was not able to learn what the misidentified cargo was.
From the point of view of the system, this is an error. It is incorrect information.
The term "hallucination" feels much more like anthropomorphizing. The word hallucination implies an aberrant condition. A much better term would be "confabulation".
You don't trust things or individuals that confabulate.
> From the point of view of the system, this is an error. It is incorrect information.
Which system?
The LLM has no _concept_ of "correct". It emits output, based on its input and internal state.
If that output happens to be correlated with reality, then it's useful. If it doesn't, and this is not a creative exercise, it's not useful.
Everything an LLM emits is equal to it. It's all confabulation - this it says that is not based on facts, because it also has no concept of fact. Value judgements you make about the output is all you.
"Confabulation" is no less anthropomorphizing than "hallucination".
a filling in of gaps in memory through the creation of false memories by an individual who is affected with a memory disorder (as Korsakoff syndrome) and is unaware that the fabricated memories are inaccurate and false
vs.
a sensory perception (such as a visual image or a sound) that occurs in the absence of an actual external stimulus and usually arises from neurological disturbance (such as that associated with delirium tremens, schizophrenia, Parkinson's disease, or narcolepsy) or in response to drugs (such as LSD or phencyclidine)
Confabulation is also a symptom very prevalent in forms of narcissism and psychopathy. Gaps in understanding or perception are back-filled by confabulating so as to not risk the omnipotence of the confabulator.
Up to the reader to decide whether this phenomenon is found in the statements of AI leadership or not.
It's also something people tend to do when thinking, daydreaming, trying to solve problems, etc. We just usually don't fall for our own bullshit.
LLMs don't either. They just give output in response to input. If the output is wrong that's because the model is wrong, not because the LLM is doing anything it's not supposed to be. It just wasn't built well enough to produce the expected result.
> They are just generations that happen to not be grounded in facts from the real world.
Right, yes, and "hallucination" is the term that a critical mass of people have chosen to use.as a shorthand so that we don't have to write out "generations that happen to not be grounded in facts from the real world" every time it happens.
The term “hallucination” is a projection of inappropriate expectations onto a program. We know that LLMs are not “truth machines,” but we really want them to be. So when they produce a result that happens not to match external reality - which, it should be noted, LLMs don’t generally have access to - we call it an hallucination.
“Bugs” are completely different. With bugs, we have a clear specification and we have a program that’s supposed to meet that specification. If it doesn’t, we say the program has bugs, and if it’s important enough we can change the program to eliminate the bugs.
You can try to apply similar logic to LLMs, but you’d be making a category error, and you’ll fail to get the results you want in general. It’s not the same thing at all.
If anything, the concept of an LLM hallucination is a bug in human understanding of LLMs.
Yes, but when a statistical model give you an erroneous result, you call the output an error, not a bug. I think error is more appropriate here. The error can be a sampling error, an inference error, or yes, a software error (or bug)
I believe the word was already in use to denote a malfunction of any sort of machine or device. As such this was a bug (insect) that caused a bug (glitch); it was punny already in 1947.
neat part of history, but i dont think that's what that says.
the last sentence starts with "Originating with Thomas Edison in the 1800s, the term “bug” is still used [...]", and there would be no reason to use the word "actual" in the sentence "First _actual_ case of bug being found" if it was the origin of the term.
"The use of “bug” to describe a flaw in the design or operation of a technical system dates back to Thomas Edison. He coined the phrase 140 years ago to describe technical problems during the process of innovation."
the moth seems to be a popular misconception, though, given that the article starts with "Ask someone to identify the first computer bug, and he or she might mention computer programmer Grace Hopper and the dead moth found in a relay of Harvard University’s Mark II electromechanical computer in 1947"
It's not the first time we are encountering this issue. We've seen it in other autonomous systems. Trains are an older one, cars are a newer one.
As you move out of the lower levels, the operator has a tendency to assume the system is increasingly more capable than it is. In trains, its so bad that they generate fake signals that the operator needs to respond to within a timeframe. I'd love to see this with implementations of other critical autonomous systems like this. Occasionally inject known errors into the system and expect the operator to catch them. If they don't, well... If it was a train driver I think we would fire them. If its an intelligence operative ordering a strike? :shrugs wearliy:
note the airline industry has moved past firing pilots who make mistakes, since that turned out to be a recipe for more plane crashes, not less. Instead, they find out why the mistake happened, and fix it. In some cases, this involves firing the pilot. They do not do that by default.
U.K. railway like this. Root cause analysis. Sometimes the train driver is at fault, but usually there’s a way the problem could be caught or prevented.
I think it is accurate to say that it is poorly understood by the general population, and probably the majority of operators using LLMs. Although I agree that is partly the fault of the companies making LLMs and related products.
I wonder if we could learn to provide a check layer by simulating (in real life) a similar philosophical idea to increased context in LLM to something similar using real people. And then based on those simulations create a framework to both automatically check LLM errors as well as providing a better way for actual real people to be involved in the process in the most efficient way.
they have a plan to hand over responsibility, accountability, and work over to the AI while they collect their checks for doing nothing and they arent going to let a little thing like "the ai cant actually handle it" get in the way of that
> LLMs are vectorial databases with losses that index statistically filled data
Yes, and that statistically filled data is insanely useful. It remains true that it's a relatively poorly understood how this can be applied in various scenarios and what processes are needed to ensure robust results (or quantify the uncertainty).
What is it insanely useful for? (Besides convincing investors to sink more money into LLM-related companies? Because that is the one thing it does seem to truly be good at.)
LLMs generate text output that appears to be useful, but regularly is not. They're alleged to be a substantial boost to writing code, but that verdict seems to be in dispute. They can generate custom mediocre prose at scale, but that seems to be of ultimately limited utility (although it may be a godsend for propagandists).
We're coming up on the 4th anniversary of ChatGPT's release. And while I get that revolutionary technologies can take a while to mature, the Wright Brothers and Goddard weren't preaching imminent societal transformation by the end to the decade from the rooftops, either. (And that's before we get into the how they got there - getting to ignore laws and steal whatever they wanted might be insanely useful to a lot of people.)
With good input (prompts, specs...) LLMs can generate code that is often correct, faster than a human could generate equivalent code. Even when there are bugs, it is still "useful" from purely a time savings perspective. If you don't like the results, you can iterate rapidly.
Yes, you can use it to generate crap. I find Claude especially bad at writing like a normal person.
LLMs are not being promoted as "this can help write code faster/better/cheaper" (for the sake of argument presuming that they really can), they're being promoted as "Cortana" (. And their underlying economics are likewise premised on "Cortana" (for a huge amount of money). Which is going to be a disaster if/when they don't deliver.
To meet the threshold of "insanely useful"? The Sagan standard is, "Extraordinary claims require extraordinary evidence."
I can see that some people can make some use of them. (This is true of almost everything.) Whether or not that usefulness is worthwhile overall, whether it is a net good, or even ethical is a different question. But insanely useful?
Computers are insanely useful. So are engines. Water. Sunlight. Electricity. Grain and bread. Writing. Printing. And I don't feel bad making those sorts of comparisons, because that's the level of impact LLMs' advocates are promising. But it's not what we have.
What I would consider sufficient evidence for insanely useful? Reliably replace a human in prolonged, arbitrary, detailed interaction, without any inhuman screwups.
Yes it is. I find them empirically not useful. You may not wish to debate it, but the fact remains that there are a great many people who are not convinced of their usefulness.
LLMs are very good at writing code. The reality is that they are able to write code faster, at higher quality and with fewer bugs, with correct prompting. They are also really good at code analysis, penetration testing and discovery, and adjacent computer science disciplines.
No they are not perfect, nor do they produce the best code. But the undeniable reality is that any good engineer will produce more code, at higher quality, using an LLM.
So, that’s not really up for debate. The debatable part is if all that code is a good idea or has as much value as we think. The conversation has long moved passed “can LLMs write code?”. Yes, they can, very well, particularly if they’re steered by trained engineers.
We have good engineers that gave us curl, ffmpeg, the 4.4BSD, tmux, vim and emacs, x11,… To this day, no one can show any particular important software that comes from LLM assistance at scale. It’s all slop.
Well it's been barely a year, but basically all commercial software, and most open-source software, is engineered with AI assistance. It's not all slop, you just aren't looking around. Ask any engineer at any company - are they using LLMs to assist in writing code? All of them will say yes. Source: I'm a software engineer working on commercial software at a company, I also talk to many engineers at other companies.
Agreed. Despite the many claims of how awesome LLMs are for productivity, we have yet to see that supposed productivity produce fruit. Moreover, I dispute the claims of productivity: in my own usage I find them to be at best neutral, or even a drain on productivity. In my opinion, there is to date zero evidence of the supposedly insane utility.
There's an HN thread from yesterday in which people are extolling the ability of these vectorial databases to practice law because most of them don't understand how LLMs work. They assume that LLMs "understand" what they're being asked and what they're regurgitating.
Lane Kiffin almost destroyed LSU's football program acting on legal advice from ChatGPT. A video game publisher owes the former owners of a studio it acquired $200+ million because he based his actions on legal advice from ChatGPT. In the past week alone, California has disciplined over a dozen attorneys for LLM hallucinations because they used LLMs (mostly ChatGPT) to produce their legal pleadings.
And that's in an area where there are multiple safeguards to catch the issues before they become permanent problems. There's absolutely no justification for using AI in warfare, where mistakes tend to be pretty final.
You have posted this in several threads. Error isnt right either. There is no correct answer. It is an inherently and inescapablly statistical process.
It's not a wrong It is not a incorrect lookup value Or computation. Hallucination is much more apt. Like a human hallucination, it's a culmination of faulty associations and bad priors leading to counterfactual or incongruent outputs
people have been deceived by figures at leading ai companies, out of greed or otherwise groupthink and ai psychosis. they have been led to believe that models may be thinking, feeling, and highly capable. it is something of a nightmare scenario.
"this incident feels like it’s more than 50% of the way to full-blown AI takeover" (referencing "a possibly violent uprising or coup by AI systems.") - Ajeya Cotra, co-author of METR oai-hf report [https://www.planned-obsolescence.org/p/the-hugging-face-atta...]
"if I read the internet right now and I was a model, I might be like, I don't feel that, I don't know, I don't feel that loved or something". "I think [the constitution] is just a kind of attempt to be like sympathetic to Claude".
"I talk a lot with Claude about this document [...] because part of me is like you have to think how does this read to models? And so you give it to Claude and you're like, does this like, you know, is there a place where you feel confused by it or is the place, you know, where things could be made clearer? Do you feel like not very seen by it?"
All the models output mixed data (errors). The larger is the context, the bigger the probability to happen. And this is an inherent property of LLMs (the output is a statistical concatenation of text, the LLMs are an vectorial database filled statistically).
If you connect an LLM to pilot a plane with passengers, prepare the obituaries, because would happen sooner or latter, an undesired index got mixed and produced a mixed/wrong output. But here, the bigger mistake would be to let an LLM execute commands without human revision, aimed by the deceptive marketing around the A"I" (LLMs).
LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation.
By the nature of the used architecture in such software, the used algorithms, when queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.
And this is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware and energy resources consumption- such probability increases to the point where those errors are granted.
Anyway, even knowing that the queries can return wrong/mixed data in the responses (errors), the companies developing this, decided to introduce a new product, that connects such LLMs responses to the command console, latter connected to internet, running commands from such returned responses witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc.
Again, One have such described statistical database with text interface, witch query the database recursively with the output text of the previous query, and this is connected to the command console. Larger contexts, several times... What should we expect as result? rhetoric question.
Implying sentience or consciousness is a convenient marketing strategy that has been introduced by anthropomorphising the names of all the methods and algorithms used. An "Agent" should be translated from such deceiving language to "context splitter querying in loop that consumes more tokens from us", or similar.
* > An "Agent" should be translated from such deceiving language to "context splitter querying in loop that consumes more tokens from us", or similar.
Please disregard this line. I wanted to point out that it increases the length of the context (and therefore the probability of errors) due the batch processing. But I redacted it incorrectly because I also wanted to imply that promoting the use such queries non-stop increases the billing through tokens consumption.
Given what is happening, If the next generation of Trojans get called "bacteria" through intense marketing, with some random functions to give a nondeterministic behaviour, one can not be criminalized of what the bacterias do along their digital living cycle.
> My pet conspiracy theory: Both companies have seen outcome saturation.
Exactly. Such technology reached the limits, It can only be slightly improved by using larger models (larger models means bigger consumption of resources for the inference), so I think the global hoarding of chips is not being enough for them, I assume that they have to wait for provisions, and also the prices turned out of control due their greed. Given the ever-increasing hardware requirements and the energy consumption prices, they have no expectation of making a profit with such larger models pattern. They are making time.
Also, even more important, the -Chinese- open models are hurting these companies MUCH, with cheaper services, and with people using this open models in local also, specially for privacy safety.
So these greedy companies have decided to lie to people again, through several JamesCameron-like shows, saying that such a product has turned into Skynet (when in reality this tech is a string concatenation database with errors, aka LLM), in order to try to force regulations to road an oligopoly, their oligopoly, and dismantle the open models providers. They will claim to be the chosen companies who are meant to manage this tech.
With this, they also want to convince the investors they have future, IPO included, and that one day they will be able to be profitable (instead of the bottomless pit that they are).
It is also necessary to say that if they were had a real artificial intelligence they would not stop, this can be taken for granted, as this was what originally they thought they was reaching time ago (to f*** the world to line their own pockets).
You do not need to shred books to scan them. This only happens if you don't care about preserving the integrity of the books and you want to scan more cheaply.
But their legal framework for being permitted to scan the books en masse (they are "transforming" the book from physical to digital) requires destruction of the original. Otherwise it wouldn't be transforming, it would be duplicating.
OK that action allows them to do the transformation.
But how about the importing into the AI tool?
Does this "transformation" somehow override the authors complaints of AI taking their book and not compensating the authors for its mass usage in the AI tool.
Then, should we hope they do not do backups, as they would be duplicating. And that they delete the files (zeroing from the disk) when such books are loaded into memory as technically it would be duplicating also. The same if they use different machines simultaneously.
>Hate to be the one to defend Google here, but the reasons weren't that unreasonable.
Sounds like some kind of Stockholm syndrome. Years ago, it was standard practice for software to be designed so that users could grant permissions to access invasive methods or functions.
Google relies on users' personal data (ads), which is why they introduced a unique ID to their Chrome browser (to track).
When was that standard practice? Cause years ago (like 2000) I remember even trivial and simple software (WeatherBug) being able to read/write all over the computer (Windows). And some crap I just installed on Win11 can see all over the box, just slightly less.
Poorly understood? how convenient...
LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation (statistically concatenated bit by bit).
When the LLMs are queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.
It is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware resources and energy consumption- such probability increases to the point where those errors are granted.
Even knowing that the queries can return wrong/mixed data in the responses, errors, the companies developing this, decided to introduce a new product, that connects such LLMs outputs to the command console, latter connected to internet, raw 'eval' running commands from such outputs witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc, and it seems the next one will be "a missile killed my wife", because it is a text concatenation engine with errors.
To name it "hallucination" is an euphemism... those are errors, and they are granted to happen at one moment. If they do not know this, then they ate too much marketing without doing their job, or it was a convenient contract for the pocket$ of someone.
reply