You can predict with certainty what a compilers output would be. Linus is conflating the fact that most people don’t understand how a compiler converts higher level programming language into machine code with the reality that no one can actually predict and audit what an LLM puts out.
You cannot, that’s why reproducible builds are hard. Modern compiler are made with compilers (egg/chicken problem) not assembly nor machine code and you always loose some perfect prediction.
Yes LLMs are less predictible than compiler but their output is readable to human, while machine code is very hard (yet not impossible).
Is that a bad things that 99% of devs have forgot how to write code cutting holes in paper? Tools are changing and while a direct comparison between the move from paper to COBOL or assembly to C is not the same as python to LLM generated it’s hard to deny that LLM will change how human write code.
Thanks for that explanation. Are you saying that if I rebuild the exact same code on the same machine I could get different results for a given compiler? I think this would render solutions like docker useless. Could you elaborate?
Yes you can get a different result. Compilers are designed to be deterministic, but a build is more than the compiler. Usually the bytes differ but the behavior is identical.
Docker is far from being reproducible!
It works on my machine ≠ reproducible.
But reproducible will make it works on your machine.
Using a compiler does not guarantee you to get byte-perfect output each time, especially not on different machines.
Talk about misleading. The differences are timestamps and metadata, and some compilers have flags to let you zero out timestamps and force it to produce a bit-for-bit identical file every time.
That is, of course, the whole point of docker (making sure the results of compiling are identical). I totally get the different machines giving slightly different results. This is not really in the same realm of the dramatically different output a LLM will give though.
No the goal of Docker is not to make sure the results of compiling are identitcal. It is to make it work on your machine, there is some nuance to it.
This is more the goal of something like nixpkgs (NixOS).
This is not really in the same realm of the dramatically different output a LLM will give though.
Yes this is not same realm but my point was that eve if LLM would give you different results each time the result is transparent and editable. Also developers tools evolve and Linus have more technical knowledge to use this tool in a way that will gave him satisfying result than me or a junior dev with 1.5 years of experience. He was against LLM generated code a few years ago, not because he hated on AI but because this tool was giving him unsatisfiying results. If compilers were not working 50% of the time he would probably write everything in assembly. LLM models and their coding harness cannot be compared to what they were 2 years ago. And most of the devs are using them. Will that transform bad devs into good ones? Probably not. Does that mean that LLM generated or assisted code is the way to go? Not necessarly. But it means that it’s a tool that find its use in the majority of development teams and thus that they find it quite useful.
There is some issues with it and I’m not here to deny them. Whether they are social, environmental, capitalist, monopolist, learning. Like every new disruptive tool we use them in ways we won’t in the future and I think it’s similar with AI.
He is saying that you have to understand your contribution, so whatever the LLM spits out, you need to thoroughly review and double-check that it makes sense. So, in particular, he’s not trying to rate these tools, he’s merely saying make sure the contribution is objectively good, then he doesn’t care how you got there.
My issue is him saying don’t use a compiler if you don’t really understand the machine language it creates. This conflates LLM with a compiler as a tool when they are inherently different. This is disingenuous because it is possible to understand and audit what a compiler creates. You cannot audit what an LLM creates in even close to the same way.
But why can you not audit what the LLM creates? You can’t do it for the general case, i.e. prove that for a given prompt it always spits out a valid solution, but you can look at what it actually spit out for your concrete prompt and then you can freely decide, if those code changes are good.
I am of the opinion that a thorough review often takes similarly long to actually writing it yourself, and there’s a high chance that you miss architectural aspects, if you don’t investigate yourself.
But yeah, again, I think you’re taking the comparison too literal.
He is just saying that LLMs are a tool. You need to know how to use your tool and produce objectively good results. Then he does not care what tool you use to get there.
You don’t have to do everything with the LLM. Tracing the logical path very much still happens by looking at the code in your editor.
This is also why I’m saying it isn’t much faster. Understanding the existing code and what changes are necessary is the bulk of the work when handwriting code as well.
You *can* tell the LLM to go ham and implement the change on its own and then only look at the diff afterwards, but anyone who thinks they understand the code changes then, is lying to themselves IMHO.
Can you elaborate on the last sentence a bit from a technical perspective? I’m not using code myself to the extent that I could really judge between LLM-generated code being good or just-works. I can just imagine compiler output being strictly deterministic, while LLM-code, due to the way a LLM works in principle, should be more …unstructured?
I honestly didn’t think you could even audit an LLM. It is possible, but not in the same way you would audit code. It is my understanding, because of the probabilistic nature of how the LLM operates, you can get different results even within the same model. This makes tracing something like logic next to impossible. Why the LLM gave the result is somewhat of a mystery. It is in effect a black box. This may change in the future, but for now LLM suffers from this issue.
You could probably trace how it comes up with the probabilities of the next token. Though the meaning of all the weights are probably going to be lost on you. Once you’ve got your probabilities and you’ve limited to some top number, it’s just a dice throw at that point.
Compilers are incredibly complex things and the often have undefined behaviors that slip in because the compiler has to interact with the operating system at large, and that means the code actually doesn’t run the same on every platform, or even every version of the same operating system. So, while they are technically deterministic, the reality is that you get surprising and unexpected behaviors in practice.
Also worth nothing that the probability distributions for an LLM are fixed at training time, and that is why the weights are static. When doing inference, models are selecting from a top-k of probable tokens at each autoregressive loop for your typical LLM. When you adjust top-k to 1 or simply set the “temperature” to 0, it will always give you the same result from the same input.
Why?
You can predict with certainty what a compilers output would be. Linus is conflating the fact that most people don’t understand how a compiler converts higher level programming language into machine code with the reality that no one can actually predict and audit what an LLM puts out.
You cannot, that’s why reproducible builds are hard. Modern compiler are made with compilers (egg/chicken problem) not assembly nor machine code and you always loose some perfect prediction.
Yes LLMs are less predictible than compiler but their output is readable to human, while machine code is very hard (yet not impossible).
Is that a bad things that 99% of devs have forgot how to write code cutting holes in paper? Tools are changing and while a direct comparison between the move from paper to COBOL or assembly to C is not the same as python to LLM generated it’s hard to deny that LLM will change how human write code.
Thanks for that explanation. Are you saying that if I rebuild the exact same code on the same machine I could get different results for a given compiler? I think this would render solutions like docker useless. Could you elaborate?
Yes you can get a different result. Compilers are designed to be deterministic, but a build is more than the compiler. Usually the bytes differ but the behavior is identical.
Docker is far from being reproducible!
It works on my machine ≠ reproducible. But reproducible will make it works on your machine.
Using a compiler does not guarantee you to get byte-perfect output each time, especially not on different machines.
Talk about misleading. The differences are timestamps and metadata, and some compilers have flags to let you zero out timestamps and force it to produce a bit-for-bit identical file every time.
Yes, that is a great explanation. Thank you.
That is, of course, the whole point of docker (making sure the results of compiling are identical). I totally get the different machines giving slightly different results. This is not really in the same realm of the dramatically different output a LLM will give though.
No the goal of Docker is not to make sure the results of compiling are identitcal. It is to make it work on your machine, there is some nuance to it.
This is more the goal of something like nixpkgs (NixOS).
Yes this is not same realm but my point was that eve if LLM would give you different results each time the result is transparent and editable. Also developers tools evolve and Linus have more technical knowledge to use this tool in a way that will gave him satisfying result than me or a junior dev with 1.5 years of experience. He was against LLM generated code a few years ago, not because he hated on AI but because this tool was giving him unsatisfiying results. If compilers were not working 50% of the time he would probably write everything in assembly. LLM models and their coding harness cannot be compared to what they were 2 years ago. And most of the devs are using them. Will that transform bad devs into good ones? Probably not. Does that mean that LLM generated or assisted code is the way to go? Not necessarly. But it means that it’s a tool that find its use in the majority of development teams and thus that they find it quite useful.
There is some issues with it and I’m not here to deny them. Whether they are social, environmental, capitalist, monopolist, learning. Like every new disruptive tool we use them in ways we won’t in the future and I think it’s similar with AI.
He is saying that you have to understand your contribution, so whatever the LLM spits out, you need to thoroughly review and double-check that it makes sense. So, in particular, he’s not trying to rate these tools, he’s merely saying make sure the contribution is objectively good, then he doesn’t care how you got there.
I got that part and I agree with it.
My issue is him saying don’t use a compiler if you don’t really understand the machine language it creates. This conflates LLM with a compiler as a tool when they are inherently different. This is disingenuous because it is possible to understand and audit what a compiler creates. You cannot audit what an LLM creates in even close to the same way.
But why can you not audit what the LLM creates? You can’t do it for the general case, i.e. prove that for a given prompt it always spits out a valid solution, but you can look at what it actually spit out for your concrete prompt and then you can freely decide, if those code changes are good.
I am of the opinion that a thorough review often takes similarly long to actually writing it yourself, and there’s a high chance that you miss architectural aspects, if you don’t investigate yourself.
But yeah, again, I think you’re taking the comparison too literal.
He is just saying that LLMs are a tool. You need to know how to use your tool and produce objectively good results. Then he does not care what tool you use to get there.
Auditing typically means tracing the logical path. How do you suggest to do this with an LLM?
I am definitely taking his words literally which is why I disagree with just this one point.
You don’t have to do everything with the LLM. Tracing the logical path very much still happens by looking at the code in your editor.
This is also why I’m saying it isn’t much faster. Understanding the existing code and what changes are necessary is the bulk of the work when handwriting code as well.
You *can* tell the LLM to go ham and implement the change on its own and then only look at the diff afterwards, but anyone who thinks they understand the code changes then, is lying to themselves IMHO.
Can you elaborate on the last sentence a bit from a technical perspective? I’m not using code myself to the extent that I could really judge between LLM-generated code being good or just-works. I can just imagine compiler output being strictly deterministic, while LLM-code, due to the way a LLM works in principle, should be more …unstructured?
I honestly didn’t think you could even audit an LLM. It is possible, but not in the same way you would audit code. It is my understanding, because of the probabilistic nature of how the LLM operates, you can get different results even within the same model. This makes tracing something like logic next to impossible. Why the LLM gave the result is somewhat of a mystery. It is in effect a black box. This may change in the future, but for now LLM suffers from this issue.
You could probably trace how it comes up with the probabilities of the next token. Though the meaning of all the weights are probably going to be lost on you. Once you’ve got your probabilities and you’ve limited to some top number, it’s just a dice throw at that point.
Sleeping agent in LLM is a scary topic and the tech itself has mathematically no solution for this: https://arxiv.org/html/2401.05566v2
Backdoors through model poisoning is so evil, I love it.
all the undefined behaviors would like to disagree with you there champ
Can you elaborate?
Compilers are incredibly complex things and the often have undefined behaviors that slip in because the compiler has to interact with the operating system at large, and that means the code actually doesn’t run the same on every platform, or even every version of the same operating system. So, while they are technically deterministic, the reality is that you get surprising and unexpected behaviors in practice.
Also worth nothing that the probability distributions for an LLM are fixed at training time, and that is why the weights are static. When doing inference, models are selecting from a top-k of probable tokens at each autoregressive loop for your typical LLM. When you adjust top-k to 1 or simply set the “temperature” to 0, it will always give you the same result from the same input.
Thanks