He is saying that you have to understand your contribution, so whatever the LLM spits out, you need to thoroughly review and double-check that it makes sense. So, in particular, he’s not trying to rate these tools, he’s merely saying make sure the contribution is objectively good, then he doesn’t care how you got there.
My issue is him saying don’t use a compiler if you don’t really understand the machine language it creates. This conflates LLM with a compiler as a tool when they are inherently different. This is disingenuous because it is possible to understand and audit what a compiler creates. You cannot audit what an LLM creates in even close to the same way.
But why can you not audit what the LLM creates? You can’t do it for the general case, i.e. prove that for a given prompt it always spits out a valid solution, but you can look at what it actually spit out for your concrete prompt and then you can freely decide, if those code changes are good.
I am of the opinion that a thorough review often takes similarly long to actually writing it yourself, and there’s a high chance that you miss architectural aspects, if you don’t investigate yourself.
But yeah, again, I think you’re taking the comparison too literal.
He is just saying that LLMs are a tool. You need to know how to use your tool and produce objectively good results. Then he does not care what tool you use to get there.
You don’t have to do everything with the LLM. Tracing the logical path very much still happens by looking at the code in your editor.
This is also why I’m saying it isn’t much faster. Understanding the existing code and what changes are necessary is the bulk of the work when handwriting code as well.
You *can* tell the LLM to go ham and implement the change on its own and then only look at the diff afterwards, but anyone who thinks they understand the code changes then, is lying to themselves IMHO.
Can you elaborate on the last sentence a bit from a technical perspective? I’m not using code myself to the extent that I could really judge between LLM-generated code being good or just-works. I can just imagine compiler output being strictly deterministic, while LLM-code, due to the way a LLM works in principle, should be more …unstructured?
I honestly didn’t think you could even audit an LLM. It is possible, but not in the same way you would audit code. It is my understanding, because of the probabilistic nature of how the LLM operates, you can get different results even within the same model. This makes tracing something like logic next to impossible. Why the LLM gave the result is somewhat of a mystery. It is in effect a black box. This may change in the future, but for now LLM suffers from this issue.
You could probably trace how it comes up with the probabilities of the next token. Though the meaning of all the weights are probably going to be lost on you. Once you’ve got your probabilities and you’ve limited to some top number, it’s just a dice throw at that point.
He is saying that you have to understand your contribution, so whatever the LLM spits out, you need to thoroughly review and double-check that it makes sense. So, in particular, he’s not trying to rate these tools, he’s merely saying make sure the contribution is objectively good, then he doesn’t care how you got there.
I got that part and I agree with it.
My issue is him saying don’t use a compiler if you don’t really understand the machine language it creates. This conflates LLM with a compiler as a tool when they are inherently different. This is disingenuous because it is possible to understand and audit what a compiler creates. You cannot audit what an LLM creates in even close to the same way.
But why can you not audit what the LLM creates? You can’t do it for the general case, i.e. prove that for a given prompt it always spits out a valid solution, but you can look at what it actually spit out for your concrete prompt and then you can freely decide, if those code changes are good.
I am of the opinion that a thorough review often takes similarly long to actually writing it yourself, and there’s a high chance that you miss architectural aspects, if you don’t investigate yourself.
But yeah, again, I think you’re taking the comparison too literal.
He is just saying that LLMs are a tool. You need to know how to use your tool and produce objectively good results. Then he does not care what tool you use to get there.
Auditing typically means tracing the logical path. How do you suggest to do this with an LLM?
I am definitely taking his words literally which is why I disagree with just this one point.
You don’t have to do everything with the LLM. Tracing the logical path very much still happens by looking at the code in your editor.
This is also why I’m saying it isn’t much faster. Understanding the existing code and what changes are necessary is the bulk of the work when handwriting code as well.
You *can* tell the LLM to go ham and implement the change on its own and then only look at the diff afterwards, but anyone who thinks they understand the code changes then, is lying to themselves IMHO.
Can you elaborate on the last sentence a bit from a technical perspective? I’m not using code myself to the extent that I could really judge between LLM-generated code being good or just-works. I can just imagine compiler output being strictly deterministic, while LLM-code, due to the way a LLM works in principle, should be more …unstructured?
I honestly didn’t think you could even audit an LLM. It is possible, but not in the same way you would audit code. It is my understanding, because of the probabilistic nature of how the LLM operates, you can get different results even within the same model. This makes tracing something like logic next to impossible. Why the LLM gave the result is somewhat of a mystery. It is in effect a black box. This may change in the future, but for now LLM suffers from this issue.
You could probably trace how it comes up with the probabilities of the next token. Though the meaning of all the weights are probably going to be lost on you. Once you’ve got your probabilities and you’ve limited to some top number, it’s just a dice throw at that point.
Sleeping agent in LLM is a scary topic and the tech itself has mathematically no solution for this: https://arxiv.org/html/2401.05566v2
Backdoors through model poisoning is so evil, I love it.