A JIT compiler generates code depending on the hot path driven by user actions - which are non-deterministic. The same happens when a compiler is deterministic but in a different way than developers think. Developers think they know what code is generated, but they don’t. A compiler translates a double loop or a stream into just one SIMD instruction - they think they know what code the compiler generates, but they don’t - so their trust in the compiler is for the wrong reasons. And even if compilers would be fully deterministic, and people would know the code they generate, compilers are so complex that determinism is not useful for verification - just because ‘someone’ looks at compiler output does not mean it’s correct.
That’s not what this argument is about. The JIT is non-deterministic based on the hot path yes. But the way that code gets turned into code IS.
The compilation of instructions is deterministic. The control flow and WHEN it gets compiled may not be, and it may smartly choose not to compile certain things based on that hot path but that’s not the same.
A compiler and interpreter is a SYSTEM. a LLM is not.
Ok, how the fuck is that relevant to the argument?? LLMs write code, then deterministically it gets compiled, type checked, linted, etc. At that point you absolutely have deterministic code that is as valid as can be verified. EXACTLY like if a human wrote it. Unless you’re suggesting that programmers never made mistakes that don’t show up in tests…
The “problem” is that any standard compliant code uniquely defines the output for any given input. A bug-free compiler translates that execution path into the exact same behaviour, just in a different formulation.
An LLM translates a more or less vague natural language description of an execution path into a precise one formulated in code.
The difference is pretty fundamental: In one case you give an exact description of the inputs and outputs, and let a compiler translate that to a different but logically equivalent description. In the other you give a non-exact description and let a stochastic machine translate that into one of many possible exact descriptions.
That doesn’t mean it’s impossible to produce good code with an LLM. It means you need to inspect the output from the LLM with far more scrutiny than you would ever inspect the assembly output from a compiler.
https://www.tabulamag.com/p/non-determinism-is-not-the-problem
Oh my god. No.
That’s not what this argument is about. The JIT is non-deterministic based on the hot path yes. But the way that code gets turned into code IS.
The compilation of instructions is deterministic. The control flow and WHEN it gets compiled may not be, and it may smartly choose not to compile certain things based on that hot path but that’s not the same.
A compiler and interpreter is a SYSTEM. a LLM is not.
Ok, how the fuck is that relevant to the argument?? LLMs write code, then deterministically it gets compiled, type checked, linted, etc. At that point you absolutely have deterministic code that is as valid as can be verified. EXACTLY like if a human wrote it. Unless you’re suggesting that programmers never made mistakes that don’t show up in tests…
The “problem” is that any standard compliant code uniquely defines the output for any given input. A bug-free compiler translates that execution path into the exact same behaviour, just in a different formulation.
An LLM translates a more or less vague natural language description of an execution path into a precise one formulated in code.
The difference is pretty fundamental: In one case you give an exact description of the inputs and outputs, and let a compiler translate that to a different but logically equivalent description. In the other you give a non-exact description and let a stochastic machine translate that into one of many possible exact descriptions.
That doesn’t mean it’s impossible to produce good code with an LLM. It means you need to inspect the output from the LLM with far more scrutiny than you would ever inspect the assembly output from a compiler.