I think the distinction between programmatic error (solvable) and operational characteristic (mitigatable) is valid in theory, but I disagree that it matters in practice.
Proactive prevention (like bounds checking) only "solves" the class of problem if you assume 100% developer compliance. History shows we don't get that. So while the root cause differs (math vs. probabilistic model), the failure mode is identical: we are deploying systems where the default state is unsafe.
In that sense, it is an apples-to-apples comparison of risk. Relying on perfect discipline to secure C memory is functionally as dangerous as relying on prompt engineering to secure an LLM.
Agree to disagree. I think that the nature of a given instance of a programmatic error as something that, once fixed, means it stays fixed is significant.
I also think that if we’re assessing the likelihood of the entire SDLC producing an error (including programmers, choice of language, tests/linters/sanitizers, discipline, deadlines, and so on) and comparing that to the behavior of a running LLM, we’re both making a category error and also zooming out too far to discover useful insights as to how to make things better.
But I think we’re both clear on those positions and it’s OK if we don’t agree. FWIW I do strongly agree that
> Relying on perfect discipline to secure C memory is functionally as dangerous as relying on prompt engineering to secure an LLM.
…just for different reasons that suggest qualitatively different solutions.
Proactive prevention (like bounds checking) only "solves" the class of problem if you assume 100% developer compliance. History shows we don't get that. So while the root cause differs (math vs. probabilistic model), the failure mode is identical: we are deploying systems where the default state is unsafe.
In that sense, it is an apples-to-apples comparison of risk. Relying on perfect discipline to secure C memory is functionally as dangerous as relying on prompt engineering to secure an LLM.