@PeterJones said in Need clarification about "built-in" language lexers:
So I asked ChatGPT why,Why would you believe that atrocity?
It was probably an off-the-cuff question, but I figured I’d take it seriously. I know that this is going off-topic, so feel free to cull this response if you like.
To tweak the old Russian maxim, it’s very much a case of “distrust until verified” (which is why I posted my question instead of just swallowing what the thing spit out.
I’m not particularly a fan of them, and I honestly believe that in time, we (as in humanity) may come to regret their invention and our likely inevitable overdependence on them.
But I’m also not an ignorant neophyte. I’m actually very well aware of the limitations and problems with LLM’s, probably more than most people, and despite that, I’ve found them to be useful in some contexts.
First, you’re not wrong to call them “random text generators,” but that really is an oversimplification. It’s not just a flat index of word frequencies. Tehy’re trained with billions (or even trillions) of parameters that encode patterns across syntax, semantics and reasoning heuristics. From a purely mathematical point of view, it’s actually pretty intersting. But saying it’s “just statistics” is a bit like saying the human brain is “just firing neurons.” Yeah, it’s technically true, but it misses the interesting part.
So yes, because they are probabilistic sequence models, they are perfectly capable of fabricating “facts” (i.e., hallucinations, especially with multi-dimensional requests or as the context window gets filled up), making overgeneralizations like missing edge cases, or have issues with compression bias, shallow chain-of-reasoning (although this one is getting a little better), ambiguity drift, context inference biases, fidelity drift when repeatedly iterating through details, context window size limitations for long conversations, etc. I have some experience dealing with each of these limitations to some extent.
So I know all that going in, and since I do, I know not to rely on them as primary sources, and also how to account for many of those problems and a number of strategies to somewhat limit and mitigate the problems (e.g., authoritative source anchoring, chunking, forcing tabular output, explicitly prompting for blanks instead of it making guesses, etc). If I’m doing anything serious, I’ll use all the tools at my disposal, but I still know that if the output isn’t testable, it’s not trustworthy and I know not to rely on it for expertise; it’s just a tool I use to speed up my info gathering. I think of it as supplementary rather than authoritative.
So it’s an occasionally useful tool that’s saved me some time by giving me a starting point to quickly gather ideas and point me to things I might not have thought of before I check with reliable sources (like you) that can actually confirm or invalidate them.
I don’t expect to change any minds about it, and in truth, I don’t really even want to, but you always take the time to thoroughly answer people’s questions, and I wanted to respect that in turn.
