The argument is not just "theft", and to the extent that "theft" is a part of it, it is not the most important part, not by large margin. The reason that "theft" is not a solid argument, is because people creating open-source often distribute it for free themselves. Piracy is not the issue.
The issue is that all attributions get stripped away by the LLM (even in cases where it is trivial to avoid). For example, as a test, I asked DeepSeek to tell me how one can do an exact substring-boolean-test in Common Lisp, and it gave me code that was, character for character, _identical_ to the code that appears in Seibel's Common Lisp Book. It clearly memorized the code. It stands to reason, that it could have also memorized where it saw it, and been trained to tell the user that the code was from that book (which is NOT in the public domain, despite being free to read on the internet)[0].
To give you an analogy, nobody has a problem with someone downloading and distributing the public domain works of Leo Tolstoy or Charles Dickens or Jane Austen or Karl Marx, etc. But to remove their names from those works, and placing your (or someone else's) name on them instead, is plainly malicious behavior. Such behavior needs to shunned.
LLMs (I believe) _can_ be trained to not plagiarize. They do not have to be original, nor to use public domain training-data, they just have to give attribution (which, as far as I can tell, requires little more than a corpus and a stack of similarity-measures).
I suspect that they are not trained this way, because fully automated plagiarism creates an ambiguous situation, where a copyleft license (like GPL or EUPL etc) can be unknowingly violated, and there is hardly any trace of the violation. Plagiarism is the product.
[0]: Such attribution can only make LLMs _more_ useful and _more_ widely and confidently integrated into workflows. The lack of such attributions, risks diluting the value of open source projects, as (1) a high-signal indicator of skill and achievement, and (2) a way to ensure the provenance and quality of the code (both very important, if you are considering to build a business on top of one or more such projects). Just look at all those repos that claim to have "made" Minecraft-like game or Unix-like OS, when all they did was have an LLM plagiarize the code for them. This is a kind of, at best, spam and, at worst, fraud.
In addition to giving credit where credit is due, attribution is important for verifying the trustworthiness of "facts". When someone tells me they asked a LLM and it says xxxxx, I want to know did it get it from a bad source, or worse, hallucinated it from thin air. And if it turns out to be good information, I might want to check out other things the guy has written.
At least with the old school web search engines, you got a hyperlink that you could use to find out more. Now the only way to learn more is to keep asking the LLM more questions. There is no way to get from the LLM to a source outside the LLM.
And I hope I am just being a Chicken Little, but if LLMs become to go-to place for searching, we will lose traditional web search engines. They won't be profitable if not enough people use them. For those who don't remember how the web was before Google's search engine, we had search engines that really didn't work, web rings, and people's list of random bookmarks. The web is much too complex and big to ever be able to go back to those methods. The web will be effectively unnavigable and dead. All we will have left will be the unattributed knowledge distilled in the LLM with no way back to the old methods. And please tell me I'm wrong.
> It clearly memorized the code
This code could appear many times in training data; but I wonder what decoding params did you actually used, because that effect appears likely with close-to-greedy way. Also, if it is a coding assistant model, chances are this skill was increased by additional sampling from that domain during training? Or maybe RL has teached model to provide verbatim answers for certain questions? So it was rewarded if that kind of answers is really helpful (I guess it was?).