My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ...

So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illicit means.

Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.

• ThePhysicist

  yesterday at 11:10 AM

  Would really like to know how their watermarking technique works, and if they can use it to store arbitrary information in the text (I assume they can if only in a limited way). I assume if the text is long enough they could add all kinds of metadata that would then be undetectable as the model that generated the text is the "cryptographic" key that encodes the data. I wonder if you can have another model run over the text and destroy the watermark. I predict an interesting cat and mouse game to develop.

  • unbelievr

    yesterday at 11:51 AM

    The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs work by estimating the most likely next token, so there's sometimes a list of possible candidates that would all work in the text (e.g. synonyms). At low "temperature", the output is a bit more deterministic and otherwise it's a weighted dice roll of which token/word to pick. An LLM can loop over existing text and figure out if the output matches something it would do, similar to checking chess moves against the best computed move for detecting cheating. But the LLM purposefully creates a pattern of alternating weighted rolls that are highly unlikely to appear in normal text, and that becomes the watermarking.

    The upside is that this has very low false positive detection rate, but the downsides are many. It only works on longer pieces of text. The system is fragile, and small edits (or rewrites by a local model) can fool the detection. Only the owner of the model is able to re-run inference at this level, so data must be sent to them for evaluation. And sometimes the token output is basically 100% deterministic because the input asks for the straight answer to a fact, or to recite a quote verbatim. That leaves no room for watermarking at all, unless the model is able to lie.

    • vintermann

      yesterday at 12:16 PM

      A way to think of it is that any time the model faces a choice, it leaks some fractional bits of information. Instead of making the choice randomly, you can put information in those bits.

      But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. Exotic prompts ("Use no more than five E's in four consecutive words anywhere in the text") seem like they will confuse the hell out of attempts to extract the bits from the output text alone.

      • Majromax

        yesterday at 1:04 PM

        > But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty.

        When you only need to encode one bit, the signal to noise ratio can be very low. If I try to write my own human words under the policy of "try a little bit to avoid the letter 'e' in every fifth word," then a sufficiently long text would still be 'watermarked' even if I only succeed in this dictum (e.g.) 10% more often than the baseline.

        • ttoinou

          yesterday at 1:14 PM

          Whereas if you fully avoid the letter e, everyone will know you are George Perec

          • r2_pilot

            yesterday at 2:54 PM

            Who is Gorg Prc?

          • cratermoon

            yesterday at 4:45 PM

            And if you lowercase everything, you're e e cummings.

            • jambalaya8

              yesterday at 5:10 PM

              most people have at least two, but generally around four to six (or more, thanks internet) interactional styles (not "selves", just things which vary depending on context, subject, and people they are interacting with). probably more. some of it is due to simple physical comfort levels (right now I am in a physical position where capitalizing is more difficult); some might be due to talking to a peer group instead of a group of kids or a priest or boss at a job, etc. unsure how that will shake out with AI but it unnerves me.

        • recursive

          yesterday at 6:03 PM

          There's more to it than encoding one bit. You also want to avoid false positives. You can encode a single bit by XORing all the bits in the UTF-8 encoding. But then you get a lot of incorrect hits. The lower your tolerance for false positives, the more it acts like you're actually requiring more bits in your payload.

      • wrsh07

        yesterday at 5:55 PM

        The scheme that Scott Aaronson describes essentially uses a specific prng, and you can then check a certain function with relatively few tokens to get a sense of whether or not a model using that scheme generated the text

        A few caveats: you need to know the key to the function (used when generating the text) and you need to know the bias it would introduce

        The point is that you do not need to know the full prefix, just a modest sample set of contiguous tokens

      • applicative

        yesterday at 1:36 PM

        In fact such ‘arbitrary’ constraints uniformly improve composition. Thus eg if I force a - largely arbitrary - technical glossary to be unrelentingly applied to a translation, every single sentence improves in quality.

        • katatue

          yesterday at 4:23 PM

          Oulipo vindicated.

    • demibabs

      yesterday at 3:33 PM

      > The system is fragile, and small edits (or rewrites by a local model) can fool the detection.

      Well no, small edits wouldn’t fool the detection as long as the seeding only uses a small run of previous tokens.

      And yeah full rewrites breaking it is by design. The watermark is just meant to tell you whether the text was generated by a watermarked model, not whether the ideas came from AI or something like that.

    • out_of_protocol

      yesterday at 8:03 PM

      Does that also mean owner must re-run very model ever released by them to detect anything?

    • bsenftner

      yesterday at 12:10 PM

      In practice, it is theater. Are they going to do this with the code output too? This is nonsense security theater for the low thinkers to have a sense that someone is in charge. When we all know nobody is in charge, anywhere.

      • pavlov

        yesterday at 12:24 PM

        Their AI model tends to write a lot of lengthy comment blocks.

        That's a fine place to put the watermark to track those users who accept the code blindly and don't delete/edit the comments.

        • kfse

          yesterday at 2:26 PM

          The model doesn't particularly "know" which tokens within its output are "comments", it's not so easy

          • alasdair_

            yesterday at 6:05 PM

            It knows when several tokens are about equal vs. times where one token is vastly preferred. In the latter case, that’s usually code or math or something similar and so it won’t alter those tokens.

      • alasdair_

        yesterday at 6:03 PM

        They say in the paper that they will not do this for code but it may happen for comments.

      • applicative

        yesterday at 1:24 PM

        EU regulations are going to force it, friend. The tsunami is coming and cannot be stopped and Anthropic has jack to do with it.

        • HarHarVeryFunny

          yesterday at 2:41 PM

          I wonder if that is entirely true. It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it ("Habsburg AI").

          In the case of Anthropic it would also be entirely unsurprising if they've been lobbying the government to force everyone to do something in their (Anthropic's) own best interest.

          • kergonath

            yesterday at 3:15 PM

            > It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it

            It’s also in their interest to demonstrate that they can be trusted and to show that they at least pay lip service to limit the obvious downsides of the tools they are selling. The use cases they sell to mainstream audiences are not affected by detection tools. The point of having a LLM do the work for you is that the work is done, and reliably. It does not matter if it is done by a LLM, and most of the time it is obvious anyway.

            • 2_foos_in_a_bar

              yesterday at 6:13 PM

              The EU mandate for AI watermarking is likely the first step in the direction of prohibiting AI for specific use cases. The pretext for outlawing (or at the very least controlling) the use of AI when the time comes will be something along the lines of data integrity or just general compliance legalese.

              So, the use cases that are being sold to mainstream audiences actually will be affected by detection tools, especially if the output is intended to be monetized in some way. In the near future the EU will likely come down with heavy intervention to prevent AI from impacting employment rates across Europe. The number of legitimate use cases for costly frontier models drops significantly once eliminating professional jobs is off the table. This is all conjecture at this point though.

              • kergonath

                yesterday at 7:14 PM

                I’m sorry, this is going to be a bit long but you made good points.

                > The EU mandate for AI watermarking is likely the first step in the direction of prohibiting AI for specific use cases.

                I am not sure how practical that would be. The cat’s already out of the bag and they won’t prevent companies in the whole world from releasing open weight models. Playing catch up by distilling flagship models is also relatively cheap; we’d see smaller companies setting up shop in friendly regimes. And I don’t see any appetite to go full child porn and criminalise the possession of a LLM. So we’d end up with a similar situation as with illegal downloads, i.e., everyone will do it and nobody will care.

                > The pretext for outlawing (or at the very least controlling) the use of AI when the time comes will be something along the lines of data integrity or just general compliance legalese.

                They could forbid using LLM for hacking, but hacking is already illegal. They could make it a factor when determining punishment, but I don’t think that would work terribly well. Most of the dangerous stuff we can do with LLMs is already illegal, or should become so. Things like propaganda, identity theft, harassment, scams. We need enforcement with teeth on these, not pointless feel-good legislation. Again, there are parallels with cryptocurrencies and torrenting software. These things have illegal uses, but it’s also really difficult to make them illegal, at least in semi-functioning democracies.

                > So, the use cases that are being sold to mainstream audiences actually will be affected by detection tools, especially if the output is intended to be monetized in some way.

                I don’t know that mainstream audiences are really against LLMs. They are mostly against AI in a nebulous sense, but even non-technical people use ChatGPT or equivalent. I think that the critical mass is already there and the tools are convenient enough that they couldn’t outlaw them without a massive uproar.

                Detection tools don’t seem all that relevant to mainstream audiences’ use of LLMs. AI companies will sell this as a safeguard against misuse, and everyone will be happy about it. The politicians will say they accomplished something, the AI companies will slowly turn public opinion, and the public will have shiny toys.

                > The number of legitimate use cases for costly frontier models drops significantly once eliminating professional jobs is off the table.

                I don’t know. They can open possibilities that we don’t necessarily consider.

                One example I have is a friend who is getting his house refurbished. He’s not an engineer or a material scientist. He does not have enough free time to read thoroughly on the many subjects involved. With a decent LLM, he could untangle the technical documents sent by the architect and the contractors to really understand what was going on and be involved, rather than passively follow the architect’s advice. For starters, the LLM was very useful in finding issues in the quotes he received when he was looking for an architect. Those were long, technical documents, with no really standardised structure and full of jargon. I don’t think that person is going to want to stop using LLMs now. Many people are having this sort of moments right now.

                > In the near future the EU will likely come down with heavy intervention to prevent AI from impacting employment rates across Europe.

                Maybe. But i don’t believe the EU is well equipped for that. Labour laws are largely local and different in each member state. The EU regulations are basically the common denominator, ore or less, and it is easy to see why: for regulations to get adopted, they need a strong enough majority in the Commission, in the Parliament, and in the Council. It is very difficult to get anything controversial that affect the sovereignty of member states passed.

                The angle of the current AI regulations is that they set the rules for the single market, which is where the EU is the most legitimate. It is difficult to see market angle for the effect of AI in labour, and I think enough member states would be keen to kill the project.

                Also, there are many influences at play, but the EU is fundamentally an economically liberal institution. It very rarely goes in the direction that reduces economic activity. Look at how clumsy it is at fighting against cheap Chinese imports. I don’t think the institutions themselves would really want to make AI illegal. Companies have too much to lose.

    • estebarb

      yesterday at 4:53 PM

      What happens if we train models (GPT or human students) using the outputs of a model with text havingbthose watermarks? Is there something preventing the watermark from being learnable?

      • ludicrousdispla

        yesterday at 6:52 PM

        no, and in fact they should be extremely easy to detect if they are substituting a token that is a least-probable synonym

    • slickytail

      yesterday at 5:16 PM

      [dead]

  • ddalex

    yesterday at 11:46 AM

    Probably they bias the RNG for selecting the next token. This can be done practically in a lot of ways, including during training.

    I suspect the signal will be significantly under the noise floor, so it's not detectable if you don't know exactly what to look for, but certainly you can submit more information then the textual contents.

    • vintermann

      yesterday at 12:21 PM

      But if the user's prompt is in the context, you don't know exactly what the RNG chooses between. I don't know what trick they use to get past that, but it seems impossible to get by it in the general case (i.e. if the prompt can be anything) and you'll probably quickly compromise quality if you try.

      • Majromax

        yesterday at 1:11 PM

        A reasonable guess about the algorithm is 'A Watermark for Large Language Models' (https://arxiv.org/abs/2301.10226). The idea is that each generated token (or bigram) seeds a strong PRNG that splits the vocabulary into a 'green' and 'red' set. The sampler then tries to select a 'green' next-token for generation.

        After-the-fact checking only needs the vocabulary splitter, which is independent of the LLM. Over a sufficiently large text non-watermarked text would expect to use green and red tokens with the baseline probability, and that difference can easily become statistically significant over sufficiently long texts.

        The basic algorithm has obvious knobs to tune, among them the initial ratio of red to green tokens and how hard the sampler tries to pick a green token. These would balance fidelity to the original distribution against watermark detectability (minimum required content length for statistical power).

        • tmp10423288442

          yesterday at 4:09 PM

          Anthropic actually tells you the approach they use, and it's not that. From their Claude Text Watermark page[0]: "Claude’s text watermark is a version of the SynthID-Text approach published by Google DeepMind in a Nature paper in 2024."

          The Nature paper is "Scalable watermarking for identifying large language model outputs"[1]. This method does not separate out tokens into separate classes, but merely uses a seed for the PRNG that selects which among the most likely tokens generated by the LLM will actually be output. This has the advantage that there's no green and red token sets, so no token is systematically favored or disfavored. If a particular token is overwhelmingly predicted to be the most likely candidate, it will almost certainly be selected, so the watermark doesn't affect that. Even if there are several choices of output token at a point that have similar probability of selection, the watermark doesn't systematically bias in favor of one token or the other.

          This is actually a quite elegant method of watermarking that, contrary to people's fears, won't adversely affect the model output. The main concern I have with it is that it appears that you can't actually test the watermark locally, without uploading it to Anthropic. I'm not sure why that's the case, since there's no particular reason the watermarking key has to be private, except if you want to prevent others from generating text with their own LLMs that is watermarked to look like it's generated by Anthropic - but everybody wants their text to not have the watermark.

          [0]: https://www.anthropic.com/news/claude-text-watermark#:~:text...

          [1]: https://doi.org/10.1038/s41586-024-08025-4

  • cocoa19

    yesterday at 5:53 PM

    Removing the watermark usefulness depends on your use case.

    If you care to avoid detection, yes, it is useful. If you care about the best possible sequence of words, then the damage is already done once watermarked.

  • cratermoon

    yesterday at 4:44 PM

    I realize that short attention spans are pervasive now, but the link to the explanation is only eight paragraphs in https://declaude.org/watermarking/

    • ozozozd

      yesterday at 8:11 PM

      Yeah kind of surprised that people didn’t have enough patience for Gruber’s writing.

      It’s quite fun and engaging.

  • nullbio

    yesterday at 12:20 PM

    I'm sure they could use it to fingerprint people, at the very least.

  • unclebucknasty

    yesterday at 1:04 PM

    Simple version: In instances wherein the otherwise statistically chosen next word is a "toss-up", watermarking removes the randomness by imposing specific choices, determined by a key. This then becomes a detectable pattern when scanned with the key (stastically—detection itself is probabilistic).

    >use it to store arbitrary information

    No additional data is embedded. The range of available data is constrained by the text being generated (i.e. the sets of "next words" per text).

    • mbreese

      yesterday at 1:56 PM

      From what I’ve read, they won’t be imposing specific choices, but using a different (biased) RNG for those “toss-up” choices. With enough sampling, you could detect if the RNG was biased or not.

      • unclebucknasty

        today at 12:21 AM

        This is what I meant by "imposing specific choices, determined by a key". Maybe "impose" or "specific" were too strong in my attempt to simplify?

        I attempted to clarify that the impositions themselves are not deterministic, by indicating that the entire process is still probabilistic.

        Maybe Anthropic's explanation is simple enough [0]:

        >When watermarking is used, choices are still made at random, but the source of the randomness is different. Instead of using an arbitrary random number generator to pick the next word, watermarking uses the key and a few words that come before to settle what word the model should pick. That is, the words that Claude picks are still random, but now, one can check the sequence of words and see if it’s consistent with the choices Claude would make if it was using the key. If it is, one can assign a probability that the text was generated by Claude.

        [0] https://www.anthropic.com/news/claude-text-watermark

  • j45

    yesterday at 6:49 PM

    It’s possible to ask each model and build it.

• andy_ppp

  yesterday at 12:16 PM

  Doesn’t this mean Anthropic can accuse anyone of using their AI to write for them?

  • mapt

    yesterday at 12:40 PM

    Your favorite anti-AI political candidate turns out to have not written their thesis, with a 73% confidence level.

  • mbreese

    yesterday at 1:59 PM

    Is watermarking really watermarking if it can’t be independently verified?

    I mean, all of these text content watermarking schemes require the company to assess if the text was AI generated or not. They aren’t going to tell us where the toss-up tokens are or what is in the red vs green pools of words.

  • jjk166

    yesterday at 3:54 PM

    Seems like exactly the sort of problem the threat of defamation lawsuits are meant to solve.

    • pjc50

      yesterday at 4:02 PM

      Evidence is a massive problem here. As well as the extremely high threshold for US defamation; political candidates routinely tell the most absurd lies about each other.

      • jjk166

        yesterday at 4:21 PM

        The high threshold is only for public figures, which must show actual malice. For everyone else (e.g. your typical student falsely accused of plagiarism), they only need to show the statement was false, published, targeted the plaintiff, and caused damage - it's still defamation even if they were simply negligent in checking the facts. The company might be able to cover themselves if they are very clear they are just giving a statistical probability and not saying their model was definitely used, but if they imply they have hidden information that points to their conclusion (which is kind of the whole point of their algorithm) then that's not a valid defense. I can't say they would lose in practice, but in theory they ought to lose.

      • inigyou

        today at 7:36 AM

        In places like Germany it's a crime to say something that makes a politician look bad, even if it's true.

  • SatishPophale

    yesterday at 2:50 PM

    yes, why would someone will use this tool for writing then?

  • marianobayu

    yesterday at 2:21 PM

    [dead]

  • MagicMoonlight

    yesterday at 1:44 PM

    [dead]

• josephg

  yesterday at 9:47 AM

  I think this is a very real concern. But I’m not sure of any way around it. Any stenographic system that you have the code for can be trivially defeated.

  I wonder if this would be a good use for homeomorphic encryption. There might be a way to let anthropic check some text without actually giving them access to the source text. Any experts around? We could use your skills!

  • zrm

    yesterday at 10:15 AM

    > Any stenographic system that you have the code for can be trivially defeated.

    They're giving you an oracle regardless, which is almost as good. Take LLM output, make some modification, ask the detector if it's LLM output, repeat until you learn what kind of changes you have to make to defeat it.

    Or don't even bother learning what to do, just make arbitrary changes until it says it's not, so when the person they're submitting to does the same check it says the same thing.

    • kevincox

      yesterday at 11:43 AM

      I assume this oracle will be behind 20 layers of anti-bot protection, CAPTCHAs and hardware attestation challenged. It will be incredibly painful to use. It won't stop the motivated attackers, but will make it too annoying for the average person.

      • teeray

        yesterday at 6:34 PM

        When all else fails, you can hire a lot of folks cheaply to effectively Mechanical Turk it with their home internet connections.

    • sebastiennight

      yesterday at 11:19 AM

      > it says the same thing

      Reference needed? I think it remains to be proven whether those detectors can be considered deterministic.

    • _flux

      yesterday at 11:48 AM

      They could (..and probably will..) store that version and then refuse the check if this attack is detected, i.e. the version is too close to a known LLM output.

      Alternatively they could also just keep saying "yes" if it's close enough to a version that was close enough.. Although that would enable the attack to allow arbitrary text to be "proven" AI, by slowly morphing close-enough generated material to the desired text. But perhaps this is not a problem they are not concerned with.

      To satisfy the letter of the law I expect it's enough to just provide the oracle, without any mitigations.

  • piker

    yesterday at 9:49 AM

    Won't we just be able to fine tune OSS models to detect these patterns across providers? It will be cat-and-mouse but my bet is it converges to a central detector that isn't affiliated with any model provider.

    • josephg

      yesterday at 12:05 PM

      > Won't we just be able to fine tune OSS models to detect these patterns across providers?

      A good fingerprint should make use of cryptographic signatures. Without knowing the keys, the fingerprint should be indistinguishable from noise (or just random token selection)

• apparent

  yesterday at 5:23 PM

  And I'm sure students will use tools to have every other paragraph written in the style of a different AI, in an attempt to defeat this fingerprinting.

  • bluecalm

    yesterday at 5:36 PM

    Yeah, wait for LLM "scrambles" that put every paragraph and then the whole text through multiple re-write/edit style cycles.

    • solid_fuel

      yesterday at 7:13 PM

      How would that change anything? The proposed watermark is applied while the output tokens are being chosen, taking that text and running it through an LLM again would just repeat the process.

      • apparent

        yesterday at 8:24 PM

        It says it only applies to passages over 200 words, so you could use a different LLM to rephrase every other paragraph and undermine the watermarking.

      • bluecalm

        today at 7:07 AM

        You take the output and run it through another LLM with "please re-write this in xxx style". Then you repeat that a few times on different part of text and glue it all together at the end.

• atroon

  yesterday at 4:16 PM

  >Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.

  Uh, yeah, why do you think it's setup this way? The frontier companies desperately desire more high quality human text and this is how they are planning to get it for free.

• ShinyLeftPad

  yesterday at 12:00 PM

  That's why detectors like Pangram exist too I think.

  • saltwatercowboy

    yesterday at 2:27 PM

    Pangram doesn't work.

    • adamgordonbell

      yesterday at 5:04 PM

      Like their fp rate is a lie? It works well in my limited testing.

      • saltwatercowboy

        yesterday at 10:20 PM

        It's idiosyncratic to the point of uselessness in mine.

    • ShinyLeftPad

      today at 2:58 AM

      what do you mean by "work"? I think it works perfectly as UGC honeypot.

    • redsocksfan45

      yesterday at 6:06 PM

      [dead]

• rolandog

  yesterday at 7:17 PM

  Exactly, so even if you are avoiding AI, any interaction with society is now being structured so you have to submit to the digital surveillance equivalent of a cavity-search machine. What a dystopia awaits the budding generations.

• stillpointlab

  yesterday at 4:34 PM

  Sounds like a business opportunity.

• Alex3917

  yesterday at 3:34 PM

  > checking any text for watermarks requires sending the entire text to Anthropic

  Couldn’t it be checked in the TEE using confidential computing to keep Anthropic’s algorithm secret?

  • ghrl

    yesterday at 5:20 PM

    I can't speak for Anthropic, but with Google's SynthID, the algorithm is actually public. However, checking (or creating) the watermark requires a symmetric key, and the providers likely wouldn't share that key.

• kergonath

  yesterday at 2:45 PM

  > And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, MistraL

  I would think that these things would eventually converge and we’d get one watermarking algorithm as an industry standard. That way, all major provider would follow it and we’d get independent software for checking. This would partly limit the efficacy of the watermarks, but on the other hand if it’s done correctly, removing the mark could still be enough of a pain that casual users would not bother. That would obviously depend on a lot of factors. It would at least add significant friction in the production of daily slop.

  Of course it wouldn’t do much for thing like foreign propaganda but that’s a whole other discussion we need to be having.

  > Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.

  Isn’t it already what they are doing right now with some of the plagiarism detection tools? Not every university is going to have a representative corpus, and yet they are all using the software. So I guess the provider is doing the work of feeding all that data to their algorithm.

  • ComputerGuru

    yesterday at 7:44 PM

    You can’t check it with only the algorithm, you need the secret seed key. Which will be different for each provider (and they’ll probably have and use multiple). And you need the llm itself, to generate the potential tokens at each step.

• PunchyHamster

  yesterday at 1:24 PM

  I expect them to work as reliably as AI text generators do now...

• saidnooneever

  yesterday at 11:24 AM

  its lovely training data. no detection? add to training set -_-.

  its also kind of laughable that somehow people are trying to prevent the outputs not to be altered. Asif you cannot manually paraphrase anything you can read. So the only solution would be, to make it utterly unreadable (which is not possible, it obviously defeats the purpose of the thing).

  Not to mention local models ofcourse :-)

  • taneq

    yesterday at 12:37 PM

    I think what we’re testing for here is LPM output that hasn’t even been skimmed by a human, let alone paraphrased.

• micromacrofoot

  yesterday at 7:26 PM

  You've discovered the perpetual mutually assured destruction money generator — guess what the best defense against LLM spam also is?

  LLMs cause a wide number of problems, the good news for our investors is that they're all solvable with LLMs.

• j45

  yesterday at 6:49 PM

  While Anthropic is sharing this publicly, there’s isn’t much reason that other models could quietly be doing this or start.

  Local models could probably catch some patterns.

• LtWorf

  yesterday at 2:30 PM

  That's just storing what they output in a database and then checking, not a watermark.

• applicative

  yesterday at 1:33 PM

  Anthropic had absolutely nothing to do with this. The Chinese models will soon be adopting such devices as well. It is an overwhelming force coming inter alia from educators worldwide.

• yesterday at 10:52 AM

• applicative

  yesterday at 1:23 PM

  I’m worried that until such tech is perfected, pervasive and uniform across all models, education will be dead, as it certainly is at the moment. Flat out dead.

  I read stacks of term papers all year and it is a reality that, apart from such schemes, we are in an extinction event for civilization.

  • logicchains

    yesterday at 4:15 PM

    Education is in the best place ever for people who actually want to learn, they can have 24/7 access to a tutor with a wide breadth of knowledge and infinite patience for stupid questions. Education is in a terrible place for people who just want to get a degree and don't care about actual learning, but such people generally don't contribute much anyway, and are the easiest to replace with AI, so no big loss.

• Topfi

  yesterday at 10:40 AM

  Why not switch it around? Solves problems of privacy (no data upload), is far more doable and it's far more important to be able to verify content has not been tampered with after originating from a human or capture device (like a camera). We also have decades of cryptographic experience in reliably signing text, images, video, etc. and it doesn't break apart because a text is too short.

  Having proof that content (especially images and video evidence) is unmodified (whether via Photoshop, Paint or a model) is far more valuable then having evidence that an image was manipulated or generated fully by a model (which still leaves other forms of manipulation), I feel the same goes for human authored vs generated text. Free to admit that using models to generate any kind of media whole-cloth is still unappealing to me and I still pay for commissioned artwork or make it with my limited abilities for what that's worth. Do like to (poorly) write my musings too and see UX as something were thoughtful contributors (like the opinionated, sometimes controversial, but certainly talented GNOME Gitlab contributors) can make a major impact.

  Code can be beautiful, interesting and serve purpose beyond execution, of course, but for most people, in most cases, it does not in the same way as audiovisual content (not limited to art). Having code just to execute and resolve a problem can have value all in itself, the code being a means to an end whose quality, let us be honest, was barely a concern in most corporations long before LLMs.

  Also have rarely (honestly never) before LLMs fully owned all parts of any code base, always relied in part on someone's prior effort in (Flutter/Dart mostly) packages, whereas when writing, drawing, etc. I have far more situations where I make something from scratch and everything there is only there because of my conscious decision. Even simple marketing mockups that, quality wise, any modern model would beat feel different when I was fully in control, where to place what, etc. Objectively worse (at my skill level), probably, but still never the same.

  Knowing something was made from scratch by a human has value to me, beyond misinformation prevention. Knowing for a fact that LLMs were used instead of importing a library, using a template, or something similar that leads to expending similar amounts of effort, I don't see that being nearly as valuable. Heck, with all the importing and my experience back then vs now, I am spending more effort actually fully reading any LLM output in my code then I spent back then auditing Flutter/Dart packages. Then again, LLM output fails far more unpredictable then those messy packages that simply got Gradle to take down my system...

  Happy to admit, I have been skeptical of watermarking LLM output being feasible for quite some time and having looked into SynthID Text and proposals being researched, I am convinced that it is challenging to impossible beyond the lowest common denominator and less important then proofing human authorship.

  It will catch people just copying LLM output into their replies without thought, which is not a negative in my book, especially if it is not discernibly affecting output quality in regular use cases. Anyone who wouldn't copy Wikipedia into their dissertation will, in my opinion, be able to bypass text watermarking as proposed however, I feel we need to be honest there.

  Thing is, if that's the case and text watermarking will only ever catch LLM created slop, is that a bigger problem then the misinformation, harm to creators due to authorship questions and accusations, making it harder to use evidence in proceedings, teachers not trusting students even when they did the work themselves, etc.? Signatures for all such cases will be difficult to implement, yes, but I feel are going to be of greater value in the not to distant future and I equally feel are not impossible, not least because idiots will always want to hide their LLM usage, whereas human authorship is something they take pride in and want to proof.

  • piaste

    yesterday at 11:52 AM

    > Why not switch it around?

    Because a malicious human will gladly copy/paste LLM text and sign it with his "I, a human, definitely wrote this academic paper" key?

    • Topfi

      yesterday at 12:14 PM

      Fair point, for text it is far harder to prevent signatures being applied to generated text vs images at the moment of capture and most approaches I can come up with to remedy this can either be bypassed (edit histories can be output by models similar to humans) or will be controversial. Taking a page out of the anti-cheat textbook, mainly written for gaming, there are methods which might hold in the medium term. Less a fan of kernel level myself, though it might be worth exploring as there has been massive investment by the games industry into making it somewhat robust, but the approach Valve has taken with VACnet could be an inspiration worth exploring that is less invasive into peoples systems. Keystroke analysis, etc. could be relied upon as a basis for signatures, harder to spoof for current day LLMs over generating edit histories.

      I will fully admit that at a point in the future, maybe not too soon, models may be trained to bypass that too, at which point we are back where we started. As a skeptic of the extend that capabilities are emergent in LLMs vs specific to training data, I am somewhat hopeful that unless models are specifically trained for evading such human detection solutions, they'd struggle to do so, but it could still end up as a byproduct of improved, lower latency computer use focused training. Not emergent as the term is used in regard to models because that is still output performance improvements clearly traceable to very specific training data, but incidental as the goal of said training data was not to bypass.

      For what it's worth, I find human authorship being verifiable to simply be the more crucial problem over watermarking model output, so if research is to focus on one, I'd rather it the former. Maybe both signing human authored content and watermarking LLM output are both only possible in the near term, I hope not but fear it that might be the case. If so, we as a society will have some major challenges ahead (beyond all the ones we'd have anyways).

      Alternatively, we could also just start scanning everyones eyeballs...

      • mrngld

        yesterday at 12:37 PM

        I've been thinking for a while that all of this is just trying to grasp tighter the last bits of sand escaping between our fingers. The end game, perhaps, is trust. Do you trust or know the source? If you don't, assume it was AI generated. If you do, accept it as authentic based on whatever they disclose, but know that it's possible they aren't being totally honest or were themselves fooled in some way, depending on the context.

        Then build our assumptions and how we operate around those trust levels in the digital realm.

        • a2ff6eeb0

          yesterday at 6:21 PM

          Well, the end game is that AI is better at thinking us, and having a human brain involved is a net negative. The goal is to let the AI do the toil if thinking for us, and we can get the rewards.

        • chancedurham

          yesterday at 7:57 PM

          [flagged]

  • intrasight

    yesterday at 12:00 PM

    Thank you for stating clearly situation. I fully agree with your assessment. For almost 10 years now I have been saying that we need to virtually watermark reality. By "virtual" I mean store the metadata about the digital capture on a public blockchain. Then my devices could have a built-in "fake vs real" detector.

    Artists, photographers, journalist, etc. are going to want and need this.

    • bluescrn

      yesterday at 12:30 PM

      Some sort of signatures to verify legitimate unmodified photographs (taken with a relatively secure device, e.g an iPhone) seems like a reasonable idea

      But even then, people will be able to point that camera at a manipulated/generated image (either printed or on a screen). Maybe that one could be solved if the photo included some depth information?

      • intrasight

        today at 11:43 AM

        I doubt that NYT, WaPo, or any other mainstream journalist would do that.

    • exhumet

      yesterday at 12:32 PM

      a lot of cameras do have an in camera hardware based cryptographic signing (i think it runs off its own chip on them?) but any modifications to the image immediately break that. so for stuff like journalism etc where youd hopefully have a lot of settings dialed in on camera for quick turn around would keep that fingerprint intact.

      • davidgay

        yesterday at 1:36 PM

        Editing is fine as long as you keep the signed original for validation purposes.

    • warkdarrior

      yesterday at 3:53 PM

      You'll be happy to learn about C2PA https://c2pa.org/

• yesterday at 6:07 PM

• SadErn

  yesterday at 12:22 PM

  [dead]

• chii

  yesterday at 9:54 AM

  > blindly trusting they won't train on any of that

  being allowed to train on any data that you can legally obtain ought to be a right for anyone.

  After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read). The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.

  • bonzini

    yesterday at 10:03 AM

    It's a bit different when "training on any data" means basically storing a lossily-compressed copy of that data, that could be spit out years later if the model decides to do so.

    • TeMPOraL

      yesterday at 10:35 AM

      It's exactly the same problem as with humans, though.

      It's part of why we sign NDAs, and why their duration is measured in years (and that's not even targeting the human retention - just duration after which information ages enough that its disclosure is not likely to negatively impact anyone who cares).

      • bonzini

        yesterday at 10:38 AM

        It's not exactly the same problem, in that you can parallelize usage of an LLM and copy it over to another computer, but cannot do the same things with a brain. Put it another way, humans do not have the processing power needed to answer hundreds of millions of queries per day, while LLMs do.

      • gmerc

        yesterday at 10:43 AM

        It’s not because humans are rate limited which is the entire point.

      • afiori

        today at 3:39 AM

        Weirdly I think humans should have more rights than machines

        • TeMPOraL

          today at 9:05 AM

          Yes, but in a big way, denying capabilities to LLMs is denying the capability for people to exercise the rights they have. LLMs are user agent tools.

  • bmacho

    yesterday at 10:35 AM

    > being allowed to train on any data that you can legally obtain ought to be a right for anyone.

    I have the opposit viewpoint to the extreme. They shouldn't be allowed to even read that data until they are very clear about what they will or not do with it.

    Can they publish it? Can they store it? Can they use the information in it on prediction markets? Etc.

    Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated.

    • moron4hire

      yesterday at 12:31 PM

      > Humans reading texts historically come with little negative consequences

      IDK, we do have laws against opening other people's mail. Those have been on the books for hundreds of years. Seems like someone figured out a while ago that certain unauthorized humans reading certain restricted text wouldn't be good.

    • UqWBcuFx6NV4r

      yesterday at 11:11 AM

      > Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated.

      Citation needed. This is sounding tautological.

      • Timshel

        yesterday at 11:42 AM

        Not really hard to find example: License plate ALPR like Flock ?

        • ragequittah

          yesterday at 3:30 PM

          And not really hard to find examples of humans reading text being way more consequential than that. See: the Bible, nazi propaganda, etc.

          • afiori

            today at 3:37 AM

            A legislative solution to Flock seems a lot more feasible than one for literacy

  • lelanthran

    yesterday at 11:50 AM

    > After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read).

    Are you a tool?

    Because humans gets rights, tools don't.

    Arguing that untrained or partially trained models should have have rights is a different argument to arguing that a trained model should get the same rights as a human.

    • slackfan

      yesterday at 4:40 PM

      What if I'm reading it for work? What am I but a tool for the corporatioN?

      • lelanthran

        yesterday at 4:47 PM

        Are you claiming that you have no human rights while you are at work?

  • saghm

    yesterday at 10:25 AM

    At the risk of stating the obvious, there are a lot of legal rights that are human-specific (voting, holding office, filling lawsuits, etc.). It's not at all obvious why you think that you as a human being legally allowed to learn from something implies that it should be legal to train an LLM on.

  • vanviegen

    yesterday at 10:12 AM

    > The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.

    But LLMs are replacing the original, just in different words.

    And what does 'legally obtain' mean in this context? Copyrighted content is usually licensed for specific purposes. So if a license is given from training your LLM, then by all means do! But what if the license is 'for personal use'... ?

  • passwordoops

    yesterday at 10:44 AM

    You are one person. The corporation is not. Scale matters

    • cookiengineer

      yesterday at 10:54 AM

      > You are one person. The corporation is not. Scale matters

      Correct, if you violate it too often to count, you have to pay around less than ~2.5ct per violation.

      So the lesson here is: Create a company to do torrenting professionally, and resell its values for higher prices. Then get sued and pay a dime on the dollar you made.

      edit: Actually it's 2.5ct per violation.

      • subscribed

        yesterday at 1:39 PM

        Anthropic settled for $3,000 per book(1) in the settlement, so it provably depends of whether the government likes you or not.

        (1) https://apnews.com/article/ai-anthropic-copyright-settlement...

  • exe34

    yesterday at 10:40 AM

    Oh so if I use mickey mouse in a completely original production that doesn't replace the existing work by Walt Disney, you reckon they'll be fine with that?

  • Gud

    yesterday at 9:59 AM

    Why would that be a legal right?

    Why should we hand over even MORE power to the owner class?

    In a fantasy world this could be possible yes.

    • rcxdude

      yesterday at 10:05 AM

      Copyright (or any other such restriction on free use of information) creates power for owners by the simple fact that it turns information into something that can be owned.

    • juggle-anyhow

      yesterday at 10:12 AM

      Make it a right, then companies/universities will think twice before using said APIs. Instead of this grey area where we will never know.

    • m12k

      yesterday at 10:07 AM

      We don't hand over more power to the owner class by making fewer things ownable.

> I want any LLM I use to choose the very best, most precise words at every single decision point.

Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo random. The output it generates is one of the possible outputs it would have generated before, just now it's deterministic and will generate the same thing every time.

• red_admiral

  yesterday at 7:53 AM

  The best discussion I've seen so far is from Scott Aaronson: https://scottaaronson.blog/?p=6823

  > To illustrate, in the special case that GPT had a bunch of possible tokens that it judged equally probable, you could simply choose whichever token maximized g [a cryptographic function]. The choice would look uniformly random to someone who didn’t know the key, but someone who did know the key could later sum g over all n-grams and see that it was anomalously large. The general case, where the token probabilities can all be different, is a little more technical, but the basic idea is similar.

  • tmp10423288442

    yesterday at 4:14 PM

    Scott Aaronson basically invented this form of watermarking, so it's not surprising he has a good description of it.

• dragonwriter

  last Sunday at 10:26 PM

  That's inaccurate in two ways:

  (1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and

  (2) The actual behavior that is approximately what you describe already usually involves use of PRNG (with a user or harness supplied seed), not a true RNG; the change to do watermarking isn't going from RNG to PRNG, it involves adding an additional set of constraints on token generation on top of the existing ones, which inherently compromises quality.

  • tempestn

    yesterday at 8:55 AM

    You're generating a pseudo random number one way instead of another way. How would that inherently compromise quality?

    • Phemist

      yesterday at 3:05 PM

      1. As watermarked text is added to the training data, watermark-related tokens will be associated more with AI outputs and thus lower quality outputs which will hasten model collapse. Especially because every provider has its own secret key and they are all training on eachother's outputs anyway.

      I guess they can at scale filter the watermarked documents (by necessarily allowing eachother to at scale checked for watermarks, but banning the labs not part of the watermarking-cabal). Makes me wonder how useful the human quality filter is on AI output - if a human judges a given output as genuinely good and posts it somewhere for the scrapers to find and take into the training sets, will these types of outputs also be filtered out?

      2. (raw, pre-watermarked) Output token probability situations where 1 output token has the majority of the probability mass associated with it, but it is not in the watermarked set, will force the model with much higher probability to walk a non-optimal latent space. E.g., if the next OBVIOUS token for a given sentence would be a point, but the model is in this way not allowed to output it, it might put a comma and start off on a whole different tangent just to make the initial non-optimal comma grammatically make sense.

      • lanternfish

        yesterday at 3:16 PM

        There are no watermark related tokens, there is no watermarked set - read the paper, it's public and not that complicated. It doesn't change the distribution of completions, and won't change the situation where one output token has the majority of probability mass.

        You're missing the fundamentals here!

        • Phemist

          yesterday at 4:46 PM

          I am going off the explanation in the declaude page (and related papers). But I see now anthropic mentions Aaronson's distortion-free watermarking.

          Random watermarking functions colour the tokens based on (small) contexts and a secret key. Given watermarking functions are randomly chosen every single time they are used (so essentially not deterministically seeded by the context and secret key), then indeed the completion distributions are unchanged. However, they are deterministically chosen, a given string of text will always have the same corresponding watermarking functions. Tokens scored high by the function see an increased probability of being the chosen completion, those scored low see a reduced probability. I dont see it is different from merely talking about it as green/red and the points hold?

          • lanternfish

            yesterday at 5:40 PM

            For a given short-context (hash function seeding) you do have detectable manipulation of the completion (how they can read the marking). But, because (assuming enough entropy in context) the hashing function is decoupled from the log-probs, the perturbations are independent from the underlying distribution, so you're still sampling from the same distribution quotient some noise.

            The only way you'd notice this is they weren't independent, and the most plausible way that happens is if you're re-completing pre-fills (resampling the same hash function against the same log-probs).

            • Phemist

              yesterday at 9:21 PM

              Ok. I dont have a good feeling for the actual completion distributions. The noise sounds problematic. I can imagine it relates to the size of the context used for hashing. You want this as long as possible so that the entropy is higher, but you also want it as short as possible, because edits invalidate the hashing for all the tokens of which the edited tokens are part of the hashing context.

              Anyway, there are lots of cases where text carries very little entropy. E.g. boilerplate code, exchanges of pleasantries, well-worn platitudes and jokes, etc. These are sequences of tokens that will be seen across many, many separate outputs. Watermarking here (on the token following the common sequence) would thus be easily detectable and noticed as a claude style. The longer the hashing context though, the lower the amount of pathological cases with low entropy. It would be interesting to understand the exact parametrization better!

    • Spooky23

      yesterday at 6:13 PM

      These are services, so "how would <x> do this today?" is irrelevant.

      The real question is: "What can manipulation of pseudo-random number generation do?"

      We know that in the cryptographic world, attacking "randomness" is a key offensive capability. It will be here as well -- if Anthropic can watermark text as generated it's LLM, will it be able to watermark outputs as generated by "Spooky23/FooCorp"? Can I pay Anthropic to steer inquiries in a way that benefits my company or governemnt?

      Pseudo-random to the end user appears random. Most treat it like a random chance. It is not.

    • TeMPOraL

      yesterday at 10:51 AM

      Same reason that you don't just replace your rand() implementation with "return 4; // chosen by fair dice roll". If you need randomness for whatever reason, biasing the generator is compromising quality.

      ( https://xkcd.com/221/ )

      In case of LLMs, you can look at it from high and low level.

      At low level - if you could do with less randomness, you can always lower temperature. You usually keep it (or for SOTA providers' chat UI, they keep it) at a level where it's about right level - high enough to allow for more creative leaps and interpretations, low enough that it doesn't go off into crazy land after the third paragraph.

      At high level - creativity is driven by randomness. If you had an author (fiction or nonfiction) you like for their both broad and deep range of insightful thoughts, would you be happy if they suddenly developed an acute porn obsession and uncontrollably added lewd subtext to every other sentence? Still creative, still deep, but now with that one strong attractor that biases their every thought in a single direction? Would you trust/enjoy their output as much as you did before?

      That, slightly exaggerating to make it more obvious, is what "loss of quality" means here.

      • Nnnes

        yesterday at 2:24 PM

        You're missing the same point that the blog post is missing. What they're doing is much less like replacing rand() with 4 and much more like setting seed(4) before generating any numbers. There is no "loss of quality" unless you're already using a temperature of 0.

  • reliablereason

    last Sunday at 10:48 PM

    (1) LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time. This is a consequence of how they are trained.

    • dragonwriter

      yesterday at 7:55 AM

      > LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time.

      LLMs are likely to get stuck even with sampling if asked to generate tokens on their own long enough, though sampling does tend to stretch out the time before that happens (as do other techniques that don't involve sampling, like applying repetition penalties directly to token logits). But LLMs generally aren't left to infinitely extend their own output, and the length response typically needed in the use case is much shorter than the would result in collapse given the kinds of inputs expected in that use case, the existence of the theoretical eventuality may not really matter.

    • case540

      yesterday at 12:09 AM

      Citation needed

      • inigyou

        yesterday at 12:34 AM

        You know you can just try it and see on any inference system thst has this knob, right?

        Related: if you don't have a limit on sampling (top-K or top-P), eventually you'll hit one of the really unlikely tokens by chance and then the model will switch to Japanese because the most likely completion after a random Japanese character in the middle of an English sentence is more Japanese writing, not a reversal back to English.

        • cadamsdotcom

          yesterday at 12:57 PM

          That could be how it works, but in practice it takes into account all previous tokens when producing the next-token distribution to sample from. So a switch back is more likely than your explanation supposes.

          • inigyou

            yesterday at 3:57 PM

            No, if you switched to Japanese the LLM wouldn't ignore it, it would "assume" there's a reason for that. The same if the previous iteration of the LLM switched to Japanese. Else you're expecting an LLM to ignore its own previous outputs and restart "thinking" from scratch with every token?

            • cadamsdotcom

              yesterday at 7:43 PM

              It's situational and I suspect there are situations where it would and others where it wouldn't. Would depend on the almost infinite variables of how training was done. You'd be right that it'd be likely to switch but while it's possible it's due to temperature, there are just so many things going on. But it would be one sensible explanation among many.

  • thatjoeoverthr

    yesterday at 8:32 AM

    1) we’re not discussing those systems. We’re discussing a chat AI product called Claude, which does not offer those knobs.

    2) Claude’s PRNG having a P is immaterial

    • daishi55

      yesterday at 12:47 PM

      Claude has those knobs, they are just not exposed to the user. They could make Claude nearly completely deterministic if they wanted to (of course it would be a far inferior product then. But they could).

      • thatjoeoverthr

        yesterday at 2:30 PM

        “Not exposed” = has no knobs. Of course all autoregressive LLMs can be operated this way but Claude, the product, employs LLMs but isn’t one.

        • daishi55

          today at 2:50 AM

          Your original statement “LLMs use randomness in a fundamental way” is incorrect. LLMs have these knobs and randomness is not an inherent property of LLMs.

  • beering

    yesterday at 12:43 AM

    > which inherently compromises quality.

    I don’t see how this follows? Tokens are chosen randomly. If you choose tokens with a different RNG in the same distribution, you’re still getting equally good or bad tokens.

    • yesterday at 1:00 AM

    • TheOtherHobbes

      yesterday at 8:50 AM

      Not all values of "equally good" are equally good.

      Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it.

      Never mind metaphors and similes, which are even more tightly constrained.

      LLM writing is still a long way from good. Sometimes you get lucky with the odd line, but there's a difference in quality between influencer slop, genre fiction, and literary fiction and/or best-in-class journalism.

      LLMs are still somewhere between the first two, and nowhere close to approaching the third.

      • Closi

        yesterday at 10:52 AM

        > Not all values of "equally good" are equally good. Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it.

        We already know that a non-zero temperature improves quality though with current models (particularly with creative writing). The assumption that always picking the 'best' token results in the 'best' output is not the current reality.

        And if you are already intentionally putting in randomness, I can imagine that it would be possible to seed the randomness in a way that is detectable but results in the same quality.

        This is obviously not true for queries where temp = 0, but at temp = 0 then it becomes easier to identify anyway. I assume this technique implies some level of temperature.

      • tempestn

        yesterday at 8:58 AM

        It's not just swapping synonyms. The way llms work is by predicting the likelihood of the next token. It's inherently probabilistic. Choices are made based on weighted random number generation, based on those probabilities. Changing how you generate the random numbers doesn't degrade the output.

      • lyu07282

        yesterday at 11:12 AM

        wow even on HN people have no clue how any of that works? all LLM generation has some inherent randomness to it, if you replace part of that randomness to be deterministically random the result of the generation with the fingerprint and without it, is INDISTINGUISHABLE. This has absolutely nothing to do with "synonym swapping". Its also again people not understanding how anything works missing the real concern, which is that nobody can ever tell if it isn't also secretly a fingerprint with the user and session id.

• npilk

  last Sunday at 10:35 PM

  I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want.

  On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense.

  Even good human writers would probably struggle with this constraint. It would be like someone interrupting your writing to tell you the next word MUST be such-and-such, and then you have to try and make it work as best you can first try, without going back to edit. The result would probably be a little clunky. (Maybe it’s impressive LLMs write as well as they do.)

  • brookst

    last Sunday at 11:31 PM

    This is the classic misunderstanding that LLMs only pick the next token at a time. Really, they are coalescing the probabilities of a range of tokens at a time. There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.

    • Maxatar

      yesterday at 12:36 AM

      >There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.

      You're mixing up two claims here, and only one of these is kind of true. Yes LLMs do internally plan ahead in a way that is emergent rather than strictly part of their architecture, so that part of your claim is true. The way you word it by saying they are "coalescing the probabilities of a range of tokens at a time" is poetic sounding jibberish though. What's actually happening is one distribution output for the next token computed from a hidden state that implicitly encodes where the text headed.

      Your claim that if an LLM does happen to pick a token "th" instead of "tw", then the LLM isn't stuck with that decision is entirely false for autoregressive LLMs which is what all of the frontier models are. Whatever an LLM picks as its output token is final, it has no ability to undo that token selection and it must continue on the basis of that choice. It can't go back on that decision and revise the output.

      If you're interested in this, Anthropic has a summary of a very technical paper on this topic that mostly deals with this issue with respect to poetry:

      https://www.anthropic.com/research/natural-language-autoenco...

      • nextaccountic

        yesterday at 4:03 AM

        Reasoning tokens are a way to escape autoregressive woes. The model can generate a draft, then ponder on it, and use this to generate a final version

        • thatjoeoverthr

          yesterday at 8:34 AM

          They’re a way to mitigate it. It still writes like an LLM and everyone can see it.

      • yesterday at 3:59 AM

      • robocat

        yesterday at 1:18 AM

        What a crazy link:

          So we train a second copy of Claude to work backwards—reconstruct the original activation from the text explanation. We consider an explanation to be good if it leads to an accurate reconstruction. We then train Claude to produce better explanations according to this definition using standard AI training techniques.
        

        Incentives to train a pathological liar. There's no baseline so can only catch out the worst of the lies/errors. Anything (including fabrications) that passes our filters is reinforced?

        • ACCount37

          yesterday at 7:55 AM

          What are you even talking about? If there was a "baseline" for how to decode activations into text we wouldn't be having this conversation.

          The choice is between "this reconstruction sucks" and "no reconstruction", and we're only now beginning to learn how to make those reconstructions suck less.

    • inigyou

      yesterday at 12:28 AM

      No, they really do one at a time. You're incorrect on that.

      Mathematically, a long chain of conditional probabilities is equivalent to a single probability over the whole range. But computationally, for that to work out, the computation for the first probability needs to somehow consider all the downstream probabilities depending on it, which obviously isn't how autoregressive language models work. They can pack in as much downstream computation as their neural architecture allows for, which is quite a lot.

      Suppose in some context you have three equally plausible conpletions after "Be": "tween a rock and a hard place", "twixed he stood there" and "lieve he can fly". To model this probability distribution of the whole sentence, the next token "tw" needs to appear at 2/3 probability and "lie" at 1/3. After "tw" would be a 1/2 chance of "ix" and a 1/2 chance of "een"; after "lie" would be a 100% chance of "ve " and in any case the rest of the sentence after that would be 100%.

      The model needs to somehow "think ahead" to know those are the possible completions. For example if "lieve he can swim like a dolphin" was another equally plausible completion, that first token would need to be 50/50 instead of 67/33. So the computation of the first token somehow needs to encode the fact that the guy thinks he can fly but not swim, even though it doesn't become relevant in the output until several tokens later.

      In practice this probably happens to some degree but definitely doesn't happen perfectly. To perfectly model the first token's probability distribution, it would have to include knowledge of the entire distribution of all possible outputs, which is just not happening. So it approximates. Surprisingly, the approximation is good enough to produce language.

      You can see this breaking down in the seahorse emoji incident from last year. When you ask the model if there's a seahorse emoji, it first completes "Yes," as if a few tokens later it's about to produce a seahorse emoji. But when it actually gets to the token that would produce a seahorse emoji, it can't because there isn't one. But it's already outputted "Yes, the seahorse emoji is" and can't just go back and change that to "No, there's no seahorse emoji." Some models would try a few times and then say there isn't one or a system error seems to be making them unable to produce one, other models (including then-current ChatGPT) would loop forever with ensuing hilarity.

      • huflungdung

        yesterday at 12:56 AM

        [dead]

    • npilk

      yesterday at 12:20 AM

      But on some level there is uncertainty, right? Even if it’s not token-specific but at the word- or phrase-level? Otherwise what does the temperature setting do? Or has architecture changed significantly in the background?

    • moralestapia

      yesterday at 12:27 AM

      There are diffusion-based models and transformer-based models (and many other "architectures"), so your comment does not make sense.

      • inigyou

        yesterday at 12:29 AM

        Are there any diffusion-based or otherwise non-transformer-based models in mainstream use?

        • Sharlin

          yesterday at 12:38 PM

          If you include non-language models, yes.

          • inigyou

            today at 7:30 AM

            We are clearly talking about language models

        • moralestapia

          yesterday at 12:31 AM

          If by mainstream you mean "ChatGPT" or "Claude", then no.

          But also, no one really knows as they're closed.

    • doctorpangloss

      yesterday at 12:02 AM

      everyone in this thread is saying something kind of imprecise and reductive and varies between models and even modifications among the models

  • danofsteel32

    yesterday at 1:15 AM

    I tried brainstorming what an agent harness for writers would look like.

    https://chainofbranches.com/conversations/2/branches/20/

    I’m not convinced it’s possible. A good nights sleep and a notepad in a quiet room still feels like the state of the art toolchain for writers.

  • mholm

    last Sunday at 11:01 PM

    This was true in the ChatGPT era. Now we're in a world with reasoning tokens, where a model can thoroughly plan out the response it wants to make. If anything, it makes the style worse.

    • npilk

      last Sunday at 11:52 PM

      Yes, models can reason and plan, which helps them write more coherently. But when they write the final output, it’s still a single generation. It would be like letting a human make notes and write an outline, but not let them use the backspace once they start typing their response.

      Presumably you could use the same reasoning trace, run multiple generations, and get different outputs (if the temperature is >0).

      But now I’m interested in playing more with Cowork or Claude Code/Codex for prose writing to see if the set of tools there affects outputs at all. I guess you might need a more custom “writing” harness.

      • mholm

        yesterday at 7:37 PM

        There's been a lot of effort into the writing space, and the models genuinely prefer this style. You can let them iterate on the same idea 100 times, rewrite sentences, determine what works best — and they'll still verb the noun, do rule of 3, and keep the same monotonous structure.

    • tomrod

      last Sunday at 11:19 PM

      Isn't this just chain-of-thought though, doing the same thing multiple times without necessarily defining one path?

      • brookst

        last Sunday at 11:32 PM

        Chain of thought attempts to guide a single forward pass.

        Reasoning tokens with tool calling tell the model to loop on a one phase of a question and call a tool to indications completion when done.

        Related, but not the same thing.

  • halJordan

    yesterday at 2:26 AM

    Humans already do struggle with this constraint. Good examples are JRR Martin, Tolkien, and Rothfuss. You cant describe the struggle of picking the next word and then act like humans don't sit at the table struggling to pick the next word.

  • hyusap

    last Sunday at 10:58 PM

    autoregressive generation doesn’t mean the model is myopic. the next-token distribution can already reflect a longer horizon plan for the output sequence.

    • npilk

      last Sunday at 11:59 PM

      Sure, but mightn’t there be several plausible long horizon plans?

      Here’s an example: I had asked Claude for some music recommendations in a certain style. Part of its output was:

      —

      *Long journey tracks*

      Clinic — “The Return of Evil Bill”

      Guided by Voices — not really, wrong band

      Silver Apples — “Oscillations”. Proto-everything, deeply repetitive, hypnotic.

      —

      So at some point there, the next token produced was “Guided” or “Guide” or whatever, and then because it can’t go back, it had to correct itself after the fact.

      Reasoning/CoT have helped a lot, but I feel like small versions of this still happen all the time.

      Human writing is like 90% editing.

      • inigyou

        yesterday at 12:31 AM

        Would be fun to run an LLM on fake output from itself. Like just force the first N tokens to say the beginning of something really stupid, and then see how it finishes the sentence. "You're absolutely right! Human feces is actually the most effective engine coolant because $<completion>"

        • Gracana

          yesterday at 1:45 PM

          Some LLM interfaces allow you to modify and “continue” an agent response. It’s very useful for guidance, including jailbreaking. Need the model to go in a certain direction? Got a refusal that you want to bypass? Just start it off in the appropriate direction and then have it continue from there.

          • inigyou

            yesterday at 3:55 PM

            Which ones?

    • inigyou

      yesterday at 12:32 AM

      It can but it is limited because it's only got a single pass through the network to fit the entire "longer horizon plan".

  • eru

    yesterday at 8:10 AM

    > [...] and go back and restructure sentences and paragraphs if we want.

    That's what LLMs in reasoning mode do, too, to the text they present to you.

  • lesostep

    yesterday at 7:54 AM

    I also really liked the quote "Your existence is not impossible, but it's also not very likely" from the Night Vale podcast.

    I feel like the existence of good writing is also not impossible but not very likely, and so of course LLM can only write mediocrity, even when taught only on great writing.

  • disillusioned

    yesterday at 12:16 AM

    It's a bit like trying to finish a sentence when you're really stoned... you vaguely remember the preceding couple of words you've said but don't really know how you got there and now you're wandering in the forest trying to stumble on coherency.

    Well, I suppose it's nearly the opposite of that experience, upon further review. But for some reason, that's where my head jumped.

    • robocat

      yesterday at 12:37 AM

      After "stoned" you triggered into a different state.

      You be a human who's brain shifted into LLM mode (chainneling Markov?).

      Or perhaps you're an LLM impersonating humanity.

      I often wonder how much LLMs are just mirroring our own brain's patterns.

    • inigyou

      yesterday at 12:29 AM

      You're thinking of Markov chains.

  • zimpenfish

    yesterday at 8:07 AM

    > Even good human writers would probably struggle with this constraint.

    But that would be a fun writing exercise, I think. Thoroughly in the oulipo wheelhouse.

    Maybe generate a Markov chain table over all of Project Gutenberg and then say every 10th word is whatever the Markov Chain thinks it should be at that point?

    Or every Nth word has a P% possibility to be constrained by the chain? Optionally with the possibility building for each skipped word to guarantee it happens at some point. Bonus with this approach is that the human can't game the words leading up to the constraint because you don't know when it will happen.

  • cush

    last Sunday at 11:08 PM

    Models can easily do multiple passes

  • scuppernong

    last Sunday at 10:46 PM

    auto-oulipo

    • Alive-in-2025

      last Sunday at 10:57 PM

      Today I learned a new word, "Oulipo". Interesting.

      But what about the general idea that they can watermark results to tell where they came from. The next step is tracking down which user got a result. I hate both of these things. Must everything we do be tracked? Next altering wikipedia results so they can tell who looked at the page or something?

      I'd like "the best answer" from an llm and don't want to be tracked, but this isn't for me, it is for them. I understand llm results are already using a varying statistical input so they aren't always the same. But I really hate watermarking and likely tracking too.

  • lazide

    yesterday at 7:50 AM

    Human writers do better because they can think, and adjust, based on context.

    They are also usually worse (which is often better!) because they are usually lazy and don’t want to spend effort they do not have too, to accomplish their goals.

    Their goals are often complex and nuanced.

    None of this is true of LLMs.

• demibabs

  yesterday at 12:07 AM

  Yeah this is my main issue with the argument. He acknowledges in the article that LLMs are already non-deterministic, but he doesn’t seem to actually understand that.

• colmmacc

  last Sunday at 11:04 PM

  I think the article is wrong on this but it's more subtle than that. Probability distributions have a peak; there is still a token with a peak probability. What's interesting about these techniques is that token by token it can actually make the peak token even more probable. A distribution doesn't have to be "flattened" to leave a watermark - it can be "amplified" and made "more peaky".

• neuroticnews25

  yesterday at 9:26 AM

  This is true and the author seems to not understand the problems with greedy (top 1) decoding or the fact that watermarking affects only high entropy tokens. But the published watermarking methods still have a slight negative effect on perplexity, so there is something more to it.

• simonh

  yesterday at 9:09 AM

  The quality of the LLM just _is_ the quality of the token probabilities it generates. Better quality token probabilities, better quality output. Worse quality token probabilities, worse quality output.

  Watermarking changes the probability calculations for reasons other than quality. It can't not compromise quality. It literally leads the LLM to occasionally chose different tokens just for watermarking purposes.

  • simiones

    yesterday at 9:18 AM

    No one uses a pure random function over the whole probability distribution described by the LLM's output. For example, there is exactly 0 probability that the chosen next token by any common API or even local LLM runner would be a token whose final value is "0.0001" if there exist at least K tokens whose value exceeds "0.7".

    Also, as long as the same sampling strategy is used during training as the one used during inference, then the LLM will actually do much better with the biased sampling strategy than it would with a fair one - because that is what it was trained to optimize.

    • simonh

      yesterday at 10:52 AM

      >No one uses a pure random function over the whole probability distribution described by the LLM's output.

      So what? By definition with this system the LLM will chose tokens it otherwise would not, purely for watermarking reasons. Yes this token may have had a decent likelihood of being chosen anyway, but it wouldn't have been chosen and now it was for reasons nothing to do with output quality.

      I'm not sure what your last paragraph is trying to say. The blue/green list system changes what output the LLM would otherwise produce. You can't train it to produce watermarked output with this system. If you tried to, there would be no delta between trained output and watermarked output for you to be able to detect.

      • simiones

        yesterday at 12:13 PM

        My main point is that sampling with a modified distribution compared to the one produced by the model is already being done, and it is generally found to increase quality, not decrease it. So there is no reason a priori to assume that the watermarked distribution would be lower quality than other schemes for altering the "raw" output distribution (such as top P, top K, temperature, etc).

        My second point is that the training of a model by definition maximizes the fitness between the final output function and the training metrics. So, if the model is trained with the watermark applied, the training process will minimize the function `model_error(input) = |watermarked_sampling(model_output(input)) - expected_output(input)|`, by definition. This means that a model trained in this way will perform better when sampled using the watermaked_sampling method than if using, say, top_k sampling.

        • simonh

          yesterday at 12:31 PM

          All those methods are applied with the specific goal of improving output quality and are applied to the extent that they do this. Watermarking has no such goal, and is not implemented for any such reason. In fact it's much more like applying another layer of random noise over the token selection process, because the sequence that generated the green token list comes from a seeded PRNG.

          >My second point is that the training of a model by definition maximizes the fitness between the final output function and the training metrics.

          Right, but the fitness in question is watermarked text fitness, not fitness for any user interests aligned metric. You're basically saying that if we train LLMs on watermarked text they'll be really good at producing text that looks watermarked, and then we'll stick an actual watermark on top of that. Screw whatever the user wanted it to be good at.

          • simiones

            yesterday at 12:43 PM

            > All those methods are applied with the specific goal of improving output quality and are applied to the extent that they do this

            Yes, that's the goal that was used, but they are quite simplistic and crude methods, not some specifically designed function, with carefully fine tuned parameters or something. So, if a basic function like top_k can improve model utility, it's not impossible to imagine that watermarking could also happen to do so, or at least not have a significant negative effect. So whether the effect is deleterious or not is an empirical question, not something we can assume ahead of time.

            > You're basically saying that if we train LLMs on watermarked text they'll be really good at producing text that looks watermarked

            No, you're misunderstanding how the training works. If we train the model's output so that it minimizes the error function after the watermark is applied on it, the model will learn how to produce the best output it can given the watermark. It will produce better text that happens to be watermarked, not "more watermarked text". Same as if you train the model on minimizing `top_k_error(input) = |top_k_sampling(model_output(input)) - desired_output(input)|`, the model will learn to produce better output under top_k sampling, not learn to produce output that's "looks more top_k".

            • simonh

              yesterday at 5:03 PM

              But as I explained, the watermark is functionally random PRNG noise overlaid on the token probabilities. It’s not something that can be compensated for because it’s not predictable if you don’t have the seed and PRNG function.

              • tsimionescu

                yesterday at 8:32 PM

                If it's functionally random PRNG, then how does it differ from any other random sampling? If it's biased PRNG, then the LLM can adapt to the bias, and coincidentally might even benefit from this bias.

  • regularfry

    yesterday at 9:43 AM

    It doesn't, though. From the outside, without access to the parameters, you can't distinguish the watermarking system from a random number generator.

    We already use an RNG at inference precisely because it leads to higher-quality output. Changing what function is generating our random numbers changes the sequence, not the randomness from the point of view of a user.

    Fundamentally the article is railing against --temp > 0.0. He doesn't know what he's talking about.

    • simonh

      yesterday at 10:55 AM

      >It doesn't, though. From the outside, without access to the parameters, you can't distinguish the watermarking system from a random number generator.

      You can't as a user tell by how much the quality of the output was degraded. True.

      >We already use an RNG at inference precisely because it leads to higher-quality output. Changing what function is generating our random numbers changes the sequence, not the randomness from the point of view of a user.

      I'm not saying it wasn't random and now it is. I know how these things work. I said that the quality of the system is in the quality of the probabilities. That quality is being degraded.

      • rbits

        today at 11:54 AM

        > I said that the quality of the system is in the quality of the probabilities. That quality is being degraded.

        How is it being degraded exactly? The probability that it picks each option will still be the same, just deterministic based on a seed generated from the text.

        LLMs already use PRNGs. This is just changing the source of the seed. And a different seed does not change the "quality" of the random numbers. Even if you are worried that it somehow might, they can just use a cryptographic PRNG, then it is literally guaranteed that the source of the seed will not affect the output in any noticable way.

      • regularfry

        yesterday at 5:57 PM

        What you're arguing for is that the deviation from true randomness is worse than the PRNGs that are already used, and that the deviation is anticorrelated with some notion of quality. You need both. I don't think you've got either.

• georgelyon

  yesterday at 5:19 PM

  Another good way to think about this is that it does change the output, but in a way that is equally likely to make it "better" as it is to make it "worse".

  • kylebebak

    today at 2:15 AM

    That is not a good way to think about this. I don't have deep knowledge of how LLM's work, but the following is accurate enough to illustrate the point.

    Let's say the LLM is in the middle of text generation and "decides" that the next token is "dog" with p=0.55, or "cat" with p=0.45. With a temperature of 0, the model always picks dog, because it's the most likely next token. With a temperature of 1 the model picks dog 55% of the time and pick cat 45% of the time.

    With this watermarking scheme, the model might alter these probabilities s.t. p_dog for this particular generated token goes up or down. Let's say it does down, s.t. p_dog is now 0.45 and p_cat=0.55. Now, with T=1 the model picks cat 55% of the time and dog 45% of the time. Regardless of whether the "watermarking function" raises or lowers p_dog, the probability distribution for this token has changed, and whatever math this trillion dollar company and its brainiacs came up with to decide that p_dog ought to be 0.55 has been "adulterated". As others have mentioned there is no way around this.

    ---

    Regarding the watermarking scheme, it works because it doesn't just alter p_dog for this single output token. It alters probabilities for many of the generated tokens (it could do this to all of the output tokens; it's an implementation detail). E.g. at token N, it favors "cat", at token N+1 it favors "house", etc. This way, if you have the secret key that lets you generate the watermarking function for any output token, you can analyze a run of tokens and check whether it's likely they were generated according to your watermarking scheme. The longer the run of tokens, the more certain this check becomes (it becomes extremely certain quite fast).

• nathan_compton

  yesterday at 3:37 PM

  I cannot believe the markdown guy is this deranged.

• exabrial

  yesterday at 12:30 PM

  This is not the same thing

• avaer

  last Sunday at 11:13 PM

  That's missing the point. It's the distribution that's the "best", not the tokens. Then Anthropic comes in and makes the distribution something other than the best. The only saving grace is that Anthropic says it's not that bad.

  Even so, I don't think it will stop here. Once this is in place, the next step is to put more and more identification into the AI generated content; might as well pack it in, it's not that bad, and if it is they won't admit it. There's no way for anyone to check. And your argument will still be technically correct but missing the point.

  • inigyou

    yesterday at 12:36 AM

    there's absolutely no reason to think Claude produces absolute best token distributions or that slight adjustments would be noticeable.

    In fact we know it's not that good because we can often tell Claude's writing apart from human writing.

  • beering

    last Sunday at 11:32 PM

    The watermark doesn’t change the distribution, only per-token selection. I think not understanding that is the source of most people’s FUD.

    • inigyou

      yesterday at 12:37 AM

      There's no difference between those two things. The distribution that matters is the distribution of tokens that are picked not the distribution of tokens the LLM model passed to the selector.

      • beering

        yesterday at 3:04 AM

        On average the distribution of selected watermarked tokens is the same as the original distribution. You can test this experimentally.

        I think there is a possible weakness in the context of the watermarker but that is not your claim iiuc.

        • inigyou

          yesterday at 11:05 AM

          The distribution of token "ple" being the same on average, but lower after "crum" and higher after "cou", is not no difference. It's irrelevant that the single-token distribution is unchanged if the joint distribution is different.

    • cubefox

      yesterday at 12:16 AM

      This comment disagrees with you: https://news.ycombinator.com/item?id=49324387

      • beering

        yesterday at 12:46 AM

        That comment merely says quality must be compromised. It doesn’t make it clear why that must be true. Empirical study seems to say that quality is not compromised, and looking at various proposed schemes, it seems intuitively true.

  • brookst

    last Sunday at 11:35 PM

    How do you know you picked the singular “best” set of tokens in your comment here?

    Could it have been equal or better with slight variations in wording?

    The slipper slop argument is too lazy to address directly. Argue A is bad because A, not because A might become B and you’ve got good arguments against B.

> "The exact words we choose when writing matter."

Then write your own damn text if you care about the exact wording so much

• apparent

  yesterday at 5:24 PM

  His complaint seems to center on conversations between him and the LLM, not copy being written for publication elsewhere. That is, if Claude is going to teach him a new skill, he wants it to pick the most accurate words possible, not "pretty accurate words, subject to watermarking techniques".

• AshleysBrain

  yesterday at 10:20 AM

  If the exact words we choose when writing matter so much, then why use a non-deterministic LLM that produces slightly different output on every run?

• eloisant

  yesterday at 12:57 PM

  Additionally, LLM are already rolling the dice on different wordings so I don't see how this watermarking makes it any less precise.

• pchristensen

  yesterday at 5:39 PM

  He's a professional writer. As the article says, even if you write your own words, this is still a problem with proofreading, copy-pasting references or quotes, and "AI checkers".

• Arubis

  yesterday at 11:45 PM

  You are responding to the original article, authored by John Gruber, who, last I checked, has been writing his own damn text multiple times every damn day for multiple decades as a primary vocation.

  I’m willing to venture that Gruber is on the list of folks that get to hold the opinion choice of words matters.

• recursive-call

  yesterday at 12:04 PM

  The exact words matter to people who people who don’t use it to write for them. A lot of people use Claude as a friend/therapist/romantic partner/etc. That’s who I imagine would be most affected

  • nemo

    yesterday at 12:13 PM

    >A lot of people use Claude as a friend/therapist/romantic partner

    People developing a para-social (pseudo-social?) relationship with a corporate robot have far bigger problems than the word-chooser in their robot "friend".

    • summarybot

      yesterday at 3:02 PM

      Very intelligent people need intelligent-others to bounce ideas off of, and the LLM can be that.

      • kergonath

        yesterday at 3:29 PM

        Very intelligent people should be able to grasp that a LLM is not intelligent.

        • sailfast

          yesterday at 5:37 PM

          Have you tried to "rubber duck" an idea with an LLM. The latest models are pretty damnably good at it.

          • kergonath

            yesterday at 6:39 PM

            I have, and I still do this occasionally. I don’t believe the plant on my desk (I don’t have a rubber duck) is intelligent. I don’t need it to be intelligent, either.

            I also do it with a LLM every now and then and, while it’s feedback is more useful than a toy’s, it does not need to be intelligent either.

      • nemo

        yesterday at 4:44 PM

        Yes, I use LLMs this way, they're not friends, not therapists, not persons, they're tools, like a word/language calculator.

      • mossTechnician

        yesterday at 7:41 PM

        Even intelligent people have demonstrated they are not immune to the damaging effects of AI sycophancy. The most recent example that comes to mind is Hank Green.

      • Helloworldboy

        yesterday at 3:23 PM

        [dead]

  • tmp10423288442

    yesterday at 4:13 PM

    LLMs never are limited to "exact words", because their output is inherently probabilistic. The method Anthropic (along with Gemini, who has been using the exact same watermark for at least a year) uses doesn't bias the output token distribution, just reseeds the PRNG in a way that can be detected after the fact: https://www.anthropic.com/news/claude-text-watermark#which-s...

  • kergonath

    yesterday at 3:26 PM

    > The exact words matter to people who people who don’t use it to write for them.

    Right. If the exact words matter, using a non-deterministic LLM is a terrible idea in the first place. I hope these people never try putting the same prompt in different sessions.

    Also, now I am curious. How would these people interact with other humans? Is there anyone on earth who would provide the exact same reply, down to every single word, if we asked them the same question more than once?

  • pjc50

    yesterday at 4:03 PM

    How would you ever know if you've been affected? How, indeed, would you know at all whether the word you get next is different from some different word next?

  • 0-_-0

    yesterday at 6:27 PM

    Affected by what exactly?

• KaiserPro

  yesterday at 9:53 AM

  I mean this is the thing that really comes off hard.

  If you want precision and clarity of your writing, then you need to hand write it. Just like when you are optimising, its common to drop to a lower level language because the compiler doesn't express what you want. Sure its hard, but you know, thats kinda the point.

  Even if you don't want that, the LLM is an average of the style it was trained to give out. Which is a homogenisation of the language to create a vague padding medium between a few generalised facts. (because a. it makes it less jarring when stuff is wrong, because its smeared over a higher amount of text and b. it looks more 'professional' because American business English is all guff and no meat)

  Also yes, two isolated phrases may have subtly different meaning, frankly, the nuance is missed on most people. If you look at the interactions on here, at least 25% of the arguments are caused by people angrily reacting to the things _they_ thought the other person was saying, rather than what the actual person was saying.

  So no its not a perversion, the LLM is, if you're gonna be picky about things.

  • throwthrowuknow

    yesterday at 11:22 AM

    The bigger problem is its effect on the way we use language. The amount of LLM generated or edited media is going to keep increasing and the media that people consume affects their own word choice.

    • KaiserPro

      yesterday at 11:44 AM

      > affects their own word choice.

      exactly. in the same way that printed books affected word choice, so did the radio.

      • Y-bar

        yesterday at 12:09 PM

        I can't remember any radio determining words or adjusting grammar of the person speaking through it. Neither can I recall there has ever been a moveable type press, laser printer, or inkjet which bastardised the words of authors.

        These were mediums _through_ which communication happened. Language models, large or small, are not any such medium.

        • KaiserPro

          yesterday at 1:58 PM

          > Neither can I recall there has ever been a moveable type press,

          Ah my friend, you are about to fall down a rabbit hole into standardisation of spelling, and the sometimes deadly debates about how to translate latin into the vernacular.

          English, as she is written, is a great example.

          for the spoken word, BBC/received pronunciation is another. I speak the way I do _because_ of BBC radio. The reason I have the accent I do is because I changed it to fit what the BBC put out, rather than what my local (impenetrable) dialect was.

          You have to remember that your language is shaped by those around you when you are young. So if you are in an insular community, it will be reflected in your language. If I was a journalist, or hell, just me, I wouldn't be letting an LLM speak for me. So the bastardisation of my voice is down to me, not the machine.

          but again, your argument is against LLMs and globalisation of culture, not finger printing.

          • pjc50

            yesterday at 4:07 PM

            There are two different things being talked about here, one of which is "what effect does widespread LLM use have on a culture", and one of which is "what is the effect on a specific text of running it through an LLM".

          • Y-bar

            yesterday at 4:02 PM

            A radio device at BBC did all this and not humans?

            > again, your argument is against LLMs and globalisation of culture, not finger printing.

            ABSOLUTELY NOT. Do not ever put words in my mouth. My argument is that radio is a medium through which humans communicate. An LLM is not.

            • psjs

              yesterday at 10:50 PM

              [1] > A radio device at BBC did all this and not humans?

              Technology mediates (human) agency.

              [2] > My argument is that radio is a medium through which humans communicate. An LLM is not.

              KaiserPro's argument is that radio (a one-way medium) mediates how humans communicate (phonetically), thus influencing how people speak.

              Now that does not explain how the printing press and radio have influenced word choice (both of which I would like to see examples of!)

              [3] > I can't remember any radio determining words or adjusting grammar of the person speaking through it.

              Now media themselves did not really have something that looked like the kind of pseudo-agency that LLMs seemingly have. There may be some kind of qualitative leap.

            • yesterday at 4:43 PM

            • KaiserPro

              yesterday at 9:27 PM

              I mean I get your point, but sadly LLMs are a tool by which humans communicate.

              In the same way that handwriting conveys more information about the writer than type, typing ones own thoughts conveys more information about the writer than prompting an LLM.

              The analogy here is hiring a speech writer to do your speeches, or dictating to a skilled typist.

              I understand the vociferousness in push back

• mangoman

  yesterday at 12:50 PM

  What are you talking about? I understand OP has the perspective of writers, but let’s say you’re asking an LLM to explain a concept and it uses green words that are actually more difficult to understand. Or you ask for an analogy to explain a topic and the analogy doesn’t quite land because the description used “gray” instead of “overcast”.

  • eloisant

    yesterday at 12:58 PM

    Watermarking or not, LLM are already using RNG to pick a variant between different words/expressions.

• SilasX

  yesterday at 2:04 PM

  OpenAI released a feature that lets you tweak ChatGPT output to sound more human and like yourself and correct inaccuracies:

  http://blog.tyrannyofthemouse.com/2026/04/open-ai-strikes-ba...

• deng

  yesterday at 1:39 PM

  I'm sure he honestly appreciates the pushback.

  • kergonath

    yesterday at 3:32 PM

    I am sure he is quite happy to have people disagree with him on HN. He usually wears it as a badge of pride. We might even have a follow-up in a couple of days about how these techy weirdos lost the plot.

    • jgruber

      yesterday at 6:56 PM

      Indeed.

      • kergonath

        yesterday at 7:20 PM

        :D

        (For the record, I read DF quite often as a Mac-minded techy weirdo, I just happen to disagree on this particular issue)

• firemelt

  yesterday at 6:47 PM

  why you act so tough bro?

• shubhamjain

  yesterday at 8:47 AM

  What an absurd take on a very reasonable concern. As much as I hate “obviously AI” writing, I don’t understand why we have to handicap the tech and prevent it from improving. This manipulation of word choices virtually guarantees AI will always sound like a robot.

  • voidUpdate

    yesterday at 8:49 AM

    Have you compared pre- and post-watermarked text to make that assertion? And I'm not sure how writing your own text if you care deeply about the word choice is handicapping the technology?

  • croon

    yesterday at 8:57 AM

    I don't think there's been a single version of the major LLM providers that haven't handicapped the tech since the beginning by changing temperature/top-P/frequency penalty/etc. All of those stray from "highest probability" token selection. What's funny is it was done specifically to make it sound less like a robot/deterministic.

  • guenthert

    yesterday at 2:40 PM

    > This manipulation of word choices virtually guarantees AI will always sound like a robot.

    You make it sound as if that were a bad thing.

  • mrguyorama

    yesterday at 3:25 PM

    The entire watermarking scheme is based on replacing a random number generator with a seeded random number generator.

    This cannot change the "voice" of the LLM. It was already letting a random number generator choose which adjectives to use. Now that random number generator encodes a tiny signal.

    But fundamentally the way it writes has not changed.

    It's not making it choose different words. It's a minor change to how it chooses between multiple nearly identical words, where in the current case it literally flips a coin.

  • ruszki

    yesterday at 11:47 AM

    Reasonable? A concern which is based on no real data?

  • moooo99

    yesterday at 8:48 AM

    With all things going on among AI bros and the AI industry as a whole, are you really that surprised there is a widespread aversion against the tech?

• doginasuit

  yesterday at 1:05 PM

  Do people not realize this will apply to ALL Claude output, not just writing you ask it to produce?

  > Marks will apply to output from supported Claude models across Claude Platform (API), Claude, Claude Code, Claude Cowork, and Claude Tag, and wherever Claude is offered.

  I've never asked an LLM to generate writing I wish to post as my own. I don't understand why we think it is a good idea to fudge all output just so people can't cheat on their homework or generate slop. It won't have any impact on those things because there will always be models that don't do this. All it will do is increase the rate of false negatives.

  It is deeply misguided regulation and Anthropic should have just said no on grounds of common sense.

  • yesterday at 2:47 PM

• kgarten

  yesterday at 8:29 AM

  One could also use butterflies to write ;) https://xkcd.com/378/

  The problem is, that LLMs worked very well for me to improve my writing. Especially as I'm not a native speaker, it was a great way to improve the legibility of my work.

  I want a tool that helps me improve my writing. A tool I can learn from. Not a tool that switches out "bananas" to "airplanes."

  I'm was using Claude Opus and now Fabel extensively for editing my texts and I find the recent updates abysmal. Not sure if it's due to the Text Watermark.

  Before Claude was great in sharpening the meaning in my writing, it's now close to unusable.

  • voidUpdate

    yesterday at 8:40 AM

    I'll be interested to see if people can actually pick out which text is watermarked and which isn't once they introduce it. It won't switch "bananas" to "airplanes". It'll switch "I really enjoy eating bananas" to "I love eating bananas" or similar

    • kgarten

      yesterday at 12:20 PM

      That’s my point of the post … I had the feeling that the English text editing skills of Claude went significantly down in August.

      I was frustrated at first not knowing what they are doing.

      After I read this post, (seeing they introduced it in August) I think it has to do with the watermarking.

      Try it on a paragraph … the connections between sentences feel clunky now.

      I will play with it more and see if that’s really the case (the watermarking making the text edits worse).

      • kergonath

        yesterday at 3:36 PM

        > After I read this post, (seeing they introduced it in August) I think it has to do with the watermarking.

        The power of confirmation bias…

        > Try it on a paragraph … the connections between sentences feel clunky now.

        We might have a definitive explanation at some point, but there are about a dozen possible reasons for something like this. For starters, is this something really significant and not something you notice because you are looking for it (again, confirmation bias)? A bit like some people still lose their minds when they see a dash, even though statistical analysis showed that they are not a significant marker of AI-generated text?

        • kgarten

          yesterday at 11:11 PM

          I’m frustrated since August with the results from Claude for English text editing … I didn’t look for that. It’s just a fact.

          This is not about em dashes …

      • 0-_-0

        yesterday at 6:56 PM

        You're hallucinating

    • soVeryTired

      yesterday at 9:49 AM

      To that point, given a corpus of writing from person A and another from person B, I have no doubt it’s easy to train a classifier to determine who wrote what. In fact I believe law enforcement agencies already have these classifiers.

      The only difference here is that Anthropic is actively trying to make the watermark undetectable.

      • voidUpdate

        yesterday at 10:03 AM

        They're not trying to make the watermark undetectable, that would defeat the point of a watermark. They're making it detectable, but not make the text obviously watermarked

        • soVeryTired

          yesterday at 10:12 AM

          Undetectable by a human reader. Come on, give the post a charitable reading.

    • yesterday at 9:31 AM

  • nicbou

    yesterday at 8:43 AM

    This is an excellent use case. It made learning German much easier. I write what I think is correct, then get a fixed version.

    When writing in English though, I use it more like a dictionary. If you want to write past a certain level, an LLM works better as a metaphor and idiom search engine.

    I also like to ask it to generate 20 ways to say the same thing. It’s a great way to simplify or smoothen sentences without losing your voice.

    • LIMEVINCE

      yesterday at 9:35 AM

      Interesting, are you being literal about requesting 20 ways of saying the same thing? It seems pretty excessive and I'm actually impressed that asking an LLM to rewrite a statement 20 times yields output that isn't excessively redundant. Does the LLM do a pretty good job reading your mind, or do you still find yourself manually piecing together pieces from the 20 suggestions into a satisfactory sentence?

      • nicbou

        yesterday at 10:53 AM

        Yep, I literally ask it to say it in 20 different ways. Sometimes a sentence structure or a combination of words will just work better. The goal is usually to simplify a sentence without losing meaning.

        I don't expect the LLM to read my mind. The unit of work is too small for intent to matter, and I'll just steer the next recommendations in a direction as needed.

        Most of the suggestions are crap, but they can contain the seeds of a good sentence.

        • jorisw

          yesterday at 3:11 PM

          And this beats just writing the piece yourself?

          • kergonath

            yesterday at 3:39 PM

            I can’t speak for them, but I do this occasionally. Every now and then among the propositions there are one or two that did not come to my mind and that are actually quite good. I still write the whole thing myself, I just ask for advice occasionally.

          • nicbou

            yesterday at 6:20 PM

            How did you understand this as not writing the piece myself?

    • camillomiller

      yesterday at 9:32 AM

      As a writer, Claude’s metaphors are trite and obvious 90% of the time. It also has a terrible penchant for an immediately recognizable emphatic voice that makes even the best outputs super cringe. But people do not notice and do not care. In our bubble and John Gruber’s bubble we really overestimate how much people must care. Truth is, we’re in this predicament because statistically speaking the people who hive a fuck are a rounding error.

      • nicbou

        yesterday at 10:57 AM

        They really are. This is why I prefer the volume approach. I might not accept any of the ideas it spits out, but it often guides me in a direction I was not considering.

        I know that most people don't care, but my online presence is a search query for interesting people, so I care about what I put into it.

  • georgemcbay

    yesterday at 8:58 AM

    > improve the legibility of my work.

    Nitpicking here in a way that I would usually avoid, but it is relevant to the conversation being had and it seems like you might appreciate the information... "Readability" would be the more correct word to use here instead of "legibility".

    Legibility is close enough for me to know what you mean based on the context, but it really applies to the visual presentation and how easy something is to read at a symbolic level (whether someone's handwriting or font choice is good or bad impacts legibility, whether someone uses good grammar or not impacts readability).

  • zahlman

    yesterday at 8:43 AM

    > One could also use butterflies to write ;) https://xkcd.com/378/

    This comparison is frankly absurd.

• allisdust

  yesterday at 8:24 AM

  LLMs are no more than pen and paper at this point. Especially for those who aren't trying to create slop. We all would want our pens to accurately reflect the strokes (well in this case thoughts) rather than adding tiny watermarks to identify that it is generated by a particular pen or a user.

  Watermarking per model is just the start. The method is cheap enough to distinguish individual users.

  • ehnto

    yesterday at 8:42 AM

    That is an insane statement, LLMs generate swaths of text from almost nothing.

    If they are adding so little value as to be as transparent as a pen and paper then why use one at all? Transcription doesn't need an LLM so that's not what you're taking about I assume.

  • voidUpdate

    yesterday at 8:27 AM

    LLMs do a whole lot more than writing your thoughts down. They write extra text. If you just want a pen and paper, use Notepad. Or better, a pen and paper

  • a2ff6eeb0

    yesterday at 12:16 PM

    What? The entire reason I use an LLM is to be able to avoid thinking about a topic.

    That's their whole damn value prop: outsourcing thinking and producing without understanding.

    I don't need to read emails in detail to respond any more.

  • Arodex

    yesterday at 8:31 AM

    >LLMs are no more than pen and paper at this point.

    Then use pen and paper. It is the same, you say, right?

    • yesterday at 10:57 AM

Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).

• wpietri

  last Sunday at 11:37 PM

  It seems to me like he started out mad and looked to justify it.

  I'm skeptical that anybody generating LLM text is really all that concerned about optimal word choice. Or even particularly good prose. But let's pretend that person exists.

  If that person tried, say, an open model and that same model with watermarking applied, I'd be eager to hear their thoughts on the prose quality. Especially if they built an experiment harness and rated a few hundred blinded examples and found a measurable difference.

  But getting this upset in advance of any demonstrated problem? It really seems to me like the point isn't the point

  • throwawayffffas

    today at 10:56 AM

    I care about optimal word choice when generating LLM texts. Because my use case is almost exclusively reading the generated text not posting it. I use LLMs to summarize, translate and review other texts. When using LLMs in that way, as a research tool watermarking is a pointless and should not get in the way of "optimal" results.

  • beering

    last Sunday at 11:49 PM

    Google has A/B tested watermarking on millions of responses. They say they observed no difference in user behavior.

    • gwd

      yesterday at 9:13 AM

      It says they observed no difference in people clicking thumbs up or down. There are loads of other behavior that they didn't observe; like, say, switching to a different LLM.

      • Fogest

        yesterday at 7:52 PM

        Agreed, if I simply didn't like the style or words an AI was using in something it wrote, I would switch to a competitors and see what it can come with. I probably wouldn't hit the thumbs down on the Gemini response as it's not that the response is wrong, I just didn't like it. I usually reserve the thumb down for when the AI is wrong.

        Also, depending on what I am asking it, I often don't want to use the thumb down or up, as this may mean my conversation is going to have some kind of human review and depending on what I am asking for, I may not want to bring attention to my stuff.

      • iainmerrick

        yesterday at 9:31 AM

        How exactly do you propose they should keep track of quality, then, if not by A/B testing?

        • gwd

          yesterday at 10:59 AM

          The question here is not to what product management advice to the Gemini team. The discussion here is whether the watermarking is noticeable.

          The easy thing to do here would be to have 1000 questions, randomly assigning one half to an LLM with a watermark, and the other half without. Then show people pairs and say, "Which one seems watermarked?" (Or, "Which text seems more natural" or "Which is a better answer" or something like that.) If they come out equal, the watermark really is indiscernible, at least to most people.

          • iainmerrick

            yesterday at 2:14 PM

            They do exactly that "which is the better answer?" test -- I've seen it pop up a few times.

            • summarybot

              yesterday at 3:08 PM

              Isn't "which one is watermarked?" a different question than "which one is better?"

              "Which diamonds are shinier, the blood diamond sourced ones or the ethically sourced ones?" ... that's not the same question as "which diamonds are blood diamonds" (to employ an extreme analogy)

              Concluding that no one could detect which ones were blood diamonds because they were "equally shiny" is not really correct now, is it?

              • iainmerrick

                yesterday at 3:38 PM

                That's true, but you don't typically explain what you're testing in this sort of (presumably) randomised trial.

                And the Daring Fireball article does complain that watermarking will reduce quality. If that's what you're trying to check, "which is better?" is the right question.

  • boredhedgehog

    yesterday at 8:35 AM

    > It seems to me like he started out mad and looked to justify it.

    Yes, but that's neither surprising nor a reason to dismiss the anger. People get angry about DRM schemes in video games, even if the slowdown these cause is practically imperceptible. They're angry -- and Gruber acknowledges that factor too -- because a stranger manipulates what they regard as their own domain, without consent by or benefit to the owner.

    It might be another instance of consequentialism vs. honor ethics. Many consequentialists don't seem to understand that something that doesn't have demonstrable consequences can still have moral implications.

    • KaiserPro

      yesterday at 9:57 AM

      > People get angry about DRM schemes

      Yes but thats a thing that degrades something in a catastrophic way, as in I can use the thing one day, and not the next.

      A different randomisation system on something that is a text generator which is designed to be unperceptable sounds like the people who are annoyed at FLAC vs MP3[1]

      Done right you won't know the difference, done badly and you will.

      [1] ex audio engineer, try me.

      • latexr

        yesterday at 12:38 PM

        > the people who are annoyed at FLAC vs MP3

        What’s the story there? I didn’t know that was a thing and I’m curious to learn more.

        • Sharlin

          yesterday at 12:40 PM

          MP3 is lossy. FLAC is lossless. So obviously a certain type of people are going to make a religious war out of it.

          • KaiserPro

            yesterday at 1:51 PM

            Essentially yes.

            Flac takes a raw .wav and effectively zips it up to shave off a certain amount of space. (there are nuances, I think the compression scheme is designed for streaming.)

            mp3 is perceptual, so throws away the stuff that humans can't hear. This yields a much smaller file.

            However its all a sliding scale like PNG vs jpeg.

            a .jpg with a quality setting of 85 will be almost identical to a .png in visual quality. However if you then edit that jpeg, the image degrades and you start to see artifacts. (hence why memes look like shite as they get older)

            Its the same with mp3s if you compress the hell out of them, say 64kbit or lower adaptive, then you'll start to hear the tell tail "schlop" noise of mp3-like compression. You might notice it most with cymbals in drum kits. cymbals are wideband noise. as in there are loads of constituent frequencies so if you remove some of the "hidden" frequencies you tend to notice, so they sound more metallic, ironically.

            But, all of this is solvable, 256+kbit is more than enough, bonus points for higher sampling frequencies. (however you need a decoder that can actually do that sample rate...)

            • Tepix

              yesterday at 4:21 PM

              Interestingly, in a test c't magazine did a long time ago, the test audience preferred mp3 256kbit over the original.

              • milesvp

                yesterday at 4:58 PM

                Could be c't magazine accidentally played the mp3 version louder. Human's have a known preference for louder music, and will tend to prefer louder samples over quieter samples. Rumor in the industry is that this was a trick MS used to try to push the WMA format, that they encoded some WMA samples used in some publicized tests at +3dB above the source sample.

      • porridgeraisin

        yesterday at 12:17 PM

        > try me

        Good luck explaining that one

    • zahlman

      yesterday at 8:48 AM

      > People get angry about DRM schemes in video games, even if the slowdown these cause is practically imperceptible.

      Any potential "slowdown" doesn't even come close to making the list of top reasons people get upset about DRM.

    • wpietri

      yesterday at 3:42 PM

      I get what you're saying, but I think it's ridiculous for people to think of LLM services generating text as either their own domain or something that they own.

      To the extent that it's anybody's, it's either Anthropic's (they run the service) or everybody's (in that we created the content it's remixing). Legally LLM prose isn't copyrightable for good reason.

    • pjc50

      yesterday at 9:39 AM

      Even if honor were real, LLMs do not have honor, and people using LLMs to write without disclosing that fact (or indeed at all, to some purists) do not have honor either.

      > because a stranger manipulates what they regard as their own domain, without consent by or benefit to the owner.

      It's LLM output! It's not your domain, it's the LLM owner's!

  • weego

    yesterday at 8:39 AM

    > It seems to me like he started out mad and looked to justify it.

    that's been his thing since it was just a blog about apple product speculation and update. It's always been tedious.

  • robomc

    yesterday at 1:06 AM

    I assume they are just passing off AI prose as their own and don't want anyone to be able to tell. Which is surprising for someone who's been blogging for a thousand years. But I don't really see any other reason for this amount of heat and FUD.

  • Kwpolska

    yesterday at 8:56 AM

    Considering Gruber's always comically butthurt about regulation, especially EU regulation, your theory seems accurate.

• krackers

  yesterday at 7:03 AM

  >fails to understand the gumbel softmax technique

  I think in this case it doesn't help that there are multiple watermarking schemes, and the easiest for people to understand is the red/green scheme by Kirchenbauer et al. (https://arxiv.org/pdf/2301.10226), which does technically distort the logits (but I'd argue only in cases where you wouldn't notice it anyway).

  I wasn't aware of this gumbel softmax scheme, it seems you're referring to https://simons.berkeley.edu/talks/scott-aaronson-ut-austin-o... ? That's really clever as it doesn't even distort the logits, basically cryptographically indistinguishable from a "real" random sample unless you have the key.

  The actual scheme Claude uses seems to be neither of those two though, they say it is SynthId-text which seems to be tournament sampling based.

• neuroticnews25

  yesterday at 9:55 AM

  >It does not affect writing quality at all, provably

  Prove it, then? It's not a claim that GumbelSoft paper makes: "Regarding generation quality (perplexity), GumbelSoft shows relatively low perplexity"

  https://arxiv.org/html/2402.12948v3

  • yorwba

    yesterday at 4:26 PM

    They do claim that the per-token output distribution remains unchanged, but the proof is relegated to Appendix B.1. The perplexity comparison includes methods that do change the output distribution.

• Art9681

  last Sunday at 11:35 PM

  If this is true then the probability of the detection tools flagging completely human generated text as AI generated is non-trivial. Let's say I write a completely original piece and the detection tool says there is a 36% probability it was generated with Claude. What then? Now it's up to the person looking at the score to cast a subjective judgement. Maybe to me, anything over 25% is unacceptable. Maybe to someone else, it must cross over the 50% threshold. This is the problem.

  Cognitive surrender.

  • fwipsy

    last Sunday at 11:56 PM

    > the probability of the detection tools flagging completely human generated text as AI generated is non-trivial

    How does that follow? AI-generated text is already not a perfect emulation of human writing. There's lots of room to affect it laterally without changing the level of quality.

    As I understand it, LLMs with temperature >0 can select from many possible outputs. All they're doing is limiting the possible outputs to ones that contain this pattern. I don't see any reason why the quality of that subset should be lower than average. The very best outputs will likely be eliminated, but so will the very worst.

    • eru

      yesterday at 8:18 AM

      To sample from the probability distribution you already need random numbers.

      If you get your random numbers from a cryptographic PRNG, then to notice the difference between that and 'real' random numbers even in theory, means you need to break the cryptography. In practice, your gut feeling about how good some text is won't break modern cryptography.

  • wasabi991011

    yesterday at 1:18 AM

    That's an interesting problem to discuss, but unfortunately TFA spends no time discussing that.

    • beering

      yesterday at 3:06 AM

      Yeah. And the detector can’t even tell you “X% chance this is watermarked” because it doesn’t know the input distribution. It can only tell you “Y% chance that an unwatermakred text would score this high” and how many history professors understand Bayes rule well enough to understand the distinction?

  • pizzly

    yesterday at 12:44 AM

    Worse, what will academic institutions decide is the threshold for detecting AI generated work. If you have a false positive how do you prove it was a false positive or we all just trust the watermark detector over the student saying "I swear I did it all by my self"

  • pessimizer

    last Sunday at 11:52 PM

    I don't think that's true. I think it's a binary 0% or near 100% probability of a watermark having been detected; the more changes to the text having been made after the text was output by the LLM and the less leeway the LLM had for probable word choices, the longer the passage necessary to see it.

    The "problem" is that seeing the watermark doesn't mean that the person claiming to be the author didn't make extensive changes to the output of the LLM, or that the LLM wasn't simply the final editor of something that the author had put a lot of work into.

    > Cognitive surrender.

    I don't know what this means. It's just drama. Don't let the LLM write for you and this is not a worry. I'm not worried about the poetry of LLM output being subtly adulterated.

    • beering

      yesterday at 12:50 AM

      No, watermark detection is not binary, you get a real number. You decide on a threshold when looking for the watermark. This is the problem - by random chance, some human text will be detected as watermarked. You can turn the detection threshold up until it guarantees <0.001 false positive rate at the expense of higher false negatives, but seems inevitable that someone gets wrongly flagged.

• urams

  yesterday at 6:18 PM

  Gruber is always happy to lead with his emotions and backfill justifications for them. See his recent debacle with App Store review: https://daringfireball.net/2026/08/retraction_app_store_reje...

• adamgordonbell

  yesterday at 5:09 PM

  It's a writer perspective versus a reader perspective maybe?

  Sometimes when you're trying to write something, it really seems like the exact words matter a lot. Suggestions made to be more direct or use a more common word here or whatever seem to really impact the thought that you're trying to communicate.

  Certainly we've all had times when trying to communicate clearly when the specific words seem very important.

• sva_

  yesterday at 7:13 PM

  > The very fact that there is generally no "best next token" with 100% certainty

  This is not entirely accurate. Sure, there's never a token with 100% certainty, but there are often tokens with 99.9% probability, but this technique of course does not change how such a token is sampled.

• tapland

  last Sunday at 11:20 PM

  Making blog posts about AI that make it apparent that the tech is going whoosh is a choice.

• CuriouslyC

  yesterday at 12:52 PM

  By definition watermarking narrows and biases the response distribution. Clever algorithms might reduce the perceptual impact and minimize some cherry picked metrics, but it's still worse.

• zahlman

  yesterday at 8:47 AM

  > The very fact that there is generally no "best next token" with 100% certainty

  Indeed.

  It's frankly bizarre to see the assumption to the contrary being made by someone who's been passionately blogging by hand for years, who also happens to be responsible for the notoriously vague, humanistic, DWIMmy Markdown standard.

• reader9274

  last Sunday at 11:13 PM

  "Smart"? Have you read his writings in the last decade? It's all nonsense, which is why I stopped reading circa 2018

  • brookst

    last Sunday at 11:28 PM

    I think he’s still generally good on business, UX, and hardware design. That’s all subjective and taste I suppose, but his taste works for me.

    On deeper tech stuff, like this utterly nonsensical misunderstanding of watermarks… yeah, classic case of a guy who is smart, and has lost the ability to realize when they’re not knowledgeable in a domain.

• docjay

  last Sunday at 11:26 PM

  [dead]

• selectively

  last Sunday at 11:22 PM

  [dead]

• conartist6

  last Sunday at 11:40 PM

  I couldn't be happier that people are mad about it. To quote Calvin, "nothing helps a bad mood like spreading it around"

• ragazzina

  yesterday at 6:18 PM

  > a smart person like this

  Can you give an example of something smart John Gruber has said or written? Because I can't think of one, but I can think of many dumb ones.

  • jgruber

    yesterday at 7:02 PM

    Agreed!

> “By definition it must make text worse … because the nature of the watermarking algorithm requires it to sometimes increase the probability of selecting a worse word choice and decrease the probability of selecting the model’s best choice.”

Gruber made an effort to but doesn't fully understand how SynthID works. LLMs select the next word randomly from a set probability distribution, so there is no "best choice" unless you run the LLM with a temperature of 0 which would give out terrible results. Anthropic runs a non-distorting version of SynthID that doesn't change the probabilities of the underlying distribution of tokens. It makes the watermark less likely to work over smaller samples but preserves text quality. I encourage the mathematically inclined to read the paper:

https://www.nature.com/articles/s41586-024-08025-4

• Marsymars

  yesterday at 9:19 PM

  > LLMs select the next word randomly from a set probability distribution, so there is no "best choice" unless you run the LLM with a temperature of 0 which would give out terrible results.

  I'm not understanding how the word with the highest probability isn't the "best choice"?

  • baobabKoodaa

    yesterday at 9:42 PM

    You will understand if you run some LLM models with a greedy sampler that does that. The text quality begins to deteriorate rapidly. This is a very counter intuitive result so I don't blame you for not understanding until you actually tried it and experienced it for yourself.

    • Marsymars

      yesterday at 9:53 PM

      > You will understand if you run some LLM models with a greedy sampler that does that. The text quality begins to deteriorate rapidly.

      Right, I've done this, and this makes sense to me, but I'm not following how that falsifies the top probability word being the best choice in any particular instance.

      "Picking only the best word at each decision point results in a worse final result" seems like an imminently reasonable hypothesis.

      • StevenWaterman

        yesterday at 11:09 PM

        I think you're using different definitions of best. If best = leads to a correct answer overall then by definition anything that leads to a bad outcome can't be best

• tarvaina

  yesterday at 8:05 PM

  I came here to quote the same sentence. Here's another way to look at it:

  Suppose there actually is a best word choice. The LLM doesn't know what it is but makes a guess. Maybe it's the best one, maybe it isn't. The probability that SynthID changes the best choice to a worse one is equal to the probability that it changes a worse choice to the best one.

  • abustamam

    yesterday at 9:05 PM

    I think that depends on the distribution of good choices and bad ones. There may be 10 choices and maybe 8 of them could be appropriate given a context, and 2 are absolutely nonsensical. Or it could be vice versa. And its a spectrum as well.

    • tempestn

      yesterday at 9:56 PM

      But the choices are weighted based on those probabilities. This doesn't affect the weightings, only how the final weighted pseudo-random selection is made.

      • abustamam

        today at 2:47 AM

        Right, but that's what makes me doubt this statement in the comment I was replying to

        > The probability that SynthID changes the best choice to a worse one is equal to the probability that it changes a worse choice to the best one.

Gruber shows here that he really doesn’t understand the basics of how LLM text generation works. It’s weird he picked this battle about the quality of writing in LLMs. Was he planning to use LLMs to write his articles?

Well, not that weird actually. He just has a hard-on against anything that comes from the EU since Apple got in trouble. If the EU said tomorrow that they want peace in the world he’d be in Fox News the next day calling for an invasion. As a former reader of Daring Fireball, it’s just sad to see.

• redfloatplane

  yesterday at 10:07 AM

  Yes, I decided to stop reading his blog relatively recently after some extremely hot takes on EU policy. I don't feel his thoughts on the matter are particularly well-thought-out, and I feel like he's just stanning for Apple from his priors rather than from any grounding in reality.

  I dunno, I guess that's what you should expect from Gruber but these EU-bashing articles lowered the enjoyment I got from his blog underneath the bar for me.

  • solid_fuel

    yesterday at 4:23 PM

    Freaking out about basic EU privacy and safety measures is always one of the biggest red flags that someone is becoming a politically motivated hack, especially when that someone lives in the US where there is an active fascist movement tearing apart the government.

    • redfloatplane

      yesterday at 8:34 PM

      But what if those measures might apply to your favourite multi-trillion-dollar globo-corp and cause them to nominally change their business practices? That's a step too far, I say.

      • solid_fuel

        today at 2:52 AM

        Oh, well then it must be stopped by any means necessary of course! Thank god there are brave and concerned citizens like John Gruber, ready to hold the line against the tyranny of regulations.

        After all, if it wasn't for him who would ever speak up for the trillion dollar corporation? Without brave heros like him, these poor vulnerable corporations would be facing all sorts of attacks from evil regulatory organizations and their dastardly scientists. Just think of the profit - all those euros - that may be lost like chaff in the wind.

  • jorisw

    yesterday at 1:37 PM

    As an EU citizen I've found myself in agreement with everything he's written regarding EU policy

    • redfloatplane

      yesterday at 3:22 PM

      Good for you! As an EU citizen I have found myself in almost total disagreement with everything he's written regarding EU policy. Horses for courses.

• llm_nerd

  yesterday at 10:34 AM

  Gruber went from the naively wrong claim that it would insert secret hidden characters (which would be trivial to remove, obviously), to quickly writing a giant essay as if he's an expert on LLMs. Like you said, he is strangely fixated on the EU, and is certain any EU rule is the worst thing in the universe, and this whole piece seems motivated by that guiding force.

  Further he later compares Gemini to Anthropic models, saying the latter "writes better", emptily ascribing this to the synthid stuff. I think he heard that Anthropic currently has superior models, but it certainly isn't because they "write better", and if anything Opus 5 now is virtually unintelligible, before the fingerprinting.

  The fingerprinting stuff sounds weird. If the EU wants it, it should be limited to the EU, and Anthropic is fully capable of doing that but clearly saw value in recognizing their own output. Is it going to destroy the quality of the output? We'll have to see, and this anti-EU piece, predicated on utter ignorance of the field, is not convincing.

  • ragazzina

    yesterday at 6:20 PM

    How is he "strangely" fixated on EU? The EU tries to limit Apple's power, Gruber shills for Apple, Gruber is against the EU. Nothing strange about it.

    • llm_nerd

      yesterday at 11:33 PM

      I completely agree with you. But that doesn't change that Gruber's fixation on the EU is strange, specifically because he's so ego-tied to Apple that he strikes out like a untrained chihuahua at his list of Apple adversaries. He has done this for many years and it's just weird.

      It's like some weird k-pop stan sending death threats to someone that dissed their favourite singer. Just super strange stuff.

>I want any LLM I use to choose the very best, most precise words at every single decision point.

Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing?

This entire article just seems so detached from the basics of how LLMs work.

• dofm

  last Sunday at 11:34 PM

  > Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing?

  No and no. I am not sure I agree with his point but I know he is not ill-informed on either of these points, because I mentioned them to him a couple of days ago.

  • brookst

    last Sunday at 11:36 PM

    It didn’t take, apparently.

    • dofm

      last Sunday at 11:41 PM

      It’s fully possible I didn’t explain it very well in the first place, but he is making a wider point.

      The point I made (quite briefly) is that watermarking is only feasible because for good writing it is necessary to use T>0, or the writing will never explore a more creative choice, and that at T=0 you don’t even need a watermark to spot LLM-generated text.

      The point he is making is consistent with this, isn’t it? Either you allow temperature to drive creativity, consistently in a way that can be influenced and analysed, or you adulterate that process for the purposes of meeting a corporate/legal directive, in a way that is proprietary and obscure. These are ethically distinct approaches, and since he disagrees with the EU objective he comes down on one side I guess.

      Me, I don’t care about the hypothetical enough.

      Not least because I think Claude writes depressingly badly and I doubt any steganographic change will enrage me less.

      • wasabi991011

        yesterday at 1:24 AM

        The point he is making is not consistent with understanding how temperature influences LLM text generation, no.

        He repeatedly states that choosing "the best word" is the most important thing to him. I don't know how you reconcile that with creativity itself, let alone probabilistic sampling.

        • eru

          yesterday at 8:27 AM

          Yes, naively understood in his sense 'best' word means that you pick the word with the maximum score, instead of sampling from the distribution.

          That doesn't actually give you the 'best' text in any human sense of the word. Just like playing the 'best' move in Poker without sampling leads you to lose a lot of money.

        • dofm

          yesterday at 1:31 AM

          I mean creativity there in the LLM sense (temperature driving more creative solutions), not the human sense, and in the context of its consistency, configurability and being amenable to analysis. The watermarking approach makes that non-reproducible, yes? Because Anthropic can and will change it as they see fit.

          • baobabKoodaa

            yesterday at 9:52 PM

            Your response seems to be missing the point completely. Gruber thinks "best" writing is produced by choosing the "best" word (highest scoring token) at each step. This is very clear from his writing.

      • beering

        yesterday at 12:56 AM

        No, you are jumping to conclusions about how watermarking works. This is some audiophile thinking that because your RNG is “pure”, you get text with an expansive soundstage or whatever. Intuitively this may be true or false depending on your personal prior but you’d need to show it mathematically. The overall token distribution shouldn’t change and the frequency at which you see the word “load-bearing” will remain the same.

        • dofm

          yesterday at 1:09 AM

          > This is some audiophile thinking that because your RNG is “pure”, you get text with an expansive soundstage or whatever.

          You are projecting that onto me, and I cannot tell you how comically poorly aimed it is.

      • Imnimo

        yesterday at 3:02 AM

        >Either you allow temperature to drive creativity, consistently in a way that can be influenced and analysed, or you adulterate that process for the purposes of meeting a corporate/legal directive, in a way that is proprietary and obscure

        This framing does not make sense to me. What do you mean by "influenced and analysed"? How have you or anyone been influencing or analyzing the randomness behind the sampling process to create better writing? What makes the unadulterated randomness "driving creativity" but a different random choice uncreative?

        • dofm

          yesterday at 11:39 AM

          > How have you or anyone been influencing or analyzing the randomness behind the sampling process to create better writing?

          You're mischaracterising or misunderstanding my point, or I mangled it.

          I mean it is possible to analyse, control, monitor, study the impact of changing temperature on the writing, yes?

          The point about watermarking is that this relationship — change the temperature, see the effect — is now being adjusted by an unstated, secret process you explicitly can't control.

          (I gather Anthropic have recently taken away this setting anyway; that was news to me.)

          • Imnimo

            yesterday at 3:25 PM

            I don't think watermarking breaks this relationship. Watermarked text is still being sampled from the model's output distribution, and adjusting the temperature still has the same affect on that output distribution.

            I think a good intuition here is that watermarking is sort of like picking a specific PRNG seed. It's not changing or interfering with the temperature - we're still sampling from the model's probability distribution. But we're making it so the analog of the PRNG seed is coupled to the previous context.

      • solid_fuel

        yesterday at 4:29 PM

        He’s not making a wider point, he’s crashing out because the EU is involved. I don’t really think it is any more complicated than that - there are no technical merits to the criticism.

      • inigyou

        yesterday at 12:39 AM

        What does he think of all the other adulterations of LLMs that already happen?

• sfink

  yesterday at 6:28 PM

  This is a reductionist counterargument. Sure, the passage you quoted does sound like he's being equally reductionist. But the underlying point does not depend on T=0. You could state it as saying that instead of minimizing error (maximizing "writing quality"), you're using some of that error for watermarking and minimizing the rest.

  Describing it in terms of a word-by-word choice is simpler, but writing quality is dependent on the interplay between words.

  "The weather today was cold and {grey,overcast}." If the next sentence is "I miss yesterday, when it was {bright,sunny}." then the choice between "grey" and "overcast" is no longer neutral. "grey" and "bright" pair together, as do "overcast" and "sunny". Or if you disagree with my aesthetic sensibilities, consider:

      The weather today was cold and {grey,gray}. The {color,colour} of the sky matched my {humorless,humourless} mood.

• kccqzy

  yesterday at 1:57 PM

  I think it is a common misconception for anyone who hasn’t actually tried implementing a LLM to think that there is a best choice of token at each step and that following every locally best choice will lead to a globally “best” writing. This is intuitive yet wrong and perhaps there is no better way to rid oneself of this misconception other than actually implementing a simple LLM.

• Gigachad

  last Sunday at 11:30 PM

  I think the author is just mad people will be able to detect and filter out their AI slop writing in the future.

  • kalleboo

    yesterday at 3:45 AM

    Gruber just hates any kind of EU regulation of US tech companies ever since they started making what he calls "product decisions" for Apple.

    • jorisw

      yesterday at 1:43 PM

      As an EU citizen and user of Apple products, I feel the same

      • kalleboo

        yesterday at 5:08 PM

        We're all entitled to our opinions, but then just say that, don't go to great lengths to misunderstand and justify technology that you don't even use yourself to justify why the regulation is bad, just say "I think regulation is fundamentally bad".

        • jorisw

          today at 6:23 AM

          I've never seen Gruber say that

  • dofm

    last Sunday at 11:36 PM

    This is not it, no. He is not using AI and it is not I think remotely in his nature to surrender that control. He is engaging with this on principle. Again I am not sure I agree with him, but then it’s a hypothetical because I am not going to get an LLM to write for me either.

    • beering

      yesterday at 1:05 AM

      Well, it cant be that he is super worried on behalf of people who publish AI slop. That’s not a credible motivation. In fact, he complained a lot about the new ChatGPT app so I can’t believe your claim that he is not using AI.

      Seems like he really likes to use LLMs and is worried that quality will be degraded. But he will never demonstrate such degradation scientifically, we don’t have anecdotes even.

      • dofm

        yesterday at 1:26 AM

        His complaint about the ChatGPT app is that it’s a shitty non-Mac-ish Mac app. Complaining about shitty non-Mac-ish Mac apps to people who hate shitty non-Mac-ish Mac apps is more or less how he became a full time writer.

    • NitpickLawyer

      yesterday at 10:04 AM

      > He is not using AI

      That's ... even worse? So we're all here in the comments trying to figure out what the author means, and what their overall point is, while clearly they don't even use the damn thing? Oof... What a waste of time for everyone involved.

      • dofm

        yesterday at 11:09 AM

        Why? I really don't understand this. Why can't a tech writer take a deep but neutral intellectual interest in something? Isn't it important that some do? Do people have to be stakeholders or clearly on one given team, pro- or anti-, for their opinion to matter? Is it that tribal?

        It seems fully logical to me that someone who writes for a living (who, as it happens, developed the very markup language LLMs use for everything) should be invested in understanding the automatic plagiarism and word calculating machine from an intellectually honest position.

        I personally am pretty severely big-two-AI-firms, increasingly anti-big-tech, but I am learning and researching uses of LLMs because for myself I really need to understand how to use them in an intellectually and (as far as is possible) ethically sound way. Learning because as a boring old freelance programmer I have to; foolish to pretend otherwise.

        So I completely understand his position — that the AI industry is hot air and crooked and scammy and weird, and some of the people involved genuinely rather dark-sided, but the technology exists and if it hints at threatening your livelihood, you need to understand it.

        From reading his work for the best part of twenty years or so (and emailing him intermittently over that time) it would seem to me that he's a lot less bearish on the tech industry than me, and a lot less fond of the EU than I am; he's more optimistic than I am. But he writes because he has to write. I should think that would make him highly invested in understanding what LLMs do.

        • NitpickLawyer

          yesterday at 1:22 PM

          > Why? I really don't understand this. Why can't a tech writer take a deep but neutral intellectual interest in something? Isn't it important that some do? Do people have to be stakeholders or clearly on one given team, pro- or anti-, for their opinion to matter? Is it that tribal?

          I don't think it's weird to ask that people commenting on x doing y with tech z at least try tech z, no?

          Imagine this pamphlet: Basting a steak in a cast iron skillet with butter and herbs is a perversion of grilling a steak on charcoal. Signed, a life-time vegan who hasn't cooked a steak in their life.

          Then imagine people jumping in the comments to discuss. Isn't it a waste of time?

          • dofm

            yesterday at 1:43 PM

            > I don't think it's weird to ask that people commenting on x doing y with tech z at least try tech z, no?

            On what specific basis do you assume he hasn't tried it? He's definitely blogged about the desktop apps, after all.

            Or are you arguing that a writer doesn't have a meaningful or valid opinion on LLM-generated writing until they have tried to pass some off as their own?

            This just seems weird to me. I mean, I have an opinion on this and I am personally never going to use an LLM to do published writing. On an intellectual level I can still see that there is nuance in it for others (for once I agree with him about an EU regulation).

            • NitpickLawyer

              yesterday at 2:09 PM

              I'm confused. You said "he doesn't use AI" and I took that as a general "he never used AI". If I was mistaken then ignore this whole thread, that's my bad.

              • dofm

                yesterday at 5:40 PM

                Ahh — that was in the context of a suggestion AI-slop-writing I was replying to (quite an accusation for an established blogger IMO).

                But one of the issues with HN threads is that you can sometimes lose the sense of what you're replying to by clicking further down the thread, and I have committed worse misunderstandings than this, so absolutely no need to apologise (and I probably need to consider this when I am replying) :-)

        • breezybottom

          yesterday at 12:57 PM

          If he writes then he should have no investment in LLMs. Humans have been able to write for thousands of years.

          • dofm

            yesterday at 5:42 PM

            You've somewhat mischaracterised what I have said.

            I suggested he as a tech writer has reason to be invested in understanding how they work. It's not really a sustainable position to not understand, is it?

  • jorisw

    yesterday at 1:42 PM

    I don't think you know the author very well

  • llm_nerd

    yesterday at 10:37 AM

    Gruber has long been a talented, excellent writer. I hugely doubt he uses AI at all, nor does he plan to.

    His first take on this situation was cutely naive, thinking they were going to inject secret hidden unicode characters. But ultimately he has a massive hate on for the EU -- they were mean to Apple once -- and it comes out in any topic that overlaps.

    • Marsymars

      yesterday at 9:31 PM

      > Gruber has long been a talented, excellent writer. I hugely doubt he uses AI at all, nor does he plan to.

      He notes in various other posts that he uses AI/LLMs and chatbots quite extensively. (I don't recall what for exactly, but not for writing his pieces.)

      • llm_nerd

        yesterday at 11:00 PM

        Good point, and I should have been clearer that I don't think he uses or will use it for writing. He's far too skilled of a writer to need it, and at best it would be a handicap.

  • cryptonector

    yesterday at 4:57 AM

    People already are, and do.

> “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter. I want any LLM I use to choose the very best, most precise words at every single decision point.

Neither of these is "better" or "more precise"; in fact, LLMs will generally choose randomly between these candidates based on temperature, and SynthID should not distort the output of an LLM any more than the default temperature settings do already.

I agree that those are not the same sentences, but if the difference matters to you, you shouldn't be using an LLM. This difference exists at the level of what sounds better and is more evocative; to an LLM, nothing sounds like or evokes anything. They simply do not write good prose.

• slowin

  yesterday at 2:20 PM

  I’m more concerned that this will negatively impact code generation. An additional constraint completely unrelated to code quality is unacceptable as far as I’m concerned. I was an Anthropic user but now I’m looking at OpenAI or even better, open models.

  • bccdee

    today at 1:05 PM

    Unless you're already manually setting temperature to zero on the models you work with, the deviation from the "optimal" path imposed by the watermark should be no greater than what you already get from the randomization.

  • jweber123

    yesterday at 9:46 PM

    According to their article, these kinds of arbitrary choices don’t come up as often with code, so it’s less likely to have watermarks:

    https://www.anthropic.com/news/claude-text-watermark#:~:text...

  • skort

    yesterday at 2:28 PM

    If you're concerned about code generation, then learn how to actually write the damn code yourself!

What a truly bizarre article. Arguments about pre-existing randomness, temperature and whatnot aside, I simply cannot comprehend what the author here really thinks the "best word" is. There's no such thing. We humans fall on familiar patterns of writing ourselves, so we may forego something with a flourish in favor of a more commonly-used word unless we put in effort to be "special", which should be used sparingly. That is to say, human writers are likely to choose a "worse" word in far more than the supposed 51% of cases, and that has no effect on the actual quality of writing in the end.

But even if there were such a thing as a truly "best word", for some context, what are the examples here? Mango vs pineapple? Gray vs overcast? In what case is one of these better, that AI would normally infer but would suddenly be "perverted" by SynthID? Do you think your emotional state and preferences are being evaluated if they aren't explicitly in memory? And if they are there, do you think that the generator will bypass those instructions in favor of the watermark instead of placing it somewhere you won't care? I just. Genuinely don't get it. There may be words that matter in specific contexts or to you as a reader, so you should bloody well put them there.

Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude.

I think that was intended, yes.

• epihelix

  last Sunday at 11:16 PM

  You know, back in the era when proofreaders were human, I never one met a proofreader who rewrote my text afresh, rather than annotating the text with a pen.

  It's still possible to use Claude to proofread - highlight grammatical, flow, structure, logic errors and make simple suggestions for you to pick and choose or adapt as you wish. No watermarking will flag your text. No flaw accusations of LLM authorship will haunt you. All will be fine.

  But if you want an LLM to rewrite your text, that's (a) not proofreading, and (b) should be flagged as LLM generated ... because it is.

  • joshkel

    yesterday at 1:42 PM

    Anthropic's own explanation (https://www.anthropic.com/news/claude-text-watermark) is not that clear-cut:

    "When Claude proofreads text written by a person, what it gives back has generally only been lightly edited; because nearly all the words are the person’s, there’s very little (if anything) for the watermark to attach to. Depending on the length of the text and how heavily Claude has edited it, those changes might not be enough to make Claude’s involvement detectable."

    (emphasis added)

    • mrguyorama

      yesterday at 3:42 PM

      You've entirely missed the point.

      A proofreader, like an english teacher, can return you your writing simply annotated and marked up, with suggestions and edits in red pen for example.

      Then you rewrite your draft into a final using those edits and notes as suggestions.

      Making an LLM act like that proof reader would likely not cause your output text to be labeled as generated, even by this system.

  • wuschel

    yesterday at 8:24 AM

    Where is the problem with using LLM generated text?

    You could use your own hypothetical house elf to do it for you, or pay someone to do it. LLMs are just cheaper for a certain set of problems.

    People will find ways to circumvent this, so this limitation will only hit the technically less adept people.

    • zahlman

      yesterday at 8:52 AM

      > Where is the problem with using LLM generated text?

      In the fact that you didn't write it.

      > You could use your own hypothetical house elf to do it for you, or pay someone to do it.

      Yes, and those would be similarly problematic (and more expensive).

      • ivan_gammel

        yesterday at 9:35 AM

        > In the fact that you didn't write it.

        This is a fact and this is generally not a problem. Customer support guy Joe did not write that email to you with a refund: someone else did it and Joe did pick the template. Alice did not write that post card to Bob, someone else did and she just googled some nice text. We deal with a lot of content that wasn’t written by the person who signed it. That content, when written by LLM, may indeed contain watermarks and nobody will care about the choice of words, because only the meaning matters in such communications.

        People pay too much attention to authenticity here, which is no more than a demonstration of an effort. LLM can and should write scientific articles because the real effort is in directing research, not summarizing it. LLMs can and should write news, because it is cheap and efficient, and real reporting is in discovery. LLM can and should write fiction and make movies, because there is no reason why creators of various junk should earn their money easily. LLMs do not replace real talent. They just emphasize for an average person how easily replaceable they are. And that‘s ok. Creative industry is a blue collar job now.

        • jorisw

          yesterday at 1:49 PM

          It's easy to name examples of where authenticity matters and where it doesn't.

          In case of the former, there is a problem worth solving.

        • Alpha3031

          yesterday at 10:54 AM

          If it's not a problem, then there should be no issue with not concealing the fact, no? Lying about things one considers inconsequential is a useful signal about one's willingness to lie with little benefit.

    • NicuCalcea

      yesterday at 12:38 PM

      If there was a ghostwriting detector, I would use that too.

    • vrganj

      yesterday at 9:44 AM

      Nobody says it's a problem. We'd just like to know.

      Factory farming also makes meat cheaper than organic practices. I'd just like to know which one I'm getting.

  • richardatlarge

    yesterday at 1:14 AM

    Nonsense. Forget proofreaders. Think editors. In publishing some editors practically wrote the books. And then theres ghostwriting ! Think of that!

    • kalleboo

      yesterday at 3:47 AM

      Part of me wishes we had the same regulation for ghostwriting etc. Nobody should be claiming to have written a book they didn't.

    • Apocryphon

      yesterday at 3:15 AM

      Okay, then it should be acknowledged if a work was AI-edited-written, or AI-ghostwritten.

• demetrius

  last Sunday at 10:34 PM

  I'm not sure the quoted statement is true. Proofreading like "point to problems in the text", if you fix the problems yourself and don't copy-paste the solutions given to you, should still be safe, shouldn't it? So, human-written text should not be falsely flagged if you use LLM for proofreading.

  And if you copy-paste the answers from LLM, I think it's only fair the end result gets flagged. You're not writing it yourself.

  • 190n

    yesterday at 12:16 AM

    > And if you copy-paste the answers from LLF, I think it's only fair the end result gets flagged. You're not writing it yourself.

    I wonder if it would even get flagged in that case, because wouldn't the probability distribution of a token when the LLM is suggesting an edit to your writing be different than the distribution of that token once it is in the context of the text it's editing?

    • beej71

      yesterday at 2:51 AM

      I don't even ask LLMs to go that far. Tell me if I've made a spelling, punctuation, or grammatical error, period. Don't rewrite a thing, because LLMs suck at that.

• skew-aberration

  last Sunday at 11:04 PM

  Can't the LLM just generate e.g diffs? Or some other intermediate language. Then the watermark is lost when the translation step is applied.

  • croemer

    yesterday at 1:35 PM

    That's a good one. Problem is that SynthID uses only roughly the last 4 tokens, so any literal test 4 tokens or longer that is generated has the watermark. If you fix typos, insert punctuation, rename variables then yes this should be undetectable if produced from a diff.

• zmmmmm

  yesterday at 12:24 AM

  The question is, when is the "pro writer" version coming that lets you control this behaviour but costs more? Like night follows day, this will happen.

  They will need to dodge around the EU requirements but it will probably just come down to an alternative method to watermark or a contractual assurance you won't mis-represent the source of the text.

  • wuschel

    yesterday at 8:27 AM

    I had the same thought.

    I hope the other providers will add a geographical limitation on this EU rule.

    (On a side note, I wish they would replace those EU beauracts with LLms).

• jleyank

  last Sunday at 10:43 PM

  Rands made this point a few days ago as I recall. Worries about having his tool corrupt his writing during editing, etc.

• smb06

  yesterday at 12:36 AM

  The thing that could change is interpreting "the whole thing as generated by Claude"

  • inigyou

    yesterday at 12:40 AM

    Well yeah, if it's output from Claude it's likely to get detected as being output from Claude.

  • beering

    yesterday at 12:59 AM

    It’s a strawman argument because if the LLM is really just “proofreading” for you, there will be little or no watermarked text in your writing. Not enough to trip the watermark detector.

• KaiserPro

  yesterday at 10:43 AM

  wait what?

  but proof reading is a linter, not a writer. the proof reader will say "I think this is clumsy can you try x,y & z"

• ButlerianJihad

  last Sunday at 10:20 PM

  It is quite just, if you think about it. Human works are copyrighted and protected at the moment of creation. All rights reserved. Yet, LLM outputs are uncopyrightable. Therefore, if Claude or any AI has processed my copyrighted work, the end result is uncopyrightable and in the Public Domain. The public has a right to know: is this a human copyrighted work, an LLM PD work, or is the human falsely claiming authorship in order to retain copyright?

  A point of confusion for me, however: is every watermark unique? Is every algorithm for watermarking going to vary amongst models and amongst model versions? Will each model publisher keep this watermarking as a trade secret, that they alone can detect? If so, this can't scale! How do you detect "JoeBob 4.3 LLM" output? By querying every single model's watermark-detector? And if they all work by re-running the model and using tokens anew? That is extraordinarily wasteful.

  If a watermark is not self-evident, or universally detectable, then it is no good. Take, for example, US currency. The security measures are published and well known. Any count-out room in retail has a big poster indicating how you can detect authentic US bills. Nobody has to accept non-US currency in the US, and so the only authenticity you need to worry about is your US bills alone. LLM watermarking has none of this in common. Currently sounding like a shitshow, if you ask me.

  • dare944

    last Sunday at 10:51 PM

    As I understand it, the current watermarking methods rely on a secret key, making the detection schemes a black box to anyone not in possession of the key. This means organizations like Anthropic are free to make any claim about authorship they want, true or not, and no one can call them on it.

    • demibabs

      yesterday at 12:27 AM

      Part of the legislation requires them to make a public AI text detector (ala GPTZero I assume).

      Wouldn’t having that be enough to eventually reverse engineer the key?

      • dare944

        yesterday at 12:35 AM

        Not if they designed the algorithm right.

      • inigyou

        yesterday at 12:42 AM

        Probably not to get the key, but you could certainly use it adversarially to remove the watermark.

        Removal may come down to changing every third token to a different one.

        • NewsaHackO

          yesterday at 6:06 PM

          Yea, they are going to have to monitor how often similar writing pieces are being submitted, or limit the access to the checker somehow.

          • inigyou

            today at 7:29 AM

            Europe trusts its institutions way too much. It may well be that only the police can access the checker.

  • fwipsy

    last Sunday at 10:24 PM

    Perhaps LLM outputs are uncopyrightable, but derivative works of copyrighted works are not automatically in the public domain.

    • ButlerianJihad

      last Sunday at 10:40 PM

      That's an intriguing twist, isn't it? It could lead to a tug-of-war.

      Working backwards: if it is possible to confirm 100% confidence that a chunk of text is LLM output, then it is "PD until proven otherwise". How can a human reliably assert human authorship of their source text? When all watermark tests fail? Is that proof of humanity now?

      If a human proves human authorship, and LLM watermarking tests positive, then is that going to be considered a "derivative work" or not? What if there is an applicable license for the source work, such as "CC-BY-ND" that prohibits derivative works?

      This has not been court-tested, and I expect that it will need testing at that level before we can have any assurances.

      • inigyou

        yesterday at 12:42 AM

        How do you assert it now? I post some text on the internet, you claim you have copyright, how do you prove that?

        • summarybot

          yesterday at 3:10 PM

          Copyright law was updated in a very helpful way in the last twenty years sometime so that as soon as you post something to the internet you have copyright. If you need a citation don't hesitate to ask someone else.

          • inigyou

            yesterday at 3:51 PM

            So you can just say some text is copyright to you and it is?

            Here is some text that is copyright to me. As you infringed my copyright, please pay my $5000 license fee for every user who has read it:

            > Copyright law was updated in a very helpful way in the last twenty years sometime so that as soon as you post something to the internet you have copyright. If you need a citation don't hesitate to ask someone else.

            • summarybot

              yesterday at 5:49 PM

              Rather casuistic take, there.

            • cindyllm

              yesterday at 7:24 PM

              [dead]

        • pizzly

          yesterday at 12:51 AM

          Some camera pointing at you while you work. Guessing the camera needs to be watermarked itself, which can be done by adding some watermark on the chip level (each camera will have different watermark), manufacturer can confirm the watermark.

          • inigyou

            yesterday at 1:02 AM

            Is that realistically how that is proven in court today?

      • last Sunday at 11:44 PM

      • pessimizer

        yesterday at 12:10 AM

        The watermark, if I'm understanding correctly, only proves that something has been touched at some point by a particular model, not that it was entirely generated by a particular model.

        As for it being "uncopyrightable" if it were the output of an LLM, I think computer people are making a very aspie interpretation of a single decision. I think it's more that the LLM (and thereby its owners) cannot itself hold a copyright on its output, that output has to be touched by a person before it is copyrightable. A particular view from the top of a mountain can't be copyrighted, for example, but a photograph of that view can be.

        I'm not sure it at all precludes a "robosigning"* sort of situation, where machines generate output, hired temps sign and claim that output, and immediately sign it over to the people who hired them (as a work-for-hire.) Copyright is stupid, artificial law, not logical.

        -----

        * https://www.mortgageauditsonline.com/what-are-robo-signers/

        • demibabs

          yesterday at 12:32 AM

          > The watermark, if I'm understanding correctly, only proves that something has been touched at some point by a particular model, not that it was entirely generated by a particular model.

          For the watermark to be detectable, the text needs to be like 75% AI generated.

          If you have an LLM “touch” one section of the article, it’s not gonna be detectable.

> One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter.

Then why are you using an LLM to write? They're not capable of understanding such nuance. They do pick randomly between two synonymous phrases, they do not use some super smart algorithm to pick the one that sounds the best.

This excuse doesn't hold any water at all - Occam's razor says the author is just super annoyed that his AI writing will be identifiable as AI writing.

• FeteCommuniste

  yesterday at 1:13 AM

  Yeah, I snorted at the sentence "The exact words we choose when writing matter." Well, then why the heck are you using an LLM to "write," man?

I’m surprised by the comments here being so favorable to anthropic. The comments are right about there being no “best” token, and yeah Gruber may have an agenda here. But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. Take the “gray” and “overcast” choices. And lets say before applying synthid the percentages were 48% and 52%. Those percentages were learned from the training data and RL. To change those percentages to 45% and 55% in a non learned way makes it seem like training wasn’t important? Or more likely they dont have the data that shows the failure modes? also, don’t these choices compound the changes to the distribution in later sampling choices?

It is a bit of a mystery to say that “its okay to choose different tokens that we would have for watermarking bc people don’t notice” as though word choice doesn’t matter. If it doesn’t matter, doesn’t that mean that intelligence is more of a commodity than they would want it to be?

• oconnor663

  yesterday at 4:44 PM

  > To change those percentages to 45% and 55%

  This seems like a fundamental misunderstanding of how this sort of watermarking works. (Either that, or I have a fundamental misunderstanding of how it works lol.) It doesn't change the probability distribution of the next token at all. If you were getting XYZ 48% of the time before, you're still getting XYZ 48% of the time. What's changed is where the random numbers come from. But as far as you're concerned, there's just as random as they were before, just like an encrypted message is indistinguishable from random bytes if you don't know the key.

  • mangoman

    yesterday at 5:21 PM

    huh, thanks for commenting this! I trusted the linked explanation post https://declaude.org/watermarking/ but actually reading the the synthid paper showed me that my understanding was wrong: https://www.nature.com/articles/s41586-024-08025-4

    It is definitely blurrier whether you can say this approach changes the distribution then. By definition, it _has_ to change the probabilities of output tokens, but it's not totally clear that the pseudorandomly generated scoring functions does affect the learned distribution.

    put another way, I think it's safer to do:

    compute distribution -> sample -> watermark from sampled options

    than it would be to do:

    compute distribution -> watermark distribution -> sample

    • Nimitz14

      yesterday at 6:19 PM

      You’re not getting it. The probabilities do not change at all. The only change is given some probabilities there is a deterministic method for determining which symbol was sampled from that distribution. The distribution or sampling process itself is not modified.

      • mangoman

        yesterday at 7:22 PM

        Based on the SynthID-Text paper https://www.nature.com/articles/s41586-024-08025-4 I agree that the LLM's learned distribution isn't modified, but I don't think it's correct to say that the sampling process is not modified. Also I just read the paper today so I could be misinterpreting things.

        As described in the paper, you're right that it doesn't affect the main sampling technique, but what they do is they sample the distribution for 2^m samples, and then use Tournament sampling to choose the tokens among those 2^m samples, and the watermark key changes the scoring of the tournament options, using the watermark key as an input to the random generator that generates the scoring functions.

        Then, to calculate the watermark, they take the text, and compute the mean g-values of the text, and a higher score means that it's more likely that it was sampled using the provided selection of tournament watermarking functions.

        let's say you had some top P words: mango, banana, pineapple, guava, and you sampled 8 times, and got each one twice in the following order:

        1. mango 2. banana 3. pineapple 4. guava 5. mango 6. banana 7. pineapple 8. guava

        without tournament sampling, you'd truly see any of those come through. But in tournament sampling, you take those 8 options, create m scoring functions based on the pseudorandom generator, and score the 'tournament' by sampling the biased distribution you create from the g values. That does change the sampling from based purely on the LLM and entropy, but i mean, if the watermark key is also generated from some entropy, it's probably representative of the original sampling options as expected?

        this is a very fascinating topic! I do still stand by my point that anthropic is the only one who can tell if something is watermarked or not and feeling icky, but the paper has mostly quelled my concern on impacting the intelligence part.

        • mrcslws

          yesterday at 10:48 PM

          I think you're correct. It does alter the distribution for each output token, implicitly giving each candidate token a different probability. Maybe that's fine, but it's not as magical as Anthropic [and a lot of commenters here] are making it out to be.

        • yesterday at 8:58 PM

• andy12_

  yesterday at 3:02 PM

  > I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model.

  But it doesn't! The distribution doesn't change at all. The only thing that changes is that sampling of that distribution becomes deterministic as per a precomputed seed.

  • sfink

    yesterday at 6:14 PM

    You could describe that as taking an input distribution and a sampling procedure and producing an output distribution. This is a difference in sampling procedure that produces a deviation in the output distribution.

    (If you don't like calling it a distribution when it's at 100% for the chosen token and 0% for all others, then look at it as an output distribution across all possible prompt inputs, or perhaps just the cluster of prompts that achieve whatever you're trying to accomplish.)

• josephg

  yesterday at 1:29 PM

  I’m sure it matters. But how much does it matter? How much (perceived) intelligence would you be willing to sacrifice for an accurate AI predictor?

  I’d sacrifice a few %, easily. Maybe 10%. The models are getting smarter at such a fast rate that I’d be willing to lose a month or two of progress to help slow down the AI cheating epidemic.

  It sounds like you expect this fingerprinting approach would dramatically reduce the intelligence of their models. But I’m sure anthropic has measured it. I doubt they would have rolled this out if the intelligence cost were that significant. I suspect the cost is less than 5%. I personally can’t tell any difference from before they added fingerprinting. I bet you can’t either.

  • mangoman

    yesterday at 1:48 PM

    10% is a lot! And why would this slow down the cheating epidemic? There’s tons of ways like declaude etc to get around the check. Also Claude’s style is very distinctive (e.g. “load bearing”) AND has changed since 4.5 quite dramatically. I might not be able to tell on a specific response, but I can tell you that I went from canceling my ChatGPT plan in November, to now reaching for it first and considering cancelling Claude because its style is getting really groan-inducing. It’s weird because I thought ChatGPT was really annoying about a year ago, and now codex is my first choice.

    Is the cheating epidemic so bad? I’m a little out of the loop there truthfully, what are the consequences of not being able to detect AI generated text in non academic settings? And in academic settings… maybe I am underestimating the challenge, but it does feel like the assignment and ways education happens needs to change?

    As an aside, I’m not totally sure why this solution feels so icky to me. There’s something Orwellian about how the phrasing of a passage embeds hidden information that only Anthropic can see i guess

    • josephg

      yesterday at 7:24 PM

      > There’s tons of ways like declaude etc to get around the check

      How effective is this on the new fingerprinting mechanisms?

      > Is the cheating epidemic so bad?

      From what I’ve heard, yeah it’s out of control. And all the existing llm detectors that academics use have a high false positive rate, which catches a bunch of innocent students in the cross fire.

      > it does feel like the assignment and ways education happens needs to change?

      Why? Was there something fundamentally wrong with how universities teach and assess?

      The sector is responding. For example by moving back to more in person exams and reducing the load of any take home exams. Is that good, for some reason?

      > There’s something Orwellian about how the phrasing of a passage embeds hidden information

      Interesting. I don’t have the same response. LLMs give me an acute sense of existential dread each time their capabilities improve. But fingerprinting doesn’t move me at all. Do some soul searching on why this bothers you. I’d love to hear why, and I bet you aren’t alone.

  • sfink

    yesterday at 6:09 PM

    In order to do anything about the cheating epidemic, this has to be robust to adversarial attacks. Color me very skeptical on that one. It seems like reprocessing a generated text and adjusting the distribution is a way easier problem than generating the text in the first place.

• elim_garak

  yesterday at 3:17 PM

  > To change those percentages to 45% and 55% in a non learned way makes it seem like training wasn’t important?

  I think (or I hope, anyway) that this overestimates how much impact the tweaks actually have on text.

  The model has some things it "wants" to say. If it wants to tell a story about how someone reacted to dreary weather, it's going to tell approximately the same story regardless of whether the dice-roll caused it to describe the weather as "gray" or "overcast". And because "gray" and "overcast" were _already_ possibilities, the tweak from 52% -> 55% is completely lost in the noise.

  But it's true that this is all based on hope. I'm confident that you could make the tweak against arbitrary prose and even a true artiste like Gruber would never be able to tell the difference. I'm less confident that there isn't some edge case somewhere that causes a tweak to be worse than 3%, especially in some narrow application where word choice _does_ matter (like law). Even then, though, laws are already written by people who are as noisy if not noisier than LLMs.

• jorisw

  yesterday at 1:34 PM

  > Gruber may have an agenda here

  Such as?

• wewxjfq

  yesterday at 1:45 PM

  You're assuming the training data is a true representation of language usage, which it isn't. In the "gray" or "overcast" case, a single weather website might already skew the count of these words.

• matheusmoreira

  yesterday at 1:24 PM

  > But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model.

  This. I want the model I'm paying for to be "pure". I don't Anthropic or anyone else messing around with it, especially not for idiotic reasons like facillitating AI stigmatization. The "safety" nonsense is obnoxious enough.

  They should train the best possible model and let the weights speak for themselves, not degrade it into some perverted form to appease people who hate AI anyway.

  • remus

    yesterday at 2:54 PM

    > I want the model I'm paying for to be "pure".

    I don't think the models are pure in any meaningful sense. The labs have some idea of what kind of output they want from the models and then they put a huge amount of effort into training the models on the right sorts of data and massaging the models afterwards to push them towards the desired output. Then at a more practical level there's the layers of filters before your prompt even hits the model (e.g. anthropic's auto-mode classifier), system prompts, response level filtering etc.

  • applicative

    yesterday at 1:29 PM

    You can’t be serious. The weights for composition are massively degraded by RLVR training for coding

    • mangoman

      yesterday at 1:54 PM

      I agree, but thats also learned, and it is done to encourage specific responses for a task, different than applying a mask to the distribution based on a key.

  • josephg

    yesterday at 1:36 PM

    > especially not for idiotic reasons like facillitating AI stigmatization.

    From the people I’ve talked to at universities, LLM based cheating in education is an unstoppable nightmare. I don’t have a problem with LLMs. But I do want the cheating to - somehow - stop. The people who cheat miss out on learning. And the people who don’t cheat have their degrees devalued by those who do cheat.

    • matheusmoreira

      yesterday at 2:09 PM

      > The people who cheat miss out on learning.

      They aren't there to learn. They are there to jump through hoops to get a degree that will let them get a job so they can make money and prosper. The learning is entirely secondary.

      The cheating will stop when there is no longer any economic incentive to be there in the first place. People with "pure" motives will refuse to cheat on their own, precisely because they want to learn and cheating prevents them from learning.

      Watermarking AI output is a treatment for symptoms. The cause is the higher education meme. Somehow, getting a degree just became the default. Can't get a good job without a degree. That meme needs to die, and higher education will never recover its integrity until it does.

      • sfink

        yesterday at 6:17 PM

        Easy to say, hard to come up with a believable alternative. In the meantime, you're flunking out a lot of people for having the integrity to not cheat and as a result not being able to keep up with an artificially inflated workload.

        You can't just destroy some signal and handwave that you'll make it up in some other way.

        • matheusmoreira

          yesterday at 6:46 PM

          > the integrity

          There is none. It's just a jobs program fueled by student loans. Higher education in the west has been corrupt for quite a while now. AI is just the final nail in its coffin.

          • josephg

            yesterday at 7:10 PM

            This is far too cynical. I use what I learned in my computer science program constantly.

            I agree that higher education shouldn’t be as required to get a good job. But reality isn't black or white. Reducing the entire sector to a corrupt degree mill throws the baby out with the bath water.

      • josephg

        yesterday at 7:12 PM

        Got any evidence for that claim?

        I’ve worked with plenty of smart, self taught programmers throughout my career. The highest paid guy I know didn’t finish high school.

        • matheusmoreira

          yesterday at 7:56 PM

          > Got any evidence for that claim?

          Bureau of Labor Statistics. Jobs requiring higher education pay roughly 2x more than those requiring high school education and roughly 3x more than those requiring no education at all.

          And even if there's no formal requirement for a degree, it doesn't automatically mean people lacking degrees will get hired either.

          Anyone who wants to get a well paying white collar job pretty much needs a degree.

          > I’ve worked with plenty of smart, self taught programmers throughout my career.

          And how did they get the job? Networking?

          > The highest paid guy I know didn’t finish high school.

          I think this ought to be the rule, not the exception.

          • josephg

            yesterday at 8:20 PM

            > Jobs requiring higher education pay roughly 2x more than those requiring high school education

            This isn’t really evidence either way. Why do companies pay twice as much for people with higher education? We can’t tell from that statistic. Maybe it’s what you learn in class that makes you twice as valuable to potential employers.

            > how did they get the job? Networking?

            Probably. After all, that’s how most people in our industry find work. Degree or not.

            • matheusmoreira

              today at 12:31 AM

              > This isn’t really evidence either way.

              I claimed job prospects is the biggest reason why people spend years educating themselves. Evidence of 2-3x higher salaries directly supports that by providing an excellent motive.

              • josephg

                today at 1:42 AM

                I can imagine two different reasons for the higher salaries. Companies may genuinely think higher education makes their employees that much more productive. And you see that in their hiring process.

                Or the degree is something else. A marker of status. Or a signal of IQ and conscientiousness, since companies are legally barred from directly issuing IQ tests.

                An observation of 2-3x higher salaries for graduates doesn't differentiate these two theories.

                But rereading your comments, maybe that was never your argument. You said:

                > The learning is entirely secondary. The cheating will stop when there is no longer any economic incentive to be there in the first place.

                Companies might be entirely rational in offering 2-3x higher salaries to people with degrees. If that is the case, it's not just a meme. And the economic incentive will remain indefinitely. And universities and companies aren't doing anything wrong here.

                Throughout your career, on the job performance matters a lot more in aggregate than your CV. Most people are hired from referrals, after all. If students value the degree over the education itself, they're cheating themselves out of all the benefits - economic and otherwise - that education can provide them throughout their career.

                • tptacek

                  today at 5:59 AM

                  Companies are not in fact legally barred from directly issuing IQ tests, and several major companies do. This is for the most part an Internet urban legend (it has a tiny and irrelevant kernel of truth, in Griggs).

What a bunch of entitled whining. How is the system to know that it's just a private conversation that won't be used in some fraudulent way? Abuse is currently rampant, yes please let's find a way to mark LLM output. The thing I'm worried about is giving the providers the power to claim provenance. Even ignoring the privacy issues, the operational hassle of having to check N providers makes these approaches at best limited. I want to see research into providing a shared public or ideally self-hostable oracle that uses some standardized method for watermark detection. Similar to asymmetric crypto where users can't reasonably find out the secret part but can do something useful with it nonetheless.

I was initially surprised that Gruber was so invested in the "quality" of AI-generated text, which in my mind is an oxymoron. But really, Gruber's interest here is with the EU. This forms part of his ongoing attacks on the EU, all because they have been forcing Apple to align with regulations.

• inigyou

  yesterday at 12:44 AM

  Can we install random unapproved apps on our iPhones yet, or is Apple aiming to just be fined a trillion dollars because they make more than that from the 30% cut?

  • rsynnott

    yesterday at 6:22 AM

    Yes: https://support.apple.com/en-mk/117767

    • inigyou

      yesterday at 11:04 AM

      This says we can only install Apple-approved apps.

      • rsynnott

        yesterday at 4:34 PM

        ... Maybe it's localised or something?

        I see a section "How to install apps from a developer’s website in the European Union". Do you see that?

        (They do require notarisation, but not review/approval).

        • inigyou

          today at 7:28 AM

          Notarisation is review/approval

• rimliu

  yesterday at 10:49 AM

  I am from EU. Alas it has a tendency to produce some idiotic regulations. Cookie banner, new packaging fee, etc. I genuinely think some Apple related ones hurt customers more than help them.

This article feels slightly incoherent. You want high quality precise writing and to use an LLM to generate it? Feels like those are diametrically opposed

• beering

  last Sunday at 11:48 PM

  Exactly. The watermark is proportional to how much text is AI generated. Either the AI really just “fixed some typos” (not enough AI content to hide a watermark) or the AI did most of the writing (enough AI content to hide a watermark).

• breezybottom

  yesterday at 1:12 AM

  This feels like the inevitable outcome of a STEM-only education system. Now people think there's a mathematical formula for picking the "best" words, instead of having to be thoughtful and creative.

This is not meant to be snarky, But almost any writing done by Claude is a perversion of writing.

I honestly can't stand the way Claude writes. This watermark change just makes it scarier.

• _kulang

  last Sunday at 10:53 PM

  I moved to Sol for my writing and it is so so much better. But it makes more mistakes. I think they have different ideas of product but it seems OpenAI is going to follow Anthropic’s lead over the next year. I think I am going to put more effort into my writing skills to remove myself from this awful situation

  • einpoklum

    yesterday at 12:35 PM

    > I moved to Sol for my writing and it is so so much better.

    No it's not. The bad part about it is that some machine is writing instead of you, not the specific stylistic idiosyncracies.

    • _kulang

      yesterday at 1:28 PM

      Sure – but I’m entitled to make a judgement on what I consider to be “good” and “bad” output from an LLM, where “good” just means “helpful to my process”.

• gitaarik

  yesterday at 3:56 AM

  Sorry, what's scary about it exactly?

LLMs are already perversions of writing, so what else is new. Oh no, the over-long circumlocution generated by three autocorrects in a trenchcoat might be slightly longer because of this and maybe people will start noticing the subtle rhythms of vaguely peculiar word choices as yet another cue that you are wasting their time with machine-generated wordslop, what a terrible fate. Your long rambling walls of machine-waffling might be 37.05% longer than they need to be instead of the mere 36.58% longer they are now.

> I want any LLM I use to choose the very best, most precise words at every single decision point.

Oh! If you want that, you should run your own model and set the generation temperature to 0 :) Because that's not what any commercial LLM is doing. Never has been. This is just making up a universe that doesn't exist so you can get mad about no longer being in the universe that doesn't exist.

The masking technique of using a subset of the statistical distribution for each next token isn't going to be meaningfully distinguishable from a natural language perspective. I honestly think its a very elegant way to implement watermarking. I've got no real opinions on how effective it will be to people actively trying to defeat it, but I suspect that the people who are trying to pretend that LLM text was something they wrote themselves are probably too lazy to put in the work to try and defeat it anyway.

> But only Anthropic will be able to determine if text was seemingly generated by Claude, and Anthropic will only be able to detect the watermarks that are applied by Claude. Claude can’t detect the hidden watermark signals generated by, say, Gemini, and Gemini can’t detect the hidden watermark signals created by Claude, because each implementation is predicated on secret keys held only by the LLM provider

Well, akshwally...

> Interoperability. Providers must implement an interoperability solution for watermark detection such as a standardized API access method, a publicly readable signpost mechanism embedded in content, or participation in a consortium detection solution by February 2, 2027

• yesterday at 1:00 AM

This seems like a non-issue, or maybe I have the wrong expectations about writing. You write a text, ask Claude to proof-read it, but then you wholesale just copy Claudes output and use that as the final text? Wouldn't you review the changes it suggests and only take those you agree with, there by completely bypassing the watermarking?

Alternatively, you ask Claude to write the whole thing and proof read it yourself. In that case I'd like to know how much you'd need to change to break the watermarking, i.e. how much of a text would you need to change for it to be considered your work and not that of Claude?

• Laurel1234

  yesterday at 1:27 PM

  [dead]

It seems fine. I use an LLM to argue with me prior to posting blog posts so that I don't post obvious incorrectness, but the UX element to it is that it constructs notes about various sections of the text and we talk about those. There's no way for the generated text to enter the blog unless I copy-paste it and I'm not going to do that because the entire point is for me to write it.

At the point that you're generating entire volumes of text from Claude you're not really trying to be a sophisticated writer. I don't see how it's going to hurt for it to choose random related words.

it serves to show just how little regard the people behind these generated-text fingerprinting schemes have for the actual craft of writing.

LLMs have never been the place I've thought to expect any commitment to the craft of writing, to be fair.

LLM output, as the author acknowledges here, is already non-deterministic. Next token probabilities are set, and tokens are chosen pseudo-randomly. As I understand it, this watermark is just going to be a matter of using a known seed and algorithm to make those pseudo-random choices, such that a signature can be detected. The important thing is, it's not replacing intentional choices with random ones, it's just generating pseudo-random results differently. Quality shouldn't be affected.

• simonh

  yesterday at 9:04 AM

  That cannot be true. The quality of an LLM's output is the quality of the probability calculations for the next token. Anything that degrades the relationship between the system's best assessment of the appropriate probability and the actual probability used is a degradation of the quality of that probability and therefore of the output.

  If this didn't have a detectable effect on the quality of the token probability calculation, the watermark wouldn't be detectable. It may be a small degradation in the quality of the output relative to the neds of many users in many situations, but it's not zero. It's literally sometimes choosing different words that it otherwise would specifically for watermarking purposes.

  • phiresky

    yesterday at 9:33 AM

    An encrypted hard drive is EXACTLY uniformly distributed random bytes if you do not know the encryption key. No one would be able to tell the difference between a drive that is just purely random numbers or is actually filled with content. (Of course excluding the usually intentionally added readable header)

    If this was not the case, the encryption would be broken, and most everyone agrees that good encryption does exist.

    The "quality of the probability calculations" as you put it is 100% in this case and any less would be a huge deal (as in - breaks all of the internet).

    So, now you just take those same random bytes and use them as the seed for your LLM token choices. The output has the _cryptographically_ proven exact same quality as if you were using a true RNG (which you likely weren't using anyways).

    You just need to know your LLM distribution and the encryption key, then with each new token you exponentially increase the chance of knowing whether it fits your encryption key. Without actually affecting the token choice in a perceivable manner.

    > choosing different words that it otherwise would

    The "that is otherwise would" is carrying all the weight here. "Otherwise" is sampling from a distribution. You just sample from the same distribution but with a cryptographically secure, seeded RNG. https://en.wikipedia.org/wiki/Cryptographically_secure_pseud...

    "Knowing the LLM distribution" seems to me like the only hard part because you don't know the context of any random snippet.

    • simonh

      yesterday at 11:02 AM

      >You just need to know your LLM distribution and the encryption key, then with each new token you exponentially increase the chance of knowing whether it fits your encryption key. Without actually affecting the token choice in a perceivable manner.

      I struggle to understand the relevance of that comment.

      The blue/green token list biasing process literally does cause different tokens to be occasionally chosen. Not only that, but because a different token was chosen at one point, this changes the probabilities of all subsequent tokens, and the resulting later token stream every time it happens. If you had access to the token stream as it would have been, and the watermarked one by the end of the text they will be very noticeably different.

      • phiresky

        yesterday at 10:58 PM

        Well I was assuming your comment was saying it's _impossible_ to do this without affecting output, so I was just describing an option on how it IS possible. I saw that token list biasing post and yes that looks pretty bad. I didn't look in enough detail to see whether it's actually clear what Anthropic and SynthID really do. I would assume though their approach is of the smart kind.

  • yesterday at 12:51 PM

  • Nimitz14

    yesterday at 6:26 PM

    You have some fundamental misunderstandings on how LLMs work.

Gruber has a ridiculous knee-jerk response to anything the EU does, so hardly a surprise he didn't come to the table with a sober facts-based response.

> I chose to depend on a private company to express my own thoughts and now I'm mad that I'm not in control of the output

Who could have seen this coming???

I commented it last time the post about Claude watermarking went viral and I'm going to say the same thing again:

"I've been working in the media and model IP space for quite many years. What this article misses a bit is the threat model for watermarking in general. Watermarking and fingerprinting have inherently weak security guarantees -- they rely a lot on security through obscurity, weak assumed adversaries to deliver. There are clear trade offs between true positives, false positives and maintaining the quality of the media. It's true for audio-visual media, models and their outputs alike. As much as I like to take shots at poor technical choices by corps and govs, this one is unjustified. Sure, inserting glyphs is bad but biased sampling is as good as it gets in 2026."

Since the announcement, there have been many people who don't seem to fully understand what a security guarantee is, what trade offs it might involve, or how popular the type of technological solution is in general (media watermarking is ubiquitous).

And naturally, there are challenges with how you will make sense of the score in your org, e.g. you wrote an email, and it's flagged as LLM-generated because you copied two generated/edited paragraphs.

Yeah, the article might disagree with watermarking as a matter of principle, comparing it to censorship. But the methodological arguments that I have read so far have been thin in these articles.

There are many reasons to hate this watermarking but affecting the output quality isn’t one of them. The central argument he’s making is wrong. Switching out one RNG for another doesn’t make the results worse.

> My error was believing Anthropic that their system wouldn’t adulterate and corrupt the semantics of the text their models generate.

Has that ever been the case? Are they not actively tweaking their models, their fine tuning, the system prompts, the tool definitions and implementations, the guard rails, tool calls, instant responses. There are hundreds of knobs that they can change daily, or between each prompt, or even half way through a generation.

> The provider must mandate in their terms-of-service that users not remove the watermarking.

So, you don't own the generated text, and can't use it freely then. What if I copy paste a section, or rewrite a section of text to my liking? What if I rewrite some lines of code that contains the mark?

Security theater, and vague enough to be used as a weapon against who the government wishes.

I hope it's left off for non-EU customers.

• inigyou

  yesterday at 12:43 AM

  You know you can just ignore EU laws outside of the EU

  • knollimar

    yesterday at 9:27 AM

    They are literally asking for Anthropic to do that rather than roll out 1 system for everyone.

Claude's writing was already easy to recognize. The fact that Anthropic complied without complaint makes me wonder if they already watermark their outputs and used the opportunity to create goodwill. Presumably they want to avoid training their new model on text generated by the previous model, so they have reasons to be able to recognize AI-generated text.

Author read the posts but didn't understand them.

> At each decision point, they’re a little more likely to pick a word from the green list than the red list.

Wrong. There is no global red and green list. It's dependent on context and balances out on average. It won't change the result when one token is predicted overwhelmingly likely.

> I want any LLM I use to choose the very best, most precise words at every single decision point.

Try running an llm like qwen 3.8 27B in Q8 locally with an intentionally very low temperature setting, it will write like a caveman crossed with a robot. You may find that an extremely literal output does not look pleasant to read for humans.

• LoganDark

  last Sunday at 11:03 PM

  That is not what that means. Generally, precise word choice requires more than autocomplete. Larger models simulate this with hidden layers.

  • walrus01

    last Sunday at 11:07 PM

    Excessively precise word choice does not result in something that looks like content written by, or palatable to humans. It looks like you gave a high school 12 grade student a science paper and told them to apply a thesaurus to at least one word in every sentence and replace it with something else.

    • LoganDark

      last Sunday at 11:27 PM

      There is a difference between precise word choice and concise word choice. You can be precisely accessible the same as you can be concisely terse.

"The difference between the almost right word and the right word is really a large matter. ’tis the difference between the lightning bug and the lightning." - Mark Twain

"Just flip a coin to pick a random synonym. Who cares?" - AI Labs

I often agree with John Gruber, but I think he’s lost the plot with this one.

The thing I don’t understand is why he seems to care so damned much about this subject--enough to write over 4,500 words on it!

John writes for a living. That’s his profession. He’s been writing for over 25 years now. When you’re that good at writing, and you care this much about your writing, you don’t allow an LLM to take over your job. I just can’t imagine that he’s in the market for LLMs and that literary excellence is his number one selection criterion.

So why is he so livid about it? It’s like being angry that wine is going to start coming in smaller bottles even though you don’t drink wine.

Even if he’s angry on behalf of other people, I don’t get it either. In my view, having LLMs write publishable content on your behalf is not a socially-acceptable use case, nor a professionally-acceptable one in most professions, even though people are abusing it for this purpose anyway. And besides, the models aren’t even all that good at it today. If you agree with that, then you certainly should not care if it’s using different phrasing than you otherwise might prefer if the meaning is similar enough.

I can't help but wonder if perhaps his hatred of EU technology regulation (which, admittedly, is mostly pretty dumb and is mainly just making life worse for users) is getting the better of him.

• knollimar

  yesterday at 9:29 AM

  This guarantees he can never use it for the one purpose he might care about though

Sorry, this article's argument just doesn't hold water. Yes, we want Claude to write "the best text" and having its word choice even very slightly varied could arguably be construed as "not the best." But "best" is highly subjective, always has been. Claude has never, and will never, write what _you_ consider the best version of a piece of text. It has many choices, influenced by all kinds of random, context-dependent weights and environmental settings. Slightly tweaking weights to prefer certain phrasings might even tilt it toward your idea of "best."

Doc Searls, the long time Editor-in-Chief of the Linux Journal, brings an editors viewpoint to this, and mixes in a number of other voices.[1]

Many writers use AI to edit their own words, and this watermarking poisons the well for that use.

I think the backlash from this could be the seed that undoes their attempted 2 Trillion IPO this fall.

[1] https://doc.searls.com/2026/08/17/you-can-hear-the-squeak-of...

The fact that Gruber points to a "must-read" article about how the watermarking works, which, in turn, was very obviously entirely LLM-generated, says a lot about his lack of experience reading LLM output.

If Gruber can't tell a fully AI-generated article from a human-written one, perhaps he shouldn't care so much.

NB: I was told yesterday it's apparently a meme to call out Claude-generated output, but here I am, as I believe it's quite relevant to the topic at hand.

• ilogik

  yesterday at 8:03 AM

  Gruber really doesn't like any EU regulation

There is no coherent position in which the watermarking is a perversion of writing but AI writing as a whole is not a worse one.

Watermarking seems feasible for creative tasks (that admit many valid results).

But if I craft a prompt that doesn't leave space for creativity, how can they include a watermark? e.g. "Rewrite the following text, replacing 'foo' with 'bar'."

I'm curious to see where they draw the line, and whether the watermarking really affects the (perceived) quality of results.

• wisemanwillhear

  yesterday at 4:11 PM

  Although that feels like an overly simplistic example to the point of not being not being a helpful example, I agree that the amount of tokens to encode the watermark is so low in text, that it wouldn't take much to erase or distort it.

• rreichman

  yesterday at 4:11 PM

  They can't include a watermark in that case, that's not the relevant use case.

The issue here isn't (just) adulteration, it's that watermarking in general is unworkable.

If all the providers use watermarking systems with different shifting logit weightings, and the keys are secret, you have to check every provider to see if it produced a given text.

Which is clearly ridiculous.

And if all providers collaborate and use the same weightings, or if the weightings are constant and not rotated cryptographically, a generic watermark remover becomes trivial.

That's not even getting into the legal complexities of businesses running open source models without watermarking locally.

• vrganj

  yesterday at 9:39 AM

  > Interoperability. Providers must implement an interoperability solution for watermark detection such as a standardized API access method, a publicly readable signpost mechanism embedded in content, or participation in a consortium detection solution by February 2, 2027

  • frm88

    yesterday at 11:00 AM

    No idea why this was flagged to death. I vouched for it because it is a direct quote from the AI transparency act: https://theaicounsel.net/wp-content/uploads/2026/07/07_26_co... It clarifies interoperability requirements.

Are there some narrow cases (like in outputting the input almost verbatim) where the probability for every best token is always much higher than second best, and in those cases would there be no watermark?

Watermarking will be one more nail in the coffin of proprietary models if the world is so fortunate.

Reminds me of printer tracking dots.

https://en.wikipedia.org/wiki/Printer_tracking_dots

• inigyou

  yesterday at 12:45 AM

  And yet we still use printers and 90% of our color documents have the tracking dots.

I agree with the substance of this article, and disagree with the author's reaction to it.

The part I agree with:

It is true that watermarking can be done by "just" swapping one PRNG for another, and it is even true that with today's LLMs, it is possible that this will not degrade the output. But it has a cost, and as things improve, that cost will matter. You are intentionally reducing the degrees of freedom in the output, and using those bits of entropy for a purpose that does not improve the quality. If you maximize your tradeoff of bits for quality, those extra bits lower the ceiling of what's possible. It's a very simple information theoretic argument, and the only plausible argument against it (that we're using those bits so inefficiently now that the new PRNG is no worse than the old) only holds in the short term.

I also agree that having TOS that forbid removing the watermarking is deeply, deeply problematic. Hell, the whole essay is well-written and persuasive, and gives good reasons why this is a poor approach.

What I disagree with, and the reason for this comment, is the entitlement.

> The idea that anything other than my needs should factor into the generation of text for me is patently offensive.

This attitude is what is patently offensive for me. This is the argument that the world is beholden to my interests. It says that worrying about negative externalities is immoral. It's another form of certain people being above the law, shareholder profit maximization über alles, might makes right, we have to do it or someone else will, "we just help people connect", {code,a gun,roofie} is just a tool.

So I agree that the watermarking has a cost. But you can't leave out that it is an attempt to reduce negative externalities of AI. Whether it's a realistic or worthwhile attempt is a whole other debate (and Gruber does a good job of debating just that in the latter part of the essay), but saying that the user's needs are the only thing that should ever be considered is reprehensible.

• vintagedave

  yesterday at 6:13 PM

  > This is the argument that the world is beholden to my interests. It says that worrying about negative externalities is immoral.

  I agree with you, in that the benefit of the commons (preventing damage AI is doing to the world etc) is better than one individual's 'right' to a perfect product.

  But I don't think Gruber is saying quite this. He's worried about genuine semantic and intelligence loss, using the example of swapping two words:

  > The semantic difference between banana and pineapple is just as noticeable to the human eye as the taste of the two are to the human tongue

  Further, is this really a useful way to reduce the damage of AI? It only appears to work if someone already believes and then proactively checks if text is produced by AI. It doesn't really address the core issues.

• Spooky23

  yesterday at 6:08 PM

  > So I agree that the watermarking has a cost. But you can't leave out that it is an attempt to reduce negative externalities of AI. Whether it's a realistic or worthwhile attempt is a whole other debate (and Gruber does a good job of debating just that in the latter part of the essay), but saying that the user's needs are the only thing that should ever be considered is reprehensible.

  Computers are tools that exist to serve. Creating some bizarro process where we are compromising the technology in service of it's owner to achieve some nebulous goal is gross.

  Anthropic is crowing about this achievement because they are afraid of the dirt cheap AI models coming out of China and eventually other places impacting their valuation. Full stop. There's some vague notion of preventing harm without any backing, but a very real cost for startups to develop a compliant watermarked AI model.

  • sfink

    yesterday at 6:42 PM

    I agree with you as long as things are at a small scale. But at a large scale, these things reshape society and what it means to be human. Social networks started out[1] as almost wholly good. The negative effects came from scale. Same with advertisement-funded websites and tons of other things that started out as being overall positive for the commons and ended up being highly negative. You can't just stick your fingers in your ears and ignore what is happening in reality.

    > Anthropic is crowing about this achievement because they are afraid of the dirt cheap AI models coming out of China and eventually other places impacting their valuation. Full stop.

    It's the "full stop" that I'm disagreeing with. Yes, you can make valid arguments about the motivations behind this, or the effectiveness of it, or whatever. You can even conclude that it's a net negative. That's my current leaning. But "computers are tools that exist to serve" is an excuse, a conscious decision to abandon responsibility. Hammers, nails, social media, biological weapons, and date rape drugs are tools that exist to serve.

    [1] Ok, fine, at least one started out as a sleazy way to talk about people's physical attractiveness and behavior anonymously with repercussions, in a way where the victims would have no meaningful recourse. For the sake of my argument, pretend we're talking about Friendster instead of Facebook, please?

    • Spooky23

      yesterday at 10:28 PM

      I just don't agree, and that's ok.

      I love your point about social media. IMO, initially they didn't know what they had with social media - it took the right kind of uninhibited sociopath to monetize/weaponize it. That's why the MySpace guy is traveling and taking photos while Zuck is building island lairs.

      This is where a functional government would step in. There's no magic in social media technology or LLMs. I'd think of them as a car. You can buy a Nissan Leaf of a Porsche 911. One is an ok basic car. The other is an (over) engineered experience in the form of a car. You cannot legally drive a Porsche to it's potential on the public highways... because we have laws that regulate driving and hold the operator accountable.

      We're thinking about Claude Code or Gemini or whatever. The people running these companies are like "I want to exceed the power of John D. Rockefeller or Stalin." If we or the EU are going to regulate AI Labs, you need to grab them by the throat and they should be screaming about it. They seem to be very pleased with themselves. Barring real regulatory teeth, we would be opening the aperature to the Chinese companies to rationalize the valuations.

A question not addressed here is whether there exists any solution to the EU requirement that doesn't suck.

If the same red/green algorithm described in the article is applied to generated code, I cannot imagine how that does not degrade code quality (probabilistically, not at every point).

This scheme has a fatal flaw, the same secret key is used for watermarking and detection. This means you need to trust that Anthropic:

* Shows the real result from their detector instead of manipulating them, you have no way to verify * Has good enough security to prevent a key leak * Rotates keys to reduce the impact of a leak (once a key is leaked anyone can rewrite text to look more/less claude generated and it becomes useless) * Will not secretly give watermark-less access to governments or high profile corporate users * Will not use multiple secret keys to track individual users. This one might be less realistic because embedding ~32 bits of signal would probably affect quality a lot more than 1 bit.

And don't forget that the detection API will work as an oracle. If it detects your content you can send it to a different model and try again until it comes back clean.

I think we will see a hidden motivation behind this as ultimately so the output can have an author, the author can be attributed and finally the output will be copyrighted and so the LLM has more value.

Everyone who invests in AI companies wants to see the value of their investment increase.

I'd give it about 3-5 years until an AI company claims copyright over code their LLM produces. This is a crucial step in that path.

• Cakez0r

  yesterday at 9:45 AM

  I think you're right that it's actually to do with attribution (i.e. They're not just watermarking your output as claude generated, but watermarking it as claude generated _by claude user id 73684_). I think it's more to do with the growing militarization of the internet. Just another brick in the wall of enter your phone number to create an account, send your id to prove your age, smile for the flock cameras, etc. Allowing the plebs to have privacy and anonymity is not allowed anymore.

• tjpnz

  yesterday at 10:20 AM

  As someone who doesn't utilize LLMs for writing production code I'm looking forward to starting a consultancy and reaping the rewards.

I've been seeing a lot of the same questions about watermarking over the last week, so I made this playground that lets you try out 3 different watermarking schemes (including a version of SynthID-Text) on your own text: https://watermark.keito.me/ to understand it experientially myself. I found it illustrative to try different examples like code, text rewriting, etc. to see how it affected the tokens. Playing around with the detector is interesting too to see how much of the watermark can remain in edited text. I hope this is helpful for others too.

Lots of faux outrage, rambling and hyperbole here from Gruber.

“Absurdly and insultingly”? Come on…

I just don't like the idea of an AI company saying that they've added a secret signature that isn't verifiable by any third party to all responses. And we're supposed to both take them at their word, and feed them all the content we want to check so they can continue gobbling up a bunch of fresh works. What is to stop them from saying "Oh yeah, that is ours. We signed it. Trust us."

The obvious conflicts here are wild.

I dont see how there would be remotely enough entropy in most model outputs for this to be close to feasible with any kind of accuracy.

Either they false positve on pretty much everything ever written, or the chances of catching a true positive is so low as to be useless.

Basically Cinavia for text, and that often falls over and is easy to remove even when there is megabytes of data streaming over a long period of time rather than 2 or 3 bits per wall of text, let alone what most people use claude for, when there is a strict dictionary and other tight output constraints.

If we wish to use “the very best, most precise words at every single decision point” then I will note that “writing” is not the best word for what LLMs are doing. A more precise synonym might be “generating”, in which case this essay becomes “Watermarking is a perversion of text generation” - which is true, albeit somewhat trivial.

This was nice but without hard evidence. I'll bet if nobody were to have said anything, nobody would nave noticed. That doesn't make it right, but it might be less intelligible and intrusive than the claims here.

Also this:

"> I want any LLM I use to choose the very best, most precise words at every single decision point."

I don't think the author quite appreciates the level of randomness here.

This is more subtle then Claude changing prose.

Again - I suggest that the author would have to run a test on themselves to determine if they can actually find a difference.

>in areas where there is an arbitrary choice between particular words or terms within the code, the watermark can be used, such as comments within code.

I wonder if this is why Opus 5 keeps writing excessively long comments, even though I keep instructing it not to (both in chat, CLAUDE.md, and in its memories)

• croemer

  yesterday at 1:36 PM

  Opus 5 does not have the watermark yet. Only models released from now on.

This is moronic. This is like being mad that the slot machine you think is lucky is occupied.

The idea that language models can write anything remotely useful yet is - mostly - a fallacy.

It's a good idea for many human endeavors to be able to identify AI writing. Communication, after all, is our main way of building the social fabric.

However - and crucially - good writing is still beyond the frontier of any model I've seen so far.

Watermarks for the things that truly matter may not be important at all.

Finally, as X commentators have shown, simply removing punctuation or changing a word here or adding an adverb there manually will screw up the whole process enormously.

The best will be the clever folks who retroactively apply the model distribution to fraud or other crimes to try to implicate the companies via watermark.

Gotta feel for their product, policy and legal team.

Watermarks are garbage because they may embed account id, IP address and deanonimize you. That's why we should be using open-weights LLM whenever possible.

• dofm

  yesterday at 12:47 AM

  I don’t disagree about open weights (though the enabling aspect there is actually open source inference, right?)

  But it feels to me like you would need a hell of a lot of text to bury even a simple account ID. The nudges they are talking about are of the order of a handful of bits over several hundred words, I think?

• alienbaby

  last Sunday at 11:41 PM

  This is the first post I've seen mention it. How traceable are the embedded codes?

  • addandsubtract

    yesterday at 12:47 AM

    There was an earlier instance of this here: https://news.ycombinator.com/item?id=48734373

As the algorithm tries to keep the stenography in place this will mean if you specifically ask for a different phrasing of one paragraph, other parts of the document will need to change to keep the supposedly impossible to detect AI watermark in place? Won't people also just quickly do analysis on this to figure it out and remove the watermark - it's hard to do this for individual messages but when you can create an infinite number of messaging to train on, I'd think analysis of how the signal works will be quite trivial.

Some here are arguing that mechanisms used by LLM providers already derail the goal of "the very best, most precise words at every single decision point", therefore the author is misguided.

The author has expressed a preference. Assume that there is a sequence of tokens, such that it is considered the absolute best by the author. This particular method of watermarking makes it less likely to generate that sequence, by definition.

I feel their argument would have been clearer and stronger if they had spent more time exploring the alternatives, and whether these alternatives would be just as effective. It is trivially easy to remove invisible tokens.

Like it or not, there is a public good to being able to identify AI generated content, and a small degredation in quality is tolerable in my opinion.

I don't think anybody has to worry about this issue though. Manual writing, coding, and proof reading continues to be an option. Where AI output is nothing to be ashamed of, the tools are available. For everyone else, there will be LLM providers that ignore EU law.

• capitalsigma

  last Sunday at 10:53 PM

  If the author has preferences on their "own writing" that conflict with Anthropic's, then they should actually write it themselves rather than paying Anthropic to do it. Private companies don't owe you anything, even less so when they're beholden to laws in foreign jurisdictions.

• Barrin92

  last Sunday at 10:59 PM

  >Assume that there is a sequence of tokens, such that it is considered the absolute best by the author

  You can't assume that because if that was the case he'd already know what sentence to write, because that's what that means.

  The notion of a best sentence requires a final cause, an end to write to. By their very nature that's not how LLMs work, so you can't 'degrade' them on that front. They can't lose a property they didn't have.

I think writing is the killer use case for local LLMs.

We've had so many advancements in LLM samplers for improved text generation (off the top of my head: min-P, adaptive-P, XTC, DRY, p-less, Top-H, Top-n-Sigma, and so many more) but hosted LLM APIs only provide three basic knobs: temperature, top-k and top-p which are old as the mountains in LLM years at this point.

One thing that doesn't help the local LLM case is that all the popular VC backed local LLM wrappers also only support the same three ancient knobs because I suppose they're more preoccupied with their next fundraise than with keeping up with the advances in tech.

I have no doubt they can benchmax their way into believing the differences caused by watermarking are imperceptible. But I'm skeptical there won't be a drop in quality that puts them at a disadvantage relative to competitors that don't do this. This adds a new constraint changing predicting the next token to predicting the next token that carries the watermark. It's hard to see how this can avoid making the watermarked output <= the desired output in quality.

Humans have been inserting benign noise into their writing without affecting the signal for thousands of years. They called it "style".

Apparently superfluous descriptions, unnecessary words, and paragraph footnotes are all fine, but gumbel softmax (or whatever's going on here) isn't.

I really don't think Anthropic has the leverage to pull this kind of thing off in the face of competition. OpenAI doesn't do it (yet), Grok doesn't do it, neither does Deepseek. I'm sure the latter two have plans to intentionally never do it.

Who is even asking for this? Sounds like something some obsessive internal employees would push on the world.

This post makes no sense. You can either care about writing and write things yourself, or you can not care and farm it out to a model. There is no third option where you care about writing and farm it out to a model.

> I want any LLM I use to choose the very best, most precise words at every single decision point.

No, you don't. If you wanted that, you would set the temperature parameter to 0. But that would lead to less desirable results, not better. LLMs do not set the temperature to 0; they typically set it 0.4-0.7.

They chose the wrong word "watermarking." Perhaps they find the statistics _after_ a certain amount of text is generated at scale, so that it doesn't affect the fidelity or integrity of the LLMs output.

There is evidence this exists already, and it's why "I have to be honest..." and "This is the right lens, ..." keep popping up.

Probably would be cheaper too. Maybe I am missing something?

> Someone with the secret key can determine which list a word will be on at each token generation point (which is how the watermarking is detected); those without the secret key cannot.

How is this supposed to work in an actual lawsuit? Will Anthropic offer some sort of tool / (paid?) webservice to check for watermarks using that "secret key", and a judge is supposed to just believe whatever that tool's verdict is? And then it takes the EU another 20 years to understand what a silly idea this was?

• ghrl

  yesterday at 9:28 AM

  Yes, exactly, that is my understanding as well. Since the watermarking is based on a symmetric key by design (as to not be easily able to test and remove the watermarks), the providers will need to offer a watermark detection API.

  I see even more problems with this. To check any text for those watermarks, it needs to be sent to dozens of AI companies to check, potentially paying them all for just determining whether it matches their watermark, and more concerningly sending all that mostly human-written, often high quality text like unpublished research or books, to AI companies that almost all proved to obtain training data through all kinds of dubious ways.

I was using Claude yesterday and the "advice" it was giving me quickly became confused and irrelevant to the prompt, even though there was not much text in the context window. Speculation: since Claude re-reads the entire chat at every turn, the "minimal" text revisions required by watermarking quickly compound such that even Claude can't follow the discussion.

> The idea that anything other than my needs should factor into the generation of text for me is patently offensive.

And this attitude is incompatible with any models produced by frontier labs. Your needs will always be subordinate to and in service of the needs of the corporation that produced the model. And we haven't even gotten to ads yet.

I guess I understand the complaint, but LLMs are already crap at writing, IMNSHO. And, yes, maybe this will make them marginally more crap, but in my mind we're talking the difference between a 30% grade and a 29%.

As a university professor, back in the early days of this when students were just copy / pasting output directly I was really hoping the model builders would implement steganographic fuzzy hashes.

Not that it's the ideal solution. But it would be really neat.

The main reason we don't see much quality degradation in LLM writing output is because they're already poor writers. This is the load bearing reason.

I was bulding a small interpreter and writing an article in ~markdown yesterday with Fable. And while it codes like a pro, it writes like a sixth grader.

Let's see how these watermarking stats hold up if/when llms start writing well.

The watermark is based on stylometry - and we all know Claude has a style. But it doesn’t have to - everyone knows caveman - so not sure there is enough room in all cases for it to work.

Heck. The entire process itself of LLM text generation is a perversion of writing.-

Further heck: It can be said it ain't even writing.-

It's good to be the King.

And what I mean by that is that companies that are at the top tend to make anti customer decisions because they have lost the concept that pleasing customers matters as priority one.

Not sure about the strength of that particular argument, but can't say I have any faith in this being used to benefit end consumers.

Somehow they'll find a way to use this for regulatory capture

It’s super ironic to see this many bots on this thread. Sorry dang.

> It’s unacceptable for a tool to sacrifice an iota of clarity, coherence, meaning, quality, etc. for the purpose of embedding hidden clues within the text to suggest its provenance.

Yeah, that about sums it up!

LLM inference already isn't deterministic, the watermarking technique only limits the space of possible random seeds. There is no reason to believe that this subset of seeds somehow produces lower quality output.

What's gonna be a real trip, is when you can tell which LLM produced it by the subtle pattern recognition you'll have developed to catch the watermarks

Doesn't this require the whole message history including the system prompt for them to check for the watermark?

Is this 'best word' with us in the room right now?

Great question buried at the bottom:

> Also, what happens if another major global market makes it unlawful for AI to secretly watermark generated text?

• elpakal

  yesterday at 4:43 PM

  That is a good question

  > We’re applying watermarking globally at launch because we don't yet have a durable way to scope it by region. However, we will continue to evaluate different approaches, and will share updates when we have them.

  So unless they figure it out, would that 'major global market' essentially need to be the US?

So the watermark can be removed by rearranging words and choice of words. This seems trivial to bypass with a local model. If I understand this correctly.

• elpakal

  yesterday at 4:45 PM

  my understanding is that it's a probability so it can never really be removed? only made smaller.

Gruber was a good voice in the industry but this article misses the mark in a lot of ways.

A company the size of Anthropic would not voluntarily jeopardize their massive valuation if they didn’t feel the resulting output would maintain a similar level of quality as before. Is there a similar worry that their system prompt, which is injected at the start of every conversation also influences token generation in an artificial way?

If regulation will ruin Claude as a product, market forces will fill the void. There are also a ton of open weight models to choose from. It’s going to be okay.

• tmgldn

  yesterday at 11:10 AM

  Gruber is not much of a details man - he helped invent Markdown (to be lauded) but ghosted its standardisation. I would be fascinated to hear Prod John MacFarlane of UC Berkeley's opinion on it all given he was heavily involved in the push to get Markdown standardised.

Ai is a perversion of writing. A useful one, but still.

I don’t see how this makes it worse.

I wonder how much they tested on non-English languages with fewer synonyms.

any watermarking ai researchers who can explain this?

what if the llm should

- repeat something verbatim (important in a compaction prompt)

- there is just one correct order of tokens for a somewhat long chain (a certain sequence of control signals)

- provide a diff of 2 inputs

without punctuation or whitespace wiggle room?

how does the drifting work?

does it postpone the drifting and drift stronger later?

what if max_tokens is set to a low number?

in what way does this not affect output quality?

• talljeff68

  yesterday at 3:28 PM

  The Anthropic annoucement addresses lots of this. They give an example of generating code where there is clearly a requirement to follow a specific syntax and therefore the watermarking will be much less effective and likely require much larger sample of generated content to build statistical confidence in the validity/existance of a watermark. For code, it is the generated comments that will be more likely/able to contribute to the watermark confidence.

  The key to understanding the watermark technology is to realize that the model is/was already using randomness to select among the top most probable tokens, often randomly picking between choices of [nearly] equal weight. The watermarking does NOT change the distribution of the random number generation nor does it affect the range of probabilities for which tokens are being considered. Instead, it only drives the sequence of the random numbers such that they form a cryptographically generated known ordering pattern that is determined by the secret key generating the pseudo-random sequence.

  As a result of the approach, assuming inference is being done with all other parameterization of token selection being the same, there should be NO impact on the output quality....the amount of variation of output is within sample of the variation in output that already exists run-to-run of the same prompt. FWIW too: Google has confirmed this experimentally as well through full scale tests and evaluation of online Gemini output in search result pages.

People are very upset, especially in the arts, that Anthropic is changing the text to watermark it, but isn't that missing the point a little bit? They're not changing _your_ text whose every word you've carefully chosen for the exact effect, they're changing text that they're generating, i.e. text you've already chosen to give up control over. LLMs can't understand emotional nuance anyway.

The phrasing of the announcement implying that phrasing and diction don't change the meaning of text is insultingly dismissive of the whole field of literature, and I can see why people might take it as an afront, but the actual technology shouldn't have a negative impact as far as I can see. It seems to me that this one is more of a PR problem than something with real-world impact.

Using LLMs for prose that matters is already a perversion of writing.

It's no more a perversion of writing than the act of using an LLM to write in the first place.

I assume this means Gruber has fully embraced generating his blog, but is too embarrassed to admit it?

There is no very best token to choose at each decision point. It is context dependent and subjective.

Text watermarking is another EU rule made without real world input. The Union is stuck on major economic crises (electricity prices for instance) because nobody can agree on anything. However, the bureaucracy forces tech into a privacy nightmare. Brussels cannot bring together its own members but it loves pretending it can govern the internet.

• bramhaag

  yesterday at 12:32 PM

  > Text watermarking is another EU rule made without real world input.

  Except for the input of the hundreds of stakeholders they consulted, Anthropic included [1]?

  > The Union is stuck on major economic crises (electricity prices for instance) because nobody can agree on anything.

  That sure seems relevant for the implementation of AI watermarking...

  > However, the bureaucracy forces tech into a privacy nightmare.

  No, this transparency allows consumers to more easily detect AI generated content.

  [1] https://digital-strategy.ec.europa.eu/en/policies/code-pract...

• einpoklum

  yesterday at 12:34 PM

  > (electricity prices for instance) because nobody can agree on anything.

  I would say that's more like because the US has arranged for Europe's fossil fuel energy sources to be disrupted or cut off:

  * Libya - NATO made a pig's breakfast of that, it's a failed state now.

  * Iran - transitive sanctions, because why not prevent non-US states from trading with each other.

  * Russia (& Kazahkhstan) - The US (with or without Ukranian involvement) bombed the NordStream pipeline(s), led the EU into the proxy war in Ukraine and a sanctions regime against Russia. Kazakh oil goes to Europe through Russia.

  * Gulf states - until recently, possible but not very convenient ; since Feburary of this year, the war on Iran messed that up badly too.

  the US is the winner here not just geo-politically, but also as an oil exporter, with the EU now depending on purchasing US-exported oil.

  • Lumich

    yesterday at 4:53 PM

    Spot on. Also, in 2003, the war on Iraq, still occupied. And the proxy war on Syria (stifling an unwelcome pipeline project). European “leaders” pretend to not comprehend how they're being screwed. Stockholm syndrome. Populations don't understand, propaganda (“free press”) working correctly.

    • einpoklum

      yesterday at 7:55 PM

      It's very weird looking in from the outside. I mean, sure, the US is the dominant power and everything, so things like Iraq and Iran could be construed as just collateral damage from their imperial maneuvers. But it has just piled on, more and more, and even when they are hit right in the face with the massive bombing of NordStream, still practically nobody tries to draw any sort of line.

      Reminds me a bit of that 'Yes Prime Minister' sketch about nuclear deterrant, when the skeptical conversant asks the PM: "So what is the last resort, Picadilli?"

      https://www.youtube.com/watch?v=IX_d_vMKswE

      • Lumich

        yesterday at 10:50 PM

        Thanks for that YPM sketch. Incidentally, I think the “nuclear deterrant” story is total bogus from the get-go. But has worked great for decades as a framework to frighten people, or at least provide justifications and impose “global order”.

        The trouble with drawing any sort of line is that when you're complicit and “aiding and abetting”, there's simply no motivation. And as long as the populace doesn't have a clue what's going on, their understanding being reduced to “evil Putin”, that's not even much of a problem.

      • Lumich

        yesterday at 10:26 PM

        « right in the face » — It's mainly Germany that's hit right in the face and where the Stockholm syndrome applies most closely. If we look at our dear neighbours, then we can see that some of them stand to profit from the Nord Stream bombings. Poland, of course, but also the honorable Norway (have a lot of gas) or the Netherlands (have the most important port). These two also happen to be more closely allied and aligned with the (F)UK/US complex, not directly “Five Eyes” (Anglo only), but “Nine Eyes” (plus France and Denmark). The war had been actively prepared since at least 2014 (military “#TrainToWin” etc), so there was enough time for scheming and dealing.

        Guilt with a corollary of subservience to “Western values” having become the predominant ideology in Germany, strategic shortsightedness and failure to properly “relaunch” after 1990, also the occasional murder of more “promising” members of the political spectrum (sans doute at the hands of our dear “friends”), brain drain to the U.S., catastrophic “investments” like Chrysler or U.S. telcom, and barely anything of it ever raised to the threshold of public debate and intelligent reflection : a lot of things combine to explain the dismal state of affairs in modern Germany.

        It cannot be ruled out that the German gov was complicit in the bombings; not wilfully, but passively, like someone too weak to resist for lack of self-esteem. Like chancellor Scholz on Feb 7, 2022, at the White House press conference where Biden threatened the pipeline:

        ▪ “If Russia invades, that means tanks and troops crossing the border of Ukraine, again, then there will be no longer a Nord Stream 2, we will bring an end to it.” ▫ “But how will you do that, exactly, since the project is within Germany’s control?” ▪ “We will… I promise you we will be able to do it.”

        There was no reaction from Scholz. Just nothing. By the way, could be I'm wrong, but that part of the press conference seemed scripted/scheduled to me. Not the exact words, but the contents.

  • lluisantoni

    today at 1:50 AM

    [dead]

Then then don’t use Claude ? What’s wrong with all these people getting vendor locked in.

I wonder if this watermarking system to lead to an increase in tortured phrases, believed to be caused by plagiarism evasion tools that change random words to applicable synonyms.

https://arxiv.org/abs/2107.06751

• Alpha3031

  yesterday at 10:45 AM

  Seems unlikely unless Anthropic invented a time machine, given that the phenomenon predated Claude 1 by two years, and their stated introduction of watermarking (August 2) by 5.

  • GaryBluto

    yesterday at 10:52 AM

    I see how that was confusingly written. I'm not suggesting Anthropic are somehow retroactively causing it, just wondering if it could create a similar effect.

    • Alpha3031

      yesterday at 11:11 AM

      Less unlikely, but I would still suggest that it is somewhat unlikely for any recent LLM to place any significant probably on, e.g. "disappointment" instead of "failure" as the appropriate token (or series thereof) after "kidney" or any similar example.

I'd encourage reading this paper, and literature on scaling laws in autoregressive models: https://arxiv.org/abs/2303.11156

Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic announcement relies on. It becomes more difficult to reliably detect text as a fixed sample count without tweaking the distribution further. Either way, it's a minor problem that will be addressed over time, compared to the issue of who can detect this without guessing or developing their own sets: providers not releasing a way to detect any such watermarks without going through them makes this entire approach hostile to the public. The EU regulation on this subject is interesting, although again most certainly not the primary driver for these practices:

"1.1.2: Signatories will ensure that AI-generated or manipulated content is marked with an imperceptible watermark, with the exception of very short text. For free-form text longer than 200 tokens, watermarking still needs to be applied, even though it may have lower reliability compared to that of watermarking very long text"

A proper, effective and useful law would have required providers to regularly release datasets to run your own verification on any text released within a fixed interval of time, presumably once out of rotation. Instead, it only talks about exposing an user interface going through their own services:

"Signatories will ensure access to their detection solution through a user interface appropriate for the audience of end-users that may eventually be exposed to the content generated or manipulated by their AI system. [...] Any restriction to the access will be limited in time until more reliable and robust detection mechanisms have emerged and have been adopted as the state of the art for detection mechanisms for the watermarking of free-form text evolves."

Most interestingly, in line with the EU's mass-surveillance program, an alternative solution to watermarking where it may not be sufficient is also suggested, although only optional for now:

"Where appropriate and taking into account potential trade-offs related to privacy and security, as well as scalability challenges and costs, Signatories may implement as an optional supplementary measure fingerprinting or logging solutions for AI-generated or manipulated content which allow for checking whether content has been generated or manipulated by their AI system. For example, direct logging may be appropriate for text content, whereas fingerprinting approaches may be preferable for audio and visual content."

• zoobab

  yesterday at 1:28 PM

  "logging solutions"

  In 2005, FFII predicted that the Data Retention directive would not pass the Courts.

  It just took 10 years for the CJEU to strike down the measure, as it is "mass surveillance".

  Logging everything your bot does by law is of the same dimension?

• troupo

  yesterday at 10:21 AM

  EU is a capitalist union first and foremost. So their regulations try to straddle the thin line between "regulate to death and tell companies exactly how to do things" and "the companies can do whatever the hell they want".

  Since most Americans are in the latter camp, anything that even hints at making companies responsible for anything is viewed as being squarely the former.

  Whereas most EU regs are "play nice, be responsible, behave like adults. If not, this can always turn ugly". Same here.

Just copy Claude’s output and shove it into Gemini and ask to rephrase.

I will be happy to move off Anthropic given the chance. They are burning all of their good will.

I don’t understand how this works for anything but prose. Is that the point? In any code or structured output, there just isn’t the flexibility, and depending on how the user requests the output be constrained there is even less (“answer only True or False”). So is it just chat responses? If I ask the API to tell me a story about Alice and Bob then it watermarks it, but when I ask it some implausibly constrained thing like write a story about Alice and Bob with each word starting in rotation with the letters alicebob, does it try to do so and hope there are roughly équiprobable tokens regularly?

• smallerize

  last Sunday at 10:14 PM

  Yes, and it says that in https://www.anthropic.com/news/claude-text-watermark

  • andy99

    last Sunday at 10:19 PM

    I should have read that, it’s actually quite reasonable and I don’t really understand the objections in TFA having read it.

    > One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter.

    Doesn’t make sense at all in light of the actual approach, they’re just choosing a different RNG. It’s not like they’re corrupting it by flipping words.

    Should add I don’t support the watermarking and requiring it is idiotic.

    • krackers

      last Sunday at 10:25 PM

      I don't understand Gruber's points either, I wonder if there is some fundamental technical misunderstanding. Does he think that the logits should be sampled from in a "pure" manner without introducing any other bias? Does he know that there's already a sampling temperature, and that most providers have probably moved on to sampling strategies other than top-k? Does he know that the word choices have already been altered irreversibly during RLHF which is how you get the obvious Claudism like "load bearing" and "seams"?

      Perhaps it would be useful to publish examples of samples with/without watermark. I'd suspect that the variability from simply sampling repeated times would dwarf any semantic differences you'd detect with the watermark.

      • troupo

        yesterday at 10:23 AM

        > I don't understand Gruber's points either,

        Gruber's point is to bash the EU. He couldn't care much about anything else.

    • smallerize

      last Sunday at 10:42 PM

      I think Anthropic should have put all the info into one blog post. Splitting it up is really confusing people.

• Cakez0r

  yesterday at 9:55 AM

  This could be why claude code has recently started to write reams of inane comments alongside the code it generates.

• chrisjj

  last Sunday at 10:13 PM

  > In any code or structured output, there just isn’t the flexibility

  Variable name perversion incoming...

Posting for visibility a reminder that this type of watermarking is comically easy to beat. See Scott Aaronson's 2023 research in which he coined the term "pineapple attack": one must simply prompt the model to add the word "pineapple" after every other word and then do a search + replace to remove the word, to obliterate the watermark altogether.

https://www.youtube.com/live/2Kx9jbSMZqA?si=0QgCPBX2_KPZ0QTU...

Mind that even in their first example, "The results of the study were quite (important | significant | substantial | notable)", the meaning is by no means interchangeable.

"Important" refers to impact, "significant" to the statistical qualities of the underlying hypothesis, "substantial" to the work involved, and "notable" is a referential judgement by the speaker. The implied normalization of words and their respective meaning also marks one of the mechanisms how "slop" is typically creeping into the productions of "broad verbose interchange replicas" (it's all interchangeable, and a choice isn't really that, a choice, isn't it?).

"The more words in the text, the more accurate the analysis will be that the text was generated by a specific AI model or not."

And this is why the latest Claude models blather so much more. All for the sake of this "watermark."

The first thing that came to my mind was "security theater." Making people think that AI is detectable could have the same effect as actually making it detectable.

Watermarks are context poisoning.

If you don’t like it don’t use it

A lot of that rant is nonsense. Is Gruber also outraged that a RNG is already involved in everything LLM produce with a non-zero temperature? Surely it already leads to widespread use of “non-optimal words”. Simply the idea that there is a single optimal choice, down to every single word, to convey a meaning and anything straying from that is adulteration is laughable. It’s very difficult to take seriously an argument based on this kind of foundation.

He had a very similar rant a couple of days ago when he somehow thought that they would use invisible characters. It’s just as useless.

Fix it for you:

Generative AI is a perversion of writing

What a strange take. LLMs themselves are a perversion of writing. I couldn't care less about the implementation details of the PRNG they use for next-token sampling (well, as long as they're not stuffing a user ID in there).

Not telling someone you used AI is a perversion of writing. Also agree that an AI proofreader should not claim authorship, but in most other cases, the AI is not reading your mind, it's only watermarking its own usage, and we kind of need more of that.

Reminder: your favorite distilled model does not treat you, the customer, as an adversary and mess with your output.... May the free market win.

• yesterday at 12:00 AM

What a stupid take. Generating text with an LLM is already a ‘perversion’ of writing. Tweaking the last random-choice step doesn’t meaningfully change that at all.

I almost never post a comment here, but everything about the author's position is offensive and self entitled. I am so enraged that I can't even beging to formulate a response without resorting to very bad language. It's sad, because until now I respected the author. But clearly, and sadly, he has been afflicted with AI brain rot, and is likely in some stage of withdrawal. I wish him a speedy and safe recovery.

I think it's pretty dishonest of Anthropic to frame their watermark as EU regulation compliance. The EU regulation, from my understanding, requires AI content to be labeled for human viewers. In the meanwhile the Anthropic new release on the watermark says this.

> The difference between watermarked and un-watermarked text will not be distinguishable to readers

https://www.anthropic.com/news/claude-text-watermark

Which is to say, it does not actually meet the EU AI act requirements which require transparency to humans. Not to mention that if the detection requires access to the base models, it makes anthropic the only entity who gets the say on if a piece of text comes out of Claude. Anthropic is both the player and the referee here.

If there is one takeaway you should have from this fiasco it is that you should be wary of using tools that doesn't serve your needs and your needs only.

• cubefox

  yesterday at 12:24 AM

  > The EU regulation, from my understanding, requires AI content to be labeled for human viewers.

  How would that work? Claude appending " written by AI" to each of its messages? That would both be impractical and useless.

  • inigyou

    yesterday at 12:47 AM

    I think there are two separate requirements? One that if you post something like an AI video on the internet or anywhere else, you must label it as AI. And another one that AI providers must watermark their outputs.

    If you get caught uploading watermarked media without the clear label, you're in big trouble, mister.

    • knollimar

      yesterday at 9:56 AM

      And it mandates that Anthropic put it in their TOS rather than make it law.

    • cubefox

      yesterday at 8:24 AM

      We were talking about text.

      • inigyou

        yesterday at 11:03 AM

        Which must also be marked as AI, I hope. (But I doubt that is the law, because the EU is cucked to big businesses interests)

        • cubefox

          yesterday at 2:02 PM

          > How would that work? Claude appending " written by AI" to each of its messages? That would both be impractical and useless.

          • inigyou

            yesterday at 3:49 PM

            Claude text in Claude is already clearly labelled as AI.

It's because the individuals who write those laws are literally trying to shove neo-Nazi policies into EU practice. I wonder how many people would make policies like this if they and their families were publicly identified and criticized as neo-Nazi elements in society.

You think someone will write Nazi-promoting AI policy like this when society is encouraged to look at their families as examples of neo-Nazi corruption? When their wives' and kids' friends spurn them while their families engage in obvious criminal activity to harm world productivity?

Critics need to be more precise.

Plagiators are having a fit these days

I needed another reason to cancel my Claude sub. Thanks Anthropic!

This is akin to adding a giant watermark on things one would made with a free product "Made with XXX". Except you're paying $200/month for it, and there's no way to disable that watermark.

I don't disagree with EU regulations, but I strongly believe the onus should be on the content publisher, not the toolmaker. If the toolmaker watermarks whatever his tool produces, it opens a giant can of worms that cannot be closed. That means anything and everything you make with this tool is no longer fully yours, it contaminates everything and makes your work traceable. Who wants that?

I was already annoyed by the fact that Claude marked everything it did on my repos under its own account (I didn't ask for any of this), but now everything is invisibly marked, even the code. Not that I care that my writing would be watermarked, since I'd rather write my stuff myself, but code? No thanks.

Meanwhile I'm running a DeepSeek V4 Flash or Pro, or a Qwen3.8, and it writes my code without a peep. Resulting repos are clean, just the way I want them. No 'Claude' account, no watermarking, nothing. I won't be looking back after having tried these new models. Whoever makes good models that don't broadcast their maker will get my business.

This watermarking will simply push people more towards Chinese models. Keep pushing in the wrong direction Anthropic. Doing this right before an IPO is a great idea.

This is the yet another embarrassing idea from the EU they came up with, if they could focus less on dumb things and more on providing real support to the development of technology maybe we could have more healthy competition that will eventually lead to better overall technology in the hands of everyone instead of acting like dumb chickens and not just being passively useless but also stopping the real labs from doing the real work into doing this bs

Claude is already a perversion of writing.

A) without water marking, llm will poison the open internet and poison their own source of fresh new material. B) without watermarking, it potentially exposes an infinite deluge of garbage getting to people , fostering hate, eventually leading to dis-engagement, ergo destroying their training pipeline. C) ...Sadly, them water marking the claude chats with absolute dementia levels of output text is beyond me, i seriously can't tolerate this bullshit "randomly replace words with "similar" words", bs anymore, the output that claude makes right now is outright *corrosive* to my eyes, eventually leading to their training pipeline dying if devs dis-engaging ( of the 3 i think this one is the worst )

I think they need to get their shit-together and realize this is a death warrant for the tech ( in my opinion ).

They just want to prevent their own slop being fed back into their AIs.

I think this would be a good guard on websites to use it as another content protection layer.

The author makes a lot of great points. I find it surprising that somebody who has such a nuanced appreciation for the subtleties of language would be in the crowd complaining about the watermarking policy. I expected this kind of complaints from mostly students interested in academic dishonesty, who generally don't have enough command over written language to notice the slight decrease in output quality.

He's complaining that a coin-flipping synthetic text extruder is using an unfair coin. The text generated by an LLM is already adulterated. His objections are so much titling at windmills.

Such a stupid regulation. Don’t make the rest of the world suffer because of one dumb law passed for a fractional share of users.

This also, just another precedent of anti-user, pro Authoritarian, from LLM companies.

Why does everyone love AI and is impressed by AI quality but when the time comes to admitting that they use the magical AI, no one wants the watermark?

How does this work for code?

His argument for needing it to write every word at its best is weak. The black box within can change quality at every moment based on many factors they he hasn’t known about, say system prompt, or other harness adjustments. Models get better every 2 months, and they write better, but he is comparing to what?

The real argument should be watermarking itself. I don’t want my shit water marked if I ask you to just rephrase a certain part.

"My writing is my work, and Anthropic's current strategy is aggressively writer-hostile."

The same absolute morons who gave us cookie consent strike again.

I swear, one of those days I will get into politics just to fight those two things, and the cottage industry of batshit crazy lawyers that gave birth to those things.

• vrganj

  yesterday at 8:44 AM

  The cookie consent banner is not the EU's fault.

  It's either don't track or ask for consent. The fact that the industry chooses to track is not on the EU.

  • aenis

    yesterday at 2:53 PM

    Its more nuanced than that. Even companies that do not track, and use only essential cookies, ask for consent, as the consensus among compliance teams and external lawyers is "its safer this way". Thats the reality which the regulators failed to anticipate.

    Of course this misses a bigger point that tracking in the web moved in a direction that requires no cookies whatsoever, and if anything, feels more pervasive than it ever was.

    And it misses the even bigger point, that the morons who legislated cookie consent did not notice either of those two realities. And the same thing is already true with the AI act; the text watermarking is trivially defeated and everyone knows it. And I'd bet it will remain a requirement for the next decade or three.

    • vrganj

      today at 6:28 AM

      What company compliance teams do is once again, the responsibility of companies, not the EU.

      And the move away from cookies was addressed. That's why we have the GDPR now.

      We can always write new regulation, but its important to handle things as they come up and not use technological change as an excuse to do nothing.

Gruber is a smart, thoughtful man. This is a bizarre take from him.

It exhibits an undeveloped understanding of LLMs, and a righteous view that generated prose should assimilate... which should be offensive to organic intelligence.

Issues with the proprietary nature of Anthropic's watermarking aside, we will look back on this as a 'thank god' moment in the history of LLMs.

While I like that LLMs won't be able to produce the entire internet anymore, I am worried that the real reason for this move is for Anthropic to claim everything is theirs even though they STOLE humanity's collective knowledge including billions in private property worth of knowledge (or maybe even trillions), and all they do is regurgitate it, but now with a watermark on top as if it was theirs.

Are we now going to live in a world in which people bitch and moan about large private corporations' LLM text generation service and whether it's good or bad etc.? As though they're supposed to be benevolent and serve the public interest? They're not and they don't. Also, write your own damn text.

> "My error was believing Anthropic"

The error is rearranging part of one's life around Anthropic.

Best writer I’ve ever seen not understand writing. Proof positive that you needn’t understand something to produce lots of it!

Awwww, little blogger can’t write his slop anymore without rest of the world knowing that he’s not the author?

"possibly build tools to take a chunk of writing and try to remove your fingerprint from it, i.e, protect anonymity,"

The standard procedure to do this, is to chain translations to other languages and back. The message remains, but the wording will pick up some noise. --Dec 30, 2008.

This is the obvious solution to watermarking text in a way that isn't easily removed by a dumb tool.

The output quality will likely suffer as stated in the article although this can be mitigated to an extent by only enabling it on more irrelevant filler text while leaving the more functional sections untouched.

The solution using Unicode tricks amounts to malicious compliance as only the most unsophisticated users are going to fail to remove the AI watermarks when trying to pass off AI slop as their own prose.

The real solution here is not having stupid EU–tier laws in the first place.

Great, now Claude will sound even more generic. It’s not only annoying - it’s also dumb. We are all correct to push back on this load-bearing issue

So Anthropic goes on the shit list with Sony Blu-ray and its cinavia watermarking.

I'll be honest, who fucking cares? Why would you use AI to write for you and then complain that people know AI wrote the code?? If you know people wouldn't like it, why even try!?

Related:

How Claude's text watermarking works

https://news.ycombinator.com/item?id=49303350

gruber is really out of his element with ai commentary, I fully support the general skepticism but he's seemingly arguing against something he doesn't quite grasp

> the only acceptable answer for why an LLM should choose bananas instead of pineapple (or coconut, or guava, or papaya...) is that it has determined that it’s the best fit for the intended meaning, tone, and sentiment of the text.

It already fails. It randomly picks between close candidates. To help fool people into believing in intelligence claim, I guess.

Using AI is the only perversion here And what a self-reporting from all the AI bros!

is that why everything Claude writes lately sounds like a riddle?

... If watermarking is a perversion of writing, what does that make 'writing' with an LLM?

Tl;dr the author is upset that the slop generator will generate slightly different slop, and calls this perversion of the text, ironically missing the fact that the entire technology takes existing human work and perverts it in order to give him the output that he seemingly enjoys.

Perhaps it's not actually irony, perhaps it is hypocrisy.

This isn't to say there aren't correct statements in the article, but its framed very strangely.

I keep seeing an irritating misconception in this space, which is that the alternatives chosen by these algorithms are supposed to mean the same things as what they're displacing. That's not true, and not how LLM generation works. Complaints that two different choices don't mean the same thing miss the entire point.

I know (and understand why) a lot of people cheer the EU’s increasingly vast regulatory environment as being “pro consumer” but I’m really tired of said regulations being inflicted on the rest of the world. If this is what Europeans want for themselves that’s fine. But I no more want their regulations to be the de facto world’s any more than I want China’s.

Wow, never has a single article revealed the incompetency of a tech writer.

Now for the human generated watermark. Timez to addd the speelling mistakes, decreaze the usegage of big words and proper gramicaly usuage. Wish I was joking.

The objection seems to be that Claude will always write worse prose than a human writer, even if the writing Claude generates is understandable.

Yeah, John. We're all OK with that.

Wait are we supposed to be mad because clankers are displacing human creative workers, or mad because clankers don't do their level best when producing creative works because they are forced to watermark text? Or is it that they use all the water (I know they don't but are we supposed to be mad about it still)?

I can't keep up with the current Chinese psycop. There should be some kind of status page like whywewantamericatofailataitoday.ai so we can keep up with it.

No, machines being used to replace human writing is a perversion of writing. Some might call it worse than cannibalism[0]. If you're only noticing now because Anthropic is changing things behind your back, well... I've got some bad news for you, but the entire cloud-hosted subset of the AI space, especially Anthropic, is premised on the fact that doing things behind your back to their models is socially preferable, or worse, should be outright mandated.

While I generally hate legal mandates to stab your customers in the back, in my opinion there is no harmless way to use AI and mandatory text watermarking is a good bare minimum. The EU probably made the right call. The entire AI space - open models included - is predicated upon worker exploitation, replacement, and deskilling; we should at least be able to know how much of our media diet has Anthropic's fingerprints on it.

A lot of hay is made over the pretraining process in which copious amounts of stolen data are trained on; but parallel to this is a huge data labeling and human feedback operation staffed almost entirely by people in third-world countries with robust English as a Second Language (ESL) programs. The thing is, AI models already watermark their text, they just happen to do so with the textual watermarks of the Indians and Nigerians that the AI companies hired to do RLHF because they were cheap. That's why certain AI models love the word "delve" so damned much. It's neocolonialism, designed specifically to do the kind of replacement the anti-immigrant idiots keep screaming their heads off about[1].

Furthermore, as we've seen with Hank Green, even non-cannibalism-adjacent AI usage is a recipe for worker deskilling and AI psychosis. The other half of the RLHF pipeline is to turn a pile of compressed text into a chatbot that feeds you a steady drip of unsourced information while praising you every time you spot one of its lies and never saying no[2]. This is a recipe for addicting your customers.

Also, this might just be because this is on daringfireball.net, but I can't help but think the author has an axe to grind against the EU because the EU mandated Apple sign third-party app stores. The fact that he's balking at Anthropic not going along with gating the watermarks to just the EU seems downstream of this - "why aren't you maximally attempting to resist the EU?"

[0] https://www.youtube.com/watch?v=YCPAIg7RUq8

[1] To be clear, upwards of none of the far-right have actually clued into the fact that AI is trained by underpaid immigrants, mainly because it doesn't fit the narratives the people running the far-right want to push. There are some AI robotics companies that are even very explicit that their robots are there primarily to launder foreign labor into rich companies and make permanent labor arbitrage.

[2] Continuing on from [1], the far-right actually really loves AI specifically because it rarely says no to even stupid ideas, even if it's also a manifestation of everything they claim to hate.

Imagine using a random word generator and getting upset that the random words it generates are not truly artisanal random words.

Whole watermark thing is just bullshit. What if I add a watermark text and another AI as well. How many watermarks and who is the real creator. It just doesn't make sense. I ll eat my shoes if this concept is still a thing in 6 months

"Anthropic's . . . Claude is a Perversion of Writing."

FITFY.

I have no sympathy for writers whining about what the AI is doing to 'their' writing. It's only your writing when you write it. There's any easy way to avoid this: don't fucking use it. Use you own brain.

[flagged]

[flagged]

[flagged]

[flagged]

If precise word choice and nuanced phrasing are the core priorities, handing off the writing process to an autoregressive model in the first place defeats the purpose. Using LLMs as a sounding board or for structural review avoids watermark exposure entirely, it only becomes detectable when someone is copying wholesale blocks of model-generated text.

The objection isn’t that normal sampling is somehow pure. It’s that watermarking uses token choice to carry an additional, covert signal. The quality cost may be small on average, but it is still an optimization constraint unrelated to meaning or style—and it turns ordinary prose into provenance metadata. The more serious questions are detection reliability, false positives, and what happens after human editing. https://prachub.com/

The interesting part is that the detector doesn't need to identify a specific token choice. It can look for a small statistical skew across many choices. That also explains why the approach is fundamentally probabilistic: paraphrasing, translation, or human editing can dilute the signal without necessarily removing every trace of it.

[dead]

> because the nature of the watermarking algorithm requires it to sometimes increase the probability of selecting a worse word choice and decrease the probability of selecting the model’s best choice.

... and the opposite is also true, sometimes it will increase the probability of choosing the "best" word choice. So watermarking makes the LLM quality better then? /s

> My initial speculation was that maybe they’d hide invisible non-printing Unicode characters in the text.

Or just em dashes? /s

[dead]

[dead]

[flagged]

• ghomst

  last Sunday at 11:30 PM

  [flagged]

Anthropic should just pull out of the EU at this point. Europeans who really want to use it can VPN, and if they did they wouldn’t be able to hide behind their various comical tech laws.

Nanny state nonsense indeed.

As a non American/non European resident all I see from the Europeans are 0 contributions to software progress at any large scale while they surely do a lot of crying and huffing and puffing and demanding. Lots of complaining and rule making but not a lot of creating is a bad look.

• breezybottom

  yesterday at 1:17 AM

  Clearly Anthropic thinks its more profitable to comply and have access to the European market, but I'm sure you know better than the people who brought it to a $2 trillion valuation.

  • avazhi

    yesterday at 6:26 AM

    I didn't say anything about profitability. Since Anthropic is all about the good of mankind etc, ostensibly profitability isn't their number one priority.

• carlosrg

  yesterday at 9:58 AM

  Hi Gruber.

tl;dr:

> It’s unacceptable for a tool to sacrifice an iota of clarity, coherence, meaning, quality, etc. for the purpose of (watermarking)

And an example of the impact of watermarking on word choice [0]:

> The results of the study were quite [important || significant || substantial || notable]

The meaning of the sentence to changes slightly even in just this tiny example. Imagine the degradation when applied across an entire response!

[0] https://declaude.org/watermarking/

What is even the point of watermarking AI slop supposed to be? All it does is provide people with the false implication that anything which is not watermarked must not be AI-generated.

I struggle to see how this could possibly be useful unless there's some sort of psy-op going on to trick people into uncritically accepting anything lacking a watermark as not being AI-generated.

I recently found there was no website covering a historical subject, with sites begging to be visited. The information was scattered and not presented in the form people would find most useful (a map).

A couple of days later it was up with the most comprehensive review of the available info summarised and referenced ready for a human researcher to explore.

This was only possible due to AI. It would have taken me weeks to chase it down and summarise it, so it would never have happened. Let's get off our high horse about AI writing.

This (Anthropic's "watermark" stance, I mean) is so fundamentally ludicrous that I have assumed it is a (wholly insincere, but arguably pragmatic, at least from their perspective) attempt to deal with the EU and their latest misguided, ham-fisted attempt to solve a real-world problem by drenching the entire world with more regulatory slop[1].

The "watermark" can be trivially defeated, but may be enough to satisfy the letter of the law, and like many people here, I would argue that if you are letting Claude write for you, you've already accepted getting the literary equivalent of turd soup, so the harm is — or at least could be — fairly minuscule.

[1]: https://digital-strategy.ec.europa.eu/en/policies/code-pract...

(FWIW I have a more favorable view than most people seem to of the EU's efforts to at least try tackle problems like this — but predictably, the bureaucratic "solutions" they come up with don't work, but do make things objectively worse)

Respectfully, you are all missing the point.

Watermarking is bad not just because of the principled stance that your tool should not be working against your own interests (the passionate argument in TFA), but specifically because it lends credence to the idea that AI detection is a valid and possible thing to do perfectly.

As technologists of course we know "oh well yes but with some confidence interval we can detect AI token bias across a large corpus of text." To JimBob in charge of publishing your paper or reviewing your PhD submission, all he knows is "anthropic says AI detection is possible so this 30% chance your paper was written by AI means you've plagiarized." Do you really think you're winning the argument with the certified, law-approved plagiarism detection machine? No, you're not, and your career is over.

It's irresponsible to develop watermarking because it is not anywhere close to a perfect science, but it will be treated like one by people with the power to ruin your lives. Even if you've never touched AI in your life, your paper is going through the "maybe it says you cheated" box, and you better hope those dice don't come up snake eyes.

• otterley

  yesterday at 3:43 AM

  I suspect there’s a niche market for software that records your writing process to help you avoid being falsely accused of using AI to write.

• wasabi991011

  yesterday at 1:37 AM

  You should submit an article about this instead of having your point buried in a comment section of an article making an unrelated argument.