Iâve never used them first hand, but crackpots sure do love claiming to solve Riemann hypothesis, P vs NP, Collatz conjecture etc and then peddle out some huge slop. My experience has solely been curiously following what the LLMâs have been generating.
You have to be very, VERY careful. With how predisposed they are to helping, theyâll turn to âdishonestyâ rather than just shut down and refuse. What I tend to see is they get backed into a corner, and theyâll do something like prove something different under the guise of another:
Theyâll create long pattern matching chains as to create labyrinths of state machines.
Theyâll keep naming functions, values and comments to seem plausible, but you have to follow these to make sure they are what they say. A sneaky little trick is to drop important parameters in functions, they appear in the call but not in the actual body.
Theyâll do something like taking a Complex value, but only working with the real projection, rounding a number, creatively making negatives not appear by abs etc etc
So even when it compiles, youâve got the burden of verifying everything is above board which is a pretty huge task.
And when it doesnât work, introducing an error or two in formal proof systems often means youâre getting exponentially further away from solving your problem.
Iâve not seen a convincing use that tactics or goals in the proof assistant themselves donât already provide