Tiled hacker news
\

Show HN: Treepeat – Code similarity detection using Tree-sitter

65 points - last Tuesday at 4:53 PM


treepeat is a tool that finds similarities in your codebase.

Find duplicate code blocks meaningful to the language (classes/functions), not just lines.

Find near-duplicates: ignore whitespace, strings, high level AST nodes such as function and names.

Find structurally similar code: anonymize identifiers, constants, etc.

Pull requests welcome: This is very much an proof of concept - I'm happy with it, but I haven't supported very many languages at present.

Languages supported: astro, bash, css, go, html, javascript, lua, markdown (plus codeblocks), python, sql, typescript, java, kotlin, rust, yaml

Source

  • ryuuseijin

    last Thursday at 9:06 PM

    This is what I wanted to help with duplicate code detection, which is now a real problem with agentic programming since the agent tends to duplicate helper functions a lot in bigger codebases.

    I ended up writing my own tool [1] that uses vector search, which works but can be unusably slow in large codebases.

    I will give this a shot.

    [1]: https://github.com/ninjaxtools/slopdex

      • 91awebsi

        yesterday at 3:45 PM

        ooh I'll give yours a shot - thanks for calling my project out on your readme!

        I was fortunate to have a contributor push me to consider optimizations a few months back, and it definitely helped improve the speed/memory metrics.

        I see the CLI knobs are somewhat similar ('min lines', 'thresholds'), however your 'search' is fascinating. I could see myself wondering "how much error handling is duplicated for HTTP responses?" and your tool would give me a start at an answer. Nice!

    • bradleyy

      last Thursday at 8:00 PM

      Thanks for posting this; I'm very interested in finding similar chunks of code in an AST-ish fashion, and using tree-sitter seems perfect.

        • 91awebsi

          yesterday at 4:00 PM

          I keep meaning to write a post for how this has worked for my projects, I think its been helpful.

          For active projects I setup a weekly-ish 'code audit' agent: it runs treepeat to find opportunities to refactor and bumps up enforced coverage numbers. Over time for some projects I lower the '--similarity' threshold to more broadly find similar structured code (potentially more ambitious refactoring)

      • Game_Ender

        last Thursday at 10:18 PM

        Thank you for a README that is readable. It might have a bit of AI help but it’s concise and makes it clear how to use the tool.

          • anaqin

            yesterday at 12:36 PM

            I dunno… I’m finding myself yearning for some emojis, a dedicated marketing web site, and a bunch of incomprehensible paragraphs that lack any soul.

            • 91awebsi

              yesterday at 3:37 PM

              Yes, a little AI on the main readme. I started the bones of the thing, but Claude has been behind the keyboard for much of the implementation.

          • adityamishra241

            yesterday at 5:30 AM

            Nice approach. AST-level similarity seems much more useful than simple text matching.

            • last Tuesday at 4:53 PM