\

Lakegres: Vector search on existing Open/Hudi/Iceberg tables

18 points - today at 5:41 PM

Source
  • today at 6:25 PM

    • jauntywundrkind

      today at 6:29 PM

      Pretty good write up, talking to the problem, surveying what and how other players do things?

      Went looking to see if there was going to be anything open (mostly it's product pitch), found:

      > Iceberg support builds on the Hudi-backed indexing work in Apache XTable (Incubating). We developed the design in the open with engineers from Uber and Walmart to support their planet-scale tables. The open RFC documents the storage layout, query path and consistency model.

      > The design is settled and ships in Lakegres today; the Hudi implementation is being upstreamed and XTable is in release-candidate testing ahead of OSS releases that other engines can integrate.

      With two links inline, https://github.com/apache/incubator-xtable/pull/935 https://github.com/apache/hudi/blob/master/rfc/rfc-109/rfc-1...

        • gardnr

          today at 6:34 PM

          The problem doesn’t really seem to be solved though. Most attempts at decentralized index storage will be limited by what this project calls nprobes. The authors propose a cheap search first approach but it’s unclear how it scales.

            • cloud8bits

              today at 6:58 PM

              [flagged]

          • cloud8bits

            today at 6:48 PM

            [dead]

        • tokYoComPleX

          today at 5:50 PM

          This is a solidly built feature with great performance eclipsing other competing solutions on the market.

            • CBLT

              today at 6:20 PM

              Since you created an account just to compliment this product, it would probably be best to explain your relationship to the product in question. Even a simple "not affiliated, just a happy user" would be great.

          • jyasveer

            today at 5:43 PM

            great

            • cloud8bits

              today at 5:41 PM

              [flagged]