Tiled hacker news
👉🏻 About this project

Show HN: Retry a command with exponential backoff and jitter (+ Starlark exprs)

80 points - 11/15/2024

Source

  • stevekemp

    last Wednesday at 2:00 PM

    I have a collection of small sysadming/scripting utilities distributed as a single binary here:

    https://github.com/skx/sysbox

    One of those is "splay" to sleep a random amount of time, before running a command. Very useful to avoid lots of things running across a fleet at the same time.

      • networked

        last Wednesday at 4:46 PM

        I actually had SysBox starred, but `splay` wasn't on the list of alternatives. This has been fixed.

    • broken_broken_

      last Wednesday at 5:10 PM

      Hey, that’s funny, I wrote a blog post about the many ways you can implement such a program, and it was discussed on HN: https://news.ycombinator.com/item?id=42103200

      • Terretta

        last Wednesday at 1:03 PM

        For Python, consider Tenacity: https://tenacity.readthedocs.io/en/latest/

        At the CLI, this is nice for not depending on Node.

      • jstanley

        last Wednesday at 8:29 PM

        My view is that you basically never want exponential backoff.

        The only time exponential backoff is useful is if the failure is due to a rate limit and you specifically need a mechanism to reduce the rate at which you are attempting to use it.

        In the common case that the thing you're trying to talk is just down, exponential backoff with base N (e.g. wait 2x longer each time) increases your expected downtime by a factor of N (e.g. 2), because by the time your dependency is working again, you may be waiting up to the same amount of time again before you even retry it! Meanwhile, your service is down and your customers can't use it and your program is doing nothing but sleeping for another 30 minutes before it even checks to see if it can work.

        And for what? What is the downside to you if your program retries much more frequently?

        I much prefer setting a fixed time period to wait between retries (would you call that linear backoff? no backoff?), so for example if the thing fails you just sleep 1 second and try again, forever. And then your service is working again within 1 second of your dependency coming back up.

        If you really must use exponential backoff then pick a quite-low upper bound on how long you'll wait between retries. It is extremely frustrating to find out that something wasn't working just because it was sleeping for a long time because the previous handful of attempts failed.

          • dragonwriter

            last Wednesday at 9:57 PM

            > The only time exponential backoff is useful is if the failure is due to a rate limit and you specifically need a mechanism to reduce the rate at which you are attempting to use it.

            Exponential backoff is applicable to any failure where the time it has so far gone unresolved is the primary piece of available data on how lilong it is likely to take before being resolved, which is a very common situation, which is why it is a good default for most situations where you don’t have a better knowable-in-advance information at hand and the probability distribution ofn time to resolve, and where delays aren't super costly (though knowledge of when delays become costly can be used to set a cap on exponential backoff, too.)

              • jstanley

                yesterday at 12:08 AM

                > and where delays aren't super costly

                This is key.

                If delays aren't costly, sure, any algorithm is fine.

                But if you want your service up as soon as possible, why spend pointless minutes calling sleep() when you could be getting things working sooner?

                  • dragonwriter

                    yesterday at 2:38 AM

                    > If delays aren't costly, sure, any algorithm is fine.

                    Not true, delays aren't the only costs. Compute, network, and developer time digging through logs all cost money, and hammering a service fruitlessly when it is down adds to all of those.

                    (It also doesn't help that “system is overloaded” may sometimes be communicated clearly, but also in many systems can manifest in... just about any other kind of error, too.)

            • jperras

              last Wednesday at 9:06 PM

              > The only time exponential backoff is useful is if the failure is due to a rate limit and you specifically need a mechanism to reduce the rate at which you are attempting to use it.

              That's what you should be using exponential backoff for. In actuality, the new latency introduced by the backoff should be maintained for some time even after a successful request has been received, and gradually over time the interval reduced.

              > I much prefer setting a fixed time period to wait between retries (would you call that linear backoff? no backoff?)

              I've heard it referred to as truncated exponential backoff.

                • dragonwriter

                  last Wednesday at 9:59 PM

                  That's only truncated exponential backoff if you do exponential backoff to some point.

                  If its just a fixed retry interval, then its... a fixed retry interval.

              • Gasp0de

                last Wednesday at 11:04 PM

                Instead of using a fixed backoff, just use a token-bucket algorithm. Try a few requests every now and then and have each successful request enable another retry.

                • throwaway314155

                  last Wednesday at 9:46 PM

                  > basically never want exponential backoff.

                  > [unless] due to a rate limit

                  Pretty common use-case for automatic retries...

                    • jstanley

                      yesterday at 12:12 AM

                      That's fine!

                      If it's for a rate limit, sure, use exponential backoff, and put a known upper bound on how long you'll back off for.

                      But don't make your application wait for 50 minutes to come back up just because the database was down for 25 minutes. That's the kind of thing I am protesting here.

                  • mplewis

                    last Wednesday at 9:51 PM

                    I've used a fixed-time retry before. It DOSed the target server. Now, I use exponential backoff.

                      • thegrim33

                        last Wednesday at 10:06 PM

                        The jitter part is important too, to spread the retries out more in time.

                • itslennysfault

                  last Wednesday at 8:18 PM

                  If I had a nickel for every time I've written exponential backoff with jitter I'd have like several nickels.

                  • evgpbfhnr

                    last Wednesday at 12:08 PM

                    Looks similar to https://github.com/rye/eb

                  • iamjackg

                    last Wednesday at 3:54 PM

                    Very cool project! Just a suggestion: since you do have pre-built releases on GitHub, you should mention that in the Installation section of your readme.

                  • netvarun

                    last Wednesday at 5:26 PM

                    This looks cool - will give it a try (hah!) Curious on why you picked starlark instead of cel for the conditional scripting part?

                      • networked

                        last Wednesday at 9:24 PM

                        All right, let me tell you the history of recur to explain this choice. :-)

                        I wrote the initial version in Python in 2023 and used simpleeval [1] for the condition expressions. The readme for simpleeval states:

                        > I've done the best I can with this library - but there's no warranty, no guarantee, nada. A lot of very clever people think the whole idea of trying to sandbox CPython is impossible. Read the code yourself, and use it at your own risk.

                        In early 2024, simpleeval had a vulnerability report that drove the point home [2]. I wanted to switch to a safer expression library in the rare event someone passed untrusted arguments to `--condition` and as a matter of craft. (I like simpleeval, but I think it should be used for expressions in trusted or semi-trusted environments.)

                        First, I evaluated cel-python [3]. I didn't adopt it over the binary dependency on PyYAML and concerns about maturity. I also wished to avoid drastically changing the condition language.

                        Next in line was python-starlark-go [4], which I had only used in a project that ultimately didn't need complex config. I had been interested in Starlark for a while. It was an attractive alternative to conventional software configuration. I saw an opportunity to really try it.

                        A switch to python-starlark-go would have made platform-independent zipapps I built with shiv [5] no longer an option. This was when I realized I might as well port recur to Go, use starlark-go natively, and get static binaries out of it. I could have gone with cel-go, but like I said, I was interested in Starlark and wanted to keep expressions similar to how they were with simpleeval.

                        [1] https://github.com/danthedeckie/simpleeval

                        [2] https://github.com/danthedeckie/simpleeval/issues/138

                        [3] https://github.com/cloud-custodian/cel-python

                        [4] https://github.com/caketop/python-starlark-go

                        [5] https://github.com/linkedin/shiv#gotchas

                          • netvarun

                            last Wednesday at 11:02 PM

                            Wow that had been quite a journey - thanks for the detailed response. We’ve been using cel internally in a golang codebase and been pretty happy with it. I’ve only know about starlark in the Bazel context - I’ve learned a couple of things from your post. Thanks :)

                            • westurner

                              last Wednesday at 11:49 PM

                              There's not yet a Python implementation of Starlark (which, like Bazel, is a fork of Skylark FWIU)?

                              All that have tried to sandbox Python with Python have failed. E.g. RestrictedPython and RPython

                                • networked

                                  yesterday at 6:07 AM

                                  No, there is no pure-Python Starlark. So far there are only Python bindings for the Go and the Rust implementation: https://github.com/laurentlb/awesome-starlark#getting-starte.... I thought about porting the Go implementation to Python. Doing it as a subgoal for porting recur seemed a little like scope creep. (Tokei says there are 16792 SLOC in the latest d4d7611 commit of starlark-go and 899 in recur 67b38c1.)

                                  • morcus

                                    yesterday at 2:43 AM

                                    I believe Starlark was renamed Skylark, even internally. Bazel is a build system that uses Starlark as a configuration language, not a fork of Starlark.

                                      • westurner

                                        yesterday at 3:04 AM

                                        Bazel is an open source rewrite of Blaze (which introduced Skylark)

                        • greatgib

                          last Wednesday at 12:37 PM

                          Typically the kind of library that is useless and root cause of the dependency hell we are now living in.

                          That kind of simple things should be a basic inside once program or at worse a simple snipper copied from stack overflow or anything like that

                            • yoavm

                              last Wednesday at 12:52 PM

                              It does not seem to be a library at all, so very little to do with dependency hell. It's something you prepend to your commands if you want them to retry until they succeed. Seems pretty useful to me.

                            • crest

                              last Wednesday at 8:30 PM

                              A non-trivial application should not add a dependency for just exponential backoff + proportional jitter, an easy to use wrapper to put around a quick script is a good idea that and the lack of such a "basic" defensive programming technique has made untold initially small problems a lot worse by creating a thundering herd.

                          • westurner

                            last Wednesday at 11:50 PM

                            Systemd does exponential retry but IDK about jitter?

                          • whatthedangz

                            yesterday at 3:57 AM

                            [dead]