tl;dr - This section of code is gnarly, I barely know how it works, and is in dire need of a refactor.
We should either provide a "max timeout" option where if an exception hasn't been resolved within x seconds then we fail with an unrecoverable exception. Or we provide a "max retry" option where we'll retry x times before failing with an unrecoverable exception.
Right now the code tries to use both and is a mess. Either way we should implement it using an exponential backoff formula in between retry attempts. Currently set to sleep for (x / 1.5)^2 seconds where x is the retry attempt count so far.
This issue will require a good understanding of threading as we're bucketing all the exceptions and making entire buckets sleep together.
tl;dr - This section of code is gnarly, I barely know how it works, and is in dire need of a refactor.
We should either provide a "max timeout" option where if an exception hasn't been resolved within x seconds then we fail with an unrecoverable exception. Or we provide a "max retry" option where we'll retry x times before failing with an unrecoverable exception.
Right now the code tries to use both and is a mess. Either way we should implement it using an exponential backoff formula in between retry attempts. Currently set to sleep for (x / 1.5)^2 seconds where x is the retry attempt count so far.
This issue will require a good understanding of threading as we're bucketing all the exceptions and making entire buckets sleep together.