Timeouts With Many Jobs

[jakemumu]

Hey! Loving this gem, but when I run a large batch of like 100k users, the statuses tab times out in my app, is there a fix for that somehow?

[Gooner91]

Yep, can confirm, the show view for individual statuses work but the index view results in gateway timeout.

[Andy9822]

I came across this issue recently and the gem was unusable in our Production environment due to the large number of jobs present in the Redis.

I quickly looked at the source code and suspected the way the Redis keys were accessed and processed was super inefficient. I tried a very simple alternative and this was my result:

P.S. zoo was the method using the gem's code for doing the Redis keys manipulation while foo was my fix attempt - which turned out to be 9x faster.
P.S.2 Ignore the jid logs, it was leftovers from my first test attempts

I don't know how maintained this gem currently is - I just left an issue here. Depending on it, I may take some time to give my idea a shot and open a PR to improve the performance so we avoid the timeout - or at least increase the size of jobs that the gem can process without causing a timeout