Blocking pops block the connection when the origin already has enough items

[prvyk]

Describe the bug

Example:

ZADD a 1 one
ZADD a 2 two
ZADD b 3 three
BZMPOP 0 2 a b MIN COUNT 2
-> redis-cli never returns.

The order in InitializeObserver() is likely wrong (what if a key is inserted after the loop checking it but before its observer being set up?), but that's not the cause here. The lock is actually freed?

This discussion may be related:
#1046

Steps to reproduce the bug

ZADD a 1 one
ZADD a 2 two
ZADD b 3 three
BZMPOP 0 2 a b MIN COUNT 2

Expected behavior

1. "a"
2. 1. 1. "one"
      2. "1"
   2. 1. "two"
      2. "2"

Screenshots

No response

Release version

v1.0.57

IDE

No response

OS version

No response

Additional context

No response

[prvyk]

Ok, I get it now. The 'block' isn't from a Garnet lock. It's because Garnet sends out a mismatched array and redis-cli waits on read. I kept thinking this is blocking API but the block is not from there. That's easy to fix **.

The race condition is a bit more complex (and maybe related to the discussion?).

I think the simplest (but slowish) solution would be to create the observer unconditionally (better make KeysToObservers a ConcurrentDictionary to be safest) and raise CollectionUpdatedEvent at the end. CollectionUpdatedEvent will call TryAssignItem which will use ObserverStatusLock to synchronize and line 295 will check for the item. At most another CollectionUpdatedEvent comes up earlier or later and locks serialize it.

The same idea of creating the observer first and locking so a concurrent CollectionUpdatedEvent is processed right can be done without the extra event, that's slightly more complex.

[Edit: No, the simplest would be to enter keysToObserversLock on the update event too, so it has to wait until InitializeObserver is done. The comment says it's for synchronization but doesn't use it on the other event. It's an extra lock though.]

**

diff --git a/libs/server/Resp/Objects/SortedSetCommands.cs b/libs/server/Resp/Objects/SortedSetCommands.cs
index d0c3626..63d2b2f 100644
--- a/libs/server/Resp/Objects/SortedSetCommands.cs
+++ b/libs/server/Resp/Objects/SortedSetCommands.cs
@@ -1685,7 +1685,7 @@ namespace Garnet.server
             while (!RespWriteUtils.TryWriteArrayLength(result.Items.Length, ref dcurr, dend))
                 SendAndReset();

-            for (var i = 0; i < result.Items.Length; i += 2)
+            for (var i = 0; i < result.Items.Length; ++i)
             {
                 while (!RespWriteUtils.TryWriteArrayLength(2, ref dcurr, dend))
                     SendAndReset();