zk-ref watcher called with wrong values

[cch1]

I have created a zk-ref and added two watchers. One is added inline at created of the ref, the other is added subsequently using add-watch.

I then run tests by manually (from the REPL) changing the zk-ref from a remote machine.

Occasionally, I notice both watchers are called with the PREVIOUS state in the new (last) argument. Immediately after this problem, I verify that the distributed ref has the correct value on the same machine where the watcher was invoked incorrectly.

set-to/watcher-called-with/zk-ref-has
1/1/1
2/2/2
3/2/3

[cch1]

Further experimenting shows sometime the watchers are not called at all.

[cch1]

This problem was observed with avout 0.5.3 and ZooKeeper client (org.apache.zookeeper/zookeeper) 3.3.6.

[cch1]

...and the server reports its version as "Server environment:zookeeper.version=3.4.4-1386507, built on 09/17/2012 08:33 GMT"

[stuartsierra]

According to the ZooKeeper Watches documentation, ZooKeeper guarantees "A client will see a watch event for a znode it is watching before seeing the new data that corresponds to that znode." Furthermore, "Because watches are one time triggers and there is latency between getting the event and sending a new request to get a watch you cannot reliably see every change that happens to a node in ZooKeeper." (emphasis added)

Therefore, Avout's implementation of clojure.lang.IRef does not -- in fact, cannot -- fulfill the API contract of Clojure's add-watch, which says that watches run synchronously after every state change.

The ZooKeeper documentation leaves some ambiguity as to whether it is possible for a client to see the old value of a node in the callback triggered by a watch. According to Getting data after the watch (ZooKeeper mailing list), a client that calls getData in a watch callback will always see the new value.

In Avout, setting the root binding of avout.config/*use-cache* to false seems to reduce the likelihood of the watch function being called with an out-of-date value, but does not completely eliminate it.