Add P2P distributed optimization to advanced examples

[francescofarina]

Description

This PR adds a new set of advanced examples in examples/advanced/distributed_optimization, showing how to use the lower-level APIs to build P2P distributed optimization algorithms.

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).

[francescofarina]

@chesterxgchen I implemented your suggested changes:

• I moved the implementation to app_opt - I changed the name of the module nvdo to p2p as the classes can potentially be used for arbitrary p2p algos, not just distributed optimization. Happy to change the name though
• added documentation to everything that's now in app_opt
• moved the SyncAlgorithmExecutor to a separate file and renamed the base.py files to base_p2p_executor.py for the executor and p2p_controller.py for the controller. The BaseP2PAlgorithmExecutor is now an ABC.
• removed pickle. For convenience right now all the executors are saving the results with pytorch.save, but that could be removed and easily reimplemented by the user if needed
• the dependencies in the examples/advanced/distributed_optimization are now specified in a requirements.txt as the core core has been moved to app_opt and can be imported directly from nvflare

Let me know what you think.

Now that it's moved to the core, I feel implementation part could be changed/improved by offloading to the user things like saving the results, storing losses via callbacks, monitoring, etc - perhaps it makes more sense to do that at a later stage though

[holgerroth]

Tested locally and runs fine. The tutorial is great! Should consider adding some CI testing in a future PR.

[chesterxgchen]

do we reset the neighbors_values once we have all of them for next round ?

[francescofarina]

We need to maintain values per iteration and delete the values after they're consumed in the algorithm. More details in examples/advanced/distributed_optimization/README.md

neighbors_values needs to maintain a dictionary of received values per iteration. This is because, different parts of a network may be at different iterations of the algorithm (plus or minus 1 at most) - this means that I could receive a message from a neighbor valid for iteration t+1 when I'm still at iteration t. Since that message won't be sent again, I need to store it. To avoid the neighbors_values to grow indefinitely, we'll delete its content at iteration t after having consumed its values and moving to the next iteration in the algorithm. We'll see that in the next section.

[chesterxgchen]

add a few more comments

[francescofarina]

add a few more comments

implemented renaming of dist opt controllers and executors + added synchronization timeout as parameter. I also re-run all the examples on GPU and works just fine.