Notebook Interactivity

[ivirshup]

Splitting out the discussion on how we do interactivity of the notebooks.

There should be some sort of interactivity (i.e. the possibility to execute the notebooks online)
– @grst

Yes, please. We should design everything with this in mind. I think that we'll need GPU support though, right @adamgayoso ?
@Zethson

Originally posted by @Zethson in #1 (comment)

Platforms

AFAICT, the two ways we could do this are colab and binder.

I think we could try to do both. The issue here is "how do we do dependencies" since we can specify an environment for binder, but not for colab.

Maybe there is some way to specify dependencies in an environment.yml, then have colab use them?

Dependencies more generally

I don't think we should have one environment that every notebook runs in. How can we manage this?

I think it should be possible to have multiple conda environments created on a binder image. Then notebooks could specify which environment to use with their kernel.

@Zethson, does the best practices manage this in anyway? I'm trying to figure it out right now, but the binder is taking a really long time to build 😅

[grst]

Allegedly it is possible to make colab work with conda by installing /usr/local in the base environment:

• https://towardsdatascience.com/conda-google-colab-75f7c867a522
• or also this answer on stackoverflow.

[ivirshup]

Idk if those are trustworthy solutions. What if you install torch in your conda environment? Then aren't there two installations of torch on your python path? The one that comes with colab and the conda one?

This seems like a more robust solution from the RAPIDs (repo). Looking at the script they run, it's basically the same as the linked answers, but with a bunch of pip uninstalls first. It may also be pretty rapids specific, so I'm not sure if this approach can be used generically for other conda packages.

[adamgayoso]

As for scvi-tools, we can stick to binder if we decide to only load pre-trained models, which seems like a decent compromise to me. As even on GPU, waiting for training is not fun

[grst]

Then aren't there two installations of torch on your python path? The one that comes with colab and the conda one?

FWIW, it should be possible to manipulate the PYTHONPATH so that it only refers to conda packages.

[ivirshup]

Summarizing a couple discussions on hosting about this:

mybinder has a very small amount of resources. E.g. they promise 1gb of memory but may give you 2gb. This would be pretty restrictive for anything more than a single small-ish dataset. So integration, reference mapping, compositional analysis, etc could be difficult. @Zethson has concluded this means mybinder probably won't work for the best practices book, and they will need alternative hosting.

I then saw you can run a jupyterlab instance on galaxy and may even be able to get a GPU. This may be well timed since they are now working on tutorials for single cell. I think we should get in contact with them about this.

further discussion about galaxy on zulip

[grst]

So what's the conclusion here for the hackathon?

• Stick with binder until there's a binderhub on galaxy
• Additional support for colab nice to have but not required

[ivirshup]

I think so

Specifically about binderhub

I have my doubts about there every being a binder hub on galaxy, but I think we could have a "link to galaxy that will start a job". I think a binder hub could happen through denbi, but then we'd have to figure this for other parts of the world.

[grst]

Do you think geographic location matters that much? Running a notebook is neither particularly low-latency nor high-traffic...

[ivirshup]

Having used US or EU hosted services from Australia: it's very noticeable. It can take seconds to execute something very simple and connections can be dropped.

[grst]

In principle it works to launch mybinder from the tutorials with one click.
Should we still keep this open to track alternatives that provide more computational resources?

[Zethson]

Yes, I'd be super keen to get support for this. (I'd normally volunteer now but I've reached my limit).

[ivirshup]

👍, I'd be happy to say we're not going to follow this one up immediately, but leave it for later.

[ivirshup]

I'm gonna say that pushing this to a later date also entails putting off:

• Google colab integration #17
• Per-notebook environments #19