Steps to getting this cookbook published in the gallery

[jukent]

I am your Cookbook advocate, and my GitHub handle is @jukent. Please tag me in this issue with any problems getting your Cookbook published!
Once we've marked this entire checklist, click here to open an issue on ProjectPythia/cookbook-gallery to publish your Cookbook!

---

• Confirm you’ve followed the entire Project Pythia Cookbook Guide.
  Take note especially of the Develop your cookbook, Authorship and the CITATION.cff file, and Gallery tags sections. Save the Generate a DOI step as the last step of this checklist.
• Confirm that the individual notebooks within your Cookbook adhere to the notebook template.
  If the template does not fit your Cookbook’s needs, that’s fine too! Simply let us know here in this issue.
• Finalize your Cookbook repository name.
  We generally encourage the <content>-cookbook name structure.
• Finalize your environment.yml.
  Specify the minimum number of packages needed to reproduce your content. Document any necessary conflicts and pinned package versions in an issue. In your Cookbook README or a content preamble, describe any unique dependencies handled outside the conda environment.
• Sufficiently document your code with markdown narrative text, supplementary media and references, and citations.
  Declare any necessary prerequisite learning for each notebook at the top; these can be materials within your Cookbook, within other Cookbooks, or outside Project Pythia altogether.
• Review whether your Cookbook needs an Appendix of terms, definitions, or concepts.
  Additionally, should your Cookbook reference other Cookbooks and learning materials to support your content? Could supporting content be added as updates to Foundations or other Cookbooks benefit your Cookbook?
• Execute the trigger-replace-links action provided to your Cookbook.
  This will update any links to the Cookbook template to refer to your finalized repository name. See Links that need updating for a new cookbook created from this template cookbook-template#183 for manual references to these links if needed.
  • Click the Actions tab for your repository.
    
  • Highlight the trigger-replace-links action in the workflows sidebar.
    
  • On the right-hand side of the page, Run workflow > on Branch: Main.
    
• Fill in all template sections of your README.
  This will serve as your Cookbook homepage
  • Title
  • Cookbook description (brief, under title)
  • Cookbook Motivation - use this as an opportunity to tell us how your Cookbook fits in the broader learning ecosystem. Who should use this book? Why is it needed? Where does its content begin and end relative to existing resources?
  • Structure - this section is an optional roadmap for Cookbooks with more complicated structure. If you only have one main body of content that progresses linearly, you can leave this out.
• Confirm that your Cookbook is successfully building and publishing via GitHub Actions.
  This can be seen in individual Pull Requests as green checkmarks ✅ for important automation, especially the trigger-book-build action. You can also view a historical list of any of these Actions in the Actions tab at the top of your Cookbook repository. Check out nightly-build and trigger-book-build of PRs, then the build/build jobs to identify code errors. Please comment in this thread if you have issues identifying the source of any build and publishing failures your Cookbook has. Common failures include
  • Incorrectly specified environment.yml
  • trigger-link-check will fail if links in your content can not be resolved. We can help ignore links that are broken even if they work on manual clicks.
  • Code errors in your notebooks themselves
• Identify a Maintainer team via GitHub handle(s) in this thread.
  This can be one or more people with availability to check in on this Cookbook, issue fixes to broken content, or with a vision for the future development of the Cookbook. This is typically (but not necessarily) one of the primary authors of the Cookbook.
• Link your Cookbook repo to Zenodo for DOI generation
  Follow steps 1-4 under Generate a DOI in the Cookbook Guide. Return here for instructions on step 5, and your final step:
• Release your Cookbook!
  • On the right-hand sidebar for your Cookbook repository, click “Create a new release”. If you don’t see this button, you may need to click on the “Releases” header first and “Create” or “Draft” a new release.
    
  • “Choose a tag”, enter a new tag name. This will be the git reference of the snapshot of code that represents this particular release of your Cookbook! We recommend using a name fitting the CalVer scheme, so something like v2024.06.13 for the date of the release, then choose “+ Create new tag: on publish” and make its Target main (unless you have the knowledge and desire to release from another branch!)
    
  • From here you can use GitHub’s nifty “Generate release notes” button to automatically draft a summary of your Cookbook release based on merged Pull Requests! Feel free to further modify the title and body text of your release notes to fit your Cookbook and best represent your authors.
  • Finally, Publish release!

[MartinSchobben]

Hello @jukent. After some internal deliberations on the development, aims, and future maintenance, we will shortly begin addressing the requirements to get the book published. Thanks for your support!

[jukent]

@MartinSchobben thanks for the update. I'm happy to help with any steps of the process or sit down and discuss what type of work needs to be done (is it mostly content or infrastructure).

[MartinSchobben]

@jukent at the moment it is mainly about the content. We are now considering a structure which divides the book in to two units with multiple chapters: 1) course materials and 2) user instructions for some of our products; satellite-based soil moisture retrieval and flood monitoring. The former will include some of our BSc/Msc teaching materials. For example,these notebooks which we have created over the last months: https://tuw-geo.github.io/microwave-remote-sensing/. Loading this page highlights already a problem. If published as a webpage with output from the cells included, the size of some notebooks becomes excessive. So we need to figure out a way to deal with this. Also, we need to figure out how to deal with including example data in general. Although for some notebooks we can likely rely on the EODC STAC catalogue, who will also be a partner on the book.

[jukent]

Thanks for the update @MartinSchobben

Please let me know if there is anything I can help with. We'd like to showcase available cookbooks during Project Pythia talks at AGU24 and AMS25. If you think it is possible to have a working version published within ~6 weeks, we'd love to include your work on this eo-datascience-cookbook as well. Let me know if you plan to make that a goal.

[MartinSchobben]

@jukent we will aim to get a working version ready within the 6 weeks mark. We are happy to hear that you want to include our contribution in the upcoming talks!

[jukent]

Glad to hear it!

[MartinSchobben]

We finally have the initial content of the book ready. I now have already two questions:

1. Is it okay if we do not adhere to the notebook template for now, as, we use, for example, the course material in it is current format during the Msc courses at TU Wien. However the template's layout could still be something that we implement later on?
2. We have also separated the conda environment files in different sections to keep the environments more manageable. We have a Makefile that allows easy constructing of the Jupyter kernels. This also work in the GH actions workflow, as can be seen. Is this permitted as well?

The only issue with point 2 is caching of the environments during the GH actions runs

[jukent]

Hi @MartinSchobben

1. Yes that is fine. The template is there to support your work, not hold it back.
2. Again I think this is okay, and if it breaks due to the cacheing issue - we'll revisit then.

[MartinSchobben]

@jukent

Great! Then we have covered all tasks except for minting of the Zenodo DOI and creating a release, which would require transferring the repo to the Pythia organization.

[jukent]

@MartinSchobben are you ready to transfer? I think you have to initiate that - but I can take care of the Zenodo DOI and release for you.

[MartinSchobben]

@jukent I can't create repositories on ProjectPythia. Should I transfer to someone that has the rights to add content there?

[jukent]

Oh - I just invited you to the organization! Once you accept the invite you should be able to transfer the repository.

[MartinSchobben]

Thanks @jukent for the help! All done now.

[jukent]

@MartinSchobben there is a runtime error with your cookbook. Could you ping me once you address that? The cookbook should be building before we add it to the gallery :)

Also thank you for minting the DOI, my Zenodo account kept not synching.

[MartinSchobben]

@jukent, rather unfortunate, it first build: https://projectpythia.org/eo-datascience-cookbook/README.html However, now the STAC catalogue (element 84) changed and broke the classification notebook. @npikall do you have time to have a look at this?

[npikall]

I will see what I can do @MartinSchobben

[MartinSchobben]

@jukent @npikall The book seems to build again. See the nightly build. What I don't understand is why the binder is not triggered in the build book workflow? I can run the binder bot now but I have not had a successful run yet.

[jukent]

Huh it looks like it was working over the weekend (sorry I was sick Thurs and Fri), but after a39f4cc it no longer builds. The error seems to be with decoding a JSON file.

The PR to add the Cookbook to the gallery is here - ProjectPythia/cookbook-gallery#224

[MartinSchobben]

@jukent I fixed the remaining issues but I can't seem to build the book when the _config.yml parameter execute_notebooks is set to binder. The binder bot gets stuck in executing the notebooks and it is each time a different one. I also find it hard to debug, as the logs are not very informative. When the book is build with the parameter cache then everything is fine.

What I still do not understand is how to create the binder environment for the book. Alternatively I can create an repo2docker workflow. Perhaps we can host an image at the EODC in the future.

Never mind, I seem to have been confused. One can now also setup the binder by clicking on the links on the book. I will also make a new release of the book, including the fixes as discussed above.

[MartinSchobben]

@jukent I think we can close this now for real!