App support in Pkg #3772

KristofferC · 2024-01-30T13:15:29Z

This is quite heavily WIP towards having "app" support in Pkg. An app is a program that you just write its name in the terminal and it starts up, without explicitly having to invoke Julia, load the package, and call a function. Every app has an isolated environment.

More details of the design can be found in this hackmd: https://hackmd.io/r0sgJar5SpGNomVB8wRP_Q

This PR requires JuliaLang/julia#52103

Here is some example usage:

(Pkg) pkg> app st

shell> rot13
zsh:1: command not found: rot13

(Pkg) pkg> app add https://github.com/KristofferC/Rot13.jl
    Updating git-repo `https://github.com/KristofferC/Rot13.jl`
  Activating project at `~/.julia/environments/apps/Rot13`
  No Changes to `~/.julia/environments/apps/Rot13/Project.toml`
  No Changes to `~/.julia/environments/apps/Rot13/Manifest.toml`

(@Rot13) pkg> app st
[43ef800a] Rot13 v0.1.0 `https://github.com/KristofferC/Rot13.jl#master`:  rot13 /home/kc/julia/usr/bin/julia 

shell> rot13 Rotate this please
Ebgngr
guvf
cyrnfr

(@Rot13) pkg> app rm rot13
[ Info: Deleted app rot13

(@Rot13) pkg> app st

cc @MasonProtter, @Roger-luo

tecosaur · 2024-01-30T14:32:02Z

Oh this looks very cool! Thanks for all the time/effort that's gone into this 🤩

One thing I'm slightly concerned about here is the approach taken to making sure that the executables are on the users's PATH on Linux/BSD systems.

There are many more shells than are hardcoded in get_shell_config_file, for instance I know multiple people using elvish, nushell (there was mention of this one on the Julia zulip too) as well as xonsh and oil to name a few. Also, isn't the fact we're relying on a hardcoded list a sign that there's something dodgy about this approach?
I've seen some shell configurations that have early-exit conditions, which mean that programmatically appending to the shell rc may not actually do anything
The hardcoded shell file paths aren't actually correct for the supported shells. For example, I can tell you that despite using zsh the current implementation would fail to work on my machine, since I have no ~/.zshrc. If the code changed to create a ~/.zshrc it would be ignored, since I have set my ZDOTDIR to ${XDG_CONFIG_HOME:-$HOME/.config}/zsh.
Setting the PATH in shell rc files doesn't affect non interactive shell sessions. If we want Julia apps to extend to say graphical apps, this is particularly relevant as usually the launcher process is a child of the desktop environment, and either does not load the shell rc file or only loads it once at login. For this, you also want to potentially modify the shell env/profile/login shells. However, changes to those configurations are only loaded at login, so you'd also need to get the user to log out and log back in again for it to take effect.
Somebody on the Julia Slack recently mentioned needing to give juliaup write permissions to their shell config in order to successfully install juliaup, and with one particular system configuration needing to broaden write permissions to all users.

I see in the design document there is some mention of putting such files in a more standard location already on the path such as ~/.local/bin, and that Cargo is mentioned in response to this. I think it's worth noting that there is a well-documented series of efforts (like this issue) to make Cargo more XDG-compliant (https://poignardazur.github.io/2023/05/23/platform-compliance-in-cargo/ does a good job outlining this, and describing a path forwards for Cargo). The Cargo discussion can essentially be summed up as "would have been good, but a bit late now".

Other lang's package managers already install things in the XDG-appropriate locations, such as Python with pip install --user (new/alternative Python package managers like poetry copy this behaviour).

I'd advocate for a ~/.local/bin approach on Linux/BSD for these reasons. To programmatically determine which executables in ~/.local/bin are managed by Julia, the executable files could be put inside a Julia-managed directory, and then symlinked to ~/.local/bin. I think this approach keeps much of the benefits of the custom-bindir added to PATH approach while avoiding the major pitfalls.

(NB: when I say ~/.local/bin I really mean ${JULIA_BIN_DIR:-${XDG_BIN_DIR:-$HOME/.local/bin}}, but that's a bit of a mouthful)

tecosaur · 2024-01-30T14:40:18Z

src/Apps/Apps.jl

+
+function bash_shim(pkgname, julia_executable_path::String, env)
+    return """
+        #!/usr/bin/env bash


On a briefer note, shouldn't the unix shim be based on sh not bash, possibly even #!/bin/sh over #!/usr/bin/env sh (somebody else should check, but IIRC that's the POSIX form)?

"Somebody else" is apparently me 😄, and my "IIRC" was wrong.

I've just had a look at https://pubs.opengroup.org/onlinepubs/009695399/utilities/sh.html and under APPLICATION USAGE there's this relevant excerpt:

Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH , ensuring that the returned pathname is an absolute pathname and not a shell built-in.

From reading around a bit more, it seems like the consensus on portability goes something like this (from most portable to least):

A binary executable that doesn't have a shebang

#!/bin/sh

#!/usr/bin/env sh

#!/usr/bin/env bash

#!/bin/bash

Bump @KristofferC

Roger-luo · 2024-01-31T10:59:23Z

nice work! I'm wondering how apps are shared across Julia versions? e.g. are they isolated by Julia versions like how the global environment are setup?

PallHaraldsson · 2024-01-31T12:13:28Z

intended to be run by the user as appname args to app. [..] It’s assumed that Julia is installed and serves as the “driver” to start up the app.

This seems useful, maybe already despite that limitation. Could it be lifted by autoinstalling Julia (runtime, right version) for you if not available? Needs not be in first version.

This is in some ways similar to Python's zipapps (which I believe is not too popular, because runtime can't be assumed, even for Linux where it's most often preinstalled), that needs separate .pyz[w] file ending, and Python installed (and are in one archive file, optionally compressed):

https://docs.python.org/3/library/zipapp.html

There is no way to say “python X.Y or later”, so be careful of using an exact version like “/usr/bin/env python3.4” as you will need to change your shebang line for users of Python 3.5, for example.

[We already have AppBundler.jl if you want to bundle the runtime, it's best if you can have one way to make an app and it can be compiled with PackageCompiler, or use AppBundler, or a combining those..., or this system. ]

KristofferC · 2024-01-31T12:29:56Z

One thing I'm slightly concerned about here is the approach taken to making sure that the executables are on the users's PATH on Linux/BSD systems.

With regards to XDG there is an argument that Pkg should follow what Julia itself does. (As you are aware) there is JuliaLang/julia#4630. juliaup also uses this method of installing Julia and since juliaup is more or less the official way to install Julia it feels like if you have managed to install Julia itself, this should be fine. So there is a tension here between doing XDG (which some people would argue is the correct way) and to fit in how things are done everywhere else in Julia and its ecosystem.

A related question, according to XDG where should the .julia/environments/apps/Package folder go?

For Windows the Cargo issue comment says:

For Windows, everything should go in ~/appdata/locallow or ~/appdata/local,since ~/.cargo is just a cache, AFAICT. This is FOLDERID_LocalAppData for SHGetKnownFolderPath, CSIDL_LOCAL_APPDATA for SHGetFolderPath, and %LOCALAPPDATA% in the environment.

How is that translated to all the files used here (shims, AppManifest.toml, app environments)?

Other lang's package managers already install things in the XDG-appropriate locations, such as Python with pip install --user

I get

❯ pip install --user httpie                       
Requirement already satisfied: httpie in /Users/kristoffercarlsson/Library/Python/3.9/lib/python/site-packages (3.2.2)

~/Library/Python/3.9/bin
❯ ls
git-filter-repo  http  httpie  https  markdown-it  pygmentize

KristofferC · 2024-01-31T12:32:42Z

nice work! I'm wondering how apps are shared across Julia versions? e.g. are they isolated by Julia versions like how the global environment are setup?

As it is right now each app entry in AppManifest.toml has an absolute path to a Julia installation. If you want to update that Julia version you would also resolve the environment. This ties into this later comment:

Could it be lifted by autoinstalling Julia (runtime, right version) for you if not available? Needs not be in first version.

one plan forward is to use Juliaup to install the Julia installation that the app is currently configured for if it does not exist. That way you would not store the absolute path to the julia installation like that.

tecosaur · 2024-01-31T13:40:55Z

With regards to XDG there is an argument that Pkg should follow what Julia itself does.

Right. I basically see Julia as currently being in a similar situation to Cargo — in that by the end of JuliaLang/julia#4630 I think I can fairly summarise the consensus as "yes this would be nice to have, but it's going to be a hassle to start using it".

Much of the value of the XDG Desktop spec comes via a network effect. Thus when the Desktop spec was new and that issue was created in 2013, the benefit was somewhat speculative. Now though, as more tools use and assume XDG compliance, it creates a growing tension between the "Julia way" and the XDG way.

In this sort of light, I see decisions like this as opportunities to choose between digging down and digging out 😛 somewhat. I still have loose plans to go back to JuliaLang/julia#4630 to see if I can help move the state of affairs closer to XDG compliance (Stefan asked me if I'd be interested in putting a PR together a few months ago, and I am once I have fewer PRs currently open).

Considering the current "Julia way" and the XDG spec, would it not be possible to put things in ~/.julia/bin as the "Julia-managed directory" that executables are written into, and make symlinks into ~/.local/bin? I might well be missing something, but it seems to me that this way the current assumptions around ~/.julia/bin hold but we also get the benefits of using the XDG-appropriate dir as outlined in my first comment.

A related question, according to XDG where should the .julia/environments/apps/Package folder go?

I made a flowchart for answering this sort of question in the BaseDirs.jl docs which might be helpful (it's not 100% accurate, but I didn't want to make it more complicated, and I think it gets 98% of the way).

If we classify .julia/environments/apps/Package as:

user-specific data
not something explicitly configured by the user
unable to be deleted without causing disruption to the system behaviour

then Data Home would be the relevant XDG Desktop component (let me know if any of those assumptions don't hold).

More generally, I find .julia/environments/ a bit interesting in that it's a mix of automatically-changed and user-modified environments. The v1.x environments are changed when the user explicitly asks for a package to be installed/removed, and so line up best as "user configuration". However, you also have environments like __pluto_boot_v2_1.8.5 which are very much not, and probably best classed as user data.

For Windows the Cargo issue comment says:

For Windows, everything should go in ~/appdata/locallow or ~/appdata/local, since ~/.cargo is just a cache, AFAICT. This is FOLDERID_LocalAppData for SHGetKnownFolderPath, CSIDL_LOCAL_APPDATA for SHGetFolderPath, and %LOCALAPPDATA% in the environment.

How is that translated to all the files used here (shims, AppManifest.toml, app environments)?

A while ago I spent an inordinate amount of time looking at the relevant behaviour/specs/comments around directories on Windows/Mac. I think I'd probably be best off pointing you to the comparison table on https://tecosaur.github.io/BaseDirs.jl/stable/defaults/ (and if you want the reasoning/links to some of the most relevant resources: https://tecosaur.github.io/BaseDirs.jl/stable/others/).

Regarding just this part of the comment:

This is FOLDERID_LocalAppData for SHGetKnownFolderPath, CSIDL_LOCAL_APPDATA for SHGetFolderPath, and %LOCALAPPDATA% in the environment.

Yea, getting the right system dirs on windows is actually a bit of a pain. See https://github.com/tecosaur/BaseDirs.jl/blob/main/src/nt.jl for a glimpse of me not having a fun time.

davidanthoff · 2024-02-16T20:32:30Z

one plan forward is to use Juliaup to install the Julia installation that the app is currently configured for if it does not exist. That way you would not store the absolute path to the julia installation like that.

My plan generally is that the Julia version in a manifest becomes the version selector for Juliaup. Presumably that would work well for apps here too?

ufechner7 · 2024-05-04T11:56:24Z

What is still needed before this can be merged?

kescobo · 2024-07-25T15:03:30Z

What is still needed before this can be merged?

In case folks haven't seen it - @KristofferC's talk from JuliaCon has a nice summary of the current status and what the open questions still are (or what they were as of a couple of weeks ago. Start at about 6:49:00 here: https://www.youtube.com/live/OQnHyHgs0Qo?si=IVg01oXigQw1JBDH&t=24545

JBlaschke · 2024-07-27T04:57:38Z

Hi Folks, I wanted to weigh in from the perspective of HPC.

If I understand this PR correctly, then the strategy chosen is to control the user environment in such a way that Julia code, Pkg environment, and default entrypoints emulate a user experience similar to a compiled executable.

This is like the approaches taken by Python zipfiles, anaconda, etc. Our experiences in running HPC systems (serving up to 10k users) so far has shown that this approach is:

detrimental to seamless user experiences: many HPC systems are sufficiently different from laptops and workstations, often resulting in mishaps due to conflicting approaches to environment configuration;
and scale very poorly when a large number of nodes are loading many small files: XDG, $HOME, and friends tend to live on shared file systems which essentially serialize metadata I/O on application launch.

Basically: we are developing HPC-native container runtimes precisely because the approach chosen in this PR performs poorly for Python. The irony here is that this considerable engineering effort is only necessary because Python can't generate compiled code.

Therefore, I think that the motivation behind this PR -- while well intentioned -- might run a real risk at being harmful to Julia as a High-Productivity HPC language. Especially because efforts to build executable applications appears to be within reach for Julia: JuliaLang/julia#55047. Furthermore, since JIT compilation adds complexity to the container build process, this should prove to be a much more seamless and scalable solution to building Julia applications than Pkg Apps.

Also, I think an approach to Julia applications that is based on compiled executables (which could be placed in any reasonable location on the filesystem) would result in a better user experience (including for non-HPC users). When developing tools to be used by others, I have opted for compiled executables as they don't rely on the user's runtime environment. This PR implicitly promises to support every edge case which a user could configure into their favorite shell, so from a mere user support perspective I think a combination of JuliaLang/julia#55047 + a distribution mechanism would be much easier to maintain.

Let me know what you think. I am happy to contribute some of my time to this.

Citing @Seelengrab , @giordano, and @tecosaur : we had a conversation on Slack that brought this to my attention (this does not imply that they share or endorse my opinion)

kescobo · 2024-07-28T14:00:33Z

If I understand this PR correctly, then the strategy chosen is to control the user environment in such a way that Julia code, Pkg environment, and default entrypoints emulate a user experience similar to a compiled executable.

This is like the approaches taken by Python zipfiles, anaconda, etc. Our experiences in running HPC systems...

First, let me say that I agree with many of the limitations that you mention, and as someone that also works on HPCs (though not at the same level), I'm glad to have people thinking about this stuff.

At the same time, I don't think we want to let the perfect be the enemy of the good here. Julia already has a lot of advantages over python when it comes to platform independence (eg binary builder), and this PR as it stands has functionality that will be extremely useful in many contexts, even if it's not perfect for HPCs at the moment, requires a bit of extra work for users to modify their own paths, etc.

I agree that we should not rely on Julia managing path stuff, but IIUC, this PR explicitly doesn't do that - it puts stuff in a julia-managed directory and relies on users to deal with it from there. This is how cargo does it too, and I've been able to use lots of those programs on my HPC.

I agree we want to move towards a place where Pkg can build binaries and seamlessly integrate them into the system environment, but I think that can be built on top of this, and I for one do not want to wait for that ideal state to get access to this functionality.

jpsamaroo · 2024-07-28T16:45:33Z

Another counter-point to JuliaLang/julia#55047 as an alternative viable solution: not all Julia programs can be made free from dynamic dispatch, yet JuliaLang/julia#55047 requires that for the programs it generates. In particular, if that were our only application deployment solution, then any program which uses Dagger.jl (which contains a dynamic-dispatch based core) would not be able to be deployed as an application, and thus users would be driven away from using Dagger in their applications if it meant that they lost application support by using Dagger. That would be a pretty harmful force to have exist within our ecosystem, as dynamic dispatch serves a very useful purpose, especially when used with care to ensure it doesn't result in unnecessary slowdowns in fast-paths.

To make things more concrete, could concerned HPC developers try out this PR on their system of interest and see where any issues arise? In particular, can we identify any currently-existing pain points in the current implementation that would make it hard for application/package authors to adopt this feature in their application? That would ground the discussion around things that we can clearly identify as issues to be resolved in some way, rather than trying to throw out this idea in its entirety on the basis of known unknowns.

Apps only write a manifest so can't rely on the path being created by `write_project`. Also moves the `mkpath` call for `write_project` to the method that actually does the filesystem writing with `write`.

KristofferC · 2025-01-16T15:01:23Z

This has some rudimentary docs and tests now. I'd prefer merging this soonish to get into 1.12 and the test coverage can be improved based on that.

fredrikekre · 2025-01-16T16:11:05Z

docs/src/apps.md

+    - You need to manually make `~/.julia/bin` available on the PATH environment.
+    - The path to the julia executable used is the same as the one used to install the app. If this
+      julia installation gets removed, you might need to reinstall the app.
+      used by the app might not be found.


Suggested change

used by the app might not be found.

fredrikekre · 2025-01-16T16:11:19Z

docs/src/apps.md

+    - The path to the julia executable used is the same as the one used to install the app. If this
+      julia installation gets removed, you might need to reinstall the app.
+      used by the app might not be found.
+    - You can only have one app installed


One per package?

kescobo

This is very exciting! Would be amazing to have this in 1.12, thanks for your work on it @KristofferC !

kescobo · 2025-01-17T15:06:55Z

docs/src/apps.md

+
+A Julia app is structured similar to a standard Julia library with the following additions:
+
+- A `@main` entry point in the package module (see the Julia help on `@main` for details)


Can this be [`@main`](@ref), or can this just be a link to that section in the docs?

kescobo · 2025-01-17T15:08:38Z

docs/src/apps.md

+
+## Installing Julia apps
+
+The installation of Julia apps are similar to installing julia libraries but instead of using e.g. `Pkg.add` or `pkg> add` one uses `Pkg.Apps.add` or `pkg> app add` (`develop` is also available).


Suggested change

The installation of Julia apps are similar to installing julia libraries but instead of using e.g. `Pkg.add` or `pkg> add` one uses `Pkg.Apps.add` or `pkg> app add` (`develop` is also available).

The installation of Julia apps are similar to installing julia libraries but instead of using e.g. `Pkg.add` or `pkg> add` one uses `Pkg.Apps.add` or `pkg> app add` (`pkg> app develop` is also available).

Or "develop rather than add is also..."

KristofferC mentioned this pull request Jan 30, 2024

Introduce -m/--module flag to execute a main function in a package JuliaLang/julia#52103

Merged

tecosaur reviewed Jan 30, 2024

View reviewed changes

KristofferC mentioned this pull request Feb 9, 2024

Shortcut to test a local package JuliaLang/julia#53258

Closed

mkitti mentioned this pull request Feb 15, 2024

Fail gracefully when shell configuration is not accessible JuliaLang/juliaup#840

Open

tecosaur mentioned this pull request Feb 20, 2024

A path to XDG compliance JuliaLang/juliaup#844

Open

KristofferC requested a review from a team as a code owner July 5, 2024 14:21

DilumAluthge removed the request for review from a team July 6, 2024 02:42

KristofferC force-pushed the kc/apps branch from af34836 to 19c83fd Compare July 25, 2024 11:32

This comment was marked as off-topic.

Sign in to view

Roger-luo mentioned this pull request Oct 12, 2024

New package: QuantDynSim v0.1.0 JuliaRegistries/General#117132

Closed

fredrikekre force-pushed the kc/apps branch from 01f3e3e to 75dcdef Compare November 7, 2024 15:17

fredrikekre previously approved these changes Nov 7, 2024

View reviewed changes

KristofferC dismissed fredrikekre’s stale review via 0374c81 November 15, 2024 14:38

fredrikekre mentioned this pull request Dec 6, 2024

Support range formatting fredrikekre/Runic.jl#120

Merged

KristofferC and others added 13 commits January 8, 2025 08:55

wip on app support in Pkg

dfa357c

also symlink app to .local/bin

289ab99

wip on app support in Pkg

7f50912

fix from REPL extension

e6a0c8d

fixes

4783717

wip

d3a583a

wip

0f18e31

update the docs a bit

f97e59f

update

5eb96eb

use manifest if it exists

ca4994f

contract status

3f78234

Run mkpath also in write_manifest

abdae83

Apps only write a manifest so can't rely on the path being created by `write_project`. Also moves the `mkpath` call for `write_project` to the method that actually does the filesystem writing with `write`.

small stuff

949d4c0

KristofferC force-pushed the kc/apps branch from 0374c81 to 949d4c0 Compare January 8, 2025 07:55

KristofferC and others added 2 commits January 8, 2025 09:21

remove PATH handling

2d6d142

add tests and some refactorings

4ff2dae

KristofferC changed the title ~~WIP: App support in Pkg~~ App support in Pkg Jan 16, 2025

actually run tests

040bacb

fredrikekre previously approved these changes Jan 16, 2025

View reviewed changes

fredrikekre mentioned this pull request Jan 16, 2025

Add app configuration fredrikekre/Runic.jl#129

Open

kescobo reviewed Jan 17, 2025

View reviewed changes

KristofferC added 2 commits January 20, 2025 11:46

fixup tests

f787f5b

fixup from review

547374b

KristofferC dismissed fredrikekre’s stale review via 547374b January 20, 2025 10:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

App support in Pkg #3772

App support in Pkg #3772

KristofferC commented Jan 30, 2024

tecosaur commented Jan 30, 2024 •

edited

Loading

tecosaur Jan 30, 2024

tecosaur Jan 30, 2024

tecosaur Jan 31, 2024

tecosaur Jan 16, 2025

Roger-luo commented Jan 31, 2024

PallHaraldsson commented Jan 31, 2024 •

edited

Loading

KristofferC commented Jan 31, 2024 •

edited

Loading

KristofferC commented Jan 31, 2024

tecosaur commented Jan 31, 2024

davidanthoff commented Feb 16, 2024

ufechner7 commented May 4, 2024

kescobo commented Jul 25, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

JBlaschke commented Jul 27, 2024 •

edited

Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

kescobo commented Jul 28, 2024

jpsamaroo commented Jul 28, 2024 •

edited

Loading

KristofferC commented Jan 16, 2025

fredrikekre Jan 16, 2025

fredrikekre Jan 16, 2025

kescobo left a comment

kescobo Jan 17, 2025

kescobo Jan 17, 2025


		A Julia app is structured similar to a standard Julia library with the following additions:

		- A `@main` entry point in the package module (see the Julia help on `@main` for details)


		## Installing Julia apps

		The installation of Julia apps are similar to installing julia libraries but instead of using e.g. `Pkg.add` or `pkg> add` one uses `Pkg.Apps.add` or `pkg> app add` (`develop` is also available).

App support in Pkg #3772

Are you sure you want to change the base?

App support in Pkg #3772

Conversation

KristofferC commented Jan 30, 2024

tecosaur commented Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roger-luo commented Jan 31, 2024

PallHaraldsson commented Jan 31, 2024 • edited Loading

KristofferC commented Jan 31, 2024 • edited Loading

KristofferC commented Jan 31, 2024

tecosaur commented Jan 31, 2024

davidanthoff commented Feb 16, 2024

ufechner7 commented May 4, 2024

kescobo commented Jul 25, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

JBlaschke commented Jul 27, 2024 • edited Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

kescobo commented Jul 28, 2024

jpsamaroo commented Jul 28, 2024 • edited Loading

KristofferC commented Jan 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kescobo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tecosaur commented Jan 30, 2024 •

edited

Loading

PallHaraldsson commented Jan 31, 2024 •

edited

Loading

KristofferC commented Jan 31, 2024 •

edited

Loading

JBlaschke commented Jul 27, 2024 •

edited

Loading

jpsamaroo commented Jul 28, 2024 •

edited

Loading