Fully automatic Hermetic / Portable bundles

[NHDaly]

The current system for creating portable, self-contained, downloadable binaries is quite fragile.

The main problem is about embedding all non-code dependencies needed by your code and its package dependencies. The main two types of such dependencies are libraries and resources.

The current steps to bundle everything up are quite manual. For resources, and libraries, the steps include the following:

• specify the path to the file, which gets copied into Libraries/ (or Resources/) (hopefully no name collisions!)
• Then you have to manually find all references to that file (hopefully stored in some variable, usually in the package's deps.jl file, and change them to be relative paths from the binary's path.
  • But only do this if you're running as a compiled binary! to tell this, we set an environment variable "COMPILING_APPLE_BUNDLE"=>true.
• And if those libraries have interdependencies, you need to make sure to update their rpaths to not point outside the binary, ...

It can be hard to even figure out which libraries to copy into the application, and this can change as your dependencies are updated. And then, even once you find them, it is very fragile to try to change the variables in the packages themselves to point to the new relative paths for the libraries/resources.

---

This Issue is to try to find a more robust, automatic, and complete solution.

Based on a conversation I had with @staticfloat at JuliaCon2018, and now with Pkg3, I think we can make this work by re-instantiating the entire .julia directory inside the application bundle before building (by simply pointing JULIA_DEPOT_PATH inside the application bundle and running Pkg.instantiate()).

The problem then is that the libraries will still be specified via absolute paths, but based on conversations with @rbvermaa, I think we can get around this via a pass over the directory tree that attempts to replace all such absolute paths with relative paths. Fortunately, most of these paths are generated via only a few tools (BinaryBuilder/BinaryProvider, BinDeps, and a few others), so they should be fairly regular. In the long term, perhaps we can even have an option or environment variable or something that can cause those tools to emit paths in the format we want directly.

This is the basic idea that I am going to try exploring over the next couple days:

• Instantiate an entire .julia tree (inside the App bundle) before compiling.
• Do a find/replace pass to replace all absolute paths within that tree to relative paths.
• Do the static compilation, pointing at that tree (via JULIA_DEPOT_PATH), so that all the relative paths are compiled into the binary
• Delete everything in the tree except the library and resource dependencies (deps/ directories) [at simplest, just delete the src/ and test/ dirs].
• Bundle and Ship.

There are likely to be some remaining problems, but maybe we can follow the wizard approach used in BinaryProvider to create a script based on an interactive session with the user that can target any irregular paths that need to be fixed, or detect them via some other automation (such as running the final binary, detecting dynamic loads outside the app, fixing them, rebuilding, and repeat).

The relative paths we use to replace the absolute paths for loading libraries will need to be relative to the executable path, so that the user can move the application bundle around after installing it on their system. On Macs, there is built-in support for this: Julia's Base.DL_LOAD_PATH loads relative to the executable load path via "@loader_path". On Linux, we'll probably have to push the path to the binary into DL_LOAD_PATH manually before building and as the first step of loading the binary before running julia_main() (which we can get from argv[0], and already set as PROGRAM_FILE in ApplicationBuilder).

[NHDaly]

This reminds me of an open question:

• Can we have static-compilation only deps? or some other kind of build-only deps? Right now, when I instantiate the project's Manifest, that means it also installs ApplicationBuilder and PackageCompiler inside the app bundle, and PackageCompiler takes a long time to build. And there's no reason for it because it's not used by the actual project. (I think.)
  • I mean in the same sort of way that there are test-only deps via the test [Targets]:

  [targets]
  test = ["Test"]
  

[NHDaly]

Ah, okay! I spent a while banging my head against this, and now i realize:

The things I described above do in fact work on the Mac! Hooray! But I was getting held up because of the isfile present in the generated check_deps() methods. For example:

function check_deps()
    global libduckdb
    if !isfile(libduckdb)
        error("$(libduckdb) does not exist, Please re-run Pkg.build(\"DuckDB\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libduckdb) in (C_NULL, nothing)
        error("$(libduckdb) cannot be opened, Please re-run Pkg.build(\"DuckDB\"), and restart Julia.")
    end

end

So even though the actual Libdl.dlopen_e line would succeed with a relative path relative to the binary (because of the DL_LOAD_PATH), the isfile check fails because it evaluates the relative path relative to pwd.

I think the easiest path forward will just be to cd() to the binary's directory in the C program, before doing julia_init (which inits all the modules, which load their dylibs). But maybe long term we can get BinaryProvider to emit code that somehow checks isfile in the context of DL_LOAD_PATH? Maybe something like Libdl.dlexists(), except that no such function exists right now.