Discussion: Flexible paths for binary dependencies generated in deps.jl files?

[NHDaly]

This is an open thread for discussing one of the big remaining problems with ApplicationBuilder/PackageCompiler: Handling binary dependencies.

The state of the world:

Many packages depend on external shared libraries, which will be referenced via ccall. Most of these packages handle the process of installing and locating these dependencies through BinDeps and friends. In these cases, they will have a build.jl that generates a deps.jl, such that the deps.jl defines const variables with hard-coded absolute paths to the libraries. e.g.:

# ~/.julia/v0.6/Gtk/deps/deps.jl:
...
@checked_lib libgtk "/Users/daly/.julia/v0.6/Homebrew/deps/usr/lib/libgtk-3.dylib"
@checked_lib libglib "/Users/daly/.julia/v0.6/Homebrew/deps/usr/opt/glib/lib/libglib-2.0.0.dylib"
...

In addition, some packages reference these libraries via ccall from their __init__ functions, which isn't overridable via the hacks we're currently using in ApplicationBuilder (eg in examples/blink.jl).

---

Problem statement:
Compiled and bundled apps need to copy these shared libraries into their bundle directory, so that they're self contained, and all code that references them must reference them via relative paths to their in-bundle locations. (The paths must be relative because the entire app bundle can be relocated after compilation.) There cannot be any references to the original installed Pkg directories evaluated while compiling, because the path provided to ccall is compiled-in, rather than reevaluated at runtime.

---

Potential solution ideas:

1. Super simple/dumb: make build.jl aware of the ENV variables ApplicationBuilder is using when compiling, and generate deps.jl files that conditionally use a simpler relative path if that ENV variable is true.

2. The newer deps.jl files generated by BinaryBuilder.jl generates variables whose paths are relative-ish: relative to the deps.jl file:

   # ~/.julia/v0.6/MbedTLS/deps/deps.jl:
   ...
   const libmbedcrypto = joinpath(dirname(@__FILE__), "usr/lib/libmbedcrypto.2.11.0.dylib")
   const libmbedtls = joinpath(dirname(@__FILE__), "usr/lib/libmbedtls.10.dylib")
   const libmbedx509 = joinpath(dirname(@__FILE__), "usr/lib/libmbedx509.0.dylib")
   ...

   So possibly, we can take advantage of this by somehow copying the deps.jl file into the bundle as well when compiling? As far as I know, though, this won't work because there's no way to copy only the deps file and trick a module into seeing that one when it runs include.
   EDIT: What I wrote above doesn't actually make sense, because EVEN IF we did all that, the resultant paths would STILL be absolute-paths: just absolutely pointing into the app bundle. But as soon as you move the app bundle anywhere, it breaks again. The ccalls MUST be providing a relative path relative to the application's cwd!

3. Add in a hook that you can somehow define that, when provided, will cause all variables to have their paths defined relative to some other path you provide (which could simply be "" in our case)... This is vague..

   1. Maybe we can do the above via something like @checked_lib, which somehow looks for your hook and then creates one definition or the other.

[NHDaly]

CC: @staticfloat

Weeeeeeeeeeeeeee here we go! XD

[NHDaly]

CC: @ranjanan

[NHDaly]

@staticfloat: After more reflection, I think that the solution will really have to be something that has the shape of 3.i above.

That is, rather than producing these global, unconditioned const definitions, i think they'll need to be created by something that's providing some option to override these to be relative paths.

For more clarity, we'd like the generated definitions to look something like this:

const libmbedcrypto = "libmbedcrypto.2.11.0.dylib")  # like this
const libmbedtls = "./libmbedtls.10.dylib"           # or like this
const libmbedx509 = "usr/lib/libmbedx509.0.dylib"    # or like this
const libxyz= "../libs/libxyz.dylib"                 # or like this

[staticfloat]

Just checking in on this again to note that Artifacts should satisfy many of the desires here, especially as you can now set JULIA_DEPOT_PATH in the environment (or use Pkg.Artifacts.with_artifacts_directory() in code) to set a global "root" under which all artifact files should be looked for.

[NHDaly]

Oh? That sounds really awesome! thanks @staticfloat!

My concern, though, is still about being able to set relative paths. I'm not entirely sure this will help. That is, if a user downloads an application bundle, they need to be able to move it around on their computer, and all of the compiled-in paths to binary files must still work.

Is that being addressed by the Artifacts design? Will setting one of those paths to a relative path be enough to get this to Just Work? I'm afraid that currently, in BinaryProvider, a lot of the generated paths are still absolute paths generated using @__DIR__.

CC:@KristofferC

[KristofferC]

The depot path is set relative to the absolute path of the executable (which is looked up at runtime). I'll have a proof of concept of it in PackageCompilerX in not too long.

[NHDaly]

Perfect, that's exactly what i was hoping to hear! :) Yes, that will work wonderfully. Thanks!

[KristofferC]

As an example JuliaLang/PackageCompilerX.jl#13.

Running PackageCompilerX.create_app("examples/MyApp/") in the repo creates the folder MyApp inside that folder with the file structure

❯ tree .
.
├── artifacts
│   └── 43563e7631a7eafae1f9f8d9d332e3de44ad7239
│       └── bin
│           └── socrates
├── bin
│   ├── MyApp
│   └── MyApp.so.
└── lib
    ├── julia
    │   ├── libamd.so -> libamd.so.2.4.6
... ... ...
    │   └── libz.so.1.2.11
    ├── libjulia.so -> libjulia.so.1.3
    ├── libjulia.so.1 -> libjulia.so.1.3
    └── libjulia.so.1.3

6 directories, 79 files

During runtime the app prints a bunch of stuff (including executing the artifact):

❯ bin/MyApp
ARGS = String[]
Base.PROGRAM_FILE = "bin/MyApp"
DEPOT_PATH = ["/home/kc/MyApp/MyApp/bin/MyApp/../"]
LOAD_PATH = ["@"]
pwd() = "/home/kc/MyApp/MyApp"
Base.active_project() = nothing
Sys.BINDIR = "/home/kc/MyApp/MyApp/bin"
[ Info: Running an artifact:
ἔοικα γοῦν τούτου γε σμικρῷ τινι αὐτῷ τούτῳ σοφώτερος εἶναι, ὅτι ἃ μὴ οἶδα οὐδὲ οἴομαι εἰδέναι.
unsafe_string((Base.JLOptions()).image_file) = "/home/kc/MyApp/MyApp/bin/MyApp.so"
Example.domath(5) = 10
sin(0.0) = 0.0

You can move the folder MyApp and the executable still works and, in particular, finds the bundled artifact.

[NHDaly]

That looks amazing. Thanks for picking this up, @KristofferC! :)