Error creating system image on Mac with Apple M1

[rallen10]

I recently bought a new MacBook Pro with an M1 chip and I am working to transition my projects from my old mac to my new one.

On one of my projects I had need to create a custom system image because I use conda to manage my python environments. The following command worked fine on my old (non-M1) Mac

python3 -m julia.sysimage sys.dylib

but on my new M1-base Mac I am getting errors

/Library/Developer/CommandLineTools/usr/bin/python3: Error while finding module specification for 'julia-py.sysimage' (ModuleNotFoundError: No module named 'julia-py')
(ilqgames_jl) ro27711@582709-MITLL:~/Projects/MAST/ilqgames.jl  (master)
$ python3 -m julia.sysimage sys.dylib
  Activating new project at `/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.s26ma20l/compiler_env`
[ Info: Installing PackageCompiler...
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `/private/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.s26ma20l/compiler_env/Project.toml`
⌃ [9b87118b] + PackageCompiler v1.7.7
    Updating `/private/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.s26ma20l/compiler_env/Manifest.toml`
⌃ [9b87118b] + PackageCompiler v1.7.7
⌅ [05181044] + RelocatableFolders v0.1.3
  [6c6a2e73] + Scratch v1.1.1
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.8.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.0
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.2.1
  [83775a58] + Zlib_jll v1.2.12+3
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
        Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ cannot be upgraded. To see why use `status --outdated -m`
┌ Error: build.log for PackageCompiler [9b87118b-4619-50d2-8e1e-99f35a4d4d9d] not found
└ @ Main ~/Library/Python/3.8/lib/python/site-packages/julia/install-packagecompiler.jl:19
[ Info: Loading PackageCompiler...
[ Info: PackageCompiler is successfully installed at /var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.s26ma20l/compiler_env
Traceback (most recent call last):
  File "/Users/ro27711/Library/Python/3.8/bin/julia-py", line 8, in <module>
    sys.exit(main())
  File "/Users/ro27711/Library/Python/3.8/lib/python/site-packages/julia/julia_py.py", line 151, in main
    julia_py(**vars(parse_args(args, **kwargs)))
  File "/Users/ro27711/Library/Python/3.8/lib/python/site-packages/julia/julia_py.py", line 41, in julia_py
    api = LibJulia.from_juliainfo(juliainfo)
  File "/Users/ro27711/Library/Python/3.8/lib/python/site-packages/julia/libjulia.py", line 203, in from_juliainfo
    return cls(
  File "/Users/ro27711/Library/Python/3.8/lib/python/site-packages/julia/libjulia.py", line 226, in __init__
    self.libjulia = ctypes.PyDLL(libjulia_path, ctypes.RTLD_GLOBAL)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Applications/Julia-1.8.app/Contents/Resources/julia/lib/libjulia.1.8.dylib, 0x000A): tried: '/Applications/Julia-1.8.app/Contents/Resources/julia/lib/libjulia.1.8.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64)))
Command '['/Users/ro27711/Library/Python/3.8/bin/julia-py', '--julia', 'julia', '--compiled-modules=no', '--color=yes', '/Users/ro27711/Library/Python/3.8/lib/python/site-packages/julia/compile.jl', '/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.s26ma20l/compiler_env', '/Users/ro27711/Library/Python/3.8/lib/python/site-packages/julia/precompile.jl', '/Users/ro27711/Projects/MAST/ilqgames.jl/sys.dylib', '']' returned non-zero exit status 1.

Anyone run into something similar?

[mkitti]

How did you install julia?

[rallen10]

Downloaded binary from: https://julialang.org/downloads/

First tried installing via conda with conda install -c conda-forge julia but this resulted in a PackageNotFoundError. I was assuming this was because an Apple M1-compatible binary has not been added to conda forge...?

[mkitti]

I'm working on conda-forge julia on arm64, but conda-forge lacks the infrastructure to build julia from source at the moment. One hurdle is the openblas dependency here:
conda-forge/openblas-feedstock#146
There are other hurdles such as figuring out how to cross-compile the primary system image.

If you are using Julia with conda environments you may be interested in the activate script:
https://github.com/conda-forge/julia-feedstock/blob/main/recipe/scripts/activate.sh

An alternative would be to install juliaup (eg. conda install -c conda-forge juliaup), which then downloads the official binaries for you. You've already done this, so it might not make a difference.

With that out of the way, there seems to be some architecture confusion on your system. Is the python on your system running an arm64 native binary or are you trying to use rosetta for your stack?

I wonder if Julia got confused and built a native arm64 library while you were trying to use x86_64 emulation? Maybe JULIA_CPU_TARGET needs to be defined.

If you start the Julia REPL, what is the output of versioninfo()?

Can you start julia via julia -J /Users/ro27711/Projects/MAST/ilqgames.jl/sys.dylib?

[rallen10]

Thanks for the follow-up.

With that out of the way, there seems to be some architecture confusion on your system. Is the python on your system running an arm64 native binary or are you trying to use rosetta for your stack?

I have not touched anything to do with rosetta (not even familiar with what it is or does); so I assume python is running native arm64

If you start the Julia REPL, what is the output of versioninfo()?

Julia Version 1.8.0
Commit 5544a0fab76 (2022-08-17 13:38 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.3.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores

Can you start julia via julia -J /Users/ro27711/Projects/MAST/ilqgames.jl/sys.dylib?

No because I cannot generate the sys.dylib in the first place.

One detail I can give is that my python3 command was not pointing to my conda environment python; instead it was point to my systems "native" python3. If I run the following command (which correctly calls my conda environemt python), I get the resulting error:

python -m julia.sysimage sys.dylib
  Activating new project at `/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.abm1_w5b/compiler_env`
[ Info: Installing PackageCompiler...
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `/private/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.abm1_w5b/compiler_env/Project.toml`
⌃ [9b87118b] + PackageCompiler v1.7.7
    Updating `/private/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.abm1_w5b/compiler_env/Manifest.toml`
⌃ [9b87118b] + PackageCompiler v1.7.7
⌅ [05181044] + RelocatableFolders v0.1.3
  [6c6a2e73] + Scratch v1.1.1
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.8.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.0
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.2.1
  [83775a58] + Zlib_jll v1.2.12+3
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
        Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ cannot be upgraded. To see why use `status --outdated -m`
┌ Error: build.log for PackageCompiler [9b87118b-4619-50d2-8e1e-99f35a4d4d9d] not found
└ @ Main ~/miniconda3/envs/ilqgames_jl/lib/python3.10/site-packages/julia/install-packagecompiler.jl:19
[ Info: Loading PackageCompiler...
[ Info: PackageCompiler is successfully installed at /var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.abm1_w5b/compiler_env
  Activating project at `/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.abm1_w5b/compiler_env`
[ Info: Loading PackageCompiler...
[ Info: Installing PyCall...
  Activating new project at `/private/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.abm1_w5b`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
   Installed Parsers ─ v2.4.0
    Updating `/private/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.abm1_w5b/Project.toml`
  [438e738f] + PyCall v1.94.1
    Updating `/private/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.abm1_w5b/Manifest.toml`
  [8f4d0f93] + Conda v1.7.0
  [682c06a0] + JSON v0.21.3
  [1914dd2f] + MacroTools v0.5.9
  [69de0a69] + Parsers v2.4.0
  [438e738f] + PyCall v1.94.1
  [81def892] + VersionParsing v1.3.0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b27032c2] + LibCURL v0.6.3
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [de0858da] + Printf
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v0.5.2+0
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.2.1
  [4536629a] + OpenBLAS_jll v0.3.20+0
  [83775a58] + Zlib_jll v1.2.12+3
  [8e850b90] + libblastrampoline_jll v5.1.1+0
  [8e850ede] + nghttp2_jll v1.48.0+0
[ Info: Compiling a temporary system image without `PyCall`...
[ Info: PackageCompiler: creating system image object file, this might take a while...
ld: warning: ignoring file /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
ld: warning: ignoring file /Applications/Julia-1.8.app/Contents/Resources/julia/lib/libjulia.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
ld: warning: ignoring file /var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/jl_pOFcr0Zgf4.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
ld: in '/Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libunwind.dylib', building for macOS-x86_64 but attempting to link with file built for macOS-arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ERROR: LoadError: failed process: Process(`g++ -march=armv8-a+crypto+simd -shared -L/Applications/Julia-1.8.app/Contents/Resources/julia/lib -L/Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia -o /var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/jl_4FaGiU/sys.dylib -Wl,-all_load /var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/jl_pOFcr0Zgf4.o -ljulia-internal -ljulia -fPIC -Wl,-rpath,@executable_path -Wl,-rpath,@executable_path/julia`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run
    @ ./process.jl:477 [inlined]
  [4] run_compiler(cmd::Cmd; cplusplus::Bool)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/ghAiI/src/PackageCompiler.jl:145
  [5] create_sysimg_from_object_file(input_object::String, sysimage_path::String; o_init_file::Nothing, version::Nothing, compat_level::String, soname::Nothing)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/ghAiI/src/PackageCompiler.jl:702
  [6] create_sysimage(packages::Vector{Symbol}; sysimage_path::String, project::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, replace_default::Bool, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, base_sysimage::Nothing, isapp::Bool, julia_init_c_file::Nothing, version::Nothing, compat_level::String, soname::Nothing)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/ghAiI/src/PackageCompiler.jl:628
  [7] (::var"#1#2")(dir::String)
    @ Main ~/miniconda3/envs/ilqgames_jl/lib/python3.10/site-packages/julia/compile.jl:27
  [8] mktempdir(fn::var"#1#2", parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:764
  [9] mktempdir(fn::Function, parent::String) (repeats 2 times)
    @ Base.Filesystem ./file.jl:760
 [10] top-level scope
    @ ~/miniconda3/envs/ilqgames_jl/lib/python3.10/site-packages/julia/compile.jl:24
in expression starting at /Users/ro27711/miniconda3/envs/ilqgames_jl/lib/python3.10/site-packages/julia/compile.jl:23
Command '['/Users/ro27711/miniconda3/envs/ilqgames_jl/bin/julia-py', '--julia', 'julia', '--compiled-modules=no', '--color=yes', '/Users/ro27711/miniconda3/envs/ilqgames_jl/lib/python3.10/site-packages/julia/compile.jl', '/var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/tmp.pyjulia.sysimage.abm1_w5b/compiler_env', '/Users/ro27711/miniconda3/envs/ilqgames_jl/lib/python3.10/site-packages/julia/precompile.jl', '/Users/ro27711/Projects/MAST/ilqgames.jl/sys.dylib', '']' returned non-zero exit status 1.

[mkitti]

Ok, this is progress.

What happens when you run

g++ -march=armv8-a+crypto+simd -shared -L/Applications/Julia-1.8.app/Contents/Resources/julia/lib -L/Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia -o /var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/jl_4FaGiU/sys.dylib -Wl,-all_load /var/folders/d8/93mjk3sn5jv6sb8gjhm9gcbw0000gn/T/jl_pOFcr0Zgf4.o -ljulia-internal -ljulia -fPIC -Wl,-rpath,@executable_path -Wl,-rpath,@executable_path/Julia

[rallen10]

@mkitti Thank you for the help (I've been working on another project for a few days, so slow to respond). Do you mind helping me understand the above command before I run it? There is a lot going on there

Thank you

[mkitti]

I extracted it from your output above. Your output says the process exited with an error, but we need more details on the error to understand what went wrong.

Basically it invokes a C++ compiler.

[mkitti]

Is conda also running native arm or is under x86_64 emulation?