Potential Landmine With JuliaCall And JuliaPkg A Python And Julia Interoperability Issue

Hey guys,

There's a potential issue brewing in the world of Python packages that wrap Julia packages, and I wanted to bring it to your attention. It's not a major problem right now, but if Julia adoption takes off and more people start creating these wrappers, it could become a significant hurdle. Think of it as a hypothetical landmine – something that might explode down the road if we're not careful. If the number of Python Packages that wrap Julia Packages remains low, then this may never become a problem (and if it does come up, we could resolve the problem on a case-by-case).

Now, before we dive in, let me preface this by saying that I haven't fully tested everything, and I'm not an expert in Python packaging (or Julia packaging, for that matter). There's a chance I've misunderstood some aspects of JuliaCall/JuliaPkg's architecture. Plus, I haven't written any code to explicitly check for this issue. So, take everything with a grain of salt. I'm just trying to raise awareness and get the conversation started.

Overview: The Shared Julia Runtime and Dependency Resolution

The core of the problem lies in the combination of two key design choices made by JuliaCall/JuliaPkg. First, they aim for all Python packages built with JuliaCall/JuliaPkg to share a single Julia runtime. This sounds reasonable at first glance – why waste resources spinning up multiple runtimes, especially if the packages share common dependencies? However, this approach introduces complexity because you need to know all Julia dependencies upfront when the runtime is initialized.

Sharing a Julia runtime across multiple Python packages that wrap Julia code is a smart idea in theory. It saves memory and time, as you avoid the overhead of spinning up multiple runtimes. However, this approach hinges on knowing all the Julia dependencies in advance, which becomes tricky when you have multiple packages with potentially conflicting requirements. The beauty of this approach is to optimize resource utilization by avoiding redundant Julia runtimes, especially when packages share dependencies. Think of it like having a central hub for all Julia-related operations within your Python environment. However, this shared runtime model introduces complexities, particularly when it comes to managing dependencies.

The second key choice is that JuliaCall/JuliaPkg don't currently resolve Julia dependencies at build time. Instead, they handle dependency resolution at runtime. This means that the process of figuring out which Julia packages are needed and installing them happens when you actually run your Python code, rather than when you install the Python package itself. To be fair, achieving build-time dependency resolution within the Python packaging ecosystem is challenging. It might require writing plugins for different Python build systems (like setuptools, flit, hatch, etc.), which is a significant undertaking.

Resolving Julia dependencies at runtime, while offering flexibility, adds a layer of complexity. It means that the dependency resolution process happens when you import JuliaCall, rather than during the installation of the Python package. JuliaCall scans for juliapkg.json files in all possible installation paths, infers global package requirements, hunts for the appropriate Julia Runtime, and installs required packages. This approach is convenient, but it can lead to performance bottlenecks, especially in cluster environments with slow shared file systems. Resolving dependencies at runtime introduces a degree of uncertainty. You don't know for sure whether all the necessary Julia packages and their correct versions will be available until you actually run your code. This can lead to unexpected errors and compatibility issues, especially when dealing with multiple Python packages that wrap Julia code.

There's an existing issue on JuliaPkg's GitHub (Issue 16) discussing this very topic. The maintainer is concerned about the potential downsides of build-time resolution, such as preventing the distribution of wheels (pre-built packages that avoid arbitrary code execution during pip install) and requiring dependency resolution every time pip install is invoked, which could significantly slow down the installation process.

The maintainer's concerns are valid. Build-time dependency resolution does come with trade-offs. It might make distributing wheels more difficult and could increase the time it takes to install packages. However, in my opinion, this is the inherent cost of supporting a shared Julia runtime in a way that seamlessly scales to an arbitrary number of Python packages wrapping Julia packages. This scalability issue is directly tied to the problem I'll describe below. The scalability challenge becomes more pronounced as the number of Python packages that wrap Julia code increases. If each package has its own set of Julia dependencies, the shared runtime needs to be able to handle the combined requirements without conflicts. Build-time dependency resolution would help ensure compatibility upfront, preventing potential issues at runtime.

That said, I understand that the