Some questions for IREE

[xushijie]

newbier for IREE, and still have a number of questions after reading many documents.

1. When using iree-compile to compile a model, eg., linalg version to a specified target-backends, what kinds of optimizations have been involved during the IR lowering? Is it possible to add customized optimization during this lowering, e.g., adding my own dialect as one pass and applying polyhedron/kernel fusion during lowering?
2. The compilation will result two kinds of codes, scheduling for the host and execution code for the device. At runtime, how do memoery management work and, how host and device communicate with other to maximize these overhead.
3. Having checked the c++template project, and any recommended documents or samples if want to implement my own runtime atop of IREE.

[ScottTodd]

1 (optimizations)

We have one list of optimizations here: https://iree.dev/reference/optimization-options/, but this is high level and incomplete. Another list of options is included on this issue: #19072, which is working towards adding groups of optional optimization options into -O style flags.

Note that IREE is a whole program compiler, so there are "optimizations" running at every level of lowering, from frontend program definitions, to program partitioning and device placement, to backend-specific lowering strategies (tile/distribute workgroups), to instruction and register selection (some of this is handled by backend targets like LLVM or GPU driver shader compilers).

To get a comprehensive list you could read the source code, compile a program with a debug option like --mlir-print-ir-after-all. You could also watch one of the presentations from the team (https://github.com/iree-org/iree?tab=readme-ov-file#presentations-and-talks, and some in other forums like https://www.youtube.com/@LLVMPROJ/videos).

As for adding custom optimizations, yes this is possible and there are many points where you could adjust behavior. Depending on where in the process you want to change things, the tools available will change too.

• At the frontend level, you can make algorithmic changes to the program itself or tag parts of programs for special handling in the compiler (e.g. custom ops). Projects like https://github.com/nod-ai/shark-ai/tree/main/sharktank and https://github.com/iree-org/iree-turbine operate at this level.
• Within the compiler and runtime we have some extension mechanisms documented here: https://iree.dev/reference/extensions/. We also have some mechanisms within each backend to run extra passes like with [GPU] Adding support for opt pass plugins during AMDGPU executable serialization #18347.
• Another approach useful for prototyping in the compiler is to run the compiler up to a certain point, dump IR, edit the IR manually or with a tool, then resume the compiler from that point onwards. This proceed is documented here: https://iree.dev/developers/general/developer-tips/#compiling-phase-by-phase.

2 (runtime memory management)

@benvanik could give a more detailed answer here, but core parts of program memory management are computed ahead of time in the compiler. We do have multiple allocators in the runtime that users can select between (e.g. a caching allocator)

3 (runtime code samples)

See https://iree.dev/reference/bindings/c-api/#samples and also https://github.com/nod-ai/shark-ai/tree/main/shortfin .

[xushijie]

That's a great reply, and will dive into each link soon.