Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bug]: Docker ROCm failing to generate #7535

Open
1 task done
RSWilli opened this issue Jan 8, 2025 · 3 comments
Open
1 task done

[bug]: Docker ROCm failing to generate #7535

RSWilli opened this issue Jan 8, 2025 · 3 comments
Labels
bug Something isn't working

Comments

@RSWilli
Copy link

RSWilli commented Jan 8, 2025

Is there an existing issue for this problem?

  • I have searched the existing issues

Operating system

Linux

GPU vendor

AMD (ROCm)

GPU model

AMD Radeon RX 7800 XT

GPU VRAM

16GB

Version number

main branch

Browser

FF 133

Python dependencies

accelerate
1.0.1

compel
2.0.2

cuda
Not Installed

diffusers
0.31.0

numpy
1.26.3

opencv
4.9.0.80

onnx
1.16.1

pillow
10.2.0

python
3.11.10

torch
2.4.1+rocm6.1

torchvision
0.19.1+rocm6.1

transformers
4.46.3

xformers
Not Installed

What happened

I am using the docker run.sh to build a ROCm docker container on NixOS. Installation works but when I try to generate an image I get RuntimeError: HIP error: invalid device function.

What you expected to happen

It works :D

How to reproduce the problem

No response

Additional context

Attaching to invokeai-rocm-1
invokeai-rocm-1  | Could not load bitsandbytes native library: 'NoneType' object has no attribute 'split'
invokeai-rocm-1  | Traceback (most recent call last):
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/bitsandbytes/cextension.py", line 85, in <module>
invokeai-rocm-1  |     lib = get_native_library()
invokeai-rocm-1  |           ^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/bitsandbytes/cextension.py", line 64, in get_native_library
invokeai-rocm-1  |     cuda_specs = get_cuda_specs()
invokeai-rocm-1  |                  ^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/bitsandbytes/cuda_specs.py", line 39, in get_cuda_specs
invokeai-rocm-1  |     cuda_version_string=(get_cuda_version_string()),
invokeai-rocm-1  |                          ^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/bitsandbytes/cuda_specs.py", line 29, in get_cuda_version_string
invokeai-rocm-1  |     major, minor = get_cuda_version_tuple()
invokeai-rocm-1  |                    ^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/bitsandbytes/cuda_specs.py", line 24, in get_cuda_version_tuple
invokeai-rocm-1  |     major, minor = map(int, torch.version.cuda.split("."))
invokeai-rocm-1  |                             ^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  | AttributeError: 'NoneType' object has no attribute 'split'
invokeai-rocm-1  | 
invokeai-rocm-1  | CUDA Setup failed despite CUDA being available. Please run the following command to get more information:
invokeai-rocm-1  | 
invokeai-rocm-1  | python -m bitsandbytes
invokeai-rocm-1  | 
invokeai-rocm-1  | Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
invokeai-rocm-1  | to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
invokeai-rocm-1  | and open an issue at: https://github.com/bitsandbytes-foundation/bitsandbytes/issues
invokeai-rocm-1  | 
invokeai-rocm-1  | [2025-01-08 21:20:52,003]::[InvokeAI]::INFO --> Patchmatch initialized
invokeai-rocm-1  | [2025-01-08 21:20:52,572]::[InvokeAI]::INFO --> Using torch device: AMD Radeon RX 7800 XT
invokeai-rocm-1  | [2025-01-08 21:20:52,736]::[InvokeAI]::INFO --> cuDNN version: 3001000
invokeai-rocm-1  | [2025-01-08 21:20:52,748]::[InvokeAI]::INFO --> InvokeAI version 5.6.0rc1
invokeai-rocm-1  | [2025-01-08 21:20:52,748]::[InvokeAI]::INFO --> Root directory = /invokeai
invokeai-rocm-1  | [2025-01-08 21:20:52,748]::[InvokeAI]::INFO --> Initializing database at /invokeai/databases/invokeai.db
invokeai-rocm-1  | [2025-01-08 21:20:52,760]::[InvokeAI]::INFO --> Pruned 1 finished queue items
invokeai-rocm-1  | [2025-01-08 21:20:52,814]::[InvokeAI]::INFO --> Cleaned database (freed 0.04MB)
invokeai-rocm-1  | [2025-01-08 21:20:52,814]::[InvokeAI]::INFO --> Invoke running on http://0.0.0.0:9090 (Press CTRL+C to quit)
invokeai-rocm-1  | [2025-01-08 21:31:35,048]::[InvokeAI]::INFO --> Executing queue item 2, session 6492ca65-7b22-412f-b8dc-514aa06ec54d
invokeai-rocm-1  | [2025-01-08 21:31:35,631]::[ModelManagerService]::INFO --> [MODEL CACHE] Loaded model 'd168830e-9053-4490-bd67-ea0fa3f67103:text_encoder' (CLIPTextModel) onto cuda device in 0.48s. Total model size: 234.72MB, VRAM: 234.72MB (100.0%)
invokeai-rocm-1  | [2025-01-08 21:31:35,701]::[ModelManagerService]::INFO --> [MODEL CACHE] Loaded model 'd168830e-9053-4490-bd67-ea0fa3f67103:tokenizer' (CLIPTokenizer) onto cuda device in 0.00s. Total model size: 0.00MB, VRAM: 0.00MB (0.0%)
invokeai-rocm-1  | [2025-01-08 21:31:35,757]::[InvokeAI]::ERROR --> Error while invoking session 6492ca65-7b22-412f-b8dc-514aa06ec54d, invocation c5da9618-d1ef-4f1b-9283-47868f497644 (compel): HIP error: invalid device function
invokeai-rocm-1  | HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
invokeai-rocm-1  | For debugging consider passing AMD_SERIALIZE_KERNEL=3
invokeai-rocm-1  | Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
invokeai-rocm-1  | 
invokeai-rocm-1  | [2025-01-08 21:31:35,757]::[InvokeAI]::ERROR --> Traceback (most recent call last):
invokeai-rocm-1  |   File "/opt/invokeai/invokeai/app/services/session_processor/session_processor_default.py", line 129, in run_node
invokeai-rocm-1  |     output = invocation.invoke_internal(context=context, services=self._services)
invokeai-rocm-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/invokeai/invokeai/app/invocations/baseinvocation.py", line 300, in invoke_internal
invokeai-rocm-1  |     output = self.invoke(context)
invokeai-rocm-1  |              ^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
invokeai-rocm-1  |     return func(*args, **kwargs)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/invokeai/invokeai/app/invocations/compel.py", line 114, in invoke
invokeai-rocm-1  |     c, _options = compel.build_conditioning_tensor_for_conjunction(conjunction)
invokeai-rocm-1  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/compel/compel.py", line 186, in build_conditioning_tensor_for_conjunction
invokeai-rocm-1  |     this_conditioning, this_options = self.build_conditioning_tensor_for_prompt_object(p)
invokeai-rocm-1  |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/compel/compel.py", line 218, in build_conditioning_tensor_for_prompt_object
invokeai-rocm-1  |     return self._get_conditioning_for_flattened_prompt(prompt), {}
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/compel/compel.py", line 282, in _get_conditioning_for_flattened_prompt
invokeai-rocm-1  |     return self.conditioning_provider.get_embeddings_for_weighted_prompt_fragments(
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/compel/embeddings_provider.py", line 120, in get_embeddings_for_weighted_prompt_fragments
invokeai-rocm-1  |     base_embedding = self.build_weighted_embedding_tensor(tokens, per_token_weights, mask, device=device)
invokeai-rocm-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/compel/embeddings_provider.py", line 357, in build_weighted_embedding_tensor
invokeai-rocm-1  |     empty_z = self._encode_token_ids_to_embeddings(empty_token_ids)
invokeai-rocm-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/compel/embeddings_provider.py", line 390, in _encode_token_ids_to_embeddings
invokeai-rocm-1  |     text_encoder_output = self.text_encoder(token_ids,
invokeai-rocm-1  |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
invokeai-rocm-1  |     return self._call_impl(*args, **kwargs)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
invokeai-rocm-1  |     return forward_call(*args, **kwargs)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 1050, in forward
invokeai-rocm-1  |     return self.text_model(
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
invokeai-rocm-1  |     return self._call_impl(*args, **kwargs)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
invokeai-rocm-1  |     return forward_call(*args, **kwargs)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 941, in forward
invokeai-rocm-1  |     hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
invokeai-rocm-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
invokeai-rocm-1  |     return self._call_impl(*args, **kwargs)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
invokeai-rocm-1  |     return forward_call(*args, **kwargs)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 285, in forward
invokeai-rocm-1  |     inputs_embeds = self.token_embedding(input_ids)
invokeai-rocm-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
invokeai-rocm-1  |     return self._call_impl(*args, **kwargs)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
invokeai-rocm-1  |     return forward_call(*args, **kwargs)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/invokeai/invokeai/backend/model_manager/load/model_cache/torch_module_autocast/custom_modules/custom_embedding.py", line 29, in forward
invokeai-rocm-1  |     return super().forward(input)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/nn/modules/sparse.py", line 164, in forward
invokeai-rocm-1  |     return F.embedding(
invokeai-rocm-1  |            ^^^^^^^^^^^^
invokeai-rocm-1  |   File "/opt/venv/lib/python3.11/site-packages/torch/nn/functional.py", line 2267, in embedding
invokeai-rocm-1  |     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
invokeai-rocm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invokeai-rocm-1  | RuntimeError: HIP error: invalid device function
invokeai-rocm-1  | HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
invokeai-rocm-1  | For debugging consider passing AMD_SERIALIZE_KERNEL=3
invokeai-rocm-1  | Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
invokeai-rocm-1  | 
invokeai-rocm-1  | 
invokeai-rocm-1  | [2025-01-08 21:31:35,788]::[InvokeAI]::INFO --> Graph stats: 6492ca65-7b22-412f-b8dc-514aa06ec54d
invokeai-rocm-1  |                           Node   Calls   Seconds  VRAM Used
invokeai-rocm-1  |              main_model_loader       1    0.010s     0.000G
invokeai-rocm-1  |                      clip_skip       1    0.001s     0.000G
invokeai-rocm-1  |                         compel       1    0.686s     0.236G
invokeai-rocm-1  | TOTAL GRAPH EXECUTION TIME:   0.697s
invokeai-rocm-1  | TOTAL GRAPH WALL TIME:   0.698s
invokeai-rocm-1  | RAM used by InvokeAI process: 1.75G (+0.426G)
invokeai-rocm-1  | RAM used to load models: 0.23G
invokeai-rocm-1  | VRAM in use: 0.236G
invokeai-rocm-1  | RAM cache statistics:
invokeai-rocm-1  |    Model cache hits: 2
invokeai-rocm-1  |    Model cache misses: 2
invokeai-rocm-1  |    Models cached: 2
invokeai-rocm-1  |    Models cleared from cache: 0
invokeai-rocm-1  |    Cache high water mark: 0.23/0.00G
invokeai-rocm-1  |

Discord username

No response
@RSWilli RSWilli added the bug Something isn't working label Jan 8, 2025
@RSWilli
Copy link
Author

RSWilli commented Jan 8, 2025

#6976 describes the same problem, although not in docker

@RSWilli
Copy link
Author

RSWilli commented Jan 8, 2025

I searched the automatic1111 issues for the issue and found AUTOMATIC1111/stable-diffusion-webui#14763 adding HSA_OVERRIDE_GFX_VERSION=11.0.0 to the .env solves the issue, this should be documented though.

@RSWilli RSWilli mentioned this issue Jan 9, 2025
Open
@RSWilli
Copy link
Author

RSWilli commented Jan 9, 2025

With the given environment variable the SD and SDXL models seem to work, but flux doesn't. I think this is because of the bitsandbytes error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@RSWilli