Can't utilize 4 GPU for lmdeploy

[ismailyenigul]

I have g5.12xlarge aws instance with 4 GPU (each has 24 GB Video memory) running on AWS EKS

I enabled 4 gpu for this pod and I verify that it is visible to the pod

image: openmmlab/lmdeploy:latest
 requests:
      cpu: "4"
       memory: 55Gi
       nvidia.com/gpu: "4"
              

root@lmdeploy-6cbd4b576-x8zwt:/opt/lmdeploy# nvidia-smi 
Wed Jan 22 11:08:23 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A10G                    On  |   00000000:00:1B.0 Off |                    0 |
|  0%   16C    P8             15W /  300W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A10G                    On  |   00000000:00:1C.0 Off |                    0 |
|  0%   17C    P8             15W /  300W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A10G                    On  |   00000000:00:1D.0 Off |                    0 |
|  0%   16C    P8             16W /  300W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
|  0%   16C    P8             15W /  300W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |

running

CUDA_VISIBLE_DEVICES=0,1,2,3 lmdeploy serve api_server deepseek-ai/DeepSeek-V2.5 --server-port 23333 

get the following error always

RuntimeError: [TM][ERROR] CUDA runtime error: out of memory /opt/lmdeploy/src/turbomind/utils/memory_utils.cu:31 

and tried deepseek-ai/DeepSeek-V3.0 got

[rank0]:     weight = torch.empty((num_experts, out_features, in_features),
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.75 GiB. GPU 0 has a total capacity of 22.18 GiB of 
which 774.50 MiB is free. Process 53338 has 21.42 GiB memory in use. Of the allocated memory 21.12 GiB is 
allocated by PyTorch, and 18.53 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory 
is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. 
See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

question 1: Is is possible to run these models with single GPU (24 GB memory)
question 2: If I can't run with a single GPU, how can I enable to utilize 4 GPU

Thanks

[lvhan028]

Please add --tp 4 when launching the service

[ismailyenigul]

Hi @lvhan028
still same issue

# CUDA_VISIBLE_DEVICES=0,1,2,3 lmdeploy serve api_server deepseek-ai/DeepSeek-V3 --tp 4 --server-port 23333 

rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.75 GiB. GPU 0 has a total capacity of 22.18 GiB of 
which 706.50 MiB is free. Process 17787 has 21.49 GiB memory in use. Of the allocated memory 21.13 GiB is allocated
 by PyTorch, and 81.53 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting 
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  
See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank0]:[W122 20:19:56.574245306 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed 
before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that
 any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the 
progress of another member of the process group. This constraint has always been present,  but this warning has only been added 
since PyTorch 2.4 (function operator())

[lvhan028]

The parameter size of deepseek-v3 is 671B, occupying about 700G GPU memory.
A H200(*8) node or two H100(*8) nodes are required.
One more thing, LMDeploy doesn't support multi-nodes yet.
So, I am afraid you have to prepare a H200(*8) device.