[Bug] The batch decoding speed of DeepSeek V3 is too slow.

[SonChoulJun]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

The hardware setup consists of H100 * 8 across 2 nodes.

The execution command is as follows:
python3 -m sglang.launch_server --model-path /workspace/DBP/sllm_mount/models/DeepSeek-V3 --tp 16 --dist-init-addr 192.168.1.10:20000 --nnodes 2 --node-rank 0 --trust-remote-code --host 0.0.0.0 --port 8010 --disable-cuda-graph

Reproduction

deepseek-V3

Environment

The hardware setup consists of H100 * 8 across 2 nodes.

[zhyncs]

Does the GPU environment have InfiniBand?

[SonChoulJun]

yes,
The InfiniBand environment is configured with the following

export NCCL_DEBUG=INFO
export NCCL_IB_HCA=mlx5_0 for communication
export GLOO_SOCKET_IFNAME=ibs108
export NCCL_SOCKET_IFNAME=ibs108
export MASTER_ADDR=192.168.1.10
export MASTER_PORT=20000

[naveenjafer]

@SonChoulJun Hey, were you able to resolve this? Pretty much the same issue

[ispobock]

Why add --disable-cuda-graph? It may affect decoding speed a lot.