Some question about layernom in MLA code

[hcyz33]

Hi，I am confused that there is a layer normalization between the down-sample and up-sample of Q. However, this layer normalization is not shown in the DeepSeek v2 paper.

Here is the code of sglang

Here is the formulate in paper

[zhaochenyang20]

@ispobock Hey Ke, could you help with this?

[ispobock]

It's added in the original implementation.
Ref: https://huggingface.co/deepseek-ai/DeepSeek-V2.5/blob/c85b5ede86f2a598af339624cac5723861e557ed/modeling_deepseek.py#L825

And also mentioned in the paper:

For other tiny details (e.g., layer normalization and the activation function in FFNs), unless specifically stated, DeepSeek-V2 follows the settings of DeepSeek 67B

[hcyz33]

Thank you！