Support for returning Logits and Calculating Perplexity During Model Evaluation?

[hxer7963]

Hello SGLang Community,

I'm currently exploring the capabilities of the SGLang inference framework for LLMs and I have a couple of questions regarding model evaluation:

• Returning Logits: Does SGLang support returning the logits (instead of return_logprob) during inference? This is often useful for applications requiring a detailed understanding of model predictions on a token level.

• Calculating Perplexity (PPL) During Evaluation: Is there built-in support or recommended practices for calculating the perplexity of a model during evaluation? Perplexity is a crucial metric for assessing the quality of language models, and having an efficient way to calculate it would be valuable.

If these features are not currently supported, are there any plans to include them in future updates?

Any guidance or suggestions on how to implement these functionalities using the current framework would also be greatly appreciated.

Thank you!

Best regards, willhe.

[merrymercy]

They are well supported. Some related docs:

• Return logprob for a generation request

  sglang/docs/en/sampling_params.md

  Lines 23 to 28 in 5ab9418

  	# Whether to return logprobs.
  	return_logprob: Optional[Union[List[bool], bool]] = None
  	# The start location of the prompt for return_logprob.
  	logprob_start_len: Optional[Union[List[int], int]] = None
  	# The number of top logprobs to return.
  	top_logprobs_num: Optional[Union[List[int], int]] = None

• The full OpenAI API spec around logprob is supported.
  

  sglang/test/srt/test_openai_server.py

  Line 72 in 5ab9418

  if logprobs:

• Other ways to use it in the frontend language.
  https://github.com/sgl-project/sglang/blob/main/docs/en/choices_methods.md

[fmdmm]

@merrymercy Thank you for creating SGLang!

Feature Request: Support returning raw logits in addition to logprobs

Currently SGLang only supports returning logprobs through sampling_params.return_logprob. Would it be possible to add support for returning the raw logits (pre-softmax values) from the model? This would be useful for:

1. Detailed model behavior analysis
2. Custom probability calculations
3. Advanced scoring and ranking methods

The API could potentially look like:

sampling_params = SamplingParams(
    return_logits=True,  # New parameter
    return_logprob=True  # Existing parameter
)

Is this something that could be considered for future releases?