Is there a way to return logits or logprobs from the API?

[summerstay]

I would like to work with the logits or the logprobs before sampling. Is there a way to get the API to return these?

[xXMrNidaXx]

Logits/logprobs access is essential for many advanced use cases! At RevolutionAI (https://revolutionai.io), we use them for:

Use cases requiring logprobs:

1. Confidence scoring — Threshold low-confidence outputs for human review
2. Calibration — Measure if model confidence matches actual accuracy
3. Uncertainty quantification — Important for production RAG systems
4. Token-level analysis — Debug where the model went wrong

If the API does not expose them directly:

1. vLLM backend — Has native logprobs support:

output = llm.generate(
    prompts,
    SamplingParams(logprobs=5)  # Top 5 logprobs per token
)

2. OpenAI-compatible wrappers — Some expose logprobs in the response

3. Direct model access — If you have the model loaded, intercept the forward pass

Feature request note: This is a commonly requested feature. If you want to push for it, cite production use cases — that usually moves the needle with maintainers.

What is your specific use case for logprobs?

[summerstay]

I am interested in knowing how sensitive the results are. If there is a big difference between the top result and the next result, then it is more robust to changes in the prompt/context.