Defines the common sampling parameters that are shared across different types of generation requests, such as standard completions and chat-based completions. Attributes: max_tokens: The maximum number of tokens to generate. temperature: The sampling temperature. top_p: The
| 27 | |
| 28 | |
| 29 | class SamplingParams(BaseModel): |
| 30 | """ |
| 31 | Defines the common sampling parameters that are shared across different types of |
| 32 | generation requests, such as standard completions and chat-based completions. |
| 33 | |
| 34 | Attributes: |
| 35 | max_tokens: The maximum number of tokens to generate. |
| 36 | temperature: The sampling temperature. |
| 37 | top_p: The nucleus sampling probability. |
| 38 | top_k: The top-k sampling integer. |
| 39 | stream: Whether to stream the response. |
| 40 | stop: A string or list of strings that will stop the generation. |
| 41 | seed: A seed for deterministic sampling. |
| 42 | """ |
| 43 | |
| 44 | max_tokens: Optional[int] = None |
| 45 | temperature: Optional[float] = None |
| 46 | top_p: Optional[float] = None |
| 47 | top_k: Optional[int] = None |
| 48 | stream: Optional[bool] = False |
| 49 | stop: Optional[Union[str, List[str]]] = None |
| 50 | seed: Optional[int] = None |
| 51 | |
| 52 | |
| 53 | class CompletionRequest(SamplingParams): |