Sampling parameters for text generation. Overall, we follow the sampling parameters from the OpenAI text completion API (https://platform.openai.com/docs/api-reference/completions/create). In addition, we support beam search, which is not supported by OpenAI. Args: n: Numbe
| 28 | |
| 29 | @dataclass |
| 30 | class SamplingParams: |
| 31 | """Sampling parameters for text generation. |
| 32 | |
| 33 | Overall, we follow the sampling parameters from the OpenAI text completion |
| 34 | API (https://platform.openai.com/docs/api-reference/completions/create). |
| 35 | In addition, we support beam search, which is not supported by OpenAI. |
| 36 | |
| 37 | Args: |
| 38 | n: Number of output sequences to return for the given prompt. |
| 39 | best_of: Number of output sequences that are generated from the prompt. |
| 40 | From these `best_of` sequences, the top `n` sequences are returned. |
| 41 | `best_of` must be greater than or equal to `n`. By default, |
| 42 | `best_of` is set to `n`. Warning, this is only supported in V0. |
| 43 | presence_penalty: Float that penalizes new tokens based on whether they |
| 44 | appear in the generated text so far. Values > 0 encourage the model |
| 45 | to use new tokens, while values < 0 encourage the model to repeat |
| 46 | tokens. |
| 47 | frequency_penalty: Float that penalizes new tokens based on their |
| 48 | frequency in the generated text so far. Values > 0 encourage the |
| 49 | model to use new tokens, while values < 0 encourage the model to |
| 50 | repeat tokens. |
| 51 | repetition_penalty: Float that penalizes new tokens based on whether |
| 52 | they appear in the prompt and the generated text so far. Values > 1 |
| 53 | encourage the model to use new tokens, while values < 1 encourage |
| 54 | the model to repeat tokens. |
| 55 | temperature: Float that controls the randomness of the sampling. Lower |
| 56 | values make the model more deterministic, while higher values make |
| 57 | the model more random. Zero means greedy sampling. |
| 58 | top_p: Float that controls the cumulative probability of the top tokens |
| 59 | to consider. Must be in [0, 1]. Set to 1 to consider all tokens. |
| 60 | top_k: Int that controls the number of top tokens to consider. Must be a positive integer. |
| 61 | min_p: Float that represents the minimum probability for a token to be |
| 62 | considered, relative to the probability of the most likely token. |
| 63 | Must be in [0, 1]. Set to 0 to disable this. |
| 64 | seed: Random seed to use for the generation. |
| 65 | stop: list of strings that stop the generation when they are generated. |
| 66 | The returned output will not contain the stop strings. |
| 67 | stop_token_ids: list of tokens that stop the generation when they are |
| 68 | generated. The returned output will contain the stop tokens unless |
| 69 | the stop tokens are special tokens. |
| 70 | bad_words: list of words that are not allowed to be generated. |
| 71 | More precisely, only the last token of a corresponding |
| 72 | token sequence is not allowed when the next generated token |
| 73 | can complete the sequence. |
| 74 | max_tokens: Maximum number of tokens to generate per output sequence. |
| 75 | reasoning_max_tokens: Maximum number of tokens to generate for reasoning per output sequence. |
| 76 | response_max_tokens: Maximum number of tokens to generate for response per output sequence. |
| 77 | min_tokens: Minimum number of tokens to generate per output sequence |
| 78 | before EOS or stop_token_ids can be generated |
| 79 | logprobs: Number of log probabilities to return per output token. |
| 80 | When set to None, no probability is returned. If set to a non-None |
| 81 | value, the result includes the log probabilities of the specified |
| 82 | number of most likely tokens, as well as the chosen tokens. |
| 83 | Note that the implementation follows the OpenAI API: The API will |
| 84 | always return the log probability of the sampled token, so there |
| 85 | may be up to `logprobs+1` elements in the response. |
| 86 | """ |
| 87 |
no outgoing calls