Function tokenizeTokenCount

core/backend/tokenize.go:20–25

tokenizeTokenCount returns the number of tokens in a backend response, treating a nil response as zero. The gRPC client returns (nil, err) on failure, and the tracing block below runs before that error is returned — so the count must be read nil-safely here. Reading resp.Tokens on a nil resp previou

(resp *pb.TokenizationResponse)

// resp previously panicked the whole HTTP handler when tracing was enabled
// (e.g. a transient tokenize failure during router probe-budget sizing).
func tokenizeTokenCount(resp *pb.TokenizationResponse) int {
	if resp == nil {
		return 0
	}
	return len(resp.Tokens)
}

func ModelTokenize(s string, loader *model.ModelLoader, modelConfig config.ModelConfig, appConfig *config.ApplicationConfig) (schema.TokenizeResponse, error) {