MCPcopy Create free account

hub / github.com/disler/benchy / functions

Functions126 in github.com/disler/benchy

⨍ Functions 126◇ Types & classes 38↳ Endpoints 6
↓ 16 callersFunctiontimeit
Context manager to measure execution time in milliseconds. Usage: with timeit() as t: # code to time elapsed_ms
server/utils.py:8
↓ 12 callersFunctionget_openai_cost
Calculate the cost for OpenAI API usage. Args: model: The model name/alias used input_tokens: Number of input tokens
server/modules/openai_llm.py:28
↓ 6 callersFunctiondeepseek_r1_distil_separate_thoughts_and_response
Parse DeepSeek R1 responses containing <think> blocks and separate thoughts from final response. Args: response: Raw model response
server/utils.py:136
↓ 6 callersFunctionsimple_prompt
(prompt_str: str, model_alias_str: str)
server/modules/llm_models.py:21
↓ 5 callersFunctioneval_result_compare
Compare expected and actual results based on evaluation type. For numeric outputs, compare with a small epsilon tolerance.
server/modules/execution_evaluators.py:7
↓ 5 callersFunctionparse_reasoning_effort
Parse a model string to extract reasoning effort. If the model contains ":low", ":medium" or ":high" (case‐insensitive), returns (base_mo
server/utils.py:181
↓ 4 callersFunctioncalculatePercentCorrect
(numberCorrect: number)
client/src/stores/autocompleteStore.ts:363
↓ 4 callersFunctionparse_markdown_backticks
(str)
server/utils.py:125
↓ 4 callersFunctiontext_prompt
Send a prompt to Ollama and get a response.
server/modules/ollama_llm.py:7
↓ 3 callersFunctionexecute_python_code
Execute Python code and return the numeric output as a string.
server/modules/execution_evaluators.py:48
↓ 3 callersFunctionget_anthropic_cost
Calculate the cost for Anthropic API usage. Args: model: The model name/alias used input_tokens: Number of input tokens
server/modules/anthropic_llm.py:28
↓ 3 callersFunctionget_deepseek_cost
Calculate the cost for Gemini API usage. Args: model: The model name/alias used input_tokens: Number of input tokens
server/modules/deepseek_llm.py:16
↓ 3 callersFunctionget_gemini_cost
Calculate the cost for Gemini API usage. Args: model: The model name/alias used input_tokens: Number of input tokens
server/modules/gemini_llm.py:28
↓ 3 callersFunctionloadDefaultState
()
client/src/stores/thoughtBenchStore.ts:4
↓ 3 callersFunctionloadDefaultState
()
client/src/stores/autocompleteStore.ts:3
↓ 3 callersFunctionloadDefaultState
()
client/src/stores/toolCallStore.ts:4
↓ 2 callersFunctionbench_prompt
Send a prompt to DeepSeek and get detailed benchmarking response.
server/modules/deepseek_llm.py:39
↓ 2 callersFunctioncalculatePercentCorrect
(numberCorrect: number)
client/src/apis/toolCallApi.ts:116
↓ 2 callersFunctiongenerate_report
( complete_result: ExecEvalBenchmarkCompleteResult, )
server/modules/exbench_module.py:250
↓ 2 callersFunctionpredictive_prompt
Run a chat model with a predicted output to reduce latency. Args: prompt (str): The prompt to send to the OpenAI API. predic
server/modules/openai_llm.py:189
↓ 2 callersFunctionresetBenchmark
()
client/src/stores/isoSpeedBenchStore.ts:61
↓ 2 callersFunctionrun_benchmark_for_model
( model: str, benchmark_file: ExecEvalBenchmarkFile )
server/modules/exbench_module.py:205
↓ 2 callersFunctionsave_report_to_file
Save benchmark report to file with standardized naming. Args: report: The benchmark report to save output_dir: Directory to save
server/modules/exbench_module.py:73
↓ 2 callersFunctiontext_prompt
Send a prompt to OpenAI and get a response.
server/modules/openai_llm.py:227
↓ 2 callersFunctiontext_prompt
Send a prompt to DeepSeek and get the response.
server/modules/deepseek_llm.py:76
↓ 2 callersFunctionthought_prompt
Send a thought prompt to DeepSeek and parse structured response.
server/modules/deepseek_llm.py:105
↓ 2 callersFunctionthought_prompt
Handle thought prompts for Gemini thinking models.
server/modules/gemini_llm.py:51
↓ 1 callersFunctionbench_prompt
Send a prompt to Anthropic and get detailed benchmarking response.
server/modules/anthropic_llm.py:80
↓ 1 callersFunctionbench_prompt
Send a prompt to OpenAI and get detailed benchmarking response.
server/modules/openai_llm.py:142
↓ 1 callersFunctionbench_prompt
Send a prompt to Gemini and get detailed benchmarking response.
server/modules/gemini_llm.py:124
↓ 1 callersFunctionbench_prompt
Send a prompt to Ollama and get detailed benchmarking response.
server/modules/ollama_llm.py:85
↓ 1 callersFunctionbench_prompt
(prompt: str, model: str)
server/modules/fireworks_llm.py:28
↓ 1 callersFunctionexecute
Execute the tests and return the output as a string.
server/modules/execution_evaluators.py:76
↓ 1 callersFunctionloadSettings
()
client/src/stores/isoSpeedBenchStore.ts:41
↓ 1 callersFunctionloadState
()
client/src/stores/thoughtBenchStore.ts:81
↓ 1 callersFunctionloadState
()
client/src/stores/autocompleteStore.ts:332
↓ 1 callersFunctionloadState
()
client/src/stores/toolCallStore.ts:218
↓ 1 callersFunctionmain
Run the Flask application.
server/server.py:184
↓ 1 callersFunctionmakeRequest
()
client/src/apis/thoughtBenchApi.ts:31
↓ 1 callersFunctionparse_model_string
Parse model string into provider and model name. Format: "provider:model_name" or "model_name" (defaults to ollama) Raises: Valu
server/modules/exbench_module.py:37
↓ 1 callersFunctionprocess_single_prompt
( prompt_row, benchmark_file, provider, model_name, index, total_tests )
server/modules/exbench_module.py:109
↓ 1 callersFunctionrun_coder_agent
Run the coder agent with the given prompt. Args: prompt (str): The input prompt for the coder agent Returns: str: The r
server/modules/tools.py:1
↓ 1 callersFunctionrun_docs_agent
Run the docs agent with the given prompt. Args: prompt (str): The input prompt for the docs agent Returns: str: The res
server/modules/tools.py:27
↓ 1 callersFunctionrun_git_agent
Run the git agent with the given prompt. Args: prompt (str): The input prompt for the git agent Returns: str: The respo
server/modules/tools.py:14
↓ 1 callersFunctionsendPrompt
(prompt: string, model: ModelAlias)
client/src/apis/autocompleteApi.ts:3
↓ 1 callersFunctionsendToolPrompt
(prompt: string, model: ModelAlias)
client/src/apis/toolCallApi.ts:3
↓ 1 callersFunctionsetState
(state: any)
client/src/stores/thoughtBenchStore.ts:100
↓ 1 callersFunctionsetState
(state: any)
client/src/stores/autocompleteStore.ts:352
↓ 1 callersFunctionsetState
(state: any)
client/src/stores/toolCallStore.ts:237
↓ 1 callersFunctionsleep
(ms: number)
client/src/apis/thoughtBenchApi.ts:11
↓ 1 callersFunctiontext_prompt
Send a prompt to Anthropic and get a response.
server/modules/anthropic_llm.py:51
↓ 1 callersFunctiontext_prompt
Send a prompt to Gemini and get a response.
server/modules/gemini_llm.py:98
↓ 1 callersFunctiontext_prompt
(prompt: str, model: str)
server/modules/fireworks_llm.py:61
↓ 1 callersFunctionthought_prompt
(prompt: str, model: str)
server/modules/fireworks_llm.py:91
↓ 1 callersFunctiontool_prompt
Run a chat model forcing specific tool calls. Now supports JSON structured output variants.
server/modules/openai_llm.py:67
Methodaccuracy
(self)
server/modules/data_types.py:147
Functionclient
()
server/tests/server_test.py:7
FunctioncopyToClipboard
(text: string)
client/src/utils.ts:3
Methodcorrect_count
(self)
server/modules/data_types.py:139
FunctionflashBenchmark
()
client/src/stores/isoSpeedBenchStore.ts:118
FunctiongetContrastTextColor
(backgroundColor: string)
client/src/utils.ts:30
Functionget_fireworks_cost
(model: str, input_tokens: int, output_tokens: int)
server/modules/fireworks_llm.py:23
Functionget_ollama_costs
Return token costs for Ollama (always 0 since it's free)
server/modules/ollama_llm.py:36
FunctionhandleCorrect
(model: ModelAlias, isCorrect: boolean)
client/src/stores/autocompleteStore.ts:369
Functionhandle_iso_speed_bench
Handle an ISO speed benchmark request with YAML input.
server/server.py:108
Functionhandle_prompt
Handle a prompt request and return the model's response.
server/server.py:26
Functionhandle_thought_bench
Handle a thought bench request and return the model's response.
server/server.py:76
Functionhandle_tool_prompt
Handle a tool prompt request and return the tool calls.
server/server.py:49
Methodincorrect_count
(self)
server/modules/data_types.py:143
Functionmodel
()
server/tests/fireworks_llm_test.py:6
Functionollama_bench
Run benchmarks on Ollama models using a YAML configuration file. Example usage: uv run python exbench.py ollama-bench benchmark_data/si
server/exbench.py:26
Functionping
()
server/exbench.py:21
FunctionresetState
()
client/src/stores/thoughtBenchStore.ts:94
FunctionresetState
()
client/src/stores/autocompleteStore.ts:346
FunctionresetState
()
client/src/stores/toolCallStore.ts:231
FunctionretryRequest
* No need for this here
client/src/apis/thoughtBenchApi.ts:18
FunctionrunAutocomplete
()
client/src/apis/autocompleteApi.ts:22
FunctionrunThoughtPrompt
(request: ThoughtRequest)
client/src/apis/thoughtBenchApi.ts:30
FunctionrunToolCall
()
client/src/apis/toolCallApi.ts:28
FunctionsaveSettings
()
client/src/stores/isoSpeedBenchStore.ts:37
FunctionstartBenchmark
()
client/src/stores/isoSpeedBenchStore.ts:71
FunctionstringToColor
(str: string)
client/src/utils.ts:10
Functiontest_anthropic_bench_prompt
()
server/tests/anthropic_llm_test.py:11
Functiontest_anthropic_text_prompt
()
server/tests/anthropic_llm_test.py:4
Functiontest_bench_prompt
(model)
server/tests/fireworks_llm_test.py:10
Functiontest_bench_prompt_metrics
(model)
server/tests/ollama_llm_test.py:42
Functiontest_cost_ordering_group1
()
server/tests/openai_llm_test.py:113
Functiontest_cost_ordering_group2
()
server/tests/openai_llm_test.py:135
Functiontest_deepseek_bench_prompt
()
server/tests/deepseek_llm_test.py:13
Functiontest_deepseek_error_handling
()
server/tests/deepseek_llm_test.py:25
Functiontest_deepseek_text_prompt
()
server/tests/deepseek_llm_test.py:6
Functiontest_gemini_bench_prompt
()
server/tests/gemini_llm_test.py:12
Functiontest_gemini_text_prompt
()
server/tests/gemini_llm_test.py:5
Functiontest_gemini_thought_prompt
()
server/tests/gemini_llm_test.py:22
Functiontest_gemini_thought_prompt_invalid_model
()
server/tests/gemini_llm_test.py:39
Functiontest_llama_3_2_latest_text_prompt
()
server/tests/ollama_llm_test.py:19
Functiontest_missing_xml_handling
()
server/tests/ollama_llm_test.py:83
Functiontest_ollama_text_prompt
()
server/tests/ollama_llm_test.py:5
Functiontest_openai_bench_prompt
()
server/tests/openai_llm_test.py:67
Functiontest_openai_text_prompt
()
server/tests/openai_llm_test.py:58
next →1–100 of 126, ranked by callers