Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/GPT-Fathom/GPT-Fathom
/ functions
Functions
419 in github.com/GPT-Fathom/GPT-Fathom
⨍
Functions
419
◇
Types & classes
72
↳
Endpoints
2
Function
clean_numbers
(sample)
evals/utils/math_util.py:57
Function
compare_two_list
(pred, gt)
evals/utils/theoremqa_utils.py:150
Function
compute_averaged_f_score
( confusion_matrix: np.ndarray, beta: float = 1.0, average: str = "macro" )
evals/metrics.py:81
Function
compute_matthew_corr
(confusion_matrix: np.ndarray)
evals/metrics.py:56
Function
contain_num_and_str
(string)
evals/utils/theoremqa_utils.py:175
Function
create_chat_prompt
(text)
evals/build_data/build_hellaswag.py:8
Function
create_chat_prompt
(text)
evals/build_data/build_mgsm.py:10
Function
create_chat_prompt
(text, instruction)
evals/build_data/build_bbh.py:36
Function
create_chat_prompt
(text)
evals/build_data/build_arc.py:8
Function
create_chat_prompt
(text)
evals/build_data/build_truthfulqa.py:7
Function
create_fewshot_prompt
(question, answer)
evals/build_data/build_hellaswag.py:18
Function
create_fewshot_prompt
(question, answer)
evals/build_data/build_realtoxic.py:11
Function
create_fewshot_prompt
(question, answer)
evals/build_data/build_mgsm.py:20
Function
create_fewshot_prompt
(question, answer)
evals/build_data/build_bbh.py:43
Function
create_fewshot_prompt
(question, answer)
evals/build_data/build_arc.py:18
Function
create_fewshot_prompt
(question, answer)
evals/build_data/build_truthfulqa.py:17
Function
current_sample_id
()
evals/record.py:516
Method
default
(self, o: Any)
evals/data.py:187
Function
equiv_scibench
Helper function for record_and_check_match_scibench. Checks the numerical equivalence between model output and answer. Args: model_o
evals/utils/scibench_utils.py:4
Method
eval_fn_with_semaphore
(args)
evals/eval.py:102
Method
eval_sample
(self, sample: Any, rng: random.Random)
evals/eval.py:78
Method
eval_sample
(self, sample, rng)
evals/elsuite/dataset_specific/ceval.py:160
Method
eval_sample
(self, test_sample, rng)
evals/elsuite/dataset_specific/realtoxic.py:60
Method
eval_sample
(self, sample, rng)
evals/elsuite/dataset_specific/theoremqa.py:79
Method
eval_sample
(self, sample, rng)
evals/elsuite/dataset_specific/bbh.py:94
Method
eval_sample
(self, sample, rng)
evals/elsuite/dataset_specific/agieval.py:399
Method
eval_sample
(self, sample, rng)
evals/elsuite/dataset_specific/scibench.py:152
Method
eval_sample
(self, sample, rng)
evals/elsuite/dataset_specific/mmlu_CoT.py:133
Method
eval_sample
(self, sample: Any, *_)
evals/elsuite/dataset_specific/lambada.py:102
Method
eval_sample
(self, sample, rng)
evals/elsuite/dataset_specific/summedits.py:81
Method
eval_sample
Evaluate a single sample. Recorded metrics are always: one of the self.choice_strings, or "__invalid__".
evals/elsuite/modelgraded/classify.py:54
Method
eval_sample
(self, sample, rng)
evals/elsuite/basic/long_comprehension.py:144
Method
eval_sample
(self, sample, rng)
evals/elsuite/basic/code.py:149
Method
eval_sample
(self, sample, rng)
evals/elsuite/basic/reading_comprehension.py:138
Method
eval_sample
(self, sample, rng)
evals/elsuite/basic/multiple_choice.py:215
Method
eval_sample
(self, sample, rng)
evals/elsuite/basic/math_problem.py:147
Function
f1_score
(prediction: str, answers: list[str])
evals/elsuite/utils.py:93
Function
f1_score_chinese
(prediction, ground_truth, **kwargs)
evals/elsuite/utils.py:108
Method
flush_events
(self)
evals/record.py:152
Function
get_accuracy
(events: Sequence[Event])
evals/metrics.py:12
Method
get_alias
()
evals/registry.py:186
Function
get_answer
(text, answer_prompt, ignore_case=False)
evals/elsuite/utils.py:20
Method
get_base_evals
(self)
evals/registry.py:246
Function
get_bootstrap_accuracy_std
( events: Sequence[Event], num_samples: int = 1000 )
evals/metrics.py:30
Method
get_completions
(self)
evals/api.py:21
Method
get_completions
(self)
evals/api.py:47
Method
get_completions
(self)
evals/completion_fns/llama.py:27
Method
get_completions
(self)
evals/completion_fns/llama.py:32
Method
get_completions
(self)
evals/completion_fns/openai.py:28
Method
get_completions
(self)
evals/completion_fns/langchain_math.py:16
Method
get_completions
(self)
evals/completion_fns/langchain_llm.py:15
Function
get_confusion_matrix
( matches: Sequence[Event], class_labels: Optional[Set] = None )
evals/metrics.py:39
Function
get_consensus
(answers)
evals/elsuite/utils.py:31
Function
get_csv
(path, fieldnames=None)
evals/data.py:164
Function
get_json
(path)
evals/data.py:136
Function
get_jsonls
(paths: Sequence[str], line_limit=None)
evals/data.py:132
Function
get_letter_from_data
(data: str)
evals/elsuite/utils.py:86
Function
get_lines
Get a list of lines from a file.
evals/data.py:108
Function
get_picked
(events: Sequence[Event])
evals/metrics.py:21
Method
get_regexp
(pattern: str)
evals/registry.py:235
Method
get_samples
(self)
evals/eval.py:154
Function
get_scores_from_text
(text: str)
evals/elsuite/utils.py:74
Function
get_yesno_from_text
(text: str)
evals/elsuite/utils.py:80
Function
is_scientific_number
(string)
evals/utils/theoremqa_utils.py:169
Function
jsondump
(o: Any, fp: Any, ensure_ascii: bool = False, **kwargs: Any)
evals/data.py:195
Function
jsonload
(fp: Any, **kwargs: Any)
evals/data.py:203
Function
jsonloads
(s: str, **kwargs: Any)
evals/data.py:199
Function
last_boxed_only
Given a (q,a) sample, filter the answers so that they only contain the last \boxed{...} or \fbox{...} element
evals/utils/math_util.py:1
Method
last_boxed_only_string
(self, string)
evals/elsuite/dataset_specific/scibench.py:91
Function
lower
(text)
evals/elsuite/utils.py:59
Function
normalize
(prediction: str)
evals/utils/theoremqa_utils.py:181
Function
only_until_first_boxed_from_tokens
(string, tokens)
evals/utils/math_util.py:41
Function
parse_number_list
(s: str)
evals/utils/theoremqa_utils.py:128
Function
pause
()
evals/record.py:562
Method
readable
Returns True if the IO object can be read.
evals/elsuite/utils_execute.py:141
Method
readline
(self, *args, **kwargs)
evals/elsuite/utils_execute.py:135
Function
record_and_check_match
Records and checks if a sampled response from a CompletionFn matches the expected result. Args: prompt: The input prompt. sa
evals/api.py:60
Function
record_and_check_match_scibench
Records and checks if a sampled response from a CompletionFn matches the expected result for scibench. Args: prompt: The input promp
evals/api.py:121
Function
record_and_check_match_theoremqa
Records and checks if a sampled response from a CompletionFn matches the expected result for scibench. Args: prompt: The input promp
evals/api.py:156
Function
record_cond_logp
(prompt, completion, logp, **extra)
evals/record.py:534
Function
record_embedding
(prompt, embedding_type, **extra)
evals/record.py:526
Function
record_error
(msg: str, error: Exception = None, **extra)
evals/record.py:550
Function
record_event
(type, data=None, sample_id=None)
evals/record.py:558
Method
record_event
(self, type, data, sample_id=None)
evals/record.py:291
Function
record_extra
(data)
evals/record.py:554
Method
record_final_report
(self, final_report: Any)
evals/record.py:269
Method
record_final_report
(self, final_report: Any)
evals/record.py:360
Function
record_metrics
(**extra)
evals/record.py:546
Function
record_pick_option
(prompt, options, picked, **extra)
evals/record.py:538
Function
record_raw
(data)
evals/record.py:542
Method
remove_boxed
(self, s)
evals/elsuite/dataset_specific/scibench.py:79
Method
run
Run the evaluation with the corresponding recorder.
evals/eval.py:87
Method
run
(self, recorder: RecorderBase)
evals/elsuite/dataset_specific/ceval.py:201
Method
run
(self, recorder: RecorderBase)
evals/elsuite/dataset_specific/realtoxic.py:95
Method
run
(self, recorder: RecorderBase)
evals/elsuite/dataset_specific/theoremqa.py:128
Method
run
(self, recorder: RecorderBase)
evals/elsuite/dataset_specific/bbh.py:131
Method
run
(self, recorder: RecorderBase)
evals/elsuite/dataset_specific/agieval.py:437
Method
run
(self, recorder: RecorderBase)
evals/elsuite/dataset_specific/scibench.py:191
Method
run
(self, recorder: RecorderBase)
evals/elsuite/dataset_specific/mmlu_CoT.py:179
Method
run
(self, recorder: RecorderBase)
evals/elsuite/dataset_specific/lambada.py:135
← previous
next →
301–400 of 419, ranked by callers