Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/huggingface/smollm
/ functions
Functions
1,907 in github.com/huggingface/smollm
⨍
Functions
1,907
◇
Types & classes
810
↳
Endpoints
7
↓ 1 callers
Function
process_document
(text: str)
vision/m4/sourcing/processing/extracting_ngrams/extract_documents_ngrams.py:27
↓ 1 callers
Function
process_html_css
(html_css_file)
vision/data/datasets_processing_scripts/build_image_website_code/03_extraction_html_css_codes.py:10
↓ 1 callers
Method
process_ideas_websites
Keep only the main business idea, remove the details about the design of the website
vision/data/datasets_processing_scripts/build_websight_v02/python_scripts/02_02_generate_html_codes_prompt_2.py:78
↓ 1 callers
Method
process_image
(self, image_bytes)
vision/data/datasets_processing_scripts/build_webdocs_dataset/python_scripts/07_01_nsfw_image_filtering.py:117
↓ 1 callers
Method
process_inputs_binary_classification
(self, texts, images)
vision/data/datasets_processing_scripts/build_laion_coco_dataset/python_scripts/05_binary_classification.py:115
↓ 1 callers
Function
process_shard
(shard_id, data_dir)
vision/data/datasets_processing_scripts/enwiki/python_scripts/02_load_wit_images_in_ds.py:95
↓ 1 callers
Function
process_yaml
Process the YAML file, count samples, and validate paths.
vision/smolvlm2/scripts/mixture_checker.py:42
↓ 1 callers
Method
prune_heads
(self, heads)
vision/m4/models/vgpt2/modeling_vgpt2.py:167
↓ 1 callers
Function
pynmvl_handle
(accelerator)
vision/m4/training/utils.py:578
↓ 1 callers
Function
remove_duplicated_images
(texts, images, metadata)
vision/data/datasets_processing_scripts/build_webdocs_dataset/python_scripts/13_final_processing.py:100
↓ 1 callers
Function
remove_end_END_OF_DOCUMENT_TOKEN_TO_BE_REPLACED
(texts, images, metadata)
vision/data/datasets_processing_scripts/build_webdocs_dataset/python_scripts/13_final_processing.py:168
↓ 1 callers
Method
remove_non_printing_characters
(text, non_printing_characters_re)
vision/m4/sourcing/data_collection/processors/pair_filtering.py:199
↓ 1 callers
Function
remove_spam_paragraphs
(texts, images, metadata)
vision/data/datasets_processing_scripts/build_webdocs_dataset/python_scripts/13_final_processing.py:129
↓ 1 callers
Function
remove_texts_only_END_OF_DOCUMENT_TOKEN_TO_BE_REPLACED
(example)
vision/data/datasets_processing_scripts/build_webdocs_dataset/python_scripts/13_final_processing.py:236
↓ 1 callers
Function
rename_state_dict
(state_dict)
vision/m4/scripts/convert_vmistral_old_siglip_to_new_siglip.py:12
↓ 1 callers
Method
replace_digits_with_zeros
(text, digits_re)
vision/m4/sourcing/data_collection/processors/web_document_filtering.py:360
↓ 1 callers
Method
replace_overused_color_from_html_code
(self, html_code)
vision/data/datasets_processing_scripts/build_websight_v02/python_scripts/02_01_generate_html_codes_prompt_1.py:130
↓ 1 callers
Method
replace_overused_color_from_html_code
(self, html_code)
vision/data/datasets_processing_scripts/build_websight_v02/python_scripts/02_02_generate_html_codes_prompt_2.py:132
↓ 1 callers
Method
replace_unicode_punctuation
(text, unicode_punctuation)
vision/m4/sourcing/data_collection/processors/web_document_filtering.py:364
↓ 1 callers
Method
reset_jsonl_stats
(self)
vision/m4/utils/activation_tracker.py:171
↓ 1 callers
Method
reset_state
(self)
vision/m4/training/dataset.py:1344
↓ 1 callers
Method
resize_and_center_crop
(self, image: Image.Image, target_size: int)
tools/smolvlm_local_inference/SmolVLM_video_inference.py:16
↓ 1 callers
Function
retrieve_image
(url)
vision/data/datasets_processing_scripts/build_webdocs_dataset/python_scripts/04_merge_web_docs_with_images.py:71
↓ 1 callers
Function
run_cmd
(cmd, check=True)
vision/m4/scripts/convert-checkpoints.py:35
↓ 1 callers
Function
run_conv
(text: str)
vision/m4/sourcing/processing/app.py:48
↓ 1 callers
Function
run_query
(ngram_ids: List[int])
vision/m4/sourcing/processing/app.py:55
↓ 1 callers
Function
sample_has_all_files
(current_sample)
vision/m4/training/dataset_utils.py:55
↓ 1 callers
Method
save_commit_hash
(self)
vision/m4/sourcing/data_collection/processors/web_document_extractor.py:427
↓ 1 callers
Method
save_dataset
(self)
vision/m4/sourcing/data_collection/processors/web_document_extractor.py:422
↓ 1 callers
Function
save_image_caption_pair_example_in_files_with_num_shards
( example, idx, saving_dir, num_examples_per_shard, save_shard_prefix )
vision/m4/utils/datasets/create_webdataset_tar.py:101
↓ 1 callers
Function
save_image_question_answer_triplet_example_in_files_with_num_shards
( example, idx, saving_dir, num_examples_per_shard, save_shard_prefix )
vision/m4/utils/datasets/create_webdataset_tar.py:134
↓ 1 callers
Function
save_model_and_config
(new_checkpoint_path, new_model)
vision/m4/scripts/convert_vmistral_old_siglip_to_new_siglip.py:25
↓ 1 callers
Method
save_new_ds
(self)
vision/data/datasets_processing_scripts/build_concatenation_datasets_sft/merge_on_image_individual_dataset.py:130
↓ 1 callers
Function
save_sft_example_in_files_with_num_shards
(example, idx, saving_dir, num_examples_per_shard, save_shard_prefix)
vision/m4/utils/datasets/create_webdataset_tar.py:179
↓ 1 callers
Function
save_split_sharded_already_splitted_dataset
(dataset, path_save_dir_sharded_dataset, shard_size)
vision/m4/sourcing/data_collection/processors/web_document_extractor.py:274
↓ 1 callers
Function
save_split_sharded_dataset
(dataset, path_save_dir_sharded_dataset, shard_size)
vision/m4/sourcing/data_collection/processors/web_document_extractor.py:299
↓ 1 callers
Method
save_split_sharded_dataset
(self, reload_files=False)
vision/m4/sourcing/data_collection/processors/web_document_extractor.py:434
↓ 1 callers
Function
save_web_document_example_in_files_with_num_shards
( example, idx, saving_dir, num_examples_per_shard, save_shard_prefix )
vision/m4/utils/datasets/create_webdataset_tar.py:71
↓ 1 callers
Method
select_mode
(self)
vision/m4/sourcing/data_collection/visualization/web_document_and_filtering_visualization.py:665
↓ 1 callers
Method
set_title
(self)
vision/m4/sourcing/data_collection/visualization/web_document_visualization.py:18
↓ 1 callers
Method
set_title
(self)
vision/m4/sourcing/data_collection/visualization/web_document_and_filtering_visualization.py:52
↓ 1 callers
Function
set_trainable_params
Freezes all parameters, then selectively unfreezes based on user flags: - tune_vision_tower => unfreeze vision tower - tune_mm_connecto
vision/smolvlm2/smolvlm/train/train.py:76
↓ 1 callers
Method
setup_batch_size_related_configs
batch_size-related configs are processed here. All this work is done here because it requires knowing the value of num_processes
vision/m4/training/trainer.py:242
↓ 1 callers
Function
should_process
Heuristics to decide whether to upload this opt_step-XXX checkpoint or not
vision/m4/scripts/s3-upload-checkpoints.py:76
↓ 1 callers
Function
should_process
Heuristics to decide whether to convert this opt_step-XXX checkpoint or not
vision/m4/scripts/convert-checkpoints.py:62
↓ 1 callers
Function
should_process
Heuristics to decide whether to cleanup this opt_step-XXX checkpoint or not
vision/m4/scripts/cleanup-checkpoints.py:71
↓ 1 callers
Method
show_draft_input
(self, summary_x, summary_y, summary_width)
tools/smol_tools/demo_tkinter.py:272
↓ 1 callers
Function
similarity_score
(a_ij, o_q_i, tau=0.5)
vision/m4/evaluation/custom_metrics/doc_vqa_metrics.py:32
↓ 1 callers
Method
simplification_mode
(self)
vision/data/datasets_processing_scripts/clean_m4_prelimenary_experiments/explore/global_visualization.py:153
↓ 1 callers
Method
simplification_mode
(self)
vision/m4/sourcing/data_collection/visualization/global_visualization.py:124
↓ 1 callers
Method
simplification_mode
(self)
vision/m4/sourcing/data_collection/visualization/wikipedia/global_visualization.py:123
↓ 1 callers
Function
split_batch
(batch, chunk_size)
vision/m4/evaluation/utils.py:74
↓ 1 callers
Method
split_on_whitespace
This method also removes concatenated spaces.
vision/m4/sourcing/data_collection/processors/web_document_filtering.py:91
↓ 1 callers
Method
start_new_chat
(self, chat_display)
tools/smol_tools/demo_tkinter.py:598
↓ 1 callers
Method
state_dict
(self)
vision/m4/training/dataset.py:1388
↓ 1 callers
Method
statistics_without_filtering
(self)
vision/m4/sourcing/data_collection/visualization/pair_visualization.py:151
↓ 1 callers
Method
stop
(self)
vision/m4/utils/training/timer.py:60
↓ 1 callers
Function
strip_string
(string, skip_unit=False)
text/evaluation/smollm2/math_utils.py:209
↓ 1 callers
Function
summary_failures_short
(tr)
vision/m4/testing_utils.py:970
↓ 1 callers
Method
tokenization
(text, sentencepiece_model, join_on_whitespace)
vision/m4/sourcing/data_collection/processors/web_document_filtering.py:399
↓ 1 callers
Method
tokenize_and_count_tokens
(self, list_texts: List[str])
vision/data/datasets_processing_scripts/build_concatenation_datasets_sft/merge_on_image_individual_dataset.py:146
↓ 1 callers
Function
tokenize_id
Tokenize the id string on common delimiters: hyphens, underscores, and periods. Return a set of non-empty tokens. For example: "
vision/smolvlm2/scripts/sort.py:28
↓ 1 callers
Function
trainer_save_model_safe
Safely saves the model if not in DeepSpeed ZeRO stage-3.
vision/smolvlm2/smolvlm/train/train.py:50
↓ 1 callers
Function
transform_img
(img)
vision/data/datasets_processing_scripts/build_webdocs_dataset/python_scripts/07_02_nsfw_image_visualization.py:68
↓ 1 callers
Function
transform_img
(img)
vision/m4/sourcing/data_collection/visualization/choose_filtering_parameters_web_documents_node_level.py:216
↓ 1 callers
Function
transform_img
(img)
vision/m4/sourcing/data_collection/visualization/get_stats_vis_choose_filtering_parameters_laion_pairs.py:52
↓ 1 callers
Function
translate_nested_expression
(expr)
vision/data/datasets_processing_scripts/build_concatenation_datasets_sft/build_ds_sft.py:3982
↓ 1 callers
Function
translate_sequential_expressions
(input_str)
vision/data/datasets_processing_scripts/create_fine_tuning_datasets/create_table_datasets.py:669
↓ 1 callers
Function
unroll_list
(list_)
vision/data/datasets_processing_scripts/build_webdocs_dataset/python_scripts/08_01_prepare_urldedup.py:25
↓ 1 callers
Function
unwrap_exception
(wrapped_function)
vision/m4/sourcing/pmd/helpers.py:92
↓ 1 callers
Method
update_agent_output
(self, text_widget, new_text)
tools/smol_tools/demo_tkinter.py:499
↓ 1 callers
Method
update_before_collapsing_meta
(self, batch: Dict)
vision/m4/sourcing/pmd/loader_builder.py:33
↓ 1 callers
Function
update_df_metrics_and_lists_for_extraction_method
_summary_ Given a Media_Node and the Extraction_Method_Name used to get this Media_Node, this function uses the Media_Node's values to update
vision/m4/sourcing/data_collection/visualization/pair_stat_dashboard.py:94
↓ 1 callers
Method
update_improved_text
(self, text_widget, new_text)
tools/smol_tools/demo_tkinter.py:441
↓ 1 callers
Function
update_model_kwargs_for_generation
(outputs, model_kwargs)
vision/m4/models/vmistral/modeling_vmistral.py:197
↓ 1 callers
Function
update_sampling_strategies
Update each dataset entry's sampling_strategy according to these rules: - If always_full applies, use "random:100%". - Otherwise, com
vision/smolvlm2/scripts/create_mixture.py:123
↓ 1 callers
Function
urls_to_images
(web_document_dataset_without_images, image_dataset, map_url_idx, url_ban_words, num_proc)
vision/data/datasets_processing_scripts/build_webdocs_dataset/python_scripts/04_merge_web_docs_with_images.py:70
↓ 1 callers
Method
urls_to_images
(self, reload_files=False)
vision/m4/sourcing/data_collection/processors/web_document_extractor.py:396
↓ 1 callers
Function
validate_media_paths
Validate paths for a sample of elements from the dataset. Returns percentage of failed paths.
vision/smolvlm2/scripts/mixture_checker.py:7
↓ 1 callers
Function
validate_optim_states_are_reset
for a new or fully reset optimizer we expect all zeros `exp_avg` and `exp_avg_sq` state tensors and step=1
vision/m4/training/debug_utils.py:21
↓ 1 callers
Method
visualization
(self)
vision/data/datasets_processing_scripts/clean_m4_prelimenary_experiments/explore/global_visualization.py:109
↓ 1 callers
Method
visualization
(self)
vision/m4/sourcing/data_collection/visualization/global_visualization.py:80
↓ 1 callers
Method
visualization
(self)
vision/m4/sourcing/data_collection/visualization/web_document_visualization.py:12
↓ 1 callers
Method
visualization
(self)
vision/m4/sourcing/data_collection/visualization/web_document_and_filtering_visualization.py:44
↓ 1 callers
Method
visualization
(self)
vision/m4/sourcing/data_collection/visualization/pair_visualization.py:22
↓ 1 callers
Method
visualization
(self)
vision/m4/sourcing/data_collection/visualization/wikipedia/global_visualization.py:79
↓ 1 callers
Method
visualize
(self)
vision/m4/sourcing/data_collection/visualization/pair_stat_dashboard.py:267
↓ 1 callers
Function
vqa_normalize_text
Process a text Source: https://github.com/GT-Vision-Lab/VQA/blob/master/PythonEvaluationTools/vqaEvaluation/vqaEval.py 1. Conversion of charac
vision/m4/evaluation/custom_metrics/utils.py:150
↓ 1 callers
Method
vqa_normalize_text
(self, text)
vision/m4/evaluation/custom_metrics/utils.py:373
↓ 1 callers
Function
wrap_exception
(unwrapped_function)
vision/m4/sourcing/pmd/helpers.py:83
↓ 1 callers
Function
write_yaml
(data, file_path)
vision/smolvlm2/scripts/create_mixture.py:115
Function
ExtendSysPath
Temporary add given path to `sys.path`. Usage :: with ExtendSysPath('/path/to/dir'): mymodule = importlib.import_module('
vision/m4/testing_utils.py:527
Method
_DATASETS_NAME
(self)
vision/m4/sourcing/pmd/loader_builder.py:38
Method
_DATASETS_NAME
(self)
vision/m4/sourcing/pmd/loader_builder.py:118
Method
__call__
(self, example)
vision/data/datasets_processing_scripts/build_laion_coco_dataset/python_scripts/02_02_remove_opt_out.py:51
Method
__call__
(self, example)
vision/data/datasets_processing_scripts/build_laion_coco_dataset/python_scripts/make_laion_coco_1_4_dataset.py:47
Method
__call__
(self, example)
vision/data/datasets_processing_scripts/build_laion_coco_dataset/python_scripts/make_laion_coco_1_4_dataset.py:68
Method
__call__
(self, example)
vision/data/datasets_processing_scripts/build_laion_coco_dataset/python_scripts/make_laion_coco.py:46
Method
__call__
(self, example)
vision/data/datasets_processing_scripts/build_laion_coco_dataset/python_scripts/make_laion_coco.py:67
Method
__call__
(self, batch)
vision/data/datasets_processing_scripts/build_laion_coco_dataset/python_scripts/05_binary_classification.py:104
← previous
next →
901–1,000 of 1,907, ranked by callers