Batches sentences from a dataset into chunks of a specified maximum number of words. Args: examples (Dataset): The dataset containin

Count the bins of the token counts using pandas. Parameters: token_counts (list): The list of token counts to count, such as dataset["__

Extract context and question from the text. Example usage: text = "question: what is ram's name? where is he from? what school is he from

This function returns the dictionary map for languages.

This function loads the language identification model.

Load the dataset from Hugging Face and count the tokens in the specified fields. Parameters: split (str): The split of the dataset to lo

Main function for executing the tokenization pipeline. Args: args (argparse.Namespace): Parsed command-line arguments.

Loads a dataset, batches the sentences, and returns the original and batched datasets. Args: dataset_name (str): The name of the dat

Parses command-line arguments. Returns: argparse.Namespace: Parsed arguments.

Plot the histogram of the token counts using matplotlib. Parameters: token_counts (list): The list of token counts to plot, such as data

Predict the source language of the text, pass the target language of the text. returns a dictionary with the source language and target language

Route the input based on the language. model,text = route_input(text,languages_dict)

Initializes the TokenizeText instance. Args: tokenizer (transformers.AutoTokenizer): The tokenizer to be used for tokeni

This function returns the instruction prompt for English.

This function returns the instruction prompt for Hindi.

This function returns the instruction prompt for Kannada.

This function returns the question answering prompt.

This function returns the translation prompt.

This function returns the translator object.

Launch the Gradio Interface

load an individual expert. choose one expert from huggingface.co/projectbaraat this function utilizes unsloth internally. model,tokenizer

Load models and tokenizers. Usage : hindi_model, hindi_tokenizer, eng_model, eng_tokenizer = load_models()

Route to the language expert based on the input. routed_model = model_route(routed_model)

Tokenizes the input text and yields the length of tokens. Args: text (str): Input text to be tokenized. Yields:

Process the text for translation. result = process_text(text, target_language,model,tokenizer, temperature=1,max_tokens=200)

Processes the total token count obtained. Args: total_token_count (int): Total count of tokens.

Select the task based on the input text. selected_prompt = select_task(input_text,languages_dict,model,tokenizer)