Functions — github.com/EZ-hwh/AutoScraper

↓ 19 callersFunctionsimplify_html

(html, reserve_attrs = ['class'])

utils/html_utils.py:75

↓ 11 callersMethodrequest_parse

A safe and reliable call to LLMs, which confirm that the output can be parsed by json.loads(). Args: query (str): the query to pr

module/reflexion_crawler_tmp.py:41

↓ 9 callersMethodget_tag

(self)

utils/step_test.py:40

↓ 8 callersMethodrequest_parse

A safe and reliable call to LLMs, which confirm that the output can be parsed by json.loads(). Args: query (str): the query to pr

module/stepback_crawler.py:105

↓ 7 callersMethodget_children

(self)

utils/step_test.py:29

↓ 7 callersMethodget_tokens

(self)

utils/step_test.py:34

↓ 7 callersMethodrequest_parse

A safe and reliable call to LLMs, which confirm that the output can be parsed by json.loads(). Args: query (str): the query to pr

module/reflexion_crawler.py:36

↓ 5 callersMethodextract_with_xpath

Xpath Parser Args: html_content (str): text of HTML xpath (str): the string of xpath Returns: li

module/reflexion_crawler.py:278

↓ 5 callersFunctionfind_common_ancestor

(html_content:str, xpath:str)

utils/html_utils.py:19

↓ 5 callersMethodget_text

(self)

utils/step_test.py:37

↓ 5 callersFunctionnum_tokens_from_string

(string: str, encoding_name: str)

utils/html_utils.py:107

↓ 4 callersMethod__init__

(self)

module/prompt.py:8

↓ 4 callersFunctionbuild_tree

(html, attrs=[])

utils/step_test.py:78

↓ 4 callersFunctiondomlm_parse

(soup:Tag, max_len)

utils/step.py:21

↓ 4 callersMethodextract_with_code

Code parser Args: html_content (str): _description_ code (str): _description_ Returns: list[str]

module/reflexion_crawler_tmp.py:410

↓ 4 callersMethodextract_with_selector

CSS Selector parser Args: html_content (str): text of HTML selector (str): the string of CSS selector Return

module/reflexion_crawler_tmp.py:388

↓ 4 callersMethodextract_with_xpath

Xpath Parser Args: html_content (str): text of HTML xpath (str): the string of xpath Returns: li

module/reflexion_crawler_tmp.py:347

↓ 4 callersMethodgenerate_sequence

(self, instruction, html_content, ground_truth = None, max_token=8000)

module/stepback_crawler.py:223

↓ 4 callersFunctionget_max_index

(parse_tree, index)

utils/step_test.py:94

↓ 3 callersMethodextract_with_sequence

(self, html_content:str, sequence:str)

module/stepback_crawler.py:372

↓ 3 callersMethodextract_with_xpath

Xpath Parser Args: html_content (str): text of HTML xpath (str): the string of xpath Returns: li

module/stepback_crawler.py:349

↓ 3 callersMethodget_index

(self)

utils/step_test.py:30

↓ 3 callersMethodget_str

(self)

utils/step_test.py:31

↓ 3 callersFunctiontree2text

(node_list, index, simplify=True, add_tags = True)

utils/step_test.py:110

↓ 2 callersMethodappend_children

(self, index)

utils/step_test.py:33

↓ 2 callersMethodgenerate_rule

(self, instruction, html_content, ground_truth = None, max_token=8000)

module/reflexion_crawler.py:200

↓ 2 callersMethodgenerate_rule

(self, instruction, html_content, ground_truth = None, max_token=8000)

module/reflexion_crawler_tmp.py:264

↓ 2 callersMethodgenerate_sequence_html

(self, instruction: str, html_content: str,

module/stepback_crawler.py:138

↓ 2 callersFunctionnormalize

(text)

run_klarna/evaluate.py:53

↓ 2 callersFunctionnormalize

(text)

run_swde/evaluate.py:42

↓ 2 callersFunctionnormalize_list

(text_list)

run_klarna/evaluate.py:82

↓ 2 callersFunctionnormalize_list

(text_list)

run_swde/evaluate.py:71

↓ 2 callersMethodreflexion_generate

(self, res, instruction, html_content, ground_truth=None, reflection_times=3)

module/reflexion_crawler.py:73

↓ 2 callersMethodreflexion_generate

(self, res, instruction, html_content, ground_truth=None, reflection_times=3)

module/reflexion_crawler_tmp.py:79

↓ 2 callersMethodrule_synthesis

(self, website_name: str, seed_html_set: list[str],

module/reflexion_crawler.py:231

↓ 2 callersMethodset_str

(self, strx)

utils/step_test.py:35

↓ 2 callersMethodset_tag

(self, tag)

utils/step_test.py:39

↓ 1 callersFunctionattrs_dict2str

(attrs)

utils/step_test.py:104

↓ 1 callersFunctionbuild_absolute_xpath

Constructs an absolute XPath expression for a given node. :param node: A lxml node for which to build the XPath :return: A string represe

utils/html_utils.py:60

↓ 1 callersFunctionbuild_dom_tree

(node, tot_node, father_index, node_list, attrs)

utils/step_test.py:48

↓ 1 callersFunctionchatgpt

(query)

utils/api.py:8

↓ 1 callersFunctiondomlm_dataset

(html_content, max_len=1024, simplify=True, prettify = True, add_tags = True, attrs=[])

utils/step_test.py:201

↓ 1 callersFunctiondomlm_parser

(parse_tree, max_len=1024)

utils/step_test.py:194

↓ 1 callersFunctionfit_tree

(parse_tree, index, curr_len, valid_index, max_len=1024)

utils/step_test.py:149

↓ 1 callersMethodgenerate_rule_html

Generate rule by asking LLM with an instruction and HTML code. Args: instruction (str): Task description html (str):

module/reflexion_crawler.py:120

↓ 1 callersFunctionget_absolute_xpath

Given an HTML string and an XPath expression, returns the absolute XPath of the element. :param html: HTML content as a string :param xpa

utils/html_utils.py:41

↓ 1 callersMethodget_attr

(self)

utils/step_test.py:42

↓ 1 callersFunctionload_file

(filename)

crawler_generation.py:110

↓ 1 callersFunctionload_file

(filename)

crawler_count.py:110

↓ 1 callersFunctionload_file

(filename)

run_swde/evaluate.py:31

↓ 1 callersFunctionload_gt

(file_path, schema_list)

run_klarna/evaluate.py:43

↓ 1 callersFunctionload_value

(html_content, item_name)

run_klarna/evaluate.py:30

↓ 1 callersMethodset_attr

(self, attrs)

utils/step_test.py:41

↓ 1 callersMethodset_text

(self, text)

utils/step_test.py:36

↓ 1 callersMethodset_tokens

(self, tokens)

utils/step_test.py:38

↓ 1 callersFunctionweb2tree

(html_content, attrs=[])

utils/step_test.py:100

Method__init__

(self, index, findex)

utils/step_test.py:20

Method__init__

(self, html, step_len = 5000, len_func = len)

utils/html_utils.py:10

Method__init__

(self)

module/prompt.py:14

Method__init__

(self)

module/prompt.py:204

Method__init__

(self)

module/prompt.py:280

Method__init__

(self)

module/prompt.py:364

Method__init__

(self, simplify=True, verbose=True, api=None,

module/stepback_crawler.py:92

Method__init__

Initial an instance of Autocrawler, including setting the pattern of rule Args: pattern (str, optional): Which kind of rule patt

module/reflexion_crawler.py:12

Method__init__

Initial an instance of Autocrawler, including setting the pattern of rule Args: pattern (str, optional): Which kind of rule patt

module/reflexion_crawler_tmp.py:12

Method__iter__

(self)

utils/html_utils.py:15

Functioncalculate_dom_depth

Recursively calculates the depth of the DOM tree. Args: - element: BeautifulSoup object or tag to calculate depth for. - depth: curre

utils/step.py:5

Methodextract_with_seq

(self, html_content:str, xpath_seq:str)

module/reflexion_crawler_tmp.py:371

Methodgenerate_rule_html

Generate rule by asking LLM with an instruction and HTML code. Args: instruction (str): Task description html (str):

module/reflexion_crawler_tmp.py:166

Methodget_father

(self)

utils/step_test.py:32

Functionnode_counter

(node)

utils/step_test.py:44

Functionparse_accessibility_tree

(html, indent='\t')

utils/html_utils.py:93

Functionpre_tokenize

(s)

utils/step_test.py:10

Methodrule_synthesis

(self, website_name: str, seed_html_set: list[str],

module/stepback_crawler.py:257

Methodrule_synthesis

(self, website_name: str, seed_html_set: list[str],

module/reflexion_crawler_tmp.py:278

Methodrule_synthesis_cul

(self, website_name: str, seed_html_set: list[str],

module/stepback_crawler.py:303

Functionweb2text

(html_content, simplify=True, prettify = True, add_tags = True, attrs=[])

utils/step_test.py:141

Functions77 in github.com/EZ-hwh/AutoScraper