MCPcopy Create free account
hub / github.com/EZ-hwh/AutoScraper / build_tree

Function build_tree

utils/step_test.py:78–92  ·  view source on GitHub ↗
(html, attrs=[])

Source from the content-addressed store, hash-verified

76 return new_node.get_index(), tot_node
77
78def build_tree(html, attrs=[]):
79 tot_node = 0
80 node_list = []
81 html = html.replace(' ',' ')
82 html = html.replace('&lt;', '<')
83 html = html.replace('&gt;', '>')
84 soup = BeautifulSoup(html, 'html.parser')
85 for element in soup(text=lambda text: isinstance(text, Comment)):
86 element.extract()
87 [s.extract() for s in soup('script')]
88 [s.extract() for s in soup('style')]
89 [s.extract() for s in soup('textarea')]
90 build_dom_tree(soup.html, tot_node, -1, node_list, attrs=attrs)
91 node_list = sorted(node_list, key=lambda x:x.get_index())
92 return node_list
93
94def get_max_index(parse_tree, index): # get the max index of the subtree (must tag node not text node)
95 if len(parse_tree[index].get_children()) == 0:

Callers 3

web2treeFunction · 0.85
domlm_datasetFunction · 0.85
step_test.pyFile · 0.85

Calls 2

build_dom_treeFunction · 0.85
get_indexMethod · 0.80

Tested by

no test coverage detected