MCPcopy Create free account
hub / github.com/amp-labs/connectors / QueryLoadableHTML

Function QueryLoadableHTML

tools/scrapper/utils.go:22–36  ·  view source on GitHub ↗

QueryLoadableHTML this method can panic. Uses browser internally to wait for all HTML parts to be fully loaded.

(url string, wait int64)

Source from the content-addressed store, hash-verified

20// QueryLoadableHTML this method can panic.
21// Uses browser internally to wait for all HTML parts to be fully loaded.
22func QueryLoadableHTML(url string, wait int64) *goquery.Document { //nolint:gocritic
23 htmlPage, err := loadHTMLPage(url, wait)
24 if err != nil {
25 log.Fatal(err)
26 }
27
28 reader := strings.NewReader(htmlPage)
29
30 doc, err := goquery.NewDocumentFromReader(reader)
31 if err != nil {
32 log.Fatal(err)
33 }
34
35 return doc
36}
37
38func loadHTMLPage(url string, wait int64) (string, error) {
39 ctx, cancel := chromedp.NewContext(context.Background())

Callers 2

queryHTMLFunction · 0.92
QueryHTMLFunction · 0.85

Calls 1

loadHTMLPageFunction · 0.85

Tested by

no test coverage detected