MCPcopy
hub / github.com/hu17889/go_spider / HttpDownloader

Struct HttpDownloader

core/downloader/downloader_http.go:31–32  ·  view source on GitHub ↗

The HttpDownloader download page by package net/http. The "html" content is contained in dom parser of package goquery. The "json" content is saved. The "jsonp" content is modified to json. The "text" content will save body plain text only. The page result is saved in Page.

Source from the content-addressed store, hash-verified

29// The "text" content will save body plain text only.
30// The page result is saved in Page.
31type HttpDownloader struct {
32}
33
34func NewHttpDownloader() *HttpDownloader {
35 return &HttpDownloader{}

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected