MCPcopy Create free account

hub / github.com/Lichanglu/pholcus / functions

Functions266 in github.com/Lichanglu/pholcus

⨍ Functions 266◇ Types & classes 39
↓ 61 callersMethodGetOutFeild
()
spiders/spider/spider.go:137
↓ 27 callersMethodPrintf
(format string, v ...interface{})
reporter/reporter.go:4
↓ 26 callersMethodGetHtmlParser
GetHtmlParser returns goquery object binded to target crawl result.
downloader/context/response.go:120
↓ 25 callersMethodPrintln
(v ...interface{})
reporter/reporter.go:5
↓ 18 callersMethodAddQueue
(param map[string]interface{})
spiders/spider/spider.go:91
↓ 18 callersMethodGetKeyword
()
spiders/spider/spider.go:42
↓ 18 callersMethodGetTemp
(key string)
downloader/context/request.go:240
↓ 14 callersMethodGetName
()
spiders/spider/spider.go:34
↓ 13 callersMethodAidRule
调用指定规则下辅助函数
spiders/spider/spider.go:62
↓ 11 callersMethodAddItem
AddField saves KV string pair to ResponseItems preparing for Pipeline
downloader/context/response.go:79
↓ 9 callersMethodGetRuleName
()
downloader/context/request.go:199
↓ 8 callersMethodSetStatus
SetStatus save status info about download process.
downloader/context/response.go:73
↓ 8 callersMethodWrite
(p []byte)
pholcus/gui/logview.go:79
↓ 7 callersMethodAdd
(sp *Spider)
spiders/spider/spiderlist.go:11
↓ 6 callersMethodCallRule
用指定规则解析响应流
spiders/spider/spider.go:56
↓ 6 callersMethodGetUrl
()
downloader/context/request.go:187
↓ 6 callersMethodLoopAddQueue
(loop [2]int, urlFn func(int) []string, param map[string]interface{})
spiders/spider/spider.go:77
↓ 6 callersMethodNewRequest
生成请求 param全部参数列表 req := &Request{ url: param["url"].(string), //必填 parent: "", //若有必填 rule: pa
spiders/spider/spider.go:114
↓ 6 callersMethodPush
存入
scheduler/src_manage.go:11
↓ 5 callersFunctionCleanHtml
(str string, depth int)
spiders/spider/common.go:14
↓ 5 callersMethodClose
Close makes log close.
common/mlog/mlog.go:29
↓ 5 callersMethodInit
(*spider.Spider)
pholcus/crawler/crawler.go:8
↓ 5 callersMethodIsSucc
IsSucc test whether download process success or not.
downloader/context/response.go:63
↓ 3 callersMethodGetId
()
spiders/spider/spider.go:38
↓ 3 callersMethodGlobalGet
(key string)
common/config/config.go:114
↓ 3 callersFunctionNewDocker
()
pipeline/collector/docker.go:17
↓ 3 callersFunctionPhoSocketAcceptData
* *@共用 * *PhoSocketAcceptData()为幽灵蛛socket数据接收函数 *conn为握手连接 *通过conn接收client发送来的Data */
pholcus/socket/PhoSocketTest.go:100
↓ 3 callersMethodRun
()
pholcus/crawler/crawl.go:54
↓ 3 callersMethodSetBodyStr
SetBodyStr saves plain string crawled in Response.
downloader/context/response.go:103
↓ 3 callersMethodStart
()
pholcus/crawler/crawler.go:9
↓ 3 callersMethodString
()
common/config/config.go:88
↓ 3 callersMethoddownloadFile
Download file and change the charset of response charset.
downloader/downloader_http.go:222
↓ 3 callersFunctiongetResponse
请求获取响应流
pholcus/keeper/login.go:73
↓ 2 callersMethodCollect
收集数据单元
pipeline/pipeline.go:18
↓ 2 callersMethodCompare
采集非重复url并返回对比结果，重复为true
scheduler/scheduler.go:10
↓ 2 callersFunctionEncodeString
(src, charset string)
spiders/spider/common.go:62
↓ 2 callersMethodGetHeader
()
downloader/context/request.go:175
↓ 2 callersMethodGetProxyHost
()
downloader/context/request.go:179
↓ 2 callersMethodGetRespType
()
downloader/context/request.go:211
↓ 2 callersMethodGetRules
()
spiders/spider/spider.go:46
↓ 2 callersMethodGlobalGetInt
(key string)
common/config/config.go:122
↓ 2 callersMethodGoRule
根据响应流运行指定解析规则
spiders/spider/spider.go:51
↓ 2 callersFunctionNewDockerQueue
()
pipeline/collector/docker.go:21
↓ 2 callersFunctionPhoSocketAcceptData
* *@共用 * *PhoSocketAcceptData()为幽灵蛛socket数据接收函数 *conn为握手连接 *通过conn接收client发送来的Data */
pholcus/socket/PhoSocket.go:100
↓ 2 callersFunctionPhoSocketSendData
* *@共用 * *PhoSocketSendData()为幽灵蛛socket数据发送函数,但是不关闭 *conn为握手连接，sendData为要发送的数据 *通过conn给client发送Data */
pholcus/socket/PhoSocketTest.go:88
↓ 2 callersMethodSectionGet
(section string, key string)
common/config/config.go:203
↓ 2 callersMethodSum
统计数据总量
pipeline/collector/collector.go:114
↓ 2 callersMethodexcel
************************ excel 输出 ***************************/
pipeline/collector/output_lib.go:18
↓ 2 callersMethodgoOutput
(dataIndex int)
pipeline/collector/collector.go:105
↓ 2 callersMethodlog
(lable string, str string)
common/mlog/filelog.go:77
↓ 2 callersFunctionreadHeaderFromFile
(headerFile string)
downloader/context/request.go:131
↓ 2 callersMethodsend
(str string)
reporter/report.go:10
↓ 1 callersFunctionAcceptAndSendClose
接收并发送，完关闭
pholcus/socket/PhoSocket.go:112
↓ 1 callersMethodAppendText
(value string)
pholcus/gui/logview.go:60
↓ 1 callersMethodAutoAdd
根据情况自动动态增加Docker
pipeline/collector/docker.go:66
↓ 1 callersMethodBulkAddQueue
(urls []string, param map[string]interface{})
spiders/spider/spider.go:84
↓ 1 callersMethodChange
()
pipeline/collector/docker.go:43
↓ 1 callersMethodChecked
Called by the TableView to retrieve if a given row is checked.
pholcus/gui/guispider.go:81
↓ 1 callersMethodCompare
(url string)
scheduler/scheduler.go:47
↓ 1 callersFunctionConf
Conf gets singleton instance of Config.
common/etc_config/etc_config.go:41
↓ 1 callersMethodCtrlLen
()
pipeline/collector/collector.go:59
↓ 1 callersMethodCtrlR
接收控制通知
pipeline/pipeline.go:14
↓ 1 callersMethodCtrlS
()
pipeline/collector/collector.go:49
↓ 1 callersFunctionDecodeString
func Encode(src string) (dst string) { data, err := ioutil.ReadAll(transform.NewReader(bytes.NewReader([]byte(src)), simplifiedchinese.GBK.NewEncoder
spiders/spider/common.go:58
↓ 1 callersMethodDownload
(req *context.Request)
downloader/downloader.go:11
↓ 1 callersMethodExchange
二次使用Queue实例时，根据容量需求进行高效转换
common/queue/queue.go:40
↓ 1 callersMethodFree
释放一个资源
scheduler/src_manage.go:15
↓ 1 callersMethodFreeOne
从调度释放一个资源空位
pholcus/crawler/crawl.go:144
↓ 1 callersFunctionGBKToUTF8
(src string)
spiders/spider/common.go:75
↓ 1 callersMethodGetBodyStr
GetBodyStr returns plain string crawled.
downloader/context/response.go:109
↓ 1 callersMethodGetChecked
获取被选中的结果
pholcus/gui/guispider.go:93
↓ 1 callersMethodGetCookies
()
downloader/context/request.go:223
↓ 1 callersMethodGetItems
()
downloader/context/response.go:87
↓ 1 callersMethodGetMethod
()
downloader/context/request.go:215
↓ 1 callersMethodGetOne
从调度读取一个请求
pholcus/crawler/crawl.go:139
↓ 1 callersMethodGetParent
()
downloader/context/request.go:195
↓ 1 callersMethodGetPostdata
()
downloader/context/request.go:219
↓ 1 callersMethodGetRedirectFunc
()
downloader/context/request.go:183
↓ 1 callersMethodGetSpiderId
()
downloader/context/request.go:252
↓ 1 callersMethodGetTemps
()
downloader/context/request.go:244
↓ 1 callersMethodGlobalGetSlice
(key string, separator string)
common/config/config.go:154
↓ 1 callersMethodHasSection
(section string)
common/config/config.go:186
↓ 1 callersFunctionHashString
(encode string)
common/util/util.go:113
↓ 1 callersMethodInit
(size int)
common/queue/queue.go:15
↓ 1 callersMethodInit
重置
pipeline/pipeline.go:22
↓ 1 callersFunctionInitDockerParam
(dockercap uint)
config/config.go:65
↓ 1 callersFunctionInitFilelog
The InitFilelog is init the flog.
common/mlog/filelog.go:33
↓ 1 callersFunctionInitSpiders
初始化蜘蛛列表,必须在用户前端输入之后执行！
pholcus/gui/guimain.go:199
↓ 1 callersMethodIsEmpty
资源队列是否闲置
scheduler/src_manage.go:17
↓ 1 callersMethodIsOutsource
()
pholcus/pholcus.go:41
↓ 1 callersFunctionIsRedirectFunc
这是一个GetCookie函数的分支，当有302跳转的时候执行此函数
pholcus/keeper/login.go:22
↓ 1 callersMethodLoad
Load reads config file and returns an initialized Config.
common/config/config.go:26
↓ 1 callersMethodLoadString
(s string)
common/config/config.go:46
↓ 1 callersMethodManage
数据转储输出
pipeline/collector/collector.go:64
↓ 1 callersFunctionNew
(capacity uint)
scheduler/scheduler.go:31
↓ 1 callersFunctionNewGUISpiderModel
(list []*GUISpiderCore)
pholcus/gui/guispider.go:36
↓ 1 callersFunctionNewLogView
(parent walk.Container)
pholcus/gui/logview.go:23
↓ 1 callersFunctionNewRSS
(src map[string]string, level []int)
spiders/spider/rss.go:22
↓ 1 callersFunctionNewSrcManage
(capacity uint)
scheduler/src_manage.go:26
↓ 1 callersFunctionNoRedirectFunc
这是一个GetCookie函数的分支，当没有有302跳转的时候执行此函数
pholcus/keeper/login.go:57
next →1–100 of 266, ranked by callers