Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/cld378632668/SinaWeiboCrawler
/ functions
Functions
84 in github.com/cld378632668/SinaWeiboCrawler
⨍
Functions
84
◇
Types & classes
12
↓ 37 callers
Method
get
根据URL,get网页 @param url @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:219
↓ 9 callers
Method
e
(int j, int m)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:91
↓ 8 callers
Method
getHTML
(String url, JTextArea jta)
src/cn/edu/xmu/cld/WeiboCrawler2/HTML.java:33
↓ 6 callers
Method
parse
(String html)
src/cn/edu/xmu/cld/WeiboCrawler2/HTMLParser.java:40
↓ 5 callers
Method
encodeAccount
(String account)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:565
↓ 5 callers
Method
getFilePath
()
src/cn/edu/xmu/cld/WeiboCrawler2/JWindowsFrame.java:55
↓ 5 callers
Method
open
()
src/cn/edu/xmu/cld/WeiboCrawler2/JWindowsFrame.java:46
↓ 4 callers
Method
getHTMLbyProxy
(String targetURL, String hostName, int port, JTextArea jta)
src/cn/edu/xmu/cld/WeiboCrawler2/HTML.java:64
↓ 3 callers
Method
d
(int j, int k)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:97
↓ 3 callers
Method
hex_sha1
(String j)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:19
↓ 3 callers
Method
makeNonce
(int len)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:578
↓ 3 callers
Method
write2txt
(Vector<String> vector, String savePath)
src/cn/edu/xmu/cld/WeiboCrawler2/FileOperation.java:28
↓ 2 callers
Method
encode
(String psw, String servertime, String nonce)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:13
↓ 2 callers
Method
getCookieStore
(CookieStore cookiestore)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiBoUser.java:32
↓ 2 callers
Method
getJs
()
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:119
↓ 2 callers
Method
html2String
由html文件得到微博 @param html @return @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/FileOperation.java:65
↓ 2 callers
Method
login
(String username, String password )
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:58
↓ 2 callers
Method
preLogin
新浪微博预登录,获取密码加密公钥 @param unameBase64 @return 返回从结果获取的参数的哈希表 @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:236
↓ 1 callers
Method
a
(int k, int j, int m, int l)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:70
↓ 1 callers
Method
action
()
src/cn/edu/xmu/cld/WeiboCrawler2/JWindowsFrame.java:316
↓ 1 callers
Method
b
(int[] A, int r)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:33
↓ 1 callers
Method
c
(int j)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:86
↓ 1 callers
Method
crawl
此函数调用底层函数,执行具体的爬取微博的过程 并保存微博搜索结果页面html文件 @param totalPage @param searchword @throws ClientProtocolException @throws URISyntaxException @throws IOExcep
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:142
↓ 1 callers
Method
create
(HttpContext context)
src/cn/edu/xmu/cld/WeiboCrawler2/HTML.java:103
↓ 1 callers
Method
create
(HttpContext context)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:216
↓ 1 callers
Method
dump
打印页面 @param entity @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:270
↓ 1 callers
Method
excute
excute()是类的主函数,主要逻辑: allIPs = getAllProxyIPs(ipLibURL) 从代理ip网站首页网址获取所有代理IP validIPs = getValidProxyIPs(allIPs) 从所有代理IP中获取有效代理Ip iPsSupportedWeibo = ne
src/cn/edu/xmu/cld/WeiboCrawler2/ProxyIP.java:337
↓ 1 callers
Method
excute
(String[] args, JTextArea jta)
src/cn/edu/xmu/cld/WeiboCrawler2/Crawler.java:43
↓ 1 callers
Method
f
(String m, int r)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:101
↓ 1 callers
Method
filterIPsSupportSpeWebsite
过滤出能用于连接特定网址的proxyIps @param IPs - the Vector<String> contains all valid IPs @param outfilePath @return @throws ClientProtocolException
src/cn/edu/xmu/cld/WeiboCrawler2/ProxyIP.java:269
↓ 1 callers
Method
getAllProxyIPs
Get all unverified proxy IP in all IP library links. @param ipLibURL specified URL "http://www.youdaili.cn/" @return a String Vector contains all unv
src/cn/edu/xmu/cld/WeiboCrawler2/ProxyIP.java:101
↓ 1 callers
Method
getCommentNum
(String iniWeibo)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:411
↓ 1 callers
Method
getDate
(String iniWeibo)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:435
↓ 1 callers
Method
getForwardNum
(String iniWeibo)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:387
↓ 1 callers
Method
getHTML
由url得到html @param url @return html @throws URISyntaxException @throws ClientProtocolException @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:203
↓ 1 callers
Method
getIPsPageLinks
Find all IP library links on the homepage of "http://www.xici.net.co/". @param ipLibURL specified URL "http://www.youdaili.cn/" @return a
src/cn/edu/xmu/cld/WeiboCrawler2/ProxyIP.java:66
↓ 1 callers
Method
getLines
(String path)
src/cn/edu/xmu/cld/WeiboCrawler2/FileOperation.java:13
↓ 1 callers
Method
getMyTime
()
src/cn/edu/xmu/cld/WeiboCrawler2/Crawler.java:34
↓ 1 callers
Method
getMyTime
()
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:172
↓ 1 callers
Method
getParaFromResult
从新浪返回的结果字符串中获得参数 @param result @return
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:248
↓ 1 callers
Method
getPraisedNum
(String iniWeibo)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:366
↓ 1 callers
Method
getProxyIPs
获取一个HTML页面中所有的IPs There are proxy ips which haven't been verified on the page like ?? http://www.youdaili.cn/Daili/guonei/1843.html, use regex to ma
src/cn/edu/xmu/cld/WeiboCrawler2/ProxyIP.java:30
↓ 1 callers
Method
getServerTime
()
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:588
↓ 1 callers
Method
getUserName
(String iniWeibo)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:448
↓ 1 callers
Method
getUserid
(String iniWeibo)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:313
↓ 1 callers
Method
getValidProxyIPs
Test all proxy IPs and select the valid ones. @param a String Vector which contains all proxy IPs @return a String Vector which contains all valid IP
src/cn/edu/xmu/cld/WeiboCrawler2/ProxyIP.java:176
↓ 1 callers
Method
getWeiboInfo
(String htmlString)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:479
↓ 1 callers
Method
getWeiboid
(String iniWeibo)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:299
↓ 1 callers
Method
h
(int[] l)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:23
↓ 1 callers
Method
htmltoString
由html文件得到微博 @param html @return @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:284
↓ 1 callers
Method
isExistResult
是否存在指定页数的搜索结果页面 @param html @return
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:593
↓ 1 callers
Method
isVerification
(String html)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:576
↓ 1 callers
Method
runBroswer
用默认浏览器打开指定网址 @param url @throws URISyntaxException @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:188
↓ 1 callers
Method
setDisplayName
(String displayName)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiBoUser.java:17
↓ 1 callers
Method
setUserName
(String userName)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiBoUser.java:23
↓ 1 callers
Method
setUserPass
(String userPass)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiBoUser.java:29
↓ 1 callers
Method
splitHTML
(String html)
src/cn/edu/xmu/cld/WeiboCrawler2/HTMLParser.java:20
↓ 1 callers
Method
write2txt
(String searchword, String dirPath, String saveTXTPath)
src/cn/edu/xmu/cld/WeiboCrawler2/HTMLParser.java:160
↓ 1 callers
Method
writeString
把String写到本地文件 @param s @param savePath @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/FileOperation.java:48
↓ 1 callers
Method
writeString
把String写到本地文件 @param s @param savePath @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:270
↓ 1 callers
Method
writeVector
把某关键字搜索到的微博全写到文件中去 这里简单写成了txt格式 @param vector @param savePath @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:521
↓ 1 callers
Method
writeVector2xml
(Vector<String> vector, String saveXMLPath)
src/cn/edu/xmu/cld/WeiboCrawler2/HTMLParser.java:201
Method
AddW
(String u, String p, String text, byte[] pic)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:380
Method
JWindowsFrame
()
src/cn/edu/xmu/cld/WeiboCrawler2/JWindowsFrame.java:131
Method
OpenFile
()
src/cn/edu/xmu/cld/WeiboCrawler2/JWindowsFrame.java:38
Method
Share
(String u, String p, String content, String pic, String surl)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:281
Method
SinaSSOEncoder
()
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/SinaSSOEncoder.java:9
Method
actionPerformed
(ActionEvent e)
src/cn/edu/xmu/cld/WeiboCrawler2/JWindowsFrame.java:320
Method
getDisplayName
()
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiBoUser.java:14
Method
getRealHTML
(String htmlPath)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/ReParser.java:27
Method
getUserName
()
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiBoUser.java:20
Method
getUserPass
()
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiBoUser.java:26
Method
getWeiboSentence
(String iniWeibo)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:327
Method
isExistResult
是否存在指定页数的搜索结果页面 @param html @return
src/cn/edu/xmu/cld/WeiboCrawler2/Crawler.java:23
Method
main
(String[] args)
src/cn/edu/xmu/cld/WeiboCrawler2/JWindowsFrame.java:563
Method
main
(String[] args)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:49
Method
main
@param args @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:50
Method
readFileImage
(String filename)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/Sina.java:551
Method
run
()
src/cn/edu/xmu/cld/WeiboCrawler2/JWindowsFrame.java:473
Method
showData
(String s)
src/cn/edu/xmu/cld/WeiboCrawler2/JWindowsFrame.java:311
Method
utf8Parser
(String utf8)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/ReParser.java:19
Method
validate
(Cookie cookie, CookieOrigin origin)
src/cn/edu/xmu/cld/WeiboCrawler2/HTML.java:105
Method
validate
(Cookie cookie, CookieOrigin origin)
src/cn/edu/xmu/cld/WeiboCrawler2/useVersion2014/WeiboCrawler3.java:218
Method
writeVector
把某关键字搜索到的微博全写到文件中去 这里简单写成了txt格式 @param vector @param savePath @throws IOException
src/cn/edu/xmu/cld/WeiboCrawler2/FileOperation.java:86