MCPcopy Create free account

hub / github.com/N0taN3rd/Squidwarc / functions

Functions175 in github.com/N0taN3rd/Squidwarc

↓ 1 callersMethodwarcNamePerURL
* @desc Returns a function that will concatenate the output path with the filenamified seedURL * producing the full path to WARC of the page being
lib/utils/warcNaming.js:32
↓ 1 callersFunctionwhich
* @desc Executes the which command for the supplied executable name * @param {string} executable
lib/launcher/chromeFinder.js:49
↓ 1 callersMethodwriteWRPlayerPagesRecord
()
lib/crawler/chrome.js:309
↓ 1 callersMethodwriteWRPlayerPagesRecord
()
lib/crawler/puppeteer.js:307
↓ 1 callersMethodyellow
* @desc Yellow colored console.log * @param args
lib/utils/colorPrinters.js:37
Method[Symbol.iterator]
* @desc Iterate over the captured network requests for the current web page * @return {Iterator<CapturedRequest>}
lib/crawler/chrome.js:369
Method[Symbol.iterator]
* @desc Iterate over the captured network requests for the current web page * @return {Iterator<CapturedRequest>}
lib/crawler/puppeteer.js:362
Method_close
* @desc Callback for process.on('exit') * @private
lib/crawler/chrome.js:398
Method_didNavigate
* @desc Callback used for Page.navigate * @private
lib/crawler/chrome.js:377
Method_emitNetIdle
* @desc Emit the network-idle event * @private
lib/crawler/netIdleWatcher.js:186
Method_globalNetworkTimeout
* @desc Called when the global time limit was hit * @private
lib/crawler/netIdleWatcher.js:162
Method_globalNetworkTimeout
* @desc Called when the global time limit was hit * @private
lib/crawler/navigationMan.js:205
Method_navTimedOut
* @desc Called when the navigation time limit was hit * @private
lib/crawler/navigationMan.js:193
Method_networkIdled
* @desc Called when the network idle has been determined * @private
lib/crawler/netIdleWatcher.js:174
Method_networkIdled
* @desc Called when the network idle has been determined * @private
lib/crawler/navigationMan.js:214
Method_onDisconnected
* @desc CB used to emit the disconnected event * @private
lib/crawler/puppeteer.js:107
Method_onWARCGenError
* @desc Listener for warc generator error * @param {Error} err * @private
lib/crawler/chrome.js:409
Method_onWARCGenError
* @desc Listener for warc generator error * @param {Error} err - The error to emit * @private
lib/crawler/puppeteer.js:371
Method_onWARCGenFinished
* @desc Listener for warc generator finished * @private
lib/crawler/chrome.js:417
Method_onWARCGenFinished
* @desc Listener for warc generator finished * @private
lib/crawler/puppeteer.js:379
MethodaddEventListeners
* @desc Register an array of eventName, eventListener pairs to an EventEmitter * @param {!EventEmitter} emitter - The EventEmitter to register the
lib/crawler/helper.js:27
MethodboldBlueGreen
* @desc Bold blue first arg, green rest * @param bb * @param rest
lib/utils/colorPrinters.js:104
FunctionconfigureURLNormalizer
* @desc wrapper around {@link https://github.com/sindresorhus/normalize-url} that * given a configuration for normalize-url returns a function that a
lib/utils/urlUtils.js:36
Methodconstructor
* @param {string} url - A starting seed * @param {symbol} mode - The mode for the seed * @param {number} depth - The crawl depth
lib/frontier/seedTracker.js:29
Methodconstructor
* @desc Create a new frontier object
lib/frontier/index.js:27
Methodconstructor
()
lib/injectManager/pageInjects/collectLinks.js:47
Methodconstructor
(collectorRef, messages)
lib/injectManager/pageInjects/collectLinks.js:160
Methodconstructor
* @param {Object} network * @param {NavigationMan} navMan
lib/crawler/chrome.js:31
Methodconstructor
* @desc Create a new ChromeCrawler instance. For a description of the expected options see the * JSDoc CrawlConfig typedef {@link CrawlConfig} *
lib/crawler/chrome.js:72
Methodconstructor
* @param {Page} page - Puppeteer page object for the page being crawled * @param {?NetIdleOptions} [options = {}] - Optional options to control fin
lib/crawler/netIdleWatcher.js:31
Methodconstructor
* @desc Create a new PuppeteerCrawler instance. For a description of the expected options see the * JSDoc CrawlConfig typedef {@link CrawlConfig}
lib/crawler/puppeteer.js:39
Methodconstructor
* * @param {CrawlControl} [options = {}] * @param {EventEmitter} [parentEmitter]
lib/crawler/navigationMan.js:32
Functioncsi
()
lib/injectManager/pageInjects/noNaughtyJS.js:57
FunctiondefaultCatcher
* @desc The default Promise.catch function * @param {Error} err
lib/runPromise.js:28
FunctiondefaultCatcher
* @desc Default catch function for promises. Prints error and message * @param {Error} err
lib/utils/promises.js:29
MethodgenInfoMetaDataRecord
* @desc Generate the WARC Info and Metadata records * @param {!Object} warcInfo - WARC record information * @property {!string} outlinks - Pr
lib/crawler/chrome.js:328
MethodgenWARC
* @desc Alias for {@link genWarc} * @param {!Object} warcInfo - WARC record information * @property {!string} outlinks - Pre-formatted string
lib/crawler/chrome.js:247
MethodgenWARC
* @desc Alias for {@link genWarc} * @param {Object} warcInfo - WARC record information * @property {!string} outlinks - Pre-formatted string
lib/crawler/puppeteer.js:245
MethodgetCrawlInjectsNoScroll
* @desc Receive param object for JS injected into every frame of the page crawled that includes disabling things that can act as an crawler trap and J
lib/injectManager/index.js:144
FunctiongetCrawler
* @desc Receive the crawl config requested crawler * @param {CrawlConfig} config - The crawl config * @returns {ChromeCrawler | PuppeteerCrawler}
lib/crawler/index.js:24
MethodgetProtocolDef
* @desc Receive the protocol definition of the remote Chrome * @param {?ChromeOptions} [options = {}] * @return {Promise<Object>} * @public
lib/launcher/chrome.js:255
MethodgetScollOnLoadInject
* @desc Receive param object for JS injected into every frame of the page crawled that scrolls the page once the `load` event has fired * @param {b
lib/injectManager/index.js:98
MethodgetScrollInject
* @desc Receive param object for JS injected into every frame of the page crawled that scrolls the page automatically * @param {boolean} [onNewDocu
lib/injectManager/index.js:112
MethodhelloFromFrame
* @desc Listens for the `outlinkgot` message sent by each child frame that contains its outlink information
lib/injectManager/pageInjects/collectLinks.js:211
Methodinit
* @emits {connected} when the required setup is done * @desc Connect to the Chrome instance the crawler will be using and setup crawler * @retur
lib/crawler/chrome.js:141
Functioninstall
()
lib/injectManager/pageInjects/noNaughtyJS.js:47
MethodkillChrome
()
lib/launcher/chrome.js:119
MethodlaunchNoConnect
* @desc Launch Chrome by finding an acceptable executable on the host system but do not connect to it * @param {?ChromeOptions} [options = {}] *
lib/launcher/chrome.js:164
MethodloadConfig
* @desc Loads the config file and performs the preliminary normalization * @param {string} configPath - Path to this crawls config file * @retur
lib/config/index.js:69
FunctionloadTimes
()
lib/injectManager/pageInjects/noNaughtyJS.js:58
Methodmagenta
* @desc Magenta colored console.log * @param args
lib/utils/colorPrinters.js:85
MethodmaybeRemoveUDataDir
()
lib/launcher/chrome.js:109
Functionmhc
(e)
lib/injectManager/pageInjects/collectLinks.js:271
MethodnavigationError
* @desc Used to have the NavigationManger emit the 'navigation-error' event * @param {Error | string} err
lib/crawler/navigationMan.js:177
MethodnewTab
* @desc Create and connect to a new tab of a running Chrome instance * @param {?ChromeOptions} [options = {}] * @return {Promise<CRI>} * @pub
lib/launcher/chrome.js:243
Functionnoop
()
userFns.js:11
FunctiononClose
()
lib/launcher/chrome.js:279
FunctiononLine
* @param {string} line
lib/launcher/chrome.js:294
FunctiononTimeout
()
lib/launcher/chrome.js:284
FunctionpageALL
(seed)
lib/utils/seedUrlToSeedObject.js:24
FunctionpageOnly
(seed)
lib/utils/seedUrlToSeedObject.js:18
FunctionpageSD
(seed)
lib/utils/seedUrlToSeedObject.js:21
MethodrawCollectInject
* @desc Receive the raw function that is used to collect outlinks from the pre-injected script {@link collect} for use with Runtime.evaluate * @see
lib/injectManager/index.js:53
MethodrawNoNaughty
* @desc Receive the raw function that helps ensure that the pages JS can not be naughty for use with Runtime.evaluate * @see https://chromedevtools
lib/injectManager/index.js:44
MethodreqFinished
* @desc Indicate that a request has finished * @param {Response | Request} info - Puppeteer Request or Response object
lib/crawler/netIdleWatcher.js:145
MethodreqStarted
* @desc Indicate that a request was made * @param {Request} info - Puppeteer Request object
lib/crawler/netIdleWatcher.js:131
FunctionrunPromise
* @desc Runs a promise using the supplied thener and catcher functions * @param {!function(): Promise<any> | Promise<any>} runnable The promise or as
lib/runPromise.js:42
Functionrunner
* @desc Launch a configured crawl * @param {string} configPath - Path to the crawls config file * @return {Promise<void>}
lib/runners/index.js:28
Methodstop
* @desc Stop the page loading and stop capturing requests * @return {!Promise<?Object>}
lib/crawler/puppeteer.js:208
MethodstopCapturingNetwork
* @desc Stop capturing the current web pages network requests
lib/crawler/chrome.js:207
MethodstopCapturingNetwork
* @desc Stop capturing the current web pages network requests
lib/crawler/puppeteer.js:200
MethodstopPageLoading
* @desc Equivalent to hitting the refresh button when it is an X * @return {Promise<any>}
lib/crawler/chrome.js:200
MethodstopPageLoading
* @desc Equivalent to hitting the refresh button when it is an X * @return {!Promise<?Object>}
lib/crawler/puppeteer.js:193
FunctionthenNoop
* @desc just a no op
lib/runPromise.js:22
FunctionthenNoop
* @ignore * @desc Default function that does nothing
lib/utils/promises.js:23
← previous101–175 of 175, ranked by callers