Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/N0taN3rd/Squidwarc
/ functions
Functions
175 in github.com/N0taN3rd/Squidwarc
⨍
Functions
175
◇
Types & classes
38
↓ 1 callers
Method
warcNamePerURL
* @desc Returns a function that will concatenate the output path with the filenamified seedURL * producing the full path to WARC of the page being
lib/utils/warcNaming.js:32
↓ 1 callers
Function
which
* @desc Executes the which command for the supplied executable name * @param {string} executable
lib/launcher/chromeFinder.js:49
↓ 1 callers
Method
writeWRPlayerPagesRecord
()
lib/crawler/chrome.js:309
↓ 1 callers
Method
writeWRPlayerPagesRecord
()
lib/crawler/puppeteer.js:307
↓ 1 callers
Method
yellow
* @desc Yellow colored console.log * @param args
lib/utils/colorPrinters.js:37
Method
[Symbol.iterator]
* @desc Iterate over the captured network requests for the current web page * @return {Iterator<CapturedRequest>}
lib/crawler/chrome.js:369
Method
[Symbol.iterator]
* @desc Iterate over the captured network requests for the current web page * @return {Iterator<CapturedRequest>}
lib/crawler/puppeteer.js:362
Method
_close
* @desc Callback for process.on('exit') * @private
lib/crawler/chrome.js:398
Method
_didNavigate
* @desc Callback used for Page.navigate * @private
lib/crawler/chrome.js:377
Method
_emitNetIdle
* @desc Emit the network-idle event * @private
lib/crawler/netIdleWatcher.js:186
Method
_globalNetworkTimeout
* @desc Called when the global time limit was hit * @private
lib/crawler/netIdleWatcher.js:162
Method
_globalNetworkTimeout
* @desc Called when the global time limit was hit * @private
lib/crawler/navigationMan.js:205
Method
_navTimedOut
* @desc Called when the navigation time limit was hit * @private
lib/crawler/navigationMan.js:193
Method
_networkIdled
* @desc Called when the network idle has been determined * @private
lib/crawler/netIdleWatcher.js:174
Method
_networkIdled
* @desc Called when the network idle has been determined * @private
lib/crawler/navigationMan.js:214
Method
_onDisconnected
* @desc CB used to emit the disconnected event * @private
lib/crawler/puppeteer.js:107
Method
_onWARCGenError
* @desc Listener for warc generator error * @param {Error} err * @private
lib/crawler/chrome.js:409
Method
_onWARCGenError
* @desc Listener for warc generator error * @param {Error} err - The error to emit * @private
lib/crawler/puppeteer.js:371
Method
_onWARCGenFinished
* @desc Listener for warc generator finished * @private
lib/crawler/chrome.js:417
Method
_onWARCGenFinished
* @desc Listener for warc generator finished * @private
lib/crawler/puppeteer.js:379
Method
addEventListeners
* @desc Register an array of eventName, eventListener pairs to an EventEmitter * @param {!EventEmitter} emitter - The EventEmitter to register the
lib/crawler/helper.js:27
Method
boldBlueGreen
* @desc Bold blue first arg, green rest * @param bb * @param rest
lib/utils/colorPrinters.js:104
Function
configureURLNormalizer
* @desc wrapper around {@link https://github.com/sindresorhus/normalize-url} that * given a configuration for normalize-url returns a function that a
lib/utils/urlUtils.js:36
Method
constructor
* @param {string} url - A starting seed * @param {symbol} mode - The mode for the seed * @param {number} depth - The crawl depth
lib/frontier/seedTracker.js:29
Method
constructor
* @desc Create a new frontier object
lib/frontier/index.js:27
Method
constructor
()
lib/injectManager/pageInjects/collectLinks.js:47
Method
constructor
(collectorRef, messages)
lib/injectManager/pageInjects/collectLinks.js:160
Method
constructor
* @param {Object} network * @param {NavigationMan} navMan
lib/crawler/chrome.js:31
Method
constructor
* @desc Create a new ChromeCrawler instance. For a description of the expected options see the * JSDoc CrawlConfig typedef {@link CrawlConfig} *
lib/crawler/chrome.js:72
Method
constructor
* @param {Page} page - Puppeteer page object for the page being crawled * @param {?NetIdleOptions} [options = {}] - Optional options to control fin
lib/crawler/netIdleWatcher.js:31
Method
constructor
* @desc Create a new PuppeteerCrawler instance. For a description of the expected options see the * JSDoc CrawlConfig typedef {@link CrawlConfig}
lib/crawler/puppeteer.js:39
Method
constructor
* * @param {CrawlControl} [options = {}] * @param {EventEmitter} [parentEmitter]
lib/crawler/navigationMan.js:32
Function
csi
()
lib/injectManager/pageInjects/noNaughtyJS.js:57
Function
defaultCatcher
* @desc The default Promise.catch function * @param {Error} err
lib/runPromise.js:28
Function
defaultCatcher
* @desc Default catch function for promises. Prints error and message * @param {Error} err
lib/utils/promises.js:29
Method
genInfoMetaDataRecord
* @desc Generate the WARC Info and Metadata records * @param {!Object} warcInfo - WARC record information * @property {!string} outlinks - Pr
lib/crawler/chrome.js:328
Method
genWARC
* @desc Alias for {@link genWarc} * @param {!Object} warcInfo - WARC record information * @property {!string} outlinks - Pre-formatted string
lib/crawler/chrome.js:247
Method
genWARC
* @desc Alias for {@link genWarc} * @param {Object} warcInfo - WARC record information * @property {!string} outlinks - Pre-formatted string
lib/crawler/puppeteer.js:245
Method
getCrawlInjectsNoScroll
* @desc Receive param object for JS injected into every frame of the page crawled that includes disabling things that can act as an crawler trap and J
lib/injectManager/index.js:144
Function
getCrawler
* @desc Receive the crawl config requested crawler * @param {CrawlConfig} config - The crawl config * @returns {ChromeCrawler | PuppeteerCrawler}
lib/crawler/index.js:24
Method
getProtocolDef
* @desc Receive the protocol definition of the remote Chrome * @param {?ChromeOptions} [options = {}] * @return {Promise<Object>} * @public
lib/launcher/chrome.js:255
Method
getScollOnLoadInject
* @desc Receive param object for JS injected into every frame of the page crawled that scrolls the page once the `load` event has fired * @param {b
lib/injectManager/index.js:98
Method
getScrollInject
* @desc Receive param object for JS injected into every frame of the page crawled that scrolls the page automatically * @param {boolean} [onNewDocu
lib/injectManager/index.js:112
Method
helloFromFrame
* @desc Listens for the `outlinkgot` message sent by each child frame that contains its outlink information
lib/injectManager/pageInjects/collectLinks.js:211
Method
init
* @emits {connected} when the required setup is done * @desc Connect to the Chrome instance the crawler will be using and setup crawler * @retur
lib/crawler/chrome.js:141
Function
install
()
lib/injectManager/pageInjects/noNaughtyJS.js:47
Method
killChrome
()
lib/launcher/chrome.js:119
Method
launchNoConnect
* @desc Launch Chrome by finding an acceptable executable on the host system but do not connect to it * @param {?ChromeOptions} [options = {}] *
lib/launcher/chrome.js:164
Method
loadConfig
* @desc Loads the config file and performs the preliminary normalization * @param {string} configPath - Path to this crawls config file * @retur
lib/config/index.js:69
Function
loadTimes
()
lib/injectManager/pageInjects/noNaughtyJS.js:58
Method
magenta
* @desc Magenta colored console.log * @param args
lib/utils/colorPrinters.js:85
Method
maybeRemoveUDataDir
()
lib/launcher/chrome.js:109
Function
mhc
(e)
lib/injectManager/pageInjects/collectLinks.js:271
Method
navigationError
* @desc Used to have the NavigationManger emit the 'navigation-error' event * @param {Error | string} err
lib/crawler/navigationMan.js:177
Method
newTab
* @desc Create and connect to a new tab of a running Chrome instance * @param {?ChromeOptions} [options = {}] * @return {Promise<CRI>} * @pub
lib/launcher/chrome.js:243
Function
noop
()
userFns.js:11
Function
onClose
()
lib/launcher/chrome.js:279
Function
onLine
* @param {string} line
lib/launcher/chrome.js:294
Function
onTimeout
()
lib/launcher/chrome.js:284
Function
pageALL
(seed)
lib/utils/seedUrlToSeedObject.js:24
Function
pageOnly
(seed)
lib/utils/seedUrlToSeedObject.js:18
Function
pageSD
(seed)
lib/utils/seedUrlToSeedObject.js:21
Method
rawCollectInject
* @desc Receive the raw function that is used to collect outlinks from the pre-injected script {@link collect} for use with Runtime.evaluate * @see
lib/injectManager/index.js:53
Method
rawNoNaughty
* @desc Receive the raw function that helps ensure that the pages JS can not be naughty for use with Runtime.evaluate * @see https://chromedevtools
lib/injectManager/index.js:44
Method
reqFinished
* @desc Indicate that a request has finished * @param {Response | Request} info - Puppeteer Request or Response object
lib/crawler/netIdleWatcher.js:145
Method
reqStarted
* @desc Indicate that a request was made * @param {Request} info - Puppeteer Request object
lib/crawler/netIdleWatcher.js:131
Function
runPromise
* @desc Runs a promise using the supplied thener and catcher functions * @param {!function(): Promise<any> | Promise<any>} runnable The promise or as
lib/runPromise.js:42
Function
runner
* @desc Launch a configured crawl * @param {string} configPath - Path to the crawls config file * @return {Promise<void>}
lib/runners/index.js:28
Method
stop
* @desc Stop the page loading and stop capturing requests * @return {!Promise<?Object>}
lib/crawler/puppeteer.js:208
Method
stopCapturingNetwork
* @desc Stop capturing the current web pages network requests
lib/crawler/chrome.js:207
Method
stopCapturingNetwork
* @desc Stop capturing the current web pages network requests
lib/crawler/puppeteer.js:200
Method
stopPageLoading
* @desc Equivalent to hitting the refresh button when it is an X * @return {Promise<any>}
lib/crawler/chrome.js:200
Method
stopPageLoading
* @desc Equivalent to hitting the refresh button when it is an X * @return {!Promise<?Object>}
lib/crawler/puppeteer.js:193
Function
thenNoop
* @desc just a no op
lib/runPromise.js:22
Function
thenNoop
* @ignore * @desc Default function that does nothing
lib/utils/promises.js:23
← previous
101–175 of 175, ranked by callers