Types & classes — github.com/commoncrawl/cc-index-table

CodeHub MCPcopy Create free account

hub / github.com/commoncrawl/cc-index-table / types & classes

Types & classes27 in github.com/commoncrawl/cc-index-table

⨍ Functions 160◇ Types & classes 27

ClassCCIndex2Table

Convert Common Crawl's URL index into a tabular format.

src/main/java/org/commoncrawl/spark/CCIndex2Table.java:35

ClassCCIndexExport

Select and export rows and columns of the index table matched by a Spark SQL query

src/main/java/org/commoncrawl/spark/examples/CCIndexExport.java:45

ClassCCIndexWarcExport

src/main/java/org/commoncrawl/spark/examples/CCIndexWarcExport.java:49

ClassCCWarcFilenameParser

Parse Common Crawl WARC filenames (full paths) and extracts crawl, segment, and subset.

src/main/java/org/commoncrawl/spark/util/CCWarcFilenameParser.java:26

src/main/java/org/commoncrawl/spark/CCIndex2Table.java:42

src/main/java/org/commoncrawl/spark/EOTIndexTable.java:39

Parse and hold a CDX line of the format: <pre> SURT-Key Timestamp JSON-Object </pre>

src/main/java/org/commoncrawl/spark/IndexTable.java:103

ClassEOTIndexTable

Convert End of Term Web Archive's CDX index into a tabular format.

src/main/java/org/commoncrawl/spark/EOTIndexTable.java:32

ClassFilenameParseError

Exception if parsing the WARC filename fails to find the required crawl, segment, and subset.

src/main/java/org/commoncrawl/spark/util/CCWarcFilenameParser.java:59

ClassFilenameParts

Encapsulate the extracted crawl, segment, and subset.

src/main/java/org/commoncrawl/spark/util/CCWarcFilenameParser.java:43

src/main/java/org/commoncrawl/net/HostName.java:42

ClassHostNameTest

src/test/java/org/commoncrawl/net/HostNameTest.java:24

ClassIndexTable

Convert a CDX index into a tabular format.

src/main/java/org/commoncrawl/spark/IndexTable.java:74

ClassJobStatsListener

Accumulate and report Spark metrics.

src/main/java/org/commoncrawl/spark/JobStatsListener.java:27

ClassNoStatisticsException

src/util/are_part_min_max_increasing.py:11

ClassNullOutputCommitter

src/main/java/org/commoncrawl/spark/util/NullOutputCommitter.java:25

ClassTestCCIndex2Table

src/test/java/org/commoncrawl/spark/TestCCIndex2Table.java:24

ClassTestCCWarcFilenameParser

src/test/java/org/commoncrawl/spark/TestCCWarcFilenameParser.java:25

ClassTestEOTIndexTable

src/test/java/org/commoncrawl/spark/TestEOTIndexTable.java:25

ClassTestIndexTable

src/test/java/org/commoncrawl/spark/TestIndexTable.java:24

ClassTestIndexTableBase

src/test/java/org/commoncrawl/spark/TestIndexTableBase.java:37

src/test/java/org/commoncrawl/net/TestURL.java:29

src/main/java/org/commoncrawl/net/HostName.java:46

ClassWarcFileOutputFormat

Hadoop output format to dump/export WARC records (gzipped byte[]) unchanged into new WARC files.

src/main/java/org/commoncrawl/spark/util/WarcFileOutputFormat.java:41

ClassWarcRecordWriter

Opens a WARC file, writes a WARC info record and appends WARC records (gzipped byte[]).

src/main/java/org/commoncrawl/spark/util/WarcRecordWriter.java:46

Parses a string representation of a URI or URL found as WARC-Target-URI and provides access to the parts of the URL/URI. Cf. {@link java.net.URL}, {@l

src/main/java/org/commoncrawl/net/WarcUri.java:30

ClassWarcUriTest

src/test/java/org/commoncrawl/net/WarcUriTest.java:24