ranks an unordered list of records based on relevance, uses the inverse document frequency which is a document-level statistic that scores how relevan

takes a document which is suspected to be an article or something that's scrapable and attempts to scrape it

tokenizes a source of text into a list of lowercase tokens with stop words and punctuation removed

I use a ported version of the Go snowball algorithm (considered a portman2.0)here. Although I would have preferred to write my own stemmer writing a g

helper method which writes the current the inverted index to disk

get the full text of a record when expanded for detail