takes in a source of text and converts into an array of stemmed tokens (filtering out stop words and punctuation) This gets called when ingesting new

compare arrays because deepEqual is inconsistent smh

helper function to add word

called at the global start

helper method which writes the current the record list to disk parameters determine which record list we write

highest-level function that is called at regular intervals to recompute the ENTIRE inverted index to integrate new data added via Apollo, resync data

takes all of the saved records and puts them in our inverted index

helper method which return a record from the associated id

write a map of tokens to their counts in our inverted index

TODO: make sourcesMap a global so we don't keep passing large maps in parameters TODO: should return map[string]schema.Data so we have control over th

helper function which should be called when the program is initialized so that the necessary files and paths exist in our database

TODO: should search titles too (and put high probability mass on those tokens) given a query string a search type (AND / OR ) returns a list of matche

takes a lists of thoughts and converts it into the require data struct we need for the api

takes a string of tokens and returns a map of each token to its frequency

method takes data and flushes it into our inverted index Note since th

Kindle does not directly provide a way to get highlights except via https://read.amazon.com/ I use a readwise extension to download my highlights into

follows similar approach to Kindle, podcast folder - put new txt files there

helper method to get small window of matching result don't send the full text back to the client cause this is too slow

idf = log(total number of documents / number of documents that contain term) - ensures tokens which are rarer get a higher score

load the sets for the first time to prevent repeated work

loads the inverted path from disk to memory opt to use a more optimized JSON decoding and encoding library than Go's native one as our inverted index