Given a list of some items, compute entropy of the list The entropy is sum of -p[i]*log(p[i]) for every unique element i in the list, and p[i

Build a HashTree with candidates cand, then count support of these candidates to filter out all those that have support not lower tha

Note that bag must be sorted

Compute frequency and entropy of this word @param length length of the document for training to get words

Compute aggregation of this word @param words_dict frequency dict of all candidate words

Generate frequent 1-item sets

@return Frequent item sets with their frequency

Generate next candidates by dynamic programming Find range [i, j) such that items in this range have same prefix e.g., [1,2,3

Partition a string into all possible two parts, e.g. given "abcd", generate [("a", "bcd"), ("ab", "cd"), ("abc", "d")] For string of leng

Generate all candidate words with their frequency/entropy/aggregation informations @param doc the document used for words generation

Treat a suffix as an index where the suffix begins. Then sort these indexes by the suffixes.

Segment a sentence with the words generated from a document @param sentence the sentence to be handled @param method segmenta

Increase frequency of this word, then append left/right neighbors @param left a single character on the left side of this word

count the child who matches bag, suppose that current node matches

deduplicate the given SORTED list

Generate all substrings of max length n for string