DefaultTokenizer implements a simple tokenizer like Unix shell.

Lexer represents a lexical analyzer.

Tokenizer is the interface that classifies a token according to words, whitespaces, quotations, escapes and escaped quotations.