Function tokenize

nlp_class3/memory_network.py:40–46

Return the tokens of a sentence including punctuation. >>> tokenize('Bob dropped the apple. Where is the apple?') ['Bob', 'dropped', 'the', 'apple', '.', 'Where', 'is', 'the', 'apple', '?']

(sent)

def tokenize(sent):
  '''Return the tokens of a sentence including punctuation.

  >>> tokenize('Bob dropped the apple. Where is the apple?')
  ['Bob', 'dropped', 'the', 'apple', '.', 'Where', 'is', 'the', 'apple', '?']
  '''
  return [x.strip() for x in re.split('(\W+?)', sent) if x.strip()]