Method tokenize

language/serene/preprocessing.py:583–584

(self, text)

    }

  def tokenize(self, text):
    return self._tokenizer.tokenize(text)

  def encode(self, s):
    full_text = tf.compat.as_text(s)