Pretrained transformer model that can be finetuned for downstream tasks. name (str): Name of the pretrained Huggingface model to use. get_spa

The outputs, split per spaCy Doc object.

A noop function, for compatibility with the Pipe API. See the `update` method for an explanation of the loss mechanics of the component.

RETURNS (List[TransformerListener]): The listener models listening to this component. Usually internals.

RETURNS (List[str]): The downstream components listening to this component. Usually internals.

Construct a Transformer component, which lets you plug a model from the Huggingface transformers library into spaCy so you can use it in your

Set no additional annotations on the Doc objects.

Remove a listener for a downstream component. Usually internals.

Add a logger that will log memory usage to the given file. Used to debug OOM errors.

Test that an empty document gets processed correctly

Test that a pipeline with a transformer+tagger+senter and some replaced listeners runs and trains properly

Test that the pipeline doesn't fail with empty input

Test that a simple pipeline does not raise an error on texts that exceeds the model max length. We should truncate instead.

Test that a simple pipeline with just a transformer at least runs

Test that a tagger with internal transformer runs and trains properly

Test that a pipeline with just a transformer+tagger+senter runs and trains properly

Test that a pipeline with just a transformer+textcat runs and trains properly. This used to throw an error because of shape inference issues -

Test that a transformer using sentencepiece trains + IO goes OK

Deprecated. A dict with the wordpiece token data.

Deprecated. Dict formatted version of the self.wordpieces data, with values converted to PyTorch tensors.