Initialize the pipe for training, using data examples if available. get_examples (Callable[[], Iterable[Example]]): Optional function that

Apply the pipeline's model to a batch of docs, without modifying them. Returns the extracted features as the FullTransformerBatch dataclass.

Prepare for an update to the transformer. Like the `Tok2Vec` component, the `Transformer` component is unusual in that it does not re

Execute a block in a temporary directory and remove the directory and its contents at the end of the with block. YIELDS (Path): The path of t

Serialize the pipe to disk. path (str / Path): Path to a directory. exclude (Iterable[str]): String names of serialization fields to

Apply the pipe to a stream of documents. This usually happens under the hood when the nlp object is called on a text and all components are

Load the pipe from disk. path (str / Path): Path to a directory. exclude (Iterable[str]): String names of serialization fields to exc

Apply a Huggingface tokenizer to a batch of texts.

Assign the extracted features to the Doc objects. By default, the TransformerData object is written to the doc._.trf_data attribute. Your

Pool transformer data into token-aligned tensors.

Create a copy of the model, its attributes, and its parameters. Any child layers will also be deep-copied. The copy will receive a di

Wrap a PyTorch HF model, so that it has the same API as Thinc models. To optimize the model, you'll need to create a PyTorch optimizer and call

Create a Huggingface transformer model from pretrained weights. Will download the model if it is not already downloaded. source (Union[str, P

Create a Huggingface transformer model from pretrained weights. Will download the model if it is not already downloaded. source (Union[str, P

Return a dict similar to the format produced by the Huggingface tokenizer, converting arrays to pytorch tensors as well.

Drop wordpieces from inputs that are too long. This can happen because the splitter is based on linguistic tokens, and the number of wordpieces

Return a new FullTransformerBatch from a split batch of activations, using the current object's spans, wordpieces and alignment. This

Create a test model that produces a FullTransformerBatch object.

Return a two-dimensional boolean mask, indicating whether wordpieces are dropped from their sequences. Drop wordpieces from the end of the se

Check whether bytes are in safetensors format (vs torch pickle).

Add a listener for a downstream component. Usually internals.

Given a list of sequences, return a batched list of indices into the list, where the batches are grouped by length, in descending order. Batc

Create a `span_getter` that uses sentence boundary markers to extract the spans. This requires sentence boundaries to be set, and may result

Set the 'window' and 'stride' options for getting strided spans. If you set the window and stride to the same value, the spans will cover

Walk over a model of a processing component, looking for layers that are TransformerListener subclasses that have an upstream_name that

Apply a Huggingface tokenizer to a batch of texts.

Flip a coin and decide whether to flush PyTorch's cache. This allows the cache to be flushed periodically without maintaining a counter. I'm

Split a TransformerData that represents a batch into a list with one TransformerData per Doc.

Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the nlp object

Initialize the transformer component.

get_spans (Callable[[List[Doc]], List[Span]]): A function to extract spans from the batch of Doc objects. This is use

Accumulate tok2vec loss and gradient. This is passed as a callback to all but the last listener. Only the last one does the backprop.

Callback to actually do the backprop. Passed to last listener.

Call the stored _backprop callback, and then clears it. This saves memory, as otherwise we hold onto that callback until the next batc

Create a `span_getter` that uses the whole document as its spans. This is the best approach if your `Doc` objects already refer to relatively

Create a copy of the model, its attributes, and its parameters. Any child layers will also be deep-copied. The copy will rece