x: [b, h, l, l] ret: [b, h, l, 2*l-1]

x: [b, h, l, l] ret: [b, h, l, 2*l-1]

Bias for self-attention to encourage attention to close positions. Args: length: an integer scalar. Returns: a Ten

r"""Returns a window function with the given type and size

Returns a scalar factor <= 1.0 that dictates gradient clipping, i.e. we will scale the gradients by this amount before applying the r

r"""Given a waveform (1D tensor of size ``num_samples``), it returns a 2D tensor (m, ``window_size``) representing how the window is shifted along

Initializes state dict for parameter 'p'. Assumes that dim 0 of tensor p is actually the batch dimension, corresponding to batched-t

x: [b, h, l, d] y: [h or 1, m, d] ret: [b, h, l, m]

x: [b, h, l, d] y: [h or 1, m, d] ret: [b, h, l, m]

x: [b, h, l, m] y: [h or 1, m, d] ret: [b, h, l, d]

x: [b, h, l, m] y: [h or 1, m, d] ret: [b, h, l, d]

r"""Returns the smallest power of 2 that is greater than x

x: [b, h, l, 2*l-1] ret: [b, h, l, l]

x: [b, h, l, 2*l-1] ret: [b, h, l, l]

Show information of parameter wihch dominanting tot_sumsq. Args: tuples: a list of tuples of (param, state, param_names)

Called only where p.numel() > 1, this updates the scale of the parameter. If we imagine: p = underlying_param * scale.

Split long text into sentences with sentence-splitting punctuations. Args: text (str): The input text. Returns:

This function does the core update of self.step(), in the case where the members of the batch have more than 1 element. Args

Do the step for one parameter, which is actually going to be a batch of `real` parameters, with dim 0 as the batch dim. Args:

A simplified form of the core update for scalar tensors, where we cannot get a good estimate of the parameter rms.

This function returns (technically, yields) a list of of tuples (p, state), where p is a `fake` parameter that is stacked (