Apply positional information to a sequence of embeddings. Takes in a sequence of embeddings with shape (batch_size, seq_length, embed_dim) and adds positional embeddings to them Args: embed_dim: (int): Dimension of the positional embedding. max_seq_length: Maximum seque
| 1357 | |
| 1358 | |
| 1359 | class SinusoidalPositionalEmbedding(nn.Module): |
| 1360 | """Apply positional information to a sequence of embeddings. |
| 1361 | |
| 1362 | Takes in a sequence of embeddings with shape (batch_size, seq_length, embed_dim) and adds positional embeddings to |
| 1363 | them |
| 1364 | |
| 1365 | Args: |
| 1366 | embed_dim: (int): Dimension of the positional embedding. |
| 1367 | max_seq_length: Maximum sequence length to apply positional embeddings |
| 1368 | |
| 1369 | """ |
| 1370 | |
| 1371 | def __init__(self, embed_dim: int, max_seq_length: int = 32): |
| 1372 | super().__init__() |
| 1373 | position = torch.arange(max_seq_length).unsqueeze(1) |
| 1374 | div_term = torch.exp(torch.arange(0, embed_dim, 2) * (-math.log(10000.0) / embed_dim)) |
| 1375 | pe = torch.zeros(1, max_seq_length, embed_dim) |
| 1376 | pe[0, :, 0::2] = torch.sin(position * div_term) |
| 1377 | pe[0, :, 1::2] = torch.cos(position * div_term) |
| 1378 | self.register_buffer("pe", pe) |
| 1379 | |
| 1380 | def forward(self, x): |
| 1381 | _, seq_length, _ = x.shape |
| 1382 | x = x + self.pe[:, :seq_length] |
| 1383 | return x |
| 1384 | |
| 1385 | |
| 1386 | class ImagePositionalEmbeddings(nn.Module): |