Create sine-cosine positional embeddings. Args: n: the number of embedding vectors, corresponding to the number of tokens (patches) in the image. d: the dimension of the embeddings k: value that determines the maximum frequency (10,000 by default)
(n: int, d: int, k: int=10000)
| 134 | |
| 135 | @staticmethod |
| 136 | def pos_encoding(n: int, d: int, k: int=10000): |
| 137 | '''Create sine-cosine positional embeddings. |
| 138 | |
| 139 | Args: |
| 140 | n: the number of embedding vectors, corresponding to the number of tokens (patches) in the image. |
| 141 | d: the dimension of the embeddings |
| 142 | k: value that determines the maximum frequency (10,000 by default) |
| 143 | |
| 144 | Returns: |
| 145 | (n, d) tensor of position encoding vectors |
| 146 | ''' |
| 147 | x = torch.meshgrid( |
| 148 | torch.arange(n, dtype=torch.float32), |
| 149 | torch.arange(d, dtype=torch.float32), |
| 150 | indexing='ij' |
| 151 | ) |
| 152 | pos = torch.zeros_like(x[0]) |
| 153 | pos[:, ::2] = x[0][:, ::2].div(torch.pow(k, x[1][:, ::2].div(d // 2))).sin_() |
| 154 | pos[:, 1::2] = x[0][:,1::2].div(torch.pow(k, x[1][:,1::2].div(d // 2))).cos_() |
| 155 | return pos |
| 156 | |
| 157 | @staticmethod |
| 158 | def generate_mask_index(bs: int, n_tok: int, device: str='cpu'): |