- File:
labml_nn/transformers/rope/__init__.py
- Line:
95
- Quote: The paper suggests using $\Theta = {\theta_i = 10000^{\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$ for the $\frac{d}{2}$ pairs of features.
I thing it should be: $\Theta = {\theta_i = 10000^{-\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$
labml_nn/transformers/rope/__init__.py95I thing it should be:$\Theta = {\theta_i = 10000^{-\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$