Awesome List:

• PositionalEncoding:
  • https://e2eml.school/transformers.html
  • https://github.com/tensorflow/tensor2tensor/issues/1591
  • https://kazemnejad.com/blog/transformer_architecture_positional_encoding/
• BatchNorm:
  • https://colab.research.google.com/drive/1Sw2GXJmylz9DvtPaBfJIeutoXNISelZy?usp=sharing
  • https://e2eml.school/batch_normalization.html
  • How BatchNorm Helps?: https://arxiv.org/pdf/1805.11604.pdf
  • Understanding BatchNorm: https://proceedings.neurips.cc/paper/2018/file/36072923bfc3cf47745d704feb489480-Paper.pdf
  • Batch-ReNorm: https://arxiv.org/pdf/1702.03275.pdf
• StreamingNorm: https://arxiv.org/pdf/1610.06160.pdf
• Online Norm: https://proceedings.neurips.cc/paper/2019/file/cb3ce9b06932da6faaa7fc70d5b5d2f4-Paper.pdf
• How optimization works:
  • https://e2eml.school/how_optimization_works_1.html
• Einops:
  • https://github.com/arogozhnikov/einops
• FlashAttention:
  • https://arxiv.org/abs/2205.14135
• Mamba:
  • https://arxiv.org/abs/2312.00752