BN_FFN_BN: Leveraging Batch Normalization for Vision Transformers

December 2021

tl;dr: Introducing BN into vision based transformers via a BN-FFN-BN structure.

Overall impression

Inherited from the NLP tasks, vision transformers take Layernorm (LN)as default normalization technique when applied to vision tasks.

This is an alternative to PowerNorm to replace LN in transformers.

Key ideas

Technical details