What Is The Difference Between The TensorFlow Batch Normalization Implementations?

TensorFlow seems to implement at least 3 versions of batch normalization: tf.nn.batch_normalization tf.layers.batch_normalization tf.contrib.layers.batch_norm These all have diff

Solution 1:

They are actually very different.

• nn.batch_normalization performs the basic operation (i.e. a simple normalization)
• layers.batch_normalization is a batchnorm "layer", i.e. it takes care of setting up the trainable parameters etc. At the end of the day, it is a wrapper around nn.batch_normalization. Chances are this is the one you want to use, unless you want to take care of setting up variables etc. yourself.

It's similar to the difference between nn.conv2d and layers.conv2d, for example.

As for the contrib version, I can't say for sure, but it seems to me like an experimental version with some extra parameters not available in the "regular" layers one.

Solution 2:

The are all based on the same paper: http://arxiv.org/abs/1502.03167 So they should do the same thing.

Some functions move around the code but the old versions are kept to keep backwards compatibility and you end up with more than one version.

I would recommend using the simplest one that lets you do your project (that is tf.nn.batch_normalization). If you need features/parameters that are not offered pick the one that works for you.

Note: tf.contrib.* is not guarateed to remain backwards compatible (the api might change in a future version).