When should tf.add(t, 0) be used instead of tensor t directly?

Question

In tensorflow.contrib.slim.batch_norm, math_ops.add(moving_mean, 0) is used to copy the value of moving_mean, which is passed to nn.moments subsequently.

Would it be a problem if we just pass moving_mean to nn.moments directly?

Are there any guidelines on the use of copy operation (tf.add(t, 0))?

This was added in [this commit](https://github.com/tensorflow/tensorflow/commit/e78fadb2). The effect is that it copies value of t, so that if another worker modifies the value of t in parallel, it will be unaffected and will use old value for the tf.moments computation. — Yaroslav Bulatov, Dec 13 '16 at 22:00
@YaroslavBulatov Thank you for sharing ways to find the answer. It's quite helpful to examine the commit logs. — Jenny, Dec 14 '16 at 03:36

score 1 · Accepted Answer · answered Dec 13 '16 at 21:54

The problem is that the order in which moving_mean is updated, could cause the gradients to use the updated version of moving_mean, instead of the original moving_mean used as shift. So to make sure the same value is used in the forward pass and in the backward pass we make an explicit copy.

score 0 · Answer 2 · answered Dec 13 '16 at 18:23

There is not much information about it around, but it may be used to convert a tensor reference to a regular tensor. Their behavior differ when you push them to the queue (if the tensor reference changes the value of the tensor). You can read more about it in this question.

When should tf.add(t, 0) be used instead of tensor t directly?

2 Answers2