Understanding the functioning of a recurrent neural network with LSTM cells

Question

Context:

• I have a recurrent neural network with LSTM cells
• The input to the network is a batch of size (batch_size, number_of_timesteps, one_hot_encoded_class) in my case (128, 300, 38)
• The different rows of the batch (1-128) are not necessarily related to each other
• The target for one time step is given by the value of the next time step.

My questions: When I train the network using an input batch of (128,300,38) and a target batch of the same size,

1. does the network always consider only the last time-step t to predict the value of the next timestep t+1?

2. or does it consider all time steps from the beginning of the sequence up to time step t?

3. or does the LSTM cell internally remember all previous states?

I am confused about the functioning because the network is trained on multiple time steps simulatenously so I am not sure how the LSTM cell can still have knowledge of the previous states.

I hope somebody can help. Thanks in advance!

Code for dicussion:

            cells = []

            for i in range(self.n_layers):
                cell = tf.contrib.rnn.LSTMCell(self.n_hidden)
                cells.append(cell)

            cell = tf.contrib.rnn.MultiRNNCell(cells)
            init_state = cell.zero_state(self.batch_size, tf.float32)

            outputs, final_state = tf.nn.dynamic_rnn(
                cell, inputs=self.inputs, initial_state=init_state)

            self.logits = tf.contrib.layers.linear(outputs, self.num_classes)

            softmax_ce = tf.nn.sparse_softmax_cross_entropy_with_logits(
                labels=labels, logits=self.logits)

            self.loss = tf.reduce_mean(softmax_ce)
            self.train_step = tf.train.AdamOptimizer(self.lr).minimize(self.loss)

Answer 1

The above is a simple RNN unrolled to the neuron level with 3 time steps.

As you can see that the output at time step t, depends upon all time steps from the beginning. The network is trained using back-propagation through time where the weights are updated by the contribution of all error gradients across time. The weights are shared across time, so there is nothing like simultaneous update on all time steps.

The knowledge of the previous states are transfered through the state variable s_t as it is a function of previous inputs. So at any time step, the prediction is made based on the current input as well as (function of) previous inputs captured by the state variable.

NOTE: A basic rnn was used instead of LSTM because of simplicity.

Answer 2

Here's what would be helpful to keep in mind for your case specifically:

Given the input shape of [128, 300, 38]

• One call to dynamic_rnn will propagate through all 300 steps, and if you are using something like LSTM, the state will also be carried through those 300 steps
• However, each SUBSEQUENT call to dynamic_rnn will not automatically remember the state from the previous call. By the second call, the weights/etc. will have been updated thanks to the first call, but you will still need to pass the state that resulted from the first call into the second call. That's why dynamic_rnn has a parameter initial_state and that's why one of its outputs is final_state (i.e. the state after processing all 300 steps in ONE call). So you are meant to take the final state from call N and pass it back as the initial state for call N+1 to dynamic_rnn. This allrelates specifically to LSTM, since this is what you asked for
• You are right to note that elements in one batch don't necessarily relate to each other within the same batch. This is something you need to consider carefully. Because with successive calls to dynamic_rnn, batch elements in your input sequences have to relate to their respective counterparts in the previous/following sequence, but not to each other. I.e. element 3 in the first call may have nothing to do with the other 127 elements within the same batch, but element 3 in the NEXT call has to be the temporal/logical continuation of element 3 in the PREVIOUS call, and so forth. This way, the state that you keep passing forward makes sense continuously