1

I am training RNNs, which I built using tf.keras.layers.GRU layers. They are taking a long time to train (>2 hours), so I am going to deploy them to the GPU for training. I am wondering a few things about training on GPU:

  1. What is the difference between tf.keras.layers.CuDNNGRU and tf.keras.layers.GRU (and also tf.keras.layers.LSTM vs. tf.keras.layers.CuDNNLSTM)? I understand from this post that CuDNNGRU layers train faster than GRU layers, but
    • Do the 2 layers converge to different results with the same seed?
    • Do the 2 layers perform the same during inference?
    • Do CuDNN layers require a GPU during inference?
    • Can GRU layers run inference on a GPU?
    • Are CuDNN layers easily deployable? I am currently using coremlconverter to convert my keras model to CoreML for deployment.
  2. Is there an equivalent CuDNN layer for tf.keras.layers.SimpleRNN (i.e. tf.keras.layers.CuDNNSimpleRNN)? I am not committed to a specific architecture yet, and so I believe I would need the tf.keras.layers.CuDNNSimpleRNN layer if I decide on SimpleRNNs and the CuDNN layer has some functionality that I need.
  3. With CuDNN layers, do I need to have tensorflow-gpu installed? Or do they still get deployed to the GPU as long as I have the relevant drivers installed?
mickey
  • 476
  • 5
  • 18

1 Answers1

1

if you are using a cuda compatible gpu, it makes absolutely sense to use CuDNN layers. They have a different implementation that tries to overcome computation parallelization issues inherent in the RNN architecture. They usually perform a bit worst though but are 3x-6x faster https://twitter.com/fchollet/status/918170264608817152?lang=en

Do the 2 layers converge to different results with the same seed?

yes

Do the 2 layers perform the same during inference?

You should have a comparable performance but not exactly the same

Do CuDNN layers require a GPU during inference?

Yes but you can convert to a CuDNN compatible GRU/LSTM

Can GRU layers run inference on a GPU?

Yes

With CuDNN layers, do I need to have tensorflow-gpu installed? Or do they still get deployed to the GPU as long as I have the relevant drivers installed?

Yes and you need a cuda compatible gpu

cookiemonster
  • 1,315
  • 12
  • 19
  • thanks! I'll look into converting from CuDNN to GRU/LSTM, because I think that should solve a majority of my issues. any idea about CuDNNSimpleRNN layers (or do they not exist, because SimpleRNNs are much simpler than GRU/LSTM)? – mickey Aug 07 '19 at 13:55
  • checking the documentation https://keras.io/layers/recurrent/ i don't think they do exist – cookiemonster Aug 07 '19 at 14:50
  • I have a follow up question... I'm re-training some LSTM models using CuDNNLSTM layers, and I'm noticing that for layers of equivalent sizes, the CuDNNLSTM layers overfit much faster than the LSTM layers did. I'm curious if you have any insight about why the CuDNNLSTM layers would converge to a solution with less epochs than the LSTM layers take? – mickey Oct 03 '19 at 14:47
  • I haven't really tried comparing both but I saw some comparison questions before. e.g. comments here https://stackoverflow.com/questions/54559111/different-results-while-training-with-cudnnlstm-compared-to-regular-lstmcell-in. Maybe you checkout previous questions or ask a new one :) – cookiemonster Oct 04 '19 at 09:51
  • Thank you! those questions are specific to tensorflow, not keras, but super helpful resources, so thanks – mickey Oct 04 '19 at 13:25