TF Objection Detection Zoo models don't have Trainable Variables?

Question

The models in the TF Objection Detection Zoo have meta+ckpt file, Frozen.pb file, and Saved_model file.

I tried to use the meta+ckpt file to train further and also to extract some weights for particular tensors for research purpose. I see that the models don't have any trainable variables.

vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)
print(vars)

The above snippet gives an [] list. I also tried using the following.

vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)
print(vars)

I again get an [] list.

How is this possible ? is the model stripped off the variables ? or are the tf.Variable(trainable=False) ? Where can I get meta+ckpt file with valid trainable variables. I specifically looking at SSD+mobilnet models

UPDATE:

Following is the code snippet I'm using for restoring.It inside a class since I am making a custom tool for some application.

def _importer(self):
    sess = tf.InteractiveSession()
    with sess.as_default():
        reader = tf.train.import_meta_graph(self.metafile,
                                            clear_devices=True)
        reader.restore(sess, self.ckptfile)

def _read_graph(self):
    sess = tf.get_default_session()
    with sess.as_default():
        vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)
        print(vars)

UPDATE 2:

I also tried with the following snippet. Simple restoring style.

model_dir = 'ssd_mobilenet_v2/'

meta = glob.glob(model_dir+"*.meta")[0]
ckpt = meta.replace('.meta','').strip()

sess = tf.InteractiveSession()
graph = tf.Graph()
with graph.as_default():
    with tf.Session() as sess:
        reader = tf.train.import_meta_graph(meta,clear_devices=True)
        reader.restore(sess,ckpt)

        vari = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)
        for var in vari:
            print(var.name,"\n")

Th above code snippet also gives [] variable list

Answer 1

After a little bit of research, the final answer to your question is No, they don't. It is quite obvious until you realize that the variables directory in saved_model is empty.

The checkpoint file provided by the object detection model zoo contains following files:

.
|-- checkpoint
|-- frozen_inference_graph.pb
|-- model.ckpt.data-00000-of-00001
|-- model.ckpt.index
|-- model.ckpt.meta
|-- pipeline.config
`-- saved_model
    |-- saved_model.pb
    `-- variables

The pipeline.config is the config file for the model saved, the frozen_inference_graph.pb is for off-the-shelf inference. Notice that checkpoint, model.ckpt.data-00000-of-00001, model.ckpt.meta and model.ckpt.index all correspond to the checkpoint. (Here you can find a nice explanation)

So when you want to get the trainable variables, the only thing useful is the saved_model directory.

Use SavedModel to save and load your model—variables, the graph, and the graph's metadata. This is a language-neutral, recoverable, hermetic serialization format that enables higher-level systems and tools to produce, consume, and transform TensorFlow models.

To recover the SavedModel you can use the api tf.saved_model.loader.load(), and this api contain one argument called tags, which specify the type of MetaGraphDef. So if you want to get the trainable variables, you need to specify tag_constants.TRAINING when calling the api.

I tried to call this api to recover the variables but instead it gave me the error that says

MetaGraphDef associated with tags 'train' could not be found in SavedModel. To inspect available tag-sets in the SavedModel, please use the SavedModel CLI: saved_model_cli

So I did this saved_model_cli command to inspect all tags available in the SavedModel.

#from directory saved_model
saved_model_cli show --dir . --all

and the output is

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
...
signature_def['serving_default']:
  ...

So there is no tag train but only serve within this SavedModel. The SavedModel here therefore is only used for tensorflow serving. This means when these files when created, were not specified with tag training, no training variables can be recovered from these files.

P.S.: the following code is what i used for restoring the SavedModel. When setting tag_constants.TRAINING, the loading cannot be completed but when setting tag_constants.SERVING, the loading is successful but the variables are empty.

graph = tf.Graph()
with tf.Session(graph=graph) as sess:
  tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.TRAINING], export_dir)
  variables = graph.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)
  print(variables)

P.P.S: I found the script for creating the SavedModel here. It can be seen that indeed there was not train tag when creating the SavedModel.