0

In the tensorflow documentation (softmax_cross_entropy_with_logits), they said "logits : unscaled log probablilty". What is 'log probability'? First, I understand that 'logits' is an 'output before normalization' or a 'score for class'.

logits = tf.matmul(X,W) + b
hypothesis = tf.nn.softmax(logits)

If I got [1.5, 2.4, 0,7] by tf.matmul(X,W) + b, then [1.5, 2.4, 0,7] is logits(score) and this was unscaled. I can understand it up to this stage. But, I can't understand why [1.5, 2.4, 0.7] is 'log probability'.

Vlad
  • 8,225
  • 5
  • 33
  • 45
KiHyun Nam
  • 77
  • 6
  • 4
    Possible duplicate of [What is the meaning of the word logits in TensorFlow?](https://stackoverflow.com/questions/41455101/what-is-the-meaning-of-the-word-logits-in-tensorflow) – Maxim Jan 28 '18 at 09:01
  • Thanks everyone! I found this post. It solved my question almost. https://stats.stackexchange.com/questions/52825/what-does-the-logit-value-actually-mean – KiHyun Nam Jan 30 '18 at 03:52

2 Answers2

3

If you interpret the output of the softmax function as probabilities (as we like to do), then it is easy to see were the "log probability" comes from:

The softmax function is

\exp{z_k}/\sum_i{\exp{z_i}},

with z_i as the components of your "logits". The denominator just takes care of normalization, i.e. it makes sure that all the outputs sum up to 1 (which makes sense if we want to interpret them as probabilities for a set of mutually exclusive classes). So, looking at the numerator, the output of the softmax function is basically just exp(z). If we interpret this as probability, then z (the "logits") is the logarithm of the un-normalized probability.

rain city
  • 227
  • 1
  • 7
1

Thanks everyone!

I found this post. It solved my question almost.

https://stats.stackexchange.com/questions/52825/what-does-the-logit-value-actually-mean

KiHyun Nam
  • 77
  • 6