3

While I learn Keras, I always see a syntax like Activation('relu')(X). I looked at the source code and found Activation is a class, so it does make no sense to me how the syntax like Class(...)(...) works.

Here is an example and use case of it: A = Add()([A1, A2])

nuric
  • 11,027
  • 3
  • 27
  • 42
Xiangyu Zhang
  • 61
  • 5
  • 2
    Objects can have a `__call__` function which is called when attempting to call the object – BallpointBen May 14 '18 at 19:34
  • As @BallpointBen mentioned it is `__call__` method that enables this behavior. It is a way defining objects that can act like functions. Also look at [this](https://stackoverflow.com/a/9663601/2099607) answer on SO. – today May 14 '18 at 19:46
  • Thank you, everyone@BallpointBen, the __call__ does solve my question. – Xiangyu Zhang May 14 '18 at 19:51

3 Answers3

3

In Keras, it's a bit more convoluted than vanilla Python. Let's break down what happens when you call Activation('relu')(X):

  1. Activation('relu') creates a new object of that class by calling the class __init__ method. This creates the object with 'relu' as parameter.
  2. All objects in Python can be callable by implementing __call__ allowing you to call it like a function. Activation('relu')(X) now calls that function with X as parameter.
  3. But wait, Activation doesn't directly implement it, in fact it is the base class Layer.__call__ gets called which does some checks like shape matching etc.
  4. Then Layer.__call__ actually calls self.call(X) which then invokes the Activation.call method which applies the activation to the tensor and returns the result.

Hope that clarifies that line of code, a similar process happens when creating other layers and calling them with the functional API.

nuric
  • 11,027
  • 3
  • 27
  • 42
1

In python, classes may have the __call__ method, meaning that class instances are callable.

So, it's totally ok to call Activation(...)(...).

The first step creates an instance of Activation, and the second calls that instance with some parameters.

It's exactly the same as doing:

activationLayer = Activation('relu')
outputTensor = activationLayer(inputTensor) #where inputTensor == X in your example

With this, you can also reuse the same layers with different input tensors:

activationLayer = Activation('relu')

out1 = activationLayer(X1)
out2 = activationLayer(X2)

This doesn't make a big difference with a standard activation layer, but it starts getting very interesting with certain trained layers.

Example: you want to use a standard trained VGG16 model to process two images and then join the images:

vgg16 = keras.applications.vgg16(......)

img1 = Input(imageShape1)
img2 = Input(imageShape2)

out1 = vgg16(img1) #a model is also a layer by inheritance
out2 = vgg16(img2)

... continue the model ....
Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
-1

Are you expecting the new keyword? Python does not use that keyword, instead uses "function notation":

Class instantiation uses function notation. Just pretend that the class object is a parameterless function that returns a new instance of the class. For example (assuming the above class):

x = MyClass()
Nate Anderson
  • 18,334
  • 18
  • 100
  • 135
  • 1
    There is similar syntax in `dask`. The OP isn't asking about the baseline syntax to instantiate an object, its the double `class()()` syntax. I can give a simple `dask` example: http://dask.pydata.org/en/latest/delayed.html – roganjosh May 14 '18 at 19:35
  • 1
    Hi, thank you for the lighting speed response! Guess I didn't make my question clear. Let me give another example here:https://github.com/keras-team/keras/blob/master/keras/layers/merge.py#L193. The usage is like: X = Add()([A1, A2]). So here we have instantiated an Add instance without any parameters by calling Add(), let's call it ADD_INSTANCE, then, what is ADD_INSTANCE([A1, A2]) doing here? – Xiangyu Zhang May 14 '18 at 19:35
  • I think it was clear I was just too eager to give a response to a class syntax that confused me when I was starting general Python! Please ignore my answer, and/or downvote maybe @roganjosh should answer re:dask – Nate Anderson May 14 '18 at 19:37
  • Actually, the reason I'm interested in this question is because I don't really know how it works either! :) – roganjosh May 14 '18 at 19:38
  • Did @BallpointBen clarify in their comment to OP? re:`__call__`? Which was another Python feature that was foreign to me – Nate Anderson May 14 '18 at 19:39