Adding the swish activation function to your Keras model

If you want to know more about the swish activation function, I can highly recommend this blog post and the paper it is based on.

Unfortunately, the Keras code given in the blog post above didn’t work for me but after a while I found the solution somewhere else. So, this is how you can use the swish activation function in Keras:

from keras import backend as K
from keras.utils.generic_utils import get_custom_objects

# needs to be defined as activation class otherwise error
# AttributeError: 'Activation' object has no attribute '__name__'    
class Swish(Activation):
    def __init__(self, activation, **kwargs):
        super(Swish, self).__init__(activation, **kwargs)
        self.__name__ = 'swish'

def swish(x):
    return (K.sigmoid(x) * x)

get_custom_objects().update({'swish': Swish(swish)})

Afterwards, just make sure to replace all activation function calls (such as “relu” for example) with “swish”.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s