If you want to know more about the swish activation function, I can highly recommend this blog post and the paper it is based on.
Unfortunately, the Keras code given in the blog post above didn’t work for me but after a while I found the solution somewhere else. So, this is how you can use the swish activation function in Keras:
from keras import backend as K
from keras.utils.generic_utils import get_custom_objects
# needs to be defined as activation class otherwise error
# AttributeError: 'Activation' object has no attribute '__name__'
class Swish(Activation):
def __init__(self, activation, **kwargs):
super(Swish, self).__init__(activation, **kwargs)
self.__name__ = 'swish'
def swish(x):
return (K.sigmoid(x) * x)
get_custom_objects().update({'swish': Swish(swish)})
Afterwards, just make sure to replace all activation function calls (such as “relu” for example) with “swish”.