0

I want to apply this method to implement Bi-LSTM with attention. The method is discussed here: Bi-LSTM Attention model in Keras

I get the following error: 'module' object is not callable

It can not apply multiply in this line: sent_representation = merge([lstm, attention], mode='mul')

from keras.layers import merge
import tensorflow as tf
from tensorflow.keras.layers import Concatenate, Dense, Input, LSTM, Embedding, Dropout, Activation, Flatten, Permute, RepeatVector
from tensorflow.keras.layers import Bidirectional

inp =Input(shape=(maxlen,), dtype='float32')
x = Embedding(max_features, embed_size, weights=[emb_matrix])(inp)
lstm = Bidirectional(LSTM(50, return_sequences=True), name="bi_lstm_0")(x)

attention = Dense(1, activation='tanh')(lstm)
attention = Flatten()(attention)
attention = Activation('softmax')(attention)
attention = RepeatVector(max_features*2)(attention)
attention = Permute([2,1])(attention)

sent_representation = merge([lstm, attention], mode='mul')
sent_representation = Lambda(lambda xin: K.sum(xin, axis=1))(sent_representation)

output = Dense(3, activation="softmax")(sent_representation)
Jindřich
  • 10,270
  • 2
  • 23
  • 44
eli
  • 35
  • 7
  • I removed the code that was irrelevant to the question. In the original code, you did the projecting with softmax twice, it would not learn like that. – Jindřich Nov 04 '20 at 08:28
  • thank you. can you please help me to solve my issue. I'm new to nlp and deep learning, im working on this attention for days and every time I get new error. – eli Nov 04 '20 at 08:31
  • because at the end I need 3 label, I can not use sigmoid or.. can you please help me? – eli Nov 04 '20 at 08:44
  • But you are already using softmax normalization with 3 labels. What is the problem then? – Jindřich Nov 04 '20 at 08:58
  • that is attention layer. – eli Nov 04 '20 at 09:04

1 Answers1

0

In Keras, merge is a module that contains layers that implement various ways of merging outputs of other layers. You need to select a method that you want to use to merge the states.

In this particular case, you want to concatenate the outputs.

Jindřich
  • 10,270
  • 2
  • 23
  • 44