Here, you can learn how to implement Bidirectional LSTM with Keras.
With a Bidirectional LSTM layer, you can see the improvement in accuracy with fewer epochs.
model = Sequential() model.add(Embedding(len(char_indices)+1, 300, batch_size=(seq_length-time_step)*batch_size, weights=[embedding_matrix], mask_zero=True, trainable=False)) model.add(Bidirectional(LSTM(512, dropout=0.15, return_sequences=True), merge_mode='concat')) model.add(Bidirectional(LSTM(512,dropout=0.15),merge_mode='concat')) model.add(Dense(len(char_indices)+1, activation='softmax'))
You should note that if the LSTM layer has
return_sequence=True, it cannot connect directly to the Dense layer.
The Dense layer can only accept two-dimensional sequences; when the LSTM layer does not have a
return_sequence, it outputs a two-dimensional sequences. However, if it has
return_sequence=True, it outputs a third-order sequences. As a result, an error occurs. For example,
ValueError: Shapes (None, None) and (3500, None, 10500) are incompatible
One way to solve this problem is to put another LSTM layer that does not have
return_sequence. This layer accepts the three-dimensional sequences from the previous LSTM layer and passes the two-dimensional sequences to the Dense layer.
In this way, the second LSTM layer seems to have little effect on the prediction results.