How to build a Bidirectional LSTM model with Keras

Here, you can learn how to implement Bidirectional LSTM with Keras.

[ad]

With a Bidirectional LSTM layer, you can see the improvement in accuracy with fewer epochs.

model = Sequential()
model.add(Embedding(len(char_indices)+1,
                    300,
                    batch_size=(seq_length-time_step)*batch_size,
                    weights=[embedding_matrix],
                    mask_zero=True,
                    trainable=False))
model.add(Bidirectional(LSTM(512,
                             dropout=0.15,
                             return_sequences=True),
                             merge_mode='concat'))
model.add(Bidirectional(LSTM(512,dropout=0.15),merge_mode='concat'))
model.add(Dense(len(char_indices)+1, activation='softmax'))

You should note that if the LSTM layer has return_sequence=True, it cannot connect directly to the Dense layer.

The Dense layer can only accept two-dimensional sequences; when the LSTM layer does not have a return_sequence, it outputs a two-dimensional sequences. However, if it has return_sequence=True, it outputs a third-order sequences. As a result, an error occurs. For example,

ValueError: Shapes (None, None) and (3500, None, 10500) are incompatible

One way to solve this problem is to put another LSTM layer that does not have return_sequence. This layer accepts the three-dimensional sequences from the previous LSTM layer and passes the two-dimensional sequences to the Dense layer.

In this way, the second LSTM layer seems to have little effect on the prediction results.