How to make two parallel convolutional neural networks in Keras?

Question

I created two convolutional neural networks (CNN), and I want to make these networks work in parallel. Each network takes different type of images and they join in the last fully connected layer.

How to do this?

Answer 1

You essentially need a multi-input model. This can only be done through keras' functional api and can work with the pretrained nets in keras.applications. To create one you can do this:

from keras.layers import Input, Conv2D, Dense, concatenate
from keras.models import Model

1) Define your first model:

 in1 = Input(...)  # input of the first model
 x = Conv2D(...)(in1)
 # rest of the model
 out1 = Dense(...)(x)  # output for the first model

2) Define your second model:

 in2 = Input(...)  # input of the first model
 x = Conv2D(...)(in2)
 # rest of the model
 out2 = Dense(...)(x)  # output for the first model

3) Merge the two models and conclude the network:

x = concatenate([out1, out2])  # merge the outputs of the two models
out = Dense(...)(x)  # final layer of the network

4) Create the Model:

model = Model(inputs=[in1, in2], outputs=[out])