How to adjust the hyperparameters of MLP classifier to get more perfect performance

Question

I am just getting touch with Multi-layer Perceptron. And, I got this accuracy when classifying the DEAP data with MLP. However, I have no idea how to adjust the hyperparameters for improving the result.

Here is the detail of my code and result:

.

from sklearn.neural_network import MLPClassifier

import numpy as np
import scipy.io
x_vals = data['all_data'][:,0:320]

y_vals_new = np.array([0 if each=='Neg'  else 1 if each =='Neu' else 2 for each in data['all_data'][:,320]])
y_vals_Arousal = np.array([3 if each=='Pas'  else 4 if each =='Neu' else 5 for each in data['all_data'][:,321]])

DEAP_x_train = x_vals[:-256]        #using 80% of whole data for training
DEAP_x_test = x_vals[-256:]         #using 20% of whole data for testing
DEAP_y_train = y_vals_new[:-256]     ##Valence
DEAP_y_test = y_vals_new[-256:]
DEAP_y_train_A = y_vals_Arousal[:-256]   ### Arousal
DEAP_y_test_A = y_vals_Arousal[-256:]

mlp = MLPClassifier(solver='adam', activation='relu',alpha=1e-4,hidden_layer_sizes=(50,50,50), random_state=1,max_iter=11,verbose=10,learning_rate_init=.1)

mlp.fit(DEAP_x_train, DEAP_y_train)

print (mlp.score(DEAP_x_test,DEAP_y_test))
print (mlp.n_layers_)
print (mlp.n_iter_)
print (mlp.loss_)

Answer 1

If you are using SKlearn, you can use their hyper-parameter optimization tools.

For example, you can use:

• GridSearchCV
• RandomizedSearchCV

If you use GridSearchCV, you can do the following:

1) Choose your classifier

from sklearn.neural_network import MLPClassifier
mlp = MLPClassifier(max_iter=100)

2) Define a hyper-parameter space to search. (All the values that you want to try out.)

parameter_space = {
    'hidden_layer_sizes': [(50,50,50), (50,100,50), (100,)],
    'activation': ['tanh', 'relu'],
    'solver': ['sgd', 'adam'],
    'alpha': [0.0001, 0.05],
    'learning_rate': ['constant','adaptive'],
}

Note: the max_iter=100 that you defined on the initializer is not in the grid. So, that number will be constant, while the ones in the grid will be searched.

3) Run the search:

from sklearn.model_selection import GridSearchCV

clf = GridSearchCV(mlp, parameter_space, n_jobs=-1, cv=3)
clf.fit(DEAP_x_train, DEAP_y_train)

Note: the parameter n_jobs is to define how many CPU cores from your computer to use (-1 is for all the cores available). The cv is the number of splits for cross-validation.

4) See the best results:

# Best paramete set
print('Best parameters found:\n', clf.best_params_)

# All results
means = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']
for mean, std, params in zip(means, stds, clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))

5) Now you can use the clf to make new predictions. For example, check the performance on your test set.

y_true, y_pred = DEAP_y_test , clf.predict(DEAP_x_test)

from sklearn.metrics import classification_report
print('Results on the test set:')
print(classification_report(y_true, y_pred))

Answer 2

As a complement to the very practical answer of @BrunoGL, I'd like to give a more theoretical answer. I'd like to suggest everyone trying to adjust hyperparameters of a simple Neural Network to read Efficient Backprop, by Lecun and others (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf). Yes it's from 1998, no it's not outdated.

It covers the impact of the main hyperparameters you have to set (activation, solver, learning rate, batches), commons traps, the problems you may encouter if you fall into them, how to spot those problems and how to solve them. A must read for everyone that want to tune a Neural Network. Plus, it's free.