scikit learn hyperparameter optimization for MLPClassifier

tune/adjust hyperparameters MLPClassifier in scikit learn

Two simple strategies to optimize/tune the hyperparameters:

Models can have many hyperparameters and finding the best combination of parameters can be treated as a search problem.

Although there are many hyperparameter optimization/tuning algorithms now, this post shows a simple strategy which is grid search. Read more here

How to tune hyperparameters in scikit learn

In scikit learn, there is GridSearchCV method which easily finds the optimum hyperparameters among the given values.

As an example:

mlp_gs = MLPClassifier(max_iter=100)
parameter_space = {
    'hidden_layer_sizes': [(10,30,10),(20,)],
    'activation': ['tanh', 'relu'],
    'solver': ['sgd', 'adam'],
    'alpha': [0.0001, 0.05],
    'learning_rate': ['constant','adaptive'],
}
from sklearn.model_selection import GridSearchCV
clf = GridSearchCV(mlp_gs, parameter_space, n_jobs=-1, cv=5)
clf.fit(X, y) # X is train samples and y is the corresponding labels

I fixed max_iter or the number of epochs.

another example

As you see, we first define the model (mlp_gs) and then define some possible parameters. GridSearchCV method is responsible to fit() models for different combinations of the parameters and give the best combination based on the accuracies.

cv=5 is for cross validation, here it means 5-folds Stratified K-fold cross validation. Read more here

n_jobs=-1 , -1 is for using all the CPU cores available.

After running the code, the results will be like this:

To see the perfect/best hyperparameters, we need to run this:

print('Best parameters found:\n', clf.best_params_)

and we can run this part to see all the scores for all combinations:

means = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']
for mean, std, params in zip(means, stds, clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))

The final step is to test the best model on the test set.

If the test set is X_test and corresponding labels is y_test we can do:

y_true, y_pred = y_test , clf.predict(X_test)
from sklearn.metrics import classification_report
print('Results on the test set:')
print(classification_report(y_true, y_pred))

In my example, there are 10 labels (MNIST data set).

Also read more here.

Very soon I will publish another good example which help us to tune hyperparameters professionally.

I would like to introduce two packages for Laravel that I have recently developed: Laravel Pay Pocket, a modern multi-wallet package, and Laravel Failed Jobs, a UI for the Laravel Failed Jobs Table. I hope they may be of help to you.

https://github.com/HPWebdeveloper/laravel-pay-pocket
https://github.com/HPWebdeveloper/laravel-failed-jobs