Technical analysis of credit risk utilizing Python and machine learning
Evaluate credit risk using six different machine learning models. Credit risk is an unbalanced classification problem in that the population of performing loans typically greatly outnumber the population of non-performing (delinquent) loans. Different techniques were used to train and evaluate different models with unbalanced classes. Credit card data from LendingClub with 68k+ records were used to evaluate the following six models:
- RandomOverSampler
- SMOTE
- ClusterCentroids
- SMOTEENN
- BalancedRandomForestClassifier
- EasyEnsembleClassifier
Models were run and evaluated by looking at the balanced accuracy score, confusion matrix, and imbalanced classification report for each model.
Terms and concepts first defined, followed by the results for each model
Balanced Accuracy Score - The difference between the predicted values and the actual values. This measure is problematic when evaluating imbalanced classes because it is important to accurately predict the smaller class.
Confusion Matrix - Breaks the number of records into the predicted values and actual values to identify: true positives, true negatives, false positives, and false negatives as seen below.
Imbalanced Classification Report -
Precision - How reliable a positive classification is (TP/(TP+FP)
Sensitivity/Recall - Measure of ability to identify all positive classifications (TP/(TP+FN)
F1 Score - aka 'Harmonic Mean' summary statistic of both precision and sensitivity (2(Precision*Sensitivity)/(Precision+Sensitivity). A very low F1 score indicates an imbalance between Precision and Sensitivity.
RandomOverSampler - 64.7% accuracy with a high number of both false negatives and false positives. Poor precision and sensitivity
Balanced Accuracy Score
Confusion Matrix
Imbalanced Classification Report
SMOTE - Slight improvement in accuracy and sensitivity
Balanced Accuracy Score
Confusion Matrix
Imbalanced Classification Report
ClusterCentroids - Poor model - decrease in accuracy and sensitivity
Balanced Accuracy Score
Confusion Matrix
Imbalanced Classification Report
SMOTEENN - Improved accuracy but still poor sensitivity
Balanced Accuracy Score
Confusion Matrix
Imbalanced Classification Report
BalancedRandomForestClassifier - 78.4% accuracy with improved sensitivity
Balanced Accuracy Score
Confusion Matrix
Imbalanced Classification Report
EasyEnsembleClassifier - 92.3% accuracy with much higher precision and sensitivity
Balanced Accuracy Score
Confusion Matrix
Imbalanced Classification Report
The EasyEnsembleClassifier algorithm is by far the most accurate machine learning model used to analyze and predict credit risk. It had a 92.3% balanced accuracy score. It also had the highest precision and sensitivity rankings of any of the other 5 models. Of the 101 high credit risk loans, only 7 loans were not predicted correctly.
The BalancedRandomForestClassifier model was the next closest with a fair 78.4% accuracy but had 31 high risk loans classified as low risk as compared to the EasyEnsembleClassifier model which only had 7.
The above analysis demonstrates the difficulty in predicting credit risk and unbalanced classifications.