Experimental Comparison of Classification Methods under Class Imbalance

Chen, Hui and Ji, Mengru (2021) Experimental Comparison of Classification Methods under Class Imbalance. EAI Transactions on Scalable Information Systems. e12. ISSN 2032-9407

[thumbnail of eai.11-6-2021.170234.pdf]
Available under License Creative Commons Attribution No Derivatives.

Download (3MB) | Preview


The class imbalance problem is prevalent in many domains including medical, natural language processing, image recognition, economic and geographic areas etc. We perform a systematic experimental comparison of different imbalance classification algorithms — ranging from sampling, distance metric learning, cost-sensitive learning to ensemble learning approaches — on several datasets from UCI, KEEL and OpenML. The algorithms included DDAE, MWMOTE, SMOTE, RUSBoost, AdaBoost, cost-sensitive decision tree (csDCT), self-paced Ensemble Classifier, MetaCost, CAdaMEC and Iterative Metric Learning (IML). As the substantial bias potentially caused by imbalance classification can be harmful for underrepresented classes which are of critical social and economic values and even lives, the main objective of our study is thus to understand the impact of imbalance ratio and the size of the utilized datasets on the performance of the above-mentioned algorithms. Our experiments show that 1) Sampling methods perform the worst and cannot be used directly for imbalanced classification, since they lack of consideration of neighborhoods based on distance. However, some classifiers can be improved after the balance of class distribution. 2) Cost-sensitive learning models should be utilized when the dataset is less imbalanced, because it is difficult to set an appropriate cost matrix for a specific dataset, which can cause performance fluctuations. 3) IML consistently shows good performance (in terms of F1 and AUCPRC), is resilient to different imbalance ratios but sensitive to the data distribution of the dataset. 4) Ensemble learning techniques generally perform better over other approaches due to theircombined intelligence of multiple basic classifiers. 5) In terms of system performance, self-paced Ensemble Classifier performs fairly well with regards to learning time, while IML and DDAE yield the longest learningtime; AdaBoost and self-paced Ensemble Classifier are two algorithms require lowest memory usage. Wealso provide our empirical recommendation for algorithm selection under different requirements and usagescenarios based on our analysis.

Item Type: Article
Uncontrolled Keywords: Imbalance Classification, ResampIing, Cost-Sensitive Learning, Distance Metric Learning, Ensemble Learning, Performance Evaluation
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
QA75 Electronic computers. Computer science
T Technology > T Technology (General)
Depositing User: EAI Editor IV
Date Deposited: 26 Jul 2021 15:12
Last Modified: 26 Jul 2021 15:12
URI: https://eprints.eudl.eu/id/eprint/5169

Actions (login required)

View Item
View Item