Heterogeneous Ensemble Learning for Multi-class Classification

Abstract

In data mining, classification is a type of supervised learning task that involves predicting output variables consisting of a finite number of categories called classes. When the number of classes is larger than two, a classification problem is called a multi-class classification problem. Multi-class classification provides more informative predictions, and is more related to real-world scenarios. In practice, the performance for a multi-class classification problem is typically measured according to the following three perspectives: accurate, reliable, and fast classification. In order to achieve the better performance for the three perspectives, this dissertation proposes to use heterogeneous ensemble learning that exploits multiple classifiers from various classification algorithms, where each classifier plays a different role to accomplish the desired functionality. For accurate multi-class classification, Diversified One-Against-One (DOAO) and Optimally Diversified One-Against-One (ODOAO) are proposed. Their main idea is to decompose the original problem into several binary sub-problems based on the one-against-one approach. DOAO finds the best classification algorithm for each class pair from the set of heterogeneous base classifiers, thereby makes various classification algorithms to complement each other. Since the best classification algorithm for each class pair is different, DOAO enables better classification accuracy. ODOAO, an extension of DOAO, construct an ensemble where a meta-classifier effectively combines the outputs from all the heterogeneous base classifiers. Heterogeneous Ensemble of One-class Classifiers (HEOC) is also proposed for accurate classification based on decomposition of the original problem into several one-class sub-problems. HEOC constructs an ensemble consisting of one-class classifiers from various one-class classification algorithms. HEOC addresses the normalization of heterogeneous base classifiers via stacking. For reliable multi-class classification, a hybrid reject option is proposed to reject ambiguous instances instead of predicting for all instances. The hybrid reject option constructs a filter classifier and a predictor classifier separately, where the filter decides whether to predict using the predictor based on the confidence for an instance, and the predictor predicts the class of the instance. Each component is trained using the best respective classification algorithm to maximize the capability of its role, thereby improve reject option performance as providing better prediction accuracy for the same degree of rejection. For fast multi-class classification, Neural Network Approximator (NNA) is proposed to reduce computational time in the test phase. NNA approximates a classifier by adopting a multiple-outputs artificial neural network as a function approximator, where each output node corresponds to a decision function in the classifier. This approximator enables fast classification speed without compromising accuracy. The effectiveness of the proposed heterogeneous ensemble methods is demonstrated through experiments on benchmark datasets and real-world applications.