Abstract
Machine learning research is active in resolving issues that cope with algorithm complexity, efficiency and accuracy in a broad scope of applications, such as face recognition, optical character recognition, data mining, medical informatics and diagnosis, financial time series forecasting, intrusion detection and military applications. In the data representing many of these applications, the issues can be related to high dimensional data with small sample sizes. With large number of features in the data, irrelevant or redundant features can lead to performance degradation due to overfitting, where the predictors may specialise on features which are not relevant for discrimination. To address this, feature selection and ensemble methods have been developed and researched. In this thesis feature selection has been investigated using feature ranking methods for multiple classifier systems. Recursive Feature Elimination combined with feature ranking is an effective method of removing irrelevant features. An ensemble of Multi-Layer Perceptron (MLP) base classifiers with feature ranking based on the magnitude of MLP weights is proposed along with the extension of this ranking to ensemble pruning. Also in this thesis ensemble pruning has been investigated for regression with emphasis given to dynamic ensemble pruning as a means of improving accuracy and generalisation. Ordering heuristics attempt to combine accurate yet complementary predictors, and thereby ordering the predictors can lead to enhanced prediction accuracy and generalisation. A dynamic method is proposed that enhances the performance by modifying the order of aggregation through distributing the ensemble selection over the entire data-set. Two more dynamic methods have been proposed that implement ensemble pruning by diverse predictor selection in the learning process. The first of these two methods simultaneously prunes and trains in the same learning process, while the second method is a hybrid method that applies different learning approaches selectively. Experimental results demonstrate improved performance for dynamic ensemble pruning on benchmark data-sets and an application in signal calibration.