Abstract
The problem of controlling model complexity and data complexity are fundamental issues in neural network learning. Some researchers have used Bayesian-learning on neural networks to control the model complexity. The Bayesian-based technique, Automatic Relevance Determination (ARD), can effectively control the complexity of data, and automatically determine the relevance of input features by controlling the distribution of corresponding groups of weights in a network. However, we found that the relevance determination made by a single ARD model is not stable and accurate. Neural network ensemble techniques were used in our research to improve the accuracy of feature relevance determination. The accuracy of the ensemble feature relevance determination was evaluated using two synthetic datasets in which the relevance of each individual input feature was pre-determined. The results showed that ensemble feature relevance determination can effectively separate relevant features, redundancies and irrelevant features from each other, and provide useful suggestions of the boundaries between these relevance levels. Thus, the features selected, based on the ensemble feature relevance determination, benefits not only non-linear models such as neural networks, but also linear models such as the linear regression model, by enabling them to classify the samples in several real-world datasets more accurately than by using all the available input features, extracted principal components and independent components from the datasets. We also found that an ensemble of ARD models is good at selecting group relevance features, but not at ranking the relevance for each individual input feature, because the relevance rank determination of an input feature can be affected by any redundancies in the data which are highly correlated with it.