Abstract
In many modern machine learning models, attention mechanisms play a crucial role in processing data and identifying significant parts of the inputs, whether these are text or images. This selective focus enables subsequent stages of the model to achieve improved classification performance. Traditionally, attention mechanisms are applied as a preprocessing substructure before a neural network, such as in encoder/decoder architectures. In this paper, we extend the application of attention mechanisms to intermediate stages of data propagation within machine learning models. Specifically, we propose a Gen-eralised Attention Mechanism (GAM), which can be integrated before each layer of a neural network for classification tasks. The proposed GAM allows for at each layer/step of the machine learning architecture identification of the most relevant sections of the intermediate results. Our experimental results demonstrate that incorporating the proposed GAM into various machine learning models consistently enhances the accuracy of these models. This improvement is achieved with only a marginal increase in the number of parameters, which does not significantly affect the training time.