Abstract
AI models have been expanding dramatically in size and the number of trainable parameters. This rapid growth has introduced many challenges, including increased computational costs and inefficiencies. Dynamic sparse training has emerged as a novel approach to address overparameterization and achieve energy-efficient artificial neural network (ANN) architectures. The highly efficient neuro-inspired sparse design remains underexplored compared to the significant focus on random topology searches. We propose the Topographical Sparse Mapping (TSM) method, inspired by the vertebrate visual system and convergent units. TSM introduces a sparse input layer for MLPs, significantly reducing the number of parameters. Unlike conventional approaches that focus on optimising sparse connectivity patterns through complex computations, our work introduces a biologically inspired sparse connectivity scheme that naturally enhances performance without the need for intricate optimisation. Notably, the number of connections is determined solely by the number of input features, independent of the number of neurons in the receiving layer. An enhanced version of TSM (ETSM) incorporates additional pruning during training to achieve a desired reduction in parameters. This streamlined framework surpasses several state-of-the-art sparse training methods, offering superior accuracy, generalization, and training efficiency. Remarkably, ETSM overcomes the conventional trade-off between simplicity and accuracy, achieving improvements in both simultaneously. Additional experiments further demonstrate that topographically structured input mapping accelerates convergence and enhances final accuracy compared to unstructured pruning. ETSM introduces a novel perspective on computationally efficient ANN design, underscoring the value of topographically sparse connectivity. These findings emphasize the potential of leveraging neurobiological principles to effectively address overparameterization challenges in ANN development.