Abstract
This paper proposes a joint precoding–filtering framework based on widely linear (WL) processing to increase the mutual information between the transmitted symbols and the filter output in finite-alphabet multiple-input multiple-output (MIMO) systems. Existing WL transceiver designs have been confined to improper signal constellations. For the first time, this work extends WL processing to proper signals, enabling its application to practical modulations such as QAM. This generalization is realized through a joint transmitter-receiver design that induces controlled impropriety via a conjugate precoding branch and exploits the resulting pseudo-covariance at the receiver. Unlike conventional improper Gaussian signaling designs, the proposed framework operates at the WL precoder level for finite-alphabet MIMO transmission and couples the induced transmit impropriety with receiver-side WL filtering. We first derive an exact per-realization average mutual information (AMI) expression for finite-alphabet inputs under instantaneous channel state information. To reduce the computational cost, we derive a closed-form lower bound on AMI, which serves as a tractable surrogate for optimizing the WL precoding and filtering matrices. We further show that the conventional linear transceiver arises as a degenerate special case of the proposed WL framework, highlighting its generality and structural flexibility. Simulation results demonstrate consistent average AMI gains over linear baselines across modulation formats and antenna configurations. At a target AMI, the proposed design reduces the required SNR by up to 3 dB relative to the linear baseline, establishing the first generally applicable WL transceiver design with superior robustness and practical advantages across diverse communication scenarios. Index Terms—Widely linear precoding, mutual information optimization, MIMO system, joint precoding and filtering design.