Integrated sensing and communication (ISAC) is regarded as one of the most promising paradigm shifts towards sixth-generation (6G) wireless networks. By exploiting the intrinsic similarities between the two functionalities, ISAC is expected to save scarce spectral resources and realize more cost-efficient, intelligent, and human-friendly wireless networks in the future.

In the first part of this thesis, the ISAC aided by a massive multiple-input multiple-output (mMIMO) system is investigated. Currently mMIMO-based ISAC usually sidesteps the contradiction between the high resolution and energy consumption. Motivated by this, an energy-efficient widely spaced array (WSA) architecture is tailored for the radar receiver, which, along with the existing communication unit, enhances the angular resolution of radar sensing. From the perspective of compressive sensing (CS), the transmit waveform, transform dictionary, and estimation algorithm regarding the proposed architecture will be discussed. A Doppler estimation and compensation framework is also proposed to obtain the velocity information of moving targets as well as to guarantee robust communication performance under high-mobility scenarios. Our simulation results show that with the proposed WSA architecture, the normalized mean square error (NMSE) performance of radar sensing can achieve up to 10 dB improvement, and the bit-error-rate performance can remain unaffected under high-mobility scenarios with velocity $100$ km/h.

In the second part of this thesis, the waveform design for ISAC is investigated, which is also regarded as an essential task for ISAC future development. Although many state-of-the-art candidates, such as orthogonal frequency division multiplexing (OFDM) and orthogonal time frequency space (OTFS), have been considered, the potential of chirp signals widely used in radar systems has not been fully explored. Considering this, the orthogonal chirp division modulation (OCDM) scheme has been introduced to facilitate ISAC. By integrating the OCDM with the mmWave frequency-modulated continuous wave (FMCW) radar technique, superb joint communication and sensing performance can be observed compared to its counterparts like OFDM and OTFS. A novel ISAC solution based on OCDM and FMCW for mmWave communications, which demonstrates significant superiority over the conventional schemes in terms of both system performance and hardware complexity. Simulation results show that compared to the conventional schemes, the proposed OCDM-FMCW scheme can achieve up to $20$\,dB improvement in terms of root mean square error (RMSE) of parameter estimation, while the required analog-to-digital converter (ADC) sampling rate can be reduced by almost $10$ times.

In the last part of this thesis, the design of antenna array and waveform has been jointly investigated. The emerging ultra-massive multiple-input multiple-output (UM-MIMO) technique pushes ISAC into the near-field area, but the waveform design dedicated for the near-field scenarios is underexplored. As a remedy, the OCDM-based method is applied to tackle the challenging near-field ISAC problem. Specifically, a comprehensive ISAC architecture is conceived, where an UM-MIMO base station adopts OCDM waveform for communications and a co-located FMCW receiver adopts the FMCW detection principle to simplify the associated hardware. For sensing tasks, several OCDM subcarriers, namely dedicated sensing subcarriers (DSSs), are each transmitted through a dedicated sensing antenna (DSA) within the transmit UM-MIMO. By judiciously designing the DSS selection scheme and optimizing receiver parameters, the FMCW-based sensing receiver can decouple the echo signals from different DSAs with significantly reduced hardware complexity. This setup enables the estimation of ranges and velocities of near-field targets in an antenna-pairwise manner. Moreover, the concept of {\em virtual bistatic sensing} (VIBS) is introduced, which incorporates the estimates from multiple antenna pairs to achieve high-accuracy target positioning and three-dimensional velocity measurement. The VIBS paradigm is immune to hostile channel environments characterized by spatial non-stationarity and uncorrelated multipath environment. Simulation results demonstrate that the proposed ISAC scheme enhances sensing accuracy, and it outperform the existing counterparts under practical channel conditions.