Abstract
In this paper, we explore a simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS)-assisted integrated sensing and communication (ISAC) secure communication system. The average long-term security rate of the legitimate user (LU) is maximized by jointly designing the receive filters and transmit beamforming of the base station (BS), and the transmitting and reflecting coefficients of STAR-RIS, and in the meantime, guaranteeing the lower bound of echo signal-to-noise ratio (SNR) and the achievable rate of LU constraint. We propose to apply two deep reinforcement learning (DRL) algorithms to solve the complex non-convex problem and maximize the long-term benefits of the system by optimizing the BS beamforming and STAR-RIS phase shifts. The simulation results thoroughly evaluate the performance of two DRL algorithms and demonstrate that STAR-RIS outperforms the conventional reconfigurable intelligent surface (RIS) in comparison with two benchmarks. Index Terms—integrated sensing and communication (ISAC), deep reinforcement learning (DRL), simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS), secrecy rate