Reconfigurable Intelligent Surfaces (RIS) is a key 6G technology to enhance the performance of future wireless communication systems, by using, as in massive multipleinput-multiple-output (MIMO), beamforming to do so. However, beamforming requires accurate channel estimation, which becomes more challenging over a cascaded channel and more complex as the number of elements on the RIS increases. Besides, jointly designing beams at base station (BS) and RIS adds to this challenge, while the number of beamforming options, which scale with the number of antennas at the base station/RIS, adds to the complexity. Existing methods for channel estimation and/or beamforming optimization tend to use computationally intensive iterative methods that are nonscalable for large RIS-aided MIMO systems. Machine learning has been introduced to help with these kind of issues in both channel estimation and beamforming optimisation for RIS. Machine learning has the following advantages over conventional approaches: 1) It is robust at solving non-linear optimization problems like the RIS optimization problem that has no closed-form solution. 2) It has a low computational complexity benefit. Once the neural network is trained, it no longer needs extra calculations on the input data. 3) It has great adaptability and flexibility. Machine learning models can continuously learn and improve, adapting to new data and changing environments, which is suitable for the highly complex and dynamic RIS environment. Motivated by these advantages, this thesis embraces the usage of machine learning for channel estimation and beamforming optimization in MIMO-RIS systems, as follows: First, a practical three-phase channel estimation method is proposed for multi-user RIS-MIMO systems. This scheme greatly reduces the number of channel coefficients to be estimated by using the correlation of a reference user with other users. Then, a supervised and reinforcement learning approach for optmizing RIS passive beamforming are proposed. Second, to reduce the computational complexity during the training phase, a selfsupervised learning scheme is proposed for joint beamforming optmization. This scheme leverages a unsupervised learning to formulate a contrastive loss between unprocessed CSI data. The proposed self-supervised learning scheme only needs 5% of the labelled dataset to predict the joint beamforming with capacity performance less than 8% away from the upper bound. Third, a generative model named score-based diffusion model is proposed for RIS-MIMO channel estimation. Diffusion model uses posterior distributions to formulate an unsupervised learning optimization. The proposed method has a near minimum mean square error in-distribution performance, and a robust out-of-distribution performance. These three contributions highlight improvements in machine learning across three key aspects: practicality, computational complexity, and adaptability. They also demonstrate that machine learning can be widely applied in wireless communication, highlighting its great potential for future research.