The 5G Core network (5GC) adopts a Service-Based Architecture (SBA) consisting of modern technologies that provide all-new service capabilities to the end users of telecommunication networks. The ability to provision, enhance, and manage services for diverse use cases while maintaining high levels of performance is a major motivation behind the architectural overhaul observed in the 5GC. Such a significant transition towards new technologies and architecture provides powerful new service capabilities but similarly introduces a range of security challenges that must be addressed to ensure the confidentiality, integrity, and availability of services. In this work, the process of identifying new security threats related to the signaling functionality of the SBA is undertaken before further analysis is performed, and solutions are proposed in the form of a novel detection capability and protection framework for the 5GC. The identification of security threats is achieved through an in-depth literature review that considers current research focused on the different adversarial techniques that are both viable and likely, based on historical data relating to signaling-type attacks on telecommunication networks. Then, through the implementation of a 5GC simulator environment, signaling attack scenarios are reproduced to generate suitable comparative real-world data that can be used to evaluate potential solutions providing security monitoring, detection, and protection capabilities. A major influence in this work is the progress made in the area of Machine Learning (ML) in providing a powerful solution in the cybersecurity domain. The wide variety of ML techniques and algorithms provides the platform to deliver security functionality capable of detecting and making informed decisions to combat both known and unknown attacks. To this end, different machine learning models are evaluated to solve security challenges posed to the 5GC in the context of signaling security. To begin, we propose a service classification model used to perform security monitoring in the SBA. After considering a security monitoring capability for 5GC signaling, we extend the initial research to propose a protection framework to provide a layer of security against a variety of signaling attacks, including unauthorized service access. Finally, we propose a detection capability applied to a specific type of signaling attack: the Packet Forwarding Control Protocol (PFCP) attack targeting the session management functionality of the 5G control plane signaling. Important yet challenging properties of any security solution are the ability to provide privacy-preserving solutions that have minimal impact on service operations. Existing approaches like Deep Packet Inspection (DPI), which is applied to network traffic, require access to packet contents, raising issues with data confidentiality and adding processing overheads. The solutions proposed in this research use network traffic metadata generated from the 5G simulator environment as input to each model. This, in turn, offers solutions that are not only viable but also enforce a privacy-preserving feature. Furthermore, the proposed models operate independently of the real system, which circumvents the requirement to perform operations on data in transit, thus avoiding delays in network signaling performance. The solutions proposed in this thesis provide viable, privacy-preserving, and independent solutions to solve some of the current and future security challenges in the context of 5GC signaling security. The solutions are evaluated using specification-compliant 5G network behavior datasets that represent fundamental 5G procedures. We believe that the proposals presented are scalable to further traffic patterns expected to be produced by 5GC signaling and to new attack scenarios based on the ML approach used to learn normal patterns in network traffic.