Abstract
Cyber security threat detection is the process of identifying anomalous and frequent patterns within related datasets. This is currently a highly labour intensive task using signatures created from previous knowledge and manual exploration, limiting the identification of novel attacks. This thesis proposes a visual analytics solution, combining data mining and visualisation methodologies, in order to overcome these limitations. The first contribution is an anomaly detection algorithm, entitled Discovering Anomalous Terms Using Mining (DATUM), combining frequent itemset mining with a variation of Term Frequency Inverse Document Frequency (TFIDF). By modifying the TFIDF algorithm to consider feature distribution and integrating with the Find Frequent Pattern Outlier Factor (FindFPOF) anomalous record detection algorithm, anomalous patterns are automatically discovered. The results show that DATUM reduces both the number of false positives without loss of anomaly detection accuracy and the sensitivity of the FindFPOF algorithm to its initialisation parameters. The second contribution is a tool entitled Interactive Visual Analytics for Cyber Security (IVACS), combining interval based frequent itemset mining to automatically identify frequent patterns without the use of signatures. Furthermore, interactive, cross-linked visualisations present the temporal evolution of these patterns from varying perspectives, optimised for different discovery tasks. IVACS has been validated through user testing, to provide automated discovery of novel attacks and a reduction in labour for the user. The final contribution is Force Directed Aggregated Parallel Coordinates (FDAPC), for the automation of cluster identification and visual clutter reduction. FDAPC models the inter-axis line segments as springs connected to axis ticks as nodes, applying a Hooke's law algorithm in order to optimise node locations through minimisation of the total system energy. Multiple case studies demonstrate that FDAPC automatically uncovers patterns within large datasets and usability testing has shown benefits to an analyst when compared to classical parallel coordinates.