Rapid Process-Chain Anomaly Detection Using a Multistage Classifier
Timely analysis uses simple methods for simple problems and deep learning for difficult problems, but only if you can tell the difference.
Many cyber-threats involve the transfer or modification of large volumes of data, which is usually stored in thousands of differently-named archives. These threats include the unauthorized gathering and preparation of sensitive data to be exfiltrated by malicious actors, and the internal or external transfers of sensitive information using obsolete, unencrypted protocols.
As it is difficult to manually investigate long lists of file names, we have developed an automated system that ranks the most interesting filenames and can quickly assess the importance of compromised files during an attack.
This process uses mathematical methods from signal processing, information retrieval, statistics, and natural-language processing to assign scores and sort a list of file names by their importance. The method also looks for recurring strings in the filenames using Term Frequency–Inverse Document Frequency (TF-IDF) statistics, and searches for rare or suspicious organization names. For instance, when given a list of several thousand numerical names of digital images, and a few large ZIP files, the method identifies the ZIP files as the most important, due to their file extension and anomalous size.
Preliminary results show that no supervised training is needed, and that the system can operate in different natural languages, e.g. English, Spanish, and Chinese with no additional changes or inputs. The system returned ordered lists of file names containing the most interesting specimens among the first 100 elements.