Is your Cybersecurity 3 Strong ( Continued)
2 of 3 - Does your security solution have a SNEM?
This blog is the continuation of the three series blog named ‘Is your Cybersecurity 3 Strong’.
Firstly, let us start with the acronym - In a cybersecurity setting that relies on AI, SNEM is a short form of ‘Single Number Evaluation Metric’.
Now, an essential, but often ignored question any CIO /CISO should be asking while evaluating a cybersecurity solution is how can they determine the effectiveness of the solution. If the answer is not a single number evaluation metric, probably they need to look elsewhere.
Getting into a bit of weeds here, security model often is a classification problem – goal is to predict if a behavior is anomalous or not.
Say, an anomaly detection software claims that its algorithm has reached a prediction accuracy of 98.5%. This implies that its error rate is 1.5%. In this scenario, how do we determine this software is doing well in accurately identifying anomalies? Given that anomalous behaviors is infinitesimally smaller in number compared to normal behaviors, if the anomaly model was trained on a dataset with over 99% non-anomalous behaviors and less than 1% anomalous behaviors, its prediction is still inferior if it has an accuracy of only 98.5%. Moreover, if it has never learned to predict anomalous behavior, model is almost useless. In fact, with this training set, if the learning algorithm was written to ignore all the observed results and the model blindly asserts all the behaviors to be non-anomalous, statistically the error rate is 0.5%!
This may give an impression that the software is doing exceptionally well to secure the network!
This is a classic skewed class problem experienced by many machine learning algorithms when their performance is often biased against the minority class. To overcome such shortcoming, it is essential that the learning algorithm has a way of self-evaluating using a single number evaluation metric. The evaluation metric can be used to fine tune hyper parameters and for choosing threshold values.
For security solutions, enterprises often have lower tolerance for false negatives than false positives. Therefore, good prediction algorithm precociously gives out high precision and high recall, maintaining a healthy trade-off between them.
Summing up, when evaluating a security solution, it is imperative to understand its error analysis methodology and confirm its ability to continuously evolve and self-learn with a proper built-in feedback loop.