Adaptive methods and tools for shallow packet inspection in anomaly detection within encrypted network traffic

Adaptive methods and tools for shallow packet inspection in anomaly detection within encrypted network traffic

Tsaplin N.A. (KIAM RAS, Moscow, Russia)
Petrov A.P. (ICS RAS, Moscow, Russia)
Kovalev D.Yu. (FRCCSC, Moscow, Russia)

Abstract

The widespread adoption of encrypted network traffic severely limits the applicability of Deep Packet Inspection in modern cloud infrastructures. This paper addresses the challenge of accurate and scalable anomaly detection using Shallow Packet Inspection – an approach that relies solely on metadata from packet headers at OSI layers 2-4, without accessing payload contents. We propose a lightweight, kernel-level SPI framework implemented as a driver for the Hyper-V Extensible Switch, named RuStatExt, which aggregates traffic into flows and extracts statistical features with negligible performance overhead. To maximize detection quality, we introduce a modified greedy feature selection algorithm and a dynamic hyperparameter tuning strategy that scales linearly with the number of monitored virtual machines. The methodology includes filtering of virtual machines, comparison of feature selection techniques, evaluation of unsupervised models, and assessment of detection quality using Precision, Recall, and F1-score. Practical validation is performed using synthetic L2-L4 attacks (SYN/UDP floods) and high-load traffic over a 1 Gb/s link. Our experiments show that Isolation Forest, combined with features selected by the proposed greedy algorithm and dynamically tuned hyperparameters, achieves an F1-score of 0.78, nearly 2 times higher than the static configuration using all features. Crucially, the RuStatExt driver introduces no statistically significant degradation in network throughput for either TCP or UDP traffic.

Keywords

Shallow Packet Inspection (SPI); anomaly detection; unwanted traffic; feature selection; greedy search.

Edition

Proceedings of the Institute for System Programming, vol. 38, issue 2, 2026, pp. 21-34

ISSN 2220-6426 (Online), ISSN 2079-8156 (Print).

DOI: 10.15514/ISPRAS-2026-38(2)-2

For citation

Tsaplin N.A., Petrov A.P., Kovalev D.Yu. Adaptive methods and tools for shallow packet inspection in anomaly detection within encrypted network traffic. Proceedings of the Institute for System Programming, vol. 38, issue 2, 2026, pp. 21-34 DOI: 10.15514/ISPRAS-2026-38(2)-2.

Full text of the paper in pdf (in Russian) Back to the contents of the volume