Evaluating the effectiveness of DNS tunnel detection using a neural network in an intrusion detection system

Evaluating the effectiveness of DNS tunnel detection using a neural network in an intrusion detection system

Marinin N.D. (NRU HSE, Moscow, Russia)
Getman A.I. (NRU HSE, Moscow, Russia; ISP RAS, Moscow, Russia; MIPT, Dolgoprudny, Moscow Region, Russia; MSU, Moscow, Russia)

Abstract

This paper investigates a method for detecting DNS tunnels in network traffic using a neural network. To achieve this, an analysis of current detection approaches was conducted, and a dataset for neural network training was prepared. The proposed model takes as input a sequence of characters extracted from DNS responses. The trained model demonstrated an F1-score close to one. To validate the proposed approach, modules of the open-source intrusion detection system Snort 3 were extended and modified. The trained model was executed using the compatible LibML module. Experimental results demonstrate detection accuracy close to one with an almost complete absence of false positives. The average DNS packet processing time with the neural network-based detection module enabled increased by 13%, while for mixed traffic consisting of multiple protocols, the processing time increased by only 2%. The analysis of the experimental data confirms that the use of neural networks effectively complements traditional security mechanisms, enabling efficient detection of covert channels encapsulated within DNS traffic without significantly affecting the performance of the signature-based detection engine.

Keywords

DNS tunneling; neural network; intrusion detection system (IDS); covert channels.

Edition

Proceedings of the Institute for System Programming, vol. 38, issue 4, part 2, 2026, pp. 123-142

ISSN 2220-6426 (Online), ISSN 2079-8156 (Print).

DOI: 10.15514/ISPRAS-2026-38(4)-22

For citation

Marinin N.D., Getman A.I. Evaluating the effectiveness of DNS tunnel detection using a neural network in an intrusion detection system. Proceedings of the Institute for System Programming, vol. 38, issue 4, part 2, 2026, pp. 123-142 DOI: 10.15514/ISPRAS-2026-38(4)-22.

Full text of the paper in pdf (in Russian) Back to the contents of the volume