A Customized Protocol Cluster Analysis Method Based on Reinforcement Learning

EasyChair Preprint 8761

Abstract

The network protocol defines the rules of communication between two or more hosts on the Internet. As an indispensable means of information transmission in the network, its structure and definition are directly related to communication security. In the formulation of network security policies, the reverse analysis of unknown protocols plays an irreplaceable role, which is extremely important to the research and evolution of network security. It has a wide range of applications in malicious code detection, deep packet detection, and efficient fuzzing. In this paper, a series of researches, implementations and comparisons are made on the related technologies of reverse analysis of customized protocols. The static analysis technology route based on network tracking is selected, and it design and implement the algorithm model RLPA, which uses reinforcement learning DQN to cluster customized protocols. At the same time, the representative common clustering methods are implemented for experimental comparison. The analysis results indicate that the unsupervised model is not effective for the resolution and classification of customized protocols, while the reinforcement learning and supervised models have superior performance. Among them, the reinforcement learning model It can maintain a stable classification effect in the face of different unfamiliar datasets.

Keyphrases: Clustering, Customized protocol parsing, Protocol Reverse Engineering, Reinforcement Learning

@booklet{EasyChair:8761,
  author    = {Peiying Wu and Xiaohui Li and Junfeng Wang},
  title     = {A Customized Protocol Cluster Analysis Method Based on Reinforcement Learning},
  howpublished = {EasyChair Preprint 8761},
  year      = {EasyChair, 2022}}