Title:Automated Dataset Generation System for Collaborative Research of Cyber Threat Intelligence Analysis

Abstract:The objectives of cyber attacks are becoming sophisticated and the attackers are concealing their identity by disguising their characteristics to be others. Cyber Threat Intelligence (CTI) analysis is gaining attention to generate meaningful knowledge for understanding the intention of an attacker and, eventually, to make predictions. Developing the analysis technique requires a high volume and fine quality dataset. However, the organizations which have useful data do not release it to the research community because they do not want to disclose threats toward them and the data assets they have. Due to data inaccessibility, academic research tends to be biased towards the techniques for steps among each CTI process except for the analysis and production step. In this paper, we propose the automated dataset generation system named CTIMiner. The system collects threat data from publicly available security reports and malware repositories. The data is stored in the structured format. We release the source codes and the dataset to the public that includes about 628,000 records from 423 security reports published from 2008 to 2017. Also, we present a statistical feature of the dataset and the techniques that can be developed using it. Moreover, we demonstrate one application example of the dataset that analyzes the correlation and characteristics of incidents. We believe our dataset promotes collaborative research of the threat information analysis to generate CTI.