Initial commit

EventId,EventTemplate

E1,<*>:<*> Served block blk_<*> to /<*>

E2,<*>:<*> Starting thread to transfer block blk_<*> to <*>:<*>

E3,<*>:<*>:Got exception while serving blk_<*> to /<*>:

E4,BLOCK* ask <*>:<*> to delete  blk_<*>

E5,BLOCK* ask <*>:<*> to replicate blk_<*> to datanode(s) <*>:<*>

E6,BLOCK* NameSystem.addStoredBlock: blockMap updated: <*>:<*> is added to blk_<*> size <*>

E7,BLOCK* NameSystem.allocateBlock: /<*>/part-<*>. blk_<*>

E8,BLOCK* NameSystem.delete: blk_<*> is added to invalidSet of <*>:<*>

E9,Deleting block blk_<*> file /<*>/blk_<*>

E10,PacketResponder <*> for block blk_<*> terminating

E11,Received block blk_<*> of size <*> from /<*>

E12,Received block blk_<*> src: /<*>:<*> dest: /<*>:<*> of size <*>

E13,Receiving block blk_<*> src: /<*>:<*> dest: /<*>:<*>

E14,Verification succeeded for blk_<*>

EventId,EventTemplate

E1,<*>Adding an already existing block<*>

E2,<*>Verification succeeded for<*>

E3,<*>Served block<*>to<*>

E4,<*>Got exception while serving<*>to<*>

E5,<*>Receiving block<*>src:<*>dest:<*>

E6,<*>Received block<*>src:<*>dest:<*>of size<*>

E7,<*>writeBlock<*>received exception<*>

E8,<*>PacketResponder<*>for block<*>Interrupted<*>

E9,<*>Received block<*>of size<*>from<*>

E10,<*>PacketResponder<*>Exception<*>

E11,<*>PacketResponder <*> for block <*> terminating<*>

E12,<*>:Exception writing block<*>to mirror<*>

E13,<*>Receiving empty packet for block<*>

E14,<*>Exception in receiveBlock for block<*>

E15,<*>Changing block file offset of block<*>from<*>to<*>meta file offset to<*>

E16,<*>:Transmitted block<*>to<*>

E17,<*>:Failed to transfer<*>to<*>got<*>

E18,<*>Starting thread to transfer block<*>to<*>

E19,<*>Reopen Block<*>

E20,<*>Unexpected error trying to delete block<*>BlockInfo not found in volumeMap<*>

E21,<*>Deleting block<*>file<*>

E22,<*>BLOCK* NameSystem<*>allocateBlock:<*>

E23,<*>BLOCK* NameSystem<*>delete:<*>is added to invalidSet of<*>

E24,<*>BLOCK* Removing block<*>from neededReplications as it does not belong to any file<*>

E25,<*>BLOCK* ask<*>to replicate<*>to<*>

E26,<*>BLOCK* NameSystem<*>addStoredBlock: blockMap updated:<*>is added to<*>size<*>

E27,<*>BLOCK* NameSystem<*>addStoredBlock: Redundant addStoredBlock request received for<*>on<*>size<*>

E28,<*>BLOCK* NameSystem<*>addStoredBlock: addStoredBlock request received for<*>on<*>size<*>But it does not belong to any file.

E29,<*>PendingReplicationMonitor timed out block<*>

E30,<*>BLOCK* ask <*> to delete <*>

## HDFS_v1

HDFS (http://hadoop.apache.org/hdfs) is the Hadoop Distributed File System designed to run on commodity hardware. Due to the popularity of HDFS, it has been widely studied in the literature. 

This log set is generated in a private cloud environment using benchmark workloads, and manually labeled through handcrafted rules to identify the anomalies. The logs are sliced into traces according to block ids. Then each trace associated with a specific block id is assigned a groundtruth label: normal/anomaly. 

We have preprocessed the dataset for easy use in research, including:

+ HDFS_templates.csv

+ anomaly_label.csv

+ Event_traces.csv

+ Event_occurrence_matrix.csv

+ HDFS.npz

### Download

The raw logs are available for downloading at https://github.com/logpai/loghub.

### Citation

If you use the HDFS_v1 dataset from loghub in your research, please cite the following papers.

+ Wei Xu, Ling Huang, Armando Fox, David Patterson, Michael Jordan. [Detecting Large-Scale System Problems by Mining Console Logs](https://people.eecs.berkeley.edu/~jordan/papers/xu-etal-sosp09.pdf), in Proc. of the 22nd ACM Symposium on Operating Systems Principles (SOSP), 2009.

+ Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, Michael R. Lyu. [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://arxiv.org/abs/2008.06448). IEEE International Symposium on Software Reliability Engineering (ISSRE), 2023.

---------------------------------------------------------------------

## HDFS_v2

HDFS (http://hadoop.apache.org/hdfs) is the Hadoop Distributed File System designed to run on commodity hardware. Due to the popularity of HDFS, it has been widely studied in the literature. 

The log set was collected by aggregating logs from the HDFS system in our lab at CUHK for research purpose, which comprises one name node and 32 data nodes. The logs are aggregated at the node level. However, three nodes have been repaired and unfortunately some logs are lost. The logs have a huge size (over 16GB) and are provided as-is without further modification or labelling, which may involve both normal and abnormal cases.

### Download

The raw logs are available for downloading at https://github.com/logpai/loghub.

### Citation

If you use this dataset from loghub in your research, please cite the following papers.

+ Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, Michael R. Lyu. [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://arxiv.org/abs/2008.06448). IEEE International Symposium on Software Reliability Engineering (ISSRE), 2023.

---------------------------------------------------------------------

## HDFS_v3_TraceBench

TraceBench is an open data set for trace-oriented monitoring, collected using MTracer on a HDFS system deployed in a real IaaS environment. When collecting, we considered different scenarios, involving multiple scales of clusters, different kinds of user requests, various speeds of workloads, etc. In addition to recording the traces when the HDFS runs normally, we also collected the traces under the situations with various faults injected. There are 17 faults we have injected, including function and performance faults (and real system bugs). The total collection time of TraceBench is more than 180 hours, resulting 364 files that record more than 370,000 traces.

The data are collected through instrumenting the HDFS system. We have converted the raw sql files to csv files and preprocessed the trace logs for easy use in research, including:

+ normal_trace.csv

+ failure_trace.csv

+ eventId.json

+ normal_taskId.json

+ failure_taskId.json

For more detailed information, please visit the dataset project: https://mtracer.github.io/TraceBench.

### Download

The raw logs are available for downloading at https://github.com/logpai/loghub.

### Citation

If you use this dataset from loghub in your research, please cite the following paper.

+ Jingwen Zhou, Zhenbang Chen, Ji Wang, Zibin Zheng, and Michael R. Lyu. [TraceBench: An Open Data Set for Trace-oriented Monitoring](http://zbchen.github.io/Papers_files/cloudcom2014.pdf), in Proceedings of the 6th IEEE International Conference on Cloud Computing Technology and Science (CloudCom), 2014.

+ Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, Michael R. Lyu. [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://arxiv.org/abs/2008.06448). IEEE International Symposium on Software Reliability Engineering (ISSRE), 2023.