BERT-Log: Anomaly Detection for System Logs Based on Pre-trained Language Model

Figures & data

Figure 1. BERT-Log architecture.

Figure 2. Grouping raw logs into log sequence.

Table 1. Log sequences parsed from HDFS dataset.

Block ID	Log sequence	Label
blk_8376667364205250596	[‘5,’ ‘5,’ ‘22,’ ‘5,’ ‘11,’ ‘9,’ ‘11,’ ‘9,’ ‘11,’ ‘9,’ ‘26,’ ‘26,’ ‘26,’ ‘2’]	0
blk_ -3102267849859399193	[‘5,’ ‘5,’ ‘22,’ ‘7,’ ‘11,’ ‘10,’ ‘14,’ ‘7’]	1
blk_535106793607016574	[‘5,’ ‘22,’ ‘5,’ ‘5,’ ‘11,’ ‘9,’ ‘11,’ ‘9,’ ‘11,’ ‘9,’ ‘26,’ ‘26,’ ‘26’]	0
blk_ -48460618822572759	[‘22,’ ‘5,’ ‘5,’ ‘5,’ ‘26,’ ‘26,’ ‘26,’ ‘11,’ ‘9,’ ‘11,’ ‘9,’ ‘11,’ ‘9’]	0
blk_ -6635163678784332104	[‘22,’ ‘5’]	1

Figure 3. Comparison of sliding window between traditional method and our proposed method.

Table 2. Log sequences parsed from BGL dataset.

Node ID	Log sequence	Label
R62-M1-NC-I:J18-U01	[590,590]	1
R55-M0-N9-C:J06-U11	[1053,1113,1056,1052,1052,1053,1052]	0
R65-M0-N4-C:J17-U11	[291,220,218,57,58]	1
R63-M1-NB-C:J02-U11	[291,220]	0
R62-M1-N7-C:J05-U11	[291,220,218,57,291,220,218,57]	1

Table

Algorithm 1: Converting raw unstructured logs from BGL dataset into log sequence

Input: $\mathrm{r}\mathrm{a}\mathrm{w}\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{s}=\left[\mathrm{l}\mathrm{o}{\mathrm{g}}_1,\mathrm{l}\mathrm{o}{\mathrm{g}}_2,\hspace{0.17em}\dots ,\mathrm{l}\mathrm{o}{\mathrm{g}}_{\mathrm{m}}\right]$ /* Inputting the raw logs*/ Progress: 1: $\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}\leftarrow \mathrm{D}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\left(\mathrm{r}\mathrm{a}\mathrm{w}\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{s}\right)$ /*Converting raw logs into events by Drain method*/ 2: $\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{s}\leftarrow \mathrm{s}\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{\_}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\left(\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{t}\mathrm{i}\mathrm{t}\mathrm{m}\mathrm{e},\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e},\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{p}\right)$ /*Producing time windows*/ 3: for $\mathrm{t}=1\hspace{0.17em}:\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{s}.\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}$do /*Traversing time windows*/ 4: for $\mathrm{i}=1\hspace{0.17em}:\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}.\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}$ do /*Traversing event list*/ 5: $\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{d}=\mathrm{f}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}$ 6: if $\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}{\mathrm{s}}_{\mathrm{i}}\left[\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\right]\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}{\mathrm{s}}_{\mathrm{t}}$ do /*Matching time window for event*/ 7: continue; 8: end if 9: for $\mathrm{j}=1\hspace{0.17em}:\mathrm{s}\mathrm{e}\mathrm{q}\mathrm{s}.\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}$ do /*Traversing sequences*/ 10: if $\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}{\mathrm{s}}_{\mathrm{i}}\left[\mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{i}\mathrm{d}\right]==\mathrm{s}\mathrm{e}\mathrm{q}{\mathrm{s}}_{\mathrm{j}}\left[\mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{i}\mathrm{d}\right]$ do /*Matching node id for event*/ 11: $\mathrm{s}\mathrm{e}\mathrm{q}{\mathrm{s}}_{\mathrm{j}}\left[\mathrm{s}\mathrm{e}\mathrm{q}\right]\leftarrow \mathrm{s}\mathrm{e}\mathrm{q}{\mathrm{s}}_{\mathrm{j}}\left[\mathrm{s}\mathrm{e}\mathrm{q}\right]+\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}{\mathrm{s}}_{\mathrm{i}}\left[\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{d}\right]$ /*Adding new event*/ 12: $\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{d}\leftarrow \mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}$ 13: break 14: end if 15: end for 16: end for 17: if $\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{d}==\mathrm{f}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}$ do 18: $\mathrm{s}\mathrm{e}\mathrm{q}{\mathrm{s}}_{\mathrm{s}\mathrm{e}\mathrm{q}\mathrm{s}.\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}+1}\left[\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\right]\leftarrow \mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}{\mathrm{s}}_{\mathrm{t}}$ /*Adding time into sequences*/ 19: $\mathrm{s}\mathrm{e}\mathrm{q}{\mathrm{s}}_{\mathrm{s}\mathrm{e}\mathrm{q}\mathrm{s}.\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}+1}\left[\mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{i}\mathrm{d}\right]\leftarrow \mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}{\mathrm{s}}_{\mathrm{i}}\left[\mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{i}\mathrm{d}\right]$ /*Adding node id into sequences*/ 20: $\mathrm{s}\mathrm{e}\mathrm{q}{\mathrm{s}}_{\mathrm{s}\mathrm{e}\mathrm{q}\mathrm{s}.\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}+1}\left[\mathrm{s}\mathrm{e}\mathrm{q}\right]\leftarrow \mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}{\mathrm{s}}_{\mathrm{i}}\left[\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{d}\right]$ /*Adding event id into sequences*/ 21: end if 22: end for Output: log sequence

Table

Algorithm 2: Log embedding

Input: $se{q}_i=i{d}_1,i{d}_2,\dots ,i{d}_k$ $seqs=\left[se{q}_1,se{q}_2,\dots ,se{q}_m\right]$ /*Inputting log sequences*/ Progress: 1: $token\leftarrow \mathrm{\varnothing }$ /*Initializing the token*/ 2: for $i=1:n$ do /*Traversing log sequences*/ 3: $temp\leftarrow \left[CLS\right]se{q}_i\left[SEP\right]$ /*Adding [CLS][SEP] for each sequence*/ 4: $token\leftarrow token+temp$ /*Forming the token*/ 5: end for 6: $token\leftarrow mask\left(token\right)$ /*Adding the mask for token*/ 7: $token\leftarrow pad\left(token\right)$ /*Padding the token*/ 8: position$\leftarrow position\left(\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{\_}\mathrm{i}\mathrm{d}\mathrm{s}\right)$ /*Position embedding*/ 9: segment$\leftarrow \mathrm{s}\mathrm{e}\mathrm{g}\mathrm{m}ent\left(token\mathrm{\_}type\mathrm{\_}ids\right)$ /*Segment embedding*/ 10: $\mathrm{X}\leftarrow token+p\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{s}\mathrm{e}\mathrm{g}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}$ /*Forming log embedding vectors*/ 11: $\mathrm{X}\leftarrow \mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\left(\mathrm{X}\right)$ /*Do the $\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}$ function*/ 12: $\mathrm{X}\leftarrow \mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left(\mathrm{X}\right)$ /*Do the $\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}$ function*/ Output: log embedding vector $X$

Figure 4. Log encoders architecture.

Table

Algorithm 3: Log encoders

Input: log embedding vector $X$ Progress: 1: $InitializeO\in R^{d\times d_q}$ /*Initializing output vectors*/ 2: for $j=1:encoder\mathrm{\_}layers.size$ do /* Traversing encoder layers*/ 3: $MatricesQ,K,V\in R^{d\times d_q}$ /*Creating Q、K、V weight matrixes*/ 4: for $i=1:heads.size$ do /*Creating multi head attention maxtix*/ 5: Divide $Q,K$ and $V$into multiple groups based on $X$ 6: end for 7: $attention\leftarrow matmul\left(Q,K.permute\left(params\right)\right)/scale$ /*Computing attention*/ 8: if $maskisnotNULL$ do 9: $\mathrm{a}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\leftarrow \mathrm{a}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}.\mathrm{m}\mathrm{\_}\mathrm{f}\mathrm{i}\mathrm{l}\mathrm{l}\left(\mathrm{m}\mathrm{a}\mathrm{s}\mathrm{k}==\hspace{0.17em}0,\hspace{0.17em}-1\mathrm{e}10\right)$ 10: end if 11: $attention\leftarrow softmax\left(attention,dim=-1\right)$ /* Normalizing attention*/ 12: $\theta \leftarrow matmul\left(attention,V\right)$ /*Computing multi head attention*/ 13: $\theta \leftarrow layerNorm\left(\theta \right)$ /*Computing by layerNorm function*/ 14: $\theta \leftarrow FFN\left(\theta \right)$ /*Computing by FFN*/ 15: $O\leftarrow O+\theta$ /*Computing output vectors*/ 16: end for Output: $encoder\mathrm{l}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{s}\mathrm{v}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{r}$: $O$

Figure 5. Structure for the log anomaly classifier.

Table 3. Summary of log messages.

Dataset	Span of time	File size	Logs	Anomalies
HDFS	38.7 hours	743,185,031 bytes	11,175,629 logs	16,838 blocks
BGL	214.7 days	1,577,982,906 bytes	4,747,963 logs	348,469 logs

Table 4. Parameters of BERT-Log.

Parameters	Value
Learning rate	2e-5
Batch size (HDFS)	16
Batch size (BGL)	24
Sequences size	510
Vocab size	30522
Hidden size	768
Hidden layers	12
Attention heads	12
Optimizer	AdamW
Loss function	CrossEntropyLoss

Table 5. Evaluation on HDFS dataset.

Model	Precision	Recall	F1-score
PCA	1.00	0.67	0.80
IM	0.81	0.62	0.70
LogCluster	1.00	0.83	0.91
SVM	1.00	0.91	0.95
LR	0.84	0.73	0.78
DeepLog	0.96	0.96	0.96
LogRobust	0.98	0.96	0.97
HitAnomaly	1.00	0.97	0.98
LAnoBERT	–	–	0.96
NeuralLog	0.96	1.00	0.98
LogBERT	0.87	0.78	0.82
UniLog	0.88	0.65	0.75
BERT-Log	0.996≈1	0.996≈1	0.993≈0.99

Table 6. Evaluation on BGL dataset.

Model	Precision	Recall	F1-Score
PCA	0.66	0.50	0.11
IM	0.21	0.54	0.31
LogCluster	0.11	0.56	0.19
SVM	0.95	0.41	0.57
LR	0.96	0.43	0.50
DeepLog	0.91	0.71	0.79
LogRobust	0.91	0.77	0.83
HitAnomaly	0.95	0.90	0.92
LAnoBERT	-	-	0.87
NeuralLog	0.98	0.98	0.98
LogBERT	0.89	0.92	0.91
UniLog	0.98	1.00	0.99
BERT-Log	0.996≈1	0.998≈1	0.994≈0.99

Table 7. Evaluation on HDFS dataset by Pre-trained Language Model.

Model	Precision	Recall	F1-score	Parameters
RoBERTa	0.997	0.997	0.995	355M
T5	0.998	0.992	0.991	220M
UniLM	0.992	0.997	0.990	330M
ELECTRA	0.997	0.991	0.989	335M
ERNIE	0.996	0.997	0.994	750M
SpanBERT	0.997	0.998	0.995	335M
BERT-Log	0.996	0.996	0.993	110M

Table 8. Evaluation on HDFS dataset by Dataset Size.

Dataset Size	Model	Precision	Recall	F1-Score
1%	SVM	1.000	0.163	0.280
	LogCluster	1.000	0.279	0.436
	LR	1.000	0.163	0.280
	DeepLog	1.000	0.365	0.535
	BERT-Log	0.980	0.987	0.969
10%	SVM	0.996	0.489	0.656
	LogCluster	1.000	0.240	0.386
	LR	0.989	0.521	0.682
	DeepLog	0.994	0.217	0.357
	BERT-Log	0.997	0.999	0.996
20%	SVM	0.968	0.539	0.692
	LogCluster	1.000	0.234	0.379
	LR	0.973	0.748	0.846
	DeepLog	0.799	0.209	0.331
	BERT-Log	0.994	0.999	0.993
50%	SVM	0.975	0.569	0.719
	LogCluster	1.000	0.194	0.325
	LR	0.974	0.567	0.716
	DeepLog	0.484	0.199	0.282
	BERT-Log	1.000	0.995	0.998

Table 9. Evaluation on BGL dataset by training ratio.

Training Ratio	Model	Precision	Recall	F1-Score
1%	SVM	0.981	0.341	0.506
	LogRobust	0.868	0.752	0.805
	A2log	-	-	-
	HitAnomaly	0.968	0.845	0.902
	BERT-Log	0.990	0.998	0.989
10%	SVM	0.973	0.397	0.564
	LogRobust	0.873	0.812	0.841
	A2log	0.74	0.22	0.34
	HitAnomaly	0.983	0.877	0.922
	BERT-Log	0.994	0.999	0.993
20%	SVM	0.982	0.383	0.551
	LogRobust	0.867	0.821	0.843
	A2log	0.77	0.25	0.37
	HitAnomaly	0.977	0.887	0.929
	BERT-Log	0.995	0.998	0.993
50%	SVM	0.970	0.405	0.571
	LogRobust	0.897	0.792	0.841
	A2log	-	-	-
	HitAnomaly	0.964	0.886	0.923
	BERT-Log	0.994	0.999	0.993

Figure 6. ROC curve comparison.