如何從日誌中獲取模式列表並從中生成報告？

我有一個 C++ 執行檔，它掃描我的一些文件，這些文件中有一些使用者 ID。掃描完成後，它會生成下面的日誌文件，(abc.log)如下所示 -

INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_32_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_32_200003_5.data"
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 45576752 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 870206432 with value badge_leaf_cat and status -2
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:120) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_32_200003_5.data is valid
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_13_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_13_200003_5.data"
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 876269533 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 17256973 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 830173693 with value badge_leaf_cat and status -2
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:120) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_13_200003_5.data is valid
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_0_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_0_200003_5.data"
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744073135142816 num attributes: 0 seeing 1 bad records from 365 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744073698151136 num attributes: 0 seeing 2 bad records from 595 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744072929739296 num attributes: 0 seeing 3 bad records from 1214 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:117) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data is corrupt

現在我需要 grep 上面的日誌並找出它已經掃描了多少文件，有多少使用者 id 失敗，以及哪些文件已損壞。

• 對於它掃描了多少文件，我需要checking file在每一行中查找單詞，並在此基礎上增加它掃描的文件數。
• 對於失敗的唯一使用者 ID，我需要failed reading user id在一行中查找單詞並在此基礎上增加計數，然後提供失敗的使用者 ID 列表。
• 如果任何使用者 id 失敗，則意味著保存該使用者 id 的文件將損壞，因此我需要is corrupt在每一行中查找單詞並找到損壞的文件名。通常，此文件database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data已損壞。

以下是我在掃描上述日誌後想看到的響應 -

Total Number of Files Scanned - 1000
Total Number of Unique User ID failed - 10000
Total Number of Files Corrupted - 5

List of Unique User Id's which are corrupt - 
UserId-A
UserId-B

Files which are corrupted - 
FileName-A
FileName-B

使用 grep 掃描日誌後，我將如何繼續並獲得上述結果？

嘗試以下腳本：

#!/bin/bash

logfile="$1"

nfiles=$(grep -c 'checking file' "$logfile")
failed_userid=($(grep -oP 'failed reading user id: \K[^ ]*' "$logfile"))
corrupted_files=($(grep -oP '[^ ]*(?= is corrupt)' "$logfile"))


echo "Total Number of Files Scanned - $nfiles"
echo "Total Number of Unique User ID failed - ${#failed_userid[@]}"
echo "Total Number of Files Corrupted - ${#corrupted_files[@]}"
echo

echo "List of Unique User Id's which are corrupt - "
for uid in "${failed_userid[@]}"; do
  echo "$uid"
done

echo

echo "Files which are corrupted - "
for corf in "${corrupted_files[@]}"; do
  echo "$corf"
done

執行它

$ ./script file.log

您的問題輸入的結果看起來像

Total Number of Files Scanned - 3
Total Number of Unique User ID failed - 3
Total Number of Files Corrupted - 1

List of Unique User Id's which are corrupt - 
18446744073135142816
18446744073698151136
18446744072929739296

Files which are corrupted - 
/database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data

簡短說明：

• -cgrep 的選項計算匹配的行
• -P啟用 perl 正則表達式語法
• -o僅匹配部分行
• (?=構造是所謂的正向前瞻（將其作為模式，但不包括在輸出中）
• \K是後視斷言（採取整個模式，但從結果中丟棄一切到這一點）

其餘的應該是顯而易見的。但是請注意，我假設文件名中沒有空格！

引用自：https://unix.stackexchange.com/questions/164524