Text-Processing

通過 jq 連接數千個巨大的 json 文件

  • September 23, 2022

我有數千個 JSON 文件，如下所示：

文件 1 ( key1:value_list1)

{"2mac:acg":["1-248","3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43"]}

文件 2 ( key2:value_list2)

{"4qld:aaa":["3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43","19-55"]}

文件 3 ( key3:value_list3)

{"6k8h:c":["1-248","2-134","3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43"]}

我想將這些文件合併為一個，應該如下所示：

{"2mac:acg":["1-248","3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43"], "4qld:aaa":["3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43","19-55"], "6k8h:c":["1-248","2-134","3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43"]}

連接模型應該是{key1:value_list_1, key2:value_list2, key3:value_list3,...,key_last:value_list_last}

感謝@thanasisp，我使用 jq 通過 j​​q -s ‘add’ file1 file2 file3 連接它們。當連接數百個文件時，它執行良好。但是如果有數千個文件，它就不起作用並返回錯誤消息：參數列表太長！所以我想知道如何解決它，以及是否有其他方法來處理它。謝謝！PS：伺服器記憶體足夠。

jq -c -s add file*

這會將與模式匹配的所有文件讀file*入jq. ( -s)--slurp選項導致從所有輸入文件創建單個數組。這個大數組的每個元素都是其中一個文件中的一個對象。數組元素組合在一起add形成一個單一的對象。

該-c選項使jq產生“緊湊”輸出。

如果文件太多，shell 將由於超出命令行的最大允許長度而無法執行命令。

如果發生這種情況，您可以使用find創建 JSON 對象流以供jq命令處理。

find . -name '*.json' -type f -exec cat {} + | jq -c -s add >final

這使用catfrom從輸入文件（名稱以目前目錄中或以下find結尾的任何正常文件）創建 JSON 對象流。.json該jq命令將這些收集到一個數組中，然後像以前一樣組合成一個對象。最終結果輸出到名為final.

請注意，如果鍵之間存在衝突（兩個或多個文件中的相同鍵），則找到的最後一個鍵及其值將覆蓋前一個鍵及其值。

引用自：https://unix.stackexchange.com/questions/718397

相關問答

Text-Processing

文本文件轉 JSON

  • October 7, 2022
檢視問答
Bash

如何使用 jq 檢查值並更改 JSON 中的其他鍵？

  • October 1, 2022
檢視問答
Text-Processing

解析冒號分隔值對

  • July 11, 2021
檢視問答
Text-Processing

使用 jq 將 Json 表數組轉換為對象

  • May 26, 2020
檢視問答
Text-Processing

在深度嵌套的 JSON 中為特定鍵名選擇特定的所有鍵路徑和值

  • February 1, 2020
檢視問答
Text-Processing

使用 JQ 將數組的 JSON 轉換為帶有標題的 CSV

  • January 30, 2020
檢視問答