{"id":26,"date":"2013-11-18T00:32:37","date_gmt":"2013-11-17T23:32:37","guid":{"rendered":"http:\/\/188.166.119.205\/?p=26"},"modified":"2026-11-10T05:52:05","modified_gmt":"2026-11-10T05:52:05","slug":"speeding-up-compressing-with-parallel-computing","status":"publish","type":"post","link":"https:\/\/banyek.net\/?p=26&lang=en","title":{"rendered":"Speeding up compressing with parallel computing"},"content":{"rendered":"

Creating backups and compressing files are always a time-consuming task, for example, to create the daily backup on the kinja related databases were taken about 6,5 hours every day. The first part of creating the backup itself is about 40 minutes long task \u2013 that\u2019s the time of the run of innobackupex, and applying the changed logfiles to db. (I\u2019ll write about this later!) The second part of the backup is to compress files before they will be copied to the storage server\u2026and this step took about 5,5 hours. I have written this part of the backup script with the old-fashioned compression utility \u2013 gzip (and tar of course!)<\/p>\n

# Normally this is the way, how you compress a tarfile on the fly:\ntar -czf backup.tar.gz \/path\/to\/backupdir\n<\/code><\/pre>\n
This is totally awesome in the most of the times, but you have to know one thing: the gzip only uses one processor core during the operation, so if you have a beefy hardware, you can\u2019t even scratch the total throughput what your machine could do.<\/p>\n
So, the solution is to parallelize all the operation.<\/p>\n
Here is a good comparison of parallel compressing software<\/a>.<\/p>\n
I\u2019ve decided to use pigz, so I\u2019ve modified the regarding part of the backup script like this:<\/p>\n
tar -c --use-compress-program=pigz -f backup.tar.gz \/path\/to\/backupdir\n<\/code><\/pre>\n
The result could be read from this article\u2019s head image: the compression itself completes in a half of an hour (this is 11 times faster than before!)<\/p>\n
So my rebuild part of the developer\u2019s database could complete before the devs starting to use it. (hm\u2026 I think this will be a new post either.)<\/p>\n","protected":false},"excerpt":{"rendered":"
Creating backups and compressing files are always a time-consuming task, for example, to create the daily backup on the kinja related databases were taken about 6,5 hours every day. The first part of creating the backup itself is about 40 minutes long task \u2013 that\u2019s the time of the run of innobackupex, and applying the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[134],"tags":[232,126],"class_list":["post-26","post","type-post","status-publish","format-standard","hentry","category-tech-en","tag-compression-en","tag-oldblog-en"],"_links":{"self":[{"href":"https:\/\/banyek.net\/index.php?rest_route=\/wp\/v2\/posts\/26","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/banyek.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/banyek.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/banyek.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/banyek.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=26"}],"version-history":[{"count":1,"href":"https:\/\/banyek.net\/index.php?rest_route=\/wp\/v2\/posts\/26\/revisions"}],"predecessor-version":[{"id":443,"href":"https:\/\/banyek.net\/index.php?rest_route=\/wp\/v2\/posts\/26\/revisions\/443"}],"wp:attachment":[{"href":"https:\/\/banyek.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=26"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/banyek.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=26"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/banyek.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=26"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}