{"id":10558,"date":"2020-12-04T20:23:17","date_gmt":"2020-12-04T19:23:17","guid":{"rendered":"https:\/\/gaia.ub.edu\/?p=10558"},"modified":"2020-12-04T20:46:47","modified_gmt":"2020-12-04T19:46:47","slug":"gaia-edr3-bulk-catalogue-available-in-a-better-compression-format","status":"publish","type":"post","link":"https:\/\/gaia.ub.edu\/?p=10558","title":{"rendered":"Gaia EDR3 bulk catalogue available in a better compression format"},"content":{"rendered":"<p>The Gaia group at the University of Barcelona (<a href=\"http:\/\/www.ieec.cat\">IEEC<\/a> &#8211; <a href=\"http:\/\/www.icc.ub.edu\">ICCUB<\/a>), in cooperation with <a href=\"https:\/\/www.dapcom.es\">DAPCOM Data Services<\/a>\u00a0(a technological spin-off company of the <a href=\"https:\/\/www.upc.edu\">UPC<\/a> and the <a href=\"https:\/\/www.ub.edu\">UB<\/a>), has <strong>published an alternative copy of the bulk data files from <a href=\"https:\/\/www.cosmos.esa.int\/web\/gaia\/early-data-release-3\">Gaia EDR3<\/a><\/strong> &#8211; the Early Data Release 3 from Gaia.<\/p>\n<p>Gaia EDR3 was published yesterday, 3rd December 2020. Besides the <a href=\"https:\/\/gea.esac.esa.int\/archive\/\">on-line catalogue<\/a>, <a href=\"http:\/\/cdn.gea.esac.esa.int\/Gaia\/gedr3\/\">bulk CSV<\/a> files were also made available for download &#8211; an interesting option for exhaustive analyses. Such files are officially offered in &#8220;<em>csv.gz<\/em>&#8221; format, that is, compressed with the widely known <em>gzip<\/em> compressor.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-9434 alignleft\" src=\"https:\/\/gaia.ub.edu\/wp-content\/uploads\/2019\/02\/logo-01_RGB_small.jpg\" alt=\"\" width=\"160\" height=\"121\" \/> DAPCOM provides <a href=\"https:\/\/www.dapcom.es\/fapec\/\">FAPEC<\/a>, a professional data compression software offering high compression ratios at high speeds. One of the options provided is the compression of tabular (CSV-like) text files, such as those from the bulk Gaia EDR3. As a service to the worldwide astronomical community, and also as a demonstration of the capabilities of FAPEC, DAPCOM and the Gaia IEEC\/ICCUB Group converted the GaiaSource files from the official Gaia EDR3 bulk CSV repository into the FAPEC format, <strong>reducing the total size from 613 GB to 495 GB<\/strong> &#8211; that is, <strong>19% smaller than with <em>gzip<\/em><\/strong>. Other data compressors like <em>bzip2<\/em>, <em>rar<\/em>, <em>Zstandard<\/em> or <em>7-zip<\/em> cannot reach this mark.<\/p>\n<p>&nbsp;<\/p>\n<p>You can now download Gaia EDR3 in <em>csv.fapec<\/em> format here:<\/p>\n<pre style=\"text-align: center;\"><a href=\"https:\/\/gaia.ub.edu\/GaiaEDR3\/\"><strong>Gaia EDR3 <em>csv.fapec<\/em> bulk download<\/strong><\/a><\/pre>\n<p>The additional tables available in the bulk Gaia EDR3 catalogue will also be converted and published during the coming days.<\/p>\n<p><strong>Free FAPEC decompression licenses<\/strong> can be obtained from the <a href=\"https:\/\/www.dapcom.es\/get-fapec\/\">DAPCOM website<\/a>. DAPCOM plans to make a new FAPEC release soon, including a freely downloadable decompressor with Python bindings.<\/p>\n<p>Have fun!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Gaia group at the University of Barcelona (IEEC &#8211; ICCUB), in cooperation with DAPCOM Data Services\u00a0(a technological spin-off company of the UPC and the UB), has published an alternative copy of the bulk data files from Gaia EDR3 &#8211; the Early Data Release 3 from Gaia. Gaia EDR3 was published yesterday, 3rd December 2020. [&hellip;]<\/p>\n","protected":false},"author":9,"featured_media":9467,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,65,46,116],"tags":[128,127],"class_list":["post-10558","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-hot-topics","category-news","category-slider","tag-fapec","tag-gaiaedr3"],"_links":{"self":[{"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/posts\/10558","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10558"}],"version-history":[{"count":1,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/posts\/10558\/revisions"}],"predecessor-version":[{"id":10559,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/posts\/10558\/revisions\/10559"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/media\/9467"}],"wp:attachment":[{"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10558"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10558"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10558"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}