{"id":9407,"date":"2018-02-07T17:19:06","date_gmt":"2018-02-07T16:19:06","guid":{"rendered":"https:\/\/gaia.ub.edu\/?p=9407"},"modified":"2019-05-14T11:39:08","modified_gmt":"2019-05-14T10:39:08","slug":"gaia-dr2-bulk-catalogue-available-in-fapec-format","status":"publish","type":"post","link":"https:\/\/gaia.ub.edu\/?p=9407","title":{"rendered":"Gaia DR2 bulk catalogue available in FAPEC format"},"content":{"rendered":"<p>The Gaia group at the Universitat de Barcelona (<a href=\"http:\/\/www.ieec.cat\">IEEC<\/a> &#8211; <a href=\"http:\/\/www.icc.ub.edu\">ICCUB<\/a>), in cooperation with <a href=\"http:\/\/www.dapcom.es\">DAPCOM Data Services S.L.<\/a> (a technological spin-off company of the <a href=\"http:\/\/www.upc.edu\">UPC<\/a> and the <a href=\"http:\/\/www.ub.edu\">UB<\/a>), has <strong>published an alternative copy of the bulk data files from <a href=\"https:\/\/www.cosmos.esa.int\/web\/gaia\/dr2\">Gaia DR2<\/a><\/strong> &#8211; the second data release from Gaia.<\/p>\n<p>Gaia DR2 was published on 25 April 2018. Besides the <a href=\"http:\/\/gea.esac.esa.int\/archive\/\">on-line catalogue<\/a>, bulk CSV files were also made available for download &#8211; an interesting option for exhaustive analyses. Such files are officially offered in &#8220;<em>csv.gz<\/em>&#8221; format, that is, compressed with the widely known <em>gzip<\/em> compressor.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-9434 alignleft\" src=\"https:\/\/gaia.ub.edu\/wp-content\/uploads\/2019\/02\/logo-01_RGB_small.jpg\" alt=\"\" width=\"160\" height=\"121\" \/>On 6 February 2019, DAPCOM released FAPEC Archiver 19.0, a professional data compression software offering high compression ratios at high speeds. One of the options provided is the compression of tabular (CSV-like) text files, such as those from the bulk Gaia DR2. As a demonstration of the capacities of FAPEC, DAPCOM converted the full Gaia DR2 bulk CSV files to the FAPEC format, <strong>reducing the total size from 554 GB to 471 GB<\/strong> &#8211; that is, <strong>15% smaller than with <em>gzip<\/em><\/strong>. Other data compressors like <em>bzip2<\/em>, <em>rar<\/em>, <em>Zstandard<\/em> or <em>7-zip<\/em> cannot reach this mark. Specifically, for the largest tables:<\/p>\n<ul>\n<li><em>gaia_source<\/em> has been reduced from 548 GB to 466 GB. We have also combined several CSV files into larger FAPEC archives to improve download transfer speeds.<\/li>\n<li><em>gaia_source_with_rv<\/em>, from 3.1 GB to 2.5 GB.<\/li>\n<li><em>light_curves<\/em>, from 2.3 GB to 1.9 GB.<\/li>\n<\/ul>\n<p>You can now download Gaia DR2 in\u00a0<em>csv.fapec<\/em> format here:<\/p>\n<pre style=\"text-align: center;\"><a href=\"https:\/\/gaia.ub.edu\/GaiaDR2\/\"><strong>Gaia DR2 <em>csv.fapec<\/em> bulk download<\/strong><\/a><\/pre>\n<p>There you will also find the scripts used for the gzip-to-fapec conversion, as well as the log files from the process, during which we checked each of the files to make sure no data was lost or corrupted.<\/p>\n<p><strong>Free FAPEC decompression licenses<\/strong> can be obtained from the <a href=\"http:\/\/www.dapcom.es\/get-fapec\/\">DAPCOM website<\/a>.<\/p>\n<p>Have fun!<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Gaia group at the Universitat de Barcelona (IEEC &#8211; ICCUB), in cooperation with DAPCOM Data Services S.L. (a technological spin-off company of the UPC and the UB), has published an alternative copy of the bulk data files from Gaia DR2 &#8211; the second data release from Gaia. Gaia DR2 was published on 25 April [&hellip;]<\/p>\n","protected":false},"author":9,"featured_media":9467,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,65,46,116],"tags":[],"class_list":["post-9407","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-hot-topics","category-news","category-slider"],"_links":{"self":[{"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/posts\/9407","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9407"}],"version-history":[{"count":13,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/posts\/9407\/revisions"}],"predecessor-version":[{"id":9495,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/posts\/9407\/revisions\/9495"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=\/wp\/v2\/media\/9467"}],"wp:attachment":[{"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9407"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9407"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gaia.ub.edu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9407"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}