TWiki> GENIUS Web>300ArchiveSystemDesign (revision 4)

Edit Attach


300 - Archive System Design

A database and query engine will be at the core of the Gaia archive system. This work package is devoted to the study of technologies, development and implementation of these elements for Gaia. The technology choices and the design of the systems should be carefully based on the real user needs, as explored and defined in WP 200. Furthermore, the designed system should support the advanced tools and activities produced in the rest of work packages.Specifically its design should support the Grand Challenges outlined in WP 200, that will require complex and massive queries. The activity should culminate in a full working implementation of the Gaia archive system. In the process, prototypes will be prepared and tested in cooperation with the end users. A relevant point is that the system should be Virtual Observatory compliant and therefore should include a VO layer and the relevant metadata.

Workpackage Breakdown Structure (draft)

310 Management

320 Database systems evaluation

330 Query interface design

340 VO layer

350 HW considerations

  • Virtualisation/Cloud
  • Mirroring

360 Data Centre issues

  • Complementary (local) datasets: both ground-based and space-based mission data

Proposal materials

The following key points should be borne in mind in preparing the proposal case for this workpackage:

  • GAP (precursor to DPAC's CU9) have already started benchmarking DB systems and hardware configurations for a database system from the point of view of offline processing (AGIS) and also possible realtime processing (IDT) presumably with the assumption that the same solution is appropriate for an end-user archive system;
  • The GENIUS proposal must complement and enhance GAP/CU9 work, not duplicate nor replace;
  • It is way beyond the scope of GENIUS (or GAP/CU9 for that matter) to exhaustively study all possible hardware/system configurations;
  • A likely hook on which to hang this section of the proposal is to point to the success of relational technology in serving legacy surveys to the user community (SDSS, UKIDSS, VISTA...), the value of exposing standard SQL interfaces directly to the consumer, and the need to find scalable relational solutions for 10s of billions of row datasets

Draft general justification text (from Bob and Nige)

(The following is taken from a proposal to the UK funding agency for CU9 work, but which was eventually removed from the proposal and is unfunded)

More topic actions...
Topic revision: r4 - 2011-10-25 - NigelHambly
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback