OUTPUTS > PILOT / USE CASE 2

SKA (SQUARE KILOMETRE ARRAY)

PROCESSING OF LARGE AMOUNTS OF DATA FROM TELESCOPE OBSERVATIONS

LOFAR (Low Frequency Array) telescope is made up of ~7000 antennas in ~51 stations across Europe, constantly producing and processing data by combining signals.

Observations are stored in the LTA (Long Term Archive) mostly on tape: 45 PB and growing.

Data is available under request and shows high potential for analysis and development of astronomy

Summary

PROCESS, in sum, is expected to help unlock the LOFAR LTA and increase its scientific output:

  • Providing additional processing capacity in a form that makes astronomers‘ workflows more efficient.
  • Allowing astronomers to focus on science instead of on preprocessing data.

Challenges

  1. Despite of LOFAR’s high potential, users find serious problems to make use of information, because of:
    • Data size. 
    • Difficulty of retrieving (up to 10 days for a request!) and processing (several packages and procedures needed).
  1. LOFAR LTA is, so far, an underused resource

Objectives

The main goal of PROCESS for SKA (SQUARE KILOMETRE ARRAY) is to offer a solution for the processing of large amounts of data from telescope observations:

  • To simplify the processing of archived data: by automating processing steps via workflow tools.
  • To achieve a reduction of Radio Astronomical Observations.
  • To make it easy to use & easy to scale up.

Methodology

Techniques and procedures

PROCESS services:

   Step 1: stage in the data.

   Step 2: launch the processing workflow.

   Step 3: stage out results.

Specific resources for the Use Case

Hardware locations: infrastructure of 

  • LOFAR station (network).
  • LOFAR archives (Amsterdam, Jülich, Poznan).
  • PROCESS compute centre (Amsterdam, Munich, Krakow).

Demo locations:  Amsterdam (LTA LOFAR archive +  DAS5 PROCESS compute centre)

Web portal: an on line tool for:

    • Selection of dataset and workflow.
    • Launching of processing pipeline.

Support tools:, based on existing software:

  • uberftp client.
  • globus-url-copy client.
  • voms-client (only on the login node).
  • CVMFS.
  • PiCaS.

Results

For the field of activity

PROCESS provides a solution for SKA (SQUARE KILOMETRE ARRAY) to run containerized workflows, improving the portability and ease of use:

-Web based with an efficient and scalable workflow structure

-Allowing scalability, both horizontal (running the same workflow in parallel) and vertical (applying multi- and many-core techniques)