VSC Training Course: Parallel I/O, December 05-06, 2017


    Delivering high performance I/O is a crucial part of modern high-performance computing (HPC) systems which is usually accomplished by providing different filesystems to ensure that the cluster can handle the varying workloads of its users. This course will start with an introduction to the basics of I/O, an overview over parallel filesystems with a focus on BeeGFS, and the storage hardware available at VSC. Different I/O strategies will be presented from an applications view. The second day will introduce the use of the HDF5 and NetCDF (NetCDF4 and PnetCDF) library interfaces, and hands-on exercises (in C/C++ or Fortran) will allow to immediately test and understand their usage. Performance hints, optimization potential, and best practices for I/O will be discussed in detail throughout the whole course.

    Many scientific applications work on data sets that are way too large to fit into the main memory of a compute node. Choosing the most appropriate filesystem depending on the purpose will ensure that CPUs are actually doing calculations instead of waiting for the next chunk of data to arrive.

    Also, numerical simulations conducted on current HPC systems face an ever growing need for scalability pushing the limitations on size and properties that can be accurately simulated. Therefore, ever larger data sets have to be processed, be it reading input data or writing results. Serial approaches on handling I/O in a parallel application will dominate the performance on massively parallel systems, leaving a lot of computing resources idle during those serial I/O phases.

    In addition to the need for parallel I/O, input and output data is often processed on different and maybe even heterogeneous platforms. Conversion processes can impose a high level of maintenance when different data representations are needed. Portable, self-describing data formats such as HDF5 and netCDF can help to solve these problems.


    Agenda (preliminary) & Content:

    05.12.2017:     VSC I/O Workshop
    09:00 - 09:15   ––––––– Registration –––––––
    09:15 - 10:00   Introduction to I/O
    10:00 - 10:45   Storage technologies
    10:45 - 11:15   ––––––– Coffee Break –––––––
    11:15 - 12:30   VSC storage infrastructure
    12:30 - 13:30   ––––––– Lunch Break –––––––
    13:30 - 14:15   Application view to I/O
    14:15 - 15:00   Performance hints and best practices for I/O
    15:00 - 15:30   ––––––– Coffee Break –––––––
    15:30 - 16:30   MPI I/O (overview)
    16:30 - 16:30   ––––––– End of first day–––––––

    06.12.2017:     Parallel I/O and Portable Data Formats
    09:00 - 09:00   ––––––– Start of second day –––––––
    09:00 - 09:30   Introduction to parallel I/O and portable data formats
    09:30 - 10:30   PnetCDF and NetCDF4
    10:30 - 11:00   ––––––– Coffee Break –––––––
    11:00 - 12:30   PnetCDF and NetCDF4 (cont.)
    12:30 - 13:30   ––––––– Lunch Break –––––––
    13:30 - 14:30   HDF5
    14:30 - 15:00   ––––––– Coffee Break –––––––
    15:00 - 16:30   HDF5 (cont.)
    16:30 - 16:30   ––––––– End of second day (course) –––––––


    Prerequisites (for the second day only):

    Unix/Linux and either C/C++ or Fortran in particular; a basic understanding in parallel programming with MPI would be advantageous but is not mandatory.


    Claudia Blaas-Schenner, Dieter Kvasnicka, Siegfried Reinwald (VSC Team, TU Wien; first day) and
    Sebastian Lührs (Jülich Supercomputing Centre; second day)



    Date, Time, and Location:

    05. - 06.12.2017, 09:00 - 16:30,
    FH Internet-Raum FH1 (TU Wien, Wiedner Hauptstraße 8-10, ground floor, red area)


    Registration for this course is closed.

    Registration is possible for the entire 2 days course or for a selected day only.

    Registration deadline is Monday, November 6, 2017, with priority rules. Acceptance will be approved on November 7, 2017. As long as seats are available there will be an extended registration period without priority rules.

    Priority for acceptance: first - active users of the VSC systems, second - students and members of Austrian universities and public research institutes, third - other applicants.


    VSC users: none
    Students and members of Austrian universities and public research institutes: none
    Students and members of other universities and public research institutes: 120 € for 2 days (60 € for 1 day)
    Others: 400 € for 2 days (200 € for 1 day)

    Information about payment will be provided with the confirmation email.

    Coffee breaks are included in the course fee (lunch is not included).


    Opens external link in new windowSlides [html/pdf] - VSC Team: VSC I/O Workshop

    Opens external link in new windowSlides [pdf] - Sebastian Lührs: Parallel I/O and Portable Data Formats


    Claudia Blaas-Schenner, vsc-seminar@list.tuwien.ac.at


    Upcoming and past events of the VSC School Seminar:

    Opens external link in new windowvsc.ac.at/training