VSC-3 – innovative oil cooling

VSC-3 is a throughput-oriented general-purpose Linux cluster with approximately 2800 compute nodes and a few special-purpose nodes such as GPUs. It is renowned for its innovative oil cooling and unparalleled energy efficiency.

VSC-3 was installed in summer 2014 at the Arsenal TU building (Objekt 214) in Vienna by Opens external link in new windowClusterVision and got a significant extension called VSC-3+ in June 2018.

The initial configuration consists of 2020 nodes, each equipped with 2 processors (Opens external link in new windowIntel Xeon E5-2650v2, 2.6 GHz, 8 cores from the Opens external link in new windowIvy Bridge-EP family) and internally connected with an Intel QDR-80 dual-link high-speed InfiniBand fabric. Energy-efficient cooling is provided by the mineral-oil based CarnotJet System of Opens external link in new windowGreen Revolution Cooling.

All pictures on the right side refer to this initial oil-cooled configuration of VSC-3 where the compute nodes are placed inside of containers filled with the cooling oil. The picture in the lower-right corner shows one bare-metal blade with two nodes on it.

The extension VSC-3+ provides an addition of 864 nodes again with 2 processors each but with a slightly different setup (Opens external link in new windowIntel Xeon E5-2660v2, 2.2 GHz, 10 cores from the Opens external link in new windowIvy Bridge-EP family) and connected with single-rail Mellanox FDR InfiniBand. The VSC-3+ extension is air cooled using about half of the racks of the predecessor system VSC-2.

VSC-3 also provides a GPU partition (50 nodes with 1 NVIDIA Pascal GeForce GTX 1080 GPU, 2 nodes with 8 NVIDIA Pascal GeForce GTX 1080 GPUs, and 2 nodes with 10 NVIDIA Pascal GeForce GTX 1080 GPUs) and the dedicated bioinformatics nodes providing Broadwell nodes with extra large RAM and additional high-speed parallel filesystems. Also available are various smaller test partitions like 4 nodes of Intel's KNL (Knights Landing), 8 Haswell nodes interconnected by OmniPath interfaces (Intel's next generation InfiniBand network), and a small testbed of AMD processors.

 

Opens external link in new windowVSC-3 in the TOP500 list (Rank 85 in November 2014)
(The TOP500 entries refer to the initial 2020 nodes configuration of VSC-3 only, we never ran a benchmark for or with the VSC-3+ extension.)

 

Press coverage on VSC-3