Design and Implementation of a Parallel I/O Runtime System for Irregular Applications

Authors: No, J.1; Park, S-s.2; Carretero Perez, J.3; Choudhary, A.4

Source: Journal of Parallel and Distributed Computing, Volume 62, Number 2, February 2002 , pp. 193-220(28)

Publisher: Academic Press

Buy & download fulltext article:

The full text article is not available for purchase.

The publisher only permits individual articles to be downloaded by subscribers.


We present the design, implementation, and evaluation of a runtime system based on collective I/O techniques for irregular applications. The design is motivated by the requirements of a large number of science and engineering applications including teraflops applications, where the data must be reorganized into a canonical form for further processing or restarts. We present two designs: “collective I/O” and “pipelined collective I/O.” In the first design, all processors participate in I/O simultaneously, making scheduling of I/O requests simpler but creating possible contention at the I/O nodes. In the second design, processors are organized into several groups so that only one group performs I/O while the next group performs the communication to rearrange data and this entire process is dynamically pipelined to reduce I/O node contention. In other words, the design provides support for dynamic contention management. We also present a software caching method using collective I/O to reduce I/O cost by reusing the data already present in the memory of other nodes. Chunking and on-line compression mechanisms are included in both models. We present performance results on the Intel Paragon at Caltech and on the ASCI/Red teraflops machine at Sandia National Laboratories. © 2002 Elsevier Science (USA).

Document Type: Research Article


Affiliations: 1: Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, Illinois, 60440 2: Department of Computer Science and Engineering, Anyang University, Republic of Korea 3: Departamento de Informatica, Universidad Carlos III de Madrid, Leganes, 28911, Spain 4: Electrical and Computer Engineering, Northwestern University, Evanston, Illinois, 60208

Publication date: February 1, 2002

Related content



Free Content
Free content
New Content
New content
Open Access Content
Open access content
Subscribed Content
Subscribed content
Free Trial Content
Free trial content

Text size:

A | A | A | A
Share this item with others: These icons link to social bookmarking sites where readers can share and discover new web pages. print icon Print this page