Performance analysis and optimization of in-situ integration of simulation with data analysis: zipping applications up

dc.contributor.authorFu, Yuankun
dc.contributor.authorLi, Feng
dc.contributor.authorSong, Fengguang
dc.contributor.authorChen, Zizhong
dc.contributor.departmentComputer and Information Science, School of Scienceen_US
dc.date.accessioned2019-05-16T18:17:29Z
dc.date.available2019-05-16T18:17:29Z
dc.date.issued2018-06
dc.description.abstractThis paper targets an important class of applications that requires combining HPC simulations with data analysis for online or real-time scientific discovery. We use the state-of-the-art parallel-IO and data-staging libraries to build simulation-time data analysis workflows, and conduct performance analysis with real-world applications of computational fluid dynamics (CFD) simulations and molecular dynamics (MD) simulations. Driven by in-depth performance inefficiency analysis, we design an end-to-end application-level approach to eliminating the interlocks and synchronizations existent in the present methods. Our new approach employs both task parallelism and pipeline parallelism to reduce synchronizations effectively. In addition, we design a fully asynchronous, fine-grain, and pipelining runtime system, which is named Zipper. Zipper is a multi-threaded distributed runtime system and executes in a layer below the simulation and analysis applications. To further reduce the simulation application's stall time and enhance the data transfer performance, we design a concurrent data transfer optimization that uses both HPC network and parallel file system for improved bandwidth. The scalability of the Zipper system has been verified by a performance model and various empirical large scale experiments. The experimental results on an Intel multicore cluster as well as a Knight Landing HPC system demonstrate that the Zipper based approach can outperform the fastest state-of-the-art I/O transport library by up to 220% using 13,056 processor cores.en_US
dc.eprint.versionAuthor's manuscripten_US
dc.identifier.citationFu, Y., Li, F., Song, F., & Chen, Z. (2018). Performance Analysis and Optimization of In-situ Integration of Simulation with Data Analysis: Zipping Applications Up. In Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing (pp. 192–205). New York, NY, USA: ACM. https://doi.org/10.1145/3208040.3208049en_US
dc.identifier.urihttps://hdl.handle.net/1805/19328
dc.language.isoenen_US
dc.publisherACMen_US
dc.relation.isversionof10.1145/3208040.3208049en_US
dc.relation.journalProceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computingen_US
dc.rightsPublisher Policyen_US
dc.sourceAuthoren_US
dc.subjecthigh performance computingen_US
dc.subjectperformance analysis and optimizationen_US
dc.subjectin-situ/in-transit workflowsen_US
dc.titlePerformance analysis and optimization of in-situ integration of simulation with data analysis: zipping applications upen_US
dc.typeArticleen_US
Files
Original bundle
Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
Fu_2018_performance.pdf
Size:
949.32 KB
Format:
Adobe Portable Document Format
Description:
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.99 KB
Format:
Item-specific license agreed upon to submission
Description: