- Browse by Subject
Browsing by Subject "HPC"
Now showing 1 - 3 of 3
Results Per Page
Sort Options
Item ElasticBroker: Combining HPC with Cloud to Provide Realtime Insights into Simulations(arXiv, 2020) Li, Feng; Wang, Dali; Yan, Feng; Song, Fengguang; Computer and Information Science, School of ScienceFor large-scale scientific simulations, it is expensive to store raw simulation results to perform post-analysis. To minimize expensive I/O, "in-situ" analysis is often used, where analysis applications are tightly coupled with scientific simulations and can access and process the simulation results in memory. Increasingly, scientific domains employ Big Data approaches to analyze simulations for scientific discoveries. However, it remains a challenge to organize, transform, and transport data at scale between the two semantically different ecosystems (HPC and Cloud systems). In an effort to address these challenges, we design and implement the ElasticBroker software framework, which bridges HPC and Cloud applications to form an "in-situ" scientific workflow. Instead of writing simulation results to parallel file systems, ElasticBroker performs data filtering, aggregation, and format conversions to close the gap between an HPC ecosystem and a distinct Cloud ecosystem. To achieve this goal, ElasticBroker reorganizes simulation snapshots into continuous data streams and send them to the Cloud. In the Cloud, we deploy a distributed stream processing service to perform online data analysis. In our experiments, we use ElasticBroker to setup and execute a cross-ecosystem scientific workflow, which consists of a parallel computational fluid dynamics (CFD) simulation running on a supercomputer, and a parallel dynamic mode decomposition (DMD) analysis application running in a Cloud computing platform. Our results show that running scientific workflows consisting of decoupled HPC and Big Data jobs in their native environments with ElasticBroker, can achieve high quality of service, good scalability, and provide high-quality analytics for ongoing simulations.Item Performance Models For Distributed Memory HPC Systems And Deep Neural Networks(2019-12) Cardwell, David; Song, Fengguang; Tuceryan, Mihran; Mukhopadhyay, SnehasisPerformance models are useful as mathematical models to reason about the behavior of different computer systems while running various applications. In this thesis, we aim to provide two distinct performance models: one for distributed-memory high performance computing systems with network communication, and one for deep neural networks. Our main goal for the first model is insight and simplicity, while for the second we aim for accuracy in prediction. The first model is generalized for networked multi-core computer systems, while the second is specific to deep neural networks on a shared-memory system.Item X-Composer: Enabling Cross-Environments In-SituWorkflows between HPC and Cloud(ACM, 2021-07) Li, Feng; Wang, Dali; Yan, Feng; Song, Fengguang; Computer Information and Graphics Technology, School of Engineering and TechnologyAs large-scale scientific simulations and big data analyses become more popular, it is increasingly more expensive to store huge amounts of raw simulation results to perform post-analysis. To minimize the expensive data I/O, "in-situ" analysis is a promising approach, where data analysis applications analyze the simulation generated data on the fly without storing it first. However, it is challenging to organize, transform, and transport data at scales between two semantically different ecosystems due to the distinct software and hardware difference. To tackle these challenges, we design and implement the X-Composer framework. X-Composer connects cross-ecosystem applications to form an "in-situ" scientific workflow, and provides a unified approach and recipe for supporting such hybrid in-situ workflows on distributed heterogeneous resources. X-Composer reorganizes simulation data as continuous data streams and feeds them seamlessly into the Cloud-based stream processing services to minimize I/O overheads. For evaluation, we use X-Composer to set up and execute a cross-ecosystem workflow, which consists of a parallel Computational Fluid Dynamics simulation running on HPC, and a distributed Dynamic Mode Decomposition analysis application running on Cloud. Our experimental results show that X-Composer can seamlessly couple HPC and Big Data jobs in their own native environments, achieve good scalability, and provide high-fidelity analytics for ongoing simulations in real-time.