- Browse by Author
Browsing by Author "Chen, Ranran"
Now showing 1 - 4 of 4
Results Per Page
Sort Options
Item Accelerating complex modeling workflows in CyberWater using on-demand HPC/Cloud resources(IEEE, 2021-09) Li, Feng; Chen, Ranran; Fu, Yuankun; Song, Fengguang; Liang, Yao; Ranawaka, Isuru; Pamidighantam, Sudhakar; Luna, Daniel; Liang, Xu; Computer Information and Graphics Technology, School of Engineering and TechnologyWorkflow management systems (WMSs) are commonly used to organize/automate sequences of tasks as workflows to accelerate scientific discoveries. During complex workflow modeling, a local interactive workflow environment is desirable, as users usually rely on their rich, local environments for fast prototyping and refinements before they consider using more powerful computing resources. However, existing WMSs do not simultaneously support local interactive workflow environments and HPC resources. In this paper, we present an on-demand access mechanism to remote HPC resources from desktop/laptop-based workflow management software to compose, monitor and analyze scientific workflows in the CyberWater project. Cyber-Water is an open-data and open-modeling software framework for environmental and water communities. In this work, we extend the open-model, open-data design of CyberWater with on-demand HPC accessing capacity. In particular, we design and implement the LaunchAgent library, which can be integrated into the local desktop environment to allow on-demand usage of remote resources for hydrology-related workflows. LaunchAgent manages authentication to remote resources, prepares the computationally-intensive or data-intensive tasks as batch jobs, submits jobs to remote resources, and monitors the quality of services for the users. LaunchAgent interacts seamlessly with other existing components in CyberWater, which is now able to provide advantages of both feature-rich desktop software experience and increased computation power through on-demand HPC/Cloud usage. In our evaluations, we demonstrate how a hydrology workflow that consists of both local and remote tasks can be constructed and show that the added on-demand HPC/Cloud usage helps speeding up hydrology workflows while allowing intuitive workflow configurations and execution using a desktop graphical user interface.Item CyberWater: An Open Framework for Data and Model Integration(2024-05) Chen, Ranran; Liang, Yao; Song, Fengguang; Xia, Yuni; Zheng, JiangyuWorkflow management systems (WMSs) are commonly used to organize/automate sequences of tasks as workflows to accelerate scientific discoveries. During complex workflow modeling, a local interactive workflow environment is desirable, as users usually rely on their rich, local environments for fast prototyping and refinements before they consider using more powerful computing resources. This dissertation delves into the innovative development of the CyberWater framework based on Workflow Management Systems (WMSs). Against the backdrop of data-intensive and complex models, CyberWater exemplifies the transition of intricate data into insightful and actionable knowledge and introduces the nuanced architecture of CyberWater, particularly focusing on its adaptation and enhancement from the VisTrails system. It highlights the significance of control and data flow mechanisms and the introduction of new data formats for effective data processing within the CyberWater framework. This study presents an in-depth analysis of the design and implementation of Generic Model Agent Toolkits. The discussion centers on template-based component mechanisms and the integration with popular platforms, while emphasizing the toolkits ability to facilitate on-demand access to High-Performance Computing resources for large-scale data handling. Besides, the development of an asynchronously controlled workflow within CyberWater is also explored. This innovative approach enhances computational performance by optimizing pipeline-level parallelism and allows for on-demand submissions of HPC jobs, significantly improving the efficiency of data processing. A comprehensive methodology for model-driven development and Python code integration within the CyberWater framework and innovative applications of GPT models for automated data retrieval are introduced in this research as well. It examines the implementation of Git Actions for system automation in data retrieval processes and discusses the transformation of raw data into a compatible format, enhancing the adaptability and reliability of the data retrieval component in the adaptive generic model agent toolkit component. For the development and maintenance of software within the CyberWater framework, the use of tools like GitHub for version control and outlining automated processes has been applied for software updates and error reporting. Except that, the user data collection also emphasizes the role of the CyberWater Server in these processes. In conclusion, this dissertation presents our comprehensive work on the CyberWater framework’s advancements, setting new standards in scientific workflow management and demonstrating how technological innovation can significantly elevate the process of scientific discovery.Item CyberWater: An Open Framework for Data and Model Integration in Water Science and Engineering(ACM, 2022-10-17) Chen, Ranran; Li, Feng; Bieger, Drew; Song, Fengguang; Liang, Yao; Luna, Daniel; Young, Ryan; Liang, Xu; Pamidighantam, Sudhakar; Computer and Information Science, School of ScienceThe CyberWater project is to build an open-data open-model framework for easy and incremental integration of heterogeneous data sources and diverse scientific models across disciplines in the broad water domain. The CyberWater framework extends the open-data open-model framework called Meta-Scientific-Modeling (MSM) that provides a system-wide data and model integration platform. On top of MSM, the CyberWater framework provides a set of toolkits, and external system integration engines, to further facilitate users' scientific modeling and collaboration across disciplines. For example, the developed generic model agent toolkit enables users to integrate their computational models into CyberWater via graphical user interface configuration without coding, which further simplifies the data and model integration and model coupling. CyberWater adopts a graphical scientific workflow system, VisTrails, ensuring data provenance and reproducible computing. CyberWater supports novel access to high-performance computing resources on demand for users' computational expensive model tasks. We demonstrate merits of CyberWater by a use case of hydrologic modeling workflow.Item Open data and model integration through generic model agent toolkit in CyberWater framework(Elsevier, 2022-06) Chen, Ranran; Luna, Daniel; Cao, Yuan; Liang, Yao; Liang, Xu; Computer and Information Science, School of ScienceThe CyberWater project is created to develop an open data and open model integration framework for studying complex environmental and water problems, where diverse online data sources can be directly accessed by diverse models without any need of users’ extra effort on the tedious tasks of data preparation for their models. We present our design and development of a novel generic model agent toolkit in the context of CyberWater, which enables users to integrate their models into the CyberWater system without writing any new code, significantly simplifying the data and model integration task. CyberWater adopts a visual scientific workflow system, VisTrails, which also supports provenance and reproducible computing. Our approach and the developed generic model agent toolkit are demonstrated, via CyberWater framework, with automated and flexible workflows through integrating data and models using real-world use cases. Two popular hydrological models, VIC and DHSVM, are used for illustrations.