PInfer: Learning to Infer Concurrent Request Paths from System Kernel Events

Abstract

Operating system kernel-level tracers are popularly used in the post-development stage by black-box approaches. By inferring service request processing paths from kernel events, these approaches enabled system diagnosis and performance management that are application-logic aware. However, asynchronous communications and multi-threading behaviors make request path patterns dynamic on the kernel event level, this causes previous methods to focus on either software instrumentation techniques or better statistical inference models. In this paper, we propose a novel learning based approach called PInfer that infers request processing path patterns automatically with high precision. PInfer first learns dynamic event patterns of inter-thread and intra-thread service processing from the training data of sequential requests. On the testing data containing concurrent requests, PInfer infers individual request processing paths by effectively solving a graph matching problem and a generalized assignment problem based on the learned patterns. We have implemented our approach in a proprietary system performance diagnosis tool, and present performance results on 40 sets of kernel event traces. PInfer achieves on average 65% precision and 85% recall for profiling concurrent request processing paths.