Selectively Decentralized Q-Learning

Abstract

In this paper, we explore the capability of selectively decentralized Q-learning approach in learning how to optimally stabilize control systems, as compared to the centralized approach. We focus on problems in which the systems are completely unknown except the possible domain knowledge that allow us to decentralize into subsystems. In selective decentralization, we explore all of the possible communication policies among subsystems and use the cumulative gained Q-value as the metric to decide which decentralization scheme should be used for controlling. The results show that the selectively decentralized approach not only stabilizes the system faster but also shows superior converging speed on gained Q-value in different systems with different interconnection strength. In addition, the selectively decentralized converging time does not seem to grow exponentially with the system dimensionality. Practically, this fact implies that the selectively decentralized Q-learning could be used as an alternative approach in large-scale unknown control system, where in theory, the Hamilton-Jacobi-Bellman-equation approach is difficult to derive the close-form solution.