Q-learning-based Optimal Control Scheme for Time-varying Uncertain Batch Processes

Industrial batch processes are extensively employed in the modern manufac-turing industry due to their efficiency and flexibility. However, controlling and optimizing batch processes is difficult because of their complex non-stationary dynamics and inherent time-varying uncertainties. In order to ad-dress these issues, this paper proposes a novel Q-learning-based optimal con-trol scheme for time-varying batch processes to achieve optimal control while reducing reliance on process modeling. First, based on the time-varying nominal model, an initial control policy is derived from dynamic programming and the principle of optimality. Nevertheless, the presence of unknown time-varying system uncertainties hinders the optimal perfor-mance of the initial control policy. To overcome this limitation, we utilize the repetitive nature of the batch process to collect operational data from multiple batches runs under the initial control policy. Then, the Q-learning-based optimal control scheme is developed to iteratively improve the initial control policy under the reinforcement learning framework. The convergence analysis demonstrates that the improved control policy gradually converges to the optimal control policy. Finally, the simulation results from the nu-merical multi-input multi-output batch system and the injection molding process demonstrate the proposed control method's effectiveness, applicabil-ity, and superior control performance.