Thermodynamic-Aware Multi-Agent Reinforcement Learning with Predictive Load Shifting for Real-Time Power Optimization and Climate Control in Phase-Change Cold Storage Systems

Abstract

Phase-change cold storage systems have received serious consideration as a solution to peak reduction of electricity demand through the storage and release of thermal energy. Nevertheless, the intricate thermodynamic characteristics, particularly in the phase transitions, have inhibited their incorporation with reinforcement learning (RL) methods. The presented paper proposes a Thermodynamic-Aware Multi-Agent Reinforcement Learning (TA-MARL) framework, where physics-aware thermodynamic constraints, including latent heat and phase fraction dynamics, are directly embedded in the RL model. The methodology involved a hierarchical multi- time scale agent architecture, which allows predictive load shifting and energy arbitrage with regard to dynamic electricity prices and cooling demand predictions. The proposed system is compared to the baseline control strategies on the basis of real-world building energy and PCM thermodynamic simulations. The findings indicate that TA-MARL is superior to the conventional rule-based control (RBC) and price-only RL in attaining 27-32% savings in electricity costs and 22-28% savings in peak demand. Also, TA-MARL does not exceed safe limits of the phase transitions in the PCM, like in other strategies, and PCM constraints are not violated throughout the strategy. TA-MARL hierarchical organization is 35%-fold faster in converging on the learning than single-agent RL models. The ability to shift the predictive load greatly diminishes the use of energy under peak hours, and it also makes sure that the variation of temperature indoors is not below -0.5 o C and above +0.5 o C. To sum up, thermodynamic physics applied to RL allows creating a safer, efficient, and economically feasible system of PCM-based cold storage. The paper has also identified the significance of thermodynamic awareness in the reinforcement learning of thermal energy storage, and prepares the future study of extending this model to a larger, multi-zone system.