A self-adaptive system is capable of maintaining its quality requirements in the presence of dynamic environment changes.To develop a self-adaptive system,, engineers have to create self-adaptation logic that encodes when and how the system should adapt itself. However, developing self-adaptation logic may be difficult due to design time uncertainty; e.g., anticipating all potential environment changes at design time is in most cases infeasible. Online reinforcement learning addresses design time uncertainty by learning the effectiveness of adaptation actions through interactions with the system’s environment at run time, thereby automating the development of self-adaptation logic. Existing online reinforcement learning approaches for self-adaptive systems exhibit two shortcomings that limit the degree of automation: they require manually fine-tuning the exploration rate and may require manually quantizing environment states to foster scalability.
We introduce an approach to automate the aforementioned manual activities by employing policy-based reinforcement learning as a fundamentally different type of reinforcement learning. We demonstrate the feasibility and applicability of our approach using self-adaptive system exemplar.We introduced and experimentally evaluated an online reinforcement learning approach to facilitate engineering of self-adaptive information systems. Our approach contributes to system engineering by increasing the degree of automation. our approach does neither require manually quantizing environment states nor manually having to determine suitable exploration parameters for the reinforcement learning algorithm to work.