Intelligent Knowledge Exploration and Processing

Intelligent Knowledge Exploration and Processing

Self adaptive logic development in self adaptive systems using online deep reinforcement learning

Document Type : Original Article

Author
kazem nikfarjam
ِDepartment of Engineering Faculty of Computer Eng , University ,Birjand Branch, ,Islamic Azad University,,Birjand,, Iran
10.30508/kdip.2023.383007.1060
Abstract
A self-adaptive system can modify its own structure and behavior at runtime based on its perception of the environment, of itself and of its requirements. One key element of a self-adaptive system is its self-adaptation logic that encodes when and how the system should adapt itself. When developing the adaptation logic, developers face the challenge of design time uncertainty. To define when the system should adapt, they have to anticipate all potential environment states. However, anticipating all potential environment changes is infeasible in most cases due to incomplete information at design time.Online reinforcement learning (RL) addresses design time uncertainty by learning the effectiveness of adaptation actions through interactions with the system’s environment at run time, thereby automating the development of self-adaptation logic. Online-RL for self-adaptive systems integrates the elements of RL into the MAPE-K loop
Existing online RL approaches for self-adaptive systems represent learned knowledge as a value function, so exhibit two shortcomings that limit the degree of automation: they require manually fine-tuning the exploration rate and may require manually quantizing environment states to foster scalability. In this paper, use policy-based deep reinforcement learning, which are structurally quite different, to automate the aforementioned manual activities. Deep RL addresses these disadvantages by representing the learned knowledge as a neural network. learned knowledge is hidden in the neural network. The results of the experiments indicate a high convergence speed of learning.
Keywords
Deep reinforcement learning
Uncertainty
: Selfadaptation logic
: Selfadaptive system
1.      Amoui, M., Salehie, M., Mirarab, S., & Tahvildari, L. (2008, March). Adaptive action selection in autonomic software using reinforcement learning. In Fourth International Conference on Autonomic and Autonomous Systems (ICAS'08) (pp. 175-181). IEEE.
2.      Arabnejad, H., Pahl, C., Jamshidi, P., & Estrada, G. (2017, May). A comparison of reinforcement learning techniques for fuzzy cloud auto-scaling. In 2017 17th IEEE/ACM international symposium on cluster, cloud and grid computing (CCGRID) (pp. 64-73). IEEE.
3.      Aschoff, R., & Zisman, A. (2011). QoS-driven proactive adaptation of service composition. In Service-Oriented Computing: 9th International Conference, ICSOC 2011, Paphos, Cyprus, December 5-8, 2011 Proceedings 9 (pp. 421-435). Springer Berlin Heidelberg.
4.      Barrett, E., Howley, E., & Duggan, J. (2013). Applying reinforcement learning towards automating resource allocation and application scalability in the cloud. Concurrency and computation: practice and experience, 25(12), 1656-1674.
5.      Bu, X., Rao, J., & Xu, C. Z. (2012). Coordinated self-configuration of virtual machines and appliances using a model-free learning approach. IEEE transactions on parallel and distributed systems, 24(4), 681-690.
6.      Caporuscio, M., D’Angelo, M., Grassi, V., & Mirandola, R. (2016). Reinforcement learning techniques for decentralized self-adaptive service assembly. In Service-Oriented and Cloud Computing: 5th IFIP WG 2.14 European Conference, ESOCC 2016, Vienna, Austria, September 5-7, 2016, Proceedings 5 (pp. 53-68). Springer International Publishing.
7.      Chen, T., & Bahsoon, R. (2016). Self-adaptive and online qos modeling for cloud-based software services. IEEE Transactions on Software Engineering, 43(5), 453-475.
8.      D'Ippolito, N., Braberman, V., Kramer, J., Magee, J., Sykes, D., & Uchitel, S. (2014, May). Hope for the best, prepare for the worst: multi-tier control for adaptive systems. In Proceedings of the 36th International Conference on Software Engineering (pp. 688-699).
9.      Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., ... & Coppin, B. (2015). Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679.
10.  Dutreilh, X., Kirgizov, S., Melekhova, O., Malenfant, J., Rivierre, N., & Truck, I. (2011, May). Using reinforcement learning for autonomic resource allocation in clouds: towards a fully automated workflow. In ICAS 2011, The Seventh International Conference on Autonomic and Autonomous Systems (pp. 67-74).
11.  Filho, R. R., & Porter, B. (2017). Defining emergent software using continuous self-assembly, perception, and learning. ACM Transactions on Autonomous and Adaptive Systems (TAAS), 12(3), 1-25.
12.  Jamshidi, P., Cámara, J., Schmerl, B., Käestner, C., & Garlan, D. (2019, May). Machine learning meets quantitative planning: Enabling self-adaptation in autonomous robots. In 2019 IEEE/ACM 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) (pp. 39-50). IEEE.
13.  Gheibi, O., Weyns, D., & Quin, F. (2021). Applying machine learning in self-adaptive systems: A systematic literature review. ACM Transactions on Autonomous and Adaptive Systems (TAAS), 15(3), 1-37.
14.  Kephart, J. O., & Chess, D. M. (2003). The vision of autonomic computing. Computer, 36(1), 41-50.
15.  Klein, C., Maggio, M., Årzén, K. E., & Hernández-Rodriguez, F. (2014, May). Brownout: Building more robust cloud applications. In Proceedings of the 36th International Conference on Software Engineering (pp. 700-711).
16.  De Lemos, R., Giese, H., Müller, H. A., Shaw, M., Andersson, J., Litoiu, M., ... & Wuttke, J. (2013). Software engineering for self-adaptive systems: A second research roadmap. In Software Engineering for Self-Adaptive Systems II: International Seminar, Dagstuhl Castle, Germany, October 24-29, 2010 Revised Selected and Invited Papers (pp. 1-32). Springer Berlin Heidelberg.
17.  Lorido-Botran, T., Miguel-Alonso, J., & Lozano, J. A. (2014). A review of auto-scaling techniques for elastic applications in cloud environments. Journal of grid computing, 12, 559-592.
18.  Mann, Z. Á. (2017). Resource optimization across the cloud stack. IEEE Transactions on Parallel and Distributed Systems, 29(1), 169-182.
19.  Moustafa, A., & Zhang, M. (2014, June). Learning efficient compositions for QoS-aware service provisioning. In 2014 IEEE International Conference on Web Services (pp. 185-192). IEEE.
20.  Nachum, O., Norouzi, M., Xu, K., & Schuurmans, D. (2017). Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems, 30.
21.  Ramirez, A. J., Jensen, A. C., & Cheng, B. H. (2012, June). A taxonomy of uncertainty for dynamically adaptive systems. In 2012 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) (pp. 99-108). IEEE.
22.  Salehie, M., & Tahvildari, L. (2009). Self-adaptive software: Landscape and research challenges. ACM transactions on autonomous and adaptive systems (TAAS), 4(2), 1-42.
23.  Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
24.  Silvander, J. (2019). Business process optimization with reinforcement learning. In Business Modeling and Software Design: 9th International Symposium, BMSD 2019, Lisbon, Portugal, July 1–3, 2019, Proceedings 9 (pp. 203-212). Springer International Publishing.
25.  Sutton, R. S., McAllester, D., Singh, S., & Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12.
26.  Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
27.  Tesauro, G., Jong, N. K., Das, R., & Bennani, M. N. (2007). On the use of hybrid reinforcement learning for autonomic resource allocation. Cluster Computing, 10, 287-299.
28.  Iglesia, D. G. D. L., & Weyns, D. (2015). MAPE-K formal templates to rigorously design behaviors for self-adaptive systems. ACM Transactions on Autonomous and Adaptive Systems (TAAS), 10(3), 1-31.
Intelligent Knowledge Exploration and Processing
Volume 3, Issue 8
Spring 2023
Pages 82-97