With the rapid growth of online education, maintaining academic integrity in virtual exams has become a major challenge for educational institutions. This research, which was simulated at the Isfahan University of Technology in a laboratory designed for online in-person exams, introduces an innovative system that utilizes a combination of the YOLO object detection algorithm and multi-agent systems (MAS) to detect and prevent cheating. In this system, the YOLO algorithm uses video analysis to detect unauthorized objects such as mobile phones and tablets, while the MAS analyzes students' abnormal behaviors, such as suspicious mouse clicks, unusual delays in responses, abnormal eye movement patterns, and the opening of unauthorized browser tabs, to identify potential cheating patterns. The system was simulated in test exams for popular general courses at Isfahan University of Technology, including Islamic Studies, Persian, and Ethics, and the results showed that it can detect cheating with 87.9% accuracy. Furthermore, with a processing speed of 0.1 to 0.15 seconds per frame, the system is capable of real-time execution. The findings of this study emphasize that combining deep learning algorithms and agent-based systems can provide an effective and scalable solution to enhance the security of online in-person exams, ultimately contributing to greater trust in online assessment processes and ensuring educational fairness.