- Information
- AI Chat
Ijrpr 7319
python programming (cs8151)
Anna University
Recommended for you
Preview text
International Journal of Research Publication and Reviews
Journal homepage: ijrpr ISSN 2582- 7421
AI Virtual Mouse Using Hand Gesture Recognition.
Abhishek R. Shukla
Department of Information Technology, B. Birla College of Arts, Science & Commerce (Autonomous), Kalyan, 421301, Maharashtra, India. abhishekshukla@gmail DOI: doi/10.55248/gengpi.2022.3.10.
ABSTRACT:
The mouse is one of HCI's (Human Computer Interaction) incredible inventions. A wireless mouse or a Bluetooth mouse still uses devices because a battery is needed for power and a dongle is needed to connect the mouse to the PC, they are not totally device-free. The proposed AI virtual mouse system can solve this problem by using a webcam or built-in camera to record hand motions and identify hand tips using computer vision. The system's algorithm makes use of the machine learning algorithm. The computer can be controlled digitally and cando left-click, right-click, scrolling, and computer cursor functions based on hand motions without the need for a physical mouse. Deep learning is the basis for the hand detection method. As a result, by eliminating human contact and the requirement for external devices to operate the computer, the suggested strategy will stop the spread of COVID-19. Keywords: virtual mouse, AI, MediaPipe, OpenCV.
1
Because of advancements in Bluetooth and wireless technology, augmented reality, and other related sectors, the devices we use every day are becoming smaller. In this study, a computer vision-based virtual mouse system that employs hand gestures and hand tip detection to replicate mouse motions was developed. The main objective of the proposed system is to replace the traditional mouse with a web camera or a built-in camera in the computer to perform computer mouse cursor and scroll tasks AI virtual mouse technology allows us to conduct mouse cursor actions like scrolling and moving the pointer while also monitoring the fingertip of a hand gesture using a built-in camera or web camera. In this study, the user uses a built-in camera or webcam together with hand gestures to operate a computer mouse, as opposed to using a wireless or Bluetooth mouse, which needs a particular peripheral such as a mouse, a dongle to connect to the PC, and a battery to operate. The suggested system uses a web camera to record, examine, and decipher the captured frames, identify various hand and hand-tip gestures, and then carry out the relevant mouse action. The Python programming language and the OpenCV package were used to construct the AI virtual mouse system. The suggested AI virtual mouse system leverages the Media Pipe package for hand tracking and hand tricks. Additionally, the desktop window may be moved around and operations like scrolling and left-and right-clicking are carried out using the Pynput, Autopy, and PyAutoGUI packages. The findings of the proposed model demonstrate a very high degree of accuracy, and it can operate in real-world applications using CPU utilization rather than GPU.
1 Problem Description and Overview:
In the worldwhere there isn't enough room to use a physical mouse or for those who have hand difficulties and can't use a mouse, the proposed AI virtual mouse system frequently wants to circumvent challenges. The COVID-19 condition also makes it risky to utilised gadgets by touching them because doing so might result in an instance in which the virus is on them. Because hand gesture and hand unfolded detection are utilised to control the laptop mouse operations by utilising a digital camera or tip intrinsic camera, the suggested AI virtual mouse may be used to resolve these concerns.
1 Objective:
I. The Hand Following Algorithm was used: - To recognise hand gestures and track hands on a laptop, the MediaPipe framework and therefore the OpenCV package are both employed [7–10]. The application uses machine learning concepts to partially track and detect hand gestures and hand tips. II. The first goal of the planned AI virtual mouse system is to form a replacement for the traditional mouse system that can perform and manage mouse functions. This will be finished the help of an internet camera that records hand gestures and hand tips so processes these frames to hold out the mouse functions, like a left click, right click, or scrolling function.
1 Media Pipe:
A framework known as Media Pipe, that is an ASCII text file framework from Google, is employed to use during a machine learning pipeline. Provided that it is created utilizing statistical data, the Media Pipe framework is useful for cross-platform development. As a multimodal framework, the Media Pipe are often used with a range of audio and video files. The Media Pipe framework is used by the developer to make and analyse systems mistreatments and, similarly, to create systems for application-related purposes. The pipeline configuration is used to hold out the processes in the media pipe-using system. Measurability on desktops and mobile devices is enabled by the pipeline's flexibility to execute on many platforms. The 3 core elements that frame the Media Pipe design are performance evaluation, a system for getting detector data, and a gaggle of reusable items referred to as calculators. A pipeline could be a design made from units referred to as calculators that are connected to at least one another by streams via which knowledge packets pass. Developers will add, remove, or redefine custom calculators anyplace within an application created with Media Pipe, where every node is a calculator, and therefore the nodes are connected by streams, and the streams along generate a data-flow diagram. Detective work and characterization of a hand or palm in real time is finished by employing a single-shot detector model. The Media Pipe uses the single-shot detector model. Since coaching palms is easier than training hands, the hand detection module first trains a model for palm detection. Additionally, for tiny things just like your palms or fists, the non-maximum suppression performs perceptibly better. Placement of joint or knuckle coordinates within the hand region constitutes a model of hand landmark, as shown in Figure 1.
Figure 1: Hand landmarks point used by Media
1 OpenCV: The computer vision library OpenCV has object identification and image processing algorithms in it. The Python programming language's OpenCV computer vision package may be used to build real-time computer vision applications. The processing of photos and videos, as well as analytical activities like face and object identification, all make use of the OpenCV package.
Figure 2: Flowchart of real time AI virtual mouse system
Figure 5: Identifying which finger is up.
Figure 6: Shows the mouse pointer gliding over the computer screen.
Figure 7: Shows a motion that guiding a computer to click left.
Figure 8: Shows a motion that instructs a computer to click left.
Figure 9: A motion that instructs the computer to scroll up.
Figure 10: A gesture that causes the computer to scroll down.
Figure 11: A gesture that instructs the computer to do nothing.
3
There are several applications for the AI virtual mouse technology. When utilizing the physical mouse, and in cases when using the physical mouse is not an option, it can be utilized to conserve space. The method improves human-computer interaction while reducing the need for devices.
Principal applications: 1) The suggested model has a considerably greater accuracy of 99% than other virtual mouse models that have been developed, and it has several uses. 2) It is not safe to use the devices since it can lead to a scenario in which the COVID-19 virus spreads by contacting them. Without utilizing an actual mouse, the suggested AI virtual mouse may be utilized to control PC mouse activities. 3) The system is capable of controlling robots and automation systems without the use of additional hardware. 4) 2D and 3D visuals may be created by hand gestures and an AI virtual system. 5) By utilizing an AI virtual mouse, gamers may play games based on virtual reality and augmented reality without the requirement of a wireless or wired mouse. 6) People who have hand problems can utilize these controls to operate the computer mouse. 7) Like HCI, the suggested technology may be used to control robots in the field of robotics. 8) The proposed method may be used for virtual prototyping and development in the disciplines of architecture and design.
4
The artificial intelligence (AI) virtual mouse system's main objective is to eliminate the need for a hardware mouse by replacing it with hand gestures that control mouse cursor functions. The suggested system may be implemented using a webcam or an integrated camera that detects hand gestures and the tip of the hand and analyses these frames to execute certain mouse commands. The results of the model reveal that the proposed AI virtual mouse system has performed extremely well, has greater accuracy than the current models, and also overcomes most of the shortcomings of the latter. Since the recommended model is more realistic, the AI virtual mouse may be used for practical purposes. Additionally, because the proposed mouse system can be manipulated virtually using hand gestures rather than the traditional physical mouse, it can be used to halt the spread of COVID-19. The model includes a number of flaws, including some difficulties with clicking and dragging to select text and a little loss in right-click mouse function accuracy. As a result, we will work to improve the fingertip detection algorithm in order to produce findings that are more accurate.
References [1] Guha, Joy, et al. “AI Virtual Mouse Using Hand Gesture Recognition.” International Journal for Research in Applied Science and Engineering Technology, vol. 10, no. 4, International Journal for Research in Applied Science and Engineering Technology (IJRASET), Apr. 2022, pp. 3070–76. Crossref, doi/10.22214/ijraset.2022.41981.
[2] A, Mr. Venkateshwar. “Hand Gesture Recognition System as Virtual Mouse for HCI.” International Journal for Research in Applied Science and Engineering Technology, vol. 9, no. 8, International Journal for Research in Applied Science and Engineering Technology (IJRASET), Aug. 2021, pp. 1098 – 101. Crossref, doi/10.22214/ijraset.2021.37503.
[3] “VIRTUAL MOUSE USING HAND GESTURE.” International Journal of Modern Trends in Engineering & Research, vol. 4, no. 3, International Journal of Modern Trends in Engineering and Research, Mar. 2017, pp–9, doi/10.21884/ijmter.2017.4069.3wz5x.
[4] Shukla, Akhilesh, et al. “Gesture Recognition-based AI Virtual Mouse.” International Journal for Research in Applied Science and Engineering Technology, vol. 10, no. 3, International Journal for Research in Applied Science and Engineering Technology (IJRASET), Mar. 2022, pp. 1583–88. Crossref, doi/10.22214/ijraset.2022.40937.
[5] Waichal, Aabha. “Survey Paper on Hand Gesture Recognition Based Virtual Mouse.” INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT, vol. 06, no. 02, Indospace Publications, Feb. 2022. Crossref, doi/10.55041/ijsrem11641.
Ijrpr 7319
Course: python programming (cs8151)
University: Anna University
