Introduction: Computer Vision in Kitchen for Computer Control to Save Your Computer From Getting Destroyed!!
Hi, My name is Piyush And I Just Turned 13 Years Old on 20th February 2024 and go to Gems Founders School.
Have you ever wanted to control your laptop in the kitchen for entertainment or watch tutorials on cooking without getting your dirty, oily food hands on your computer?
I got this idea when my parents wanted a way to control a device without touching it with their messy or dirty hands while cooking and making food, so to counter this, I made a program using Computer Vision to Track gestures and has full control over Mouse features including the scroll wheel function!
Current Features:
- Mouse Cursor control
- Mouse Left click
- Mouse Right click
- up-down Arrow Keys for Volume Control, Scrolling
- Type Shortcut
- Desktop Switcher
- You can add as many of your own Functions (eg. Screenshot, Customizable Hotkeys and open any app or image automatically using the Locate Function)
If you want to check out more features then go and explore this link for all the functions:
Click Here - I will be explaining soon on how to edit the code to add your own functions
*All The YouTube Videos Included Are Videos Made By Me*
Supplies
Items:
- Any Laptop That can Run Python and has a camera (If it does not have a camera, then you need an External Camera)
- Any External Camera for Space in the Kitchen (OPTIONAL)
Software and Libraries Used:
- PyCharm application ( App download for Mac - App download for Windows - App download for Lunix)
- Libraries: CVZone, CV2, PyAutoGUI
Step 1: Download Pycharm and Start
➜ Use the link above to Download PyCharm, Then make a new Project and make sure you are on the same Window
and have the Same Settings as me (Step 1 Picture). Then Press Create
➜ Make and Name Your Python File (Step 2 Picture).
➜ To download the Libraries, Go to Settings> Project Interpreter > "+" (Step 3 picture)
➜ Search Libraries Name > Install (Step 4 Picture)
➜ After downloading all three Libraries, Paste the code which is in the next step
Watch the Video for Step By Step Guide:
Step 2: Understand the Code and Add Your Own Gesture :) (Optional)
In this Video, You will understand All the Important things in the Simple Code and Learn How to add your OWN Features!
PyAutoGUI Library Functions:
https://pyautogui.readthedocs.io/en/latest/quickstart.html#general-functions
Go to the Description For TimeStamps to Save some Time if Needed
Code:
# importing libraries
import cv2
import pyautogui
from cvzone.HandTrackingModule import HandDetector
# defining variables and objects
cap = cv2.VideoCapture(0)
detector = HandDetector(detectionCon=0.3, minTrackCon=0.7)
pyautogui.FAILSAFE = True
currentposset = False
######################Settings
MouseTrackingOn = True
MouseClickingOn = True
UpDownKeysOn = True
SwitchPagesOn = True
PauseControllOn = True
CUSTOM1 = True
##############################
while True:
# read camera and flip image
success, frames = cap.read()
frames = cv2.flip(frames, 1)
hands, img = detector.findHands(frames, flipType=False)
if hands: # check if there is a hand detected
hand1 = hands[0]
lmList1 = hand1["lmList"]
# bbox1 = hand1["bbox"]
center1 = hand1["center"]
handType1 = hand1["type"]
if handType1 == 'Right': # check if its right hand (you can change)
fingers1 = detector.fingersUp(hand1) ## fingers1 stores a lift of the fingers up
print(fingers1)
if MouseTrackingOn: # if mouseTracking is True then execute the code for mouseTracking
if not currentposset:
currentTipXPos, currentTipYPos = lmList1[8][0:2]
currentposset = True
if fingers1[0] == 1 and fingers1[1] == 1 and fingers1[2] == 0 and fingers1[3] == 0 and fingers1[
4] == 0 and currentposset:
differX, differY = (lmList1[8][0] - currentTipXPos) / 5, (lmList1[8][1] - currentTipYPos) / 5
MousePos = pyautogui.position()
pyautogui.moveTo(MousePos[0] + differX, MousePos[1] + differY)
else:
currentposset = False
length, info, img = detector.findDistance(lmList1[8][0:2], lmList1[4][0:2], img, (255, 0, 0),
10) # gets length between 2 points (lmList1[8][0:2] and lmList1[4][0:2].)
if MouseClickingOn: # if mouseClicking is True then execute the code for mouseClicking
if length < 50 and fingers1[2] == 0 and fingers1[3] == 0 and fingers1[4] == 0:
pyautogui.click(interval=1, button='left')
elif length < 50 and fingers1[2] == 1 and fingers1[3] == 1 and fingers1[4] == 1:
pyautogui.click(interval=1, button='right')
if UpDownKeysOn: # if upDownKeys is True then execute the code for scrolling up and down
if fingers1[0] == 1 and fingers1[1] == 1 and fingers1[2] == 1 and fingers1[3] == 0 and fingers1[4] == 0:
pyautogui.press('up')
elif fingers1[0] == 0 and fingers1[1] == 1 and fingers1[2] == 1 and fingers1[3] == 0 and fingers1[4] == 0:
pyautogui.press('down')
if SwitchPagesOn: # if SwitchPages is True then execute the code for Switching Pages (Mac Function)
if fingers1[0] == 1 and fingers1[1] == 1 and fingers1[2] == 1 and fingers1[3] == 1 and fingers1[4] == 0:
cv2.waitKey(1000)
pyautogui.keyDown('ctrl')
pyautogui.press('left')
pyautogui.keyUp('ctrl')
cv2.waitKey(10) # delay
elif fingers1[0] == 0 and fingers1[1] == 1 and fingers1[2] == 1 and fingers1[3] == 1 and fingers1[4] == 0:
cv2.waitKey(1000)
pyautogui.keyDown('ctrl')
pyautogui.press('right')
pyautogui.keyUp('ctrl') #
cv2.waitKey(10) # delay
if PauseControllOn:
if fingers1[0] == 1 and fingers1[1] == 0 and fingers1[2] == 0 and fingers1[3] == 0 and fingers1[4] == 1:
pyautogui.press('k')
cv2.waitKey(10) # delay
if CUSTOM1:
if fingers1[0] == 1 and fingers1[1] == 0 and fingers1[2] == 0 and fingers1[3] == 1 and fingers1[4] == 1: # ADD YOUR OWN GESTURE
# pyautogui.
# Go to https://pyautogui.readthedocs.io/en/latest/quickstart.html#general-functions
# and add your function
cv2.waitKey(10) # delay
cv2.imshow("Camera", img) # display image
cv2.waitKey(1)
Step 3: Main Steps on How to Use Program
➜ Firstly, If you have and want to use an External Camera, Place the Camera in A Comfortable Place where You can Easily Show Your Hand to it And You Get More Space because You can Keep Your Computer Somewhere Else But is Completely Your Choice.
➜ Then, Press the Run Located in the Top right Corner of the program.
➜ Finally, use the Gestures Provided in the Next Step And Control Your Computer!
(If you have Coded More Gestures, Use them Too! )
If There Are Any Problems or Errors, Please feel free To Contact Me!!!
Step 4: Gestures!
Stationary Position:
Stationary Position Is When You Keep All Your Fingers Up to not do any Actions
(Like the State when you are not touching your Mouse and The Computer Does not carry out any Actions)
----------------------------------------------------------------------------------------------------------------------------------
Gesture 1 - Mouse Cursor Control:
➜ For Controlling the Mouse Cursor, you first will show the Stationary Position Gesture and move to the position where you want to start the drag. Then you will show the Gesture For Cursor Control and move your Hand Which will start to
move the Mouse. When you want to stop, Show the Gesture of the Stationary Position to Freely Move. I made it like
this so it will replicate the process of A Touchpad.
The Gesture for Cursor Control is Keeping All the Fingers Down except The Index Finger.
----------------------------------------------------------------------------------------------------------------------------------
Gesture 2 - Mouse Clicking:
➜ For Controlling the Left and Right Click, you first will show the Stationary Position Gesture. Then, you will show the Gesture For Either The Left Click or The Right Click and it will Perform The Action. Make Sure your Cursor is in the Position you want it to Click!
The Gesture for Left Clicking is Keeping All the Fingers Down except The Thumb and The Index Finger. Then touch the Tip of your Thumb and Index Finger Together.
The Gesture for Right Clicking is Keeping all your fingers up and touching the tip of your Thumb and Index Finger.
----------------------------------------------------------------------------------------------------------------------------------
Gesture 3 - Up Down Keys:
➜ For Controlling the Up and Down Keys, you first will show the Stationary Position Gesture. Then, you will show the Gesture For Either The Up or The Down Key and it Start scrolling whether it is a Long Page or Volume Control. Make Sure you Click on the Window to Select and Take Action With.
The Gesture for the Up Key is Keeping the Thumb, Ring, and Pinky Finger down and the Index and Middle Finger up
The Gesture for the Down Key is Keeping only the Ring and Pinky Finger down and The First Three Fingers Up
----------------------------------------------------------------------------------------------------------------------------------
Gesture 4 - Switching Desktop Page (Apple Computer Feature):
➜ For Controlling the Desktop Switching, you first will show the Stationary Position Gesture. Then, show the Gesture For Either Switching the Desktop to the Left or Right
The Gesture for Switching the Desktop Space to the Left is Keeping the Thumb And Pinky Finger down and the Index, Middle Finger, And Ring Finger up
The Gesture for Switching the Desktop Space to the Right is Keeping Only the Pinky Finger down and the Thumb, Index, Middle Finger, And Ring Finger up
----------------------------------------------------------------------------------------------------------------------------------
Gesture 5- Switching Desktop Page (Apple Computer Feature):
➜ For Controlling the Desktop Switching, you first will show the Stationary Position Gesture. Then, show the Gesture For Either Switching the Desktop to the Left or Right
The Gesture for Switching the Desktop Space to the Left is Keeping the Thumb And Pinky Finger down and the Index, Middle Finger, And Ring Finger up
The Gesture for Switching the Desktop Space to the Right is Keeping Only the Pinky Finger down and the Thumb, Index, Middle Finger, And Ring Finger up
----------------------------------------------------------------------------------------------------------------------------------
Gesture 6- Pause Control (You can Edit To change The Key Pressed, By Default it's 'k'):
➜ For Controlling the Desktop Switching, you first will show the Stationary Position Gesture. Then, show the Gesture For Pausing and if the video is Paused, It Will Play and if it's Playing, then it will Pause The Video. Make Sure you Click on the Window to Select and Take Action With.
The Gesture for Pause/Play is Keeping the Thumb And Pinky Finger up and the Rest Down
Step 5: Demonstration
I Tried To Recreate a Situation When In the Kitchen With Wet or Dirty Hands
Step 6: Final Thoughts and Conclusion
I Think This Project Came Out Amazing and I Am Sure That This Will Be The Future Of Controlling Computers With Computer Vision and Will Help People In General by Doing Some Physical and Intuitive Activities when using devices.
Another use i found for this is when eating the food; when eating most foods, your hands get dirty, but you really want to watch youtube, Guess what! You have the perfect device
To Support me, Go To my Amazing YouTube Channel from the Videos Above and Give A Fist Bump To That Red Button!
Thank You and Hope You Enjoyed Following The Project With Me!
This is an entry in the
Engineering in the Kitchen - Autodesk Design & Make - Student Contest
![Tim's Mechanical Spider Leg [LU9685-20CU]](https://content.instructables.com/FFB/5R4I/LVKZ6G6R/FFB5R4ILVKZ6G6R.png?auto=webp&crop=1.2%3A1&frame=1&width=306)
