Project for UCLA ECEM202A/CSM213A (Fall 2021)
Arm command aims to streamline and simplify the user interaction with smart devices in one’s home. By leveraging a Lidar, RGB camera, and an intel NUC, we allow the user to simply point to a smart device and toggle it on/off by raising their thumb. We utilized MediaPipe’s hand detection model to identify the user’s hand, and then created our own algorithm to create a vector of the index finger to then determine the object being pointed to. By training a YOLO object detector model, we are currently able to indentify a lamp and a computer monitor, allowing the user to control both by simply making a thumbs up. The object’s hitbox were created based on the location in the image and then extruded to become a 3d object. We then project tyhe finger’s vector trajectory and check for any collion with the hitboxes. Our first success metric is related to response time: time elapsed from the moment the user controlled a device until when the action happened. In the current setting, we measured an average response time of less than 1 second. Another success metric is related to detection accuracy: how close can objects be and still be accurately controllable. In our testing, as long as the objects are not overlapping, the system will work as expected.
As home automation becomes more popular, users may seek a more intuitive way of interacting with smart home devices such as lights and other electronics. However, the current approach used by the majority of the main automation services require the user to name individual devices or scenes to then activate/deactivate with voice commands. As the number of devices in one’s house increases, this process can become somewhat of a burden. But what if we could control devices simply by pointing at them?
By using a small computer, a camera, and a Lidar, we are able to first detect a person’s fingers and determine which object the user is pointing to. Then, with a simple command (raising the thumb) the smart home device will toggle on/off.
When the system boot up, it will first take an image and scan all the smart devices, then building connections with them through wifi based on the configuration files and storing their coordinates. When a person shows up, the system will track his left hands. Calculate the index finger’s vector and do pointing check with devices. When the system detects a device is being pointed, it will track the user’s thumb skeletons to detect a thumbs up. After detecting that gesture, the system will send toggle command to device.
We use the Mediapipe hand model to detect hand skeletons in realtime. And we calculate the index finger vector based on its finger base and finger tip.
We trained a YOLO v5s model to perform the object detection of both the lamp and computer monitor. We then create a boundary cube around the objects to serve as a hitbox.
We then calculate the intercept between the finger vector and the objects’ hitbox to determine which object is being pointed to.
By using the fingers landmarks, we determine if the user does a thumbs up to then toggle on/off the pointed object. Communicating to the Kasa smart plugs through wifi.
When evaluating the system, we first wanted to make sure that it was functioning as expected. As such, the main metric for user interaction/experience was the time taken from the moment the user interact with the system to when the action is triggered. In this case, this means the delay from the moment the user points and raises a thumb, to the moment the smart device toggles on/off. We ran an experiment and measured the time taken over 10 distinct interactions. After collecting the data, we calculated an average delay of less than 1 second, meaning it was unnoticeable. When comparing it to a system such as Alexa, our system presented a shorted delay due to the fact that it does not need to listen to the command and interpret it using a NLP algorithm.
The second metric we measured was the accuracy of our object detection model and the generated “hitbox” for the objects. In this case, we wanted to see how close can two objects be before the system does not behave as expected due to conflicting hitboxes. For our surprise, as long as the objects are not overlapping each other, it performs as expected. This is mainly due to the fact that we are only using one camera, thus limiting our point of view to a stationary observer.
The Arm command project successfully developed a gesture-based control system for smart home devices using Lidar and camera only, bypassing the need for wearables and voice commands. The system utilized MediaPipe for hand landmark detection and a YOLO model for object identification, achieving an average response time under one second, notably faster than voice-activated systems. Challenges included ensuring sensor accuracy and computational efficiency, as well as addressing potential privacy concerns.
Our project met its initial objectives, demonstrating a novel, efficient method for smart device interaction that holds promise for individuals with speech impairments. While the response time and accuracy fulfilled the project’s success metrics, future work should aim to refine the system’s ability to operate in more complex environments, lower the latency and enhance privacy safeguards.