Members of group:

Frederico Godwyn (2502029433)
William Deli (2540119702)
Jason (2501971370)

LA40 Class

Background Problem

In last few years, supermarket industries have been facing significant challenges because of online shopping that has been increasing lately. One of the impact of those problem is a decrease in the number of visitors to physical supermarkets. Consumer prefer to choose convenient shopping from home than visit supermarket. This kind of problem made supermarket to find innovative way to for taking back consumer. Other than that, waiting process that taking so much time to pay their items has making the consumer to have a bad experience in supermarket.

In other side, supermarket facing challenges in operational and efficiency. They have to put so much money to pay the cashier every month, taking so much operating costs in the middle of competition with online shop. Other than that, payment system that is being used by most of supermarket is not efficient. Payment process that really slow and lack of payment option that flexible become a problem for consumer that hoping fast and practical payment. In this problem we have been developing a smart cart that going to solve all of this problem, called MICHELIE. MICHELIE has a purpose to speed up the shopping process, reduce waiting process, and optimize operating costs. By using MICHELIE the shopping experience by consumer will be increased by significant. Because it is using computer vision, object detection and tracking, weight sensor and mobile app to make consumer experience increasing.

Problem Formulation

What are the advantages of MICHELIE?
What are the disadvantages of MICHELIE?

Purpose

To create a tool aimed at shortening the process of scanning items.
To create a tool aimed at streamlining the payment process.

Benefits

Facilitates the payment process through mobile phone payments.
Enhances Operational Efficiency, as this technology allows business owners (supermarkets) to operate without the need for cashiers.

What does MICHELIE Stands for:

Multifunctional: Michelie is more than just a regular shopping cart. It can do several things like help you find items, keep track of what you're buying, and even help you check out faster.
Intelligent Cart: Michelie is smart. It uses technology to learn what you like to buy and can make shopping suggestions. It's like having a personal shopping assistant.
High-end Electronics: Michelie is built with advanced technology. This includes things like sensors to know what’s in the cart, a screen to interact with, and connections to work with your smartphone.
Logistics Integration: Michelie is designed to work well with the store's system. It helps keep track of what's being bought and can guide you to where items are in the store.
Efficiency: Michelie makes shopping quicker and easier for you, and it also helps the store manage things better, like keeping track of stock.

Supplies

Jetson nano
ESP32
4x Loadcell
4xHX711
Shopping cart
Webcam
Jumper wire
Terminal Block
MDF Board
Loadcell mounting made by 3D print

Step 1: Feature

Object Detection: By utilizing object detection technology, MICHELIE can identify each item and distinguish it from others. This feature ensures that every product in the cart is accurately recognized and cataloged.
Object Tracking: With object tracking technology, MICHELIE is able to determine whether an item has been picked up or not. This capability is crucial for real-time monitoring of the items being added to or removed from the cart.
Weight Measurement: By employing an HX711 sensor and a load cell, MICHELIE can verify whether an item has been placed in the cart or not. This weight measurement is an essential part of confirming the presence of items, complementing the data obtained from object detection and tracking.
Mobile App: The mobile app allows users to view the items in their MICHELIE cart and facilitates the payment for these items using the app. This adds a layer of convenience for the user, enabling them to manage their shopping and checkout process seamlessly from their mobile device.

Step 2: Block Diagram

Step 3: Flowchart

Step 4: How It Works

Collect and Train Data: Gather and train data for object detection using the SSD-Mobilenet-V1 model. Focus on several products: Mie Sedap, Indomie Goreng, Smax Ring, and Pringles, with a total of 1500 images.
Insert Data into Code: Place the collected data in the format "Yourmodel".onnx into the provided code. Ensure to update the model directory in the code.
Use Jetson Nano: Run the program on a Jetson Nano, which uses CUDA cores for enhanced object detection and tracking performance compared to a standard PC.
Connect ESP32 to Jetson: Establish a serial connection between the ESP32 and Jetson Nano using the RX and TX pins for sensor communication.
Run Load Cell Code in ESP32 and AI Jetson Code: Execute the provided upload code for the load cell and the Python code for the Jetson Nano.
ESP32 and Jetson Integration: Once the code is running, the ESP32 and Jetson will be connected, ready to detect and verify items using the load cell.
Run Mobile App Code: Launch the code for the mobile app to enable user interaction with MICHELIE.
Item Interaction: Place items in the cart. Detected items will appear in the mobile app and can be paid for directly through the app.

Step 5: Demo

https://youtu.be/9nw5ffZmN4E

Step 6: Mobile App

Our innovative mobile application serves as a crucial companion to the Smart Cart, offering a seamless and interactive shopping experience. This state-of-the-art app is ingeniously designed to display real-time data of items added to the Smart Cart, enhancing the shopping process by keeping the user informed of their selections and total expenses as they shop.

One of the key functionalities of this application is its integrated payment system. Shoppers can conveniently use the app to execute transactions for the products in their cart. This feature not only expedites the checkout process but also reduces the need for physical interactions at traditional payment counters, aligning with modern contactless payment trends.

In addition to its primary shopping functions, the app is tailored to provide personalized promotional offers. By analyzing the shopping habits and preferences of users, it smartly suggests relevant deals and discounts on products, thereby elevating the shopping experience and offering value to our customers. This targeted approach in marketing ensures that the offers are not just general advertisements but valuable, tailor-made propositions that resonate with the individual preferences of each shopper.

Step 7: Analysis

The device we have engineered offers notable efficiency and convenience in retail shopping. It streamlines the shopping experience by allowing users to bypass traditional checkout lines. With our innovative solution, customers can simply add products to their shopping cart, and the device automatically scans and registers each item. This process not only accelerates the shopping journey but also enhances the overall user experience. Furthermore, our custom-developed mobile application adds another layer of convenience, enabling users to effortlessly pay for their purchases directly from their smartphones. The app also provides real-time monitoring of the items in the cart and offers a feature for users to top up their account balance, ensuring a seamless shopping experience.

Despite its many advantages, our device does encounter a significant challenge: the computational limitations of the Jetson Nano. This constraint affects the speed and efficiency of the product scanning process. The Jetson Nano, while powerful for its size, struggles to keep up with the demands of real-time object detection and data processing required for instantaneous product recognition. This results in a slower scanning process, which can potentially impact the user experience. We are actively exploring solutions to mitigate this limitation, including potential hardware upgrades or software optimizations, to ensure our device meets both the speed and accuracy needs of our users.

Step 8: Conclusion & Recommendations

Conclusion

From the product we have developed, it is entirely feasible to create a smart cart using the Jetson Nano. To construct such a smart cart, it is essential to incorporate an HX711 sensor and a load cell to determine the weight of items. Logic is then applied to ascertain whether an item is being added to or removed from the cart, a process that is crucial for confirming the results of object tracking. The Jetson Nano serves as the computing power for all components connected, including the webcam. The ESP32 acts as a bridge between the sensors and the Jetson Nano. Cloud Firestore is utilized as a means of communication between the Jetson Nano and the mobile application.

Recommendations

It is advisable to use logic that combines object detection with the load cell data to expedite the computation process. Additionally, using acrylic as the base material for the cart is recommended, as acrylic tends to be lighter than MDF (Medium-Density Fibreboard), which can enhance the overall efficiency and maneuverability of the smart cart.