XAND 2.0 is an upgrade to our previous project (XAND - Amigo Virtual) from Integration Workshop 2. Last semester, we created a virtual assistant combined with a virtual pet, like an Alexa for kids. This time, we want to give our virtual pet a physical body.

Architecture and physical structure

The system will be composed of 3 main pillars: the robot, the AI server and the control application.

• Structure: The base will be a 2WD Chassis kit, powered by DC motors controlled by a motor driver. The embedded “brain” will be a Raspberry Pi 3 that will manage the sensors and actuators: A camera for computer vision, an ultrasonic sensor to detect obstacles, a USB microphone and a speaker for voice interaction, and an LED ring for status indicators.
• AI (Virtual Assistant): The assistant function will be enhanced by replacing the Gemini API with Ollama, an open-source software that runs LLMs directly on a PC. This gives us more control over the model and ensures faster responses.
• Control App: The mobile app will work as a configuration and manual control center, offering a status dashboard, battery monitoring, Wi-Fi configuration, a virtual joystick, and a real-time camera feed.

Detailing the main features

"Go to..." Navigation System

What it is: The user will be able to "teach" the robot paths to specific locations in the house (such as "bedroom" or "kitchen") and then, by voice command, ask it to move to one of these saved spots.

How it will work: First, the user defines and names a starting point. From there, they activate the "recording mode." Using the virtual joystick, they manually guide the robot to the next spot and save this new location with another name. The Raspberry Pi will record the sequence of movements between these points. Later, when commanded by voice, for example, "Hey Xand, go to the kitchen," the robot will execute the saved steps to reach the destination.

Object Tracking with Computer Vision

What it is: The robot will be able to identify and follow a colorful ball.

How it will work: The robot's camera will capture images in real-time. Using the OpenCV library in Python, the software will filter the image to isolate a pre-configured color (defined in the app). Based on the colored object's position and size within the image frame, the robot will adjust its motor speeds to approach or center itself with the object, maintaining a safe distance measured by the ultrasonic sensor. After reaching the ball, it will return to its starting position.

Pre-defined Movements

What it is: The robot will perform automated actions to interact playfully with the user.

How it will work: The user can ask the robot to perform an action, such as "dance." In response, the robot will execute a pre-programmed sequence of movements that simulates dancing or another action.

Assistant Features

What it is: General assistant-like responses provided by the model running on the server.

How it will work: The user can converse with the robot via the microphone about topics beyond its physical functions. These interactions depend solely on data processing, internet access, and the ability to listen and speak.