1. Project Overview
This project develops an automated LEGO sorting system using the Hiwonder JetArm robotic arm, computer vision, coordinate mapping, and inverse kinematics.
The system uses a camera to detect LEGO pieces within a workspace. The detected image coordinates are converted into real-world robot coordinates, allowing the JetArm to automatically locate, pick up, and sort the LEGO pieces into designated bins.
The overall workflow is:
Camera Input → LEGO Detection → Pixel Coordinates → Coordinate Mapping → Inverse Kinematics → Pick and Place → Sorting Output
2. Computer Vision Detection
Python and OpenCV are used to process the camera input.
LEGO pieces are detected by identifying contours, generating bounding boxes, and calculating the center point of each detected object. These center points provide the pixel coordinates required for the coordinate-mapping stage.
YOLO was also explored as an AI-based object detection method using trained object-detection models.
3. Coordinate Mapping
Pixel coordinates obtained from the camera cannot be directly used as robotic arm coordinates.
A homography-based mapping system is therefore used to convert image coordinates into real-world coordinates within the JetArm workspace.
Reference points placed around the workspace are used to calculate the homography transformation. This method reduces errors caused by camera perspective distortion and provides more accurate target coordinates compared with simple pixel scaling.
4. Robotic Arm Control
After the target position has been calculated, inverse kinematics is used to convert the target coordinates into the required JetArm joint positions.
The JetArm then performs a pick-and-place sequence:
- Move toward the detected LEGO piece.
- Lower the arm toward the target.
- Close the gripper.
- Lift the LEGO piece.
- Move toward the appropriate sorting bin.
- Release the LEGO piece.
5. Hardware
The main hardware used in the system includes:
- Hiwonder JetArm robotic arm
- Camera
- NVIDIA Jetson
- Robotic gripper
- LEGO sorting workspace
- Sorting bins
The NVIDIA Jetson performs the main processing and sends movement commands to the JetArm.
6. Results
The developed system is capable of detecting LEGO pieces, determining their positions, and controlling the JetArm to automatically perform sorting operations.
The use of homography-based coordinate mapping significantly improved the positioning accuracy compared with direct pixel scaling.
7. Challenges
Several challenges were encountered during development:
- Camera perspective distortion affecting coordinate mapping
- Lighting, reflections, and overlapping objects affecting detection
- Complexity of inverse kinematics
- Small coordinate errors causing the gripper to miss the target
Further development may focus on improving calibration, inverse kinematics, path planning, object detection accuracy, and 3D localization.