Building an Autonomous License Plate Reading Robot
The source code for CNN training pipeline is hosted on GitHub, found here.
As part of our ENPH 353 course at UBC, my team and I tackled an exciting challenge: creating an autonomous robot capable of navigating a simulated environment, identifying license plates, and avoiding obstacles. This project combined computer vision, machine learning, and robotics to create a system that could “see” and interact with its surroundings. Here’s how we did it.
The Challenge
Our task was to build an agent that could:
- Navigate a predefined course
- Identify characters on parked cars’ license plates
- Avoid NPC (non-player character) obstacles
We aimed for a simple yet robust approach, focusing on reliability and performance.
License Plate Recognition: A Three-Step Process
Step 1: Perspective Transform
Initially, we tried using SIFT (Scale-Invariant Feature Transform) for homography, but it didn’t work well with our simplified images. Instead, we developed a custom approach:
- Apply an HSV mask to isolate blue car pixels
- Determine the corners of the white backing for the parking ID and license plate
- Use a 4-point transform to “flatten” the off-angle image
This method allowed us to consistently extract the license plate region, regardless of the car’s position or angle.
Step 2: Character Segmentation
Once we had a flattened image of the license plate, we segmented it into individual characters. We converted these to binary images to simplify the input for our neural network.
Step 3: Convolutional Neural Network (CNN)
We trained a CNN to recognize individual characters. Some key points of our approach:
- Used synthetic data generation to create a large training dataset
- Augmented the data to introduce artifacts similar to those in the simulation
- Fine-tuned the CNN architecture and training parameters
After several iterations, we achieved a 97.6% accuracy on our validation set.
Robot Control: Navigating the Course
For controlling the robot, we used a combination of PID (Proportional-Integral-Derivative) control and hard coding. We implemented a finite-state machine to manage different behaviors:
- Line Following: Used PID control to follow lines on the course
- Obstacle Detection: Detected pedestrians and vehicles using color thresholding
- Parked Car Detection: Identified parked cars to read their license plates
Challenges and Learnings
Data Quality: We found that the quality and diversity of our training data significantly impacted our model’s performance. Synthetic data generation and augmentation were crucial.
Robustness: Our initial approaches often worked well in ideal conditions but failed in edge cases. We had to continuously refine our methods to handle various scenarios.
Integration: Combining computer vision, machine learning, and robotics required careful integration and testing. Each component had to work seamlessly with the others.
Performance Tuning: We spent considerable time fine-tuning our CNN and control algorithms to achieve the desired accuracy and reliability.
Results and Reflections
Our final system performed well, successfully navigating the course and reading license plates with high accuracy. The project gave us hands-on experience with:
- Computer vision techniques
- Machine learning model development and training
- Robotics control systems
- System integration and testing
Conclusion
This project was a fantastic opportunity to apply theoretical knowledge to a practical, multidisciplinary challenge. It highlighted the importance of robust design, iterative improvement, and the power of combining different technologies to solve complex problems.
The skills and experience gained from this project have broad applications, from autonomous vehicles to industrial automation. As AI and robotics continue to advance, projects like these provide valuable insights into the challenges and possibilities of creating intelligent, autonomous systems.