Robotics 104-1 Final project: A self-driving mobile platform
Lou, Seddik, Benjamin Derville and Chun-Kan Gary Chow

Abstract— In this paper, we present our work on building an autonomous car. We first describe our choosen approach and the reason behind this choice, we then describe what we manage to implement and then we finish by reviewing our work done. I. INTRODUCTION
Autonomous driving is maybe the most popular current project in machine learning, and especially deep learning.
Billions of dollars are assigned to create self-driving car.
Google lead the movement with their Google Car created from their lab Google X, followed by Tesla who manage to quickly catch up with their big fleet of cars driving on the roads at each instant. Uber just created a research center for this work, GM partner with Lyft, George Hotz, an americain car made his own home-made simple self-driving car and even Nvidea created a demo. In this sense, this work is not really original or innovative. We however thought this work was not yet popularized to students projects. This works is also interesting because it’s not ’achieved’ even by the current state-of-the-art, there are still lots of works to be done. II. T HEORY

will be very hard to distinguish between them and not useful for a long-term solution. Additionally we would encounter a major problem when defining the color and shape of a car. So instead we design an machine-learning algorithm which uses a similar approach like Google and Tesla does.
By training our own classifier for each element in traffic signs, which requires most of the computation, it allows our mobile platform to detect traffic signs and differentiate between them. Once the parameters are available, we can computationally cheap detect our classified objects. The result of our very fast image processing allows us to integrate the method for real-time applications.
1) Haar Feature-based Cascade Classifier: For our mobile platform we decided to use the Haar Feature-based
Cascade Classifier, which provides us a satisfying prediction accuracy based on a relative small amount of samples. The main advantage of this classifier is that it consists of basics decision-trees classifiers. By taking the integral of our area of interest we can compute our feature very cheap, results as a perfect fit for low CPU computers with cameras. The
OpenCV Cascade classifier takes the edge-, line and centersurround features into account.

A. Mobile platform
We initially thought about using the university material, but it felt for us that it wasn’t aligned with our objectives.
We wanted to touch not only software, but also hardware. We also wanted something that we could extend or modify with easy, understand the limit and the capacity of our robot. We finally prefered something that we could continue to work on, even after the class was over. We decided to buy cheap novice hardware to build the robot. We already had a Beaglebone that appeared to work great as remote controller, the brain of our robot. By searching online and at Taiwan market, we found needed hardware and even a platform to control the robot. B. COMPUTER VISION
C. Object recognition
One sufficient task of a selfdriving-car is to able to perceive its environment and interact with it. So one major challenge is to recognise objects like traffic signs, passengers and cars. In our final project we have focused to design a generic method to recognise traffic signs. As we are looking for a generic approach, we have change our definition of a traffic sign to contains only stop signs and traffic light. One simple approach would be color combined with a shape detection . But as a lot of traffic signs are quite similar, it

D. Lane detection using Hough tranformation
Inspired by a different Youtube videos, we saw the potential to predict the desired trajectory by detecting the lanes of a road. For this the most common method is to get the edges and apply a Hough transformation on it.

Houghspace 1. The Houghspace is a space to describe lines consisting of the vectors ρ and θ . It is very similar to a polar coordinate system. ρ = x ∗ sin(θ ) + y ∗ cos(θ )

1) Hough transformation: The Hough transformation takes the edges as input, do a mapping to the Hough space and storage them in an accumulator. By providing the method thresholds like the minimum line length, line gap, the method can interpretate the accumulator and yield lines of infinite length. Afterwards it converts those infinite lines to finite lines. 2) Progressive Probabilistic Hough Transform: The
Hough transform is not a very fast algorithm for finding infinite lines in images of a certain size and since additional analysis is required to detect finite lines, this is even slower.
A way to speed up the Hough Transform and finding finite lines at the same time is the Progressive Probabilistic Hough
Transform. The idea of this method is to transform randomly selected pixels in the edge image into the accumulator. When a bin in the accumulator corresponding to a particular infinite line has got a certain number of votes, the edge image is searched along that line to see if one or more finite line(s) are present. Then all pixels on that line are removed from the edge image. In this way the algorithm returns finite lines.

E. Lane detection using Colordetection and Shadow Removal
As it later turns out that our first approach using Lane detection with Hough transformation works for a real road but not for our self-designed road, we decided to change our approach and design a robust color detection for our
Lane detection. Our goal was to make our algorithm robust against light and shadow, while it should only detect ’red’.
For Example if we use a RGB based threshold, we would allow skin color and orange to be detected by our algorithm, as we need to set very extreme threshold to make it robust for a shadowed and brighted red. This problem could be solved by using the HSV color spaceand a Shadow Removal algorithm, which will now be introduced:
1) HSV: A coloured image is usually presented in a
RGB format. HSV, sometimes referred as HSB stands for
(Hue, Saturation and Brightness), is a cylindrical coordinate representation of an RGB image, which enable us to set the influence of incoming lights and shadows.

2) YCbCr: The algorithm that detects shadows and remove it will be described in the Implementation section. To understand the big picture of the algorithm, we will introduce the YCbCr color space based on the luma-chroma color theory. It can be calculated from a corresponding gammaadjusted RGB image:
Y = KR R + (1 − Kr − Kb )G + Kb B
Cb = 0.5

B −Y
1 − Kb

Cr = 0.5

R −Y
1 − Kr

In the YCbCr color space, the Y represents luminance information while Cb and Cr represent the color information
(difference between red/blue to the luminance).
F. Autonomous driving
We decoupled our work required in multiple steps. Gradually complexifying the system and also using the current system to build the next. The first objectif was to set up a car that would follow a simple red line. To achieve this, we would first create a remotely controllable car that can capture input, like camera vision and front-sensor but also state/output. We would then run the car manually, recording all the experiences, repeating the process in differents place, in different conditions. We would then give train a supervised algorithm that try to emulate the output of the microcontroller, that is the directions and intensity of the four wheels based on the captured image and front-sensor.
The second objectif would be to use deep reinforcement learning, for example, the famous, DeepQLearning created by the DeepMind team, to improve the driving performance.
We would start by implementing on the car the previously learned model to follow a red line. By implementing a line detection, we could train the robot with the algorithm by feeding it with bad reward whenever he deviate from the line and feeding it good reward whenever he follow it closely.
Our third objectif would be to add signs, starting with the stop sign. We would first add a stop sign on the road. We would then train a stop sign detector. Either by manually extracting images that the robot captured or by using internet dataset of images to train it. We would then modify the reinforcement learning, by feeding him with bad reward if we doesn’t stop for few seconds whenever a stop appear in front of the robot. We could also retrained our model from scratch, with all the data aggregated, by overwriting the behavior to state that the motors are stop when detecting

a sign. This approach have the benefits of directly producing a model with new behavior but require lots of computation and doesn’t allow to to control the importance of an action: which is more important; stopping at a sign or avoiding an accident? This maybe one of the last interesting point to work on: adapting to new policy/environment quickly and prioritizing some actions over other.

III. I MPLEMENTATION

3) Raspberry Pi: We quickly bought a raspberry pi 2 and set up a new robot. The hardware was slighly different but we manage to build it without too much problems. We however had a problem with the voltage. The battery that we bought to power the robot was apparently of a very bad quality, with a non-constant voltage. This created the Raspberry pi to power off the wi-fi dongle when we tried to used it. It took us some time to realize where the problem came from. We then tried other battery that worked well. We quickly adapted the code to control the robot, and we could the control it again.

A. Building the mobile platform
1) Beaglebone + Arduino: Our first approach was to use the Beaglebone as the ”brain” of the robot and using the
Arduino mega with a motor shield as the hardware to control and ”sense”.
We set up this approach, however, when testing this, we discovered that the communication that we set-up between the two, using some GPIO, was error-prone and had a big latency (more then 0.5s), we decided to remove the Arduino.

B. Remote Control
The remote control is a simple script that is launched from an ssh connection that allow to control the robot with a keyboard. It register every moves made and the instant state/output of the robot.
We also built a server with flask and nginx to be able to stream the camera vision of the robot. Unfortunately, this work was lost with the Beaglebone.
C. Computer Vision

2) Beaglebone + L293D: Our second approach at building the platform was simply to use the beaglebone black directly with an L293D, a dual h-bridge micro-controller.
By linking two motors by two, we could use one merely micro-controller to control the robot.
This approach was quite successful. After implementing the remote control, using the keyboard on one remote laptop connected through ssh, we were able to control the robot and make it move.
However, went deciding to fixing a bit the robot so that no component would fall in case of a brusk moment, we broke our Beaglebone. It appear that the glue bought in the hardware store was attacking plastic, which created short circuit on the Beaglebone. Just before we could work on the autonomous driving, we were back to square one.

The Computer Vision algorithm were all coded on a iOS platform in Python 3.4 with the Opencv2 library. As one of our major goal was to find our own way to design an effficient fast image processing algorithm, it was essential to code everything by our own. By this way we had the full control of our algorithms, got a deep understanding into the image processing and learned which for Example which filter is at which situation the best to use. So basically our algorithms consists of the Object recognition, Lane tracking and Obstacle avoidance.
1) Object recognition: As we stated above, we were using
Machine-learning algorithms for the Object recognition. At the beginning we had to collect enough images to train our classifier. Here is a table containing the amount of samples:

Object
Traffic light
Stop sign

Positive
40
50

Negative
800
1000

Once we finished to collect the images, we cropped each of them to the samplesize (25x25). The next step is to convert them into a .vec file in order to create a list of samples to simplify the workflow. Opencv2 allowed us to train our classifier very straightforward and after 22 hour of computation time we ended up with two .xml files, which contains our trained parameter. By using the Opencv2
LightCascade.detectMultiScale method we could evaluate our classifier and look if in our gray-scaled video-frame were any of our desired objects.
A picture, which shows that the algorithm detecting a stop sign. Moreover, we extended our algorithm to distinguish between a red and green traffic light. Once a traffic light is detected, the program reduced the frame to our area of interest and as the traffic light is after the cropping in the center, we have applied a 25x25 Kernel Gaussian blur on it to reduce the noise. With the Opencv2 minMaxLoc method we could detect the brightest spot and by comparing the frame in respect to the brightest spot , we can deduct the mode of the traffic light.
Here is a picture we took, which shows the multi object detection, while differentiating between a red with a green traffic light.

2) Lanedetection using Hough transformation: At the beginning we have cropped the frame to the area of interest and applied a (3x3) Gaussian blur to reduce the noise.
Afterwards we the Opencv2 Canny method detects all the edges. With these edges as a input, the Progressive
Probabilistic Hough Transform was used to get the lines.
After differentiating between the left and right side of the

image, a filter was used to collect all the lines with the same gradient. Using a linear model, the algorithm can determine, when y=ax+b than we are on the left side with a > 0, and on the right side for a < 0.
3) Shadow detection: First we transform our image to the YCbCr color space. Next, focusing on the Y channel, its histogram is computed. Histogram dissension gives us a higher contrast image in the Y channel. After that, the mean of the image in the Y channel is computed. Then slidingwindow iteration through the image is performed. The sliding window size is 33 matrices. In order to decide which pixels belong to the shadow, two approaches are followed. First, shadow pixels that have intensity less than one standard deviation of the whole image are classified. This step cannot identify the shadow regions properly as some shadow pixels are identified as non-shadow pixels. So next, the non-shadow point?s mean and standard deviations for the sliding window are computed. Now, the pixels that have intensity less than the one standard deviation of the windows are considered shadow pixels. Afterwards isolated pixels are removed by using morphological operations (cv2.dilate and cv2.erode).
After removing the shadow in the binary image, we convert the shadow-free image back to the BGR color space and can proceed with the Lanedetection. Here is a workflow chart of the Shadow detection algorithm.

4) Lanedetection using color detection: First of all the shadow removal detection was applied to preprocess the image by removing the shadows. After cropping to our area of interest, the HSV colorspace was used to filter out non red pixels. The reason to use the HSV colorspace instead of the RGB is to make the detection robust and stable, as
HSV allows us to set the influence of light and shadow in the picture. Additionally very extreme thresholds would be needed for a stable algorithm when using the RGB colorspace, but which also detects skin and orange like colors. So using the HSV colorspace instead, could solve this problem. In our case we needed to filter our image with two different thresholds, as the Hue value H for red can be either be [0,15] or [350,360]. After thresholding those two images, morphological operations got rid of error sources like noise for Example, before blending the two frames to one back together. Finally the gradient can be calculated computationally cheap by calculating the centroid with a simple contour detection method combined with the use of

Fig. 1.

Line detection algorithm

sparse matrices.
5) Obstacle detection: For the obstacle detection we can use a depth analysis of an image. But as we want to reduce the image processing time as low as possible, we decided to buy a sonar sensor and use the data of the sonar sensor to detect obstacles in front of us.
IV. D IFFICULTIES
As projects never goes smoothly from the beginning, here is a list of our major difficulties in the final project:
• Hardware problems as stated in the Building the robot part • Let the Raspberry Pi Camera takes very fast focus. (At the beginning it could took every 0.4s a picture)
• Implementing two working thread on the Pi, one for the motor control and one for the image processing
• Approach for the shadow removal and detection algorithm
• Design the lane prediction algorithm as robust of possible
• Hough Transformation for curved lines
• Getting familiar with the procedure of creating samples for the training
• Thinking of a way to calculate whether a traffic light is red or green
• Motivation and Commitment of our team
V. W ORK D ISTRIBUTION
Here is a list of the work, which each teammember has done: • Chun-Kan Gary Chow:
– Got familiar with the Computer Vision theory
– Collected samples and trained the classifier
– Implemented the Object recognition
– Implemented the Lane tracking using Hough transformation
– Implemented the Shadow detection algorithm and the color detection
– Assisted to build to mobile platform
– Wrote the final report
• Benjamin Derville:
– Bought the basic equipment for our mobile platforms

•

•

– Built all three different versions of our robot
– Found solutions to build our mobile platform
– Implemented the remote control and camera connection
– Solved the problem of the delayed picture shooting
(2 fps to 30 fps)
– Setted up the communication to the raspberry
– Implemented a server to communicate with the
Beagle bone (robot version 2)
– Wrote final report
Seddik Bouafia
– Supportive contribution
– Bought missing and needing parts for our mobile platform Lou
– Bought the paper for our self designed road
VI. C ONCLUSION AND O UTLOOK

The main goal of our final project was to get a deeper insight into the self-driving technology with our mobile platform. We succeeded in implementing an object recognition algorithm, which provided us a very precise detection rate.
We were kinda surprised of this positive result and even more impressed how fast one can dive into this amazing technology. Recently Deep learning algorithm is getting more and more popular and an next outlook could be do use
Deep learning algorithms to further improve the detection rate. In contrast our current lane tracking algorithm using shadow removal and color detection is definitely a very simplified approach. An additional PID Controller to regulate the steering angle can ensure an accuracy of 99%. But for real time tests we seek for an accuracy of 99.9%, while assuming that we are driving on a real road. So an optimised
Hough transformation algorithm for the lane detection combined with a Deep learning algorithm, which save for each camera frame its current state (steering angle and speed) can achieve this goal. George Hotz showed in his last video, that the actual researchers and companies are using exactly this technology to train their controlling algorithm, as it has the potential to be 99.9% accurate with just a relative small amount of code. As our mobile platform is self made and the theory can be mostly found in the internet, we can easily keep working to improve our self driving mobile platform.
We are really thankful for the Professor and the TA to give us the opportunity and motivation to work on such an emerging topic and to have a practical project, as our home universities have a very theoretical teaching method. Furthermore, as the deep learning algorithms are just amazing, we believe that the self driving technology will be nearly perfect in the next ten years and that this final project is an invaluable lesson.