T.C. BAHÇEŞEHİR UNIVERSITY

VISION BASED TARGET TRACKING CONTROLLED SENTRY

Capstone Project

Fikret Taygun Duvan

İSTANBUL, 2011

T.C. BAHÇEŞEHİR UNIVERSITY

FACULTY OF ENGINEERING

DEPARTMENT OF MECATRONICS ENGINEERING

VISION BASED TARGET TRACKING CONTROLLED SENTRY

Capstone Project

Fikret Taygun Duvan

Advisor: Dr. Khalid Abidi

İSTANBUL, 2010

T.C. BAHÇEŞEHİR UNIVERSITY
FACULTY OF ENGINEERING DEPARTMENT OF MECATRONICS ENGINEERING

Name of the project: Vision Based Target Tracking Controlled Sentry Name/Last Name of the Student: Fikret Taygun Duvan Date of Thesis Defense: 23/01/2011

I hereby state that the graduation project prepared by Your Name (Title Format) has been completed under my supervision. I accept this work as a “Graduation Project”. Dr. Khalid ABIDI

I hereby state that I have examined this graduation project by Your Name (Title Format) which is accepted by his supervisor. This work is acceptable as a graduation project and the student is eligible to take the graduation project examination.

Asst. Prof. Yalçın Çekiç Head of the Department of Mechatronics Engineering

We hereby state that we have held the graduation examination of Your Name and agree that the student has satisfied all requirements.

THE EXAMINATION COMMITTEE
Committee Member 1. Khalid ABIDI 2. ………………………….. 3. ………………………….. Signature ……………………….. ……………………….. ………………………..

ACADEMIC HONESTY PLEDGE

In keeping with Bahçeşehir University Student Code of Conduct, I pledge that this work is my own and that I have not received inappropriate assistance in its preparation.

I further declare that all resources in print or on the web are explicitly cited.

NAME

DATE

SIGNATURE

ABSTRACT

VISION BASED TARGET TRACKING CONTROLLED SENTRY Fikret Taygun Duvan

Faculty of Engineering Department Mechatronics Engineering

Advisor: Dr. Khalid Abidi JANUARY, 2011, 43 pages

After the 2nd World War pigeon population started increasing rapidly.(1) Large cities were affected from the overpopulation more than the smaller ones because the food supply was nearly unlimited for the pigeons. The contents of the pigeon droppings harm the historical buildings and statues when resolved in water. The only way to protect these historical heritages is to keep the pigeons away from them. Therefore this work focuses on manufacturing a system to keep the pigeons away from the specified areas. A vision based target tracking system might be useful in keeping them away from the buildings. The system consists of an image processing software, a target tracking algorithm, a computer controlled turret and integration of all these systems. This work explains the design processes of the mentioned system.

Key Words: Target Tracking, Image Processing, Servo Control

ÖZET
VISION BASED TARGET TRACKING CONTROLLED SENTRY Fikret Taygun Duvan Mühendislik Fakültesi Mekatronik Mühendisliği Bölümü Tez Danışmanı: Dr. Khalid Abidi OCAK, 2011, 43 sayfa

2. Dünya savaşının bitimiyle beraber sokak güvercinlerinin sayısında inanılmaz bir patlama yaşandı.(1) Büyük şehirler bu kontrolsüz artıştan çok daha fazla etkilendiler. Bunun en büyük nedenlerinden biri, güvercinlerin büyük şehirlerde sınırsız yemek kaynakları bulmalarından ileri geliyordu. Güvercin dışkısının suda çözünmesiyle beraber tarihi binalara ve eserlere zarar verdiğinin ortaya çıkması ile birlikte bu konuda farklı önlemler alınmaya çalışıldıysa da pek etkili olan bir yöntem henüz görülmedi. Makalenin içeriğini oluşturan konu ise , tarihi eserleri ve binaları güvercinlerin zararlı etkilerinden korumak için üretilecek hedefe otomatik yönelen bir su silahıdır. Bu makale sistemin modelleme, yazılım, design ve üretim aşamalarını kapsamaktadır.

Anahtar Kelimeler: Hedef izleme, Görüntü işeleme, Servo kontrol mekanizmaları

Table of Contents
ABSTRACT .......................................................................................................................................6 ÖZET .................................................................................................................................................7 LIST OF TABLES ..............................................................................................................................9 LIST OF FIGURES .......................................................................................................................... 10 1. INTRODUCTION ........................................................................................................................ 11 1.1. Historical Background on Target Tracking ............................................................................. 15 2. MATERIALS & METHODS ........................................................................................................ 16 2.1 Manufacturing and Control of the Turret ................................................................................. 16 2.1.1 Computer aided design ..................................................................................................... 17 2.1.2 Motor selection ................................................................................................................. 19 2.1.3. Performance Test for the Turret System ........................................................................... 20 2.1.4. Servo Control .................................................................................................................. 22 2.2. Image Processing and Target Tracking ................................................................................... 23 4. 5. DISCUSSION ........................................................................................................................... 25 CONCLUSION ......................................................................................................................... 26

Appendix A ...................................................................................................................................... 27 Physical Dimension Constraints .................................................................................................... 27 Appendix B ...................................................................................................................................... 29 Motor Selection for the Base ......................................................................................................... 29 Motor Selection for the Barrel ...................................................................................................... 31 Appendix C ...................................................................................................................................... 32 Part Specifications ........................................................................................................................ 32 Appendix D ...................................................................................................................................... 34 Source Code.................................................................................................................................. 34 Appendix E....................................................................................................................................... 37 Part Drawings ............................................................................................................................... 37 Bibliography...................................................................................................................................... 38

LIST OF TABLES
Table 1 Camera specifications ........................................................................................................... 17 Table 2 Desired properties for the motor selection process ................................................................ 19 Table 3 Motor specifications ............................................................................................................. 19

LIST OF FIGURES
Figure I Application of target tracking on sports ................................................................................ 11 Figure II Sample gun turret drawing .................................................................................................. 12 Figure III Pigeon nest on the preventative spikes ............................................................................... 13 Figure IV Simplified drawing of the turret mechanism ....................................................................... 16 Figure V ............................................................................................................................................ 18 Figure VI ........................................................................................................................................... 18 Figure VII .......................................................................................................................................... 18 Figure VIII ......................................................................................................................................... 18 Figure IX Performance test schematic for the turret .......................................................................... 20 Figure X The view from the system’s camera..................................................................................... 20 Figure XI Sample Performance Test Graph ........................................................................................ 21 Figure XII Checker board ................................................................................................................... 23 Figure XIII Sketch showing the proportions between the distance of the object and the area covered in the field of vision .......................................................................................................................... 24 Figure XIV Showing the preparations of the experiment.................................................................... 24 Figure XV Vitruvian Man by Da Vinci ................................................................................................. 27 Figure XVI Sketch Showing the vision sight of the camera and the calculated min distance for the object ............................................................................................................................................... 29 Figure XVII Sketch showing the corresponding length of 1 degree in 1.075 meters away from centre29

1. INTRODUCTION
Tracking a target basically means estimating or trailing a target’s projectile in consecutive images which is achieved by processing an image. Image processing can be defined as signal processing for which the input is an image and the output is a set of characteristics or parameters related to the image. Most image-processing techniques involve treating the image as a two-dimensional signal and applying standard signal-processing techniques to it. Target tracking systems have been implemented to numerous civilian and military appliances. Air traffic controllers, ocean surveillance and avionics applications or sport events for generating viewer statistics i makes use of target tracking systems for civilians(2). In spite of the fact that target tracking systems are being used in civilian applications its roots go back to the invention of the radars used by military forces. In the autumn of 1922, Albert H. Taylor and Leo C. Young at the U.S. Naval Aircraft Radio Laboratory were conducting communication experiments when they noticed Figure I Application of target tracking on sports that a wooden ship in the Potomac River was interfering with their signals; the system was using separated transmitting and receiving antennas and detecting targets due to changes in the signal. In 1930, Lawrence A. Hyland working with Taylor and Young, at the U.S. Naval Research Laboratory (NRL) in Washington, D.C., used a similar arrangement of radio equipment to detect a passing aircraft. This led to a proposal by Taylor for using this technique for detecting ships and aircraft ii. Guidance and control of manned and unmanned vehicles and weapons are becoming increasingly important for all branches of the armed services in order to decrease their casualties in combats. Therefore use of unmanned ground, sea, and aerial vehicles that must guide themselves autonomously to their target is increasing. In the work of Ajmal S. Mian (3), it is suggested that radars are expensive. Also they use active illumination of aircrafts with electromagnetic waves resulting for them to give away position data which is undesirable from military point of view. Hence video surveillance tracking units are low cost, unable to be detected and in the military, visual aircraft tracking i ii

See Figure I http://en.wikipedia.org/wiki/History_of_radar

can be used to aim and guide weapons. He suggested some modifications to an existing algorithm commonly known as the KLT (Kanade-Lucas-Tomasi) algorithm because it was initially proposed by Lucas and Kanade(4) and later developed fully by Tomasi and Kanade.(5) Without going into too much detail, the algorithm can be described as the composition of two different algorithms which are optical flow estimation and feature based tracking. Optical flow based techniques exploit the fact that there is minimal change in images that are taken at small intervals of time, such as in video, even in the presence of relative motion between the camera and the objects in the scene. On January 2006 a student from University of Johannesburg named Gideon Ferreira submitted a dissertation on stereo-vision based autonomous gun turret(6). The project focuses on understanding the mechanics and the working system of the gun turrets which are being used by the armed forces all around the world iii . Ferreira modelled a sample system and manufactured his design. Instead of a real shooting mechanism he employed a laser sight also he uses two lowcost webcams for target detection. Although he did an outstanding job on the whole system there are some points that are not included in his work. First of all, the range of the system is not defined in any part Figure II Sample gun turret drawing of the report. Which means that there is no way to assess the system in the means of speed characteristics, what object the system will recognize and from what the distance will the objects are going to be recognizable. Although there is a speed calculation which is done in order to select motors for the system, the relations between the maximum needed speed for the specified object to be detected and motors’ actual speed are missing. What’s more in the performance test section for the designed controller the settling time, overshoot and steady state values are not defined beforehand, meaning that the controller is not designed to meet any predefined values or design criteria. Also the physical dimension constraints are not quantified, the design seems to be formed randomly. The work mentioned above has a number of common points in the means of design, system mechanics and software with the work that is covered by this project. The structure of iii See Figure II Sample gun turret drawingFigure II

this project derives inspiration from the mentioned work thus filling the spotted gaps in that one is one of the goals of this report. After specifying the gaps on the previous work, the specific aim of the project should be clarified in order to determine the desired functions. After World War II street pigeoniv population is increased world-wide, mostly in larger cities.(1) Food was abundant from pigeon enthusiasts and rubbish, an extensive food supply, supported the overpopulation.(7) As a result of pigeon overpopulation most of the historical buildings and statues are in danger since when resolved in water, contents of the pigeon droppings v become harmful. Also pigeons which are searching for food destroy the vegetation, causing problems for the farmers.(7) In Europe, in cities like London, Paris and alike there are some preventive apparatuses which are mounted on the sides of the buildings or statues in order to protect them against the dropping threat. These apparatuses are angled spikes which are not actually working well as can be seen on Figure III. The indicated reasons are the basis of this project. Designing a system which will track and keep the desired animals away from the specified areas might be useful both for protecting the mentioned buildings and keeping pigeons or other harmful creatures away from the agricultural areas. For that reason manufacturing an autonomously guided water pistol might be a solution. However it is not such an easy task since it involves integration of different systems. First a target tracking system is needed to be designed in order to locate and aim at targets. Following that a computer controlled turret mechanism has to be modelled and the final step will be the integration of the two systems.
Figure III Pigeon nest on the preventative spikes

The need for such a product is mentioned above; the next step is determining the desired functions for the system. The first desired function for the system concerns the mobility of the mechanism. Regarding to the fact that the need for the system to be placed in areas which are not easy to reach, making the design in such a way that will allow the mechanism to be

iv v

Known as Columba Livia or Rock dove. However it is going to be referred as pigeons throught the article. The droppings contain water-soluble salts which comprise halite, sylvite, potassium calcium sulphate, aphthitalite, apatite group minerals, weddellite and gypsum (24).

carried using one hand is one of the desired functions. This design constraint is thoroughly discussed in Appendix A. The camera which is employed in the system needs to recognize the specified objects and track them during the time interval when objects are in range of the system. Since the flight speed of the target is known, the range of the system can be calculated. By doing this calculation another design constraint for the system is obtained. More information on the range of the system is discussed in the following chapters. While manufacturing the system another important point is the maximum available angular velocity of the system. This plays a significantly important role on tracking the desired target. After motion is recognized by the tracking system, the performance of the system depends on the turret’s tracking abilities. Therefore the turret should be able to track the desired target. The required angular speed for the turret system is calculated in Appendix B. A laser sight is going to be fitted to the system instead of a water pistol, since manufacturing a fully working shooting mechanism, calculating the dynamics of the system, creating a control system and preparing a target tracking software is a hard subject. Also just one semester is not enough time to design and manufacture such a mechanism. Therefore the project is divided into stages and the first stage is going to be manufacturing a turret using a laser sight instead of a water pistol. The desired functions for the system are listed below:  The physical dimensions of the design should allow the mechanism to be carried easily using one hand.  The tracking system needs to recognize the motion and track the desired targets. The range of the system has to be specified.  The turret needs to follow the commands which are sent from the tracking system and aim at the desired object. The maximum angular speed needed for the system has to be calculated.

1.1. Historical Background on Target Tracking
After performing meticulous literature search on target tracking, it became evident that there are two fundamental approaches against the topic. One of these approaches is recognition-based tracking which is examined in the work of Bray(8), Lowe(9) and Schalkoff and Mc Vey (10). The main idea which is explained in these cited sources is that this method is basically recognizing the object in consecutive images and extracting position data. The accuracy of the method heavily depends on the recognition of the object rather than its movement which means if the moving object is not recognized the system will not extract any position data from the image. On the other hand the motion-based recognition technique does not need to recognize the object but needs to detect the motion(11). Since we are not living in a perfect world all of the measurements that are performed include errors. In the case of target tracking the main disturbance is the noise involved in the images. In order to obtain better results from the processed images, noise filtering methods are created(12). According to the results of the literature search it became clear that, Kalman filtering system is one of the most used techniques amongst the others(2)(13)(14). It is an updated form of the Bayesian filtering method (15). Kalman filter is successful in smoothing random deviations from the actual path of the targets, improving in its ability to predict the path of each target as more measurements from the tracker were processed(16).

2. MATERIALS & METHODS
The turret is going to be used to aim and track a moving target and will be controlled by the image processing software. The mechanism itself is modelled on SolidWorks. The design is based on predefined performance characteristics such as size, shape, material selection and the actuator selection for the turret system. The manufacturing process of the project could be divided into three stages: 1. Manufacturing and control of the turret. 2. Image processing software for target tracking. 3. Integration of the two separate systems.

2.1 Manufacturing and Control of the Turret
A small scale model of an actual gun turret has to be manufactured. As mentioned earlier in the text this project is the first step to reach the real objective of protecting the historical monuments. Therefore a laser sight is going to be fitted inside the barrel, instead of an actual shooting mechanism. The turret mechanism will have two degrees of freedom since the system needs to have a rotating base in horizontal axis and a laser fitted barrel rotating in the
Figure IV Simplified drawing of the turret mechanism

vertical axis. See Figure IV to get a clear idea. In order to provide the mentioned conditions, the system has to employ two actuators.

While designing the system the most important criterion which is taken into consideration is the size of the device. In order to provide mobility to the mechanism, the area that will be covered should be smaller than 0,073 m2.See Appendix A for calculations. The desired angular velocity for the base is ω = 4.6425 rads/sec. Appendix B contains both the motor selection processes and the obtaining method of the angular speed.

The specifications of the webcam, which is selected for the project, are shown in Table 1. Deriving from these properties the maximum velocity for a detectable object is calculated in terms of pixels per seconds. The width of the images that are taken by the webcam is 640

pixels and the frame rate is 30fps. That means the time interval for a frame to be replaced by the following is t=0.033 seconds. In order for the image processing algorithm to track the object there has to be more than one frame which the object has captured in. For the motor speed calculations in Appendix B the number of frames required is taken as 5 frames since calculating the exact processing time is impossible because even the lighting conditions effects the processing time.
Resolution Logitech Pro 4000 Quickcam 640x480(pixels) Frame Rate 30 (frames per second) Angle of Sight 44o

Table 1 Camera specifications

The mechanism is going to be in a stationary position if there is no object detected by the image processing software. This condition results in a narrow vision area for the mechanism since the angle of sight of the webcam is 44 degrees.
2.1.1 Computer aided design

There are a number of benefits of using computer aided design or CAD software. Some of the benefits are visible in the design process, while the others may not be visible directly but result in improvement in the quality of product and better control over designing process. (17)Benefits of using CAD software instead of traditional designing processes emerge from the following factors: 1) How complex the engineering drawing is: For highly complex drawings the traditional drawing process consumes lots of time. 2) The details required in the drawing: If more details are required, it can be done much faster with CAD. 3) The number of repeated parts in the drawings: There is a feature of saving the repeated drawings in CAD software and they can be used in any other drawing without having to draw them again. There is also library feature in CAD software, where a number of readymade drawings of most frequently used components are available readily. Dassault Systèmes SolidWorks Corp.’s SolidWorks is employed as the 3D CAD software in this project because of the familiarity with the software. All of the individual parts for the turret mechanism were designed and assembled using the mentioned CAD software (see Appendix E). Before deciding on the final shape of the design 5 different hand sketches were done. The reason behind deciding on this form is its ease of production and maintenance

conditions. Figure VIII, Figure VII, Figure V and Figure VI shows the SolidWorks screenshots of the complete assembly. During the design process the drawings for the servos were taken from the part library which is supplied by the manufacturer company. The webcam is placed on the front side of the base in a way that laser pointer and vision sight of the camera is facing the same directionvi.

Figure VI

Figure V

Figure VII

Figure VIII

vi

The reason behind this orientation is going to be discussed in detail later in the following chapter.

2.1.2 Motor selection

Prior to going into the details of the selection process it should be clarified that servo motors are going to be utilized for the project. The benefits of using servos are listed below:  Linear relationship between the speed and electric control signal.  Steady state stability.  Wide range of speed control.  Linearity of mechanical characteristics throughout the entire speed range.  Low mechanical and electrical inertia.  Fast response. Before deciding on the motors, it is essential to know the inertial properties of the design. SolidWorks’ built-in function for calculating the mass and inertial properties is used in order to attain the required values. The selection was based on the following parametersvii.  Torque generated by motor.  Cost of motor.  Maximum rotational speed of motor. The detailed calculations on determining the motors can be found in Appendix B. Table 2, shows the results of the calculations which the motor selection process was based on.
Moment Of Inertia 0.0119445 kgm2 0.0117599 kgm2 Required Torque 2.2411 kgcm 2.2065 kgcm Required Speed 44.33rpm 44.33rpm

Base Motor Barrel Motor

Table 2 Desired properties for the motor selection process

Hitec RCD’s products were chosen because of their good cost-performance ratio. The model that provides the desired conditions is Hitec HS-322HD for both the base and barrel mechanisms. Table 3, contains some of the important specifications of the selected motors. All the specs of the motor is included in Appendix C.
HS-322HD 3.0 kg/cm 4.8V 0.19sec/60° = 5.51174978 rads/sec = 52.633 rpm

Torque Voltage Rotational Speed
Table 3 Motor specifications

vii

Note that while choosing the motors, power calculations should also be included but for the type of servo motors that are employed in this project power data is not available. Therefore power calculations are not included.

2.1.3. Performance Test for the Turret System

In order to assess the performance of the turret system the expected results are needed to be shown in this section. The desired function for the turret system is to track the pigeons which are recognized by the tracking system viii . In order to assess the system performance the following test is going to be done on the system. d The turret system will be placed in “d” meters away from a base-ball throwing mechanism. Such a mechanism is needed for Figure IX Performance test schematic for the turret this performance test since the speed of the thrown object has to be known. See Figure IX for the orientation of the devices. After the turret is locked onto its target the target will always stay in the centre of the horizontal axis since the camera and the laser sight is going to be facing to the same direction and their orientation is going to be coaxial which can be seen from the CAD drawings in the upper section. The mentioned concept 320 pixels 320 pixels is visualized in Figure X , since the resolution of the system 640 x 480 pixels one can say that the object will always stay in some Figure X The view from the system’s camera point of the 320th pixel in the horizontal axis if the system is working in the way it is supposed to work. This is going to be viii The recognition of the pigeons depends on the maximum speed that the tracking system is able to recognize which is calculated in Appendix B

the performance test variable for the system. In other words the ball is going to be thrown at various speeds (5m/s to 20 m/s) for each different distance value. Of course the distance values are going to be changed keeping the range specifications of the system in mind. After completing the test the resulting graphs are going to be plotted. The error values are going to be observed in order to do iterations on the system if necessary. See Figure XI for the sample graph. The anticipated performance test for the system is not conducted since the mentioned Figure XI Sample Performance Test Graph mechanism could not be acquired. Instead of doing the above experiment another model was constructed. The camera is placed on a table and a 5 kg mass is released from different heights in front of the camera from a distance of 1 meter. The speed of the mass is calculated using the basics physics formula below:

The model above allows the project to be tested in terms of the speed for object tracking. Results are shown below: Height 5 cm 10 cm 15 cm 20 cm 25 cm Calculated Speed Tracking Result 0.989 m/s Passed 1.400 m/s Passed 1.715 m/s Passed 1.979m/s Passed 2.213 m/s Failed However this gives a general idea about the performance of the system, it is not a fully sufficient performance test for the system. Because experiment the sequence should be applied for different distance (distances that are inside the range of the system) values but it is not possible to implement the above model for different distances because of the required space conditions.

2.1.4. Servo Control

A servo mechanism is an automatic system that uses system’s error as feedback. In theory, the feedback is used to control the physical position or other parameters of the motor. Basically one can define servos as smart motors. For example if you are using a DC motor to open/close your garage door, the motor will start and stop at the desired point using a sensor or an external button which means that it is not possible to tell how many degrees the motor will turn or control it. In the case of a servo, one can precisely define how many degrees the motor will turn at a desired speed. This feature makes servos a great choice for robotic applications. A servo is a closed-loop system with negative feedback.(18) Making the feedback positive will result in having an oscillating mechanism. So for the servo to operate properly, the feedback must always stay negative, otherwise the servo becomes unstable. The servo can almost become an oscillator, in which case it overshoots and rings following a rapid change at the input. There is one point in here that needs to be clarified in order to avoid confusion. A negative feedback means that the input and feedback signals are antiphase. For example if the input signal is a sinus wave, the output signal is also a sinus wave with a displacement of 180 degrees in phase. A great advantage of servo motors is that they are very easy to control in comparison with other types of motors (DC, Stepper) that require an H-Bridge or external circuitry to drive the motor and encoders for getting position data. The servo motor has all this circuitry internally. In the case of this project a servo controller board is employed to drive the servos. The specs of the controller board are included in Appendix C. The controller uses the serial port protocol to communicate with PC’s. In this application the terminal software of the controller board is used to manage the communication protocols on the serial port. In order to explain the system in more detailed fashion the following could be said. First and foremost the image data is processed by the image processing algorithm which is explained in detail in the following section and the x and y position data is extracted from the image in terms of pixels. The servo control algorithm gets the height and width values from the image which is 640x480 pixels in this case. In order to find the centre of the image the values are divided into halves. The x and y coordinates of the object is compared to the centre. The result gives the direction of movement on each axis for the servos. To compute the exact position displacement for the servos, the subtraction of the x or y coordinate from the centre is vectorially added to the previous position of the servo (0 to 255).The result divided into a iterated number which depends on the resolution of the image and the value obtained is send to the servos as the new position data. This sequence is repeated for each frame of the camera succeeding the camera to track an object. The code for the algorithm is included in Appendix D.

2.2. Image Processing and Target Tracking
Target tracking is an elaborate assignment since it employs a number of different techniques, sensors or tools working together. Amongst today’s applications Intel’s Open Source Computer Vision Library (OpenCV) is highly popular. For this project National Instrument’s LabVIEW software was employed at first because once getting familiar with the software there are toolboxes included specifically for machine vision and image processing. Although it is a very useful feature there are other aspects which makes using OpenCV a better option. Because of LabVIEW’s unique interface understanding the software is a complex issue. Also the documentation for the software is not easily available for ordinary users since it is commercial software. An academic copy of the software which has been purchased by BUTECH was acquired and after spending weeks on understanding LabVIEW the methodology is changed and OpenCV is employed for this project as a final solution. Before going into details of the image processing algorithm I would like to discuss the operating conditions of the system which are the minimum and maximum distances of the object to the system. The minimum distance, for the tracking system to recognize the pigeons, is explained in Appendix B. Hence calculating the maximum distance is not easily done without constructing a model. In order to calculate the maximum distance one
Figure XII Checker board

must define the minimum area, in sight of vision, for the tracking system to recognize the motion. Since this project specifically targets the pigeons, the section area of a pigeon’s body is going to be the area that is going to be used to calculate the minimum area (Average pigeon section area=20cm x 10cm). The minimum area which can be detected by an image processing application is 9 pixels.(19) However the problem is converting the mentioned 9 pixels into a measurable unit in order to define a maximum distance which the pigeons can be detected from.

Since pixels are relative units of measurement there is no mathematical way to convert pixels into centimetres or any other real unit. The only solution for the problem is setting up an experiment. The procedure is as follows. The webcam which is selected for this project is placed on an even floor from a known distance away from a checker board whose dimensions are also known. Each square in the checker board will correspond to some pixels. Since the areas of the squares are known in centimetres, calculating the number of pixels in an area corresponding to one of the squares will allow the conversion from pixels to centimetres to be done. As a result of this experiment However counting the pixels without an image processing Figure XIII Sketch showing the proportions between the distance of the object and the area covered in the field of vision tool is not possible. After doing the conversion of the mentioned 9 pixels into centimetre squares the maximum distance for the system to recognize the pigeons can be calculated. Figure XIII clearly shows that when the checker board is in position 1 it covers a length of 4 units out of 8, after the distance is doubled the length covered is 4 units out of 16. If this calculation was done in two dimensions (area calculation) instead of just one dimension than the result would’ve clearly shown that the change of the area covered by the object in the field of vision is inversely proportional to the square of its distance from the camera. This relation is going to be used while calculating the range for the system.

Figure XIV Showing the preparations of the experiment

After the experiment is done it became evident that for the camera used in this system in 640x480 resolution 1 cm corresponds to 8.5 pixels in 1 meter distance. From the constructed model above this data gives a max distance of 4.34613 meters as the system range. In order to be able to track an object the system needs distinctive features such as a specific shape or colour for the object. In the case of this project colour is chosen as the distinctive feature and the image processing algorithm is structured according to it. This technique is called object tracking based on colour. Before explaining the mechanics of the object tracking algorithm one must be aware of the fact that each pixel has a value in RGB (Red, Green and Blue) colour space defining the colour properties. The values differ from 0 to 255 for each colour for example to obtain white the values should be R=255, G=255, B=255. This property is used to identify the colour of a pixel and extract the desired colour from the background of the image. The operating mechanism of the software is explained below. First of all the RGB values are read from each pixel and recorded. Using the selected colour’s values, red (R=255, G=0, B=0) in our case, a threshold is set. The reason for setting the threshold is distinguishing the desired colour from the others. Basically the pixels which has the value greater than the threshold is equalized to the red’s value and the remaining ones which are under the threshold is equalized to black (R=0, G=0, B=0). This gives an image with the whole background in black and the object in red. Consequently the object is extracted from the background and the pixel locations can be calculated. The calculated locations are used to create a rectangle around these pixels and the geometrical centre of the rectangle is computed. This results in getting the x and y coordinates of the object in terms of pixels allowing us to locate the object in the image frame. The source code is included in Appendix D. In the case of more than one pigeon is in the field of view of the camera, processing algorithm focuses on the closest (biggest) one and ignores the others.

4. DISCUSSION
The performance test shows us that the system is not able to reach the flight speed of a pigeon which was aimed initially. The reason behind this fact can be explained by two reasons. First of all the inadequacy of the system to reach that speeds is not the mechanic system or wrong motor selection. The source of the problem is the narrow angle of sight of the camera and the time required for the image processing software to operate. There are two ways of avoiding this. One of them is using a better computer which has a stronger CPU than the existing one. For example if the servo control software is stopped and only the image processing part is run the processing speed increases significantly. Also the

lighting conditions effects the processing speed. The other solution might be using a different image processing algorithm which would be more system friendly. An infra-red thermal camera was intended to be used in the system in order to provide the ability of working in night conditions to the system but because of the cost of such cameras it is not considered. If it could have been possible the system would be much more accurate since the body temperature of the pigeons is 41 degrees a good distinctive feature in order to eliminate wrong target detections.

5. CONCLUSION
In order to successfully achieve the goal which is mentioned in the introduction part a mechanic pan & tilt system was constructed. Following this an image processing algorithm is designed and implemented using OpenCV. Using a servo controller the data gathered using the image processing algorithm is sent to two servos, giving the system the ability to track an object. All of the desired functions are tried to be implemented to the system. Although the system cannot reach the desired speed the current maximum speed of the design is also sufficient. Remembering the need for this system we can say that the pigeons will be stationary or moving much slower speeds than we aimed when they are landed. Therefore the system is available for the next enhancements which will be discussed in the future work section.

6.

FUTURE WORK

Most of the objectives of the project were achieved although some aspects can refined. The list below summarizes the most important ones:  Stereo vision can be employed for the system in order to get the distance data of the object for a better lock on to the target object. Image processing algorithm can be refined and different techniques can be employed. In order to provide mobility to the whole system, embedded systems such as MiniARM 2244 can be used instead of a PC for processing purposes. A wide angle lens can be mounted on the camera for a better angle of sight.







Appendix A
Physical Dimension Constraints
In order to grasp where the physical dimension constraints emerge from one should bear in mind that the goal of this capstone project is to design and manufacture a mechanism to keep the unwanted animals away from a specified area. Although the finished product can be implemented for a number of different uses, this project specifically focuses on pigeons because of the mentioned reasons in introduction. After underlining this fact to give an idea of how the design process was carried out, the design criteria could be listed as follows:  Mobility for usage conditions.  Providing the minimum required space for the components of the mechanism. (laser sight, motors and other required parts)  Ease of production and maintenance.  Production cost. To be able to meet the conditions defined above, the size of the design has to be calculated accordingly. The calculations below specify the upper limit for the area that the design will cover: Since the first criterion is mobility, the system has to have such dimensions that can be handled and moved easily by manpower. Keeping in mind that this appliance might be placed in areas that are hard to reach, the dimensions should exceed the limit for a person to be able to lift mechanism using one hand. To calculate such value, one should find the average arm span of a femaleix first. Since the height of a human equals to her/his arm span, (See Figure XV for the world famous piece Vitruvian Man by Da Vinci showing anatomic proportions of human body.) world’s female height average can be calculated. Say Lfemale_avg for the female’s height average value and imagine a female forming a circle with both arms wide spread in the front side of the body. The diameter of the circle could be calculated from the following formula:

Equating the diameter into the radius value in the equation below:

Figure XV Vitruvian Man by Da Vinci ix Any kind of engineering design should always consider the extreme conditions thus female arm span is selected for the calculations because the average female arm span is significantly shorter than an avarage male’s arm span.

Equation () gives the upper limit of the area covered by the device for a female to be able to carry it with both hands. As mentioned earlier, the mechanism should be able to be carried using one hand. Thus dividing the Acircle value by 2 will give the Amax value which is taken as the upper limit for the space that the mechanism will cover.

Calculations are as follows: x xi

The upper limit for the area that is going to be covered by the design is

x

xi

Netherlands has the world’s tallest height average for females with 1.693 meters. (22) India (rural regions) has the world’s shortest height average for females with 1.021 meters. (23)

Appendix B
Motor Selection for the Base
To be able to select the motors that will satisfy the desired conditions the following calculations are made:  Defining the ωmax for the base:

The flight speed of the pigeons range from 4 to 20 m/s(20), to calculate the needed speed for the turret mechanism, one must first define the minimum distance available for the tracking system to be able to track an object. See Figure XVI in order to get a clear idea of the mentioned minimum distance which is the radius of the circle in the figure. All of the calculations can be seen below:

Figure XVI Sketch Showing the vision sight of the camera and the calculated min distance for the object

Figure XVII Sketch showing the corresponding length of 1 degree in 1.075 meters away from centre

o

The minimum distance of the object to the camera: Speed of the pigeon: (Since the system will be working when the pigeons are about to land or already landed the speed is taken as follows)

The minimum time interval for the camera to detect an object: xii The minimum distance that the pigeon has to travel inside the vision sight of the camera in order for the camera to be able to detect motion:

To find the minimum distance of the object to the camera one must calculate the radius of the circle which has a 0.6 meter arc corresponding to an angle of 44xii degrees:

The radius of the arc is 1.075m. In order to calculate ωmax one must translate the speed of the pigeon to angular velocity. To do this a circle is drawn in SolidWorks with the calculated dimensions and a change of 1 degree is calculated in terms of meters. 1 degree corresponds to 0.0188 meters when the object is 1.075 meters away from the centre, see Figure XVII. If 0.0188 meters corresponds to 1 degree than 5 meters corresponds to 266 degrees. Assuming that the speed of a pigeon will be 5 m/s, the angular speed becomes 266 degrees/second. Final step for calculating ωmax is as follows:

xii

See Table 1 for the camera specifications.



Calculating the required torque for the base: Calculating the required acceleration:

The moment of inertia for the base system is calculated in SolidWorks which is mentioned before, in the article. The moment of inertia for the system is hence for the calculations a safety factor of n = 0.5 is added to the value obtained from SolidWorks in order to create room for iterations which might be needed when manufacturing the system.

The required torque is:

Motor Selection for the Barrel
The calculations for selecting a motor for the barrel system are pretty much the same calculations which are done above. Calculating the required torque and power:

gcm

Appendix C
Part Specifications

Appendix D
Source Code
#include #include #include void colordetect(IplImage* image); int main() { CvCapture* capture=cvCaptureFromCAM(CV_CAP_ANY); IplImage* frame=0; IplImage* hsv=0; int key=0; cvNamedWindow("Cam",CV_WINDOW_AUTOSIZE); frame=cvQueryFrame(capture); hsv=cvCreateImage(cvGetSize(frame),IPL_DEPTH_8U,3); while(key!='q') { frame=cvQueryFrame(capture); colordetect(frame); cvShowImage("Cam",frame); key=cvWaitKey(10); } cvReleaseCapture(&capture); cvReleaseImage(&frame); cvReleaseImage(&hsv); return 0; } void colordetect(IplImage* image) { IplImage* red=cvCreateImage(cvGetSize(image),IPL_DEPTH_8U,1); int i=0; int j=0; int width=image->width; int height=image->height; //image ait bilgiler bunları kullanarak pixel valu okuyacağız int step=image->widthStep; int channel=image->nChannels; uchar* data=(uchar*)image->imageData; //red int rstep=red->widthStep; int rchannel=red->nChannels; uchar* rdata=(uchar*)red->imageData; //draw int x1=image->width; int x2=0; int y1=image->height; int y2=0; //data[i*step+j*channel+2] red pixel value //data[i*step+j*channel+1] green pixel value //data[i*step+j*channel+0] blue pixel value

for(i=0;i(data[i*step+j*channel+0]+50)) { rdata[i*rstep+j*rchannel]=255; } else { rdata[i*rstep+j*rchannel]=0; } } } //çizdirme işlemi yapıyoruz for(i=0;i midx + range then pan = pan+(cogX-midx)/horizFactor end if cogY = GetVariable("COG_Y") if cogY < midy - range then tilt = tilt-(midy-cogY)/vertFactor elseif cogY > midy + range then tilt = tilt+(cogY-midy)/vertFactor end if if if if if pan > 250 then pan = 250 pan < 5 then pan = 5 tilt > 250 then tilt = 250 tilt < 5 then tilt = 5

SetVariable "PAN_SERVO", pan SetVariable "TILT_SERVO", tilt end if

Appendix E
Part Drawings

Bibliography
1. Simms, Eric. The public life of the street pigeon. London : Hutchinson , 1979 . p. 144. 2. Ground target tracking-a historical perspective . Chong, Chee-Yee, Garren, D. and Grayson, T.P. 2000, Aerospace Conference Proceedings, 2000 IEEE, Vol. 3, pp. 433-448. 3. Realtime Visual Tracking of Aircrafts. Mian, A.S. Canberra : s.n., 1-3 Dec. 2008. Computing: Techniques and Applications, 2008. DICTA '08.Digital Image. pp. 351-356. 4. An iterative image registration technique with an application to stereo vision. Kanade, B. Lucas and T. 1981. International Joint Conference on Artificial Intelligence. pp. 674–679. 5. Good features to track. Tomasi., J. Shi and C. 1994. IEEE Conference on Computer Vision and Pattern Recognition. pp. 593–600. 6. Ferreira, Gideon. Stereo Vision Based Target Tracking for a Gun Turret Utilizing Low Performance Components. Electrical and Electronic Engineering, University of Johannesburg. Johannesburg : s.n., 2006. Dissertation. 7. Regulation of the Street Pigeon in Basel. Haag-Wackernagel, Daniel. 2, Wildlife Society Bulletin, Vol. 23, pp. 256-260 . 8. Tracking Objects Using Image Disparities. Bray, A. J. 1, February 1990, Image Vision Computing, Vol. 8, pp. 4-9. 9. Three-Dimensional Object Recognition from Single Two-Dimesional Images. Lowe, D. G. 3, 2987, Artificial Intelligence, Vol. 31, pp. 355-395. 10. A Model and Tracking Algorithm for a Class of Video Targets. Vey, R. J. Schalkoff and E. S. Mc. 1, 1982, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vols. PAMI-4, pp. 2-10. 11. Motion Tracking with an Active Camera. Murray, Don and Basu, Anup. 5, May 1994, Vol. 16, pp. 449-459. 12. Weeks, A. Fundamentals of Electronic Image Processing. s.l. : Wiley-IEEE Press, 1996. p. 576. 13. Digital Image Enhancement and Noise Filtering by Use of Local Statistics. Lee, Jong-Sen. 2, March 1980, Pattern Analysis and Machine Intelligence,IEEE Transactions, Vols. PAMI-2, pp. 165-168. 14. Armin Gruen, Olaf Kuebler,Peggy Agouris. Automatic extraction of man-made objects from aerial and space images. Basel : Birkhäuser Verlag, 1995. 15. Kalman filtering in two dimensions. Woods, J. and Radewan, C. 4, July 1977, Information Theory, IEEE Transactions, Vol. 23, pp. 473- 482. 16. Welch G, Bishop G. An Introduction to the Kalman Filter. University of North Carolina. [Online] 24 Jul 2006. [Cited: 18 January 2011.] http://www.cs.unc.edu/~welch/media/pdf/kalman_intro.pdf. 17. Extending CAD tools and techniques. De Micheli, G. 1, January 1993, Computer, Vol. 26, pp. 8587.

18. ENGINEERING, YEDITEPE UNIVERSITY DEPARTMENT OF MECHANICAL. SERVO MOTOR SYSTEMS. s.l. : http://me.yeditepe.edu.tr/courses/me402/lab%20manuals/Servo_Motor_System.pdf. 19. Louban, Roman. Image Processing of Edge and Surface Defects: Theoretical Basis of Adaptive Algorithms with Numerous Practical Applications. s.l. : Springer, 2009. p. 168. 20. Flight kinematics of black-billed magpies and pigeons over a wide range of speeds. Tobalske, B. and Dial, K. 2, 1996, The Journal of Experimental Biology, Vol. 199, pp. 263-280. 21. N. Mohan, T.M. Undeland, W.P. Robbins. Power Electronics, Converters, Applications and Design. New York : John Wiley & Sons, 1995. 22. A Tutorial on Particle Filters for Online Nonlinear/Non-Gaussian Bayesian Tracking. M.S. Arulampalam, S. Maskell, N. Gordon, T. Clapp. 2, 2002, IEEE Transactions on Signal Processing, Vol. 50, pp. 2-3. 23. Zelfgerapporteerde Medische Consumptie. Gezondheid En Leefstijl. [Online] Central Bureau of Statistics , 16 March 2009. [Cited: 20 01 2011.] http://statline.cbs.nl/StatWeb/publication/?DM=SLEN&PA=03799ENG&D1=242-253,255266&D2=27-44&D3=0&D4=a&LA=EN&HDR=T&STB=G1,G2,G3&VW=T. 24. Diet and nutritional status of rural adolescents in India. Venkaiah, K., et al. 11, November 2002, European Journal of Clinical Nutrition, Vol. 56 , pp. 1119–25. 25. Soluble salt minerals from pigeon droppings as potential contributors to the decay of stone based Cultural Heritage. HERAS, M. G., et al. 3, May,June 2004, European Journal of Mineralogy, Vol. 16, pp. 505-509.