You are on page 1of 82

D

e
l
f
t
U
n
i
v
e
r
s
i
t
y
o
f
T
e
c
h
n
o
l
o
g
y
Bachelor Thesis
UAV Camera System for Object Searching and Tracking
Harald Homulle 4032578
Jrn Zimmerling 4047621
June 2012
Faculty of Electrical Engineering,
Mathematics and Computer Science
UAV Thesis (June 22, 2012)
This Bachelor thesis is made using L
A
T
E
X in Nimbus Roman 11 pt.
Preface
The third year of the Bachelor Electrical Engineering is completed with the bachelor thesis. With the
bachelor thesis students show their academic capability, and ability to design a electronic system in a
structured manner. At the TU Delft the bachelor thesis is embedded in the nal bachelors project where
students design and build a prototype for a client.
From this perspective Johan Melis and Thijs Durieux are our clients. The two master students from
Aerospace Engineering are setting up a competition for Unmanned Aerial Vehicles (UAV). The contes-
tants of the competition have to built a UAV with the ability to perform the subsequent tasks:
track a ground vehicle;
track an aerial vehicle;
y a track as fast as possible.
Our team, consisting of six Electrical Engineering students, developed an electronic system which per-
forms the tracking of a ground vehicle. At the same time a group of ten Aerospace Engineering students
designed and built a xed wing UAV that can y as fast as possible. Although our electronic system
was tested on a quadrocopter UAV, it was designed for the xed wing UAV developed at the Aerospace
faculty.
Within a time span of seven weeks, the goal of this project was to implement a tracking system on a
UAV with the ability to send a picture of the tracked vehicle to a ground station. Our team was split up
in three groups: a transmission and visualization, a position and control and a camera system group.
The authors of this thesis have developed and implemented the imaging part. A camera system was
developed which has the ability to detect an object and generate control signals for the UAV in order to
follow the object. Furthermore, formulas are deduced for the estimation of the location and velocity of
the ground vehicle.
The design choices and decisions regarding the camera system are described in this thesis . An
overview of the hardware and software of the camera system is given. The choices made in the design
trajectory are motivated and the performance of the implemented system is shown.
Harald Homulle & Jrn Zimmerling
Delft, June 2012
Acknowledgements
We would like to thank our supervisor Dr. Ir. C.J.M. Verhoeven for the nancial support and his enthused
way of supervising. We would like to acknowledge our clients B.Sc. J. Melis and B.Sc. T. Durieux
for their support and giving us the opportunity to participate in the UAV Project. The Microelectronics
Department deserves special thanks for the provision of budget and two quadrocopters. We would like to
express our thanks to M.Sc. M. van Dongen for his support on producing a PCB for the Toshiba Camera.
Ir. S. Engelen for the technical discussions and the support of a Leopard Imaging Camera Module. We
are grateful to M. de Vlieger for handling our orders and her help with various administrative tasks.
Furthermore we like to thank the Raspberry Pi foundation for their support with a Raspberry Pi. Thanks
also to our team members, whose team spirit, technical discussions and support were invaluable.
i
ii
Summary
An embedded electronic camera system for unmanned aerial vehicles (UAV) was developed in the scope
of the nal bachelor project at the TU Delft in 2012. The aim of the project was to develop a prototype
for a new UAV competition.
The electronic system of the prototype had to able to track an object, make a picture of it and send
it to a ground station, where the picture and ight data is visualized. To whole project is divided into
three subgroups each designing a specic part of the system. This thesis provides the design choices and
performance results regarding the image processing and object detection of the whole system.
It was considered that a literature study would usefully supplement the choice of hard- and software.
The literature study was extended by creating an overview of existing hardware for embedded video
processing.
To perform embedded image processing the hardware components were selected rst. It was decided
that the system consists out of a camera, a processing board and a power converter. As camera a webcam
was chosen, because it is robust, fast, small and it has the ability to buffer data, so no buffering on the
processing board was needed. The Beagleboard, an ARM based prototyping board, was picked as a pro-
cessing board, because it is the fastest of the compared boards. Furthermore it doesnt use much power, is
compatible with the hardware used by the other project groups and enough drivers for peripheral devices
are available. A buck converter was selected to down convert the voltage of a battery to the lower input
voltage needed by the Beagleboard, as it had the highest efciency of the considered converters.
The Beagleboard had to be capable of multiple sensor communication, real time image processing and
running location and velocity estimation algorithms. It was decided to run Linux on the Beagleboard,
because it allows high level programming.
For prototyping reasons, C++ was used to implement the algorithms, because it is easy to implement,
fast and can be handled well by the Beagleboard. The Open Computer Vision library was selected to
make the implementation of the readout of the webcam fast and the implementation of the algorithms
facile. Algorithms were implemented for colour detection, real location estimation of the object and
velocity estimation.
Different object detection approaches were considered and colour detection by thresholding each
pixel of the captured video frame, was selected as the most convenient algorithm, because it is the fastest
and least complex algorithm of the compared methods. The design brief laid explicit constraints on a
high frame rate such that colour thresholding was selected, as being the fastest algorithm.
The pixel coordinate of the object in the frame was found by a centre of mass calculation of the
selected pixels. A moving average lter was implemented to remove noise from the object detection
algorithm.
Trigonometric based algorithms were deduced to estimate the real location from the pixel coordi-
nates. The velocity of the object was found by the displacement of the object between two frames over
the time between those frames.
The designed prototype was tested in two different approaches; a test setup with a static camera and one
with a moving camera, mounted on a quadrocopter. In both congurations it was shown the built system
is capable of reliable object detection, location and velocity estimation.
iii
From the results of these experiments, it could be concluded that the system can achieve an accuracy
of up to 99 % in the static and 90 % in the moving camera setup for the estimation of both the location
and the velocity. Furthermore, around 14 frames per second on a resolution of 320 240 could be pro-
cessed using the Beagleboard and running all algorithms described above.
The prototype built within this bachelor project and documented in this thesis, is able to full the tasks
dened in the assignment. It therefore meets the specications regarding the image processing and
object detection dened in the design brief of the clients. Some suggestions for future improvements
of the system are testing it on the xed wing UAV and further enhancing and optimizing the various
algorithms.
Although the system is not ready for the UAV competition yet, the rst milestone on the way to a
UAV with camera system has been reached within this thesis. The work performed in the scope of this
thesis clearly contributed to the electronic system of the nal xed wing UAV demonstrating the task of
the new UAV competition.
iv
Contents
Preface i
Acknowledgments ii
Summary iii
List of Figures vii
List of Tables ix
List of Abbreviations xi
1 Introduction 1
2 UAV Camera Systems in Literature 3
2.1 Object Detection Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Stabilization of the Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Object Detection with a Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.4 Velocity and Motion Vector Calculation . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.5 Object Tracking with a Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 Build Setup & System Overview 7
3.1 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1.1 Design Brief . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1.2 Camera Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.1.3 Processing Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2 Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.1 Analogue (CMOS) Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.2 Digital (CMOS) Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.3 USB Webcam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3 Video Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3.1 CMUcam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.3.2 Leopardboard DM365 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.3.3 Beagleboard XM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3.4 Raspberry Pi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3.5 FPGA Spartan-3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.4 Selection of the Hardware Components . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4.1 Selection of the Processing Unit . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4.2 Selection of the Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.5 Power Converter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.6 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
v
4 Object Detection & Tracking Algorithms 19
4.1 Proposed Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.2 Colour Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2.1 RGB or HSV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2.2 Thresholding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.3 Stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.4 Position Estimation of the Detected Object . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.5 Velocity Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.6 Optical Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.7 Generation of steering signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.7.1 Quadrocopter Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.7.2 Fixed Wing Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.8 Implementation and Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.8.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.8.2 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5 System Results 31
5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.1.1 Static Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.1.2 Moving Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.2 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2.1 Statistical Experiment Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2.2 Static Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2.3 Moving Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.2.4 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.2.5 Image Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.2.6 Power and Weight . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.3 Experimental Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6 Conclusion 41
6.1 Evaluation of the Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6.2 Further Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Bibliography 45
A Design Brief 49
B System 51
C C++ Code 53
D Results 63
vi
List of Figures
3.1 Analogue CMOS camera 20B44P. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2 Digital CMOS camera Toshiba TCM8230MD & additional needed breakout board. . . . 10
3.3 Leopard digital camera module LI-VM34LP. . . . . . . . . . . . . . . . . . . . . . . . 11
3.4 Conrad mini webcam. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.5 CMUcam 4 (Board & Omnivision 9665 camera). . . . . . . . . . . . . . . . . . . . . . 12
3.6 Leopard board DM365. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.7 Beagleboard XM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.8 Raspberry Pi. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.9 Spartan-3 Develompent Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.10 Buck converter TPS5430 and AR.Drone battery. . . . . . . . . . . . . . . . . . . . . . . 17
3.11 Hardware setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.12 Overview of the full electronic UAV system. . . . . . . . . . . . . . . . . . . . . . . . . 18
4.1 Comparison of the RGB and HSV colour spaces. . . . . . . . . . . . . . . . . . . . . . 21
4.2 Example of a set of pixels that are checked by the algorithm. . . . . . . . . . . . . . . . 22
4.3 3D sketch used to estimate the location of the target. . . . . . . . . . . . . . . . . . . . 23
4.4 Top view of the sketch given in Figure 4.3. . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.5 Steering signals generation; quadrocopter approach. . . . . . . . . . . . . . . . . . . . . 27
4.6 Image from a demo video of a red car driving on a road. . . . . . . . . . . . . . . . . . . 28
4.7 Image from a demo video; colour detection. . . . . . . . . . . . . . . . . . . . . . . . . 28
4.8 Image from a demo video; optical ow. . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.1 Semi-Outdoor testing setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.2 Indoor testing setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.3 Moving camera testing setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.4 Two captured images from the semi-outdoor umbrella tracking experiment. . . . . . . . 34
5.5 Objects on the predened matrix as seen by the webcam. . . . . . . . . . . . . . . . . . 35
5.6 Indoor path of object. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.7 Indoor time line. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.8 Noisy and ltered pixel location in the static camera setup. . . . . . . . . . . . . . . . . 36
5.9 Path of the red cap in the moving camera experiment. . . . . . . . . . . . . . . . . . . . 37
B.1 Functional block diagram of the UAV system. . . . . . . . . . . . . . . . . . . . . . . . 51
D.1 Projection points of pixels under different resolutions. . . . . . . . . . . . . . . . . . . . 66
vii
viii
List of Tables
2.1 Overview of object detection / recognition techniques . . . . . . . . . . . . . . . . . . . 3
3.1 Overview of cameras. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Overview of processing boards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.3 Camera calibration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.4 Overview of power converters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.1 Overview of available object recognition algorithms. . . . . . . . . . . . . . . . . . . . 20
5.1 Performance overview on two test setups. . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.2 Performance overview for software resizing. . . . . . . . . . . . . . . . . . . . . . . . . 38
5.3 Performance overview for hardware resizing. . . . . . . . . . . . . . . . . . . . . . . . 38
5.4 Size overview for different image storage formats. . . . . . . . . . . . . . . . . . . . . . 39
5.5 Overview of the measured power consumption and weight of the system. . . . . . . . . . 39
6.1 The various system requirements that are achieved in the present prototype. . . . . . . . 41
D.1 Measurement results accompanying Figure 5.7. . . . . . . . . . . . . . . . . . . . . . . 63
ix
x
List of Abbreviations
AR.Drone Parrot AR.Drone quadrocopter (Prototyping)
ARM Advanced RISC Machine
B byte
b bit
BMP Bitmap
CCD Charge-Coupled Device
CMOS Complementary Metal Oxide Semiconductor
COCOA Moving Object Detection Framework
D/A Digital/Analogue conversion
DC Direct Current
FoV Field of View
FPGA Field-Programmable Gate Array
fps Frames per Second
GPIO General Purpose Input/Output port
GPS Global Positioning System
GPU Graphical processing unit
GV Ground Vehicle
HSV Hue, Saturation and Value
Hz Hertz
I
2
C Inter-Integrated Circuit
I/O Input/Output ports
IR Infra Red
JPEG Joint Photographic Experts Group
LDO Low-Dropout Regulator
LDR Laser Detection Radar
LVFG Lyapunov Vector Field Guidance
MODAT Moving Objects Detection and Tracking Framework
N.A. Not Applicable
OpenCV Open Computer Vision Library
OS Operating System
PCB Printed Circuit Board
PNG Portable Network Graphics
RGB Red, Green and Blue
RMS Root Mean Square
SD Secure Digital (Multimedia Flash Memory Card)
Sonar Sound Navigation and Ranging
TLD Track, Learn and Detect
TVFG Tangent Vector Field Guidance
UART Universal Asynchronous Receiver/Transmitter
UAV Unmanned Aerial Vehicle
USB Universal Serial Bus
V Voltage
xi
VHDL VHSIC Hardware Description Language
VPSS Video Processing Subsystem
W Watt
YUV Luminance and two Colour Chrominances
xii
Chapter 1
Introduction
Unmanned aerial vehicles, also known as drones, already perform a wide range of tasks in the military
sector [1]. These tasks include surveillance, territory exploration, reconnaissance and even attacking.
The history of many military inventions, like GPS or the internet, shows that the civil market often
adapts or enhances military products for civil use [2]. The same seems to happen with UAVs as the
growing number of non-military applications shows. Applications like wildre protection [3], dike or
pipeline surveillance or even lming are growing in the last years.
The newUAVcompetition is an initiative by Johan Melis and Thijs Durieux. The competition consists
of three tasks, rst the contestants have to show the speed and manoeuvrability of their UAV, second a
ground object has to be found in a eld and nally two UAVs will have to nd each other. To show the
achievability of these goals, a prototype for the competition is desired.
Two teams will be working on this prototype: ten students of Aerospace Engineering and six students
of Electrical Engineering. The aerospace team will design and produce a xed wing UAV
1
, the goal of
the electrical team is to develop the electronics of the system.
The electronic system of the developed UAV is roughly divided into three parts. One team deals with
the connection of the UAV to the ground station. Furthermore, they handle and visualize the information
exchanged between ground station and UAV. The second team works on the control of the UAV, the
sensors and the merging of the subsystems. The last team, formed by the authors of this thesis, develops
a camera system that recognizes and tracks objects. The electronic system cannot be tested on the xed
wing UAV of the Aerospace team and will therefore be tested on a quadrocopter
2
AR.Drone.
It is the aim of this thesis to describe the design and implementation process of the camera system and
the design choices made.
An overview of existing object detection and tracking methods is given to investigate which algo-
rithm, camera and processing unit are most suitable for a UAV. Thereafter the system specications are
given as well as the experimental performance of the camera system. Furthermore the chosen algorithm
used to detect and track objects is explained.
The requirements for the whole system are listed in the design brief in Appendix A. From those
requirements, a ow chart for the various tasks is gathered in Appendix B, where all tasks of the nal
system are shown. According to Appendix B the aims of the camera subproject are
to record a video in real time;
to adapt the video for further processing on an embedded platform;
to detect a ground vehicle;
to estimate the location of the ground vehicle;
and to estimate the velocity of the ground vehicle.
1
A xed wing UAV is essentially a small unmanned air plane with wings rather than rotors.
2
A quadrocopter UAV is a helicopter-like aircraft with four horizontal rotors, often arranged in a square layout.
1
2
The studies of UAV systems were usefully supplemented with a literature study of related elds. A lot of
research has been carried out on these subjects. In order for the raw camera data to be processable with
common object detection and tracking algorithms, video stabilization may be required.
Therefore a wide range of stabilization algorithms, which differ in complexity and accuracy, have
been proposed by various research groups [47].
Object recognition for UAVs can be either based on colour [810] or based on motion [6, 11, 12].
Stabilization can also be done after the object is detected by implementing a moving average lter [13]
on the detection data.
The estimation of the motion of objects can be calculated by optical ow methods [14], so the motion
vector of an object can be extracted from a video stream.
All papers report computational limitations of their algorithms due to the embedded system and re-
quirement of real time data analysis, therefore special attention will be paid to the speed and complexity
of the algorithms used in this thesis.
This thesis is organized as follows. In chapter 2 the current technological level of all related elds is
discussed. Design choices are made in chapter 3 regarding a camera, a processing board and a power
supply. The overviewof the camera systemand the total systemis also given in that chapter. The software
and algorithms are reported in chapter 4; formulas for the calculation of the position and velocity of the
object are deduced. The performance of the camera system is given in chapter 5. The conclusions of this
thesis and some suggestions for further work are provided in chapter 6.
UAV Thesis UAV Camera System
Chapter 2
UAV Camera Systems in Literature
In this chapter a brief overview of the current technical level with respect to related elds is given. By
analysing the literature and research performed on the eld of UAV camera systems and image process-
ing, this chapter seeks to point out the state of the art. Using the overview given in this literature study,
design, implementation and algorithm choices can be made in the following chapters.
In section 2.1 a discussion of various object detection methods shows that object detection with a
camera is desirable for the goals of this project. Subsequently section 2.2 gives an overview of software
and hardware methods in order to stabilize camera pictures. Several camera object detection algorithms
are a topic of discussion in section 2.3. Section 2.4 discusses techniques to extract the motion vector of
an object from a video stream. Finally, section 2.5 shows different options to track an object once its
location and motion are known.
2.1 Object Detection Methods
Depending on their application, camera, sonar, or laser systems are used to detect objects. Differences
between the named methods can be found in Table 2.1 where these methods are listed and compared.
The size and price range from small and cheap solutions up to large and expensive ones.
Table 2.1: Overview of object detection / recognition techniques
Attribute Camera Camera Sonar [1517] LDR [18]
Colour IR
Dimensions Two dimensional Two dimensional Three dimensional Three dimensional
Information Three colour layers One heat layer Shape / Distance Shape / Distance
Detection basis Colour / Motion Heat Shape Shape
Size Very small Very small Large Large
Weight Very small Very small Large Large
High Data Rate Low Resolution Costly
Disadvantages Good Contrast Needed High heat contrast needed Range 20 m Mainly Distance Acquaintance
Ambient Light Needed Mainly used under water Large and heavy
Advantages High Detail Works at night Works at night High resolution
It can be seen that camera systems have the advantage of being very small, cheap and having a high
resolution. However camera systems produce a much higher data rate than needed for object detection
and a colour contrast is needed for reliable detection.
For Sonar (Sound Navigation and Radar) and LDR (Laser Detection Radar) systems, the detection is
based on the shape of the object. Sonar detection is done with sound and is normally used underwater,
LDR is based on the same principle, but uses lasers instead of sound. The reection of either the sound
or light waves on objects is used to create a 3D shape view of the area of interest. If the exact shape of
the object is known, this shape can then be detected on the 3D shape view.
3
4 2.2. STABILIZATION OF THE CAMERA
The resolution of these systems is related to the used wavelength in both applications. The wavelength
of light waves used for LDR is much smaller than the wavelength of acoustic waves used for Sonar so
the resolution of LDR is much higher. However both systems are quite large and more expensive than
camera systems.
As described in the design brief (in Appendix A) a camera is required. Therefore optical object
recognition based on cameras is studied in more detail in the following sections.
2.2 Stabilization of the Camera
The camera can be stabilized both in hardware and in software. A (more) stable image results in better
tracking results and thus a higher accuracy of the algorithm. The hardware approach will increase the
weight of the system, but will result in a stable image. The software approach will increase the processing
power needed and results are mostly worse than using hardware to control the motion.
Li and Ding [7] suggest the use of servos to control and minimize the camera movement. Stabilization
inside the camera can be performed using gyroscopes to measure camera movement and shift the sensor
in the opposite direction to balance the movement [4].
In order to stabilize a video in software the movement of the camera is computed by comparing mul-
tiple frames with each other. The video is stabilized by shifting the frames to correct for the computed
movement. The approaches found in literature differ by the algorithm to estimate the camera move-
ment. [19] suggests to estimate the camera movement by a spacio-temporal approach, where blocks are
matched in subsequent frames.
Another approach implemented in the COCOA framework for UAV object tracking proposed by [5,6]
is Ego Motion Compensation. [5, 6] combine a feature based approach similar to [19] with a gradient
based approach as proposed by [20]. The algorithm tends to adept itself during the process and learns to
discriminate shapes, stabilization is based on shifting the image according to where the shape is detected.
A combination of hardware and software video stabilization is described by [4]. Instead of estimating
the movement with an algorithm, the movement is measured with gyroscopes. The frame correction is
similar to software approaches and is based on image shifting.
A total different approach is not stabilizing the camera or the camera sensor, but stabilizing the mea-
surement data. This can be done by implementing a moving average lter as described by [13] and
weighting the previously measured values with a certain factor to establish a weighted average.
2.3 Object Detection with a Camera
In general two types of object recognition are described in literature: Colour recognition [810] and
Movement recognition [6, 11, 12]. Colour recognition is based on the detection of particular colour bulbs
of the object. For recognition based on colour [8], the processing is quite easy.
In each single frame the pixels with Red, Green and Blue (RGB) components are checked for a
particular colour condition. After this colour thresholding the centre of mass of all pixels satisfying the
condition is calculated. This algorithm is quite fast and needs just a few calculations per pixel. Similar
is detection based on a certain brightness threshold, a black and white image is used in this method. A
bright object can be tracked by thresholding the luminance of the pixels [9].
Depending on the tracked object, converting the video to other colour spaces, such as Hue, Saturation
and Value (HSV) or Luminance and two Chrominances for the colours (YUV) , can be benecial for
the reliability of the algorithm. This technique can be implemented on a FPGA for fast parallel process-
ing [10].
To detect moving objects in a video stream, subsequent frames are compared to each other in order to
detect movement. This requires more processing power since multiple frames are correlated to each
other whereas colour recognition can be performed by thresholding single frames.
Recognition based on motion is more complex than recognition based on colour [11]. Several frames
UAV Thesis UAV Camera System
CHAPTER 2. UAV CAMERA SYSTEMS IN LITERATURE 5
are buffered to extract the motion of objects within those frames. Assuming the targeted object is the
moving object in the processed frames results in object recognition. On steady cameras, a reference
frame is stored and pixels are checked for new objects appearing in the scene [12]. On moving cameras
(like on UAVs) this is even more difcult, since the camera itself is already moving.
The Moving Object Detection and Tracking (MODAT) framework stitches subsequent camera frames
to a map, on which motion is detected using a Gaussian Mixture Learning technique. This technique
learns to discriminate background movements from the real target objects. A fast parallel processing unit
is needed to calculate the motion in real time [6].
In both methods the camera and processing adds noise to the image, which leads to the detection of
undesired objects (noise). Various noise reduction methods have been suggested in literature. Noise
reduction can be achieved by applying some blur to the image, which removes detail and noise. Similar
a median lter can remove variations between pixels in the image and eliminate noise as well as huge
differences [6, 9]. A method of noise compensation used after object detection is called binary erosion,
which checks the pixels and their neighbours for connectivity. Assuming the object consist of more than
one pixel, only connected pixels are recognized as an object [10].
2.4 Velocity and Motion Vector Calculation
Besides the location of the object its velocity is needed, in order to track an object. The estimation of the
objects location is discussed in section 2.3, whereas the estimation of the velocity and motion is covered
in this section.
The calculation of a motion vector eld from a video stream is called optical ow. The method is
based on the assumption that the changes in two subsequent image frames are small and the colour of the
object stays the same. Now the displacement of each pixel in two subsequent frames is calculated. The
resulting vector eld contains the direction and magnitude of movement of each pixel. On the UAV used
in this project, the camera itself moves, thus the motion vector of the object is given by the difference
between the background and object motion.
In literature many different ways of calculating the motion vector eld on video streams are discussed.
[14] discusses an optical ow algorithm to estimate the motion of objects. The proposed algorithm is
focused on the motion of edges since the error rate of common optical ow algorithms is quite high at
the edges of objects.
The calculation of the motion eld of a wavelet transformed video is discussed by Lui et al. [21]
and [22]. The wavelet transform of an image consists of the contour lines of an image. By calculating
the movement of the contour lines of objects it is easier to assign a single motion vector to an object and
to separate different objects.
Another accurate but computational intense method of motion estimation is proposed by Farnebck
[23]. This algorithm is based on intense tensor computations and is not suited for embedded solutions.
However Farnebck suggests to only calculate the displacement of pixels in the near neighbourhood of
the object to save computation time.
Overall the methods to calculate the motion of objects discussed in literature are computationally intense.
The development of motion detection algorithms suitable for embedded platforms is beyond the scope
of this thesis. So only simple optical ow methods have been studied within the scope of this thesis.
2.5 Object Tracking with a Camera
Object detection over multiple frames can lead to object tracking and is necessary for autonomous UAVs.
The generation of steering signals from a captured video in order to track an object is a complex task,
especially for large velocity differences between the object and UAV. The camera system has to calculate
UAV Camera System UAV Thesis
6 2.5. OBJECT TRACKING WITH A CAMERA
the movement of the tracked object in order to y in its direction.
[7, 9, 24] propose the use of servomotors to change the position of the camera relative to the UAV.
This minimizes the chance of losing the tracked object.
[24] describes the main two algorithms for path planning with UAVs: Tangent vector eld guidance
(TVFG) and Lyapunov vector eld guidance (LVFG). According to Chen et al. [24] a combination of
both algorithms is desired for UAVs with larger turning limit circles than the tracked object.
Common UAV autopilots, such as the Paparazzi project [25], implement such path planning algo-
rithms. By supplying GPS coordinates and object speed to the autopilot, it is able to delineate a path
to the specied location. Even complex tasks, like ying in an eight shape above a location, can be
performed by Paparazzi.
UAV Thesis UAV Camera System
Chapter 3
Build Setup & System Overview
The literature study in chapter 2 points out that the accuracy and reliability of real time object detection
and motion estimation is mainly limited by the computational power of the processing unit. It might
therefore not be unexpected that the choice of hardware components and especially the core processing
unit of the system is one of the key points of this thesis. The algorithms presented in chapter 4 are directly
affected by the hardware setup presented in the following sections.
In this chapter an overview of the hardware setup and the overall system is given. Section 3.1 points
out the system requirements as dened in the design brief. Based on these requirements several cameras
and processing units are analysed. The hardware conguration is explained in section 3.2 for the camera
and in section 3.3 for the processing unit. Both sections give an overview of existing hardware in order
to rate each of them with respect to the system requirements. The hardware choices made are motivated
in section 3.4 The selection of the power converter is discussed in section 3.5. In section 3.6 an overview
of the nal system is provided.
3.1 System Requirements
In this section the requirements regarding the camera system are explained. First the explicit require-
ments as noted in the design brief (Appendix A) are listed. Thereafter requirements from a technical
point of view based on ndings in the literature study are stated.
3.1.1 Design Brief
In the design brief the following requirements regarding both the camera and the processing unit are
listed:
Speed
The speed of the system is high enough to track vehicles up to 55 km/h at a maximum UAV
speed of 100 km/h;
The speed of the image processing reaches a minimum amount of at least 15 frames per
second (fps) , implying that a picture is taken at least every 2 m.
Object Tracking
The object can be detected and followed from at least 50 m.
Power
The system can y one hour on a separate battery;
Therefore it was decided that the systems power consumption is below 5 W .
Weight
The maximum payload of the quadrocopter UAV is 100 g;
The maximum payload of the xed wing UAV is 250 g;
The power source is excluded from this weight.
7
8 3.1. SYSTEM REQUIREMENTS
Finalizing
The system is documented in a manner which allows other persons to understand and expand
the system;
The system has enough computational power and free I/O ports in order to extend the system
after delivery.
Test platform
As a test platform the quadrocopter AR.Drone was used. Although the system is designed for a xed
wing UAV, a quadrocopter is used for several reasons. First of all the destination platform, a xed wing
UAV, is developed at the aerospace faculty simultaneously with the camera system, such that the xed
wing UAV cannot be used as a test platform. Since a quadrocopter is able to hover, it can be used indoor
in contrast to a xed wing UAV which needs to move forward to take off. This allows weather and wind
independent indoor testing. Finally, in the case the control over a quadrocopter is lost, it simply keeps
hovering or crashes down. Whereas a xed wing UAV becomes a projectile it is more likely for people
to get hurt or for the UAV to be damaged. Because of these reasons a quadrocopter is used as testing
platform.
3.1.2 Camera Requirements
Besides the requirements mentioned in the design brief, extra requirements derived from literature, the
available timespan and the framework of the project are given:
Usable resolution;
Available drivers for a camera;
Field of View (FoV) < 45

;
Auto colour / light adjustment;
Maximum price of e250 (Processing board, camera and power converter);
Delivery in less than one week.
In literature the difculty of processing large resolutions in real time on embedded devices is described.
Therefore only cameras with a maximum resolution of around 640 480 are examined.
In the context of this thesis real time is dened as a soft real time criterion. A system is a soft real
time system in the case the result of an algorithm loses its value after a certain deadline. In this context
the term real time is interpreted as directly processing the image data before capturing a new frame. Thus
the location of the target object is updated in the speed of the frame rate. A lower frame rate or delays
in the algorithm result in a lower quality of service and accuracy of the system. But the system can still
operate with a low frame rate, so that the real time criterion is not a hard criterion.
The eld of view should be in the range 30-45

. Lower than 30

results in a narrow FoV, so that the


UAV should y high before being able to oversee a large area. Higher than 45

results in a wide FoV,


which results in large areas when ying on a normal high. Therefore the camera FoV needs to be in the
specied range.
Since the system is used indoor as well as outdoor, auto colour and light adjustment is desired to
prevent the image from getting dark or bright.
Because of the limited time the availability of drivers is an important prerequisite. Furthermore, the
choice of products is limited to those deliverable in less than one week in view of the short project scope.
3.1.3 Processing Requirements
The processing takes place on an embedded device. This device has to be capable of handling the
video data stream, recognizing and tracking the objects, controlling the UAV and processing sensor
communication. Since the whole project is divided over three groups the processing board has to be
compatible with the hardware choices made by other groups.
UAV Thesis UAV Camera System
CHAPTER 3. BUILD SETUP & SYSTEM OVERVIEW 9
The wireless transmission and visualization group demanded that the processing board can commu-
nicate via UART with a XBee for wireless data transmission to the ground station.
The position and control group of the UAV project requires that the processing unit can communicate
via UART with a barometer, via I
2
C with a GPS sensor and with the AR.Drone via its the serial UART
port. Thus the processing board needs to support UART and I
2
C communication via multiple I/O ports.
As the system is still in development after this bachelor thesis the processing board should still have
free I/O ports which leave room for expansion at the delivery of the system. The processing power should
be adequate to do those calculations in real time: minimum 15 fps encoding/decoding.
3.2 Camera
As mentioned above a lightweight, easy interfaceable camera is desired. The available camera types are
Mono colour cameras;
Full colour cameras;
Infra red (IR) cameras.
Mono colour and IR have the advantage of having only one data layer, instead of three for full colour.
This results in a decrease in processing time for mono colour and IR cameras, since less data has to be
handled.
Colour has the advantage of containing more information, which makes object extraction based on
colour possible. In literature colour cameras and colour recognition techniques are most common.
IR cameras only provide information about the heat of an object. No IR camera is used, since the
heat of the tracked object is not dened and it cannot be assumed that an IR contrast of the object and
background is given.
The use of a full colour camera allows a more reliable detection, than a mono colour camera. Al-
though the use of a full colour camera results in a higher data rate, the system will benet from a more
accurate and reliable object detection. Because of this a full colour camera is used.
The availability of colour cameras is abundant. A selection of cameras that satises the requirements
mentioned above are listed and explained below. An overview of the camera modules is given in Ta-
ble 3.1. In total 5 cameras were compared, three of them were a digital camera, one analogue camera
and one webcam.
Table 3.1: Overview of cameras.
Attribute TCM8230MD [26] LI-VM34LP [27] Omnivision 9665 [28] 20B44P [29] Conrad Mini Webcam
Delivery speed One week One day Two weeks (with CMUcam) One week One day
Open source software No No Yes No No
Drivers available No Yes Yes No Yes
Resolution 698 502 752 480 1304 1036 640 480 640 480
Frame rate 30 fps 60 fps 30 fps 25 fps 30 fps
Data type Digital Digital Digital Analogue Digital
Data output Parallel Parallel Parallel - Serial
Output format RGB or YUV RGB or YUV RGB or YUV CVBS RGB
Size 6 6 4.5 mm 10 10 mm 4 5 mm 171712 mm 20 20 mm
Power 120 mW 200 mW 80 mW 550 mW 1.5 W
Costs e7.50 e85 e89 e89 e15
Features
Auto luminance control
Sub sampling
Luminance control (via I
2
C)
Sub sampling
Automatic white balance
Automatic luminance control
Back light compensation
Ultra wide vibration spec.
Auto luminance & colour control
Sub sampling
Advantage
Cheap, Small
Not over dimensioned
Drivers
High frame rate
High frame rate
Compatible with processing units
-
Easy to implement
Robust
Disadvantage
No breakout board
No drivers
Expensive
No auto luminance control
Not available without CMUcam
Long delivery
A/D conversion needed
No sub sampling
Computationally intense
UAV Camera System UAV Thesis
10 3.2. CAMERA
3.2.1 Analogue (CMOS) Camera
Almost all cameras are based on a CCD (Charge-Coupled Device) or CMOS (Complementary Metal
Oxide Semiconductor) sensor. Both devices produce digital images, where CMOS is the cheaper tech-
nique [30]. Although both sensors produce digital videos, most CMOS cameras available convert the
video on chip to a composite video format . This format is an analogue video standard, where the com-
plete video stream is modulated on one wave form. It includes the three colours, the clock frequency and
offset data.
For (colour) object recognition techniques a digital image is required. An A/D video converter either
as a standalone device, or integrated on a microprocessor is needed to deliver digital pictures with an
analogue camera. An analogue CMOS camera is shown in Figure 3.1.
Figure 3.1: Analogue CMOS camera 20B44P [29].
3.2.2 Digital (CMOS) Camera
Cameras that lack the D/A conversion of an analogue CMOS camera directly offer a parallel digital
output. Those cameras are also congurable via a I
2
C bus, where the output resolution and colour format
can be set. The digital outputs of the camera can be directly connected to a microprocessor for further
image processing and object recognition.
The advantages over the analogue method are the lack of an extra needed (standalone) A/D converter
and an adjustable data output format (I
2
C). In Figure 3.2(a) a camera of about e7.50 is shown [31].
This is a very small digital camera of 6 6 mm and weights less than a gram [26]. The problem with
this camera is the absence of a prefabricated breakout board. Such a breakout board [32] is depicted in
Figure 3.2(b).
(a) Toshiba TCM8230MD Camera (b) Toshiba TCM8230MD Breakout
Board
Figure 3.2: Digital CMOS camera Toshiba TCM8230MD [26] & additional needed breakout board [32].
There are also more expensive, prefabricated boards with a camera already attached to them. Those
boards are ready to be used and no manual work is needed. Such a board with a camera similar to the
above mentioned Toshiba camera costs e85 [33] and is made by Leopard Imaging [27]. In Figure 3.3
an example of an out of the box camera module is given.
The main advantage over the Toshiba camera is that no breakout board is needed. The camera has an
interface that can be attached to the camera header of a Leopard Imaging Board and a Beagleboard, both
processing boards are described in the following section.
UAV Thesis UAV Camera System
CHAPTER 3. BUILD SETUP & SYSTEM OVERVIEW 11
Figure 3.3: Leopard digital camera module LI-VM34LP [27].
3.2.3 USB Webcam
A USB webcam is almost the same as a digital camera, but it is more robust because of the case its in.
Furthermore it has its own small microprocessor that adapts the video data for transmission over USB.
Besides the advantage of general available drivers and support, USB webcams transmit the video in a
serial format rather than parallel like a digital CMOS camera.
Another difference is that the data is normally compressed, whereas the data of the parallel cameras
is uncompressed. Normally, if the read out speed of both cameras is equal, a digital CMOS camera
captures more video frames per second than a USB camera. Figure 3.4 shows an image of the Conrad
mini webcam. This webcam is small and only weights 25 g which is less than other webcams available
on the consumer market.
Figure 3.4: Conrad mini webcam.
3.3 Video Processing
Since the system is doing real time (video) processing, the processing should take place on the UAV.
Therefore an onboard microprocessor is needed to run those applications.
Furthermore the total system has to communicate with a GPS-sensor, a barometer, a XBee and the
AR.Drone itself. From the design requirements perspective the board should have enough I/O ports to
be easy expandable. It might therefore not be unexpected that the processing board needs multiple USB
-ports and GPIOs (General Purpose Input/Output) , which are also accessible as UART or I
2
C ports.
For both setups, either with analogue or digital cameras, the microprocessor has to process the digital
(or digitalized) video data. Not all microprocessors are suitable for the purpose of video processing. In
Table 3.2 an overview of the available video processing boards described below is given.
UAV Camera System UAV Thesis
12 3.3. VIDEO PROCESSING
Table 3.2: Overview of processing boards.
Attribute CMUcam 4 [28] Leopardboard 365 [27] BeagleBoard XM [34] Raspberry Pi [35] SPARTAN-3 [36]
Delivery speed One week One day One day On Request One Day
Price (excl. BTW) e90 (incl. camera) e100 e160 e35 e120
Open source Yes No Yes Yes N.A.
Processor Parallax P8X32A ARM9 26EJ-S ARM Cortex A8 ARM11 76JZF-S XC3S200 FPGA, 200K Gates
Generation - ARMv5 ARMv7 ARMv6 -
Speed 80 MHz 300 MHz 1 GHz 700 MHz 50MHz
RAM Not specied 128 MB 512 MB 256 MB 1 MB
Memory Slot SD SD SD SD N.A.
Power Consumption < 2.5 W < 2 W < 2 W < 4 W Not specied
Voltage 5 V 5 V 5 V 5 V 5 V
Size Not specied 76 64 mm 83 83 mm 86 54 mm Not specied
Parallel Camera Port Yes Yes Yes Yes

Yes
Weight Not specied 54 g 74 g 44 g Not specied
Advantage
Cheap
Low power
Easy interface with LI-camera
Fast GPU
Camera interface
Light
Cheap
Fast parallel processing
Optimal use of Hardware
Disadvantage Slow processor Not enough I/O ports
Heavy
Expensive
Unknown delivery time
Slow prototyping
No drivers for peripherals

= No camera drivers available.


3.3.1 CMUcam
There are some products that combine both a camera and microprocessor. They provide a camera and a
circuitry board with a microprocessor that can be ashed to run a specied code (for example an object
recognition code). The CMUcam [28] is an example of such an all in one solution. The software that can
be ashed on the microprocessor is open source and can be adapted to run an object recognition code.
The microprocessor however is slow compared to other boards and can only analyse 10 fps at a low
resolution (160 120 pixels). Also the storage capacity is small, so the possibility to buffer data is
limited. It is also not suitable for further needs like place estimation and velocity calculations.
Figure 3.5: CMUcam 4 (Board & Omnivision 9665 camera) [37].
3.3.2 Leopardboard DM365
The company Leopard Imaging develops a system similar to the CMUcam, however it is not open source.
Various Leopard Boards are available with a (more advanced) microprocessor on it, namely an ARM
processor. Besides that, they also have a special Video Processing Subsystem (VPSS) , which will be
able to handle (a part of) the video processing [27]. The Leopardboard 365 (Figure 3.6) is one example
of the available boards.
The board has a parallel video port that is compatible with Leopard Imaging camera modules. The
microprocessor is fast enough to process 50 million pixels per second, which would be more than ade-
UAV Thesis UAV Camera System
CHAPTER 3. BUILD SETUP & SYSTEM OVERVIEW 13
quate for the purpose of the project.
The microprocessor has the ability to run the place and velocity calculation besides the recognition
code. However, the board only has a single USB and no UART port, which makes the board unsuitable
for communication with multiple sensors and a XBee.
Figure 3.6: Leopard board DM365 [27].
3.3.3 Beagleboard XM
The Beagleboard XM is a prototyping board, widely used by hobbyists. The board is based on the ARM
Cortex A8 processor which runs at 1 GHz. From the boards discussed in this thesis this board has the
most advanced processing unit. Furthermore a wide range of GPIOs and four USB ports are available on
the board. This makes the board suitable as main processing board for the whole UAV (camera) system.
The Beagleboard XM [34] is shown in Figure 3.7. A general purpose microprocessor is the core
of this board, a Linux OS can be run on the board for easy prototyping. The board is also compatible
with the Leopard Imaging Modules via the camera header. Although there are drivers for some Leopard
camera modules available, only a small range of Linux kernels support Leopard Imaging cameras.
Compared to the other processing boards the Beagleboard has the disadvantage of being quite heavy
(74 g). The nal product has to have a weight of less than 250 g, but the AR.Drone which is used as
testing platform, as a prove of concept, has a maximum payload of 100 g. This makes the Beagleboard
unsuitable for the prove of concept on the AR.Drone, but suitable for the product targeted in the design
brief.
Figure 3.7: Beagleboard XM.
UAV Camera System UAV Thesis
14 3.3. VIDEO PROCESSING
3.3.4 Raspberry Pi
Similar to the Beagleboard, the Raspberry Pi [35] is an ARM based processing board. However it has an
older version of the ARM processor. The main advantage over the Beagleboard is its smaller size, weight
and lower price. The main disadvantage is its slower processor and its long delivery times. Specications
are listed in [35]. The Raspberry Pi is shown in Figure 3.8.
Figure 3.8: Raspberry Pi [38].
3.3.5 FPGA Spartan-3
In contrast to the other processing boards presented in this section the Spartan-3 is not a microprocessor
but a FPGA. A FPGA is taken into account as Price et al. [10] report a signicant speed up of vision
algorithms on FPGAs. Images can be processed parallel on a FPGA rather than subsequently as on a
microprocessor. An image can be processed in parallel and therefore a FPGA can process an image in
fewer clock cycles than a microprocessor. However the clock speed of the Spartan-3 is twenty times
lower than the one of the Beagleboard. On a FPGA the hardware can be congured dedicated for a
specic problem.
The FPGA Spartan-3 Starter Kit [36] is shown in Figure 3.9. The board has three 40-pin expansion
connectors and a serial port. Furthermore a range of leds, switches and buttons are provided to simplify
the prototyping process.
The documentation does not contain any information on the size or weight of the board. This is a
disadvantage since especially the weight is an important issue for this project. Furthermore two other
subgroups of the project work in high level programming languages which are hard to map to a FPGA
platform. Furthermore the drivers for cameras and sensors are not provided in VHDL format, the primary
language to congure FPGAs. Therefore a microprocessor which handles sensor communication would
be needed in cooperation with the FPGA in the case a FPGA is chosen.
Figure 3.9: Spartan-3 Develompent Board [39].
UAV Thesis UAV Camera System
CHAPTER 3. BUILD SETUP & SYSTEM OVERVIEW 15
3.4 Selection of the Hardware Components
In this section the design choices, regarding the hardware setup presented in section 3.6, are motivated
and presented. In the rst part subsection 3.4.1 the selection of the processing board is discussed. In
subsection 3.4.2 the camera choice is motivated.
3.4.1 Selection of the Processing Unit
The choice for a processing board was made based on the previous section and Table 3.2.
The CMUcam4 is not suited for the camera system, since the processor is just fast enough for object
recognition. However the processing board is not only responsible for object recognition, but has to read
out all sensors, send data via a data link and estimate the location of the target. The CMUcam4 cannot
handle this amount of processing steps.
The Leopardboard DM365 does not meet the requirements either. The number of I/O ports and speed
of the processor is much smaller compared to the Beagleboard and the Raspberry Pi. The Leopardboard
not only runs at a slower clock speed than the Beagleboard and the Raspberry Pi, but also uses an obsolete
processor.
Although the delivery times for a Raspberry Pi on the consumer market are above 3 months, the ofcer
for education of the Raspberry Pi foundation was persuaded this project deserves a Raspberry Pi via a
letter of the authors. Because of the available I/O ports and its low weight the Raspberry Pi is suitable
for this project. However the Raspberry Pis processor performance is inferior to the performance of
the Beagleboard. Furthermore the Raspberry Pis power consumption is twice as high as the one of the
Beagleboard, according to its specications. It is therefore not used in the prototype.
Although being an elegant and fast solution, the FPGA Spartan-3 is not used in the prototype. The
lack of drivers for peripheral sensors and cameras as well as the not specied weight and size of the
board are the most important issues for not using the FPGA. In view of time the system would only
be implementable with a FPGA and microprocessor in coexistence. However the system would get
gratuitous complex with this approach.
Thus for the nal system and prototype the Beagleboard XM is the best choice since it has a parallel
camera port and the largest computational power, compared to the other boards. Its power consumption is
low. The Beagleboard has a wide range of GPIO ports that support I
2
C as well as UART communication.
The computational power and free I/O ports allow to extent the system after this project, which is a
requirement of the clients.
3.4.2 Selection of the Camera
The choice of a camera was made, based on the overview given in section 3.2 and Table 3.1. To keep
the prototype simple and fast implementable, it was chosen to use a camera with digital output, so the
20B55P was not chosen as its output is an analogue composite format.
In section 3.4 it was decided that the CMUcam4 had to less computational power to reach the system
requirements. Thus the Omnivision 9665 sensor was not used, as it is dedicated for the CMUcam and
has quite long delivery times.
The nal system has to guarantee operation under various light conditions, like bright sunlight,
shadow and even very cloudy weather or indoor light conditions. Therefore easy luminance control
is a must. The luminance of the LI-VM34LP needs to be controlled externally in software, which is
more complicated than auto luminance control. Thus the LI-VM34LP is not used in the system although
it has the highest frame rate of the compared cameras. Furthermore the camera board is over ten times
more expensive than the TCM8230MD and more than ve times as expensive as the Conrad mini web-
cam. This is not a reasonable price for a camera of equal specications and therefore the LI-VM34LP
was not chosen.
For the TCM8230MD no drivers and breakout board are available. Although a breakout board can
be designed and fabricated on the faculty within one day, writing drivers can easily cost a week. In view
UAV Camera System UAV Thesis
16 3.5. POWER CONVERTER
of time it was decided to use the Conrad mini webcam since its technical specications meet the design
requirements. Furthermore it is easy implementable under the condition an operating system is used on
the processing platform. Although the USBwebcamconsumes much more power than the other cameras,
the processing unit and camera consume only about 3.5 W according to their specications. Thus with
the combination of Beagleboard and webcam, the power requirement of maximum 5 W is kept. Since
the camera needs to be mounted outside the UAV the system will benet from a robust camera like a
webcam.
An important parameter of a camera is its eld of view. However the FoV of the Conrad mini webcam
is not given in its documentation. So the eld of view angles of the camera had to be determined. By
measuring the visible height and width of an image. The camera was mounted at a height h of 0.77 m
looking downwards to the ground. In this setting, the visible width w = 0.636 m and length l = 0.468 m
were measured with a ruler. From these measurements the angles were calculated using Equation 3.1.
=
360

arctan
_
w
2 h
_
and =
360

arctan
_
l
2 h
_
(3.1)
For the used Conrad mini webcam, the results are given in Table 3.3.
Table 3.3: Camera calibration.
Horizontal Angle 44.9

Vertical Angle 33.8

3.5 Power Converter


On a UAV the whole system is powered via a rechargeable battery. The whole system may have a
maximum power consumption of 5 W. All devices are powered by the main processing board which
itself runs on a 5 V DC source. The AR.Drone runs on a 11.1 V battery, but the power supply of the nal
xed wing UAV is chosen by students at Aerospace Engineering. To make the system as transferable as
possible the power converter should be able to handle a wide range of input voltages. Since the power
consumption directly inuences the maximum ight time, a power converter with a high efciency is
desired. Due to the limited budget, the price of the converter is an essential parameter as well. In
Table 3.4 a selection of power converters is listed with the major design attributes.
Due to the limited payload of the testing platform, the weight is important, however all components
weight less than a gram, so this is not an issue.
Table 3.4: Overview of power converters.
Attribute Buck converter LDO Voltage divider
TPS5430 [40] L7805ABP [41] Simple resistors
Delivery speed One day One day One day
Max. Power output 15 W 5 W >15 W
Price e6 e0.22 e0.11
Efciency 95% 44.3% (@ 11.1 V) <44.3% (@ 11.1 V)
Range input voltage 5.5-38 V 7-35 V xed
Accuracy 1.5% 2% Depends on used resistors
UAV Thesis UAV Camera System
CHAPTER 3. BUILD SETUP & SYSTEM OVERVIEW 17
The efciency of the Low-Dropout Regulator (LDO) and Voltage divider is strongly dependent on
the used input voltage and output current. According to [42] the efciency of a LDO can be calculated
according to,
Efciency =
I
out
V
out
(I
out
+I
quiescent
)V
in
100%, (3.2)
where I
out
denotes the output current, V
out
the output voltage, V
in
the input voltage and I
quiescent
the
quiescent current needed for correct device operation. For the example of an AR.Drone and the LDO
L7805ABP the efciency is 44.3% according to 3.2.
The voltage divider is not used as it only performs at a single input voltage. In order to not double
the power consumption of the system with an inefcient power converter, the buck converter TPS5430 is
chosen as power converter for the system. The TPS5430 and the battery of the AR.Drone are shown in
Figure 3.10.
Figure 3.10: Buck converter TPS5430 and AR.Drone battery.
3.6 System Overview
In this section a system overview of the entire UAV camera system is provided. As concluded in sec-
tion 3.4 and section 3.5 the camera system consists of the Conrad mini webcam, the Beagleboard XM as
central processing unit and the TPS5430 buck converter. The hardware setup is shown in Figure 3.11.
Figure 3.11: Hardware setup with a webcam, the Beagleboard, the buck conveter and the AR.Drone battery.
UAV Camera System UAV Thesis
18 3.6. SYSTEM OVERVIEW
Furthermore the position and control group decided to measure the position of the UAV with a GPS
sensor and its height with a barometer. The wireless transmission group chose XBee Pros to transmit
data from the UAV to the base station. In order to communicate with all these sensors in an easy way, a
USB to UART shield is used. This shield provides easy software access to the sensors.
The function of the system is depicted in Figure 3.12. All tasks performed by the Beagleboard are
coloured blue, whereas the tasks of the ground station are shown in red. The three groups are each
responsible for a specic task. The position and control group reads the output of the GPS sensor,
the barometer and AR.Drone sensors. After processing this data it is passed to the transmission and
visualization group in order to be sent to the base station for visualization.
Furthermore, the data is passed to the camera system. The object detection and image processing
part is responsible for capturing an image and detecting the target object. In the position and velocity
estimation part the location of the object is calculated from the captured image and the sensor data
passed from the position and control group. The latter produces control signals from the target position
and velocity, which are send to the AR.Drone in order for the UAV to track the target.
The transmission and visualization group is responsible for transmitting all ight data to the base
station and visualize it. The user can request a picture from the webcam. The picture is captured by the
webcam and compressed in the image processing part of the system.
Control Signals AR.Drone
Sensor Output AR.Drone
Position &
Control
GPS Sensor Outpout
Barometer Sensor Outpout
Camera Control Signals
Camera Image
Signal
Translation
Position and
Velocity
Estimation
Object
Detection
& Image
Processing
Signal
Processing
Target
Postition &
Velocity
Speed,
Position,
Height UAV
Compressed
Camera Image
Object
Location
Sensor
Data
System
Input
Signal Processing for Wireless Transmission
Visualization
User Input
Camera
System
Transmission
& Visualization
Figure 3.12: Overview of the full electronic UAV system; the chart shows the interaction between the different
project groups.
In order to program on a high level and run multiple applications at the same time on an embedded
platform an operating system is required. Furthermore, an operating system is responsible for memory
management and handling peripheral devices. Although the activity of the operating system slows down
the camera system, it makes prototyping easier and faster.
The Beagleboard runs a minimal version of Ubuntu 11.10. with a XFCE graphical environment.
This operating system supports a wide range of peripheral devices. The graphical user interface allows
software development on the Beagleboard which saves valuable time.
UAV Thesis UAV Camera System
Chapter 4
Object Detection & Tracking Algorithms
In this chapter the algorithms used to track an object and estimate its parameters are studied. In sec-
tion 4.1 object detection algorithms are a topic of discussion to nd a fast, reliable algorithm, which
suits the need of embedded real time object detection. Furthermore it is concluded that colour threshol-
ding matches the need of this project, such that section 4.2 explains the colour thresholding algorithm
in more detail. The algorithm tends to nd the object in a picture and give the image coordinates. The
position and velocity estimations nd the real position and speed of the object that is detected in the rst
algorithm. The derivation of algorithms for position and velocity estimation are given in section 4.4 and
section 4.5, respectively.
The optical ow algorithm explained in section 4.6 has the possibilities to calculate motion of points
in subsequent video frames. This can be used to calculate the motion of the object and surroundings and
so the real speed of the object. This is a more complicated algorithm compared to the previous algorithm
for speed estimation.
The generation of steering signals is described in section 4.7. Finally the implementation and simu-
lation results are given in section 4.8, in addition the system results are covered in chapter 5.
4.1 Proposed Algorithms
A wide range of object detection algorithms have been proposed in literature as shown in chapter 2. In
this section four object detection algorithms are a topic of discussion: colour thresholding, mean/variance
model and Gaussian mixture model background subtraction as well as the TLD algorithm better known
as the predator algorithm.
The algorithms are compared on their reliability and accuracy of object detection. Special attention
has been paid to their computational complexity and use of memory, since an embedded platform can
only handle algorithms with low complexity in real time. These algorithm properties are listed together
with their main advantages and disadvantages in Table 4.1.
With colour thresholding the colour values of each pixel are checked for a certain threshold condition.
Pixels that pass the condition are detected as object. Thus a single computational operation is performed
on each pixel. This makes the algorithm fast compared to the other options and suitable for real time
applications. However pixels that belong to the background might pass the thresholding condition and
disturb the outcome.
Background subtraction is a more complex algorithm than colour thresholding. For background sub-
traction an image of the background as seen by the camera image is stored in a reference frame. Once an
object enters the frame it can be detected by comparing the reference frame with the captured frame.
19
20 4.1. PROPOSED ALGORITHMS
Table 4.1: Overview of available object recognition algorithms.
Attribute
Colour Background subtraction [43] TLD [44]
thresholding mean/variance model Gaussian mixture (Track, Learn, Detect)
Accuracy
Reliability
Low Medium High Very high
Computational
Complexity
Low Medium Medium Very high
Memory
Intensity
Low Medium High Very high
Detect Multiple
Objects
No Yes Yes Yes
Advantages
Fast Can handle multiple Objects
Easy implementable Adapt to background Learning algorithm
Can deal with Filters high frequency Can deal with luminance,
luminance changes background movement object and background changes
Disadvantages
Inaccurate Computational complex and memory intensive
Background may not vary fast Complex algorithms
Only detects Inaccurate under fast Stores a lot of
one Object changing luminance conditions images for learning feature
Lots of versions of background subtraction have been developed. However the two main approaches
have been included in these investigations: the mean/variance model and the Gaussian mixture model.
In the mean/variance model the reference frame contains the mean and the variance of each colour
component averaged over a number of frames, for every pixel. The background is thus modelled by a
single Gaussian distribution for each pixel. The mean and variance of each pixel is updated after each
iteration to make the algorithm adaptive.
In the case the Gaussian mixture model is used as background model, the background colour distri-
bution is modelled by multiple Gaussian curves. Thus more complex backgrounds, like trees moving
in the wind, can be modelled with the Gaussian mixture model. The drawback however is the fact that
more data needs to be stored for this algorithm.
Although background subtraction is mainly used on static camera systems, [45] described background
subtraction for mobile platforms.
The recently developed TDL algorithm, also known as predator algorithm, is the most advanced ob-
ject detection algorithm studied in this context. The algorithm can track multiple objects at the same
time. Furthermore the algorithm is a learning based algorithm which stores images of the object in each
geometrical and luminance position to optimize its tracking. The algorithm includes object detection
based on shape as well as colour. However the algorithm requires advanced computers to be run on a
reasonable frame rate. Thus the algorithm is too complex for embedded platforms.
As the nal system needs to operate at high frame rates to not lose the tracked object, it was decided to
implement the colour thresholding algorithm. The algorithm is easy implementable, fast and can deal
with changing light conditions. It can track a single object which limits the systemto only be functionable
when no other object of the specied colour appears in the processed frame.
The background subtraction methods require more processing power than colour thresholding, but
their use results in more reliable object detection. The algorithms have not been used as a high frame
rate is more important for the purpose of object tracking with fast moving UAVs, than an accurate and re-
liable algorithm. Furthermore background subtraction algorithms become unreliable under fast changing
luminance conditions as the reference frame contains the background image and different light condi-
tions.
UAV Thesis UAV Camera System
CHAPTER 4. OBJECT DETECTION & TRACKING ALGORITHMS 21
4.2 Colour Detection
To detect an object in a captured video frame, colour thresholding on pixels is implemented.
4.2.1 RGB or HSV
Two colour spaces are shown in Figure 4.1. The RGB and HSV colour space are compared on their
capability of colour separation. In subsection 4.2.2, red pixels need to be found.
(a) RGB colour space (b) HSV colour space
Figure 4.1: Comparison of the RGB and HSV colour spaces [46].
In the RGB colour space this is done by setting three thresholding criteria, however as can be seen in
Figure 4.1(a) the transition between colours is very smooth.
In the HSV colour space, on the other hand, the separation of the colours is well-dened as can be
seen on Figure 4.1(b). This is because the colour is only stored in the hue component. Thus if red needs
to be detected, only thresholding on the hue component is sufcient, to get red pixels. For an even better
result, thresholding on the saturation and value component yields only pixels that are not too dark or
bright for example. Therefore the HSV colour space is used to do colour thresholding on the captured
video frames.
4.2.2 Thresholding
After converting the incoming frame to the HSV colour space, colour thresholding is done, using the
following principle.
In the converted frame, each pixel will be processed and tested for three threshold criteria concerning
its hue, saturation and value components. If the pixel passes the thresholding, its location in the current
frame is stored. A counter is incremented every time a pixel location is added. After all pixel locations
are added up, the number is divided by that of the counter, which results in the centre of mass of the
object.
This principle is illustrated in Figure 4.2. In Figure 4.2(a) the original set of pixels from a picture
is shown. The algorithm nds the red pixels by checking their HSV values and colouring them black
if they satisfy the given criteria. The calculated centre of mass is nally coloured white, as shown
in Figure 4.2(b). This algorithm is given in pseudo code below. The image is loaded and the HSV
components are thresholded, the pixel values written back are given as RGB colours.
UAV Camera System UAV Thesis
22 4.3. STABILIZATION
(a) Original (b) Result after the described algorithm
Figure 4.2: Example of a set of pixels that are checked by the algorithm; black is accepted; centre of mass is white.
1: N
y
pixels
= picture width in pixels
2: N
x
pixels
= picture height in pixels
3: count = 0
4: x
centre
= 0
5: y
centre
= 0
6: for j = 0 N
x
pixels
do
7: for i = 0 N
y
pixels
do
8: if image(i, j, HSV) < (T
H
1, T
S
1, T
V
1)| image(i, j, HSV) > (T
H
2, T
S
2, T
V
2) then
9: image(i, j, RGB) (0, 0, 0)
10: x
centre
= x
centre
+i
11: y
centre
= y
centre
+ j
12: count ++
13: end if
14: end for
15: end for
16: X
obj
centre
= x
centre
/count
17: Y
obj
centre
= y
centre
/count
18: image(X
obj
centre
,Y
obj
centre
, RGB) (255, 255, 255)
4.3 Stabilization
In view of time it was decided that attaching servomotors to stabilize the camera, as proposed by [7],
would make the system more complex than desired for a simple prototype. Further means of hardware
stabilization were found to be either to heavy or not useful. Sensor shifting based stabilization is only
done inside (expensive) cameras and can therefore not be used.
Stabilization in software on the contrary would demand to much system resources and slow down the
processing. The third approach, a moving average lter [13], is therefore implemented.
A weighted moving average lter is not only a low process power demanding algorithm, but also a
practical way to remove noise and obtain a stable system output. The displacement of the pixel locations
of the tracked objects centre is a weighted average of the previous locations. An example of a weighted
moving average lter of length l is given by

FT
p+l;p+l+1
=
l

n=1
a
n

FT
p+n;p+n+1
(4.1)
where a
n
is the nth lter coefcient. The lter coefcients themselves have to satisfy normalization
criteria
l

n=1
a
n
= 1. (4.2)
UAV Thesis UAV Camera System
CHAPTER 4. OBJECT DETECTION & TRACKING ALGORITHMS 23
A disadvantage of using a moving average lter is that the speed of the system is directly related to the
length of the moving average lter. The length of the moving average lter and the weighting coefcients
is a trade-off between stability and speed of the system.
4.4 Position Estimation of the Detected Object
After the detection of the object in a camera frame (section 4.2), the location of the target vehicle can be
estimated. This location is needed in order to calculate its velocity and to generate steering signals for
the UAV.
The aim of this section is to derive a simple algebraic expression to estimate the location of the target
Ground Vehicle (GV) relative to the position of the UAV. These expressions depend on given parameters
like the sensor outputs and the position of the target in the image frame.
M
P
F
h
P

B
A
D C
B'
A'
D'
C'
d

T
x
y

P - Position of the UAV


F - Base point of the UAV
T - Position of the Target
M - Projection of the midpoint
d - Distance from M to F
h - Height of the UAV
- Camera angle
- Horizontal FoV angle
- Vertical FoV angle
A - Projection of the top-left
B - Projection of the bottom-left
C - Projection of the top-right
D - Projection of the bottom-right
- Horizontal target angle
A

- Captured image
Figure 4.3: 3D sketch used to estimate the location of the target.
F
B
A
D
C

T
x
y

FTx
FTy
FT
x
- Vertical target distance
FT
y
- Horizontal target distance
Figure 4.4: Top view of the sketch given in Figure 4.3.
In Figure 4.3 a sketch of the conguration of UAV and GV is shown. A top view of the conguration is
depicted in Figure 4.4. The UAV is located at the point P, its rectangular projection on the ground, the
UAV Camera System UAV Thesis
24 4.4. POSITION ESTIMATION OF THE DETECTED OBJECT
foot point, is called F and the target GV is located at point T. The barometer measures the height h of
the UAV and the GPS sensor measures the location of point P, so the location F is known as well.
Furthermore, the horizontal eld of view angle , and vertical eld of view angle are given by
calibrating the camera. The camera is mounted in an angle of . The horizontal angle of the UAV to the
ground can be taken into account by redening as the relative mounting angle without effecting the
following derivation. The relative mounting angle is dened as the sum of the absolute mounting angle
and the angle of the UAV to the horizontal ground. For simplicity the UAV is assumed to be parallel to
the ground at all time. However for xed wing UAVs the horizontal angle of the vehicle to the ground is
not neglectable, since xed wing UAVs can y under large horizontal angles.
The trapezium ABCD is dened as the projection of the captured image A

onto the ground as


shown in Figure 4.3. The projection of the midpoint of the image is dened as M. The distance d from
the foot point to M can be calculated using basic trigonometric functions,
d =h tan(). (4.3)
The location of the object in the captured image is given by the discrete pixel coordinate of the object
with respect to the upper left image corner A

X =
_
X
obj
centre
Y
obj
centre
_
. (4.4)
This coordinate is the output of the object detection algorithm and denotes the centre of mass of the
object in the captured frame. The projection of this coordinate on the ground is the relative position of
the target with respect to the UAV. Thus the pixel coordinate (0, 0) corresponds to an object located at A
and a pixel coordinate of (N
x
pixels
, N
y
pixels
) to an object located at C.
According to this the two dimensional reference coordinate system xy is dened where

x is dened
along the ight direction

FM and

y is dened along

BC. The relative coordinates of the target in the
dened coordinate system can be written as

FT =
_
FT
x
FT
y
_
. (4.5)
Since the pixels form a uniform grid in the camera image, one can assign a specic camera angle to
each pixel. The angle between neighbour pixels is dened as

in the horizontal and

in the vertical
direction. The pixels are uniformly distributed over the total eld of view angle so the angle between
neighbour pixels is dened as

=

N
y
pixels
1
and

=

N
x
pixels
1
. (4.6)
Since the exact angle of each vertical pixel is known, FT
x
can be calculated similar to Equation 4.3 as
FT
x
=h tan
_
+

2

(X
obj
centre
)
N
x
pixel
1
_
. (4.7)
From the top view shown in Figure 4.4, it can be seen that the target angle with respect to the central
camera axis can be written as
=

_
Y
obj
centre

N
y
pixels
2
_
=

N
y
pixel
1

_
Y
obj
centre

N
y
pixels
2
_
, (4.8)
where N
y
pixels
denotes the total number of pixels in the y direction.
Thus the relative location FT
y
can be written as
FT
y
=FT
x
tan
_

_
=FT
x
tan
_

N
y
pixel
1

_
Y
obj
centre

N
y
pixels
2
__
. (4.9)
UAV Thesis UAV Camera System
CHAPTER 4. OBJECT DETECTION & TRACKING ALGORITHMS 25
4.5 Velocity Estimation
Next to the position of the tracked ground vehicle its velocity is required in order to generate steering sig-
nals. A rst approach based on the time derivation of the position estimation is presented. In section 4.6
a more advanced optical ow algorithm is given.
The velocity of the tracked object is extracted from the displacement of the object in two subsequent
frames. To determine the objects velocity the displacement of the tracked object with respect to the UAV
and of the UAV itself is required.
The displacement of the object relative to the UAV can be calculated with the algorithm given in
section 4.4. The GPS coordinate is measured at each captured frame so that the absolute position and its
displacement are known as well.
The relative positions of the object are denoted as

FT
1
=
_
FT
x
1
FT
y
1
_
and

FT
2
=
_
FT
x
2
FT
y
2
_
, (4.10)
in the rst and second frame, respectively. The averaged positions as found by the moving average lter
are used. Furthermore the location of the of the UAV is given by

P
1
=
_
P
x
1
P
y
1
_
and

P
2
=
_
P
x
2
P
y
2
_
, (4.11)
in the same two frames. The xdirection of the coordinate system is dened by the UAVs ight direction,
whereas the ydirection is dened orthogonal to the xdirection.
The displacement vector of the object with respect to the ground between frame one and two

D
1;2
,
can be calculated as

D
1;2
=
_

FT
2
+

P
2
_

FT
1
+

P
1
_
. (4.12)
In the capturing algorithm the time between two frames is measured with a timer in order to be able to
calculate the frame rate. In the context of velocity estimation the time between two frames is denoted as
t. This time is needed to calculate the velocity of the object from its displacement.
The velocity, dened as the displacement per time interval, is calculated with

V
1;2
=

D
1;2
t
(4.13)
The absolute velocity is given by the length of

V
1;2
denoted as,

V
1;2

=
_

V
x
1;2
2
+

V
y
1;2
2
. (4.14)
4.6 Optical Flow
As stated by [14] the motion can be calculated with an optical ow algorithm. For all optical ow
methods the brightness of the object is assumed to be constant and the displacement is said to be small.
The basic idea of optical ow is nding the displacement of the object: the motion between two image
frames. This motion is between the rst frame at time t and the second frame at time t +t. Because the
intensity (I) of the object is assumed to be constant, this gives
I(x, y, t) = I(x +x, y +y, t +t). (4.15)
With the assumption that the movement is small, the above equation can be expanded using Taylor series,
this results in
I(x +x, y +y, t +t) = I(x, y, t) +
I
x
x +
I
y
y +
I
t
t. (4.16)
UAV Camera System UAV Thesis
26 4.7. GENERATION OF STEERING SIGNALS
Assuming the higher order terms of the Taylor expansion are neglectable. Combining 4.15 and 4.16 it
holds that
I
x
x +
I
y
y +
I
t
t = 0 (4.17)
and dividing the above equation by t yields
I
x
x
t
+
I
y
y
t
+
I
t
t
t
=
I
x
V
x
+
I
y
V
y
+
I
t
= 0. (4.18)
The V
x
and V
y
are components of the displacement over time of I(x, y, t), better known as the image
velocity or optical ow. This nally leads to its most simple form
I
x
V
x
+I
Y
V
y
= I
t
. (4.19)
However this equation has two unknowns and thus cannot be solved.
Various implementations of the basic algorithm are given, which introduce additional conditions, so
the system can be solved. The most used method is the differential method of calculating the optical
ow. For this method partial derivatives of the image signal are used. The most common implementation
is the one given by Lucas and Kanade: the so called Lucas-Kanade method [47]. It is beyond the scope
of this thesis to give a detailed description, but the algorithm basically works on a local area around the
point of which the ow needs to be calculated I(x, y, t). In the second image the pixels of a certain local
area (with size
x
and
y
) around the I(x, y, t) are investigated. Then V (V
x
and V
y
) is found to be the
vector that minimizes the residual function [47]
(V
x
,V
y
) =
x+
x

i=x
x
y+
y

j=y
y
_
I(i, j, t) I(i +x, j +y, t +t)
_
2
(4.20)
Now that the tracking is basically explained, it is still unclear what will or can be tracked. In an image
not all blocks of pixels are as traceable as other ones. This has to do with the surrounding features, on
a uniform road for example, the old pixel can be anywhere in the new image. Therefore nding high
contrasts is the basis of nding good features. On those high contrast lines, the highest contrast points
can be chosen to be tracked in the algorithm.
The nal result is a vector eld of the selected pixels, a vector points in the direction of the displace-
ment between the two frames. This vector eld can be used to estimate the motion of the object by
comparing its motion vector to that of the surroundings and subtracting them. This will yield the speed
of the object relative to the ground. The advantage is that the own speed is not needed for this technique
compared to the simple velocity algorithm of section 4.5.
4.7 Generation of steering signals
With the algorithms described above, the position of the object in the image is found. Although two algo-
rithms for path planning were studied in section 2.5, only the generation of steering signals is discussed
below. For the quadrocopter a simple rst order approach is given, for the xed wing UAV this approach
is not usable and therefore only path planning is possible.
4.7.1 Quadrocopter Approach
As a rst test approach the hovering property of the quadrocopter is used. The steering signals are
generated according to Figure 4.5.
The picture is divided into nine quadrants. When the object is in one of those quadrants, steering
signals will be generated belonging to that quadrant. The goal is to get the object in the centre of the
UAV Thesis UAV Camera System
CHAPTER 4. OBJECT DETECTION & TRACKING ALGORITHMS 27
frame, so steering signals are generated to full this. When the object is in quadrant LF the UAV has to
go left and forward, in this way the object will be shifting towards the centre. When the object is in the
centre the UAV can hover at its current position till the object moves away.
This method is quite easy to implement and can have good results on the quadrocopter, on the xed
wing UAV more complex algorithms should be applied to the known position of the object, because it
cannot hover on a spot.
L
LF
LB
S
F
B
RF
R
RB
L - Move Left
R - Move Right
F - Move Forward
B - Move Backward
And logic combinations
S - No Movement
Figure 4.5: Steering signals generation; quadrocopter approach.
4.7.2 Fixed Wing Approach
On the xed wing UAV the usage of steering signals is not useful, therefore path planning is needed to
derive a smooth ight path to the object. Path planning can be done by an autopilot, such as Paparazzi
[25], if given an estimated GPS and velocity of the object. The autopilot calculates the optimal ight
path to its destination. This technique is therefore optimal for the use on a xed wing UAV, which cannot
hover on a spot.
4.8 Implementation and Simulation
The algorithms needed to be tested before they were implemented on the Beagleboard. Below both the
implementation and simulation are described.
4.8.1 Implementation
C++ was used for the implementation of the algorithms, cause of its easy implementation, its high pro-
gramming level and primarily the availability of the Open Computer Vision Library (OpenCV). OpenCV
is an extensive C++ library that holds many functions regarding image processing, feature tracking, video
stabilization and more. Because OpenCV has many built in functions, readout of a webcam and process-
ing of video data is simplied, which speed ups the time required for writing video capture functions and
memory allocation.
4.8.2 Simulation
The colour detection algorithm and optical ow have been tested on a demo video. The other algorithms
require unknown parameters, like height and camera angles. Those algorithms were only tested in the
experimental test setup, given and described in chapter 5.
A demo video of a red car driving on a road is used to test these two algorithms. The video is taken
from a helicopter, which is similar to a video from a quadrocopter. An image of the demo video is shown
in Figure 4.6.
UAV Camera System UAV Thesis
28 4.8. IMPLEMENTATION AND SIMULATION
Figure 4.6: Image from a demo video of a red car driving on a road.
Testing was done using C++ with OpenCV. The algorithms were directly implemented and tested on a
pc. Below the simulation results using the demo video in OpenCV are given.
Colour Detection
The basic implementation of section 4.2 was followed and tested on the demo video. In the algorithm the
threshold values were chosen for H > 160 and H < 15. This gives the result as depicted in Figure 4.7.
On the image the pixels that satisfy the given threshold values are coloured black and the centre of mass
white (enlarged for better visibility). It can be seen that the car is the only object detected on the image,
Figure 4.7: Image from a demo video; colour detection.
the centre of the object is coloured white, with this the operation of the colour detection algorithm is
demonstrated.
UAV Thesis UAV Camera System
CHAPTER 4. OBJECT DETECTION & TRACKING ALGORITHMS 29
Optical Flow
Various implementations of the optical ow algorithm are predened in OpenCV. But before the optical
ow can be calculated, some points in the image have to be selected. This is done with the nd good
trackable features algorithm, also dened as an OpenCV function.
Figure 4.8: Image from a demo video; optical ow.
On Figure 4.8 the vector eld of the tracked features is given. The red dots are the current positions
of the features the algorithm selected and is now tracking. The white lines are the vectors connecting
the current positions to where they were in the last frame. It can be seen that the vectors of the car
are pointing in the direction the car is driving in. The surrounding vectors are pointing in the opposite
direction. From this it follows that the car is being chased and the chopper is ying in the direction the
car is driving in.
UAV Camera System UAV Thesis
30 4.8. IMPLEMENTATION AND SIMULATION
UAV Thesis UAV Camera System
Chapter 5
System Results
In chapter 4 the algorithms were tested with a demo video. In this chapter the performance of the
hardware setup described in chapter 3 together with the algorithms explained in chapter 4 is tested. First
of all the experimental setup is described in section 5.1. Two setups are described: a static and a moving
camera setup. In the following section 5.2 the results of each setup are presented. The results are nally
discussed in section 5.3.
5.1 Experimental Setup
The system was tested in two setups. Firstly using a static camera and secondly using a moving camera
on the quadrocopter. The static setup was split up into an indoor and an outdoor experiment to test the
system under various light conditions.
5.1.1 Static Camera
The outdoor setup is described rst, followed by the indoor setup. The outdoor setup is called semi-
outdoor, as the camera was placed inside a building looking outside, since the setup was not mobile at
the time of testing.
Semi-Outdoor
On the semi-outdoor setup, the camera was placed inside a building, behind a window, looking outside.
The height was set to 6.5 m and the camera angle to 72

. As target object a red umbrella was tracked.


The setup is depicted in Figure 5.1. This test was mainly done to see, whether the algorithms perform
well under outdoor conditions. The rst goal was to nd only the umbrella as a target and no other
objects. The second goal was to nd the place of the umbrella relative to the camera.
Indoor
In a room a matrix was drawn onto the ground. A master view of the matrix is depicted in Figure 5.2(b).
The camera was placed in point F at a height of 0.77 m, looking in an angle = 66

downwards upon the


matrix. The area visible in the captured image is coloured blue. A 3D view of the camera setup with the
object 1 m in front of the camera is depicted in Figure 5.2(a). The goal of this experiment was to detect
the object, nd its estimated location and compare it to the real location. Thereafter the estimation of the
speed and the operation of the moving average lter were tested. The moving average lter was tested
by shaking the camera.
31
32 5.1. EXPERIMENTAL SETUP
P - Position of the Camera
F - Base point of the Camera

FT
x
- Vertical target distance

FT
y
- Horizontal target distance
T - Target = Umbrella
h - Height = 6.5 m
- Camera angle = 72

- Horizontal FoV = 44.9

- Vertical FoV = 33.8

Figure 5.1: Semi-Outdoor testing setup.


(a) 3D View.
P - Position of the Camera
F - Base point of the Camera
T - Target = Red Cap
h - Height = 0.77 m
- Camera angle = 66

- Horizontal FoV = 44.9

The blue area can be seen


on the recorded image.
1 m
1 m
F
(b) Master View.
Figure 5.2: Indoor testing setup.
5.1.2 Moving Camera
The camera was mounted on the quadrocopter UAV. The integration of the other projects was not com-
plete at the moment of testing, so ight angles and height had to be estimated and assumed constant in
the algorithms. The ight speed was not available, so the inuence of the UAV its speed on the velocity
estimation could not be tested. The aim of the experiment was to y over an object, nd the position of
the object and test the generation of the steering signals.
Furthermore the accuracy of the location estimation algorithm is tested by hovering on a spot and
moving the object across the matrix dened in Figure 5.2(b). It was suspected that the hovering quadro-
copter adds noise to the location estimation algorithm. Therefore this setup is more realistic since the
camera system is used on non-stable platforms.
The setup is depicted in Figure 5.3. The height of the UAV and thus the camera was estimated to be
1.4 m. The angle of the camera looking down = 45

. The UAV was hovering on a spot of 1 m behind


the object. The same red cap as in the indoor setup was used as a target.
UAV Thesis UAV Camera System
CHAPTER 5. SYSTEM RESULTS 33
P - Position of the Camera
F - Base point of the Camera
T - Target = Red Cap
h - Height = 1.5 m
- Camera angle = 45

- Horizontal FoV = 44.9

The blue area can be seen on the


recorded image.
Figure 5.3: Moving camera testing setup.
5.2 Experimental Results
In this section the results of the experiments described above are reported. Furthermore, the image
processing algorithms are benchmarked with an accuracy test and a speed test. The rst test is used to
nd the accuracy; in the setup this includes the detection, but more importantly the calculation of the
relative distance and speed. The second test is to nd the performance; the number of frames handled
each second.
Section 5.2.1 quanties the accuracy such that the experimental results can be compared with each
other. The results of the static camera setup are provided in subsection 5.2.2 whereas the results of
the moving camera approach are given in subsection 5.2.3. The results of the algorithm benchmarking
on various camera resolutions are given in subsection 5.2.4. Since the system has to be able to take a
picture and transmit it wireless to a base station, the data size of images is a topic of investigation in
subsection 5.2.5.
5.2.1 Statistical Experiment Evaluation
To quantify the system results measured in the described experiments the accuracy of the algorithms
needs to be dened.
The accuracy is found by mounting the camera under a certain angle on a well-dened height. In
the eld of view a red coloured object is detected and the real distance to the object is measured. The
measured distance is compared to the outcome of the algorithm to nd its accuracy. To test the accuracy
of the velocity calculation algorithm, a similar method is used. The real location and the predicted
location are compared, the difference is the inaccuracy.
To quantify the accuracy the accuracy index A is dened as one minus the relative root mean square error
A =
_
_
_
_
1

P
p=1
1
X
2
p
_

X
p
X
p
_
2
P
_
_
_
_
100% (5.1)
where

X
p
is the location measured in the p-th measurement and X
p
is the actual location. Furthermore P
is the number of measured locations.
5.2.2 Static Camera
The results of the two static camera settings are given below. The results of the semi-outdoor approach
are given rst, followed by the indoor experiment.
UAV Camera System UAV Thesis
34 5.2. EXPERIMENTAL RESULTS
Semi-Outdoor
The semi-outdoor approach showed that the algorithms can detect objects in various outdoor light condi-
tions. The umbrella used as target object was detected in the bright sunlight as well as in the shadow of a
tree. It may thus be concluded that the object detection algorithm works under outdoor light conditions.
This experiment was mainly used as a qualitative experiment, since it is hard to measure the camera
height and object location in the given setup. However some quantitative result are given as well.
The real distance between the camera and the umbrella was measured and compared to the distance
estimated by the algorithm. Captured images are shown in Figure 5.4. On 5.4(a) the measured distance
was 16 m; the estimated distance 19 m. In 5.4(b) the real distance was set on 21 m, estimated 25 m. The
difference of some meters is caused by the inaccuracy in camera height, in the angle , and the difculty
of measuring the real distance.
(a) Tracking Close (b) Tracking Far
Figure 5.4: Two captured images from the semi-outdoor umbrella tracking experiment.
Indoor
At the indoor setup the accuracy of the algorithm was determined, because distances and angles could be
measured better than at the semi-outdoor setup. A red cap was used as target, as depicted in Figure 5.5,
where the matrix as seen by the camera is shown. The red cap was moved along a path to test the
reliability of the system on moving objects. The path of the red cap is visualized in Figure 5.6(a).
During the experiment, the location and velocity as estimated by the system were stored in a log le
and plotted with Matlab later. The estimated positions can be seen in Figure 5.6(b). The estimated points
on the matrix match with the real points on the ground. The path between two matrix spots is not straight,
as the red cap was moved by hand without the use of a ruler.
To show the path over time, a Matlab plot was made of the objects estimated position versus the frame
counter, shown in Figure 5.7(a). The measurement data from the experiment that covers this section is
listed in Appendix D Table D.1. The estimated velocity of the object can be found in Figure 5.7(b). It
can be seen that the object is rst found on location (x, y) = (3 m, -1 m), this is in accordance with the
starting point set in Figure 5.6(a). Around frame 60 the object starts moving towards the point (x, y) =
(1 m, 0 m). So the same time the x-coordinate changes 2 m, the y-coordinate changes only 1 m. Looking
at the speed, the velocity in the x-direction is twice the velocity in the y-direction, which is consistent to
the double displacement over the same time as found in the distance estimation. The second movement
is found to be in only in the x-direction, this can be found in both velocity and distance plots. On the
velocity plot some noise can be seen, this is caused by very small movements over multiple frames which
are not smoothed by the moving average lter.
The accuracy of the location estimation was calculated from the log le (Table D.1) according to
Equation 5.1. In the horizontal direction an accuracy of 99,3 % and in the vertical direction an accuracy
UAV Thesis UAV Camera System
CHAPTER 5. SYSTEM RESULTS 35
Figure 5.5: Objects on the predened matrix as seen by the webcam.
1 m
1 m
(a) Drawing of the path of the Ob-
ject.
(b) Objects estimated distance (Matlab plot).
Figure 5.6: Indoor path of object.
of 98,6 % was reached in the static indoor setup. The errors made are mainly due to measurement errors
in the camera angle and the alignment of the camera with respect to the matrix. The discretisation error
due to the projection of the discrete pixels on the ground plays a minor role.
The accuracy in the horizontal direction is higher than in the vertical direction, as the projection of
the image pixels on the ground have an higher spacing in the vertical direction for large distances. The
projection of the image pixels on the ground is shown in Figure D.1 in Appendix D. One can clearly see
that the distance between neighbouring pixel projections increases with increasing distance between the
target and the camera.
The accuracy is strongly dependent on the measurement error of the camera angle. Suppose a target
has been detected in the middle of the image and a camera angle of 66

has been selected. A measure-


ment error of 1

causes an error in the vertical position estimation of 4.2 %. This error is even higher for
greater camera angles or targets positioned in the top row of the image. This shows the high sensitivity
of the position estimation algorithm with respect to the camera angle. This sensitivity can be reduced by
choosing lower camera angles.
UAV Camera System UAV Thesis
36 5.2. EXPERIMENTAL RESULTS
(a) Object Estimated Location.
(b) Object Estimated Velocity.
Figure 5.7: Indoor time line.
In the case noise is added to the static camera system by shaking the camera the impact of the moving
average lter can be analysed. In gure Figure 5.8 the horizontal pixel location of the target is shown
before and after the moving average lter. The averaged pixel location does not vary more than 3 pixels
even though the noisy pixel location varies more than 14 pixels. One can clearly see that the noise applied
by shaking the camera is removed by averaging over multiple samples. In this example a moving average
lter of length 10 has been used. The drawback however can be found in the speed of the system. The
system gets slower as the lter becomes longer, as changes in the coordinate are averaged with previous
coordinates. The accuracy of the algorithm drops by 1,1% in x- and y-direction when applying noise.
Figure 5.8: Horizontal pixel location of the target as detected by the camera system before (blue) and after (black)
the moving average lter. Noise has been applied to the system by shaking the camera. The length of the moving
average lter is 10.
5.2.3 Moving Camera
As described in the setup of the moving camera, the webcam was mounted on the AR.Drone. In the
experiment the AR.Drone was hovering around a spot. The estimated location of the target object with
respect to the UAV is depicted in Figure 5.9. The red cap was placed 1 m in front of the AR.Drone, thus
on (x, y) = (1 m, 0 m). Thereafter it was moved to (x, y) = (2 m, 1 m) and the last point can be found on
(x, y) = (2 m, -1 m).
UAV Thesis UAV Camera System
CHAPTER 5. SYSTEM RESULTS 37
The AR.Drone its position and camera angle were estimated, however the real values vary due to
drifting of the AR.Drone. Due to this drifting the object seems to be shifting around its location, which
can be seen as noise and is partially corrected by the moving average lter. A second problem discovered
in the experiment is that the AR.Drone cannot maintain height, which implies the object also seems to
shift forward and backward with respect to its real position. This problem can of course not be xed by
the moving average lter.
One can clearly see that the results of the moving camera system are less accurate than the results of
the static camera; thus the accuracy is lower. The accuracy drops to 85.7 % in the x-direction and 89.1 %
in the y-direction.
Figure 5.9: Path of the red cap in the moving camera experiment on the ground as estimated by the camera system.
The camera was mounted on the AR.Drone, such that camera angle and height cannot be assumed constant.
5.2.4 Performance
The performance is independent of the experimental setup, as it depends on the algorithm, the operating
system (OS) and the platform used.
The same algorithm was benchmarked on the Beagleboard XM using Linux Ubuntu and on a laptop
using the same camera and OS. For this comparison, a resolution OF 640 480 was used. The results
of this performance benchmarking are given in Table 5.1.
Table 5.1: Performance overview on two test setups; resolution of 640 480.
Platform Beagleboard XM Dell Vostro 1710
Processor ARM Cortex A8 @ 1 GHz Intel Core2Duo @ 2.6 GHz
RAM 512 MB 3 GB
OS Ubuntu ARMEL (11.10) Ubuntu 32 Bit (12.04)
Capturing [fps] 13 28
HSV tracking [fps] 4.5 25
Optical Flow [fps] 3 18
Because the number of frames that can be handled at full resolution on the Beagleboard is not suf-
cient according to the design brief, a second test with images scaled in software was performed. Results
UAV Camera System UAV Thesis
38 5.2. EXPERIMENTAL RESULTS
for those smaller resolutions are given in Table 5.2. It can be seen that the capture rate is xed at 13-
14 fps. Under this condition, the processor is always fully loaded at 80 %, where the other 20 % are used
by the operating system.
Table 5.2: Performance overview for software resizing.
Resolution Capturing [fps] HSV tracking [fps] Optical Flow

[fps]
Full 640 480 13 4.5 4-2
Half 320 240 13 8.5 9-6
Third 213 160 13.5 10 10-8
Quarter 160 120 14 11 10-8

= Optical Flow is divided in no movement (upper bounds) and movement (lower bounds).
While enabling HSV colour tracking, the number of frames per second drops to 4.5 fps at full resolution,
whereas it is possible to do the calculations at 8.5 fps at the half resolution of 320 240. It can be further
increased by scaling to lower resolution, but it is limited to around 13-14 fps (the capture rate).
The main problem with this technique is the fact that the scaling is done after capturing a frame. Thus
a frame is captured at 640 480 and afterwards scaled down. Therefore, after testing software scaling,
also hardware scaling was implemented. The camera is set to capture images at a resolution lower than
640 480, so that no software scaling is needed. In Table 5.3 the performance results for hardware
scaling are given. It can be seen that the number of frames that can be captured is not any more limited
to 13 fps. Furthermore the performance increase for HSV tracking at half resolution is almost 50 %, for
a quarter of the original resolution this is even almost 120 %.
Table 5.3: Performance overview for hardware resizing.
Resolution Capturing [fps] HSV tracking [fps]
Full 640 480 13 4.5
352 288 24 12
Half 320 240 26 14
176 144 29 18
Quarter 160 120 31 24
The optical ow algorithm gets more intensive with increasing movement, therefore the number of
frames per second drops while increasing the camera or object movement. Furthermore the optical ow
algorithm is too slow to be used in this real time application, this can be improved by either decreasing
the resolution the Optical Flow is working on, or by decreasing the number of points to track. However
an algorithm with varying computational time is not desirable in real time applications. Thus the velocity
estimation is based on trigonometric functions.
However the fps benchmarking listed in Table 5.2 and Table 5.3 indicate the maximum possible
frame rate of the setup. In the nal prototype, the object detection algorithm is not the only program
running on the microprocessor. The sensor readout, transmission of data and steering of the UAV are
tasks of the processing unit as well. The programs performing the listed tasks have not been provided
at the day of testing. The benchmarking was performed with object detection, location calculation and
velocity estimation only. However the object detection is the most intense application running on the
Beagleboard.
Although it is suggested to use a resolution of 320 240 it order to detect even small objects, a
resolution of 176 144 has to be used to reach the system requirement of 15 fps. In the case a lower
frame rate does not affect the application signicantly a resolution of 320 240 should be used.
UAV Thesis UAV Camera System
CHAPTER 5. SYSTEM RESULTS 39
5.2.5 Image Size
The camera system has the ability to capture, compress and transmit an image from the UAV to the
ground station. Therefore the data size of different image compression algorithms is a topic of interest.
In Table 5.4 an overview of the size of stored image frames is given. The following three formats
were compared: JPEG, PNG and BMP. Those formats are the most common compressed image formats.
OpenCV has a standard function to write a captured frame to any of these formats. The frames were
stored under their default compression ratios. The JPEG images clearly have the smallest le size and
are therefore preferred as image storage format for sending pictures through the XBees. The compression
of an image takes the same time for all image format, such that choice of format is only based on data
size.
Table 5.4: Size overview for different image storage formats; default compression.
Resolution JPEG [kB] PNG [kB] BMP [kB]
Full 640 480 60 335 900
Half 320 240 18 93 225
Quarter 160 120 6 27 57
5.2.6 Power and Weight
The nal system consumes 4.5 W and weights 175.5 g as shown in Table 5.5. A maximum power
consumption was not specied in the design, but a requirement of 5 W was added as a requirement for
the system. The system consumes less than 4.5 W and therefore meets design criteria. In the short time
span of the project is was not possible to design a system under 100 g such that only subsystems could be
tested on the AR.Drone. The design criterion of 250 g was reached as the system only weights 175.5 g.
Table 5.5: Overview of the measured power consumption and weight of the system.
Device Power Weight
Beagleboard 2.5 W 72 g
Camera 1.5 W 25 g
GPS Sensor 75 mW 10 g
Barometer 25 W 0.5 g
XBee 200 mW 20 g
USB to UART N.A. 25 g
Buck converter N.A. 23 g
Efciency
Buck Converter
95% N.A.
Total 4.5 W 175.5 g
5.3 Experimental Discussion
In section 5.2 the experimental results were listed. The accuracy, performance, image size and power
were calculated or measured. In this section the outcomes of the experiment are a topic of discussion.
Firstly, the accuracy is derived for both the static camera and the moving camera. As expected the
accuracy for the static camera is higher than for the moving camera. Whereas the static setup reaches up
to 99 %, the moving setup is limited to around 90 %. This is mainly caused by the disturbance and drift
UAV Camera System UAV Thesis
40 5.3. EXPERIMENTAL DISCUSSION
of the AR.Drone. On the xed wing it is suspected that a higher accuracy can be achieved, because the
xed wing UAV is more stable when ying on its cruise speed.
In the nal prototype the camera angles and ight height are measured by the Position and Control
system developed by another subgroup of the project. Thus the position estimation is based on actual
ight data and not on a static height and a xed camera angle. This has a positive effect on the accuracy
as well.
Furthermore it was found that the accuracy is highly dependent on the camera angle . Changing the
camera angle from 66

to 67

already results in a maximum deviation of 14,4% in the estimated x-


location, in the case an object is detected in the top row. In the middle of the picture the deviation due to
the change in camera angle results in a deviation of 4,2%.
Changing the camera angle form 45

to 44

results in and maximum deviation of 4% on the top row


of the image and of 3,5% in the middle of the picture. So to increase the accuracy of the camera system
a low camera angle ( 45

) should be chosen. But in order to control a large area the opposite holds
true. So the choice of the camera angle is a trade-off between accuracy and view area. For the prototype
on the AR.Drone a good trade-off between view area and accuracy has been found empirically with a
camera angle of 45

.
On the xed wing UAV the use of smaller camera angles is suggested, as a xed wing UAV operates
at greater heights such that the covered area is larger than on the AR.Drone.
The performance was measured in the second step. As showed, changing the resolution of the captured
frames, affects the frame rate directly. As stated in the design brief, around 15 fps need to be captured.
By halving the resolution to 320 240 a frame rate of 14 fps has been achieved. It has been shown that
hardware resizing is more effective than software resizing.
Changing the resolution however, directly affects the size of the object that can be detected, as can
be seen in Appendix D Figure D.1, a height h of 0.77 m and a camera angle = 66

(as in the indoor


setup) were chosen to visualize the impact of the resolution. A bigger distance between two blue lines,
holds that the object is harder to detect. The consequence is that objects that are further away are harder
to detect.
Figure D.1(b) shows the projection of the pixels upon the ground in a resolution of 320 240, all
object, except the one on 6 m, are well covered with multiple pixel lines. In the experiment is was
conrmed that the object on 6 m was hard to detect.
Changing the resolution to 176 144 as shown in Figure D.1(c), increases the distance between
image points and objects from 4 m on are be hard to cover. However the frame rate increases to 18 fps,
such that faster movements can be detected. Therefore it is a trade-off between the speed and the size of
objects that are detected at a certain distance.
As objects are harder to detect on images with smaller resolution it is recommended to use images
sized 320 240 and accept the 4 fps drop in frame rate. The increase in reliability of the object detection
compensates the decrease in frame rate.
Thirdly the image sizes was a topic of investigation. A fast transmission of images is preferred such
that small images are desired. The smallest images are stored using the jpeg format and therefore this
format is be used to store and transfer images to the ground station. At time of writing the bit rate of the
transmission group was not yet determined, such that the exact image resolution for image transmission
has not been chosen.
Finally the power of the system had to be less than 5 W, which is achieved. Thanks to a good power
converter the losses in the system are small, most power is consumed by the Beagleboard, which makes
sense, because all processing and algorithms are done there.
UAV Thesis UAV Camera System
Chapter 6
Conclusion
In this chapter some conclusions from the work performed over the last two month are drawn. The proto-
type is evaluated in section 6.1 with respect to its design requirements listed in section 3.1. Furthermore
the test results from chapter 5 are summarized in the evaluation section. Finally, in section 6.2 sugges-
tions for future work are given and explained. Special attention has been paid to this section, since the
camera system is still in development and students will work on this project in the near future. The listed
point for further work should act as a reference point for students working on the camera system.
6.1 Evaluation of the Prototype
In this section the prototype developed in this project is evaluated. By comparing the system results
presented in chapter 5 with the system requirements, which are applicable to the camera subsystem.
In Table 6.1 the system requirements regarding the camera system are listed. System requirements
that have been achieved are marked (). One can clearly see that the aim of this project has been
achieved: design, build and test an embedded UAV object detection platform.
Table 6.1: The various system requirements that are achieved () in the present prototype.
System Requirements Prototype Comments
Software
Processing 15 fps On a resolution of 320240 with thresholding
Location estimation Implemented with an accuracy of 99% (static)
Velocity estimation Accuracy dependent on location estimation
Image stabilization A moving average lter is implemented
Hardware
Capturing 15 fps On a resolution of 640480
Power consumption <5 W System needs 4.5 W
weight <250 g The total system weights 175 g
weight <100 g The total could thus not be tested on the AR.Drone
Costs < e250 The camera system costs e175
Expandable (free I/Os) The Beagleboard has free USB and GPIO ports
Testing
On AR.Drone The camera system has been tested on the AR.Drone
On Fixed wing UAV The whole system has not been tested on xed wing
Object detection on 50 m Strongly dependent on size of object
Documentation
Well documented Each design choice and code included in thesis
Suggestions future work List included for further development
41
42 6.1. EVALUATION OF THE PROTOTYPE
The system requirement of capturing and processing 15 fps has been achieved. It has been shown that
the system is capable to detect objects and estimate their location and velocity within known ranges of
accuracy. The system can reach an accuracy of up to 99 % for the static and 90 % for the moving camera
system.
The accuracy also depends on the x- and y-direction, by which the y-direction shows the highest
accuracy of the two. This is caused by the values of the angle and height of the camera of which the
inuence is bigger for the x-direction. A change of 1

in the camera angle can lead to a deviation in the


x-direction up to 14,4 % around 66

, around 45

this deviation goes down to around 4 %.


Therefore it is advised to measure the camera angle very accurate and preferably with a dedicated
sensor. In the y-direction this deviation is hardly seen, therefore the accuracy in this direction is higher.
Furthermore the empirical ndings of this project show that the possible size of detected objects heavily
depends on the resolution used. On the larger resolutions the system can detect objects at 50 m sized
2 m
2
.
However it is suggested to use larger objects in the UAV competition as the reliability and accuracy
increases using larger objects. When using larger objects, the use of smaller resolution is suggested, as
it increases the processing speed of the system and fast moving objects can be detected too. A resolution
of 320 240 is recommended for the nal system.
In the case fast objects need to be tracked and the design requirement of 15 fps needs to be satised
a resolution of 176 144 should be used. This leads to a less reliable object detection compared to the
resolution of 320 240.
The size of the compressed images varies with resolution and storage format, this mainly depends on the
compression ratio of the various formats. It is advised to use the highest compression ratio for sending
images over the transmission link to the ground station, to minimize the data that needs to be transferred.
A soft real-time criterion is met by the system as the system can guarantee an update of the targets
location in the speed of the frame rate. The quality of the object tracking benets from a higher frame
rate but is still able to operate in the case at low frame rate.
At last the power and weight requirement was satisfactionally achieved; the power consumption was
maximum 4.5 W and the weight of the system is 175 g excluding the battery. However the maximum
payload of the AR.Drone is 100 g, since the AR.Drone is designed as toy and not as expandable UAV
platform. The subsystem described in this thesis could be tested on the AR.Drone, but the whole system
could not be tested on the AR.Drone, since it is too heavy.
The system has not been tested on the xed wing UAV yet, since the targeted platform was still in
development at the aerospace faculty at the time of delivering this thesis. Therefore the stability and
accuracy on xed wing ights is unknown and could not be estimated.
The prototype can be used to extend the system as it is still in development in order to complete
all tasks required for the UAV challenge it is built for. The number of free ports on the Beagleboard
is still greater than the number of used ports, which indicates the possibilities for future development.
Although the system is not ready for the UAV competition yet, the rst milestone on the way to an UAV
with camera system has been reached within this thesis. The work documented in this thesis will clearly
contribute to the electronic system of the nal xed wing UAV demonstrating the task of the new UAV
challenge.
UAV Thesis UAV Camera System
CHAPTER 6. CONCLUSION 43
6.2 Further Work
During the tests of the camera system, some points for further work have been discovered. The points
listed below could not have been addressed in the given time.
Further system development should therefore concentrate on the following points listed by their per-
tinence.
I Interface the system with the autopilot of the xed wing UAV. In order to test the system on the
xed wing UAV designed at the aerospace faculty the camera system has to be interfaced with its
autopilot to pass the steering signals.
II Test the system on a xed wing UAV. As the xed wing UAV was developed simultaneously with
the camera system, it could not be tested on the xed wing UAV. Since the design requirements
target the use of the system on a xed wing UAV, testing it on such a system is indispensable.
III Take the side wards angle of the UAV into account in the position estimation. In view of time
only the forwards angle has been taken into account while calculating the position of the target. The
system gains more accuracy once the side wards angle is taken into account as well. For the xed
wing this is a must.
IV Improve object detection algorithm. Although the algorithm can detect object with a high relia-
bility the prototype would benet from an algorithm which can detect objects with various colours.
This can be implemented by choosing a contrast detection or a moving object detection algorithm.
However special attention should be paid to the computational complexity of those algorithms since
the frame rate get drops signicantly while choosing computationally complex algorithms.
V Further optimization of the algorithms for an embedded platform. Although all algorithms
were used on an embedded platform they are not optimized to run on embedded devices. Using
faster and smarter algorithms will improve the usability on the embedded platform. Minimalizing
the operating system and disabling all other processes will further improve the speed.
VI Including a system for plane detection. Next to the detection of a ground vehicle a UAV has to be
detected by the system as well, according to the competition description. Another camera mounted
on the top of the UAV might be used to complete this task. A IR camera might be useful for this
task as planes have an infra-red contrast with respect to the sky.
VII Transferring the system to a single PCB. For prototyping reasons the system currently consists
out of a Beagleboard with a lot of peripheral devices. To reduce the weight of the system and make
it more robust against vibrations and acceleration, the system should be transferred to a single PCB.
UAV Camera System UAV Thesis
44 6.2. FURTHER WORK
UAV Thesis UAV Camera System
Bibliography
[1] G. M. E. Dempsey, The US army roadmap (2010-2035) UAS, US Army UAS Center of
Excellence, pp. 35, 2010. [Online]. Available: http://www.fas.org/irp/program/collect/uas-army.
pdf
[2] Remarkable progress made in military products for civilian use, Peoples Daily, dec. 2000,
Accessed on 2012-05-3. [Online]. Available: http://english.people.com.cn/english/200012/19/
eng20001219_58213.html
[3] Wildre imaging ights by nasas ikhana uav conclude, NASA, oct. 2007, Accessed on 2012-04-
26. [Online]. Available: http://www.nasa.gov/centers/dryden/news/Features/2007/wildre_socal_
10_07.html
[4] D. Sachs, S. Nasiri, and D. Goehl, Image stabilization technology overview,
2009, Accessed on 2012-04-23. [Online]. Available: http://www.invensense.com/shared/pdf/
ImageStabilizationWhitepaper_051606.pdf
[5] S. Ali and M. Shah, COCOA: tracking in aerial imagery, in Airborne Intelligence, Surveillance,
Reconnaissance (ISR) Systems and Applications III, vol. 6209, no. 1, 2006.
[6] A. Ibrahim, P. W. Ching, G. Seet, W. Lau, and W. Czajewski, Moving objects detection and track-
ing framework for uav-based surveillance, in Image and Video Technology (PSIVT), 2010 Fourth
Pacic-Rim Symposium on, nov. 2010, pp. 456 461.
[7] Z. Li and J. Ding, Ground moving target tracking control system design for uav surveillance, in
Automation and Logistics, 2007 IEEE International Conference on, aug. 2007, pp. 1458 1463.
[8] I. Mondragon, P. Campoy, M. Olivares-Mendez, and C. Martinez, 3d object following based on vi-
sual information for unmanned aerial vehicles, in Robotics Symposium, 2011 IEEE IX Latin Amer-
ican and IEEE Colombian Conference on Automatic Control and Industry Applications (LARC),
oct. 2011, pp. 1 7.
[9] L. Feng, M. Chen Ben, and L. K. Yew, Integration and implementation of a low-cost and vision-
based UAV tracking system, in Control Conference, 2007. CCC 2007. Chinese, jun. 2007, pp. 731
736.
[10] A. Price, J. Pyke, D. Ashiri, and T. Cornall, Real time object detection for an unmanned aerial
vehicle using an fpga based vision system, in Robotics and Automation, 2006. ICRA 2006. Pro-
ceedings 2006 IEEE International Conference on, may 2006, pp. 2854 2859.
[11] K. Dastner, B. Kohler, and F. Opitz, 3D object extraction and tracking for infra red and day light
camera systems, in Radar Symposium (IRS), 2011 Proceedings International, sept. 2011, pp. 737
742.
45
46 BIBLIOGRAPHY
[12] P. Angelov, R. Ramezani, and X. Zhou, Autonomous novelty detection and object tracking in video
streams using evolving clustering and takagi-sugeno type neuro-fuzzy system, in Neural Networks,
2008. IJCNN 2008. (IEEE World Congress on Computational Intelligence). IEEE International
Joint Conference on, jun. 2008, pp. 1456 1463.
[13] S. W. Smith, The Scientist and Engineers Guide to Digital Signal Processing, 1st ed. California
Technical Pub, 1997, Chapter 15.
[14] H. Hayashi, M. Kudo, J. Toyama, and M. Shimbo, Estimation of velocity vectors from a video
stream using discontinuity of optical ow, in Knowledge-Based Intelligent Information Engineer-
ing Systems, 1999. Third International Conference, dec. 1999, pp. 447 450.
[15] Z. Liu, X. Tian, and D. Zhou, Man-made object detection algorithm of sonar image based on
texture analysis, in Signal Processing, 2006 8th International Conference on, vol. 4, 2006.
[16] Y. Pailhas, Y. Petillot, and C. Capus, High-resolution sonars: what resolution do we need for
target recognition? EURASIP J. Adv. Signal Process, vol. 2010, pp. 42:142:13, feb. 2010.
[Online]. Available: http://dx.doi.org/10.1155/2010/205095
[17] BW5000 sonar data sheet, Blueview, 2012, Accessed on 2012-04-30. [Online]. Available:
http://www.blueview.com/media/Data_Sheet_-_BV-5000_v8.pdf
[18] C. Grillenzoni, Robus object detection from laser scanner data, Quaderni di Statistica, vol. 8,
2006.
[19] A. Brooks, Real-time digital image stabilization, in EE 420 Image Processing Computer Project
Final Paper, mar. 2003.
[20] S. Mann and R. Picard, Video orbits of the projective group a simple approach to featureless
estimation of parameters, Image Processing, IEEE Transactions on, vol. 6, no. 9, pp. 1281 1295,
sep 1997.
[21] H. Liu, A. Rosenfeld, and R. Chellappa, Two-frame multi-scale optical ow estimation using
wavelet decomposition, in Acoustics, Speech, and Signal Processing (ICASSP), 2002 IEEE Inter-
national Conference on, vol. 4, may 2002, pp. IV3588 IV3591.
[22] J. Magarey and N. Kingsbury, Motion estimation using a complex-valued wavelet transform,
Signal Processing, IEEE Transactions on, vol. 46, no. 4, pp. 1069 1084, apr. 1998.
[23] G. Farneback, Fast and accurate motion estimation using orientation tensors and parametric mo-
tion models, in Pattern Recognition, 2000. Proceedings. 15th International Conference on, vol. 1,
2000, pp. 135 139 vol.1.
[24] J. Chen and D. Dawson, Uav tracking with a monocular camera, in Decision and Control, 2006
45th IEEE Conference on, dec. 2006, pp. 3873 3878.
[25] The paparazzi project. ENAC University, may 2012, Accessed on 2012-6-7. [Online]. Available:
http://paparazzi.enac.fr
[26] TCM8230MD CMOS camera v1.20 data sheet, Toshiba, 2004. [Online]. Available:
http://www.sparkfun.com/datasheets/Sensors/Imaging/TCM8230MD.pdf
[27] Leopard imaging products, Leopard Imaging, 2012, Accessed on 2012-04-25. [Online].
Available: https://www.leopardimaging.com/
[28] CMUcam, Carnegie Mellon University, 2012, Accessed on 2012-04-26. [Online]. Available:
UAV Thesis UAV Camera System
BIBLIOGRAPHY 47
http://www.cmucam.org
[29] 20B44M pictures, RHPstore, 2012, Accessed on 2012-05-13. [Online]. Available: http:
//www.rhpstore.com/images/
[30] CCD vs. CMOS, Teledyne DALSA, 2012, Accessed on 2012-05-21. [Online]. Available:
http://www.teledynedalsa.com/corp/markets/ccd_vs_cmos.aspx
[31] TCM8230MD cmos camera product price, Sparkfun, 2012, Accessed on 2012-04-26. [Online].
Available: http://www.sparkfun.com/products/8667
[32] MUX, TCM8230MD breakout board description, 2011, Accessed on 2012-04-26. [Online].
Available: http://sigalrm.blogspot.com/2011/03/tcm8230md-breakout.html
[33] Leopard imaging boards (camera) price, Leopard Imaging, 2012, Accessed on 2012-04-25.
[Online]. Available: http://shop.leopardimaging.com/category.sc?categoryId=10
[34] Beagleboard-XM development board rev c 1.0 data sheet, BeagleBoard.org, apr. 2010. [Online].
Available: http://www.farnell.com/datasheets/1537539.pdf
[35] Raspberry pi development board model b hardware overview, Camebridge University, may 2012,
Accessed on 2012-05-5. [Online]. Available: http://elinux.org/RPi_Hardware
[36] Spartan-3 starter kit board user guidet, ug130 (v1.1), XILINX Inc, 2005. [Online]. Available:
http://www.digilentinc.com/Data/Products/S3BOARD/S3BOARD_RM.pdf
[37] CMUcam 4, Carnegie Mellon University, 2011, Accessed on 2012-04-26. [Online]. Available:
http://www.cmucam.org/projects/cmucam4/
[38] Raspberry pi news, Cut The Record, mar. 2012, Accessed on 2012-05-25. [Online]. Available:
http://cutthecord.nl/2012/03/19/raspberry-pi-pas-leverbaar-in-juli/
[39] I. Digilent, Spartan-3 board, product information, jun. 2012. [Online]. Available: http:
//www.digilentinc.com/Data/Products/S3BOARD/S3BOARD-top-400.gif
[40] TPS5430 3-a, wide input range, step-down swifttm converter, data sheet,
Texas Instruments, nov. 2006. [Online]. Available: http://nl.rs-online.com/web/p/
dc-to-dc-converter-switching-regulator-chip/0182212/
[41] L78xx, L78xxC L78xxAB, L78xxAC, positive voltage regulator ics, data sheet, STMicroelec-
tronics, 2012. [Online]. Available: http://www.st.com/internet/com/TECHNICAL_RESOURCES/
TECHNICAL_LITERATURE/DATASHEET/CD00000444.pdf
[42] D. B. Lee, Understanding the terms and denitions of ldo voltage regulators, application report
slva079, Texas Instruments, oct. 1999. [Online]. Available: http://www.ti.com/general/docs/lit/
getliterature.tsp?baseLiteratureNumber=slva079&track=no
[43] Y. Long, X. Xiao, X. Shu, and S. Chen, Vehicle tracking method using background subtraction and
meanshift algorithm, in E-Product E-Service and E-Entertainment (ICEEE), 2010 International
Conference on, nov. 2010, pp. 1 4.
[44] Z. Kalal, J. Matas, and K. Mikolajczyk, P-N Learning: Bootstrapping Binary Classiers by Struc-
tural Constraints, Conference on Computer Vision and Pattern Recognition, 2010.
[45] E. Hayman and J.-O. Eklundh, Statistical background subtraction for a mobile observer, in Com-
puter Vision, 2003. Proceedings. Ninth IEEE International Conference on, oct. 2003, pp. 67 74
UAV Camera System UAV Thesis
48 BIBLIOGRAPHY
vol.1.
[46] SharkD, Comparison of different color solids for the hsl, hsv and rgb color models.
aug. 2008, Accessed on 2012-6-4. [Online]. Available: http://et.wikipedia.org/wiki/Pilt:
Color_solid_comparison_hsl_hsv_rgb_cone_sphere_cube_cylinder.png
[47] J. Y. Bouguet, Pyramidal implementation of the lucas kanade feature tracker, 2000, Accessed on
2012-05-14. [Online]. Available: http://www.intel.com/research/mrl/research/opencv/
UAV Thesis UAV Camera System
Appendix A
Design Brief
1. Introduction
For this Bachelor Thesis Project within the curriculum of Electrical Engineering at the department
of EEMCS of Delft University of Technology, a system will be designed for implementation on
an Unmanned Aerial Vehicle (UAV). This system will be a combination of hardware and software.
It will focus on the professional market and, with minor modications, is to be used in the UAV
Competition organized by Thijs Durieux and Johan Melis. This document will specify project
requirements and design constraints.
2. Requirements derived from the intended use
2.1 The system must exible to the extent that it can be implemented on both so-called quadro-
copters
1
and xed-wing UAVs. A detailed denition of exible:
i. The system must be able to function properly on a xed-wing UAV with a velocity of
100 km/h.
ii. The battery power, the number of available I/O ports and the available processing power
must be sufcient to allow for an extension of the system with a second camera and
additional sensors. This requirement is related to the future possibility of tracking other
airborne vehicles.
2.2 The system must be capable of detecting a ground-based moving vehicle from the air.
2.3 The system must be capable of tracking (i.e., following) the ground-based vehicle at a dis-
tance of no more than 50 m.
2.4 The velocity of the detected vehicle may not exceed 55 km/h.
2.5 The system must be capable of taking a photograph of the detected ground-based vehicle.
2.6 The system must be capable of transmitting the photograph to a ground station.
2.7 The system must be able to determine its position (x and y) with a resolution of 2 m.
2.8 The system must be able to determine its height (i.e., z) with a resolution of 1 m.
2.9 The system need not autonomously control a UAV.
2.10 The system need not withstand rain or extreme humidity.
2.11 The mass of the system excluding camera and battery may not exceed 250 g.
2.12 The system must function at least 1 hour with its own battery.
2.13 The system must be delivered and documented in such a way that work can be continued by
people not regularly afliated to the project and our project team.
2.14 A (scaled) demonstrator must be delivered, consisting of a AR.Drone that is capable of track-
ing an RC model car and transmitting an aerial photograph of this car to a base station. The
control and position signals transmitted by the UAV must be visualized in a graphical user
interface (GUI) on the ground station.
1
Helicopter-like aircraft with four horizontal rotors, often arranged in a square layout.
49
50
3. Requirements derived from the (ecological) situation of the system in its surroundings
3.1 The system must comply with all legislation concerning radio wave transmission.
3.2 The altitude of the system may not exceed 70 m.
3.3 The system must be able to communicate with its ground station in all possible geometrical
positions.
3.4 The system must be able to communicate with its ground station over a range of 1000 m.
3.5 The system is to be built from standardized components and shall therefore comply with
environmental standards such as RoHS.
i. The lifetime of the system will be determined by the lifetime of the components.
ii. Requirements with respect to recycling and disposal are determined by the components
of the system.
4. Requirements concerning the system as such
4.1 The system must generate and transmit a signal if the battery power drops below a predeter-
mined threshold value.
4.2 The system must be able to cope with ight vibrations to the extent that its functionality is
not inuenced during ight.
4.3 The system must send a signal to the autopilot if the connection between the system and
its ground station is lost for more than t seconds. This threshold value t must be set to the
desired value digitally.
4.4 The system must send signals to the autopilot protocol or bus (to be determined) concerning
its position and the (estimated) position of the tracked vehicle.
4.5 The data link between the system and the ground station must be suitable for transmitting
autopilot signals with a maximum bandwidth of 2 MHz.
MoSCoW model draft
Must have
All points mentioned in the design brief.
Should have
Possibility of detecting an airborne vehicle (e.g., another UAV).
Possibility of tracking the detected airborne vehicle.
Could have
Fully functional built-in autopilot.
Possibility of dening geographical operating boundaries of the UAV.
Would have (would be)
A fully autonomous xed-wing UAV that meets the demands of Thijs Durieux and Johan Melis.
UAV Thesis UAV Camera System
Appendix B
System
GPS Sensor
Height Sensor
Camera
Laptop Input
Battery Voltage
Demux
Inputs
Data Request (picture, location)
Send signals
Calculate
Remaining
Time
Start landing procedure
Recording
Video
Raw Video
Stream Adapted
Image

Video
Stream
Recognize
Object
Flight Hieght
Determination
Flight Height
UAV Location
Determination
GPS Location
Object Location
Calculation GPS Location
Object
Motion Vector
Calculation
Motion Vector
Control Signals
Generation
Control Signals
Control Signals
Translation
(Driver)
Auto Pilot / Parrot
Show
Data
Send
Data
Filtered Data Data
Filtration
GUI Output
1 2 3
4 5
6
Figure B.1: Functional block diagram of the UAV system; camera subsystem in gray.
In Figure B.1 the total systemis shown, the parts covered in this thesis are depicted in gray. The following
steps are made.
1. The video is recorded using a camera (section 3.2).
2. The raw video data is adapted to be used on a processing board (section 3.3).
3. On the images from the video stream, the object is detected (section 4.2).
4. The location of the detected object is calculated (section 4.4).
5. Besides the location, also the velocity of the object is calculated (section 4.5).
6. Finally steering signals for the UAV are found (section 4.7).
51
52
UAV Thesis UAV Camera System
Appendix C
C++ Code
The nal C++ code using the OpenCV libraries for video processing is printed below.
webcamCalc.h
1 # i f n d e f WEBCAMCALC_H_INCLUDED
2 # d e f i n e WEBCAMCALC_H_INCLUDED
3
4 # i n c l u d e " opencv2 / vi deo / t r a c k i n g . hpp "
5 # i n c l u d e " opencv / cv . h"
6 # i n c l u d e " opencv2 / i mgpr oc / i mgpr oc . hpp "
7 # i n c l u d e " opencv2 / hi ghgui / hi ghgui . hpp "
8 # i n c l u d e <s ys / t i me . h>
9 # i n c l u d e " t i me . h"
10 # i n c l u d e " math . h"
11 # i n c l u d e <s s t r eam >
12 # i n c l u d e <f s t r e am >
13 # i n c l u d e < s t r i n g >
14 # i n c l u d e <i os t r e a m >
15 # i n c l u d e <ve c t or >
16
17 # d e f i n e PI 3. 14159265
18 # d e f i n e LENGTH_AVERAGE 5 / / Lengt h of t he moving a ve r a ge f i l t e r .
19 # d e f i n e s c a l e f a c t o r 3 / / Set s c a l e f a c t o r s f o r r e s i z i n g .
20 # d e f i n e s c a l e f a c t o r s t o r e 1
21
22 us i ng namespace s t d ;
23 us i ng s t d : : s t r i n g ;
24
25 / / Func t i on t o c a l c u l a t e t i me d i f f e r e n c e bet ween two f r ames .
26 i n t di f f _ms ( t i me va l t 1 , t i me va l t 2 )
27 {
28 r e t u r n ( ( ( t 1 . t v_s e c t 2 . t v_s e c ) 1000000 ) + ( t 1 . t v_us e c t 2 . t v_us e c ) ) / 1000;
29 }
30
31 / / Func t i on t o c a l c u l a t e t he a ve r a ge over a s e t of dat a , s t a b i l i z e s t he out pu t of
t he p i x e l c o o r d i n a t e s .
32 voi d Movi ngAverage ( ve c t or <i nt > &xdat a , ve c t or <i nt > &ydat a , doubl e &out x , doubl e &
out y )
33 {
34 doubl e out x_ol d =out x ;
35 doubl e out y_ol d =out y ;
36 i n t c ount i ng = 0;
37 out x = 0;
38 out y = 0;
39
40 / / Compute out y and out x .
41 f o r ( i n t i ndex =0; i ndex < xda t a . s i z e ( ) ; ++i ndex ) {
42 i f ( xda t a [ i ndex ] == 0 && yda t a [ i ndex ] == 0 ) {
53
54
43 out x = out x + out x_ol d ;
44 out y = out y + out y_ol d ;
45 ++c ount i ng ;
46 }
47 e l s e {
48 out x = xda t a [ i ndex ] + out x ;
49 out y = yda t a [ i ndex ] + out y ;
50 }
51 }
52
53 out x = out x / xda t a . s i z e ( ) ;
54 out y = out y / yda t a . s i z e ( ) ;
55
56 i f ( c ount i ng == xda t a . s i z e ( ) ) {
57 out x = 0;
58 out y = 0;
59 }
60
61 / / s h i f t e v e r y t h i n g
62 f o r ( i n t i ndex = ( xda t a . s i z e ( ) 1) ; i ndex > 0; i ndex ) {
63 xda t a [ i ndex ] = xda t a [ i ndex 1] ;
64 yda t a [ i ndex ] = yda t a [ i ndex 1] ;
65 }
66
67 }
68
69 / / Conver t s an I n t t o an s t r i n g t o s ave p i c t u r e s wi t h a number .
70 s t r i n g c o n v e r t I n t ( i n t number )
71 {
72 s t r i n g s t r e a m s s ;
73 s s << " . / i mages / " ;
74 s s << number ;
75 s s << " . j pg " ;
76 r e t u r n s s . s t r ( ) ;
77 }
78
79 / / The Red d e t e c t i o n f u n c t i o n us i ng HSV t h r e s h o l d i n g and c a l c u l a t i n g t he c e n t e r of
mass .
80 voi d Det ect Red ( cv : : Mat &image , i n t &p i c x c e n t e r , i n t &p i c y c e n t e r , bool &gaus ) {
81 / / Def i ne s t a r t val ues , ge t i mage wi dt h and h e i g h t .
82 i n t nc= i mage . rows ;
83 i n t nl = i mage . c o l s ;
84 i n t x c e n t e r = 0 ;
85 i n t y c e n t e r = 0 ;
86 i n t count = 0 ;
87
88 / / Gaus s i an Bl ur s e t t i n g s used i n t he HSV De t e c t i o n Al gor i t hm .
89 cv : : Si z e g a u s _ s i z e ;
90 g a u s _ s i z e . wi dt h = 13;
91 g a u s _ s i z e . h e i g h t = 13;
92
93 cv : : Mat hs v_f r ame ;
94
95 / / Conve r t i ng t o t he HSV c o l o r space , t h i s gi ve s much b e t t e r r e c o g n i t i o n r e s u l t s
t ha n us i ng RGB t h r e s h o l d i n g f o r t h i s pur pos e .
96 cv : : c vt Col or ( image , hsv_f r ame , CV_BGR2HSV) ;
97
98 i f ( gaus ) {
99 cv : : Ga us s i a nBl ur ( hsv_f r ame , hsv_f r ame , ga us _s i z e , 0 , 0) ;
100 }
101
102 / / Cycl e t hr ough a l l p i x e l s and check f o r t h r e s h o l d .
103 f o r ( i n t j =0; j <nl ; j ++) {
UAV Thesis UAV Camera System
APPENDIX C. C++ CODE 55
104 f o r ( i n t i =0; i <nc ; i ++) {
105 / / Each p i x e l i s c o l o r e d bl a c k i f i t i s d e t e c t e d beyond a c e r t a i n t r e s h o l d .
106 / / Usi ng t he HSV f r ame f o r checki ng t he p i x e l s , t ha n c o l o r i n g them i n t he
o r i g i n a l p i c t u r e .
107 i f ( ( hs v_f r ame . at <cv : : Vec3b >( i , j ) [ 0 ] > 160 && hs v_f r ame . at <cv : : Vec3b >( i , j )
[ 1 ] > 150 && hs v_f r ame . at <cv : : Vec3b >( i , j ) [ 2 ] > 35) | | ( hs v_f r ame . at <cv
: : Vec3b >( i , j ) [ 0 ] < 15 && hs v_f r ame . at <cv : : Vec3b >( i , j ) [ 1 ] > 170 &&
hs v_f r ame . at <cv : : Vec3b >( i , j ) [ 2 ] > 35) ) {
108 i mage . at <cv : : Vec3b >( i , j ) [ 0] = 0 ;
109 i mage . at <cv : : Vec3b >( i , j ) [ 1] = 0 ;
110 i mage . at <cv : : Vec3b >( i , j ) [ 2] = 0 ;
111
112 x c e n t e r = x c e n t e r + i ;
113 y c e n t e r = y c e n t e r + j ;
114 count ++;
115 }
116 }
117 }
118
119 / / Now c e n t e r of mass c a l c u l a t i o n , onl y i f r ed p i x e l s d e t e c t e d .
120 i f ( count ! =0) {
121 x c e n t e r = i n t ( x c e n t e r / count ) ;
122 y c e n t e r = i n t ( y c e n t e r / count ) ;
123
124 / / Col or t he c e n t e r of mass of t he d e t e c t e d p i x e l s .
125 / / Cycl e t hr ough t he p i x e l s ar ound t he c e n t e r
126 f o r ( i n t j =( yc e nt e r 2) ; j <( y c e n t e r +2) ; j ++) {
127 f o r ( i n t i =( xc e nt e r 2) ; i <( x c e n t e r +2) ; i ++) {
128 i f ( i >=0 && j >=0 && i <= nc && j <= nl ) / / Cl i p t o bounda r i e s .
129 {
130 i mage . at <cv : : Vec3b >( i , j ) [ 0] = 255 ;
131 i mage . at <cv : : Vec3b >( i , j ) [ 1] = 255 ;
132 i mage . at <cv : : Vec3b >( i , j ) [ 2] = 255 ;
133 }
134 }
135 }
136 }
137
138 / / Expor t c a l c u l a t e d c e n t e r of mass X and Y
139 p i c x c e n t e r = x c e n t e r ;
140 p i c y c e n t e r = y c e n t e r ;
141 }
142
143 / / From i mage p i x e l c o o r d i n a t e s , t he r e a l x and y c o o r d i n a t e s a r e c a l c u l a t e d .
144 voi d Ca l c u l a t e P o s i t i o n ( doubl e &p i c x c e n t e r , doubl e &p i c y c e n t e r , i n t &p i c h e i g h t , i n t &
pi cwi dt h , doubl e &he i ght , doubl e &a ngl e be t a , doubl e &a n g l e t h e t a , doubl e &
a n g l e d e l t a , doubl e &r e l a t i v e x , doubl e &r e l a t i v e y )
145 {
146 / / I mpl e me nt a t i on of t he f u n c t i o n t o c a l c u l a t e t he r e l a t i v e d i s t a n c e t o t he
o b j e c t .
147 / / Needs FoV, P o s i t i o n Angle , Hei ght of camer a aboven ground , c o r d i n a t e s of
o b j e c t i n p i c t u r e f r ame .
148
149 / / De c l a r e two temp v a r i a b l e s .
150 doubl e temp ;
151 doubl e temp2 ;
152
153 / / F i r s t l y Ca l c u l a t i o n of Re l a t i v e X i s done .
154 temp = a n g l e b e t a + a n g l e d e l t a / 2 a n g l e d e l t a p i c x c e n t e r / ( p i c h e i g h t 1) ;
155 temp2 = h e i g h t t a n ( temp PI / 1 8 0 . 0 ) ;
156
157 / / Secondl y Re l a t i v e y i s c a l c u l a t e d .
158 temp = ( a n g l e t h e t a / ( pi cwi dt h 1) ) ( p i c y c e n t e r pi c wi dt h / 2 . 0 ) ;
UAV Camera System UAV Thesis
56
159 r e l a t i v e y = t emp2 t a n ( temp PI / 1 8 0 . 0 ) ;
160 r e l a t i v e x = t emp2 ;
161 }
162
163 / / Ve l o c i t y as t he d i f f e r e n c e bet ween t he c u r r e n t mass and f or mer mass c e n t e r over
t i me .
164 voi d Ca l c u l a t e Ve l o c i t y ( doubl e &x , doubl e &y , doubl e &x_ol d , doubl e &y_ol d , doubl e &
f ps , doubl e &v e l o c i t y , doubl e &vel x , doubl e &ve l y )
165 {
166 doubl e temp ;
167 doubl e dx = x x_ol d ;
168 doubl e dy = y y_ol d ;
169
170 ve l x = dx f ps ;
171 ve l y = dy f ps ;
172
173 temp = dxdx + dydy ;
174 temp = s q r t ( temp ) ;
175 v e l o c i t y = temp f ps ;
176 }
177
178 voi d Ca l c u l a t e S t e e r i n g ( doubl e &pi cx , doubl e &pi cy , cv : : Si z e &s i z e , uns i gned c ha r &
s t e e r i n g )
179 {
180 i f ( pi c x == 0 && pi c y == 0) / / no o b j e c t
181 s t e e r i n g =85;
182 e l s e i f ( pi c x < ( s i z e . h e i g h t ) / 3 )
183 {
184 i f ( pi c y < ( s i z e . wi dt h ) / 3 ) {
185 s t e e r i n g =0; }
186 e l s e i f ( pi c y < ( 2( s i z e . wi dt h ) / 3 ) ) {
187 s t e e r i n g =60; }
188 e l s e { s t e e r i n g =204; }
189 }
190 e l s e i f ( pi c x < ( 2( s i z e . h e i g h t ) / 3 ) )
191 {
192 i f ( pi c y < ( s i z e . wi dt h ) / 3 ) {
193 s t e e r i n g =15; }
194 e l s e i f ( pi c y < ( 2( s i z e . wi dt h ) / 3 ) ) {
195 s t e e r i n g =85; }
196 e l s e { s t e e r i n g =240; }
197 }
198 e l s e
199 {
200 i f ( pi c y < ( s i z e . wi dt h ) / 3 ) {
201 s t e e r i n g =51; }
202 e l s e i f ( pi c y < 2( s i z e . wi dt h ) / 3 ) {
203 s t e e r i n g =195; }
204 e l s e { s t e e r i n g =255; }
205 }
206 }
207
208 # e n d i f / / WEBCAMCALC_H_INCLUDED
webcam.cpp
1 # i n c l u d e " webcamCalc . h"
2
3 cv : : Po i n t 2 f pt ;
4
5 i n t main ( i nt , c ha r )
6 {
7 / / De c l a r e s t e e r s i g n a l s .
8 uns i gned c ha r s t e e r i n g = 85 ;
UAV Thesis UAV Camera System
APPENDIX C. C++ CODE 57
9
10 / / I n p u t s .
11 doubl e he i ght , a nbe t a ;
12 p r i n t f ( " Pl e s e e n t e r t he camer a h e i g h t i n met er : \ n" ) ;
13 s c a nf ( "%l f " ,& h e i g h t ) ;
14 p r i n t f ( " Pl e s e e n t e r t he camer a a ngl e i n degr e e : \ n" ) ;
15 s c a nf ( "%l f " ,&a nbe t a ) ;
16
17 / / Open t he d e f a u l t camera , i s nor mal l y / dev / vi deo0 on l i n u x .
18 cv : : Vi deoCapt ur e c a p t u r e ( 0 ) ;
19
20 / / Check i f c a p t u r e i s opened .
21 i f ( ! c a p t u r e . i sOpened ( ) ) {
22 p r i n t f ( " Camera f a i l e d t o open ! \ n" ) ;
23 r e t u r n 1;
24 }
25
26 c a p t u r e . s e t ( CV_CAP_PROP_FRAME_WIDTH, 320) ;
27 c a p t u r e . s e t ( CV_CAP_PROP_FRAME_HEIGHT, 240) ;
28
29 / / Bool eans t o s ee i f t he v a r i o u s a l g o r i t h ms a r e r unni ng .
30 bool n i g h t = f a l s e ;
31 bool r ed = f a l s e ;
32 bool f l ow = f a l s e ;
33 bool cap = f a l s e ;
34 bool g a u s s i a n = f a l s e ;
35 bool r o t a t e = f a l s e ;
36
37 / / De c l a r e v a r i a b l e s f o r t he p o s i t i o n c a l c u l a t i o n a l gor i t hm , t he y wi l l nor mal l y
be s e t by o t h e r f u n c t i o n s as ba r ome t e r ed .
38 i n t pi c x = 0 , pi c y = 0;
39 ve c t or <i nt > x _ p i x e l _ d a t a (LENGTH_AVERAGE, 0 ) ; / / Cont a i ns t he l a s t x _ pi x e l
l o c a t i o n s .
40 ve c t or <i nt > y _ p i x e l _ d a t a (LENGTH_AVERAGE, 0 ) ; / / Cont a i ns t he l a s t y _ pi x e l
l o c a t i o n s .
41 doubl e aver age_x =0;
42 doubl e aver age_y =0;
43
44 doubl e t h e t a = 4 4 . 9 ;
45 doubl e d e l t a = 3 3 . 8 ;
46 doubl e r e a l x = 0 , r e a l x _ o l d = 0;
47 doubl e r e a l y = 0 , r e a l y _ o l d = 0;
48 doubl e s peed = 0 , s peedx = 0 , s peedy = 0;
49
50 / / De c l a r e v a r i a b l e s f o r t he c o n t r a s t and b r i g h t n e s s a l g o r i t h m .
51 doubl e a l pha = 0 . 5 ;
52 i n t be t a = 30;
53
54 / / De c l a r e v a r i a b l e s f o r t he o p t i c a l f l ow a l g o r i t h m .
55 c ons t i n t MAX_COUNT = 100;
56 cv : : Te r mCr i t e r i a t e r mc r i t ( CV_TERMCRIT_ITER| CV_TERMCRIT_EPS, 2 0 , 0 . 0 3 ) ;
57 cv : : Si z e s ubPi xWi nSi ze ( 10 , 10) , wi nSi ze ( 31 , 31) ;
58 cv : : ve c t or <cv : : Poi nt 2f > p o i n t s [ 2 ] ;
59
60 / / Image s t o r e p l a c e s and s i z e s .
61 cv : : Mat frame , pr evGr ay , gr ay , image , c a pt ur e f r a me ;
62
63 / / Set t he t i me r f o r t he f ps c a l c u l a t i o n .
64 s t r u c t t i me va l s t a r t , end ;
65 doubl e f ps , s ec ;
66 i n t c ou n t e r = 0;
67 / / s t a r t =t i me (NULL) ;
68 ge t t i me of da y (& s t a r t , NULL) ;
UAV Camera System UAV Thesis
58
69
70 / / S e t t i n g up a f i l e t o wr i t e t o .
71 of s t r e a m r e s u l t s ;
72 r e s u l t s . open ( " r e s u l t s . t x t " ) ;
73 r e s u l t s << " Sec , Pi ct ur eX , Pi ct ur eY , AverageX , AverageY , RealX , RealY , SpeedX , SpeedY ,
Speed \ n" ;
74
75
76 / / Get f i r s t f r ame and not e i t s s i z e .
77 c a p t u r e >> f r ame ;
78
79 cv : : Si z e s i z e = f r ame . s i z e ( ) ;
80 / / p r i n t f ( " Hei ght , Width : %i , %i \ n " , s i z e . he i ght , s i z e . wi dt h ) ;
81
82
83 / / Set t o r e c o r d vi deo and s ave t o f i l e .
84 /
85 cv : : Vi deoWr i t er r e c or d ( " webcam0 . a vi " , CV_FOURCC( D , I , V , 3 ) , 11 , f r ame . s i z e
( ) , t r u e ) ;
86 i f ( ! r e c or d . i sOpened ( ) ) {
87 p r i n t f ( " Vi deoWr i t er f a i l e d t o open ! \ n " ) ;
88 r e t u r n 1;
89 }
90 /
91
92 / / Open a window t o d i s p l a y c a p t u r e d i mages .
93 cv : : namedWindow( " vi deo " , 1) ;
94
95 / / S t a r t i n g t he l oop wher ei n each f r ame i s pr oc e s s e d . Thi s i s t he b a s i s of t he
a l g o r i t h m .
96 f o r ( ; ; )
97 {
98 / / Get a new f r ame from camer a .
99 c a p t u r e >> f r ame ;
100
101 / / cv : : r e s i z e ( c a pt ur e f r a me , frame , cv : : Si z e ( c a pt ur e f r a me . c o l s / s c a l e f a c t o r ,
c a pt ur e f r a me . rows / s c a l e f a c t o r ) ) ;
102
103 / / Rot a t e f unc t i on , f o r i f t he camer a i s ups i de down , t u r n 180 de gr e e s i mage .
104 i f ( r o t a t e )
105 cv : : f l i p ( frame , frame , 1) ;
106
107 / / The b a s i c a l g o r i t h m t o d e t e c t t he r ed p i x e l s , c a l c u l a t e t he c e n t e r of mass
and f i n d t he r e l a t i v e d i s t a n c e .
108 i f ( r ed ) {
109 Det ect Red ( frame , pi cx , pi cy , g a u s s i a n ) ;
110 x _ p i x e l _ d a t a [ 0] = pi c x ;
111 y _ p i x e l _ d a t a [ 0] = pi c y ;
112 Movi ngAverage ( x_pi xe l _da t a , y_pi xe l _da t a , aver age_x , aver age_y ) ;
113 r e a l x _ o l d = r e a l x ;
114 r e a l y _ o l d = r e a l y ;
115 Ca l c u l a t e S t e e r i n g ( aver age_x , aver age_y , s i z e , s t e e r i n g ) ;
116 p r i n t f ( " St e e r i n g Si gn a l %i \ n" , s t e e r i n g ) ;
117
118 i f ( pi c x ! = 0 && pi c y ! = 0) {
119 p r i n t f ( " Obj ec t De t e c t e d on i mage a t %f , %f \ n" , aver age_x , aver age_y ) ;
120 Ca l c u l a t e P o s i t i o n ( aver age_x , aver age_y , s i z e . he i ght , s i z e . wi dt h , he i ght ,
anbet a , t h e t a , d e l t a , r e a l x , r e a l y ) ;
121 p r i n t f ( " Obj ec t Real Di s t a nc e %f , %f \ n" , r e a l x , r e a l y ) ;
122 }
123 }
124
UAV Thesis UAV Camera System
APPENDIX C. C++ CODE 59
125 / / Anot her a l gor i t hm , not compl et e , i t us e s HSV c o l o r s pace and t h r e s h o l d i n g
and Canny edge d e t e c t i o n t o f i n d o b j e c t s .
126
127 / / The o p t i c a l f l ow al gor i hm , i s t o sl ow on t he embedded p l a t f or m
u n f o r t u n a t e l y .
128 / / The code i s t a ke n from t he t u t o r i a l code of t he OpenCV l i b r a r y .
129 i f ( f l ow ) {
130 f r ame . copyTo ( i mage ) ;
131 c vt Col or ( image , gr ay , CV_BGR2GRAY) ;
132
133 / / The good f e a t u r e s t o t r a c k a l g o r i t h m has s e t p o i n t s t o be t r a c k e d i n a
v e c t o r .
134 i f ( ! p o i n t s [ 0 ] . empty ( ) )
135 {
136 cv : : ve c t or <uchar > s t a t u s ;
137 cv : : ve c t or < f l o a t > e r r ;
138 i f ( pr evGr ay . empty ( ) )
139 gr ay . copyTo ( pr evGr ay ) ;
140
141 / / The Real o p t i c a l f l ow a l g o r i t h m us i ng Lucas Kanade t r a c k i n g .
142 / / Bot h v e c t o r s of t he c u r r e n t and l a s t p o s i t i o n a r e s t o r e d .
143 cal cOpt i cal Fl owPyr LK ( pr evGr ay , gr ay , p o i n t s [ 0 ] , p o i n t s [ 1 ] , s t a t u s , e r r ,
wi nSi ze , 3 , t e r mc r i t , 0 , 0. 001) ;
144 s i z e _ t i , k ;
145 f o r ( i = k = 0; i < p o i n t s [ 1 ] . s i z e ( ) ; i ++ )
146 {
147 i f ( ! s t a t u s [ i ] )
148 c on t i n u e ;
149
150 p o i n t s [ 1 ] [ k++] = p o i n t s [ 1 ] [ i ] ;
151
152 / / Thi s c o l o r s t he i mage on t he s p e c i f i e d s p o t s and dr aws t h e i r
v e c t o r s .
153 i f ( p o i n t s [ 1 ] [ i ] . x<690 && p o i n t s [ 1 ] [ i ] . x >30) {
154 c i r c l e ( image , p o i n t s [ 1 ] [ i ] , 3 , cv : : Sc a l a r ( 0 , 0 , 255) , 1, 4) ;
155 l i n e ( image , p o i n t s [ 0 ] [ i ] , p o i n t s [ 1 ] [ i ] , cv : : Sc a l a r ( 255 , 255 , 255) , 2
, CV_AA, 0) ;
156 }
157 }
158 p o i n t s [ 1 ] . r e s i z e ( k ) ;
159 }
160 }
161
162 / / Takes an i mage of t he c u r r e n t f r ame and s t o r e s i t .
163 i f ( cap ) {
164 / / s avef r ame i s t he r e s i z e d f r ame
165 cv : : Mat s avef r ame ;
166 s avef r ame = cvCr eat eMat ( f r ame . c o l s / s c a l e f a c t o r s t o r e , f r ame . rows /
s c a l e f a c t o r s t o r e , CV_8UC3) ;
167 / / r e s i z i n g
168 cv : : r e s i z e ( frame , s avef r ame , cv : : Si z e ( f r ame . c o l s / s c a l e f a c t o r s t o r e , f r ame .
rows / s c a l e f a c t o r s t o r e ) ) ;
169 cv : : i mwr i t e ( c o n v e r t I n t ( c ou n t e r ) , s avef r ame ) ;
170 cap = f a l s e ;
171 }
172
173 / / F i n a l l y showi ng t he f r ame on s c r e e n .
174
175 i f ( ! f l ow ) {
176 cv : : imshow( " vi deo " , f r ame ) ;
177 }
178 e l s e {
179 cv : : imshow( " vi deo " , i mage ) ;
UAV Camera System UAV Thesis
60
180 s t d : : swap ( p o i n t s [ 1 ] , p o i n t s [ 0 ] ) ;
181 swap ( pr evGr ay , gr ay ) ;
182 }
183
184 / / Ca l c u l a t e t he f ps and t he n p r i n t .
185 / / s l e e p ( 1 ) ;
186 ge t t i me of da y (&end , NULL) ;
187 ++ c o un t e r ;
188 s ec = di f f _ms ( end , s t a r t ) / 1 0 0 0 . 0 ;
189 f ps = 1 . 0 / s ec ;
190 ge t t i me of da y (& s t a r t , NULL) ;
191 p r i n t f ( " FPS = %f \ n" , f ps ) ;
192
193 i f ( r ed && pi c x ! = 0 && pi c y ! = 0 ) {
194 Ca l c u l a t e Ve l o c i t y ( r e a l x , r e a l y , r e a l x_ol d , r e a l y_ol d , f ps , speed , speedx ,
s peedy ) ;
195 p r i n t f ( " Obj ec t Speed %f , X: %f , Y: %f \ n" , speed , speedx , s peedy ) ;
196 }
197
198 i f ( aver age_x == 0 && aver age_y == 0) {
199 r e a l x = 0 , r e a l y = 0 , s peedx = 0 , s peedy = 0 , s peed = 0;
200 }
201
202 r e s u l t s << s ec << " , " << pi c x << " , " << pi c y << " , " << aver age_x << " , " <<
aver age_y << " , " << r e a l x << " , " << r e a l y << " , " << s peedx << " , " <<
s peedy << " , " << s peed <<" \ n" ;
203
204 / / I f r e c or d i s a c t i v a t e d , t he n add f r ame t o r e c or de d vi deo .
205 / / r e c or d << f r ame ;
206
207 / / De t e c t i n g t he us e r i n p u t but t ons , wi t h t h e i r a c c or di ng a c t i o n s s e t .
208 c ha r c = ( c ha r ) cv : : wai t Key ( 1 ) ;
209 i f ( c == 27 )
210 br eak ;
211 s wi t c h ( c )
212 {
213 c a s e f :
214 / / S t a r t i n g t he o p t i c a l f l ow a l gor i t hm , f i r s t f i n d i n g t he good
f e a t u r e s t o t r a c k .
215 f l ow = ! f l ow ;
216 f r ame . copyTo ( i mage ) ;
217 c vt Col or ( image , gr ay , CV_BGR2GRAY) ;
218 goodFeat ur es ToTr ack ( gr ay , p o i n t s [ 1 ] , MAX_COUNT, 0. 01 , 10 , cv : : Mat ( ) ,
3 , 0 , 0 . 0 4 ) ;
219 c or ne r SubPi x ( gr ay , p o i n t s [ 1 ] , subPi xWi nSi ze , cv : : Si z e ( 1, 1) ,
t e r mc r i t ) ;
220 br eak ;
221 c a s e c :
222 / / Capt ur e t he ne xt p i c t u r e .
223 cap = t r u e ;
224 br eak ;
225 c a s e r :
226 / / S t a r t s d e t e c t i n g r ed s p o t s on t he p i c t u r e .
227 r ed = ! r ed ;
228 br eak ;
229 c a s e n :
230 / / Ac t i v a t e s t he mode t o c o r r e c t b r i g h t n e s s and c o n t r a s t .
231 n i g h t = ! n i g h t ;
232 br eak ;
233 c a s e g :
234 / / Bl ur i mage HSV.
235 g a u s s i a n = ! g a u s s i a n ;
236 br eak ;
UAV Thesis UAV Camera System
APPENDIX C. C++ CODE 61
237 c a s e t :
238 / / I f i mage i s ups i de down , t h i s r o t a t e s t he i mage wi t h 180 de gr e e s .
239 r o t a t e = ! r o t a t e ;
240 br eak ;
241 d e f a u l t :
242 ;
243 }
244 }
245
246 / / The camer a wi l l be d e i n i t i a l i z e d a u t o ma t i c a l l y i n Vi deoCapt ur e d e s t r u c t o r .
247 / / The r e c or de d vi deo wi l l be c l o s e d a u t o ma t i c a l l y i n t he Vi deoWr i t er
d e s t r u c t o r .
248 r e s u l t s . c l o s e ( ) ;
249 r e t u r n 0;
250 }
UAV Camera System UAV Thesis
62
UAV Thesis UAV Camera System
Appendix D
Results
Table D.1: Measurement results accompanying Figure 5.7.
Frame Sec Picture X Picture Y Average X Average Y Real X Real Y Speed X Speed Y Speed
35 0,071 46 43 46 43 3,181 -0,940 0 0 0
36 0,075 46 43 46 43 3,181 -0,940 0 0 0
37 0,074 46 43 46 43 3,181 -0,940 0 0 0
38 0,076 46 43 46 43 3,181 -0,940 0 0 0
39 0,071 47 43 46,2 43 3,175 -0,938 -0,097 0,029 0,101
40 0,075 46 43 46,2 43 3,175 -0,938 0 0 0
41 0,075 46 43 46,2 43 3,175 -0,938 0 0 0
42 0,076 46 43 46,2 43 3,175 -0,938 0 0 0
43 0,071 46 43 46,2 43 3,175 -0,938 0 0 0
44 0,075 47 43 46,2 43 3,175 -0,938 0 0 0
45 0,075 46 43 46,2 43 3,175 -0,938 0 0 0
46 0,075 46 43 46,2 43 3,175 -0,938 0 0 0
47 0,071 46 43 46,2 43 3,175 -0,938 0 0 0
48 0,075 46 43 46,2 43 3,175 -0,938 0 0 0
49 0,075 46 43 46 43 3,181 -0,940 0,091 -0,027 0,095
50 0,075 46 43 46 43 3,181 -0,940 0 0 0
51 0,071 46 43 46 43 3,181 -0,940 0 0 0
52 0,075 46 44 46 43,2 3,181 -0,939 0 0,023 0,023
53 0,075 46 44 46 43,4 3,181 -0,937 0 0,023 0,023
54 0,075 46 43 46 43,4 3,181 -0,937 0 0 0
55 0,071 47 44 46,2 43,6 3,175 -0,933 -0,097 0,052 0,110
56 0,075 48 44 46,6 43,8 3,161 -0,928 -0,182 0,076 0,197
57 0,075 49 45 47,2 44 3,141 -0,920 -0,270 0,102 0,288
58 0,075 50 46 48 44,4 3,114 -0,909 -0,355 0,148 0,384
59 0,071 53 48 49,4 45,4 3,069 -0,887 -0,642 0,302 0,709
60 0,075 55 50 51 46,6 3,018 -0,863 -0,674 0,323 0,747
61 0,076 58 51 53 48 2,957 -0,835 -0,803 0,374 0,886
62 0,076 61 52 55,4 49,4 2,887 -0,804 -0,924 0,402 1,008
63 0,071 65 56 58,4 51,4 2,803 -0,766 -1,176 0,537 1,293
64 0,076 69 60 61,6 53,8 2,719 -0,726 -1,109 0,529 1,229
65 0,076 73 62 65,2 56,2 2,630 -0,685 -1,176 0,532 1,291
66 0,08 76 65 68,8 59 2,546 -0,645 -1,051 0,507 1,167
67 0,066 83 70 73,2 62,6 2,449 -0,597 -1,459 0,718 1,626
68 0,074 88 73 77,8 66 2,355 -0,554 -1,268 0,590 1,399
69 0,078 92 76 82,4 69,2 2,268 -0,514 -1,122 0,505 1,230
70 0,076 96 79 87 72,6 2,186 -0,477 -1,077 0,496 1,186
71 0,071 104 86 92,6 76,8 2,093 -0,434 -1,307 0,603 1,439
72 0,076 109 91 97,8 81 2,013 -0,396 -1,054 0,503 1,168
73 0,076 115 95 103,2 85,4 1,935 -0,359 -1,023 0,486 1,132
Continued on next page
63
64
Table D.1 Continued from previous page
Frame Sec Picture X Picture Y Average X Average Y Real X Real Y Speed X Speed Y Speed
74 0,08 120 101 108,8 90,4 1,860 -0,321 -0,941 0,469 1,051
75 0,067 133 110 116,2 96,6 1,768 -0,278 -1,374 0,650 1,521
76 0,076 142 116 123,8 102,6 1,681 -0,239 -1,143 0,513 1,253
77 0,077 151 120 132,2 108,4 1,593 -0,203 -1,144 0,463 1,234
78 0,076 159 128 141 115 1,509 -0,167 -1,112 0,468 1,207
79 0,072 173 144 151,6 123,6 1,416 -0,127 -1,288 0,563 1,406
80 0,076 182 154 161,4 132,4 1,338 -0,091 -1,028 0,475 1,133
81 0,077 189 162 170,8 141,6 1,269 -0,057 -0,896 0,434 0,996
82 0,072 193 167 179,2 151 1,211 -0,027 -0,797 0,425 0,903
83 0,076 199 171 187,2 159,6 1,160 -0,001 -0,674 0,337 0,754
84 0,076 202 173 193 165,4 1,125 0,015 -0,464 0,211 0,510
85 0,076 201 172 196,8 169 1,102 0,024 -0,294 0,124 0,319
86 0,071 202 172 199,4 171 1,088 0,029 -0,210 0,071 0,222
87 0,076 201 172 201 172 1,078 0,032 -0,119 0,032 0,123
88 0,075 201 171 201,4 172 1,076 0,032 -0,030 -0,001 0,030
89 0,078 202 172 201,4 171,8 1,076 0,031 0 -0,007 0,007
90 0,072 201 171 201,4 171,6 1,076 0,031 0 -0,007 0,007
91 0,076 202 173 201,4 171,8 1,076 0,031 0 0,007 0,007
92 0,076 202 173 201,6 172 1,075 0,032 -0,015 0,007 0,016
93 0,076 202 172 201,8 172,2 1,074 0,032 -0,015 0,007 0,016
94 0,072 202 173 201,8 172,4 1,074 0,033 0 0,007 0,007
95 0,076 201 172 201,8 172,6 1,074 0,033 0 0,007 0,007
96 0,076 201 172 201,6 172,4 1,075 0,033 0,015 -0,007 0,016
97 0,076 202 172 201,6 172,2 1,075 0,032 0 -0,007 0,007
98 0,072 202 173 201,6 172,4 1,075 0,033 0 0,007 0,007
99 0,076 202 173 201,6 172,4 1,075 0,033 0 0 0
100 0,077 202 173 201,8 172,6 1,074 0,033 -0,015 0,006 0,016
101 0,072 202 172 202 172,6 1,073 0,033 -0,016 0 0,016
102 0,076 202 172 202 172,6 1,073 0,033 0 0 0
103 0,076 202 173 202 172,6 1,073 0,033 0 0 0
104 0,075 202 173 202 172,6 1,073 0,033 0 0 0
105 0,073 202 172 202 172,4 1,073 0,033 0 -0,007 0,007
106 0,077 202 172 202 172,4 1,073 0,033 0 0 0
107 0,078 201 172 201,8 172,4 1,074 0,033 0,014 0 0,014
108 0,073 202 172 201,8 172,2 1,074 0,032 0 -0,007 0,007
109 0,076 203 173 202 172,2 1,073 0,032 -0,015 0 0,015
110 0,076 202 172 202 172,2 1,073 0,032 0 0 0
111 0,076 202 173 202 172,4 1,073 0,033 0 0,007 0,007
112 0,071 202 173 202,2 172,6 1,072 0,033 -0,016 0,007 0,017
113 0,075 202 173 202,2 172,8 1,072 0,034 0 0,007 0,007
114 0,078 201 173 201,8 172,8 1,074 0,034 0,029 0,001 0,029
115 0,076 201 173 201,6 173 1,075 0,034 0,015 0,007 0,017
116 0,072 196 170 200,4 172,4 1,082 0,033 0,094 -0,019 0,096
117 0,077 187 169 197,4 171,6 1,099 0,031 0,223 -0,021 0,224
118 0,076 178 166 192,6 170,2 1,127 0,028 0,371 -0,040 0,373
119 0,073 172 165 186,8 168,6 1,163 0,025 0,485 -0,050 0,487
120 0,122 155 163 177,6 166,6 1,222 0,020 0,487 -0,039 0,488
121 0,073 140 162 166,4 165 1,300 0,016 1,073 -0,053 1,075
122 0,071 132 164 155,4 164 1,385 0,014 1,189 -0,033 1,189
123 0,076 119 164 143,6 163,6 1,485 0,013 1,319 -0,006 1,319
124 0,075 111 164 131,4 163,4 1,601 0,013 1,549 0,003 1,549
125 0,078 104 165 121,2 163,8 1,710 0,016 1,397 0,033 1,397
126 0,071 97 167 112,6 164,8 1,812 0,021 1,434 0,076 1,436
127 0,076 94 167 105 165,4 1,911 0,025 1,300 0,052 1,301
128 0,077 94 167 100 166 1,981 0,029 0,911 0,050 0,913
129 0,076 94 167 96,6 166,6 2,031 0,033 0,662 0,049 0,664
Continued on next page
UAV Thesis UAV Camera System
APPENDIX D. RESULTS 65
Table D.1 Continued from previous page
Frame Sec Picture X Picture Y Average X Average Y Real X Real Y Speed X Speed Y Speed
130 0,072 94 167 94,6 167 2,062 0,035 0,426 0,035 0,427
131 0,076 96 165 94,4 166,6 2,065 0,033 0,041 -0,026 0,048
132 0,076 94 169 94,4 167 2,065 0,036 0 0,027 0,027
133 0,075 94 169 94,4 167,4 2,065 0,038 0 0,027 0,027
134 0,072 94 168 94,4 167,6 2,065 0,039 0 0,014 0,014
135 0,076 94 169 94,4 168 2,065 0,041 0 0,027 0,027
136 0,076 94 168 94 168,6 2,071 0,044 0,082 0,042 0,092
137 0,075 94 169 94 168,6 2,071 0,044 0 0 0
138 0,072 94 169 94 168,6 2,071 0,044 0 0 0
139 0,076 94 168 94 168,6 2,071 0,044 0 0 0
140 0,076 94 168 94 168,4 2,071 0,043 0 -0,013 0,013
141 0,073 95 165 94,2 167,8 2,068 0,040 -0,043 -0,043 0,060
142 0,076 91 162 93,6 166,4 2,077 0,033 0,124 -0,092 0,154
143 0,076 95 165 93,8 165,6 2,074 0,029 -0,041 -0,054 0,068
144 0,076 94 169 93,8 165,8 2,074 0,030 0 0,013 0,013
145 0,071 94 169 93,8 166 2,074 0,031 0 0,014 0,014
146 0,121 94 168 93,6 166,6 2,077 0,034 0,026 0,026 0,037
147 0,074 94 165 94,2 167,2 2,068 0,037 -0,127 0,039 0,133
148 0,071 95 165 94,2 167,2 2,068 0,037 0 0 0
149 0,076 94 168 94,2 167 2,068 0,036 0 -0,013 0,013
150 0,076 94 169 94,2 167 2,068 0,036 0 0 0
151 0,076 94 168 94,2 167 2,068 0,036 0 0 0
152 0,072 94 168 94,2 167,6 2,068 0,039 0 0,042 0,042
153 0,076 94 168 94 168,2 2,071 0,042 0,041 0,041 0,058
154 0,076 94 168 94 168,2 2,071 0,042 0 0 0
155 0,076 94 168 94 168 2,071 0,041 0 -0,013 0,013
156 0,073 94 168 94 168 2,071 0,041 0 0 0
157 0,077 94 168 94 168 2,071 0,041 0 0 0
158 0,076 94 168 94 168 2,071 0,041 0 0 0
159 0,076 93 167 93,8 167,8 2,074 0,040 0,041 -0,013 0,043
160 0,072 93 168 93,6 167,8 2,077 0,040 0,044 0,001 0,044
161 0,076 93 168 93,4 167,8 2,081 0,040 0,041 0,001 0,041
162 0,075 94 168 93,4 167,8 2,081 0,040 0 0 0
163 0,072 93 168 93,2 167,8 2,084 0,040 0,044 0,001 0,044
164 0,076 94 169 93,4 168,2 2,081 0,042 -0,042 0,026 0,049
165 0,076 94 168 93,6 168,2 2,077 0,042 -0,041 -0,001 0,041
166 0,076 94 169 93,8 168,4 2,074 0,043 -0,041 0,013 0,043
167 0,071 92 168 93,4 168,4 2,081 0,043 0,089 0,002 0,089
168 0,076 89 166 92,6 168 2,093 0,041 0,167 -0,024 0,169
169 0,076 87 167 91,2 167,6 2,116 0,040 0,297 -0,022 0,297
170 0,076 84 166 89,2 167,2 2,149 0,038 0,434 -0,020 0,434
171 0,072 79 165 86,2 166,4 2,200 0,035 0,711 -0,047 0,712
172 0,077 75 166 82,8 166 2,261 0,033 0,788 -0,016 0,788
173 0,077 71 166 79,2 166 2,328 0,034 0,878 0,013 0,878
174 0,076 68 167 75,4 166 2,404 0,035 0,992 0,015 0,992
175 0,071 61 168 70,8 166,4 2,501 0,039 1,372 0,055 1,373
176 0,076 59 169 66,8 167,2 2,592 0,046 1,194 0,086 1,197
177 0,076 56 170 63 168 2,684 0,053 1,210 0,091 1,213
178 0,075 54 170 59,6 168,8 2,771 0,060 1,167 0,096 1,171
179 0,071 52 170 56,4 169,4 2,858 0,066 1,230 0,086 1,233
180 0,076 49 169 54 169,6 2,927 0,069 0,907 0,040 0,908
181 0,077 50 169 52,2 169,6 2,981 0,070 0,698 0,016 0,699
182 0,071 49 168 50,8 169,2 3,024 0,068 0,607 -0,028 0,608
183 0,076 49 168 49,8 168,8 3,056 0,066 0,415 -0,030 0,416
184 0,076 49 168 49,2 168,4 3,075 0,063 0,253 -0,034 0,255
185 0,075 49 168 49,2 168,2 3,075 0,062 0 -0,020 0,020
Continued on next page
UAV Camera System UAV Thesis
66
Table D.1 Continued from previous page
Frame Sec Picture X Picture Y Average X Average Y Real X Real Y Speed X Speed Y Speed
186 0,071 49 169 49 168,2 3,081 0,062 0,091 0,002 0,091
187 0,076 49 168 49 168,2 3,081 0,062 0 0 0
188 0,076 49 168 49 168,2 3,081 0,062 0 0 0
189 0,077 49 169 49 168,4 3,081 0,064 0 0,020 0,020
190 0,071 50 169 49,2 168,6 3,075 0,065 -0,091 0,019 0,093
191 0,076 49 169 49,2 168,6 3,075 0,065 0 0 0
192 0,076 50 169 49,4 168,8 3,069 0,066 -0,085 0,018 0,087
193 0,074 50 169 49,6 169 3,062 0,068 -0,087 0,018 0,088
194 0,072 49 169 49,6 169 3,062 0,068 0 0 0
195 0,076 50 169 49,6 169 3,062 0,068 0 0 0
196 0,076 49 169 49,6 169 3,062 0,068 0 0 0
197 0,076 50 169 49,6 169 3,062 0,068 0 0 0
198 0,072 50 169 49,6 169 3,062 0,068 0 0 0
199 0,076 49 169 49,6 169 3,062 0,068 0 0 0
200 0,076 49 169 49,4 169 3,069 0,068 0,084 0,002 0,084
(a) Resolution 640 480
Figure D.1: Projection points of pixels under different resolutions; height h = 0.77 m, camera angle = 66

.
Continued on next page
UAV Thesis UAV Camera System
APPENDIX D. RESULTS 67
Continued from previous page
(b) Resolution 320 240
(c) Resolution 176 144
Figure D.1: Projection Points of Pixels under different resolutions.
UAV Camera System UAV Thesis

You might also like