Embedded Systems

~ Kiran Divekar
 Agenda
Basic Embedded systems
Embedded Linux
Tools, Development
Points to optimize
 Embedded Linux
Run on Limited Resource
Real Time
Dedicated Device Drivers
 Sixnet VersaTRAK IPm

Embedded Linux
PowerPC CPU
16MB DRAM 16MB Flash
Serial & Ethernet Ports
Linux and IEC1131 Programming/Modbus
 Sharp Zaurus

Lineo Embedix
206 MHz StrongARM
64 MB DRAM 16MB Flash
3.5 display (320x240) 64K
colors
Opera browser & Qtopia
QT/Embedded GUI
Cell and Web Phones

Telepong Mobile
Phone

GITWiT Mobile
Aplio/PRO IP Phone
Phone
 TiVO Set Top Box

Home grown port of Embedded Linux

54MHz PowerPC
Multi GB hard disk
 Axis 2120 Network Camera

uCLinux
Built-in Ethernet port
100 MHz ETRAX CPU
16 MB RAM
 Humanoid Robots

Univ. of
Tokyo/Kawanda
Ind.
Dual Pentium CPU
RT-Linux
Height: 53 inches
Weight: 121 lbs.
Isamu
 Humanoid Robots
HOAP

Fujitsu
RT-Linux
Height: 48 cm
Weight: 6 kg
100 units/yr
 What is an Embedded OS?

An "embedded system" is any computer system or

computing device that performs a dedicated function or
is designed for use with a specific embedded software
application.

Embedded systems may use a ROM-based operating

system or they may use a disk-based system, like a PC.
But an embedded system is not usable as a
commercially viable substitute for general purpose
computers or devices.
 What makes a good Embedded
OS?
Modular
Scalable
Configurable
Small footprint
CPU support
Device drivers
etc, etc, etc...
 What is Real Time?

A real time system is one in which the

correctness of the computations not only
depends upon the logical correctness of
the computation but also upon the time at
which the result is produced. If the timing
constraints of the system are not met,
system failure is said to have occurred.

- Donald Gillies
 What is Real Time?

Real time in operating systems:

The ability of the operating system to

provide a required level of service in a
bounded response time.

- POSIX Standard 1003.1

 Hard vs. Soft Real Time

Hard
guaranteed worst-case response times
absolutely, positively, first time every time
Soft
Kinda, sorta, usually
 What makes a good RTOS?

Multi-threaded and pre-emptible

Thread priority has to exist because no
deadline driven OS exists
Must support predictable thread
synchronization mechanisms
A system of priority inheritance must exist
 Who are the Embedded OS
players?
Wind River Systems
VxWorks
pSOS
QNX Software Systems
QNX
Green Hills Software
Integrity
QualComm : REX OS
 Who are the Embedded OS
players?
Mentor Graphics
VRTX
Palm Computing
PalmOS
Symbian
SymbianOS, S60, S80, S90
 Microsoft

Embedded NT/XP
Real-time control
Windows CE (CE.NET)
Internet devices
Pocket PC 2002
Handheld PCs and PDAs
 Commercial Embedded Linux

AMIRIX Embedded Linux

derived from Debian
Coollogic Coollinux
combines Linux and Java for Internet apps
Coventive Xlinux
kernel can be as small as 143KB
Esfia RedBlue Linux
400K, designed for wireless apps
 Commercial Embedded Linux

KYZO Pizza Box Linux

SAMBA based file, print, CD server
Lineo Embedix
supports real time and high availability apps
LynuxWorks BlueCat
general purpose embedded solution
MontaVista Linux
general purpose embedded solution
 Commercial Embedded Linux

Neoware NeoLinux
Red Hat derived for information appliances
PalmPalm Tynux
Internet appliuance and multimedia
Red Hat Embedded Linux
general purpose embedded solution
RedSonic Red-Ice Linux
runs from DiskonChip or CompactFlash
 Commercial Embedded Linux

Bluecat Linux
TimeSys Linux GPL
low latency enhanced kernel
Tuxia TASTE
distro targets Internet appliances
 Open Source Embedded Linux
Embedded Debian Project
convert Debian to an embedded OS
ETLinux
for PC104 SBCs
uCLinux
for microprocessors that dont have MM
uLinux (muLinux)
distro fits on a single floppy
 Commercial Linux RTOS

FSMLabs - Open RT Linux

Lineo - Embedix Realtime
LynuxWorks - BlueCat RT
MontaVista Software - Real Time Extensions
REDSonic - REDICE Linux
TimeSys - Linux/Real-Time
 Open Source Linux RTOS

ART Linux - real time extension based on

RTLinux
KURT - event schedules with 10us resolution
Linux-SRT - for soft real time apps like
multimedia
Qlinux - provides Quality of Service guarantees
RTAI - hard Real Time Application Interface
 RTLinux

A hard real-time mini operating system

runs Linux as its lowest priority execution thread
Linux thread completely preemptible
Real time threads and interrupt handlers never
delayed by non-realtime operations
Supports user level programming
MiniRTL implementation fits on a floppy
 Whats so special about Linux?

Multiple choices vs. sole source

Source code freely available
Robust and reliable
Modular, configurable, scalable
Superb support for networking and
Internet
No runtime licenses
Large pool of skilled developers
Whats so special about Linux?
Process and Virtual Addressing

GUI Applications
Applications
GUI
Kernel
Device Driver

Hardware
 Linux on PC
Memory

#
#
#hello world :-( PC User Apps
#
#

Keybo Kernel
ard
IO
Devices
Disk File File System
System
File system on disk
Linux on Embedded System

Memory

User Apps
CPU IO
Kernel
RAM ROM

File System

File system on chip

 UI
Printf Scanf

Kernel Kernel
Device
Driver Display Adaptor Keyboard Driver

#
# Keybo
#hello world :-( ard
#
#
 UI on Embedded System
Printf Scanf

Kernel Kernel
Device
Driver Display Adaptor Keyboard Driver

Any Output Any Input

Device Device
 Linux on Embedded System

LCD Button

Touch
UART
Replacing Replacing screen
Displayer Keyboard
LED UART

Network Network
 UI on UART
Printf Scanf

Kernel
Kernel
Device
Driver UART Driver
Txd Rxd
Layout
System Bootloader
Kernel
Memory ROM
Rootdisk
Usage
(Static)

RAM
 System Bootloader
Kernel
Memory ROM
Rootdisk
Usage
(Running) Kernel

Rootdisk
RAM

User
Memory
Debugging Using The Ethernet

Ethernet

RS232
 Linux

Local kernel

Booting
RAM

RAM Disk

0x0xffe00000
u-boot
Zipped Linux
ROM kernel
Zipped root
image
Remote Booting (U-Boot on MPC860)
Zipped Linux
Linux
kernel 192.168.0.3

kernel
DHCP server

RAM TFTP server

Zipped kernel
Linux image
(/home/TFTPdir/uImage)

NFS server
0x0xffe00000

u-boot MyIP: 192.168.0.253 Root Image for

Server IP: 192.168.0.3 MPC860 Linux
ROM Kernel Name: uImage (/home/NFSdir/)
NFS position: /home/NFSdir
 Bootloader
Redboot
U-boot
Homemade bootloader
Grub ???
Function of Bootloader

Bootloader ROM
Kernel

Rootdisk
Init Board
Also support some basic
debugging function
Loading Kernel
From ROM or From
Remote Server RAM
Load Root Image
Starting Kernel
 Root Disk
Host Target Development environment
Arm targets : Linux supports 10 distinct ARM
Advanced RISC Machine

CPUs, 16 different platforms, and more than 200 related boards

http://www.arm.linux.org.uk
SCSI, USB, I2C buses
Parallel, Serial ports
MTD storage
 Host Target
Host: Development machine
Target: Embedded target system

Make ARCH=arm CROSS-

COMPILE=arm-linux-
 Cross-compilation
Contains: gcc, binutils, glibc headers
uClibc based toolchains (create much smaller
binaries):
Download one from http://buildroot.uclibc.org/
Handhelds.org (glibc: bigger binaries):
ftp://ftp.handhelds.org/projects/toolchain/
Build it yourself using crosstool (glibc)
Code-sourcery.
 Components
Kernel uImage
Loadable modules if non-GPL
RFS
Init scripts
Busybox (configrable)
Apps
Done !!!
 Kernel uImage
Configuration
Make allnoconfig ?
Arch/<arch>/configs/

B.S.P. [ Board Support Package ]

 RFS
Jffs2 images
Squashfs images
Initrd.gz for GRUB
# Create an empty ramdisk image
dd if=/dev/zero of=/tmp/ramdisk.img bs=$BLKSIZE count=$RDSIZE
# Make it an ext2 mountable file system
/sbin/mke2fs -F -m 0 -b $BLKSIZE /tmp/ramdisk.img $RDSIZE
# Mount it so that we can populate
mount /tmp/ramdisk.img /mnt/initrd -t ext2 -o loop=/dev/loop0

# Populate the filesystem (subdirectories)

..# Finish up...........
umount /mnt/initrd
gzip -9 /tmp/ramdisk.img
 Download File To
Development Board
PC
Development
1010010 Board
1101100
1111011
1110101 ROM
1101010
 RFS
Jffs2, squashfs images
Netboot
setenv x_bootargs_root 'root=/dev/nfs rw nfsroot=$(serverip):$(rootpath) ip=$
(ipaddr):$(serverip):$(bootargs_end);'
setenv serverip 192.168.1.1
setenv ipaddr 192.168.1.10
setenv rootpath /home/test/exports/

NFS exports : /etc/exports

Export directory with proper perms
 NFS

PC
With NFS Server
Development
Ethernet Board
1111011 Cable
1110101 (Running
1101010 Linux)
 uclibc
C Compiled with Compiled statically
program shared libraries

glibC uClibc glibC uClibc

Hello 4.6 K 4.4 K 475 K 25 K

World

Busybox 245 K 231 K 843 K 311 K

 Tools
Chroot: runs with specified root
Minicom:
Sshd : ssh server
Lighttpd : HTTP server
UI tookkits: I dont know
Browsers: Webkit
 Files

ls Busybox
/dev/console Device files
myapp cross-compiled application

watchdog stability module

 Optimizations
1. kernel optimizations
2. memory
3. performance
4. Size
5. Security
 Startup Optimizations
System startup /etc/inittab
SysV init:
Starts services sequentially. Waits for the
current startup script to be complete to start the
next one! While dependencies exist, some tasks
can be run in parallel!
Initng: http://initng.thinktux.net
New alternative to SysV init, which can start
services in parallel, as soon as their
preconditions are met
 Memory Optimizations
Upfront memory allocations
Implement and manage memory pools
This reduces the overheads of allocation and
freeing of memory.
Adds pool maintenance overhead
 Security Tools
Selinux
Grsec
Lids
Lidsadm
Listconf
Tools which can set capabilities, restrictions
and access permissions on a file.
 Network Performance
TCP / IP tuning
/etc/sysctl.conf
net/core/rmem_max = 8738000
net/core/wmem_max = 6553600
net/ipv4/tcp_rmem = 8192 873800 8738000
net/ipv4/tcp_wmem = 4096 655360 6553600
NAPI for drivers.
 Using processor acceleration
instructions
liboil - http://liboil.freedesktop.org/
Library of functions optimized for special
instructions from several processors
(Altivec, MMX, SSE, etc.)
Mainly functions implementing loops on
data arrays:
type conversion, copying, simple
arithmetics, direct cosine transform,
random number generation...
 Prelinking (1)
Applies to executables using shared libraries
To load and start an executable, the
dynamic linker has a significant amount of
work to do (mainly address relocation)
It can take a lot of time for executables
using many shared libraries!
In many systems in which executables and
shared libraries never change, the same
job is done every time the executable is
started.
 Use simpler Unix executables
Big, feature rich executables take time to load.
Particularly true for shell scripts calling the
bash shell!
Idea: replace standard Unix / GNU executables
by lightweight, simplified implementations by
busybox (http://busybox.net).
 Shells: reducing forking
fork / exec system calls are very heavy.
Because of this, calls to executables from
shells are slow.
Select Shells -> Standalone shell in
busybox configuration to make the busybox
shell call applets whenever possible.
Pipes and back-quotes are also implemented
by fork / exec.
You can reduce their usage in scripts. E.g.
cat /proc/cpuinfo | grep model
Replace it with: grep model proc/cpuinfo
 Use faster filesystems
Run faster by using the most appropriate
filesystems!
Compressed read-only filesystem (block
device):
use SquashFS (http://
squashfs.sourceforge.net)
NAND flash storage: if you do not need
compression,
 Use faster filesystems (2)
Use RAM filesystems for temporary,
speed critical files with no need for
permanent storage. Documentation/
filesystems/tmpfs.txt
Good to benchmark your system with JFS
or XFS too. XFS is reported to be the
fastest to mount (good for startup time),
and JFS to have the lowest CPU
utilization.
Speed up applications with tmpfs
When enough RAM is available, the OS keeps recently
accessed files and applications in RAM (page cache).
This significantly speeds up any new usage. However,
depending on system activity, this may not last long.
For programs that need fast startup even if they haven't
been run for a long time:
copy them to a tmpfs filesystem at system startup!
Caution: don't use ramdisks instead!
Ramdisks duplicate files in RAM and unused space
cannot be reclaimed.
 Use a profiler
Using a profiler can help to identify
unexpected behavior degrading application
performance.
For example, a profiler can tell you in which
functions most of the time is spent.
Easy to do with Valgrind: http://valgrind.org/
 Linux-Tiny
Goal: reduce the disk footprint and RAM size
of the Linux kernel
Set of patches against the mainstream Linux
kernel.
Mergeability in mainstream is a priority.
Many changes have already been merged in
recent kernels.
All features can be selected in kernel
configuration (CONFIG_EMBEDDED).
 Linux-Tiny ideas (1)
Remove kernel messages (printk, BUG,
panic...)
Hunt excess inlining (speed vs. size tradeoff)
Hunt excess memory allocations
Memory (slob instead of slab) allocator more
space efficient for small systems.
Simpler alternative implementations of kernel
functionalities with less features, or not
supporting special cases.
 Linux-Tiny ideas (2)
Remove some features which may not be
needed
in some systems.
Compiling optimizations for size.
A smaller kernel executable also saves
RAM
(unless executed in place from storage).
 Reducing kernel RAM
Booting without sysfs support (if your
apps can do without it):

Kernel XIP
Kernel code is not copied into RAM, thus
saving the size of the uncompressed
kernel executable (at least 1 MB, often 2
or 3 MB).
 Static or dynamic linking? (1)
Static linking
All shared library code duplicated in the
executables
Allows not to copy the C library in the
filesystem.
Simpler and smaller when very few
executables (busybox)
Library code duplication: bad for systems
with more executables, best for small
systems with few executables!
 Static or dynamic linking? (2)

Dynamic linking
Shared library code not duplicated in the
executables
Makes much smaller executables
Saves space in RAM (bigger executables
take more RAM)
 How to use uClibc?
Need to compile all your executables with
a uClibc toolchain.
Ready-to-use toolchains
You also have to copy the uClibc files
from the toolchain to the /lib directory in
the target root filesystem.
 Need for stripping
To remove debugging information, use the
strip command.
This can save a very significant amount of
space!
gcc -o hello hello.c (output size:
4635 bytes)
strip hello (output
size: 2852 bytes, -38.5%)
Don't forget to strip libraries too!
 Are my executables stripped?
You can use the file command to get the answer
file hello
hello: ELF 32-bit LSB executable,
Intel 80386, version 1 (SYSV), for
GNU/Linux 2.2.5, dynamically linked
(uses shared libs), not stripped
strip hello
hello: ELF 32-bit LSB executable,
Intel 80386, version 1 (SYSV), for
GNU/Linux 2.2.5, dynamically linked
(uses shared libs), stripped
 How to strip
Some lightweight tools, like busybox, are
automatically stripped when you build them.
Makefiles for many standard tools offer a
special command:
make install-strip
Caution: stripping is architecture
dependent.
Use the strip command from your cross-
compiling toolchain:
arm-linux-strip potato
 sstrip: super strip
Goes beyond strip and can strip out a few
more bits that are not used by Linux to start
an executable.
 Library Optimizer

http://libraryopt.sourceforge.net/
Examines the complete target file system,
resolves all shared library symbol
references, and rebuilds the shared
libraries with only the object files required
to satisfy the symbol references.
Can also take care of stripping executables
and libraries.
 Compiler space optimizations
Regular compiler optimizations simplifying
code also reduce size
You can also reduce the size of
executables by asking gcc to optimize
generated code size:
gcc -Os -o husband husband.c
-Os corresponds to -O2 optimizations
except the ones increasing size, plus extra
size-specific ones.
 Restartable applications

When RAM is scarce, can be useful to

abort applications that are not in use.
Better to do it before the Linux Kernel
OOM (Out Of Memory) killer comes and
makes bad decisions.
To support this, design your programs with
the capability to save their state when they
are aborted and return to it when they are
restarted. [ HA ???]
 Compressing filesystems
Can significantly increase your storage
capacity
MTD (flash or rom) storage: use jffs2
Block storage: use SquashFS (
http://squashfs.sourceforge.net) instead
of CramFS for read-only partitions. It
compresses much better and is much
faster too.
 Tickless kernel
Kernel configuration: NO_HZ setting in Processor type
and features
To implement multitasking, the processor receives a
timer interrupt
at a given frequency (every 4 ms by default on Linux
2.6). On idle systems, this wakes up the processor all
the time, just to realize there is nothing to do!
Idea: when all processors are idle, disable the timer
interrupt, and re-enable it when something happens (a
real interrupt). This saves power in laptops, in
embedded systems and with virtual servers!
 Suspend hidden GUIs

Idea: suspend hidden user interfaces to save

CPU and power.
Send a suspend (stop) signal:
kill -SIGTSTP <pid>
Send a continue signal:
kill -SIGCONT <pid>
 Software suspend

http://www.suspend2.net/
Lots of great features for notebook users,
such as RAM suspend or hibernate to disk.
Unfortunately, restricted on some Intel
compatible processors and targeting only
machines with APM or ACPI
Not addressing the requirements of
embedded systems
Reducing power consumption -
Summary
Tickless operation
CPU frequency control
Suspend hidden GUIs
Software suspend (for PC embedded
systems)
Kernel and application XIP
Power management in device drivers
 Kernel Function Trace
Kernel Function Trace (KFT)
Kernel function tracing system, to answer
questions such as:
What functions are called?
How long to the functions take?
How many times are the functions called?
In which functions is most time spent?
Many details and a very good presentation:
http://elinux.org/Kernel_Function_Trace
 Use a swap partition
For embedded systems with a hard disk
Even if your system never gets out of RAM, and swapping
is not used much in embedded systems, it could make
sense!
With a swap partition, the system can swap out memory
pages which haven't been used recently, making more
RAM available for caching frequently accessed files.
This should improve overall throughput, but may also
reduce system responsiveness. Bad for real-time
systems!
Caution: do not use flash storage as a swap partition!
 Use mlockall()
Parts of the address space of a process may be
swapped out or not loaded yet (on-demand paging:
the kernel only loads process pages with are
actually accessed, and loads other ones when
actually accessed). This causes unpredictable
delays in process execution.
The mlockall() system call can be used to make
sure that the entire address space of a running
process is loaded in RAM.
Useful for real-time applications needing
predictable response time.
 Debugging
Gdb server on Target
Target remote over serial/network
Unstripped binaries at host
Mechanisms to export cores.
Accidental reboots.
 Remote
Debugging Remote PC
Running Linux

Ethernet
GDB
Cable

GBD Server
Debugging Using RS232 Port

Ethernet

RS232
 Unified tools

Android : software platform and operating

system written by Google and the Open Handset
Alliance
Openwrt
Scratchbox
Code-sourcery

Metroworks
 Further info
Home page:
http://handhelds.org/
Mailing lists:
http://www.handhelds.org/email_lists.html
 For Further Info
Web Sites

LinuxDevices.com
http://www.linuxdevices.com/
Embedded Linux Consortium
http://www.embedded-linux.org/index.php3
All Linux Devices
http://alllinuxdevices.com/
Embedded Linux StarGate
http://linux-embedded.com/
 For Further Info
Web Sites/Magazines
Dr. Dobbs Embedded Systems
http://www.ddjembedded.com/about/

Embedded Linux Journal

http://embedded.linuxjournal.com/

Embedded Systems Programming

http://www.embedded.com/mag.htm
54 48 41 4E 4B
59 4F 55