You are on page 1of 28

… in 60 Minutes

Click to Presentation
edit Master title
style
•Welcome, we will
Click to edit thebegin shortly!
outline text format
•Your phone will be muted throughout the
Second Outline Level
presentation, kindly keep it muted.
Third
•This session Outline
will be Level
recorded for future
reference. Fourth Outline Level
•Questions? Please use theOutline
Fifth Chat functionality
Level
on WebEx to submit questions. They will be
Sixth Outline
answered toward the end of the presentation.
Level
SeventhThankOutline
youLevelClick to edit
for attending!
Master subtitle
Click to edit Masterstyles
presenter style

1 Copyright © Ciena Corporation 2014. All rights reserved. Confidential & Proprietary.
The Gnu Debugger
Basic program building blocks
code
data
context
On-target usage
Embedded debugging with gdbserver
Core file handling
What Constitutes a Process?
A process contains:
Program code.
Executable and library ELF text segments.
Program data.
Executable and library ELF data segments:
global/static variables.
Specialised structures, e.g. the heap: dynamic data.
Execution context.
CPU registers.
Stack (a memory contraption to save/restore
registers).
GDB Building Blocks
GDB understands ELF (Executable and Linkable
Format).
Part of the SysV ABI since release 4 (1988).
Container for programs, shared libraries and core
files.
GDB uses ptrace() to control user processes.
Stop and resume execution.
Intercept signals.
Examine or modify memory.
Examine or modify registers.
Executable ELF Segments
> cat /proc/$(pgrep sleep)/maps
00100000-00103000 r-xp 00000000 00:00 0 [vdso]
0fe64000-0ffd9000 r-xp 00000000 b3:01 454447 /lib/libc-2.20.so
0ffd9000-0ffe8000 ---p 00175000 b3:01 454447 /lib/libc-2.20.so
0ffe8000-0ffea000 r--p 00174000 b3:01 454447 /lib/libc-2.20.so
0ffea000-0ffee000 rwxp 00176000 b3:01 454447 /lib/libc-2.20.so
0ffee000-0fff0000 rwxp 00000000 00:00 0
10000000-10008000 r-xp 00000000 b3:01 146126 /bin/sleep
10017000-10018000 rwxp 00007000 b3:01 146126 /bin/sleep
48000000-48021000 r-xp 00000000 b3:01 454284 /lib/ld-2.20.so
48027000-48028000 rw-p 00000000 00:00 0
48030000-48031000 r--p 00020000 b3:01 454284 /lib/ld-2.20.so
48031000-48033000 rwxp 00021000 b3:01 454284 /lib/ld-2.20.so
bfafa000-bfb1b000 rw-p 00000000 00:00 0 [stack]

> readelf -l $(which sleep)


Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
PHDR 0x000034 0x10000034 0x10000034 0x000e0 0x000e0 R E 0x4
INTERP 0x000114 0x10000114 0x10000114 0x0000d 0x0000d R 0x1
[Requesting program interpreter: /lib/ld.so.1]
LOAD 0x000000 0x10000000 0x10000000 0x07088 0x07088 R E 0x10000
LOAD 0x007088 0x10017088 0x10017088 0x00120 0x004a0 RWE 0x10000
DYNAMIC 0x0070ac 0x100170ac 0x100170ac 0x000c8 0x000c8 RW 0x4
NOTE 0x000124 0x10000124 0x10000124 0x00020 0x00020 R 0x4
GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x10
Library ELF Segments
> cat /proc/$(pgrep sleep)/maps
00100000-00103000 r-xp 00000000 00:00 0 [vdso]
0fe64000-0ffd9000 r-xp 00000000 b3:01 454447 /lib/libc-2.20.so
0ffd9000-0ffe8000 ---p 00175000 b3:01 454447 /lib/libc-2.20.so
0ffe8000-0ffea000 r--p 00174000 b3:01 454447 /lib/libc-2.20.so
0ffea000-0ffee000 rwxp 00176000 b3:01 454447 /lib/libc-2.20.so
0ffee000-0fff0000 rwxp 00000000 00:00 0
10000000-10008000 r-xp 00000000 b3:01 146126 /bin/sleep
10017000-10018000 rwxp 00007000 b3:01 146126 /bin/sleep
48000000-48021000 r-xp 00000000 b3:01 454284 /lib/ld-2.20.so
48027000-48028000 rw-p 00000000 00:00 0
48030000-48031000 r--p 00020000 b3:01 454284 /lib/ld-2.20.so
48031000-48033000 rwxp 00021000 b3:01 454284 /lib/ld-2.20.so
bfafa000-bfb1b000 rw-p 00000000 00:00 0 [stack]

> readelf -l /lib/libc-2.20.so


Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
PHDR 0x000034 0x00000034 0x00000034 0x00140 0x00140 R E 0x4
INTERP 0x162a34 0x00162a34 0x00162a34 0x0000d 0x0000d R 0x4
[Requesting program interpreter: /lib/ld.so.1]
LOAD 0x000000 0x00000000 0x00000000 0x1741d4 0x1741d4 R E 0x10000
LOAD 0x1745b0 0x001845b0 0x001845b0 0x04c90 0x07648 RWE 0x10000
DYNAMIC 0x175f10 0x00185f10 0x00185f10 0x000f0 0x000f0 RW 0x4
NOTE 0x000174 0x00000174 0x00000174 0x00020 0x00020 R 0x4
TLS 0x1745b0 0x001845b0 0x001845b0 0x0000c 0x0005c R 0x4
GNU_EH_FRAME 0x162a44 0x00162a44 0x00162a44 0x01d2c 0x01d2c R 0x4
GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x10
GNU_RELRO 0x1745b0 0x001845b0 0x001845b0 0x01a50 0x01a50 R 0x1
Dynamic Data Segments
> cat /proc/$(pgrep sleep)/maps
00100000-00103000 r-xp 00000000 00:00 0 [vdso]
0fe64000-0ffd9000 r-xp 00000000 b3:01 454447 /lib/libc-2.20.so
0ffd9000-0ffe8000 ---p 00175000 b3:01 454447 /lib/libc-2.20.so
0ffe8000-0ffea000 r--p 00174000 b3:01 454447 /lib/libc-2.20.so
0ffea000-0ffee000 rwxp 00176000 b3:01 454447 /lib/libc-2.20.so
0ffee000-0fff0000 rwxp 00000000 00:00 0
10000000-10008000 r-xp 00000000 b3:01 146126 /bin/sleep
10017000-10018000 rwxp 00007000 b3:01 146126 /bin/sleep
48000000-48021000 r-xp 00000000 b3:01 454284 /lib/ld-2.20.so
48027000-48028000 rw-p 00000000 00:00 0
48030000-48031000 r--p 00020000 b3:01 454284 /lib/ld-2.20.so
48031000-48033000 rwxp 00021000 b3:01 454284 /lib/ld-2.20.so
bfafa000-bfb1b000 rw-p 00000000 00:00 0 [stack]
Program Context
> cat /proc/$(pgrep sleep)/maps
00100000-00103000 r-xp 00000000 00:00 0 [vdso]
0fe64000-0ffd9000 r-xp 00000000 b3:01 454447 /lib/libc-2.20.so
0ffd9000-0ffe8000 ---p 00175000 b3:01 454447 /lib/libc-2.20.so
0ffe8000-0ffea000 r--p 00174000 b3:01 454447 /lib/libc-2.20.so
0ffea000-0ffee000 rwxp 00176000 b3:01 454447 /lib/libc-2.20.so
0ffee000-0fff0000 rwxp 00000000 00:00 0
10000000-10008000 r-xp 00000000 b3:01 146126 /bin/sleep
10017000-10018000 rwxp 00007000 b3:01 146126 /bin/sleep
48000000-48021000 r-xp 00000000 b3:01 454284 /lib/ld-2.20.so
48027000-48028000 rw-p 00000000 00:00 0
48030000-48031000 r--p 00020000 b3:01 454284 /lib/ld-2.20.so
48031000-48033000 rwxp 00021000 b3:01 454284 /lib/ld-2.20.so
bfafa000-bfb1b000 rw-p 00000000 00:00 0 [stack]
CPU Register Sets
powerpc arm

general-purpose registers user-mode registers


r0 r8: parm 5 r16 r24 < > = 0 CR0 r0: parm 0 r8
r1: stack ptr r9: parm 6 r17 r25 < > = 0 CR1 r1: parm 1 r9
r2 r10: parm 7 r18 r26 < > = 0 CR2 r2: parm 2 r10
r3: parm 0 r11 r19 r27 < > = 0 CR3 r3: parm 3 r11
r4: parm 1 r12 r20 r28 < > = 0 CR4 r4 r12
r5: parm 2 r13 r21 r29 < > = 0 CR5 r5 r13: stack ptr
r6: parm 3 r14 r22 r30 < > = 0 CR6 r6 r14: link reg
r7: parm 4 r15 r23 r31 < > = 0 CR7 r7 r15: program ctr

PC CPSR
LR special-purpose registers
N >
Z C V Q > I F T mode: usr
CTR
31 26 8 4 0

x86 mips

eax: return value esi zero t0 s0 t8


ebx: glob offset edi at t1 s1 t9
ecx esp: stack ptr v0: retval 0 t2 s2 k0
edx ebp: (frame ptr) v1: retval 1 t3 s3 k1
eflags eip: program ctr a0: arg 1 t4 s4 gp: global ptr
a1: arg 2 t5 s5 sp: stack ptr
a2: arg 3 t6 s6 fp: (frame ptr)
do not preserve a3: arg 4 t7 s7 ra: link reg
value across function call
preserve
reserved
The Stack
powerpc arm x86 mips
r1 saved r1 r13 esp sp
local variables function args space for a0
return address space for a1
local variables space for a2
saved registers
space for a4
saved registers local variables

local variables
saved registers

saved r1 return address saved registers return address


return address more function args a0
return address
more function args a1
local variables function args
a2
local variables
a4
saved registers more arguments
saved registers
local variables local variables
return address
saved r1 saved registers saved registers
local variables
return address
local variables return address return address
saved registers
function args a0
a1
saved registers
return address a2
... ... ...
...
GDB at Run-Time
GDB obtains its run-time library ELF

data from two sources: GDB


library ELF
program ELF

Symbol and debug


information from
program and library ELF
/proc/<pid>/mem
files.
Process memory and
execution context
directly from the running target
process
process using ptrace().
Shortcuts are available
through the /proc file
GDB Invocation
Launch a program through GDB
shell> gdb sleep
(gdb) set args 100
(gdb) run
Launch GDB, attach to a process
shell> gdb
(gdb) attach <pid>
(gdb) continue
Launch and attach GDB all at once
shell> gdb -p <pid>
(gdb) continue
Basic GDB Commands
Dump the stack Print local variables
(gdb) backtrace (gdb) info locals
Move up one stack Print global variables
frame (gdb) info
(gdb) up variables
Show process Examine stack memory
information (gdb) x/40xw $sp
(gdb) info proc Print one variable
all (gdb) print
Display registers main_arena
(gdb) info List the next
registers instructions
GDB-Process Interactions
GDB target
process
attach

process
stops
continue

process
resumes
ctrl-c

process
stops
info reg

x/40xw $sp

b <address>

continue

process
resumes

process
stops
Remote GDB
library ELF
library ELF
program ELF
GDB is too heavyweight GDB

to run on embedded
target systems.
ELF symbol and debug
workstation

information is stripped. target system

GDB itself is a resource gdbserver


/proc/<pid>/mem

hog.
A lightweight process
runs on target and
target
relays GDB commands. process

GDB runs on a
workstation with access
Cross-Compilation Considerations
The default workstation gdb can debug native
(x86_64) programs.
Must use cross-compiled gdb included in
architecture-specific toolchain.
By default, cross-compiled gdb loads the
workstation (x86_64) system libraries.
Must supply cross-compiled gdb with appropriate file
system paths.
dbg.sh
Helper script dbg.sh invokes toolchain gdb, cross-
compiled for the target, with appropriate shared
library paths.
saos-8.x CTX:
shell> make/tools/dbg.sh gdb saos-
thorin releasefs ctx 10.1.24.124:65000
bin/cesd-ctm
saos-6.x dernhelm:
shell> make/tools/dbg.sh gdb saos-sds
tarfs dernhelm 10.1.25.119:2159
ciena/bin/leos
on any target, use “ddd” instead of “gdb” to
Firewall Considerations
workstation target system

On saos-6.x, the
firewall must be opened GDB gdbserver

to let GDB connect to


the gdbserver.
On saos-8.x, the CTX target system
workstation
firewall must be opened
and the GDB control GDB NAT
packets must be
gdbserver
forwarded over the
backplane.
Firewall Helper Script (Saos-6.x)
Helper script “krn” opens up port 2159 and
launches gdbserver attached to leos.
bash# krn debug on
bash# krn debug off # when done
To debug another program, open the firewall
manually:
bash# iptables -A INPUT -p tcp –dport
2159 -j ACCEPT
bash# gdbserver :2159 –attach <pid>
bash# iptables -D INPUT -p tcp –dport
2159 -j ACCEPT # when done
Firewall Helper Script (Saos-8.x)
On CTX, re-run firewall.sh script with magic file,
to open up ports 65000-65010.
8700P> touch /rel/firewall_debug
8700P> /ciena/scripts/firewall.sh
CTX translates port to 65000 + LM number
1-A-2> gdbserver :65000 –attach $(pgrep
cesd-pslm)
shell> make/tools/dbg.sh gdb saos-
thorin releasefs pslm-200-20
10.1.24.124:65002 bin/cesd-pslm
Bonus: direct LM telnet via port 65100 + LM
number
shell> telnet 10.1.24.124 65102
Breakpoint
A breakpoint replaces an (gdb) break healthAgentUptimeGet

opcode with a trap


gdbserver
instruction.
When the CPU executes
the trap, the kernel
signals a SIGTRAP. PTRACE_POKETEXT

/proc/<pid>/mem
GDB intercepts the
SIGTRAP and halts the
process.
Breakpoint Commands
Insert a breakpoint
(gdb) break healthAgentUptimeGet
(gdb) break *0x109aa1f0
Remove a breakpoint, all breakpoints
(gdb) delete 1
(gdb) delete breakpoints
Define breakpoint commands
(gdb) commands
print healthAgentData.uptime
continue
end
Single-step: source, opcode
(gdb) s # awkward on optimised code
Death Signals
SIGABRT SIGABR
T

The program hit an program


error condition and self-
ended. SIGSEGV

SIGSEGV MMU
The program tried to program
access an invalid
memory location. SIGBU
S

SIGBUS MMU IO
The program failed an program
IO operation.
GDB with Core Files
When a process library ELF
library ELF

crashes, a core file is GDB


program ELF

produced. It uses ELF


and contains:
Dynamic data pages. core ELF

Execution context.
GDB obtains this data
by reading yet another
ELF file.
dbg.sh (again)
With a static core file, the toolchain GDB and
proper shared library paths must still be set. The
same helper script can be used.
saos-8.x CTX:
shell> make/tools/dbg.sh gdb saos-
thorin releasefs ctx /path/to/core
bin/cesd-ctm
saos-6.x dernhelm:
shell> make/tools/dbg.sh gdb saos-sds
tarfs dernhelm /path/to/core
ciena/bin/leos
GDB Helper Scripts
The tedious bits of core analysis are simplified by
useful helper scripts.
(gdb) cd /your/view/src/software/saos-
sds/tools/gdb
saos_helper.py
(gdb) source saos_helper.py
(gdb) help lsig
(gdb) help lps
glibc_heap.py
(gdb) source glibc_heap.py
(gdb) help heap_check
(gdb) help heap_dump
Core Examination Tip
Here is the most useful gdb command for
SIGSEGV and SIGBUS:
(gdb) x/i $pc
Shows the instruction that caused the error.
Was it a read or a write?
Where does the address come from?
Once the source of the address is figured out (i.e.
mapped to a variable or to a function parameter),
move up one stack frame and repeat.
Attitude Tip

You might also like