Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Network packet capture in Linux kernelspace

An overview of the network stack in the Linux kernel Beraldo Leal
beraldo@ime.usp.br http://www.ime.usp.br/~beraldo/
Institute of Mathematics and Statistics - IME University of Sao Paulo - USP

25th October 2011

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

1 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Outline

Introduction Network stack Packet ingress ow Methods to capture packets

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

2 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Introduction

Sniers; Improvements in packet reception; Linux kernel network subsystem;

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

3 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Sniers

tcpdump, wireshark, snort, etc; Using the well-known library libpcap; Not suitable for > 10 Gbps; Packet loss;

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

4 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Improvements in packet reception

Commodity hardware for packet capture; 3COM Intel endace, ... Many Interruptions NEW API or NAPI (interruption coalescence) zero-copy Direct Memory Access - DMA mmap()

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

5 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Linux kernel network subsystem

Kernel number of les: 36.680

1 2

net/ number of les: 1.293 ( 3.5% ) drivers/net/ number of les: 1.935 ( 5.27% ) Kernel SLOC: 9.723.525 net/ SLOC: 480.928 ( 5% ) drivers/net/ SLOC: 1.155.317 ( 12% )

1 2

kernel 3.0.0 source: wc, nd, cat, etc..

25th October 2011 Network packet capture in Linux kernelspace 6 / 25

Beraldo Leal

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Network stack
L5: Application
http, ftp, ssh, telnet, ... (message)

L4: Transport
tcp, udp, ... (segment)

L3: Network
ipv4, ipv6, ... (datagram/packet)

L1/2: Link / host-to-network

ethernet, token ring, ... (frame)

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

8 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Important data structs:

net device include/linux/netdevice.h sk buff include/linux/skbuff.h

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

9 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Important data structs:

net device (include/linux/netdevice.h) unsigned int mtu unsigned int flags unsigned char dev addr[MAX ADDR LEN] int promiscuity

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

10 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Important data structs:

sk buff (include/linux/skbuff.h) struct sk buff *next; struct sk buff *prev; ktime t tstamp; struct net device *dev; unsigned int len; unsigned int data len; u16 mac len; u8 pkt type; be16 protocol; sk buff data t transport header; (old h) sk buff data t network header; (old nh) sk buff data t mac header; (old mac)

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

11 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Important sk bu routines

alloc skb(); dev alloc skb(); kfree skb(); dev kfree skb(); skb clone(); skb network header(skb); skb transport header(skb); skb mac header(skb);

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

12 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Packet ingress ow
When working in interrupt driven model, the nic registers an

interrupt handler; This interrupt handler will be called when a frame is received; Typically in the handler, we allocate sk buff by calling dev alloc skb(); Copies data from nics buer to this struct just created; nic call generic reception routine netif rx(); netif rx() put frame in per cpu queue; if queue is full, drop! net rx action() decision based on skb->protocol; This function basically dequeues the frame and delivery a copy for every protocol handler;
ptype all and ptype base queues
25th October 2011 Network packet capture in Linux kernelspace 13 / 25

Beraldo Leal

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Packet ingress ow

ip v4 rcv() will receive the ip datagram (if is a ipv4 packet); ip checksum, check ip headers, .... ip rcv finish() makes route decision (ip forward() or

ip local delivery())
ip local delivery() defrag fragmented packets, and call

ip local deliver finish()

ip local deliver finish() nd protocol handler again; tcp v4 rcv(), udp rcv(), or other L4 protocol handler ...

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

14 / 25

ip_local_deliver_nish() (net/ipv4/ip_input.c) "nd protoco! hand!er or send icmp_dst_unreach

<continue>
NF_IP_FORWARD

"a)er&*& Net$or% ip_local_deliver() (net/ipv4/ip_input.c) defra& fra&mented packets ip_rcv_nish() (net/ipv4/ip_input.c) "nd route and hand!e IP options

NF_IP_"O#A"_IN

ip_for$ard() (net/ipv4/ip_for'ard.c) hand!e route a!ert( send redirect if necessary( decrease ))*( verify if fra& is possib!e (mtu) ip_error() (net/ipv4/route.c) routin& error, send icmp pkt <...>

NF_IP_PRE_ROU IN!

ip_rcv() pac%et_rcv() arp_rcv() (net/ipv4/ip_input.c) <tcpdump_process> (hand!e arp re#uests verify skb, IP headers <dhcpd process> and rep!ies) and IP checksum <...>

netif_rx() (net/core/dev.c)

input_#ueue $cpu%

net_rx_action() (net/core/dev.c) decision based on skb >protoco! "e!d "a)er&+',& Ph)sical'"in%

Net$or%&Drivers (drivers'net'()

Applica&ion
,serspace 'ernelspace

)oc'e&$La#er (ne&*core*soc'+c) &cp_v(_do_rcv() (net/ipv4/tcp_ipv4.c) check for socket state __&cp_v(_loo',p() (net/ipv4/tcp_ipv4.c) check for socket in %)T*+" with dst_port

generate %&'( error

&cp_v(_rcv() (net/ipv4/tcp_ipv4.c) check for tcp headers

,dp_rcv() (net/ipv4/udp.c) check for udp headers

#...$

La#er$($ Transpor&

ip_local_deliver_nish() (net/ipv4/ip_input.c) nd protocol handler or send icmp_dst_unreach

#continue$
NF_IP_FORWARD

La#er$%$ Ne&"or' ip_local_deliver() (net/ipv4/ip_input.c) defrag fragmented packets ip_rcv_nish() (net/ipv4/ip_input.c) nd route and handle %( options

NF_IP_LO AL_IN

ip_!or"ard() (net/ipv4/ip_forward.c) handle route alert; send redirect if necessary; decrease TT ; verify if frag is possi!le (mtu) ip_error() (net/ipv4/route.c) routing error" send icmp pkt

NF_IP_PRE_ROUTING

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Methods to capture packets

protocol handler register a function to handler packets with dev add pack() netlter hooks userspace tools; socket AF PACKET, libpcap, ...

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

17 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

struct packet type my proto; int my packet rcv(struct sk bu skb, struct net device dev, struct packet type pt, struct net device orig dev) { printk(KERN ERR + 1!\n); kfree skb(skb); return 0; } static int hello init(void) { printk(<1> Hello world!\n); my proto.type = htons(ETH P ALL); my proto.dev = NULL; my proto.func = my packet rcv; dev add pack(&my proto); return 0; } static void hello exit(void) { dev remove pack(&my proto); printk(<1> Bye, cruel world\n); } module init(hello init); module exit(hello exit);

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

18 / 25

Introduction Network stack Packet ingress ow Methods to capture packets 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

University of Sao Paulo - USP

int my packet rcv(struct sk bu skb, struct net device dev, struct packet type pt, struct net device orig dev) { switch (skb>pkt type) { case PACKET HOST: printk(PACKET HOST ); break; case PACKET BROADCAST: printk(PACKET BROADCAST ); break; case PACKET MULTICAST: printk(PACKET MULTICAST ); break; case PACKET OTHERHOST: printk(PACKET OTHERHOST ); break; case PACKET OUTGOING: printk(PACKET OUTGOING ); break; case PACKET LOOPBACK: printk(PACKET LOOPBACK ); break; case PACKET FASTROUTE: printk(PACKET FASTROUTE ); break; } printk(%s 0x%.4X 0x%.4X \n, skb>dev>name, ntohs(skb>protocol), ip hdr(skb)>protocol) kfree skb(skb); return 0; } Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 19 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Netlter hooks

iptables = userspace; netlter = kernelspace; Netlter is merely a series of hooks in various points in a

protocol stack;
packet ltering, network address [and port] translation

(NA[P]T) and other packet mangling;

www.netfilter.org

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

20 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

References

br.kernelnewbies.org/node/150 has many links

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

23 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Thankyou! Question?

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

24 / 25

Introduction Network stack Packet ingress ow Methods to capture packets

University of Sao Paulo - USP

Network packet capture in Linux kernelspace

An overview of the network stack in the Linux kernel Beraldo Leal
beraldo@ime.usp.br http://www.ime.usp.br/~beraldo/
Institute of Mathematics and Statistics - IME University of Sao Paulo - USP

25th October 2011

Beraldo Leal

25th October 2011

Network packet capture in Linux kernelspace

25 / 25