VARNISH-CACHE *

Lessons Learnt from Performance Testing

* https://www.varnish-cache.org
Tuesday, 25 October 11

WHY? FAST!
If

your throughput is currently measured in the hundreds per

second, think of varnish in the thousands per second.

Theoretical

limits in hundreds of thousands per second.*

* http://kristianlyng.wordpress.com/2010/10/23/275k-req/
Tuesday, 25 October 11

GET OUT OF JAIL. FREE.

Crappy

cache control?

Expensive
Not

CDNs? 20/GB?

enough hardware?

Backend

Tuesday, 25 October 11

struggling?

IT WONT
Do

HTTPS... terminate SSL with something else, hopefully

your load balancers.

Tuesday, 25 October 11

INSTALL THE LATEST

RHEL

/ CentOS

rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnishrelease-3.0-1.noarch.rpm

yum install -y varnish

Debian
apt-get install varnish

Tuesday, 25 October 11

USE 64BIT*
Its all about virtual memory: Things like
stack size suddenly matter on 32bit. If you
must use Varnish on 32-bit, youre somewhat
on your own. However, try to fit it within
2GB. I wouldnt recommend a cache larger
than 1GB, and no more than a few hundred
threads (Why are you on 32bit again?)

* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/
Tuesday, 25 October 11

SYSTEM CONFIG
The

minimum.

echo 'fs.file-max=1000000' >> /etc/sysctl.conf

echo 'net.ipv4.ip_local_port_range = 1024 65535' >> /etc/sysctl.conf
sysctl -p
echo '* soft nofile 1000000' >> /etc/security/limits.conf
echo '* hard nofile 1000000' >> /etc/security/limits.conf
echo 'session required pam_limits.so' >> /etc/pam.d/login
ulimit -n 1000000
ulimit -n -H

Do

not set tw_reuse to 1 (sysctl). It will cause problems for

NAT based clients.

Enable
Tuesday, 25 October 11

keepalives on load balancers.

THREADS
Adjust

default thread* settings in /etc/sysconfig/varnish

thread_pool_add_delay=2
thread_pools = <Number of cpu cores>
thread_pool_min = <800/number of cpu cores>
thread_pool_max = 4000
session_linger = 50
sess_workspace = <16k to 5m>

The only thing that made a real difference while tuning Varnish
was the number of threads. And making sure it actually caches.
Beyond that, it really doesnt matter much. Our defaults are
good.

* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/
Tuesday, 25 October 11

STORAGE
Memory
Got

is fast, duh.

enough RAM to store the cache?

Yes! then use -s malloc
Unsure? then use -s file

Tuesday, 25 October 11

VCL
Varnish

Configuration Language

Looks like C and compiled into C but

~ has regex tilda operator
Assignments use set keyword.
Only has conditional if tests.

...

Has a bunch of variables and some built in functions.

Defined

by backends, probes, ACLs and subroutines which

hook into the varnish workflow.

Tuesday, 25 October 11

VCL
Default VCL is defined in
/etc/varnish/default.vcl
Then create your custom VCL
-f /var/custom.vcl

in

so varnish can fall through to the default.vcl

Can also include "/var/other.vcl";
Keep

it simple within the VCL workflow*

Do most of the work in vcl_recv and vcl_fetch

* https://www.varnish-cache.org/trac/wiki/VCLExampleDefault
Tuesday, 25 October 11

LOOKUP AND MISS

request
hash

request

lookup
fetch

received from client

object and miss

response from backend

deliver

Tuesday, 25 October 11

to client

recv
hash
hit

pipe
miss

fetch
deliver

pass

LOOKUP AND HIT

request
hash

received from client

request

lookup

object and hit

recv
hash
hit

pipe
miss

fetch
deliver

Tuesday, 25 October 11

to client

deliver

pass

PASS AND DELIVER

request

received from client

recv
hash

pass
fetch

through to backend
response

deliver

Tuesday, 25 October 11

to client

hit

pipe
miss

fetch
deliver

pass

PIPE
request
pipe

received from client

to backend until client or

backend closes

recv
hash
hit

pipe
miss

fetch
deliver

Tuesday, 25 October 11

pass

PROBES
Define

common health
checks for multiple
backends.

window is how many of the

latest polls are probed for

health.

threshold is how many

polls must succeed to be

healthy.
Tuesday, 25 October 11

probe health {
.url = "/healthCheck?onFailure=true";
.expected_response = 200;
.interval= 15s;
.timeout = 5s;
.window= 4;
.threshold = 4;
}

BACKENDS
Define

backend host, port

and health checks inline.
{ .backend = { .host = "10.10.172.1"; .port = "8001"; .probe = health; } }

Or

named.

backend mysite {
.host = "www.example.com";
.port = "http";
.probe = "health";
}

Tuesday, 25 October 11

DIRECTORS
Group

backends together using different algorithms.

Random
Client (based on identity)
Hash (based on url)
Round robin
Fallback

Tuesday, 25 October 11

YMMV
I

found it better to balance

in front of Varnish using a
hardware load balancer ...

Random

seemed a little too

random using Varnish
directors however was
probably an artefact of the
load generation.

Tuesday, 25 October 11

DESIGN DECISIONS
Share

nothing.
Easy to scale horizontally.
Great if your site is not transactional.
Backend session affinity via LB set cookie.
LB

Tuesday, 25 October 11

Set-Cookie=BackendID

varnish

varnish

varnish

app

app

app

DESIGN DECISIONS
Share

everything.
Varnish loves RAM, scale vertically.
Varnish cache becomes the shared source of truth.

LB
varnish
app
Tuesday, 25 October 11

app

app

SHARE NOTHING. FALLBACK.

LB

director varnishA fallback {

{ .backend = { .host = "a";} }
{ .backend = { .host = "b";} }
{ .backend = { .host = "c";} }
}

director varnishB fallback {

{ .backend = { .host = "b";} }
{ .backend = { .host = "a";} }
{ .backend = { .host = "c";} }
}

director varnishC fallback {

{ .backend = { .host = "c";} }
{ .backend = { .host = "b";} }
{ .backend = { .host = "a";} }
}

Tuesday, 25 October 11

Set-Cookie=VarnishID

varnish

varnish

varnish

app

app

app

TERMINATE SSL.
Get

the LB to set a Port header.

Useful when user switches from HTTP to HTTPS so varnish
knows which backend port to route to.
LB

Set-Cookie=VarnishID
Port=https

varnish

app
:80
:443

if (req.http.Port ~ "https") {
set req.backend = appa_80;
} else {
set req.backend = appa_443;
}

Tuesday, 25 October 11

VCL TRICKS
Rewrite headers to/from production.com.au
testdomain.com

to

vcl_recv
set req.http.Host = regsuball(req.http.Host, "testdomain\.com", "com\.au");
set req.http.Referer = regsuball(req.http.Host, "testdomain\.com", "com\.au");

vcl_deliver
set
set
set
set

resp.http.Host = regsuball(resp.http.Host, "\.com\.au", "\.testdomain\.com");

resp.http.Referer = regsuball(resp.http.Host, "\.com\.au", "\.testdomain\.com");
resp.http.Location = regsuball(resp.http.Location, "\.com\.au", "\.testdomain\.com");
resp.http.Set-Cookie = regsuball(resp.http.Set-Cookie, "Domain=.*;", "");

Tuesday, 25 October 11

VCL TRICKS
Add some debug
vcl_deliver

headers for hit/miss in firebug in

if (obj.hits > 0) {
set resp.http.X-Varnish-Hit = "++";
set resp.http.X-Varnish-Hits = obj.hits;
set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(\d+).*", "\1");
} else {
set resp.http.X-Varnish-Hit = "--";
set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(\d+).*", "\1");
}

Tuesday, 25 October 11

VCL TRICKS
Normalize the Accept-Encoding
usage in vcl_recv

headers to reduce cache

if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
remove req.http.Accept-Encoding;
}
}

Tuesday, 25 October 11

VCL TRICKS
Remove

Google Analytics cookies in vcl_recv

set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");

set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");

Remember, varnish

will pass any request with cookies by

default. You will need to over-ride this behaviour if your web
app uses cookies.

Allow lookup of requests with basic auth (in test) or cookies

in vcl_recv but beware of creating a per-user-session store.
if (req.http.Authorization || req.http.Cookie) { return (lookup); }

Tuesday, 25 October 11

VCL TRICKS
Pass

on the client IP to the backend servers using the XForwarded-For header in vcl_recv
remove req.http.X-Forwarded-For;
setreq.http.X-Forwarded-For = client.ip;

Tuesday, 25 October 11

VCL TRICKS
Let varnish keep
vcl_fetch

objects for 2 hours beyond their TTL in

set beresp.grace = 2h;

Let

varnish serve stale objects up to 30 secs old for healthy

servers, otherwise 2 hours in vcl_recv

if (! req.backend.healthy) {
set req.grace = 2h;
} else {
set req.grace = 30s;
}

Can

also use saint mode to disable backends for a period of

time, otherwise if all backends unavailable, use grace.

Tuesday, 25 October 11

VCL TRICKS
Force

a grace on error. Restart keyword can be called any

where and will return request all the way back to vcl_recv
sub vcl_error {
if (req.restarts == 0) {
set req.http.X-Serve-Graced = "1";
return(restart);
}
}

Tuesday, 25 October 11

TTL
Varnish

will obey cache control headers from backend

Cache-Control: s-maxage
Cache-Control: max-age

Can

also override TTL control in VCL, but best do it from the

backend
set obj.ttl = x

Tuesday, 25 October 11

REDUCING CACHE SIZE

Normalize
Set

/ sanitize request headers

Content-Length on backends

Serve
Pipe

(static) digital assets from VFS cache

large objects like videos

Tuesday, 25 October 11

MONITORING VARNISH
Handy command
varnishstat
varnishlog
varnishtop
varnishadm

Tuesday, 25 October 11

line tools

VARNISHSTAT

varnishstat

Hit rate % (x100)

Average per second
Average/sec since boot
Sum total

Filtered

varnishstat -f \
client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_fail

Tuesday, 25 October 11

VARNISHSTAT
Log

all statistics to disk ...

#!/bin/bash
logdir=/var/log/perf/`hostname`/latest
mkdir -p $logdir

while true
do
varnishstat -x >> "$logdir/varnishstat.xml"
sleep 60
done

Then

Tuesday, 25 October 11

graph as you see fit ...

VARNISHLOG

varnishlog debugging (-d to dump shared memory)

12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12

BackendOpen
TxRequest
TxURL
TxProtocol
TxHeader
TxHeader
TxHeader
TxHeader
TxHeader
TxHeader
RxProtocol
RxStatus
RxResponse
RxHeader
RxHeader
RxHeader
RxHeader
RxHeader
RxHeader
RxHeader
Fetch_Body
Length
BackendClose

Tuesday, 25 October 11

b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b

xx_80[0] x.x.x.x 45546 x.x.x.x 80

GET
/
HTTP/1.1
Host:
Referer:
Cookie:
X-Forwarded-For: 1.2.3.4
X-Varnish: 2092287889
Accept-Encoding: gzip
HTTP/1.1
401
Authorization Required
Date: Mon, 24 Oct 2011 05:46:18 GMT
WWW-Authenticate: Basic realm="XX Authorised Personel Only"
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 336
Connection: close
Content-Type: text/html; charset=iso-8859-1
4 0 1
336
xx_80[0]

VARNISHTOP
Monitor top URLs hitting
varnishtop -i txurl

backend

Monitor top received URLs

varnishtop -i rxurl

Tuesday, 25 October 11

VARNISHADM

varnishadm for administration

# varnishadm
CLI connected to 127.0.0.1 6082
200
----------------------------Varnish Cache CLI 1.0
----------------------------Linux,2.6.32-131.12.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit
Type 'help' for command list.
Type 'quit' to close CLI session.
varnish> help
200
help [command]
ping [timestamp]
auth response
quit
banner
status
start
stop
vcl.load <configname> <filename>
vcl.inline <configname> <quoted_VCLstring>
vcl.use <configname>
vcl.discard <configname>
vcl.list
Tuesday, 25 October 11

MORE HELP?
Talk

to the devs, read their blogs.

http://kristianlyng.wordpress.com
http://ingvar.blog.redpill-linpro.com/

Get

some online training.

https://www.varnish-software.com/products-services/training/
varnish-administration-course

Consider

the Varnish Moral License.

http://phk.freebsd.dk/VML/

Tuesday, 25 October 11