You are on page 1of 56

Network Design Patterns:

N-Tier Data Centers


Deepak Kakadia, Staff Engineer, Network Architect,
Sun Microsystems
Richard Croucher, Chief Architect, Professional
Services, Sun Microsystems
Sun BluePrints OnLineOctober 2003

http://www.sun.com/blueprints
Sun Microsystems, Inc.
4150 Network Circle
Santa Clara, CA 95045 U.S.A.
650 960-1300
Part No. 817-3997-10
Revision 1.0, 10/9/03
Edition: October 2003

Copyright 2003 Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, California 95045 U.S.A. All rights reserved.
Sun Microsystems, Inc. has intellectual property rights relating to technology embodied in the product that is described in this document. In
particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed at http://
www.sun.com/patents and one or more additional patents or pending patent applications in the U.S. and in other countries.
This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation.
No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors,
if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers.
Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in
the United States and other countries, exclusively licensed through X/Open Company, Ltd.
Sun, Sun Microsystems, the Sun logo, Sun BluePrints, Sun ONE, Sun StorEdge, Sun Fire, SunDocs, Jump Start, iPlanet, Enterprise Java Beans,
Java Database Connectivity, and Solaris are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other
countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the US
and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.
The OPEN LOOK and Sun Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges
the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun
holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Suns licensees who implement OPEN
LOOK GUIs and otherwise comply with Suns written license agreements.
U.S. Government RightsCommercial use. Government users are subject to the Sun Microsystems, Inc. standard license agreement and
applicable provisions of the Far and its supplements.
DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES,
INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT,
ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.
Copyright 2003 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, Californie 95045 Etats-Unis. Tous droits rservs.
Sun Microsystems, Inc. a les droits de proprit intellectuels relatants la technologie incorpore dans le produit qui est dcrit dans ce
document. En particulier, et sans la limitation, ces droits de proprit intellectuels peuvent inclure un ou plus des brevets amricains numrs
http://www.sun.com/patents et un ou les brevets plus supplmentaires ou les applications de brevet en attente dans les Etats-Unis et dans
les autres pays.
Ce produit ou document est protg par un copyright et distribu avec des licences qui en restreignent lutilisation, la copie, la distribution, et la
dcompilation. Aucune partie de ce produit ou document ne peut tre reproduite sous aucune forme, par quelque moyen que ce soit, sans
lautorisation pralable et crite de Sun et de ses bailleurs de licence, sil y en a. Le logiciel dtenu par des tiers, et qui comprend la technologie
relative aux polices de caractres, est protg par un copyright et licenci par des fournisseurs de Sun.
Des parties de ce produit pourront tre drives des systmes Berkeley BSD licencis par lUniversit de Californie. UNIX est une marque
enregistree aux Etats-Unis et dans dautres pays et licencie exclusivement par X/Open Company Ltd.
Sun, Sun Microsystems, le logo Sun, Sun BluePrints, Sun ONE, Sun StorEdge, Sun Fire, SunDocs, Jump Start, iPlanet, Enterprise Java Beans,
Java Database Connectivity, et Solaris sont des marques de fabrique ou des marques dposes, ou marques de service, de Sun Microsystems,
Inc. aux Etats-Unis et dans dautres pays. Toutes les marques SPARC sont utilises sous licence et sont des marques de fabrique ou des marques
dposes de SPARC International, Inc. aux Etats-Unis et dans dautres pays. Les produits portant les marques SPARC sont bass sur une
architecture dveloppe par Sun Microsystems, Inc.
Linterface dutilisation graphique OPEN LOOK et Sun a t dveloppe par Sun Microsystems, Inc. pour ses utilisateurs et licencis. Sun
reconnat les efforts de pionniers de Xerox pour la recherche et le dveloppement du concept des interfaces dutilisation visuelle ou graphique
pour lindustrie de linformatique. Sun dtient une licence non exclusive de Xerox sur linterface dutilisation graphique Xerox, cette licence
couvrant galement les licencis de Sun qui mettent en place linterface dutilisation graphique OPEN LOOK et qui en outre se conforment aux
licences crites de Sun.
LA DOCUMENTATION EST FOURNIE "EN LTAT" ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES
OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT
TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A LAPTITUDE A UNE UTILISATION PARTICULIERE OU A
LABSENCE DE CONTREFAON.

Please
Recycle

Network Design Patterns: N-Tier


Data Centers
This article describes recommendations and best practices for designing and
implementing N-Tier architectures for web-based service delivery infrastructures. It
documents a set of fully tested configurations, comparing solutions using two
different network equipment providers.
The design and implementation of optimal N-Tier architectures requires a thorough
analysis of functional and non-functional requirements, including performance,
availability, security, scalability and flexibility. The degree to which non-functional
requirements are met and exceeded often contributes to the quality factor of an
architecture. In this paper, we present some proven design principles and concepts,
which we hope are of value towards the successful delivery of N-Tier architecture
solutions.
For the purposes of this paper, we distinguish network architecture from design
as follows:
Architecture is a high-level description of how the major components of the system
interconnect with each other from a logical and physical perspective.
Design is more of a process that specifies, in sufficient detail for implementation,
how to construct a network of interconnected nodes that meets or exceeds functional
and non-functional requirements (performance, availability, scalability, and so on).
There currently exists a sufficient level of available public information describing the
construction of N-Tier data center architectures from a software standpoint.
However, from a systems and network architecture perspective, the best way to go
about constructing these architectures might not be as clear. This article attempts to
fill that void, by describing:

The recommended network design patterns and reasons behind these designs

Actual implementations that have been tested, tuned and proven in actual
deployments

This article also describes two popular approaches to network architectures, each
with its own merits and limitations.
This article addresses the following topics:

Defining Architectures on page 2


Implementation Overview on page 10
Designing the Network Configuration on page 11
Configuring the Physical Network on page 35
Network Security on page 48

The focus of this article is limited to data flow. Systems and network management is
a topic in itself and is not discussed in this paper. Readers should have sufficiently
detailed information to extend and modify this discussion to their particular needs.

Defining Architectures
This section breaks architecture considerations into two sections:

Logical Architectures on page 2


Physical Architectures on page 6

Logical Architectures
A typical high-level functional overview of an N-Tier data center architecture is
shown in FIGURE 1. The tiers are logically segregated by functionality as follows:
1. Web service tierThe set of load-balanced, front-end web servers that handles the
initial ingress client HTTP web requests. Each web server typically includes a
servelet engine or JSP engine, and the capability to execute locally hosted cgi/bin
scripts which execute native code.
2. Application service tierThe set of application server instances that might span
one large multi-CPU server or several smaller multi-CPU servers. This tier provides
greater functionality and sophicated processing capabilities and services, such as EJB
containers, an ORB, or connector modules that integrate with legacy servers. The
application server usually interfaces with the web server. In recent years, the web
server has been included in the application server text image for increased
performance.
3. Naming service tierThe set of servers, possibly load balanced, that provides
secondary infrastructure support services for managing and maintaining naming
information for hosts addressing data, configuration data, and so on. This can
include services such as DNS and LDAP.

Network Design Patterns: N-Tier Data Centers October 2003

4. Data service tierThe set of highly available servers that provides persistent data
storage services for critical corporate data. Due to the the sensitive nature of the
data, this tier is usually highly protected in terms of security and availability. We
briefly describe new virtual tiered firewalling techniques that provide increased
security without sacrificing performance, using hardware-based firewalls.

t
ien

ss

ce

ac

Cl
ce

Re

arc

ier

et

vic
ser

We

rs

tie

es

tur

ec
hit

en
fer

ce
rvi

ier

et

tion

vic
ser

lica

p
Ap

ice

v
ser

min

tier
S
DN

Na

AP

ier
cle
Ora

et

LD

ta

Da

ic
erv

SA

FIGURE 1

Logical High-Level Overview of N-Tier Architecture

Note Client access represents all the client systems that can access the reference
architecture (client systems, web browsers, and so forth). This access is not
considered part of the N-Tier architecture
The actual reference implementation described leveraged Sun ONE technology.
The Sun ONE software stack provides the software needed to deploy applications
using this reference implementation. The tiers of the N-Tier data center architecture,
shown in FIGURE 1, are mapped to the Sun ONE software as follows:

Web service tierSupported by Sun ONE Web Server, Enterprise Edition software

Naming service tierSupported by Sun ONE Directory Server software

Application service tierSupported by Sun ONE Application Server software

Defining Architectures

Data service tierProvided by Oracle 9i database software, which is not part of


the Sun ONE software stack

FIGURE 2 illustrates in greater detail how the logical N-Tier architecture is mapped to

the software systems architecture, and how Sun ONE and other software
components fit to create a complete system.

Network Design Patterns: N-Tier Data Centers October 2003

ta
Da ce
i
v
r
se r
tie

cle

Ora

n
atio
plicice
p
A rv
se r
tie

AP

LD

AP

LD

ute

Ro

Firewall

et
ern
Int r
o t
e
ran
Int

DN
b
Weer
v
r
se

ute

Ro

DN

FIGURE 2

Firewall

b
Weer
v
r
se

Firewall

b
We ce
vi
r
e
s r
tie

g
min
Na vice
ser r
tie

BC
JD
ns
ctio
ne
n
BC
Co ing
JD
s
m
tion
Na
g nnec
n
i
g Co
g
ssa
B
min
Me l
EJ ner
Na
i
i
a
a
ng
i
t
M
ag
t
con
B rity Mess
rvle r
J
e
E necru
S ine
il
ta
taSi e
Ma
on
t
A
con
c
e
l
D
rv
rity
JP
Se iner
tion Secu
a
a
r
t
t
is ing
con
DA
min itor
JP
Ad mon
B
tion
d
R
tra ing
n
O
s
a
i
n
mi itor
TP
Ad mon
HT er
B
d
R
v
n
O
a
ser
TP
s
HT er
nce
v
sta
n
i
ser
ver
ser
n
o
i
at
plic
t
ap
en
E
ON
oym
l
n
p
de
Su
nd
ly a
b
em
e
er
ass
rvic
n,
Se ation
tain
o
n
i
t
r
a
co
e integ
cre
ice
cor
ice
erv
v
s
S
r
n
Se
atio ces
plic rvi
Ap eb se
w
es
e
vic
rvic
ser
Se ery
b
licy
We
liv
po
de
nd
a
y
ntit
Ide atform
Pl

cy

ga

Le

Logical N-Tier Software Systems Architecture

Defining Architectures

Physical Architectures
This logical architecture is then mapped to a physical realization as shown in
FIGURE 3, which describes the physical architecture. This mapping is influenced
heavily at this point by non-functional requirements, such as performance and
availability. For example, redundant servers and redundant network connectivity are
a result of availability requirements and possibly performance requirements.
Wirespeed multilayer switches might satisfy performance requirements, provided
that the switch performs packet processing within timing constraints. Most Layer 2
and Layer 3 switches are implemented using ASICs, which are commonly available.
Higher layer packet service implementations differentiate vendors from each other.
Some vendors, such as Foundry in the ServerIron product line, implement Layer 7
using general purpose CPUs. This is much slower that an ASIC or an FPGA
implementation.

Network Design Patterns: N-Tier Data Centers October 2003

Client 1

Client 2

Client
access

L2-L3
Edge switch

Edge
network
Core switches

Web
service
tier
Sun Fire 280R Sun Fire 280R Sun Fire 280R Sun Fire 280R

Naming
service
tier
Sun Fire 280R Sun Fire 280R Sun Fire 280R Sun Fire 280R

Application
service
tier

Data
service
tier

FIGURE 3

Sun Fire
6800

Sun Fire
6800

T3

Sun Fire
6800

T3

Sun Fire
6800

Physical N-Tier Architecture Showing Hardware Components

Defining Architectures

TABLE 1 through TABLE 3 describe the actual tested hardware components and

configurations. This architecture is tested and proven. However, it is easily


extensible for adapting to specific customer requirements while keeping the
fundamental pattern intact. For example, increasing or decreasing the number of
redundant servers of a particular tier directly impacts the degree of availability and
performance (if configured as Active Active), but does not violate the fundamental
architecture.

TABLE 1

Memory
(RAM)

Storage
Capacity

Operating
Environment

Web servers (4) Sun Fire 280R Two 900servers


MHz CPUs

2 GB

36 GB

Solaris 9 10/02
with patches

Sun ONE
Web Server
6.0, Service
Pack 5,
Enterprise
Edition

DNS servers (2) Sun Fire 280R


servers

Two 900MHz CPUs

2 GB

36 GB

Solaris 9 10/02
with patches

n/a

Directory
server

(2) Sun Fire 280R


servers

Two 900MHz CPUs

4 GB

36 GB

Solaris 9 10/02
with patches

Sun ONE
Directory
Server 5.1

(2) Sun Fire 6800


servers

24 1-GHz
CPUs

64 GB

200 GB

Solaris 9 10/02
with patches

Sun ONE
Application
Server 7.0

Data servers (2) Sun Fire 6800


servers

24 1-GHz
CPUs

32 GB

36 GB

Solaris 9 10/02
with patches

VERITAS
Volume
Manager 3.5
software,
Oracle 9i
Release 2
software

n/a

200 GB

n/a

n/a

Service Tier

Function

Web

Naming

Application Application
servers
Data

Database
storage

Tested Systems Configuration: Sun Servers and Storage


Equipment

Features

(2) Sun StorEdge n/a


T3 Arrays

Network Design Patterns: N-Tier Data Centers October 2003

Software

TABLE 2

Tested Network Configuration 1 - Extreme Networks Equipment

Function

Equipment

Features

Operating Environment

Edge switch

(1) Extreme Networks


Summit7i switch

24 1-Gb ports

Extreme OS 6.2.2 build 18

Core switch

(2) Extreme Networks


BlackDiamond 6800
switches

Chassis switch with 7


blades. Each blade has 8
1-Gb ports (56 ports
total)

Extreme OS 6.2.2 build 18

TABLE 3

Tested Network Configuration 2 - Foundry Networks Equipment

Function

Equipment

Features

Operating Environment

Edge switch

(1) Extreme Networks


Summit7i switch

24 1-Gb ports

Extreme OS 6.2.2 build 18

Core switch

(2) Foundry BigIron


4000 Layer 2/3

Server loadbalancing
switch

(2) Foundry
ServerIron XL loadbalancing switches

Defining Architectures

Implementation Overview
Once the design and component selection is complete, follow these implementation
steps to set up your service tiers and their software:

To Implement Systems as Service Tiers

1. Install computer systems and storage hardware.


2. Establish network connectivity.
3. Configure partitions and domains on the Sun Fire 6800 servers in the application
and data service tiers.
4. Install and configure the Solaris Operating System on all systems.
5. Configure the Sun StorEdge T3 arrays on the Sun Fire 6800 servers in the data
service tier.
6. Install Oracle 9i Release 2 database software on the Sun Fire 6800 servers in the
data service tier.
7. Install the Sun ONE software on the following systems:
a. Web service tierInstall the Sun ONE Web Server software on the four Sun
Fire 280R servers in this service tier.
b. Naming service tierConfigure DNS on two of the Sun Fire 280R servers.
c. Naming service tierInstall and configure the Sun ONE Directory Server 5.1
software on the other two Sun Fire 280R servers.
Refer to the Sun ONE Directory Server 5.1 documentation collection (see Related
Resources on page 54 for more information).
d. Application service tierInstall and configure the Sun ONE Application Server
software on two Sun Fire 6800 servers.

Note See http://docs.sun.com for installation and hardware guides.

10

Network Design Patterns: N-Tier Data Centers October 2003

Designing the Network Configuration


One can argue that the heart of any architecture composed of multiple nodes is the
network architecture. The network architecture is vital to assembling components or
building blocks in such a way that the system delivery is high performance, flexible,
highly available, scalable, and secure. We describe a recommended methodology for
creating these architectures. We also describe actual implementations which show
the contrasts between two different network vendors, and how to optimally leverage
each of them. This section discusses the following topics:

Logical Network Architecture on page 11


IP Services on page 11
Designing Networks for Availability on page 19
Networks and VLANs on page 23
Inter-Tier Traffic Patterns on page 32

Logical Network Architecture


The logical network design is composed of segregated networks, implemented
physically using virtual local area networks (VLANs) defined by the network
switches. The internal network uses private IP address spaces (for example,
10.0.0.0) as specified in RFC 1918 for security and portability advantages. Look
ahead to FIGURE 10 for a high-level overview of the logical network architecture.
The management network provides centralized data collection and management of
all devices. Each device has a separate interface to the management network to
avoid contaminating the production network in terms of security and performance.
The management network is also used for automating the installation of the
software using Solaris JumpStart technology.
Although several networks physically reside on a single active core switch, network
traffic is segregated and secured using static routes, access control lists, and VLANs.
From a practical perspective, this is as secure as separate individual switches.

IP Services
The following subsections provide a description of some of the more important
emerging IP services that are often important components of a complete network
design for a Sun ONE deployment. These IP services are divided into two categories:

Designing the Network Configuration

11

Stateful Session Based - This class of IP services requires that the switch
maintain session state information, so that a particular clients session state is
maintained across all packets. We show that this requirement has severe
implications for highly available solutions, and limits scalabiity and performance.

Stateless Session Based - This class of IP services does not require that the switch
maintain any state information associated with a particular flow.

Many functions can be implemented by either network switches and appliances or


by the Sun ONE software stack. In this section, we describe:

How these new IP services work


What benefit they provide
Availability strategies

Later sections describe similar functions that are included in the Sun ONE integrated
stack.
Modern multilayer network switches perform many advanced Layer 3 IP services in
addition to basic routing. These services are implemented as functions that operate
on a packet by modifying the packet headers and controlling the rate at which they
are forwarded. IP services include functions such as Server Load Balancing, Quality
of Service, Application Redirection, Network Address Translation, and others. In this
section we start our discussion with Server Load Balancing, an important service for
data centers, and then describe adjacent services which can be cascaded.

Server Load Balancing - Class 2 Stateless


The Server Load Balancing (SLB) function maps incoming client requests destined
for a virtual IP (VIP) address and ported to a real server IP address and port. The
target server is selected from a set of identically configured servers, based on a predefined algorithm that considers the loads on the servers as criteria for choosing the
best server at any instant.
The purpose of server load balancing is to provide one layer of indirection to
decouple servers from the network service that clients interface with. Thus, the
server load-balancing function can choose the best server to service a client request.
Decoupling increases availability because if some servers fail, the service is still
available from the remaining functioning servers. Decoupling increases flexibility
because servers can be added or removed without impacting the service. Other
redirection functions can be cascaded to provide compound functionality.
SLB mapping functions differ from other mapping functions, such as redirection.
Redirection makes mapping decisions based on criteria such as ensuring that a
particular client is redirected to the same server to take advantage of caches, cookie
persistence, and so on. FIGURE 4 shows an overview of the various mapping
functions, and how they can intercept a request and rewrite the IP header according
to the provisioned configuration rules.

12

Network Design Patterns: N-Tier Data Centers October 2003

Client

Packet A
srcIP:a.b.c.d srcPort: 123 dstIP:VIP1 http://www.a.com/index.html

VIP1 = a.b.c.d:123
URL
String
Match

HTTP
Header

Cookie

Cache

VIP2 = e.f.g.h:456
SSL
Session
ID

SLB

Custom
Algorithm
First Function rewrote srcIP and port, so that the real server
will reply to this switch, which is at srcIP e.f.g.h and port 456.
Dest is set to the SLB function

Round
Robin

Least
Connections

SLB function, finds the best server, and rewrites the


dstIP of the target real server - Real IP

Packet A1

Packet A2

srcIP:e.f.g.h srcPort: 456 dstIP:VIP2 http://www.a.com/index.html

srcIP:e.f.g.h srcPort: 456 dstIP:RealIP http://www.a.com/index.html

FIGURE 4

Real Dest. IP
[1,2,3,...n]

IP Services Switch Functions on Incoming Packets

FIGURE 4 shows that a typical client request is destined for an external VIP, with IP
address a.b.c.d and port 123. The SLB algorithm eventually intercepts the packet,
and rewrites the destination IP address corresponding to the real server, which was
chosen by a particular algorithm. The packet is then returned as indicated by the
source IP address.

Application Redirection - Class 2 Stateless


The Application Redirection function intercepts a clients HTTP request and
redirects the request to another destination, which is usually a group of cache
servers. Application Redirection rewrites the IP destination field. This is different
from proxy switching, where the socket connection is terminated and a new
connection to the server is created to fetch the requested web page.
Application Redirection serves the following purposes:

Reduces the load on one set of web servers and redirects it to another set, usually
cache servers for specific content

Intercepts client requests and redirects them to another destination for control of
certain types of traffic based on filtered criteria

FIGURE 5 illustrates the functional model of Application Redirection, which only


rewrites the IP header.

Designing the Network Configuration

13

Client
servergroup 1
DEST = A

client http request


DEST = A
srcIP:a.b.c.d srcPort: 123 dstIP:VIP1 http://www.a.com/index.html

srcIP:a.b.c.d srcPort: 123 dstIP:VIPA http://www.a.com/index.html

Application Redirection
Filter has a defined destination IP
Client request meets filter criteria,
request is intercepted, IP dest is
rewritten to new destination IP addr
DEST = B

servergroup 2
DEST = B
srcIP:a.b.c.d srcPort: 123 dstIP:VIPB http://www.a.com/index.html

FIGURE 5

Application Redirection Functional Model

Content Switching - Class 1 Stateful


Content Switching is also known as Layer 7 processing, proxy switching, or URL
switching. This function accepts a clients incoming HTTP request, terminates the
socket connection, and creates another socket connection to the target web server,
which is chosen based on a user-defined rule. The Content Switching function then
fetches the requested web page and returns it to the client.
FIGURE 6 shows an overview of the functional Content Switching mode:

14

Network Design Patterns: N-Tier Data Centers October 2003

servergroup 1
stata
VIP
servergroup 2
dnsa
Layer 7 Switching Function
- terminate socket connection
- get URL
- check against rules
- forward to servergroup/slb function
- or get valid cookie, with server ID
and forward it to the same server

client http
request

http://www.a.com/SMA/stata/index.html servergroup1

servergroup 3
statb

servergroup 4
cacheb

http://www.a.com/SMA/dnsa/index.html servergroup2
http://www.a.com/SMB/statb/index.html servergroup3
http://www.a.com/SMB/CACHEB/index.html servergroup4

servergroup 5
dynab

http://www.a.com/SMB/DYNA/index.html servergroup1
FIGURE 6

Content Switching Functional Model

Content Switching with full Network Address Translation (NAT) serves the
following purposes:

Isolates internal IP addresses from being exposed to the public Internet.

Allows reuse of a single IP address. For example, clients can send their web
requests to www.a.com or www.b.com, where DNS maps both a.com and b.com
domains to a single IP address. The proxy switch receives this request, with the
packet containing an HTTP header in the payload that contains the target domain
(a.com or b.com), and redirects this request to the appropriate group of servers.

Allows parallel fetching of different parts of web pages from servers optimized
and tuned for that type of data. For example, a complex web page may need GIFs,
dynamic content, and cached content. With Content Switching, one set of web
servers can hold the GIFs, and another can hold the dynamic content. The proxy
switch can make parallel fetches and retrieve the entire page at a faster rate than
would otherwise be possible.

Ensures requests with cookies or Secure Sockets Layer (SSL) session IDs are
redirected to the same server to take advantage of persistence.

FIGURE 6 shows that the clients socket connection is terminated by the proxy
function. The proxy retrieves as much of the URL as is needed to make a decision
based on the retrieved URL. In this example, various URLs map to various server

Designing the Network Configuration

15

groups, which are virtual IP addresses. At this stage, the request may be forwarded
to the server, or passed off to the SLB function that is waiting for traffic destined for
the server group.
The proxy is configured with a virtual IP, so the switch forwards all client requests
destined for this virtual IP to the proxy function. The proxy function rewrites the IP
header, particularly the source IP and port, so that the server sends back the
requested data to the proxy instead of directly to the client.

Network Address Translation - Class 1 Stateful


Network Address Translation (NAT) is a critical component for security and proper
traffic direction. There are two basic types of NAT: half and full. Half NAT rewrites
the destination IP address and MAC address to a redirected location such as a web
cache, which returns the packet directly to the client because the source IP address is
unchanged. Full NAT is where the socket connection is terminated by a proxy, so the
source IP and MAC are changed to that of the proxy server.
NAT serves the following purposes:

SecurityPrevents exposing internal private IP addresses to the public.

IP Address ConservationRequires only one valid exposed IP address to fetch


Internet traffic from internal networks with invalid IP addresses.

RedirectionIntercepts traffic destined for one set of servers and redirects it to


another by rewriting the destination IP and MAC addresses. The redirected
servers are able to send back the request directly to the clients with half NAT
translated traffic because the original source IP address has not been rewritten.

NAT is configured with a set of filters, usually a quintuple Layer 3 rule. If the
incoming traffic matches a certain filter rule, the packet IP header is rewritten or
another socket connection is initiated to the target server, which itself can be
changed, depending on the rule.

Secure Sockets Layer (SSL) Session ID Persistence - Class 1


Stateful
SSL can be implemented in a variety of ways: in software, hardware, or both. SSL
can be terminated at the target server, an intermediate server, an SSL network
appliance, or at an SSL-capable network switch. An SSL appliance, such as Netscaler
or Array Networks, tends to be implemented with a PC board and have a PCI-based
card containing the SSL accelerator ASIC. Hence the SSL acceleration is
implemented in libraries, which offload only the mathematical computations. The
rest of the SSL processing is implemented in software, directing selective functions
to the hardware accelerator. Clearly, one immediate limitation is the PCI bus. Other

16

Network Design Patterns: N-Tier Data Centers October 2003

newer SSL devices have an SSL accelerator integrated into the datapath of the
network switch. These are very advanced products, just starting to emerge from
startups such as Wincom Systems.
FIGURE 7 shows an example of switch activity. Once a client has made initial contact
with a particular server, which may have been selected based on SLB, the switch
ensures that subsequent requests are forwarded to the same SSL server based on the
SSL ID that the switch has stored during the initial SSL handshake. The switch keeps
state information based on the clients initial request for HTTPS on port 443, which
contains a hello message. This first request is then forwarded to the server selected
by the SLB algorithm or by another function. The server responds to the clients
hello message with an SSL session ID. The switch then intercepts this SSL session
and stores it in a table. The switch forwards all of the clients subsequent requests to
the same server, as long as each request contains the SSL session ID in the HTTP
header. There may be several different TCP socket connections that span the same
SSL session. State is maintained by the SSL Session ID in each HTTP request sent by
the same client.

Client

SSL tunnel - SSL session ID


HTTP Session
HTTP Session
HTTP Session
HTTP Session

FIGURE 7

Network
Switch

SSL Server1

1
2
3
4

Network Switch with Persistence Based on SSL Session ID

An appliance can be added for increased performance in terms of SSL handshakes


and bulk encryption throughput. FIGURE 8 shows how an SSL appliance could be
deployed. Client requests first come in for a particular URL using the HTTPS
protocol on port 443. The switch recognizes that this needs to be directed to the
appliance, which is configured to provide that SSL service. A typical appliance, such
as Netscaler, can also be configured to provide Content Switching and Load
Balancing in addition to SSL acceleration. The appliance then reads or inserts
cookies and resubmits the HTTP request to an appropriate server, which maintains
state based on the cookie in the HTTP header.

Designing the Network Configuration

17

Sun ONE
Web Servers

Client

Internet

Multilayer
Switch
http
SSL
Session Persistence
Based on SSL Session ID

http
Session Persistence
Based on Cookie
https

SSL Accelerator Appliance


Key Exchange and Bulk Encryption
FIGURE 8

Tested SSL Accelerator Configuration

Cookie Persistence - Class 1 Stateful


The HTTP/1.0 protocol was originally designed to provide static pages in one
transaction. As more complex web sites evolved, which required multiple HTTP gets
on the same server, it was later discovered that performance was severely limited by
the tearing down and opening of TCP socket connections. This problem was solved
by HTTP 1.1, which allowed persistent connections. Immediately after a socket
connection, the client could pipeline multiple requests. However, as more complex
web sites evolved for applications such as the shopping cart, persistence across
multiple HTTP 1.1 requests was required. The problem was further complicated by
proxies and load balancers that interfered with the traffic being redirected to the
same web server. Another mechanism was required to maintain state across multiple
HTTP 1.1 requests. The solution was the introduction of two new headers in the
HTTP request, Set-Cookie and Cookies, as defined in RFC 2109. These headers
carried the state information between the client and server. Typically, most loadbalancing switches have enough intelligence to ensure that a particular clients
session with a particular server is maintained, based on the cookie that is inserted by
the server and maintained by the client.

18

Network Design Patterns: N-Tier Data Centers October 2003

Client Tier

External Network
Connectivity

FIGURE 9

Network Tier
Layer 3 redundancy
session based services
requires session sharing.
Other stateless services
failover with no problem.

Services Tier
Sun Servers with IPMPdual NICs

Network Availability Strategies

Designing Networks for Availability


FIGURE 9 shows a cross-sectional view of the tier types and the functions performed
at each tier. It also shows the availability strategies for the network and web tier.
External tier availability strategies are outside the scope of this article. Our
discussion is limited to the services tiers: Web, Application Services, Naming, and so
on.

Designing network architectures for optimal availability requires maximizing two


orthogonal components:

Intra AvailabilityRefers to maximizing the function that considers the


estimated failure probability of the components themselves. The components that
cause the failure are only considered by the following equation:

FAvailability = MTBF (MTBF + MTTR)


Where MTBF is Mean Time Between Failures and MTTR is Mean Time to
Recovery.

Designing the Network Configuration

19

Inter AvailabilityRefers to minimizing the impact of failures caused by factors


external to the system by the surrounding environment, such as single points of
failure (SPOFs), power outages, or a technician accidently pulling out a cable.

It is not sufficient to simply maximize the FAvailability function. The SPOF and
environmental factors also need to be considered. The networks designed in this
article describe a highly available architecture which conforms to these design
principles.
FIGURE 10 provides an overview of the logical network architecture. This shows how
the tiers map to the different networks, which are also mapped to segregate VLANs.
This segregation allows inter-tier traffic to be controlled by filters on the switch, or
possibly a firewall (which is the only bridge point between VLANs).

20

Network Design Patterns: N-Tier Data Centers October 2003

Client
network
172.16.0.0/24
External
network

External
network
192.168.10.0/24
Load
balancer

Load
balancer

Web service
network
10.10.0.0/24
Application
services network
10.30.0.0/24

Naming
services network
10.20.0.0/24

Management
network
10.100.0.0/24

Management
network

FIGURE 10

Production
network

Access
to all
networks

Backup
network
10.110.0.0/24

Database
service network
10.100.0.0/24
Device
network
10.0.0.0/24

Logical Network Architecture Overview

The following list describes each subnetwork shown in FIGURE 10:

External networkThe external-facing network that directly connects to the


internet. All IP addresses must be registered, and it is advisable to secure them
with a firewall.

Designing the Network Configuration

21

The following networks are assigned non-routable IP addresses, as specified by


RFC 1918, which can also be based on the following:

22

10.0.0.0 - 10.255.255.255 (10/8 prefix)


172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
192.168.0.0 - 192.168.255.255 (192.168/16 prefix)

Web Services networkA dedicated network containing web servers. Typical


configurations include a load-balancing switch. This switch can be configured to
allow the web server to return the clients HTTP request directly, or require the
load-balancing device to return the request on behalf of the providers web server.

Naming Services networkA dedicated network consisting of servers that


provide LDAP, DNS, NIS, and other naming services. The services are for internal
use only, and should be highly secure. Internal infrastructure support services
must be sure that requests originate and are destined for internal servers. Most
requests tend to be read intensive, hence the potential use of caching strategies for
increased performance.

Management networkA dedicated service network that provides management


and configuration for all servers, including the JumpStart installation option for
new systems.

Backup networkA dedicated service network that provides backup and restore
operations. This network is pivotal to minimizing disturbances to other
production service networks during backup and other network bandwidthintensive operations.

Device networkA dedicated network that attaches IP devices for storage, as


well as other devices.

Application Services networkA dedicated network, typically consisting of


larger multi-CPU servers that host multiple instances of the Sun ONE Application
Server software image. These requests tend to only require low network
bandwidth, but can span multiple protocols, including HTTP, CORBA,
proprietary TCP, and UDP-based protocols. The network traffic can also be
significant when Sun ONE application server clustering is enabled. Every update
to a stateful session bean triggers a multicast update to all servers on this
dedicated network so that participating cluster nodes update the appropriate
stateful session bean. Network utilization increases directly proportional to the
intensity of session bean updates.

Database networkA dedicated network, typically consisting of one or two


multi-CPU database servers. Network traffic typically consists of jdbc traffic
between the application server and the web server.

Network Design Patterns: N-Tier Data Centers October 2003

Networks and VLANs


Each service is deployed in a dedicated Class C network where the first three octets
represent the network number. The design represents an innovative approach where
separate Layer 2 devices are not required because the functionality was collapsed
into the core switch. Decreasing the management and configuration of separate
devices while maintaining the same functionality is a major step toward cutting
costs and increasing reliability.
FIGURE 11 shows how a traditional configuration requires two Layer 2 switches. A

particular VLAN spans the six segments, which gives each interface access to the
VLAN on failover.

Client network
Edge switch
Master
switch
(layer 3)

Standby
switch
(layer 3)

Layer 2
switch

Layer 2
switch
Network
interface 1

Network
interface 0

FIGURE 11

Sun server

Traditional Availability Network Design Using Separate Layer 2 Switches

The design shown in FIGURE 12 results in the same network functionality, but
eliminates the need for two Layer 2 devices. This is accomplished using a tagged
VLAN interconnect between the two core switches. Collapsing the Layer 2
functionality reduces the number of network devices. This provides a reduced risk
of unit failure, lower cost, and reduced manageability issues.

Designing the Network Configuration

23

Client network

Edge
switch

Master core switch 1

Standby core switch 2

Network
interface 1
Network
interface 0

FIGURE 12

Sun server

Availability Network Design Using Large Chassis-Based Switches

Solaris IPMP Host Network Interface Redundancy


Introduction
Solaris 8 introduced a novel technology that added network load balancing and path
failover with IP Multi Path (IPMP) built into the base operating system. IPMP is
used to provide failover between multiple physical network interfaces. Its
advantages over NAFO and other network interface redundancy approaches are
support for outbound load balancing, and a single mechanism to support both
clustered and non-clustered nodes. Failover is typically fast enough to preserve
application session connectivity. The default failover time is ten seconds. Normal
TCP/IP recovery ensures no loss of data. Interfaces are collected into groups. The
minimum number of interfaces assigned to a group is two. Where larger domains
are built, the interface group can scale to the total number of installed network
interface cards (NICs) that are attached to the same VLAN.
24

Network Design Patterns: N-Tier Data Centers October 2003

Note Currently, configuration of Quad Fast Ethernet cards is limited to eight


cards, and configuration of Gigabit Ethernet cards is limited to six cards. Additional
I/0 cards can be added by blacklisting and adding as needed. Please consult a Sun
Systems Engineer for the latest configuration rules.

Solaris IPMP Host Network Interface Redundancy Internals


FIGURE 13 describes the internal components of Solaris IPMP. The key is the Group
Identifier (GID) associated with each physical MAC. IP forwards packets destined
for a particular GID, as opposed to a physical interface.

Designing the Network Configuration

25

Sun server

thd

pa

in.m

atio

plic

Ap

ea

mh

a
tre

TC

.1

97

.6.

10
VIP
TIP

.1

97

.
0.6

ha

:b1

9b

:ff:

t2

20

:
8:0

t3

or

or

C
MA
ce0
Y
PH

or

2
R

3
t2
or
P

t3

or

or

or

t2

or

t1

or

et

t2

or

La

ye

or

t1

or

t2

La

GID

TIP

:b2

9b

:ff:

0
0:2

8:
C
MA
ce1
Y
PH

.53

97

.6.

10

ha

GID

ye

VIP

.54

97

.6.

10

P
R

t2

or
P

or
t3

P
P

or

t1

et

or
t2

or

La

ye

or
t

or
t1

or

t2

La

ye

or

t3

or

Network switch

Network switch

FIGURE 13

IPMP Internals

Further, IP has the ability to loadbalance across all physical interfaces that have a
common GID. As can be seen, each physical interface has a unique MAC and a
unique test IP address (TIP). The Virtual IP address associated with each physical
interface that belongs to the same GID has two modes of operation:
26

Network Design Patterns: N-Tier Data Centers October 2003

1. Active Active The VIP must be specified using the addif command for each
active interface. Only active interfaces are load balanced for outgoing packets.
2. Active StandbyThe VIP is not specified on a particular interface, and the
standby command is issued for the standby interface. In this case, the standby
interface does not participate in outbound load balancing. It associates with the VIP
only in the event that an active interface fails and remains down.
It is important to configure the network switch carefully when using IPMP. As
shown in FIGURE 13, there are two types of forwarding tables: Layer 3 for IP-based
forwarding, and Layer 2 forwarding. It is important that the MAC addresses are
different: the switch must have a unique port associated with each MAC address.
Otherwise the switch may get confused, depending on the network forwarding
implementation. Also, if a standby interface has just failed over, the VIP will be
associated with the old, failed MAC and the switch may incorrectly forward to the
failed NIC. It is important to use tested network switches that know, when a link
fails, to remove the appropriate entries from the forwarding table are re-ARPed for
an updated entry.
IPMP detects failures using two methods:

Link levelThe NIC knows when the link is physically down. The driver issues a
message to IP to perform failure detection operations based on the configurations.

IP levelThe daemon in.mpathd detects Layer 3 failures with the default router.
Periodically in.mpathd tests the connectivity between the TIP and the default
router, ensuring correct Layer 3 connectivity. In the event of a failure, a message is
sent to IP to to perform failure detection operations, based on the configurations.

Solaris IPMP Host Network Interface Redundancy


Availability
A typical highly available configuration includes a Sun server that has dual NIC
cards, which increases the availability of these components by several orders of
magnitude. For example, the Gigabit Ethernet card, part number X2069A, by itself
has an MTBF of 199,156 hours, and assuming approximately 2 hours MTTR, has an
availability of 0.999989958. With two cards, the MTBF increases so that availability
becomes 9 9s at .9999999996 availability. This small incremental cost has a big
impact on the overall availability computation.
FIGURE 14 shows the Sun server redundant NIC model using IPMP in Active Standby
mode. The server has two NICs, ge0 and ge1, with fixed IP addresses a.b.c.d and
e.f.g.h. The virtual IP address w.x.y.z is the IP address of the service. Client
requests use this IP address as the destination. This IP address floats between the
two interfaces: ge0 or ge1. Only one interface can be associated with the virtual IP
address at any one instant. If the ge0 interface owns the virtual IP address, then data

Designing the Network Configuration

27

traffic follows the P1 path. If the ge0 interface fails, the ge1 interface takes over and
associates the virtual IP address, and then data traffic follows the P2 path. Failures
can be detected within two seconds, depending on the configuration.
.
HCS

HCS
P2
P1

VLAN
IPMP-dual NIC
ge0:a.b.c.d
w.x.y.z
ge1:e.f.g.h

FIGURE 14

High-Availability Network Interface Cards on Sun Servers in Active Standby

Layer 3Integrated Virtual Router Redundancy Protocol


(VRRP) RFC 2338 and IPMP
By combining the availability technologies of routers and server NICs, we can create
a reusable cell that can be reused in any deployment where servers are connected to
routers. This reusable cell is highly available and scalable. In most LAN
environments, there are many servers or client nodes, with one default router. If that
default router fails, or the link that connects the hosts to the default router fails, then
connectivity to the outside network is down. To solve this problem, Virtual Router
Redundancy Protocol (VRRP), as defined in RFC 2338, was introduced. VRRP is
basically a multicast protocol used by switches to monitor one anothers health.
When a backup switch detects a failure of the master switch, it assumes the IP
address of the master. This is the same IP address used by all servers of a particular
VLAN as the default route. To integrate IPMP and VRRP so that both redundancy
protocols interoperate properly, care must be taken to ensure correct operating and
corrective operations in the event of a failure. FIGURE 15 describes a working
configuration. Lines 1 and 2 show the VRRP protocol used by the routers to monitor
one another. If one router detects that the other has failed, the surviving router
assumes the role of master and inherits the IP address and MAC address of the
master.
Lines 3 and 5 in FIGURE 15 show how a switch can verify that a particular connection
is up and running. This connection can be port-based, link-based, or based on Layers
3, 4, and 7. The router can make synthetic requests to the server and verify that a
28

Network Design Patterns: N-Tier Data Centers October 2003

particular service is up and running. If the router detects that the service has failed,
the VRRP can be configured on some switches to consider this failure as part of the
election algorithm and tie the failure to the priority of the VRRP router.
Simultaneously, the server is also monitoring links. Currently, IPMP consists of a
daemon, in.mpathd, that constantly pings the default router. As long as the default
router can be pinged, the master interface (ge0) assumes ownership of the IP
address. If the in.mpathd daemon detects that the default router is not reachable,
automatic failover occurs, which brings down the link and floats over the IP address
of the server to the surviving interface (ge1).
In the lab, we were able to tune IPMP and VRRP to achieve failure detection and
recovery within one second. There is a trade-off, however. Because the control
packets are on the same network as the production network, and because VRRP is a
CPU-intensive task, false failures are possible if the switches, networks, or servers
become overloaded. This is because the device might take longer than the strict
timeout to respond to the peers heartbeat.

VRRP

2
VRRP
6

4
ge1
ge0

FIGURE 15

in.mpathd

Design Pattern for IPMP and VRRP Integrated Availability Solution

N-Tier Data Center Logical Network Design


The logical network design for the N-Tier data center (FIGURE 16) incorporates server
redundant network interfaces and integrated VRRP and IPMP.

Designing the Network Configuration

29

Clients

172.16.0.1

192.168.0.1
Edge switch
Master

Slave
192.168.0.2

192.168.0.2

10.50.0.1

10.10.0.1

10.40.0.1

10.10.0.1

10.50.0.1

10.20.0.1

10.20.0.1

10.40.0.1

10.30.0.1

10.30.0.1

Servers

FIGURE 16

30

Logical Network Architecture With Virtual Routers, VLANs, and Networks

Network Design Patterns: N-Tier Data Centers October 2003

TABLE 4 summarizes the eight separate networks and associated VLANs.

TABLE 4

Network and VLAN Design

Name

Network

Default Router

VLAN

Purpose

client

172.16.0.0

172.16.0.1

client

Client load generation

edge

192.16.0.0

192.16.0.1

edge

Connects client network to internal


reference architecture network

web

10.10.0.0

10.10.0.1

web

Web services

ds

10.20.0.0

10.20.0.1

ds

Directory services

db

10.30.0.0

10.30.0.1

db

Database services

app

10.40.0.0

10.40.0.1

app

Application services

dns

10.50.0.0

10.50.0.1

dns

DNS services

mgt

10.100.0.0

10.100.0.1

mgt

Management and administration

san

10.0.0.0

10.0.0.1

san

Devices network

The edge network connects redundantly to the internal reference architecture


network. One of the core switches has ownership of the 192.16.0.2 IP address,
which means that switch is the master and the other is in slave mode. When the
switch is in slave mode, it does not respond to any traffic, including ARPs. The
master also assumes ownership of the MAC that floats along with the virtual IP
address of 192.16.0.2.

Note If you have multiple NICs, make sure each NIC uses its unique MAC
address.
Each switch is configured with the identical networks and associated VLANS that
are shown in TABLE 4. An interconnect between the switches extends each VLAN, but
is tagged to allow multiple VLANs to share a physical link. (This requires a network
interface, such as cards that use the ce driver, that supports tagged VLANS.) The
Sun servers connect to both switches in the appropriate slots, where only one of the
two interfaces will be active.
Most switches support Routing Information Protocol (RIP and RIPv2), Open Shortest
Path First (OSPF), and Border Gateway Protocol v4 (BGP4). However, static routes
provide a more secure environment. A redundancy protocol based on Virtual Router
Redundancy Protocol (VRRP, RFC 2338) runs between the virtual routers. The MAC
address of the virtual routers floats among the active virtual routers, so that the ARP
caches on the servers do not need any updates when a failover occurs.

Designing the Network Configuration

31

Inter-Tier Traffic Patterns


When a client makes a request, the request can be handled two ways depending on
the type of request: A web server might return information to the client directly, or it
might forward the request to an application server for further processing.
In the case where the clients request is for static content, such as images, the request
is handled directly by the web service tier. These requests are handled quickly, and
do not present a heavy load to the client or server.
In the case where the client requests dynamically generated content that requires
Java server pages (JSP) or servlet processing, the request is passed to the application
service tier for processing. This is often the bottleneck for large-scale environments.
The application server runs the core of the application that handles the business
logic to service the client request, either directly or indirectly. While handling the
business logic, the application server can use many supporting resources, including
directory servers, databases, and even other web application services.
FIGURE 17 illustrates how the data flows through the various system interfaces
during a typical application services request. TABLE 5 provides a description of each
numbered interaction in the illustration. Load balancers provide availability,
performance and scalability by spreading the incoming load across multiple web
servers. The web servers retrieve static content from an NFS server. To prevent NFS
from becoming the bottleneck, the web server can cache this content data onto a
local disk using cachefs. This preserves their statelessness but reduces NFS
network traffic and associated latencies. New changes to the web server content are
pulled according to demand onto the web servers as the data is accessed. Newly
added web servers can have their caches preloaded using the cachefspack [1m]
command.

It is important to note that the following is highly extensible and sufficiently


modular to allow for flexible variations according to customer needs.

32

Network Design Patterns: N-Tier Data Centers October 2003

Clients

12

Switching
services

Load
balancer

11

Web services

Static content
NFS server

5
4

Directory
services

10
Application
services

8
9

7
6

Database
services

FIGURE 17

Logical Network Describing Application Data Flow

Designing the Network Configuration

33

TABLE 5

34

Sequence of Events for FIGURE 17

Item

Interface1

Interface2

Protocol

Description

Client

Switch

HTTP/
HTTPS

Client initiates web request. Client


communication can be HTTP or HTTPS
(HTTP with secure socket layer). HTTPS
can be terminated at the switch or at the
web server.

Switch

Web server

HTTP/
HTTPS

Switch redirects client request to


appropriate web server.

Web Server

Application
server

Application
server web
connector
over TCP

The web server redirects the request to


the application server for processing.
Communication passes through a web
server plug-in over a proprietary TCPbased protocol.

Application
server

Directory
server

LDAP

The Java 2 Enterprise Edition (J2EE)


application hosted by the application
server identifies the requested process as
requiring specific authorization. It sends
a request to the directory server to verify
that the user has valid authorization.

Directory
server

Application
server

LDAP

The directory server successfully verifies


the authorization through the users
LDAP role. The validated response is
returned to the application server.
Application server then processes
business logic represented in J2EE
application.

Application
server

Database
server

JDBC

The business logic requests data from a


database as input for processing. The
requests may come from servlets, Java
Data Access Objects, or Enterprise
JavaBeans (EJBs) that in turn use Java
DataBase Connectivity (JDBC) to access
the database.

Database
server

Application
server

JDBC

The JDBC request can contain any valid


SQL statement. The database processes
the request natively and returns the
appropriate result through JDBC to the
application server.

Network Design Patterns: N-Tier Data Centers October 2003

TABLE 5

Sequence of Events for FIGURE 17 (Continued)

Item

Interface1

Interface2

Protocol

Description

Application
server

Web server

Application
server Web
connector
over TCP

The J2EE application completes the


business logic processing, packages the
data for display, usually through a JSP
that renders HTML, and returns the
response to the web server.

Web server

Switch

HTTP/
HTTPS

Switch receives reply from web server.

10

Switch

Client

HTTP/
HTTPS

Switch rewrites IP header and returns


request to client.

Configuring the Physical Network


The next step involves implementing a real network based on the logical network
design. This process involves:

Physical Network Connectivity on page 38


Configuring Switches on page 40

There are several approaches to realize a network that functionally satisfies the
logical architectural requirements.
The Sun ONE N-Tier Data Center is vendor independent, so you can use the
network equipment that best suits your environment. In the lab, we built two
different network configurations. One configuration uses Extreme Networks Black
Diamond 6800 Core Switches, (FIGURE 18), and the other uses Foundry Networks Big
Iron Core Switch and Server Iron XL Load Balancer (FIGURE 19). The Extreme
Networks switch that we used has built-in load balancing, so there is no need for an
external load-balancing device. The Foundry Networks products required a separate
load-balancing switch.
Similar implementations could be realized using products from Cisco or Nortel.

Configuring the Physical Network

35

Client 1

Client 2

Client
access

L2-L3
edge switch

192.168.10.1

10.50.0.1

Extreme
switches

Extreme switch
192.168.10.2

Core

Core

Extreme switch
192.168.10.3
10.30.0.1
10.40.0.1
10.20.0.1
10.10.0.1

Web
service
tier
Sun Fire 280R Sun Fire 280R Sun Fire 280R Sun Fire 280R
10.10.0.100 10.10.0.101 10.10.0.102 10.10.0.103

Naming
service
tier
Sun Fire 280R Sun Fire 280R Sun Fire 280R Sun Fire 280R
10.20.0.100 10.20.0.101 10.20.0.102 10.20.0.103

Application
service
tier

Data
service
tier

FIGURE 18

36

Sun Fire
6800
10.40.0.100

Sun Fire
6800
10.40.0.101

T3
Sun Fire
6800
10.30.0.100

T3

Sun Fire
6800
10.30.0.101

Sun ONE N-Tier Network Configuration With Extreme Networks Equipment

Network Design Patterns: N-Tier Data Centers October 2003

Client 1

Client 2

Client
access

Level 2-3 edge switch

192.168.10.1
10.50.0.1

Master core
192.168.10.2

Standby core
192.168.10.3

Foundry
switches

10.30.0.1
10.40.0.1
10.20.0.1
10.10.0.1
Server load-balancer switches

Web
service
tier

Sun Fire 280R Sun Fire 280R Sun Fire 280R Sun Fire 280R
10.10.0.100 10.10.0.101 10.10.0.102 10.10.0.103

Naming
service
tier

Sun Fire 280R Sun Fire 280R Sun Fire 280R Sun Fire 280R
10.20.0.100 10.20.0.101 10.20.0.102 10.20.0.103

Application
service
tier

Data
service
tier

FIGURE 19

Sun Fire
6800
10.40.0.100

Sun Fire
6800
10.40.0.101

T3
Sun Fire
6800
10.30.0.100

T3

Sun Fire
6800
10.30.0.101

Sun ONE N-Tier Network Configuration With Foundry Networks Equipment

Configuring the Physical Network

37

Physical Network Connectivity


FIGURE 20 is a standardized diagram showing the IP addresses for both the Extreme
Networks and Foundry Networks configurations.

The Extreme Networks configuration would be mapped directly to mls1 and mls2,
because these switches incorporate SLB functionality. Foundry, however, requires
additional load-balancing appliances.
FIGURE 20 elements are described further in TABLE 6.

38

Network Design Patterns: N-Tier Data Centers October 2003

ge0:172.16.0.102/24

ge0:172.16.0.101/24

Client2

Client1

172.16.0.1/24
Edge

hme0:10.100.16.102
10.100.16.1

192.168.0.1/24
6

hme0:10.100.16.101
10.100.168.2

10.100.168.2
mls1

192.168.0.101/24

192.168.0.102/24

mls2

10.30.0.1/24

192.168.0.2/24

192.168.0.2/24

10.30.0.1/24

10.20.0.1/24

10.10.0.1/24

10.10.0.1/24

10.20.0.1/24

192.168.0.2/24

192.168.0.2/24

ge0:10.10.0.101/24 ge0:10.10.0.103/24 ge0:10.10.0.105/24 ge0:10.10.0.107/24


web1

web1

web1

web1

ge1:10.10.0.102/24 ge1: 10.10.0.104/24 ge1:10.10.0.106/26 ge1:10.10.0.108/26

ge0:10.40.0.101/24

ge1:10.40.0.102/24

ge1:10.40.0.106/24

ge0:10.40.0.105/24

app1

app1

app2

app2

ge0:10.40.0.103/24

ge1:10.40.0.104/24

ge1:10.40.0.108/24
hme0:10.100.10.105

ge0:10.10.0.107/24

hme010.100.10.101

ge0:10.20.0.103/24

ge0:10.20.0.101/24

ds2

ds1
hme0:10.100.20.101

ge1:10.20.0.102/24

ge0:10.30.0.101/24

ge0:10.30.0.103/24

db1
hme0:10.100.30.101

FIGURE 20

ge1:10.20.0.104/24

hme0:10.100.20.103

db2
ge1:10.30.0.102/24

hme0:10.100.30.103

ge1:10.30.0.104/24

Physical Network Connections and Addressing

Configuring the Physical Network

39

TABLE 6

Physical Network Connections and Addressing

Switch

Description

Port

Speed (Mb/s)

Base
Address

Netmask

edge

Client network to external


network router

1,2,3,4

1000

172.16.0.1

255.255.255.0

edge

External network - mls1

5,6

1000

192.168.10.1

255.255.255.0

mls1

External network

1000

192.168.10.2

255.255.255.0

mls1

Web/application service
router

3,4,5,6

1000

10.10.0.1

255.255.255.0

mls1

Directory service router

7,8

1000

10.20.0.1

255.255.255.0

mls1

Database service router

9,10

1000

10.30.0.1

255.255.255.0

mls2

External network

1000

192.168.10.2

255.255.255.0

mls2

Web/application service
router

3,4,5,6

1000

10.10.0.1

255.255.255.0

mls2

Directory service router

7,8

1000

10.20.0.1

255.255.255.0

mls2

Database services router

9,10

1000

10.30.0.1

255.255.255.0

Configuring Switches
A high-level overview of the switch configuration is shown in FIGURE 21.

40

Network Design Patterns: N-Tier Data Centers October 2003

t7

i
mm

ks
or

Su

etw

N
me

tre

t
en .1
cli 6.0
2.1

Ex

ge .2
ed68.0
.1
92

ge .2
ed68.0
.1
92

1
b
we .0.1
0
1
.
0

ge .2
ed68.0
.1
92

1
b
we .0.1
0
0.1

ot

ot

ot

ot

ot

ot

ot

ot

Sl
Sl

Sl

Sl
Sl

Sl
Sl
Sl

ds.0.1
0
2
.
10 b
1
d .0.1
0
3
.
10 p
ap .0.1
.40
NS
0
1
,D
S
p
1
p
DN
.0.
,a
.50
db
0
,
1
ds
b,
e
w
tec
n
n
08
rco
68
e
t
d
in
t 1
on
rp
mg0.0.
am
es
i
.10
kD
10 Blac
rks
o
tw
Ne
e
m
tre
Ex
FIGURE 21

ot

ot

ot

ot

ot

ot

ot

ot

Sl

17

ds.0.1
0
2
.
10 b
1
d .0.1
0
3
.
10 p
ap .0.1
40
NS
.
,D
10 S
p
.1
ap
DN
b,
0.0
d
5
.
,
10
ds
b,
e
w
tec
n
n
8
co
80
er
6
t
n
d
i
t 1
on
rp
mg0.0.
es
am
i
.10
kD
10 Blac
rks
o
tw
Ne
e
m
tre
Ex

Sl
Sl
Sl
Sl

Sl
Sl
Sl

Collapsed Design Without Layer 2 Switches

Configuring the Physical Network

41

Configuring the Extreme Networks Switches


For the Sun ONE N-Tier, two Extreme Networks BlackDiamond switches were used
for the core switches and one Summit7i switch was used for the edge switch.

Note Network equipment from Foundry Networks can be used instead. Refer to
Configuring the Foundry Networks Switches on page 43.

To Configure the Extreme Networks Switches


1. Configure the core switches.
The following example shows an excerpt of the switch configuration file.
#
# MSM64 Configuration generated Thu Dec 6 20:19:20 2001
# Software Version 6.1.9 (Build 11)
By Release_Master on 08/30/01 11:34:27
configure slot 1 module g8x
configure slot 2 module g8x
configure slot 3 module g8x
configure slot 4 module g8x
configure slot 5 module g8x
configure slot 6 module g8x
configure slot 7 module f48t
configure slot 8 module f48t
.....................................................
configure dot1q ethertype 8100
configure dot1p type dot1p_priority 0 qosprofile QP1
configure dot1p type dot1p_priority 1 qosprofile QP2
configure dot1p type dot1p_priority 2 qosprofile QP3
configure dot1p type dot1p_priority 3 qosprofile QP4
.....................................................
enable sys-health-check
configure sys-health-check alarm-level log
enable system-watchdog
config qosprofile QP1 minbw 0% maxbw 100% priority Low minbuf 0% maxbuf 0 K
config qosprofile QP2 minbw 0% maxbw 100% priority LowHi minbuf 0% maxbuf 0 K

42

Network Design Patterns: N-Tier Data Centers October 2003

2. Configure the edge switch.


The following example shows an excerpt of the switch configuration file.
#
# Summit7i Configuration generated Mon Dec 10 14:39:46 2001
# Software Version 6.1.9 (Build 11)
By Release_Master on 08/30/01 11:34:27
configure dot1q ethertype 8100
configure dot1p type dot1p_priority 0 qosprofile QP1
....................................................
enable system-watchdog
config qosprofile QP1 minbw 0% maxbw 100% priority Low minbuf 0% maxbuf 0 K
....................................................
delete protocol ip
delete protocol ipx
delete protocol netbios
delete protocol decnet
delete protocol appletalk
....................................................
# Config information for VLAN Default.
config vlan Default tag 1
# VLAN-ID=0x1 Global Tag 1
config vlan Default protocol ANY
config vlan Default qosprofile QP1
enable bootp vlan Default
....................................................

Configuring the Foundry Networks Switches


This section describes the network architecture implementation using Foundry
Networks equipment instead of Extreme Networks equipment. The overall setup is
shown in FIGURE 22.

Configuring the Physical Network

43

Client

Client

NS5200
Netscreen
firewall

SLB0
Server load
balancer

FIGURE 22

44

Client

S7i
Extreme Networks
Summit 7i

MLS0
BigIron
layer 2/3 switch

MLS1
BigIron
layer 2/3 switch

Servers

Servers

Web
service
tier

Servers

Servers

Directory
service
tier

Servers

Servers

Application
service
tier

Servers

Servers

Database
service
tier

Servers

Servers

Foundry Networks Implementation

Network Design Patterns: N-Tier Data Centers October 2003

Client

NS5200
Netscreen
firewall

SLB1
Server load
balancer

Foundry Networks Master Core Switch Configuration


The following code box shows part of the configuration file for the BigIron master
core switch (called MLS0 in the lab).
module 1 bi-jc-8-port-gig-m4-management-module
module 3 bi-jc-48e-port-100-module
!
global-protocol-vlan
!
vlan 1 name DEFAULT-VLAN by port
vlan 10 name refarch by port
untagged ethe 1/1 ethe 3/1 to 3/16
router-interface ve 10
!
vlan 99 name mgmt by port
untagged ethe 3/47 to 3/48
router-interface ve 99
!
hostname MLS0
ip default-network 129.146.138.0/16
ip route 192.168.0.0 255.255.255.0 172.0.0.1
ip route 129.148.181.0 255.255.255.0 129.146.138.1
ip route 0.0.0.0 0.0.0.0 129.146.138.1
!
router vrrp-extended
interface ve 10
ip address 20.20.0.102 255.255.255.0
ip address 172.0.0.70 255.255.255.0
ip vrrp-extended vrid 1
backup priority 100 track-priority 20
advertise backup
ip-address 172.0.0.10
dead-interval 1
track-port e 3/1
enable
ip vrrp-extended vrid 2
backup priority 100 track-priority 20
advertise backup
ip-address 20.20.0.100
dead-interval 1
track-port e 3/13
enable
!
interface ve 99
ip address 129.146.138.10 255.255.255.0
!
end

Configuring the Physical Network

45

Foundry Networks Standby Core Switch Configuration


The following code box shows a partial listing of the configuration file for the
BigIron standby core switch (called MLS1 in the lab).
ver 07.5.05cT53
!
module 1 bi-jc-8-port-gig-m4-management-module
module 3 bi-jc-48e-port-100-module
!
global-protocol-vlan
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 99 name swan by port
untagged ethe 1/6 to 1/8
router-interface ve 99
!
vlan 10 name refarch by port
untagged ethe 3/1 to 3/16
router-interface ve 10
!
!
hostname MLS1
ip default-network 129.146.138.0/1
ip route 192.168.0.0 255.255.255.0 172.0.0.1
ip route 0.0.0.0 0.0.0.0 129.146.138.1
!
router vrrp-extended
interface ve 10
ip address 20.20.0.102 255.255.255.0
ip address 172.0.0.71 255.255.255.0
ip vrrp-extended vrid 1
backup priority 100 track-priority 20
advertise backup
ip-address 172.0.0.10
dead-interval 1
track-port e 3/1
enable
ip vrrp-extended vrid 2
backup priority 100 track-priority 20
advertise backup
ip-address 20.20.0.100
dead-interval 1
track-port e 3/13
enable
interface ve 99
ip address 129.146.138.11 255.255.255.0
!
!
!
!
!
sflow sample 512
sflow source ethernet 3/1
sflow enable
!
!
!
end

46

Network Design Patterns: N-Tier Data Centers October 2003

Foundry Networks Master Server Load Balancer Configuration


The following code box shows a partial listing of the configuration file used for the
master Server XL server load balancer (called SLB0 in the lab).
ver 07.3.05T12
global-protocol-vlan
!
!
server source-ip 20.20.0.50 255.255.255.0 172.0.0.10
!
!!
!
server real web1 10.20.0.1
port http
port http url "HEAD /"
!
server real web2 10.20.0.2
port http
port http url "HEAD /"
!
!
server virtual WebVip1 192.168.0.100
port http
port http dsr
bind http web1 http web2 http
!
!
vlan 1 name DEFAULT-VLAN by port
no spanning-tree
!
hostname SLB0
ip address 192.168.0.111 255.255.255.0
ip default-gateway 192.168.0.10
web-management allow-no-password
banner motd ^C
Reference Architecture -- Enterprise Engineering^C
Server Load Balancer-- SLB0 129.146.138.12/24^C
!!
end

Configuring the Physical Network

47

Foundry Networks Standby Server Load Balancer Configuration


The following code box shows a partial listing of the configuration file used for the
standby Server XL server load balancer (called SLB1 in the lab).
ver 07.3.05T12
global-protocol-vlan
!
!
server source-ip 20.20.0.51 255.255.255.0 172.0.0.10
!
!!
!
server real s1 20.20.0.1
port http
port http url "HEAD /"
!
server real s2 20.20.0.2
port http
port http url "HEAD /"
!
!
server virtual vip1 172.0.0.11
port http
port http dsr
bind http s1 http s2 http
!
!
vlan 1 name DEFAULT-VLAN by port
!
hostname SLB1
ip address 172.0.0.112 255.255.255.0
ip default-gateway 172.0.0.10
web-management allow-no-password
banner motd ^C
Reference Architecture - Enterprise Engineering^C
Server Load Balancer - SLB1 - 129.146.138.13/24^C
!

Network Security
For the N-Tier network design, firewalls were configured between each service tier to
provide network security. FIGURE 23 shows the relationship between the firewalls
and the service tiers.

48

Network Design Patterns: N-Tier Data Centers October 2003

Client

Client

Intranet/Internet
Edge
switch
Firewall
Web service
tier
Firewall
Application service
tier
Firewall
Database service
tier

FIGURE 23

Firewalls Between Service Tiers

In the lab, one physical firewall device was used to create multiple virtual firewalls.
Network traffic was directed to pass through the firewalls between the service tiers,
as shown in FIGURE 24.
The core switch is only configured for Layer 2 with separate port-based VLANs. The
connection between the Netscreen device and the core switch uses tagged VLANS.
Trust zones are created on the Netscreen device, and they map directly to the tagged
VLANs. The Netscreen firewall device performs the Layer 3 routing. This
configuration directs all traffic through the firewall, resulting in firewall protection
between each service tier.

Network Security

49

Client

Client

Intranet/Internet
Edge
switch

Netscreen
device

VLAN*

Core
switch

Core
switch
VLAN* Netscreen
device

Database VLAN

Database VLAN

Application VLAN

Application VLAN

Web VLAN

Web VLAN

Web service
tier

Application service
tier

Traffic

Database service
tier

*Web, application and database traffic multiplexed on one VLAN


FIGURE 24

50

Virtual Firewall Architecture Using Netscreen and Foundry Networks


Products

Network Design Patterns: N-Tier Data Centers October 2003

Netscreen Firewall
The following code box shows a partial example of a configuration file used to
configure the Netscreen device.

set auth timeout 10


set clock "timezone" 0
set admin format dos
set admin name "netscreen"
set admin password nKVUM2rwMUzPcrkG5sWIHdCtqkAibn
set admin sys-ip 0.0.0.0
set admin auth timeout 0
set admin auth type Local
set zone id 1000 "DMZ1"
set zone id 1001 "web"
set zone id 1002 "appsrvr"
set zone "Untrust" block
set zone "DMZ" vrouter untrust-vr
set zone "MGT" block
set zone "DMZ1" vrouter trust-vr
set zone "web" vrouter trust-vr
set zone "appsrvr" vrouter trust-vr
set ip tftp retry 10
set ip tftp timeout 2
set interface ethernet1 zone DMZ1
set interface ethernet2 zone web
set interface ethernet3 zone appsrvr
set interface ethernet1 ip 192.168.0.253/24
set interface ethernet1 route
set interface ethernet2 ip 10.10.0.253/24
set interface ethernet2 route
set interface ethernet3 ip 20.20.0.253/24
set interface ethernet3 route
unset interface vlan1 bypass-others-ipsec
unset interface vlan1 bypass-non-ip
set interface ethernet1 manage ping
unset interface ethernet1 manage scs
unset interface ethernet1 manage telnet
unset interface ethernet1 manage snmp
unset interface ethernet1 manage global
unset interface ethernet1 manage global-pro
unset interface ethernet1 manage ssl
set interface ethernet1 manage web
unset interface ethernet1 ident-reset
set interface vlan1 manage ping
set interface vlan1 manage scs
set interface vlan1 manage telnet
set interface vlan1 manage snmp
set interface vlan1 manage global
set interface vlan1 manage global-pro
set interface vlan1 manage ssl
set interface vlan1 manage web

Network Security

51

set interface v1-trust manage ping


set interface v1-trust manage scs
set interface v1-trust manage telnet
set interface v1-trust manage snmp
set interface v1-trust manage global
set interface v1-trust manage global-pro
set interface v1-trust manage ssl
set interface v1-trust manage web
unset interface v1-trust ident-reset
unset interface v1-untrust manage ping
unset interface v1-untrust manage scs
unset interface v1-untrust manage telnet
unset interface v1-untrust manage snmp
unset interface v1-untrust manage global
unset interface v1-untrust manage global-pro
unset interface v1-untrust manage ssl
unset interface v1-untrust manage web
unset interface v1-untrust ident-reset
set interface v1-dmz manage ping
unset interface v1-dmz manage scs
unset interface v1-dmz manage telnet
unset interface v1-dmz manage snmp
unset interface v1-dmz manage global
unset interface v1-dmz manage global-pro
unset interface v1-dmz manage ssl
unset interface v1-dmz manage web
unset interface v1-dmz ident-reset
set interface ethernet2 manage ping
unset interface ethernet2 manage scs
unset interface ethernet2 manage telnet
unset interface ethernet2 manage snmp
unset interface ethernet2 manage global
unset interface ethernet2 manage global-pro
unset interface ethernet2 manage ssl
unset interface ethernet2 manage web
unset interface ethernet2 ident-reset
set interface ethernet3 manage ping
unset interface ethernet3 manage scs
unset interface ethernet3 manage telnet
unset interface ethernet3 manage snmp
unset interface ethernet3 manage global
unset interface ethernet3 manage global-pro
unset interface ethernet3 manage ssl
unset interface ethernet3 manage web
unset interface ethernet3 ident-reset
set interface v1-untrust screen tear-drop
set interface v1-untrust screen syn-flood
set interface v1-untrust screen ping-death
set interface v1-untrust screen ip-filter-src
set interface v1-untrust screen land
set flow mac-flooding
set flow check-session
set address DMZ1 "dmznet" 192.168.0.0 255.255.255.0
set address web "webnet" 10.10.0.0 255.255.255.0

52

Network Design Patterns: N-Tier Data Centers October 2003

set address appsrvr "appnet" 20.20.0.0 255.255.255.0


set snmp name "ns208"
set traffic-shaping ip_precedence 7 6 5 4 3 2 1 0
set ike policy-checking
set ike respond-bad-spi 1
set ike id-mode subnet
set l2tp default auth local
set l2tp default ppp-auth any
set l2tp default radius-port 1645
set policy id 0 from DMZ1 to web "dmznet" "webnet" "ANY" Permit
set policy id 1 from web to DMZ1 "webnet" "dmznet" "ANY" Permit
set policy id 2 from DMZ1 to appsrvr "dmznet" "appnet" "ANY" Permit
set policy id 3 from appsrvr to DMZ1 "appnet" "dmznet" "ANY" Permit
set ha interface ethernet8
set ha track threshold 255
set pki authority default scep mode "auto"
set pki x509 default cert-path partial
_____________________

About the Authors


Deepak Kakadia is a staff engineer and network architect in the Reference
Architecture Engineering Group for Sun Microsystems in Menlo Park, California.
Deepak has been with Sun since 1994. He has previously worked with Corona
Networks, Digital Equipment Corp., and Nortel Networks. He has a Bachelors
Degree in Computer Systems Engineering, a Masters Degree in Computer Science,
has completed Ph.D qualifying exams, and has completed his SCPD certificate in
Networking through the Department of Electrical Engineering at Stanford
University. He has filed four patents in the area of Networking and Systems
Management.
Richard Croucher is Chief Architect of Suns Professional Services in EMEA. He is
also an SMI Technical Director. Richard joined Sun in 1995. He has nearly 20 years of
experience with UNIX and nearly 30 years of experience in the IT and electronics
industries. He holds a Higher National Certificate in Applied Physics, a Higher
National Diploma in Electronics, and has received a post-graduate Certificate of
Advanced Study in Non-Destructive Testing and Materials Science from Brunel
University. He was elected to membership of the British Computer Society in 1993.

About the Authors

53

Acknowledgements
Sincere thanks to Bill Sprouse, Kemer Thomson, Brad Carlile, and many others who
provided tremendous support, guidance, and direction to this article.

Related Resources
Sun ONE Directory Server 5.1 documentation collection
Note: This product was formerly known as the iPlanet Directory Server
software. You can locate the documentation collection at:
http://docs.sun.com
Search for the phrase directory server 5.1. Start with the iPlanet Directory
Server 5.1 Deployment Guide.

Ordering Sun Documents


The SunDocsSM program provides more than 250 manuals from Sun Microsystems,
Inc. If you live in the United States, Canada, Europe, or Japan, you can purchase
documentation sets or individual manuals through this program.

Accessing Sun Documentation Online


The docs.sun.com web site enables you to access Sun technical documentation
online. You can browse the docs.sun.com archive or search for a specific book title
or subject. The URL is http://docs.sun.com/
To reference Sun BluePrints OnLine articles, visit the Sun BluePrints OnLine web site at:
http://www.sun.com/blueprints/online.html

54

Network Design Patterns: N-Tier Data Centers October 2003

You might also like