OMICRON

Page: 1 K02 03 20060309

K02 03 20060309
Monitoring of Ethernet
Monitoring of Ethernet
Messages
Messages
Dr. Alexander Apostolov
OMICRON Page: 2 K02 03 20060309
Scope
Scope
Discuss Ethernet architectures supporting
IEC 61850 projects
Not defined in the standard but essential for a
real project !
Background on Ethernet principles and
technologies
Practical examples
OMICRON Page: 3 K02 03 20060309
SIMPLEX
Transmission in one direction only
HALF DUPLEX
Two way means of transmission but data can only
travel in one direction at a time
FULL DUPLEX
Transmission in both directions simultaneously
Transmission Definitions
Transmission Definitions
OMICRON Page: 4 K02 03 20060309
PROTOCOL
Rules and procedures that communications networks
use to communicate on the communications medium
CONNECTION
Communications are Connection Oriented or
Connectionless
Transmission Definitions
Transmission Definitions
OMICRON Page: 5 K02 03 20060309
Point-to-point
Transmission Definitions
Transmission Definitions
IED
Engineering
Station
Modem
Modem
Laptop
IED
OMICRON Page: 6 K02 03 20060309
Shared Access
Transmission Definitions
Transmission Definitions
IEC 61850
IED
Legacy
IED
IEC 61850
IED
Proxy
Server
Substation
HMI
RS485
Ethernet
Ethernet
Switch
Substation
HMI IED IED IED
Ethernet
OMICRON Page: 7 K02 03 20060309
NETWORK ACCESS
CARRIER SENSE METHODS
CSMA (Carrier Sense Multiple Access)/CD
(Collision Detection)
Medium Access Control (MAC)
TOKEN ACCESS METHOD
The device that has the Token has access to
transmit
RESERVATION METHOD
Each device has a predefined time slot to transmit
Transmission Definitions
Transmission Definitions
OMICRON Page: 8 K02 03 20060309
Medium - Ethernet devices attach to a common medium that
provides a path along which the electronic signals will travel:
- historically, this medium has been coaxial copper cable
- more commonly a twisted pair
-fiber optic cabling.
Segment - a single shared medium as an Ethernet segment.
Nodes - devices that attach to that segment are stations or
nodes.
Frame - The nodes communicate in short messages called
frames, which are variably sized chunks of information.
Transmission Definitions
Transmission Definitions
OMICRON Page: 9 K02 03 20060309
RS 232 - 9/25 pin D connector
EIA 232 - Maximum data rate 20kb/s
- Maximum distance 50 feet
RS 485 - 2/4 conductors
EIA 485 - Maximum data rate 10Mb/s
- Maximum distance 4000 feet
Optical fiber - Immunity to electrical interference
- Advantages in distance and speed
Wireless
Transmission Medium
Transmission Medium
OMICRON Page: 10 K02 03 20060309
Noise corruption of data
Parity bit check
Two coordinate parity check
Checksums
Cyclic redundancy check
Unauthorized access
Password protection
Modem dial back
Firewalls
Data Security
Data Security
OMICRON Page: 11 K02 03 20060309
Presentation
Session
Transport
Network
Data Link
Physical
Application Selects appropriate service for application
Provides code conversion, data reformatting
Coordinates interaction between end
application process
Provides for end to end data integrity and
quality of service
Switches and routes information
Transfers unit of information to other end
of physical link
Transmits bit stream to medium
Open Systems Interconnection
Open Systems Interconnection
(OSI) Model
(OSI) Model
OMICRON Page: 12 K02 03 20060309
OSI Stack
OSI Stack
Application
Presentation
Session
Transport
Network
Data link
Physical
PDU PCI
PDU PCI
PDU PCI
PDU PCI
PDU PCI
Commands or data
PDU
fragment
T H
PDU
fragment
T H
PDU
fragment
T H
PCI =Protocol Control Inform.
PDU =Protocol Data Unit
H =Header
T =Trailer
Frames (Ethernet, token ring, etc)
OMICRON Page: 13 K02 03 20060309
Communications Process
Communications Process
Upper Layer
Lower Layer
Entity with
services to offer
SAP
SAP
Upper Layer
Lower Layer
Entity with
services to offer
SAP
SAP
Peer-to-peer dialog
OMICRON Page: 14 K02 03 20060309
Ethernet Frame
Ethernet Frame
Pre SFDDA SA Length
Type
MAC Data +Pad FCS
7 1 6 6 2 46-1500bytes 4
Pre: The Preamble is an alternating pattern (7 bytes) of 1 and 0 that
tells receiving stations that a frame is coming
SFD: Start-of-frame delimiter (1 byte: 10101011) indicating that the
next bit is the left-most bit in the left-most byte of the destination
address.
DA: Destination address (6 bytes) identifies which station(s) should
receive the frame
SA: Source addresses (6 bytes) identifies the sending station
OMICRON Page: 15 K02 03 20060309
Ethernet Frame
Ethernet Frame
Pre SFDDA SA Length
Type
MAC Data +Pad FCS
7 1 6 6 2 46-1500bytes 4
Length Type: Number of MAC-client data bytes that are contained
in the data field of the frame
MAC Client Data: A sequence of n bytes (46=< n =<1500) of any
value. (The total frame minimum is 64 bytes). The Pad contains (if
necessary) extra data bytes in order to bring the frame length up to
its minimum size. A minimum Ethernet frame size is 64 bytes from
the Destination MAC Address field through the Frame Check
Sequence.
FCS: The Frame Check Sequence is a 32-bit cyclic redundancy
check (CRC) value
OMICRON Page: 16 K02 03 20060309
Media Access Control (MAC) Address -
This is the physical address of any device,
such as the NIC in a computer, on the
network.
The MAC address has two parts, each 3
bytes long.
The first 3 bytes identify the company that
made the NIC.
The second 3 bytes are the serial number
of the NIC itself.
Ethernet
Ethernet
OMICRON Page: 17 K02 03 20060309
Unicast - A transmission from one node
addressed specifically to another node.
Multicast - When a node sends a
packet addressed to a special group
address. Devices that are interested in
this group register to receive packets
addressed to the group.
Broadcast - When a node sends out a
packet that is intended for transmission
to all other nodes on the network.
Ethernet
Ethernet
OMICRON Page: 18 K02 03 20060309
Network An interconnected group of nodes or
stations linked by communication channels
Node The interface point where one or more
functional units are connected
LAN Local area network (<5km)
WAN Wide area network
Network topology Pattern of nodes and their interconnection
Network Terminology
Network Terminology
OMICRON Page: 19 K02 03 20060309
Ethernet Basic Principle
Ethernet Basic Principle
Collisions are an issue for real time automation
Ethernet
1. Listen and broadcast if the wire is free
2. If there is a collision then re-transmit in an interval [0, 2
N-1
]
OMICRON Page: 20 K02 03 20060309
Switches principles
Switches principles
Switches eliminate collisions and are thus
systematically used for industrial applications
Internal Switches
Queue
OMICRON Page: 21 K02 03 20060309
Switches: Performances
Switches: Performances
Performances bottleneck is no longer the communication
Network but possibly the applications (ms timefame)
Pure delay (store & forward): 5-10s
Maximum frame size: 1536 bytes
1536 bytes @ 100 Mbps = 123 s
Goose size << 1536 bytes
OMICRON Page: 22 K02 03 20060309
Switches:
Switches:
Unicast
Unicast
1
2
OMICRON Page: 23 K02 03 20060309
Switches: Multicast
Switches: Multicast
1
2 2
OMICRON Page: 24 K02 03 20060309
Switches: Broadcast
Switches: Broadcast
1
2 2
2
2
OMICRON Page: 25 K02 03 20060309
Wall
Wall
-
-
to
to
-
-
wall performances
wall performances
example (ms)
example (ms)
Process
Binary Input
Process
Binary Output
Filtering [2-10]
Logical [1-50]
Communication [2-15]
Relaying [1-7]
To
Switching [0.1]
Communication [2-15]
Logical [1-50]
[5-75]
[4-72]
To + 9/147
OMICRON Page: 26 K02 03 20060309
Switches: other features used in
Switches: other features used in
Substation Automation
Substation Automation
Substation environment !
Electro-magnetic compatibility
DC power supply with 20ms voltage dips, possibly
redundant
Priority management (802.3p):
Capability to have priority queues in order to further
boost the communication performances
Virtual LAN VLAN (802.3q):
Capability to create logical groups of devices in order
to filter the messages not belonging to one group
OMICRON Page: 27 K02 03 20060309
Switches: other features used in
Switches: other features used in
Substation Automation
Substation Automation
Management
Capability to detect a switch failure: SNMP or watch-
dog
Redundancy management
No standard today for hard real time redundancy
Redundancy between switches (not applications)
See further slides
OMICRON Page: 28 K02 03 20060309
Base architectures
Base architectures
DOUBLE STAR
The most secured
scheme since multiple
failures are tolerated
except the two central
switches
simultaneously
RING
The most common scheme since
good performance/cost ratio
(save 2 central switches vs. start
configuration)
Failure of a single fiber or device
does affect the system once
reconfiguration is completed
STAR
Basic scheme since
failure of the central
switch leads to the
total communication
failure
OMICRON Page: 29 K02 03 20060309
Redundancy management
Redundancy management
A way to improve system availability and reliability
Behavior is not defined in IEC 61850
Typical example where the architect needs to define what
shall be done and first check on paper that it can be
supported by the IEDs
May be applied to:
Communication infrastructure: tolerate the loss of an optical
fiber or/and the loss of a central switch
Clients: capability for a client to continue the tasks initiated
by a first client (example: Graphical user interface)
Servers: capability for a client to switch to a redundant server
if the first one is not operational (example: CT/VT sensor)
99.99% availability requests the 3 types of redundancy
OMICRON Page: 30 K02 03 20060309
Redundancy management at
Redundancy management at
communication level
communication level
Defined between Ethernet switches
If a device has two Ethernet port it must also have an integrated
switch
Spanning tree mechanisms
Recalculation of the route between switches using an internal
protocol between switches and avoiding loops
Base: 802.3d. Typically 30 s reconfiguration time
Fast: 803w. Typically 100 ms reconfiguration time, some
implementations claims 5 ms per switch (N x 5ms with a ring made of
N switches)
Other mechanisms
Detects the failure of the adjacent switch to reconfigure
For ring topology, less than 1 ms reconfigurations
Sends two signals at the same time and keep the first one arrived
No switch-over time, but need to keep a correct buffer size (dual homing)
Substation applications enable to get extremely fast
Switch-over time
OMICRON Page: 31 K02 03 20060309
SWITCH
SHM
Self healing ring
manager
opto
Et her net Por t s 10/100 Base TX
opto
Opt i cal Ri ng
Primary Fiber
Secondary Fiber
EEPROM
Port MII
N
100Mbs Full duplex
Failsafe
output relays
Example (ring)
Example (ring)
Repeaters
(IEE 802.3 Ethernet switch)
+ Self Healing Manager (SHM)
OMICRON Page: 32 K02 03 20060309
Example (ring)
Example (ring)
EP
RS
RP
ES
Primary fibre
Secondary fibre
A B
C
D
E
1 2 3
5 6 7
9 10 11 8
4
switch switch
switch
switch
switch
During nominal situation, Ethernet packet goes in
primary fibre always in the same direction, and
only a checking frame (4 bytes) is sent every 5ms
in secondary fibre in the opposite direction.
OMICRON Page: 33 K02 03 20060309
EP
RS
RP
ES
Primary fibre
Secondary fibre
A B
C
D
E
1 2 3
5 6 7
9 10 11 8
4
switch switch
switch
switch
switch
Example (ring)
Example (ring)
If the connection between 2 switches is broken
The Ethernet network will continue to run correctly.
Both SHM start immediately the network self-healing.
At one side, the messages received are no more emitted to
the primary fibre but to the secondary fibre.
On the other side of the cut off, the messages received to secondary
are emitted to primary and the new topological loop is closed.
OMICRON Page: 34 K02 03 20060309
Other communication elements
Other communication elements
Need to design the communication infrastructure
according to the system constraints and requirements
OSI Model
Physical
Data Link
Network
Transport
Session
Presentation
Application
Hardware
Software
Technology Protocol
802.3
802.3
IP
TCP
MMS/SNTP
Com.
Element
Bridge/Switch
/Proxy
Router
Gateway/
Proxy
IEC Com.
Services
GOOSE
Report, Control,
File
OMICRON Page: 35 K02 03 20060309
Proxy use
Proxy use
Use proxy to keep sufficient performances in large systems
or/and enable a progressive commissioning
Server Server Server
Client Client
IP Routable
Fast Peer-to-Peer
Client
T104,
http, etc.
GUI
Proxy
Bay
Bay
Bay
Bay
Bay
Bay
Gateway
Proxy
Server (resp.
subscriber, clients)
performance is
dependant on the
number of clients
(resp. pubslisher,
servers)
A proxy can replicate
the real time status &
measurement of a
series of servers and
be possibly
redundant
OMICRON Page: 36 K02 03 20060309
IEC 62351
IEC 62351
Data and Communication Security series of standard
Currently at CD stage (votes till August 2005)
IEC 62351-6: Security for IEC 61850 profiles
Relies on IEC 62351-4: Profiles including MMS
Relies on IEC 62351-3: Profiles including TCP
OSI Model
Physical
Data Link
Network
Transport
Session
Presentation
Application
Hardware
Software
Technology Protocol
802.3
802.3
IP
TCP
MMS/SNTP
IEC Com.
Services
GOOSE
Report, Control,
File
Security
Services
Authentification,
Replay
Authentification,
Encryption, Replay
Authentification,
Replay