IT Application Monitoring Using PI AF and Notifications

Josiah Thoen - AEP

2008 OSIsoft, Inc. | Company Confidential

AEP Who We Are By Assets

One of the largest U.S. electricity generators ( ~ 38,000 MWs) with a significant cost advantage in many of our market areas Largest consumer of coal in the Western Hemisphere

Operations within four RTOs PJM SPP ERCOT MISO

A leading consumer of natural gas Major wind power developer (#3 in U.S. in 2005) 39,000 miles of transmission 186,000 miles of distribution Coal transportation 7,000 railcars 2,230 barges and 53 towboats 5 million customers
2008 OSIsoft, Inc. | Company Confidential

AEP IT - Service Performance

Monitoring AEP IT Assets for over 7 years
Servers Hardware, Operating Systems (Windows and Unix) Middleware Platforms IIS, BEA WebLogic, etc. Applications In house custom applications Vendor Applications
BlackBerry Server, Lotus Notes, etc.

2008 OSIsoft, Inc. | Company Confidential

AEP IT - PI Infrastructure
PI Server Collective
2 Servers 8 CPU, 8gb RAM, 85gb SAN Drive Space Software:
PI Server

Analytics Server
2 CPU, 2gb RAM, 30gb Local Drive Space Software: ACE (Primary)

AF & Notification MS SQL Express ACE (Backup)

Stats:
Tags: 94,049 Modules: 54,076 AF Elements: ~10,000 AF Element Templates: ~25 Notifications: 331 Data Storage: 1gb every 23days

Data Acquisition Node

2 CPU, 2gb RAM, 30gb Local Drive Space Software: IT Monitor Interfaces Custom PISDK monitoring services

2008 OSIsoft, Inc. | Company Confidential

AEP IT and PI the beginning

PI IT Monitor Pilot
Customer IT Application Support Teams Primary IT Operations Secondary Business Units - Secondary

Purpose Production Server and Application visibility

Result Infrastructure monitoring approach Application Monitoring
5

2008 OSIsoft, Inc. | Company Confidential

AEP IT and PI - Problems

Time Series data is not relational
Convoluted naming conventions and overloaded tag attributes Network > Datacenter > Server > Components > Application > Middleware > Database > and back

Maintenance nightmare
Everything is manual Numerous duplicate displays

No Event Management
All this data but no way to knowing when something is broken.

2008 OSIsoft, Inc. | Company Confidential

AEP IT and PI - Solutions

Problem: Time Series data is not relational
Solution: AF Relational Meta Data on top of Time Series data.

Problem: Maintenance nightmare

Solution: AF Template based management of elements and their attributes

Problem: No Event Management

Solution: PI Notifications Template based triggered event delivery

2008 OSIsoft, Inc. | Company Confidential

AF Implementation
WebMethods BrokerClient QueueLength monitoring
SoftwareAG WebMethods SOA, B2B messaging platform Pub-Sub Message Brokering
Publisher sends a message to a Broker. Broker delivers the message to one or more Subscribers of that message.

Used in many Tier 1 and 2 application services and business processes

Facilitates communication with external counter parties Used Internally to send data between applications.

2008 OSIsoft, Inc. | Company Confidential

Business Need
Monitor the Broker to make sure that client queues are not getting full.
Publishing clients send messages to the Broker Broker delivers the message to subscribing clients queue. Subscribing clients retrieve messages from their queue. Implications of failure: Client may be unavailable. Broker may be having problems.

2008 OSIsoft, Inc. | Company Confidential

Functional Requirements
Store the Broker Client Info data. Alerting:
Warn when queue contains 25+ messages. Alarm when queue contains 75+ messages. Alerts must send a Clear message when the alerting condition is gone.

Alerting Recipient:
Email notification to those interested. AEP IT Command Center: 24x7 Support Center. All Events go to Central Event Management System.

2008 OSIsoft, Inc. | Company Confidential

10

Monitoring Process
Data Acquisition PISDK Custom Monitor
Invoke a service within the WebMethods environment every 60 seconds. Parse data. Do exception testing Send data to PI. Automatically Add new Broker Clients to AF as they are discovered. Automatically create tags for Broker Clients if they do not exist.

2008 OSIsoft, Inc. | Company Confidential

11

AF Element Templates
BrokerClientInfo:
Defines the attributes of a BrokerClient as it exists in WebMethods. Each Attribute references a PI Point. Appname = \\%Server%\BrokerClientInfo_%..\..\Element%_%..\El ement%_%Element%_%Attribute%_Info QueueLength = \\%Server%\BrokerClientInfo_%..\..\Element%_%..\El ement%_%Element%_%Attribute%_Count

2008 OSIsoft, Inc. | Company Confidential

12

BrokerClientInfo Template

2008 OSIsoft, Inc. | Company Confidential

13

BrokerClientInfo Element

2008 OSIsoft, Inc. | Company Confidential

14

AF Element Structure
WebMethods
Brokers <BrokerServer>
<Broker> <BrokerClient>
<BrokerClient> <Broker> <BrokerClient> <BrokerClient>

<BrokerServer>
<Broker> <BrokerClient> <BrokerClient>

2008 OSIsoft, Inc. | Company Confidential

15

AF Element Structure

2008 OSIsoft, Inc. | Company Confidential

16

Notification Process - Requirements

Notification is based on the QueueLength attribute of a BrokerClientInfo.
Needs to be customizable for each client. Needs to be disabled for certain clients. Default Warning = 25. Default Alarm = 75.

2008 OSIsoft, Inc. | Company Confidential

17

Notification Process - Template

Attribute Template:

2008 OSIsoft, Inc. | Company Confidential

18

Notification Process Warn Trigger

IF Notification enabled And Warn enabled And QueueLength >= Warn value And QueueLength < Alarm value

2008 OSIsoft, Inc. | Company Confidential

19

Notification Process Alarm Trigger

IF Notification enabled And Alarm enabled And QueueLength >= Alarm value

2008 OSIsoft, Inc. | Company Confidential

20

Delivery Channel
WebService
Sends Target and Trigger value to our EventManagement Web Service. Target = AFElement Path Value = CRITICAL, WARN, OK

Event Management Web Service

Gets additional details from AF using the Target. Creates an Event Management message and sends it to our Event Management server. Closes the previous event using the EventId attribute from the AF element. Sends new EventId back to AF Targets EventId attribute.

2008 OSIsoft, Inc. | Company Confidential

21

Event Management View

BMC Impact Explorer

Transpara

2008 OSIsoft, Inc. | Company Confidential

22