Zenoss Admin Guide

Version 0.22.0
Table of Contents
1 Introduction............................................................................................................... 4
2 Architecture............................................................................................................... 4
3 Key Concepts ............................................................................................................ 5
3.1 Path Navigation .................................................................................................. 5
3.2 Classification...................................................................................................... 5
3.3 zProperties.......................................................................................................... 5
4 Quick Start ................................................................................................................ 5
4.1 Testing SNMP .................................................................................................... 5
4.2 Adding Devices with SNMP............................................................................... 5
4.2.1 Adding remote Windows boxes ................................................................... 5
4.2.2 Adding remote Linux boxes ......................................................................... 6
4.2.3 Adding Cisco devices................................................................................... 6
4.3 No SNMP........................................................................................................... 6
4.3.1 With ssh....................................................................................................... 6
4.3.2 With portscan............................................................................................... 6
5 Device Management .................................................................................................. 7
5.1 Adding a Device................................................................................................. 7
5.2 Editing a Device ................................................................................................. 7
5.3 Searching for a Device........................................................................................ 7
5.4 Process Monitoring............................................................................................. 7
5.5 File System Monitoring ...................................................................................... 7
5.6 Windows Monitoring.......................................................................................... 8
5.7 Zenmodeler ........................................................................................................ 8
5.8 Production State ................................................................................................. 8
5.9 Maintenance Windows ....................................................................................... 8
5.10 Devices Tabs .................................................................................................... 8
5.10.1 Status ......................................................................................................... 8
5.10.2 OS ............................................................................................................. 9
5.10.3 Hardware ................................................................................................... 9
5.10.4 Software..................................................................................................... 9
5.10.5 Events........................................................................................................ 9
5.10.6 History....................................................................................................... 9
5.10.7 Perf ............................................................................................................ 9
5.10.8 PerfConf .................................................................................................... 9
5.10.9 Edit ............................................................................................................ 9
5.10.10 Manage .................................................................................................... 9
5.10.11 Custom .................................................................................................... 9
5.10.12 zProperties ............................................................................................. 10
5.10.13 Changes ................................................................................................. 10
5.11 Custom Schema .............................................................................................. 10
6 Event Management.................................................................................................. 10
6.1 Dashboard ........................................................................................................ 10

2 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
 6.1.1 Navigational Bar ........................................................................................ 10
6.1.2 Systems-Level Event Summary.................................................................. 11
6.1.3 Devices with Events................................................................................... 11
6.1.4 Infrastructure Issues ................................................................................... 11
6.2 Event Console .................................................................................................. 11
6.3 Event Concepts................................................................................................. 11
6.3.1 Event Life Cycle ........................................................................................ 11
6.3.2 De-duplication ........................................................................................... 11
6.3.3 Begin / end correlation ............................................................................... 12
6.3.4 Classification ............................................................................................. 12
6.4 Managing Events .............................................................................................. 12
6.4.1 Event Classes............................................................................................. 12
6.4.2 Event Class Mapping ................................................................................. 12
6.4.3 Applying Event and Device Context .......................................................... 13
6.4.4 SNMP Trap Mapping................................................................................. 13
6.5 Alerting (Events through Email or Pager) ......................................................... 14
6.5.1 User Alerting Rules.................................................................................... 14
7 Performance Management ....................................................................................... 15
7.1 RRDTemplates ................................................................................................. 15
7.1.1 Name Binding Reference ........................................................................... 15
7.2 Other RRDTemplate Objects ............................................................................ 15
8 Availability Monitoring........................................................................................... 16
8.1 Zenping ............................................................................................................ 16
8.2 Zenstatus .......................................................................................................... 16
8.3 Nagios Plug-ins ................................................................................................ 16
9 Modeling Maps ....................................................................................................... 17
10 Zenoss General Admin .......................................................................................... 18
10.1 Startup............................................................................................................ 18
10.2 Zenoss Daemons............................................................................................. 18
10.2.1 Configuring Daemons .............................................................................. 18
10.3 MySQL Event Backend .................................................................................. 18
10.3.1 Saving Live Zenoss events ....................................................................... 18
10.4 RRD Files....................................................................................................... 19
10.5 Maintenance Scripts........................................................................................ 19
10.5.1 Example daily script................................................................................. 19
10.5.2 Example Weekly Script............................................................................ 19
10.5.3 Log Rotate Script ..................................................................................... 19
11 Appendix: Remote MySQL Instance ..................................................................... 20
12 Appendix: Network Auto-Discovery ..................................................................... 20
13 Appendix: ZProperties........................................................................................... 21
13.1 Events............................................................................................................. 21
13.2 Devices........................................................................................................... 21
13.3 Services .......................................................................................................... 23
13.4 Networks ........................................................................................................ 23
13.5 Manufacturers................................................................................................. 23

3 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
1 Introduction
The Zenoss system can be used to access many different types of management
information. All information in Zenoss is accessed via a standard web browser. Behind
the Zenoss interface are several different data management systems. These systems
manage events, performance, availability, and configuration information.

This document will give an overview of the architecture of the Zenoss system and
describe how to perform common management tasks. At the core of the Zenoss system is
the Zope web application development environment. As an administrator it may be
useful at times to have access to the ZopeBook, which describes how to use Zope. This
book can be found at the main Zope web site http://www.zope.org.

2 Architecture
Zenoss is made up of twelve different daemons. They are described below:
1. zeo – the backend object database that stores the configuration model.
2. zope – the web application development environment used to develop the console.
3. zenping – high performance asynchronous testing of ICMP status.
4. zenperfsnmp – high performance asynchronous SNMP performance collection.
5. zenmodeler – high performance automated model population using SNMP, SSH,
and Telnet to collect its information. Zenmodeler works against devices that have
been loaded into the DMD.
6. zendisc – a subclass of zenmodeler that walks the routing table to discover the
network topology and then pings all discovered networks to find active IPs and
devices.
7. zenagios – runs Nagios plug-ins on the local box or on remote boxes through SSH
8. zensyslog – collection of and classification of syslog events.
9. zenstatus – perform active TCP connection testing of remote daemons.
10. zenprocess – process monitoring using SNMP host resources mib.
11. zentrap – receives traps and turns them into events
12. zenxevent – receives events through xml-rpc
Zenoss also has a set of programs that run under windows to perform Event log collection
and windows service monitoring using the native windows management interface (WMI).
These services must be run under windows:
1. zenwinmodeler – auto-discovery of services running on a windows box.
2. zenwin – monitoring up/down availability of windows services
3. zeneventlog – collection of event log events.

4 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
3 Key Concepts
3.1 Path Navigation
Zenoss uses hierarchies much like a file system does to organize information. These
hierarchies are navigated using paths just like a UNIX file system or a web URL. The
paths are navigating objects in a database though not an actual file system.

3.2 Classification
Many of Zenoss’s hierarchies are used to classify IT entities, things like Devices
(computers, routers, switches, etc) or Events (status information sent out by devices).
Once an item is properly classified the system understands more about the item. This
makes proper classification an important activity in the system. Often classification
happens automatically with the ability for manual override later. As Zenoss matures
auto-classification will be come more common.

3.3 zProperties
Zenoss allows configuration to be specified using its hierarchical organization system.
zProperties are properties that control different modules of Zenoss. They can be set at
any level of a hierarchy and values set at lower levels of the hierarchy override those
above. zProperties are described further in Managing zProperties.

4 Quick Start
Once Zenoss is up and running and you have followed the INSTALL.txt, the device
database needs to be populated. Zenoss can perform auto-discovery, devices can be
loaded one at a time through the web UI or you can batch load devices formatted in an
XML file. To be able to model a device Zenoss will need a valid SNMP, SSH, or Telnet
connection to the device. Add the Zenoss server to your device database by clicking
“Add Device” and adding it with its IP.

4.1 Testing SNMP

To test whether or not a device has SNMP run:

snmpwalk -v1 -c communityString gate system

If the command does not time out, your SNMP is working correctly.

4.2 Adding Devices with SNMP

4.2.1 Adding remote Windows boxes

By default Windows may not have any SNMP installed. To install SNMP, go to Start 
Control Panel  Add or Remove Programs  Add/Remove Windows Components.
Check the box for Management and Monitoring tools and install them. Next, you need to

5 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
turn the services on and configure them, so go to Control Panel  Administrative Tools
 Services and start both SNMP Service and SNMP Trap Service. Set the SNMP
Community string in the SNMP Service properties to the community string of your
SNMP. If you would like to monitor WMI, refer to the section on Zenwin.

If you want processor and memory monitoring, install SNMP-Informant, go to

www.snmp-informant.com and download SNMP for Windows.

To collect Windows Event logs or log files from a windows box using syslog you can use
SyslogAgent from http://syslogserver.com/syslogagent.html. Windows event log can
also be monitored using zenwin’s native WMI connection but this requires a second
windows box that runs zenwin.

4.2.2 Adding remote Linux boxes

To configure a Linux machine for monitoring, it must have SNMP installed. To do this,
obtain the package net-snmp. You need to configure your snmp once installed.

4.2.3 Adding Cisco devices

Cisco devices come with SNMP on them. You have to setup the SNMP on them to be in
the same community as the rest of your network. Syslog monitoring information to
come.

4.3 No SNMP

4.3.1 With ssh

If a device has no SNMP and you do not wish to put SNMP on it then you can model it
with ssh. To do this, go to the Add Device page. Set discover to “None” and the
operating system to Linux (must have Linux to do ssh monitoring at this point). Once the
device is added, navigate to the device and view its zProperties tab. Set
zCommandUsername and zCommandPassword to the username and password of the
device. Set zSnmpMonitorIgnore to “True”. Set zCollectorIgnorePlugins to
“snmp|portscan”. Then, set zTransportPreference to “ssh. Next, click the manage tab of
the device and “Collect Configuration”.

4.3.2 With portscan

You can also do a modeling of IP services by doing a portscan. To model the device with
a portscan, go to the zProperties tab of a device. Change the zTransportPreference to
“portscan”. Remodel the device by going to the manage tab and clicking “Collect
Configuration”.s

6 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
5 Device Management
5.1 Adding a Device
To add a device, click on Add Device by the management tab. The only fields that are
required to be filled in when creating a device are Device Name and Device Class Path.
All other fields may be filled in later in the “Edit” tab of the specific device.

5.2 Editing a Device

If you wish to edit a device, first locate the device either by using the “Browse By” menu
or by searching for the device in the search box at the top of the application. The device
can then be edited by selecting the “Edit” tab of the device and filling in the appropriate
information.

5.3 Searching for a Device

There are quite a few simple ways to browse for a device. The first and most obvious
way to search for a device is to type in the name or IP of the device in the “Search” field
at the top of the application. However this is not very useful if you cannot remember the
name of the device or are not looking for a specific device. For this reason you can use
the “Browse by” menu section. Browse by allows you to browse by Systems, Groups,
Locations and Reports. Browsing by Systems allows you to browse by some common
types of Devices such as File and Printer and Infrastructure and are arranged
hierarchically. Browsing by Groups allows you to browse by groups that you can set up
to organize your devices. These groups can be named whatever you like and are arranged
hierarchically. Browsing by Locations is basically the same as the others, is just allows
you to arrange your devices into groups based on location and are arranged hierarchically
as well.

5.4 Process Monitoring

Zenoss is able to monitor processes CPU and memory usage. To access this, simply click
on the Processes option on the Dashboard. You can add a process monitor by giving it a
name and specifying the process via a regular expression. Clicking on a specific process
will take you to an interface that shows all instances of that process running across
machines that have it monitored. If the process has multiple instances the Zenoss will
monitor the sum of CPU and memory utilization of all processes as well as the count of
total processes running. However, if the process has only a single instance, CPU
utilization and memory usage are graphed for the single process. To perform process
monitoring the device SNMP agent must have a reasonable HOST-RESOURCES mib.

5.5 File System Monitoring

File system monitoring is only usable if the system has a good HOST-RESOURCES mib.
File systems monitor the amount of blocks used and show total bytes, available bytes,
used bytes and percent used. You can set thresholds to alert when a specific event occurs
such as the disk reaches 90% utilization.

7 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
5.6 Windows Monitoring
To monitor a Windows machine, you will likely want to use the Zenwin component (set
up separately) to monitor WMI. To use Zenwin, please refer to the README.txt located
in the zenwin folder. If you do not want to monitor WMI, adding a Windows machine is
the same as anything else.

All Windows services are by default not monitored. If you would like to monitor a
specific service, it is very simple to turn it on. Navigate to the Windows device and click
the OS tab. Click on the service you wish to monitor and change the value of monitor to
“True”.

5.7 Zenmodeler
Zenmodeler goes through the list of devices known to Zenoss and performs auto-discover
against each devices sub-components (such as interfaces, file systems, processes,
ipservices, etc). By default the system is setup to perform complete remodeling every 6
hours. This may be too often for large deployments. Often this process is run only once
per day from cron as described above.

5.8 Production State

Production State is an important concept in Zenoss. It determines whether or not a device
is monitored and can be used to control several elements of the event system such as
whether or not an event will produce a remote alert (email or page). Typically devices
start off their life in state “Pre-Production.” In this state devices are monitored by default
but no remote alerting occurs and events aren’t shown on the Dashboard. Once a device
is in full “Production” state monitoring is occurring and remote alerts are sent. If service
needs to be performed on a device its state can be set to “Maintenance” to block
temporarily any remote alerts.

5.9 Maintenance Windows

Zenoss maintenance windows allow scheduled production state changes of a device or all
devices in a System, Group, or Location. Maintenance windows are defined on the
Manage tab of these objects. A maintenance window has a start time, duration, repeat
cycle and start and end production state. A typical window changes the production state
to Maintenance at its start and Production at its end.

5.10 Devices Tabs

The following tabs are available when an individual device is selected.

5.10.1 Status
The status tab is the default tab that is shown when you click on a device. It shows much
of the data of the device and its status (SNMP, ping, uptime, etc.). The status page also
shows how the device is organized such as which groups it is under, what kind of system
it is and where the data is being stored.

8 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
5.10.2 OS
The OS tab contains logical OS components such as interface, services, file systems, etc.
It consists of lots of the non-physical elements of the device.

5.10.3 Hardware
The hardware tab gives you information on the devices available/used memory,
available/used swap and information on the CPU(s).

5.10.4 Software
The software tab gives a list of installed software and can be sorted by Manufacturer,
Name or Install Date.

5.10.5 Events
The events tab is very similar to other events menus. Events may be sorted by quite a
few things including, but not limited to, component, eventClass and count. They may
also be filtered by severity, state and a regular expression applied over all the events.
Events may be move, mapped and acknowledged here as well.

5.10.6 History
The history tab shows the events that have finished with the event life cycle by some
means and have been archived in the history.

5.10.7 Perf
The Perf tab has performance graphs of the device you are looking at. It displays Load
Average 5 Min, CPU Utilization, CPU Idle, Free Memory and Free Swap. This can be
changed to update hourly, daily, weekly, monthly or yearly.

5.10.8 PerfConf
PerfConf gives more detailed data on where the performance data is coming from, allows
you to set thresholds and tells you the specifics of the graphs.

5.10.9 Edit
See the Editing a Device section.

5.10.10 Manage
Using the Manage tab will allow you to change the location of the device, remodel the
device, reset the manage IP, reset the SNMP community, rename the device, clear
heartbeats and delete it.

5.10.11 Custom
The Custom tab allows you to set Custom values in the custom fields defined in Custom
Schema. For more on custom schema, see the Custom Schema section.

9 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
5.10.12 zProperties
zProperties are the properties that the device and all devices further down the hierarchical
tree will inherit. For a complete listing of zProperties, see the tables in the larger
zProperties section.

5.10.13 Changes
The Changes tab logs user changes via the Zope interface.

5.11 Custom Schema

Custom schema can be added to devices to allow the user to add in specific data fields for
tracking on the devices. Properties can be added to the device that will be applied over
all devices. These properties must be named in the following convention: cXxxxx where
X is the name of the property. It must start with a lower case c and be followed by an
upper case letter. This can be especially useful if there are my data fields that you wish
to track but are not covered by Zenoss.

6 Event Management
The Zenoss Event Management system can collect events from syslog, Windows event
log, SNMP traps, and XML-RPC. Processing is performed on raw events to integrate
them tightly into the Zenoss model. Specifically, an event is run through a set of rules to
determine its class which can then provide additional information such as event severity
or up-down correlation.

6.1 Dashboard
The Dashboard is the initial menu of Zenoss. The Dashboard shows Systems-level event
summaries, devices that currently have events with severity of at least Error magnitude,
and infrastructure issues along with a navigational bar. There is a search function in
which all or some of a machines name may be typed in to search for it, as well as an IP
address. User settings may be accessed through the “Settings” link in the top right of the
screen. Below the settings link is the time and date that Zenoss was last updated. Every
60 seconds there is an AJAX call that refreshes the data fields and polls for new data. If
the poll fails, it will display “Lost Connection to Zenoss”.

6.1.1 Navigational Bar

The Main Views section of the Navigational Bar consists of the Dashboard, Events,
Devices, Services, Networks, and Manufacturers. Clicking Events will take you to the
Event management page where you can monitor event status, events, history, properties
and changes made to events. The Devices section allows you to manage sub devices and
a summary of events by severity. You can also look at events sorted by severity followed
by device name, history of device events, PerfConfig, zProperties for Devices, and recent
changes made to the Devices.

10 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
6.1.2 Systems-Level Event Summary
The Systems-Level Event Summary provides a business oriented model of the system a
fraction sorted by where the events are occurring and the severity level of it and is
divided into relevant sub-categories to provide organizational ease. The
acknowledged/total events are represented. Clicking on “Systems” in the title of the
module will take you to a listing of System events, which can be sorted and managed
easily. The categories can be customized with ease by selecting the “Browse By:
Systems” and doing whatever you wish with the add, move to, and delete buttons. You
can also navigate the Systems categories by selecting the link to go to its page.

6.1.3 Devices with Events

This section displays the name of the device, acknowledgement status, and amount of
Critical and Error severity level events. “Devices” can be clicked on to take you to the
Devices event log, which is default sorted by severity. Clicking on the specific device
will take you to the device log of that device. The events are shown in fractions of
acknowledged/total. If they are acknowledged it shows the user that acknowledged it.

6.1.4 Infrastructure Issues

Zenoss Infrastructure Issues will show up any problems with a Zenoss daemon. It may
not be running or there may be some other problem if it shows up.

6.2 Event Console

When looking at events, there are quite a few sorting and filtering schemes available.
Events can be sorted by clicking on “device”, “component”, “eventClass”, “summary”,
“firstTime”, “lastTime” and “count”. You can also filter by state and severity or by using
the “Filter” field. “Filter” is a regular expression on all text seen in the event list. Events
can be selected and either Acknowledged, moved to History, or Mapped into a specific
location.

6.3 Event Concepts

6.3.1 Event Life Cycle

The Zenoss Event Life Cycle is a very straightforward process. The first step of the life
cycle is the creation of the event. The default state of the event is set to “New”. The
event can then be Acknowledged, Suppressed or “dropped” with an Event class rule.
From there, an event will be archived into the Event History database in one of four
ways. The event can manually be put into the history database; it can be put into the
database due to auto clear correlation (bad event happens, good event for the same thing
happens, move bad event to history), an event class rule, and an inactive timeout.

6.3.2 De-duplication
If a single event is submitted multiple times for some reason, instead of the event
clogging up the event log with hundreds or perhaps even thousands of events a counter is
incremented.

11 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
6.3.3 Begin / end correlation
An event enters the Event Life Cycle at the start of the event and at the end of the event,
it leaves the life cycle. If a positive event that corresponds to a negative event is
received, the negative event is cleared.

6.3.4 Classification
When an event is found, it is automatically identified and placed in the correct location
and is correctly labeled with the corresponding class. Classes can cause actions to fire
and/or add more information to an event (such as correlations, severity modification, or
custom actions).

6.4 Managing Events

6.4.1 Event Classes

The first integration step performed on a received event is assignment of the event class.
Event classes are added to the system using the hierarchal scheme found throughout
Zenoss. Events are mapped to Event Classes by Event Class instances. Event Class
instances are looked up by a non-unique key called EventClassKey.

6.4.2 Event Class Mapping

For syslog events EventClassKey is most commonly the “Tag” of the syslog event.
Often the syslog tag maps to the process name from which the event came.

Because the EventClassKey is non-unique further matching may need to be performed to

find a unique instance. This matching can be done through mechanisms, regular
expression match or Python expression evaluation. Because there will be list of instances
against which these rules will be evaluated, the order of evaluation is important. On the
event instance screen the “Sequence” tab allows all instances for a particular
EventClassKey to be ordered. When performing regular expression matches on an event
extraction directives can be used to populate attributes of the event. These directives
follow the Python format for named extractions in the form (?P<keyName>\S+). When
creating a regular expression the original event text can be added to the example field and
upon save the regular expression will be tested against this text.

As mentioned above instances can also be matched by specifying a Python expression.

This expression will be evaluated with the variable name “evt” bound to the current
event. An example:
evt.priority>4

If the EventClassKey lookup returns no results a second lookup will be performed using
the key “defaultmapping” . Default mappings can be used to match large ranges of
events by regular expression.

12 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
6.4.3 Applying Event and Device Context
Once an event has been mapped to its class two things happen its Event and Device
contexts are applied. EventClass context application is done by looking up the zProperty
list zEventProperties. Any property names found in zEventProperties are set in the same
way that other zProperties are found, except that when looking up the property the prefix
zEvent_ is added to the property name. When the zEventProperties value is changed
placeholders are created for each of the properties in the list on the zProperties screen.

A common property to update is severity and infact severity is added by default to

zEventProperties. To update the severity of a classified event change the value of
zEvent_severity in the event class path of the event.

After the event context has been applied the device context is applied. During this
process the productionState, location, DeviceClass, DeviceGroups, and Systems, are
added to the event. Once this is done a zEventProperties update is attempted as described
above but using the device class path instead of the event class path. This allows a
particular device or class of devices to override the default values for any given event.

6.4.4 SNMP Trap Mapping

If a trap comes in and is classified as an "/Unknown" event type, this is because Zenoss
does not know what you want to do with this event.

To map this event, first select the checkbox next to the event, select a category next to the
Map button at the bottom of the page. Each category does different things to the event:
changing its severity, moving it to the history table, etc. For now, select "/App" and press
the Map button.

This will take you to the edit screen for a new "Mapper". These are the rules used to map
this event to the "/App" category. This rule, since it matches the Trap by a very specific
OID, is all you need.

In the "transform" section of the mapper, you can put some code to modify the summary.
For example, lets say you want to set the summary string to "spam Filter Detects Virus".
You would put this in the transform edit area:

evt.summary = "spam Filter Detects Virus"

A trap has a header with some standard (and mostly useless) information. But then it has
a sequence of attribute/values You can see these values as event details if you click on
the last column of the event.

You have indicated you want the value for the OID ".1.3.6.1.4.1.9789.1500.2.5" as the
summary. If you had the MIB loaded, you could do this:

evt.summary = evt.spamFilterDetectsVirus

13 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
But... we have the OID and the data is still in there. We just need to use the slightly more
cryptic:

evt.summary = getattr(evt, ".1.3.6.1.4.9789.1500.2.5", "Unexpected missing OID")

The "device" object for the event has been made available, too:

evt.summary = getattr(evt, ".1.3.6.1.4.9789.1500.2.5", "Unexpected missing OID") + "

from device " + device.getId()

6.5 Alerting (Events through Email or Pager)

The daemon zenactions provides functionality for sending emails or pages based on
events received. It continuously evaluates every user’s paging rules against the event
database.

6.5.1 User Alerting Rules

Each user has their own set of alerting rules. These are configured by going to the
settings link in the upper right of the UI. Once there you will see an "Action Rules" tab.
To add an alerting rule enter your rule name in the add field something like "email_alert"
for instance.

By default a rule for new severity 4 and greater production events will be created. If
action is email the event will be emailed. If its page you will need to have an snpp
paging server setup (see externallibs dir of install directory sendpage does this). Lots of
wireless phone systems have SMTP to SMS gateways so you might just use email.

By default email alerts will be sent to the email address defined in the main user settings
tab and pager alerts will go to the pager address. You can override this by filling in the
optional address field.

The delay field is the number of seconds to wait before sending the alert. If an event
clears before delay time no alert is sent. Rules can be modified using the GUI provided.
You can see a list of valid fields in the popup called “Add Filter”.

Email messages can have a user specified subject and body. The first defines the alert to
be sent when a failure is detected. The second defines the clear alert to be sent once the
failure is closed. These fields are python format strings. At the bottom of the page is a
list of the fields available for an alert. A clear events fields are accessed by prefixing the
field name with “clear”. For instance the field prodState becomes clearProdState. There
is also a special field clearOrEventSummary which will print the clear summary or if it
does not exist the original alert summary. This is useful for the subject of a clear alert.
In the case where an alert has no clear (it was deleted for the UI for instance) a meaning
full subject will still be created.

14 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
7 Performance Management
The daemon zenperfsnmp performs SNMP performance collection for the Zenoss system.
It is fully asynchronous and tries to be very efficient in the way in which it polls devices.
Most servers are actually polled using only one UDP packet. Zenperfsnmp can also
process a simple min/max threshold against any data source it polls.

7.1 RRDTemplates
The top level performance configuration object is an RRDTemplate. RRDTemplates
define the data sources to collect, any thresholds and how the data sources should be
graphed. RRDTemplates are defined in the PerfConf tab of any device tree object or on
the collected object itself. Binding of a template to an object is based its name. By
default binding is done by looking for templates with the same name as an objects
meta_type. For instance all devices in the system have the meta_type “Device” so their
RRDTemplate is called Device. Templates are inherited in the same way that zProperties
are so the template closest to the object is its definition. In the PerfConf tab on an actual
object there is a “Local Copy” button this will create a copy of the current template on
the local device, which can then be customized. If the custom copy is no longer
necessary it can be deleted by clicking the “Remove Local” button.

7.1.1 Name Binding Reference

Device – the device object itself (these oids don’t have an snmp index number)
FileSystem – file system object currently uses host resources MIB
Interface – interfaces are bound using their interface type for example (ethernetCsmacd)
HardDisk – hard disk object for I/O stats. Windows boxes with Informat MIB.

7.2 Other RRDTemplate Objects

To begin collecting information on an object an RRDDataSource must be added to the
object. Its most important values are the name, OID and rrdtype. OIDs on a Device
template most often need to end in .0 don’t forget to add it! The rrdtype field is usually
COUNTER or GAUGE and should match the SNMP type of the OID being polled. Line
Type needs to be filled in only if the default is not acceptable. By default the first data
source on a graph is an area the rest lines.

RRDThresholds define thresholds for datasources. A threshold can be applied to

multiple data sources. Thresholds can be min or max (ie less then x send a trap or greater
than y send a trap). The most common thresholds define only a max value. This value
can either be an absolute number like 85 or can be a python expression where the value is
calculated relative to attributes of the object to which the threshold is applied. When
using a python expression the name “here” will be applied to the target object. So to set
an 85% threshold relative to an interfaces’ speed the expression would be:
here.speed*.85/8. We divide by 8 at the end because the speed is stored in units of
bits/second and the data comes into the RRD file as bytes/second (and is scaled when the
graph is generated).

RRDGraphs designate global graph options such as which data sources should be shown
together on a graph, what the y-axis units are, the size of the image created, the width of
15 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved
.
lines in the graph, if the graph should use a log based y-axis scale, if the data should be
stacked or not, if a summary should be generated, and what the min and max y-axis
values should be.

8 Availability Monitoring
The availability monitoring system within Zenoss provides active testing of the IT
Infrastructure. The system currently consists of zenping Zenoss’ layer 3 aware topology
monitoring daemon and zenstatus a TCP status tester.

8.1 Zenping
There isn’t much configuration work to be done setting up zenping. The most important
element of this daemon is that Zenoss has built a compete model of the your routing
system. If there are gaps in Zenoss’ routing model the power of zenpings topology
monitoring will not be available. This issue can be seen in the zenping.log file.

8.2 Zenstatus
Zenstatus performs monitoring of TCP services. It is configured by turning on
monitoring of a service under the “Services” root on the Navigation Toolbar. Service
monitoring can be turned on a service class but this can be overridden on any service
instance. For instance “SMTP” will be monitored by default. But it may not be a critical
service on all boxes. If this is the case it may be removed on specific devices. Also if the
service is configured to only listen on localhost (127.0.0.1) the service will not be
monitored.

8.3 Nagios Plug-ins

Zenoss has the ability to run Nagios plug-ins though the process zenagios. Zenagios can
run plug-ins locally and remotely using a native SSH transport. When run Zenoss will
track the return code of each plugin and create events with the output from the plug-in.
Zenoss will not track performance information from a plug-in.

Nagios plug-ins are configured using a Nagios Template that is much like the
RRDTemplates used for performance monitoring. So a template named “Device” will
bind to all devices below the template definition. Within each template is a list of
commands that will run. The commands can be any program that follows the Nagios
plug-in standard (inputs are command line arguments output is first line of stdout plus a
return code) as defined in http://nagiosplug.sourceforge.net/developer-guidelines.html

A Nagios command has several fields:

• name – name of the command object
• enabled – should this command be used on a give device.
• component – the component name to use when zenagios sends events to Zenoss
• event class – the event class to used when sending events to Zenoss
• severity – the default severity to use when sending events to Zenoss.

16 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
 • cycle time – how often a command should be run in seconds
• command template – the command to run.
The command template string built using Zope TALES. Several variables are passed
when evaluating the template. They are:
• zNagiosPath – the path to Nagios plug-ins on a given box it comes from the
zProperty zenNagiosPath. zNagiosPath will be automatically added to a
command if it is absent form the beginning of the beginning of the command.
• devname – the device name of the device against which the command is being
evaluated.
• dev – the device object against which the command is being evaluated.
• here – the context of evaluation. for a device this == dev for a component such as
a filesystem or interface this is the component object.
• compname – if this command evaluates against a component this is its name as a
string
• now – the current time.
Template values are accessed like shell variables here are some examples:

Run an http check against all devices using the uri /zport/dmd
check_http –H ${devname}-u /zport/dmd

In a template named FileSystem the following command will run against all FileSystems
on a device.
check_disk –w 10% -c 5% -p ${compname}

9 Modeling Maps
Zenoss uses plugin maps to map real world information into the standard Zenoss model.
Input to the plugins can come from SNMP, SSH or Telnet. Selection of which plugins to
run against a device is done by matching the plugin name against the zProperties:
zCollectorCollectPlugins and zCollectorIgnorePlugins. Plugins that match
zCollectorCollectPlugins are collected ones that match zCollectorIgnorePlugins are
ignored.
• DeviceMap – collect basic information about a device such as its OS type and
hardware model.
• InterfaceMap – collect the list of interfaces on a device.
• RouteMap – collect the routing table from the device.
• IpServicesMap – collect the ip services running on the device.
• FileSystemMap – collect the list of filesystems on a device.

17 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
10 Zenoss General Admin
10.1 Startup
Zenoss can be controlled entirely by the bin/zenoss script. To provide automatic startup
on Linux, link zenoss to the /etc/rc.d/init.d directory
ln -s $ZENHOME/bin/zenoss /etc/rc.d/init.d

It is important to either add $ZENHOME to roots environment or to the

zenoss script itself.

10.2 Zenoss Daemons

All Zenoss daemons share some similar commands. They have the following commands:
• run – start daemon but don’t put it into the background. This is good for
debugging.
• start – start as a daemon running in background detached from the shell.
• stop – stop the daemon.
• restart – stop/start the daemon. Sometimes the start command will run before the
daemon has terminated. If this happens just re-run the command.
• status – check the status of a daemon will print out the current process number if
running.
• help – display a list of all options for the daemon.

10.2.1 Configuring Daemons

Any daemon can be configured by adding key/value pairs to a file named
$ZENHOME/etc/DAEMONNAME.conf. Valid keys are the long option of any
command line option. These can be listed by using the daemons “help” command.

10.3 MySQL Event Backend

MySQL should be backed up following the MySQL manual.

10.3.1 Saving Live Zenoss events

Zenoss stores its "live" events in an in memory MySQL table called “status”. To save
this data to disk every 5 minutes and reload do the following.
1. edit zeneventsdump so that it has the proper MYUSER, MYPASS and
MYSQLHOME
2. make MYSQLHOME/backup and set ownership to mysql and chmod 777
3. add zeneventsdump to cron on some cycle interval (e.g. 5 minutes)
4. create the file initfile.sql in MYSQLHOME/backup with the contents shown in
zeneventsdump
5. add this to the start script the mysql_safe add --init-
file=MYSQLHOME/backup/initfile.sql

18 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
10.4 RRD Files
RRD files can be copied to a second drive or stored on a reliable storage mechanism
(RAID). Don’t use RAID-5 as write performance is very important for these files.

10.5 Maintenance Scripts

10.5.1 Example daily script

#!/bin/bash
export ZENHOME=/usr/local/zenoss
export PYTHONPATH=$ZENHOME/lib/python
rm -rf /tmp/renderserver/
cd $ZENHOME
bin/zenmodeler run --logpath $ZENHOME/log -v30 >>
log/zenmodeler.err.log 2>&1
bin/repozo.py -B -f var/Data.fs -r var/backup >> log/repozo.log 2>&1
/usr/local/mysql/bin/mysqldump -u root --password=\!mercy\! events >
var/backup/events.sql

To backup or restore the Zeo database the repozo command is used. See its help page for
more details.

The Zeo database needs to be “packed” periodically to reclaim space. To do this you
should set up a cron job that runs the following command once a day.

10.5.2 Example Weekly Script

This example script will pack the zeo database once a week and cleanup backups more
than two weeks old.

#!/bin/bash
export ZENHOME=/usr/local/zenoss
export PYTHONPATH=$ZENHOME/lib/python
cd $ZENHOME
bin/zeopack.py -p 8100
find $ZENHOME/var/backup -name \*fsz -mtime +14 -exec rm {} \;
find $ZENHOME/var/backup -name \*.dat -mtime +14 -exec rm {} \;

10.5.3 Log Rotate Script

If your system uses logrotate to manage files put the following in /etc/logrotate.d/zenoss
to manage Zenoss’ log files.

/usr/local/zenoss/log/*.log {
weekly
rotate 2
copytruncate
}

19 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
11 Appendix: Remote MySQL Instance
If you are interested in using a remote MySQL database for Zenoss, set it up like normal.
Once install go to
[IPWhereZenossIsInstalled]:8080/zport/dmd/ZenEventManager/manage and click the
Properties tab. Change the database field to the address of the MySQL database. Do the
same to [IPWhereZenossIsInstalled]:8080/zport/dmd/ZenEventHistory/manage.

12 Appendix: Network Auto-Discovery

To perform auto-discovery the machine on which Zenoss is installed must have an SNMP
agent running. Start the auto-discovery engine by running in ZENHOME:
bin/zendisc run --routers

This command will first model the monitoring machine and then walk through the
routing tables of all routers it can find. Auto-discovery will go as far as valid SNMP
access is found or until a network is discovered in the DMD that has its zAutoDiscover
property set to False.

Routers discovered through this process will be placed in the device path
/Network/Router.

Performing full discovery of all devices on the network run zendisc without the --routers
flag. This will need to be done as root so that a raw socket for ICMP pinging can be
created.
bin/zendisc run

When devices are discovered they are placed into the /Discovered device path. They
should then be moved to a more specific part of the tree. Servers are normally organized
by OS so windows machines might go to /Server/Windows. Other information can be
added to a device, like its Business System or its Location, using the Edit tab on a device.

Discovery can also be run on an existing network object.

bin/zendisc run --net 10.2.1.0

This command will ping all devices on the 10.2.1.0 network and the attempt to perform
SNMP discovery on them. 10.2.1.0 must exist in the Networks root of the DMD. To
add a network to the system enter its IP in CIDR format i.e. 10.2.1.0/24 for the class C
network in the add field at the bottom of the networks screen.

20 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.
13 Appendix: ZProperties

13.1 Events

Property
Property Name Type Description
Location to which an event will be stored.
Possible values are: status, history and drop.
Default is status meaning the event will be an
“active” event. History sends the evnet directly to
the history table. Drop tells the system to discard
zEventAction string the event.
A list of classes that a clear event should clear in
zEventClearClass lines addition to its own class.
Allows you to override the severity value of an
event. If this is -1 it is ignored. Possible values
zEventSeverity int are 0 – 5.

13.2 Devices

Property
Property Name Type Description

Allows you to set the timeout time of the

zCollectorClientTimeout int collector client in seconds

Use to control which collector plug-ins will be

loaded for a particular device. This value is a
regular expression that will be matched against
ZCollectorCollectMaps string the plug-in names that will be collected.
zCollectorCollectPlugins string Model plugins to collect (regex).

Use to control which collector plug-ins will be

loaded for a particular device. This value is a
regular expression that will be matched against
zCollectorIgnoreMaps string the plug-in names that will not be collected.

zCollectorIgnorePlugins string Model plugins not to collect (regex).

zCommandCommandTimeout float Time to wait for a command to complete.

zCommandExistanceTest string ***
zCommandLoginTimeout float Time to wait for a login prompt.
zCommandLoginTries int Number of times to attempt login.
Password to use when performing command
zCommandPassword string login.
Port to connect to when performing command
zCommandPort int collection.

21 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.

zCommandProtocol
zCommandSearchPath
zCommandUsername
zFileSystemMapIgnoreNames
zFileSystemMapIgnoreTypes
zIfDescription
zIpServiceMapMaxPort
zNagiosPath
zPingInterfaceDescription
zPingInterfaceName
zPingMonitorIgnore
zProdStateThreshold
zPythonClass
zRouteMapCollectOnlyIndirect
zRouteMapCollectOnlyLocal
zSnmpCommunities
zSnmpCommunity
zSnmpMonitorIgnore
zSnmpPort
zSnmpTimeout
zSnmpTries
zSnmpVer
zTelnetEnable
zTelnetEnableRegex
zTelnetLoginRegex
22 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved
Protocol to use when performing command
string collection. Possible values are ssh, telnet.
lines ***Places to search for the commands?***
Username to use when performing command
string collection.
Regular expression of file system names to
string ignore.
lines DO NOT USE
Show the interface description field in the
boolean interface list.
int Highest port to scan, default to 1024.
Path where Nagios plug-ins are installed on the
local Zenoss box or a remote box in the case
String where SSH is used to run the command.
A catalog query string that will find interfaces to
string be pinged by description.
A catalog query string that will find interfaces to
string be pinged by name.
boolean Whether or not to ping the device.
Production state threshold at which Zenoss will
begin to monitor a device. Default of 500
int equals Pre-Productions.
string DO NOT USE
Only collect routes that are directly connected
boolean to the device.
Only collect local routes (these are usually
manually configured vs. those learned through
boolean a routing protocol.)
Array of SNMP community strings that the
ZenModeler will try to use when collecting
lines SNMP information.
Community to be used when collecting SNMP
information. If it is different than what is found
by ZenModeler it will be set on the modeled
string device.
Whether or not to ignore monitoring SNMP on a
boolean device.
int Port that the SNMP agent listens on.
float Timeout time in seconds for an SNMP request
int Amount of tries to collect snmp data
string SNMP version used. Valid values are v1, v2.
When logging into a Cisco device issue the
enable command to enable access during
boolean command collection.
Regular expression to match the enable
string prompt.
string Regular expression to match the login prompt.
.
 Regular expression to match the password
zTelnetLPasswordRegex string prompt.
List of regular expressions to match the
zTelnetSuccessRegexList lines command prompt.

zTelnetTermLength boolean On a Cisco device set term length to Zero.

Preferred protocol for model collection.
Possible values are snmp, command and
portscan. Used to choose the collector plug-in
when more than one plug-in of a given type is
zTransportPreference string found.
zWinEventlog boolean Whether or not to send the log.
The password used to remotely login if it is a
zWinPassword string Windows machine.
The user name used to remotely login if it is a
zWinUser string Windows machine.

13.3 Services
Property
Property Name Type Description
Determines what severity to send for the specified
zFailSeverity int service.
zHideFieldsFromList lines Fields to hide from Services instance list
zMonitor boolean Tells whether or not to monitor a service.

13.4 Networks
Property
Property Name Type Description
Should zendisc perform auto-discovery on this
zAutoDiscover boolean network
List of netmask numbers to use when creating
network containers. Default is 24, 32 which will
make /24 networks at the top level of the networks
zDefaultNetworkTree lines tree if a network us smaller than /24.

13.5 Manufacturers
Property Name Property Type Description
zDeviceClass string FUTURE USE
zDeviceGroup string FUTURE USE
zSystem string FUTURE USE

23 of 23 Copyright © 2002 Zenoss, Inc. All Rights Reserved

.