You are on page 1of 112

Technical Best

Practices
Oracle Utilities Application Framework (4.3.0.6.0)

WHITE PAPER / JUNE 2018


DISCLAIMER
The following is intended to outline our general product direction. It is intended for information
purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any
material, code, or functionality, and should not be relied upon in making purchasing decisions. The
development, release, and timing of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.

2 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Table of Contents
Introduction ................................................................................................ 10

Caveat ........................................................................................................................................10

Conventions Used In This Whitepaper .......................................................................................10

Oracle Utilities Application Framework Overview ...................................... 10

Architecture Changes ................................................................................ 12

Installation Best Practices .......................................................................... 13

Read the Installation Guide ........................................................................................................13

Ensure the prerequisites are installed ........................................................................................13

Environment Practices ............................................................................... 13

Using multiple administrators......................................................................................................14

Using Oracle WebLogic Domain Templates ...............................................................................15

General Best Practices .............................................................................. 15

Limiting production Access .........................................................................................................15

Regular Collection and Reporting of Performance Metrics .........................................................16

Respecting Record Ownership ...................................................................................................16

Backup of Logs ...........................................................................................................................17

Post Process Logs......................................................................................................................17

Check Logs For Errors ...............................................................................................................18

3 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Optimize Operating System Settings ..........................................................................................18

Optimize connection pools .........................................................................................................19

Read the available manuals .......................................................................................................20

Technical Documentation Set Whitepapers available.................................................................21

Secure default userids ................................................................................................................24

Consider different userids for different modes of access ............................................................24

Don’t double audit.......................................................................................................................25

Use Identical Machines ..............................................................................................................25

Regularly restart machines .........................................................................................................25

Avoid using direct SQL to manipulate data .................................................................................26

Minimize Portal Zones not used .................................................................................................26

Routine Tasks for Operations .....................................................................................................26

Typical Business Day .................................................................................................................27

Login Id versus Userid ................................................................................................................28

Hardware Architecture Best Practices ........................................................................................29

High Availability Architecture ......................................................................................................32

Failover Best Practices ...............................................................................................................33

General Troubleshooting Techniques .........................................................................................34

Technical vs User Logs ..............................................................................................................36

Overriding System Date .............................................................................................................37

4 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Transaction Timeouts .................................................................................................................38

Java Garbage Collection Guidelines ..........................................................................................42

Security Configuration ................................................................................................................43

Shortcuts When Processing Templates ......................................................................................43

Accessing JMX for Oracle WebLogic .........................................................................................44

UCP JMX Interface .....................................................................................................................45

Using Java Mission Control with Oracle Utilities Application Framework ...................................46

Overload Protection ....................................................................................................................47

Resource Management ..............................................................................................................48

Data Management Best Practices ............................................................. 48

Respecting Data Diversity ..........................................................................................................49

Removal of Staging Records ......................................................................................................49

Partitioning..................................................................................................................................50

Compression ..............................................................................................................................52

Database Clustering ...................................................................................................................52

Backup and Recovery ................................................................................................................53

Client Computer Best Practices ................................................................. 53

Make sure the machine meets at least the minimum specification .............................................53

Internet Explorer Compatibility Mode Settings ............................................................................54

Popup Blockers ..........................................................................................................................54

5 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Internet Explorer Caching Settings .............................................................................................54

Clearing Internet Explorer Cache ...............................................................................................55

Optimal Network Card Settings ..................................................................................................55

Network Best Practices.............................................................................. 55

Network bandwidth .....................................................................................................................55

Ensure legitimate Network Traffic ...............................................................................................56

Regularly check network latency ................................................................................................56

General Networking Guidelines ..................................................................................................57

Web Application Server Best Practices ..................................................... 58

Make sure that the access.log is being created ..........................................................................58

Examine Memory Footprint.........................................................................................................58

Turn off Debug............................................................................................................................59

Load balancers ...........................................................................................................................59

Preload or Not? ..........................................................................................................................60

Hardware or software proxy........................................................................................................61

What is the number of Web Application instances do I need? ....................................................62

Determining Active Users ...........................................................................................................62

Defining external LDAP to the Web Application Server ..............................................................64

Appropriate use of AppViewer ....................................................................................................64

Fine Grained JVM Options .........................................................................................................65

6 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Customizing the server context ..................................................................................................67

Clustering or Managed? .............................................................................................................67

Monitoring and Managing the Web Application Server using JMX .............................................69

Password Management solution for Oracle WebLogic ...............................................................70

Error configuring Oracle WebLogic credentials ..........................................................................71

Corrupted SPLApp.war ...............................................................................................................71

Web Application Server Logs .....................................................................................................72

Enabling additional Java options ................................................................................................73

Using setUserOverrides.sh for Oracle WebLogic 12c ................................................................73

Native vs Embedded Oracle WebLogic Mode ............................................................................74

CLIENT-CERT Support ..............................................................................................................76

Implementing Work Managers ....................................................................................................77

Implementing the Custom Authentication Security Provider .......................................................78

Business Application Server Best Practices .............................................. 79

Cache Management ...................................................................................................................79

Using JMX with the Business Application Server .......................................................................79

Replicating the txrpt statistics .....................................................................................................80

Database Connection Management ...........................................................................................81

XPath Memory Management ......................................................................................................82

Enabling Service Timings ...........................................................................................................83

7 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Oracle WebLogic Datasource Support .......................................................................................83

Database Best Practices ........................................................................... 87

Regularly Calculate Database Statistics .....................................................................................87

Ensure I/O is spread evenly across available devices ................................................................88

Use the Correct NLS settings (Oracle) .......................................................................................89

Monitoring database connections ...............................................................................................89

Building the Data Model .............................................................................................................91

Using Database Compression ....................................................................................................96

Configuration Migration Assistant Best Practices ...................................... 97

Optimizing Locations for Migrations ............................................................................................97

Daisy Chaining Jobs ...................................................................................................................98

Use Auto-approve to Streamline Migrations ...............................................................................99

Store Exports in Code Repositories..........................................................................................100

Decide your Promotion Model ..................................................................................................100

Optimization Techniques ......................................................................... 100

Consider Standard Images .......................................................................................................100

Architecture Optimizations ........................................................................................................102

Optimize Monitoring..................................................................................................................104

Reduce Queries using Views....................................................................................................105

Use CMA for all your migrations ...............................................................................................105

8 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Consider Automation ................................................................................................................107

Preparing Your Implementation For the Cloud ........................................ 107

Move extensions to the database .............................................................................................107

Simplify your integrations .........................................................................................................109

Remove operational extensions ...............................................................................................109

Rationalize your environments .................................................................................................110

Think differently for extensions .................................................................................................110

9 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
INTRODUCTION
Oracle Utilities Application
Implementation of the product at any site introduces new practices into the IT group to maintain the Framework is a common design
health of the system and provide the expected service levels demanded by the business. While and runtime framework across
configuration of the product is important to the success of the implementation (and subsequence most of the Oracle Utilities
maintenance), adopting new practices can help ensure that the system will operate within acceptable product line. As this is standard,
common best practices have
tolerances and support the business goals.
emerged. This whitepaper
outlines the most common
This white paper outlines the common and best practices used by IT groups at sites using Oracle
practices.
Utilities Application Framework based products and Oracle internal studies, around the world, that
have benefited sites in some positive way. The recommendations in this document are based upon
experiences from various sites and internal studies, which have benefited from implementing the
practices outlined in the document

Note: For publishing purposes, the word product will be used to indicate all Oracle Utilities Application
Framework based products.

Note: This whitepaper has been updated to include advice for all versions of Oracle Utilities
Application Framework V4.x. This document does not include any advice for Oracle Utilities
Application Framework V2.x.

Note: The term Oracle Utilities Cloud refers to the Oracle Utilities SaaS offerings only. Oracle Utilities
products on Platform As A Service and Infrastructure As A Service can use the same techniques as
1
the SaaS or on-premise implementations .

Note: Advice in this document is primarily applicable to the latest version of the Oracle Utilities
Application Framework at time of publication. This advice may apply to other versions of the Oracle
Utilities Application Framework and may be applied at site discretion.

Caveat
While all care has been taken in providing this information, implementation of the practices outlined in
this document may NOT guarantee the same level of (or any) improvement. Not all practices outlined
in this document will be appropriate for your site. It is recommended that each practice be examined in
light of your particular organizational policies and use of the product. If the practice is deemed
beneficial to your site, then consider implementing it. If the practice is not appropriate (e.g. for cost and
other reasons), then it should not be considered.

Conventions Used In This Whitepaper


The advice in this document applies to any product based upon Oracle Utilities Application Framework
versions 4.0 and above. Refer to the installation documentation to verify which version of the
framework applies to your version of the product. For publishing purposes the specific facilities and
instructions for specific framework versions will be indicated with the appropriate footnotes.

ORACLE UTILITIES APPLICATION FRAMEW ORK OVERVIEW


The Oracle Utilities Application Framework is a reusable, scalable and flexible java based framework
which allows other products to be built, configured and implemented in a standard way.

1
Including any cloud service not operated by Oracle. Refer to the conditions related to that individual cloud service for any additional
advice or conditions.

10 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
When Oracle Utilities Customer Care & Billing was migrated from V1 to V2, it was decided that the
technical aspects of that product be separated to allow for reuse and independence from technical
issues. The idea was that all the technical aspects would be concentrated in this separate product (i.e.
a framework) and allow all products using the framework to concentrate on delivering superior
functionality. The product was named the Oracle Utilities Application Framework (oufw is the product
code). The framework was then used across existing and new products to support a common
technology platform across Oracle Utilities.

The technical components are contained in the Oracle Utilities Application Framework which can be
summarized as follows:

 Metadata – The Oracle Utilities Application Framework is responsible for defining and using the
metadata to define the runtime behavior of the product. All the metadata definition and management
is contained within the Oracle Utilities Application Framework.

 UI Management – The Oracle Utilities Application Framework is responsible for defining and
rendering the pages and responsible for ensuring the pages are in the appropriate format for the
locale.

 Integration – The Oracle Utilities Application Framework is responsible for providing the integration
points to the architecture. Refer to the Oracle Utilities Application Framework Integration Overview
(Doc Id: 789060.1) whitepaper available from My Oracle Support for more details.

 Tools – The Oracle Utilities Application Framework provides a common set of facilities and tools
that can be used across all products.

 Technology – The Oracle Utilities Application Framework is responsible for all technology
standards compliance, platform support and integration

The figure below summarizes some of the facilities that the Oracle Utilities Application Framework
provides:

Oracle Utilities Application Framework Features

OBJECTS UX INTEGRATION SECURITY TECH

Meta Data Driven Cross Browser JAX-WS/JAX-RS SSL/TLS Oracle Database


Support

Object Based Device Support SOAP Authorization Oracle WebLogic

Schema Based UI Management JMX Authentication Java


Identifier

Configuration rather Personalization XML/JSON SSO Support HTML5/JS


than code

Common Objects UI Builder Staging WS-* Standards Platforms

Common Services Saved Views Command Utilities Database Vault Groovy

ConfigTools Interactive Query Oracle Enterprise Audit Vault XPath/Xquery


Manager support

Localization Support Multi-lingual SOA Integration Identity Management Resource Control


Suite

11 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Debug Support Wizards (BPA) Service Bus Adapters LDAP Integration Traceability

Technology Federation
Integration

Real Time Integration Whitelisting

There are a number of products from the Utilities Global Business Unit are built upon the Oracle
Utilities Application Framework. These products require the Oracle Utilities Application Framework to
be installed first and then the product itself installed onto the framework to complete the installation
process.

There are a number of key benefits that the Oracle Utilities Application Framework provides to these
products:

 Common facilities – The Oracle Utilities Application Framework provides a standard set of
technical facilities that mean that products can concentrate in the unique aspects of their markets
rather than making technical decisions.

 Common methods of configuration – The Oracle Utilities Application Framework standardizes the
technical configuration process for a product. Customers can effectively reuse the configuration
process across products.

 Common methods of implementation - The Oracle Utilities Application Framework standardizes


the technical aspects of a product implementation. Customers can effectively reuse the technical
implementation process across products.

 Quicker adoption of new technologies – As new technologies and standards are identified as
being important for the product line, they can be integrated centrally benefiting multiple products.

 Multi-lingual and Multi-platform - The Oracle Utilities Application Framework allows the products
to be offered in more markets and across multiple platforms for maximized flexibility.

 Cross product reuse – As enhancements to the Oracle Utilities Application Framework are
identified by a particular product, all products can potentially benefit from the enhancement.

Note: Use of the Oracle Utilities Application Framework does not preclude the introduction of product
specific technologies or facilities to satisfy markets. The framework minimizes the need and assists in
the quick integration of a new product specific piece of technology (if necessary). One of the goals of the Oracle
Utilities Application Framework is
to provide up to date technology.
This means making architecture
ARCHITECTURE CHANGES changes to maximize flexibility.

Over the last few releases of the Oracle Utilities Application Framework the architecture has been
optimized to take advantage of the latest technological advances, provide flexibility and support
varying deployment models. The architectural changes over the last few releases include:

 IWS replaces XAI – The Inbound Web Services capability which houses web services in the Oracle
WebLogic domain rather than within the product itself, which was the architecture of XML
Application Integration (XAI). This allows greater flexibility in configuration and support for advanced
security using Oracle Web Services Manager. For more information refer to Migrating from XAI to
IWS (Doc Id: 1644914.1) available from My Oracle Support.

 OSB replaces MPL – The Multi-purpose Listener capability was a limited service bus which has

12 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
since been replaced with an adapter to Oracle Service Bus. This allows greater implementation
flexibility, increased performance and a wide range of source and target technology
implementations. For more information refer to Oracle Service Bus Integration (Doc Id: 1558279.1)
available from My Oracle Support.

 Simpler Architecture – In past releases, it was possible to separate the Web Application Server
and Business Application Server. This technique was not popular with implementations due to the
increased configuration complexity and complications when patching. Based upon feedback from
customers and partners the Web Application Server and Business Application Server have been
architecturally combined to support a more streamlined and less costly architecture emphasizing
different channels representing different users or modes of access to the product.

 Channel Based Architecture – Over the last few releases, the architecture has been moving
towards a channel based architecture where the channels into the product are able to be separated
from a configuration and architectural point of view. This allows each channel to be segregated,
managed, monitored and optimized for the traffic within that channel. At the moment, online, web
services and batch are on separate channels. More channels will be added in progressive releases.

 Native Based Installation – Over the last few releases, the architecture has been standardized to
use more of the Oracle WebLogic stack, natively, to improve license value, reduce costs and allow
for flexible architectures. This has meant that the original alternative embedded mode has been
Installation is one of the first steps
removed to encourage the use of this new capability. This also allows for support for Oracle's
in any implementation. There are
Maximum Availability Architecture. techniques available to help
minimize the effort involved.

INSTALLATION BEST PRACTICES


During the initial phases of an implementation, a copy of the product will need to be installed. During
the implementation a number of copies of additional copies will be installed, including production. This
section outlines some practices that customers have used to make this process smooth.

Read the Installation Guide


One of the most important pieces of advice in this document to implement is to read the installation
guide that is supplied with the product. It provides valuable information about what needs to be
installed and configured as well the order of the installation. Failure to follow the instructions can cause
unnecessary delays to the installation.

If you are upgrading to a new version, read the new installation guide as well as it will contain
instructions on how to upgrade to the new version as well as details of what has been changed in the
new version.

Ensure the prerequisites are installed


When installing there is a number of third parties prerequisite software that must be obtained (i.e.
downloaded) prior to the actual installation of product software can commence. Read the Installation
Guide and Quick Installation Guide to download and install the prerequisite software prior to installing
product.
A copy of the Oracle Utilities
Note: For customers who are upgrading, the installation of product and its related third party software
product is an environment. Using
is designed so that more than one version of product can co-exist. established Environment
Management principles can
reduce implementation costs.
ENVIRONMENT PRACTICES

Note: There is a more detailed discussion of effective Environment Management in the Environment

13 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Management document of the Software Configuration Management Series (Doc Id: 560401.1)
whitepapers available from My Oracle Support. Refer to that document for further advice.

When installing product at a site, each copy of product is regarded as an environment to perform a
particular task or group of tasks. Typically, without planning this can lead to a larger than anticipated
number of environments. This can have a possible negative flow on effect by increasing overall
maintenance effort and increasing resource usage (hardware and people), which may in turn cause
delays in implementations. Customers to minimize the impact of environments on their
implementations have used the following advice:

 At the start of the implementation decide the number of environments to use. Keep this to a
minimum and consider sharing environments between tasks. Another technique associated with this
is to specify an end date for each environment. This is the date the environment can be removed .
from the implementation. This can force rethinks on the number of environments that are to be used
at an implementation and may force sharing.

 Consider all the resources necessary for the environment. For each environment, consider the
impact on the hardware and maintenance effort including the following:

– The time and resources it takes to install the environment.

– The time and resources it takes to keep the environment up to date including application of single
fixes, rollups/service packs and upgrades. Do not forget application and management of
customization builds.

– The time and resources to maintain the configuration migration and information lifecycle
management facilities for multiple environments, if used at an implementation. This includes the
setup and regular migrations that will be performed.

– The time and resources it takes to backup and restore environments on a regular basis. In some
implementations, having different backup schemes for environments based upon tasks and
update frequency for that environment, i.e. more updated = more frequent backup, may provide
some savings.

– The time and resources to manage the disk space for each environment including regular
cleanups.

 Consider using Pluggable Databases or multiple schemas. Environments may be setup so that
the database can be reduced to a single database instance with each environment having a
different schema/owner. This will reduce the memory footprint of the DBMS on the machine but may
reduce availability of the database instance is shut down (all environments are affected). For non-
production, most customers create a database instance for each environment or use pluggable
databases in Oracle 12c and above.

Using multiple administrators


By default, when installing product a single administrator account (usually referred to as splsys) is
used to install and own the product. This is the default behavior of the installation and apart from
specifying a different userid than the default splsys, it is possible to use other userids to own all or
individual environments.

For example, if the conversion team wishes to have the ability to start, stop and monitor their own
environments, you can create another administrator account and install their copies of product using
that userid. This allows the conversion team to control their own environments. If you did not have the
ability to use multiple administrators than they may have access to all environments (as you would

14 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
have to give them access to the splsys account).

One of the advantages of this approach is that you can delegate management of a copy product to
other teams without compromising other environments. Another advantage is that you can quickly
identify UNIX resource ownership by user rather than trying using other methods.

The only disadvantage is that to manage all copies of product you will need to logon to the additional
administration accounts that own the various copies.

Using Oracle WebLogic Domain Templates


Note: In Oracle Utilities Application Framework 4.3.0.6.0 and above, the product specific domain
templates have been replaced with the base Oracle WebLogic templates supplied with Oracle
WebLogic.

In Oracle Utilities Application Framework 4.3.x and above, a number of Oracle WebLogic domain
templates have been provided to simplify the creation of the Oracle WebLogic domain for the
installation. These templates can be used with the Domain Creation wizard supplied with Oracle
WebLogic. The templates are located in the $SPLEBASE/tools/domaintempates. The following
templates are supplied:

Product Specific Domain Templates

TEMPLATE USAGE

Simple A simple template with a single server housing administration and the product.
This template is suitable to non-production environments.

Complex A more complex template with a simple cluster for the product and separate
administration server. This template is suitable for an initial setup for a
production system. Once established it is recommended to use the Oracle
WebLogic console to extend the domain according to your individual
requirements.

The name of the domain templates adhere to the following naming convention:

Oracle-Utilities-<template>-Linux-<version>.jar

Where:

<template> Domain Template type (Simple or Complex)

<version> The version of Oracle WebLogic the template is optimized for. Ensure Implementations of Oracle Utilities
the correct version is used to ensure optimization. Application Framework based
product all over the world use
The use of domain templates simplifies the native installation process as outlined in the Installation various techniques to minimize
Guide supplied with the product. costs and maximize flexibility.

GENERAL BEST PRACTICES


This section outlines some general practices that have been successfully implemented at various
product sites.

Limiting production Access


One of the guiding principles at successful sites is that production access is restricted to the
processing necessary to run their business. This means that other non-mainstream work, such as ad-

15 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
hoc queries, are either very limited or NOT performed on production at all. This may sound logical but
a few sites have allowed access to production from inappropriate sources, which has had an adverse
impact on performance.

For example, it is not appropriate to allow people access to the production database through ad-hoc
query tools (i.e. such as SQL Developer, SQL*Plus etc). The freestyle nature of these tools can allow
a single user to wreak havoc on performance with a single inefficient SQL statement.

The database is not optimized for such unexpected traffic. Removal of this potentially inefficient
access can typically, improve performance.

Regular Collection and Reporting of Performance Metrics


One of the major practices that successful customers perform is the regular collection of performance
statistics, analysis of the statistics and reporting pertinent information to relevant parties within the
organization as well as Oracle. Collection of such information can help identify bottlenecks and badly
performing transactions, as well as help understand how the product is being used at your site. They
offer proof of both good and bad performance and typically allow sites to gauge the extent of any
issue.

The product contains a number of collection points in the architecture that are useful for real time and
offline collection of performance related data. Information on the collection points are documented in
the Performance Troubleshooting Guideline Series (Doc Id: 560382.1) whitepapers available from My
Oracle Support. Using the guide, decide which statistics are important to the various stakeholders at
your site, decide the frequency of collection and format of any output to be provided. Use your sites
Service Level Agreement (SLA), if it exists, for guidance on what to report.

Respecting Record Ownership


In Oracle Utilities Application Framework, the concept of ownership of records is used on each row in
the database, for configuration tables. A data element was added to data to indicate the owner of the
object and is used to protect key data supplied with the product from alteration or deletion. It is used
by the online system to prevent the online users accidentally causing critical data failures. The owner
is also used by the upgrade tools protect the data from deletion.

The ownership of the record determines what you can do with that record as outlined in the following
table.

Common Record Ownership

OWNER COMMENTS

Framework If the record is owned by Framework then implementation teams cannot alter
or delete the record from the database as it is deemed critical to the running
of the Framework. This is usually meta-data deemed important by the
Framework team. For example the user SYSUSER is owned by the Framework

Product If the record is owned by the product (denoted by the product name or Base)
then some changes are permitted but deletion is not permitted as the record
as it is necessary for the operation of the product. The amount of change will
vary according to the object definition.

Customer Modification If the record is owned by Customer Modification then the implementation has
added the record. The implementation can change and delete the record (if it
is allowed by the business rules).

16 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Basically you can only delete records that are owned by Customer Modification. All other records are
maintained by various utilities supplied with the product as part of upgrade and patch deployments.

Note: Whilst, it is possible to alter or delete the records at the database level, if permitted by database
permissions, doing this may produce unexpected results when applying patches or during execution of
processes, so respect the ownership of the records.

Backup of Logs
By default product removes existing log files from $SPLSYSTEMLOGS (or %SPLSYSTEMLOGS% for
Windows platforms) upon restart. This is the default behavior of the product but may not be desirable
for effective analysis as the logs disappear.

To override this behavior the following needs to be done:

 A directory needs to be created to house the log files. Most sites create a common directory for all
environments on a machine. The size allocation of that directory will depend on how long you wish
to retain the log files. It is generally recommended that logs be retained for post analysis and then
archived (according to site standards) after processing to keep this directory relevant. Typically
customers create a subdirectory under $SPLOUTPUT (or %SPLOUTPUT% for Windows platforms) to
hold the files.

 Set the SPLBCKLOGDIR environment variable in the .profile (for all environments) or
$SPLEBASE/scripts/cmenv.sh (for individual environments) to the location you specified in the
first step. For Windows platforms then the environment can be set in your Windows profile or using
%SPLEBASE%/scripts/cmenv.cmd.

 Logs will be backed up at the location specified in the format


<datetime>.<environment>.<filename> where <datetime> is the date and time of the
restart, <environment> is the id of the environment (taken from the SPLENVIRON environment
variable) and <filename> is the original filename of the log.

Once the logs have been saved you must use log retention principles to manage the logs under
SPLBCKLOGDIR to meet your sites standards. Most sites archive the logs to tape or simply compress
them after post processing the log files.

Post Process Logs


The logs written by the various components of product provide valuable performance and diagnostic
information. Some sites have designed and developed methods to post process those logs to extract
important information and then report on it to relevant parties.

If the logs are retained by your site, the consider post processing the logs on a regular basis before
they are archived or deleted permanently. One approach is to extract that information from the logs
and loading the extracted data into some analysis repository or log analysis cloud services for regular
and trend reporting. The diagram below illustrates the process:

17 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 1. Post Processing Logs

Details of the logs written by the product are documented in the Server Administration Guides shipped
with the product. Use these guides to determine what data to extract from the logs for post processing.

Note: Customers using the Application Management Pack for Oracle Utilities can also use the log
viewing and filtering capability within Oracle Enterprise Manager to query log entries.

Check Logs For Errors


One of the most important tasks for a site is to regularly track errors output into logs. Whenever an
error occurs in product, an error record is written to the appropriate log for analysis. Some sites
regularly check these logs for these errors and using the information in the log, address the error
condition.

Figure 2. Filtering Logs

Viewing and checking for errors on a regular basis to quickly reduce the amount of error that may
occur can detect trends and common problems. The Performance Troubleshooting Guideline Series
(Doc Id: 560382.1) whitepapers available from My Oracle Support outlines the logs and error
conditions contained within those logs.

Optimize Operating System Settings


Note: The Oracle Utilities Application Framework generally does not have any specific operating
system settings other than the ones recommended for Oracle WebLogic and Oracle Database. Refer
to the installation guides for those products for guidance.

18 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
One of the most important configuration settings for product is the operating system itself. The
Installation Guide provided with your product highlights the fact that the operating system parameters
must be set to optimal values for the product to perform optimally. Some sites have experienced large
improvements in performance by heeding this advice. Sites that have decided to ignore this advice
have experience bad performance till the settings were corrected.

Typically, the optimization of the operating system is performed during the implementation and uses
the following principles:

 The value of an individual operating system setting is the maximum value of any product on that
machine. For example, typically if Oracle is installed on the same machine, the values for those
products are used. The settings used in this way are usually are sufficient for the other products on
that machine.

 If the machine is dedicated for a particular product or tier, then refer to the documentation in the
installation guide and the particular vendor's site for further advice on setting up the operating
system in an optimal state.

Optimize connection pools


One of the settings that will affect performance is the size of the connection pools at each layer in the
architecture. Insufficient pool sizes can cause unnecessary transaction queues that can cause
unnecessary delays. Conversely setting the pool sizes too high can cause higher than usual resource
usage on a machine also causing adverse performance. So a balance needs to be struck for
optimization.

During the implementation the size of the connection pools is determined and configured (with relevant
growth tolerances) depending on the usage patterns and expected peak/normal traffic levels. The
goal, typically, is to have enough connections available at normal traffic levels to minimize queuing
and also have the right tolerances to cater for any expected peak periods. Therefore, it is
recommended:

 Set the number of initial connections to the normal number of connections expected.
Remember this is not the number of users that are connecting but the expected number of
concurrent connections under normal load.

 Set the tolerances for pool growth (usually a maximum pool size and a connection
increment) to the peak load expected at any time. This tolerance will have to be tracked to
determine the optimal level. Do not be tempted to set this to a very large value as memory and
network bandwidth calculations are usually dependent on the values specified and wastage of
resources needs to be minimized.

The product has up to three connection pools to configure:

 Client connections – These are the number of active connections supported on the Web
Application Server from the client machines. Remember that in an OLTP product the number of
connections allocated is always less that the number of users on the system. It needs to be
sufficient to cater for the number of actively running transactions at any given point of time. Refer to
Configuring the Client Thread Pool Size for more information about pool sizing.

Note: Remember it is possible to set a different client connection pools per channel. For example,
using Work Managers you can limit online and/or web service calls.

 Database connections – These are the number of pooled connections to the database. The
Framework holds these connections open so that the overhead of opening and closing connections

19 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
is minimized. For Version 2.x of the product, the number of connections allocated is dictated in each
individual web applications hibernate.properties file using ucp or JDBC managed connection
pools within the Oracle WebLogic JNDI.

The figure below illustrates the connection pools available for the Oracle Utilities Application
Framework:

Figure 3. Connection Pools

Refer to the Server Administration Guide provided with your product for advice on the configuration
and monitoring of the connection pools.

Read the available manuals


The Oracle Utilities Application Framework product includes a set of documentation that should be
downloaded with the software and read as part of the implementation and support of product. The
following technical documentation is available on the distribution web:

 Installation Guide – Installation documentation for the base product including supported platforms
and required patches.

 Server Administration Guide – Documentation on how to configure and operate the server
components of the product.

 Batch Server Administration Guide - Details of the configuration settings and common operations
for the batch component of product.

 Developer documentation – This is detailed documentation on the customization aspects of the


implementation including standards for implementations. This is supplied with the online
documentation and with the Oracle Utilities SDK. The ConfigTools documentation is included in the
online documentation.

 Online documentation – This is comprehensive functional help, development guidance and

20 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
implementation documentation available via the online help installed with the product, via
doc.oracle.com and available in PDF format from Software Delivery Cloud.

Technical Documentation Set Whitepapers available


Apart from the product based documentation there are a number of whitepapers that provide specialist
and supplemental information for use during and post implementation. The table below lists the current
available technical documentation as well as the Knowledge base Id within My Oracle Support where Whitepaper are documents on
specialist topics that provide
the documentation resides:
advice for implementations.

Available Technical Whitepapers

REFERENCE WHITEPAPER TITLE CONTENT

560382.1 Performance Troubleshooting Guideline A series of whitepapers outlining the tracking


Series points available in the architecture for
performance and a troubleshooting guide
based upon common problems.

560401.1 Software Configuration Management Series This series of documents outlines a set of
generic processes (that can be used as part
of the site processes) for managing code and
data changes. This series includes
documents that cover concepts, change
management, defect management, release
management, version management,
distribution of code and data, management
of environments and auditing configuration.
The individual whitepapers are as follows:
 Concepts - General concepts and
introduction.
 Environment Management - Principles
and techniques for creating and managing
environments.
 Version Management - Integration of
Version control and version management
of configuration items.
 Release Management - Packaging
configuration items into a release.
 Distribution - Distribution and installation
of releases across environments
 Change Management - Generic change
management processes for product
implementations.
 Status Accounting - Status reporting
techniques using product facilities.
 Defect Management - Generic defect
management processes for product
implementations.
 Implementing Single Fixes - Discussion
on the single fix architecture and how to
use it in an implementation.
 Implementing Service Packs -

21 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Discussion on the service packs and how
to use them in an implementation.
 Implementing Upgrades - Discussion on
the upgrade process and common
techniques for minimizing the impact of
upgrades.

773473.1 Oracle Utilities Application Framework A whitepaper outlining the security facilities
Security Overview in the Oracle Utilities Application Framework.

774783.1 LDAP Integration for Oracle Utilities A whitepaper outlining the common process
Application Framework based products for integrating an external LDAP based
security repository with the framework.

789060.1 Oracle Utilities Application Framework A whitepaper outlining all the various
Integration Overview common integration techniques used with
the product (with case studies)

799912.1 Single Sign On Integration for Oracle A whitepaper outlining a generic process for
Utilities Application Framework based integrating a single sign on product with the
products Oracle Utilities Application Framework.

807068.1 Oracle Utilities Application Framework A whitepaper outlining the different variations
Architecture Guidelines of architecture that can be considered. Each
variation will include advice on configuration
and other considerations.

836362.1 Batch Best Practices for Oracle Utilities A whitepaper outlining the common and best
Application Framework based products practices relating to batch processing.

1177265.1 What's New in Oracle Utilities Application This whitepaper outlines the changes since
Framework V4? the V2.2 release of Oracle Utilities
Application Framework.

1290700.1 Database Vault Integration This whitepaper outlines the Database Vault
integration available with Oracle Utilities
Application Framework V4.1 and above.

1299732.1 BI Publisher Integration Guidelines This whitepaper outlines some guidelines for
integration available with Oracle BI Publisher
for reporting.

1308161.1 Oracle SOA Suite Integration This whitepaper outlines the integration
between Oracle SOA Suite and the Oracle
Utilities Application Framework.

1308181.1 Oracle WebLogic JMS Integration This whitepaper outlines the inbuilt
integration between Oracle WebLogic JMS
and the Oracle Utilities Application
Framework.

1375600.1 Oracle Identity Management Suite This whitepaper outlines integration between
Integration the product and components of Oracle
Identity Management Suite.

1375615.1 Oracle Utilities Application Framework This whitepaper outlines common security
Advanced Security requirements and outlines how the security

22 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
within the product and components of Oracle
Identity Management Suite can be used to
implement that requirement.

1474435.1 Oracle Application Management Pack for This whitepaper outlines the features and
Oracle Utilities Overview functions of the Oracle Application
Management packs available for Oracle
Enterprise Manager.

1506855.1 Integration Reference Solutions This whitepaper outlines all the integrations
with Oracle technology possible with Oracle
Utilities Application Framework with solution
strategies.

1558279.1 Oracle Service Bus Integration This whitepaper describes direct integration
with Oracle Service Bus including the new
Oracle Service Bus protocol adapters
available for Outbound Messages and
Notification.

1561930.1 Using Oracle Text for Fuzzy Searching This whitepaper describes how to use the
Name Matching and fuzzy operator facilities
in Oracle Text to implement fuzzy searching
using the @fuzzy helper function available
in Oracle Utilities Application Framework
V4.2.0.0.0 and above.

1606764.1 Audit Vault Integration This whitepaper describes the integration


with Oracle Audit Vault to centralize and
separate Audit information from Oracle
Utilities Application Framework products.
Audit Vault integration is available in Oracle
Utilities Application Framework 4.2.0.1.0 and
above only.

1643845.1 Private Cloud Planning Guide This whitepaper outlines how to the
recommended architecture of implementing
Oracle Utilities applications on Oracle's
Private Cloud.

1644914.1 Migrating XML Application Integration to This whitepaper outlines the features of
Inbound Web Services Inbound Web Services (IWS) which replaces
the XML Application Integration (XAI)
functionality. It covers techniques used to
migrate from XAI to IWS.

1682436.1 ILM Planning Guide This whitepaper outlines the Information


Lifecycle Management based product data
management solution to help minimize
storage costs for Oracle Utilities Application
Framework based products.

1929040.1 ConfigTools Best Practices This whitepaper outlines techniques to


implement customizations using the
ConfigTools functionality of Oracle Utilities
Application Framework

2014163.1 Oracle Utilities Testing Accelerator This whitepaper outlines the features and

23 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Overview functions available in the Oracle Utilities
Testing Accelerator.

2132081.1 Migrating From On Premise To Oracle This whitepaper outlines the process of
Platform As A Service moving an Oracle Utilities product from on-
premise to Oracle Cloud Platform As A
Service

2196486.1 Batch Scheduler Integration This whitepaper outlines the Oracle Utilities
Application Framework based integration
with Oracle’s DBMS_SCHDEULER to build,
manage and execute complex batch
schedules.

2211363.1 Enterprise Manager for Oracle Utilities This whitepaper outlines the process of
Whitepaper: Service Pack Compliance converting service packs to allow the
Application Management Pack for Oracle
Utilities to install service packs using the
patch management capabilities.

2214375.1 Web Services Best Practices This whitepaper outlines the best practices of
the web services capabilities available for
integration including Inbound Web Services,
REST integration, Outbound Messages and
Real Time Adapters.

2404696.1 Implementing Oracle In Memory Option This whitepaper outlines the process for
implementing Oracle In Memory Database
Store with Oracle Utilities Application
Framework products.

Oracle WebLogic 12.2.x Configuration This whitepaper outlines how to setup a


Guide domain for Oracle WebLogic 12.2.1.x and
above.

This documentation is updated regularly with each release of product with new and improved
information and advice. Announcements of updates to whitepapers may be tracked via The Shorten
Spot blog.

Secure default userids


There are a number of default users (and associated default passwords) that are supplied with the
installation of product. It is recommended that the default users and their passwords be altered
according to the site security standards. Refer to the Security Guide provided with the product for the
list of supplied default users.

Consider different userids for different modes of access


Note: It is not possible to configure product to use different database accounts for access. All modes
of access will share the relevant pool of database connections as a single database user.

In Oracle Utilities Application Framework version 4.0.1 and above the authorization userid is available
as the CLIENT_IDENTIFIER on Oracle database sessions.

By default the application is configured to either use SYSUSER, SPL or IWS to access the product for
online, web services and in background processing. This means any audit or staging records are

24 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
associated with a common userid. Some implementations have created additional userids to use as a
filter for reporting, traceability and auditing purposes. The following guidelines may be used in this
area:

 Create a different userid for integration transactions. This allows tracking of integration within
the architecture. It is also possible to assign each transaction a different userid for integration, as it
is passed as part of the transaction but usually most customers consider this overkill.

 Create a different userid for each background interface. This allows security and traceability to
be tracked at a lower level.

 Create a generic userid for mainstream background processes. This allows tracking of online
versus batch initiation of processes (especially To Do, Case and Customer Contact processing).

Note: Remember that any product user must be defined to the product as well as the authentication
repository. In Oracle Utilities Application Framework 4.2.x and above, there is a separation between
authentication and authorization identifiers.

Don’t double audit


The product has an auditing facility that is soft configured. The facility can be enabled by configuring
the auditing parameters (location of the audit data, audit rules etc) against the meta data definitions of
the tables. This ensures that any online or web service based updates are audited according to those
rules. Auditing is used to track online changes to critical entities.

The financial component of product already has a separate auditing facility, as all customers generally
require it. Any changes to financial information such as payments, adjustments, bills etc are registered
in the Financial Transaction tables. Therefore enabling auditing on those entities is not required and
constitutes double auditing (i.e. auditing information is stored in two places).

While the impact of the double auditing may be storage related, enabling auditing on bills, for example,
can have a performance hit on online bills. Customers with large numbers of bill segments per bill (i.e.
several hundred) have experienced negative performance impact during online billing when double
auditing is enabled on financial entities. This does not affect batch performance as auditing is not used
in batch.

Use Identical Machines


The flexibility of the technology used by the product allows the ability to mix-and-match different
hardware for a configuration. While this may be attractive and allow for some innovative solutions, it
makes overall manageability and operations harder. Hence, it should be avoided.

Having identical hardware allows for ease of stocking spare parts, better reproducibility of problems
(both software and hardware), and reduces the per platform testing cost. This cost, in many cases, will
surpass the savings from reusing existing disparate hardware.

Regularly restart machines


It is generally a good practice to restart servers periodically. This recovers slow memory leaks,
temporary disk space build-up, or other hidden problems that may manifest themselves only when the
server is up for such a long duration. This is a simple way to avoid unexpected or unexplained failures.

Most hardware vendors have recommendations on optimal time intervals to restart machines. Some
vendors strongly encourage the issue for maintenance reasons. Check with your vendor for specifics
for your platform.

25 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Avoid using direct SQL to manipulate data
Note: Issuing SQL data manipulation language (DML) statements other then SELECT statements
directly against base tables can cause data integrity to be compromised and can invalidate your
product warranty. All data update access should be through maintenance objects that ensure data
integrity is maintained.

Unless the outcome can be verified as correct, you should not use ANY direct SQL statement against
product database as you may corrupt the data and prevent the product from operating correctly.

All the data maintenance and data access in the product is located in the Maintenance Objects. The
Maintenance Objects validate ALL changes against your sites business rules and the rules built into
the product. If you are using the objects to manipulate the data then integrity is guaranteed as:

 All the validations including business rules, calculations and referential integrity are contained within
the Maintenance Objects.

 The Maintenance Object performs a commit when all validations are successful. If any validation is
failed the whole object is rolled back to a consistent state. In background processing, a commit is
performed after a number of Maintenance Objects have been processed (known as the Commit
Interval). At that point the last commit point is registered on the Batch Control for restarting
purposes. If the background process fails between commit points, the database is rolled back to the
last commit point.

 All access modes (online, web services, batch/background processing) from code supplied with the
product use the Maintenance Objects for processing. This means that integrity is guaranteed across
all modes.

 Any customizations (algorithms etc) using the Oracle Utilities SDK will should be using the
Maintenance Objects.

 Using incorrect SQL may violate any of the validations and even make the system unusable.

 If you have to manipulate data within the product, use one or more, of the following provided
methods:

– The browser user interface.

– Inbound Web Services.

– Conversion Toolkit.

– Software Development Kit.

Minimize Portal Zones not used


In the Oracle Utilities Application Framework, portals were introduced to all sites to decide what zones
and their sequence should for different user groups. For performance reasons, it is recommended that
you configure portal preferences to collapse zones that are not needed every time a portal is
displayed.

The system does not perform the processing necessary to build collapsed zones until a user expands
the zone, so configuring them as initially collapsed improves response times. This is especially
relevant for the To Do zones that may take a while if the number of To Do records is excessive.

Routine Tasks for Operations

26 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
After the implementation of the product has been completed there is a common set of tasks that IT
groups perform to maintain the system. The table below lists these tasks:

Common Tasks

TEMPLATE USAGE

Perform Backups Perform the backup of the database and file system using the site
procedures and the tools designated for your site.

Post Process Logs Check the log files for any error conditions that may need to be
addressed. Refer to Post Process Logs and Check Logs For Errors for
more details.

Process Performance Data Collate and process day's performance data to assess against any Server
Level targets. Identify any badly performing transactions.

Perform Batch Schedule Execute the batch schedule agreed for your site. This will include
overnight, daily, hourly and ad-hoc background processes.

Rebuild Statistics Oracle recommends that the database statistics for the product schemas
to be rebuilt on a regular basis so that the any access path chosen by the
Cost Based Optimizer is optimized.

File Cleanup On a regular basis, the output files from the background processes and
logs will need to be archived and removed to minimize disk space usage.

Storage Manage Data not The Oracle Utilities Application Framework features can use Information
required Lifecycle Management to minimize storage. Refer to Information Lifecycle
Management for more details.

Run Cleanup Batch Jobs There are a number of background processes that remove staging
records that have been already successfully processed. Refer to Removal
of Staging Records for more details.

Note: The tasks listed above do not constitute a comprehensive list of what needs to be performed.
During the implementation you will decide what additionally needs to be done for your site.

Typical Business Day


One of the patterns experienced at sites is the notion of a common definition of a business day.
Typically during the implementation the business day is defined for planning purposes. It defines when
the call center is at peak or non-peak, background processing can be performed and when backups
are performed during the business day.

The figure below illustrates a simplified model of a typical customer business day:

27 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 4. Example typical business day

Note: The above diagram is for illustrative purposes only and could vary for your site.

Typically a business day contains the following elements:

 There is a peak online period where the majority of call center business is performed.
Typically this is performed in business hours varying according to local custom.

 There is a call center off peak period where the volume of call center traffic is greatly
reduced compared to the peak period. Typically in call centers, which operate 24x7, this
represents overnight and weekends. At this time the call center is reduced in size (usually a
skeleton shift). Some sites do not operate in non-peak periods and rely on automated technology
(e.g. IVR) to process transactions such as payments etc.

 Backups are either performed at the start of the peak period or the end of the peak period. The
decision is based upon risk around failure of the background processing and its risk to the impact of
online processing. The product specific background processes can be run anytime but avoiding
them during peak time will maximize the available computing resources to the successful processing
of call center transactions. The backup at the end of the peak period is the most common patterns
amongst product customers.

 Background processes are run at both peak and off peak times. The majority of the background
processing is performed at off-peak times to maximize the computing resources to the successful
completion of the background processing. The background processing that is run at off peak times
is usually to check ongoing call center transactions for adherence to business rules and process
interface transactions ready for overnight processing.

 Monitoring is performed throughout both peak and off peak times. The monitoring regime used
may use manual as well as automated tools and utilities to monitor compliance against agreed
service levels. Any non-compliance is tracked and resolved.

The definition of the business day for you site is crucial to schedule background processing and set
monitoring regimes appropriate for the traffic levels expected.

Login Id versus Userid


Note: This facility is available in Oracle Utilities Application Framework V4 and above only.

In the past releases of the Oracle Utilities Application Framework the userid that could be used to login
was restricted to 8 characters in length. In Oracle Utilities Application Framework V4 and above, it is
possible to use a user identifier of up to 256 characters in length.

In Oracle Utilities Application Framework V4 and above the concept of a Login Id is supported. This
attribute is the used by the framework to authenticate the user. For backward compatibility the 8
character userid field is still used for auditing purposes internally. Therefore both Userid and Login Id
should be populated. They can be different or the same values.

Note: The Login Id can be changed post creating the user identity to support name change,
acquisitions etc. The short User Id is not changeable as records in the product already use this value.

28 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
The Login Id can be set manually, via Oracle Identity Manager or set in a Pre-Processing Algorithm on
2
the User Business Object to auto generate a value.

Hardware Architecture Best Practices


Note: There is a more detailed discussion of effective architectures in the Oracle Utilities Application
Framework Architecture Guidelines (Doc Id: 807068.1) whitepaper available from My Oracle Support.
Refer to that document for further advice.

The product can be run on various combinations of hardware based architectures. When choosing an
architecture that is best suited to a site there are a number of key factors that must be considered:

 Cost. When deciding a preferred architecture, the total cost of the machine(s) and infrastructure
needs to be taken into a consideration. This should the ongoing costs of maintenance as well as
power costs.

 IT Maintenance Effort. When deciding a preferred architecture, the manual or automated effort in
maintaining the hardware in that architecture needs to be factored into the solution.

 Availability. One of the chief motivations for settling on a multi-machine architecture is requiring the
architecture to support high availability. When deciding a preferred architecture, the tolerance and
cost of availability needs to be factored into the solution.

SINGLE SERVER ARCHITECTURE

If the site is cost sensitive and/or the availability requirements allows it, then having all the architecture
on a single machine is appropriate. This is known as the single server architecture. This configuration
is popular with some sites as:

 The cost of the hardware can be minimal (or least very cost effective).

 Maintenance costs can be minimized with the minimal hardware.

 Virtualization software (typically part of the operating system or third party virtualization software)
can be used to partition the machine into virtual machines.

The one issue that makes this solution less than ideal is the risk of unavailability due to hardware
failure. Customers that choose this solution, typically address this shortcoming by buying a second
machine of similar size and using it for failover, disaster recovery as well as non-production. In
essence, if the primary hardware fails then the backup machine assumes the responsibility for
production till the hardware fault is resolved. In this case, additional effort is required to keep the
secondary machine in synchronization with the primary.

The diagram below illustrates the single server architecture:

2
Available in Oracle Utilities Application Framework version 4.3.0.4.0 and above only for Identity Manager integration.

29 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 5. Example Single Server Architecture

SIMPLE MULTI-TIER ARCHITECTURE

One of the variations on the single server architecture is the simple multi-tier architecture. In this
hardware architecture, the database server and Web Application Server/Business Application Server
are separated on different machines.

This is chosen by customers who want to optimize the hardware for the particular tier (settings and
size of machine) and therefore separate the maintenance efforts for each server. For example,
Database Administrators need only access the Database Server to perform their duties and set the
operating system parameters optimized for the database.

Unfortunately the solution can have a higher cost than the single server solution and still does not
address the unavailability of any machine in the architecture. Customers that have used this model
adopt a similar solution to the single server architecture (duplicate secondary machines at a secondary
site) but also have the option of having both machines in the architecture being the same size and
shifting the roles when availability is compromised. For example, if the database server fails, the Web
Application Server can be configured to act as a combination of the Database Server and Web
Application Server.

The figure below illustrates the Simple Multi-Tier Architecture:

30 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 6. Example Simple Multi-Tier Architecture

Machines in this architecture can be the same size or different sizes depending on the cost/benefits of
the various variations. Typically customers use a smaller machine for the Web Application Server as
compared with the database server.

MULTIPLE WEB APPLICATION SERVERS

To support higher availability for the product, some sites consider having multiple Web Application
servers. This allows online users to be spread across machines and in the case of a failure be diverted
to the machine that is available. To achieve this, the site must use a load balancer (see Load
balancers discussion later in this document). At the time of failover the load balancer will redirect traffic
to the available server. This is made possible as the product is stateless.

The Web Applications Servers are either clustered or managed. Refer to the discussion in the
Clustering or Managed? section of this document for advice.

This architecture is quite common as it represents flexibility as one of the Web Application Servers can
be dedicated to batch processing in non-business hours making the architecture more cost effective.
Typically the Web Application Server software is shutdown to allow batch processing to use the full
resources of the machine while allowing users (usually a small subset) to process online transactions.

The only drawbacks with this solution are a potential higher cost than a multi-tier solution and the
potential impact of database unavailability. Customers that use this architecture overcome the
potential unavailability of the database by either using a secondary site to act as the failover or using
one of the Web Applications in a failover database server role. The latter is less common, as most
customers find it more complex to configure, but is possible with this is a possibility with this
architecture.

The figure below illustrates the Multi-Tier Architecture:

Figure 7. Example Multi-Application Server Architecture

31 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Machines in this architecture can be the same size or different sizes depending on the cost/benefits of
the various variations. Typically customers use a smaller machine for the Web Application Server as
compared with the database server.

High Availability Architecture


Note: Customers/partners should consider the Maximum Availability Architecture guidelines for Oracle
Fusion Middleware and Oracle Database when designing high availability architectures.

The most hardware intense solution is where all the tiers in the architecture have multiple machines for
high availability and distribution of traffic. The solution can vary (number of machines etc) but have the
following common attributes:

 There is no single point of failure. There is redundancy at all levels of the architecture. This
excludes redundancy in the network itself, though this is typically out of scope for most
implementations.

 Consider Segmentation. The number of servers will depend on segmentation of the traffic between
call centers, non-call centers, interfaces and batch processing. It is possible to reuse existing
servers or setup dedicated servers for different types of traffic.

 Flexible Availability. Availability can be managed with either hardware based solutions, software
based solutions or a valid combination of both.

 Consider the number of active users. The number of users will dictate the number of machine to
some extent. Experience has shown, that a large number of users tend to be better served, from a
performance and availability point of view, by multiple machines. Refer to the What is the number of
Web Application instances do I need? for a discussion on this topic.

 Consider how to manage multiple servers. The Web Applications Servers are either clustered or
managed. Refer to the discussion in the Clustering or Managed? section of this document for
advice.

 Use Database High availability solutions. Database clustering is typically handled by the
clustering or grid support supplied with the database management system.

This solution represents the highest cost hardware from both hardware and a maintenance
perspective. Historically customers with large volumes of data or specific high availability requirements
have used this solution successfully.

The figure below illustrates the High Availability Architecture:

32 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 8. Example High Availability Server Architecture

Note: The Mobile Server is only available in Oracle Utilities Application Framework V4.3.0.5.0 and
above.

Failover Best Practices


Failover occurs when a server in your architecture becomes unavailable due to hardware or software
failure. Immediately after the failure the active components architecture route the transactions around
the unavailable component to an alternative or secondary component (on another site) to maintain a
level of availability. This routing can be done automatically through the use of high availability
software/hardware or manually by operators.

The Oracle Utilities product architecture supports failover at all tiers of the architecture, using either
hardware or software based solutions. Failover solutions can be varied but a few principles have been
adopted successfully by existing customers:

 Failover solutions that are automated are preferable to manual intervention. Depending on the
hardware architecture used the failover capability can be automated.

 Availability goals play a big part in the extent of a failover solution. Sites with high availability
targets tend to favor more expensive, comprehensive hardware and software solutions. Sites with
lower availability (or no goals) tend to use manual processes to handle failures.

 Failover is built into the software used by the products. For example, Web Application Server
3

vendors have inbuilt failover capabilities including load balancing, which is popular with customers.

 Hardware vendors will have failover capabilities at the hardware or operating system level. In
some cases, it is an option offered as part of the hardware. Sites use the hardware solution in
combination with a software based solution to offer protection at the hardware level. In this case, the

3
This may entail an additional license from the relevant vendor.

33 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
hardware solution will detect the failure of the hardware and work in conjunction with the software
solution to route the traffic around the unavailable component.

 Failover is made easier to implement for the product as the Web Application is stateless. The
users only need connection to the server while they are actively sending or receiving data from the
server. While they are inputting data and talking on the phone they are not consuming resources on
the machine. For each transaction the infrastructure routes the calls across the active components
of the architecture.

 At the database level the common failover facility used is the facility provided by the
database vendor. For example, Oracle database customers typically implement RAC. Failover
configuration at the database is the least used by existing sites, as the cost of having additional
hardware is usually prohibitive (or at least not cost configurable).

 Sites wanting to have failover and disaster recovery but cannot afford both consider a
solution which combines both. In this case, the disaster recovery configuration is used as a
failover for non-disasters.

For any failover solution to be effective, the site typically analyses all the potential areas of failure in
their architecture and configures the hardware and software to cover that eventuality. In some cases,
sites have chosen NOT to cover eventualities of extremely low probability. Using hardware Mean Time
between Failure (MTBF) values from hardware vendors can assist in this decision.

When designing a failover solution then the following considerations are important:

 Determine what the availability goals are for your site.

 Determine the inbuilt failover capabilities of the hardware and software that your site is using. This
may reduce the cost of implementing a failover solution if it is already in place.

 List all the components that need to be covered by a failover solution. Review the list to ensure all
aspects of "what can fail?" are covered.

 Design your failover solution with all the above information in mind that you can automate (within
reason) for your site. Ensure the solution is simple and reuses already available infrastructure to
save costs.

Commonly sites use the following failover techniques in the architecture:

Common Failover Solutions

TIER COMMON FAILOVER SOLUTIONS

Network Load Balancer (hardware for large numbers of users; software based for
others). Consider redundant load balancers for no single point of failure
requirements.

Application Server Use inbuilt clustering/failover facilities unless load balancer is doing this.
Consider hardware solutions for batch or interface servers.

Database Server Use inbuilt failover facilities in database unless hardware solution is more
cost effective.

General Troubleshooting Techniques


Whilst the troubleshooting features of the product are documented in detail in the online help,

34 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Performance Troubleshooting Guideline Series (Doc Id: 560382.1) whitepapers available from My
Oracle Support and other manuals there are a number of techniques and guidelines that can be used
to help identify problems:

 Check the logs in the right order – The log files are usually the best spot to look for errors as any
error is automatically logged to then by the product. The most efficient method is to look for the logs
from the bottom up as if the error appears in the lower ranks of the architecture that is more likely
4
where the error occurred . Typically you look for records of type ERROR in the following logs (located
in $SPLEBASE/logs/system):

Log Files

LOG FILE COMMENTS

spl_service.log Business Application Server log. In some versions of the Oracle Utilities
Application Framework this log does not exist as it is included in the
spl_web.log. Errors in here can be service or database related.

spl_xai.log/spl_iws.log Web Services Integration also known as XML Application Integration


(XAI) or Inbound Web Services log. This log file is exclusively used for the
XAI Servlet. More detail can exist in the xai.trc file if tracing is enabled.

spl_web.log Web Application Server log. This is typically where errors from the
browser interface are logged. If errors are repeated from the
spl_service.log then the issue is not in the Web Application Server
software but in the Business Application or below.

spl_mobile.log Mobile Server log. This is typically where errors from the mobile REST
interface are logged.

Note: There are other logs that are related to the Java EE Web Application Server used that exist in
this directory or under the location specified in the Java EE Web Application Server.

 First error message is usually the right one – When an error occurs in the product, it can cause
other errors. Usually the first occurrence of any error is usually the root cause. This is more
apparent when a low level error occurs which ripples across other processes. For example, if the
database credentials are incorrect then the first error will be that the product cannot connect to the
database but other errors in the product will appear as meta data cannot be loaded into various
components. In this case fixing the database error will correct the other errors as well.

 Not all errors are in fact errors – The product will issue errors if components are missing but are
able to overcome this issue. For example, if meta-data is missing the system may resort to using
default values. In most cases this means the product can operate without incident but the cause
should be resolved to ensure correct behavior.

Note: In some versions, such errors are reported as a WARNING rather than an ERROR.

 Tracing can help find the issue – The products include trace facilities that can be enabled to help
resolve the error. This information is logged to the logs above (and other server logs) that can be
used for diagnosis as well as for support calls. Refer to Online and Batch tracing and Support
Utilities for more information about these tools.

4
The theory is that the first place the error occurs is the most likely candidate tier.

35 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 There are usually a common set of candidates – When an error occurs there are a number of
typical candidates for causing issues:

– Running out of resources – The product uses the resources allocated to it that are available on
the machine. If some capacity is reached in the physical machine (memory or disk space are
typical resource constraints) or logical, via configuration, such as JVM memory allocations, then
the product will report a resource issue. In some cases, the product will directly report the problem
in the logs but in some case it will be indirectly. For example, if the disk space is limited then a log
may not be written which can cause issues.

– Incorrect configuration – If the product configuration files or internal configuration are incorrect
for any reason, they can cause errors. A common example of this is passwords which either are
wrong or have expired. File paths are also typical settings to check.

– Missing metadata – The product is meta-data driven. If the metadata is incorrect or missing then
the behavior of the product may not be as expected. This can be hard to detect using the usual
methods and typically requires functionality testing rather than technical detective work.

– Out of date software – All the software used in the solution, whether part of the product or
infrastructure, has updates, patches and upgrades to contend with. Upgrading to the latest patch
level typically can address most issues.

Refer to the Performance Troubleshooting Guideline Series (Doc Id: 560382.1) whitepapers available
from My Oracle Support for more techniques and additional advice.

Technical vs User Logs


Note: This facility is only available in Oracle Utilities Application Framework 4.3.0.5.0 and above only.

For security reasons the logging in the product has been changed to separate the business
information for an error and the technical information. This is used primarily in the Oracle Cloud
implementations of the product but is also available for on-premise implementations by default.

When implemented there are two logs for each tier with the following content:

Separated Log Files

LOG FILE CONTENTS

*-tech.log Technical information about the server including java errors, cache messages and other
technical related messages. This log will not contain any business related data associated
with the error for security purposes.

*-user.log Log contains any business related messages associated with the server including any
business data related with the errors and business data for traces.

COMBINING LOGS

If you do not want separate logs and want the traditional combined log then you can configure the
product to combine the logs using the following steps:

 Alter the log4j2.properties file, using a custom template to add the following section to the
configuration:

! =========== Full-log appender ===========

36 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
appender.fullLog.type = RollingFile

appender.fullLog.name = fullLog

appender.fullLog.append = false

appender.fullLog.fileName=app-full.log

appender.fullLog.filePattern=app-full.log.%d{yyyy-MM-dd}

appender.fullLog.policies.type = Policies

appender.fullLog.policies.time.type = TimeBasedTriggeringPolicy

appender.fullLog.policies.time.interval = 1

appender.fullLog.policies.time.modulate = true

appender.fullLog.strategy.type = DefaultRolloverStrategy

appender.fullLog.strategy.max = 20

appender.fullLog.layout.type = PatternLayout

appender.fullLog.layout.charset = UTF-8

appender.fullLog.layout.pattern = %X{userId} - %X{transactionId} %X{wlECID}


%d [%t] %-5p (%c{3}) %m%n

 Remove the appender.userLog.* and appender.techLog.* entries to remove the individual


logs after combining them.

Overriding System Date


By default, the product obtains the date from the database server using an internal call. This is to
ensure a consistent recording of the date and time across the system, irrespective of the time zone of
the user. It is possible in some products, to adjust this time to the local time using the user time zone
feature of Oracle Utilities Application Framework V4.0 and above.

It is possible to set a date for testing purposes at the system level as well as at user level.

Note: This facility is not recommended for use in production environments. It is only recommended in
non-production environments such as testing and development.

Note: To avoid issues with data values, to enable this feature, after configuring at the system or user
level the setting spl.runtime.options.allowSystemDateOverride must be set to true in the
spl.properties file for online, business application server and/or batch.

SYSTEM WIDE

To set a specific date for an environment, for testing purposes, a Feature may be added using the
Feature Configuration menu option. The feature to use is a General System Configuration feature. You
may create a General System Configuration feature if one does not exist on your environment. The
System Override Date option may be added to this feature and the date specified in international ISO
format (e.g. YYYY-MM-DD). An example of this feature is shown in the figure below.

Note: Only one General System Configuration feature should exist per environment.

37 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 9. Adding a feature for System Override Date

Once saved, this date will be used, across ALL users on that environment, instead of the date for
online and Web Service operations. You may need to flush the online cache to reflect the change
across the system. Refer to the Server Administration Guides supplied with the product for techniques
on how to flush the cache. To reverse the configuration, remove the System Override Date option from
the Feature Configuration.

USER SPECIFIC DATE

Note: This feature was added in Oracle Utilities Application Framework V4.1 and consequently is only
available for that version and above.

Whilst the system override is not appropriate as some users require specific dates, especially for
testing, then it is also possible to override the system date per user. This will force the online to use
the override value for the user, if it exists, instead of any system override or system date.

To achieve this, an Override System Date characteristic can be added to the individual user record
using the User menu option as shown in the figure below. As with the System Override, the
Characteristic value should be in ISO format (e.g. YYYY-MM-DD).

Figure 10. Adding a System Override Date per user

As with the System wide override, the user should refresh the cache to reflect the change. To reverse
the configuration, remove the characteristic from the user record.

Transaction Timeouts
By default, transactions are subject to time limits imposed at the infrastructure level (at a network or
database level). In most cases, sites do not impose any explicit time limits.

The idea of time limits on transactions is to catch any long running online or web service transaction
from causing inefficiencies in traffic volumes across your configuration. To use time limits effectively,
the site would want to set limits on a number of key common transactions to keep a cap on resource
usage across the enterprise.

38 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
In Oracle Utilities Application Framework V4.1 and above, a set of optional configuration settings has
been added to allow sites to specify global time limits and transaction level time limits on individual
calls within a transaction.

Note: For Oracle Utilities Application Framework V4.1 this capability is enabled using patch 10356853.

The concepts of timeouts contained in the product are as follows:

 Timeouts may be set globally or overridden on individual services, business objects, business
services or service scripts.

 Timeouts are tracked throughout the transaction execution but the timeout is explicitly checked prior
to any database access to ensure the timeout has not been reached. If the transaction has multiple
database access statements, the current cumulative transaction time is checked at each statement.
If no database access is made by the transaction then timeouts are not checked and therefore not
enforced.

 Timeouts tolerances are taken from the parent object. For example, if a service script calls business
objects and/or business services then the time allocated to the service script applies across all the
calls not the timeouts of the individual business objects and/or business services.

 The timeout values only apply to online transaction and web services calls, therefore batch
transactions are not subject to using this timeout facility.

 When a timeout limit is exceeded, an appropriate error message is returned and the transaction is
rolled back to the last commit or savepoint.

 Timeout values are not precise. The transaction will not be terminated at the exact timeout value.
When timeouts occur, additional processing time for transaction rollback and networking may need
to be executed, which is performed after the timeout occurs, before returning control back to the
originating user.

Most of the transactions in the product are subject to one timeout. If the transaction is part of a portal
with multiple transactions, each individual zone is subject to its own timeout. If timeouts are not
consistent across a portal then some zones may be timed out whilst others continue processing.

CONFIGURATION OF TIMEOUTS

To configure the use of timeouts for online or Web Service traffic a number of configuration settings
need to be specified in configuration files within the Web Application Server and Business Application
Server. The parameters that control timeouts are as follows:

Timeout Parameters

PARAMETER COMMENTS

ouaf.timeout.business_object.<bocode> Maximum amount of time (in seconds) for business


object <bocode> can execute before timeout. This
timeout will override
ouaf.timeout.business_object.default
when executing this specific business object. The
values for <bocode> may be any valid business
object.

ouaf.timeout.business_object.default Maximum amount of time (in seconds) an


invokeBO call can last. All queries issues by the

39 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
business object will have life time remaining time of
execution of this business object call. This is a
general timeout and can be overridden for an
individual business object, if desired.

ouaf.timeout.business_service.<bscode> Maximum amount of time (in seconds) for business


service <bscode> can execute before timeout.
This timeout will override
ouaf.timeout.business_service.default
when executing this specific business service. The
values for <bscode> may be any valid business
service.

ouaf.timeout.business_service.default Maximum amount of time (in seconds) an


invokeBS can execute before timeout. All queries
issues by the business service will have life time
remaining time of execution of this business service
call. This is a general timeout and can be
overridden for an individual business service, if
desired.

ouaf.timeout.query.default Maximum amount of time (in seconds) an individual


query can run if it is not restricted by a service or
some other timeout. For instance, if the online
application is issuing a query, which is not a part of
a service call, a script or a Business Object read,
the query will be affected by this timeout.
Otherwise, the timeout will be set to remaining time
of a logical transaction it belongs to (service call,
script and Business Object execution).

ouaf.timeout.script.<scriptname> Maximum amount of time (in seconds) for script


<scriptname> can execute before timeout. This
timeout will override
ouaf.timeout.script.default when
executing this specific service script. The values for
<scriptname> may be any valid service script.

ouaf.timeout.script.default Maximum amount of time (in seconds) a service


script call can execute before timeout. All queries
issues by the script will have life time remaining
time of execution of this script call. This is a
general timeout and can be overridden for an
individual service script, if desired.

ouaf.timeout.service.<service> Maximum amount of time (in seconds) for service


<service> can execute before timeout. This
timeout will override
ouaf.timeout.service.default when
executing this specific service. The values for
<service> may be any valid application service

ouaf.timeout.service.default Maximum amount of time (in seconds) a service


call can execute before timeout. All queries issues
by the service will have life time remaining time of
execution of this service call. This is a general
timeout and can be overridden for an individual

40 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
service, if desired.

For example, in general:

ouaf.timeout.service.default=300

For example, to specify values for logical transactions that can be overridden with values specific for a
service, script or business object operation:

ouaf.timeout.service.default=300

#specified a value for individual service

ouaf.timeout.service.CILTPD=600

In the example above a value specific for service CILTPD was specified.

To implement the transaction timeouts for your site then the following files need to be updated:

 Specify the value of ouaf.timeout.query.default in the Web Application server


spl.properties file:

$SPLEBASE/etc/conf/root/WEB-INF/classes/spl.properties (Linux/Unix)

or

%SPLEBASE%\etc\conf\root\WEB-INF\classes\spl.properties (Windows)

 Specify the other timeout parameters in the Business Application Server spl.properties file:

$SPLEBASE/etc/conf/service/spl.properties (Linux/Unix)

or

%SPLEBASE%\etc\conf\service\spl.properties (Windows)

Note: Changing this file manually may lose changes across upgrades. If the changes need to be
preserved across upgrades then it is recommended to implement a custom template for this file. Refer
to the Server Administration Guide supplied with your product for details of this process.

IMPLEMENTING GUIDELINES

The timeout specification is flexible and powerful but can be overused if over configured. The following
guidelines should be taken into account when using this facility:

 Specify global defaults (*.default parameters) according to your site expectations and any
service level agreements.

 Consider the same common value for global defaults unless the customizations at your site dictate
different values for each object type default.

 Only set overrides on specific objects/services/scripts on the subset of these objects that are critical
to your site and require explicit different timeout values from the relevant defaults. In other words,
avoid setting overrides on all services individually.

 In portals, each zone may have different timeout values. Take this into account when designing

41 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
timeout values.

 Review timeout values with the business on a regular basis to see if the value currently set is
appropriate for the site.

Java Garbage Collection Guidelines


One of the major features of java is its ability to perform garbage collection. In the past programmers
would have to manage their memory allocation, and more importantly, memory de-allocation to
manage the memory footprints of their products. If this was not performed then the program may run
out of physical or logical memory and stop executing (i.e. crash). Java attempts to help developers by
automatically clear up as much memory when it hits a critical threshold. It basically looks for inactive
bits of code or objects and cleans them up to free up memory for active processes. Hence the term
garbage collection, which implies that java collects (i.e. removes) garbage (i.e. inactive) code and
objects.

Java performs garbage collection automatically when it detects a memory tolerance has been
reached. This has the advantage that it can help prevent out of memory conditions within the java
virtual machine. Now I did not use the word guarantee as if the java virtual machine is extremely active
then garbage collection may not be able to find enough to prevent an issue. This is usually a rare
occurrence but can happen.

Whilst garbage collection is an advantage in terms of memory management it has a darker side. When
garbage collection is triggered, all activity within the java virtual machine freezes for the garbage
collection to do its work efficiently. In most cases, the amount of time it freezes is short but if the
garbage collection activity is frequent then the garbage collection time can impact performance of a
java application.

Whilst the default garbage collection regime shipped with the versions of the java virtual machine are
adequate for most sites, it is possible to tweak the garbage collection tolerances and algorithms that
are used to speed up the garbage collection process or make sure that garbage collection is not called
as often as it may be using the defaults.

The key here is to ensure that the whilst you cannot avoid garbage collection happening you want to
minimize its impact by minimizing the frequency it occurs, and when it occurs, minimizing it's impact.

There are a few guidelines to consider when tuning java garbage collection for applications:

 Consider using Parallel Garbage Collection – In later versions of java, the ability to garbage
collect using multiple CPU's in parallel was introduced to minimize the time garbage collection was
executing.

 Tweaking Garbage Collection tolerances – By default the java virtual machine has a specific set
of tolerances for initiating garbage collection. This may be able to be tweaked to decrease the
frequency and duration of the garbage collection. The java virtual machine documentation supplied
by your virtual machine vendor will outline the options to set this.

 Tweaking memory parameters – By default java allocates regions of memory within a java virtual
machine to manage the lifecycle of classes and objects. If a product is using more of one region
over another, this can force garbage collection to occur. Each vendor of the java virtual machines
offers s set of options to tweak the garbage collection algorithm and allocation of memory regions.
Refer to Memory Management in Java for more information about java memory.

 Watch your CPU usage – When tweaking the garbage collection parameters or memory settings
you need to watch the amount of CPU used at the time of garbage collection. This will ensure that

42 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
while solving one problem, the impact of garbage collection on performance, does not introduce a
new issue, excessive CPU.

For more information about garbage collection and tuning it, refer to Java Garbage Collection Tuning
Guide. Also review the Garbage collection information supplied by the vendor of the java virtual
machine for specific advice for your platform.

Security Configuration
One of the features of the product is the ability to configure the authentication component of security.
As with other Java EE based applications the product supports the standard set of settings and
configurations inherent in the Java EE standard. The web.xml file controls the behavior of the
authentication method used in the login-config section of the configuration file.

There are a number of settings and they have additional configuration settings that must be adhered
to:

Login Configuration Settings

SETTING COMMENTS

BASIC This setting uses the operating system login dialog as the product authentication
dialog. The setting may also indicate the realm-name used. This setting is useful
for basic environments and also can be used by some Single Sign On solutions
that detect this setting.

FORM This is the default setting where the product (or implementation) supplies a
JSP/HTML based login dialog (and error dialog) in the form-login-config
section. This is the most common option for the product. Implementers can
implement their own forms according to site standards if desired.

CLIENT-CERT This is more advanced two way SSL based authentication. This is typically used
for Single Sign On and identify federation implementations.

Note: Oracle WebLogic supports fallback mechanisms for authentication methods.

For a more in-depth discussion of this topic refer to the Oracle WebLogic security documentation.

Typically most customers use the default FORM based login option.

Shortcuts When Processing Templates


When updating the configuration settings on a server via the ENVIRON.INI, the initialSetup.sh
utility is needed to reflect all the changes into configuration files via templates. This will automatically
update ALL files. It is possible to shortcut this process and update individual configuration files if the
changes are isolated to specific files.

The following commands can be used:

perl processTemplate.plx –t <list of templates>

where

<list of templates> List of templates with comma delimiters to process

For example:

43 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
perl processTemplate.plx -t jarservice.xml.template,spl.properties.template

Note: Use of processTemplate.plx can lead to mismatched configuration files with EAR/WAR
files. If any change affects the EAR/WAR file it is recommended to run initialSetup[.sh] utility to
apply the configuration changes.

Accessing JMX for Oracle WebLogic


Note: Refer to Accessing WebLogic MBeans with JMX for information about accessing JMX.

Oracle WebLogic supports an extensive JMX interface to expose runtime statistics. To enable this
facility the following configuration process should be performed:

 Enable the JMX Management Server in the Oracle WebLogic console at domain  Configuration 
General  Advanced Settings option. Enable both Compatibility Mbean Server Enabled and
Management EJB Enabled (this enables the legacy and new JMX interface). Save the changes and
restart the server to reflect the change.

Note: For Oracle Utilities Application Framework V4.2.0.0.0 and above, this facility is enabled by
default.

 In the startup of the Oracle WebLogic server in the $SPLSYSTEMLOGS/myserver.log (or


%SPLESYSTEMLOGS%\myserver.log on Windows) you will see the BEA-149512 message
indicating the Mbean servers have been started. The message will indicate the JMX URL that can
be used to access the JMX Mbeans. The URL is in the format:

service:jmx:iiop://<host>:<port>/jndi/<mbeanserver>

where:

<host> Oracle WebLogic host name

<port> Oracle WebLogic port number

<mbeanserver> Valid Values:

weblogic.management.mbeanservers.runtime

weblogic.management.mbeanservers.edit

weblogic.management.mbeanservers.domainruntime

 Ensure that you execute the splenviron[.sh] utility to set the appropriate environment
variables for the desired environment.
 Execute the following jconsole command to initiate the connection to the JMX Mbean server

Windows:

jconsole -J-
Djava.class.path=%JAVA_HOME%\lib\jconsole.jar;%WL_HOME%\server\lib\wljmxcli
ent.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote

Linux/Unix:

jconsole -J-
Djava.class.path=$JAVA_HOME/lib/jconsole.jar;$WL_HOME/server/lib/wljmxclien
t.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote

44 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 To connect to the JMX classes, use specify the Remote process URL from the previous steps (i.,e.
service:jmx:iiop...) using the credentials specified for the Oracle WebLogic console.

The JMX classes are now available.

For example:

Figure 11. Example Oracle WebLogic metrics

Refer to the Oracle WebLogic JMX MBean Reference for more information

UCP JMX Interface


Note: This facility is available in Oracle Utilities Application Framework V4.2.0.0.0 and above.

Note: For backward compatibility purposes this setting is disabled.

Note: If your site is using Data Sources within Oracle WebLogic, then this section is not applicable.
Enabling the JMX within Oracle WebLogic will expose those metrics.

The JMX interface for the product can be extended even further by exposing the UCP connection
pooling metrics to track statistics for database connections. To implement this facility, the following
process should be implemented:

 In the environment to implement this change, copy the hibernate.properties.web.template


to cm.hibernate.properties.web.template within the templates directory.

 Edit the cm.hibernate.properties.web.template and change the


hibernate.ucp.jmx_enabled from false to true. For example:

45 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
hibernate.ucp.jmx_enabled=true

 Execute the initialsetup[.sh] –t option to apply the custom template and replace the
hibernate.properties within the $SPLEBASE\etc\config\service (or for Windows use
%SPLEBASE%/etc/config/service).

 Restarting the online product will enable the statistics.

As the UCP is in the server path for the Java EE Web Application Server, the UCP metrics are
accessible from the Java EE JMX capability. Once it is connected the UCP JMX statistics are
available. For example:

Figure 12. Example UCP metrics

Refer to the UCP Administration java documents for more information on the statistics tracked.

Using Java Mission Control with Oracle Utilities Application Framework


Note: This facility is only available in Oracle HotSpot SDK Verson 7u40 and above and is only
compatible with Oracle Utilities Application Framework V4.2.0.2.0 and above only.

Note: This facility is not appropriate for Inbound Web Services tracking.

Note: JMX for online and batch MUST be enabled for this facility to work. Refer to the Server
Administration Guide for more details.

The Java Mission Control provides developers with low level diagnostics for Java programs. This

46 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
facility is useful for diagnosing performance and coding issues in the implementation process. It is
possible to use this facility, with the right version of Java, for online and batch tracking at the Java
level. It also can be used with Oracle Enterprise Manager to use the Java Virtual Machine Diagnostics
(JVMD) capability.
5
To use this facility there are two basic options that must be added to the command line for Oracle
WebLogic and the Oracle Coherence startup. These options are:

-XX:+UnlockCommercialFeatures -XX:+FlightRecorder

These flags can be added to the product configuration using any of the following techniques:

Java Mission Control Settings

SCOPE INSTRUCTIONS

Online/Mobile/Web Perform either one of the following:


Services  Add the above java options to the setUserOverrides.sh script within the
Oracle WebLogic domain as documented in Customizing Domain Wide Server
Parameters.
 Add the above java options to the individual server startup options in the
Oracle WebLogic console.

Batch Add the above options to the threadpool.*.be templates used by bedit
using the com.ouaf.batch.jvmoptions variable as outlined in the Server
Administration Guide.

Once connected the Flight Recorder and Java Mission Control features are available via the JMX URls
outlined in the Server Administration Guide.

Overload Protection
Note: It is recommended to use the Oracle WebLogic console or WLST to maintain work managers or
use the overload protection features of Oracle WebLogic. Customers using embedded installations
can set the overload protection for the product server in the Installation of the default domain. Refer to
the Installation Guide supplied with the product for details.

By default in Oracle WebLogic the domain uses a global work manager to manage connections. The
issue with the default global work manager is that the setting is effectively unlimited connections. In
non-production, this value is never reached ordinarily but it can cause issues in Production platforms.
If the global default is used then the server may experience an out of memory condition before hitting
the global connection limit. In Oracle Utilities Application Framework implementations there are a
number of ways of addressing this:

 Overload Protection – Oracle WebLogic contains an overload protection setting which tells the
server what to do in an overload situation. Typically there is a setting to handle out of memory
conditions with two settings no-action (default) or system-exit. In a production, where high
availability is typically configured, it is recommended to set this value to system-exit. For more
information about overload protection refer to Avoiding and Managing Overload.

 Using application scoped Work Managers – It is possible to implement an application scoped


work manager configuration, allocated to product servers, to implement tolerances to avoid out of

5
There are other options that are available that can be used to further filter the result sets. Refer to Running Flight Recorder for
other options

47 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
memory conditions by setting constraints low enough to avoid out of memory conditions. The value
of the constraints will depend on how busy your end user traffic is in relation to the individual
servers. The application scoped work manager configuration may use a Max Threads Constraint or
6
a Capacity Constraint . For more information about work managers refer to Using Work Managers
to Optimize Scheduled Work.

Note: It is recommended not to use Execute Queue functionality with Oracle Utilities Application
Framework as that is designed for legacy support. Use of work managers is recommended as an
alternative to the Execute Queue functionality.

 Stuck Thread Handling – As part of the work manager definition it is also possible to specify server
specific stuck thread handling, which directs how WebLogic should handle stuck threads and the
tolerances affecting the condition. It is possible to reuse the server definitions of stuck thread
tolerances (default), specify whether stuck threads are ignored or specify work manager specific
tolerances. For more information about stuck thread handling using work managers refer to Using
Work Managers to Optimize Scheduled Work.

Resource Management
By default, resources in the product are set to the default tolerances supplied with Oracle WebLogic
and the Oracle Database. Whilst these defaults, usually unlimited access, may be appropriate for non-
production environments, it may not be appropriate for production environments.

There are a few resource management capabilities that can be used by the product to set appropriate
resource limits:

 Work Manager Support – It is possible to setup Application Scoped Work Managers to specify
constraints to prevent out of memory or overload issues on product servers. These control client
connections to the servers to ensure optimal resource usage on the product servers. Refer to Using
Work Managers to Optimize Scheduled Work for more information about this capability.

 Database Resource Plan Support – It is possible to set and manage database resources at
various levels using the Oracle Database Resource Manager. This allows finite control at the
database level to resources. For more information about resource management, refer to Using the
Database Resource Manager to Manage Database Server Resources (Doc Id: 2067783.1) available
from My Oracle Support and Managing Resources with Oracle Database Resource Manager.

 Transaction Timeout Support – The Oracle Utilities Application Framework has a facility that
allows global and service transaction timeouts to be set to help limit resource usage. These provide
a method of policing transactions to operate within acceptable tolerance. Refer to Enabling Service
Timings for more information about this facility.

Setting these tolerances to match your business performance expectations and/or service level
agreements will depend on the traffic usage experienced for your site. Using the available monitoring
facilities can help determine tolerances.

Data is one of the commodities in


DATA MANAGEMENT BEST PRACTICES an implementation that must be
managed throughout the lifecycle.
Once a product has been put into product one of the issues that needs to be managed is the quantity
of data that accumulates over time. While storage is relatively cheap, as compared to the past,

6
These are the only two constraint types supported in the current release. Theoretically Minimum Threads Constraint is also
supported but tends not to be used by the majority of implementations.

48 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
maintenance of an optimal amount of storage is both cost effective and maintains a stable level of
performance.

Data management techniques used with products varies according to the types of data stored within
the product.

Respecting Data Diversity


One of the most important considerations for a site is to respect the diversity of the data contained in
the product where you are trying to manage the data from. Different types of data require different
types of management. Requirements for managing data are typically driven by business practices,
industry practices or even government legislation (typically driven by tax requirements).

Products are typically is divided into a number of data types and each of these data types needs to be
managed in the database for a varying length of time as the product typically has different uses for
them. In most products the data types can be categorized as follows:

Data Types

DATA TYPE DEFINITION HOW TO MANAGE

System Data Data provided in the initial This is managed by Oracle through installation,
installation of the product. This is upgrades and patches.
required for the product to
operate at the base level.

Configuration Data driving the configuration of Maintained by a subset of individuals. Kept


Data the product (e.g. Menus, rates, indefinitely and only represents small part of any
security, reference data etc). database. This data should be migrated using
Configuration Migration Assistant.

Master Data Data pertaining to customers Maintained by end users. Kept indefinitely but
such as personal records, can be driven by government legislation such as
addresses, account information, privacy laws or industry rules.
contracts, meters, crews, assets
etc).

Transaction Data Day to day data relating to any Data is still active is retained for operational
interaction or activity against the reasons. Historical data is deleted or archived
Master Data (e.g. Bills, Tasks, according to business rules or government
Meter Reads, Cases, Payments, legislation. Typically storage and retention of this
Contacts etc). data is managed using Information Lifecycle
Management (ILM) techniques and technology.
Refer to the ILM Planning Guide (Doc Id:
1682436.1) available from My Oracle Support for
more information.

The table above illustrates the various differences between the types of data and their usual data
retention rules. During an implementation and post implementation, you must be aware of the data
types and then plan the data retention rules accordingly.

Removal of Staging Records


The product uses a staging concept for most of the major interfaces. This involves a process, known
as Process X, to load the staging tables and then a base product background process is run to
validate and copy the valid staging data into the relevant main tables. When records are loaded
initially, the status of the records is set to Pending indicating they are ready to process. Once the

49 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
relevant base product background process processes them, then the status is changed to either
Completed (for valid records) or Error (for invalid records). Invalid records can be corrected using the
relevant staging online query to manually resolve the error.

This is summarized in the figure below:

Figure 13. Import Staging Process Overview

It is assumed that completed staging records are no longer required, after a period of time, as the data
they contain has been reflected have been reflected in the main tables. There is no business reason to
keep completed staging records after they have been completed for long periods of time.

Regular cleanups of the staging tables to remove completed records will have great performance
benefits on interfaces. Successful sites run the provided purge jobs to improve performance and
reduce disk space usage.

To decide when to run these purge jobs and what parameters to pass to them the following is
recommended:

 Work out with the business at the site how long they wish to retain the number of completed
records. You can stress to them that NO important data is lost in purging completed records as their
data is reflected in main tables. The value is used for the NO-OF-DAYS batch parameter passed to
the job. The value is the number of days not the number of business days (e.g. A value of 14 for
NO-OF-DAYS means 2 weeks).

 For the To Do Purge job, there are additional parameters to decide the specific To-Do type to purge
or ALL (DEL-TD-TYPE-CD and DEL-ALL-TD-SW). Work with the business to decide if this job is to
be run once (for all To Do types) or multiple times for each To-Do Type. Successful customers run
it to delete all To Do types to reduce the number of jobs to run.

 Decide the frequency based upon data growth of each table. Ideally these purge process should be
run each business day at the end of the nightly batch schedule to keep the optimum but should be
run once a week at a minimum.

Partitioning
One of the most popular data management techniques is the use of partitioning on tables. Partitioning
enables tables and indexes to be split into smaller, more manageable components.

Partitioning allows a table, index or index-organized table to be subdivided into smaller pieces. Each
piece of database object is called a partition. Each partition has its own name, and may optionally
have its own storage characteristics, such as having table compression enabled or being stored in
different tablespaces. From the perspective of a database administrator, a partitioned object has
multiple pieces which can be managed either collectively or individually. This gives the administrator
considerably flexibility in managing partitioned objects. However, from the perspective of the product,
a partitioned table is identical to a non-partitioned table; no modifications are necessary when

50 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
accessing a partitioned table using SQL.

Partitioning has known benefits:

 Divide and Conquer - With partitioning, maintenance operations can be focused on particular
portions of tables. For example, a database administrator could back up a single partition of a table,
rather than backing up the entire table. For maintenance operations across an entire database
object, it is possible to perform these operations on a per-partition basis, thus dividing the
maintenance process into more manageable chunks.

 Parallel Execution of SQL – Most databases will sense that the table is partitioned and run SQL
statements (including SELECT and INSERT statements) in multiple threads. Each of the partitions
can be thought of as an individual table and the database uses this.

 Pruning – Queries operating on one partition can run substantially faster due to reduced size of the
data to search.

 Partition Availability - Partitioned database objects provide partition independence. This


characteristic of partition independence can be an important part of a high-availability strategy. For
example, if one partition of a partitioned table is unavailable, all of the other partitions of the table
remain online and available; the product can continue to execute queries and transactions against
this partitioned table, and these database operations will run successfully if they do not need to
access the unavailable partition.

 Storage Manage Data – Partitioning is one of the techniques used in Information Lifecycle
Management (ILM) to implement savings using storage and separating data for finite management
of data. Refer to the ILM Planning Guide (Doc Id: 1682436.1) from My Oracle Support.

When using partitioning you should ensure that major processes accessing the table do not cross
partition boundaries. Crossing from one partition to another can cause slight delays as physically the
table has been separated into individual files per partition. This situation may be avoided when
designing the partitioning regime for the table.

The key to success to partitioning is recognizing which tables are candidates for partitioning and what
partitioning scheme to use. Partitioning must be planned and designed into a database to ensure that
the partitioning regime is optimal for your products.

The ideal candidates for partitioning are large tables with a small number of indexes. The benefits of
partitioning are optimal for large tables rather than applying the principles across all tables. The
minimal number of indexes is a criterion to minimize the likelihood of crossing partition boundaries in
SQL.

Once the tables are chosen to be partitioned then the next step is to decide the number of partitions to
implement. The rule of thumb is to choose the number of partitions so that any SQL that accesses the
table using the indexes will minimize crossing partition boundaries. If your product is multi-threaded
then each thread of the process needs to remain within a partition. In this case the number of
partitions should be equal to the number of threads (or a divisor). For example, if a major process runs
in 10 threads then the number of partitions could be 10, 5 or 2. Each of the numbers ensures that each
thread stays within a partition.

Once the number of partitions is chosen the next step is to decide which partition scheme you can
use. Database vendors have implemented numerous ways of dividing a table into partitions. Each of
these schemes (and sometimes combination) tells the database how to split the data into the various
partitions as well as how to access the partitions. The most common partitioning scheme used is
known as range partitioning where a range of values (index based) is used to designate the partition a

51 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
record is placed within. Refer to the partitioning documentation provided by your database vendor for
details of all the different schemes that can be used to partition your table data.

Table partitioning represents the easiest method of data management and is usually the first data
management technique used before other techniques are considered.

Note: Refer to the DBA Guides shipped with each product for product specific partitioning advice.

Compression
Note: Database level compression varies from one database version to another. In some cases, it is
included as an optional component of the database and in other cases; it is a separate option that
must be obtained from Oracle.

A technique that is starting to emerge from the database vendors is compression of data. This can be
done at a database level (global) or a table level and typically requires no changes to a product to
implement.

As the data is stored and retrieved it is compress and decompressed before passing back to the
product. As far as the product is concerned it is unaware that the data is compressed or not. This
appeals to database administrators as they can experiment with compression without the need to
involve the product developers.

Database systems have not heavily utilized compression techniques on data stored in tables. One
reason is that the trade-off between time and space for compression is not always attractive for
databases. A typical compression technique may offer space savings, but only at a cost of much
increased query time against the data. Furthermore, many of the standard techniques do not even
guarantee that data size does not increase after compression.

Over time, database vendors have addressed the trade-off by implementing unique compression
techniques. It has come to a stage where virtually no negative impact on the performance of queries
against compressed data; in fact, it may have a significant positive impact on queries accessing large
amounts of data, as well as on data management operations like backup and recovery. Each database
vendor will supply guidelines to effectively use of compression to minimize any overhead for all SQL
statements (including INSERT’s, UPDATE’s etc) and which tables are the best candidates for
compression.

Database Clustering
One of the more advanced features that have emerged as a valid data management technique is the
ability for databases to be clustered. This is a relatively new technique for data management, as most
people associate clustering with availability rather than management of data volumes.

Database clustering provides the ability for a database to be spread on more one machine but seem to
the product as a single database. The database management system manages all the synchronization
and load balancing of transactions automatically. It was designed to support the availability of the
database in case of a hardware failure in one of the nodes of the cluster.

Experience within the industry has shown that using the clustering capabilities can also improve
performance when large amounts of data are involved. Logically clustering enables the database to
access more power and spreading the workload across machines.

This technique is applicable where the volume of the data is impacting database performance. One of
the major symptoms is CPU usage on database is consistently high, no matter what tuning is
performed at the database and product level. This implies that the database is CPU bound and while

52 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
there may be an option to add more CPU’s to the server, considering clustering the data becomes a
viable alternative.

While implementing clustering has been made progressively easier with each release of the database
management system, implementing clustering must be planned using the guidelines outlined by the
database vendor. Refer to the documentation provided on clustering by your database vendor.

Backup and Recovery


One of the most critical components of the implementation and ongoing support of product at a site is
the ability to backup the data and software to ensure business continuity. Equally important is the
ability to easily restore that data if the need arises.

Typically a site will have a preferred regime and set of tools that is used to achieve a backup and
recovery of all systems that the site. When implementing product this regime and set of tools is
typically reused to cater for the products and business needs.

When considering a backup regime for product the following should be considered:

 There is nothing within product technically that warrants a particular approach to Backup and
Recovery. Most customers continue to use their existing approaches.

 There is nothing within product technically that warrants a particular backup and recovery tool. Most
customers use the native tools provide with their platforms, for cost savings, but some customer
have purchased additional infrastructure to take advantage of faster backups/recoveries or
additional features provided by such tools.

 If your site does not have a backup regime already the following can be considered default industry
practice:

 Use Hot Incremental backups on production during the business week to reduce outage times.

 Do a FULL backup (Hot or Cold) once a week at least to reduce recovery times.

 Verify backups after they are taken to reduce risk of delayed recoveries.

 On non-production, consider either the same regime as production or consider regular FULL
backups at peak periods in an implementation.
Whilst the client is a browser
there are configurations that can
CLIENT COMPUTER BEST PRACTICES optimize the client experience.

Even though product is browser based there are some practices on the client machine that affects
performance. This section outlines the practices about the client machine that have proved beneficial.

Make sure the machine meets at least the minimum specification


As part of the installation documentation for each installation of product, the minimum and
recommended hardware for the client is specified. Typically Oracle takes the following into
consideration when specifying this information:

 The minimum and recommended hardware as specified by Microsoft for the operating system used
for the client.

 A typical set of other applications running on the machine, typically Office style applications.

While all care is taken in specifying the hardware will cost in mind, experience has shown that

53 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
customers need to review the specification in light of their internal standards.

Internet Explorer Compatibility Mode Settings


The Oracle Utilities Application Framework is compatible with a wide range of versions of Internet
Explorer. For releases Oracle Utilities Application Framework V4.2.x and below the URL used for the
product must be defined on compatibility mode for backward compatibility. There are a number of
ways to define the URL as using compatibility mode:

 At runtime the user can add the URL at first login using the browser compatibility mode settings.
This is explained in an article on the Microsoft site.

 If using Internet Explorer 11 then it is possible to set the compatibility from the menu as explained in
an article on the Microsoft site.

 If sites want to implement automatic group policies to define the product URL's using compatibility
mode then refer to the Enterprise Mode article on the Microsoft site.

In Oracle Utilities Application Framework V4.3.x and above, compatibility mode is no longer required.

Popup Blockers
The browser interface to the product uses popup windows for initial searches on some transactions.
Commercial and inbuilt pop blockers may interfere with the display of these windows. It is
recommended to provide overrides for these blockers for the relevant URL’s used for the
environments used onsite.

The popup blocker may block the initial popup search windows on some transactions but may not
affect subsequent searches that are explicitly requested by the end user.

Internet Explorer Caching Settings


Note: The Oracle Utilities Application Framework supports a wide range of browsers and their
respective versions. Refer to the Installation Guide for details of browser support.

The Internet Explorer settings used must match the recommended settings as outlined in the product
Installation Guide, which includes:

 Internet Explorer cache settings should be set to Automatically NOT Every visit to EVERY page for
production use. Certain elements on the browser user interface pages are cached on the client for
performance reasons. Incorrect setting of the cache settings in Internet Explorer will increase
bandwidth usage significantly and degrade performance, as screen elements will be retrieved on
each rather than from the cache. The correct setting is shown below:

Figure 14. Example Cache Setting

 Java script must be enabled. The product framework uses javascript to implement the browser user

54 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
interface.

 HTTP 1.1 supports must be enabled. If you use a proxy to get to the server, then also check Use
HTTP 1.1 through proxy connections.

Figure 15. HTTP 1.1 Settings

Clearing Internet Explorer Cache


Between upgrades, it is advisable to manually clear the Internet Explorer cache to remove any
elements that may be still in the cache that are not applicable to the new version. This is a rare
situation but sometimes clearing the cache can ensure corrections in caching or inappropriate
elements left over from upgrades from being incorrectly displayed.

Optimal Network Card Settings


Typically the manufacturers of NIC devices provide a number of configuration settings to allow further
optimization of network transmit and receive settings. Typically the defaults provided with the card are
sufficient for the needs of the network traffic transmitted and received by the machine.

It may be further optimal to investigate whether changing the settings can improve performance at
your site (particularly the number of network buffers used). Altering the settings may improve
performance but also may adversely affect performance (due to higher CPU usage). Typically the
majority of customers use the default settings provided by the manufacturer Typical most modern networks
are fast and reliable. Optimizing
the use of the network can
improve consistency of
NETW ORK BEST PRACTICES performance.
The product ships data a network between the clients and the various components of the architecture.
This section outlines some of the practices to optimize the network elements of a configuration.

Network bandwidth
One of the most common questions asked about the product is the network footprint of the Oracle
Utilities Application Framework based product. This question is difficult to answer precisely for a
number of reasons:

 The amount of data sent up and down the network is dependent on how much change is done by an
individual user at the front end of the product. Only the elements changed by the end user are
transmitted back to the server. The more the user changes the more the data is transmitted. Given
the numerous possible permutations and combinations for data changes at any given time, this can
be hard to estimate.

 The Oracle Utilities Application Framework supports partial object faulting. This means the
framework only sends data to the client that is being displayed. In screen with more than one tab,
the framework only sends the data for the tabs that are accessed by the end user. This means only
part of the overall object required by the screen. Most users tend to operate on a small number of
tabs but this can vary from transaction to transaction.

55 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 All transmission between the client and server are compressed using HTTP 1.1 natively supported
compression. This can reduce the actual size of the data transmission considerably depending on
the content of the changes.

 Screen data is cached on the client machine that can be reused. The product takes advantage of
the caching facilities in the HTTP 1.1 protocol and the browser caching functionality. For example,
screen definitions and graphics are stored on the client machine to reduce network footprint. Upon
every transmission of a screen element the data in the cache is tagged with an expiry date to
indicate the life of the element in the cache. Use of client side caching can reduce the network traffic
considerably with some customers reporting up to 90% reduction in network traffic when this
caching is enabled.

It is possible to track network bandwidth using a log analyzer against the W3C standard access.log
produced by your Web Application Server. Refer to the Performance Troubleshooting Guideline Series
(Doc Id: 560382.1) whitepapers available from My Oracle Support for more information about this log.

Ensure legitimate Network Traffic


One of the major factors on performance is the amount of legitimate traffic on the network. The traffic
to and from product shares the bandwidth with all other traffic on the network. If there is any network
congestion than all transactions from all network-based applications will be adversely affected.

Some customer sites have found that traffic that is not legitimate can adversely affect network
performance. Traffic that is considered not legitimate includes:

 Traffic generated from viruses and Trojans – There are a plentiful number of viruses and Trojans
in the general Internet network that can cause bandwidth issues. Most sites have regular virus
protection to minimize the impact to your network but not all. While it is not a requirement within
product to have such protection, the industry in general recognizes the need for such protection.

 Unauthorized large transfers – Large transfers of data can adversely affect performance as it can
soak up bandwidth if the transfer is not configured correctly. There have been instances of large
FTP transfers slowing down traffic on lower bandwidth networks.

Ensuring that only legitimate traffic is on a network can provide greater bandwidth for all applications
(including product) and improve consistency.

Regularly check network latency


In a network, latency, a synonym for delay, is an expression of how much time it takes for a packet of
data to get from one designated point to another. In some usages, latency is measured by sending a
packet that is returned to the sender and the round-trip time is considered the latency. The greatest
impact on performance is inconsistency latency.

The latency assumption seems to be that data should be transmitted instantly between one point and
another (that is, with no delay at all). The contributors to network latency include:

 Propagation - This is simply the time it takes for a packet to travel between one place and another
at the speed of light.

 Transmission - The medium itself (whether optical fiber, wireless, or some other) introduces some
delay. The size of the packet introduces delay in a round trip since a larger packet will take longer to
receive and return than a short one.

 Router and other processing - Each gateway node takes time to examine and possibly change
the header in a packet (for example, changing the hop count in the time-to-live field). This is a

56 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
common cause of network latency.

 Other computer and storage delays - Within networks at each end of the journey, a packet may
be subject to storage and hard disk access delays at intermediate devices such as switches and
bridges.

Minimizing latency or latest ensuring consistent latency is the goal of most of the product sites. A
discussion of latency and how to measure it is contained in the whitepaper Performance
Troubleshooting Guideline Series (Doc Id: 560382.1) whitepapers available from My Oracle Support.

General Networking Guidelines


One of the common areas of configuration issues is setting up incorrect networking for the product. In
the past, networking was simple with host resolution being straightforward. Over the last few years,
more complex domain resolution technologies, proxies, firewalls, demarcation and virtualization have
complicated networking to the stage that simple configuration will not always guarantee successful
networking. To help minimize issues with the product the following guidelines need to be followed:

The product infrastructure such as the Java EE Web Application Server and Java itself needs access
to the hosts and port named in the configuration. When specifying the host name in configuration files,
ensure the host housing that component can connect (directly or via name resolution) to the host
specified (even it is the local host). The table below outlines what each component needs access to:

Networking Components

COMPONENT COMMENTS

Online Needs access to Database to load cache at startup time and for transaction
processing. Can use JDBC Data Sources or Universal Connection Pool for
connection. Database location should be configured using SQL*Net via Oracle Client.

Mobile Needs access to Database for transaction processing. Can use JDBC Data Sources
or Universal Connection Pool for connection. Database location should be configured
using SQL*Net via Oracle Client.

Web Services Needs access to Database for transaction processing. Can use JDBC Data Sources
or Universal Connection Pool for connection. Database location should be configured
using SQL*Net via Oracle Client.

Batch Needs access to Database for transaction processing. For On-Premise applications
uses UCP for direct connection. Database location should be configured using
SQL*Net via Oracle Client.

 Ensure ports are available and unique for the host as defined in your firewall. Inability to connect to
ports will result in a failed startup.

 If there are issues then consider using localhost as your hostname in the configuration for the
relevant components (mainly Web Application Server and Business Application Server). If using
localhost consider installing the loopback adapter for your operating system. Use of the loopback
adapter and localhost is highly recommended if using dynamic server addresses and/or dynamic
7
server names (such in virtualization) .

7
This is recommended for Oracle Virtualization customers.

57 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 When using multiple network connections, ensure the product uses the correct network
connection(s) to operate.

 If using CLUSTERED mode in batch, ensure that the multicast protocol is enabled, if desired; the
configured multicast address and port are available through your firewall and networking
Oracle Utilities Application
configuration.
Framework based products are
web based applications that are
housed in a Java EE container.

WEB APPLICATION SERVER BEST PRACTICES


The Web Application Server is used by product to serve the pages to the client and contains a control
data cache. There are a number of practices that sites find useful for maintaining the health of the
Web Application Server.

Make sure that the access.log is being created


The access.log contains useful information that can be used for tracking bandwidth and usage
patterns to make changes to configuration.

One of the key log files for traffic analysis is the access.log. This is a log generated by every hit on the
system from the end users. Every element of the screen is logged, asynchronously, including the time
and userid. This log must be configured/enabled to be generated by the configuration. Refer to the
Web Application servers documentation on how to enable this log to be generated.

The log is generated in W3C common log format and can be analyzed by third party log analyzers for
further analysis. A full description of the log, it usefulness and the log analyzers that can read the log
are documented in the whitepaper Performance Troubleshooting Guideline Series (Doc Id: 560382.1)
whitepapers available from My Oracle Support.

Some customers use the log for various purposes:

 It is possible to track errors and trends from the log using the log analyzers.

 It is possible to parse the log at a low level and determine the number of concurrent users and the
users that have used the system (and interestingly conversely who has NOT used the system).

 It is possible to track flows of individual sessions, known as click streaming, to track the screens and
data used for the screens.

 It is possible to determine the criteria used by users for searches. This is useful for detecting
wildcard searching.

This log is useful but it is large so needs to be managed.

Examine Memory Footprint


One of the common experiences for ALL the Java EE Web Application Servers that product runs upon
is that there seems to be a Java Virtual Machine (JVM) limit on exactly how many concurrent online
users a server will support. Typically the experience has been that between 300-400 concurrent users
are served by each instance of a JVM.

There are a number of techniques that are available to maximize this:

 Increasing the java memory parameters used for the JVM – This can be a configuration setting
change or a script change. Typically customers change the default settings to either 512MB, 1GB or

58 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
2GB per JVM.

Note: In Oracle Utilities Application Framework V4.0 and above the JVM Options can be configured
using parameters. Refer to the Server Administration Guide provided with your product for more
details.

 Creating additional servers within the instance to cater for the load - A server entry for each
new server is setup in the same Oracle WebLogic domain. The port number can be the same (if the
server is housed on a separate machine, known as clustering) or a different port number (i.e.
managed servers). A proxy is required to have a common connection point and to implement load
balancing. The memory footprint will be the same size for each server.

Turn off Debug


One of the development features of product is the ability to output useful debugging information as
part of the running of the application. While this information is useful for development environments it
is not useful for production or other performance sensitive environments.

Most customers change the debug setting to false to disable global debug information. It is possible to
debug individual transactions using the interactive debug facility.

Note: This requires the Application Descriptors for all applications to be updated.

Load balancers
Oracle Utilities product customers who have more than one Application Server (physical or logical)
must use a load balancer to route the traffic evenly across the available servers. This load balancer
can be either software based, such as a web server with the appropriate plugin from the Application
Server vendor, or a hardware based load balancer (such as BigIp or other Layer 7 switches).
Experience has shown that customers with a large number of users (typically greater than 1500) tend
to use hardware load balancers and smaller customers use software based load balancers.

Using load balancers with product may not guarantee that load is evenly distributed, as the
transactions do not have a consistent resource load factor. The resource load factor for any product
depends on the transaction type and the data used in that transaction. For example, Search
transactions are different from maintenance transactions and resource usage of any search is
dependent on the criteria used. Two executions of the same search will have different response and
resource usage profiles. Factored on top of that is the fact that the load on a server is a summation of
the all the transactions sent to it and that transactions vary from second to second, minute to minute,
hour to hour etc. The best you can do is

When installing a load balancer there are a number of algorithms for load balancing offered:

Load Balancing Algorithms

ALGORITHM PROCESSING COMMENTS

Round Robin Traffic is routed to each server on a This is the most common used by
rotating basis. implementations and the
recommended setting.

Random Traffic is routed randomly to the Not commonly used but may be used
servers. if traffic is random enough.

Weighted Round-Robin Variation on Round Robin but allows Not generally used by
Allocation support for clusters where all servers

59 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
are not the same size. implementations.

IP Address Traffic is routed using client IP Has been used by customers but
address as the identifier where found that has limitations if used with
servers are assigned IP address virtual servers such as Terminal
ranges. services or Citrix.

Load Load factors of transactions are Not used with product, as most load
measured and used to determine factors are inconsistent across
which server is best suited. transaction invocations.

Typically most customers use Round Robin as it is simple and given load is unpredictable can yield
the best results. Most customers understand that on some periods of time the load will not be
balanced but on average the load is relatively balanced. Remember that each transaction time is a
function of how much data is changed

If using load balancing the following additional advice is applicable:

 Ensure that the load balancer does not interfere with Internet Explorer caching. This may result in a
low cache hit rate and increase bandwidth used.

 Ensure that the load balancer supports HTTP 1.1 headers to support compression.

 Ensure that the balancer supports Passive and Active Cookie persistence for session cookies. The
Web Application Server uses session cookies, for passing security credentials between the client
and the server. The load balancer must not compromise this facility.

 Ensure that the load balancer supports SSL persistence, if SSL is used, to ensure that encryption
and decryption are not compromised.

Preload or Not?
One of the startup features of product V1.5, and above, is the preloading of pages to save time. This
preloading process dynamically rebuilds the screen definitions from the XML meta data on startup.
While this setting (by default) enables the startup to pre-build them (instead of on first invocation) the
startup of the Web Application Server is delayed while the preload process is executing. The startup of
the server is delayed until the last of the screens is preloaded.

While the preloading of individual screens is very quick (measured in milliseconds) building all screens
(1000+) can cause significant delays to initial availability AFTER a restart. It is possible to influence the
amount of preloading using two parameters in the Web Application Descriptor called:

 preloadAllPages – This parameter affects how much preloading is taking place if it is preloaded.
A value of true preloads every screen for product. A value of false preloads screens off the Main
menu only (the screens the end users will be using).

 disablePreload – This parameter controls whether preload is performed or not at all. This
parameter affectively overrides the preloadAllPages parameter.

The effect of changing the parameters is outlined in the following table:

Preloading

PRELOADALLPAGES DISABLEPRELOAD EFFECTS

60 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
true true Pages are not preloaded at all. First invocation of the
screen by the first user in that screen loads the screen for
all users. Can cause slight delay in initial screen load for a
single user but application startup is quicker.

true false All pages are preloaded including administration and


utilities menu. This setting is not recommended for
production as it delays Web Application Server startup
unnecessarily.

false true Pages are not preloaded at all. First invocation of the
screen by the first user in that screen loads the screen for
all users. Can cause slight delay in initial screen load for a
single initial user but application startup is quicker.

false false Default. Pages on the Main menu are preloaded. This
delays the startup of each managed server but ensures
screens are loaded quicker for ALL users.

Changing of this parameter affects availability rather than performance but should be considered if
availability is critical or you are not using all the screens in product.

It is recommended that the following settings be implemented if you do not use the entire product or
you want startup to be quicker:

preloadAllPages false

disablePreload true

Note: This requires the Application Descriptors for all applications to be updated.

Hardware or software proxy


You will need to proxy connections if you use clustering or a number of managed servers. The choice
of software or hardware proxy is site specific. Large customers prefer hardware proxies and smaller
ones software proxies.

If the implementation uses multiple servers then a proxy is needed to group the servers into a cluster
or managed configuration for load balancing purposes. There are two alternatives for such a proxy:

 Software – Each of the Web Application Server's supported by product provide a plugin to use a
HTTP server such as Oracle Traffic Director, Apache, Oracle HTTP server, Netscape or IIS as a
proxy. Typically the plugin is installed within the HTTP server and configured to define the server
address and scheme of load balancing.

 Hardware – Increasingly the network router manufacturers are making hardware products that act
as network proxies or load balancers (known as Layer 7 load balancers). Hardware such as BigIp,
WebSwitch, NetScaler etc are increasingly performing load balancing within intelligent hardware. In
this case, you simply configure the servers and ports to a virtual address in the hardware and the
load balancing scheme to use.

Customers with multiple servers are either using a hardware or software proxy. The larger scale
customers favoring hardware based solutions. The only thing to remember with a proxy is to make

61 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
sure the following are taken into account:

 The proxy server must support the browser caching scheme and not disable it or adversely affect its
operation. This will increase network through put.

 The proxy server must support session cookies. It must be configured to support the passing and
processing of session cookies as they are used for security tokens in product. Failure of this point
will result in the security dialog being displayed before EVERY screen.

What is the number of Web Application instances do I need?


One of the most common questions for an implementation is how many Web Application Servers do I
need to support the number of users that we have planned to be attached to the product for
production? The answer to this depends on the JVM you are using and its limitations.

Tests and experience has shown that the Java Virtual Machine has an internal limitation on the
number of threads that can be safely supported for transactions. This is not a sever limitation but
represents the number of active transactions (i.e. Users) that are supported on a Web Application
Server at any time.

Tests have shown that this number varies between 300 – 500 users on a single Web Application
Server JVM instance. The number varies according to the JVM version used and the vendor that
supplies the JVM. This number represents maximum number of simultaneous active users hitting the
Web Application Server at peak time.

The easiest method for finally determining the number of instances this will become is to divide the
number of users expected on the system, at worst case, by 300 and then round up to the next integer.
For example, to support 750 users then you can specify 3 instances, to support 500 then you specify 2
instances etc. This method assumes worst case. Regular monitoring of the actual number of
connections will reveal whether this needs to be altered.

Determining Active Users


One of most important metrics to determine is the number of active concurrent users at both the
lowest ebb and the peak to determine the capacity needed at different times in the business. The
definition of active users is as follows:

 An active user is any individual user or thread that is actively using any component of the
architecture from the client to the database. This means they are performing active work and are
consuming resources in the architecture.

 This number does not include users that are idle whilst in the system, for example, sitting on a
screen in think time or logged in but not actively using the system. Oracle Utilities Application
Framework based products are stateless which means these users are not consuming resources on
the architecture.

 The number of registered or named users does not correspond to the number of active users. While
the number of registered users is useful for licensing or general planning it is not a metric to use for
capacity.

The reason that the number of active users is important is that each user is consuming some server
resources. These resources need to be sufficient to meet the load required to meet the demands of
these active users.

There are two metrics that need to be determined for active users:

62 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 Minimum Active Users – These are the number of active users at the times where the system is at
minimum capacity. This represents the absolute minimum you need to support in terms of system
resources. Typical times for these activities would be the overnight call center traffic or the start of
the business day where you have minimal traffic.

 Maximum Active Users – These are the number of active users at peak times in your business.
This is a measurement done when the busiest time has happened or is happening to determine the
maximum resources needed to be provided.

Now, the discussion above has talked about users but this really applies to Web Services Clients
accessing the Inbound Web Services, REST clients accessing the REST capability, Mobile client
accessing the Mobile Server and batch threads concurrently running on the threadpools. So there are
different ways of capturing this information over time:

Capturing Active User Information

TIER METRIC WAYS OF CAPTURING THIS INFORMATION

Online Users Use the Oracle WebLogic MBean ThreadPoolRuntimeMBean with


the attribute ExecuteThreads at peak and non-peak times. The
total across the domain would represent the minimum.

If you want to separate to individual servers then use the Oracle


WebLogic MBean ServerChannelRuntimeMBean with the
attribute ConnectionsCount at peak and non-peak times.

Batch Threads Use the sum of the NumberOfMembers attribute from the Batch JMX
API as documented in the Server Administration Guide.

Web Services Clients This can use the same method as Online (especially for REST which
are embedded in the servlet). This should represent all the
connections from all the integrations at any time.

Mobile Clients This can use the same method as Online (especially for REST which
are embedded in the servlet).

Database Database If using Oracle WebLogic JDBC Data Sources, then use the Oracle
Connections WebLogic MBean JDBCDataSourceRuntimeMBean with the
CurrCapacity + WaitingForConnectionCurrentCount
attributes.
If using Universal Connection Pool8, then use the UCP Mbean
UniversalConnectionPoolMBean with the (maxPoolSize –
availableConnectionsCount) + PendingRequestsCount
attributes.

Note: You might notice that the database active connections are actually calculations. This is due to
the fact that the metrics capture the capacity within a limit and need to take into account when the limit
is reached and has waiting requests.

Once the data is collected it is recommended to be used for the following:

 Connection Pool Sizes – The connection pools should be sized using the minimum values

8
This applied to Universal Connection Pool Version 12.2 and above.

63 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
experienced and with the maximum values with some tolerances for growth.

 Number of Servers to setup – For each channel, determine the number of servers based upon the
numbers and the capacity on each server. Typically at a minimum of two servers should be setup for
the minimum high availability solutions. Refer to Oracle Maximum Availability Architecture for more
advice.

Defining external LDAP to the Web Application Server


Note: A detailed discussion of LDAP integration is available in the LDAP Integration (Doc Id:
774783.1) whitepaper available from My Oracle Support.

Lightweight Directory Authentication Protocol (LDAP) is promoted as a means to leverage an


organizational directory as a principal registry for product user authentication. Therefore as part of the
security setup of product you may need to integrate to an onsite LDAP security repository. This is
supported directly by the Web Application server software and product does not require additional
configuration.

Each of the Web Application Server vendors has specific instructions for integrating LDAP but the
same process is followed:

 Determine LDAP Query - The LDAP query to find the users is required to be determined. Even
though LDAP is a standard protocol determined by the Internet Engineering Task Force (IETF) the
repository structure itself will vary from vendor to vendor and even the same vendors repository
structure will vary from customer to customer as it can be altered to suit the business model. This is
the hardest part of the process, as the query needs to be correct else it will not return the right
records. It is akin to submitting the wrong SQL statement. There are tools, like adfind (for
Microsoft ADS for example) to help you with this process.

 Define LDAP settings to Web Application Server - Input the query and credentials to access the
LDAP repository. This will vary between Web Application servers but basically you need to define
the following:

– The location (host) of the LDAP server(s)

– The port numbers for the LDAP server(s) (usually 389)

– The credentials used to read the LDAP server(s) (userid/password)

– The LDAP query to get the users (and sometimes groups for some Web Application Servers).

– (Optional) Cache settings to save data retrieved from the LDAP server for performance reasons.

Note: Ensure that the LDAP you have specified contains a definition of the administration account you
use to start/stop/administrate product, else if you have made a mistake it may not be possible to
restart the Web Application Server. To reduce the risk of this happening, some sites define two
repositories, one to the LDAP server and one to the default security repository provided by the Web
Application Server vendor as a precaution. The latter is used to house the administration accounts you
do not want to store in the company LDAP.

 Restart to reflect changes - Restart the Web Application Server.

For more information refer to Configuring LDAP Providers for Oracle WebLogic.

Appropriate use of AppViewer

64 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Note: The AppViewer in Oracle Utilities Application Framework V4.3.0.6.0 and above is dynamic and
does not require generation.

The AppViewer is a component of product that displays meta-data in a more useable format. In past
versions of the product, it was preloaded with every product environment. Typically the information is
used for design and development work only.

To make the best use of the AppViewer the following advice is offered:

 The AppViewer is provided blank intentionally. It must be primed using a predefined set of Batch
jobs. This will take data from the meta-data (including ANY customizations) and generate it. You will
need to run the jobs regularly if you update the meta-data regularly and want the information
reflected in the Application Viewer.

AppViewer Batch Control

BATCH CONTROL USAGE

F1-AVALG Generate AppViewer XML file(s) for Algorithm data (includes javadocs and
groovydocs).

F1-AVBT Generate AppViewer XML file(s) for Batch Control. This is useful for run book
information.

F1-AVMO Generate AppViewer XML file(s) for Maintenance Object data

F1-AVTBL Generate AppViewer XML file(s) for Table/Field data

F1-AVTD Generate AppViewer XML file(s) for To Do Type

 The introduction of the batch jobs, means you can decide which information is important for your site
to display in the AppViewer. For example, if you wish not to have To Do Types documented then
you can omit that information by not running that job. If you wish to populate ALL the information
then you can use the genappvieweritem.sh command.

 Consider only populating the information in any design and development environments to save disk
space. The AppViewer can extend to a number of gigabytes if fully loaded which can slow
deployment times significantly as deployment validates each file.

Fine Grained JVM Options


The utilities provided with the Oracle Utilities Application Framework invoke a Java command line for
the Web Application Server, Business Application Server and batch components of the architecture.
Whilst the memory arguments and java options are standardized in the utilities, some sites have found
that changing the defaults provided allows for improvements in performance and stability.

In the past releases of the Oracle Utilities Application Framework prior to V4 this meant manually
changing the scripts provided as utilities, with the product, which can be overridden in upgrades. In
Oracle Utilities Application Framework V4 and above it is possible to set the memory requirements
and additional JVM options from configuration parameters. The following table lists the settings that
can be altered using the configureEnv.sh utility:

JVM Memory Options

SETTING COMPONENT USAGE

65 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
ANT_ADDITIONAL_OPT ant Additional java options for the ANT make tool.

ANT_OPT_MAX ant Maximum memory size for ANT make tool.

ANT_OPT_MIN ant Minimum memory size for ANT make tool.

BATCH_MEMORY_ADDITIONAL_OPT Batch Additional java options for Batch Threadpool


workers.

BATCH_MEMORY_OPT_MAX Batch Maximum memory for Batch Threadpool


workers.

BATCH_MEMORY_OPT_MAXPERMSIZE Batch Maximum permanent generation size for Batch


Threadpool workers.

BATCH_MEMORY_OPT_MIN Batch Minimum memory for Batch Threadpool


workers.

WEB_ADDITIONAL_OPT Web Additional java options for Java EE Web


Application Server.

WEB_MEMORY_OPT_MAX Web Maximum memory for Java EE Web


Application Server.

WEB_MEMORY_OPT_MAXPERMSIZE Web Maximum permanent generation size for Java


EE Web Application Server.

WEB_MEMORY_OPT_MIN Web Minimum memory for Java EE Web Application


Server.

Note: The *MAXPERMSIZE parameters are not required for Java 7 or above.

The values for these settings will vary according to your site needs and the JVM vendor used at your
site. The following guidelines should be considered when changing these values:

 The additional java options supported by each JVM vendor, is slightly different to take advantage of
specific platform requirements by the JVM. Refer to the JVM options documentation provided with
your JVM. For Oracle based JVM's refer to Oracle JVM VM Options.

 Ensure any options specified are within the constraints and restrictions of the JVM. For example,
setting invalid values may result in failure or unexpected behavior.

 Do not specify the –Xms, -Xmx or –XX:PermSize parameters as additional options as these are
have dedicated settings already.

The following common settings have been used by customers:

Additional JVM Options

OPTION USAGE

-XX:+UseParallelGC Use Parallel Garbage Collection

-XX:+MaxFDLimit Bump the number of file descriptors to maximum (Solaris


Only)

66 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
-XX:+UseGCOverheadLimit Use a policy that limits the proportion of the VM's time that is
spent in Garbage Collection before an OutOfMemory error is
thrown.

-XX:+UseLargePages Use large page memory. See Large Memory Pages for more
details.

-XX:-HeapDumpOnOutOfMemoryError Dump heap to file when java.lang.OutOfMemoryError is


thrown. Commonly used by Oracle Support if necessary.

-XX:HeapDumpPath=<path and name> Location and name of dump file. Commonly used by Oracle
Support if necessary.

-XX:+PrintGC Print message when garbage collection occurs

-XX:+UnlockCommercialFeatures Used with FlightRrecorder to enable Java Mission Control

-XX:+FlightRecorder Allow Flight Recorder to be used on this JVM for Java


Mission Control

Customizing the server context


In past versions of the Oracle Utilities Application Framework the URL used by the product was fixed
on certain platforms. The URL included the context spl or splapp depending on the platform and the
Java EE Web Application server used. In Oracle Utilities Application Framework V4 and above, it is
now possible to specify a custom context as part of the installation process. This allows the following
URL to be used:

https://<host>:<port>/<context>/cis.jsp

as the default URL with the following settings:

<host> The hostname for the Web Application Server.

<port> The port number allocated to WL_PORT at installation time. To avoid the port
number a value of 80 may be specified. This value can only be specified once per
Web Application Server machine.

<context> The server context that can be set using WEB_CONTEXT_ROOT at installation time.
This value must be valid for the Java EE Web Application Server and is restricted
to a single value text value without any embedded blanks or special characters
(such as the directory character).

Clustering or Managed?
One of the decisions that must be made, when dealing with multiple web application servers, is to
whether the servers will be clustered or managed. The attributes of each style are outlined below:

 Clustered – A cluster is a group of servers running a Web application server simultaneously,


appearing to the users as if it were a single server (usually managed by a separate administration
server). The advantages of using a cluster are that you can manage the servers as a group and also
the servers communicate to each other to monitor availability. Clusters can load balance within
themselves as they are in constant communication with each other. The disadvantages are that
there is an overhead in communication (usually each server uses multicast to communicate to the
other servers in a cluster) and each server must use a different IP address and port number. This

67 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
means clusters can only operate on one machine per server. The figure below summarizes a
cluster:

Figure 16. Example clustered server architecture

 Managed – A managed set of Web Application servers that are independent of each other. They
can be housed on a single machine or multiple machines and can be housed on machines of
differing size. The advantage of managed servers is that each server can be targeted for specific
user groups and can be managed independently. There is no additional communication between the
servers. A separate administration server can manage the servers but that role can be taken by one
of the managed servers if desired. The disadvantages are that the load balancing
software/hardware housed between the users and the managed servers performs the load
balancing and that deployment must be performed individually. The figure below summarizes
managed servers:

Figure 17. Example managed server architecture

There are no clear winners between clustering and managed Web Application Servers as the main
factors in the decision are:

68 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 Amount of hardware – Clustering requires a hardware server per server as the cluster tends to
need different hosts for each server. Sites where a small number of servers are deployed cannot
use clustering.

 Maintenance Effort – Clustering can reduce maintenance overhead if there are a large number of
servers involved. Managed servers require individual maintenance.

 Tolerance for multi-casting – Some sites ban multi-casting as it constitutes can be perceived as
an unacceptable overhead on the network. Deploying a private network between the servers can
minimize this, though this is more expensive.

 Flexibility – Many sites use managed due to its flexibility in routing particular traffic to particular
servers. For example, setting up specific servers for non-call center traffic (e.g. Web Services,
interfaces, depots).

Whether your site uses clustering or managed servers does not factor into high availability solutions as
customers have deployed high availability solutions using either technique.

Note: For more information refer to Administering Clusters for Oracle WebLogic Server for information
on clustering.

Monitoring and Managing the Web Application Server using JMX


In Oracle Utilities Application Framework Version 4.0 it is possible to enable JMX performance
statistics to allow collection, management and monitoring of JVM information for the Web Application
Server. For backward compatibility, the JMX enabled facilities are disabled by default. To use this
facility you must execute the configureEnv.sh utility with the –a option (Advanced Menu) and
specify the following settings:

JMX Options

OPTION USAGE

JMX Enablement System Userid Userid used for logging onto JMX Mbeans

JMX Enablement System Password Password to be used for JMX Enablement System Userid

RMI Port for JMX Web Port number to allocate to the JMX for the Web Application
Server

This information is added to the spl.properties file in the $SPLEBASE/etc/conf/root/WEB-


INF/classes subdirectory for the environment, for the Web Application Server. An example of the
applicable settings is shown below:

spl.runtime.management.rmi.port=..

spl.runtime.management.connector.url.default=service:jmx:rmi:///jndi/rmi://
hostname:../oracle/ouaf/webAppConnector

jmx.remote.x.password.file=scripts/ouaf.jmx.password.file

jmx.remote.x.access.file=scripts/ouaf.jmx.access.file

ouaf.jmx.com.splwg.base.support.management.mbean.JVMInfo=enabled

ouaf.jmx.com.splwg.base.web.mbeans.FlushBean=enabled

69 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
The following settings are important to the JMX monitor:

 The spl.runtime.management.connector.url.default is the JMX url to be used in the


JMX console or JMX browser.

 The jmx.remote.x.password.file and jmx.remote.x.access.file are the default


security setup for the JMX. These are for basic security setup. For more information about the files
and alternative security setups refer to Monitoring and Management Using JMX Technology.

 The ouaf.jmx.* settings enable individual beans at startup time. These may be enabled at
runtime.

Once the Web Application Server component is started; the JMX Mbeans defined in this configuration
are started and a JSR160 compliant JMX console or JMX browser can be used to connect to the JMX
Mbeans. The remote URL and credentials are provided as configured above.

Within the JMX console or JMX browser there are a number of specific facilities that are available:

 It is possible to manage the data within the Web Application Server cache from JMX. In past
releases of Oracle Utilities Application Framework this was possible using utility URLS's which
required the IT group to logon to the product to issue commands. This is still possible but can be
replaced with JMX console commands. This is controlled by the FlushBean Mbean.

 It is possible to get environmental information about the Web Application Server Java Virtual
Machine (JVM) for support purposes. . In past releases of Oracle Utilities Application Framework
this was possible using utility URLS's which required the IT group to logon to the product to issue
commands. This is still possible but can be replaced with JMX console commands. This is controlled
by the JVMInfo Mbean.

 It is possible to get internal JVM information about the Web Application Server using the
JVMSystem Mbean. This is an extension of the base Java MXBeans (Package
java.lang.management). By default these are disabled and can be seen by executing the
enableJVMSystemBeans operation from the BaseMasterBean. When enabled the following
additional areas can be monitored via JMX for the Web Application Server:

– Class Loading statistics

– Memory statistics

– Operating System statistics (statistics vary by platform).

– JVM Runtime information (additional to JVMInfo)

– Thread statistics – Statistics on individual java threads.

Note: No confirmation (i.e. Are You Sure?) dialog is provided with most JMX consoles or JMX browser
so care should be taken when issuing commands.

Password Management solution for Oracle WebLogic


One of the common requests for an enhancement is the ability for users to change their application
passwords from within the product. Typically password management is scoped outside the product's
domain as it is considered infrastructure. This does not mean the product need not provide the
interface to change the password, but it is the infrastructure's responsibility to provide a mechanism to
change the passwords used in the security store.

70 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
The issue becomes then if the infrastructure provides such an interface for the product to hook into.
There are a number of patterns in this area:

 Customers implement an identity management solution to manage the passwords, expiry and rules.
In this case the implementation needs to interface to the identity management solution by calling the
appropriate facilities in the identity management solution around passwords. Of course, the Java EE
Web Application Server used is then interfaced into the identity management solution or the related
security store to provide the authentication mechanism.

 Customers link the security store for authentication directly to the security configuration of the Java
EE Web Application Server. In this case, the Java EE Web Application Server provides the interface
to the password change facility.

In the latter case, if you are a customer using Oracle WebLogic, there is an example JSP available
under Password Change Sample to allow an application to change the passwords, irrespective of the
security used. This example can be altered to suit your sites standards and linked to the product as a
custom JSP via a Navigation key to link to the appropriate menu.

Note: The code provided in the article above is provided as-is and is only a sample.

Error configuring Oracle WebLogic credentials


When the product is installed with Oracle WebLogic, the security repository used by the environment
is populated with an initial Administration System userid (usually system or weblogic) to be used to
create other credentials post installation. To use this user within Oracle WebLogic it must encrypted
(along with the password) before it can be used. The installer calls a java class within Oracle
WebLogic to encrypt this userid and password, but if the path to Oracle WebLogic is incorrect,
specified in the WEB_SERVER_HOME (or WL_HOME ) parameter the installer will return this error when
attempting to encrypt the user:

…<crit> Error occured while running java -


Dweblogic.RootDirectory=…/splapp weblogic.security.Encrypt :

Output is Exception in thread "main" java.lang.NoClassDefFoundError:


weblogic/security/Encrypt

Caused by: java.lang.ClassNotFoundException: weblogic.security.Encrypt

Could not find the main class: weblogic.security.Encrypt. Program will


exit.

To fix this issue set the WEB_SERVER_HOME using the configureEnv.sh –i utility (or set WL_HOME)
to access the appropriate security encryption classes.

Corrupted SPLApp.war
By default, the product installer uses archive mode for the product deployment. When using archive
mode the product utilities build the product into a set of Java EE WAR and EAR files prior to
deployment.

The WAR and EAR build is performed by the initialSetup.sh utility. Refer to the Server
Administration Guides for the product for a detailed description of the options and operations
supported by this utility.

71 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
If, for any reason, the WAR or EAR files are not built completely, and are therefore are corrupted, then
the product start may abort. This can manifest in a number of error messages depending on the nature
of the corruption:

… <info> ERROR: …/splapp/applications/SPLApp.war war file does not exist.


Problem with the environment. Exiting.

or

weblogic.management.DeploymentException: Unexpected end of ZLIB input


stream

at
weblogic.application.internal.EarDeploymentFactory.findOrCreateComponentMBe
ans(EarDeploymentFactory.java:189)

To resolve this issue then rerun the initialSetup.sh utility to recompile the WAR and EAR files.

Web Application Server Logs


In the Server Administration Guide for your product the product specific logs are outlined including the
formats and location. Given the product runs within a Java EE Web Application Server, that server
also has a set of configuration files that can be used for diagnostic information.

The table below outlines the default set of Java EE Web Application Server log files:

Log Files

LOG FILE USAGE LOG TYPE

Server Log Log for each server in the domain containing Oracle WebLogic
messages

Domain Log Rollup of server messages at the domain level. Oracle WebLogic
Any error that occurs in any server in the
domain will send message to this log as well as
the server log.

Access Log HTTP Access Log (also known as Apache Log) Oracle WebLogic
which holds accesses from user interface.

Audit Log (optional) This is an additional log provided by the Audit Oracle WebLogic
Provider in Oracle WebLogic to log security
events such as logon, password violations,
logoff etc.

Web Application Log (spl_web) Captures product web application messages for Product Log
user interface

Business Application Log Captures product business application server Product Log
(spl_service) messages

Inbound Web Services (spl_iws Captures product web services messages Product Log
or (spl_xai)

Initial Setup (initialSetup) Captures template and configuration generation Product Log

72 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
messages

Configuration (configureEnv) Captures configuration utility messages Product Log

Refer to the Oracle WebLogic documentation and Server Administration Guide for details of the logs,
location and format.

Enabling additional Java options


It is possible to set specialist additional options on the java command that is used to run the
component for development or debug purposes. For Oracle Utilities Application Framework V4.0 and
above, you can achieve this by specify relevant options the values on the following settings:

Additional Option Parameters

TIER PARAMETER

Web/Business Application Server WEB_ADDITIONAL_OPT

Batch BATCH_MEMORY_ADDITIONAL_OPT

You specify the values as you would on the java command line as outlined by your Java vendor. For
example, for Oracle WebLogic/Oracle Java customers:

 It is possible to enable java debug (using jdb) to debug your java code using the –Xrunjdwp option.

 It is possible to enable verbose class loading into the log files using the –verbose java option.

Note: The combination of java options that can be used must be valid for the JVM version and vendor
used.

Note: For customers on previous versions of the Oracle Utilities Application Framework, these setting
must be manually set in the scripts used to initiate the JVM. Please note, changes to any base scripts
may be overridden when initialSetup.sh is executed.

Using setUserOverrides.sh for Oracle WebLogic 12c


In Oracle WebLogic 12c, there is a feature where if the script setUserOverrides.sh exists in
$DOMAIN_HOME/bin then this script will be called at startup by nodeManager or the Administration
server at startup time. This script is user defined and is useful for the following situations:

 The SPLEBASE setting can be set in this script to allow configuration files to get used from the file
system rather than within the EAR/WAR file at runtime. For example:

SPLEBASE="/u01/utilities/product"

export SPLEBASE

 The java custom memory and additional parameters can be added to the server startup using the
USER_MEM_ARGS environment setting. For example:

USER_MEM_ARGS="-Xms1024m -Xmx4096m -XX:CompileThreshold=8000 -


XX:PermSize=500m -
Djava.security.auth.login.config=/u01/utilities/product/splapp/config/java.
login.config -XX:+UnlockCommercialFeatures -XX:+FlightRecorder"

73 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
export USER_MEM_ARGS

 The Library LD_LIBRARY_PATH path can be set (if needed). For example:

LD_LIBRARY_PATH="/u01/utilities/product/runtime:$LD_LIBRARY_PATH"

export LD_LIBRARY_PATH

 The classpath using EXT_PRE_CLASSPATH can be manipulated that is sent to

EXT_PRE_CLASSPATH="…"

export EXT_PRE_CLASSPATH

Note: If there are multiple servers on this domain ensure the script takes this into account using the
SERVERNAME variable.

Native vs Embedded Oracle WebLogic Mode


Note: In Oracle Utilities Application Framework 4.3.0.5.0 and above, embedded more is no longer
supported.

By default, when using Oracle WebLogic, the Oracle Utilities Application Framework based product is
installed in embedded mode. This means that the configuration files and files within the product are
used directly by Oracle WebLogic, including configuration files necessary for Oracle WebLogic itself
(e.g. config.xml). The installation process generates the necessary configuration files and then using
generated versions of Oracle WebLogic utilities points the Oracle WebLogic runtime to the files in the
product environment. The term embedded is used to describe the fact that Oracle WebLogic uses files
embedded in the product rather than its own files; it simply provides the runtime for the environment.

The embedded approach has advantages and disadvantages:

Embedded Mode Advantages and Disadvantages

ADVANTAGES DISADVANTAGES

Simple and easy to implement configuration, ideal Changes to domain configuration within Oracle
for development and other non-production WebLogic console must be reflected in configuration
environments files using user exits or custom templates to retain
changes across patches/upgrades.

One Oracle WebLogic installation can be shared Does not support clustering without complex manual
across many environments on the same host changes to configuration files

Default security setup Administration Server deployed with product

Common configuration change scenarios are Enterprise Manager does not recognize Oracle
handled by configuration settings WebLogic targets without manual configuration of
discovery

Limited changes to some features (such as domain)

Utilities provided must be used to manage product. You


cannot use Oracle Enterprise Manager or WebLogic
console to make configuration changes

74 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
This setup is designed for development and other non-production environments where you need
multiple copies of the product on a single host but may not be appropriate for production environments
where advanced security setup and clustering are typically required.

The alternative is to install the product in what is termed, native mode. Typically Oracle WebLogic
Java EE Web Applications are deployed directly to Oracle WebLogic and managed that way. This has
the advantage of gaining full access to the Oracle WebLogic facilities like advanced configuration and
more importantly the ability cluster the product across multiple nodes. Oracle Utilities Application
Framework V4.x and above, can be installed using this mode with minor changes to the installation
process. It is also possible to convert an embedded installation into a native installation with minor
changes, if migration to this mode is appropriate.

The native mode allows the product to have access to support using the features of Oracle WebLogic
with fewer configuration steps than embedded mode. The advantages and disadvantages of this mode
are outlined in the table below:

Native Mode Advantages and Disadvantages

ADVANTAGES DISADVANTAGES

Direct support for Clustering/Managed Servers Support for multiple environments per installation
requires multiple domains

Changes to domain typically do not require any Requires some manual effort in setting up domain,
manual changes to templates. This reduces risk. servers and security for environment though this is
minimized by using the console or Oracle Enterprise
Manager.

Administration Server can be separated from


product servers.

Enterprise Manager automatically detects targets


and all the functionality from the console is
available

Native facilities in Oracle WebLogic and/or Oracle


Enterprise Manager can be used to operate and
monitor the product

Configuration and operational facilities of Oracle


WebLogic can be used (including documented
variations)

The figure below illustrates the differences between the two modes:

75 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 18. Native vs Embedded architecture

The two modes have different attributes and different approaches applicable to different situations.
The following recommendations should be consider when deciding which mode to use:

 It is not impractical use different modes for different environments. One mode will not usually satisfy
all the needs of all environments.

 It is recommended to use native mode for production implementations as it offers flexibility, cluster
support, separation of the Administration function and the ability to use the advanced configuration
elements of Oracle WebLogic as well as Oracle Enterprise Manager (if applicable).

 It is recommended to use native mode if each environment is housed in a separate virtual machine,
which is common in virtualized implementations. This will allow configuration at the virtual machine
level to be used and reduces maintenance efforts.

 It is recommended to use embedded mode if more than one copy of the product exists on the same
virtual or non-virtual host. The ability to share a common copy of Oracle WebLogic is reduces the
maintenance efforts for multiple environments.

 It is recommended to use embedded mode for development environments where java based
development is taking place. This setup supports the use of the expanded mode features of Oracle
WebLogic used by the Oracle Utilities SDK, which requires access to expanded directories for multi-
user development.

CLIENT-CERT Support
Note: In Oracle Utilities Application Framework V4.2.0.2.0 and above, CLIENT-CERT is supported
from the configureEnv.sh utility directly.

One of the additional configuration options for the authentication of the product is to implement a
Single Sign On solution or implement client certificates. Whilst most of the configuration for these
features is performed in the Single Sign On product and/or Java EE Application Server, the Oracle
Utilities Application Framework has to be configured to use that facility.

In most cases to use these facilities the login configuration for the product has to be changed from
FORM or BASIC to CLIENT-CERT. This informs the product that the credentials will be passed directly
from the Java EE Application Server (via a single sign on solution, security providers or via client
certificates).

For releases of the Oracle Utilities Application Framework prior to V4.2.0.2.0, CLIENT-CERT can be
supported using templates:

 Logon to the machine that houses the environment to change as the product administrator.

 Take a copy of the web.xml.template to cm.web.xml.template in the same directory the


original is located in the templates subdirectory. This will inform the Oracle Utilities Application
Framework to use this new template instead of the base template.

 Edit the cm.web.xml.template file and replace the login-config section with a section
configuring the CLIENT-CERT configuration. For example:

Replace:

<login-config>

76 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
<auth-method>@WEB_WLAUTHMETHOD@</auth-method>

<form-login-config>

<form-login-page>@WEB_FORM_LOGIN_PAGE@</form-login-page>

<form-error-page>@WEB_FORM_LOGIN_ERROR_PAGE@</form-error-page>

</form-login-config>

</login-config>

With:

<login-config>

<auth-method>CLIENT-CERT</auth-method>

</login-config>

Note: For Oracle Utilities Application Framework V4.x customers, this may need to be repeated for the
templates for AppViewer (web.xml.appViewer.template) and online help
(web.xml.help.template) if you wish to include those components in the same solution.

 Ensure the environment is shutdown prior to implementing any changes.

 Execute the initialSetup utility to implement the changes and rebuild the EAR files.

Note: As the web.xml file has been changed and EAR file rebuilt, customers using native mode will
have to redeploy the SPLWeb application to reflect the change.

 Optionally, changes can be verified by viewing the web.xml files generated under the
$SPLEBASE\etc\conf subdirectory of the product installation.

 Restart the product.

The product is now configured to use CLIENT-CERT.

Implementing Work Managers


Oracle Weblogic supports Work Managers which allow resource constraints to be allocated to servers.
The main use for Work Managers is to provide service assurance by preventing traffic overload of
servers. By default, Oracle WebLogic provides a global work manager with no limits.

It is possible to use Work Managers to constrain the traffic for online transactions and Web Services
traffic for Oracle Utilities Application Framework based products. Once a Work Manager is deployed
Oracle WebLogic will track traffic till the configured resource limit is reached. If the resource limit is
reached, the server where the limit is attached will refuse more traffic to assure existing traffic has
enough resources to complete, till the limit is not exceeded once more. To use Work Managers
effectively, it is recommended that clustering or multiple managed servers be used to maximize
availability.

From the Oracle WebLogic console, perform the following:

 Define a Capacity Constraint for use the server. You can optionally deploy the constraint directly to
the product server definition or via a custom Work Manager for tracking.

77 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 Define a Work Manager and associate the fore-mentioned Capacity Constraint with the Work
Manager. Deploy the Work Manager to the product server(s).

Once implemented, Work Managers can be monitored from the Oracle WebLogic console.

Implementing the Custom Authentication Security Provider


Note: Refer to the Security Guide for details on how to install and configure the Authentication
Security Provider.

In Oracle Utilities Application Framework V4.3.0.4.0 and above, a custom OUAF based Authentication
Security Provider for Oracle WebLogic is available. In past releases, a number of security checks were
performed at login time. Use of this Authentication Security Provider allows implementations to decide
when these checks are performed by the domain where there is a complex security configuration such
as having multiple authentication sources or federated solutions. Use of the Authentication Security
Provider is optional.

This security provider allows implementations to add additional checks to their security domain with
the following functionality:

 The custom Authentication Security Provider connects to the product, via a preconfigured data
source, to perform additional security checks.

 The Security Provider checks that the user exists within the product, using the authentication
identifier as a key, on the User Object. If the user does not exist, a security violation is returned to
be processed in the Oracle WebLogic security domain as configured.

 If the user exists in the product, the Security Provider checks that the user is enabled. If the user is
disabled on the User Object, a security violation is returned to be processed in the Oracle WebLogic
security domain as configured.

 The Security Provider can be configured to ignore privileged accounts if desired. The list of excluded
users can be configured on the Provider.

This Authentication Provider is designed to work with other providers (including other Authentication
Security Providers) to provide greater control over a complex security setup.

A typical use case is as follows:

 The site wants to implement an external authentication source such as a SAML based solution,
Microsoft Active Directory, Oracle Unified Directory, OpenLDAP or Novell NDS LDAP Server. This
involves configuring an authentication provider with the links to the relevant product.

 The site does not want to include privilege accounts in their primary security authentication source
so they use the embedded LDAP as an additional source. They would configure both sources and
have the rules setup in such a way that the use must be a member of either source.

 The OUAF based Authentication Provider can be also added to the mix to check that the user is
active and part of the application after it has been authenticated by the primary source to ensure the
user is active and defined. This also would check regularly to implement a lockout of a user that is
disabled on the User Object as part of a user decommission process.

In the above use case, the configuration of the OUAF based Authentication Security Provider is used
to catch user profiles before they are passed to the login process. This is certainly an extra layer of
security to catch users that are actively disabled during a session.
In the architecture the Business
Application Server contains the

78 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
business logic to be used with
each channel.
BUSINESS APPLICATION SERVER BEST PRACTICES
The Business Application Server is used by product to process the business logic. Whilst most of the
advice for the Web Application Server can be reused with the Business Application Server there are a
number of practices and general advice that is specific to this tier in the architecture.

Cache Management
One of the features of the Oracle Utilities Application Framework is the implementation of a level 2
cache within the architecture to provide performance benefits for commonly used configuration
information. Generally the cache is managed by the Oracle Utilities Application Framework
automatically with little or no interaction from operators. By default, the cache is reloaded as needed or
every eight (8) hours, whichever occurs first. Some elements of the cache such as security information
is refreshed on a more frequent basis (every 30 minutes).

There are a number of cache management utilities to manually cause all or parts of the cache to
refresh manually. These utilities are documented in the Server Administration Guide for your product.

While these utilities are rarely used in production, they can be used, by appropriately authorized
personnel to make sure the cache contains the correct information. Typically the manual refresh is
required if the configuration data is changed and needs to be reflected as soon as possible.

Using JMX with the Business Application Server


In Oracle Utilities Application Framework Version 4.0 it is possible to enable JMX performance
statistics to allow collection, management and monitoring of JVM information for the Business
Application Server. For backward compatibility, the JMX enabled facilities are disabled by default. To
use this facility you must execute the configureEnv.sh utility with the –a option (Advanced Menu)
and specify the following settings:

JMX Options

OPTION USAGE

JMX Enablement System Userid Userid used for logging onto JMX Mbeans

JMX Enablement System Password Password to be used for JMX Enablement System Userid

RMI Port for JMX Web Port number to allocate to the JMX for the Web Application
Server

This information is added to the spl.properties file in the $SPLBASE/etc/conf/service


subdirectory for the environment, for the Business Application Server. An example of the applicable
settings is shown below:

spl.runtime.management.rmi.port=…

spl.runtime.management.connector.url.default=service:jmx:rmi:///jndi/rmi://
host:…/oracle/ouaf/ejbAppConnector

ouaf.jmx.com.splwg.ejb.service.management.PerformanceStatistics=enabled

jmx.remote.x.password.file=scripts/ouaf.jmx.password.file

jmx.remote.x.access.file=scripts/ouaf.jmx.access.file

79 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
The following settings are important to the JMX monitor:

 The spl.runtime.management.connector.url.default is the JMX url to be used in the


JMX console or JMX browser.

 The jmx.remote.x.password.file and jmx.remote.x.access.file are the default


security setup for the JMX. These are for basic security setup. For more information about the files
and alternative security setups refer to Monitoring and Management Using JMX Technology.

 The ouaf.jmx.* settings enable individual beans at startup time. These may be enabled at
runtime.

 Once the Business Application Server component is started; the JMX Mbeans defined in this
configuration are started and a JSR160 compliant JMX console or JMX browser can be used to
connect to the JMX Mbeans. The remote URL and credentials are provided as configured above.

 The only Mbean available with the Business Application Server is the PerformanceStatistics
Mbean. This Mbean collects object performance data for analysis. For customer familiar with the
Oracle Tuxedo product, this facility is similar to the txrpt facility available for performance analysis.

 The statistics are collected by the Mbean from the time the Mbean is enabled until the environment
statistics are reset. By default, the Mbean is enabled at startup time but may be disabled (or re-
enabled) at any time using the disableMbean or enableMbean operations from the
PerformanceMbeanController Mbean.

 When using this Mbean there are a few recommendations:

– The completeExecutionDump operation returns a CSV of the performance statistics of


individual application services to the JMX console or JMX browser. This represents the current
state of the statistics at that time.

– The reset operation resets the statistics within the Mbean to start collection. This operation is
handy to ensure performance over a selected period.

There are other operations and attributes that return individual value information that may of interest.
Refer to the Server Administration Guide provided with your product for a detailed description of what
statistics are available.

Note: No confirmation (i.e. Are You Sure?) dialog is provided with most JMX consoles or JMX browser
so care should be taken when issuing commands.

Replicating the txrpt statistics


One of the features customers of past releases of V1.x of Oracle Utilities Customer Care And Billing
used to use to gather performance data was the txrpt facility within Oracle Tuxedo. The utility would
take performance data gathered from every service call and produce summary statistics per hour. The
statistics were the number of calls and the average response time for each defined service. The
txrpt utility collected the statistics from log files that were enabled in the Oracle Tuxedo
configuration. This information was useful in tracking the performance of individual services within the
product against a site SLA.

With the advent of Oracle Utilities Application Framework V2.x and the removal of Oracle Tuxedo from
the architecture meant that this information was not available for collection as easily as originally. In
Oracle Utilities Application Framework the implementation of the PerformanceStatistics Mbean
allows for collection of performance information in a similar fashion txrpt. To achieve the same
results as txrpt the following should be performed:

80 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 On the hour boundary the completeExecutionDump operation must be executed by your JMX
console or JMX browser to extract and save the CSV information to a file. The file should have the
date and time of the collection for reference reasons.

 After collection of the statistics has been completed, the reset operation should be executed from
your JMX console or JMX browser.

The information in the files can be collated according to the desired analysis required by your site to
summarize the information. The CSV can be loaded into a database for analysis or into your sites
preferred spreadsheet or analysis tool. Remember that the date and time of the collection is not
recorded in the data only the data itself.

Note: While this process can be manually done using a JMX console such as jconsole, it is
recommended that the JMX console or JMX browser automate the collection of the process in the
background. Refer to the documentation of the JMX console and JMX browser to configure your
console or browser to achieve this.

This facility is flexible for a number of reasons:

 The time period for collection is not limited to hourly as txrpt was. The time collection period can
be increased or decreased according to your site standards. For example, you might want to collect
the data every 10 minutes.

 The statistics are live and can be queried regardless of the collection process.

 The level of information is higher than the original txrpt. The following additional information is
collected and summarized:

– The data is now also summarized by the type of transaction that is performed. This will allow the
site to assess the performance of reads, updates, deletes, inserts etc separately.

– The last transaction recorded is detailed including the user. This information is useful for checking
against other statistics to assess where the performance is at the present moment.

 Statistics are already calculated by the utility prior to analysis. The txrpt utility only collected the
average. This facility collects the average, minimum (best case) and maximum (worst case)
performance statistics in the collection period.

Database Connection Management


Hibernate and UCP are used to provide a pool of connections to the database for the various
components of the product. A separate pool exists for online, IWS and background processes. The
size of the pool can be set in the hibernate.properties.

The size of the pool can vary from mode of component to component with the following guidelines:

 The minimum pool size of the product should be set to the average number of connections needed
for the mode of access. By default it is set to one (1) which is sufficient for non-production, but for
each new connection required for the traffic the database connection needs to be established prior
to use. The establishment of an individual database connection can cause delays to the transaction
using the connection as it waits for the connection to be established. This negates the benefit of
pooling connections. Track the number of connections used at normal traffic load and specify that as
the minimum. This will establish the connections at startup time and avoid the overhead of creating
connections on the fly. Ideally you want to avoid creating connections on the fly unnecessarily.

 The maximum pool value should be set to cover any peak load you may experience. Initially the

81 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
values can be artificially inflated but after monitoring the number of connections open at peak times
can optimize the value.

 The total number of database connections from all pools connecting to an individual database
should not exceed the number of configured users/connections for that database. Exceeding the
number of configuration users can cause database connection failures and delays in transactions.

 Typically customers have indicated that a good rule of thumb to use is that at any time one third of
the defined users are active for normal traffic and two thirds are active at peak.

Note: This is a rule of thumb and may NOT apply to the traffic patterns at your site. It is recommended
to start with an agreed value and then monitor to optimize the values as necessary.

Refer to the Server Administration Guide for your product for additional advice on this facility.

XPath Memory Management


Note: This facility is available for Oracle Utilities Application Framework V4.1 and above.

With the popularity of the Configuration Tools facility within the product for customer extension the
increase load of XPath may cause memory issues under particular user transaction conditions (in
particular high volume patterns). As with most technology in the Oracle Utilities Application
Framework, the XPath statements used in the Configuration Tools are cached for improved
performance. Increased load on the cache may cause memory issues at higher volumes.

To minimize this the Oracle Utilities Application Framework has introduced two new settings in the
spl.properties file for the Business Application Server, where the dimensions of the XPath
statement cache are defined. These settings allow the site to optimize the control the XPath cache to
support caching of commonly used XPath statements but allowing for optimal specification of the
cache size (to help prevent memory issues).

The settings are shown in the table below:

XPath Options

SETTING USAGE

com.oracle.XPath.LRUSize Maximum number of XPath queries to hold in cache across all


threads. A zero (0) value indicates no caching, minus one (-1)
value indicates unlimited or other positive values indicate number
of queries stored in cache. Cache is managed on a Least Reused
basis. For memory requirements, assume approximately 7k per
query). The default in the template is 2000 queries

com.oracle.XPath.flushTimeout The time, in seconds, when the cache is automatically cleared. A


zero (0) value indicates never auto-flush cache and a positive
value indicates the number of seconds. The default in the
template is 86400 seconds (24 hours).

Note: The templates provided with the product have these settings commented out. To use the
settings uncomment the entries in the generated configuration files.

In most cases the defaults are sufficient but can be altered if the following is guidelines are:

82 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 If there is memory issues (e.g. out of memory) then decreasing the LRUSize or decreasing the
flushTimeout may result in a reduction in memory issues. LRUSize has a greater impact on
memory than flushTimeout.

 If decreasing value the value of the LRUsize causes performance issues consider changing the
flushTimeout initially only and ascertain if that works for your site.

There are no strict guidelines on the value for both parameters as cache performance is subject to the
user traffic profile and the amount and types of XPath queries executed. Experimentation will assist in
determining the right mix of both settings for your site.

Enabling Service Timings


Note: This facility is designed for development use only and is not recommended for production use. It
is recommended to use the JMX facility for production if tracking facilities are required for production.

If the Performance Statistics JMX call is not valid for your site it is also possible to configure log4j to
display service performance information in the spl_service.log. This is possible by adding the following
line to the $SPLEBASE/splapp/businessapp/properties/log4j.properties or
$SPLEBASE/etc/conf/service/log4j.properties file by adding the following line:

log4j.logger.com.splwg.base.api.service.ServiceDispatcher=debug

Once this is set the debug messages will written to the


$SPLEBASE/logs/system/spl_service.log in with the message type
api.service.ServiceDispatcher type with a Start and End message. Both the Start and End
message outline the service name called, execution mode and the End message includes the timing
for the service in ms. For example:

USER1 weblogic.kernel.Default (self-tuning)'] DEBUG


(api.service.ServiceDispatcher) Start 'FWLEFKRP read

USER1 - 682001-206-1 2011-12-09 16:00:37,932 [[ACTIVE] ExecuteThread: '1'


for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG
(api.service.ServiceDispatcher) End 'FWLEFKRP read, time 18.430 ms

Oracle WebLogic Datasource Support


Note: The instructions below apply to multiple versions of Oracle Utilities Application Framework. Use
the appropriate instructions for the appropriate version of Oracle Utilities Application Framework used.

Note: For background on the JDBC Datasource Support within Oracle WebLogic, refer to Oracle
WebLogic Server Data Sources.

Note: For tuning advice for JDBC Datasources refer to Tuning Data Source Connection Pools.

By default, the online (and XAI) components of the Oracle Utilities Application Framework use
Universal Connection Pool for connection pooling. If you wish to use Oracle WebLogic JDBC based
connection pooling for integration to other products or to use the GridLink features of Oracle
WebLogic, then Oracle Utilities Application Framework needs to be configured to use Oracle
WebLogic Data Sources.

83 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
The advantages of using data sources are:

 Maintenance of the pool characteristics – It is possible to maintain the pool tolerances and
connection information from the Oracle WebLogic console or Oracle Enterprise Manager. In some
cases these tolerances can be changed without an outage to the product. The figure below
illustrates the typical pool characteristics managed from Oracle WebLogic:

Figure 19. Example Pool tolerance configuration

 Advanced Monitoring – The pool management capabilities of Oracle WebLogic includes a set of
statistics that are calculated for the pooling that can be tracked to ascertain the health of the pool.
These statistics can be obtained using JMX, via the Oracle WebLogic console or via Oracle
Enterprise Manager. The figure below illustrates the capability to display the statistics to the Oracle
WebLogic console:

84 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 20. Example Pool statistics

 GridLink Support – Using the Oracle WebLogic Data Sources allows the ability to use GridLink
based data sources that provides easier configuration for failover and RAC configuration. Refer to
the Oracle WebLogic GridLink documentation for a discussion of this feature set.

 Diagnostic analysis – The Oracle WebLogic Data Sources have advanced diagnostic capabilities
to monitor and detect database connectivity and for resource profiling. This information is available
from the Oracle WebLogic console, JMX and Oracle Enterprise Manager. Refer to the Monitoring
documentation for Oracle WebLogic for a discussion of the facilities.

The Oracle Utilities Application Framework can be configured to use Data Sources using the following
process:

 Define the JDBC datasource to the product database, using the Oracle WebLogic console, with the
following attributes:

 The JNDI Name for the datasource should contain a directory name such as jdbc. For example,
jdbc/demodb. The name of the JNDI should reflect your specific requirements.

 Ensure that Global Transaction Support is disabled on the JDBC connection. This is not appropriate
for this integration.

 The XA JDBC driver may be used but the product does not take advantage of the XA feature set.

 The Instance or Service driver may be used but if there is no site preference use the Service driver.

 Unless otherwise stated, the Statement Cache Algorithm should be set to the default setting of LRU.

 The Database User and Password to use should be any database user with read/write access to the
product such as CISUSER or SPLUSER. The value can correspond to the value of the DBUSER
configuration variable in the ENVIRON.INI, if no site preference exists.

Note: For Oracle Utilities Application Framework 4.3 and above, this manual process described below

85 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
is not required. The installation asks for the JDBC_NAME as part of the installation which is the fully
qualified JNDI name for the data source.

For products using Oracle Utilities Application Framework V4.x then the above XML code should be
placed in the CM_config.xml.exit_3.include in the templates directory.

To use the new JDBC datasource for online the following must be performed:

 Copy the hibernate.properties.web.template to


cm.hibernate.properties.web.template within the relevant directory. This overrides the
base template with the custom template.

 Add the following lines to the top of the cm template file:

hibernate.connection.datasource = <jdbc_jndi>

hibernate.connection.username = <jndi_user>

hibernate.connection.password = <jndi_user_password>

where :

<jdbc_jndi> Full JNDI name for the JDBC connection

<jndi_user> A user to access the JNDI (Substitution variable


@WEB_WLSYSUSER@ can be used in the template).

<jndi_user_password> The password for the user to access the JNDI (Substitution variable
@WEB_WLSYSPASS@ can be used in the template).

 Remove the following lines from the cm template file depending on the version of the Oracle Utilities
Application Framework:

– All hibernate.connection.url entries including any ifdef statements

– All hibernate.ucp.* entries

 Change the hibernate.transaction.factory_class to


org.hibernate.transaction.JDBCTransactionFactory

 Change the hibernate.connection.provider_class to


org.hibernate.connection.DatasourceConnectionProvider

Note: For later versions of Hibernate use the


org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProvide
rImpl class for hibernate.connection.provider_class.

 Save the file. For Oracle Utilities Application Framework V4.2 and above the sample cm template is
shown below:

hibernate.connection.driver_class = @DBDRIVER@

hibernate.connection.datasource = jdbc/ouafdb

hibernate.connection.username = @WEB_WLSYSUSER@

hibernate.connection.password = @WEB_WLSYSPASS@

86 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
hibernate.dialect = @DIALECT@

hibernate.show_sql = false

hibernate.max_fetch_depth = 2

hibernate.transaction.factory_class =
org.hibernate.transaction.JDBCTransactionFactory

hibernate.jdbc.fetch_size = 100

hibernate.jdbc.batch_size = 30

hibernate.query.factory_class=org.hibernate.hql.internal.classic.ClassicQue
ryTranslatorFactory

hibernate.cache.use_second_level_cache = false

hibernate.query.substitutions = true 'Y', false 'N'

hibernate.connection.provider_class=org.hibernate.connection.DatasourceConn
ectionProvider

hibernate.connection.release_mode=on_close

#ouaf_user_exit hibernate.properties.exit.include

#ouaf_user_exit hibernate.properties.web.exit.include

 Execute the initialSetup.sh command to apply the changes and new templates.

Note: It is possible to execute the initialSetup.sh –t command to apply the changes. This avoids
an EAR rebuild.

 Restart the server to verify the connection is active.

DATABASE BEST PRACTI CES


The Database Server is responsible for the storage and management of data. There are a number of
practices that sites find useful for maintaining the health of the Database Server. The database in the Oracle
Utilities Application Framework
architecture stores and managed
Regularly Calculate Database Statistics
the data for the Oracle Utilities
Database statistics are important for the performance of all SQL in the product. Keeping them up to products.
date ensures the database has the most up to date information to make the appropriate access path
decisions.

When any table in the system grows (or shrinks) by a larger than normal rate, the access paths to that
table may change causing inefficiencies. For the database to make the correct decision, it uses a set
of statistics to assess all available paths. This is an important factor in performance. It is therefore
recommended that database statistics be recalculated, using dbmsstats, on a regular basis to
maintain up to date statistics.

The frequency will depend on the volume and size of your database. It is recommended that statistics
most tables be calculated once a week at minimum unless their growth factors do not affect the path
chosen by the DBMS.

87 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Note: CISADM is used as an example in the guidelines below. If your site uses another schema owner,
then substitute that owner in the examples below.

The following guidelines can be used to assist:

 It is possible to check what is the Last Analyzed Date on product tables are current (or not) by
running the following SQL.

SELECT table_name, last_analyzed FROM dba_tables WHERE owner = 'CISADM';

 It is possible to check check what is the Last Analyzed Date on indexes are current (or not) by
running the following SQL.

SELECT index_name, last_analyzed FROM dba_indexes WHERE owner = 'CISADM';

 If the Indexes are older by a week or more than consider gathering Statistics on them. You can also
use the below SQL which tells approximate number of INSERTs, UPDATEs, and DELETEs for that
table, as well as whether the table has been truncated, since the last time statistics were gathered.

SELECT * FROM USER_TAB_MODIFICATIONS;

Note: The MONITORING attribute must be set on individual objects to use this facility.

 It is recommended to gather statistics while no active purging activities are occurring on the
database.

 It is recommended to use the dbms_stats package for collecting statistics. An estimate percentage
of 10 percent is generally sufficient. Set the degree parameter to a higher level to enable parallel
collection of statistics. It is suggested to set the block_sample parameter to false.

 The Method option while gathering statistics on tables should be set to FOR ALL COLUMNS SIZE
AUTO. This will make sure that Oracle automatically determines which columns require histograms
and the number of buckets (size) of each histogram.

 Gathering statistics separately for indexes is generally faster than the cascade=true option while
gathering table statistics.

 It is recommended to not collect statistics on all the tables at a single batch run at a single point of
time. Dividing the tables into multiple groups and then executing statistics calculation for each group
at different time frames will minimize any disruption due to statistics calculation.

 Depending on the stability of the query performance, it is suggested that the statistics collection
frequency can be altered to maintain query performance.

A discussion on the statistics calculation is also discussed in Performance Troubleshooting Guideline


Series (Doc Id: 560382.1) whitepapers available from My Oracle Support.

Ensure I/O is spread evenly across available devices


Ensuring I/O across devices is becoming less of a problem with progressive versions of the database
handling this automatically and the introduction of intelligent SAN technology.

One of the key practices that are key to performance of a database is the elimination of hot spots in
the disk architecture by ensuring that I/O is spread across all available devices. This is known as the
Database Topology. For example, placing the database physical files on a single disk is not optimal as
multiple concurrent requests queue to use the disk and would result in higher than expected disk wait

88 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
times. By spreading the load across disks, the opportunity for wait times is minimized and increases
throughput. It is therefore recommended that the disk architecture be designed for the physical
database files so that as much I/O as possible is spread across all disks.

A discussion on the database topology and its implications is outlined in Performance Troubleshooting
Guideline Series (Doc Id: 560382.1) whitepapers available from My Oracle Support.

Use the Correct NLS settings (Oracle)


One of the configuration settings that can affect the sorting and processing of date data is setting the
correct language for the connection to the database. For Oracle customers it is recommended that the
NLS setting is correct for your region is set correctly at installation time. Refer to the Globalization
Support Guide for Oracle for details of the valid settings for your region.

Monitoring database connections


In Oracle Utilities Application Framework V4.0 and above, it is possible to use a different user per
access method (online, batch, etc) it is limited to a single user per access method. This can be limiting
when trying to track individual sessions at the database level as the connections can be difficult to
distinguish.

It is possible to track individual connections using two attributes of the v$session system view:

 CLIENT_IDENTIFIER – The application user used for the duration of the transaction is now placed
in the CLIENT_IDENTIFIER for the duration of the transaction using the connection. For
compatibility purposes, the short userid is placed in this column (not the Login Id). If the connection
is idle, the column is blank.

 MODULE – The module that is executing and using the database connection is now populated in the
MODULE field of v$session. If the connection is active, the MODULE will contain the text UGBU
Idle to denote it as an idle connection using by the product. The value of this will vary according to
the object type:

Module Values

OBJECT TYPE VALUE

Application Service Application Service Name is displayed. This can be translated using the
CI_MD_SVC_L table where SVC_NAME is the service name in MODULE and DESCR is
the description of the service.

Business Object Business Object Code is displayed. This can be translated using the F1_BUS_OBJ_L
table where BUS_OBJ_CD is the Business Object name in MODULE and DESCR is the
description of the Business Object.

Business Service Business Service Code is displayed. This can be translated using the F1_BUS_SVC_L
table where BUS_SVC_CD is the Business service code in MODULE and DESCR is the
description of the Business Service.

Service Script Script Code is displayed. This can be translated using the CI_SCR_L table where
SCR_CD is the script code in MODULE and DESCR is the description of the script.

89 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Note: If more than one language pack is installed on the product then LANGUAGE_CD must be
populated to return the description in the desired language.

Note: To use the MODULE feature the hibernate.connection.release_mode must be set to


on_close in the hibernate.properties file. This is the default for Oracle Utilities Application
Framework V4.2 and above, earlier releases require manual changes to configuration files.

 ACTION – The transaction type that is requested for the MODULE is now populated in the ACTION
field of v$session. If the connection is idle, the column is blank. The table below lists the valid
action values populated:

Action Values

ACTION VALUE

ADD Service is attempting adding a new instance of an object to the system.

CHANGE Service is attempting changes to an existing object in the system.

DEFAULT_ITEM Service is resetting its values to defaults. For example, by pressing the Clear button
on the product UI toolbar.

DELETE Service is attempting to delete an existing object.

EXECUTE_BO Service is a business object and is executing (Business Objects only)

EXECUTE_BS Service is a business service and is executing (Business Services only)

EXECUTE_LIST Service is a list based service and is executing (List Services only)

EXECUTE_SEARCH Service is a search based service and is executing (Search Services only)

EXECUTE_SS Service is a service script (including BPA scripts) and is executing (BPA and Service
Scripts only)

READ Service is attempting to retrieve an object from the system

READ_SYSTEM Service is a common Oracle Utilities Application Framework based service that is
executing

VALIDATE Service is issuing a validation action

 CLIENT_INFO - The contents of the Database Tag characteristic type (up to 64 characters) on the
9

individual user record is now populated in the CLIENT_INFO field of v$session. If the connection
is idle, the column is blank. A value of up to 64 characters can be used. If the database tag is not
used this value is blank. For example:

9
Refer to the database documentation for the length of the v$session columns

90 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 21. Example Database Tag characteristic

10
An example of this monitoring information from v$session, which shows the connection from
SYSUSER executing a read transaction on the User object (CILTUSEP):

Figure 22. Example Pool tolerance configuration

Information in these columns is only populated when an active transaction is actually executing
against the database. When a connection is idle, this status is indicated in the MODULE field.

Building the Data Model


One of the common questions regarding product is the availability of the total data model in a
particular tool (such as Oracle Data Modeler or similar). The product contains a large number of tables
and it is generally impractical to display a full model due to its size (to make it legible).

There are a number of sources of information that can replace a full data model and present the data
mode information into bite sized chunks:

 The data model information is contained in the Data Dictionary component of the Application
Viewer.

 The Conversion documentation, provided with the product documentation, contains a summary set
of data models that basically outline the major entities in the product.

Note: Not all Oracle Utilities Application Framework products include a conversion capability.

Each of the Business Process manuals for the product outlines the functionality and contain data
models specifically for that component.

WHY IS THERE NO REFERENTIAL INTEGRITY BUILT INTO THE DATABASE?

Typically referential integrity of a database is managed by the database itself. In product this is not so
as the Maintenance Objects contain ALL the business logic including referential integrity. The reasons
for this are varied:

 From a maintenance cost point of view, all the code is in one place. This reduces maintenance

10
Example does not include Database Tag sample

91 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
effort.

 Databases implement all or nothing referential integrity. This means that referential integrity is
checked whether the data has changed or not. From a performance point of view this is potentially
wasting time. The Maintenance Objects in product decide when to enforce referential integrity rules.

 Most of the referential rules in product are optional. If there is a value in the foreign key field it is
checked, if there is no value (blanks, zero or nulls) then the referential integrity is not checked
unless it is a mandatory column. This is not possible in database imposed referential integrity.

 If the database controlled referential integrity then the application has no control on when it is
imposed in the course of a transaction. Maintenance Object controlled referential integrity allows
finer levels of control on when referential integrity is enforced in the transaction flow.

 Each database implements referential integrity in a slightly different way. To reduce maintenance
costs, code differences are kept to a minimum.

 Maintenance Object enforced referential integrity is more efficient as far as product is concerned
and translates to superior performance across many database types.

BUILDING THE DATA MODEL

All is not lost though. The Oracle Utilities Application Framework maintains its own data dictionary in
the form of meta-data that is used by the Oracle Utilities Software Development Kit.

If you insist that you want the data model in a tool or adorning a large wall then the following is
recommended process to be used to generate the data model using the meta-data:

 Export the CISADM schema (with no data) as a backup using the database export utility or use SQL
Developer to clone the schema to a ghost schema.

 Create constraints from the meta-data structure. The two Oracle pl/sql scripts below can be used to
achieve this. The name of individual constraint are documented in the meta data. Run the utility and
created the constraints in the database.

FUNCTION TO JOIN

create or replace function join

p_cursor sys_refcursor,

p_del varchar2 := ','

) return varchar2

is

l_value varchar2(32767);

l_result varchar2(32767);

begin

loop

fetch p_cursor into l_value;

92 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
exit when p_cursor%notfound;

if l_result is not null then

l_result := l_result || p_del;

end if;

l_result := l_result || l_value;

end loop;

return l_result;

end join;

show errors;

SCRIPT TO CREATE CONSTRAINTS

SET serverout ON size 1000000

SET echo OFF

SET feedback OFF

SET linesize 300

spool constraints.sql

DECLARE

CURSOR c1

IS

SELECT tbl_name,

CONST_ID,

REF_CONST_ID,

table_name

FROM ci_MD_CONST,

user_indexes

WHERE CONST_TYPE_FLG='FK'

AND TRIM(index_name)=SUBSTR(REF_CONST_ID,5,7) AND


LENGTH(TRIM(REF_CONST_ID))>8

AND TRIM(tbl_name) IN (SELECT TRIM(table_name) FROM user_tables)

UNION ALL

SELECT tbl_name,

93 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
CONST_ID,

REF_CONST_ID,

table_name

FROM ci_MD_CONST,

user_indexes

WHERE CONST_TYPE_FLG='FK'

AND TRIM(index_name)=TRIM(REF_CONST_ID) AND LENGTH(TRIM(REF_CONST_ID))<=8

AND TRIM(tbl_name) IN (SELECT TRIM(table_name) FROM user_tables)

UNION ALL

SELECT UNIQUE tbl_name,

CONST_ID,

REF_CONST_ID,

table_name

FROM ci_MD_CONST,

user_indexes

WHERE CONST_TYPE_FLG='FK'

AND TRIM(index_name)=TRIM(REF_CONST_ID)

AND TRIM(tbl_name) IN (SELECT TRIM(table_name) FROM user_tables)

ORDER BY 1;

---

stmt VARCHAR2(400);

field_list VARCHAR2(300);

BEGIN

FOR r1 IN c1

LOOP

stmt := 'alter table ' || trim(r1.tbl_name) || ' add constraint ' ||


trim(r1.const_id);

dbms_output.put_line(stmt);

SELECT

JOIN(CURSOR

94 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
(SELECT trim(fld_name)

FROM ci_md_const_fld

WHERE const_id = r1.const_id

ORDER BY seq_num

))

INTO field_list

FROM dual;

stmt := 'foreign key (' || field_list || ')';

dbms_output.put_line(stmt);

SELECT

JOIN(CURSOR

(SELECT trim(fld_name)

FROM ci_md_const_fld

WHERE const_id = r1.ref_const_id

ORDER BY seq_num

))

INTO field_list

FROM dual;

stmt := 'references ' || trim(r1.table_name) || ' (' || field_list || ');';

dbms_output.put_line(stmt);

END LOOP;

END;

spool OFF;

EXIT;

 Run the constraints.sql file created in the previous step to create the RI using the ghost
schema owner.

 Load the data model in the tool of your choice. Load the data model with the constraints in the
desired tool. This should build the data model.

Note: This may take a while for the WHOLE data model.

 Removed the newly created constraints. This is to return the database back to the original condition.

95 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
set serverout on size 1000000

set echo off

set feedback off

set linesize 300

spool drop_constraints.sql

select 'ALTER ' || tbl_name || ' drop constraint ' || CONST_ID || ';' from
ci_MD_CONST where CONST_TYPE_FLG='FK' order by tbl_name, CONST_ID;

spool off;

@drop_constraints.sql

exit;

 Reload the database. You then have the data model in your tool and the database returned to its
original state.

Using Database Compression


One of the features of the database is ability the compress the data. The compression facilities in
Oracle have a number of attractive features:

 Compression applies to disk as well as memory (in the database buffer cache). This saves both
storage costs and means that more data can be loaded into memory. Whilst there is a CPU
overhead with compression, due to the compression and decompression activities, the memory
processing savings may cancel out any overhead to yield performance improvements.

 Compressed data can co-exist with uncompressed data. For example, compression can be enabled
so that any new data is compressed automatically.

Oracle offers various levels of compression.

 Basic – Each edition includes a basic compression algorithm.

 Advanced Compression – Advanced Compression is an option on the Enterprise Edition of Oracle


and offers higher levels of compression as well as compression optimized for various activities such
as OLTP.

 Hybrid Columnar Compression – This is the newest and most advanced compression algorithm
that offers the highest level of compression and compression optimizations. At the present time this
is offered within Oracle ExaData with hardware assisted compression, to reduce compression CPU
overheads.

Compression can be used with Oracle Utilities Application Framework products as it is transparent to
the underlying product code with all the options described above. Guidelines for compress are
available from the Database Administration Guide. Implementation advice is available in the Advanced
Compression whitepaper.

For Oracle Utilities Application Framework based applications the following additional advice is
recommended:

 Don't Compress Staging Tables – Staging Tables are basically queue tables where data is

96 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
inserted and then later the data is mostly deleted as it is transferred to the main table. Compression
is not suited to those types of tables.

 Low Cardinality Tables Offer Better Compression – Tables where there is a low cardinality will
take more advantage of the compression algorithms. This also translates to the backups of these
tables.

 Transaction Tables Are Key Candidates – In terms of value, transaction tables are the main
tables that would benefit from compression. Other tables can be compressed but the value may not
be as high as transaction tables.

CONFIGURATION MIGRATION ASSISTANT BEST PRACTICES

Note: Configuration Migration Assistant is only used to migrate configuration data only. It is not
supported for non-configuration data. Configuration Migration Assistant
is provided with all Oracle Utilities
The Configuration Migration Assistant was provided in Oracle Utilities Application Framework 4.2 and Application Framework based
products to manage configuration
above for migrating Configuration Data from one environment to another. The capability provides a
Data optimally and protecting data
facility to export configuration data and imports that data, with optional approvals, into a target
integrity.
environment whilst protecting data integrity.

The capability adds the following to the Oracle Utilities Application Framework:

 Migration Objects. A set of migration objects are added to the product to define the configuration
objects that are supported to be migration and the instructions on how to migrate configuration
objects as well as hold migration import information for approvals.

 Migration Batch Controls. A set of batch jobs that perform the migration export and migration
imports to implement the capability in the background.

 Migration Portals. A set of portals to manage the migration instructions and review the migrations
including approvals for individual changes.

For more information about each of the above, refer to the online documentation shipped with the
product.

Optimizing Locations for Migrations


One of the first parts of the implementation of the Configuration Migration Assistant is to decide the
location of the migration exports and imports for use across the environments. This location is either
defined in the Migration Assistant Configuration Master Configuration Record or using the appropriate
environment variable (e.g. F1_CMA_FILES or CLOUD_LOCATION_F1_MIGR_ASSISTANT_FILES).
For example:

97 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 23. Master configuration for Migration Assistant Configuration.

Individual settings are required for the import and export locations used for the individual
environments. There are a number of variations that can be used when setting up the locations:

 Consider Common Locations. One the techniques that can be useful is to setup a common
location for all migrations. That would maximize reuse and flexibility whilst minimizing any controls.
That way, migrations would simply be deposited into the common location and loaded from that
location into any target environment. The common location can be shared across machines to
provide maximum flexibility.

 Consider Gated Locations. An alternative to the common location is what is known as gated
locations. These are locations that are environment specific and are gated as it requires a manual
(or automated) process for copying the file from one location to another before the import can be
performed. This is useful where you have a formal approval process for migrating data so you want
to control when the exports are made available.

 Consider Cloud Storage – In Oracle Utilities Application Framework 4.3.0.6.0, a File Adapter has
been added that supports local and remote storage on an Oracle Cloud Object Storage Service
instance. Refer to the release notes for this version of the Oracle Utilities Application Framework for
more details.

Daisy Chaining Jobs


The engine used for the Configuration Migration Assistant is batch based with a number of jobs to
perform both exports and imports. These jobs can be run individually to achieve the results or can be
daisy chained using some additional configuration. The latter method is used in the Oracle Utilities
Cloud implementation of Configuration Migration Assistant to minimize effort and costs.

Once the daisy chain method is configured only the F1-MGDIM (Migration Data Set Import Monitor)
and F1-MGDPR (Migration Data Set Export Monitor) jobs need to need to be executed to import and
export configuration data, respectively.

Note: These instructions only apply to the Import process as the Export process is already a single
process.

There is a multistep process for configuring the Enter Algorithms and Post Processing Algorithms to
tell the product to minimize the export and import processes.

Note: Regardless of this configuration approvals are still supported as that is indicated on the
individual import migration request.

98 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Note: To take full advantage of this new configuration, enable Automatically Apply on Imports.

SET ENTER ALGORITHMS

The first step is to configure the import process, which is a multi-step process, to auto transition data
when necessary to save time. This is done on the F1-MigrDataSetImport business object and
setting the Enter Algorithm on the following states:

Enter Algorithm Values

STATUS ENTER ALGORITHM

PENDING F1-MGDIM-SJ

READY2COMP F1-MGOPR-SJ

READY2APPLY F1-MGOAP-SJ

APPLYING F1-MGTAP-SJ

READYOBJ F1-MGOPR-SJ

READYTRANS F1-MGTPR-SJ

SET POST PROCESSING ALGORITHMS

The last step is to set the Post Processing algorithms on the Import jobs to instruct the Import Monitor
Batch Controls, listed below, to run multiple steps within its execution.

Post Processing Algorithm Values

BATCH CONTROL POST PROCESSING ALGORITHM

F1-MGOPR F1-MGTPR-NJ

F1-MGTPR F1-MGDIM-NJ

F1-MGOAP F1-MGDIM-NJ11

F1-MGTAP F1-MGDIM-NJ12

Use Auto-approve to Streamline Migrations


On the Migration Data Set Import there are a number of options that dictate the behavior of the import
process. One of the options available in Oracle Utilities Application Framework V4.3.0.5.0 and above,
is to automatically apply changes. For example:

11
For multi-lingual solutions, consider adding an additional Post Processing algorithm F1-ENG2LNGSJ to copy any missing
language entries
12
For multi-lingual solutions, consider adding an additional Post Processing algorithm F1-ENG2LNGSJ to copy any missing
language entries

99 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 24. Example Automatic Apply option

By setting Automatically Apply to Yes (Default is No) and used in conjunction with setting Default
Status For Add and Default Status for Change to Approved, synchronizes the changes for an import
automatically in a single step. This can allow for direct migrations of changes without the need for
approvals.

Store Exports in Code Repositories


Configuration Migration Assistant exports its configuration data to a file. This file can be loaded in an
source code repository with any associated code for developers to keep data with their code for
completion

Note: The file should not be altered manually after export as that may cause unexpected results.

Decide your Promotion Model


With the introduction of facilities like Configuration Migration Assistant the environments that are
regarded as sources, targets and the relationship between them should be expressed as a Promotion
Model. The would list all the environments you want to manage using Configuration Migration
Assistant and the linkages for flow of data between those environments. Once the model is agreed
then the export and import locations can be set accordingly to implement the flow of data.

For more information on promotion models, refer to the Software Configuration Management Series
(Doc Id: 560401.1) series of whitepaper available from My Oracle Support.

OPTIMIZATION TECHNIQUES
With the implementation of Oracle Utilities products on the cloud a series of optimizations were
implemented to reduce risk and reduce costs for those implementations. This section will summarize
the techniques and practices we have implemented to optimize the implementation for the cloud.
It is possible to optimize your
Note: Techniques discussed in this section are used for Oracle Cloud implementations and are implementation based upon
discussed in generic terms to make them possibility applicable to on-premise, PaaS or IaaS techniques used by partners and
the Oracle Utilities Cloud Service
implementations.
to reduce costs and risk.

Consider Standard Images


One of the first activities performed with the product is to install it. Typically this is done manually and
repeated for each successive environment. While the installation technology has improved over each
progressive release, the manual process can take time and is prone to manual errors. In the cloud
manual installation, is impractical once you scale across installation. Therefore an alternative
approach is required.

One technique that is popular across cloud implementations of all sorts of products is known as

100 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
imaging. The general idea is that you perform one manual install and use that install as a master copy
used to build other environments using appropriate imaging technology such as Docker, LXC etc.

Note: It is also possible to use the Domain Template Builder shipped with Oracle WebLogic to help
standardize the images if no imaging technology is available.

When using the imaging technology consider the following:

 Standardize your technical configuration. When you create an image to be reused you will want
to minimize the effort of configuration once that image is used, therefore standardize all the settings
in the technical configuration so that they are reused across instances. This extends to domain
settings, behavior settings and any technical connection settings. Whilst some settings need to be
different for each instance, those can be handled with automation or template changes.

 Keep it simple. To maximize reuse and reduce costs, keep your configuration as simple as
possible with little variation. Simpler configurations are easier to manage and have less
maintenance costs.

 There are some instance parameters. Whist most of the behavior settings (e.g. expressed in
properties files) can be standardized then there will always be some configuration settings that will
have to differ. The table below lists the most common instance specific settings.

Instance Specific Setting

ENVIRONMENT SETTING USAGE

BSN_WLHOST This is the host name for Business Application Server. The value of
localhost may be used if desired. If you are housing more than one domain
per host then you will also need to alter port numbers as well. Refer to the
Server Administration Guide for port numbers.

COHERENCE_CLUSTER_* If you are not using the Batch Edit facility then you can set the coherence
cluster information for the instance.

DBNAME This is the database service name. This contains the Pluggable Database or
Database instance name.

DBSERVER This is the host name for the database server.

DESC This is the description of the environment used for documentation purposes
only.

ENVIRONMENT_ID This is an unique identifier used for Oracle Enterprise Manager to link
disparate installations into a single environment.

SPLENVIRON This is the identifier for the environment

WEB_ADMIN_SERVER This is the host name for the administration server

WEB_WLHOST This is the host name for Web Application Server. The value of localhost
may be used if desired. If you are housing more than one domain per host
then you will also need to alter port numbers as well. Refer to the Server
Administration Guide for port numbers.

WLS_DOMAIN_HOME This is the domain home used for the instance. If you have multiple domains
on a machine then his needs to be populated.

101 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 Oracle Enterprise Manager Support. Oracle Enterprise Manager can be used to orchestrate
common processes including data migration and domain migrations.

The process is summarized as follows:

 Create Image. Use the imaging technology to create an image to reuse. You do not have to install
imaging software as creating a tar/zip of the base environment is a basic image technique.

 Use Domain Builder. Use the Oracle WebLogic Domain Builder to create a reusable domain
template.

 Copy Image to Target location. Copy the image to the target location. If the copy is done manually
then update the /etc/cistab file.

 Create Target Domain. Use the Oracle WebLogic Configuration Wizard to create a new domain
using the domain template created earlier.

 Reconfigure Product. The product configuration files need to be realigned to the new locations,
host names and port numbers. This can be updates to ENVIRON.INI and then reflected using the
initialSetup.sh utility.

 Deploy Product into Domain. Using the Oracle WebLogic console, wlst, Oracle Enterprise
Manager or Oracle Fusion Middleware Control, deploy the product Java EE files to the newly
created domain.

Note: For the database component, it is recommended to use the database tools within Oracle
Enterprise Manager.

The following figure illustrates the process:

Figure 25. Imaging Process

Note: If your environment contains Java based extensions, then use the SDK utilities to create a
release or patch before migration and reapply after migration, if necessary.

Architecture Optimizations
Note: For a full set of guidelines for architectures it is highly recommended to use the availability
guidelines as expressed in the Oracle Maximum Availability Architecture.

The architecture decisions of an implementation help determine the operational aspects of the
solution. To maximize the efficiency of the architecture and improve flexibility the following architecture
optimizations should be considered:

102 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 Implement channel based architecture. One of the major features of the Oracle Utilities
Application Framework is a set of channels that represent the different ways of interacting with the
product. The channels can be separated, or combined, to support flexible level of configurations
across the Oracle Utilities spectrum. In nimble implementations, each channel can be clustered or
managed as a separate unit using Oracle WebLogic Clusters and Oracle Coherence Clusters. This
has many advantages:

– Improved channel sizing. It is possible to size individual channels to suit individual traffic
patterns. This allows you to reflect the proportions of capacity you want to devote to each
channel. For example, if you have a small call center and more integration, you can size those
channels accordingly.

– Dynamic capacity. One of the big advantages of this architecture in the cloud is the ability to
distribute capacity dynamically as traffic patterns change. For example, it is possible to distribute
more online during peak office hours and then divert some of that capacity to batch overnight.

– Taking advantage of resource controls. One of the features of Oracle Utilities Application
Framework is that the database connectivity is now more traceable. Each channel can be
distinguished individually on the sessions view (v$sessions). While this is important for
monitoring, it is also useful for focused resource management. Within the Oracle Database a
capability called Resource Manager allows administrators to set limits on connections based on
their session attributes. This capability allows resource limits and other performance controls to
13
be enforce on database resources . For example, it is possible to reserve capacity for critical
channels at specific periods. This would ensure that channels do not steal resources from each
other and preserve performance as much as possible.

The following figure illustrates the channel architecture:

Figure 26. Channel Based Architecture

 Use domain based clustering. One of the major advantages of implementing this channel based
architecture is that channels can be clustered separately or combined into clusters as necessary.
The use of clustering tends to reduce costs as management is performance at the cluster level

13
Oracle Database Resource Manager control resources on the database server only but is an effective resource management
capability for Oracle Utilities products.

103 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
rather than individual server levels. Load balancers control the flow of transactions to ensure even
distribution and clusters are critical to high availability solutions. One of the other features that may
be practical is dynamic clustering, in which the cluster may start automatically additional capacity if
the capacity reaches a specific limit. This reduces server overwork and outage due to reduced
capacity at peak times.

 Using application delivery controllers to segregate traffic. One of the newest techniques
14
available is using an application delivery controller , proxy server or load balancer to control the
resources for each channel. The idea of this technique is that all the Oracle WebLogic based
channels are combined into a single cluster, to minimize costs, and then use configuration in the
relevant proxy server or load balancer to control the number of concurrent connections to each
protocol. The idea is that as each channel either uses a specific individual protocol or server URL
pattern that you can use the advanced features of a proxy server or load balancer to manage the
traffic into that architecture. This assumes that the application delivery controller, proxy server or
load balancer has this functionality and that providing the rules for that makes sense to reduce
costs.

Optimize Monitoring
Note: For a full list of metrics available, refer to the Server Administration Guide shipped with the
Oracle Utilities product, Oracle WebLogic MBean Reference and Database Monitoring documentation.

One the features of the Oracle Utilities Application Framework is that it uses existing infrastructure
metric frameworks and provides its own metrics to allow administrators to monitor the product using
standard protocols. With the vast number of overall metrics available it is not always efficient to
monitor everything all the time. The following techniques should be considered when optimizing
monitoring:

 Prioritize by importance. Decide the parts of the product that are important to your business users.
Once you know which parts are important then decide what you want to know about those
components to work effectively. For example, most sites want to know availability as a prime metric
so track that across all the critical components.

 Decide the underlying metrics. Once the components that are important are decided it is
important then to decide what, from the set of metrics that are available, are relevant to track. For
example, once you know that online is important to track, you would target availability metrics and
then critical performance metrics at a level that makes sense to you. As an administrator, you might
not be interested in metrics at the java class level that are available as that is too low level.
Generally start to monitor with more than you need and then you can reduce if necessary.

 Set service level targets. One of the most important aspects of monitoring is setting appropriate
targets. The setting of targets focusses the collection of metric to catch tolerances for critical
processes. The Oracle Utilities Application Framework allows the configuration of service level
targets in the form of Batch Level of Service and Performance Metrics objects to collect specific
tolerances and express them in various ways in the product.

 Decide the monitoring technology. In the cloud we provide a Cloud portal called Oracle Utilities
Cloud Service Foundation which displays preset metrics and allows some agreed controls for the
products in that service. Outside the cloud, Oracle Enterprise Manager (or Oracle Cloud Control)
can be used to collect metrics regularly and assess them against service level targets across the
architecture. Alternatively, running the standard logs, through appropriate log analyzation services

14
Such as Oracle Traffic Director or an equivalent.

104 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
or log analysis services, can trap errors and help work out that is happening on an ongoing basis.

 Do not forget history. While assessing the metrics at the present time are important, it is also
important to track trends. This means capturing data over a longer period of time to ascertain trends,
patterns and anomalies in the metric. This will allow the assessment of capacity and determine
regular traffic patterns associated with time or business functions.

Reduce Queries using Views


Note: This feature is explained in detail in the User Guide or online help shipped with the Oracle
Utilities product.

One of the features of the ConfigTools feature of the Oracle Utilities Application Framework is the
Saved Search feature. This feature allows authorized users to save a view of the query zone they are
on including the sorting, column placement (if supported by the zone type) and filters used. The user
then can recall that view at any time and even replace the default view used when the zone is loaded.
This not only allows a greater degree of personalization but it offers an opportunity for reducing some
costs. For example:

Figure 27. Saved Search feature

When query zones are built they usually preset the sorting, filters and column placement for a
particular process. In some cases implementers may build multiple zones to cover a variety of use
cases. If this is the case, then using Saved Search with a few guidelines can reduce your costs:

 Superset query elements. Consider combining all the columns, filters etc into a single query zone
and let the users design the specific view they want to use.

 Add extra column information. Consider adding additional columns or filters to make the zone as
generic as possible. This may save enhancing the query zone in the future by adding the ability to
display this information.

Note: There is a physical limit to the number of columns that can be defined for an individual query
zone so respect that limit when designing query zones.

Use CMA for all your migrations


As part of the cloud implementation, Java based extensions and database changes are not permitted

105 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
to reduce costs of the cloud implementation. There are a number of reasons why these types of
extensions are not permitted:

 Risk Reduction. For security reasons, access to low level resources such as the operating system
and database are not permitted in a cloud environment. Allowing Java based extensions and
database changes would be risky as they require that low level access.

 Cost Reduction. In a non-cloud environment you have three common ways of extending the
product. Extensions can be built in Java using the Oracle Utilities SDK, online using the browser
using the Oracle Utilities Application Framework ConfigTools component and/or making database
changes for custom objects. Each of those technologies uses a different method of deployment and
co-ordination across those methods can introduce both increased cost and risk. In the cloud to
reduce that risk and cost, all extensions are managed in ConfigTools and using generic objects
(such as FACT and others) to remove the co-ordination effort altogether and make the software
stack standard. The diagram below illustrates this co-ordination and the extension types:

Figure 28. Migration in traditional methods and on the cloud

To reduce risk therefore the cloud uses Configuration Migration Assistant exclusively to load the
environment (loading accelerators) and migrating all extensions across environments using the
following guidelines:

 Daisy chained jobs. The Configuration Migration Assistant jobs are setup so the minimum
numbers of jobs are executed to export and import the data across all environments.

 Auto-apply enabled. By default auto-apply is enabled, therefore data that is approved is


automatically applied without the need for additional manual approvals. If the migration is sensitive
this behavior can be overwritten on individual Migration Import Requests.

 Executed continuously. The Migration Export Monitor and Migration Import Monitor are configured

106 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
to run as Timed Jobs to run them in the background continuously. That means migrations are as
nimble as they can be.

 Focused security settings. Only approved users can create Migration Import Requests or
Migration Export Requests via the Oracle Utilities Cloud Service Foundation which is only included
in cloud implementations. This can be done on premise by setting up appropriate application
services to allocate to those functions.

 Standardized locations. The cloud implementation standardizes the locations of the export and
import files with a combination of shared locations for non-production and a separate remote
location for production use (for security reasons). These are set using the environment variables
provided by the Oracle Utilities Application Framework. Refer to the Server Administration Guide for
more details of the environment variables available.

Consider Automation
In the cloud world, any manual effort costs money and reduces effectiveness. Whilst the cloud
automation tools are not supplied with the product, automation is possible for customer's on-premise:

 Oracle Enterprise Manager. Oracle Enterprise Manager with the appropriate additional packs can
provide automation across the stack to cover deployment, incident management, software lifecycle
management, monitoring etc. It also includes an automation technology that can be used to
automate your processes. Refer to the following documents, whitepapers etc:

– Supercharging DevOps with Oracle Enterprise Manager

– Oracle Enterprise Manager Cloud Control Site

– Automated Patching using Oracle Enterprise Manager

– Advanced Uses of Oracle Enterprise Manager

– About Deployment Procedures

 Third party automation tools. The Oracle Utilities Application Framework and related technology
provides a set of interfaces using command lines, JMX, REST etc that can be used with third party
tools to automate the product processes.

PREPARING YOUR IMPLE MENTATION FOR THE CLOUD


One of the most common questions we get is how to prepare an on-premise application for
implementing into the Oracle Utilities Cloud offerings.

Note: Customers considering Platform As A Service or Infrastructure as a Service can use the same There are techniques that can be
principles as on-premise implementations. They may optionally follow these recommendations to save used at any time to ease the
transition from an on-premise
costs on those platforms.
implementation to the Oracle
Utilities Cloud Service.
Move extensions to the database
As pointed out in the last section under Use CMA for all your migrations it is recommended to move all
your extensions to the database to take advantage of the lower cost migration afforded by the
Configuration Migration Assistant. It is recommended that the following be done:

 Move Java extensions to ConfigTools. In the Oracle Utilities Cloud, Java is not available for use
for extensions for security reasons and to avoid additional costs due to deployment co-ordination
activities. Any existing Java code needs to be migrated to either Groovy or scripting using the

107 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
ConfigTools capability. Refer to the ConfigTools Best Practices (Doc Id: 1929040.1) available from
My Oracle Support for additional advice on building extensions with ConfigTools.

 Use Plug-In Batch for your batch extensions. Typically batch programs are written in Java using
the Oracle Utilities SDK. As with the last recommendation, Java is not available for batch programs.
It is recommended that any Java based batch processes be converted to use the Plug-In Batch
capability. A number of algorithms may need to be built to configure the new capability. Refer to the
ConfigTools Best Practices (Doc Id: 1929040.1) available from My Oracle Support for additional
advice on using the Plug-In Batch capability.

 Use common objects instead of custom tables. In the Oracle Utilities Cloud, changes to the
database are not permitted for security reasons and to avoid additional costs due to deployment co-
ordination activities. Any custom tables should use the generic objects provided with the product.
These objects are empty maintenance objects that can be customized, via business objects, to store
custom data, if necessary. Additionally characteristics and XML based CLOB's are still available on
product objects as a method of storing custom data for extensions. Refer to the online
documentation for details of all these capabilities.

 Convert XML Application Integration to Inbound Web Services. To reduce costs, it is highly
recommended to convert all the XML Application Integration based services into Inbound Web
Services. This will allow more flexible security options and consistency in integration. For more
details, it is recommended to read the Migrating from XAI to IWS whitepaper (Doc Id: 1644914.1)
and Web Services Best Practices whitepaper (Doc Id: 2214375.1) available from My Oracle
Support.

 Convert XSL to managed content. In the Oracle Utilities Cloud, access to the file system is
restricted so therefore any XSL file cannot be accessed on the cloud. To retain those file it is
recommended that the Managed Content capability in the Oracle Utilities Application Framework be
used to store the XSL file in the database. Refer to the online documentation for details of this
capability.

 Convert all file paths to use the File Storage Extendable Lookup . In the Oracle Utilities Cloud,
15

access to the file system is restricted, it is therefore recommended to parameterize existing file
paths using File Storage Extendable Lookup capability to allow easier migration of file paths to valid
paths for the Oracle Utilities Cloud. Generic file paths need to be configured on the Extendable
Lookup for F1-FileStorage. While on-premise, the Native File Storage Adapter may be used to
replicate existing capabilities. When migrating to the Oracle Utilities Cloud, this can be converted
over to the new location or using the Oracle Cloud Object Storage Adapter, if necessary. Convert all
FILE-PATH parameters and any configuration to use the lookup with the file-storage protocol.
Refer to the online documentation for details of this capability. An example of the File Storage
Extendable Lookup is shown below:

15
File Storage Configuration is available in Oracle Utilities Application Framework 4.3.0.6.0 and above.

108 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Figure 29. Example File Storage Extendable Lookup

Simplify your integrations


On the Oracle Utilities Cloud, the product becomes a black box that provides a series of services, via
SOAP and/or REST. When preparing to migrate to the Oracle Utilities Cloud, isolate your integrations
so that the on-premise product serves the same purpose to save costs in migrations using the
following techniques:

 Isolate your integrations outside the domain. All integration technologies such as JMS etc,
should not be in the product domain. These should be created outside the domain, in remote
domains for example, and product domain or JNDI entries configured to connect to that remote
domain. The domain setup for Oracle Utilities Cloud is predefined and cannot be extended.

 Consider using Oracle SOA Suite or Oracle Service Bus to drive your integrations. As the
product is a provider of services, you may need to use middleware of you need to orchestrate your
integrations. This separates the responsibility of the integration into segments with each participate
fulfilling a specific role. This would also improve integration visibility. When migrating to the cloud, it
is possible to house the Oracle SOA Suite/Oracle Service Bus on-premise or in the cloud via the
relevant Oracle Cloud Service.

 Consider Integration Cloud Service. One of the prebuilt integrations available to the Oracle
Utilities Cloud is the Oracle Integration Cloud Service. This service allows for prebuilt integrations or
simple orchestrations to be used for integrations between hybrid and cloud implementations. Refer
to the Oracle Integration Cloud Service for more information.

 Staging programs need to use Plug-In Batch. Imports and Export programs are traditionally been
built in Java using the Oracle Utilities SDK. In the Oracle Utilities Cloud, these Java programs must
use Plug-In Batch (or use the JCA/File capabilities of the Oracle SOA Suite/Oracle Service Bus) to
provide a mechanism for importing and exporting data.

 Factor in Oracle Cloud Agent. If the integration source remains on-premise consider that the if the
integration uses any Oracle Cloud offering as a way of integrating to that source that an Oracle
Cloud Agent must be installed to provide a secure method of allowing the Cloud to connect to on-
premise resources.

Remove operational extensions


The Oracle Utilities Cloud provides a prebuilt operational environment via the Oracle Utilities Cloud
Service Foundation where some basic operational capabilities are permitted for specific environments.
Any operational extension, such as custom UNIX scripts, templates etc are not available in the Oracle

109 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
Utilities Cloud environment. A change of operations is required using the following recommendations:

 Become familiar with capabilities of the Oracle Utilities Cloud Service Foundation. The Oracle
Utilities Cloud Service Foundation will become the operations workbench for day to day information
about your Oracle Utilities Cloud Service and the interactions with that service.

 Become familiar with the operational REST API. The Oracle Utilities Cloud Service exposes a
number of metrics and capabilities via a REST based API. This API is used by the Oracle Utilities
16
Cloud Service Foundation but may also be used externally for operational integration .

 Rationalize your operation utilities. As part of the migration to the Oracle Utilities Cloud Service,
some operational aspects must be rationalized that are provided as part of the service and alter your
existing operations to use the new API set or use the Oracle Utilities Cloud Service Foundation.

Rationalize your environments


In the Oracle Utilities Cloud Service a preset number of environments are provided as part of the
service. Additional environment can be purchased or use alternative platforms. When migrating to the
Oracle Utilities Cloud Service review your existing environments for migration to the cloud and whether
an existing provided environment can fulfil expectations or additional environments are necessary.
One of the drivers to the Oracle Utilities Cloud Service is reduction in costs so examine the need for
any additional environment carefully in terms of both benefit and costs before committing to additional
environments.

Generic environment management advice is available from the Software Configuration Management
Series (Doc Id: 560401.1) available from My Oracle Support.

Think differently for extensions


The Oracle Utilities Cloud Service typically offers a lower cost of ownership over an on-premise
implementation of the corresponding product. One of the reasons of the cost savings is that the Oracle
Utilities Cloud Service offers additional prebuilt configuration to speed up implementation. To take
advantage of the cost savings the way that implementations extend the product should consider the
following:

 Consider what the base delivers. In most on-premise implementations, the base install comes
with an initial installation which contains the basics for the implementation to build upon. In the
Oracle Utilities Cloud Service, a cloud accelerator can also be loaded to offer more options and
flexibility. Examine what is delivered in the cloud installation and reuse as much of the base as
possible, including replacing already used extensions with base functionality, to minimize costs.

 Keep it simple. As pointed out in the other advice, the cloud should be considered a product with
set interfaces both internal and external. By considering the product as a product, rather than a tool
to manipulate to your business requirements, costs can be kept to a minimum through reduction of
complexity. For example, interfaces can treat the API as a service interface and separate
technology and middleware out as capability.

 Remember the Cloud is a specific implementation of the product. It is important to remember


that the Oracle Utilities Cloud Service is a specific implementation of the product. The functionality is
not restricted though the implementation options are focused on reduction of complexity and
reduction of costs.

16
External consoles must be able to support the REST protocol with the security policies implemented on the Oracle Utilities Cloud
Service.

110 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
 No build necessary. With the use of the Oracle Utilities SDK, the concept of builds was relevant.
This is due to the deployment aspect of Java and related technologies. In the Oracle Utilities Cloud
world, builds become irrelevant as all the extensions are built live on the development environment.
To exhibit some level of control developers needs to understand the ConfigTools development
environment and its important differences:

– Use Revision Control. If version control is required then the Oracle Utilities Application
Framework includes a Revision Control capability that support team management with check in
and check out capability. Refer to the online documentation for details of this capability.

– Use Configuration Migration Assistant for migrations. To migrate extensions from one
environment to another use Configuration Migration Assistant. You can perform checkpoints and
spot migrations.

– Only use Bundling for non-production environments. The Bundling facility is designed for use
for individual objects or a small number of objects. It is not available for production use.

111 W HITE PAPER / Technical Best Practices - Oracle Utilities Application Framework (4.3.0.6.0)
ORACLE CORPORATION

Worldwide Headquarters
500 Oracle Parkway, Redwood Shores, CA 94065 USA

Worldwide Inquiries
TELE + 1.650.506.7000 + 1.800.ORACLE1
FAX + 1.650.506.7200
oracle.com

CONNECT W ITH US
Call +1.800.ORACLE1 or visit oracle.com. Outside North America, find your local office at oracle.com/contact.

blogs.oracle.com/theshortenspot linkedin.com/in/theshortenspot twitter.com/theshortenspot

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the contents hereof are
subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed
orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any
liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be
reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or
registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks
of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0618
OUAF Technical Best Practices - Oracle Utilities Application Framework
June 2018

You might also like