Tuning SAP Oracle AIX v.2.0

IBM Americas ATS
IBM SAP Technical Brief
Tuning SAP with Oracle on AIX
Mark Gordon IBM Solutions ATS
Version: 2.0 Date: May 3, 2013
2013 International Business Machines, Inc.
IBM Americas ATS 1. ACKNOWLEDGEMENTS ................................................................................................................ 5 1.1. 1.2. 1.3. 2. 3. 4. 5. 6. 7. VERSION 2.0 ................................................................................................................................... 5 VERSION 1.2 ................................................................................................................................... 5 INITIAL VERSION............................................................................................................................. 5
DISCLAIMERS ................................................................................................................................... 5 TRADEMARK..................................................................................................................................... 5 FEEDBACK ......................................................................................................................................... 6 VERSION UPDATES.......................................................................................................................... 6 INTRODUCTION................................................................................................................................ 7 ORACLE CONCEPTS........................................................................................................................ 9 7.1. 7.2. 7.3. 7.4. LOCAL AND JOIN PREDICATES ........................................................................................................ 9 EXPLAIN ......................................................................................................................................... 9 ACCESS AND FILTER PREDICATES ................................................................................................. 10 CHECK FOR EXECUTION PLAN IN CACHE ........................................................................................ 12
8.
ANALYZING A PROBLEM WITH A SPECIFIC PROGRAM OR TRANSACTION ............ 14 8.1. COMPONENTS OF SAP RESPONSE TIME ......................................................................................... 14 8.2. MAJORITY OF TIME IS CPU ON APPLICATION SERVER ................................................................... 15 8.2.1. Performance problem is high CPU time.............................................................................. 15 8.2.1.1. Other common ABAP problems.................................................................................. 20 8.2.1.2. Summary of Performance problem is high CPU time ................................................. 20 8.3. MAJORITY OF TIME IS DATABASE REQUEST TIME ......................................................................... 20 8.3.1. Types of problems causing high DB request time................................................................ 20 8.3.1.1. Indexes do not support predicates................................................................................ 20 8.3.1.2. Misuse of SAP Data Model ......................................................................................... 21 8.3.1.3. SELECT in LOOP instead of FOR ALL ENTRIES.................................................... 21 8.3.1.4. Data skew causes wrong execution plan to be chosen................................................. 22 8.3.1.5. Invalid or missing Oracle statistics .............................................................................. 22 8.3.1.6. Range Predicates cause wrong access plan to be chosen............................................. 22 8.3.1.7. Unnecessary SQL......................................................................................................... 23 8.3.1.7.1. Table could be buffered on application server but is not buffered .............................. 23 8.3.1.7.2. SAP instance buffers are too small .............................................................................. 23 8.3.1.7.3. FOR ALL ENTRIES with empty internal table .......................................................... 23 8.3.1.7.4. Invalid parameters in ABAP SQL Call........................................................................ 23 8.3.1.7.5. SQL is not needed for business process....................................................................... 23 8.3.2. FBL3N indexes do not support local predicates............................................................... 24 8.3.2.1. Summary of FBL3N .................................................................................................... 28 8.3.3. Program performs too many SQL operations...................................................................... 29 8.3.3.1. Summary of Program performs too many SQL operations ......................................... 30 8.3.4. Invalid parameters cause unnecessary SQL ........................................................................ 30
2013 International Business Machines, Inc. Page 2
IBM Americas ATS 8.3.4.1. Summary of Invalid parameters cause unnecessary SQL............................................ 32 8.3.5. MB51 Index does not support business process requirements ......................................... 32 8.3.5.1. Summary of MB51 Index does not support business process requirements ............ 35 8.4. MAJORITY OF TIME IS NOT CPU OR DB REQUEST TIME ................................................................. 36 9. SYSTEM PERFORMANCE PROBLEMS ..................................................................................... 37 9.1. PERFORM SQL CACHE ANALYSIS ................................................................................................. 37 9.1.1. Indicator of inefficient SQL ................................................................................................. 37 9.1.2. SQL Bget ratios that are generally not problems ................................................................ 38 9.1.3. Local predicates do not match indexes................................................................................ 39 9.1.4. Reasons that the predicates do not match indexes............................................................... 40 9.1.5. Actions when predicates do not match indexes.................................................................... 41 9.1.6. High Impact SQL caused by missing indexes on custom table............................................ 42 9.1.6.1. Summary of missing custom index.............................................................................. 44 9.1.7. High-impact SQL caused by incorrect use of SAP data model ........................................... 45 9.1.7.1. Summary of incorrect use of data model ..................................................................... 47 9.1.8. High-impact SQL when optimizer chooses wrong execution plan ...................................... 47 9.1.8.1. Summary of wrong choice of access path.................................................................... 52 9.1.9. High-impact SQL is a symptom of SAP buffer problem ...................................................... 52 9.1.9.1. Summary of High-impact SQL is a symptom of SAP buffer problem........................ 57 9.1.10. High Impact Unnecessary SQL............................................................................................ 58 9.1.10.1. Summary of High Impact Unnecessary SQL............................................................... 59 9.1.11. High Impact SQL is a symptom of Oracle statistics problem.............................................. 59 9.1.11.1. Summary of access path problem caused by statistics................................................. 62 9.1.12. High impact SQL caused by inefficient ABAP programming.............................................. 62 9.1.12.1. Summary of High impact SQL caused by inefficient ABAP programming ............... 64 9.1.13. High impact SQL caused by range predicate estimate........................................................ 65 9.1.13.1. Summary of High impact SQL caused by range predicate estimate ........................... 68 9.1.14. SQL not needed because of business customization ............................................................ 69 9.1.14.1. Summary of SQL not needed because of business customization............................... 72 9.2. CREATE CANDIDATE LIST FROM ST03N........................................................................................ 73 9.2.1. ST03N with table statistics................................................................................................... 73 10. SYSTEM HEALTH CHECK ....................................................................................................... 74 10.1. CPU ACTIVITY .......................................................................................................................... 74 10.2. I/O ACTIVITY ............................................................................................................................ 77 10.2.1. Good I/O performance in Oracle......................................................................................... 78 10.2.2. Symptom of I/O hotspot on disk ........................................................................................... 78 10.3. ORACLE REVIEW....................................................................................................................... 79 10.3.1. Concurrent I/O..................................................................................................................... 80 10.3.2. Database Hit Rate................................................................................................................ 80 10.3.3. Database delays ................................................................................................................... 81 10.4. SAP BUFFERED TABLE STATISTICS ............................................................................................ 83 10.4.1. Not buffered but could be buffered ...................................................................................... 83 10.5. EVALUATE MEMORY IN AIX ..................................................................................................... 84 2013 International Business Machines, Inc. Page 3
IBM Americas ATS 10.5.1. 10.5.2. 10.5.3. 10.5.4. 10.5.5. 11. 11.1. 11.2. 11.3. 11.4. 12. 64K page use........................................................................................................................ 84 AIX Restricted Tunables concept ......................................................................................... 84 AIX Alternative Memory Management ................................................................................ 84 AIX paging ........................................................................................................................... 85 Evaluating increasing memory for Oracle or SAP .............................................................. 88 USE THE SAP DATA MODEL ...................................................................................................... 91 USE ARRAY OPERATIONS ON THE DATABASE............................................................................. 91 CHECK WHETHER THE DATABASE CALL CAN BE AVOIDED ......................................................... 91 WRITE ABAP PROGRAMS THAT ARE LINE-ITEM SCALABLE ...................................................... 91 APPENDIX: REFERENCE MATERIALS................................................................................. 93
FOUR GUIDELINES FOR AVOIDING PERFORMANCE PROBLEMS............................. 91
IBM Americas ATS
1. Acknowledgements
1.1. 1.2. 1.3. Version 2.0 Version 1.2 Initial Version
Thank you to Damir Rubic and Steve Poon, who reviewed and proposed improvements.
Thank you to Damir Rubic and Steve Poon, who contributed new material on AIX and Oracle.
Thank you to Walter Orb of IBM Germany, who was for several years the pSeries SAP performance lead for IBM Americas Solutions ATS, and who showed me many of the processes used in this paper. Thank you to Marty Carangelo, Dale Martin, and Ralf Schmidt-Dannert, for their contributions on Oracle and pSeries performance. Thank you to Phil Hardy and Damir Rubic, who reviewed the paper and offered many improvements.
2. Disclaimers
IBM has not formally reviewed this paper. While effort has been made to verify the information, this paper may contain errors. IBM makes no warranties or representations with respect to the content hereof and specifically disclaims any implied warranties of merchantability or fitness for any particular purpose. IBM assumes no responsibility for any errors that may appear in this document. The information contained in this document is subject to change without any notice. IBM reserves the right to make any such changes without obligation to notify any person of such revision or changes. IBM makes no commitment to keep the information contained herein up to date. Most examples in this paper are based on ECC 6 with Oracle 11. The processes and guidelines in this paper are the compilation of experiences analyzing performance on a variety of SAP systems. Your results may vary in applying them to your system. Examples have been created and/or edited to clarify points for the paper.
3. Trademark
ABAP, SAP, R/3, and SAP Netweaver are the trademark(s) or registered trademark(s) of SAP AG in Germany and in several other countries. IBM, RS/6000, System p, pSeries and AIX, AIX/L, AIX/L, Total Storage, Power PC, Power VM, are registered trademarks of IBM Corporation. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. UNIX is a registered trademark of The Open Group.
IBM Americas ATS
4. Feedback
Please send comments or suggestions for changes to gordonmr@us.ibm.com.
5. Version Updates
Version 1.0 initial version Version 1.1 add SM04 Sessions Version 1.2 o Rename document to Tuning SAP with Oracle on AIX o added Section 8.3.1 (and sub-sections) types of DB access problems o added SQL cache example Section 9.1.6 o added invalid statistics example Section 9.1.11 o removed BDCPV example of access path choice problem, and replaced with Section 9.1.8 and LTAP/LTAK example o added ST06N and AIX virtualization CPU display in Section 10.1 o Added Section 9.1.10 empty FOR ALL ENTRIES internal table o Added SAP memory management (Sections 10.5.1 and 10.5.3) o Added Oracle related information in Sections 10.3.1. Version 2.0 major updates for ECC6 and Oracle 11 o Added Section 7 Oracle Concepts. o Updated Section 8.2.1 - Performance problem is high CPU time. o Updated Section 8.3.2 with FBL3N example to replace MIRO. o Updated Section 8.3.3 - Program performs too many SQL operations. o Added Section 8.3.4- Invalid parameters cause unnecessary SQL. o Added Section 8.3.5 - MB51 Index does not support business process requirements. o Updated Section 9.1.7 with new VBRP example. o Added Section 9.1.8- High-impact SQL when optimizer chooses wrong execution plan. o Updated Section 9.1.10- High Impact Unnecessary SQL. o Added Section 9.1.12 - High impact SQL caused by inefficient ABAP programming. o Added Section 9.1.13 - High impact SQL caused by range predicate estimate. o Added Section 9.1.14 - SQL not needed because of business customization. o Added Section 9.2.1- ST03N with table statistics. o Removed sections on summaries of performance tools.
IBM Americas ATS
6. Introduction
There are two intended audiences for this paper Oracle DBAs and SAP BASIS administrators. Either may be doing performance analysis on an SAP system with Oracle on AIX. The goal of this paper is to provide each audience with material that is useful and new: An SAP Basis administrator experienced with other databases should find the ORACLE specific tuning tools and techniques helpful, while the experienced ORACLE administrator is presented with SAP specific tuning tools and techniques. This paper covers the two most common types of performance problems database performance and inefficient ABAP coding. While there are other causes of problems in SAP (e.g. network performance, external RFC interfaces, SAP instance configuration, SAP sort, etc), database and ABAP performance are the most common and generally have the biggest impact. In order to provide the most benefit in the smallest paper, these other issues are not included in this paper. This paper has a process-based approach, where different goals are pursued via different processes and tools. To fix a problem reported for a specific program, we will perform elapsed time analyses of programs, determine where time is spent, and optimize these long running parts. This includes interpretation of ST03N and STAT/STAD records, and using ST05 and SAT. The paper will demonstrate how to use the SAP stats to obtain database performance statistics, identify I/O bottlenecks and SAP problems, etc. The benefit of this approach is that it is focused on an area that has been identified as a business problem. To check for inefficient use of DB resources and improve overall database server performance, we will use ST04 statement cache analysis. The value of this approach is that it offers a benefit in reducing resource usage and increasing system efficiency. The disadvantage is that one may be finding and solving problems that no end-user cares about. For example, if we can improve the elapsed time of a batch job from 2 hours to 10 minutes, but the job runs at 2:00 AM, and nobody needs the output until 8:00 AM, it may not really be a problem. Even if it is not a business problem, there may still be a benefit to addressing a problem of this type as part of optimizing resource consumption, in order to reduce the computing resources required to support business requirements. To do a system health check, review AIX paging and CPU usage, Oracle I/O statistics, ST10 and ST02 buffering. The operating environment needs to be running well for good performance, but problems in these areas can be symptoms of other problems. For example, inefficient SQL can cause high CPU usage or high I/O activity. Therefore, a health check should be done together with analysis of SQL and ABAP problems. This paper has many examples, and it describes what is good or bad in each example. There are not always specific rules given on what is good or bad, such as Database request time over 40% of elapsed time is bad and under 40% is good. Rather, this paper tries to focus on an opportunity-based approach, such as: Look for where a program (or the SAP and database system) spends time. Ask If I fix a problem in this area, will people notice and care that it has been fixed? 2013 International Business Machines, Inc. Page 7
IBM Americas ATS It will discuss how to estimate the impact of solving a problem. System wide performance analysis (such as a statement of cache analysis, or ST03 analysis) will generally turn up several candidates. By estimating the impact of fixing these problems, one can decide which to address first. When doing this analysis, it is important to identify and track specific issues. Often, a performance issue may not have enough impact to merit a new index, or an ABAP change. In this case, we want to track that we have analyzed it, and chosen not to do anything, so that we dont waste time discovering it again next year. This paper refers to a number of SAP Notes. An OSS userid, or userid that allows access to service.sap.com, is a prerequisite for anyone doing performance analysis on an SAP system, whether the person is an Oracle DBA, AIX administrator, or SAP BASIS Administrator. This paper breaks performance management into three parts, which are discussed in Sections 8, 9, and 10: Analyzing a problem with a specific program or transaction System performance problems System Health Check. Since AIX-level symptoms such as paging, excessive CPU use, and high I/O rates can be symptoms of application and SQL problems, one should start by reviewing the SAP and Oracle indicators, before taking action based on the AIX level indicators.
IBM Americas ATS
7. Oracle Concepts
Here, we have a brief review of some of the concepts and terms that will be used for Oracle.
7.1.
Local and Join Predicates
Local predicates on columns qualify the rows to be returned to the application. Join predicates specify the relationship between tables used to join rows in the tables.
Figure 1: SQL with local and join predicates
In Figure 1, T_00.MANDT=:A0 is a local predicate, and T_00.MBLNR=T_01.MBLNR is a join predicate. Local predicates are called filtering or selective when they return few rows. Selecting a single document by its document number (where BELNR = ) would be return few rows. Selecting all the rows in a client (where MANDT = ) is not selective, since there is generally only one client on a system.
7.2.
Explain
We can explain SQL to evaluate whether it is executed efficiently. The explain output will provide some clues that we can use to determine whether the SQL can be improved.
Figure 2: Explain example
IBM Americas ATS In Figure 2, note that the estimated cost is much higher than #rows and the estimated CPU cost is also very high. These are Oracles estimate of the cost of executing the statement, not the actual runtime statistics. When the cost is very high relative to the rows, it can point to a problem where Oracle cannot retrieve the result efficiently. An example is when the indexes do not support the local or join predicates.
7.3.
Access and Filter Predicates
Access Predicates are evaluated in the index. The more selective they are, the fewer index and table pages to be retrieved. Filter predicates are applied for index or table rows on the buffer pages retrieved for the access predicates. It is much more efficient to apply local predicates as access predicates than as filter predicates. In order to find out why the statement cost estimate is high, in Figure 2 drill into the Access Predicates field or the Filter Predicates field under INDEX SKIP SCAN RFBLG~0 to display the Oracle access and filter predicates used on the index.
Figure 3: Access and Filter Predicates
We can see that the reason that this statement has a high cost is that the local predicates on BELNR and GJAHR are applied as filter predicates on the index pages that have been retrieved for the MANDT access predicate. In order to determine why local predicates are processed as access predicates or filter predicates, we need to check the indexes on the table. Drill into the RFBLG~0 field in Figure 2 to display the columns in the index.
Figure 4: RFBLG~0 Index
When we compare the index columns in Figure 4 with the local predicates in Figure 2, we see that the local predicates match the first, third, and fourth columns of the index. Since there is no local predicate on the second column, Oracle used index skip scan, and has to apply GJAHR= and BELNR= as filter predicates. 2013 International Business Machines, Inc. Page 10
IBM Americas ATS This example is actually a logic error in the program. As we can see from the columns in the unique index RFBLG~0 shown in Figure 4, BELNR is unique only when combined with BUKRS and BELNR. If BUKRS is not specified in the SQL, it is possible to retrieve more than one document. If the SQL has local predicates on MANDT, BUKRS, BELNR, GJAHR as shown in Figure 5, then Oracle uses index range scan and processes all the local predicates as access predicates, and the cost estimate is much lower than in Figure 2. There are no Filter Predicates. Performance will be better.
Figure 5: Explain with added local predicate on BUKRS
IBM Americas ATS
7.4.
Check for execution plan in cache
When you explain a statement, Oracle will attempt to display the execution plan that is being used. If that is not possible, you will see the following message:
Figure 6: Explain cannot display cached plan
If Oracle cannot show the cached execution plan, it is possible that the plan you are seeing in the explain output is different from the plan that was used when the statement was executed.
IBM Americas ATS If Oracle can display the cached execution plan, you will see a message in the explain output similar to the following:
Figure 7: Explain with cached execution plan
IBM Americas ATS
8. Analyzing a problem with a specific program or transaction

If a problem has been identified, then the first step is to determine where most of the response time is spent. Based on this first step, there are different tools that are used to monitor the ABAP, database, RFCs, etc.
8.1.
Components of SAP response time
SAP note 364625 describes the different components of SAP dialog step response time. This paper is focused on programs that have high database request time, or high CPU time. These are the two most frequent symptoms of performance problems.
Figure 8: STAD transaction display of time components
IBM Americas ATS
8.2.
Majority of time is CPU on application server
When the majority of time is CPU on the application server, we will use SAT (formerly SE30) to analyze the ABAP program.
8.2.1. Performance problem is high CPU time

As an example, assume that we have been asked to investigate the performance of ZSUMMARY. We check ST03N as shown in Figure 9.
Figure 9: ST03N ZSUMMARY
Average times are somewhat slow (0 Time column is 174 seconds per dialog step), with CPU on the application server (0 CPU column) over ten times length of database request time (0 DB Time column). Since CPU time is the majority of elapsed time in the example in Figure 9, use SAT to trace the ABAP runtime. We arrange for a user to run ZSUMMARY so we can trace it. Run SAT and click the Own Standard Variant button and Change.
Figure 10: SAT own standard variant selected
IBM Americas ATS Then press change (the pencil) in Figure 10 to change the own standard variant options to set default settings.
Figure 11: SAT gather stats on internal tables
Enable statistics for internal tables as shown in Figure 11 By default these are off, but inefficient internal table accesses are a very common source of high CPU use in custom ABAP programs. In the Duration/Type tab, select Per Call Position.
Figure 12: SAT Aggregation level
In Figure 12, select Aggregation level by call, so that different calls on the same table are reported separately. Aggregation level None will cause the trace to grow very big very quickly, but is used if you need to determine the hierarchy relationships between calling and called programs. 2013 International Business Machines, Inc. Page 16
IBM Americas ATS Save and go back to the main screen shown in Figure 10. Then press Switch On/Off on the SAT main screen to show the list of work processes here in Figure 13.
Figure 13: SAT start/end measurement
Select the process to be traced, and press Start measurement, trace a while, then press End measurement. Return back to the main SAT screen shown in Figure 10. Press Format performance data, and select trace to be formatted. After the trace is formatted, you can evaluate it.
Figure 14: SAT analysis
Display the trace selected in the Evaluate tab.
Figure 15 : ZSUMMARY Desktop
As shown in Figure 15, the program spent over 98% of elapsed time processing Internal Tables on the application server. 2013 International Business Machines, Inc. Page 17
IBM Americas ATS Press the Times tab and then sort the list by Net to see where the time was spent.
Figure 16: SAT sorted by Net time
Select the slow Read Table statement in Figure 16 and press ABAP to see the ABAP.
Figure 17: ZSUMMARY source code
This is custom code it starts with Z. We will send it to the developers to fix it. The problem with this ABAP is shown below in Figure 19. You can calculate the time per READ TABLE using the Hit Lists button in Figure 14.
Figure 18: Hit Lists
In Figure 18, 308,697,854 uSec / 9645 calls = 32,005 uSec (or 32 ms) per call, which is very slow.
IBM Americas ATS READ TABLE calls should be a few microseconds. Inefficient access reading internal tables is a very common performance and scalability issue. In the SAT transaction, press the Tips and tricks button to see various suggestions on ways to improve ABAP programming. Tips and tricks contains several examples of common problems with internal tables.
Figure 19: SAT tips and tricks
One can review Tips and tricks for suggestions on ways to improve ABAP programs. In this case, the problem is that the ABAP ZSUMMARY shown in Figure 17 is doing a linear search of the internal table.
Figure 20: SAT binary search test
When a large internal table is read without the BINARY SEARCH option, the program becomes much slower as the number of items processed grows and the table grows. Hashed internal tables can also be used to improve the performance of READ TABLE.
IBM Americas ATS
8.2.1.1. Other common ABAP problems

Sorting an internal table too frequently, rather than using a sorted or hashed ITAB is another common performance problem that causes high CPU.
Figure 21: Frequent ABAP SORTs
High CPU use and slow performance of LOOP AT statements can also be improved by using SORTED or HASHED internal tables.
Figure 22: Slow LOOP AT
8.2.1.2. Summary of Performance problem is high CPU time

Check the response time components of the program. If a majority of time is spent in CPU time on the application server, use SAT to trace the program. We find that the majority of ABAP time is on a READ TABLE statement. This is a custom program, so we would send this to the development team to investigate ways to speed up processing the internal tables, such as using a hashed itab or sorted itab with BINARY SEARCH option on READ TABLE.
8.3.
Majority of time is Database Request time
If the majority of program or transaction elapsed time is DB time, the first question we need to ask is whether the DB time is high because of a problem in retrieving the data. There are a number of different problems that manifest themselves as high elapsed DB time.
8.3.1. Types of problems causing high DB request time
8.3.1.1. Indexes do not support predicates

Local predicates are the column operator value clauses in SQL. In order to execute these efficiently, the selective local predicate column must be in an index. If the column is not in an index, then Oracle must read data pages from the table to check whether the row contains the value which is sought. 2013 International Business Machines, Inc. Page 20
IBM Americas ATS Join predicates are the table1.columnA = table2.columnB clauses in SQL. Just as with local predicates, selective join predicates must be indexed for good performance. In many cases, the correct way to address this is to modify the ABAP so that it uses the indexes correctly. There is an example of this in Section 9.1.12. In some cases, new indexes are needed to support a business process that is not part of original SAP functionality. There are example of this in Sections 8.3.2, 8.3.5, and 9.1.6.
8.3.1.2. Misuse of SAP Data Model

SAP often stores data redundantly in more than one table. For example, a delivery document might contain the order number. And an order document might contain the associated delivery number. The delivery tables are indexed by delivery number, and order tables are indexed by order number, so if a program has the value of an order number and wants to retrieve the associated delivery, the program could theoretically use either table. But, if the program selects from the delivery table using order number, there is no index for order number and the select is slow. If the program selects from the order table, there is an index and the select is fast. There are three SAP notes (185530, 191492, and 187906) that describe common errors in use of the data model and how to fix them. There is an example of this problem in Section 9.1.7.
8.3.1.3. SELECT in LOOP instead of FOR ALL ENTRIES

If an ABAP program has a list of keys in an internal table, there are two ways to use the table to select rows from the database: LOOP AT internal_table SELECT from DB_TABLE where COLUMN = internal_table-column ENDLOOP
Figure 23: LOOP with select
Each SELECT call in the LOOP will make a call between the application server, and the database server. For better performance, an array operation should be used: SELECT from DB_TABLE for all entries in internal_table WHERE COLUMN=internal_table-column 2013 International Business Machines, Inc. Page 21
IBM Americas ATS The LOOP AT will make one call to the DB for each value, the FOR ALL ENTRIES will group the values together, and SAP will create an IN list (there is one column=value in the WHERE) or an OR (if there is more than one column=value clause in the WHERE. There is an example of this in Section 8.3.3.
8.3.1.4. Data skew causes wrong execution plan to be chosen

When Oracle estimates the number of rows that will be returned on an ABAP select, it uses the number of distinct values in a column and the number of rows in the table to estimate the number of rows with each column value. For example if a column AUFNR has 1,000 distinct values, and its table has 10,000 rows, then Oracle assumes that each value specified for AUFNR will return 10 rows. However, if many rows contain the same value, then the data is skewed -- that is the distribution of values in rows is not uniform. If data is skewed, Oracle requires additional histogram statistics, combined with an ABAP HINT, in order to use values at execution time to determine the skew, and choose a better execution plan. There is an example of this in Section 9.1.8.
8.3.1.5. Invalid or missing Oracle statistics

Cost based optimizers collect statistics about tables and indexes (number of rows, distinct values of columns, etc) that are used to optimize SQL statements. If the statistics are missing, or not collected in the correct way, the optimizer may choose the wrong execution plan. There is an example of this issue in Section 9.1.11.
8.3.1.6. Range Predicates cause wrong access plan to be chosen

Similar to the problem with skew, prepare with parameters can cause problems with range predicates. When preparing with parameters, Oracle uses estimates of filtering for range predicates, such as <. <=, BETWEEN, etc. If the runtime values are selective (e.g. date >= today), or not selective, (e.g. date <= 12/31/9999), then the Oracle estimate will be wrong, and a slow access plan may be chosen. As with Section 8.3.1.4, ABAP HINTs can be used to pass the values to Oracle, in order to get a better estimate and access plan. There is an example of this issue in Section 9.1.13.
IBM Americas ATS
8.3.1.7. Unnecessary SQL 8.3.1.7.1. Table could be buffered on application server but is not buffered
Tables that are seldom changed, and where the application can tolerate a small interval between when rows are changed and the changes are available on all application servers can be buffered on the SAP application server to offload the database. There is an example of this issue in Section 10.4.1.
8.3.1.7.2. SAP instance buffers are too small

If the SAP generic or single record buffers are too small, then tables that could be buffered on the application cannot fit into buffer, and must be read from the DB server. There is an example of this problem in Section 9.1.9.
8.3.1.7.3. FOR ALL ENTRIES with empty internal table

The ABAP FOR ALL ENTRIES statement uses an internal table with a list of keys to be retrieved for a column or columns. If the internal table is empty, the local predicates on the columns are not generated in the SQL, which generally causes the program to retrieve rows that it does not need. There is an example of this problem in Section 9.1.10.
8.3.1.7.4. Invalid parameters in ABAP SQL Call

If the ABAP does not correctly validate the inputs and SQL results at runtime, it is possible to have invalid parameters in the DB calls selection conditions that would never retrieve a row. There is an example of this in Section 8.3.4.
8.3.1.7.5. SQL is not needed for business process

Based on customization and business processes, some table columns may be used, or not, by SAP. If they are not used, they will contain no data, so selections from the table based on the column will never retrieve data. There is an example of this in Section 9.1.14.
IBM Americas ATS
8.3.2. FBL3N indexes do not support local predicates

We have been asked to investigate the performance of the FBL3N transaction. As a first step, check the response time characteristics of FBL3N using ST03N.
Figure 24: MIRO ST03N times
Figure 24 shows that the average FBL3N dialog step runs about 1 second (0 Time column), and about 90% of the elapsed time (0 DB Time column) is database request time. The 0 Time and 0 DB Time units are milliseconds. The T Response Time unit is seconds. While the average response time is good, there may be some long running dialog steps that have slow response time. In Figure 25 we use STAD transaction to display statistics records for FBL3N. Note that most have very short response time, but there is one that runs over two minutes (126,265 ms), where almost all the elapsed time is DB time.
Figure 25: STAD FBL3N
IBM Americas ATS Since DB request time is the majority of time (see Figure 25), and individual SQL calls may be long, arrange to have the transaction run by a user who is experiencing long response times, and trace the transaction with ST05. Here is the output of an ST05 SQL trace.
Figure 26: FBL3N ST05 trace
In Figure 26, note that there are some slow SELECTs on BSIS each takes over one second and return just a few rows. (The duration is in microseconds.) In general, a statement that is executed efficiently using indexes will take at most few milliseconds per row. Press the ABAP display button, which is the sheet of paper button in Figure 26, in order to see the ABAP source code.
Figure 27: FB3N ABAP display
The cursor will be positioned at the source line in the ABAP. In Figure 27, the ABAP is SAP code. You can also display the program attributes to confirm the owner of the program.
IBM Americas ATS
Figure 28: Program Attributes
Review the execution plan used by Oracle -- press the Explain button in Figure 26.
Figure 29: FBL3N BSIS
IBM Americas ATS Drill into Access predicates for BSIS~0 in Figure 29, to see which local predicates are applied in the index.
Figure 30: BSIS Access Predicates
MANDT, BUKRS, and HKONT are processed in the index. Drill into the table name BSIS in Figure 29, to display the indexes and indexed columns.
Figure 31: BSIS~0 Index
Comparing the local predicates in Figure 29 with the index columns in Figure 31, we see that none of the other local predicates on BUDAT or BLART can be evaluated in the index. Drill into Filter Predicates for BSIS table in Figure 29, to verify that BUDAT and BLART are evaluated on the table rows.
Figure 32: BSIS Filter Predicates
IBM Americas ATS Since the statement runs very slowly and returns few rows, we can infer that one or both of the the local predicates on BUDAT or BLART provides some filtering if MANDT, BUKRS or HKONT had provided all the filtering, the statement would run quickly, since evaluating access predicates in the index is very efficient. With efficient index access, it would have taken a few ms per row, instead of taking about one second per row. In Figure 29, compare #Rows on the index and table, to see Oracles estimate of the selectivity of the Filter Predicates on the table Oracle estimates that 697 rows will qualify from the index, but only one will qualify after the filter predicates on the table are evaluated. So, the problem seems to be that some of the local predicates in the SQL are not processed efficiently in this SQL. Since Figure 27 showed that the program is SAP code, we would: First, check the SAP data dictionary definitions Second, search SAP notes for this problem Third, open an OSS message In this case, we search SAP SMP and find that there is a SAP note that seems to address this issue it proposes an index on BSIS for (MANDT, BUKRS, HKONT, and BUDAT).
Figure 33: SAP note for FBL3N BSIS
8.3.2.1. Summary of FBL3N

The transaction was identified as a candidate to review. We checked the components of response time using ST03N, and found that the majority of time was DB time. The transaction was traced with ST05, and slow SQL was found. The slow SQL statement was initiated from an SAP ABAP program. We checked SAP SMP, and found a note that recommended a new index. We can either create the proposed index, or open a message to SAP to review this issue. 2013 International Business Machines, Inc. Page 28
IBM Americas ATS
8.3.3. Program performs too many SQL operations

A different example of inefficient SQL is when a program uses single row selects instead of array selects. SAP has a construct (FOR ALL ENTRIES) that uses an internal table in an ABAP to specify the key values to be fetched from a table in Oracle. The FOR ALL ENTRIES (FAE) can be executed as an IN list (if there is only one variable column in the internal table) or as an OR (if there are two or more variable columns in the internal table. In this example, we have been asked to review the performance of a program that is slow. In checking the stats for the program, we see that database request time (0 DB column) is long, nearly 40 minutes, but the average time for each sequential read is short, 2.5 ms.
Figure 34: ZFB001 program - lots of fast DB accesses
Average CPU time (0 CPU) is about of database request time (0 DB), so we need to look at the database accesses with ST05 to investigate. We trace the transaction with ST05, and then list the trace.
Figure 35: ZFB001 ST05
The program behavior is interesting. It repeatedly selects rows from the same table, BKPF. Display the ABAP source.
Figure 36: ZFB001 program source
IBM Americas ATS The program is looping through an internal table (LOOP AT t_bel). For each entry in the table, the program selects from BKPF. If it is possible to change the program to do a single FOR ALL ENTRIES select on BKPF from the internal table, the program would probably run much faster, since reading N rows at once from Oracle is generally faster than reading 1 row N times. Since the program is custom code (Zxx), we send it back to the ABAP team, to review whether the program could be restructured to use SAP array fetch operations.
8.3.3.1. Summary of Program performs too many SQL operations

ST03N statistics showed long database time and relatively small CPU time, but per-call sequential DB request times were good (2.5 ms). We used ST05 to trace the SQL and also confirm the SQL response times. The trace showed that the same table was repeatedly accessed. The ABAP program is looping through an internal table, fetching one row at a time from the database. The program needs to be re-evaluated, to determine if the single-row access to the database can be converted into an array operation, such as FOR ALL ENTIRIES.
8.3.4. Invalid parameters cause unnecessary SQL

Logic errors, such as not validating input parameters or the results of SQL can create the situation where SQL is executed with parameters that will never retrieve a row, since the parameters specify keys that cannot exist. This type of problem can be found when analyzing an SQL trace, or by examining the Bind variables (e.g. Figure 70) when a statement is explained in the Shared Cursor Cache. In this example, we have been asked to analyze the performance of the program ZCOITEMS, a custom program that runs slowly. The majority of elapsed time was DB request time, so we use ST05 SQL trace.
Figure 37: ZCOITEMS ST05
Note that Hits is zero on all lines no rows are retrieved. The value being sought is (space).
IBM Americas ATS We summarize the trace (Trace list > Summarize by SQL statement), to see if this occurs frequently.
Figure 38: ZCOITEMS summarize
And we see that 98% of the calls to LFA1 are identical.
Figure 39: ZCOITEMS summary
Press the page icon (Display Call Positions in ABAP Program) to display the ABAP source location.
Figure 40: ZCOITEMS ABAP
And we find the location in the ABAP which makes the calls. Last, we use SE11 to review the table (or call the developers), to see if space is a reasonable value to use in the program.
IBM Americas ATS
Figure 41: ZCOITEMS LFA1 display
LIFNR is the vendor or creditor account number. It does not seem to make sense to specify LIFNR = , since there is no vendor or creditor with the account number . If this had been a status flag, then it could make sense to search for a status of space. We return the program to the ABAP developers, to review how the data is retrieved and checked before the call to LFA1 is made.
8.3.4.1. Summary of Invalid parameters cause unnecessary SQL

The program ZCOITEMS spends the majority of its time in DB calls. We trace with ST05, and see that the program frequently selects from LFA1 where LIFNR = (space). Summary of the trace shows that duplicate selects on LFA1 are over 90% of the calls on LFA1. SE11 shows that LIFNR is a customer or vendor number, so it is unlikely that space is valid. The program is returned to development to review.
8.3.5. MB51 Index does not support business process requirements

In this example, we have been asked to review performance of MB51, which has been reported to have long runtimes.
Figure 42: MB51 performance statistics
Based on the ST03N data in Figure 42, the performance seems fine an average of 3.6 seconds. Well have one of the users run the transaction, and trace it. Since DB time is the large majority of time, we use ST05 SQL trace. 2013 International Business Machines, Inc. Page 32
IBM Americas ATS
Figure 43: MB51 ST05 summarized trace
We run the ST05 SQL trace and summarize the trace, and find in Figure 43 that there are some slow selects on MKPF. The first line shows 47 rows retrieved in 464 seconds. 10 rows per second is a very slow rate. If the indexes support the SQL, it should take at most a few milliseconds per row. Display the ABAP source, and we see that it is SAP standard code.
Figure 44: MB51 ABAP Source
The ABAP already contains the &SUBSTITUTE VALUES& HINT, so that the statement will be prepared with values each time it is executed. When preparing a statement with values, histograms must also be gathered on the table, for best access plan choice.
IBM Americas ATS Explain the statement in Figure 44, and note that the Oracle estimate for the MSEG branch of the join is much higher than the MKPF branch of the join, even though the #Rows estimate is similar. A custom index (Z00) is used on MSEG. We need to evaluate the index, to see if it has been created correctly to optimize access to the table.
Figure 45: MB51 with custom index used
First, drill into the Access Predicates on MSEG~Z00, to see which local predicates are applied in the index (Access Predicates) and which are applied on table rows in the data pages (Filter Predicates).
Figure 46: MB51 Access and Filter Predicates
The local predicates BWART BETWEEN and WERKS = are Access Predicates in the index, but WERKS = is applied again as a Filter Predicate after the table is read. We need to look at the definition of the index to see why this is happening. If the index was designed to support this 2013 International Business Machines, Inc. Page 34
IBM Americas ATS SQL, all three local predicates on MANDT, WERKS, and BWART would be Access Predicates and there would be no Filter Predicates.
Figure 47: MB51 Custom Z00 Index
In Figure 47, we can see the problem the BWART column (which has a range predicate BETWEEN) is ahead of WERKS in the index. For the most efficient index access, columns with equals predicates, such as MANDT and WERKS in this example, should be before columns with range predicates. In this example, the index does not efficiently support the local predicates the range predicate on BWART causes the WERKS = local predicate to be a Filtering Predicate so it is applied on index pages after accessing the index b-tree Assuming that this index is designed to optimize the business process of reporting on movement types (BWART) for a plant (WERKS), and that the end-user will generally specify a single plant, the best design for the index would be (MANDT, WERKS, BWART, MATNR). We want to move the column that has the range predicate (BWART) after MANDT and WERKS in the index. In all the other examples in this paper, we try to avoid creating custom indexes. The table in Section 9.1.5 shows the decision matrix. But, in some cases custom indexes are necessary to support a business process that is not supported by the default indexes. The business process reporting on all the materials (MATNR) in a plant (WERKS) is not supported by the default MSEG indexes. This index would enable that process to be done efficiently.
8.3.5.1. Summary of MB51 Index does not support business process requirements
We are asked to review MB51. ST03N statistics show good average response times, but when we trace execution for a user, we see long running selects on MKPF that return few rows. Explain shows that a custom index is being used, and that one of the local predicates is applied as a Filter Predicate, which is not as efficient as having all local predicates processed as Access Predicates. We review the index design and local predicates, and find that there is a range predicate on the second column of the index. We review the purpose of the index, and change the column order of the index to have the columns with equals predicates in the lead, followed by columns with range predicates.
IBM Americas ATS
8.4.
Majority of time is not CPU or DB request time
There are a number of SAP response time categories that can also indicate performance problems that are not covered in this paper. SQL and ABAP, which are covered in this paper, are the source of the vast majority of performance problems. The paper Tuning SAP on DB2 for z/OS on System z on IBM Techdocs (http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100287 ) contains detailed explanations about additional SAP sources of program delays (such as ENQ delay, SAP sort, local I/O, commit work and wait, RFC, GUI, network, etc), their symptoms and possible fixes.
IBM Americas ATS
9. System performance problems

9.1. Perform SQL Cache Analysis
In order to find inefficient SQL that may have a system-wide impact, because of using excessive CPU or doing excessive I/O, review the SQL cache. SAP note 766349 describes the columns in the ST04 Shared Cursor Cache display.
9.1.1. Indicator of inefficient SQL

The primary indicator shown in ST04 of inefficient SQL is when both Bgets/exec and Bgets/row are high. A Bget (buffer get) is when oracle references a page in Oracle database buffer. A high ratio for both of these indicators is a sign that Oracle must search a large amount of data to find the results. Searching many pages will consume extra CPU and may cause extra I/O operations. Normally, it might take 5 buffer gets to retrieve a single row from a single table, and it can be much fewer when performing array operations that retrieve many rows with a single call to the DB. SAP and Oracle display CPU and elapsed time statistics, as well as the efficiency (Bgets/row and Bgets/exec) statistics. The CPU and elapsed time data is useful for estimating the impact of the statement on the system or program.
Figure 48: ST04 display of SQL cache for Oracle 11
Note that if no rows are returned by a statement, Bgets/row is 0. When Bgets/row and Bgets/Exec are high, it is a sign of inefficient SQL. If Bgets/row is high, and Bgets/Exec is low, it means the statement is executed efficiently, but seldom returns a row.
IBM Americas ATS
Figure 49: ST04 SQL cache - problem example
In Figure 49, the first statement in the list looks like it is very inefficient each execution takes on the average 3.5 seconds to return 4.9 rows. There are 11,456 Bgets/exec and 2,239 Bgets/row, which is much higher than the 5 Bgets/row that one might see if the SQL were efficiently accessing the data.
Figure 50: ST04 SQL cache - moderately inefficient SQL
Even when the Bgets/row is lower than the example shown in Figure 49, if a statement is executed frequently enough it can cause an important impact on the system. By addressing and fixing several of these problems, one can make a measurable decrease in CPU usage on the system. The higher Bgets/row and Bgets/exec are, the more likely that the SQL is inefficient.
9.1.2. SQL Bget ratios that are generally not problems

When Bgets/exec is high and Bgets/row is low, SAP is doing array operations, which affect many rows per database call. This is generally an efficient way to access the database.
IBM Americas ATS When Bgets/exec is low and Bgets/row is high, this shows that the SQL seldom retrieves rows. If Proc rows is 0, check whether the table contains data, and if the statement is needed in the program. When Bgets/exec is low, and Bgets/row is low, then the data has been retrieved efficiently.
9.1.3. Local predicates do not match indexes

As described above in Section 8.3.1, one of the main causes of inefficient SQL is that the SQL local predicates generated from the ABAP do not match the available indexes. Local predicates are the WHERE COLUMN operator value clauses in the SQL. If the predicates contain columns that are not in indexes, or if the indexed columns are not selective (that is they return many rows) then Oracle generally cannot access the data efficiently.
Figure 51: Sample explain
In Figure 51, the local predicates are: MANDT = VGBEL =
IBM Americas ATS Drill into Filter Predicates on Figure 51, and we see in Figure 52 that all the local predicates are applied on the table. This is to be expected, since Oracle is using TABLE ACCESS FULL, rather than using an index.
Figure 52: VPRB Filter Predicates
One can drill into the table or indexes shown in Figure 51, to see the indexes on the table.
Figure 53: Display indexes from explain
Now that we have predicates and index columns, one can compare to determine if the local predicate columns are contained in indexes. Comparing Figure 51 and Figure 53, we see that the only indexed predicate column is MANDT. Since there is only one value of MANDT, Oracle must look at all the rows of the table, in order to find the rows matching the VGBEL = predicate.
9.1.4. Reasons that the predicates do not match indexes

There are three basic scenarios: The program is not using the SAP data model correctly, The ABAP needs to be changed to match the available indexes, and The business process may require a new index on the table Many kinds of information are redundantly stored in SAP. For instance, the transaction data for a billing document may contain columns with the document number for the delivery or sales order being billed. But just because the information is present does not mean that is the right way to retrieve it. Billing documents, to continue the example above, are indexed to support lookup by 2013 International Business Machines, Inc. Page 40
IBM Americas ATS billing document number. If a program tries to read the billing tables using the invoice column, in order to find the associated invoice document, the column with the invoice number will not be indexed, and access will be slow. If this is the situation, the solution is not to create an index, but to use the SAP data model correctly. There is an example of incorrect use of the SAP data model below in section 9.1.6. SAP has several SAP notes that describe wrong and right ways to use the SAP data model to retrieve application information: MM SAP note 191492 SD SAP note 185530 PP SAP note 187906 And if we look at SAP note 185530, we find this SQL and a solution:
Figure 54: Note 185530 - VBRP where VGBEL
9.1.5. Actions when predicates do not match indexes

It can occur that specific business processes require new indexes on tables. Here are some guidelines on how to approach the problem when you find a program where the predicates do not match the indexes: ABAP Creator Table Creator Action when predicates do not match indexes SAP SAP Check data dictionary for standard indexes which are not active in DB Check SAP notes Open a message to SAP Custom Custom Rewrite program if possible Evaluate table access patterns, whether table can be buffered Create index Custom SAP Check data dictionary for standard indexes that are not active in DB Review use of data model (will frequently fix problem) Change program Create index (should very seldom be necessary)
IBM Americas ATS
9.1.6. High Impact SQL caused by missing indexes on custom table

In this example, the SQL cache is sorted by CPU time, to bring the statements with longest total execution time to the top. Note that the top statement performs over 12,000 Bgets for each row processed, which is inefficient. With effective indexes, selecting a single row from a table will take 4-5 Bgets. Each execution is over 3 seconds (the time is in microseconds), and with effective indexes, selecting a row would normally take a few milliseconds.
Figure 55: Custom table SQL
We select the statement and press the explain button to see the execution plan.
Figure 56: Custom table explain
It is SELECT FOR UPDATE, so part of the long averaged elapsed time could be row lock contention. But in Figure 55 each execution used over 1 CPU second to process over 16K bufgets, so the statement is also very inefficient. 2013 International Business Machines, Inc. Page 42
IBM Americas ATS We see in Figure 56 that Oracle is using INDEX SKIP SCAN, which means that not all the local predicates match the leading index columns some local predicates are applied on the index pages as Filter Predicates. The cost is very high, relative to estimated rows. Drill into Access Predicates or Filter Predicates in Figure 56 to see how the local predicates are applied.
Figure 57: Custom Table Access and Filter Predicates
The entire index is read on each execution SERNR is a Filter Predicate, and there is only one MANDT, as shown in the index statistics in Figure 58.
Figure 58: Custom Table index
From the SQL list in Figure 55, we can also display the ABAP text by pressing the ABAP Source button.
Figure 59: Custom program for custom table
IBM Americas ATS Goto > Attributes to check who wrote it.
Figure 60: Custom Program Attributes
The ABAP is custom development it was not created by SAP. If created and modified by SAP, it would standard ABAP. We compare the local predicates in the SQL statement MANDT = SERNR = With the columns contained in the index MANDT WERKS MATNR ZOX_RUECK SERNR ZCONF_CNT And now we can see the source of the problem the local predicates in the SQL do not match the leading columns of the index. At this time, we need to determine what action should be taken. From the table in Section 9.1.5, we see that possible actions might be to rewrite the SQL, or buffer the table in SAP, or create an index. In this case, let us assume that we cannot re-write the SQL to match the available index. So, we will create a new index (MANDT, SERNR) on the table.
9.1.6.1. Summary of missing custom index

The statement was identified by sorting the SQL cache by CPU time. The top statement performed over 12,000 bufgets per row, which is inefficient. Each execution took over 1 second, which was about 1,000 times as long as it would take with a proper index. When we explained the statement, and compared the local predicates to the available indexes, we found that there is no index that supported the ABAP selection for SERNR = i_update_status-sernr. The solution was to create a new index (MANDT, SERNR) for this program.
IBM Americas ATS
9.1.7. High-impact SQL caused by incorrect use of SAP data model

Sort the SQL cache by CPU Time, to bring the high impact statements to the top.
Figure 61: VBRP high impact statement
The first statement in the list is very, very inefficient it performs over 9,000 Buffer gets per row, and retrieves only 35 rows per execution. When all the local predicates in an index are evaluated efficiently in an index, there are at most only a few Buffer gets per row. Explain the statement to see what execution plan is used.
Figure 62: VBRP explain
It is TABLE ACCESS FULL the entire table is read without an index. Note the local predicates, which we will compare with indexes. MANDT = VGBEL = VGPOS =
IBM Americas ATS Drill into VPRB in Figure 62 to see the indexes on the table. There is only one index, and it does not contain VGBEL or VGPOS.
Figure 63: VBRP index
The SQL has a local predicate on MANDT, but Figure 63 shows that MANDT has only one value, so Oracle chose to read the entire table (TABLE ACCESS FULL) as shown in Figure 62. Use the ABAP Source button in Figure 61 to look at the source code.
Figure 64: Program accessing VBRP
The program name starts with Z this is custom code. From the table in Section 9.1.5, we know that the first action is to examine if the program is using the data model correctly. SAP note 185530 describes several incorrect and correct lookups for SD. This problem is contained in the note:
Figure 65: SAP note with examples of using the SD data model
IBM Americas ATS The action for this problem is to send it back to the developer with reference to SAP note 185530. In this proposed fix, for SD documents there is a table VBFA (document flow) that contains the predecessor and successor documents for sales documents. For example, given a sales order number, one can find all the subsequent documents related to the sales order, or given an invoice number, one can find all the predecessor documents that lead to that invoice document.
9.1.7.1. Summary of incorrect use of data model

Statement is found in ST04 SQL cache with very high Bgets/exec and high Bgets/row. Explain the statement, the entire table is read. Check the indexes, there is no index that supports the local predicates. Check the owner of the program. It is not SAP, so we suspect that the programmer is looking in the wrong place for the information. Check SAP note 185530, since the problem here is in use of the SD tables, and find the proposed fix. Send the program back to developer to re-work.
9.1.8. High-impact SQL when optimizer chooses wrong execution plan

While the Oracle costbased optimizer generally does a good job of determining the best way to access tables based on the predicates in the SQL, it can go wrong. Here the SQL cache has been sorted by elapsed time, to find statements with a high impact on response time.
Figure 66: ST04 SQL cache for skew example
The statement performs about 30 buffer gets per row, which is high for an array operation of 3,000 rows per execution. With efficient indexes, a single row read is about 5 bufgets, and for array operations there are fewer bufgets per row, often less than one bufget per row. A high ratio of Bgets per row usually means that the local predicates that filter well are not being applied in the index, of if the statement is a join, they are not being applied on the first table accessed by Oracle.
IBM Americas ATS Explain the statement, to see what it is doing.
Figure 67: BKPF explain
From Figure 57, get the local predicates, which will be needed later: MANDT = BUKRS= BSTAT = XBLNR = BUDAT = STBLG = The ST04 statistics in Figure 66 tell us that the current execution plan is not an efficient way to access the data, since each execution takes over 33 buffer gets for each row. Drill into the BKPF~1 index in Figure 67, and we see that this index matches the columns (MANDT, BUKRS, BSTAT, and XBLNR) in the local predicates.
Figure 68: BKPF~1
IBM Americas ATS Drill into the BKPF table in Figure 67 to get the row count for the table.
Figure 69: BKPF rows
There are 6.2 M rows, and 3.6M distinct values of XBLNR, so without histograms, Oracle will estimate that there are about 2 rows per XBLNR value. If there is skew in the distribution of values in XBLNR, this estimate will be wrong. We press the Bind Variables button in Figure 67 to see history of variables used at runtime. Review the Bind Variables to see if there are patterns or trends in the variables that users specify.
Figure 70: BKPF Bind variables
Comparing with the SQL in Figure 67, we see that the value for XBLNR is space (:A3 is space). We can use ST04 SQL Command Editor to check for skew in the columns.
Figure 71: BKPF SQL to count XBLNR
The transaction TAANA can also be used to do counts of rows with a value.
IBM Americas ATS The result in Figure 72 shows that the value (space) occurs very frequently over three million rows in the table have the value XBLNR = . The index BKPF~1 is not a good choice in this situation, since there are so many rows with value XBLNR= . If XBLNR were a value other than space, then BKPF~1 would have been a good choice.
Figure 72: BKPF XBLNR count
Next, check the indexes on BKPF to see if there is an index that would support the BUDAT = local predicate. BKPF~2 matches MANDT, BUKRS, BSTAT, and BUDAT. You can use the query in Figure 71, modified for BUDAT column, to confirm that the data in BUDAT is not skewed, and that it would be a better choice for Oracle to use.
Figure 73: BKPF~2 index
Figure 73 shows that there is already an index on BUDAT. Oracle is not choosing it. When there is skew in data values, or when range predicates are used to select very small or very large percentages of the rows, the default SAP prepare method (prepare with parameters) may cause Oracle to choose a bad execution plan, since Oracle cannot make a good estimate of cost in these situations. In this case, Oracle is over-estimating the selectivity of the XBLNR = local predicate. The default way that statements are prepared by SAP is that parameters are used. This enables a statement to be shared by many different threads, but it has the disadvantage that Oracle cannot compare the execution-time values to the Oracle histogram statistics, to determine when an infrequently occurring (or frequently occurring) value is being sought. SAP provides an option to use an ABAP hint to change this default. See SAP note 129385 regarding how to prepare a statement with literals or values. For earlier versions of SAP and Oracle that do not have the Bind Variables option in ST04 explain, it is generally necessary to trace the SQL using ST05 to get the parameters.
IBM Americas ATS Using the ABAP Source button in Figure 66, we can display the ABAP source. It is custom code, since the name starts with Z.
Figure 74: BKPF source
Since there is skew, we need to check if histograms are being collected on the table, since Oracle will need the histograms in conjunction with parameter values to determine when the program is selecting frequently-occurring values or seldom-occurring values. Use DB02 Storage tab to view columns with histograms. Figure 75 shows that there are currently none.
Figure 75: DB02 display columns with histograms
If you use SAP to control statistics, then update DBSTATC for BKPF to collect statistics using EH Method for Estimate Histograms The options for DBSTATC are described in SAP note 106047.
Figure 76: DBSTATC
IBM Americas ATS Last, the ABAP must be modified to add the Open SQL HINT &SUBSTITUTE VALUES&, as described in SAP note 130480. &SUBSTITUTE VALUES& is used when the access plan should vary based on the runtime values of the SQL, such as when there is skew in the distribution of a column used in the statement. By passing the values of the parameters to Oracle, Oracle can compare the value with the histograms, and use that to choose the best execution plan. SAP note 176754 discusses a variety of issues that may impact the Oracle Cost Based Optimizer in choosing the best execution plan. It is necessary that two steps are done to fix this problem the histogram statistics gather the data that shows Oracle that there is skew in values of a column, and the ABAP HINT passes the value used when the statement is executed, so that Oracle can compare the table statistics to the execution value, to determine whether a frequently occurring or infrequently occurring value is being retrieved.
9.1.8.1. Summary of wrong choice of access path

Statement is found in SQL cache with many buffer gets per row. Explain the statement, and we see that four of the six local predicates are evaluated in index BKPF~1. We check the bind values, and then look at the distribution of values in XBLNR, and we determine that there is skew in XBLNR, and the program is fetching the commonly occurring value (space). There is another index that contains BUDAT, and since there are over 5,000 distinct values of BUDAT, we modify the program to use the &SUBSTITUTE VALUES& HINT so that SAP will pass the values to Oracle when the statement is executed. We also gather histogram statistics on BKPF, so that Oracle will know that XBLNR has data skew.
9.1.9. High-impact SQL is a symptom of SAP buffer problem

We have ST05 traced and summarized the trace for a program with long DB time, in order to find the SQL that is slow.
Figure 77: A611 in SQL cache
Note in Figure 77 that the selects on A611 take the largest part of DB request time, but A611 is set to be fully buffered (BfTp = ful).
IBM Americas ATS Display the ABAP source using the ABAP Source button in Figure 77, and it is SAP code.
Figure 78: A611 ABAP source SD_COND_ACCESS
Transaction ST02 shows SAP buffer management activity generic key DB accs is much higher than for other buffer types, and there are swaps.
Figure 79: ST02 Buffer Display
IBM Americas ATS In Figure 79, drill into the Generic Key line.
Figure 80: ST02 Generic Key
In Figure 80, press Buffered Objects. Figure 81 shows that the table with the highest number of DB calls, A611, is in error state.
Figure 81: A611 Error State
Place the cursor on a buffer state, and press F1 to display the help for Buffer State. Error means the buffer could not be loaded.
IBM Americas ATS
Figure 82: ST02 Buffer Status
The table is configured as buffered on the application server, but there must be some sort of problem, since the call statistics in Figure 81 show over 1 million calls. When a table is buffered on the application server, there should be very few calls to the database server to retrieve rows from the table. It may be that the table is frequently changed and invalidated and then re-loaded or that the table does not fit in the application server buffer. These can be checked in Figure 81 there are no changes. The table may also be read by the ABAP in a way that bypasses buffering on the application server. See SAP note 47239 regarding buffered table behavior. Since BYPASSING BUFFER is not used in the ABAP above, and the table is in error state, we can infer that the problem is that the SAP Generic Buffer area needs to be enlarged. We can check the size of the table to verify that it can be buffered, and then increase the generic buffer parameters zcsa/table_buffer_area and zcsa/db_max_buftab. As an alternative, one can change the table to be buffered on generically on some of the key columns, rather than fully buffering the table. Each generic range will require less space, and the most active parts of the table would be loaded into SAP buffer.
IBM Americas ATS
Figure 83: DB05 A611
One can use DB05 to evaluate the number of generic ranges to choose. In Figure 83, note that there are 250 distinct values of WERKS, but only one distinct value of MANDT, KAPPL, and KSCHL. If we were to choose to buffer this table generically, then buffering on 4 columns would be a reasonable choice, each WERKS would have its own range in the SAP buffer.
IBM Americas ATS As shown in Figure 84, use SE13 to change the number of columns in the generic key, if you want to buffer in smaller ranges.
Figure 84: SE13 A611
9.1.9.1. Summary of High-impact SQL is a symptom of SAP buffer problem

ST05 summarized trace shows that a buffered table is top contributor to DB time. Look at the program source, and it is SAP code. Check the buffers on the application server, see that the table is in error state. This usually means that the generic buffer is too small. The frequent calls to A611 are a symptom of another problem, the small generic buffer size. Either increase the size of the generic buffer, or change to buffer the table in smaller segments.
IBM Americas ATS
9.1.10.
High Impact Unnecessary SQL
In this example, we have ordered the SQL cache by rows per execution, in order to search for statements that retrieve large numbers of rows. The top statement looks OK, based on the efficiency metrics the ratio of Bget/row is low. But, the question we ask is why would a program need to retrieve 16 million rows on each execution of an SQL statement?
Figure 85: EKET 16 million rows
We explain the SQL to see what the program is doing.
Figure 86: Explain EKET
In the explain output, we see something unusual. The only local predicate is MANDT =, so all the rows in the table for the productive MANDT are being selected. Usually we would only see SQL with MANDT= and no other local predicates in SQL that loads fully buffered tables. We need to check the program that is issuing the SQL.
IBM Americas ATS
Figure 87: EKET program source
Figure 87 shows that the program is a custom program (the name starts with Z) Note that the ABAP has selection conditions (WHERE EBELN = ) that are not in the Oracle SQL why does this happen? ABAP SQL works like a template for building the DB SQL. When the WHERE condition is converted to SQL local predicates, if there is no value specified for a variable or if an internal table used with FOR ALL ENTRIES is empty, that column is not included in the SQL. Here, the problem is that the internal table I_EKPO is empty, so the DBSL does not create local predicates on EKET-EBELN and EKET-EBELP. Since MANDT is implicitly added to the SQL, we get an SQL statement with only MANDT =. The internal table used in FOR ALL ENTRIES places restrictions on the rows to be retrieved, so if the internal table is empty, then all rows are retrieved. In this case, the fix is to modify the ABAP to check for whether I_EKPO is empty, and if it is empty to skip the SELECT.
9.1.10.1. Summary of High Impact Unnecessary SQL

The SQL in cache is efficient (low Bgets / row) but many rows are processed. We check the SQL, and the only local predicate is MANDT = there are no other selection conditions. Checking the ABAP, there is a WHERE clause with EKET columns referencing an internal table. If the internal table is empty, then the DBSL does not create a local predicate on EKET. The fix is to modify the ABAP to check whether the table is empty, and skip the SELECT if it is empty.
9.1.11.
High Impact SQL is a symptom of Oracle statistics problem
In this example, we have ordered the SQL cache by total elapsed time, in order to find statements that have the largest impact on runtime (at the system level). Note that the top statement has over 6,000 bufgets per execution, but 0 bufgets per row. This is because no rows were returned. See Section 9.1.2 regarding ratios of bufgets to executions and rows.
IBM Americas ATS This example is taken from a test system. In general, problems with statistics on productive systems are encountered soon after new tables or new SAP functionality are transported into prod, or when errors have been made in existing procedures for gathering DB statistics.
Figure 88: D010INC high bgets/Exec
When we explain the statement, we see in Figure 89 that the table D010INC is being read without an index, using TABLE ACCESS FULL.
Figure 89:D010INC explain
IBM Americas ATS Click on the table name (D010INC) in Figure 89 to see the table and its indexes and index statistics.
Figure 90:D010INC indexes
SAP retrieves the statistics here from Oracle. At the time the statistics were analyzed, the table was empty. But the table is clearly no longer empty, since each execution of the SQL (see Figure 88) currently reads over 6,000 pages of data from the table. We can analyze the table to generate new statistics (note the Analyze button in Figure 90), and after the statistics are generated, when we explain the statement, the index is used.
IBM Americas ATS
Figure 91: D010INC read with index
If this were a productive system and the table were an SAP business table, we would need to review the statistics gathering process for the table, to verify that it was correct, and we would need to analyze statistics for the table, in order for Oracle to choose a better execution plan.
9.1.11.1. Summary of access path problem caused by statistics

We sorted the SQL cache to find statement with high elapsed time. The top statement did over 6,000 bufgets per execution, and never returned a row. This is inefficient. We explained the statement and examined the table and index statistics, and found that the table had statistics gathered when it was empty, and the table had grown. See SAP notes 932975 and 1020260 (and other note they reference) regarding Oracle tables that require special statistics to ensure proper optimization.
9.1.12.
High impact SQL caused by inefficient ABAP programming
In this example, we have a statement that is high in total elapsed time, and where the statement is inefficiently executed. The statement performs 39 bufgets per row returned.
Figure 92: CRCO ST04 Shared Cursor Cache
IBM Americas ATS Explain the statement, and we see that there are both Access Predicates (predicates applied in the index) and Filter Predicates (predicates applied after the index rows are read) on the CRCO~0 index.
Figure 93: CRCO Explain
Drill into the Access Predicates line in Figure 93, and we see that OBJID is applied as a Filter Predicate, which is less efficient than when it is an Access Predicate.
Figure 94: CRCO Predicates
IBM Americas ATS If we drill into the CRCO~0 index in Figure 93, we can see the problem there is no local predicate on the OBJTY column, which is the second column in the index. Local predicates on columns after OBJTY, such as OBJID, are processed as Filter Predicates.
Figure 95: CRCO~0 index
In Figure 95, note that there is only one value for OBJTY. Since there is only one value, it should be possible for the ABAP program to include the value, so that the first three columns of CRCO~0 can be used as access predicates. We check the ABAP source using the ABAP Source button in Figure 92.
Figure 96: CRCO program
This is a custom program, so our action is to send it back to the developer, to review how to specify OBJTY = as part of the ABAP SQL.
9.1.12.1. Summary of High impact SQL caused by inefficient ABAP programming

SQL in the shared cursor cache has symptom of inefficiency high bufgets per row. We explain the statement, and see that the only access predicate is MANDT=. The local predicate on OBJID is a filtering predicate, which is evaluated by oracle after the index rows are retrieved by MANDT. There is only one value of OBJTY, the second column in the index, so it may be possible to add OBJTY to the ABAP SQL. The program is a custom program, so we send it to development to review how to modify the ABAP to match the CRCO~0 index better.
IBM Americas ATS
9.1.13.
High impact SQL caused by range predicate estimate
Similar to problems with skew discussed in Section 9.1.8, range predicates can also lead to incorrect access path selection, since ABAP SQL is normally prepared with parameters. In this example, we have an SQL statement with high total elapsed time, and high bufgets per row.
Figure 97: BSIS range predicate statement statistics
Explain the SQL.
Figure 98: BSIS explain
The cost estimate is low, so Oracle thinks this should be an efficient statement. Drill into the Access Predicates for the INDEX RANGE SCAN. 2013 International Business Machines, Inc. Page 65
IBM Americas ATS
Figure 99: BSIS Access Predicates
Then drill into the index BSIS~Z03. Since there are no filter predicates in Figure 99, we know we will find that MANDT, BUKRS, HKONT, and BUDAT make up the first four columns.
Figure 100: BSIS~Z03 Index
We can use the ASH Sample button Figure 98, to display where the time is spent in execution of the statement. The cost estimates in Figure 98 showed that Oracle estimated the cost of accessing the index and the table as roughly the same. But when we display the ASH Sample data, we see that in reality Oracle spends 99% of elapsed time on the table. This can occur if the index columns do not filter as well as Oracle estimated.
IBM Americas ATS
Figure 101: BSIS ASH Sample
There is a range predicate on BUDAT (BUDAT <=), so the selectivity will depend on the value of BUDAT specified at runtime. By default, SAP prepares statements with parameters, and in this case Oracle uses an estimate of the filtering for range predicates such as <, <=, BETWEEN, etc. If the runtime values filter very well, or not at all, the Oracle estimate may cause a bad access plan to be chosen. Press the Bind Variables button in Figure 98.
Figure 102: BSIS Bind Variables
In Figure 102, we can see the date is 12/31/9999. The local predicate BUDAT <= 12/31/9999 provides no filtering at all. If the ABAP can be modified to use the &SUBSTITUTE VALUES HINT, then Oracle can make a better estimate of the filtering of the BUDAT local predicate, and will choose an access path that gives better filtering in the index.
IBM Americas ATS
Figure 103: BSIS prepared with values
The cost estimate is higher now than in the original explain in Figure 98, but it is a more accurate estimate. Oracle now chooses an index where BUDAT is not processed in the index.
Figure 104: BSIS with values access and filter predicates
Oracle still applies MANDT, BUKRS, and HKONT as access predicates, and can also use GJAHR as a filter predicate. This will be a better access plan than was originally chosen, since one more local predicate will be applied on the index, rather than on the table. Another possible action is to investigate if this is caused by user behavior. If users are entering BUDAT <= 12/31/9999 in the selection conditions, and if they could remove this selection without impacting the validity of the report, then taking BUDAT out of the SQL would be a way to execute the statement more efficiently, without having to modify the ABAP.
9.1.13.1. Summary of High impact SQL caused by range predicate estimate

We find a statement in the SQL cache with high bufgets per row, and high total elapsed time. The statement matches four columns on a custom index, but the ASH sample data shows that much more time is spent evaluating rows than in the estimate. We examine the bind variables, and see that the BUDAT <= local predicate is not selective. We can either modify the program to prepare with values, or determine whether this is caused by user behavior.
IBM Americas ATS
9.1.14.
SQL not needed because of business customization
We have an SQL statement in the statement cache that has high total elapsed time, but never retrieves rows. Each execution ran 800 ms, and returned no rows. With efficient use of an index, each execution should take just a few ms.
Figure 105: AUFK statement statistics
Since no rows are returned, we know ahead of time that the combination of local predicates eliminates all the rows in the table they are very selective. The table could also be empty. That will be checked below. Explain the statement, and we see local predicates being applied in the index as Access Predicates, and on the table as Filter Predicates.
Figure 106: AUFK explain
The filter predicates applied on AUFK table.
IBM Americas ATS
Figure 107: AUFK filter predicates
The access predicates applied in the AUFK~D index are MANDT and PSPEL.
Figure 108: AUFK~D access predicates
Since the statement runs nearly one second on each execution. We know that MANDT and PSPEL are not sufficient for the selectivity. If they were, then the statement would be very fast, since applying access predicates in the index b-tree is fast.
Figure 109: AUFK~D index
There is only one value of PSPEL. If the SQL contains that value, then PSPEL = will not be selective. We check the Bind Variables to see what is used by the program at runtime.
Figure 110: AUFK Bind Variables
IBM Americas ATS We can create an AD-HOC variant in TAANA to analyze the distribution of data in the PSPEL column. We find that the value is always empty PSPEL column is WBS element. The program is using the PSPEL value that occurs in all rows. The PSPEL = local predicate is not selective, so the AUFK~D index was not a good choice.
Figure 111: AUFK TAANA
Display the program. It is a custom program.
Figure 112: AUFK Source
At this point we know that the filtering is not on MANDT or PSPEL.
IBM Americas ATS If we had histograms on the table and prepared with values, we could get Oracle to choose a different access plan, as shown in Figure 113, where we explain the statement using the literal value for PSPEL.
Figure 113: AUFK with values
But, we need to consider the fact that the SQL never returns a row. We would discuss with the programmer and experts on the business function whether it can ever return data, given the selection conditions. It may be that the business process configuration in use would never return data for the WERKS AUART combination (LHAR YHAR in Figure 110), and that the statement should be removed from the ABAP.
9.1.14.1. Summary of SQL not needed because of business customization

The symptom of inefficient SQL is high bufgets per execution and high total elapsed time. One notable thing about this SQL is that it seems to never return a row. We explain the statement, and see that the index being used has only one value for each column, and check the bind variables and TANA and confirm that the SQL is seeking a commonly occurring value, so the index is not selective. Preparing with values allows Oracle to choose a better access plan, but the fact that no rows are retrieved points to the fact that the SQL may be unnecessary. We review the business process with ABAP and functional experts, to determine whether the SQL call can be removed from the ABAP.
IBM Americas ATS
9.2.
Create candidate list from ST03N
Run ST03N, sort by total database time (T DB Time).
Figure 114: ST03N by DB time
Transactions which have the longest DB time, and thus probably the largest impact on the database server, are at the top of the list. Transactions or programs that have DB time that is much higher than CPU time, for instance ZERROR or ZTU in Figure 114, frequently have ABAP or SQL problems that contribute to the long DB time. At this point, we cannot tell which is occurring, but one can trace the programs with ST05, as shown in Section 8.3, to determine whether there is a problem or not.
9.2.1. ST03N with table statistics

If the SAP parameter stat/tabrec is enabled, then SAP will gather table call statistics for each dialog step. These statistics can be displayed, to determine which table accounts for the long DB time for a transaction or program.
Figure 115: ST03N Table Statistics
In Figure 115, RFEBBU00 calls to AVIP are slow on average it retrieves 9 rows per second. Either the program makes many calls to AVIP that return no rows, or the SQL is slow. As shown in Section 8.3, we can use ST05 to trace the program and limit the trace to AVIP. 2013 International Business Machines, Inc. Page 73
IBM Americas ATS
10. System Health Check

10.1. CPU activity
As discussed above, SQL problems can contribute to high CPU use on the DB server, and ABAP coding problems can contribute to high CPU use on the application server. If the system has high CPU utilization, then first search for SQL and application problems that may be causing high CPU use. For native or dedicated LPAR environments, SAP ST06 (or OS07) transaction has statistics on recent use, and on historical CPU use. Since ST06 has statistics based on hourly averages, it will not show CPU constraints until they are very severe. It is generally better to monitor CPU use using an AIX based tool, and gather statistics on 5 or 10 minute intervals, in order to be able to calculate average and peak activity. For SPLPAR environments, ST06N (or OS07N) is required together with an updated saposcol, in order for SAP to correctly report CPU activity. See SAP note 994025 regarding SAP and AIX requirements for SAP monitoring in a virtualized AIX environment. Important Note: The older OS monitors in SAP (ST06 and OS07) will give misleading and incorrect output, when used in an SPLPAR virtualized AIX environment, since they do not report on entitlement or physical CPU usage. They only report User and System percentage, which are not useful for identifying CPU constraints with SPLPAR.
IBM Americas ATS
Figure 116: ST06N with AIX virtualization support
Figure 116 shows an LPAR that is CPU constrained the Capacity Consumed is the same as Available Capacity. Dont use Idle time as a measure of whether the LPAR CPU is at maximum.
IBM Americas ATS On an SPLPAR virtualized LPAR, the traditional User and System CPU measurements are not useful when evaluating CPU availability or constraints.
Figure 117: ST06N over Entitlement
In Figure 117, there is 37% idle but Guaranteed Capacity (aka Entitlement) is only 1.2 CPUs, and the LPAR is currently using 5.13. If there were demand from other LPARs for their Entitlements, it is possible that this LPAR could be reduced to as little as 1.2 CPUs of usable CPU resources. In general, it is recommended to size the LPAR Entitlement based on capacity requirements, and to set the number of VCPUs to be at most about 2-3 times the entitlement for the LPAR. In Figure 117, the Entitlement (Guaranteed Capacity) is only 10% of the CPU count, so either the entitlement is too low, or there are too many VCPUs. ST06N is can be used by Basis or DBA team-members to get real-time snapshots of CPU or I/O activity. For longer term monitoring and capacity planning, one can use nmon, sar, or other tools to gather and report on the utilization. While it may be acceptable for an application server running batch work to run at 100% utilization, if an application server supporting dialog or critical interfaces is frequently hitting 100% utilization, it can have a significant impact on response time and end-user satisfaction.
IBM Americas ATS If the DB server has high CPU activity: Evaluate the impact of inefficient SQL and applications as in Section 9.1. If there is a SAP application instance on the DB server, investigate whether it can be moved off. If no work can be moved off, evaluate moving some batch jobs to run outside of peak time. Acquire more CPU power for the DB server If one application server has high CPU activity Evaluate the impact of inefficient ABAP code on excessive CPU use as in Section 8.2. Evaluate changes in the SAP login groups to balance the workload differently Change the mix of batch/update/dialog to reduce the amount of work on the server Acquire more CPU power
10.2.
I/O activity
I/O activity can be monitored at the Oracle level from SAP using ST04, or as an alternative, one can use AWR to monitor it at the Oracle level directly from Oracle. Oracle AWR has the benefit that they can be configured to automatically gather performance statistics, which can be used for historical reporting and comparison. ST04 file statistics are gathered by logging in to view the current statistics, or can be manually reset for an interval. ST04 and AWR both report average read times from the perspective of the DB. Another reason for starting with the Oracle view of I/O activity is that there are several different configuration problems that can cause the symptom of slow I/O at the Oracle level, though physical I/O at the AIX level is fast. The actual problem is serialization on a resource (e.g. AIO server processes, filesystem buffers, or filesystem i-nodes) in AIX after the I/O is issued by Oracle, but before AIX sends the I/O to the disk. For additional information on AIX settings related to I/O queue settings, see http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105745. ST04 > Wait Event Analysis > Filesystem requests can be used to view read and write activity and times, as viewed by Oracle.
IBM Americas ATS
10.2.1.
Good I/O performance in Oracle
In the following section of an Oracle v$filestat report (ST04 > Wait Event Analysis > Filesystem requests) note that the average I/O response times are small single digit, and they are not widely skewed. Here, the I/O response times are good.
Figure 118: ST04 file system requests with good response times
10.2.2.
Symptom of I/O hotspot on disk
Figure 119 is Oracle v$system_event, which shows the categories of delay. It can be displayed from ST04 Detail Analysis. db file sequential read is generally the top delay category, and with good I/O performance the average waits should be in the 5-10 ms range.
Figure 119: V$SYSTEM_EVENT
Here, the average is 33 ms, which is very high. This indicates an I/O performance problem.
IBM Americas ATS If we display v$filestat (also from ST04 Wait Event Analysis > Filesystem Requests), we can check whether I/O is slow on all datafiles, or only some.
Figure 120: ST04 file system requests
Figure 120 shows that there are widely varying I/O response times on the most frequently read datafiles. When there is such variation in the response time of read-intensive files, it is usually symptom of an I/O constraint at the physical disk level, where the data files are not spread across enough disks. (A disk may be called a vpath, hdiskpower, LUN, etc it is the AIX representation of the physical location where the data is stored). In order to prevent this from happening, we recommend using a database layout where each LV is configured to reside on many hdisks. In SMIT, this would be done by creating the LV on several hdisks, and specifying Range of physical volumes maximum. See the paper Configuring IBM TotalStorage for Oracle OLTP Applications, which is document number WP100319 at http://www.ibm.com/support/techdocs, for more details about configuring a stripe and spread layout for the database. The solution for I/O hotspots is to analyze the I/O at the physical level in the storage system, and if overloaded spots are found, to spread the datafile across more disks: Determine name(s) of file(s) with slow response time ST04 or AWR Determine the filesystem that contains the filename using ls or find commands Determine LV that contains the filesystem using df or lsfs List PVs (or vpaths) in the LV using lslv l If ESS, list LUNs for the vpaths using lsess Use ESS expert to check I/O activity for clusters, ranks, LUNs
10.3.
Oracle Review
There are a couple quick checks one can do, as part of the system health check. See the SAP note 618868 for more information on oracle performance FAQs. This paper just briefly discusses the main elements of Oracle configuration and performance. For more details, please see the IBM Whitepaper Oracle Architecture and Tuning on AIX 2013 International Business Machines, Inc. Page 79
IBM Americas ATS (http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100883) contains detailed and upto-date information on configuration and tuning Oracle.
10.3.1.
Concurrent I/O
In section 9.2, cio, or concurrent I/O was mentioned as one way of improving I/O performance. Concurrent I/O is a feature of JFS2 provided in AIX 5.2 ML01 and newer versions. CIO includes the performance benefits of direct I/O, plus the additional performance benefit from eliminating contention on the inode lock. Concurrent I/O is intended for use by applications that do their own data serialization, such as Oracle. Do not place Oracle binaries on filesystems mounted with the cio mount option. Oracle 10g and 11g will open database files located on the JFS2 file system with the O_CIO option if the filesystemio_options initialization parameter is set to either directIO or setall. (The filesystemio_options=setall setting should be used, as asynchronous I/O is disabled with the directIO option.) With Oracle 10g or 11g, the filesystems may also be mounted with the cio option. For more information, refer to: SAP Note 948294 - AIX JFS2: mount options to use with Oracle 10g Oracle Doc ID: Note: 257338.1, Direct I/O (DIO) and Concurrent I/O (CIO) on AIX 5L. For optimal performance when using direct I/O and concurrent I/O, the JFS2 blocksize (agblksize) must be set according to the Oracle db_block_size: For database files (.dbf): For db_block_size 4096, use agblksize = 4096. For db_block_size = 2048, use agblksize = 2048. For redo log and control files: Put the redo log and control files on a separate, dedicated filesystem and use agblksize = 512. The agblksize is specified during filesystem creation time. To check agblksize, use the lsfs q command as root user. For example,
# lsfs -q /orafs Name Nodename Mount Pt VFS Size Options Auto Accounting /dev/oralv -/orafs jfs2 165847040 rw yes no (lv size: 165847040, fs size: 165847040, block size: 4096, sparse files: yes, inline log: no, inline log size: 0, EAformat: v1, Quota: no, DMAPI: no, VIX: yes, EFS: no, ISNAPSHOT: no)
10.3.2.
Database Hit Rate
ST04 can be used to review the hit rate in the database. Since SAP often executes many SQL operations per dialog step, the goal should generally be to have a hit rate in the high 90s. A 96% hit rate is two times the misses of a 98% hit rate, that is it has two times as much I/O delay. A 94% hit rate is three times the misses of a 98% hit rate. The impact of cache misses quickly grows as hit rate decreases. 2013 International Business Machines, Inc. Page 80
IBM Americas ATS Since inefficient SQL can cause the symptom of low hit rates, the SQL cache analysis process in Section 9.1 should be the first action, when a low hit rate is seen. After the SQL analysis, one can review adding more memory to Oracle (in order to increase hit rate) by using the process in Section 10.5.5.
Figure 121: ST04 database overview
10.3.3.
Database delays
The oracle v$system_event view can be used to review the causes of delay in oracle. These delay statistics are recorded in AWR. As mentioned above regarding hit-rates, many Oracle delays (buffer busy waits, enqueue, etc) can be symptoms of SQL or application design problems, and may not fixed with database parameters. Normally, db file sequential read is the largest source of delay. information on Oracle wait events. See SAP note 619188 for more
Improving database delays, like improving hit rate, is not a goal in itself. They are used as part of the process to understand why SAP DB request time is high, and to formulate actions. For example, if DB request time is high, and average I/O times are high, then examining how to improve I/O times is a reasonable action. If DB request times are high, and I/O times are low, then one investigates how to improve the efficiency of SQL, or the ABAP.
IBM Americas ATS
Figure 122: ST04 - Oracle v$system_event delay statistics
While a workload is running, you can use ST04 > Wait Event Analysis > Session Monitor, to display wait events for the Oracle processes. Associating an Oracle wait event with its SQL statement can make it easier to determine the cause of the delay.
Figure 123: ST04 Oracle sessions
IBM Americas ATS
10.4.
SAP buffered table statistics
There are two quick checks that can be done: Tables that are not buffered, but which should be, and Tables which are buffered, but whose settings should be changed.
10.4.1.
Not buffered but could be buffered
The most common problem with buffered tables is tables that could be buffered, but are not. Generally, these are custom tables. Most SAP tables have the appropriate settings already enabled. Figure 124 is a segment of an ST10 report for not buffered tables, ordered by calls to the database. Impact on the database is more a function of the buffer gets per row (as shown in Section 9.1) and not calls, but calls or rows are the only orders available with ST10.
Figure 124: ST10 not buffered tables
There is one table in this list that may be buffering candidates: ZTCODE_CHECK. It is a custom table, so we must check: How large is the table Will it be changed frequently (it looks like not, from ST10 stats) Can the application tolerate a short period where the buffered versions are different, when the table is changed in the database and changes are propagated to application servers. Since the table is read with direct read, if the table is reasonably small (e.g. <5 MB), not changed much, and the application can tolerate inconsistency, then they could be fully buffered. If the tables are very large, then they might be single record buffered, to save buffer space by buffering only the referenced rows. ABAP SELECT SINGLE (SAP direct read) can be read from either the generic or single record buffer. SAP SELECT can only be read from the generic buffer. 2013 International Business Machines, Inc. Page 83
IBM Americas ATS
10.5.
Evaluate memory in AIX
There are three different SAP memory management options, which are described in more detail below. The latest version supported for your SAP kernel should be used. The three options are ES/TABLE = UNIX_STD should not be used with AIX ES/TABLE = SHM_SEGS for SAP kernels w/o support for ES/SHM_SEGS_VERSION=2 ES/TABLE = SHM_SEGS & ES/SHM_SEGS_VERSION=2 latest and best version See below for information on kernel levels required for each. SAP Instance memory parameter settings can have an impact on AIX memory use. For example, the latest version of SAP EM (using ES/SHM_SEGS_VERSION = 2) has more efficient disclaim processing, which can reduce the amount of real memory needed for SAP EM. If there is paging in AIX, please review the latest versions of the SAP notes referenced in Section 10.5.3.
10.5.1.
64K page use
64K page size (medium pages) can be used with both SAP and Oracle to improve performance. SAP note 1387644 describes the prerequisites for using 64K pages with SAP, and how to configure SAP to use 64K pages. With Oracle 11g, 64K pages are used by default for SGA on AIX.
10.5.2.
AIX Restricted Tunables concept
Since AIX 5.3, six tuning commands (vmo, ioo, schedo, raso, no and nfso) are available to tune the operating system for the specific workload. Beginning with AIX 6.1, some tunables are now classified as restricted use tunables. The default values for restricted tunables are considered to be optimum for most AIX environments and should only be modified at the recommendation of IBM support professionals. As these parameters are not recommended for user modification, they are no longer displayed by default, but can be displayed with the F option (force). The F option forces the display of restricted tunable parameters when the options a, -L, or x are specified alone on the command line to list all tunables. When F is not specified, restricted tunables are not included in a display unless specifically named in association with a display option.
10.5.3.
AIX Alternative Memory Management
In the older versions of SAP kernels (4.XX and 6.XX), extended memory bottleneck would occur if the high numbers of user contexts existed on the system. To alleviate that problem, SAP implemented a AIX memory management (ES/TABLE=SHM_SEGS) that allowed individual user context and overall extended memory size to grow beyond several GB. New memory management was based on the several different AIX memory allocation functions like disclaim(), late swap allocation etc Several benefits are achieved using this method: Eliminating potential memory bottlenecks, User context change is much faster, particularly for large contexts, A much higher throughput is achieved, especially for higher context change rates (local updates )
IBM Americas ATS Variable PSALLOC=EARLY must never be set if this Extended Memory implementation is activated. The profile parameter that would initiate the utilization of the new Extended Memory Management is: ES/TABLE=SHM_SEGS
Additional parameters are required and important for the successful implementation of the new memory management: EM/TOTAL_SIZE_MB: allows a maximum of 16 GB extended memory and can be larger, if required. This can be used to limit the total amount of Extended Memory, to prevent a swap bottleneck. ES/SHM_PROC_SEG_COUNT: specifies the number of shared memory segments that are used as shared and privately. ES/SHM_MAX_PRIV_SEGS: specifies the number of shared memory segments that are used privately. ES/SHM_USER_COUNT: the maximum number of users in all open sessions. ES/SHM_SEG_COUNT: the maximum number of AIX system segments that ESSHM uses. ES/SHM_SEG_SIZE: size of the AIX system segments (in megabytes) To properly implement the AIX alternative memory management following notes should be reviewed: OSS Note: 445533 (up to SAP 4.6D), OSS Note: 789477 (as of 6.10). In the specific situation that is primarily related to the large memory requirements based on the new AIX memory management, the memory management performance anomaly can occur. This is primarily related to the method that is used to map and disclaim the AIX memory pages used by the SAP kernel. To improve the performance of the memory management the following variable should be included in the profile of the associated application server: ES/TABLE = SHM_SEGS ES/SHM_SEGS_VERSION = 2 To get more details about the memory mgmt. process used in this case, please check the following notes: OSS Note: 856848, OSS Note: 1088458.
10.5.4.
AIX paging
The standard tools (nmon, vmstat, sar, topas) can be used for monitoring paging. The page space page-ins and page-outs (paging to disk) are the important indicator to check. Paging should not occur regularly. 2013 International Business Machines, Inc. Page 85
IBM Americas ATS If you encounter AIX paging, a quick check would be to look at the virtual memory manager parameters. AIX memory pages are categorized as being computational or file pages. AIX will always try to utilize available physical memory (subject to vmo parameter settings). What is not required to support computational page demand will tend to be used for filesystem cache. In AIX 5L, a number of the default vmo settings in AIX 5.3 are not optimized for Oracle database workloads and should be modified. (The defaults were changed in AIX 6.1 and should not require modification with 6.1 or 7.1.) Check the settings for the following vmo parameters using the vmo a command (as root user): minperm% =3 maxperm% = 90 maxclient% = 90 lru_file_repage = 0 (with AIX 7.1, lru_file_repage is no longer settable) Refer to SAP Note 973227 - AIX Virtual Memory Management: Tuning Recommendations. Note: In AIX 5.2 environments with large physical memory, set maxperm%, maxclient% such as not to exceed 24GB of memory. If you encounter AIX paging on a system with an SAP instance, and you have reviewed that the correct EM configuration is used, consider paging first as a symptom of an application problem, and approach it from the application statistics in SAP. Since SAP memory use can vary greatly, as programs allocate and free SAP memory, ABAP problems such as the slow internal table access described in Section 8.2 can contribute to paging problems. If a report processing many line items runs quickly, while it is running it will acquire more memory, and then release it. If the program runs too long because of inefficient ABAP coding, then the program will need the memory for longer than it should. When several programs do this, it can cause paging. If the programs were coded efficiently, then they would quickly finish, and this would reduce the likelihood that they would all be running simultaneously. After having found the programs and users with large memory requirements, one can consider running the reports in batch on a server specially configured for large memory requirements, or the huge reports can be automated to run at night, when there may be less demand for memory. If the total SAP demand for memory is larger than available memory, one can reduce SAP memory quotas for EM and heap, but if the workload is unchanged, this will cause programs to go into PRIV mode (when EM is exhausted) or to abend when running out of heap.
IBM Americas ATS Check for running programs using large amounts of memory with SM04 > goto > memory:
Figure 125: SM04 > goto > memory
If a user is running several transactions, only one of the transactions will be displayed in SM04. Choose a user, and press Sessions to see the names of the transactions.
Figure 126: SM04 Sessions
One can then examine these transactions further to determine which use the memory. Use table TSTC to see transaction name and code. VA05 is List of Sales Orders.
IBM Americas ATS ST03N has historical statistics on memory usage.
Figure 127: ST03N memory use statistics
10.5.5.
Evaluating increasing memory for Oracle or SAP
If you have evaluated SAP or Oracle memory use, and would like to increase the buffer sizes in Oracle or SAP, how do you determine whether there is memory available, or whether adding memory will cause paging? The free page information on commands such as vmstat does not always reflect how much memory is really available to be added to SAP or Oracle buffers. If vmstat shows free memory, then there are truly free memory pages that can be added to SAP or Oracle buffers. Check vmstat (or ST06) over the course of several days, and if it consistently shows free memory, then use the minimum free pages to determine the amount of memory that is available to add to Oracle or SAP. Do not try to allocate all free memory to SAP or Oracle.
IBM Americas ATS
Figure 128: ST06 displays free memory
There are situations with files are mapped to memory, where ST06 or AIX vmstat may show almost no free pages, but there is still a lot of memory that can be added to SAP or Oracle, such as when a server is used for ftp file transfers or NFS file-serving. Since DIO&CIO is now the recommended setting for Oracle datafiles, the database data files would not be memory mapped. Unless a file is opened for DIO (direct I/O) AIX maps file pages into available physical memory, in order to help performance. There are AIX parameters on the vmo command that set the limits of memory-mapped files. On a database server, first check that SAP note 78498 or 973227 (depending on SAP and AIX version) has been implemented to establish limits on real memory used for memory-mapped files.
IBM Americas ATS Check the amount of real memory that is being used for memory-mapped files. This can be done with the command svmon G.
Figure 129: svmon -G output
The total of AIX memory mapped pages is the sum of pers and clnt columns, for both pin and inuse rows. In Figure 129, 234+1,585,658=1,585,892, which is over 1/3 of real memory. Since SAP and Oracle memory use can vary widely from hour to hour, run the svmon command periodically for several days or a week, to determine the minimum number of (persistent + client) pages over the period. This minimum (pers+clnt pin+inuse) should be compared with vmo maxperm, to determine if there is available memory. Make changes gradually, and dont over-allocate the memory in Oracle and SAP buffers, as that can cause AIX paging. Dont add memory to Oracle or SAP just because AIX shows that there are available pages. If Oracle hit rates are low, or SAP buffers need to be increased to support the workload, then increasing the size of the Oracle and SAP buffers is reasonable.
IBM Americas ATS
11. Four guidelines for avoiding performance problems

As seen in the examples above, there are a few general rules for avoiding performance problems.
11.1.
Use the SAP data model
If youre evaluating the performance of custom code, and it runs slowly because the predicates dont match the indexes on SAP tables, the odds are very good that the program is looking for the data in the wrong place. Most SAP business documents (e.g. sales order, purchase order, delivery note, etc) can be found using the document number with a standard SAP table and standard SAP index. In addition to the example in Section 9.1.6, See SAP notes 185530, 187906, and 191492 for examples of incorrect and correct use of the SAP data model. The symptom of this problem is in the ST04 SQL cache - high buffer gets per exec and high buffer gets per row. This happens when the predicates on the SQL do not match the index columns on the table and Oracle has to read many blocks of data to retrieve the result.
11.2.
Use array operations on the database
If the program builds internal tables that contain keys for selects on other tables, evaluate whether an array operation such as FOR ALL ENTRIES can be used to perform array selects, rather than using LOOP AT with individual database calls. The symptom of this problem is seen in ST05 traces, where a program makes frequent calls to a table, and each call accesses few rows.
11.3.
Check whether the database call can be avoided
There are several versions of this problem: A table is set as buffered, but the application server generic or single record buffers are too small to hold the table rows. The cure for this is to increase the size of the SAP buffers. A table is not set to be buffered, but should be. In this case, the table is usually read-only, and the application can tolerate a small interval when the data is not in synch on all the application servers. In this case, the table attributes should be changed to buffered. A program is repeatedly fetching the same information from the database. This problem can be detected by using the duplicate selects function in ST05. The program needs to be examined to see how it can be changed so that it does not have to repeatedly go back to the database for the same information
11.4.
Write ABAP programs that are line-item scalable
If the program will process many lines in a report: Use BINARY SEARCH on the READ TABLE from itab statements with a SORTED internal table Or, evaluate whether internal tables need to be defined as HASHED 2013 International Business Machines, Inc. Page 91
IBM Americas ATS The symptom of this problem is high CPU use for a program, where CPU use does not scale with additional line items e.g. a 100 line report takes 1 second CPU, but a 1000 line item report takes more than 10 seconds CPU, and a 10,000 line report takes much more than 100 seconds CPU. These scalability problems get worse as the report lines increase.
IBM Americas ATS
12. Appendix: Reference Materials

Oracle Architecture and Tuning on AIX, Damir Rubic (IBM) o http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100883 Diagnosing Oracle Database Performance on AIX Using IBM NMON and Oracle Statspack Reports, Dale Martin (IBM) o http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101720 Configuring IBM TotalStorage for Oracle OLTP Applications, Dale Martin (IBM) o http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100319 Oracle 11g ASM on AIX with Oracle Real Applications Clusters (RAC), Rebecca Ballough (IBM) o http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102169 Breaking the Oracle I/O Performance Bottleneck, Dale Martin (IBM) o http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3885 Oracle Database and 1TB Segment Aliasing, Ralf Schmidt-Dannert (IBM) o http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105761 IBM POWER7 AIX and Oracle Database performance considerations, Sanjay Ruprell, Peter H Barnett, Wolfgang Tertel o http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102171 AIX 5.3 Performance Management Guide, IBM AIX 6.1 Performance Management Guide, IBM AIX 7.1 Performance Management Guide, IBM IBM AIX Version 6.1 Difference Guide, IBM Redbooks o http://w3.itso.ibm.com/abstracts/sg247559.html?Open IBM AIX Version 7.1 Difference Guide, IBM Redbooks o http://w3.itso.ibm.com/abstracts/sg247910.html?Open Improving DB performance with AIX Concurrent I/O: Kashyap/Olszewski/Hendrickson, IBM "PowerVM Virtualization on IBM System p: Introduction and Configuration", IBM Redbooks o http://www.redbooks.ibm.com/redpieces/abstracts/sg247940.html "PowerVM Virtualization on IBM System p: Managing and Monitoring", IBM Redbooks o http://www.redbooks.ibm.com/redpieces/abstracts/sg247590.html 2013 International Business Machines, Inc. Page 93
IBM Americas ATS IBM System p and AIX Information Center o http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp (AIX 5.3) o http://publib.boulder.ibm.com/infocenter/systems/scope/aix/index.jsp (AIX 6.1) o http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp (AIX 7.1) o `My Oracle` (Metalink) Support web site: o https://support.oracle.com o Oracle 9i Documentation website: o http://www.oracle.com/technology/documentation/oracle9i.html o Oracle 10g R2 Documentation website: o http://www.oracle.com/pls/db102/homepage o Oracle 11g R2 Documentation website: o http://www.oracle.com/pls/db112/homepage
IBM Americas ATS Figure 1: SQL with local and join predicates ................................................................................................ 9 Figure 2: Explain example ............................................................................................................................. 9 Figure 3: Access and Filter Predicates......................................................................................................... 10 Figure 4: RFBLG~0 Index........................................................................................................................... 10 Figure 5: Explain with added local predicate on BUKRS ........................................................................... 11 Figure 6: Explain cannot display cached plan ............................................................................................. 12 Figure 7: Explain with cached execution plan............................................................................................. 13 Figure 8: STAD transaction display of time components............................................................................ 14 Figure 9: ST03N ZSUMMARY .................................................................................................................. 15 Figure 10: SAT own standard variant selected ............................................................................................ 15 Figure 11: SAT gather stats on internal tables............................................................................................. 16 Figure 12: SAT Aggregation level............................................................................................................... 16 Figure 13: SAT start/end measurement ....................................................................................................... 17 Figure 14: SAT analysis............................................................................................................................... 17 Figure 15 : ZSUMMARY Desktop.............................................................................................................. 17 Figure 16: SAT sorted by Net time.............................................................................................................. 18 Figure 17: ZSUMMARY source code......................................................................................................... 18 Figure 18: Hit Lists ...................................................................................................................................... 18 Figure 19: SAT tips and tricks ..................................................................................................................... 19 Figure 20: SAT binary search test ............................................................................................................... 19 Figure 21: Frequent ABAP SORTs ............................................................................................................. 20 Figure 22: Slow LOOP AT .......................................................................................................................... 20 Figure 23: LOOP with select ....................................................................................................................... 21 Figure 24: MIRO ST03N times ................................................................................................................... 24 Figure 25: STAD FBL3N ............................................................................................................................ 24 Figure 26: FBL3N ST05 trace ..................................................................................................................... 25 Figure 27: FB3N ABAP display .................................................................................................................. 25 Figure 28: Program Attributes ..................................................................................................................... 26 Figure 29: FBL3N BSIS .............................................................................................................................. 26 Figure 30: BSIS Access Predicates.............................................................................................................. 27 Figure 31: BSIS~0 Index ............................................................................................................................. 27 Figure 32: BSIS Filter Predicates ................................................................................................................ 27 Figure 33: SAP note for FBL3N BSIS ........................................................................................................ 28 Figure 34: ZFB001 program - lots of fast DB accesses............................................................................... 29 Figure 35: ZFB001 ST05 ............................................................................................................................. 29 Figure 36: ZFB001 program source............................................................................................................. 29 Figure 37: ZCOITEMS ST05 ...................................................................................................................... 30 Figure 38: ZCOITEMS summarize ............................................................................................................. 31 Figure 39: ZCOITEMS summary ................................................................................................................ 31 Figure 40: ZCOITEMS ABAP .................................................................................................................... 31 Figure 41: ZCOITEMS LFA1 display......................................................................................................... 32 Figure 42: MB51 performance statistics...................................................................................................... 32 Figure 43: MB51 ST05 summarized trace................................................................................................... 33 Figure 44: MB51 ABAP Source .................................................................................................................. 33 2013 International Business Machines, Inc. Page 95
IBM Americas ATS Figure 45: MB51 with custom index used................................................................................................... 34 Figure 46: MB51 Access and Filter Predicates............................................................................................ 34 Figure 47: MB51 Custom Z00 Index........................................................................................................... 35 Figure 48: ST04 display of SQL cache for Oracle 11 ................................................................................. 37 Figure 49: ST04 SQL cache - problem example ......................................................................................... 38 Figure 50: ST04 SQL cache - moderately inefficient SQL ......................................................................... 38 Figure 51: Sample explain ........................................................................................................................... 39 Figure 52: VPRB Filter Predicates .............................................................................................................. 40 Figure 53: Display indexes from explain..................................................................................................... 40 Figure 54: Note 185530 - VBRP where VGBEL ........................................................................................ 41 Figure 55: Custom table SQL ...................................................................................................................... 42 Figure 56: Custom table explain .................................................................................................................. 42 Figure 57: Custom Table Access and Filter Predicates ............................................................................... 43 Figure 58: Custom Table index.................................................................................................................... 43 Figure 59: Custom program for custom table .............................................................................................. 43 Figure 60: Custom Program Attributes........................................................................................................ 44 Figure 61: VBRP high impact statement ..................................................................................................... 45 Figure 62: VBRP explain............................................................................................................................. 45 Figure 63: VBRP index................................................................................................................................ 46 Figure 64: Program accessing VBRP .......................................................................................................... 46 Figure 65: SAP note with examples of using the SD data model................................................................ 46 Figure 66: ST04 SQL cache for skew example ........................................................................................... 47 Figure 67: BKPF explain ............................................................................................................................. 48 Figure 68: BKPF~1...................................................................................................................................... 48 Figure 69: BKPF rows ................................................................................................................................. 49 Figure 70: BKPF Bind variables.................................................................................................................. 49 Figure 71: BKPF SQL to count XBLNR..................................................................................................... 49 Figure 72: BKPF XBLNR count.................................................................................................................. 50 Figure 73: BKPF~2 index............................................................................................................................ 50 Figure 74: BKPF source............................................................................................................................... 51 Figure 75: DB02 display columns with histograms..................................................................................... 51 Figure 76: DBSTATC.................................................................................................................................. 51 Figure 77: A611 in SQL cache .................................................................................................................... 52 Figure 78: A611 ABAP source SD_COND_ACCESS ............................................................................... 53 Figure 79: ST02 Buffer Display .................................................................................................................. 53 Figure 80: ST02 Generic Key ...................................................................................................................... 54 Figure 81: A611 Error State......................................................................................................................... 54 Figure 82: ST02 Buffer Status ..................................................................................................................... 55 Figure 83: DB05 A611................................................................................................................................. 56 Figure 84: SE13 A611 ................................................................................................................................. 57 Figure 85: EKET 16 million rows ............................................................................................................... 58 Figure 86: Explain EKET ............................................................................................................................ 58 Figure 87: EKET program source................................................................................................................ 59 Figure 88: D010INC high bgets/Exec.......................................................................................................... 60 Figure 89:D010INC explain ........................................................................................................................ 60 2013 International Business Machines, Inc. Page 96
IBM Americas ATS Figure 90:D010INC indexes ........................................................................................................................ 61 Figure 91: D010INC read with index .......................................................................................................... 62 Figure 92: CRCO ST04 Shared Cursor Cache ............................................................................................ 62 Figure 93: CRCO Explain............................................................................................................................ 63 Figure 94: CRCO Predicates........................................................................................................................ 63 Figure 95: CRCO~0 index ........................................................................................................................... 64 Figure 96: CRCO program........................................................................................................................... 64 Figure 97: BSIS range predicate statement statistics................................................................................... 65 Figure 98: BSIS explain............................................................................................................................... 65 Figure 99: BSIS Access Predicates.............................................................................................................. 66 Figure 100: BSIS~Z03 Index....................................................................................................................... 66 Figure 101: BSIS ASH Sample ................................................................................................................... 67 Figure 102: BSIS Bind Variables ................................................................................................................ 67 Figure 103: BSIS prepared with values ....................................................................................................... 68 Figure 104: BSIS with values access and filter predicates .......................................................................... 68 Figure 105: AUFK statement statistics ........................................................................................................ 69 Figure 106: AUFK explain .......................................................................................................................... 69 Figure 107: AUFK filter predicates ............................................................................................................. 70 Figure 108: AUFK~D access predicates...................................................................................................... 70 Figure 109: AUFK~D index ........................................................................................................................ 70 Figure 110: AUFK Bind Variables .............................................................................................................. 70 Figure 111: AUFK TAANA ........................................................................................................................ 71 Figure 112: AUFK Source ........................................................................................................................... 71 Figure 113: AUFK with values.................................................................................................................... 72 Figure 114: ST03N by DB time................................................................................................................... 73 Figure 115: ST03N Table Statistics............................................................................................................. 73 Figure 116: ST06N with AIX virtualization support................................................................................... 75 Figure 117: ST06N over Entitlement........................................................................................................... 76 Figure 118: ST04 file system requests with good response times ............................................................... 78 Figure 119: V$SYSTEM_EVENT .............................................................................................................. 78 Figure 120: ST04 file system requests......................................................................................................... 79 Figure 121: ST04 database overview........................................................................................................... 81 Figure 122: ST04 - Oracle v$system_event delay statistics ........................................................................ 82 Figure 123: ST04 Oracle sessions................................................................................................................ 82 Figure 124: ST10 not buffered tables .......................................................................................................... 83 Figure 125: SM04 > goto > memory ........................................................................................................... 87 Figure 126: SM04 Sessions ......................................................................................................................... 87 Figure 127: ST03N memory use statistics ................................................................................................... 88 Figure 128: ST06 displays free memory...................................................................................................... 89 Figure 129: svmon -G output....................................................................................................................... 90

Tuning SAP Oracle AIX v.2.0

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Tuning SAP Oracle AIX v.2.0

Uploaded by

Copyright:

Available Formats

IBM Americas ATS

IBM SAP Technical Brief

Tuning SAP with Oracle on AIX

Mark Gordon IBM Solutions ATS

Version: 2.0 Date: May 3, 2013

2013 International Business Machines, Inc.

2013 International Business Machines, Inc. Page 2

FOUR GUIDELINES FOR AVOIDING PERFORMANCE PROBLEMS............................. 91

2013 International Business Machines, Inc. Page 4

IBM Americas ATS

2013 International Business Machines, Inc. Page 5

IBM Americas ATS

2013 International Business Machines, Inc. Page 6

IBM Americas ATS

2013 International Business Machines, Inc. Page 8

IBM Americas ATS

Local and Join Predicates

Figure 1: SQL with local and join predicates

Figure 2: Explain example

2013 International Business Machines, Inc. Page 9

Access and Filter Predicates

Figure 3: Access and Filter Predicates

Figure 4: RFBLG~0 Index

Figure 5: Explain with added local predicate on BUKRS

2013 International Business Machines, Inc. Page 11

IBM Americas ATS

Check for execution plan in cache

Figure 6: Explain cannot display cached plan

2013 International Business Machines, Inc. Page 12

Figure 7: Explain with cached execution plan

2013 International Business Machines, Inc. Page 13

IBM Americas ATS

8. Analyzing a problem with a specific program or transaction

Components of SAP response time

Figure 8: STAD transaction display of time components

2013 International Business Machines, Inc. Page 14

IBM Americas ATS

Majority of time is CPU on application server

8.2.1. Performance problem is high CPU time

Figure 9: ST03N ZSUMMARY

Figure 10: SAT own standard variant selected

2013 International Business Machines, Inc. Page 15

Figure 11: SAT gather stats on internal tables

Figure 12: SAT Aggregation level

Figure 13: SAT start/end measurement

Figure 14: SAT analysis

Display the trace selected in the Evaluate tab.

Figure 15 : ZSUMMARY Desktop

Figure 16: SAT sorted by Net time

Figure 17: ZSUMMARY source code

Figure 18: Hit Lists

2013 International Business Machines, Inc. Page 18

Figure 19: SAT tips and tricks

Figure 20: SAT binary search test

2013 International Business Machines, Inc. Page 19

IBM Americas ATS

8.2.1.1. Other common ABAP problems

Figure 21: Frequent ABAP SORTs

Figure 22: Slow LOOP AT

8.2.1.2. Summary of Performance problem is high CPU time

Majority of time is Database Request time

8.3.1. Types of problems causing high DB request time

8.3.1.1. Indexes do not support predicates