a010029

Part One: A Tactical Overview of Power/AIX

Virtual Memory Management Mechanisms
Earl Jew [updated since IBM Technical University/Cairo January 12-14, 2016]
Senior IT Consultant for IBM Power Systems and System Storage
IBM STG Lab Services Power Systems Delivery Practice

© Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
 ABSTRACT

Part One: An Overview of Power/AIX Virtual Memory Management

Mechanisms

For the day-to-day AIX system administrator, managing the

complexities of the Power/AIX VMM (Virtual Memory Manager) boils-
down to only a few operational concepts, numeric indicators and
AIX:vmo/ioo kernel settings. This session introduces the Power/AIX
VMM mechanisms and describes the indicators and settings that
govern how to optimize VMM for performance and efficiency.

Earl Jew (earlj@us.ibm.com) 00-1-310-251-2907 cell

Senior IT Management Consultant - IBM Power Systems and IBM Systems Storage
IBM STG Lab Services Power Systems Delivery Practice
400 North Brand Blvd., c/o IBM 7th floor, Glendale, California, USA 91203

© Copyright IBM Corporation 2016. Technical University/Symposia materials 1

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Learning What-to-know How-to-see-it Then-do-this

(a010029) Part One: A Tactical Overview of Power/AIX Virtual Memory

Manager Mechanisms

(a010030) Part Two: The Four Dimensions of POWER7 and POWER8 Affinity

(a010031) Part Three: How to Use Power/AIX Historical, Cumulative Statistics to

Indicate Performance Issues

(a010032) Part Four: How to Use Power/AIX Real-Time Statistics to Indicate

Performance Issues

(a010033) Part Five: Remedial Tactics for Performance Tuning the Indicated Issues
of Power/AIX Workloads

© Copyright IBM Corporation 2016. Technical University/Symposia materials 2

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Learning What-to-know How-to-see-it Then-do-this

(a010029) Part One: A Tactical Overview of Power/AIX Virtual Memory Manager

Mechanisms

(a010030) Part Two: The Four Dimensions of POWER7 and POWER8 Affinity

(a010031) Part Three: How to Use Power/AIX Historical, Cumulative Statistics

to Indicate Performance Issues

(a010032) Part Four: How to Use Power/AIX Real-Time Statistics to Indicate

Performance Issues

(a010033) Part Five: Remedial Tactics for Performance Tuning the Indicated Issues
of Power/AIX Workloads

© Copyright IBM Corporation 2016. Technical University/Symposia materials 3

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Learning What-to-know How-to-see-it Then-do-this

(a010029) Part One: A Tactical Overview of Power/AIX Virtual Memory Manager

Mechanisms

(a010030) Part Two: The Four Dimensions of POWER7 and POWER8 Affinity

(a010031) Part Three: How to Use Power/AIX Historical, Cumulative Statistics to

Indicate Performance Issues

(a010032) Part Four: How to Use Power/AIX Real-Time Statistics to Indicate

Performance Issues

(a010033) Part Five: Remedial Tactics for Performance Tuning the Indicated
Issues of Power/AIX Workloads

© Copyright IBM Corporation 2016. Technical University/Symposia materials 4

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 This is a semi-contiguous chunk of AIX LPAR main memory.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 5

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX These are AIX processes and the pinned memory buffers of AIX.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 6

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
User Computational Memory

This is memory used to house executable code comprised of

user processes/threads, shared memory, TCP/IP connec-
tions, etc. (aka Computational Memory). User Computational
Memory is created&generated when a program is started; it is
the code that is executed on CPUs and generally resides in
real memory, i.e. DB2/Oracle rdbms, SAP, application code.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 7

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Processes are Computational Memory

AIX:ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Mar 27 - 9:25 /etc/init
root 1966248 3342568 0 Mar 27 - 1:24 /usr/sbin/syslogd -M all
root 2031806 1 0 Mar 27 - 1:47 /usr/sbin/getty /dev/console
root 2097340 1 0 Mar 27 - 10:38 /usr/sbin/syncd 60
root 2359470 3342568 0 Mar 27 - 0:00 /usr/sbin/portmap
root 2556120 1 0 Mar 27 - 0:00 /usr/ccs/bin/shlap64
root 2621674 3342568 0 Mar 27 - 0:07 /usr/sbin/xntpd
root 2752680 1 0 Mar 27 - 0:00 /usr/lib/errdemon
root 2818236 3342568 0 Mar 27 - 0:00 /usr/sbin/nimsh -s
oracle 3014662 1 0 Mar 27 - 0:01 ora_arc2_euw
root 3080386 3342568 0 Mar 27 - 0:00 /usr/sbin/qdaemon
oracle 3145736 1 1 06:52:00 - 1:09 ora_p091_wiggle
root 3342568 1 0 Mar 27 - 0:03 /usr/sbin/srcmstr
root 3473596 3342568 0 Mar 27 - 0:00 /usr/sbin/biod 6
root 3539118 1 0 Mar 27 - 0:46 /usr/lpp/PD/bin/ovcd
oracle 3604490 1 0 Mar 27 - 0:04 ora_mman_euw
oracle 3670222 1 0 Mar 27 - 0:05 ora_dbrm_euw
root 3801328 3539118 0 Mar 27 - 60:40 /usr/lpp/PD/lbin/perf/coda
root 3866858 7799444 0 08:02:05 - 0:00 ksh /home/pabflore/ubin/dmp5 -l pabflores@wiggle -a -p 2/7 -s -z plan_group_dim
root 3932384 1 0 Mar 27 - 0:00 /opt/opsware/agent/bin/python /opt/opsware/agent/pylibs/shadowbot/daemonbot.pyc -…
root 3997868 1 0 Mar 27 - 1:18 /usr/bin/topasrec -L -s 300 -R 1 -r 6 -o /etc/perf/daily/ -ypersistent=1 -O typ…
root 4194522 3342568 0 Mar 27 - 0:00 /usr/sbin/automountd
root 4325564 3539118 0 Mar 27 - 0:02 /usr/lpp/PD/lbin/eaagt/opcle -std
root 4390966 3539118 0 Mar 27 - 0:41 /usr/lpp/PD/lbin/eaagt/opcmsga
oracle 4522144 1 0 Mar 27 - 3:53 ora_mmnl_euw
oracle 4653162 1 0 Mar 27 - 0:18 ora_mmon_euw
oracle 4784300 1 2 06:52:03 - 1:08 ora_p124_wiggle
oracle 4849888 1 1 06:51:59 - 1:09 ora_p073_wiggle
root 4915408 3539118 0 Mar 27 - 1:03 /usr/lpp/PD/lbin/agtrep/agtrep -start
root 5111874 1 2 Mar 27 - 168:12 /opt/BESClient/bin/BESClient
oracle 5374072 1 0 Mar 27 - 0:05 ora_diag_euw
oracle 5439610 1 0 Mar 27 - 3:55 ora_vktm_euw
oracle 5570670 1 0 Mar 27 - 0:25 ora_psp0_euw
oracle 5636176 1 0 Mar 27 - 16:22 ora_dia0_euw
oracle 5701716 1 0 Mar 27 - 0:10 ora_dbw0_euw

© Copyright IBM Corporation 2016. Technical University/Symposia materials 8

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Shared Memory Segments are Computational Memory

AIX:ipcs -bm
IPC status from /dev/mem as of Thu Apr 2 07:47:14 PDT 2015
T ID KEY MODE OWNER GROUP SEGSZ
Shared Memory:
m 1048576 0x52e74b4f --rw-rw-rw- imnadm imnadm 36028
m 1048577 0x9308e451 --rw-rw-rw- imnadm imnadm 97948
m 1048578 0xe4663d62 --rw-rw-rw- imnadm imnadm 96
m 3 0xc76283cc --rw-rw-rw- imnadm imnadm 42268
m 4 0x298ee665 --rw-rw-rw- imnadm imnadm 2844
m 50331653 0x0101216b --rw-rw-r-- root system 4192
m 255852550 0x78000100 --rw-rw-rw- root system 33554432
m 32505863 0x01012141 --rw-rw-r-- root system 4192
m 49283081 0x4d320660 --rw-rw---- oracle dba 935342080
m 10 0x0101212f --rw-rw-r-- root system 4192
m 11 0x010120a3 --rw-rw-r-- root system 4192
m 19 0x01012217 --rw-rw-r-- root system 4192
m 745537556 0x01012295 --rw-rw-r-- root system 16
m 31457301 0x06347849 --rw-rw-rw- root system 65544
m 137363478 0x78100100 --rw-rw-rw- root system 33554432
m 173015063 0x4103b5bf --rw------- root system 53764
m 24 0x0c6629c9 --rw-rw-rw- root system 2994648
m 25 0x31012100 --rw-rw-rw- root system 131176
m 1048602 0x4303b5bf --rw------- root system 424
m 246415387 0x047f57a4 --rw-rw---- oracle dba 21474861056
m 915406876 0x78001441 --rw-rw-rw- root system 222696000
m 29 0x01012063 --rw-rw-r-- root system 4192
m 205520926 0x7800153c --rw-rw-rw- root system 2656
m 922746911 0x01012317 --rw-rw-r-- root system 4192
…

© Copyright IBM Corporation 2016. Technical University/Symposia materials 9

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 TCP/IP Connections are Computational Memory

AIX:netstat
Active Internet connections
PCB/ADDR Proto Recv-Q Send-Q Local Address Foreign Address (state)
f1000500105e7bb8 tcp6 0 0 ::1.pvalarm *.* LISTEN
f1000500016d83b8 tcp4 0 0 wiggleworm8.ssh sea-ax0191.d.54221 ESTABLISHED
f10005001733fbb8 tcp4 0 0 loopback.EvMgrC loopback.33950 ESTABLISHED
f100050017337bb8 tcp4 0 0 loopback.33950 loopback.EvMgrC ESTABLISHED
f1000500024293b8 tcp4 0 0 wiggleworm8.ncube-lm wiggleworm8.43295 ESTABLISHED
f1000500005233b8 tcp4 0 0 wiggleworm8.43295 wiggleworm8.ncube-lm ESTABLISHED
f10005001056c3b8 tcp4 0 0 wiggleworm8.33845 lax-backup00.8405 ESTABLISHED
f100050001311bb8 tcp4 0 0 wiggleworm8.44917 sea-dc04.dav.ldap ESTABLISHED
f10005000a031bb8 tcp4 0 0 wiggleworm8.ssh turtle7tusaj0.33036 ESTABLISHED
f10005001064bbb8 tcp4 0 0 wiggleworm8.ssh turtle7tusaj0.33038 ESTABLISHED
f100050011356bb8 tcp4 0 0 loopback.33947 loopback.33948 ESTABLISHED
f10005001135e3b8 tcp4 0 0 loopback.33948 loopback.33947 ESTABLISHED
f100050000546bb8 tcp4 0 0 loopback.33958 loopback.33959 ESTABLISHED
f100050000540bb8 tcp4 0 0 loopback.33959 loopback.33958 ESTABLISHED
f1000500105ecbb8 tcp4 0 0 wiggleworm8.pvalarm *.* LISTEN
f100050010588bb8 tcp4 0 0 wiggleworm8.ncube-lm den-2sxfry1-.56790 ESTABLISHED
f100050000cc9bb8 tcp4 0 0 wiggleworm8.ncube-lm den-jb5tyy1-.65303 ESTABLISHED
f1000500105983b8 tcp4 0 0 wiggleworm8.33829 den2-backup0.8405 ESTABLISHED
f100050000a42bb8 tcp4 0 0 wiggleworm8.ncube-lm *.* LISTEN
f1000500105773b8 tcp4 0 0 wiggleworm8.33841 dr-cvma03.da.8405 ESTABLISHED
f10005000a05d3b8 tcp4 0 0 wiggleworm8.ncube-lm sea-sqldist0.1134 ESTABLISHED
f1000500105e3bb8 tcp4 0 0 loopback.pvalarm *.* LISTEN
f1000500105613b8 tcp4 0 0 wiggleworm8.33849 orl2-backup0.8405 ESTABLISHED
f10005001161f3b8 tcp4 0 0 wiggleworm8.44916 sea-dc03.dav.msft-gc ESTABLISHED
f1000500105b03b8 tcp4 0 0 wiggleworm8.ssh sea-ax0016.48671 ESTABLISHED
f1000500105a03b8 tcp4 0 0 wiggleworm8.33826 ded-backup00.8405 ESTABLISHED
f100050001072bb8 tcp4 0 0 wiggleworm8.ncube-lm sea-lx0028.d.48220 ESTABLISHED
f1000500013a13b8 tcp4 0 0 wiggleworm8.ncube-lm sea-lx0028.d.48240 ESTABLISHED
f1000500105bc3b8 tcp4 0 0 loopback.3d-nfsd *.* LISTEN
f10005000a3833b8 tcp4 0 0 wiggleworm8.ncube-lm sea-resc04.d.teredo ESTABLISHED
f10005001056f3b8 tcp4 0 0 wiggleworm8.33844 irv-backup02.8405 ESTABLISHED
f1000500007a9bb8 tcp4 0 16 wiggleworm8.ncube-lm 172.16.50.36.50564 ESTABLISHED
f10005000a200bb8 tcp4 0 0 wiggleworm8.ncube-lm 172.16.50.36.50566 ESTABLISHED

© Copyright IBM Corporation 2016. Technical University/Symposia materials 10

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX Computational Memory includes all of both AIX and User
Computational Memory in the same list. This is the highest
User Computational Memory

priority memory to keep resident in main memory. It is ideally

60%-80% of a healthy LPAR’s main memory.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 11

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

Free Memory is actually a list of memory pages that are

immediately available for allocation/assignment upon request;
freemem it is not a list of only contiguous pages. This list is the next
most important memory after Computational Memory.
Remember: “Free Memory drives everything.”

© Copyright IBM Corporation 2016. Technical University/Symposia materials 12

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

freemem
File Cache

Generally what is not Computational or Free Memory

will be used as File Cache, the lowest priority memory.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 13

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

When Free Memory is Low, it is replenished by a very

freemem
high-priority AIX kernel process called lrud.
File Cache

© Copyright IBM Corporation 2016. Technical University/Symposia materials 14

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

When Free Memory is Low, it is replenished by a very

freemem high-priority kernel process called lrud.
File Cache

The lrud process scans the File Cache for “old” least
recently used File pages to reassign as Free Memory.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 15

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

Computational Memory grows by using Free Memory.

But it can only grow if Free Memory is available.
freemem
File Cache

© Copyright IBM Corporation 2016. Technical University/Symposia materials 16

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 The Dynamic Nature of the AIX VMM Mechanisms

AIX AIX AIX AIX AIX

Computational Memory

Computational Memory
Computational Memory

Computational Memory

Computational Memory
freemem
freemem freemem
freemem
freemem

Cache
File
Cache

Cache

Cache

Cache
File

File

File

IBM Systems Technical Events | ibm.com/training/events File

© Copyright IBM Corporation 2016. Technical University/Symposia materials
may not be reproduced in whole or in part without the prior written permission of
17

IBM.
 AIX
Computational Memory

Sometimes the Free Memory list can be completely

empty. This is Not Good, but Not Ugly. Just Slower.
Cache
File

© Copyright IBM Corporation 2016. Technical University/Symposia materials 18

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

When Free Memory is Empty, anything that needs

Free Memory must wait for Free Memory to be freed
by lrud. The File Cache is scanned for “old” pages to
Cache

replenish Free Memory, and the File Cache “shrinks”.

File

© Copyright IBM Corporation 2016. Technical University/Symposia materials 19

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

freemem When Free Memory is Empty, anything that needs

Free Memory must wait for Free Memory to be freed
Cache
File

by lrud. The File Cache is scanned for “old” pages to

replenish Free Memory, and the File Cache “shrinks”.
© Copyright IBM Corporation 2016. Technical University/Symposia materials 20
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

freemem
And as the File Cache “shrinks”, there are fewer and
Cache
File

fewer “old” pages to be easily scanned&freed by lrud.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 21

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

But also as the File Cache shrinks, there are fewer File
freemem Cache “hits”, and thus there are more hard page faults
File Cache
to the disk storage system – which itself also needs
Free Memory to hold its content in the File Cache.
© Copyright IBM Corporation 2016. Technical University/Symposia materials 22
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

Because of this, there is often a sudden unquenchable

demand for Free Memory to grow Computational
freemem Memory AND to service disk storage IO at the same
File Cache time. This is called Free Memory Thrashing.
© Copyright IBM Corporation 2016. Technical University/Symposia materials 23
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX
Computational Memory

At some point, the File Cache becomes too small to

support any Free Memory demand via lrud. This is the
freemem
File Cache
point when “AIX eats itself to feed itself”; this is Ugly.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 24

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX eats itself to feed itself

AIX
pagingspace_vg Introducing the AIX:LVM:pagingspace_vg
on fast SAN disks that is built on fast SAN storage hdisks;
this is an AIX pagingspace that is equal in
size to the LPAR’s allocation of RAM.
Computational Memory

0% full
Albeit an ancient relic, it is still needed
today. Do not argue: It is still needed!!

All production LPARs should have an AIX

pagingspace built in this fashion.

At some point, the File Cache becomes too small to

support any Free Memory demand via lrud. This is the
freemem
File Cache
point when “AIX eats itself to feed itself”; this is Ugly.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 25

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX eats itself to feed itself

AIX
pagingspace_vg When “AIX eats itself to feed itself”, lrud
on fast SAN disks begins copying “old” least recently used
Computational Memory pages to the AIX
Computational Memory

pagingspace before reassigning these

1,2,3% full Computational Memory pages to the Free
Memory list, (aka pagingspace-pageout’s).

Unfortunately writing any content to a disk

storage system is substantially slow when
Free Memory is immediately needed.

At some point, the File Cache becomes too small to

pagingspace
pageout support any Free Memory demand via lrud. This is the
freemem
File Cache
point when “AIX eats itself to feed itself”; this is Ugly.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 26

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX eats itself to feed itself

AIX
pagingspace_vg When “AIX eats itself to feed itself”, lrud
on fast SAN disks begins copying “old” least recently used
Computational Memory pages to the AIX
Computational Memory

pagingspace before reassigning these

3% full Computational Memory pages to the Free
Memory list, (aka pagingspace-pageout’s).

Unfortunately writing any content to a disk

storage system is substantially slow when
Free Memory is already sorely needed.

At some point, the File Cache becomes too small to

freemem support any Free Memory demand via lrud. This is the
File Cache
point when “AIX eats itself to feed itself”; this is Ugly.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 27

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX eats itself to feed itself

AIX
pagingspace_vg When “AIX eats itself to feed itself”, lrud
on fast SAN disks begins copying “old” least recently used
Computational Memory pages to the AIX
pagingspace before reassigning these
Computational Memory

3% full Computational Memory pages to the Free

Memory list, (aka pagingspace-pageout’s).

Unfortunately writing any content to a disk

storage system is substantially slow when
Free Memory is already sorely needed.

But soon again the File Cache becomes too small to

freemem
support any Free Memory demand via lrud. When AIX
File Cache
eats itself to feed itself again, it pagingspace-pageouts.
© Copyright IBM Corporation 2016. Technical University/Symposia materials 28
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX eats itself to feed itself

AIX When “AIX eats itself to feed itself”, lrud

pagingspace_vg
on fast SAN disks begins copying “old” least recently used
Computational Memory pages to the AIX
pagingspace before reassigning these
Computational Memory

4,5% full Computational Memory pages to the Free

Memory list, (aka pagingspace-pageout’s).

Unfortunately writing any content to a disk

storage system is substantially slow when
Free Memory is already sorely needed.

pagingspace
But again the File Cache soon becomes too small to
pageouts
support any Free Memory demand via lrud. When AIX
freemem
File Cache
eats itself to feed itself again, it pagingspace-pageouts.
© Copyright IBM Corporation 2016. Technical University/Symposia materials 29
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX eats itself to feed itself

AIX
pagingspace_vg When “AIX eats itself to feed itself”, lrud
on fast SAN disks begins copying “old” least recently used
Computational Memory pages to the AIX
pagingspace before reassigning these
Computational Memory

5% full Computational Memory pages to the Free

Memory list, (aka pagingspace-pageout’s).

Unfortunately writing any content to a disk

storage system is substantially slow when
Free Memory is already sorely needed.

freemem Remember: “Free Memory drives everything”.

File Cache

© Copyright IBM Corporation 2016. Technical University/Symposia materials 30

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 HMC LPAR profile AIX:lparstat -i (1 of 2)

AIX:lparstat –i
Thu Apr 2 07:44:50 PDT 2015
AIX wiggleworm8 1 6 00X999XX9X00
6100-07-08-1339
Node Name : wiggleworm8
Partition Name : wiggleworm8
Partition Number : 10
Type : Shared-SMT-4
Mode : Uncapped
Entitled Capacity : 3.50
Partition Group-ID : 32778
Shared Pool ID : 4
Online Virtual CPUs : 6
Maximum Virtual CPUs : 6
Minimum Virtual CPUs : 1
Online Memory : 98304 MB
Maximum Memory : 131072 MB
Minimum Memory : 4096 MB
Variable Capacity Weight : 128
Minimum Capacity : 0.10
Maximum Capacity : 6.00
Capacity Increment : 0.01
Maximum Physical CPUs in system : 16
Active Physical CPUs in system : 16
Active CPUs in Pool : 6

© Copyright IBM Corporation 2016. Technical University/Symposia materials 31

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 HMC LPAR profile AIX:lparstat -i (2 of 2)

…
Active CPUs in Pool : 6
Shared Physical CPUs in system : 13
Maximum Capacity of Pool : 600
Entitled Capacity of Pool : 350
Unallocated Capacity : 0.00
Physical CPU Percentage : 58.33%
Unallocated Weight : 0
Memory Mode : Dedicated
Total I/O Memory Entitlement : -
Variable Memory Capacity Weight : -
Memory Pool ID : -
Physical Memory in the Pool : -
Hypervisor Page Size : -
Unallocated Variable Memory Capacity Weight: -
Unallocated I/O Memory entitlement : -
Memory Group ID of LPAR : -
Desired Virtual CPUs : 6
Desired Memory : 98304 MB
Desired Variable Capacity Weight : 128
Desired Capacity : 3.50
Target Memory Expansion Factor : -
Target Memory Expansion Size : -
Power Saving Mode : Disabled

© Copyright IBM Corporation 2016. Technical University/Symposia materials 32

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the amount of only AIX memory

AIX These are AIX processes and the pinned memory buffers of AIX.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 33

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the amount of only AIX memory

AIX:vmstat –v
25165824 memory pages
19112512 lruable pages
9201 free pages
4 memory pools
8028287 pinned pages  AIX is generally comprised of only pinned memory pages
80.0 maxpin percentage
3.0 minperm percentage
90.0 maxperm percentage
75.8 numperm percentage
14492435 file pages
0.0 compressed percentage
0 compressed pages
75.8 numclient percentage
90.0 maxclient percentage
14492435 client pages
0 remote pageouts scheduled
857 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
1972 filesystem I/Os blocked with no fsbuf
9900 client filesystem I/Os blocked with no fsbuf
209695 external pager filesystem I/Os blocked with no fsbuf
42.4 percentage of memory used for computational pages

© Copyright IBM Corporation 2016. Technical University/Symposia materials 34

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the percentage of Computational Memory

AIX Computational Memory includes all of both AIX and User

Computational Memory in the same list. This is the highest
User Computational Memory

priority memory to keep resident in main memory. It is ideally

60%-80% of a healthy LPAR’s main memory.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 35

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the percentage of Computational Memory

AIX:vmstat –v
25165824 memory pages
19112512 lruable pages
9201 free pages
4 memory pools
8028287 pinned pages  Generally AIX is comprised of only pinned memory pages
80.0 maxpin percentage
3.0 minperm percentage
90.0 maxperm percentage
75.8 numperm percentage
14492435 file pages
0.0 compressed percentage
0 compressed pages
75.8 numclient percentage
90.0 maxclient percentage
14492435 client pages
0 remote pageouts scheduled
857 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
1972 filesystem I/Os blocked with no fsbuf
9900 client filesystem I/Os blocked with no fsbuf
209695 external pager filesystem I/Os blocked with no fsbuf
42.4 percentage of memory used for computational pages  aka COMP%

© Copyright IBM Corporation 2016. Technical University/Symposia materials 36

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the amount of Free Memory

AIX
Computational Memory

Free Memory is actually a list of memory pages that are

immediately available for allocation/assignment upon request;
freemem it is not a list of only contiguous pages. This list is the next
most important memory after Computational Memory.
Remember: “Free Memory drives everything.”

© Copyright IBM Corporation 2016. Technical University/Symposia materials 37

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the amount of Free Memory

AIX:vmstat –v
25165824 memory pages
19112512 lruable pages
9201 free pages  This is the number of Free Pages on the Free Memory list
4 memory pools
8028287 pinned pages  Generally AIX is comprised of only pinned memory pages
80.0 maxpin percentage
3.0 minperm percentage
90.0 maxperm percentage
75.8 numperm percentage
14492435 file pages
0.0 compressed percentage
0 compressed pages
75.8 numclient percentage
90.0 maxclient percentage
14492435 client pages
0 remote pageouts scheduled
857 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
1972 filesystem I/Os blocked with no fsbuf
9900 client filesystem I/Os blocked with no fsbuf
209695 external pager filesystem I/Os blocked with no fsbuf
42.4 percentage of memory used for computational pages  aka COMP%

© Copyright IBM Corporation 2016. Technical University/Symposia materials 38

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the number of File Cache pages

AIX
Computational Memory

freemem
File Cache

Generally what is not Computational or Free Memory

will be used as File Cache, the lowest priority memory.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 39

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the number of File Cache pages

AIX:vmstat –v
25165824 memory pages
19112512 lruable pages
9201 free pages  This is the number of Free Pages on the Free Memory list
4 memory pools
8028287 pinned pages  Generally AIX is comprised of only pinned memory pages
80.0 maxpin percentage
3.0 minperm percentage
90.0 maxperm percentage
75.8 numperm percentage
14492435 file pages  This is the number of JFS/JFS2/NFS/VxFS File Cache pages
0.0 compressed percentage
0 compressed pages
75.8 numclient percentage
90.0 maxclient percentage
14492435 client pages  This is the number of JFS2 File Cache pages
0 remote pageouts scheduled
857 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
1972 filesystem I/Os blocked with no fsbuf
9900 client filesystem I/Os blocked with no fsbuf
209695 external pager filesystem I/Os blocked with no fsbuf
42.4 percentage of memory used for computational pages  aka COMP%

© Copyright IBM Corporation 2016. Technical University/Symposia materials 40

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the percent of File Cache

AIX:vmstat –v
25165824 memory pages
19112512 lruable pages
9201 free pages  This is the number of Free Pages on the Free Memory list
4 memory pools
8028287 pinned pages  Generally AIX is comprised of only pinned memory pages
80.0 maxpin percentage
3.0 minperm percentage
90.0 maxperm percentage
75.8 numperm percentage  This is the percent of JFS/JFS2/NFS/VxFS File Cache
14492435 file pages  This is the number of JFS/JFS2/NFS/VxFS File Cache pages
0.0 compressed percentage
0 compressed pages
75.8 numclient percentage  This is the percent of JFS2-only File Cache
90.0 maxclient percentage
14492435 client pages  This is the number of JFS2 File Cache pages
0 remote pageouts scheduled
857 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
1972 filesystem I/Os blocked with no fsbuf
9900 client filesystem I/Os blocked with no fsbuf
209695 external pager filesystem I/Os blocked with no fsbuf
42.4 percentage of memory used for computational pages  aka COMP%

© Copyright IBM Corporation 2016. Technical University/Symposia materials 41

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 What is the trigger for pagingspace-pageouts

AIX
pagingspace_vg When “AIX eats itself to feed itself”, lrud
on fast SAN disks begins copying “old” least recently used
Computational Memory pages to the AIX
Computational Memory

pagingspace before reassigning these

1,2,3% full Computational Memory pages to the Free
Memory list, (aka pagingspace-pageout’s).

Unfortunately writing any content to a disk

storage system is substantially slow when
Free Memory is immediately needed.

At some point, the File Cache becomes too small to

pagingspace
pageout support any Free Memory demand via lrud. This is the
freemem
File Cache
point when “AIX eats itself to feed itself”; this is Ugly.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 42

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 What is the trigger for pagingspace-pageouts

AIX:vmstat –v
25165824 memory pages
19112512 lruable pages
9201 free pages  This is the number of Free Pages on the Free Memory list
4 memory pools
8028287 pinned pages  Generally AIX is comprised of only pinned memory pages
80.0 maxpin percentage
3.0 minperm percentage  This value is the trigger for pagingspace-pageouts
90.0 maxperm percentage
75.8 numperm percentage  This is the percent of JFS/JFS2/NFS/VxFS File Cache
14492435 file pages  This is the number of JFS/JFS2/NFS/VxFS File Cache pages
0.0 compressed percentage
0 compressed pages
75.8 numclient percentage  This is the percent of JFS2-only File Cache
90.0 maxclient percentage
14492435 client pages  This is the number of JFS2 File Cache pages
0 remote pageouts scheduled
857 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
1972 filesystem I/Os blocked with no fsbuf
9900 client filesystem I/Os blocked with no fsbuf
209695 external pager filesystem I/Os blocked with no fsbuf
42.4 percentage of memory used for computational pages  aka COMP%

© Copyright IBM Corporation 2016. Technical University/Symposia materials 43

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the Percent Used of the pagingspace

The AIX:LVM:pagingspace_vg is built on

pagingspace_vg fast SAN storage hdisks; this is an AIX
on fast SAN disks
pagingspace that is equal in size to the
LPAR’s allocation of RAM.

1% full All production LPARs should have an AIX

pagingspace built in this fashion.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 44

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the Percent Used of the pagingspace

lsps –a; lsps -s

Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum
hd6 hdisk0 rootvg 20480MB 1 yes yes lv 0 # realigned for clarity

Total Paging Space Percent Used

20480MB 1%

© Copyright IBM Corporation 2016. Technical University/Symposia materials 45

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the Percent Used of the pagingspace

lsps –a; lsps -s

Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum
hd6 hdisk0 rootvg 20480MB 1 yes yes lv 0 # realigned for clarity

Total Paging Space Percent Used

20480MB 1%

What happens when the AIX:pagingspace grows too full to store the demanded
pagingspace-pageouts?

© Copyright IBM Corporation 2016. Technical University/Symposia materials 46

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the Percent Used of the pagingspace

lsps –a; lsps -s

Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum
hd6 hdisk0 rootvg 20480MB 1 yes yes lv 0 # realigned for clarity

Total Paging Space Percent Used

20480MB 1%

What happens when the AIX:pagingspace grows too full to store the demanded
pagingspace-pageouts?

Yeah, I know that it rarely ever happens, but still… What happens when it’s full?

© Copyright IBM Corporation 2016. Technical University/Symposia materials 47

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 How to note the Percent Used of the pagingspace

lsps –a; lsps -s

Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum
hd6 hdisk0 rootvg 20480MB 1 yes yes lv 0 # realigned for clarity

Total Paging Space Percent Used

20480MB 1%

What happens when the AIX:pagingspace grows too full to store the demanded
pagingspace-pageouts?

Yeah, I know that it rarely ever happens, but still… What happens when it’s full?

AIX purposely CRASHES to preclude data corruption!!

© Copyright IBM Corporation 2016. Technical University/Symposia materials 48

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 AIX:vmstat –s for the count of pagingspace-pageouts

uptime ; vmstat –s
07:45AM up 5 days, 14:35, 7 users, load average: 68.90, 55.77, 53.68
13783904583 total address trans. faults
18464514733 page ins
2091137989 page outs
0 paging space page ins
0 paging space page outs  Rule#1: Acceptable Tolerance is 5-digits/90days Uptime
0 total reclaims
6953205800 zero filled pages faults
89715 executable filled pages faults
38420778917 pages examined by clock
5 revolutions of the clock hand
20036911968 pages freed by the clock
46545239 backtracks
137657525 free frame waits
0 extend XPT waits
1779946655 pending I/O waits
20598490595 start I/Os
2333693437 iodones
8560316549 cpu context switches
1900589120 device interrupts
70007091 software interrupts
1230095024 decrementer interrupts
7598754 mpc-sent interrupts
7598754 mpc-received interrupts
676750577 phantom interrupts
0 traps
32508428126 syscalls

© Copyright IBM Corporation 2016. Technical University/Symposia materials 49

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 What are free frame waits?

uptime ; vmstat –s
07:45AM up 5 days, 14:35, 7 users, load average: 68.90, 55.77, 53.68
13783904583 total address trans. faults
18464514733 page ins
2091137989 page outs
0 paging space page ins
0 paging space page outs  Rule#1: Acceptable Tolerance is 5-digits/90days Uptime
0 total reclaims
6953205800 zero filled pages faults
89715 executable filled pages faults
38420778917 pages examined by clock
5 revolutions of the clock hand
20036911968 pages freed by the clock
46545239 backtracks
137657525 free frame waits  Huh?
0 extend XPT waits
1779946655 pending I/O waits
20598490595 start I/Os
2333693437 iodones
8560316549 cpu context switches
1900589120 device interrupts
70007091 software interrupts
1230095024 decrementer interrupts
7598754 mpc-sent interrupts
7598754 mpc-received interrupts
676750577 phantom interrupts
0 traps
32508428126 syscalls

© Copyright IBM Corporation 2016. Technical University/Symposia materials 50

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 What are free frame waits?

AIX
Computational Memory

When Free Memory is Empty, any request for Free

Memory must wait for it to be freed by lrud.
These waits are counted as “Free Frame Waits”.
Cache
File

© Copyright IBM Corporation 2016. Technical University/Symposia materials 51

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Rule#1 pagingspace pageouts; Rule#2 free frame waits

uptime ; vmstat –s
07:45AM up 5 days, 14:35, 7 users, load average: 68.90, 55.77, 53.68
13783904583 total address trans. faults
18464514733 page ins
2091137989 page outs
0 paging space page ins
0 paging space page outs  Rule#1: Acceptable Tolerance is 5-digits/90days Uptime
0 total reclaims
6953205800 zero filled pages faults
89715 executable filled pages faults
38420778917 pages examined by clock
5 revolutions of the clock hand
20036911968 pages freed by the clock
46545239 backtracks
137657525 free frame waits  Rule#2: Acceptable Tolerance is 5-digits/90days Uptime
0 extend XPT waits
1779946655 pending I/O waits
20598490595 start I/Os
2333693437 iodones
8560316549 cpu context switches
1900589120 device interrupts
70007091 software interrupts
1230095024 decrementer interrupts
7598754 mpc-sent interrupts
7598754 mpc-received interrupts
676750577 phantom interrupts
0 traps
32508428126 syscalls

© Copyright IBM Corporation 2016. Technical University/Symposia materials 52

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 More about AIX:vmstat –s address translations

uptime ; vmstat –s
07:45AM up 5 days, 14:35, 7 users, load average: 68.90, 55.77, 53.68
13783904583 total address trans. faults  the count of virtual-to-real memory address translations
18464514733 page ins
2091137989 page outs
0 paging space page ins
0 paging space page outs  Rule#1: Acceptable Tolerance is 5-digits/90days Uptime
0 total reclaims
6953205800 zero filled pages faults
89715 executable filled pages faults
38420778917 pages examined by clock
5 revolutions of the clock hand
20036911968 pages freed by the clock
46545239 backtracks
137657525 free frame waits  Rule#2: Acceptable Tolerance is 5-digits/90days Uptime
0 extend XPT waits
1779946655 pending I/O waits
20598490595 start I/Os
2333693437 iodones
8560316549 cpu context switches
1900589120 device interrupts
70007091 software interrupts
1230095024 decrementer interrupts
7598754 mpc-sent interrupts
7598754 mpc-received interrupts
676750577 phantom interrupts
0 traps
32508428126 syscalls

© Copyright IBM Corporation 2016. Technical University/Symposia materials 53

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 More about AIX:vmstat –s VMM-initiated IOs

uptime ; vmstat –s
07:45AM up 5 days, 14:35, 7 users, load average: 68.90, 55.77, 53.68
13783904583 total address trans. faults  the count of virtual-to-real memory address translations
18464514733 page ins  Generally JFS/JFS2 file system VMM-initiated READs
2091137989 page outs  Generally JFS/JFS2 file system VMM-initiated WRITEs
0 paging space page ins
0 paging space page outs  Rule#1: Acceptable Tolerance is 5-digits/90days Uptime
0 total reclaims
6953205800 zero filled pages faults
89715 executable filled pages faults  Code used to create computational memory objects
38420778917 pages examined by clock
5 revolutions of the clock hand
20036911968 pages freed by the clock
46545239 backtracks
137657525 free frame waits  Rule#2: Acceptable Tolerance is 5-digits/90days Uptime
0 extend XPT waits
1779946655 pending I/O waits  IOs not completed yet when the requesting process returns to CPU
20598490595 start I/Os  Generally the sum of the above page ins and page outs
2333693437 iodones  Generally the start I/Os coalesced into fewer merged-data IO operations
8560316549 cpu context switches
1900589120 device interrupts
70007091 software interrupts
1230095024 decrementer interrupts
7598754 mpc-sent interrupts
7598754 mpc-received interrupts
676750577 phantom interrupts
0 traps
32508428126 syscalls

© Copyright IBM Corporation 2016. Technical University/Symposia materials 54

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 More about AIX:vmstat –s lrud freeing memory

uptime ; vmstat –s
07:45AM up 5 days, 14:35, 7 users, load average: 68.90, 55.77, 53.68
13783904583 total address trans. faults  the count of virtual-to-real memory address translations
18464514733 page ins  Generally JFS/JFS2 file system VMM-initiated READs
2091137989 page outs  Generally JFS/JFS2 file system VMM-initiated WRITEs
0 paging space page ins
0 paging space page outs  Rule#1: Acceptable Tolerance is 5-digits/90days Uptime
0 total reclaims
6953205800 zero filled pages faults
89715 executable filled pages faults  Code used to create computational memory objects
38420778917 pages examined by clock  Rule#3: Pages scanned by lrud when free memory is low
5 revolutions of the clock hand  Rule#4: Completed lrud scans of the entire File Cache list
20036911968 pages freed by the clock  Rule#3: Pages freed by the above lrud scanning
46545239 backtracks
137657525 free frame waits  Rule#2: Acceptable Tolerance is 5-digits/90days Uptime
0 extend XPT waits
1779946655 pending I/O waits  IOs not completed yet when the requesting process returns to CPU
20598490595 start I/Os  Generally the sum of the above page ins and page outs
2333693437 iodones  Generally the start I/Os coalesced into fewer merged-data IO operations
8560316549 cpu context switches
1900589120 device interrupts
70007091 software interrupts
1230095024 decrementer interrupts
7598754 mpc-sent interrupts
7598754 mpc-received interrupts
676750577 phantom interrupts
0 traps
32508428126 syscalls

© Copyright IBM Corporation 2016. Technical University/Symposia materials 55

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 More about AIX:vmstat –s system event counts

uptime ; vmstat –s
07:45AM up 5 days, 14:35, 7 users, load average: 68.90, 55.77, 53.68
13783904583 total address trans. faults  the count of virtual-to-real memory address translations
18464514733 page ins  Generally JFS/JFS2 file system VMM-initiated READs
2091137989 page outs  Generally JFS/JFS2 file system VMM-initiated WRITEs
0 paging space page ins
0 paging space page outs  Rule#1: Acceptable Tolerance is 5-digits/90days Uptime
0 total reclaims
6953205800 zero filled pages faults
89715 executable filled pages faults  Code used to create computational memory objects
38420778917 pages examined by clock  Pages scanned by lrud when free memory is low
5 revolutions of the clock hand  Completed lrud scans of the entire File Cache list
20036911968 pages freed by the clock  Pages freed by the above lrud scanning
46545239 backtracks
137657525 free frame waits  Rule#2: Acceptable Tolerance is 5-digits/90days Uptime
0 extend XPT waits
1779946655 pending I/O waits  IOs not completed yet when the requesting process returns to CPU
20598490595 start I/Os  Generally the sum of the above page ins and page outs
2333693437 iodones  Generally the start I/Os coalesced into fewer merged-data IO operations
8560316549 cpu context switches  Count of threads (of processes) switching on/off SMT logical CPUs
1900589120 device interrupts  Count of time-sensitive hardware device operations needing a logical CPU
70007091 software interrupts  Count of time-sensitive software interrupts needing a logical CPU
1230095024 decrementer interrupts  Tick-tock timing interrupts of only active logical CPUs (~1.95ns each)
7598754 mpc-sent interrupts
7598754 mpc-received interrupts
676750577 phantom interrupts  Device interrupts from other LPARs that pop-up here, thus “phantom”
0 traps
32508428126 syscalls  The workload of all processing boils-down to executing system calls for services

© Copyright IBM Corporation 2016. Technical University/Symposia materials 56

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 More about AIX:vmstat –s a Key Ratio

uptime ; vmstat –s
07:45AM up 5 days, 14:35, 7 users, load average: 68.90, 55.77, 53.68
13783904583 total address trans. faults  the count of virtual-to-real memory address translations
18464514733 page ins  Generally JFS/JFS2 file system VMM-initiated READs
2091137989 page outs  Generally JFS/JFS2 file system VMM-initiated WRITEs
0 paging space page ins
0 paging space page outs  Rule#1: Acceptable Tolerance is 5-digits/90days Uptime
0 total reclaims
6953205800 zero filled pages faults  Memory used to create computational memory objects
89715 executable filled pages faults  Code used to create computational memory objects
38420778917 pages examined by clock  Pages scanned by lrud when free memory is low
5 revolutions of the clock hand  Completed lrud scans of the entire File Cache list
20036911968 pages freed by the clock  Pages freed by the above lrud scanning
46545239 backtracks
137657525 free frame waits  Rule#2: Acceptable Tolerance is 5-digits/90days Uptime
0 extend XPT waits
1779946655 pending I/O waits  IOs not completed yet when the requesting process returns to CPU
20598490595 start I/Os  Generally the sum of the above page ins and page outs
2333693437 iodones  Generally the start I/Os coalesced into fewer merged-data IO operations
8560316549 cpu context switches  Count of threads (of processes) switching on/off SMT logical CPUs
1900589120 device interrupts  Count of time-sensitive hardware device operations needing a logical CPU
70007091 software interrupts  Count of time-sensitive software interrupts needing a logical CPU
1230095024 decrementer interrupts  Tick-tock timing interrupts of only active logical CPUs (~10ms each)
7598754 mpc-sent interrupts
7598754 mpc-received interrupts
676750577 phantom interrupts  Device interrupts from other LPARs that pop-up here, thus “phantom”
0 traps
32508428126 syscalls  The workload of all processing boils-down to executing system calls for services

© Copyright IBM Corporation 2016. Technical University/Symposia materials 57

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Summary of AIX:vmstat –s

uptime ; vmstat –s
07:45AM up 5 days, 14:35, 7 users, load average: 68.90, 55.77, 53.68
13783904583 total address trans. faults  the count of virtual-to-real memory address translations
18464514733 page ins  Generally JFS/JFS2 file system VMM-initiated READs
2091137989 page outs  Generally JFS/JFS2 file system VMM-initiated WRITEs
0 paging space page ins
0 paging space page outs  Rule#1: Acceptable Tolerance is 5-digits/90days Uptime
0 total reclaims
6953205800 zero filled pages faults  Memory used to create computational memory objects
89715 executable filled pages faults  Code used to create computational memory objects
38420778917 pages examined by clock  Rule#3: Pages scanned by lrud when free memory is low
5 revolutions of the clock hand  Rule#4: Completed lrud scans of the entire File Cache list
20036911968 pages freed by the clock  Rule#3: Pages freed by the above lrud scanning
46545239 backtracks
137657525 free frame waits  Rule#2: Acceptable Tolerance is 5-digits/90days Uptime
0 extend XPT waits
1779946655 pending I/O waits  IOs not completed yet when the requesting process returns to CPU
20598490595 start I/Os  Generally the sum of the above page ins and page outs
2333693437 iodones  Generally the start I/Os coalesced into fewer merged-data IO operations
8560316549 cpu context switches  Count of threads (of processes) switching on/off SMT logical CPUs
1900589120 device interrupts  Count of time-sensitive hardware device operations needing a logical CPU
70007091 software interrupts  Count of time-sensitive software interrupts needing a logical CPU
1230095024 decrementer interrupts  Tick-tock timing interrupts of only active logical CPUs (~10ms each)
7598754 mpc-sent interrupts
7598754 mpc-received interrupts
676750577 phantom interrupts  Device interrupts from other LPARs that pop-up here, thus “phantom”
0 traps
32508428126 syscalls  The workload of all processing boils-down to executing system calls for services

© Copyright IBM Corporation 2016. Technical University/Symposia materials 58

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 More about AIX:vmstat –v AIX logical memory pools

AIX:vmstat –v
25165824 memory pages
19112512 lruable pages
9201 free pages  This is the number of Free Pages on the Free Memory list
4 memory pools  The count of AIX logical memory pools
8028287 pinned pages  Generally AIX is comprised of only pinned memory pages
80.0 maxpin percentage
3.0 minperm percentage  This value is the trigger for pagingspace-pageouts
90.0 maxperm percentage
75.8 numperm percentage  This is the percent of JFS/JFS2/NFS/VxFS File Cache
14492435 file pages  This is the number of JFS/JFS2/NFS/VxFS File Cache pages
0.0 compressed percentage
0 compressed pages
75.8 numclient percentage  This is the percent of JFS2-only File Cache
90.0 maxclient percentage
14492435 client pages  This is the number of JFS2 File Cache pages
0 remote pageouts scheduled
857 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
1972 filesystem I/Os blocked with no fsbuf
9900 client filesystem I/Os blocked with no fsbuf
209695 external pager filesystem I/Os blocked with no fsbuf
42.4 percentage of memory used for computational pages  aka COMP%

© Copyright IBM Corporation 2016. Technical University/Symposia materials 59

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 More about AIX:vmstat –v AIX IO buffer exhaustion

AIX:vmstat –v
25165824 memory pages
19112512 lruable pages
9201 free pages  This is the number of Free Pages on the Free Memory list
4 memory pools  The count of AIX logical memory pools
8028287 pinned pages  Generally AIX is comprised of only pinned memory pages
80.0 maxpin percentage
3.0 minperm percentage  This value is the trigger for pagingspace-pageouts
90.0 maxperm percentage
75.8 numperm percentage  This is the percent of JFS/JFS2/NFS/VxFS File Cache
14492435 file pages  This is the number of JFS/JFS2/NFS/VxFS File Cache pages
0.0 compressed percentage
0 compressed pages
75.8 numclient percentage  This is the percent of JFS2-only File Cache
90.0 maxclient percentage
14492435 client pages  This is the number of JFS2 File Cache pages
0 remote pageouts scheduled
857 pending disk I/Os blocked with no pbuf  AIX pbuf exhaustion
0 paging space I/Os blocked with no psbuf  AIX psbuf exhaustion
1972 filesystem I/Os blocked with no fsbuf  AIX fsbuf exhaustion
9900 client filesystem I/Os blocked with no fsbuf  AIX fsbuf exhaustion
209695 external pager filesystem I/Os blocked with no fsbuf  AIX fsbuf exhaustion
42.4 percentage of memory used for computational pages  aka COMP%

Pagingspace-pageouts are triggered when numperm% and/or numclient% is less than/

equal to minperm%. Generally, a high COMP% is what causes the File Cache to grow too small.
© Copyright IBM Corporation 2016. Technical University/Symposia materials 60
IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Summary of AIX:vmstat –v

AIX:vmstat –v
25165824 memory pages
19112512 lruable pages
9201 free pages  This is the number of Free Pages on the Free Memory list
4 memory pools  The count of AIX logical memory pools
8028287 pinned pages  Generally AIX is comprised of only pinned memory pages
80.0 maxpin percentage
3.0 minperm percentage  This value is the trigger for pagingspace-pageouts
90.0 maxperm percentage
75.8 numperm percentage  This is the percent of JFS/JFS2/NFS/VxFS File Cache
14492435 file pages  This is the number of JFS/JFS2/NFS/VxFS File Cache pages
0.0 compressed percentage
0 compressed pages
75.8 numclient percentage  This is the percent of JFS2-only File Cache
90.0 maxclient percentage
14492435 client pages  This is the number of JFS2 File Cache pages
0 remote pageouts scheduled
857 pending disk I/Os blocked with no pbuf  AIX pbuf exhaustion
0 paging space I/Os blocked with no psbuf  AIX psbuf exhaustion
1972 filesystem I/Os blocked with no fsbuf  AIX fsbuf exhaustion
9900 client filesystem I/Os blocked with no fsbuf  AIX fsbuf exhaustion
209695 external pager filesystem I/Os blocked with no fsbuf  AIX fsbuf exhaustion
42.4 percentage of memory used for computational pages  aka COMP%

© Copyright IBM Corporation 2016. Technical University/Symposia materials 61

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Trademarks and notes

IBM Corporation 2016

 IBM, the IBM logo and ibm.com are registered trademarks, and other company, product, or service names may be
trademarks or service marks of International Business Machines Corporation in the United States, other countries, or
both. A current list of IBM trademarks is available on the web at “Copyright and trademark information” at
www.ibm.com/legal/copytrade.shtml
 Other company, product, and service names may be trademarks or service marks of others.
 References in this publication to IBM products or services do not imply that IBM intends to make them available in all
countries in which IBM operates.
 IBM and IBM Credit LLC do not, nor intend to, offer or provide accounting, tax or legal advice to clients. Clients should
consult with their own financial, tax and legal advisors. Any tax or accounting treatment decisions made by or on behalf
of the client are the sole responsibility of the customer.
 IBM Global Financing offerings are provided through IBM Credit LLC in the United States, IBM Canada Ltd. in Canada,
and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients. Rates and
availability are based on a client’s credit rating, financing terms, offering type, equipment type and options, and may
vary by country. Some offerings are not available in certain countries. Other restrictions may apply. Rates and offerings
are subject to change, extension or withdrawal without notice.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 62

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Session Evaluations

YOUR OPINION MATTERS!

1 2 3 4
Submit four or more session
evaluations by 5:30pm Wednesday
to be eligible for drawings!
*Winners will be notified Thursday morning. Prizes must be picked up at
registration desk, during operating hours, by the conclusion of the event.

© Copyright IBM Corporation 2016. Technical University/Symposia materials 63

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.
 Continue growing your IBM skills

ibm.com/training
provides a comprehensive
portfolio of skills and career
accelerators that are designed
to meet all your training needs.

If you can’t find the training that is right for you with our
Global Training Providers, we can help.

Contact IBM Training at dpmc@us.ibm.com

Global Skills Initiative

© Copyright IBM Corporation 2016. Technical University/Symposia materials 64

IBM Systems Technical Events | ibm.com/training/events may not be reproduced in whole or in part without the prior written permission of
IBM.