Professional Documents
Culture Documents
www.cmgindia.org
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
What we refine
Introduction
Virtualization Basics
vSphere Platform Basics
Scoping
Approach
Data Collection and Analysis
IO Profiling
Optimizing
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
Introduction
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
Virtualization Basics
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
Virtualizing x86
Non-virtualized (native) system
OS is designed run on bare metal hardware
Assumption of full ownership with full privileges
Use the CPU privilege levels to restrict the access
Operating System code run with higher privilege
Application code has less privilege
Privilege escalations are either faulted or ignored
Challenges (Virtualization)
Hardware is shared among multiple VMs
Cannot let guest run with highest privilege
Virtual Machines need to be isolated and contained
Binary Translation
Combination of Direct Execution and Binary
Translation (BT)
Virtual Machine Monitor (VMM) tracks the execution
from guest
Just in time translation of privileged execution
Translate the code that can cause privilege
escalation
Use translation cache for reusable translations
Direct execution of user level code (Ring 3)
Impacts
Overhead of translation and cache
Translated code is not as optimal as original code
Impacts
Exit to VMM can be expensive
In guest updates (page table) costlier
to track
IN/OUT instruction and memory
mapped I/O have higher overhead
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
Memory Virtualization
Virtual Memory
Application see a contiguous fixed address space - Virtual
Address (VA)
OS maps Virtual Address to Physical Address (PA)
Uses Page Tables to store this VA to PA mapping.
(normal page size: 4K)
CPU walks the page table for any translations
Address in %CR3 = page table location
Uses TLB (Translation look-aside Buffers) to cache
the translations
App 1
0GB
0GB
4GB
Physical
In a Virtual Machine
Additional address translation is required
VA -> PA -> MA
VMM uses shadow page table to map Guest VA to MA
and replaces the guest page tables
CPU uses VA -> MA mapping (no two level walks)
Reduced cost at execution
App 2
4GB
App 1
0GB
App 2
4GB
0GB
4GB
VM - Physical
Machine
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
Memory Virtualization
Impact of using Shadow Page Tables
Tracing the guest page table writes are expensive
Cost of Propagating the changes to shadow page table
Page faults need to be intercepted
Hidden page faults (Missed in Shadow page table)
True page fault (Forward to guest)
Context switches to be monitored
CR3 updates need to be intercepted to replace the guest page
table with shadow page table
Resources (memory) for maintaining shadow page tables
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
Benefits
Impact
TLB fill is expensive and page walk is costlier
Using large pages will absorb some costs here (2MB v/s 4KB)
Smaller page tables fewer levels to traverse
TLB capacity increase thus lesser TLB miss
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
10
I/O Virtualization
Software based I/O virtualization
I/O devices are virtual
IN/OUT instructions are monitored and completed by VMM
Impact of traps will induce additional latency
Hardware Assistance
PCI pass through
A physical I/O device dedicated to guest
Depends on availability of hardware support (IO-MMU)
Guest has control over I/O device and transacts directly
SR-IOV
A single device can be shared across multiple guests
Multiple Virtual Functions appear as if they are physical devices
Needs hardware support and capable devices
Good for latency sensitive applications
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
11
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
12
vSphere ESXi
vSphere ESXi is a Type 1 hypervisor runs directly on the bare metal
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
13
CPU Scheduler
Designed around being fair (Fairness) with emphasis on
responsiveness and resource utilization
Share based algorithm with option to guarantee resources
Highly Scalable
Supports 128 vCPUs /4 TB RAM per Virtual machine
480 Logical CPUs on the host
1024 Virtual Machines/4096 vCPUs per host
1:32 over commitment ratio
15
Co Scheduling
SMP Virtual Machines (vSMP) pose unique challenge for optimized
resource usage
Guest expects all CPUs to be scheduled at the same time (Co-Scheduling)
But all CPUs might not have enough workload (waste??)
Optimization
Do not schedule siblings if they are idle
Provides an illusion of synchronous progress
Skew in execution can be dangerous (no forever run)
Keep a check on skew and keep it bounded
Strict Co-Scheduling
If the skew is beyond threshold stop all sibling CPUs and schedule them
together
Can cause CPU fragmentation
Relaxed Co-Scheduling
Decisions to stop or proceed done at each siblings
Not all siblings to be stopped (Only which far ahead should stop)
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
16
Load Balancing
Running worlds need to be load balanced across PCPUs to have
proper resource utilization and responsiveness
Worlds can be migrated across PCPUs (Pull or Push)
Not All migrations can be good (Cache/NUMA advantages at risk)
17
Memory Scheduler
Proportional share based allocation (as CPU)
Will reclaim the allocation if found idle
Over commitment friendly memory conservation algorithms
Transparent Page Sharing
Memory Ballooning
Memory Compression
Swap to SSD (Low Latency Swap)
Swap to File
18
VMkernel
Virtual Disk
Virtual Mode RDM, VMDK, Snapshot
Virtual SAN
Block Device
NFS
TCP/IP
Physical Infra
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
19
NIC Emulation
VMkernel
Virtual Switch
I/O Filters/Chain (Pre)
VLAN
Team
Mirror
Offloads
Physical Infra
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
20
Scoping
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
21
22
23
24
25
Physical Machine
Virtual Machine
26
27
Redefine
28
Required..
Good to have
Tools/Metrics
vCenter Data Collection, ESXTOP, vSCSI Stats, Net Stats, vRealize
Operations Manager
Guest OS Level Statistics
Perfmon, SAR, vmstat, iostat
Benchmarking Tools
IOMeter, Netperf, Uperf, Database Stress tools
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
29
Approach
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
30
Where to look
Application Level
App specific Performance data
Guest OS - CPU/Memory, I/O Statistics
Virtualization layer
Connectivity Layer
Peripheral Devices
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
31
Methodology
Test or
Monitor
Collect
Tune
Data
Analyze
Data
Validate
Data
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
32
Benchmarking
Real Program
Run the program and measure the performance with respect to
response/completion times
Micro Benchmark
Performance of a specific activity of a program is tested
Short run and involves no/small set of IO
Ex: Memory Allocation/De Allocation, Program Loop
Component Benchmark
Measure raw performance of components
CPU/ Memory, Network IO, Storage IO
Kernel Benchmarking
Targeted towards profiling kernel performance
Synthetic Benchmarks
Specifically written tests to mimic the operations from a program/platform
33
Data Collection
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
34
CPU Utilization
Used by Application
Privileged (System) Time V/s User Time
Load Averages or Processor Queue Length
Memory Utilization
Available (Free)
Pages/Sec (Swap I/O), Page Faults
I/O
Read/Writes, Throughput
Queue Lengths
Latency/Wait Times
Network Packets Received, Errors, Discarded, Resets
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
35
vSphere Layer
vCenter Real time/Historic performance charts
Start here as it provides historical data for comparison
Minimum Collection Interval 20 Seconds (Refresh: 1 min)
Enough if the problem window is longer (> ~3 minutes)
36
Analysis
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
37
CPU Resources
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
38
CPU Statistics
What to Look
39
CPU Usage
Used and Run
A high value for used suggests CPU bound workload
Compare with Demand
Will adding more CPU to VM help?
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
40
Co-Stop: All siblings vCPUs are not making progress at same rate
Stop the vCPU which is ahead till slower vCPUs can catchup
41
Memory Stats
42
43
VM Stats in UI
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
44
Storage Stats
Esxtop storage views
Adapter (d)
Disk Device (u)
VM (v)
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
45
Latency Why?
Majority of the Device Latencies are caused by external
infrastructure
Busy Storage Device
Load factor
Burst I/O resulting from Backup, OS/Antivirus updates,
Replication, Boot Storm
Noisy neighbor issues
46
Network Stats
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
47
IO Profiling
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
48
vSCSI Stats
Data collection tool which provides into I/O pattern
of a Virtual Machine
Can be used to profile the I/O requirements of an
Application
I/O Size (Length)
Queue Size (Otstanding I/O)
Read/Write Ratio
How Random (or How Sequential)
Inter-arrival rate (How demanding)
Latency (Read/Write)
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
49
Net Stats
Can be used to profile Network I/O
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
50
Optimization
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
51
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
52
Workload Placement
Planning should consider Average as well as Peak utilization
Consolidation Ratios
Make sure sufficient resources for CPU bound critical workloads
available
Higher consolidation ratios are acceptable for desktop/test workloads
53
I/O Optimization
Storage
Optimal path policies to be used (Vendor involvement might be
required)
Keep an eye on storage utilization and load
Pick right RAID level based on the usage pattern
Storage I/O Control can help to reduce the impact of busy conditions
Storage DRS can keep the load balanced
Network
Use a hardware with offload features (Savings on CPU time)
Segmentation, Checksum etc..
Jumbo frames have lesser CPU overhead
May not help if average size of I/O transactions are small (~1500)
Network I/O Control can help to prioritize the workload
BIOS/Drivers
Recommended to keep the driver and BIOS level up to date
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
54
Optimization
Use virtual machine settings to mark the sensitivity of the VM
Exclusive access to CPU resources
Limit the de-scheduling of VM
Use of coalescing and other optimization methods are limited
Use SR-IOV based pass-trough for further reduction in latency
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
55
Questions?
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
56
Thank You
Presented at CMG India 2nd Annual Conference 2015. Copyright VMware Inc, 2015
57
Appendix
Performance Monitoring Utilities: resxtop and esxtop
https://pubs.vmware.com/vsphere60/index.jsp#com.vmware.vsphere.monitoring.doc/GUID-A31249BFB5DC-455B-AFC7-7D0BBD6E37B6.html
vScsiStats
http://cormachogan.com/2013/07/10/getting-started-with-vscsistats/
NetStats
Command line tool
From a shell session run net-stats h for details
58