SpringOne 2GX 2010 Day 2:Virtualizing Your Java Applications

Virtualizing Your Java Applications: Best Practices (Justin Murray)

Emerging Technology Track 8:30 AM – 10:00 AM

From the conference materials:

In this session, you will learn the key things to consider when you take your application to a virtualization platform, in particular to VMware ESX. The choice of configured memory sizes, virtual CPUs, timekeeping strategies and general virtual machine monitoring will be explored. Best practices will be detailed in each section. An example performance test project will also be shown.

Justin Murray holds a technical alliances role at VMware, where he has worked for three years. His main work at VMware is in helping ISV partners to test, prove and optimize performance of their products on the VMware vSphere platform. Justin also helps VMware's customers with virtualizing their business applications and infrastructure and advises them on best practice approaches to virtualizing Java-based applications in particular. Previous to joining VMware, Justin worked at HP on management software and on Java on HP-UX.

From my personal notes:

• Use of VMware ESX (not VMware ESXi or VMware Server) is the focus during this presentation.
• Technically, no changes are needed when moving Java to ESX.
• This session shares insights based on real world customer scenarios by Justin Murray from VMware.
• There are only 7 best practices here, all of which are well documented in the VMware white paper on this subject, although not everything presented in this session exists in the white paper.
• Only about 25% of attendees (by raise of hands) appear to have already started virtualizing Java applications, and a small portion are currently considering doing so.
• The Hypervisor gets out of your way most of the time, passing down Java instructions directly to the JVM most of the time.
• If you follow these best practices, getting within 5% to 10% of the performance achieved without VMware is obtainable.
• There is a large wave of applications currently being migrated to VMware in the industry.
• The classic use case of virtualization is server consolidation.
• The ratio of physical to virtual servers is typically 5, although this number can be much higher.
• About 90% of servers typically run at about 14% or lower of CPU capability.
• Running multiple Java instances within a virtual machine is a better way to use the additional power available while at the same time providing isolation across applications.
• It is much easier to isolate performance problems when using virtual machines.
• Isolation at the OS and JVM layers enables greater flexibility for patch management.
• Rapid provisioning can be performed by storing a "golden" virtual machine in a library and copying for other instances.
• VMware HA (High Availability), DRS (Distributed Resource Scheduler), FT (Fault Tolerance).
• It is recommended that only 1 to 3 JVMs be running in each virtual machine, although Murray has seen customers run dozens per virtual machine.
• Java is probably the easiest platform to move from physical to virtual.
• The Hypervisor is essentially a customized OS that specializes in hosting virtual machines.
• There are 3 layers of memory that support Java: physical server memory, virtual machine memory, and JVM memory.
• Do not overcommit your ESX server.
• The ESX Hypervisor has no idea that memory references are associated with Java.
• JVM garbage collection acts the same way when running on a virtual server.
• Transparent page sharing enables memory overcommitting, but Murray advises to stay out of that scenario when using Java.
• Memory management between guest physical memory and machine memory is now handled by the latest hardware.
• Use the "very, very latest" hardware for Java systems – right now the Intel 5500 and the AMD Shanghai – typical results demonstrate 25% performance increase.
• Extended page tables (EPT) are available with both the Intel 5500 and AMD Shanghai.
• The motivation of ESX is to share as much as possible.
• Switch off the page sharing algorithm between virtual machines.
• Ballooning enables emergency retrieval of memory, but VMware has no idea that it is affecting the JVM.
• ESX has a SWAP file for each virtual machine, but we do not want swapping to occur because of resultant performance degradation.
• The following features need to be turned off: page sharing, ballooning, and swapping.
• How big should the virtual machine be? The virtual machine needs to be sized to leave adequate space for the Java heap and other memory demands of the JVM code, for any concurrently executing process that needs memory from the same guest OS to prevent swapping in the guest OS.
• The memory "Reservation" value in the vCenter client needs to be set to the size of the active memory for the virtual machine, but a problem is that people often do not trust themselves when setting this value, so use the configured memory if you are in any doubt about "active memory".
• Using large memory pages provides up to a 20% performance boost.
• Page sharing between virtual machines is based on content – it is possible that using the same guest OS for multiple virtual machines will increase this capability.
• A single Java thread executes on a vCPU at any given time.
• What is the right number of vCPUs to assign to a given Java process? Perhaps 1 vCPU is all that is needed. It is possible that Java software being investigated is not as multithreaded as one might think. The majority of Java threads should not be in the running state, but in the waiting state or the ready to run state.
• It is not possible to look at the static threads and determine how many vCPUs should be used.
• Generally speaking, it is alright to have more vCPUs than physical CPUs, provided that they are not running at 100% all of the time.
• It is feasible to have overcommitted vCPUs, not overcommitted physical CPUs.
• Run load testing with varying amounts of vCPUs to determine the effect on a Java application – start with the lowest number of vCPUs that is practical.
• More vCPUs is not always better.
• Set limits on real CPU power when maximizing utilization of ESX host with multiple vCPU virtual machines.
• Use vCenter and "esxtop" command. The level of granularity is finer with esxtop – about every 3 seconds as opposed to every 10 to 15 seconds using vCenter.
• Be careful with the amount of I/O logging.
• The "esxtop" command is modeled after the traditional Linux "top" command.
• Use "esxtop" with "netstat -a" to determine any network related bottlenecks.
• Use "esxtop -n".
• Synchronize between ESX host and virtual guest OS with external NTP source for Linux or w32time for Windows.
• Lower the clock interrupt rate on Linux machines by using a more modern kernel if possible.
• Use as few vCPUs as are practical for your application.
• Kernels for best timekeeping: for ESX 3.5 use SLES 10 SP2 (use VMI for 32 bit VMs), RHEL 5.4. (Note: missed recommendations for ESX 4.0.)
• Use your actual application code in sizing exercises rather than representative code or subsets of what you think represents your code.
• For easier monitoring and load balancing, use one JVM or a small number of JVMs per virtual machine.
• Use "esxtop" output data to see where resource consumption occurs in your Java application.
• What you lose when having more than one JVM per virtual machine is that when you use vMotion you will not know exactly which JVMs are being moved.
• White paper will be updated again in the next few months.
• A tool called "Capacity Planner" can be run against virtual machines to gather statistics to determine how many can be virtualized without even thinking.
• Capacity Planner monitors context switches per second, disk I/O, memory consumption, and other concurrent process besides Java consumption.
• Use the latest versions of ESX/vSphere and hardware with virtualization assist if possible.
• Latest ESX 4.1 combined with latest hardware can provide as little as 0% overhead – the latest can handle 200k IOPS greater than 95% of Oracle databases in the world.
• In some cases, virtualized platforms have even greater performance than physical counterparts.
• Performance brief white paper.