number 9

glassfish v2
comes with JDK 1.6,
which has
DTrace providers built into Hotspot
that let you monitor your JVM.

backstory

• ‘goldfish’ is a local zone (’virtual’ Solaris instance).
• ‘vera’ is the global zone (the ‘main’ Solaris instance).

The server runs Solaris Express .
Glassfish runs on a JVM in the ‘goldfish’ zone.

I want to DTrace that JVM.

By default, local zones don’t have enough privilege to run dtrace (as it lets you peek into the kernel).
I was about to change that when I realised something.

zones aren’t VMs

I tend to assume that’s obvious, until I talk to non-Solaris users.

• Solaris zones (unlike VMware guests or Xen domains) don’t run in their own virtual machine
  • zones all share the same kernel (and a lot of the OS)
  • they suffer practically no performance overhead
  • you can give a zone its own filesystems/NICs/schedulers/resource controls or re-use the global zones resources.
  • this gives them very low memory/storage overhead
• zones can’t run Windows
  • I don’t see that as a problem
• you can run linux (of sorts) in a zone
  • ask yourself why you want to
• for storage, you can create one zpool in the global zone to hold all your zones
  • this avoids the massive wastage of : 4 VMs on a host, each with 7 filesystems, each 30% full (which is how most VMware installs seem to run).
  • you can snapshot everything in the global zone, or delegate to each zone. you can change your mind easily too.
• the zone is a natural place to apply resource controls
  • BUT this is possible without zones
  • processes sharing a zone can be further divided into projects if you need fine grained control
• the global zone can see (and access) all processes in all zones
  • processes running in the zone only see other processes in that zone

This last point is a huge benefit of zones which I think a lot of people overlook, or mistake for a negative. It also makes my job today much easier.

VMware hosts or Xen dom0s monitor at the VM/domain level:
xen has xentop, vmware has ‘VMware tools’( ESX has a bloated GUI I am trying to drink away).

You can start a console in a vm (like ‘zlogin’ on Solaris) but you might as well SSH into them. It’s no easier to keep track of your processes than if they were on remote machines.

In a global Solaris zone, the processes are all there alongside you. Just use your usual monitoring tools -
ps, prstat, etc. are all zone-aware. Accounting is a doddle too.

back to the point

If you don’t treat zones as VMs, there’s a much simpler way to do this.
Instead of granting ‘goldfish’ dtrace privileges I can simply monitor the JVM from ‘vera’.

i like to watch

This 1-liner (from the DTrace wiki)
fires at every JVM method call, printing the classname of the called instance and the called method:

   vera # dtrace -qn 'hotspot*:::method-entry { printf("-> %4s.%s\n", stringof(copyin(arg1, arg2)), stringof(copyin(arg3, arg4))); }'
   dtrace: buffer size lowered to 2m

NB: this is running as root in the global zone, which has more than enough privilege for dtrace.

Trouble is, I got no output.
Turns out the ’method-entry’ Hotspot probe is disabled by default for performance reasons.
To enable the method-entry probe, you pass the ’-XX:+ExtendedDTraceProbes’ flag when the JVM starts.

the doctor will see you now

Having to bounce the JVM (or run it in a ‘debug’ mode by default on a production box) would suck.
Luckily, the JDK comes with a tool called jinfo – it lets you read system properties, command line flags etc. from a running JVM.
What the manpage doesn’t say is that since JDK 1.6 jinfo can also set those properties on the running JVM.

On other OSes, we’d have to drop into the VM as root (or do some remote JVM voodoo). On Solaris, we can see all processes in all zones from vera
(actually we could do this bit from the zone, since it doesn’t need special privileges).

vera # jinfo -flag +ExtendedDTraceProbes $(pgrep -z goldfish java)

’pgrep -z goldfish foo’ returns the PID of all processes called foo in the ‘goldfish’ zone
(I’m only running 1 java process in the goldfish zone, so I know pgrep will find the right one). jinfo sets the flag on the PID returned by pgrep.

Immediately, the probe starts to fire
and the dtrace window starts spewing class and method names:

   vera # dtrace -qn 'hotspot*:::method-entry { printf("-> %4s.%s\n", stringof(copyin(arg1, arg2)), stringof(copyin(arg3, arg4))); }'
   dtrace: buffer size lowered to 2m
   -> java/util/HashMap/HashMap$HashIterator.newKeyIterator
   -> java/util/HashMap$KeyIterator.<init>
   -> java/util/HashMap$KeyIteratorhIterator.<init>
   -> java/util/HashMap$HashIteratorctorImpl.<init>
   -> java/lang/Objectenterprise/server/ss/provider/ASSelector.<init>rise/server/ss/provider/ASSelector
   -> java/util/HashMap$HashIteratorver/ss/spi/ASSocketFacadeUtils.hasNexti/ASSocketFacadeUtils
   -> java/nio/channels/spi/AbstractSelectorSSocketServiceProxy.beginServiceProxy
   -> java/nio/channels/spi/AbstractInterruptibleChannelce.blockedOn
   -> sun/misc/SharedSecretsrise/server/ss/ASSocketService.getJavaLangAccesscketService
   -> java/lang/Threadenterprise/server/PEMain.currentThreadrver/PEMain
   ...
   ...

There are other interesting probes there (that don’t fire several hundred times a second) – garbage collection,
method compilation, classloading.
See the full probe list for more.

When we get what we came for, we can let the JVM run at full speed again:

vera # jinfo -flag -ExtendedDTraceProbes $(pgrep -z goldfish java)

and dtrace shuts up.

to recap: VMware can kiss my ass

1. on a 1.6 JVM under Solaris we can switch a running JVM into profiling/debugging mode without needing to restart anything.
2. we can monitor processes (JVMS or otherwise) in all our ‘virtual machines’ using standard UNIX tools.
3. we can HUP/KILL/generally bugger about with said zoned processes without needing to ‘zlogin’ if the mood takes us (and we feel like driving the zone admin insane)
4. I am a decent packaging system away from being a raving Solaris zealot.