thinking cap

Now that
I finally got off my ass and upgraded to a recent Solaris Express
I can have a proper look at Duckhorn.

Zones and resource management (RM) were made for each other
, but it could be a bit of an involved process. Project Duckhorn
set out to properly integrate zones and RM.

hold still

I’ll use my glassfish zone, ‘goldfish’
as a guinea pig. Fire up zonecfg in the global zone:

vera # zonecfg -z goldfish
zonecfg:goldfish> set cpu-shares=20
zonecfg:goldfish> set max-lwps=200
zonecfg:goldfish> add capped-memory
zonecfg:goldfish:capped-memory> set physical=500m
zonecfg:goldfish:capped-memory> set swap=750m
zonecfg:goldfish:capped-memory> end
zonecfg:goldfish> exit
vera #

LWPs = ‘lightweight processes’, or threads. CPU shares are used to determine how much CPU the zone gets
(see below). We also set some limits on RAM and swap.

do what I mean, not what I say

So, reboot the zone to apply these limits.

vera # zoneadm -z goldfish reboot
zoneadm: zone 'goldfish': enabling system/rcap service failed: entity not found
vera #

Ah, I forgot to install rcapd (the daemon which enforces resource limits).

vera # mount /cdrom && cd /cdrom/Solaris_11/Product/
vera # yes | pkgadd -d . SUNWrcapr SUNWrcapu
[ output snipped ]

Try again:

vera # zoneadm -z goldfish boot
zoneadm: zone 'goldfish': WARNING: The zone.cpu-shares rctl is set but
zoneadm: zone 'goldfish': FSS is not the default scheduling class for
zoneadm: zone 'goldfish': this zone.  FSS will be used for processes
zoneadm: zone 'goldfish': in the zone but to get the full benefit of FSS,
zoneadm: zone 'goldfish': it should be the default scheduling class.
zoneadm: zone 'goldfish': See dispadmin(1M) for more details.
vera #

So, the zone boots, but you’re warned that you’ve forgotten something.

CPU shares are used by the ‘fair share scheduler’ (FSS) which is another of Solaris’ really nice features.

FSS : burstable cpu quotas

Full details are in FSS
but briefly:

FSS dishes out resources amongst active zones based on their relative number of CPU shares.

i.e. If ‘goldfish’ is the only zone asking for CPU, it gets all the CPU.

Only when there’s contention
(e.g. goldfish goes bananas and I SSH into vera to reboot it) does FSS look at the shares.

The global zone gets 1 share by default, so the calculation is:

• goldfish (20 shares) gets ( 20 / (20 +1) = ) about 95% CPU
• vera (1 share) gets ( 1 / (20 + 1) = ) about 5% CPU

which will hopefully be enough to kill the runaway process or halt the goldfish zone.

If you need hard maximums and minimums (for licensing purposes)
look at CPU caps .
Both allocation strategies work on fractions of CPUs (although zoneadm will transparently manage processor sets
and resource pools if you prefer to use those).

tick tock

To make FSS the default scheduler:

vera # dispadmin -d
dispadmin: Default scheduling class is not set
vera # dispadmin -d FSS
vera # dispadmin -d
FSS     (Fair Share)
vera # reboot

the brown noise

We now have some reasonable protection against a runaway zone taking the global zone
with it.

To test this I’m using a shell-based forkbomb –
the following 13 characters are poison to any unix box you paste them into.

  ){  :& };:

Login to vera and run ‘prstat -Z’ to watch the zones (global and goldfish).
Then forkbomb ‘goldfish’:

goldfish #  ){  :& };:
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
...
...

As expected, we’re hitting the max-lwps limit (200 LWPs). There’s no noticeable impact on vera.

If we remove that safety net:

vera # zonecfg -z goldfish
zonecfg:goldfish> clear max-lwps
zonecfg:goldfish> commit
zonecfg:goldfish> exit
vera # zonecfg -z goldfish reboot

And forkbomb again:

goldfish #  ){  :& };:
-bash: fork: Not enough space
-bash: fork: Not enough space
-bash: fork: Not enough space
...
...

This time, we hit the ‘capped-memory’ limit.

The load is tremendous – as high as 350 (!) – but
thanks to FSS (and that 1 CPU share) we can still run ‘zoneadm -z goldfish halt’.

doctor, it hurts when I do this

As a control, I’ll remove the memory capping (but keep the CPU shares).
This is pretty stupid, as the box will thrash itself to death.

vera # zonecfg -z goldfish
zonecfg:goldfish> remove capped-memory
zonecfg:goldfish> exit
vera # zoneadm -z goldfish reboot

goldfish #  ){  :& };:

Sure enough vera and goldfish lock up completely.

The problem is that without a memory cap, the forkbomb will exhaust system swap.
Even though the global zone still gets a share of the CPU, it can’t start any processes
so it’s impossible to run any administrative commands.

So, Don’t Do That. Always cap zone memory (physical and swap) to reserve some for the global zone.

project management

You probably gathered
I’m a great fan of zones. Easy to use RM makes them really powerful.

But RM isn’t limited to zones. Resource controls, CPU shares, etc. can also be applied
to ‘projects’ – user-defined lists of processes within a single zone.
You could define projects in goldfish (one
for the appserver, one for something else in there) to further divvy up the zones allocation
of resources.

There’s an awful lot of goodness linked to at
The Opensolaris Zones Community page .

My favourite two are The Sun BluePrints Guide to Solaris Containers ,
and Mennos blueprint which has some good practical stuff on Oracle
projects.