fast zone cloning on Solaris 10

Glassfish seems like a natural successor to Tomcat.
The clustering features look interesting, but I only have the one machine.

Hmm. I’m going to need a shitload of zones.

send in the clones

The ‘zoneadm clone’ command creates a zone by copying an existing zonepath (to avoid going through the install twice).
On Solaris Express, zones on ZFS can be cloned in about a second
Solaris 10 (update4) has to actually copy the files, so we’ll use a trick to avoid that.

the master plan

• build 1 ‘template’ zone on ZFS
• configure it to a ‘standard build’
• take a ZFS snapshot of the zonepath
• ZFS clone the snapshot N times to make N zonepaths
• run zonecfg and hook up each zonepath
• boot them
• ssh in and install whatever you like

build your template zone

We’ll quickly make a bog-standard
‘whole root’
zone .

This takes more disk (and longer to install) than a sparse zone,
but gives you maximum flexibility (you can write to /usr, etc.).

zfs create -o mountpoint=/zones vera/zones
zfs create -o compression=on vera/zones/template
zonecfg -z template "create -b; \
    set zonepath=/zones/template ;\
    commit ; exit"
chmod 700 /zones/template/
time zoneadm -z template install

As I said, that takes a while (a sparse zone installs in about 5 minutes):

real    21m30.749s
user    1m18.566s
sys     3m35.917s

Good job we only have to do it once.

tweak it like you mean it

You could clone the zonepath now (skip ahead to ’say cheese’), but
since I tend to setup my machines the same way, I’ll customize things first.

First thing to do is boot the zone, and complete the system identification.

zoneadm -z template boot
zlogin -C -e. template

The zlogin command means :

• get me a console (-C) login to do system setup
  • sysconfig runs on the zone console, so a straight zlogin isn’t enough
• type ’..’ (-e.) to be dropped back to the main zone
  • the default sequence is .#, which will kill your ssh session to the global zone

You’ll see a counter as the SMF database is generated on first boot
(which takes a few minutes. again, we only need to do this in the template)::

[Connected to zone 'template' console]
 37/138

Then go through the standard Solaris sysconfig
(doesn’t matter what you enter – this is overridden on a per-zone basis).

When that’s done, the zone will reboot itself (hit ’..’ to exit zrogin).

Now do your ‘standard build’. My list :

• change roots shell and prompt
• copy my public SSH keys so I can ssh in as root
• setup sendmail
• turn off some daemons

Since that’s what I did for my original solaris install
I can just copy files to do most of this.

zlogin template usermod -s /usr/bin/bash root
cp /.bash_profile /zones/template/root/
cp /etc/ssh/sshd_config /zones/template/root/etc/ssh/sshd_config
cp -Rp /.ssh/ /zones/template/root/.ssh/
cp /etc/mail/sendmail.cf /zones/template/root/etc/mail/sendmail.cf
cp /etc/mail/aliases /zones/template/root/etc/mail/aliases
cp /etc/mail/aliases.db /zones/template/root/etc/mail/aliases.db
for i in webconsole sendmail autofs
do
zlogin template svcadm disable $i
done

say cheese

     zlogin template
     # sys-unconfig # this also halts the 'template' zone
     zoneadm -z template detach
     zfs snapshot vera/zones/template@clean
     zoneadm -z template attach

(the last ‘attach’ command makes patching the zone slighty easier).

going around the houses

Now we can use that to create a new zonepath for our DB zone, ganesh:

zfs clone vera/zones/template@clean vera/zones/ganesh

Life is a LOT easier if you separate your OS from your data, so I also give the zone its own ZFS filesystem – what we call ‘delegating a dataset’ – to install
its apps etc on
(note that although the zonepath is on ZFS, the zone is not ‘aware’ of that, so you can’t create zfs filesystems on it).
This also lets zone admins run their own snapshots etc. (snapping from the global zone works too, so choose your preference)

zfs create -o mountpoint=none vera/delegated/ganesh
zfs set quota=5G vera/delegated/ganesh

zonecfg supports ‘create -a’ to attach a pre-built zoneroot and generate a
config for it. We also

• set it to boot at system startup (’autoboot’)
• add a network address (’add net’)
• apply some simple resource controls (’add cpu-shares/max-lwps/capped-memory’)

    zonecfg -z ganesh "create -a /zones/ganesh;set autoboot=true; \
    add net; set physical=iprb0; set address=10.1.0.1/24; end; \
    set cpu-shares=20; set max-lwps=400; \
    add capped-memory; set physical=400m; set swap=500m; end; \
    add dataset ; set name=vera/delegated/ganesh; end; \
    commit; exit"
    zoneadm -z ganesh attach

feed some prepared answers to sysconfig:

sed s/ZONENAME/ganesh/ \
/zones/scripts/sysidcfg.template > /zones/ganesh/root/etc/sysidcfg

and finally boot it

zoneadm -z ganesh boot

attack of the clones

That’s the database taken care of.
We now have 3 more to do, and this is pretty easy to script.
I threw something together to do the job for me.
It’s pretty stinky (I don’t really speak shell) but should be easy for you to roll your own
You’ll need the script and the template for sysidcfg

cd /zones/scripts
wget http://files.hellooperator.net/solaris/zones/s10/scripts/bang_one_out.s10u4.sh
wget http://files.hellooperator.net/solaris/zones/s10/scripts/sysidcfg.template

Now the payoff:

time for i in kingfish rippyfish turnipfish
 do
   /zones/scripts/bang_one_out.s10u4.sh $i
 done

real    0m14.409s
user    0m2.459s
sys     0m1.097s

zoneadm list -iv
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              native   shared
   6 ganesh           running    /zones/ganesh                  native   shared
  25 kingfish         running    /zones/kingfish                native   shared
  27 rippyfish        running    /zones/rippyfish               native   shared
  29 turnipfish       running    /zones/turnipfish              native   shared

did you see that?

That’s 15 SECONDS to do what took 20 minutes the first time. Except these zones are configured and booted ready to ssh into.

Oh, and there are 3 of them.

I use zone cloning like Jumpstart – a way to
get a known, reproducible base OS as a building blocks for other things.

You can clone zones whatever FS they’re on, but it will take
longer to copy files than to snapshot+clone (especially for whole root zones).

The great thing about ZFS snapshots and clones is that a clone only uses disk space for the changes from its parent snapshot. It’s not obvious at the filesystem level:

du -hs  /zones/template /zones/ganesh
 2.1G   /zones/template
 2.3G   /zones/ganesh

But you can see it in the dataset (the ‘USED’ field below):

zfs list  vera/zones/template vera/zones/ganesh
NAME                  USED  AVAIL  REFER  MOUNTPOINT
vera/zones/ganesh    35.1M  28.6G  2.11G    /zones/ganesh
vera/zones/template  2.13G  28.6G  2.10G  /zones/template

Finally, remember you can clone any zone.
A common
problem we have is our test and dev. systems getting out of step with our production
boxes. If they’re zones
(and they will be if I have a say in it), you can easily clone
the live box (and its database zone) to get a testbed for upgrades, config changes, etc. that is as close to reality as you can get.