number 9

I ran my Glassfish zone
through the quickstart
and it bore up fairly well; it doesn’t piss memory, and autodeploy is reliable (2 improvements on my
last tomcat deployment).

Now I’ve got a database ,
I’m almost ready to deploy something useful, but first it’d be nice to have the appserver managed by the OS (rather than firing it up by hand).

Glassfish SMF integration
makes it easy to run a domain as an SMF service, so the first thing I need is a domain.

domain attraction

A glassfish domain is a group of ‘server instances’ (JVMs to you and me).
Each domain has its own logs, collection of webapps, admin accounts, etc.
(see the Administration Guide p36 onwards for a detailed overview).

I’m not clustering (yet), so my domain just has 1 JVM -
I’ll keep it under /domains :

goldfish # zfs create tank/delegated/goldfish/domains
goldfish # zfs set mountpoint=/domains tank/delegated/goldfish/domains

To create a domain you run asadmin create-domain (see ‘asadmin help’ for suprisingly good documentation)

   goldfish # /j2ee/bin/asadmin create-domain \
   --adminport 4848 --instanceport 80 \
   --profile cluster --user admin \
   --domaindir /domains rollerdisco
   Please enter the admin password> <choose a password>
   Please enter the admin password again> <confirm password>
   Please enter the master password [Enter to accept the default]:>
   Please enter the master password again [Enter to accept the default]:>
   On Unix platform, port numbers below 1024 may require special privileges.
   Domain being created with profile:cluster, as specified on command line or environment.
   Security Store used should be JKS
   goldfish #

manifestation

To automate domain startup, store the admin and ‘master’ (keystore) passwords in a file
which you can feed to asadmin tasks :

goldfish # cat > /domains/rollerdisco/.aspass
AS_ADMIN_PASSWORD=whatever-you-specified
AS_ADMIN_MASTERPASSWORD=only-need-this-if-you-went-for-non-defaultchangeit
^D

Now tell asadmin create-service to build an SMF manifest for your domain.

   goldfish #  /j2ee/bin/asadmin create-service --passwordfile/domains/rollerdisco/.aspass \
   > --type das --serviceproperties net_privaddr /domains/rollerdisco/
   java.lang.IllegalArgumentException: Present Platform, OS: SunOS version: 5.11 is not Solaris 10. This facility is not supported on platforms other than Solaris 10.
   goldfish #  # >_<

Damn it! Solaris Express shows up as ‘5.11’ in uname – and since glassfish is only checking for solaris 10, it refuses to build a service. We need to persuade it to run anyway.

Luckily, Angelo wrote a
DTrace script to tamper with the uname() syscall .
We just need the PID of the shell we’re running asadmin from.

goldfish # uname -a
SunOS goldfish 5.11 snv_61 i86pc i386 i86pc
goldfish # echo $$
8539

Then run Angelos D script up in the global zone
(which has DTrace privileges ) :

vera # ./unamespoofer.d 8539
Changing output of uname for pid 8539 and its descendants...

and now if we retry

   goldfish # uname -a
   SunOS goldfish 5.10 snv_61 i86pc i386 i86pc
   goldfish #
   goldfish #  /j2ee/bin/asadmin create-service --passwordfile /domains/rollerdisco/.aspass \
   > --type das --serviceproperties net_privaddr /domains/rollerdisco/
   The SMF Service was created successfully. Here are the details:
   Name of the service:application/SUNWappserver/rollerdisco
   Type of the service:Domain
   Configuration location of the service:/domains
   Manifest file location on the system:/var/svc/manifest/application/SUNWappserver/rollerdisco_domains/Domain-service-smf.xml.
   The service could be enabled using svcadm command.
   Command create-service executed successfully.
   goldfish # # \o/

You can cancel that D script now – glassfish itself doesn’t mind the OS version
(this bug -#1308 – should be fixed in glassfish b45).

One last gotcha: on rebooting, the service went into maintenance mode.
The logs mentioned it couldn’t find ’/j2ee/bin/asadmin’, but it was there when I checked.

/j2ee is a ZFS filesystem holding my glassfish install.
Turns out the glassfish manifest doesn’t list any dependencies, so
SMF tries to start it before the ZFS filesystems are brought up.

So edit the created manifest, and right below the line, add:

  <dependency
        name='network'
         grouping='require_all'
        restart_on='none'
        type='service'>
        <service_fmri value='svc:/milestone/network:default' />
  </dependency>

  <dependency
        name='filesystem-local'
        grouping='require_all'
        restart_on='none'
        type='service'>
       <service_fmri value='svc:/system/filesystem/local:default' />
  </dependency>

Then reimport it:

goldfish # svccfg import /var/svc/manifest/application/SUNWappserver/rollerdisco_domains/Domain-service-smf.xml

(filed as bug #2910 – the fix will be in b46).

Now we can start the service:

goldfish # svcadm enable rollerdisco

It’ll take a minute or two, but should come up OK.

priviligious fanatics

SMF also lets Glassfish run as root relatively safely (even if we weren’t in a zone).

A normal root shell can do pretty much anything it likes:

goldfish # ppriv -S $$
4139:   -bash
flags = 
        E: zone
        I: basic
        P: zone
        L: zone

Look at the glassfish service:

goldfish # ppriv -S $(pgrep java)
3081:   /j2ee/jdk/bin/java -client -XX:MaxPermSize=192m -Xmx512m -XX:NewRatio=
flags = PRIV_AWARE
        E: basic,net_privaddr
        I: basic,net_privaddr
        P: basic,net_privaddr
        L: zone

‘basic’ privilege lets you fork, exec, symlink, write files you own, etc. It’s the same rights
the ‘nobody’ user has¹.
By default, this is the privilege SMF grants glassfish:

goldfish # svccfg -s rollerdisco 'listprop start/privileges'
start/privileges  astring  basic,net_privaddr

Remember : On Solaris 10 (or higher ), it’s the privilege that matters, not the fact that you’re root (root just happens to have that privilege by default).

For example, net_privaddr lets you open ports < 1024 . If I hadn’t specifically
asked for it (the ’—serviceproperties’ option to create-service) glassfish wouldn’t be able
to run on port 80, even as root (which is why create-domain warned me about it).

¹ Admittedly, root probably owns more files than nobody on the average server.

I need a database to do anything useful with glassfish .
Here are my PostgreSQL install notes.

the install

I chose postgresql 8.2 as part of my SXCE install .
If you don’t have it already, you need to:

globalzone # cd /cdrom/Solaris_11/Product
globalzone # pkgadd -d . SUNWpostgr-82-client SUNWpostgr-82-contrib \
SUNWpostgr-82-docs SUNWpostgr-82-libs SUNWpostgr-82-server \
SUNWpostgr-82-server-data-root SUNWpostgr-82-tcl

I might as well run it in a zone (partly to keep things tidy in case I screw up).
With sparse zones, I only need to install packages in the global zone and all zones can use them (one set of packages to maintain == happy sysadmin).

I’ll use the zone cloning script
I mentioned the other day, and slap on some resource caps while I’m at it :

globalzone # /zones/bangoneout.sh elephantom 1.2.3.4/24
globalzone # zonecfg -z elephantom "set max-lwps=300; add capped-memory; set physical=400M; set swap=512M; end; exit;"

ZFS snapshots make backing up the DB a lot easier, so I’ll give the zone a chunk of my zpool to manage:

globalzone # zfs create tank/delegated/elephantom
globalzone # zfs set mountpoint=none tank/delegated/elephantom
globalzone # zfs set quota=5G tank/delegated/elephantom
globalzone # zonecfg -z elephantom 'add dataset; set name=tank/delegated/elephantom; end'

From here on, we treat the zone as we would any other server:

globalzone # zoneadm -z elephantom reboot
globalzone # zlogin elephantom
[Connected to zone 'elephantom' pts/2]
elephantom #

creating the database

PostgreSQL integrates nicely with Solaris -
there’s RBAC support (a ‘PostgreSQL administration’ profile for DBA tasks),
DTrace providers and SMF integration in recent SXCE builds:

elephantom # svcs postgresql
disabled       12:21:40 svc:/application/database/postgresql:version_81
disabled       12:21:40 svc:/application/database/postgresql:version_82

I’ll make a ZFS filesystem and tell the version_82 instance to use it:

elephantom # zfs create tank/delegated/elephantom/data
elephantom # zfs set mountpoint=/data tank/delegated/elephantom/data
elephantom # chown postgres:postgres /data
elephantom # svccfg -s postgresql:version_82 'setprop postgresql/data = /data'
elephantom # svcadm refresh version_82

The rest of the install is the same as any UNIX.

Install the database as usual:

elephantom # su - postgres
$ /usr/postgres/8.2/bin/initdb /data
....snip usual initdb messages.....
$ exit
elephantom #

It’s probably a good idea to take a snapshot now, before we tweak stuff.
Note we can do this from within the zone since we :

elephantom # zfs snapshot vera/delegated/ganesh/data@pristine

The default config ( /data/postgresql.conf ) needs a few tweaks. I set:

   wal_sync_method = fsync
   full_page_writes = off
   listen_addresses = '*'
   # logfle is /data/server.log
   log_connections = on
   log_disconnections = on
   log_hostname = on

and edited /data/pg_hba.conf
to allow access from my glassfish zone
(all inter-zone traffic goes over loopback, so there’s no need to change your firewall).

Now start the server via SMF:

elephantom # svcadm enable postgresql:version_82

and create the user and the db:

elephantom # su - postgres
$ PATH=/usr/postgres/8.2/bin:$PATH
$ createuser -PREDS dbuser
Enter password for new role:
Enter it again:
CREATE ROLE
$ createdb -O dbuser zonedb
CREATE DATABASE
$ exit

Finally, I’ll check I can login from the glassfish zone:

glassfishzone # /usr/postgres/8.2/bin/psql -h elephantom.mydomain -U dbuser zonedb
Password for user dbuser:
Welcome to psql 8.2.3, the PostgreSQL interactive terminal.
Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit
zonedb=> \q
glassfishzone #

I stuck a 200Mb memory cap on the zone
- the above config seems quite happy in there.

Just a quick ‘note to self’.

I noticed my NIC ( iprb0 ) is running in half duplex mode (autoneg doesn’t work with our switches):

vera # kstat iprb:0|grep duplex
   duplex                          half

A google finds very out of date information. I finally found the answer in the manpage, of all places

man iprb(4) says you need to edit /kernel/drv/iprb.conf

and set ForceSpeedDuplex to ‘4’ (for 100Mbit full dup):

vera # echo 'ForceSpeedDuplex=4;' > /kernel/drv/iprb.conf

(if you have more than 1 iprb NIC, you’ll have to set them all – see the manpage for details)

Then do a reconfigure reboot:

vera # reboot -- -r

after the reboot:

vera # kstat iprb:0|grep duplex
   duplex                          full

Now that
I finally got off my ass and upgraded to a recent Solaris Express
I can have a proper look at Duckhorn.

Zones and resource management (RM) were made for each other
, but it could be a bit of an involved process. Project Duckhorn
set out to properly integrate zones and RM.

hold still

I’ll use my glassfish zone, ‘goldfish’
as a guinea pig. Fire up zonecfg in the global zone:

vera # zonecfg -z goldfish
zonecfg:goldfish> set cpu-shares=20
zonecfg:goldfish> set max-lwps=200
zonecfg:goldfish> add capped-memory
zonecfg:goldfish:capped-memory> set physical=500m
zonecfg:goldfish:capped-memory> set swap=750m
zonecfg:goldfish:capped-memory> end
zonecfg:goldfish> exit
vera #

LWPs = ‘lightweight processes’, or threads. CPU shares are used to determine how much CPU the zone gets
(see below). We also set some limits on RAM and swap.

do what I mean, not what I say

So, reboot the zone to apply these limits.

vera # zoneadm -z goldfish reboot
zoneadm: zone 'goldfish': enabling system/rcap service failed: entity not found
vera #

Ah, I forgot to install rcapd (the daemon which enforces resource limits).

vera # mount /cdrom && cd /cdrom/Solaris_11/Product/
vera # yes | pkgadd -d . SUNWrcapr SUNWrcapu
[ output snipped ]

Try again:

vera # zoneadm -z goldfish boot
zoneadm: zone 'goldfish': WARNING: The zone.cpu-shares rctl is set but
zoneadm: zone 'goldfish': FSS is not the default scheduling class for
zoneadm: zone 'goldfish': this zone.  FSS will be used for processes
zoneadm: zone 'goldfish': in the zone but to get the full benefit of FSS,
zoneadm: zone 'goldfish': it should be the default scheduling class.
zoneadm: zone 'goldfish': See dispadmin(1M) for more details.
vera #

So, the zone boots, but you’re warned that you’ve forgotten something.

CPU shares are used by the ‘fair share scheduler’ (FSS) which is another of Solaris’ really nice features.

FSS : burstable cpu quotas

Full details are in FSS
but briefly:

FSS dishes out resources amongst active zones based on their relative number of CPU shares.

i.e. If ‘goldfish’ is the only zone asking for CPU, it gets all the CPU.

Only when there’s contention
(e.g. goldfish goes bananas and I SSH into vera to reboot it) does FSS look at the shares.

The global zone gets 1 share by default, so the calculation is:

• goldfish (20 shares) gets ( 20 / (20 +1) = ) about 95% CPU
• vera (1 share) gets ( 1 / (20 + 1) = ) about 5% CPU

which will hopefully be enough to kill the runaway process or halt the goldfish zone.

If you need hard maximums and minimums (for licensing purposes)
look at CPU caps .
Both allocation strategies work on fractions of CPUs (although zoneadm will transparently manage processor sets
and resource pools if you prefer to use those).

tick tock

To make FSS the default scheduler:

vera # dispadmin -d
dispadmin: Default scheduling class is not set
vera # dispadmin -d FSS
vera # dispadmin -d
FSS     (Fair Share)
vera # reboot

the brown noise

We now have some reasonable protection against a runaway zone taking the global zone
with it.

To test this I’m using a shell-based forkbomb –
the following 13 characters are poison to any unix box you paste them into.

:(){ :|:& };:

Login to vera and run ‘prstat -Z’ to watch the zones (global and goldfish).
Then forkbomb ‘goldfish’:

goldfish # :(){ :|:& };:
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
...
...

As expected, we’re hitting the max-lwps limit (200 LWPs). There’s no noticeable impact on vera.

If we remove that safety net:

vera # zonecfg -z goldfish
zonecfg:goldfish> clear max-lwps
zonecfg:goldfish> commit
zonecfg:goldfish> exit
vera # zonecfg -z goldfish reboot

And forkbomb again:

goldfish # :(){ :|:& };:
-bash: fork: Not enough space
-bash: fork: Not enough space
-bash: fork: Not enough space
...
...

This time, we hit the ‘capped-memory’ limit.

The load is tremendous – as high as 350 (!) – but
thanks to FSS (and that 1 CPU share) we can still run ‘zoneadm -z goldfish halt’.

doctor, it hurts when I do this

As a control, I’ll remove the memory capping (but keep the CPU shares).
This is pretty stupid, as the box will thrash itself to death.

vera # zonecfg -z goldfish
zonecfg:goldfish> remove capped-memory
zonecfg:goldfish> exit
vera # zoneadm -z goldfish reboot

goldfish # :(){ :|:& };:

Sure enough vera and goldfish lock up completely.

The problem is that without a memory cap, the forkbomb will exhaust system swap.
Even though the global zone still gets a share of the CPU, it can’t start any processes
so it’s impossible to run any administrative commands.

So, Don’t Do That. Always cap zone memory (physical and swap) to reserve some for the global zone.

project management

You probably gathered
I’m a great fan of zones. Easy to use RM makes them really powerful.

But RM isn’t limited to zones. Resource controls, CPU shares, etc. can also be applied
to ‘projects’ – user-defined lists of processes within a single zone.
You could define projects in goldfish (one
for the appserver, one for something else in there) to further divvy up the zones allocation
of resources.

There’s an awful lot of goodness linked to at
The Opensolaris Zones Community page .

My favourite two are The Sun BluePrints Guide to Solaris Containers ,
and Mennos blueprint which has some good practical stuff on Oracle
projects.

I’ve been putting off reinstalling Solaris Express for a while,
but this week I finally went to b61 ( ships with crossbow , duckhorn , cpu caps, direct boot, a pony, wireless drivers, a load of iSCSI/ZFS fixes)

My last solaris install
was a ‘kick the tyres’ thing; now I’m using it for important things
(like my home directory ) I want something I can upgrade.

I’m going for a Live Upgrade
capable setup
(Briefly, it’s an alternate root partition you upgrade and then boot into. I’ll go into details again, but for now it boils down to ‘leave a free partition on one of the disks’).

The machine is a general purpose server, with a ZFS storage pool for building
iSCSI and
zones on.

Anyway, here are the install notes. Hopefully of some use to others.

hardware

It’s the same P4 destkop as before, but I scrounged some addons:

• 3Ghz P4
• 1 Gb Ram
• 2×40Gb IDE disks
• 1 x PCI 100Mbit NIC
• 1 X PCI-X Gbit NIC (the back half hangs out of the motherboard PCI slot. Looks stupid, but works.)

software

Get the latest SXCR DVD ISO
(there are CDROMS images if you insist). You need to register (free) with Sun Online to get the file(s).

• accept the license
• download the 5 zipped parts of the DVD
  • gotcha: get the right files! x86 is below SPARC.
  • gotcha: Firefox is rubbish at downloading big files. Use something else – konqueror works fine.
• unzip them, then cat them together to make the DVD image
• check md5sum of ‘big.iso’, test it boots using qemu, etc.
• burn it

pick an installer, any installer

I don’t see the point of the GUI install -
if you want one, see Dennis Clarkes walkthrough . You’ll need at least 768Mb of RAM.

To get a text installer:

• choose ‘Solaris Express’
• choose ‘4. Solaris Interactive Text (Console session)’
• let it (try to) DHCP its interfaces
• ‘Setting up Java. Please wait…’ : wait
• ‘Beginning system identification’:

NB the text installer doesn’t give you the option to install the compilers, so
install them from the DVD later.

sysidconfig

This should be familiar by now
but:

• configure keyboard layout : choose ‘uk-english’
• Select a Language: choose ‘1. English’
• hit F2 to get through to the ‘Network Connectivity’ screens
  • Networked: Yes
  • configure ‘iprb0’ (my WAN link)
    • DHCP: yes
    • IPv6: no
  • Kerberos Security: no
  • Name service: None (will be set via DHCP)
  • Default NFSv4 domain
• Time Zone: Europe/Britain(UK)
• check/edit clock time
• set root password

pre-install chitchat

• choose a ‘Standard’ install (as opposed to upgrade or flash archive)
• choose ‘Manually eject CD/DVD’ and ‘Manual Reboot’
• choose to install from CD/DVD
• accept license
• extra Geographic regions : none
• Sytem Locale : default (POSIX C)
• Web Start location : none

tweaking the install

lazy option If this section seems like a pain in the arse, just choose to install everything and skip to the next bit.

I only have 2×40Gb and I want a decent sized zpool, so I want to keep
the root FS reasonably small. I have no need for X so I’m going to choose
‘Core’ (979Mb) and hit F4 to customize it.

Make sure the below have [X] next to them.
Occassionally one part of a component has an insane amount of sub-dependencies,
(dynamic resource pools for example) so I’ve deselected that part.

Solaris packaging is primitive. It can highlight dependencies, but
it expects you to add those dependencies (and their dependencies, ad infinitum).
Choose ‘resolve dependencies after selecting all packages’ or you will be here all fucking day.

My shopping list:

• ‘Documentation Tools’ ( the ‘man’ command)
• ‘Fair Share Scheduler’ (zone resource controls)
• ‘Freeware Other Utilities’ (less)
• ‘Freeware shells’ (bash + zsh)
• ‘GNU wget’
• ‘Live Upgrade Software’
• ‘On-line Manual Pages’
• ‘PostgreSQL 8.2’ (all but ‘JDBC 3 driver’)
• ‘Secure Shell’
• ‘Solaris Zones’
• ‘System Accounting’
• ‘System and Network Administration Framework’ (needed by ‘Live Upgrade’)
• ‘The XSLT library’ (needed by ‘PostgreSQL 8.2’
• ‘core software for resource pools’ (all but ‘Dynamic Resource Pools’)
• ‘gcmn – Common GNU package’ – (needed by ‘GNU wget’)
• ‘gtar – GNU tar’ (to untar CentOS image for linux branded zones)
• ‘iSCSI Target’
• ‘lx Brand’ (for linux branded zones)

I come out of this with a total size of ‘1096Mb’. Hit F2 to continue.

disks

I’m going to make both disks the same size and leave space on the second disk for
a Live Upgrade ABE.

• ‘select disks’ – choose both
• preserve data – no
• choose ‘manual layout’. Select ‘overlap’ and customize it

c0d0

Slice	Mount Point	Slice (MB)
0	/	3129
1	swap	1027
3	/zfs	35000

c1d0

Slice	Mount Point	Slice (MB)
0	/alt	3129
1	swap	1027
3	/zfs2	35000

/zfs and /zfs2 will be the halves of my zpool mirror. /alt is the ABE.
None of the 3 are mounted (or should even be formatted) -
I’m just reserving the space here to save time later.

make install go fast now

• hit F2 to confirm the summary (or go back if you typoed)
• F2 past ‘mount remote file server’
• check the summary and hit OK to kick off the install
  • hit Ok if it warns you about boot device – it’s just saying ‘your BIOS is set to boot off CD, but you told me you wanted to boot off your hard disk’.

I hit ok and packed up to go home – it was done by the time I was ready to leave.

first boot

First, set the BIOS to boot off the hard disks. The first boot will take a minute or two,
mainly because Solaris is building its SMF database. It’s about 30 seconds after that.

tweaks

setup firewall

vi /etc/ipf/ipf.conf
svcadm enable ipfilter
ipfstat -io # to ensure it's loaded ok

set up root how I like it

usermod -s /usr/bin/bash root
mkdir /root && chmod 700 /root
vi /etc/passwd # change roots homedir to /root

setup sendmail

vi /etc/mail/sendmail.cf # set 'DSmysmarthost.com'
echo 'root:  me@mydomain.com' >> /etc/mail/aliases
newaliases && svcadm restart sendmail

fix manpage indexes

catman -w

setup a CDROM

   mkdir /cdrom
   echo "/dev/dsk/c1t1d0p0  -  /cdrom  hsfs  -  no  -" >> /etc/vfstab

setup root access via SSH using RSA keys

vera # vi /etc/ssh/sshd_config # set 'PermitRootLogin yes'
vera # svcadm restart ssh
me@desktop $ sftp root@solaris
mkdir .ssh
chmod 700 .ssh
put .ssh/id_dsa.pub /root/.ssh/authorized_keys
exit
me@desktop $ ssh root@vera # this should use the key now
vera # vi /etc/ssh/sshd_config # set 'PermitRootLogin without-password'
vera # svcadm restart ssh

cleanout ‘markers’ from vfstab

vera # umount /zfs
vera # umount /zfs2
vera # umount /alt
vera # rm -r /zfs /zfs2 /alt
vera # vi /etc/vfstab # remove the 3 entries

setup the zpool

vera # zpool create -f tank mirror c0d0s3 c1d0s3
vera # zfs set mountpoint=none tank

(’zpool’ needs reassurance (’-f’) these are the right partitions because the installer built filesystems on them)