Glassfishv2 and PostgreSQL (open source, free) now officially wizz all over
BEA, IBM and Oracle (proprietary, hideously expensive).
Feels good to have backed a winning horse. There’s another beta out now too that I’ll be upgrading
my existing install to when I get five minutes.
In the meantime, congratulations to the GF team.
I want my Roller install
to use LDAP authentication (instead of its own account database).
LDAP auth means cleartext passwords, so I need to run the site over SSL.
Each Glassfish domain has it’s own keystore, which is protected by what the docs call the
‘admin master password’ (not the same as the ‘admin password’).
The master password is just a Java keystore password,
so if you didn’t say otherwise at domain creation time
it defaults to ‘changeit’.
You can check by trying to list the entries in your keystore:
goldfish $ keytool -list -keystore \
/domains/rollerdisco/config/keystore.jks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
s1as, Apr 22, 2007, PrivateKeyEntry,
Certificate fingerprint (MD5): 80:08:13:50:00:00:80:08:13:50:00:00There’s just one entry – s1as – which is the SSL keypair for the domains admin webapp (om port 4848).
As of version 2, the Glassfish admin UI lets you choose keypairs from this keystore, but it can’t import into the keystore itself.
So you’ll have to break out the command line.
As you’ll see, I really hope they add that feature soon.
If you’re creating a new keypair, things are relatively sane – see
Ryans howto .
I’m unlucky enough to have an existing apache-style’server.key/server.crt’ PEM keypair I want to use.
This turns out to be a massive pain in the arse. You’d think ‘keytool -import’ would do the trick,
but that only lets you add the certificate – keytool assumes it created the inital private key so doesn’t
provide a way to import one.
Googling found some very old tools for doing this, most of which only work with older, PKCS12 based keystores.
Let me save you an afternoon. The Jetty project have a tool called
PKCSImport
which is by far the least shitty way of doing this.
First, merge the two files using everyones favourite crypto swiss army knife:
goldfish $ openssl pkcs12 -export -out roller.p12 \
-inkey roller.key -in roller.crtIt asks you for an export password. ‘foo’ works fine 
Then get the Jetty jarfile and run the tool against your PKCS keypair:
goldfish # /usr/sfw/bin/wget http://kent.dl.sourceforge.net/sourceforge/jetty/jetty-6.1.4rc0.zip
goldfish # unzip jetty-6.1.4rc0.zip
goldfish # /j2ee/jdk/bin/java \
-cp jetty-6.1.4rc0/lib/jetty-6.1.4rc0.jar \
org.mortbay.jetty.security.PKCS12Import \
roller.p12 /domains/rollerdisco/config/keystore.jks
Enter input keystore passphrase: foo
Enter output keystore passphrase:
Alias 0: 1
Adding key for alias 1 This imports your certificate and keypair and creates a new keystore entry.
Unfortunately the entry is called ’1’, which is hardly self-documenting, so
I’ll rename that to the domain name it’ll run on:
goldfish # /j2ee/jdk/bin/keytool -changealias \
-keystore /domains/rollerdisco/config/keystore.jks \
-alias 1 -destalias roller.yourdomain.comGlassfish already serves webapps over https, it’s just on a non-standard port of 8181
(the—domainproperties option to asadmin create-domain can specify this. Yay hindsight).
Go to the admin webapp, and browse to :
Configurations → server-config → HTTP Service → HTTP listeners
under there you’ll see 3 listeners
First change the http-listener-2 port from 8181 to 443, then click over to the SSL tab.
Enter the keystore alias name for the keypair you just imported in ‘Certificate Nickname:’ and tick the TLS and SSLv3 boxes.
Also select at least the ‘common ciphers’.
We’re now running the same webapp (roller) over both the http and https.
Finally, go back to http-listener-1 and set the SSL redirect port to 443 – this tells the webapp to redirect to SSL for
sensitive operations.
Alternatively, turn off the http listener (untick ‘Listener: enabled’) to ensure all traffic comes over https.
My (b33) glassfish v2 build
is a bit long in the tooth
(the latest promoted build is b50).
It bundles a JVM, but the nightly builds don’t, so my first pre-upgrade step is to install a standalone 1.6 JVM.
I’ll need a few zones for playing with glassfish clustering and they’ll all need JVMs. Ideally, we want a central copy in the global zone that is visible from all the local zones.
That way:
Originally, this ‘howto’ involved getting the tarball from the JEE page, installing it to a zfs filesystem and loopback mounting it into each zone read-only.
Today, I realized the Solaris packages for JDK 1.6 give us all the benefits of that with none of the hassle. They’ll install in the global zone and be visible (as part of /usr) in all local zones. I won’t even have to restart the zones.
Get JDK 6u1 from the Java SE download page
(the Solaris x86 packages – tar.Z on the ‘Download’ page) .
Extract it In the global zone and add the packages it contains:
vera # uncompress jdk-6u1-solaris-i586.tar.Z
vera # tar xf jdk-6u1-solaris-i586.tar
vera # yes | pkgadd -d . SUNW*It’s now visible in all the zones:
vera # zlogin goldfish
goldfish # java -version
java version "1.6.0_01"
Java(TM) SE Runtime Environment (build 1.6.0_01-b06)
Java HotSpot(TM) Client VM (build 1.6.0_01-b06, mixed mode, sharing)The only minor niggle is that the /usr/java symlink can’t be edited in the local zones, but it’s easy enough to set a PATH explicitly if you want a different JVM.
glassfish v2
comes with JDK 1.6,
which has
DTrace providers built into Hotspot
that let you monitor your JVM.
The server runs Solaris Express .
Glassfish runs on a JVM in the ‘goldfish’ zone.
I want to DTrace that JVM.
By default, local zones don’t have enough privilege to run dtrace (as it lets you peek into the kernel).
I was about to change that when I realised something.
I tend to assume that’s obvious, until I talk to non-Solaris users.
This last point is a huge benefit of zones which I think a lot of people overlook, or mistake for a negative. It also makes my job today much easier.
VMware hosts or Xen dom0s monitor at the VM/domain level:
xen has xentop, vmware has ‘VMware tools’( ESX has a bloated GUI I am trying to drink away).
You can start a console in a vm (like ‘zlogin’ on Solaris) but you might as well SSH into them. It’s no easier to keep track of your processes than if they were on remote machines.
In a global Solaris zone, the processes are all there alongside you. Just use your usual monitoring tools -
ps, prstat, etc. are all zone-aware. Accounting is a doddle too.
If you don’t treat zones as VMs, there’s a much simpler way to do this.
Instead of granting ‘goldfish’ dtrace privileges I can simply monitor the JVM from ‘vera’.
This 1-liner (from the DTrace wiki)
fires at every JVM method call, printing the classname of the called instance and the called method:
vera # dtrace -qn 'hotspot*:::method-entry { printf("-> %4s.%s\n", stringof(copyin(arg1, arg2)), stringof(copyin(arg3, arg4))); }'
dtrace: buffer size lowered to 2m
NB: this is running as root in the global zone, which has more than enough privilege for dtrace.
Trouble is, I got no output.
Turns out the ’method-entry’ Hotspot probe is disabled by default for performance reasons.
To enable the method-entry probe, you pass the ’-XX:+ExtendedDTraceProbes’ flag when the JVM starts.
Having to bounce the JVM (or run it in a ‘debug’ mode by default on a production box) would suck.
Luckily, the JDK comes with a tool called jinfo – it lets you read system properties, command line flags etc. from a running JVM.
What the manpage doesn’t say is that since JDK 1.6 jinfo can also set those properties on the running JVM.
On other OSes, we’d have to drop into the VM as root (or do some remote JVM voodoo). On Solaris, we can see all processes in all zones from vera
(actually we could do this bit from the zone, since it doesn’t need special privileges).
vera # jinfo -flag +ExtendedDTraceProbes $(pgrep -z goldfish java)’pgrep -z goldfish foo’ returns the PID of all processes called foo in the ‘goldfish’ zone
(I’m only running 1 java process in the goldfish zone, so I know pgrep will find the right one). jinfo sets the flag on the PID returned by pgrep.
Immediately, the probe starts to fire
and the dtrace window starts spewing class and method names:
vera # dtrace -qn 'hotspot*:::method-entry { printf("-> %4s.%s\n", stringof(copyin(arg1, arg2)), stringof(copyin(arg3, arg4))); }'
dtrace: buffer size lowered to 2m
-> java/util/HashMap/HashMap$HashIterator.newKeyIterator
-> java/util/HashMap$KeyIterator.<init>
-> java/util/HashMap$KeyIteratorhIterator.<init>
-> java/util/HashMap$HashIteratorctorImpl.<init>
-> java/lang/Objectenterprise/server/ss/provider/ASSelector.<init>rise/server/ss/provider/ASSelector
-> java/util/HashMap$HashIteratorver/ss/spi/ASSocketFacadeUtils.hasNexti/ASSocketFacadeUtils
-> java/nio/channels/spi/AbstractSelectorSSocketServiceProxy.beginServiceProxy
-> java/nio/channels/spi/AbstractInterruptibleChannelce.blockedOn
-> sun/misc/SharedSecretsrise/server/ss/ASSocketService.getJavaLangAccesscketService
-> java/lang/Threadenterprise/server/PEMain.currentThreadrver/PEMain
...
...
There are other interesting probes there (that don’t fire several hundred times a second) – garbage collection,
method compilation, classloading.
See the full probe list for more.
When we get what we came for, we can let the JVM run at full speed again:
vera # jinfo -flag -ExtendedDTraceProbes $(pgrep -z goldfish java)and dtrace shuts up.
It’s been a while since I cared about j2ee, but it looks like Glassfish runs rails via jruby
(and I like a free nano as much as the next guy). Reason enough to kick the tyres.
I like to run services in a dedicated zone
(especially if I’m expecting to try, hate and delete them over the course of 15 minutes).
vera # zonecfg -z goldfish
goldfish: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:goldfish> create
zonecfg:goldfish> set zonepath=/zones/goldfish
zonecfg:goldfish> set autoboot=true
zonecfg:goldfish> add net
zonecfg:goldfish:net> set address=10.0.0.5/24
zonecfg:goldfish:net> set physical=iprb0
zonecfg:goldfish:net> end
zonecfg:goldfish> commit
zonecfg:goldfish> exit
vera #
vera # zoneadm -z goldfish install
vera # zoneadm -z goldfish boot
vera # zlogin -C goldfishThe last command connects us to the zones console (where sysidconfig sets up timezone, root password, etc.).
I want to create snapshots and filesystems within the zone, so
I’ll let the zone have 4GB of my zpool, using the ‘add dataset’ command.
vera # zfs create tank/delegated ; zfs create tank/delegated/goldfish
vera # zfs set quota=4G tank/delegated/goldfish
vera # zfs set mountpoint=none tank/delegated/goldfish
vera # zonecfg -z goldfish 'add dataset; set name=tank/delegated/goldfish;end'
vera # zoneadm -z goldfish rebootNB: mountpoint must be ‘none’ or you’ll get an error:
could not verify zfs dataset tank/delegated/goldfish: mountpoint cannot be inherited zoneadm: zone goldfish failed to verify
(this ensures we don’t inadvertantly leak information into the zone).
The zone can now see the dataset (its parents are visible too but you obviously can’t access them).
Note the quota we set earlier.
goldfish # zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 24.7G 11.5G 1.50K legacy
tank/delegated 49K 11.5G 24.5K legacy
tank/delegated/goldfish 24.5K 4.00G 24.5K none
goldfish # zfs create tank/delegated/goldfish/j2ee
goldfish # zfs set mountpoint=/j2ee tank/delegated/goldfish/j2ee
goldfish # zfs list tank/delegated/goldfish/j2ee
NAME USED AVAIL REFER MOUNTPOINT
tank/delegated/goldfish/j2ee 24.5K 4.00G 24.5K /j2eeThe global zone can still see the dataset, snapshot it, etc.
but it has a special property set – ‘zoned’ – which tells the global zone to ignore the mountpoint property (for security again).
vera # zfs list tank/delegated/goldfish/j2ee
NAME USED AVAIL REFER MOUNTPOINT
tank/delegated/goldfish/j2ee 24.5K 4.00G 24.5K /j2ee
vera # zfs get zoned tank/delegated/goldfish/j2ee
NAME PROPERTY VALUE SOURCE
tank/delegated/goldfish/j2ee zoned on inherited from tank/delegated/goldfish
vera # ls /j2ee
/j2ee: No such file or directory
vera # zfs set mountpoint=/wee tank/delegated/goldfish/j2ee
cannot set property for 'tank/delegated/goldfish/j2ee': 'mountpoint' cannot be set on dataset in a non-global zoneYou can manually unset the zoned property if you like doing really stupid things.
Now I ssh into the zone,
download the Java EE 5 SDK Update 3 Preview with JDK
and run it.
Plenty of Java-based installers crap out if you don’t have X installed.
Sun thoughtfully added a ’-console’ flag to the installer to allow text-only installs.
More importantly, when the X installer craps out it tells you to try rerunning it with the ’-console’ option.
If that sounds like an obvious thing to do you clearly haven’t run many commercial software installers.
goldfish # chmod +x java_ee_sdk-5_03-preview-solaris-i586.bin
goldfish # ./java_ee_sdk-5_03-preview-solaris-i586.bin -consoleProduct: Java Platform, Enterprise Edition 5 SDK Location: /j2ee Space Required: 265.56 MB ------------------------------------------------ Java 2 SDK, Standard Edition 6.0 Sun Java System Message Queue 4.0 Sun Java System Application Server Platform Edition 9 Sample Applications Java BluePrints Your First Cup: An Introduction to the Java EE Platform Ready to Install
When the installer completes, it tells you
goldfish # /j2ee/bin/asadmin start-domain domain1
Browse to http://yourzonesip.com:4848 , log in with the admin credentials, job done.
Course, you don’t have to install it in a zone, but it takes 5 minutes and this way you don’t have to worry about running it as non-root.
Zones are great for hosting several glassfish instances on one physical server:
This will make it much easier to try out things like clustering.
First impressions of Glassfish itself are:
I still won’t be surprised to find it floating upside down tomorrow.
Well, it’s still there I’ve since
deployed Roller to it
and added some resource caps
(about 600Mb seems to be much more than it needs).
Oh, and I won the Nano 