number 9

(I have avoided that cliche all year, give me a break)

I’ve run various releases of Roller on various Glassfish versions, and am pretty happy with what I’ve settled on.

Here are some gotchas it’s taken me a while to Google fixes for.

Glassfish v2

Glassfish v3 is going to be awesome when it’s done, but it’s not a production release.
Get the Sun-branded release – it’s the same codebase, but has a pretty installer and the
option of commercial support down the line if you ever need it.

Roller bowler bowler penny a pitch

Roller 3 is obsolete now and clanky to setup. 4.0 is much much easier to deploy (the install guide even has instructions for Glossfish2), but JavaMail only works on 4.0.1. Dave Johnson (snoopdave, Roller developer) made a pre-built Roller 4.0.1 tarball available a while back.

It’s good, you should use it.

friends don’t let friends use DB2

No-one uses DB2 unless they’re forced to (no-one you’d want to talk to, anyway).
Roller hates it even more than I do. Avoid it unless you enjoy pain.

Better options I’ve tried are:

• PostgreSQL works fine with Roller I love it (but it can be tricky to cluster)
• MySQL is what most Roller guys use. I can’t bring myself to recommend it (we have previous) but I’d expect less Roller bugs talking to it.
• we’re currently on an Oracle 10g backend, it works so far. Someone else supports it, so that’s ok with me.
• Derby/JavaDB worked well enough, but it’s CLI is pretty dreadful (cheers IBM) and you might need to shell into it one day. Good within NetBeans, when you’re too lazy to install a real DB (did I mention the PostgreSQl installer yet)?

fustercluck

Glassfish clusters are easy to setup and actually fun to look after. If you want Glassfish to HA your sessions you should add a <distributable /> element to your web.xml. Roller doesn’t use sessions for much, but it’s not a lot of work, so you might as well.  Other than that, it’s pretty straightforword.
Roller clusters are a different kind of fun.

Out of the box, Roller will do horrible things to you if you try to cluster it.
My favourites (so far)

• search is broken
• caching doesn’t work (well, it caches, it jut does it inconsistently on random nodes)
• scheduled tasks get run twice or not at all, often corrupting your schedule as they do it
• file uploads need a shared filestore.

I got round these with the following roller-custom.properties:

# hook in an external search engine instead
search.enabled=false

# these should be mounted from a reliable fileserver
uploads.dir=/glassfish/uploads
planet.aggregator.cache.dir=/glassfish/planet

# memcache instead of file-cache, scales better anyway
#
# see my last post for details
#
cache.defaultFactory=net.java.roller.tools.cache.memcached.MemcachedLRUCacheFactoryImpl
cache.memcached.default.servers=clusternode1:11211,clusternode2:11211
# it's a bit verbose by default
log4j.category.net.java.roller.tools.cache.memcached=INFO

# give each node a unique clientId - see below
config.expandedProperties=uploads.dir,search.index.dir,tasks.clientId
tasks.clientId=${roller.nodename}

wait, what?

tasks.clientId needs to be unique for each node in your Roller cluster.
Since we run the same WARfile on each node, we make that an alias to a JVM system property, ‘roller.nodename’, that’s different on each cluster node.

In Glassfish, you can do this by creating a cluster-wide JVM option which is set to a system property, and give each server instance a different value for system property.

Assuming we have a cluster called ‘rollercluster’ with 2 server instances ‘instance01’ and ‘instance02’, this should do the trick:

asadmin create-jvm-options --target roller-cluster '-Droller.nodename=${NODENAME}'
Command create-jvm-options executed successfully.
asadmin create-system-properties --target instance01 NODENAME=instance01
Command create-system-properties executed successfully.
asadmin create-system-properties --target instance02 NODENAME=instance02
Command create-system-properties executed successfully.

Restart your server instances, and
‘ps’ should show ‘-Droller.nodename=instance02’ on the server running instance02.

(Thanks to Anil Gangolli and Kedar Mhaswade on the roller and glassfish lists for helping figure this out)

One downside to embedding your database server is that Glassfish needs more memory.

I’d been messing around with a few webapps and yesterday got Roller4 running. After a few clicks around the app, I started to see ‘PermGen space: java.lang.OutOfMemoryError’ errors. 

More power, Doctor

The fix is pretty obvious : have the JVM allocate more PermGen space.

It’s simple to do this through the admin UI ( ‘Application Server’ -> ‘JVM Settings’ -> ‘JVM Options’).

Or you can do it on the command line:

 hypnotoad:~ $ asadmin create-jvm-options \-XX\\:MaxPermSize=128m
created 1 option(s)
Command create-jvm-options executed successfully.
hypnotoad:~ $ 

You need to restart the server for it to actually take effect.

If you’re dealing with lots of small files, a
NFS and ZFS combination
can run slower than you’d like.

For example, untarring the apache source tree on the NFS client:

planb:$ time tar xvf httpd-2.2.4.tar.gz
real    2m9.594s
user    0m0.356s
sys     0m0.508s

Ouch.

BenR mentioned turning off the ZIL
(ZFS intent log ) to speed up ZFS+NFS over aggregated Gbit ethernet – I doubted my 100Mbit link would have the same bottleneck for anything,
but it was worth a go.

Tell the ZFS/NFS server to switch off the ZIL:

vera# echo 'set zfs:zil_disable=1' >> /etc/system

Then either reboot, or run:

vera # echo ‘zil_disable/W 1’ | mdb -kw
vera # zpool export tank
vera # zpool import tank

In my case, I might as well reboot.

planb:$ mv httpd-2.2.4 outoftheway
planb:$ time tar xzf httpd-2.2.4.tar.gz
real    0m4.862s
user    0m0.368s
sys     0m0.440s

Holy crap.

There are some
implications to the ‘correctness’ of this from the NFS clients point of view but on the ZFS box itself it’s non-lethal, so I think
I’ll keep it (I’m snapshotting the share three times a day, so I’m
reasonably safe if when Linux shits itself).

Now that the dust has settled a little , I thought it was time to upgrade the blog to 4.0.x.

I was running typo 2.6.0 with rails frozen at 1.0.0 , so to be honest I was expecting trouble. WEBrick ran so fast on my dummy run I thought it was worth the risk.

If you’re thinking of trying it yourself:

CYA

• backup everything, including the database
• switch your typo theme back to ‘azure’

The last one saves you a lot of hassle if you’re using a ‘tweaked’ theme – if typo can’t find default.rhtml, it won’t run (it’s not as easy to switch themes from the console as it used to be, as the Settings model has been replaced).

fitter, happier

• download and extract the typo tarball (gems on textdrive are a pain in the arse.) – this includes rails 1.1.6 in vendor¹/.
• copy over your existing config/database.yml
• migrate the database

 rasputnik typo-4.0.2 $ RAILS_ENV=production rake migrate

Watch it Just Work (it does on postgresql, anyway),
start it up and have a poke around.

start chopping

On Textdrive, they recently enforced resource limits. After a minute or two I started getting 500 errors as the fcgi backends bloated up and were killed off.
Luckily I’d been warned about this.

• go into ‘text filters’ under the admin screens and disable any text filters you don’t use (I’m textile only here)
• sidebars have to be removed manually , like so:

 rasputnik typo/ $ cd components/plugins/sidebars
 rasputnik sidebars $ ls
 aimpresence               delicious_controller.rb    tada
 aimpresence_controller.rb flickr                      tada_controller.rb
 amazon                    flickr_controller.rb        tag
 amazon_controller.rb      fortythree                 tag_controller.rb
 ...
 ...
 ...
 rasputnik sidebars $ rm -r forty* backpack* amazon* upcoming*
 rasputnik sidebars $ # restart typo
 rasputnik sidebars $ killall -9 ruby && ~/bin/spawn-typo.sh

That should save you a few Mb. I’ve still had the preview pane crash once while I was writing this, but otherwise it seems a lot happier.

UPDATE – crash bang wallop

Well it’s been a month, and it’s finally stablized. Make sure to do the extra steps here (mainly removing Sparklines) if you’re resource restricted.

¹ actually, it doesn’t. Looks like someone dropped a bollock (run ‘rake rails:freeze:gems’ if you want to lock it down).