number 9

I posted the other day about
Solaris Express ZFS / iSCSI integration .
As I said, there’s no TPGT support in the ZFS offering
yet
so to present targets on specific interfaces you
need to use the iscsitadm commands directly.

As usual, the manpages are good, so treat this as an introduction to them.

start the daemon

The only setup needed is to assign a config directory

vera # svcadm enable iscsitgt
vera # iscsitadm create admin -d /etc/iscsi

make the TPGT

An iSCSI TPGT (Target Portal Group Tag) is just a list of IP addresses.
When you assign a target to the TPGT, you make it reachable on those IPs.

I want my targets on iprb0, so I create a new TPGT and add iprb0s IP address to it.

vera # ifconfig iprb0
....
        inet 1.2.3.4 ....
....
vera # iscsitadm create tpgt 1
vera # iscsitadm modify tpgt -i 1.2.3.4 1
vera # iscsitadm list tpgt -v 1
TPGT: 1
    IP Address: 1.2.3.4

sanity check

You are running a firewall, right? iSCSI is on tcp/3260, so a line like this in /etc/ipf/ipf.conf would be a good idea:

pass in quick on iprb0 proto tcp from iscsiclient to 1.2.3.4 port = 3260 flags S keep state

make the targets

I want eight targets, 2Gb each. So I make a zvol for each, then make a target on it:

vera # zfs create tank/iscsiluns
vera # for i in first second third fourth fifth sixth seventh eighth
do
 zfs create -V 2G tank/iscsiluns/$i
 iscsitadm create target -b /dev/zvol/rdsk/tank/iscsiluns/$i $i
done

The important bit here is the ’-b’ flag to ‘back’ the targets with zvols.

The zvol targets are ‘onlined’ immediately.
By default, iscsitadm backs targets with files.
These are zero-filled before being started, so they take ages to online (and by default
they get created in your ‘admin’ directory which may fill your root filesystem).

vera # iscsitadm list target -v eighth
Target: eighth
    iSCSI Name: iqn.1986-03.com.sun:02:d456321a-e2f9-ca43-8537-88a0d23b4a33.eighth
    Connections: 0
    ACL list:
    TPGT list:
    LUN information:
        LUN: 0
            GUID: 0
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 2.0G
            Backing store: /dev/zvol/rdsk/tank/iscsiluns/eighth
            Status: online

add targets to the TPGT

vera # for i in first second third fourth fifth sixth seventh eighth
do
iscsitadm modify target -p 1 $i
done

vera # iscsitadm list target -v eighth
Target: eighth
    iSCSI Name: iqn.1986-03.com.sun:02:d456321a-e2f9-ca43-8537-88a0d23b4a33.eighth
    Connections: 0
    ACL list:
    TPGT list:
        TPGT: 1
    LUN information:
        LUN: 0
            GUID: 0
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 2.0G
            Backing store: /dev/zvol/rdsk/tank/iscsiluns/eighth
            Status: online

The volumes are of course still zvols, so you can snapshot/clone them as usual:

vera # zfs list -t volume
NAME                     USED  AVAIL  REFER  MOUNTPOINT
tank/iscsiluns/eighth   36.5K  18.1G  36.5K  -
tank/iscsiluns/fifth    36.5K  18.1G  36.5K  -
tank/iscsiluns/first    36.5K  18.1G  36.5K  -
tank/iscsiluns/fourth   36.5K  18.1G  36.5K  -
tank/iscsiluns/second   36.5K  18.1G  36.5K  -
tank/iscsiluns/seventh  36.5K  18.1G  36.5K  -
tank/iscsiluns/sixth    36.5K  18.1G  36.5K  -
tank/iscsiluns/third    36.5K  18.1G  36.5K  -

As you might have gathered, I’m a recent Solaris convert.

Thought I’d mention that Sun are doing a ‘shipit’ like program where they’ll send you the latest stable Solaris 10 release and Sun Studio (their enormous know-it-all toolchain) on DVD.

If you take my advice (remember, I’m a regular Nostradamus when it comes to picking the Next Big Thing)
, you can get it here .

UPDATE: Ceri just pointed out Get Opensolaris , which is the SXCR equivalent of the above. Looks nice too.

Ubuntu is a good Linux, but reiserfs let me down this autumn (nothing personal, the other linux filesystems suck too), and I had
a particularly stupid installer destroy ^_~/.mozilla last month.

I badly wanted a ZFS home directory (not badly enough to go near ZFS-FUSE; I’m not deranged ).
Nothing fancy, just a mirrored zpool for redundancy and regular snapshots to protect against pilot error.

(update: FreeBSD 7.0 now has excellent ZFS support which might be helpful if you don’t mind leaving Linux)

To avoid having to migrate fully, I moved my home directory to an NFS share on my Solaris machine (iSCSI wouldn’t help; I’d have to mkfs.crappyfs the LUN on the Linux end anyway. Besides, all the Linux iSCSI initiators I tried stank).

It works a treat.

planb$ pwd
/home/sisred
planb$ md5sum misc/oncall.txt
68db8407afbe4965918e3b425e2d5abd  misc/oncall.txt
planb$ rm misc/oncall.txt
planb$ echo 'shit'
shit
planb$ date
Sun Jan  7 21:16:12 GMT 2007
planb$ ls .zfs/snapshot/
[snip - lots of directories]
zfs-auto-snap-2007-01-05-06:00:00/
zfs-auto-snap-2007-01-05-12:00:00/
zfs-auto-snap-2007-01-05-18:00:00/
zfs-auto-snap-2007-01-06-00:00:00/
zfs-auto-snap-2007-01-06-06:00:00/
zfs-auto-snap-2007-01-06-12:00:00/
zfs-auto-snap-2007-01-06-18:00:00/
zfs-auto-snap-2007-01-07-00:00:00/
[snip - lots of directories]
planb$ cp -p .zfs/snapshot/zfs-auto-snap-2007-01-07-18\:00\:00/misc/oncall.txt misc/oncall.txt
planb: $ md5sum misc/oncall.txt
68db8407afbe4965918e3b425e2d5abd  misc/oncall.txt
planb$ uname -rs
Linux 2.6.17-10-386
planb$

Course, this works just as well with any OS that can mount NFS.
The only difference to running directly on Solaris is speed (due to 100Mbit link) and I have to ssh
over to run zfs(1).

Here are my notes (again, they’re mostly a paste of my notes, so ask if anything is unclear).

setup a dedicated link

This is the only remotely tricky bit, and it’s optional.
Both machines are on my desk, so I hooked them up with a crossover cable to keep NFS off the LAN.

I’m using NICs I found in a bin. Once they’re fitted:

both sides

I’ll choose a network of 10.10.10.0/30 for the link.
Each end gets a hostname of ‘red$host’ (the cable is red. Imagination bypass.).

Add this to /etc/hosts on each end:

10.10.10.1  redsun
10.10.10.2  redlinux

and then disable firewalls on those interfaces (eth2 on Linux and e1000g0 on Solaris in my case).

the linux end

add an entry to /etc/network/interfaces

auto eth2
iface eth2 inet static
  address 10.10.10.2
  netmask 255.255.255.252

then bring up the interface with sudo ifup eth2

By default, Linux tosses a coin to choose the interface number at boot, so now and then eth0 and eth2 will
swap.
I’m sure this is useful to some people, but if you prefer a sane networking setup , you can tie them down in /etc/iftab:

eth0 mac 00:13:20:b2:22:23
eth2 mac 00:90:27:12:4f:11

the solaris end

Before you can use a NIC, you need to plumb it in (this hooks up all the kernel processing streams to it, hence the name).

ifconfig
Bring up the interface with:

sun # ifconfig e1000g0 plumb
sun # ifconfig e1000g0 inet 10.10.10.1/30 up

For this to be done automatically at boot:

sun # echo redsun > /etc/hostname.e1000g0
sun # echo "10.10.10.0    255.255.255.252" >> /etc/inet/netmasks

You should be able to ping across the link now.

create a home filesystem

sun # zfs create tank/home
sun # zfs set compression=on tank/home
sun # zfs create tank/home/sisred
sun # zfs set mountpoint=/export/home/sisred tank/home/sisred

create a user

I gave up trying to persuade linux to do NFSv4, so I’m using v3.
That means the uids on each end need to match.

Since I’m moving an existing homedir, I create a Solaris user with matching UID/GID.
(run ‘id’ on the linux host – in my case, ‘1000’ for both).

sun # groupadd -g 1000 sisred
sun # useradd -c "Dick Davies" -g 1000 -u 1000 \
-d /export/home/sisred -m \
-s /usr/bin/bash sisred
sun # chown -R sisred:sisred ~sisred

No need for a password (I use via key-based SSH).

share it out

sun # zfs set sharenfs=on tank/home/sisred
sun # svcadm enable nfs/server
sun # svcs -xv
sun #

The second command will make sure NFS starts at reboot. ‘svcs -xv’ will give useful output if something didn’t work.

Shares are read/write by default. The ‘sharenfs’ property can also take a string of options to share(1M) –
so zfs set sharenfs=ro tank/home/sisred creates a readonly share, for example.

move into your new $HOME

linux # sudo mkdir /nfshome
linux # sudo chown sisred:sisred /nfshome
linux # sudo apt-get install portmap # for NFS locking
linux # sudo mount redsun:/export/home/sisred /nfshome
linux # rsync -va /home/sisred/ /nfshome/

(You need to run the last command as the user due to NFS ‘root squash’).

If it looks OK, you can make it permanent by editing /etc/fstab and remounting:

redsun:/export/home/sisred /home/sisred  nfs proto=tcp        0       2

now you’re just showing off

If performance is a bit slow on small files, take a look here for tuning.

You need something to automate snapshots up at the Solaris end. Two options are:

1. Tim Fosters SMF snapshots
   - some good features (automated ’ zfs -i send/recv’ of snapshots to a remote host, cleanup of old snapshots etc).
2. roll your own

UPDATE: I’ve since switched to NFSv4, which was easy when you know how. See here