number 9

I posted the other day about
Solaris Express ZFS / iSCSI integration .
As I said, there’s no TPGT support in the ZFS offering
yet
so to present targets on specific interfaces you
need to use the iscsitadm commands directly.

As usual, the manpages are good, so treat this as an introduction to them.

start the daemon

The only setup needed is to assign a config directory

vera # svcadm enable iscsitgt
vera # iscsitadm create admin -d /etc/iscsi

make the TPGT

An iSCSI TPGT (Target Portal Group Tag) is just a list of IP addresses.
When you assign a target to the TPGT, you make it reachable on those IPs.

I want my targets on iprb0, so I create a new TPGT and add iprb0s IP address to it.

vera # ifconfig iprb0
....
        inet 1.2.3.4 ....
....
vera # iscsitadm create tpgt 1
vera # iscsitadm modify tpgt -i 1.2.3.4 1
vera # iscsitadm list tpgt -v 1
TPGT: 1
    IP Address: 1.2.3.4

sanity check

You are running a firewall, right? iSCSI is on tcp/3260, so a line like this in /etc/ipf/ipf.conf would be a good idea:

pass in quick on iprb0 proto tcp from iscsiclient to 1.2.3.4 port = 3260 flags S keep state

make the targets

I want eight targets, 2Gb each. So I make a zvol for each, then make a target on it:

vera # zfs create tank/iscsiluns
vera # for i in first second third fourth fifth sixth seventh eighth
do
 zfs create -V 2G tank/iscsiluns/$i
 iscsitadm create target -b /dev/zvol/rdsk/tank/iscsiluns/$i $i
done

The important bit here is the ’-b’ flag to ‘back’ the targets with zvols.

The zvol targets are ‘onlined’ immediately.
By default, iscsitadm backs targets with files.
These are zero-filled before being started, so they take ages to online (and by default
they get created in your ‘admin’ directory which may fill your root filesystem).

vera # iscsitadm list target -v eighth
Target: eighth
    iSCSI Name: iqn.1986-03.com.sun:02:d456321a-e2f9-ca43-8537-88a0d23b4a33.eighth
    Connections: 0
    ACL list:
    TPGT list:
    LUN information:
        LUN: 0
            GUID: 0
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 2.0G
            Backing store: /dev/zvol/rdsk/tank/iscsiluns/eighth
            Status: online

add targets to the TPGT

vera # for i in first second third fourth fifth sixth seventh eighth
do
iscsitadm modify target -p 1 $i
done

vera # iscsitadm list target -v eighth
Target: eighth
    iSCSI Name: iqn.1986-03.com.sun:02:d456321a-e2f9-ca43-8537-88a0d23b4a33.eighth
    Connections: 0
    ACL list:
    TPGT list:
        TPGT: 1
    LUN information:
        LUN: 0
            GUID: 0
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 2.0G
            Backing store: /dev/zvol/rdsk/tank/iscsiluns/eighth
            Status: online

The volumes are of course still zvols, so you can snapshot/clone them as usual:

vera # zfs list -t volume
NAME                     USED  AVAIL  REFER  MOUNTPOINT
tank/iscsiluns/eighth   36.5K  18.1G  36.5K  -
tank/iscsiluns/fifth    36.5K  18.1G  36.5K  -
tank/iscsiluns/first    36.5K  18.1G  36.5K  -
tank/iscsiluns/fourth   36.5K  18.1G  36.5K  -
tank/iscsiluns/second   36.5K  18.1G  36.5K  -
tank/iscsiluns/seventh  36.5K  18.1G  36.5K  -
tank/iscsiluns/sixth    36.5K  18.1G  36.5K  -
tank/iscsiluns/third    36.5K  18.1G  36.5K  -

I spotted iSCSI in the autumn and it seemed to tick a lot of boxes
(think SANs without the hype, HBAs, fibre channel, vendor lock-in or horrendous cost).

b54 SXCR has really nice ZFS iscsi integration :

zfs create tank/iscsi
zfs set mountpoint=legacy tank/iscsi
zfs set shareiscsi=on tank/iscsi

Then start the iSCSI target daemon¹. Nothing is shared yet.

sol# svcadm enable iscsitgt
sol# iscsitadm list target
sol#

Thanks to ZFS inheritance, any zvols (not filesystems, I checked) created under tank/iscsi will automatically become LUNs:

zfs create -V 4G tank/iscsi/lefty
zfs create -V 4G tank/iscsi/righty

And that’s it.

iscsitadm list target -v
Target: tank/iscsi/lefty
    iSCSI Name: iqn.1986-03.com.sun:02:5ee25ac8-4930-ea8e-bc86-9d475793e277
    Alias: tank/iscsi/lefty
    Connections: 0
    ACL list:
    TPGT list:
    LUN information:
        LUN: 0
            GUID: 0x0
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 4.0G
            Backing store: /dev/zvol/rdsk/tank/iscsi/lefty
            Status: online
Target: tank/iscsi/righty
    iSCSI Name: iqn.1986-03.com.sun:02:9cdcfa4d-a1dc-40f2-d70d-f294d5d9151f
    Alias: tank/iscsi/righty
    Connections: 0
    ACL list:
    TPGT list:
    LUN information:
        LUN: 0
            GUID: 0x0
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 4.0G
            Backing store: /dev/zvol/rdsk/tank/iscsi/righty
            Status: online

Renaming the ZVOLs causes the iSCSI target/alias to update (the IQN is preserved so clients should be OK) :

zfs rename tank/iscsi/lefty tank/iscsi/pinky
zfs rename tank/iscsi/righty tank/iscsi/perky
iscsitadm list target
Target: tank/iscsi/pinky
   iSCSI Name: iqn.1986-03.com.sun:02:5ee25ac8-4930-ea8e-bc86-9d475793e277
   Connections: 0
Target: tank/iscsi/perky
   iSCSI Name: iqn.1986-03.com.sun:02:9cdcfa4d-a1dc-40f2-d70d-f294d5d9151f
   Connections: 0

When you do the usual zfs magic, iscsitadm seems to keeps up:

zfs snapshot tank/iscsi/lefty@snap
zfs clone tank/iscsi/lefty@snap tank/iscsi/sinister
iscsitadm list target
Target: tank/iscsi/righty
    iSCSI Name: iqn.1986-03.com.sun:02:4cf907ba-db89-4403-c1bb-ed5db6fcb76b
    Connections: 0
Target: tank/iscsi/lefty
    iSCSI Name: iqn.1986-03.com.sun:02:25fb9c0e-c58f-edcc-e8bc-d9686c0f0e88
    Connections: 0
Target: tank/iscsi/sinister
    iSCSI Name: iqn.1986-03.com.sun:02:c734e4a8-7d65-e427-80c7-dda523e814b7
    Connections: 0

Note the new IQN for the clone.
Adding

zfs set readonly=on  tank/iscsi/sinister

gets you the benefit of the .zfs/snapshot directory NFS clients would see if you’d set sharenfs=on (OK, you’re sharing a snapshot of the entire LUN rather than the filesystem, but clients can still get their old files without bugging you, so it does the job).

Course, you can share them out read/write too.
Combining that with ‘zfs promote’ should be more than enough rope for anyones needs

Having all the config in ZFS itself means that
when I upgrade the server, I just need to zpool import -f tank and I get my iscsi LUNs along with everything else.

There are a few things lacking – there’s no way to configure TPGT groups, it’s 1 LUN per target – but nothing I need badly. Discovery works fine.

(UPDATE: if you need TPGT support, you can use iscsitadm directly until it’s integrated into SXCR)

¹ Without the last step, you’ll get an error (when inheriting the shareiscsi property) when you create the first zvol:

cannot share 'tank/iscsi/lefty': iscsitgtd failed request to share
filesystem successfully created, but not shared

which is bollocks, fortunately. It autostarts iscsitgtd for you.