Configuring Apple XSAN 2.0 clients for Stornext MDCs and volumes

Wednesday, March 10th, 2010 at 18:30

Configuring Apple XSAN 2.0 clients for Stornext MDC’s and volumes

(If you’re just looking for the solution, scroll down to the “Solution” heading.)

This is a relatively informal post in part meant as a rant and in part a solution…

BACKGROUND

Thanks to Apple’s super proprietary nature and the relative idiot proofing of any documentation, getting XSAN clients to connect to Stornext volumes requires some out of the box thinking.

We have just finished installing a new SAN system to take care of the whole company’s data requirements including the Avid Media Composer systems used in G Media’s post production.  I’ll go into this in more detail in other posts.  For the purposes of this post I can describe the system broadly as follows.

2 x 24TB array enclosures configured with a series of specifically created RAID groups set up as RAID5, 6 or 50 depending on the application.

This presents 4 main volumes:  media, data, hafs and complete.

These are presented to Quantum Stornext metadata controllers and formatted with SNFS.

All the Avid Media Composers and several back office servers are SAN attached Stornext clients over 4Gb FC.  There is also a Bridge machine that is fibre attached and serves Stornext distributed LAN client connections to a further series of servers.

The installation of all the Windows clients was really quite easy.  A simple point and shoot installer, give it the IP addresses for the two Meta Data Controllers (MDCs) and map volumes to Windows drive letters.  Getting a Mac to connect was a different story!

For those who don’t know, Apple’s XSAN product is Quantum Stornext under the hood.  It’s basically a version of the MDC code ported to Darwin.  What this means is you can buy a client version of Stornext for various other OSs and connect them to your Apple XServe RAID.  That is if you were silly enough to buy one.

What Apple don’t advertise is you can do the reverse:  run a Stornext set of disks and MDCs, and connect a Mac to it using the XSAN client.  What neither Quantum or Apple will tell you up front is some pair of idiots from both sides did some deal that says only Apple is allowed to sell Apple Stornext software.  What this lead to is Apple refusing to port the Distributed Ethernet Client (DLC) for the Mac, sighting incompatibility with their XServe RAID product.  On the up side, XSAN clients are cheaper than either the SAN or DLC Stornext clients so you just have to deal with the fact you’re going to need the fibre infrastructure to support them.

THE LEAD UP

So!  You get your XSAN license fancy little box in the mail and open it to find even more designer packaging and a manual that is both understandable by three-year-olds and tells you absolutely nothing.  On the disk is an installer, that runs and finishes.  Then you launch the XSAN admin utility, conveniently located in /Applications/server so you don’t find it the fist time you look.  Running this results in an initial setup phase that first asks the question:  Create a new SAN or join an existing one.  So you click on ‘Join existing SAN’ (as we already have one) and it asks you for a server, username and password.  It doesn’t tell you what they’re for but its easy to assume its for the primary MDC.  At this point you’ll find yourself going around in circles because it can’t connect to the MDC and never will this way.

THE FAULT FINDING:

(From here I’ll describe the process I went through determining what was going on.  It may give someone out there some other ideas or at least a better understanding of what the back end of Stornext looks like.)

Note that what we’re trying to do is get the Mac to see and mount two of the SN volumes called ‘media’ and ‘complete’.  What follows are the notes at the time:

The MDCs are both listed in DNS, all the FQDN’s work from any given machine.  The MDCs are not binded to the AD though, and would be a difficult task to do.  Neither is the Mac we’re trying to install the client on, this may be easier to do.

After installing XSAN and launching XSAN admin, you’re given the option to create a new SAN or connect to an existing one.  When you hit ‘connect to an existing one’ it asks for a server, username and password.  If you put in the IP, the name or the FQDN of the MDCs it comes back and tells you it can’t find the server on the network.  Interestingly if you put in its own IP, username and password, it comes back and tells you the machine is not a server.  So at least it’s looking …  at something.

All the different OS clients are very similar once installed and all have the same set of utilities and config files.  Where to find them differs on the OS.  Using them is generally similar in the CLI but certain syntaxes may change depending on the OS.  The Mac client behaves very similarly to the AIX Unix client, however the files are located in \Library\Filesystems\Xsan instead of \usr\Filesystems\stornex.

The ‘cvadmin’ utility in \Library\Filesystems\Xsan\bin immediately returns “no FSSs are active” and states that “the XSAN file system services may be stopped on 127.0.0.1.”  I messed around with some of the commands to no avail.  Weirdly if I tried “start media 192.168.0.135″ it said the FSM already existed, but didn’t do anything about it.

Also, the ‘fsmpm’ process isn’t running either.  Presumably because the initial config hasn’t been completed to start it.  This would explain why ‘cvadmin’ can’t find any FSSs, I suppose.  You can start ‘fsmpm’ by running ‘xsand’ directly from \Library\Filesystems\Xsan\bin.  However, ‘cvadmin’ comes back with the same result after this.

The Mac can see all the luns OK from disk-manager.  One interesting thing is cvadmin’s ‘disks’ command returns nothing.  However, if you start the ‘fsmpm’ process (by running ‘xsand’ directly from Library\Filesystems\Xsan\bin) then the ‘disks’ command returns a series of what look like the volumes but segmented (I’m assuming by LUN) across multiple mount-points, located where you’d expect them in \dev\rdisk(x).  Further attempts at ‘start’, ‘activate’ or ‘fsmlist’ commands return nothing interesting.  Nor does running ‘cvadmin’ again.

All the other clients have a “fsnameservers” file in the config folder.  So I created a plaintext file called fsnameservers, put in the IP numbers of the MDCs (in order) and put the file in \Library\Filesystems\Xsan\config.  No change after a restart.

If you run ‘mount_acfs’ from \Library\Filesystems\Xsan\bin it comes back and says “cannot mount, license authorization failure”.

And that was the clue that I needed. Basically, because the initial install hadn’t completed and I hadn’t put in a  serial number, the installer hadn’t put all the config files where they should be and activated everything.

THE SOLUTION

First we need to fool the XSAN installer that we’re going to be using an Apple MDC.  In this case, the machine we’re installing the client on will be that MDC.  At that first screen you want to create a new SAN, not join an existing one.  The prompts are fairly easy from here, add the system, add a serial number, tell it that this system is the MDC and finish the install.  At this stage you can close the admin utility and never need it again.

Navigate to /Library/Filesystems/Xsan/config/ in the Finder and open the fsnameservers file in text edit.  Make sure text edit is set up to save as plaintext not rich text and won’t add file extensions.  The file will contain the IP number of the machine your on.  Delete this and put in the IP numbers of the Stornext MDCs, in order (primary then secondary), save and close it.

You will need to have the root user enabled to continue.

Open a terminal and type:

#su
(enter root password)
# rm /Library/Filesystems/Xsan/config/.auth_secret
# kill –HUP `cat /var/run/fsmpm.pid`
# /Library/FileSystems/Xsan/bin/cvadmin

You should now see all the available Stornext volumes.  It is possible that ‘acfs_mount’ needs to restart first. So to be on the safe side, restart the machine, open a terminal and just type:

#su
(enter root password)
# /Library/FileSystems/Xsan/bin/cvadmin

To mount my volumes (‘media’ and ‘complete’) I did the following:

/#  cd Volumes
/Volumes#  su
/Volumes#  mkdir media
/Volumes#  mkdir complete
/Volumes#  mount -w -t acfs media /Volumes/media
/Volumes#  mount -w -t acfs complete /Volumes/complete

And you’re away!

What I will cover in future posts is how Avid Media Composer handles (or rather doesn’t handle very well) shared media on Stornext volumes.

And thenI found this article buried deep on Apples support site.

Tags: , ,