This is your Content-Aware scaling.

And this is your Healing Brush and Content-Aware Scaling on steroids. Because every now and then something comes along in the post-production world that makes you lose your freaking mind. In a good way.

Your first question? How in Holy Hell (no, that is not an oxymoron) is this possible? How can an algorithm know how to do this without the aid of fairy dust, Rainman or Richard Branson? Well, a couple of fancy-pants Adobe researches have paired up with a professor from Princeton and a PhD student to whip up a delicious algorithm called PatchMatch. Which, surprisingly, is not a board game for hip, iPhone-wielding quilt-sewing sugar-induced ADD twenty-somethings. It’s actually a “Randomised Correspondence Algorithm for Structural Image Editing” or “Algorithm to put half the world’s photographers out of business and the other half into lazy, jaded and slightly over-caffeinated Photoshop monkeys”. But AWESOME monkeys, so it’s OK! Yes!

So when will we expect to see this feature in Photoshop? Adobe Creative Suite 5 will, according to Adobe, hit the stores on April 12 2010. Which coincidentally is the date we plan for CEO Garion Hall to order it for us, thanks! Nevertheless, CS5 is expected to include a number of advanced technologies, like three dimensional brushes and warping tools. It also is expected to bring more aggressive use of the GPU, including improved utilization of the CUDA GPU computing technology found on NVIDIA graphics cards.

OK, so we wouldn’t know a randomised correspondence algorithm from a bottle of medium spiced salsa, but we’re already thinking of the many applications for the Content-Aware fill. Photobombers ruining our superb photographic technique, the occasional Facebook friend who never wants to be tagged and will therefore be content-awared out of the photo completely, the ugly power line obscuring our otherwise misleadingly romantic sunset photos, the unfortunately placed hand in group photos, the junk mail in our Inbox, some of your hair (when you need a haircut), the person in front of you at the cash register with too many groceries, Nickleback, your girlfriend with the weird fetish of dressing up like herself and pretending to be obnoxious, politicians, utility bills, Lady Gaga, cracks on the pavement you continually trip over no matter how many times you try to remember where they are, dirty clothes, the cast of every reality television show in existence, and pickles.

That’s what this guy did. Not the drug lords part, the part with robots and blowing stuff up.

So we thought, yeah, this is a cool video. The post production is a little rough, but he definitely has some skills and he knows how to use them. But then we wonder, well, there were no actors, there was no plot, no narrative. Just robots. And destruction. And robot destruction. How is he going to direct a 30 million dollar film? As it turns out, he won’t be writing his own script, but will be directing someone else’s. Either way, we hope he’s up for the challenge.

Yeah but, we hear you ask, do they actually go anywhere? Can these backyard directors actually succeed? The answer is yes. Sometimes yes. OK, occasionally yes. Although it usually helps if Peter Jackson happens upon a few shorts you’ve done in your spare time. And it usually helps if you use your own camcorder, shoot yourself playing the part of a robot, rotoscope, and then you have something like Tetra Vaal. Or a documentary about extraterrestrials marooned in Johannesburg, called Alive in Joburg. Adidas gives you a small budget to make a viral campaign about an artificial intelligence gone rogue that kinda sorta doesn’t really relate at all to shoes. Or yellow. But it’s called Yellow. Then you go ahead and make some freaking awesome shorts for Microsoft’s Halo 3 video game:

Peter Jackson asks you to direct the Halo movie. The studios pull out and the project falls through. So instead, you make a movie version of Alive in Joburg, using the same gritty style of cinematography with seamless CGI so that it becomes a critical award-winning success. This would be called District 9. Which you should know already because if you haven’t seen District 9 we don’t want to be friends with you.

Another newcomer to YouTube filmmaking: the guys behind What’s in the Box?, which went viral last year despite using ballerinas in soldier costumers (you’ll see what we mean if you watch it all the way through) and have since been approached by big wigs in Hollywood. Meanwhile, we here at G Media Post Production, are scratching our heads, possibly with our pants off (hey, it’s Friday), and wondering why we hadn’t thought of this. or why our CEO, Garion Hall, hasn’t thought of this. We’d like to think Garion will ask us to make a video that includes robots, pizza, sex and explosions and robots. Oh wait, that’s Transformers. OK, so robots, explosions and something resembling a plot. But when Peter Jackson comes knocking on our door we’d have to put our pants on. So maybe we’ll just stick to being awesome at what we do and leave the robots for Hollywood. Or Hollywood for the robots.

Our criteria for choosing a SAN solution had some very specific needs. But these needs don’t nessisariy just apply to our situation:
1) Multiple Avid NLE’s connected to a single storage pool
2) Non-proprietary disk hardware
3) Very scalable in both connections and storage size
4) A pay as you expand pricing model
5) Sufficient bandwidth for multicam HD editing
6) Would work over and existing fibre-channel infrastructure.

At the time we had an Avid Unity LANShare LP which had done its job well enough but could not be expanded beyond 8TB and 6 clients. The jumping off point was the CEO (Garion Hall) tasking me with investigating alternative means to exceed the 6 clients with the intention of employing more editors or at lease more editing machines to increase post production output. So with that in mind I went about outlining the kind of system we would want. In such a system we also wanted to move away from closed box, proprietary systems that lock you into their pricing and support contracts. We wanted the flexibility to expand the system as we needed it, the ability to do so ourselves with third party hardware that we could support ourselves.
Naturally we had specific bandwidth requirements that needed to be sustained (which is simple maths) and in our case we already had a 4Gb fibre-channel infrastructure that we wanted to continue using, or at the very least, not have to replace (at great expense).

There are hundreds of SAN solutions on the market for enterprise and HPC-scientific computing. By comparison there are very few designed specifically for post production. (Though that is changing at an ever increasing rate). Nearly all of the non-post specific solutions have two things in common that I believe make them poor choices for post production use. Firstly is the assumptions the traditional SAN model makes:

1) Low number of high speed clients. (I.e. A number of servers will be connected to a number of SAN controllers and then those servers will serve files out to a network.
2) Very large number of low speed clients
3) A very high volume of read/write requests (per user per file per second)
4) Generally very small file sizes. (from >1KB to under 50MB)
5) Read/writes are done non-sequentially, all over the platers/arrays/volumes

So not only are they set up to be the opposite of post production requirements:
1) Small-medium number of high speed clients (each with a user)
2) Little or no low speed clients
3) Low volume of read/write requests from workstations (often only 2-4 per user and only 1 file per user.)
4) Generally extremely large file sizes. (anywhere up to 10GB or more, around 10,000 times as big)
5) Read/writes are done sequentially (linear, long r/w from consistent plater/array/volume locations)

There is also an assumption that a single SAN controller (basically an intelligent array controller) will have the bandwidth to control many “slave” arrays. Or, put simply, there may be say ten 16drive arrays but there’s only a pair of 4Gb connections to it. So if your working in film then that huge tower of god knows how many petabytes is really only usable by two people. To further frustrate the situation, once you’ve reached the limit for a controller you need to buy a second one. (or maybe you want to buy lots of controllers so more people can get bandwidth) Then you have not one concentric storage pool but multiple SAN’s all needing individual configuration, management, fault tolerance, HA considerations and DR strategies. And to top it off, now each of your clients needs to connect to multiple SAN’s. Why did we want a SAN again?

The second major drawback to enterprise systems is their pricing structure. Because they assume a large number of slow clients and a large volume of internally controlled data, it gets prohibitively expensive to configure a system for the needs of post production. For instance, for me to replace our old Avid Unity with a solution from NetApp I would need six SAN controllers and the management modules and back end switching to support them. For me to connect six high speed clients, running non server OS’s and have 8TB would have cost me more than five times the cost of the Unity. About $250K and that’s to only MATCH the unity, not scale up from it! (and to completely knock this on the head, none of them support os X, which makes up about 75% of the post production market for workstations.)

So, then you turn to post production friendly SAN solutions. These have drawbacks too. Primarily, they’re nearly all just as proprietary and closed-box as Avid. And most of them are about the same price. So, as we are running Media Composer only I’m back to just buying a new Unity!
The exceptions to this are systems such as Autodesk Stone and Apple XSan. I will get back to them later.

FEAR NOT, there is another way! Well, two actually but they stem from the same idea. Clustered file systems (basically a fancy high speed NAS) and heterogeneous file systems (a controllerless or headless SAN with a special filesystem on it). Both share some core functionalities:

1) Single, managed storage pool
2) Scalable numbers of high and low speed clients
3) Infinitely scalable storage size (well, effectively infinite)
4) Client platform agnostic (any OS can connect to it, within reason. An Amiga500 for instance wont have a client available!)

Sounds good right. Well there are some drawbacks and some good parts. Lets start with the clustered systems.

Clustered storage is basically a genesis on the basic SAN model but takes away the “head” and the basic NAS model but with shared storage and connections across the whole cluster. Clustered systems are made up of “nodes” or blocks of storage that are intelligent, can operate individually but when connected to each other, share their storage and connections out as a single entity. On paper this is brilliant. In real life, there are some hurdles but we’ll get to that. Most of these systems rely having a back end interconnection between each node and a front end interface for the clients to attach to. Systems such as Isilon and BlueArc use Infiniban to interconnect the nodes and allow 1Gb and 10Gb Ethernet connections for the clients. This way any client connected to any node can access data from all the nodes at high speed. It’s a good model and its very effective. But again we’re pushed back to a very proprietary system because the internal file management (and indeed the file system itself on many of these products) is very complex and has to have 100% hardware/software reliance. On the up side, they’re nearly indestructible and child’s play to expand. So for post production environments that don’t have on the ground ITS, they are a really good solution. Isilon (which I’ve used) and BlueArc (which I haven’t used but gets rave reviews from studios around the world) are good examples for ease of use and have very good FT/HA. In an Isilon cluster, adding a node (9TB, ~6 usable) is a task a receptionist could do with a one page instruction page. No, really it is that easy. Plus, you can literally walk up to a cluster and yank out a drive or even a whole node and NOTHING WILL HAPPEN. (apart from someone like me getting a lot of urgent emails from it). This is because the filesystem and the data are spanned across the whole cluster. It just goes about re-striping and shunting stuff around until its happy.. The only thing your editors will notice is the usable space of the volume will have suddenly gone down. For us here at G Media though it came down to cost. Not just the initial outlay but the cost of expansion. Because each node is intelligent and includes all the hardware it means each node is, by comparison to a straight array (non-SAN) hugely expensive:
• Non-managed array $/TB ~$1,000
• Avid Unity $/TB ~$5,000
• Isilon $/TB ~$10,000
You see with things like Isilon you’re paying for the hardware, the software license, the filesystem license and the support PER NODE.

Also note that while these systems behave as a SAN they are in fact a super high performance NAS solution. And thus use protocols such as NFS and even SAMBA for the clients. This has its own issues attached. For instance, we couldn’t get older Avid Media Composer systems to get anything over 50MB/s over 1Gb Ethernet and the HP workstations (xw8200 & xw8400) simply would not let a 10G card run jumbo frames. Ultimately, we couldn’t playback more than two streams of DCVProHD1080i50. Not enough for us with 2, 3 or even 4 camera multi-cam shoots. I wont go into the other issues with running media storage on NAS but I will say that even if we had newer workstations (some of them actually are) and could use 10Gb Ethernet with jumbo frames it still means installing 10G infrastructure on CX4 copper (as Isilon didn’t support fibre), CX4 10G Ethernet switches, CX4 high speed connection heads for the cluster. AHHHGG! Very expensive.

So, heterogeneous file systems:
Heterogeneous file systems, such as SGI’s CXFS and Quantum Stornext, work on the principle of removing the “head” of a traditional SAN and allowing it to have control over any storage you choose to present it (in the form of LUNs) over what ever connection you choose to use. So the “head” becomes transparent (or virtual really) in the sense that while it controls the filesystem traffic between client and storage (like a traditional controller) It’s not the conduit (and bottle neck) for the traffic, the clients are attached to the storage directly. Clever right… This is achieved by splitting control requests and data transmission up. File requests and OS updates (metadata) are sent over standard Ethernet (either in band with an existing LAN or across its own private LAN). Data r/w is done over a storage connection (FC, IB, 10G what ever) directly. The control requests are all handled by a (or several) servers called metadata servers. They talk to all the clients and to the storage and point everyone in the right direction.

Here at G-Media, we finally decided on Quantum StorNext, so that’s the system I will describe. All of the NLE’s, the storage arrays, tape library and MDC’s are connected via fibre channel through FC switching. (the fabric for this switching is zoned specifically for this task, a topic I will discuss at some other time). At the same time all the NLE’s and the MDC’s are also connected to a LAN. When an NLE (client) requests a file it makes that request via the LAN to the MDC, the MDC holds the metadata for where on the arrays the files are, instructs the client (and thus the clients HBA) where to go, the client HBA sends SCSI commands over FC to the array controllers which send back the data and finally the client returns any updates to the MDC.

Additionally, the MDC’s are responsible for creating and maintaining the filesystem on the storage. The major advantage is you can use any storage hardware you can imagine, provided it can be mounted by the MDC’s. Even a USB stick can be added to the storage pool (not that I suggest you do this!!!). Expanding storage is as simple as adding arrays, presenting the LUNs to the MDC, getting the MDC to stripe them with the file system and add them to an existing volume. All the user will see is the available space go up.

The MDC’s themselves are just normal server hardware running RHEL, nothing proprietary or fancy at all. And a pair of them can run as an HA fail-over configuration. Adding more clients is a per seat license fee with no additional costs for file system licensing, server based connection licenses or additional hardware (assuming you have a spare port on your FC switch). You can also run a virtual SAN client over Ethernet that will actually run at 100MBps (not 30MBps like normal CIFS) called a Distributed LAN Client or DLC. And if you’re really keen you can run 10G Ethernet or IB for these DLC’s.

Clients are available for just about any imaginable platform, including OS X. In fact, the Mac license is an XSan license because XSan IS StorNext under the hood, just pre-packaged by Apple for Apple. In fact you can connect PC’s to an XServe RAID XSan by buying a StorNext license (that’s another story though). The same is true for Autodesk Stone. All StorNext under the hood. So really when you think about it if you buy XSan your paying a premium for shittier hardware that makes it hard to connect anything but Mac’s to. Whereas you could be running an enterprise wide solution for every system on better, more redundant, faster hardware with better HA and FT credentials if you just go to StorNext. I should quickly point out that Autodesk Stone is NOT shitty hardware or a bad implementation. With them your getting good stuff, and paying for the Autodesk badge.

While going down the path of a self configured and managed heterogeneous file system may not be the right choice for every post production operation, if you have capable IT people on staff it could save you time and money, especially in the long run. On the other hand, its defiantly NOT the kind of system you want to have only off site phone support for with no one technical around to maintain it. Especially because you will end up with multiple vendors, manufactures and suppliers all of whom will doubtlessly blame each other when something goes wrong.

Smoke has been around for a while, often used in high-profile television programs such as “House”, “NCIS”, CSI: NY” and “Entourage”, Super Bowl commercials and even the packaging, promotions and content for the 2008 Beijing Olympics. All in all, Smoke is in its own field as an all-in-one toolset. Editing, titles, high-end colour correction, image stabilisation, tracking, keying, compositing (both 2D and 3D), paint, rotoscoping, retouching ­— whatever you can think of, Smoke can do it. No more bouncing around between Final Cut “not quite” Pro and Color or Motion or After Effects. Sure, you can do your fancy tricks in After Effects in a few hours, or you could do it in Smoke in five minutes. And now, you can do it on a shiny, sexy Mac.

And so, a pair of Post Production Ninjas wandered down to the venue/pub for the launch/free beer. It was clear from the get go that Autodesk knew how to put on a product launch (take that, Apple). With our hands clutching generous helpings of Smoke for Mac brochures and finger food stolen from the buffet, we sat down for what would an extended yet highly entertaining run-through of just how Smoke works, on a Mac no less. Using real footage from a real television program, we were led through some of the very cool things Smoke can do. And we were certainly impressed. Not only can Smoke rotoscope and key as fast as it takes us to devour a tray of savoury pastries, it’s colour correction cube (in 10-bit no less) is freaking awesome and made us pee our pants a little. All in all, Smoke is intuitive, efficient and a pleasure to use.

After the launch, we were eager to get back to the bunker to download ourselves a trial version of Smoke and take it for a spin. The trial version does not check for specific hardware requirements; you can even run it on a Macbook Pro if you were that way inclined — although you’ll need a resolution of 1920 otherwise you’ll lose a few buttons on the sides. The full version of Smoke will require a Mac Pro that meets Autodesk’s requirements. Oh, and did we mention Sparks? The third-party plugins for Smoke? A lot of people have been asking about Sparks on the Mac, and it seems that Sparks are tentatively delayed for now. This, we were told, all depends on how warmly Smoke for Mac is received by the industry.

(Read: how many copies of Smoke for Mac are sold.) So there’s a fair chance we might see Sparks sometime this year. One can hope, right?

Of course, there is a price. $US14,995 to be exact. And that’s not including the annual subscription for $US1,995. Sounds pricey, yes, but not when you consider it used to cost upwards of $80,000 for one of these bad boys, you can hardly complain. And also consider Smoke could replace your Avid or Final Cut Studio, along with After Effects and possibly others, all on your existing Mac Pro, it’s not too unreasonable after all. Bringing Smoke into this mid-range price bracket is big news, and allows small to medium-sized post production houses the possibility of adding a Smoke in their arsenal; something that will surely pay for itself very quickly. Now if only we can convince Garion Hall the same thing!

(If you’re just looking for the solution, scroll down to the “Solution” heading.)

This is a relatively informal post in part meant as a rant and in part a solution…

BACKGROUND

Thanks to Apple’s super proprietary nature and the relative idiot proofing of any documentation, getting XSAN clients to connect to Stornext volumes requires some out of the box thinking.

We have just finished installing a new SAN system to take care of the whole company’s data requirements including the Avid Media Composer systems used in G Media’s post production.  I’ll go into this in more detail in other posts.  For the purposes of this post I can describe the system broadly as follows.

2 x 24TB array enclosures configured with a series of specifically created RAID groups set up as RAID5, 6 or 50 depending on the application.

This presents 4 main volumes:  media, data, hafs and complete.

These are presented to Quantum Stornext metadata controllers and formatted with SNFS.

All the Avid Media Composers and several back office servers are SAN attached Stornext clients over 4Gb FC.  There is also a Bridge machine that is fibre attached and serves Stornext distributed LAN client connections to a further series of servers.

The installation of all the Windows clients was really quite easy.  A simple point and shoot installer, give it the IP addresses for the two Meta Data Controllers (MDCs) and map volumes to Windows drive letters.  Getting a Mac to connect was a different story!

For those who don’t know, Apple’s XSAN product is Quantum Stornext under the hood.  It’s basically a version of the MDC code ported to Darwin.  What this means is you can buy a client version of Stornext for various other OSs and connect them to your Apple XServe RAID.  That is if you were silly enough to buy one.

What Apple don’t advertise is you can do the reverse:  run a Stornext set of disks and MDCs, and connect a Mac to it using the XSAN client.  What neither Quantum or Apple will tell you up front is some pair of idiots from both sides did some deal that says only Apple is allowed to sell Apple Stornext software.  What this lead to is Apple refusing to port the Distributed Ethernet Client (DLC) for the Mac, sighting incompatibility with their XServe RAID product.  On the up side, XSAN clients are cheaper than either the SAN or DLC Stornext clients so you just have to deal with the fact you’re going to need the fibre infrastructure to support them.

THE LEAD UP

So!  You get your XSAN license fancy little box in the mail and open it to find even more designer packaging and a manual that is both understandable by three-year-olds and tells you absolutely nothing.  On the disk is an installer, that runs and finishes.  Then you launch the XSAN admin utility, conveniently located in /Applications/server so you don’t find it the fist time you look.  Running this results in an initial setup phase that first asks the question:  Create a new SAN or join an existing one.  So you click on ‘Join existing SAN’ (as we already have one) and it asks you for a server, username and password.  It doesn’t tell you what they’re for but its easy to assume its for the primary MDC.  At this point you’ll find yourself going around in circles because it can’t connect to the MDC and never will this way.

THE FAULT FINDING:

(From here I’ll describe the process I went through determining what was going on.  It may give someone out there some other ideas or at least a better understanding of what the back end of Stornext looks like.)

Note that what we’re trying to do is get the Mac to see and mount two of the SN volumes called ‘media’ and ‘complete’.  What follows are the notes at the time:

The MDCs are both listed in DNS, all the FQDN’s work from any given machine.  The MDCs are not binded to the AD though, and would be a difficult task to do.  Neither is the Mac we’re trying to install the client on, this may be easier to do.

After installing XSAN and launching XSAN admin, you’re given the option to create a new SAN or connect to an existing one.  When you hit ‘connect to an existing one’ it asks for a server, username and password.  If you put in the IP, the name or the FQDN of the MDCs it comes back and tells you it can’t find the server on the network.  Interestingly if you put in its own IP, username and password, it comes back and tells you the machine is not a server.  So at least it’s looking …  at something.

All the different OS clients are very similar once installed and all have the same set of utilities and config files.  Where to find them differs on the OS.  Using them is generally similar in the CLI but certain syntaxes may change depending on the OS.  The Mac client behaves very similarly to the AIX Unix client, however the files are located in \Library\Filesystems\Xsan instead of \usr\Filesystems\stornex.

The ‘cvadmin’ utility in \Library\Filesystems\Xsan\bin immediately returns “no FSSs are active” and states that “the XSAN file system services may be stopped on 127.0.0.1.”  I messed around with some of the commands to no avail.  Weirdly if I tried “start media 192.168.0.135″ it said the FSM already existed, but didn’t do anything about it.

Also, the ‘fsmpm’ process isn’t running either.  Presumably because the initial config hasn’t been completed to start it.  This would explain why ‘cvadmin’ can’t find any FSSs, I suppose.  You can start ‘fsmpm’ by running ‘xsand’ directly from \Library\Filesystems\Xsan\bin.  However, ‘cvadmin’ comes back with the same result after this.

The Mac can see all the luns OK from disk-manager.  One interesting thing is cvadmin’s ‘disks’ command returns nothing.  However, if you start the ‘fsmpm’ process (by running ‘xsand’ directly from Library\Filesystems\Xsan\bin) then the ‘disks’ command returns a series of what look like the volumes but segmented (I’m assuming by LUN) across multiple mount-points, located where you’d expect them in \dev\rdisk(x).  Further attempts at ‘start’, ‘activate’ or ‘fsmlist’ commands return nothing interesting.  Nor does running ‘cvadmin’ again.

All the other clients have a “fsnameservers” file in the config folder.  So I created a plaintext file called fsnameservers, put in the IP numbers of the MDCs (in order) and put the file in \Library\Filesystems\Xsan\config.  No change after a restart.

If you run ‘mount_acfs’ from \Library\Filesystems\Xsan\bin it comes back and says “cannot mount, license authorization failure”.

And that was the clue that I needed. Basically, because the initial install hadn’t completed and I hadn’t put in a  serial number, the installer hadn’t put all the config files where they should be and activated everything.

THE SOLUTION

First we need to fool the XSAN installer that we’re going to be using an Apple MDC.  In this case, the machine we’re installing the client on will be that MDC.  At that first screen you want to create a new SAN, not join an existing one.  The prompts are fairly easy from here, add the system, add a serial number, tell it that this system is the MDC and finish the install.  At this stage you can close the admin utility and never need it again.

Navigate to /Library/Filesystems/Xsan/config/ in the Finder and open the fsnameservers file in text edit.  Make sure text edit is set up to save as plaintext not rich text and won’t add file extensions.  The file will contain the IP number of the machine your on.  Delete this and put in the IP numbers of the Stornext MDCs, in order (primary then secondary), save and close it.

You will need to have the root user enabled to continue.

Open a terminal and type:

#su

(enter root password)

# rm /Library/Filesystems/Xsan/config/.auth_secret

# kill –HUP `cat /var/run/fsmpm.pid`

# /Library/FileSystems/Xsan/bin/cvadmin

You should now see all the available Stornext volumes.  It is possible that ‘acfs_mount’ needs to restart first. So to be on the safe side, restart the machine, open a terminal and just type:

#su

(enter root password)

# /Library/FileSystems/Xsan/bin/cvadmin

To mount my volumes (‘media’ and ‘complete’) I did the following:

/#  cd Volumes

/Volumes#  su

/Volumes#  mkdir media

/Volumes#  mkdir complete

/Volumes#  mount -w -t acfs media /Volumes/media

/Volumes#  mount -w -t acfs complete /Volumes/complete

And you’re away!

What I will cover in future posts is how Avid Media Composer handles (or rather doesn’t handle very well) shared media on Stornext volumes.

And thenI found this article buried deep on Apples support site.