A good guide us to place LUN's into Volumes grouped by similar Snapshot or Snapmirror regimes (as this functionality occurs at the Volume level). I think the techy term for grouping LUN's in this way is a 'Consistancy Group' - anything you need to get Snap'd together should be kept in the same Volume.

Another thing I picked up was that when allocating space for LUN's be sure to allocate twice the space you need to allow for Snapshots. This space requirement supercedes the default 20% allocated at the Volume level. For LUN based Snapshots the agent software on the host itself (eg SnapDrive for Windows or SnapDrive for Exchange) manages the Snapshot - it interacts with the SAN to ensure this happens properly but the SAN itself has no knowledge of whats inside the LUN.

What this means is that if every block in the LUN changes you need at least as much space again for the Snapshot or you'll get a disk-space error. Its unlikely this would occur - a situation in which it might would be a drive defragment which touched every block.

If you're considering getting a SAN or already have one you should check out the Toasters mailing list. I've searched the interweb for equivalent lists for HDS and EMC but haven't found any equivalent (that isn't actually hosted by the vender itself).

Its completely independent of NetApp but is an excellent place to ask questions or search for answers in the list archives.

A good overview of the list is here.

When you get down to the nitty gritty of configuring an iSCSI connection theres not actually a nice guide to getting this done. Sure there are plenty of docs and white-papers on the topic but many of them are either to detailed or not detailed enough (NetApp, Sun and Microsoft are all guilty of making something which should be simple more difficult than it needs to be).

From a Solaris perspective there are a couple of really good guides that fill in the blanks between the Solaris & NetApp documentation:

* OpenSolaris and iSCSI: NetApp Makes it Easy

* iSCSI Examples

A cunning way to do your SAN backups:

Schedule a job to mount your LUN's to the backup server and backup a SnapShot to tape from there. Requires a bit of scripting and tweaking but it should provide much more flexibility than trying to backup each server individually.

That way you can avoid being reamed by backup software vendors on a per host basis. You may still opt to do an NTBackup to file for servers and applications but the databases will reside on the SAN and get backed up to tape.

I'm not a storage person so it took me awhile to get my head around the terminology. I suspect Sysadmins who host databases get harassed regularly by their DBA's about this stuff on a regular basis and as a result are much more intimately acquainted with this stuff than I am. One feature that helps the Sysadmin stay out of DBA initiated RAID-config-hell is that DataOnTap (the NetApp OS) only supports RAID 4 or RAID DP (similar to RAID 6) - note that the 'D' in DP is for 'Diagonal' not 'Dual'.

In NetApp land -

An Aggregate is a collection of disks - this is fairly straightforward. One thing to remember is that for every aggregate you lose a disk to parity data - fine if you have multiple shelves of disks or groups of disks of different capacity (eg you might aggregate 7 15k 72Gb disks and another aggregate of 7 10k 300Gb disks) but not really needed if you have only a single shelf with all disks the same. I guess there are plenty of reasons you might want different aggregates on a single shelf but if you're not doing anything fancy you may as well stick with one). Its easy enough to expand an aggregate by adding disks but its not easy to downsize an aggregate.

A Volume is a chunk of space within an aggregate - note that by default the DataOnTap OS is in vol0 within aggr0. Don't mess with vol0! If you're doing CIF's or NFS sharing (eg NAS type functionality) then you'd dish up your shares at a volume level. Volumes come in two type - the old style TradVolume and the newer FlexVolume - unless you have a special requirement you're better off with a FlexVolume which lets you resize it on the fly.

A LUN is a SCSI term (Logical Unit Number) to reference an individual disk. From an NetApp iSCSI point of view (probably Fibre-Channel too) a LUN is a big file that exists in a Volume. From a server perspective the LUN is visible as a raw disk device to make merry with. A LUN can be easily resized (up or down) but be aware that the NetApp has no inherent knowledge of whats inside a LUN - this has implications for SnapShots - if you need to Snap a LUN's contents you'll want to get SnapDrive and/or SnapManager (depending on the app using the LUN) which acts as an agent on the server (which does understand the LUN's contents) to initiate a Snap within the LUN from the NetApp.

In terms of layout we were initially tempted to create a Volume per application (eg Exchange, Oracle, SQL etc) with multiple LUN's inside each volume. We're now looking at a Volume per LUN as this will give us more flexibility in terms of Snapshots & SnapMirroring (or restore from Backup).

We bit the bullet and bought two NetApp 'toasters' - a FAS270 for our DR/UAT site and a FAS270c ('c' for clustered) for our Prod site.

For 2Tb of storage apiece they were actually pretty cheap - we'll SnapMirror between the two sites, we'll use the SnapManager tools for SQL, Oracle and Exchange and iSCSI as our transport medium (Fibre Channel is to expensive and complicated although we will use it between our backup server and tape drive).

So I'll be collecting some tips here as we put real apps on these systems.

You can team your Nic's in a couple of ways - either singletrunk or multitrunk mode. Single mode is purely for failover - one connection dies and the other will pick up the connection. Multi mode provides failover, link aggregation and load-balancing.

If you have more than two Nic's you can do this via the web interface, if you've only got two then you'll have to use the console (obviously you can't reconfigure a Nic if its already being used for something else; eg if you 'ifconfig e0 down' you'll lose your connectivity to configure the trunking).

To create a multi trunk virtual interface called multitrunk1 with e0 and e1 issue the following command on the console:

vif create multi multitrunk1 e0 e1

Then to configure it do the usual:

ifconfig multitrunk1 [ip address] netmask [netmask address]

And you can brink it up or down in the same way as any other interface.

One important point to note is that if you do this from the console be sure to update /etc/rc & /etc/host to reflect the vif or you'll lose the interface after a reboot. The web interface does write this info to these files but its worth double-checking that the updates have been made.

Some discussion on how the NetApp simulator came to be and how it is used.

The latest simulators even come as pre-built VMWare images you can use via VMWare Player.

You also get demo keys for the majority of the add on modules you're likely to use so you can try before you buy.

So I've been reading a lot about iSCSI - mostly positive (practical and pragmatic) but a few negative (largely academic or have enough money to go with Fibre Channel).

I suspect people had their doubts when moving from hubs to switches and 10mb to 100mb (coax to utp was a no-brainer so there was no doubt there :-)

Theres an interesting Usenix presentation with a slide on Differences Between NFS and iSCSI.

It does compare a transport mechanism with a distributed file system though. Confusing.

If you use Oracle over NFS why use a SAN if you can get away with a lower cost NAS ? Then again NFS is Unix-centric and you can't do Windows specific stuff beyond simple file-sharing with NAS (eg SQL, Exchange). Many DBA's seem to dismiss or deride Oracle over NFS while quite a few seem to think its a great way to go (although it would be good to read a practical unbiased example that wasn't from a SAN vendor).

The results of the benchmarking in the paper do seem to favour iSCSI in situations where data is not shared between systems and in meta-data intensive applications.

There also seems to be some debate about the merits of using a TOE adaptor in terms of adding yet another layer of complexity between your application server and its data. On top of this there seems to be a small but growing market for WAFS adding yet another layer to your LAN/WAN.

I guess iSCSI has been around for a few years now which in IT terms means it should have achieved a certain level of maturity which should allay any fears . . .

Useful benchmarking tool - IOMeter - ex Intel but now Open Source.

Use it to simulate load and test performance on storage devices (local, NAS or SAN).

Interesting article on how it is used in performance testing iSCSI over a Cisco MDS 9000-series Multi-protocol Switch.

Via StorageMojo - CleverSafe looks very cool. Its a geographically distributed storage grid with built in encryption.

If you watch the flash demo you'll see a chunk of data split up to over eleven locations when its written and when its read back only half of the locations need to respond (due to the built in redundancy). Its like RAID but on a grand-scale.

Whats better is that it looks like they're building an open-source community around the product.

I'm going to sound like a bit of a NetApp schill but I found another pointer to to the statistically probability of a RAID reconstruct failure - Expect Double Disk Failures With ATA Drives. 9% with four drives and 48% with 16 drives. Nasty.

Went to a small vendor seminar to showcase some NetApp technologies and came away with some interesting information -

* The probability of a write failure is pretty small (usually in the legalise small print) but this small possibility increases as disk space increases (which is why a generic RAID of small disks is more reliable than a RAID of really big disks). Those consumer grade 500Gb and 1Tb disks are looking slightly less attractive now. In a failure situation if a disk dies and you goto reconstruct the array you could conceivably end up with a second failure due to a tiny write error - then you're screwed.

* NetApp get around this by using a variation on RAID 6 DP (like RAID 5 but with two parity disks) - any performance hit (and its significant if you set this up using a normal controller) is offset by NetApps smart controller (thats why storage vendors charge a premium for data security). This problem and NetApps response is vividly illustrated in this post to the 'Toaster' (NetApp nickname) mailinglist.

* Fibre-channel is big with Unix shops and iSCSI is big with Windows shops. Surprisingly NFS over IP is still popular in Unix-land too.

* Snapshotting now encompasses databases and mailstores. The snapshot facility places a much much lower performance overhead than a similar EMC device (granted they would say that). Apparently companies are moving away from tape based backup to disk based - keeping tapes around purely for occassional snapshots and compliance reasons.

* NetApp do 'thin provisioning' - essentially you can lie about your storage capacity (present 1 physical TB as 2 virtual TB). This was apparently implemented based upon lies developers would tell their admins, dba's and storage managers - once everyone had added in their own comfort factor it was discovered that only about 40% of the capacity was utilised and the rest was wasted. Pooling storage in a NAS or SAN and over-subscribing it means you can shuffle the space around depending on your needs at that time. Apparently the key is the forecasting tools which will help you to predict when you'll run out of space. It also tends to work better in multi-terabyte shops rather than gigabyte shops.

* You can now stream snapshots between filers in different locations (for DR / BCP / Replication) over any IP link (one NZ client does this over dialup to a location half way around the world) - this is possible due to the small 4k block size used by NetApp for storage - at the device level it only replicates changed blocks rather than the entire changed file.

Its always nice to hear vendor 'war stories' - apparently after eBay had their extended site outage in 2001 they called in Oracle who looked into the database side and found no problem with the backend software, some more (extensive) digging pinpointed the fault in disk firmware code - when the disk faulted the error was propogated up through the application layers and eventually killed the site. After this Oracle came up with their HARD initiative (essentially a database designed and implemented for the extremely paranoid) which computes its own checksums on data as its written (so it provides an extra layer of redundancy over the storage layer).

Another interesting Oracle specific tale outlined their datacenter - which uses blades and NetApp appliances extensively (storing petabytes of data). The interesting thing is that they worked through the economics of using a Fibre Channel HBA infrastructure for their blades and went with NFS over IP instead - working out that 1 x blade + 2 FC HBA's (for redundancy) was much more expensive than 1 x blade + 2 built in teamed Gb NIC's (and they were willing to wear the performance penalty). NFS also allows them to manage a central pool of storage rather than carving out chunks for direct attached storage. Interesting.

Apparently a big leap of faith is for DBA's to allow the device to handle the and manage the storage rather than thinking about sindle-count. Once they get over that they can forget about the storage and focus on the database.

I can't believe the level of confidence required to trawl Suns campus and come away with a brand spanking new Thumper storage server. This sort of passion is to be commended :-)

Nice article on iSCSI to demonstrate its cost-effectiveness at LinuxGazette. A little light on detail unfortunately but some interesting ideas nonetheless.

Company has 10 SCSI drives in 1U enclosure - InfoStation from StorCase.