My journey with NetApp

Comments Broken

2012-05-28T13:41:00.002+05:30

Yeah, it was a fool of me to believe on Google and think there can't be an error like that. Anyway I did upgrade the dated look of my blog, and with new look everything was fine, except comment section. I did try to restore the old config but all the restore methods failed even restore from template backup which I did saved before making change failed.

So guess what, as to this writing comments are only working from IE8, will check other browsers also will let you know if that does work.

Thanks,

pNFS in NetApp

2012-05-28T02:13:00.000+05:30

In my last post I discussed about pNFS features and architecture, and promised that in next post I will discuss about NetApp’s implementation of pNFS. So here it is.

Since 1992, the time when NetApp was formed it is always known for its solid NFS implementation and frontrunner of NFS design and standardization. Today when NetApp has gone unified, and support not only NFS but SMB, FC, iSCSI, FCOE and some other less known protocols also, still it is a major driving force to NFS design. The latest example is pNFS where NetApp has proved again by delivering pNFS to its ONTAP 8.1 Cluster Mode offering.

In my opinion they have done a wonderful job with their implementation of pNFS, though currently it’s limited to support of file only, but heck yeah it’s wonderful. In NetApp’s implementation of pNFS not only data is distributed across all the nodes but metadata also. So if you have got metadata intensive workload you can tackle that as well by using round robin from your DNS infrastructure and instantly you have a cluster which not only scales linearly with data nodes but metadata nodes also.

Another niche feature is live volume transition. It gives flexibility to move a live volume in any node of cluster if ever required, that too non-disruptively. Thanks to pNFS, it keeps same NFS handle while data being served from new node so you don’t need to unmounts and remount the filesystem and since we use cluster wide namespace no change is required at its directory location also. Isn’t this sweet, no more struggling to get downtime to move a volume, huh!

And the best part is that it’s built around industry standard set by IETF, so there is no requirement of special client software and you can use it along with NFS version 3, 4 or 4.1 without pNFS supported client.

The only setback from using it across your environment is limited client support. As of today, only Fedora 16 with kernel 2.6.39 and RHEL 6.2 can be used, however soon we will see its support in other distributions also as it’s already in kernel mainstream and it will not be long when other distributions start shipping it.

References:
http://media.netapp.com/documents/wp-7153.pdf
http://media.netapp.com/documents/tr-4063.pdf

pNFS

2012-05-27T21:59:00.000+05:30

As my nature, I was lazing around over the weekend when I noticed the newly published best practice document on pNFS and ONTAP 8.1 Cluster Mode. Soon I realized this is what c-mode is made for; parallelism. How? Let’s see that.

First let’s start with what is Parallel NFS or in short ‘pNFS’?

pNFS is extension of NFS version 4.1 which adds support of parallelism to existing NFS version 4.

Well, that was the shortest answer I could write however here’s little more detail on it.

pNFS is part of NFS 4.1, the second minor version to NFS version 4 which adds support of session, and directory delegation, along-with parallelism. The idea to use SAN filesystem architecture and parallelism in NFS was originated from Gary Grider of Los Alamos National Lab and Lee Ward of Sandia National Lab, later it was presented by Gearth Gibson, a professor at Carnegie Mellon University and founder and CTO of Panasas, Brent Welch of Panasas, and Peter Corbett of NetApp in a problem statement to Internet Engineering Task Force (IETF) in 2004. Later, in 2005 NFSv4 working group of IETF commended drafts and in 2006 it folded into the 4.1 minor version draft. It’s published under RFC 5661 describing NFS version 4.1 with parallel support and RFC 5662 detailing protocol definition codes. pNFS is not limited to file, support for block data (RFC 5663) and object based data (RFC 5664) are also added, so now it’s possible to access not only file but object (OSD) and block (FC/iSCSI) based storage also over NFS.

pNFS, being open source, does not require additional software or drivers on the client that are proprietary to enable. Therefore, the different varieties of NFS can coexist at the same time and supported NFS clients can mount the same file system over NFSv3, NFSv4, and NFSv4.1/pNFS.

It is widely accepted by industry and jointly developed by Panasas, NetApp, Sun, EMC, IBM, UMich/CITI and many more however at the time of writing only Fedora 16 with kernel 2.6.39 supports all three layout types (blocks, files and objects) whereas RHEL 6.2 support only files layout.

Now question comes, what was the need for it?

We all love NFS for its simplicity, and from the time it was designed by Sun in the era of 10Mb Ethernet, it scaled well to 100Mb and then gigabit Ethernet, however with the advent of 10Gb and 40Gb links, single stream designed protocol wasn’t enough to scale it further. Industry has already used TOE Ethernet cards, link aggregation and bigger boxes but that wasn’t sufficient to utilize the bandwidth and CPU powers we have available now. So what was left to deal with? Parallel NFS.

So, what’s so different from earlier version of NFS?

pNFS is not much different from its ancestors it just separates metadata from data. Unlike traditional NFS Versions 3, 4, and 4.1, where metadata and data are shared on the same I/O path, with pNFS, metadata and data travels on different I/O paths. It allows metadata server handles all the metadata activities from the client, while the data servers provide a direct path for data access.

+-----------+

|+-----------+ +-----------+

||+-----------+ | |

||| | NFSv4.1 + pNFS | Metadata |

+|| Clients |<----------Metadata------------>| Server |

+| | | |

+-----------+ | |

||| +-----------+

Data |

||| |

||| Storage +-----------+ |

||| Protocol |+-----------+ |

||+----------------||+-----------+ Control |

|+-----------------||| | Protocol|

+------------------+|| Data |------------+

+| Server |

+-----------+

Figure 1: pNFS Architecture

As a result of this, in a clustered storage system with multiple nodes you mount only a directory or root namespace but you get direct access to data from each nodes. The metadata server (MDS) handles all nondata traffic such as GETATTRs, SETATTRs, ACCESS, LOOKUPs, and so on. Data servers (DSs) store file data and respond directly to client read and write requests. A control protocol is used to provide synchronization between the metadata server and data server.

Ok, but where does pNFS add value?

Large files and high number of concurrent users. With the advent of parallel computing with multi node cluster, single job gets divided amongst n nodes and when a job arrives at computational nodes they all try to access same data from one storage location which soon becomes bottleneck however with pNFS multiple storage system nodes responds with parts of file in parallel, increasing the aggregated bandwidth and lowering the latency. At the same time when many small files are accessed by large number of concurrent users single storage system can get chocked however with pNFS all nodes of storage system hosting the file, share user load.

Great, but how all this work?

In principal, pNFS uses parallelism used by RAID-0 however at different level. As in RAID-0 data is spread across multiple disks for faster response, same way in pNFS one file/filesystem is spread across multiple nodes in clustered storage array for faster response.

For example when client sends read requests for a file to storage system, storage system replies with file metadata along with layout details, detailing node address, data location, and striping information, after getting layout details client knows list of cluster nodes having parts of the file. Now client directly contacts to all the data nodes simultaneously for the file and nodes reply with the parts of files they have which clients later assembles.

I think It’s enough for now, next post will detail about pNFS implementation by NetApp

Read for scholars:

http://www.pnfs.com/

http://datatracker.ietf.org/wg/nfsv4/

http://www.pdl.cmu.edu/pNFS/index.shtml

http://www.citi.umich.edu/projects/asci/pnfs/linux/

http://www.pnfs.com/docs/LISA-11-pNFS-BoF-final.pdf

http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-pnfs-00.txt

http://www.pdl.cmu.edu/pNFS/archive/gibson-pnfs-problem-statement.html

How to quit SSH session by typing 'exit' in ONTAP

2012-05-23T00:08:00.000+05:30

Just a little trick to remove annoyance for new NetApp admin.

Usually when we want to terminate a SSH session on ONTAP you type Ctrl+D as typing 'exit' returns with message to press 'Ctrl+D' to exit. It's perfectly fine and there isn't any harm as that gracefully closes the session but as most of us are more use to of typing 'exit' to terminate the session, find it odd.

So for those who prefer typing 'exit' to close the session can do so by enabling telnet.distinct.enable option.
Default setting for this option is off which in effect mirrors telnet and console session but if you don't use this feature, which I believe most of us don't, can enable it and feel more at home.

I am back

2012-05-19T12:48:00.000+05:30

Yeah, I was absense from blogesphere for some time but guess what it was worth that because during my last a year and half I have learned so much about NetApp and its related technologies that I really fealt what a n00b I was.

Anyway I must say thankyou to all of you who encouraged me and patiently waited for my return.

Thanks.

How to restore data from aggregate snapshot

2010-12-03T00:13:00.000+05:30

Today one of our user found himself in wet pants when he noticed his robocopy job has overwritten a folder, rather than appending new data to it. Being panicked he run to me looking for any tape or snapshot backup of his original data, which unfortunately wasn’t there as previously he confirmed that they don’t need any kind of protection.

Now at this time I had only place left where I can recover the data, aggregate level snapshots; so I looked at aggregate snapshots and saw it goes back to time when he had data in place. Knowing that the data deleted from volume is still locked in aggregate’s snapshot I was feeling good that I have done a good job by having some space reserved for aggregate level snapshot, which no one ever advocated.

Now the next step is to recover the data, but problem was that if I revert aggregate using “snap restore –A” then all the volumes in that aggregate will be reverted which will be bigger problem. So had to go on a different way, use aggregate copy function to copy the aggregate’s snapshot to an empty aggregate and then restore the data from there.

Here’s the cookbook for this.

Pre-checks:

The volume you lost data from is a flexible volume
Identify an aggregate which is empty so it can be used for destination (could be on another controller also)
Make sure the destination aggregate is either equal or larger than source aggregate
/etc/hosts.equiv has entry for the filer you want to copy data to and /etc/hosts has its IP address added, in case of copying on same controller loopback address (127.0.0.1) should be added in /etc/hosts file and local filername should be in hosts.equiv file
Name of aggregate’s snapshot which you want to copy

Example:

Let’s say the volume we lost data was ‘vol1’, the aggregate which has this volume is ‘aggr_source’, the aggregate’s snapshot which has lost data is ‘hourly.1’ and empty aggregate where we will be storing data to is ‘aggr_destination’

Execution:

Restrict the destination aggregate using ‘aggr restrict aggr_destination’
Start the aggregate data copy using ‘aggr copy start –s hourly.1 aggr_source aggr_destination’
Once the copy is completed online the aggregate using ‘aggr online aggr_destination’
If you have done copy on same controller, system will rename the volume ‘vol1’ of ‘aggr_destination’ to ‘vol1(1)’
Now export the volume or lun and you have your all lost data available.

So here’s the answer to another popular question, why do I need to reserve space for aggregate level snapshot. Do you have the answer now?

Most destructive command in Ontap

2010-10-12T04:24:00.000+05:30

There are some commands which shake me when I run or even when I am close to them, but never thought I could be make my filer so close to death by just mistyping a command.

Yes, indeed I did it by typing 'gbd' rather than 'dbg', these two are so close to each other that my buttery fingers didn't realize that I mistyped a command and by the time I could realize it was too late. Sigh!

Little background on this 'gbd' command.

This command is in diag mode and so debug, however whereas 'dbg' captures filer debug info on console or a file, 'gbd' sends kill signal to all the processors which stops all the work on filers and everything just hangs. The only way to recover your filer is by hard rebooting it, either by RLM or physically tipping the power button.

I don't understand why brilliant NetApp engineers have made it so simple, why couldn't they use a command like 'use_this_to_kill_your_system' or something like that and I swear no one would ever type it.

Anyway I did it and I admit, that I should have checked before hitting return which I didn't. But guess what I was lucky enough to not to do it on a prod system and this test/dev system was having only a bunch of NFS clients were connected to it which made it sort of invisible to client systems due to the nature of NFS protocol.

What is the other command which you think shouldn't be so easy along with this?

How to do host/user/group or netgroup from filer

2010-10-12T00:32:00.000+05:30

Often we want to do nslookup for a host or NIS/LDAP lookup for a user or group for troubleshooting purpose. You have a unix system handy and you do it from there however what if you suspect results are not same as what your filer may be getting?

If you are troubleshooting CIFS issue, you are in luck with command 'cifs lookup' however, if you are dealing with DNS or NFS issue then you are out of luck, unless you go into advanced mode. Yes, you go inside advance mode and you get access to lot of other commands including one very nifty command 'getXXbyYY', which is incredibly useful but hidden from the view of admins for some strange reason, really I am not sure why NetApp thinks this shouldn't be available to end user as every time I do troubleshooting I feel the need of this and by no way I see this to be making any sort of changes on filer.

Anyway here's the command, though command says using "man na_getXXbyYY" for additional info however I couldn't locate it on systems therefore I use

test1*> getXXbyYY help
usage: getXXbyYY
Where sub-command is one of
gethostbyname_r
gethostbyaddr_r
netgrp
getspwbyname_r
getpwbyname_r
getpwbyuid_r
getgrbyname
getgrbygid
getgrlist

For more information, try 'man na_getXXbyYY'

Please remember this command is not available in admin mode and search order depends of your /etc/nsswitch.conf entry, so before you start thinking that this isn't working as expected please check these two things first.

Though all the subcommands are self explanatory however I have added small description for each of them.

gethostbyname_r - Resolves host name to IP address from configured DNS server, same as nslookup
gethostbyaddr_r - Retrieves IP address for host name from configured DNS server, same as reverse lookup
netgrp - Checks group membership for given host from LDAP/Files/NIS
getspwbyname_r - Displays user information using shadow file
getpwbyname_r - Displays user information including encrypted password from LDAP/Files/NIS
getpwbyuid_r - Same as above however you provide uid in this command rather than user name
getgrbyname - Displays group name and gid from LDAP/Files/NIS
getgrbygid - Same as above however you provide gid in this command rather than group name
getgrlist - Shows given user's gid from LDAP/Files/NIS

Examples:

test1*> getXXbyYY gethostbyname_r landinghost1
name: landinghost1
aliases:
addresses: 10.21.242.7

test1*> getXXbyYY gethostbyaddr_r 10.21.242.7
name: landinghost1
aliases:
addresses: 10.21.242.7

test1*> getXXbyYY netgrp support-group testhost1
client testhost1 is in netgroup support-group

test1*> getXXbyYY getpwbyname_r root
pw_name = root
pw_passwd = _J9..gsxiYTAHEtV3Qnk
pw_uid = 0, pw_gid = 1
pw_gecos =
pw_dir = /
pw_shell =

test1*> getXXbyYY getpwbyuid_r 0
pw_name = root
pw_passwd = _J9..gsxiYTAHEtV3Qnk
pw_uid = 0, pw_gid = 1
pw_gecos =
pw_dir = /
pw_shell =

test1*> getXXbyYY getgrbyname was
name = was
gid = 10826

test1*> getXXbyYY getgrbygid 10826
name = was
gid = 10826

test1*> getXXbyYY getgrlist wasadmin
pw_name = wasadmin
Groups: 10826

Execute command from a file on Ontap

2010-08-29T05:47:00.000+05:30

Occasionally we want to quickly or periodically run a set of pre defined commands on filers, like when we are making little change in network configuration and want to minimize the network down time or when we are creating a volume and know snap reserve, snap schedule, de-dupe, autosize or anything else which needs to be changed after every volume creation. If you are executing commands from unix terminal then a better way would be to keep all the commands in a text file and would do something like

bash>for $i in vol1
>do
>snap sched $i 0
>snap reserve $i 0
>blah $i
>blah $i
>done

and everything would be done, but imagine how would you do if you are doing through console. Unfortunately Ontap doesn't supports any kind of script not even for loop so we have to run each and every line of command either by typing on console or doing a copy-paste from text file from our desktop.

However there is a better way, use notepad to create set of command as you would execute on console with correct order, copy it off to filer and use 'source' command to execute each and every line from it. I know it's not such a brilliant idea as still you have to copy and paste everything to a file on filer however it's wee better than executing each and every command on console.

Think about you have to re-run /etc/rc so either you can use 'rdfile /etc/rc' to print everything on console and copy all the the line and execute on console or just run 'source /etc/rc' and let it run all the commands for you. You can also use 'source -v /etc/rc' to print the commands on console but not to execute, just to get an idea if there are any junk characters or any unwanted command inside the file, just as a precaution you better be sure that all the commands are valid and correct as if a command fails source doesn't stops there and just goes to next command in list.

Use it and I am sure you will like it next time when you are making some changes on filer which needs ten different commands to be executed.

How to check unplanned downtime detail for a NetApp filer

2010-08-28T23:47:00.002+05:30

Every now and then someone ask us what is uptime of system and we just type 'uptime' on system console to get the detail instantly.

This is really handy command to know when the system was last rebooted and how many operations per protocol it has served since then. Wouldn't our life be little easy if managers get satisfy with this detail? Alas! but that doesn't happen and they ask us to give all the details since we have acquired the system or 1st January and then we go back to our excel sheet or ppt we have created as part of monthly report to pull the data.

How about if we can get same information from system with just a command, wouldn't that be cool. Fortunate enough we have little known command 'availtime' right inside Ontap which just do the exact same function and specifically created after thinking about our bosses.

HOST02*> availtime full
Service statistics as of Sat Aug 28 18:07:33 BST 2010
System (UP). First recorded 68824252 secs ago on Mon Jun 23 04:16:41 BST 2008
   Planned downs 31, downtime 6781737 secs, longest 6771328, Tue Sep 9 15:07:33 BST 2008
   Uptime counting unplanned downtime: 100.00%; counting total downtime: 90.14%
NFS (UP). First recorded 68824242 secs ago on Mon Jun 23 04:16:51 BST 2008
   Planned downs 43, downtime 6849318 secs, longest 6839978, Wed Sep 10 10:11:43 BST 2008
   Uptime counting unplanned downtime: 100.00%; counting total downtime: 90.04%
CIFS (UP). First recorded 61969859 secs ago on Wed Sep 10 12:16:34 BST 2008
   Planned downs 35, downtime 17166 secs, longest 7351, Thu Jul 30 13:52:25 BST 2009
   Uptime counting unplanned downtime: 100.00%; counting total downtime: 99.97%
HTTP (UP). First recorded 47876362 secs ago on Fri Feb 20 14:08:11 GMT 2009
   Planned downs 8, downtime 235 secs, longest 53, Wed Jan 20 14:10:18 GMT 2010
   Unplanned downs 16, downtime 4915 secs, longest 3800, Mon Jul 27 16:01:02 BST 2009
   Uptime counting unplanned downtime: 99.98%; counting total downtime: 99.98%
FCP (DOWN). First recorded 68817797 secs ago on Mon Jun 23 06:04:16 BST 2008
   Planned downs 17, downtime 44988443 secs, longest 38209631, Sat Aug 28 18:07:33 BST 2010
   Unplanned downs 6, downtime 78 secs, longest 21, Fri Feb 20 15:24:44 GMT 2009
   Uptime counting unplanned downtime: 99.99%; counting total downtime: 34.62%
iSCSI (DOWN). First recorded 61970687 secs ago on Wed Sep 10 12:02:46 BST 2008
   Planned downs 21, downtime 38211244 secs, longest 36389556, Sat Aug 28 18:07:33 BST 2010
   Uptime counting unplanned downtime: 100.00%; counting total downtime: 38.33%

I am not sure why NetApp has kept this command in Advanced mode but once you know this command I bet next time you will not refrain yourself going inside advance mode to see how many unscheduled downtime you had since last reset.

A shorter version of this command is just 'availtime' it also shows the same information as 'availtime full' however it truncates letters from output and denotes Planned with P and Unplanned with U which is very good if you want to pass it in script.

HOST04*> availtime
Service statistics as of Sat Aug 28 18:07:33 BST 2010
System (UP). First recorded (20667804) on Wed Sep 23 09:35:49 GMT 2009
   P 5, 496, 139, Fri Dec 11 15:58:19 GMT 2009
   U 1, 1605, 1605, Wed Mar 31 17:01:41 GMT 2010
CIFS (UP). First recorded (20666589) on Wed Sep 23 09:56:04 GMT 2009
   P 7, 825, 646, Thu Jan 21 19:08:03 GMT 2010
   U 1, 77, 77, Wed Mar 31 16:34:54 GMT 2010
HTTP (UP). First recorded (20664731) on Wed Sep 23 10:27:02 GMT 2009
   P 3, 51, 22, Thu Jan 21 19:17:25 GMT 2010
   U 4, 203, 96, Thu Jan 21 19:08:03 GMT 2010
FCP (UP). First recorded (20477735) on Fri Sep 25 14:23:38 GMT 2009
   P 3, 126, 92, Thu Jan 21 19:07:57 GMT 2010
   U 4, 108, 76, Wed Mar 31 16:34:53 GMT 2010

In order to reset the output use 'reset' switch and it will zero out all the counters, make sure you have recorded the statistics before you reset the counters as once you reset the counters you will not be able to get details of system uptime since system was built so you may like to do only after you acquire a new system, have done all the configuration and now it's the time for it to serve user requests.