.ForEach(delegate (Blog blog){

NFS on Server 2008 == annoying

jeriley.nospam@nospam.gmail.com (Jesse) — Mon, 29 Jun 2009 08:03:00 -1000

Got a linux box that has a nfs share - say its at 192.168.1.10 and the share under /etc/exports is

/some_nfs_share 192.168.1.0/24(rw,no_root_squash)

No problem right? On another linux box within the 192 subnet, the following works quite nicely. Assume folders exist.

mount.nfs -rw 192.168.1.10:/some_nfs_share /myfiles

No sweat, mounts right up and I get read-write so I know I did it correctly and ready for the Win2k8. Hop over, install the unix file share stuff (NFS, SMB) and do this ...

mount -o fileaccess=777 nolock casesensitive=yes -u:root -p:* \\192.168.1.10\some_nfs_share *

I can read files ...sort of and I can't write anything. In the windows application logs, I get Event ID: 16397, Source: NfsClnt with this crap

Windows(R) Lightweight Directory Access Protocol (LDAP) failed a request to connect to Active Directory Domain Services(R) for Windows user <Domain\MyUserAccount>.

Without the corresponding UNIX identity of the Windows user, the user cannot access Network File System (NFS) shared resources.

Verify that the Windows user is in Active Directory Domain Services and has access permissions

The hell? WHY are you trying to authenticate to LDAP? I just gave you a user/pass, why would I need more than that? Ok, FINE I bust out a local account, take that. Nope, exactly same error. Authen fail. I guess I'll have to go uber-lame and go ftp. I hate ftp.

Customer Service - Nexsan

jeriley.nospam@nospam.gmail.com (Jesse) — Thu, 25 Jun 2009 06:34:00 -1000

Back in January, I was tossed into a fire regarding an environment from your absolute worst nightmare. Yes, that bad. It's been called a "technology showcase", "how to do everything wrong", "what happens when you have no plan" and so on -- a real top notch problem suited for those who are ~~stupid enough~~ given the opportunity to fix something so terrible, I didn't really know where to start.

When I got my mind around this environment and some of the terrible things within it, I found something rather surprising -- lack of support and willingness to assist by vendors of hardware and low-level software -- that were under warranty. Some of which when contacted return an email saying "we don't recognize your email address, go away", others go into a black hole. Then came one whoop-ass exception - Nexsan.

For those who've never heard of them, Nexsan makes Storage Area Network devices and other storage centric solutions (archiving, compliance) for various industries such as Financial, Health care, etc. I was unfamiliar with Nexsan before this, but now, I'm a big fan for a bunch of reasons.

First off, finding a contact. I was AMAZED at how hard it was to get a simple "support@" or "contact@" addresses from some other site, but Nexsan does it right -- "support@nexsan.com" what a concept! Simple, easy. Sooo how's the response? Well, you get the auto-reply as confirmation and normally you get a reply within an hour or so. That's more than reasonable. I typically expect 4 business hours give or take.

Second, willingness to help. This is where Nexsan "gets it". Most companies I've found give you a not-so-greatly-trained person that is probably looking though a book to find your problem and are more than happy to push you off to the next person. This never happened, not once. My first contact person, Darrell Snipes asked that I give him a call and discuss our situation. After a short talk, he emailed me some recommendations for performance, a best practice document, a new firmware version, when our current support was going to run out and how to send him back the logs so he could take a deeper look ... and he followed up. What else could you want from a device you know nothing about? I couldn't think of one!

Third, following though and going beyond. At where I work, we push for exceeding client's expectations all the time -- someone must have told them the same thing. Our support ran out and, of course, we ran into a problem. With no more contract, nothing, I expected the rightfully-so "too bad, so sad". I contacted them asking if we could open up a pay-per-incident ticket and this is the response I got ...

Jesse,
If your unit is out of warranty/support as a industry leading company we don’t mind providing you with a help Monday to Friday 7AM to 5 PM PST.
However we will not be able to issue any RMA for damaged parts or anything similar.
Please let me know if you need any additional help,

It isn't a failure and wow, how great is that. I sent a reply asking them about a problem and again, totally reasonable, expecting the stated 7am to 5pm PST -- I sent my reply around 7:45pm fully NOT expecting a reply until the next day. At 11:42pm EST Chad Steele responded to my question, saying it was very fixable (firmware, we're WAY out of date) and to boot, 11:42pm is not within the 5pm PST time frame. Define : Awesomeness.

I have a short list of companies on my "list". Companies I do not hesitate recommending or sending anyone to -- Nexsan is definitely on that list. I highly recommend them.

Reverse phishing?

jeriley.nospam@nospam.gmail.com (Jesse) — Fri, 12 Jun 2009 02:35:00 -1000

I'm convinced there's a new type of phishing expedition going on in an attempt to hack search engine results. Why? Comments on my blog that don't add up. For one, the magic key words the person leaves as their name. I've dropped this as an image because I want none of those getting picked up by a search engine but its pretty clear they want an email.

The IP traces back to .ph ...hmmm, think they're could be a "California Orange County Lawyer"? Hardly not. Website comes back to nothing, which surprises me, but I've seen them come back to random, infested sites. Icing on the cake, the half right, confusing context english. Ahhh I love it. Curious, should I send an email to that account (after I create a new fake one of course)? Could be fun...

Update : 6/17/09 - here's another one I got. I post "interesting matters here"! Yay!

Rolling off the Net Admin - 12 lessons learned

jeriley.nospam@nospam.gmail.com (Jesse) — Fri, 29 May 2009 03:43:00 -1000

My past rash of posts have mostly focused on hardware, servers, linux and other related devices and I'm glad to say, I'm heading back into development. Granted, its VB but its development. So what have I learned in my nearly 6 month absence? Quite a few things, some of which will change my approach on a variety of things.

0. Doing lists of things is cool. I like lists. I like sites that give me lists. It gives me a clear beginning and end but also limits the amount of content for each point. I don't have to explain in great detail how I get to the point, it's already spelled out and I can get to explaining it.

1. Outages suck on any platform. Windows, Linux, don't matter. When the system goes down, regardless of the reason, its always like cockroaches when the lights turn on -- everyone darts around wildly until someone gets an answer ...or the lights go out.

2. If you know nothing about linux, doing everything is near impossible without someone who does or one uber well written book or ...ten. Forums, web sites, etc are not very effective when you want to know how to do something linux. There are rare exceptions to this (Nagios website is _very_ nicely documented for instance), but an overwhelming majority of them assume you already know 99% of the problem or issue you are searching for - worse, it doesn't get better when you do have a specific problem. There's so many distros, code bases, dependencies, etc you might as well be kicked out into the Congo with a bag full of money and t-bone steaks pinned to your body.

3. Few Linux admins understand anything about Windows, other than they don't like it. Yes, this is a bold statement, but I've found compounding evidence that it is true. Opinions run rampant and a common approach is "if I don't understand it, it must be doing it wrong" -- which is a total setup for failure. I've learned how to make linux do things ...why can't linux admins do the same in windows? The arguments of changing things, seeing things and modifying every little aspect is completely, utterly a lack of an ability to understand and drive to FIND how one OS does something and 99.999% of the time is unnecessary.

4. The art of asking the right question is still solid gold. See #2. Being able to ask those who have a good idea of how it works is great, knowing how to ask them the right question to get an answer is best. Ask your questions carefully and fully. Listen a lot, speak less.

5. Things that are simple in windows are not in linux. I don't care if every linux admin on the planet says otherwise, it's simply not true. Compiling code, making then making installs is the most bass akwards way of getting something to work that I have ever seen. If I download something and run it, I expect it to install. Linux does not do this very well. Further, the code you download, build and install may not work on the next version of OS or another distro. I won't have to worry about this as much with windows applications ...they typically tell you that up front.

6. Once setup, linux services are easier to replicate than in windows. It might be a pain to get up and running at first, but once its setup -- copy paste that conf file out and you can clone it just about anywhere, quickly and easily and you can do it with just about anything. Windows, that's not always the case.

7. The configuration process is nearly identical for every service in linux. Most services come with a /etc/<service>, /etc/<service>.conf where most of the action occurs. Lots of services are setup this way. Some of them have a check config command of some kind so you can test it before you punt it out there. Also, some of it will hot-replace one instance with another without killing people using your system. Now that's kinda cool.

8. A linux OS version 5 and version 5.3 could mean months ...or years have passed between them. We have been using CentOS 4.4 and 5.0 - current version is 5.3. At first glance, assumption could be that it's only a minor revision from 5 to 5.3. Later I found out it's been 3 years between them, 4.4 was 5+ years. Minor revision numbers aren't.

9. Admins are ignored as much as Devs. Tangent story time. The Army core of engineers told the city of New Orleans years ago that, at best, it was DESIGNED to withstand a category 3 hurricane. When a category 5 hit, everyone when into an uproar that it, the levy, failed. No, wrong, it exceeded its tolerance and therefore should not have been expected to withstand that level of force at all. It's like taking a common home and building it to 20 stories and wondering why the foundation fell apart. The warnings of devs and admins are ignored ...until the category 5 hits. Then its "oh wow, this works so much better". Yep, tried to warn ya!

10. Free versions of stuff still kinda suck. VMware 2, CentOS, and don't get me started about the bajillion other items within the app itself -- a lot of these things suck. No support, spotty "community" following (much less help), limitations because after all, they want you to pay for the big boy version, makes these things not so great. If you want the real toys, pony up and pay for it. Trail versions still suck more, but free has its drawbacks. Cept for nagios ...I'm a fan of nagios ...and PNP4Nagios. Both are free and ain't that bad. Needs a better gui though.

11. Mind stretching is rewarding the most when it's something you know nothing about. Going into this I knew how to download linux ...sometimes and knew next to nothing about it. Now I've got SVN on a home server just for kicks along with it doing some other random junk "just because I can". It's equally nice to know I can do it again whenever and it wasn't a fluke.

To wrap things up - am I a fan of linux? No. Between rpms, dependencies, compiling code that isn't mine, sudo-ing and other silliness (did you know root can be called reut?), it isn't my first choice. Does it have a place? Sure. I would use linux at home to monitor sensors all over my house but I wouldn't use it in something that requires a living application, by that I mean something that's going to move constantly and change it's "shape". It was interesting to work on and downright annoying at others, but I'm glad I did - "it's just another club in the bag".

Return to Area 51 ...or telnet into a cisco

jeriley.nospam@nospam.gmail.com (Jesse) — Thu, 21 May 2009 08:21:00 -1000

Having a wide range of experience like I have, you often find yourself back in a land once long forgotten -- today was one of those days. I returned to very low level network land to which I haven't been in over 8 years. We've been having a problem with a port on a switch recently, freaking out and disabling itself. Not just any switch, a cisco 3600 series. I hold cisco's in high regard, and layer 3 switches are pretty high on the uber w00t-tastic scale, so just seeing this error was a new experience all in and of itself. I present the faulty port.

Faulty eh? Take a look at the port stats and sure enough, there's a BOAT LOAD of errors, more than I've ever seen. A "rule" is 1% is considered excessive, and we are hitting 1.2% solidly. Checking out the receive detail, getting a lot of FCS (meaning CRC didn't check out) along with Undersize errors (packets are not the right size) -- so what does this mean? Nothing specific, it's just info, and in this case, it is incomplete. Off I went onto the switch, into the telnet console and wanted to know what IT was saying about this. Sometimes the graphical interfaces don't tell the whole story ...this is one of those times. I had to really dig into the archives to remember how to get into a cisco the way I wanted (logged in, config mode) so I ran show interface gi0/3 (show me interface gigabit bank 0 port 3) and I get some hotness ...

GigabitEthernet0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is <removed>
Description: <removed>
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 254/255, txload 13/255, rxload 4/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:24, output 00:00:00, output hang never
Last clearing of "show interface" counters 5d22h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1912000 bits/sec, 688 packets/sec
5 minute output rate 5479000 bits/sec, 755 packets/sec
     135263414 packets input, 2818426259 bytes, 0 no buffer
     Received 174958 broadcasts (0 multicast)
     0 runts, 81 giants, 0 throttles
     1254363 input errors, 1254282 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 8537 multicast, 0 pause input
     0 input packets with dribble condition detected
     153425270 packets output, 896631553 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Again, reconfirms CRC errors but notice no frame errors. This is a big huge hint - it means the size of the packet is OK but the packet itself isn't checking out. This leads me to believe its a cable that's run too long or unshielded. Cables are susceptible to EM, electromagnetic interference and every wire is an antenna, period, I don't care how long, if its got ferris metal in it, it's an antenna. I put money on unshielded.

Security - part 4 of X - Policy.

jeriley.nospam@nospam.gmail.com (Jesse) — Wed, 29 Apr 2009 23:15:00 -1000

Ah, policy. The magic document that says how things should be done. I love them the most when they haven't been reviewed in 6 months. Why? The world wasn't the same 6 months ago: New laws, new trends, ideas, methods, products, people (remember them?), all of them are changing in 6 months. Why am I picking on 6 months? I just made it up - if your shop is more dynamic, lean and fast paced, it might even be shorter. Most major companies don't change them for years, good companies review them often. Notice I didn't say change?

Policies are great when they are simple, clear and most importantly, understandable. Years ago, Steve Riley (no relation) gave a talk about asking for the rules governing a firewall and received 2 reams of paper. No surprise, it wasn't effective. As he put it "an exception, to the exception, to the exception" let traffic in the network that never should've been allowed. Due to its complexity, no one knew what the firewall was doing. This should serve as the initial indication that it's time to review the policy, simplify it and put it into practice.

This is totally my opinion, but policy should have solid, clear reason behind it and every item within it. Keeping company morale isn't one of them. Why? Morale doesn't mean a thing if the business goes belly up because someone let their kid download porn and the media finds out a system with customer data was compromised by a big DUH. Ok, so it's not always that drastic (is it?), but its not the driving force, the business IS.

A while back when I was running a terminal service cluster, we disabled flash. Why? Some sites spiked the cpu load to high values and since we couldn't control where they went (business driven), we just controlled the flash. No problem, flash disabled, performance remained consistent for all users instantly. Within 24 hours while walking though the halls I was stopped by one of our TS users and asked, and I quote "Hey, why doesn't mountain dew dot com come up anymore?" to which I explained our policy (it slows the system down, shared resources, etc) and I was amazed by the response : "well, we need to turn it back on, I need to get to mountain dew dot com, this is unacceptable." and for the record, this was a municipal govt agency. This user went to their supervisor, and I use that term loosely, and I was told to "just turn it back on". Yes, I'm serious, and no, the policy stayed so long as there wasn't "acceptable, business driven reason" -- news flash, this will not earn you popularity points and be prepared to defend it.

This comes full circle when any policy is challenged and it SHOULD be challenged. If there IS a business reason, the policy will need to be changed. Ultimately it's the business that makes the policy, more specifically, those in power. Without their buy in, there's no weight. Without weight, it won't be enforced.

So how do you know when there's a policy problem? One clear indication is when you hear "it's always been done that way"; the translation is "we don't know why, the process sucks but its what we're use to." For a slightly humorous look on how this happens, check out the Five Monkeys experiment (not sure if it was ever conducted, but either way, it tells the story well). That should be a green light to start ripping into it and finding out WHY it's done that way.

So what's your company policy on access? On disasters? On hiring? On firing? On lay offs? On sensitive documents? On system usage? ...you do have one right? Does it still apply and make sense?

Production Outage Planning - 10 (or so) points

jeriley.nospam@nospam.gmail.com (Jesse) — Fri, 27 Mar 2009 02:51:00 -1000

I've recently went through a number of production planned outages of a group of systems we've recently taken over. I like these outages because of that magical word planned. This isn't planned like you put it in your planner and write down a time, no this is a concrete list with no surprises, no unclear roles, everything is laid out and everyone knows what they are doing. This doesn't seem hard and it really isn't, just takes some attention. Being my 2nd or 3rd one on this current project and I'm noticing some good stuff and possible fail routes that can easily be avoided.

0. Know everything you can about the systems humanly possible. If you have never restarted server X, how do you know it will come back and operate as you expect? Are services set to start manually? How do you know? "It should" doesn't cut it. Worse, what happens if it does not come back, at all? Do you have another system to replace it? How do you direct traffic to that server? Does the backup server work? Does it work completely and 100%? Do you have access to stand up another server from nothing if you need to? These are absolutely necessary and can cause you to invoke the UpdateResume(); method -- and quite frankly, this is the scariest part. No documentation is your enemy here.

1. Solid communication among the team is a total, 100% must. It only takes one parasitic team member can destroy this whole process and render the outage a total failure. Ask other team members "what do you think" if you feel certain members are feeling uncomfortable or one person is doing all the talking. There should be a lot of people jumping in and commenting, confirming, asking "what if". Talk it out as much as possible.

2. Everyone's role is the same as it is day to day. This means your DBA talks about risks and procedures to the database, the devs talk about the code and the risks to the app, etc. This doesn't necessarily mean there isn't cross talk but it does mean the DBA should _NOT_ outline a code deployment.

3. Include at least one of every role on you project, even if they're not directly involved in the deployment. Just because your developer isn't pushing any code doesn't mean they should not be aware that you are changing a script to start a particular service that could affect something the dev team is doing down the road, along with mimicking the internal systems to production. Do not forget QA if there is a code push of any kind; they should be involved in every outage as well.

4. Run though the plan, outloud, multiple times, in the middle of the room. This goes back to number 3 and if you forget to include someone you didn't know you should've -- sometimes a BA will ask "hey will this have any impact on <some obscure system you didn't know about that will be totally screwed when you do this>?" This happens almost every time in the beginning, as time goes on, the question goes the other direction and it's the sys admin or devs or DBA asking the BA.

5. Ask the tough questions and don't rush it. If one team member responsible for system X and can't explain why doing a certain upgrade, setting change, etc will or will not be beneficial, then that is not to be included in the plan. On the same though, allow that person to figure out why and/or verify what they believe. If that means putting it off completely, then so be it, don't force an answer that you want to hear -- expect and ask for real evidence, something that says "THIS is why we are doing this". On the same page, do NOT be rushed into an outage because it will be nothing but pain. Push back and be clear about WHY.

6. Have a failover/rollback plan. Code breaks, servers fail, systems stop working -- it happens, so plan for it. Backup/zip/move/copy what you are changing out -- and that means onto a totally different system and understand how to get it back to where it was. Make sure you've tested this type of rollback as well. Just know "how to do it" is very different from "I know it will work". Understand the risks, be realistic about their impact and prepare for it to all go wrong.

7. Pre-outage steps are solid gold! Do everything you can early to make it easier. If code can be copied up into a pre-determined directory, do it. If a config file can be set aside and ready, do it. Verify the outage window everyone's thinking about has been given the rubber stamp. Verify everyone has access, etc. Pre-steps can save you a ton of time and make your outage that much cleaner.

8. Make a timeline and start from the ends. Say your window is 6 hours. It'll take you 5 hours to complete your task, you've got one hour extra. Wrong, if it takes 3 hours to roll back your changes. Start at the ends. If it'll take you 30 minutes to shutdown the systems and 3 hours to roll back, you've only got 2.5 hours to do your work. Make your timelines from the ends and work toward the middle. This will save much pain and suffering.

9. Execute the plan and nothing more. It's ok for the devs to say "well if we can just change this while we’re at it" bring down the hammer immediately. It's too late and not part of the plan, BUT it does mean it can go into the next one. Have them write it down and include it on the next process and stand your ground.

9b. Have alternate ways to do the same thing. Sometimes reindexing a database takes 2 hours, others it takes 15 minutes. Consider service shutdowns and "cleaner" ways of running. This will increase your risk, but sometime save an outage from being completely worthless fail whale waste of time to a total success (been there, done that, got the t-shirt).

10. Do a post-opt review. What did you learn? What surprises did you get and why? Can they be stopped from happening again? How long did the whole thing take? Why did it take longer, shorter than you expected? What can you do differently? These answers will make subsequent plans far more refined and get you that much closer to perfection.

11. Catch up on sleep. Ok, this one is more of a reminder for myself to cash in on weekends and time that I don't spend on outages to restock on sleep. Make no mistake; being up at midnight to 5am and into work at 8:30am isn't glorious by any means, but the rewards will come when you return back to a normal day and maybe even leave early a few times.

I'm sure that if you follow these, your plans will be utterly painful at first but will get better over time. It's not fun, easy nor exciting, but it's a good, solid way to make sure you and your client are not surprised (or at least it's kept to a minimum).

Bringing HBA SAN and LUNs together with .Net

jeriley.nospam@nospam.gmail.com (Jesse) — Wed, 11 Mar 2009 05:33:00 -1000

After writing my last post on tracing the two items together, the magical lazy comment came about to -- there has to be a better way. Of course there is, no one in their right mind would even THINK to do this by hand time and time again. So I went hunting for a way to pull this info into something useful, like an application of some kind. Sure enough, there is an API out there for SSH for .net called SharpSSH and it works, with a bit of thinking.

I approached this problem with the idea being simple - if I can get the result from a console into a string, I can do whatever I want with it from there. One of the commands does exactly that -- SshExec.RunCommand(string). Thanks to my ~~documentation~~ blog post, I know everything I need to do, now to work the string values when they come back.

One of the commands I use quite frequently is more. So my first attempts with SharpSSH was to use more because it was a sure fire way to get everything, except for one problem. If you've ever used more at least twice, there's a condition that exists when a screen is smaller than the requested information, "pages" are created so you can read parts at a time. This poses a serious issue since using this method, I can't control this. I COULD have executed a space command if it held out longer than, say 5 seconds but that seemed a bit cumbersome for what I wanted. After all, most of what I need is within 100 lines or so. Knowing this, I decided to go with tail -1000.

First, you have to create a connection and authenticate. SharpSSH makes this rather simple by doing the following...

SshExec exec = new SshExec(server, user, pass);
exec.Connect();
if (exec.Connected)
{ }

So far, stupid easy. Next, you do the exec.RunCommand(string) as such...

string scsiInfo = exec.RunCommand(@"tail -1000 " + procScsiLocation);

if (string.IsNullOrEmpty(scsiInfo))
{ }

Here, I simply ask for the /proc/scsi/<adapter> info. Since I'm working with servers that are all the same, I made these static but this could be pulled back with a simple ls on /proc/scsi/ command and a selection could be made, probably with radio buttons. The result of this can be ugly, but each line is terminated with a \n which makes it very predictable and easily split into a list of strings (my favorite).

List<string> scsiInfoList = new List<string>(Regex.Split(scsiInfo, "\n"));

I used Regex.Split instead of String.Split strictly based on personal preference. With this newly created list, I can create test conditions that filter out the junk I don't want and append to a string builder what I DO what. Feeding the result into a simple multiline textbox makes this easy to read. From there, you can do just about everything you need to do.

Tracing volume groups with an HBA in linux

jeriley.nospam@nospam.gmail.com (Jesse) — Mon, 09 Mar 2009 09:32:00 -1000

I've been off in Linux land for a while and I've ran into HBAs, fiber channels, SANs and volume groups. Honestly, they're a billion times more complex than they EVER need to be but if you are unlucky enough to run across such things, here's how I did it, do it and will continue to find it was the professor in the library with the candlestick.

First, log into your linux box and head over to /proc/scsi. In here you should see a folder with a semi-recognizable driver, in this case a qlogic 2000.

[root@mybox scsi]# ls
device_info qla2xxx scsi sg

If you want to know EXACTLY what card is installed, a lspci -v will display this for you ... in this case, ISP2312.

10:01.0 Fibre Channel: QLogic Corp. ISP2312-based 2Gb Fibre Channel to PCI-X HBA (rev 02)
        Subsystem: QLogic Corp. Unknown device 0100
        Flags: bus master, 66MHz, medium devsel, latency 128, IRQ 130
        I/O ports at 5000 [size=256]
        Memory at fdff0000 (64-bit, non-prefetchable) [size=4K]
        [virtual] Expansion ROM at d1000000 [disabled] [size=128K]
        Capabilities: [44] Power Management version 2
        Capabilities: [4c] PCI-X non-bridge device
        Capabilities: [54] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
        Capabilities: [64] #06 [0080]

So anyway, jump into your HBA's driver directory and you'll find a file or two, probably just numbers, 0, 1, 2, etc. In my case, I have 1 and 2. Run a more 1 on this and you'll get a dump of information similar to this. I've snipped the top because it's irrelevant.

...

SCSI Device Information:
scsi-qla0-adapter-node=200000145e249594;
scsi-qla0-adapter-port=210000145e249594;
scsi-qla0-target-0=5000402101fc137a;
scsi-qla0-target-1=5000402201fc137a;
scsi-qla0-target-2=5000402001fc137a;
scsi-qla0-target-3=5000402301fc137a;

FC Port Information:
scsi-qla0-port-0=2001000402fc137a:5000402101fc137a:010000:81;
scsi-qla0-port-1=2101000402fc137a:5000402201fc137a:010400:82;
scsi-qla0-port-2=2001000402fc137a:5000402001fc137a:020400:83;
scsi-qla0-port-3=2101000402fc137a:5000402301fc137a:020000:84;
scsi-qla0-port-4=200000145e249595:210000145e249595:020600:1;

SCSI LUN Information:
(Id:Lun) * - indicates lun is not registered with the OS.
( 0: 0): Total reqs 3, Pending reqs 0, flags 0x0*, 0:0:81 00
( 0: 8): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:81 00
( 0:15): Total reqs 80236, Pending reqs 0, flags 0x0, 0:0:81 00
( 0:17): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:81 00
( 0:21): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:81 00
( 0:25): Total reqs 80253, Pending reqs 0, flags 0x0, 0:0:81 00
( 1: 0): Total reqs 3, Pending reqs 0, flags 0x0*, 0:0:82 00
( 1: 1): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:82 00
( 1:11): Total reqs 80236, Pending reqs 0, flags 0x0, 0:0:82 00
( 1:19): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:82 00
( 1:21): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:82 00
( 1:25): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:82 00
( 2: 0): Total reqs 3, Pending reqs 0, flags 0x0*, 0:0:83 00
( 2: 8): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:83 00
( 2:15): Total reqs 80257, Pending reqs 0, flags 0x0, 0:0:83 00
( 2:17): Total reqs 80253, Pending reqs 0, flags 0x0, 0:0:83 00
( 2:21): Total reqs 80256, Pending reqs 0, flags 0x0, 0:0:83 00
( 2:25): Total reqs 80262, Pending reqs 0, flags 0x0, 0:0:83 00
( 3: 0): Total reqs 3, Pending reqs 0, flags 0x0*, 0:0:84 00
( 3: 1): Total reqs 80312, Pending reqs 0, flags 0x0, 0:0:84 00
( 3:11): Total reqs 80236, Pending reqs 0, flags 0x0, 0:0:84 00
( 3:19): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:84 00
( 3:21): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:84 00
( 3:25): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:84 00

This tells us a bunch of things, useful things we'll need later. Most importantly the adapter-node/adapter-port IDs that I have bolded. This is the ID we need to find out WHAT exactly this server has access to. Out on the SAN, find this information and which volumes the ID has access to. Those volumes should have a serial number of some kind which is the next step. This part I can't help you with because every SAN is different, but I can tell you the SAN I am working with gave me an ID of 603BD80A that is 1000.0GB -- according to my linux server, I don't have one of those volumes mounted of that size, a df -h tells me this. So what the hell? There's a volume group that's combined multiple volumes into 1, hints the name v that happens to be over 1000GB, as vgdisplay tells me so...

[root@mybox ~]# vgdisplay
--- Volume group ---
VG Name               datagroup
System ID
Format                lvm2
Metadata Areas        4
Metadata Sequence No 8
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               1
Max PV                0
Cur PV                4
Act PV                4
VG Size               2.73 TB
PE Size               4.00 MB
Total PE              715292
Alloc PE / Size       715292 / 2.73 TB
Free PE / Size       0 / 0
VG UUID               pY6jLp-rZcZ-MVQS-BKH2-XmHx-xJhb-5BpxSa

Ok, so what? I haven't proved anything, and worse, what if there's multiple groups that have bigger than 1000GB? Next step ... use that serial number we got earlier off our SAN to determine WHAT "drives" are what. Time to hit the multipath -ll ...and look for our serial number

mpath13 (36000402001fc137a603bd80a00000000)
[size=931 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=1][active]
\_ 2:0:2:25 sdaj 66:48 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 1:0:0:25 sdf 8:80 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 1:0:2:25 sdp 8:240 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 2:0:0:25 sdz 65:144 [active][ready]

Sweet, we got a hit and its the right size. So multipath13 is really sdaj, sdf, sdp and sdz. Ok, so what? Call on the powers of pvdisplay to tell you where it's really living...

[root@mybox ~]# pvdisplay /dev/sdaj
--- Physical volume ---
PV Name               /dev/dm-15
VG Name               datagroup
PV Size               931.39 GB / not usable 0
Allocatable           yes (but full)
PE Size (KByte)       4096
Total PE              238437
Free PE               0
Allocated PE          238437
PV UUID               EUt6gs-8AIF-sAXe-q2MH-3L27-v57p-P6aESM

There you have it, it IS in fact living inside your datagroup volume group -- but what about the other drives sdf, sdp and sdz? Do another pvdisplay on THOSE and notice the PV UUIDs are the same...

[root@mybox ~]# pvdisplay /dev/sdf
--- Physical volume ---
PV Name               /dev/dm-15
VG Name               datagroup
PV Size               931.39 GB / not usable 0
Allocatable           yes (but full)
PE Size (KByte)       4096
Total PE              238437
Free PE               0
Allocated PE          238437
PV UUID               EUt6gs-8AIF-sAXe-q2MH-3L27-v57p-P6aESM

This come in very handy when doing an IO check and noticing a high state of wait on a particular drive and pvdisplay works on those as well. In order to get the whole volume group, we would have to trace back the rest of them to complete the picture to get the full 3000GB assigned on the SAN.

Problems you can't test for

jeriley.nospam@nospam.gmail.com (Jesse) — Mon, 02 Mar 2009 21:23:00 -1000

Right now I'm waiting on a QA person to verify the code push that just happened, which seems like no big deal ...until I mention that half the team was unable to connect in for most of the time I've come to find out that things happen that you NEVER see coming and I'm talking about really odd ball things you can NEVER plan for. This is a perfect example.

This evening, after making contact with everyone involved, I disabled the load balancer and let the others know its time to do their thing. One at a time, each server went down, files were being copied, everything was as it should be, noooo problems ...until about 1 hour into it, I lost my VPN connection. Not good. Why? Last time this happened it was (still is) a huge fire causing all sorts of pain and suffering and it happened right @ 1am on the dot. I message the other two connected in and they report no problems. About that time I ask the 4th person -- they lost connection about the same time I did. Long story short, road runner must have been doing and outage around the same time we were. All DNS requests resolved, but IP routes fell flat. Trace routes did their usual hop a few then disappear (no ping reply). The other two (At&t / WOW) had no problems. After about an hour and a half, we were back up.

So how do you plan for this? Indirectly, you do, by having multiple people in multiple places if at all possible. If you're all at work and that has an issue, it would've been a fire sale and people would've been freaking out (rightfully so) -- your app is down in a remote location, you just disabled your network gear and there's no way to get it back quickly other than to get the connection you just lost, back. At the same time, because we had multiple networks at our disposal, we were able to nail down that indeed it WAS road runner and not the production systems perceived to be lost. Imagine the conversations about what really happened ...and what the REAL answer was?

Security - Part 3 of X

jeriley.nospam@nospam.gmail.com (Jesse) — Thu, 11 Dec 2008 06:31:00 -1000

In the first two parts of this covered the idea of security and physical security. We now move into access security. Access security is a highly religious topic, bent with emotions, misinformation, egos, principals and typically (worthless) corporate policy so prepare for battle on this one if you dare tackle it. Let's dive right into this using our previous example of an office building and I'm looking for a port to plug into the network. How long will that take? Virtually no time, they're everywhere, which leads me to my first not-really bold statement of this post.

Network access cannot be controlled. If you haven't heard this yet, let this be the end all. Under no circumstance can any company in today's world do business without having network access available to the world, period. Clients, employees, your own staff need access to the net at a moments notice for a variety of business reasons (or not so business reasons) and the network should not get in the way - costs too much money and overhead. Generally speaking, the network should never be seen nor heard but "just there". So accepting this, the next issue is anything device attached to your network can't be trusted, regardless if it actually has your policy on it ...at least initially. Taking the same mindset from the previous post of doing "what if" with a slight tweak, we're going to add "and I wanted to affect <blank>". We're going to say the network policy is basic - some sites are blocked, most ports are blocked except the usuals (http, smtp, some other services) and we ask "What if I came in with a laptop/pda/some-wireless-device and plugged it into the network and wanted to affect the entire network?"

That's a hugely vague question isn't it? The first one that comes to my mind is network speed -- why? It's fairly easy to flood the bejesus out of a network with multicast packets (refereed to as a multicast/broadcast storm). I did this once, by total accident while doing a norton ghosting -- meant to select unicast and missed, but it pointed to some flaws in network management, which brings up another point that requires a slight tangent -- not all attacks are intentional. Some of them are completely, totally innocent. They might be stupid, or misinformed or just downright unknowing, but sometimes it happens and a well protected system will resist these accidental attacks as well. So, back to the multicast storm - I've started one and tons of junk data is now flooding the network at an amazing rate, taking the system down to a crawl (think 14.4 modem speed). This fix is easy - simple flood control turned on will limit the effect if not eliminate it (depends on size of the network, infrastructure, etc).

Ok, cheap shot, that "hole" is plugged, what's my next vector? Well, I'd would like a login, a username and pass, maybe some network share info so I launch a man in the middle attack and get some unsuspecting user (someone that comes in late and hasn't logged in is ideal) and get some hash values, sids and a username, run my favorite password cracker and gain some entry. Sound hard? Nope, I've done it before as a demo, proof of concept and had a secretaries password busted in under 3 minutes. Even worse, recent advances in hardware and of all things, gaming video cards have made it possible to crack that tougher password MUCH faster. This is where a decision would have to be made in regards to what you are protecting. Should you enforce long, complex passwords? Should you bump your domain to NTLMv2 or kerbrose? Those are decisions that will have to be weighed and decided per the requirements (cost vs risk) but lets say you do kerbrose but not long passwords so that should protect them a bit more.

Now the passwords are heavy on the encryption (not to be confused with hashing), I can't flood the network anymore, now what? Oh look, a pc with a hard drive -- let's clone it! If this doesn't make you wince in pain, I don't know what will. Notice I didn't say steal it, I said clone. I'm more interested in the cache of users and passwords on the drive than anything else first and foremost. The random slew of documents and other junk on the drive is a nice bonus, maybe I can find other apps to attack (we'll get to that later) but for now, I want a copy to see what I can see. Now, since I'm downright evil, I want to change an HR document and toss it back on the network that clearly has promises that are way out of line (you get a corvette for your 5 year anniversary).

Data's now on a server somewhere, encrypted, so changing stupid stuff isn't easy ...how about a keylogger like I mentioned in the other posts? I'm going to stop here because I think the point is made -- they're impractical to control in most situations. Enforcing long passwords is known to increase your helpdesk tickets (see : expensive) -- talk about a hassle but you should get the idea by now. Their ARE many ways to fix these problems with quarantine networks, 802.11x authentications, IPSec (a GREAT IDEA), radius and others. If you've never heard of these, look into them, now. Next post, I want to talk about the touchy subject of policies and how I feel they should be approached.

Security - part 2 of X

jeriley.nospam@nospam.gmail.com (Jesse) — Sun, 07 Dec 2008 23:21:00 -1000

In part 2 of X, I'm going to cover the aspects of physical security. This doesn't seem like rocket science, but again, I'm surprised how the same breaches happen again and again so it's necessary to cover it. Physical security is often overlooked. If a bad guy has access to the server room, network gear, etc, he owns it, not you. The biggest, baddest security is worthless if it's taken out by pulling a power cord -- game over and time to update the resume.

Personally, I think physical security is the easiest to figure out. Look around a room, figure out the logistics of people coming and going and put something in between it. Also, consider the ifs - "If I made it here, what would be my next easy target". Sticking with the previous post's style, we're going to create a server room to help illustrate the various aspects of this particular topic. Also, get into the mindset of being downright evil and I mean EVIL. It will help you expose real problems that could be easy to protect and its fun.

Let's say the server room is on the 3rd floor of an office building. It's a real room, has a door with a lock, an AC unit for environmental control and 50 servers of various kinds. Let's start with the first if. "If I wanted to get in, how would I start?" By simply walking in the front door. I know, lame, but if there's nothing providing access control to the whole building, getting in is the easiest part. Now there's a guard. The guard doesn't necessarily control who comes in when, but we'll come back to the guard's role as we move though this. Now I move to the server room.

"If I got to the server room, what would be my next easy target?" The ceiling. Surprised? How many office buildings have you seen that have ~~false~~ free hanging ceilings? The first thing I would do is look around for an adjoining office, jump up on the desk, push up some tiles and hop up and over - fancy locks and doors are now worthless. Also, if the walls are weak, ie, made of drywall, how long would it take to punch though that? Seconds. The door itself - could it take a brute force kick?

Notice I didn't talk about locks. Why? As hard as it was to get my head around it, I do not consider locks a security piece. Many companies would have you believe if you have a lock on something, that's it, no one can get into it which is dead wrong. At best, locks are a delayed access control device, nothing more. Example : if I put a big, $500 lock on a door made of 3/16" plywood, I'm not attacking the lock, I'm attacking the door. Game over. Removing the idea of a lock making something "secure" helps in this thought process.

Now, moving on into the server room itself. "If I made it into the server room, what would be my next easy target" -- this one is stupid easy, I'm looking for incredible impact with the least amount of effort - the network cables. Most companies have their network cables funneling into the room in one BIG pipe - cut that and I win! No level of documentation will save you on this one because that bundle of cables goes to 500 different locations all over the building and you shouldn't (can't) splice the wires back together or worse, fiber, so now what? Told you to get into an evil mindset! Next I'm looking at the power, another big win if I knock it out. A switch, a panel, something that I can blow to kingdom come (not just throw the switch, I want that puppy to fry, spark and smoke - real damage). What about the AC (or now more commonly called the EU, environmental unit)? If I can disable that in some way, especially on a long weekend in July, it'll be the gift that keeps on giving! How quickly would YOU notice?

Ok, I'm having fun wrecking a server room, but let's say all those are protected. The power/network wires are under the floor, the servers are the only thing I really have access to. Regardless if the servers are in a rack or otherwise sitting out, if I can get to the cables going INTO them, this opens a whole new list of problems. Many KVMs have an input and 4, 8, 16 outputs. Now I install a keylogger device that transmits wirelessly, game over. I can also pull cables at random and take out other systems one by one (or with one big pull).

The servers are all in enclosed racks, no wires outside at all, I can see the lights blinking, and thats it. Ok, fine, lets just take the whole thing! Roll that puppy out the door! Most companies don't assemble their racks, so it's going to fit rolling out the door. Oh, and nevermind the wires your ripping up while you do it, maybe it'll be just intertwined enough to pull other stuff out as you go (or just cut it). At this point, the guard comes into play. Without that guard, no one would think twice, much less say anything but any (good) guard would stop that in a heartbeat. Easier fix for this is just bolt the thing to the floor in some manner, most racks come with a kit. If not, build your own. No really.

Ok, now what? I can't get into the server room because there's concrete walls to the ceiling, a nice steel door with no visible lock and it wouldn't matter because the servers are locked down tighter than the outside. I can't get to the wires, I can't get to anything inside that room. Ok, so let's plug our laptop into the network and see what we can find -- which I'll talk more about in the next posting.

Hopefully this has got you thinking. Granted yes, some of the things covered here are a bit extreme but not unrealistic. I'm sure some are thinking "that person would be caught and throw in prison forever! No one is stupid enough to do that" -- I beg to differ and ultimately, if someone does something amazingly destructive to your companies ability to be productive, it doesn't make ANY difference what happens to the person that did it, none, zero because it's already done and money is already being lost.

Cost wise, with the exception of the guard and the true floor-to-ceiling walls, none of them are all that expensive. Most important, consider your needs. You may not need the 5000$ RFID system where a simple key-code door will do. That will be left to cost analysis and ~~religion~~ policy.

Security - part 1 of X

jeriley.nospam@nospam.gmail.com (Jesse) — Fri, 05 Dec 2008 06:07:00 -1000

In this world we live in today, there are many issues that hit in the security realm but I'm astonished at how the same issues keep coming up so I feel it's necessary to go over them as best I can. Some things may shock you, freak you out, piss you off and generally make you think I'm full of it. That's ok, I welcome it!

When I first started writing this, I realized there's a lot to to cover, so I am going to make this a multi-part posting. It won't be geek-cool in a lot of places, nor will it be super exciting but informative and thought provoking. I will attempt to the best I can to cover points, counter points, countermeasures and realistic expectations. Some of them will be overlapping, cross referencing, contradicting and downright confusing but stick with it, at the end it should make sense ...sort of. So let's drive right into it.

Security is really risk management, not necessarily protection or what we typically think of as being safe. It seems backwards, but it's true. For this to make any bit of sense, I'm going to use a house as an example, your house. Look around your house and think about what keeps people out, where they keep people out and how it keeps those people out. How does it let people in?

Quick observations would show a front door that has a lock, maybe even a deadbolt, windows with locks, some lights for the front, perhaps motion activated and maybe a garage door with internal access. Those are the obvious, but what about the less obvious? How about dogs (detectors), a safe (heavy protection), internal door locks (light protection)? What is even less obvious ...what about you (security policy) and anyone that lives in the house? There's a lot of things going on and I haven't even got to the point where someone wants in! Now, say someone comes up to the front door, knocks (makes a request) -- does that person get in immediately? The first reaction is "no! of course not!" -- What about big gatherings during the summer say a graduation or retirement? Enter the all mighty "It depends".

"It depends" will get used often, but for good reason - it refers to the policy and the decisions based on that policy. We all have them in some capacity but we don't think about it much, it just happens. For example: grandma comes over. Grandma knocks or you see her coming, you open the door, let her in and she is now inside. Is grandma now part of the internal policy? YES! How? Grandma can let people in, open/unlock doors, windows, and overall apply her own policy. Now, I don't mean to pick on grandma, but if someone comes to the front door with a big smile and says they know you or your family, do you think grandma will be the first to say "no!" ...maybe not, and that could be a problem depending on who it is. Instead, if grandma goes to answer the door, without thinking you policy her policy -- "Who is it grandma?" and await a reply (request) of who it is. If that person is unknown or something isn't quite right, you go and check it out (authenticate). This also varies with the type of neighborhood, where you grew up, where grandma grew up and so on. If you live in a downtown area, more likely than not you'll look out your peep hole before you open the door whereas if you live out in the country, you might just yell "come in!".

I'm going to take this to the extreme. Let's say the house is "high" security. Cameras, bullet proof glass, steel reinforced doors, heavily armed guards (with guard dogs!), iron gates, laser turrets with a large open, unobstructed lawn surrounding the house (seriously, you didn't think that was just for looks did you?) -- and grandma comes over. In and of itself, grandma's way of getting in is the same -- a request is made and once inside she can apply her own policy, albiet less dramatic, still can cause problems.

Same goes for every type of security. Security in this context does not mean protection necessarily it means management of realistic risk. Take the first example -- most people feel perfectly comfortable and "safe" inside their home, as they should, but does that -really- stop someone from entering? NO! The second example with the extreme (insane) security doesn't either. It's the amount of effort required for someone unauthorized to enter and ~~wreak havok~~ come for a visit.

"How do you figure? You mean to tell me that my family isn't as important as someone in a stupid crazy high security compound?" NO! and this is where it gets really interesting, at least I think so anyway. I'm going to use another set of examples. First up, a business of 50 employees. They are in the (yep, I'm goin there) widget business. They're not big enough to have a full IT staff, and more importantly, it isn't their business, but they do just fine. Another business, Flashing 12 Inc has 50,000 employees across 24 countries and has a full, robust IT staff but like Widget Express, they're not in the business of IT. Each has a laptop stolen with "personal information" -- by law, this must be reported and both companies report it stolen. Which company is effected more? Widget Express. As a matter of fact, because of the stolen laptop, the company goes bankrupt from all the bad press, lost business and cost to cover the breach - the other company can.

So what am I getting at? Widget Express can't handle a major disruption, much like many families cannot handle someone coming in and stealing a bunch of their stuff. Am I saying that every family should have a laser turret out front? As cool as that would be it's not practical; the cost does not justify the potential loss. What can (should) be done is handle the areas that make sense. Example: a lock on the front door, first floor windows are locked, lights are on and so on. Easy, cost effective things that will clearly make it more difficult or even better make enough noise to take notice. Most importantly they must be maintained -- having the 2nd house but leaving the tunnel outside of the compound open makes all those fancy, flashy things WORTHLESS.

In the next post, I hope to break this down into smaller pieces and look at the aspects of security - risk management, potential loss, and a bunch of other stuff regarding the more physical aspect of security.

Dropdown lists with the same value, different text

jeriley.nospam@nospam.gmail.com (Jesse) — Mon, 01 Dec 2008 05:00:00 -1000

I've been working on some calculators that are rather heavy on the number side, but because of the nature of what these things are, there's some issues that totally surprised me. If you have two values in a dropdown and do a postback, unexpected things happen.

On a page, paste this into the source...

<asp:DropDownList ID="DropDownList1" runat="server" AutoPostBack="true" >
    <asp:ListItem Value="3">ItemA</asp:ListItem>
    <asp:ListItem Value="4">ItemB</asp:ListItem>
    <asp:ListItem Value="3">ItemC</asp:ListItem>
    <asp:ListItem Value="4">ItemD</asp:ListItem>
    <asp:ListItem Value="9">ItemE</asp:ListItem>
</asp:DropDownList>

Load it up and watch what happens (select ItemA, then ItemB, ItemC, ItemD, ItemE) and note the behavior. This is because (as I found out) .net keeps track of it in viewstate by value, not index, text or anything else (as demonstrated).

"Why is this a problem? When would you have two values the same, that'll never happen!" - wrong! In front of me is an xml doc with pressure values on it for the project I'm working on -- guess what, the pressure for itemC is the same as itemW, but the names are very different so this is a real world, totally practical reason why this should not work this way.

So how do you fix it? Great question. If I ever come up with a good one, I'll post up an answer ...unless someone else does? I have a feeling it will be saving the index, the value and the text into viewstate or a combination thereof by using an inherited control. Just a thought.

NFL Live is a joke

jeriley.nospam@nospam.gmail.com (Jesse) — Thu, 20 Nov 2008 15:23:00 -1000

I'm sitting here, right now, watching the horrible service known as "NFL LIVE". They advertise it for sprint "nfl mobile live" and it has made me a firm believer in NEVER buying the service, ever.

On paper, its a good deal - they broadcast the game online. But what they don't tell you is they don't let you watch much of the game, at all. What do I mean? I've been watching the Bengals/Steelers game for an entire half. I've seen maybe a total of 10 plays. At first I thought it was an injury time out or they were doing something else to kill time -- no, they're not. It's terrible!! What's worse, the picture quality leaves much to be desired. More so, the audio quality sounds like I'm in a sewer pipe.

They want someone to pay for this junk? I'm sorry guys, go back to the drawing board and start alllllll over.