Changing your disk scheduler on a Linux virtual machine to increase performance.

First some background of what we do with storage at ipHouse in our VMware environments.

We really like NFS. Architecturally it’s simpler than block based storage; you just need a good local area network and a storage system that can export a file based protocol. There’s no need for specialized hardware or intelligent host bus adapters, just let the storage array handle the storage. Virtualization lends itself to file based storage quite well. VMDKs are just files after all. I kind of snickered when VMware first came out with their VAAI storage extensions. It seemed, to me, like they were enhancing block-level storage devices to do a lot of what NAS based storage already does.

While I was taking my VCP4 class my colleges, most of whom were from big companies, snickered when I mentioned that our storage was on a NAS. A “filer” for them was a place for document sharing and storage. There was “no way” it would ever be fast enough, or good enough to backend their virtualized infrastructure. I’ve seen that notion fade more and more as ZFS has opened the doors for storage startups; and the big players are fighting back with their own specialized NAS devices. There are some really cool ideas floating around: NAS devices that are scale-out, that are optimized for virtualization, and that can do in-line deduplication of data.

That being said…

I have learned that there are some OS level tweaks that can enhance performance on virtual machines. Most x86 operating systems seem to be optimized for single disks, or internal RAID setups. Understandable as that has traditionally been the bulk of their install base. This means that the OS can manage disk queuing better that the dumb RAID card, or the dumber hard drive. CFQ, the default disk scheduler as of kernel 2.6.18 does this. As I understand it CFQ breaks synchronous read/write requests into queues, and assigns timeslices to each queue, weighted by IO priority. The effect is that higher priority processes get longer queues which keeps IO requests from the same process close together. Great idea when the OS has direct access and is managing the storage. Not so great when the storage is handled remotely; the array on the other side is doing the scheduling. All of that optimization is ostensibly ignored. So for a virtual machine it’s better to switch to a simpler algorithm and let the storage array handle the write queuing.

From my reading (and testing) It’s better to switch to the noop scheduler. Noop simply shoves all requests into a first-in-first-out (FIFO) queue and can merge requests. It is simple, fast, and is great for flash storage (no mechanical latency) or for situations where optimization is handled by another device. Like a NAS! Perfect for virtualization!

I discovered this after getting a snippet of a shell script to try from Mike (who got it from a potential vendor that is a big storage geek). This wasn’t new information as Mike had mentioned this almost 18 months ago in passing but neither he nor myself ever tested it. After giving me the info, again, he suggested that I “test this out, and let me know if it works.”.

I’m still testing it, so caveat emptor, but I thought I’d share it with you.

***WARNING DO NOT DO THIS ON A VM WITH SNAPSHOTS***

#!/bin/sh

grep '' /sys/block/sd*/queue/scheduler
for d in /sys/block/sd*; do
echo noop > $d/queue/scheduler
done
grep '' /sys/block/sd*/queue/scheduler

This switches the scheduler from cfq to noop on all “SCSI” disks in the virtual machine.

He also added the following tweak to increase the read-ahead from 256 sectors to 1000 sectors, which caches more disk data for faster read times, after printing what the OS has mounted.

#!/bin/sh

mount
blockdev --getra /dev/sd?
blockdev --setra 10000 /dev/sd?
blockdev --getra /dev/sd?

Again, I’m still testing this on my personal stuff, but, qualitatively, things feel a lot faster. If anything, I haven’t crashed my Linux systems.

Anyways, I hope that helps!

MinneDemo happens two to three times a year and I think this has to be one of the most exciting MinneStar events (although, I have yet to attend MinneBBQ).

As you enter the area of presentations you feel a great surge of energy from the presenters and the crowd. There is an excellent mix of different age groups that attend. The first hour is for networking, eating, and of course – drinking. Once you have had your fill of food and conversation, you head into a medium-sized auditorium to take your seat, wait, and feel the anticipation build.

All of the presenters have a spark that ignites the entire audience sucking them into the presentation.

What are people presenting at MinneDemo?

Presenters show off real, working technology products created locally in Minnesota. One not only feels energy but a sense of pride in our great state!

I am just going to touch on two of the different presenters that I enjoyed wholeheartedly but before I do that I would like to congratulate Code 42 software, the company behind CrashPlan for landing a $52.5M growth capital investment round.

In case you want to catch the entire show there is video footage of the event and more great links on the video page.

Brahmageddon is an iOS game that is a lot like wack-a-mole but, with a spice of Hindu mythology. The company that makes the game, Bust Out Solutions, is based in Minneapolis, Minnesota. It was quite entertaining to watch the game being played and hearing the Hindu music in the background. The graphics are magical and I would love to meet the artist who created the Hindu demons. The game takes on the same effect that Angry Birds.What I mean by this is it is a fast paced game that is easy to understand with a pinch of challenge to keep the players enthralled. It is quite simple and an easy way to acquire entertainment on the go. I could easily see myself playing this game on the bus ride home from work. The audience for the game seems like it would cover all demographics. It was also encouraging that the creators still find the game quite fun to play.

Red Stamp’s presentation takes the quirky, stylish greeting cards that you would buy at say a Paper Source and puts it into a mobile correspondence. It was a pleasure to watch a company that moved with the times and did not just stay in the paper world. I think Red Stamp represents a beautiful evolution in the greeting card. Red Stamp is really catering to the crowd that just does not have time to stop by a store or order a card and have it shipped their way. I think this crowd of busy people is growing and Red Stamp is growing right along with them. It is refreshing to see a company that instead of saying “no we won’t do that” said “hey, we are up to the challenge and can make this happen”.

If you would like a kick in the butt to remember what dreams you used to have and what it is like to strive to make those dreams come true, you should check out MinneDemo.

And again, if you would like to see all the presentations, check out the video footage.

First, its become common practice for many pages to have at least three stylesheets; one for all media, one for screen-specific instructions, and another for print. Other stylesheets might be imported for specific pages or resources, such as a lightbox library or HTML form suite. Instead of putting each stylesheet existing in a separate file which must be loaded and parsed separately, consider consolidating them as much as possible by using @media directives within the CSS file. For example,


// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
// combined.css
// these styles apply everywhere, all the time

body { color: black; }
h1 { font-size: 150%; font-weight: bold; }

// – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - -
// these styles only apply on-screen

@media screen {
body { background: url(‘background.jpg’); }
div.nav a:hover { font-weight: bold; }
}

// – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - -
// these styles only apply in-print

@media print {
div.nav { display: none; }
p { text-align: justify; }
}

// – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - -

Placing all your stylesheets in the <head> section of your web page is best practice, and also improves apparent performance since the web browser can load and parse the style information before it begins rendering the page body. Associating stylesheets via <link> tags instead of @import is better, since not all browsers handle @import directives promptly.

Also consider using Expires headers to instruct web clients to cache CSS files for a long time. In apache, this is easily achieved through the mod_expires instructions,


ExpiresActive On
ExpiresByType text/css "access plus 1 year"


If you need to retain the ability to change your CSS files more frequently, add a version number to the filename and update it as necessary. For example, combined-1.0.css

I look at many cloud providers and I see the opposite. Their services were designed for expedience instead of permanence. They make it hard and, at times, very expensive to actually keep data around. Usually you have to attach a “disk” (or “volume”) to any machine that has data you want to keep and you have to pay for that privilege. You also better have backups because you have no idea about the underlying storage or data retention policies.

Any data that you absolutely need could mean you’re paying two or three times what you’d expect in order to keep it.

To my hoarder eyes the cloud is one big data furnace. It’s a dangerous place for your information to stay.

Enterprise data storage is expensive. I’ve often joked that virtualization is a scheme to sell storage arrays. It’s a tricky game of performance, space, and redundancy. Disks fail, flash is expensive, you never have enough RAM or CPU. There are dozens of types of arrays for hundreds of applications, retention policies, regulations; it’s a mess! When you have a service that has hundreds of thousands of customers then it may make sense that you discourage persistent data. You want people to consume your resources, pay their bill, and move on. Expedience instead of permanence. I’ve often been asked: Why online storage is so expensive when hard drives are so cheap? Well, this is why.

We built the ipHouse vmForge product with the idea that a virtual data center (VDC) replaces co-located infrastructure. The storage is persistent from the get-go. Is it any wonder that Mike has been loath to call it a ‘cloud service’?

This means that there are severe implications for any storage array that we put in place. We have to make sure that anything we put in place not only performs well but also goes the distance. It’s still a very good idea to do backups, though they probably will not be nearly as large, as most customers just need to back up a few key files or the database dumps that happen regularly. (you are backing up your database, right?)

Well, that’s my opinion anyways. Now I’m going to go back home and work on my basement.

• Centralized management over several to many access-points.
• Unified access policies.
• Ease of deployment.
• Rogue AP scanning for PCI/DSS compliance.

The lesson I learned: Feature freeze is a good thing. Know when to stop fixing.

Now early in the project, I had a pretty good idea of what pieces needed to go together but I did not have a very good idea of how to get there. I put down a quick design and while I was doing that I started to see problems..

• Pieces did not fit together.
• Some things were missing.
• This was not going to work.

Time to start learning. I love learning.

So this is the good part. This is the fun part. This is where things go click. More on that in a moment.

I strive to make every day a boring day on my production servers. That doesn’t mean that they don’t do cool stuff. That doesn’t mean I don’t like my job or that I find it dull. I don’t. I just like to be beyond the point of being surprised, pleasantly or otherwise, when I am doing something for a client. That’s the goal.

Now technology moves too fast to be expert in everything. There will always be opportunities to learn something new, but my goal is to make the systems I run not scratch that itch for new learning. All that is to say that I have a craving for learning new stuff that is not and should not be filled by the day-to-day work I fit my learning projects around.

So there I am in the middle of doing a general something new. I have a general idea of how it goes, but there is a part missing or a process that I don’t know how to do. I know the next step, but not quite how to get there. Then I learn how, or I learn it is not going to work and I find a new way. My favorite learning is when I learn something new that brings two formerly unrelated pieces together in my mind and they fit together. That click is one of my favorite experiences in life.

After a while of this, things started to make a lot more sense. I had something that would mostly work. There were still some things to optimize, and some things to work out. More of the fun part. Here’s the problem: It gets addicting to learn stuff. If you’ve ever gone to Wikipedia and seen hours magically vanish, you know what I am talking about.

At some point though, it is time to stop fixing for a while. Freeze the design and commit to finish the thing, even though it is broken. Make version 1.0. There are a lot of things you can learn from seeing the finished system even with its flaws that you can’t see by looking at the parts.

I’m not saying leave it broken. Make version 2, and version 3, if you want but at some early point, freeze the specification, stop making changes, stop making fixes, go finish. Exercising the discipline to stop following new opportunities to improve my project, stop fixing, and stop learning (for a moment) meant the difference between having an imperfect but completed project and having a whole ton of good ideas.

So I stopped and did the drudge work. I finished. I made something imperfect. My reward? In addition to the problems I deliberately ignored, I see lots of little details that need fixing, which I would never have seen without finishing.

Off to version 2.0. I love learning.

Debugging what is going wrong with a VPN setup is difficult. The IKE protocol is “chatty”, and negotiates back and forth between the two ends for several rounds. The GUI offers not much help, it is either  UP or Down. Most of the real debugging happens inside the CLI.

One problem in particular that has always bugged me is that you need access to the end machines involved to initiate traffic across the link. The network admin typically doesn’t have direct access on the computers on either side of the VPN in order to initiate that traffic. I’ll show you a method that can be used to initiate traffic from that network as well.

Here are some basic steps to troubleshoot VPNs for FortiGate.

In IKE/IPSec, there are two phases to establish the tunnel. Phase1 is the basic setup and getting the two ends talking. Then IKE takes  over in Phase2 to negotiate the shared key with periodic key rotation as well as dealing with NAT-T (NAT tunnelling), and all the other “higher-end” parameters.

The first trouble shooting step is to verify your parameters are all correct and matching.

For Phase1, is the end gateway dynamic or static? Fortigate to Fortigate can use both Main and Aggressive modes for dynamic connections, but many other brands can not. In general, if you are supporting a dynamic IP client end, you will have to use Aggressive mode Phase1, so make sure that mode is set for dynamic clients. If this a static config, you should use Main mode for Phase1, which is a bit more secure on the initial handshake.

For Phase2, are both sides setup to use PFS? Replay Detection? Dead-peer detection? While most VPN setups include a set of encryption and hash algorithms, you only need one that are the same. The reason for the set is to offer many choices. In practice, just pick one that your base client supports and go from there. Now-a-days, AES256/SHA1 is probably supported across the board, and that is all I ever use. You don’t have to match the set of them exactly, each side just needs a common one to talk.

After that all checks out, we need to see what IKE is doing that is failing.

So SSH or console into the CLI.

If this is debugging a VDOM
(like in this case), you may have to switch into the root VDOM if you
are the system admin of the firewall as opposed to a VDOM admin.

fgt300C-fw # config vdom
fgt300C-fw # edit root
current vf=root:0

fgt300C-fw (root) #

as the diag commands are only available in the individual VDOMs or from the root VDOM for the system admin.

To enable debug logging on the console (should be default) do

fgt300C-fw (root) # diagnose debug console

To enable debugging output

fgt300C-fw (root) # diagnose debug enable

Phase1 debugging isn’t too useful. IKE/Phase2 debugging is where the problem almost always is. Lets turn on full debugging logs there.

fgt300C-fw (root) # diagnose debug application ike -1

Now, the problem I’ve always run up against is getting the tunnel to trigger to open up with traffic running on the link. You either have to conference in somebody with access to help you, or use this nifty trick…

Open another SSH connection to the FW CLI.  (If this is a VDOM, you’ll have to ‘conf vdom; edit “vdom3″ to get into
the VDOM context where the network is you want to troubleshoot).

Set the ping source IP address to be in the inside network of the host you are trying to troubleshoot..

fgt300C-fw (vdom3) # execute ping-options source 172.30.3.254

And now, ping away from the CLI in order to bring up the tunnel interface

fgt300C-fw (vdom3) # execute ping 192.168.0.1

(assuming 192.168.0.1 is an existing host only reachable via the VPN tunnel, and the ping service is allowed through the tunnel).

fgt300C-fw (vdom3) # execute ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=46.9 ms
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=47.3 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=45.5 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=66.3 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=45.7 ms

--- 192.168.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 45.5/50.3/66.3 ms

The trick here is that you are source as the network you are setting up, which should trigger the tunnel to come up if it isn’t up already, and you can see real live traffic. I don’t know how many times I’ve been stuck on a conference call waiting for whoever had access to do something to get around to doing the test I asked of them.

Back in the first debug window, you should see a whole bunch of IPSec and IKE messages fly past on the screen.

You have to learn to pick out the lines that are important, and zone in on them as everything is flying by. Learn to pause the display (or do a quick ‘diag debug dis’ to stop the output). Scrolling back and zeroing in on the one error out of 100 lines is going to be your key skill here.

If all is well, you should get something about the SA being established with the SPI value (not important).

ike 3:MyVPN_GW:18690:MyVPN:49143: added IPsec SA: SPIs=939fc892/b54d030

and of course, if it is configured for SNMP, something like

ike 3:MyVPN_GW:18690:MyVPN:49143: sending SNMP tunnel UP trap

is a nice confirmation that all is well with the VPN.

If you are seeing a lot of errors repeating with Phase1, and you see messages like

ike 3:MyVPN_GW:18698: sent IKE msg (P1_RETRANSMIT): ....

Most likely the problem is a mismatch preshare key for the VPN tunnel, as it isn’t passing out of P1 (which doesn’t have much to negotiate).

Also check again if this is dynamic client (generally requiring Aggressive mode) or a static connection that probably should be set to Main mode, but could be using Aggressive Mode.

If you don’t have a common encryption alg/hash, you should see some errors like..

ike 3:MyVPN_GW:18707: no SA proposal chosen

As it can’t find a matching SA between the two ends using the same encryption algorithm/hash combo to encrypt the tunnel. Fixup the encryption alg/hash and everything should go better.

The hardest problems to detect are different keylength timers (you’ll just have to review them on both sides to make sure your P1 and P2 keylife timers are identical on both sides). Problems that you encounter with different timers show up as a VPN that works for a while, but then stops work, and won’t come up unless you bounce both sides. With valid timers the same on both sides, the VPN should keep up and key rollovers happen automatically.

Also, DPD may not always negotiate. One side may have it on and let a VPN connection stay up for a certain time until the timer kicks off and closes the connection for the lack of keep-alive packets. Make sure both sides have it on, or both sides have it off.

There are a few other error conditions that may come up, but these are the more common errors.

The most important thing with the low level debugging like this is to learn to pick out the important error lines from all the rest of the junk flying by. It just takes practice. You may want to deliberately break an existing setup just to see what happens. But once you can zero in on that one error line out of a 100 that is important, it will be a lot easier to troubleshoot what problems may come at you.

You can even do it by hand if you have access to the backend storage. Mike once one-upped me by piping the VMX through sed. That’s cheating but all’s fair I guess. Cheater.

The vmForge VDC allows you to clone vApps and the individual machines contained therein. It automatically edits the config, can handle numbering the machine, and makes everything nice and easy. This is a killer feature in my book.

A lot of cloud providers are instance based. You select the operating system, push it out, and rely on automated services to configure them for you. Most of the time, you don’t get persistent storage. If you do, it’s usually a volume you attach to the instance and has nothing to do with its operating system. By using a vmForge VDC you can do the opposite. You can create a machine, configure it how you like, and then clone it. Configure once, and be done. Then you can keep a copy of it in your catalog for later deployments. Each clone is exactly that: a complete copy of your original system.

You may think that’s really cool! But wait, there’s more! (sorry, couldn’t resist)

When you build virtual machines in your VDC you are building them in vApps. A vApp is a logical container that holds virtual machines, internal networks, and can do things like set boot/shutdown order and power-down semantics.

When creating a vApp you also have the option to “fence” it. Fencing isolates the layer-2 networks within the vApp from any outside network. This means you can have internally consistent ip addressing inside the vApp. You can then “template” the vApp by moving it to your catalog and deploy it over and over and over again. That means that your preconfigured, multi-server application can be redeployed with a few mouse clicks!

Ultimately, cloning is about saving time. You get to use conventional tools to set up and multiple machines quickly and easily. You don’t have to learn any arcane scripting language, nor trust and maintain a complicated configuration service like Chef or Puppet. You just set up servers, push them out, and start to use them.

So, clone away!

It gets you out of hardware. Out of substantial up-front costs, management and repair, depreciation, and end-of-life planning.

It gets you out of data centering. Out of power, cooling, and cabling overhead and management.

It might even get you out of this. With a virtualized infrastructure, you can get access to and administer your servers and network from almost anywhere. From your office, your home, the beach…

What else could you be getting out of with a virtual data center?

ZFS Version 28 adds some of the more important features of ZFS: Deduplication, triple parity RAIDZ3, and RAIDZ. This means that I can have full featured storage devices, running ZFS natively, via FreeBSD.

As a bonus I don’t have to learn Solaris.

You can run ZFS in Linux but you would either have to run via FUSE which is file system emulation in user-space, not in the kernel. Or download it and build it yourself. In my opinion, both of those options are idiotic. I’m not willing to jump through those kind of hoops just to run a filesystem in Linux. I’d rather have native, in kernel support for it. Until now, your choice was either run Solaris (or a fork of Solaris) or run an outdated version of ZFS via FreeBSD.

One project that should directly benefit of this: FreeNAS. FreeNAS is a customized installation of FreeBSD designed to operate as a NAS and iSCSI SAN. It has a pretty slick ajax/web interface as of version 8 but so far had missed out on key ZFS features.

One reason I want run up FreeBSD 9 and ZFS is to better learn ZFS troubleshooting and administration. FreeNAS aside, there are a lot of vendor supported storage devices that are coming into the market based on ZFS. I want to troubleshoot those devices on a lower level. Before this, I would have to install Solaris. This means that I would actually have to navigate to Oracle’s Website. No thank you.

In-line deduplication is on of my favorite impractical features of all time. It unfortunately, required gobs of memory (8 GB RAM for every 1 TB of storage, if memory serves) Hopefully, someone smart will figure out how to do it on flash, in a practical way, as rebuilding those tables after a power failure would suck. (see Mike’s post about Tegile – a company actually doing such in production today)

Obviously I don’t know a lot about ZFS yet which is why I’m glad I get to learn via FreeBSD.

If only I could convince ipHouse to give me a little more storage space on my personal VDC…hint hint!