So, a little more than a week after its launch, employees were allowed to purchase iPhone 4′s for our personal use.  And I snapped one up.  The 32GB models were in low supply, so I opted for a 16GB, even though I had already filled my previous 16GB iPhone 3G model.  First impressions — I really love the iPhone 4.  The camera is fantastic, the screen is really amazing, and its speedy (which any original iPhone 3G owner can appreciate).   And iOS 4 is a great enhancement when you have the better processors.   I can attest, though, it runs like mud on an iPhone 3G.

That said, I found the attenna issue to be a real thing.  If I touch the left side near the break point, my signal diminishes.  Even with iOS 4.01 update, I still see signal go down.  Apple did discover that they had a software error that caused misrepresentation of signal, but the iOS 4.01 update didn’t resolve all the problems for me.   However, I believe that a case will fix the problem as Consumer Reports and Apple have both reported.  And so, today, I downloaded the iPhone 4 Case Program app from the App Store and selected my free case.  The shipping time shows 3 to 5 weeks to arrive (which is the only bummer), but Apple did a great job of making this program easy to use from a customer’s perspective.

To order your free case from Apple, get the app, launch it and then click the blue button.  You are asked to log into your iTunes account and then the app apparently checks the fulfillment system against your serial and/or iTunes account to see if you have a pending order.  If you have a pending order, it will display a message — otherwise, it displays a list of the available cases.  I was impressed with the selection — cases from Speck, Belkin and other vendors.  I selected my Speck case and placed the order.  The app automatically pulled my address and name information from my iTunes account and so one more tap and the order was complete.  It took only about 4 or 5 taps on screen to complete my order — which is a testament to Apple’s mentality and method of creating easy customer experiences.

That all said, I’m not sure that Apple PR or Steve Jobs handled the situation very gracefully, given the type of experience they strive to create for the customer.  But that is my personal opinion and no one can discount that Apple has made an exceptional device and people want them.  Even with the antenna problem, it is the best iPhone yet.  I can’t wait to see what is up their sleeves next.    That marks one thing off of my Apple to-buy list, but a new iMac and iPad are still on the list…  just waiting until the house comes a little further and we see how all of the finances for it works out.  We are trying to be very budget conscious right now.

To quote the HP engineer on the call — there were just too many combinations and possibilities to be able to certify all available firmwares — and so the idea of a release set began in January.  The release sets are available in a compatibility matrix on the HP Bladesystem page (http://h18004.www1.hp.com/products/blades/components/c-class.html) — look at the Compatibility tab.

The good news is that in our environment, we are close to compliance with the January release set.  The only thing out of compliance for the January release is our blade ROM, PMC and iLO firmware, but, as we were also informed, this is not a good situation to be in.

Something I already knew is that we have a bottom-up firmware topology for the HP Bladesystem — meaning that we have to update the bottom level components first and then move up.  I knew this applied to the interconnects and onboard administrator modules, but it also extends to blade servers also.   In addition, the blade servers are at the lowest level and should be updated first — particularly the iLO.

The reason for this is that new OA and interconnect firmware may introduce features and if the PMC or iLO is not aware of these features due to out-dated firmware, erroneous might be passed and all sorts of things could happen — at worst, random server reboots.

iLO firmware 1.81 is particularly susceptible to the random reboot for Windows blades.  If you are running 1.81 and have Windows OS loaded, you should upgrade to 1.82 as soon as possible.  See http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?locale=en_US&objectID=c01802766 for more information.  Other OSes – Linux, VMware, XenServer – are not affected.

A week after our conversation, I began playing with WordPress on my own webserver and started putting together a website mock up with a template from WooThemes and content from their existing site.  It was fairly easy and I like playing around with websites like that.  WordPress worked pretty well for the basic site.

WordPress also provided them with an easy way to keep the website up-to-date from the road or from their iPhones as they traveled and snapped photos.  WordPress is supported and updated as security problems were found and should help them stay secure in the future.

Useful Plugins for Musicians
After putting their existing information into the site, I started looking at what else would be good on the site.  I located the fantastic website of developer Dan Coulter, http://blogsforbands.com/.  He had developed several WordPress plug-ins for bands, including a gig calendar and discography.  I added the discography plug-in to a new install of WordPress and began adding songs from the past album that I had.  It worked wondefully — allowing me to post each album, the songs, words to the songs and links to buy the songs from iTunes, Napster or Amazon’s MP3 store. It also allowed a link to buy a physical copy…  but from where?

Enter another plug-in – Tips & Tricks HQ – developers of the WordPress eStore.  I had used WP E-Commerce in the past, but it didn’t want to install on my musician’s web host and it didn’t easily offer digital downloads, but WordPress eStore did…   We added the plugin and began populating it items, like T-Shirts and CD’s.

Our musician friends utilize ReverbNation heavily and so I was able to find their widgets (which our friends were already using on their original site) and place those into text widgets on the sidebar.  The cool thing about this approach is that it allows them to update their music players, gig calendar and mailing list all within ReverbNation and have it feed their website.  Likewise, their website is feeding ReverbNation new mailing list addresses for future mailings and it is collecting stats of who is listening to their music.   ReverbNation is a free service, but offers enhanced and additional pay-for services.

As we were going live and testing everything, I found a couple things – like emails being sent from the eStore were sending from “WordPress”, but Tips & Tricks HQ had another plugin to allow us to customize the friendly name of the sender.

If any other bands or musicians are looking for a solution, I’ll be the first to recommend WordPress!

I had been contemplating getting a Mac Mini to replace my aging Apple TV at home and to offer us the capability of watching NetFlix or Hulu directly on our TV.   I had found dongle cables to offer video and audio to HDMI on the last revision of Mac Mini and was just waiting primarily on our house build to get going to see how our money looked as we were finishing the project.  I have been trying to be very good about my technology purchases – since I felt like we had higher priorities.

And today, I am glad we waited.  I hope to add a new Mac Mini to our collection at home soon as our media Mac – our home base for iTunes content and photos.  I still feel like I need to wait, but my wishlist of Apple products keeps growing…  An iPad, a new iPhone 4 and now add a Mac Mini.  I also want to replace my aging iMac with a new version for video editing…

The biggest issue with the Mac Mini media “server” I see going forward is software, given today’s hardware revision.  I had blogged before (here and here) about wanting some of the iTunes capabilities brought over to iPhoto – to be able to setup a library and have all my photos automatically push up to it and to be able to pull down photos easily in iPhoto from a central library.

I’ve mentioned my friend before with the agency, and good photo management is a problem that they have in their all Mac environment.  They currently have their photos on a file share, but iPhoto is much better solution since you could add keywords and ratings and all the other metadata that iPhoto allows.  But this same need applies to the multi-Mac home.

I had been worried about losing my digital home movies.  That data is really too large to really push out to a backup service like Mozy and when I load new movies, its usually to the tune of 20 or 30GB at a time, which would take weeks to push up.  In addition to that data, I also have Movies and TV shows that we have purchased through iTunes.

So, I knew that I wanted some sort of external storage with at least mirroring capabilities to protect the movies as best I could.  Back in January, I was close to buying a Western Digital mirrored external hard drive from the Apple store.  I chose instead to get a 1TB, single external drive with Firewire as an upgrade to my 500GB Time Machine which was almost full, and wait for a better solution.

Since it was first released, I have always been in love with the Drobo.  I describe it as an mini-EVA to my co-workers, because it mirrors a lot of the HP Storageworks EVA functionality – like drives auto-leveling and automatically striping data across disks in the disk group.  Maybe, I just like the idea of having something that advanced attached to my home computers…  and I’ll be the first to admit that they don’t compare — they are apples and oranges.

But I do love what the Drobo offers, and so I have been watching them for some time.  Since its introduction, the Drobo had gone from a single device with USB only, to an enhanced version which has USB2 and Firewire (my choice of devices), a version which adds eSATA with USB and Firewire, and several larger devices, the Drobo Elite and the Drobo Pro, which feature iSCSI and NAS functionality targeted towards SMB’s.

As a side note, the Drobo Pro is even VMware certified, and I feel like it is a great solution for small to medium businesses looking for shared storage for an vSphere deployment.  This past week, we attended the Charlotte Regional VMware Users Group meeting, and actually got to see one of the Drobo Elite units on display.

The price has held steady on the Drobo that I have been watching.  At a $399 price point, I couldn’t justify buying it and then having to purchase drives to go inside.  But, as I said before, B&H was offering a great deal – the Drobo device at $100 off its normal MSRP.  It was a good $50 less than I could find it anywhere else, and so I bit — err bought.  I was able to equip it with two Western Digital Caviar Green (my wife would be proud) 1TB drives from NewEgg for a cost of $138.  And so, last night I finished up my transition of data and have everything I wanted protected on the Drobo – and I’m happy…

While we were first deploying VI3 and performing our consolidation, the primary focus was on the non-critical systems and moving them into the virtual infrastructure to get the best utilization of hardware.  Since completion of this phase, the next focus became moving some of our mission critical systems to VMware in order to establish disaster recovery for our non-clustered systems.  Disaster recovery through VMware is accomplished by 1) relocating the boot and data onto SAN storage which is replicated to our secondary data center and 2) by the ability to utilized VMware HA in the event of hardware failure to establish resiliency we do not have on a single-server, hardware deployment.

As we have expanded VMware’s role in our data center, new challenges have emerged.  First, when a network issue is occurring, we don’t have our traditional monitoring tools (like PRTG) in a position where they are able to alert for large changes in traffic.  In our physical environment, HP agents are run and PRTG is able to query against these systems with SNMP to retrieve information about traffic.  In the virtual environment, we don’t run these agents (because they are largely non-applicable since these are virtual boxes).  Our preferred way to monitor is through something that can look directly into the virtualization layer and retrieve information.

When we began our virtualization initiative, we saw some monitoring solutions at regional VMware user group meetings, but at that point – we didn’t see a lot of value in the products.  As we progressed through the beginning of phase two, we knew that monitoring and visibility into the virtualization beyond what vCenter allowed was a big thing for us.  We demoed several products from companies I had seen or talked with at VMworld and we finally decided upon two candidates to demo within our environment.

The first product we tested was Hyper9 — and we were really impressed with this product.  This product began as a search tool – basically a Google for virtual infrastructure.  It was really cool and the search interface was really good for answering questions as asked or posed by management.  It also was quick to drill down and get to information that we were searching for.  One of its most powerful features is it’s vmDNA – where it tracks changes in the VM and can compare two point-in-time views to see what has changed on the server.  This, we felt, was really powerful.  The place where we saw a deficiency in the product was related to alerts and alarming.  Hyper9 had separated this into a separate product which is good from a flexibility stand-point, but bad from a central management and alarming standpoint.  This piece of the puzzle just didn’t seem to be enterprise ready.  The second deficiency was in relation to network monitoring – something that was on our must-have list of requirements.   Hyper9 came pre-populated with lots of common pain point searches that allowed us to quickly look for problem areas — large snapshot files, datastores that are low on disk space, and tools that are out of date or not running.  The problem we saw with this was that the searches were passive (had to be executed at a specific time — scheduled — in the secondary application) and did not actively alert when a condition was met.

The second product we had decided on demoing was Vizioncore’s vFoglight application.  vFoglight is based on Quest Software’s Foglight engine (Quest owns Vizioncore), which is an enterprise class monitoring and visualization engine.  Vizioncore wrote the VMware cartridge which handles the monitoring for VMware infrastructure by pulling information directly from vCenter — with no impact to the ESX hosts.   It is strong in reporting and alarming.  The configuration, terminology and organization of the application is different from anything else we looked at, so learning the layout and how it operates was a large learning curve.  Fortunately, Vizioncore offers good training for free on their website.

vFoglight was exponentially more complex than the Hyper9 product, but it offered the additional features we were looking for in the networking areas.  It also had the integrated alarming and notifications within a centralized console – not a in a separate application.   Most of vFoglight’s intelligence was built around the alert and alarm conditions which triggered alarms in an alarms panel which could be tracked and researched (how often has this happened before), where notes could be stored and where individual members of our team could acknowledge an alarm or add notes on how the issue was resolved.   In addition, one of the internal developer had created a “vBundle” of common, useful reports which is available free to add to the system.

Because this software was built on the Foglight engine, it was not only possible but may be very advantageous for us to add additional cartridges to monitor specific databases or application or physical hosts.  The fewer monitoring systems we have to maintain or watch, the better, in my opinion.  Time will tell.

Ultimately, we made the decision to go with vFoglight.  The downside to the decision was its price.  We paid well for all this functionality.  But, so far, it has worked extremely well.  We finished our production installation last week and have about a week’s worth of data.  The Foglight system works much better over a long time as statistics and predictions become more accurate because of historical data warehoused.  At this point, we are still learning the application and how to do things in it and letting it collect data.

The service engineer for our account was extremely helpful in setting up the application.  They have a sizer application which determines the number of objects in your infrastructure and then provides recommendation on how to best hardware configuration, database options and estimated data growth over 1 year and 3 years for planning.  I hope to have more information on vFoglight as I continue to learn about it.

Our errors look like this:

After several cases with VMware and HP technical support, we are no closer to resolving the issues.  VMware support, for its part, has done a good job of telling us what ESX is reacting to and seeing.  HP support, on the other hand, has been circling around the problem but has made little progress in diagnosing the issue.  We have had an ongoing case for several months and our primary EVA resource at HP has continually examined the EVAperf information and SSSU output that we have sent to HP for analysis.  Those have turned up nothing, and yet the messages continue from VMware.

The errors in the log make sense to me – we are losing a path to a data disk (sometimes even a boot-from-SAN ESX OS disk!) – but why HP cannot see anything in our Brocade switches or within the EVA is beyond me.   Our ESX hosts, whether blade or rack-mounted hardware, are seeing the problems across the board.  The one cluster we waited to upgrade never saw the issues in ESX3.5, but sees them now in ESX4.  And perhaps it is a VMware issue that is just too sensitive in monitoring its storage, but I suspect its something else.   The messages don’t seem to affect operation on the hosts, but it certainly makes investigating problems difficult when trying to determine what is a real problem versus just another failed path message.  Anyone else seeing this?

Transition
The transition to the dvSwitches was an unexpected complication.  In vCenter, the port group names, although identical, are presented differently in vCenter.  Because of this, you could not simply VMotion from an ESX3.5 node to ESX4.  As a solution, I decided to make an temporary standard vSwitch on one node of the new ESX4 cluster as the destination for all my vMotions.  Each of my dvSwitches on the ESX4 cluster had two uplinks, so I stole one uplink for each of my temp standard vSwitches.  This allowed me to seemlessly change the network configuration of each VM after it vMotioned from the standard port group to the distributed port group.   Although it was more steps and trouble, it allowed me to make the transition during the day while production workloads stayed online with no impact to them.

Design Considerations
vNetwork Distributed Switch has certainly saved me time, and its only been in my VMware environment a really short time.  I like several things that the distributed switch allows – such as persistent ports, port statistics and the ability to maintain the distributed switch in one place across all nodes in my cluster.  The downside to this centralized administration is an increased dependence on vCenter Server.

Rich Bramley of VM/ETC posted on the challenges of distributed switches with virtualized vCenter instances — largely the problem of a cyclical relationship that could leave you between “the vDS rock and a hard place,” as he puts it.   He and I drew the same conclusions and ultimately, I have decided to keep a mix of distributed and standard vSwitches within my environment.

Perhaps I am overly cautious, however, I run my service console, VMotion, FT Logging all across VMware Standard Switches.  All VM traffic across the vNetwork Distributed Switch.  The exception to the “all VM traffic” rule will be when  if we introduce a virtualized vCenter instance on our cluster.  The dependance on vCenter for dvSwitches and having a virtualized vCenter assigned to a distributed port group makes for a potential disaster, since the VM cannot access or bind to his port on the distributed vSwitch without vCenter running.  Also of note,  Rich reports that vCenter on a distributed switch is NOT supported by VMware because of the cyclical relationship between the two.

In previous version of ESX, the desired storage setting was fixed path for the EVA.  In our case, we simultaneously presented the ESX3.5 and ESX4 hosts to the same LUNs, meaning some were fixed and some were set to the ESX4 default, which was MRU.  This caused problems.  After initial issues, we backed away and presented one LUN at a time, performed our VMotions and then unpresented the LUN from the old cluster.  This prevented any flapping issues between controllers.

The best practices document describes two settings for best practice with an EVA that is ALUA aware (EVA 4000/6000/8000 and EVA 4400/6400/8400):

• Set the default path selection policy to round robin (VMW_PSP_RR)
• Set the IOPS Limit to 1

The path selection policy is the bigger deal.   The default path selection policy in ESX4 is most recently used (MRU).  This is still a valid policy and works with the EVA, but it is not their best recommendation because Round Robin allows the ESX host to use two paths or four paths (depending on EVA model) to send traffic to the ESX host which can provide greater performance.

I was not sure what the IOPS setting actually does based on the documentation, but Duncan Epping has since posted about the IOPS=1 setting and about how in a real-world environment this setting may not make a difference.  I agree with his conclusions.

Now, I should not and do not want to take credit for the instructions below… I did not come up with this solution — simply a user.  The esxcli commands came directly from the HP Best Practices Guide, but the scripted IOPS setting changes for the LUNs came from another blog - Virtual IEF.  This solution has worked brilliantly in our environment.  Virutal IEF has a great, in-depth posting about how the EVA LUN ownership works and what makes an optimal path.

Setting Changes for Round Robin PSP
To change the default path selection policy (PSP) to Round Robin. If LUNs are already presented, you will need to reboot to allow the LUNs to rescan and set to Round Robin instead of the default most recently used (MRU) setting.

 esxcli nmp satp setdefaultpsp --satp VMW_SATP_ALUA --psp VMW_PSP_RR

Change the IOPS limit value for the interactive session.

 for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i ;done

Verify the IOPS value is correct, run this script and check values.

 for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin getconfig --device $i ;done

Your output should look like:

 Device: naa.600508b4001087e400009000012b0000
 I/O Operation Limit: 1
 Limit Type: Iops
 Use Active Unoptimized Paths: false
 Errors:
 Unable to find device with the name naa.600508b4001087e400009000012b0000:1
 Byte Limit: 10485760

Unfortunately, the IOPS value will reset on reboot many times. To ensure that this value is correctly set, you are able to set a value in the post boot script. Add the same for statement to/etc/rc.local to set the IOPS value on each reboot.

 for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i ;done

On a personal note, my wife and I are in the final stages of planning our new home, which we hope to begin building in the coming months.  Any free time that I would have had to blog about my experiences has been consumed with house plans, builder meetings and other items to prepare for this major undertaking.  My wife is attempting to chronicle our build on her blog, My Green Glasses.  The house will be a certified green home, Energy Star certified and *possibly* LEED certified.  We know that we will be close to meeting requirements for a LEED certification and we are looking at what additional things need to be done to make it happen and whether it is worth it or not.