The first thing to realize, is that vShield Edge is written for security folks to use and not for virtualization administrators. The rules you apply are very similar to those found with in the O’Reilly book Building Internet Firewalls. Which is, actually, one of the better books out there on how to properly create the rules for your firewalls. My copy is well used and marked as I used to build my own firewalls.

I had, what I expected was a simple use for vShield Edge. I wanted to use Edge to protect a View environment while using my existing virtual firewall for my standard DMZ and network. However, I also wanted the View virtual desktops to talk to my new vCloud environment. But the first task was to add in View so that it could talk to my existing environment then add in the vCloud mechanisms. Figure 1 depicts what I wanted to happen, in this case, all I am adding is View into the mix with vShield Edge.

Figure 1: Simple Network w/vShield Protecting View

So why use vShield Edge and not my existing virtual firewall? Mainly because, I already serve on port 80 and 443  various existing web services and for View to work not only do I need to serve up port 80 but 443 for View traffic amongst others. So instead of adding in reverse proxies and other complicated features to give me access to more services hanging on port 80 and 443, I assigned a new IP to a vShield Edge Firewall.

vShield Edge, unlike VMsafe type firewalls (vShield App, Trend Micro Deep Security, Reflex Systems vTrust, Juniper vGW, etc.) sits between two vSwitch portgroups (whether on the same vSwitch or different vSwitches). In my case, different portgroups on two vSwitches for the external, and two portgroups on the same vSwitch for the internal side of the traffic.

However, there needs to be a few more things added to the diagram to make View work in a secure fashion. That is the View Connection Broker, and the View Security Server. These two would fall into different aspects of my network to protect them. Figure 2, shows their placement. But the key is why I placed these two services where I did.

Figure 2: vShield w/View Components

The View Security Server acts as a gateway between the desktops and the connection broker and the actual client. In essence, you get a SSL Tunnel from the View Client to the Security Server, and from there communication happens to broker, which hands off to the security server a desktop to which it can connect.  If the Security Server was not there, there would not be an SSL tunnel and the hand off would be between the client and the connection broker. Which means you would need to open quite a few more holes in your firewall. The connection broker could live within the View environment, but you really want to protect this server from the hostile environment of the desktops, so in essence the View trust zone (orange in figure 1 and 2) should only contain the desktops.

In our diagram are two vShield Edge (vSE) components, each with their own specific rules. The first is truly a bastion firewall, while the right most (in figure 2) is a way to keep the hostile desktop zone from reaching the internal VMs except on well known ports. But first, how to configure each Edge firewall so that you can actually use them. Just installing and turning on Edge is not enough.

Step 1:

Remember vShield is designed for security folks who understand firewalls, so a bit of information first:

Figure 3: Edge Internals

There are two sides and really two sets of tables within an Edge firewall. The first is the External Interface, with its associated firewall. The second is an Internal Interface with its own associated firewall. You can go in and out of each of these interfaces. So for example, if you want to have traffic go from the external to the internal interfaces. You need to allow, Incoming on the External (into the external interface into the middle area of the Edge firewall), and Outgoing on the Internal interface (or into the internal network). There exists a middle area, which is between these two interfaces which is covered by Network Address Translation (NAT) and routing rules.

Step 2:

Set up the NAT, to allow all those IPs on the Internal interface to masquerade as the external IP address. This we would think would be setup by default, but that is not the case. You need to setup the source NAT your self. In other words, you need to modify the source address of the VMs inside the vSE to masquerade as the external IP address. The actual, external interface is blocked out in Figure 4, however lets assume it is W.X.Y.Z and we want ANY internal IP to map to W.X.Y.Z.

Figure 4: vSE SNAT

Step 3:

Setup external and internal firewall rules to allow the following services through:

• ICMP (to allow people to ping your external addresses to see if they are alive)
• UDP based DNS (53/udp)
• HTTP (80/tcp)
• HTTPS (443/tcp)
• PCoIP (4172/tcp and 4172/udp)
• RDP (3389/tcp)

You also want the same rules for outgoing so that you can respond to the requests appropriately.We will assume the internal IP address is A.B.C.D and the External is W.X.Y.Z. So what does this mean in vSE speak? Enter the following rules:

So why 4 rules for every port we want opened? Remember Step 1, where we have two distinct interfaces? We need to deal with each interface separately. We allow these ports In if we also need to allow them Out, so that we have continued communication.  We could also try to set up the rules like we do for icmp and DNS-UDP, but  I found all 4 are needed for each port.

Step 4:

Use the default ‘block all’ Traffic Police to anything not already listed as open. During testing you may have to set the Traffic Policy to allow, but we really want this to be Blocked per Figure 5.

Figure 5: Traffic Policy

Step 5:

Allow Communication to the View Desktops. We do this via a route. The View desktops sit on the network I.J.K.0/24, and the Internal side of the vSE is A.B.C.254. So the static route looks like Figure 6. The route says to take all traffic destined for I.J.K.0/24 and pass it through the A.B.C.254 address. Which happens to be the

Figure 6: View Static Route

Step 6:

Using vShield App or vShield Edge, allow the following to communicate from the View Desktops and Security Server:

• Active Directory
• Internal DNS
• View Connection Broker
• and Ultimately my internal vCloud Director

Is that All?

But is this everything? Actually it could be but if you look closely at Figure’s 1 and 2, you will notice that that there is no firewall between the View Zone  and the Internal Zone, well not one you can see. The first approach to this was to use vShield App to keep the View Zone from the Internal Zone, but you can also use a vShield Edge appliance to achieve the same behavior. So that you end up with the following overall design of an existing environment (Figure 7).

Figure 7: Internal Protected from View Desktops using vShield Edge

What is very interesting here, is that vShield Edge can be used to protect bastions, but also to create separation between various trust zones while limiting access between the view desktops and the real internal aspects of your network.

Thanks to @vTexan and his VMware View 5 Install Guide. Without this helpful page, I would not have been able to proceed with my own implementation. Please be aware, however, that the steps @vTexan gives are not necessarily the most secure. Adding in the properly placed firewalls is just one aspect of a secure view implementation, but there is also the proper choice of role based access controls (RBAC) for active directory. The View Connection Broker, requires Active Directory access, but should be limited to a non-domain admin for a newly created View only OU into which the desktops should go. Why, because View needs to be able to modify all aspects of Active Directory for the desktops, so the permissions should be such that the Active Directory user used by View should be limited in scope on what it can modify.

An Aside

I tried to use vShield App to protect the View trustzone from the Internal trust zone but vShield App fails badly if the vShield App appliances suddenly become unavailable. Given that I had 3 such failures, in 8 hours, that were fairly catastrophic, I chose to use on vSE to protect my trust zones.

Conclusion:

So now, we have a fully protected View environment using several vSE’s to provide adequate protection. In addition, we now have a cook book of steps necessary to get vShield Edge up and running (Steps 1-4). Lastly, we have expanded those steps to those required to allow View to communicate properly from the Security Server to the VMs in question.

• Automated backup during a predefined backup window
• Notification of ANY issues immediately
• Automated backup testing

So what was the problem with VDR? The integrity check continually failed and therefore the backups continually failed. Yet, it worked for a while! Which was the frustrating part. The solution was the following:

• Unmount the Destination (which is a Windows VM that talks to my DISC GmbH Bluray backup library for long term storage)
• Remount using the Share name, I previously created instead of the F$ mechanism available to windows
• Edit all my backup jobs to point to the new Destination
• Test by backing up critical components

I believe that VDR does not understand mount points ending in $, which could be given that it is a Linux VM and $ is a special character in nearly every shell and 4th generation language.

So now, I wait until I know for certain this will work, in the meantime I will be testing other tools. Backup failures are NOT something I like to see. Ever.

UPDATE: This was definitely the solution, been running strong for quite a while.

The setup of the scsi-target-utils was pretty straight forward, I pointed the /etc/tgt/targets.conf to my 7.8GB LVM LUN for use for iSCSI. Which meant I had to use VMFS v5 to access it all. Which is my intention. But then that is where several issues occured:

• vSphere 5 requires the iSCSI vmkernel port to hang off a dedicated vSwitch and not a vNetwork Distributed Switch (vDS). This is a change from vSphere 4.

This predicated a slight change to my networking which led to the creation of a dedicated network just for iSCSI and the use of  one of my free Flex-10 NICs just for iSCSI. Unfortunately, in order to change the networking within the Server Profile within Virtual Connect for Flex-10, you first have to shutdown the node. Since my server upgrade, I have more than enough memory to run everything my nodes so that powering off one will not adversely effect anything. So, I went through the process of upgrading server profiles, putting nodes into maintenance mode, shutting them down, saving the virtual connect profile, rebooting the node and viola it was done. If I had more than 4 nodes, this would have been a week end nightmare. How bigger shops manage this is with quite a bit of scripting.

• vSphere 5 would not allow me to migrate my vmkernel device from a vDS back to a vSwitch.

The second problem I faced was that I had to recreate all my vmkernel ports for iSCSI instead of migrating them off the vDS on which they were set. Because there was no migration path available. You can migrate from a vSwitch but apparently not back to one.

• vSphere 5 required me to bind a vmkernel device for iSCSI utilization.

Not a huge issue, but it is a definite change from setting up iSCSI with older versions of vSphere, where all you needed to do was have the proper IP address and it would route appropriately.

• vSphere 5 good not see the iSCSI Server yet, the iSCSI Server could ping the vmkernel devices.

This is where the mojo comes in. The first problem I had was iptables, it was set to deny iSCSI traffic as well as iSNS traffic. That was fixed temporarily by disabling iptables. However, it still would not connect. The solution ended up being a change to how iSNS traffic worked. I disabled the iSNSAccessControl parameter from On to Off, restarted iSNS and suddenly things started to work.

My final /etc/tgt/targets.conf file consists of the following:

default-driver iscsi
iSNSServerIP W.X.Y.Z
iSNSServerPort 3205
iSNSAccessControl Off
iSNS On
<target iqn.2008-09.com.example:server.target1>
        backing-store /dev/vg_iscsi/lv_iscsi
        lun 1
</target>

And for my security settings I ended up adding these rules to my /etc/sysconfig/iptables file:

-A INPUT -m state --state NEW -m tcp -p tcp -s W.X.Y.0/24 --dport 3260 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -s W.X.Y.0/24 --dport 3205 -j ACCEPT

Now I have a functioning iSCSI server with close to 8TBs available for use by vSphere 5. It will come in handy for some work I have to do on my SAN. Oh, and the hardware upgrade for this all purpose node? 32GBs of Memory plus a Sandybridge quad core running very fast. The 1GB dedicated link I am using is through a switch that will only be used for iSCSI, so all my iSCSI accesses are segregated from the rest of my network. That should help with security as well as performance!

• Install vSphere 5
• Join to vCenter
• Apply Host Profile

Yet it was not that simple.  I found that the Host Profile was asking for a Mac Address which I did not know, nor could I discover as it was not just asking for any Mac Address but one produced by ESX for a VMware vmknic. The solution was not obvious but something easily handled.

• Using the dvSwitch Switch Editor add the hosts in by hand.
• Using the Manage Virtual Adapters Editor go to each dvSwitch and add in the appropriate vmkernel devices (I had 4)

Now I thought I was ready to apply the profile, but not yet!

• Present any Storage Devices to the new Node
• Rescan the Storage for LUNs and Datastores
• Change the name of Datastore1 to be hostname:Datastore1 to be in keeping with my naming convention

Now, apply the profile.

There were still a few errors with the profile about a missing LUN, but no LUN is actually missing. Editing the profile to remove the offending LOCAL LUN did not change the behavior. Nor can I use Host Profiles to add my local administrative user anymore.

Following along the next steps are:

• Attach a VMware Upgrade Manager Profile
• Scan for Updates
• Remediate any Updates

So how do you get 100% Host Profile Compliance? I am not sure you can. All the local drives within HP blades actually show up as remote drives, and since they are not the same across the board, the profile can never be compliant. However, the host profile can be adjusted by using the steps in VMware KB 2002488. With this change, my hosts are now compliance once more. However, what this does, is disable both the Pluggable Storage Architecture (PSA) configuration and the Path Selection Policy (PSP) Configuration for all your local and remote devices. Any PSA and PSP changes will now need to either be done by hand or scripted in some fashion.

So why not upgrade all 4 nodes to vSphere 5 at this time? Support and Licensing reasons. I still have customers using vSphere 4, and having at least one node with that level will be helpful. The other is for Licensing reasons, I would need at least two more licenses for the 4th node. But this does lead to other issues. I cannot yet upgrade to VMFS-5, VM Hardware v8, or upgrade VMware Tools yet.

Which was that the database server I was using had a full drive.

None of my current management tools actually told me this, instead they went on running like everything was fine, when clearly it was not. So which tools did not show this information? VMware vCops Advanced, Solarwinds VMAN, and vKernel to mention just a few. But they do show issues with other nodes that had full disks, so I wonder what is so special about a Windows 2008 R2 Server running MSSQL?

This is the type of Alert I would expect to be emailed to me from all these tools. Yet, no email was sent.

The solution was simple, free up and add disk space to the VM. I removed from MSSQL a few old and unused databases, which gave me just enough breathing room to maintain my systems. The next step was to add more disk space. I did this by expanding the virtual disk and adding space to it using the build in Windows 2008 disk extend capability.

Now to find out why the tools did not report this accurately or even email me when this happened. Perhaps because the server was full and no writes could take place? Which means the tools need to change for such critical issues?

• Place node in Maintenance mode, evacuating all running VMs
• Power down node
• Remove node, add in/exchange memory modules, replace node
• Power on node, using ILO watch to be sure new memory was found properly
• Exit Maintenance mode

It was seamless, and once more done with Zero downtime.

However, adding in my new BL460c G7 into my HP C3000 enclosure was not that simple. Those steps started with:

• Place BL460c in Slot 4
• Power On
• Error ensued about placement of the riser card.
• Moved Riser card, Power on
• All was great

Okay so far it is pretty standard.

First Gotcha: The G7 blades include an ILO 3, which has several new security controls, all goodness, but to apply the ILO licenses to use the advanced features (like virtual media), I had to apply the license. To do this, you needed to Trust All to apply the license. So we moved on from here:

• Change ILO Trust mode to Trust by Certificate and import the HP SIM Server certificate.
• Mount the FW 9.30 ISO and upgrade the blade firmware using virtual media
• On Reboot set the Dynamic Power Mode per the the article vSphere Upgrade Saga: Staging Upgrade to vSphere 5.

Second Gotcha: To configure the P410i in the box, be sure to press any key to see extra Boot messages, else you will NOT be able to see the time to press F8 (Fn-Command-F8 if trying to use a Mac Book Pro via the Remote Desktop Client). It took installing vSphere 5 on this node to realize I had no disk on which to install.

Third Gotcha: Once everything was installed, I attempted to apply a Virtual Connect Server Profile so that the blade can properly access my networking. Even though the blade was found within the Enclosure within HP SIM, virtual connect could NOT see the enclosure. Actually, the Onboard Administrator said it could not participate in Virtual Connect.

Fourth Gotcha: The device would fail to reboot due to a power issue, which also affected the blade in Slot 3.

A bit of searching and I hit upon a possible solution: Move the blades in Slot 3 and 4 to Slots 5 and 6. Which means all my blades are now in the same cooling Zone within the chassis. Even with 6 Fans, the c3000 has two Zones that encompass more than just cooling.

Now, continuing with my new blade install:

• Join new blade to vCenter Cluster
• Apply vSphere 5 Profile
• …

Viola that worked. Virtual Connect Manager could see the blade, the server profile could be applied. Now, I begin to wonder what will happen when I put in another blade, but I will cross that bridge when I get there.

My issue with EVC stemmed from trying to enable it on an existing set of nodes so that I can easily transfer VMs between them. Exactly how EVC was planned to be used. However, in my environment somewhere along the line EVC was disabled. To enable it, I had to find the setting that would enable. I tried to first enable it for the chipset I had, that did not work. Then I enabled it for the previous chipset, of course that did not work. The solution was to enable it for the chipset I did not already have. In other words, I am running the chipset before Westmere, so in order for EVC to enable on my cluster, I had to set EVC mode to be Westmere chips. In other words, I am masking off all Westmere special functions.

This is what confused me, we all know we use EVC to mask off what we do not want, but if you have an existing cluster, you need to max off the chipset ABOVE the one you are on. That is, unless you desire to shutdown all your existing VMs, set EVC to the mode you desire and then power them al back on.

That was not something I was willing to do.

So, now I add to my pre-stage steps, setting EVC to mask off chipset functionality above what I already have. This will become a fairly major requirement when I add a Westmere based blade into my cluster.

Here are the steps I followed:

Step 0: Review any updates to firmware for my HP BladeSystem c3000 just in case something new came out since the last time I looked. There was a new upgrade for the Onboard Administrator.

Step 1: Given this upgrade was available the first step was to upgrade the Onboard Administrator firmware, which I could do directly talking to the Onboard Administrator web interface. Quite simple, and easy. Unlike previous Onboard Administrator ungrades there was no issues with loosing power or connectivity to any of the blades. While doing this, I made out a shopping list to ensure even more redundancy: Another Onboard Administrator as well as a replacement Insight Display Module.

Step 2: Upgrade the Flex10 Virtual Connect firmware. This is required to use vSphere 5. You must upgrade any Flex10/Virtual connect firmware to version 3.30 in order to use vSphere 5. Step 2 was to upgrade the secondary Flex10 first using the command line Virtual Connect Support Utility (VCSU) using the update command.

Step 3: Upgrade the primary Flex10 Virtual Connect firmware. There was an oddness with this upgrade using the VCSU, and that was the VM running the VCSU lost connectivity which seemed to cause a reboot. So something inside the VCSU failed and rebooted, which is really a horrible failure mechanism. However, I only lost network connectivity for a few seconds. Which was also much better than the last Flex10 Upgrade I did.

Step 4: Upgrade the firmware within my spare blade using the HP Firmware DVD v9.30. There was an ILO update, so to be sure all firmware was upgraded, I booted from the DVD twice just to be sure I caught everything.

Step 5: Install vSphere 5 ESXi onto the spare blade just upgraded. The trick is to ensure you pick the proper disk, in this case local disk instead of a SAN drive. I double checked this before proceeding. The install is so much easier than that of ESX.

Step 6: Configure the Management Network for the spare blade to not use DHCP but to use the provided IP Address, Netmask, and hostname.

Step 7: Decided to upgrade my existing vCenter instead of using the Linux appliance, mainly because I already have a Windows configuration for vCenter and the Database server.

Step 8: Backup your vCenter Database and SSL certificates. vCenter 5 will remind you of this as well and nicely ask you to ensure you are happy with your backup.

Step 9: Perform vCenter Upgrade

Step 10: Perform vCenter Client Upgrade

Step 11: Install vSphere Web Client for new License Reporting functionality. This required me to find out which process held port 9443 open, which turned out to be something running Java. However, what I do not know. There was a Java update to be made, once I did that, this error went a way. So I went through and removed all my third party tools from the vCenter server which eventually fixed the issue. A part of this step was to register your vCenter Server with the Web Client Server so that it can communicate.

Step 12: Upgrade vCenter Update Manager. Once more this is a polite reminder to make a backup first of your VUM database which requires you to select the I am happy with my backup checkbox. I find that a great improvement. While VUM upgraded fine, it did say it wanted to reboot my host, eventually that will happen as a matter of course. Need to test if everything comes back up on boot.

Step 13: Install all the vCenter Support Tools, Dump Collector, Syslog Collector, and Auto Deploy but not the Authentication Proxy. These are all new for vCenter 5. The key to these installs, is that if it asks, yes you want the Integrated VMware vCenter Server Installation. The Authentication proxy requires IIS which could cause issues so we hold off on that for now. Now reboot your vCenter server and make sure everything runs.

Step 14: Reconnect the ESX hosts to vCenter which deployed the new vCenter Agents onto the hosts.

Step 15: Create a new cluster so that I can enable EVC mode for Nahalem or better processors. This will help with new blades once they arrive. Somehow EVC was disabled on my other nodes.

Step 16: Configure VUM and import the vSphere 5 ESXi image for later use. The VUM the client for some reason did not enable by default which took going to VMware’s KB Article 1015223 to solve.

Step 17: Configure the newly installed spare node using the old nodes Host Profile. If you have Enterprise Plus licenses the use of Host Profiles for this type of staging is incredibly useful. Even though I am moving from ESX to ESXi, use of Host Profiles to at least get the network configuration correct is very important.  If you do not have Enterprise Plus, you can use an Evaluation Mode to do this. The gotcha here is that I am moving from ESX to ESXi so some of the Host Profile elements will not apply. So not only do I do a clone of the Host Profile, apply that profile, but I need to delete and recreate the Host Profile once the spare node is fully configured else there will be compliance issues.

Step 18: Reboot Node and enter the BIOS to set HP Power Profile to Custom and HP Power Regulator to OS Control Mode per VMware KB Article 1018206. Save and reboot. You can do this as a part of any of the node upgrade steps previously mentioned, but without this you will not be able to take advantage of the power controls within vSphere 5.

Step 19: Upgrade vSphere 5. This is a good test of your VUM install from Step 16. As of this writing there are at least 2 patches for vSphere 5. One requires a reboot.

At the moment, the system is staged for migration of VMs and upgrade of the other nodes. Staging such an upgrade is important as now we know that the firmware upgrades work, I can enable the proper EVC mode, and the other systems are still manageable.

First I should state every site is different and what I do for my site may or may not work for yours, however, sharing this information should help you to decide on how to proceed. First, I have a constant battle and need to do the following on a regular basis:

• Optimize and resize my images. Pagespeed and Yslow not only want you to specify image width and height attributes within the HTML img tag but they also want those images to be already scaled to the specified width and height as well as perform further lossless compression on image files.
• Use Google Pagespeed to optimize my unoptimized images for me, then replace the ones on the site with the optimized ones. You have to be careful with extensions as Pagespeed will change the extensions and names of the files to what it wants, but you need to copy them over using the appropriately named files.
• To properly specify image height and width attributes, I ended up modifying some custom plugins to always output the width and height tags.
• Modified the theme (or create a child theme) to call my customer plugin functions. You could on the other hand apply this logic to the entire content, but instead I applied it to just the specific functions that required it. That is to do the following style of function, which hard codes the width and height values, and echo’s out the output just like the original function. This approach requires you to look at the existing HTML output to determine the proper strings for the preg_replace.

  function my_plugin_function($args) {
        ob_start();
        plugin_function();
        $output=ob_get_contents();
        ob_end_clean();
        $output=preg_replace("/<img src/","<img width=48 height=48 src",$output);
        $output=preg_replace("/cookie.domain/","cookieless.domain",$output);
        echo $output;
  }

• Modify the image locations to use a cookie-less domain, otherwise you are sending out kilobytes of data per image when someone is logged into the site. I use the same approach as I did above but add another line (the one with cookie.domain in it above) to replace the existing URL with my cookie-less domain URL, which can be either an entirely new domain or a sub-domain. Either way, it serves up purely static content. This one change, has produced the biggest win.

Now that I have my daily or weekly tasks completed, it is time to launch into other tools to use to improve overall performance. First, let me say, I have used quite a few optimizing plugins within my WordPress install and some work better than others. My first approach was to do it all by hand, which did work, but was not complete enough. The key to optimizing the site is to do the following:

• Specify Expire Headers
• Enable Compression
• Specify Entity Tags for improved browser caching
• Reduce Request Size
• Minify HTML, Javascript, and CSS

To do this, I started by using the following:

• Modify .htaccess file by hand to include Compression, Expire Headers, And Entity Tags, however, this became to hard to manage
• Use the W3 Total Cache plugin which handled the creation of .htaccess files for me and to a certain extent reduced the request size and applied minification rules. But alas, a later upgrade required me to not only chance my SEO plugin from All in One SEO to WordPress SEO, it also required me to enable more modules within Apache. But also failed to work for some of my more complex lightbox code. So, I ended up disabling the W3 Total Cache plugin but I keep it around so that I can recreate the .htaccess files as needed. I believe this is all related to the fact that W3 Total Cache, just could not find my document root appropriately.
• Create my own minify tools for various plugins such as GT Tabs. Some plugins do not get their files picked up appropriately by any minify plugin, so I usually have to edit them to add something like this at the front and end of the PHP style script. Granted you could also probably call some other minify tool, but this type of code works as well, mainly because you do not know if the other minify tool is loaded already. Many plugins like Easy Fancyboxalready include this type of minification of CSS.

  <?php
  ob_start("my_minify_function");
  function my_minify_function($buffer) {
       /* remove comments */
       $buffer = preg_replace('!/\*[^*]*\*+([^/][^*]*\*+)*/!', '', $buffer);
       /* remove tabs and newlines */
       $buffer = str_replace(array("\r", "\n", "\t"), '', $buffer);
       /* and squeeze some more */
       $buffer = str_replace(array(", ", ": ", " {", "{ ", " }", "} ", "; ", " 0;"), array(",", ":", "{", "{", "}", "}", ";", ";"), $buffer);
       return $buffer;
    }
  ?>
  ...Normal file CSS Code...
  <?php ob_end_flush(); ?>

• Better WordPress Minify is another plugin which will help with minification of CSS and JS which instead of using filters, hooks into the enqueue sub-system to ensure the minification of CSS and JS takes place for all those that are added using the wordpress enqueue calls. The good thing about this plugin, is that not only does this do all the heavy lifting for plugins that use the enqueue mechanism, but it also allows you to point to a cookie-less domain to perform the minification. Most CSS and JS files are static, so using a cookie-less domain also reduces your request size. The bad thing is that any CSS and JS added outside of the enqueue sub-system is not optimized.
• DB Cache Reloaded Fix plugin helps by creating cache files for common db queries that can stick around for as many minutes as you request. Unlike page and object caching, db caching is most helpful on a site that does not have alot of site updates, as they may not show up right away. This one plugin improved over all performance by removing the more than occasional spike in access time for pages. <== NEW

The use of W3 Total Cache to create and manage my .htaccess files for browser caching, gzip compression, adding Expire Headers, and Entity Tags is very handy even if I cannot leave the plugin on all the time. Better WordPress Minify was the big winner here for my site as it handles minification quite well, and has enough controls to ensure things get performed appropriately. DB Cache Reloaded Fix smoothed out and lowered overall response time on the site.

However, we are not done yet, there are lots of little things that can be done to improve overall site performance using wordpress.

• Modify CSS of some plugins to remove conditional CSS statements as some add conditionals specific to IE6
• Reduce redirects which show up when you use Gravatars. There is a default gravatar which you can disable in the system, but it turns and does a redirect to the ‘blank.gif’ 1×1 image inside the wordpress distribution. Redirects like this are time consuming so I added another little bit of code to optimize the selection of my gravatar to be not only from a cookieless domain but to bypass all the redirects.

  function my_get_avatar($avatar,$id="",$size="",$default="",$alt="") {
        return "<img height=1 width=1 src=cookieless.domain/blog/wp-includes/images/blank.gif>";
  }
  add_filter('get_avatar','my_get_avatar');

• Force the template files for my theme to be served up by a cookieless domain.

  function my_template_directory_uri(template_dir_uri,$template="",$theme_root_uri="") {
        return preg_replace("/cookie.domain/","cookieless.domain",$template_dir_uri);
  }
  add_action('template_directory_uri','my_template_directory_uri');

• Edit the wordpress-popular-posts.php file of the WordPress Popular Postsplugin to use the functions wp_register_style/wp_enqueue_style over the old method of adding an action to wp_head to output the CSS includes (which is commented out in the following code. This change allowed me to make use of the Better WordPress Minify plugin for this plugins small style sheet plugin, and also fixed a CSS/JS ordering problem per Google Page Speed.

  wp_register_style('wpp.css',plugin_dir_url(__FILE__).'style/wpp.css');
  wp_enqueue_style('wpp.css');
  //add_action('wp_head',array(&$this, 'wpp_print_systlesheet'));

• Edit the theme to pass width and height for the logo image by placing the following at the appropriate place within the header code just before the closing /> of the logo imgHTML tag. Each theme will be different.

  $imagep=TEMPLATEPATH.'/images/'.$bfa_ata['logo'];
  list($width,$height)=getimagesize($imagep);
  echo '" width='.$width.' height='.$height;

There are still lots of little things to do with the site to achieve high YSlow and Pagespeed ratings, but they will require a bit more coding. While these tools and techniques greatly assist with this, the incrementals from now become harder and harder. One that would give me a huge increase in ranking would be to use CSS Sprites for small images that I load quite a bit, but to do this, not only do I need to create the sprites but modify plugins and code to use them appropriately. Still left to do are:

• Combine small images into CSS Sprites for those small images on the site. Some are easier than others.
• Leverage Browser Caching as any CSS/JS that appears by calling a PHP script has a 30 minute cache time.
• Optimize the order of styles and scripts
• minize the request size by offloading more to a cookieless domain
• minify inline javascript
• inline some small CSS that Better WordPress Minify did not pick up
• specify a few more entity tags
• consider removing query strings from static resources
• play around with deferring javascript from loading
• remove AlphaImageLoader CSS commands for IE, unfortunately, if I do not keep these one plugin does not work very well.

If some how I can fix all these issues, I feel the score will climb quite a bit for YSlow and Pagespeed, but the rest of the items to fix require more coding, research, and testing to achieve a higher score. Specifically I will need to:

• modify plugins or hook into the wp_enqueue_* code in wordpress to change to a cookieless domain, however, wp_enqueue_* functions do not have a filter or action chain which I can use to modify the URL to a cookieless domain. In essence the WP_Dependencies class in WordPress would need to be modified to allow such filters. Not everything can be handled by WordPress Better Minify, specifically elements that require version numbers.
• Modify more plugins to use wp_register_style/wp_enqueue_style to remove inline CSS as necessary
• Modify Better WordPress Minify to better minify CSS/JS as it apparently is not as tight as it could be.
• Find a way to set Expire Headers and Entity Tags for those CSS/JS that contain .php in the name instead of .css or .js
• Find a way to get CSS Sprites to work across all browsers.

Almost there, but still have quite a bit of work to do.

First what do you do if you were hacked (assume working in the directory /usr/share/wordpress, but really wherever your wordpress install lives):

• Make a backup of your database
• Get a complete list of your existing themes and plugins
• Download the latest WordPress zip file
• Disable your existing WordPress installation (copy index.php to some other name)
• Move your old WordPress install from /usr/share/wordpress to /usr/share/wordpress.old
• Make a new /usr/share/wordpress directory and install the latest WordPress into this directory
• Restore your existing database
• Use the default theme
• Compare the Exploit-DB for WordPress against your list of plugins and themes
• Search Google for specific 0-day attacks against your version of WordPress, if any exist patch them as explained
• Reinstall your Theme if not on the above list
• Reinstall your Plugins if not on the aforementioned list, if you cannot find your plugins because they have gone the way of the dodo and become available, then inspect every file for any form of malware and then copy from your ‘hacked’ backup directory. However, if you do not know what malware looks like, then get a professional to help. <== NEW

After a few hours work your WordPress site should be back to normal hopefully without the latest list of exploits.

If you have bbpress also installed, repeat for bbpress as well as the two can be linked together in many ways. <== NEW

If you were not attacked, but wish to prevent it from happening, or if you were attacked how to prevent it from happening again. First, modify your installation just a bit to alleviate further attacks:

• Change your Administrator and Editor passwords to a > 16 character pass phrase
• Replace your NONCE salts within your wp-config.php file by getting a new set from https://api.wordpress.org/secret-key/1.1/salt/
• Ensure no files or directories are world or group writable
• Remove any unused themes and plugins (they can still be attack vectors even when not activated) <== NEW
• Clear any Cache files in wp-content/cache/* or other locations <== NEW
• Change the Permissions on /index.php to be non-writable (chmod ugo-w index.php) <== NEW

Then install some WordPress security plugins. These first are a must:

• Ultimate Security Checker – Provides a list of easily fixable security vulnerabilities in the standard WordPress environment. Some of these end up being hard to pass without modifying other plugins. One in particular is W3 Total Cache, where you need to remove the ‘X-Powered-By’ headers.
• Secure WP – Provides useful tools to disable attack vectors and giving up too much knowledge of your environment.
• WP Security Scan – Use WP Security Scan to verify there are no vulnerabilities in your fresh installation or your database, use this tool to modify your administrative username and database table prefixes from their current settings. Also scan for other mis-configurations
• Clean Options – Go through all your options and remove any orphaned and unused options while also inspecting the options for malware. Not only will this clean up your current options, but also speed up WordPress.
• WordPress File Monitor Plus – This is the most important plugin. Configure it to run every hour or sooner so that you can get a list of changes to your WordPress files. If anything changes, inspect immediately for malware. If anything is added to your list of WordPress files, it could be malware. Simply look at the files for base64 encoded data that looks cryptic and not as plain PHP code. I then inspect my Apache log files for who perpetrated the attacks if possible and block them from happening again using firewall rules.
• WordPress Firewall 2 adds some important but basic request verification against well known attack vectors <== NEW

These three add more logging into your WordPress environment and provide details that takes other knowledge to use:

• WP MalWatch – Provides a way to look for malware in common places, while not always useful, it could expose critical issues.
• Exploit Scanner – Provides a way to look for malware in uncommon locations, takes a bit of Javaascript and PHP coding knowledge to understand the output, but does bring to light certain critical issues.
• WP Cron – Provides a way to inspect those cron jobs WordPress requires to work, ensure there are only expected jobs in the list.

In addition, to the above elements, I also made the following changes:

• Disable the ability to directly access wp-config.php and other files using a .htaccess file (if using Apache)
• Disable the ability to access non-image files from your uploads directory. Granted this only denies reading files of the appropriate extension, but you also need to ensure any upload software inspects for images only into this directory.
• Changed permissions on critical WordPress files to be non-writable by all so that they cannot be modified. However, this does require you be diligent whenever you update WordPress to make the files readable once more, then revert them to non-writable by all.
• Implement a web performance management suite such as New Relic RPM to detect when your WordPress site is accessing external web traffic. In Figure 1, for example we see an increase in ‘Web External’ traffic that represented a hack to a WordPress site. Most malware wants to call home, this one did to a recently created domain. New Relic RPM furthermore allows you to view all External Services your website calls, it is a very good idea to review that list to determine if all external traffic is expected or not. <==NEW
• Sign Up for something like Website Defender to automatically scan the site every few days. <== NEW

Figure 1: New Relic RPM showing External Web Traffic

Given how WordPress sites are written today, some Web External traffic is to be expected, but when there is a sudden increase, it is also time to research to determine why this is happening.

Lastly, I created a development site into which I can install and test any new updates to plugins and WordPress. Until they get tested, they do not get onto the production WordPress installation.  By being diligent and relying on good attack intelligence you can stay on top of the current batch of WordPress attacks as well as prevent them from happening.  It is important to not only to review exploit databases for known vulnerabilities but to also understand how the attacks were perpetrated so that you can investigate and mitigate future security attacks.

We also need to remember to clean up any bbpress installation as well. bbpress, which is forum software, has links to WordPress when you use the two together.