The New Server:

2x Xeon 3.2 Ghz Dual Core (4 cores total)
2 GB Memory
2x 73GB SCSI Drives

Software:

Linux: CentOS 5.0
Apache: 2.2
MySQL: 4.3 (on a separate server)
PHP: 5.1.6
Webpage: CakePHP Framework 1.2.2.8120

Website workload:

95% Read / 5% Write

Benchmark:

Apache Bench: ab -c 20 -n 1000 http://webpage

Base: 8.68 Pageviews/Second

Admittedly the webpage is heavy. I took a stock apache config, and on my first run I got 8.68 pageviews/second.  I was pretty happy with this, however my CPU utilization was 100%.  While this is normally a good thing, I knew that the CakePHP framework was eating up a lot of CPU cycles that were unecessary.  My next step was to fix CakePHP up a little.

CakePHP CacheTime Patch: 9.40 Pageviews/Second

There is a small performance bug in the code where it processes an entire page for caching even though the page won’t be cached.  Here’s the patch.

Even though this patch reduced the code path, we were still doing a lot of processing of code. While PHP is relatively light, it is still an interpreted language. There’s always a way of speeding an interpreted language up. In PHP’s case, it’s turning on APC.

Turn on APC: 18.35 Pageviews/Second

APC caches PHP byte code so it doesn’t need to get interpreted every single run.  This nearly doubles performance and drops the CPU utilization down to 85%.  Which means we have a new bottleneck.  Originally I though it was disk IO, but iostat shows little disk activity.  And since I’m running a local apache bench, it’s not network.  This leaves the database.  Even though the database is running on another server and that server CPU is only 30% utilized, there is latency involved in going to another server.  I figure it’s time to fire up memcached and point CakePHP at it (this proves to be a mistake).

CakePHP using memcached: 17.20 Pageviews/Second

I actually lose performance running cake’s caching through memcached.  After some investigation, I see that it only uses it to store paths, and realize that when using the default file cache backing it’s still sitting in memory and a file open/read is much quicker than making a connection to memcached and pulling the information.  I turn memcached back off for Cake’s caching.

Going back to the drawing board, I turn on database profiling and see that the session management is taking 500 milliseconds! (I had been using the database for storing sessions)  After some investigation, I find that CakePHP has an undocumented way of storing sessions in it’s cache. This time I decide to use the APC cache instead of memcached (using APC for Cake’s internal path/model caching makes no performance impact).  I can get away with APC for session caching since I only have 1 webserver (will have to go to memcached if I ever have more than 1 webserver).

Session Caching in APC: 20.55

The session caching did improve performance.  And it also improves my CPU utilization to 95%.  Still no disk IO.  Looking at the database profiles, I see a few queries which are a bit heavy.  I decide to integrate memcached into my CakePHP models.  Based on some code I found here, I reworked to be able to use in models as this is a more appropriate place than the controller.

Memcache for heavy calls: 21.55 Pageviews/Second

This helped some, and brought my CPU utilization to just over 99%.  While, you would think that this is good enough as it would be able to handle a front page Digg a few times over, I didn’t want to take any chances.  I needed to reduce the code path to get pageviews up.  That’s when I decided to code up “Lock Down Mode”. 

If the server gets hit hard enough causing the load to go above 10, the code will switch over to “Lock Down Mode”.  This reduces much of the site functionality, and makes most pages static instead of dynamic.  It will be good enough for 99% of the users and keep the site up.

Lock Down Mode: 108.70 Pageviews/Second

This seemed to do the trick. That should be enough to handle Digg, Slashdot, StumbleUpon and Reddit all at once.

Conclusion

While my work isn’t done, this is good enough for now. Going from 8.68 pageviews/second to 108 pageviews/second is not too bad. The secret to scaling a website is looking at what the current bottleneck and the figuring out how to address it.

Maybe one of these days I’ll write a post on speeding up the client side and moving to multiple servers.

Disclaimer: Yes, I know that this is not a truly valid test as I ran Apache Bench locally which was taking up CPU resources as well as not hitting the network path. But, it was good enough to show relative performance improvements. Additionally, there were numerous small tweaks along the way that I didn’t mention and many webpage specific performance improvements. Before all the website improvements, the “base” number was probably closer to 5 pageviews a second.

• Turn back
• Get through it by any means necessary
• Get to a vantage point to see if the reward is worth the effort to overcome the obstacle

For a long time, I was very much the second person.  Call it an ego thing, call it stubbornness, but I did whatever it took to achieve a goal.  Sometimes at great sacrifice to myself.  I don’t believe this is necessarily a bad thing, as I think we all need to build tenacity.

However, after the past few years, I’ve come to change my thinking.  I’ve come to value the “resources” it would take to acheive some goals, and I realize that often a goal is not worth the sacrafice.  So, before I dedicate my time and energy, I put some due dilegence into making sure the outcome is worth it.  This saves unneccessary effort.

The takeaway from all of this is to keep the end result in perspective and realize what it’s value is.

The first thing I do is look at the new technologies to see if any of them are viable as a productivity enhancement.  If one of them is, I’ll try it out and integrate when appropriate.  The second thing I do is learn something ancillary to my current area of focus.  This usually is something which is not required to perform my job, but could aid tremendously.  By doing this “homework”, when you’re back in crunch time, you’ll be greatly more effective.

For example, when I was doing a lot of coding, I’d spend my downtime improving my .emacs file so I could code much more effectively.  For ancillary learning, I learned about proper interviewing techniques.  This came in very handy when I was interviewing people to join my team as I had some of the best hires of the group.

The average person career is able to rise above the brilliant person’s when they are able scale more effectively.  To scale effectively it requires the ability to delegate.  If one of your direct reports is able to perform a job that you are performing at the same quality, then you are “paying” too much to get that task done (assuming you make more than your direct).  Instead, it should be delegated.  With that extra time, you should be able to perform the more important tasks to a higher quality, or else take one of your boss’ tasks.

As a person moves through their career it becomes less about how brilliant you can make a presentation, but rather about the people under you who will determine how far your career goes.

Plant the “promotion seed” in your manager’s mind far before you think you’re up for a promotion.  This is done by simply asking your manager if you will get a promotion at the next opening.  While it may seem innocuous, by having them think about you getting a promotion, you have planted the seed for consideration of a promotion.

Managers have an internal clock of how long they need to consider someone for a promotion before they act on it.  By making them “consider” you earlier, you have started that internal clock sooner.  This has the double effect of additional pressure on a manager knowing that you expect a promotion from them.  If there is no pressure, then the manager thinks you’re okay with your current level and has no guilt about not promoting you.

Do this whenever you get a new manager, or right after you just got a promotion.  For example, I waited 2 months from getting a promotion before I planted the next seed.

• DISC - Helps with human interaction to understand how _other_ people try achieving success
• Twin Pillars - Marketing in its most simplistic form. However, applicable to daily life
• “Strategic Focus” - Strategy in it’s simplest form. Can break down a company quickly with this

DISC: (Dominance, Influence, Steadiness, Conscientious)

Once you understand how someone goes about achieving success, you can understand how to speak to them so they “hear” you. I love this model because it’s easy to see each of these traits and it is not nearly as complex as Myers-Briggs. For example a dominant personality, will want to achieve success for prestige. So, to influence them, you tell them that by doing something, they will look good to the rest of the organization.

Twin Pillars - Segmentation and Differentiation

Marketing doesn’t get any more basic than this.  To be successful, identify customer needs (segmentation), and then deliver more value to the customer (differentiation) than the competition.  While it seems simple, there can be a large drill-down into each of these pillars.  From my experience, this concept is lost to most technical people.

Strategic Focus

This model says that a company’s strategy must be either operational efficiency (a Dell/Walmart), product innovation (Google), or customer intimacy/relationships (most local businesses).  A good company will focus on one, and do it very well.  If a company is lucky it can do 2.  However, no company can do all three well.  The model says that depending on what a companies strategic focus will determine what it should invest in.

1. Wealth without work
2. Pleasure without conscience
3. Knowledge without character
4. Commerce without morality
5. Science without humanity
6. Worship without sacrifice
7. Politics without principle

The value of models is that it allows a person to simplify a lot of information down to something which can be retained in an individual’s head. By “freeing up space” they can focus on the next set of information. This allows a person to retain much more data than they could otherwise. More data leads to a better decision.

The additional benefit of a model is that when another person knows the model, communication is much crisper and clearer. The other person does not have to churn on categorizing what you are saying, but instead can focus on what your point is.

Models also allow a person to jump up the experience ladder when thinking about a situation. They can do an accurate strategy analysis for the first time using SWOT. Not using a model would require years of experience to do just as well.

In MBA school, I learned a ton of models. For the most part, they were worthless because they we too complex for me to keep straight in my head, or else they were too specialized to be applicable in the majority of situations.

A truly great model has the following attributes:

• Simple - 3 or 4 categories tops
• Covers the majority of case - There will always be corner cases which models miss
• Others can understand it quickly - If you need to teach it, it’s quick
• Requires little experience to apply

A great model will not cover the every situation, but it will certainly reduce the need to start from scratch in decision making.

Next time I will cover my favorite models and touch on some of the worse models.

I finally wrapped up MBA school and spent the last few months getting my life back in order.  I’m now ready to get back to blogging.

This blog will now focus on:

• Human Behavior - Understanding why we do what we do
• “Little Things” - Items in life that make a huge difference
• Technology - Trends, Performance, and Emerging Technology
• Entrepreneurship - Business Ventures

Additionally, I will try keeping the posts concise and go for quality over quantity.

I’m looking forward to it.