I asked a veteran recruiter who has worked for many placement firms and now runs his own small firm about this.  Specifically, I asked him these questions:

Are you getting taken advantage of if your salary is 110k and you are billed to clients at $150/hr?  Is it better to go 1099/W2 instead?  When you were working as a recruiter for the big consulting companies, did you try to steer consultants to go perm instead of W2 or 1099 because you would make more money off them?  Where consultants laid off if they were on the bench more than a couple of weeks?  Did you place less-qualified people on projects because they were the only one’s available?

We never cared if a person was salaried or a 1099.  I guess a salaried person is more likely to stay with you longer because it is a pain to change benefits (especially if they are really good benefits) when you change jobs, but other than that, the percentage spreadsheets I mentioned before (see Billing rates – other side of the fence, part 2) always spit out a number and it didn’t have a bias towards salaried vs. 1099.

As for the specific question about a salary of $110K on a bill rate of $150/hr, it depends on the benefits (for example, are you taking insurance for a single person, a married couple, or a family?).  Also, what other benefits come with the salary? 401(k) match?  Tuition reimbursement?  Utilization bonus (if you work over a certain number of billable hours in a calendar year, do you get a bonus?).  I’d need a clearer picture of what comes with the $110K salary to know if it is a bad deal for the consultant.  I’d think at a rate of $150/hr the vendor could afford a higher salary, just my gut though until I know more about the vendor benefits.

Also, this type of stuff applies across different industries.  I know a person in a law firm that has an hourly billable rate of $380/hr.  She has a nice salary, but it isn’t anything out of this world.  They scoreboard the entire firm to show how much money you bring in through your billable hours, how much new business you won, etc.  There is a lot hidden behind the curtain in all industries, so it is hard to compare apples to apples on these things sometimes.

As for bench policy, it was always a case by case basis.  Has the person been with the firm for a long time?  Meaning, has the firm made a decent profit on this person over the years.  At the $17B firm where I worked, there was a lady that had been there 30 years.  She was rarely on the bench, but if she ever did need bench time, she had earned it and it was there no matter what.  A general rule of thumb is a week of bench per year of service.  Also, the person’s skill set was a factor – was the person in a marketable space (Java people in some cities are gold, you’d give that person bench no matter what because they’d be out on another assignment for you very soon.  On the contrary, if the person was a mainframer, that work isn’t as hot in most cities after Y2K, so that person probably had a shorter shelf life.  One vendor (a $1B company) was a huge mainframe shop and after Y2K they had a ton of employees that were no longer billable.  They kept them all on the bench and retrained them in Java.  It was a huge investment/loss to the vendor and Wall Street killed them for the move and their stock price sank.  So even when they “did the right thing” and took care of their people, Wall Street slammed them.

This story was told to me: A client is looking for a contractor and contacts a staffing firm to find candidates.  The staffing firms sends a resume that looks real good.  A technical interview is done via the phone, the candidate really knows his stuff and nails the interview, and he is hired.  A few weeks later the new hire arrives and begins work.  But something does not seem right.  The contractor does not seem to be that sharp, nothing like he was during the interview.  His coding skills are not that good.  After a few weeks, the client does some digging, and it is discovered this person is not the person who did the interview!

It’s called the “bait and switch”, and the hope is the client remains unaware it is a different person.  I’m guessing sometimes they get away with this switch, but you can imagine how upset the client is when they find out they have been tricked.  It seems the staffing firm is tricked also. This happened twice, and both cases involved sponsored candidates from India.  In one of the cases there was a group of seven who all lived together and one of them was really sharp (the “ring leader”) and would do all the interviews for the other six, who were junior-level.  The resumes had the real name of the person but the experience was the ring leaders.  Also, when the junior programmers were placed at a client the ring-leader would help out the junior programmers if they were struggling on the project they were on.  And the client was none the wiser.

This then happened to me, and shows the extent some people will go to trick the client.  The client I was at did a phone interview on a candidate that I participated in.  The candidate did very well in the phone interview.  Since the client had experienced the same bait-and-switch as above, the next step in the interview process is a required a face-to-face meeting.  So they did a video interview on Skype, were we used Skype for the video and used a land-line for the voice.  Once again the candidate did very well but….it turns out, we were talking to a different person on the land-line than who was on the video!  We got suspicious when we saw how the video and voice were so out of sync, and doing some more digging turned up they were in fact different people.  The guy on the video was pretending to be the guy talking on the land-line.  Crazy!

I asked my recruiter friend about this, and he replied:

I saw it happen more often earlier in my career, not so much now (but it still does happen on rare occasions).  Detroit in particular had a ton of foreign national firms set up show here in the 90’s and when this “bait & switch” became a trend, the Big 3 put an end to it.  They’d make candidates give some form of ID (SS# or some other identifier).  There would also be harsh penalties such as removal from the vendor blanket for firms that repeatedly used this tactic.  The bait & switch is a rookie move.  The staffing world can be a greasy business – unfortunately nothing really surprises me these days.

Some placement firms use a fixed hourly rate markup, but many use a markup percentage.  One placement company I am talking with does just that, and they like to have a 68% mark up.  I can never understand why they use a percentage for mark up, especially one that high.  After all, 68% of $50/hr and 68% of $150/hr is a $68/hr difference in margin!

In talking with a recruiter who has been in the industry a while, and is now a recruiter for his one-man shop, here was his response:

My answer is that each deal should be dealt with on a case-by-case basis.  Percentages are dangerous for the reasons you mentioned – and that goes both ways.  The candidate is punished when the rate is high, the vendor is punished when the rate is low.  Having said that, most mid to large vendors (probably those at $50M or more) will have a percentage based formula where they bake their fixed costs into an Excel spreadsheet (insurance, rent, management overhead, vendor mgmt. system fees, etc) and use a percentage system to evaluate a deal.

I’ve worked in a few decent sized companies.  At those firms we used excel spreadsheets pre-populated with the fixed amounts mentioned above (rent doesn’t change often, maybe annually, same goes for overhead, vendor fees, etc).  In my experience, the spreadsheet was probably updated once a year, but the fixed costs are essentially left alone for the most part.  The spreadsheet allowed for two variable data entry points: (1) employee cost and (2) the client bill rate.  Employee cost was basically a candidate’s salary (or hourly rate) and health benefits (how many people are being insured – single, married or family benefits).  The client bill rate is what it is, the rate the end client is paying the vendor.  These two variables were set against the fixed costs mentioned earlier and the end result was a percentage.  There were guidelines as to what percentages were acceptable (if above “X” percent, the sales guy gets a better cut, if below a certain percent, the sales guy wouldn’t make anything on the deal).

Publicly traded companies definitely follow this model because they have to report their quarterly earnings to Wall Street and these firms are typically graded on their margins.  So management will bake in rewards or penalties for the sales/recruiting teams to keep everyone rowing in the same direction.  So when a sales guy says he can’t do a deal, it is probably because the deal is at a certain margin where he’ll get punished with minimal commission.  Some percentage systems involve a sliding scale that accounts for how high/low the bill rate is and higher rate deals would allow for a higher percentage to go to the candidate.  This makes sense for the reasons you mentioned (20% of $50 if far different than %20 of $150 – and things like rent, insurance, vendor fees are all fixed costs, they don’t go up or down based on the amount in the bill rate).  Having said that, some factors (insurance, workers comp, unemployment) do go up with a candidate that has a higher compensation, so those are some things to keep in mind.

All of the above basically applies to bigger and/or publicly traded companies.  It doesn’t apply to the small guy – at least in my mind it doesn’t as we don’t have the overhead, the same pressure from Wall Street or from a parent company – we don’t answer to anyone but ourselves.  For me, my rule of thumb is pretty simple: I have a “walk away” point where I won’t do a deal if it falls below a certain NET hourly amount.  I don’t care about the percentage.  I care more about the bigger picture:

- Who invested the most time in making the deal work?
- Does the vendor have some additional value (for example, are they a preferred vendor?)
- Supply and demand (and market rates) also come in to play (did I find this guy in a day, or did it take me 6 weeks?)
- Did the candidate bring the deal? (and did the vendor do nothing?)
- Dd the client give a low bill rate?  Is there much room to work with?

If I’ve spent a lot of time cultivating a relationship with a client, to the point that when I make a suggestion, my candidate gets to cut to the front of the line and basically gets an interview on my recommendation – now I have value and my margin reflects that.  Additionally, if there turns out to be multiple candidates available for that role, then that also comes in to play.  If I talk to 5 candidates that have the same skill set (and assuming all things are equal) then it comes down to which candidate is cheapest.  My rate will go up and down some to show the end client who is more expensive but generally the lower cost candidate gives a bigger margin.  I guess I work backward and factor in the above points – since running my own firm I have never (and will never) start with a percentage in mind. It is all of the above (and some other things that I’m forgetting) that factor in to what type of a NET margin I’ll accept, then a percentage comes out of it and I look at it more as a reference point, nothing more.

And to put that in perspective, I don’t charge clients $150/hr then pass on $60/hr to my teammate and keep $90/hr for myself.  My teammates make the lion’s share of the bill rate, and my hourly NET is still a nice little number that supports me and my family.  I live by the rule that “pigs get fat, hogs get slaughtered”.  Take what you need to live a nice life, but don’t get greedy because it’ll eventually catch up with you.  I want candidates that have worked with me to feel that they got what the deserved.  It leads to future referrals and repeat business with candidates.  It is a small world, no need to get greedy…

For percentage markups, it seems that a common ground of was 80/20 (in favor of the consultant) was a good deal and 70/30 was more the norm.  In my opinion, the ratio slides from bigger companies needing more (like 60/40 or 65/35) to smaller companies who can tolerate 80/20.  The reason I say this is that I lived it first-hand 19 years ago and am living it again now.  When I stated my own career in 1994, it was for a midsize company (a $400M company that went through an IPO, they are still doing well to this day).  When they were a smaller company (pre-IPO), they signed up a lot of employees for 80/20 deals.  As the company grew, and the company overhead, infrastructure, service offerings, sales/recruiting force, etc. grew, those once “decent” 80/20 deals became the company albatross, putting a serious drain on the overall company margins.  It was my job to meet with the employees and try to re-work those 80/20 contracts.  The message to the employee was “don’t worry about the split anymore, worry about  your Net hourly take home pay”.  If you were making $40/hr on a $50/hr bill rate, don’t focus on maintaining 80% of the new bill rate, instead give me a new hourly number (you are at $40/hr, do you want $45/hr?) and let me worry about the hourly bill rate to support that.  Then I had to go to the client and fight for whatever bill rate increase I could get – and I was challenged to get it away from an 80/20 margin because the company had outgrown that.  So sometimes focusing on the percentage or the split can be the wrong way to look at it.  And for the past four years running my own small firm, I never look at it (other than as a reference point).  As I said above, I have a walk away point based on a NET hourly number that starts my negotiation (or ends the negotiation) and I go from there.  The many different factors (duration, amount of work, supply and demand, amount of money tied up in the deal, risk) all go in to what number I’ll tolerate.  It is a case-by-case situation – at least for the small guy.

To answer about the 68% markup: For a large firm that seems pretty high.  Note that W2 employees have FICA, FUTA, SUTA, Liability and Workers’ Comp which could arguable account for roughly 20% of the margin.  But even after you take off the 20% and get it down to 48%, that is still too high in my opinion.

But all those tasks can be done much easier using DQS, which also has a lot more features available.  DQS enables you to build a knowledge base and use it to perform a variety of critical data quality tasks, including correction, enrichment, standardization, and de-duplication of your data (see Data Quality Services Books Online).  Or you can perform matching manually within the Master Data Services Add-in for Excel which leverages the matching functionality in Data Quality Services.  And there is a DQS cleaning transformation that you can use in SSIS (see Overview of the DQS Cleansing Transform).

MDS has limited data cleansing via business rules which can apply default values and change values.  The best approach is to use DQS to clean the data from the source and then copy that data into MDS.

More info:

Data Conversion in SSIS

Cleanse and Match Master Data by Using EIM – Microsoft

Placement companies (staffing firm, recruiting firm, consulting firm…they go by many names) are needed.  I have used them many times to find projects.  They deserve a cut of the margin, as a lot of unseen work goes into them finding that open position at a company and contacting you about it.  It is a question of how much of a margin is fair?  And how much do they really need to “break even”?  I have seen independent recruiters happy with $15/hr margins, and I have seen consultants getting taking advantage of (i.e. making a salary equivalent of $50/hr and the client bill rate is $250/hr and in addition the consultant has to travel 100%).  And I can never understand why some placement companies use a percentage for mark up.  After all, 68% of $50/hr and 68% of $150/hr is a $68/hr difference in margin.  Most of the big placement companies say they need around $40/hr off of W2 rates to break even.  The last placement company I talked to said to break even they need 4% of 1099/C2C and 17% for W2.  I asked a veteran recruiter who has worked for many placement firms and now runs his own small firm about this:

Short answer:
$40/hr seems a little much to “break even” and I’d question it.  Having said that, there is a lot of hidden cost in running a big firm, so that number isn’t a far off as you’d think.  I’ll try to detail it as best I can in the “long answer” below.  On the other end of the spectrum, where the little guy doesn’t have nearly the overhead (personally my overhead cost isn’t in the big guy’s stratosphere) but the little guy has a lot more at risk.  A bad deal (read: bad client) or a series of late payments, etc – those are killers.  If you have a small placement company, you can probably appreciate the pain of paying your teammates on time and a NET 30 day payment stretching out to a NET 90 day payment and beyond.  You become a bank and that isn’t fun.  So the little guy doesn’t need the same mark-up, but there are some variables that impact them differently.

Long answer:
These thoughts are based on 19 years of personal experience in which I’ve worked for a big firm (multi-billion dollar) a few small firms (obviously myself, but I’ve also worked for two other small firms) and the middle guy ($400 million or so).

(1) From my experience with the big/middles guys (from working with (a) ConsultingCompanyA at the time I was there it was a $17 Billion dollar company and from working with (b) ConsultingCompanyB, which went through an IPO and was in the $400 Million space when I was there.  Here are the things that influenced mark-up:

• Office space/physical infrastructure
• Internal office employees (sales, recruiting, managers, regional VPs, payroll, support staff, executives, etc)
• Insurance, payroll tax, FICA, unemployment costs (for W2 people)
• Legal/internal fees for H1-B candidates (visas, filings, support)
• Advertising
• Tools/software
• Cost of being a preferred vendor.  Some large companies actually make you kick-back money, in the range of 3.5%.  Plus, in some industries there are “rate cards” which put max rates on certain skill sets which forces all the vendors to bid against one another and makes the business a commodity play.  It negatively affects the big guy because their max rate is set, they cannot go higher than the maximum rate on the card, and then a candidate can play one firm against the other for the best deal and it becomes a game of chicken between the big firms on who will take the lowest margin

(2) From my experience of working with smaller firms, and from where it really hits home – working for myself – here are the things that influence mark-up:

• Some of the above (insurance and advertising for example)
• Banking/Loans
• Payroll: this is a killer if your clients don’t pay in a timely fashion.  As I mentioned in my “short answer”, you realize that you’re a bank and the “Banking” point above becomes a larger influence.  The current people you have billing fund future people billing.  If that cycle gets disturbed, it is an issue
• Risk: if a client has had delayed payments in the past, I’ll mark up my next deal to them higher to account for that risk
• Skill set: if a skill set takes a long time to staff, that mark-up will be higher
• Rarity of a skill set: in some cities, Java talent is impossible to find.  I’m not going to give that away at a lesser mark-up as an easier skill set to find like a Business Analyst
• Level of skill set: I am working with a client to staff their CIO position.  There are very few people in this talent pool and they are coming from my personal Rolodex – something which I hold near & dear to my heart.  That calls for a much different mark-up (and this segues in to a conversation about perm/direct placement vs. contract vs. contract to hire).  You can see where the subject of “mark-ups” covers a lot of stuff and can get confusing/complicated
• Is your candidate on someone else’s payroll: I have a few partners where I get talent from.  When they take care of the payroll responsibility, some of the points above are eliminated
• The cost of the resource: Does the candidate cost $125/hr?  Is the person a $40/hr resource?  One ties up more payroll, incurs greater risk if the client doesn’t pay on time, etc
• The duration of the opportunity: Is it a 3-month assignment or a 12-month assignment?  The shorter the gig, the higher the mark-up
• Benefits: does the candidate require them?  Is it for one person or a family?

When a consultant brings the solution to the staffing firm, the rules change.  For example, when you already know the end client/hiring manager and they will pick you without an interview and you just need to select a firm on a preferred vendor list, a majority of the stuff on the “big guy” mark-up list drops off.  You didn’t need their offices in other cities to land this position, you didn’t need their recruiting/sales/VPs/etc, you didn’t need their H1-B process, all that cost gets dropped from the equation in my opinion.

If you are that person that the end-client wants to hire and you need to go through a preferred vendor, I’d go to the end-client and ask their purchasing dept who is the smallest of the big guys (or the easiest ones to work with).  Sometimes a boutique firm is on the list, not a high volume churn company.  Some large companies have to have a certain minority spend, so sometimes there is a little guy in the form of a minority vendor.  They’d love to have a deal fall in their lap, it is found money and they’d be very willing to work with you on a lesser mark-up.  If you came to me and said “CompanyA has this position, I know the manager and he already said he’d take me, I just need to find a preferred vendor” – we’d have a very open dialogue where I’d point out my fixed costs (I have a 3.5% rebate I need to account for, I have internal finance people who have to run your paychecks, etc) and we’d weed out all the stuff that doesn’t impact you.

Here are the reasons I blog:

• I can document solutions I encounter for future use.  Sort of my own personal Google
• It helps to improve my writing skills
• I learn new technology by blogging about it.  The best way to learn is by teaching, especially when I don’t know enough about a topic
• To raise my personal brand
• It’s fun!
• To prove to clients I know my stuff (credibility)
• I enjoy sharing knowledge
• It helps me to remember the things I blog about better because I am researching it and writing it
• It’s a way for recruiters to find you
• It’s a way to become “known” in the BI community
• It’s a big plus if you are trying to become a SQL Server MVP
• It helps as a consultant when I can use my blog as a solution to a customer’s problem
• I use it as a replacement for client documentation.  They want you to document a solution, a work-around, etc.  Don’t just write-up something in an email or Word doc…blog it!

Creating a blog is real easy.  It took me only one afternoon to learn how to blog and to post my first one.  I use the web hosting company BlueHost (cost is $5/month), but you can find others at WordPress.org.  You can also use WordPress.com for free, but by using a web hosting company you get more WordPress features (more control of your site, use of more plug-ins).  For a comparison see WordPress.com and WordPress.org, WordPress.org vs WordPress.com – Which One Should I Use?, WordPress.com or WordPress.org? Which One’s Right for You?, and WordPress.com Or WordPress.Org…That Is The Question!

Note you can use your own domain name (i.e. www.jamesserra.com) when using a web hosting company as well as with WordPress.com (For $13/year…see Add a Domain).

These are the main reasons I hear why people don’t blog, and I said the same things before I finally started blogging.  Here they are, with my answers as to why they should not prevent you from blogging:

1. “I can’t think of anything to blog about”.  I always have “Can I make this into a blog” in the back of my mind.  I keep a list of topics in OneNote (shared among all my computers as well as my iPhone).  That list quickly went from 5 items to 20, to 50, to 100, and now it sits at about 300.  Once you get in that blogging mindset, you will have plenty of topics.
2. “I don’t know what to write about that others will find interesting”.  Not every blog you write has to be interesting to everyone.  Some blogs lots of people will find interesting, others only a few.  So what?  If you blog often, just about everyone will find at least some of your blogs useful.  Even if you are new to SQL Server, blog about what you learn or a problem you solve, as other people who are new to SQL Server will find your blogs helpful
3. “I worry that I will plagiarise other blogs”.  I have written about 300 blogs, and there are maybe 3-4 blogs I wrote that where “original” and no one else had written about at the time.  My blogs are usually a mash-up of other blog posts I have read on a given topic.  Written in a way that I understand, and hopefully others who are like me will understand them better.  Almost every blog out there was covered by another blog.  But maybe my way of writing helps someone to understand a topic better than other blogs.  Or maybe someone found an answer to a problem on my blog that was also answered on other blogs but those other blogs did not show up on search results
4. “I fear I will be vulnerable to comments\criticism.  After all, everyone in the world can see my blog and comment on it”.  Well, in the 2+ years I have been blogging, I have written about many things that I am not an expert on, and no one has viciously attacked a post of mine.  And if they did, I can always choose not to post their comment.  But hey, I know I may post something that is inaccurate, no matter how much I research a topic.  No big deal, if it’s pointed out I just correct my blog post.  That is the whole point of posting anyone, to share knowledge and have the help of others to correct any mistakes

Some tips:

• Blog consistently, and at least once a week.  For the first year I blogged three times a week, and now blog twice a week
• Create a twitter account if you don’t already have one and use the option in WordPress to publicize your blog post to twitter when it is posted.  This will get more visits to your blog site
• Queue your blogs.  This way, in moments of inspiration, you can write a bunch of blogs and wait to publish them.  Schedule them for the same days and times (i.e. every Tuesday and Thursday at 10am).  Never do weekends
• I frequently go back and edit posts, adding content and links after reading a good article/blog post on the same subject
• Blogging can be a way for logging problems and solutions for clients

The WordPress add-ins I use:

• Akismet: Used by millions, Akismet is quite possibly the best way in the world to protect your blog from comment and trackback spam
• All in One SEO Pack: Out-of-the-box SEO for your WordPress blog
• BackWPup: is a very easy way to do a complete backup of your entire WordPress site
• Feedburner Email Widget: Allows you to add a Feedburner Email Subscription widget to one of your sidebars
• flickrRSS: Allows you to integrate the photos from a Flickr RSS feed into your site
• Google Analytics Dashboard: Google Analytics graph integration
• Google Analytics Plugin: Optimized Google Analytics Plugin for WordPress
• Google XML Sitemaps: This plugin will generate a special XML sitemap which will help search engines like Google, Yahoo, Bing and Ask.com to better index your blog
• Jetpack by WordPress.com: Bring the power of the WordPress.com cloud to your self-hosted WordPress. Jetpack enables you to connect your blog to a WordPress.com account to use the powerful features normally only available to WordPress.com users
• MobilePress: Turn your WordPress blog into a mobile website/blog
• Primary Feedburner: Redirect your website feeds to feedburner
• Social Media Widget: Adds links to all of your social media and sharing site profiles. Tons of icons come in 3 sizes, 4 icon styles, and 4 animations
• Subscribe to Comments Reloaded: Subscribe to Comments Reloaded is a robust plugin that enables commenters to sign up for e-mail notifications. It includes a full-featured subscription manager that your commenters can use to unsubscribe to certain posts or suspend all notifications
• SyntaxHighlighter Evolved: Easily post syntax-highlighted code to your site without having to modify the code at all. Uses Alex Gorbatchev’s SyntaxHighlighter
• WordPress Editorial Calendar: The Editorial Calendar makes it possible to see all your posts and drag and drop them to manage your blog
• Yet Another Related Posts Plugin: Returns a list of related entries based on a unique algorithm for display on your blog and RSS feeds
• Weaver theme

More info:

Rock Stars, Normal People, and You

How to Start a Blog

Blogging for a Living

The Importance of Regular Blogging

The One Tip That Took My Blog To New Levels. Stop Caring.

THIS BLOG POST SHOWS I STOPPED CARING

Want a great blog? Should you stop caring or stop being afraid?

Blogging for Beginners

Why do we Blog? Looking back …

Writing a Technical Blog: Why to do it and what to write about

So you want to blog about SQL Server?

• They bring experience from prior projects on what works and what does not.  So you avoid common mistakes and do things the right way
• The know all the tools to use for shortcuts.  You can spend days or weeks writing your own solution, or you can be made aware of a tool that will do the job quickly and better
• You will receive architecture decisions that are correct, hence saving the large cost of having to “redo” a project because of an inexperienced architect
• They code faster.  I have seen coders who are 2x-4x faster than the average person
• They do not need supervision: They can work on their own and ask few questions, so you are not wasting your time or saying “I would have just done this myself if I knew I would be spending all this time answering questions”
• They can bring in colleagues who are good. Need more resources on a project?  Ask the independent consultant who he has worked with that would be a good fit for the current project.  Would you rather use that person or ask a placement firm to find someone?
• They have excellent people skills, which is the #1 priority for a succesful project
• They are much more efficient with their time.  They have learned over the years to maximize their time and get the most work done in that 8-hour day.  They are the masters of time management
• They work hard.  To feel like they are earning their rate, they put in the extra effort and spend non-billable hours keeping up with technology and learning so they can make sure they are finding the best solutions for their client
• They know colleagues they can reach out to help solve problems or bounce ideas off of.  They have built relationships over the years with other experts, so they can get advise to make sure their client’s project is successful
• The have experience with many tools so know the best one for the job.  The biggest project failures are the ones where the wrong technology or tool was chosen from the start.  Experienced consultants will have been on a project similar to yours and will know which tools made that project successful
• With an independent consultant you get them on the job instead of a large consulting company doing a bait and switch (showing you impressive resumes who will work on your project, but then replacing them with much-less experienced consultants because there are the only ones available)
• They are still a lot less expensive than the big consulting companies.  And chances are the independent consultant will be better than what you will get with a big consulting companies because guess what happens to the top talent at a big consulting company?  They become highly paid independent consultants!
• The end result will be the project is done quicker and with a better solution.  So the extra money spent on a higher-priced consultant will likely result in bigger savings to the company

As BI consultant and longtime colleague of mine Garrett Edmondson pointed out, there are some inherent risks as well. 

It can be very difficult for an organization to determine if the consultant (independent or otherwise) actually has the required skills because in many instances the organization lacks those skills internally.  Otherwise, why would the consultant be needed?

You should own your organizational deficiencies and enlist the help of a very knowledgeable/expensive consultant to help fill either a fulltime or contract position.  A little investment on the front end can yield tremendous results.  Also, the very knowledgeable/expensive consultant can train and mentor a junior BI developer by actually implementing a solution with him. 

Just because someone comes from a large consulting company does not necessarily mean that they have the required skills.  In my experience most large consulting companies actually have very small DW/BI practices, and that void is filled in by smaller consulting companies that specialize in DW/BI.

A few things can be done mitigate the risks:

In my humble opinion, a true data warehouse/ BI expert is worth their weight in gold.  Ask for references and then actually contact them!

Most hardware/software vendors have a list of recommended partners.  They are usually on the recommended list because they have been successful in the past, which might be a good indication of future results. 

While it might have been possible in the past to have one guy know it all when it comes to BI, I don’t think that is still possible.  Do you really think that someone can truly be an expert in all of the following?

• Master Data Management – MDS
• Data Cleansing – DQS
• ETL – SSIS
• Data Warehouse SMP Hardware Architecture (all IO components see: FastTrack)
• - oRAID/LUN/FILEGROUP/DATABASE FILE configuration
• - HBA, Fiber Channel Switch, RAID Controllers, RAID enclosures
• Data Warehouse MPP Architecture – PDW
• Dimensional Modeling
• Data Warehouse Life Cycle
• DW project management
• Unstructured Data – HDFS
• MapReduce – Java
• Unstructured Data Hardware Architecture – Hadoop Cluster
• Data Mining/Machine Learning Algorithms
• CRISP-DM Methodology
• MDX
• SSAS Cubes
• Reporting – SSRS/PowerPivot/PerformancePoint
• Geospatial data
• “Cloud” computing
• In memory/Columnstore technologies….

I could easily double the size of the list.  It simply is not possible to know it all when it comes to BI and finding an organization that truly has expertise in all areas of BI is rare.

From Programmer to Business Intelligence Expert

Many years ago, when I would attend conferences, I would just go back to my hotel and not bother meeting anyone.  Big mistake.  Over the years I have seen the value in meeting people.  So, I make it a point to pre-arrange as many breakfasts/lunches/dinners/beers/coffees with people I would like to meet.  Since this was a BI conference, I tried to meet every BI expert in the industry, and nearly did so!  Not only did we share great insights into the wonderful world of BI, but I shared personal stories with many who I now consider friends.  And I look forward to meeting up with everyone at the next conference.

Below are all the people I met (43, and I know I missed a few names – sorry!), with many of them taking the time out of their busy day to chat with me, including a number who I shared a meal with.  How many people did you meet at your last conference?

Thomas LaRock
Douglas McDowell
Mark Tabladillo
Artus Krohn-Grimberghe
Peter Myers
Drew Minkin
Rod Colledge
Rob Collie
Mark Ginnebaugh
Bill Graziano
Karen Lopez
Josh Fennessy
Chris Webb
Kevin Kline
Jen Stirrup
Adam Jorgensen
Paul Turley
Craig Utley
Marco Russo
Brian Mitchell
Brian Walker
Rob Kerr
Scott Stauffer
Amy Lewis
John Hoang
Stacia Misner
Erika Bakse
Julie Koesmarno
Hope Foley
Neil Hambly
Devin Knight
Dan Clark
Angel Abundez
Bob Bloem
Ted Krueger
Greg Kramer
Bob Phillips
Aaron Nelson
Jason Thomas
Jorge Segarra
Todd McDermid
Dave DuVarney
Wendy Pastrick

More info:

PASS Business Analytics Conference Day 1 Takeaways and PASS Business Analytics Conference Day 2 Takeaways

PASS Business Analytics Conference Keynote Day #1 and PASS Business Analytics Conference Keynote Day #2

PASS Business Analytics Conference Summary

Day 1: PASS Business Analytics Conference, The Art of Analytics and Day 2: PASS Business Analytics Conference, New 3D Mapping Analytics Tool for Excel and Day 3: PASS Business Analytics Conference, Analytics in Action

PASS Business Analytics Conference (BAC) Recap

PASS BA Conference Key Note Day 1; “FUN has its own ROI”

PASS Business Analytics Conference–Recap

PASS BA Conference: Interview with the PASS Board

More info:

Microsoft Talks Analytics, Introduces New Preview of BI Technology

Public preview of project codename “GeoFlow” for Excel delivers 3D data visualization and storytelling

Geoflow? Is that it, Microsoft?

Download GeoFlow Preview for Excel 2013

Public Preview GeoFlow for Excel 2013

GeoFlow Released for Public Preview

Exploring Excel 2013 for BI Tip #6: GeoFlow–The Latest Excel Visualization

GeoFlow Public Preview Available

Day 2: PASS Business Analytics Conference, New 3D Mapping Analytics Tool for Excel

Telling Even Better Stories with Microsoft BI and GeoFlow

Public Preview GeoFlow for Excel 2013

Analyze a Twitter feed with Excel 2013, DataExplorer and GeoFlow

From Data to Insight: Getting Public Data with Data Explorer & Building a Power View & GeoFlow

Here are the PowerPoint presentation: What exactly is Business Intelligence? and Building an Effective Data Warehouse Architecture.

What exactly is Business Intelligence?

BI is the “Gathering of data from multiple sources to present it in a way that allows executives to make better business decisions”. I will describe in more detail exactly what BI is, what encompasses the Microsoft BI stack, why it is so popular, and why a BI career pays so much. I will review specific examples from previous projects of mine that show the benefits of BI and its huge return-on-investment. I’ll go into detail on the components of a BI solution, and I will discuss key concepts for successfully implementing BI in your organization.

Building an Effective Data Warehouse Architecture

You’re a DBA and your boss asks you to determine if a data warehouse would help the company. So many questions pop into your head: Why use a data warehouse? What is the best methodology to use when creating a data warehouse? Should I use a normalized or dimensional approach? What is the difference between the Kimball and Inmon methodologies? Does the new Tabular model in SQL Server 2012 change things? What is the difference between a data warehouse and a data mart? Is there any hardware I can purchase that is optimized for a data warehouse? What if I have a ton of data? During this session James will help you to answer these questions so your response to your boss will provoke amazement and lead to a big raise. Or at least help to lead you down the correct path!

Here are the PowerPoint presentations:  Overview of Microsoft Appliances and Building an Effective Data Warehouse Architecture.

Overview of Microsoft Appliances

Learn how SQL Server 2008 can scale to hundreds of terabytes for BI/DW solutions. This session will focus on Fast Track solutions and appliances, reference architectures, and Parallel Data Warehousing (PDW). It will also cover performance numbers and lessons learned from one of the very first production PDW deployments in the world and how a successful BI solution was built on top of it using SSAS.

Explore all the different appliances – HP Business Decision Appliance (BDA), HP Business Data Warehouse appliance (BDW), HP Enterprise Data Warehouse Appliance (EDW), and HP Database Consolidation Appliance (DBC) – and how they can save you significant time and money instead of building on your own. If you’re involved in the decision making for purchasing one or more servers for SQL Server, this session will show you options to ordering a server and internally installing the hardware, OS, and SQL Server.

Building an Effective Data Warehouse Architecture

You’re a DBA and your boss asks you to determine if a data warehouse would help the company. So many questions pop into your head: Why use a data warehouse? What is the best methodology to use when creating a data warehouse? Should I use a normalized or dimensional approach? What is the difference between the Kimball and Inmon methodologies? Does the new Tabular model in SQL Server 2012 change things? What is the difference between a data warehouse and a data mart? Is there any hardware I can purchase that is optimized for a data warehouse? What if I have a ton of data? During this session James will help you to answer these questions so your response to your boss will provoke amazement and lead to a big raise. Or at least help to lead you down the correct path!

I will be arriving Tuesday night in Chicago and attending a pre-conference session on Wednesday.  Thursday and Friday are the conference sessions, and I will be presenting on Friday at 11:15am in the Sheraton 2.  My topic is “Building an Effective Data Warehouse Architecture“.  I also have an alternate presentation ready to go in case a speaker has to cancel at the last-minute.  That presentation is “Overview of Microsoft Appliances“.

I was fortunate enough to be asked to sit at the bloggers table during the keynote speeches, so make sure to check my blog during the times below for updates:

• Thursday, 8:00-9:15am CST: Microsoft Director of Program Management for BI Kamal Hathi and Technical Fellow Amir Netz
• Friday, 8:00-9:15am CST: Award-winning economist and Freakonomics author Dr. Steven Levitt

Also, I am staying an extra day in Chicago to present at SQLSaturday.  I will again present “Building an Effective Data Warehouse Architecture” at 4pm EST, as well as a just added “What exactly is Business Intelligence?” at 10:30am EST.

Please shoot me an email if you will be attending and would like to meet up.  Hope to see you there!

• Report Builder – Using “SQL Server”, auto-detects joins if source system has foreign-key relationships (by selected “Auto Detect” relationships on the “Design a query” screen).  Otherwise will have to create your own joins.  If use “Analysis Services”, will get a different query designer, and has the benefit of not needing to create joins as a cube has them built-in
• PowerPivot –  Using “SQL Server”, auto-detects joins if source system has foreign-key relationships (via “Select Related Tables” button on the “Table Import Wizard” screen).  Otherwise will have to create your own joins.  If use “Analysis Services”, will get a different Table Import Wizard, very similar to one in Report Builder, and has the benefit of not needing to create joins as a cube has them built-in, but the result returns just one flattened table.  I like to think of PowerPivot as essentially a way of making an analysis services cube from a relational source using Excel as the design tool
• PerformancePoint – If use “Analysis Services” has the benefit of not needing to create joins as a cube has them built-in.  “SQL Server” can only be used to represent tables as KPIs on scorecards or have them appear as data values within filters (see http://www.jamesserra.com/archive/2012/10/using-performancepoint-against-tabular-data/)
• Excel Pivot Tables – If use “Analysis Services” has the benefit of not needing to create joins as a cube has them built-in.  If use “SQL Server”, can only use one table
• Power View – Can only connect to the Tabular model and the multidimensional model (which is in CTP, see Microsoft SQL Server 2012 With Power View For Multidimensional Models CTP)
• Visual Studio Reporting Services (SSRS) - Using “SQL Server”, auto-detects joins if source system has foreign-key relationships (when adding tables on the “Query Designer” screen).  Otherwise will have to create your own joins.  If use “Analysis Services”, will get a different query designer, very similar to one in Report Builder, and has the benefit of not needing to create joins as a cube has them built-in

• No check out feature
• Difficult to extract a prior version (see http://www.ssistalk.com/2012/10/12/ssis-2012-export-old-version-of-a-deployed-project/)
• Downloading a project into VS is not quick as it involves exporting and unzipping
• Can’t revert just one SSIS package – must do the whole project
• Can’t compare checked-in version with your changes (pending changes)
• Does not support shelvesets
• No integration with Visual Studio
• Can’t make it part of a continuous integration solution
• Can’t deploy just one package, must deploy the entire project

The bottom line is the best use for Project Versions is to quickly revert to a prior project version if a new version causes issues (a quick rollback).  It is not really a source control solution.

Benefit of views

• Can be modified by anyone, even outside of BIDS/SSDT
• Can provide default values when needed
• Simple computation can be carried out by views
• Renaming fields leads to better understanding of the flow
• Can present a star schema, even if the underlying structure is much more complex
• Can be analyzed by third-party tools to get dependency tracking
• Can be optimized without ever opening BIDS/SSDT
• For security reasons, to limit the rows retrieved by joining with a security table

Benefit of views in SQL Server Integration Services (SSIS):

• Simpler code inside SSIS packages
• No need to open the package to understand what it is reading
• Easily query the database for debugging purposes
• Query optimizations can be carried out separately

Benefit of views in SQL Server Analysis Services (SSAS):

• Renaming database columns to SSAS attributes
• Clearly exposing all the transformations to DBA
• Simplifying handling of fast variations
• Full control on JOINs sent to SQL Server
• Exposing a start schema, even if the underlying structure is not a simple star schema

Forrester Research: “Business Intelligence is a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insights and decision-making.”

Gartner: “A broad category of applications and technologies for gathering, storing, analyzing, sharing and providing access to data to help enterprise users make better business decisions.”

Me: “Gathering of data from multiple sources to present it in a way that allows executives to make better business decisions”.

“Get the right data at the right time to make the right decision for the organization”.

“It is the application of knowledge derived from analyzing an organization’s data to effect a more positive outcome”.

“It transforms data into knowledge”.

So why use BI?

• Understand the health of the organization
• Collaborate on a shared view of business drivers
• Reduce the time to decision
• Opportunity to save costs
• Reduce maintenance and usage cost due to multiple reporting systems and silos of information
• Reduce costs (losses) associated with decisions made on basis of inaccurate information
• Be able to compete in the market with today’s pace and stay ahead of competition due to use of right business intelligence

Tangible Benefits:

• Enable critical data and analysis tools as identified for multiple business groups
• Provide data integrity, simplification and standardization for the business areas
• Provide dynamic and interactive reporting
• Allow for data mining and predictive analytics
• Reports create simplicity and reduce Business Technology dependence for report creation and updates
• Cost savings by reducing the business users effort to create these reports
• Rapid access to data from all sources
• Standard KPI and dashboard reports

Intangible Benefits:

• Improve efficiency and accuracy of decision support system for management and executive decision-making
• Reduce manual consolidation efforts thereby improving productivity
• Provide transparency

The bottom line is BI is no longer a luxury afforded by a few large companies.  It is now considered an essential part of the IT portfolio.

Examples of common business questions that BI can answer:

• How are the sales territories performing?
• How are the sales people performing?
• Which customers are likely to buy from us?
• What products do our customers buy together?

Many companies have a ton of data, but don’t know what to do with it.  They have business analysts who spend days or weeks pulling together data from different sources to create reports, many times in Excel.  These analysts would like clean data that is more accurate and more timely so they can spend more of their time analyzing data instead of building reports.  They often have many of these problems with their current reporting solution:

• Are using reports from 3rd-party app that is hitting production data
• Need to integrate data from a variety of data systems, often in different formats
• Reports are slow, getting timeouts, and inflexible
• Need to combine, cut and paste reports together to form other reports
• Data systems are not optimized for analytical queries, don’t contain all the data needed, and are not available all the time
• The systems do not have universal definitions (no “single version of the truth”)
• Does not manage historical context
• Employees may not the sufficient skills, tools, or permissions to query data systems
• Want to use data in other front-end tools to do ad-hoc querying and data mining 

So decision makers need:

• Reliable, secure access to data to do their job effectively
• Flexibility in the ways they access data
• Tools to browse and analyze data and view reports
• Low time-to-impact; low latency query results

So what is the best solution?  There is no single solution scenario to describe what a BI Solution looks like.  There are however some broad categories of solutions that describe common approaches to BI solutions.  In order of increasing complexity and time investment:

1. Operational Reporting: Provides improved access to existing data from operational systems.  Usually involves the building of an Operational Data Store (ODS)
2. Business Process / Activity Management: Provides improved analysis and reporting capabilities for specific business processes or activities.  See BPM and BAM and Business Activity Monitoring and Business Intelligence.  Usually involves a real-time summary of business activities from multiple transactional data sources to operations managers and upper management (“Operational Intelligence”)
3. Data Mart / Integrated Reporting and Analytics: Provides improved tools and access to business users of an application to enhance its value by improving decision-making.  This is the most common solution.  Usually involves the Kimball methodology when building the data warehouse
4. Enterprise Data Warehousing / Information Management: Provides comprehensive integration of critical information across the enterprise.  Also breaks down the barriers between applications.  Usually involves the Inmon methodology when building the data warehouse

By understanding all of the solution scenarios, you can decide how to approach building your first BI solution.  Some developers may elect to start small by simply providing improved access to operational data as described in the Operational Reporting scenario.  Other developers may decide to tackle the larger problem of providing comprehensive integrated reporting and analytics capabilities to their application as described in the Data Mart scenario.  Truly adventurous developers may decide to tackle the larger problem of Enterprise Data Warehousing, and attempt to provide their customer with an Enterprise Information Management solution which solves the larger problem of providing an integrated view of enterprise information across application boundaries.

More info:

DBA’s, want to earn the big bucks? Learn BI

Business discovery: the next step in business intelligence?

Importance of Data Management for Business People

Microsoft Business Intelligence at a Glance Poster

Putting BI to Work for Your Organization

Three Mistakes to Avoid in your Business Intelligence Project

Why Corporate BI and Self-Service BI Are Both Necessary

Video Business Intelligence Can Benefit Any Organization

How to Start Microsoft BI

Five Pillars for Creating a Business Intelligence Culture in an Organization