Zujava.com is built using Drupal 7, and makes heavy use of fields/entities. The main feature of the site is creating an article. An article can be made up of any number of sections. We have different section types available (like free text, a link list, Amazon products, eBay, etc.). To create an article you give it a title and an introduction and then add, order, and customize any number of sections  you want.

Each section is a field collection of custom fields added to the content types. I used custom code to build up the article display, section controls (adding, editing, ordering, etc), and blocks for the site. The home page is a collection of blocks and the article side bar are still more blocks. Nearly all of the blocks are created at the code level and I use custom logic to effect their display.

When possible I am creating content with render arrays, but this is probably only 15% of the time. I found the lack of documentation on render arrays a challenge when starting development.

We had two priorities for the site during development:

1. Make is really fast.
2. Make it faster.

The dirty little secret of Drupal performance is you have to use less modules. They don’t tell you this in Drupal school. I took a rather militant approach to module selection during development and refused to add any module to the site unless absolutely necessary. It’s very easy to handle placement and appearance logic in Drupal 7, so I don’t use Panels or Context. The only page that needed Views was the blog list page, so I wrote a few custom lines of code for that. I will hold out against Views as long as possible. (They are all great modules, it’s just the code bases are huge.)

For simple tasks like adding the Google Analytics tracking code I just used custom logic. The site makes heavy use of hook_cron for the billing system and the integration with Google Analytics (to download traffic data for individual pages to show to page authors). At some point, I will need to upgrade to better batch/job handling. The fantastic Amazon Module made Amazon integration a breeze.

Making search engine sitemaps work well is hard, so I had to use the XML sitemap module. For spam I’m using CAPTCHA, though I expect that will need an upgrade to Mollon at some point.

The site runs on Pantheon’s multi-tenant V2.0 platform.

I wanted to move Droptopia to better hosting at about the same time I found out about Pantheon Drupal. Pantheon isn’t Drupal hosting, it’s a platform for Drupal. Baked into its service is a load of developer goodies like Varnish, Memcached, Apache Solr, and Jenkins. Code is managed in a hosted Git repository and you get three sites for each Drupal site — dev, stage, and production — complete with 1-click deployment scripts you run from the web dashboard to manage releases.

Migrating to Pantheon was dead-simple. The first step was to tar up a dump of the database and the web root. Then I used the Pantheon import wizard and pointed it to the tar file:

The Pantheon wizard for importing a new site

Once you hit the “Confirm” button you get taken to a cool log screen that shows you the progress of the import in real-time. The Droptopia export was over 300 MB and Pantheon had the site completely set up in less than four minutes.

The best part was Pantheon noticed during the import that my site was running Drupal 6.20 core and did the Drupal core upgrade for me while the site was being imported. That is ridiculously slick.

I’ve pushed a few minor code updates using the deployment workflow. So far everything is easy and the site is running great. Drupal just got a lot less painful.

The problem with using Drupal is you get so intoxicated with the modular architecture it provides that you start seeing any web site as a Drupal implementation waiting to happen. I was bored one weekend when I thought, hey, I can build a Sortfolio in Drupal.

I got to work building the site and populating it with Drupal shops. I found a great design studio called Statik that took pity on my poor design skills and provided an excellent theme for the site.

Droptopia was born:

Droptopia: Find the perfect Drupal shop for your next project

As of this writing Droptopia lists over 100 amazing Drupal shops specializing in everything from churches to the Omega theme to Enterprise Drupal. Know a great shop we don’t have? Let us know. Work at a great shop we don’t list? Create a portfolio for free.

When we show a client new to Drupal the content list they quickly come to realize this page isn’t very friendly. There really isn’t much you can do to enhance this page. Take it or leave it.

There are several alternatives, like the excellent Content Management Filter module. You get loads of helpful filters and you can set the total number of rows to view in the results set:

The Content Management Filter Drupal module

You can also setup the Views and Views Bulk Operations modules to add in the bulk control from the content list to content listings you create as views. It’s pretty flexible.

But I wanted a drop-in solution that was a bit sexier than the Content Management Filter — complex filters are so 1999 — and easier to install and setup. So I created the Content Finder module:

The Drupal Content Finder module

The module is designed to be dead simple. You get just a single text box. That is it. For ease of  use, the module always remembers the last sort and search you did. The content list will look exactly the way you left it when you return, be that five minutes later or five days.

You can also use specific filters in the search box — each called an “axis” — for type, status, created, createdafter, and createdbefore. Examples:

• A search for at returns all nodes with “at” in the title.
• A search for type:page returns all nodes of type “page”.
• A search for status:published at returns all published nodes with “at” in the title.
• A search for createdafter:4/8/2011 returns all nodes created after April 8th, 2011.
• A search for status:published createdafter:4/8/2011 at returns all published nodes with “at” in the title related after April 8th, 2011.

Content Finder is by no means as good as the Content Management Filter module. But it’s a start. We already have a great patch in the queue for adding in auto-suggest for search axes as you type, and more enhancements are on their way.

Check it out and let me know what you think.

At this point I had one project fail I didn’t control and one project fail I did control. Once I had decided to abandon that project I was frustrated and aimless. In these times I tend to do two things: drink beer and write code (not necessarily in that order). I didn’t know what to do next.

And then there was Drupal

The projects for the firm I work with part time – Urban Insight – were increasingly based on Drupal. Drupal is a powerful, modular, and popular framework for developing a web site (less a framework for developing a web app). The community supporting Drupal is probably its greatest asset, quickly followed by the massive number of modules available for free to extend functionality, and the sheer beauty, power, and massive pain of the chaotic hook system architecture.

Drupal was surging ahead in popularity and within a very short time Urban Insight had a pile of Drupal-powered client sites that we maintained. I believe in being a zealous steward of technology for my clients. In Drupal, this means ensuring the Drupal configuration remains secure and tuned, and that security patches are installed quickly.

With Drupal’s power comes complexity, and this complexity can be hard to manage. When you have dozens of Drupal sites it’s hard to keep track of the core version, module list, and configuration of each one, and keep it accurate at all times.

So I had this thought. If I built a little web app dashboard to track all of our Drupal sites it would give us a single place to monitor the live configuration of every site we had. I threw together some mock-ups and on the (as I later learned) wise advice of Rework I tore out features that would take too long or were too hard (which really translated into hard to use, as I later discovered), and focused on what I could build in three months.

I got an alpha working and showed it to our CEO. He said he saw its value and committed to being the first customer, when it was ready to launch. I was excited. When I started to build this I honestly didn’t think it had potential as a business. I was just building it to make my life easier at Urban Insight.

Go marketing young man

I wasn’t going to make the same mistakes I made with my previous effort, so I tried to focus on marketing. I did learn how important the marketing was, but I had not learned how to market.

I remembered some advice I read previously about design: all designers start by ripping something off. So I ripped something off. For the home page, I found one I really liked (Pulseapp for the curious) and started my design by ripping of theirs. You aren’t doing anything unsavory in starting by copying because in the course of adapting it to your needs it becomes your own.

The other neat thing that happened in the process of copying is that as you replicate specific components of a design you start to understand why they did this little thing in this certain way and you have this nice aha moment or clarity. I am a lousy designer but starting by copying helped me design one order of magnitude less lousy.

I used this technique for the home page (thank you Pulseapp!), the tour page (thank you 37 Signals!), and the tips page (thank you again 37 Signals).

Do only what is important, or less really is less

I didn’t build payment integration until after the beta. I lied really. I launched the beta, said you could sign up for a paid place once we left beta but with no intention of building it unless people used the damn thing. I wasn’t wasting time.

I also resisted the urge to build an administrative app. You always need to build some kind of for admins to look at logs, see account information, and do various account management tasks. But these admin apps can be a real boondoggle. Giant charts and graphs are cool to look at, but do they actually help me get stuff done faster?

Using some advice I got (yet again) from Rework I decided to let need define development of the admin app. When a support request came in I forced myself to build an admin feature that would speed up troubleshooting the issue the next time it happened. This meant it took slightly longer to address a support issue the first time around, but we are talking only hours, not days or weeks.

With this pure need-driven approach to development the time it took to handle troubleshooting dropped tremendously. When an issue came in, I had a screen or a link that helped me identify the issue in a few seconds, instead of a few minutes of manual database hunting and pecking. It also led me to bake in as much self-help as possible into the app when things went wrong. When the tool encountered a specific issue with a user I was now able to have it return specific steps to address the issue. The most common issues really addressed 80%+ of the issues being encountered.

I had little support to triage, and when I did, it was easy to handle.

Droptor is born

The Droptor home page

I finally had it. A tool I needed, a tool I used, a tool others found useful, and a design that didn’t suck (too much). I launched the beast in beta, released the module on Drupal.org, got some users, and planned for launch.

I baked in payment processing, planned for launch day marketing, and ran a promotion for my current beta users to sign up before launch for a steep, lifetime discount. Several users took advantage of the offer.

Droptor launched on July 17, 2010.  I released three major upgrades over the course of the next five months, culminating in a Droptor 3.0 release on January 2, 2011.

The product needs to scale in revenue

I imagined the Droptor customer base as being in two distinct groups that I would reach one after the other. The first group would be firms like Urban Insight, Chapter Three, and Phase 2 that do a lot of Drupal work and have many Drupal sites they need to maintain and support on behalf of their clients. There are easily more than 100 of these kinds of customers.

The second group of users would be organizations – like a university or a large company – where Drupal has grown like an infection from being used on a little project they had to put together quickly two years ago to conquering the home page. I’ve personally spoken to these IT departments and heard this story over and over again.

When I decided to open up Droptor to other users, I thought I had a clear idea of the customer base and the potential size of that base. I thought it was large enough to be a first step to build a small business.

But my assumption that a firm would have many Drupal sites to support on behalf of their clients proved incorrect. It now appears far more common that a firm will hand over a finished project to a client-side IT department. My plan was to use the revenue from the first customer group to reach the far harder second group was not working.

Marketing is hard

Yet again, I found marketing very hard. I ran a Google Adwords campaign for months that did drive traffic to the site. This kept a steady pace of free sign ups coming in, but after some time it was clear these were the wrong users. These folks were going to Google to figure out how to setup a Drupal site, and were using Droptor to help them figure that out.

I certainly don’t have any issues with these folks using Droptor for this purpose, and I would never stop them. But they aren’t the users who are ever going to set up a paid plan and track dozens on Drupal sites for their work. It wasn’t worth paying for customers that wouldn’t increase revenue. I stopped the Google Adwords campaign.

Being part of the Drupal community was probably the most valuable marketing I could do. Just posting the module on the Drupal.org site was valuable in getting attention. I also used Twitter to put a public face towards users and was able to respond quickly when users reported issues (and win public praise).

I also spent money on putting a glossy handout in the swag bag of every conference attendee for the 2011 DrupalCon conference in Chicago. This is the Drupal event of the year. A majority of everyone in attendance is a potential customer so I thought this would be a perfect marketing opportunity.

Sadly the ad had essentially no impact. The problem must lie with the product, the ad, or the marketing method. I had budget constraints so I had to design the ad myself. But even with that, it must be an issue with the product.

The DrupalCon Chicago Droptor ad

But the most disheartening moment with marketing came when I spoke with several people at DrupalCon Chicago 2011 who had the exact problem managing Drupal sites that Droptor was meant to solve, and stared at me blankly when I said “I have a tool that solves this problem.” For whatever reason, Droptor didn’t resonate. I really don’t know why.

I feel like I did two simple marketing things right. First, I setup Google alerts for “drupal monitoring” and “droptor”, an easy and well-known method to track conversations and opportunities to talk about your product. Second, I carefully tracked whenever praise came in – be it email, Twitter, or a support ticket. This made preparing marketing materials much easier.

The payment plans were wrong

In retrospect, I think another problem was the payment model of Droptor. The pricing module was $2 per Drupal site scanned per month. I came up with this model to make it easy for consultants to bill the cost of Droptor back to the individual clients they were supporting (assuming they had a fleet of Drupal sites for many different clients). What I didn’t realize is the cost was so trivial that passing along the cost didn’t matter.

I should have used a traditional billing model where there was a free plan and several paid plans for larger number of sites (with additional features made premium). This pricing structure would have made the tool seem more professional, and I believe the additional cost would have added credibility.

What I’m proud of

I’m happy with how two things came out: providing customer support and building a product. I made customer support the top priority above all other tasks and responded to issues quickly and completely. Nothing disarms a frustrated user faster than a rapid response, honesty, and an “I’m sorry.”

I am also proud of the fact that I built a real product, maintained that product, upgraded that product, supported that product, and got a (few) people to pay for that product. The feeling of getting paid for an act of pure creation is amazing.

Where Droptor is today

I am averaging a little less than one free sign up a day (during the week), and very few paid sign ups. As of this writing, we are syncing over 450 Drupal sites for over 400 users. We have about 25 paying customers. Droptor revenue is very modest, but does cover our expenses (other than time) of hosting, backup, and payment processing.

Revenue isn’t growing, and I am no longer convinced that revenue can scale. It’s decision time. Refocus and continue to work on Droptor or discontinue it and move on.

Major lessons learned

Most or all of what I’ve learned is already out there, but these lessons are all personal to me so I want to share them here clearly. The lessons are:

• Build something you will use
• Google Adwords will get you traffic in exchange for money. The critical question to answer is, “Is it the right traffic?”
• Use a standard pricing model, with plans based on size and extras for the higher plans
• In design, copy something good. If unsure, find a similar work you like and start by copying it
• Use a Twitter account to provide easy communication to users
• Spend almost as much time building the marketing material for a feature as you do to build the feature
• Communicate value to potential users, not features
• Have a guaranteed first user. Fuck, have five
• Let necessity drive development, not wish list
• Let pain points guide your efforts
• Hard to make features are probably hard to use
• Easy is hard, so focus on that instead of hard-to-build features
• Track testimonials and current clients for use in marketing

What is next?

I have no idea.

The failures have been incredibly frustrating for me on a personal level, though very educational. I would love to work with a larger team on a new project. I really don’t know why, but I can’t wait to try, try, try again.

Why I’m doing this.

Before I dive and talk about the next app I launched, I want to be clear on what my goals are in starting a web business:

• I like building stuff on the web. I have been coding and using computers all of my life.
• I want to build permanent support for the work/life balance that I have found is most productive for my output.
• I want to be in charge.
• I want to be business for 30 years, not three.

Now, in terms of my goals for money I really, truly am not doing this to get rich. I am doing this to make money, though. But money to me means slowly growing a company over time and making it profitable on day one. I’m not interested in selling a business to Google or Facebook, and I am not setting out to build something I think they would want to buy.

I’m interested in profits, margins, and value. My business heroes are Warren Buffet, Jason Fried, and Joel Spolsky. I don’t care about social networks or photo sharing apps. I don’t want a $10 million dollar funding round. I don’t want an audience before I have revenue. I want to run a business that makes profits and does cool stuff, building a team of like-minded people over time.

The idea

The home page of Prospector App

I designed and led the team that built several internal web applications for the higher education market. One of these was essentially a customized CRM for admissions recruiting. Perfect, I thought. The value of a CRM in an organization is proven – look at Salesforce – so organizations will want them. I could combine my domain knowledge of higher education admissions with having already built a web CRM for recruitment. Sweet.

So I started coding. And coding. And coding. And designing (poorly). I took the premise of what I had learned building an internal .NET app for recruitment, took out the stuff I didn’t think made sense (the peril of services), and rebuilt it as a multi-account web app with the excellent CakePHP framework.

Development wasn’t hard, except for the payment integration. I don’t know what it is with payment gateway vendors, but the APIs are usually painful (Authorize.NET, I’m looking at you) or the multitude of plans is very mind-boggling (read: Paypal Propayments Flopro Webmasters Elite). Integration is a pain, testing is a pain, and handling all of the edge cases is a huge pain.

I wanted people to be able to upgrade, downgrade, and cancel in the application, without having to contact support. This took more time than I wanted to allocate. In retrospect, I should have launched a free beta and waited to build payment integration if people signed up. This is a valuable lesson I didn’t fully learn until this happened and I read Rework by 37 Signals (an excellent book, highly recommended). Solve real pain points, not anticipated ones.

My Three Major Mistakes

I finished the thing. I had my head deep underwater, coding for several months, happily working away. Then I actually finished it and I had no idea what to do with it. I didn’t need to use it myself, I didn’t know how to market it, and I didn’t have any good connections to the community of potential users (higher-education admissions professionals). I had this big thing, and I had no idea what to do with it.

Of course I had read about the danger of building something you didn’t use, but I didn’t really understand why it was so bad until I experienced firsthand. If I had a co-founder who had connections or access to the admissions community, it would have been a far different experience. But I didn’t, I just myself, my code, and my time. Once the thing was done I had no idea what to do with it.

Another major mistake was not caring about the design of the public pages for the application. I didn’t spend much time thinking about how to market the product on the web site itself, or even how to market it generally. The design sucked and I had no idea what to do next.

I think the mistake with marketing came down to the fact that I didn’t start thinking about marketing on day one. In retrospect, I should have kept this in the back of my mind during development. When I finished a feature, I should have adding it to a tour page, or a FAQ, or something. But I didn’t. I just put marketing off to a later stage. I will worry about that later, I thought.

The final major mistake was that I didn’t have a guaranteed first customer. A guaranteed first customer is a friend or a colleague who both needs the thing and would be happy to try it in production and give you feedback. I didn’t have anyone who I knew for certain would be the first user. This should have been a major red flag, but I missed it completely.

To summarize, I made loads of mistakes on this project, including:

• Spent too much time on payment integration before the product was proven
• Built something I didn’t need or would even use
• Didn’t have a connection to the community of potential users
• Put marketing off to the end instead of making it part of the process from day one
• Didn’t have a guaranteed first customer

The project stalled on completion. The technology was finished, the code was complete, but so was the project. I moved on to Act 3.

I don’t like failure. I really don’t. I get why you shouldn’t be afraid of it. I totally agree that there is loads to learn from it (read on, I know this first hand!). But I don’t like talking, sharing, or thinking about failure. I don’t even really talk about my startup work to my family, save for my partner. I mean, who doesn’t like winning?

I’m generally a private person, and don’t want to share something until it’s been a success. But I’m at this transition point where I have been doing my startup for more than two years, it hasn’t worked, I am unbelievably frustrated, and I need to make some important decisions on what to do next. I am forcing myself to write this post as an exercise to improve the thinking I can do about the experience, learn more, and hopefully get some outside advice.

Chatting on the web

A web chat in action

My dive into boot-strapped web apps started four years ago when the firm I worked at wanted to develop a web-based tool for admissions teams to chat with students. The tool was called University Web Chat, and later was renamed to Milo Web Chat.

There were other great chat systems available at the time (like Campfire), but none that supported the use-case we were trying to provide to teams of admissions recruiters. That is, the ability to host a chat with a small recruitment team (known, registered) for a larger student audience (unknown, unregistered). We focused on ease of use, simple custom branding, and serving the admissions market exclusively.

I was the lead architect and developer of the first working version of the tool, though several other Urban Insight developers helped with development throughout the lifespan of the app. It worked, was easy to use, and scaled to chats with 70+ people participating. As the developer, it was very satisfying to have people use something I built.

We used a basic freemium pricing model where the basic account of up to 4 chatters at once was free and a premium plan where the limit was lifted to 100 chatters cost money. We struggled to find the right pricing model, and I never felt like we had it right.

Account growth – both paid and free – was slow. Two years after launch the leadership decided to wind-down Web Chat. I was bummed, but unsurprised.

So what went wrong?

I looked to my own frustration. It was a very challenging product management experience for me because as a services organization we always had to give client work first priority. This meant that we could only sporadically give Web Chat attention. There wasn’t a consistent allocation of hours for development or marketing allocated over time. With uneven spurts of time to work on the project, we spent more time getting into the groove then we did actually getting good work done. Looking back, I think the project was doomed to fail from the beginning with this approach.

While I felt (and feel) the lack of consistent attention was the biggest cause of failure, we also struggled with messaging. Despite all the great advice online contrary to this approach, I focused on features. Features make sense to me. A careful evaluation of features is how I buy a product.

I assumed this must be how everyone else makes a purchase. I thought, just show the amazing features, clearly, and people will go gangbusters for how awesome our product is! Boy was I wrong. I didn’t start to understand how critical this was until the end of the project.

Still, a key strength we brought to the product was a focus on a niche area (higher-education admissions) where we had real experience. There are a lot of admissions professionals out there – complete with their own conference – so I don’t think the market was too small for Web Chat to be a moderate success.

Web Chat didn’t work out, but I learned some valuable lessons:

• You need to be able to give you product consistent attention.
• Don’t focus on features, focus on value
• Even simple is hard for a user
• Users don’t think like you ever

Next steps

Web Chat was doomed, I was frustrated, but I was also blown away by the potential of a business built around a web application. You never have to install your app on client hardware, you only have to upgrade one system when new releases came out, it’s a no cost business to start (only my time is expensive), and since childhood web coding has pure fun. I also wanted to be in control so I could make (what I thought) were the right decisions.

Luckily, my employer was very supportive and very flexible and I was able to go part time so I could focus half my week on my own product. At last, time! At last, control! At last, web apps!

Sadly, I was not all out of failure. The brain dump continues in Act 2.

The conversion didn’t create any immediate problems with site performance, but we did start to see delays for certain tasks that used to be speedy with MyISAM, such as imports from a dump file. To make it easy to replicate the issue we created a simple script that looped through 25 node IDs and called noad_load() and noad_save() for each one. The script ran fast on a MyISAM copy of the database, but was dog slow on InnoDB (2 seconds compared to 14 seconds).

Our assumption was this was an InnoDB configuration issue so we made a variety of basic InnoDB settings refinements. But these didn’t help. I wanted the team to have a clear direction on how to proceed with troubleshooting and make sure we were covering all the bases. The solution? A checklist!

Here is the checklist we created:

1. Post the problem to Drupal Stack Exchange for ideas
2. Try profiling the table
3. Run load tests on other hardware to verify issue is is not hardware related
4. Identify a the slowest query in the node_load()/node_save() calls and use explain to review it
5. Identify the total index sizes for each table
6. Determine if we need to allocate more ram to InnoDB
7. Make sure we don’t have hard drive fragmentation
8. Try adjusting the InnoDB log settings on the stage server

Now, we had been assuming this must be an InnoDB configuration issue because we don’t use MyISAM all that much, and that was the only thing that changed. But I thought, what the hell, let’s check some random stuff too.

Turns out this issue had nothing to do with InnoDB configuration. Once we hit item four we quickly traced the problem to a single SELECT query taking up more than 90% of the execution time of our load test. After adding an index to the table in question we reduced the load test time from 14 seconds to under 2. Beer time!

As a developer on my team said afterwards: “I am going to remember three things in database tuning: index, index, and index.”

The team blogs about the stuff we use for our work, like Drupal, jQuery, SEO, and security. We are just getting started, but we are posting about once per week.

If you are interested, this page lists my posts.

It’s very easy to understand, but requires discipline to do consistently. Simply, when something comes back to me I check that is does or is what is claimed. I don’t micromanage and don’t bother the team member when it’s being worked on. But when they say it’s done, I check it out.

My goal is to check 100% of what comes back to me, but this isn’t always practical. Sometimes the issue can’t be checked because we can’t safely recreate the issue. Sometimes the state of the data isn’t right and it is non-trivial to recreate that state. Still, these are rare. I am able to validate 90-95% of what comes back to me.

What do I check? If it’s a bug, I check the reproduction steps. If it’s a feature, I check our requirements document or specification for every intended piece of functionality and verify they work. If it’s a configuration change that our hosting provider has made to one of our managed servers, I validate the configuration is live at the command line or in the web root. If it’s a firewall change, I run a port scanner.

Why is this so important? Because as a project manager your most valuable asset with a client is your reputation. If you frequently give work to the client that is incomplete you will have no credibility. If your client doesn’t have faith in you, then they won’t have faith in your organization, or in your ability to make the project successful. The project will be doomed.

However, if your work consistently reflects what you say, then you gain credibility with the client and they will have confidence in you, your recommendations, and ultimately your ability to make their project a success.

My first tip to new project managers? Trust but verify.