Implement the Logic

How to combine DelayedJob with Shopify API Limits gem

2011-06-11T00:00:00Z

As anyone who has done any serious web application development using the cloud (in my case Heroku) to host an App supporting Shopify shops, you need to be aware of the API call limits of both the shop itself and across all the shops the App is installed in. The 300|3000 rule – you can call one shop 300 times continuously or make 3000 calls to all shops continuously.

We used to have to write wrapper code to monitor and count API calls on top of writing complex program logic which was clunky to say the least. The ShopifyAPI development team kicked in by providing API call limits in the response headers, allowing an App to smartly self-monitor these limits during processing.

This was great except the development team forgot that the ShopifyAPI gem was not providing ActiveResource headers, making this small bit of progress tough to use for most people. Along came @christocracy who quickly hacked a small modification to ActiveResource and the ShopifyAPI gem providing simple and non-intrusive monitoring methods in support of API calls. In addition to now being able to know where the App stands with these limits, it is also important to run longer running API calls in background jobs (AKA delayed jobs) so as to not block the web App from servicing other calls. How to take advantage of this wicked combination of DelayedJob and the ShopifyAPI credit limits?

shopify-api-limits

Assume a user selects ALL their (Orders|Products) in their Shop, and this amounts to a total of 603 items (a list of ID numbers), just over twice a shops typical API limit for continuous calls. To send these 603 (Orders|Products) to the App I setup the shop(s) with an injected link in the (Orders|Products) Action Drop Down menu. At the moment Shopify Admin has a bug limiting selection to 50 items, but let’s assume we can send all 603 ID’s as a GET parameter to our App.

Our App receives 603 ID’s and needs to “touch” each (Order|Product) once (or twice, eg: read/update). For example by doing this simple operation:

That burns 2 API calls right there. So we are in some trouble since we can only do 300 API call continuously.

I decided to send ALL 603 id’s to my method that will enqueue the request to process this list into the Delayed Jobs table I use. Delayed Jobs are run automatically by Worker threads that spin themselves in and out of existence, so as not to burn the App’s credit card charges for worker fees to swiftly. I use Heroku and workers cost a nickel each per hour. The key to a Delayed Job is that you can control when it runs, what runs, and it does not affect your App since it runs in it’s own thread.

When a job runs, the object that is instantiated can have code that checks the number of available API calls it can make. If there available API calls, we can interact with the Shop. If there are not enough API calls to complete a desired process, we have do take some steps to ensure we complete all the tasks that were specified.

We are processing a list of (Orders|Products) items so we need to keep track of the ones we have already processed and the ones we have not. This is easy by keeping track of the index of where we are in the iteration of the list of items. We can spawn a new Delayed Job, and since we know we have only partially processed our list of items, we instantiate the new job with the items of the list we have not processed yet.

Additionally when setting up a new job we can also specify when to run the job. If we tell it to run 601 seconds in the future, we should have a fresh slate of API calls available if the limit was reset. That is the essence of the code provided in the following gist. Of course, other actions could also be taking place that delay the availability of API calls, so the process should continue itself until it receives enough calls to complete. This means jobs that start and encounter a 503 response (no API calls can be made temporarily) should spawn a new job in the future, and terminate themselves properly.

I tested this code out and it works well. It processes as many API calls as possible before hitting the limit, at which point it spawns a new delayed job to start 10 minutes later. Watching the logs for activity, 10 minutes pass and the next delayed job was picked up by the worker thread assigned to run jobs. This job completed before running out of API calls and terminated gracefully, thus leaving the App in a good state, the shop in a good state and all was well that ended well.

Thanks to Chris for his excellent shopify_api_limits gem for making this possible.

The we are Sorry! App

2011-06-03T00:00:00Z

We are all Happy, not!

Scenario. You are running a very busy Shopify business, closing hundreds of orders per day. Your product is great, Shopify works great, and your customers are by an large happy.

Occasionally, no matter how well things are going, a customer is going to phone up and complain about something. The product was not quite what they expected. The delivery was late. The book was bent. The fruit was soft. The neon green was more like neon taupe. And on and on.

So you pull up their order and you’re looking at your Shopify Admin screen.

You installed the We are Sorry! App and now you have that link at your disposal. You click it and the magic happens. The App looks up the customer, generates a 10% one time use discount for them, and fires off an email to them with the good news. The results speak for themselves.

What an easy way to make your customers happy again. Say you’re sorry today!

You can also completely customize the actual discount. It can be cash amount, a percentage, anything you can do with the built-in Marketing tab discount codes.

When Shopify upgrades the Discount Coupon operations, the App will follow suit, so you can take advantage of one-click apologies.

You can see the App in action by trying it for yourself. It currently lives at sorry

Lesson learned about ShopifyAPI Billing

2011-05-25T00:00:00Z

I am in the midst of no longer giving away simple (or complex) Apps for free, but instead, hooking into the Shopify Billing system built into the API, so I can receive some lunch money from my hard work.

So, I receive access to a store, as the store admin, deploy my code, and test out the Recurring Charge calls. It all works as expected in my experiments, so I call up the shop keeper and let her push the actual button confirming that yes, she is willing to pay the small monthly fee for using my small App to relieve her of tedious work.

I get another call to deploy the same kind of App. So I set out to see how that all works by accepting a staff account to the new store, and wiring it all up. After some initial glitches with my sloppy code migration, I hook into the App and it is approved, but I always end up at the “confirmed” page of my App sans a valid charge ID. WHA HAPPEN? WHUZZUP?

I load up my App with debugging and see that what ends up happening is that my App finds itself Authorized in the Shop, but the redirect ends up at the Shop Admin Account page. It gradually dawns on me, the problem is simple. I am NOT the Shop Admin, I am a pee-on staff member. So all my testing of the “Do you Accept to Pay XX$ for this App” are falling on deaf ears.

I asked my client to login and click on the App login… she is directly confronted with the correct screen, from Shopify, asking her if she accepts the charge for the App. She agrees, and ends up at my Apps confirmed script with the charge ID. All is well. Lesson learned.

Using the Shopify API command line interface (CLI)

2011-05-15T00:00:00Z

A good many developers probably have installed the shopify_api gem from rubygems.org in order to built an Application to provide new functionality to shops. In fact, without even wanting to develop a full application, this gem is great for quick and dirty manipulation of stores. I am constantly using the Ruby console IRB to check if certain inventory makes sense, or other tidbits. It is almost essential to use this gem to advantage. If you have ever used Textmate on OSX and hooked up the handy Shopify Bundle from Meeech, then you are very used to typing in an API key and password to a domain to edit the assets, theme, etc. Turns out, there is yet another way to interact with shops, and it also involves the command line.

Ruby developers are generally very familiar with rakefiles and rake tasks. Unlike makefiles and the make command, two things generally reserved for people that think wave equations are fun for breakfast distractions unlike the rest of us that just think waves are pretty can like to jump in them, rake and rakefiles are comprehensible and indispensable.

As with anything progressive, rake was deemed to be ripe for a re-factoring and during the glory days of Merb, before Rails switched to Merb-ishness, Merb had Wycats and Thor. Thor is like rake, but even more fun some would say. Thor tasks are yet another way of running ruby tasks so that developer life is easier and more automatic. Turns out, the Shopify developer gang built a Thor task command-line interface (CLI) into the Shopify gem. I finally got around to playing with it this rainy weekend, many months after it was introduced to me at a beer drinkup I attended with the Shopify nerds.

When you install the Shopify API gem

gem install shopify_api

you get this command line interface for free! In order to make it a little more compatible with my current environment, I tweaked it a little as per the gist here: cli.rb

If you copy that code into your favourite text editor and save it as cli.rb, you can experiment with this neat option. Once you save this file to your system (*nix compatible), you can mark it as executable with

chmod +x cli.rb

If you then type cli.rb list you will see nothing!!! Yay… unless you see errors, which would be mean you messed something up. If that is the case, and some error chunks are blown, edit the file till none are thrown.

Once going, try this:

cli.rb add mysite

You will be prompted that mysite.myshopify.com will be created for you, or you could type in something else.

Now, you fill in the API key, and API password for the site, and all of a sudden, this is a good thing. A configuration file has been made especially for this site. Big deal you say. Bah… Do that crap all the time… Okay.

So next, try:

cli.rb console

And now, you’re in a full-blown IRB session, authenticated to that store. Wonderful. Now you can ask questions like:

ShopifyAPI::Product.count

and get an instant answer. No messing around. I used to just use IRB and paste in my Shop’s Auth Key, but that is too painful, I can never remember all 280 digits off the top of my head. Even connecting to Heroku and asking my Shop for the credentials is a pain, and looking inside my note application also sucks. This is just a simpler, cleaner way to make a config file and hang out.

There are enough other commands to keep it all nice and organized. In fact, the current list is:

Tasks:
  cli.rb add CONNECTION        # create a config file for a connection named CONNECTION
  cli.rb console [CONNECTION]  # start an API console for CONNECTION
  cli.rb default [CONNECTION]  # show the default connection, or make CONNECTION the default
  cli.rb edit [CONNECTION]     # open the config file for CONNECTION with your default editor
  cli.rb help [TASK]           # Describe available tasks or one specific task
  cli.rb list                  # list available connections
  cli.rb remove CONNECTION     # remove the config file for CONNECTION
  cli.rb show [CONNECTION]     # output the location and contents of the CONNECTION's config file

Neat. Enjoy. I know I will.

Fulfillment Messaging App

2011-04-29T00:00:00Z

When running a Shopify store, sometimes you want to treat your orders a little different than the normal flow allows. It can really depend on your shipping alliances, your inventory and what your customers expect for communications.

When an order is booked, the customer and shop keeper receive an email that serves as a first communication. There are quite a few other email templates that can be fired off once the initial order is dealt with. Fulfillment seems to present some issues that I have dealt with recently using custom Applications.

One scenario is that when a shop is receiving a lot of orders, it is painful to fulfill them one by one. I created an App that accepts up to 250 orders at once from the Shopify Admin, and fulfills them automatically. Although that relieves the shopkeeper from clicking on 250 separate orders, it revealed some other problems. Sometimes it is important to fulfill those orders and NOT send an email. Sometimes it is important to fulfill those orders and send a CUSTOM message. My recent App allows for that by letting the Shopkeeper create custom email messages. When they select orders for special fulfillment, each order’s customer gets the custom email. One super thing about Heroku is the easy integration with Delayed Job. Tobi from Shopify wrote this code to take on the running of background tasks. In my Apps, I now delegate all potentially long running tasks to background jobs. It is quite neat to see this in action. I watch the job queue as it fills and empties itself during normal operations of the App. The Delayed Jobs spin up their own worker force too and this helps them to work fast. Once all the work is done, they kill themselves off too.

Another scenario that cropped up was the fact that sometimes an order will need to be fulfilled more than once. That occurs when a shopkeeper accepts payment for a quantity of more than one item, while at the same time recording future delivery dates. At each delivery date, an already fulfilled order needs to be fulfilled again. Luckily one of my Apps is able to present the shopkeeper with future delivery dates so that these future fulfillments happen in an orderly and proper fashion. Along with multiple fulfillments, the shopkeeper can rely on the built-in shipping update letter to go out, or use the custom messages they store in the App.

These custom Fulfillment Apps are a great way to eliminate clicks in the day to day administration of Shopify stores that deal with fulfillments in ways just slightly different than the usual pattern.

My Fulfillment Apps are well tested at this point with many thousands of orders booked. If you have a twist on fulfillment you have to address with your Shopify store, drop me a line.

Shopify App Development for Vendors

2011-04-13T00:00:00Z

I have now successfully deployed and tested over 15 kinds of Shopify App systems to support various clients. The pattern I have adopted has grown more stable and flexible as I continuously experiment with the basics.

Sinatra is my favourite Ruby framework for many reasons. It is very easy to create Domain Specific Languages (DSL) processing, which is perfect for Shopify Apps. It handles models and Active Record well, has support for my favourite template system in Haml and comes in a small but very extensible package of only 2500 lines of code. Compare that to popular Rails, at 250,000 lines of code.

I run Sinatra using the thin webserver on Heroku, one the best cloud-based hosting companies I have ever worked with. Heroku accept an entire App codebase as a simple git remote, and so I can simply git push an entire App to see the magic in action.

A common request from Shopify store owners is to have some sort of external access whereby a third party, a Vendor perhaps, can login to a web application, participate in some aspect of Shopify and yet not have access to the Shopify site itself. I addressed this with an App recently.

Using Sinatra, and Rack, I setup three Applications in the one App, which for this blog we can simply name VendorApp.

VendorApp encapsulates three smaller specialized applications. ShopifyApp, PublicApp and AdminApp. These would be available at three separate URLs.

http://vendorapp.heroku.com/ is the way the public (external vendors) login and see their reports, upload their desired products, etc.
http://vendorapp.heroku.com/admin is where the Shopkeeper logs in and determines what his vendors can see, what they have been doing, etc.
http://vendorapp.heroku.com/shopify is where the ShopifyAPI lives and this is where sales reports can be tallied, orders filtered, products added etc.

Using Warden for Authentication, we can ensure vendors that try the PublicApp login can see only their sales, only their products, etc. With the same code we can also authenticate for access to the AdminApp, where the shopkeeper sees all his Vendors, all their products etc. The admin can control these accounts with a few clicks.

A simple sample workflow for this Application is as follows. Every product has a Vendor. So when an Order is paid for, using a Webhook, we capture the order, and extract the products and vendors. If the Vendor exists we add the quantity and price of the sold product to the Vendor Sales. A vendor has many sales. Now, when the Vendor logs in to the PublicApp, we show them how many products they have sold, and how much money they earned. The Shopkeeper can set a percentage for the Sales, so that a Vendor may see 30% of any Sales.

If you have any interest in this kind of App, drop me a line and perhaps I can install a version of this App into your Shopify store.

Related Products Shopify Application

2010-05-22T00:00:00Z

Related products recently caught my attention recently when evaluating how to best render a Product, along with what the store owner thinks are related products, hoping to increase chances of selling one or more products.

The algorithms I found published on the forums were by and large based on the following algorithm:

Loop through the collection.all for products
inside this loop:
- Loop through all the tags the product has
- Loop through all the tags the product from the outer collection.all loop has
- look for any matches between tags and if one is found, render the product

Imagine for a second you have a store with 1000 products. You want to display two related products to the one you are currently showing. With this algorithm, the outer loop has to run 1000 times to provide on of each product in the store. Inside this loop, you have to loop through not only every tag the Product itself has, but also the tags each and every other Product has, looking for any matches. This is terribly inefficient. Although there are probably ways to accomplish related products without this algorithm, more than a few stores are using it.

I have an application working on Heroku that provides support for using tags to connect to related products. Try out heroku to get a sense for what it can do.

The application presented here works a little differently than the above algorithm. Once installed in your Shopify store, the application provides a link to “Manage Relates Products”. Pressing this link and opening in a new tab or window, brings up the application which will work with the product specified. The application will present a simple list of any established related products if any. It accepts tags in a text box. If you submit the tags, the application will search the store for all products with a matching tag(s). These products will be assigned to a Metafield for the product. The application will display the found matching products in a list.

At this point, if you were to view the Product using the established product.liquid template, nothing would appear different at all. There is no code in place to render any related products. To take care of this, we can include a liquid snippet called “related_products” to detect if any related products are assigned to the product. If so, render some new DOM nodes, and the data representing the product. With a little javascript, we can then turn the data representing the related products into nice clickable elements.

As an example of how tagging might be used to related products to one another, I have added four records from my collection to the homepage of my store at hunkybill. I used the Ramones record as a sample product. The following figures help to explain the rationale behind the related products. Each record is tagged a different way, appropriate simply as example tags, and not necessarily for music snobs to debate. When showing the Ramones record, if I chose the tag 1970’s, I would want a related record to show up, which happens to be Peter Tosh (figure 1). If I changed my mind and decided the Ramones were suggestive of Reggae, I would want Peter Tosh and Fishbone to show up as related products (figure 2).

A simple Liquid snippet that looks for the Metafield related products is good enough to render inside the product.liquid template. The DOM markup is easily styled and can be modified for almost any purpose. If you use a DOM inspector to examine related_products_data you can see the Metafield string of product data. This container div should have a style of display:none to ensure no one actually visually sees this data as it is ugly.

{% assign mf = product.metafields.related_products %}
{% unless mf == empty %}
  <div id="related_products_ct">
    <div id="related_products_data">
      {{product.metafields.related_products.related_products}}
    </div>
    <h2>Related Products</h2>
    <ul id="related_products_list">
    </ul>
  </div>
{%else%}
<p>No Related Products</p>
{% endunless %}

Once the product.liquid template and the included snippet for “related_products” has rendered, the DOM can be checked to see if any of those elements actually exist. If they do, it is simple to read the Metafield string that contains the related products into a Javascript variable. The application stored the data using JSON encoding, so we simply reverse this by parsing the string as JSON. Now we have an array of objects we can iterate, with access to anything interesting about the product. We have the images, the variants, the prices, and can therefore render a nice looking related product with nothing but CSS and some DOM elements as needed. For the purpose of this example, I have simply used a list to render a link to the related product using it’s title. The simplest code to do this can be added to any shop.js file (although the dialect of this example is jQuery, it would be dead simple to use any other flavour of Javascript you like).

(function($) {
  // Product has related products
  $.processRelatedProducts = function () {
    var data = JSON.parse($('#related_products_data').text()); 
    var list = $('#related_products_list');
    $.each(data, function(idx, obj){
       list.append($("<li>").append($("<a>").attr({href: '/products/'+obj.handle}).text(obj.title)));
    });
  } 

})(jQuery); 

$(document).ready(function() {
   // DOM is loaded so we are ready to process whatever we want
   if($('#related_products_data')) {
     $.processRelatedProducts();
   }
});

This application has plenty of room for improvement. For example, showing all the tags available like the Shopify admin uses for setting tags, limiting how many products end up as related products, searching only designated collections for products to add to related products, all easy modifications. The code for the application is open-sourced github for anyone who wants to hack away and make it better. Since some people do like to use tags to relate products, even if Shopify does not believe this is an appropriate use of tags, I hope that my application here can make at least some people happier in managing their related products.

Now offering Shopify Training

2010-04-09T00:00:00Z

With about four full years of experience playing with Shopify’s system, the Shopify API, the Javascript API, Ruby, Rails, and Liquid, I feel prepared to teach what I have learned to others.

I will soon open my first and very own Shopify store, with custom courses as the product for sale. I will likely keep it simple, offering time in exchange for money. You can request a course to learn anything you want about Shopify, without fear of confusing or incorrect answers from the public forums.

Courses will be offered via screen sharing, Skype or perhaps even just the good old telephone system if desired. I am probably going to prepare some online slideshows as well, to assist in the course content delivery, and to keep focus…

Shopify Vision, Compass and Sass Combo

2010-03-29T00:00:00Z

I have not used Vision lately in any of my Shopify development, but that has recently changed due to some new work habits.

Vision

Currently hooked up to the latest version. Installed it in my /workspace. It runs using the command line, ruby vision.rb. I symlink a Shopify store into Vision’s /themes directory allowing me to keep the actual code in my Shopify Github account on a per client basis. Every project I work on gets assigned to a directory in my Github account. When adding a new client or store, I simply do a git add <new_client_store>, and all my work is versioned and available to any of my computers, be it a laptop or desktop, home or away. Vision does not provide all the luxury of a real Shopify store, but for 80% or so of what I need, it is fine. I can quickly download a copy of a site once it is localhost:3232 approved, and then upload it to the client’s account for live tests.

Shopify’s Liquid templates have access to images in the /assets directory using the filter image_url. This convention is not pretty in that most people usually want to associate images during development with a location like /images. In fact, Compass, talked about further down has no plans to ever support keeping images in the same directory as the stylesheets, so setup of Compass will take some effort to work with Shopify. When referencing an image in a Shopify stylesheet, eg: background: transparent url(‘foo.png’) 0 0 no-repeat !important; it is important to note that Compass will only ever produce a background: transparent url(‘/foo.png’) 0 0 no-repeat !important; or a background: transparent url(‘../foo.png’) 0 0 no-repeat !important; which means trouble when using Sass mixins that use images. There are not so many of those though, so for that aspect of development, the workaround is to use the mixin, examine the generated CSS, and then copy it back to the Sass where it can be converted to Sass and allow for the mixin to be removed. Clunky, but it actually rarely comes into play.

Compass

Compass can be started with a number of options, included a stand-alone project, as well as one rendering one of the well known CSS frameworks like BlueprintCSS, 960gs, YUI, or Susy. I have had success with BlueprintCSS; 960gs has been somewhat harder to deal with, and I am sure YUI and Susy work well as there are few complaints about those options.

Once a stand-alone project is created, it can be added to Github for versioning the same as the Shopify site. I only add .gitignore entries for the .sass-cache files which you don’t want versioned anyway. A line like .sass-cache/*/ added to .gitignore will not store any files or directories under the .sass-cache directory for example.

In the config.rb file, you can setup compass to work with relative files or not, and some other configurations. For the most part, I leave this at the defaults and check to make sure that any Sass files compile without errors using the command line compass -w. Keeping a terminal tab open with that alerts me to any errors in my Sass. Since compass is compiling Sass into CSS in the /stylesheets directory by default, it is important to link these into the Shopify project. In order to take advantage of Github, we cannot simply symlink the CSS files into the Shopify /assets directory. What I do, is tinker with the config.rb to tell compass to compile the Sass directly into the Shopify /assets directory. This works well, and means all the generated CSS and Sass code, are properly versioned and available to all my computers.

Sass

Sass is a way of writing CSS using a higher level of code abstraction, practically like writing Ruby. Sass has excellent constructs like variables and the ability to declare a block of code by name for use as a mixin. For example, I can create a variable called !bg_color = #22ff44 and then refer to that in any further Sass files that descend from where that was declared. Sass file organization is up to the developer, and I choose to use a similar pattern to any Ruby views whereby I load in base classes as partials, and then put together all my Sass source in just one or two main files. That helps to keep the clutter at bay, and allows me the freedom to count on mixins and variables wherever and whenever I need them.

With respect to Shopify for example, I keep a partial file called cart.sass handy for all the declarations specific to rendering a Shopping cart in the style required by the client. Any grid setup is stored in a partial called grid.sass and likewise, basic text formatting is in a partial called _text.sass. A main Sass file would then just @include partials/text.sass to take advantage of the text setup for the site. A header example might be:

@import compass/utilities/text.sass
@import compass/utilities/links.sass
@import compass/utilities/lists.sass
@import compass/layout.sass
@import partials/base.sass
@import partials/cart.sass
@import text.sass

I used to work by tweaking Liquid templates and saving them, and then previewing the results. This was a terrible workflow that seriously diminished my capacity to work on Shopify sites with enthusiasm. Now with compass and the provided workflow advantages, developing Liquid, Javascript and integrating CSS is excellent and fun.

The latest Vim, Scribe and Textmate editors all allow code to be saved as soon as the code editing window loses focus. Compass has a watch command that compiles code as soon as changes are detected in a Sass file. Hence, the split second after moving focus from code editing to a browser, the code has been saved, compass notices this, and it compiles Sass to CSS, allowing reviews in the browser. Even further, Firefox (which I don’t use much anymore) has a new watcher plugin called xrefresh which will auto-refresh the browser when it notices changes to provided files. This implies that as focus switches from code editing to the browser, you can always see the latest greatest code without doing much more than clicking the mouse once when done. The author of Sass, Nathan Weizenbaum has even provided the community with FireSass plugin for Firefox, allowing developers to see the exact line of Sass code responsible for CSS issues in the browser. I have not quite gotten that far to have tried that, but it is nice.

Identifying Code Smells in Shopify

2010-02-22T00:00:00Z

The Shopify inventory system is based upon Products. Once you define a Product, you can decide what Variants of this product exist, and add them as needed. One sure fire code smell with inventory here is when you see a system full of identical products, added to the inventory without using Variants. This smells for the reason that it is avoiding an efficient internal algorithm for rendering products.

If you have 50 chairs for sale, and they are all the same chair, same cost, same style, same everything, except, they vary in color, do not create 50 chairs one for each color. Create one chair, with 50 variants, one for each color.

This simple code smell just sent me reeling from the reek and now it is time to fix it.