Zumba Tech

Zumbatech takes on #hackforchange

2015-06-15T00:00:00+00:00

On June 6, 2015, a team of engineers from Zumbatech decided to contribute in an all-day hackathon event called Hackforchange. This is a national effort for civic hacking that brings engineers and designers together to make a positive impact in our communities.

Let’s not re-invent the wheel

The day before the event, our team got together to figure out what we were going to work on that would give the most impact to our community. We chose to work on generating visualizations of Florida vendor transactions. The first thing we noticed is that a couple of projects had already been underway to create restful APIs and alternate data formats for this data. We decided that an API that is specific to this data set wouldn’t be very reusable for other data sets, and it would still take an engineer’s effort to visualize the data from those APIs.

We wanted to make something that is generic enough to work with any sort of data set, be flexible enough for other engineers to create tools and visualizations via an API, and be easy enough for non-engineers to construct visualizations that fit their needs. A daunting task, especially for it to be mostly completed in a single hackathon. After some planning and discussion, we came up with a solution ready for hacking!

Hackathon

Bright and early on that Saturday morning, we arrived in the LAB Miami offices to work on a project we called Datamnom. The idea of the project is to make a generic ingestion program that can take in multiple data sources and populate an Elasticsearch index. Once the data is in Elasticsearch, a tool called Kibana can be hooked up to the Elasticsearch index we populated to create visualizations.

After writing a prototype nodejs program and setting up a Vagrant environment, we had Kibana up and running with data to visualize:

Here is team Zumba demoing our work to the Florida CFO, Jeff Atwater:

via Miami Herald

Presentation and Reception

After importing about 8 years worth of Florida Vendor transactions, the team presented our idea and applications to the group. Other groups that were working with this data set decided to use our tools to make their own visualizations.

Team @zumbatech presents FL State Payments API and visualization app. #hackforchange pic.twitter.com/R48vs6AnUZ
— Code For Miami (@CodeForMiami) June 6, 2015

Using @zumbatech's Payments API, @robdotd and team visualize printing costs across FL departments. #hackforchange pic.twitter.com/eB0BpAgpyV
— Code For Miami (@CodeForMiami) June 6, 2015

State challenge group is visualizing Vendor Data in a bunch of different ways and automating new viz. #hackforchange pic.twitter.com/ZGZYnUrcS6
— Code For Miami (@CodeForMiami) June 6, 2015

Great presentation from @CodeforFTL and group! #hackforchange @knightfdn @CFJBLaw @wyncode @socrata pic.twitter.com/sEidId7pE9
— Code For Miami (@CodeForMiami) June 6, 2015

Conclusion

We had a fun time at LAB Miami hacking together a project we think can really help lawmakers, researchers, and reporters visualize public data in a way that allows them to ask the right questions and help our communities.

W3C Push API Crash Course

2015-05-29T00:00:00+00:00

W3C’s Push API is exciting. Almost as exciting as all the possibilities that arise from having a persistent presence in your users’ browser. At Zumba, we already have ideas for it, and I was in charge of doing the gritty work of getting notifications to show up in the browser, but also developing a way to manage who gets those notifications. I had a lot of questions along the way that I will lay out and give you straight answers. Let’s get pushing already!

The ServiceWorker

The ServiceWorker is where the Push API lives. A ServiceWorker is a JavaScript file that defines activities that are allowed to run continuously, well after the life-cycle of a web-page. If you have heard of WebWorkers, ServiceWorkers are quite similar, but they will continue to work in the background even after a user has closed the web-page, which is key to being available to show notifications.

ServiceWorker Registration

Before we get started with registering, you need to set up a Google Application that can be used with Google Cloud Messaging to actually send the notifications; it is free for development and takes about 2-3 minutes.

Google Application Setup

Go to Google Developer Console and create a new application.
Note your new applications Project Number:
Go to the new application and under APIs & auth click APIs and look for “messaging” and enable:
- Google Cloud Messaging for Chrome
- Google Cloud Messaging for Android
Under the same sidebar menu, go to Credentials and generate new Public API access keys for server. Keep them handy for a little later on.

manifest.json

Google Chrome relies on a simple JSON file to detect your websites Google Cloud app. It is placed in the root of your website and it goes a little something like this:

Replace application_project_number with the project number of your Google App from the steps above. Moving on!

ServiceWorker Life-Cycle

ServiceWorkers have a registration process that goes through the following steps:

install: currently being installed
waiting: waiting on registration
active: registered and running

Scope

ServiceWorkers also have a scope which is set to the directory under which it can be found. For example, if your service worker script is located at /static/js/muh/scripts/ServiceWorker.js Then its scope will be: /static/js/muh/scripts/. This will not work; your users will not be doing their browsing in your static assets, so placing this script further up the directory tree is highly recommended. Service Workers also will not install if not served over HTTPS.

Registering

Now that you know what to expect with setting up a suitable environment for your service worker let’s get it installed.

A good installation script should also consider that their might be a service worker already in place and running. To check for that let’s do the following:

Now that we have the installation status: we know if registration of the service worker is necessary. Assuming registration is necessary:

ServiceWorkerRegistrations are important to push because they hold the key to our next step which is the PushManager. Once we have a reliable way to either register a ServiceWorker or retrieve an existing ServiceWorker’s registration, we can continue to…

The PushManager

The PushManager is a creatively named API that manages our push subscriptions. The entry point for it can be found on that ServiceWorkerRegistration I mentioned above. You should take the same approach with this as with service worker registrations. The functions you should know about are: getSubscription and subscribe.

As you can see here, we check for an existing push subscription, then if one is not found we attempt to get permission for one. This attempt is when the user is prompted to approve notifications.

Putting It Together

So, putting the two together, you probably end up with something like this:

This will auto-register a service-worker and will subscribe your website to push notifications and send the push subscription to your server.

PushSubscription

Providing that everything went well, and the user agreed to be notified, you will be given a PushSubscription object that will contain details about how to execute a push notification. (PushSubscription) This object contains an endpoint attribute which is the entry point for push notifications. (The Push API is still in flux, but I would recommend keeping the endpoint; Google is the only one running a W3C compliant push server but I am sure there will be more to come and this will be what distinguishes them.)

What distinguishes this user is the subscriptionId, which is like a device ID and will be how you reach this particular user with your push notifications.

Using the PushSubscription

Once you have the subscriptionId, you can send it to Google Cloud Messaging endpoint like so:

Put your Google Cloud project API key next to key= and put the users subscriptionId in registration_ids and this will send notifications but we need to tell the service worker how to receive them, and what to do with them.

Receiving Notifications with ServiceWorker

ServiceWorkers run in a completely different global context than a DOMWindow. The global object in a ServiceWorker is self, and the event to look out for is onpush. A very simple notifier service worker will look something like this:

This will extract that push data out of the event and show it as a notification.

The Bad News

W3C’s Push API is very new. Google Chrome is the only browser to begin working on implementation and it still isn’t finished. With that in mind, the Chromium development team has intentionally blocked arbitrary data from being sent through the Push API for now (chromium issue #449184). The reason stated is that as of right now, the API does not mandate encryption of incoming push messages. Without mandatory encryption, anyone can easily use a Man-in-the-middle attack to put bad data into push messages.

As stated in the linked chromium issue, push messages are really just fancy pings at this point. But those pings can be used to tell clients to fetch the latest data from your servers.

I will continue to follow the Push API and keep this article updated if any important developments occur. Drop me a line on twitter if you have any questions. Happy pushing!

Other Helpful Resources:

Relational Database Design

2015-05-22T00:00:00+00:00

Schema design is something that is very important to any application where data is to be stored and retrieved. It is also a topic that is often debated, controversial and discarded as something simple to be done during development. However, Proper planning BEFORE development is done will help save a lot of time down the road.

Over my last 15 years in application development, I have been refining techniques, best practices and conventions with regard to designing database schemas so that they can be used in real world enterprise solutions. In this article I will discuss some things I have learned so far and share my opinions and experiences. The views expressed in this article are my own and may or may not reflect that of Zumba.

Overview - The Basics

Wikipedia explains a relational database as a digital database whose organization is based on the relational model of data, as proposed by E.F. Codd in 1970. This model organizes data into one or more tables (or “relations”) of rows and columns, with a unique key for each row. Generally, each entity type described in a database has its own table, the rows representing instances of that type of entity and the columns representing values attributed to that instance. Because each row in a table has its own unique key, rows in a table can be linked to rows in other tables by storing the unique key of the row to which it should be linked (where such unique key is known as a “foreign key”). Codd showed that data relationships of arbitrary complexity can be represented using this simple set of concepts.

That’s a basic description of a relational database. When designing a database schema, it is important to know the purpose of the application(s) in which it will be used and where the data may be transported at a later time.

Where to begin

So, now that we have the basics covered and some definitions listed, it’s time to dig in and start designing. The first thing you need to do is decide (or learn) the purpose of the application. You will also want to know if there is to be any reporting performed on the data and if there is a data warehouse that will process this data for further consumption at a later date.

For the purpose of this article, we will assume that this is an application that handles products that are delivered to the end user digitally like an interactive book. The following are some questions you will want to ask the product owners at the start of a project. Most of questions should be asked in general for all aspects of the software design.

Who is the audience of the application?
- Who will be using the application and how?
Are there administrative areas?
- If so, will there be different statuses / states of items?
- Do changes to these need to be monitored and logged?
- Will there be a need to see real-time statistics?
Does admin activity need to be logged and reported?
Does regular user activity need to be logged and reported?
Will there be a need for real-time reporting of user activity?
Will the application need to interact with 3rd party systems?
How long must data be stored to be compliant with any rules that must be followed?

There are other questions that may be more specific to the type of application.

Naming Conventions

Just as coding standards and variable naming conventions are important, so are the naming of tables and columns. This is one of the more controversial subjects on database design. The interesting thing about “best practices” is they change and flip flop regularly. So, as such, many will agree and disagree with what I view as proper naming. Also, in some cases frameworks may require you to use certain naming conventions. If you are bound by such restrictions, then try to use as much of this as possible.

So let’s just get to it then.

Tables

The name should describe what is in the table and be very obvious to someone looking at the list of tables in your database.

For tables that will map back to an object in code, use singular object names like ‘product’, ‘user’, ‘color’, etc.
Join Tables that facilitate 1 to many relationships will have the main object listed first and the attribute name second and joined together with an underscore (eg. ‘product_language’, ‘user_product’).
Tables that store logged data will have a suffix of ‘_log’ (eg. ‘activity_log’).
Tables that store historical states of objects or object/attributes will contain a suffix of ‘_hist’ as in “history” (eg. ‘product_status_hist’).
When extending another table, typically a table in a framework that you want to be able to update without issue, you will add a meta table named the same as the table being extended with the suffix ‘_meta’ (eg. ‘user_meta’).
If this application will be packaged within an existing database, then all tables should contain the same prefix. The prefix should be small, kept to <= 3 characters + an underscore (eg. ‘zba_’).

Columns

Names of your columns should be as self-explanatory as possible when someone is simply looking at a list of names without seeing the full structure of the table to see types.

Try to avoid reserved words or words that are used in the SQL language. For instance, don’t name a description column ‘desc’. However, a suffix of ‘_desc’ is fine. More on that later.
Most objects will have a “Name” element. It is very easy to just call the column ‘name’ and be done with it.
- However, if you use the table name (or portion of) and add the suffix of ‘_name’, that will help to ensure that your query will not be ambiguous and that you will be able to pull data from a single query without having to specify an alias for each column name that is named generically.
- In fact, any commonly used elements of an object should have their column name begin with the table so that the same simple usage can be achieved. The convention for this would be to use ‘TABLE_’ as a prefix to the generic name.
When adding a column that is a foreign key to another table’s id, it should be named as the foreign table name + ‘_id’ (eg. in the product table, you would create the column ‘color_id’ to reference the color table).
- This format should be used for any foreign key columns. The pattern is ‘FOREIGN-TABLE_FOREIGN-TABLE-COLUMN’.
When adding a column that is used as a flag (1 or 0, true or false, etc.), it should contain the prefix: ‘is_’. So, an active flag would be named as ‘is_active’, published flag would be ‘is_published’.

Constraints

Constraints are used to create restrictions on Foreign Keys and indexes or just to put a restriction on the data going into the column.

Foreign Key: Use the prefix ‘fk_’ and follow this format: ‘fk_MAIN-TABLE_ref_FOREIGN-TABLE_FOREIGN-TABLE-COLUMN’ (eg. ‘fk_product_ref_color_id’).
Unique Index: Use the prefix ‘udx_’ and follow this format: ‘udx_MAIN-TABLE_MAIN-TABLE-COLUMN’ (eg. ‘udx_product_product_name’).
Non-Unique Index: Use the prefix ‘idx_’ and follow this format: ‘idx_MAIN-TABLE_MAIN-TABLE-COLUMN’ (eg. ‘idx_product_product_name’).
Multi-column indexes follow the same convention as single column unique and non-unique, except, you just keep appending the column names that are used.

Defaults Conventions

Now that the naming conventions are covered, let’s talk about some conventions when creating tables and how to put this to good use in your applications.

Standard Tables

Just about every web application will need one or all of the following tables.

user: This table may already exist if you are using an existing framework.
- If using a framework, you may want to extend the user data in a table called ‘user_meta’.
role: Just as you need users to use your system, you will want to differentiate those users. If using a framework in your application, this table may already exist.
status: Something in your application will likely have a need for different statuses. This is typically a small table and should likely make use of bit masks (discussed later).

There are other standard tables based on what type of application you are building. product might be a another common table. But if your app does not deal with products, this might not be needed.

Standard Columns

All tables should have the following column.
- A primary key column named simply ‘id’. This column can be simply a number or a large UUID string. Unless there is a really good reason, the ID should generally be an integer of whatever size you think the table may grow. The UUID can be another column in the table.
  - If numeric, It should be set to autoincrement (in databases that support it, or create a trigger and sequence in DBs such as Oracle to achieve the same thing).
  - If using a UUID string style, there is no autoincrement. You will need to handle the generated value another way.
Nearly all tables should have the following columns. The only exception to this is when maintaining integrity is not important.
- An ‘is_active’ column with a default of 1 or true. Instead of deleting from this table, you will set this to 0 or false. This requires that your queries that select data be aware of the column and its purpose.
- A ‘created’ column with a default as specified below. This should be set on row creation and not altered afterwards.
- A ‘modified’ column with a default as specified below. This will contain a timestamp of when the most recent change was made to the record.

Column Types

Basic Columns

Flag columns should be stored as either boolean column types (if supported) or 1-byte integers where they will hold either a 0 or 1.
Description or other columns that will hold markup or a lot of text should be some type of TEXT column based on what is appropriate for the length of the data and what is supported by your database.
String values should be stored in VARCHAR type columns. The length should be adequate to support the max length of the data. I, personally, like using binary increments. However, there is no benefit of doing so. Just be sure it is large enough to support the data being stored.
When adding a foreign key column, be sure to set its type to be the same as the column that it references.

Date and Time Columns

For columns that store date and time data, there are two ways to approach this. Perform you calculations and procedures in the database or the code.

database
- When offloading calculations and functions onto the database server, You will want to use datetime column types that are supported by the database server you are using.
- At a minimum, you should choose a date and time column that that is as granular as 1 second.
- You should use the same column type for all of your date-time columns so it will be obvious and predictable to anyone using the data.
- For the ‘created’ column, you will specify the default as whatever equates to NOW() for the column type chosen.
- For the ‘modified’ column, you will want to create a trigger that will set the value to whatever value was used for the default of the ‘created’ column ON UPDATE
- The Date and Time column type should support timezone as well so that the time can be meaningful and re-produced reliably.
code
- When writing code that can be used on multiple databases or when you want to perform manipulations or calculations in the code or directly in the SQL, then you should use a numeric type to store the data.
- The data should be stored as a unix timestamp (Number of seconds since epoch). So the column must be at least as large as an INTEGER(10).
- The Integer type should also be unsigned.
- For the ‘created’ column, you will specify the default as whatever equates to CURRENT_TIMESTAMP in the database or 0 and handle the value in the code.
- For the ‘modified’ column, you will want to have code that is centralized in whatever base object everything extends to set this value to be the current timestamp.
- Timestamps should be set using UTC timezone with the proper offset for the timezone desired. An additional column to store the timezone may be required as well.

Joining records

Join tables are the most common way to store many-to-many relationships. These tables typically just contain the IDs of both tables and the standard columns. Sometimes, there may be some additional meta data. But that is up to how they are being used.

Another option that works for attribute tables that have a relatively small number of records is to use bit masks. To use a bit mask:

Create your attribute table as normal except for the addition of 1 column called ‘bitmask’.
The value would start at 2 for the first row and then go up by the power of 2 from there.
- An easy way to achieve this is to create a trigger that will set the column value = the product of ‘id’ x 2 AFTER INSERT if possible.
- The data type of the bitmask column should be an unsigned integer that is large enough to account for all of your records as the power of 2 increases.
Your main object table will then have a column called ‘FOREIGN-TABLE_bitmask’ (eg. ‘language_bitmask’).
- To store the relationships, you will simply OR the languages bitmask values together and store that sum in the language_bitmask column.
  - Just a tip, you can simply add all of the values together and it will yield the same final number as OR-ing them all together.
- To query the data (Example: pull product and language data), you would join language lang with product prod ON (prod.lang_bitmask & lang.bitmask = lang.bitmask).

The examples shown above are more pseudo code as I’m trying to not be database specific. But you should be able to adapt them to your environment fairly easily. Bitmasks cut down on the number of joins and will provide some performance improvements. Just make sure the columns are indexed.

Warehousing

One of the benefits of following these conventions and standards is that it will make the data very easy to be reported on and warehoused for Business Intelligence reporting and insights. In order to tie data together from multiple applications and systems, a Data Warehouse is used. There are many different commercial solutions for this in existence (Cognos, Oracle BI, Tableau, …etc.). However, they all basically store processed data in a database of their own.

The database in a data warehouse will not be designed the same as the database used in an application. Multiple queries per second are not as important as data integrity. The structure is set up in a way to facilitate the generation of reports as fast as possible while still maintaining the integrity of the data.

ETL

In order for the warehouse to collect data from many disparate systems, it uses what is called an ETL process. ETL stands for Extract, Transform, Load. Essentially, what it does in a basic sense is

Pull data from the application database.
Transform the data or parse it so that it will fit into its schema and be able to be used with the data from the other systems.
Insert / Update records in the warehouse database with the newly transformed data.

Why Is This Relevant

The way that the ETL process determines what should be brought into the warehouse is based on multiple factors. However, the main one is based on when data changes. So columns like ‘created’ and ‘modified’ help to determine what records have changed since it last ran. The ‘is_active’ column is used to help with letting the warehouse know when data is “deleted” (A soft delete).

Having a consistent numerical id column allows the warehouse to tie back records to the records in the application database tables. The warehouse will use its own IDs, but will need to be able to keep track of changes in the application. And the ID column is the best one to use.

Conclusion

This article is a general guideline for schema design and is intended to be a starting point for application database design. It should be extended and altered where necessary. Following these guidelines will help to build a solid foundation for any application. I hope this has been informative and/or thought provoking.

Thanks for reading.

How We Do CSS at Zumba

2015-03-31T00:00:00+00:00

As you may or may not know Zumba’s website consists of a robust app that includes a fully custom ecommerce shop that supports international transactions, class and training search and registration, Zumba Instructor Network (ZIN) admin portal and many other apps. Our in-house tech department is responsible for managing the interactive products, developing, testing and deploying the projects.

I was inspired by the teams at Ghost and Github to write this article to give you an idea of how we do CSS at Zumba.

Intro - Browser and Device Support

We try to keep our code as light, modular and DRY as possible. We use Modernizr, Autoprefixer and some Foundation components. We try to keep our nesting no more than 4 levels deep and we try to make sure that our naming convention lends itself to reusable components.

We support the last few versions of popular browsers (Chrome, Safari and Firefox). At the time of writing this we currently support IE 8+. About 2% of our users still use IE 8; we are not out of the woods yet.

We made the leap to a responsive application with our recent redesign of the site. Therefore, we support a wide range of desktop, tablet and mobile devices. At the time of writing this we don’t currently support smart watches. We will soon, though… so, “watch” out for that… sorry.

Preprocessor

We use Sass with Compass as our preprocessor. Our team is proficient and has much training on this popular pre-processor and it’s large following helps.

Folder/Sandwich Structure

Here’s the fun stuff - we structured our css based on a few important factors:

+ Modularity and Reusability

Develop components that can be reused throughout the application.

+ CSS Rules

Consider the size of the compiled stylesheets per section of the site to not exceed IE9’s rules limit of 4,095. Our previous site was compiled into two massive stylesheets and we learned the hard way. We currently break it down per section of the site. See examples below.

+ Light stylesheets

Keeping the stylesheet declaration low kills two birds with one stone. We break the stylesheets down into individual sheets that contain only the rules for each section of the site—this has the added benefit of avoiding unnecessary or redundant styles per section. Doing this helps us keep our stylesheets light.

Can I have my sandwich with a side of css? Zumba is a happy and energetic fitness lifestyle company. That doesn’t mean we can’t have our sandwich and eat it too. We decided to structure our css like a sandwich order.

Plate

The plate represents the foundation of the code and is reused throughout the site. Example:

plate
|— _angular.scss
|— _animations.scss
|— _fonts.scss
|— _icons.scss
|— _mixins.scss
|— _typography.scss
|— _variables.scss

Toppings

These can be added to your sandwich to make it as delicious as you want it. These consist of mini components that can be reused throughout the site, but is not necessary for every page. Example:

toppings
|— _button.scss
|— _input.scss
|— _shares.scss
|— _tooltip.scss

Sandwich

These nom noms can contain a combination of toppings to make a complete sandwich or component. These are usually a bit more complex and a little more specific. Sandwiches can be used throughout the entire site and can be customized per section of the site to meet that section’s needs or requirements. Example:

sandwich
|— _accordion.scss
|— _featured.scss
|— _form.scss
|— _hero-slider.scss
|— _modal.scss
|— _pagination.scss
|— _rangeslider.scss

Example of our overall structure:

assets-build/css-blt
|— checkout/
|— classes/
|— consumer/
|— header-footer/
|— plate/
|— sandwiches/
|— shop/
|— toppings/
|— _settings.scss
|— classes.scss
|— consumer.scss
|— header-footer.scss
|— main.scss
|— shop.scss

main.scss - Consists of the basics of our pages’ layout.

CSS structure

This is an example of how we expect our stylesheets to be laid out. This is not an actual file.

Each file is named after the page or the page module it supports
We never style ID’s only Classes
Keep nesting down to a minimum and no deeper than 4 levels including pseudo selectors
Group related properties wherever possible
Media queries go after properties and the order should reflect mobile first

Naming

Variables and Mixins

Our variables and mixing naming is pretty specific in order to avoid any confusion. We have been here long enough to understand that the company likes to make simple to drastic changes to support the business in general. We have to develop keeping scalability in mind and this variable/mixin strategy just makes more sense for us.

Mixin

Color Variables

Classes

Our preferred technique is a more descriptive naming convention over deep nesting.

Media Queries

For the most part we target based on viewport width. We have a feature or two that depends on the detection of touch. We stick to three basic breakpoints and rarely make exceptions for custom breakpoints. We don’t want to compromise code quality to consider every viewport width consideration.

Typography

Linting

Currently our linting consists of pull request code reviews. This is what we look for:

Levels of nesting
Indentation
Flag the use of ID’s
Spacing after selectors and properties
Formatting of properties
Code that’s commented out
Formatting of selectors - lowercase and hyphened preferred
Null values should be 0 or none instead of 0px
Use of color names should use variables
Duplicate selectors or properties

The fun never ends

This post and our style guide that is soon to come will never be a done deal written in stone. What we build today is cutting edge and future proof until the future is now and we realize how crazy we were when we developed using archaic methods and tools. The tech world moves at an insane pace and that’s what makes our job fun. New technologies will emerge and we will use them to make our applications and our lives as developers better. Therefore, this is relevant for the time being and I hope that it covers us for a very long time, but we all can’t be surprised if we have to come back and revise this to support new new. Until then, we will enjoy this sandwich.

Zumba^® Technology Day

2014-12-30T00:00:00+00:00

Let me start out by saying, that before I started working here at Zumba^®, my perception of companies that were not “tech companies” was completely wrong. I was very naive, and thought that ‘those’ companies did not have the correct culture to nurture and expand my career as a software engineer. Not all companies are the same, but at least there are some companies, such as Zumba^® that strongly believe in technology. Our team is constantly innovating, and we are on top of the latest technology. Don’t believe me? Here is some inside knowledge about our Zumba^® Technology Day.

Nowadays, a lot of the big tech companies have some sort of program that incentivizes it’s employees to tinker with ideas that are not part of their assigned work. Apparently this concept was started by 3m and then was made famous by Google with their “20 percent program”, in which employees get 20% of their time to work on side projects or ideas. Following in their footsteps, we decided to implement a similar program in-house called “Zumba^® Technology Day”, where the technology department gets at least one day a month (usually the last Thursday of the month) to actually experiment and come up with new ideas that can benefit the company.

The way our tech day works is pretty simple. It started simply with a google doc that we used as master list of ideas that anyone at anytime could go in and update. Over time, we refined the process to a Kanban project in Jira. This change made it easier to maintain and team members can now take ownership of what they want to work on and invite other team members to contribute. Note that we do not force anyone to use the google doc or the Jira project. In fact, we only have 2 actual rules:

The only interrupts that will be taken are business critical interrupts.
The Technology team must work on something that stabilizes our current systems or designs for the future. This includes documenting what we now have, designing a future vision, prototyping something that will help solve a business problem, documenting an improvement to the development process, automating tests, writing scripts, bashing a bunch of pet peeve bugs, researching an alternative technology, or the like.

It’s been a year and half since we established the Zumba^® Technology Day in our department and during this time it has proven to be a success. Not only some of the ideas/projects that we have implemented started with a tech day and became actual projects that are now vital part of our company, but it has also allowed us to have a day where we can interact with each other in a different setting. We have created a work setting that we can work without the pressure of a deadline and restrictions from the business- It’s our time to be creative!

If you have been missing working with exciting technologies and experienced engineers on solutions to complex problems, then Join the team! We have several openings for software engineers and frontend developers.

Caching CakePHP 2.x routes

2014-10-26T00:00:00+00:00

At Zumba we are continuously looking for optimization in our applications. These optimizations help to reduce the server loads, consequently reducing the number of servers and saving money. Besides that, it gives a better user experience for the end user by serving content faster and in some cases saving on consumer bandwidth (specially for mobile users).

This week we profiled our app using Xdebug profiler and we identified the router was responsible for a big part of the request time. In our main app we use over 130 custom routes which makes CakePHP generate an object for each route, and consequently parse and generate a regex for each route. This takes significant time and many function calls to do it.

In order to optimize the routing time, we started looking at options to optimize our routing process. After some research and deep checking in our codebase as well as CakePHP’s code, we found we could cache the routes easily. The solution we found is easily applicable for many other CakePHP applications. Basically it consists of exporting the compiled routes into a file and loading it instead of connecting them on every request to the Router. This strategy is similar of how FastRoute caches their routes.

First, we moved all the Router::connect() to another file called routes.connect.php. On the routes.php we added the logic for the caching. In the end, we ended up with something like this:

We had a couple options here. For example, we used var_export instead of serialize. It changes how the application caches the file and adds some extra steps to the process. Using serialize you can just cache in memory (using some cache engine like APC), avoiding writing routes to the disk and avoiding to change the default route class.

We chose to use var_export and dump the output to a file because it allows PHP to opcache the file, avoiding to re-parse everything again on every request. Using serialize it generate a string that needs to be parsed and executed on every request. Depending of your app and the number of routes that you have, the use of serialize is simpler and faster than var_export. Give both approaches a try and compare the performance between them.

Using var_export brings some consequences, which is the requirement of implementing the magic method __set_state. This method is not implemented in CakePHP core until the version 2.6. I opened the PR cakephp/cakephp#4953 to support it on CakePHP 2.6+. So, to solve this problem in CakePHP prior to 2.6 we created a class that extends Cake’s CakeRoute to implement the magic method and this looks like it:

PS: This code is not compatible with PHP 5.2. If you are using PHP 5.2, stop everything and upgrade your PHP version.

With this class and changing the default route class in the routes.php you can cache the routes using var_export. If you have plugins with routes, you may need to do some additional changes. If you have plugins but these plugins doesn’t have any route, Cake automatically create a route to it using the PluginShortRoute (which also doesn’t implement the magic method). It means you probably will have to remove these classes from your routes before build the cache. Knowing these limitations you can also workaround this type of issue by creating another extended classes.

If you are wondering why we create a temporary file and rename it to the final filename it is to avoid a concurrency issue between one request writing the file and another request reading the file at the same time. It could be avoided by using file lock, but it would stop all concurrent requests until the cache is finally done and stored on the disk. Using a temporary file and renaming it is an atomic operation, so it is avoided. Thanks Jose Diaz Gonzalez (@savant) for pointing it out.

We used the sha1_file() function to clear any kind of cache when a route is changed in routes.connect.php. Everytime you change the file, the SHA1 of the file will change and consequently it will generate another cache file. This allows the cached file to regenerate for seamless deployments where routes where changed.

I would like to present some numbers of these changes, but it is very subjective because it depends a lot from the hardware (specially the disk), the number and type of routes, etc. What I can say is for our case the time to load the routes are half of the time from connecting on every request and to generate an URL using Router::url() (or via some helper/controller) it is four times faster on the first route that hits the last route (usually the generic one from Cake).

One interesting thing we found on the tests were that loading the cached file and routing to the first route was faster than connecting all the routes via Router::connect() and matching the first route (which just compile one route).

In summary, the changes to cache the routes are small for most of the applications. There is different approaches that you have to test and decide which one fits better for your application. Also, some limitations could block your cache, but most likely there is a workaround for it. If you can’t find one, contact me and I can try to help you.

Developing With AngularJS? Forget jQuery Exists.

2014-08-02T00:00:00+00:00

Before I begin, take a moment to remember how hard our lives were before jQuery ironed out the treacherous wrinkles of cross-browser development. Client-side Javascript is to jQuery as mammals are to sharks with frickin’ laser beams attached to their heads. Zumba® Tech engineers are big fans of jQuery.

Now, if you are writing an AngularJS application, do us all a favor and forget jQuery exists.

Here’s the rub — the power of simple jQuery scripts can sometimes be incongruent with the goals and maintainability of an AngularJS application. jQuery is a library with a lot going on under the hood, but it is primarily used for these things:

Performing AJAX operations
Animating HTML elements
Handling browser events
Querying, traversing, and manipulating the DOM

There are caveats when using jQuery to execute any the above actions within an AngularJS app, but this post is focused on the last two: Events and DOM interactions.

Separation of Concerns

Querying for CSS selectors is very difficult to do without implicitly tying your code to the DOM; it’s nigh impossible. This can lead to brittle code in the long run. For example, when you write $('.soso') you create an implied contract that the HTML template will contain an element with the soso class. If you later remove the soso class and add a new awesome class to the element, you have broken the contract and the Javascript may stop working.

At a fundamental level, the principle of separation of concerns was violated. The soso class was given meaning outside of its implied styling. Worse still, nothing in the HTML template indicated that the soso class was more than just a boring presentational element of the page.

DOM traversal is also problematic. For example, using jQuery you can query for a specific element, get that element’s siblings, find any links that are children of those siblings, and bind a click event to those links. This kind of script is easy to write, but it is tied to the structure of the template. Adding or removing DOM elements in the hierarchy can very easily break the script.

Manually Querying the DOM Is Like Writing SQL

Here’s some pseudo code for you. Use your imagination.

SELECT DOMElement FROM document WHERE document.id = "#navBar"

This is conceptually equivalent to writing $('#navBar').

Querying the DOM is a low level task that is akin to writing SQL statements. You would never sprinkle SQL statements around your backend codebase, so why do the same thing inside your AngularJS directives?

Indigestion

One of the awesome features of AngularJS is two way data binding:

Any changes to the view are immediately reflected in the model, and any changes in the model are propagated to the view.

AngularJS periodically loops over the properties of the “model” and updates the DOM if the data has changed. This is called the $digest cycle. When you use ngClick or scope.$watch, or even the $timeout service, AngularJS will automatically kick-off a $digest cycle for you.

However, when you manually bind click events via $(element).on(), AngularJS is not aware of those event handlers. Data in the model could be updated, but the view will likely remain stale. You could manually call scope.$digest or scope.$apply, but this quickly becomes messy and bug-prone. If you use lots of manual $digest calls, you are bound to start getting AngularJS errors about trying to start a digest cycle when one is already in progress. Trust me.

Refactoring a Problematic Directive

Breaking away from the jQuery methodology is hard. I figure that the best way to illustrate these concepts is by example. So, I’ve cooked up a brittle, jQuery-style directive that I’m going to refactor into a resilient, reusable “AngularJS Way” component.

The Template

Let’s say we want the content paragraphs to toggle visibility when the user clicks on the corresponding links in the list, and we want the li elements to indicate the active paragraph. The following directive gets the job done.

The Brittle Directive

The above code is somewhat clean on the surface, but there are a lot of implied dependencies on the DOM. For example, the directive will only work if the links remain children of li tags. The directive will break if the DOM is modified to put the links inside of a div or p tag.

Every time a link is clicked, the code traverses the DOM. That’s expensive in a large system. It could be refactored to cache the DOM traversal, but doing this introduces a new dependency on the state of the DOM. For example: What if some of the HTML gets changed to be inserted dynamically? Suddenly the variables will be empty at linking time, yet again breaking the directive. Event delegation mitigates this, but we still don’t solve the problem of hidden functionality. When examining the HTML, it is not clear that there is behavior attached to the links and paragraphs.

Use the Scope

We need to divorce the data/state from the template’s attributes and classes. To do that we can introduce ngClass, ngClick, and ngIf to the template. We can then keep track of the links’ states inside the directive scope, and leave the DOM manipulation to Angular.

The New and Improved Template & Directive

There are no dependencies on the DOM at all. It’s just a directive that wraps an object called scope.active that contains some boolean values. The booleans get flipped on and off when the select function is called. This directive is relatively immune to DOM updates. You could change all of the classes in the template; you could delete and add new HTML; you could change a tags into div tags or p tags into span tags. Nothing will break in the JS file. There are no classes being toggled in the directive. If a developer decides to change the classes on the li elements from active to awesomesauce, everything still works. This is awesome.
There is no DOM traversal happening at all. This is awesome; and it’s fast.
Looking at the HTML reveals how this widget works. It is declarative, which is awesome. (there’s a theme here)
Since all of the click operations are handled by ngClick, everything will remain inside the digest cycle, and I don’t have to use scope.$apply anywhere. It will just work.

jQuery Under the Hood

AngularJS ships with jqLite:

jqLite is a tiny, API-compatible subset of jQuery that allows Angular to manipulate the DOM in a cross-browser compatible way. jqLite implements only the most commonly needed functionality with the goal of having a very small footprint.

The interesting thing here is that Angular uses jqLite to perform DOM manipulation such as adding and removing classes for the ngClass directive. This allows ngClass and other built-in directives to be an abstraction that separates the queries from your code.

Summary

jQuery is a badass tool for scripting interactions with the DOM. It’s a global hammer that you can whip out anywhere in the code and smash out your feature on the spot. This is awesome, but it leads to brittle, non-reusable, hard to maintain code (inside AngularJS, at least).

TL;DR

When trying to solve a problem or add functionality to an AngularJS app, start by forgetting about jQuery solutions. Reach for directives like ngClick and ngClass. Doing so will likely result in a more elegant solution.

Enforce code standards with composer, git hooks, and phpcs

2014-04-14T00:00:00+00:00

Maintaining code quality on projects where there are many developers contributing is a tough assignment. How many times have you tried to contribute to an open-source project only to find the maintainer rejecting your pull request on the grounds of some invisible coding standard? How many times as a maintainer of an open-source project (or internal) have you had a hard time reading code because there were careless tabs/spaces mixed, if statements with no brackets, and other such things. Luckily there are tools that can assist maintainers. In this post, I’ll be going over how to use composer, git hooks, and phpcs to enforce code quality rules.

There are a couple of things to keep in mind. First, you want this process to be as simple as possible. Secondly, you want it to be easy to run when necessary. Finally, you want it to be universally accepted as part of your contribution procedure.

There Is No Catch

What if I told you that it could be done without the developer even knowing it’s happening?

Most modern PHP projects use composer as their dependency manager. Before you can make anything work, you need to run composer install. This is where the magic happens.

Phpcs Dependency

First, we need a development dependency specified to install phpcs. It looks something like this:

{
    "require-dev": [
        "squizlabs/php_codesniffer": "2.0.*@dev"
    ]
}

Install Scripts

Composer has a handy schema entry called scripts. It supports a script hook post-install-cmd. We will use this to install a git pre-commit hook. Adding to our example above:

{
    "require-dev": [
        "squizlabs/php_codesniffer": "2.0.*@dev"
    ],
    "scripts": {
        "post-install-cmd": [
            "bash contrib/setup.sh"
        ]
    }
}

This will run a bash script called setup.sh when the command composer install is run.

Setup the Git Pre-commit Hook

In our setup.sh, we will need to copy a pre-commit script into the .git/hooks directory:

#!/bin/sh

cp contrib/pre-commit .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit

This will copy our pre-commit script from the contrib directory to the hooks section of the special git directory and make it executable.

Create the Pre-commit Hook

Whenever a contributing developer attempts to commit their code, it will run our pre-commit script. Now all we need to do is run the code sniffer rules on relavent files specific to this commit:

This script will get the staged files of the commit, run a php lint check (always nice), and apply the code sniffer rules to the staged files.

If there is a code standards violation, the phpcs process will return a non-zero exit status which will tell git to abort the commit.

Bringing it all together

With all of these things in place, the workflow is as follows:

Developer runs composer install.
PHP Code Sniffer is installed via a dev dependency.
The post-install command automatically copies the pre-commit hook into the developer’s local git hooks.
When the developer commits code, the pre-commit hook fires and checks the staged files for coding standards violations and lint checks.

This is a relatively simple setup that can save pull request code reviews a significant amount of time preventing back-and-forth on simple things such as mixed tabs/spaces, bracket placement, etc.

Asynchronous Control Flow with jQuery.Deferred

2014-01-16T00:00:00+00:00

Zumba® Tech is a big fan of javascript promises; they free our code from callback hell by allowing us to use a more elegant and abstract structure. With promises, asynchronous control flow becomes intuitive (first a and b, then c) and fun. Today, I’ll show you how to adapt a third party library that uses callbacks to a promise interface.

We’ll be using jQuery.Deferred

Chris Saylor blogged about the node.js async module last November. I’m going to focus on the client side for this post, but the principles apply to any environment.

There are many great promise libraries:

Q — A tool for making and composing asynchronous promises
RSVP.js — tools for organizing asynchronous code
when.js — the promise library from cujoJS

There’s even a spec for promises in ES6.

However, until the native spec is fully supported we’ll be using jQuery.Deferred ^[1] on the client.

A Typical Scenario

We recently built a new internal app to handle some administrative tasks. We used Bootstrap to get up and running quickly, and we stumbled upon a great library for handling our dialog boxes called bootbox.js from Nick Payne.

Here is a typical snippet of code, straight from bootbox:

With this code, bootbox will open a dialog containing the "Hello world!" text. The callback is fired after the user clicks on the "ok" button and the Example.show("Hello world callback"); code is executed.

This is already sweet, but soon we found ourselves writing a lot of boilerplate when we implemented the confirm dialogs:

That’s not overly messy, but handling these conditions in every confirmation dialog with inline callbacks didn’t appeal to us.

Wrapping Bootbox With an Adapter

First we created a new object that wraps the bootbox logic. We called it Popper (we’re not always clever with lib naming).

Returning a Deferred Object

The above snippet isn’t very impressive. However, now that we have a wrapper we can adapt the bootbox interface to use a deferred object:

In effect, confirming the bootbox dialog will resolve the deferred object, and canceling the dialog will reject the deferred object. Our implementation code now looks like this:

This has a couple of advantages: * There are no more messy if blocks to parse; the code is more “flat”. * We’ve changed the flow of the application from using a single, multipurpose callback into using separate callbacks with unique concerns.

jQuery.ajax returns a Deferred

There were many cases where we would fire off an ajax request after the user confirmed the dialog. Since jQuery returns a deferred object from $.ajax, we were able to envelop that logic into our adapter as well:

A Few Steps Further

The above code is nice, but there are still a few outstanding issues: * .done and .fail are not very semantic method names for confirming or canceling a dialog. * It would be nice to register callbacks that fire as soon as the buttons are clicked, and additional callbacks that fire after the ajax responds.

To solve these issues, we created a map of semantic method names (ok maps to done, cancel maps to fail, etc). Popper’s methods return a new object that contains these semantic methods, making our implementation code look beautiful:

The Source

Here’s a link to the full Popper.js source.

Conclusion

I hope I’ve expanded your toolbelt a little by showing you how to use a promise pattern in a real-world scenario.

Do you like the Popper lib? Leave us a comment!

[ 1 ] We know that jQuery.Deferred objects are not Promise/A+ compliant, and we sympathize with why that’s non-optimal. However, since we are already using jQuery in most of our projects, jQuery.Deferred solves 95% of our needs without requiring us to depend on an additional library.

Introducing Json Serializer - An open source library to serialize to JSON

2014-01-06T00:00:00+00:00

After the release of some of our libraries, such as Symbiosis, MongoUnit and CSV Policy, we are proud to announce our newest open source project: Json Serializer. The Json Serializer–as the name suggests–serializes content in JSON format. It is a very simple concept, but very useful in some cases.

Use Case

Applications are becoming increasingly more complex, and companies are worried about their consumers. At Zumba we are no different. Our ecommerce system handles thousands of consumers daily and we have many new features coming. Without a decoupled and maintainable system we can’t grow very fast.

In our shopping cart we implement some entity classes and some other classes to group the entities together. For example, we have entities to represent the cart item, the cart message, the promotions/coupons applied, etc. All these entities are associated to a main cart entity that represents the consumer cart. This cart entity is stored in our database so users can switch between devices and/or come back a few days later and add the items to the cart, etc. The system does not rely on the user session.

For some time we just serialized the cart using the PHP serialize() function and stored it in a binary field in a MongoDB collection. It worked very well until we had to extract the data to use in marketing emails, analytics, etc. The serialize function doesn’t generate an easily encoded string that is easy to parse by another system that is not PHP based. We couldn’t keep creating code around our shopping cart to have analytics or marketing information either. The solution was to store the cart in a format that would be easy for PHP to quickly serialize/unserialize and also easy to integrate with partners or different tools. The solution was to encode the entities in JSON.

JSON is very well supported by many languages and tools. Since we are using MongoDB to store the carts we can easily generate queries directly from MongoDB to have some statistics. This change decreased the performance of managing carts, but in the main scope it was a gain.

What about `JsonSerializable` in PHP 5.4?

The JsonSerializable interface is used with the json_encode() function to serialize an object. This means it is very useful to convert your class to JSON when you are responding with the entity to your API, for example; however, it doesn’t support unserialization.

What about using `json_encode` with the `Serializable` interface?

Nope. If your entity implements Serializable you can define how to serialize and unserialize, but PHP wraps their serializable code around it which means the output is not 100% JSON.

Hmmm, so what about JMS Serializer?

JMS Serializer is a great library and does serialize to JSON, but it doesn’t have as many features as this library. For example, when you de-serialize, you have to pass the class where the data was generated. Also, they don’t support nested encoding.

Why not just create a method to aggregate all the data and respond as JSON?

Yes, you can. But are you sure you want to make a method to cover all situations and nested levels to aggregate everything? In our case it would be painful and would require more code than this library. If it works for you, go for it. You probably will get better performance, and it will probably be easier to store the JSON however you want (I mean not in the same structure of your classes, as the JsonSerializer does).

You are convincing me, but I have so much crap on my classes. Can I ignore some of the properties?

Not a problem. The library supports the magic methods __sleep() and __wakeup(). In each class you can define what will be stored in the JSON and what to execute when unserialized. It works exactly the same way as the serialize() and unserialize() functions.

Ok, but my code is a little complex. I have some double linked lists, some recursing objects, etc.

This is also not a problem. The library handles code recursion. You can call the serialize method and enjoy it.

F*cking amazing, but I doubt it will work in my old environment.

The library supports PHP 5.3.6 and above. We don’t need any other dependency. If you are using any PHP version before that, stop now, read some blogs, and go upgrade your system! PHP 5.3 only has a few months of support left.

Is the library compatible with `Serializable` or `JsonSerializable`?

Yep. Each one has different functionality, so everything is fine. You can still use Serializable with serialize() and unserialize() functions. You can use JsonSerializable with json_encode() to respond to your API and our library to serialize the object for internal storage.

You are talking too much; you’re like a car salesmen. Do you have some code to show?

Indeed, let’s go to an example:

Is that clear?

Wow, amazing! How can I get it?

You can install via composer (package zumba/json-serializer), or download directly from GitHub. If you find something wrong, or something that we can improve, feel free to open a GitHub issue. If you want to make a pull request, even better. :)

You are welcome to check our other open source projects on https://github.com/zumba.

Tame Stale Pull Requests with Drill Sergeant

2013-12-07T00:00:00+00:00

Our tech team uses a process similar to the Github Flow, whereby all changes are pushed to the main branch of a repo by way of Github pull requests. We use this internally, but it is also the most common mechanism for open source projects to get community contributions on Github. We think it’s a great way to encourage code reviews and incremental improvements prior to going into a main production pushable branch, however it’s not without its pain points.

Stale Pull Requests `==` Death to Open Source

How many times have you come across an open source project that does almost what you need? How many of these have pull requests that are one or more months old? It’s discouraging when you know that you could contribute and fix whatever problem may exist with the project but you know that the maintainer has abandoned or is not responsive to merging pull requests. As both a maintainer of open source projects, it’s critical to stay up-to-date and aware of open pull requests.

Introducing Drill Sergeant

As a part of our development lifecycle, stale pull requests mean that QA doesn’t have the opportunity to begin testing until it is merged. Therefore, it is vital that the dev team know about pull requests to be reviewed and merged as soon as possible. We developed an open source tool called Drill Sergeant to send reports via email of any pull request older than a specified period.

Reporting for Duty

An example report from running Drill Sergeant against the CakePHP project configured for one week staleness looks something like this:

The command for the above:

GITHUB_TOKEN=atokenhere drillsergeant -e "myemail@address" -r "cakephp/cakephp" -s 168 # 24 * 7

Schedule It with Crontab

This tool is meant to be run on a schedule and it’s easy to setup to get daily reports:

0 0 * * *	GITHUB_TOKEN=atokenhere drillsergeant -e "myemail@address" -r "cakephp/cakephp" -s 168

Install It

The full instructions can be found in the readme of the project

Handling Asynchronism with Promises

2013-11-06T00:00:00+00:00

While experimenting with Node.js for a coding game we are working on called Node Defender¹ for the Ultimate Developer Conference, many asynchronous problems arose that we were unaccustomed to dealing with since we are primarily a PHP shop (read: synchronous). The examples covered in this post are specific to MongoDB, but the essence can be applied to any asynchronous process.

Intro

First, let’s discuss the problem. The game server is a Node.js app that hosts both the websocket communication as well as a leaderboard of top scores and top categories. In the backend, we use MongoDB to store all the games that occur and use MongoDB’s aggregation framework to get the top 10 scores and the top scores in each category: Kills, damage, highest round achieved, and “wave clears”. Retrieving the top score for each category requires separate queries to MongoDB; each query is asynchronous. How would we go about sending one request to the leaderboard that executes these 4 queries and returns them all without causing the event loop of Node.js to hang waiting for synchronous requests? The answer is “promises.” Martin Fowler does a good writeup about what a promise is in javascript in this post.

The Aggregate

Each game is represented in MongoDB as a document containing various values for calculating the score. In order to get the highest value in various categories the following aggregate query needs to be run:

For a quick explanation:

The sorter controls which category by which we’re going to sort; descending in this case.
The $project is the projection of the aggregate we’d like MongoDB to return. In this case: the player’s name and the value of the category.
Finally, the $limit of 1, we just want the top value.

Async All the Things

Where does the promise come into play? With the async library, we can use a mapping to get the above aggregate for all required categories, execute in parallel, and return the results to one callback called the promise. In typical promise setups, there would be separate functions for a success, failure, and always. In our case, we’re going to setup our promise to receive all conditions and let that method determine how to handle the success and failure results.

We use the async “map” method to accomplish this. It allows us to pass an array of categories to the above aggregate, run the getTopCategory method against each one, compile the results, and then call our promise.

Explanation:

The map array contains the categories we want to run with the aggregate queries.
We bind the db (which is the connector to MongoDB) to the getTopCategory method, which allows this method to have direct access to the db instance without having to pass it in as a parameter.
Finally, the promise anonymous method attached to the Async.map parameter gets the results (or errors) from all of the category aggregates and compiles them into a single list to be consumed by the calling method.

Conclusion

Using the “promise” pattern with the async library, we can make concurrent asynchronous requests and return the results of asynchronous queries into one unified usable list back to the callee. This prevents our request from blocking other requests to Node because the event loop isn’t waiting on a series of synchronous queries to MongoDB.

Footnotes

Node Defender is a javascript programming game where you defend against waves of murderous server side code that wants nothing more than to kill and disconnect your client. The client code is open source, and we will be doing a series of blog posts about the construction of the project as well as the infrastructure used to run it.

Incorporating Mongounit into Multi-datasource Models with Traits

2013-10-30T00:00:00+00:00

A while back we open sourced Mongounit, a PHPUnit extension for testing models utilizing mongodb. One key issue that we’ve discovered as we incorporate MongoDB into more of our data models is that extending Mongounit’s TestCase class limits that unit test towards Mongo only as the datasource. Since only a portion of our data is in Mongo while the remaining is in MySQL, limiting a test case to work with one datasource or another is too limiting.

Workarounds

The first workaround we employed (as we use PHPUnit’s dbunit extension) was to separate the tests of methods that use MongoDB versus methods that use MySQL into separate unit test clases. This worked for a while until we encountered situations where it was either not convienient to split into separate unit tests, or a method was interacting (or calling via protected methods) both data sources.

The second workaround was to manually set and clear data in mongo in the test cases, which works of course, but causes a lot of duplicate code strewn throughout the test code base.

Enter 5.4 and Traits

PHP does not support the concept of multiple inheritance which could have easily solved our problem. However, in PHP 5.4, a usable alternative is available. We are fortunate that our code base is workable on PHP 5.4, although not everyone is in the same position. For those not on PHP 5.3 and below, one of the above workarounds will have to suffice.

One of the nice features of 5.4 was the introduction of traits. Traits provide a simple way to re-use common methods and was a great solution to our lack of multiple inheritance. The first step was to actually create a useful trait in Mongounit to serve as means of attaching mongo support to test cases that extends the dbunit extension.

You can see Mongounit’s trait implementation in Github available as of version 1.1.3.

The next step was to use the trait in our dbunit based test case. This means seting up the “setup” and “teardown” methods from the trait as well as implementing the abstract methods required for connecting to mongo and retrieving fixture data. These methods are the same as what is available in the TestCase, so in our case, it was a matter of moving the implementation into the trait method.

Now that our test case has implemented the trait, we now have access to setup fixtures for both MySQL and Mongo and utilize both within the same test case.

Conclusion

Using some of the “newer” features of PHP, we are able to make our test cases more flexable and easier to use without having to reimplement entirely what mongounit does into a dbunit extended class.

Symbiosis: Instanced Dispatcher Support Added

2013-09-22T00:00:00+00:00

A new version of Symbiosis v1.2 has been released.

Introducing Instanced Dispatchers

Prior to v1.2, all events were globally registered via a statically defined event manager. This posed issues when some models needed to have events contained to themselves, instead of available in the global event space.

To solve this issue, we’ve added an Event Registry. This has the same API and functionality of the Event Manager, but in an instanced state. Now you can create an Event Registry anywhere and register events specifically to it. The Event Manager remains backwards compatible, but under the hood, it is using a static instance of the new Event Registry system.

Composer update

We’ve also improved the composer setup to properly use the PSR-0 standard. As such, the autoloader has been removed. If you were using Symbiosis prior to v1.2 and were using the autoloader directly, you will need to update to use the composer autoloader.

Conclusion

Symbiosis is now more flexable to implement. Our next steps will be to improve the plugin system to be more (and less) coupled to the event system.

See the full changeset: https://github.com/zumba/symbiosis/pull/15/files

Join the Zumba Engineering Team

2013-08-06T00:00:00+00:00

Our engineering team is in the process of adopting new technologies to solve complex business problems, and we’re looking for additional engineers to help us out.

Introduction

Zumba offers a unique opportunity of providing a startup feel with the security of an established company. Our business is growing, and as such our technology stack also needs to grow with it. This allows us to work on newer technologies (when it makes sense), so there’s always something to learn.

What to expect

The Zumba system operates on AWS linux based instances with a mix of MySQL and MongoDB driving our applications in a service oriented architecture. Our deployments work on a continous integration strategy as we use unit testing coupled with Jenkins CI server to deploy quality code to production.

Our projects are scheduled and structured with the scrum methodology in two-week sprints.

If you join our engineering team, you will be up and making your first commit within a few hours as your environment is managed by our vagrant setup.

Various positions offer unique opportunities to work on fascinating technologies:

Frontend

Use sass to style and build rich applications.
Work with backbone.js and require.js to make our applications more interactive.
Drive the user experience using modern web frameworks and mixins.

Positions:

Frontend Developer

Backend

Work with advanced PHP techniques as more of our application moves to PHP 5.4.
Solve difficult performance issues by utilizing different datastores, such as Memcached, MongoDB, and Elasticsearch.
Help us explore and improve our architecture with the JVM and Scala.
Continuously improve our continous integration strategy.
Work with other engineers that are excited about the technologies they bring to the table.

Positions:

Join Us

If working with exciting technologies and experienced engineers on solutions to complex problems is what you’ve been missing, then Join the team! We have several openings for software engineers and frontend developers.

Introducing Zlogd - An open source universal logging agent

2013-06-20T00:00:00+00:00

Logging is a critical part to an application’s life-cycle. It helps identify problems when they occur and track trends of events.

There are many ways and products to accomplish this depending on your goal, but today we’re introducing our open-source logging agent to help: Zlogd.

Use case

In many instances in web applications, multiple servers are involved, each involving one or many log files. When tracking down a problem or examining for trends, this becomes very problematic.

Zlogd is meant to solve this problem as being a universal logging agent. Zlogd runs locally on each server and listens to a socket file. The application then will start logging output to the socket file where Zlogd will package and ship the log message to a central logging location.

Zlogd Workflow

In our proposed stack configuration, our central logging server will contain the following:

Logstash agent for receiving all messages from all the Zlogd instances.
Elastic search for Logstash to store the messages
Kibana3 to have one interface to examine the logs

The workflow would then lend itself to this configuration:

Benefits

Since our stack is mostly PHP, that means each request is synchronous. If there are multiple loggers enabled (ie, write to file, ship to newrelic on warning/error, etc), the customer is on the hook for waiting for that log event to process through all those loggers. This shortens the logging time during the request by only having to write to a single logger (the Zlogd sock file) thereby increasing application performance.

This also facilitates the central logging paradigm, which allows the consolidation of all application logs.

Conclusion

Zlogd hopes to solve the problem of fractured logging and be flexible enough to work with any log collecting apparatus. This is a relatively new system, and as such we will be working the kinks out of this first version release. Feel free to fork and open a pull request to implement your flavor of log transport (graylog2, splunk, loggly, etc).

Reference

Middleman.js Project Open Sourced

2013-02-11T00:00:00+00:00

We’re on a roll this week, and it’s only Monday! Yesterday Chris Saylor introduced Mongounit. Today I am going to show you another open source project we’ve written called Middleman.js

Middleman is a javascript library that can be used in a browser or as a node.js module. It allows you to hook into the execution of any function to apply prefiltering to that function’s arguments; globally change the function’s execution context; pipe that function’s arguments to several others; or, overload the function to behave differently. It does all this for you seamlessly, so you can simply call the original function as you normally would.

Example

For a practical example we need to start with some assumptions: You have a new mobile project that communicates with a RESTful server API. After a few weeks of development some project requirements change and now every ajax url must start with /api.

You don’t want to manually find every ajax call in the project and prepend the url strings with /api.
You don’t want to force your team to remember to do this as you develop.
You want to implement this new requirement as seamlessly as possible.

jQuery’s powerful ajax methods are well known, but let’s say you’re working with a more simplified library like Zepto.js which was built for mobile. Middleman.js can solve this problem with just a few lines:

Seamless

You can call $.ajax as you normally would. The filter method will be executed first, and the array args that is returned by the filter method will be passed to the original $.ajax method as parameters.

Object references are passed by value in Javascript, which is why you don’t see me doing anything with the ajaxSettings variable after I’ve modified it’s url property. See this for more details.

Context

In the previous example, Middleman will use the lib object for context when executing the ajax method (in this case Zepto.js). However, Middleman can also take an optional context parameter that will be used instead, like this:

Notice that the context of the stringify method became Array. Had I passed context : Object, instead, the output would have been [object Array] instead of a,b,c,d. You’ve probably seen this kind of behavior in other popular libraries like Underscore.js’s _.bind method.

Final word

Currently there is no way to disable Middleman’s meddling once you’ve called the map function. We’re planning to incorporate an internal registery in a future release that will allow you to inspect the Middleman object to see what methods have been overloaded, and also make it easy for Middleman to flip it’s filters on or off at a granular level.

Repository - Middleman.js
License - MIT

Mongounit Project Open Sourced

2013-02-10T00:00:00+00:00

Introducing Mongounit.

Mongounit is a PHPUnit extension modeled after dbunit that allows for fixture-based unit testing of mongo db backed code.

One of our more recent projects has given the team exposure to MongoDB. As such, we needed an easy way to test the models that utilize mongo in a similar fashion to how we test models that talk to mysql. Using this framework, it’s easy to implement mongo test cases to easily create fixture data in collections, or simply clear collections between test cases.

See an example Testcase

More info

Repository - Mongounit
License - MIT
CI Build - Travis CI

Mocking Singleton PHP classes with PHPUnit

2012-11-26T00:00:00+00:00

In many of our projects, utilities and vendor classes are implemented with a singleton pattern. If you’re not familiar, a singleton pattern looks something like:

In this post, we’ll cover a nice way to inject a PHPUnit mock object for use in testing methods that utilize singleton classes.

Inception

First, we need to identify how this sort of mechanism is mocked. The key aspect of the singleton class is the protected static self attribute. This is what we’re most interested in for injecting our mock object. In our example singleton class, the self attribute is protected (which is usually the case), so we’ll need the use of the reflection class in order to “unprotect” and inject.

Mock Class

Now we will look at the code to do the above. First, a sample class that implements our singleton as part of a normal method:

In this exercise, we will mock Someclass’s hello method to make it return a different string. This is a super-simplified example, normally you would want to do this sort of thing for classes that do network based things, such as an emailer, REST service connector, etc.

Let’s create our mock class:

The expects method creates the mock object and injects that mock object into the self attribute. The method you test that utilizes the singleton will get the mock object when they call the getInstance method.

Example Use

Here is our PHPUnit test case. We instantiate the class we want to test, mock the singleton object, and alter the output of the method.

Conclusion

This technique is really nice for common utility classes. We typically use them in the setUp and tearDown methods, so we can test the other parts of the method that are important for their logic, and leave a single test to test the actual functionality of the singleton class we’re mocking.

Some CakePHP optimizations

2012-11-05T00:00:00+00:00

Our site and system has a lot of throughput and it make us use more instances and try to reduce the load in every part. It makes the company happy (save money) and also make the customer happy (faster load). On this article I will go over few things in terms of architecture and some code changes/strategies that could make your application faster as well.

Architecture

All our infra-structure is hosted by Amazon AWS. So, some tips are only for Amazon, but others can be reused by other hosting.

Install APC or any other opcode cache. It is kind of trivial, but just to make sure.
Move things out of the box. We moved all our assets to S3 and CloudFront. It helps in two ways:
reduce the load in host boxes (EC2), giving more CPU and memory for the PHP applications
Allow the browser to make more concurrent requests, since the host is usually different (ok, you can do the same with the same box).
Scale a little vertical before go horizontal. We had an experience of using many boxes of small size in Amazon EC2. But after some math and tests, we identified that getting few large instances is better than many small instances. The server handle more request simultaneously and it is faster, and cheaper. The cheaper is kind of weird to hear, because these boxes are expensive, but it is true. You can replace like 5 medium instances (m1.medium) by one high-memory extra large (m2.xlarge).
Avoid network requests. They could be fast, but you always have the overhead of the protocol, transmission, etc. I will give one example on the code portion.

Code

Store the cake caches locally. All our cache is based on Memcached. The internal cake caches (_cake_core_ and _cake_model_) wasn’t different. But these caches don’t change that often, so why use network for that? So we simply changed these 2 caches to use APC instead of Memcache. It reduced couple milliseconds on each request with a ridiculous change.
Do you change your files often in production? Do you have an automatized deployment process? Do you restart your webserver every deploy? If you answered yes for these questions, you may disable the configuration apc.stat on production. This configuration, when enabled, makes the APC check every request if the file you are trying to load was modified to re-create the opcode cache. If you don’t change the files in production, why re-check it? Disabling it make your code run mostly in memory only.
Check your APC configurations. Don’t let APC run out of memory enough to cache all your code.
Do you use cake query cache or cached view based on the model? If no, create a method on your AppModel called _clearCache and make it returns true. Why? By default Cake do some inflections and try loop thru the caches on the app/tmp/cache/ folder to delete the cached files when you make some database change operation (INSERT/UPDATE/DELETE). We did that and we got a huge improvement. One reason is because our sessions are stored in database, causing a insert/update in every single request.
View cache. You can use cake view cache or other system, like Varnish. Depending of your system cake view cache is enough and give more flexibility, since you can still add some PHP in the caches. Just be careful with restrict pages. Sometimes the cache will make it public for people with the URL. See more details on CakePHP Documentation.
requestAction and elements. Most of the sites has some dynamic content which need to render several times, for example the site menu. Usually the content comes from database and you have to add the request to the model into the AppController. It causes one or more query per request, which probably is unnecessary to do in all requests, since it will be the same for few minutes or even hours. The solution is use the requestAction inside the element and use the cache element. All built-in CakePHP and very simple to use. In the end, after the first request (which could be a little bit slower than usual because the requestAction) the code will get the raw html rendered from the cache, saving a good time and CPU from your database.

Conclusion

The changes above show few actions you could do to improve your site performance and save money with your hosting. There is much more tips to say, but I guess these are the most simple to implement in short time and take advantage of the results.

Some of the tips or concept are valid for non-CakePHP sites as well.

Schema.org Product Microdata and Breadcrumbs

2012-09-19T00:00:00+00:00

This week I was looking to improve our shop SEO and one of the things was the microdata for HTML5. It is really cool feature from HTML5, helping crawlers to understand more about the site and providing a better results for users. For example, Google shows rich snippets in their results, so you can see page author, breadcrumbs, review information, product prices, etc. You can see few examples on Google Rich Snippets Testing Tool.

Looking at Google Webmaster documentation, they recommend do use the microdata from schema.org, which is a schema made by search providers like Bind, Google, Yahoo! and Yandex. The schema contains a lot of types useful for you website. This works very well in many cases, but some times it requires layout changes to fit well.

Before I explain the problem, a little description of microdata in HTML5. Most of HTML tags can have the attributes itemprop, itemscope, itemtype. The itemprop define the property name and usually the content of the tag is the value, for example a name. itemscope define a block of properties, for example a product. itemtype contains the URL which contais the definition of the itemscope.

It means that if you want to describe a product, you create a some itemscope with, for example, a div and define all the properties there, like name, price, reviews, etc. It sounds great and worked for almost everything in the page, except one thing: breadcrumbs! The Product type do not have any property for categories or breadcrumbs, it is part from the WebPage type. But you can not define property for multiple scopes or in a different scope with microdata.

Unfortunately I didn’t find any solution with schema.org only, then I had to use schema.org and the deprecated data-vocabulary. The code in the end was something similar with that:

This is not the best solution, but it was the balance between do not change the layout because the microdata and have a nice presentation in search provider results. I hope the schema.org covers it once breadcrumbs are not restricted to WebPage type or any of the sub-types.

Do you have a better solution using microdata?

CakeFest 2012

2012-09-18T00:00:00+00:00

This post is a little bit late, but two of our team participated in CakeFest 2012. This was a really nice event, with friendly people, and great topics.

If you missed the event, you can check the slides on Joind.in. If you would like to see the pictures, Facebook.

Thanks Zumba^® for this opportunity. Thanks to everybody that helped to organize CakeFest and everybody we met there.

Our First Presentation - Javascript the important bits

2012-09-11T00:00:00+00:00

Part of being a web developer is having to work with multiple languages and multiple technologies. We are in the process of moving all our front-end javascript to backbone.js, and I thought it would be a good idea to have a refresher on javascript and jQuery to get the team in the right mindset during this transition.

Our first presentation has been put on our public slideshare account: Javascript: the Important Bits.

In the near future, we’re also doing something similar for PHP, including new 5.4 features, namespaces with autoload, and other similar topics.

Symbiosis Project Open Sourced

2012-08-21T00:00:00+00:00

I’m excited to announce that we have open-sourced our first project: Symbiosis. Symbiosis is a PHP library that allows for easy integration of an event driven plugin system to any PHP project. It also allows the event system to be used stand-alone from the plugin system.

Plug Me In

In a previous post, I described how we used a rudamentary event system to modularize our cart system. Now, we’ve taken a big step in making that sort of modularization easy to implement in many of our projects, and hopefully many of yours. There are many differences between our example in the previous post and our open-sourced version:

The event that gets passed to the listeners is now an object. Instead of just passing an array of data, we pass an event object. Not only is the data available, but several key flow control methods as well. Such as stopping propagation of the event, suggesting the client prevent further actions, as well as an easy way to trigger the event.
We’ve included the plugin system. Prior, we wrote off how to include the plugin, but with an event driven mechanism, I realized that this was an important step. The plugin manager included with Symbiosis includes a couple of key attributes. While developing with a rudamentary version of the plugin manager, I quickly realized that we needed to be able to control the order of some plugins. I’ve added a priority order to the plugin framework that allows you to set the order in which the plugins are executed. I’ve also added the ability to disable plugins from startup.

Example of Symbiosis in Action

With a few lines of code, you can see how easy it is to include a plugin system in your PHP project:

First we construct our plugin:

The plugin must extend \Zumba\Symbiosis\Framework\Plugin to implement all the proper methods and have default attributes.

Next, we register the plugins in the application bootstrap:

And finally, somewhere in our application, we need to trigger the event that the plugin is listening:

When the event is triggered, the anonymous function we registered in our sample plugin should execute:

Conclusion

We found this project useful in our own applications, and we hope that you may find it useful as well. We’ve included a way of installation either via a git submodule, download, or a composer install. The project has 100% code coverage in PHPUnit tests, and we feel comfortable with it’s stability.

You can see more examples and full documentation on the Symbiosis Wiki.

If you see anything that could be improved, let us know by opening an issue in the github project, or submit a patch.

Creating bash completion to your console application

2012-08-20T00:00:00+00:00

The number of console applications are growing more and more in web application environment. Many frameworks of many languages uses it and save a lot of time from developers and sysops to do some operations or even create cron jobs.

In our company it is not different. We use tons of jobs to do many operations and sometimes makes hard to remember the exactly name of the jobs. To simplify a little bit, we implemented support to bash completion.

Our jobs has a “starter” (common executable file, which load the configurations, basic stuffs, and run the target operation), similar the frameworks. In our case, we call /app_folder/job ClassName [methodName] and the job is just a shell script to execute php job.php $@. The job.php do the bootstrap part and loads the classname from the first parameter in a specific namespace. For example, our base namespace is Zumba\Job, if you pass job Engineering it will try to load the class Zumba\Job\Engineering. By default we call the method execute, but if you pass the second parameter we try to run that method instead of execute, making one job class reusable for multiple actions.

This weekend I saw the bash completion for CakePHP from Andy Dawson and had an idea to do the same for our service application, because we frequently forget the exactly job class or method name and add extra steps to verify these names before execute the job. I read his code, made some research and finally get our bash completion working fine.

First thing you need to do is make your application returns the available options for the first and second parameter. It means you will be able to get the available options from console by executing some command. PS: If your application do not change often or you don’t want or can update your app, you can have the parameters hard coded in the bash completion script.

In our case, I could add a job class to get that informaiton, but I prefered to use a symbolic method and parse with an if in the job.php file. I used that way to avoid create a job class for this proposal and be one more in the list and share the structure from the others methods, which is a little complicated (we have some logs, register execution, etc.). But few free to do the way you want. In the end, you can run my script like job __check__ to get the available classes or job __check__ SpecificJobClass to get the available methods.

The PHP portion is basically that:

With that, from the console we are able to execute the check (job __check__) and the list of available jobs: Job1 MyCustom MyApp TopSecret StopTheCompany

This was the first step from the script. The second part is get a list of the available methods, ie when execute the job __check__ StopTheCompany we get something like: shutdownServers dropDatabase removeAppCode

Now is time to make the bash interact with your code. You need to create a script in your /etc/bash_completion.d/ folder, with permission 0644. It will be like the code below.

To explain a bit, the last line will tell bash to complete the executable job using the function _my_application (defined few lines above). The function basically test the number of parameters and call your application with the current parameters and transmit to the bash completion. In our case, we have just two parameters, as you can see.

After save the file in the /etc/bash_completion.d restart your bash/console/terminal and you can try to use the completion. I guess I don’t need to put some examples how to use or how it looks like. If you never used bash completion it is not the place to get started.

Creating a testing interface for your API

2012-08-16T00:00:00+00:00

Couple months ago we started an integration with Ooyala, using JSON API. They provide the documentation to the methods and I saw they have a simple interface to test their methods: Ooyala API Scratchpad. This interface was very useful while we integrate with them and I thought: “Why we don’t have one interface like that for our API?”

I started a page with Twitter Bootstrap to have a similar functionality, which the goal was to get an interface easy to developers see the response for multiple HTTP protocols, set the parameters, etc. The initial page looks like this:

For the request, I used jQuery AJAX. It means that some HTTP methods (usually PUT and DELETE) are not supported by old browsers, but we don’t care as it is an internal tool and everybody uses Chrome or Firefox. Doing the request you get a JSON response formatted, like that:

Developers liked this page, but it wasn’t good enough to me. Then I get the response from Creating Self Documentation to your API (also thru AJAX) and made an autocomplete integrated with the parameters generation. Resulting something like that:

This exemplifies how the annotations are important. We get all the API methods, method descriptions, method parameters, HTTP method, etc. from the annotation and generate nice pages like that.

The productivity of using this page was greatly improved with not having to toggle between the tester and the documentation. As it is an internal tool, why not complement adding some request log in this tool to show possible errors or behaviors?! I got some help from ChromePHP (that put log messages on the response header), integrated with our log system and then parsed it from the response, showing in the page as well.

After have it in our API, we started this blog and looking the Disqus API I saw they have a similar tool called Disqus Console. Again, it helped me with some integration. If you have an API, make this type of tool for you client. He definitely will like it.

Conclusion

I spent one day to create this tool and people are saving time every day. It means sometimes you should take a break and spend some time doing simple tools to gain in long term.

You can see what annotations, some open source projects and few lines of code are able to do. Use it! As I told in the annotations post, it is a simple thing with many benefits.

Do you want the code of that page? Sure, why not?! https://gist.github.com/3365894.

Tech Team at Zumba Convention 2012

2012-08-13T00:00:00+00:00

This weekend happened the Zumba® Instructor Convention 2012, a meeting between thousands of ZIN™ members. It happens every year and all the home office is invited to attend and participate. It is an odd experience that we hadn’t in any other company and it’s a great opportunity to get feedback from our users and not only from our boss, that sometimes don’t reflect the reality.

As the convention happened during business days, we closed the IT department and went there to have some fun, exercise (even if it’s just walking around, because as you know mostly IT people are not dancers, further between professionals), party, interact and for some craziness.

This kind of opportunity is nice to see that you helped to make that happen, people are happy and having fun together because you made your job. It is very gratifying and, my friend, no money can pay it. Back in the office you can see the results too, just look in the face and behavior from people, they look more friendly and relaxed, what is improving the productivity.

Below you can see a picture with few people from our department in the convention. Yes, we are with this sleepy-face because we were in the Daddy Yankee show in the previous night.

Left to right: Daniel Matsinger, Girish Raut, Cristiano da Silva, Chris Saylor, Ralph Valdes, Juan Basso

If you think it is just one more convention, look how is that convention…

Frisby.js or: How I Learned to Stop Worrying and Love the Service Test

2012-08-11T00:00:00+00:00

Writing a public facing API can be a daunting task, and making sure your endpoints behave how your customers and partners expect is always a challenge. In this article, I will go over how to use NodeJS, Frisby.js, and Jasmine-node to test your API, and how to involve your customers and partners in the process.

Behavior Driven Development

Frisby.js and Jasmine are very conducive to the idea of BDD. When creating an API (especially in conjunction with a partner), BDD is a fantastic way to drive coordination between ~~business requirements~~ user stories and code. Frisby.js gives a nice set of methods to turn a user story into a functioning test case.

Writing Frisby.js Test Cases

Suppose you have a user story about getting a user’s profile by ID and returning some basic information. The test you would write would be similar to this:

To execute the above test, execute the following from the command line:

jasmine-node --verbose

Integrating with Partners and Customers

Communicating problems, changes, and features of an API to a partner or customer can be problematic. Our approach was to separate this kind of testing framework for the API from our internal API repository. This way, we can share the API test repository, allow the tests to show some example uses of the API, and provide an avenue for partners to report an issue.

With the use of Github, a partner can fork our testing repo at any time, write a test case for the issue they are having, and submit a pull request. We can then verify by reproducing the issue from their pull request test case and truely know that we’ve resolved the partner’s issue when their own test case passes with the fix.

Caveats

There is a small drawback to service-level testing (as opposed to unit testing). Since you’ll be making actual requests to your API, you don’t have the luxury of fixtures creating clean databases after the test case is run. So, depending on the data changed in the request, you’d have to chain a cleanup method after the test case is run.

Also, as opposed to unit tests, the failed test will not pinpoint in the code where the error happens, so you’d need to rely on other tools, such as logging, to find the issue in the source.

Conclusion

With a BDD approach and the use of some clever testing tools, it’s easier than ever to test an externally available service for full coverage and to provide a better channel of communication with partners to resolve bugs and implement features.

Creating Self Documentation to your API

2012-08-05T00:00:00+00:00

When we started our internal web-service we had a problem between the team that works in the web-service application and the front end developers because the front end developers didn’t know exactly what was the expected parameters, endpoints, HTTP verb, etc. This increased the delay in delivery features and in some cases caused a few bugs.

Looking at the problem, Chris Saylor had a great idea to use the controller annotation and generate a page with that. He started with some proof of concept, which was very well accepted by the front end developers and than Chris and I improved that page content and layout.

As part of the page content improvement, we started to use our own annotations, similar many projects do, ie: - PHPDoc - PHPUnit - Doctrine - Symfony

We created tags like @inputParam, @optionalInputParam, @returnParam, etc. It helped us to create a more useful documentation page, with more relevant information for front end developers instead of a regular PHP documentation that doesn’t help front end developers at all.

In our web-service we created a method that reponse HTML or JSON, depending of Accept header. If the browser requests the site, it accepts text/html and we provide a HTML page using Twitter Bootstrap with jQuery and make an AJAX request to the same URL, getting the JSON version (that is usually what our web-service response) and generating the HTML with all URI and parameters required for that.

Below is an example of our JSON response:

With jQuery we loop thru all the controllers and actions and generate the page. Also we have a search with jQuery to filter results, making the life much easier to front end developers.

Before you ask why we separate the HTML in one file and the content in an AJAX, the answer is simple. Our web-service is designed to performance and always response in JSON (yes, we can support different outputs, but it is our internal web-service and we don’t want to add more complexity). The HTML portion is a static file, not a PHP file. Another reason is make this method reusable, like we have for our web test interface (I will describe in another post).

Using Application Events to Hook in Plugins

2012-08-04T00:00:00+00:00

In many instances, having a plugin system (even for closed-source applications) is a convenient and safe approach to adding functionality to a product. It minimizes risk by not having to modify the core of the source. In this article, I’ll be discussing how we implemented a plugin system for our cart software to allow for plugins.

Benefits of Plugins versus Adding the functionality in Place

It prevents gargantuan classes to handle every situation that your software might need to handle.
Allows for easier testing of the individual plugins.
Provides various ways of including the plugins (including ways for filtering in specific situations).

Event Horizon

Application events come in many shapes and sizes. We chose to utilize a static class for “registering” the event to a callback method.

This adds a callable entry to the events array per event “tag”. Notice that $events can be an array, so assigning multiple events to the same callback is very easy to do. Now all we need is a way to trigger the events we register to this class at runtime.

When triggering an event, we simply pass an array of data to the callback that was registered. So all our callbacks should have a signature similar to: public function someCallback($data = array()).

Now we have all the tools we need in order to start registering and triggering events.

Many frameworks have started to provide this kind of application runtime events (which inspired this kind of system), such as CakePHP 2.x, as well as ORMs such as Doctrine2.

Plug and Play

Now that we have our event system in place, it’s time to use it for our plugins. There are many ways to include a “plugin” structure that I won’t cover here. What I will cover is once you have your plugin included, how to use our event system to call it.

We have several uses for plugins for our cart system. Chief among them is refreshing the status of the items in the cart on certain actions, such as when adding an item to the cart. When structuring our plugin using an event system, we just need to expose a registration method that registers its own callbacks to all the events that it should listen for and react to when operations occur on the cart. For example:

Conclusion

With this type of setup, when we need additional things to happen on certain actions, we don’t have to go back and modify the core part of the app and risk introducing regression errors. We simply write a plugin, register the event, and can concentrate the unit tests on the plugin callbacks themselves.

CakePHP and Code Coverage

2012-08-03T00:00:00+00:00

Recently we started a new API application in CakePHP 2.2 and put some code and unit tests. When we put the code in Jenkins CI to a continuous integration we setup the code coverage and in the reports we got couple CakePHP core files as not covered.

Our application doesn’t care about cover Cake files, and we also don’t want to include Cake core tests in our CI. Leaving these files in CI give reports that our code is not well covered, which it is not true and hide the uncovered code from our app.

As CakePHP, we also use PHPUnit in other project, but there we use the phpunit.xml configuration file. Using the configuration file is easy to put folders/files on black and white lists, but it is not possible when using CakePHP because it has their own runner system. Our solution was figure out a way using code. We changed our test suite to do it, see below.

With the code above you can run in your console: ./Console/cake test app AllApp --coverage-html=./report

You can replace the --coverage-html by --coverage-clover to use in Jenkins (or both, like in our case).

Kicking Off the Blog

2012-07-21T00:00:00+00:00

Zumba Fitness^®’s techology department will begin blogging about the technology that we use, the problems that we encounter, and the cool things that’s in development.

We’ll also use this as a platform to announce projects that our team has decided to open-source. We look forward to feedback (and hopefully some pull-requests) from the community.

Zumba Tech

Zumbatech takes on #hackforchange

Let’s not re-invent the wheel

Hackathon

Presentation and Reception

Conclusion

W3C Push API Crash Course

The ServiceWorker

ServiceWorker Registration

Google Application Setup

manifest.json

ServiceWorker Life-Cycle

Scope

Registering

The PushManager

Putting It Together

PushSubscription

Using the PushSubscription

Receiving Notifications with ServiceWorker

The Bad News

Other Helpful Resources:

Relational Database Design

Overview - The Basics

Where to begin

Naming Conventions

Tables

Columns

Constraints

Defaults Conventions

Standard Tables

Standard Columns

Column Types

Basic Columns

Date and Time Columns

Joining records

Warehousing

ETL

Why Is This Relevant

Conclusion

How We Do CSS at Zumba

Intro - Browser and Device Support

Preprocessor

Folder/Sandwich Structure

+ Modularity and Reusability

+ CSS Rules

+ Light stylesheets

Plate

Toppings

Sandwich

Example of our overall structure:

main.scss - Consists of the basics of our pages’ layout.

CSS structure

Naming

Variables and Mixins

Mixin

Color Variables

Classes

Media Queries

Typography

Linting

The fun never ends

Zumba® Technology Day

Caching CakePHP 2.x routes

Developing With AngularJS? Forget jQuery Exists.

Separation of Concerns

Manually Querying the DOM Is Like Writing SQL

Indigestion

Refactoring a Problematic Directive

The Template

The Brittle Directive

Use the Scope

The New and Improved Template & Directive

jQuery Under the Hood

Summary

TL;DR

Enforce code standards with composer, git hooks, and phpcs

There Is No Catch

Phpcs Dependency

Install Scripts

Setup the Git Pre-commit Hook

Zumba^® Technology Day

What about `JsonSerializable` in PHP 5.4?

What about using `json_encode` with the `Serializable` interface?

Is the library compatible with `Serializable` or `JsonSerializable`?

Stale Pull Requests `==` Death to Open Source