It is helpful for developers to have  tools that can analyze different codebases quickly. A rather comprehensive list of source code analyzer tools for each platform is listed here: (http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis) .  Since an in-depth comparison of the multitude of analyzer tools is beyond the scope of this article, all figures and analysis were done by using Understand by Scitools, a typical albeit premium source code analysis tool.

Understand by SciTools

Understand by SciTools has the ability to scan Ada, Cobol, C/C++, C#, Fortran, Objective-C, Java, Jovial, Pascal, PL/M, Python and others. Like many multi-language analyzer programs it is not free, however, the benefits of such a program are enormous. (For the purposes of this demonstration, a deprecated and unused codebase was analyzed.)

After source code files have been parsed, you’ll see multi-windowed view like the following:

Figure 1. Parsing of project sources

Figure 2. IDE source windows.

Pros of Source Code Analyzer tool

The ability to see how a variable/method in the project is analyzed, how it is used (or unused) in the project, what methods call it or what methods are called by it are a quick way to get reference information. This can also be done quite easily in various IDEs of the respective languages – Xcode, Eclipse, IntelliJ, etc. What sets premium multi-language source code analysis tools apart from IDEs is the ability to see graphically how the source code is structured and to be able to run metrics on them as a whole.  In Figure 3, for example, we notice that the androidTestApp has 113 references to the androidAdAgent library. The flurryAdTestApp has about 25 dependencies on such project. Further analysis of this project reveals that the flurryAdTestApp is a generic sample project for testing ad functionality while the androidTestApp is a more universal testing application. There are many more benefits to know this internal dependencies – for example, knowing how complex a dependency exists makes it easier to understand how much QA is required if such code is refactored.

Figure 3. Internal architecture complexity

Figure 4. UML class diagram

Figure 4 shows the overall UML class structure of the project. This is particularly useful if you need to refactor or re-engineer your code base to specific design patterns. Some of these analyzer tools allow you to directly manipulate the uml diagrams and the underlying code structure is changed as a result.

Figure 5. Unused variables and parameters

The ability to see unused variables and parameters is particularly useful in reducing code bloat and keeping the codebase lean.

Figure 6. Check code for various coding standards.

Quite a few of these premium analyzer tools have code check algorithms that allow you to be notified of overly complex algorithms (big-O, number of lines, cyclomatic dependencies, and unused code paths). Overly complex programs are difficult to comprehend and have many possible paths making them difficult to test and validate. Most analyzer tools allow you to record/program your own macros for determining proper code validation:

- Improper use of .equals() and .hashCode()

- Unsafe casts

- When something will always be null

- Possible StackOverflows

- Possible ignored exceptions

- Cyclomatic Compexity (modified, strict)

Cons of Source Code Analyzer tool

One drawback of using such source code analysis tools is that you have to configure the project to find all relevant sources for each project type – Failure to have a project properly configured could result in too much or too little detail that’s useful. Source code that is significantly modular across libraries would be difficult to analyze.  Also, many tools that are out there are quite one-dimensional in that they may check coding style but may not be able to provide detailed analysis on code complexity and improper algorithmic complexity. One open source tool that shows promise is sonar (http://www.sonarsource.org) but it’s big drawback is that it requires a web server and a database. Other possible issues are that some code analyzers analyze the bytecode while others analyze the source. Whatever source analyzer tool is chosen it may not be comprehensive enough for the organizational needs.

At Flurry, we have multiple sdk codebases – Objective-C, java (android), BlackBerry10, windows phone 8/7, windows 8, HTML5. Each additional analyzer tool needs a significant learning curve. We try to keep the tools to a minimum while still trying to get coverage on the different codebases.

Source Code Analysis As Part of the Development Environment

There are many programs that analyze source code but only few provide support for multiple languages and are open-source. The most popular ones for java are PMD, FindBugs and Checkstyle.  However a tool that is simple, multi-language and open-source would be ideal.

The ability to easily see the UML structure of a program, understand it’s code complexity, and see all the dependencies with the click of a button can easily replace invalid comments and outdated documentation. A good source analyzer tool is only one part of a toolbox that should be available to the developer. Unit tests, code reviews, and pair programming should always have priority but a source code tool can definitely help the developer (both new and experienced) to keep track of large codebase that he may be working on.

Author: Richard Brett

Synopsis

Our process is designed to help us discover as much as we can about each candidate as efficiently as possible. We begin with a phone screen, which is a low effort, low cost handshake between us and the candidate. This gives us a basic sense of their programming knowledge and their compatibility with us. Then, we follow up with a code test that evaluates their ability to design and implement a solution to the proposed challenge. Finally, we conduct two onsite interviews, one of which covers the design decisions made in their code test and their technical abilities, and another which covers their creative problem solving skills and delves into their past experiences.

During the two interviews, we assess the following:

• Communication skills

• Problem solving skills

• Design ability

• Team fit

  
  

  
  

Giving Hints

Most candidates will get stuck solving a problem at least once during the interviews. This is our opportunity to direct the conversation with a hint. A good hint can sometimes be very difficult to give. If an interviewee has chosen to answer a problem solving question in a way that is unusual but not necessarily incorrect, and then gets stuck, it is up to us to think ahead and supply a hint that helps them get to where they want to go. This can be especially challenging if the interviewee knows more about the subject than we do. The alternative is to give a hint that sets the candidate onto a path that leads to a known good solution, but this has several drawbacks. First, it wastes time because the interviewee must start again. Second, the candidate often still has their original idea in mind, which distracts them. Finally, pointing someone in a completely different direction is jarring and throws them off, especially if they start analyzing whether they have jeopardized their chances at an offer by getting the question wrong. Ideally, a good hint will provoke a thoughtful discussion about how to arrive at a solution. This also introduces a collaborative element to the interview and gives the interviewee a chance to teach us something. By digging in like this, we gain a wealth of information about their ability to communicate and how they approach problems.

Reviewing the Code Test

The code test tells us several things about the candidate:

• How they write code in a natural setting

• How they structure blocks of code – is it organized into logical units of work?

• How they test their code – are their tests concise and relevant? What do the tests cover? Is their code designed to be testable?

• How they design their algorithms – what approach does the candidate take?

Reviewing the test with the candidate mimics an actual code review. We discuss the compromises they made, such as performance, readability, and testability. We also explore the design decisions the candidate made to gain insight into how they prioritize their development practices. This also gives us a sense of how they respond to feedback: do they give justifications for their design decisions? Do they readily acknowledge mistakes?

Author: Jon Miller

One of the best ways of connecting with mobile application customers is via remote push notifications. This is available on both Android (GCM) and iOS (APNS). These services allow developers to send messages directly to their users and this is an extremely valuable tool to announce updates, send personalized messages and engage directly with the audience. Google and Apple provide services that developers can send push messages to and they in turn deliver those messages to their users.

The Problem

It’s not unusual for apps these days to have in the order of millions and even tens of millions of users. Testing a Push Notification backend can be extremely hard. Sure, you can set up a few test devices to receive messages but how do you know how long it would take your backend to send out a large number of push messages to the Google and Apple servers? Also, you don’t want to risk being throttled or completely blacklisted by either of those services by sending a ton of test data their way.

The Solution

The solution is to write a mock server that’s able to simulate the Google/Apple Push Notification Service, and a test client to hit it with requests.

The Google service is completely REST based, so a script that executes a lot of curls in a loop can do that job. Also, it’s fairly straightforward to write a simple HTTP server and accepts POSTs and sends back either a 200 or some sort of error code.

Apple’s APNS, however, presents a few challenges. It’s a binary format listed here. Since the protocol is binary, you need to write some sort of mock client that can generate messages in the specified format. At Flurry, we’ve been playing around with Node.js to build scalable services and it’s fairly straightforward to setup an Apple APNS test client and server.

The Client

https://gist.github.com/rahuloak/4949310

The client.connect() method connects to the mock server and generates test data. The Buffer object in Node is used to pack the data into a binary format to send it over the wire. Although the protocol lets you specify a token size, the token size has been set to 64 bytes in the client since that’s typically the token length that gets generated. Also, in our experience, the APNS server actually rejects tokens that aren’t exactly 64 bytes long. The generateToken() method generates 64 byte hex tokens randomly. The payload is simple and static in this example. The createBuffer method can generate data in both the simple and enhanced format.
 
What good is a client without a server, you ask? So without further ado, here’s the mock server to go with the test client.

The Server

https://gist.github.com/rahuloak/4949381

After accepting a request, the server buffers everything into an array and then reads the buffers one by one. APNS has an error protocol, but this server only sends a 0 on success and a 1 otherwise. Quick caveat: Since the server stores data in a variable until it gets a FIN from the client (on ‘end’) and only then does it process the data, the {allowHalfOpen:true} option is required on createServer so that the client does not automatically close the connection.

This setup is fairly basic, but it is useful for many reasons. Firstly, the client could be used to generate fake tokens and send them to any server that would accept them (just don’t do it to the APNS server, even in sandbox mode). The data in the payload in the above example is static, but playing around with the size of the data as well as the number of blocks sent per request helps identify the optimal size of data that you would want to send over the wire. At the moment, the server does nothing with the data, but saving it to some database or simply adding a sleep in the server would be a good indicator of estimated time to send a potentially large number of push messages. There are a number of variables that could be changed to try and estimate the performance of the system and set a benchmark of how long it would take to send a large batch of messages.

Happy testing!

4th grade budding girl geek in the making 2nd row 2nd girl from the left

TechWomen brings emerging women leaders in Science, Technology, Engineering and Mathematics  from the Middle East and Africa together with their counterparts in the United States for a professional mentorship and exchange program. TechWomen connects and supports the next generation of women entrepreneurs in these fields by providing them access and opportunity to advance their careers and pursue their dreams.

http://www.techwomen.org/

When cabling, here are some practices Flurry lives by:

Label everything

There has never been a situation where you’ve heard someone say, “I wish I hadn’t labeled this.” Labeling just makes sense. Spend the extra time to label both ends of the network and power cables. Your sanity will thank you. If you’re really prepared, print out the labels on a sheet ahead of time so they’ll be ready to use.

Cable length

When selecting cable length, there are two schools of thought. There are those who want exact lengths and those who prefer a little extra slack. The majority of messy cabling jobs are from selecting improper cable lengths so use shorter cables where possible. A good option is custom made cables. You get the length that you need without any excess. This option is usually expensive in either time or money. The other option is to purchase standard length cables. Assuming that you have a 42U rack, the furthest distance between two network ports is a little over six feet. In our rack build outs, we’ve had great results using standard five foot network cables for our server to switch connections. 

Cable management arms

When purchasing servers, some manufacturers provide a cable management arm with your purchase. They allow you to pull out your server without unplugging any cables. For this added benefit, they provide bulk, retain heat, and reduce proper airflow. If you have them, we suggest that you don’t use them. Under most circumstances, you would unplug all cables before you pull out a server anyway.

No sharp bends

Cables do require a bit of care when being handled. A cable’s integrity can suffer with any sharp bends so try to avoid this. In the past, we have seen port speed negotiation and intermittent network issues cause by damaged network cables.

Use mount points

As you group cables together, utilize anchor points inside of the rack to minimize stress on cable ends. Prolonged stress on the cable ends can cause the cable and socket it’s connected in to break. Power ends are also known to unplug. The weight of the bundled power cables can gradually unplug it at any moment. Using anchor points will help alleviate directed stress to the outlet.

Less sharing

Isolate different types of cables (power, network, kvm, etc) into different runs. Separating cable types will allow for easy access and changes. Bundled power cables can cause electromagnetic interference on surrounding cables so it would be wise to separate power from network cables. If you must keep copper network and power cables close together, try to keep them at right angles. Standing at the back of the rack, network cables are positioned on the left hand side of the rack while power cables are generally on the right in our setup.

Lots and lots of velcro

We’ve seen the benefits of velcro cable ties very early on. It’s got a lot of favorable qualities that plastic zip ties do not. They’re easy to add/remove and also retie. They’re also great when mounting bundled cables into anchor points inside of the racks. If your velcro ties come with a slotted end, do not give into the urge to thread the velcro into the ends. It’s annoying to unwrap and rethread. Don’t be shy to cut the velcro to length, either; using just the right length of velcro can make it easier to bundle and re-bundle cables. 

Now that you have these tips in mind, let’s get started on cabling a Flurry rack.

1. Facing the back of a 42U rack, add a 48 port switch in about the middle of the rack (position 21U (21st from the bottom). Once you have all your servers racked, now the fun part being, cabling. Let’s start with the network.

 2. From the top most server, connect the network cable to the top left port of your switch, which should be port 1.

3. As you go down the rack, connect the network cables on the top row of ports from left to right on the switch (usually odd numbered ports). Stop when you’ve reached the switch.

4. Using the velcro cable ties, gather together the cables in a group of ten and bundle the cabled groups with the cable ties. Keep the bundle on the left hand side of the rack. You will have one group of ten and one group of eleven that form into one bundled cable.

5. For the bottom set of servers, start with the lowest server (rack position 1U) and connect the network cable to the bottom left most port on the switch.

6. Starting from the bottom up, connect the network cables on the bottom row of ports from left to right on the switch (usually even numbered ports).

7. Doing the same as the top half of the rack, gather together the cables in a group of ten and bundle the cabled groups with the cable ties. Keep these bundles on the left hand side of the rack. You’ll end up with two bundles of ten that form into one bundled cable. Look pretty decent?

8. Now, lets get to power cabling. In this scenario, we will have three power distribution units (pdus), one on the left and two on the right side of the rack. Starting from the top of the rack, velcro together five power cables and plug them into one of the pdu strips on the left side of the rack from the top down.

9. Take another two sets of four bundled power cables and plug them into the other pdu strips on the right hand side also following the top to bottom convention. You should end up with a balanced distribution of power plugs.

10. Take a bundle of six power cables and plug them into the pdu strip on the left hand side.

11. Take another two sets of four power cables and plug them into the two pdu strips on the right hand side.

12. Start from the bottom up, bundle the power cables in groups of five. You will end up with two sets of five power cables and a bundle of four.

13. Plug the bundle of four power cables into the pdu on the left hand side.

At this point, you can take a step back and admire your work. Hopefully, it looks sort of like this:

Good cabling can be an art form. As in any artistic endeavor, it takes a lot of time, patience, skill, and some imagination. There is no one size fits all solution, but hopefully this post will provide you with some great ideas on your next rack build out.

Before your custom filter can be used, it will have to compiled, packaged in a jar, and deployed to all the regionservers. Restarting the HBase cluster is necessary for the regionservers to pick up the code in their classpaths. Therein lies the problem – an HBase restart takes a non-trivial amount of time (although rolling restarts mitigate that somewhat) and the downtime is significant with a cluster as big as Flurry’s.

This is where dynamic filters come in. The word ‘dynamic’ refers to the on-demand loading of these custom filters, just like loading external modules at runtime in a typical application server or web server. In this post, we will explore an approach that makes this possible in the cluster.

How It Works Now
Before we dive into the workings of dynamically loading filters, let’s see how regular custom filters work.

Assuming the custom filter has already been deployed to a jar in the regionservers’ classpath, the client can simply use the filter, e.g. in a full table scan, like this

https://gist.github.com/4227003

This filter will have to be pushed to the regionservers to be run server-side. The sequence of how the custom filter gets replicated on the regionservers is as follows:

1. The client serializes the Filter class name and its state into a byte stream by calling the CustomFilter’s write(DataOutput) method.
2. The client directs the byte array to the regionservers that will be part of the scan.
3. Server-side, the RegionScanner re-creates the Scan, including the filter, using the byte stream. Part of the stream is the filter’s class name, and the default classloader uses this fully qualified name to load the class using Class.forName().
4. The filter is then instantiated using its empty constructor and configured using the rest of the filter byte array (using the filter’s readFields(DataInput) method (see org.apache.hadoop.hbase.client.Scan for details).
5. The filter is then applied as a predicate on each row.

(1) myfilter.jar containing our custom filter resides locally in the regionservers’ classpath
(2) the custom filter is instantiated using the default Classloader

Once deployed, this definition of our custom filter is static. We can make an ad hoc query using the combination of filters, but if we need to add, extend or replace a custom filter, it has to be added to the regionserver’s classpath and we have to wait for its next restart before those filters can be used.

There is a faster way.

Dynamic Loading
A key takeaway from the previous section is that Filters are Writables – they are instantiated using the name of the class and then configured by a stream of bytes that the Filter understands. This makes the filter configuration highly customizable and we can use this flexibility to our advantage.

Rather than create a regular Filter, we introduce a ProxyFilter which acts as the extension point through which we can load our custom Filters on demand. During runtime, it will load the custom class filter itself.

Let’s look at some example code. To start with, there is just a small change we have to make on the client; the ProxyFilter now wraps the Filter or FilterList we want to use in our scan.

https://gist.github.com/4227336

The ProxyFilter passes its own class name to be instantiated on the server side, and serializes the custom filter after.

https://gist.github.com/4227434

On the regionserver the ProxyFilter is initialized in the same way as described in the previous section. The byte stream that follows should minimally contain the filter name and its configuration byte array. In the ProxyFilter’s readFields method, the relevant code looks like this.

https://gist.github.com/4227524

This is very much like how the default Scan re-creates the Filter on the regionserver with one critical difference – it uses a filterModule object to obtain the Class definition of the custom filter. This module retrieves the custom filter Class and returns it to ProxyFilter for instantiation and configuration.

There can be different strategies for retrieving the custom filter class. Our example code copies the jar file from the Hadoop filesystem to a local directory and delegates the loading of the Filter classes from this jar to a custom classloader [3].

To configure the location of the directory the module searches for the filters.jar in HDFS, add the following property in hbase-site.xml.

<property>
<name>flurry.hdfs.filter.jar</name>
<value>/flurry/filters.jar</value>
</property>
(1) The custom filter jar resides in a predefined directory in HDFS
(2) The proxyfilter.jar containing the extension point needs to reside locally in the regionserver’s classpath
(3) The ProxyFilter is instantiated using the default ClassLoader
(4) If necessary, the rowfilter.jar is downloaded from a preset Path in HDFS. A custom classloader in ProxyFilter proceeds to instantiate the correct filter. Filter interface calls are then delegated to the enclosed filter.

With the ProxyFilter in place, it is now simply a matter of placing or replacing the jar in the Hadoop FS directory to pick up the latest filters.

Reloading Filters
When a new Scan is requested on the server side, this module first checks up on the filter.jar. If this jar is unchanged, the previously loaded Classes are returned. However, if the jar has been updated, the module repeats the process of downloading it from HDFS, creating a new instance of the classloader and reloading the classes from this modified jar. The previous classloader is dereferenced and left to be garbage collected. Restarting the HBase cluster is not required.

The HdfsJarModule keeps track of the latest custom filter definitions using a separate classloader for the different jar versions

Custom classloading and reloading can be a class-linking, ClassCastException minefield, but the risk here is mitigated by the highly specialized use case of Filtering. The filter is instantiated and configured per scan and its object lifecycle limited to the time it takes to do the scan in the regionserver. The example uses the child-first classloader mentioned in a previous post on ClassLoaders that searches for a configured set of URLs before delegating to its parent classloader [2].

Things to watch out for

• The example code has a performance overhead as it makes additional calls to HDFS to check for the modification time of the filter jar when a filter is first instantiated. This may be a significant factor for smaller scans. If so, the logic can be changed to check the jar less frequently.
• The code is also very naïve at this point. Updating the filter.jar in the Hadoop FS while a table scan is happening can have undesired results if the updated filters are not backward compatible. Different versions of the jar can be picked up by the RegionServers for the same scan as they check and instantiate the Filters at different times.
• Mutable static variables are discouraged in the custom Filter because they will be reinitialized when the class is reloaded dynamically.

Extensions
The example code is just a starting point for more interesting functionality tailored to different use cases. Scans using filters can also be used in MapReduce jobs and coprocessers. A short list of possible ways to extend the code:

• The most obvious weakness in the example implementation is the ProxyFilter only supports one jar. Extending that to include all jars in a filter directory will be a good start. [4]
• Different clients may expect certain versions of Filters. Some versioning and bookkeeping logic will be necessary to ensure that the ProxyFilter can serve up the correct filter to each client.
• Generalize the solution to include MapReduce scenarios that use HBase as the input source. The module can load the custom filters at the start of each map task from the MR job library instead, unloading the filters after the task ends.
• Support other JVM languages for filtering. We have tried serializing Groovy 1.6 scripts as Filters but performance was several times slower.

Using the Proxyfilter as a generic extension point for custom filters allows us to improve our performance without the hit of restarting our entire HBase cluster.

Footnotes
[1] Class Reloading Basics http://tutorials.jenkov.com/java-reflection/dynamic-class-loading-reloading.html
[2] See our blog post on ClassLoaders for alternate classloader delegation http://tech.flurry.com/using-the-java-classloader-to-write-efficient
[3] The classloader in our example resembles the one described in this ticket https://issues.apache.org/jira/browse/HBASE-1936
[4] A new classloader has been just been introduced in hbase-0.92.2 for coprocessors, and it seems to fit perfectly for our dynamic filters https://issues.apache.org/jira/browse/HBASE-6308
[5] Example code https://github.com/vallancelee/hbase-filter/tree/master/customFilters

Thankfully, Android provides some tools to make this easier such as the AsyncTask utility class and the StrictMode API. You can also use good software development practices such as adhering to strict code style and requiring careful code review of code that involve the UI thread. Unfortunately, these approaches require diligence, are prone to human error, or only catch errors at runtime.

CheckThread for Android

CheckThread is an open source project authored by Joe Conti that provides annotations and a simple static analysis tool for verifying certain contracts in multithreaded programs. It’s not a brand new project and it’s not very difficult to use, but it hasn’t had a very high adoption for Android apps. It offers an automated alternative to exclusively using comments and code review to ensuring no bugs related to the UI thread are introduced in your code. The annotations provided by CheckThread are: @ThreadSafe, @NotThreadSafe, @ThreadConfined

ThreadSafe and NotThreadSafe are described in Java Concurrency in Practice, and CheckThread enforces the same semantics that book defines. For the purposes of this blog post, the only annotation that we’ll be using is ThreadConfined.

Thread confinement is a general property of restricting data or code to access from only a single thread. A data structure confined to the stack is inherently thread confined. A method that is only ever called by a single thread is also thread confined. In Android, updates to the UI must be confined to the UI thread. In very concrete terms, this implies that any method that mutates the state of a View should only be called from the UI thread. If this policy is violated, the Android framework may throw a RuntimeException, but also may simply produce undefined behavior, depending on the specific nature of the update to the UI.

CheckThread supports defining thread policies in XML files, so while it would be possible, it’s not necessary to download the source of the Android framework code and manually add annotations to it. Instead, we can simply define a general thread policy to apply to Android framework classes.

Time for an Example

The following example demonstrates how to declare a thread policy in XML, annotate a simple program and run the CheckThread analyzer to catch a couple of bugs.

CheckThread’s XML syntax supports patterns and wildcards which allows you to concisely define policies for Android framework code. In this example we define two Android specific policies. In general this file would contain more definitions for other Android framework classes and could also contain definitions for your own code.

The first policy declares that all methods in Activity and its subclasses that begin with the prefix “on” should be confined to the main thread. Since CheckThread has no built-in concept of the Android framework or of the Activity class we need to inform the static analyzer which thread will call these methods.

The second policy declares that all methods in classes ending with “View” should be confined to the main thread. The intention is to prevent calling any code that modifies that UI from any other thread than the UI thread. This is a little bit conservative since there are some methods in Android View classes that have no side-effects, but it will do for now.

https://gist.github.com/4113656

Having defined the thread policy, we can turn to our application code. The example app is the rough beginnings of a Hacker News app. It fetches the RSS feed for the front page, parses the titles and displays them in a LinearLayout.

This first version is naive; it does network IO and XML parsing in Activity.onCreate. This error will definitely be caught by StrictMode, and will likely just crash the app on launch, so it would be caught early in development, but it would be even better if it were caught before the app was even run.

https://gist.github.com/4113662

In this code, we make a call to the static method getHttp in the IO utility class. The details of this class and method are not important, but since it does network IO, it should be called from off the UI thread. We’ve annotated the entire class as follows:

https://gist.github.com/4113669

This annotation tells CheckThread that all the methods in this class should be called from the “other” thread.

Finally, we’re ready to run the static analyzer. CheckThread provides several ways to run the analysis tool, including Eclipse and Intellij plugins, but we’ll just use the Ant tasks on the command line. This is what CheckThread outputs after we run the analyzer:

https://gist.github.com/4113676

As you can see, CheckThread reports an error: we’re calling a method that should be confined to the “other” thread from a method that’s confined to “MAIN”. One solution to this problem is to start a new thread to do network IO on. We create an anonymous subclass of java.util.Thread and override run, inside of which we fetch the RSS feed, parse it and update the UI.

https://gist.github.com/4113683

We’ve annotated the run method to be confined to the “other” thread. CheckThread will use this to validate the call to IO.getHttp. After running the analyzer again, we discover that there’s a new error reported:

https://gist.github.com/4113686

This time, the error is caused by calling the method setText on a TextView from a different thread than the UI thread. This error is generated by the combination of our thread policy defined in XML and the annotation on the run method.

Instead, we could call the Activity.runOnUiThread with a new Runnable. The Runnable’s run method is annotated to be confined to the UI thread, since we’re passing it to a framework method that will call it from the UI thread.

https://gist.github.com/4113689

Finally, CheckThread reports no errors to us. Of course that doesn’t mean that the code is bug free, static analysis of any kind has limits. We’ve just gotten some small assurance that the contracts defined in the XML policy and annotations will be held. While this example is simple, and the code we’re analyzing would be greatly simplified by using an AsyncTask, it does demonstrate the class of errors that CheckThread is designed to catch. The complete sample project is published on Github.

The Pros and Cons of Annotations

One drawback that is probably immediately obvious is the need to add annotations to a lot of your code. Specifically, CheckThread’s static analysis is relatively simple, and doesn’t construct a complete call graph of your code. This means that the analyzer will not generate a warning for the code below:

https://gist.github.com/4113695

While this may appear to be a significant problem, it’s my opinion that in practice it is not actually a deal breaker. Java already requires that the programmer write most types in code. This is seen by some as a drawback of Java (and is often cited incorrectly as a drawback of static typing in general). However there are real advantages to annotating code with type signatures, and even proponents of languages with powerful type inference will admit this, since it’s usually recommended to write the type of “top-level” or publicly exported functions even if the compiler can infer the type without any hint.

The annotations that CheckThread uses are similar; they describe an important part of a method’s contract, that is whether it is thread safe or should be confined to a specific thread. Requiring the programmer to write those annotations elevates the challenge of writing correct multithreaded code to be at the forefront of the programmer’s mind, requiring that some thought be put into each method’s contract. The use of automated static analysis makes it less likely that a comment will become stale and describe a method as thread safe when it is not.

The Future of Static Analysis

The good news is that the future of static analysis tools designed to catch multithreaded bugs is looking very bright. A recent paper published by Sai Zhang, Hao Lü, and Michael D. Ernst at the University of Washington describes a more powerful approach to analyzing multithreaded GUI programs. Their work targets Android applications as well as Java programs written using other GUI frameworks. The analyzer described in their paper specifically does construct a complete call graph of the program being analyzed. In addition, it doesn’t require any annotations by the programmer and also addresses the use of reflection in building the call graph, which Android specifically uses to inflate layouts from XML. This work was published only this past summer, and the tool itself is underdocumented at the moment, but I recommend that anyone interested in this area read the paper which outlines their work quite clearly.

The secret behind this flexibility is how developed the Java toolchain has become. One tool (Eclipse)  in particular has Eclipsed the rest and become the dominant IDE for Java developers. Eclipse is free, with integrations like JUnit support, and a host of really great plugins making it the de facto standard in Java development, displacing IntelliJ and other options.  In fact, entry level developers rarely even think about the compilation step, because Eclipse’s autocompilation keeps your code compiled every time you save a file.

There’s Always a Catch

Unfortunately no technology is magical and while this set up rarely causes issues, it can. One interesting case arises when the developer is using the Eclipse 1.6 compiler compliance and the target environment uses Sun’s 1.6 JDK compiler.  For example at Flurry, during development we rely on Eclipse’s JDT Compiler, but the code we ship gets compiled for deployment on a Jenkins server by Ant using Sun’s JDK compiler. Note that both the developer and continuous integration environment are building for Java 6, but using different compilers. 

Until recently this never came up as an issue as the Eclipse and Sun compilers, even when running on different operating systems, behave almost identically.  However, we have been running into some interesting (and frustrating) compiler issues that are essentially changing “Write Once, Run Anywhere” into “Write Once, Compile Using Your Target JDK, Run Anywhere.”  We have valid 1.6 Java code using generics, which compiles fine under Eclipse, but won’t compile using Sun’s javac.

Let’s See an Example

An example of the code in question is below. Note that it meets the Java specification and should be a valid program. In fact, in Eclipse using Java 1.6 compiler compliance the code compiles, but won’t compile using Sun’s 1.6 JDK javac.

https://gist.github.com/4035987

Compiling this code using javac in the Sun 1.6 JDK gives this compiler error:

https://gist.github.com/4036089

“Write Once, Run Anywhere” never made any promises about using different compilers, but the fact that our toolchain was using a different compiler than our build server never bore much thought until now.

Possible Solutions

The obvious solution is to have all developers work on the same environment as where the code will be deployed, but this would defer developers from using their preferred environment and impact productivity by constraining our development options. Possible solutions we have kicked around :

1. Have ant compile using the Eclipse incremental compiler, (using flags  -Dbuild.compiler=org.eclipse.jdt.core.JDTCompilerAdapter and of course -Dant.build.javac.target=1.6). This side steps the problem by forcing the continuous integration system to use the same compiler as developer laptops, but is not ideal as this was never an intended use of the Eclipse compiler. 
2. Move to the 1.7 JDK for compilation, using a target 1.6 bytecode. This solves this particular issue, but what happens in the future?
3. Change the code to compile under Sun’s JDK. This is not a bad option but will cost some speed of development found in the simplicity of Eclipse’s built in system. 

My experience has been that Eclipse is a well worn path in the Java world, and its a little surprising that this hasn’t come up before for us given the heavy use of generics (although there are lots of generics issues which have been logged over at bugs.sun.com, like http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6302954 which has come up for us as a related issue – the “no unique maximal instance exists” error message). 

Switching to use the Eclipse compiler for our deployable artifacts would be an unorthodox move, and I’m curious if anyone out there reading this has done that, and if so, with what results.

We had a discussion internally and the consensus was that moving to 1.7 for compilation using a target 1.6 bytecode (#2) should work, but would potentially open us up to bugs in 1.7 (and would require upgrading some internal utility machines to support this).  We aren’t quite ready to make the leap to Java 7, although its probably time to start considering the move. 

For now, we coded around the compiler issue, but its coming up for us regularly now, and we are kicking around the ideas on how to resolve.  In the near term, for the projects that run into this generics compile issue, developers are back to using good ole ant to check if their code will “really” compile.  Its easy to forget how convenient autocompilation has become, and the fact that it isn’t really the same as the build server’s compiler.

Recently we have begun moving parts of the Flurry backend over to a new set of services running on NodeJs.  NodeJS is a fast, flexible application framework that is especially suited to developing RESTful API services.

Restify

To help us build these new backend components we’re using the NodeJS “Restify” framework.  Restify provides an extensive amount of low-level functionality to help you correctly implement RESTful services – much in the same way the “Express” NodeJS module helps you rapidly develop web-based applications – for those of you familiar with Express, you’d feel right at home in Restify.  We’ve found Restify to be a fantastic framework for development – easy to use, understand and implement.

Mocha

To facilitate unit testing, we’re using the Mocha Javascript unit testing framework for everything we do and have found it to be flexible and easy to use.  Mocha makes it really easy to write unit tests for your NodeJS code – so easy in fact, we decided we wanted to use Mocha for our regression testing.

After some trial and error, we have settled on a fairly simple setup, which we have found works well and is easy to implement.  The following steps outline the process, and for this small tutorial we’ll build the requisite “Hello World” API that simple returns “Hello World” when called.

Before we get started, we first want to make sure that both Restify and Mocha are installed for use in our new Node project:

https://gist.github.com/3821529

Once those are installed, we’re ready to create our sample “Hello World” API service, as well as setup the Mocha regression test cases.

Here’s the contents of the app.js file that we will be using for testing:

https://gist.github.com/3821547

You can see that unlike other app.js examples you may have seen, this one is very small and simply makes a function call to StartServer() which has been exported from the start_server.js file.  This function simply starts the server – we’ll cover that below.

Start up the NodeJS service in Mocha

Before we can do any regression testing against our “Hello World” service, we must first start up the Hello World service.  To do this, we are going to create a special “before” Mocha hook – this will run before any of the other regression test files are run by Mocha, and will allow us to start the service so it can be tested.

Within your directory, create a sub directory called “test”.  All of our regression test and unit test files are going to be located inside.  Create a new file called “01-start-server.js” with the following contents:

https://gist.github.com/3821576

This file will be picked up as the first file in the directory (that’s why we started the filename with 01), and the before() function will be executed, which in turn requires and runs our StartServer() function.   The StartServer() function is defined in the start_server.js file:

https://gist.github.com/3821649

It’s purpose is to actually initialize the Restify listeners and start the service.

Create a Mocha regression test

Now that we have a way to automatically start the service before we need to test it, we can go about the business of writing our regression test cases.  Since our Hello World service is so simple, we’re just going to have one test – we’re going to test to be sure our call to the /hello service returns an HTTP response code of “200”, indicating our request was “OK”:

https://gist.github.com/3821607

Initializing an HTTP Client

At the very top of our file that contains the test cases for our Hello World service you can see we are using a feature of Restify – the Restify JSON Client.  The JSON Client is a small wrapper around NodeJS’s http module that makes it easy to call URL based web services that return their content in JSON format.  The JSON client will automatically JSON.parse the response body and make it available for your use (as the “data” parameter in our callback function).

Once we’ve created our client, we can then use the client to make a GET call to our /hello service URL.  If we encounter an error connecting to the service, our “err” parameter will contain the error.  The “data” parameter will contain the data returned from the call, so we will want to check that to be sure it contains the data we requested. 

Running the Regression Test

Now that we have our test in place, the next step is to actually run it, which is as easy as typing “mocha” in your project directory:

https://gist.github.com/3821733

Mocha will first run the “01-start-server.js” file in the test directory – this starts our service.  Next, it will move on to the service_hello_tests.js file and run our solitary regression test.  If the service responds as we have outlined in our test, the test will be marked as passed by Mocha.

Using this simple setup we can add as many additional tests as needed – either extending our “hello” service, or writing additional test cases for new services.

Using Mocha for both unit testing and regression testing allows us to save time by only having to deal with one testing framework – Mocha is flexible enough to make running unit tests in more complex scenarios fairly straight forward and easy to do.  Now if only it could write your unit tests for you

While Flurry’s growth is great news for our customers, employees, and investors, it creates challenges for Release Management to support the growing number of applications, the growing number of developers, and the growing number of  development and test environments.  We need to manage all of that well and do so quickly and reliably.

In short, we need continuous integration to rapidly build, deploy, and test our applications.

Deployment Tools @ Flurry
 
To support our continuous integration, we setup three core tools.

• Source Control: We use Github Enterprise to manage our source code and various configuration files. We use a variation of the Git Flow development process where all features are developed on individual branches and every merge is in the form of a pull request (which is also code reviewed).
• Continuous Build: We use Jenkins to build code and deploy to our QA and test environments and to run our JUnit tests.  Jenkins is set up to automatically run JUnit tests when code developers check-in code and when they create pull requests for their branches.  Jenkins also runs Selenium tests with SauceLabs every night against the QA environments and Production.
• Task Tracking: We use Jira (with the Greenhopper agile plugin) for ticket management for planning and tracking enhancements and bug fixes.  All three tools are well integrated with various plug-ins that allow them to share information and to trigger actions.

Challenges at Scale
 
Our setup for continuous integration has served us well but has some challenges.

• Too Many Jobs: We have more than 50 Jenkins jobs.  We have over 130 deployment scripts and more than 1,600 configuration files for the CI tools and applications.  Each new application and each new QA environment adds to the pile.  While we are whizzes at writing bash scripts and Java programs, this is clearly not scalable in the long term.
• Slow Deployments: For security reasons, our Jenkins server cannot deploy war files and config files directly to Production servers.  For Production deployments, we run a Jenkins job that copies the files built by other jobs to a server in the data center over a locked-down, one-way secure tunnel.  Ops staff then manually runs various scripts to push the files to the Production servers and restart them.  This is inefficient in terms of time and people resources.
• Test Overrun: Our JUnit test suite has over 1,000 test cases which take about an hour to run.  With the increase in the number of developers, the number of test runs triggered by their pull requests is clogging the Jenkins build server. We have biweekly releases to Production which we would like to be able to cut down to a daily basis or at least every few days.  The blocker to this is that the build, deploy, test, analyze, and fix cycle is too long to allow that.

Improving the Process: Design for Speed

The speed of an engineering team is directly related to the speed of release and deployment so we needed to get faster. We have taken a number of steps to address the challenges.

• We optimized our JUnit test cases by removing unnecessary sleep statements and stubbed out the deployments to our test CDN which reduces network wait time.
• We upgraded the build system to bigger, faster hardware and parallelized the JUnit test runs so that we can run multiple test jobs at the same time.
• We added dedicated Jenkins slave servers that can share the burden during times of heavy parallel building. 

Overall we have reduced the time to run the entire test suite to 15 minutes.

To make it easier to manage the Jenkins jobs, we removed a lot of old jobs and combined others using parameterized builds.  We renamed the remaining Jenkins jobs to follow a standard naming convention and organized them into tabbed views.  We now have a dozen jobs laid out where people can find them.

Jenkins

All of the improvement steps have helped, but we needed more fundamental changes.

Improving the Process: RPM Based Software Deployments
 
We changed our build and deployment process to use RPM repositories where every environment has its own local repository of RPMs.  In the new process, the Jenkins job builds the war files then bundles up each war file along with its install script.  The job also builds RPMs for each application’s config files, email templates, data files and the config files for HBase and Hadoop.  Once all of the RPMs are built, the job rsyncs the RPMs to the target environment’s repo.  It then runs ssh yum install against each server in the environment to get it to update itself.  Once all the servers are updated, the job restarts all of the servers at once.  The job is parameterized so that users can build and deploy a single app, a set of apps, all apps, or just some config files.

The developers have access to Jenkins so that they can update the QA environments at will without having to involve Release Engineering.

The RPM-based build and deployment process gives us several advantages.  The install scripts are embedded into the RPMs which reduces the cluttered hierarchy of scripts called by the Jenkins jobs.  The same tools and processes for deployments in the Dev and QA environments can now be safely used in Production.

By having a repo for each environment, we only have to deploy the RPMs once to that repo.  Each sever in the environment then pulls the RPMs from its repo.  This save a lot of time and network bandwidth for our remote environments whose servers used to get files directly from Jenkins.

RPMs support dependencies which instruct yum to deploy a group of other RPMs before deploying the given RPM.  For example, we can set an application’s RPM to be dependent of the application’s config file RPM, so that when we install the application, yum automatically installs the config file RPM.  The dependency feature also allows us to set up a parent RPM for each class of server where the parent RPM is dependent on all of the application RPMs that run on that class of server.  We simple execute yum install with the parent RPM, and yum downloads and installs all of the application RPMs and their dependent config file RPMs needed for that server.  In the future we will add dependencies for Java, Jetty, and various OS packages to the parent RPMs.  This will allow us to kick start a new server and fully provision it at the push of a button.

Conclusion

As with any change in process and tools, there were a few gotchas.  The Jenkins slave server was easy to set up, but there were a lot of tools and configurations needed to support our junit test runs that had to be copied from the Jenkins master server.  We also found a few places where the concurrent junit tests runs stepped on common files.

Overall, the changes have sped up and cleaned up our build and deployments.  They have allowed us to better manage what we have and to handle our future growth.