Continuously Deployed

Ruby Caches - Improving Caching Via Tooling

2025-02-10T00:00:00+00:00

A series of posts will explore and detail some of the current Rails caching code. This post takes a deeper look at the network requests involved in caching.

These posts attempt to explain in more detail, but please do read the official Rails Caching Guide docs, which are also very good.

Rails Cache Posts:

Ruby Cache Improvements

If you want to drive performance improvements for a library, you will need to understand some of the use cases and the structure of the code. We have covered how to use the Rails cache, and we took a deep look at the caching code workflow, including where time is most often spent, as well as digging into cache network requests to see where network IO time is spent. Now, if you want to start to improve caching, you can look at adding features to help avoid issues like cache stampede, or you can focus on reducing the latency for using the cache. This post will look at lowering latency when using the cache.

Understanding latency with benchmarks and profiles

If we want to make something faster we want data to help drive the improvements, often the best way to speed something up is using benchmarks and profilers. When I reach for one of these tools, I usually want both. I can use the benchmarks to see where I am in a range of options and conditions, then use the profiler to dig into specific use cases that don’t match my needs or expectations to target specific improvements. After fixing the profile, the increased performance is usually evident in the relevant benchmarks. Tooling like this can be a great feedback loop for driving improvements. It can also help avoid regressions. This is why, on many of the projects I work on, my team and I set up benchmarks and profiling as part of the CI pipeline. We have a complete history of performance improvements, and we can fail CI for significant regressions or at least investigate the regression, which is sometimes required for a bug fix or new feature.

A PR adding the benchmark and profile scripts to Dalli.

Benchmarking Dalli with GitHub Actions

Let’s examine how we can set up a suite of benchmarks for the Dalli Rubygem, which powers the Memcached flavor of Rails.cache.

create a new GitHub action with the content below .github/workflows/benchmarks.yml

name: Benchmarks

on: [push, pull_request]

jobs:
 build:
 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v4
 - name: Install Memcached 1.6.23
 working-directory: scripts
 env:
 MEMCACHED_VERSION: 1.6.23
 run: |
 chmod +x ./install_memcached.sh
 ./install_memcached.sh
 memcached -d
 memcached -d -p 11222
 - name: Set up Ruby
 uses: ruby/setup-ruby@v1
 with:
 ruby-version: 3.2
 bundler-cache: true # 'bundle install' and cache
 - name: Run Benchmarks
 run: RUBY_YJIT_ENABLE=1 BENCH_TARGET=all bundle exec bin/benchmark

This adds a new action that will install everything needed, run two instances of Memcached, and then run our benchmark script.

add the benchmark script bin/benchmark this file is large, so you can get it from the Dalli repo, some brief snippets below

#!/usr/bin/env ruby
# frozen_string_literal: true

# This helps benchmark current performance of Dalli
#
# run with:
# bundle exec bin/benchmark
# RUBY_YJIT_ENABLE=1 BENCH_TARGET=get bundle exec bin/benchmark
require 'bundler/inline'
require 'json'

gemfile do
 source 'https://rubygems.org'
 gem 'benchmark-ips'
 gem 'logger'
end

require_relative '../lib/dalli'
require 'benchmark/ips'

...a bunch more code...

if %w[all set].include?(bench_target)
  Benchmark.ips do |x|
    x.config(warmup: bench_warmup, time: bench_time, suite: suite)
    x.report('client set') { client.set('key', payload) }
    x.report('raw sock set') do
    sock.write("ms sock_key #{payload.bytesize} T3600 MS\r\n")
    sock.write(payload)
    sock.write("\r\n")
    sock.flush
    sock.readline # clear the buffer
  end
  x.compare!
  end
end

The benchmark lets us measure the iterations per second of all the command cache actions get, set, get_multi, and read_multi. For the benchmark, I compare against extremely simplified memcached access via a raw Ruby socket. This makes it easier to see what we can still improve, ensuring the abstractions in the Dalli gem are very performant and offer very low overhead compared to the fastest pure Ruby implementation you could expect.

This benchmark runs on all PRs and outputs the current iterations per second of all the methods we have configured. We can then make PRs and include the before-and-after expectations of the benchmark data.

Profiling Dalli with GitHub Actions

We also set up a similar CI action for profiling, so we are always capturing profiles and can look back if a PR has some unexpected change after we role it out. The same profiling tooling, while nice in CI, is also part of the very fast feedback loop along with the benchmark script when doing performance work.

create a new GitHub action for profiling .github/workflows/profile.yml

and add this content to the new file

name: Profiles

on: [push, pull_request]

jobs:
 build:
 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v4
 - name: Install Memcached 1.6.23
 working-directory: scripts
 env:
 MEMCACHED_VERSION: 1.6.23
 run: |
 chmod +x ./install_memcached.sh
 ./install_memcached.sh
 memcached -d
 - name: Set up Ruby
 uses: ruby/setup-ruby@v1
 with:
 ruby-version: 3.4
 bundler-cache: true # 'bundle install' and cache
 - name: Run Profiles
 run: RUBY_YJIT_ENABLE=1 BENCH_TARGET=all bundle exec bin/profile
 - name: Upload profile results
 uses: actions/upload-artifact@v4
 with:
 name: profile-results
 path: |
 client_get_profile.json
 socket_get_profile.json
 client_set_profile.json
 socket_set_profile.json
 client_get_multi_profile.json
 socket_get_multi_profile.json
 client_set_multi_profile.json
 socket_set_multi_profile.json

Similar to the benchmark, this installs the required things, gets the Memcached server up and running, and then kicks off our profile script. The action then uploads all of the profiling results, which can be retrieved after any CI run.

The profiling script is long, so you can get that from the Dalli repo, but below are some snippets

#!/usr/bin/env ruby
# frozen_string_literal: true

# This helps profile specific call paths in Dalli
# finding and fixing performance issues in these profiles should result in improvements in the dalli benchmarks
#
# run with:
# RUBY_YJIT_ENABLE=1 bundle exec bin/profile
require 'bundler/inline'
require 'json'

gemfile do
 source 'https://rubygems.org'
 gem 'benchmark-ips'
 gem 'vernier'
 gem 'logger'
end

require_relative '../lib/dalli'
require 'benchmark/ips'
require 'vernier'

... lots more setup code...

if %w[all set].include?(bench_target)
  Vernier.profile(out: 'client_set_profile.json') do
    start_time = Time.now
    client.set(dalli_key, payload, 3600, raw: true) while Time.now - start_time < bench_time
  end

  Vernier.profile(out: 'socket_set_profile.json') do
    start_time = Time.now
    while Time.now - start_time < bench_time
      sock.write("ms sock_key #{payload.bytesize} T3600 MS\r\n")
      sock.write(payload)
      sock.write("\r\n")
      sock.flush
      sock.readline # clear the buffer
    end
  end
end

Again, like the benchmarking script we can target all the most important caching actions, in many we can also compare the profile of the raw socket to our Dalli implementation to help find differences.

Developer workflow leveraging the benchmark and profiling tools

It depends on your needs, but let’s say we want to improve writing to the cache; perhaps we have noticed that some write-heavy workloads are not performing as well with Dalli as other cache implementations. Beyond having the benchmarks as part of CI, several applications include benchmarking scripts so we can compare benchmarks across various caching backends, which helps us spot unexpected differences.

Capture a Benchmark Baseline

We can capture a baseline by running the benchmark script and targetting the set method.

Warming up --------------------------------------
 client set   304.000 i/100ms
 raw sock set   827.000 i/100ms
Calculating -------------------------------------
 client set      2.998k (± 8.9%) i/s  (333.52 μs/i) -     29.792k in  10.009832s
 raw sock set      8.254k (± 2.8%) i/s  (121.15 μs/i) -     82.700k in  10.027442s
Comparison:
 raw sock set:     8254.3 i/s
 client set:     2998.4 i/s - 2.75x  slower

Profile the Target method

Then, we can profile the set method and look for unexpected spikes.

on this flame graph an important section with string manipulation jumps out

Why are we spending more than 6% of our time modifying a string?

The profiles help you track down where time is spent and possibly where time is wasted.

Refactor, iterate, fix hot spots

Now you have a good loop, make a small change, generate another profile run, and attempt to fix hotspots.

primary win is by writing the value directly to the IO object
- This avoids copying a potentially very large string, which can cause numerous allocations when resizing the buffer
some smaller wins avoiding some other string manipulation

After each code change, you can look at the new profile flame graph and see if it fixes the issue you were targeting.

Then, finally, run your benchmarks again to see if fixing what you saw in the profile data improves your microbenchmark.

Warming up --------------------------------------
 client set   724.000 i/100ms
 raw sock set   828.000 i/100ms
Calculating -------------------------------------
 client set      7.434k (± 2.4%) i/s  (134.51 μs/i) -     74.572k in  10.037368s
 raw sock set      8.209k (± 4.5%) i/s  (121.82 μs/i) -     81.972k in  10.011659s
Comparison:
 raw sock set:     8208.7 i/s
 client set:     7434.2 i/s - 1.10x  slower

Rerun the benchmark

When you have fixed all the issues in the profile, you rerun the benchmark, and hopefully, the improvement makes a noticeable difference. While things are different in real-world applications, often, what shows up in your microbenchmark will translate into real-world improvements.

Ship your performance Improvement

You can now ship a PR that includes the before-and-after benchmark data and screenshots showing the improvement in the flame graph.

This basic workflow was followed for a real PR improving the set performance for dalli, which you can check out for complete details.

Invest in the toolchain

Since we invested in making these benchmark and profile scripts both available to run locally for developers and as part of CI, it makes it easier for anyone to make performance improvements. It also simplifies any bugs reported around performance as we should be able to review it within our tool chain or we can add the new use case as a path to optimize in our scripts.

Over time, this can help bring consistent improvement to a library and help avoid accidental regressions.

NOTE: At the moment, these improvements are on a research fork of Dalli, but we have talked with the Dalli maintainer and will start porting various improvements into the released gem.

Ruby Caches - Rails Local Cache Strategy

2024-12-17T00:00:00+00:00

A series of posts will explore and detail some of the current Rails caching code. This post takes a deeper look at the network requests involved in caching.

These posts attempt to explain in more detail, but please do read the official Rails Caching Guide docs, which are also very good.

Rails Cache Posts:

Rails Cache Local Cache Strategy

A semi hidden feature of Rails is the ability to use a local cache strategy along with rails cache. You might already be using it in your app without realizing it.

Let’s take a look at how it works. When you want to use it and when you might want to avoid it.

What is Active Support Local Cache Strategy?

There are some abstractions to support cache strategies in Rails, but sadly the only one actually in Rails is the ActiveSupport::Cache::LocalCacheStore. Although, I am aware of code that leverages cache strategies in private code repositories to many various layered cache strategies.

Local Cache Strategy

Caches that implement LocalCache will be backed by an in-memory cache for the duration of a block. Repeated calls to the cache for the same key will hit the in-memory cache for faster access. –Local Cache Strategy

Basically, this is fancy memoization that tries to avoid making repeated network calls to a remote cache store. It is a bit fancier because it tries to handle some of the complexities for you. For example, in a Rails web app, it has Rack middleware to clear the memoized cache when the request is finished. Also, it is implemented in a way that is thread safe.

It isn’t used in all of the Rails cache stores as it doesn’t make sense for the local MemoryStore to use it as they are both in memory stores. It was also removed from Rails FileStore when it had to use serialization to avoid mutation related bugs, which we will describe more a bit later.

How Strategy::LocalCache works

You will see it used in both the Redis and Memcached cache stores. It is included in those classes like so:

prepend Strategy::LocalCache

This will mean that every time you write to the cache, it will also write to the memory local cache. And every time you read from the cache, it will first check the local cache. It will try to serialize the value to the local cache and deserialize it when you read from the local cache, this avoids an issue where you can modify a value and possibly not get what you expected from the cache, due to runtime modifications of the original object stored in the cache.

It clears the local cache when the request is finished, this is done by the Rack middleware. which is included like so:

module Strategy
  autoload :LocalCache, "active_support/cache/strategy/local_cache"
end

While the idea is simple, the local cache has evolved over time to handle some of the complexities around mutation, memory usage, and serialization.

TODO: update my mermaid diagram to include flow with local cache strategy

Local Cache History

Originally, it was just a hash of the in memory objects being used in the cache. This was simple, but it had issues with mutations. First there was an issue with mutation on the value read from the cache.

data = cache.read('key') # => "foobar"
data << "bar"
cache.read('key') # => "foobarbar"

After that was fixed, there was a similar issue with modifiying the value written to the cache as seen in this example.

my_string = "foo"
cache.write('key', my_string)
my_string << "bar"
cache.read('key') # => "foobar"

The various fixes were introduced using dup_value! which was a deep dup of the value. These fixes did solved the mutation issue, but it had a larger performance hit, as it was effectively calling Marshal on every cache write and read @value = Marshal.load(Marshal.dump(@value))

The performance problems ended up being reported as a Rails issue, large performance regression of the local cache strategy. It had a decent reproducible benchmark, and investigation resulted in a number of changes for the local store and a good explanation of the challenges. the

A few fixes such as avoiding as much of the performance hit and avoiding use of Marshal as possible were put into Rails. The cache moved away from the duplicate serialization and added some hot path optimizations for dupable entries, but still requires some serialization overhead to protect against mutation. The improvements helped significantly especially for simpler cache objects, but for large serialized object the overhead was still significantly worse, like 300X slower

the local cache will only save the network round trip (and internal Redis/memcached latency).

After the bug fixes and performance fixes were in, the local cache no longer made sense for the file store as it wasn’t just a memoized value in a hash, but required serialization to avoid mutation bugs.

In the discussion it was pointed out that there are some reasons to side step the local cache strategy and just use request local memoization. Especially for larger cache objects where the serialization overhead is more significant.

When to not use local cache?

If you are going to read the value multiple times in the same request, you can wrap the value from the cache and memoize it in a local variable. This can be done around either the fetch or read method. This is often valuable if many different parts of the codebase are trying to get from the cache through different call paths.

Depending on your needs you can have a simple memoized method that is an accessor or a memoized hash accepting cache keys to handle related caches. Something like below if your class has the correct lifecycle. If you need kepe the cache tied to the request lifecycle, you can use ActiveSupport::CurrentAttributes to keep the memoized value in the request local scope.

class SomeCacheableClass
    def memoized_cache_fetch(key)
        @memoized_hash[key] ||= Rails.cache.fetch(key) { yield }
    end
end

If you are using memoization you are back to extremely fast performance because it skips the serialization penalty and hash lookups are very fast, but you need to be careful to not modify the value you are memoizing, or at least be aware that it can be mutated. As you can see, returning a memoized value will be orders of magnitude faster than anything that requires serialization or network calls.

Comparison:
cache memcached 1x memoized_cache_fetch: 11633145.8 i/s
cache memcached 1x fetch:     4209.4 i/s - 2763.63x  slower
cache memcached 1x fetch bypass local store:     2023.2 i/s - 5749.82x  slower

So if you see significant slowdown when repeatly getting the same data out of the cache don’t be afraid to memoize it, if you look at profiling data or a flame graph you will see duplicate deserialization is slowing you down and can be avoided.

Other Rails Cache Strategies?

Since the current local cache strategy is the only one in Rails, maybe we could show how to build a not mutation safe but faster hash lookup strategy. Basically re-introducing the original hash strategy. Sometimes the trade offs are worth it, when it is clear what the pain and cost would be. We can look at how to build that in a future post.

Ruby Caches - Rails Cache Network Requests

2024-11-07T00:00:00+00:00

A series of posts will explore and detail some of the current Rails caching code. This post takes a deeper look at the network requests involved in caching.

These posts attempt to explain in more detail, but please do read the official Rails Caching Guide docs, which are also very good.

Rails Cache Posts:

Rails Cache Network Requests

Most of the larger Rails applications end up using a cache server that is accessed over the network. This is because many of the simpler caches, like filestore and memory, do not work for a horizontally scaling Rails deployment. In contrast, you can set up a localhost Redis or Memcached server, which is generally just for development or small-scale applications. Let’s look at what happens with a typical Rails cache configured with a memcached server.

Let’s not look at serialization or compression but at what is going on to get data over the network.

Rails Cache Network Calls

In the previous post, Rails cache workflows, we covered how a Rails.cache.read will make a network request to the memcached Server to get the data.

Rails.cache.write("key", "my cached value")
result = Rails.cache.read("key")
puts result
=> "my cached value"

The Rails.cache.read call will use the Dalli Ruby client to request data from a Memcached server. What does that look like? Let’s look at the code to use Dalli directly to better understand what is happening. The Rails memcached store just provides helpers for working with Dalli.

dalli_key = "dalli_key"
dalli = Dalli::Client.new('localhost:11211')
dalli.set(dalli_key, "my cached value")
result = dalli.get(dalli_key)
puts result
=> "my cached value"

Usually, when interacting with a cache, you will use Rails.cache; outside of Rails, you might use Dalli (or Redis) directly. While these gems are great, it is good to know that they are simple network clients under the hood. For example, instead of using Dalli, we could do all of this with far less robust error handling (IE don’t use this code in production). Hopefully, this shows the layers of abstraction between Rails, the underlying caching library, and the Ruby code used to make network requests. Each layer of abstraction adds value and convience, but understanding all the layers can be helpful. For example, when you want to help performance optimize something, knowing which layer is slowing things down can be helpful.

sock = TCPSocket.new('127.0.0.1', '11211', connect_timeout: 1)
sock.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, true)
sock.setsockopt(Socket::SOL_SOCKET, Socket::SO_KEEPALIVE, true)

# set a cache value via a raw socket
sock.write("set sock_key 0 3600 15\r\n")
sock.write("my cached value")
sock.write("\r\n")
sock.flush
sock.readline # clear the buffer

# read a cache value via a raw socket
sock.write("get sock_key\r\n")
sock.readline
result = sock.read(15)
puts result
=> "my cached value"

Caches and Network Failures

Simple enough, but as with anything over the network, what happens if the network goes down?

dalli.get("dalli_key")
=> No server available (Dalli::RingError)

That isn’t good; we can’t have trying to use a cache raise exceptions on misses. Well, Rails actually doesn’t. While it uses Dalli under the hood, it has a nicer API that helps avoid things like that.

Rails.cache.read("some_non_existing_key")
=> nil

A miss for a key not found in the cache and a miss because the cache was down will both return nil. So, the Rails.cache API makes it a little simpler to work with network errors. It is still worth considering how your code handles these errors. You often don’t want a nil but instead want to get the data from a different source.

Testing Network Errors with Toxiproxy

OK, we don’t want errors. How do we test what our code will do and how to handle it properly? There are several ways to test network error conditions, but one I like to use is toxiproxy. It can simulate network and system conditions for chaos and resiliency testing, making it easy to test things like the endpoint has gone down.

You can read the toxiproxy ruby docs for instructions on how to get started, but the example code below should provide the basics.

# add to Gemfile
gem "toxiproxy"

Also, update your test_helper.rb

Toxiproxy.populate([{
  name: "memcached",
  listen: "localhost:11222",
  upstream: "localhost:11211",
}])

Then you can add toxiproxy code to any of your tests, in this case, to target what happens when memcached isn’t available.

# in book.rb
def save_to_cache
  Rails.cache.write(Book.key_for_id(id), self, ttl: 600)
end

def self.get_from_cache(id)
  Rails.cache.read(key_for_id(id))
end

def self.key_for_id(id)
  "Model.book.#{id}"
end

# in book_test.rb
test "uses cache get" do
  @book.save!
  @book.save_to_cache
  assert_equal(@book, Book.get_from_cache(@book.id))
  # Note: toxi proxy is not parallel test safe
  Toxiproxy[/memcached/].down do
    assert_equal(nil, Book.get_from_cache(@book.id))
  end
end

The test above makes it easy to see that the Rails cache won’t raise and will just return nil

The example above also shows why you don’t often see a Rails.cache.read in isolation. I often use Rails.cache.fetch. Let’s add some code and write a test to see how that works.

# in book.rb
def self.fetch_from_cache(id)
  Rails.cache.fetch(key_for_id(id)) { Book.find(id) }
end

# in book_test.rb
test "uses db on cache fetch failure" do
  @book.save!
  @book.save_to_cache
  assert_equal(@book, Book.get_from_cache(@book.id))
  # toxi proxy is not parallel test safe
  Toxiproxy[/memcached/].down do
    assert_equal(@book, Book.fetch_from_cache(@book.id))
  end
end

In this case, you can see that using Rails.cache.fetch will more cleanly handle any network issues to your cache server. It also makes it easy to implement read-through caching on any query.

Cache Request Details

While I have some other details I want to get into with Toxiproxy, and how to optimize caching code. This post has gotten long enough, in this post we dug through all the layers and exposed some of the details, and also showed some tooling that can help test and verify network failure conditions. We can build on this further in the next time.

Ruby Caches - Rails Cache Workflows

2024-10-24T00:00:00+00:00

A series of posts will explore and detail some of the current Rails caching code. This post explores the steps and type of work involved in caching.

These posts attempts to explain in a bit more details, but please do read the official Rails Caching Guide docs, which are also very good.

Rails Cache Posts:

Rails Cache Workflows

What happens as part of caching… Like most software, it isn’t magic. This post will break things down a bit so we can talk about and dig in deeper on some of the specifics. I mostly work with and cache about low-level usage of Rails.cache while much of the Rails Caching Guide focuses on view layer caching. My experience has always required more complex caching at the data layer, not just a lazily deferred view layer. Rails view layer caching is built on these same primitives, so when we talk about low-level caching, we mean the Active Support Cache API for interacting with the cache directly. While it supports many things, we are going to dig in on the most commonly used features:

Rails.cache.read: this method gets data from the cache
Rails.cache.write: this method puts data in the cache
Rails.cache.fetch: this method either get from the cache or from the backing data store and then sets it in the cache for the next time

Let’s look at what is happening with these methods so we can see how they help abstract away and handle many shared operations. We will see common patterns like:

serialization (and deserialization)
compression (and decompression)
cache store API specifics

Rails.cache.read

When calling Rails.cache.read, the application code is trying to find some piece of data in the cache.

result = Rails.cache.read(key)

What does this do? It depends, but let’s assume we have the Rails MemCache Store configured as our cache store and a valid Memcached server, which does contain the data stored for the cache key.

ActiveSupport::Cache::Store.read
- normalize the key
- normalize cache key versions (supports recycling keys)
- instruments the cache call (for tracing, logging, testing, stats)
- set entry equal to the response from the configured ActiveSupport::Cache::MemCacheStore.read_entry call
  - this method does two things deserialize_entry(read_serialized_entry(key, **options), **options)
    - read_serialized_entry: this will use the Cache store-specific way to get the serialized data out of the cache
      - for the MemCacheStore this means it will use the Dalli Memcached Client gem to request the data over a socket
    - deserialize_entry: is back in the base class and uses the coder to get the cache entry and metadata
      - @coder.load(payload): this will first have the coder decompress the raw cache data if needed, then it will deserialize it
  - checks meta data on the entry to handle things like cleaning up and not returning expired entries
  - returns the cache entry.value back to the caller of cache.read

All the above can occur in less than 1ms for a remote network cache call when on a fast cloud regional deployment. It is a lot to take in, let’s diagram it.

Rails.cache.write

When calling Rails.cache.write the application code is trying to store some piece of data in the cache.

book = Book.first # some active record book object
Rails.cache.write(key, book, ttl: 3600)

What does this do? It depends, but let’s assume we have the Rails MemCache Store configured as our cache store and a valid Memcached server.

ActiveSupport::Cache::Store.write
- normalize the key
- normalize cache key versions (supports recycling keys)
- instruments the cache call (for tracing, logging, testing, stats)
- builds the CacheEntry object (meta data and value wrapper)
- calls ActiveSupport::Cache::MemCacheStore.write_entry to set the value
  - this method does two things ` write_serialized_entry(key, serialize_entry(entry, **options), **options)`
  - serialize_entry: is back in the base class and uses the coder to serialize the data
    - @coder.dump_compressed(payload): (or just dump if no compression needed) This will have coder serialize and then compress the data
  - write_serialized_entry: this will use the Cache store-specific way to set the serialized data to the cache
    - for the MemCacheStore this means it will use the Dalli Memcached Client gem to send the data over a socket
- returns true if set successfully

All the above can occur in less than 1ms for a remote network cache call when on a fast cloud regional deployment. It is a lot to take in, let’s diagram it.

Rails.cache.fetch

A very common pattern with caching is to try to get a piece of data from the cache, and if you can’t find it, get the data from another data store and save it to the cache so you will get it next time. Rails has the Rails.cache.fetch helper to simplify this.

Rails.cache.fetch(key, ttl: 3600) {
  Book.first # some active record book object
}

It is easiest to think of fetch in terms of both the previous read and write. While it duplicates some of the underlying code, it does not always directly differ from those methods. Let’s break this down for the Memcache store in the case where there is a miss on the cache read.

Rails.cache.fetch
- entry = read_entry: this does most of the same as read, and we will in this case, assume it was a miss and return nil
- save_block_result_to_cache(name, key, options, &block)
- call the block, which in this case is Book.first
- then it is like cache.write with that value

While I very briefly covered fetch, it can do many other interesting things that help with the complexity and edge cases when using a cache, such as race conditions, nil handling, forcing updates, etc.

The most common cache call in most of the Rails apps I work in is Rails.cache.fetch it is extremely powerful for very basic caching.

Notes on Using Rails Cache

The purpose of this post was to get a bit deeper into the foundational concepts and steps related to caching with the active support cache store API. So, we can further explore some of the important concepts like serialization, compression, and specific cache store implementations. It is important to realize that for each different store type, Rails relies on a client; in this case, we talked about Dalli being the Memcached client, but if you use the Redis store, Rails will end up using the Redis gem. While Rails will default to using compression and serialization built into Ruby, other options (often better options) also rely on the gem ecosystem outside of Rails.

In my example, I cached an active record object. The Rails caching guide rightly warns against this, as we get into serialization and versioning, we can cover solutions to problems with caching Active Record objects
These examples should also make it clear that much is happening behind the scenes for such a simple API (can’t get much easier than read('key")), it is important to understand the different pieces when wanting to optimize performance further when working on caching
- for complex data structures, serialization can take longer than network calls
- for certain size data and data formats, avoiding compression can make sense
- knowing how caching breaks down into these steps can help when benchmarking and profiling caching approaches that work best for your data

We will dig in a bit more in a future post.

Creating Consistent Characters Across Images

2024-10-22T00:00:00+00:00

I have been creating children’s books with my kids using AI for a while now. I recently talked to someone, and they were doing the same thing but using just the ChatGPT bot and creating a single image for a short story. I will cover some of my recent progress working on this side project, as well as some of the methods I use.

The Text Isn’t the Problem

Early on, generating amusing enough stories for the kids was super easy. My daughter would request something like, “I want a book about me being a puppy doctor.” Some simple prompt trial and error and parsing would easily make a cute book of any desired length and could even adjust the vocabulary to the appropriate reading level. One could even go for specific styles with rhymes, humor, first person, etc… I will cover the text generation in another post sometime, but once I have an excellent short story, I will try to generate images.

Early Image Problems

The first book’s images were hilariously bad. They were so bad that it was part of the humor. We would all laugh at the incredibly silly images, but immediately, my kids would start to point out all the inconsistencies of the images on each page.

Why doesn’t the book’s Sasha look at all like her?
Why does she have long brown hair on one page, short on another, and orange hair somewhere else?
Why do the clothes change on every single page?
Why do the age, height, etc, vary constantly?

These images were from Dalli-2 and early stable diffusion. I covered some of the processes in a post, Ai Book Building.

Early Attempts to Improve the Images

I started trying a few methods to improve the quality of the images created.

Improved and Consistent Image Prompts

My first attempts to resolve this were just improved prompts. I would add a trick that would regularize the image prompt with additional details describing the character whenever they were to be in the illustration. This was slightly better, but not by much. Most of the quality improvements were from improved Stable Diffusion models.

Prompts included consistent characters Theo the astronaut, a purple good wizard, and Wilbur the Whale

Image2Image Support

I next allowed the book creator to select and upload images that would be used along with image2image and prompts to generate semi-recognizable images that were stylized across the book. The Img2Img technique had several problems:

You end up wanting a unique image per page
The closer the initial image is to the actual desired content, the better quality
If the photos aren’t consistent, neither is the output
The output quality was hit-and-miss
It was extremely time-consuming, now resulting in photo shoots just to seed the book generation
I don’t want to upload a bunch of pictures of my kids to any of the cloud AI companies, so I run it all locally on my own machine

I was stuck at this for a while, so I made a post about the process of using img2img to make book covers. Combining this with different Stable Diffusion models and styles can get good results, but sometimes it takes many iterations or is difficult to get a cartoon or children’s book illustration-like style while also still being recognizable as the character.

While time-consuming, this technique allowed me to put my kids into their books finally

Training LoRA models

“A LoRA diffusion model is a type of model that uses Low-Rank Adaptation (LoRA) to fine-tune a diffusion model for a specific domain, character, style, or concept” – LoRA Models can be trained in the cloud

Eventually, I learned how to take a small collection of high-quality images of one of my kids and train a LoRA model. The tools for local training are a bit hard to work with, but they also can be run in the cloud fairly quickly. Making a LoRA model for use to generate images on a local machine takes about one hour of machine time on my RTX 480. After some trial and error, I can build Stable Diffusion 1.5 or XL models that could fairly frequently build recognizable characters in any situation. The quality of any given image varied a lot, and the older Stable Diffusion models couldn’t set scenes in the books very well. It was close, but not good enough to take the quality of the books to the next level. Combining this technique and custom Stable Diffusion models that better match the scene/theme of the book would help, but only a little, and it was a lot of work. These could occasionally produce very good results, but a large number of image generations per page were required.

Training local LoRA models, was a pretty big breakthrough in terms of more easily making repeatable recognizable characters

I will post on how to train LoRAs on your local machine sometime easily, but you can check out Flux Gym to get started.

Flux LoRA Models

The next big breakthrough came when Flux dev was released. This image model was far superior in building scenes from prompts, so you could put characters in any scene the book would require… The trick of detailed descriptions of a character and a scene also went a long way. However, now I have seen how recognizable and good a trained LoRA model could look with image models. I needed a way to generate Flux LoRA models, but on my RTX 4080 graph card with 16GB of RAM (or I could get it all running in the cloud). The problem was that tooling for training local Flux LoRAs required 24GB, but folks kept tweaking things and layering various improvements. It is slow, but folks are training models on 8GB graphic cards these days.

I can now take a small, high-quality collection of photos of my subject and generate a pretty good Flux LoRA in about 1 hour on my local machine. Example images for input are below.

Example training set of images, this trains a person model with a focus mostly on the face, but will capture basic size / age

Consistent Characters

Now, after spending a bit more time and getting better at training LoRA models and improving the tools, I can fairly easily build a model for any character I want. I generally just need 30-40 images and a bit of time cropping and adjusting the image inputs. You can do this quickly, but the better your input images are, the better your model turns out; I am sure I can improve my process of collecting the input images, but the output quality is already good enough for making fun and easily recognizable consistent characters across one of my kid’s book ideas. You can get consistent outfits with a few details in the prompt, you can get wildly imaginative scenes. It is easy to generate various styles: 2d, 3d, photo-realistic, cartoon, watercolor, etc.

I now have LoRA models of the whole family and can generate funny and silly images. The kids enjoy just requesting silly images, not even full books now. The styles for the image outputs are very flexible, from oil paintings to movie animation styles.

A bunch of random LoRA images showing the variety

A prompt across LoRA Models

To give a bit more of an idea of how the training works and the output it produces, here are a few sample prompts and the output images showing my family inserted into book themes.

Prompt “ultra-realistic photo of ((MODEL_TARGET_NAME dressed in a cowboy hat and jeans)), hyper detail, cinematic lighting, magic neon, Canon EOS R3, Nikon, f/1.4, ISO 200, 1/160s, 8K, RAW, unedited, symmetrical balance, in-frame, 8K”

Prompt “a colorful cartoon image of ((MODEL_TARGET_NAME as an astronaut in space)), hyper detail, cinematic lighting, colorful, 8K”

I build all of these books using a custom Rails app that can use local or remote text and image generators. I will like share some more posts on some of the scripts and code that makes it easier to automate all these processes.

Ruby Caches - Rails Cache Comparisons

2024-10-20T00:00:00+00:00

A series of posts will explore and detail some of the current Rails caching code. This post compares a few of the cache store options.

These posts attempts to explain in a bit more details, but please do read the official Rails Caching Guide docs, which are also very good.

Rails Cache Posts:

Rails Cache Initialization
Rails Cache Comparisons

Rails Caching Comparisons

Rails offers many caching options, most of the difference will matter more as you scale your system architecture. The performance difference of the various options for small scale systems and single box deployments is not that significant. For example, using the Rails in-memory cache works very well for single process applications, it however doesn’t scale to the most common Rails deployments which is Puma with a few workers and threads. At that point you will often see much higher performance with a cache like Memcached or Redis, at that point to maintain extremely low latency you might choose an on box Memcached, which removes the network latency of a remote cache. That will work well until you need to horizontally scale your web workers (or background jobs). At that point you will likely consider remote cache solutions like Memcached, Redis, or the latest SolidCache.

Rails Cache Stores

We can cover a quick overview of the various cache stores and their trade offs. Also, note you can make a custom store that builds off any of these or offers interesting options like layered caching combining both a machine local cache and a remote cache.

Store	Serializes	Compression	Local Store	Cross Proccess	Node Local	Remote	Cluster
memory store	✓	No	No	No	No	No	No
file store	✓	✓	No	✓	✓	No	No
memcached store	✓	✓	✓	✓	✓	✓	✓
redis store	✓	✓	✓	✓	✓	✓	✓
solid cache store	✓	✓	✓	✓	Maybe	✓	✓

Another point that is harder to list in the table is what is memory limited vs disk space limited. The majority of the cache systems are memory limited, but the file store and solid cache (which is backed by a database) is disk space limited.

If there are other considerations or trade offs that should be added to the table let me know.

Compression

The Rails cache for a long time has supported gzip compression, but recently added support passing in your own compression library. This makes it easier to use faster compression libraries like zstd, snappy, and l4z_flex. In general, so long as you are maintaining another dependency already, using a more modern compression library will yield better performance. Compression is even more valuable when you consider the network overhead of transmitting the cache value bytes over the network.

the compression was done against an active record object including a large number of associations with some random string data
the data was serialized using the default Marshal serializer
then we just compressed the serialized bytes

original_value = Book.all.to_a
cache_value =  Marshal.dump(original_value)

rows << ["No Compression", cache_value.bytesize]
rows << ["BrotliCompressor", Compressors::BrotliCompressor.deflate_byte_size(cache_value)]
rows << ["SnappyCompressor", Compressors::SnappyCompressor.deflate_byte_size(cache_value)]
rows << ["ZstdCompressor", Compressors::ZstdCompressor.deflate_byte_size(cache_value)]
rows << ["L4zCompressor", Compressors::L4zCompressor.deflate_byte_size(cache_value)]
rows << ["LZ4FlexCompressor", Compressors::LZ4FlexCompressor.deflate_byte_size(cache_value)]

puts "benchmarking Rails #{@cache} cache compression..."
table = Terminal::Table.new :headings => ['Compressor', 'Bytesize'], :rows => rows
puts table

This shows the byte size of the cache value after compression using each of the various compression libraries.

+-------------------+----------+
| Compressor        | Bytesize |
+-------------------+----------+
| No Compression    | 1023931  |
| BrotliCompressor  | 577629   |
| SnappyCompressor  | 895496   |
| ZstdCompressor    | 587085   |
| L4zCompressor     | 879179   |
| LZ4FlexCompressor | 878160   |
+-------------------+----------+

Obviously, the final bytesize isn’t the only consideration, you also have to consider the CPU overhead of compressing and decompressing the data. The tradeoff on final size also matters less if you aren’t making cache calls over the network. So let’s also benchmark the speed of the compressors, in this case we are just taking the serialized bytes and compressing them, and measuring the speed of the compression (we aren’t decompressing them).

Below is the output of the benchmark, you can see that snappy is the fastest compression library, followed by lz4_flex (a rust based compression library).

Warming up --------------------------------------
                Gzip     3.000 i/100ms
              Brotli     3.000 i/100ms
                Zstd    26.000 i/100ms
              Snappy   316.000 i/100ms
                 LZ4   111.000 i/100ms
             LZ4Flex   196.000 i/100ms
Calculating -------------------------------------
                Gzip     35.028 (± 2.9%) i/s -    177.000 in   5.054752s
              Brotli     35.899 (± 2.8%) i/s -    180.000 in   5.015287s
                Zstd    278.536 (± 1.8%) i/s -      1.404k in   5.042192s
              Snappy      3.087k (± 2.9%) i/s -     15.484k in   5.020421s
                 LZ4      1.101k (± 2.5%) i/s -      5.550k in   5.042399s
             LZ4Flex      1.959k (± 3.5%) i/s -      9.800k in   5.008819s

Comparison:
              Snappy:     3086.8 i/s
             LZ4Flex:     1959.3 i/s - 1.58x  slower
                 LZ4:     1101.4 i/s - 2.80x  slower
                Zstd:      278.5 i/s - 11.08x  slower
              Brotli:       35.9 i/s - 85.98x  slower
                Gzip:       35.0 i/s - 88.12x  slower

I honestly recommend benchmarking this on the system you deploy on as I have seen some variability in performance between different systems.

Serialization

While Marshal has been the default serializer for Rails for a long time, it is not the fastest serializer. The fastest serializer is MessagePack, which has been recently supported in Rails for serialization in a number of places, not just the cache.

I recommend using MessagePack for serialization, not only is it faster it avoids a lot fo the downsizes of Marshal and is far less verbose than JSON. There is not really any good reason to not use MessagePack for serialization.

benchmarking serializers (serialize + deserialize)...
ruby 3.3.4 (2024-07-09 revision be1089c8ec) [arm64-darwin23]
Warming up --------------------------------------
             Marshal     6.000 i/100ms
                JSON     6.000 i/100ms
            AS::JSON     6.000 i/100ms
         MessagePack     7.000 i/100ms
Calculating -------------------------------------
             Marshal     62.984 (± 4.8%) i/s -    318.000 in   5.067421s
                JSON     61.427 (± 3.3%) i/s -    312.000 in   5.082445s
            AS::JSON     61.546 (± 3.2%) i/s -    312.000 in   5.075183s
         MessagePack     78.024 (± 2.6%) i/s -    392.000 in   5.028982s

Comparison:
         MessagePack:       78.0 i/s
             Marshal:       63.0 i/s - 1.24x  slower
            AS::JSON:       61.5 i/s - 1.27x  slower
                JSON:       61.4 i/s - 1.27x  slower

Just serialization:
Warming up --------------------------------------
             Marshal    11.000 i/100ms
                JSON     7.000 i/100ms
            AS::JSON     7.000 i/100ms
         MessagePack    15.000 i/100ms
Calculating -------------------------------------
             Marshal    114.629 (± 3.5%) i/s -    583.000 in   5.090881s
                JSON     74.783 (± 2.7%) i/s -    378.000 in   5.058664s
            AS::JSON     76.329 (± 2.6%) i/s -    385.000 in   5.047530s
         MessagePack    155.636 (± 3.2%) i/s -    780.000 in   5.017039s

Comparison:
         MessagePack:      155.6 i/s
             Marshal:      114.6 i/s - 1.36x  slower
            AS::JSON:       76.3 i/s - 2.04x  slower
                JSON:       74.8 i/s - 2.08x  slower

There are a few objects that are harder to serialize with MsgPack, but generally it is easy to add a custom helper to support them, there are a few odd cases where you can technically find Message Pack is slower than Marshal, but I have not seen this in real world data.

How to optimize and benchmark for network architecture and latency

I will make another post soon that gets more into how to optimize and benchmark for network architecture and latency. I originally tried some techniques that were not the best and wanted to share what worked and what didn’t. If you are just adding caching to a small application, hopefully you can avoid needing to be highly optimized at this level. When starting to look at remote caching, it will be important to consider things like the latency between app servers and the cache server, for example if you have regionally distributed app servers you may want co-located caches in each region, vs a global cache across regions. As this topic opens up many more questions, we can devote a full post to the topic.

Conclusion

We can combine all of the benchmarks to benchmark end to end with any given serializer, compression library, and cache store. The end to end benchmarks make it a bit harder to see the impact of the various changes and most of the time ends up disappearing into the call to the cache store, so the data normalizes the results.

I would recommend starting new applications (or new caching setups) with:

Serialization: Message Pack
Compression: one of the modern compression libraries (Snappy or LZ4Flex)
Cache Store: I would then recommend using the memcached store as it works well even on a local machine installation, and scales across load balanced clusters for extremely large deployments with very simple deployment hassle.

A note on why not Redis, it can be a great choice, but the api for redis is more complex than memcached. The large scale deployments can be a bit more complex. The real issue is that if you make a redis available for caching, you will likely find folks starting to use it for other things. For example trying to share it across background jobs, or using it for other parts of the system architecture. The configuration you want for a caching redis is different than for a background jobs redis, then folks make compromises like having high availability redis as it is needed for jobs, and then using it for caching, which is a far more expensive and not needed configuration for caching. Redis also supports Lua scripts and more complex data types, which can all be great solutions when properly discussed and considered, but you don’t want to find them becoming hidden complexity that folks build on top of what was supposed to be a simple caching solution. Memcached avoids most of these problems by offering a very robust, but more restrictive API.

Ruby Caches - Rails Cache Initialization

2024-10-17T00:00:00+00:00

Rails Cache Initialization

A series of posts will explore and detail some of the current Rails caching code. A good place to start is how the Rails cache is configured and loaded.

This attempts to explain in a bit more details, but please do read the official Rails Caching Guide docs, which are also very good.

The Rails Cache Configuration

Often, folks use different Rails cache configurations in different environments. It isn’t uncommon for folks to have a NullCacheStore in a test or a memory-only cache-store. Then, in development, they have a Memcache or Redis cache with short-lived TTLs, and then a production cluster of cache servers. Let’s see how Rails accomplishes letting folks load different caching configurations.

The Rails cache configuration code by default lives in project/config/environments/<ENV>.rb:

project
│   README.md
│   ...    
│
└───config
│   │   ...
│   │
│   └───environments
│       │   test.rb
│       │   development.rb
│       │   production.rb

A basic development configuration might look something like:

  # Enable/disable caching. By default, caching is disabled.
  # Run rails dev:cache to toggle caching.
  if Rails.root.join("tmp/caching-dev.txt").exist?
    config.action_controller.perform_caching = true
    config.action_controller.enable_fragment_cache_logging = true

    config.cache_store = :redis_cache_store, {
          connect_timeout: 3,  # Defaults to 1 second
          read_timeout: 0.4, # Defaults to 1 second
          write_timeout: 0.4, # Defaults to 1 second
          reconnect_attempts: 2,   # Defaults to 1
          expires_in: 1.day,

          error_handler: ->(method:, returning:, exception:) {
            Rails.logger.error "Redis error: #{exception}, method: #{method}, returning: #{returning}"
    }
    }

    config.public_file_server.headers = {
          "Cache-Control" => "public, max-age=#{2.days.to_i}"
    }
  else
    config.action_controller.perform_caching = false

    config.cache_store = :null_store
  end

Cache Initialization

The configuration sets up all the basic options and tells the application the type of cache to build for a given environment. During Application startup, Rails will build the cache, so that you get your active support cache store when you call Rails.cache.

How does it build that cache store?

The application bootstrap railtie, initialize_cache
This bootstrap will read the cache config, then feed it into ActiveSupport::Cache.lookup_store
Which will load any Rails cache, including custom ones, so long as the symbol you pass can be required like so require "active_support/cache/#{store}" * the pattern should be a symbol of :redis_cache_store will look for active_support/cache/redis_cache_store.rb * it will then call new on your cache store, passing in all the additional options you provided in your config

Make Your Own

Given all this, you should be able to create your own custom Active Support Cache Store. Or you can configure the standard stores and set up multiple caches beyond just Rails.cache. If you have different caching needs, you can make a new cache the same way Rails does.

AnotherCache = ActiveSupport::Cache::RedisCacheStore.new(
  connect_timeout: 3, read_timeout: 0.4, write_timeout: 0.4, reconnect_attempts: 2,  expires_in: 1.day
)

What happens when a Cache Store Initializes

It can depend on the implementation, but let’s take a closer look at MemCacheStore#new

It will:

creates Dalli client(s) * either a pool with the option pool: true or with specific pool configuration options * or a single client
It can handle a single cache host or a collection of hosts to be treated as a load-balanced cluster
It initializes most of the objects in memory

It does not immediately create connections to Memcached; something in the code will need to call a cache method that requires an active connection, and it will lazily initialize the connection. Some folks do this purposefully to warm the connection, while others prefer a lazy connection. Either way, when you have a custom cache store, be aware of this and how it relates to forking web servers; if you lazily connect, that can happen after fork. If you eagerly connect, you likely need to ensure your cache store is fork-safe (memcache and redis are fork-safe).

After this initialization is complete, you are ready to call Rails.cache.method wherever you need it in your code.

Gardyn 2.0 four years later

2024-10-11T00:00:00+00:00

Gardyn 2.0 years later

I posted my first review of the Gardyn 2.0 a few years ago. I had had the Gardyn for 2 seasons at that point. Now it has been another couple of seasons, but more importantly, it has required some additional repairs.

Since then, I have wanted to cover the Gardyn and some of my basement grow setup.

Gardyn 2.0 still worth it years later?

The simple answer is yes, I am happy with it. I occasionally get a subscription for a few months. Because of my blog post and my surprisingly popular Google Photos garden photo gallery, I have even gotten a few months free from referrals. Thanks, folks.

At this point, I have continued to maintain and use my Gardyn system. I do that seasonally, shutting it down in the summer to focus on my outdoor gardening. Also, I keep some of my bubbled buckets and my lettuce rack going all year. The Gardyn is a bit more maintenance because it grows 30 plants, and I am busy harvesting outside.

The reasons I like the gardyn and still recommend it, it is silly easy and just will make you successful and get you into growing your own greens, it can motivate you to get into the hobby and ensure you really can’t mess it up and kill all your plants. You can always make it more challenging by seeding your own pods, growing things outside of their recommended plants, etc. It inspired me to get bubbler buckets and retrofit an old AeroGarden to just grow with the LED grow light I bought for the bubblers.

The Gardyn is the perfect entry into this hobby, with a nice setup and beyond the small AeroGardens.

Gardyn Maintenance and Repairs

I don’t do all the required deep cleanings and water dumps that Gardyn recommends… I try to clean out the water tank and such occasionally, and I do one deep cleaning a year where I disassemble and clean and reassemble the entire Gardyn. That is part of the reason I don’t run it in the summer, so I can take it apart and clean it. I do recommend Gardyn hydroboost or a similar hydroponic additive to reduce cleanings.

I have twice had most of my plants die because the pump died… The first time, it took me a long time to sort things out, request the official part, repair, and get back up and running. It was not a fun repair, and I lost a lot of plants… The second time which was less than a year after the first repair the pump gave out in late spring, so I just shut the Gardyn down for the summar a bit early, and I had already cleared many of my plants. So it was no big deal, but I wanted to be able to more easily fix it going forward… So I bought a small external water pump and timer, which was cheaper than the official pump and looked more robust… I drilled holes and ran the cables (grey in the image). This works great as I am starting off my new season. Replacing this pump will now only cost about $20m, and if the Gardyn LEDs go out, I might replace them with more external LED grow lights, while I keep using the tub towers and base for growing.

I have read that the newer Gardyns are easier to clean and disassemble, and have made the pump less likely to have water get in an corroding the connections, which is the problem I have had.

Given my Gardyn is over four years old, while this is a bit too many repairs for one to be entirely happy about it, I am pretty happy as it eased me into all this and now I am comfortable and confident to go much further in a DIY fashion because I have learned so much with the initial hand holding of the Gardyn. Also, anything with electronics, water, lights, and various nutrients is a rough environment and repairs are to be expected at some point.

Conclusion

I still need to find out where all the traffic and folks that find my Google Photos album show up from, but I wanted to update my original review with some of these details, and I will continue to add some new photos and updates to the photo album. As the 5th year winter basement garden project season kicks off, I am looking forward to lots of greens and peppers, while my daughter Sasha absolutely loves picking and eating the small cucumbers we grow in the basement.

Rails World 2024

2024-10-01T00:00:00+00:00

Conferences for Remote Developers

I posted a quick note on ruby.social that got some notice and a few folks pinged me.

If you work remotely, going to a conference with a large number of other remote coworkers is a great way to build and refresh in-person relationships. I got to talk in person with 20+ coworkers who were previously only Slack icons and discuss technology and project decisions with the CEO in person. If you work remotely, it is worth finding these IRL touch points from time to time.

I wanted to add some more thoughts, but it gets long on a thread, so moving to a blog post.

Conferences for Remote Developers (long version)

It has been a few years since I visited many Ruby conferences. Attending the most recent RailsWorld was a reminder of some of their value.

I work remotely, so it was a really great way to connect with co-workers, as my employer, Shopify, was the city sponsor in Toronto. This meant that hundreds of employees, both remote and local, attended pre-events, conferences, and outside-of-conference meetings.
It was a reminder that these are real people, many of whom I have seen only through GitHub icons, PRs, and issues for years.
It is fun to have a user of an open-source library I maintain find me and chat about how much they enjoy the project and how they are using it in unique ways.
It is fun to see some speakers’ growth over the years. One of the speakers at the conference gave a keynote talk; I also happened to be in attendance years ago when she gave her very first conference talk. Another easy giving their final talk after ten years of conference talks.

Talking about the work others do and how we use it inspired us to write and discuss how to use these tools and help share which tools are valuable to our work. We should also try to share more of the work we do that isn’t public or shared and could be interesting to the community. Non-private things that are locked away for no particular reason are closed source code.

I hope to make a few more posts about some of the tools we use in the coming months. I will also revisit my open source repos, respond to issues, and update them, ensuring they are all tested and work well with the latest Rails releases. Finally, I will clean up and get my current side project back online so I can share some of the fun things I play around with when learning.

All of that is to say that the conference did exactly what it should have. It inspired me and made me happy to be a part of the Ruby community, and it was great to put faces and real people to the great work I have been enjoying and following along with online for years.

Rails World 2024 Summary

As usual, DHH kicked things off with his keynote address, setting the theme for many of the more detailed follow-up talks.

Keynote tl;dr; DHH wants folks to have a fast onramp but be proud to learn and dig into the details. Driving home, a new tag line from “Hello World” to “IPO” was repeated throughout many of the talks and is clearly a theme.

What is Rails going for?

Many great Rails companies have been around and are still growing, but Rails lost some of the newcomer momentum and is trying to get that back. As Rails became more complex and built more complexity into the base system, it became much harder to set up a brand new app, deploy it, and reliably use all of the features out of the box. Generally, we need a managed DB and Redis out of the box, along with multiple processes for job workers and grinding small single deployments (cheap Heroku dynos) to a halt.

Rails is investing in ensuring out of the box, all the initial solutions can start with extremely simple to-manage defaults that can deploy to single boxes without hassle, and using the same code can scale to much larger databases or purpose-built technologies like Memcache and Redis. Many of the talks filled in the details.

Solid Queue revisits Action Bobs, building out and active job adapter using a data base backed queue that can ship at a starting point with an sqlite DB.
This was a great talk and also gets into some very cool new DB features like skip lock where modern DBs can unlock extremely performant queuing solutions
Solid Cache, another option for Rails.cache and again brand new simple apps can start with sqlite, early scaling can share the production DB, and this scales all the way up to very large distributed shared caches using collections of multiple DBs.
Solid Cable, action cable on the DB starting with sqlite for new apps.

So what does this mean after Rails new? You can start out with 4 DBs that are each single SQLlite files. As you grow in usage you can start to migrate each of those to the best fit for your needs. Also, please don’t underestimate how much it can scale on modern multi-core and SSD hardware.

Rails World 2024 Theme

The theme was clear and then sprinkled with talks on how this useful Rails as a one-person framework can make it easy for a single dev or small team to get a lot of leverage. Speaker justin-searls, giving his final Rails talk, showed how to build low code but very modern feeling apps with low maintenance costs. Diving into optimizing performance, profiling, and how to write fast Ruby with less allocations.

Not all talks were only about the technology as Eileen Uchitelle reminds us how many of the largest and oldest Rails apps have struggled either with microservices or componentizing a modular monolith. Pointing out, rightly in my opinion, that neither solves what is a people problem, not a tech problem. The problem comes from prioritizing features and new code over good maintenance, refactoring, and constantly driving to improve code quality. Also, we often have folks not trained in rails writing a significant amount of code on a team without others familiar with rails, and companies don’t incentivize training and maintenance, often giving promotions to new high-visibility projects vs reducing toil and simplicity.

While the talks were good, the best part for me was seeing and chatting with many developers I interact with online and contributing to projects I use, contribute to, and learn from. It was great to meet @jhawthorn and tell them about how I started using their new profiler vernier (please check it out and see how it can finally untangle performance issues across threads), asking the author of Solid cache about how it may work with a particular architecture setup, or even discussing with colleges if we can perhaps open source some libraries that might help push the community ahead a bit further as folks are facing similar challenges as we see internally.

It was great meeting old friends and talking to some new friends at Rails World. Thanks to everyone who helped make it a nice event.

Increasing Collaboration Confidence with bin/setup

2023-06-12T00:00:00+00:00

A Repo Quick Start

As many organizations with many microservices and employee churn, you can and should expect to often step into a new codebase that needs some smaller updates. It can be really helpful to be able to very quickly go from cloning a repo to being able to run tests and check things in a developer environment. The goal of a number of different helper scripts is to help reduce the total maintenance cost of repositories over their lifetime and accelerate anyone onboarding into the repository.

enter bin/setup

The goal of bin/setup is to go from git clone to being able to run the test suite and boot the app in development mode. The workflow should allow someone to be running any repository very quickly.

> git clone someproject.git
> cd someproject
> bin/setup
> Bootstrapping someproject
> ...
> # checks various depenencies and installs what is missing
> # ... redis, postgres, gems, etc...
> # checks if DBs exist if not creates them and sets them up
> # ... db create, db migrate, db seed, etc...
> ...
> All complete, now you can run `bin/start` for a webserver or `bin/test` to run the tests

The script might have a few assumptions depending on where you work. We have some that if on a Mac will require Homebrew, we have some that might bail if you don’t have Docker. The required dependencies are very small and the script can point you at the documentation to get the minimum requirements up and running.

Language Agnostic and Composable Scripts

While I think bin/setup is the most valuable. You can see I already referenced a few other scripts. Some of the scripts are just one-line aliases, but it still keeps the concept language agnostic. I can get up and running on a Go, Ruby, and Node app on the same day to upgrade a feature touching on the frontend and a number of internal services. In Ruby, for example, the scripts can be very simple.

example bin/test for a Ruby repository:

#!/bin/sh
bundle exec rspec 

example bin/start for a Ruby repository:

#!/bin/sh
PORT=${PORT:-3000} RACK_ENV=development RAILS_ENV=development foreman start -f Procfile.dev --color

The bin/setup script is by far the most complicated, but normally looks something like below:

#!/usr/bin/env ruby

require "fileutils"

APP_ROOT = File.expand_path("..", __dir__)

def system!(*args)
  system(*args) || abort("Command failed: #{args}")
end

def executable?(command)
  ENV["PATH"].split(File::PATH_SEPARATOR).map do |path|
    (ENV["PATHEXT"] ? ENV["PATHEXT"].split(";") : [""]).map do |extension|
      File.executable?(File.join(path, "#{command}#{extension}"))
    end
  end.flatten.any?
end

# WARNING: not portable (macOS and Linux only)
def running?(process_name)
  system("{ ps -ef; docker ps 2>/dev/null; } | grep #{process_name} | grep -v grep > /dev/null 2>&1")
end

def database_exists?(database_name)
  puts %(running database_exists? check for #{database_name})
  system(%(psql -lqt | cut -d \| -f 1 | grep -qw #{database_name}))
end

def create_database!
  system!("bundle exec rails db:create")
end

def check_dependencies
  # Check all dependencies first, and guide the user to the right place
  # to learn more if any are missing.
  #
  # you can add anything you might need here like redis, rabbit, kafka, etc
  # you can either warn or try to detect and install
  unless executable?("psql")
    abort <<~WARNING
      ############################ WARNING ############################
      PostgreSQL is required for using this setup, for tests, the
      server, or the console! For more information, see
      https://internal-wiki.com/psql
    WARNING
  end

  unless running?("postgres")
    abort <<~WARNING
      ############################ WARNING ############################
      PostgreSQL must be running for using this setup, for tests, the
      server, or the console! For more information, see
      https://internal-wiki.com/postgres
    WARNING
  end

  unless require "bundler"
    abort <<~WARNING
      ############################ WARNING ############################
      Bundler is required for using this setup, for tests, the server,
      or the console! For more information, see
      See https://internal-wiki.com/bundler
    WARNING
  end
end

def run_setup
  # This section will go through the basic steps to setup the app to the point specs or server start would work
  puts "Checking dependencies using Bundler"
  system("bundle check") || system!("bundle install")

  puts "Copying example files into place if needed"
  FileUtils.cp "config/database.yml.example", "config/database.yml" unless File.exist?("config/database.yml")

  puts "Creating tmp folder if needed"
  FileUtils.mkdir "tmp" unless File.exist?("tmp")

  create_database!("app_name_development") if !database_exists?("app_name_development")
  if database_exists?("app_name_development")
    puts "Migrating development database"
    system!({"RAILS_ENV" => "development"}, "bin/rails db:migrate")
        puts "seeding development database"
    system!({"RAILS_ENV" => "development"}, "bin/rails db:seed")
  end

  create_database!("app_name_test") if !database_exists?("app_name_test")
  if database_exists?("app_name_test")
    puts "Migrating test database"
    system!({"RAILS_ENV" => "test"}, "RAILS_ENV=test bin/rails db:migrate")
  end

  puts "Installing foreman"
  system! "gem install foreman"
end

FileUtils.chdir APP_ROOT do
  check_dependencies
  run_setup

  puts <<~SUCCESS
    The app is setup
    run tests try:
        bin/test
    To run as a server, try:
        bin/start
  SUCCESS
end

Verify The Scripts on CI

If these scripts aren’t run often and kept up to date, they are bound to break and go un-noticed. It is helpful to verify the scripts on CI. It also generally had the dual purpose of helping keep the script working on both Docker and Macs. Let’s take a look at the things we can run on CI to help verify our app can be setup and run for folks immediately.

While the examples I am giving are for CircleCI, they should easily integrate into any continuous integration toolchain.

Verifying `bin/setup` & `bin/start` on CI

  check_bin_setup:
    <<: *base-job
    steps:
      - checkout
      - restore_cache:
          key: *gem_cache_key
      - run:
          name: Run setup
          command: bin/setup
      - run:
          name: Run setup again (idempotency check)
          command: bin/setup
      - run:
          name: Boot dev server in the background
          command: bin/start
          background: true
      - run:
          name: Start the rails server, wait for it to be available, then make a request and verify the response.
          command: |
            dockerize -wait tcp://localhost:3000/ -timeout 1m
            STATUS_CODE=`curl -s -o /dev/null -w "%{http_code}" -H "Accept: application/json;" -H "Content-Type: application/json"' http://localhost:3000/app_name/healthcheck`
            if [ $STATUS_CODE != "200" ]; then
                echo "Server failed to return a 200"
                exit 2
            fi
          environment:
            RAILS_ENV: development
            RACK_ENV: development

Bonus Points Verify Eager Load on CI

Long as we are verifying things work as expected on CI, another thing that can help avoid an incident is verifying the app works correctly with eager loading. We add a Rake task like below and have our CI testing script call it.

desc "Loads the Rails Application with 'eager_loading=true' as our production deployment eagerloads and we don't want to find out about a missing depedency that late."
task :eager_load_check do
  require File.expand_path('./config/environment', File.dirname(__FILE__))
  Rails.application.eager_load!
end

Enable Any Developer to Quickly Jump Into Any Repo

Suppose your teams make it easy to jump into any repository. In that case, you will help reduce siloing and encourage understanding not only the systems that teams own but working in and helping contribute to the systems in the ecosystem they participate in. It will promote collaboration and exploration and reduce the friction when folks move between teams and projects. There are other steps teams can take to reduce the friction of getting started, but maintaining these few scripts is a low-cost and high-value return for teams managing many repositories.

Making Book Covers with Img2Img

2023-02-27T00:00:00+00:00

I have been having fun making children’s books with my kids using AI. I have been working on some techniques to make them even more personalized. I have recently been building some of the images for the books using image-to-image. I will describe a bit of the process and some of the options I use.

Examples

Below are some examples of the kinds of image options I get when working from photos to make images for the books.

Book Cover Process

If you want to make a similar stylized book cover or image, my basic outline is below.

Get a good description of the image you want. In my example, “A young boy pointing a magic wand and casting a spell.”
Capture a photo that will fit the theme * This is a fun part with my kiddos. We might dress up with Halloween costumes, setup some scenery or backgrounds * the closer the original image is to what you are looking for the better you will be able to bring your kid into the book
either use a local stable diffusion img2img script or an online free tool * I have been using Ruby scripts to run stable diffusion batch jobs locally * You can also often find a free hugging face stable diffusion web-ui model and use the img2img tab
Generate several images adjusting the “Denoising strength” to be more or less like your original image
Photoshop or edit to adjust any other details

example of generating options while using the huggingface free web ui models

What does adjusting Denoising strength do?

Stable Diffusion Denoising Strength is documented as:

Determines how little respect the algorithm should have for image’s content. At 0, nothing will change, and at 1 you’ll get an unrelated image. With values below 1.0, processing will take less steps than the Sampling Steps slider specifies.

As it states, the closer to zero the more like the original image it will be, by the time it is 1, you are just getting images based on the text prompt you provide. The input image will be ignored. I personally find that 0.4 - 0.6 is the best range for my use cases.

the original image with a few examples of images generated while adjusting the denoising strength

What’s next?

I am looking at ways that I can improve the continuity of the style and look of the characters across pages. If I could seed a book with 3-4 photos of my kids possibly in custom for the theme, could I have all pages make not just a little boy who is an astronaut, but make them all look like the same little boy, etc?

If you want to see any of the completed books, the links below are to the amazon pages for the books. As I am still new to this, the later books, I believe to be better than the first few. I have a couple more that are nearly ready to be published.

Github Organization Repositories by Ruby Version

2022-12-28T00:00:00+00:00

Every year as Ruby releases a new version at Christmas, and many folks adjust their supported versions of Ruby. Where I work adjusting the supported versions of Ruby, deprecating older versions. This means folks have a window of time to find and update the offending older gems and microservices that would no longer be supported. This primarily relies on teams that have ownership of apps following the guidance and eventually folks nudging teams before we deprecate the Docker container that enables deployment of those versions.

I thought it would be nice to pull this out as a burndown list or a chart that can show all the repositories and group them by the Ruby version. This isn’t something I could figure out how to easily do via GitHub code search, but with a bit of work, I could get the data I needed via Github’s GraphQL API. Let’s take a look at how to do that and how we can visualize it.

Displaying Repositories Grouped By Ruby Version

This is what is looks like when you have a valid github personal access token (PAT) added to the notebook.

You can also filter in to find only repos matching a specific Ruby version.

Querying Organization Repositories including Ruby Version

This does rely on the fact that our company generally requires a .ruby-version, but the same approach could be adjusted to pull the version out of a Dockerfile, CI file, or deployment scripts.

The basic Github GraphQL Query is below, note the object(expression: "main:.ruby-version") section:

async function get_repos(lang, first, after) {
  let lang_filter_line = " ";
  if (lang !== "*") {
    lang_filter_line = ` language:${lang} `;
  }
  let query_line = `query: "org:${githubOrganization} ${lang_filter_line} archived:false sort:updated-desc", type: REPOSITORY, first: ${first},`;
  if (after !== null) {
    query_line += ` after: "${after}"`;
  }

  return await github4`query {
  search(${query_line}) {
    repositoryCount
    pageInfo {
        endCursor
        hasNextPage
    }
    edges {
      node {
        ... on Repository {
          name
          nameWithOwner
          collaborators(first: 20) {
            nodes {
              name
            }
          }
          repositoryTopics(first: 10) {
            nodes {
              topic {
                name
              }
            }
          }
          stargazers {
            totalCount
          }
          object(expression: "main:.ruby-version") {
            ... on Blob {
              text
            }
          }
        }
      }
    }
  }
}`;
}

There are several functions to pull things like the organization’s teams, tags, etc. The basic pattern can be extended to find gems, deprecated projects, projects without a team, projects with invalid codeowners files, or more. Once you have data for a file like the .ruby-versions file, you can process it to make it friendly for display, this is the snippet I use to process each repository.

  node.ruby_version =
    node?.object?.text?.trim()?.replace("ruby-", "") || "unknown";

I have been using Observable for visualizing and sharing things like that recently. I have a number of internal company notebooks, and shared out A Observable notebook displaying Trends on Ruby.Social a few days ago.

You can clone the notebook from here yourself, to make a private version for your organization.

Building Books with AI

2022-12-20T00:00:00+00:00

A Brief Note on Controversial AI Images

I know there is some debate about AI Art and copyright. I am not going to get into that a lot. The current models don’t offer enough protections for Artists. Regardless of where we are with the models we have right now. I believe AI art which is considered fully responsible and reasonable use, will be reached. The full details and depth of that issue are outside the scope of this post.

Creating Childrens Books with AI

I was playing around with some of the newer AI tools. Originally, using GPT3 to write a book with my son. Then I started using DALL·E 2 and Stable Diffusion to generate images to illustrate the book. As the holidays approached, I wrote another for the kids cousins. This year grandparents, cousins, and some aunts and uncles will receive printed copies of books my kids helps write and illustrate with the help of AI. It has been a fun a rewarding project, and is easy to do. It is wild that you can work with a 3 year old to build a book they will feel invested in and proud of.

Outline of the Steps

Find a topic the kid is interested in. I generally took a favorite classic fairy tale story
Generate some of the pages with GPT3, allowing the child to accept or reject pages or add in details of interest
- for example, My son wanted to become a squirrel and feature squirrels in the book
- My daughter wanted to be dressed as a puppy and have pigs in the book
Once you have a good story, move on to illustrations. I created one image per page.
- I often tried the exact text of the page to make an image but found it needed adjusting
- I most often used my prompt, digital art to get art-like images as opposed to anything looking like photos, 3d, or the like
Build out the format in a Google Doc or word doc, print it out, and read it to the kids, taking edit notes
Finalize the book
- make final edits
- decide on the final formats, which likely have requirements
- I used Amazon Kindle Direct Publishing: Self Publishing
- This requires specific margins, 24 pages of content, making a cover, etc

Unique Personalized Gifts

It is fun to make these with the kids or generate a bunch of images myself when I have the time to review them with the kids later. Writing short stories and inserting friends, family, inside jokes, and funny family context is amusing, and the result is unique. After making a few more of these, I am sure I will have the process and flow down well enough to make a genuinely personalized book for friends in a bit of time. Publishing several particular children’s books isn’t something I really could have imagined doing before experimenting with these tools. Now I could see all sorts of interesting mixed media creative experiments I could build with my kids.

What’s Next?

I am looking at moving to really put us into the stories and have started to work with Stable Diffusions Img2Img to generate images that the kids could act out or help design. This way the kiddos can get even more involved with some of the creative aspects and make the book even more special and unique to themselves.

Thoughts on Software Complexity

2022-11-23T00:00:00+00:00

NOTE: this was originally a mastodon thread, but feels like small linked blog posts are a better format

As there seems to be a tech crunch going on, several companies are responding in different ways, including layoffs which increase the maintenance cost per developer… This naturally leads to companies looking at re-orgs, shutting down services, and current maintenance costs.

The constant reshuffling of engineers, I find, is universally disliked, and companies seem to be pretty bad at it, although there seem to be some growing trends to try to do it better.

Current popular re-org strategy around team topologies.
A post where the BBC covers how it applied team topologies and how it went.

Pair the above re-org reading with a good rebuttal with the structure and process fallacy. Which points out that without fixing underlying problems a re-org isn’t likely to be much help.

Finally, take a look at the collapse of complex software… Even if you spend time fixing ways of working, org structure, and processes as the above articles suggest if the economic times are forcing a shrinking team to maintain a large and complex system that already has out of control dependencies… The post points to two end states: collapse and rebuilds or build early with sustainable, tried-and-true techniques.

Some choice quotes from the collapse of complex software:

more hierarchies, more bureaucracies, deeper intertwinings of social structures. Early on, this makes sense: each new level of complexity brings rewards, in terms of increased economic output, tax revenue, etc. But at a certain point, the law of diminishing returns sets in, and each new level of complexity brings fewer and fewer net benefits, dwindling down to zero and beyond.

I have definitily seen where additional layers start to bring negative value.

complexity has worked so well for so long, societies are unable to adapt. Even when each new layer of complexity starts to bring zero or even negative returns on investment, people continue trying to do what worked in the past.

I am a architect but often feel powerless to stop the rapidly increasing complexity

Architects are brought in to “fix” the system. They might wheel out a big whiteboard showing a lot of boxes and arrows pointing at other boxes, and inevitably, their solution is… to add more boxes and arrows. Nobody can subtract from the system; everyone just adds.

This seems to be what Musk is trying to do cutting microservices left and right at twitter (I disagree with how he went about it in a very de-humanizing way).

software companies can keep hiring new headcount to manage their existing software (i.e. more engineers to understand more boxes and arrows), but if their labor force is forced to contract, then that same system may become unmaintainable. A rapid and permanent reduction in complexity may be the only long-term solution.

This also seems fairly real from my industry experience.

I think whether software follows the boom-and-bust model, or a more sustainable model, will depend on the economic pressures of the organization that is producing the software. A software company that values growth at all cost, like the Romans eagerly gobbling up more and more of Gaul, will likely fall into the “add-complexity-and-collapse” cycle.

How to reduce complexity?

That is likely to big of a topic for this post, but I can through in a bonus link for where to start… Take more careful consideration of your dependencies. If you want to help organization reduce maintenance costs over time, cull your dependencies. Which I generally agree with. Also, make sure you consider your own internal libraries and services as dependencies.

Gardyn 2.0 Review

2022-11-10T00:00:00+00:00

Gardyn Review

Overall I am happy with the Gardyn indoor hydroponic growing system. I am partially through season two of growing with my Gardyn 2.0 unit and wanted to write up a more detailed review. I feel like the Gardyn has struck a pretty good balance on a DIY vs all-included system. I wish it were a bit more DIY, and I shy away from the yearly subscriptions, which I will explain later. The system grows a good amount of plants, works well year-round, is relatively clean, and, honestly, is very pleasing to look at. I almost wish I had a better space to keep it more in view (My setup is in the back of the unfinished basement). I researched several options and I am happy with where I landed. Suppose you want a more DIY experience and control to expand and modify the system over time. In that case, this isn’t the right setup for you, but if you want a super easy-to-maintain garden supplying you with fresh herbs, lettuces, and small veggies, you won’t be disappointed.

I am sharing out my gardyn affiliate link and if anyone wants to buy one, feel free to use it, or feel free to ignore it. It can save you a bit on a purchase, and I can get a few months of subscription for free.

Pros and Cons

At a glance, these are the points that stand out as I reflect on my experience using the Gardyn 2.0 over the last 18 months.

Pro:

easy to maintain
fresh greens all the time
holds up great with travel. We have left the gardyn for a week at a time, often and without issue.
I love having a new, one-month-old, and 2-month-old variety of my favorite plants
The cameras are kind of silly and fun to check in on plants while traveling
Access to replacement parts, extra food, plants, etc, all via the store
Replacement of failed ycubes is nice and easy
fun activity with the kids
- they love to harvest and eat lettuce right as it grows
- planting and helping with the baby plant nursery is always fun
- they help with the watering by refilling the tank
Easy to DIY hack:
- I often make my own ycubes
- I have extra grow lights near it to help some plants grow and to continually prep nursery
- I surrounded with Highly Reflective Mylar Films, 82x 47Inch, Metallized Foil Covering Sheet, to increase light efficiency
- one can add a watering line or extended tank

Con:

harder to clean and needs more frequent cleaning than I expected
upfront cost is still pretty high
The subscription model
- it is always trying to push memberships
- The membership isn’t beneficial (the AI reminders aren’t worth a monthly fee)
- you definitely don’t need 10 ycubes a month once you have built up a good collection of plants
- it is just far too expensive

Gardyn Tips

Some tips based on how I use my gardyn.

How I use the Gardyn

I use my gardyn for about 9 months a year. I have too much outdoor gardening going on in the summer months and don’t really need or want to deal with the gardyn for those peak summer months. I really enjoy doting on my basement garden in the winter. I enjoy learning new salads and recipes to help cook up and eat through the abundance of whatever I am growing any given month. I also time my monthly subscription having a few months at the start of each new growing season to help me kick off the next growing year with a fun collection of plants before going more fully DIY.

DIY Pods with Purchased Seeds

I bought cheap seed packs, and standard grow cubes, and I recycle the plastic part of the ycubes. This way, I can make my own cubes avoiding the $5 per cube cost for non-members. I have had great success getting basically any lettuce and many herbs growing with this DIY method.

Plant Cycling

I try to plant a few basils every few weeks when I first get the garden going… That way, I always have mature basil plants and some younger, more baby plants at any give time. I can grab a few leaves or if one is starting to bolt, I can cut the whole thing down and make pesto. This same technic works well for things that bolt, like cilantro as well.

Plants and Seeds

I am happy with the selection of plants that Gardyn sells

Liked
- Celery (it doesn’t go big… but the greens are great in soups, stalks in stir fry, and to add crunch to a salad)
- All the Basils, I think that I go 80% Genovese 20% Thai basil. This mix is what makes my pesto so good
- All the lettuces, the Swiss Chard is exceptionally colorful and fun
- All the Kales
- Most Herbs, I particularly like Chives, Cilantro, Thyme, Oregano, Parsley, and Rosemary
OK
- Cherry Tomatoes, it amuses the kids, and they eat them right off the vine, but they never get big or sweet
- Peppers, they are small but are still lovely to have around to throw in a salad or recipe to add a few fresh veggies
- Mint, it grows fine. It even flourishes, but I don’t know what to do with it.
Disliked
- Snap Peas
- Mini Strawberries also grow very large and basically produced very little… The kids did kind of like them though
- Green Beans, any of the beans… They grow huge and drop leaves all over the floor… I got maybe 10 beans
- Eggplants… Fine, I’m not too fond of eggplant, but my wife and kids do… It never produced anything one could eat)
- Flowers, Can’t eat them, not interested ;)

What Does and Doesn’t Work?

The results for different folks will vary given the natural climate (sunlight, humidity, heat). I have had great luck with all the greens and done OK with a few smaller peppers. I have generally been disappointed or unimpressed with most of the vegetables. The pea pods/beans end up growing and shedding a lot of leaves and becoming hard to grow anything else near, but they barely produce any vegetables. The baby strawberries were cute but so small and infrequent that it was just more of a game for my kids to find and eat them and then complain they were too sour. The cherry tomatoes are similarly too small and sour, so my kids aren’t that into them. Other than all the lettuces that grow great, many of the herbs were a hit as well. I loved having thyme, cilantro, chives, and basil to add to any fresh recipes. Mint grows well, but I never really know what to do with it.

A timeline or usage

setup with free month and initial first 30 cubes
ending membership with many pods, I find I can grow and harvest most far longer than they indicated
buying pods at non-member cost
cleanings
learning how to recycle and reuse pods
yearly shutdown during summer
yearly deep clean… It was really worth taking the whole thing apart and deep-cleaning EVERYTHING
restart monthly membership to get new cubes
restarting Gardyn nursery in early fall
as I wrap up my final outdoor harvests, I already have a bounty of greens in the basement to lift my spirits.

Final Thoughts

I enjoy my odd basement gardening hobby and all the home cooking it inspires. A large photo gallery of my gardyn in action hopefully gives a good idea of what you can expect if you decide you want some more plants in your life.

Ownership in an age of microservices

2022-08-27T00:00:00+00:00

This is the second post in a series about organization structures, teams and ownership, and application health.

Ownership challenges abound in microservices

While there are many great resources out there that discuss microservices, where they may or may not be a good fit, and how some companies have decided to move away from them, and back to a monolith, in this post I am going to focus on some of the unique challenges that microservices present with regard to ownership. Ownership is frequently called out as a common problem with microservices, and both Uber, working towards macro-services with a Domain-Oriented Microservice Architecture, and AirBnb’s journey towards macro-service Domain aggregation are aimed at improving the clarity of ownership of domains of data and the teams and services that support them.

Microservices and evolving models of ownership

There are many different models of ownership, and one of the reasons microservices took off was to avoid the “tragedy of the commons” problem that often occurs in large shared codebases. We break the systems apart in an attempt to avoid shared codebases that tend to degrade over time. As companies moved to many smaller systems, a number of different ownership models emerged:

Single person owner
Orphan codebase, due to attrition or team re-assignment
Modular Monolith, shared ownership, whether clear or unclear
Team Ownership (my preferred)
Workgroup/Guild/Steering Committee Ownership

With these came some more modern code collaboration patterns, such as

Internal tickets
Internal open-source (my preferred)
Trusted outsider
Tour of duty
Embedded expert

These are all explained very well in a post patterns of cross-team collaboration. I personally like team ownership, acting as stewards of the long term vision and domain of services, and influencing changes via an internal open-source model. The team, in this model, plays a similar role to that of the primary contributors in an open source project.

Strong ownership takes time

In a decomposed, microservices ecosystem I personally believe that having a team with a long history of knowledge-building and maintaining an app will lead to better outcomes. The institutional knowledge of operating the service feeds back into the development practices of the team. Over time, the team learns how to mitigate risk, anticipate needs, and more quickly and flexibly change and evolve the system. Strong team ownership also helps with communication, especially during incident response as everyone knows which team owns a given app, which adds confidence that the team also has deep knowledge of the system and will be able to support it. A stable relationship between business domain, services, and team brings clarity and shared knowledge of where responsibilities and vision for the domain belongs. In my last post, I will discuss how reorgs and diverging team names across systems causes organizational drift and confusion.

Who is responsible for the Distributed Monolith?

One issue that comes up in microservices is the emergence of a distributed monolith. As microservices have been around for a fairly long time at this point (in tech life cycles that is) there are places that have “legacy” microservice systems, perhaps better described as a distributed monolith. As a company first splits up a legacy monolith, they often make many learning mistakes, like setting up poor system boundaries which leads to distributed transactions, or taking shortcuts, like giving multiple applications read/write access to a single shared database (hello shared DB, my old friend). The early extracted set of services often have some “winners” that prove the value of a well-defined and executed extraction, as well as some less than ideal systems that become part of the distributed monolith, the remains of the legacy monolith that folks have failed to finish extracting. At some point, companies need to refactor, delete, and reduce the maintenance of microservices the same way companies end up building strategies to unravel or modernize legacy monoliths, via Modular Monolith and other techniques.

The problem with poorly extracted and tightly coupled services is that they don’t have clear domain boundaries, which makes it difficult to build strong ownership and knowledge around. These services end up with the same tragedy of the commons issues of a large, shared code base.

As part of any thinking about reorgs, considering these systems is key. The new structure should have clear domain boundaries, and take future ownership of these systems into account. A steering group, architects, or your senior ICs can help think through high-friction boundaries and make recommendations. Often, I see companies try to reorg while only thinking about the new work, as opposed to taking into account these hard to maintain poorly understood systems. This creates a lot of technical drag on the team(s) trying to own them, as well as on any team that owns a feature that is yet to be extracted from these systems. If the reorg doesn’t take this into account, it might make the situation worse by trying to brush these problem areas under a rug.

Reorgs and technology investments to help address confusing ownership at Airbnb

Reorgs impact clear ownership

At the same time, companies are also very keen on reorgs, in part due to the recent popularity of the inverse conway maneuver. This attempts to use the fact that with Conway’s law, particularly in microservices, your org structure ends up designing your architecture, at least in terms of system boundaries. The Team Topologies book about this is very compelling, but I think many organizations attempt it without fully understanding the domains and systems that teams currently own. Without a process to understand and purposefully move applications to proper owning teams, the reorg can end up causing more chaos and confusion, increasing maintenance costs, and lowering MTTR for critical user journeys, not to mention reducing trust within the org. When done well, a reorg can lead to stronger and better understood domains owned by teams with a coherent set of domain and system boundaries. This leads to strong ownership and deeper domain knowledge within the team, enabling them to be more productive and responsive to the needs of the business.

When the company needs to freeze hiring or contract, how are legacy systems assigned ownership and supported. Reorging when a company is growing leads to more positive ownership opportunities than reorging when a company is at a stable size or shrinking. How do we set up ownership so that we can withstand ups and downs. –Erica Tripp

As leadership sees some of the issues falling out of the current org structures, they often will reach for the reorg, but reorgs themselves don’t change the ways of working, and often can make maintaining legacy systems harder if you move folks away from areas they have built domain expertise.

changing structure and process without addressing leadership, cultural and ways of working issues, usually exacerbates existing problems rather than improving performance. –The Structure and Process Fallacy

In terms of clear ownership in microservices some even propose that, “Ideally each team has only one service”. If you are a growing company that has been evolving microservices, I doubt this is the situation. After a number of reorgs, and folks coming and going, a team’s responsibilities more often look like a giant group of only semi-related services and libraries.

A overloaded but not entirely uncommon diagram showing a teams repo ownership

If you end up having teams with ownership responsibilities as shown above, with more services than team members, it is unlikely there is strong ownership with deep understanding of all the systems involved.

What do we need to know about our teams?

We need to better understand the reality of what our teams are working on and if they are supported to work on the goals we set out for them. If the team’s goals align with the systems they have, if they are set up for success or are in a declining spiral of more maintenance than they can healthily support while also delivering new value. There is a lot we should know about how our teams currently look, so that we can re-balance in ways that ensure all teams are healthier and more effective. While we need to also consider domains, specific systems, and team’s charters to re-define an org structure, there are a lot of things that should be considered about the team health directly.

Is the team overloaded?
- Do they maintain too many repos for the number of folks on the team?
- Do they have too many weeks on-call per developer?
- Are they supporting legacy systems with high maintenance burdens, or all newer and actively supported systems?
- Are the systems they support well bound to a coherent set of domains?
- Are the apps they currently support healthy? (I will dig in more into what is a healthy app in a future post).
Is the team well rounded?
- Does the team have some tenured folks that can navigate the org and have historical context?
- Is the team a good mix of Sr / Jr folks, for the type or work?
- Is the manager a good fit for the team (A newish manager with a Jr and non-tenured team, leaves everyone trying to learn and navigate at the same time)
How is attrition impacting the team?
- Teams with higher attrition than others are likely significantly impaired, consider what really happens when someone leaves a team

What do we need to know about our repos and apps?

We can’t just talk about teams without looking at the details. At some point we want to also look at all the repos they own, what they most actively work on, and the health of those systems. Especially after several rounds of reorgs, teams often end up with some things that don’t fit well into their charter, but responsibility for a system has to end up somewhere. It is best to at least make it clear where the responsibility ends up, even if it immediately requires some discussion about how the legacy load might not be healthy, so that it can be addressed. I will follow up on this question in a future post, digging into how to think about the health of repos and the value and risk they create for companies.

Strong ownership drives quality

There are many other posts talking about the impacts on ownership of microservices. I think there are still a lot of good questions on how best to handle it. I am mostly trying to make the case that thinking through the domain boundaries is not enough, one has to think of what it takes to have deep knowledge and strong ownership of a domain. It takes having a healthy team that isn’t spread too thin. Ideally they have a cohesive set of services and responsibilities that form a common domain. If you move teams into what seems like better domains as part of a reorg but don’t look at or address the other responsibilities a team may already have, strong ownership and the attempts at more clarity of domains will not succeed.

Operations vs Ownership vs HR Organizational Structures

2022-08-22T00:00:00+00:00

Organizational structure drift causes issues

When your HR/Management org structure starts to drift from your operational and project development structure. The organization’s communication, shared understanding, system ownership, and incident response times suffer.

Communication: meetings references teams that no longer exist, in terms of ownership of runtime systems and deliverables. Folks that should be in the room aren’t invited.
Shared Understanding: If folks in different roles are referring to teams with different names which sometimes reference slightly different groups of people, understanding of who is working on which systems will become blurry.
system ownership: running operations for a system and not having input into its long term ownership in direction leads to increased pages and incidents with no vision for a stable and coherent system.
incident response: folks see incidents and can’t map the product feature and ownership to the system, team, and pager on-call / at fault to follow up on investigations

How org structure drift happens

If org charts, managers, and teams are reorged a few times eventually a newly formed team can inherit some legacy systems as it fits their newly defined mission. These inherited systems somehow have moved away from the original authors or recent operators. One day the company will find out that a critical system seems to be owned by a team totally unfamiliar with it, often during an incident. This poor outcome does clarify the issue of re-orgs that don’t reflect operation and code ownership responsibilities.

The tale of the teams from before…

In this tale, an incident occurs with various services having problems, this might be figured out by an outside team that leverages the service that has started to fail. It becomes clear this upstream service needs attention and that no one is clear who is responsible for the system… It hasn’t had updates for months, no deploys for weeks, the code owners file references a team that was reorged out of existence. During the incident the downstream team can even page that service’s owner to find that a responder knows nothing about the service and didn’t know it was assigned to their team… or that the service was inherited and only one person really knows anything about it and they are on vacation.

At this point during an incident, possibly in the middle of the night, you have folks with very little context about a system making decisions to “fix the service without safety” or “wake up people who likely don’t know how to help”. The failure of the moment must get resolved, and good people will dig in and figure it out, but during any good post-mortem (incident review). It should come up that there wasn’t clear ownership of that system.

it wasn’t clear which product team owned the service
- or that team no longer actively worked on it, overlooked “legacy” system that is actually critical
a team clearly owned it but knew nothing about it
- this was inherited after a reorg… none of the original engineers originally involved are involved anymore…
- The new team have spent only minimum maintenance efforts to keep it running.
a eng team still clearly owned the service, but has been trying to deprecate it for years without enough support to make significant progress
it actually isn’t clear who owns that any more… The team that owned it was re-orged away and it wasn’t noticed there is now a pager rotation with no people remaining.

If you have been around the block a few times you have likely seen unhealthy system maintenance before. It might happen slowly over time as teams come and go, without paying attention organizational structure drift has started to result in increased risk, increased burnout, lowered productivity.

What does organizational drift look like?

Depending on the current situationm needs folks end up having to try to use different tools and shared-language to find the right folks to help solve the problem. Below is some fake/scrubbed data based on Company Org charts mapping to apps, the PagerDuty teams mapped to apps, and Github teams mapped to apps (using CODEOWNERS files)

Company Org Chart	PagerDuty Teams	Github Teams

diagrams showing the teams in terms of org charts, operations (PagerDuty), and code owners (Github)

Just from a visual check these diagrams diverge. When one starts to talk about the differences, various organizational issues become extremely obvious. From teams with far too many apps and responsibilities, to teams that simply do not exist anymore in terms of the org chart, but have sets of employees maintaining legacy systems in terms of code ownership or operational support.

These diagrams don’t even cover that we have Eng Org team charts which don’t 1:1 align to the Product Org charts. If there is an incident occurring for say a 3rd party payment in an international market, that hasn’t triggered pager alerts… Where would you look for support? The International Expansion Engineering team? The Product person in the chart of payment flows? The Engineering Payments team? What if the payment issues are only for a certain product feature? Even if you know the specific application that has the code in it, do you reach out to the operation support team or the code owners of the system?

How many different ways do folks try to interact with a team?

The value in having teams with consistent naming across multiple views makes it easier to find the right team whether it is for customer support, an active incident, or possibly just project collaboration or questions. There are many lenses where clear lines to or from teams can be helpful.

customer relations support: user voice queue
pagers / on-call: pager-duty
code owners: github
metric owners: Datadog, can we map to services or teams?
real-time discussions: Slack, which channel for the team? or a project channel?

How to fix it…

Everyone wants a data driven business and often can’t even get their organizational data to align across systems. Having a source of truth and being able to map it to any and all different use cases as views of the data is a powerful idea. Management should try to keep those views of teams as close as possible and have good reasons for when they drift (migrating from a out-dated to new view of responsibilities as an example). Companies need to make it very easy to find and look up the team organizational information and fix discrepancies.

The fix is aligning on a source of truth and restructuring the other mappings to follow. You can start to enforce all other systems to map back to the source off truth in order of importance. Do this in code or an automated process, flag mis-matches, do not only rely on humans or you be back at it again as it quickly falls out of sync.

When a company has shared views of what teams exist, who is on those teams, and what they are working on we can start to use data to look at ensuring product and system goals have appropriate staffing to support the goals, and we can see and respond to changes in the organization such as sudden unexpected resignations putting critical systems at risk. I will talk more about how we can look at team and service health in future postings.

Fun with Rails Query Tracing

2022-06-27T00:00:00+00:00

Finding the Origins of Database Queries

There are a number of reasons and times when trying to find the specific origin of a DB query comes up. Most often in legacy systems which have spiraled into extremely complex systems. A common cause is a shared database and trying to figure out which application is causing some specific query. Even in a single application, trying to tease out DB queries that have embedded themselves deep within a shared library. Whatever the cause, trying to find the source of a DB query comes up as a fairly common code detective task from time to time. Let’s look at a few approaches to tracking down the query.

Adding Query Comments

A really helpful tool for Rails is Marginalia, which can append comments about the source of a query to the query itself as a comment. This lets one leverage traditional DB tools like slow query logs, etc, and cross-reference from the query back to the code source. The output would look something like the below, depending on the tool you used to look at the queries.

NOTE: A new Ruby query commenter has recently come up as an option. Implementing an open standard that integrates with open tracing and can join with trace-ids to other observability data. It is an implementation of Google’s SqlCommenter standard, which has a Rails Implementation which extends Marginalia with additional information.

my-service 	SELECT COUNT(*) FROM some_table WHERE some_table.some_reference_id = ??? /*application:my-service,controller:endpoint_a,action:show*/ 	1,181
my-service 	SELECT some_table.* FROM some_table WHERE some_table.id = ??? LIMIT ??? /*application:my-service,controller:endpoint_b,action:create*/ 	3,227

Some tools strip the comments or need to be configured to keep them. You can also set up your Rails logs to output this as well, but it can be really verbose. I leave it off by default but can toggle it on when debugging something specific.

# add in config/initializers/marginalia.rb
Marginalia.application_name = "my-service"

if ENV["ENABLE_AR_LOGGING"]
  ActiveRecord::Base.logger = config.logger
  ActiveRecord::Base.logger.level = Logger::DEBUG
end

DD Database APM

If you are already using an observability tool, many offer rich support for databases these days. For example, Datadog now offers database monitoring. This can be a really powerful tool, and help to understand and drive the deprecation of tables over time. A few quick notes

If you have multiple applications connecting to a shared DB, ensure a unique DB Username so that datadog can attribute queries to specific applications
DD, unfortunately, strips query comments, it would be really nice if they kept the query comment in the DB Monitoring both for query metrics and samples.
query_metrics - understand query load, speed, etc especially a historical view
query_samples - all the details around specific queries
The APM application traces can help pinpoint where the code initiates the database span.

Fun with Rails LogSubscriber

While the others are great general tooling, what if you need something more specific, or you just want to try to find specific queries by running tests and seeing the call stack? Well, this being Ruby of course we can hack together something for development.

Finding the Query Needle In the Haystack

Looking for call chains causing specific active record queries? Why not log out the call stack info or drop a debugger to inspect the code whenever a particular query is executed. Combining this with a robust test suite should help track down all the offending callers in no time.

Rails or more specifically ActiveRecord/ActiveSupport have all sorts of cool tools. A really useful one is LogSubscriber, it is a part of the Active Support Instrumentation / Notifications ecosystem, which overall is really cool. It can be used to hook into many of the common abstractions in Rails, for example adding custom action controller logging.

In our case, let’s listen to all active record queries, with a log subscriber, and capture any of interest. This is some throw-away code I add when needed, I track down whatever I am looking for and remove it when my work is complete.

# add in spec_helper.rb, or other test setup files
module ActiveRecord
  class LogSubscriber < ActiveSupport::LogSubscriber
    def sql(event)
      # NOTE: I add a global $ignore_query == false && if I need to say ignore all the factories or before/after spec specific queries to help
      # only find callers in application code.
      if /FROM "some_table" WHERE "some_condition"/.match?(event.payload[:sql])
        Rails.logger.info "SQL FOUND #{caller_locations[15...150]}" 
        binding.irb if ENV["QUERY_BINDING"]
      end
    end
  end
end

ActiveRecord::LogSubscriber.attach_to :active_record

Remarkable2 Review

2022-06-25T00:00:00+00:00

I bought a Remarkable2 early after launch (I believe, I was in batch 3). I haven’t really said much about it, in some sense because it doesn’t feel particularly cool, but what is clearly special is that I still use it nearly every day. I have had the device for over a year now and still enjoy it. It isn’t really something to show off like a cool tech toy, but after settling into some basic usage patterns, it is a nearly daily tool for me and has effectively replaced pen and paper, allowing me to digitize and archive my notes when I want. The device isn’t perfect and future iterations hopefully improve the writing experience, but it has after many attempts gotten good enough that I rarely reach for traditional pen and paper. Although, as the company tries to move from selling a device to making profits off a subscription service, I am more likely to look for alternatives that aren’t subscription based.

TL;DR: I have used this almost daily for 18 months, and have hundreds of pages of notes on it, it has become a part of my daily workflow.

History of Digital Notes Attempts

I have been wanting quick digital notes for decades and tried a number of solutions, most of which fell short so quickly, that I never developed any habit of using them on a day-to-day basis… The first attempts might date me and remind folks of a time when many companies were pushing for quick digital notes with palm pilots Graffiti script and making progress on natural digital note conversion for Pocket PCs. In recent years, it feels like there has been less interest in natural writing on digital devices, although Apple, brought some interest back with the Apple Pencil.

Pocket PC, with pen
Palm Pilot (handspring)
Samsung Galaxy, tablet with pen
iPad
live scribe
- This was an interesting one, but the requirement of special paper, and needing to having a bluetooth phone app running and syncing caused to much friction.
Samsung Galaxy S21 Ultra, with pen

Why the Remarkable Works for Me

I basically use the remarkable as an endless stream of notes. I have a primary notebook that I just add pages to the end ever growing. I used to have mutliple notebooks like work, personal, sketching, etc… Even the friction of having to switch from one notebook to another was to much and I would break my habit of writing something down. I then back up and archive that file… If I need something in particular digitized or want to work more on some particular page (say a sketch of a graph), I export it that to a single file. That is kind of it, my primary use case is just replacing the pen and notepad with an endless digital notepad. I love it because it really doesn’t work for much else than pen and paper replacement. It isn’t fancy, and many of the extra features haven’t been to useful to me, for example other than being a neat trick streaming it to my screen over a zoom as a whiteboard isn’t particularly engaging for anyone.

An example of what my list of notes often look like on the remarkable

Alternative Usages

OK, yes from time to time I do use the remarkable for other things, but the only thing that is a daily habit is taking quick notes, most often during meetings to serve as a reminder before moving the notes to better long term and more collaborative storage.

sketching up a graph or chart to think through the design before spending more time on a detailed chart using real data
books, it is much easier for me to buy books on kindle as I am hooked into that store and it requires finding non DRM (or hacking around it) content to read on the remarkable, and the lack of backlight limits usable reading locations/time.

Comics On The Remarkable

Look how great these look, seriously an amazing reading experience.

Black & White Comics Look Gorgeous on the Remarkable

These comics, are very cool… but without color, most folks would prefer comics on an iPad. There are a lot of great older comics that were always black and white, which you can buy and put on the remarkable. I bought msot of mine from Comixology before Amazon bought them. I am not sure if they still support non DRM comic formats that work on the remarkable.

Long Form Articles are Easy on the Eyes

I also read long-form articles on it, during the day with good lighting, I mostly reach tech and blog entries this way to give my eyes a break from the computer screen. I use the highlighter while reading to highlight sections of text to help me remember them or reference them later. I still use my Kindle at night for the backlight and it is simpler to buy content on the kindle, while it takes a bit of effort on the remarkable. For long artciles in the morning with my coffee the Remarkable is a great fit.

Sketching

It is fun to sketch and draw whie watching TV, it isn’t something I do a ton, but it can be a nice way to let the mind wander.

doodling on the Remarkable

A Future Full of Improvements in Digitial Notes

Even though this product has found a place in my daily workflow, it doesn’t mean it has solved the problem… In fact, it still means I am chasing the same dream, now I just have something acceptable enough that I don’t entirely give up on it. A short list of things that need to be improved.

No backlight
Slow initial response (getting out of sleep/power off and into notes is annoying at work I have it set up to basically always be awake) but it just doesn’t work for taking a quick note before bed… I just email myself from my phone
Digital conversion notes to digitized text are still pretty bad, I convert drawings, charts, etc… but otherwise things are just stored/archived as basically PDF drawings.
- I still dream of being able to hand write in the markdown format for show blog posts
Article sync isn’t perfect to send remarkable strips of images and graphics… The PDF view is less easy on the eyes but keeps images as part of the file* the pen capture resolution is still notably less sharp and responsive than a pen… My wife’s cursive which is already hard to read with a good pen and paper is particularly difficult as the remarkable is just not as fine or subtle of the line as it could be.
Not every device should require a service, as with all software the companies Remarkable is now trying to have a monthly/yearly subscription as part of using the device. I am trying to find and remove all subscriptions I can from my life. I don’t feel like I need to pay for the device and for each month I use it. Most of the cloud features are useless to me, I just need it to back up a file.
Color, I really love the e-ink experience but it is still common where I don’t read on the device because articles have pictures, charts, or leverage color… I keep hearing e-ink color is just around the corner
A ridiciously good drawing environment. I want layers, colors, smoothing curve line drawing mode… Like the best of photoshop with e-ink
Higher resolution writing capture, use a really nice pen and sign your name and then compare to your remarkable signature… There is still a way to go
Give me the Web
- Web browser… Yeah, I know e-ink web browsing? Yes, I think it would be great for textually rich sites…
- Email, I love to do email on the remarkable, if instant text digitization worked well

In the end, it is great and I am happy with it, but it is still kind of an expensive toy, I am sure most folks wouldn’t find it worth it and would be better off with a good phone or tablet. I know I have grown accustom to it, and would miss it if it broke and was no longer a tool I could reach for.

example of browsing multiple pages on the Remarkable

NFTs, Art Investing, and Collectibles

2022-05-19T00:00:00+00:00

I am still skeptical of Crypto in general, I do own some coins for fun / investment, but consider it an interesting and very risky investment. As far as NFTs built on top of the blockchain, I looked into it and decided that in its current form and iteration it isn’t compelling to me… I think there are some real concerns around the crypto ecosystem for now, and I am happy to watch and wait. Hearing all the swirl around NFTs and art investments did get me thinking a bit more about how I spend money on Art and where I could perhaps do some more purposeful spending to support art.

Art As An Investment

Does anyone else having these ads follow them around?

NFTs, brought art investment to the software investment market… and it keeps growing! Or so they say… as NFTs collapse back down.

I know there are a number of startups trying to “open up” the art market to collective investment both for traditional art masterpieces and other digital markets. Positioning it as diversification in investments to get money out of the markets, move away from large cash stock piles, etc… It’s all a bit much for me, but with NFTs and these types of non-traditional investments being marketed all over the internet. The ads did make me think is there any art I would like that might be an investment? Should we always be treating art as something that needs to hold or increase value? I hope not, but if I am not really buying any art that isn’t supporting artists, perhaps I should buy more art and I can pretend it is small investments.

I have never really had or held art as an investment or followed the art market, but I had as a kid been more into various collectibles… baseball cards, comics, POGs (yes, I have hundreds of them), and magic the gathering cards. I offloaded most of my collections over time but always held on to some toys, comics, and magic cards… Turns out a few of my magic cards in terms of a percentage growth have beat my index funds! I guess, if I can consider collectibles as pop art, I have some experience and it feels a lot more fun to participate in than NFTs.

Oh wow that box of magic cards in my basement has a handful of dual lands and dozens of other older interesting cards.

I don’t really want to be in a world where all art is an investment, so while it might be fun to buy some pop art/collectibles that could increase in value, I also wanted to look at art that just makes me happy and can help support artists.

Comics

Comics? Seriously? Yeah, why not with NFTs as art, it feels like folks are really looking to get into collectibles again. Along with the current state of blockbuster movies being mostly comics, I feel like the value of comic collectibles going up over time seems likely… Old nerds watch marvel movies nostalgic for our childhood comic days, soon folks will be nostalgic for early comic movies and the comics of the time that inspired them… A number of comic movies are a generation behind like Spawn, Maxx, and others that might due for a movie revisit. Seems like some of the comics around the movie reboots will jump in value whenever that nostalgia hits.

A few comics after I framed them

Theo smiling after we put up Iron Man (current favorite) on his wall

Also, if I am wrong and I probably am, it isn’t a real big investment, and it is cool pop culture art you can hang on your wall, a gift to kids, cousins, and friends. I picked up a number of comics and got them framed for my and some friends’ kids. Instead of paying a ton of money in ETH for a digital NFTs image that at the moment seems mostly like bragging rights, my kiddos have their favorite comic hero up on their wall.

Spiderman before framing

Spiderman comics after framing

Commissioned Art

There are lots of ways to commission custom art, and it isn’t always insanely expensive. After our dog passed, our family had a commissioned piece to remember our dog Spot.

Our dog Spot

After a small family reunion-type vacation, we had an artist take a group photo and turn it into a drawing that we sent to all the families…

Supporting Small Artists

On sites like Etsy, you can find much smaller and more unique art that is possibly a better fit for your tastes than things you will find in a more gallery or commercial art market. You can support smaller more niche artists that have a quirky hobby or just haven’t found a large commercial following. Anyways, it is a nice way to be able to support.

Etsy Art - Many affordable options for commissioned custom art
I even know a developer who has commissioned custom art for her public speaking slides!

Photographers

This is probably the most common way many non-arty folks help support artists… Hiring a photographer. Wedding photos, new baby photos, etc… We have enjoyed the experience and the photos so we have increased this practice by hiring photographers for larger family trips when cousins get together. Major life milestones, etc… It is fun and we get some great photos that end up on walls, Christmas cards, etc.

A family photo during a vacation

Local Art Fairs

While I haven’t attended many of these recently, we are planning to hit up First Friday Art Walks in Denver’s Art District on Santa Fe soon. I remember as a kid visiting Art fairs in Springfield, IL often and collecting little small drawings or paintings intended to be sold to kids for $3-5, these pieces proudly hung on my walls and inspired my own creations. I hope my kids enjoy and remember the experience of finding little pieces of art that speak to them. I even found a favorite artist, whom every year, I would seek out their small photo-realistic pencil drawings of animals in the kid’s art tent.

Pencil drawings, I bought as a kid… I still have a gorilla, dog, and my favorite frog

A painting my parents bought in Springfield and was handed down to me

Original art

There are various places to pick up original art. We like to purchase some art while traveling.

A painting we bought when in Vietnam, shipped back, and then framed

While traveling met a local painter and loved many of her pieces but the color scheme wouldn’t work for our house, they mentioned commissioning pieces with loose guidelines like color themes and taking it from there. We are actively considering commissioning a piece, and it wouldn’t cost more than any of the paintings they already had for sale, which had standard prices based on canvas size.

Interested in just finding a custom piece online? If you have any favorite sites let me know.

SingulArt - reasonable prices for custom paintings (we haven’t purchased from here yet, but are currently considering a piece).

Redis & Sidekiq

2022-04-10T00:00:00+00:00

Redis & Sidekiq

A collection of notes about working with Sidekiq and Redis. A previous post about Ruby & Redis, briefly touched on some things, but I will get into more specifics in this post.

Redis and Background Jobs

A common usage of Redis for Rubyists is for background jobs. Two popular libraries for jobs are Sidekiq and Resque. At this point, I highly recommend Sidekiq over Resque as it is more actively maintained and has more community support around it. I am not going to get into too many specifics of Sidekiq and Resque, but talk a bit more about how they use Redis. There are always some gotchas when working with Redis, ask folks about sometimes an incident that occurred because of a keys, flushall, or flushdb command. Some of these commands are destructive which is always something to be careful with, but they also all have very slow performance characteristics. It is worth noting how some of the calls in Resque and Sidekiq scale with queue depth, which is critical to understand.

One of the most desirable properties in queue or stream based systems is increased efficiency as the queue depth grows. Without that property, slight mode shifts can push the system into the state of perpetually growing backlog. Who here hasn't a seen a graph like this: 1/ pic.twitter.com/opKuB7Y9tA
— Joe Magerramov (@_joemag_) April 4, 2022

A Incident Caused by our Sidekiq/Redis Observability

UPDATE: We no longer think the line below is the culprite, we observe latency growth and decline with queue size, but we are unsure of the cause and unable to reproduce. As seen, in the NOTE above the Sidekiq latency call is O(1+1) and therefor fast and predictable.

We got into trouble when moving from Resque to Sidekiq because our observability instrumentation was frequently making an O(S+N) call (Sidekiq’s queue latency). It wasn’t much of a problem until we had one of our common traffic spikes that results in a brief deep queue depth. Our previous Resque code didn’t have any issues and had some similar instrumentation being sent to Datadog. While our Sidekiq code had been live for days, this behavior where our processing speed decreased with queue depth hadn’t been observed or noticed. The problem came to light when on a weekend (of course) a small spike caused a background job backlog, as we an expected common case. The latency went way up due to our instrumentation and we started processing jobs slower than we enqueued them. This fairly quickly filled our entire Redis leading to OOM errors.

Analytics Monitoring our Recovery

These charts are from after the incident. We moved to a new Redis to get things back up and running during the incident, and after things were under control worked on draining the old full Redis, in a isolated way that couldn’t impact production load. In this graph, you can see as we reduce the queue size the latency of our Redis calls also reduces in step. I included CPU to show how hard we were taxing our Redis, this chart isn’t 1:1 as we were adding and removing workers and making some other tweaks, but the queue size -> latency is a direct correlation.

Code Culprite

NOTE: Update Mike responded that he doesn’t think the latency call is the issue so we are further investigating

As mentioned it wasn’t any of our normal code that was really the problem, it was this line that was part of our instrumentation and observability tooling. Sidekiq::Queue.new(queue_name).latency. As with any incident, there were a ton of other related things, but it is worth noting that this seemingly simple line could have some hidden gotchas or an outsized impact on your Redis performance. As that latency call scales linearly with queue size, it is calling Redis’s Lrange under the hood which is an O(S+N) operation.

Sidekiq / Redis Performance

A colleague @samsm, helped dig into this incident by putting together the queue size -> latency charts above as well as all the helpful tables I am sharing below. These showe how Sidekiq calls translate into their Redis implementation operation details and the operational costs.

Redis / Sidekiq Math

Doing the math on various Sidekiq operations: how much will they impact Redis?

Big O Complexity Notation 101

Big O Notation

Sidekiq Operation Complexity

Additional Sidekiq / Redis Reading

Some additional reading if you want to dig in further on working with Sidekiq and Redis

Sidekiq Best Practices has some useful tips and calls out gotchas
Sidekiq in Practice book, fastest way to get it right the first time if you are just learning Sidekiq
Blog Post on Sidekiq testing
Sidekiq Testing Wiki Entry
Redis Commands Documentation

CDNs

2022-04-04T00:00:00+00:00

I am starting a series of self-reference materials on staples of web application delivery. A place to collect and document my learnings and understanding of various technologies I often reach for when designing systems. The first in this series was about Ruby and Redis While this one will focus in on Content Delivery Networks. These reference pages will be updated over time and evolve as my usage or focus changes.

Content Delivery Network (CDN)

A Content Delivery Network can help serve high traffic or high performance websites as well as offer a number of features.

A content delivery network, or content distribution network (CDN), is a geographically distributed network of proxy servers and their data centers. The goal is to provide high availability and performance by distributing the service spatially relative to end users. CDNs came into existence in the late 1990s as a means for alleviating the performance bottlenecks of the Internet[1][2] as the Internet was starting to become a mission-critical medium for people and enterprises. Since then, CDNs have grown to serve a large portion of the Internet content today… – wikipedia

Minimum of what you want your CDN to be doing for you

Smart routing: last-mile network distribution
Speedy established TLS: speed up your TLS handshakes
DDOS Protection: Cached pages are kind of already protected but many CDNs offer DDOS protection (you can also do this at your load balancer layer)
Serving assets: handling serving assets to avoid having static file load hit dynamic servers
Caching: at least assets, but even better for HTML / API content
Compression: gzip and brotli

CDN Setups

Often you might want different configurations and settings for different purposes and uses of CDNs. For example, it is fairly common to have an asset CDN with long-lived caches for assets, while you might want applications to be able to specify find grained caching headers for HTML and API content.

Why You Should have A CDN

It can help even if you aren’t quite ready to fully leverage it. CDNs have a lot of hidden advantages, and with some early setup can be very quickly utilized to handle insane amounts of traffic. Handling loads of largest advertising campaigns for the initial experience even if all those new signups have to be pushed into a queue to handle when you can. As this tweet points out even after folks think of all the obvious reasons to have CDNs, there are often many other clever ways to use CDNs.

There are a bunch of benefits to putting a CDN on top of your web application, such as reduced TTFB, caching, and DDoS protection. But there's also a huge benefit that people don't talk about: No user-side DNS caching. You can switch origins within minutes, not days/weeks/months.
— Jack Ellis (@JackEllis) April 3, 2022

A Start Up Story

My first startup got a viral link taking it down and by the time we got page caching on our rails server a few hours had passed and we lost the bump… For this first bit of traction, a CDN could have made all the difference. The content would have been easy to protect with the single viral page behind the CDN and appropriate content headers. That startup never succeeded, but early on much time was wasted trying to improve app layer caching and performance where a CDN could have been a big help.

CDN Gotchas

While CDNs are great, nothing is free and every abstraction adds some complexity to your system. Understanding the values it can provide and also understanding some of the gotchas cna help your team decide if it is the right decision for the system.

Accidentally caching private pages/data!
caches including things like a set cookie (sessions, return_to, etc) that should be user-specific
CSRF, many traditional protections like the built-in Rails CSRF don’t work well with cached pages
difficulties with various security implementations like content-security-policy nonce implementations
Having different rules for cache keys and what information is sent to the origin
you may only cache on specific cookies and headers…but you might want your origin to receive all headers to help with debugging or other info, this is an additional mental load when understanding a request-response cycle.

Understand the reliability risk and how to mitigate it

I have written previously about request depth and availabiltiy. A CDN is an additional layer in the request depth of your application stack… It also only has a 99.9% AWS SLA, as the entry point to your systems which means that is the upper limit on your overall SLA pretending for a moment everything else was 100% reliable. Now, AWS and most CDNs actually have far better real-world uptime and success rates during normal operations, but all of the major CDNs have had notable major outages. Including Cloudfront having a major outage the day before thanksgiving. There are a few protective measures one can take a team really needs to ensure higher availability of their site, neither suggestion is cheap.

* A team can implement and support High Availability Origin Failover as a protective layer within Cloudfront, to protect against origin level failures. * A team can implement multi-CDN DNS failover This can be done in an automated or manual fashion depending on complexity and cost concerns.

CDNs vs Load Balancers

These days CDNs can do a lot of what load balancers do, by routing different types of requests to different back-end “origin” servers. Geo target routing to support locale and nearest reach servers… While this overlaps with load balancers, I generally end up with a configuration where I have both a CDN and load balancers in place. My setup looks often looks like so: web browser -> CDN -> ALB -> Application Servers.

CDNs Enhancing Request Payloads

As part of the ability to route different requests, CDNs now often handle basic IP Address Geolocation, allowing one to route requests from different countries, language support, city, postal code, or more to different origins. Even if your application doesn’t use different origins, the Geolocation information is often extremely useful. You can avoid additional network calls or 3rd party integrations to leverage the CDN’s built-in geolocation support. For example, AWS Cloudfront can provide Geolocation info on all requests to the CDN to impact caching logic as well as header hints to your application servers. For example, it is helpful to use this data to detect if a user might be accessing your site on the wrong domain given the country they are in. Example additional header data.

CloudFront-Viewer-Country-Name: United States
CloudFront-Viewer-Country-Region: MI
CloudFront-Viewer-Country-Region-Name: Michigan
CloudFront-Viewer-City: Ann Arbor
CloudFront-Viewer-Postal-Code: 48105
CloudFront-Viewer-Time-Zone: America/Detroit
CloudFront-Viewer-Latitude: 42.30680
CloudFront-Viewer-Longitude: -83.70590
CloudFront-Viewer-Metro-Code: 505

Note: inside a Ruby app, Rails/Rack will format the headers adding in HTTP and upcasing, so you could access this data in a typical Rails app like so request.headers["HTTP_CLOUDFRONT_VIEWER_CITY"].

CDNs Are Evolving

This post covers more traditional CDNs and features common consumer-facing sites should consider leveraging. CDNs are now moving into the realm of cloud infrastructure with Lambda@Edge and other CDNs allowing code (most commonly Javascript) to be deployed to the CDN network. The features and options supported when looking at having fully supported runtime code at the edge opens up new architecture and application stack options. I am looking at FAAS and edge deployed code, but haven’t leveraged it in any significant production environment yet. It is definitely a space to keep an eye on that promises to simplify global app distribution while maintaining extremely performant consumer experiences.

Additional CDN Links

Optimize Rails App Performance With Rails + Amazon CloudFront
CDNs aren’t just for caching
CDN comic
Understanding HTTP/S, CDNs and Edge Proxies
Optimizing high availability with CloudFront origin failover
Cloudfront SLA - 99.9%
Cloudfront 2 hour outage
When a CDN goes down
- The best defense for this is DNS fallback routing around the CDN… If you configure a failover routing policy. For example on AWS, Route53 would check the health of your Cloudfront distribution and if it’s not healthy then the traffic would failover directly to your load balancer.

Ruby & Redis

2022-03-26T00:00:00+00:00

Ruby & Redis

A collection of notes and some tips about using Redis.

Redis Setup

Redis is super easy to setup, and in dev mode often just works right out of the box, but as you leverage and scale it inproduction, you might want to think more about it’s setup beyond just setting a default REDIS_URL ENV var. Often a basic Redis for simple product is just setup like so…

Redis.current = Redis.new(url: ENV['REDIS_URL'])

This has some issues:

The usage of Redis.current is deprecated
Above is using all default options like timeout
A shared redis is blocking across threads and can be much slower for highly threaded server configurations, for example Puma.

A better setup adding in configurable options:

Redis.new(
  url: ENV['REDIS_URL'],
  timeout: ENV.fetch("REDIS_TIMEOUT", 1).to_i,
  reconnect_attempts: ENV.fetch("REDIS_RECONNECT_ATTEMPTS", 3).to_i,
  reconnect_delay: ENV.fetch("REDIS_RECONNECT_DELAY", 0.5).to_f,
  reconnect_delay_max: ENV.fetch("REDIS_RECONNECT_DELAY_MAX", 5).to_f
)

If you are wanting to configure a Redis and use it across threads, using a Redis connection pool is recommended.

pool_size = ENV.fetch("RAILS_MAX_THREADS", 10)
redis_pool = ConnectionPool.new(size: pool_size) do
  Redis.new(
    url: ENV['REDIS_URL'],
    timeout: ENV.fetch("REDIS_TIMEOUT", 1).to_i,
    reconnect_attempts: ENV.fetch("REDIS_RECONNECT_ATTEMPTS", 3).to_i,
    reconnect_delay: ENV.fetch("REDIS_RECONNECT_DELAY", 0.5).to_f,
    reconnect_delay_max: ENV.fetch("REDIS_RECONNECT_DELAY_MAX", 5).to_f
  )
end

Although this means when using it you need to grab a pool connection first

# original style, which is deprecated and would block across threads
Redis.current.get("some_key")

# utilizing a pool
redis_pool.with do |conn|
  conn.get("some_key")
end

thx @ericactripp, for sharing the link about connection pools

Redis in Common Libraries

All the above helps when you are working with Redis directly, but often we are configuring common libraries with Redis, how many of them are able to leverage the same kinds of benifits like a connection pool?

sidekiq: If you use sidekiq it already has a connection pool under the hood.
resque: Currently, resque doesn’t support a connection pool… It does have away to avoid re-connecting for each worker by using inherit_socket: true as described as one option to avoid needing a connection pool.
flipper: The current flipper redis, is a single Redis connection, but creating a redis adapter that used a pool, looks like it would be pretty easy.
Rails.cache: Rails redis cache you are likely using a single Redis instance, a shared redis connection, support for connection pools was added, but folks need to opt into it with some additional options, see the example configuration below.

# common config that won't leverage a redis connection pool
config.cache_store = :redis_cache_store, {
  url: ENV["REDIS_URL"],
}

# by setting the pool side and timeout, you can leverage a connection pool with your Redis
config.cache_store = :redis_cache_store, {
  url: ENV["REDIS_URL"],
  pool_size: 8,
  pool_timeout: 6
}

Investigate Combining Redis Calls

If you have an app that is making many sequential Redis calls, there is a good chance you could make a significant improvement by leveraging Redis pipelining or Mget. I think that that the Flipper codebase is a great way to learn and see various Ruby techniques. It is high quality and has a wide adoption so you can trust it has been put through the paces. If you want to dig into combinging calls, read about the differences between pipeline and mget in terms of code and latency.

Pipeline Syntax Change

As long as we are updating some of our calls, worth being aware of another depracation. “Pipelining commands on a Redis instance is deprecated and will be removed in Redis 5.0.0.”

redis.pipelined do
  redis.get("key")
end

# should be replaced by

redis.pipelined do |pipeline|
  pipeline.get("key")
end

Redis Usage Beyond The Basics

If you are looking to do a bit more than the default Rails.cache capabilities with Redis, you will find it supports a lot of powerful feature, and can along with pipelining be extremely performant. If you are looking to push for as performant as you can, setup hiredis-rb as your connection for redis-rb. It uses C extensions to be as performant as possible. This post goes into some details where direct caching wiht Redis can provide more powerful capabilities than using Rails.cache

Redis CLI

A few useful tips around using Redis and understanding how your application is using Redis.

brew install redis: install redis via homebrew
brew uninstall redis: uninstall redis via homebrew
brew info redis: Get info on currently installed redis
redis-cli ping: Check if redis service is running
redis-cli monitor
redis-cli slowlog get 100

Exciting Things Are Happening with Ruby Redis

A good deal of things will be changing in Redis-rb 5.0, we mentioned Redis.current and the redis.pipelined changes. These changes and others help support a move to a simpler and faster redis-client under the hood.

A move to simplify the redis-rb codebase and drop a mutex looks like it will roll out redis-client, which can significantly speed up some use cases. It looks like sidekiq for example with move to this in the near future.

New prototype Redis driver cuts Sidekiq job processing time from 61 sec to 39 sec. Incredible improvement! https://t.co/ZzIw7koMiq
— Mike Perham 🇺🇦 (@getajobmike) March 23, 2022

Update: Looks like that perf win was a bit to good to be true.

Sadly I have to walk this back. Looks like there was a bug in the new driver and the performance is not significantly different. https://t.co/2o7cgAUX0K
— Mike Perham 🇺🇦 (@getajobmike) March 28, 2022

Micro-Service Request Depth Availability

2022-01-18T00:00:00+00:00

Micro-Service Request Depth Availability

In systems that use micro-services, often the growth and interaction of the services grow organically over time. While it is enabling teams to move quickly and integrate whatever they need it leads to some known bad patterns in terms of micro-service interactions that have serious impacts on availability. This post explains two of micro-service integration anti-patterns calling them “The Mesh” (I prefer Distributed Monolith) and “Services in Depth”. In both, the issue is a single request into your system can fan out to many individual services both in breadth and depth. Most micro-service systems I have seen have a mix of both fan-outs in-breadth and deep lines of service depths in some cases.

Understand the Implecations of Deep Service Call Depth

Let’s consider service depth, as the simpler version of the problem to reason about. When a team is investing going micro-services some breath and depth calls are to be expected but understanding what it means and how to consider the impacts in the designing of the system. Below we will consider that each application has an aggregate request availability of 99.9%.

What does the request success rate look like for a request with 6 micro-service call depth?

> (0.999) * 100
=> 99.9
> (0.999 * 0.999) * 100
=> 99.8001
> (0.999 * 0.999 * 0.999) * 100
=> 99.7002999
> (0.999 * 0.999 * 0.999 * 0.999) * 100
=> 99.6005996001
> (0.999 * 0.999 * 0.999 * 0.999 * 0.999) * 100
=> 99.5009990004999
> (0.999 * 0.999 * 0.999 * 0.999 * 0.999 * 0.999) * 100
=> 99.4014980014994

Assuming each service has an aggregate availability of 99.9%, A service call depth of 6 has a request availability of %99.4

This is likely a lot lower than teams expected. Also, often depending on your infrastructure it is a lot easier to stack up to six network calls than you may think. Also, while I am not covering the impact to latency in this post, understand it will have a large and negative impact on latency for a deep request call stack.

Visualizing The Request Failure Rate

A nice way to think about the combined success rate is by thinking of each network hop as having a small opportunity for failure. These failure threads peel off as requests navigate the micro-service call stack. As the complexity of the network communications increases and the call depth deepens, the likelihood of failure increases as well. This would include things like your load balancer, DBs, App servers, and application caches.

Each network hop is an opportunity for failure, in the above showing 7 failure opportunities

Request Depth Failure Trends

Another way to visualize this is just a simple bar chart showing a decline of expected availability as service depth grows.

Micro-Service Request Availability Calculator

The below calculator will let you quickly estimate your theoretical availability based on the estimated SLA across multiple service calls. Consider each part of your infrastructure (CDNs, load balancers, DBs, Caches) as well as the total services involved in a successful response to a request.

Micro-Service Call Depth Availibility Calculator

AWS SLAs

So far we have been talking about micro-services and their availability, we should consider the services often used to host our application code into the cloud. A very popular cloud for hosting service is AWS, which publishes all AWS SLAs. Let’s look at this from the perspective of a typical AWS application, assuming a single app stack (no micro-services), a pretty standard setup, and assuming the app code has a runtime SLA of 99.9%, the combined math leaves a total theoretical max request success expectation of %99.6. If you are building something that needs extremely high reliability, are you able to keep availability promises?]

You can see as you stack AWS services, regardless of your application stability, request success percentage decreases…

Service	SLA	% Success Math	Request % Success
cloudfront	99.9	0.999	99.9%
ALB	99.99	0.999 * 0.9999	99.89%
ECS	99.99	0.999 * 0.9999 * 0.9999	99.88%
Custom App	99.9	0.999 * 0.9999 * 0.9999 * 0.999	99.78%
Elasticache	99.9	0.999 * 0.9999 * 0.9999 * 0.999 * 0.999	99.68%
RDS (Postgres)	99.95	0.999 * 0.9999 * 0.9999 * 0.999 * 0.999 * 0.9995	99.63%

Mitigations / Considerations

When you realize as the system scales and grows and the number of total microservice dependencies a typical request into your system may have, it is worth thinking about and considering some mitigation strategies. Opposed to pushing towards Five Nines: Chasing The Dream?, embrace failure and resilience, find an acceptable and achievable level of availability for your service. Then invest in mitigation techniques and strategies to deliver a reliable client experience on unreliable internet. A few examples of mitigations are listed below.

Client Side Retries: A good reason to have client-side retries and avoid implementing retries at all levels of the infrastructure (some other special cases may make sense to avoid full round trips). See Google’s SRE book, sections Client-Side Throttling and Deciding to Retry from the Handling Overload chapter.
Be Wary of Circular Graphs: Detect circular graphs, even if this can technically be supported in your infrastructure, it may be best to avoid as a way to force folks to think through more robust and scalable solutions.
Avoid Duplicate Service Calls: This happens when you might have a very common piece of data that a service calls, before you know it all your micro-services call this in high demand service. You might have an initial request fan out to 3 micro-services that all call this common data service under the hood. This often happens for something like user data.
- Consider common data and look at data forwarding, which can early in the request processing add metadata that is sent to all upstream requests. Avoiding all upstream requests from making individual network requests for the data.
Agree on Constraints: Consider alerting on requests that exceed an agreed maximum service call depth or circular call graphs.

Conclusion

There are a lot of benefits of microservices, but I feel like the expectations around reliability and latency are often overlooked when folks move from a larger shared codebase and adopt microservices. The companies are looking for faster deployments and teams that move independently and do not fully grasp that they slowly have turned every method call or DB join into a remote network request with all the failures and performance characteristics that come with it.

Revisiting Front-Ends

2022-01-08T00:00:00+00:00

Revisiting Front-Ends

I have been a ‘full-stack’ developer for a long time. These days depending on where you work and how your org works being full-stack isn’t really viable anymore. Given the growing complexity of both the front-end and back-end end systems, it is more and more required to specialize. That being said, I feel like there are good reasons to understand and think across the front-end boundaries. For example, if you care about user performance how you design backend APIs and deploy front ends can have a massive impact. From fully supporting and leveraging CDNs, pre-fetching, cached API-queries, and more. Anyways, as my front-end skills fell further behind and some exciting changes have been occurring in the front-end world it was a good time to spend a little time refreshing my knowledge and sharpening my tools.

Where I started

I decided to look at a couple of different projects and ways to explore the space. This hasn’t brought me fully up to speed with the amazing front-end folks I work with, but I have learned a lot and enjoyed working in a bit more visual space.

I worked on a side project modernizing its look and design updating Semantic-UI/Fomantic-UI designs.
I picked up, Modern Front-End Development for Rails and worked through some exercises.
I dug into CDNs, caching, and compression options (Brotli)… modernizing our setup at work
I helped automate and setup lighthouse tracking on projects at work, and fixed some low hanging fruit
I played around with hotwire
I picked up Tailwind CSS for a few toy projects
I converted my blog to Tailwind CSS from an old customized Twitter bootstrap theme
I started working on some visualizations in D3.js, diagramming network traffic, failure rates, and org/system structures.
I built a presentation as a Tailwind / D3 microsite vs a slide show

Some of the next things coming up?

I will likely convert over the Semantic-UI/Fomantic-UI site to Tailwind
I am helping support our frontend team on efforts to decouple our Front End deployment from Rails
I will be digging into our custom react design system a bit more at work and porting over a few pages to it* Adding some more visualizations directly to this blog

Tailwind

After trying out a few frameworks including a bit of a deep dive on Semantic-UI/Fomantic-UI. I wasn’t satisfied, the growing buzz around Tailwind pulled me in. I still have a lot to learn and a ways to go, but it is better matching my needs/desires for front-end support than anything has in a long time. As I played with Tailwind, I needed a few projects to drive a bit more real-world usage.

Converting the Blog to Tailwind

This blog you are reading moved from Twitter Bootstrap now supports Purge CSS for the various pages, has Tailwind layouts, templates, and dev support. Nothing too complicated, but I feel it looks much better than it did. Simpler header, more readable font / white space. Dropped most of the sidebar, etc… I had to reformat some of my Markdown and post tags, I wrote a conversion script to reformat my post history.

Tailwind Learning Sources

There are a lot of great resources out there and I wanted to share a few. I have also been using Tailwind on some test Rails projects, so some of the links are more Rails -> Tailwind specific.

Learning & Exploring
- Tailwind Play, this real time configurable exploritory playground for Tailwind is a great way to quickly fool around with ideas and see results. I used this a lot before working on templates and files. You can also share ideas.
- Center elements with Tailwind CSS
- Tailwind breakout grid
- Tailwind preflight, I recommend understanding what preflight does and what it ‘resets’
Rails & Tailwind
Tailwind Blogs / Jekyll (My templates are heavily based on some of the below, but customized a bit)
- Setting up docs with Tailwind CSS & GitHub Pages
- Tailwind Docs
- purpletual-blog-theme (Tailwind & Jekyll starter kit)
- Jekyll Starter Tailwind
- jekyll-atlantic theme - This is the base of my theme

Other Learning Projects

Other than the blog, a few other examples from my recent front-end exploration exploration.

Coverband Web built in Semantic-UI

D3 Network request flow chart

Presentation breaking down org charts, team / system relationships, and network request flows (D3 and Tailwind)

D3 Request Max SLA Calculator (interactive visualization)

Final Thoughts

It is good to revisit and resharpen skills in an area even if you aren’t planning to be an expert. While I don’t really do full-stack work in my daily workflow anymore, I am often heavily influencing our system design and architecture as it relates to microservices, mobile, and the future of our front-ends. I want to ensure I am still looking closely enough to know what questions to ask and understand when folks are sharing ideas and concerns… I want and need to know the landscape, so to speak, including some of the toolchains, pain points, and benefits over older styles of development. A quick bit of focused exploratory work can help one stay fresh while also not slowing down or getting in the way of the experts doing the real front-end work where I work. I am able to be a better and more capable partner in discussions and designs.

As I continue learning more about Tailwind and Visualization tools, please share any good links with me.

Principal / Staff Engineer Resources

2022-01-04T00:00:00+00:00

Principal / Staff Engineer Resources

A friend who recently was promoted to a Staff Engineer, and wanted to learn a bit more about the role asked me if I had any good resources. I put together a small set of resources that could be helpful when aspiring or transitioning into Principal or Staff level engineering roles. I figured it would be good to share out with others who might be interested. I broke up the resources into a few different categories, so folks can dig in where relevant. It is worth noting that higher-level technical roles have a lot of variance between companies. Folks will find that they can often shape the role or find a company that defines the role in a way that will be interesting to them. Don’t feel like you need to fit into a tightly defined box, long as you are providing high-level value, there is a place for you to grow and apply your technical experience. You will hear of staff engineers who never code, while others with the same title still consider software development a core part of their job responsibilities. Whatever way you want to grow and increase the value of your work there is always more to learn as you expand your technical software career and look to have a broader impact on software development.

Staff and Leadership Resources

A few things actually focus on this narrow niche of technology roles. Overall, these are all excellent resources that I highly recommended. This list includes sources that continually create and add new content.

Staff Eng - easily the best community and set of materials out there for learning more at the top of the technical track.
- a newsletter
- a book, Staff Engineer: Leadership beyond the management track
- a podcast, Staff Eng Podcast
- a list of staff engineer learning resources
Software Lead Weekly - email newsletter with great links of posts being discussed in the community.
LeadDev StaffPlus - LeadDev runs some of the best advanced eng confrences. Staff Plus, is the best conf that touches on all the cross functional aspects that you will be involved with as you advance in a technical career.
- A lot of past material is turned into articles, videos, and other online content.
- See my Lead Dev London Summary to get a better idea for the feel.
CTO Connection - Community, confs, newsletter for leaders. Frequent interviews and videos shared via email updates. This is more management focused then the others. It also has been a bit more hit and miss, and seems to have been less relavent for me lately.

Staff and Leadership Articles

There are many good articles as well, but obviously these are more of one offs.

Keeping Up with Tech in General

As you are shaping technology decisions that can have impacts for years, plan to try to keep up on some of what is happening with the industry in general. Find sources that relate to your field, attend conferences, read newsletters, or listen to podcasts. Figure out the best way for you to low effort keep your ear to the ground. A few resources, I have enjoyed.

The Changelog podcast, excellent discussions with engineers
Leading From Incidents, It Will Never Work in Theory - Short summaries of recent results in empirical software engineering research
Greater Than Code Podcast - A podcast that is more about the who, why, and community… vs the technical how.

Distributed Team Resources

My friend and I both work on remote-first teams. A few resources around remote work, and how to lead tech outside of centralized teams.

First, if you are working remote long term, it is worth getting a good remote audio/video setup
I don’t think top down leadership scales particularly well for software, I think it is even less effective when distributed. Scaling Architecture Conversationally sets out a number of approaches to encourage a more team sourced achitecture with guidance.
Hashicorp: Distributed, Async, and Document Driven - learn from how Hashicorp builds distributed software
Remote, while this company seemed to go off the Rails (pun intended) and loose a lot of the good will they had in the remote community, much of the advice can still be useful.

Books

Although there are lots of great posts, talks, and threads about technology leadership, sometimes nothing can give the big picture and the depth of a book. The shortlist of books below, stand out in my mind as worth the time.

Accelerate - If you want to shape how a software org functions, ensure you are basing it on research backed successes. This book summaries what is actually working best based on research.
The First 90 days - While I highly disagree with a few things in this book, there is enough that is actionable and will help you have an impact as well as understand what is motivating other new leaders, that it is worth reading.
The Manager’s Path - Even if you are looking to stay on the Staff / technical track many parts of this book will help you grow with parts of your job.
Working Backwards - Another book I don’t agree or like all the recommendations, but has enough practical and actionable advice on how Amazon scaled and managed to stay agile with massive growth, that it is worth reading.
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems - Even if you aren’t handling big data (yet), this book will let you understand when and why to reach for distributed systems and tools.
Domain Driven Design - One of the most important things while scaling and hardest to fix later is bad domain modeling, deeply learn about building good data models.

Ruby Specific

I am mostly operating in the Ruby world, and my friend is also at a Ruby-based company, so some Ruby-specific resources. As you grow as a tech leader in some sense you will care less about a specific technology, but there are reasons to keep an ear to the ground for the technologies your business has invested so much in. As I have grown my career, I have found it valuable to stay up to date with things going on in the community. If your company is invested in Ruby it means your hiring, training, operations, and infrastructure costs are all related to Ruby, ensure you are using it at its best.

Ruby Weekly - easiest way to see the most discussed articles in and around the Ruby community
RubyFlow - a more techy stream of Ruby links and project updates
Speedshop - a few great ways to soak in Ruby operational and performance tips Rails performance Blog, Ruby Performance Newsletter
Ruby Central Youtube channel - various confrence talks
Ruby Confreaks Recordings
Rails Security Email List - ensure you see critical security issue announcements
Sustainable Web Development with Ruby on Rails - If you are currently struggling with scaling Rails as the dev team grows, this book should help guide ways to move forward that are less likely to come back to haunt you.

Enjoy the Role

Good luck with the new role and growth. As always there are lots to learn, but there is a growing community out there to find friends and mentors and talk about what you want out of your career. The highly defined ladder is changing and roles are more malleable as we move to more hybrid and distributed ways of working. Feel free to explore and help shape the ways folks can lead in technology, it doesn’t have to be a path to being a manager, director, or VP in all cases anymore. It is an exciting time to be a technical learner and leader.

If you have any good articles, sites, or books please share them with me, as I am always looking to learn more about what others in this area are doing.

Availability, Outages, Reliability, and SLAs

2021-11-23T00:00:00+00:00

TODO:

Grammarly

Availability, Outages, Reliability, and SLAs

As companies mature, eventually downtime moves from speculative loss of revenue or customer growth, to directly impacting revenue and decreasing active customer trust. The tradeoffs of move fast and find market fit changes when a business shifts to driving revenue from a working business plan. This shifts the dynamics and companies working with processes and models that worked during early startup growth often struggle with the shift, understanding where new products and features can still learn and change fast, while more established critical systems require different approaches. In this post, I want to take a look at some of the different and related concerns around building reliable services.

Outages and Uptime

When talking about reliability folks can mean different things… Measuring uptime in terms of total site availability. When talking about outages and uptime, folks generally mean the entire experience is available and aren’t concenred with invididual request level success. If the site is generally a bit slow and functions within normal bonds it is considered to be available and not suffering from an outage, even if the p99 is not looking good.

Planned Vs Unplanned Downtime

In all the discussion below we are focused on unplanned downtime and the risk assement to a business and service. Planned downtime should be part of the plan and help with risk mitagation. It is built in and schedulable for nearly all cloud services, because one way to mitagate risk is controlling for planned downtime during the lowest impact to users and the business. Further discussion of planned downtime is out of scope of this post.

Why not just build for High Availability and Five Nines from the Start?

A quick note since some folks try to push for massive scale or availability far before it is needed. I recommend not not Five nines: chasing the dream?, from the start. It doesn’t really work and it isn’t worth it, and most companies have learned that it isn’t worth the opportunity cost.

Google doesn’t do it
Amazon Doesn’t do it
If you build off cloud services, and if you build off their clouds you likely really aren’t either (we will cover this more later)

I have worked at a number of companies at various sizes and stages of growth, and seen companies grow to global scale while still never reaching high availability at a multi-region level. This isn’t to say you may not want to design for that level of resilience depending on your business, but you might be surpised at the number of companies that have gone public before needed to tackle that problem.

users typically don’t notice the difference between high reliability and extreme reliability in a service, because the user experience is dominated by less reliable components like the cellular network or the device they are working with. Put simply, a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability! –Google SRE Book

Difference between Outages and Normal Operations

One thing to note is most of the time core services don’t have total outages anymore, but they do happen and there are ways to prepare for them… A more common day to day impact on users of your systems is operational efficiency in terms of both success rate and performance.

Total Infrastructure Outages

Much of this is outside your control and you are dependent on your provides, but there are some patterns with providers or across multiple providers to mitgate incrustructure issues. The ability to change these more often lies with platform and infrastructure teams, but often comes with high cost, increased complexity, and little value outside of the risk mitigation. Having high availability around redundant CDN providers with DNS failovers as an example is an expensive solution to a problem that seems to occur every few years for CDN providers.

AWS Cloudfront
AWS Region
AWS S3
Fastly outage

This category, is where you might want to be thinking more in terms of Time-based availability opposed to Aggregate availability These are often rare occurances where massive systems fail and the only way to prepare for them is building multiple redundant systems ahead of time.

Total Application Services Outages

These are system level issues that cause outages that are under your application, platform, and infrastructure engineers control. A number of stragies can be applied both at platform and application level to reduce risk

configuration change mistake
application code logic mistake
data store failure (DB migration mistake)
system overload
bad host issues
thundering herd

This category, is where you might want to be thinking more in terms of Time-based availability, as these types of outages aren’t just expected built in failure rates that all services should be built to handle as part of normal opperations, these are trigged by faulty changes, unexpected load, out of expectation infrastructure issues.

Total 3rd Party Integration Outages

You might have some 3rd parties that are in portions of your critical path… Similar to infrastructure outages you are slightly dependant on your partners, but unlike infrastructure most mitigations can be handled by Eng or Platform teams opposed to infrastructure teams. For example, if you offer multiple logins you should ensure a single auth provider can’t take down your login page or consume so many web workers while timing out that it is impossible to process other login sources successfully.

braintree
paypay
stripe
FB Auth
Google Auth
Apple ID
authzero
onelogin

Daily Service Quality

This is the impact of day to day expected success rates, latency, and failures that should be expected during normal operations. The effect of these issues is somehting your application teams have the ability to impact and control the outcomes of. Note: that all your third parties also have spikes in latency, error rates, and can for partial outages be mitigated by eng and platform teams.

Micro Service SLAs

The issue with SLAs is that folks often want to aim for extremly high SLAs like “five nines” 99.999% uptime… Companies also want high fully independent teams running and managing microservices… Let’s take a look at how this plays out. Every time you add another service dependency you reduce the theoritical maximum SLA you could provide.

Basic Single Application Stack Request Flow

In this example I am using Sankey diagrams to attempt to visualize the flow of a request from clients through an example infrastructure stack.

A single request (like a request for an html page or single API request)

Micro Service Requests

An Example of a Microservice Architecture.

Includes: a web front-end, API service, and common supporting services (DB and Redis Cache)

Microservice SLAs

micro

> (0.999) * 100
=> 99.9
> (0.999 * 0.999) * 100
=> 99.8001
> (0.999 * 0.999 * 0.999) * 100
=> 99.7002999
> (0.999 * 0.999 * 0.999 * 0.999) * 100
=> 99.6005996001
> (0.999 * 0.999 * 0.999 * 0.999 * 0.999) * 100
=> 99.5009990004999
> (0.999 * 0.999 * 0.999 * 0.999 * 0.999 * 0.999) * 100
=> 99.4014980014994

AWS SLAs

Let’s look at this from the perspective of a typical AWS applications, assuming a single app stack (no micro services), a pretty standard setup with app code of 99.9% still leaves a total SLA max of %99.6

cloudfront   ALB       ECS      YOUR APP    Elasticache   RDS (Postgres)
(0.999   *  0.9999  * 0.9999  *  0.999   *    0.999        *  0.9995)   * 100
=> 99.63052065660449

You can see as you stack AWS services, regardless of your application stability, request success percentage decreases…

Service	SLA	% Success Math	Request % Success
cloudfront	99.9	0.999	99.9%
ALB	99.99	0.999 * 0.9999	99.89%
ECS	99.99	0.999 * 0.9999 * 0.9999	99.88%
Custom App	99.9	0.999 * 0.9999 * 0.9999 * 0.999	99.78%
Elasticache	99.9	0.999 * 0.9999 * 0.9999 * 0.999 * 0.999	99.68%
RDS (Postgres)	99.95	0.999 * 0.9999 * 0.9999 * 0.999 * 0.999 * 0.9995	99.63%

Impacts of the internet

Now let’s consider that even if one has achived all of this, are your customers actually receiving “five nines” of service? If we consider their provider ISP (assuming home internet connections via ISPs), local wifi setup, or even worse a mobile connection… You can already drop expectations to at least %99.9, really you are trying to make your service appear as stable as their internet, with as many network failures as the customer has to endure with any other serivce.

If failures are being measured from the end-user perspective and it is possible to drive the error rate for the service below the background error rate, those errors will fall within the noise for a given user’s Internet connection. While there are significant differences between ISPs and protocols (e.g., TCP versus UDP, IPv4 versus IPv6), we’ve measured the typical background error rate for ISPs as falling between 0.01% and 1%. -- Embracing Risk, Site Reliability Engineering

WHAT ABOUT COST ^^ Google covers as well

Learning with Game Days

2021-01-08T00:00:00+00:00

photo credit Neramitevent@pixabay

Learning with Game Days

Many different companies and posts talk about why and how to run Game Days. I won’t rehash all of that in this post, instead I will give a basic intro and link to some sources then dive into some more specific Game Days I have recently been involved with.

A game day simulates a failure or event to test systems, processes, and team responses. The purpose is to actually perform the actions the team would perform as if an exceptional event happened

Below are a few recent Game Day examples, with some details of what we did, what we learned, how we broke something in a “safe” manner, and a runbook to run your own. In all cases, one important factor of running the Game Day is having a way to stop if the simulated incident starts to cause a real incident. When planning a simulated emergency make sure you have a planned way to escape out of the test if something unexpected is occurring.

Safe and Confident Deployments

For various reasons some of our systems were relatively slow to deploy. This means if something bad happened, it could take awhile to revert, pass CI, deploy etc… We finally got access to a rollback tool which was much faster than the normal deploy process to jump back to a recently deployed version of the code. While the tool was available for awhile many folks had their first experience using it during a real incident! Trying to find docs and understand the commands while stressed about some service being broken. Not the ideal way to learn… We got a group together with a preplanned and scheduled time, we then broke a ‘hidden’ endpoint on production, used rollback, fixed the code and showed how to “roll forward” back into the standard development flow. Since our tool did a bunch of cool things like freezing the deploy when you roll back, alerting various folks that it was going on, etc we got to see and feel the full experience. We had folks running the commands who had never had to do a rollback in production before, which made the exercise really valuable.

Runbook

schedule with a group of interested folks
announce to everyone in relevant channels so they don’t believe a real incident is occurring
deploy a non-important change
roll back and verify all the alerts, notifications, and commands
watch dashboards confirming how quickly the rollback worked and shows how to monitor deploy progress
fix the non important change
unfreeze deploys (if your system supports something like this)
deploy the main branch and restore the standard flow.

Upstream Partner Timeouts

We have a feature that I will cover more at some point in a future post, called chaos traces, similar to chaos monkey it lets you inject some chaos into your system to see how it behaves. In this Game Day, after the release of a new partner integration, we used our chaos trace tool to validate how the integration handle timeouts and errors. The code author and the PR reviewer paired up following the deploy and used chaos traces to verify that when the integration had an API Error or if the integration was running slow that the app handled it how everyone expected. By injecting say 3 seconds of latency into the respond time, we verify that the app would handle the timeout and give the customer a reasonable UX experience.

Runbook

schedule with a group of interested folks
announce to everyone in relevant channels so they don’t believe a real incident is occurring
cause partner error scenarios (if you don’t have chaos functionality a feature flipper, or ENV toggle may work)
watch dashboards confirming expected logs / metrics
screen shot / share the UX experience of the error states with the team
have confidence that edge case integrations are not leaving users stuck in a confusing state

Simulating Traffic Spikes

I didn’t participate directly in this Game Day, but I helped write some of the code and reviewed PRs. As a new service was being integrated into a high traffic flow, before the new service went live the team wanted to load test it. In the past we have often used simulated load, but it is hard to realistically generate traffic that looks like production… We tried something new and did a Game Day around production load… The team had areas of site that would soon make the new experience visible fire and forget all get requests to the new services without displaying and results… They even had a scale factor so we took traffic on a normal day and generated 3X production traffic to the new backing service… Since all the requests were fire and forget it didn’t slow down the user experience and let us work on performance tunning the new service under real heavy load.

Runbook

schedule with a group of interested folks
announce to everyone in relevant channels so they don’t believe a real incident is occurring
have a feature toggle to enable or disable the fire and forget load
enable and start to scale up the load factor
watch dashboards
- ensuring the Game Day isn’t impacting existing services or users
- learn how the new service response to heavy and increasing load and fix any performance issues
toggle back off the simulated load, and move forward with confidence the new service can handle real traffic

Cache Failures

Caching is hard, eventually you might have to break a bad cache… If this cache has data for a few million users, sometimes very bad things happen to performance while the system tries to rebuild a cache. This is a future Game Day, that we have not yet done, but are currently planning after running into a cache issue. We were able to quickly bust the cache and fix a service that was in a invalid state, but it got us thinking… As the service and data set keeps growing, how long can we rely on that? At what point will we break cache and start to see a cascading of timeouts throughout all of our client services as we buckle under the load… Seems like a perfect thing to Game Day, we plan to put a feature toggle in place to let us ramp up cache_skips for a percentage of requests. Turn up the dial and ensure we can hit 100% without to large of an impact to our performance and stability.

Runbook

schedule with a group of interested folks
announce to everyone in relevant channels so they don’t believe a real incident is occurring
have a feature toggle turn up cache skips
start to turn up while watching dashboards and dial back down if there is any unexpected performance issues
identify any scaling issues and address and repeat until comfortable
enjoy the confidence that next time someone needs to bust the entire cache it shouldn’t be a problem

Event Bus Failures

After some recent issues with our Event Bus, we put a number of protections in place that we thought would ensure our service could handle a significant unplanned outage without causing any interruption to clients. We made several changes to how we were handling events

moving from synchronous to asynchronous sending of events where possible
on any synchronous message queue retries in background jobs for later
increase our background job storage to be able to handle millions of stalled and failed events waiting for recovery
autoscaling background workers to be able to quickly process a big backlog of events without going to fast to overload the system

OK, great next time there is an incident our service will keep humming along without any issue… Well, would it really hold up? How could we be sure? Of course by now, y’all realize that instead of waiting for the next time our events queue has an incident, we can run a Game Day! That is what we did, we made a toggle to disable our access to the event queue server… Then flipped it off, we got to see if our protections worked verifying each doing what was expected… If we were wrong we could very quickly restore the system… During the incident the week prior, our internal background job storage overflowed in about 15m as we helplessly watched in horror… As we watched for 15 minutes during this Game Day our new monitor and connected alert showed the estimated time remaining we could buffer all events before running into problems. Showing how we could now handle hours of downtime, giving the teams plenty of time to get paged, and work on a resolution if needed.

Runbook

schedule with a group of interested folks
announce to everyone in relevant channels so they don’t believe a real incident is occurring
toggle off access to the event queue service
watch dashboards and improve any lacking observability to be confident in being able to understand your system
scale up any resources that are more impacted than expected
use the background job stalled jobs load to calculate and set alerts and storage needed for appropriate response times
enjoy the confidence that next time the event service has unexpected maintenance your service won’t drop any data and the services clients won’t see any errors.

Conclusion

Game Days can be used to work through what could be a difficult or exceptional situation in a controlled manner at a planned time and the support of a team. When done well, they can teach folks how to be better prepared for incidents, better understand their systems, and have more empathy for how other folks process and understand system in the stress of a real incident. They are a great way to ensure everyone on the team is ready when something unexpected next happens.

Performance of JSON Parsers at Scale

2020-12-02T00:00:00+00:00

photo credit qimono@pixabay

Performance of JSON Parsers at Scale

In a recent post, benchmarking JSON Parsers (OJ, SimdJson, FastJsonParser). This compared the parsers based on local microbenchmarks. In the end, I recommended for almost all general use cases go with OJ. Saying that FastJsonParser might be worth it for specific use cases. I want to do a quick follow up on sharing what happens when microbenchmarks meet real-world data, scale, and systems.

TL; DR; you probably just want to use OJ as originally recommended, even on data where FastJsonParser wins in a microbenchmark, the real-world data was undetectable. While moving from StdLib to OJ was a 40% latency improvement holding up across multiple services.

Microbenchmarks

As often the case microbenchmarks come with a lot of issues. In this case, my microbenchmarks showed with a single example of real-world data that FastJsonParser was that faster and had the lowest memory usage… OJ was about 1.55x slower in both with_keys and normal string key benchmarks. I benchmarked against two JSON pieces, a very small fake JSON payload, and a real-world large payload pulled from one of our production systems. For the specific examples, I used and with no other concerns yes FastJsonParser is faster, but that doesn’t mean it will translate into a real-world performance win.

Given that we had previously seen 40% latency improvements when moving to OJ, it seemed like another 50% speed lift would be worth it, so I set out to test FastJsonParser on some of our production systems.

Last week / Yesterday -> OJ released with 40% latency improvement

What Does Real World & At Scale Mean?

In my case, I started with a single app having all API calls use FastJsonParser to parse responses as well as when pulling JSON out of caches. The single app had a smaller JSON payload than I benchmarked with but had a very high throughput. After deployment, there was no detectable change in latency… Why not?

At that point, the way the app was performing was already fairly well optimized
According to DataDog trace spans JSON parsing was taking up less than 1ms of response time
Um… what is 50% faster on 1ms of a response, where JSON parse wasn’t even in the top 10 time-consuming spans of building the response? Nothing really

OK, I figured I picked a bad test case… I had originally benchmarked with a large JSON collection blob that passed through multiple systems. I decided to target 5 applications that worked together that used and served the original data I used to benchmark. This broke down like so:

1 front end app
4 microservices sending different JSON payloads

In total that large JSON collection data was passed through 3 of the 5 apps, with other JSON data coming from the other services. I figured this would have a bunch of small wins that would add up to show reduced latency for the front end application. Since all the small gains would eventually roll up to its final response time.

After sending out 5 PRs, getting approvals, deploying, I played a waiting game watching graphs and collecting data… NOTHING, I could see nothing. No errors, no problems, no performance impact.

Why Wasn’t It Faster?

I think similar to the single app example, even in this case OJ had really already captured the majority of the wins. JSON.parse was no longer in the top 10 spans of any of the 5 apps I updated. It previously was a part of the critical performance path… It no longer was… I am guessing there might have been tiny improvements, but nothing I could see with the naked eye… For most of these services, a 1ms improvement in each service, wouldn’t have been visible with all the random network noise.

I think just network latency outweighed any further improvement on JSON parsing… None of the payloads were large or complex enough to drive a significant cost. This goes back to the original point, you need to really have a good reason to spend the extra time with FastJsonParser to drive further improvements over OJ’s Json.parse drop-in replacement which also ensures all the Rails toolchain and middleware is using the improvements. Since FasterJsonParser requires the developer to explicitly call FastJsonparser.parse I only did that where we handled API calls, it took more work and it wasn’t an improvement. If you have spans where JSON.parse is showing significant time in your application traces, it could be different for you.

via GIPHY

Microbenchmark -> Welcome to the real world

Conclusion

Unless you are maintaining a gem and are avoiding dependencies, I highly recommend using OJ for your applications. It requires very low effort and holds up in microbenchmarks and across many different services and real-world data.

benchmarking JSON Parsers (OJ, SimdJson, FastJsonParser)

2020-11-15T00:00:00+00:00

photo credit Tumisu lt: @pixabay

UPDATE: Added FastJsonParser

After some feedback on reddit (thx @f9ae8221b), pointing out a JSON gem I wasn’t aware of, I updated the benchmarks to also support FastJSONparser and cover symbolize_keys, which is important for my companies use cases (which a co-worker pointed out) and can cause significant performance issues if you have to do that independently of JSON parsing.

Performance Benchmarks between OJ, SimdJSON, FastJsonparser, and StdLib

I was recently looking at the performance of some endpoints that process large amounts of JSON, and I wondered if we could do even better than we do in terms of performance for that processing. Across our company we have recently switch most of our apps from the Ruby StdLib JSON to OJ, but I had read about SimdJSON and was curious if we should look further into it as well. In this article I will tell you a bit about each of the Ruby JSON options and why you might want to consider them.

OJ

OJ is a Ruby library for both parsing and generating JSON with a ton of options. I would basically say if you don’t want to think too much but care about JSON performance just set up the OJ gem, and it should be the best option for most folks. It is well known, tested, and trusted in the community with a ton of support.

A drop in replacement, set it forget it, it is faster and better
Has built in Rails support, for some Rails JSON quirks
Supports both generation and parsing

SimdJSON

The SimdJSON Ruby library doesn’t have a lot of talks, documentation, or attention… but it is binding to the fastest JSON parser out there… It offers parsing speeds that nothing else can touch, and if you are trying to parse extremely large and dynamic JSON, it might just be the best option for you.

Only supports parsing
Doesn’t support symbolize_keys
The fastest option out there (uses simdjson C libs)
Not a lot of community activity

FastJsonparser

While simdJSON is a fast gem, it doesn’t have much support and the way it handles rescuing errors could leak memory. While I didn’t see such issues in my limited production rollout that is worth noting. The user @f9ae8221b pointed out the memory issue and that the gem FastJsonparser also wraps simdjson and has wider community support. I had never heard of the gem, and was already trying to patch SimdJSON to support symbolize_keys. Luckily FastJsonparser already supports that option. It still is faster than OJ, and requires a bit more work to intgrate, but looks like a better option than SimdJSON when you are looking for improved parsing speed. The user still mentions it could have some production issues, so I will have to report back as I roll it out to various systems.

Only supports parsing
Does support symbolize_keys
The fastest option out there (uses simdjson C libs)
Larger community support

StdLib Ruby JSON

It is built in, seriously if you do much with JSON in a production system, just use OJ, unless you want to dig in deeper or find some specific reason it won’t work for you. The Ruby library is fine and will work for any quick check, but if you have any reason to care about performance, OJ is a easy to use drop in replacement… A note of when you shouldn’t use it? If you are authoring a gem, reduce your hard dependencies as much as possible, if you call JSON.parse and a hosting app is using OJ, your gem will use OJ and be faster… You shouldn’t force users of your gem to require OJ.

Benchmarking the methods

Let’s see the difference with favorite Ruby benchmarking gem benchmark-ips, which gives a bit more readable reports than the standard benchmark lib. These are just quick micro-benchmarks, with all the issues that come with them, but the performance impact has been further validated by deploying to production systems with measurable impacts on the response time. The product use case included far larger JSON payloads and with much higher variability to the data, making me think the results would apply to most web service like systems.

Benchmarking JSON Parsing (without symbolize_keys, added FastJsonparser)

We will load up the various libraries, and some weird fake HASH/JSON data. Then benchmark parsing it for a number of seconds…

require 'benchmark/ips'
require 'json'
require 'oj'
require 'simdjson'
require 'fast_jsonparser'
require 'memory_profiler'
require 'rails'

json = {
  "one":1,
  "two":2,
  "three": "3",
  "nested": {
    "I": "go",
    "deep": "when",
    "i": "need",
    a: 2
  },
  "array":[
    true,
    false,
    "mixed",
    "types",
    2,
    4,
    6
  ]
}.as_json.to_json.freeze

puts "ensure these match"
puts  Oj.load(json, symbol_keys: false) == Simdjson.parse(json) &&
        Simdjson.parse(json) == JSON.parse(json, symbolize_names: false) &&
        FastJsonparser.parse(json, symbolize_keys: false) == Simdjson.parse(json)

Benchmark.ips do |x|
  x.config(:time => 15, :warmup => 3)

  x.report("oj parse") { Oj.load(json, symbol_keys: false) }
  x.report("simdjson parse") { Simdjson.parse(json) }
  x.report("FastJsonparser parse") { FastJsonparser.parse(json, symbolize_keys: false) }
  x.report("stdlib JSON parse") { JSON.parse(json, symbolize_names: false) }

  x.compare!
end

# Let's check memory as well...
report = MemoryProfiler.report do
  100.times { Simdjson.parse(json.dup) }
end
puts "simpdjson memory"
report.pretty_print

report = MemoryProfiler.report do
  100.times { Oj.load(json.dup) }
end

puts "OJ memory"
report.pretty_print

Benchmark Results (without symbolize_keys)

This shows as claimed that SimdJSON and FastJsonparser outperform OJ even on pretty small and contrived JSON examples. The Performance gap holds up or sometimes looks more significant when looking at more realistic production payloads seen in some of the product systems I work with. Note if you need symbolize_keys or want a bit more community support I would go with FastJsonparser.

ensure these match
true
Warming up --------------------------------------
            oj parse    12.697k i/100ms
      simdjson parse    17.276k i/100ms
FastJsonparser parse    17.834k i/100ms
   stdlib JSON parse     8.662k i/100ms
Calculating -------------------------------------
            oj parse    121.709k (± 3.5%) i/s -      1.828M in  15.040973s
      simdjson parse    171.253k (± 4.3%) i/s -      2.574M in  15.060276s
FastJsonparser parse    190.436k (± 3.2%) i/s -      2.853M in  15.000218s
   stdlib JSON parse     93.032k (± 3.4%) i/s -      1.403M in  15.102830s

Comparison:
FastJsonparser parse:   190436.3 i/s
      simdjson parse:   171252.9 i/s - 1.11x  (± 0.00) slower
            oj parse:   121709.5 i/s - 1.56x  (± 0.00) slower
   stdlib JSON parse:    93032.1 i/s - 2.05x  (± 0.00) slower

Benchmarking JSON Parsing (with symbolize_keys)

require 'benchmark/ips'
require 'json'
require 'oj'
require 'simdjson'
require 'fast_jsonparser'
require 'memory_profiler'
require 'rails'

json = {
  "one":1,
  "two":2,
  "three": "3",
  "nested": {
    "I": "go",
    "deep": "when",
    "i": "need",
    a: 2
  },
  "array":[
    true,
    false,
    "mixed",
    "types",
    2,
    4,
    6
  ]
}.as_json.to_json.freeze

puts "ensure these match"
puts  Oj.load(json, symbol_keys: true) == Simdjson.parse(json).deep_symbolize_keys! &&
        Simdjson.parse(json).deep_symbolize_keys! == JSON.parse(json, symbolize_names: true) &&
        FastJsonparser.parse(json) == Simdjson.parse(json).deep_symbolize_keys!


Benchmark.ips do |x|
  x.config(:time => 15, :warmup => 3)

  x.report("oj parse") { Oj.load(json, symbol_keys: true) }
  x.report("simdjson parse") { Simdjson.parse(json).deep_symbolize_keys! }
  x.report("FastJsonparser parse") { FastJsonparser.parse(json) }
  x.report("stdlib JSON parse") { JSON.parse(json, symbolize_names: true) }

  x.compare!
end

Benchmark Results (with symbolize_keys)

This is the other main reason to use FastJsonparser depending on the integrations in your apps you might rely on symbolized_keys… We had added that at a very low level in our shared ApiClient, and the performance implications of having to symbolize_keys as a second pass make a big difference. This shows how the simdjson performance win doesn’t hold up when you need symbolized_keys.

ensure these match
true
Warming up --------------------------------------
            oj parse    13.455k i/100ms
      simdjson parse     7.752k i/100ms
FastJsonparser parse    19.458k i/100ms
   stdlib JSON parse     8.546k i/100ms
Calculating -------------------------------------
            oj parse    134.285k (± 4.5%) i/s -      2.018M in  15.060313s
      simdjson parse     75.825k (± 7.2%) i/s -      1.132M in  15.022033s
FastJsonparser parse    208.199k (± 3.1%) i/s -      3.133M in  15.061737s
   stdlib JSON parse     86.504k (± 3.5%) i/s -      1.299M in  15.035736s

Comparison:
FastJsonparser parse:   208199.1 i/s
            oj parse:   134285.4 i/s - 1.55x  (± 0.00) slower
   stdlib JSON parse:    86503.7 i/s - 2.41x  (± 0.00) slower
      simdjson parse:    75825.4 i/s - 2.75x  (± 0.00) slower

Using Large JSON Data

The results are very similar for a much larger production 120K JSON payload, pulled for a live system. (NOTE: these benchmarks were run on a different machine)… In this case we are showing nearly a 2X performance boost.

without symbolize_keys:

Warming up --------------------------------------
            oj parse    62.000  i/100ms
      simdjson parse    79.000  i/100ms
   stdlib JSON parse    42.000  i/100ms
Calculating -------------------------------------
            oj parse    622.377  (± 3.9%) i/s -      9.362k in  15.066907s
      simdjson parse    815.699  (± 4.5%) i/s -     12.245k in  15.045902s
   stdlib JSON parse    426.656  (± 3.5%) i/s -      6.426k in  15.083428s

Comparison:
      simdjson parse:      815.7 i/s
            oj parse:      622.4 i/s - 1.31x  (± 0.00) slower
   stdlib JSON parse:      426.7 i/s - 1.91x  (± 0.00) slower

with symbol_keys:

ensure these match
true
Warming up --------------------------------------
            oj parse    71.000  i/100ms
      simdjson parse    29.000  i/100ms
FastJsonparser parse    82.000  i/100ms
   stdlib JSON parse    41.000  i/100ms
Calculating -------------------------------------
            oj parse    726.191  (± 1.5%) i/s -     10.934k in  15.059977s
      simdjson parse    294.947  (± 2.4%) i/s -      4.437k in  15.052250s
FastJsonparser parse    909.828  (±10.2%) i/s -     13.530k in  15.026051s
   stdlib JSON parse    497.749  (± 3.6%) i/s -      7.462k in  15.011659s

Comparison:
FastJsonparser parse:      909.8 i/s
            oj parse:      726.2 i/s - 1.25x  (± 0.00) slower
   stdlib JSON parse:      497.7 i/s - 1.83x  (± 0.00) slower
      simdjson parse:      294.9 i/s - 3.08x  (± 0.00) slower

The MemoryProfiler (nor production deployments, server metrics) on either small or large JSON objects didn’t really show any substantial difference, so I wouldn’t be too concerned with memory when picking these libraries.

Conclusion

If you have a Ruby service that is parsing large quantities of JSON, it might be worth taking a look at the newer and less known FastJsonparser. While the gem is less documented and takes a bit more work to integrate into your app than OJ. If you are looking for a drop in replacement OJ is still the way to go, but for some use cases SimpdJSON or FastJsonparser will be worth the extra effort. If you are using Rails with a production deployment I can’t really see any reason to not use OJ for the significant performance benefits that come with it. The OJ library made it as easy as possible to use as a drop in replacement and if you rely on nearly any particular JSON quick of the past they have options to help you stay fully compatible. I know as we look towards Ruby 3 we are also hoping to move away from some of the native extension C libraries, but when it comes to very low level repetitive application tasks vs application logic, sometimes it is hard to beat and worth the integration and dependency cost.

Ruby: Understanding create_or_find_by vs find_or_create_by

2020-10-22T00:00:00+00:00

photo credit geralt: @pixabay

Performance Benchmarks & Considerations between `create_or_find_by` & `find_or_create_by`

I was recently optimizing an endpoint and got to think through some interesting differences between two Active Record methods that help you either find an existing record or create a new one. At first glance, it seems either is fine with some notable differences around their race conditions.

`find_or_create_by`

The find_or_create_by method has been around longer and is more familiar to many Rubyists. The race condition is called out in the linked docs, excerpt below.

Please note this method is not atomic, it runs first a SELECT, and if there are no results an INSERT is attempted. If there are other threads or processes there is a race condition between both calls and it could be the case that you end up with two similar records.

This lead to Rails 6 adding the newer methods…

`create_or_find_by`

The new create_or_find_by methods have a more rare race condition (on deleted ids), but can prevent a more common insert race condition on duplicates… It is well described in this post, Rails 6 adds create_or_find_by, along with some downsides. For example without a unique DB constraint it will create duplicates (ex: add_index :posts, :title, unique: true). These issues are also called out in the docs linked above, excerpt below.

The underlying table must have the relevant columns defined with unique constraints.
While we avoid the race condition between SELECT -> INSERT from #find_or_create_by, we actually have another race condition between INSERT -> SELECT, which can be triggered if a DELETE between those two statements is run by another client. But for most applications, that’s a significantly less likely condition to hit.
It relies on exception handling to handle control flow, which may be marginally slower.
The primary key may auto-increment on each create, even if it fails. This can accelerate the problem of running out of integers, if the underlying table is still stuck on a primary key of type int (note: All Rails apps since 5.1+ have defaulted to bigint, which is not liable to this problem).

Benchmarking the methods

While the docs are good at calling out the race conditions, they are not as clear about the performance implications… In fact, they could lead one to believe that create_or_find_by is always slower from this line, “may be marginally slower”… The reality is you need to know the usage characteristics of where you will be calling these methods to pick the one with the best performance characteristics.

Both of the methods will either find or create a record, and they try those in different orders… If you expect to most often find the record vs most often create a record that has a big impact on the performance. Let’s see how much by breaking out my favorite Ruby benchmarking gem benchmark-ips, which gives a bit more readable reports than the standard benchmark lib. These are just quick micro-benchmarks, with all the issues that come with them, but the performance has also been validated by deploying to production systems at scale.

Benchmarking All Finds

In this case, we are going to benchmark a case that simulates a code path that is all finds, and no creates… If you have an endpoint that creates once in a user’s lifecycle and then forever is hitting the find, you likely will have a much higher find vs create ratio close to this benchmark.

require 'benchmark/ips'
ActiveRecord::Base.logger = nil
Post.destroy_all

Benchmark.ips do |x|
  x.config(:time => 10, :warmup => 3)
  x.report 'create_or_find_by' do
    Post.create_or_find_by!(title: "create_or_find_by")
  end
  x.report 'find_or_create_by' do
    Post.find_or_create_by!(title: "find_or_create_by")
  end
  x.compare!
end

results:

As expected, when you would find an existing record all the time find_or_create_by is much faster, approximately 4X faster!

Warming up --------------------------------------
  create_or_find_by     49.000  i/100ms
  find_or_create_by    204.000  i/100ms
Calculating -------------------------------------
   create_or_find_by     450.791  (± 7.8%) i/s -      4.508k in  10.063664s
   find_or_create_by     2.078k (± 6.9%) i/s -     20.808k in  10.061016s

Comparison:
  find_or_create_by:     2078.1 i/s
  create_or_find_by:     450.8 i/s - 4.61x  (± 0.00) slower

Benchmarking All Creates

In this case, we will benchmark where nearly all the calls are creating new records… This would simulate an endpoint that is generally creating brand new records and very rarely should find an existing record.

require 'benchmark/ips'
ActiveRecord::Base.logger = nil
Post.destroy_all

Benchmark.ips do |x|
  x.config(:time => 10, :warmup => 3)
  x.report 'create_or_find_by' do
    Post.create_or_find_by!(title: "create_or_find_by #{rand}")
  end
  x.report 'find_or_create_by' do
    Post.find_or_create_by!(title: "find_or_create_by #{rand}")
  end
  x.compare!
end

results:

In a case where you are always creating it is faster to create_or_find_by but the overall difference is less dramatic.

Warming up --------------------------------------
  create_or_find_by     73.000  i/100ms
  find_or_create_by     44.000  i/100ms
Calculating -------------------------------------
  create_or_find_by     722.939  (± 8.3%) i/s -      7.227k in  10.069582s
  find_or_create_by     522.615  (± 9.6%) i/s -      5.192k in  10.028946s

Comparison:
  create_or_find_by:      722.9 i/s
  find_or_create_by:      522.6 i/s - 1.38x  (± 0.00) slower

Conclusion

When you are working with create_or_find_by or find_or_create_by ensure you are considering how and which race conditions might affect your code. If it is easier to just have your app handle DB constraint errors and retry directly, most of the time using find_or_create_by is going to be simpler and more performant… If you reach for create_or_find_by ensure you understand the additional complexity and performance impacts depending on the expected hit and miss ratio for your use case.

Ruby: Patching StdLib in Gems

2020-07-15T00:00:00+00:00

photo credit patches: AnnaER @pixabay

Why Patch Ruby StdLib Code in Gems

Well, the Ruby community does this a lot, it can unlock powerful enhancements, features, observability, and more…

Here are some examples of patching Ruby’s StdLib (standard library). Let’s just look at a few that patch a single piece of Ruby, Net::HTTP. Many libraries want to tap into what is happening around the network.

DataDog’s dd-trace-rb gem, patches many Ruby StdLib features like Net::HTTP
MiniProfiler also patches Net::HTTP
WebMock in tests patches Net::HTTP

Sometimes opposed to patching upstream Ruby code, one can just have adapters/wrappers around them, while related it is a much different approach and you can see how Faraday handles adapting Net::HTTP as an example of that approach. Which is safer, but requires upstream apps to change their code to use the libraries’ APIs as opposed to modifying existing behavior.

Gems Patch Ruby StdLib, So What?

The problem comes up with multiple gems trying to patch the same method. From the examples above, there are multiple ways to attempt to modify the original code, which doesn’t always play nicely together.

alias, alias_method, and the like
prepend, class/module extension ways of extending a method and using super
replacing constants, I don’t know the common term for what WebMock does to patch Net::Http

If you have multiple gems patching the same upstream Ruby StdLib (or Rails) class or function, you can run into issues. This is a known Ruby ‘Bug’ along with a known solution to detect and patch in the same way.

Example: Errors: stack level too deep

The reason I am writing this up is that I had a bug in Coverband for months, thx bug reporters(@) I appreciate it, that made no sense to me… I couldn’t reproduce it, I didn’t have any great stack traces, I had no idea what area of code the issue was even in… I couldn’t even investigate the issue. At the time all I really knew about the bug? Exception: Stack level too deep error.

After months, of once in awhile taking a look but not understanding the problem… I got a new bug report from @ hanslauwers… Which, added some details, specifically that the gem AirBrake and Coverband, both were patching Resque… but in different ways…

A few days prior to the above report, I saw while working on another project this excellent description of a problem that had been solved in the MiniProfiler project, the readme documents how to resolve Net::HTTP stack level too deep errors… So the new bug report made my spidey sense tingle, and I was finally able to fix it.

How to handle applications differences

I ended up following the same pattern as MiniProfiler, which described the problem and the fix excellently in it’s readme.

If you start seeing SystemStackError: stack level too deep errors from Net::HTTP after installing Mini Profiler, this means there is another patch for Net::HTTP#request that conflicts with Mini Profiler’s patch in your application. To fix this, change rack-mini-profiler gem line in your Gemfile to the following:

… examples …

This conflict happens when a ruby method is patched twice, once using module prepend, and once using method aliasing. See this ruby issue for details. The fix is to apply all patches the same way. Mini Profiler by default will apply its patch using method aliasing, but you can change that to module prepend by adding require: [‘prepend_net_http_patch’] to the gem line as shown above.

The readme, explains the issue, has code examples for how app’s integrating the gem can resolve the issue, and links to the original Ruby “Bug”, which explains the issue in detail and discusses approaches to solve the problem

Coverband’s Patching Solution

This is the PR that was merged after understanding the problem and approach I took to resolve the problem. Again, heavily patterned off the MiniProfiler solution.

In the end, it is a pretty simple fix, but it took time and various folks participating in the bug report to understand. If you see an open github issue that still seems relevant, add some comments and details. You never know if you will be the trigger that helps folks understand and resolve the issue.

I know patching always gets a bad wrap in Ruby, and it can be hard to fully understand and debug, but it is also extremely powerful. It is good to understand the gotcha’s that can occur, and how to work around those issues, especially if you are shipping shared code that can patch other shared code like Ruby’s StdLib.

Rails Flaky Spec Solutions

2020-01-14T00:00:00+00:00

photo credit flaky wall: pixabay

Introducing Rails Flaky Spec Examples

I have written about how our teams are dealing with flaky ruby tests in legacy projects. In this post, I want to show, how we can teach about common testing pitfalls and how to avoid them.

In this post, I want to introduce a new project Rails Flaky Spec Examples. I created the example flaky Rails spec suite for an internal code clinic at Stitch Fix. We ran it as a 1-hour workshop to teach about common flaky test issues and how to resolve them. I am hoping that over time, I can continue to grow the example base and talk about some of the more complex flaky tests and how we could perhaps more systematically avoid them. As I work with this project over time, I hope to make it into a good learning tool for the community.

Running the flaky and thens stable suite

Why A Rails Suite? One Problem with Flaky Test Posts

While there are a number of great blog posts on flaky specs:

The majority of the posts don’t have runnable examples. While they might have some code snippets showing examples in the post they don’t have a runnable project. You can try to paste some of the examples into a file, but they reference dependencies, like an active record model without the related migration or dependencies. Often the snippets get too simplified to show how the errors look in a real large legacy project.

Since the examples aren’t runnable it makes it a bit harder to use them as a teaching tool, or show more complex tests or CI interactions. This project aims to be in the sweet spot, where it is still small enough to easily understand the issues, but it is part of a real runnable app that can be extended to highlight more complex Rails and JS testing issues. Adding things like common libraries (Faker, Capybara, etc) and different levels of tests including browser-based javascript tests and the related race conditions.

While this project isn’t a real-world example which are the best sources of flaky specs, sometimes real-world examples are hard to easily understand. Many of the examples in this project were extracted from real world examples. If you really want to dive into a fully developed complex code base that has Flaky specs, the best source for that with tagged flaky specs comes from @ samsaffron/discourse.org in their tagged collection of flaky heisentest which is described more in the excellent post, tests that sometimes fail.

This project allows devs, to run spec examples, see the failures, and try to fix the flaky specs either themselves or with a small group. If they get stuck example solutions are readily available. It should also be relatively easy to extend the project to add examples extracted from real-world projects. I would love to get some flaky test submissions for difficult flaky spec issues.

Project Structure

The project is designed to have two versions of every spec the flaky version, and the stable version.

You can see this in the folder structure each spec folder has a sub-directory solved

This lets me use Rspec tags to run either the solved or flaky specs.

config.define_derived_metadata(file_path: Regexp.new('/solved/')) do |metadata|
  metadata[:solved] = true
  ENV['SOLVED_SPECS'] = 'true'
end

With these dynamic tags and a default .rspec with --tag ~solved:true we can now run either the flaky or stable suite.

flaky: bundle exec rspec
stable: bundle exec rspec --tag solved

Example: Solving A Flaky Spec

Let me show the expected workflow when learning with the project…

Run the suite and pick a failure that looks interesting…
Read the Related Code
Modify the Spec, try to fix it
Compare your answer to the provided solution (remember the is more than one way to solve many of these issues)

1. Pick a Failure

Let’s run the suite and pick a failure. In this case spec/models/post_example_e_spec.rb, Post ordered expect many order posts to be in alphabetical order, looks interesting.

bundle exec rspec
Run options: exclude {:solved=>true}

Randomized with seed 5788
...Capybara starting Puma...
* Version 4.3.1 , codename: Mysterious Traveller
* Min threads: 0, max threads: 4
* Listening on tcp://127.0.0.1:64214
...FF.FF.FFF.F.FFF

Failures:
...

3) Post post ordered expect many order posts to be in alphabetical order
   Failure/Error: alphabet.each { |el| Post.create!(title: Faker::String.random(2), body: el) }

   ActiveRecord::RecordInvalid:
     Validation failed: Title has already been taken
   # ./spec/models/post_example_e_spec.rb:12:in `block (4 levels) in <top (required)>'
   # ./spec/models/post_example_e_spec.rb:12:in `each'
   # ./spec/models/post_example_e_spec.rb:12:in `block (3 levels) in <top (required)>'
...

Finished in 10.2 seconds (files took 3.33 seconds to load)
21 examples, 11 failures

Failed examples: ...
rspec ./spec/models/post_example_e_spec.rb:10 # Post post ordered expect many order posts to be in alphabetical order

Let’s take a closer look at the code involved. In this case, from the comments, we see this spec is flaky on its own and doesn’t require the full suite to be flaky.

require 'rails_helper'

class Post < ApplicationRecord
  validates :title, uniqueness: true

  scope :ordered, -> { order(body: :asc, id: :asc) }
end

# Classification: Randomness
# Success Rate: 80%
# Suite Required: false
RSpec.describe Post, type: :model do
  describe "post ordered" do
    it "expect many order posts to be in alphabetical order" do
      alphabet = ('a'..'z').to_a
      alphabet.each { |el| Post.create!(title: Faker::String.random(2), body: el) }
      expect(Post.ordered.map(&:body)).to eq(alphabet)
    end
  end
end

3. Modify The Spec

For the above example, since we require the title to be unique, but we are using a small random value… We can see collisions occur. There are a number of ways to solve this, perhaps we don’t need randomness at all in this case!

require 'rails_helper'

class Post < ApplicationRecord
  validates :title, uniqueness: true

  scope :ordered, -> { order(body: :asc, id: :asc) }
end

# Classification: Randomness
# Success Rate: 80%
# Suite Required: false
RSpec.describe Post, type: :model do
  describe "post ordered" do
    it "expect many order posts to be in alphabetical order" do
      alphabet = ('a'..'z').to_a
      alphabet.each { |el| Post.create!(title: el, body: el) }
      expect(Post.ordered.map(&:body)).to eq(alphabet)
    end
  end
end

Run the spec a few times to ensure it now always passes.

4. Compare Answers

Now you can look at the file in the solutions folder. It will contain a solved spec with additional details explaining why there was an issue, and how it was solved. Occasionally offering more than one solution. Now you can repeat the steps until you have no more errors in the spec suite.

Using the Project for A Workshop

This is how we turned the project into a code clinic or workshop. We ran it entirely remotely, scheduling multiple 1-hour sessions folks could sign up for. There was a brief set of slides, explaining the project and getting folks bootstrapped and installed. Then we used Zoom to breakout into small groups of 4 devs, to solve the specs as a little team. Regrouping at the end to discuss and share our solutions. The workgroup format was broken down like below:

introduce the project and the workflow (10m)
solve a single flaky test with the whole group mob programming style (10m)
break up into groups and have the groups solve flaky tests (30m)
regroup, share solutions, and discussion (10m)

If you try running a workshop, let me know I would be curious how it went.

Here are some other helpful links related to flaky Rails tests.

Flaky Test Detection, help detect and find tests causing flakiness
make it easier to debug flaky tests, 5-ways to improve flaky test debugging
test-smells, examples of bad JS and Ruby test patterns
fixing flaky tests like a detective
Tips and Tricks for Dubugging and Fixing Slow/Flaky Capybara Specs
How CSS Animations Can Break Your Tests
Flaky Tests - A War that Never Ends

What I learned about Software Development from building a Climbing Wall

2019-11-20T00:00:00+00:00

Theo liked to imitate as he was learning to walk.

TL;DR: This post is just for fun!

I didn’t really learn about programming, this is just a catchy title, and I wanted to share a big project I have continued to work on that has nothing to do with programming. I thought I could make a kind of funny post by stringing together a bunch of programming ‘wisdom’ that could really be associated with nearly anything.

climbing wall’s current state

Prototype, Iterate, Test, and Iterate More

Climbing Wall: If you aren’t sure how much you will use the wall, or how much effort you want to put into it you can start small. Build and expand over time.

Software: In software, most of the time these days folks take an agile / lean approach and try to deliver working MVPs along the life of the project to deliver continuous value and learning.

Prototype

I started really small, initially with a training board. I installed this above the stairs entering the basement.

A training board lets you strengthen your fingers, and do weird pull-ups

Iterate

I then added a few holds directly into the wall near the original board so I could string together a few moves.

A first box of bulk climbing holds, a test mounting on a board

Iterate Again & Again

Then I added a single set of boards on the wall… and started to expand out from there.

empty wall, mounts, one board, and more

Project Planning

Negotiate with the Team

Climbing Wall: I did have to discuss everything with my wife before I started drilling a bunch of holes in our wall. This was a bit of a process, starting with the training board, the wall holes (and an agreement that I would patch any holes… hmm I still have to do some of those. A discussion that became easy as she found she enjoyed adding frequent climbing into her exercise routine as much as I did.

Software: In software you are always working with stakeholders, PMs, designers, other developers, and hopefully directly with some customers. Nearly everything built requires negotiation and compromises on time, features, UX, etc.

negotiation with my partner

It Starts with A Plan

Initially, I started with just a small idea, but it expanded out, especially after my wife decided she also really enjoyed climbing on the wall as well.

planning and materials

Ensure Your Project has a Reasonable Learning Curve

Climbing Wall: On a climbing wall you want it to be fun for beginners, kids, and more experienced climbers. Just like in software you want a project to be accessible and learnable by new hires and developers of different skill levels.

Our wall was a big hit with so many kids, I have added a lot of easier holds and built a lower route so small children can jump on the wall and have a great time.

Software: You can’t build a team, recruit, and mentor folks if you have a software project that is all expert level. Ensure it is easy to set up the apps development environment and it is easy to add features, test, and deploy safely so new folks can learn with confidence.

Theo Climbing, click for video

Set Milestones

On personal projects as in software, you want to set short, medium, & long term goals.

Short: something at home to help with training for climbing
Medium: I want to connect all the reasonable basement walls for a long route
Long: I want to connect the original training board to the route along with ceiling holds
Longer: I want to add a removable incline that can add a step grade when good climbers are visiting

Adding Ceiling Holds was a longer-term goal

Celebrate Your Wins

Climbing Wall: In general, whenever I expanded the climbing wall, I would quickly add some holds and celebrate by climbing my new longer route.

Software: Your team should be proud and get to celebrate after shipping something big. Also, ensure developers are sharing the things they learned along the release with the team. Having space to make investments and to pay down debt, requires that everything can’t always be moving at maximum speed all the time. Celebrate the progress folks are making.

Climbing the Routes as the wall is in progress

Erin testing our new “door crossing” problem

Plan For Growth

Climbing Wall: On the wall, I would build and leave space to add more when I had the time.

Software: In software, you want code that is flexible and easy to adapt. This doesn’t mean to over optimize, but know when to be specific and when to offer flexibility (the rule or threes can help with this).

Gaps When, low on Supplies

Make it work, make it right, make it fast

Climbing Wall: The wall was built to keep up and extend my skill level… When I didn’t have the parts or the time, I would sometimes make something fast and leave gaps to extend the routes.

Software: In software, make sure you can get it working, this ensures you solve the hard problems. Then make it right soft the edge cases and the gotchas. Then make it fast and scalable.

color coded climbing routes

Learning & Growth

Climbing Wall: As I worked on the wall project I became better with tools, building, designing routes, & more. I got comfortable and started to think up some more complex projects.

Software: Software takes practice, you will get better the more you build things. Learning which practices to follow and which don’t scale well.

You Can Learn Anything

Climbing Wall: I seriously know very little about building things, tools, construction, or really even climbing. All the information you need is available online to learn so much about any topic that interests you.

Software: You are always learning in software. A new framework, language, domain, etc… The field changes so fast that you have to keep learning to stay up to date. To know when something is a fad or is really worth investing time in deep learning.

You Get Better with Practice

Climbing Wall: Originally, I barely knew what size drill bits, bolts, nuts, and holds… Now I can put all this together and set up a new board in almost no time at all.

Software: It is good to keep practicing… Often this is how you learn to navigate all the grey areas of programming. The best solution to a problem isn’t always black and white, often the best practices have edge cases… Learning what to bend and what to break comes with practice and experience. Feeling the pain of maintaining systems over time, knowing what will stick around and what code often just gets removed.

over time, building became faster

It Takes A Team

Climbing Wall: You can’t hold up 4x4 plywood and drill it in yourself… Building a climbing wall requires teamwork. Collaboration to successfully complete the project.

Software: In software, most projects can’t be done by a single developer anymore. It takes collaboration, coordination, and teamwork to build something that lasts.

A few of the friends who have helped out

Learn From Prior Art

Climbing Wall: I read a number of things to learn how to build a climbing wall, this free build a climbing wall e-book from Atomik is great. No reason to try to learn from scratch.

Software: Not often are you building a program from scratch with no prior art. Learn from existing frameworks, applications, books, and open source. Build on the shoulders of giants as they say.

need inspiration? google climbing walls

Start Cheap and Upgrade Later

Climbing Wall: I started with some cheap holds, but over time I upgraded to nicer holds over time as I spent more time on the wall and as I expanded it. In the end, I really love atomik climbing holds, and I buy most of my equiptment there.

Software: In a software startup or a feature, you want to find the fastest and cheapest way to verify the value of something. When you know there is a value and have been able to build something sustianable (or I guess in startup world, with a hockey stick growth), you might want to move on from “it works” to it is best in class… Particularly, for things that aren’t part of the core company business value.

Cheap bulk grey holds, later upgraded to various specialty holds

Safely Removing Image Assets from Rails

2019-10-28T00:00:00+00:00

photo credit cleaning: pixabay

Why Cleanup Rails Image Assets?

Why would we want to more safely delete image assets?

a clean repo is easier to jump into and maintain over time
cruft that isn’t in use can be confusing over time
image assets can slow down your test and deploy pipeline
- Rails tests frequently need to dynamically or initialize building all assets, this is often a slow hit on a Rails test suite
Deployment needs to work with assets as well, often multiple steps
- building all assets rake assets:precompile
- compressing asset bundle
- uploading assets to a CDN

All of this time adds up, assets compilation on a large legacy Rails app still adds around 40 seconds to the build time, down from 1m30s in the past. While asset preparation and deployment before cleanup and optimization of that flow and files was adding over 3 minutes to our deploy time.

How To Safely Delete Image Assets

OK, hopefully now you would love to delete all the images in your app/assets/images folder that you don’t use… What images are safe to delete or out of use? I have looked at a number of ideas for this.

grepping with various scripts
using log aggregation search results to ensure no hits were being made of an image asset
using Sprockets options, unknown_asset_fallback alone, doesn’t make you entirely safe…

What I wanted was a way to delete a folder of images or a single image that I believed was no longer in use, but have the build fail if there was any reference to the image. I wanted Rails to fail in these cases:

a page is loaded in dev mode referencing a missing asset
tests are run against a page referencing a missing asset (ActionDispatch::IntegrationTest, request spec, etc)
bundle exec rake assets:precompile

Sprockets Unknown Asset Fallback

Not surprisingly, other folks have wanted this and Sprockets has a built-in option config.assets.unknown_asset_fallback, which gets close to what I wanted. From the docs, this option claims to:

When set to a truthy value, a result will be returned even if the requested asset is not found in the asset pipeline. When set to a falsey value it will raise an error when no asset is found in the pipeline. Defaults to true.

So let’s set it to false: Rails.application.config.assets.unknown_asset_fallback = false. Now if you have a deleted image referenced like below:

<%= image_tag("deleted_image.svg") %>

You will get an error when visiting the page or running tests:

bundle exec rake

...S.......E

Error:
HomeControllerTest#test_should_get_index:
ActionView::Template::Error: The asset "deleted_image.svg" is not present in the asset pipeline.
    app/views/home/index.html.erb:6:in `_app_views_home_index_html_erb___957919561084124106_70092585694780'
    test/controllers/home_controller_test.rb:5:in `block in <class:HomeControllerTest>'

This doesn’t make one entirely safe, as images that are referenced in your scss, css, or other styles would still not cause an error. They would silently lead to broken images.

Patch To Force Asset Compilation To Fail on Unknown Assets

Sadly, I couldn’t find any option or configuration that would cause compiling stylesheets to fail. I thought this would block my goal of being able to remove a ton of assets safely with confidence… Well, after lots of digging, I figured out how to patch sprockets-rails so that it will blow up and fail when it references an unknown asset.

You can apply this patch in your: config/initializers/assets.rb

Now if you have a file in your styles, like app/assets/stylesheets/application.scss reference an image, your asset pipeline will blow up when the image is missing.

.broken-image-demo {
  background-image: image-url('deleted_image.svg');
}

Depending on how your tests run, they will fail when precompiling assets, or a failure will occur when you call rake assets:precompile as shown below.

bundle exec rake assets:precompile
...
Done in 1.32s.
rake aborted!
Sprockets::Rails::Helper::AssetNotFound: path not resolved: deleted_image.svg
/Users/danmayer/projects/coverband_demo/config/initializers/assets.rb:56:in `rescue in compute_asset_path'
/Users/danmayer/projects/coverband_demo/config/initializers/assets.rb:51:in `compute_asset_path'
/Users/danmayer/.rvm/gems/ruby-2.6.2/gems/actionview-5.2.2.1/lib/action_view/helpers/asset_url_helper.rb:200:in `asset_path'
...

Asset Failure Demo

If you want to see this in action, feel free to clone coverband demo. Install gems and get the tests passing. Then read the comments and run tests or compile assets when uncommenting the example lines to trigger the expected errours. Key Files:

config/initializers/assets.rb, this shows all the setuo and configuration needed
app/assets/stylesheets/application.scss, an example of stylesheets referencing a missing image
app/views/home/index.html.erb, an example of a view file referencing a broken image

A Final Note

On an old legacy application we were able to delete over 50% of the total asset disk size, by clearing out old unused image assets. This made it easier to find and navigate our assets folder, and it significantly sped up both our test suite and deployment. While I wouldn’t expect most projects to have as much image cruft sitting around, especially with older applications, it is easy for these unused assets to really build up over time.

While the above, should make it easier to delete image assets and do housekeeping yourself, it still takes a bit of time. You need to go through a process:

find a likely set of unused images
delete them, run tests
add back images that were still used
repeat until satisfied

This obviously looks like a process that can be automated to help you clean up all your image assets automatically. That is totally true, and I will cover how to do that in a future post. This post covers what is a prerequisite to being able to automate the cleanup, ensuring that your app will properly and very loudly fail when an image was removed which is still required.

Flaky Ruby Tests

2019-09-07T00:00:00+00:00

Restoration of a Ruby Test Suite

I want to talk about some recent work to restore a Rails app’s test suite to a usable state. The goal went beyond the test suite, to restoring trust in the continuous deployment pipeline, but this post will mostly focus on the Rspec suite. The reason I started on this work was that the current state of deployment was “dangerous”, various folks preferred to avoid the project as it was difficult to work in and release, but still critical to our overall architecture. At it’s very worst, deploys were taking over 30 minutes, with a failure rate of the deployment pipeline of 45%. The issue became clear and high priority to me when one day, I had two small PRs to release, due to bad luck with the failure rate, it took me nearly 6 hours to get my changes deployed. A constant distraction that dragged on through meetings, and other work. Making the pains of the team extremely clear and personal, I decided an effort to get things back into a safe state should be taken on.

I wanted to share some of what I learned as there has been some recent discussion in the Ruby community about fixing flaky tests @samsaffron @tenderlove @ctietze @SonjaBPeterson @saramic.

Over the years one of the most complex liabilities we carried in our test suite has been flaky tests. This problem is so hard some people just give up. We are slowly cataloging the problems here: https://t.co/7AdwOGNfNw , I hope to write about it.
— Sam Saffron (@samsaffron) May 7, 2019

A Restoration Plan

Taking a step back, thinking about what needed to happen and how to get there was the first step. I thought this would fit in well to the old agile advice…

Make It Work. Make It Right. Make It Fast.

Kent Beck

Make It Work

What didn’t work about the deployment? In my opinion, it was broken because:

doesn’t meet a 95% or better success rate
deploys are too slow, to watch and review if changes succeeded, 10 minutes of less
test suite relying on CI parallelism is to slow to ever run locally, local suite run needs to be possible in 1hr or less.

With a definition of what success looks like to make it work, then I was able to start to dig into the details of how to get there.

Delete Flaky Tests That Aren’t Valuable

This is often a hard one for some teams. An honest discussion of the purpose and value of tests is likely needed. I found good success by having a small PR removing a few flaky tests, and pointing to similar tests in the suite that exercised functionality in a more reliable way. For example, moving from a complete feature spec that tested several endpoints in a long workflow, to a couple of feature tests exercising individual endpoints, along with unit tests for the underlying service providers. The team might need to have a discussion ahead of time, or you might be surprised that others quickly agree that really flaky tests aren’t providing value.

Flaky tests are worse than useless tests
— Aaron Patterson (@tenderlove) August 30, 2019

Fixing Flaky Tests

The primary cause of our unstable deploy pipeline was flaky tests. Given we were having deployments fail 45% of the time we had a large number of flaky tests causing issues. Let’s dive into some of the techniques for resolving flaky test failures.

andrewhalliday

Divide and Conquer, with Rspec-Retry

Initially quarantine helps to reduce their damage to other tests, but you still have to fix them soon.

Martin Fowler, Eradicating Non-Determinism in Tests

Since before I joined the project, it has used rspec-retry as a way to isolate some flaky tests. Yes, this is a band-aid, but in terms of getting back to a “make it work” baseline, it is a wonderful tool. This is how I used it. For a period of about a month, I watched every failure on our CI spec suite. Every time a test failed more than once, I would add it to an internal wiki marking the test as flaky. Myself and other folks from the team, when they had time, would “adopt” a test and try to fix it, if one timeboxed an hour or two and couldn’t figure out and fix the underlying issue, we would tag it flaky, so that rspec-retry would run the test multiple times trying to achieve success. We ran our flaky tag specs in a special CI job, bundle exec rspec --tag retry_flaky_test isolated from our other tests. This CI job had a success rate of 99%, so the flaky tests would pass on retry, and be split off from others. Then with logging, and debugging we could dig in deeper in resolve the underlying issues and move the test back into the standard test suite. This is great because it very quickly got the test suite back into usable condition and tags all the future work still needing to be addressed and captures metrics about which tests are most flaky, or no longer are flaky as we resolve issues. At our current stage, we still need to go back and resolve a number of flaky tests, but they no longer slow down or block our CI.

Isolate Flaky Test Recipe:

capture data and identify flaky tests (use metric tracking, or just eyeball if from your CI history)
quickly try to fix them timeboxed to a short amount of time
if you can’t fix them, tag them for rspec-retry, to isolate them and remove them as a blocker for CI
Find a way to distribute this process across folks on the team, and explain the plan on how to follow through with cleanup.

Fix the Worst Offenders

From above you likely will find some worst offender tests or hopefully patterns that impact multiple tests. These even with flaky test rspec-retry may still fail to often to be reliable. If you dig into a few of the tests (during the timeboxing) you are likely to find some patterns. For example @TildeWill, fixed a whole class of flaky tests related to Capybara negative matchers. We also fixed entire categories of failing tests that weren’t properly using Capybara’s asynchronous matchers, each of these fixes added around 6% to the success rate of our CI suite per PR.

Common Flaky Test Issues

I won’t cover all the various types of flaky tests in as much detail as you can find in @samsaffron’s post, Tests that sometimes fail. Here are some of the most common issues we found while resolving issues.

fix timing issues (timezone dependant)
stale state issues due to non DB stores (global vars, redis, memcache, Rails.cache, etc)
fix order / dependency issues… Tests that only pass in a specific order
- running specs in documentation mode can really help find ordering issues… Run it this way every time so you have a clear log of what ran when ``time bundle exec rspec –format documentation`
Capybara
- devise auth issues 1 2
- not using the aysnch matchers
  - example: expect(page.current_url).to eq(/something/) is bad, switch to waiting version expect(page).current_path.to eq(/something/) which is good.
- using sleep opposed to correct waiting matchers
- in geneal wait_for_ajax is dangerous
VCR allowing non-stubbed network requests can be dangerous, try to get your suite passing with, VCR.configure { |c| c.allow_http_connections_when_no_cassette = false }

Tips and Tools to Fix Flaky Tests

A few tips to help debug and fix flaky tests. I found each of these scripts extremely valuable in moving forward our success rate.

Single Flaky Test Case

Quickly, verify a test that fails randomly even in isolation, with this great tip.

from @_swanson

Single Flaky Spec File

Quickly, check the success rate of a single file. I use this to report a before and after of a fix in a PR.

Calculate Success Rate of a Branch on CircleCI

Push a fix on a branch and run it a number of times to see if it improves the overall success rate.

Additional Resources on Resolving Flaky Tests

BuildPulse, software service that helps identify flaky tests.
Capybara Cheat Sheet, ensure you are following the best practices, to avoid common mistakes.
Write Reliable, Asynchronous Integration Tests With Capybara
Capybara::SlowFinderErrors
Tearing Down Capybara Tests of AJAX Pages

Make It Right

Most of the remaining work on restoring this test suite now falls into this category. The deployment pipeline succeeds at over a 95% success rate at around 10m. These are acceptable numbers for our project. What we haven’t done is resolve all of the flaky tests which pass because of retry attempts. Until we can move all the tests to be fully reliable there is work to be done.

Make It Fast

I will dive into this more in a future article, but with some effort, the team was able to get our CI deployment pipeline down from over 30m on avg to only taking 10m on average. The CI jobs to run just our tests are down around 5m, with the full deployment jobs taking longer. I expect as we continue to make improvements and fix some of the known bad actors in our test suite, this number will continue to go down. Why did we make so much progress on “Make It Fast” before finishing “Make It Right”? Well, we needed a better and faster feedback loop to find and fix flaky tests, as well as to make it right. A fast feedback loop is really required to make progress quickly. Until we could increase the iteration cycles, we could only have so many flaky fix PRs make it through the pipeline in a day, and at the beginning testing locallt wasn’t really possible. In terms of make it fast, I did want to mention there are still two efforts under way.

Local Test Suite Speed

If the test suite it to slow to ever run locally it is also hard to test and ensure it reliably runs anywhere other than on CI. Initially, the test suite was so slow, it would either crash or stall out most of the time. Occasionally with many failures, it would complete after 9+ hours… After using CI to drive most of the fixes, now the local spec suite reliably runs on a single machine in 45 minutes. This is still far to slow for my liking but is headed in the right direction.

Deployment Pipeline Improvements

The CI deployment pipeline is the test suite, but also much more. This article isn’t going to focus on deployment improvements but without changes related to the tests or fixing flaky test failures. Various improvements cut our deployment in a third, I will detail this more in a future article. This involved breaking down all the continuous deployment steps finding inefficiencies, redundancy, and improving parallelization.

Building SVG Badges in Ruby

2019-04-10T00:00:00+00:00

Building SVG Badges in Ruby

A while ago I needed to create a simple dynamic SVG. After a bit of tinkering, it was easy enough to build a simple Ruby class to do what I needed to do.

SVG created from the below code

It was a super quick thing to put together and solved a problem I was having. I was reminded of that today when I had another quick issue that I wanted to resolve. I wanted to pull some data not available by a services APIs, the data I needed was easily accessible to me in their webviews, so I quickly hacked together a web scraper, pulling the data I needed into a google sheet.

Being a Developer, Unblocks You

A thing I have always loved about being a developer is you can solve your own problems. You have to be careful to not get sucked into it and wasting a bunch of time, but you also aren’t blocked just because something you need isn’t already available. If you build a quick hack ensure it is just that a quick hack and that you won’t regret putting it into your workflow. The below SVG example was a tiny helper for devs, while today’ journey helped pull metrics on CircleCI related to metrics I want to pull for myself monthly. In either case, if it breaks it is no issue and can be fixed in a few minutes.

Being able to solve the problems you run into along the way is one of the great parts of being a developer.

Full Ruby SVG Badge Code

class SvgFormatter
  def initialize(output = nil)
    @output = output || STDOUT
  end

  def format(result)
    percentage = SOME_DATA_SOURCE.round(1)
    File.open('badge/results.svg', 'w') { |f| f.write(template(percentage)) }
  rescue RuntimeError => e
    @output.puts e.message
    @output.puts 'SvgFormatter was unable to generate a badge.'
  end

  private

  def template(percentage)
    file_content = <<~SVGTEMPLATE
      <?xml version="1.0"?>
      <svg xmlns="http://www.w3.org/2000/svg" width="90" height="20">
        <linearGradient id="a" x2="0" y2="100%">
          <stop offset="0" stop-color="#bbb" stop-opacity=".1"/>
          <stop offset="1" stop-opacity=".1"/>
        </linearGradient>
        <rect rx="3" width="90" height="20" fill="#555"/>
        <rect rx="3" x="51" width="39" height="20" fill="#007ec6"/>
        <rect rx="3" width="90" height="20" fill="url(#a)"/>
        <g fill="#fff" text-anchor="middle" font-family="DejaVu Sans,Verdana,Geneva,sans-serif" font-size="11">
          <text x="24.5" y="15" fill="#010101" fill-opacity=".3">Title</text>
          <text x="24.5" y="14">Title</text>
          <text x="68.5" y="15" fill="#010101" fill-opacity=".3">#{percentage}%</text>
          <text x="69.5" y="14">#{percentage}%</text>
        </g>
      </svg>
    SVGTEMPLATE
    file_content
  end
end

Zola Electric Career

2019-02-14T00:00:00+00:00

Zola Electric Career

Tomorrow is my last day at Zola Electric, it is time for a different challenge. It has been an amazing ride over the last 4+ years. Helping grown from a single country with non connected devices to smart devices that can power a large variety of appliances. During my time with Zola it has grown to provide energy access to over 1 million people without grid access in Africa. It has been a rewarding and challenging experience. I wanted to share my favorite story from working at Zola over the last few years.

My first installation in Tanzania

Want to see it in action, visit the Coverband Demo Site, visit a few pages and watch the coverage report change as you use the site.

Satisfying Work

On my first work visit to Arusha, we had scheduled a number of activities for me to better understand the business and our operations. One of which was to go watch an installation occur near Arusha. On arrival, the children were playing in the yard and extremely excited to see us visit. After a bit of discussion and planning of where cabling, lights, and the M-series box will go we get started. I volunteer to install the panel, with no ladder available, I climbed a tree and jumped onto the tin roof attached the solar panel in place. Using a screwdriver to part the roof panels to run the panel cabling through the roof to the box. I came down to help finish install some of the lights. The mother holding the youngest baby turns on her Zola lights for the first time and the kids’ cheer and dance by her feet while a smile of pride and happiness fills her face. It is possibly the best and most satisfying work moment of my life.

The happy mother after installation of lighting system

More Photos!

Climbing to the roof

Jumping over to the roof, sure hope it holds me!!!

Hanging a light as part of the installation, seen earlier with the Mother below

Coverband 3

2018-10-03T00:00:00+00:00

Coverband 3

Coverband gives deep insight into the production usage of every line of code in your Ruby app. Find dead code, find hotspots, see code paths that might be unexpectedly in high usage.

This release marks deep dives into Ruby performance, benchmarking, and driving almost all features based on improving performance. The release is a majory performance improvement, resolving some issues were Coverband 2 could have outliers that performed worse than the original Coverband 1 gem.

Me first and the Gimme Gimmes, a punk cover band, image from Wikimedia

Want to see it in action, visit the Coverband Demo Site, visit a few pages and watch the coverage report change as you use the site.

What breaking changes are there?

drops Tracepoint
drops Ruby <= 2.3.0
drops JSON Gem dependency
drops various other features not needed without Tracepoint
- memory cache, sampling, restricted to app folders, etc

What is new?

standardizes on Coverage array format vs sparse hash
rewrites store methods, for 60X perf!
- implemented for Redis and File store
improved mountable web interface

What exactly is faster?

tldr;

Storing Coverage data to a data store. While this didn’t happen often when it did it could generate thousands of Redis requests on large apps, which could make for a very slow outlier request…

Long Version

The primary improvements come down to changing the storage format and moving from method of storage that required one call per file and assummed a small list of files, to a 2 pass stoage for all data. Basically, the previous storage mechanism created an N+1 type issue. Coverband 2 had solved the issue of capturing Coverage data, which was no longer a performance concern, in fact it captured all files by default as there wasn’t really a reason to filter for performance anymore… This created a new problem when trying to store coverage data. Eventually a very specific benchmark was added that simulates trying to store coverage reports for apps with 2934 files tracking coverage. In Coverband 3 by changing the storage format and the storage plan the storage benchmark improved by nearly 60X!

Simulate repeatedly storing coverage reports with Coverband 2.0.3:

rake benchmarks:redis_reporting
runs benchmarks on reporting large sets of files to redis
Warming up --------------------------------------
       store_reports     1.000  i/100ms
Calculating -------------------------------------
       store_reports      0.814  (± 0.0%) i/s -     13.000  in  15.982911s

Simulate repeatedly storing coverage reports with Coverband 3.0.0:

rake benchmarks:redis_reporting
runs benchmarks on reporting large sets of files to redis
Warming up --------------------------------------
       store_reports     4.000  i/100ms
Calculating -------------------------------------
       store_reports     47.111  (± 4.2%) i/s -    708.000  in  15.066259s

How did this impact Rails Apps

In general, depending on various settings the outliers weren’t that noticable or it made Coverband unusable for folks. I configured a Rails app with non ideal, but not terrible options, to show what this could look like and how Coverband 3 resolves the issue. If configured idealy even Coverband 2 would mostly only show in outliers, so this test is showing how even in a bad setup V3 performs with nearly no impact.

The benchmarks below are a further iteration of my AB Benchmarking Rails apps process. Now much easier to read and compare, by graphing by following this Apache Bench via Gnuplot guide.

No Coverband	Coverband 2	Coverband 3

mean: 27.272 [ms]	mean: 35.762 [ms]	mean: 28.460 [ms]
36.67 [#/sec] (mean)	27.96 [#/sec] (mean)	35.14 [#/sec] (mean)

Notice that with no Coverband and Coverband 3, the majority of requests show little variation, while Coverband 2 has frequent outliers pushing request timing all over the place. See full sized images of the benchmarks below.

What is next?

I am glad you asked, as we introduced a roadmap and set of feature discussions for upcoming Coverband improvements. Check out the changes.md for the latest, but in short here is some of what we have in the works.

full Railties support, near zero config Rails integration
solving the coverage drift problem…
adding out of band data collection methods.
droppoing simplecov dependency
adding an Active Record adapter

Please give Coverband a shot and add any issue you have to the repo.

Footnotes

For those crazy few that really want all the data on the benchmarks…

No Coverband

rake benchmarks:coverband_demo_graph
Benchmarking coverband-demo.herokuapp.com (be patient)

Concurrency Level:      10
Time taken for tests:   54.543 seconds
Complete requests:      2000
Failed requests:        0
Requests per second:    36.67 [#/sec] (mean)
Time per request:       272.717 [ms] (mean)
Time per request:       27.272 [ms] (mean, across all concurrent requests)

Percentage of the requests served within a certain time (ms)
  50%    262
  66%    269
  75%    273
  80%    276
  90%    288
  95%    302
  98%    328
  99%    350
 100%   1400 (longest request)

Coverband 2

rake benchmarks:coverband_demo_graph
Benchmarking coverband-demo.herokuapp.com (be patient)

Concurrency Level:      10
Time taken for tests:   71.524 seconds
Complete requests:      2000
Failed requests:        0
Requests per second:    27.96 [#/sec] (mean)
Time per request:       357.619 [ms] (mean)
Time per request:       35.762 [ms] (mean, across all concurrent requests)

Percentage of the requests served within a certain time (ms)
  50%    325
  66%    353
  75%    378
  80%    398
  90%    489
  95%    545
  98%    622
  99%    682
 100%   1442 (longest request)

Coverband 3

rake benchmarks:coverband_demo_graph
Benchmarking coverband-demo.herokuapp.com (be patient)

Concurrency Level:      10
Time taken for tests:   56.919 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      16565120 bytes
HTML transferred:       14974000 bytes
Requests per second:    35.14 [#/sec] (mean)
Time per request:       284.597 [ms] (mean)
Time per request:       28.460 [ms] (mean, across all concurrent requests)

Percentage of the requests served within a certain time (ms)
  50%    273
  66%    282
  75%    290
  80%    296
  90%    319
  95%    344
  98%    381
  99%    410
 100%   1358 (longest request)

Lead Developer London

2018-07-18T00:00:00+00:00

A Unique Conference

I attended #LeadDevLondon this year, it was a nice and different conference. Generally, I have attended more language-specific conferences. There were basically no language-specific talks at all during the conf, as everything was a bit more general about building and working well with technology teams. Given that these days a large part of what I do is manage teams of devs, it was good to hear other folks talk through some of the challenges and solutions they have gone through during their journey.

My swag bag full of stickers

Some Favorite Slides

For my very favorite slide see the section with my tweets below ;) Otherwise, enjoy some favorite slides with little to no context ;) You can generally find the full presentation slides on the talks page and the full videos of the #LeadDevLondon talks on youtube.

@alicegoldfuss explaining containers

@alicegoldfuss explaining what else ops does ;)

@alexhillphd going over the art of code reviews, this talk had my favorite slide of the conf

@cmccarrick how to scale yourself and make important decisions

@tara_ojo on how to help Jr’s with good 1:1s

@nmeans lessons learn from 3 mile island

@ClareSudbery

@dbussink shows how distributed his team is

@dbussink explains how to make that an advantage

@WebDevBev on how to run awesome internships

@jqiu25 the journey to safe ways to edit production data

Four Slides From The Legacy Code Talk

I guess I really liked what @ramtop had to say about legacy code

A Talk Pattern I Want To See

One thing I noticed and really came away with from the talks is that every piece of advice and best practice really fits into part of a companies growth curve. Even the most agreed on advice can’t and shouldn’t apply the same to a single person startup and a company of thousands of people and hundreds of engineers. This made me crave a talk that opposed to saying how some team approaches a challenge and the ways it works for them, but how a best practice or process changes and adapts to different growth stages of a company…

As a single talk, an example let’s consider continuous delivery of the growth of a team. While the end best practices involve style checkers, peer code review, automated test suites, deployment to one or more staging, and feature flag or percentage based releases to production, before/after metric analysis on the impact of deploys… It would not be possible to start there nor would it be sensible to set up all that complexity when just getting started… I feel like the flow would naturally look something more like this.

1 person startup -> manually run CLI deployment task from the dev machine
2-3 people -> CI test suite, manual CLI deployment
3-8 people -> static style checker, CI test suite, code review, staging deployment with stakeholder / QA review, push-button deployments and rollbacks automated on CI
8-16 -> all the above, with multiple staging environments, some integrated with 3rd party sandboxes, feature flag based releases, metrics tied to specific deployed releases, and deployment queue to help schedule deployments
etc, etc, etc…

I think it would be great to see detailed talks on tech org structures, testing, monitoring, alerting, etc..

My Tweets

Some of the tweets I shared during the conf.

Great to see some old friends at #LeadDevLondon and introduce them to Theo... @mariagutierrez got a nice hug... pic.twitter.com/bUIjOUrdff
— Dan Mayer (@danmayer) June 27, 2018

My favorite slide from #LeadDevLondon so far.. Also, why our team tries to have linters as part of our CI build... Have higher level PR discussions... pic.twitter.com/9gandnj3Q5
— Dan Mayer (@danmayer) June 27, 2018

Thanks #LeadDevLondon after the conf, Theo got to see the analytical engine and difference engine at the London Science Museum... Which is fun since I am reading him a book about Ada Lovelace and Charles Babbage. pic.twitter.com/sK6VxRnhaQ
— Dan Mayer (@danmayer) June 28, 2018

The Most Important Part, Stickers

Seriously, thanks to all the sponsors and friends handing out stickers. I love collecting stickers at conferences to come back and cover my climbing wall. I am building a large collage covering the entire climbing wall in layers of stickers. After returning from a conference I always have a huge collection of stickers to use, which is great. In this case, I basically was able to make an entire tile of my climbing wall a #leaddevlondon tile full of stickers I picked up during the trip.

If you don’t know what to do with all the stickers you have collected at various conferences, feel free to send them my way ;)

Benchmarks Bugs

2018-06-10T00:00:00+00:00

Benchmarking Bugs

On my previous post I covered benchmarking Rubygems… I was working on some follow up to improve the benchmarks and discovered a bug rendering one of the benchmark comparisons invalid. In this post we will cover what I had missed, how to avoid some gotchas, and ways to improve the readability of your benchmarks.

image from pixabay

Spot The Bug

This is a simplified example from the source code. The bug is hard to notice without knowing the internals of the library, but if you understand Ruby’s Coverage library you might spot it.

namespace :benchmarks do
  desc 'set up coverband with coverage redis'
  task :setup_coverage do
    clone_classifier
    $LOAD_PATH.unshift(File.join(classifier_dir, 'lib'))
    require 'benchmark'
    require 'classifier-reborn'

    Coverband.configure do |config|
      config.root               = Dir.pwd
      config.percentage         = 100.0
      config.logger             = $stdout
      config.collector          = 'coverage'
      config.memory_caching     = ENV['MEMORY_CACHE'] ? true : false
      config.store              = Coverband::Adapters::RedisStore.new(Redis.new)
    end
  end
  
    def run_work
    puts "benchmark for: #{Coverband.configuration.inspect}"
    puts "store: #{Coverband.configuration.store.inspect}"
    Benchmark.bm(15) do |x|
      x.report 'coverband' do
        SAMPLINGS.times do
          Coverband::Collectors::Base.instance.sample do
            work
          end
        end
      end

      x.report 'no coverband' do
        SAMPLINGS.times do
          work
        end
      end
    end
    Coverband::Collectors::Base.instance.stop
    Coverband::Collectors::Base.instance.reset_instance
  end

  desc 'runs benchmarks coverage'
  task run_coverage: :setup_coverage do
    puts 'Coverband <!--more--> Coverage configured with to use default redis store'
    SAMPLINGS = 5
    run_work
  end
end

The issue is that the benchmark is trying to compare running Coverband with Ruby’s Coverage lib against code, which doesn’t collect usage data. When using Ruby’s Coverage is loaded and started, it changes how ALL other code is loaded and interpreted. Which means once the Coverage library is loaded both benchmarks will run with the performance impact. The only difference is once is collecting and reporting the coverage to Redis, via Coverband. Since the Coverage library effect the Ruby runtime it is best to run the benchmarks as entirely isolated processes to avoid blending the impacts of one benchmark into the other.

Spot the Bug in the Stats

Even if one doesn’t notice the issue in the code, the output from the benchmarks should make the issue stand out. In my previous post I noticed that I could no longer detect performance impacts of Coverband, that seemed to good to be true.

                      user     system      total        real
coverband         0.320000   0.010000   0.330000 (  0.322387)
no coverband      0.320000   0.000000   0.320000 (  0.321767)

It also didn’t match with real world Rails benchmarks. At the time in that post, I believed my benchmarks just weren’t sufficient to capture the small impact. Wanting to make my benchmarks easier to read and to be able to detect the known performance impact is what caused me to find the bug.

Making Benchmarks More Readable

As I was working to detail the the performance impact of Coverband, I decided I should port my benchmarks from the std-lib benchmark, to the excellent benchmark-ips gem by @evanphx. The gem has output in the format below. The new format in comparison to the default format, makes it much easier to understand performance differences. In this case because of the bug to see clearly a missing and expected difference. The improved Gem also helped me find better iterations to detect changes on my existing and working benchmarks.

Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
        no coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband     14.690  (±27.2%) i/s -    149.000  in  12.045429s
        no coverband     15.112  (±33.1%) i/s -    151.000  in  12.240970s

Comparison:
        no coverband:       15.1 i/s
           coverband:       14.7 i/s - same-ish: difference falls within error

The improved format along with increased iterations (5 seconds warmup and 12 seconds runtime) made it clear that there was truly ZERO difference between the Coverage benchmarks, which was wrong. This makes it very clear a there was a bug in my methodology.

Fixing the Issue

Once I realized Coverage was loaded for both runs, it was pretty clear what was wrong. I needed to do a couple things.

The Coverage lib needs to be both required and started before any code that we plan to track (this also wasn’t done in the previous benchmark, so actual coverage data wasn’t being collected correctly.)
Since Coverage changes how Ruby actually interprets the code when loaded. I needed to run the two timed tasks independently and finally compare the data.

It was pretty easy to port my old benchmark code to benchmark-ips, and to clean things up along the way. As well as handle both of the mentioned issues. See the new benchmark code below (simplified here, see Github for the full src code details).

namespace :benchmarks do
  desc 'setup standard benchmark'
  task :setup do
    clone_classifier
    $LOAD_PATH.unshift(File.join(classifier_dir, 'lib'))
    require 'benchmark'
    require 'benchmark/ips'

    # NOTE: When we require files is what makes performance impact of Coverage interesting (moving this above or below Coverage.start gives detectable vs not detectable performance impacts
    require 'classifier-reborn'
    if ENV['COVERAGE']
      puts 'Coverage library loaded and started'
      require 'coverage'
      ::Coverage.start
    end
    require 'redis'
    require 'coverband'
    require File.join(File.dirname(__FILE__), 'dog')
  end

  desc 'set up coverband with coverage Redis'
  task :setup_coverage do
    Coverband.configure do |config|
      config.root               = Dir.pwd
      config.percentage         = 100.0
      config.logger             = $stdout
      config.collector          = 'coverage'
      config.memory_caching     = ENV['MEMORY_CACHE'] ? true : false
      config.store              = Coverband::Adapters::RedisStore.new(Redis.new)
    end
  end
  
  def run_work(hold_work = false)
    suite = GCSuite.new
    #puts "benchmark for: #{Coverband.configuration.inspect}"
    #puts "store: #{Coverband.configuration.store.inspect}"
    Benchmark.ips do |x|
      x.config(:time => 12, :warmup => 5, :suite => suite)
      x.report 'coverband' do
        Coverband::Collectors::Base.instance.sample do
          work
        end
      end
      Coverband::Collectors::Base.instance.stop
      x.report 'no coverband' do
        work
      end
      x.hold! 'temp_results' if hold_work
      x.compare!
    end
    Coverband::Collectors::Base.instance.reset_instance
  end
  
  desc 'runs benchmarks coverage'
  task run_coverage: [:setup, :setup_coverage] do
    puts 'Coverband Coverage configured with to use default Redis store'
    run_work(true)
  end

  desc 'compare Coverband Ruby Coverage with normal Ruby'
  task :compare_coverage do
    puts 'comparing with Coverage loaded and not, this takes some time for output...'
    puts `COVERAGE=true rake benchmarks:run_coverage`
    puts `rake benchmarks:run_coverage`
  end
end

As you can see to handle the hard part of running the benchmarks independently and comparing them I used a cool feature of benchmark-ips. Loading the Coverage library or not is now controlled by ENV['COVERAGE'] making it easy to run with or without a change to how Ruby interprets the code.

Using benchmark-ips hold!

As I was starting to look at how to compare data on two runs, I checked to see if the new benchmarking gem I was using could help me. The documentation covered having a hold! feature, which seemed exactly what I needed.

If you are comparing multiple implementations of a piece of code you may want to benchmark them in separate invocations of Ruby so that the measurements are independent of each other. You can do this with the hold! command.
– benchmark-ips, independent-benchmarking

Sounds great, but the hold! wasn’t detailed very well in the documentation. In fact another user had created an issue for the project trying to figure out how to properly use the hold!. I decided to read through the source code to figure it out and verify that it could do what I needed.

PR with example benchmark-ips `hold!` Usage

Well long as I take the time to figure it out, and I know others have struggled to understand the usage. We might as well try to make the usage clear to everyone. The project has an examples folder detailing several usages. Adding a small PR with a hold usage example, gave a simple usage scenario. Which was a simplified version of the approach I used to benchmark the Ruby Coverage library performance impacts in Coverband.

Updated Benchmark Conclusions

After fixing the bug, benchmarking Coverage is still very interesting. Even small changes in when Coverage is started and which files are required, can have a massive impact on the overall performance. Take a look at diff below, which shows moving a single line require 'classifier-reborn' to be above or below when we require and start Coverage, which means the libraries code is either included or not in the Coverage collection.

-    require 'classifier-reborn'
     if ENV['COVERAGE']
       puts 'Coverage library loaded and started'
       require 'coverage'
       ::Coverage.start
     end
+    require 'classifier-reborn'
     require 'redis'
     require 'coverband'
     require File.join(File.dirname(__FILE__), 'dog')

The first benchmark shows when classifier-reborn is required prior to setting up Coverage and therefor it’s data is excluded from the results.

rake benchmarks:compare_coverage
comparing with Coverage loaded and not, this takes some time for output...
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband     11.950  (±16.7%) i/s -    140.000  in  12.041935s

Pausing here -- run Ruby again to measure the next benchmark...
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
        no coverband     1.000  i/100ms
Calculating -------------------------------------
        no coverband     14.497  (±20.7%) i/s -    161.000  in  12.020487s

Comparison:
        no coverband:       14.5 i/s
           coverband:       12.0 i/s - same-ish: difference falls within error

In this second benchmark we ensure the Coverage library is loaded and started prior to requiring classifier-reborn, which means we are capturing all the usage of that library code.

rake benchmarks:compare_coverage
comparing with Coverage loaded and not, this takes some time for output...
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband      9.117  (±11.0%) i/s -    109.000  in  12.108073s

Pausing here -- run Ruby again to measure the next benchmark...
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
        no coverband     1.000  i/100ms
Calculating -------------------------------------
        no coverband     14.184  (±14.1%) i/s -    164.000  in  12.017139s

Comparison:
        no coverband:       14.2 i/s
           coverband:        9.1 i/s - 1.56x  slower

In these examples the only difference is that the library classifier-reborn is either included in the recorded coverage or ignored by the Coverage library, based on when we require it. The improved benchmark code:

is easier to read than it was previously (both code and benchmark output)
it clearly captures and allows for easy performance tests of Coverband using Coverage
It makes easily clear how much of a win Coverage is over the older TracePoint API
- Coverage showing either no diff or 1.56x slower (depending on the coverage scope)
- Tracepoint showing around 3.95x slower

Full Benchmark Output

Here is the full benchmark comparison output across all benchmarked configurations of Coverband.

Coverband tracepoint configured with file store
Warming up --------------------------------------
           coverband     1.000  i/100ms
        no coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband      3.348  (±29.9%) i/s -     39.000  in  12.095922s
        no coverband     13.921  (±21.6%) i/s -    160.000  in  12.048469s

Comparison:
        no coverband:       13.9 i/s
           coverband:        3.3 i/s - 4.16x  slower

Coverband tracepoint configured with default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
        no coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband      3.726  (± 0.0%) i/s -     45.000  in  12.286824s
        no coverband     14.736  (±27.1%) i/s -    150.000  in  12.059519s

Comparison:
        no coverband:       14.7 i/s
           coverband:        3.7 i/s - 3.95x  slower

comparing with Coverage loaded and not, this takes some time for output...
Coverage library loaded and started
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband     12.173  (±16.4%) i/s -    143.000  in  12.017026s

Pausing here -- run Ruby again to measure the next benchmark...
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
        no coverband     1.000  i/100ms
Calculating -------------------------------------
        no coverband     15.045  (±13.3%) i/s -    177.000  in  12.024386s

Comparison:
        no coverband:       15.0 i/s
           coverband:       12.2 i/s - same-ish: difference falls within error

Driving Performance Impact Improvements Based on Learnings

Now with an ability to detect performance differences when running Coverband with Ruby’s Coverage library, I can build features and benchmarks to show how to reduce the performance impact on real world use cases.

For example in general with Coverband the goal is to show the coverage of application code ignoring framework and gem code. Knowing that which files are loaded at what time related to Coverage start can impact performance. I can build a feature to ensure that Coverband sets the Coverage library to ensure ONLY the application code is being tracked by Coverage, which will help ensure the lowest possible performance impact.

I will be working on this feature (Coverband Safelist) and new benchmarks to show how much application developers could reduce the performance burden on standard Rails apps by using a Safelist. Opposed to allowing various Rails framework code to be tracked along side their application code which is required by the current version of Coverband, given how Rails loads code. Look forward to another post, when that feature and benchmarks are ready.

Ruby Gems Benchmarks

2018-05-29T00:00:00+00:00

RubyGems Benchmarks

This follows up on my previous post focused on Ruby Benchmarking. This post will focus on benchmarking a gem as part of it’s long term maintenance. Building a community standard around Gem Benchmarking has even been suggested as a good way to help Ruby fine tune performance over time.

image from pixabay

Why Benchmark A Gem

If you maintain a Gem, which could have large performance impacts on folks integrating the gem. It can make sense to try to build performance testing into the Gem, helping to ensure that changes and new features don’t impact the performance in unexpected ways overtime. It also will help for you to publish performance impacts to give confidence to folks adding your gem as a dependency.

For example I maintain a Gem, Coverband, which records production code coverage. Obviously, this can have a major impact on performance as it tracks every line executed on production. During the life of Coverband, nearly every decision about features and Gem release involved some Benchmarking. The project was initially developed to help remove dead code from a large old monolithic rails app, before the first release could be put on production it went through a number of performance tests and only ran on staging until the perf impact could be acceptably controlled.

Coverband Benchmark Timeline

Coverband code has gone through several strategies to mitigate the performance impact. Let’s take a look at quick timeline of major changes. As Coverband has changed, we can check and compare the performance impacts with the Coverband Benchmark Tasks, which are explained later in this post.

2013: initial release
- performance cost of 100% recorded coverage, around 8X slower Rails requests
- initial release performance costs weren’t well explained with no released benchmarks
- Performance costs were mitigated by sampling a percentage of requests
- As well as safelist & blocklisting files to track
2014: released coverband_ext (C extension for fast access to tracepoint API) in
- benchmarks for 100% recorded coverage showed this 1.25X slower Rails requests
- This was the first time I released solid benchmarks on coverband
  - benchmarks were done on a sample Rails app & a large production app at my current place of employment
  - benchmarks were done by hand in a non repeatable process
2015: Various performance improvements introduced
- Redis pipelining
- Redis zadd
- benchmarking was done by hand, in a non repeatable way, to confirm actual performance impacts
2016: performance tests introduced into Coverband repository by @kbaum8
- The new performance tests were used to propose that with various improvements between Ruby 1.9.x and Ruby 2.1.x that we could drop support for Coverband_ext
- Moved from set_trace_func to Ruby trace_point
- The micro-benchmark showed that Ruby trace_point was more than good enough an the C extension no longer provided significant performance improvements
- See the source for integrated micro-benchmark performance tests
- At these point most features and changes were checked with the performance tests to ensure the project was always getting faster
- improved file filtering
- Adding support for line usage count vs just used or unused for example didn’t incur additional overhead
2017:
- Multiple backend stores added
- The benchmark performance tests were refactored, so performance could be compared across multiple backend stores
2018:
- Attempt to patch the Coverage reentrant Ruby bug
  - in this quest I propose Coverage.pause and Coverage.resume
  - and in response @tenderlove says my goals are supported by peek_results which was added about 3 years ago, and that the performance impacts I had been assuming were not likely correct.
  - I attempt to prove that the performance impact would be significant, and end up proving @tenderlove is correct ;) Which is what started the deeper dive into Ruby Benchmarking
- The results from the above cause me to update the Coverband benchmarks and use those benchmarks to help prove the significant win of using Coverage vs trace_point
- Extending the benchmark code to report across different collector methods, shows massive win for the new Coverage based collector, at least a 4X improvement.

Benchmark Code

Below are some selected examples of the Coverband micro-benchmark performance tasks.

There are many ways to setup benchmarks for a Gem. In the Coverband examples are simple Rake tasks. This is opposed to performance tests, which folks have often employed for Rails app benchmarks. Given all the configuration and runtime impacts, the Rake approach worked well, but it requires reviewing the data by hand as nothing is tracked or charted in a machine readable format over time via CI.

The example below has been simplified a bit from the full source linked above.

namespace :benchmarks do
  # leaving out some helper methods

    desc 'set up coverband tracepoint collector to redis'
  task :setup do
    clone_classifier
    $LOAD_PATH.unshift(File.join(classifier_dir, 'lib'))
    require 'benchmark'
    require 'classifier-reborn'

    Coverband.configure do |config|
      config.redis              = Redis.new
      config.root               = Dir.pwd
      config.percentage         = 100.0
      config.logger             = $stdout
      config.collector          = 'trace'
      config.memory_caching     = ENV['MEMORY_CACHE'] ? true : false
      config.store              = Coverband::Adapters::RedisStore.new(Redis.new)
    end
  end

  desc 'set up coverband with coverage collector to redis'
  task :setup_coverage do
    clone_classifier
    $LOAD_PATH.unshift(File.join(classifier_dir, 'lib'))
    require 'benchmark'
    require 'classifier-reborn'

    Coverband.configure do |config|
      config.root               = Dir.pwd
      config.percentage         = 100.0
      config.logger             = $stdout
      config.collector          = 'coverage'
      config.memory_caching     = ENV['MEMORY_CACHE'] ? true : false
      config.store              = Coverband::Adapters::RedisStore.new(Redis.new)
    end
  end

  def work
    5.times do
      bayes_classification
      lsi_classification
    end

    # simulate many calls to the same line
    10_000.times { Dog.new.bark }
  end

  def run_work
    puts "benchmark for: #{Coverband.configuration.inspect}"
    puts "store: #{Coverband.configuration.store.inspect}"
    Benchmark.bm(15) do |x|
      x.report 'coverband' do
        SAMPLINGS.times do
          Coverband::Collectors::Base.instance.sample do
            work
          end
        end
      end

      x.report 'no coverband' do
        SAMPLINGS.times do
          work
        end
      end
    end
    Coverband::Collectors::Base.instance.stop
    Coverband::Collectors::Base.instance.reset_instance
  end

  desc 'runs benchmarks on default redis setup'
  task run: :setup do
    puts 'Coverband tracepoint configured with default redis store'
    SAMPLINGS = 5
    run_work
  end

  desc 'runs benchmarks coverage'
  task run_coverage: :setup_coverage do
    puts 'Coverband Coverage configured with to use default redis store'
    SAMPLINGS = 5
    run_work
  end
end

desc 'runs all benchmarks'
task benchmarks: ['benchmarks:run', 'benchmarks:run_coverage']

Benchmark Results

Below is a sample of the output generated when all the benchmarks are run. For each test, it configures Coverband and outputs the configuration settings, along with the same code executed with and without Coverband. The key point being the output below for the new Coverband implementation.

                      user     system      total        real
coverband         0.320000   0.010000   0.330000 (  0.322387)
no coverband      0.320000   0.000000   0.320000 (  0.321767)

![Stopwatch](https://chart.googleapis.com/chart?chtt=Time%20in%20Sec%20Vs%20Benchmark%20Type&chxt=x,y&cht=bvs&chxr=1,0,1.5&chds=0,1.5&chd=t:1.350,1.3100,0.32,0.32&chco=76A4FB&chbh=23,100,75&chs=500x325&chxl=0:

Tracepoint%20to%20File

Tracepoint%20to%20Redis

Coverage%20to%20Redis

Without%20Coverband)

User time from benchmark results below graphed

While it is easy to see the performance impact of the previous tracepoint collector, at 1.350000 vs 0.320000 the current benchmark can’t even detect a performance slowdown using the new Coverage collector. While this means, I should further extend the Gems performance tests, the new implementation is significantly and easily proved to be far more performant than the previous implementation.

rake benchmarks
Coverband tracepoint configured with file store
benchmark for: #<Coverband::Configuration:0x007fc069235c68 @root="/Users/danmayer/projects/coverband", @redis=nil, @root_paths=[], @ignore=[], @additional_files=[], @include_gems=false, @percentage=100.0, @verbose=false, @reporter="scov", @collector="trace", @logger=#<IO:<STDOUT>>, @startup_delay=0, @memory_caching=false, @store=#<Coverband::Adapters::FileStore:0x007fc0692359e8 @path="/tmp/benchmark_store.json">, @disable_on_failure_for=nil>
store: #<Coverband::Adapters::FileStore:0x007fc0692359e8 @path="/tmp/benchmark_store.json">
                      user     system      total        real
coverband         1.350000   0.000000   1.350000 (  1.354026)
no coverband      0.310000   0.000000   0.310000 (  0.321970)
Coverband tracepoint configured with default redis store
/Users/danmayer/projects/coverband/test/benchmarks/benchmark.rake:127: warning: already initialized constant SAMPLINGS
/Users/danmayer/projects/coverband/test/benchmarks/benchmark.rake:134: warning: previous definition of SAMPLINGS was here
benchmark for: #<Coverband::Configuration:0x007fc069235c68 @root="/Users/danmayer/projects/coverband", @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>, @root_paths=[], @ignore=[], @additional_files=[], @include_gems=false, @percentage=100.0, @verbose=false, @reporter="scov", @collector="trace", @logger=#<IO:<STDOUT>>, @startup_delay=0, @memory_caching=false, @store=#<Coverband::Adapters::RedisStore:0x007fc06911c8b8 @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>>, @disable_on_failure_for=nil>
store: #<Coverband::Adapters::RedisStore:0x007fc06911c8b8 @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>>
                      user     system      total        real
coverband         1.310000   0.000000   1.310000 (  1.344789)
no coverband      0.320000   0.000000   0.320000 (  0.318863)
Coverband Coverage configured with to use default redis store
/Users/danmayer/projects/coverband/test/benchmarks/benchmark.rake:141: warning: already initialized constant SAMPLINGS
/Users/danmayer/projects/coverband/test/benchmarks/benchmark.rake:127: warning: previous definition of SAMPLINGS was here
benchmark for: #<Coverband::Configuration:0x007fc069235c68 @root="/Users/danmayer/projects/coverband", @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>, @root_paths=[], @ignore=[], @additional_files=[], @include_gems=false, @percentage=100.0, @verbose=false, @reporter="scov", @collector="coverage", @logger=#<IO:<STDOUT>>, @startup_delay=0, @memory_caching=false, @store=#<Coverband::Adapters::RedisStore:0x007fc0691acfa8 @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>>, @disable_on_failure_for=nil>
store: #<Coverband::Adapters::RedisStore:0x007fc0691acfa8 @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>>
                      user     system      total        real
coverband         0.320000   0.010000   0.330000 (  0.322387)
no coverband      0.320000   0.000000   0.320000 (  0.321767)

Additional Resources on Performance Testing Gems

Some other examples of benchmarking a Gem or even single commits.

Ruby Benchmarking

2018-03-25T00:00:00+00:00

Ruby Benchmarking

If you work on a large app, publish, gems, or are interested in the Ruby community at some point you will likely want to dig into performance and benchmarking. Often what you think will help performance doesn’t have as much of an impact as you would guess. Which is why being able to set up quick experiments to verify the impact of code changes on performance and memory can be important. Benchmarking is great as it can help prove, your change had the impact you want. Ruby performance and the measurement of it is getting more discussion at the moment because Ruby 3X3 set out a goal to make Ruby 3 three times faster than Ruby 2. Learn more in this talk, Ruby3x3: How are we going to measure 3x?

This post will talk about measuring the performance of Ruby and benchmarking your own code and applications. In a future post, I will dig more into good practices around benchmarking Ruby gems.

image from pixabay

Ruby Benchmarking Projects

A number of projects have been proposed and have support from the community for benchmarking Ruby… These project all will help the community improve the performance of Ruby, and can be used to help you measure the impact or Ruby or Gem changes you would like to try. If you are interested in contributing to Ruby or helping the community with improved performance or better benchmarking I encourage you to check out these projects. A great way to learn more about benchmarking is to dig into projects that are already out available and looking for more contributors.

Ruby Benchmarking Basics

When I need to quickly benchmark something, I look no further than Ruby’s simple Benchmark Module. I don’t have all the usage memorized off the top of my head, but have ended up at this great blog post, timing Ruby code is easy with Benchmark, a number of times over the years. It is a quick and simple post that will help get basics up and running in no time at all. For some additional Ruby benchmarking support also see the resources below.

Learning by Benchmarking A Proposed Ruby Change

I previously covered how to build Ruby from scratch on OS X. I posted that as I was working on a feature, I was hoping to get into Ruby. I proposed a change to Ruby’s Coverage that I hoped would lead to large performance improvements adding Coverage Pause & Resume support. A little discussion with @tenderlove, lead me to try to prove the value of the feature in terms of performance. In the end, I actually proved my feature idea was unnecessary and that Ruby’s Coverage was significantly faster than Ruby’s Tracepoint. Below, I will walk through some of the steps and code, I used while driving towards better benchmarks and understanding of what was really impacting performance while trying to collect the line of code runtime.

First Attempt: A Micro Benchmark

For some specific changes a micro benchmark, that focuses in very narrowly on the specific changes under test can be all that is needed and make for faster and easier iteration while testing changes. Often these are the easiest to setup and be more easily repeatable with various versions and settings to compare a number of changes.

My first attempt was to use a small repository, that I had used to show an issue with Ruby’s coverage, see my Benchmarking Coverage Example. The code was extended it so it could be run in 3 modes:

Ruby without Coverage loaded: ruby example.rb
Ruby with Coverage as it exists in the current release: COVERAGE=true ruby example.rb
Ruby with my suggested Coverage feature put to use: ENHANCED_COVERAGE=true ruby example.rb

require 'benchmark'
require 'coverage'

WITH_COVERAGE = !!ENV['COVERAGE']
WITH_ENHANCED_COVERAGE = !!ENV['ENHANCED_COVERAGE']

Coverage.start if WITH_COVERAGE || WITH_ENHANCED_COVERAGE
require 'bigdecimal/math'
require './app'
require './app_proxy'
Coverage.pause if WITH_ENHANCED_COVERAGE

ITERATIONS = 2_000
UPTO = 1_000
coverage_data = nil

# warm up
AppProxy.process(App, {iterations: 1, up_to: UPTO})

Benchmark.bm do |bm|
  if WITH_ENHANCED_COVERAGE
    bm.report { coverage_data = AppProxy.process(App, {iterations: ITERATIONS, up_to: UPTO, coverage: false, enhanced_coverage: false}) }
  end
  bm.report { coverage_data = AppProxy.process(App, {iterations: ITERATIONS, up_to: UPTO, coverage: WITH_COVERAGE, enhanced_coverage: WITH_ENHANCED_COVERAGE}) }
end

puts "coverage"
puts coverage_data
puts "done"

The initial results, even with a high number of iterations basically couldn’t show a real difference. In the case, I was trying to show, since it wasn’t a dramatic improvement and wasn’t obviously measurable like the number of objects allocated to memory, proved to not be a good case for a microbenchmark.

Second Attempt: Build On Rails Ruby Bench

As I abandoned the idea of a microbenchmark being able to show the differences of the performance impact when using Ruby’s code coverage. I sought a more realistic example of how Ruby code is frequently used in production. I found @codefolio’s project, rails_ruby_bench, which I mentioned previously intends to help measure performance impacts for Ruby 3X3. Unfortunately, I ran into some issues getting this benchmark to run locally and I would need to do a good deal of work to embed the code I wanted under test into this project and Discourse which the project uses as the Rails app under performance testing.

While I didn’t end up pursuing this route, it did help push me towards a good direction to more realistically measure the performance impacts I wanted to see. By setting up a sample Rails application and testing full request cycles, I would have a much more realistic measure of my changes. I plan to follow up on this project more in the future and think it is an ideal way to test many Ruby or Gem changes that would have performance or memory impacts on standard Rails applications.

Third Attempt: Sample Rails App

Combining the learnings from the microbenchmark and the ideas in Rails Ruby Bench, I wanted to be able to quickly test a number of different scenarios by just setting a few environment variables, and then pull meaningful results on exercising the full Rails app with the changes. To measure the impact on the full Rails stack I turned to Apache’s AB, a simple HTTP benchmarking tool I have used many times over the years.

For each test, the Rails application would be run in a different mode, then benchmarked via AB. I created a new repository coverage_rails_benchmark. The project README covers all the steps for anyone to run their own set of benchmarks and records all the results of my performance tests.

The benchmarking script itself is extremely simple, the script below will make 2000 requests making 5 concurrent requests at a time to the endpoint listed. The code can be found in bin/benchmark.rb

puts `ab -n 2000 -c 5 "http://127.0.0.1:3000/posts"`

For example to get the most basic results of Ruby without Coverage loaded, one would follow the two steps below in two different terminals.

start the Rails server in basic mode: IGNORED_COVERAGE=true RAILS_ENV=production bin/rails server
execute the benchmark: ruby ./bin/benchmark.rb

This would output a bunch of data about the benchmark results.

Benchmarking 127.0.0.1 (be patient)

Server Software:
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /posts
Document Length:        3631 bytes

Concurrency Level:      5
Time taken for tests:   8.391 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      8572000 bytes
HTML transferred:       7262000 bytes
Requests per second:    238.34 [#/sec] (mean)
Time per request:       20.978 [ms] (mean)
Time per request:       4.196 [ms] (mean, across all concurrent requests)
Transfer rate:          997.58 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0      11
Processing:     6   21  10.8     19     192
Waiting:        6   20  10.7     19     190
Total:          6   21  10.8     19     192

Percentage of the requests served within a certain time (ms)
  50%     19
  66%     23
  75%     25
  80%     26
  90%     30
  95%     33
  98%     38
  99%     42
 100%    192 (longest request)

For my various comparisons, I ended up caring the most for the mean time per request in the above example 20.978 [ms] (mean) was the starting point baseline I comparing with all other results. My final sample benchmark support 7 distinct modes that could be run in two different distinct settings. Unlike with my microbenchmark performance differences were extremely obvious with a full spread of fasted benchmarking mode running at 17.735 [ms] (mean) vs the slowest mode taking 85.069 [ms] (mean). Showing the worst mode was nearly 5X slower in a simple benchmark.

Benchmarking Conclusions

The various modes and options made it clear that I had initially been optimizing for the wrong thing, and that while there was significant performance improvements that could be made, it wouldn’t be related to my suggested Coverage feature proposal to support pause and resume. Instead, it showed that collecting coverage data was always very fast, but pulling and processing that data could be very slow. Opposed to trying to sample data collection the goal should be to reduce as much as possible the frequency one processes the results and to filter it down to the smallest set of results needed. Let’s take a look at a high-level summary below, or see the full benchmark details.

No Coverage Support: 17.735
Coverage Running (Ignore Coverage): 18.131
Coverage Stopped (Ignore Coverage): 18.268
Coverage Paused (Ignore Coverage): 18.717
Coverband Coverage (Ignore Coverage): 18.759
New Pilot Version Coveraband Coverage (Collect Coverage): 19.227
Coverage Running (Collect Coverage, but only into memory): 21.141
Coverage Resume (Ignore Coverage): 23.930
Coverage Resume (Collect Coverage, but only into memory): 26.720
Coverband Coverage (Collect Coverage): 39.421
Coverband Tracepoint (Collect Coverage): 46.979
Coverband Tracepoint (Ignore Coverage): 47.500
Coverage (Collect Coverage, send to Rails.logger): 85.069

In the end, my goal of changing Ruby’s Coverage was to be able to significantly reduce the performance overhead of my gem Coverband’s ability to collect runtime data. The results from the benchmark gave me all the data I needed to see that a different approach, where I dropped TracePoint in favor of Coverage but worked to reduce the frequency of checking the results would lead to far better performance improvements than my initially suggested feature. In the end, I pursued that approach, and was able to reduce the overhead with a sample rate of 100% from 2.5X slower to only being 1.08X slower! The details on that, I will cover in another post with some specifics on how to build performance benchmark testing into a Gem.

Ruby Benchmarking Learnings

I came away with a deeper understanding of Ruby performance benchmarking, and a much faster Gem that I will be able to release shortly. Beyond that some additional thoughts on Ruby benchmarking,

For some changes microbenchmarks aren’t helpful
There are some existing great projects to help folks benchmark changes to the Ruby language
Ruby’s Coverage is significantly Faster than collecting line usage via Ruby’s TracePoint functionality
Specific to Ruby’s Coverage library
- The number of files instrumented with coverage has a big impact on performance, making micro benchmarks useless for that library
- Calling coverage.peek_results to access the data is the biggest cost, much larger than collecting it.
- Simply logging, processing, or trying to do anything with data is often slower than collecting it
- Performance wins will come in reducing processing the data vs collecting it
Writing an maintaining good benchmarks can be challenging, I didn’t touch on the issues of running benchmarks with “background noise” and running multiple times to tease that out
Without measuring performance, guessing what will have a large impact is often wrong

Let’s Go Faster

via GIPHY

In a follow-up post, I will dig into the details of benchmarking a Gem over time, and the specific changes that helped to reduce the performance overhead of Coverband making it an order of magnitude faster. While there are code changes and using different features of Ruby, large performance wins also can come from a fundamentally different approach to solving the problem, which is the case with the changes needed to improve Coverband.

Rack Proxy Tour

2018-02-26T00:00:00+00:00

Rack Proxy Tour

I wanted to share a quick tour of a flexible Ruby tool. I have turned to rack-proxy a number of times, throughout the years. It is a tiny and super useful Rack middleware that can quickly be adapted to perform a number of useful functions. While it is a small library it is a tool that has been handy for quick and lasting workarounds a number of times. It is a sharp tool, so be careful with it… Especially because it is severely lacking documentation… Perhaps I should send a documentation PR one of these days.

image from pixabay

Rack Proxy Examples

Some of the various ways I have used Rack Proxy over the years, while I wouldn’t recommend all of them, sometimes a quick hack is needed and Rack proxy can be a powerful tool for that.

subdomain based pass-through to multiple apps
useful for handling awkward redirection rules for moved pages
fan out a single API request to multiple concurrent backend requests and merging results
authentication / authorization prior to proxying requests to a blindly trusting backend
avoiding CORs complications by proxying from same domain to another backend

Example Code

In the example below we will have our rack proxy middleware, handle user authentication and authorization then make an authenticated request to another service. In this case, any request to our Rails applications host with the path /example_service/ will pass through to the target service.

require 'rack-proxy'

class ExampleServiceProxy < Rack::Proxy
  def perform_request(env)
    request = Rack::Request.new(env)

	 # path matches our target &&
	 # user auth (devise in this case) found a logged in user
    if request.path =~ %r{^/example_service} &&
      env['warden'] &&
      env['warden'].user

		# have a user but using CanCan check if user has needed permissions
      if env['warden'].user.can?(:access, :access_example_service)
        token = "Bearer #{Settings['service.token']}"
        service_url = Settings['service.url']

        @backend = URI(service_url)
        env['rack.backend'] = @backend
        
        # while documentation says you only need on of these,
        # I needed to set them all to have the expected results
        env['REQUEST_PATH'] = env['REQUEST_URI'] = env['PATH_INFO'] = '/api/target_service_path'
        
        # target service fails on cookies
        env['HTTP_COOKIE'] = ''
        env['HTTP_AUTHORIZATION'] = token
        super(env)
      else
        Rails.logger.info "example_service: 401 user #{env['warden'].user.inspect} denied"
        [401, {}, ['Unauthorized!']]
      end
    else
      @app.call(env)
    end
  end
end

Then you can just hook up the middleware as needed, using something like below (in application.rb for example).

config.middleware.use(ExampleServiceProxy)

If you want to be able to access the devise user env['warden'].user you will need to make sure your middleware is inserted after the devise middleware. You can quickly check the order by printing out the Rails middleware stack.

> rake middleware

use Raven::Rack
use BufferedLoggingMiddleware
...
use Warden::Manager
use ExampleServiceProxy
...
use OtherExamples

Building Your Own Ruby

2018-02-17T00:00:00+00:00

Building & Using Your Own Ruby

If you want to work on changes or learn more about the internals of the Ruby language, you can alter the source and build your own from scratch. It isn’t that hard or scary, you will learn a bit more about Ruby just by building it. This is a quick start to building Ruby from scratch on OSX, and using it to run your local apps.

If you want to read a great guide about contributing to Ruby, it also covers how to build even easier via a Dockerfile.

Get the Source

Use git or SVN to pull the source repo.

I like git, so I forked the Ruby project.
Then clone it: git clone git@github.com:danmayer/ruby.git
go into the directory and take a look around.

Build Trunk Before Modifying

Before you make any modifications, I recommend you get the current trunk building and test running your apps with it. Then you can create a branch and see the impact of any modifications you would like to try out.

Building Yourself

I recommend using the Ruby-buil instructions below as I continued to hit issues with building from scatch with my own options related to SSL.

If you want to give it a shot thought, run the 4 commands below and it will compile and install Ruby from src on your system.

aclocal
autoconf
bash -c './configure'
make && make install

If you get this error about OpenSSL:

openssl:
	Could not be configured. It will not be installed.
	Check ext/openssl/mkmf.log for more details.
*** Fix the problems, then remove these directories and try again if you want.

It is a bit hard to resolve on OS X yourself, but luckily we can just use ruby-build to do the work for us, see below.

Building via Ruby-build

Ruby build will detect and use SSL from homebrew and avoid the broken OSX implementation.

install ruby-build: brew install ruby-build
copy the current dev trunk target: cp /usr/local/Cellar/ruby-build/20171226/share/ruby-build/2.6.0-dev /usr/local/Cellar/ruby-build/20171226/share/ruby-build/2.6.0-mine
edit the file (...ruby-build/2.6.0-mine) to point to your fork: ... install_git "ruby-trunk" "https://github.com/danmayer/ruby.git" ...
build from your git to the ~/.rubies/ directory: ruby-build 2.6.0-mine ~/.rubies/ruby-2.6.0mine
- run from the build dir: /usr/local/Cellar/ruby-build/20171226/share/ruby-build
make sure to open a new shell so ChRuby (or RbEnv) find the new ruby.
you can now reference the new build in your .ruby-version file in any project

If you want to build from your local git to avoid pushing to a remote branch while testing this is how your build file should end up.

install_package "openssl-1.1.0g" "https://www.openssl.org/source/openssl-1.1.0g.tar.gz#de4d501267da39310905cb6dc8c6121f7a2cad45a7707f76df828fe1b85073af"  mac_openssl --if has_broken_mac_openssl
install_git "coverage_pause" "/Users/danmayer/projects/ruby" "feature/coverage_pause" ldflags_dirs autoconf standard_build standard_install_with_bundled_gems verify_openssl

If all is working as expected you should see this.

ruby-build 2.6.0-coverage ~/.rubies/ruby-2.6.0coverage
ruby-build: use openssl from homebrew
Cloning https://github.com/danmayer/ruby.git...
Installing coverage_pause...
ruby-build: use readline from homebrew
Installed coverage_pause to /Users/danmayer/.rubies/ruby-2.6.0coverage

Playing Nice with Other Rubies?

If you are like most Rubyists you have a number of Rubies installed, beyond the default OSX Ruby, using something like RBenv, RVM, or ChRuby.

The default OS X Ruby should be: /usr/bin/ruby
If you build from scratch the default target will be installed to: /usr/local/bin
- This can cause some issues with various Ruby environment managers
- adjust your path to target this Ruby or your normal one
- Another reason I recommend building via ruby-build, as it will work easier with most Ruby environment managers
Ruby build will put ruby into the ~/.rubies
- with ChRuby it will automatically pick these up

Then you can reference your new custom ruby in a .ruby-version file at the root project directory, for example:

# .ruby-version
ruby-2.6.0coverage

Running an App with your Custom Ruby

After building your own Ruby and setting the .ruby-version you should be good to go. You can verify you are running your Ruby by adding some simple print statement. Pick a favorite Ruby method and just add a print statement like so…

printf( "hello from my method!\n" );

push the change to the branch your referenced in your Ruby build steps
- there are ways to build locally, but I have just targetted git branches
rebuild Ruby: ruby-build 2.6.0-coverage ~/.rubies/ruby-2.6.0coverage
enter your project iwth the set ruby-version
run ruby -v to make sure it matches expectations
- if you get chruby: unknown Ruby: ruby-2.6.0coverage either refresh chruby or open a new terminal so it picks up the new build
now run things as you normally would…

# in Rakefile
desc "call coverage running"
task :call_coverage_running do
  require 'coverage'
  Coverage.running?
end

rake call_coverage_running
hello from Coverage.running

In the above output, I can see that I am runnin my custom Ruby branch as I had added a print messaged to the Coverage.running? method.

Testing your Custom Ruby

If you are trying to do some real work on Ruby other than exploring, you will want to be able to run the tests on files after you modify them, and add your own tests.

Running Tests

For example to run a specific test file, you can run the below command.

make test-all TESTS=test/coverage/test_coverage.rb
Run options: "--ruby=./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disable-gems" --excludes-dir=./test/excludes --name=!/memory_leak/

# Running tests:

Finished tests in 1.580204s, 12.0238 tests/s, 74.6739 assertions/s.
19 tests, 118 assertions, 0 failures, 0 errors, 0 skips

ruby -v: ruby 2.6.0dev (2018-01-13 trunk 61811) [x86_64-darwin15]

Running Specs

Below we can see how to run a single specific spec file

make test-spec MSPECOPT=spec/ruby/library/coverage/start_spec.rb
generating x86_64-darwin15-fake.rb
x86_64-darwin15-fake.rb updated
$ /Users/danmayer/projects/ruby/miniruby -I/Users/danmayer/projects/ruby/lib /Users/danmayer/projects/ruby/tool/runruby.rb --archdir=/Users/danmayer/projects/ruby --extout=.ext -- /Users/danmayer/projects/ruby/spec/mspec/bin/mspec-run -B ./spec/default.mspec spec/ruby/library/coverage/start_spec.rb
ruby 2.6.0dev (2018-01-13 trunk 61811) [x86_64-darwin15]
[\ | ==================100%================== | 00:00:00]      0F      0E

Finished in 0.002753 seconds

1 file, 0 examples, 0 expectations, 0 failures, 0 errors, 0 tagged

January Software Links

2018-02-03T00:00:00+00:00

Some of my favorite links from January. Also, I will use this month to highlight a group of folks who have been sharing advice and helping me grow for a number of years now. Thanks!

If you are growing as an engineering leader find a set of mentors. Thanks, @glv @mariagutierrez @mad_typist @SteveAkers @dramsay @ryan_owens @ioutfitter @chadfowler I always appreciate the thoughtful advice. pic.twitter.com/zJyBM9cNlo
— Dan Mayer (@danmayer) February 3, 2018

The above folks in a mentoring group recommend good books, bounce ideas off each other, and help folks prep for big talks, events, or projects. Thanks!

Software Development

Data

Two great Postgres Links from @tpitale
- Postgres Indexes Under the Hood
- Postgres Configuration Recommendation Tool

Ruby

Tech Management

best books for first time managers

Random

Links by wsyperek

December Software Links

2017-12-31T00:00:00+00:00

Some of my favorite links from December. If you have any good links I missed pass them my way.

What you learn over 5 years is that every dependency, every line of code, every test, every feature is a DEBT or outage, waiting to happen.
— David Copeland (@davetron5000) November 26, 2017

continuing my trend of having something from @davetron5000 each month, and this month I have two ;)

Software Development

one-click PDF microservice
on Kafka and Redis streams
distributed system with one process representing every user / shared resource in Elixir, great talk @myobie
Understand why Serverless matters another great write up by @davetron5000

OK, updated my pre and post production testing diagram based on feedback I got. Still something I missed out? pic.twitter.com/c3Q2wUSQdI
— Cindy Sridharan (@copyconstruct) December 28, 2017

Ruby

GraphQL Performance in Rails
Memory and Performance improvements for Pluck
- If you have to iterate through huge collections and don’t need the full object this is a huge win
Malloc Can Double Multi-threaded Ruby Program Memory Usage
- Sidekiq users can reduce memory use by 1.5-4x with a few config/environment changes

DevOps

Rapid Release at Massive Scale deployment at Facebook

Tech Management

Random

10 years in and nobody has a use case for blockchains
- yes this is a bit of a clickbait title, but many points still hold true. The claims of distributed aren’t holding up to well with real world usage of blockchains.

Links by wsyperek

On Tech Challenges

2017-12-18T00:00:00+00:00

Coding Challenge

The company I am with, Offgrid-Electric, has evolved our hiring process over time to try to ensure we can find a diverse and strong distributed team. There are many interesting processes related to hiring a distributed team, but I will quickly focus on the coding challenge, technical challenge, or whatever folks are calling it these days.

image by FotografieLink

Offer Challenge Options

This post is kicked off by the thread below.

We accept any that the applicant is most comfortable with... Share existing code and walk us through what was done, pairing, code exercise take home, or tech challenge live on a call. I find some people are more comfortable with one method or another so we try to do anything
— Dan Mayer (@danmayer) December 18, 2017

Folks followed up asking for more details on other challenge options we provide and why, which I will try to breakdown below.

What is the point?

First off why have a coding exercise in person, take-home, or other? The goal is to allow a candidate to demonstrate skills beyond being great to talk with during interviews. That they can back up the conversation with problem-solving skills that match the team’s expectations. There are far too many posts and stories joking about how few developers can code, which I will skip over, but often a team wants a way to help ensure that a candidate has skills that will make them successful if they join. At OGE we try to allow for the widest possible options to prove out that a candidate would be technically qualified or the position.

Tech Challenges Options

We offered a few standard options and have added more and evolved our challenges over time.

Take Home Challenge: We have someone on our team develop a challenge for any job description we are hiring for. The challenge is prepared before we even post a job opening. In general, we have someone on the team develop the challenge and complete it themselves to ensure they fully understand the challenge and have a good estimate of the time required to complete it (generally 2-4 hrs). This falls somewhere into the category of building a small application that does X, Y, and Z given this sample data set. Given folks likely have a current job we give plenty of time to work on the challenge and include instructions for how to submit the challenge and things we expect to see.

Pair Programming: We had some folks talk to us that enjoy pair programming and find it a good way to understand the way our current team works and thinks. We offer to pair on the take-home challenge and work on it with the candidate at a scheduled time, or to work on some of the code we are currently working on, or to pair on some code that the candidate has recently been working on. Some folks are very comfortable with this kind of exercise while others get very nervous pairing with someone they are just getting to know. Some folks have paired a lot in the past, or it might be a fairly new experience for them. Unless your team practices 100% pair programming, I believe you should offer some alternatives to just pairing. I like having a take-home challenge that can also be offered as a pairing exercise.

Open Source Work: This is a great option for folks that already have a lot of publicly available code they are proud of and showcases their skills. There is no reason to make someone spend extra hours of their time when you could have them walk you through code they are passionate about and have already been working on. I have enjoyed learning about some candidates projects even if it wasn’t a great fit and the enthusiasm for something they have recently worked on in their own time shows through. It is important to have other options beyond just open source. While this is a great alternative, it can’t be the primary or expected way to check off the technical challenge. You can find more detailed reasoning about this on various diversity hiring posts, but not everyone has the same privilege to spend extra time outside of work on open source, some jobs don’t allow it, it isn’t everyone’s cup of tea.

Technical Questioning: While I prefer to do something with code like the options listed above, sometimes that isn’t the best fit for whatever reason. In these cases, we have done some technical white boarding type interviews over video chat. In my interview, at OGE, I walked through how to split out a monolith application into smaller component applications and discussed the kind of DevOps you need to support such transitions. This format has also worked well for QA interviews. I try to avoid trick questions / brain teasers. Asking data modeling questions tends to work pretty well over a whiteboard for an example of good technical questions.

Choose your own Adventure

Taking this even further, given our only goal with this part of the interview process is to come away with some confidence that the person is technically capable in a set of skills… I often ask the candidate after presenting our standard challenge options listed above if that have any other way they would like to prove out technical skills. I will list some things candidates have suggested or done before.

Technical Writing / Educational Materials: We have had some folks apply that have created online materials for courses. Sharing video series that teach programming concepts, blog posts detailing how to use some technology, or videos of talks given at a conference covering technically relevant material.

School Projects / Papers: If you are talking to someone recently coming out of university they might have some great projects they could share with you. If it is a group project just get into specifics about what pieces they directly contributed. We recently had a candidate submit their Ph.D. thesis as a way to confirm their technical skill.

Code Reading / Walkthrough: A few candidates that didn’t have open source code but as consultants had learned to quickly look at projects and distill how they worked. In this case, they were also curious to see how our systems worked. After some discussion, we walked through sections of our companies source code with the candidate reading the code and explaining what it did and the recommendations they would have to improve that section of code. While this is a bit more challenging on the interviewer’s side, it can give candidates a great picture of the kind of code and the quality of the project they will be getting involved in. If the person can quickly jump in and absorb parts of the project’s domain and explain it back to you, it can give you the confidence you need to move forward.

In the end, it comes down to just making sure you set a candidate up to show you their best work. You don’t want to setup a situation that will have a great candidate so nervous they don’t show what they are capable of. Being a globally distributed team, in the end, I don’t expect all our team to work the same way, so I wouldn’t expect all candidates to succeed in the same style of challenges. You can always find a developer that loves to pair or hates it, or gets very nervous working out a problem on a whiteboard or could lecture in front of a whiteboard for hours on end. All of that is fine, let em shine.

November Software Links

2017-12-02T00:00:00+00:00

Some of my favorite links from November. If you have any good links I missed pass them my way.

Software Development

A follow up to last months part one, part two of human code reviews
Security Alerts from Github
50K scholarships to Udacity via Google
embracing metrics vs testing

☠️ "Dead Code" can rot a code base over time. Here are a few common types of dead code. Be diligent and thorough about removing them as the code evolves. Even dead code can still cause problems. pic.twitter.com/vPQ7njyhEv
— Jason McCreary (@gonedark) November 22, 2017

intentionally and proactively providing an extra-normal level of support and manual help to users in order to learn about the barriers people are hitting

service and support driven development

Data

Running PGBouncer turned a potential catastrophe into a non-event for us. https://t.co/IIbk3vPzit
— David Copeland (@davetron5000) November 28, 2017

more about PgBouncer
scaling connections in Postgres
Querying across RDS and Redshift, checkout dblink. Enabling some powerful data warehouse jobs. Thanks Alex from @anjlabs
A story of rolling out DB partitioning on a live DB and notes on how to do it.

Ruby

Reduce open DB connections from Sidekiq in Rails
Rails 5.2 beta lots of good stuff
- huge improvements in Redis cache
- active storage
- faster boot time with bootsnap

Devops

Things I wish I knew about terraform before jumping into it
DuckRails: quickly mock API endpoints
dependbot automate dependency updates thanks @dmitrytsepelev

Tech Management

working more than 50 hours makes you less productive
thoughts from a 20 year startup CTO veteran thanks @dramsay
Review of Remote: Office Not Required by @moyheen who is part of our OGE team.
Case study: How complexity creeps in

Random

Links by wsyperek

Active Record Database Documentation

2017-11-12T00:00:00+00:00

Documentation Folders CC image from Pixabay

Active Record Database Documentation

This post will cover how to use the Active Record feature to add comments to your tables, columns, and indexes. This feature makes it easy to keep your database documentation up to date as you can add descriptions at the same time as you add or update tables and columns.

The best documentation about the feature is contained in the original Pull Request support comments in database
The feature was also well covered in Rails 5 supports adding comments in migrations by Prajakta Tambe

Documentation Embedded with Change Process

Why would we want to use Rails to build our database documentation?

I believe documentation close to code and embedded in the code change process has a better chance of staying up to date and relevant.

I also think we want to add human context to our information in same tools we build our database. We want to do this in source code that is trackable for the same reason we run database migrations in Rails opposed to just having a DBA make schema changes outside of our application code change process.

When we embed our database documentation in our standard code change process we easily get many advantages.

See DB comments change over time because they are part of Git
Search in code editor tools (and github)
Documentation can be reviewed as part of PRs by a data team, analysts, or other folks who might be the target documentation audience

By having the documentation embedded in the database directly, other values can be unlocked.

The documentation is embedded in most DB explorer tools (SQL workbench, Postico, etc).
Single source of truth documentation. It is easy to generate and push to documentation repositories (markdown, html, confluence). Either from Rails, CI, or any other tool in your workflow (see examples below).

Postico OSX Postgres client showing comments as you explore DB structure

Code Samples

Below are some code samples to help you get started with a workflow around database documentation.

Migration

A migration adding comments to a previously existing table. You can add descriptions to call out deprecated fields, gotchas, planned refactorings, or add historical context that may be helpful to the next developer trying to understand what the field means.

class AddContactComments < ActiveRecord::Migration[5.1]
  def change
    msg = 'Contacts table holds individual details about our contacts, it is associated with leads and customers'
    change_table_comment(:contacts, msg)

    change_column_comment(:contacts, :first_name, 'the contacts first name')
    change_column_comment(:contacts, :last_name, 'the contacts last name')
    change_column_comment(:contacts, :house_latitude, 'the house_latitude the contact lives at')
    change_column_comment(:contacts, :house_longitude, 'the house_longitude the contact lives at')
    change_column_comment(:contacts, :house_location_accuracy, 'the accuracy range we captured the GPS with')
    change_column_comment(:contacts, :deleted_at, "the date the contact was 'hidden' from our DB")
    
    # call out gotchas
    # tasks have assignee_id while contacts still use agent_id in the DB, this is a recommended refactoring
    msg = 'the agent_id field is for who the contact is currently assigned to, various places in the code and API it is referenced by assignee_id'
    change_column_comment(:leads, :agent_id, msg)
  end
end

Ruby DB Docs Access

To generate documentation which could be pushed to a wiki, html, confluence, or elsewhere you can iterate through a tables columns and fetch the comments.

> ActiveRecord::Base.connection.table_comment('leads')
=> "Leads table holds individual details about leads, related associations, event timestamps, and joins to contact"	
> Contact.columns_hash['literacy'].comment
=> reading level: {"-1"=>"none", "0"=>"no_read", "1"=>"limited_read", "2"=>"read_fluent", "none"=>"none", "no_read"=>"no_read", "limited_read"=>"limited_read", "read_fluent"=>"read_fluent"}

# iterate through all the columns on a table and output them to your documentation file or API
Contact.columns.each do |c|
  puts c.comment 
end

SQL DB Docs Access

Obviously you don’t need Rails to get at this information, you can pull it out with raw SQL as well. Covered in this post: Querying table, view, column and function descriptions

# get the all the table comments in your DB
SELECT c.relname As tname, CASE WHEN c.relkind = 'v' THEN 'view' ELSE 'table' END As type, 
    pg_get_userbyid(c.relowner) AS towner, t.spcname AS tspace, 
    n.nspname AS sname,  d.description
   FROM pg_class As c
   LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
   LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
   LEFT JOIN pg_description As d ON (d.objoid = c.oid AND d.objsubid = 0)
   WHERE c.relkind IN('r', 'v') AND d.description > ''
   ORDER BY n.nspname, c.relname ;

# get all the comments for the contacts table
SELECT a.attname As column_name,  d.description
   FROM pg_class As c
    INNER JOIN pg_attribute As a ON c.oid = a.attrelid
   LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
   LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
   LEFT JOIN pg_description As d ON (d.objoid = c.oid AND d.objsubid = a.attnum)
   WHERE  c.relkind IN('r', 'v') AND  n.nspname = 'public' AND c.relname = 'contacts'
   ORDER BY n.nspname, c.relname, a.attname ;

October Software Links

2017-11-02T00:00:00+00:00

A few days late for the end of the month, but that is OK… Some of my favorite links from this month. Also, if you aren’t following our new OGE tech blog, it has some great content. I included a few posts from our blog this month.

Software Development

Tech Management

Random

Links by wsyperek

Databases Across Environments

2017-10-26T00:00:00+00:00

DB Syncs CC image from Pixabay

Syncing Databases Across Environments

It seems that often when a business grows, at some point it is hard to create useful QA, development, and staging data to cover all the cases that can occur on production. Eventually, there is a need to try to replicate certain cases or debug various issues that is much easier when you can pull the production data to another system to experiment and debug safely. Nearly, everywhere I have worked eventually we need to clone large portions of the production DB to staging… Quickly followed by wanting to pull staging data to local development. While sometimes various privacy and security concerns must be taken into account, in general we are just talking about replicating a database or tables from one place to another. I will cover some approaches to moving DBs.

DB Rotation on AWS

If you are in a cloud environment use the tools they provide when you can and avoid building a custom solution. We use AWS, and we run our Postgres DB via the RDS service. We have backup and retention rules and the ability to restore snapshots to different names. We leverage snapshots to move the DBs from one environment to another. If you are using Google cloud or Heroku there are similar options. I am not going to layout the code, but you can also do it easily via the AWS GUI. Our basic setup when in the cloud is detailed below.

Production: has a nightly snapshot taken
Staging:
- about once every other month we restore the staging DB from a production snapshot
- after the snapshot is restored some scripts run (clean up, sanitize, and truncate large un-needed tables) so it will run better on a smaller DB instance
- Careful about various test data that is expected by managers and your QA team. You might have destroyed a lot of work
- We handle that by QA regression suite will detect our QA team data is missing and recreate example QA data for the team
- staging has nightly snapshots taken
Dev Stacks: deployed development environments
- these are created dynamically by developers from git branches
- they restore a new DB from the latest staging snapshot
- then run migrations and whatnot on the branch to get it to the correct state
- we don’t take any back ups of these stacks which are traditionally destroyed after a few days of use

Full DB to local Development

While the AWS toolchain is great you can’t restore an AWS snapshot to your local Postgres DB. For that we just created a simple script that will dump Postgres and upload it to S3 in a compressed format. We have some options to exclude some extra large tables that we generally don’t care about for development, you could always pull those later using the method for individual tables mentioned later.

Full Postgres to S3 Dump Rake Task

  def db_to_dump
    ActiveRecord::Base.connection_config[:database]
  end

  def db_host
    ActiveRecord::Base.connection_config[:host]
  end

  def db_port
    ActiveRecord::Base.connection_config[:port]
  end

  def db_user
    ActiveRecord::Base.connection_config[:username]
  end

  def db_pass
    ActiveRecord::Base.connection_config[:password]
  end

  ###
  # Run this on the target environment you wish to dump (staging dev stack)
  # bundle exec rake data:dump_full_db
  #
  # The DB files should get uploaded to S3 ready 
  # to be pulled into another environment
  ###
  desc 'dump full DB to S3'
  task dump_full_db: :environment do
    tables_to_exclude = %w[really_big_tables papertrail_versions]
    exclude_lines = tables_to_exclude.map { |table| "--exclude-table-data=#{table}" }.join(' ')
    full_db_file = '/tmp/full_dev_dump_data.sql'

    begin
      s3 = AWS::S3.new(access_key_id: env['S3_ACCESS_KEY_ID'],
                     secret_access_key: env['S3_SECRET_ACCESS_KEY'])
      bucket = s3.buckets['dev-db-bucket']
    
      # HACK this is needed so you don't pass the password
      File.open(pg_pass_file, 'w') { |f|
        f.write("#{db_host}:#{db_port}:#{db_to_dump}:#{db_user}:#{db_pass}")
      }
      `chmod 600 #{pg_pass_file}`
      
      `pg_dump #{exclude_lines} -O -v -x -F c -f #{full_db_file} -h #{db_host} -p #{db_port} -U #{db_user} #{db_to_dump}`

      [full_db_file].each do |file|
        if File.exists?(file)
          puts "uploading #{file}"
          path = Pathname.new(file)
          obj = bucket.objects[path.basename.to_s]
          obj.write(path)
        end
      end
    ensure
      `rm #{pg_pass_file} &> /dev/null`
    end
  end
  
  ###
  # Run this on the environment you wish to create or restore
  # the most recent dump.
  #
  # bundle exec rake reload_full_from_s3
  ###
  desc 'reload dev DB from S3'
  task reload_full_from_s3: :environment do
    unless ENV['SKIP_DOWNLOAD']
      s3, bucket = s3_and_bucket
      full_db_file = '/tmp/full_dev_dump_data.sql'

      [full_db_file].each do |file|
        puts "downloading #{file}"
        path = Pathname.new(file)
        obj = bucket.objects[path.basename.to_s]

        File.open(path.to_s, 'wb') do |s3_file|
          obj.read do |chunk|
            s3_file.write(chunk)
          end
        end
      end
    end

    db_to_dump = ActiveRecord::Base.connection_config[:database]
    ActiveRecord::Base.remove_connection

    puts `psql -c "drop database #{db_to_dump};"`
    if $? != 0
      puts 'drop DB failed (you likely need to close console or app)'
      exit 1
    end
    puts `psql -c "create database #{db_to_dump};"`
    puts `pg_restore  --verbose --dbname #{db_to_dump} -F c #{full_db_file}`
  end

Single table to Staging or Development

OK that makes it easy for a new dev to get a copy of the latest full DB, what if you just want to pull the most recent payments, customers, or products… A much faster way it to load a table or two.

Postgres Single table S3 Dump & Load Rake Task

  ###
  # Run on target environment to dump a table
  # TABLE_NAME=phone_block_lists bundle exec rake db:dump_db_table
  ###
  desc 'dump single DB table to S3'
  task dump_db_table: :environment do
    table = ENV['TABLE_NAME'] || 'users'
    table_file = "/tmp/#{table}.sql"

    begin
      s3, bucket = s3_and_bucket
      write_pg_pass
      `pg_dump --no-owner --no-acl -v -F c -h #{db_host} -p #{db_port} -U #{db_user} --table public.#{table} #{db_to_dump} > #{table_file}`

      [table_file].each do |file|
        if File.exists?(file)
          puts "uploading #{file}"
          path = Pathname.new(file)
          obj = bucket.objects[path.basename.to_s]
          obj.write(path)
        end
      end
    ensure
      `rm #{pg_pass_file} &> /dev/null`
    end
  end
  
  ###
  # Run on environment where you want to load the table
  # TABLE_NAME=phone_block_lists bundle exec rake db:reload_table_from_s3
  ###
  desc 'reload dev DB table from S3'
  task reload_table_from_s3: :environment do
    table = ENV['TABLE_NAME'] || 'administrative_areas'
    table_file = "/tmp/#{table}.sql"
    db_to_dump = ActiveRecord::Base.connection_config[:database]

    unless ENV['SKIP_DOWNLOAD']
      s3, bucket = s3_and_bucket

      [table_file].each do |file|
        puts "downloading #{file}"
        path = Pathname.new(file)
        obj = bucket.objects[path.basename.to_s]

        File.open(path.to_s, 'wb') do |s3_file|
          # this is slow perhaps move to AWS CLI download
          obj.read do |chunk|
            s3_file.write(chunk)
          end
        end
      end
    end

    puts `psql --dbname #{db_to_dump} -c "truncate table #{table}"`
    puts `pg_restore  --verbose --data-only --dbname #{db_to_dump}  -F c #{table_file}`
  end

Database Helper Scripts

These are small examples extracted out from our database helper rake task. We have a number of other little helpers as well. It can be useful to build up some small simple tools to help you team move data around environments. It can also really simplify the process of bring on new developers to the team. Hope the simple examples above will be helpful to some folks.

Planning A Remote Team Summit

2017-10-25T00:00:00+00:00

Planning A Remote Team Summit

It is essential for distributed teams to have face to face time. We try to get the entire department to get together at least once a year. In general, smaller groups within our department also get together from time to time to collaborate on specific projects as well. If you read about organizations that do remote work well, they organize one or more team summits/retreats a year.

The team enjoying the rare chance to use post-its in person

Value of Distributed Team Summits

At our summits we work through pain points in our processes and structure, we build trust, and collaborate taking advantages of some of the in-person advantages inherent to being face to face… For example whiteboards (we miss you), post-it-notes, sketches, and more. The value of the summit will long outlast any specific decision or activity. The team will walk away more confident in having challenging discussions, reaching out across organizational boundaries, and assuming best intentions first when issues arise. The most valuable part and the prominent goal of bringing everyone together is team bonding and building trust.

Summit Planning

Planning required for a proper summit shouldn’t be underestimated. At OGE, we put together a small workgroup that met weekly to plan the event. The group developed a schedule, planned sessions, activities, and assigned follow up tasks to specific people. Our workgroup used slack, confluence, and weekly video chats to plan our summit.

Logistics:

The logistics around planning a summit for a distributed team summit gets increasingly complicated as your team grows. Early gatherings might start as simple as flying a handful of people and booking hotel rooms. As an organization expands, it will require coordination among dozens of people across time zones, countries, and schedules.

Start planning early:

Finding dates for your event that work across dozens of people and countries holidays will be a challenge.
Many folks may need visas and you will want to start on this with plenty of time (OGE, unfortunately, ended up having a Visa issue preventing one person from attending this year)
Avoid high hotel and airfare costs by being able to book well in advance

Travel Help:

Depending on your organization it may be helpful to coordinate flights, hotels, etc.
Also, it might be easier to let some things be handled ad-hoc and allowing for expensing ad-hoc costs (ex: travel to/from the Airport)
A reminder on guidelines around expensing will help put people at ease and avoid unexpected expenses
Depending on the situation folks might need help with Visas (ex: letters from businesses requesting their travel for an event, proof of hotel accommodations, etc.)
Booking blocks of rooms at hotels or whole Airbnb houses can help with cost and logistics by keeping the group close to each other and the venue
Booking a venue, when small you might meet at a local office, but as you grow a local office likely can’t handle an influx of the full international team

Day to Day help:

Having someone from the local team available to help with planning can be very valuable
Coordinating lunches can save a lot of time in the schedule. (self-exploring, forming groups, and finding lunch in an unknown city lead to lots of missed afternoon meetings)
As there will likely be needs for some communication and standard work, planning on reliable and fast internet connections is highly recommended
Some recommendations on dinners, things to do, how to use local transit, etc

Discussing our DevOps challenges in person

Communications, keeping everyone up to speed

It is important to communicate about your event to various people in the organization before, during, and after the event. I will cover some essential communications to keep in mind.

Prior to the event

Exec / Budget: This communication has to happen far before any event. This happens as you build a team and invest in a distributed workforce. If a growing budget expense for an international summit isn’t understood as necessary all the way up the organization, your group probably shouldn’t be building an international team. Make sure folks know while sometimes you save on salary by not hiring everyone in expensive markets or having office space for all employees there are some overhead costs, such as increased travel budgets to ensure the team can get together from time to time.

Stakeholders and Partners: Let stakeholders know they will have limited access to the team and deliverables likely won’t be released (or make progress) during the event. Sharing a bit about why it is necessary for a distributed team to work on processes, planning, and collaboration in person from time to time. Let folks know it is coming up as soon as you have dates. Follow up sending reminders immediately before the event. The reminder email should include clear guidelines on expectation on what work will or will not happen that week. Also, officially cancel any regularly scheduled meetings, and explain how to raise any emergency issues that may arise (phone numbers, SMS, and contact info for backups that might not be able to attend the event, etc.).

Internal team: set expectations that we are there to communicate and work together, and to not be on laptops on slack and in remote meetings much during the week. The costs and value of getting together are too valuable to end up spending it as a regular week. Share travel tips, schedules, a map with notable locations, and contact info to a local person to help resolve any unexpected travel issues.

During the Event

Stakeholders and Partners: Send a quick reminder as the event starts. Folks sometimes enjoy seeing some photos or updates as the event occurs.

Internal team: If any of the schedule plans change make sure to share with everyone. Makes sure folks have a way to access changes when away from venue internet. Either in person or async during the event take little check-ins to make sure the event is going well, the discussions are valuable and that nobody is struggling with some unexpected challenge related to travel, accommodations, etc.

After the Event

Everyone: We have both slack and workplace where the organization shares news. The team should share a write-up with the company explaining what happened, where, and some updates on things you did. Share some photos, let other departments know what your team is up to and celebrate a bit of the rate time you get together.

Internal team: follow up on action items, plans and re-enforce decisions that were made at the event.

Setting an Event Format

There are many formats and ways to plan events, so I won’t try to cover them all. Instead, I can say it is good for the group planning the event to find some structure they can agree on. Then work together to roll it out. Our planning group read The Advantage and loosely followed some of its guidelines as well as some of our ideas to plan the event. I think any book or document can help guide planning towards a successful event. In the end, our high-level agenda looked like below having topic theme focuses each day, with the last day planned to follow up on topics and questions that came up during the event.

Day 1: Trust
Day 2: Method
Day 3: Impact
Day 4: Innovation
Day 5: Acceleration

Was the summit a success?

The primary meeting room

How did our event go? I think it went well and accomplished many of our goals. Everyone came away saying it was a great event and that we should do them more often. We sent out a survey to all the attendees and scored higly across the board. For a bit, more details on our most recent event read our Distributed Teams Amsterdam Summit Recap.

September Software Links

2017-09-25T00:00:00+00:00

Software Development

Data

Readable plots with python, with 538 style
Data Mapping, High Resolution Settlement Layer thanks @alepietrobon
How to grow a data driven team
Reading an explain analyze query plan

Ruby

Random

The moon blew up without warning

Random Links by Sean MacEntee

Markdown to Confluence Wikis

2017-08-27T00:00:00+00:00

Markdown converted for Atlassian’s Confluence

If you ever have a need to convert a bunch of markdown or html files to confluence it is really easy. Their Rest-API is pretty clean and if you use Ruby there is a small Confluence API gem to make it even easier.

Why Move To Confluence?

Why would we move our documentation from Markdown in a git repo, to Confluence? Well our team is growing and we interact with more and more non developers, who could benefit and contribute to our documentation. While some of our documentation was very technical much of it wasn’t. It would be helpful to our Support, IT, Project Management, and Stakeholders. Markdown and git aren’t the best way to get other folks comfort able reading or contributing to our documentation.

Increased collaboration
Simplified access control to read
Ability to mix ticketing, widgets, and wiki docs

That was enough for us to decide on moving our documentation over.

Bulk Conversion & Upload

I didn’t want to take the time to move over 80+ documents by hand. So I wrote a script to help. I created a page in our confluence space for imported docs. I set that as a parent page and then just converted all of the Markdown to HTML and uploaded it in a single step. Linking to the original document in case anything was lost in translation. Now we can pull content and link to the old content, but start building out newer documentation on our confluence space.

Show Me the Code

You can view the gist or see the code below.

#!/usr/bin/env ruby

require 'rubygems'
require 'confluence/api/client'
require 'kramdown'


username = 'your@email.com'
password = 'your_pass'
url      = 'https://your_co.atlassian.net/wiki'
space    = 'DP'
# if you don't want your files at the root give a parent page to nest them under
parent_page_id = 5564541
path     = '/Users/danmayer/projects/DevDocuments'
original_root_path = 'https://github.com/OffgridElectric/DevDocuments/blob/master'

###
# see https://github.com/amishyn/confluence-api-client for usage
###
client = Confluence::Api::Client.new(username, password, url)

errors = []
files = Dir.glob("#{path}/**/*")
converted = 0

files.each do |file|
  if file.to_s.match('.md')
    converted += 1
    next if converted <= 3
    relative_path = file.to_s.gsub("#{path}/",'')
    title = file.to_s.gsub("#{path}/",'').gsub('/','_').gsub('.md','')
    content = File.read(file)
    content += "\n\n\n [original document](#{original_root_path}/#{relative_path})"
    html_content = Kramdown::Document.new(content).to_html

    puts "creating #{title}"
    page = client.create({type:"page",
                      title: title,
                      space: {key: space},
                      ancestors:[{id: parent_page_id}],
                      body: {storage:{value: html_content,
                                      representation: "storage"}}})
    errors << page if page['statusCode'].to_i > 200
  end
end

puts "converted #{converted}"
puts "errors:"
puts errors.inspect

puts 'done'

Sharing Knowledge & Breaking Silos on Distributed Teams with Brown Bags

2017-08-24T00:00:00+00:00

Over the last 2 years or so our team has built up a collection of about 80 internal talks -credit arcticpenguin

I have worked with distributed teams for years now. At Offgrid-Electric (OGE), our tech team is extremely distributed (across timezones, continents, and cultures). One thing that can always be a challenge is feeling like a part of the team. One thing that I have found useful both with team building, and to help with knowledge sharing through the team is brown bags. The term might take you back a bit, but the concept is easy enough. A less formal presentation, talk, or discussion generally with coffee, lunch, or a snack. Obviously, this can be done with co-located teams as well, but with a distributed team and a few best practices they can be very effective.

Brown Bags at OGE

We don’t have too many formal rules, but we try to keep things humming along and have found some good ideals along the way.

The most important is record your brown bags, this unlocks a number of great things. If people can’t all make it across timezones they can catch up later. You can catch up on talks you missed while on vacation. Most importantly, you are building an amazing onboarding resource over time. When folks start at OGE, I have a short standard list of brown bags I always recommend. Then I personalize other recommendations based on their interests and projects they will be getting involved with.

Diving Deep into our Call Center Automation with Dmitry Tsepelev

As you build up a team video collection keep both the video recordings and the slides somewhere accessible by the whole team. We have a shared google drive folder, and dump all the materials in there. It is getting to the point we might need categorized folders, or at least we should adopt a naming convention like 2017–06–15-data-changes.mp4 to help highlight what is dated over time.

Try to follow some scheduled cadence. Our loose schedule is to have a brown bag every two weeks. While we don’t achieve this all the time, if there are meeting heavy weeks or behind schedule projects, we generally catch up and get back into the flow.

To help keep the talks coming and inspire additional ideas create a brown bag signup board. We have talks people want to give as well as talks people want to hear that folks can try to get other people to give. We have a simple shared Asana board for this, but any task management tool would do.

An early version of Annyce Davis “First Do No Harm” talk given as a brown bag.

Another great thing to come out of our team brown bags over the years, has been helping people prep conference talks. If you work with new or experienced speakers encourage folks to practice their talks to the team and get some initial feedback. It is a great way to figure out what is working and what isn’t before giving a talk to a bigger group and to help calm your nerves.

Invite other folks, break the silos, and share cross-functional knowledge that might be a bit outside of everyone’s day to day. Originally our talks were generally just the software team. Then occasionally with the hardware or data teams. Now we generally, depending on the topics, invite a larger group involving IT, Data, Software, Project Managers, Hardware, and Stakeholders.

Make it fun, keep it loose, talks don’t have to be long, be formal slide presentations, or even be about technology. We have occasionally paired a few short lightening talks into a single brown bag. Folks have talked about travel or just given accounts of their experiences while visiting our consumer markets in Africa.

While this has worked for us well, obviously there are always ways to improve. What other ways do team’s use to share knowledge? What ways could we improve our brownbags even more? We would love to hear how it goes if your team starts up internal brown bags, let us know.

Finally, to help get you started I can share some slides from one of our recent internal talks, information architecture change process.

Information Architecture Changes

2017-08-13T00:00:00+00:00

Visualizing DB Schema Changes, even at a quick glance better structures stand out

Information Architecture Changes

Creating a robust Data Model for your business & application is one of the most important things to get right. We still don’t have great tools available for discussing proposed database schema changes. Fixing a bad data model after data has started flowing on a production system is more complicated & time consuming than folks estimate. A series of not fully thought out data decisions early on can cut a team’s velocity significantly. This is due to time being spent on trying to fix bad & invalid data, adding missing DB constraints & validations, and refactoring towards a more appropriate data model.

What can we do to ensure a more robust model from the start and increase the confidence we have in data model changes?

A Proposal For More Communicative Data Model Changes

Below, I will lay out a proposal we are iterating on as part of our OGE team best practices.

Our Goals:

Ability to communicate anticipated impacts of our data model changes clearly across Development, Project Management, & Business Intelligence teams
Ability to quickly visualize changes (faster feedback loop)
Reduce bad data models making it to production
Support both interactive & asynchronous communication styles that work best for our distributed team
Ability to see high and low level details using the same data
Easily integrates into our tool chain

While this does add additional overhead while working on data changes to our system. It helps us to collaborate and socialize the data model. Working towards ubiquitous language across our teams and with our stakeholders.

It is important to note this is for data schema changes not for data migrations, which I have written about how we handle in the past.

Data Model Change Process

The process creates some artifacts that help support a robust understanding and conversation around the data changes.

A Pull Request (PR) is used to group the artifacts, description and reasoning for the change, and to focus the discussion
A high level visual asset on changes, to quickly understand the lay of the land, so to speak (including before and after images)
A low level view showing column level removals and additions to the schema, colorized via git diff support

High Level Visual Assets

The high level view are images that can include class or class and field information about the models under change. The PR will typically include both the before and after images so one can see how the classes and associations will change over time. If fields are moved you can also see a class shrinking as fields move to newly associated objects. This type of view can help see the bigger relationships and structures of the data model.

A high detail visual representation of the expected end result

Low Level Details

By building image models from a textual diagramming format (DOT files), one can get much more details and control for the diagrams.

The low level textual details will result in a git diff easily viewed as part of the PR. It makes clear specific fields and associations that are added and removed. While this might be verbose for some folks, it is the level of details needed for a developer implementing and the business intelligence analyst reviewing. Seeing the specific fields makes it clear if the data will meet their needs. It can also be very helpful to discuss low level details such as field type.

A textual diff showing low level details when refactoring out a phone number

Example Live Refactoring Your DB Model

It might be easier to understand the goal with a quick animation to show the flow, than just describing it. Here is the flow one can expect to get into when working on DB model / schema changes.

Team Discussion Workflow

This flow allows for a quick feedback loop, where folks in a room or on a screen share meeting could quickly diagram a model together. Making quick and small edits to the textual representation and generating new versions to discuss. This flow requires some initial work by a developer to prep the original model and DOT files.

Discuss original model image
Edit DOT files during discussion
Generate new model image

Loop until the group is satisfied with the results

Detailed Developer Workflow

This should walk through the steps required to implement this process on your own project. This process could easily be implemented and integrated into various frameworks, but our specific details use tooling for Ruby on Rails to further simplify the efforts of generating these artifacts.

Generate or update initial model on dev branch
- generate dot file: FOCUS=true VERBOSE=true OUTPUT=doc/customers.dot rake diagram:models:customers
- generate image: neato -Tpng doc/customers.dot > doc/customers.png
- check in the un-edited DOT & image files
- commit to dev branch
- this will serve as your before to diff against later
Create a feature branch to work on your changes
- new branch: git checkout -b feature/db_phone_change
- edit to reflect your changes: open doc/customer.dot
- convert updated DOT file to updated image: neato -Tpng doc/customer.dot > doc/customers.png
- view PNG & refactor until happy
- create a PR and upload the before and after images, diff will be automatically created for the DOT file changes.

Change Process Presentation

This is a version of the presentation I gave internally when getting our team on board. Feel free to have a look Information Architecture Change Process

Code & Development Details

See below for details on how to implement this yourself.

Dependencies

There are many tools that could do this. If you don’t have the ability to automatically create diagrams from your code or access to graphviz, you could build out much of the diagrams by hand with a tool like draw.io. If you are using Rails, I recommend the below setup.

Add to your `gemfile` in your dev/test group

We use the Railroady gem to generate our initial DOT files & images.

group :test, :development do
  gem 'railroady'
end

Install `graphviz` and tools if you don’t already have it

brew install graphviz

Setup some rake tasks for custom focused diagrams

By default Railroady has some great Model diagrams.

rake diagram:models:all
rake diagram:models:brief

They tend to be to verbose by default. You can easily customize the output to focus in on the models under discussion. Below is some quick code to highlight how to customize the output.

namespace :diagram do
  namespace :models do

    def excluded_models
      (['PaperTrail::Version']+(ENV['EXCLUDE_MODELS'] || '').split(',')).join(',')
    end

    ####
    #
    # Convert dot files to images
    # neato -Tpng doc/customer.dot > tmp/customers.png
    #
    ####
    def generate_for_files(files, output_file)
      files = files + ['app/models/user.rb'] if ENV['WITH_USER']
      options = ENV['VERBOSE'] ? '--alphabetize' : '--brief'
      if output_file.ends_with?('dot')
        `EXCLUDED_MODELS=#{excluded_models} railroady -M -s #{files.join(',')} --show-belongs_to #{options} > #{output_file}`
        doc = File.read(output_file)
        doc = doc.gsub('\l',"\\l\n")
        File.open(output_file, 'w') {|f| f.write(doc) }
      else
        `EXCLUDED_MODELS=#{excluded_models} railroady -M -s #{files.join(',')} --show-belongs_to #{options} | neato -Tpng > #{output_file}`
      end
      puts "find the file in #{output_file}"
    end

    desc 'Generates an class diagram for all customer models.'
    task :customers do
      output = ENV['OUTPUT'] || 'tmp/customers.png'
      customer_ignores = if ENV['FOCUS']
                           %w(Tagging Tag Ticket Task).join(',')
                         else
                           ''
                         end
      ENV['EXCLUDE_MODELS'] = ENV['EXCLUDE_MODELS'].to_s + customer_ignores
      files = %w(
      app/models/customer.rb
      app/models/contact.rb
      app/models/lead.rb
      app/models/sms_notification.rb
      app/models/phone_block_list.rb
      app/models/call.rb
      )
      generate_for_files(files, output)
    end
  end
end

Most Valuable Code

2017-07-20T00:00:00+00:00

Obama writing JS with code.org.

The Value of Code

While both of the below examples are obviously a bit exaggerated they do capture the point. The code we write has value because it solves some challenge.

The Best Code is No Code At All
-Jeff Atwood

Programming is a means to an end, not an end in itself. You should be trying to do as little of it as possible to make the thing that you want.
-Jake Levine

Most Valuable Code I’ve Written

Given that I think the most valuable code I have currently ever written is a stupid and simple method. This is a recreation of the method below avoiding un-important internals.

def ensure_safe
  begin
    yield
  rescue Exception => exception
    message_exception = exception.exception("ensure_safe rendered the page, saved exception: #{exception.message}")
    message_exception.set_backtrace(exception.backtrace )
    if Rails.env.production?
      ExceptionReporter.report(message_exception)
    else
      Rails.logger.error "******* Error *******"
      Rails.logger.error "error safe render, on production this would report the error, but render nothing"
      Rails.logger.error "error: #{message_exception.class} #{message_exception.message}\n  #{message_exception.backtrace.join("\n ")}"
      raise exception
    end
    return ''
  end
end

The above method when added to application_helper.rb could be called in view wrapping a render method.

<div>
 <%= ensure_safe { render 'show_results', results: data } %>
</div>

When used on important pages, it would generally result in all the primary content being rendered. Then it would silently return an empty partial to the user, while reporting the exceptions up to our developer team. Often the bug was in a navigational, promotional, advertising, or upsell element. Many times the error would only occur in specific data combinations which is why it wasn’t caught in testing. It prevented many smaller error from ever taking down our primary content or purchase flow ever again, which was huge. In total this probably prevented hundreds of thousands of errors from ever reaching our end users eyes.

Stands the Test of Time

My thoughts about this particular piece of code isn’t new. In face below is what I had to say about it when I moved on from LivingSocial a few years ago in a email farewell to fellow engineers.

ensure_safe: I still consider this the most valuable piece of code I have ever written. Luckily, the era of frequent exceptions and downtime has passed, but back when we were pushing 12+ minute deploys and we didn’t have ls-deploy rollback when a bug went live we would often have 20+ minutes of purchase flow downtime. Knocking out deals#show or a purchase page would halt all revenue for about 20 minutes while we scrambled to revert and redeploy. The silly method would let some portions of the view safely explode while sending us exceptions but keeping the site running. It often headed off 10-30K exceptions per incident often with no users the wiser that an issue occurred. I wish there was a graphite stat counter on how many times that method saved the site from going down ;)

What Piece of Code Surprised you?

What silly piece of code has surprised you over the years? Something that was perhaps thought of as a quick fix or temporary solution that has stood the test of time of proved so valuable that it has endured? I am sure other folks have some great examples of the method that has become a myth or perhaps infamous for it’s craziness.

Phone Number DB Types

2017-06-26T00:00:00+00:00

tweeting about DB phone number formats

How to best store & query phone numbers in Postgres DB

Update: 06-27-2017, See the update & round 2 section below for improved and corrected data

After some frustration cleaning up some of our information architecture related to phone numbers. I posted the tweet at the top of this article.

It actually lead to far more responses, questions, and discussion about the best way to store numbers… It made me think that perhaps our quick attempts to extract the best process out of multiple production table cleanups, could be missing something. I decided it would be pretty easy to try to dig in a bit deeper for a more data backed answer. I definitely don’t claim to be a DB performance expert, so perhaps we had reached some wrong conclusions.

a portion of the sub-discussion from the original tweet

a pretty strong statement saying we got this wrong

Open Questions

The premise we ended up with was that it was worth a tradeoff of some complexity to store phone numbers as bigint vs varchar in our Postgres DB.

Everyone admits it is easier to work with a string number and it avoids the leading 0 problem. Is there a valid reason to use bigint vs varchar?

The thread brought up a few open questions:

What is the difference in DB table storage size?
What is the performance difference
- the difference on insertion time?
- the difference on query time?

OK, we should be able to answer these fairly easily.

Benchmark Setup

Create Simple Tables with our unique index

class DemoTables < ActiveRecord::Migration[5.1]
  def change
    create_table :demo_int_phone_numbers do |t|
      t.bigint :phone_number, limit: 8, null: false
    end
    add_index :demo_int_phone_numbers, [:phone_number], unique: true

    create_table :demo_string_phone_numbers do |t|
      t.string :phone_number, limit: 8, null: false
    end
    add_index :demo_string_phone_numbers, [:phone_number], unique: true
  end
end

Create our simple AR models

class DemoIntPhoneNumber < ActiveRecord::Base
end

class DemoStringPhoneNumber < ActiveRecord::Base
end

Build a simple benchmark script

namespace :demo do
  task :perf => :environment do
    DemoStringPhoneNumber.destroy_all
    DemoIntPhoneNumber.destroy_all

    GC.start
    puts "benchmarking insertions"
    Benchmark.bm(7) do |bm|
      bm.report do
        (0...200_000).each do |val|
          DemoStringPhoneNumber.create!(phone_number: val)
        end
      end
    end

    GC.start
    Benchmark.bm(7) do |bm|
      bm.report do
        (0...200_000).each do |val|
          DemoIntPhoneNumber.create!(phone_number: val)
        end
      end
    end

    GC.start
    puts "benchmarking queries"
    Benchmark.bm(7) do |bm|
      bm.report do
        (0...200_000).each do |val|
          DemoStringPhoneNumber.where(phone_number: val).first
        end
      end
    end

    GC.start
    Benchmark.bm(7) do |bm|
      bm.report do
        (0...1200_000).each do |val|
          DemoIntPhoneNumber.where(phone_number: val).first
        end
      end
    end
  end
end

Table Size Results

I was wrong on table size… Looking back as we moved phone numbers from many tables to a single phone_numbers table a number of them were using different types, including varchar, which did make the tables bigger… but that was because a number of the old tables hadn’t set a limit meaning it defaulted to varchar(255). Once you set a limit bigint and string with limit: 8 will result in the same size of table.

select pg_size_pretty(pg_relation_size('demo_int_phone_numbers'));
=> 8656 kB

select pg_size_pretty(pg_relation_size('demo_string_phone_numbers'));
=> 8656 kB

Insertion Performance Results

I actually didn’t expect a difference on this but someone mentioned there could be a slower insertion time… It looks like nothing noticeable on insertion which is nice.

# benchmarking insertions
              user     system      total        real
        385.130000  35.700000 420.830000 (568.695180)
              user     system      total        real
        371.370000  34.110000 405.480000 (543.861259)

Query Performance Results

While numerous performance blog posts say that a Postgres btree index is more performant for integers than strings. Sample data doesn’t show as big of a difference as I was expecting. The performance benefit is questionable. It is there, but doesn’t seem worth the effort. Since this is a quick and dirty benchmark one would have to be more systematic to prove out a real difference. I think my simplification of just having sequential numbers inserted could not realistically simulate how a production index ends up.

# 200,000 queries against String Table
              user     system      total        real
        194.320000  15.400000 209.720000 (265.271056)
        
# 200,000 queries against Int Table        
              user     system      total        real
        160.390000  13.010000 173.400000 (210.774260)

In that end this does show a small difference, but nothing to be impressed about.

Bad Assumptions

From our cleanup of multiple tables we had multiple issues and conditions which lead me to conflate some of the issues.

Our DB had a number of columns that were varchar(255) opposed to using a correct limit. This entirely counts for the table size difference.
I am curious if having unused space in a varchar field can still impact query performance, I am not sure about this.
We weren’t enforcing a unique index, instead sometimes relying on Rails validates :field_name uniqueness: true, and sometimes not even enforcing uniqueness. Which I think could impact query time more significantly when there are duplicates. Proving this out would require more benchmarking.

Conclusions

Let’s revisit our questions:

Is there a valid reason to use bigint vs varchar?

Not really, there may be slight query performance wins, but far less than I thought.

What is the difference in DB table storage size?

None, I was wrong

What is the performance difference on insertion time?

None, which I expected but some other folks expected to see.

What is the performance difference on query time?

Slight difference, more detailed benchmarking would be good… Initial results show this to be far less significant than initially thought. I will say this one is dependent on the performance characteristics you see in production. That being said my assumptions here were still a bit flawed, and I was surprised by the results.

Update: Limit Not What You Think…

OK so this post made sense to folks, but then people were asking why I am limiting the phone number to 8, which wouldn’t work in various countries… in fact it wouldn’t work for our countries either… I will need to rerun my benchmarks and I believe get different results.

We have historically put limit in our int fields when doing something other than the default. The limit has a different meaning for numerical and string types for Rails DB migrations.

The :limit on bigint or int in Rails refers to the byte size not a length limit. bigint limit: 8 is redundant as 8 bytes is the max. Prior to Rails supporting the bigint type, the way to get a bigint was to actually use int limit: 8. In fact if you look at the schema after creating a bigint limit: 6 the limit is just ignored by postgres (and by mysql it seems). The DB storage type ends up as a bigint with 8 bytes of storage. This post explains the oddness of int limits well.

Limit on string dos what you expect and limits the length of the string. The way I was generating phone numbers never tested this as I simply generated between (0...200_000).

# storing a typical TZ number for us
DemoIntPhoneNumber.create!(phone_number: 2551234567890) 

# storing an actual max, which is far larger than we need
DemoIntPhoneNumber.create!(phone_number: 9223372036854775807)

# lets try those with our string limit version
DemoStringPhoneNumber.create!(phone_number: 2550677939180)
ActiveRecord::ValueTooLong: PG::StringDataRightTruncation: ERROR:  value too long for type character varying(8)

# and clearly the max won't work at all...

My benchmark isn’t valid as varchar(8) isn’t really the same as bigint. The fact that tables size was previously the same was a bad coincidence, which I think shows that bigint is actually more efficient.

Let’s try again, varchar actually needs to hold 13 for our max length number, and 15 for German numbers. So we will compare bigint to varchar(15), even though bigint can technically hold numbers of length 19.

Benchmarking Round 2

Setup Changes

Let’s create two new tables the first verifies with the type bigint that the limit option is ignored it just creates a table with an 8 byte bigint Postgres type. The second creates a new string table with a varchar(15)

class DemoTablesTwo < ActiveRecord::Migration[5.1]
  def change
    create_table :demo_smallint_phone_numbers do |t|
      t.bigint :phone_number, limit: 6, null: false
    end
    add_index :demo_smallint_phone_numbers, [:phone_number], unique: true

    create_table :demo_bigstring_phone_numbers do |t|
      t.string :phone_number, limit: 15, null: false
    end
    add_index :demo_bigstring_phone_numbers, [:phone_number], unique: true
  end
end

OK then we will need a quick AR class

class DemoBigstringPhoneNumber < ActiveRecord::Base
end

Then the benchmark improvements

namespace :demo do
  task :perf => :environment do
    DemoBigstringPhoneNumber.delete_all
    DemoIntPhoneNumber.delete_all
    base_number = 2550000000000
    max_number = (base_number + 150_000)

    GC.start
    puts "benchmarking insertions"
    Benchmark.bm(7) do |bm|
      bm.report do
        (base_number...max_number).each do |val|
          DemoBigstringPhoneNumber.create!(phone_number: val)
        end
      end
    end

    GC.start
    Benchmark.bm(7) do |bm|
      bm.report do
        (base_number...max_number).each do |val|
          DemoIntPhoneNumber.create!(phone_number: val)
        end
      end
    end

    # lets not just have sequential numbers lets get some better data by inserting arbitrary numbers
    (75_000).times do
      val = max_number + rand(90_000_000)
      begin
        DemoBigstringPhoneNumber.create!(phone_number: val)
        DemoIntPhoneNumber.create!(phone_number: val)
      rescue => err
        # ignore duplicates that occur because of rand
        puts err
      end
    end

    GC.start
    puts "benchmarking queries"
    Benchmark.bm(7) do |bm|
      bm.report do
        DemoBigstringPhoneNumber.find_each do |val|
          DemoBigstringPhoneNumber.where(phone_number: val.phone_number).first
        end
      end
    end

    GC.start
    Benchmark.bm(7) do |bm|
      bm.report do
        DemoIntPhoneNumber.find_each do |val|
          DemoIntPhoneNumber.where(phone_number: val.phone_number).first
        end
      end
    end
  end
end

Table Size Results

After running a number of tests, the tables size issue does become more clear. This now matches my original expectation.

select count(*) from demo_int_phone_numbers; => 339669
select pg_size_pretty(pg_relation_size('demo_int_phone_numbers'));
=> 14 MB

select count(*) from demo_bigstring_phone_numbers; => 339669
select pg_size_pretty(pg_relation_size('demo_bigstring_phone_numbers'));
=> 17 MB

Insertion Performance Results

Again no real difference.

Query Performance Results

Results are are starting to show a bit more more of a difference. Likely, more valid as this time we aren’t just querying sequential numbers but we have some randomness introduced.

# queries against String Table varchar(15)
              user     system      total        real
        377.940000  28.550000 406.490000 (550.554212)
        
# queries against Int Table        
              user     system      total        real
        302.260000  23.320000 325.580000 (403.738332)

Revisited Conclusions

Let’s revisit our questions:

Is there a valid reason to use bigint vs varchar?

Perhaps, the query improvements are smaller than I thought, but noticeable. The table size difference is real, but also not that large. It would really depend on your needs and the size of your tables. In the end Google says never store phone numbers as numeric data (rule #24), but given a known set of countries and query performance, you can examine the tradeoffs.

What is the difference in DB table storage size?

Yes, there is a difference. The type bigint is a more efficient way to store 15 digit numbers.

What is the performance difference on insertion time?

None, which I expected but some other folks expected to see.

What is the performance difference on query time?

The larger set of test data and introducing randomness into the data seems to show an increasing improvement for bigint queries. While it is still small than I anticipated, it does seem to have a real impact.

Unused DB Columns, via logs

2017-06-20T00:00:00+00:00

a search for ‘unused columns’ brings up this image from Flickr user @denisbin.

Detecting DB Column Usage

I have always been interested in ways to automate & cleanup old unused code. Removing the clutter helps focus in on the things that matter and make clear patterns emerge.

This post continuing on some database cleanup work. The first post of the series is, cleanup unused tables in your database

After taking on full tales, I wanted a way to find unused fields. I came up with two easy ways to try to automate detecting this in a Rails app.

Previously covered: Query the DB’s paper_trail history
In this post: We will cover query logging and searching field usage via a log aggregator.

DB Field Usage Output

The output depends on your log search tool. We currently use Sumologic for our log aggregation so all the examples will focus on Sumologic. That being said it would be simple do to this in Splunk and other log aggregation tools.

Aggregated Search Log

If you are running a system that is on more than a single server you should setup aggregated log search. Distributed Logging is basically a requirement for micro-services, and honestly the moment your logs aren’t on a single server.

In Rails to extract column usage out of your logs you need to enable ActiveRecord logging on the application you are targeting. In general this isn’t enabled on production, but you can enable it for a period of time to query later. There can be a slight perf hit for logging query details, but more likely it just increases you log volume, as the perf isn’t very noticeable.

Ways to enable logging:

# enable debug on production add to environments/production.rb
config.log_level = :debug

# debug just for ActiveRecord
ActiveRecord::Base.logger.level = :debug

ActiveRecord Log Format

The specifics of your log header, ender, single line logging, etc… The specifics shouldn’t matter. If you turn on default ActiveRecord logs you should have something like below. If that is making it to your log querying tool you should be good to go.

TS=2017-06-20T14:46:34.881881 PID=00063 USER=000000 LEVEL=I |   [1m[35mSQL (1.6ms)[0m  [1m[32mINSERT INTO "tracks" ("latitude", "longitude", "accuracy", "action", "action_id", "timestamp", "battery_percentage", "used_memory", "max_memory", "created_at", "updated_at", "gps_enabled") VALUES ($1, $2, $3, $4, $5, $6, $7, ...

Verify you are seeing that in your log collector and then it is as simple as crafting a good query.

DB Field Log Search

This will be a bit different for any tool, but basically you will be looking to extract the table and all the fields into your tools parsed fields. At that point you can run a query over whatever time period you want to see the count or last timestamp of a given table / field usage.

Find all table / fields updated along with the number of updates in the last search period.

_sourcename=*app.log *insert into |
 parse "INSERT INTO \"*\" (*)" as table, table_fields |
 count(table_fields) by table, table_fields" |
 sort by _count

Find all field usage where the field matches the world ability… Just a simple example of narrowing in on fields you think are likely dead.

_sourcename=*app.log *insert into |
 parse "INSERT INTO \"*\" (*)" as table, table_fields |
 count(table_fields) by table, table_fields |
 sort by _count |
 where table_fields matches "*ability*"

Find all fields with 0 results

This is an exercise for the reader. As it is simple to do human queries on fields you already suspect. If you want to detect anything unused without a good human guess you will need to automate your querying. Basically you will need to implement API access to your log query tool, then iterate through each table and field querying for the last usage, over whatever time period you care about I didn’t do this exercise as using papertrail to detect unused DB columns worked well enough for me.

Paternity Leave

2017-06-15T00:00:00+00:00

I am on paternity leave, actually my second of 2 staggered Paternity leaves. I am about 1 week into my second leave, hanging out with little Theo. I have some time off to bond with my kid, but more importantly my wife has some time to go back to work and stay focused knowing, I’ve got this.

Men’s Paternity Promotes Gender Equity

My wife started sharing a bunch of information about paternity leave when we started discussing having a child. The choices men make to be more involved with the kid can help women who choose to work to avoid some of the associated career setbacks that often come with becoming a mother. There is some interesting research going on here, it seems to impact both work and home life. Increasing men’s participation in chores and playing with children. I suspect that has something to do with the empathy gained, spending time at home alone with the kiddo.

Gotta make that dollar…

A mother’s future earnings increased 7% for every month that her partner took parental leave.
– Could Paternity Leave Policies Close The Wage Gap?

USA, get it together ;)

79: Number of countries that have laws requiring paternity leave
– Paternity Leave By The Numbers

Tech industry is leading the way (change.org most generous leave in tech), keep pushing it forward.

this year alone, Amazon, Netflix, and Microsoft have all announced extended parental leave. This is of particular importance because for the first time, we’re seeing a move towards expanded leave for fathers as well as for mothers.

found that men increased their time spent on household chores by 23% after taking paternity leave.

Another study by the Columbia University School of Social Work found that men who took more than two weeks of leave were more likely to handle childcare tasks like diapering, feeding, and playing, and were more likely to contribute to household chores.
– Women Benefit Most When Men Take Paternity Leave

behavior-modification, how scientific ;)

a behavior-modification tool that has been shown to boost male participation in the household, enhance female participation in the labor force, and promote gender equity in both domains.

Among labor economists, overly long maternity leaves are now recognized as creating a barrier to pay equity. At home, meanwhile, long leaves result in women doing most of the housework and child care. (referencing that it is better to have some leave for mom and dad opposed to long mom only leave)

policy makers decided to make men an offer they would feel ashamed to refuse

study suggests that paternity leave might give men a new mind-set, prompting them to trade more money for more time at home, more flexibility, or both.
– Daddy Track: The Case for Paternity Leave

It’s good for for Father / Child Relationships

It helps you bond with your baby, and it has been showm to increase the interest and time a father spends with his kids through out life. Ever hear about men jealous of the relationship between kids and their mothers? It is easy to see why kids could be closer to their moms, but taking paternity leave helps you to develop your own bond.

Daddy Paternity Leave Activities

What do we spend our days doing?

Bi-lingual song & story time at the library
Visiting Daycares trying to get a spot locked down (the good times can’t go on forever).
Long walks to parks
Running errands, Theo enjoys looking around shops
Gardening, while I haven’t yet got him to pull weeds, he does seem to love watching me do it.
Baby Dance party
Crying and refusing to calm down (OK this part isn’t very fun)
Teaching the kid about Operation Ivy, and other Punk Rock.
Writing code and blog posts, the kid has to sleep sometime.
Visit a museum, kiddo loves being carried around to look at new things

Paternity Leave Stigma

96 percent of men are back at work after two weeks, while 13 percent do not take a single day off.
–Boston College Research

While there still is some stigma and even well intentioned folks unintentionally may throw a few jokes your way. It is getting better, and the best way to help keep the progress going is normalize dad’s taking of as much time as mom’s do.

I say, “Be the change, you want to see.”

I can tell you people are extremely nice to me, when I am out as a dad ‘alone’ with out baby. Random folks hold doors, elevators, make nice comments, etc. I am not even kidding that earlier today an old woman pulled her car over to the side of the road and shouted out the open window, “It is so nice to see dad’s taking their kids for a walk too.” So while it is shifting this way, it is still rare enough to see a man caring for a baby that folks are surprised to see it.

So if you can again, take some time, Share the “burden.” Enjoy some time with your kid, while they like you.

Dude, Check Your Privilege!

Yep, I hear that.

Taking maternity or paternity leave in the US at all means you are pretty privileged. Only 12% of Americans have access to paid parental leave and the US is one of only three countries left in the world that do not guarantee paid maternity leave.

As with most of my posts and life, I understand my ability to take leave, work from home, and live with reduced income from my wife and I taking leave is lucky. As is the flexibility of my career / current job to support staggered paternity leave (split between baby’s birth and Mom returning to work). As are many other things, the key point is if Paternity leave is available and viable to you as a mother or father, take it.

Also, it is fine if a Mom chooses not go back to work, a father taking paternity leave is still a good thing that lets you bond with your child and promotes all people’s choices.

Unused DB Columns

2017-05-22T00:00:00+00:00

a search for ‘unused database’ brings up this awesome wikimedia image for some reason.

Detecting DB Column Usage

I am always interested in ways to automate and cleanup old unused code. Removing the clutter helps focus in on the things that matter and make more clear patterns emerge. Continuing on some work to cleanup unused tables in a database, I really wanted a way to find unused columns / fields. I came up with two easy ways to try to automate detecting this in a Rails app. The first is to query the DB’s paper_trail history, which is easy if you are already using paper_trail. The second I will cover in another post soon.

Column Usage Output

What does column usage look like, well for each table it is a list of fields and the last date it was modified. The simple output should be pretty clear.

____________________________________________________________
field usage for Customer
************************************************************
barcode: 2017-05-20
email: No updates in 3 months
fixed_location_id: No updates in 3 months
agent_id: No updates in 3 months
account_id: 2017-05-20
house_latitude: No updates in 3 months
house_longitude: No updates in 3 months
house_location_accuracy: No updates in 3 months
...
notes: 2017-05-20
last_paid_at: 2017-05-20
____________________________________________________________
field usage for Lead
************************************************************
fixed_location_id: No updates in 3 months
custom_location_name: No updates in 3 months
agent_id: 2017-05-20
captured_by_id: 2017-05-20
converted_at: 2017-05-20
service_level_id: No updates in 3 months
source: 2017-05-20
latitude: 2017-05-20
longitude: 2017-05-20
status: 2017-05-20
assigned_at: No updates in 3 months
...

Obviously, a developer will need to interpret this, some tables aren’t often updated it doesn’t mean they have no purpose. For example we have some configuration tables that are only modified when we integrate new partners into our system. So they will show up in my list with “No updates in 3 months” but that can be quickly filtered out.

Detect Columns via Paper_Trail

If you use the gem paper_trail to track the history of changes on some of your models, you are already recording when any column was last changed. All one has to do is come up with the queries to pull the data out. Doing this by hand would be a huge pain, but luckily it is easy to hook into our code and automate the process. We just find all models that support paper_trail, then iterate through the fields.

Show Me the Code

The below functions and tasks can be added to any rake task file.

def find_ar_models
  Rails.application.eager_load!

  array = ActiveRecord::Base.descendants.collect { |x| x.to_s if x.table_exists? }.compact - ['ActiveRecord::SchemaMigration']

  array.reject { |x| x.split('::').last.split('_').first == "HABTM" }
end

desc 'List when column was last updated'
task :list_recent_columns => :environment do
  ignore_models = %w(RailsSettings::Settings Devise::Oauth::Client)
  models = find_ar_models
  models = models.reject{|model| ignore_models.include?(model.to_s) }
  
  ignored_columns = %(id created_at updated_at)
  results = {}
  
  models.each do |model|
    model_results = {}
    model = Kernel.const_get model
    next unless model.paper_trail.enabled?
    puts "_"*60
    puts "field usage for #{model}"
    puts "*"*60
    model.attribute_names.each do |attr|
      next if ignored_columns.include?(attr)
      result = PaperTrail::Version.where(item_type: model.to_s)
                 .where("created_at >= ?",[3.months.ago])
                 .where("object_changes ILIKE ?",["%#{attr}%"])
                 .order("created_at DESC")
                 .limit(1).first
      model_results[attr] = result&.created_at&.to_date || 'No updates in 3 months'
      puts "#{attr}: #{model_results[attr]}"
    end
    results[model.to_s] = model_results
  end
end

Quick Note on Performance

Make sure you have indexes on created_at and item_type or this will be very slow. If you don’t find another way to limit. For example we don’t have an index on created_at so I did a query to find the max_id of 3 months ago and use that as the limit opposed to created_at which makes the queries run without any perf issues.

Weekly Software Updates

2017-05-18T00:00:00+00:00

Stay in Sync

When your team is distributed, it can be hard to keep in sync, feel a part of a team share knowledge, pass on learnings, and best practices. One thing I found helpful when I was part of a of partially co-located and part distributed. An async weekly software update, collecting announcements, important dates, and great links. My manager Doug Ramsay and my next manager Jessie Link both followed this practice in some form or another… The basic format was some announcements, team updates, and interesting links. I honestly can’t recall if the weekly update had a name, but in my mine I know I always called Jessie’s updates, “Jessie Link’s Links”. I believe originally the updates were sent to the whole team via email, then email / campfire, and eventually Slack.

This is different than team stand ups or more formal processes and can be looser sometimes sharing funny pics, posts, or birthdays. I always enjoyed reading Jessie & Doug’s links and it has been great to continue the practice.

Software Update Template

As I have been helping keep a fully distributed team working together and up to speed. I have found it helpful as well. I have slightly changed the format over time, but the basics are in the template below.

As another week starts, ...funny_or_motivating line...
	
:keyboard: Dev first-responder: <>
:todo: PM Release Coordinator: <>
:calendar: Holidays / Vacations
     * 
:speaker: Updates:
     * 
:newspaper: Articles:
     * 

What to Cover

Rotations: If there is any sort of rotation like a First Responder list who is on rotation that week. We have that concept both for the dev team and our PM team. So they are listed first.

Calendar: While all the specifics on the calendar should be on some sort of team shared calendars it is good to call out and place reminders for the team in this section. People that might be out of office or specifically calling out holidays that impact parts of the company.

Updates: In this section I note any big items occurring during the week, such as a large anticipated release. Internal or external talks by our team. It’s a great place to call out something awesome someone on the team recently did. New folks joining the team, hiring announcements, etc…

Articles: This is a collection of good and relevant links I have seen in the last week or so. Sometimes sent in by the team (share the link and credit who passed it along), sometimes if is one of teams blog posts, or something I picked up on twitter. This is a great way to help introduce new ideas that can be discussed by the team later.

Software Update Example

An example of a filled out example below, to help show what this translates into each week.

As another week starts, the sun still creates energy
    
:keyboard: Dev first-responder: John
:todo: PM Release Coordinator: Galt 
:calendar: Holidays / Vacations
     * Dan is working from Paris Monday / Tuesday
     * Annyce is at Google I/O on T - F
     * John is off on Friday
     * Tanzania folks are out of office for a Holiday Friday
:speaker: Updates:
     * Software Support is out this week so let’s try to pay extra care to #software-bugs
     * welcome Konrad, onboarding new dev on the 21st
     * We are looking for a good PM pass it around: https://medium.com/electric-africa/product-manager-service-logistics-41f5b404a8dd
:newspaper: Articles:
     * Sinatra hits 2.0, https://github.com/sinatra/sinatra/tree/v2.0.0
     * Best talks from Railsconf, http://confreaks.tv/events/railsconf2017
     * Switching to Kotlin, https://medium.com/@magnus.chatt/why-you-should-totally-switch-to-kotlin-c7bbde9e10d5

Which when rendered becomes.

As another week starts, the sun still creates energy

:keyboard: Dev first-responder: John
:todo: PM Release Coordinator: Galt
:calendar: Holidays / Vacations
- Dan is working from Paris Monday / Tuesday
- Annyce is at Google I/O on T - F
- John is off on Friday
- Tanzania folks are out of office for a Holiday Friday
:speaker: Updates:
- Software Support is out this week so let’s try to pay extra care to #software-bugs
- welcome Konrad, onboarding new dev on the 21st
- We are looking for a good PM pass it around
:newspaper: Articles:

Distributed Team Tips

2017-04-26T00:00:00+00:00

Distributed Team Tips

credit: wikimedia.org

A awhile ago a developer friend @Qrush asked folks for thoughts on how distributed teams should work on twitter. He asked specific questions, I responded and thought it might be helpful to share the slightly cleaned up answers with others.

How Distributed?

A quick note to make the tips a bit more relevant. Our company is highly distributed across timezones, cultures, and languages. This means some things that work for people just dealing with East Coast and West Coast, don’t work so well when handling from Denver to Tanzania. Obviously, keep it simple and if you don’t need as many processes around managing timezones, don’t worry about it.

What’s one unique thing you feel your team does differently than other orgs to work better with everyone across different timezones/asynchronously?

At Offgrid Electric we have done a couple things, I don’t think there is any one magic bullet but it is a combination of efforts that helps to make a highly distributed team work.

Time Zone Management Guidelines: I wish I could take credit for this, but it was started and created above my pay grade. The company had group discussions, surveys, and maintains a shared doc. The doc covers tips and examples to help folks with time zone etiquette and best practices.
- It also manages a company calendar showing the ‘recommended’ hours for cross time zone meetings.
Maker Day Wednesday (No Meeting Wednesday): This is on the company calendar for all the groups that are software stakeholders (operations, finance, PM, etc).
- This is frequently people’s favorite day of the week and often when they get the most done.
Encourage Async Communications:
- conversations should be in tickets (Asana/Jira), documents (lots of google docs attached to tickets with comments, screen shots, etc), or github (PRs)
- slack is for more real time questions, not to record decisions, prioritize work, or cover details.
Meetings:
- Have a document outlining some meeting best practices the company encourages.
- Our company doc pushes “core meeting hours”. Tuesday and Thursday have core hours that work for nearly all timezones, the bulk of meetings should be scheduled during these hours.
- Record meetings, don’t expect everyone in every time zone will always be able to make meetings that are on weird hours, record the video and let them watch async
- Add most people to calendar invites as optional unless they have an active role to play, otherwise they can catch up via the recording Async. This let’s people know the priority of making a meeting that might be at a difficult time.
- In general the company outlines the types of meetings and suggests sending materials for meetings 24 hrs in advance. This is less for team standups and more for sharing with exec or wider audiences.
- I also have a quarterly reminder to go over all my meetings particularly recurring ones… Cut, reduce, shorten, whatever you can.
Respect Cultures Holidays: We have all the countries holidays on our calendars, don’t schedule meetings important to someone during one of their countries holidays.
Shared Team Calendar: The team should add anything on it that would impact their ability to work during normal hours…
- It is really helpful to just know someone is out for an afternoon, if you had planned to sync up with them and miss because of timezones when they are out that is frustrating and can delay decisions by 24hours or more.
In Person Time Matters: Try to get together small groups a couple times a year and everyone at least once a year.
- Remember when budgets get tight the first thing cut is travel. If you are highly distributed this is going to hurt more than you think. Protect it if you can, explain the importance to execs and anyone pushing back on budgets.
- I know some folks that build travel cost into salary hidden away in the budget so it can’t really be taken away later.
Documentation: Maintain lots of documentation…
- Software Team Docs: We try to keep FAQs, expensing policies, travel policies, etc in Git… Want to change things, send a PR.
- Cross Org Docs: Our team shares lots of documentation with other groups, these docs we keep in google docs. This is more accessible and easier to work with other groups.
- Templates: Have a useful format for something? Make a template, then copy, rinse, and repeat.
- Organize, Prune, Archive, Comment, Share, and do whatever to try to keep things up to date and useful.
Microhubs: Try to get 2 people in cities were you hire… While we are all over we tend to have little clusters of a handful of people. It makes it nice to send that group out to dinner for the holidays, just have something in person.
Default Time Zone: Pick the COMPANY timezone. At our company all hours are given in EAT (East African Time), folks know to adjust from where they are to that time when there are any announcements.
- Google Labs has an awesome calendar feature to have 2 timezones display on your calendar at all times.
- We store all dates in UTC as most devs, but most of our tools (New Relic, Sumologic, etc) display data in EAT.
Distributed Bonding: It is hard but do what you can to let distributed folks just get to know each other and stay engaged.
- We do lots of brownbags, some more serious, some just kind of fun and interesting… Keep it light but encourages time to get folks chatting.
- Slack / Team Jokes, I am sure folks love GIFs if they are distributed.
- Emoji :)

Whats your main method of communication? Chat/email/something else? If you could change anything about how that works to help remote folks: what would it be?

Slack, it is our primary comms, and it is a bit noisy… I would encourage people to turn it off a bit more often for focus work.

Make sure there is a agreed on video conferencing software for the company, that it is known to work well in the locations you operate. We have found that Zoom works very well for us and works even with low bandwidth connections.

We are hiring

Know any great project managers? Do you want to work with me on off-grid electricity in Africa?

Product Manager, Logistics

Alexa Compliment My Wife

2017-03-18T00:00:00+00:00

Alexa, tell my wife she’s pretty

Erin holding baby Theo

I have a hobby of playing around with new technology while ignoring the super bowl on TV. This year I decided to learn some of the Alexa API. I had received an Echo for Christmas and thought it would be fun to build a really simple app. I decided to make a little app to help me compliment my wife. We try to compliment each other daily, something we learned in our marriage class ;)

Alexa Skill: “My Wife”

I had the basic functions working by the end of the Super Bowl. After tweaking it a bit more and adding a name variable I decided it was good enough to publish. It is a silly skill, but it was something fun to learn and do in a couple of hours. It has also inspired me to start working on some other Alexa skills. Without further ado, feel free to install the MyWife Alexa skill

User: “Alexa, tell my wife she is pretty.”
Alexa: “Hey Wife, Wow, that confidence looks sexy on you.”
User: “Alexa, tell my wife set name to Erin”
Alexa: “Wife’s name is now set to Erin”
User: “Alexa, tell my wife Erin she is sexy”
Alexa: “Dang Girl is your name Wifi ? Because I’m feeling a connection!”
User: “Alexa, tell my wife I love her”
Alexa: “I want to grow old with you.”

I plan to add some more fun features eventually, but this was amusing enough for me to start with.

Alexa Links

I have been having fun with our Echo’s, you can see how I have been using Alexa for newborns to help us adjust to our baby. I plan to try to do even more with Alexa in the future.

If you want to get started with Alexa programming. Amazon made it super simple, start with the Node.js Alexa SDK Lambda skill, and work from there. See the simple example tutorials, I liked above with my code. I think there are some really cool possibilities opened up by nearly always on voice UI.

sign up as an Amazon Alexa developer
read about the Alexa skills kit for Node.js and here
getting started with skills kit
a good trivia game tutorial
a nice Fact skills step by step guide

Code

This is the full source code, feel free to use it to learn some Alexa basics. It basically extends some concepts from the sample space facts and high low game examples.

Alexa For Newborns

2017-03-05T00:00:00+00:00

A Newborn & Alexa

Theo doing his best Spock hand

Erin and I recently welcomed, Theo, our first baby into the world. We had to learn lots of new things to take care of Theo and ourselves. Particularly, we had to learn how to do many things with a single hand or even better hands free. As I struggled to do things while feeding a baby and trying to keep him quiet so Erin could sleep, I started to get a bit more creative and utilize Amazon’s Alexa more to ease some tasks.

The crib light to check on baby: red light, 10% and 90% brightness, can change colors.

Introduced to Alexa

I had received an Echo Dot for Xmas, we set it up for some music. I got two Alexa plugs and a lightbulb. While kind of neat the home automation wasn’t that helpful. We mostly used it for news flash, Pandora, and some silly fun. I programmed an Alexa app to compliment my wife for me, “Alexa, tell my wife she is pretty.” This makes Alexa spew random funny compliments much to our amusement, but that is the subject of upcoming post. My wife enjoyed following pregnancy tips using the What To Expect skill during her pregnancy as well. We even played around with a fun baby name Alexa skill. In general prior to our baby’s arrival Alexa was kind of neat, but I hadn’t really thought of it as something that would be useful or actually helpful for raising a newborn.

The Lightbulb Moment

The lightbulb went off in my head about how useful Alexa could be when I was exhausted bottle feeding the baby in the middle of the very first night we had him home. The baby was crying after not getting quite enough milk from the last breast feeding, Erin really needed some sleep, and I was supplementing with donor milk in a bottle.

As I sit in a room that is too dark, I am bored and sleepy as I forgot my phone and kindle. Theo is more playing than eating, so I just sit in the silence for over an hour trying to stay awake. I realize that Alexa could solve my problem. I could be entertained without my phone with music, audio books, and more. As for the lights being able to control and dim those by voice would do the trick. When I change the baby and try to get him into deep sleep before returning to our bedroom, I could also have Alexa sooth the baby with white noise back into a deep sleep. All of this sounded great, I wanted an Echo in our nursery right now, and if I had one, I could have used it to order one ;) The next morning in a sleepy haze I ordered another echo and some colored echo-enabled lights.

Lights, Sounds, & Actions

A brief list of our Alexa usage. See the section below for a far more detailed example of how we put Alexa to use.

Lights

The nursery has two lights that can be controlled together or independently

Crib Light: We want to check on Theo occasionally through out the night or one of us needs to pick him up to soothe him out of the room. This light is our primary way to do that. Off course it is Red Light to help us sleep. Theo’s Room: Generally when I walk into the Nursery my hands are full, baby, bottle, extra blankets, etc. Control either the full room or the bedside lamp.

Sounds

Music
Books (Audible / Kindle)
White Noise machine
Podcasts
News Alerts

Actions

Baby Stats
Alarms
Stopwatch
Add to shopping / todo list when your hands are full

Alexa Bedtime Routine

I continued to think of ideas and started to use the echo in other ways. The list below shows how we use Alexa as part of our nighttime baby routine. You can see how we use red lights around all sleep area lighting to reduce light in the middle of the night impacting our sleep.

The two lights in our bedroom including the Crib light, which is the one we use the most.

Due to Theo’s low birth weight and weight loss we had to track feedings, pees, & poops.
- We did this on paper, but towards the end of our tracking discovered the Alexa skill Baby Stats
To encourage milk production and get Theo to gain weight we had to follow a “Triple feed diet”
- Feed at least every three hours (unless baby requests food earlier)
  - "Alexa, set alarm for 3 hours"
- Start feeding with breast feeding at least 10 minutes a side
  - install the Stopwatch skill.
  - "Alexa, open stopwatch and start"
  - "Alexa, ask stopwatch for my status"
  - "Alexa, ask stopwatch to stop"
- Use the Breast pump for 15 minutes
  - "Alexa, set alarm for 15 minutes"
- Bottle feed 1.5 oz
  - "Alexa, ask baby stats to add feeding 1.5 ounces"
During our feed cycle which takes about an hour, we can get some entertainment if we are away from our phones / partner.
- "Alexa, open Pandora"
- "Alexa, open audible book"
- "Alexa, open Kindle" (This is a really cool feature where it will use the Alexa voice to read your kindle book to you.)
After a feeding (and a diaper change), time to calm the baby
- "Alexa, dim Theo's room to 5 percent" (less light and red light)
- "Alexa, open ocean sounds"
- There are a number of white noise options this is my favorite.

As I finally leave our Nursery, "Alexa, turn off Theo's room", and return to our bedroom, "Alexa, turn on crib light", I set the baby back into his crib. I return to bed and watch Theo settle for awhile. Eventually, satisfied he is down for sleep, "Alexa, turn off baby crib" and the room goes dark and I return to sleep.

Cleanup your DB, find unused tables

2017-02-08T00:00:00+00:00

As systems grow sometimes tables and models come and go. Often things don’t get fully cleaned up or removed from the system. As time goes on it is often harder and harder to know what is safe to remove, as people involved with the code may have moved on. I have used lots of little tricks to clean up dead code. I even maintain Coverband to find code usage in production. One trick Tom Copeland pointed out to me was checking the max created at on a given table to see when it was last used. It was so simple and helpful I ended up wanting to do that across all tables not just individual models.

Rake DB cleanup

I first tried to do this in pure SQL with a subquery, but when I was having trouble making it work, I found a post basically saying don’t do that. It’s point was if you are going to run a subquery over a collection of all your tables us code not SQL. In the end I made a simple Rake task to output the data I wanted. You should be able to drop it into your own project and get a good idea of tables that might not be needed and get to drop the related code as well. In my case it brought about 14 tables to our attention to schedule for cleanup.

namespace :database do
  desc 'List all tables in surge DB and last usage, help find tables to remove'
  task :list_recent => :environment do
    table_data = {}
    skip_large_slow_tables = ['some_table', 'versions']

    sql = "select * from information_schema.tables where table_schema='public' and table_type='BASE TABLE';"
    tables_array = ActiveRecord::Base.connection.execute(sql)
    tables_array.each do |table|
      table_name = table['table_name']
      next if skip_large_slow_tables.include?(table_name)
      puts "running: #{table_name}"
      begin
        sql = "select max(created_at) from #{table_name};"
        max_result = ActiveRecord::Base.connection.execute(sql).first['max']
      rescue
        max_result = 'N/A'
      end
      table_data[table_name] = max_result
    end
    table_data.sort_by{|k,v| v.to_s }.each do |pair|
      table = pair.first
      max = pair.last
      puts "#{table}:#{max}"
    end
  end
end

2016 in Review

2017-01-22T00:00:00+00:00

2016 in review

As 2017 gets going, it is nice to look back a bit. The year has certainly given much to think about.

On the Homefront

We have a baby on the way
- Probably related we now have wills and life insurance in place ;)
We moved our household to 100% renewable electricity (still have natural gas as well)
- reduced our energy usage about 30%
- beat out our energy efficient neighbors: “You’re using 9% less energy than your efficient neighbors.”
We saw Hamilton in NYC, which we had been wanting to do since we first hear the NPR broadcast of the soundtrack. It was as inspiring as I had hoped.
I wrote a little bit:
- 5 dev blog posts
- 3 travel posts: life stages through burritos, Aruba Advice, & Sewer Cover Photos
I worked on some open source:
- Getting back into Coverband
- releasing my first Clojure project
- working on learning React Native and Clojure script

Books

I Read 26 books, over my goal of 18!

I also finished converting my Grandfather’s book from the last known paper copy to an e-book. I will clean this up a bit more and likely try to release it in 2017.

Best books:

Best book: The Quest: Energy, Security, and the Remaking of the Modern World, this was a long read, but I really enjoyed the history of and energy perspective.
Be Prepared: A Practical Handbook for New Dads: by far the most fun new father book I read.
Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future: while everyone has their flaws this was inspiring and makes me want to be more driven with purpose.
Saddest Book, but good to know: We Wish to Inform You That Tomorrow We Will be Killed With Our Families: Stories from Rwanda

Book Quotes

Some quotes I highlighted that caught my eye.

The Architecture of Open Source Applications, Volume II: Structure, Scale, and a Few More Fearless Hacks

When we started this project, we were losing team members too often. In itself, this is bad. However, the lack of accurate up-to-date documentation meant that most of the technical understanding of the release process was documented by folklore and oral histories, which we lost whenever a person left. We needed to turn this situation around, urgently.

Relates to how teams need to bust open silos and make sure multiple people understand systems.

We did this by making sure that we set aside time to fix at least one thing—anything!—after each release. We implemented this by negotiating for a day or two of “do not disturb” time immediately after we shipped a release.

How to get a problem area out of a tar pit…

These improvements snowballed until we found ourselves past the tipping point, and able to make time to work on significant major improvements. At that point, the gains from release optimizations really kicked in.

and how you can tell you have finally overcome that weakness.

One of the most important duties of a project’s lead architect is to defend the architecture against expedient “fixes” that would break it and cause functional problems or severe maintenance headaches down the road. Arguments over this can get quite heated, especially when defending architecture conflicts against something that a developer or user considers a must-have feature. But these arguments are necessary, because the easiest choice is often the wrong one for the longer term.

Team leadership, in architectural integrity is important.

The general rule of thumb is that rather than hard-coding constants, developers use run-time-settable MCA parameters, thereby allowing power users to tweak run-time behavior.

Mirroring some thoughts on permissions.

There is no place for the “not invented here” religion when trying to solve highly complex engineering problems; it only makes good logistical sense to re-use external code whenever possible. Such re-use frees developers to focus on the problems unique to Open MPI; there is no sense re-solving a problem that someone else has solved already.

A good reminder to build onto of the shoulders of those before you.

Differences are good. Developer disagreements are good. Embrace challenges to the status quo; do not get complacent. A plucky grad student saying “Hey, check this out…” can lead to the basis of a whole new feature or a major evolution of the product.

Listen to others, and value the people you work with opinions

The lesson learned from the TAP fiasco was that to have reasonable maintainability, persistent data needs an explicit schema. More generally, it was a lesson about adding complexity to a project: when considering introducing a novel system for solving a problem, make sure the complexity of that solution is well understood and tested and that the benefits are clearly worth the added complexity before committing the project to it.

A reminder that flexible data storage is nice at a feature start, but can add endless complexity to data analysis later on.

The user interface is the most important part of any product. It’s the only part of your program visible to the outside world and if you get it wrong the world will hate you.

This one goes out to Nick S. and is a good reminder that in the end it is about the users. It isn’t done until they are using it.

Be Prepared: A Practical Handbook for New Dads

Don’t let the dog lick your baby’s face for the first few months. The baby’s immune system is still immature, and your dog’s tongue has been to places you’d rather not think about.

Ha amazing, also lots of other great advice on prepping your dog for the baby.

Experts agree that reading to even very young babies is a good idea. They say that hearing words read aloud helps to “map” a baby’s brain to focus on, and eventually recognize, certain sound patterns, the building blocks of language.

We read Harry Potter to the baby belly

You may think this will never happen to you, and it probably won’t, but stick the key in your wallet anyway. The difficulty of pacifying a six-month-old through a car window is only matched by the humiliation of explaining to the 911 operator the nature of your emergency.

On keeping an extra set of keys for when you eventually lock the baby in the car!

Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future

“It’s like anything else where you find out that the last ten percent is where all the integration happens and things don’t play together,”

Agreed, frequently even with a problem solved, rolling out a integrated and useful solution takes so much effort in that last 10%.

In one case someone spotted an error in a software file in the hours right before a launch. SpaceX’s engineers changed the file, checked how it affected the test hardware, and, when no problems were detected, sent the file to the Falcon 9, waiting on the launchpad, all in less than thirty minutes.

As organizations grow speed and quick responses get overlooked… Being able to adapt quickly is incredibly important.

People who await guidance or detailed instructions languish.

Being able to self direct and provide value is incredibly important.

If the rules are such that you can’t make progress, then you have to fight the rules.

Don’t let process be the excuse for a bad decision or bad execution.

We Wish to Inform You That Tomorrow We Will Be Killed with Our Families: Stories From Rwanda

Plenty of more moderate views could be heard, but who listens to moderates in times of revolution?

Rings true in modern times

Are Your Lights On?

EACH SOLUTION IS THE SOURCE OF THE NEXT PROBLEM

So true when you solve any performance bottleneck you are just uncovering the next one.

We never get rid of problems. Problems, solutions, and new problems weave an endless chain. The best we can hope for is that the problems we substitute are less troublesome than the ones we “solve.”

Another way of saying the above.

TRY BLAMING YOURSELF FOR A CHANGE—EVEN FOR A MOMENT.

Too often from the start we look to deflect blame.

Moonwalking with Einstein: The Art and Science of Remembering Everything

You can exercise daily and eat healthily and live a long life, while experiencing a short one. If you spend your life sitting in a cubicle and passing papers, one day is bound to blend unmemorably into the next—and disappear. That’s why it’s important to change routines regularly, and take vacations to exotic locales, and have as many new experiences as possible that can serve to anchor our memories. Creating new memories stretches out psychological time, and lengthens our perception of our lives.

A good life tip in a memory book.

What separates experts from the rest of us is that they tend to engage in a very directed, highly focused routine, which Ericsson has labeled “deliberate practice.”

I think I do need to do more focused and directed practice. My learning is often to scattered.

we’re too busy thinking about what we’re going to say next, instead of paying attention

I am guilty of this far to often.

The Quest: Energy, Security, and the Remaking of the Modern World

To create an entire system required considerable funding. Although not called such at the time, one of the other inventions that could be credited to Edison and his investors was venture capital.

Interesting.

The challenge for developing countries is to increase reliability, ensure that power supplies keep up with economic growth, and avoid shortfalls that constrain growth. It is also to deliver electricity to the 1.6 billion people who have no access at all to electricity but instead burn kerosene or scrounge for wood or collect dung.

This is in part why I do what I do.

That is why, despite what is often said, increased renewable or nuclear power would have very little impact on oil use unless accompanied by very widespread adoption of electric cars that plug into the electric grid.

A good point about

That states can serve as the laboratories of democracy.

Do we really allow states to veer that far from one another?

“There’s great momentum in China,” Shi said. “We used to pursue the American dream. Now everybody is pursuing the Chinese dream.

Another reason why we need to do better as a country.

Public Service Company of Colorado, a subsidiary of Xcel Energy, currently has the largest share of its total electricity coming from wind of any utility in the nation, almost 15 percent.

Woohoo, go Colorado. This program is actually how we got our home to 100% renewable energy, by participating in their wind program.

We need to create a business model in which reducing megawatts is treated the same way from an investment point of view as producing megawatts.

Yes this is a very interesting idea, but capital doesn’t reward reduction at the moment, how would one go about this?

For its first 40 years, the oil industry had been a lighting business. Its main product was kerosene, poured into lamps and used around the world for illumination. John D. Rockefeller became the richest man in the world as an illumination merchant.

Interesting and makes obvious why light is the very first thing one gets into with solar energy in emerging markets.

Get Lucky: How to Put Planned Serendipity to Work for You and Your Business

Attending events and seminars put on by different departments in your organization, joining extracurricular activities where other employees are also participating, even sitting with new people in the cafeteria at lunch are all ways [to increase serendipity]

It is good to remember to spend time with other people in the company than just your group.

Conferences in your area of expertise, collaborative meet-ups, seminars, networking events, or other smaller get-togethers with like-minded individuals are all terrific ways to put yourself in an unfamiliar situation within a familiar environment. … The more you can open up space in your schedule to attend events as frequently as time allows, and to attend as wide a range as possible as opposed to just the same old conference circuit, the more likely you are to have exactly that serendipitous encounter you didn’t even know you needed.

Make time to spend time with folks working on similar problems as yourself. You never know what comes out of it.

Pixar University’s roster of classes includes improvisational theater, drawing, and screenwriting as well as classes on every aspect of computerized filmmaking, and classes are open to all employees at every level.

That is awesome, clearly I should step outside my box more often.

“The real problems are the ‘m & ms’—the managers and the meetings. Managers’ real jobs are to interrupt people … and managers most of all call meetings, and meetings are just toxic; they’re just terrible poisonous things during the day at work.”

Our teams favorite day is no meeting Wednesday.

Even those of us who have lifelong obsessions find it easy to forget them amidst the immediacy of these trifles. But that just means it’s all the more critical to make time and space in your mind for the things that matter to you. … commitment balances the short-term view with long-term aspirations.

More on needing focused, dedicated learning.

Too often, however, when it comes to our work environments we focus overwhelmingly on managing by objectives. We talk of hard work, discipline, and financial incentives.

We need to stay flexible

he learned that experiments done for the sake of curiosity often yield the most practical results.

Try new things!

We need to make something with it, and then put it out into the world with all the courage and determination we can muster.

Do something with your passions.

Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency

The shortcut that is most frequently employed is to assume that individual workers are entirely fungible. … The problem is that human workers are not entirely fungible. Even the most dedicated advocate of matrix management will understand that there are limits to how many ways an individual can be divided.

This is still a problem I see far to often where management doesn’t understand the true cost of task shifting.

Knowledge workers aren’t fungible. Treating them as if they were will increase busyness but make it harder for them to get useful work done.

Try to get a team that can focus on the things they are good at.

We collected information about the environment and recorded each and every task switch. To no one’s surprise, people who were interrupted less often performed better.

How do we avoid this issue with slack and other tools constantly vying for attention.

Fragmented knowledge workers may look busy, but a lot of their busyness is just thrashing, switching continually from one activity to another.

Agreed when large portions of time by a whole team is just trying to keep up with company communications, announcements, and sync with related teams… It significantly handicaps the ability to get real work done.

The poor worker who is there at midnight has probably stayed at his or her desk after regular closing just in the hope of having some peace and quiet for a little real work. What a shame that the organization can’t give such workers a decently interrupt-and noise-free environment during standard hours.

When the best work is being done outside of the office and outside of office hours something is wrong.

This can lead to a positive obsession with process. Process obsession is the problem. … Process obsession is as common in knowledge-worker organizations today as the common cold … and, in my opinion, about as desirable.

When something slips through the cracks rarely is more process the solution.

employee evaluation standards that don’t help you build meaningful longterm relationships

This is frequently not kept in mind when developing review processes.

Real quality has little to do with defects, but our so-called quality programs have everything to do with defects.

This is extremely true, this is an issue with how we structure the whole concept of QA teams.

it would suggest that a real Quality Program should spend one-ninth or less of its resources on defect prevention and removal, and the rest on assuring product uniqueness, usefulness, market impact, change of customer work modes, etc

This is a interesting point, but basically now gets into what product managers actually, meaning that it would just be an issue with terminology for many groups (and perhaps the sizing of various teams).

Defect prevention and removal efforts may add sufficient overhead to the overall process so that it is too slow and unresponsive to market needs. … The strongest argument for new product integrity and usefulness has been effectively silenced. … Cutting out the inconsequential would be, for most of them, the most important act of quality improvement. … The prescription “Make less and choose much more carefully what you make ” is a tough one to swallow. The problem is that the “quality” that quantity has is power.

Some more on what actually makes quality points again to product managers.

You can’t grow if you can’t change at all. This may not sound like a revelation to you, but I frequently come across organizations that frankly admit they can’t change, that they are stuck in stasis, but nonetheless consider growth a normal expectation.

A good reason to always be pushing for change, not more of the same.

I have identified as barriers to change, such as reduced slack, standard process, poured-in-concrete processes and voluminous documentation of everything that will hold still for it.

Another hit against overly process driven

It’s nontrivial for a company and everyone in it to know “who we are.” A little bit easier, however, is to know “who we aren’t.” When even that knowledge is missing—when there is no basis in the company to say about a given cockamamy scheme “it just isn’t us”—the company clearly lacks vision.

Making sure to spend time and effort to figure out a real vision is important.

Leadership is the ability to enroll other people in your agenda. Meaningful acts of leadership usually cause people to accept some short-term pain (extra cost or effort, delayed gratification) in order to increase the long-term benefit.

I liked this quote on leadership

In order to enable change, companies have to learn that keeping managers busy is a blunder. … Making it okay to take risks and succeed in risky endeavors requires that you also make it okay to fail.

change takes time ;)

Staying Organized with Markdown

2016-11-26T00:00:00+00:00

It is good to be able to keep track of the daily work you do and to keep a focus on important larger initiatives. While working as a team having software that supports robust communication and integrations is necessary, I find it overkill for personal daily use, quick tracking, and reflection over time.

I have instead followed a different practice for years, that I have shown a few people who have also found it helpful. I keep it as simple as possible with a few Markdown files, which I check into a private git repo.

Markdown notes work well because it is quick and free form. I still occasionally take notes on paper, but I transfer things into my Markdown notes at the end of each day. I have different sections and templates that I can use for weekly, quarterly, and yearly tasks. Nothing is faster than a few bullet lists of todos for day to day work, which keeps me focused on what I need to get done each day. It also lets me quickly throw in anything that came up suddenly which took my attention, letting me see why any goals for today slipped to tomorrow.

These notes eventually turn into a story and keep my history, when I reflect back on OKRs each quarter, I go through my daily standups to pull out the bigger picture of what I accomplished for the quarter. My history of work I have captured for the last 3 or 4 years this way and can extract patterns and themes, and if I want to accomplish something over the long term build it into the short term goals to see improvements over the years.

I have refined this system for a few years, and it has been the best thing at keeping me on track and focused for a long while now. I hope you can find parts of it helpful as well.

Tools for Markdown Standups

The tools to support this kind of process is super light weight, which is one of the advantages. I can even edit these through the GitHub editor on my phone if I am away from my computer.

Text Editor (MacDown)
Git (I check in files when I make significant changes or at least weekly)

Project Structure

Having a bit of structure with multiple files and sections of your daily file can be helpful. I keep all of these notes in a single folder, notes, which I then have several documents that are active.

notes (directory)
- archive (directory)
- standup.md (the primary file I work on daily)
- scratch.md (a staging area for notes on anything that often gets moved to official documentation, project ReadMes, emails, google docs, code, etc. at a later time)
- okr_q1.md, okr_q2.md, etc. (my quarterly goals and notes reflecting on them at the end of the quarter)
- random.md various documents in progress, individual documents I use when staging something larger to be shared with the team when more complete.

Notes Templates

I keep various levels of notes in my standup file… Here is a blank template that gives the high-level structure, below I will explain a few of the sections in more detail.

# Links

* various links
* sometimes sub link sections

# Goals

### Daily

* daily reminders
* check calendar
* share standup
* etc

### Weekly

* review Asana
* review Jira
* weekly reflection
* etc

### Monthly

* branch cleanup
* AWS usage monitoring
* etc

# Weekly Monday template
see template below, but a template to fill out each Monday

# Current

* current high level
* team goals

# Symbol Key
see template below, a key for notes

# Standup

### Monday, Oct 11th

### Tuesday, Oct 12th

### Wednesday, Oct 13th

### Thursday, Oct 14th

### Friday, Oct 15th

# History
All the previous week's standups get moved here. Then I start a clean standup.md file each year archiving the previous one.

Goals Section

This section lets me keep a short list of daily, weekly, monthly todos/goals. It is a little reminder for overall pace, focus, and schedule. Often things that are recurring end up with a calendar reminder as well, but this is a quick place to check to make sure I am staying on top of things.

Weekly Monday Template

At the start of each week, I share a bit of a week’s summary with the team. It is a great way to make sure people are on the same page and remind of anything out of the ordinary like Holidays in one of the countries where we operate. I fill this out each Monday and share it in our #software-team slack channel.

It is also a good place to call out birthdays, kudos for something well done, link to a recent post-mortem or anything else that might be worth bringing a little extra attention.

### Weekly Monday Template

Notes for Software Team about the coming week
* Dev first-responder: <name>
* PM Release Coordinator: <name>
* Out of office / Holidays / Vacation
   * TZ, US, Russia
   * Dev vacays etc...
* Updates:
   * info about a big release or test planned for the week
   * dates and times for brown bags, etc
* Articles:
   * some good links
   * from recent articles

Current Section

The current section lets me add some higher level goals and keep them just above my weekly standup notes, so I keep them in mind. Current helps make sure my day to day work is building towards some of the larger themes and that daily tasks don’t always push out larger goals.

### Current

* Follow up on Allware teams gathering
* Reschedule web and Android quarterly walkthroughs
* Update system architecture diagrams
* Improve ETL pipeline process

Symbol Key

I like to be able to check things off my list so to speak, these symbols I try to apply to my daily standup todos so I can see the status. Often if something doesn’t get a checkmark, it gets copied into the next day, so I can continue to make progress against it.

### Symbol Key

✓ : done  
☯ : progress  
♺ : continuation from previous day  
⚠ : notice, missed, didn't get to it, problems

Daily Standup Section

Daily Standup, is the workhorse of the system. The section where I put in what I plan to do for a day, but also adjust it to what I am doing during the day. At the start of the week, I write down a few of the things I know for the full week. As the week progresses, there is more and more detail and often my days get fully booked up with work scheduled.

We share our text standups in a Slack channel, so I try to share mine at the end of most days or at the end of the week. I used to share at start of day, but to often unexpected items end up changing my plans.

# Standup

### Monday, Oct 11th

* meetings: Stakeholders, PM Weekly, Squad IPM, Squad Retro, 1:1/Name
* post weekly software template
* tickets
   * links to Asana tickets
   * links to Jira tickets
* permissions fix (this is likely something that came up mid-day)
* resolve Solr issues (this is likely something that came up mid-day)
* research X
* documentation for Y (link to doc)  

### Tuesday, Oct 12th

* meetings: Call Center, Another Squad IPM, New Country Launch, Data Team Standup
* tickets
   * links to Asana tickets
   * links to Jira tickets
* recruiting followups
* schedule interviews
* review large PR (this is likely something that came up mid-day)

### Wednesday, Oct 13th

* etc
* ...

Final Thoughts

The markdown system is fast and flexible. Whenever I need a new section, to quickly organize thoughts, I just create a header and start writing, when the section is no longer useful in the day to day I pull it out.

Sometimes you have a full day planned out, and something blows up, and the entire standup gets updated, that is OK it is quick and easy to roll today’s tasks into tomorrow section. The ability to very quickly see an overview and modify it is really where this wins. I can’t move 6 tickets in Asana and Jira from one day to another without about 20 clicks, but I can do that in seconds in my Markdown file.

Obviously, this will need to be adjusted to your needs. It has worked very well for me with few changes over the last few years. Previously, I even kept the whole file in an organization public git repo so anyone at the company could see what I was up to. The scratch file became useful to several folks, and I got PRs updating sections, which had become little references for the team. I believe the git tips / cool Git one-liners sections was one that had some mileage. Also, the list of links I had built up to all sorts of resources became useful to others over time. Currently, I keep my notes in a private repo. Private notes due to me doing more management, and my notes often include thoughts and feelings of the team which I need to keep in mind, but also keep privately. A private repo lets me keep my system as useful as possible, keeping it useful to me without worrying about sharing thoughts given in confidence.

If you have any suggestions or other organizational tips, feel free to share. I hope this will be helpful to a few other folks as it has been to me over the years.

Managing DB Schema & Data Changes

2016-11-21T00:00:00+00:00

As you work with a growing team and codebase, occasionally database changes will cause some surprises and issues during deploys. This seems to be especially true in the Rails world, but isn’t exclusively a Rails app problem. Rails has a simple way to handle schema changes, which works well for small additive changes, but struggles on larger changes to an existing DB.

As I have worked with a few teams, it is worth calling out some issues and learnings about how to help avoid some issues.

deleting tables
changing columns
slow and missing indexes
long running migrations
data migration vs schema changes
communication issues between data and software teams

Team Policies

At Offgrid Electric, we have shared github wikis with documentation, process, and policies. Over time the team has built up processes related to making database changes to our application. These policies help us avoid some off the pitfalls, while encouraging good communication. We adjust them over time, but it might be helpful for other organizations as they begin to run into any issues around database change process. Below is a partial scrubbed version of our documentation.

Schema Changes Process

Changes to the data model (adding/removing tables/columns) should involve active communication with the data team to ensure:

New structures (tables/columns) include all of the information we might need for future analysis in a format that is conducive to performing that analysis
Structural changes/removals (tables/columns) do not break existing data pipeline processes or reports

The following processes should help us ensure these points.

Process for Adding New Tables / Columns

If a new process is being modeled in the database the Business Intelligence (BI) and Data Infrastructure (DI) teams should be actively involved in the data modeling process to ensure all of the information needed for analysis is being collected.
Migration PR’s should have a :thumbsup: from someone on the data-infrastructure team prior to merging.
The DB migration to should be merged and run on staging prior to production. If there is any possible impact to dataware house it should remain on staging for a day (so the nightly data tasks run against staging) prior to being merged and deployed to production.

Process for Modification or Removal of Tables / Columns

A ticket should be created in the Data Warehouse Jira project backlog. The ticket should list the proposed modifications and ask that someone confirm there are no ETL or BI processes that utilize the column/table that is being changed. Completion of this ticket should be a prerequisite for a PR.
Migration PR’s should have a :thumbsup: from someone on the data-infrastructure team prior to merging.

Database General Guidelines

Avoid adding columns with a json data type. Our data warehouse uses AWS Redshift, which does not have robust support for JSON column types. Which means that any data that starts off as json in our DB becomes, incredibly hard to query. Since it’s hard to know what we will want to query in the data warehouse, it’s safest to assume “all of it.” If that’s the case, then we should strongly avoid json column types in Surge.

Adding large indexes

Creating an index on a large table can be slow, if you follow the process recommended here, it won’t lock the tables during the indexing process.

avoid locking
if you indexing will take more than 10 minutes do it outside of a deploy, following the Modifying Large Tables process.

postgres concurrent indexing in Rails

Modifying Large Tables

Why we need a process to modify large tables. This isn’t handled well by automatically running Rails migrations on deploy.

Migration happen on deploy, any migration longer than 10 min will time out
If we take down old workers during the deploy we will have an outage while the migration finishes (this used to occur on old deploy process, but no longer should be a problem)
Large DB changes can be a bit risky and effect performance it is good to schedule them.

Solution:

To avoid this we commit the migration but run it manually from a console.

Make a PR with the migration only, no code changes
Commit the migration with a leading underscore _ to it won’t run automatically
- for example db/migrate/_20160817125700_add_transaction_date.rb vs db/migrate/20160817125700_add_transaction_date.rb
Have data team review and approve migration as recommended for all schema changes as mentioned above
After approved, schedule a time to run migration during low site traffic volume (after 10pm EAT)
- after data team approval and schedule is set make sure to do the below process on staging at least a day before the scheduled production run
merge the PR to master
deploy to master, MIGRATION SHOULD NOT RUN ON DEPLOY
connect to production console, and manually execute the migration
- rename migration and remove the _
- in this example: mv db/migrate/_20160817125700_add_transaction_date.rb db/migrate/20160817125700_add_transaction_date.rb
- execute the migration: bundle exec rake db:migrate

Alternate Complex Solution:

Rarely there is a reason to bypass migrations entirely and make a change manually against the DB. In these special cases which normally means a DBA is going to perform some magic to migrate something without locking, the process is slightly different.

Basically the same as above BUT:

migration file should never run in production, we still want it for dev, testing, and likely staging though. In this case skip the _ naming part of the process mentioned above.

class AddTransactionDate < ActiveRecord::Migration
  def up
    unless Rails.env.production?
      add_column :account_transactions, :transaction_date, :datetime
      execute "UPDATE account_transactions SET transaction_date = created_at"
      ...
    end
  end

In all non production environments, deploy and let the migration run
In production have the DBA do their magic and then insert the migration into the schema table the migration should never even execute on production
- ActiveRecord::Base.connection.execute("INSERT INTO schema_migrations (version) VALUES('20130817125700')")
At this point merge to master and deploy. The migration should be skipped, if anything went wrong the unless Rails.env.production? should also protect against the migration accidentally running.

Data Migrations vs. Schema Migrations

Data migrations often can take far over 10 minutes, if you are iterating through and modifying a large table we have had some that take hours. This isn’t a good first for the standard Rails migration or standard deployment processes. So we looked for some other options

We previously ran release tasks, kind of following the thoughtbot process. These are basically managed rake tasks, but occasionally folks weren’t sure what had been run by whom and when. While this worked OK, we thought we could do better.

Our team ended up creating rails-data-migrations which is a lightweight wrapper around standard Rails migrations to create data migrations. These use the same mechanism to store the timestamp into the schema_migrations as a standard migration, but they don’t run on default db:migrate. The data migrations are included in their own db/data_migrations and include a generator to create new migrations (rails generate data_migration migration_name).

data migrations don’t run by default on deploy
we can kick them off in console
they will only run once and it is easy to check if anyone has run them
you can run all missing data migrations against a environment with a single command.

Impacting Work

2016-10-16T00:00:00+00:00

A job that will inspire you daily

The work we do at Off Grid Electric, has a clear impact. The mission is inspiring and the immediate benefits we deliver are obvious and extremely rewarding.

Our team is amazing, and driven to rise to a challenge. We face unique challenges operating without physical addresses, 60% network coverage, and customer support for people without internet. We take distributed work both challenges and benefits seriously, building a team and process to enjoy it’s freedoms.

installing our system on a roof in a remote village of Karatu, Tanzania

We are hiring

Our Software team is hiring the positions below. There are other non software positions as well

Sr. Data Engineer, big data, ETL, and data infrastructure.
Full Stack Ruby, APIs, integrations, and IOT.
Sr. Android Engineer, majority of our workforce is on motorcycles with an Android.
QA Engineer, test automation, team management, web/mobile testing.

Don’t take my word for it

“Across sub-Saharan Africa, fully 590 million people lack access to power. And it’s a life-or-death issue: Indoor air pollution from wood stoves now kills 3.5 million people per year, more than AIDS and malaria combined.” –Brad Plumer: Washington Post

Over the years, I have learned to respect Bill Gates for his passionate endeavors to change the world. His recent post highlighted the importance of the work we do at Offgrid. We deliver directly on #1 and provide free power to schools helping enable #4.

There are four objectives I think we should prioritize:

Provide everyone on earth with affordable energy without contributing to climate change.

Develop a vaccine for HIV and a cure for neurodegenerative diseases.

Protect the world from future health epidemics, which might be more infectious than Ebola and more deadly than Zika.

Give every student and teacher new tools so all students get a world-class education.

–Accelerating Innovation with Leadership

Another, current inescapable name in the technology world these days is Elon Musk, who is know for his passion for Solar energy.

“this handy fusion reactor in the sky, where you don’t have to do anything—it works, it shows up every day, and it produces ridiculous amounts of power.” –the deal with Solar

“This is because the overarching purpose of Tesla Motors (and the reason I am funding the company) is to help expedite the move from a mine-and-burn hydrocarbon economy towards a solar electric economy, which I believe to be the primary, but not exclusive, sustainable solution.” –Original Tesla Master Plan (recently doubled down in Master Plan, Part Duex

Musk clearly believes in it because he is an investor in Offgrid Electric, through his involvement with SolarCity. He recently talked of the kind of impact we are working to deliver.

“This allows you to go completely off grid. It’s analogous to the way that mobile leapfrogged landlines.” –Tesla Energy Will Help World’s Poor Leapfrog the Grid

Offrid Electric in Tanzania

Our global headquarters is in Arusha Tanzania. While we are launching in other countries, Tanzania is by far the most developed part of our business. We recently rebranded Offgrid Electric’s offerings from M-Power to Zola in Tanzania.

Office in Arusha, TZ, field visit with Sales team, hanging out employee cottage

We have partnerships and sell at gas stations through out Tanzania.

Banner ad outside gas station, Demo unit inside Total gas station

We sell most frequently to areas with limited or no access to the power grid. Often these are more modern villages just to remote for access, but we also sell to Maasai villages, which can spread over large distances into the mountains.

Panoramic view from hilltop down to base of Maasai village, we have a number of customers in the village

Feature Branch Staging Servers

2016-07-31T00:00:00+00:00

Continuous Deployment for Feature Branches

The ideal web software deployment process lets you quickly deploy features as they are ready and tested. I previously wrote about continuous-deployment previously, and most of those thoughts remain. I do think that continuous deployment can have some other issues beyond automated testing that make it a challenge, in this post I want to focus more on the human feedback loop opposed to the automated one.

Human QA
Stakeholder feedback
Long running branches (large groups of related features, 3rd party integrations, etc)

We had each of the above cases, at Offgrid-Electric. All the cases point towards a staging environment. In normal feature development flow is a time where you want share your work, which should be deployed and available to some set of users. Let’s explore how a single staging doesn’t solve everything.

Staging Servers

The first thing people often do for QA and stakeholders is setup a staging server. When a feature is complete, you can merge it to staging and deploy for feedback. While this flow works fine for a small number of features and developers, it runs into issues as you scale the team.

Staging has a mix of features that are in various points of QA or getting feedback from stakeholders.
The code changes start to pile on top of each other, and can’t be cleanly applied to master
Entangled features and iterations of fixes now can’t be released until all that has been merged into the staging branch are stakeholder & QA approved.
Long running branches that won’t be ready for weeks, can’t be merged in at all, where do you get feedback for those?
When QA is working on testing one feature they are often running into bugs for another feature complicating testing and debugging.
Training? We have a lot of internal tools that can’t be released until users are informed and trained on the new system. The feedback loop on training, can block features for weeks.

At this point sometimes folks setup multiple staging servers or use tools like xip.io/ to share developers boxes. These solutions are bandaids for and only solve small pieces of the problem. In the end, you want to be able to take any branch and dynamically create a deployed environment and keep it around as needed. This gets beyond the simple continuous deployment of dev branch to the staging server and master to production. How do you scale continuous deployment?

Feature Branch Staging

With the cloud and related tools, it isn’t much harder to go from continuous deployment of staging and production to dynamic branch based staging servers. We do this with Docker and AWS tools, but there are many ways to automate configuring an entire environment and deployment. I will cover some of the challenges a bit later, as there are a number of “gotchas”.

What does it look like?

A developer starts a feature branch from master. They work on the feature say feature/new_cool_page, and check it into git. The team CI (Continuous Integration) server is already running the tests on each commit. At some point, the developers are ready to share and get external feedback from stakeholders, PM team, and QA. The developers can run some command create_branch_staging. Which will stand up a new server deployment using their branch and return a URL based on the branch name new_cool_page.staging-domian.com. From then on future git pushes can deploy to the feature branch if CI finds a matching staging environment, or you can continue to deploy to the feature staging server manually.

Once you have a feature on a viewable staging branch, there can be a lot of feedback and discussion among the team. Often PRs with comments from devs will link to issues on the feature staging server. When everything is finally ready to go, you merge the branch to master and deploy to production. Destroy the feature staging environment destroy_branch_staging and celebrate another victory!

We still manage a “shared” staging server as well, this is for any features that might need to sit and “bake” with additional usage for awhile (think major library updates or infrastructure changes), alternatively in that case the feature branch is just collapsed to our old dev branch and deployed to staging.

The process enabled by being able to create multiple environments dynamically solves most of the issues with a shared staging server. It significantly increased our velocity. Increased the confidence of our smaller deploys, and made it simpler to prototype and throw away ideas. Beyond the developer process. It additionally has other nice benefits.

Forcing function to fully automate your infrastructure (encouraging DevOps), basically immutable infrastructure.
Easy to test infrastructure changes independantly from code changes (ha-proxy, Nginx config, different memory CPU and instances, etc.)
Depending on how you solve some of the data replication challenges (mentioned shortly), it is built in testing for backup and restore
Discover bottlenecks in your infrastructure earlier before production
Helps force environment simplification and understanding.

Challenges

While this sounds great, it does bring up some additional challenges from having a more traditional CI deployments to staging and production environments. It isn’t quite as simple as the prior case of deploying to single environment that was often configured by hand and had it’s data smoothed over the ages. Think about all the dependencies that go into running your app and how they relate to one another, such as how you seed your staging environment data. Here are some of the challenges we faced.

Seed Data: our staging uses an old snapshot of production with additional QA data layered on top
- our solution: daily snapshots of staging. Each feature branch starts with the most recent RDS (AWS Postgres) snapshot to dynamically create a new DB. Then any migrations since the snapshot are applied on top. (We can easily pull our production snapshots as well if we need to debug production data issues)
Multiple Data Sources: Our app requires Postgres, Redis, and Solr and each needs data seeded per environment
- our solution: each data store has its individual requirements.
  - Postgres: RDS snapshots as mentioned above
  - Solr: we trigger a snapshot of staging Solr, and script creating a new Solr namespace from the snapshot for each server.
  - Redis: we treat Redis as a rebuildable store, it can start empty for any given instance. We spin up a new Redis for each server, but you could leverage namespaces for this.
Multiple Services: Where do you point to your other services? By make sure it is configurable
- our solution: By default, all feature-stagings point to our other staging services. Where they point is controlled by environment variables (12-factor app) for any staging deployment you can alter the ENV to point to a specifically deployed alternative.
Additional Cost: You still want to be careful about the number of staging, but also how you decide to share resources.
- our solution: We approach this a few ways
  - shared resources: you can deploy feature-staging-lite (webapp only deploy (no data sources, crons, or background workers). Instead it uses shared-staging data sources. This is ideal for UI only changes and requires far fewer resources.
  - shared servers: Beyond just sharing actual data sources. We sometimes spin up full new servers like RDS, but for Solr, we just utilize namespaces. Allowing a single Solr server to host ALL the feature branches search indexes.
  - reduced hardware: While we run shared-staging very close to production. We run far fewer web-workers, with less CPU and memory for most of our dev stacks. We can adjust if needed. Along with Docker we run multiple deployments on the same servers.
  - staging sweeper: Cleaning up after staging environments, you don’t want an infinitely growing list of servers and databases sitting around. We monitor staging servers and they self-destroy after 3 days of inactivity.
Additional Monitoring: you will need to monitor some of your staging tools. For example, it’s important that things like staging sweeper keep running.
- our solution: Most of our monitoring and metrics are build with server environment names in mind.
Exception Overload: Managing multiple exception streams, can be confusing.
- our solution: We keep exceptions in very clear production, shared-staging, feature-staging streams at the very least, often including BRANCH_NAME for any metrics / alerts
Scaling Staging: At first people might be cautious using feature staging, but it becomes so useful next thing you realize you could be maxing out your staging infrastructure.
- our solution: We had to build in auto-scaling up and down based on the demand for the number of branch-staging servers. If someone deploys and there aren’t enough servers in the cluster, it adds more. It will shrink back down to size as staging servers are destroyed.
Custom Tooling: When you have a single staging server, folks just connect to it and do what they need. Now it can be hard to find where your service is running and what is going on.
- our solution: where you can avoid tooling, using things like log aggregation. You don’t need to connect to staging to view logs. If you need a console we have a script docker:console which will create a docker task and connect it to your environment and SSH you over. Also, keep it simple we rely on a lot of battle tested AWS tooling.

Our Approach

We worked with the very talented Trek10, to develop our production and staging infrastructure as well as our deployment process. We leveraged a few technologies, some open source, some standard AWS tooling, and some custom scripts or services we or Trek10 developed.

The basic tech stack:

CloudFormation: This lets us manage all the related resources and services together as one.
- ECS: Easy way to have multiple services and tasks in our environment.
  - Docker: This makes it easy to package up our app and dependencies… Easy to CI as well as scale out as multiple services and tasks.
- RDS: We can create copies from snapshots of any given database for other branches
- Elastic-Cache: Similar to RDS the tools around using Redis from elastic cache makes it super easy to manage
- AWS-lambda: If we need to do anything fancy (say scale our production DB size to something more cost effective for staging). We can script that in AWS Lambda and CloudFormation will trigger our events and wait for their results. Anything CloudFormation can’t handle out of the box, we can handle this way.
- Route-53: You will need easy to integrate dynamic DNS to go this route. We are always adding and tearing down DNS names.
- ELB: Less required for feature staging but a piece we use for production high availability.
- Ruby: We have some simple Ruby glue scripts we kick off to interact with all of this stuff ;)

As for the detailed specifics, we will save that for another post. Hopefully, something our partners Trek10 will cover in much more detail soon.

The Perfect Staging

In a perfect world, you would never have to think about this stuff and environments would just appear when and how you need them. With no concern for the cost and complexity of getting there. That isn’t the case but depending on your project and how early you start to automate away manual tasks, it is easier than ever to get pretty close to the ideal. You need to look at your project to see if it makes sense, how often are you having a real issue with features getting in each other’s way, does your organization have significant stakeholder feedback or Human QA against features? If you haven’t shipped anything and aren’t worried about zero downtime deployment, again not worth your time. I think the cost of getting this kind of staging and development process was too high a few years ago, but it can be built out for many projects in a cost effective manner with enormous benefits. Is it the holy grail, no, but it is something I would have a hard time living without anymore.

Research, Proof of Concept, Pilot template

2016-01-17T00:00:00+00:00

Learning Project Template

When we start a small project to learn and answer questions for the organization. It is good if we can be clear about the type of project and the specific learning goals of the project. This template aims to help create a path towards focused learning.

Project Name

Start with a small paragraph describing the project. List the specific limitations, simplifications, and assumptions in the project. What is roll out / testers / pilot usage if planned, or approach to research if no initial rollout planned. State explicitly if this is a research, proof of concept, or pilot project.

timeline for research / POC / Pilot
- timebox: before check-in / extension
involved contributors

Research: This often is documenting, searching, having discussions, proposing plans and alternative approaches. Research may include small code samples that explore or verify if something works as expected or is a viable option. Most often is used to drive the discussion further.

POC (Proof of concept): This would be a step deeper than pure research. It tries to proof out an idea in a way that answers goals of the project under more real-world conditions. POCs aren’t necessarily for end users, often just for the developers. POCs, unlike pure research should be trying to work with real world business data/conditions opposed to lab conditions of research only.

Pilot: Pilots are often intended for release to a small group of initial users. While pilots are tests of the underlying code and architecture, it is also often testing the business value, UX, performance and usability in real-world conditions. A pilot can be ongoing for a time to allow close contact with a small group of users to iterate towards a solution.

Current Status Quo

We are probably doing this project because we have no solution, or the existing solution has known flaws. Let’s give a brief statement on the current state, which could be as simple as “does not exist.”

Goals

A list of defined things we are trying to learn from the project. Below are just some examples that might be relevant to different levels of learning projects.

Research
- What questions are we trying to answer?
- What decisions do we want to be able to discuss or make as a result of this research?
- Can we identify and reduce the scope of unknowns with this research?
- Is it faster, cheaper, better to do any of the proposed solutions?
POC
- What are we proving out? (or what existing challenge does this solve)
- What alternatives are we skipping to focus on this POC?
- What does this replace if there is an existing solution?
- Level of effort:
  - Did the POC level of effort match expectations?
  - Cost more / less?
  - expose flaws in our assumptions?
- Should we continues to pursue this approach?
- We halted the POC early due to learning X, causing us to abandon the approach. Often this result would still answer many of the questions above.
Pilot
- We are working to prove this works for our users?
- The new way is better by X% than the existing way.
- Does this work in a real world scenario?
- Are there flaws in our assumptions before we fully build this out for all users?
- Can we support this at scale?

Configuring a new OS X Yosemite development machine

2015-12-10T00:00:00+00:00

This is a update from how I setup OS X Mavericks years ago. Basically, it is more for my own reference, but that post seemed to be helpful to some others.

These are the steps I took to setup a new OS X 10.10.3 (Yosemite) machine for development. I have heard on twitter that each time you setup a new machine you get 1% better and at this rate you can become good at it in a few hundred years! Ha while that seems to be true I am getting happier with the process and I am now at least trying to document it. This post won’t serve as a rich tutorial on the configuration process, but should give some ideas on good ways to setup your own machine or ideas on how to fix some common gotchas I ran into with my setup.

I know this is a very opinionated approach without much explanation, so if you are new to development I recommend finding richer tutorial as a this is largely intended for my own documentation.

new instructions

same xcode-select –install
install brew manually from http://brew.sh/
brew install gpg
keys slightly different gpg -o key.tar.gz --decrypt latest_keys.tar.gz.gpg
brew install zsh
same verify zsh
got your keys working clone your dotfiles so your terminal is better: git clone git@github.com:danmayer/dotfiles.git
install rbenv git clone https://github.com/sstephenson/rbenv.git ~/.rbenv
install postgres http://www.postgresql.org/download/macosx/ or http://postgresapp.com/

Initial setup steps

verify full disk encryption
update password on account / machine (if one was previously configured)
install all OS X updates
install X code (this and the updates can be slow get this started as soon as you can)
- cmd line xcode-select --install

While Xcode installs

install chrome
setup chome sync so it will pull your plugins like lastpass
while X code installs install size up, or your prefered window manager
install dropbox, get sync started
get your old ssh keys (or you can make new ones and upload them to services like github)
- tar czf key.tar.gz .ssh/
- gpg -c key.tar.gz
- scp it over from other machines or encypted file sync via Dropbox
- gpg -o key.tar.gz --decrypt latest_keys.tar.gz.gpg
- tar xvzf key.tar.gz
download and install virtualbox
download and install aquamacs (or your favorite editor)
download and install flux for light filtering

After Xcode, install brew

fix various SSL errors like Faraday::SSLError: SSL_connect returned=1 errno=0 state=unknown state: certificate verify failed
- you need to run brew install openssl
- and brew link openssl --force
- re-install your rubies! Don’t forget to rbenv rehash
install iterm2
install zsh brew install zsh
- change default shell to zsh chsh -s /bin/zsh
- verify your running zsh after opening new terminal ps -o comm $$
install Pygments sudo easy_install Pygments #my dotfiles use Pygments
git clone dotfiles, or however you sync your environment
- sync your non git dot files encypted via scp or pgp dropbox sync (I keep .env_extras that contains some sensative keys so it is not stored on github)
configure VPN (if needed)
install Mou (markdown editor)
install campfire client (Propane)
install IM client, Adium
- configure Adium for gchat
brew install qt
- at the time required HEAD brew update; brew install qt --HEAD
gem install thrift -v '0.9.0' -- --with-cppflags='-D_FORTIFY_SOURCE=0'

brew install Mysql55

initialize mysql:

mysql_install_db --verbose --user=`whoami` --basedir="/opt/boxen/homebrew/Cellar/mysql55/5.5.30/" --datadir=/opt/boxen/homebrew/var/mysql55 --tmpdir=/tmp

gem install mysql2 -v '0.2.18' -- --with-mysql-dir=/opt/boxen/homebrew/opt/mysql55/
brew install imagemagick

Get your code

begin to git clone all the things you love
start to rbenv install all the Ruby versions you need
bundle install all the things
verify tests are passing or apps start

Updates Since installed

rbenv to get latest ruby 1.9.3-p484 thanks @obfuscurity

cd /opt/boxen/rbenv/plugins/ruby-build

  git pull; cd
  rbenv install 1.9.3-p484
  rbenv global 1.9.3-p484
  gem install bundler

configure default global node.js nodenv global v0.10.21

Possible Improvements

Should I install emacs starter kit or start over? I need to fix my emacs setup
Automated way to reset all the OS X default apps to open file types?
Automate some of the app downloads and installs
Automate some of the common ruby version installs

Developer Team Onboarding Playbook

2015-08-25T00:00:00+00:00

photo credit: opensourceway

Developer Team Onboarding Playbook

I’m working on building out a plan to improve our engineering onboarding at Offgrid Electric. As we start to add engineers to the team at a faster pace, the cost of not getting onboarding right increases. I’m sure the playbook is slightly different from team to team, and I think it is important to note that I am developing this playbook with a distributed team in mind.

I reached out to some mentors, past & current teammates, & twitter to develop a plan. I would love further suggestions, if you feel I am missing something or breaking Rule One of Management: First, Do No Harm.

This is my current draft on building out an onboarding playbook:

Have a home for onboarding plans, checklists, & materials
- different pieces in git, wiki, or project management software
- execute / improve against playbook with each hire
- plans are available to everyone including the new hires to help improve
Checklists
- accounts to be setup (and who is responsible for setting them up)
- email lists, groups, permissions to be granted (and who is responsible for setting them up)
- things that need to happen in the first day, first week, first month
Send welcome / Team Introduction email
Assign a team buddy
- helps answer questions on company, process, code, & culture
- partner for first staging deployment
- partner for first production deployment
- many teams make use of a buddy, the concept is covered well in, Make New Teammates Feel Welcome from the Start, of Zapier’s guide to remote work.
Day one plan
- 1-1 with hiring manager, verify basic accounts setup together
- get on team communication tools (Slack, Asana, etc)
- git commit to documentation
- time to install dev tools
- 1-1 with buddy
Initial getting started on the project
- Follow project setup instructions, sending PRs for any missing, dated, or confusing information along the way
- tests passing
- development server success
- mobile app builds (mobile devs)
- mobile app can connect to dev server (mobile devs)
Initial smaller quick win tickets for the first week(s)
Intro Getting Started Pairing Schedule
- a small number of schedule pairing events in the first few weeks
- scheduled pairing time with buddy
- scheduled pairing time with team mates, help with knowledge sharing and getting to know folks
Schedule initial check-ins
- with team lead 1 week, 2 weeks, 1 month, 3 months
- with buddy
- with business partners (if sensible)
- with other key players (QA, etc)
Scheduled items should be added to new dev’s calendar

Thanks @mariagutierrez, @glv, @tempo, @sonia_pdx, & @davetron5000 for ideas and inspiration.

If you have ideas, please share in the comments or hit me up on twitter @danmayer

Storing and deploying graphite configs

2015-08-23T00:00:00+00:00

Storing & deploying graphite configs

When I worked @livingsocialeng a engineer on the team Josh Sharpe improved how we dealt with graphite dashboards. He wrote Rake tasks to push and pull our graphite dashboards into each project’s git repo. This made it easier to collaborate on dashboards and see the history of a dashboard. It is really handy to keep your dashboard under version control. If you don’t know why you want graphite dashboards or why it would be good to keep them in version control, checkout a practical guid to graphite monitoring.

At the time, I thought awesome, used the tasks and went back to work. As I have moved on to a new job, and am working on graphite dashboards. I realized I really missed the tasks. I checked around and didn’t find a gem to handle storing graphite dashboards with a project. So I looked into how to build the tasks myself, there likely isn’t a gem because it is really pretty simple.

Download existing dashboards

The format is a bit verbose, so it is best to download an existing dashboard to edit. I added a Rake task, which just pulls a dashboard so it can be edited as a template to build a new dashboard.

task :pull_dashboard_skeleton do
  curl -u user:pass https://graphite.host.com/dashboard/load/DashName.json > temp.json
end

Curling an existing graph will pull down JSON that looks something like this.

{
  "state": {
    "name": "My Dash",
    "defaultGraphParams": {
        "width": 1024,
        "from": "-24hours",
        "until": "now",
        "height": 450
    },
    "graphSize": {
        "width": 1024,
        "height": 400
    },
    "refreshConfig": {
        "interval": 60000,
        "enabled": false
    },
    "timeConfig": {
        "relativeStartUnits": "hours",
        "relativeStartQuantity": "2",
        "relativeUntilQuantity": "",
        "startTime": "9:00 AM",
        "endTime": "5:00 PM",
        "type": "relative",
        "relativeUntilUnits": "now"
    },
  "graphs": [
        [
            [
            ],
            {
              "target": [
                "alias(drawAsInfinite(stats_counts.namespace.deploys), \"deploys\")",
                "alias(summarize(sumSeries(stats_counts.namespace.env.key.*.*),\"1min\"), \"current something\")",
                "alias(timeShift(summarize(sumSeries(stats_counts.namespace.env.key.*.*),\"1min\"),\"1d\"), \"previous 24h something\")"
              ],
                "hideLegend": "false",
                "title": "\"something 24 hour Comparison\"",
                "height": "420",
                "width": "1024",
                "_salt": "1433982521.214"
            },
            ""
        ],
        [
            [
            ],
            {
              "target": [
                "alias(sumSeries(stats_counts.namespace.env.other.*.*), \"total other\")"
              ],
                "hideLegend": "false",
                "title": "\"other graph\"",
                "height": "420",
                "width": "1024",
                "_salt": "1433982521.214"
            },
            ""
        ]
    ]
  }
}

Uploading a static dashboard

For a static dashboard, you can make a really simple Rake task. I ended up putting it in lib/tasks/graphite.rake. In the task point at where in your project you want to store the dashboard file, in this example config/graphite_dashboards/DashboardName.json. Then the task can just use curl to post that data up to your graphite server.

task :update_graphite_dash do
  data = File.read('config/graphite_dashboards/DashboardName.json')
  data = CGI.escape(JSON.parse(data)['state'].to_json)
  state = "state=#{data}"
  `curl -d '#{state}' -u user:pass -X POST https://graphite.host.com/dashboard/save/DashboardName`
end

This lets you keep any dashboards the team build in git. Allowing you to edit the JSON in your own editor which is more nicer than using the Graphite online editor. Once you have this it is easy to extend the concept even further to use data to generate dashboards.

Generating dynamic dashboards

While, previously I had only pushed and pulled dashboards. I ran into cases where I really needed the same dashboard, but slightly scoped by a particular set of data. I realized now that I had the dashboard posting code inside Ruby, it would be easy to dynamically generate dashboards based on data. Again, adding another Rake task, I was able to iterate through our regions and generate region specific dashboards. The trick here, is to use ERB to allow you to embed ruby into the dashboard template. A simple trick, but very effective at letting me zoom in on any dashboard to a specific scope.

task update_dynamic_dashboard: :environment do
  Regions.each do |region|
    data = ERB.new(File.read('config/graphite_dashboards/DynamicRegionDash.json')).result(binding)
    data = CGI.escape(JSON.parse(data)['state'].to_json)
    state = "state=#{data}"
    `curl -d '#{state}' -u user:pass -X POST http://graphite.host.com/dashboard/save/Dynamic-#{region.name}-Dashboard`
  end
end

The actual JSON for the dashboard is pretty much the same, but with the region embed a few places like so.

 {
   "state": {
      "name": "Dynamic-<%= region.name %>-Dashboard",
      ...
       "alias(summarize(sumSeries(stats_counts.namespace.env.key.<%= region.name %>.*),\"1min\"), \"current something by region\")",
       ...

That is all there is to it. Hopefully, this will be helpful to someone else to get their graphite dashboards under version control. It didn’t take long to google around to figure it out, but this should save time from finding the exact formats and urls needed to make things work.

Working at Off Grid Electric

2015-08-10T00:00:00+00:00

Some folks have asked about how I ended up at a job like Off Grid Electric, so I thought I would share a bit of that. I recently took a sabbatical to travel for about 5 months with my wife. While I was traveling, I was thinking about my career, the future, and what is important to me. I felt like I hadn’t been pushing myself enough lately, and I had lost some of the passion I always liked to bring into my work.

TL;DR

“Across sub-Saharan Africa, fully 590 million people lack access to power. And it’s a life-or-death issue: Indoor air pollution from wood stoves now kills 3.5 million people per year, more than AIDS and malaria combined.” –Brad Plumer: Washington Post

Stories illustrating the importance of our mission, and meeting our customers in Tanzania, helped restore a passion to my work.

Travel Shifts Perspectives

There is a great post by how stuff works called can travel change you? I would say it can change you if your open to it. I was thinking about my position in the world and looking to change things up. So I started getting interested in international positions, that could improve some of the major challenges in the world.

Seeing many different countries in my world definitely changed my thoughts on what were major challenges in the world:

clean water
sufficient sewage systems
access to power
clean energy
functional mail systems
trusted government
free press
access to education
income inequality

Climbing onto a roof to install solar panels

For awhile, I had been a bit stuck in an echo chamber of dev tools, consumer startups, and incremental improvements on social communications. While there is cool working going on there, I was a bit burnt out on it. Looking from outside my normal communities, I realized all the issues I found most important in the world to be outside of my current focuses. I wanted to help solve big problems, I knew I was passionate about. I wanted work on something outside the frequent tech joke that resonates a bit too loudly on occasion.

“The best minds of my generation are thinking about how to make people click ads. That sucks.” –Former Facebook dev & Cloudera co-founder Jeff Hammerbacher

Although, there are some good counterpoints showing the legacy of amazing tools the industry is building. Hilariously, I saw another version of this just today.

“Meanwhile the “best minds” of our generation work on drone kale smoothie delivery.” –@thomasfuchs

Investing Your Time

There are great ways to support problems you care about, but in the end support often comes down to your time or your money. While I don’t have lots of money to invest in these important issues, like Bill Gates dedicating to invest $1 billion clean energy, I can invest my time. I can make working on these hard problems a part of my career path. A long career dedicated to working on big problems to help focus my direction for the future.

I started to think that many of the best engineers I know work at a cross section of engineering and knowing another domain extremely well. I decided I could look for work where I could leverage my software skills while learning more about a big issue I am passionate about improving. These thoughts were still forming in my head as we travel around.

Around that time, I started to come across a post on my social media. I must have seen it three or four times before I decided it was something I needed to follow up on. I do know that Micah, whom I had know through @techstars had posted the job to Facebook. His post was the final push to motivate me to finally click through and contact @Off Grid Electric. The job post really seemed to be a great fit for what I was looking for, a dream job for someone who loves adventure.

Know What You Want

After a bit of time talking with Thor Muller, it became clear that Off Grid Electric also satisfied other parts of a job I find important.

We are a small, highly-distributed software team (US, Nigeria, Tanzania, Russia).
We allow remote workers
We are driven by a mission, which I find personally compelling
Flexible and travel-friendly work
I wanted to work on a project with international components
While technology is important, it is just one tool to solve problems
A long term position - I want to invest time in a company and be able to grow with it.
An opportunity to work closely with end users

The list goes on, there was a lot to make everything seem right to me. It is good to take some time and reflect deeply on what you want, and what you want to be doing X years from now. Then find a path that can take you in that direction. I feel fortunate to have found that role.

Debugging in the Karatu regional office with Dmitry

Look cool? More photos from my recent work trip to Tanzania.

Singing your tune? We’re Hiring

I know other folks that are looking to make changes. If this is the kind of work that speaks to you, ping me I would love to chat. Feel free to reach out over email (danmayergmail) or twitter [@danmayer](http://twitter.com/danmayer).

Open Development Positions @OffGrid

We also have non-software jobs, if you aren’t about writing code all day ;)

Not your thing?

Obviously, the gig won’t be for everyone. If you’re looking for a different kind of change, I should also give a shout out to @livingsocialeng.

Like a bigger team, more interested in consumer products at scale, LivingSocial is Hiring, and is has many brilliant devs focused on big engineering challenges. I am not kidding, it is frequently described by developers past and present as the best job they have ever had.

If neither interests you, but you’re looking for something new, feel free to get in touch. I know various teams trying to pick up Ruby, Clojure, and Mobile devs.

pair programming readme toc

2015-06-02T00:00:00+00:00

Pairing Around the Globe

I met a number of Ruby devs when traveling. One developer I met Samnang Chhun have been keeping in touch and wanting to pair on something. We hooked up online for a bit, while chatting we thought it would be nice if github readme’s had a nice table of contents. So we decided to hack something up real quick.

It is kind of amazing that we can meet people and setup connections online, that much later are still growing and teaching us new things. The Ruby community has always excelled at this. Samnang posted about pairing on this as well.

Considering we only had a little bit of time, the hack was obviously something simple. It was fun to hack on something while catching up, and to take something from ID to usable code while on a single call. You can drag the bookmarklet into you nav bar to quickly add a TOC onto any github page.

(Readme TOC)

Samnang is a Rails developer currently looking for remote contracting or full time Ruby gig’s. If you are looking for someone get in touch with him.

Making A Programming Presentation

2015-03-25T00:00:00+00:00

I wrote this shortly after I gave my talk, but never cleaned up the draft. I cleaned it up a bit this morning and thought I should just publish it as is.

I don’t think there are many posts talking about how to get into presenting at programming conferences. I doubt I’m even particularly good at it, but I still felt I might have some information to share. If you are interested in speaking at programming conferences and think you have knowledge to share, hopefully this post can help you on your path.

The “git log” of my RubyNation talk

I wrote a post about Coverband code coverage on the Livingsocial techblog
I gave a 5 min. lightning talk at MountainWest RubyConf. (slides: Coverband lightning talk)
I decided I wanted to expand on this subject and started seeking some help putting together my first talk proposal. Asking co-workers and online friends for help improving my proposal. I got lots of great feedback.
I submitted my proposal to a couple conferences I was interested in speaking at.
- GoRuCo Conf, not accepted
- RubyNation, accepted
I expanded the talk to give a longer (10 min.) lightning talk at Railsconf, as part of the LivingSocial Lightning Talks (video) sessions. (slides: Coverband Railsconf talk)
I expanded the talk out and gave the presentation internally as part of our LivingSocial speaker series to our developers. Asking for feedback from our team which helped me to improve the slides and talk.
I worked feedback into my presentation and gave dry runs whiled hooked up to a projector. Making sure I was comfortable with my slides and talk.
I gave my presentation at RubyNation and tried to remember what Russ Olsen told me, “everyone watching really wants to see you succeed.” I am sure I learned a lot in the process even if it didn’t go quite as well as I hoped.
- final slides: Production Code Analysis: Tales of deleting 200K+ LOC
- video of my talk (link not yet available, should be soon)

Developing an Idea

I recommend starting with a small idea. Giving smaller lightning talks or short talks at local Ruby meetups to judge if there is enough material to go deeper and enough interest to justify more effort. Watch some presentations and see what kind of style you meshes well with you. Same goes for looking at various slides, find a style and format that you like. Write a short blog post, or even a outline for yourself to see if you have enough material to build a talk.

Seek out people to help give you feedback, in all stages of the process. On the initial idea, while working on the proposal, to listen to shorter “beta” versions of your talk, and as you develop it into a full talk.

Problems I had

Since I gave this as a few lightning talks and internal talks. I ran into various issues, I tried to learn from each of them to improve my final result.

Projectors, keep text off the edges of slides. Conference projectors could be set up poorly or blur on edges.
Bring water!
Try to present on your own laptop, you will be comfortable making changes or fixing issues.
White and grey backgrounds work better than very dark or black backgrounds on projectors that don’t have great contrast.
Try not to read from your slide notes. Speak more from your thoughts evoked from your slides (I didn’t do great at this).
Any live demos practice configuring the 2 screen setup a few times (I had an issue where the demo showed on my laptop screen but didn’t go to the projector screens).
Get good at setting up two monitors, and swapping them. Know the key commands. The screen set to be the dominate screen swapped right before my talk, and I had to sort it out
Smaller code snippets (really the audience can’t read large blocks of code)
Animated gifs! While a easy way to get a laugh. It’s still are appreciated by the audience and keeps people engaged, go for it.
Talks convey concepts and practices better than the nitty gritty of code and specifics. Details might do better in a blog post.
Get good with whatever slide presentation software you end up using. I used deck.js which is pretty cool, but still not as powerful as things like keynote.
Nervousness, while I learned a lot. I think I learned public speaking isn’t really my thing. I am far more comfortable writing a post or working out ideas ahead of time. I might continue to work on speaking, but in the end I doubt it is my best format for me to share ideas.

Cache me if you can

2015-02-21T00:00:00+00:00

Recently, I was looking at a few endpoints to speed up. We were expecting to send large volumes of traffic to a few URLs and wanted to make sure they were fast. While digging in, I found about 10,000 unexected requests per minute (RPM), going between one of our back end services and another. Sometimes the same internal service request was occurring multiple times in a single front end request. I thought this can’t be right, what is going on? Let’s take a look.

I tracked the code issue down to something like this:

def get_object_data(object_id)
  @object_data = Rails.cache.fetch("cache-key-name-#{object_id}", expires_in: 1.minutes) {
      SomeInternalApi.getCoolObject(object_id)
  }
end

At first glance it seems fine, it looks like this is a normal cache call and should never make the same external request twice during the same request, let alone the same minute. Do you see the issue yet?

Remember to take special care if you need to cache nil

Looking back at the logs, I noticed this was only occurring for objects that were no longer available. A minority of requests for sure, but they would happen over and over… Because the API client was turning the 404 response into nil. Unlike the case with an object Rails.cache.fetch won’t cache nil. To read a bit more about features of Rails.cache.fetch or how it might not match expectations read the cache.fetch documentation.

There are two places I see this error pop up:

forgetting the nil case in Rails.cache.fetch, as was the case I just found
forgetting the nil case in a method with Memoization (@var ||= getVar(id)) remember that rails.cache.fetch

That makes sense, how do I fix it?

There are many ways to deal with this problem. I don’t know if I think one is better than another necessarily. Honestly, long as the caching method takes nil into account, I am happy. All of the solutions I have seen are pretty obvious when reading the code.

NullObject Solution

One nice solution, I’ve seen was done by @gfmurphy (according to git blame). It basically creates a Null Object, which can be cached. The Null Object solution looks something like this.

  class DataGetter
    NullData = Struct.new()
 
    def get_slow_data_rails_cache
      @slow_data_or_nil = Rails.cache.fetch("cache_key_#{params[:id]}", :expires_in => 30.minutes) {
        SomeService.slow_data(params) || NullData.new()
      }
      @slow_data_or_nil.is_a?(NullData) ? nil : @slow_data_or_nil
    end
 
    def get_slow_data_memoize
      @slow_data_or_nil ||= SomeService.slow_data(params) || NullData.new()
      @slow_data_or_nil.is_a?(NullData) ? nil : @slow_data_or_nil
    end
  end

Existence Check

Another solution I have seen a number of times, so I don’t know who to give original credit to, I will call the existence check.

class DataGetter
  # this solution only works for memoization
 
  # This will not cache nils
  def get_slow_data__bad
    @slow_data_or_nil ||= SomeService.slow_data(params)
  end
 
  # This will cache nils, hooray!
  def get_slow_data__good
    return @slow_data_or_nil if defined?(@slow_data_or_nil)
    @slow_data_or_nil = SomeService.slow_data(params)
  end
end

I am sure there are many other ways to handle caching for nils. I know I have seen others, but if you have a favorite way to solve this issue send it my way. If not just wanted to share that it is something to keep in mind. I have seen this issue a lot, and I am sure I have made this mistake my fair share of times. It is an easy thing to overlook, but can have fairly large consequences if nil can be an expected response.

Safer Refactoring on Legacy Systems

2015-02-18T00:00:00+00:00

A small refactoring project I took on turned into a crazy journey. Filled with frustration, production bugs, and quick fixes…

started one of those, "it will be an easy little refactoring"… Now I am lost in a sea of yak hair on a ship crewed by yaks… What happened?
— Dan Mayer (@danmayer) February 10, 2015

As you can see from the time between these two tweets it also, was burning away on my back burner annoying me for 8 days. I worked on other things for most of that time while it went through some PR review and QA, but it was still a task to check in on each day.

Hooray! Started as an estimated 2 hr refactoring -> 9 hours of coding, 1 1/2 week of QA / PR review -> 138 prod errors -> fix -> it's live!
— Dan Mayer (@danmayer) February 18, 2015

There has got to be a better way

TLDR; I did a refactoring, which I believe should have been done. After I completed the release, I realized how I could have done it safer and with less stress all around. Read the breakdown and two solutions to see a better way to handle situations like this.

First let me start by saying I am very aware of responsible refactoring, which is a great thing to keep in mind. I knew when I first started this refactoring it would be a little dangerous, but decided to take it on for a few reasons.

we had extracted the functionally of the app into a gem
the gem had made it into usage in ALL other consumer facing applications
the legacy app code was basically a duplicate
the legacy app code had fallen out date
the legacy app code was missing some features
the refactoring has been suggested since the initial creation of the gem and beyond being out of date the code methods and results were beginning to diverge.

OK, this made me think it was worth the risk and there were risks.

the legacy app is poorly tested
the legacy app was notoriously hard to manually test
the code wasn’t designed into the app well in the first place, it was all over the place, my final change illustrates that well git diff => Showing 43 changed files with 130 additions and 492 deletions.
while the original code was out of date and didn’t always match the results of the newer gem, we might have started to “code around” the issues, you know “it’s not a bug, it’s a feature.”

As @tcopeland said, “ha whew. Replacing lots of global-ish methods is tough”

Quick breakdown of the problem

Since every situation is a bit different. I want to try to give a bit better idea of the problem I faced.

#app/controllers/application_controller.rb
include HelpfulModule

# Hey cool, it is mixed in all our controllers, lets really make this global
# we can just include a bunch of helper methods,
# and sprinkle usage of this all over the view layer as well!
helper_method :helpful_check?
helper_method :give_me_some_type_name
...# a surprisingly large set of these
helper_method :is_some_special_thing?

#app/controllers/api/various_base_controller(s).rb
include HelpfulModule

#app/controllers/modules/helpful_module.rb
module HelpfulModule
  protected

	def helpful_check?
	 !!(rand > 5)
	end
	
	def give_me_some_type_name
	  case
   	    when (helpful_check?) then true
        when is_some_special_thing? then false
        else true
      end
	end

	... #many more methods
	
	def is_some_special_thing?
	  return true
	end

end

#many controller and module call sites
app/controllers/modules/auth_support.rb
app/controllers/home_controller.rb
...

#many view layer call sites
app/views/cool/index.hml.erb
app/views/something/_partial.hml.erb
...

The way I did it

I didn’t realize the code had spread so far and wide through out the app. I could thought I’d simply create the object one place in the request and then switch all the calls to it. Basically the change was something like below.

in app/controllers/application_controller.rb

I removed the module: git diff => - include HelpfulModule.new

I removed all the various helper methods git diff =>

- helper_method :helpful_check?
- helper_method :give_me_some_type_name
...# a surprisingly large set of these
- helper_method :is_some_special_thing?

I added an accessor

 def helpful_thing
   @helpful_thing ||= Gem::HelpfulModule.new(request, "stuff")
 end

I removed the module rm app/controllers/modules/helpful_module.rb

I then updated all those many many call sites git diff =>

  #app/controllers/home_controller.rb
  - is_some_special_thing?
  + helpful_thing.is_some_special_thing?

After thinking, I had made all the changes. I ran the test suite. I had a bunch of errors and failures. I missed various calls. Some places didn’t have access to the initialized object. I fixed both good and bad tests (tests which basically had hard coded expectations), and got everything passing. I looked at my diff and realized I had been all over the code, changing far more files than expected and having to fix more unexpected failures than imagined. I knew this had become riskier than initially imagined. Also the time it took me was far longer to get to reach a completed state than I thought it would. I knew that the tests in the legacy app weren’t covering the change well enough and went though manual testing myself along with related fixes. I then passed it off to the mobile team to help QA since it would effect some mobile APIs. We finally deployed it, and boom exceptions. A quick rollback and fix, deploy… All seems good a few hours later some error reports come in, with another minor issue which didn’t raise any exceptions… Another fix released. Finally, the long 8 day journey of the minor refactoring is over.

The way I should have made the change

I should have broken this into two steps. It would have made the initial estimate far more accurate at a few hours. It would have significantly reduced the risk. It also would have reduced the scope of where I needed to focus testing, both manual and automated to catch any unexpected changes.

Replace global methods implementation in module wrapper to simply make calls to the new gem. Initially leaving all callers as they are
After that has been successfully deployed, slowly move callers to direct calls, where it makes sense. Otherwise leave the level of abstraction as a single container for all interactions with the gem.

That change looks something like this

#app/controllers/modules/helpful_module.rb
module HelpfulModule
  protected

	def helpful_check?
	 helpful_thing.helpful_check?
	end
	
	def give_me_some_type_name
	  helpful_thing.give_me_some_type_name
	end

	... #many more methods
	
	def is_some_special_thing?
	  helpful_thing.is_some_special_thing?
	end

   private
   
   def helpful_thing
     @helpful_thing ||= Gem::HelpfulModule.new(request, "stuff")
   end
end

Which is obviously a much simpler and less invasive change. It is easier to reason about and test. It is something I could have completed much faster and released with more confidence.

Conclusion

Yes this is actually pretty common refactoring advice. It has been around nearly as long as the concept of refactoring. Make this type of refactoring in two or more steps opposed to doing it all at once. For me the point was that it is easy to forget the challenges of working with a large and legacy production system. The complexity demands additional attention both in terms of adding features and making “smaller” refactorings. A big part of any change to a large complex production system at the heart of a companies systems, should be a roll out plan. Always think of the safest way to try to move forward. Sometimes that is feature flags, A/B testing, limiting to employee users, and sometimes it is breaking up a refactoring into smaller easier and safer steps, like I should have.

Build robots with my nephew

2015-02-09T00:00:00+00:00

My nephew is interested in robots. I told him that we could build a robot together and he got really excited. Originally my plan was to get Lego mindstorms, but considering it is $350. I am hoping to find a cheaper way to test out the waters. Also, at his current age I assume he will want to keep what we build and may be a bit rough with it. I figure when he is a bit older Lego mindstorms is the way to go.

Anyone know of simple and fun robot kits?

Nephew just turned 6
Looking for something we can put together
Simple action like pushing a button to cause the robot to drive forward for X seconds
Looking to build something my newphew could keep and play with.

If anyone has any ideas shoot me an email, comment, or tweet.

Thanks! Current ideas (shared, sent, and found)

Results

I ended up going with, the tin can robot. Which worked out great. Since he is 6 and the product is recommended for 8 and up, it was a lot more of me putting it together and him helping, but it did maintain his interest the entire time and we were able to finish the robot. I did have issues screwing nearly all of the joints together. I gave up on a number of them and ended up just taping a few pieces together with invisible tape. Overall, still worked out great, he learned some things it sparked his interest, and he asked great questions.

Best Tech Travel Articles

2015-02-03T00:00:00+00:00

I posted the best articles I read while traveling and my travel book list on our family blog. On my dev blog, I figured I could share my slightly nerdier posts.

I thought it might be fun to share some of my favorite’s of the year. Many of these links come from Doug Ramsay, who sends a weekly engineering update along with the best links he has seen. Besides Doug’s emails many links come from some of my awesome LivingSocial colleagues. Enjoy, and remember to send to Kindle to save your eyes.

Programming

Career

Startups

Interesting

Ruby in South East Asia

2015-01-21T00:00:00+00:00

Rubyists are a welcoming community everywhere

Some great things are going on in the Ruby communities in SE Asia. As we were traveling I decided I would like to meet some Ruby communities along the way in other parts of the world.

I reached out to Rails Girls, and their organizers were amazing, , specifically Linda Liukas. Linda was extremely helpful putting me in touch with local organizers around the world. Rails Girls was by far the most helpful group to help get me in touch with a bunch of other great Rubyists. It was amazing to me that shooting a single email to Rails Girls, resulted in at least 5 organizers responding to me and putting me in touch with other contacts to help me meet up with Rubyists while traveling. Huge thanks to from Rails Girls that helped ping pong my email request around the world.

I also met some organizers by searching for local Ruby user groups, and finding Cape Town Ruby Brigade. Specifically, wanted to thank Steve Barnett the organizer of Cape Town Ruby Brigade. Steve helped try to setup a meetup with me while I was in South Africa, even though it didn’t end up working out with my schedule.

Everyone was extremely accommodating and happy to try to get together, when I sent out a few emails with my travel schedule. Our community is amazing that way. I talked with other developers and startups as well, but my initial connection was always through the Ruby community. It is amazing to show up in a city where you don’t know anyone, and have Rubyist welcome you to their work, apartments, cars, and favorite restaurants. #Rubyfriends who share a love for a programming language, and trust the community enough to know we are instantly with friends.

My interest in international developer communities

As I told friends I met up with while traveling. I see developer teams shrinking while expanding their reach. I think in the future startups will increasingly want some international talent on their team. Making it easier to provide continuous support and improvement to their products across time-zones. Making it easier to build support for different languages, currencies, and communities which have different expectations in each country. As remote development gets easier and tools continue to improve, being able to launch sites and applications quickly across the globe will increasingly be a huge advantage. While this is something American developers aren’t as experienced with, my conversations with people and companies in other SE Asia show this global from start is already common in the way they work. For example the Malaysian startup, Watch Over Me, co-founded by Chin Xin-Ci, has their primary user base in the US, UK, and Australia (not in Malaysia). Startups that can work internationally and utilize the growing talent around the world will be ahead of the curve. I think it is how small teams will operate in the future to compete against ever growing multinational corporations, which can employ much larger teams across the globe. Seeing communities which are growing from a handful of developers to communities that are attracting interest and investment from VC firms, is pretty cool. Developers can be a great global community which helps build a middle class across borders and unlock opportunities in countries which might not have as strong of an economy. I think that is one of the reasons software can be an amazing career.

Similarity between Cambodia and Malaysia communities

While I want to talk a bit about each country specifically, it was interesting to see some of the similarities between the two communities. The size of the Ruby community is fairly small in both countries. The community is centralized around the major city of the country (Phnom Phen and Kaula Lumpur). Both communities are struggling with online payment as the government hasn’t fully supported online payment options in the country. While both countries official language isn’t English the majority of apps and sites are built in English. In both countries Android was more common, but mostly because of cheaper units, the developers I met seemed more into iPhones. In part I am guessing because the English language apps often weren’t targeting local populations (Android), instead international tech consumers (iPhone). In both countries, programming seemed to be introduced mostly at the college level, with no computer programming classes earlier in the educational system. The developers I met relied on a lot of self learning on the internet, and in the local community to continue to improve their skills. The local communities being fairly small seemed well connected both to the local community and to the international community at large. The groups I talked with all seemed much more comfortable with working across borders, languages, time zones, and currencies than most US devs. Which isn’t surprising, I just think this will give international developers a good edge in the future. Everyone I talked with seemed to love what they did and were really happy that developing software was such a international career.

Cambodia

First I want to thank Katherine Pe, who through Rails Girls, connected me with Samnang Chhun. Samnang organizes the Cambodia Ruby users group and is a core member of a web development meetup group called Share Vision Team. Samnang organized a lunch, for me to meet some of the developers from the area. It was great to meet freelancers, organizers, and some developers from InSTEDD iLabs SE Asia. Thanks for spending time chatting with me @ungsophy, @channaly, and @ksokmesa.

While out at lunch, I learned a lot about pieces of the current development community in Cambodia. I didn’t know much going in, so learning basic info like most of the community being in Phnom Penh, was interesting to me. Below is a short list of thoughts I had after visiting.

Khmer is the official language of Cambodia, Google play doesn’t support Khmer language.
- This means most sites and apps are in English
- Hard to target older Khmer / French speaking locals
- Focus is on younger people (who mostly know English) and other English speaking countries
small tight knit community, developers from various companies often get together
InSTEDD headquarters, employees many local devs. Seems to be a central role in the community.
- InSTEDD iLab, is sponsored by google.org and Rockefeller Foundation
- working on some awesome project’s like disease tracking in Africa
- not just developers! A interdisciplinary team of public health experts, scientists, and software engineers
Payment systems aren’t in place so many local online companies have to deal with cash (or target other countries)
a large of the community seems to be working for companies based in other countries, small number of local focused startups
There seemed to be a noticeable mobile focus in some of our conversations
Software development was regarded as a good and growing career
The developer community seems most active on twitter

Malaysia

The community in Malaysia was amazing, it was clearly growing fast and had great government support. I first want to thank Winston Teo, who organizes the Singapore Ruby User Group and Red Dot Ruby Conf. I was put in touch with Winston through Rails Girls, who put me in touch with Josh Teng, who became my guide into Malaysia’s developer community. Josh is co-founder of televate.io and teaches a beginner web development class using Ruby called Code Division. The class is being run in collaboration with the Malaysian Global Innovation and Creativity Centre (MaGIC). It was great to get to meet the Code Division class and many people involved in MaGIC. Magic seems to be the center of the Malaysian tech startup scene. I actually just missed the MaGIC startup career fair, which had 70+ startups involved and sounds like it was a great event. I got to spend a little more time with the community in Kuala Lumpur (KL), so I learned a bit more about the community there.

MaGIC operations and former board member, Khailee Ng, is the 500 startups SE Asia managing partner
- I know @davemcclure from his talks with techstars back in the day. Also, from http://500.co/, investing in a few friends startups.
- I find the 500 startups interest in SE Asia especially encouraging for the startup and developer community there.
- Prior to starting a fund specifically aimed at SE Asia, 500 startups had already invested in a number of COs from the area
It was great to meet some people working at MaGIC like Imran Sulaiman, Dr Lau Cher Han to help support the developer community.
Thanks for chatting with me Dhendy Ferdian, who was working too much ;), on various projects
Amazing government support, much of which is funneled through MaGIC
- Developers could get re-imbursed for iphones, github, code climate, and other developer tools
- Government has helped build many developer training programs through MaGIC Academy
- Government is funding scholarships making MaGIC academy affordable to nearly everyone.
Small set of Ruby developers around 50 devs in KL
- The Web Development Bootcamp is aiming to double the number of Ruby developers in the city
- Malay and SE Asia startups have a big demand for more Ruby developers in this area
- advanced schools aimed at leveling up junior developers and exposing them to more complicated systems.
- programs to place Jr. developers in apprenticeships at companies with established developers
local population mostly have androids, but tech community is into iPhones
Google play does support the official language (Malay), but most sites and app still focus on English.
A growing startup scene, GrabTaxi, ShopperApp, iMoney with support from government, investors, and mentors.
small startups like GoGet, a task-rabbit like service. I met the CTO, Fun Wei Tan Tai, at MaGIC.
The government is into some protectionist policies seemingly cracking down on Uber, while helping push the local GrabTaxi. It was fun to get to try the home grown, GrabTaxi service while around KL.
a fun loving community, into the techie lingo, memes, and seemingly SV culture.
Online payment systems aren’t fully supported. Many companies have to deal with cash or focus on countries with established online payments
many developers work for companies out of the country (often in Singapore)
- although compared to Cambodia, a larger percentage seems to be working for home grown companies
The developer community seems most active on FB

Thanks one more time to Chin Xin-Ci co-founder of Watch Over Me, for taking the time to chat with me about her company’s growth and struggles as a startup. It was awesome to get such good insight into a Malaysian startup.

I really enjoyed helping with Ruby code reviews for the Code Division bootcamp class, as well as chatting with the teacher and students over dinner. A really big thank you goes out to Josh Teng, who showed me a great time around KL and introduced me to many amazing people. It was a blast to see where you work, your apartment, and get driven around town by a local Rubyist showing off an awesome startup city. Thanks!

Final thoughts

I heard a lot about Singapore, when reaching out to the Ruby community in SE Asia and in person. I wish my travels had taken me through Singapore as it seems like the programming and startup hub of SE Asia from what I heard. Hopefully I can make a trip to visit the community in Singapore in the future.

It was interesting to get to see some of the amazing growth happening in Cambodia and Malaysia. Overall I was impressed and the visit left me feeling energized and excited about the Ruby community. Happy with how close and supportive the global Ruby community can be. I am not sure if this post will be very useful to anyone else, but it was nice to reflect on my thoughts and remember some of the people and conversations. If nothing else, it is good to be able to once again thank everyone who helped connect me to various developers, organize meetups, and take the time to talk with me. It was really a pleasure to get to know the communities and people. I think it will be very exciting to watch as SE Asia continues to grow on the tech scene and to see partnerships between SE Asian and US companies. I hope I can help some of the folks I met along the way in the future, and I hope to get a chance to visit again. In fact, I hope I can work with some of the people I met sometime in the future.

How to improve profit from tourists for small international businesses

2014-12-18T00:00:00+00:00

As we have traveled for months we often have to rely on online reviews. We are forced to as we often show up in cities with no contacts on the ground and plans to see some attractions. This means when it comes to hotels, food, ATMs, taxis, bars, and other activities we turn to the net. It seems many smaller hotels and restaurants don’t know how to leverage the internet to increase their business. For instance we are in Elmina, Ghana and it only has 1 restaurant according to TripAdvisor, which I am here to tell you is not true.

Some little B&B and restaurants do become active with basic online presence and often tell us about massive growth in their business. It becomes so important they protect there online reputations, diligently responding to any complaints and trying to improve on common customer issues. So how can you attract international travelers, with a low amount of effort? I thought I should share what I consider to be the basic steps for having an online presence that tourists can find.

Have at least a basic website
Ensure correct location information on your establishments on mapping sites
Ensure correct additional information and photos of your establishments pages
For lodging that includes a restaurant ensure both hotel and food listings
For lodging, include an option to pay with credit card

Let’s dive a bit more into the details of the list.

Have at least a basic website.

Your business has to have a website, even if very basic. A single page will do, you can consider managing a google places or facebook page as a website if you don’t know how to create your own. It must list contact information (phone, email, physical address), accurate map location (a link to the correct location on a online map), and links to any other relevant online presence you officially manage (facebook, TripAdvisor, etc).

Ensure correct location information on your establishments on mapping sites

I can’t count the number of times we WANTED to visit a restaurant and could not find it. TripAdvisor has many restaurants with inaccurate location information. Google maps can’t find the business at all. The official website doesn’t link to any map or even have an address that shows up when entered into online map services.

add your location or fix the location of your establishment on google maps, bing maps, apple maps, mapquest
verify the location information is correct on popular review sits like TripAdvisor
if you are a hotel, B&B, check location information on hotels.com, agoda, priceline, or where ever else you have previously accepted reservations from
if you have a facebook page add or correct your location information.
make sure your website links to a valid address on a online map service like google maps, if your address does not yet work on the maps link to direct latitude and longitude on the map which can ensure the pinpoint is in the correct location.

Ensure correct additional information and photos of your establishments pages

Maintain your online pages, adding photos, ensuring correct information (address, phone number, website, map locations). I would recommend maintaining at least these 3 pages:

TripAdvisor
Google Local (google places, google plus, google maps all seem to fall under the name google local for business)
Facebook page

For lodging that includes a restaurant ensure both hotel and food listings

If the establishment is both a hotel and has a restaurant open to anyone, you need two listings on TripAdvisor. One for the hotel and one for the restaurant. Even if the restaurant doesn’t have a distinctly different name, it should be listed as HOTEL NAME Restaurant for the second food listing. This will let travelers find food in cities like Elmina where most of the dinning establishments are attached to lodging. So go submit a second restaurant listing now.

For lodging, include an option to pay with credit card

This is the last item and it is really a bonus point, 1-4 are far more important than this last item. If the establishment offers overnight accommodations, either work with a service AirBNB, Agoda, or any others to have a way to accept online payments via credit cards. Travelers frequently end up in situations where getting enough cash can be difficult. Also, having online reservations that can accept credit card payments through your own website is acceptable, but be careful as I personally don’t always trust small business websites to handle credit card information and often prefer to book through a 3rd party for that reason. I have seen some hotels try to accept credit cards via email or over http non secure internet connections which isn’t OK. If you can’t work with any of the services at least make sure you can accept a credit card in person even if you need to charge an additional credit card fee.

It really is that easy

That’s it, by having a website and managing a few brand pages, many lost tourists who are looking to buy things will come knocking at your door. Besides a bit of time and the cost of internet access, these resources are basically free. The best part is you will get more customers and they will be happier, and require less help. Being able to find information about location, hours, and prices online will reduce the time you spent answering the same questions from tourists calling it. There is a lot more you can do online and here might be ways to spend money on smart online advertising, but if your in a small tourist town you can probably get by, with the tips above. Happy customers will help do the rest for you.

Coding while traveling

2014-11-15T00:00:00+00:00

I set off to travel for about 5 months, during this time I wouldn’t be working. Of course since I actually really love to program, I planned on writing some code while traveling around. My old Mac book was dated, heavy, and starting to have some issues charging (as well as a 45 minute battery life). So I started to think about the machine I would bring with me on my travels. After a bit, of consideration the decision was between a Chromebook or Mac book Air.

The Macbook has nearly all the advantages, but they are expensive and a theft target. A target to the point that some people I been traveling with hide the fact that they have macs (cases, stickers, etc). To cover the cost I looked into a older used Air, but I couldn’t find a good one at the time. Since we are spending a lot on the trip and spending time on my computer wasn’t the trip’s goal I was still interested in the Chromebook.

Any Chromebook has the advantage of being extremely cheap and having amazing battery life. If it is broken, lost or stolen you can buy one or have one shipped to nearly any country for a reasonable price. In the end I went with a Samsung Chromebook. I bought it a few months before my trip to make sure I would be pretty happy using it.

I knew developing on the Chromebook wouldn’t be at all like I was used to, but I thought it would still work out well for me to write some blog posts, edit some photos, learn some new libraries, and work on some of my Open Source Software projects.

Previously, I had been teaching some people how to program. Since everyone had different machines I ended up teaching with each student using Nitrious.io. Since Nitrous has worked so well for me in the past, I planned to do much of my development using that while traveling. Nitrous has a Chromebook app, which is almost great it has a few bugs related to copy and paste that becoming ridiculously frustrating.

Beyond Nitrious, I planned on also writing code using a variety of other methods.

Github online text editor
SSH to my EC2 machine
SSH to my Digital Ocean machine

I figured I had a good set of options, and I am quite comfortable using Emacs over a screen session on a SSH connection. Before starting my trip at home my Chromebook development plan went great, I was very happy with the situation, while it wasn’t as good as a Mac, I figured it would good enough during my trip. If you look at my development methods above, you will easily see they all suffer from the same problem. They rely on having a working internet connection. Which wasn’t a problem at all before I traveled ;)

No worries, I had heard how everywhere in Asia was leaps and bounds ahead of the US for internet ;) Also, rumors of cheap 4G data SIM cards where available in nearly all countries… The problem with that theory is that it doesn’t match the kind of travel I’ve been doing, namely travel on the cheap. We are also, often in remote areas far from major cities. While Asia often has great high speed internet it was never good at a $30 a night hostel;)

While we do have international data with T-mobile, which has been amazing and made keeping in touch easy. The coverage is spotty and often we are limited to edge speeds. If you are out of the cities you have no access. The same issue applies to buying a SIM card, which is only helpful when you are near cities. I am currently writing this post in a tiny town in Uganda with a view of a rainy Nile river, which has 30 people sharing a single 4G connection.

I didn’t think having good access to the internet would be such a problem, but it has been continually difficult to find a decent connection. My backup plan for this was local text editors and recent Chromebook apps like Chome Development Kit. While interesting the backup plan hasn’t been very useful.

First, I end up wanting the net to pull various JS libraries, CSS, or look up solutions. The other issue is that I don’t want to built a pure JS application. I want to build an app that interacts with 3rd party APIs, or a simple server back end. Both of which quickly put me back in the situation where I need net to build anything interesting. While it has been fun to play with a few D3 javascript visualization scripts locally, it hasn’t really let me work on the kind of projects I was hoping to play with while traveling.

I assumed between good internet and occasionally spending some time on local development I would be pretty satisfied. In the end, that hasn’t been the case. I do regret going with the Chromebook and if I could do it all again I would have gotten a Mac book Air. I am sure, I would have spent a bit too much time on the computer, but I would have been able to keep up with my skills better. Since working on development projects hasn’t worked out. I have had time to read a large amount, as well as spend a good amount of time writing. The Chromebook as a distraction free text editor, is hard to beat. Without net or the ability to do much else, I have been able to write a surprising amount.

Final thoughts on the Chromebook

If you do get a Chromebook I can say the older version didn’t have the horsepower to be that great at photo editing. Any large number of DSLR images slow it to a crawl. My wife has the newer Samsung Chromebook 2, while it doesn’t have a built in SD card reader, adding a USB reader will give you a much more usable travel laptop. If it wasn’t for wanting to develop offline a bit, I actually think the Chromebook is a pretty ideal long term travel machine. With all the wireless features turned off. I can get a solid 7 hours out of the machine, which gives me a number of usages between charges. It is cheap, fast to boot, and can jump online faster than my Mac can search for WiFi networks in reach.

C Extension Notes

2014-07-24T00:00:00+00:00

These are unorganized notes from building a c extension, I found it in the drafts folder and figured it might as well be set free.

Building a Ruby Gem with a C extension

Notes

Examples

Stackprof.c
fast_stack.c (older simpler version)
FastStack.rb (older simpler version)

Gotchas

If you are compiling locally over and over, it appears that the .o files from install gems will stick around. You need to gem uninstall the native gem and reinstall it to pick up your changes.

Perf Progress

ab -n 100 -c5 http://admin.lsdev.co:80/

No Coverband

Time taken for tests: 3.980 seconds / 3.999 seconds

Ruby Coverband

Time taken for tests: 31.489 seconds / 29.064 seconds

Ruby Coverband C Ext V1

Time taken for tests: 14.679 seconds / 12.033 seconds

Ruby Coverband C Ext V2 with Regex

Time taken for tests: 6.131 seconds / 6.321 seconds

PL app

ab -n 50 -c5 http://pipeline.lsdev.co:80/pipeline/cities

No Coverband

Time taken for tests: 31.347 seconds / 31.298 seconds

Ruby Coverband

(moved to, so slow ab -n 10 -c5 http://pipeline.lsdev.co:80/pipeline/cities, then multiplied by 5 to keep the numbers in line) Time taken for tests: 336.53 seconds / 331.89 seconds

Ruby Coverband C Ext V2 with Regex (back to the default 50 requests)

Time taken for tests: 42.785 seconds / 43.343 seconds

C code to debug

    rb_eval_string("puts 'self is:'");
    rb_funcall(rb_mKernel, rb_intern("puts"), 1, self);
    rb_eval_string("puts 'klass is:'");
    rb_funcall(rb_mKernel, rb_intern("puts"), 1, klass);

Warnings I need to fix

    make
    compiling coverband_ext.c
	coverband_ext.c:44:23: warning: incompatible pointer types passing 'void (VALUE, VALUE, VALUE, ID, VALUE)' to parameter of type 'rb_event_hook_func_t' (aka 'void (*)(rb_event_flag_t, VALUE, VALUE, ID, VALUE)') [-Wincompatible-pointer-types]
    rb_add_event_hook(trace_line_handler_ext, RUBY_EVENT_LINE, 0);
	                      ^~~~~~~~~~~~~~~~~~~~~~
	/opt/boxen/rbenv/versions/1.9.3-p448/include/ruby-1.9.1/ruby/ruby.h:1427:45: note: passing argument to parameter 'func' here
	void rb_add_event_hook(rb_event_hook_func_t func, rb_event_flag_t events,
	                                            ^
	coverband_ext.c:45:15: error: use of undeclared identifier 'bindval'
    rb_iv_set(bindval, "@tracer_set", Qtrue);
	              ^
	coverband_ext.c:53:24: warning: incompatible pointer types passing 'void (VALUE, VALUE, VALUE, ID, VALUE)' to parameter of type 'rb_event_hook_func_t' (aka 'void (*)(rb_event_flag_t, VALUE, VALUE, ID, VALUE)') [-Wincompatible-pointer-types]
	  rb_remove_event_hook(trace_line_handler_ext);
	                       ^~~~~~~~~~~~~~~~~~~~~~
	/opt/boxen/rbenv/versions/1.9.3-p448/include/ruby-1.9.1/ruby/ruby.h:1429:47: note: passing argument to parameter 'func' here
	int rb_remove_event_hook(rb_event_hook_func_t func);
	                                              ^
	2 warnings and 1 error generated.
	make: *** [coverband_ext.o] Error 1  

C Extension Code examples

Working with ifdef to detect and deal with ruby version differences

    #if defined(RUBY_VM)
      #include <ruby/re.h>
      #include <ruby/intern.h>
    
      #if defined(HAVE_RB_PROFILE_FRAMES)
        #include <ruby/debug.h>
      #else
        #include <vm_core.h>
        #include <iseq.h>
        // There's a compile error on 1.9.3. So:
        #ifdef RTYPEDDATA_DATA
          #define ruby_current_thread ((rb_thread_t *)RTYPEDDATA_DATA(rb_thread_current()))
        #endif
      #endif
    #else
      #include <st.h>
      #include <re.h>
      #include <intern.h>
      #include <node.h>
      #include <env.h>
      typedef rb_event_t rb_event_flag_t;
    #endif

flame graphs in miniprofiler

Advice from my grandfather

2014-07-01T00:00:00+00:00

Advice from my Grandfather, “Show up”

My grandfather had some advice for me about being good at your job. I am not exactly sure when he shared the advice with me, it might have been before college or when I visited after college right before my first job. But I think of it often –

“90% of being good at something is to just keep showing up”

Or the short version, “show up”. Faced with something hard, keep showing up and work at it. Take on the hard challenge no one else is taking on, even the dull challenges which are important occasionally. Push yourself, take on challenges just out of your comfort zone.

I think in the end this is a bit related to the common advice that it takes 10,000 hours to become a expert. Want to get to 10,000 hours? Keep showing up and working at your craft, it can take over 10 years to become an expert.

Showing up: motivates me

if there is there a really important project that no one seems to find ‘cool’ enough to work on, show up
if there are unexpected issues days before a release, dig in and solve it for the release.
if the team is finished with the 90% interesting part of the project but that last mile needs to get done, show up for the team to help pull it through to the end.
If production issues are taking systems down and no one seems to remember how it works, dig in.

Showing up is important for a variety of reasons. The actions you take shape the person you are. Being ready to pitch in often, and your team will appreciate your efforts and know your willing to help get whatever job done. Your manager and your company will appreciate you for it. You will be trusted to make other good decisions because you have shown up and come through in the past.

Showing up: is a great way to learn

Want to be a leader or become one of the most knowledgable on the team? Showing up is a great way to always learn. You might not start as the best, but if you are one of the people that consistently shows up and steps in when needed, you will become one. How do you become skilled in all areas of a career, by taking on tasks and challenges when they need to be done even if you don’t know the solution.

You can see knowledge spread to people who work this way, some of the junior developers co-workers have on multiple occasions stuck around to watch over the shoulders of more experience devs while they dove into and solved an emergency. Sure enough, next time similar issues came up, the junior developers who had watched before stepped in and jumped on the tasks themselves while a senior dev checked in to see if they needed help. Often solving issues before any of the more senior devs were available to help. It is amazing how much knowledge gets shared this way.

If you like to learn new things showing up is always a great thing to do as well. Who wants to just get really good at solving the same problem over and over? Unknown mystery exceptions occurring a handful of times? Add monitoring, metrics, read up on the related systems, put in probes to verify what you ‘think’ you understand. Track it down and suddenly you know more about a subsystem than anyone else around, and solved a problem lurking in the system.

The number of things I have learned this way has been to many to count, it is something I have always like about being a developer, there is no real way to trap me to a specific job. As software eats the world, I am learning the skills and tools to quickly make myself productive in nearly any area of a business. System security, intrusion detection, business analytics, performance, debugging, real time dashboards, big data, personalization, mobile, infrastructure… All things I have mostly learned on the job, because we needed someone to take on a new challenge.

So if you want to learn a ton, be one of the best on your team, become irreplaceable at your company, expand your skill set, and become overall excellent at your job. Keep showing up, especially when no one else can help out at the moment.

Know the systems? Congrats you’ve earned the the “tour guide” badge

Once you’re a go-to to answer questions about the systems, a large part of your job is teaching all that knowledge to others on the team. As Adam keys describes in Developers are From Mars, Developers are From Venus, you will reach a tour guide level of knowledge.

One word of warning

This doesn’t mean you need to work all the time. It doesn’t mean let people or business walk all over you assigning you more work than one can handle. It doesn’t mean you should always be doing the work no one else would be willing to. It does mean taking on any challenge and not ignoring important parts of the business, to only work on fun or traditional ‘developer’ projects.

Talk Proposal Help

2014-03-25T00:00:00+00:00

I wrote my first CFP and submitted a talk to GoRuCo. Sadly, I wasn’t selected, but I think thing I have something worth working on. I got some additional feedback that I hadn’t received in time to update my GoRuCo proposal. I have incorporated the changes and now plan to submit my talk idea to RubyNation (6 days left of CFPs). I would love to get some additional feedback on other ways to improve the proposal or make sure it is interesting to the audience.

If you have any feedback on the proposal below you can use draftin to submit your changes to my document.

Talk Title

Production Code Analysis: Tales of deleting 200K+ LOC

Talk Description:

In a long-lived Ruby application with many contributors, it’s common to feel like you have lost control of the codebase. Refactoring is difficult in these situations, but tools are available to help you regain the control you once had. I will cover driving refactoring based on production metrics and analysis as a solution to this problem.

Start simple: what you can learn from New Relic and other performance monitors
Fill in missing pieces with event tracking, email tracking, partial tracking, i18n locale tracking, and one-off trackers
Quickly comparing performance between two implementations
Introduce Coverband code coverage in production, which detects unused code.

Talk Format:

30 Min

Intended Audience:

Intermediate

Speaker Bio:

I moved to Ruby from Java while trying to create a startup. We were funded by Techstars then O’Reilly AlphaTech Ventures, and briefly known in the Ruby community for Devver (cloud accelerated testing) and Caliper (code metrics). Failing to find a revenue stream, I joined the fast growing company LivingSocial. At LS, I have worked on internal, mobile, consumer, and platform initiatives. Currently tech lead for Consumer Deals, helping tame our mono-Rails beast. I write about code at http://mayerdan.com

Has this talk been given in the past? (If so, where?):

No, this is the first CFP I have ever written, I have been kind of scared of public speaking ;)

note: Since writing this CFP I gave a portion of this talk as a 5 minute lightning talk at mountain west Ruby conf

Links to past talks you’ve given (video, audio, etc):

None. Mostly I have written about code on my blog, http://mayerdan.com. For instance, I introduced Coverband on my blog.

Other Samples:

Investigating Api Developer Tooling

2014-01-29T00:00:00+00:00

Evaluating API development tools

I have been evaluating various API tools lately. In part as I develop more APIs myself for personal projects, as well as the importance they play in the systems I work on professionally. I have learned that all web projects will have some sort of API backend and multiple front ends: web, mobile apps, native desktop, internal reporting tools, etc, etc.

If you can generate any sort of interesting data eventually people will want it in other formats. This has changed how I look at data sources, with data I am starting to think about API first. Many other developers are starting to focus more on internal and external APIs as well, which is why there is a variety of great tooling and competing standards growing related to it.

I had been playing around with some API tools such as Apiary, and Fdoc on my side projects. At work, various discussions about the future of our APIs has recently become a focus. Adam Keys, Cloves Carneiro, and Tim Schmelmer are leading the charge in having better tooling around our APIs. The discussions lead to some features we wanted to support, which caused me to start to take a much closer look at some of the API tools available. I decided to try to take notes as I dug in to research a variety of options.

image courtesy of marc_smith

Features we wanted

The most common features we wanted after discussing our APIs looking over existing tools.

Support for multiple languages. (preferably all the ones we use in the company)
Support for automatically generating client code from an IDL definition
Support for generating service-side generation of API documentation, based on the IDL definition of the APIs
Support for cURL / browser based service interaction, to analyze requests / responses in a “human-readable” format

Other features were mentioned, but this was the most common and most desired list of features. Since we are considering tools for integration for professional apis, I wanted solutions that would work self-hosted / behind the firewall. Which makes some of the hosted solutions like Apiary less interesting although large parts of their solution is open sourced, like API Blueprint.

The growing number of APIs we wanted something similar to Google APIs Discovery Service. Google’s tools are great, but not really an option as they haven’t open sourced the document generation, service discovery server, or much server side language tooling. What has been open sourced is a client code generator, which doesn’t have as much language support as we would like (Java, Java/GWT, .NET, and PHP). Also, open sourced was the google-apis-explorer which while cool doesn’t quite cover full documentation needs. These tools would work for you if you build out APIs that match their schema, but their supported languages are limited and don’t include my primary language Ruby.

Why not just standard restful JSON apis

Because this is the state of most APIs I use today. Without any tooling just building rest-like APIs response formats, tool chains, documentation is slightly different. Often documentation is or not vary accessible. API Client are a mix of thin configurations of rest-client or large custom client gems with heavy models. Across many APIs this makes the tests, caching, usability, and usage expectations a new challenge with each API.

I want to get more benefits than just moving from various custom presenters and JSON formatters to a consistent usage of a JSON serializing tool / specific JSON formats. Since most of the APIs I work with are in Ruby there are a number of tools that solve only that limited set of issues. I am just looking for a bigger win than the various Libraries below provide:

RABL
Jbuilder
Acts_as_API
Roar
Grape (although admittedly this one starts to get closer, with some of the optional integrations with other gems)
and more…

For people interested in just having a solid Ruby api with a great format and code to help you build that one of the above options might be a great choice. The change log has a good post on crafting JSON output, which shows some examples, although it is a little dated. Also, watch the various Railscast API videos before you get started, it can help save you from a silly mistake.

Benefits Beyond JSON Formats

One piece of the puzzle is using standardize JSON formats. Try not to just make up your URIs and data representation yourself, I promise you will regret it. Right now many APIs I use and some I have developed (I know for shame) are hand rolled and the structure of the data isn’t ideal. Many apis are fairly loose and inconsistent in terms of how they serialize data. Some JSON format standards could help us improve that, as well as provide validators that could be integrated into the testing process, etc. Others only help in terms of defining where endpoints exist, how endpoints are accessed, and what the responding data looks like. While these are interesting, they are more interesting when paired with tooling based on these standards. Most of the tools mention in my list utilize one of these formats and some could support multiple formats in conjunction which seems promising.

JSON API
- A format by Yehuda Katz and Steve Klabnik.
- The default ember data format.
- The format restricts how you represent your JSON data, but to a list of tried and true best practices for representing your objects.
- Example implementations exist for Ruby and other languages.
JSON-RPC
- A JSON standard with a highly supported version 2.0 spec.
- Defines rules for making JSON remote procedure calls, has client support in many languages.
- Defines good input, client, and error behavior.
- Doesn’t define or enforce good JSON data serialization of your objects. Allows for any format.
JSON Schema
- A format to describe APIs in JSON.
- The basis of a growing list of tools, including some on this list like swagger.
- While this format describes how your JSON api data is represented. It doesn’t enforce any good data serialization habits, allows for any format.
Apache Avro JSON
- While Avro has a whole set of tooling one can choose to just support its format
- The format helps define and enforce good data serialization habits.
- Good language and platform support.
- Useful beyond APIs Hadoop and log processors, could utilize object and message serialization. Which would improve some issues we have seen with pure JSON.

Why not Thrift, Protocol Buffers

The largest reason is in the end I want human readable APIs. I want to be able to debug and test with standard curl commands. We want to be able to use simple Ajax calls without additional tooling. The benefits of Thrift and Protocol Buffers seem to not be large enough to force our stack to always use non standard tooling and limit our choices in terms of integrating other options. Finally the support for these formats isn’t quite as strong as we would like in terms of native mobile support. The only one of the data serialization and RPC systems that seems to not have as many limitations is Apache Avro, which will be covered in more data in the comparison list.

Api Tooling Options

Tools with offerings most similar to Google APIs Discovery Service

ioDocs: Host or Self Serve API docs, built and supported by Mashery
- pros:
- Large community support and great documentation
- Very nice interactive documentation, documentation live demo
- Open source support for the Google Discovery Document Format
- Built on the JSON Schema standard
- Allows for service discovery for all APIs
- Could be used with a JSON format, adding format validation to the tool chain for future APIs would be easy
- Supports various authentication mechanisms
- Many example integrations
- cons:
- No server code or comment integration, so the JSON docs need to be maintained and kept in sync by hand
- Client code generation has limited support and doesn’t Ruby or Scala
- Less tooling available than other options
- While trying to be a open source implementation of Google’s Discovery Service it is missing pieces and will always be behind it since it relies on it’s tooling some places
Swagger: An API stack built by Reverb
- pros:
- Very usable, interactive documentation, Live Demo of the interactive docs
- Built on the JSON Schema standard
- Good community support, active swagger google group, highly responsive IRC room
- Picked up and in use by several large companies like Salesforce
- All related tools open sourced, with active contributions
- Stub server support
- Code generation support in many languages include our primary focuses (Javascript, Ruby, Scala, Andriod, iOS)
- Docs can be written for existing APIs, opposed to having to build new APIs to support the format
- Has integrated server side support for generating the swagger JSON docs
- Integrated Server code (or annotated comments) more likely to keep in docs and clients in sync with changes
- Allows for service discovery for all APIs
- Could be used with a JSON format, adding format validation to the tool chain for future APIs would be easy
- Supports various authentication mechanisms
- cons:
- Client code generation is mixed quality, good JS and Scala libs, while Ruby isn’t so great
- Server code generation is mixed quality, again Scala being a great example with Ruby being OK
- Current Ruby server side implementation likely would require comments in code, as the framework integration seems too immature
- The Ruby code both server and client would likely need heavy contributions
- While encourages good restful behavior, doesn’t really enforce any data format standards
- Some of the tools are still in early / active development. Because of this documentation is a bit lacking, errors are difficult to interperate. This leads to a larger learning curve than some other tools
RAML: A YAML description language for rest-line JSON apis, built by Mulesoft
- pros:
- Very polished tooling & extremely good documentation, seems to have community support via RAML workgroup
- Sharable API notebooks to share example usage and scripts, might be an interesting way to share configuring and usage.
- Interactive Documentation, Live Documentation Demo
- Nearly all components open sourced
- Many example integrations
- Stub Server support
- cons:
- Early code generation tools, not yet stable. Ruby not supported
- Documentation of the API lives entirely outside of standard code flows and tools
- While a open format, it is kind of a odd non-standard format

Highly Integrated with Ruby

Rails-API combined with ActiveModel::Serializer
- pros:
- Supports a common and standardized format JSON API
- Heavily integrated with Rails
- Great Javascript client support
- Enforced good data structure for API
- Simple model for building dynamic clients
- cons:
- No built in documentation support
- Little in the way of tooling: validations, client generators, or other tools
- Still a work in progress
Rocket Pants: opinionated rails api framework
- pros:
- Highly integrated into rails and ActiveRecord (AR error support, AR object mapping support.)
- Most standard rails coding style
- build in support for easy caching
- Easy Ruby client generation via ApiSmith integration
- Consistent and enforced good data structure for data serialization
- cons:
- Doesn’t include pretty documentation generation support
- Also a pro, but heavily tied to rails and no other language support
- Custom format doesn’t yet support JSON Schema
- Client generation is Ruby only
FDoc: Documentation format and verification, built by Square
- pros:
- Works with Rails or Sinatra
- Integrated testing and verification support
- Supports JSON schema standard
- Includes documentation page generation support
- Company sponsored support via Square
- cons:
- The documentation pages leave a lot to be desired. Not interactive, not very skimable. Very bare bones
- No client generation support
- Limited non Ruby language support
- The doc files are kept in separate files, no server code or comments integration
Interpol: A toolkit for working with API endpoint definition files, built by Seomoz
- pros:
- Integrated Ruby testing tools to validate endpoints
- Stub server support
- Development tools to ensure API endpoint responses match current schema, keep things in sync
- Documentation generation support
- Request parsing validation libraries
- Supports JSON Schema
- Would allow support of existing APIs
- cons:
- Definition not integrated via code or comments, but kept as a separate YAML file
- The documentation pages are pretty bare bones, not interactive
- No client generation support
- While you define how data is serialized it doesn’t enforce any good serialization habits
- Ruby only for server side integrations

Other Options

Apache Avro
- pros:
- Better native support than Thrift or Protocol Buffers, see why Flurry chose Avro
- While supporting a pure JSON format they have a faster binary format as well
- Good platform and language support
- “Simple integration with dynamic languages. Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages.”
- Advantages in integration with Hadoop and log processing
- cons:
- while sort of human readable it can be a bit more difficult
- while sort of curl-able, writing valid curl calls can be a bit more difficult curl curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"adSpaceName":"splash_screen","location":{"lat":12.231212,"lon":23.3435},}' http://localhost:8080
- Avro Ruby code needs some love and performance improvements
- No built in documentation generators
- A bit heavy weight with a higher learning curve than others
- No way to support old existing APIs could only move over as part of upgrades
Barrister
- pros:
- built on the JSON-RPC 2.0 standard
- Integrated Ruby server and client side support
- Generates static documentation pages, based on nice Docco standard. Small Example Doc.
- Fairly bare bones clients
- Extremely simple learning curve, could get up and running quickly
- Could work with existing APIs
- Server integration likely doesn’t support our needs
- IDL generated as a separate file, no tight server code or comments integration
- Supports multiple languages
- cons:
- Doesn’t enforce good data serialization or URI practices
- Scala support not listed yet
- Smaller more limited tools than most
- Seems to have smaller community support
- Likely would need to hand roll some supporting tooling for both clients and servers
Roll Your Own
- pros:
- Complete control over all pieces and stages of the tool chain
- Heroko’s JSON schema support for platform API. Heroku did it! They support standard formats but rolled their own versions of document and client code generators. Seems very close to Swagger format and mentions Swagger standard, but doesn’t seem fully compatible and they went their own way for some of their tooling
  - Heroics: Heroku’s Ruby client generation from JSON Schema
  - Generating a Go client a tutorial building a Go client
- Leverage tools released from other companies efforts
- We can really mix and match some of the formats and tools mentioned above trying to get the best of everything
  - For instance one can choose to use JSON-api (or Avro JSON) format with nearly any of the documentation / verification tools
  - We could take a system like FDoc or Barrister and add tooling to extend it to generate more robust swagger or ioDocs output
- cons:
- Higher learning curve
- Higher maintenance costs
- Could make a bad call, all the other projects would have more community involvement
- May be harder to support existing APIs
- Likely overall looser integration than working with a fully integrated stack
- There are pretty big projects so it might be a large undertaking, depending on features you want to support

Next Steps Towards Better APIs

It is great that there are so many good options being built to improve tooling around developing APIs. Obviously, there are so many options it isn’t really clear what the best option is. The next step seem to be to pick out and implement a couple of the options to explore them in more detail. Either trying a different option for a couple different services or implementing several of the options on the same API to see which seems like the best fit.

In fact I have already started to do this with Swagger, which is why it has a more detail describing it than several of the other options. One nice thing about researching all of these options and implementing one of them, has been learning how bad many of my APIs are. Seriously, trying to layer tooling like this on top of an awkward API and data response objects makes it abundantly clear that a poor planning was involved. It immediately makes the serializations used for the models seem awkward and overly verbose and the URI endpoints poorly chosen. Going through the exercise should improve your APIs. Building a new API with any of the tools or libraries above would likely result in a better API because it would force you to really think about the contract the API is describing when it is being implemented.

I will follow up with more specifics on some example implementations of some of the options above. So stay tuned and see you next time.

Self learning Programming, teach yourself to code

2014-01-13T00:00:00+00:00

I think everyone in today’s world should be learning a little about how to code. If you use a computer at some point you will want to better understand how it works, or you will want to be able to create something on it as opposed to using it only for consumption. Even learning some of the fundamentals of programming will help you understand how the applications you use to socialize or work function. The basics of creating a webpage, applying styles, and adding interactivity/data storage will come in handy far more often than you can imagine even if you never become a “developer”.

Since the early days of the internet, you could basically find everything you needed to learn about programming online. This is especially true today given the explosion of online education that has happened in the last few years not just for programming, but for learning nearly anything one would want to study. I have been really interested in online and self learning education in general. As various people have asked about programming or wanting help making basic web pages, I have been specifically looking at developer education online. It is amazing how quickly the space is growing and how great the resources have become. Both for those who is interested in learning just a bit to make their own web pages, and for for those that want to learn it all and make a career out of software development. As education approaches the career level there are offline components as well, but even those programming bootcamps are a new and interesting education model..

Note: Since I program primarily in Ruby, the resources are slightly skewed towards Ruby. Many of the resources, offer multiple languages and there are additional programs focused on other languages that might interest you.

Self Guided Free - Where To Start

Before going to a school or even an online class that has a schedule and assignments, I recommend starting with self learning at your own pace. Spend some time on nights and weekends and learn some of the basics and see what parts of programming might interest you.

I really recommend everyone starts by learning the basics of web programming. The beginner friendly courses at CodeAcademy is where I recommend everyone starts:

HTML & CSS
Javascript
Finally a “backend” language like Ruby or Python

After that, learning more and getting a deeper understand of a programming language of computer science basics in general makes sense.

TryRuby.org: free interactive Ruby tutorial great way to play around and learn some Ruby basics quickly
If you are interested in Ruby, I highly recommend the free Ruby in 100 minutes tutorial by Jumpstart lab
- Jumpstart Lab offers further self learning Ruby and Javascript tutorials
CodeAcademy: free mobile 1 hour programming intro app, a fun way to learn with spare time on your phone.
Khan Academy: one of the first free online classes, offers a great self guided CS set of courses
Rubymonk: free interactive tutorials
Code.org: part of the “hour of code” initiative, it offers tutorials and additional resources to get started learning to code
Why’s Poignant guid to Ruby - the most interesting and artist programming book you will ever find.

Free - Unstructured

Free - Structured

Coursera: Free university support online courses, these couses have start times enrollment periods, and require keeping up with the classes pace.

Minimal Costs / Free Intros

code school: free beginner courses($29 a month for intermediate and advanced courses)
RubyOffRails: videos, mentorship, office hours($39 a month and 80% Discount for Women)
Udemy: self paced video training technology courses (some free)
Udacity: Self paced small cost self contained online coursed (some free)
One Month Rails: self paced video tutorials and online community ($99)
Pramatic Studio: self paced video courses, from the authors of the most famous Ruby book
Rails Tutorial (free book, $50 for training videos)
Learn the Command Line the Hard way (other materials from the Learn X the hard way series)

Face to Face Courses (1-5 days) - Get a Taste or Level Up

Jumpstart Lab
5 days with Ruby and Rails
often you can find 3 day programming courses on eventbright, for example Learn Ruby + Rails the Hard and Fast Way
Many conferences offer train camps before or during conference. Just check upcoming conferences. For example training at Lonestar Rubyconf.

Bootcamps - Make a Career Out Of It

Launch Academy
- $12,500
- 6 weeks interactive
- 10 week in person bootcamp
- career placement support
GSchool
- $20,000
- 24 week in person bootcamp
- career placement support
The Starter League
- $12,000
- 12 weeks in person bootcamp
- career placement support
bloc.io
- $4,250
- on-line video / mentoring bootcamp
- approximately 12 weeks, but learn at your own pace
- online meeting with mentor 3 times a week.
Betamore Academy
- $3,000 - $3,400
- 10 or 12 weeks
- front-end or back-end course focus, in person bootcamp
- career placement support
coding dojo
- $9,850
- 12 weeks in person bootcamp
The Flatiron School
- $12,000
- 4 weeks online
- 12 week in person bootcamp
- career placement support
Dev Bootcamp
- $12,200
- 9 week in person bootcamp
- career placement support
HackReactor
- $17,780
- 12 weeks in person bootcamp
- career placement support (98% graduate hiring rate)
Mobile Makers
- $9,000
- 8 weeks in person bootcamp
- mobile / iOS focus
- career placement support
Metis - with the guys from ThoughtBot
- $12,500
- 12 weeks in person bootcamp
- career placement support

No Silver Bullet

I think all of this is really promising and interesting, but the industry is young. Mistakes will be made - some will be poorly trained, and some will be flat out ripped off while seeking knowledge. Make sure to research your choices, and just like college you really will get out of it what you put into it. As LeVar Burton says on Reading Rainbow, “But you don’t have to take my word for it.”

Thanks

Thanks goes out to Andrew Thal, Ed Weng, and Emily Rocheleau for reading the draft of this post and making improvements. I am sure anyone who read the post understands it better for the help.

Update

A friend is running a to help people find a dev camp that fits them well. Check out Coursereport.com - A programming bootcamp directory that features reviews, ratings for over 100 bootcamps worldwide.

Coverband Production Code Coverage

2013-12-17T00:00:00+00:00

Rubyists have long used code coverage to help guide development. While our coverage reports are generated from tests, I wanted to see coverage of the production system as it was getting exercised by our users. After discussing with a number of developers and some failed attempts, I built coverband to record production code coverage.

Why production code coverage

Dead code is a bug waiting to happen. It makes reasoning about a system more complicated, maintaining large codebases is hard enough without thousands of lines of un-executed code.

If your team is managing an aging production system, a rotating group of developers and years of refactoring can create many dead code paths. It is often easy to find all the endpoints being used with data from New Relic, or any other web analytics. It is much harder to find all the helpers, unused model methods, and other code that has been refactored out of use. Using Coverband, we have found entire models which no longer get executed by any code path, but still had test coverage. It also made it easy to see conditional paths which are no longer followed in all directions. Dead code often has test coverage slowing your test suite, and occasionally developers have to spend time maintaining or fixing it when upgrading Ruby versions or frameworks. The less code you have the easier a system is to reason about, and the more clear the abstractions become.

How Does it work

Initially I was trying to use Ruby’s standard library Coverage. I tried to use Coverage on both Ruby 1.9.3 and Ruby 2.0 and ran into issues with it crashing. It didn’t seem to play nicely with the idea of sampling the requests. It would often run smoothly for a short while, but would eventually segfault even on tiny example applications.

This led me to look for other solutions. Eventually I came across set_trace_func, which was added in 1.9.x to give Ruby some basic tracing functionality. This has been improved for Ruby 2 with the TracePoint api. While there are a few weird quirks with Ruby tracing where it doesn’t quite capture everything you expect, in general it works very well to allow capturing live code usage. Coverband uses a combination of Stdlib’s Coverage to take a small baseline measurement of code executed while booting. Then it uses a Rack::Middleware layer enabling set_trace_func to get runtime coverage.

Since collecting the coverage data does have a performance overhead, I wanted to be able to slowly gather data without affecting all requests. So Coverband implements a sampling strategy allowing a small percentage of requests to be monitored. On smaller apps, I have a large percentage of requests being sampled while on large, high traffic sites I have been running Coverband with as little as 1% sampling and am still collecting very useful data.

While I think there are various improvements that can be made to Coverband, after running it on a few production projects it has already proved to be useful. Our team has been able to find large chunks of unexecuted code, leading to a series of my favorite commits, the elusive negative change commit.

This post will not cover the full usage and setup, as Coverband is under active development. The current detailed instructions can be found in the project’s Readme. If you want to try it out, follow the setup instructions on the Coverband github page. Any issues or feature requests can be passed along as github issues.

What do you get

To avoid reinventing the wheel, Coverband coverage output looks just like most Ruby users have come to expect. It outputs in a Simplecov compatible format. This makes it easy to use any Simplecov formatter, which is what Coverband does by default.

Thanks Livingsocial

This post is a cross-post about coverband on the Livingsocial tech blog. I work at Livingsocial and we are hiring. At work we are using coverband in production on some applications to help drive refactoring and cleanup efforts.

Configuring a new OS X Mavericks development machine

2013-12-01T00:00:00+00:00

These are the steps I took to setup a new OS X 10.9 (Mavericks) machine for development. I have heard on twitter that each time you setup a new machine you get 1% better and at this rate you can become good at it in a few hundred years! Ha while that seems to be true I am getting happier with the process and I am now at least trying to document it. This post won’t serve as a rich tutorial on the configuration process, but should give some ideas on good ways to setup your own machine or ideas on how to fix some common gotchas I ran into with my setup.

I know this is a very opinionated approach without much explanation, so if you are new to development I recommend finding richer tutorial as a this is largely intended for my own documentation.

Initial setup steps

verify full disk encryption
update password on account / machine (if one was previously configured)
install all OS X updates
install X code (this and the updates can be slow get this started as soon as you can)
- options download install pretty much all the requested packages
- cmd line xcode-select --install

While Xcode installs

install chrome
while X code installs install size up, or your prefered window manager
install dropbox, get sync started
get ssh keys
- tar czf key.tar.gz .ssh/
- gpg -c key.tar.gz
- scp it over from other machines or encypted file sync via Dropbox
- gpg –output key.tar.gz –decrypt key.tar.gz.gpg
- tar xvzf key.tar.gz
download and install virtualbox

After Xcode, install / run Boxen

Installing Boxen will take a long time, so you can continue any download and configuration steps while it runs although some things you want to install are dependent on Boxen completing.

Boxen setup steps

cd our-boxen-master/
sudo mkdir -p /opt/boxen
sudo chown ${USER}:staff /opt/boxen
git clone https://github.com/boxen/our-boxen /opt/boxen/repo
cd /opt/boxen/repo
script/boxen

install iterm2
install coco-emacs or Aquamacs
install zsh brew install zsh
- change default shell to zsh chsh -s /bin/zsh
- verify your running zsh after opening new terminal ps -o comm $$
install Pygments sudo easy_install Pygments #my dotfiles use Pygments
git clone dotfiles, or however you sync your environment
- sync your non git dot files encypted via scp or pgp dropbox sync (I keep .env_extras that contains some sensative keys so it is not stored on github)
configure VPN (if needed)
install Mou (markdown editor)
install campfire client (Propane)
install IM client, Adium
- configure Adium for gchat
brew install qt
- at the time required HEAD brew update; brew install qt --HEAD
gem install thrift -v '0.9.0' -- --with-cppflags='-D_FORTIFY_SOURCE=0'

brew install Mysql55

initialize mysql:

mysql_install_db --verbose --user=`whoami` --basedir="/opt/boxen/homebrew/Cellar/mysql55/5.5.30/" --datadir=/opt/boxen/homebrew/var/mysql55 --tmpdir=/tmp

gem install mysql2 -v '0.2.18' -- --with-mysql-dir=/opt/boxen/homebrew/opt/mysql55/
brew install imagemagick

Get your code

begin to git clone all the things you love
start to rbenv install all the Ruby versions you need
bundle install all the things
verify tests are passing or apps start

Updates Since installed

rbenv to get latest ruby 1.9.3-p484 thanks @obfuscurity

cd /opt/boxen/rbenv/plugins/ruby-build

  git pull; cd
  rbenv install 1.9.3-p484
  rbenv global 1.9.3-p484
  gem install bundler

configure default global node.js nodenv global v0.10.21

Possible Improvements

Should I install emacs starter kit or start over? I need to fix my emacs setup
Automated way to reset all the OS X default apps to open file types?
Automate some of the app downloads and installs
Automate some of the common ruby version installs

Sinatra using respond_to with url extensions (.json)

2013-10-07T00:00:00+00:00

Since I work most of the time in Rails, I found that using the Sinatra responds_to provided in Sinatra-Contrib to be a bit confusing. It didn’t match my expectations mostly in terms of setting content type based on url extensions. It worked fine when curling a url with the proper accepts settings curl -i -H "Accept: application/json" but not when using a browser with the .json extension on the end of the url.

I ended up solving this by setting the request.accept array based matching a optionally appended format on the url. It was also hard to use named url parameters along with a optional format parameter across all the urls I wanted to support. I saw one solution that allowed a optional format on a url, but it required modifying the path of each url. I didn’t want to alter each of my uris with complex regex. I eventually got everything working as I would like, with a bit of hacking.

I added a before filter that checks the current request, if it matches the extension json (although this would work for any format), I set the request.accept headers. I then remove the extension from the path_info so that the urls work with the standard Sinatra route helpers. After adding that hack, the Sinatra-Contrib respond_to works as I expected.

Sinatra optional format before filter

require 'sinatra/contrib'

before /.*/ do
  if request.url.match(/.json$/)
    request.accept.unshift('application/json')
    request.path_info = request.path_info.gsub(/.json$/,'')
  end
end

#...

get '/projects', :provides => [:html, :json] do
  @projects = Project.projects
  respond_to do |format|
    format.json { @projects.to_json }
    format.html { erb :index }
  end
end

Is there a better way?

Perhaps I am missing something, it took me a bit to figure out a good solution to this. Normally I find Sinatra matches my expectations very well and follows the principle of least surprise. That is why I decided to share my solution and see if there was just some other more simple way to solve the problem.

Anyways, using this I was able to make the churn-site api pretty nice to browse over a browser. Especially if you have the JSONview chrome extension which I highly recommend.

JS bookmarklet to switch between environment and shell function to open branch in github

2013-09-24T00:00:00+00:00

I got annoyed manually typing a new host to switch between development, staging, and production environments for some of my projects. After having yet another typo take me off to the wrong place and loosing my train of thought. I decided to whip up a better solution.

Switch environments using bookmarklets

Basically when switching environments I was manually applying a regex replacement in my head. Then I would type out that replacement in a error prone way. Instead let’s just write it once and let the browser take care of it. To do that we create a bookmark prepended with javascript: so the browser will run it. Then manipulate the current document.location to take you where you want to go.

Example url swapping bookmarklets

#bookmark named to_development
javascript:(document.location.href = document.location.href.replace(document.location.host,'localhost:3000'))
	
#bookmark named to_churn_staging
javascript:(document.location.href = document.location.href.replace(document.location.host,'churn-site-staging.heroku.com').replace(':3000',''))
	
#bookmark named to_churn_production
javascript:(document.location.href = document.location.href.replace(document.location.host,'churn.picoappz.com').replace(':3000',''))

This is a really simple solution, which is very handy. Recently, I have found myself building a hand full of tiny custom bookmarklets to help with little tasks. More than JS bookmarklets, you can use tricks like this all over the place.

Beyond bookmarklets, tiny shell helper functions

I have been automating tasks with shell functions as well. The same basic idea as the bookmarklets above of having a tiny function to replace a repetitive error prone task. I think encoding logic like this into a bookmarklet or a shell function is a great way to remember the tricks you learn, and encode the knowledge in a way that is sharable. In fact @tcopeland shared a great shell function with me recently, to quickly jump from the terminal to my branch view in the browser.

function open_branch { open http://github.com/danmayer/\`basename \`pwd\`\`/tree/$(git symbolic-ref head| sed -e 's/.*\///g'); }

The code above will open my browser to github on my current project and the branch I am currently working on.

I use a number of other shell functions to simplify my day to day tasks. I highly recommend building some of your own functions and getting them checked into your dotfiles repo (you do have your dotfiles in a git right?). If you have any great bookmarklets or shell functions definitely let me know by commenting or sending a note to @danmayer on twitter.

update

In response to this post, Andy A. sent me a improved version of the bash open_branch function. Beyond being more flexible it adds some additional functionality like displaying your PRs and latest commit. Check out his improve github helper shell functions

Smokescreen: Smoke tests for Ruby

2013-09-19T00:00:00+00:00

This post is a cross-post with the livingsocial tech blog. Livingsocial was nice enough to help opensource smokescreen so I could blog about it.

There is a staggering number of definitions of smoke testing, and a good subset about how it relates to software development. I think the most common definition related to software can be quoted from the wikipedia article, “A subset of test cases that cover the most important functionality of a component or system.” You can also find some good explanations covering smoke tests on stack overflow.

As our test suite grew we had a need for quickly ‘smoke testing’ a project before deployment. We had a need because it was no longer reasonable to run the full test suite before every small commit. We wanted to run a smart subset of tests before deploy. Our smoke tests wouldn’t cover everything, but it would catch silly mistakes (syntax error, broken config files, etc) and check for catastrophic errors that could impact key actions, like our purchase flow. Obviously this is only necessary for projects that have very large and slow test suites, if your project has a suite that completes in 5 minutes or less congrats!

At the moment running smokescreen prior to deploy is part of our deployment ‘checklist’, but it could be easily integrated directly into our deployment tools. Eventually one could see integrating smoke tests directly into the deployment process to run on the production boxes prior to starting up new unicorn workers. While it doesn’t protect us from every mistake it gives us confidence to deploy as frequently as we do while reducing the risk of making larger mistakes.

There are a number of existing gems related to smoke testing available, smokeit a tool for smoketesting urls, similarly antismoker helps build rake tasks to test urls which can be run as part of CI. Our primary need wasn’t to quickly check endpoints for http 200 success responses, but to get fast feedback on the critical functions of our app. We wanted to always run a minimum set of tests to cover our most important actions, as well as run tests likely to be impacted by the recent changes. We ended up building smokescreen to meet our needs, originally as a rake task and eventually extracting it out to a gem.

Setting up Smokescreen

Setting up smokescreen is fairly easy, include the gem in your Gemfile, and then require it in your primary Rakefile or in any of your lib/tasks/other_rakefiles.

#in Gemfile
group :development, :test do
  gem 'smokescreen'
end

#in a Rakefile
require 'smokescreen'

#below require in rake file
Smokescreen.configure(:critical_tests => ["test/functional/purchases_controller_test.rb", "test/functional/important_controller_test.rb"])

After you require smokescreen make sure to configure the critical tests. Just pass in an array of tests that you want to be be considered part of the critical set of tests.

After configuring Smokescreen

You can view all the tasks together running bundle exec rake -T smoke, as you can tell there are four main groupings of tests current, deploy, previous, and recent.

rake test:smokescreen:critical:functionals                # Run tests for critical:functionals
rake test:smokescreen:current:all                         # run most important tests as well as tests for currently changed files (git diff)
rake test:smokescreen:current:changed                     # run tests related to currently changed files (git diff)
rake test:smokescreen:current:changed_tests               # run currently changed test files (git diff)
rake test:smokescreen:current:functionals                 # Run tests for functionals
rake test:smokescreen:current:functionals:changed_tests   # Run tests for functionals:changed_tests
rake test:smokescreen:current:units                       # Run tests for units
rake test:smokescreen:current:units:changed_tests         # Run tests for units:changed_tests
rake test:smokescreen:deploy:all                          # run most important tests as well as tests for files changed since the last deploy
rake test:smokescreen:deploy:changed                      # run tests related to files changed since last deploy
rake test:smokescreen:deploy:functionals                  # Run tests for functionals
rake test:smokescreen:deploy:units                        # Run tests for units
rake test:smokescreen:previous:all                        # run most important tests as well as tests for previously changed files (last commit, git show)
rake test:smokescreen:previous:changed                    # run tests related to previously changed files (last commit, git show)
rake test:smokescreen:previous:changed_tests              # run previously changed test files (last commit, git show)
rake test:smokescreen:previous:functionals                # Run tests for functionals
rake test:smokescreen:previous:functionals:changed_tests  # Run tests for functionals:changed_tests
rake test:smokescreen:previous:units                      # Run tests for units
rake test:smokescreen:previous:units:changed_tests        # Run tests for units:changed_tests
rake test:smokescreen:recent:all                          # run most important tests as well as recently effected tests (git diff + git show)
rake test:smokescreen:recent:changed                      # run tests for currently changed files (git diff)
rake test:smokescreen:recent:changed_tests                # run recently changed test files (current and previous commit test files)
rake test:smokescreen:recent:functionals                  # Run tests for functionals
rake test:smokescreen:recent:functionals:changed_tests    # Run tests for functionals:changed_tests
rake test:smokescreen:recent:units                        # Run tests for units
rake test:smokescreen:recent:units:changed_tests          # Run tests for units:changed_tests

Running Smokescreen

The most common task I run is rake test:smokescreen:recent:all, as it is the most comprehensive. That task runs the critical tests, test files related to currently change files (as found by git diff), and test files related to recently changed files (as found by git show). It is easy to target more specifically if you choose.

I also find running rake test:smokescreen:previous:changed_tests or rake test:smokescreen:previous:all handy when doing code review on another developers cherry-picked commit. Making sure the tests don’t have one of the “it works on my machine” bugs.

Smokescreen has only been used on a small set of projects, and doesn’t currently support anything other than test:unit. Now that it is extracted out from a single application, I plan to user it in other projects and hopefully it will be easy to make some improvements to the tool. I already know there are easy wins that could be made to improve the process of matching likely related test files to recent changes. Other possible future features are parallel tests support, recently failing tests from CI or previous local test runs, and rspec support.

Finally wanted to mention, as we have improved our test suite, and moved CI to run a massive split of parallel tests, smokescreen has been less important than in the past, but it is still something I run nearly daily and have enjoyed having in my toolbox.

Introducing Churn Site to measure code churn

2013-09-16T00:00:00+00:00

Inspiration for the project

I have been interested in code metrics for awhile. I have built a few dev tools, and in the past tried to build startup based on code metrics. Recently I have been working on various side projects. As part of that, I wanted to put a little more work into the churn gem that I maintain. I decided building a site around code churn would force me to much more familiar as a client/user of the gem. Which would help me find more of the rough edges inspiring me improve the the project. I also thought I could learn some interesting stats by running churn against a larger variety of projects. Finally, I just wanted to play around with the idea of disposable temporary worker servers, which I have been toying around with and it seemed like a great fit for running churn against arbitrary projects.

Check out churn-site now, or read a bit more about the project.

Churn-Site details

I put the app together pretty quickly using Sinatra (using Sinatra-Template), Chart.js, and personal projects that aren’t yet ready for release. The site is fairly simple, it has a list of projects and allows anyone to add projects or trigger churn builds against a project. For each project it collects churn data over time per commit and charts the results over time. To simplify collecting the churn data it supports github webhooks, which make it simple to keep churn up to date.


	churn data for Metric-Fu

At the moment the most interesting data is probably the average churn of a method in your project and the number of methods with churn greater than that. Which I think gives interesting insight to problematic methods at a much more specific level than only looking only at file level churn (which is the standard unit for many churn tools).

I am hoping to add class and method break downs for additional languages, but at the moment Ruby is the only language supported for anything beyond file level data.

If the churn project interests you, there is a shared project tracker with upcoming features queued up.

If you are more interested in the site, feel free to check out the source code to churn-site, although it will likely be a bit hard to contribute to yet as it relies on some of my other yet to be released projects. I think after a bit more cleanup, and adding support for uploading metrics directly from the churn gem, it will be much easier to contribute to the churn-site. If you have ideas for additional graphs of ways to break down the churn data let me know.

I am interested in how the community can make code metrics simpler while still being able to build rich dashboards like Metric-Fu or Code Climate. I reached out to the Metric-Fu Google Group, with some additional details about the project to get some early feedback. From that email thread, @kerrizor was nice enough to send my first PR for the site. Then share some good ideas for future features such as, showing the velocity of churn over time and showing a breakdown of the percentage of commits that involve a file or class.

Fast Prototyping with Sinatra

2013-08-20T00:00:00+00:00

Quickly prototyping apps with Sinatra

When I am quickly playing around with a new app or idea. I often want to be very minimal and fast. Half the time I never even get the app up and running anywhere other than my machine. Sometimes I never get past an index.html with some words and inline JS. I have found that being able to quickly setup a nice basic development environment makes it more likely I will play around with ideas, and explore new projects.

After various approaches I have found that for me, I am very happy to do my fast prototyping in Sinatra. I can quickly get an application or api started with Sinatra, with a bit of help it provides all the basics I need. After a couple projects, I have simple deployment of the projects down for either EC2 or Heroku. I can start up and get into the details of my idea amazingly quickly, which is great. After starting a couple projects with Sinatra I was tired of taking the time to setup some of the basic Sinatra boiler plate, the way I liked. Eventually after refactoring a few projects, I found a Sinatra style and setup that I like. I standardized on gems and configurations that solved common problems that I ran into. I built out a skeleton app to help me get up and running on my projects more quickly. It is simply my personal Sinatra Template

Why use a Sinatra Template (or skeleton)?

Starting a brand new app takes time and much of the initial configuration is the same. This doesn’t mean I don’t fully understand the app, Sinatra Template tiny and it has been abstracted out of a few real projects. I built on it and then improved on my template as I began using it to generate newer apps.

If I choose Sinatra over Rails for a web project, in part I am assuming less front end needs, overall smaller project, and a codebase that requires less structure. If this is more than a prototype, microsite, or expected to grow into a larger application, I recommend just starting with Rails. For that you can take a look at quickly starting Rails apps with Composer, which is another post in this series. So why not just start with a blank Sinatra app. Sinatra is very barebones, my Sinatra Template gives me the basics I expect.

environments (multiple procfiles), server startup (foreman), console (configured Pry), tests, my preferred license, development mode app reloading (shotgun), better_errors in development, flash message support, sessions, logging, admin authorization
A solid starting design, Sinatra Template has Twitter Bootstrap configured and ready to go out of the gate
I am setup to perform most of my basic JS needs via Jquery which is ready to go from the beginning
Having a nice familiar base app as a starting point helps me more quickly get to building application specific pieces of code and get my idea on the page. Using a normal configuration with gems I am familiar with.

Really I am not advocating others to use my Sinatra Template, it is so small and simple that I recommend building your own. Feel free to use mine (or one of the many listed at the bottom of this post) as a example and starting point. Then customize it the way you like, as you built multiple Sinatra apps. Become familiar with your setup, and update it as you improve your Sinatra projects. When I add a new gem or technique to a couple of my projects I back port the features to my original template. Things like the better_errors gem, Pry, and Foreman weren’t originally part of my Sinatra template.

Nothing to development with Sinatra Template

The example below is creating a new application called test_project.

cd ~/projects/
git clone https://github.com/danmayer/sinatra_template.git
cd ~/projects/sinatra_template
rake clone_web project=test_project

This should give you output like

building project: test_project
running mkdir test_project

running cp -R ./sinatra_template/web/ ./test_project/

running cp ./sinatra_template/README.md ./test_project/

running rm -rf .git

running git init
Initialized empty Git repository in /Users/danmayer/projects/test_project/.git/
running rm -rf .rvmrc

running rm -rf tmp/*

running echo "rvm use ruby-1.9.3-p392 --create"  >> .rvmrc

running find ./ -type f -exec sed -i '' 's/SINATRA_TEMPLATE/test_project/g' {} \;

Then you can start to work on your project

cd ~/projects/test_project
#agree to the rvm ruby version
bundle install
foreman start
open http://localhost:3000

#also a single test should exist and pass
bundle exec rake

From development to production

Great we have an app how do we get it out there for the world to see? For personal projects I mostly deploy initially to Heroku and only move to a larger more real production environment if the project really demands it. Getting a basic app to Heroku is easy. Obviously in the instructions below you need to give your app a different name.

heroku apps:create sinatratestproject
git add .
git commit -a -m "initial commit"

git push heroku master
#thats it you should be live
open http://sinatratestproject.herokuapp.com

Sinatra from nothing to production

Following the steps above it how I start most of my Sinatra apps these days. Getting me up and running very quickly so that I can focus on the goals at hand and not worry about some of the initial app creation boilerplate. I have used this on a couple projects now and I am pretty happy with my Sinatra setup. I have continued to patch and evolve my base Sinatra template, and I expect it will continue to improve over time.

I built both blog2ebook and the Code Churn site using Sinatra Template, if you would like to checkout some examples.

Like the idea but not my Sinatra Template, Alternatives

Thoughts on Continuous Deployment

2013-08-04T00:00:00+00:00

Manual Continuous Deployment vs. Automated Continuous Deployment

Recently there has been discussion around continuous deployment where I work. I have always liked the practice. I strongly believe in quick and small changes to production apps leading to the most value for the users and being the safest way to quickly iterate towards a larger goal. However our primary project has always practiced a looser version of continuous deployment, a manual continuous deployment. The manual but continual deploy process suites the current state of our project, while still giving us many of the advantages. I believe there are some times when following strictly automated continuous deployment should be considered dangerous, and I thought it would make sense to explain my concerns a bit.

First let me describe two different continuous deployment processes

Manual Continuous Deployment:

Manually trigger by developers, not automated by CI servers
Commiter / Deployer, follows a checklist of basic deploy guidelines
In general any significant change is deployed nearly immediately after merging to masters, sometimes this means commits go out in clusters or groups

Automated Continuous Deployment:

Automated by CI server, deploys after non failing test suite run
Each commit is tested individually and deployed individually

At first look every developer is probably thinking clearly automated deploy is the way to go. We should remove repetitive processes from the developer flow! Trusting developers with check lists is error prone, it is hard to coordinate deploys. I would agree that in the ideal project with a perfect infrastructure, that is the correct choice. The reality is most large projects have some baggage and state of the project means it might not be the best or a realistic choice for every project.

Issues associated with automated continuous deployment

There are many nice befits of having a fully automated continuous deployment system. I’m skipping over arguments about never automating deploys and moving forward with the assumption, that for most web applications which don’t require high levels of human testing, have large security implications, or have lives at stake that it should be a goal. Even with the goal of automated continuous deployment, there are still some primary problems that can make a project not yet ready for automated continuous deployment. Below is a list that I would consider requirements for automating deployment.

Deploys have no negative impact on system and end users
Fast tests
Fast deploys
A developer, usually the code author should be present during deploy
Easy safe override of continuous deployment
Clear dashboard around the deploy queue, continuous deployment status, manual deployment interruptions
Strong application monitoring and alerting

Deploys have no negative impact

There are still projects out there that have brief downtime during deploys. Others just have negative performance issues for users. Projects might experience a brief blip of exceptions or temporary slow down related to breaking many large caches. All of these impact how often you want to deploy. Until you can safely deploy without having any negative impact you still likely want to have a developer batch up deploys that make sense. Developers should still want to deploy the smallest change possible to effect any of the server and business metrics, but likely don’t need to deploy every test, copy, and minor tweak as it is merged into master. Regardless of automated continuous deploys or not, solving this limitation is highly recommended.

Fast Tests and Fast Deploys

Fast tests and Fast deploys are fairly similar. Both impact the total throughput of commits and deploys possible during a day for automated deploys. If you have a 50 minute test suite and a 10 minute deploy, it effectively will take 1 hour for a commit to hit production. For automated continuous deployment you really want to deploy each commit that passes CI individually. One of the large gains from a system like this is that each commit goes out one at a time so that monitoring and issues can be traced back to the initial deploy that caused a change.

Let’s pretend developers all work in one time zone and only work a 8 hour day (Ha). That would impose a limit that only be 8 individual commits to the project a day, or the team ends up facing a backlog queue of deployments (some of which occur outside of normal working hours). That is a a bad idea, and if this is something a project enforces, quickly developers will ‘hack’ around the rule to do bad things like bundling many unrelated changes per commit just to get it through the deploy queue.

Both Fast Tests and Fast Deploys are very importantly related to the next important issue…

A developer should be present for deploy

Automated tests don’t catch everything. Even if there are no exceptions you might have performance regressions, unexpected behavior, or other issues. Deploying when no one is around and isn’t aware their feature just went live is a bad idea. Preferably the code author and code reviewer, should be present and monitoring systems they expect to be impacted. Occasionally only one of the two, really needs to be there depending on the magnitude of the change.

If you have slow tests or a slow deploy, quickly you will see a problem where 40+ commits are merged on a single day, because you have a big team that knows small safe changes is a best practice. Many of the commits will related to improving tests, developer only tools, minor copy changes, and other simple ‘safe’ changes. These changes are treated the same by a completely automated system and when mixed with larger features, developers end up checking in code and having it deploy a undetermined time later.

Our deployment checklist notes developers should be around after a deployment of their code for around 20 minutes in case any issues arise. In a automated world, with long times between commits and deployment this becomes a very tricky issue. This issue alone is the biggest blocker of moving to automated deployments. Until your tests and deployments are super fast, allowing humans to filter what to deploy when adds tremendous value.

Manual overrides, and dashboards

When a project is running automated continuous deploys, at some point unexpected exceptions will start to occur. Exceptions either from a data change, external api issue, or simple a bad commit automatically deployed. At this point developers should have the ability to react quickly if deemed necessary.

This means possibly doing a rollback deploy of the last commit, or quickly making a change and committing that into the project. Either a rollback or manual commit should be able to be deployed by a developer. It should automatically freeze future continuous deployments. It should make that clear on the applications deployment dashboard. The dashboard should make clear ‘automated deploys disabled because of deploy rollback of commit SHA’, or ‘automated deploys disabled manual override deploy by user X with commit SHA’.

A single one button click on the dashboard should reenable automated continuous deployment, but it should be a smart re-enable, either asking the user which she to begin deploying again with, or immediately moving to HEAD on master and skipping any previous commits queued up. Obviously if someone manual deploys a fix you don’t want to go back and deploy commits prior to it, or redeploy the same commit that user deployed themselves.

Related if there are special tests going on the dashboard should allow one click disabling of continues deploy which could be temporarily necessary while debugging or isolating a single system. These dashboards should also make it clear the expected deploy time for commits currently pushed to the project. Even if there is a fast test / deploy cycle. Developers might not want to push 2 or 3 commits at the end of the work day if there are already 10 commits queued up.

Strong application monitoring and alerting

Really for production applications having good monitoring and alerting in place is important. I think because of the ability for a deploy queue to get backed up and occasionally trigger outside of a developers expectations, that monitoring becomes even more of a importance. Not just on simple things like response time, exception rate, but deeper business metrics like successful logins, purchases, and other important metrics. You will want to clearly see which deploy accidentally effected a metric and trigger alerts as a developer might not be able to as closely watch each individual commit being deployed.

Choosing which is right for your team and project

After considering some of the real world implications of automated continuous deployment. I didn’t feel it was right for our team on most of our projects at the moment. Both because we would need a bit of additional tooling around deployment and dashboards, and because our tests are far to slow. For each project the team should be able to discuss and work towards automating continuous deployment, removing any problems that might be a blocker. Removing any of the blockers will be a big win for improving the development process on the project.

Some projects accepting certain deployment shortcomings might be acceptable. For instance manual overrides, freezes and intelligent restarting of automated deploys isn’t much of an issue for a project with a 20 second test suite. Which will generally finish faster than one could realistically debug any production issue. I’m personally hoping we can successfully move more of our projects towards fully automated systems, but I think it is important to know how it impacts a project and balance the value of the automated system vs the risks for a given project. I hope we can eventually have all our project’s to a place where automated continuous deployment makes sense.

Additional Reading

Etsy has some great processes around deployments. I recommend checking out a couple of their posts / slides.

Non Etsy articles further discussing continuous deployment.

Easily Pushing Static Sites to S3

2013-07-30T00:00:00+00:00

Simple Static Site Hosting on S3

I wanted a simple way to build and host static sites on S3. I already had various apps storing files and pulling data from S3. I figured I could whip up a tiny app to make it easy to build up a static site and deploy it to S3. I decided to go with a simple rake task that can take template ERB files along with some data and merge it with the contents of a public folder and sync that rendered static files over to S3.

The first site I am using this technique on is a collection of some of my little side projects. I called the site PicoAppz and plan on hosting a number of projects there as subdomains, so I don’t have to buy a new domain for each of my little one off projects.

You can see the source code for PicoAppz on Github, the code started incredibly small and has grown as I added automatically pulling the latest screenshots of my apps, rebuilding the site on deploy, and I have started to build a Sinatra based admin app which will let me control some of the meta data.

Static Site Publishing Code

The primary piece of code you would want to look at to grok the static site generation flow would be lib/pico_appz.

Two primary entry points called from rake

  def build
    copy_public_folder
    render_templates_to_tmp
    upload_to_s3
  end

  def preview
    copy_public_folder
    render_templates_to_tmp
    `open ./tmp/index.html`
  end

Then this basically breaks down to these small 3 steps

  def copy_public_folder
    puts 'copying'
    `cp -r ./public/ ./tmp`
  end

  def render_templates_to_tmp
    puts 'building'
    Dir["./lib/views/**/*.erb"].each do |file|
      puts "processing #{file}"
      template = File.read(file)
      rendered_file = Erubis::Eruby.new(template).result({:title => "picoappz", :data => app_data, :gh_data => fetch_github_data})
      output_file = file.gsub(/\.erb/,'').gsub(/\/lib\/views/,"/tmp")
      File.open(output_file, 'w') {|f| f.write(rendered_file) }
    end
  end

  def upload_to_s3
    puts 'uploading'
    Dir["./tmp/**/*.*"].each do |file|
      unless File.directory?(file)
        mimetype = `file --mime-type -b #{file}`.gsub(/\n/,"")
        mimetype = 'text/css' if file.match(/\.css/)
        filename = file.gsub(/\.\/tmp\//,'')
        puts "uploading #{file} to #{filename}"
        write_file(filename, File.read(file), :content_type => mimetype)
      end
    end
  end

The trickiest bit in there is that you need to have the correct mime type to set on files when you upload them to S3. It took me a couple attempts to find something that would be safe to call on either OSX or Ubuntu, but so far this seems to work everywhere I run it.

Setup DNS

I was a little confused after reading some of the information about configuring DNS for S3 sites. Most of the tutorials you will find are pretty good, but some are out of date particularly with respects to hosting root domains without a ‘www’ vs with ‘www’. I was planning on hosting a number of sites on subdomains, and decided I wanted the main site to be with ‘www’ always on.

I use DNSimple to manage my domains, it was easy to set up two manual entries in my DNS to cover the root and www versions of the page.

URL picoappz.com 600 http://www.picoappz.com (This is a alias that will have all root traffic forward to ‘www’)
CNAME www.picoappz.com 600 www.picoappz.com.s3-website-us-east-1.amazonaws.com (This CNAME points at my s3 bucket for the site.)

For AWS S3, you just create a new bucket and configure some settings

Make sure to have ‘Enable website hosting’ turned on
I set my index document to index.html which I upload as the root of the site

Under permissions add a bucket policy similar to

  {
    "Version": "2008-10-17",
    "Statement": [
      {
        "Sid": "AddPerm",
        "Effect": "Allow",
        "Principal": {
          "AWS": "*"
        },
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::www.picoappz.com/*"
      }
    ]
  }

That should be pretty much it. Now I plan on being able to very quickly add and build out other mini-sites and side projects that I can easily host on S3. If you are interested a bit more about how it fetches screenshots and the code that actually handles write_file to upload to S3 check out the source, both are in the lib directory of the project on github. It should be pretty easy to follow, make sure you set all the require ENV variables in your environment.

The Result

[master][~/projects/picoappz] rake build
copying
building
processing ./views/index.html.erb
uploading
uploading ./tmp/404.html to 404.html
uploading ./tmp/css/application.css to css/application.css
uploading ./tmp/css/bootstrap-image-gallery.min.css to css/bootstrap-image-gallery.min.css
uploading ./tmp/css/bootstrap-responsive.css to css/bootstrap-responsive.css
uploading ./tmp/css/bootstrap-responsive.min.css to css/bootstrap-responsive.min.css
uploading ./tmp/css/bootstrap.css to css/bootstrap.css
uploading ./tmp/css/bootstrap.min.css to css/bootstrap.min.css
uploading ./tmp/img/Blog2Ebook-clipped.png to img/Blog2Ebook-clipped.png
uploading ./tmp/img/Blog2Ebook-full.png to img/Blog2Ebook-full.png
uploading ./tmp/img/Blog2Ebook-thumb.png to img/Blog2Ebook-thumb.png
uploading ./tmp/img/blog2ebook.png to img/blog2ebook.png
uploading ./tmp/img/blog2ebook_sm.png to img/blog2ebook_sm.png
uploading ./tmp/img/Churn-clipped.png to img/Churn-clipped.png
uploading ./tmp/img/Churn-full.png to img/Churn-full.png
uploading ./tmp/img/Churn-thumb.png to img/Churn-thumb.png
uploading ./tmp/img/churn.png to img/churn.png
uploading ./tmp/img/churn_sm.png to img/churn_sm.png
uploading ./tmp/img/GitHub-Mark-120px-plus.png to img/GitHub-Mark-120px-plus.png
uploading ./tmp/img/GitHub-Mark-32px.png to img/GitHub-Mark-32px.png
uploading ./tmp/img/GitHub-Mark-64px.png to img/GitHub-Mark-64px.png
uploading ./tmp/img/glyphicons-halflings-white.png to img/glyphicons-halflings-white.png
uploading ./tmp/img/glyphicons-halflings.png to img/glyphicons-halflings.png
uploading ./tmp/img/nothing_calendar-clipped.png to img/nothing_calendar-clipped.png
uploading ./tmp/img/nothing_calendar-full.png to img/nothing_calendar-full.png
uploading ./tmp/img/nothing_calendar-thumb.png to img/nothing_calendar-thumb.png
uploading ./tmp/img/nothing_calendar.jpg to img/nothing_calendar.jpg
uploading ./tmp/img/nothing_calendar.jpg-clipped.png to img/nothing_calendar.jpg-clipped.png
uploading ./tmp/img/nothing_calendar.jpg-full.png to img/nothing_calendar.jpg-full.png
uploading ./tmp/img/nothing_calendar.jpg-thumb.png to img/nothing_calendar.jpg-thumb.png
uploading ./tmp/img/nothing_calendar.pdf to img/nothing_calendar.pdf
uploading ./tmp/img/nothing_calendar.png to img/nothing_calendar.png
uploading ./tmp/img/nothing_calendar_sm.png to img/nothing_calendar_sm.png
uploading ./tmp/img/NothingCalendar-clipped.png to img/NothingCalendar-clipped.png
uploading ./tmp/img/NothingCalendar-full.png to img/NothingCalendar-full.png
uploading ./tmp/img/NothingCalendar-thumb.png to img/NothingCalendar-thumb.png
uploading ./tmp/index.html to index.html
uploading ./tmp/js/application.js to js/application.js
uploading ./tmp/js/bootstrap-image-gallery.min.js to js/bootstrap-image-gallery.min.js
uploading ./tmp/js/bootstrap.js to js/bootstrap.js
uploading ./tmp/js/bootstrap.min.js to js/bootstrap.min.js
uploading ./tmp/js/jquery-2.0.3.min.js to js/jquery-2.0.3.min.js
uploading ./tmp/js/load-image.js to js/load-image.js

Executable Ruby in embeddable widgets and other terrible ideas

2013-07-13T00:00:00+00:00

I want to first say that all that follows is basically a terrible idea, but I enjoy building evil projects once in awhile. If you are looking for a nice solution to running Ruby from the web checkout repl.it, TryRuby, or codepad.

In this post I show examples of how I setup Ruby code that can be executed on my statically generated blog. This is sort of a Ruby follow up to Javascript formatting and highlighting on Jekyll and Github pages. Although, it doesn’t show how to setup your own version as that wouldn’t be very safe or secure.

Once I had javascript code execution working, I really wanted to be able to do the same with Ruby. I could even envision some real world use cases for running Ruby from a page. I happened to be working on another project that seemed like a good fit to making Ruby from JS happen. I am going to start with some examples before getting into how the remote Ruby runner works.

(The first run might take as long as 60 seconds to return a result, more on why below)

Example Ruby code snippet

require 'test/unit'

class Client
  attr_accessor :name, :age
  
  def initialize(name, age)
    self.name = name
    self.age = age
  end

  def can_drink?
    age > 21
  end

end

ClientTest = Class.new  Test::Unit::TestCase do

   def test_can_drink
     client = Client.new('bruce', 20)
     assert_equal false, client.can_drink?
   end

end

Example Ruby code snippet that results in a file artifact

require 'pdfkit'

kit = PDFKit.new('http://www.reddit.com/')
Dir.mkdir('./artifacts') unless File.exists?('./artifacts')
file = kit.to_file('./artifacts/temp_pdf_kit.pdf')

Neat, How’s it work?

The Ruby script running is based on a couple projects that I am not really ready to release. I can explain the basic functions and components, but I am not saying this is a good idea. I also wouldn’t take this as a template for how anyone should try to embed Ruby examples into a blog. I must say that for just running some simple Ruby code this is in fact huge overkill. The pieces of my code running system are built to handle much more than simple script requests, I just realized it would be a simple feature to add to my other project for fun. Here are the steps taken to run Ruby from a static HTML site.

A Javascript library sends Ruby code to my Heroku app called deferred-server.
The deferred-server verifies the code is trusted based on the signature (code can only be signed by valid users of deferred-server)
If the code is trusted the app launches a deferred-server (a sleeping ec2 instances, but could basically be any cloud provider)
On booting the deferred-server starts a instance of server-responder, which handles various requests to a deferred server, either script based or project based.
The deferred-server app sends the script payload to the now running server, hitting the configured server-responder with a package of work to do and where to store the future_result.
The Javascript at this point knows where to look for the future_result and is polling for completed results.
When the JS finds completed results it pulls down the results as a JSON package and can handle them for the given situation (In this case the ruby-runner displays the results and provides a link to any run artifacts).
After X amount of time of inactivity the deferred-server is put back to sleep saving on cost and resources.

That is insane, I want to try it!

Unfortunately, the code for this isn’t really ready for general use. In fact it is quite a mess. I won’t get into why for now, but I can say that I have been playing around with many different development ideas and styles on my side projects and some really didn’t work well at all! I also have yet to come up with a security model that would make this really safe to share.

As mentioned earlier it can take a long time for the first request to respond. This is because I am not just hitting one server that is ready to handle my Ruby request. I am hitting a go between server which actually manages other servers and boots them up on demand. Once the script-responder server is up and running the response time is actually pretty good even with the go between server.

I could modify and combine small pieces of the two projects to allow anyone to easily setup a Heroku instance to execute arbitrary Ruby code. Which might be the best way to release and share this functionality. If there is any interest in being able to quickly and easily embed Ruby example code on blogs, let me know and I can likely whip that up. Just being able to run code on Heroku, wouldn’t satisfy my needs as I wanted to run code which couldn’t be installed onto Heroku systems.

Other examples of running Ruby code

I will post more on deferred-server and server-responder at some point in the future. For now you can read more about the deferred-server script runner in the examples section.

Also, you can check out my first real world usage to run custom Ruby on a static page. It uses a custom JS script on my resume, which runs embedded JS to on the fly build a PDF of my resume (click the PDF link to see it in action, view the page source to see how it works). This has been running ‘in production’ for months without issue. The source for custom Ruby on the page is pretty hilarious.

<div class="ruby-runner-custom" style="display:none;" data-sig="IhFw4kpAHfpiBkbNv0s4ALR8tjc=" data-auto-init="false">
  <pre class="code">
    require 'pdfkit'
    kit = PDFKit.new('http://resume.mayerdan.com/?no_pdf=true')
    file = kit.to_file('./artifacts/dan_mayer_resume.pdf')
  </pre>
</div>
<script src="http://git-hook-responder.herokuapp.com/javascript/code-runner.js" type="text/javascript"></script>
<script>
var addPDFLink = function(currentPluggin) {
  var element = '#formats';
  $('#formats').append(' | <a class="run-button" href="#">PDF</a>');
  $(element).find('.run-button').click(function(e) {
    $(element).find('.run-button').text('generating...');
    currentPluggin.runExample();
        e.preventdefault;
    return false;
  });
};
$('.ruby-runner-custom').codeRunner({'initialize_method' : addPDFLink, 'follow_files' : 'true'});
</script>

I also use the deferred code running functionality in a slightly different way with a Ruby project. It follows the same basic flow, but opposed to JS the Ruby app can make requests to itself running on a deferred server. I use that functionality on my blog2ebook project to perform tasks that aren’t allowed or would take to long on Heroku, like running kindlegen to generate ebooks. This allows a app to serve as a front end on Heroku while doing larger backend tasks on a more robust server.

I will keep playing around with these systems and hopefully be sharing more and putting the to good use soon.

introducing Blog2Ebook

2013-06-20T00:00:00+00:00

Take your Blogs to go

I recently have been working improving my ability to quickly prototype projects from the ground up. I feel like in my day to day work I am frequently working on very large established systems. Where slow steady progress and refactoring are king. Working on smaller apps to try out new ideas and development methods can be a fun and educational habit. I built this app initially very quickly to solve a problem of my own. I have then expanded it a bit to build out out a app that might be useful for some to some other people.

The result is a tiny app, Blog2Ebook

What does it do?

Blog2Ebook supports importing from 3 different formats to make a Kindle book:

RSS feed to Kindle
Single post to Kindle (useful because some https sites no longer allow bookmarklets)
Copied article text to Kindle

You might be thinking there are tons of solutions for this. Like Readability and Amazon’s send to Kindle. True but each seem to have some problems and is missing something I want.

Nothing seems to fetch & format a full blog the way I want.
Readability is mostly setup for single articles as a bookmarklet, but the newer JS restrictions on some https sites like github prevent the bookmarklet from working
Amazon allows you to email files to the Kindle, but sometimes you just want to paste some text. You have to create and attach a text file to the email opposed to just having the email body be the content.

So I built a little app and learned a ton about eBook formatting, publishing, and various eBook formats. I think I might want to self publish something in the future and this has prepared me with some knowledge of the self publishing problems and process.

Supporting very simple eBook publishing of a full blog with all content is the primary goal of the project. The other features are secondary and basically around because they were easy to add on and solved some problems. At this point I am starting to look a bit more into how to best format the content and embed related content into the books.

App details and Resources

You can find the project Blog2Book, convert an RSS feed to Kindle eBook, hosted on Heroku.

The source code is available on github, as Blog2Ebook is open source

If you have thoughts, problems, or ideas send feedback to issues for the project.

Technologies Used

I will likely post a bit more about the development process and specific examples of tech usage in another post, but I wanted to provide some details on the tech supporting the project.

Built using Sinatra Template, a personal project I maintain to quickly bootstrap small Sinatra projects. I will describe it more in a future post, but it makes it easy to get up and running with a web or API project using Sinatra and the gems and features I am experienced using.
Single Article Publishing, document content parsing
- Readability’s Api to better support single article publishing
- I also checked out Pismo and Ruby Readability which is a Port of arc90’s readability project to ruby. In the end single article support most often uses Readability’s API as I found it gave the best results.
Redis Cloud by Garantia Data A awesome cloud redis provider
Mailgun to power email, which is the kindle delivery mechanism.
RSS feed processing was done with rest-client and nokogiri
Amazon’s KindleGen to package up HTML content into .mobi files. Which allows packaging images in with the content as well as embedding book meta data. (mobi support with images is on a work in progress branch on not live yet)

Because this is using the free tier of some of the supporting services, I limited daily usage of the app to below the free tier limits. So please be kind to the service, if it starts to run out of capacity I will take a look at expanding it out.

Quick Application Diagrams with Yard

2013-06-02T00:00:00+00:00

Thanks to following awesome people on twitter I learned something I didn’t know about yard. That is can be used to create nice application diagrams. I saw a tweet from Matt Aimonetti, about generating diagrams with yard. I was interested in trying it out myself. After quickly installing graphviz which I apparently no longer had on my machine, I was rewarded with some nice diagrams for my rails apps. It also works well for gems as you can see from the churn graph below. Unfortunately it doesn’t seem to work completely out of the box on Sinatra apps, so I might need to learn a bit more configuration.

brew install graphviz
gem install yard
yard -n
yard graph --dependencies | dot -Tpdf -o app_diagram.pdf

Above was all that was needed to get rails app diagrams like those below.

Churn

Small Rails apps, Hearno

and finally NothingCalendar2

fast start rails with composer

2013-05-27T00:00:00+00:00

Quickly getting started with Rails 3.2

This post explains how I quickly get an application started for Rails 3.2.x.The post is part of a series covering quickly prototyping projects.

When I start a new rails project, I want to quickly get going with a login system, admin and other user roles, a nice default app template / style, and a JS framework I can build on. After looking at several options I ended up going with Rails Composer. It has a number of packages that work out of the box. It is very configurable, it is very easy to get up and running. Railsapps used to have a lot of free tutorials coving using rails-composer, now they sell detailed tutorials to support their work. You can still generally just read the read me for a project and go through the basic options to get everything working. I generally follow the Rails App for Devise with CanCan and Twitter Bootstrap example for my basic app base and further configure it myself from there.

Why use Rails composer?

Starting a brand new app takes time and a lot of the initial configuration is very much the same. The composer options I choose are well understood common Rails gems I would often choose for my initial setup in the first place. I understand all the pieces that I setup with Rails composer so I don’t feel like I am giving up any understanding by starting with the generated app. I would recommend making sure you ‘could’ setup any of the pieces Rails composer configures for you, and understanding them before using them in a generated project. If I am programming in my personal time I often have a problem I want to solve or a specific idea I want to test, getting to the point I can work on the primary focus with little lost has a big value to me.

If I choose Rails over Sinatra for a web project, in part I am assuming I will have higher front end needs for the project. Which likely means I will have accounts, logins, roles / rights, and want a more polished look and feel. Out of the box Rails requires you to setup a lot just to meet those basic needs. With Rails composer, I am basically good to go.

A solid starting design, Rails composer will give me Twitter Bootstrap configured and ready to go out of the gate
I am setup to perform most of my basic JS needs via Jquery which is ready to go from the beginning
Accounts, roles, and rights are covered by : Devise, Cancan, and Rolify. Quickly covering my basic account management needs.
The composer setups up a great way to share application specific configuration that isn’t checked into the code repo via Figaro. Which also supposed publishing the environment to Heroku my preferred production deployment for personal projects.
Having a nice familiar base app as a starting point helps me more quickly get to building application specific pieces of code and get my idea out there. Using a normal configuration with gems I am familiar with.
Stand on the shoulders of giants, I don’t want to spend my time building yet another authentication system, building in features like forgot password, email confirmation, etc.

Nothing to development with Rails composer

The example below is creating a new application called nothingcalendar2, which is a new version of a app I created awhile ago called nothing calendar to try out some JS ideas. Now I want to convert it to a modern rails application and add some new features.

rvm use ruby-1.9.3

#this will ask you some configuration options
rails new nothingcalendar2 -m https://raw.github.com/RailsApps/rails-composer/master/composer.rb

cd nothingcalendar2
bunlde install --local
bundle exec rake db:create
bundle exec rake db:migrate

#set the data to match your accounts / info
open config/application.yml

#this will use the data set in the last step to create accounts in the dev DB
bundle exec rake db:seed

# You have a app
rails s
open http://localhost:3000/

# check the tests
bundle exec rake

#create a new repo
https://github.com/new
git remote add origin git@github.com:danmayer/nothingcalendar2.git
git push -u origin master

From development to production

Great we have a app how do we get it out there for the world to see? For personal projects I mostly deploy initially to Heroku and only move to a larger more real production environment if the project really demands it. Getting a basic app to Heroku is easy. Obviously in the instructions below you need to name your app something else.

heroku apps:create nothingcalendar2

#edit Gemfile
#add the group option to sqlite3 as you don't want it to install on heroku
#set group to development and test
#you could use postgres locally since that is what you are using on heroku in production. That is the recommended setup but for a toy project many people don't have postgres locally
gem 'sqlite3', :group => [:development, :test]
#add postgres to Gemfile only for production group
gem 'pg', :group => [:production]
bundle install

git push heroku master
heroku run rake db:migrate
#push the settings in your config/application.yml to heroku
bundle exec rake figaro:heroku
#seed the database with admin account
heroku run rake db:seed

#you should be live
open http://nothingcalendar2.herokuapp.com/

Rails from nothing to production

Following the steps above it how I start most of my rails apps these days. Getting me up and running very quickly so that I can focus on the goals at hand and not worry about some of the initial details of having a basic infrastructure to build on. I have used this on a couple projects now and I am pretty happy with the process. I will likely continue to use this to quickly test out ideas, although for many simple progress I prefer the even lighter weight Sinatra setup, but that is for another post.

I think this is a great way to get going. Not everyone agrees with me for the opposing view checkout, Why we don’t use a Rails template by thunderboltlabs.

Like the idea but not Rails-Composer, Alternatives

Performance Issues and Bug Clusters

2013-05-08T00:00:00+00:00

There is common advice in programming that bugs tend to cluster in the code. I have always found that true. In part because if there is a hard problem without a clean solution it is easier to make a mistake. Another reasons is one bug results in one hack with a bad fix adding it’s own bug. Creating bugs all the way down. I think devs in general know about the issue of bugs clustering, but I don’t think everyone knows what to do about it or that it doesn’t only apply to bugs.

OK, bugs cluster now what?

Depending on the on the seriousness of the bug, likelihood of a one off mistake or potential cluster, there are a couple of different things that might be worth doing:

For any bug it is worth at least doing a brief audit to see if there are similar mistakes in nearby code.
- Did this class use the same variable with the same mistake?
- Was this method accidentally called with the wrong object anywhere else
- Was this a data error, logic mistake, typo, misunderstanding? (How likely is this to be a one off bug vs. a cluster)
- Do you remember debugging something like this in a related area fairly recently, If so likely worth digging in more.
- A quick check can find and save many related bugs from popping up, over and over.
Was this a Really serious bug?
- Audit all related code. Looking at all calling code and everything relying on results from the buggy section of code.
- Did you just find a bug in authentication, encryption, payment services, etc… You better believe you should have one or two devs look for similar errors.
- Where else in the code is there anything similar to this that could be impacted
- Why didn’t a test catch this bug?
- After the bug is fixed was it serious enough there needs to be a follow up post-mortem, meeting, log audits, discussions with the original code author(s)?
Hmmm this was just a tiny bug but there is similar code all over the place.
- Did you just fix the same bug 3 places in arrow? If so time to Isolate buggy issue.
- If there are many related pieces of code dealing with the same issue often in slightly different ways, it is extremely likely there is a bug.
- It also means that if you hack something to fix the bug, you likely only fixed one manifestation of the bug, or by fixing the known case created a new unknown bug.
Is this bug to big of problem to fix it all right now? Fix the initial known bug. Then in a rare time where comments are better than code. Comment on the hack pointing to related past bugs and potential other bugs. If you have even built up the mental model of a solution or two perhaps leave that information as well. We all have had to hack around some bug just to make it through the day, but if you come back a second time to fix a bug and find a comment like that, you have some refactoring in your future, dig in and get to it.
If you wrote the code and just authored a bug fix for a issue, perhaps ask another dev to give a quick code review of the code, class, or method. Fresh eyes can do wonders to illuminate a subtle deeper issue.
Likely a obvious one, but write a test that fails then fix the bug… Exceptions on production you say! Fine fix, deploy, then locally revert the fix and get that test around the error case.
A much harder thing to fix but good to understand. Was the bug cluster caused by a bad data model. Code that isn’t representative of the problem and task at hand? If so there might be a larger discussion or refactoring to start to consider.

Clusters, it’s not just for bugs

Code patterns are everywhere: styles, library usage, thoughts, developers, late night bad hacking, good day, bad day, new shiny obsession, dead feature, big O blindness, N+1, slow methods, service abuse…

Pretty much any pattern the human mind can recognize can and will cluster in code. If you are trying to fix up, undo, remove, or improve code it helps to understand a bit of context. Just a little extra context can explain why the code exists the way it does. Git blame your lines, methods, and features to learn some possible historical context. See what other code was added or modified around the same time. I find that helpfully when sleuthing for all other possible related bugs… I actually find more often that this sort of ‘code cluster’ is more useful when removing a feature or doing performance work. It is good to know that code pattern form clusters like this in general but, I have found that following up on the patterns particularly for performance issues can have huge payoffs. If you find a simple but large performance win, dig into the code cluster finding all related code.

I was working on performance problems for a specific Rails action. I found that it was doing the most naive and simple thing possible, which is great and in the past never really cause an issue. As the data size changed it was blatantly obvious that it was making hundreds of unnecessary calls per request and reducing the result set down for display to the first 5. Basically get_massive_result_collection[0…5], Ok simple fix, added ability to pass a limit get_massive_result_collection(:limit => 5). I realized that when the collection method was first added to the code, it’s usage had been spread all through out the app. I got the important performance fix deployed to production and saw a order of magnitude difference proving this was a pretty valuable discovery. I immediately created a ticket to follow up on all calls to the existing collection method in the app and find how many were abusing it. Over the next week or two I was able to knock incredible amounts of time of nearly every page load as the method was being used in application layouts, common partials, and spread since it was originally implemented. The code changes where each simple after tracking down the initial performance mistake. I personally will always spend a bit of time spelunking the code now to follow up on related code after discovering a large performance win.

Humans are pattern recognition machines

Seriously we are are pattern recognition machines, get good at it and use it! Keep a eye out for the patterns in your code. Understand that it can help you with bug fixing, refactoring, performance, dead code removal, and more. No piece of code is written in complete isolation, if it was it really wouldn’t be able to do much. So consider the code clusters the surround your changes, and can help you prevent a bug that has gone undetected.

Repetitive Hash Syntax, My quest for a Self Referring Hash

2013-03-03T00:00:00+00:00

For years a tiny code style issue has bothered me in Ruby. I find myself frequently building up a set of variables and then building a Hash where the keys all match the variables. It is a small nit-pick, and never wastes much time, but it has always annoyed me. The wasted duplication to have {:some_key => some_key} for a long list of variables, when the key and the variable are named the same. Convention over configuration and all that. Ruby is a beautiful and flexible language. Why can’t we make it simpler and cleaner to create a Hash based of some locally defined variables. It turns out more complicated than one would think, because of trying to bind to the current variable scope.

In the end what I wanted was a way to create a hash that would looks something like this, which I like to call a SelfReferringHash

    calculation = 7
    word        = "a word"
    data        = "more data"
    
    {:calculation <=, :word <=, :data <=}
    #in this new fake syntax I imaged '<=' to mean points to self

    #or perhaps being able to call a method on a array
    [:calculation, :word, :data].bind_into_hash

    #or perhaps something a bit more normal
    Hash.build_from_vars([:calculation, :word, :data])
    
    #all methods would have the same result of a hash like below
    {:calculation => 7, :word => "a word", :data => "more data"}

###Inspiring a solution

I mentioned this annoyance years ago to Ara Howard, and he whipped up a example very quickly, but it didn’t feel quite right. Ara proposed a few iterations with various syntax’s :result.to_h{} or [:a, :b, :c].to_h{} but the syntax was distracting. So I kind of forgot about the issue and forgot about the issue for awhile.

I then recently came across this post Playing Around with Ruby Hashes. Where Alex wants to add various features such as setting a arbitrarily deep value on the Hash without all the intermediate checks and creation of empty hashes

    hash[player][:stats][category][statistic] = value
    
    #without littering my code with a bunch of lines like

    hash[player] ||= {:stats => {}}
    hash[player][:stats][category] ||= {}

The post build some fun features onto Ruby’s Hash and the post inspired me to play around with building a cleaner solution.

###Attempting a solution

While one of Ara’s symbol to hash solutions was pretty close to what I wanted, it Monkey patched the Array and and Symbol I wanted to avoid that. I decided that I should make a new object and subclass Hash opposed to tacking my code onto any existing objects in the system.

    class SuperHash < Hash

      def initialize(*args, &context)
        if args && args.first.is_a?(Array) && context
          super()
          hash = SuperHash.from_array(args.first, &context)
          hash.each_pair{|key, value| self[key]=  value}
        elsif args && args.first.is_a?(Array) && context.nil?
          raise "requires context block for binding"
        else
          super()
        end
      end

      def self.from_array(array, &context)
        raise "requires context block for binding" if context.nil?
        array.flatten.inject(SuperHash.new){|h, val| h.update({val => (eval(val.to_s, context))}) }
      end

    end

    calculation = 7
    word        = "a word"
    data        = "more data"

    my_hash = SuperHash.from_array([:calculation, :word, :data]){}
    puts my_hash.class
    puts my_hash.inspect

    my_hash = SuperHash.new([:calculation, :word, :data]){}
    puts my_hash.class
    puts my_hash.inspect

    begin
      my_hash = SuperHash.new([:calculation, :word, :data])
      puts my_hash.class
      puts my_hash.inspect
    rescue => err
      puts err
    end

    my_hash = SuperHash.new()
    puts my_hash.class
    puts my_hash.inspect

Running that code results in the following output, with the SuperHash.new([:calculation, :word, :data]){} ending up being the ‘cleanest’ way to invoke the code.

    SuperHash
    {:data=>"more data", :calculation=>7, :word=>"a word"}
    SuperHash
    {:data=>"more data", :calculation=>7, :word=>"a word"}
    requires context block for binding
    SuperHash
    {}

In the end, this might be interesting if I had some other things to patch onto the Hash object. Unfortunately it seems like this still isn’t really that useful and it is a bit to awkward. I could never avoid the primary issue I had with Ara’s examples of having to awkwardly pass the context. All solutions seem to require you to pass a empty block to be able to bind to the context and access the variables at that time. Having a initializer or a class method which requires {} just seems to awkward to be useable. So after all this time I am still left feeling, unsatisfied. While this is a very simple problem to understand and Ruby is a very flexible language. The issues of binding to a context and variable scoping makes building a more beautiful and usable version of a self referring Hash out of my reach.

If anyone has suggestions or ideas on how to build something that is a bit cleaner or closer to my goal and doesn’t resort to passing a empty block, I would love to see any other solutions. Perhaps, I am missing some more interesting way to bend Ruby to fit my mind.

Setting mime-type / content-type when uploading files to S3

2013-02-17T00:00:00+00:00

I was having a issue where after uploading a file to S3, it wouldn’t allow any user to view the file with a browser, but only download the file. The issue turned out to be that S3 defaults to binary/octet-stream as the content-type. I realized, I needed to set the content-type / mime-type when uploading files to preserve their mime-type and how they are treated in the browser for the user.

I am using the Fog.io library to upload files and manage EC2 servers. It turns out that the ruby std lib File object doesn’t have any methods to help with mime type. While there are many gems to help with the task I didn’t really want to add another gem dependency or me around with what seem to be a large list of not recently maintained gems. After some quick google-fu, I found a solution that would work well enough on any unix like system. I just shelled out and called file -Ib and parsed it to get a mime type.

   directory = connection.directories.create(
                                              :key    => "a-bucket-name",
                                              :public => true
                                              )
   filename = './somefile.txt'
   content = File.read(filename)
   mimetype = `file -Ib #{filename}`.gsub(/\n/,"")
   file_options = {
      :key    => filename,
      :body   => content,
      :content_type => mime_type,
      :public => true
    }
    file = directory.files.new(file_options)
    file.save

After preserving and uploading correct mime types, the files worked as expected. If a user clicks a txt file they see the contents in browser, same with images, etc. Anyways, that solved the issue for me. I am kind of surprised S3 doesn’t have some sort of auto-detect mime type option.

Jekyll JS code formatting and running

2012-12-15T00:00:00+00:00

This post is an example of how to setup JS code formatting for Jekyll on Github pages along with a ability to allow users to execute the example JS code.

example JS code snippet:

alert('hello');
function adder(a,b){
 return a + b;
};
adder(5,8);

This is just for fun and should make it a bit easier to blog about some JS code examples in the future. It also allowed me to play around with some JS code to build a simple Jquery plugin for my blog.

Instructions

To make this work on github pages for your own blog basically follow the Pygments for Jekyll instructions. Remember to link the generated Pygments css file in your layout file. Then I just hacked together a really quick JQuery plugin which I include on my Jekyll layout template, starting from Jquery.exampleRunner, I modified the plugin to find any div with class js-runner and insert a run button which will include the results. You can view the source by checking this blogs source, or view my version of jquery.exampleRunner source directly.

First Responder, helping large development teams coordinate and work together

2012-11-18T00:00:00+00:00

The team I am working on has been responsible for maintaining a large central code base. This code base is a older code base that spans to many responsibilities. It’s stability and performance is important, because of this central roll. It is a point that many other teams need to interact with. Any downtime or large exceptions can have a significant impact on the company or many other teams relying on this project as a API.

As the number of developers and teams within our company has grown so has how we manage code going into this central project. We began requiring the majority of code going into the project to have code reviews. We also went from only having developers who had a lot of experience with the central project to many developers who only touch it a hand full of times and work on other projects for their day to day development. This generally meant when a developer from another project needed a small change they needed some help finding their way around the project and to understand the context where they would be making changes.

The changes and growth lead the team I am on to being overwhelmed with requests to review code, answer questions, and help other teams integrate changes into the project. At first we tried to just deal with this ad hoc as situations arose. While this sort of worked for awhile, it lead to a number of distractions and interruptions for all the developers on our team. It also made it hard to plan and schedule our own features as we didn’t know when or which one of our developers might get pulled of task to help another team work with the codebase. We needed a better way to work with other developers and teams, and a way to help review the constant flow of new code being pushed into the project. Code needed to be reviewed often coming in from all the other devs on our team freed up to focus on getting things done.

For me personally the constant flow of interruptions was endangering my ability to have and longer period of focus, or get any significant work done because of the interruption tax.

interruptions are the enemy of work. They are the enemy of productivity, they are the enemy of creativity, they are the enemy of everything.

–Jason Fried of 37Signals

Introducing a first responder

We decided to schedule one person from our team a week to do nothing but help other developers get code into our project. The first responder (on-call, bug catcher, or on-duty) developer does the majority of the code review during their week. They take emails coming in for live bugs and if it is something tiny they fix it immediately or determine if another developer on the team needs to immediately focus on that issue. They help PM get estimates on upcoming work to at least ballpark if it is worth discussing. They look over other teams proposed features, branches, and Pull Requests to varying degrees. Sometimes all that is needed is a code review of the final changes, other times having a quick meeting with a team to discuss the requirements of their feature and what changes in the central project would need to occur to support their feature. Occasionally the first-responder will end up pair programming with another developer to help share knowledge of the codebase and help them integrate their changes.

We rotate who is the first responder on the team each week. It can actually be pretty hectic and a bit scary. Some days you will get slammed with so many incoming requests bouncing quickly from one idea to another, and all the context switching will wear you out. Other days, you get to learn a lot about a part of the code base that you might not have been as familiar with previously, or with another team’s work which is new to you. It is definitely a challenging week that requires a different set of skills than our normal development duties.

This has freed up the rest of our team’s developers to better focus on the features and work that we had planned. It makes it less likely a unexpected hiccup will knock all our devs off schedule. It has also reduced the tiny little interruptions that destroy developers ability to focus and do real development.

When our team introduced the first responder, we emailed all developers at the company. We shared our plan for a first responder and a email address to get in touch with our team’s first responder. The email went to the first responder who is on duty that week, but also allows the rest of our team to have a high level of what is going on by lurking on email threads. We let teams know that if there is a more immediate need they can jump in our campfire room and ask questions where anyone from the team can help respond, and we always update the room info to let people know who is the first responder this week. We asked that people try email and campfire channels prior to falling back to IMing a developer directly. We also added all the information to our projects README, so that new developers joining a team can find out quickly how to best interact with the central project.

We aren’t the only team at our company that has been creating the first responder position, in fact we weren’t even the first. Other teams naturally evolved similar positions. Since we have been using the first responder on our team for awhile now. I think I can safely say it has been a big improvement. I have been happy to see more teams adopting similar rotation roles as well, as it has been great to have someone as a point of contact when trying to work with a project when you less familiar with it.

While the satisfaction of being the first responder is different than normal tasks, helping someone ship a big new feature for their team can be very rewarding. So can mentoring more junior developers to help them solve a problem in a better way. There is also a certain joy of helping review and ship 10 - 15 smaller deployments to production, getting a bit in the code review zone of a review deployment / cycle. I know that I believe it has helped increase the overall quality of the code getting to the production system. It has also helped us to streamline and more quickly process a rapid flow of pull requests without slowing down the development pace as much (So has standardizing and sharing a set of expectations for pull requests for our project, but more on that in another post). I don’t like adding to much process to the development flow, but I think the right amount of process helps keep healthy balance between Stables and Volatiles, ensuring they can work together while trying to get some of the benefits of each group. If your team seems to be frequently knocked off task by outside requests or facing a growing backlog of necessary code reviews, I highly recommend implementing the team first responder.

Additional Sources

A Protocol for Code Reviews
reasons interruptions hurt productivity
37signals Getting Real chapter, Alone Time
Large collection of sources on StackOverflow about about developer interruptions
Our first responder contact methods (email, group chat, IM) in part help people to follow Tactics for avoiding colleague interruptions

Unsolved First Responder Issue

A developer on my team pointed out one problem we have been having with the first responder rotation. We work on a distributed team, so we have rotations that involve developers in different timezones. If the first responder is on the west coast and a early morning issue occurs someone needs to respond. Currently a east coaster will drop what they are doing to step in to the first responder role to help with whatever live bug fighting is going on. We don’t really have a good solution for this issue yet.

ratio of bugs per line of code

2012-11-11T00:00:00+00:00

The more development I do the more I feel like increased Lines Of Code (LOC), nearly always results in increased bugs.

I know that seems obvious at first, but hear me out, as many refactorings, abstractions, ‘cleaner’ code increases the LOC.

I have been reviewing a lot of code lately and the larger the added LOC are the more concerned I am about being able to fully understand and review the code. Even when I fully understand the purpose of the new code, I know lurking in each line could be a subtle but important bug. Seems obvious at first, but shouldn’t the overall complexity of the change and level of difficulty of the code be a large part of the equation as well? Most of the time the answer is not really. Reviewing new code going into a project each line introduces a new place for a bug to exist, even when each individual line is incredibly simple. So reviewing a ‘simple’ code change of 600 LOC is still often times far more risky than a complex change of 100 LOC (^1).

My team has been working on refactoring and cleaning up a large older code base. One of the biggest issues now is just the sheer volume of unused, near duplicated, or overly abstracted code. The cognitive load associated with understanding all the implications of a change, and who might be relying on a specific quirk in a piece of existing code.

This made me think about many of the great developers I know and how they try to avoid creating large code bases. Which is why I think eventually many great developers start to really favor the most succinct code to accomplish their task. Often it is slightly less readable (chaining injects, transformations, and reductions to a collection in one line), but it leaves such a small surface area for bugs to linger. Which is why I think so many incredibly smart engineers who enjoy the beauty of code move from writing verbose ‘readable’ explicit code with big objects and reusable modules, to condense and succinct but powerful code. Often favoring simpler objects, relationships, and data structures.

I think the best example of developer trajectory of verbosity to conciseness comes from Steve Yegge, who gives a outrageous example of beginner code compared to expert code, to illustrate his point in Portrait of a N00b. While he admits that example is a bit of a joke, he does seem to have some support for my premise.

I happen to hold a hard-won minority opinion about code bases. In particular I believe, quite staunchly I might add, that the worst thing that can happen to a code base is size.

– Yegge in codes worst enemy

In fact many developers talk about how LOC is the enemy of software projects. The linked post references 37 Signals (several times), Seth Godin, and heavily quotes Steve McConnell all supporting how much more difficult and less time efficient it is to manage large code bases.

I think in part the reason large codebases are such a problem is that as the code grows, so does the number of existing bugs in the system. The ratio of bugs per lines of code is pretty constant for a project based on the methods the team has followed while adding features and code to the project. Many of the articles I am linking to, point out the the cost of adding code doesn’t grow linerly but get’s orders of magnitude worse as the code base grows.

Bug to code ratios

The idea of bugs per lines of code isn’t really a new idea. Steve McConnell, the primary source for the previously mentioned post, has written extensively on defects per lines of code. Covering average bugs per LOC stats in his great book Code Complete

(a) Industry Average: "about 15 - 50 errors per 1000 lines of delivered
code." He further says this is usually representative of code that has some
level of structured programming behind it, but probably includes a mix of
coding techniques.

(b) Microsoft Applications: "about 10 - 20 defects per 1000 lines of code
during in-house testing, and 0.5 defect per KLOC (KLOC IS CALLED AS 1000 lines of code) in released
product (Moore 1992)." He attributes this to a combination of code-reading
techniques and independent testing (discussed further in another chapter of
his book).

(c) "Harlan Mills pioneered 'cleanroom development', a technique that has
been able to achieve rates as low as 3 defects per 1000 lines of code during
in-house testing and 0.1 defect per 1000 lines of code in released product
(Cobb and Mills 1990). A few projects - for example, the space-shuttle
software - have achieved a level of 0 defects in 500,000 lines of code using
a system of format development methods, peer reviews, and statistical
testing."

Going further McConnell talks about the value of defect tracking. As well as the speed of writing quality software in relationship to the defect rate, and bugs related to a team’s software development process maturity.

I think McConnell places too high of cost on bugs, at least for modern agile web development. He is mostly discussing the issues in the context of shipped (Microsoft, NASA, and defense), opposed continuously deployed web apps where most bugs can be solved quickly and the mean time to recovery matters more than have a extremely low bug rate. (yes +1 for Facebook’s Move Fast and Break Things). I do agree with McConnell that more bugs in the software will also slow the speed of development, as it increases the cognitive load.

Working on a large project with many developers of varying skill, I often push for easily readable code. I think it is important that anyone on the team can work on the code. When more easily readable code starts to increase total code size, these two ideas are at odds. That dichotomy is what brought all of this to the front of my mind. I have been increasingly been hesitant to believe a refactoring just for the sake of slight readability at the cost of increased lines of code is a good thing.

I do think heavily functional and succinct code is far better for back end systems and complex functions and less likely to cause maintainability problems. So while I might slowly be writing more chained and dense code, I don’t think it works everywhere. I have yet to see how that kind of succinct ‘clean’ code can handle user input exceptions, and highly conditional view layers. I think part of it might be why experienced developers like to work on frameworks and more isolated services where one can produce and enforce such ‘pretty’ code.

Final Thoughts

Keep your code tiny. Fight extra complexity and lines of code and strike down upon it with great vengeance & furious anger…

–not quite what was said in Pulp Fiction

I think bugs to LOC ratio is one of the main reasons having a large codebase is bad. I believe the cognitive load of having to write code that functions around all the bugs/quirks in a code base is why progress slows so much on larger code bases. I think large codebases being bad is one of the reasons why SOA, Micro-Service Architecture, Heroku’s Lightweight Web Services, and Github’s building mini-apps is the solution all larger apps end up moving towards.

While I still place a incredibly high value on the readability of code, and often find a Collection#each accumulating into another variable more readable than Collection#inject. I think developers should error on the side of less code, unless there are clear improvements to having more code. Especially when trying to refactor to improve the readability of the code.

Based on this I feel like refactorings that are introducing more lines of code for the sake of readability are often just moving the complexity around. It often feels like a win, but it has a good probability to introduce more bugs. In some cases is actually only more clear to the refactoring author.

Related to refactoring making code larger, while small testable methods are great. If each method just is another layer of abstraction deferring to yet another method before the meat of the problem is solved, it is less likely to be a improvement. Abstractions have a cognitive load and each leaky abstraction layer is another potential buggy line of code. Design patterns can be great but knowing when they are overkill takes time.

I am not saying all refactoring is bad, in fact I think bug LOC ratios increase the importance of some refactorings. A refactoring that doesn’t make the code easier to read (and likely not any more difficult) but delivers the same functionality while reducing the total LOC is a big win. A win in terms of maintainability, it reduces the far too frequent bugs where multiple files/functions need to be ‘kept in sync’ to ensure correct functionality.

One failing in my keep the project as small as possible idea, is user input and UX. All those dedicated to succinct code, please show me great examples of user input error handling and complex conditional view layers. I still haven’t seen good ways to approach these issues short of writing more code and trying to make all of the conditions as clear and small as possible. (Often products made for developers solve this by only allowing for tiny number of possible conditions. I don’t think that approach works in reality when working with designers and UX teams, who often have clear use cases for each conditional)

Additional Sources

(1) Obviously different bugs have different costs, and critical path bugs are much worse than minor presentational bugs. I am ignoring this issue to get to the larger point.

Bitnami EC2 https and environment variables setup"

2012-10-28T00:00:00+00:00

I have a project where I am using the Bitnami ruby stack. I used bitnami, because I wanted to avoid having to deal with a bunch of server setup just to play around with a box. It has worked well, but I have had some stumbling blocks, originally I had trouble installing nokogiri on Bitnami boxes, after getting past that I still needed to search around for solutions to setting up environment variables on Bitnami boxes. Then finally I needed to look up configuring https for my new box so I could pass secure API keys.

Addining environment variables on Bitnami servers

To add Bitnami environment variables for the Bitnami user (the default login user), just update the standard users .bashrc located at /home/bitnami/.bashrc

To add environment variables for web processes and servers started by Bitnami’s default boot up, add variables to /opt/bitnami/scripts/setenv.sh

Add environment variables the standard way to either or both configs

export SOME_ENV_VAR='accessible to servers'
export ANOTHER_VAR='still here'

In my case I am using a Sinatra server behind apache2 served by passenger, so I can then just access the variables with ENV['SOME_ENV_VAR'].

Setting up Apache2 / Passenger for https on Bitnami stack

Since I was passing some API keys over to my Sinatra server, I wanted to make sure I could hit the API via https. It turns out Bitnami has pretty good docs on setting up a Bitnami Apache2 server with http enabled. For me basically just the few steps enabled https

emacs /opt/bitnami/apache2/conf/httpd.conf
#uncomment LoadModule ssl_module modules/mod_ssl.so
#uncomment Include conf/extra/httpd-ssl.conf
emacs /opt/bitnami/apache2/conf/extra/httpd-ssl.conf
# update the default virtual host for https traffic listening on 442

<VirtualHost _default_:443>
  ServerName www.server.com
  DocumentRoot /opt/bitnami/apps/my_sinatra_app/public
  <Directory /opt/bitnami/apps/my_sinatra_app/public>
      Allow from all
      Options -MultiViews
  </Directory>

#   General setup for the virtual host                                                                                               
DocumentRoot "/opt/bitnami/apps/my_sinatra_app/public"
ServerName www.server.com:443
ServerAdmin you@example.com
ErrorLog "/opt/bitnami/apache2/logs/error_log"
TransferLog "/opt/bitnami/apache2/logs/access_log"

That got me part of the way there, but the instructions on creating a self-signed cert could have been a bit better and more clear, here is what I ran to generate my cert.

$ cd /opt/bitnami/apache2/conf
$ openssl genrsa -des3 -out privkey.pem 1024 
$ openssl req -new -key privkey.pem -out cert.csr
$ openssl rsa -in privkey.pem -out server.key
$ openssl x509 -in cert.csr -out server.crt -req -signkey server.key -days 365
$ apachectl restart    

That should do it, you should now be able to reach the same Ruby app behind https that was initially running on http. This let me get to the next step of playing around with my personal project, but really I need to get all of the Bitnami box setup automated with a chef script, but that is for another time.

Fix to install Nokogiri on bitnami EC2 instance

2012-10-21T00:00:00+00:00

Recently I set up a ec2 server, using the great Bitnami ruby stack. I used bitnami, because I wanted to avoid having to deal with a bunch of server setup just to play around with a box. It worked great, but for some reason, I couldn’t get the Nokogiri gem to install on the box. After trying many many solutions, I found this post on StackOverflow about installing nokogiri 1.5.2 on bitnami. This was exactly what I was looking for, so my solution came down to, running the following command, with all libs, etc explicitly passed in.

sudo gem install nokogiri -- --with-xml2-dir=/opt/bitnami/common --with-xslt-dir=/opt/bitnami/common --with-xml2-include=/opt/bitnami/common/include/libxml2 --with-xslt-include=/opt/bitnami/common/include --with-xml2-lib=/opt/bitnami/common/lib --with-xslt-lib=/opt/bitnami/common/lib

Thanks StackOverflow, and thanks Bitnami

Rails tests gotcha with I18n.locale

2012-01-14T00:00:00+00:00

A example of a bad I18n rails test… Found some tests failing in a project because tests in different files would change I18n.locale, changes like this make your tests fail if the tests are not run in the standard order. since this took me awhile to track down, I thought I should share a quick example for others to find. So if you are having a issue like this remember to add a teardown method that resets I18n.locale to the default local.

<script src="https://gist.github.com/1568648.js?file=I18n_bad_test.rb"> </script>

Ruby processing: making it snow

2011-12-27T00:00:00+00:00

ruby, processing, graphicsI re-found a fun holiday project I did a couple years ago making snow with Ruby processing.

Cleaning up files after hacking

2011-12-17T00:00:00+00:00

This blog was hacked awhile ago. It was annoying, I hadn’t updated my blog software for awhile and there was a security hole. It was a good learning experience of how to deal with the intrusion. It was also nice to be able to quickly whip together some scripts to help clean everything up. This let me see which files were recently modified

`find . -type f -mtime -3 | grep -v "/Maildir/" | grep -v "/logs/"``

The exploiters were modifying my .htaccess and adding a few files of their own, this deleted their files

    <script src="https://gist.github.com/1491781.js?file=gistfile1.txt"> </script>

This code helped remove exploit code that was injected into all of the pages on the site.

    <script src="https://gist.github.com/1491783.js?file=remove_code.rb"> </script>

After upgrading my software and installing security patches, a bit of other cleanup, and file permission fixes everything was back to normal.

ActiveRecord / Rails case insensitive ordering

2011-11-16T00:00:00+00:00

Rails, Ruby, ActiverecorderIt is pretty common to want to have the results of a query sorted by order. I wanted the results sorted by name on a user object. Oddly it is hard to find how to make a string based order case insensitive on google. So for other having the same issue, it is simple, just add lower(field) in the order clause and it will sort in a case insensitive manner. Now I don’t care when a user doesn’t capitalize the first letter of their name. User.order('lower(name)').all

Announcing NothingCalendar

2011-11-14T00:00:00+00:00

Ruby, Rails, learning, projectsI wanted a small project to play around with. A project that was small and simple that I could actually get done in my free time. An app I would use myself. I ended up deciding to build NothingCalendar, which was inspired by CalendarAboutNothing. It let’s users track events, anything they feel like tracking over time. Mark off days you worked out, kept to your diet, worked on writing, studied, quit smoking, or anything else. The idea was a simple way that I could add marks on a calendar on my phone, and have it sync-able and shareable. I also wanted it to be useable when you are offline, since phones loose connection often in metros, planes, lines, etc. One goal with the project is to keep it is small enough that I can build it and play around with different ideas. It is such a small app, that I could rewrite the entire thing with a new methodology if I wanted. Which is exactly what I plan on starting to do. I wanted a app that I could practice new skills and techniques on, while still being useful, and something I would use on a day to day basis. I started working on the project with the idea to build up a chain of doing some open source experimental development having a playground to try things and learn. A small app where I could push myself to look at new technologies, and to force myself to release them into a production environment, opposed to another one off script or tests that never see the light of day. I highly recommend forcing yourself to do a bit of development every day, it is recommended by a bunch of smart people (Andrew Hunt and David Thomas’s The Pragmatic Programmer: From Journeyman to Master, Chad Folwer’s The Passionate Programmer, Paul Graham’s Hackers and Painters). I was in part inspired by Chris Strom (@eee_c), who has done incredible work keeping up a chain and blogging about what he learns. NothingCalendar is a testament to what you can do in about 15 minutes a day. That is about how much time I have spent on it most evenings. Just forcing myself to do a bit every day, has led to my longest streak on CalendarAboutNothing (66 days at the moment). Just remembering to code a bit everyday, keeps the project in your mind, keeps you improving it. It also makes refactoring little things a perfect task, you only have a few minutes take something on. Complete a tiny refactoring of from a todo list and check it in. I frequently just check in #TODO comments, and grep for them when I don’t have much time or an idea of what to work on. After a tiny todo, call it a day and know you will have more time sometime soon to do something more substantial. Please give NothingCalendar a try, and let me know how it goes or any problems / feature ideas. I will will be working more more on it each day, and trying to rewrite large parts of the app using some new ideas.

working with Facebook Dialog Feed and urls with params

2011-11-13T00:00:00+00:00

Facebook’s documentation always sucks. There are many examples of working with facebooks dialog feed, but none of them include passing urls that have params. They are all simple examples like share ‘http://example.com’, well if you want to share ‘http://example.com?foo=bar&buzz=baz’ you are kind of out of luck. It took me far longer than I care to admite to figure out the right way to get facebook to share a link that included params. I was using ruby and rails, creating a link with _url and params, and then trying to encode it and send it to facebook. Below is what ended up working. Note that it seems to be a bit of an odd encoding, but it allows one to send urls with params and encoded ‘?’ and ‘&’ properly through facebook and have the share url include the params your expecting. facebook_shared\_url = action\_url({:foo => 'bar', :buzz => 'baz'})facebook\_url = "https://www.facebook.com/dialog/feed?app_id=#{FACEBOOK_CLIENT_ID}&redirect_uri=#{URI.encode(facebook_shared\_url).gsub('&','%26')}&description=#{text}&name=#{via}&link=#{URI.encode(facebook_shared\_url).gsub('&','%26')}"

Grep a filtered list of files

2011-11-12T00:00:00+00:00

I am working on a project that has view formats. Frequently when grepping the project, I want to grep only one format. I added a couple bash aliases to help me pipe a list of files to grep. Grepping through a specific list of files can be useful in many situations. Here is how to grep a list of files or set up quick aliases to make it easy to do. First you need to get a list of files, in my case I do this with find and looking for the touch format alias find_touch_views='find app/views/ -name "*.touch.erb"' This generates the list of files you want to grep, now you just pipe that list to grep using xargs to grep each file for a specific word. alias grep_touch_views='find_touch_views | xargs grep -Ri "$0"' in the end this lets me run grep_touch_views touch-nav Modify these to fit your needs and add them to your ~/.bash_profile alias find_touch_views='find app/views/ -name "*.touch.erb"'alias grep_touch_views='find_touch_views | xargs grep -Ri "$0"' which gives me a list of all the views that contain touch-nav.

Git clone from local filesystem to fetch local branches

2011-10-29T00:00:00+00:00

Github, Git, environment, So I am working on a project that has a very long test suite. Yes, that is a problem and we should start to do something about it. For now that problem exists but it leads to wanting to do some weird things. I wanted to work on code in a branch, while my tests were running in another. I had cloned the project twice so I had two versions, of the project, the issue was the branch I wanted to work in hadn’t been pushed remote. I then realized since it is git I could easily solve the problem by pointing my second version of my project at the first and fetching all of my local branches! Git is pretty great that you can solve things like this. It also means you could easily just run git off a shared network filesystem, no reason to really setup a fancy git server. Here are the steps make a second version of a project and then fetch the local branches from your first version # if you haven’t clone a second version of the project do that from the filesystem or the git server> cd project_2> git clone /Users/danmayer/projects/project_1> Cloning into project_1… # in the second version of your project add a remote repo pointing at the original version of the project on your filesystem> cd project_1> git remote add local /Users/danmayer/projects/project_1/> git fetch local> git branch #=> a list of the branches on your first projects directory This is a pretty simple thing to do, but when I ran into the issue it took me a bit of time to realize how simple it was to solve this problem. It also just illustrates just how versatile git really is.

Simplifying new gemset creation for RVM

2011-10-23T00:00:00+00:00

Ruby, environment, developmentI found myself annoyed that each time I created a new project I had to go through the same steps to create and setup my gemset. I am also trying to be better about automating things I do over and over. I decided I should put something in my .bash_profile to simplify my frequent task of making a new gemset on a brand new project. The manual way I ended up creating gemsets in the past, was:* cd to project_directory* rvm use ree (my preferred local ruby run time)* rvm gemset create projectGemSet* echo ‘rvm use ree@projectGemSet’ > .rvmrc* cd ../* cd project_directiory* agree to trust the .rvmrc file That is pretty simple, but gets a bit annoying to do over and over. I figured I could very quickly whip up a solution in my bash profile. Unfortunately, due to my shell scripting skills kind of sucking, I ended up using Ruby. Basically after adding my new ruby script to my .bash_profile, I can now just type ‘init_gemset newgemset’, and I will be left in my project using my new gemset. The final script is below.

    <script src="https://gist.github.com/1257690.js?file=init_gemset.rb"> </script>

One thing to note is the time saved with automating a repetitive task vs. time spent to automate it. In the end because of spending a decent amount of time trying to write this as a shell script without using Ruby. I likely spent more time getting this script working, than it will ever save me. It also isn’t even fully reusable for other people in it’s current state (the .bash_profile function needs a full path to the ruby script in it, and my preferred RVM ruby is hard coded). It could be made more easily reusable / sharable (likely making it a gem, or sending a pull request to RVM itself with this as a feature). Doing this the fully ‘right’ way would just add more time spent, on something I initially did to save a few moments. I wanted to get back to what I was originally working on, and I had something that would work for me. So I will leave it here as a simple gist. Working on this also made me think about what is worth spending time automating. In terms of savings this likely wasn’t worth much, but there are many times that a repetitive task is really begging for automation.

ActiveRecord Association objects can act weird (bug in association#any?)

2011-10-02T00:00:00+00:00

Rails, RubyI found a weird issue when working with ActiveRecord AssociationProxy objects, they weren’t proxying the method correctly. The issue arose when trying to use the #any? method on an array which should be true when empty. When calling this on a AssociationProxy object that has 0 items it was returning false. Oddly enough inspecting or to_s the association and then calling #any? resulted in the correct behavior. It was so weird though that after looking at other options I ended up using the present?, which always behaved how I expected it to. Even weirder is the issue only presented itself in the rails server debug and production modes. It always acted as one would expect in rails console. I started to dig in more, but ended up yak shaving a lot of other stuff to try to learn exactly what the cause was. If anyone understands the issue better, I would be happy to know more. Until then, be aware that at least in the 2.3.X versions of rails, ActiveRecord AssociationProxy#any? acts funky and should not be trusted. Use a work around like present? or length > 0 (both worked correctly in console and on dev/production server environments). This is one of these issues where there is so much going on in the rails stack that it is hard to determine where the issue lies. So for those googling this issue, beware AssociationProxy.any? on has_many relationships. Example code:

<script src="https://gist.github.com/1250980.js?file=AR_bug.rb"> </script>

Google Closure Compiler And Ruby

2011-07-04T00:00:00+00:00

I have repeatedly needed to compress / minify a collection of javascript files. Which I had been doing by hand with Google’s closure compiler. I started to get annoyed with this process and went looking for a better solution. If you are using Rails, there is a great solution using Jammit asset packager library for Rails. I was not using rails, and just needed something that could write the files on demand. A bit of quick searching came up withh this post Compressing Javascript Code With Google Closure Compiler And Ruby. It basically met my needs, but needed to be modified a bit to take a collection of files and write that collection of compresses files back to disk. Here is the solution I ended up with:

    <script src="https://gist.github.com/1063584.js"> </script>

Emacs swap buffer window method

2011-06-05T00:00:00+00:00

So I really like split screen in emacs. I am in that mode pretty much all the time. One thing that annoyed me was opening a file on the side of the screen I wanted. Sometimes I want code left, tests right others it is project A left project B right or often server code left, client code right. Anyways I often opened a file on the wrong side of the screen and wanted to move the buffer. Previously I always closed the buffer, then switched to the other emacs window (C-x o) then reopened the file. This annoyed me to know end, so I finally went looking for a solution. I found something that was close, but not what I wanted, which swaps two buffers positions left and right, so I started with that code and modified it to do just what I want. I then bound the function to “C-x /” which wasn’t in use and I can now swap buffer windows with ease. Also, kind of fun to do a bit of lisp/emacs hacking and breaking out of my Ruby world a bit.

    <script src="https://gist.github.com/1009137.js?file=swap_buffer_window.el"> </script>

If you notice I have it also close the current tab for the buffer and move the cursor context to the new buffer on the other side of the screen. Most people hate and don’t use tabs in emacs so that one tab closing could be commented out. I love tabs because I am a very spacial person and the tabs let me keep the physical location of the buffers in my head. This was written to run in Aquamacs, but will work equally well in standard emacs. Just drop it in your .emacs file or emacs config directory. Code can be found at this gist, Emacs swap buffer window

Force all Sinatra traffic to https

2010-08-07T00:00:00+00:00

I wanted to have all traffic on a site go through https. Since the site was on heroku. @env['rack.url_scheme'])=='https' wasn’t a sufficient way of detecting if on https. You need to also check (@env['HTTP_X_FORWARDED_PROTO'].

Below I wrote a simple before filter which you could place in the production configuration. After talking with the author of rack-ssl-enforcer I got that patched to work on heroku as well. So that is an even simpler option.

Anyways, I spent far to long googling sinatra https, sinatra ssl, sinatra over https, sinatra secure, and heroku https, and couldn’t find good pages on this for awhile. So here is the info for anyone else wanted to host secure Sinatra pages on heroku.

      <script src="http://gist.github.com/512887.js?file=sinatra_https_redirect.rb"> </script>

Adding Custom JavaScript Handlers for PhoneGap Android

2010-07-10T00:00:00+00:00

One of the advantages I mentioned about PhoneGap, was being able to write custom native code, and interface with it via JavaScript. Their Aren’t great example of how to do this out there, so I figured a quick blog post was in order. I search around for how to do this when getting started with PhoneGap, and the best example I could find was, Native messages for Android PhoneGap. Following this example, I was able to get my own Javascript interface working. Basically you create a class (using java for android example):

<script src="http://gist.github.com/470919.js?file=CustomJS.java"> </script>

I create this private class in the main Java class which extends DroidGap (the class that boots up PhoneGap and creates the webview UI).

<script src="http://gist.github.com/470920.js?file=extendDroidGap.java"> </script>

Then in your html you can call in javascript bridged Java functions like var myData = CUSTOM.getData(); which will return the java data (in this case a string) to the javascript. I have used the JS bride to begin new Android Activities, Close the WebView and move entirely to Native code. Background the WebUI to do something native for awhile, then return the WebView back to the front, Etc.

<script src="http://gist.github.com/470922.js?file=customJSPhoneGap.js"> </script>

This can be done similarly for the iPhone and Objective-C to Javascript with PhoneGap. This lets you leverage extra native flexibility while still building the majority of your app as JS, CSS, and HTML.

Mobile Development and My Recommendation for PhoneGap

2010-07-06T00:00:00+00:00

In the last month or so I have been exploring mobile development. I wrote basic native Android and iPhone apps. I also build cross platform apps using Rhodes, Titanium, and PhoneGap. I thought it would be good to share my thoughts and experiences on the mobile dev space. If you want full control and are doing a lot of local heavy processing (CPU bound) native is the way to go. I program to the web data without access I don’t do much interesting. When the whole app basically relies on live data I feel javascript, HTML, and CSS is the way to go. Not only am I already familiar with it, but basically the app can share code with the mobile site. Also, using JS, HTML, and CSS gives the most flexibility in terms of supporting multiple devices. Since I wasn’t using anything specific to a single mobile device or really pushing the hardware, I quickly moved away from fully native apps. It wasn’t worth the time and extra effort to write nearly the same app in Java, Objective C, and whatever else. Game developers, and those doing a bunch with video, photos, etc are probably most interested in native apps and probably won’t be as interested in other options. This lead me to explore Rhodes, Titanium, and PhoneGap as alternatives to purely native apps. All of which let you leverage HTML and CSS for building a UI. Rhodes lets you write code in Ruby using a MVC framework. You leverage html (erb/haml) views using a Ruby back end to make remote requests, interact with a DB, and interact with native phone components. It is a cool framework, and obviously since it is Ruby, I enjoyed writing the code. However, it obscured the actual process of making an app for a device a bit much for my comfort. If something went wrong anywhere in the stack or if you wanted to break out of the Rhodes way to do something native on a platform it was difficult to do. It tries to manage the entire app development end to end hiding any per device customizations away from you. Since I wanted to be able to use some existing libraries for Android and iPhones, this limitation was a dealbreaker. Titanium and PhoneGap use the same basic approach as each other providing a javascript library, which interfaces with native code for each device. You write an app in JS/HTML/CSS and make calls to the JS library to interact with native elements like the accelerometer, GPS, buttons, etc. I think it’s a big win to be able to write most of the app in familiar web languages basically utilizing Ajax and local storage to give a nice mobile experience. Getting up and running on either Titanium or PhoneGap was painless. Again, in the end I wanted more control over some native components which caused me to choose PhoneGap over Titanium. While the concept is the same for both of these products, their approach is very different. Titanium is a bit more all or nothing of an approach, you develop in Titanium’s little world and it is best to not go out ofit. PhoneGap takes a extremely lightweight and minimalist approach to getting your project running on each device. It gives you basically the bare necessities and gets out of your way. PhoneGap doesn’t try to configure the build environment and it doesn’t manage simulators. To create a app with PhoneGap for any mobile platform you point to the same web directory (HTML, CSS, JS, and images), then generate a native app for the device. At that point you have the same native app project you would have if you were to start from scratch but you have a few nice libraries and code started for you. If you don’t like the way PhoneGap does something, that is fine delete it, override it, extend it. You need to do something native PhoneGap doesn’t support add your own JS handlers and interact with new or existing native code. I needed to allow for FaceBook connect login on Android, which PhoneGap doesn’t support no problem. FaceBook maintains a Android SDK, which I imported into my PhoneGap project and added JS handlers which call to the FaceBook Android native code. There is a bug in Androids SSL cert verification for WebKit that was causing errors for HTTPS access, drop into java override the SSL Cert verification and fixed that. This would be difficult to do with Rhodes or Titanium because they don’t allow you easily write and interact with your own native code. The project has to work only through the API provided to you to stay cross compatible. Which is great in some ways, but very limiting if you want to break out and just tweak a few little things. I have been extremely happy working with PhoneGap, it rocks. It lets you build a native mobile app as a locally cached and privileged mobile website. You can leverage existing mobile site code for the native app. It will still be better than a pure website experience. It lets you piggyback on skill web developers already have. It is a great way to make use of HTML5 local DBs, local data, and HTML templates. It encourages you follow best practices for building a mobile site that works offline and requests only minimal JSON so it can fast over slow internet. Before building an app with PhoneGap I recommend building a mobile site. Learn about best practices for the mobile web. If you are new to developing sites for mobile, I highly recommend following the recommendations from Yehuda Katz on building great mobile sites. He discusses some things specific to Rails 3, but mostly just good JS practices. In many ways when building a PhoneGap app you are just building a mobile website. You have access to utilizing some native things that you can’t with pure HTML/CSS. Local storage is a good example, it is supported by HTML5, but the earlier Android webkit didn’t support it. Using PhoneGap you have local storage on old devices and it can use standard HTML5 local storage on newer devices. Last but not least, PhoneGap has a great active community. Drop into #phonegap on freenode and people are always around to help out. The mailing list has a constant flow of questions and answers. While PhoneGap docs are a little out of date, there are people actively working on improving them. There are many example applications and open code on github to check out how to use PhoneGap. Also, the code is small and simple enough it is a great project to get involved in and give back. After working with it awhile, you will start to understand the internals and could help contribute patches, features, and plugins to the community.

The Ruby Fitbit Api Has Been Updated

2010-07-03T00:00:00+00:00

—| layout: posttail authors: [“Dan Mayer”] title: “The Ruby-fitbit API has been updated” category: tags: [] —

More about it can be found on here, Ruby-fitbit api on github. Also, the original announcement can be found on my blog. Basically there is now detailed access to any of the data types for any given day. This can be used to backup your fitbit or gather the data for your own charting purposes. Which I will likely start taking advantage of soon for my fitbit blog widget. Example: ~/projects/ruby-fitbit(master) > ruby bin/ruby-fitbit MYEMAIL MYPASS steps 06/22/2010Calories Burned 1012Steps Taken 0Milkes Walked .00Activity Levels Durations:Sedentary 16hrs 7minLightly 0minFairly # => {fitbit.fairly_active}Very 0minsteps data for data Tue Jun 22 00:00:00 -0400 2010+——————————–+——-+

+——————————–+——-+

Tue Jun 22 21:50:00 -0400 2010

91.0

Tue Jun 22 05:05:00 -0400 2010

0.0

Tue Jun 22 16:15:00 -0400 2010

0.0

Tue Jun 22 10:40:00 -0400 2010

0.0

Tue Jun 22 19:55:00 -0400 2010

69.0

Tue Jun 22 14:20:00 -0400 2010

15.0

Tue Jun 22 08:45:00 -0400 2010

1.0

Tue Jun 22 03:10:00 -0400 2010

0.0

Tue Jun 22 23:35:00 -0400 2010

0.0

Tue Jun 22 01:15:00 -0400 2010

0.0

Tue Jun 22 18:00:00 -0400 2010

0.0

Tue Jun 22 12:25:00 -0400 2010

0.0

Tue Jun 22 06:50:00 -0400 2010

0.0

Tue Jun 22 21:40:00 -0400 2010

11.0

Tue Jun 22 10:30:00 -0400 2010

0.0

Tue Jun 22 04:55:00 -0400 2010

0.0

Tue Jun 22 16:05:00 -0400 2010

0.0

Tue Jun 22 19:45:00 -0400 2010

543.0

Tue Jun 22 14:10:00 -0400 2010

0.0

Tue Jun 22 08:35:00 -0400 2010

347.0

Tue Jun 22 03:00:00 -0400 2010

0.0

Tue Jun 22 23:25:00 -0400 2010

0.0

Tue Jun 22 01:05:00 -0400 2010

0.0

Tue Jun 22 17:50:00 -0400 2010

0.0

Tue Jun 22 12:15:00 -0400 2010

0.0

Tue Jun 22 06:40:00 -0400 2010

0.0

Tue Jun 22 21:30:00 -0400 2010

25.0

Tue Jun 22 10:20:00 -0400 2010

0.0

Tue Jun 22 04:45:00 -0400 2010

0.0

Tue Jun 22 15:55:00 -0400 2010

0.0

Tue Jun 22 19:35:00 -0400 2010

19.0

Tue Jun 22 14:00:00 -0400 2010

0.0

Tue Jun 22 08:25:00 -0400 2010

0.0

Tue Jun 22 02:50:00 -0400 2010

0.0

Tue Jun 22 23:15:00 -0400 2010

0.0

Tue Jun 22 06:30:00 -0400 2010

0.0

Tue Jun 22 00:55:00 -0400 2010

0.0

Tue Jun 22 17:40:00 -0400 2010

0.0

Tue Jun 22 12:05:00 -0400 2010

0.0

Tue Jun 22 21:20:00 -0400 2010

23.0

Tue Jun 22 15:45:00 -0400 2010

0.0

Tue Jun 22 10:10:00 -0400 2010

0.0

Tue Jun 22 04:35:00 -0400 2010

0.0

Tue Jun 22 19:25:00 -0400 2010

535.0

Tue Jun 22 02:40:00 -0400 2010

0.0

Tue Jun 22 13:50:00 -0400 2010

0.0

Tue Jun 22 08:15:00 -0400 2010

0.0

Tue Jun 22 23:05:00 -0400 2010

0.0

Tue Jun 22 11:55:00 -0400 2010

0.0

Tue Jun 22 06:20:00 -0400 2010

0.0

Tue Jun 22 00:45:00 -0400 2010

0.0

Tue Jun 22 17:30:00 -0400 2010

0.0

Tue Jun 22 21:10:00 -0400 2010

0.0

Tue Jun 22 15:35:00 -0400 2010

0.0

Tue Jun 22 10:00:00 -0400 2010

0.0

Tue Jun 22 04:25:00 -0400 2010

0.0

Tue Jun 22 19:15:00 -0400 2010

595.0

Tue Jun 22 08:05:00 -0400 2010

0.0

Tue Jun 22 02:30:00 -0400 2010

0.0

Tue Jun 22 13:40:00 -0400 2010

0.0

Tue Jun 22 22:55:00 -0400 2010

0.0

Tue Jun 22 17:20:00 -0400 2010

0.0

Tue Jun 22 11:45:00 -0400 2010

0.0

Tue Jun 22 06:10:00 -0400 2010

0.0

Tue Jun 22 00:35:00 -0400 2010

0.0

Tue Jun 22 21:00:00 -0400 2010

0.0

Tue Jun 22 04:15:00 -0400 2010

0.0

Tue Jun 22 15:25:00 -0400 2010

0.0

Tue Jun 22 09:50:00 -0400 2010

0.0

Tue Jun 22 19:05:00 -0400 2010

425.0

Tue Jun 22 13:30:00 -0400 2010

0.0

Tue Jun 22 07:55:00 -0400 2010

0.0

Tue Jun 22 02:20:00 -0400 2010

0.0

Tue Jun 22 22:45:00 -0400 2010

0.0

Tue Jun 22 00:25:00 -0400 2010

0.0

Tue Jun 22 17:10:00 -0400 2010

22.0

Tue Jun 22 11:35:00 -0400 2010

0.0

Tue Jun 22 06:00:00 -0400 2010

0.0

Tue Jun 22 20:50:00 -0400 2010

0.0

Tue Jun 22 09:40:00 -0400 2010

0.0

Tue Jun 22 04:05:00 -0400 2010

0.0

Tue Jun 22 15:15:00 -0400 2010

0.0

Tue Jun 22 18:55:00 -0400 2010

62.0

Tue Jun 22 13:20:00 -0400 2010

0.0

Tue Jun 22 07:45:00 -0400 2010

0.0

Tue Jun 22 02:10:00 -0400 2010

0.0

Tue Jun 22 22:35:00 -0400 2010

0.0

Tue Jun 22 05:50:00 -0400 2010

0.0

Tue Jun 22 00:15:00 -0400 2010

0.0

Tue Jun 22 17:00:00 -0400 2010

35.0

Tue Jun 22 11:25:00 -0400 2010

0.0

Tue Jun 22 20:40:00 -0400 2010

0.0

Tue Jun 22 15:05:00 -0400 2010

0.0

Tue Jun 22 09:30:00 -0400 2010

0.0

Tue Jun 22 03:55:00 -0400 2010

0.0

Tue Jun 22 18:45:00 -0400 2010

0.0

Tue Jun 22 02:00:00 -0400 2010

0.0

Tue Jun 22 13:10:00 -0400 2010

0.0

Tue Jun 22 07:35:00 -0400 2010

0.0

Tue Jun 22 22:25:00 -0400 2010

0.0

Tue Jun 22 11:15:00 -0400 2010

0.0

Tue Jun 22 05:40:00 -0400 2010

0.0

Tue Jun 22 00:05:00 -0400 2010

0.0

Tue Jun 22 16:50:00 -0400 2010

0.0

Tue Jun 22 20:30:00 -0400 2010

0.0

Tue Jun 22 14:55:00 -0400 2010

0.0

Tue Jun 22 09:20:00 -0400 2010

0.0

Tue Jun 22 03:45:00 -0400 2010

0.0

Tue Jun 22 18:35:00 -0400 2010

0.0

Tue Jun 22 07:25:00 -0400 2010

0.0

Tue Jun 22 01:50:00 -0400 2010

0.0

Tue Jun 22 13:00:00 -0400 2010

91.0

Tue Jun 22 22:15:00 -0400 2010

445.0

Tue Jun 22 16:40:00 -0400 2010

0.0

Tue Jun 22 11:05:00 -0400 2010

0.0

Tue Jun 22 05:30:00 -0400 2010

0.0

Tue Jun 22 20:20:00 -0400 2010

0.0

Tue Jun 22 14:45:00 -0400 2010

0.0

Tue Jun 22 09:10:00 -0400 2010

0.0

Tue Jun 22 03:35:00 -0400 2010

0.0

Wed Jun 23 00:00:00 -0400 2010

0.0

Tue Jun 22 07:15:00 -0400 2010

0.0

Tue Jun 22 01:40:00 -0400 2010

0.0

Tue Jun 22 18:25:00 -0400 2010

0.0

Tue Jun 22 12:50:00 -0400 2010

0.0

Tue Jun 22 22:05:00 -0400 2010

80.0

Tue Jun 22 16:30:00 -0400 2010

0.0

Tue Jun 22 10:55:00 -0400 2010

0.0

Tue Jun 22 05:20:00 -0400 2010

0.0

Tue Jun 22 20:10:00 -0400 2010

0.0

Tue Jun 22 03:25:00 -0400 2010

0.0

Tue Jun 22 14:35:00 -0400 2010

0.0

Tue Jun 22 09:00:00 -0400 2010

370.0

Tue Jun 22 23:50:00 -0400 2010

0.0

Tue Jun 22 12:40:00 -0400 2010

0.0

Tue Jun 22 07:05:00 -0400 2010

0.0

Tue Jun 22 01:30:00 -0400 2010

0.0

Tue Jun 22 18:15:00 -0400 2010

0.0

Tue Jun 22 21:55:00 -0400 2010

69.0

Tue Jun 22 16:20:00 -0400 2010

0.0

Tue Jun 22 10:45:00 -0400 2010

0.0

Tue Jun 22 05:10:00 -0400 2010

0.0

Tue Jun 22 20:00:00 -0400 2010

0.0

Tue Jun 22 08:50:00 -0400 2010

13.0

Tue Jun 22 03:15:00 -0400 2010

0.0

Tue Jun 22 14:25:00 -0400 2010

0.0

Tue Jun 22 23:40:00 -0400 2010

0.0

Tue Jun 22 18:05:00 -0400 2010

0.0

Tue Jun 22 12:30:00 -0400 2010

0.0

Tue Jun 22 06:55:00 -0400 2010

0.0

Tue Jun 22 01:20:00 -0400 2010

0.0

Tue Jun 22 21:45:00 -0400 2010

17.0

Tue Jun 22 05:00:00 -0400 2010

0.0

Tue Jun 22 16:10:00 -0400 2010

0.0

Tue Jun 22 10:35:00 -0400 2010

0.0

Tue Jun 22 19:50:00 -0400 2010

155.0

Tue Jun 22 14:15:00 -0400 2010

40.0

Tue Jun 22 08:40:00 -0400 2010

203.0

Tue Jun 22 03:05:00 -0400 2010

0.0

Tue Jun 22 23:30:00 -0400 2010

0.0

Tue Jun 22 01:10:00 -0400 2010

0.0

Tue Jun 22 17:55:00 -0400 2010

0.0

Tue Jun 22 12:20:00 -0400 2010

0.0

Tue Jun 22 06:45:00 -0400 2010

0.0

Tue Jun 22 21:35:00 -0400 2010

53.0

Tue Jun 22 10:25:00 -0400 2010

0.0

Tue Jun 22 04:50:00 -0400 2010

0.0

Tue Jun 22 16:00:00 -0400 2010

0.0

Tue Jun 22 19:40:00 -0400 2010

438.0

Tue Jun 22 14:05:00 -0400 2010

0.0

Tue Jun 22 08:30:00 -0400 2010

12.0

Tue Jun 22 02:55:00 -0400 2010

0.0

Tue Jun 22 23:20:00 -0400 2010

0.0

Tue Jun 22 06:35:00 -0400 2010

0.0

Tue Jun 22 01:00:00 -0400 2010

0.0

Tue Jun 22 17:45:00 -0400 2010

0.0

Tue Jun 22 12:10:00 -0400 2010

0.0

Tue Jun 22 21:25:00 -0400 2010

32.0

Tue Jun 22 15:50:00 -0400 2010

0.0

Tue Jun 22 10:15:00 -0400 2010

0.0

Tue Jun 22 04:40:00 -0400 2010

0.0

Tue Jun 22 19:30:00 -0400 2010

499.0

Tue Jun 22 02:45:00 -0400 2010

0.0

Tue Jun 22 13:55:00 -0400 2010

0.0

Tue Jun 22 08:20:00 -0400 2010

0.0

Tue Jun 22 23:10:00 -0400 2010

0.0

Tue Jun 22 12:00:00 -0400 2010

0.0

Tue Jun 22 06:25:00 -0400 2010

0.0

Tue Jun 22 00:50:00 -0400 2010

0.0

Tue Jun 22 17:35:00 -0400 2010

0.0

Tue Jun 22 21:15:00 -0400 2010

67.0

Tue Jun 22 15:40:00 -0400 2010

0.0

Tue Jun 22 10:05:00 -0400 2010

0.0

Tue Jun 22 04:30:00 -0400 2010

0.0

Tue Jun 22 19:20:00 -0400 2010

604.0

Tue Jun 22 08:10:00 -0400 2010

0.0

Tue Jun 22 02:35:00 -0400 2010

0.0

Tue Jun 22 13:45:00 -0400 2010

0.0

Tue Jun 22 23:00:00 -0400 2010

0.0

Tue Jun 22 17:25:00 -0400 2010

0.0

Tue Jun 22 11:50:00 -0400 2010

10.0

Tue Jun 22 06:15:00 -0400 2010

0.0

Tue Jun 22 00:40:00 -0400 2010

0.0

Tue Jun 22 21:05:00 -0400 2010

0.0

Tue Jun 22 04:20:00 -0400 2010

0.0

Tue Jun 22 15:30:00 -0400 2010

0.0

Tue Jun 22 09:55:00 -0400 2010

0.0

Tue Jun 22 19:10:00 -0400 2010

291.0

Tue Jun 22 13:35:00 -0400 2010

0.0

Tue Jun 22 08:00:00 -0400 2010

0.0

Tue Jun 22 02:25:00 -0400 2010

0.0

Tue Jun 22 22:50:00 -0400 2010

0.0

Tue Jun 22 17:15:00 -0400 2010

0.0

Tue Jun 22 11:40:00 -0400 2010

0.0

Tue Jun 22 06:05:00 -0400 2010

0.0

Tue Jun 22 00:30:00 -0400 2010

0.0

Tue Jun 22 20:55:00 -0400 2010

0.0

Tue Jun 22 04:10:00 -0400 2010

0.0

Tue Jun 22 15:20:00 -0400 2010

0.0

Tue Jun 22 09:45:00 -0400 2010

0.0

Tue Jun 22 19:00:00 -0400 2010

161.0

Tue Jun 22 13:25:00 -0400 2010

0.0

Tue Jun 22 07:50:00 -0400 2010

0.0

Tue Jun 22 02:15:00 -0400 2010

0.0

Tue Jun 22 22:40:00 -0400 2010

0.0

Tue Jun 22 00:20:00 -0400 2010

0.0

Tue Jun 22 17:05:00 -0400 2010

0.0

Tue Jun 22 11:30:00 -0400 2010

0.0

Tue Jun 22 05:55:00 -0400 2010

0.0

Tue Jun 22 20:45:00 -0400 2010

0.0

Tue Jun 22 09:35:00 -0400 2010

0.0

Tue Jun 22 04:00:00 -0400 2010

0.0

Tue Jun 22 15:10:00 -0400 2010

0.0

Tue Jun 22 18:50:00 -0400 2010

0.0

Tue Jun 22 13:15:00 -0400 2010

0.0

Tue Jun 22 07:40:00 -0400 2010

0.0

Tue Jun 22 02:05:00 -0400 2010

0.0

Tue Jun 22 22:30:00 -0400 2010

0.0

Tue Jun 22 05:45:00 -0400 2010

0.0

Tue Jun 22 00:10:00 -0400 2010

0.0

Tue Jun 22 16:55:00 -0400 2010

37.0

Tue Jun 22 11:20:00 -0400 2010

0.0

Tue Jun 22 20:35:00 -0400 2010

26.0

Tue Jun 22 15:00:00 -0400 2010

0.0

Tue Jun 22 09:25:00 -0400 2010

0.0

Tue Jun 22 03:50:00 -0400 2010

0.0

Tue Jun 22 18:40:00 -0400 2010

0.0

Tue Jun 22 01:55:00 -0400 2010

0.0

Tue Jun 22 13:05:00 -0400 2010

75.0

Tue Jun 22 07:30:00 -0400 2010

0.0

Tue Jun 22 22:20:00 -0400 2010

22.0

Tue Jun 22 11:10:00 -0400 2010

0.0

Tue Jun 22 05:35:00 -0400 2010

0.0

Tue Jun 22 00:00:00 -0400 2010

0.0

Tue Jun 22 16:45:00 -0400 2010

0.0

Tue Jun 22 20:25:00 -0400 2010

0.0

Tue Jun 22 14:50:00 -0400 2010

0.0

Tue Jun 22 09:15:00 -0400 2010

0.0

Tue Jun 22 03:40:00 -0400 2010

0.0

Tue Jun 22 18:30:00 -0400 2010

0.0

Tue Jun 22 07:20:00 -0400 2010

0.0

Tue Jun 22 01:45:00 -0400 2010

0.0

Tue Jun 22 12:55:00 -0400 2010

0.0

Tue Jun 22 22:10:00 -0400 2010

268.0

Tue Jun 22 16:35:00 -0400 2010

0.0

Tue Jun 22 11:00:00 -0400 2010

0.0

Tue Jun 22 05:25:00 -0400 2010

0.0

Tue Jun 22 20:15:00 -0400 2010

0.0

Tue Jun 22 03:30:00 -0400 2010

0.0

Tue Jun 22 14:40:00 -0400 2010

0.0

Tue Jun 22 09:05:00 -0400 2010

74.0

Tue Jun 22 23:55:00 -0400 2010

0.0

Tue Jun 22 12:45:00 -0400 2010

0.0

Tue Jun 22 07:10:00 -0400 2010

0.0

Tue Jun 22 01:35:00 -0400 2010

0.0

Tue Jun 22 18:20:00 -0400 2010

0.0

Tue Jun 22 22:00:00 -0400 2010

0.0

Tue Jun 22 16:25:00 -0400 2010

0.0

Tue Jun 22 10:50:00 -0400 2010

0.0

Tue Jun 22 05:15:00 -0400 2010

0.0

Tue Jun 22 20:05:00 -0400 2010

0.0

Tue Jun 22 08:55:00 -0400 2010

0.0

Tue Jun 22 03:20:00 -0400 2010

0.0

Tue Jun 22 14:30:00 -0400 2010

16.0

Tue Jun 22 23:45:00 -0400 2010

0.0

Tue Jun 22 18:10:00 -0400 2010

0.0

Tue Jun 22 12:35:00 -0400 2010

0.0

Tue Jun 22 07:00:00 -0400 2010

0.0

Tue Jun 22 01:25:00 -0400 2010

0.0

+——————————–+——-+289 rows in setdone

Classy Joe Nassy

2010-06-28T00:00:00+00:00

This is just a tribute to a developer I work with, saying “keep it classy.” This is classy Joe, AKA classy Joe Nassy.

Another page dedicated to classy joe.

Javascript document.addEventListener PhoneGap deviceready issues

2010-06-26T00:00:00+00:00

I had some issues with the deviceready callback. I had seen some similar posts, and figured I could save someone some time. Basically when using document.addEventListener order matters a lot. It all comes down to exactly when this line is loaded and what is already defined. <pre>document.addEventListener(“deviceready”, deviceSetup, true);</pre> We have custom JS, and originally it was loaded second, so the deviceready event was fired before, the listener was set.

    <script src="lscustom.js" type="text/javascript" charset="utf-8"> </script>
    <script src="phonegap.js" type="text/javascript" charset="utf-8"> </script>

After that I was still having an issue and it turns out the method you define register as the callback needs to be defined before you add the listener. So having this in our lscustom.js works.
var deviceSetup = function(){ console.log(“Device Setup”); }; document.addEventListener(“deviceready”, deviceSetup, true); While, the below fails. document.addEventListener(“deviceready”, deviceSetup, true); var deviceSetup = function(){ console.log(“Device Setup”); };

Anyways since I ended up wasting a bit of time searching and debugging this issue, I figured I would share my debugging session.

Code Highlighting on MovableType

2010-06-21T00:00:00+00:00

def code_highlighting puts "this is a highlight"end I found a syntaxhighlighter plugin, which makes it simple to use SyntaxHighlighter on Movable Type. After downloading and unzipping the plugin in your MT directory. Just add the <pre> MTSyntaxHighlighterInclude </pre> tag to your templates and then choose the SyntaxHighlighter format when making a post.

Appcelerator Titanium older SDKs

2010-06-16T00:00:00+00:00

If you have a need to work with a app on a older SDK Titanium doesn’t make it very easy for you. In fact they don’t seem to have historical files available for download or any info on how to install older SDKs. You can get a old version of the 0.8.x (in this case 0.8.2) SDK here:Titanium 0.8.2 SDK download And after a bunch of searching I found that you should move the file on OSX to this location:/Library/Application Support/Titanium/mobilesdk/osx/0.8.2 After that restart Titanium and the SDK should be available to you in the SDK dropdown when you create or import projects.

Ruby Dev Available for Contracting

2010-06-12T00:00:00+00:00

Avdi a ex-Devver, developer, has just announced that ShipRise LLC is open for business. He is looking to do freelancing/contracting for Ruby and other hard development problems. I can attest to Avdi’s skill in breaking down hard problems (like distributed test running) into elegant simple solutions. If you are looking for a great dev to work with you on a project, ping him. Besides Ruby and general coding, Avdi is ready to help with “Dispersed team facilitation”, which he has long term experience with. If you have a small distributed team of developers, he can help get everyone on the right path to work together, while being apart. He is gather a ton of information to help remote workers at his new blog Wide Teams.

New gig at LivingSocial

2010-06-03T00:00:00+00:00

livingsocial, ruby, jobsI just started my new gig at LivingSocial. I had been a fan of their product ever since moving to DC. I had loved how it got groups of my friends to get together and plan something. I also really enjoyed how it gave me new ideas of cool things to do in my city. Especially since I just moved to DC and didn’t really know what to do. We have bought deals to restaurants we wanted to try, restaurants we already knew and loved, kennedy center brunch with a group of friends, a wine tour with a group of 5 of us going. We have really enjoyed the savings, but even more so that it helps get us out there doing and trying new things. Anyways I will be joining the development team, obviously doing Ruby stuff, since that is what I am focused on now. The team is already impressive and has accomplished a lot and scaled the site to request levels I have yet to deal with. I am excited to get to work and learn with the team, while bringing some of my skills from past experiences to help with the team. Yesterday, I pushed my first code live to production, which made me an official member of the team. Now to dig in and wrap my head around this code base.

Ruby Fitbit API

2010-05-26T00:00:00+00:00

Fitbit has been slow to release an official API, so I started work on a screen scraping version. It will become a nice gem interface to the real API when it is released. It is pretty limited at the moment, only able to pull the most basic current days info. You can check it out here ruby-fitbit on github. Example output<pre>~/projects/ruby-fitbit(master) > ruby bin/ruby-fitbit my@email.com MYPASSCalories Burned 834Steps Taken 552Milkes Walked .23Activity Levels Durations:Sedentary 11hrs 28minLightly 19minFairly 16minVery 0min done</pre> I also connected a quick sinatra app on Heroku to it so that I could embed the data in my personal blog as a widget. I will clean that up and release the code for it soon. Here is the widget at the moment. I need to add good CSS that is customizable by the user. It would also be good to make it easy for others to host their own widgets opposed to just my own data.

Converting Movable Type to Tumblr

2010-05-23T00:00:00+00:00

Continuing my work on converting this blog to be a dev blog, I moved over all non development and computer related posts to my personal blog. I just converted about 1,000 posts and 3,000 comments from MovableType to Tumblr and Disqus. I decided I would host my personal blog on Tumblr, which has been interesting so far. Sucking the old posts and comments out of MovableType and pushing them to Tumblr wasn’t as easy as I had hoped. Although it gave me a chance to enjoy on of my favorite parts of being a developer. If something doesn’t exist and I want it, I can built it. So I set off to write a script that would take a Movable Type blog export and import all of the posts into Tumblr and push all of the comments to Disqus. While it wasn’t the easiest thing to do, it wasn’t that difficult either. I have slowly built up a collection of various utilities to help port from Movable Type to Wordpress, or from wordpress.org to wordpress.com. Things that help clean up formatting or remove javascript and replace it with something allowed on wordpress.org. I added my conversion script to the inconveniently named MT_WP_Converter git repo. View MovableType to Tumblr Converter Script

Quick Gotchas:

There are lots of older or bad example code on how to use the disqus API out there, I had a problem posting comments because all of the POST endpoints must end with a ‘/’, which I didn’t know. This forum post helped me fix an issue posting to disqus with rest client.
Remember to turn off the facebook publishing for tumblr, before doing the import or testing it. I didn’t remember this and published the same test post on facebook about 40 times. Oops!

Improving my Emacs Configurations

2010-05-14T00:00:00+00:00

Emacs, Aquamacs, emacs starter kit, developmentI have been using Emacs as my default editor for awhile now. My emacs setup was a hodge podge of various plugins, code from friends, blogs, etc… It wasn’t organized and was hard to maintain and fine tune. I decided it was time to get my emacs config under control, and under git so it was portable and easy to revert update etc. I looked at a couple options for existing Emacs configuration setups for Ruby. Emacs Starter Kit Topfunky’s Emacs Starter Kit Aquamacs Starter Kit In the end I ended up going with Aquamacs Starter Kit, namely because I have been a fan of Aquamacs for awhile, and already new my way around it’s system pretty well. The original Starter kit, from Technomancy, doesn’t play nicely with Aquamacs and leads to some issues. Some people have forks that are supposed to work better with Aquamacs, but it seems the kit built from the ground up for Aquamacs works best. Topfunky’s project is based on the original emacs starter kit, but has been customized to work well with Carbon Emacs, which is more cross compatible than Aquamacs, so your emacs config is more likely to be portable to all unix systems as well as OS X with Carbon emacs. This looks like a really interesting option, but I decided I liked some of the nice extra sugar that Aquamacs provides. First install Aquamacs starter kit There were a few things that I couldn’t stand from the defaults in the Aquamacs Starter Kit and removed also some modifications that I added. ### Disabling/Removing some Starter Kit features You likely installed your starter kit to a location like, ‘/Users/danmayer/Library/Preferences/Aquamacs Emacs/aquamacs-emacs-starter-kit’, go there to find the files for modification. Disabling the emacs Twitter client, sorry I like my twitter to be entirely separate from my dev environment. Then edit ‘init.el’:;;(autoload 'twitter-get-friends-timeline "twitter" nil t);;(autoload 'twitter-status-edit "twitter" nil t);;(global-set-key "\C-xt" 'twitter-get-friends-timeline);;(add-hook 'twitter-status-edit-mode-hook 'longlines-mode) Disable Ruby-electric, which tries to complete various matched chars and statements as you type like ‘, {}. (), begin/end, if/end, etc… It drives me nuts and breaks my thought process. Edit ‘misc-mode-tweaks.el’:;;(require 'ruby-electric);;(add-hook 'ruby-mode-hook;; (lambda nil;; (require 'ruby-electric);; (ruby-electric-mode);; (flymake-mode-on))) ### Additions to Starter Kit I saw a Emacs package to add flog score inline to ruby method scores and thought that sounded like a great addition to help keep my methods concise, so I added that, to my .emacs: ;; ruby-complexity flog scores for methods http://github.com/topfunky/emacs-starter-kit/tree/master/vendor/ruby-complexity/(add-to-list 'load-path "~/.emacs.d/vendor/ruby-complexity/") (require 'linum)(require 'ruby-complexity)(add-hook 'ruby-mode-hook (function (lambda () (flymake-mode) (linum-mode) (ruby-complexity-mode)))) I also liked some of the options from defunkt’s Textmate minor mode. First ‘cd ~/.emacs.d/vendor; git clone git://github.com/defunkt/textmate.el.git’ Again added to my .emacs file: ;; textmate bindings for some nicer key combos for common actions(add-to-list 'load-path "~/.emacs.d/vendor/textmate.el")(require 'textmate)(textmate-mode) I am just starting to get going with improving my Emacs setup, so I am sure I will likely be making additional changes. The current changes above are checked into my fork of the project, feel free to pull the above changes if you’d like. If you have any great Emacs tips, tricks, or plugins I should add let me know as I am sure it can just keep getting better, but I am pretty happy for now.

Introducing Ruby Resume, a project to create and host your resume

2010-05-10T00:00:00+00:00

Ruby, Github, Open source, Programming, Heroku, Markdown, Tools, ResumeThe Ruby Resume project, is a project I started to make it easier for Ruby developers to create, host, share their resume, and keep it up to date. It is an open source project, that anyone can use to help manage their resume online. It offers a variety of options and you can take or leave any part you wish. Basically, you fork the project, alter some things for your needs, and can contribute interesting additions back to the shared Ruby Resume project. The project uses Sinatra, Markdown, and a collection of Rake tasks to get the job done. What does the Ruby Resume project do?- Supports deploying your resume to Heroku in a variety of formats.

Easily deploy your app to any Sinatra compatible host
Allows simple publishing of your resume to your github personal page.
It makes it simple to publish your resume as a gem. I got the idea for a personal resume gem from Eric Davis.
It uses Markdown, which integrates well with Github Jobs
It currently suppots HTML, LaTeX, and Markdown. Soon it will support PDF, RTF, etc… I built this because I had to publish and start updating my resume again after not dealing with it for 3 years. I wanted something that would simplify the whole process. I wanted my resume under git, and I wanted to be able to quickly deploy any changes online and support a large number of formats. Anyways check out the source on Github and the Readme which gives simple instructions on how to use the project. Or what the video below which demonstrates how to use this project for your own resume. Live Examples:- My resume on Heroku
My Github personal page
My personal resume gem is, danmayer-resume, which can be installed using gem install danmayer-resume, then execute danmayer-resume

This blog will be changing soon

2010-05-04T00:00:00+00:00

thoughts BlogIt will becoming my development blog. I am currently working on updating and reconfiguring the system, but you can look forward to many changes around here. If you want to follow and read my more personal blog posts, head over to WastedBrains, which will host my non development blogging and thoughts.

A archive of my last project

2010-05-01T00:00:00+00:00

Since Caliper is shutting down on April 30th, I made a little video to show what it did, and explain it’s purpose. It isn’t really promotional, but if people ever want a reference of something I worked on this should help. </embed> Caliper Screencast from dan mayer on Vimeo.

Dan Mayer Resume

2009-04-18T00:00:00+00:00

Daniel Mayer dan@mayerdan.com

Refactoring makes me happy

2008-07-29T00:00:00+00:00

I still get such a kick out of refactoring and shrinking down code. I try not to shrink it to the point that it is harder to read. I think some people get a little overzealous with refactoring and obscure so much of the code that it is no longer that readable, or everything is 20 levels deeper than it should be when you are in a debugger trying to trace a programs execution. Anyways I was cleaning up a bunch of little TODOs or ugly sections of code that had been building up in some of our files. It felt good, and should make things even simpler moving forward.

Quicksort in Processing.js

2008-05-28T00:00:00+00:00

I always liked computer visualization. I think it is fun and interesting, and drawing Mandelbrot for the first time brought a smile to my face. I always thought I understood sorting, AI, search, and other algorithms best when you could really see them in action. I always enjoyed little videos or applets of a bunch of different sorting algorithms comparing their differences and speeds. I don’t really have time right now to do a whole bunch, but when I saw processing ported to javascript I thought I really would have to write a visualization for at least one sort just for the fun of playing around with a cool project. So here is a link to my Quicksort in Processing.js, it will only work in the very newest browsers (FF3, Opera 9.5, and other beta browsers… feel free to try it out on anything worst thing that will happen is just an empty box… but it will not likely do anything on older browsers). Anyways, just some fun coding to play around with… If anyone can figure out how to make the double buffering / frame rate look good I would love a fix to make the animation run smoothly.

iTunes Stalling on Windows XP

2008-02-25T00:00:00+00:00

I have been meaning to write this little tutorial ever since I fixed my iTunes on windows XP. I have a pretty good computer (3.4 ghz 1Gb ram), so I was confused when switching from WinAmp to iTunes to find my music randomly stalling. If i was running something process intensive like compiling or even just loading a new program my music would stutter on XP. I search around the web and found that others were having this problem, but not many had a good solution. After browsing around the web, I found that the stuttering in my music was caused when iTunes didn’t get enough of the CPU to process the music file. So when something would jump up and use 90% of my CPU iTunes would happily stop processing the music because it didn’t have a high enough process priority to keep a couple percent of CPU for itself. When playing music in the background iTunes only needs 3%-4% to play the music. So now knowing the problem I searched for solutions and many just said that every time they started iTunes they would open the Windows Task Manager go to the processes tab and right click on iTunes to set the priority of the process to high. This did in fact work, but now I had to do that every time that I openned iTunes which was really annoying. So I continued looking for a better solution. There are special windows programs out there, most of them cost money, that will monitor which processes are running on the system and automatically change the process priority on any app you tell the system to increase the priority to whatever settings you choose. This seemed like and OK solution but now having an app running all the time checking my processes and just some other app that could cause problems, seemed bad. It also seems that processes aren’t as stable if you switch their priority after they begin, it is best to set the priority as they are started. So off searching for a solution I found that you can start processes in does and pass what priority you want to start the process. So I decided to just hack up a little script which would launch iTunes from dos with a process priority of ‘High’. to run a windows app at a different process priority than the default you can create a file named “appName.bat” (in this case iTunes.bat), place this file in the same directory as the executable you want to run with a higher priority. Then edit the bat file (you can edit it in notepad). Add this line to the file start /high iTunes.exe (or start /priority_setting application.exe) Save your bat file… Now all you have to do is create shortcuts to the new bat file instead of the shortcuts pointing directly the application. To create a shortcut right click on your bat file and select ‘Create Shortcut’. If you want the shortcut to have the same icon as the original app instead of the dos icon, right click the shortcut and got to properties, then the ‘Shortcut’ tab, and click Change Icon. Select browse an application to search for an icon in, then select the application you are creating a shortcut for (in this case iTunes.exe), it will display the available icons select the one your want. Anyways, I hope this helps anyone else having trouble getting audio to play smooth on windows XP with iTunes. I have also used this method to speed up some other processes that I always want running full speed on windows, with pretty good success.

online resources for computer science students

2008-01-20T00:00:00+00:00

50 killer online resources for computer science students. I thought it was a decent list and figured I would share it. Dave sorry their is stuff about computers on my blog, I know it bothers you… You never called me when you were in Denver, so um… were even or something.

Ruby on Rails built in string shortcuts

2007-10-16T00:00:00+00:00

There are some great string munipulation shortcuts in rails. I thought this was a good write up and little collection of some of the most useful functions. Rails string shortcuts

Always use lenth on ruby collections

2007-09-07T00:00:00+00:00

This bug never got me, but it is a good little ruby best practice to follow. So remember when ever using ruby collections use .length call not the .size call… Why use length on all ruby collections.

Amazing Image Resizing

2007-08-21T00:00:00+00:00

This is the coolest image resizing I have ever seen, I love that your can resize to erase objects out of photos, awesome… Amazing Image Resizing

Finished Rails Book

2007-08-12T00:00:00+00:00

I finally finished a reading the rails book all the way through. I keep putting it off and reading other books because reading straight through a programming book is boring. Yippie for finishing that up, and I learn some more little rails tricks that should be helpful. Next up Linux System Administration and Backup and Recovery!

Using Rails link_to_if

2007-07-31T00:00:00+00:00

There are a lot of great helper methods in rails, and I need to remember to use more of them, one I over looked for a long time that I have found useful is link_to_if <%= link_to_if logged_in?, “#{item.name}”, :controller => ‘items’, :action => ‘show’, :id => item %> replaces the much longer and more verbose: <%- if logged_in? %><%= link_to , “#{item.name}”, :controller => ‘items’, :action => ‘show’, :id => item %><%- else %><%= item.name %><%- end %> I like being able to replace five confusing lines with one single simple solution. The link_to_if helper writes a link if the condition is met, otherwise it just outputs the text that would have been in the link. Very cool. Also if you want to use Ajax forms with multiple submit buttons in Rails, here is the solution. Lastly if you are trying to write a selector and want to make sure the test is returning the html you expect you can add puts @response.body into the test after the page call and see the response. This has been useful to me a few times.

Highly useful assert_select

2007-06-26T00:00:00+00:00

I find myself using the assert_select a lot while doing testing and rails. I have had to keep looking up different examples depending on the situation. I have found that this page seems to have most everything I need when I can’t remember it off the top of my head. Rails assert select examples

Ruby rails testing

2007-06-12T00:00:00+00:00

My roommate made a great post on rails testing so I thought I would share that here. Testing faster in ruby on rails.

Ruby email to rss

2007-05-20T00:00:00+00:00

A simple ruby email2rss example When I was searching for some basic ruby scripts to download pop email I came across the email 2 Rss project on Ruby forge. Since I noticed there was no code available for the project yet I figured I would make a quick version up and make the code available, so anyone else searching for similar examples would have a starting point. This code is just a functional example, but doesn’t have any real options or error checking so use at your own risk. Download the source to Ruby Email to RSS Feed <pre>require ‘net/pop’require ‘rss/maker’ version = “2.0” # [“0.9”, “1.0”, “2.0”]destination = “/archive/emailFeed.xml” # local file to write #create a new email Feedcontent = RSS::Maker.make(version) do

rss

rss.channel.title = “Dan Email Feed” rss.channel.link = “http://www.pretheory.com” rss.channel.description = “My newest emails” rss.items.do_sort = true # sort items by date Net::POP3.start(‘pop.yourDomain.com’, 110, ‘userName’, ‘myPass’) do

pop

if pop.mails.empty? puts ‘No mail.’ else i = 0 pop.each_mail do

# or “pop.mails.each …” subject = m.header.split(“\r\n”).grep(/^Subject: /)[0] subject = subject.gsub(“Subject: “,””) subject = subject.gsub(“:”,””) subject = subject[0,15] if(subject.length > 15) link = ‘http://www.gmail.com’ body = m.pop body = body[0,25] if(body.length > 25) #create feed item item = rss.items.new_item item.title = subject item.link = link item.date = Time.now item.description = body i += 1 end puts “#{pop.mails.size} mails popped.” end endend File.open(destination,”w”) do

f.write(content)end puts ‘Done. Thanks.’</pre>

Ruby Email Archiver

2007-05-20T00:00:00+00:00

Checking and receiving email via ruby. If you need to get emails and check emails via a ruby script this should get you started in the right direction. I needed to copy down about six thousand emails from my pop server, so I wrote up this little script. Ruby makes email simple, you just need to know where to look and unfortunately most searches result in finding a 3rd party library that costs a pretty penny for email actions in ruby. What you really need is the Net::POP3 or the Net::IMAP. After going through the quick documentation and seeing some of the available code examples, writing the code is straightforward. This is just a very simple example, with hardly any error checking. Download the source to Ruby Email Archiver <pre>require ‘net/pop’ Net::POP3.start(‘pop.yourdomain.com’, 110, ‘userName’, ‘myPass’) do

pop

if pop.mails.empty? puts ‘No mail.’ else i = 0 pop.each_mail do

f.write m.pop end #if you want to delete msg after archive #m.delete i += 1 end puts “#{pop.mails.size} mails popped.” endend</pre>

Rails casts

2007-05-18T00:00:00+00:00

A great way to keep up with some rails tips and ideas. I have started watching rails casts when I am eating a snack, working out, or just have a few minutes. It is a great and simple way to keep rails ideas fresh in your head. Most of the tips are fairly basic for now, but I am hoping they start dipping into more complex rails solutions. Or how to break out of the framework when it makes sense. Railscasts

Really good post on regular expresions

2007-05-17T00:00:00+00:00

I found that this was a really good post on regular expressions. Just a good beginner walk through that will have you more comfortable looking at any regex you run into while debugging someone else’s code, or if your trying to add your own regular expressions yourself. Regular expression walk through

I am now at Pretheory

2007-05-16T00:00:00+00:00

I know I don’t actually post to this blog much at all, but I still occasionally add some stuff on it and I might slowly be bringing it back to life. So with that in mind I thought I should announce that I am now the co founder of a start up. Pretheory. If you want to keep up with the various projects we are working on the best place to check is over at the Pretheory blog. Currently we are programming in Ruby on Rails, and we are hoping to have our first project launched by early August. So if you have been following this blog at all, keep and eye on Pretheory now as well.

Good Startup Resources

2007-05-14T00:00:00+00:00

Here is a list of some good start up resources that I have begun to check out on the web. Paul Graham’s essays
Clay Shirky’s essays
OnStartups
Small Business Hub
Startup news
programming.reddit.com
Good start up books to read
Hopefully I will grow this list a bit more shortly, but for now that is some good information to get your started.

Eclipse Plugin Callisto

2007-01-31T00:00:00+00:00

This is a pretty cool project you can use via the update manager you just tell it the type of development you want to do (java swing/j2ee/java webapps/c++/ and others). Anyways it is pretty good and gets a bunch of plugins together and manages them for you, so if newer versions of one thing still require an older version of another plug in it deals with any incompatibility issues. You select what you want and then it will say you need other dependencies click the button that says get required. That seems to be it. Callisto peace,Dan

Spam bots try anything

2007-01-14T00:00:00+00:00

Well I have on many of my sites had to deal with and fight the onslaught of spam bots, but I still find them amusing some times. After a bug occurred on a production machine, I was looking at the logs and ran into many errors looking like this:

  2007-01-11 18:15:05,938 [TP-Processor4395]
  ERROR com.realestate.search.core.SearchController - java.lang.NumberFormatException: For input string: "http://valtrex-gs.****.com"2007-01-11 18:15:05,980 [TP-Processor4395]
  ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/].[jsp] - Servlet.service() for servlet jspthrew exceptionjava.lang.NumberFormatException: For input string: "http://valtrex-gs.***.com"       at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)

It seems this was a pretty dumb, but probably mildly successful bot. It started some search and found tons of web pages, every time it encountered a form, it would attempt to fill out the majority of the fields with the URL hoping to make it into generated pages, most common search terms, comments, user names, or anything that would possibly link to the URL and increase the sites page rank. I found it hilarious, because somewhere on our site it was trying to fill out number, date, and other fields with the URL which we threw errors on format exceptions and probably returned them to the form with an error about that field. The bot hasn’t given up, but hasn’t slammed us very hard. Either way it is another amusing tale, of spam bots running amok on the web. At least they can make me laugh sometimes…

Mysql null date

2006-12-18T00:00:00+00:00

I had an issue with java and hibernate returning a null date. java hibernate java.sql.SQLException: Value ‘0000-00-00’ can not be represented as java.sql.Date This just took a little longer to find the fix on a google search than I thought it should have so I post what I ended up doing to resolve the situation here: ?zeroDateTimeBehavior=convertToNull you add this to your jdbc connection url and it will convert the 0000-00-00 to the proper type for java to work with. More info here: http://dev.mysql.com/doc/mysql/en/cj-upgrading.html#cj-upgrading-3-0-to-3-1 thanks to this thread, which had many other suggestions, but most of them were not helpful, but the one above saved my day. http://forum.java.sun.com/thread.jspa?threadID=649520&messageID=3824019 you can add this on the end of your hibernate connection file, like so…jdbc:mysql://{DB_IPADDRESS}:{DB_PORT}/{DB_NAME}?zeroDateTimeBehavior=convertToNull&profileSql=true

Soap Sucks

2006-11-19T00:00:00+00:00

I have said it for a long time now that XML and SOAP are bloated and most of the time the wrong way to do anything. Well it seems lead api developers are agreeing with me, it might have been the decade of XML but I hope to slowly see it fade to a hardly used only when needed standard, instead of the if it doesn’t use XML it obviously isn’t designed well. Soap i wasted so much time and clock cycle for projects that you never did well and were always replaced. let me say now I’d never choose to use SOAP and WSDL again. I was wrong

Java String isEmpty()

2006-11-19T00:00:00+00:00

I have heard the java string class will finally get the isEmpty method in an upcoming release. Lets hope it is in 1.6, because I can’t wait much longer, I can’t for the life of me see why this was ignored for so long. Why have I had to put this in so many freakin if statements when this could have been fixed by adding a very easy and simple but amazingly easy to read and useful method. String randWord; /random code or back in forth between many different classes */ String someFunction() {if(randWord!=null && !randWord.equals(“”)) { /do something/} ….} all of this could just be replaced and make the code so much more readable with the isEmpty function, becoming the following… if( !randWord.isEmpty() ) { /do something*/} I want this functionality and I want it now… I guess since java is now opensource I should just go override the String class to add one simple function! So much value and so simple.

Nerds are great

2006-10-12T00:00:00+00:00

Some days they make me proud to be a nerd… This just cracks me up,Mozilla Bug Report

Ubuntu Server 6.06

2006-07-10T00:00:00+00:00

I installed Ubuntu server on one of my machines, one of the nice things about it, is that it from install it configures a working LAMP (Linux, Apache, MySQL and PHP) set up. Which is nice and saves lots of configuration. Unfortunately, I still need to get Subversion, Ruby, Tomcat, and other things installed and running on the system. Anyways since, I was going to have to set up a bunch of things and planned on using the system as a development machine I then installed Ubuntu desktop on the system as one of the aptitude scripts, which installs all the packages require for the desktop configuration. After that though, I realized that I was not able to complete any install with out causing an error in clvm, anything you install would cause a clvm error and make your install fail. So here is the fix if you install the Ubuntu server from CD and then install the desktop: Ultimately this post by npcomplete2000, was the solution for me:http://www.ubuntuforums.org/archive/index.php/t-186356.html removing clvm caused no problems for me and was something I never thought I would need. After that I could install other applications with out problems. More error information so those searching for error codes will find this post:Starting Cluster LVM Daemon clvmd could not connect to cluster managerConsult syslog for more informationinvoke-rc.d: initscript clvm, action “start” failed.dpkg: error processing clvm (–configure): subprocess post-installation script returned error exit status 3Errors were encountered while processing: clvmE: Sub-process /usr/bin/dpkg returned an error code (1)

Ruby on Rails Gem error could not find gem

2006-07-10T00:00:00+00:00

I got this gem error on Ubuntu 6.06 over and over then finally I followed these instructions to install Ruby on Rails, Install Ruby on Rails on Ubuntu 6.06 and I changed one line: sudo gem update –system IS WRONG!!!! run: sudo gem update and thing all the sudo gem installs work perfectly fine for me. Why I have no idea. I hate computers sometimes, but at least ruby and rails now work for Ubuntu 6.06 on my system.

Old Computer Science Links

2006-07-03T00:00:00+00:00

General Links University of Colorado Web2gether Project L3D labs at CU My code online Machine Learning MITs class page on Machine Learning The Use of Java in Machine Learning AI: a modern approach Java Neural Networks Text Categorization Links Intelligent Information Retrieval Tutorial on Text Mining Open source spam filtering project Yet Another Multipurpose Chunk Annotator Weka Links WEKA Spam analysis LSA/LSI Links LSA @ CU LSI @ UTK SVM Links Technical Paper Describing SVM classification SVM Info / Lessons Comprehensive List of SVM publications Large Scale SVM Learning SVM Spam Filter Research paper SVM Tools: SVDLIBC SVMseq Tiny SVM SVM-Light

Coding tan lines

2006-05-21T00:00:00+00:00

This is funny, and probably far to true.

Java skills wanted in the market

2006-05-03T00:00:00+00:00

I have been looking around at what companies seem to want currently from a mi level Java programmer, it seems to be all over the map. I also have notice a big rise in the demand for C#/ASP programmers as well. So if your starting to hunt for a new Java developer position here is a list of things you might want to study up on a bit before going out on the job hunt:

  [Java RMI](http://java.sun.com/j2se/1.4.2/docs/guide/rmi/javarmiproperties.html)[Java Swing Basics / tutorials](http://java.sun.com/docs/books/tutorial/uiswing/TOC.html)[EJB basics / tutorials](http://java.sun.com/developer/onlineTraining/EJBIntro/EJBIntro.html)[Some basic internationalization](http://java.sun.com/developer/technicalArticles/Intl/IntlIntro/)[Java Date formatting](http://home.tiscali.nl/~bmc88/java/sbook/016.html) or more [Date timezones](http://java.sun.com/j2se/1.5.0/docs/api/java/util/SimpleTimeZone.html)[Java Applet security](http://www.securingjava.com/chapter-two/)[Compressing serialized objects or other output files in Java](http://java.sun.com/developer/technicalArticles/Programming/compression/)<a href="http://www.ccs.neu.edu/home/kenb/com3337/rmi_tut.html>more about Java RMI[Threading issues of Array List](http://java.sun.com/j2se/1.4.2/docs/api/java/util/ArrayList.html)[I know we all extend classes and work with interfaces, but they always ask questions about it at interviews](http://www.javacoffeebreak.com/java104/java104.html)There are also many [testing](http://www.webopedia.com/TERM/W/White_Box_Testing.html) positions [available](http://www.faqs.org/faqs/software-eng/testing-faq/section-13.html) which want experience with [JUnit](http://supportweb.cs.bham.ac.uk/documentation/tutorials/docsystem/build/tutorials/junit/junit.html).

So if your currently on a Java position and looking around a bit, here are some areas you should really brush up on a bit before starting your interview rounds.

Ant options

2006-01-30T00:00:00+00:00

if your running out of Java Heap space memory while running ant you probably just need to increase the memory size for ant when it is running to do this you set the enviroment variable ANT_OPTS. ANT_OPTS=-Xmx128M in windows you right click my computer go to properties, and then click advanced then set new enviroment variable name ANT_OPTS value -Xmx128M. If your running cygwin you must close current instances and reopen them to get the new settings to take effect, after opening up cygwin again try >echo $ANT_OPTS which should output your setting. If you still receive the error you can go higher with your memory. I have never had to go higher than 256.

Junit testing server states

2006-01-19T00:00:00+00:00

Alot of Junit docs and faq tells you that have bad design if your trying to extend test suite so that there is only one setUp() and testDown() for a whole sweet of tests, but that just isnt true when your trying to tests proper order of state changes each transaction should change the state of the server dpending on its current state, if the server is started up new for each test your only testing the start up state… my single unit tests have dependencies on previous tests that has cuased the server to be in its current state… So no where gives good documentation on how to do this the faq is un usefully short showing how to extend testSuite but not how to use it… so here is the best page for how to actually use it just ignore hte Turbine2 stuff: 16. Q: How can I setup Turbine once for my JUnit Test cases? And remember that Junit executes your tests in order alphabetically and numerically after the intial keyword test. ie TestA will run before TestB so if you running tests in order for state make sure that you have your tests named appropriately to be sure they will run in the expected order. hope this helps someone running into problems running Juint with a single setUp across a suite of tests.

Apache2 integrating Tomcat 5.5 on windows

2006-01-19T00:00:00+00:00

I have been trying to do this off and on for over a year and a half and every time i run into problems and I get pissed and quit. I have found hundreds of pages that all say different ways of how to solve the problem of integrating tomcat 5.5 with apache2. Non of them worked for me, but after combining tips from various install guides I can show you what finally worked for me. After wasting over all the time hours and hours… here is all that I did small changes to the apache configure file and small ones to tomcats… restart both of there servers and WHOOPIE it just works. Apache2 with Tomcat 5.5 on a windows server.

  httpdconf changes:    LoadModule proxy_module modules/mod_proxy.soLoadModule proxy_connect_module modules/mod_proxy_connect.soLoadModule proxy_http_module modules/mod_proxy_http.so    <VirtualHost *:80>ServerName special.myname.comDocumentRoot "C:/Program Files/Apache Group/Apache2/htdocs"    <ifmodule mod_proxy.c>ProxyRequests OnProxyPass /cool http://66.***.***.**3:8082/your/webapp/rootdir/## these are comments on the dir#C:\Program Files\Apache\Tomcat 5.5\webapps\you\webapp\rootdir\# if anyone was confused to how that would map to windows tomcat stuff#ProxyPass /*.jsp http://66.***.***.**3:8082</ifmodule>    tomcat's server.xml:        <connector acceptcount="100" disableuploadtimeout="true" enablelookups="false" connectiontimeout="20000" port="8082" maxsparethreads="75" minsparethreads="25" proxyport="80" maxthreads="150" />

thats it... you might have to change some of your jsps for how they find local directories and get all the lookups for the local directories to match your mapping, but you got it... it should all work now. Congradulations... if that didnt work for you here are some sites I was looking at for help:

  [Apache proxy forwarding](http://www.drewnoakes.com/snippets/ConfiguringApacheToRedirectASingleHostToMultiplePorts/)    [Apache 2 tomcat 5.5](http://forum.sun.com/thread.jspa?threadID=22381&tstart=210)    best of luck post comments here and i can try to help you if they are problems i have run into before.

Broken MailEnable

2005-12-20T00:00:00+00:00

Problem: Mail Enable Bad Mail folder became HUGE! Or MailEnable Bad Mail folder filled up! MailEnable was stuffing my hard drive with badmessages and outgoing that i didnt think it was allowed to send. Some spammers obviously found a way to attack my mail server and start making it work for them… Anyways it filled an entire 80GB hard drive with bad emails… The folders where so large that explorer would crash when trying to delete the files. So I search around and found this solution to emptying the folders. http://support.microsoft.com/default.aspx?scid=kb;en-us;555408 After you make sure it is working remove the /s from the end of the line you wont see that it is deleting one at a time, but the process goes about 10x faster because you aren’t printing between each of the deletes. I also dissabled my entire mail server until i can figure out what the hell is going on and how to stop the spamming bastards that are trying to take over my server. windows server 2003 too many files to delete solved.Mail Enable can open config solved.Email Spam removed from system solved. The problem now is how slow it is going while deleting the files.

No Fluff just Stuff

2005-11-14T00:00:00+00:00

Well I am done with the Rocky Mountain Software Symposium (RMSS), No Fluff just Stuff. Which was excellent. I learned a ton I wish I could go to one of these ever 3 to 6 months… Anyways I learned alot and have some great stuff that I can put into use on my projects at work. I also learned a ton of just really cool and interesting stuff. That will make me a better programmer all the time, as well as get me moving on new technologies faster than others. Anyways, it was really cool and interesting. Hopefully I will get to use and play with some of the code before i forget all of the cool stuff that i just learned… check the new RSS feed reader that is on the front of wastedbrains to see something I coded up real quick… I highly recommend the no fluff just stuff tour to everyone.

Ruby on Rails

2005-11-10T00:00:00+00:00

I have been learning the language ruby, and have begun playing around with ruby on rails. Which is a development framework for web based applications. I have been really impressed with ruby on rails so far, I was worried about how extendable it would be to less general problems, but it seems to be fairly extendable. Which is cool. I highly recommend giving it a shot and I think this was the best tutorial I have gone through while learning ruby. Ruby on rails tutorial, building a cookbook I also recommend Agile web development with Rails, from the Pragmatic Programmers

CSS columns guides

2005-09-08T00:00:00+00:00

I needed some good CSS layouts so i looked around tinkered a bit and got what i want, and I thought i should share some of the best resources i found. I would like to give thanks to everyone that has made their CSS and XHTML layouts available as it is a very cool thing to do and help out with other web developers. Great threecolum css with header and footer Center floating three column a list apart liquid header Ordered columns float model Skidoo layouts

Apache XML RPC Asych

2005-08-08T00:00:00+00:00

Exception in thread “Thread-0” java.lang.NullPointerException at org.apache.xmlrpc.XmlRpcClient$XmlRpcClientAsyncThread.run(XmlRpcClient.java:271) If your getting any similar errors while trying to run executeAsync from the XmlRpcClient (org.apache.xmlrpc.XmlRpcClient) class. The problem is this bug, which has been fixed by apache, but hasn’t made it into the binaries or the src releases because they havent been updated fro some time. To get the fix use the anonymous cvs, or apply this patch to your src. It took me a few hours to figure out the problem with this and fix it after finding the patch. So if you build the jars and use the xmlrpc-2.0-beta that is built as the jar you include in your project that should fix everything.

java in cygwin

2005-07-21T00:00:00+00:00

man I had some serious problems running java in cygwin with a shell script (sh script)… So I am going to post my script to help you out… there are a couple things of note first… java a windows program expects windows paths your cygwin script proably uses unix paths. cygwin sh scripts treat ‘;’ as a special character so when using multiple classpaths you must escape it ‘\;’ to make it work as expected… # test.shJAVAEXE=$JAVA_HOME/bin/java.execd ../IEP_HOME=C:\dev\dsmayer\sandbox\iepIEP_BIN=C:\dev\dsmayer\sandbox\iep\binIEP_LIB=C:\dev\dsmayer\sandbox\iep\libIEP_JAR=C:\dev\dsmayer\sandbox\iep\lib\iepio.jarIEP_IST=C:\dev\dsmayer\sandbox\iep\lib\istcustom.jarIEPB_JAR=C:\dev\dsmayer\sandbox\iep\lib\backport-util-concurrent.jarIEP_PROPS=C:\dev\dsmayer\sandbox\iep\properties echo java -cp $IEP_JAR\;$IEP_IST\;$IEPB_JAR RCS.example.RCSregechojava -cp $IEP_JAR\;$IEP_IST\;$IEPB_JAR RCS.example.RCSreg &java -cp $IEP_PROPS\;$IEP_JAR\;$IEPB_JAR\;$IEP_IST RCS.example.clients.ResourceClient &java -cp $IEP_PROPS\;$IEP_JAR\;$IEPB_JAR\;$IEP_IST RCS.example.clients.UserClient &

Java Source Code Examples

2005-06-28T00:00:00+00:00

A great collection of simple java source exmples. These are organized well and provide a wealth of information if your having problems doing something you know has been done before. http://www.java2s.com/ Java Source Examples

Computer Science Research Papers

2005-06-13T00:00:00+00:00

I have definately violated some of these and I think it would be good for people that are trying to do some serious analysis to follow these I do hate nothing more when marketing makes grapsh that essentially lie. Anyways if you have a computer science or any science for that matter paper, these are great rules to go by to remain creditable. A Rubic for IT Analysis Papers

Programming Links

2005-06-02T00:00:00+00:00

Since I haven’t been putting much interesting up here of my own code lately I figure I could point you to some things of note and interest: Java Form SpellChecking This is a great tool to allow users of your site or webapp to have access to spell checking much like from MS word. Java Image Galleries Ever wanted a great image gallery solution? Art of Illusion A cool 3d rendering studio entirely in Java. A good webservices and JAVA XML tutorial with lots of basic information

Converting XML with XSLT

2005-05-31T00:00:00+00:00

Problem: Converting XML to Excel I had to convert some XML to sort and display as we wished in excel. After reading a bunch of articles about XML-FO and other stuff about cacoon and other Java solutions I decided to go with just using a fairly simple XSLT stylesheet and convert the XML to HTML tables with the excel extension. This is even the recommended solution by microsoft (Which is probably because excel has a bloated and often changing format.) So converting this way leads to a well and easy and nice looking excel file and has the benifeit that anyone without excel can also view the file in a standard web browser. The microsoft tutorial on converting XML with XSTL is very good. If you have any problems or need more advanced XSTL this is also another very usefull set of XSTL examples. The basic idea is to make the stylesheet so that it can sort and create a easy to view html table. Then you can either let excel open the XML which can find the stylesheet and apply it for the user, or you can use a program to do the conversion for your user and output excel or html files. I needed to generate the files for my users so I went with Xalan from the apache foundation. Xalan is a XSL stylesheet processors in Java & C++ The example is simple to modify for whatever you need. I used the example file SimpleTransform, which can be found in the installation directory at \xalan-j_2_6_0\samples\SimpleTransform The result is simple and easy, below I will include my XSLT file to show how easy it is toe create simple large tables from a nice little XSLT file. I couldnt include it as text cause it tried to render it so here is a link to it...[Download file](http://WWW.bandddesigns.com/ml/arch/trans.xsl)

Java string to inputstream

2005-04-06T00:00:00+00:00

If you need to make a string into an input stream it is really easy. Many people say you shouldnt convert a string into an input stream and that it is bad code design though. Since I was using someone elses library that only took input streams and all of my input was created as strings i really had no choice but to do the conversion (which is wastefull of memory cause you essentially have exact copies of the same data). Making a string into an inputstream can be done like this: ByteArrayInputStream bs = new ByteArrayInputStream(site.getBytes()); If you have any problems leave a comment.

CVS stuff

2005-04-05T00:00:00+00:00

I always am forgettting this and that about CVS and this is just a really good resource on all the CVS stuff you would need to know when working with a regular CVS repository. Command Line CVS

java code tar

2005-03-30T00:00:00+00:00

I have released a development version of java tar with added support for various GNU tar features like multi file, verification, and breaking the 8GB barrier. It is pretty nice so if your into learning some java code or working with tar there is alot of good info here and well documentated source code. Java Tar upgrades

Java Tar

2005-03-30T00:00:00+00:00

Java Tar source. This might not be the final draft, but it is getting pretty close. Most features seem to be working well and the source is very well documentated. So if you want or need the new GNU features you can get them from java tar here. The main java tar build hasnt incoporated these yet but all of the modifications have been sent off to them, so hopefully they will incorporate them soon. Java Tar homepage Until then feel free to download the java tar source from me. You can edit and modify as much as the orginal liscense allows, which says it is public domain use it as you wish. I would like to give a big thanks to Timothy Gerard Endres, who is the orginal author of Java Tar. If you have any questions comments or problems let me know. Download Java Tar

Java Tar and problems

2005-03-30T00:00:00+00:00

I have been working on adding a bunch of GNU tar features to java tar. I will release the source soon, but I just wanted to comment on the lack of good documentation on the format that makes it very hard to work with the GNU tar format… Simple things like regular tar takes char 32 as a null and ignored character… and so does GNUtar for the regular tar fields but in the offset field needed for multi volume tars 32 crashes the program giving you header errors. It only accepts char 48 as a null which in the other size field it accepts either 32 or 48… things like this are commented no where and i only discoved by writting and editing the GNU tar source. Which is also far under documentated in the code and very hard to follow. GNU Tar size field: If your working with GNU tar the size field is just like the standard tar field in ustar… except which is no where to be found in their documentation when you support unlimited size files. If you have a file larger than 8GB to support it you must write the number as bits in twos compliment notation. Also after doing the you have to flip the sign bit (the very most left 0) to a 1. Which would normally mean you have a negative number and now if you decode this as a twos compliment you end up with a huge negative number… but not encoded it normally as a positive number and flip that bit. I only figured this out after manually reading and decoding gnu tar and stanttard ustar headers for a long time. The is nothing that talks about how the support for unlimited files was added to gnu tar… well hopefully if you are having the same problem you found this page. So upgrading GNU tar or other programs to support GNU tar should be easier. I have added support for these gnu features to javatar:Multi volumeverificationunlimited filesizefast single file extractionTar Table of contents (xml of the files and there offsets with in the tar) If your interested in this or have any questions about GNU tar or JAva tar feel free to send me an email and ask.

Java copy file code

2005-03-10T00:00:00+00:00

just cause i didnt find code on this fast enough here is code to copy a file using java. It reads the file in while writting it out using buffered readers and writters:

   public static void copy(String from, String to) throws IOException{   InputStream in = null;    OutputStream out = null;    try {        in = new FileInputStream(from);        out = new FileOutputStream(to);        int length = 128*10240; // danger!        byte[] bytes = new byte[length];        int read=0;        for(;;){        read=in.read(bytes,0,length);        if(read==-1){            break;        }        out.write(bytes,0,read);        }        } finally {            if (in != null) {                in.close();            }            if (out != null) {                out.close();            }        }    }

my neglected blog

2005-02-28T00:00:00+00:00

alright here is a post to my neglected blog. It always falls behind… Anyways i have been doing a ton of programming, but sadly none of it is very interesting or new right now. Alot of the stuff I am doing I have done before, but I am trying to build up a system which I can use to explore some new ideas and I need a nice simple toolset built up before doing anything else all the interesting. So it might be awhile, before I have any really interesting things to post on here, but i will try and we will see when I get a chance to get back to my AI interests.

GNU tar errors on windows

2005-02-08T00:00:00+00:00

I ran into a very odd error while running GNU tar on windows XP under dos or the cmd.exe (dos emulator). The error must be incredibly rare since i only found one other post with similiar problems. I will first explain the error and then how it was fixed or resolved. here is the one link that I found with the similiar GNU windows tar error tar error

  my error:C:\dev\dsmayer\sandbox\exe>tar cvf tester.tar newstar: Cannot add file news: No such file or directory (ENOENT)tar: Error exit delayed from previous errors

news was a standard directory. the error we found was occuring because we were in the C:\dev\*.* directory apparently dev and aux (as the other user was having that error) are some sort of key words that screw up the program, I dont know why. I do know that moving tar.ext and my directory to a folder c:\temp or any other c:\*.*.\dsmayer\sandbox\exe and it would work. So if your having any erros similiar with GNU tar I suggest changing directories and seeing if that fixes your problems. Taring under cygwin in this directory worked fine it was only a problem when taring in dos. Good luck post any questions, comments, or other issues relating to GNU tar on windows here.

Jakarta org.apache.commons.fileupload

2005-02-07T00:00:00+00:00

Using this package caused a little more problems than one would have believed, but it was by far still the best option out there. I found some other beens, but they only would successfully upload Ascii files and were not writen properly to write out image files or other binary data. After having a problem with a few other packages I went to use Jakarta’s java fileupload and after figuring out multiple issues and reading alot of documentation I got all of it working as I wanted. I thought I should share th links I ended up using that were usefull and helped me get everything working.

File upload wrapper take a look at ProcessFileUpload.jsp

Jakarta’s how to use the fileupload package

If you have any problems let me know and I can probably help you out.

Again Sorry

2004-11-03T00:00:00+00:00

Again I am truly sorry that i haven’t been posting anything here. School, My job hunt, and other things have b een taking my time. I have written quite a bit of code and hopefully will post it all soon. I have alot of little pocket pc applications and I also have done decision trees, neural networks, and probabilistic agents. I might have a pretty cool final project for my AI course to post but that will really depend on what I have time to end up doing for a project.

sorry busy

2004-10-12T00:00:00+00:00

Sorry there has been no updates and no new code posted for awhile. In fact comments were broken for the last couple months. Well I have fixed the comments and will slowly be getting back into some of the coding stuff. I have been busy with my job hunt, which seems to be going pretty well, but I dont want to jinx it. I have been doing some interesting stuff in my AI course and learning alot, we have been working with Agents and Probablistic Inference… We are now doing Utility funtions and feedback learning systems, which is pretty cool. I wish i knew more about knowlege representation. Since coding up the problems takes me so long in part because i dont know how to represent all of the data for the problems. Oh well all is going well and I am learning, and I guess that is what that matters. The concepts I get and the coding I can figure out with time.

Intelligent Agents

2004-09-09T00:00:00+00:00

Well I had to write a little one page paper for AI class about agents so here it is incase anyone else is ever thinking about a simple little agent that plays a wierd cave game called Wumpus world. Also, I guess some of the Ideas are a little interesting. The code to run and test everything is provided in my extended entry if you want it. (The code isn’t the prettiest or well commented seeing as I had other work to do and had to finish the entire assignement in less than two days, but it works and the smart agent does do better thant he simple one.)

Agents

I created the two agents, the dumb agent that was mostly random and the more intelligent agent. The dumb agent got an avg of -450 which was close to the naive agent the professor had coded up, which averaged -390. So everything with my world and agent seemed pretty close. I then began work on the intelligent agent. I found it significantly harder than expected to add features that improved performance. Many times adding rules that I thought would help improve the agents performance and these rules would actually result in far lower scores. After trying a few pretty simple ideas I developed what my final agent became. My final agent had an average score of -60 which wasn’t great, but still significantly better than the dumb agent. I added one extra constraint to the game for fun, since I am a vegetarian I decided that I would never kill the Wumpus. So my algorithm would simple avoided the Wumpus in attempts to navigate to the gold. This lead to more impossible maps and therefor a lower overall score. It will still be interesting to see how my animal friendly agent performed in comparison to others.My more intelligent agent worked on the right hand on the maze wall idea. I would go forward until i found a area that could present a problem. A problem being either a stench or a breeze, if this problem was found i would turn around and walk the other direction and then try a different route, with my right hand facing the problem. This worked well at avoiding pits since I also had a higher percent of the time the choice of moving forward, and always would move forward if there was no chance of danger. This quickly lead to the problem of certain pits providing an infinite loop. Lowering my score by getting in a safe, but useless route. To fix this if I encountered the same pit problem multiple times i would then just walk threw the sensed breeze in hopes that the pit was not the direction I was going. This could have been improved by first trying alternate routes around the pit, but could have then left the problem of many different infinite loops.The improvement of the agent was significant and noticeable, but also illustrated the difficulties of simple relying on a simple set of rules. I think a more effective route would have been to have the agent slowly walk along any known safe route while mapping problem areas to his known portion of the map and only after exhausting all safe possibilities (and trying to create safe possibilities by killing the Wumpus) picking at random a unsafe point of passage that would lead to the most possible options for a next move.

fighting back

2004-08-08T00:00:00+00:00

Well lately all the spam on my various websites had been getting out of control. They have machines that go to blogs and post comments with links to ads for words to raise their rating at google. It sucks it has been annoying me and the others that i host for. So i coded up a little thing i am hoping will help. We will see if it actually does. The good news is that i learned where code resides for the comments and i can add in more code and such if i need to to start blocking all the spammers. Fun huh. In the next couple days we will see if it works. It is a good use of learning time since today i was reading learning perl and working on perl programming and the entire blogging system was written in perl. Yeaaa I am learn a language that people told me to learn better before applying to google. if you have any problems leaving comments anywhere tell me with email…

cleaning and moving

2004-07-19T00:00:00+00:00

Today I am collecting all of the source and all of the data from News Shaker. I am preparing to move everything to a new machine that is on a live connection that everyone from the outside can get to. I will then continue to add the rest of the features mentioned below. The last week has been used to add these features, while making little to no process on improving the results of the learning. I am going to talk with a few people about how to improve the learning before working with that part of the project to much more. Overall most of the code seems quite stable enough but alot needs to be done on the user interface to make it useable by normal people that aren’t accustom to odd design and testing set ups. It must be cleaned up before it could ever really be used.

Feature requests

2004-07-06T00:00:00+00:00

I need to add the following features into news shaker to make it more usefull: Done (has an X if finished): X * Delete category and all related sites X * Delete category place all remaining sites in another category X * Ability to have one category be a sub category of another * 2 level categorization (related to the sub category idea above) X * Automated “real world” testing with accuracy for all categories after a new model build. Should consist of 20 unseen and unmodeled sites that are hand categorized and then have them categorized. X * A way to save the results from the real world testing in the database and display them. * A way to post articles that aren’t links but are actually html files into the system. (This also allows visitors to view this file.) X * ability for people to vote for a file that is in the wrong category to be recategorized X * Increased categorization speed * Start a test from a new UID and then track where all the results go and view each result individually. * Making sure that two of the same sites are never added to the database * Checking and updating sites and getting rid of no longer existant ones. X * Ability for users to report errors and admins to view them and delete X * Ability for users to request categories and admins to view them and delete X * Ability for administrator to recategorize based on users votes to recategorizeOn another note i have increased accuracy on testing to the 94% overall accuracy on known documents and i am getting and average of 25% for unknown documents, which isn’t horrible but i would like to do much better. I have now began to study and look into a transductive approach that i might begin to use, depending on the results of the next bits of testing.

News Shaker

2004-06-29T00:00:00+00:00

The last couple of weeks i have done alot of work on news shaker. I have done lots of testing. I all of the categories (about 12) to an approximated error average of 88%. For 12 categories this is really good. First i began by adding more and more data to the categories and rebuilding the models. This initially was increasing the percents but it ceased to help after all of the categories had about 90 documents in them. I then began to play with the weight of the positive terms. This was highly successful after increasing the weighting on all of the positive training vectors I could successfully take all of the training data and recategorize it with 88% accuracy with the remaining documents not wrongly categorized but declared to be of an unknown category. I then started real world testing giving all of my category unseen documents that were hand categorized. The results for the few real world tests i have done so far have been fairly poor, showing only 15-20% accuracy. I am not sure why that varying how the model is made dramatically increases percent of categorization of known documents but seems to have no effect on unseen documents. This currently is the problem i am working on. It is possitive to get known values accuracy for my models to range from 85% to 93%. After a little more real world testing and some other discussion i might be able to come to a conclusion as to what is going on between known and unknown examples.

New NewsShaker Feature

2004-06-18T00:00:00+00:00

After waiting weeks of meaning to add this feature I finally did it. It actually took me less than an hour when I thought i was going to have to write all sorts of new code and that everthing would somehow end up being far more complex than I wanted it to be. Simple feature added, now instead of telling the system to crawl an entire site, you can tell the system to add a single page to the database. This makes it easier when finding an article, that links to entirely useless data, but should be added. So I am glad i finally took the time to add this simple feature. It was also good to see that I still remember alot more of the code structure on the spidering system than i would have thought I remembered. Starting next week I am going to finish making the system entirely automated. I should be able to finish that in a couple days. Then I am going to make the system very general so it doesn’t have to remain so specific to special education and then the same code base for newsshaker would be adoptable to other systems such as the HAMCOD project (which is a horrible name, but since I am more interested in just working on the idea for now I am not going to spend any time working on a name until success full. Man I could make some amazing progress on the system if i could get about 3 people coding on these machine learning systems. Oh well it is good to be back and making some progress again.

Initial Description of HAMCOD

2004-06-17T00:00:00+00:00

This is a description of a idea that may be the next phase of my project as I continue my work with text categorization. It right now is just a initial idea and outline of an idea so that I have some thought to begin working with when I get to the next point. HAMCODHuman Assisted Machine Categorized Open Directory A collaborative human assisted machine categorized directory. That will extend the functionality of the ODP (http:www.dmoz.com) project. This project will combine a already large and extensive base of human knowledge, with text categorization and social collaboration techniques to increase the amount of well categorized and defined websites, without the need of such high levels of human interaction that are required for DMOZ. This projects goal is to eventually use machine learning techniques to replace the slow and time consuming process of human categorization. This will have humans interact with the machine learning process as it contributes to and works with the categorized directory. This will have humans say when something is miss categorized and they will be able to recommend new re-categorizations. Or recommend deletion from the directory. The spider would crawl for new pages that aren’t listed in the category, if a new page is discover by the spider, the system will attempt to categorize it to the lowest level of a category that it can. So if there is a main category like “shoes” it might have a hierarchy like this: 1.____Shoes________Car______Cheese2.___

________

_________

3.Nike__Reebok__Vans__FordToyota____American__Cheddar The system would first do categorization on the level of is this page about shoes. If it is determined to be about shoes it would them use different models knowing that it is a shoe to try to determine what shoe company the page is about. In its early stages it will only work through the Computers category of the DMOZ project. Which is already contains 149,512 websites. To first determine if a site belongs in the computers category, I will need to get about 400,000 websites at random from other parts of the DMOZ directory. Alternatively, I could assume that if I can’t find an appropriate sub category in computers that the site doesn’t belong in the computers category at all. Initially I will design the system with no log in or registration required. Since use will be practically non existent. Once use begins I will require an email address and to have that email address to be confirmed. This will make it rather hard for companies or individuals to artificially increase their rankings with in the directory. With in each category the sites will need some sort of ranking system initially I won’t worry about ranking the sites. I will just assume that as you get specific enough there will only be a small number of sites in each category. (This seems to be true for DMOZ)

Done testing and work on SVMMail

2004-06-16T00:00:00+00:00

After going through a little less than 3000 emails. I have finished testing nad doing any work on SVMMail. It still is sitting at a 97.5% accuracy. I am sure this could be increased, but I need to move all of my focus back to my primary project, News Shaker. On the News Shaker front. I have added about 350 new manually categorized sites across the database. I am going to rebuild all of the models and see if the increased training data brings my percents up to a more reasonable level. Then once I have a little better percent accuracy I will begin all of the auto categorization code and just start to let the system go crazy and see how many sites it can categorize correctly when left to its own devices. Should be an interesting time next week. That is if the machine boots up. Someone was working on my system and now it freezes on boot up. I am sure all my data is still there and I have a fairly recent back up but hopefully this can be sorted out before the begining of next week. I also have began reading Managing Gigabytes which is about compressing and indexing documents and images. I also ordered a new book about machine learning and artifical intellegence that i will begin reading soon. Perhaps they will provide me with some new ideas on how to improve my system.

SVMMail testing update

2004-05-24T00:00:00+00:00

I have now gone through about 800 emails with my SVM / SMO mail filter. I am still getting about 97.5% accuracy. I have blocked over 700 spams from my mail account. I am taking off (to europe) and I am sure my mail will fill up with about 600 emails while I am gone so I will have a much larger test results when I return. Good luck to anyone else out there working with or thinking about SVMs / SMOs for spam filters. It looks like it is a winning combination. I hope my various code and articles can help you on your way. peace,Dan Mayer

SVM Mail feature vector

2004-05-20T00:00:00+00:00

I have gotten a few emails and questions from others researchers in the community and I decided that I would begin to answer questions on my site rather than through email so any others could also benefit from the answer. So here is the first response I am posting on the web. Feel free to contact me if you have any other questions and I can try to respond. I found your project while googling for various alternatives to spamfiltering; I’ve been thinking about trying SVM for mail filtering myself,but I’m slightly at loss as to what features to use. A bag-of-words model comes naturally to mind, but it is not the mostefficient computationally; are you perhaps using it or something similar?First the text content is run through a common words stop list. This removes all very common and useless words when it comes to categorization. Removing words such as �the�, �is�, �that�, and so on. Then the text is converted to the SVM format of feature vectors. The feature vectors are created using a relative frequency to the total word space. Feature # : frequency in document / frequency in word space So let�s say you have 3 documents in your space. All of the words in all of the documents would end up in your word space with a value of the total occurrences in all of the documents. Then a ratio for each document with the same word would be used for that feature. Each feature represents one word in the document. Lets do a tiny example: Documents1: �There is a little cat�2: �Where is the little cat�3:�There is a little dog, very little� SpaceThere:2Where:1Is:2A:2Very:1The:1Little:4Cat:2Dog:1 So each of the words would be assigned a feature number in order of appearance, for instance �There� would be 1 and �Is� would be 2. Then to make the vector you would take each document and calculate the frequency ratio. For example we will do this for the 3rd document. There(1) : � Is : � a : � Little: 2/4 Dog : 1/1 Very : 1/1 Replacing the words with their feature numbers and the fractions with values would result in the final document vector based on frequency. 1:.5 2:.5 3:.5 4:.5 5:1 6:1 You mentioned bag of words which is another way you can create vectors to represent the text. I thought about this method and decided that the frequency would provide a better representation of each document. Neither way is that computationally affective I guess. You can leave out all features with 0 since they will be ignored in SVM anyways. I also leave out all ratios that result in less than 0.000001 because I figure that means the feature is really worthless in comparison to the space. What makes this fairly efficient is that the dictionary with the word counts is saved. So while building the model is time consuming getting new text and making comparisons is really pretty fast. Then you don�t have to rebuild the model each time you place something in a category you can only rebuild as needed or on a set time like once a week. If you would like to see all of the libraries and math that I created to create these vectors download the source code to Text2SVM which I use to do these conversions. This also allows you to save a dictionary since each word must relate to the same feature value in the model and with any future files you plan to test against the model. This allows you to create new vectors that will match up with your old model.

SVMMail vs Apple Mail

2004-05-19T00:00:00+00:00

Well it looks like my SVM mail is very similar to apple’s Mail. Apples mail also uses a vector representation and training to achieve 98%+ accuracy (the claim). After reading through this, it makes sense why my filter is achieving such good accuracy. They are using a little more complicated vector analysis tool than I. I use SVMlight, while Apple is using LSA (Latentic Semantic Analysis), which i used to work with but I found the tools far less developed and harder to work with. It was causing me all sorts of problems to tell LSA to do the simple clustering I was doing very simple with SVMlight. The main reason it looks like they are using LSA is they first reduce the space vector and then using LSA on the reduced vector they are claiming a major performance increase. This is really believable, especially since LSA offers quick tools for folding new information into a model instead of recreating it. Anyways, I am happy to learn that the approach I took with my mail filter appears to be very similar to one of the best computer companies out there. It gives me even more reason to believe that I am on the right trail with all of my various Vector based learning algorithms. Explaining the Apple Mail Filter P.S. This means that my filter that has barely been developed and has many enhancements possible is achieving about .5% less than apples filter!

News Shaker Update

2004-05-18T00:00:00+00:00

After doing some initial work with the 8 category problem, I have run into some problems. Nothing that can�t be solved but just some initial hiccups as expected. The very first run through I was getting approximately 30% accuracy on my categorizations. Better than random guessing but still pretty worthless. After changing to a different layout of the model, I am now getting around 43%. Which 43% (on average of the 8 models some are higher) also sucks. I now have about 5 different ideas after talking with a professor at CU at how to improve my overall percents. I am trying to get over 75% accuracy once I have about that level (which isn�t that high) I am hoping with some user feedback on the site that the model will train and improve itself. Which would be really cool, and possible since pretty much the whole process is automated now. I first was taking all of the categories and creating a positive and a negative vector. The positive was all of the categorized data in the model. The negative was all of the other data in all the other categories. This wasn�t doing so well, so I removed the general category from the negative vector. I also removed the uncategorized data from the negative vector since it is possible these could fit in the category. Doing this increased my model accuracy from the 30% to the 43%. I am now considering other things I could do to improve the accuracy. One of the things I am considering is a two level model. The first would only say if the model relates to special education the second level would then categorize within the special education category. This would allow me to quickly dump anything I know isn�t related to special education at all. It would also allow me on the final site to have users help with the categorization process. Anything that couldn�t be categorized better than just special education related could be placed in a general category. The general category users could view and then place in the proper category which would in turn help train the system. I am also now considering a move from SVMlight to libSVM. Apparently libSVM offers some better options and optimizations, but still uses the same input format. This is important because text2SVM, took awhile and was written with SVMlight in mind. I have done some other optimizations on text2SVM which isn�t included in the released source because the project has begun to become less general and more specific to my project. It has improved and become far faster though. If I move to libSVM this would allow me to get results of a categorization attempt as a percent. If I had percents I could compare the results to different models better which would be useful since the value comparison between models isn�t scaled the same. One of the problems I am running into is testing time. It takes about 2 1/2 hours or so to create and run a new test. It requires a few different steps. If I run them all at once my machine runs out of memory and crashes. So I have to run the steps one at time even though the code is completely automated, it can�t run as such without time to dump the memory. Perhaps I will have to start looking around CU for a gigantic machine that I can use to do testing much faster. The spam filter has gone through over 500 emails now and has an accuracy of 97.5% on unseen new email. This is great, if it wasn�t so specialized to my mail I would make the filter available to everyone. That�s it for now. The good news is I think I am still headed in the right direction and I think I will end up with a capable system. The bad news is that I think it is going to be harder and more time consuming than originally planned. I will be busy with some other stuff and out of town over the next 3 weeks so there will probably be little updated information available on the project.

SVM Mail testing

2004-05-11T00:00:00+00:00

Today I got a chance to let SVM mail sort 252 messages that it has never seen before. The messages were moved sucessfully fromt he inbox to either my real folder, or my spam folder. Out of the 252 messages the system incorrectly categorized 8 emails. Three of the 8 were actually the same message sent out from the frienster network. I have added those messages to real training model. This means on the first really good real world test my filter achieved an 96.8% accuracy. This comparess well to the 98% estimated acuuracy of the model, by categorizing the training data 98% correctly. The system has been through about 400 emails, and my accuracy is now at 97.25 %. I have only retrained the model once including little bits of the new info I have collected. I places all of the incorrectly identified emails in the proper categories and retrained the system. Then just to see if that would make it correctly identify those emails I recategoried them, it cut the misclassifications in half, but some where still missclassified. It seems that forwarded messages with attachments are what it will missclassify still. Other emails seem good. I am currently not working on or extending this project because it was just a testing project for some of my code, which I am now focusing back on my News Shaker project, which is using SVM classification to create a google news like site.

SVMMail

2004-05-07T00:00:00+00:00

I reached a great milestone with SVMMail today. I will be doing more test and releasing more information next week. The initial results are a 97% accuracy on the filter. Also with real world testing (so far a low number of 65 mails), there was only 2 errors (1 false positive) in prediction. I had training data of 550 real emails and 713 spam emails (all of which i collected in the 3 weeks or so that I ahve been working on this project.) I am really excited that I have past the stumbling blocks that I was on the last 3 days where I was actually getting a 0% accuracy because a bug the was generating a pretty much random model. There is currently no web interface and it is all just run directly from java (jbuilder in my case). I will add features like that and the ability to track how many of each type of error my system makes later. This is a great I am really happy with how this is working out.

SVM Spam Filtering

2004-04-18T00:00:00+00:00

Support Vector Machine (SVM) Spam filtering. After working more with SVM and my software to utilize it moving along nicely, I am begining to do work on a spam filter for my email. I have been getting a much larger amount of email in the last couple months. The spam is highly recognizable and simple to see patterns in, 90% of it is related to buying prescription drugs. I figured that SVM should be able to detected this very easily. Since i have already written, a few java email applications, I thought it would be easy enough to write a program that will let me log on and list things as spam and then use that to train and SVM model, which takes far less training data than many other machine learning methods. For now i have done a very simplistic design, the first step will, be building a simple webmail system, it will really only be used to mark as spam, and check the spam folder to make sure there was no false positives. Then I will use my normal mail system to read and respond to the mail. When i first log into my simple webmail all new mail will be download and categorized. I will then be able to submit any missed items as spam. Anything newly categorized as spam or not will be places in either the spam, or real mail section of the database. The database will store who the mail was from, the subject, and the body of the message. I have now created two folders in my mail system. old and spam. I have moved all my old real mail to the old folder and all spam to the spam folder. I have about 300 spam messages and about 520 real messages. I have finished hte code for the mail checking and sorting using java. I now imported my Text2SVM software and used the to create the orginal model file. Now similiarly to my NewShaker software all that is left to do is have autocategorization. For spam this should be a little simpler since there are only two classification levels. This should let me learn the process that I will need to use for Newsshaker, but at a little simpler level. I created a model using 170 of real mail, vs spam. With it I was getting an 80% recognition on my test data, I think with some simple fixes in the way I am creating my text from my mail, stop listing, and stemming I should be able to increase this rate. I should also be able to increase the rate once I add more training data. (This was created as a sub project to create and test code that will be used for newsshaker)
Here is my initial design.

Programming progress

2004-04-13T00:00:00+00:00

Today I accomplished alot on many different programming projects. 3 to be correct. Anyways, I got my comp org program almost done which is nice. I made some really good progress on my text categorization project for work. Then finally for my 3d graphics class i made some good progress on my very simple 3d shooter.

I am happy to have accomplished so much, but on the downside it is 10:30 and i have been programming the entire day on one project or another. Argg…

I swear i will have to do something fun and exciting for all of you soon. Then i will have something worth writting about again, but until then here is a picture of my 3d shooter in progress

BROKEN image

A step closer to fully automated

2004-04-13T00:00:00+00:00

After recent success with my models i wanted to do some much more involved and usefull tests. The only problem was half of the stuff i was doing by hand. I had written some java software Text2SVM toat would help witht he conversions and such, but i had to give it the names of the files and everything myself. I now integrated Text2SVM more into newsshaker. It uses the database to find out the names of all the categories. It recursively sorts through all the text files in given directories. It is quite nice. I ran into some problems where i was running the entire system out of memory even with a full gig given to java, which would cause way to much swapping and slow the hold system down. I rewrote my code to break up the steps into smaller parts using much less memory and it now doesnt crash and runs 2 or 3 times faster. I now can go straight from the database to 9 SVMlight formated text training datasets for models. The next step will be to write some code the generates the models using an interface between java and SVMlight. After that i will be writting some code to keep all the results of the testing organized and worthwhile, which i will be storing in the database. The final step will then be writting code that takes many random documents from my database and trys to categorize them, and stores if they were correctly told to be place in the category from which they came. I will then be storing a matrix of attempts to categorize a category and where it was actually categorized. This should be highly exciting because it will really let me see the info I need to know to make my system worka dn to know how reliably it works, and where the problems are occuring. It would let me for instants see if the space between special education parenting is to similiar and close to special education school to be determined, but if combined would serve as a valid and seperate categorization from everything else. School will probably keep me busy for awhile, but it is nice to be making some good progress and see a first beta version coming into very close view. Then i will have a wonderfull testing and development application for text categorization. Hopefully by the end of summer i will have everything generalized to the point that anyone could add categories and begin maintaining a category and seeing how they can get different results with different information retrival schemes, or using diffferent categorization algorythms besides SVMs. Anyone have some other algorythms worth looking into for text categorization, besides LSA/LSI and SVMs?

Busy

2004-04-09T00:00:00+00:00

Alright i have been really busy on my senior project for the last couple weeks so I haven’t had time to really get any work done. I did however have a meeting with a CU AI professor who is familiar witht he use of SVMs. He thought i was on the right track and was approaching everything properly. He answered the one last question i had before i can really set the system up to start working on its own for categorization. So that was exciting. Now all i really need is a week or two to really spend some time coding. If i had the time i really think i could make a very impressive first run beta of the entire system. Then with some time i think it oculd turn into a pretty cool application. I am hoping to tweak some of the hard coded values to allow for much more flexability allowing hte system to be set up anywhere and work on categorizing any categories the user wants. The categories are currently hard coded into a few of the functions and in the end that wont be the best way to do things. Seperate good news is that my senior project just passed a 1 and 1/2 hour live test with 80 law students and a professor with no errors and no problems. That is the longest time the system has been used continously and by far the most users on the system at once. It is really kind of cool to think about the fact that their was 80 law students actively using something that i was a large part of creating. We are going to run a larger more extensive test on monday where the professor will be braodcasting a bunch of questions in class. We will see how that works out in the end, I am excited about it. The site will move soon, but if anyone is interested in looking at what my senior project can do take a look at http://blackout.cs.colorado.edu/mtroom3/jsp It is an interactive classroom for the law school, so that professors can more easily get feedback and quiz large classes of students. Finally a good friend of mine from my research lab was notice by google and they called her up to talk to her and ask for her resume. So that is some pretty exciting news, that they are actively searching out talent. Perhaps as my project matures they will stumble apon what i am doing.

FIRST GOOD SUCCESS!!

2004-03-30T00:00:00+00:00

Using my Text2SVM after learning how to increase java memory size so that i could do a large test was very successful. I used about 2,700 documents and then put them into SVMlight. I used 90% training data and 10% testing data leaving around 270 tests. I achieved 95% accuracy in training SVM to recognize one text category from another. I am highly excited! Tomorrow i will be testing with several of my smaller categories today i tested with my largest category. If these trends contrinue…. eeehhhh. I was really only hoping to achieve over 80% correct. So lets hope this is the start of something truely wonderful. I know that the results continuing this high are very unlikely, but i am still really excited about the first good results. I am pumped about doing more testing tomorrow!!

Current Percent levels

2004-03-30T00:00:00+00:00

These are the current success levels for models based on different categories. These models were built with the current data in the News Shaker database. They were converted to SVMlight models with the use of Text2SVM without stemming enabled.

NewsShaker: Features needed / bug report V.001

2004-03-29T00:00:00+00:00

Features:* from empty DB to fully functional (no manual data entry)* No code should contain hard coded value refrencing categories* Easy way to define where search should put results default (in a category or in the uncategorized)* Ability for user to say they think the categorization is incorrect (if enough users say this it should be moved to the new category)* Store all SVM results int he database* a way to visualize the distances between the categories..* how many categorizations per category have been correct or incorrect* a way to start over the correct and incorrect after generating new SVM models* SVM weighting with C towards the positive examples* a way to let a user create their own account and their own categories to manage themselves* When administrator is adding a URL to crawl. Should be able to pick depth to crawl and default category that all results will be placed in. * Search entire database or by category using Lucecne* Ability to add single page to a category* More administration features* Ability to start or stop auto categorization* caching the front page* Text2SVM integration* Text2SVM configuration file* ability to create the SVM categories from the web as administrator* create only the dictionary and store it as one function* use stored dictionary to create all needed spaces for SVM models* ability to distinguish between multiple categories instead of single boolean. Bugs:* If the user chooses to move a page from one category to another but doesn’t choose a new category it should do nothing and give them an error.* Crawling from the web doesn’t work anymore* SVM first word spacing?? with the first character removal??? is this still a bug?* Counts for the categories should be switch to be autocounted* Text2SVM runs out of memory on large examples* front page loads to slowly* categories are manually counted let MYSQL do the counting!

Text2SVM

2004-03-09T00:00:00+00:00

This is the main page for “Text2SVM” a java program that converts text documents to the format require for SVMlight. The project is currently in beta. I figured that i should release a early working version for anyone else that has been finding it very difficult to find software to help them work with SVM’s. This project is distibuted as is with no garuntees. So use it try it out. Right now it is written very specifically to handle text files conversion to a single format for SVM light. This should be capable of being edited to support many other formats though. To read more about SVMlight visit their page. If you make any good modifications or changes of this program I would love to hear about it please contact me at ddmayer (at) colorado (dot) edu (spam pertection). You can also leave comments on this page so other users could help solve your problems. To download the code click here. This first release 0.001 include these features:-Import entire directory of files-Save dictionary to generate other word vectors based on same SVMmodel-export files in SVMlight format-Stemming to increase the occurance of words like (stopping and stopped)-The ability to import saved dictionaries and continue adding to the model-debugging code left in What is SVM, and why would i want to convert text to a word vector?Support Vector Machines. They are a method or clustering and machine learning. In this examples we are working with Text categorization using SVM. This programs purpose is to take a bunch of documents in a category and a bunch of documents unrelated to that category. It then creates word vectors out of all the documents. The word vectors then are converted to SVMlights format and given positve weight and negative weight. Anytime the document vector has a possitive weight that means we consider it part of our category. A negative weight means it is outside of our category. After the text is written in this format you can use SVMlight to compare unknown text and see if it belongs in this category. KNOWN ISSUES:- occassionally cuts the first letter off of a document.- uses high amount of memory to run the program add -Xmx256x then the program your trying to run. wc.java is known to run out of memory with large amounts of text.

Week of stuff

2004-03-08T00:00:00+00:00

Well this is going to be a busy week. Instead of doing homework on saturday. I went to ft. Collins because my friend Matt wanted me to come up. Also since steve is in town i have been hanging out with him and doing stuff and there for not getting my work done. Today oddly enough instead of spending my time working on my various school projects, I worked for 6 hours on my work project. I accomplished quite a bit and I am now ready to introduce my Machine Learning blog. It follows my work and progress on some projects that i am working on and serves to be a center for most of my coding and source code. I am sure that doesn’t interest 90% of the people that read my blog, but oh well. I am planning to be really busy until spring break. Which is kind of annoying. I wanted to ask this girl out, but i really dont have any time to go out on a date before spring break. Also, the whole spring break thing is still up in the air. I seriously have no clue what I will be doing. Scott and Dom still dont know when they are off work, how much time they will have and such. Also, I am pretty sure they both would rather just spend it with thier girlfriends. So I am really thinking about just going to vegas with some of my other friends. Also Megan, one of my good friends from highschool is coming to colorado the same week as my break for her springbreak. I would really like to see her, but it is likely that no matter what I do i will be out of town. Maybe I should just flip a coin a couple of times and figure it out. Well now I am off to a statistics review session.. yeaaa school from 7-9! You got to love being inside the entire day when it is beautiful outside.

Text to vectors progress

2004-03-04T00:00:00+00:00

I finished the basic text to vector, but in its current form it is kinda useless. It only outputs to system.out in a format that isn’t usefull yet. Also it doesn’t save the dictionary that it uses to generate the vectors. So you can’t generate any new word vectors to run comparisons with. I will be adding in features to save the generation dictionary, to import a saved dictionary, and to output the feature word vectors in the SVM light format. I also have recently found some cool java stemming software that I will add in to my project after I get the first usable version out. If your interested in the stemming software here are the links: Porter Stemming Algorythm Lancaster Stemmer

Converting documents to word vectors

2004-03-04T00:00:00+00:00

After unsuccessfully searching all over the web for source that would do this, I began writting my own code to convert all of my documents to word vectors based on a overall word space. I have a program that came with LSA called Pindex that does this but for LSA and it seems to give some odd results that I am not sure if they are compatable with SVM’s. (I know that the formatting is different, which I will be converting to SVMlight formatting, but I am not sure if the data is valid.) I am now making a word vector for every document that will just be its total ratio to a total spaces count of those words. I will then have a seperate java app that allows you to give it two sets of documents and it will generate the proper SVM light input file. Giving one set of documents the positive value and the other set the negative. Then I should be able to create take any new text create its word vector based on the same dictionary and then run comparisons in SVMlight. I am hoping to have the text to vector done this weekend. It shouldn’t be to difficult. Then I will release it on the web since i had a next to impossible time trying to find anything like this on the web.

Space Altering

2004-03-02T00:00:00+00:00

I ran some initial results that would tell me how closely related all the documents within a category were related. we were looking for numbers in the .75 range. The first attempt was actually returning the .339 for one category and .328 for another. These were rather low. To improve the mean of the categories we thought creating a larger space with an overall larger vocabulary would help the documents to be more related. We increased the space from 987 total documents to 1798 documents. I didn’t add any documents to either of the test categories, only to the overall space (documents that were really not related to either catgory). I also added a simple stoplist filtering out some common but irrelavent words. This didn’t seem to help the relational mean between the two categories. The first category lowered from .339 to .320 while the second increased from .338 to .349. Now I have to decide wether to greatly increase the space, or greatly increase the documents within the categories (which i have to do by hand so it is a slow and time consuming process.)

Very cool clustering

2004-03-01T00:00:00+00:00

This isn’t quite related to my own machine learning research but it is cool to see what others out there are doing with machine learning. This is a very cool search engine that using text clustering. http://vivisimo.com

Setting Up

2004-03-01T00:00:00+00:00

This is going to be a research blog kept by Dan Mayer to show his progress on his computer science research into machine learning. I will be setting up a collection of all the work and research that I have done on machine learning. I am working on a project for the L3D labs at the University of Colorado. I am working specifically with text categorization, currently focused on using LSA and SVM to categorize. I will post all the links, projects, examples, and resources that I have used and learned from during my progress on this project. This is documenting the work on my project News Shaker, which will be similiar to google news. It uses a web spider and text categorization to organize the data into particular categories. The categories I am currently working with are special education related. The topics of the categories were choosen so that this project would become a sub project of Web2gether, which is a site to help the special education community. Currently all of the categories seen on the News Shaker site are manually sorted and loaded into the system after the spider crawls specific topics on the web. I am now working with LSA and SVMs to begin the text categorization of newly crawled websites and results. This project is also setting up a easy to use web managed content management system. This would allow for the quick set up and design of any topic based categorization using the methods developed while creating the original News Shaker which is focused on special education.

open gl progress

2004-02-16T00:00:00+00:00

Well first sorry about the site being down for awhile, my hosting service had some problems, but they have it fixed. Second, i finished my openGL project for fractal terrains today. This project took me entirely way to long, but it has been hard to move back to c++ after having all the nice things java provides you for so long. Anyways i am slowly getting back on track. So take a look and i hope you all like my little fractals. It is alot cooler if you actually have the program since you can rotate and do all sorts of stuff with the terrain then. Oh well back to laying and now moving.

finished opengl

2004-02-02T00:00:00+00:00

I just finished my first opengl progam. I know this one isn’t 3d but it could be. I just had a hard time moving back to c++ at first since i am so used to java and remembering all the wierd ways to pass things in c++ sucked. Anyways simple but wonderfull since i am done with my first working and well documentated opengl program. I think i have a good understanding of how to work with opengl now as well. Anyways it is a fractal look and enjoy. i know it is simple.

Well today was a big

2004-02-02T00:00:00+00:00

Well today was a big relief… Since my server at work died twice on friday, I have been nervous all weekend that i might have lost all my work. today we recovered the server and i made sure to make backups of all of my files. So now even if the server dies (since we really dont know what the problem was) i wont loose any work. My project at work it moving along nicely. Their are about 900 sites in the database currently and we should be categorizing on the fly by the end of this week. Which will be a huge step. If all goes well i might have a highly successfully project right away. If not with some refinement over a few weeks we should be able to make it fairly successful.

Cool project

2004-01-15T00:00:00+00:00

I just learned that the final project in my computer graphics course is to make a simple 3d Doom like game. I am excited. Depending on the models and if we can make them or we are just using stock models I will try to put a bunch of my friends in the game so that I can kill them and they can all kill each other. Yeaaaa violence. Either way it is going to be my first few steps in the direction of 3D graphics which is really cool and exciting.

OpenGL Resources

2003-11-08T00:00:00+00:00

This is a collection of various tips and things I use or have done in OpenGL. This is actually up on the web more as a refrence to myself than anything else. This is a collection of good links, programs, tutorials, and source code. That I have found usefull, created, or thought would be usefull to others. So enjoy any and all of my OpenGL info. Projects: 3D shooter I wrote, it could obviously be improved alot but it is a good easy start.
You can download the source and the exe and everything and mess with it yourself. This was written in Visual Studio.net Get it
I also made a fractal program into a openGL winamp plug in. If your interested in doing conversions to a winamp plug in yourself, you can download the source and the DLL for winamp and mess around with this.
Get the winamp code Links: A group of scanned 3D object models, and how to read and write them. NEHE the best place to go for open GL resources.

C++ resources

2003-11-04T00:00:00+00:00

This is a collection of various tips and things I use or have done in C+. This is actually up on the web more as a refrence to myself than anything else. I keep learning this stuff, but since I dont use it all that frequently I tend to forget how to do something exactly the next time i need to. So i will just keep a bunch of files that i use for refrence when coding. I don’t use C++ as much now since I have been working mostly with Java at my job. Anyways here is some c++ stuff:

MySQL resources

2003-11-02T00:00:00+00:00

This is a collection of various tips and things I use in MySQL. This is actually up on the web more as a reference to myself than anything else. I keep learning this stuff, but since I don’t use it all that frequently I tend to forget how to do something exactly the next time i need to. So here are little commands I use and some tiny descriptions. If you have some tips tricks or think i am doing something the hard way please feel free to comment and share. If you have any MySQL questions please feel free to post them if you think it is something I might be able to help you with. That said here is mysql information…

<b>MySQL Commands:</b>

//to start mysql on the command line Mysql �u root �p (It will then ask for your password)

//To start mysql server Sudo bash�bin/safe-mysqld� from the mysql root directory

//How to select data using rules involving more than one table from MYSQL: Select * from user leftjoin link on user.id = from Left join msg on to=msg.id where user.name = �john�; Database:mysql -u root �p ***

//snowmass //to create a text dump of a database mysqldump �-user [user name] �-password=[password] [database name] > [dump file]

//to create a new databasemy sql> create database somedb;

//granting privileges to a user grant all privileges on *.* to NewsShaker@�localhost" > identified by 'passwrd';

//shows all the tables of a database show tables;

//creates a new table in the database your currently using create table "tablename" ("column1" "data type", "column2" "data type", "column3" "data type");

//creates a link table (which is really the same as other tables but just//a way to associate data between tablescreate table link (catID bigint, pageID bigint);

//deletes an entire tabledrop table Pages;

//deletes an entry from table idgen where the item has a uid of 12delete from idgen where uid=12;

//show only the colums of a table not the datashow columns from tablename;

//To change the data of an already existing entry update tablename set columname = "whatever" where columnname = "something";

//to alter a table if you need to add a new field alter table tablename add column_name column_type after column_name2;

//creates a dump of database called newsshaker ./mysqldump �u root �p newsshaker > ../../../ddmayer/newsshakerdb.txt//at command line in the mysql/bin directory

//to recreate the database from the dump file ./mysql -u root -pPASSWORD newsshaker < ./newsshakerdb.txt//at the command line in the mysql/bin directory

//to get the count in a category since a date: select count(*) as co from Pages as p left join link as l on p.UID = l.PageID left join categories as c on c.UID = l.CatID where c.name like 'autism' and p.date > 20031201000000 group by c.UID;

//to get only 5 in order as the very newest: select p.UID,p.date,c.UID from Pages as p left join link as l on p.UID = l.PageID left join categories as c on c.UID = l.CatID where c.name like 'autism' and p.date > 20031201000000 order by p.date DESC limit 5;

//to get a whole category: select p.UID,p.date,c.UID from Pages as p left join link as l on p.UID = l.PageID left join categories as c on c.UID = l.CatID where c.name like 'autism';

//add a new user to your mysql system mysql> GRANT ALL PRIVILEGES ON *.* TO 'monty'@'localhost' -> IDENTIFIED BY 'some_pass' WITH GRANT OPTION;

//getting random results from a mysql query or mysql table mysql_query("SELECT * FROM table ORDER BY RAND() LIMIT 1")

//renaming multiple tables at the same time. RENAME TABLE old_table TO backup_table, new_table TO old_table, backup_table TO new_table;

Links:

MySQL home

My SQL commands tutorial

Java MySQL tutorial

Great java Mysql developer tutorial

Linux / Unix Resources

2003-11-02T00:00:00+00:00

This is a collection of various tips and things I use in linux. This is actually up on the web more as a refrence to myself than anything else. I keep learning this stuff, but since I dont use it all that frequently I tend to forget how to do something exactly the next time i need to. So here are little commands I use and some tiny descriptions. If you have some tips tricks or think i am doing something the hard way please feel free to comment and share. If you have any Linux / Unix questions please feel free to post them if you think it is something I might be able to help you with. That said here is what I got… Commands:// secure copy. This allows you to copy a file to another computer over SSHscp filename compname:~/. (exactly like this no spaces may occur anywhere in this) //if something is in the background or you accidently suspended it you//can bring it back to the front by typingfg (in the command line) //if you have sudo access you can do about anything on the systemsudo cmd (any command you want, then it will ask for your password) //to quickly view a fileless filename.txt //to open and edit a fileemacs filename //to add to your path and edit many other settings this in your home directoryemacs .cshrc //to get to your home directorycd (or sometimes cd $HOME) //FTP stuffftp domain.comftp> cd directoryftp> put filenameftp> get otherfileftp> quit //To add users to the systemsudo adduser //edit a users password if your root or can sudosudo passwd username //to change your users password while logged on the systempasswd //to give users sudo accessudo visudo//to run a program and keep the console availableprogram & //if you dont have the proper permissions for a filechmod 666 filename (this gives read and write access to everyone, so it isn’t secure, man chmod if you need a specific file rights) TomCat:To start tomcat run �bin/start-tomcat� from tomcats home directoryTomcat installs /var/tomcat/webappsTomcat really /usr/local/Jakarta-tomcat

Java Resources

2003-11-02T00:00:00+00:00

This is a collection of various tips and things I use in Java. This is actually up on the web more as a refrence to myself than anything else. I keep learning this stuff, but since I dont use it all that frequently I tend to forget how to do something exactly the next time i need to. So here are little commands I use and some tiny descriptions. If you have some tips tricks or think i am doing something the hard way please feel free to comment and share. If you have any Java questions please feel free to post them if you think it is something I might be able to help you with. That said here is some java stuff… First off i think the best way to work with java code and projects is using J Builder. It is available free for personal use. I highly recommend it. It is available for linux and windows. get it from Borland If your going to develop web applications I also highly recommend apache tomcat. It is a free java (JSP) server. It is also available for both windows and linux which is great. get it from Apache Tomcat If a java program keeps running out of memory, this is easy to fix. The java virtual machine is only given so much memory. So running:”java -Xmx256m PROGRAMNAME” should fix the problem by giving the virtual machine 256 mb of memory. You will see this error as a java.lang.outofmemory error. If your using J builder go to project->run->Java VM parameters: and add the -Xmx256m to the parameter line. Here’s how you could rewrite that statement using the ?: operator: System.out.println(“The character “ + aChar + “ is “ + (Character.isUpperCase(aChar) ? “upper” : “lower”) + “case.”); The ?: operator returns the string “upper” if the isUpperCase method returns true. Otherwise, it returns the string “lower”. The result is concatenated with other parts of a message to be displayed. Using ?: makes sense here because the if statement is secondary to the call to the println method. Once you get used to this construct, it also makes the code easier to read. Files:One of the first Java projects I did the source and some info on a AOL Instant messenger (AIM) chatterbot. Built using simpleaim, megahal and some of my own work. Most of the work was in combining the programs learning how they worked and such though. I did add a nice logger to it as well. This still needs alot of work to be that impressive. download MegaDan.zip A part of my machine learning, for project News Shaker. Text2SVM converts text documents into the proper format for use with SVMlight. It is fairly well commented and contains the full source code. This is a good easy way to learn to work with reading and writting files since it has alot of IO. Visit this projects own page, Text2SVM. My edited and improved version of weblech. Weblech is a web spider. I added many features and costumized it for my own purposes. This crawler uses google to find files that are relavant to whatever topics your working on. Download the improved weblech from here Links:A great starting point for building a java web spider (Weblech) My edited and improved version of weblech A great Java Word Counter / dictionary builder Stop List Parsing Text Java MySQL tutorial Java Porter Stemming Algorythm Java Lancaster Stemmer Sorting a hash table in java Java sort class (sorts most types) a great java machine learning site Pretty much everything you would need to work with java Email How to work with multipart email messages Great Java Swing Tutorials, especially the stuff on the grid bag layout Great webservices and java XML info The 10 best and unervalued least known about java libraries. some of these would be great to use in many projects without reimplementing there feautres yourself.

Continuously Deployed

Ruby Caches - Improving Caching Via Tooling

Ruby Cache Improvements

Understanding latency with benchmarks and profiles

Benchmarking Dalli with GitHub Actions

Profiling Dalli with GitHub Actions

Developer workflow leveraging the benchmark and profiling tools

Capture a Benchmark Baseline

Profile the Target method

Refactor, iterate, fix hot spots

Rerun the benchmark

Ship your performance Improvement

Invest in the toolchain

Ruby Caches - Rails Local Cache Strategy

Rails Cache Local Cache Strategy

What is Active Support Local Cache Strategy?

How Strategy::LocalCache works

Local Cache History

When to not use local cache?

Other Rails Cache Strategies?

Ruby Caches - Rails Cache Network Requests

Rails Cache Network Requests

Rails Cache Network Calls

Caches and Network Failures

Testing Network Errors with Toxiproxy

Cache Request Details

Ruby Caches - Rails Cache Workflows

Rails Cache Workflows

Rails.cache.read

Rails.cache.write

Rails.cache.fetch

Notes on Using Rails Cache

Creating Consistent Characters Across Images

The Text Isn’t the Problem

Early Image Problems

Early Attempts to Improve the Images

Improved and Consistent Image Prompts

Image2Image Support

Training LoRA models

Flux LoRA Models

Consistent Characters

A prompt across LoRA Models

Ruby Caches - Rails Cache Comparisons

Rails Caching Comparisons

Rails Cache Stores

Compression

Serialization

How to optimize and benchmark for network architecture and latency

Conclusion

Ruby Caches - Rails Cache Initialization

Rails Cache Initialization

The Rails Cache Configuration

Cache Initialization

Make Your Own

What happens when a Cache Store Initializes

Gardyn 2.0 four years later

Gardyn 2.0 years later

Gardyn 2.0 still worth it years later?

Gardyn Maintenance and Repairs

Conclusion

Rails World 2024

Conferences for Remote Developers

Conferences for Remote Developers (long version)

Rails World 2024 Summary

What is Rails going for?

Rails World 2024 Theme

Increasing Collaboration Confidence with bin/setup

A Repo Quick Start

enter bin/setup

Language Agnostic and Composable Scripts

Verify The Scripts on CI

Verifying bin/setup & bin/start on CI

Bonus Points Verify Eager Load on CI

Enable Any Developer to Quickly Jump Into Any Repo

Making Book Covers with Img2Img

Examples

Book Cover Process

What does adjusting Denoising strength do?

What’s next?

Github Organization Repositories by Ruby Version

Verifying `bin/setup` & `bin/start` on CI