Richard Crowley’s bloghttp://rcrowley.org/index.xml2015-12-14T06:51:31ZRichard CrowleyDevStructure Redux
http://rcrowley.org/2015/12/13/devstructure-redux.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley
<article>
<h1>DevStructure Redux</h1>
<p>Five years ago Matt Tanase and I built a tool called <a href="http://devstructure.com/blueprint/">Blueprint</a> under the banner of a company we called DevStructure. We believed configuring development environments was too difficult and that existing configuration management tools were too hard to use. This, we concluded, resulted in development environments that were too far removed from their production environments and thus offered little proof that software will work well once deployed to production.</p>
<p>We started with a paid semi-hosted development environment and eventually backed off to a set of open-source tools we hoped to support with production hosting offerings. To cut a long story short, we never made it that far. As a business, DevStructure was a failure.</p>
<p>Early this morning while standing in the dark gently bouncing my two-month-old daughter back to sleep I was thinking about what we built and why we failed. If I was ever bitter about our failure as a business I’m long since over it but now I see the world through DevStructure-colored glasses; I see shades of what we tried to do in all sorts of tools being released all across our industry. Perhaps that’s confirmation bias but it keeps me thinking about how I’d approach the problem today. And let’s be very clear: The management of development environments that are reliable predictors of how software will behave in production environments is still a major threat to software quality five years later.</p>
<p>However, since I adore <a href="https://slack.com/jobs">my job</a> and since chasing this idea would be a costly distraction for us right now, DevStructure Redux isn’t going to be coming from me anytime soon. Consider this a free idea or idle ramblings, whichever you choose.</p>
<p>Blueprint was trying to put a better user interface on configuration management and even the idea of infrastructure as code but the tools we built were quite hostile towards code review. Nowadays, I believe code review is essential so I’ve resumed seeking code-centric user interface improvements.</p>
<p>Today’s configuration management suffers as a user interface in part because it is built on command-line tools readily available on the servers being managed. Most operators can resist the temptation to <code>apt-get install ...</code> in production but basically no one can resist in a development environment. And why would they? The edit-save-run cycle is a drag and over time most Chef or Puppet runs tend towards oh-my-god-this-is-slow.</p>
<p>So this is where my mind was at 5 o’clock this morning. I thought that if configuration management was a concern of the operating system then the interface it ultimately exposed could be made more amenable to code review. Here we go with the hand-waving.</p>
<p>Code review-conducive package installation is the easiest to tackle. Whatever the interface, there just needs to be a file or directory tree naming the packages to be installed and specifying their versions. When changes are made the operating system needs to add and remove packages to suit. Changes may be allowed via editor, command-line tool, or pulling changes from an upstream source code repository. Hypothetically this could be as simple as files like <code>/etc/packages/<em>mysql-server</em></code> that contain the desired version number. Version control <code>/etc</code> and send all the changes through code review.</p>
<p>Immediately after installing a package folks tend to configure the newly-installed software. Detecting meaningful changes to these files was always a major problem for Blueprint. Well-behaved packages mark their configuration files and include hashes of the packaged content to enable detection of changes. In my opinion we’d be better-served by packages devoid of configuration files. Programs would thus be required to function sanely by default or fail immediately at the author’s discretion. Imagine how wonderful it would be if this requirement were imposed on all packages: <code>/etc</code> would be empty on a fresh operating system installation! If the defaults don't suit you, create <code>/etc/<em>my.cnf</em></code> containing your desired configuration and commit it, too.</p>
<p>While we’re moving configuration files out of packages, let’s get rid of maintainer scripts, too. I want to control when services are started and restarted. I want to control user and group IDs. And I reserve the right to manage these actions and resources differently in my infrastructure than you do in yours. Store service dependencies in <code>systemd</code> units like <code>/etc/systemd/system/<em>mysql.unit</em></code>. Store users and groups in <code>/etc/passwd</code> and <code>/etc/group</code> but without all the cruft or better yet in <code>/etc/passwd.d</code> and <code>/etc/group.d</code> directories. And version them just like packages and configuration files.</p>
<p>Oddly, despite my desire to move maintainer scripts from packages out into configuration management, I also want to get rid of <code>exec</code> or <code>execute</code> resources that allow today’s configuration management tools to run arbitrary commands. Puppet’s documentation has long said that if you find yourself using a lot of <code>exec</code> resources you’re wrong and should be writing your own types and providers. Everyone ignored that because <code>exec</code> resources are easy. The real problem underlying all of this, though, is that it is easier to manage many aspects of a server today via commands than via files (<code>iptables</code>(8), <code>mount</code>(8), <code>sysctl</code>(8), to name but a few) and tough to manage small changes to large sets of defaults (basically every default configuration file). This needs to change if we're to make anything but superficial lipstick-on-a-pig progress. It's no accident this is starting to sound a little bit like Plan 9.</p>
<p>If we were starting DevStructure again today our company would look a whole lot more like CoreOS than like Chef or Puppet Labs. My vision for a minimal operating system, however, is quite different than that of CoreOS. I envision a base installation chock full of all the tools I could ever need to introspect the system. I envision a package manager that enables and encourages fat packages, explicit dependency versioning and vendoring, and then gets out of the business of configuration and orchestration. I envision configuration management being the only interface for adding and removing software or changing its configuration. I envision that all aspects of the operating system will be designed to be version-controlled. I envision packaging to be the obvious best way to deploy your own software into production.</p>
<p>DevStructure Redux would sell an appliance or service (customer’s choice) that turn this minimal operating system distribution into a streamlined development-to-production pipeline. At a minimum, it would need to accept and serve package artifacts, integrate with test runners and code review tools, store package artifacts installed from the Internet so precise versions are available to all servers, manage server inventory, and orchestrate reconfiguration and deployments. The toughest product challenge is, I think, how to infiltrate brownfield environments. Failure is imminent if this product gives anyone the impression it’s an all-or-nothing proposition; customers need to derive value immediately and in proportion to their investment.</p>
<p>Alas, I will not have the opportunity to take up this cause again for a long time because I love my job. Our development environments at Slack are not terribly portable to other applications or workflows but they are amazingly production-like and highly optimized for our 100+ engineers getting work done. Seeing this scale does nothing but reinforce my belief that development environments should be as close of a match to production as possible. I’m not satisfied with the latest batch of abstractions over the ever more complex software systems being built today but I’ll keep searching. I sincerely hope someone builds what I’ve described above because I’d love to buy it.</p>
</article>
talks/gophercon-2014.htmlhttp://rcrowley.org/talks/gophercon-2014.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley<a href="http://rcrowley.org/talks/gophercon-2014.html">http://rcrowley.org/talks/gophercon-2014.html</a>Certified
http://rcrowley.org/2014/04/15/certified.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley
<article>
<header>
<time datetime="2014-04-15" pubdate>2014-04-15</time>
<h1>Certified</h1>
</header>
<p>It’s a scary Internet out there. Between the growing value of the data exchanged online and the ever-present threat that the NSA decides you’re interesting we’re left with only one responsible thing to do: Protect the confidentiality and integrity of our and our users’ data.</p>
<p>One important part of any strategy to protect our users’ data is encrypting that data when in motion and that means anywhere, anytime. It’s not good enough to only encrypt your payment pages. It’s not good enough to only encrypt traffic between users’ browsers and your load balancers. Every connection that crosses a network should be made via TLS. Period.</p>
<p>Traditionally this is either confusing because OpenSSL or expensive because Verisign. Today I’m introducing a new option. <a href="https://github.com/rcrowley/certified">Certified</a> will help you generate and manage an internal CA for your company and its <a href="https://github.com/rcrowley/certified/wiki">wiki</a> includes a growing collection of examples for the various operating systems, server softwares, and programming languages we all use.</p>
<p>Be safe out there.</p>
</article>
talks/gosf-2014-01-15.htmlhttp://rcrowley.org/talks/gosf-2014-01-15.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley<a href="http://rcrowley.org/talks/gosf-2014-01-15.html">http://rcrowley.org/talks/gosf-2014-01-15.html</a>Go web services with Tiger Tonic
http://rcrowley.org/articles/tiger-tonic.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley
<article>
<header>
<time datetime="2013-12-13" pubdate>2013-12-13</time>
<h1>Go web services with Tiger Tonic</h1>
</header>
<p><em>Cross-posted from <a href="http://blog.gopheracademy.com/day-13-tiger-tonic">Go Advent Day 13 - Go web services with Tiger Tonic</a>.</em></p>
<p>Go is unique among mainstream programming languages in that its standard library web server is not a complete afterthought. The Go language is well-suited for engineering complex networked services and Go’s standard library recognizes that many (if not most) of those services communicate via HTTP. Sprinkle some <a href="https://groups.google.com/forum/#!msg/golang-nuts/BNUNbKSypE0/E4qSfpx9qI8J">Google scale</a> on it and your web applications and services can really hit the ground running.</p>
<p>The standard library sets the tone but it’s far from the end of the story of how to effectively build web services in Go. And as most Go stories do, ours begins with an interface:</p>
<p><pre><code>type Handler interface {
ServeHTTP(ResponseWriter, *Request)
}</code></pre>
<p><code>http.Handler</code> may not look like much but its simplicity is the key to its universality. The standard library provides a number of powerful implementations of this interface but most serious web applications and services are quick to find shortcomings, differences of opinion, and higher-level abstractions that require us to implement <code>http.Handler</code> ourselves.</p>
<h2>Introducing Tiger Tonic</h2>
<p><a href="https://github.com/rcrowley/go-tigertonic">Tiger Tonic</a> packages several <code>http.Handler</code> implementations that make engineering web services easier. They strive to be orthogonal like the features of the Go language itself. And because everything is an <code>http.Handler</code> it’s possible and even easy to use Tiger Tonic in conjunction with your own or anyone else’s handlers.</p>
<p>Tiger Tonic eschews HTML templates, JavaScript and CSS asset pipelines, cookies, and the like to remain squarely focused on building JSON web services. Why JSON? It’s become the lingua franca for serialization in web services everywhere. Why the focus on web services? The authors love <a href="http://dropwizard.codahale.com">Dropwizard</a> and missed all it brought to the Java community so they worked to reproduce that feeling in Go.</p>
<h2>Multiplexing requests</h2>
<p>Most web services respond differently to requests for different URLs so they reach for the standard <code>http.ServeMux</code>. Then these web services start handling requests like <code>POST /games/{id}/bet</code> (an example from <a href="https://developers.betable.com">Betable</a>) and <code>http.ServeMux</code> gets in the way. <code>tigertonic.TrieServeMux</code> is here to help.</p>
<pre><code>mux := tigertonic.NewTrieServeMux()
mux.Handle("POST", "/games/{id}/bet", handler)</code></pre>
<p><code>tigertonic.TrieServeMux</code> multiplexes requests differently than the standard <code>http.ServeMux</code>. URL components surrounded by braces like <code>{id}</code> above are wildcards added to <code>r.URL.Query</code> and otherwise the request must match the method and path of the matching handler exactly. In the author’s experience this is the least surprising thing to do. Plus it gives the framework the opportunity to respond 404 and 405 to requests it cannot fulfill. See? Your web service is already a good HTTP citizen!</p>
<p>And as cool as that is it’s even cooler that your web service is still an <code>http.Handler</code> that you can <code>http.ListenAndServe</code> without a second thought.</p>
<h2>JSON in, JSON out</h2>
<p>So now the question is how we respond to all the requests we’re now multiplexing. This is where we start to specialize on web services. Once again, the standard library has giant shoulders on which to stand. Import <code>encoding/json</code> and you’re off to the races, right? Well:</p>
<pre><code>func handlerFunc(w http.ResponseWriter, r *http.Request) {
err := json.NewEncoder(w).Encode(&MyResponse{/*...*/})
if nil != err {
fmt.Fprintln(w, err)
}
}</code></pre>
<p>That’s a bit clumsy. <code>tigertonic.Marshaled</code> allows you to construct handlers from functions that automatically deserialize request bodies according to the type of the final argument and serialize response bodies from the error or response object returned.</p>
<pre><code>var handler http.Handler = tigertonic.Marshaled(func(
url.URL, http.Header, *MyRequest,
) (int, http.Header, *MyResponse, error) {
return http.StatusOK, nil, &MyResponse{/*...*/}, nil
})</code></pre>
<p>This proves to be a powerful higher-level abstraction that removes JSON serialization from the web service programmer’s long list of concerns. And just as before, handlers returned by <code>tigertonic.Marshaled</code> understand how to respond 400, 406, and 415 like Roy Fielding intended.</p>
<h2>Testing</h2>
<p>Perhaps the best feature of this new abstraction is its effects on testing web services. No disrespect to the <code>net/http/httptest</code> package or <code>httptest.ResponseRecorder</code> but Go web services should test their responses not their serializations. For example:</p>
<pre><code>func TestAdvent(t *testing.T) {
s, _, rs, err := handler(
mocking.URL(mux, "POST", "/games/ID/bet"),
mocking.Header(nil),
nil,
)
if nil != err {
t.Fatal(err)
}
if http.StatusOK != s {
t.Fatal(s)
}
if "win" != rs.Outcome { // Merry Christmas!
t.Fatal(rs)
}
}</code></pre>
<p>That’s far more precise than matching strings with regular expressions and far less verbose than deserializing the JSON response.</p>
<p>For those keeping score at home: yes, <code>tigertonic.Marshaled</code> also returns an <code>http.Handler</code>.</p>
<h2>Extra batteries</h2>
<p>Though <code>tigertonic.TrieServeMux</code> and <code>tigertonic.Marshaled</code> are the main attractions, Tiger Tonic packages a number of other useful handlers that make building web services with Go easier:</p>
<ul>
<li><code>tigertonic.TrieServeMux</code> has <code>HandleNamespace</code> to match and remove prefixes from requested URLs.</li>
<li><code>tigertonic.HostServeMux</code> supports virtual hosting of many domains in a single Go process.</li>
<li><code>tigertonic.First</code> and <code>tigertonic.If</code> enable handler chaining a la Rack or WSGI middleware.</li>
<li><code>tigertonic.Counted</code> and <code>tigertonic.Timed</code> emit metrics about all your requests via <a href="https://github.com/rcrowley/go-metrics"><code>go-metrics</code></a>.</li>
<li><code>tigertonic.WithContext</code> and <code>tigertonic.Context</code> add support for strongly-typed per-request context.</li>
<li><code>tigertonic.HTTPBasicAuth</code> is a specialization of <code>tigertonic.If</code> that conditionally handles requests if an acceptable <code>Authorization</code> header is present.</li>
<li><code>tigertonic.CORSBuilder</code> and <code>tigertonic.CORSHandler</code> facilitate setting the basic CORS response headers.</li>
<li><code>tigertonic.Server</code> has <code>CA</code> and <code>TLS</code> methods that simplify listening for TLS connections.</li>
<li><code>tigertonic.Configure</code>, in conjunction with method values, makes it easy to read configuration files.</li>
</ul>
<h2>Go forth</h2>
<p><a href="https://github.com/rcrowley/go-tigertonic">Tiger Tonic</a> is available on GitHub an includes a complete <a href="https://github.com/rcrowley/go-tigertonic/tree/master/example">example</a> that covers all the handlers included.</p>
<p><code>http.Handler</code> is the common currency for Go web frameworks of all shapes and sizes. The handlers in the <code>tigertonic</code> package are meant to make engineering web services more correct, more efficient, and more testable but because they’re handlers they’re at your service and up to the challenge just as long as you stick to the humble little <code>http.Handler</code> interface.</p>
</article>
talks/basho-chat-2013-12-03.htmlhttp://rcrowley.org/talks/basho-chat-2013-12-03.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley<a href="http://rcrowley.org/talks/basho-chat-2013-12-03.html">http://rcrowley.org/talks/basho-chat-2013-12-03.html</a>talks/opsmatic-2013-11-19.htmlhttp://rcrowley.org/talks/opsmatic-2013-11-19.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley<a href="http://rcrowley.org/talks/opsmatic-2013-11-19.html">http://rcrowley.org/talks/opsmatic-2013-11-19.html</a>talks/strange-loop-2013.htmlhttp://rcrowley.org/talks/strange-loop-2013.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley<a href="http://rcrowley.org/talks/strange-loop-2013.html">http://rcrowley.org/talks/strange-loop-2013.html</a>talks/surge-2013.htmlhttp://rcrowley.org/talks/surge-2013.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley<a href="http://rcrowley.org/talks/surge-2013.html">http://rcrowley.org/talks/surge-2013.html</a>talks/puppetconf-2013.htmlhttp://rcrowley.org/talks/puppetconf-2013.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley<a href="http://rcrowley.org/talks/puppetconf-2013.html">http://rcrowley.org/talks/puppetconf-2013.html</a>talks/agile-2013.htmlhttp://rcrowley.org/talks/agile-2013.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley<a href="http://rcrowley.org/talks/agile-2013.html">http://rcrowley.org/talks/agile-2013.html</a>Graceful stopping in Go
http://rcrowley.org/articles/golang-graceful-stop.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley
<article>
<header>
<time datetime="2013-04-27" pubdate>2013-04-27</time>
<h1>Graceful stopping in Go</h1>
</header>
<p>It’s rude to deploy early and often if deploys interrupt user requests so we build our Go services at Betable to stop gracefully without interrupting anyone. The idea is to stop listening, presumably so a new process can take over, and let all open connections respond to any in-progress requests before finally stopping service. Incidentally, we use <a href="https://github.com/rcrowley/goagain"><code>goagain</code></a> to restart without even stopping listening but that’s beyond the scope of this article.</p>
<p><code>main.main</code> does four things: listens, makes a <code>Service</code> and sends it into the background, blocks until signaled, and afterwards stops the service gracefully. Listening and signal handling are right out of the standard library playbook except that, to my great annoyance, you need a <code>*net.TCPListener</code> or <code>*net.TCPConn</code> and not merely a <code>net.Listener</code> or <code>net.Conn</code> to call <code>SetDeadline</code>.</p>
<p><code>service.Serve(listener)</code> accepts connections and handles each one of them in its own goroutine. It sets a deadline so <code>listener.AcceptTCP()</code> doesn’t block forever and between successive iterations checks to see if it should stop listening.</p>
<p><code>service.serve(conn)</code> reads and writes and likewise does so on a deadline. It sets a deadline so <code>conn.Read(buf)</code> doesn’t block forever and between writing a response and reading the next request or after the <code>conn.Read(buf)</code> deadline checks to see if it should close the connection.</p>
<p>The various goroutines decide to close connections and listeners if the service’s channel receives a value which, because nothing ever sends on this channel, is only possible after <code>service.Stop()</code> has closed the channel. Recall the built-in <a href="http://golang.org/ref/spec#Close"><code>close</code></a> function that causes channels to forevermore receive the zero-value of the channel’s element type.</p>
<p>The final piece of the puzzle is waiting for all goroutines to die which is implemented via the standard library’s <code>sync.WaitGroup</code>.</p>
<p><a href="https://gist.github.com/rcrowley/5474430">https://gist.github.com/rcrowley/5474430</a> has it all:</p>
<script src="https://gist.github.com/rcrowley/5474430.js"></script>
</article>
Go acknowledgement test
http://rcrowley.org/2013/04/19/golang-ack-test.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley
<article>
<header>
<time datetime="2013-04-19" pubdate>2013-04-19</time>
<h1>Go acknowledgement test</h1>
</header>
<p>I have been writing a lot of Go lately at <a href="https://betable.com">Betable</a> and the channels of channels pattern comes up so often that I broken down and benchmarked it. I’m publishig this more widely at the insistence of Paul Hammond who would agree that this is yet another of the many and persistent reminders that Rob Pike is smarter than I am.</p>
<p>In summary, it is faster to allocate a one-time-use channel for a goroutine to acknowledge a request than to use a <code>sync.Mutex</code>.<p>
<script src="https://gist.github.com/rcrowley/5423530.js"></script>
</article>
talks/nodejs-2013-02-19/http://rcrowley.org/talks/nodejs-2013-02-19/2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley<a href="http://rcrowley.org/talks/nodejs-2013-02-19/">http://rcrowley.org/talks/nodejs-2013-02-19/</a>Federated Graphite
http://rcrowley.org/articles/federated-graphite.html2015-12-14T06:51:31Z2015-12-14T06:51:31ZRichard Crowley
<article>
<header>
<time datetime="2012-08-03" pubdate>2012-08-03</time>
<h1>Federated Graphite</h1>
</header>
<p>Graphite is a nearly undocumented thing of sweetness. We at Betable collect, store, and graph many tens of thousands of metrics and we quickly reached the limits of a single-instance Graphite infrastructure. That’s OK, though, because Graphite does actually support some pretty reasonable federated architectures if you know where to look. Consider this your treasure map.</p>
<p>Aside: <a href="https://gist.github.com/2311507">https://gist.github.com/2311507</a> is how I build Debian packages for Graphite which I deploy via <a href="https://github.com/rcrowley/freight">Freight</a>.</p>
<h2>Firewalls</h2>
<p>You probably already have your firewalls configured to allow your entire infrastructure to send metrics to Graphite on port 2003. Your Graphite almost-cluster is going to need a bit more permission to communicate with itself.</p>
<p>Open ports 2014, 2114, and so on as you need between all Graphite nodes. These are how the <code>carbon-relay.py</code> instance on each Graphite node will communicate with the <code>carbon-cache.py</code> instance(s) on each Graphite node.</p>
<p>Also open the port on which the web app listens to other Graphite nodes. This is how the web apps will communicate with the other web apps.</p>
<h2><code>carbon-cache.py</code></h2>
<p>Each <code>carbon-cache.py</code> instance should listen on a non-loopback interface and on a port that’s open to the <code>carbon-relay.py</code>s. I have found two instances of <code>carbon-cache.py</code> per Graphite node to be advantageous on the Rackspace Cloud (YMMV). Their pickle receivers listen on 2014 for instance <code>a</code> and 2114 for instance <code>b</code> on each Graphite node.</p>
<p>Each <code>carbon-cache.py</code> instance must have a name unique with respect to all <code>carbon-cache.py</code> instances on all Graphite nodes. For example: <code>1.2.3.4:2014:a</code>, <code>1.2.3.4:2114:b</code>, <code>5.6.7.8:2014:c</code>, <code>5.6.7.8:2114:d</code>. This isn’t obvious from the documentation or even from reading the code, so keep this in mind.</p>
<h2>Whisper databases</h2>
<p>I started the migration from one Graphite node to two by <code>rsync</code>ing all Betable’s Whisper databases from the old node to the new node.</p>
<pre><code>rsync -avz --exclude=carbon graphite-ops.betable.com:/var/lib/graphite/whisper /var/lib/graphite/</code></pre>
<p>The original plan (which is executed below) was to go back later and “garbage collect” the Whisper databases that didn’t belong after the Graphite cluster was up-and-running. If you’re feeling adventurous, you should use <code>whisper-clean.py</code> from <a href="https://gist.github.com/3153844">https://gist.github.com/3153844</a> to choose which Whisper databases to <code>rsync</code> and delete up-front because a Whisper database on the local filesystem will be preferred over querying a remote <code>carbon-cache</code> instance.</p>
<h2><code>carbon-relay.py</code></h2>
<p>Each Graphite node should run a single <code>carbon-relay.py</code> instance. It should listen on 2003 and 2004 so your senders don't have to be reconfigured.</p>
<p>List all <code>carbon-cache.py</code> instances on all Graphite nodes in <code>DESTINATIONS</code> in <code>carbon.conf</code>, each with its <code>PICKLE_RECEIVER_INTERFACE</code>, <code>PICKLE_RECEIVER_PORT</code>, and instance name. Sort the instances by their name. This is what allows metrics to arrive at any Graphite node and be routed to the right Whisper database.</p>
<p>To use consistent hashing or not to use consistent hashing? That’s your own problem. I use consistent hashing because I have better things to do than balance a Graphite cluster by hand.</p>
<p>With this configuration, Graphite’s write path is federated between two nodes. Its read path, however, appears to be missing half its data.</p>
<h2>Web app</h2>
<p>Each Graphite node should run the web app, even if you don’t plan to use it directly.</p>
<p>Each web app should list the <strong>local</strong> <code>carbon-cache.py</code> instances in <code>CARBONLINK_HOSTS</code> each with its <code>CACHE_QUERY_INTERFACE</code>, <code>CACHE_QUERY_PORT</code>, and instance name. Sort the instances by their name as before so the consistent hash ring works properly. If you’re not using consistent hashing, sort the instances by their name to appease my OCD.</p>
<p>Each web app should list the other web apps in <code>CLUSTER_SERVERS</code>, each with its address and port. If a web app lists itself in <code>CLUSTER_SERVERS</code>, it’s gonna have a bad time.</p>
<h2>Garbage collection</h2>
<p>Once you’re satisfied with your Graphite cluster, it’s time to collect the garbage left by <code>rsync</code>ing all those Whisper files around. <code>whisper-clean.py</code> from <a href="https://gist.github.com/3153844">https://gist.github.com/3153844</a> does exactly that.</p>
<p>(Of course, the usual disclaimers about how this deletes data but is only known to work for me apply so tread lightly.)</p>
<p>On <code>1.2.3.4</code>:</p>
<pre><code>DJANGO_SETTINGS_MODULE="graphite.settings" python whisper-clean.py 1.2.3.4:a 1.2.3.4:b -5.6.7.8:c -5.6.7.8:d</code></pre>
<p>On <code>5.6.7.8</code>:</p>
<pre><code>DJANGO_SETTINGS_MODULE="graphite.settings" python whisper-clean.py -1.2.3.4:a -1.2.3.4:b 5.6.7.8:c 5.6.7.8:d</code></pre>
<p>If you happen to have been smarter than I was with your <code>rsync</code>ing, this step probably won’t be necessary.</p>
<h2>Configuration management</h2>
<p>None of this configuration is done by hand. We use Puppet but Chef works just fine. I highly recommend using <a href="http://docs.puppetlabs.com/guides/exported_resources.html">Exported Resources</a>, <a href="https://github.com/rcrowley/puppet-related_nodes"><code>puppet-related_nodes</code></a>, or <a href="http://wiki.opscode.com/display/chef/Search">Chef Search</a> to keep the Graphite cluster aware of itself.</p>
<p>Betable’s Graphite cluster learns its topology from <code>related_nodes</code> queries like</p>
<pre><code>$carbonlink_hosts = related_nodes(Graphite::Carbon::Cache, true)
$cluster_servers = related_nodes(Package["graphite"])
$destinations = related_nodes(Graphite::Carbon::Cache, true)</code></pre>
<p>that pick up Graphite nodes and the <code>graphite::carbon::cache</code> resources declared with each <code>node</code> stanza. These enforce that instance names (<code>a</code>, <code>b</code>, <code>c</code>, <code>d</code>, etc.) are unique across the cluster. Each <code>graphite::carbon::cache</code> resource generates a SysV-init script and a <code>service</code> resource that runs a <code>carbon-cache.py</code> instance.</p>
<p>The <code>carbon.conf</code> and <code>local_settings.py</code> templates Puppet uses are in <a href="https://gist.github.com/3248921">https://gist.github.com/3248921</a>.</p>
</article>