From Excel to TSV to Rust

Wed, 22 Jan 2020 00:00:00 +0000

I'm trying to build a simple card game, but that's not even the horrifying part of this story. The horrifying part is that I'm keeping my data in Microsoft Excel.

It's perfect! It flattens the data into something that can be more easily expressed as an object, and the relationships between cards are easy to model. Should one card always be 20% more expensive than another, or half as expensive? Easy solution: use an Excel formula. Spreadsheets were built for this. But reading a spreadsheet from a game is madness.

Initially I exported the spreadsheet as Tab-separated Values (TSV), a simple format that obeys CSV rules and Excel can export natively. This happily reduced the problem to reading TSV, but as I wrote a Rust parser for reading that TSV there were enough unwraps and error handling occurring that I decided it would be easier to move all this error-prone part further back in my asset pipeline.

So I decided to turn TSV directly into a .rs</code> file. This is the madness, the horrifying part which is why someone ought to take my keyboard away.

My go-to language for hacky things is Ruby, so I wrote a small Ruby script which can read the TSV and then echo it back out using an ERB template to turn it into Rust code.

# tsv2rs.rb
#!ruby

require 'csv'
require 'erb'

TSV='ARCOMAGE_CARDS.txt'
RST='src/cards.rs.erb'
RS ='src/cards.rs'

def value_or_default(value, default)
  if value == nil or value.empty?; default; else value; end
end

File.new(RS, File::CREAT|File::TRUNC|File::RDWR, 0644)
    .write ERB.new(File.read(RST)).result(binding)
</code></pre>
It's very simple, aside from the helper function value_or_default</code> it's just
two lines: open a file, then write out the result.</p>
The ERB template is where frightening things happen, where we write code which
writes code. But it's not that scary, actually. I want something static, but
also in a Vec</code>, so lazy_static</code> showed up. After that it's very simple Rust
constructors.</p>
use lazy_static::lazy_static;
use crate::card::*;

lazy_static! {
  pub static ref CARDS: Vec<Card> = vec![<%
  first = true
  CSV.foreach(TSV, col_sep: "\t") do |row|
    # skips header rows, assumes your first column is always a number
    next unless row.first.match? /^[0-9]+$/
  %>
  <% unless first %>,<% end %>
  <% first = false %>
  Card {
    atlas_row: <%= row[0] %>,
    atlas_col: <%= row[1] %>,
    category: Category::<%= row[2].capitalize %>,
    card_name: "<%= row[3] %>".to_owned(),
    cost: PlayCost {
      bricks: <%= value_or_default row[4], 0 %>,
      ...
    }
    ...
  }
  <% end %>
  ];
}
</code></pre>
There's some complexity in keeping track of the first time the loop has been
iterated over so a comma can be printed between items, but otherwise it's plain
ERB that shouldn't look terribly unfamiliar to anyone who's used Ruby on Rails
before. Or Jinja with Python.</p>
I highly suggest running cargo fmt</code> afterward, because ERB isn't well known for
cleaning up whitespace effectively.</p>
Following this, it was time to glue everything together. For this I found that
PowerShell worked marvelously. I found a StackOverflow</a> answer which
provided the basics of a script which can save Excel files as other formats,
then the Microsoft documentation</a> provided the right format code for the
TSV I want. Finally, the command runs the Ruby script from earlier.</p>
# xlsx2tsv2rs.ps1
$ExcelFile = Join-Path $(Get-Location) ARCOMAGE_CARDS.xlsx
$TsvFile = Join-Path $(Get-Location) ARCOMAGE_CARDS.tsv
$TsvFmtCode = -4158
$Excel = New-Object -Com Excel.Application
$Excel.DisplayAlerts=$False
$WorkBook = $Excel.Workbooks.Open($ExcelFile)
$WorkBook.SaveAs($TsvFile, $TsvFmtCode)
$Excel.quit()
ruby tsv2rs.rb
</code></pre>
The natural next step for this is to use a build.rs</code> file to invoke this
automatically when cargo build</code> is run, however, there's a reason why I don't</em>
think that's a good idea: it destroys cross-platform compatibility. The way it
is now, I can rebuild my static objects when I'm on a Windows PC which has
PowerShell, Excel, and Ruby. While PowerShell is now cross-platform, the only
other native platform for Excel is Mac. I'm not even sure if PowerShell on Mac
would interface with Excel for Mac the way it has on Windows.</p>
And that's how I learned to convert Excel to Rust.</p>


Level up your Crate
Wed, 27 Mar 2019 00:00:00 +0000
This is adapted from a talk given on 27 March 2019 at Desert Rust</a>, a meetup group of Rust enthusiasts in Mesa, Arizona. If you live in the area or are just visiting during a meetup, please stop by - we'd love to see you there!</p>

You probably already know how to make a Rust crate.</p>
cargo new --lib crate
</code></pre>
You've probably written some Rust code in it, code that is good and ought to be shared. You've published it to crates.io, and nothing happened. Nobody appreciated it, why?</p>
When I was a young boy and still foolish, my brother had a soccer (football) coach who would tell him, "if you look good, you play good." In the spirit of that council, I believe that even the best software will be ignored if it's not dressed up to impress.</p>
The following tips can help juice up your crate and help it sell itself to potential users.</p>
The elements of a solid crate, unless you know better ones, are:</p>

README</li>
Documentation</li>
Tests</li>
Continuous Integration (if applicable)</li>
</ul>
README</h2>
The README is your 60-seconds in an elevator pitch to your potential user. This is the part of your craft where you should throw modesty to the wind, because this is advertising, and advertising is war.</p>
I organize my README into four parts. Everything else is extraneous and belongs in separate documents. These parts are:</p>

What is this? What does it do?</li>
How does this work? No, how does it work for me? (Best approached with a short code sample).</li>
How do I get it? (This is Rust, so I assume it's a crate).</li>
Can I use this? By this I mean licensing - not everybody can use every license of software.</li>
</ol>
I try to be as customer-centric as possible when writing these. The template for my idea of a perfect README looks like the following:</p>
vim README.md
# Crate Name
Elevator pitch! What does this do, and why is it awesome
at doing it?

```
fn main() {
  println!("Code sample!");
}
```

Elaboration about what this does, with more code samples,
or a link to a wiki or some other documentation.

# License
[2-Clause BSD](https://opensource.org/licenses/BSD-2-Clause)
</code></pre>
It's short, and it frontloads the most pertinent information to someone who's shopping for crates and may have dozens of tabs open for research.</p>
Remember to add your README to your Cargo.toml</code> file so it gets picked up by cargo build</code> and shows upon your Crates.io listing page as well, such as with throttle</a>!</p>
vim Cargo.toml
[package]
readme = "README.md"
</code></pre>
While it may seem trivial, this is another point of contact to make it easy for potential customers to quickly learn about your crate.</p>
Badges</h3>
Badges are little images which automatically updated and tell consumers information about your crate. By being on Crates.io, you get the these two badges for free,</p>

a crates.io current version badge, which looks like , and</li>
a docs.rs current version badge, which looks like .</li>
</ol>
Use these badges to link readers on your Github/Gitlab page to crates.io or to docs.rs.</p>
I like to add these to my README directly below the Crate name, but just before the elevator pitch.</p>
vim README.md
# Crate name

[![Crates.io][cios]][cio] [![Docs.rs][drss]][]drs

Elevator pitch!

...

[cio]: https://crates.io/crates/crate-name
[cios]: https://img.shields.io/crates/v1/crate-name.svg
[drs]: https://docs.rs/crate-name
[drss]: https://docs.rs/crate-name/badge.svg
</code></pre>
You can also add these links to your Cargo.toml</code> so they are visible from your Crates.io listing as well.</p>
vim Cargo.toml
[package]
documentation = "https://docs.rs/crate-name"
repository = "https://github.com/..."
</code></pre>
Documentation</h2>
Documentation is what separates code for me from code for you - and code for future me who has forgotten how everything works.</p>
You've probably already worked out that Rust documentation is in CommonMark (or Markdown) format. Docs for ordinary items, such as structs, fields, and functions, are given by three slashes.</p>
/// Control the rate of novertrunnions.
pub mut novertrunnion_choke: f32;
</code></pre>
Bigger ideas, such as how an entire crate is to be used or a module within that crate, are given by module docs, which use two slashes and a bang!</p>
//! For a number of years now, work has been proceeding
//! in order to bring perfection to the crudely conceived
//! idea of a transmission that would not only supply
//! ...
</code></pre>
Tests</h2>
You probably don't write bugs, but I do, and I try to prove that my code works using tests. Writing good tests is among the harder things to do, and entire books have been written on the subject, so I'll spare you any incomplete thoughts on the subject here.</p>
The default crate template comes with a stub for you to fill in some tests:</p>
vim src/lib.rs
...
#[cfg(test)]
mod tests {
  #[test]
  fn it_works() {
    assert!(1 + 1 == 2);
  }
}
</code></pre>
Cargo comes with a built-in test runner, so running those tests is remarkably easy.</p>
➜  retry git:(master) ✗ cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running target/debug/deps/mysteriouspants_retry-6c4c9e88511db7ca

running 2 tests
test tests::succeeds_after_two_retries ... ok
test tests::fails_after_exhausting_retries ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests mysteriouspants-retry

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
</code></pre>
Going further, we arrive at my favorite Rust feature, the part that I think takes Rust from a toy language to something capable of building durable software: doctests.</p>
cargo test</code> will try to run code samples in your documentation, so if you change your crate's implementation, you can easily catch idiosyncrasies in your documentation's code examples. This encourages you to document using prose and</em> code, creating APIs with rich reference material.</p>
/// The main winding was of the normal lotus-o-delta type
/// placed in pandndermic semi-boloid slots of the stator,
/// ...
///
/// ```rust
/// # extern crate mysteriouspants_rockwell;
/// # use rockwell::TurboEncabulator;
/// # fn main() {
/// // create a new TurboEncabulator
/// let mut encabulator = TurboEncabulator {
///   cardinal_grammeters: 3
/// };
/// assert_eq!(encabulator.cardinal_grammeters, 3);
/// # }
</code></pre>
Any code sample line that does not begin with a #</code> (octothorpe) will appear in your documentation, letting you de-noise what shows up on docs.rs.</p>
Travis</h2>
The last thing I look for in a well-constructed crate meant for third-party consumption is some kind of Continuous Integration. For Open-Source, the easiest I know of it Travis CI</a>.</p>
Tell Travis how to build your Crate by telling it that you're writing Rust in a .travis.yml</code> file.</p>
vim .travis.yml
language: rust
rust:
  - stable
  - beta
  - nightly
matrix:
  allow_failures:
    - rust: nightly
fast_finish: true
</code></pre>
Using Travis gets you another badge for your README, as well.</p>
vim Cargo.toml
[badges]
travis-ci = { repository = "github-user/repo" }

vim README.md
[other badges from before] [![Build Status][traviss]][travis]

...

[travis]: https://travis-ci.org/github-user/repo
[traviss]: https://travis-ci.org/github-user/repo.svg?branch=master
</code></pre>

If you've followed these suggestions, I consider your crate as having leveled up! Potential users can</p>

see what your crate is about at a glance in your README, both on your repo page and on crates.io,</li>
read detailed documentation and examples to confirm that what you've written is what they're after, and</li>
see that your code does everything you say that it does with a passing CI build!</li>
</ul>

These suggestions are</em> unless you know better ones. If you do, please open an issue or PR on github</a> and be clear it's this post you think ought to be revised.</em></p>


Headfirst into Rust - Throttle and Mutability
Sat, 01 Sep 2018 00:00:00 +0000
In the first iteration of my Throttle, I packaged the state of the
throttle into a Cell</code></a> so that an otherwise immutable throttle
could still function.</p>
pub struct Throttle<TArg> {
  delay_calculator: Box<Fn(TArg, Duration) -> Duration + Send + Sync>,
  state: Cell<ThrottleState>
}
</code></pre>
This has the interesting side effect of making the throttle unsafe in
threaded environments. Two threads could attempt to mutate the state at
the same time, which would have undefined results.</p>
Fishing for opinions, the suggestion to use an RwLock</code></a> to
control access to the Cell</code> came up. The implication being that, if two
threads try to modify the state of a throttle at the same time, then the
second thread to do so will spin wait until the throttle becomes
available again. This is acceptable behavior, since the throttle may
just wait that thread again anyway as soon as it acquires the lock.</p>
Looking at RwLock</code>, however, it is not a happy story. Getting a lock
may fail and that requires propagating that failure to the caller. This
would make the implementation for acquire look something like the
following:</p>
impl <TArg> Throttle<TArg> {
  pub fn acquire(&self, arg: TArg): LockResult { ... }
}
</code></pre>
Callers would have to match on the result or call unwrap</code> on the result
to determine if there was an error or not, which is more complicated
than I want the API to be in its most basic case.</p>
In my opinion, how to handle concurrency failures is not a concern of
the throttle, so instead I chose to make the throttle itself mutable.</p>
pub struct Throttle<TArg> {
  delay_calculator: Box<Fn(TArg, Duration) -> Duration + Send + Sync>,
  state: ThrottleState
}

impl <TArg> Throttle<TArg> {
  pub fn acquire(&mut self, arg: TArg) { ... }
}
</code></pre>
Rust's borrow checker forces the caller to handle thread safety (or
unsafety), in a way that is most sensible to the caller. In the simple
case: there is only a single thread, so the API remains clean of any
concurrency; in the complex case: the consumer can use whatever
concurrency methods are necessary.</p>
In the future I think it would be wise to refactor this down into a
trait for Throttles, with specializations for different kinds of
throttling - simple throttles, closure-controlled throttles, threadsafe
closure-controlled throttles, throttle-buckets, threadsafe throttle
buckets, and so on. But as I iteratively dive deeper into the Rust
language, such things are fun thoughts that can stand to be pondered a
while longer.</p>


Headfirst into Rust - Building a Throttle
Wed, 06 Jun 2018 00:00:00 +0000
This is a development journal of my first Rust crate, Throttle</a>, which I built over the course of a weekend and a few weeknights. I based my crate authorship standard on literally a single livecoding session by Jon "Jonhoo" Gjengset</a>, took courage in the Jeremy Clarkson motto "how hard can it be?" and set to the task. The follows how I wrote some really bad code and made a crate.</p>
Most of the code listings are reconstructions and not completely accurate to what I eventually hammered out which compiled.</em></p>
The Throttle</h2>
I decided to build a TPS throttle because I want to build some tooling around Squizz' zKillboard</a> for EVE Online. First of my concerns is that I don't get blocked for spamming his website too much - and to slow down a program I reckon I want a throttle. I started by using the default crate template I got out of IntelliJ, and in the testing section wrote out how I wanted to use the throttle. I got something like the following:</p>
// simple throttle configured for 10 TPS
let throttle = Throttle::new(10);

let iteration_start = Instant::now();

for _i in 0..11 {
    throttle.acquire();
}

// prove that it waited by showing that one second has elapsed
assert_eq!(iteration_start.elapsed().as_secs() == 1, true);
</code></pre>
The basic idea is that by calling acquire</code> you're really saying "call thread::sleep</code> if and only if I need to slow down," and by making it an object it could even be used by multiple threads of execution at the same time (using a lock). That way multiple threads could be pulling work, and all safely not viciously slaughtering zKillboard.</p>
I started with a perusal of the standard library and found some APIs which will form the backbone of the throttle: Duration</code></a>, Instant</code></a>, and sleep</code></a>.</p>
Thus armed, I charged in and created a struct which looked something like the following, and then the first problem happened.</p>
#[derive(Copy, Clone)]
enum ThrottleState {
    Uninitialized,
    Initialized {
        previous_invocation: Instant
    }
}

pub struct Throttle {
  tps: f32,
  previous_invocation: Cell<ThrottleState>
}

impl Throttle {
  fn new(tps: f32) -> Throttle { ... }
  fn acquire(&self) { ... }
}
</code></pre>
Yes, the first problem happened before I had even implemented the methods. My first problem was a single thought:</p>

wouldn't it be cool if the throttle could be dynamic, getting faster and slower?</p>
</blockquote>
I write Java in my day job (trying to transition to Kotlin, but for present purposes the reflex is the same), so the concept of "thing that tells how long to delay" feels like a dependency and should not be an aspect of the Throttle</code> itself, and it didn't feel like something that I should slap Cell<f32></code> as the type for tps</code> and call it done, either.</p>
Well you say dependency and so I try to make a bean or at least a struct. That was my first problem.</p>
Storing a Trait, or Don't Write Java in Rust</h2>
My first reflex was to create a trait for the thing which controls the variance of the Throttle</code>.</p>
pub Trait DelayCalculator<TArg> {
  fn calculate(arg: TArg) -> Duration;
}

impl <TArg> DelayCalculator<TArg> for f32 {
  fn calculate(&self, arg: TArg) -> Duration {
    return Duration::from_millis((1.0 / self) * 1000.0);
  }
}
</code></pre>
So in my head the nifty thing is that now f32</code> itself implements DelayCalculator</code>, so I should be able to keep my example and pass a straight f32</code> into Throttle::new</code> and it should just work.</em></p>
But when I modified Throttle</code> itself, things broke apart.</p>
pub struct Throttle<TArg> {
  // ERROR: DelayCalculator is not sized
  delay_calculator: DelayCalculator<TArg>,
  previous_invocation: Cell<ThrottleState>
}
</code></pre>
An experienced Rustacean is likely chuckling a bit, but for the uninitiated, this actually makes a lot of sense only after you stop to consider what you're describing the memory layout to be. To put a Throttle</code> somewhere the compiler needs to know how big it is. But a Trait is unsized because it could be implemented by something as small as u8</code> or as large as an array of them making a bitmap image of your mom (so filling all addressable memory). There is no way to know.</p>
The compiler helpfully suggests putting the field in a Box</code>, or storing it on the heap, so Throttle</code> would contain a pointer to allocated memory of any size, which I tried. But it felt wrong. A Throttle</code> is simple enough it ought to be expressable without any recourse to the heap!</p>
The next option is to make it a generic parameter, making the Throttle</code> look something like the following:</p>
pub struct Throttle<TDelayCalculator, TArg>
    where TDelayCalculator : DelayCalculator<TArg> {
  delay_calculator: TDelayCalculator,
  previous_invocation: Cell<ThrottleState>
}
</code></pre>
This is better, maybe even good. It makes it harder to work with Throttle</code>, however, as you need to know about the DelayCalculator</code> it uses to size it and therefore store it. Also a point of detraction, it's heavier. Java-style heavier. It requires the introduction of a new type where perhaps one is unneeded.</p>
Next I decided to try to make it a functional interface.</p>
Storing a Closure, or Template All The Things</h2>
By making the Throttle</code> take a delay calculator as a functional interface I hoped to ameilorate the need for making additional implementations like some deranged late nineties Java developer. I also hoped to elide the need for using those Box</code>es and generics. This was not to be.</p>
pub struct Throttle<TArg> {
  // ERROR: Fn<TArg> -> Duration is not sized
  delay_calculator: Fn<TArg> -> Duration,
  previous_invocation: Cell<ThrottleState>
}
</code></pre>
Oops, same errors as before. But it again makes sense. A closure can capture its surrounding state, which means that any closure may have a different size depending on what local state it captured.</p>
So it's back to using a generic:</p>
pub struct Throttle<TArg, TDelayCalculator> 
    where TDelayCalculator : Fn<TArg> -> Duration {
  delay_calculator: TDelayCalculator,
  previous_invocation: Cell<ThrottleState>,
  delay_arg_type: PhantomData<TArg>
}
</code></pre>
But this simply bubbles the issue of the TDelayCalculator</code> being unsizeable back up to the calling code. In the trivial case it doesn't matter because it will live on the stack. But if it's packaged into a client it becomes unergonomic or the whole thing must be Box</code>ed. Yuk.</p>
It feels like the Box</code> is unavoidable, so for now it's inside the Throttle</code> to keep the struct as portable as it possibly can be.</p>
pub struct Throttle<TArg> {
  delay_calculator: Box<Fn(TArg, Duration) -> Duration>,
  state: Cell<ThrottleState>
}
</code></pre>
This makes the constructors of the struct very simple:</p>
pub fn new<TDelayCalculator>(delay_calculator: TDelayCalculator) -> Throttle<TArg>
    where TDelayCalculator: Fn(TArg, Duration) -> Duration + 'static {
  return Throttle {
    delay_calculator: Box::new(delay_calculator),
    state: Cell::new(ThrottleState::Uninitialized)
  };
}

pub fn new_tps_throttle(tps: f32) -> Throttle<TArg> {
  return Throttle {
    delay_calculator: Box::new(move |_, _|
      Duration::from_millis(((1.0 / tps) * 1000.0) as u64)),
    state: Cell::new(ThrottleState::Uninitialized)
  };
}
</code></pre>
Documentation that also Tests</h2>
Thus far Rust has been delightful. The language is terse, yet expressive, and the compiler, while unforgiving, is helpful with well-written error messages with suggestions to resolution. But perhaps the most impressive to me is how documentation is written for Rust. Beyond a good README file, which has been shown to be the single most important factor in the adoption of an open-source project, the documentation can either be an asset or a liability, depending on the quality and accuracy.</p>
Rust aims to prevent documentation drift by simply compiling it.</p>
The module documentation for Throttle, the top of the package and the introduction when a potential consumer first sees the documentation page, looks like the following:</p>
//! A simple throttle, used for slowing down repeated code. Use this to avoid
//! drowning out downstream systems. For example, if I were reading the contents
//! of a file repeatedly (polling for data, perhaps), or calling an external
//! network resource, I could use a `Throttle` to slow that down to avoid
//! resource contention or browning out a downstream service.
//!
//! This ranges in utility from a simple TPS throttle, "never go faster than *x*
//! transactions per second,"
//!
//! ```rust
//! # extern crate mysteriouspants_throttle;
//! # use std::time::Instant;
//! # use mysteriouspants_throttle::Throttle;
//! # fn main() {
//! // create a new Throttle that rate limits to 10 TPS
//! let throttle = Throttle::new_tps_throttle(10.0);
//!
//! let iteration_start = Instant::now();
//!
//! // iterate eleven times, which at 10 TPS should take just over 1 second
//! for _i in 0..11 {
//!   throttle.acquire(());
//!   // do the needful
//! }
//!
//! // prove that it did, in fact, take 1 second
//! assert_eq!(iteration_start.elapsed().as_secs() == 1, true);
//! # }
//! ```
</code></pre>
The documentation is itself in CommonMark, so it's remarkably easy to write. But the code sample is compiled, so when I run cargo test</code> I see that it runs the documentation code listing as well:</p>
PS C:\Users\xpm\Projects\mysteriouspants-throttle> cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target\debug\deps\mysteriouspants_throttle-62cc10bf6d8c5f7f.exe

running 1 test
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests mysteriouspants-throttle

running 4 tests
test src\lib.rs -  (line 10) ... ok
test src\lib.rs -  (line 34) ... ok
test src\lib.rs - Throttle<TArg>::new_variable_throttle (line 76) ... ok
test src\lib.rs - Throttle<TArg>::new_variable_throttle (line 92) ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
</code></pre>
And to the user, the person viewing the webpage, it looks like any normal code listing.</p>
// create a new Throttle that rate limits to 10 TPS
let throttle = Throttle::new_tps_throttle(10.0);

let iteration_start = Instant::now();

// iterate eleven times, which at 10 TPS should take just over 1 second
for _i in 0..11 {
  throttle.acquire(());
  // do the needful
}

// prove that it did, in fact, take 1 second
assert_eq!(iteration_start.elapsed().as_secs() == 1, true);
</code></pre>
The combined effect is that I ended up putting more of my tests in the documentation, both as a test-driven measure, and to make sure my tests are really showing how the crate works from a consumer-driven perspective. If I change the API contract, the test should break right in the documentation, so the documentation should not be able to drift away from how the crate actually works, so long as I'm still running the tests.</p>
Final thoughts</h2>
Rust is a language I've had my eye on for a very long time now. It promises to be close to the metal, fast, and expressive, focusing on letting the programmer build abstractions that don't cost in performance as much as possible. It's portable, and it has a great build system with an easy way of taking on new dependencies.</p>
I personally prefer it over more popular modern languages like Go, because Rust dares to ignore the draw of garbage collection to say that the programmer can write proper code with a low water mark in memory (I distrust garbage collectors).</p>
In short, Rust values the things that I value, and for that I enjoy it.</p>


Let's Encrypt on Nginx Setup
Wed, 31 Aug 2016 00:00:00 +0000
Encryption is a thing now, 314, proliferated to the point that Google
will downrank site which do not have it. I've never exactly cared who,
if anyone, reads this site, but I thought I'd give Let's Encrypt a try -
if for no better reason than because I'd like to use HTTP2 on a web app
I'm writing.</p>
These notes are really for my benefit, though it might benefit you as
well.</p>
First, install Let's Encrypt.</p>
sudo apt install -y letsencrypt
</code></pre>
For every domain, modify the virtual server definition to look like this:</p>
sudo vim /etc/nginx/sites-available/mysteriouspants.com
  # a reflector domain that just redirects to www.,
  # and serves .well-known files
  server {
    listen 80;
    server_name mysteriouspants.com;
    access_log off;

    root /var/www/empty;

    location /.well-known {
      root /var/www/letsencrypt;
      try_files $uri /dev/null =404;
    }

    location / {
      return 301 https://www.$host$request_uri;
    }
  }

  server {
    listen 443 ssl http2;
    server_name mysteriouspants.com;
    access_log off;

    ssl_certificate /etc/letsencrypt/live/mysteriouspants.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mysteriouspants.com/privkey.pem;

    return 301 https://www.$host$request_uri;
  }
sudo vim /etc/nginx/sites-available/www.mysteriouspants.com
  server {
    listen 80;
    server_name www.mysteriouspants.com;
    access_log off;

    root /var/www/empty;

    location /.well-known {
      root /var/www/letsencrypt;
      try_files $uri /dev/null =404;
    }

    location / {
      return 301 https://$host$request_uri;
    }
  }

  server {
    listen 443 ssl http2;
    server_name www.mysteriouspants.com;
    access_log off;

    ssl_certificate /etc/letsencrypt/live/mysteriouspants.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mysteriouspants.com/privkey.pem;

    root /var/www/www.mysteriouspants.com;
  }
sudo service nginx reload
</code></pre>
Make some directories. The empty</code> directory in particular is for the
root of the redirection mirrors.</p>
sudo mkdir /var/www/empty
sudo mkdir /var/www/letsencrypt
</code></pre>
Actually grab your first SSL certificate. It will ask you to accept a
EULA, then it will populate /var/www/letsencrypt</code> with some challenge
files. Their servers will ask your domains for those files to prove that
you control the domains - so they're not just handing out certificates
willy-nilly.</p>
sudo letsencrypt certonly --webroot -w /var/www/letsencrypt \
  -d mysteriouspants.com \
  -d www.mysteriouspants.com
</code></pre>
This command should complete with no errors, if it does, your Nginx
configuration is likely messed up.</p>
At this point, your web pages are actually SSL'd now that your SSL
certificates exist. Harden up your Nginx server by adding this set of
ciphers I totally didn't just pull off some webpage.</p>
sudo vim /etc/nginx/nginx.conf
  ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
</code></pre>
Finally, Let's Encrypt certificates are only good for 90 days, so it's
best to renew them. It's simple enough to renew it using a cron job.</p>
sudo vim /etc/cron.daily/letencrypt
  #!/bin/sh

  letsencrypt renew
  service nginx reload
  # reload any other services that may need to take the new certificates
</code></pre>
And there you have it.</p>


Deploying Elixir/Phoenix
Mon, 18 Jul 2016 00:00:00 +0000
This page is something of a living blog post, and it does get updated
from time to time. I suggest you look at the file's history</a>
if you need to know how fresh/stale this guide is.</em></p>
Well 314, it has been a while, has it not? I have recently completed a
bit of an adventure (or completed the adventure nearly enough to share,
at any rate). I've gone and fallen in love with the Elixir programming
language</a>, and the star of the Elixir world right now is
Phoenix</a>, a web framework. Now, cutting websites it's nearly
as sexy as it was in the late nineties, but this nearly makes up for it.</p>
Phoenix's one-two punch of a Rails-ish paradigm and a nice database
layer in Ecto makes it a true joy to work with. Where Phoenix gains on
Rails, however, is in Elixir, a language which keeps much of the
happiness of Ruby while handing you the speed of Erlang. Even on
database-backed pages Phoenix returns pages in tens of milliseconds.</p>
So it's some hot stuff, and it makes me happy. But with all web apps,
there comes a time when running it on my laptop is not enough. So I
embarked upon a journey to learn to deploy Elixir/Phoenix apps to a
server using edeliver</a> and exrm.</p>

The overall model of deployment is to commit and push code to your source repository. edeliver will then pull that to a build server (a server that resembles your production server - in OS and architecture, so you can build a compatible package). On this build server it uses exrm to compile a release (or upgrade, which can patch your running code - pretty dope), and downloads that to your computer. You then upload this package to your production server(s).</p>
These exrm releases themselves are pretty cool. They are self-contained
releases, they include an Erlang VM, all the package and dependencies to
make your app run. Gone are the days of hammering rubygems.org from
three dozen machines every deploy!</p>
For my build server, I simply set up a new user account on my Digital
Ocean droplet, running Ubuntu Linux, which runs this site and my IRC
bouncer. My production server was similarly yet another account on this
server. edeliver, however, supports a plurality of production servers.
Cleverly, it uses SSH, so you can use keys to make things happen without
a single password getting in the way.</p>
Setting up a Build Server</h2>
SSH to the server as root</code> or any sudo</code>-capable user.</p>
We'll be using asdf</a> to manage our development dependencies
(erlang, elixir, and nodejs). This allows us to install a plurality of
versions in case you have to build different projects requiring
different versions from the same build user.</p>
First we'll install some prerequisites via apt</code>.</p>
sudo apt-get install -y git vim unzip \
    build-essential autoconf m4 libncurses5-dev \
    libwxgtk3.0-dev libgl1-mesa-dev libglu1-mesa-dev \
    libpng3 libssh-dev unixodbc-dev
sudo update-alternatives --set editor /usr/bin/vim.basic
</code></pre>
Now create a build user.</p>
sudo useradd --shell=/bin/bash build
</code></pre>
You should install any applicable SSH key for the build</code> user into
~build/.ssh/authorized_keys</code> now.</p>
Login as the build</code> user now and install asdf and its plugins.</p>
git clone https://github.com/asdf-vm/asdf.git ~/.asdf
echo '. $HOME/.asdf/asdf.sh' >> ~/.profile
echo '. $HOME/.asdf/completions/asdf.bash' >> ~/.profile
source ~/.profile
asdf plugin-add erlang https://github.com/asdf-vm/asdf-erlang.git
asdf plugin-add elixir https://github.com/asdf-vm/asdf-elixir.git
asdf plugin-add nodejs https://github.com/asdf-vm/asdf-nodejs.git
</code></pre>
This next step will install the actual tools. This can take a while, so
I suggest separating each command by a comma, then running off to get
something to eat. It can take a moment.</p>
asdf install erlang 20.1
asdf global  erlang 20.1
asdf install elixir 1.5.2
asdf global  elixir 1.5.2
asdf install nodejs 8.7.0
addf global  nodejs 8.7.0
</code></pre>
Note:</strong> Erlang requires somewhere slightly north of 1GB of RAM to
build successfully, or it dies somewhere in a WxWidgets library compile.
I suggest that you enable a swapfile</a> during the build if you
have less than 2GB of free RAM before starting the build. As this will
degrade the SSD hardware you're probably running on, I suggest you
remove the swapfile after the build is completed.</p>
If this worked, then installing Hex and Rebar will work.</p>
mix local.hex
mix local.rebar
</code></pre>
Your build server is almost ready to get to work! Create the
my-app_prod.secret.exs</code> file, which should follow the following
template.</p>
vim my-app_prod.secret.exs
  use Mix.Config

  config :my_app, MyApp.Repo,
    adapter: Ecto.Adapters.Postgres,
    username: "u",
    password: "p",
    database: "d",
    hostname: "localhost",
    template: "template0",
    pool_size: 10
</code></pre>
I'll leave the configuration of your PostgreSQL server to you - there
are plenty of tutorials and how-to's for it. Suffice it to say, you
should substitute real values where I have left placeholders.</p>
Setting up a Production Server</h2>
Some of these steps will be identical to the steps performed on the
build server. If these servers are the same, you can safely skip the
redundant steps.</p>
First, SSH to the new production server as root</code> or a sudo</code>-capable
user. First we'll install Elixir and some Erlang dependencies, as well
as PostgreSQL. Again, if this isn't your database host, then you might
try omitting some. Also, I'm not too sure if the Erlang dependencies are
even necessary. If someone does know, please drop me a message.</p>
sudo useradd -s /bin/bash my-app
sudo mkdir -p ~my-app/.ssh
# add relevant keys to ~my-app/.ssh/authorized_keys
sudo chown -R my-app:users ~my-app
</code></pre>
On deploy and start the application will be running on port 4001</code> or
whatever port you configure in your prod.exs</code> file, so firewall that
sucker up.</p>
sudo ufw deny 4001
</code></pre>
Next configure Nginx to reverse proxy to port 4001</code>. Be sure to replace
the IP address with the public IP address of your production server.</p>
sudo vim /etc/nginx/sites-available/my-app.com
  map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
  }

  server {
    listen 80;
    server_name my-app.com;

    access_log off;

    location /.well-known {
      root /var/www/letsencrypt;
      try_files $uri /dev/null =404;
    }

    location / {
      return 301 https://$host$request_uri;
    }
  }

  server {
    listen 443 ssl http2;
    server_name my-app.com;
    access_log off;

    ssl_certificate /etc/letsencrypt/live/my-app.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/my-app.com/privkey.pem;

    location / {
      proxy_pass http://127.0.0.1:4001;

      include proxy_params;

      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
    }
  }
sudo ln -s /etc/nginx/sites-{available,enabled}/my-app.com
sudo service nginx reload
</code></pre>
Finally, set the app to autostart on machine boot. So that random things
like rebooting for a kernel update doesn't leave the app high and dry
until someone pokes you on Slack.</p>
sudo vim /etc/init.d/my-app-prod.conf
  description "my-app-prod"

  ## Uncomment the following two lines to run the
  ## application as www-data:www-data
  setuid my-app
  setgid my-app

  start on runlevel [2345]
  stop on runlevel [016]

  expect stop
  respawn

  env MIX_ENV=prod
  export MIX_ENV

  ## Uncomment the following two lines if we configured
  ## our port with an environment variable.
  # env PORT=4001
  # export PORT

  env HOME=/home/my-app/my_app
  export HOME

  pre-start exec /bin/sh /home/my-app/my_app/bin/my_app start

  post-stop exec /bin/sh /home/my-app/my_app/bin/my_app stop
</code></pre>
Configuring your Phoenix Project</h2>
There isn't a whole lot to do here, but some of these steps are
important and missing one can lead to about a day of very confused
debugging.</p>
vim config/prod.exs
  use Mix.Config

  config :my_app, MyApp.Endpoint,
    http: [port: 4001], # configure the port to rproxy to here!
    url: [host: "my_app.com", port: 80],
    cache_static_manifest: "priv/static/manifest.json",
    server: true # DON'T FORGET THIS LINE

  config :logger, level: :info

  import_config "prod.secret.exs"
</code></pre>
There will be a mess of comments in there, I haven't explored what they
all do yet.</p>
The next file to change is mix.exs</code>, which you need to add edeliver to
as a dependency.</p>
vim mix.exs
  # ...

  def application do
    [mod: {MyApp, []},
     applications: [:phoenix, :phoenix_pubsub, :phoenix_html, :cowboy,
                    :logger, :gettext, :phoenix_ecto, :postgrex, :edeliver]]
  end

  # ...

  defp deps do
    [{:phoenix, "~> 1.2.0"},
     {:postgrex, ">= 0.11.2"},
     {:phoenix_ecto, "~> 3.0"},
     {:phoenix_html, "~> 2.6.1"},
     {:phoenix_live_reload, "~> 1.0.5", only: :dev},
     {:gettext, "~> 0.11"},
     {:cowboy, "~> 1.0.4"},
     {:edeliver, "~> 1.4.3"}]
  end
</code></pre>
You'll notice that we've added edeliver to both applications and deps.
Note that there is a release version in this file. You can and should
increment that from time to time. It'll show up in your builds -
speaking of which, we need to configure edeliver to know about all the
work we've just done! Do this by creating an edeliver configuration
file, that should look something like this:</p>
mkdir -p .deliver
vim .deliver/config
  #!/usr/bin/env bash

  APP="my_app" # name of your release

  # You can experiment around with this to see what you like best.
  # I'm a retard, so I build a lot without incrementing the version
  # number, so I like having the extra info. More disciplined developers
  # can probably forgo that, however.
  AUTO_VERSION="commit-count+branch+git-revision"

  BUILD_HOST="build.my-app.com" # host where to build the release
  BUILD_USER="build" # local user at build host
  BUILD_AT="/home/build/builds/my_app" # build directory on build host

  # I don't use any staging servers, so these are really kind of
  # moot point for me. #YOLO
  STAGING_HOSTS="stage-01.my-app.com"
  STAGING_USER="my-app-stage"
  TEST_AT="/home/my-app-stage"

  PRODUCTION_HOSTS="my-app.com"
  PRODUCTION_USER="my-app"
  DELIVER_TO="/home/my-app"

  # runs the phoenix.digest mix command, which gets rid of a missing
  # manifest file error
  pre_erlang_clean_compile() {
    status "Installing NodeJS dependencies"
    __sync_remote "
      [ -f ~/.profile ] && source ~/.profile # load profile (optional)
      set -e # fail if any command fails (recommended)
      cd '$BUILD_AT' # enter the build directory on the build host (required)
      # prepare something
      mkdir -p priv/static # required by the phoenix.digest task
      # run your custom task
      APP='$APP' MIX_ENV='$TARGET_MIX_ENV' npm install
    "
    status "Building frontend items"
    __sync_remote "
      [ -f ~/.profile ] && source ~/.profile # load profile (optional)
      set -e # fail if any command fails (recommended)
      cd '$BUILD_AT' # enter the build directory on the build host (required)
      # prepare something
      mkdir -p priv/static # required by the phoenix.digest task
      # run your custom task
      APP='$APP' MIX_ENV='$TARGET_MIX_ENV' npm run deploy
    "
    status "Running phoenix.digest" # log output prepended with "----->"
    __sync_remote " # runs the commands on the build host
      [ -f ~/.profile ] && source ~/.profile # load profile (optional)
      set -e # fail if any command fails (recommended)
      cd '$BUILD_AT' # enter the build directory on the build host (required)
      # prepare something
      mkdir -p priv/static # required by the phoenix.digest task
      # run your custom task
      APP='$APP' MIX_ENV='$TARGET_MIX_ENV' $MIX_CMD phoenix.digest $SILENCE
    "
  }

  # copies the prod.secret.exs file you keep sequestered on your build
  # machine into the build directory.
  pre_erlang_get_and_update_deps() {
    # copy it on the build host to the build directory when building
    local _secret_config_file_on_build_host="/home/build/my-app_prod.secret.exs"
    if [ "$TARGET_MIX_ENV" = "prod" ]; then
      status "Linking '$_secret_config_file_on_build_host' to build config dir"
      __sync_remote "
        ln -sfn '$_secret_config_file_on_build_host' '$BUILD_AT/config/prod.secret.exs'
      "
    fi
  }
</code></pre>
Building and Deploying a Release</h2>
All this work, and it boils down to some very simple commands!</p>
mix edeliver build release
</code></pre>
Copy the release tag from the output, and deploy that sucker.</p>
mix edeliver deploy release to production --version=<<that release tag>>
mix edeliver start production
mix edeliver migrate production # runs your database migrations!
</code></pre>

All that work, but look at what was gained! A very slick deployment
process that encourages sustainable versioning. These have been my notes
from the adventure, which I hope you benefit from in some way. If you
find any error, I encourage you to let me know, preferably by making a
pull request on this file</a> with the fix.</p>
Happy coding, and more importantly, happy deploying, 314!</p>


Wed, 02 Mar 2016 00:00:00 +0000
8 July 2011, 11:26:46 AM Eastern Daylight Time</em></p>
"A thousand strangers chanted together with the countdown. And then it
happened.</p>
</a></p>
"The massive clouds of smoke billowed out the sides of the tower poking
out above the trees. The shuttle rising slowly, and then faster. And
then the noise shaking your guts from over 3 miles away. As the shuttle
gets higher you see this column of fire pushing it up faster and faster.</p>
"And then it was gone: lost in the clouds. The last time human eyes will
see a shuttle leave a launchpad.</p>
"I actually teared up, but I'm pretty sure nobody noticed me wiping my
eyes with my shirt. The folks closest to me know that this isn't exactly
an uncommon event for me, but I felt it all come back: A little kid
alone in his basement building a puzzle of the shuttle, dreaming of
space. </a> Seeing Challenger through the library window on a day when I was
too young to understand what it meant for 7 people to die in the name of
science exploration. I learned about computers because of all this
stuff. The Space Shuttle was an honest to god spaceship. It filled the
gap between science fiction and science reality.</p>
"I have a 3 year old now, and I told him before I left that I was going
to go see a Spaceship, and I think he thought I was lying. He knows the
Millennium Falcon is a spaceship, but that it's also pretend. Atlantis</em>
was real. I saw it with my own eyes with a tower of fire underneath it.
It shook my body and I hope my sons have something to inspire them the
same way.</p>
"But where is the shared mission to be bigger, louder, higher, faster and
further than anything that came before? What are we doing that will
inspire a 6 year-old to sit in the dark in his basement and dream over a
puzzle?"</p>
- Rob "CmdrTaco" Malda</p>
science.slashdot.org</a>

washingtonpost.com</a></p>

  
  Launch Day</i> by Stanley Von Medvey</figcaption>
</figure>


The Case of the Leaky Pool
Tue, 04 Dec 2012 00:00:00 +0000
It always has to happen right before your system administrator leaves on vacation. The game is set, the stakes are high, and nobody is willing to fold. It was at a time like this that the Java Virtual Machine (JVM) on our second production server was discovered to be leaking memory like a gangster leaking blood in the Valentine's Day Massacre. Unlike the mafiosos in Chicago, we couldn't figure out why our JVM was bleeding.</p>

Note that this post is best read in a 1920's-style detective voice. I tried to have fun with the writing style, so just give it a go, OK? You might want to go re-read that first paragraph again, 314, just for practice. Additionally, some of the technical information might be slightly off-base (though I'm pretty certain that it's not too far from the truth;) if you know better, please consider doing me a favor and letting me know in the comments section! I'd love the correction!</em></p>
Like any good beat, we got two production servers with a fat load balancer in front. app01.prod</code> is usually the point man, with app02.prod</code> ready to take the heat if app01.prod</code> gets iced. The cool kids call it a "hot failover," but me? I just call it smart.</p>
And being smart paid off. On Monday we looked at our Munin graphs to check in on the situation. The admin likes to do that before he checks out. Whether for peace of mind, or through grizzled experience doesn't matter as much. Our monitoring reported all systems normal - except app02.prod</code>.</p>

  
  app02.prod.proxy memory-use-day4</figcaption>
</figure>
Unlike most smart shops, we were running a different setup. app01.prod</code> was running our code using Apache Passenger and MRI Ruby 1.8.7. It was the first gig we could get in the club of the corporate private cloud. The bouncer wouldn't let any cool toys inside the party, so we had to dance to the music they played.</p>
But that was a year and a half ago, and the bouncer changed his mind. And we got good at smuggling.</p>
We were running into teething issues with MRI 1.8.7. MRI/YARV 1.9.3 has been roaming the streets for a while now, and we're not sure we want to be stuck on yesterday's ride. With MRI/YARV 2.0 just around the corner, it made sense to make a push for a better stack. So for the past four months we've been in the garage trying to refit the old car for a new trick: JRuby.</p>
Particularly handy is the JRuby way of deploying to J2EE servers, which we can push into the private cloud like we own the joint. The whole organization is chill with anything Java, so we're golden, right? Just grab a server, stand it up, and we're in to the hottest cloud in town. A quick snoop around the scene shows a few promising candidates. I could deploy through GlassFish and cut WAR files using Warbler. I like GlassFish. I used it before at an old job. We go way back, GlassFish and I.</p>
Or I could try this new kid on the block. It supports clustering, queues, and scheduled tasks. Long-running requests (for WebSockets and such) are included. I wryly thought back to a few dirty hacks I had running with delayed_job</code>. Yeah, this TorqueBox</a> kid is the right tool for the job. I tossed out the hacks and spruced up the app, and faster than you could say "UTF-8" it worked.</p>
The sysadmin packaged up TorqueBox into a Debian package so we could install it in our private cloud on our production servers. Because of some legacy decisions in our build servers we had to take the whole .tar.gz</code> file and rename it .png</code>, then un-tar that in a post-install script. Yeah, we got good at smuggling. But it beat the bouncers, and we had our best tools inside the club for the first time.</p>
But it's not all roses when you're trying to deploy something as complex as this. You look at a cron-style task schedule the wrong way and you can find yourself in a world of pain. Like I did this past week.</p>
The leak was huge. We weren't really sure how it could leak 18GB of memory on a 6GB dime, but it had. And the sysadmin was on his way out the door. In order to keep our production systems reliable he tore down app02.prod</code>, which was running our experimental TorqueBox server, and restored it to running the same Apache/Passenger MRI 1.8.7 gig that app01.prod</code> is running. If app01.prod</code> gets iced, app02.prod</code> will be at least as good as it was. And ain't nobody wanted to think about what would happen if they both went down in a hail of requests. There are some things too horrifying to plan for.</p>
This put my investigation in a pickle. The memory leak was only manifesting on app02.prod</code>, and without the extant bug, it was my job to try and replicate it. I started spinning up VirtualBox VMs and tried to start replicating the bug in an environment as close to prod as was possible. But again, the sysadmin saved me.</p>
He installed Munin on a little-known dame sitting in the other aisle, app01.stage</code>, where we try out our new kicks before promoting them to a full prod showing.</p>

  
  app01.stage.proxy leak-day2</figcaption>
</figure>
Stage was showing the same kind of problem as prod: an 18GB memory leak. app01.stage</code> had been booted around the same time we last rebooted app02.prod</code>, so the similar velocity in leak proved that we had the same dirt bag in our sights. The sysadmin left for the week, leaving me with app01.stage</code> and a bug to find.</p>
I spun down my VMs and SSH'd into app01.stage</code> to take a look. I didn't think I'd find anything, but it was worth a try. The system was stable, and it seemed to think that the culprit JVM was only taking 800MB of memory.</p>
A clue!</p>
I looked at the graph again. Munin says it was taking well beyond the memory space, yet there wasn't any swap activity. Years of tracking down bugs hadn't prepared me for this. But a crash-course in memory mapping I'd taken when experimenting with libdispatch</code>'s I/O facilities had prepared me for this. Funny how the gigs you take for fun end up saving you in the end!</p>
What I learned from libdispatch</code> is that all programs are incurably greedy.</p>
They allocate a city of memory, but only ever use a few blocks. Maybe they just keep the rest for a rainy day? Back in the old days of DOS and tommy guns this wasn't a concern. You only ever ran a program at a time, and so nobody cared that the racketeers were running away with the city hall. There was one capo and he set The Rules. Life was simpler back then.</p>
But the world got bigger, and more gangs started vying for the limelight. And they all wanted the same city - all of it. So we give it to them. All of it. And things got more complex.</p>
When a program allocates memory, it runs through a call to either malloc</code> or calloc</code>, and a few others. But those are the big two. malloc</code> returns unformatted memory, while calloc</code> fills the memory with zeroes. malloc</code> and calloc</code> themselves chains down through the local runtime memory manager to see if it has free memory in a held page.</p>
Memory is cordoned off into pages, like the blocks of a city. Unlike a city block, only one program can own a single page at a time - we can't split 'em up. So the local runtime has a small memory manager which will try and give you parts of existing pages it holds when you need more memory. If you have a 4KB page, and you have used 1KB of it, and you ask for 2KB of memory, it should give you part of that existing page. There are tools like guard malloc which change this slightly, but for everyone else this is the way it works.</p>
If the local memory manager doesn't have the memory, it hits the operating system kernel.</p>
Now, kernels are smart. They know that programs are greedy, and so they cheat them. When the program malloc</code>'s a fat load of memory, the kernel just pretends to give it what it wants. But in reality it just gives it pages of memory. Address space. Imaginary property. A deed with no land. Smoke and mirrors. A mirage. It even does this for calloc</code>.</p>
In Linux and Darwin (a FreeBSD fork), the kernel allocates a page of all-zero memory. Whenever a made man demands a leveled (zero-filled) block, the kernel gives him one, but it just forwards to that pre-made zero page. The mafioso can go look at the page and read it all. Everything's zeroes. And that page is shared between potentially hundreds of gangsters at a time. It's a city of addresses which all go to the same vacant lot. It's deception on a grand scale!</p>
The trick happens when they try to build a safe house on that block. The moment that they try to write to that zero-filled page, the kernel gets wise and creates a zero-filled page, then hands that right over to the program. Because of CPU-level magic, the address doesn't change - everything is still distinct. If a page size is 4KB (which is typical), and a program asks for 2040KB of zero-filled pages, that program will get 510 pages which all map back to a single zero-filled page.</p>
The program has 2040KB of address space allocated, but is only taking 0KB of physical memory.</p>
The kernel beat the system, and over-sold the city.</p>
When that program tries to write to that first page, the kernel fixes the situation and hands the program a real page, so now the program has 509 references to the shared page and one reference to a real, private page. It's a slick trick called "Copy on Write."</p>
Now, the linearly increasing line on the Munin graph is showing the committed memory</em>, which is address space given to programs, but which may or may not actually be used. That the real memory graph down below is incredibly low suggests that the program, in this case the JVM, is allocating lots of memory, but never touching it. Because of the fancy footwork on part of the kernel, this prevented the program from crashing due to a memory shortage.</p>
Believe it or not, this narrows down the culprit significantly. But it requires some savvy with the JVM itself.</p>
The JVM, as a virtual machine, allocates a heap space, usually something fairly large, like 64MB. It also enforces a heap space ceiling, usually at 256MB. In my server's case, the JVM started with 64MB and had a ceiling of 512MB. But it also has a "permanent generation" space, which in this case was 256MB. This is where class definitions live, and other such things.</p>
Anything Java, and by extension JRuby, is going to live in the Java heap space. Because of how the heap works, it won't be living in mapped pages, so it should always be in physical RAM. To prove this, I used a tool called VisualVM to get a bug in their court. I watched the gang, and they never knew any better.</p>
At first I installed VisualVM on my iMac, and then enabled the remote JMX connector on app01.stage</code>. But then the corporate firewall blocked everything. A few experiments with SSH port forwarding similarly failed. I needed a way in, or I couldn't see if I was right.</p>
So I cheated. I ran wget</code> on app01.stage</code> to put VisualVM on the staging server, which is running headless. Then I ran SSH with X11 forwarding to put that window onto my local X server (XQuartz for Mac).</p>
ssh -X app01.stage /root/visualvm/bin/visualvm
</code></pre>
I had a way in. And it didn't look bad. These mafia types are much nicer than the drug runners from LA.</p>

  
  app01.stage.proxy visualvm-threadleak-day2</figcaption>
</figure>
The heap space is normal. So it's something causing a memory allocation that's going deeper in the JVM.</p>
Another clue.</p>
This narrows down the list of suspects a lot. Nothing pure-Java or pure-JRuby will be doing this. It has to be a native call. Something is leaking inside the JVM itself.</p>
I called in my friends. TorqueBox and JRuby have friendly, helpful communities. I hopped on IRC, my favorite diner, freenode.net</code>, and poked the folks in #torquebox</code> and #jruby</code>. They confirmed my suspicions throughout. I'm on the right track.</p>
After an APB to Google, I found a tool called pmap</code>, which shows all allocated memory space for a process. Given that I was profiling a JVM with a full enterprise stack, running in a clustered High-Availability mode, there was a lot of stuff. I saw the Java heap space, the permanent generation space, as well as a few memory-mapped files.</p>
The kernel is tricky again. When you read a file into memory, it will pretend to do so. Using the same ideas behind copy-on-write, it will expose the whole file's length to your address space, but will swap out the actual memory. This partial in-memory caching can save big on space without impacting performance too much. pmap</code> shows you files mapped to memory, which really helps to figure out what memory is doing what. Throw away all the files and you're left with potentials for leaks.</p>
But most interesting to me were hundreds of 2040KB "anonymous" pages. An anonymous page isn't mapped to a specific file or network stream. It's memory the program has allocated because the program needs memory. The Java heap space was a huge 300MB anonymous memory block. The permanent generation space was a similarly large 250MB anonymous memory block.</p>
But what are these 2040KB blocks? And they were all</em> exactly</strong> 2040KB.</p>
By piping pmap</code> into grep</code>, I started counting them:</p>
pmap 3248 | grep -c "2040K\\ rwx--\\ \\ \\ \\ \\[\\ anon\\ \\]"
</code></pre>
I would run that, wait a little while, then run it again. I was getting another 2040KB allocation every minute. Then I noticed the VisualVM window behind that. I noticed the VM thread count.</p>
1,740 threads sounds a little high, and it's climbing. I know it's an enterprise application server, but 1,740 is gratuitous. And I have 1,741 2040KB pages.</p>
And the default stack size for a thread is 2040KB. After a few more minutes of watching, I'm sure. There's a huge thread leak. Something is creating threads and never destroying them when it's done.</p>
VisualVM has a thread tab, which shows pure chaos: among the normal threads are around 1,550 threads, all helpfully named things like pool-1350-thread-1</code>.</p>
Another clue.</p>
It's not just leaking threads, it's leaking thread pools with just one thread in them. That narrows it down a lot. Lots of code creates threads, but less code creates thread pools.</p>
At this point I'm sure as Machine Gun Kelly's spit that it's thread leaks. A thread allocates a stack which lives outside the heap space, which fits the M.O. of the leak perfectly. My TorqueBox and JRuby associates on IRC heartily agreed.</p>
My application doesn't perform any manual threading, so it's all going to be something from a underlying framework. The list of possibilities was nauseatingly endless, however:</p>

oauth</code>, which shouldn't be spawning thread pools every minute. But it's possible. It has to whack an external server and check back, so it could be using a thread.</li>
jdbc-mysql</code>, which could very well be naughty with threads. I wasn't too sure though. This one's been through a lot, and he doesn't seem like the type to simply throw caution to the wind and start doing illegal things.</li>
torquebox</code>, which gets me crazy fun distributed caching and session stores, as well as exposes a logger which chains down to the JBoss AS 7 logger.</li>
torquebox-messaging</code>, used to enqueue a delayed job.</li>
</ul>
The hunt went on. I started commenting out code and disabling parts of my app to try and shake out the leak, which helpfully reproduced on my iMac without complaint. But the app was too big, too complex to start tearing out suspects. I needed another in.</p>
So I started a new, blank Rails app. I started a TorqueBox server, installed the app, and watched the thread count. The leak wasn't there - good.</p>
I added a few render-able pages, and checked the session for a user id (to verify that it wasn't happening somewhere in the Infinispan cache). Still no leak.</p>
I added a scheduled task. I had a scheduled task to run every 15 minutes and another to run every hour on my production app, so I was sure that it wasn't in Quartz, the system which dispatches the scheduled tasks. Still no leak.</p>
Then I added something innocent. I added a timeout</em>. TorqueBox, via Quartz, will enforce a timeout on your tasks, notifying them when they should stop running. And sure enough, that timeout spawned a thread pool with a single thread to effect that notification, but there was no code to clean up that thread when it finished.</p>
I looked at my production application. I had made a mistake. My hourly task was firing every minute, hence the leaked thread pool every minute. If I had caught that mistake I might have suspected the scheduled task earlier, and thereby caught the bug faster. Or I could have fixed it and never seen the bug again because one hour is far too intermittent to try and track down a single thread leak. My timeout was correctly set at 59 minutes, but even so, after that 59 minutes the timeout should have fired and cleaned itself up.</p>
However, the important thing is that I caught the leaker red-handed. I had the whole police department surrounding the building, so I sent in a SWAT team to rustle out the no-good. I sent in one of the TorqueBox lead developers, Benjamin Browning. (Or rather he decided to help fix this of his own volition, but just work with the writing style a bit, please</em>). He raced through the TorqueBox (and JBoss Polyglot) code base and discovered the culprit</a>.</p>
And Benjamin Browning, being the awesome person he is, fixed it</a>. People using TorqueBox 2.2.0 will never have to worry about their scheduled tasks leaking timeout threads.</p>
As I sat at my desk in the middle of the bullpen that afternoon I leaned back in my old chair and grinned. I was a heck of a chase, but in the end, I found the bad guy, and we brought him to justice.</p>
There were a few other steps I didn't mention, such as a cryptic spawn-stack trace I saw using a trial version of YourKit analyzer. This is also my first attempt at writing like a pulp-fiction crime novel. I hope you enjoyed it, 314!</em></p>

From Excel to TSV to Rust

Level up your Crate

Badges</h3> Badges are little images which automatically updated and tell consumers information about your crate. By being on Crates.io, you get the these two badges for free,</p>

Headfirst into Rust - Throttle and Mutability

Headfirst into Rust - Building a Throttle

Let's Encrypt on Nginx Setup

Deploying Elixir/Phoenix

Setting up a Build Server</h2> SSH to the server as root</code> or any sudo</code>-capable user.</p>

Configuring your Phoenix Project</h2> There isn't a whole lot to do here, but some of these steps are important and missing one can lead to about a day of very confused debugging.</p>

The Case of the Leaky Pool