David Trejo's Thoughts

My new blog is at http://dtrejo.com/blog

Sat, 09 Aug 2014 09:59:00 -0700

Please read my new blog at http://dtrejo.com/blog

Faster git workflow?

Sat, 12 Oct 2013 19:04:00 -0700

Normally, git status -s shows me the following image, and then I spend some time entering in the paths to edit or add the files I’ve changed.

But what if git status -s showed me this?

$ git status -s
g1  M lib/child_process.js
g2 M  lib/sys.js
g3 ?? newfile.md

And then I could do this:

$ vim g1
$ git add g3
...

Does anyone know of a tool that allows this?

If it doesn’t exist, maybe I’ll make it :)

GOOD NEWS: scm breeze does all these things! Thanks for the tip @glebm

Cheers,

My pull request workflow for wtfjs.com

Thu, 05 Sep 2013 18:38:38 -0700

Here’s how I merge pull requests quickly, and make sure they work well on the site:

open github, find a pull request
then run this script

sh NUM=72 curl -s https://github.com/brianleroux/wtfjs/pull/$NUM.patch > $NUM.patch git am --signoff < $NUM.patch

then I edit the post if the person made a typo, and eventually publish.
sometimes I git rebase -i origin/master and then fixup and reword if the person made many small commits for a single post. I also use git commit --amend, which I have aliased to git amend and git ammend in case I make a typo.
While rewording or amending, I also add “closes #39” for each issue, to make sure that I merged the issues, and none are closed accidentally via the web UI.

That’s all!

Cheers,

Metrics and Operational Awareness at Voxer

Fri, 18 Jan 2013 22:11:00 -0800

note: written in august 2012 during my Voxer internship. some links are broken as the sky project has yet to be released.

Please read the article on Voxer’s no single point of failure architecture, as it explains how process rings work, and how voxer’s load balancing works.

What are the salient implementation features of our metrics solution?

Client side (on your app servers, which emit data)

metrics are coalesced into 1440 byte udp packets to optimize network efficiency
handy node.js bindings which allow you simply call sky.counter("http_status|200") whenever a certain call suceeds or fails
keys emitted with the form http_status|200 are rolled together with keys of the form http_status|404, giving you both information on, say, 404s, 200 OKs, AND the high-level view of how many http requests you are doing per minute.

Server side (on the skyservers which aggregate metrics data)

skyservers form a ring, and clients load balance themselves so as not to overwhelm a single server
intra-ring skyserver communication is also coalesced for efficiency.
load balancing allows you to add more skyservers as the number of your appservers grows
every minute, skyservers talk to a single redis server to roll-up their data
the redis intermediary in front of riak allows for accurate counters, because redis performs atomic INCR operations

Persisting data

a rollup process continuously pulls keys out of redis and puts the latest time period of data into redis, simultaneously performing 5-minute rollups of the 1 minute data, if applicable.
riak as our persistence medium allows us to keep historical data forever, as we can increase the size of our riak cluster by adding a machine.

Viewing metrics and adding new ones

the redis server keeps a list of keys that have recently been emitted, and exposes this list to our graphing UI, meaning that to begin recording a new metric, you need not do any configuration, simply start recording it on your app servers. The skygraphs UI will simply pick it up and start showing it to you.
the skygraphs UI will also hit redis for the latest minute’s data, giving you real-time access to what is going on right now in your infrastructure.

Why didn’t we just use etsy’s statsd and whisper and carbon?

Etsy wrote a library called statsd, which works with whisper (the backend) to record metrics data. There are a couple reasons we chose not to use these technologies.

rings of servers recieving metrics is the only way we can handle the volume of data being recorded by our app servers
riak allows us to store historical data forever (just add another machine)
we think our graphing UI is pretty snazzy, especially since it has heat maps

Forthcoming: If you’d like to read about skygraphs, our awesome graphing UI, please check out this article.

Thanks for reading!
David Trejo
@ddtrejo & DTrejo on github

Node Single Point of Failure

Fri, 18 Jan 2013 21:03:00 -0800

note: written in august 2012 during my Voxer internship

Voxer believes that any single point of failure is bad. Very bad (this is why we use riak). To achieve this, we’ve written a couple libraries to allow node processes to communicate across machines and in concert. The first is poolee by Danny Coates which does http load balancing on the client, and the second is ring*, which organizes processes that compose a service into a cooperating whole (each process owns part of the keyspace, and thus part of the work).

Here’s how this looks:

Requests emanate from clients via poolee, which has a list of all API servers. Poolee chooses a random API server to speak with, and will stop talking to the API server if it becomes unresponsive. Health checks on unresponsive servers are performed with using exponential backoff (e.g. 1 second, 8 seconds, 16 seconds, etc), so as not to flood the network.

The ring of API servers kicks in! Each client request has a distinguishing characteristic (e.g. geographic region), which dictates a specific API server as the “home” or “owner” of that data, and it is tasked with responding to that request. You may have noticed that poolee chooses a random node, but it turns our that this random node knows it is not the owner of the request, and can forward it to the true “home” node of that request.

ring is consistent, which means that when an API server goes down, the other servers will fill in for the downed server without changing the homes of data stored on any nodes other than one neighboring node. The efficiency of home re-shuffling is what makes the ring consistent. If this doesn’t make sense, it is because it’s difficult to explain, wikipedia does a better job of explaining consistent hash rings.

Failure

Poolee and ring work together and allow us to do rolling restarts without service interruption, restart single machines when SSL has memory leaks, and remove machines from our clusters.

Thus, Voxer achieves a no single point of failure architecture using the power of node.js.

*Forthcoming: docs and more information on ring.

Thanks for reading!
David Trejo
@ddtrejo & DTrejo on github

Salary Negotiation Resources (CS focused)

Sun, 06 Jan 2013 13:25:00 -0800

This is a dump of my personal notes; expect nothing!

General

- http://www.offer-letters.com/

- http://www.jamesaltucher.com/2011/11/how-to-negotiate-in-three-easy-lessons/

- http://www.iwillteachyoutoberich.com/blog/negotiate-like-an-indian-i-taught-m…

- http://mindyourdecisions.com/blog/2009/08/04/how-to-negotiate-a-pay-raise-wit…

- why you should talk about salary with co-workers and friends: http://news.ycombinator.com/item?id=4767273

- summary/review of secrets of power negotiating book: http://www.korokithakis.net/posts/secrets-power-negotiating/

Startup-specific

- http://accountalent.com/?p=380

- How I negotiated my startup compensation

http://blog.keen.io/post/29904565692/how-i-negotiated-my-startup-compensation [article crit:http://news.ycombinator.com/item?id=4413569]

- Should you join that start-up? Running the numbers.

http://auscillate.com/post/238

- [great comments] http://news.ycombinator.com/item?id=2928339

Books

- ask for it on amazon; be sure to do excercises with whichever friend of yours is best at negotiating

- summary/review of secrets of power negotiating book: http://www.korokithakis.net/posts/secrets-power-negotiating/

Scientific

http://link.springer.com/chapter/10.1007%2F978-3-642-00142-0_66

http://link.springer.com/article/10.1007%2Fs11211-008-0084-x

http://psycnet.apa.org/index.cfm?fa=search.displayRecord&UID=2001-17820-002

unread

http://news.ycombinator.com/item?id=2309317

http://news.ycombinator.com/item?id=2438980

Mounting a remote file system over ssh on Mac OSX Lion...

Mon, 30 Jul 2012 11:17:00 -0700

Mounting a remote file system over ssh on Mac OSX Lion 10.7.4

You hate having to ftp and/or git push. You like being able to using finder and your local editor configurations.

Here’s how to mount a remote filesystem over ssh:

download http://osxfuse.github.com/ and http://macfusionapp.org
at this point, if you try to use macfusion, you will see this error.
download http://www.tuxera.com/mac/macfuse-core-10.5-2.1.9.dmg
open up macfusion, and it should work!

These instructions were put together from the sshfs website as well as a blog post which tells you how to fix macfuse errors. There is more discussion on this over at http://www.matthewratzloff.com/blog/2012/03/12/macfusion-on-os-x-lion/.

My Github Pull Request Workflow Let’s say I’m...

Fri, 06 Jul 2012 12:01:00 -0700

My Github Pull Request Workflow

Let’s say I’m taking pull requests for node_redis.

Here’s how I go about it.

export NUM=214 # for convenience
curl -s https://github.com/mranney/node_redis/pull/$NUM.patch > $NUM.patch

# see what changed, and make sure it applies cleanly
git apply --stat $NUM.patch
git apply --check $NUM.patch

# add the commits to master
git am --signoff < $NUM.patch

# if I need to make some changes, I do the following and put "e" next
# to the commits that need changes
git rebase -i origin/master

Yes, rebase can be scary. I recommend reading about how to undo anything using git’s reflog, even a botched rebase or merge.

How To: All Your Sites Hosted Free on Nodejitsu

Thu, 26 Apr 2012 14:50:00 -0700

Just moved all my sites to Nodejitsu so I wouldn’t have to pay for hosting (~$120/year). Other benefits: no need to sys-admin, redundancy, simplicity.

Here’s my setup:

Sign up for Nodejitsu (free) with GimmeSomeBeta.jit.su
Make a node app, possibly a simple static site using tako
Sign up for Cloudflare (free), and make sure to turn off all security (you don’t need it).
Follow nodejitsu’s DNS guide
Deploy your app (jitsu deploy)
Visit it at your domain name!

Cheers and happy hosting!
David

Pro tip: if you’d like to host different nodejitsu apps on different subdomains, make a *.yourdomain.com A record in Cloudflare. When you deploy an app you want served on a subdomain, just add the appropriate "domains": [ "thing.yourdomain.com" ] property to your package.json.

Want a Nodejitsu Beta invite?

Fri, 09 Mar 2012 09:59:00 -0800

Those of you who’ve been waiting, wait no longer — get a nodejitsu beta invite at gimmesomebeta.jit.su!

Cheers!

Convert all line endings to LF

Mon, 02 Jan 2012 14:48:00 -0800

Why? So that you can check your node_modules folder into git, whether or not there are mixed line endings in files.

Use dos2unix, which you can get using macports:

sudo port install dos2unix

And then use it:

dos2unix filetoconvert.js

Or if there are many offending files:

cd path/to/bad/files/
find . | xargs dos2unix

And then git add whatever you like and you’ll be good to go!

How to detect that node.js is running on Windows

Fri, 23 Dec 2011 22:38:00 -0800

Do something like this:

if (process.platform === 'win32') {
  console.log('This is windows');
} else {
  console.log('%s is not windows, I hope!', process.platform);
}

Needed this to update run.js to work on Windows. Still not done making it work, but sometime soon it will happen.

The main reason I need to do this is because fs.watchFile works on Mac/Linux, but only fs.watch works on windows. Once I have it working correctly I’ll post an update with code.

Cheers!
David

Scraping Made Easy with jQuery and SelectorGadget A few days ago...

Fri, 21 Jan 2011 19:35:00 -0800

Scraping Made Easy with jQuery and SelectorGadget

A few days ago I was doing a TON of scraping, and as you know, without the right tools, scraping can be a REAL pain. Out of my pain comes your pleasure — here’s a list of scraping tools and resources which will make your life MUCH easier the next time you need some information from a crufty old website. If you’re short on time, skip to the end and read the tl;dr.

Scraping with jQuery and Node.js

Scraping with jQuery is a real pleasure. Here’s some example code to get you started, it gets the top three articles with their point values from Hacker News. The key part:

var window = jsdom(agent.body).createWindow()
    , $ = require('jquery').create(window);
  
  // scrape!
  var titles = $('.title a')
    , points = $('.subtext span');
    
  var printme = $.map(points, function(el, i) {
      if (i < 3) {
        return $(el).text() + '\t' + $(titles[i]).text();
      }
    });
  
  console.log(printme.join('\n'));

To get the example working, make sure you’ve installed node.js and npm). You may find run.js very helpful for auto-rerunning your scraper whenever you make a change.

Test your code live against the page with jqueryify

Now that you can run any jQuery code you’d like, you can start testing your code against the page. I like to use firebug or Google Chrome’s built in console. Not all pages have jQuery included, and sometimes the version of jQuery will be old, so it’s best to overwrite it with your own version using the jqueryify bookmarklet (it’s also a good idea to change the version of jQuery used by the bookmarklet to match the one used by jsdom to avoid any strange bugs).

Find the shortest selector with Selector Gadget

Next up is finding the correct selectors for the information you want to extract. Selector Gadget makes it very easy to find the least complicated selector that still does the job. Of course you can always choose selectors by hand if you don’t fancy selector gadget, but it is super helpful for crufty nasty sites.

Scrape dynamic pages that use javascript to populate information (arg!)

At this point you can easily find selectors and test on the page live, but there is danger ahead if you encounter a site that uses javascript or AJAX to populate the information. Your tests will work just fine, but when you load the page programmatically, the page’s javascript won’t be run. There’s a few ways to get around this:

Use a regex to extract the values from the javascript written to script tags in the page.
Pretend to be the page and Make requests to the AJAX urls to get the information you need
URL hack. There may still be a URL hack you haven’t tried which will give you the information you need. (in this case, request is your friend.)
Run the page’s javascript programmatically, then scrape the information after it’s been slotted into the page. (See jsdom’s documentation.).

Regex: tools to make your life easier

txt2re.com is a regex generator. That’s right, generator, not tester. This means you enter in a string, click the parts you’d like to match, and then copy and paste the regex into your code. For me, this is pure bliss.

regexpal.com helps you quickly test your regular expressions against test data. Protip: only type the regex, omit the / at the front and end which you’d normally use in your javascript.

tl;dr

Use jQuery and node.js for your scraping. Example code
Test on the live page with the jqueryify bookmarklet
Find the best selectors with Selector Gadget
For pages with inline script tags containing information, use txt2re.com to generate regular expressions, and regexpal.com to test them.
And finally, don’t forget that URL hacking is your friend!

Thanks for reading — if you have any feedback, please leave a comment below!

PS If things are going wrong, here’s the list of common gotchas:

use the same version of jQuery to test as the one you use in your scraper
make sure the information you need is not loaded by javascript on the page

Touch Tone Tanks featured on TechCrunch TV!

Tue, 07 Dec 2010 15:47:00 -0800

Our app, Touch Tone Tanks, was featured at 6:45 on TechCrunch TV.

See the video of Touch Tone Tanks

Check out 6:45 of the above video :)

- David Trejo

PS yeah that was pretty repetitive.

Always be Coding

Mon, 22 Nov 2010 15:42:00 -0800

It is very difficult to always be coding. Here is a list that helps me make progress even when I’m presented with a hairy complicated technical problem. This list also helps soldier on even if I’m not feeling particularly motivated, or in the mood.

Mind Tricks

I’ve ordered this list from hardest to easiest in an attempt to trick my brain into doing at least one of these things:

Finish the whole project in one day. BAM.
Pick an isolated sub-problem and solve that
Take a 10 minute break and go outside, then come back and look at the problem again
Write pseudocode
Write really horrible and ugly and wrong pseudocode, with tons of TODOs and FIXMEs.
Find a friend who is doing work, and make them guilt you into working with them [New! A highly effective tip suggested by Charles Yeh]
write tests and example cases to help focus your implementation [New! Big hat tip to Jeremy Hoffman]

On a somewhat different axis:

Take a pen and paper and sit outside, far away from any screens, and just draw out the problem and how it flows

I actually love drafting with a pen and paper, as you described. Somehow it’s easier for me to draw out bullet points, boxes, lines, and bits of pseudocode than it is for me to conceive of it on a screen. A trick I’ve been doing lately, especially when I have to learn a new system or API with many components, is to put each key function or object on a Post-it note. I stick the Post-Its on my desk, or on a single sheet in my notebook if I want to carry my diagram with me (or draw lines between post-its instead of on them). When I need to change my diagram, I don’t have to erase anything, I can just move or replace the applicable Post-Its. High-tech! — Jeremy Hoffman

Write pseudocode with a pen and paper
Take nap, but ONLY after you’ve done one of the above things for at least 5 minutes!

If you have any mind tricks, please share!

– David

PS Make sure to use calendaraboutnothing.com to really keep yourself honest (suggested by jettiene of #node.js — thanks!).

Watch me demo my phone-controlled game! 7pm tonight! (#nytm...

Tue, 02 Nov 2010 11:34:00 -0700

Watch me demo my phone-controlled game! 7pm tonight! (#nytm #node.js #twilio #socket.io)

Update: You can now see a video demo of touch tone tanks! (It turns out we didn’t crash and burn)

Good afternoon!

Later today Paul, Justin, and myself will be demoing our phone-controlled tanks game* in front of 900 people at New York Tech Meetup! (I hope we don’t crash and burn)

You can watch us live at livestream.com/nytechmeetup at 7pm tonight.

Cheers,

David

* In case you’re curious about how this works:

First you set up a projector or other large screen, then you bust out your phones, call our phone number, and press the buttons on your keypad to shoot or move around. The tones are sent to the game over all kinds of crazy internet infrastructure (Twilio, node.js, and socket.io if you’re curious. We’ve even open-sourced the communication bits of it so you can build your own phone controlled game or web application. See nodaphone for documentation and source.)

Node.js for Server Newbs This is the story of how a webserver...

Fri, 03 Sep 2010 13:26:00 -0700

Node.js for Server Newbs

This is the story of how a webserver noob and thrifty college student got access to a server on which to run node.js.

This guide is for those who have used node.js locally, but want to run it on a server and show their projects to the world.

This assumes (almost) no familiarity with how hosting works, but does assume some familiarity with node.js and building from source. This guide will help you install node.js on an Ubuntu 10.4, 64-bit machine.

Here we go:

Sign up for web hosting at webbynode. Why did I choose webbynode? It offers hosting for $10/month with root access. Only prgrmr competes†, but they have $4/month worth of support, which is scary for those new to hosting, namely myself. I also shied away from webfaction ($9.50/month), because while they do allow you to install node.js in your home directory, there are all kinds of configuration headaches (I think). In any case, these instructions are specific to webbynode with the Ubuntu setup. Update: to get 99 cents off with webbynode, use davidt at checkout.

Under deploy, click the bare linux image, then choose ubuntu 10.4, 64-bit. This may take a a minute or so. When it is active, continue on.

On your dashboard you should see the IP of your webby, it looks like this: x.x.x.x

Open up a console* on your local machine, and run these at the command line:

localmachine$ ssh root@x.x.x.x

(Enter your password to log into your webby server)

This is key, as it allows you to do everything else:

webby# apt-get update
webby# apt-get install git-core g++ make curl libssl-dev

(python is already installed)

Now to download the node files and build them (make sure to use the latest version from http://nodejs.org/#build):

webby# curl -O http://nodejs.org/dist/node-v0.2.0.tar.gz
webby# tar xvzf node-v0.2.0.tar.gz
webby# cd node-v0.2.0
webby# ./configure
webby# make
webby# make install
webby# node --version
>0.2.0

You’re ready to write some JS and serve people up!

Cheers!

-David

PS In order to start serving your site from yourdomain.com, you’ll need to follow webbynode’s instructions for changing the dns setting with whoever you bought your domain from. Here’s their guide: http://guides.webbynode.com/articles/webbymanager/dns-setup.html Don’t forget to migrate your blog and other things!

Bonus:

Set up security on your machine with http://guides.webbynode.com/articles/security/ubuntu810-ufw.html

Use Tim’s guide to restart your node process when it crashes [just pay attention to the upstart parts, if you want to keep moving parts to a minimum]

http://howtonode.org/deploying-node-with-spark

† that I know of — let me know if you find other good options!

* If you are using a pc you may need to install something like putty so that you can use ssh.

$1100 clock made with jQuery The QlockTwo, made in Germany, is...

Sun, 15 Aug 2010 20:57:00 -0700

$1100 clock made with jQuery

The QlockTwo, made in Germany, is a beautiful device.

I set out to steal some of that beauty and made a version of the QlockTwo using jQuery. You can see it live right here.

Building this clock was a ton of fun. It took me a couple of days, and I learned about the relativity of time (ooh sophisticated).

Here are some fancy psychological things that go on when you use the clock:

The first half of the hour creeps along — you don’t notice the dots as much as the words, so most of the time it’s later than what you see in words. The only way to stay accurate is to pay lots of attention to the dots, which requires more thought and means it takes you longer to read the clock.

The second half of the hour races. The words are always either in synced with or ahead of the real-time. This means it that you’ll finish meetings earlier, and be more punctual because you’ll give yourself extra time to get places. Just make sure you always schedule your appointments on the hour.

Technical Details

I select the words to be highlighted in a really jenky way. It is very dependent on the structure of the table (yeah I tried to use divs, but I don’t have the skill to keep all the words aligned horizontally and vertically and keep everything square too). Here’s roughly how the code works:

Javascript asks for the time
Time is transformed into words by a for loop which used to be a GIGANTIC composite switch and if else statement. It was a mess.
The text in all the table cells is put together into one long string. There is a one-to-one relationship between the table cells and the letters in the string. This means that if I match the word “OCLOCK” in the string, and the O in OCLOCK is the 50th character in the string, then I know I can just light up the 50th table cell and it will contain the “O.” That wasn’t the most clear way of saying things. Here’s how it goes: I want to highlight OCLOCK, the code finds the position of OCLOCK in the string, then highlights the table cells in that position.
Every 400 milliseconds the clock’s face is refreshed. It is important to me that the clock change times in a smooth manner, and it is important that the clock light up in a visible way when you load the page. To achieve a smooth fadein, there is a css transition of .12 seconds. This makes things smooth when the clock changes times. The 400 millisecond fadein leaves the page time to load when you first fire it up, so you get to experience the initial glow when the clock lights up.
Now you might be thinking, if the CLOCK REFRESHES EVERY 400 MILLISECONDS THEN WHY ISN’T IT FLASHING LIKE CRAAAAAZYYY?? That’s how it was initially, and it was ugly as sin (if you consider sin ugly). Here’s the solution: compare the previously lighted letters with the new ones that need to be lighted, unlight the ones not present in the new list, and then light up the new ones. This turns out to be a bit complicated and it turns out Venn diagrams made with water colors are a key tool of the modern-day programmer.
What other cool stuff does the clock have in it? I stole a glow from midtonedesign.com, which adds the halo behind the clock (cheers Jonotan). I also chose a font from the google font directory that mimics the QlockTwo’s font quite well. And of course I threw in some text-shadows to make things glow. Mandatory.

Conclusion

jQuery is tons of fun to work with, and the #jquery channel is full of helpful people. I emailed QlockTwo and apparently my version is a “Nice Application with JQuery and CSS!”

The time distorting aspects of the clock are quite fun, so make sure to put one up in your office so you can wrap up boring meetings more quickly.

Cheers,

David

Hotkeys for Hacker NewsI was bored so I made gmail-style hotkeys...

Sat, 14 Aug 2010 16:43:00 -0700

Hotkeys for Hacker News

I was bored so I made gmail-style hotkeys for Hacker News. You can install the chrome extension here.

I thought I’d add support for Vim and Emacs style hotkeys, but then I didn’t.

Hope you like the chrome extension. If you are concerned about privacy, it *only* runs on http://news.ycombinator.com/*

Have a great day,

David

(DTrejo on HN)

PS Here’s the source if you fancy seeing it: http://github.com/DTrejo/HN-Hotkeys Most of the action goes on in hnkeys.js

How to Stay Up-to-date with Node.js

Mon, 02 Aug 2010 10:45:00 -0700

Node.js is changing every day and new releases are very frequent. Here is how I stay up to date with the latest release version of Node.js.

This guide assumes you already have node.js installed and you are familiar with the command line.

1. Install npm (full npm install guide)

curl http://npmjs.org/install.sh | sh

Or update it if you’ve already installed it

npm install npm

2. Install nave with npm

npm install nave

3. Use nave to run the version of your choice in a subshell

nave use 0.1.102
# or even better:
nave usemain stable

4. All done! Repeat from step 3 whenever a new version of node is released. By the way, npm is great, you can even use it to update itself :)

What tools do you use to stay up to date? If there is a better way, please let me know!