2012-05-04T15:43:03-07:00 http://code.disqus.com/ 2012-05-04T00:00:00-07:00 http://justcramer.com/2012/05/04/distributing-work-without-celery We’ve been migrating a lot of data to various places lately at DISQUS. These generally have been things like runningconsistancy checks on our PostgreSQL shards, or creating a new system which requires a certain form of denormalized data. Itusually involves iterating through the results of … 2012-05-03T00:00:00-07:00 http://justcramer.com/2012/05/03/using-travis-ci I’ve been using Travis-CI for a while now. Both my personal projects,and even several of the libraries we maintain at DISQUS rely on it for Continuous Integration. I figured it was about time to confessmy undenying love for Travis, and throw up some notes about the defaults we use in our … 2012-04-24T00:00:00-07:00 http://justcramer.com/2012/04/24/sticking-with-standards More and more I’m seeing the “requirements.txt pattern” come up. This generally refers to projects (but not just), andseems to have started around the same time as Heroku adopting Python. I feel like this is something that matters in thePython world, and because I have an … 2012-04-08T00:00:00-07:00 http://justcramer.com/2012/04/08/using-arrays-as-materialized-paths-in-postgres Something we’ve been casually working on at Disqus for quite some time is an improved pagination method for threaded comments. This is obviously pretty important to us, it drives the very foundation of our product. It also happens to be an area that’s somewhat challenging, and has a … 2012-03-17T00:00:00-07:00 http://www.dctrwatson.com/2012/03/mutt-and-gmail/ Per recommendation from a neckbeard friend, Aaron, I set out to try out Mutt as my email client. Since my email is hosted by Gmail, there’s a little extra configuration needed than just setting up an IMAP inbox. Also, since people actually send multimedia emails, I wrote a small patch for Mutt ... 2012-03-16T00:00:00-07:00 http://www.dctrwatson.com/2012/03/limits-conf-and-daemons-on-ubuntu/ I recently was setting up a couple ElasticSearch and RabbitMQ instances when I noticed RabbitMQ was still reporting an abysmally low fd limit in its log file at startup. I double checked my /etc/security/limits.conf and sure enough, limits were properly set to 64000. Yet for some reason it was st... 2012-02-09T00:00:00-08:00 http://anton.kovalyov.net/2012/02/09/mock-testing-long-polling/ Yesterday, I asked Twitter if anyone knew how to mock long polling requests. The projet I’m working on uses these types of requests quite extensively, and I need a good way to test them. Q: In JavaScript, how would you mock a long poll?— Anton Kovalyov (@valueof) February 8, 2012 The re... 2011-12-30T00:00:00-08:00 http://www.dctrwatson.com/2011/12/realtime-postfix-stats-aggregator/ With a user base as large as Disqus‘, there is a ton of new comment and reply notification emails to send. Indubitable there is a user that accidentally (maybe even purposefully) subscribes to extremely hot threads. When they start receiving a stream of emails, their email provider doesn’t ap... 2011-12-30T00:00:00-08:00 http://www.dctrwatson.com/2011/12/django-celery-eventlet-and-debugging-blocking/ I recently wrote a couple Celery tasks that are purely IO bound. So instead of using the default multiprocessing execution pool, I used the Eventlet execution pool. With just a small change in Celery settings, I was off to the races. Wrong! After some amount of time, it just sits at 100% CPU and ... 2011-11-10T00:00:00-08:00 http://justcramer.com/2011/11/10/scaling-schema-changes I frequently get asked how Disqus deals with schema changes. It’s a fair question, since we operate a fairly large amount of servers, but I also tend to think the answer is somewhat obvious. So let’s start with the problem of schema changes at scale (in PostgreSQL).Generally you have …