Life Editor

Ask HN: How do you log application events?

noreply@blogger.com (MathBee) — Tue, 28 Apr 2015 02:21:00 +0000

We are currently inserting our logs in an sql database, with timestamp, logType, userId, userAgent and description columns. It makes it trivial for us to debug any event by just querying the db. However, after three and a half years of continued use, the table is now way too large.
How do you guys log application events in such a way that extracting information from it is easy, but still keep the size of the logs manageable?

thaumaturgy 8 hours ago

Ehm, the contrast between my answer and everyone else's here makes me feel surprisingly greybearded, but...Application logging has been a solved problem for decades now. syslog or direct-to-disk in a reasonable format, let logrotate do the job it's faithfully done for years and let the gzipped old files get picked up by the offsite backups that you're surely running, and use the standard collection of tools for mining text files: grep, cut, tail, etc.I'm a little weirded out that "my logs are too big" is still a thing, and that the most common answer to this is "glue even more complexity together".
reply

nlh 7 hours ago

100% agree. Don't try to build your own when there are some excellent free (and commercial) ones that are battle-tested.grep, cut, tail, etc. work quite well if you're working on a single machine or small number of machines.
The "ELK stack" (ElasticSearch, Logstash, Kibana) is a step up in complexity but gives you much more power than command line tools.
There are also some great commercial solutions that abstract away some of that complexity if you don't feel like rolling your own (Scalyr, Splunk, SumoLogic, etc.).
But regardless of the path you take, don't reinvent the wheel!

FIFA 2014 World Cup live stream architecture

noreply@blogger.com (MathBee) — Tue, 28 Apr 2015 02:16:00 +0000

We were given the task to stream the FIFA 14 World Cup and I think this was an experience worth sharing. This is a quick overview about: the architecture, the components, the pain, the learning, the open source and etc.

The numbers

GER 7×1 BRA (yeah, we’re not proud of it)
0.5M simultaneous users @ a single game – ARG x SUI
580Gbps @ a single game – ARG x SUI
=~ 1600 watched years @ the whole event

The core overview

The project was to receive an input stream, generate HLS output stream for hundreds of thousands and to provide a great experience for final users:

Fetch the RTMP input stream
Generate HLS and send it to Cassandra
Fetch binary and meta data from Cassandra and rebuild the HLS playlists with Nginx+lua
Serve and cache the live content in a scalable way
Design and implement the player

If you want to understand why we chose HLS check this presentation only in pt-BR. tip: sometimes we need to rebuild some things from scratch.

The input

The live stream comes to our servers as RTMP and we were using EvoStream (now we’re moving to nginx-rtmp) to receive this input and to generate HLS output to a known folder. Then we have some python daemons, running at the same machine, watching this known folder and parsing the m3u8 and posting the data to Cassandra.
To watch files modification and to be notified by these events, we first tried watchdog but for some reason we weren’t able to make it work as fast as we expected and we changed to pyinotify.
Another challenge we had to overcome was to make the python program scale to x cpu cores, we ended up by creating multiple Python processes and using async execution.
tip: maybe the best language / tool is in another castle.

The storage

We previously were using Redis to store the live stream data but we thought Cassandra was needed to offer DVR functionality easily (although we still uses Redis a lot). Cassandra response time was increasing with load to a certain point where clients started to timeout and the video playback completely stopped.
We were using it as Queue-like which turns out to be a anti-pattern. We then denormalized our data and also changed to LeveledCompactionStrategy as well as we set durable_writes to false, since we could treat our live stream as ephemeral data.

Finally, but most importantly, since we knew the maximum size a playlist could have, we could specify the start column (filtering with id > minTimeuuid(now – playlist_duration)). This really mitigated the effect of tombstones for reads. After these changes, we were able to achieve a latency in the order of 10ms for our 99% percentile.

tip: limit your queries + denormalize your data + send instrumentation data to graphite + use SSD.

The output

With all the data and meta-data we could build the HLS manifest and serve the video chunks. The only thing we were struggling was that we didn’t want to add an extra server to fetch and build the manifests.
Since we already had invested a lot of effort into Nginx+Lua, we thought it could be possible to use lua to fetch and build the manifest. It was a matter of building a lua driver for Cassandra and use it. One good thing about this approach (rebuilding the manifest) was that in the end we realized that we were almost ready to serve DASH.
tip: test your lua scripts + check the lua global vars + double check your caching config

The player

In order to provide a better experience, we chose to build Clappr, an extensible open-source HTML5 video player. With Clappr – and a few custom extensions like PiP (Picture In Picture) and Multi-angle replays – we were able to deliver a great experience to our users.
tip: open source it from day 0 + follow to flow issue -> commit FIX#123

The sauron

To keep an eye over all these system, we built a monitoring dashboard using mostly open source projects like: logstash, elastic search, graphite, graphana, kibana, seyren, angular, mongo, redis, rails and many others.
tip: use SSD for graphite and elasticsearch

The bonus round

Although we didn’t open sourced the entire solution, you can check most of them:

Source: http://leandromoreira.com.br/2015/04/26/fifa-2014-world-cup-live-stream-architecture/

DIY Device Cloud

noreply@blogger.com (MathBee) — Wed, 28 Jan 2015 07:28:00 +0000

Motivation and Goals
What is a device cloud?
I consider a device cloud to be a service that provides both a broker for communication between devices and a host for applications that interact with devices. The broker allows devices to communicate to other devices or applications without having to be tightly coupled or directly connected to each other. The device cloud is also an easily accessible host for applications that interact with devices, such as visualizations of data or control of device functionality.
Why do you want a device cloud?
- Discovering and connecting to your devices over the internet is hard! Dynamic IP addresses, NAT, firewalls, and more prevent you from easily accessing your home network and devices. Hosting your own device cloud on the internet allows devices to connect and communicate without the trouble of opening connectivity to your home network.
- Not all devices can be connected and online all the time. A device cloud allows for asynchronous communication between devices. For example a sensor or actuator can periodically connect to your device cloud to record data or receive new instructions; because the device isn't connected all the time it can greatly reduce its power consumption.
- You have total control over the data and services available to your devices. You aren't limited by what a 3rd party provides or concerned about what they might do with your data. You own the entire infrastructure and can mold it to your needs.
Project Goals
- Automate the setup and maintenance of services that form a personal device cloud. Setup should be an easily repeatable process from a single command or script.
- Target running the device cloud on cheap, low end cloud infrastructure like Amazon EC2's micro instance free tier.
- Build on existing tools, services, and protocols--don't reinvent the wheel! Many of the services to build a device cloud exist today, they just need to be put together into an easy to use package.
- Document how to connect popular development hardware like the Raspberry Pi, Beaglebone Black, and Arduino to a device cloud.
MQTT - Broker That Ties Everything Together

MQTT is a publish-subscribe protocol that's perfect as a broker for device to device communication. Created by Andy Stanford-Clark and Arlen Nipper, it has been used in projects like Andy's own 'Twittering House'. You can see more from Andy in this TEDx talk:
Mosquitto is an open source MQTT broker implementation that's under active development, has an established reputation, and is well documented. I've done some informal testing with Mosquitto and it seems like the perfect broker to use in this project.
Ansible - Automation That's Easy

Ansible is part of a new generation of open source IT automation tools which aim to simplify the automation of tasks like installing software and setting up servers. You can see more about Ansible from its creator in this recent PyCon talk:
Ansible strives to be simple and requires very little support infrastructure or complex setup. Even though the project is only a few years old, it is already one of the largest on github and home to a healthy ecosystem of modules that can automate almost any server task.
I've been investigating Ansible and expect to make heavy use of it as the primary means of automating the setup and maintenance of the device cloud. The primary output of this project will likely be a collection of Ansible playbooks and tasks that help you build and manage your own device cloud.
Architecture Sketch

Here's a high level look at how I see different components fitting together to form a device cloud:

The MQTT broker, mosquitto, is at the center of all communication and lives on a server in Amazon EC2, Microsoft Azure, etc. Along with the broker other applications like visualizations or control apps are hosted on the same server and can communicate with the broker directly. I have some early thoughts on these apps and will save that for a followup post.
Devices in your home network, or really anywhere, connect to the MQTT broker using an encrypted SSL MQTT channel. Normally SSL is a tremendous pain to configure and manage (especially with self-signed certificates), however I've found it's easy to automate the certificate creation tasks using Ansible so it should be painless. I want the device cloud to be secure by default and not require someone to be an expert in security to setup or run the system.
Devices can be any hardware that's powerful enough to speak MQTT's protocol. Embedded Linux boards like the Raspberry Pi, Beaglebone Black, Arduino Yun, Intel Galileo, etc. will be very easy to configure and use with the device cloud. I hope to even provide Ansible tasks that can deploy and configure MQTT client tools and libraries on any embedded Linux device automatically.
What about devices which don't support SSL, like an Arduino & CC3000? In this case mosquitto has a concept of a bridge server which can act both as an MQTT broker and client. Any MQTT messages sent to the bridge will be relayed back up to the parent broker and vice-versa. With this setup it will be possible to build a small, low power Arduino + CC3000 device that periodically connects to the bridge over an unencrypted channel to send/receive commands. Because the bridge lives inside your home network (which is secured with wireless encryption, right?) it's still relatively secure. Certainly more secure than opening the broker in the cloud up to unencrypted MQTT traffic! Again Ansible tasks should be able to turn any Linux gadget into a bridge for the device cloud easily.
Github Home & Documentation

I've started a repository on github that will be the home for all the software output from this project. Since documentation will be an important part of this project I've also created a base for the documentation at http://diy-device-cloud.readthedocs.org/ using the excellent readthedocs.org service. The documentation is actually built from files in the github repository for the project (under the docs folder) based on the sphinx tool. Because the docs are in the repository it's easy to keep them versioned and up to date. For now I've put up a couple docs that recap what I've covered so far in the project logs here.
MQTT Broker Setup On Amazon EC2

In the past couple days I've added playbooks to the project github source which allow you to provision an MQTT broker server on Amazon's EC2 cloud and set up a basic device cloud. Check out this video to see a walkthrough of setting up a server and communicating with a Raspberry Pi:
I'm still a little new to using Ansibile, but so far am really happy with how simple it's making the process of standing up a server from scratch.
Also I purchased the 'diydevicecloud.net' domain name and am making subdomains of it available to anyone who wants to host their device cloud from an easily reachable address with the excellent FreeDNS.afraid.org dynamic DNS service. You can find the domain here and just need a free account on FreeDNS to get a subdomain. I've even integrated into the Ansible playbook a step to register a regular cron job that will keep your broker server IP address up to date with your FreeDNS domain. Watch the video to see more details of the process.
You aren't limited to using this domain either--there are many other domains on FreeDNS, or you can even buy your own domain and get a static IP in Amazon's cloud.
The next area I want to focus on are client apps. As a part of this project I want to document a few end to end examples of MQTT applications, such as measuring sensor data or remotely controlling devices.
Server Deployment Updates
I've done some cleaning up and refactored the Ansible playbooks to use roles as suggested by Ansible's best practices. I'm not super happy with how the system is configured--right now global configuration is in the inventory file under the [all:vars] group. This is a good spot to configure the entire system in one file but feels a little unwieldily. I'll be looking more into other options here to see if there's a simpler and more flexible way to configure the system.
I've also added quite a few roles to form the base of the server, including:
- MQTT server: installs mosquitto server and configures it to use SSL secured MQTT communication.
- MQTT client: installs mosquitto client tools, Paho MQTT python client library, and the SSL certs for communication with the server. This role can be installed on both the Ubuntu-based server and Debian-based devices like the Raspberry Pi or Beaglebone Black.
- Security: installs and configures fail2ban brute force attack protection package. Ubuntu server is also pretty well locked down by only allowing SSH login with keys (i.e. no password auth allowed), and denying root the ability to login. I want to look more into other useful security software / configs, but am pretty happy with security for now.
- Web server: installs nginx web server. There's not much configured right now, but I plan to put some basic web apps on to help manage the server & communicate with devices.
- SMTP email relay: installs postfix mail transport and configures it to relay mail to an SMTP server like gmail. This is handy for allowing the server to send emails, like from fail2ban warnings or perhaps notifications of MQTT events. Surprisingly this was by far the most complex package to automate with Ansible, mostly because installing postfix requires answering some question prompts which are difficult to do with automation.
One more major package I want to install is a Python WSGI app server that can host web applications written in Python. I plan to make a few simple web apps for reading and writing MQTT messages and would prefer to write them in Python with the Flask web app framework. I've looked into some options here and it looks like it comes down to either uWSGI or Gunicorn. From my testing uWSGI is kind of painful to setup so I might go for Gunicorn.
Service Architecture
Here's a deeper look at the services for the DIY device cloud:

This diagram shows a simple device cloud 'hello world' application that turns an LED on from a web page over the internet. You can see in the bottom left a web page which is hosted on the broker server. When a button is clicked (1) it communicates to a python web application which is hosted by gunicorn (python WSGI web app server) and nginx (main web server). That web application talks directly to the MQTT broker on the server (2), which will inform any connected clients of MQTT messages. A device runs a device service which is just a simple python script that talks to the broker (3) and turns on/off the LED with GPIO libraries (4). The device service is hosted by the excellent supervisor tool so it will automatically run in the background on the device--all the messy details about properly running a background process are abstracted away by supervisor.
This diagram highlights the two main extension points I see where you can plug in your own code to build your device cloud:
- Web services, written as python WSGI applications (using a simple framework like flask).
- Device services, written as simple python scripts that are run automatically by supervisor.
The ansible deployment scripts for both the broker server and devices take care of setting up all the services, configuring them, etc. You only need to place your web service and device service code in a certain location and follow a few simple conventions.
I made a video of creating this simple hello world application and will post it shortly when Youtube is finished processing it.
Device Cloud Hello World
This is a video of how to create a 'hello world' for the device cloud--blinking an LED from a webpage over the internet:

Source: http://hackaday.io/project/1109/logs

How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS

noreply@blogger.com (MathBee) — Wed, 28 Jan 2015 07:25:00 +0000

Introduction

Asynchronous, or non-blocking, processing is a method of separating the execution of certain tasks from the main flow of a program. This provides you with several advantages, including allowing your user-facing code to run without interruption.
Message passing is a method which program components can use to communicate and exchange information. It can be implemented synchronously or asynchronously and can allow discrete processes to communicate without problems. Message passing is often implemented as an alternative to traditional databases for this type of usage because message queues often implement additional features, provide increased performance, and can reside completely in-memory.
Celery is a task queue that is built on an asynchronous message passing system. It can be used as a bucket where programming tasks can be dumped. The program that passed the task can continue to execute and function responsively, and then later on, it can poll celery to see if the computation is complete and retrieve the data.
While celery is written in Python, its protocol can be implemented in any language. It can even function with other languages through webhooks.
By implementing a job queue into your program's environment, you can easily offload tasks and continue to handle interactions from your users. This is a simple way to increase the responsiveness of your applications and not get locked up while performing long-running computations.
In this guide, we will install and implement a celery job queue using RabbitMQ as the messaging system on an Ubuntu 12.04 VPS.

Install the Components

Install Celery

Celery is written in Python, and as such, it is easy to install in the same way that we handle regular Python packages.
We will follow the recommended procedures for handling Python packages by creating a virtual environment to install our messaging system. This helps us keep our environment stable and not effect the larger system.
Install the Python virtual environment package from Ubuntu's default repositories:

sudo apt-get update
sudo apt-get install python-virtualenv

We will create a messaging directory where we will implement our system:

mkdir ~/messaging
cd ~/messaging

We can now create a virtual environment where we can install celery by using the following command:

virtualenv --no-site-packages venv

With the virtual environment configured, we can activate it by typing:

source venv/bin/activate

Your prompt will change to reflect that you are now operating in the virtual environment we made above. This will ensure that our Python packages are installed locally instead of globally.
If at any time we need to deactivate the environment (not now), you can type:

deactivate

Now that we have activated the environment, we can install celery with pip:

pip install celery

Install RabbitMQ

Celery requires a messaging agent in order to handle requests from an external source. This agent is referred to as a "broker".
There are quite a few options for brokers available to choose from, including relational databases, NoSQL databases, key-value stores, and actual messaging systems.
We will be configuring celery to use the RabbitMQ messaging system, as it provides robust, stable performance and interacts well with celery. It is a great solution because it includes features that mesh well with our intended use.
We can install RabbitMQ through Ubuntu's repositories:

sudo apt-get install rabbitmq-server

The RabbitMQ service is started automatically on our server upon installation.

Create a Celery Instance

In order to use celery's task queuing capabilities, our first step after installation must be to create a celery instance. This is a simple process of importing the package, creating an "app", and then setting up the tasks that celery will be able to execute in the background.
Let's create a Python script inside our messaging directory called tasks.py where we can define tasks that our workers can perform.

sudo nano ~/messaging/tasks.py

The first thing we should do is import the Celery function from the celery package:

from celery import Celery

After that, we can create a celery application instance that connects to the default RabbitMQ service:

from celery import Celery

app = Celery('tasks', backend='amqp', broker='amqp://')

The first argument to the Celery function is the name that will be prepended to tasks to identify them.
The backend parameter is an optional parameter that is necessary if you wish to query the status of a background task, or retrieve its results.
If your tasks are simply functions that do some work and then quit, without returning a useful value to use in your program, you can leave this parameter out. If only some of your tasks require this functionality, enable it here and we can disable it on a case-by-case basis further on.
The broker parameter specifies the URL needed to connect to our broker. In our case, this is the RabbitMQ service that is running on our server. RabbitMQ operates using a protocol called "amqp". If RabbitMQ is operating under its default configuration, celery can connect with no other information other than the amqp:// scheme.

Build Celery Tasks

Still in this file, we now need to add our tasks.
Each celery task must be introduced with the decorator @app.task. This allows celery to identify functions that it can add its queuing functions to. After each decorator, we simply create a function that our workers can run.
Our first task will be a simple function that prints out a string to console.

from celery import Celery

app = Celery('tasks', backend='amqp', broker='amqp://')

@app.task
def print_hello():
    print 'hello there'

Because this function does not return any useful information (it instead prints it to the console), we can tell celery to not use the backend to store state information about this task. This is less complicated under the hood and requires fewer resources.

from celery import Celery

app = Celery('tasks', backend='amqp', broker='amqp://')

@app.task(ignore_result=True)
def print_hello():
    print 'hello there'

Next, we will add another function that will generate prime numbers (taken from RosettaCode). This can be a long-running process, so it is a good example for how we can deal with asynchronous worker processes when we are waiting for a result.

from celery import Celery

app = Celery('tasks', backend='amqp', broker='amqp://')

@app.task(ignore_result=True)
def print_hello():
    print 'hello there'

@app.task
def gen_prime(x):
    multiples = []
    results = []
    for i in xrange(2, x+1):
        if i not in multiples:
            results.append(i)
            for j in xrange(i*i, x+1, i):
                multiples.append(j)
    return results

Because we care about what the return value of this function is, and because we want to know when it has completed (so that we may use the results, etc), we do not add the ignore_result parameter to this second task.
Save and close the file.

Start Celery Worker Processes

We can now start a worker processes that will be able to accept connections from applications. It will use the file we just created to learn about the tasks it can perform.
Starting a worker instance is as easy as calling out the application name with the celery command. We will include a "&" character at the end of our string to put our worker process in the background:

celery worker -A tasks &

This will start up an application, and then detach it from the terminal, allowing you to continue to use it for other tasks.
If you want to start multiple workers, you can do so by naming each one with the -n argument:

celery worker -A tasks -n one.%h &
celery worker -A tasks -n two.%h &

The %h will be replaced by the hostname when the worker is named.
To stop workers, you can use the kill command. We can query for the process id and then eliminate the workers based on this information.

ps auxww | grep 'celery worker' | awk '{print $2}' | xargs kill

This will allow the worker to complete its current task before exiting.
If you wish to shut down all workers without waiting for them to complete their tasks, you can execute:

ps auxww | grep 'celery worker' | awk '{print $2}' | xargs kill -9

Use the Queue to Handle Work

We can use the worker process(es) we spawned to complete work in the background for our programs.
Instead of creating an entire program to demonstrate how this works, we will explore the different options in a Python interpreter:

python

At the prompt, we can import our functions into the environment:

from tasks import print_hello
from tasks import gen_prime

If you test these functions, they appear to not have any special functionality. The first function prints a line as expected:

print_hello()

hello there

The second function returns a list of prime numbers:

primes = gen_prime(1000)
print primes

If we give the second function a larger range of numbers to check, the execution hangs while it calculates:

primes = gen_prime(50000)

Stop the execution by typing "CTRL-C". This process is clearly not computing in the background.
To access the background worker, we need to use the .delay method. Celery wraps our functions with additional capabilities. This method is used to pass the function to a worker to execute. It should return immediately:

primes = gen_prime.delay(50000)

This task is now being executed by the workers we started earlier. Because we configured a backend parameter for our application, we can check the status of the computation and get access to the result.
To check whether the task is complete, we can use the .ready method:

primes.ready()

False

A value of "False" means that the task is still running and a result is not available yet. When we get a value of "True", we can do something with the answer.

primes.ready()

True

We can get the value by using the .get method.
If we have already verified that the value is computed with the .ready method, then we can use that method like this:

print primes.get()

[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79,

83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151, 157, 163, 167,

173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, 239, 241, 251, 257, 263,

269, 271, 277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349, 353, 359, 367,

373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443, 449, 457, 461, 463,

467, 479, 487, 491, 499, 503, 509, 521, 523, . . .

If, however, you have not used the .ready method prior to calling .get, you most likely want to add a "timeout" option so that your program isn't forced to wait for the result, which would defeat the purpose of our implementation:

print primes.get(timeout=2)

This will raise an exception if it times out, which you can handle in your program.

Conclusion

Although this is enough information to get you started on using celery within your programs, it is only scratching the surface on the full functionality of this library. Celery allows you to string background tasks together, group tasks, and combine functions in interesting ways.
Although celery is written in Python, it can be used with other languages through webhooks. This makes it incredibly flexible for moving tasks into the background, regardless of your chosen language.

Source: https://www.digitalocean.com/community/tutorials/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps

Creating a browser based EmonHub log console using python, flask, socketio, MQTT

noreply@blogger.com (MathBee) — Wed, 28 Jan 2015 07:24:00 +0000

Written in Python using the flask web framework this example subscribes to a MQTT topic to which logs are posted from emonhub and pushes messages up to the browser using socket.io. The results are displayed in a html box that looks like a ubuntu linux terminal window. The example includes basic session based authentication with a hardcoded username and password see app.py.
Username: demo, password: demo
Being able to view the emonhub log from a browser rather than having to login via SSH could make debugging more convenient.

Install

sudo pip install Flask
sudo pip install Flask-SocketIO
sudo pip install mosquitto

Installation of Flask-SocketIO can be slow on a raspberrypi.
see: https://flask-socketio.readthedocs.org/en/latest/
Code is based on the flask-socketio example

Changes to emonhub

In emonhub.py:
In the import section at the top add:

import mosquitto

Just below the import section add:

mqttc = mosquitto.Mosquitto()
mqttc.connect("127.0.0.1",1883, 60, True)

class MQTTLogHandler(logging.Handler):
    def emit(self, record):        
        mqttc.publish("log",record.asctime+" "+record.levelname+" "+record.message)

Lower down, just below the line (line 324):

logger.addHandler(loghandler)

add:

mqttloghandler = MQTTLogHandler()
logger.addHandler(mqttloghandler)

Source:

https://github.com/emoncms/development/tree/master/Tutorials/Python/socketiowebconsole

Web Scraping With Scrapy and MongoDB

noreply@blogger.com (MathBee) — Sun, 11 Jan 2015 09:29:00 +0000

In this article we’re going to build a scraper for an actual freelance gig where the client wants a Python program to scrape data from Stack Overflow to grab new questions (question title and URL). Scraped data should then be stored in MongoDB. It’s worth noting that Stack Overflow has an API, which can be used to access the exact same data. However, the client wanted a scraper, so a scraper is what he got.

Updated on 01/03/2014 – refactored spider. Thanks, @kissgyorgy.

As always, be sure to review the site’s terms of use/service and respect therobots.txt file before starting any scraping job. Make sure to adhere to ethical scraping practices by not flooding the site with numerous requests over a short span of time. Treat any site you scrape as if it were your own.

Installation

We need the Scrapy library (v0.24.4) along with PyMongo (v2.7.2) for storing the data in MongoDB. You need to install MongoDB as well (not covered).

Scrapy

If you’re running OSX or a flavor of Linux, install Scrapy with pip (with your virtualenv activated):

$ pip install Scrapy

If you are on Windows machine, you will need to manually install a number of dependencies. Please refer to the official documentation for detailed instructions as well as this Youtube videothat I created.

Once Scrapy is setup, verify your installation by running this command in the Python shell:

>>> import scrapy
>>>

If you don’t get an error then you are good to go!

PyMongo

Next, install PyMongo with pip:

$ pip install pymongo

Now we can start building the crawler.

Scrapy Project

Let’s start a new Scrapy project:

$ scrapy startproject stack

This creates a number of files and folders that includes a basic boilerplate for you to get started quickly:

├── scrapy.cfg
└── stack
    ├── __init__.py
    ├── items.py
    ├── pipelines.py
    ├── settings.py
    └── spiders
        └── __init__.py

Specify Data

The items.py file is used to define storage “containers” for the data that we plan to scrape.

The StackItem() class inherits from Item (docs), which basically has a number of pre-defined objects that Scrapy has already built for us:

import scrapy


class StackItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    pass

Let’s add some items that we actually want to collect. For each question the client needs the title and URL. So, update items.py like so:

from scrapy.item import Item, Field


class StackItem(Item):
    title = Field()
    url = Field()

Create the Spider

Create a file called stack_spider.py in the “spiders” directory. This is where the magic happens – e.g., where we’ll tell Scrapy how to find the exact data we’re looking for. As you can imagine, this is specific to each individual web page.

Start by defining a class that inherits from Scrapy’s Spider and then adding attributes as needed:

from scrapy import Spider


class StackSpider(Spider):
    name = "stack"
    allowed_domains = ["stackoverflow.com"]
    start_urls = [
        "http://stackoverflow.com/questions?pagesize=50&sort=newest",
    ]

The first few variables are self-explanatory (docs):

name defines the name of the Spider.
allowed_domains contains the base-URLs for the allowed domains for the spider to crawl.
start_urls is a list of URLs for the spider to start crawling from. All subsequent URLs will start from the data that the spider downloads from the URLS in start_urls.

XPath Selectors

Next, Scrapy uses XPath selectors to extract data from a website. In other words, we can select certain parts of the HTML data based on a given XPath. As stated in Scrapy’s documentation, “XPath is a language for selecting nodes in XML documents, which can also be used with HTML.”

You can easily find a specific Xpath using Chrome’s Developer Tools. Simply inspect a specific HTML element, copy the XPath, and then tweak (as needed):

Developer Tools also gives you the ability to test XPath selectors in the JavaScript Console by using $x – i.e., $x("//img"):

Again, we basically tell Scrapy where to start looking for information based on a defined XPath. Let’s navigate to the Stack Overflow site in Chrome and find the XPath selectors.

Right click on the first question and select “Inspect Element”:

Now grab the XPath for the <div class="summary">, //*[@id="question-summary-27624141"]/div[2], and then test it out in the JavaScript Console:

As you can tell, it just selects that one question. So we need to alter the XPath to grab allquestions. Any ideas? It’s simple: //div[@class="summary"]/h3. What does this mean? Essentially, this XPath states: Grab all <h3> elements that are children of a <div> that has a class of summary. Test this XPath out in the JavaScript Console.

Notice how we are not using the actual XPath output from Chrome Developer Tools. In most cases, the output is just a helpful aside, which generally points you in the right direction for finding the working XPath.

Now let’s update the stack_spider.py script:

from scrapy import Spider
from scrapy.selector import Selector


class StackSpider(Spider):
    name = "stack"
    allowed_domains = ["stackoverflow.com"]
    start_urls = [
        "http://stackoverflow.com/questions?pagesize=50&sort=newest",
    ]

    def parse(self, response):
        questions = Selector(response).xpath('//div[@class="summary"]/h3')

Extract the Data

We still need to parse and scrape the data we want, which falls within <div class="summary"><h3>. Again, update stack_spider.py like so:

from scrapy import Spider
from scrapy.selector import Selector

from stack.items import StackItem


class StackSpider(Spider):
    name = "stack"
    allowed_domains = ["stackoverflow.com"]
    start_urls = [
        "http://stackoverflow.com/questions?pagesize=50&sort=newest",
    ]

    def parse(self, response):
        questions = Selector(response).xpath('//div[@class="summary"]/h3')

        for question in questions:
            item = StackItem()
            item['title'] = question.xpath(
                'a[@class="question-hyperlink"]/text()').extract()[0]
            item['url'] = question.xpath(
                'a[@class="question-hyperlink"]/@href').extract()[0]
            yield item

We are iterating through the questions and assigning the title and url values from the scraped data. Be sure to test out the XPath selectors in the JavaScript Console within Chrome Developer Tools – e.g., $x('//div[@class="summary"]/h3/a[@class="question-hyperlink"]/text()') and $x('//div[@class="summary"]/h3/a[@class="question-hyperlink"]/@href').

Test

Ready for the first test? Simply run the following command within the “stack” directory:

$ scrapy crawl stack

Along with the Scrapy stack trace, you should see 50 question titles and URLs outputted. You can render the output to a JSON file with this little command:

$ scrapy crawl stack -o items.json -t json

We’ve now implemented our Spider based on our data that we are seeking. Now we need to store the scraped data within MongoDB.

Store the Data in MongoDB

Each time an item is returned, we want to validate the data and then add it to a Mongo collection.

The initial step is to create the database that we plan to use to save all of our crawled data. Open settings.py and specify the pipeline and add the database settings:

ITEM_PIPELINES = ['stack.pipelines.MongoDBPipeline', ]

MONGODB_SERVER = "localhost"
MONGODB_PORT = 27017
MONGODB_DB = "stackoverflow"
MONGODB_COLLECTION = "questions"

Pipeline Management

We’ve setup our spider to crawl and parse the HTML, and we’ve set up our database settings. Now we have to connect the two together through a pipeline in pipelines.py.

Connect to Database

First, let’s define a method to actually connect to the database:

import pymongo

from scrapy.conf import settings


class MongoDBPipeline(object):

    def __init__(self):
        connection = pymongo.Connection(
            settings['MONGODB_SERVER'],
            settings['MONGODB_PORT']
        )
        db = connection[settings['MONGODB_DB']]
        self.collection = db[settings['MONGODB_COLLECTION']]

Here, we create a class, MongoDBPipeline(), and we have a constructor function to initialize the class by defining the Mongo settings and then connecting to the database.

Process the Data

Next, we need to define a method to process the parsed data:

import pymongo

from scrapy.conf import settings
from scrapy.exceptions import DropItem
from scrapy import log


class MongoDBPipeline(object):

    def __init__(self):
        connection = pymongo.Connection(
            settings['MONGODB_SERVER'],
            settings['MONGODB_PORT']
        )
        db = connection[settings['MONGODB_DB']]
        self.collection = db[settings['MONGODB_COLLECTION']]

    def process_item(self, item, spider):
        valid = True
        for data in item:
            if not data:
                valid = False
                raise DropItem("Missing {0}!".format(data))
        if valid:
            self.collection.insert(dict(item))
            log.msg("Question added to MongoDB database!",
                    level=log.DEBUG, spider=spider)
        return item

We establish a connection to the database, unpack the data, and then save it to the database. Now we can test again!

Test

Again, run the following command within the “stack” directory:

$ scrapy crawl stack

Hooray! We have successfully stored our crawled data into the database:

Conclusion

This is a pretty simple example of using Scrapy to crawl and scrape a web page. The actual freelance project required the script to follow the pagination links and scrape each page using the CrawlSpider (docs), which is super easy to implement. Try implementing this on your own, and leave a comment below with the link to the Github repository for a quick code review. Need help? Start with this script, which is nearly complete. Cheers!

You can download the entire source code from the Github repository. Comment below with questions. Thanks for Reading!

Source: https://realpython.com/blog/python/web-scraping-with-scrapy-and-mongodb/?utm_source=Python+Weekly+Newsletter&utm_campaign=3387594e10-Python_Weekly_Issue_172_January_1_2015&utm_medium=email&utm_term=0_9e26887fc5-3387594e10-312723701

Web Scraping - It’s Your Civic Duty

noreply@blogger.com (MathBee) — Sun, 11 Jan 2015 09:24:00 +0000

Introduction

More and more information from local, state and federal governments is being placed on the web. However, a lot of the data is not presented in a way that is easy to download and manipulate. I think it is an important civic duty for us all to be aware of how government money is spent. Having the data in a more accessible format is a first step in that process.

In this article, I’ll use BeautifulSoup to scrape some data from the Minnesota 2014 Capital Budget. Then I’ll load the data into a pandas DataFrame and create a simple plot showing where the money is going.

My purpose in writing this is not to make any specific political statements about this data set. I chose this data because:

I personally found it interesting
I think it translates well across other states and across the world
It highlights several useful python tools both in and outside of the stdlib
The process is applicable to other domains outside of government data
It is a manageable size so you can understand it using basic tools

The Data

I live in MN so thought I would take a look at what sort of budget information is available to us via the various state websites. To be honest, there is a lot of information but it seems like the vast majority is stored in a PDF or on anHTML page.

I applaud the state for making the data available but it is not easy to analyze the data in the way it is currently presented. As I looked through the Minnesota government website, I found this 2014 Capital Budget page which is actually pretty straightforward to understand.

The first part of the document contains a high level summary of all the projects receiving capital dollars as well as how the capital budget will be funded.

The second part of the document has a lot of detail on each of the summary items. For the purpose of this exercise, I am only going to scrape the summary section but the same basic principle can be applied to the detailed line items.

One final note, I realize that this data set is not that large and that you could easily type it all into Excel. However, if we were to scale this to pull in more data, you quickly get to the point where hand typing the data just does not make sense. The principles I walk through will scale to much larger sets. I hope it has the added bonus that you will learn something as well. I know I enjoyed working on this little project.

The Tools

For this particular task, I am going to use 2 very common python tools for scraping the site:

BeautifulSoup to parse the data
Requests to get the data from the website.

Strictly speaking, Requests is not being used for much in this case but I think it makes sense to start using it. If/when you start getting more complicated situations, you’ll be happy you are already using it.

Scrapy is another powerful tool for doing web scraping but for my needs BeautifulSoup was perfect so that’s what I’m sticking with for this article. Maybe I’ll look at it for a future article.

Once I scrape the data, I’ll convert it to a pandas DataFrame so that I can analyze and plot the data.

One final note, I’m trying to use idiomatic python as much as possible. My current environment is python 2.7 but I’ll use theprint_function to make the python 3 conversion much easier. Also, I’m going to use the defaultdict to streamline the processing of the data. This was first introduced in python 2.5 and is pretty handy when working with dictionaries where the values are lists.

Start the Process

Import all the modules we need:

from __future__ import print_function
import os.path
from collections import defaultdict
import string
import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

Now we need to initialize the variables. I’m going to use two dictionaries. One will store all of the expense items and the other will include the funding source. Note, I am not going to store the total. We can calculate it so we’ll skip that piece of data. I am using the defaultdict to make it easy to append the values I scrape:

url = "https://www.revisor.mn.gov/laws/?year=2014&type=0&doctype=Chapter&id=294"
# Init the variables
# Use a defaultdict with an empty list because it eases the DataFrame creation
expense_lines = defaultdict(list)
funding_lines = defaultdict(list)
funding = False

Use requests to get the data and pass it to BeautifulSoup. In my final script, I’m going to store the HTML to disk so that I don’t need to hit the website every time I run it. I won’t show it in this section in order to keep the code short.

result = requests.get(url)
c = result.content
# Now that we have the data, let's process it
soup = BeautifulSoup(c)

Understand Your HTML

The key to understanding any scraping is looking at the HTML and understanding how you want to pull your data out.

In this case, I downloaded the HTML into an editor and collapsed some of the data. It is very helpful that there is a div that wraps the data I need:

<p class="enacting_clause">BE IT ENACTED BY THE LEGISLATURE OF THE STATE OF MINNESOTA:</p>
  <div class="article" id="laws.1.0.0">
    <h1 class="article_no">ARTICLE 1</h1>
    <h1 class="article_header">APPROPRIATIONS</h1>
    <div class="bill_section" id="laws.1.1.0">

Within that div, there are mutliple tables which ultimately contain the info we need:

<div class="bill_section" id="laws.1.1.0">
     <table>
       <col style="width:100%"/>
       <tr>
         <td style="border-bottom: 1px solid transparent; border-right: 1px solid transparent;" valign="bottom">
           <h1 class="bill_sec_no">Section 1. </h1>
           <h3 class="headnote">
             <span class="new" style="text-decoration: underline">CAPITAL IMPROVEMENT APPROPRIATIONS.</span>
           </h3>
         </td>
       </tr>
     </table>
     <p class="first">
     </p>
     <table>

Finally, here is a row of data:

<tr>
          <td style="border-bottom: 1px solid transparent; border-right: 1px solid transparent;" valign="bottom">
            <span class="new" style="text-decoration: underline">University of Minnesota</span>
          </td>
          <td style="border-bottom: 1px solid transparent; border-right: 1px solid transparent;" valign="bottom">
            <span class="new" style="text-decoration: underline">$</span>
          </td>
          <td style="border-bottom: 1px solid transparent; border-right: 1px solid transparent;" valign="bottom">
            <span class="new" style="text-decoration: underline">119,367,000</span>
          </td>
        </tr>

In the example above, we want to parse out two pieces of data - the description (Universty of Minnesota) and the amount (119,367,000). Another item to note is that the number comes through with commas as well as parenthesis for negative values so we are going to need to clean it up a little. I also found that I pulled in a lot of extra white space in the process, so using string.strip is a good idea.

Here is the clean up function we’ll use:

def convert_num(val):
    """
    Convert the string number value to a float
     - Remove all extra whitespace
     - Remove commas
     - If wrapped in (), then it is negative number
    """
    val = string.strip(val).replace(",","").replace("(","-").replace(")","")
    return float(val)

Parse the HTML

Now that we know how to get to our tables, use BeautifulSoup’s powerful API to get at our data.

# After looking at the data, we can see that the summary has a div id we can use
summary = soup.find("div", {"class":"bill_section","id": "laws.1.1.0"})

# Get all the tables in the summary
tables = summary.find_all('table')

# The first table is not useful header info
# The second table contains all the we need (the list is 0 indexed)
data_table = tables[1]

Parse each row in the table and add to the appropriate dictionary depending on whether it is a funding line or expense line:

#Go through each row of the table and pull out our data
for row in data_table.find_all("tr"):
    cells = row.find_all("td")
    # Ignore lines that don't have 3 cells of data because it is just spacing
    if len(cells) == 3:
        line = (string.strip(cells[0].text), convert_num(cells[2].text))
        # Once we get to the total line we start getting the funding lines
        if line[0] == "TOTAL":
            funding = True
            # We don't want to capture the total because we can calc it
            continue
        if funding:
            funding_lines[line[0]].append(line[1])
        else:
            expense_lines[line[0]].append(line[1])

Convert the Data

Our dictionaries contain the data we need, let’s add them to a pandas DataFrame using DataFrame.from_dict() :

# Create the DataFrame using from_dict
expense_df = pd.DataFrame.from_dict(expense_lines,orient='index')
funding_df = pd.DataFrame.from_dict(funding_lines,orient='index')
# Label our column
expense_df.rename(columns={0: 'Amount'}, inplace=True)
funding_df.rename(columns={0: 'Amount'}, inplace=True)

First, we look at a small subset of the spending lines:

expense_df.head()

	Amount
Administration	127000000
Bond Sale Expenses	900000
Minnesota State Academies	11354000
Public Facilities Authority	45993000
Housing Finance Agency	20000000

Then look at how it is funded.

funding_df

	Amount
State Transportation Fund	36613000
Trunk Highway Fund	7950000
Bond Proceeds Fund (User Financed Debt Service)	39104000
Bond Proceeds Cancellations	-10849000
Maximum Effort School Loan Fund	5491000
Bond Proceeds Fund (General Fund Debt Service)	814745000

We can check our totals too to make sure we processed the data correctly.

expense_df["Amount"].sum()

893054000.0

funding_df["Amount"].sum()

893054000.0

It looks like everything was processed correctly. Now, we can analyze the data any way we want.

Plot The Data

In this specific case, I am going to generate a simple horizontal bar graph so that it is easy to see where the biggest expenditures are.

First, I’ll sort both sets of data:

expense_df = expense_df.sort(columns='Amount')
funding_df = funding_df.sort(columns='Amount')

Making Nice Plots

If you don’t learn anything else from this article, don’t forget that you can make your default plots look a lot nicer by using a simple pandas settings - pd.options.display.mpl_style = 'default'

Set our default plot style so it looks nicer:

pd.options.display.mpl_style = 'default'

Plot horizontal bar chart

expense_df.plot(kind='barh', figsize=[6, 10],title="2014 MN Capital Budget Spending")

For comparison, here is what the image looks like if you don’t set pd.options.display.mpl_style

Regardless of the format, I think you’ll agree that viewing the capital budget in this plot yields a lot more insight than the raw HTML data.

Final Thoughts

This little project has been useful for me and I hope it provides a starting point for you to understand how to use various python tools to scrape the web. In this case, I learned a little bit that I think could be applicable to lots of other projects. I also am curious about this little slice of data and intend to look into it some more and see what insight I can glean.

For reference, here is the complete code for this example. This version will download the data to a file and use that locally instead of hitting the site each time.

#Parse 2014 MN Capital budget - https://www.revisor.mn.gov/laws/?year=2014&type=0&doctype=Chapter&id=294
#Store the summary in a DataFrame for eventual manipulation
from __future__ import print_function
import os.path
from collections import defaultdict
import string
import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

filename = "MNBudget-2014.html"
url = "https://www.revisor.mn.gov/laws/?year=2014&type=0&doctype=Chapter&id=294"

def convert_num(val):
    """
    Convert the string number value to a float
     - Remove all extra whitespace
     - Remove commas
     - If wrapped in (), then it is negative number
    """
    val = string.strip(val).replace(",","").replace("(","-").replace(")","")
    return float(val)

# As we work through the process, it is easier to
# download it once and work with the saved copy instead of
# trying to hit the server each time
# Just delete the output file to force a new download
if os.path.isfile(filename):
    print("Loading the data via the file.")
    f = open(filename, 'r')
    c = f.read()
else:
    print("Fetching the data via the URL.")
    result = requests.get(url)
    c = result.content
    f = open(filename,'w')
    f.write(c)
f.close()

# Init the variables
# Use a defaultdict with an empty list because it eases the DataFrame creation
expense_lines = defaultdict(list)
funding_lines = defaultdict(list)
funding = False

# Now that we have the data, let's process it
soup = BeautifulSoup(c)

# After looking at the data, we can see that the summary has a div id we can use
summary = soup.find("div", {"class":"bill_section","id": "laws.1.1.0"})

# Get all the tables in the summary
tables = summary.find_all('table')

# The first table is not useful header info
# The second table contains all the we need (the list is 0 indexed)
data_table = tables[1]

#Go through each row of the table and pull out our data
for row in data_table.find_all("tr"):
    cells = row.find_all("td")
    # Ignore lines that don't have 3 cells of data because it is just spacing
    if len(cells) == 3:
        line = (string.strip(cells[0].text), convert_num(cells[2].text))
        # Once we get to the total line we start getting the funding lines
        if line[0] == "TOTAL":
            funding = True
            # We don't want to capture the total because we can calc it
            continue
        if funding:
            funding_lines[line[0]].append(line[1])
        else:
            expense_lines[line[0]].append(line[1])

# Create the DataFrame using from_dict
expense_df = pd.DataFrame.from_dict(expense_lines,orient='index')
funding_df = pd.DataFrame.from_dict(funding_lines,orient='index')
# Label our column
expense_df.rename(columns={0: 'Amount'}, inplace=True)
funding_df.rename(columns={0: 'Amount'}, inplace=True)

expense_df = expense_df.sort(columns='Amount')
funding_df = funding_df.sort(columns='Amount')

print(expense_df["Amount"].sum())
print(funding_df["Amount"].sum())

#Set some nicer defaults for plots
pd.options.display.mpl_style = 'default'

expense_bar = expense_df.plot(kind='barh', figsize=[7, 13],title="2014 MN Capital Budget Spending")
plt.savefig("MN-2014-Expense.png")

funding_bar = funding_df.plot(kind='barh',title="2014 MN Capital Budget Funding")
plt.savefig("MN-2014-Funding.png")

An Architecture for Real-Time Geo-tracking with Python, Celery, RabbitMQ, and More

noreply@blogger.com (MathBee) — Thu, 18 Dec 2014 10:20:00 +0000

Source: http://blog.pivotal.io/pivotal/case-studies-2/an-architecture-for-real-time-geo-tracking-with-python-celery-rabbitmq-and-more

If you look at a map online and someone asked what you see, most people will answer, “a map.”

That’s not the answer you would get from Ragi Burham. He sees the underlying geometry, vectors, styling, raster data, GUI controls, application logic, and data store. That’s because Burham is an expert in the area of mapping solutions, and his background includes over a decade of geo-related work with organizations like ESRI, NIMA, National Geographic, Navteq, TeleAtlas, and his own company, AmigoCloud.

At Pivotal, we get excited about people like Ragi Burham—an open source minded software engineer with a new way of doing things. At PyCon in March of 2013, Burham shared a talk titled, “Realtime Tracking and Mapping of Geographic Objects using Python”—it’s another way to look at geography-based big, fast data architectures.

Below, we recap the “Normal” and “Real-time” architectures that Burham shared in his talk. These two approaches use OpenLayers, Varnish, Nginx, Gunicorn, Django,TileStache, Memcached, Mapnik, PostgreSQL, PostGIS, Socket.io, Node.js, Celery, andRabbitMQ. Much like our recent post on scaling social media with Celery and RabbitMQ, Burham adds these two components along with Socket.io and Node.js to help his real-time architecture work in a fundamentally differently way and scale.

If you have 26 minutes, you can also view the video below.

http://www.youtube.com/embed/WQQr0UM17gI

A “NORMAL” GEO-BASED ARCHITECTURE
THE CHALLENGES WITH REAL-TIME GEO-BASED ARCHITECTURES

Beyond an explanation of underlying geometry, vectors, styling, raster data, and architecture, Burham covered some of use cases and requirements driving the specifics of what geo-based systems do—determining the amount of information on the map, omitting certain types of mapping objects, making calculations like speed or direction, alerting that something has crossed a boundary, and more.

He began outlining his “Normal” GeoStack architecture by listing the most popular open source data sets and covering some of the pros and cons of OpenStreetMap,NAIP, SRTM, and Natural Earth Data. With the data sources covered, he filled in the details of the architecture. This included a client with OpenLayers, a JavaScript library that helps with rendering maps (another option is Leaflet). Varnish was used in front of servers with the following components: Nginx, Gunicorn, TileStache, Memcached, and Mapnik. Then, a PostgreSQL database stores the map information with the PostGIS spatial database extension.

Because rendering style can take a lot of time, Varnish is used for speeding things up and can also be used as a reverse proxy. For those unfamiliar, TileStache is used to serve map tiles based on rendered geographic data the Mapnik toolkit is used for map development. Here is a screen shot of his normal architecture:

One of the key challenges with this type of architecture is when you need to get state from the database. Clients end up polling for many real-time tracking scenarios. This presents a problem when, for example, you have 1000s of vehicles confirming state every second. Polling the database at this rate can create an overwhelming and unnecessary amount of traffic. And it doesn’t scale.

But, what if the server could notify a client? To evolve the architecture, Burham proposes four additional components—Node.js with Socket.io and Celery with RabbitMQ.

By adding Socket.io to the client, we get a way to use websockets across browsers and allow data to be transported bi-directionally. We can send information from the server back to the client specifically and only when it is needed. Varnish changes purpose slightly in this architecture—it still does the HTTP caching, but now it also acts as a reverse proxy. You can configure small lines of code on Varnish to deal with traffic differently. For example, if websocket traffic comes in, you can tell it to go to server A, and, if other types of traffic come in, use server B. In this case, Node.js is connected with Varnish and offers a way to push or publish notifications back out to clients.

Now, we need a way to take an incoming client’s geo-position update, asynchronously decide if some alert should happen, and distribute a notification back out for only certain clients. This is where we add Celery and RabbitMQ to the architecture. With these two components, we can support asynchronous tasks and queues. If you aren’t familiar with Celery, it is a task queue based on distributed message passing. Your code can call Celery tasks to execute concurrently and operate in either asynchronous or synchronous modes. This is great for geographic use cases like crossing a boundary or fence—when something is entering or leaving a polygon. You don’t need to calculate when an update happens in real-time, you can receive a task, make a calculation, and decide to notify others later. With the addition of RabbitMQ (Celery’s default broker), we then transform these tasks and calculations into a Pub/Sub channel where Node.js is listening. Whenever an appropriate event is triggered, we can push a message down the channel, and Node.js forwards the information back to the appropriate clients using Socket.io.

Flask performance optimization

noreply@blogger.com (MathBee) — Thu, 18 Dec 2014 07:33:00 +0000

Source: http://www.vadimkravcenko.com/flask-performance/

Here are my performance optimization guidelines that I try to follow

General

After transitioning from PHP to Python development I was quite surprised by the lack of performance optimization tips and scaling advice in general. So with this post I will try and lay out the basic ideas of scaling a website from 10 to 50,000 users with Flask Framework.

Server Setup

It all starts with your server setup, and the server depends on your budget. There are several options that you can chose from:

4GB RAM 8 Core VPS from Linode (80$)
4GB RAM 2 Core VPS from DigitalOcean (40$)
Or a dedicated server from any of hundreds of hosting companies.

But I would strongly advise taking something with >= 2GB RAM. This much RAM and a decent CPU will help you scale to hundreds of thousands of views per day. (With the proper amount of caching of course)

Frontend Reverse Proxy

Now that we have our server, we need to setup a frontend proxy. Your choice is between Apache, Lighttpd or Nginx. I would suggest using Nginx because it’s event-driven (that means amazingly low memory footprint) and possibility to serve 10,000 concurrent users with no problems at all.

Caching static pages with Nginx itself

Before the users even gets to your dynamic site, it’s usually a good idea to cache some pages with nginx. For example you have a page with a list of posts that updates only once a day, or a comments page, that updates only once a week. For such pages you can setup Nginx caching, so that the user wont even get to the python interpeter. A guide on how to setup Filesystem cache is here, or if you want to setup Nginx + Memcached guide.

UWSGI and Gevent

Now that the user got to the dynamic part of your application, we need to decide what kind of web server we want to use. There are different benchmarks here and here that praise different kinds of web servers. But I personally prefer UWSGI, because it’s fast and lightweight plus there are many different CLI options that you can provide. For example: Auto-reload python compiled files on every request. (Good for development).
Now we add some asynchronous IO and lightweight multi-threading to it with Gevent and boom we have a very fast web server, able to handle thousands of concurrent connections.

Flask PyPy

If you’re not using any 3rd party extensions that are mainly written in C (Like the MySQLdb driver for python) your should consider trying PyPy for more performance boost. Flask is fully compatible with PyPy although some patching will be required to use Gevent with PyPy (source)
Here are some benchmarks CPython vs PyPy: here and here

Flask-Cache

This should be the first extension that you install. It’s easy to setup and easy to use.
Example caching with decorator:

@cache.cached(timeout=500, key_prefix='daily_posts')
def get_posts():
    posts = get_all_posts_with_multiple_joins_heavy_db()
    return [x.content for x in posts]

cached_posts = get_posts()

Memcached/Redis

Another advice I can give you is storing EVERYTHING, that can be cached, in memory. It doesn’t matter what kind of system you prefer (Although a horizontally scalable structure would be nice) Memcached, Redis, Voldemort. The main point is, cache, cache, cache. Don’t do anything dynamic if you can get the results from cache.

Jinja2 template bytecode caching

With Jinja2 you have several options of caching bytecode of compiled templates:

FileSystemBytecodeCache
MemcachedBytecodeCache

Use them.

Consider using ElasticSearch

If you’re doing a lot of queries like this:

SELECT * FROM posts WHERE author='someauthor'

Then you should definitely consider replacing these kinds of queries with ElasticSearch. Here are some articles on ES performance: here and here
Although ES is not a database, you can feed data into it from your RDBMS and then serve the data to user from it.
The main points are:

ES is easily scalable. You can scale it from 1 to 10 servers “with just a click of a button”
Very fast. Based on Apache Lucene the FULL-TEXT search capabilities are extremely fast.
Easy to use. You can add remove documents with requests to the REST API.
Active development and big community.

Code Profiling and Optimization

Now we’ve come down to the last thing you must regularly do. Profile your application! Check out how many queries you’re doing per page, how long does the sorting last, maybe there are ways you can optimize the nested for loops? There’s always something that can be rewritten.
Good for you there’s a good profiling tool for Flask that makes it easy to find the bottlenecks of your application

Conclusion

I’m pretty sure everyone has their own “performance guidelines” that they usually follow with their apps. But this one is mine, and I hope you found out something useful here.

What is the technology stack behind Pinterest?

noreply@blogger.com (MathBee) — Thu, 18 Dec 2014 07:28:00 +0000

Paul Sciarra, I'm a co-founder.

377 upvotes by Quora User, Yash Nelapati, Ben Silbermann, (more)

We use python + heavily-modified Django at the application layer. Tornado and (very selectively) node.js as web-servers. Memcached and membase / redis for object- and logical-caching, respectively. RabbitMQ as a message queue. Nginx, HAproxy and Varnish for static-delivery and load-balancing. Persistent data storage using MySQL. MrJob on EMR for map-reduce.

Quora’s Technology Examined

noreply@blogger.com (MathBee) — Thu, 18 Dec 2014 07:27:00 +0000

Source: http://www.bigfastblog.com/quoras-technology-examined

Quora has taken the tech and entrepreneurial world by storm, providing a system that works so fluidly that it is sometimes hard to see what the big fuss is all about. This slick tool is powered, not only by an intelligent crowd of askers and answerers, but by a well-crafted backend created by co-founders who honed their skills at Facebook.

It is not surprising that, with all the smart people using this smart tool, there are many pondering on how it works so well. The NoSQL boffins scratch their heads and ponder such questions as, “Why does Quora use MySQL as the data store rather than NoSQLs such as Cassandra, MongoDB, CouchDB, etc?“.

In this blog post I will delve into the snippets of information available on Quora and look at Quora from a technical perspective. What technical decisions have they made? What does their architecture look like? What languages and frameworks do they use? How do they make that search bar respond so quickly?

Components Of Quora

The general components that make up Quora are…

You can ask questions
You can answer questions (anonymously if you desire)
You can comment on answered questions
You can vote-up or vote-down answers to questions
Questions can be assigned to topics
You can write a post (a informative statement, rather like a orphaned answer or blog post)
You can follow questions, topics or other users
A super-fast auto-complete search-box at the top, which doubles as the method for entering new questions

The last point, the super-fast auto-complete search-box, is one of the defining features of Quora. You can see immediately, as you begin to enter a question, whether somebody else has already asked that question or if there is a topic or post on the subject. Let’s start there…

What’s Cooking Under That Hood?

Only the questions, topic labels, user names or post titles are indexed and served up to the search-box. There is no full-text search, so searching the content of questions and answers will not work. The text that is indexed is tokenized so that words in a different order will still be matched. Prefix matching enables best matches to be shown before the entire word is entered. For instance, typing “mi” might immediately show “Microsoft” in the results.

There is some simple stemming of words, since “nears” matches “near”, but “pony” does not match “ponies”. “Topic-aliases” allow for similar matches on topic names, such as “startup” and “start-up”. These topic-aliases have been manually entered by users. Otherwise these would not match.

If a duplicate question is redirected to another question (a feature of Quora), then that original question will still appear in the search results, since it increases the chances of a match. There is no n-gram indexing, so slight mis-spellings will not match. For instance, “gooogle” (with an extra “o”) finds nothing.

Previously, they did use an open source search server, called Sphinx. It supports the features they are using above, but they have since moved from this due to real-time constraints. Their new solution is built in-house and allows them better prefix indexing and control over the matching algorithms. They built this in Python.

What libraries does Quora use for search?
Adam D’Angelo, Quora Founder (Nov 13, 2010)
Our search is custom-written. It doesn’t use any libraries aside from Thrift, and Python’s unicode library, which we use for unicode normalization.

Speedy Queries

Did I mention that the search-box is fast? I did some tests and found the responses to be impressive. Queries are sent over AJAX as a GET request. Responses come back as JSON with the rendered HTML embedded inside the JSON. Rendering of the results on the server-side, as opposed to rendering them in JavaScript, seems to be due to the need to highlight matching words in the text. This is sometimes too complex for JavaScript. For instance, typing “categories” might highlight the world “category” in the result text.

I was seeing responses of roughly 50 milliseconds per query from my Linode machine. Quora does not short-change you when sending requests. From within the browser, I found typing “Microsoft” (9 characters) would result in nine requests to the Quora search server, no matter how fast you type. As you will see later, the server is in control, so if it did become over-loaded, then it could update the results less frequently without changing the JavaScript.

Quora uses persistent connections. A HTTP connection is established with the server when you start typing the search query. This connection is kept open and further requests are made on this same open connection. The connection will terminate (times-out) if not used for 60 seconds. If a connection times-out then a new connection is established when typing begins.

To simulate the typing of a word into the search-box, I sent the following requests, character-by-character, across a persistent connection. For instance “butler” is six requests (“b”, “bu”, “but” … “butler”).

"butler" (6 chars) duration: 0.393 secs 0.065 secs per query
"butler monkeys" (14 chars) duration: 0.672 secs 0.048 secs per query
"fasdisajfosdffsa" (16 chars) duration: 0.746 secs 0.046 secs per query

That last query was used to test if there was a slow-down for a word that would obviously not be in a caching layer. I saw no slow-down. This means that they are not caching, caching is only used to take the load off the backend search engine or they are doing something smarter (e.g. if there is no match for “fasd” then there will be no match for “fasdi”, so abort).

Is Quora going to implement full-text search?
Adam D’Angelo, I made a lot of the early Quora … (Sep 1, 2010)
Yes, eventually. We haven’t implemented this yet because we’ve prioritized other things, but we will definitely do it in the future.

Webnode2 And LiveNode

Webnode2 and LiveNode are some of Quora’s internal systems, which were built for managing the content. Webnode2 generates HTML, CSS and JavaScript and is tightly coupled with LiveNode, which is responsible for managing the display of the content on the webpage. Charlie Cheever says that he were to start a similar project without LiveNode, then the first thing he would do is rebuild it.

They seem very pleased with the technology they have built and struggled to find its weaknesses. One weakness is that it is tricky for LiveNode to keep track of what is happening within the browser as it pushes changes from the server. If users A and B are viewing the same question then ones interactions will affect the other. For instance, if user A up-votes an answer then that answer will be promoted and will visibly move up the page. This display change will be pushed over AJAX to user B’s browser. Any prior browser-side change that user B made, such as expanding a comments section, might be lost.

LiveNode is written in Python, C++, and JavaScript. jQuery and Cython is also used.

While they would like to open-source LiveNode and have tried to keep code separation, doing so right now would be too much work and would take time away from their main goal, which is making Quora better.

Charlie Cheever points out that webnode2 is unrelated to the “free and easy website builder” called Webnode at webnode.com.

Amazon Web Services

Amazon EC2 and S3 is used for their hosting. While this is not as cost-effective in the long-term as running your own servers, it is perfectly designed for fast growing companies like Quora.

Ubuntu Linux

Quora uses Ubuntu Linux as its OS of choice. No major surprises there. It is easy to deploy and manage on Amazon EC2. Adam D’Angelo points out that he used Debian Linux at high school and college and stuck with it because “it works and there hasn’t been a compelling reason to switch”.

Static Content

You only need to look at the source HTML of any Quora webpage to see that they are using Amazon’s distributed content delivery network, Cloudfront. URLs are in the form…

http://d2o7bfz2il9cb7.cloudfront.net/main-thumb-670336-25-7kmigSSkkdusoE6gHRkdQsXfjuTCaxQs.jpeg

CloudFront is used for all static images, CSS and JavaScript (except for Google’s Analytics JavaScript, which is hosted by Google). Images are uploaded to the EC2 machine, then resized and uploaded to S3. This is managed using the Python S3 API.

HAProxy Load-Balancing

Quora uses HAProxy at the front-line, which load-balances onto the distributed Nginx servers behind them.

Nginx

Behind the load-balancer, Nginx is used as a reverse-proxy server onto the web-servers.

To understand more about this setup I recommend reading “Using Nginx As Reverse-Proxy Server On High-Loaded Sites“.

Pylons And Paste

Pylons, a lightweight web framework, is used as their main web-server behind Nginx. They use the defaultPylon + Paste stack.

Pylons was selected much like you would select a pumpkin for Haloween. They scooped out the insides, such as templates and the ORM, and replaced it with their own technology, written in Python. This is whereLiveNode and webnode2 reside.

MochiMedia was also one of the inspirations for using Pylons, since they were using it themselves.

Python

Coming from Facebook, it was a good bet that Charlie and Adam would choose PHP for their development language. As Adam points out, “Facebook is stuck on that for legacy reasons, not because it is the best choice right now“. From this experience they knew that choosing technologies, especially programming languages, for the long-run was very important. They also looked at C#, Java, and Scala. Discounting Mono, C# would be a choice of more than just the language. It would require them to build on-top of a Microsoft stack. Python won over Java because it is more expressive and quicker to write code than Java. Scala was too new. Adam mentions speed and the lack of type-checking as drawbacks with Python, but they both already knew the language reasonably well. Where Python lacks speed for performance critical backend components, they opt to write them in C++. They saw Ruby as a close match to Python, but their experience with Python and lack of experience in Ruby, made Python the winner. Python 2.6, to be precise.

Additional benefits for using Python are the fact that data-structures that map well to JSON, code readability, there is a large collection of libraries and the availability of good debuggers and reloaders. Browser-server communication using JSON is major component of what Quora does, so this was an important factor.

No IDEs are used for development as most use the Emacs text editor. Obviously this is personal choice, and would change as the team grows.

PyPy, a project that aims to produce a flexible and fast Python implementation, was also mentioned as something that might give them a speed-boost.

Thrift

Thrift is used for communications between backend systems. The Thrift service is written in C++.

Why would you write a Thrift service in C++?
Adam D’Angelo, I’ve written a lot of Python, in… (Sep 4, 2010)
Mainly if you want to keep data in memory between requests, and want to keep your Python code stateless. Writing a Python wrapper around a C library involves some memory management with reference counting that requires some understanding of the Python internals, but writing a thrift interface is simple. You also isolate failures this way – if the service goes down it won’t take the Python code down with it.

Tornado

The Tornado web framework is used for live updating. This is their Comet server, which handles the large volumes of open connections used for long-polling and pushes updates to the browsers.

Long Polling (Comet)

Quora does not display just static web pages. Each page will update will new content as questions, answers and comments are submitted by other others. As Adam D’Angelo points out, one of the best ways to do this currently is with “long polling”. This is different to “polling”. With polling the browser will repeatedly send requests to the server saying “Any updates?” and the server will respond with “No”. A few seconds later it will ask again, “How about now?”. “No”. “How about now?”, “No, already!”. This puts the client (web-browser) in the driver’s seat. This is backwards because the client does not how long to wait before asking again. If the client asks the server too frequently then it will unduly overload the server. If it pings the server too infrequently, then server will be sitting on updates while it waits for the client to request them and the end-user will not see updates immediately.

Long polling, also known as Comet, puts the server in control, by making the client wait for responses. The conversation between the client and server is the same, but instead of the client waiting before making another request, the server waits before it makes the response. The server can keep the connection open for a long period of time (e.g. 60 seconds) while it waits to see if any updates come in. When updates do come in it can respond immediately to the client. On receiving the update, the client then immediately sends a new request for more updates. The server, once again, delays responding until it knows something worth telling the client or enough time has past that it would be rude not to respond.

The benefit to long-polling is that there is less back-and-forth between the client and server. The server is in control of the timing, so updates to the browser can be made within milliseconds. This makes it ideal for chat applications or applications that want really snappy updates for their users.

The down-side is that you are going have lots of open connections between the clients and your servers. If you have a million users (Quora will soon) and, if only 10% of them are online on your site, you will need an architecture that can hold open at least 100,000 concurrent connections. This assumes they only have one tab open to your site. Right now I have 7 tabs open for quora.com in my browser. Each tab usually has multiple connections open to quora.com. In short, Quora must maintain a lot of open connections.

The good news is that there are technologies specifically designed for this. It costs very little to hold open connections in memory if you free up all the resources used for that connection. For instance, Nginx (Quora uses this for proxying requests) is a single-threaded event-based application and uses very little memory for each connection. Each Nginx process is actively dealing with only one connection at a time. This means it can scale to tens of thousands of concurrent connections.

How do you push messages back to a web-browser client through AJAX? Is there any way to do this without having the client constantly polling the server for updates?
Adam D’Angelo, Quora (Sep 29, 2010)
There is no reliable way to do this without having the client polling the server. However, you can make the server stall its responses (50 seconds is a safe bet) and then complete them when a message is ready for the client. This is called “long polling” and it’s how Quora, Gmail, Meebo, etc all handle the problem.
If you have a specialized server that uses epoll or kqueue, you should be able to hold on the order of 100k users per server (depending on how many messages are going). This is called the “c10k” problem. http://www.kegel.com/c10k.html

MySQL

Just like Facebook, where co-founder Adam D’Angelo previously worked, Quora heavily uses MySQL. In answer to the Quora question “When Adam D’Angelo says “partition your data at the application level”, what exactly does he mean?“, D’Angelo goes into the details of how to use MySQL (or relational-databases generally) as a distributed data-store.

The basic advice is to only partition data if necessary, keep data on one machine if possible and use a hash of the primary key to partition larger datasets across multiple databases. Joins must be avoided. He sites FriendFeed’s architecture as a good example of this. FriendFeed’s architecture is described by Bret Taylor in his post “How FriendFeed uses MySQL to store schema-less data“. D’Angelo also states that you should not use a NoSQL database for a social site until you have millions of users.

It is not only Quora and FriendFeed who are heavily using MySQL. Ever heard of “Google”? It is hard to imagine, since everything Google does has to scale so well, but in the words of Google, “Google uses MySQL [...] in some of the applications that we build that are not search related”. Google has released patches for MySQL related to replication, syncing, monitoring and faster master promotion.

How does one evaluate if a database is efficient enough to not crash as it’s put under increasing load?
Adam D’Angelo, Quora (Oct 10, 2010)
One option is to simulate some load. Write a script that mimics the kinds of queries your application will be doing, and make sure it can handle the amount of load you want it to be ready for (especially as the size of the dataset changes).

Memcached

Memcached is used as a caching layer in front of MySQL.

Git

Git is used for version control.

JavaScript Placement

If you look at Quora’s source you will see that the JavaScript comes at the end of the page. Charlie Cheeversuggests that this give the feeling of a quicker loading page, since the browser has content to display before the JavaScript has be seen.

Charlie Cheever Follows “14 Rules for Faster-Loading Web Sites”

Steve Souders, author of High Performance Web Sites and Even Faster Web Sites, lists the following rules for making websites faster. This list is mentioned by Charlie Cheever, the co-founder of Quora, as one of the reasons for Quora’s speed.

“One resource we used as a guide is Steve Souders’ list of rules for high performance websites:http://stevesouders.com/hpws/rules.php”
– Charlie Cheever, Quora

Steve Souders’ 14 rules are…

Make Fewer HTTP Requests
Use a Content Delivery Network
Add an Expires Header
Gzip Components
Put Stylesheets at the Top
Put Scripts at the Bottom
Avoid CSS Expressions
Make JavaScript and CSS External
Reduce DNS Lookups
Minify JavaScript
Avoid Redirects
Remove Duplicate Scripts
Configure ETags
Make AJAX Cacheable

Conclusion

Quora is a great example of a modern tech start-up. They are very small team who understand the technologies they are using very well. They have made considered choices in the technology they have selected and have a good vision of which components would be better written from scratch. They seem keen to share these in-house technologies with the open-source community and I look forward to when they have the time to make this a reality.

Using Nginx As Reverse-Proxy Server On High-Loaded Sites

noreply@blogger.com (MathBee) — Thu, 18 Dec 2014 07:13:00 +0000

Source: http://kovyrin.net/2006/05/18/nginx-as-reverse-proxy/

Two weeks ago we have started new version of one of our primary web projects and have started very massive advertisement campaign to promote this web site. As the result of that advertisements, our outgoing traffic has been increased to 200-250Mbit/s from only one server! In this article I will describe, how to build stable and efficient web site with two-layer architecture (with frontend + backend web servers) or how to modify your current server configuration to get additional resources to handle more requests.

First of all, let me describe general structure of web-server and how it handles clients requests:

Client initiates request to your server.
His browser connects to your server.
Your server (as for Apache) creates new thread/process to handle request.
If client requested dynamic content, web server spawns CGI process or executes dynamic content handling module (i.e. mod_php) and waits while request will be processed. When it receives result web-page, it sends it to client.
If client asked for some static file, web server sends this file to client
Client’s browser receives answer, closes connection to web server and displays content.

As you can see, when there are many requests coming to your server, your server needs to create many parallel threads/processes and keep them running while client will close connection. If client has slow connection, web server process will wait too long and resource consumption will increase very fast.

What we can do in such situation? Simple solution is to buy more memory and more CPUs for your server and wait while web server load will crash your server. But there is more efficient solution! You can simply put some small piece of software (nginx, for example) behind your big web server and let it handle all requests to static content and to pass all dynamic requests to primary web-server. With this solution your big server will spawn additional threads/processes only for static pages and it will return answers to small frontend very fast and then can free resources to use them to handle another queries. Small frontend can wait very long time while client will receive his content and will close connection – backend server will not consume resources for such long time!

Here you can see simple diagram of proposed web server configuration:

As additional benefit from such configuration you can get very useful feature of managed downloads that will be described below.

If your server contains some static resources, which can be downloaded not by all users (content provider can provide mp3 files only to users with positive balance or some site can provide downloads only to logged-in users), in generic configuration you need to create some script to handle this downloads and to create some ugly links like http://some.service.com/down.php?file=xxx.mp3 and additionally your users will not be able to resume downloads (except such cases when your script so complex, that it handles Ranges HTTP-header)…

In configuration with nginx frontend, you can create simple URL-rewriting rule, that will pass all requests to pretty URLs like http://your.cool-service.com/files/cool.mp3 to some simple script /down.php automatically and, if this script has returned X-Accel-Redirect header, will send requested file to user automatically with Ranges header support and when user will download his cool content, your backend server can handle other requests. Your users even will not know that your script controls their downloads. Simple diagram for described algorithm is following:

Let me bring your attention to interesting fact: If you only accelerate your site with described technique and do not want to create download control system, you do not need to modify any of your scripts on your backend server! They will work as in original configuration!

So, the last thing you need to boost your web server with nginx reverse proxying technique is following configuration file snipet:

    server {
        listen       80;
        server_name  some-server.com www.server-name.com;

        access_log  logs/host.access.log  main;

        # Main location
        location / {
            proxy_pass         http://127.0.0.1:8080/;
            proxy_redirect     off;

            proxy_set_header   Host             $host;
            proxy_set_header   X-Real-IP        $remote_addr;
            proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

            client_max_body_size       10m;
            client_body_buffer_size    128k;

            proxy_connect_timeout      90;
            proxy_send_timeout         90;
            proxy_read_timeout         90;

            proxy_buffer_size          4k;
            proxy_buffers              4 32k;
            proxy_busy_buffers_size    64k;
            proxy_temp_file_write_size 64k;
        }

        # Static files location
        location ~* ^.+.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js)$ {
            root   /spool/www/members_ng;
        }

    }

Full version of sample config file you can get here.

Notice: If your backend scripts are using user IP addresses for some purposes, you will need to installmod_rpaf module to use X-Real-IP header provided by nginx instead of real user’s IP address.

That is all! Now you can simply install nginx on your server, configure it and your server will be able to handle more traffic with the less resources that it uses now! Everything will be done transparently for your currently written scripts and if you want, you will be able to provide download handling with simple trick, that I will describe in my next post

If you have some questions, do not hesitate to ask them here in comments – I will try to answer all of them. If you liked this article, you can support author by taking a look at advertisements on this page or simply

How FriendFeed uses MySQL to store schema-less data

noreply@blogger.com (MathBee) — Thu, 18 Dec 2014 07:00:00 +0000

Source: http://backchannel.org/blog/friendfeed-schemaless-mysql

BACKGROUND

We use MySQL for storing all of the data in FriendFeed. Our database has grown a lot as our user base has grown. We now store over 250 million entries and a bunch of other data, from comments and "likes" to friend lists.

As our database has grown, we have tried to iteratively deal with the scaling issues that come with rapid growth. We did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. However, as we grew, scaling our existing features to accomodate more traffic turned out to be much less of an issue than adding new features.

In particular, making schema changes or adding indexes to a database with more than 10 - 20 million rows completely locks the database for hours at a time. Removing old indexes takes just as much time, and not removing them hurts performance because the database will continue to read and write to those unused blocks on every INSERT, pushing important blocks out of memory. There are complex operational procedures you can do to circumvent these problems (like setting up the new index on a slave, and then swapping the slave and the master), but those procedures are so error prone and heavyweight, they implicitly discouraged our adding features that would require schema/index changes. Since our databases are all heavily sharded, the relational features of MySQL like JOIN have never been useful to us, so we decided to look outside of the realm of RDBMS.

Lots of projects exist designed to tackle the problem storing data with flexible schemas and building new indexes on the fly (e.g., CouchDB). However, none of them seemed widely-used enough by large sites to inspire confidence. In the tests we read about and ran ourselves, none of the projects were stable or battle-tested enough for our needs (see this somewhat outdated article on CouchDB, for example). MySQL works. It doesn't corrupt data. Replication works. We understand its limitations already. We like MySQL for storage, just not RDBMS usage patterns.

After some deliberation, we decided to implement a "schema-less" storage system on top of MySQL rather than use a completely new storage system. This post attempts to describe the high-level details of the system. We are curious how other large sites have tackled these problems, and we thought some of the design work we have done might be useful to other developers.

OVERVIEW

Our datastore stores schema-less bags of properties (e.g., JSON objects or Python dictionaries). The only required property of stored entities is id, a 16-byte UUID. The rest of the entity is opaque as far as the datastore is concerned. We can change the "schema" simply by storing new properties.

We index data in these entities by storing indexes in separate MySQL tables. If we want to index three properties in each entity, we will have three MySQL tables - one for each index. If we want to stop using an index, we stop writing to that table from our code and, optionally, drop the table from MySQL. If we want a new index, we make a new MySQL table for that index and run a process to asynchronously populate the index without disrupting our live service.

As a result, we end up having more tables than we had before, but adding and removing indexes is easy. We have heavily optimized the process that populates new indexes (which we call "The Cleaner") so that it fills new indexes rapidly without disrupting the site. We can store new properties and index them in a day's time rather than a week's time, and we don't need to swap MySQL masters and slaves or do any other scary operational work to make it happen.

DETAILS

In MySQL, our entities are stored in a table that looks like this:

CREATE TABLE entities (
    added_id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
    id BINARY(16) NOT NULL,
    updated TIMESTAMP NOT NULL,
    body MEDIUMBLOB,
    UNIQUE KEY (id),
    KEY (updated)
) ENGINE=InnoDB;

The added_id column is present because InnoDB stores data rows physically in primary key order. The AUTO_INCREMENT primary key ensures new entities are written sequentially on disk after old entities, which helps for both read and write locality (new entities tend to be read more frequently than old entities since FriendFeed pages are ordered reverse-chronologically). Entity bodies are stored as zlib-compressed, pickled Python dictionaries.

Indexes are stored in separate tables. To create a new index, we create a new table storing the attributes we want to index on all of our database shards. For example, a typical entity in FriendFeed might look like this:

{
    "id": "71f0c4d2291844cca2df6f486e96e37c",
    "user_id": "f48b0440ca0c4f66991c4d5f6a078eaf",
    "feed_id": "f48b0440ca0c4f66991c4d5f6a078eaf",
    "title": "We just launched a new backend system for FriendFeed!",
    "link": "http://friendfeed.com/e/71f0c4d2-2918-44cc-a2df-6f486e96e37c",
    "published": 1235697046,
    "updated": 1235697046,
}

We want to index the user_id attribute of these entities so we can render a page of all the entities a given user has posted. Our index table looks like this:

CREATE TABLE index_user_id (
    user_id BINARY(16) NOT NULL,
    entity_id BINARY(16) NOT NULL UNIQUE,
    PRIMARY KEY (user_id, entity_id)
) ENGINE=InnoDB;

Our datastore automatically maintains indexes on your behalf, so to start an instance of our datastore that stores entities like the structure above with the given indexes, you would write (in Python):

user_id_index = friendfeed.datastore.Index(
    table="index_user_id", properties=["user_id"], shard_on="user_id")
datastore = friendfeed.datastore.DataStore(
    mysql_shards=["127.0.0.1:3306", "127.0.0.1:3307"],
    indexes=[user_id_index])

new_entity = {
    "id": binascii.a2b_hex("71f0c4d2291844cca2df6f486e96e37c"),
    "user_id": binascii.a2b_hex("f48b0440ca0c4f66991c4d5f6a078eaf"),
    "feed_id": binascii.a2b_hex("f48b0440ca0c4f66991c4d5f6a078eaf"),
    "title": u"We just launched a new backend system for FriendFeed!",
    "link": u"http://friendfeed.com/e/71f0c4d2-2918-44cc-a2df-6f486e96e37c",
    "published": 1235697046,
    "updated": 1235697046,
}
datastore.put(new_entity)
entity = datastore.get(binascii.a2b_hex("71f0c4d2291844cca2df6f486e96e37c"))
entity = user_id_index.get_all(datastore, user_id=binascii.a2b_hex("f48b0440ca0c4f66991c4d5f6a078eaf"))

The Index class above looks for the user_id property in all entities and automatically maintains the index in the index_user_id table. Since our database is sharded, theshard_on argument is used to determine which shard the index gets stored on (in this case, entity["user_id"] % num_shards).

You can query an index using the index instance (see user_id_index.get_allabove). The datastore code does the "join" between the index_user_id table and theentities table in Python, by first querying the index_user_id tables on all database shards to get a list of entity IDs and then fetching those entity IDs from the entitiestable.

To add a new index, e.g., on the link property, we would create a new table:

CREATE TABLE index_link (
    link VARCHAR(735) NOT NULL,
    entity_id BINARY(16) NOT NULL UNIQUE,
    PRIMARY KEY (link, entity_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

We would change our datastore initialization code to include this new index:

user_id_index = friendfeed.datastore.Index(
    table="index_user_id", properties=["user_id"], shard_on="user_id")
link_index = friendfeed.datastore.Index(
    table="index_link", properties=["link"], shard_on="link")
datastore = friendfeed.datastore.DataStore(
    mysql_shards=["127.0.0.1:3306", "127.0.0.1:3307"],
    indexes=[user_id_index, link_index])

And we could populate the index asynchronously (even while serving live traffic) with:

./rundatastorecleaner.py --index=index_link

CONSISTENCY AND ATOMICITY

Since our database is sharded, and indexes for an entity can be stored on different shards than the entities themselves, consistency is an issue. What if the process crashes before it has written to all the index tables?

Building a transaction protocol was appealing to the most ambitious of FriendFeed engineers, but we wanted to keep the system as simple as possible. We decided to loosen constraints such that:

The property bag stored in the main entities table is canonical
Indexes may not reflect the actual entity values

Consequently, we write a new entity to the database with the following steps:

Write the entity to the entities table, using the ACID properties of InnoDB
Write the indexes to all of the index tables on all of the shards

When we read from the index tables, we know they may not be accurate (i.e., they may reflect old property values if writing has not finished step 2). To ensure we don't return invalid entities based on the constraints above, we use the index tables to determine which entities to read, but we re-apply the query filters on the entities themselves rather than trusting the integrity of the indexes:

Read the entity_id from all of the index tables based on the query
Read the entities from the entities table from the given entity IDs
Filter (in Python) all of the entities that do not match the query conditions based on the actual property values

To ensure that indexes are not missing perpetually and inconsistencies are eventually fixed, the "Cleaner" process I mentioned above runs continously over the entities table, writing missing indexes and cleaning up old and invalid indexes. It cleans recently updated entities first, so inconsistencies in the indexes get fixed fairly quickly (within a couple of seconds) in practice.

PERFORMANCE

We have optimized our primary indexes quite a bit in this new system, and we are quite pleased with the results. Here is a graph of FriendFeed page view latency for the past month (we launched the new backend a couple of days ago, as you can tell by the dramatic drop):

In particular, the latency of our system is now remarkably stable, even during peak mid-day hours. Here is a graph of FriendFeed page view latency for the past 24 hours:

Compare this to one week ago:

The system has been really easy to work with so far. We have already changed the indexes a couple of times since we deployed the system, and we have started converting some of our biggest MySQL tables to use this new scheme so we can change their structure more liberally going forward.

Getting a ubuntu clone in VirtualBox to connect to the network

noreply@blogger.com (MathBee) — Tue, 09 Dec 2014 10:18:00 +0000

In order to clone a VM on Virtual Box, you first need to save the machine state of a given VM, or completely power off the VM. You cannot clone an actively running VM.

Once a VM is powered off or saved, you can clone the VM by following this guide.

Go to "Machine" -> "Clone" on VirtualBox GUI menu. It will open up VM clone wizard as follows. Type in a new machine name.

On the next screen, you will get to choose "Clone type". If you want to create an independent copy of your VM, choose "Full clone".

If there is any snapshot of an original VM, you can choose between copying the current machine state only, or copying the current machine state, plus all available snapshots.

Finally, click on "Clone button" to proceed with cloning. Once cloning is done, you can power on the cloned VM.

Solve networking problems of cloned Linux VM

When cloning a Linux VM, additional steps may be necessary to make the networking of its cloned VM work.

For example, for Ubuntu/Debian VM, you may notice that there is no active network interface, upon powering on the cloned VM. The problem is due to the fact that the Linux system of the cloned VM stores the original VM's MAC address(es) for its network interface(s) in its udev configuration file, even though VirtualBox has generated new MAC address(es) for the cloned VM.

Since Linux 2.6 kernel, udev device manager system has been introduced to handle devices including networking devices, hard drives, USB devices, etc. The udev of the original VM assigns device name "eth0" to the VM's MAC address, and this (eth0, MAC address) mapping rule is stored in /etc/udev/rules.d/*-persistent-net.rules.

The cloned VM will use the same udev rule to attempt to assign device names. However, since VirtualBox has generated a different MAC address for the cloned VM, it cannot assign "eth0" to the cloned VM's primary interface, and ends up assigning a different name (e.g., "eth1") to the interface. Of course, because there is no network configuration for eth1, networking cannot work.

$ sudo vi /etc/udev/rules.d/*-persistent-net.rules

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.0 (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:d9:c8:cc", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.0 (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:2d:49:3e", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

In this example, the first line is a copy from the original VM, and the second line was appended by the cloned VM. "08:00:27:d9:c8:cc" is the MAC address assigned to the original VM's interface, which is named "eth0", while "08:00:27:2d:49:3e" is the MAC address assigned to the cloned VM's interface, which is assigned "eth1".

To fix this problem, remove the first line, and replace "eth1" with "eth0" in the second line. Thus, a correct udev rule for the cloned VM looks like:

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.0 (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:2d:49:3e", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

Finally, reboot the cloned VM.

One more thing: make sure to chance hostname of the cloned VM in /etc/hostname and /etc/hosts, since the cloned VM has the same hostname of the original VM.

Source: http://xmodulo.com/how-to-clone-or-copy-virtual-machine-on-virtualbox.html

Kickstarting Flask on Ubuntu - Setup and Deployment

noreply@blogger.com (MathBee) — Tue, 09 Dec 2014 03:53:00 +0000

This tutorial details how to setup a Flask application on a server running Ubuntu.

Since this process can be difficult, as there are a number of moving pieces, we’ll look at this in multiple parts, starting with the most basic configuration and working our way up:

Part 1: Setting up the basic configuration
Part 2: Adding Supervisor
Part 3: Simplifying deployment with Git Hooks
Part 4: Automating with Fabric (with an example video!)

Updated (10/28/2014): Added info on how to add a new user on Ubuntu

We’ll specifically be using:

Ubuntu 14.04
nginx 1.4.6
gunicorn 19.1.1
Python 2.7.8
Pip 1.5.4
virtualenv 1.11.4
Flask 0.10.1
Supervisor 3.0b2

Assuming you already have a VPS running an Ubuntu operating system, we need to set up a web server on top of the operating system to serve static files – like stylesheets, JavaScript files, and images – to end users. We’ll use nginx as our web server. Since a web server cannot communicate directly with Flask (err Python), we’ll use gunicorn to act as a medium between the server and Python/Flask.

Wait, why do we need two servers? Think if Gunicorn as the application web server that will be running behind nginx – the front facing web server. Gunicorn is WSGI compatible. It can talk to other applications that support WSGI, like Flask or Django.

Need access to a web server? Check out Digital Ocean, Linode, or Amazon EC2. Alternatively, you can use Vagrant to emulate a Linux environment. This setup was tested on both Digital Ocean and Linode.

The end goal: HTTP requests are routed from the web server to Flask, which Flask handles appropriately, and the responses are then sent right back to the web server and, finally, back to the end user. Properly implementing a Web Server Gateway Interface (WSGI) will be paramount to our success.

Let’s get to it.

Part 1 – Setup

Let’s get the basic configuration setup.

Add a new User

After SSH’ing into the server as the ‘root’ user, run-

$ adduser newuser
$ adduser newuser sudo

-to create a new user with ‘sudo’ privileges.

Install the Requirements

SSH into the server with the new user, and then install the following packages:

$ sudo apt-get update
$ sudo apt-get install -y python python-pip python-virtualenv nginx gunicorn

Set up Flask

Start by creating a new directory, “/home/www”, to store the project:

$ sudo mkdir /home/www && cd /home/www

Then create and activate a virtualenv:

$ sudo virtualenv env
$ source env/bin/activate

Install the requirements:

$ sudo pip install Flask==0.10.1

Now set up your project:

$ sudo mkdir flask_project && cd flask_project
$ sudo vim app.py

Add the following code to app.py:

from flask import Flask, jsonify

app = Flask(__name__)


@app.route('/')
def index():
    return 'Flask is running!'


@app.route('/data')
def names():
    data = {"names": ["John", "Jacob", "Julie", "Jennifer"]}
    return jsonify(data)


if __name__ == '__main__':
    app.run()

Within VIM, press “i” to enter the INSERT mode. Add the code, then press “escape” to leave INSERT mode to go into COMMAND mode. Finally type “:wq” to save and exit VIM.

Set up a static directory-

$ sudo mkdir static

-and then add an index.html (sudo vim static/index.html) file with the following html:

<h1>Test!</h1>

Configure nginx

Start nginx:

$ sudo /etc/init.d/nginx start

Then:

$ sudo rm /etc/nginx/sites-enabled/default
$ sudo touch /etc/nginx/sites-available/flask_project
$ sudo ln -s /etc/nginx/sites-available/flask_project /etc/nginx/sites-enabled/flask_project

Here, we remove the default nginx configuration, create a new config file (called flask_project), and, finally, set up a symlink to the config file we just created so that nginx loads it on startup.

Now, let’s add the config settings to flask_project:

$ sudo vim /etc/nginx/sites-enabled/flask_project

Add:

server {
    location / {
        proxy_pass http://127.0.0.1:8000;
    }
    location /static {
        alias  /home/www/flask_project/static/;
    }
}

So, HTTP requests that hit the / endpoint will be ‘reverse proxied’ to port 8000 on 127.0.0.1(or the “loopback ip” or “localhost”). This is same IP and port that gunicorn will use.

We also indicate that we want nginx to directly serve the static files from the “/home/www/flask_project/static/” directory rather than routing the requests through gunicorn/WSGI. This will speed up our site’s load time since nginx knows to serve that directory directly.

Restart nginx:

$ sudo /etc/init.d/nginx restart

Profit!

$ cd /home/www/flask_project/
$ gunicorn app:app -b localhost:8000

The latter command manually runs gunicorn on localhost port 8000.

Open your browser and navigate to http://your_domain_name_or_ip_address.

Again, you should see the “Flask is running!” message. Test out the other URL, /data, as well. If you navigate to http://your_domain_name_or_ip_address/static, you should see “Test!”, indicating that we’re serving static files correctly.

Part 2 – Supervisor

So, we have a working Flask app; however, there’s one problem: We have to manually (re)start gunicorn each time we make changes to our app. We can automate this with Supervisor.

Configure Supervisor

SSH into your server, and then install Supervisor:

$ sudo apt-get install -y supervisor

Now create a configuration file:

$ sudo vim /etc/supervisor/conf.d/flask_project.conf

Add:

[program:flask_project]
command = gunicorn app:app -b localhost:8000
directory = /home/www/flask_project
user = newuser

Profit!

Stop gunicorn:

$ sudo pkill gunicorn

Start gunicorn with supervisor:

$ sudo supervisorctl reread
$ sudo supervisorctl update
$ sudo supervisorctl start flask_project

Make sure your app is still running at http://your_domain_name_or_ip_address. Check out the Supervisor documentation for custom configuration info.

Part 3 – Deployment

In this final part, we’ll look at using a post-receive Git Hook along with Git, of course, to simplify the deployment process.

Configure Git

Again, SSH into the remote server. And then install Git:

$ sudo apt-get install -y git

Now run the following commands to set up a bare Git repo that we can push to:

$ sudo mkdir /home/git && cd /home/git
$ sudo mkdir flask_project.git && cd flask_project.git
$ sudo git init --bare

Quick tip: displaying the git branch on your prompt will help remind you of where you are at while trekking through the terminal.
Consider adding this to your bash profile on production:
parse_git_branch() {
    git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* $.*$/ (\1)/'
}

export PS1="\u@\h \W\[\033[32m\]\$(parse_git_branch)\[\033[00m\] $ "

Configure the Post-Receive Hook

$ sudo vim hooks/post-receive

Add:

#!/bin/sh
GIT_WORK_TREE=/home/www/flask_project git checkout -f

Now on every push, the new files will copy over to the app directory, “/home/www/flask_project”.

Then make the file executable:

$ sudo chmod +x hooks/post-receive

Profit!

Back in your local Flask directory (“flask_project”) add a new Git repo along with the following remote:

$ git init
$ git remote add production root@<your_ip_or_domain>:/home/git/flask_project.git

Make sure to update the IP or domain name.

Make some changes to your code in the app.py file:

@app.route('/data')
def names():
    data = {
        "first_names": ["John", "Jacob", "Julie", "Jennifer"],
        "last_names": ["Connor", "Johnson", "Cloud", "Ray"]
    }
    return jsonify(data)

Commit your local changes, then push:

$ git add -A
$ git commit -am "initial"
$ git push production master

SSH into your server and restart gunicorn via Supervisor:

$ sudo supervisorctl restart flask_project

Check out your changes at http://your_domain_name_or_ip_address/data.

Part 4 – Automating

Do you really want to manually configure a server? Sure, it’s great for learning, but it’s super tedious, as you can tell. Fortunately, we’ve automated the process with Fabric. Along with setting up nginx, gunicorn, Supervisor, and Git, the script creates a basic Flask app, which is specific to the project that we’ve been working with. You can easily customize this to meet your own specific needs.

You should remove the username and password from the file and either place them in a separate config file that stays out of version control or set up SSH keys on the remote server so that you do not need a password to login. Also, be sure to update the env.hosts variable to your IP or domain name.

Setup

To test this script (fabfile.py) clone the repo and start with a clean, freshly provisioned server with Ubuntu 14.04. Then navigate to the “flask-deploy” directory. To set up the basic configuration on the remote server as well as your app, run the following command:

$ fab create

Your app should now be live. Test this out in your browser.

Deployment

Want to set up deployment with Git Hooks? Initialize a local repo within your Flask project directory (if necessary). Then make some local changes to your Flask app, and run the following command to deploy:

$ fab deploy

Check your app again in the browser. Make sure your changes show up.

Status Check

Finally, you can check that the Supervisor process is running correctly, to ensure that your app is live, with the following command:

$ fab status

Again, this script is specific to your project at hand. You can customize this to your own needs by updating the config section and altering the tasks as necessary.

Rollback

To err is human…

Things are bound to go wrong from time to time once you have code on production. Everything may work well on your local development environment only to crash in production. So it’s important to have a strategy in place to quickly revert a Git commit. Take a quick look at therollback task from the fabfile.py, which allows you to revert changes quickly to get a working app back up.

Test it out by purposely breaking your code and then deploying to production, and then run:

$ fab rollback

You can then update your code locally to fix the bugs, and then re-deploy.

Example Video

Just press play…

Conclusion and Next Steps

Want to take this workflow to the next level?

Add a pre-production (staging) server into the mix for testing purposes. Deploy to this server first, which should be an exact copy of your production environment, to test before deploying to production.
Utilize Continuous Integration and Delivery to further eliminate bugs and regressions with automated testing and decrease the time it takes to deploy your app.

Source: https://realpython.com/blog/python/kickstarting-flask-on-ubuntu-setup-and-deployment/

Using nginx as HTTP load balancer

noreply@blogger.com (MathBee) — Tue, 09 Dec 2014 03:51:00 +0000

Introduction

Load balancing across multiple application instances is a commonly used technique for optimizing resource utilization, maximizing throughput, reducing latency, and ensuring fault-tolerant configurations.

It is possible to use nginx as a very efficient HTTP load balancer to distribute traffic to several application servers and to improve performance, scalability and reliability of web applications with nginx.

Load balancing methods

The following load balancing mechanisms (or methods) are supported in nginx:

round-robin — requests to the application servers are distributed in a round-robin fashion,
least-connected — next request is assigned to the server with the least number of active connections,
ip-hash — a hash-function is used to determine what server should be selected for the next request (based on the client’s IP address).

Default load balancing configuration

The simplest configuration for load balancing with nginx may look like the following:

http {
    upstream myapp1 {
        server srv1.example.com;
        server srv2.example.com;
        server srv3.example.com;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://myapp1;
        }
    }
}

In the example above, there are 3 instances of the same application running on srv1-srv3. When the load balancing method is not specifically configured, it defaults to round-robin. All requests are proxied to the server group myapp1, and nginx applies HTTP load balancing to distribute the requests.

Reverse proxy implementation in nginx includes load balancing for HTTP, HTTPS, FastCGI, uwsgi, SCGI, and memcached.

To configure load balancing for HTTPS instead of HTTP, just use “https” as the protocol.

When setting up load balancing for FastCGI, uwsgi, SCGI, or memcached, use fastcgi_pass, uwsgi_pass,scgi_pass, and memcached_pass directives respectively.

Least connected load balancing

Another load balancing discipline is least-connected. Least-connected allows controlling the load on application instances more fairly in a situation when some of the requests take longer to complete.

With the least-connected load balancing, nginx will try not to overload a busy application server with excessive requests, distributing the new requests to a less busy server instead.

Least-connected load balancing in nginx is activated when the least_conn directive is used as part of the server group configuration:

    upstream myapp1 {
        least_conn;
        server srv1.example.com;
        server srv2.example.com;
        server srv3.example.com;
    }

Session persistence

Please note that with round-robin or least-connected load balancing, each subsequent client’s request can be potentially distributed to a different server. There is no guarantee that the same client will be always directed to the same server.

If there is the need to tie a client to a particular application server — in other words, make the client’s session “sticky” or “persistent” in terms of always trying to select a particular server — the ip-hash load balancing mechanism can be used.

With ip-hash, the client’s IP address is used as a hashing key to determine what server in a server group should be selected for the client’s requests. This method ensures that the requests from the same client will always be directed to the same server except when this server is unavailable.

To configure ip-hash load balancing, just add the ip_hash directive to the server (upstream) group configuration:

upstream myapp1 {
    ip_hash;
    server srv1.example.com;
    server srv2.example.com;
    server srv3.example.com;
}

Weighted load balancing

It is also possible to influence nginx load balancing algorithms even further by using server weights.

In the examples above, the server weights are not configured which means that all specified servers are treated as equally qualified for a particular load balancing method.

With the round-robin in particular it also means a more or less equal distribution of requests across the servers — provided there are enough requests, and when the requests are processed in a uniform manner and completed fast enough.

When the weight parameter is specified for a server, the weight is accounted as part of the load balancing decision.

    upstream myapp1 {
        server srv1.example.com weight=3;
        server srv2.example.com;
        server srv3.example.com;
    }

With this configuration, every 5 new requests will be distributed across the application instances as the following: 3 requests will be directed to srv1, one request will go to srv2, and another one — to srv3.

It is similarly possible to use weights with the least-connected and ip-hash load balancing in the recent versions of nginx.

Health checks

Reverse proxy implementation in nginx includes in-band (or passive) server health checks. If the response from a particular server fails with an error, nginx will mark this server as failed, and will try to avoid selecting this server for subsequent inbound requests for a while.

The max_fails directive sets the number of consecutive unsuccessful attempts to communicate with the server that should happen during fail_timeout. By default, max_fails is set to 1. When it is set to 0, health checks are disabled for this server. The fail_timeout parameter also defines how long the server will be marked as failed. After fail_timeout interval following the server failure, nginx will start to gracefully probe the server with the live client’s requests. If the probes have been successful, the server is marked as a live one.

Load Balancing and Reverse Proxying with Nginx

noreply@blogger.com (MathBee) — Tue, 09 Dec 2014 03:49:00 +0000

Nginx is a modern, open-source, high-performance web server. It is capable of handling a huge number of concurrent connections easily (see the C10K problem). Over a year ago, I wrote about using nginx as a load balancer and remote proxy. Since then, my understanding of nginx and best practices in its configuration have progressed significantly. I’ve decided to refresh my blog post to provide some of this additional knowledge.

As I explained in my previous post, nginx relies on a non-blocking I/O, event-driven model, which allows it to easily handle a large number of incoming concurrent client connections with ease. This makes it an excellent choice as aload balancer and reverse proxy. In contrast, the traditional Apache HTTP model relies on a limited number of synchronous threads which may block on I/O.

Nginx running on a single server handles incoming client requests and distributes them to a pool of upstream application servers that actually fulfill the requests. The pool of application servers can be easily scaled up or down to handle changes in traffic levels. This flexibility provides a way to scale the capacity of almost any web application quite easily.

Following are some specific scenarios and nginx configuration examples that I have used when setting up and maintaining applications and network infrastructures for both Atomic Object and our clients. I lead up to a fairly practical configuration implementation that I’ve used recently.

Simple Scenario

Situation: We have a JRuby application running on Apache Tomcat. The application gets a significant amount of traffic and is no longer performing as well as required. We have hit the limit of how much additional RAM/CPU resources we can easily add to the server. Instead, we decide to scale horizontally and add more servers.

First, we setup a new nginx server to a accept the connections, and distribute them to our Tomcat application servers.

upstream jruby_application  {
  server 10.0.0.2:8080 max_fails=1 fail_timeout=10s;
  server 10.0.0.3:8080 max_fails=1 fail_timeout=10s;
  # and so on: server 10.0.0.x;
}
 
server {
  listen 80;
 
  root /var/www/html;
  try_files /maintenance.html $uri $uri/index.html $uri.html @proxy;
 
  location @proxy {
    proxy_pass  http://jruby_application;
  }
}

Explanation: We set nginx to listen on port 80. When a request comes in, nginx first checks to see if the /var/www/html/maintenance.html file exists. If it does, it serves it. Otherwise, it continues down the list of checks so see if the request can be handled locally. If the request cannot be served locally, nginx passes it to the@proxy location block, which will proxy the request to the upstream JRuby applications running on port 8080.

This is extremely powerful as it allows nginx to intercept certain requests before they are proxied to upstream applications. This is useful for static assets such as images and files. Additionally, the check for a maintenance page allows us to selectively put the entire site in maintenance mode simply by creating a file.

Scenario with HTTP Headers

Situation: Our new cluster of servers has been running great. However, we notice that the IP addresses of clients aren’t being properly passed to the JRuby application. Instead, all clients have an IP address of 10.0.0.1.

Fortunately, all that we need to do is to tell nginx to set or pass on certain HTTP headers to the upstream JRuby application.

server {
  listen 80;
 
  root /var/www/html;
  try_files /maintenance.html $uri $uri/index.html $uri.html @proxy;
 
  location @proxy {
 
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
    proxy_pass  http://jruby_application;
  }
}

Explanation: By adding some specific HTTP headers via the proxy_set_headerdirective, we can ensure that the JRuby application has access to the correct HTTP headers from the client. In addition, we can set certain headers based on characteristics of the client connection. In this instance, X-Real-IP is set to the remote address of the client. However, the client itself may just be a proxy server forwarding a request on behalf of a client (common with certain CDN’s).X-Forwarded-For is set to the ultimate client IP address (end user) based on the HTTP headers provided with the request.

Scenario with SSL/HTTPS

Situation: Our JRuby application now needs to be able to secure client requests. It would be a pain to setup SSL on Apache Tomcat, so we decide to use nginx as the SSL endpoint. In our current setup, all connections to upstream JRuby application servers are on a private network, and so do not need to be separately secured.

server {
  listen 443;
  server_name jrubyapp.example.com;
 
  root /var/www/html;
  try_files /maintenance.html $uri $uri/index.html $uri.html @proxy;
 
  ssl on;
  ssl_certificate cert.crt;   
  ssl_certificate_key cert.key; 
 
  location @proxy {
 
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
 
    proxy_pass  http://jruby_application;
  }
}

Explanation: We set nginx to listen on port 443 (HTTPS), specify that nginx should enable the SSL engine, and use the provided SSL certificate and SSL certificate key. Nginx does not have a separate directive for providing a SSL chain certificate (such as with Apache HTTP), so any chain certificates need to be appended to the primary certificate (e.g. cat newcert.crt chain.crt > cert.crt). Additionally, we set the X-Forwarded-Proto HTTP header so that the JRuby application can detect that the client connected securely over HTTPS.

Scenario Redirecting all Traffic to HTTPS

Situation: We decide that all traffic to our JRuby application should be secured with SSL. This tends to be for non-public-facing websites such as particular client application instances. While it would be better that the connections were originally made with SSL, we can provide a redirect as a convenience to users where appropriate.

server {
  listen 80;
 
  root /var/www/html;
  try_files /maintenance.html $uri $uri/index.html $uri.html @secure;
 
  location @secure {
    rewrite ^ https://jrubyapp.example.com permanent;
  }
 
}
server {
  listen 443;
  server_name jrubyapp.example.com;
 
  root /var/www/html;
  try_files /maintenance.html $uri $uri/index.html $uri.html @proxy;
 
  ssl on;
  ssl_certificate cert.crt;   
  ssl_certificate_key cert.key; 
 
  location @proxy {
 
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
 
    proxy_pass  http://jruby_application;
  }
}

Explanation: We set nginx to listen on both port 80 (HTTP) and port 443 (HTTPS). On port 80, nginx first checks for the /var/www/html/maintenance.html page, and then checks if it can otherwise handle the request locally. If it cannot, it returns a redirect to HTTPS. The permanent keyword tells nginx to respond with a HTTP 301(moved permanently) as opposed to a HTTP 302 (moved temporarily). Note that the server block for port 80 has no means of actually proxying the request.

Scenario with Proxy Redirect

Situation: Our upstream JRuby application does not always respond to clients with the appropriate URI. For instance, the JRuby application occasionally responds with a redirect, but specifies a protocol of HTTP instead of HTTPS as the JRuby application itself is running as HTTP (nginx is handling the SSL termination). We can intercept any of these responses and ensure that the client is redirected properly.

server {
  listen 443;
  server_name jrubyapp.example.com;
 
  root /var/www/html;
  try_files /maintenance.html $uri $uri/index.html $uri.html @proxy;
 
  ssl on;
  ssl_certificate cert.crt;   
  ssl_certificate_key cert.key; 
 
  location @proxy {
 
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
 
    proxy_pass  http://jruby_application;
    proxy_redirect http://jrubyapp.example.com https://jrubyapp.example.com;
  }
}

Explanation: The proxy_redirect directive tells nginx to look for any instances ofhttp://jrubyapp.example.com in the HTTP headers of responses from the upstream JRuby application. If such an instance is found, it is dynamically rewritten tohttps://jrubyapp.example.com before the response is sent to the client. In this way, we ensure that any redirects or callbacks generated by the JRuby application are properly constructed before responding to a client.

Failure to handle cases such as these can result in nasty redirect loops or cause clients to attempt to connect with HTTP to HTTPS ports, or vice versa. Failing to setup proper proxy_redirect directives when they are needed is one of the most common instances of ‘under-configuring’ nginx.

The Final Configuration

Situation: After several iterations, we finally have a nginx configuration to load balance and reverse proxy for our JRuby application. Our nginx server can handle serving assets locally, display a maintenance pages, proxy to upstreams, forward HTTP headers, and handle both HTTP and HTTPS requests.

upstream jruby_application  {
  server 10.0.0.2:8080 max_fails=1 fail_timeout=10s;
  server 10.0.0.3:8080 max_fails=1 fail_timeout=10s;
  server 10.0.0.4:8080 max_fails=1 fail_timeout=10s;
  # and so on: server 10.0.0.x;
}
server {
  listen 80;
 
  access_log  /var/log/nginx/jrubyapp.example.com.access.log;
  error_log  /var/log/nginx/jrubyapp.example.com.error.log;
 
 
  root /var/www/html;
  try_files /maintenance.html $uri $uri/index.html $uri.html @secure;
 
  location @secure {
    rewrite ^ https://jrubyapp.example.com permanent;
  }
 
}
server {
  listen 443;
  server_name jrubyapp.example.com someotherurl.com;
 
  access_log  /var/log/nginx/jrubyapp.example.com.ssl.access.log;
  error_log  /var/log/nginx/jrubyapp.example.com.ssl.error.log;
 
  root /var/www/html;
  try_files /maintenance.html $uri $uri/index.html $uri.html @proxy;
 
  if ($host != 'jrubyapp.example.com') {
    rewrite ^/(.*)$ https://jrubyapp.example.com/$1 permanent;
  }
 
  ssl on;
  ssl_certificate cert.crt;
  ssl_certificate_key cert.key;
 
  location @proxy {
 
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
 
    proxy_pass  http://jruby_application;
    proxy_redirect http://jrubyapp.example.com https://jrubyapp.example.com;
  }
 
   client_max_body_size 64M;
}

Explanation: The above configuration is nearly identical to the one that has been built up through each of the scenarios. Notably, we’ve added aclient_max_body_size directive to allow clients to upload large files through the proxy, access_log and error_log directives so that we can properly log access requests and aerrors, and a check to be sure that a client is using the correct ‘canonical url’ when accessing the application.

Additional Resources

The nginx wiki is great, and the following pages can be helpful references for the above configuration:
The Reactor Pattern: The design pattern that nginx uses for event handling concurrently.
Moving from Apache to nginx: Some syntax hints, and processing differences.
Why running nginx in front of Apache is a good idea: Considerations for splitting the handling of requests between nginx and Apache.

Source: http://spin.atomicobject.com/2013/07/08/nginx-load-balancing-reverse-proxy-updated/

Eclipse jsp the function fn:replace undefined

noreply@blogger.com (MathBee) — Fri, 23 May 2014 04:11:00 +0000

I am using maven2 and Eclipse Kepler. And when I run "maven clean", the code compiles successfully and shows the message "BUILD SUCCESS". But I get the red squiggly underlines under fn:replace call in one .jsp file.

I checked the pom.xml file, and found it include the reference to JSTL as below:
<dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>jstl</artifactId>
            <version>1.2</version>
            <scope>runtime</scope>
</dependency>

After Google the Internet, I found the solution, which is to include the following line in the .jsp file
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %>

and run clean project, then the error is fixed.

Creation of Amazon EC2 Key Pairs

noreply@blogger.com (MathBee) — Tue, 20 May 2014 07:26:00 +0000

You can use both Amazon EC2 console and command line "ec2-create-keypair" to create a Key pair.

The details is in Amazon AWS web site: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html

Amazon EC2 command ec2-run-instances error client.invalidamiid.notfount

noreply@blogger.com (MathBee) — Tue, 20 May 2014 07:21:00 +0000

I was using the command-line ec2-run-instances to create an EC2 instance. However I kept getting the following error message:

Client.InvalidAMIID.NotFound: The image id '[ami-xxxxxxxx]'

does not exist Error starting (256):

The solution is that I need specify the region using the option --region --region ap-southeast-1



One example is

Design Patterns for Large-Scale Real-Time Learning

noreply@blogger.com (MathBee) — Thu, 15 May 2014 10:26:00 +0000

A presentation slides from Sean Owen

http://qconlondon.com/dl/qcon-london-2014/slides/SeanOwen_DesignPatternsForLargeScaleRealTimeLearning.pdf

Lambda Architecture

noreply@blogger.com (MathBee) — Thu, 15 May 2014 10:17:00 +0000

The Lambda Architecture is proposed by Nathan Marz, which is a generic, scalable and fault-tolerant data processing architecture.

There are some useful resources online:
1. The first Lambda Architecture post wrote by Nathan Marz
http://www.databasetube.com/database/big-data-lambda-architecture/

2. An simple introduction.
http://lambda-architecture.net/

3. Lambda Architecture: A state of the art
http://www.datasalt.com/2014/01/lambda-architecture-a-state-of-the-art/

DLMS/COSEM

noreply@blogger.com (MathBee) — Mon, 28 Apr 2014 09:43:00 +0000

DLMS or Device Language Message Specification and COSEM or Companion Specification for Energy Meeting is a specification that specifies an interface model and communication protocols for data exchange with metering equipment. And DLMS/COSEM has been adopted by the IEC TC13 WG14 into the IEC 62056 series of standards.
COSEM specifies the interface model, which provides a view of the functionality of the meter as it is available at its interface(s). The interface consists of attributes and methods, where the attributes values represent all the data in meters and the methods describe the allowable operations to be performed on the attributes.
DLMS is an application layer protocol that defines how the data can be accessed and transported.
The DLMS/COSEM specification is document in 4 “books”.

The Blue book “COSEM – Identification System and Interface Classes” specifies the interface model
The Green book “DLMS/COSEM – Architecture and Protocols” specifies the architecture and the protocols with DLMS on top, for the various media specific communication profiles. The communication media includes RS-232, RS-485, PSTN, GSM, GPRS, IPv4 PPP and PLC.
The Yellow book specifies conformance test plans for COSEM object interface model.
The White book is a glossary of terms.

Two good web sites for preparing programmer’s interview

noreply@blogger.com (MathBee) — Fri, 25 Apr 2014 09:42:00 +0000

1. Leetcode: http://discuss.leetcode.com/
2. GeeksforGeeks: http://www.geeksforgeeks.org/

Edit The Opencart Footer Area

noreply@blogger.com (MathBee) — Wed, 23 Apr 2014 10:05:00 +0000

Firstly, in order to remove the “Powered by Opencart” text from the OpenCart Footer, you need open the file at: catalog/view/theme/[Your Theme]/template/common/footer.tpl.

And then you simply delete all of the text between the <! and the -> symbols to remove the comments and OpenCart Powered By text at the bottom of the pages.

Secondly, we want to add an image about Paypal payment at the bottom of the OpenCart shop web page. In the same footer.tpl, at the same location of the deleted OpenCart Powered By text, add the following code:
<div align=”center”><img src=”http://vivoharmony.com/image/paypal.jpg” style=”margin: 15px 15px”></div>

Lastly, we also want to insert the copyright text at the bottom of the OpenCart shop web page. Thus, after the Paypal image, we add the following code:
<div align=”center”><p>Copyright &#169 2014 Vivoharmony. All Rights Reserved. </p></div>

The &#169 is to display the copyright sign “© ” in html.

Life Editor

Ask HN: How do you log application events?

FIFA 2014 World Cup live stream architecture

The numbers

The core overview

The input

The storage

The output

The player

The sauron

The bonus round

DIY Device Cloud

How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS

Introduction

Install the Components

Install Celery

Install RabbitMQ

Create a Celery Instance

Build Celery Tasks

Start Celery Worker Processes

Use the Queue to Handle Work

Conclusion

Creating a browser based EmonHub log console using python, flask, socketio, MQTT

Install

Changes to emonhub

Web Scraping With Scrapy and MongoDB

Installation

Scrapy

PyMongo

Scrapy Project

Specify Data

Create the Spider

XPath Selectors

Extract the Data

Test

Store the Data in MongoDB

Pipeline Management

Test

Conclusion

Web Scraping - It’s Your Civic Duty

Introduction

The Data

The Tools

Start the Process

Understand Your HTML

Parse the HTML

Convert the Data

Plot The Data

Final Thoughts

An Architecture for Real-Time Geo-tracking with Python, Celery, RabbitMQ, and More

Flask performance optimization

General

Server Setup

Frontend Reverse Proxy

Caching static pages with Nginx itself

UWSGI and Gevent

Flask PyPy

Flask-Cache

Memcached/Redis

Jinja2 template bytecode caching

Consider using ElasticSearch

Code Profiling and Optimization

Conclusion

What is the technology stack behind Pinterest?

Quora’s Technology Examined

Components Of Quora

What’s Cooking Under That Hood?

The Search-Box

Speedy Queries

Webnode2 And LiveNode

Amazon Web Services

Ubuntu Linux

Static Content

HAProxy Load-Balancing

Nginx

Pylons And Paste

Python

Thrift

Tornado

Long Polling (Comet)