The post Sysadmin Sunday 127 appeared first on Server Density Blog.

The post Sysadmin Sunday 126 appeared first on Server Density Blog.

Initially, these were installed locally on a developer’s machine and set up via Apache vhosts running each component separately.

This soon became unmaintainable on a daily basis without a lot of work. We were spending too much time discussing whether a certain component was up to date and fighting bugs caused by API incompatibilities between versions. As we added more services to deal with things beyond the core of the product, this just got worse.

The answer was Vagrant.

What is Vagrant?

Essentially, vagrant is a command line tool for managing virtualbox instances (other backends are now available in the latest version). You use a pre built box (or package your own) to create a fresh virtual machine with all your tools installed, accessible via SSH from the host machine.

Once a vagrant box is configured, it’s just a case of vagrant up, waiting a while and then a system is up and running.

Our vagrant box

We went through some iterations and experimentation to find something suitable for the way we wanted to work, combined with what we could actually achieve in the box. This is the end result for now, and I’ll list some modifications that we have planned but haven’t found the time to do.

The box is split in two, a base and an environment box:

The Base Box

The first stage in the vagrant build was to build a customised base box. We based it on Ubuntu 12.04, 64-bit build as that’s what we planned to use in production. Once that was chosen, I collected a set of dependencies and development tools that were required for each service (MongoDB, Apache, Node.js, vim, screen, etc). These were then deployed into the base box using a fairly standard puppet manifest with some available modules. At this stage, it installs just the tooling, dependencies and makes required system level config changes (networking setup, DNS entries).

Once this box is built, it’s uploaded to a development webserver so all the team have access to it. The Vagrantfile and the puppet manifests live in git, alongside the development box. This means that the base box can be recreated/tweaked/reviewed by anyone at any time if that’s a necessity.

Using a base box like this loses some flexibility. Every time you want to add something new at the base level, you have to rebuild and re upload the entire box. But it saves deployment time in the next stage, which overall results in a win.

You can temporarily work around this by adding a dependency into the environment box, but if you’re not strict with how you manage this it becomes a bit of a mess of where everything is, and you lose the advantage of having a pre-built base box.

The development environment

This is a little less straightforward than the base box build.

Starting with a Vagrantfile with the base box imported/declared, we added puppet modules to handle installing our agent into the box so it can report to itself, and a module to install apache vhosts.

Once that was done, we created a puppet module that can handle installing all of the services from git. This includes a clone/update, build process (buildout/composer), and finally run any test or development data scripts that we have. This was mostly a mash of already available bits and abuse of the exec puppet command.

The Vagrantfile is just a file of Ruby code, so it was easy to add something that set the box hostname based on the username of the host, and import a separate settings file for overriding the defaults for the vagrant settings (code checkout locations being the main one).

The final stage for this was to add a script that will kill and then start all the code that runs as a service (tornado/celery mainly). This grew out of an ugly hack involving starting lots of things in screen, and hasn’t really been updated to anything else. It does have a convenient advantage that `screen -list` will tell you exactly what is running, and a total number at the bottom for quick verification that everything started okay.

The code is checked out into a shared directory between the host and the guest, ultimately living on the host. This uses nfs for performance which means we can edit the code using the host editors and tools but the code will still run inside the vagrant box.

Debugging is taken care of by xdebug being configured to point to the host IP for the PHP services, and some work with WingIDE remote debugging for the python services, again with Wing configured to connect into the vagrant box.

Once the box was up and running, we added settings, configs and a custom domain (using vagrant-dns) to enable decent separation and ensure we don’t accidentally hardcode production/development URLs (or at least, that these are easier to catch if it does happen).

The main feature of this box is that the puppet manifests run with each provision. These update and redeploy the code each time, simplifying the update process to a single command, across every service and repository that we have deployed.

The Advantages

• Reproducible environment for everyone involved.
• The puppet manifests mean that just a vagrant provision then waiting is enough to bring everything up to date with the latest master.
• Self contained stack, you can see what is running at any point.
• Closer to production. We mostly develop on OSX, but deploy to Linux, this gives us both.
• Shared URLs for testing. We can pop a URL from our vagrant machines into Hipchat, and other members of the team can use it locally, without having to change it. vagrant-dns is a big win for us there.
• Easy to install. Install virtualbox, vagrant, get the Vagrantfiles, run vagrant up.
• While there’s nothing in the vagrant configurations that couldn’t be as easily done with some scripting for the host machine and remove the need for virtualisation, it’s handy that when it goes wrong a fresh rebuild is just a destroy and up away. Extremely useful for testing system wide settings.

The Disadvantages

• A from-nothing vagrant up takes 25 minutes and downloads about 2Gb (1.2Gb for the base box, the rest for code + dependencies + extras).
• vagrant provision to update to latest can take up to 10 minutes depending on speed of the connection and the host machine, so it’s not that easy to ‘just test’ something. You can update the individual services manually, but then you have the problem that we started with, making sure that everyone has the same code.
• It’s hard work for the host machine. The box we have configured has 2 cores and 2Gb of RAM allocated. On a 4Gb Macbook Air, that can start getting a little close to resource starvation, particularly with an IDE and a debugger running.
• Debugging isn’t as easy as I’d like, setting up the debugger is fairly involved in settings, and you can only really debug one thing at a time. Can be awkward when you’re trying to trace values across multiple services.
• The configuration we have isn’t as close to production as I’d like (no nginx, no caching, no centralised logging) but this is just a matter of spending more time.
• It doesn’t entirely solve ‘it works for me’. It just becomes ‘it works on my vagrant’. Fortunately, instances of that seem to be a lot less common.
• Random virtualbox/vagrant/host problems. We’ve had boxes crash, networks go away and all manner of strange things. At least with the code living on the host machine, we’ve not lost work when that happens.

While it seems that there’s more disadvantages than advantages, overall the reproducibility and simplicity of reducing updating to a single command far outweigh the drawbacks of working in a virtualised environment.

Future plans

Most of the future plans for this revolve around gradually bringing it in line with the production environment without losing the flexibility that we have gained.

• Use the production puppet manifests where possible. Our infrastructure is entirely puppet controlled, so I’d like to increase the reuse where possible.
• Create a version that uses the multi-vm capability of vagrant to simulate a cluster, with each service separately. Could be handy for looking at scaling/communication problems.
• See if we can reduce provision time even further, with possible build optimisations. This may then transfer to our deployment system.
• Move to vagrant 1.2 and test out some other backends.
• Remove the screen based development start script and move to something more production-like. (This is only used in the vagrant box, the live deployments use proper init scripts.)

Overall, the use of vagrant has been a big win for us as a company, and has reduced a lot of the problems we were having. There’s still some work to be done until we’re completely happy with it, but I’d recommend that anyone looking at building this type of project take a serious look to see if it suits them.

The post Many projects with Vagrant and Puppet appeared first on Server Density Blog.

The post Sysadmin Sunday 125 appeared first on Server Density Blog.

Shiny SOA

Server Density v2 was re-designed from the ground up as a distributed set of HTTP services, following much in the same vein as Netflix or Amazon’s Service Orientated Architecture approach. What that means is that each distinctly different moving part of the project was separated into its own service (when it made sense to do so), e.g. “users”, “devices”, “cloud” etc. That means you gain full separation of concerns, the users service doesn’t need to care what the sharding policy for devices is as long as the RESTful HTTP API it provides conforms to a known specification.

That’s all fine and dandy I hear you say, but if the brave new world of SOA were so cut and dry you wouldn’t be writing this blog post! Well true, the issues that arise are very different from those when you develop everything as “one app” and here are some of the harder ones:

• Authentication and sessions – do all services talk to the same session store?
• If we want to do “real time” in the browser, e.g. using WebSockets, does that mean all of our backend services need to be WebSocket aware, or even socket.io aware (if we use that library)?
• Will we need to use rewrites to mitigate cross domain request issues in the browser?
• Does the UI need to be aware of all the possible hosts/ports/paths to services on the backend?
• With multiple backend services possibly providing push updates to the frontend, do we need multiple socket connections from the browser?

So how do we go about tackling these? Step this way, I’ll try not to bore you…

Authentication, authorisation and sessions

As we wanted to avoid duplication of methods across multiple services we decided on building a thin layer service that sits between the two. Any request performed by a user must first go via this thin layer before being routed to the backend. By doing this, we get to perform authentication and authorisation in one central place rather than having every service check if user is currently logged in and if they should be allowed to perform an action.

We created two separate services to handle authentication:

1. The auth app handles performing user account lookups with the backend users service before creating a session in MongoDB and forwarding the user to the main UI. This allows us to both scale auth separately and allows for single sign-on capabilities if we want to go down that road. [1]
2. Our main thin layer service uses the same MongoDB session store to check if a user is authenticated before appending their account details to the request and then acting as a HTTP router to the requested service. Any request that doesn’t also have a corresponding authenticated session is immediately stopped, with an appropriate HTTP error status code. [2]

[1] In order to access the session storage in the same way, the auth app and UI share a core library with session middleware.
[2] Backend services are not externally routed, our Nginx load balancers only allow access via the thin layer, and most services do not even have externally accessible network addressees.

By accepting account details appended to the request query string, an individual service doesn’t have to perform any extra user detail lookups and for the most part just has to use the provided user ID and account ID to restrict database queries. This will change slightly as we introduce more fine grained access controls to some areas of some services, but for the most part this covers our needs.

WebSockets, socket.io and services

One decision that was made early on was that we didn’t want to be performing inefficient polling for updates like we do in v1. As reliable as this is in terms of being proven technology (nothing can be simpler than issuing a GET every minute), there are now several alternative solutions for “[more] real time” updates, under the umbrella of the HTML5 specifications:

• Server Sent Events - a stream of events sent as a special part of the HTTP response body during a request that can be listened for using a JavaScript API.
• WebSockets - a two way persistent socket connection created by issuing an HTTP upgrade header from a regular request that can be opened, listened to and have messages sent over using a JavaScript API.

Take note that both have advantages and disadvantages – while WebSockets provide efficient two way messaging, because they’re effectively a different protocol that runs on separate ports, some users may have issues connecting through firewalls. There are also other networking issues to consider such as whether your load balancer supports them.

Server Sent Events have the upper hand in that they are just an extension of regular ol’ HTTP and can just be used on a longpoll connection. Any messages you might need to send upstream can be sent as extra regular concurrent HTTP requests (the open request and event API is asynchronous like any regularXMLHTTPRequest).

For our needs, WebSockets seemed like the best fit and most concerns over compatibility (both networking and browser support) can be circumvented by using a library such as socket.io or SockJS. They provide a convenient wrapper around various socket or socket-like browser messaging solutions, e.g WebSockets, Flash sockets, HTTP longpolling etc., with any failure in creating a messaging channel failing back to a more reliable method (in most cases longpolling).

We eventually chose socket.io due to it’s stability and well established support across platforms, but what about backend support for sockets? Or even libraries like socket.io? There is fairly good support in our core platform (Python) and even some support penetrating our secondary platform (PHP) for socket.io but we didn’t want to have to support our wrapper library across every service. Instead we decided to keep to standard web technology like WebSockets and instead leave compatibility between the frontend abstraction and backend transport up to the thin layer service.

This made our choice of platform for the thin layer quite natural given the following constraints:

• Very fast with a preferably event based networking API.
• First class socket.io and WebSocket support.
• As an added bonus, a mature HTTP routing/proxying library wouldn’t go amiss.

Node.js met all these requirements perfectly.

N.B. We could have also chosen Python to write it in because we already use both Gevent and Tornado which both have thirdparty socket.io compatibility libraries available and provide fast event based networking capabilities.

Despite this, we chose Node.js because it is the defacto target platform for socket.io. This allows us to keep up with the latest version of the library and API. We also chose to write the thin layer in CoffeeScript because our frontend is written in CS and we have lots of experience from other parts of the system. Since the thin layer is a long running daemon process, we can deal with a few cycles spent at startup to compile the CS to JS.

Routing frontend requests to services

Due to cross domain restrictions (CORS alleviates many of these issues), session cookie and SSL certificate requirements, we require the ability to proxy requests to separate services via a single thin layer address, using the URL path to seamlessly route to a backend address.

As we’re already using Node.js we were able to take advantage of the excellent node-http-proxy library, which pretty much does exactly this. We add a few extra goodies onto the plain routing in http-proxy, such as configurable HTTP headers for cross domain requests and CSRF protection.

Our two main requirements are authentication handling and address mapping, both for HTTP requests and socket connections. To enable this we added a Connect style plugin system that allows addresses to be remapped easily for different purposes beyond the mapping built into http-proxy or for data to be injected into requests before forwarding, like adding account details to query strings after querying MongoDB.

How simple is the plugin API? Just add one or several methods like get, post, put, delete, map or socket to your plugin class and the routing handlers will do the rest.

One of http-proxy‘s features is WebSocket routing, however due to using socket.io this doesn’t get used, instead…

Aggregating multiple backend connections over a single frontend socket

With the socket.io server side handling built into the thin layer we are able to take advantage of having full control over the stack between thin layer and backend service, by terminating the abstracted socket at the Node.js instance and then creating one or more real WebSocket connections to the backend services.

Instead of creating new connections from the frontend to services we just append a different ID to our messages and the thin layer routes the messages to the appropriate service, crating a new WebSocket if it needs to. On receipt of messages from this connection, it appends the same ID to them before sending them back to the frontend.

This socket aggregation also means connections are lazy, only created as needed by the frontend, but once created they remain and will continue to push “real time” updates back to the UI.

Another extension of this method is the ability to push other messages over a single connection. Once you have a connection to a service over a WebSocket you can do anything over that socket you might have done with individual requests; this was part of the reason for choosing sockets. However, we are also able to tell the thin layer to send HTTP requests with a socket message and to tag the response with an ID before sending back as a socket message.

This doesn’t save us much in terms of requests on the backend but this part of our infrastructure is relatively cheap to scale due to various tricks we can employ such as SSL termination. Instead it saves the user’s browser from sending and receiving lots of expensive HTTP requests.

Sockii to the rescue!

Sockii is our internal name for the thin layer we created – it can do everything described above and some more besides, all from a simple JSON config.

We will continuously improve Sockii, and it’s currently sparse documentation, and as of today we’ve released it under a BSD license available for everyone to fork and deploy!

So when you’re using Server Density to see real time updates of your cloud instance build status or the response time of your websites, you know a little about what’s connecting our backend services to your browser client.

The post Introducing Sockii: HTTP and WebSocket aggregator appeared first on Server Density Blog.

The post Sysadmin Sunday 124 appeared first on Server Density Blog.

Every minute our monitoring nodes issue requests to a public URL/IP, usually a website. They question whether whatever you’re monitoring is running, provide you with response time statistics and gather any other useful information. Oh, and if it’s down – we’ll let you know (even if it is 2am).

Just because your website is up in London, it doesn’t necessarily mean that it’ll perform the same to visitors in Australia. So we monitor performance from locations all over the world. This provides you with an overview of how your service is performing – regardless of visitor location.

The Process

To deliver you all of this information we decided to create a director-actor (server-client) infrastructure. The ‘director’ within our data centre, and the ‘actor’ being a machine placed somewhere in a distant location like Sydney (~16,000 km from London). The ‘director’ propagates checks for each of the ‘actors’ locations, which perform the requested check and return the result to the ‘director’.

The Tech

To facilitate the director-actor (server-client) architecture, we chose a message broker software to act as a transport layer between both entities. For this we’ve used the following tool set: Python, Celery, RabbitMQ and Gevent.

We’ve previously created our own basic queuing system using MongoDB but given Celery supports MongoDB as a backend, we started using that. We know how to scale Mongo in production and it has excellent replication/failover capabilities, but we hit some problems which forced us to use RabbitMQ instead.

If you consider HTTP checks for the POST method, it may be that some users send sensitive data, or use some sort of authorisation for the targeted URL (or both). The monitoring nodes are hosted with various vendors around the world and communicate with our central directors over the internet rather than our secure, internal network. This means we need to ensure communication is secure, which either means a VPN or SSL. We didn’t really want to deal with managing a VPN of a large number of nodes around the world so decided to investigate SSL transports instead.

Celery is smart and robust tool which is really easy to extend. But being in the stage of prototyping means that we do things fast and adjust later, this way we decided to go with the RabbitMQ broker with had full support for SSL encryption inside Celery backends, also MongoDB as a backend was still considered an a experimental option. Have a look inside the Celery docs to decide which broker will work best for you.

We do intend to switch to using MongoDB for the Celery backend eventually, and have submitted a patch to add SSL support to Celery. RabbitMQ is working well but it’s another component in our architecture and given our existing experience with MongoDB, we prefer to reduce the number of 3rd party tools where possible. It also has better failover support.

It was better to solve this problem by looking into what we didn’t want to do as opposed to what we did. We knew we didn’t want to do raw socket connection, write our own broker or worry too much about scalability at this stage. This is the solution that we chose as a result. Admittedly, the tech used may well change in the future – but it’s solving the problem well right now, and we’re still experimenting with it.

The Director

The diagram above presents a simplified structure of the current architecture layout for Celery queues. Checks are described by the location on which we are going to execute them. We allow users to create checks from more than one location, so the same URL will often land on different queues at the same time.

The director assembles a check from the information provided by a user. In a URL check for example, that information could be: target URL, check method (GET, POST, PUT, DELETE), payload, timeout, etc. Information in this form will be serialised and allocated to the right location queue.

Server Density currently supports 9 locations with more on the horizon.

The Actor

At the other end of the process (a couple of thousands kilometres away), our actors are working hard to handle the check location queues. They’re designed to take traveling checks from the queue, interpret the message received and perform the requested actions.

We were concerned about how to handle things here. Primarily because we don’t want to evaluate status just by doing one URL check, here’s why:

>>> import timeit
>>> timeit.repeat(
... "requests.get('http://example.com')",
... "import requests",
... number=1,
... repeat=5)

[0.9308030605316162,
2.1385269165039062,
3.0700418949127197,
6.3459131717681885,
1.9396629333496094]

Previously Server Density did one check every minute but we found this had its problems. There are hundreds of things that can go wrong when requests are being sent around the world, alerting users that a website was down for example.

As a result v2 now sends 3 checks each time and records the fastest response – not the average. One check out of the three could timeout for example bringing the entire average down. In this use case people want to see the shortest response time figure. This will be put on the responses queues and send back to the dDirector, which then acts upon the information. We still store the results of all checks and these will be exposed in the UI for the user.

The Scale

Since we are using Celery gevent we can benefit from the fact that one queue can be handled by more than one worker (and vice versa). So what are the implications of this? If we see any scale related problems we have few options:

1. Increase the concurrency level. We are using gevent and can therefore benefit from greenlets.
2. Increase the number of processes. Python has this thing called GIL, right? By deploying another process you can try to bypass this and force your box to do things in parallel.
3. Increase the number of machines. Install more workers with information on which queue(s) to consume.

Conclusion

The most important change is the check methodology – doing 3 checks per location. Further, every location executes the check at the same time so you can see the response time from every location at once, rather than one at a time. Instead of having to do a lot of the queue management functionality ourselves within ZeroMQ, using Celery we can pass that over to the library and dedicate more time to check methodology. It also provides more ways to inspect the queue status and ties into our own monitoring so we know sooner when things are going wrong.

The post Using Celery for queuing requests appeared first on Server Density Blog.

The post Sysadmin Sunday 123 appeared first on Server Density Blog.

For Server Density v2 I have been benchmarking MongoDB with different tweaks so we can get maximum performance for our high throughput clusters, but make cost savings for our less important systems. A lot has been said about various choices of write concern, deploying to SSDs and replication lag but there aren’t really any numbers to base your decision on.

This set of MongoDB benchmarks is not about the absolute numbers but is designed to give you an idea of how each of the different options affects performance. Your own queries will differ but the idea is to prove general assumptions and principles about the relative differences between each of the write options.

Test methodology

These MongoDB benchmarks test various options for configuring and querying MongoDB. I wrote a simple Python script to issue 200 queries and record the execution time for each. It was run with Python 2.7.3 and Pymongo 2.5 against MongoDB 2.4.1 on an Ubuntu Linux 12.04 Intel Xeon-SandyBridge E3-1270-Quadcore 3.4GHz dedicated server with 32GB RAM, Western Digital WD Caviar RE4 500GB spinning disk and Smart XceedIOPS 200GB SSD.

The script was run twice, taking the results from the second execution. This avoids slowdown cause by initially allocating files, collections, etc – MongoDB only creates databases when they’re first written which adds a bit of time to the first call but isn’t really relevant in real world usage.

import time
import pymongo
m = pymongo.MongoClient()

doc = {'a': 1, 'b': 'hat'}

i = 0

while (i < 200):

	start = time.time()
	m.tests.insertTest.insert(doc, manipulate=False, w=1)
	end = time.time()

	executionTime = (end - start) * 1000 # Convert to ms

	print executionTime

	i = i + 1

This is a dummy document because I'm not trying to simulate a real application here. Document size, number/size of indexes and the type of operation will all play a part in the actual numbers. This is only testing inserts but there are other optimisations you can make with updates, particularly ensuring documents don't grow. However, this is sufficient for what I'm trying to show in these tests - the relative difference between the write options.

Write concern

The write concern allows you to trade write performance with knowing the status of the write. If you're doing high throughput logging but aren't concerned about possibly losing some writes (e.g. if the mongod crashes or there is a network error) then you can set the write concern low. Your write calls will return quickly but you won't know if they were successful. The write concern can be dialed up to including error handling (the default) so the write will be acknowledged (not necessarily safe on disk).

It's important to know that an acknowledgement is not the same as a successful write - it simply gives you a receipt that the server accepted the write to process. If you need to know that writes were actually successful one option is to require confirmation the write has hit the journal. This is essentially a safe write to the single node with the option to go further to request acknowledgement from replica slaves. It's much slower to do this but guarantees your data is replicated.

• w=0 is the fastest way to issue writes, with an average execution time of 0.07ms, max of 0.11ms and min of 0.06ms. This setting disables basic acknowledgment of write operations, but returns information about socket excepts and networking errors to the application.
• w=1 takes double the time to return, with an average execution time of 0.13ms, max of 0.32ms and min of 0.11ms. This guarantees that the write has been acknowledged but doesn't guarantee that it has reached disk (the journal), so there is still potential for the write to be lost - there's a 100ms window where the journal might not be flushed to disk. Setting j=1 protects against this.
• j=1 (spinning disk) is several orders of magnitude slower than even w=1, with an average execution time of 34.19ms, max of 34.28ms and min of 34.10ms. The mongod will confirm the write operation only after it has written the operation to the journal. This confirms that the write operation can survive a mongod shutdown and ensures that the write operation is durable.
• j=1 (SSD) is x3 faster than a spinning disk with an average execution time of 11.18ms, max of 11.24ms and min of 11.11ms.
• There is an interesting ramp up for the initial few queries every time the script is run. This is likely to do with connection pooling and opening the initial connection to the database, whereas subsequent queries can use the already open connection.
• Some spikes appear during the script execution. This could be the connection closing and being recreated.

This means that you can reasonably use the default w=1 as a safe starting point but if you need to be sure data has gone to a single node, j=1 is the option you need. And for high throughput you can half query times by going down to w=0.

SSD vs Spinning Disk

It's a safe assumption that SSDs will always be faster than spinning disks, but the question is how much - and is that worth paying for them? The more data you store, the more expensive the SSD will be - higher capacity SSDs are available but they are fairly cost prohibitive. However, MongoDB supports storing databases in directories which can be mounted to their own devices, giving you the option of putting certain databases on SSDs.

Putting your journal on an SSD and then using the j=1 flag is a good optimisation. You need the --directoryperdb config flag and you can then mount the databases on their own disks. The journal is always in its own directory so you can mount it separately without any changes if you wish.

Replication

If you specify a number greater than 1 for the w flag then this will require n number of replica slaves to acknowledge the write before the query completes. I tested this in a x4 node replica set with the primary and a slave in the same data centre (San Jose, USA) as the execution script and the remaining x2 nodes in a different data centre (Washington DC, USA).

The average round trip time between the nodes in the same data centre is 0.864ms and between different data centres is 71.187ms.

• w=2 required acknowledgement from the primary and one of the 3 slaves. Average execution time was 14ms, max of 867ms and min of 1.6ms.
• w=3 required acknowledgement from the primary plus 2 slaves. Average execution time was 310ms, max of 1329ms and min of 96ms. The killer here is the range in response times, which are affected by network latency + congestion, communication overhead between 3 nodes and having to wait for each one.

Using an integer for the w flag lets MongoDB decide which nodes must acknowledge. My replica set has 4 nodes and I specified 2 and 3 but I didn't get to choose which ones were part of the acknowledgement. This could be local slaves but could also be remote, which is probably responsible for the range in response times where a remote slave happened to return faster than the local one. More control is possible using tags.

Conclusion

It's fairly clear that these MongoDB benchmark results validate the general assumptions that SSDs are faster and there is a fairly variable latency involved with replicating over a network, particularly over long distances. What this experiment shows is the differences between the write concern options so you can make the right tradeoff between durability and performance. It also highlights that you can significantly improve performance if you need the journal based durability by adding SSDs.

MongoDB benchmarks raw results

The post MongoDB Benchmarks appeared first on Server Density Blog.

The post Sysadmin Sunday 122 appeared first on Server Density Blog.