The two hardest data types to model are e-commerce and genealogy. There really just isn’t any good way to do it in a relational database. MongoDB is the answer Genealogy developers have been looking for for a very long time. A flexible schema that allows adhoc searching, horizontal scale and full data integrity. The following presentation covers MongoDB fundamentals, how to build a graph in MongoDB, how model genealogical data.

The principles explained apply equally well in RDBMSs, MongoDB and other databases.

Potential scenarios to protect against

1. Drive failure

2. Machine failure

3. Switch failure

4. Power circuit failure

5. Data center failure

6. Intrusion

7. Fat fingers

8. Programmer error

Raid

To prevent drive failure use multiple drives in a single machine for high availability. RAID 10 provides the best performance with high availability. RAID 10 consists of a minimum of 4 disks which are split into mirrored pairs. The raid controller stripes across the pairs.

Replication

Replication is the act of mirroring the data from one server onto another. It will protect against any failure of one (or more) of machines. Commonly two or more machines are used. If properly configured it can also protect against most of the failures identified above. I will present a few different configurations and what they protect against.

Replica Sets

In addition to standard Master Slave replication MongoDB provides an additional replication configuration called replica sets. Replica sets are similar to master slave, but they are set aware with automatic failover and recovery. Replica sets are a minimum of 3 nodes. 3 nodes can be 3 database nodes, or 2 database nodes + 1 arbiter. The arbiter doesn’t handle queries or store data, but is around to provide a third perspective to cast a vote when determining the status of the set. In master slave you can have the same setup but it requires you to participate as the arbiter.

Configuration 1 :

2 nodes + 1 arbiter (or 3 nodes) on same rack (same switch and power circuit).

• Protects against single drive failure (if not using RAID 0 or 10).
• Protects against single node failure.

Configuration 2 :

2 nodes + 1 arbiter on 3 different racks (different switch and power circuit).

• Protects against the above + failure of switch or power circuit.

Configuration 3 :

4 nodes + 1 arbiter on 3 different racks (different switch and power circuit).

• Protects against the above + two nodes failing or a node failure at the same time as a switch or power failure.

Configuration 4 :

4 nodes + 1 arbiter in 3 different data centers.

• Protects against all the above + data center failure.

Configuration 5 :

Any combination of the above but swap out the arbiter for a delayed secondary with priority 0 (hidden).

Some Protection against data loss in the event of Intrusion, Fat finger (sysadmin / DBA accidentally deleting / changing something), programmer error.

Only partial protection as any data written during the delay window will be lost. If not caught during the window (or windows) provides no protection.

Backups

A backup consists of a dump of the data ideally stored in a remote secure location. It’s important to ensure that the backup format used has an easy and consistent import mechanism. Any scripts you use to backup should have a counterpart restore written along with them. The worst possible time to discover issues with your backup is when you are trying to restore them. Make sure restoration is rock solid.

For permanent disaster recovery and for compliance with a variety of industry regulations it’s important to keep (offsite) backups. Depending on your data size, business and type of data you may want to take backups more or less regularly. A solid backup plan is the only way to ensure that the bulk of your data is never lost in the event of a disaster. It is often not enough to have replication be your sole form of backup as it doesn’t protect against intrusion, fat finger (sysadmin / DBA accidentally deleting / changing something) or programmer error outside of the delay window and not at all without a delayed replicant.

Conclusion

It’s important to understand the pitfalls and how to prevent them. Also recognize that there’s a balance that must be struck. Multiple data centers add additional complexity and bring a number of challenges along with their additional protection.

Lastly, Replication != backup. Please backup your data. If someone accidentally or intentionally maliciously performs a drop operation that same operation will be replicated across all of your machines. The only safety net is a good backup strategy.

The full recording can be found at 10gen’s site at http://www.10gen.com/events/hybrid-applications

 Other Resources:

• http://www.10gen.com/presentations/mongosv-2011/mongodb-at-customink-adoption-operations-and-community
• http://jwage.com/2010/08/25/blending-the-doctrine-orm-and-mongodb-odm/
• http://www.10gen.com/presentations/mongosf2011/craigslist
• http://spf13.com/post/ecommerce-in-la
• http://spf13.com/post/mongodb-sort-conference-2011
• http://spf13.com/post/mongodb-e-commerce-and-transactions

 EaseUS® Partition Master 9.1 Home Edition

As Partition Magic alternative, EaseUS Partition Master Home Edition is a ALL-IN-ONE partition solution and disk management freeware. It allows you to extend partition (especially for system drive), manage disk space easily, settle low disk space problem on MBR and GUID partition table (GPT) disk under Windows 2000/XP/Vista/Windows 7 (SP1 included) 32 bit and 64 bit system.

 

Process Hacker

Effectively a better version of Task Manager, Process Hacker is a free and open source process viewer and memory editor with unique features such as powerful process termination. It can show services, processes and their threads, modules, handles and memory regions.

TeraCopy

TeraCopy is designed to copy and move files at the maximum possible speed. The program skips bad files during the copying process, and then displays them at the end of the transfer so that you can see which ones need attention. TeraCopy can automatically check the copied files for errors by calculating their CRC checksum values. It also provides a lot more information about the files being copied than its Windows counterpart. TeraCopy integrates with Windows Explorer’s right-click menu and can be set as the default copy handler. It replaces Explores copy and move functionality and instead provides excellent move and copy support.

 

WinDirStat

WinDirStat is a disk usage statistics viewer and cleanup tool for Microsoft Windows. It works well and is very fast.

Write Monkey

Write Monkey is a knock off of the Write Room full screen distraction free applications for OS X. It’s quite well done and free.

Putty & Putty Connection Manager

PuTTY is a free implementation of Telnet and SSH for Windows and Unix platforms, along with an 

xterm

 terminal emulator. Putty Connection Manager can be used as tabbed version of Putty. It works on all the editions of Windows OS i.e Windows XP, Server 2003,Vista, Server 2008 and Windows 7.

 

I really enjoyed my time in LA and am looking forward to returning. I was impressed by how many companies are using MongoDB in LA and how great the interaction was at the Meetup.

An introduction to using MongoDB with PHP.

Walking through the basics of schema design, connecting to a DB, performing CRUD operations and queries in PHP.
MongoDB runs great in the cloud, but there are some things you should know. In this session we’ll explore scaling and performance characteristics of running Mongo in the cloud as well as best practices for running on platforms like Amazon EC2.

It was a really fun event to attend. I was overwhelmed with the interest and excitement around MongoDB. The MongoDB talks were always standing room only with over 70 people in the first session and over 100 people in the second. The LDS church has some very interesting data challenges including the worlds largest genealogy database with billions of individuals and relationships and many more times that in documents and records. There was a lot of interest in MongoDB and it was nice to see the excitement around it.

Since the conference I’ve gotten lots of positive feedback from participants via twitter and email.

Here’s how I approached this when I was at OpenSky.

1. Databases on Metal

For certain operations operating directly on servers in your own data centers still makes sense. IO heavy operations such as databases continue to see considerably better performance benefits from operating directly on the hardware. Additionally these machines benefit from specifically tuned hard drives and controllers built with higher IO in mind. For all the right reasons, the virtualized and commoditized cloud can’t and won’t compete here, it’s just not cost effective for them to do so.

2. Vital in house

I’ve been a cloud customer far too long to depend on it’s reliability. Cloud servers can and will fail. It’s been my experience that this happens at a much higher rate than traditional servers. When uptime is the most essential, a traditional approach will serve you better.

3. Appendages in the cloud

I’ve always been a big believer in using the best tool for the job. Use the cloud for what it’s built to do. Not everything is vital. There are many supporting pieces of your infrastructure where perfect uptime isn’t critical. What the appendages are will depend entirely on your business. At OpenSky we currently operate our blog and marketing servers on EC2. We leverage S3 for archival backups. We utilize email delivery servers on the cloud. This is a small set of what we will eventually have there, but it provides a good insight into our approach.

4. Scale in the cloud

By operating a core selection of servers in house it enables us to scale up our web nodes in the cloud. Since our ecommerce application isn’t particularly database heavy (thanks in large part to mongoDB) our scalability bottleneck is on our web servers. Keeping a core set of them in house to handle things like checkout and administrative operations permits us to scale the bulk of our traffic to the cloud.

Currently HP and Rackspace are two companies that are providing turnkey hybrid cloud computing offerings.

Related articles

• Competition for the Cloud Heats Up (spf13.com)
• CIOs not fretting about cloud security (CIO Enterprise Forum)
• HP Cloud Strategy, 3 pillars to address varying needs (CIO Enterprise Forum)

I gave a pretty significatly revised presentation on MongoDB and E-Commerce with emphasis on transactions. If you saw the presentation at MongoNYC this summer, it’s worth a revisit.

This has been a big week for 10gen. We’ve just made the headlines with two big stories: First we closed a new round of financing and second, we’ve release our 2.0 version of MongoDB.  Both are huge milestones for an amazing company and I’m thrilled to be a part of it.

A part of the new funding will go towards expanding… which of course means we (and I) are hiring.

My role at 10gen is to be the voice of our developers. I’m assembling a team of developers, hackers, evangelists, and sysops (dev ops) focused on building applications with MongoDB. Our job is to ensure that developers have the best experience working with MongoDB.

This includes

• Building and maintaining drivers and tools
• Supporting our clients like Craigstlist, SAP and foursquare.
• Blogging
• Giving presentations
• Working with ODMs and other 3rd party libraries
• Working with open source projects
• Writing automation scripts for things like backups.
• Being the voice of a language for 10gen.
  

Sound interesting?

@ me on twitter

oh, and go and …

Over the next two years I led OpenSky to become the first Ecommerce site built on MongoDB and one of the first PHP sites to use MongoDB. Along the way I gave a few presentations and we created the open source library Doctrine MongoDB ODM, an extension of the Doctrine project.

Fast forward to the present and Mongo is changing the world. Their install base is more than all of the other NoSQL solutions combined. Disney, Craigslist, Foursquare, Shutterfly, Etsy, SourceForge, NY Times, IGN, Disqus, Chartbeat, EA, Sailthru, SongKick, Topsy, Turntable, Squarespace, OpenSky and hundreds of other companies have chosen MongoDB as their database Solution. The future looks very bright for MongoDB.

Why all this talk about MongoDB? I have recently joined 10gen (the MongoDB company) as Chief Solutions Architect. In this role I will lead the public side of the engineering organization including integration, evangelism, support and consulting. I will be bringing to this role my experience as an engineer, architect and executive building innovative products and companies since 1995.

I look forward to the new challenges, opportunities and friends this new role will provide and hope to help advance the adoption, ability and usability of MongoDB

Perhaps you’ll recognize these words, “About five years ago I started to notice an odd thing. The products that the database vendors were building had less and less to do with what the customers wanted. … So, what is this growing disconnect?” Those words were written in 2004 by Adam Bosworth, a veteren of Microsoft, Google and BEA. In the 7 years since things have only gotten worse. Open source products came to maturity (if you can call it that), but none improved on any of the challenges Bosworth outlines. He points out 3 things that everyone wants in a database, but nobody is providing.. well nobody except MongoDB.

1. Dynamic Schema

He writes:

Dynamic schema so that as the business model/description of goods or services changes and evolves, this evolution can be handled seamlessly in a system running 24 by 7, 365 days a year. This means that Amazon can track new things about new goods without changing the running system. It means that Federal Express can add Federal Express Ground seamlessly to their running tracking system and so on. In short, the database should handle unlimited change.

Mongo with it’s document design handles this elegantly and gracefully. We’ve made loads of changes to our database without even needing to worry about migrations. The responsibility to maintain changes falls on the application and isn’t a constrained by the database.

2. Dynamic Partitioning

He Writes:

Dynamic partitioning of data across large dynamic numbers of machines. … The only issue is that it needs to be dynamic so that as items are added or get “busy” the system dynamically load balances their data across the machines. In short, the database should handle unlimited scale with very low latency. It can do this because the vast majority of queries will be local to a product or a customer or something over which you can partion.

Mongo has solved this as well with it’s automatic sharding. It runs pretty much exactly as he requests. Mongo automatically distributes data among shards and balances them dynamically.

3. Modern Indexing

He Writes:

Google has spoiled the world. Everyone has learned that just typing in a few words should show the relevant results in a couple of hundred milliseconds.

He is a bit vague on what he means here, so taken one way, everyone has solved this, but then again if taken that way, it was solved long before 2004 when this was written. I’m going to take it to mean real indexing across highly distributed data stores. So you can find a set of data across many nodes all extremely quickly. I’d like to add this to automatic indexing. There is no reason not to do this using a similar approach to how mongo tackled sharding. Again as he ins’t clear on what he is asking for it’s not clear if MongoDB has solved it or not. I think with their map reduce framework and the ability to create combined indexes on rich structured documents I think they are heading in the right direction, but I don’t think they’ve quite solved this yet.

Related articles

• Craigslist Adopting MongoDB (readwriteweb.com)
• MongoDB Competes on Speed and Flexibility (pcworld.com)
• Ten things I didn’t know about MongoDB (slowping.com)
• Blending Mongo and RDBMS for ecommerce (spf13.com)