The Build Mine

Data Loss

2009-01-18T10:40:00.000-08:00

The machine that I had most of my work on has failed and I don't know if I have a good backup now. I'm trying to find what I can but most of the links off this blog probably won't work anymore. The work that Okoloth is doing is much more advanced than I've done so I highly suggest you check his site out for the state of the art on armory mining.

More on Armory Data Standardization: Data modelling a character

2008-06-10T15:54:00.000-07:00

I'm continuing to work on this. One of the big challenges is doing this in the face of the impending release of WotLK, level 80 and revamping of talent trees. Ideally, not only do I need to model a character in space (all of a character's elements) but I need to model it in time as well. All the analysis I did last year will be useless because talent trees will be different. Because of this requirement, it's a lot harder than "just grab the XML and shove it in a database".

I gotta think about this some more, but I want to be able to track characters in time, kind of like the Internet Wayback Machine.

Started a WoW Blog

2007-10-25T18:24:00.000-07:00

I started a blog just to jot down random stuff about my experiences playing WoW. It's really hard to work on data collection when it keeps you from playing the game at all. (I don't have a lot of spare time)

Big Red Pet

The Inevitable Hiatus Post.

2007-10-24T14:44:00.001-07:00

If it wasn't obvious by now, I haven't spent a lot of time working on this lately. I'm firmly committed to discussing the finer points of builds (and point people once again to the largely empty forum I've set up for this type of discussion) so hopefully I can start up some discussion.

If you're particularly proud of your build, I highly encourage you to sign up on the forums and write up a post in your class forum about it. If we can get enough people to do this, discussion will surely follow.

New Build Shop: Priest 14/0/47

2007-09-19T14:01:00.001-07:00

The Build Shop over at WoW Insider is experimenting with a new format, now it's interview-style. Their first target is Skosiris from the oh so awesome Wowhead. Check it out.

I'm out of state visiting relatives so this is quick. I'm currently rewriting everything for speed.

Armory to Data Miners - It's ok!

2007-08-27T16:47:00.000-07:00

I finally got around to looking at the recent batch of armory updates, and noticed this little gem:

XML file size reduction: The XML files used by the Armory have been optimized for speed, partly by streamlining the amount of code in the files. As a result, third-party sites that mine Armory data may need to make adjustments to account for the new file configurations.

There you go, implicit permission from Blizzard to suck down data.

Report - 20070814232234

2007-08-17T16:52:00.001-07:00

Eventually there will be 7 reports. The base report consists of all level 70s broken down by class, with the top 20 specs for each class. The remaining 6 reports are further broken down by the following metrics: battlegroup, faction, gender, race, realm, and realm type. There is also an index page as a jumping off point to all reports. Unfortunately at this time battlegroup and realm breakdowns aren't working properly. I need to design custom XSL templates for them as they don't mesh with the other reports as easily. Once those templates are created I'll link to them here. Enjoy!

Report - Spec Breakdown by Realm Type

2007-08-16T00:02:00.001-07:00

Ok, I've got all the characters from my first run loaded with updated armory data. This time I store the raw XML, (and much props to Okoloth for a nice method to merge all armory data into a single XML file) then when I want to run a report at a point in time, I export the information I'm interested in to a database for quick searching. After writing insane scripts to generate HTML from the database last time, I've learned my lesson. Now the data is exported in XML and I'm using XSL to transform the data into tables. The first report I've generated is a breakdown of each class' top 20 specs, by realm type.
For each class you'll see the top 20 specs for RP-PvP, PvE, PvP and RP servers. Unfortunately, in order to make sure the output shows up in all browsers properly, I'm not serving raw XML with an XSL stylesheet directly, but using PHP to load the XML and XSL and output the resulting HTML. The raw XML is available here. You will note that the XML also contains the top 10 builds for each spec. I didn't use that data in the HMTL report because I wanted to keep the tables somewhat simple, and by using XSL I can tweak the stylesheet to include that data once I figure out a good way to represent it. (plus I can make it prettier much more easily, if I ever learn how)

Taking a quick look at the data for Hunters, 41/20/0 dominates both PvE and PvP realms, but interestingly enough, a larger percentage of Hunters with that spec are on PvP servers. (7920 vs. 5938, or 7.861% vs. 5.894%) Also, while the first Survival spec on PvP servers is only ranked #9 (0/21/40), the first PvE Survival spec is at #7. Though to be fair, the percentage of the class is higher in the PvP spec. In fact, it's pretty clear that out of the Hunters represented in this table there are more PvP Hunters than PvE.

Please feel free to talk about this report either here or on our forums.

New Build Shop up - Warlock ??/41+/??

2007-08-15T11:43:00.000-07:00

WoW Insider has a new Build Shop up, this time focusing on a Demonology spec. This one's a bit open-ended, since the requirement is just 41+ points in demonology. I've updated the existing 70s I have now and fetched their (relatively) current character sheets. Scanning the top 20 Warlock specs on my current report, #5 is 7/43/11, and #6 is 20/41/0. Coming in at #16 is 18/43/0, #18 is 5/45/11, and #20 is 9/41/11.

Obviously, I've got a new report out, but I'm redesiging the report generation to be more general and flexible. The goal here is to support live queries against that database on various criteria. I know for a fact that making this live will kill my server, so I need to figure out a way to ensure that doesn't happen. In any case, you should eventually be able to do top N spec/builds per class, with these additional variables: faction, race, gender, realm, realm type and battlegroup. I think a lot of people will be interested in the realm type breakdown and if there truly is a difference between how PvE and PvP players choose their talents.

765k * 5 XML files == 7.5 gigs of data

2007-08-07T22:37:00.000-07:00

I'm almost a third of the way through reimporting all the characters from the first run into the new db. At the moment I'm only sucking down and storing the raw xml from armory and haven't done any processing. While that's going on I'm desigining a new database table structure with the concept of tracking characters over time in mind.

In the meantime, does anyone have any pointers to good tutorials on XSL? Outputting raw html is kind of a pain so being able to do XSL transformations on XML output would probably make things a bit simpler.

2nd Fetch update - 416k and counting

2007-08-03T13:26:00.000-07:00

After 3 days or so, I've fetched new data on 416,000 of the 2.5 million characters I have so I should have new data by the end of august. I need to figure out the best way to multithread this process so I can grab data faster. As a side note, the XML data for those 416,000 characters is taking up close to 4 gigs of drive space. Coming up with a more efficient storage method is next on the list.

What constitutes being a particular spec versus a hybrid?

2007-08-01T16:14:00.001-07:00

There seems to be a difference of opinion on this one. Prior to TBC, it was generally agreed that you were "specced" to a particular talent tree if you had at least 31 points in that talent which means just over 50% of your talent points are in a single tree. Now that TBC is out and there are an additional 10 points to allocate, some people say that you're a hybrid unless you have at least 41 points in a tree, which is at least 67%. What do you think? Personally I've stuck with 31 points in a single tree as being specced. I'll run some numbers and update this post shortly.

Update: Here's my quick class breakdown. Each Tree will have two numbers, number of characters that have 31 or more points in that tree, and number of characters that have 41 or more points in that tree. The remaining number are hybrids by that definition.

Total number of 70s: 782,247

Druid: 76371
Balance: 11053 (14.47%) , 9370 (12.27%)
Feral Combat: 44857 (58.74%) , 43897 (57.48%)
Restoration: 20421 (26.74%) , 17934 (23.48%)
Hybrids: 40 (0.05%) , 5170 (6.77%)

Hunters: 96671
Beast Mastery: 26328 (27.23%) , 24733 (25.58%)
Marksmanship: 62945 (65.11%) , 59058 (61.09%)
Survival: 7101 (7.35%) , 3419 (3.54%)
Hybrids: 297 (0.31%) , 9461 (9.79%)

Mages:104760
Arcane: 41297 (39.42%) , 12353 (11.80%)
Fire: 33784 (32.25%) , 28360 (27.07%)
Frost: 29506 (28.17%) , 27602 (26.35%)
Hybrids:173 (0.17%) , 36445 (34.79%)

Paladins:69338
Holy: 50493 (72.82%) , 45571 (65.72%)
Protection: 9991 (14.41%) , 9124 (13.05%)
Retribution: 8526 (12.19%) , 7974 (11.40%)
Hybrids: 328 (0.47%) , 6669 (9.62%)

Priests:91094
Discipline: 10577 (11.61%) , 4728 (5.20%)
Holy: 44361 (48.70%) , 9712 (10.66%)
Shadow: 35781 (39.28%) , 35030 (38.45%)
Hybrids: 375 (0.41%) , 41624 (45.69%)

Rogues:94760
Assassination: 24234 (25.57%) , 19420 (20.49%)
Combat: 47267 (49.88%) , 40052 (42.27%)
Subtlety: 21193 (22.36%) , 13901 (14.67%)
Hybrids: 2066 (2.18%) , 21567 (22.76%)

Shamans:51279
Elemental: 13797 (26.90%) , 6326 (12.34%)
Enhancement: 16685 (32.54%) , 12951 (25.26%)
Restoration: 20743 (40.45%) , 20274 (39.54%)
Hybrids: 54 (0.11%) , 11728 (22.87%)

Warlock:85389
Affliction: 39612 (46.39%) , 32815 (38.43%)
Demonology: 27128 (31.77%) , 21758 (25.48%)
Destruction: 18074 (21.17%) , 13926 (16.31%)
Hybrids: 575 (0.67%) , 16890 (19.78%)

Warrior:112765
Arms: 41118 (36.46%) , 15151 (13.44%)
Fury: 16159 (14.33%) , 13553 (12.02%)
Protection: 55178 (48.93%) , 53906 (47.80%)
Hybrids: 310 (0.27%) , 30155 (26.74%)

Armory Miner Gets Props!

2007-08-01T16:05:00.000-07:00

Okoloth just got a mention on WoW Insider! It's good that people are realizing the power of the Armory.

Standardizing Armory data storage

2007-08-01T11:15:00.000-07:00

There are a number of projects out there to fetch and operate on Armory data. However, there doesn't seem to be much interaction between these projects. The simplest way to cache Armory data is to just fetch the 5 XML files for each character and store them as blobs in a table. However, that's not really conducive to doing complex queries and searching it doesn't scale. I'm working on a database representation of character data (arena teams is on the table next) and I'm wondering if anyone else out there is interested in working on this. The goal is to create a freely available database schema that any Armory-related project could take advantage of for reasonably quick offline searching. If you're interested, drop me a line at thebuildmine@gmail.com. Once the schema's done, then work can begin on a fetch and management library for various scripting languages. (perl, php, python, ruby, etc.)

Round 2 gathering has started

2007-08-01T11:11:00.001-07:00

After going through the first grab of 2.5M characters, I've gone back and done some refinements on data gathering and optimization. I'm currently acquiring the latest data on all the characters I've gotten before, sorted by level descending, since what we're most interested in in the short term is data on level 70s. This time I'm fetching all of the armory data and storing it instead of just the character sheet for more complex queries and comparisons. Eventually I was get things to a point where live searches can be done.

Forums up!

2007-07-27T17:01:00.000-07:00

I'm happy to announce that I've put up forums for discussion on the results discovered from datamining the Armory.

Racial Report

2007-07-27T11:36:00.000-07:00

I respun the report and added the racial breakdowns. I don't have class numbers by race offhand, but I can run those this weekend. The original reports are there too, but they're the same as before since it's all scripts running against the same data. Over the next couple of weeks I'll be overhauling my schema and collection tools now that I've gone through this exercise once.

First Report

2007-07-26T00:10:00.000-07:00

Ok, here's a link to the first "full" report. (I am fully aware that my web design fu is negative. Hopefully CSS will save me someday.) This structure of report should mostly stay the same (I hope). For each class I've generated 4 spec/build reports, one specialized for each talent tree (31 points or more in a given tree is considered specialized) and overall. Each report lists the top 10 specs, and the top 10 builds within each spec. There's way too much information for me to come close to discussing this, so if you find something you want to discuss, blog about it and let me know and I'll mention your blog here.

Shockadins on WoW Insider

2007-07-25T11:03:00.000-07:00

Last night WoW Insider posted an article about a Shockadin build for 40/0/21, and is catching a lot of flak for it, saying the author didn't do enough research, etc. Research? That's what we're here for. Let's take a look-see! (Remember this data is not up-to-the-second accurate. This is data obtained a couple of months ago.)

Quick numbers for non-clickers: 1412/69338 level 70 paladins share this spec. That's barely over 2%.

Since I haven't even touched my pally since the BC talent trees came out, I can't comment much.

The top build is shared by 20 people. In the Holy tree, the top build ditches 2/2 Improved Lay on Hands and 3/3 Sanctified Light in favor of 5/5 Improved Seal of Righteousness. Not having ISoR seems to be a common complaint in the article's comments, so that's consistent. As for the Retribution tree, the top build puts 3/3 in Improved Seal of the Crusader, 5/5 Deflection (instead of 3/5), and 3/3 Crusade (instead of 2/3), ditching Conviction and Seal of Command.

The first negative comment seems to be close to the voice of the "majority" of the spec. However, the third build does forgo ISoR....

So? Is there a valid criticism for WoWI not doing their homework? In the end, these types of situations are exactly what I'm trying to address.

Build Results Layout - How do you want to see it?

2007-07-17T17:06:00.000-07:00

So after a number of problems over the past month or so, I've got almost 2.5 million characters in the database. Live queries are out of the question, because my server just doesn't have the oomph for it. So for each class, I'm going to run the top 10 overall specs, and the top 10 builds of those specs, but I'm also going to run the top 10 specs (and top 10 builds within those specs) from each tree, to show a little more diversity specwise. This will take a while, but I'll try to have stuff up before the end of the month.

New Build Shop up - Priest 23/38/0

2007-07-17T17:03:00.000-07:00

WoW Insider's Build Shop has a new entry: A Priest PvE healing build. I've got a 60 priest waiting in the wings but I suspect he'll be taking advantage of this build after hitting the cap.

Just under 2.5M characters and Europe info

2007-06-22T11:29:00.001-07:00

The armory has been acting wonky for the past few days, so I've suspended data acquisition for now. I've got just under 2,500,000 characters in place, and that's just enUS. I'm in the process of tweaking things to support the EU armory so I should get even more data from there. I'm also redesigining the schema somewhat to improve performance. While I'd like to put up web pages with live queries, I think what will actually happen is daily or weekly runs with result pages. This weekend I'll start running comparisons between the 1M checkpoint and where the data is now.

New Build Shop up - Hunter 5/13/43

2007-06-13T10:16:00.000-07:00

WoWInsider's Build Shop has a new build up, conveniently enough a hunter build. Let's take a look.

Well, I'm up to almost 2 million characters, and guess what: no one's got this build. In fact, only 45 characters even have this *spec*. Needless to say, none of them share a build. This leads me to believe this build was submitted either as a theory or a joke.

So what have we got? Part of me questions putting points in BM at all if you're going for an uber-CC build, since why do you need to worry about attack speed if your enemies are always under control? I would think those 5 points would be better off in either Mortal Strikes (otherwise why did they bither putting 5 points in Lethal Shots?) or horror of horrors, Imp Stings to pump up Wyvern Sting.

And since we're getting into the MM tree, why no points in Efficiency? Mana's a concern, since you're going to be tossing lots of spells around, so I would think Efficiency would be a priority. Go For the Throat is fine, I guess, but again, given the implied intent of the build, we don't care about damage, we don't care about pets, so putting those points in Imp Feign Death seems like it might be better (I know, Imp FD is bogus, I'm just saying) for dealing with loss of control when 4+ enemies are involved.

Tier 1 in SV is an exercise in trying to get to higher tiers. Sure, two points in Savage Strikes, but the other 3 points can go wherever. I'm guessing Humanoid Slaying was chosen for the same reason Eliah guessed, for PvP. For PvE I'd probably pick Hawkeye too. :P Tier 2 and Tier 3 are more of the same, Entrapment, Imp Wing Clip and Clever Traps are necessary. 5 points in Survivalist is tier fodder and extra health can't hurt. Trap Mastery is key to the intent, Survival Instincts and Killer Instinct are tier fodder but are the best of what's available. Resourcefulness is key for long fights. Lightning Reflexes gets you to Wyvern Sting, so you might as well grab Expose Weakness for more damage and to get you up to Master Tactician and Readiness. I would think that filling out Thrill of the Hunt would be useful (again, the whole mana thing) but overall I feel this is a build to mess around with and not a particularly serious one.

1st Million - Warriors

2007-06-11T17:20:00.000-07:00

44,771 level 70 warriors in the first million. The largest represented class. Not hunters, not mages... warriors:

4 Arms specs, 1 Fury spec, and 5 Protection specs. Being a tank class, this breakdown is expected.

Arms - 16708 (37.3%)
Fury - 6466 (14.4%)
Protection - 21472 (47.9%)
Nospec'd - 125 (0.2%)
Unspec'd - 7 (0.01%)

1st Million - Warlocks

2007-06-11T16:42:00.000-07:00

Warlocks! There's 34,001 of them in the first million. Here we go:

7 Affliction specs, 2 Demonology specs, and 1 lone Destruction spec. Ok, let me see if I can say this right: So far, of all the classes, the Warlock top spec has the lowest percentage of any other class. This can be a good or bad thing. Either the warlock talents are so good, characters can spec a number of different ways and still be very effective, or warlocks are nubs and don't understand their own class and are top notch at PvP by accident. I'll go with the latter just to piss off Inowar. I note with amusement that his spec is not in the top 10.

Anyhoo, here's the spec breakdown:

Affliction - 15384 (45.2%)
Demonology - 10887 (32.0%)
Destruction - 7501 (22.06%)
Unspec'd - 229 (0.6%)
Nospec'd - 4 (0.01%)

Update - Yeah, that second spec was a typo, it's 20/41/0.