tagShadow – MentatJack

Stuff – How I Met Your Mother 2.16

mentatjack — Sat, 17 May 2014 19:20:38 +0000

The 2 story lines in Stuff involve clearing out the detritus of previous relationships and the trials of supporting the creative endeavors of our friends. This post is about the former.

The picture I have of myself and my high school girlfriend was taken on the top of one of the World Trade towers. The picture I have of my college girlfriend was taken at my graduation. She’s wearing sunglasses and standing with J. Scattered around my apartment are pictures of my family, including some from my wedding. I think I’d like to put a wedding picture and the first 2 pictures in a frame together. I hope anyone who loves me in the future will love me in part because of what each of those relationships cultivated in me.

I’ve been thinking about pulling together a website based on the idea that TV should inspire reading. It would basically be a ton of reading recommendations based on television shows and specific episodes. That idea has me thinking more about what I’m watching. I’m not sure what I’d recommend based on this episode of HIMYM.

The season finale of Fringe is another story. Seeing Leonard Nimoy in an office in one of the World Trade towers was probably many a person’s first glimpse at the idea of multiple dimensions. I’ve got the start of a great recommendation list for books and stories about multiple dimensions at TagShadow.

So that’s what I’m watching right now: Fringe and How I Met Your Mother. Also The Shield, but I have no idea what I’d recommend someone read related to Los Angeles gangs and corrupt cops.

I continue to read

mentatjack — Thu, 15 Aug 2013 03:06:21 +0000

What I’ve been doing while not posting here.

I’m 15 books into my 20 book goal for the year. I may just have to raise that goal up a few notches.
I’ve started quite a few reviews for this blog, but I seem to have lost all confidence in my ability to collect my thoughts…
thus I’ve been reading about literary analysis. We’ll see if that helps me gather my thoughts.
I started a tumblr to track the short fiction I read. See my reviews of The Emperor’s Old Bones by Gemma Files and The Best Science Fiction of the Year Three by Ken Macleod
I continue development on the next iteration of TagShadow, although data entry has stalled. I really need to bring some other people onto this project…

The Year’s Best Science Fiction and The Finite Canvas

mentatjack — Thu, 16 May 2013 05:53:39 +0000

I was adding The Year’s Best Science Fiction: Thirtieth Annual Collection to TagShadow and got sidetracked by one of the stories that’s available to read online.

The Finite Canvas by Brit Mandelo is an intense story that revolves around a conversation between two women. They tell each other stories and are drawn to each other by a need they share. It’s way more bloody and awesome than that sounds and I highly recommend it.

Header Image “Illusion”

mentatjack — Thu, 01 Oct 2009 02:07:01 +0000

I’ve updated the header image, with a section of Illusion by Sessarego. Be sure to follow those links for a full image of the painting and other beautiful paintings by the artist. Post frequency has dropped dramatically here, but I’ve made an TON of progress on TagShadow. I decided to use bbPress for my forum and I’m pleased with the results. It would please me greatly for you to sign up (I’ve enabled openId, so that should be painless) and in general spread the word.

TagShadow: 100 tags, 500 books and 5 visualized components.

mentatjack — Wed, 16 Sep 2009 07:23:13 +0000

This iteration has 2 improvements over the last version. The biggest is that I ran this on 100 tags instead of just 10. That in and of itself creates some clearer groupings. The second improvement is even more exciting.

I decided to use the color of the dots to convey some meaning. The first and second principle components are mapped respectively to the x and y axis, just like before. Now I’m also mapping the 3rd, 4th and 5th principle componets to Red, Green and Blue, respectively. Hopefully this makes exploring groupings a bit clearer.

My next feature, since it’s been requested by almost everyone I’ve shown this to, will be a mechanism to know which dots correspond to which tags. I have a pretty good idea how I’m going to go about this. Even when that feature is available I highly encourage users to just explore.

Updates: TagShadow and RIP

mentatjack — Mon, 14 Sep 2009 07:12:02 +0000

I made significant progress with TagShadow over the weekend. Even if it’s not clear exactly what I’m up to, you can see progress of some sort in this series of screenshots. My latest plot has 500 products and is a projection of the data from 10 different tags.

Also I wrote for a few hours on Saturday.

Somehow I found time to edit another one of these:

My favorite part about that one is my bookmark. And I pimp my Noir and Speculative Fiction post.

Pricinipal Component Analysis: The Java Iteration

mentatjack — Sat, 12 Sep 2009 19:31:47 +0000

Much more satisfying than my earlier attempt to do this by hand, I’m actually building the tools that will eventually power TagShadow. I’m step by step implementing wikipedia’s instructions on Computing PCA using the covariance method. I fully understand the folly of trusting wikipedia this closely, but I promise to compare against other sources.
package matrixtest; import Jama.*; /** * * @author MentatJack */ public class PCA { public static Matrix meanVector(Matrix X){ Matrix u = new Matrix(X.getRowDimension(),1); double temp; double x[][] = X.getArray(); for(int i = 0; i < x.length;i++){ temp = 0; for (int j=0; j < x[0].length;j++){ temp += x[i][j]; } u.set(i,0,temp/x.length); } return u; } public static Matrix getCovariance(Matrix M){ Matrix h = new Matrix(1,M.getColumnDimension(),1); Matrix u = meanVector(M); Matrix B = M.minus(u.times(h)); return B.times(B.transpose()); } }

Next I have to sort the eigenvectors…

Tagshadow: Jama, Weka and Scatter Plots

mentatjack — Fri, 11 Sep 2009 16:52:43 +0000

There’s nothing quite like the double edged blade of progress for producing equal parts spectacular failure and insight. I linked to some of my experiments displaying scatter plot book recommendations. Following feedback I’ve iterated over various displays with less and less “graph” feel. You’ll get a look at that when my first dimension reductions see the light.

On that front I’ve made some progress with Jama, which allows me to represent and manipulate matrices in Java. I ran my lovecraft example data through the paces of SVD, QR, LU, and such, but the real test will be when I try and manipulate the significantly larger amazon Tag matrix.

I DID use the larger data set with Weka, a VERY neat tool that exposes a TON of interesting ways to manipulate large sets of data. This is where I had the largest insights and failures. The failures involved quickly discovering the constraints of my memory as I repeatedly crashed the application with too much data. When I found a subset of the data (and an appropriate heap size) it could handle I actually got some of my first plots of Principal Component data. On one hand I’m giddy and on another I realize that I still have a long way to go. Weka’s Knowledge Flow functionality (complete with flashbacks to MatLab in college) allows for some crazy fast prototyping and experimenting.

I’ve got earlier editions of Numerical Recipes: The Art of Scientific Computing and Data Mining: Practical Machine Learning Tools and Techniques on hold at the library, so it should be a fun weekend.

Plotting Science Fiction versus Vampires

mentatjack — Sun, 06 Sep 2009 23:13:35 +0000

If you click on the above image, you’ll be taken to an interactive version of the plot with some explanation. As I see the completion of core Tag shadow functionality on the horizon I need to start asking if this makes ANY sense to anyone else. Let me know what you think, and be sure to check out the other 2 plots I made, Cyberpunk vs Space Opera and Alternate History vs Magic.

Question about free data

mentatjack — Fri, 04 Sep 2009 16:21:08 +0000

I’m considering making some of the data that I generate for TagShadow available so that others can make use of it in their own projects. My question is this: How do I license data?

My initial thought was Creative Commons, but their FAQ suggests that software should use a different license, and points me at Free Software Foundation and the Open Source Initiative. But my question is about the data and not the software used to generate it. My plan is to spend some time on those sites and look at the various Terms of Service and Usage Agreements attached to web services that generate data that I’ve found useful at one point or another. While I’m researching myself, I thought I’d ask the world at large.

TagShadow: A Random Glimpse

mentatjack — Sun, 23 Aug 2009 07:03:30 +0000

Today was a crash course in php, mySQL and Amazon Web Services. The true fruits of today’s labor are still a ways off, but since I have some data, I figured I’d use it for something. That data (still collecting) is a list of the Science Fiction and Fantasy book on amazon. I built a very simple page that grabs a random chunk of 40 books and outputs the covers. Of course these all link to amazon.

I’ve actually had some fun refreshing this page. The image above is a screen capture that featured a bunch of Zelazny books. I’d not realized that he’d collaborated with both Philip K. Dick and Alfred Bester. Explore some random books and let me know what gems you find.

Ontology, Tags and Such. Oh My!

mentatjack — Thu, 20 Aug 2009 05:42:59 +0000

I read a spectacular article today, Ontology is Overrated: Categories, Links, and Tags. It’s like it was written just for me. The section labeled mind reading really helped reign in the feature creep on TagShadow. Whenever I look at a tag cloud my immediate reaction is to start pruning away all the junk data. After reading this I’m pretty sure I’m comfortable stepping back and letting the data speak for itself.

Another decision helped me solidify my priorities with TagShadow. Amazon has decent tag data. I’m going to set up a sort of sandboxed alpha version that uses the logic I’m developing, but instead of user input everything will run on amazon data. Hopefully that’ll let everyone get an idea what I’m trying to build. Once the logic works for amazon’s canned tag data, I’ll implement a version that’ll handle users and run off the data they provide.

Having deliverables to work towards makes me a happy developer. Knowing what I don’t have to code is priceless. I’ve had the chance to experiment pretty extensively with jQuery the past couple days. At the office it was exceedingly useful and tonight I made some updates to TwitterPlusOne It’s been a good day.

Principal Component Analysis: The Lovecraft Experiment (part 1)

mentatjack — Sat, 18 Jul 2009 22:21:55 +0000

The Setup

I’ve been tweeting my linear algebra experiments today. Here’s a quick rundown of my goal. I want to take the data (tags) people use to describe stuff (in my case books) online and visualize that in such a way that it’s readily apparent which things are similar. The easiest way to map a whole bunch of points (books again) is to plot them on a plane … which gives me 2 dimensions to work with. The problem then becomes one of reducing 1000’s of dimensions (one for each tag) into 2.

Principal Component Analysis is to tool for this job. From the wikipedia page, this is the description of PCA that informs my metaphor:

If a multivariate dataset is visualised as a set of coordinates in a high-dimensional data space (1 axis per variable), PCA supplies the user with a lower-dimensional picture, a “shadow” of this object when viewed from its (in some sense) most informative viewpoint.

I acquired a copy of an academic paper on PCA, but first I decided to see how far I could get with a simple example and the insight that wikipedia could provide. I got this far:

The discussion on the PCA page pointed me at the page for Sample Covariance Matrix which was much clearer on the actual method for generating said matrix. Now it looks like I need to calculate the Spectral Decomposition using the suggested method of Singular Value Decomposition. I’m a bit stuck at Householder Transformation, so I figured I’d document my progress so far.

The Data

I decided to take one of my most recent reads, Wonder Boys (WB) and compare it to Atrocity Archives (AA). To play with 3 variables (one more than the 2 I’ll end up plotting) and still end up with a nice 3×3 matix, I included Call of Cuthulu (CC). For the purposes of this exercise, I’m describing these three books with the following tags:

Lovecraft
Fantasy
Modern

In the table below, I’ve set on a 1-10 scale how much the book relates to the particular tag. In my final application that would be compiled by the number of users tagging a book and the number of times each tag was used for that book.

	Lovecraft	Fantasy	Modern
WB	8	0	10
AA	9	8	10
CC	10	10	1

We can quibble extensively over the qualitative nature of the data above (in general I love such discussions), but right now I’m solely interested in the analysis. No matter how I go about the calculation, I’m going to need a vector that represents the mean for each row:

This is how I’m going to calculate the covariance:

In the above equation, x is the 3×3 matrix defined by my table above. x-bar is the “mean vector,” (6,9,7) and N ends up being 3. The results, which I computed by hand, so as to reacquaint myself with the linear algebra:

Stay Tuned. In part 2 of this series I will figure out what to do with my pretty covariant matrix. There will likely be a part 3 with a pretty graphic and some thoughts on the whole experience.

A special thanks goes out to Wolfram Alpha for generating the image of the covariant matrix above.

If there are any friendly mathematicians out there that want to point me at some resources, I’d be eternally grateful. Also, my final build of the code to run all of this will probably be in Java, although I’ve been eyeing Scala as something I’d like to play with. Feel free to contact me if you’re interested in helping out on the coding side of things. And last, but not least, the end use for all of this is going to be a Quantitative Visual Book Recommendation Engine. I’m going to need a ton of user input for this, so feel free to let me know if you’re excited by this project and want to help in any way. As always, I can be emailed at mentatjack dot com with the username mentat1.

development: Currently Reading Web Slice

mentatjack — Tue, 16 Jun 2009 02:18:37 +0000

There’s a new browser on the block, Internet Explorer 8. I’ve been obsessing recently about microformats, so I was intrigued by its “Web Slice” functionality. To test this out I wrapped my Currently Reading widget in a slice. I changed “Currently Reading” to “Current Input,” since I didn’t have a quick way to change the id of the h2 tag on that text … I’ll add a “Currently Listening To” slice later to test multiple slices.

I’ve tested this with IE8 and I’ll test in Firefox with WebChunks later. I’d love input or questions on this experiment

#scifibookclub 2: Spin by Robert Charles Wilson

mentatjack — Thu, 11 Jun 2009 04:19:41 +0000

I figured this was as good a time as any to test the hCalendar microformat.

June 26th : 30th, 2009 #scifibookclub 2

We’ll be reading Spin by Robert Charles Wilson, winner of the 2006 Hugo Award for best novel.

This hCalendar event brought to you by the hCalendar Creator.

Web Design and Consulting for Writers and Publishers

mentatjack — Wed, 27 May 2009 05:03:09 +0000

I’m obsessing about design more than usual as I delve deeper into TagShadow. Enter Clockpunk Studios. They specialize in websites for writers and I may very well hit them up for some help making my project pretty once I get my matrices all in a row. I particularly like that they seem to understand what a complete web strategy is. I develop for a company with 30k products and we manage to find people all over the web interested in our handmade art, so I’ve become pretty good at spotting the holes in a marketing solution. The folks over at Clockpunk Studios seem to have pretty much every base covered.