FolkMind

Others who are working on similar lines as FolkMind

Sun, 13 Nov 2005 20:55:06 +0530

Quintura is a compamny that is working on solving the search problem. The following is a list of problems they are working on:

1. Inability to define, manage and visualize the context for your keyword
2. Documents ranked by a search engine according to an algorithm specific to a search engine, and not specific to your interests
3. Over-ranked commercial and under-ranked non-commercial search results
4. You spend time on the first few pages of the search results, and don't have time or patience to go beyond the first and second page
5. Your search results don't always capture your trusted and valued sources

Interested in joining the FolkMind team?

Sun, 13 Nov 2005 16:58:46 +0530

When I put up the recruitment notice at SourceForge I got around five response. Before taking in any one, I thought of defining some basic expectations from my side.

First I need to know if you are really interested in the idea behind FolkMind.
A good way of demonstrating this to me is by sending me a MindMap of what you think of FolkMind. A FreeMind is a good tool with which you can do this (and like the name implies it is free :-) ). If you are not using this tool, send me the stuff that you create as a picture.

Second I need to know how good you are with implementing stuff and if you are used to the open source model of working. A pointer to any implementation that you have done (or help build) would be good enough. Obviously it should be available online – preferably on source forge.

If interested please send the above mentioned details to folkmind at gmail dot com

Digg FolkMind

Sun, 13 Nov 2005 16:31:00 +0530

According to Digg.com, this is what they are all about:

Digg is a technology news website that combines social bookmarking, blogging, RSS, and non-hierarchical editorial control. With digg, users submit stories for review, but rather than allow an editor to decide which stories go on the homepage, the users do.

Also the readership of digg is nearing that of slashdot.org. They are so successful that people are already making digg clones. One such example is shoutwire.

I have posted FolkMInd on Digg. So if you like the idea of FolkMind please help me spread the word.
* Link to Digg the first article.
* Link to the second blog

Thanks

FolkMind is getting some attention!

Sun, 13 Nov 2005 16:11:59 +0530

The total number of links returned by Google when you search for FolkMind has crossed 500. That is nice...
MSN is also returning similar number of links, but Yahoo is way behind.

Some interesting facts about Google results:
* The first two links that Google returns point to my Blog and the primary blog entry on the subject.
* The third link points to the SourceForge project on FolkMind.

With MSN the results are similar, but the order is the reverse.

Yahoo return only about 120 links, but the first few results are very similar to MSN.

So Yahoo and MSN seem to give more priority to a higher ranking web site.

Is FolkMind better than Google?

Sun, 13 Nov 2005 02:52:19 +0530

I have been discussing the idea behind FolkMind with one of my friend and the conversation went something like this:

Folk: So George, I have heard that you have coined this new term ‘FolkMind’. Tell me more about the idea behind it. I didn’t get time to go through the long blog that you have written on this.

George: It is about a new way of retrieving information from the Internet, using the collective intelligence of…

Folk: Oh, then that’s what it is, another search engine! Haven’t you heard about Google!!

George: Well not exactly. A good search engine will help you find information, but it doesn’t help you in understanding it. But FolkMind will!

Folk: What do you mean by ‘make you understand’?

George: Let’s take an example. Say you are searching for something like ‘Web 2.0’. Google returns you more than 15 million records.

Folk: Exactly you got the information that you wanted. It was that simple.

George: But that is too much information and like I mentioned in another blog, too much information is no information.

Folk: But the links that you got from Google are sorted based on the relevance. You know they use this algorithm called ‘PageRank’ which evaluates what the Internet community has to say about the subject being searched for.

George: To handle as huge an information source as the Internet, you require a totally decentralized community driven approach like PageRank. That’s obvious! But I still have to go through multiple pages to find out and understand what I was looking for.

Folk: Do you expect to find pages that exclusively contain what you are looking for?

George: Well, I don’t expect that! When you ask a question to an Oracle, you expect her to give a single answer instead of 50 different instructions, which you have to club together. Similarly, wouldn’t it be nice if the search engine could return you a page that is generated for me based on my query?

Folk: You mean an auto-generated page? Come on George, you must be kidding. I would prefer to read through 10 man-made pages to get what I want, than to go through all the junk returned by a bot.

George: Well, have you seen a newspaper that is made completely using an algorithm? The newspaper is called Google News!

Folk: Yes. I know about it. But news is different you know..

George: How come? As you would already know, Google News allows you to customize the news based on your preferences. So, in a way, it is also a search that you perform and the algorithm generates a page for you.

Folk: Hey, but Google News also just lists a set of links; you still have to navigate to the source page to make sense out of the information

George: That could be because Google doesn’t own the news and hence expects you to visit the original page. Well, coming back to the discussion on search, the result of a query need not be text that reads like a story; instead it could be a diagram that represents the essence of it. That way any bot that generates the result wouldn’t have to worry too much about structural stuff like grammar and readability. Think of the benefits – a picture is better for easy and quick comprehension as you would be using both halves of the brain when you ‘read’ a picture. Also, if you bookmark the picture instead of the text article, you will be able to recollect the matter by just glancing through it instead of reading the whole stuff again.

Consider the following facts:
* 60% of what you read are structure words and not content.
* 80% of the stuff that you read is forgotten within 24 hours.

Folk: What are you trying to say - that I am wasting my time in reading?

George: If your intension is to comprehend as much stuff in the shortest possible time, then reading verbose text is not the way for doing it. Say for example, suppose you were asked to review 50 code files. Would you start by reading from File1, File2 etc. or would you look for a UML diagram and then drill down for more details?

Folk: I would prefer UML diagrams and I agree that pictures are better than normal text for comprehension. But how can you compare structured data like code with text that you find on the Internet: You cannot create ‘UML’ like diagrams out of free text.

George: Yes. You are right. But if you do not want very precise data, why not represent the text as a MindMap. If you need more detail on any specific data you can always look at the original text by clicking on the appropriate section on the MindMap.

Folk: Hmm.. Sounds interesting. So, I do a Search and the result that I get is a MindMap that is generated by combining the related information from different places on the Web. What is returned is knowledge instead of links to thousands of sites where I can find the data.

George: Also the knowledge is presented in an easy to comprehend manner – as a MindMap.

Folk: But I know of a few other sites that do similar stuff, like Grokker and …what is the other name ...yes, Kartoo. How is FolkMind different?

George: Consider this. When you do a Google Search for say “Microsoft”, you get a few hundred million links. Anything beyond the first few pages are not useful for me. The tools that you mentioned above are designed to just solve this very problem. The value addition provided by these tools is that they can better represent the search results graphically. Grokker does a good job of categorizing the results. Kartoo goes one step ahead by also finding out common links between the results. So they are search enhancement tools. But as you see, FolkMind is more focused on retrieving “knowledge”.

Folk: Can you explain this better?

George: Sure. What you expect out of a search is not a list of probable links or their graphical representation. What you need is the “knowledge” based on your context.

Folk: Context? You mean..

George: When I do a Search, I am trying to extend my current knowledge along a particular axis. If a search engine really knows “me”, then it can return more relevant information. This would want the search engine to be aware of my current knowledge set

Folk: Hmm. I agree. This could be the reason behind the current trend of personalized search, search history and so on… so that the search engine knows my interests. So how does FolkMind satisfy this need?

George: To start with, FolkMind is a digitized version of the knowledge that you have. It helps you in extending you current knowledge by utilizing the Internet. The Internet is the biggest source of information and in a way represents the virtual brain of humanity.

Folk: A digitized version of my knowledge – what do you mean?

George: Let me take bookmarks as an example to explain this. When ever you find something interesting on the web you bookmark it. Right?

Folk: Yes – Obviously I won’t be able to recollect every thing that I have ever read and bookmarks help in recollecting stuff later on.

George: Precisely. Now assume that instead of bookmaking stuff, you use FolkMind to create a MindMap and store that instead of the bookmark. So when you have to recollect stuff, you can glance through the MindMap and be done with it instead of going through the whole text. Also assume that when adding new maps, FolkMind helps you in cross-linking it with previously stored maps. Over time this store will become a huge collection of interconnected concepts.

Folk: I can imagine. In a way this would resemble the human brain itself. Just like in the brain, we have neurons connecting to other neurons; FolkMind would have concepts connected to other concepts. Now I get what you mean by ‘digitized version of ones knowledge’!

George: The value of such a store will increase as the amount of data in it increases.

Folk: But George, that would be a lot of effort to create such a store. Do you think people will be willing to put in that extra effort?

George: Think of it like this. On one hand we have the extra effort to populate the MindMap and on the other hand you have the benefit of recollecting stuff easily. So if you reduce the effort in creating the MindMap store, the benefit could be enough motivation to put in that extra effort.

Folk: Well, you never know. Most people could be like me –Very lazy :-)

George: Ha Ha .. I know. That’s where the community can help. Take the example of Wikipedia. Only a fraction of the people takes the effort to update the articles. But many people use it.

Folk: How do you propose to reduce the effort for creating the MindMaps and that too using the community? I thought you just said that FolkMind would be a local application!

George: Think of FolkMind like Google Desktop. Yes, It runs locally but it can connect to the FolkMind server to get more data. Let me take an example. Assume FolkMind runs as a browser plug-in, always accessible to you. When you browse to a page, FolkMind can do some ‘Natural language processing’ on the text and will try to automatically create a MindMap and this is done in the background. Also the resulting MindMap is cached. The similarity with Google Desktop is that the former tries to index what ever you see; FolkMind tries to create a MindMap out of it. You can imagine that this auto generated MindMap would not be perfect. That’s where the user can help. If he is really interested in a particular article, he can invoke the plug-in and brings it to foreground. This will result in the MindMap graphically displayed over the article as an overlay. Also it presents tools for the user to quickly make corrections to the MindMap, like add, delete or modify nodes or connection. When a user bookmark this article, the MindMap thus generated is also linked with it.

Folk: Ok, now I understand how Natural Language Processing done by FolkMind can help reduce the effort put in by the user in creating MindMaps. How can the community help here.

George: This is the most important aspect of FolkMind. Any MindMap that is generated by the user is also saved on the server in his profile. The advantage of this is that wherever you connect, your MindMaps are always available to you. Also, FolkMind server can also share this data with other users. So the content that you helped create is used to increase the accuracy of MindMaps generated when someone else looks at the same page. Also you benefit from what others have contributed. This collective value addition is key to increasing the accuracy and relevance of the MindMaps generated. As more and more people use it the content becomes accurate and relevant. Over time more pages get stored and cross-linked.

Folk: WOW that’s a very powerful idea. FolkMind server can help create a Concept Map for the whole Internet. This will help change the way we look at the Internet. Instead of considering the Internet as a collection of web pages connected together using hyperlinks, using FolkMind we can visualize the Web as a collection of interlinked concepts. So, instead of navigating through hyperlinks, you can navigate through concepts.

George: This will also change the way you search for stuff. Instead of doing keyword searches you can do concept searches. And since FolkMind already knows about your context or knowledge set based on what you have already stored in your profile, it can retrieve you stuff that is very relevant to you or even suggest new stuff. It is like Amazon suggesting things to you based on what you have looked for before.

Folk: So if my knowledge set already contains stuff like Web 2.0 and AJAX, FolkMind can suggest to me things like Atlas as and when they emerge on the Internet.

George: Precisely. It can help you create new useful connections between existing concepts. In essence it can help you create new ideas!

Folk: Now I am really exited about this. How far have you finished this?

George: FolkMind is still in a very early stage of development. I have created a project on SourceForge. If you are interested you can join in. Also you can help spread the word by digging it here.

Introducing the first movers

Sat, 12 Nov 2005 16:45:50 +0530

It is encouraging to see that many people have shown interest to help build FolkMind.
The following are the official members in the FolkMind SourceForge project:

* C.Sylesh - A former colleague of mine. A really smart guy.
* Ann Catherine - My wife. We have been brainstorming this idea much before I started speaking about it.

There are others who are un-officially working for FolkMind. They will soon join the SourceForge project.

Too much of information is no information

Mon, 24 Oct 2005 00:27:59 +0530

I recently saw the movie PI, which is about a Math genius trying to figure out patterns in nature. I didn't like it very much, though IMDB users seem to have a different opinion.

One of the things that is said about the movie is that mathematicians have always tried to find repeating patterns in the expansion of PI and have come up with all possible sequences of numbers. This is another way of saying that there is no pattern in the expansion of PI (You won't believe it, but people have expanded pi to almost six and a half billion digits and you can get it from here).

To me this is similar to saying 'too much of information is no information'!; unless you have the right tools that can process all this data and present it in a meaningful way. This is very significant for the web which is the biggest source of information. Hence the need for better tooling - like FolkMind.

Let’s take News.com as an example. Instead of presenting users with a long list of new stories, they have added tooling that can help readers make more sense of the information and help them realize the connection between stories – In short help users realize ‘The Big Picture’.

So if we need to handle al the information that is available on the Internet, we need radically different tooling that can help us visualize the data in radically different way.

Created SourceForge project for FolkMind

Tue, 18 Oct 2005 00:47:38 +0530

I have created a SourceForge project for implementing FolkMind - a better Web 2.0 browser as described in my previous blog.

I wanted to name the project as 'FolkMind - The browser for the virtual brain for humanity'. But because of SourceForge restrictions on the length of the name I had to cut it short as 'FolkMind - The virtualbrain of humanity'.

The project will be available at the following location: [http://folkmind.sourceforge.net]

If you are interested in contributing to this project, please do join in - I will definitely need some help.

Tags: Web 2.0 FolkMind

FolkMind – a killer app for the Web 2.0 era

Fri, 14 Oct 2005 11:45:02 +0530

Explosion of content on the Internet

Currently, a lot of information is being published in the Internet as blogs, which are rich in content, frequently published and scattered across numerous sites. MSDN Blogs alone hosts around 2000 blogs. It is very difficult to aggregate this information and get a consolidated view of the same. RSS readers and content aggregators have tried to help us achieve this to some extent, but these tools cannot scale up to handle the voluminous and widespread content in the Internet.

The brain as a better model for describing the Internet

The Internet in its current form represents the collective intelligence of humanity and it has many characteristics that are similar to the human brain. Just like new connections are made between neurons in our brain as we learn new things, as new concepts emerge on the Internet, the links between the documents that describe the concept gets stronger – quite similar to the organic growth that we see in the brain. Also in the brain the connections between the neuron are more important than the individual neuron (though the neuron forms an integral part of the whole, they are far too many to be considered individually). Similarly in the web the concepts that emerge out of connected documents are more interesting than the individual document itself. One might argue that a document(s) authored by a few authoritative person(s) on a subject might be a better representation than the collection of documents made by the whole crowd. Though it might sound counter intuitive, in his work titled 'the wisdom of crowds' James Suriowecki explains that a better representation and a more accurate picture emerges out of the collective decision of the crowd than the work of a selected few. This is also probably one of the factors behind the success of Wikipedia as an accurate encyclopedia.

Currently the most prominent means of getting information from the Internet is through search. This approach is good enough if we already know what we are looking for – or if we know the right question to ask. But if we visit the Internet with the intension of finding something new, a better approach would be to navigate the Internet directory or taxonomy.

Benefit of Folksonomy in favor of Taxonomy

Lets take the previous use-case of browsing through MSDN blogs, looking for something 'new’ and ‘interesting’ – this is a good example of a situation where I wouldn’t know what exactly to search for. Instead of individually going through each blog, I can extract the statistically unique terms to form a taxonomy. My intension in doing this exercise is to distill the contents of more than 2000 blogs into a few words and then pick out (from the resultant set) those terms that I find to be interesting. This can be easily done using TagCloud and the result is as follows.

You can see the original cloud here

Is this auto-generated taxonomy good enough representation of what is published through MSDN blogs? In my opinion-No. This result is purely statistical in nature and I would compare it to the result of Google index with out 'page rank'. It does not take into account the collaborative content selection and filtering that happens usually on the Internet. It is this additional data that makes the data more relevant. A better approximation would be to use the celebrative tagging also referred to as 'Folksonomy'. The term folksonomy is defined in wikipedia as "a neologism for a practice of collaborative categorization using freely chosen keywords. More colloquially, this refers to a group of people cooperating spontaneously to organize information into categories, typically using categories or tags on pages, or semantic links with types that evolve without much central control."

This kind of tagging allows for the kind of multiple, overlapping associations that the brain itself uses, rather than using rigid categories. Such flexibility in using tags is both good and bad. On one hand we have tags like 'blog' and 'blogs' appearing as different tags. On the positive side, a photo of a smiling baby might be tagged 'baby', 'happy' and 'cute'. So in effect folksonomy produces results that more accurately reflect the population's conceptual model of the information.

The need for better tooling

If the brain is a better representative model of how the Internet works then we need different kind of tools to navigate and retrieve information from it. To be able to cope up with the vast amount of information, it should be capable of navigating across concepts instead of across documents. And once we locate our exact match, we should be able to drill down to it. To explain this new UI, lets take the example of Google Earth. To be able to locate a particular spot on earth (which is not previously tagged by Google - hence not available to search), we can take two approaches. The brute force approach would be to hunt through all the locatable points on the surface, until we reach our point. A more efficient strategy would be to zoom out (in other words elevate our self to a higher altitude), where we get an over all picture and then drill down to our point of interest. So to handle more complex problems, we need to create better abstractions. Another benefit of higher abstractions is that, at higher level we can easily spot associations and connections between locations (or concepts) that are hard to find at ground level. In the analogy, just as we are able to navigate across continents, countries and states, we should be able to navigate across concepts that emerge out of the Internet. Another vital feature that is missing in the tools that are currently available is the ability to discover and make connections between concepts. It is this lack of tooling that led me to envision FolkMind.

The vision of FolkMind

To me the new killer app for the Internet should help me in working at any levels of abstraction. The higher the abstraction, the more volume of complexity and data I can handle. Also at any level of abstraction, I should be able to navigate between concepts that are visible at that level and observe new connections that were not apparent to me at a different abstraction. And when I want to dig deeper, it should help me in exploring more on that subject. At the lowest level of abstraction, it would resemble a browser. The mass of the content that is on the Internet will still be on HTML, which is doing a good job of capturing presentation information, and a browser is suitable to view this. In short this application should act as a seamless extension to mind and help me in generating ideas by creating new connection between concepts about which I have little or no previous knowledge by leveraging the collective intelligence of humanity. A 'mind map' would be a good UI (an example of such a UI is shown below) for representing the above-mentioned vision. Wikipedia defines the term 'mind map' as 'a pictorial representation of how a central concept is linked to other concepts and issues'.

As a start we can create a mind map of the existing folksonomy that already exists on the Internet (with data from sites like del.icio.us) and then add new content and nodes to it. In his article entitled "Using Wikipedia and the Yahoo API to give structure to flat lists", Matt Biddulph explains a simple method for automatically converting a set of terms into a connected graph. To me the idea of linking together concepts is quite powerful. Once we reach a critical mass of concepts defined in such a mind map, it can transform itself from a concept management tool to an idea generation tool.

A new person who logs into FolkMind can start with the most popular folksonomy terms and from that point browse related concepts or he can start by searching for a particular concept. Each node in the mind map can be tagged with additional information like a short description, its relevance (based on algorithms similar in principle to the one used by Google page rank), and additional information. This is the highest level of abstraction and at this level the user is more concerned with the connections between concepts than the individual documents that contributed to that concept. With usage the folksonomy gets richer and more concepts and connections between concepts emerge. Once the relevant concepts are identified, the next step is to drill down to the individual documents that pertain to that subject.

If the Internet can be considered as the virtual brain that represents the collective intelligence of humanity, then FolkMind is the pictorial representation on the same expressed as a mind map – hence the name.

How does FolkMind fit in as a Web 2.0 application

Let me explain how FolkMind application demonstrates the traits that are commonly observed in Web 2.0 applications. FolkMind can be a thick client that connects to the FolkMind server to retrieve its content but it uses the client side processing power for rich interactive UI and for local cache. As the user interacts with the UI, any change is reflected back to the server (this is similar to how Google Earth works). Though a user is given the option of marking certain connections as private, all other new connections and nodes that are created by the user will be stored on the server and will be visible to other users. This action is similar to a person creating a bookmark and tagging it using del.icio.us. Thus a user pursuing selfish interests (the motive behind creating a new connection or node is for his own benefit) build collective value for the rest of the users as a side effect. This phenomenon (also referred to, as the network effect) is critical to the success of a Web 2.0 application. As more content gets added to the system and as more users join in, the value of existing users will grow.

Conclusion

As we have seen in this article, the vision behind FolkMind is to be a powerful application with an intuitive, interactive UI that can harness the power of Internet by being capable of handing huge volume of data. Eventually, this will become the virtual brain of humanity!

References

What Is Web 2.0 by Tim O’Reilly
Personal Brain
Using Wikipedia and the Yahoo API to give structure to flat lists

Tags: Web 2.0 FolkMind

Updated: Fixed some typos and added a few tags