• Fast traversals:
  • Jumping from one vertex of the graph to another should be possible in O(1)
• Online processing:
  • “Standard queries” (<–whatever this means) should compute within miliseconds.
  • As an example: Local recommendations e.g. similar users in a bipartite “User – Band” graph should be possible to process online in less than a second.
• Query language:
  • A programming model that supports pattern matching and traversals with one (or possibly several) starting nodes
  • No SPARQL (too general for a reasonable graph application) support needed.
  • Support for reading and writing new data (to disk)!
• Distribution effort:
  • The programmer should not have to care about the distribution techniques.
  • He should just be able to use the technology.
• Fault tolerance:
  • The system has to run stable if working computers are added or removed.
  • Probably by introducing redundancy in some way [1]
• Persistence:
  • Transactions and persistence are important for any data base service.

It is very clear that this wish list is very high level. But I think these are reasonable assumptions from which we can break down the problem and discuss pros and cons of all the techniques needed to built such a system.   

[1] on the Redundancy discussion:

Depending on the techniques used, introducing redundancy has probably two positive effects on:

1. Fast traversals
2. Fault tolerance

On the other hand it has a deep impact on

1. Persistence (which is hard to achieve in a distributed setting anyway is even harder to achieve once redundancies are included.)

It is not clear if we really need redundancy. Maybe there are some other techniques that enable us to find our goals but I personally have the feeling that a good model for redundancy will “solve” the problem.

relation to the reading club

I already found the time to look over our courrent reading assignments. Especially the VLDB paper (Topology partitioning applied to SPARQL, HADOOP and TripleStores) and the Challenges in parallel graph processing strengthen my confidence that an approach described above seems very reasonable.

What is your oppinion?

Do you think I am missing some features or should keep a focus on one particular feature? What about methods to achieve those goals? I am happy to discuss your thoughts!

Related posts:

1. From Graph (batch) processing towards a distributed graph data base
2. Google Pregel vs Signal Collect for distributed Graph Processing – pros and cons
3. neo4j based social news feed demo on wikipedia graph running

Readinglist for next Meeting (Wednesday March 7th 2pm CET)

• memcached paper: To understand how for distributed shared memory works which could essentially speed up approaches like Signal Collect
• Beehive: to see a p2p aproach for graph distribution. 
• Challenges in parallel graph processing. For obvious reasons since it points out the large picture.
• http://www.boost.org/doc/libs/1_48_0/libs/graph_parallel/doc/html/index.html The boos library is a general parallel graph processing framework. In any case it is interesting and good to understand what is going on there.
• Topology partitioning applied to SPARQL, HADOOP and TripleStores Shows how a speedup of 1000x can be achieved due to smart partitioning of a graph

Again while reading an preparing stuff feel free to add more reading wishes to the comments of this blog post or drop me a mail!

Summary of yesterdays meeting

As written in the introduction we agreed that the papers where interesting but not heading in our direction. Claudio pointed out that everyone should consider the following set of questions.

• Do we want the graph to be mutable or is it supposed to writable or is it supposed to be read only?
  • writing makes sens. If it is read only it is called batch processing
  • Writing is hard you care about locking consistancy

• Do we want to answer queries (Cypher/gremlin/whatever)?
• Do we want to provide an API for processing?
• How big is the data set we want to support
  • many people do in memory
  • If you go to the disk you open a whole new bottle of topics
  • One approach would be to solve the problem in memory first.

I am very confident that it was a good idea to start with graph processing but that we are taking the right steps now to go in the direction of real distributed graph data base systems. I think there are some more questions and high level assumptions that one has to fix which I will post in a few days on this blog. Sorry I am in a hurry for this day / rest of the week.

Infrastructure

Schegi just suggested to create a Mailingliste for the reading club or to switch to Google Groups. He pointed out that a private blog is kind of a weired medium to be so central. What is your opinion on that? Do we need some other / more formal infrastructure?

Related posts:

1. Wishlist of features for a distributed graph data base technology
2. Google Pregel vs Signal Collect for distributed Graph Processing – pros and cons
3. Reading club on Graph databases and distributed systems

His promoter suggested something like:

1. You should send a press release to all music related websites
2. You should show the videos to your  friends and ask them to reshare it with their friends
3. You should make a riffle regarding the video

Before I start with my thoughts. In this blog article I will explain some fundamental concepts of online marketing. This is not only interesting for musicians but for any brand! The concepts of branding will be questioned hard.

The aked question is very fundamental for online (music) marketing

The oberservation of this musician is quite right and holds true for many musicians but also for other products. Actually it is a nice application of the Pareto Principle (8.9% of bands make up 91.1% of plays on Last.fm). And the problem exists not only on the web. Once you start a new thing with a lot of enthusiasm you will most certainly get to realize that no one really cares despite the fact that your product / music is good and your receive some nice feedback. I looked at some of the social media response for this particular artist and found out that he really is in the lucky position to receive a lot of great feedback and see that quite a lot people cared. Still it is  hard for him to increase his reach and inform people about this very product id est introducing the music to them.

Problem of overcoming the boarders of one’s own social circle

The entire problem breaks down to overcoming one’s own social circle. If you understand that the entire world is a network (it is really hard to fully understand all the implications of this) you understand that the big problem in (online) marketing is the following:

You have your ego network – or let’s call it social circle of people – that you can reach with an idea. You will realize within your  social circle some ties are better and some are worse. Some ties are so great that they might even help you out (for example because they are your record label or promoter or booker or die hard fans). But in the end they can be all combined as one person in the network that has a (somewhat bigger) social circle. And the social circle will look like the following pictures which I took from Marc Smith Blogpost about social networks in the news (A tutor of mine at Webscie summer school 2011)

In this series of pictures you can actually see how weak your social circle is (and again for a music band the circle might be bigger at all but the entire picture will look similar:

Even though you can reach anyone in our worldwide friendship network with only 6 people in between your social circle will only reach a view ten thousand people! So the question is: “how to get the other 7 billion?”

To make this even more clear: As a b(r)and you might “know” an amazing number of let us say 3500 fans (mabe you reach even more) and become a “rather” central node in the social graph of people and brands. But let uns see why this is not of any help!

1. you know 3500 fans
2. Those fans already have 410’000 (almost half a million) people in their ego networks which probably makes the use of Facebook so tempting. 
3. Your fans friends altogether know 51 mio people. 
4. Finally these people that you already know over three hops will know the entire rest of the world e.g. the other 7 bn people 

Do you really think your music / product / message / idea is so great that your friends will tell their friends who will tell their friends who will tell the others? That is really all it takes! 4 hops. Sorry to disappoint you. This is almost impossible to achieve!

The classic answer before the viral web was born: Advertising

Well the problem existed a long time before the web was born. People already have figured out the solution: You ask someone who has a huge reach to tell his audience about your new idea! Less sophisticated we can summerize this idea in one word: “advertising” you go to any media pay them some money and they will talk about you. This worked amazingly well in earlier days. There wasn’t that much media and if you had your product in some media you could be certain that you had at least a brand awareness. But there is a limitation of advertising…

A Problem of advertising and some advantages and disadvantages of online advertising

Of course once you run advertising campaigns your will receive some attention. No matter how good or bad your product is. That was the principle during the Dot com Bubble  where many start ups spent way too much advertising dollars and money to built reach instead of focusing on a great product. This is the reason why every ad campaign should carefully measure several things (fortunately in online marketing this is easier than anything else!):

• Conversionrate – as the percentage of people that convert to (fans, custormers, emailadresses,… basically whatever goal you have for the campaign)
• Bounce rate – as the amount of people that clicked your ad but left your page / product right away
• The price you pay per click (or per 1 thousand impressions)
• The value to you for a new fan (customer).

Once you know these numbers you can easily calculate weather an ad campaign is usefull for you or not! And you can calculate those numbers easily. The first three are given or measured during the ad campaign. If you don’t know how to do this. Contact me. The last one has its own paragraph.

What is the value of one new fan?

For the value of a new fan you could do the following. Count the number of fans you have (e.g. size of news letter / facebook fans …) and then look at your last year revenue (sum up: merch + concerts + sold music) devide this number by the number of fans and you see how much revenue one fan produces in one year. I know it is only a rough estimation but I wonder how many bands have calculated this number! Look at other products and markets.Everyone who is in the business of direct marketing maintains a customer data base which is a solid asset to the business. Groupon for example gives away 6 Euro for every new Customer someone finds. (This means that Groupon thinks a new Contact is worth at least 6 Euro.) 

As a musician you should know this number. Assume a new Fan is worth 6 Euro to you this would mean that knowing 5 thousand fans is worth as much as the production of an entire record. Could this be true? Actually I will try to start some calculations together with my Collegues from In Legend soon and try to calculate how much we should spend to gain a new fan!

Problem of aggressive marketing vs spreading of new information

On the web the problem is that people have so much choice that advertisting only helps to some degree. With Google adwords for example you only pay for clicks. So you definately obtain the attention. On the other side people drift away very quickly and their span of attention is very small. Another drawback is that many things on the web are free. Even though advertisting for music on the web is so much cheaper than offline advertising it is not standard to make use of advertisting on the web. Recordlabels and promoters so far refuse to invest money for it. That is sad because there is a lot of potential in online advertising (especially once you measure your conversion and bounce rates as well as the click through rates.)

Luckily those rates can be measured before you run an ad campaign. Investing money to increase reach should therefore never be done to a complete new product. A complete new product should be tested first. You meassure the feedback. Once you realize that people are happy you are save paying money to increasing your reach. For example in in legends case the youtube pandemonium video received much better feedback than vortex video and so far the soul apart video seems to receive the best feedback overall. By user ratings / average daily views as well as user comments. So you better go out an make advertising with your best performing content. Will you care that this is not your newest piece of work or even the first video? Hell no! Advertising is always targeted to people that don’t know you. You can even show them a 10 year old video and they might not realize its age! Once they like you they can still discover your newer stuff. 

Attention the above suggestion drastically changes if you are a mature b(r)and. In that case of course your star products are already well known and you should use promotional power and your brand recognition to get out the new product. This leads to the next paragraph.

Do all these calculations depend on the maturity of the band?

The answer is clearly yes! A band that is mature will still need promotion and advertising but also a lot of messages spread from almost alone. Some other things are easier achieved (mag title stories and so on…) But the obvious message is the following. The cost of advertising and your conversion rate will most probably stay constant and remain independent of the bands maturity. Metallica has to pay the same price for youtube ads as In Legend has to. But the fan is worth more if a band is not mature yet. A fan from the first record might go to many tours and buy a lot of products and also help spreading the word in a viral manner. For a mature band chances are pretty high that people who see the ad already know the brand but already decided not to like the band. So even though a young band has not much money and can not be sure that the product is already sufficiently good (which will lead to a drop in conversionrate i.e. more expensive ads or higher acquisition cost per new fan) ads make much more sense if the band is totally unknown. Be couragous invest some money and bootstrap your product! You are also courageous by going on the stage. So please also enter the web stage and built your reach! I know it’s less fun than rocking the audience in a life concert but the effect should be pretty much the same.

Other non advertising impact factors to overcome the boarders of one’s social circle

Since advertising is very expensive and probably not sustainable one can wonder if there are other ways to overcome the boarders of your social circle. I would say there are. The best way to do so in my pure empirical experience is transparicy, openess, the trust in other people and the interactive communication with them. Take me as an example. When I went to china I had basically no one to talk to about my interests. Now I started blogging sharing my ideas and – some people already called me crazy – parts of my intellectual propertie. Guess what! By doing so I was able to find more and more people with similar interests from all around the globe that I would never have met in my own social circle giving my valuable feedback to my ideas. My Reading club on distributed graph data bases is just a recent example of this added value from transparency.

So as a band here are some things you could and should do besides advertising your star products (and yes it is not the mainstream and might take some courage to do so!):

1. Interact with your fans. Treat them with respect. Not just by telling them but by really showing them. Don’t be so “kind” to share only news about you. That’s not interacting that’s publishing! Join the discussion on current topics (in music, news, …) or respond to what your fans say! Find out if it is relevant and stay in a conversation. I know it is hard since fans can become annoying and as a person it is already hard where a band has much more fame and more people and ideas to take care of. 
2. And please don’t be fake interactive by asking questions like “Heya Legends..
   A new week has begun, what is your sound for depressing mondays and what gets your through the week???” Asking questions especially if they don’t really carry a meaning is not interactive. That is just embarissing. That’s the reason why this kind of questions don’t take your marketing anywhere.
3. Integrate your fans! That is the by far most promesing strategy. As a band with already 3’500 fans there is so much diversity, creativity and so on that you will be able to achieve extra ordinary things. I am sure fans of you will have access to recording studios will have access to cheap videos maybe there are webdesigner and photographers. If you interact with your fans in a very smart way you don’t even have to ask them if there is a REPLACE BY WHATEVER YOU NEED. You will just know who is it. This leads me to 4th
4. Use the upcoming social network google+. First of all it is very obvious that it will kill Facebook on the long term (probably even on the short term) but more important it supports you to follow your fans since you can put them into circles. So all photographs go in one circle. All reviews go in another circle, all bloggers in the next circle, all technicians, all bookers, all concert organizers and so on. Streetteamer go in one circle. Follow those people they like your music and could be valuable for you.   
5. Speaking of streetteam: I once talked to a record label coworker. She told me in her experience the single best promotion tool for a band is a streetteam. Unfortunately I have seen many streetteams also did not receive the respect they deserved from band mambers. A band’s streetteam has an incredible impact. I once wondered how a poster of my favourite band came to be placed on a train station of a minor German city. It was much later that I realized that this must have been streetteam members… On the web by being interactive you can build your own global streetteam with almost no cost – besides time! Treat your streetteam. Have a streetteam meeting at least once a year and have all bandmembers come and have a great party with them. You can also make virtual meetings.
6. last but not least: Do all the things your promoter suggested at the very top of my post. Those are the core homework choires. 

For the last lines. Have fun making music and enjoy the most recent video of my band

Related posts:

1. Question by Filip Stilin (House on Mars): What do you think of Bandcamp?
2. you should use Youtube promoted videos for your music band
3. Stop Facebook – Filterbubble of facebook’s news stream & wall

So after I read both papers as well as Claudios overview on Pregel clones and took some notes here are my thoughts but first a short summary of both papers.

Summary of Google Pregel

The methodology is heavily based on Bulk Sychronous Parallel Model (BSP) and also has some similarties to MapReduce (with just one superstep). The main idea is to spread the data over several machines and introduce some supersteps. For each superstep every vertex of the graph calculates a certain function that is given by the programmer.
This enables one to process large graphs which are distributed over several machines. The paper describes how to use Checkpoints to increase fault tolerance and also how to make good use of the Google File System in order to partition the graph data on the workers. The authors mention that smarter hashing functions could help to distribute the vertices not randomly but rather in a way they are connected on the graph which could potentially increase performance.
Overall the goal of Google Pregel seems to enable one to process large graph data and gain knowledge from it. The focus does not seem to increase the usage of the calculation power of the distributed system efficiently. In stead it rather seems to create a system that makes distribution of data – that will not fit into one machine – possible at a decent speed and without much effort for the programmer by introducing methods for increasing fault tolerance.

Summary of Signal Collect

Signal Collect as a system is pretty similar to Google Pregel. The main difference is that the authors introduce a threshold score which is used to decide weather a node should collect its signals or weather it should send signals. Using this score the processing of algorithms can be accelerated in a way that for every super step only signals and collects are performed if a certain threashhold is hit.
From here the authors say that one can get rid of the superstep model and make the entire calculation asynchronous. This is done by introducing randomization on the set of vertices on which signal and collect computations have to be computed (as long as the threasholdscores are overcome)
The entire system is implemented on a single machine but the vertices of the compute graph are processed by different workers (in this setting Threads). All Threads are able to share the main memory of the system which makes message passing of Signal and Collect computations unnecessary. The authors show how the increasing number of workers actually antiproportionally lower the runtime of the algorithm in the asynchronous setting. They also give evidence that different scheduling strategies seem to fit the needs for different graphs or algorithms.

Discussion of Pros and Cons

• From the text above it seems very obvious that Signal Collect with its Asynchronous Programming model seems superior. But – in opposite to the authors – I have hidden to mention the drawbacks of one small but important detail. The fact that all the workers share a common knowledge which they can access due random access in main memory of the machine allows their model to be so fast while being asynchronous. It is not clear how to maintain this speed with a real distributed system. So in this way Signal Collect only give a proof of concept that an abstract programming model for graph processing exists and it enables fast distribution in theory.
• Pregel actually is a real frame work that can really achieve distribution of large data to clusters of several thousand machines which for sure is a huge pro.
• Signal Collect proposes to be more general than Pregel since Pregel can only respect one vertex type and edges are stored implicitly. Whereas Signal Collect is able to store RDF Graphs. I personally understand that Signal Collect can only send signals from one vertex to another if and edge exists and is also not able to add or remove edges or vertices. In this sense I still think that Pregel is the more general system. But I guess one can still argue on my point of view.
• Pregel’s big drawbacks in my opinion are that the system is not optimized for speed. As already discussed in the last meeting of the reading club Map Reduce – with its one Superstep attitude – is able to start Backup tasks towards the end of the computation in order to fight stragglers. Pregel has to wait for those stragglers in every superstep in order to make synchronous Barriers possible.  
• Another point that is unique with Pregel is the deep integration with Google File System (btw. I am almost through the google file system paper and even if you already know of the idea it is absolutely worthwhile reading it and understanding the arguments for the design decisions of the google file system). So far I am not sure weather this integration is a strong or a weak point. This is due to the fact that I can’t see all the implications. However it gives strenght to my argument that for a distributed system some things like network protocols and file systems should be considered since they seem to have a strong impact on the entire system. 
• Both systems in my opinion fail to consider partitioning of the graph and a different network protocol as an important task. Especially for Pregel I do not understand this since it already has so much network traffic. Partitioning the graph might increase start up Traffic on the one hand but could increase overall traffic on the long term. 

Outlooks and personal thoughts:

I am considering to invite the authors of both papers to next weeks reading club. It would be even more interesting to discuss these and other questions directly with the guys who built that stuff. 

Also I like Schegi’s idea to see what happens if one actually runs several neo4j servers on different machines and just use a model similar to Signal Collect or Pregel to perform some computations. In this way a programming model could be given and research on the core distribution framework – relying on good technologies for the workers – could be done.

For the development of the first version of metalcon we used memcached. I read a lot that memcached scales perfectly horizontal over several machines. I wonder how an integration of memcached to Signal Collect would work in order to make the asynchronous computation possible in a distributed fashion. Since random access memory is a bottleneck in any application I suggest to put the original memcached paper on our reading list.

One last point to mention is that both systems still don’t seem to be useful as a technology to built a distributed graph data base which enables online query processing.

Related posts:

1. From Graph (batch) processing towards a distributed graph data base
2. Some thoughts on Google Mapeduce and Google Pregel after our discussions in the Reading Club
3. Claudio Martella talks @ FOSDEM about Apache Giraph: Distributed Graph Processing in the Cloud

Kind of amazing that he actually did this. I am really looking forward to the time where these kind of events are not amazing anymore but rather standard like they should be. It is also interesting to see what technology he chose. Well I guess it was not for Google Plus’ publicity but rather for the brand recognition of Youtube. Anyway I didn’t see Facebook around there. Also I think questions like this should not be decided by how a president behaves it still is interesting to see how Obama chose Google over Facebook in these days…

Related posts:

1. how to invite people to Google plus
2. Google 2011 Q4 Earnings
3. Google, Facebook & co. are not free!

Summary

First take away which was well known is that there is a certain stack of Google papers and corresponding Apache implementations:

1. Google File System vs Apache Hadoop filesystem
2. Google Big Table vs Apache HBase
3. Google Map reduce vs Apache Hadoop
4. Google Pregel vs Apache Giraph

The later ones are all based eather on GFS or HDFS. Therefore we agreed that a detailed understanding of GFS (Google file system) is mandatory to fully understand the Map Reduce implementation. We don’t want to commonly discuss GFS yet but at least think everyone should be well aware of it and give room for further questions about it on next weeks reading club.

We discussed map Reduce’s advantage of handling stragglers over Pregel’s approach. In map reduce since it is a one step system it is easy to deal with Stragglers. Just reassign the job to a different machine as soon as it takes to long. This will perfectly handle stragglers that occure due to faulty machines. The superstep model in pregel has – up to our knowledge – no clear solution to these kind of Stragglers (to come up with a strategy to handle those would be a very nice research topic!) On the other hand Pregel has another kind of Stragglers that come from super nodes. There are some papers that are fixing those problems one of them is the paper that will be read for next week.

We had the discussion that partitioning the data in a smart way would make the process more efficient. We agreed that for Map Reduce and Pregel where you just want to process the graph on a cloud this is not the most important thing. But for a real time graph data base the partitioning of data will most certainly be a crucial point. Here again we saw the strong connection to Google File System since the Google File system does a lot of the partitioning in the current approaches.

Achim pointed out that Microsoft also has some proprietary products. It would be nice if someone could provide more detailed resources. He also wished that we could focus on the problems first and then talk about distributing. His solution was to make this top down.

We also discussed if frameworks that use map reduce to process large graphs have been compared with Pregel or Apache Giraph so far. This evaluation would also be a very interesting research topic. For that reason and to better understand what is happening when large graphs are processed with map reduce we included the last two papers for reading.

Feedback from you guys

After the club was over I asked everyone for suggestions and I got some usefull feedback:

• We should prepare more than one paper
• google hangout in combination with many people in the room is a little hard (introduce everyone in the beginning or everyone brings a notebook or group of people should sit in front of one camera)
• We need more focus on the paper we are currently discussing. Understanding problems should be collected 1 or 2 days before we meet and be integrated into the agenda.
• We need some check points for every paper. everyone should state: (what do i like, what do i not like, what could be further research, what do i want to discuss, what do i not understand) 
• We need a reading pool where everyone can commit

New Rules

In order to incoperate the feedback from you guys I thought of some rules for next weeks meeting. I am not sure if they are the best rules and if they don’t work we will easily change them back.

• There is a list of papers to be discussed (see below)
• At the end of the club we fix 3-6 papers from the paper pool that are to be prepared for next week
• before the club meets everyone should commit some more papers to the pool that he would like to read the week after (you can do this on the comments here or via email)
• If more people are in the same room they should sit together in front of one camera
• Short introduction of who is there in the beginning
• use the checkpoints to discuss papers
• no discussions of brand new solutions and ideas. Write them down, send a mail, discuss them at a different place. The reading club is for collectively understanding the papers that we are reading.

Last but not least. The focus is about creating ideas and research about distributed real time graph data base solutions. That is why we first want to understand the graph processing stuff.

Reading tasks for next week

for better understanding the basics (should not be discussed)

• Google File System

To understand Pregel and another approach that has not this rigid super step model. The last paper introduces some methods to fight stragglers that come from graph topology.

• Signal Collect
• Google Pregel
• HipG: Parallel Processing of Large-Scale Graphs

And finnaly two more papers that discuss how map reduce can be used to process large graphs without a pregel like frame work.

• Pegasus Paper
• Design Patterns for Efficient Graph Algorithms in MapReduce

More feedback is welcome

If you have some suggestions to the rules or other remarks that we havn’t thought of or if you just want to read other papers feel free to comment here in this way everyone who is interested can contribute to the discussion.

Related posts:

1. Reading club on Graph databases and distributed systems
2. Google Pregel vs Signal Collect for distributed Graph Processing – pros and cons
3. From Graph (batch) processing towards a distributed graph data base

Teaching classes is over for this term so for the next couple of weeks I want to spend a lot of time working on some research topics that are on my mind. My goal is to finnaly write down my PhD proposal and have a well organized written structure for the rest of my PhD time.

The main topic for 2012 in the scientific part of my life can be summerized by these bullets points:

• Graph data bases
• distributed systems
• distributed computing
• distribution of graph data bases
• dynamic hash tables
• peer to peer networks
• graph data base query languages (since this seems to have a deep impact on the technologies that suport everything)
• real time graph processing

So the reading club will read and most importantly understand and discuss papers that belong to those categories.

I will start with the following selection of papers:

• Google Pregel
• Map Reduce
• Beehive

Time and Place

The reading club will take place in D116  the “Kreuzverweisraum” and will take place every wednesday 2 pm CET . 

For next week I expect from anyone who wants to join that the Map Reduce paper will be read by Wednesday.

I will keep anyone up to date with the results from the reading club and the anouncements for next weeks readings.

How to join on the web!

I shared a google plus circle with all people who are interested:
https://plus.google.com/115250982031867883098/posts/AhSZgvbKYs8
You can contact me to be included to the circle. The circle will be invited to a hangout every wednesday about 2 pm CET (central european time)
If anyone knows a better technology for the telco feel free to tell me.

Related posts:

1. From Graph (batch) processing towards a distributed graph data base
2. Some thoughts on Google Mapeduce and Google Pregel after our discussions in the Reading Club
3. Google Pregel vs Signal Collect for distributed Graph Processing – pros and cons

• Topics were Blueprints (a common graph accass api) created by Marko Rodriguez as 
• Problem of real time graph processing 
• Benchmarking issues (we need standards for benchmarking). 
• A guy from OrientDB raised the question weather Graph databases should really have ACID transaction?
• Max De Marzi raised the question about are graphs changing while processing or are they rather static?
• Achim Pointed out that Relational databases are actually a special case of Graph databases. He demands vendors to generalize more and consolidate the technologies… 

The room was not as full as the talks before but sill half of the seats have been filled as you can see on this short video:

My thoughts on acid discussion

I think that the ACID question was interesting. Alistair from neo4j gave a fine response to this saying that it clearly depends on the usecase and the kind of transactions that you really need. He compared to the relational data base world where you might have the option of switching ACID off.

He says that “in neo4j you cannot shut off ACID as neo4j believes that for most of their customers this is the best choice. But he admits that there are use cases where you migh want to shutoff ACID.

Changing graphs vs static graphs

I think that this is also a very important question. On one hand we have static models like Giraph that are able to find answers on huge static graphs on the other side you have situations like graphity where you have fast fluctuation in data. Unfortunately for the later there is no technology I know of (besides hidden Facebook and Twitter and Googleplus) .

the entire disussion was recorded here:

Related posts:

1. Nils Grunwald from Linkfluence talks at FOSDEM about Cascalog for graph processing
2. From Graph (batch) processing towards a distributed graph data base
3. Google Pregel vs Signal Collect for distributed Graph Processing – pros and cons

In this scenario obviously big data is envolved and the data carries a natural structure of a graph. He sais a system to process the data has the following constrains:

• The processing should not compromise the rest of the system
• Low maintenance costs
• Used for queries and rapid prototyping (so they want a “general” graph processing solution as customer needs changes)
• Flexible, hard to tell which field or metadata will be used beforehand.

He afterwards introduces their solution Cascalog based on Hadoop and is also inspired by cascading a workflow managment system and datalog a subset of prolog which as a declarative, expressive language is very concise way of writing queries and enable quick prototyping

For me personally it is not a very interesting solution since it is not able to answer queries in realtime which of course is obvious if you consider the technologies it is based on. But I quess for people that have time and just do analysis this solution will properly work pretty well!

What I really liked about his the solution is that after processing the graph you can export the data to Gephi or to Neo4j to have fast query processing. 

Hey then explained alot specific details about the syntax of cascalog:

nils grundwald from linkfluence talks about cascalog at fosdem

Related posts:

1. Claudio Martella talks @ FOSDEM about Apache Giraph: Distributed Graph Processing in the Cloud
2. From Graph (batch) processing towards a distributed graph data base
3. Birds of a feather: Graph processing future trends in Graph Devroom

here the video streams from the graph dev room:

Related posts:

1. Nils Grunwald from Linkfluence talks at FOSDEM about Cascalog for graph processing
2. From Graph (batch) processing towards a distributed graph data base
3. Birds of a feather: Graph processing future trends in Graph Devroom