Left Image: Here’s some sample data that is derived from NACTAS 2007 North American Social Technographic Online Survey.
Although I primarily rely on the proven Forrester Research methodology, when it makes sense, I tap into my network to discuss concepts and get examples. While only a part of methodology is evident to the public (what you see on this blog, or on twitter) this post will make it crystal clear what works and what doesn’t.
Forrester methodology
For all of my reports, I follow the research methodology that many other analysts have used before me for the past 25 years. This tested process may involve analyzing Forrester’s large technographic and consumer data banks, work with our data research team, conduct interviews, surveys, talk with clients (buyers), talk with vendors (sellers) discuss with fellow analysts, venture capitalists, and research online it’s a thorough way to see the market holistically.
A social computing analyst uses social technologies
As an analyst that covers Social Computing for the Interactive Marketer, it’s natural to use the very tools that I’m researching in fact, it was one of the factors in how I earned the role. Using the tools is a great way to learn what works and what doesn’t, a key step before I make recommendations to clients. Also, this blog serves as a supplamant to clients as well as myself, for example I’m known for building many “industry index” lists in the space, it helps me to keep track of who’s doing what, and since I screen each submission, the signal to noise ratio is higher than a wiki.
In tune with my market: the Interactive Marketer
At Forrester, we’re client centric, and as such, I serve the Interactive Marketer. Recently, we ran a survey of my blog readers, and found out that the largest group of my blog readers (14k subscribed) are interactive marketers or work at an interactive agency. Also having a twitter network of 10k followers is an amazing resource. Fortunately, this audience is a match to the role that I serve, making the following two uses cases possible:
When crowdsourcing makes sense
These tools make it easy to flesh out concepts and find examples but deeper research is always required, here’s some examples where it does work:
Use Case 1: Fleshing out concepts
I’ve found that asking questions from my blog and from my Twitter account are helpful for certain information gathering, but not for other needs. For example, I used a discussion on my blog and twitter from social media experts to vet a definition for the term “Online Community”, given this is the crowd of experts on this topic (and they are very willing to collaborate), this was an appropriate way to use this. I then scrubbed the answers, and was able to extrapolate the general key ideas (crowd sourcing is often rough on the edges) and come to agreement with my editor. In the endnotes of the report Online Community Best Practices we referenced the crowdsourcing effort. Please note in that report Online Community Best Practices, I interviewed 17 companies, and relied ton Technographic data, a standard procedure in report methodology less than 5% of the report was crowdsourcedUse Case 2: Finding case studies and examples
Also, Forrester reports are chockfull of examples of companies doing it right and wrong. Given my network is tuned into what’s happening with brands and social media, I can easily get an answer if I don’t know it. It’s a direct example of using a social computer (and since my coverage is on social computing) this is a good fit. Of course, finding the examples is just the start, often, I need to understand the case study, and that often involves a typical research phone interview.For both of the above use cases. additional follow-ups and analysis are required.
Traditional methodologies are core
Asking my blog readers and twitter followers to provide broad responses that represent mainstream North America, Europe, or Asia isn’t effective. Fortunately, Forrester has a great deal of Technographic data (we’ve large data teams) at my disposal around user behaviors and consumer preferences I don’t need to crowd source crowd source to get this, and rely on this for my data collection. I’m currently focused on a Forrester Wave of the online community platforms, this particular methodology does not call for any social media use I’m using a tried and true methodology, learn more about it on the Forrester site.
Sometimes, I engage in discussion on twitter that appeals to me personally: movies, music, events, art, sports, and occasionally politics. Being the curious individual I am, I’ve been known (before Forrester) to ask questions of my network and get responses if’s a fascinating use of social media. Although some may not see the separation, it’s important to note that these discussions often have nothing to do with my role as an analyst.
Summary:
I primarily rely on Forrester’s data, process, and methodologies that have been tested and refined over the last 25 years. This includes input from our large survey data, interviews, client meetings, vendor briefings, and talking with fellow analysis. When it makes sense, I can rely on my large network of Interactive Marketers and Social Computing experts to flesh out concepts and find examples although additional analysis is almost always required.
As I discover new ways to use these tools, I’ll update this post, I’d love to hear your responses
Jeremiah,
The two use cases for crowdsourcing that you mention are indeed useful. You get the value from online community, and you also provide value back by maintaining this blog and facilitating Twitter conversations.
With commercial research, the difference is that you sell it. As the community does not receive much value from it, it does not contribute as much as it could. Take the example of Innocentive – if Forrester were willing to pay for contributions, I’m sure you would get much more value out of crowdsourcing.
Net, I think crowdsourcing should be treated as an investment – the more you put into it, the more it pays off.
Cheers,
Darek
I agree Jeremiah – crowd-sourcing is brilliant for idea generation and collating information. And it is a two-way process – people can equally give and receive.
However, there will always be the requirement for someone like yourself to skilfully interepret the results. Taking a mean result of crowdsourcing in itself may produce lowest common denominator catch-all findings; but used as supporting or additional evidence for more analytical studies can be very powerful
You forget to mention one great application of crowdsourcing: Identify what research topics your audience want you to focus on.
I assume that based on the numbers of answers to your post and the passion in them you can identify what case studies, topics or analysis is more demanded in your research area.
Luis, Partner rSitez
Thanks Luis
As mentioned above, there are many inputs coming to me: clients (firms), vendors, VCs, press, fellow analysts, so yes, I watch the conversation carefully, to help determine where my research agenda is going to go.
One question I realize that needs to be answered, is that brands are unsure about how to find –and manage– social media services: agencies, firms, boutiques.
Simon, thanks. I’ve noticed a real lack in quality between most public wikis and what a vetted list on a blog would look like, so yes, a filter is needed.
Darek, thanks, an interesting point. There are brands and vendors that are really anxious to tell us their stories, getting feedback and input isn’t a challenge, the real challenge is making sense of it all for clients.
With that said, please scan my archives, see tags “web strategy, community marketing, social network, and industry index” I give away a lot of my knowledge back to the marketplace –at no charge. I learned early on you get as much as you give.
Good points all around, thanks.
I agree with you on the use cases you highlighted. One challenge as we all know with social media in general is the validation of the identity of the user participating in a community.The participants of the community is the key. But at the same time, how do you overcome the issue of a competitor particpating in a community to skew the analysis? As you rightly pointed out traditional methodologies are the core.
Suresh, competitors jump in the discussion all the time (online and offline), it’s part of my job to vet that out.
One of the best questions I ask vendors (and brands is) who’s your competition –when you ask this enough, I really learn the marketplace.
Jeremiah, indeed your blog is among my favourites – and that’s because you provide quality content to the community free of charge. One of the good points raised in this conversation is that the nature of your research makes crowdsourcing difficult, given that you assess vendors, who may want to contribute and bias the research results. Do you think that with some specific framework for crowdsourcing you could limit this bias and get more value out of it? Think about Amazon book recommendations, could a framework of such kind work in your context?
Darek
We analysts have to filter the bias in every type of interaction: online or in person, it’s simply not going to go away.
A true opensource model of research will allow networks of people to connect to rate and rank products. These networks however won’t be strangers, but folks you know, or those that have established relationships.
It’s very helpful Jeremiah to have the balance in the crowdsourcing discussion and I feel we all too often go off at a tangent and assume it is the holy grail for all product development challenges. I have blogged your post over at the crowdsourcing 101 @ http://www.youth-marketing-buzz.com/2008/10/buzz-words-crowdsourcing.html
Would Design by Humans or springleap.com be examples of crowdsourcing?
I agree Jeremiah – crowd-sourcing is brilliant for idea generation and collating information. And it is a two-way process – people can equally give and receive.
However, there will always be the requirement for someone like yourself to skilfully interepret the results. Taking a mean result of crowdsourcing in itself may produce lowest common denominator catch-all findings; but used as supporting or additional evidence for more analytical studies can be very powerful