October 20, 2015
Are commenters on specialized forums like Hacker News good at predicting the success of startups?
Developers use Hacker News as a place to post betas of their products, or stories about newly released products, so that the expert community can weigh in with their criticisms, questions and predictions. Many developers end up stressing over the comments they receive–but should they?
To find out, Venngage did a Hacker News study investigating the kinds of responses startups receive when they post their products and services on Hacker News. The study used a sample of 10 original HN threads for startups that have since become successful: Dropbox, Airbnb, Codeacademy, Instacart, Quora, Meteor, Zenefits, Stripe, Heroku, and the now deceased Homejoy (which scaled quickly but ultimately failed to turn a profit).
A natural language algorithm (Jacob Perkin’s Sentiment Analysis with Python NLTK) was used to categorize the first 100 comments in each thread automatically as either Positive, Negative or Neutral. Then, in order to obtain a smaller and more concentrated sample for analysis, the top comments for each thread were categorized manually.
The results showed that the responses on Hacker News tended towards negativity, with the top comment in each thread reflecting the general sentiments of the rest of the comments (with the exception of Homejoy and Dropbox). About half the comments were negative, a third positive and the rest were neutral or tangential. While there seemed to be a crowd bias towards more tech-heavy businesses like Codeacademy and Dropbox, while businesses geared towards a more general market like Airbnb, Quora and Instacart were received more negatively. Also, businesses that posed a perceived threat towards user security, like Quora, which required a Facebook login to be used, were picked apart by commenters.
Ultimately, the Hacker News crowd was shown to be no better at predicting the success of a business than a larger and more general crowd, despite their specialized knowledge. In fact, in many cases, their tendency to focus on the nit-picky details seemed to distract commenters from the actual business model. And certain things that Hacker News commenters found annoying, like the restricting structure of Meteor, ended up being one of its assets for a more general userbase.
Here is an infographic of the study’s findings.
Did you like this article?
Get more delivered to your inbox just like it!