Startup Wednesday: Think About All the Data

December 2, 2009

11:38 pm

One thing is certain these days, we all deal with a lot of data, as individuals and more importantly as companies.  As individuals, we are bombarded with data from RSS feeds, Twitter, Facebook, SMS, and so on.  It takes a lot of our energy to intake, process and retain all of this so we can, in the end, be able to utilize it to our advantage.  When you think about it, companies are not different, just a different scale.  So it is vital as a new startup to consider all your data requirements early on and design for the worst case scenario, so to speak.

Lets think about Twitter for just a few minutes.  Way back in the day when they started out, it was a small project and they most likely did not foresee the huge data demands they would run into in the near future.  The early adopters tweeted status updates, and each one went into its own special place in a database under that username … Then users grew, following was added to be allow people to weed out the noise.  All the while the amount of data Twitter began collecting and having to store and deal with was increasing in an exponential fashion.  Pretty soon it had become so bad that Twitter was literally unable to stay up under the weight of all of the data and went through a bit of a bad spell.  I am sure we all remember the “Fail Whales” right?  Anyway, it took a bit of money and a few terrific engineers to retool the system on the fly.  It didn’t knock them out of business, but it sure did cause a lot of hassle.  If however, they had envisioned from the beginning how their data demands would have grown, to a “worst case” scenario, they may have been able to design for it on the front end and avoid some of the pitfalls that they did face.  Now I understand none of us have a crystal ball, but if you are starting a company, you should aspire to be HUGE right ?  So take Twitter’s lesson and learn and plan accordingly.

In this light I wanted to share a video of a talk taken at a Startup2Startup (started by Dave McClure),

a group of Silicon Valley geeks, entrepreneurs, & investors dedicated to educating and helping the next generation of internet startups. We meet monthly over dinner to discuss relevant topics in technology & entrepreneurship, connect with new people & companies, and share our knowledge & experience.

Anyway, the video below is the first of a series of three from there last get together.  In this portion, Roger Magoulas of O’Reilly Media, talks Big Data.  Its a good topic, so listen and enjoy.

Did you like this article?

Get more delivered to your inbox just like it!

Sorry about that. Try these articles instead!

Jimmy Gardner is a Senior Unix/Windows Engineer, a lover of technology and photography. Follow him online at @jjgardner3 and enjoy his portfolio at

  • Shares

Leave a Reply

  • (will not be published)