Source: http://priceonomics.com/introducing-the-...

Many readers of this blog might have no idea how we make money. The short answer is that we sell data to companies. We also write books that many of our awesome readers purchase. But our biggest source of revenue comes from companies who hire us to crawl data for them. Unlike other sites that have a strong content-focus, our philosophy is to make money by selling things, mostly to businesses.

Tech companies and hedge funds pay us between $2K to $10K per month to crawl web pages, structure the information, and then deliver it to them in analyzed form. This is a pretty significant amount of money because acquiring data is a burning problem for some companies.

Why is web crawling such a pain point? Well, unless it’s your primary focus, you’re constantly re-inventing the wheel: setting up a backend infrastructure, figuring out how to browse this particular web page and encode it properly, figuring out how to queue and rate-limit crawl tasks, how parse the information, and how to keep things running when they break a few months later when the website updates.

Even we were reinventing the wheel almost every time for our first few customers. But we figured if we did this enough times, not only would we get better at it, but we could build the product we wished existed for crawling and analyzing web data. And that’s exactly what we’ve been working on.

Today, we’re introducing the Priceonomics Analysis Engine, a tool that helps you crawl and analyze web data both at scale, or just one page at a time. It’s still what you would call a “minimum viable product”, in skeletal form, but we wanted to release it to get feedback from developers and anyone who might want to use it.