Release highlights January 27th 2017

By Stefan Avivson on February 2nd, 2017

Pipes Debugging

Now you can debug your pipe executions both live – and after they’ve stopped. This makes pipes many times easier to debug when something fails and gives deep insight into how your pipes behave – and how far they are in processing.



Read more about pipes debugging by clicking here

Data set deduplication & combine

Data sets have also gotten a face lift with 2 new powerful options for deduplication and combining 1 data set with another.



Read more about deduplication & combine by clicking here


We’re also introducing a brand new robot type – we’ve called it an AutoBot.



AutoBots provides a mapping between any url and your Extractor robots – allowing you to interact with a single AutoBot that then forwards your request to potentially 100s or 1000s of extractor robots defined within. This feature allows you to perform web scraping on a professional level never seen before!

Read more about AutoBots by clicking here

Complex Data Types

We’ve given all field definition interfaces a face lift as well – and introduced the ability to have complex types all over. This means you can now have arrays and objects in your output from your robots – as complex as you’d like. We’ve also added some new step actions to the Extractor robot that further enables you to leverage this new feature.
Additionally we’ve made it possible to use data types as input and / or output for all robot types.



Read more about data types by clicking here

Facebook Pipe Actions

We’ve also launched a new set of pipe actions that allows you to query and interact with the Facebook Graph API – directly from ! We support querying places and events based on areas, and several lookups based on their internal Facebook ID’s. This is part of our continued efforts to bring more social data onto the platform – which now includes data from Foursquare, Facebook and Flickr.



Release highlights 13th October 2016

By Stefan Avivson on October 14th, 2016

Today’s release is the biggest in the history of! It includes new features we are really proud to announce – the first three in particular! Please read on…

Read more…

Web scraping or web data extraction?

By Jacob Laurvigen on September 20th, 2016

If you are a developer you might have noticed a change in wording surrounding what would normally be described as web scraping. This is now called web research or as we call it, web extraction. So why don’t we just call it web scraping? Depending on geographic location the understanding of web scraping and if it’s a good or bad thing ranges from “Web Scraping is a natural tool for data research” to “this is a gray zone”.


What web scraping really is

What web scraping really is, is to gather data from the web and the purpose for this could be just as different as why people read books, newspapers or any other source of knowledge. But what you can’t do is of course copying a text, image etc. and publish/present/re-sell this as your own work. Just like a journalist reads other news papers you would need to write your own version or simply use this newly acquired knowledge to navigate. What it really is, is applying robotics to a manual job as you already would be doing when you start your computer and open a browser to look for information. What the web scraping robots (or web extraction robots as we call it) do is to automate a string of events to save you or your organisation of e.g. writing all this information down manually or pressing F5 all day long to look for changes. So web scraping is helping you automate a manual and time-consuming jobs and yes and that IS a good thing!


Read more…

Don’t Be the Business with Rotten Teeth!

By Jacob Laurvigen on September 14th, 2016

My name is Jacob Laurvigen, and in my work as chief innovation officer at the virtual robot company, I get to work with some of the most innovative companies every single day. But it is not all joy, and in this article, I will try to cover some of the frustration that innovative and disrupting technology can cause for even the most well-managed businesses.

For most of us, discovering a new technology is a joyful moment, and if this technology can make your business perform better, it is an even more joyful moment. This has most certainly been the case with so-called Big Data technology, and in the forefront of the disruptive tool, we have external data (all the data connected to the World Wide Web). You can collect enormous amount of data and transform this data in your own personal data refinery that gives you a live datafeed with all the information you need on financial markets, your products, your competitors’ products and just about anything you can think of and put into an analytical process.
Read more…