Automating Big-Data Analysis and Replacing Human Intuition with Algorithms

A new and unique computer system from MIT has outperformed human intuition using its algorithms, and it’s amazing, and perhaps a little frightening: the Data Science Machine beat out over 600 human teams in finding predictive analysis.

Big-data analysis consists of searching for buried patterns that have some kind of predictive power.

But choosing which “features” of the data to analyze usually requires some human intuition.

In a database containing, say, the beginning and end dates of various sales promotions and weekly profits, the crucial data may not be the dates themselves but the spans between them, or not the total profits but the averages across those spans.

MIT researchers aim to take the human element out of big-data analysis, with a new system that not only searches for patterns but designs the feature set, too.

To test the first prototype of their system, they enrolled it in three data science competitions, in which it competed against human teams to find predictive patterns in unfamiliar data sets.

Of the 906 teams participating in the three competitions, the researchers’ “Data Science Machine” finished ahead of 615.

In two of the three competitions, the predictions made by the Data Science Machine were 94 percent and 96 percent as accurate as the winning submissions.

In the third, the figure was a more modest 87 percent. But where the teams of humans typically labored over their prediction algorithms for months, the Data Science Machine took somewhere between two and 12 hours to produce each of its entries.

Read more at Automating Big-Data Analysis and Replacing Human Intuition with Algorithms

Share your opinions with us in the comment box. Subscribe to get updates in your inbox.

Share on FacebookShare on Google+Share on LinkedInTweet about this on TwitterEmail this to someone

Is ETL Development doomed?

Is ETL Development doomed?

There seems to be a couple of tracks for this. First is the pure development automation tools, such as Varigence MIST. If you are technically minded, take a look at this product demo video – though I suggest skipping to about 25 minutes in to see the real meat as it does go on a bit. It looks mindbogglingly powerful but is clearly shooting at the ETL pro who wants to churn stuff out faster, more consistently and with less fiddling about. MIST is limited to SSIS/AS (for now) and I’m not sure how far it will go as it’s clearly aimed at the developer pro market, which is not always the big buyers. I expect to be playing with it more over the next few weeks on a live project so should be able to get a better view.

The second path appears to be more targeted at eliminating ETL developers in their entirety. AnalytixDS wraps up metadata import (i.e. you suck in your source and target metadata from the systems or ERWIN), do the mapping of fields and apply rules, then “push button make code”. Obviously there’s a bit more to it than that, but the less you care about your back end and the quality of your ETL code (cough Wherescape cough) the more likely this product will appeal to you. Say hello, business users, who are the big buyers (though I look forward to troubleshooting your non-scalable disasters in the near future).

Do you have any opinion? Share with us by leaving your comments below of send us a message.

Share on FacebookShare on Google+Share on LinkedInTweet about this on TwitterEmail this to someone

Five Data Mining Techniques That Help Create Business Value

Five Data Mining Techniques That Help Create Business Value

The term data mining first appeared in the 1990s while before that, statisticians used the terms “Data Fishing” or “Data Dredging” to refer to analysing data without an a-priori hypothesis. The most important objective of any data mining process is to find useful information that is easily understood in large data sets. There are a few important classes of tasks that are involved with data mining:

  1. Anomaly or Outlier detection
  2. Association rule learning
  3. Clustering analysis
  4. Classification analysis
  5. Regression analysis

Data mining can help organisations and scientists to find and select the most important and relevant information. This information can be used to create models that can help make predictions how people or systems will behave so you can anticipate on it. The more data you have the better the models will become that you can create using the data mining techniques, resulting in more business value for your organisation.

If you have any opinion about how data mining help to create business value, post it in the comment box. And contact us for discussion.

Share on FacebookShare on Google+Share on LinkedInTweet about this on TwitterEmail this to someone

Cloud Database for Dummies (Free E-Book)

Cloud Database for Dummies

Building a Database in the Cloud? E-Book Explains How.

It’s free for download as long as you have a free Oracle account.

Clear and concise, practical, filled with time-saving tips—the reviews are in on Building a Database Cloud for Dummies.

This quick-reference guide, organized into six short chapters and supplemented with helpful illustrations, provides a clear overview of the cloud and step-by-step instructions on deploying database as a service. Download the complimentary e-book today and learn how to:

  • Build a vision and business case for the cloud
  • Operate and deploy a database in the private cloud
  • Maximize success, with a list of the top 10 things to consider when starting

Cloud Database

Cloud database allow businesses to drive down costs while increasing business agility and IT performance. See for yourself how to save money, simplify management, and improve resource efficiency with a database cloud.

Share on FacebookShare on Google+Share on LinkedInTweet about this on TwitterEmail this to someone