Big Data Solutions In Action

Creating the Digital Brain


Creating the Digital Brain


• FEATURES • By Kaushik Das




Oil spills from mining accidents can cost tens of billions per incident. The famed BP Oil spill in the Gulf cost $40 billion alone for the company, never mind the uncalculated cost of the region impacted.

How could this type of economic and environmental disaster be avoided? Our answer is smart systems. And they are not just a pipe dream, the Pivotal Data Science team has been working hard over the past few years to provide real, practical solutions to answer this type of issue for the oil and gas industry, as well as others.

The idea is to instrument the drilling rig to be a smart system. Drawing on concepts of the Internet of Things (IoT), where any machine from personal wearables, like smart phones and Fitbits, to industrial equipment like jet engines, power turbines and drilling rigs can be constructed as a system of sensors and actuators, essentially creating the the sense organs and limbs of the smart system.  Put these together with the remarkable advancements in big data technologies which enable us to pull petabytes of data into a Data Lake that we can use as a basis to employ complex machine learning models efficiently, and we have the basic two ingredients of a smart system that could monitor and prevent accidents, downtime and even ensure energy efficiency.

Digital Brain = Data Lake + Data Science

How will a smart offshore oil platform work?  Let’s look at the three elements—the sensors, the brain, and the actuators.

The sensors in a drilling rigs measure temperature, pressure, and Monitoring While Drilling (MWD) variables, which can include seismic, gamma ray and high frequency electromagnetic data, as well as hydraulic and mechanical variables.  To create the digital brain, we load all of this data along with measurements made off the drill like those of drilling fluid properties and previous seismic data into a Data Lake.  Here, the all the data is stored together and we can extract patterns in the data.  Since the data for one oilfield involving multiple boreholes can run into hundreds of variables and billions of rows we use a parallel modeling package like MADlib to extract patterns from the data in an efficient manner.  This involves clustering the data and then regressing over the rate of penetration of the drill.

Once we have a good model, we can operationalize it by checking the actual rate against the predicted rate.  If the predicted rate is different, we flag it as an anomaly.  We also create a library of anomalies and label them.  Armed with that dataset, we are in a position to monitor the drilling and take appropriate action if we detect an anomaly.  For instance, if the anomaly is associated with a blowout, we can set the brain to stop drilling by activating the actuators (the control system) and send a red alert to the control room to initiate a response team to investigate.  They may find that this anomaly is just be an indication that the drill bit is wearing out earlier than anticipated and needs to be changed.  Or, importantly, they may discover a serious threat and be able to stop a catastrophe like the BP oil spill.

In the case of the former, this has an added bonus of helping us to do predictive maintenance, and improve the productivity of our operations.  It is an example of the application of Data Science methodology on the appropriate technology.

The Technology Behind the Science

Here at Pivotal we have been making great strides in building the platform that houses the Data Lake.  This includes a parallel storage system based on HDFS with parallel database (namely, HAWQ andGreenplum) and in-memory (Gemfire) modules with a variety of other components that make it easy to ingest and store data, compute on it, and take action.  The whole platform is based on open standards, which makes it highly compatible with a lot of third-party software and effectively future-proofs it.


Other Applications for the Digital Brain

Another example of making a system smart would be putting a digital brain into a smart grid.  Right now we have smart meters collecting an enormous amount of data regarding power usage from every business and household.  This maps the entire range of activity in any city!  But how do we get value from that?

The answer is once again to load this data into a Data Lake and look at the frequency content of the time series signals we get from every smart meter.  This enables us to cluster every meter and find out outliers or anomalies.  Then again, as the system tracks the changing of the behavior of a cluster or even individual smart meters, we can identify anomalies as they happen. Over time, we can train our model and label these anomalies as meter malfunctions, meter tampering or vegetation management (a tree or a branch falling on a power line).  Now we have created a Smart Grid than can be our eyes and ears everywhere across the power grid, and can take the appropriate action to prevent downtime and at the same time achieve optimal performance.

Download the Pivotal Data Science Lab datasheet.

The possibilities of the digital brain to transform industry are manifold. We are working steadily to apply this methodology widely, including another industry we are talking about this week at Strata—smart cities and the connected car .

The true potential of the IoT will be realized when we are able to create digital brains and transform the IOT from just things to a self-aware systems.  This will never eliminate the human element, rather it will make human intervention more effective and reduce delays and scope of error in action.

Jeff Immelt of GE has said that “zero unplanned downtime” is a key goal for GE’s use of the Industrial Internet.  But we can take this even further—what about zero unplanned outages, zero industrial accidents and zero environmental disasters?

The opportunity is right here, let us all make it happen!

Ten Hot Big Data Trends

Ten Hot Big Data Trends



As you enter the world of big data, you’ll need to absorb many new types of database and data-management technologies. Here are the top-ten big data trends:

  • Hadoop is becoming the underpinning for distributed big data management. Hadoop is a distributed file system that can be used in conjunction with MapReduce to process and analyze massive amounts of data, enabling the big data trend. Hadoop will be tightly integrated into data warehousing technologies so that structured and unstructured data can be integrated more effectively.
  • Big data makes it possible to leverage data from sensors to change business outcomes.More and more businesses are using highly sophisticated sensors on the equipment that runs their operations. New innovations in big data technology are making it possible to analyze all this data to get advanced notification of problems that can be fixed to protect the business.
  • Big data can help a business initiative become a real-time action to increase revenue.Companies in markets such as retail are using real-time streaming data analytics to keep track of customer actions and offer incentives to increase revenue per customer.
  • Big data can be integrated with historical data warehouses to transform planning. Big data can provide a company with a better understanding of massive amounts of data about their business. This information about the current state of the business can be combined with historical data to get a full view of the context for business change.
  • Big data can change the way diseases are managed by adding predictive analytics.Increasingly, healthcare practitioners are looking to big data solutions to gain insights into disease by compare symptoms and test results to databases of results from hundreds of thousands of other cases. This allows practitioners to more quickly predict outcomes and save lives.
  • Cloud computing will transform the way that data will be managed in the future. Cloud computing is invaluable as a tool to support the expansion of big data. Increasingly, cloud services that are optimized for data will mean that many more services and delivery models will make big data more practical for companies of all sizes.
  • Security and governance will be the difference between success and failure of businesses leveraging big data. Big data can be a huge benefit, but it isn’t risk-free. Companies will discover that if they are not careful, it is possible to expose private information through big data analysis. Companies need to balance the need to analyze results with best practices for security and governance.
  • Veracity, or truthfulness, of big data will become the most important issue for the coming year. Many companies can get carried away with the ability to analyze massive amounts of data and get back compelling results that predict business outcomes. Therefore, companies will find that the truthfulness of the data must become a top priority or decision making will suffer.
  • As big data moves out of the experimental stage, more packaged offerings will be developed.Most big data projects initiated over the past few years have been experimental. Companies are cautiously working with new tools and technology. Now big data is about to enter the mainstream. Lots of packaged big data offerings will flood the market.
  • Use cases and new innovative ways to apply big data will explode. Early successes with big data in different industries such as manufacturing, retail, and healthcare will lead to many more industries looking at ways to leverage massive amounts of data to transform their industries.

Ten Hot Big Data Trends

IBM, Apple forge enterprise app pact

IBM, Apple forge enterprise app pact: Watson, meet iPad

Summary: Apple gets a big leg up in the enterprise courtesy of IBM’s vast army. IBM gets to show off its analytics and industry specific apps running exclusively on iOS.

Larry Dignan
ginni and tim
It’s safe to say Apple gets the enterprise (and the profits involved) now. Ginni Rometty and Tim Cook create a win-win pact.

IBM and Apple said they have forged an enterprise pact where the two companies will collaborate on exclusive industry-specific applications built on iOS.

Apple maintains enterprise dominance; Windows Phone lags

IBM rolls out MobileFirst, eyes role as enterprise mobility enabler

Apple boasts enterprise sweet spot for the iPad

The deal makes sense on many fronts. First, industry-specific apps will lock down Apple’s iOS market share in the enterprise. Apple’s iOS market share vs. Android in the enterprise is the inverse of the consumer space. IBM gets to package iOS apps, embed its analytics tools, and then use its services and channel to sprinkle the apps into corporations.

And here’s another win-win: Apple gets a key enterprise partner without having to exclusively build and market to corporations. IBM gets Apple’s cool factor. In other words, consumerization will only go so far for Apple’s enterprise ambitions. Apple CEO Tim Cook gets the enterprise and is an IBM alum.


The details of the deal—dubbed IBM MobileFirst for iOS—break down like this:

  • Apple and IBM will create more than 100 vertical-focused enterprise apps built only for the iPhone and iPad. Target markets include retail, healthcare, banking, travel and transportation, telecommunications and insurance starting in the fall.
  • IBM’s cloud services such as device management, security and analytics will be optimized for iOS. Private app catalogs and productivity suites will be available. Services will be available on IBM’s Bluemix development platform.
  • AppleCare will be tailored for enterprise deployments with support on-site via IBM.
  • There’s a commitment to use IBM’s Fiberlink MaaS360 for mobile device management.
  • Apple is standardizing on IBM’s analytics and big data apps.
  • IBM will package device activation, supply and management for the iOS partnership. IBM will also sell industry-focused iPhones and iPads as a bundle.
  • Big Blue’s 100,000 consultants will push Apple wares in the field.
  • And finally, IBM’s financing arm will be in on the deal.

Cook said:

“We’re putting IBM’s renowned big data analytics at iOS users’ fingertips, which opens up a large market opportunity for Apple. This is a radical step for enterprise and something that only Apple and IBM can deliver.”

The market opportunity reference is critical. Apple has been knocked for lack of an iTV or iWatch (at least for now), but if it mines the enterprise better it’ll keep the cash cow going for years.

On the IBM side of the equation, IBM CEO Ginni Rometty said the alliance will transform “the way people work, industries operate and companies perform.”

Bottom line: The pact between IBM and Apple give both parties credibility and likely sales wins.

Forrester analyst Frank Gillett cheered the deal:

The Apple IBM partnership is a landmark agreement. Given IBM’s market strength and coverage, this partnership gives Apple enterprise capabilities and credibility at one stroke — and gives IBM a premium advantage in the race for mobile enterprise leadership. Look for Google and leading enterprise suppliers to seek partnerships that offer a credible alternative.

Winners and losers

IBM buys virtual assistant maker Cognea to give Watson personality from ‘suit and tie to kid next door’

IBM aims to speed up enterprise app development

Clearly, Apple is the biggest winner of the bunch, but IBM also gets its device management software into the flow. IBM has been investing heavily in mobility, specifically mobile commerce. Apple gives IBM consumerization cred.

And now for those losers:

  1. Android. Android has a ragtag band of partners in the enterprise, an operating system that has taken security knocks, and multiple versions that make the platform hard to manage. There’s a reason iOS leads in enterprise market share. Gillett’s point that Google will need partnerships is well taken. The problem is that it’s going to be hard to match IBM’s coverage in one stroke.
  2. Samsung. Samsung’s business-to-business unit has been the biggest champion of Android in the enterprise. An IBM exclusive with iOS basically locks out Android in the industry-specific application department.
  3. SAP and Oracle. Both enterprise software giants have been pushing their apps in corporations with a focus on industries they dominate. For companies thinking mobile first, IBM just plowed its way into the conversation.
  4. Microsoft. The software giant’s biggest play was to get Windows shops — and there are a ton of them in the enterprise — to go with Microsoft on the mobile front too. IBM and iOS will derail those plans somewhat, but not entirely. Microsoft’s mobile device management and collaboration platform will be strong.
  5. BlackBerry. BlackBerry is caught in the middle of an iOS and Android enterprise war. That position is going to hurt.


Topics: Enterprise SoftwareAppleIBMMobility

[ 2013] Open Source Backup with Bareos: Architecture, Features and Bacula comparison — Maik Aussendorf » Bareos is a reliable network open source software to backup, archive and restore files from all ma… 2013 — Christophe Villeneuve » Le projet “The sharebox” est un fork de la “piratebox”, dont son but est le partage de …

ref. 635 websphere mq administrator Fecha: 26 de julio de 2011 newx seleccionara, para importante multinacional …

This is the sixth of seven tutorials designed to help you get started with Disaster Recovery. This tutorial covers failing back from the cloud to your vSpher…
Video Rating: 5 / 5

Webinar: Get Your Hadoop On – Leveraging Big Data with Informatica

With the rise of social computing, sensor data, mobile computing and geospatial, organizations are faced with ever-increasing volumes and new types of data. …
Video Rating: 0 / 5

Ash Kulkarni, general manager, Enterprise Data Integration, Informatica, discusses how Informatica 9.1 is helping customers tackle Big Data challenges. Liste…

RabbitMQ with EasyNetQ Tutorial

A quick introduction to RabbitMQ using the EasyNetQ client library, this video has an accompanying written tutorial on my blog:…
Video Rating: 5 / 5

Recorded at SpringOne2GX 2013 in Santa Clara, CA. Speaker: Durai Arasan ETRADE has revamped their SOA platform with RabbitMQ as a core messaging system and S…
Video Rating: 5 / 5