Big Data…the most renowned term in the IT industry these days. What is it? Whats in it for a software professional? Have a glance here!This Article originally published at Srihari’s Knowledge Library
Gartner defines this word as “Big data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.”
Confused? Well, lets understand it.
High volumes of Data
Big data is a science of information processing when the data is in huge volumes like some peta or exa bytes. (1 petabyte = 1024 terabytes and 1 exabyte = 1024 petabytes). In other words, one exabyte is equal to a 100 crore GB of data.
As an individual we may know only about terabytes (TB) of data. At an enterprise level data sizes vary from petabytes to exabytes. Wondering which enterprise generates this much data? Think about facebook. Millions of people keep posting into their timelines at any point of time at a volume of single comment to an album upload of their personal event. Combining all this data will give a tons of load to facebook servers. Why should facebook worry? Keep the huge size clusters available to store all this data. No, its not just about storing data. Facebook has to analyze this data, find the trends like what people are talking about, what is the most discussed news/ event, etc. It is required to take the business decisions at organization level based on such analysis. If you think about the remote sensing data, genome code etc, the situation is still more complex.
DBMS/ RDMBS Systems
Since the existing data solutions can process the structured/ organised information, there is a need of new techniques to process unstructured data. Also process of data in DBMS/ RDBMS happens only when the complete set of data is available. In this competitive world, business decisions to be taken so quickly which requires the processing of data while it is flowing in.
Hence big data works on huge sets of structured/ unstructured data, while it is flowing in, if required and process it and provides the analysis and/ or reports as required. Different phases of data processing in Big Data includes capture (collecting the data), curation (extract the important data and preserve it), storage (save data on to the disk/ cluster), search (finding the required patterns), sharing and analysis (generate the data trends).
Some big facts …
- 90% of data in the world is unstructured
- Human genome code took 10 years to process. Using Big data, it can be processed in one week
- Data production in 2009 would increase to 4400% by 2020
In a day,
- The largest telescope that is expected to be operational by 2024 would gather 14 exabytes of data
- Surveillance cameras around the globe collects 413 petabytes of video data
- 46 petabytes of data flows through AT&T’s networks
- Google processes 24 petabytes
- Walmart generates 2.5 petabytes of data
- Facebook processes 0.5 petabytes
Big Data Technical Frameworks
There are many big data frameworks available and Hadoop is the one and most famous framework. It is an open source framework and widely used in, across different products and enterprise editions of different companies. Other frameworks include Spark, Storm, HPCC etc.
Programming Languages used in Big Data
One of the widely used programming language for data processing is R. NoSQL and Python are also playing a key role as data processing languages.
Most of the global IT giants utilized this opportunity and converted the Big Data science into solutions to fulfill the needs of different organizations based on their data analysis model. Few of such solutions are:
- Google BigQuery
- SAP Hana
- IBM Infosphere
- Microsoft Azure
- Oracle Big Data Appliance
- EMC Alpine, Pivotal
- HP Vertica, HAVEn
and many more!
Big Data Job Opportunities
Being an emerging technology, big data has lot of scope to create lot of job opportunities in IT space. The roles include Data Engineer, Data Scientist, Data Analytics Engineer, Data Architect, Data Warehouse Analyst, Business Intelligence Analyst, Data Modeler etc.
Big Data Certifications
Here is the list of top 10 certifications from the best institutions:
- Certified Analytics Professional – The Institute for Operations Research and the Management Sciences (INFORMS)
- Certification of Professional Achievement in Data Sciences – Columbia University
- Certificate in Engineering Excellence Big Data Anlytics and Optimization – International School of Engineering (INSOFE)
- Mining Massive Data Sets Graduate Certificate – Stanford University
- Certificate in Analytics: Optimizing Big Data – University of Delaware
- EMC Data Scientist Associate (EMCDSA) – EMC
- Cloudera Certified Professional : Data Scientist – Cloudera Inc
- Cloudera Certified Developer for Apache Hadoop – Cloudera Inc
- Cloudera Certified Administrator for Apache Hadoop
- Revolution R Enterprise Professional – Revolution Analytics
1. HP Vertica Certification
2. Oracle Business Intelligence Foundation Suite 11g Certified Implementation Specialist
3. SAS Certified Predictive Modeler using SAS Enterprise Miner 7 Credential
For more info you can visit…
Big Data Wiki
R Data Processing Language