Hadoop - how to learn

By -

Enterprise level data analysis and processing requirement is keep on changing day by day and it is proved that in order to make correct judgments better to deep dive into as much data as possible. Companies are adopting BIG DATA technologies to analysis the big volume of data that comes to their eco system. Some years back market analysts thought that this trend is purely due to peer pressure but later pioneers who implemented Hadoop (Open source big data frame work) started reaping the benefit. Most of the companies are adopting Hadoop framework for their data problems.


There are multiple technologies which can handle big volume of data such as massively parallel processing systems like green plum, Microsoft SQL Server MPP systems, Teradata etc.  Problem with these database is cost involved for implementation and maintenance. Hadoop is completely open source and can implement using in-expensive commodity servers. There are lot of players in the market who is specialized in their own version of Hadoop and provides support to implement such as Apache hadoop, Cloudera’s Distribution including Apache Hadoop, IBM distribution of Apache Hadoop, DataStax Brisk, Amazon Elastic MapReduce, and Horton works etc.


I choose to learn Horton works Hadoop distribution as this is the only one company which provides support for windows. Horton works and Microsoft have tie up to build BI technologies to work on Hadoop framework.


 Whoever interested to learn Hadoop can download the free sandbox from here http://hortonworks.com/products/hortonworks-sandbox/. Sandbox is a personal, portable Hadoop environment that comes with a dozen interactive Hadoop tutorials provided by Horton networks.
To install hortonworks sandbox, you need to install virtual box in your machine. I have installed oracle virtual VM VirtualBox managerto install this and you can find the same from here   http://www.oracle.com/us/technologies/virtualization/virtualbox/overview/index.html



Open the virtual machine as shown in Fig 1 and import the downloaded sandbox appliance which will start installing the hortonworks Hadoop!!

 
Fig 1















Enjoy learning…


Comments

Post a Comment

Popular posts from this blog

Microsoft BI Implementation - Cube back up and restore using XMLA command

By -
Image
Current BI implementation require to deploy the cube in multiple location to make it available  locally in each location to improve the performance. Cube development happening in one location say X but since the company is having presence in around the world and so business users sits in different locations X, Y and Z. Internet connectivity between X, Y and Z are not good and accessing the cube deployed in X from Y and Z is very slow and it impact the end user productivity. To overcome this issue, cube has to be deployed in Y and Z locally and upload the connection in their local SharePoint as mentioned below.  Fig 1 Now let see how we can automate all these process from taking back up of the cube database to restore it in each location. Follow the below mentioned steps to automate back up and restore of cube database. - select the back up option by right click on the cube database to be backed up and generate the XMLA script after providing necessary proper
Read more

Databricks - incorrect header check

By -
This post I would like to show you how we can fix the problem of " Incorrect header check " received while fetching the data from hive table. Actual message " SparkException: Job aborted due to stage failure: Task 0 in stage 63.0 failed 4 times, most recent failure: Lost task 0.3 in stage 63.0 (TID 3506, 10.1.1.7, executor 28): java.io.IOException: incorrect header check.   at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)     at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)     at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)     at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)     at java.io.InputStream.read(InputStream.java:101) " To better understand the scenario, let me explain how the data got loaded in the hive table. we are storing the data from source systems to RAW layer. H
Read more