Hadoop - Pig
PIG is a script used in Hadoop to fetch data and generate different varieties of result set. Here I would like to explain the basic use of PIG script. Lets consider one table called census_education which contains census education data for each county.
Concept here is slightly different from normal SQL. Filtering, grouping etc are not doing in one step instead multiple steps.
In the first statement LOAD program pulls the record from the table census_education which is highlighted in red and store the output result into a variable called a. USING keyword refer the table metadata from the HCatalog while pulling the records from the table.
Second statement pulls only required data for analysis and store it in another variable b. similarly third statement group the records and store it in another variable c.
Foreach iterate through the entire record in the output result of c and find the average for highly educated peoples for each city ( city is a dimension column in the output result of c).
finally we can push output data to table or other variable by using dump.
Fig 1 |
Concept here is slightly different from normal SQL. Filtering, grouping etc are not doing in one step instead multiple steps.
In the first statement LOAD program pulls the record from the table census_education which is highlighted in red and store the output result into a variable called a. USING keyword refer the table metadata from the HCatalog while pulling the records from the table.
Second statement pulls only required data for analysis and store it in another variable b. similarly third statement group the records and store it in another variable c.
Foreach iterate through the entire record in the output result of c and find the average for highly educated peoples for each city ( city is a dimension column in the output result of c).
finally we can push output data to table or other variable by using dump.
Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea.
ReplyDeleteSpoken English Classes in Chennai
Spoken English in Chennai
German Language Course in Chennai
IELTS Coaching centre in Chennai
French Language Classes in Chennai
Japanese Classes in Chennai
Japanese Language Classes in Chennai
Spoken English Classes in OMR
Spoken English Classes in Porur
Spoken English Classes in Adyar