Building Yarn and Hive on Spark - Edureka Blog

Building Yarn and Hive on Spark - Edureka Blog Hive Yarn Get Electrified By Spark Back Home Categories Online Courses Mock Interviews Webinars NEW Community Write for Us Categories Artificial Intelligence AI vs Machine Learning vs Deep LearningMachine Learning AlgorithmsArtificial Intelligence TutorialWhat is Deep LearningDeep Learning TutorialInstall TensorFlowDeep Learning with PythonBackpropagationTensorFlow TutorialConvolutional Neural Network TutorialVIEW ALL BI and Visualization What is TableauTableau TutorialTableau Interview QuestionsWhat is InformaticaInformatica Interview QuestionsPower BI TutorialPower BI Interview QuestionsOLTP vs OLAPQlikView TutorialAdvanced Excel Formulas TutorialVIEW ALL Big Data What is HadoopHadoop ArchitectureHadoop TutorialHadoop Interview QuestionsHadoop EcosystemData Science vs Big Data vs Data AnalyticsWhat is Big DataMapReduce TutorialPig TutorialSpark TutorialSpark Interview QuestionsBig Data TutorialHive TutorialVIEW ALL Blockchain Blockchain TutorialWhat is BlockchainHyperledger FabricWhat Is EthereumEthereum TutorialB lockchain ApplicationsSolidity TutorialBlockchain ProgrammingHow Blockchain WorksVIEW ALL Cloud Computing What is AWSAWS TutorialAWS CertificationAzure Interview QuestionsAzure TutorialWhat Is Cloud ComputingWhat Is SalesforceIoT TutorialSalesforce TutorialSalesforce Interview QuestionsVIEW ALL Cyber Security Cloud SecurityWhat is CryptographyNmap TutorialSQL Injection AttacksHow To Install Kali LinuxHow to become an Ethical Hacker?Footprinting in Ethical HackingNetwork Scanning for Ethical HackingARP SpoofingApplication SecurityVIEW ALL Data Science Python Pandas TutorialWhat is Machine LearningMachine Learning TutorialMachine Learning ProjectsMachine Learning Interview QuestionsWhat Is Data ScienceSAS TutorialR TutorialData Science ProjectsHow to become a data scientistData Science Interview QuestionsData Scientist SalaryVIEW ALL Data Warehousing and ETL What is Data WarehouseDimension Table in Data WarehousingData Warehousing Interview QuestionsData warehouse architectureTalend T utorialTalend ETL ToolTalend Interview QuestionsFact Table and its TypesInformatica TransformationsInformatica TutorialVIEW ALL Databases What is MySQLMySQL Data TypesSQL JoinsSQL Data TypesWhat is MongoDBMongoDB Interview QuestionsMySQL TutorialSQL Interview QuestionsSQL CommandsMySQL Interview QuestionsVIEW ALL DevOps What is DevOpsDevOps vs AgileDevOps ToolsDevOps TutorialHow To Become A DevOps EngineerDevOps Interview QuestionsWhat Is DockerDocker TutorialDocker Interview QuestionsWhat Is ChefWhat Is KubernetesKubernetes TutorialVIEW ALL Front End Web Development What is JavaScript â€" All You Need To Know About JavaScriptJavaScript TutorialJavaScript Interview QuestionsJavaScript FrameworksAngular TutorialAngular Interview QuestionsWhat is REST API?React TutorialReact vs AngularjQuery TutorialNode TutorialReact Interview QuestionsVIEW ALL Mobile Development Android TutorialAndroid Interview QuestionsAndroid ArchitectureAndroid SQLite DatabaseProgramming Yarn Get Electrifie... Comprehensive HIVE (4 Blogs) Become a Certified Professional Hive Yarn Get Electrified By Spark Last updated on Apr 22,2020 9K Views Awanish Awanish is a Sr. Research Analyst at Edureka. He has rich expertise... Awanish is a Sr. Research Analyst at Edureka. He has rich expertise in Big Data technologies like Hadoop, Spark, Storm, Kafka, Flink. Awanish also...1 Comments Bookmark Become a Certified Professional In this blog, let us see how to build Spark for a specific Hadoop version.We will also learn how to build Spark with HIVE and YARN. Considering that you have Hadoop, jdk, mvn and git pre-installed and pre-configured on your system.Open Mozilla browser and Download Spark using below link.https://edureka.wistia.com/medias/k14eamzaza/Open terminal.Command: tar -xvf Downloads/spark-1.1.1.tgzCommand: lsOpen spark-1.1.1 directory.You can open pom.xml file. This file gives you the information about all the dependencies you need.Do not edit it to stay out of trouble. Command: c d spark-1.1.1/Command: sudo gedit sbt/sbt-launch-lib.bashEdit the file as below snapshot, save it and close it.We are reducing the memory to avoid object heap space issue as mentioned in below snapshot.Now, run the below command in the terminal to build spark for Hadoop 2.2.0 with HIVE and YARN.Command: ./sbt/sbt -Pyarn -Phive -Phadoop-2.2 -Dhadoop.version=2.2.0 -DskipTests assemblyNote: My Hadoop version is 2.2.0, you can change it according to your Hadoop version.For other Hadoop versions# Apache Hadoop 2.0.5-alpha-Dhadoop.version=2.0.5-alpha # Cloudera CDH 4.2.0-Dhadoop.version=2.0.0-cdh4.2.0 # Apache Hadoop 0.23.x-Phadoop-0.23 -Dhadoop.version=0.23.7 # Apache Hadoop 2.3.X-Phadoop-2.3 -Dhadoop.version=2.3.0 # Apache Hadoop 2.4.X-Phadoop-2.4 -Dhadoop.version=2.4.0It will take some time for compiling and packaging, please wait till it completes.Two jars spark-assembly-1.1.1-hadoop2.2.0.jar and spark-examples-1.1.1-hadoop2.2.0.jar getscreated.Path of spark-assembly-1.1.1-hadoop2.2.0 .jar : /home/edureka/spark-1.1.1/assembly/target/scala-2.10/spark-assembly-1.1.1-hadoop2.2.0.jarPath of spark-examples-1.1.1-hadoop2.2.0.jar : /home/edureka/spark-1.1.1/examples/target/scala-2.10/spark-examples-1.1.1-hadoop2.2.0.jarCongratulations, you have successfully built Spark for Hive Yarn.Got a question for us? Please mention them in the comments section and we will get back to you.Related Posts:Get Started with Apache SparkApache Spark Lighting up the Big Data WorldApache Spark EcosystemApache Spark with Hadoop-Why it matters?Start your Training in Apache Spark Scala Today.Recommended videos for you Secure Your Hadoop Cluster With Kerberos Watch Now Ways to Succeed with Hadoop in 2015 Watch Now Hadoop for Java Professionals Watch Now Big Data Processing with Spark and Scala Watch Now Pig Tutorial Know Everything About Apache Pig Script Watch Now Filtering on HBase Using MapReduce Filtering Pattern Watch Now Spark SQL | Apache Spark Watch Now Python for Big Data Analytics Watch Now What is Big Data and Why Learn Hadoop!!! Watch Now Apache Spark Will Replace Hadoop ! Know Why Watch Now Top Hadoop Interview Questions and Answers Ace Your Interview Watch Now Introduction to Apache Solr-1 Watch Now Apache Spark Redefining Big Data Processing Watch Now 5 Things One Must Know About Spark Watch Now Hadoop Tutorial A Complete Tutorial For Hadoop Watch Now MapReduce Tutorial All You Need To Know About MapReduce Watch Now MapReduce Design Patterns Application of Join Pattern Watch Now Hive Tutorial Understanding Hive In Depth Watch Now What Is Hadoop All You Need To Know About Hadoop Watch Now Is Hadoop A Necessity For Data Science? Watch NowRecommended blogs for you Operators in Apache Pig: Part 1- Relational Operators Read Article Using Big Data to Boost Telecom’s Marketing Capabilities Read Article What are the Best books for Hadoop? Read Article Jobs In Hadoop Read Article Hadoop Developer-Job Responsibilities Skills Read Article A Beginners Guide to Understanding Big Data Hadoop Read Article What is Hadoop? Introduction to Big Data Hadoop Read Article Top Big Data Technologies that you Need to know Read Article How to Set Up Hadoop Cluster with HDFS High Availability Read Article MapReduce Example: Reduce Side Join in Hadoop MapReduce Read Article Map Side Join Vs. Join Read Article A Day In The Life Of A Hadoop Administrator Read Article Why You Should Choose Python For Big Data Read Article Setting Up A Multi Node Cluster In Hadoop 2.X Read Article ELK Stack Tutorial Discover, Analyze And Visualize Your Data Efficiently Read Article Real Time Storm Project Read Article NameNode High Availability with Quorum Journal Manager Read Article Apache Spark with Hadoop Why it Matters? Read Article Applying Hadoop with Data Science Read Article Hadoop Certification Become a Certified Big Data Hadoop Professional Read Article Comments 1 Comment Trending Courses in Big Data Big Data Hadoop Certification Training158k Enrolled LearnersWeekend/WeekdayLive Class Reviews 5 (62950)