Hadoop/bigdata Developer Resume
Dallas, TX
SUMMARY
- Over 6+ years of IT experience as a Developer, Designer & quality reviewer with cross platform integration experience using Hadoop, Java and J2EE.
- Good Understanding of the Hadoop Distributed File System and Hadoop Ecosystem.
- Hands on experience in installing, configuring and using Hadoop ecosystem components such as MapReduce (MRv1), YARN (MRv2), HIVE, PIG, SQOOP, FLUME and OOZIE.
- Hands on experience on Hortonworks and Cloudera Hadoop environments.
- Experience inHadoopadministration activities such as installation and configuration using Hortonworks and Cloudera Distributions.
- Good knowledge on Hadoop Cluster architecture and monitoring the cluster.
- Strong understanding of Hadoop Daemons and MapReduce concepts.
- Worked in handling Hadoop Ecosystem Projects such as Hive, Pig, HBase and Sqoop.
- Hands on experience in importing and exporting data using Sqoop from Relational Database Systems like Oracle, MySQL, Teradata to HDFS and vice versa.
- Worked in developing UDFs for HIVE and PIG using Java.
- Strong understanding of NoSQL databases like HBase, MongoDB and Cassandra.
- Hands on experience in creating integration between Hive and HBase.
- Hands on experience in loading semi structured and unstructured data into HDFS using Flume or Kafka.
- Handling different file formats on Parquet, ORC, Avro, Sequence file and flat text files.
- Strong understanding of other distributed execution engines like TEZ and Spark. Hands on experience in developing Spark jobs using Spark Core, Spark SQL and Spark Streaming.
- Used Spark Streaming and also have good knowledge of Storm and its integration with Kafka.
- Experience in Amazon Web Services (AWS) like EMR, S3 and EC2 services.
- Experience working with cluster monitoring tools like Nagios, Ambari and Cloudera Manager.
- Experience in analyzing, designing and developing ETL strategies and processes, writing ETL specifications using Talend.
- Extensively used Informatica Power Center for Extraction, Transformation and Loading.
- Hands on experience in DevOps tools like Maven, Git/GitHub and Jenkins.
- More than 3 years of experience in JAVA, J2EE, Web Services, SOAP, HTML and XML related technologies demonstrating strong analytical and problem solving skills, computer proficiency and ability to follow through with projects from inception to completion.
- Extensive experience working in Oracle, DB2, SQL Server and My SQL database and Java Core concepts like OOPS, Multithreading, Collections and IO.
- Hands on JAXWS, JSP, Servlets, Struts, Web Logic, Web Sphere, Hibernate, Spring, JBoss, JDBC, RMI, Java Script, Ajax, jQuery, Linux, Unix, WSDL, XML, HTML, AWS and Scala and Vertica.
- Developed applications using Java, RDBMS, and Linux shell scripting.
- Experience in complete project life cycle of Client Server and Web applications.
- Have good interpersonal, communicational skills, strong problem solving skills, explore/adopt to new technologies with ease and a good team member.
TECHNICAL SKILLS
Hadoop/BigData Technologies: Hadoop (Hortonworks, Cloudera) HDFS, Map Reduce, Pig, Hive, HBase, Zookeeper, Oozie, Sqoop, Flume, Impala, Spark, Storm, Kafka, Cassandra, MongoDB, Solr, Talend, Tableau.
Cloud Services: Amazon Web Services (AWS)
Languages/Scripts: Java JDK1.4/1.5/1.6 (JDK 5/JDK 6), C/C++, Python, Scala, HTML, SQL, PL/SQL
Frameworks: Hibernate 2.x/3.x, Spring 2.x/3.x, Struts 1.x/2.x
Web Services: WSDL, SOAP, REST, Jersey
Client Technologies: JQUERY, Java Script, AJAX, CSS, HTML 5, XHTML
Operating Systems: UNIX, Windows, LINUX
Application Servers: IBM Web sphere, Apache Tomcat, Web Logic
Web Technologies: JSP, Servlets, JDBC, Java Beans, JavaScript
Databases: Greenplum, Oracle 8i/9i/10g, Netezza, MySQL 4.x/5.x, Teradata
IDE Tools: Eclipse 3.x, IBM Web Sphere Application Developer, IBM RAD 7.0, IntelliJ
Build/CI Tools: ANT, Maven, Jenkins
Methodologies: Waterfall, Agile/Scrum
Version Control: SVN, CVS, Git/GitHub
Development Tools: TOAD, SQL Developer, SOAP UI, Putty
PROFESSIONAL EXPERIENCE
Confidential, Dallas, TX
Hadoop/Bigdata Developer
Responsibilities:
- Worked on Big Data Hadoop cluster implementation and data integration in developing large - scale system software.
- Assessed existing EDW technologies and methods to ensure our EDW/BI architecture meet the needs of the business, enterprise and allows for business growth.
- Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems like Oracle and vice-versa.
- Hands on collecting and aggregating log files and copy them to HDFS using Flume for further analysis.
- Developed MapReduce programs to parse the raw data, populate staging tables and store the refined data in partitioned tables in the EDW.
- Developed MapReduce jobs to convert data files into Parquet file format and included MRUnit to test the correctness of MapReduce programs.
- Created Hive queries that helped market analysts spot emerging trends by comparing fresh data with EDW reference tables and historical metrics.
- Developed business specific custom UDF's in Hive and Pig.
- Optimized MapReduce codes, Pig scripts, Hive queries and involved in performance tuning and analysis.
- Developed workflows in Oozie to automate the tasks of extraction of data from warehouses and weblogs into HDFS and pre-processing with Pig.
- Created HBase tables to load large sets of structures, semi-structured and unstructured data coming from NoSQL and a variety of portfolios.
- Experience in creating integration between Hive and HBase.
- Worked with cloud services like Amazon EC2 and Amazon S3 and involved in ETL, Data Integration and Migration using Talend Open Studio.
- Hands on experience in creating RDD’s using Apache Spark, Spark Streaming and applying operations like Transformations and Actions on them.
- Developed Spark jobs written in Scala to perform operations like data aggregation, data processing and data analysis.
- Developed a data pipeline using Apache Kafka and Spark Streaming to run analytics on the web log files.
- Used Spark API over Hortonworks Hadoop YARN to perform analytics on Hive data.
- Implemented a POC with Spark SQL to interpret complex JSON records. Used Cassandra as a storage for Spark analytics.
- Experience in collecting metrics for Hadoop clusters using Ambari.
- Experience working with off-shore teams and communicating daily status on issues, road-blocks.
Environment: Hadoop, Ambari, Centos Linux, HDFS, MapReduce, Kafka, Spark, Spark Streaming, Spark SQL, Parquet, Pig, Hive, Sqoop, Oracle, Oozie, Cassandra, Talend, GitHub.
Confidential, CA
Hadoop Developer
Responsibilities:
- Installed, configured and maintained Apache Hadoop clusters for application development and Hadoop tools like Hive, Pig, HBase, Zookeeper and Sqoop.
- Handled importing of data from various data sources using Sqoop, performed transformations using Hive, MapReduce and loaded data into HDFS.
- Developed Pig Latin scripts to extract and filter relevant data from the web server output files to load into HDFS.
- Optimized MapReduce jobs to use HDFS efficiently by using various compression mechanisms.
- Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior.
- Configured MySQL database to store Hive metadata.
- Implemented business logic by writing UDFs in Java and used various UDFs from Piggybanks and other sources.
- Used Hive to analyze the partitioned and bucketed data and to compute various metrics for reporting.
- Installed Oozie workflow engine to run multiple Hive and Pig jobs.
- Hands on experience in writing Hive UDFs to handle different Avro schemas.
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
- Developed Java RESTful web services to upload data from local to Amazon S3, listing S3 objects and file manipulation operations.
- Successfully ran all Hadoop MapReduce programs on Amazon Elastic MapReduce framework by using Amazon S3 for Input and Output.
- Experience in Amazon S3 for storing objects and sharing data between Hadoop.
- Connected Tableau from client end with AWS IP addresses to view the end results.
- Continuous monitoring and managing of the Hadoop cluster using Cloudera Manager.
- Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.
Environment: Hadoop, MapReduce, HDFS, Hive, Pig, Java, SQL, Cloudera Manager, Sqoop, Flume, Oozie, Java, Teradata, Informatica, Tableau, Eclipse, GitHub.
Confidential, Milwaukee, WI
Hadoop Developer
Responsibilities:
- Processed data into HDFS and analyzed the data using MapReduce, Pig and Hive to produce summary results fromHadoopto downstream systems.
- Used Sqoop widely in order to import data from various systems/sources like MySQL into HDFS.
- Used Pig as ETL tool to do transformations, event joins and some pre-aggregations before storing the data onto HDFS.
- Responsible for developing data pipeline using Flume, Sqoop and Pig to extract the data from weblogs and store in HDFS.
- Applied Hive quires to perform data analysis on HBase using Storage Handler in order to meet the business requirements.
- Hands on experience with NoSQL databases like HBase in storing URL's and images.
- Created components like Hive UDFs for missing functionality in HIVE for analytics.
- Used different Serde’s for converting JSON data into pipe separated data.
- Used different file formats like Text files, Sequence Files and Avro file format.
- Developing Scripts and Batch Job to schedule a bundle (group of coordinators) which consists of variousHadoopPrograms using Oozie.
- Exported the analyzed data to the relational databases using Sqoop.
- Assisted in creating and maintaining technical documentation to launch Hadoop clusters and for executing HIVE queries and PIG scripts.
- Assisted in cluster maintenance, cluster monitoring, adding and removing cluster nodes and Troubleshooting.
Environment: MapReduce, HDFS Sqoop, Flume, LINUX, Oozie,Hadoop, Pig, Hive, Hbase, HadoopCluster, Amazon Web Services.
Confidential
JAVA Developer
Responsibilities:
- Created design documents and reviewed with team in addition toassisting the business analyst project manager in explanations to line of business.
- Involved in different phases to gather requirements, document thefunctional specifications,design,datamodeling, development, testing of the applications.
- Developed the web tier using JSP to show account details andsummary.
- Designed and developed the UI using JSP, HTML, CSS and JavaScript.
- Utilized JPA for Object/Relational Mapping purposes for transparent persistence onto the SQL Server database.
- Used Tomcat web server for deployment purpose.
- Involved in creation of Test Cases for JUnit Testing.
- Used Oracle as Database and used Toad for queries execution and also involved in writing SQL scripts, PL/SQL code for procedures and functions.
- Developed application using Eclipse and used build and deploy tool as Maven.
- Used Log4J to print the logging, debugging, warning, info on the server console.
- J2EE Front-EndandBack-Endsupportingbusiness logic, integration, andpersistence.
- UsedJSPwithSpring Frameworkfor developingUser Interfaces.
- Developed the front-end userinterfaceusingJ2EE,Taglibs, Servlets, JDBC, HTML, DHTML, CSS, XML, XSL, XSLTandJavaScriptas perUse Case Specification.
- IntegratedSecurity Web Servicesfor authentication of users.
- UsedHibernate Object/Relational mappingandpersistence frameworkas well as aData Access abstraction Layer.
- Data Access Objects(DAO) framework is bundled as part of theHibernate Database Layer.
- Designed Data MappingXMLdocuments that are utilized byHibernate, to call stored procedures.
- Responsible for Testing and moving the application into Staging and Production environments.
- Responsible for Project Documentation, Status Reporting and Presentation.
- UsedCVS version controlto maintain theSource Code.
Environment: J2EE/Java, JSP,spring, Spring MVC, Hibernate, Eclipse 3.0, MySQL, JDBC, JBoss, ANT, JavaScript, XML, SOAP, WSDL, CVS.
