We provide IT Staff Augmentation Services!

Sr. Big Data/hadoop Developer Resume

SUMMARY:

  • Over 8+ years of professional experience in IT, which includes Analysis, Design, Coding, Testing, Implementation and support in Java and Big Data Technologies working with Apache Hadoop Eco - components.
  • 4+ years of exclusive experience in Hadoop and its components like HDFS, Map Reduce, Apache Pig, Hive, Sqoop, HBase and Oozie
  • Involved in writing the Pig scripts and Pig UDFs to pre-process the data for analysis
  • Experience in creating Hive External and Managed tables and writing queries on them
  • Hands on Experience in troubleshooting operational issues and identifying root causes of Hadoop Cluster
  • Expertise in managing data from multiple sources and transform large sets of data
  • Extensively used Sqoop to import data into HDFS from RDBMS and vice-versa.
  • Designing and creating HIVE external tables using shared meta-store instead of the derby with partitioning, dynamic partitioning and buckets.
  • Experience in Oozie and workflow scheduler to manage Hadoop jobs by Direct Acyclic Graph (DAG) of actions with control flows.
  • Experience in integrating Hive and HBase for effective operations.
  • Good understanding in working with various compression techniques like Avro, Snappy, LZO
  • Experienced in working with Spark ecosystem using Spark-SQL and Scala queries on different data file formats like .txt, .csv etc.
  • Hands on experience in migrating Map Reduce jobs into Spark RDD transformations using SCALA.
  • Good experience in Cloudera, Hortonworks & Apache Hadoop distributions.
  • Strong understanding of NoSQL databases and hands on work experience in writing applications on NoSQL databases like HBase, Cassandra and MongoDB, Redis, Neo4j.
  • Working knowledge on major Hadoop ecosystems PIG , HIVE , Sqoop, and Flume.
  • Experience in implementing Custom Partitions and Combiners for effective data distributions.
  • Experience in Writing Map Reduce jobs for text mining for predictive analysis
  • Experience in analyzing data using Cassandra QL, Hive QL and Pig Latin programs.
  • Experience in implementing Custom Partitions and Combiners for effective data distributions.
  • Good working knowledge with Map Reduce and Apache Pig
  • Experience in application development using Java, J2EE, EJB, Hibernate, JDBC, Jakarta Struts, JSP and Servlets.
  • Experience in using various IDEs Eclipse, My Eclipse and repositories SVN and CVS.
  • Experience of using build tools Ant and Maven.
  • Working with relative ease with different working strategies like Agile, Waterfall and Scrum methodologies.
  • Excellent communication and analytical skills and flexible to adapt to evolving technology.

TECHNICAL SKILLS:

Languages: C, C++, Python, Java, J2EE, SQL, PL/SQL, Scala, UML, XML

Hadoop Ecosystem: HDFS, MapReduce, Spark Core, Spark Streaming, Spark SQL, Hive, Pig, Sqoop, Flume, Kafka, Oozie, Zookeeper.

Databases: Oracle 10g/11g, SQL Server, MYSQL, DB2

No SQL: HBase, Cassandra, MongoDB

Application / Web Servers: Apache Tomcat, JBoss, Mongrel, Web Logic, Web Sphere

Web Services: SOAP, REST

Operating systems: Windows, Unix, Linux

Microsoft Products: MS office, MS Visio, MS Project

Frameworks: Spring, Hibernate, Struts

PROFESSIONAL EXPERIENCE:

Confidential

Sr. Big Data/Hadoop Developer

Roles &Responsibilities:

  • Moved all crawl data flat files generated from various retailers to HDFS for Further processing.
  • Written the Apache PIG scripts to process the HDFS data.
  • Created Hive tables to store the processed results in a tabular format.
  • Developed the sqoop scripts in order to make the interaction between Pig and MySQL Database.
  • Writing the script files for processing data and loading to HDFS
  • Writing CLI commands using HDFS.
  • Developed the UNIX shell scripts for creating the reports from Hive data.
  • Completely involved in the requirement analysis phase.
  • Created two different users (hduser for performing hdfs operations and mapred user for performing map reduce operations only)
  • Ensured NFS is configured for Name Node
  • Setting Password less Hadoop
  • Written PIG scripts to process the Credit Card and Debit Card Transactions for Active customers by joining the data from HDFS and Hive using HCatalog for various merchants
  • Responsible for writing Lucene search program for high-performance, full-featured text search of Merchants
  • Written Python UDFs to process the RegEx and return the valid Merchant codes and names using streaming
  • Created Hive scripts for joining the raw data with the lookup data and for some aggregative operations as per the business requirement.
  • Setting up cron job to delete Hadoop logs/local old job files/cluster tempfiles
  • Setup Hive with MySQL as a Remote Metastore
  • Moved all log/text files generated by various products into HDFS location
  • Written Map Reduce code that will take input as log files and parse the logs and structure them in tabular format to facilitate effective querying on thelog data
  • Loading data from UNIX file system to HDFS and vice versa.
  • Implemented Data Ingestion in real time processing using Kafka.
  • Created External Hive Tables on top of parsed data.

Environment: Hadoop, HDFS, Map Reduce, Apache Pig, Hive, SQOOP, Linux, MySQL, Spark, Hbase, Hortonworks, HDP 2.6.5 Struts, Mongodb, Kafka, Eclipse RCP, Hadoop,, Spring, Spring, Spark

Confidential, Chicago, IL

Sr. Big Data/Hadoop Developer

Roles &Responsibilities:

  • Imported data using Sqoop to load data from MySQL to HDFS on regular basis.
  • Developed Scripts and Batch Job to schedule various Hadoop Program.
  • Wrote Hive queries for data analysis to meet the business requirements.
  • Created Hive tables and working on them using Hive QL.
  • End-to-end performance tuning of Hadoop clusters and Hadoop MapReduce routines against very large data sets.
  • Exported analyzed data to HDFS using Sqoop for generating reports.
  • Used MapReduce and Sqoop to load, aggregate, store and analyze web log data from different web servers.
  • Written Java UDFs to convert to upper case of card names & process dates to suitable format in PIG & Hive
  • Responsible for build the Docker containers and scheduled the Oozie workflows to run the sprints
  • Periodically reviewed Hadoop related logs and fixing errors and preventing errors by analyzing the warnings.
  • Involved in Analyzing system failures, identifying root causes, and recommended course of actions.
  • End-to-end involvement in data ingestion, cleansing, and transformation in Hadoop.
  • Developed Hive queries for the analysts.
  • Used Impala to read, write and query the Hadoop data in HDFS from Cassandra and configured Kafka to read and write messages from external programs.
  • Optimizing existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD's.
  • Cluster co-ordination services through Zookeeper.
  • Written the Spouts and Bolts after collecting the real stream customer data from Kafka broker to process and store into HBASE.
  • Analyze the log files and process through Flume
  • Experience in optimization of MapReduce algorithm using combiners and partitions to deliver the best results and worked on Application performance optimization.

Environment: Hadoop, HDFS, Map Reduce, Apache Pig, Hive, SQOOP, Linux, MySQL, Spark

Confidential, Columbus, Ohio

Hadoop Developer

Roles &Responsibilities:

  • Imported data using Sqoop to load data from MySQL to HDFS on regular basis.
  • Developed Scripts and Batch Job to schedule various Hadoop Program.
  • Wrote Flume configuration files for importing streaming log data into MongoDB with Flume
  • Performed masking on customer sensitive data using Flume interceptors.
  • Used IMPALA to analyze data ingested into Hive tables and compute various metrics for reporting on the dashboard .
  • Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.
  • Wrote Hive queries for data analysis to meet the business requirements.
  • POC developed to pull Realtime twitter streaming using kafka, flume and spark
  • Used parquet compression format for hive table creation
  • Implemented Kerberos security Implementation
  • Created users in Active Directory and map the roles in each group for the users in Apache Sentry
  • Involved in LDAP implementation for different types of accesses in AD for Hue, Hive, Pig
  • Complete caring of Hive and Spark tuning with partitioning/bucketing of ORC and executors/driver’s memory
  • Involved in Extracting, transforming, loading Data from Hive to Load an RDBMS.
  • Optimized MapReduce Jobs to use HDFS efficiently by using various compression mechanisms.
  • Involved in extracting, transform and load the data into HBase Database
  • Implemented Partitioning and bucketing in Hive based on the requirement.
  • Involved in transforming Data within a Hadoop Cluster
  • Involved in Using MapReduce to Parse Weblog Data for MapReduce to convert raw weblog data into parsed, delimited records.
  • Written Hive UDFs to extract data from staging tables
  • Developed hive external tables
  • Created Hbase tables to store Json data

Environment: Eclipse, jdk1.8.0, Hadoop2.8, HDFS, MapReduce, Pig0.15.0, Hive2.0, HBase, Kerberos, Apache-Maven3.0.3

Confidential

Java Developer

Roles &Responsibilities:

  • Analyzing the feasibility Documents
  • Coding the business logic methods in core java.
  • Involved in development of the Action classes and Action Forms based on the Struts framework.
  • Participated in client-side validation and server-side validation.
  • Involved in creation of struts configuration file and validation file for skip module using struts framework.
  • Developed java programs, JSP pages and servlets using Spring framework.
  • Involved in creating database tables, writing complex TSQL queries and stored procedures in the SQL server.
  • Worked with AJAX framework to get the asynchronous response for the user request and used JavaScript for the validation.
  • Used EJBs in the application and developed Session beans to implement business logic at the middle tier level.
  • Actively involved in writing SQL using SQL Query Builder.
  • Used JAXB to read and manipulate the xml properties.
  • Used JNI for calling the libraries and other implemented functions in C language.
  • Handling Server Related issues, new requirement handling, changes and patch movements.
  • Developed the Restful Web Services for various XSD schemas.
  • Used Servlets to implement Business components.
  • Designed and Developed required Manager Classes for database operations.
  • Developed various Servlets for monitoring the application.
  • Designed the UML class diagram, Sequence diagrams for Trade Services.
  • Designed the complete Hibernate mapping for SQL Server for PDM.
  • Designed the complete JAXB classes mapping for various XSD schemas.
  • Developed the Restful Web Services for various XSD schemas.
  • Involved in writing JUnit test Classes for performing Unit testing.

Environment: Eclipse neon, jdk1.8.0, Java, Servlets, JSP, EJB, xml, SQL server, Struts, JUnit and Eclipse, SQL, UNIX, UML, Apache-Maven3.0.3, Struts, Mongodb, Kafka, Eclipse RCP

Confidential

Java Developer

Roles &Responsibilities:

  • Identifying reviewing, assessing and resolving production issues
  • Configure and maintain the associated application components and environments (as required).
  • Provide application support to management, team members and end users
  • Having experience on the sales functionality and order management
  • Worked on the Email template creation with HTML code as per the requirements
  • Email notification configured based on the requirement
  • Involved in writing programs for XA transaction management on multiple databases of the application.
  • Developed java programs, JSP pages and servlets using Cantata Struts framework.
  • Involved in creating database tables, writing complex TSQL queries and stored procedures in the SQL server.
  • Worked with AJAX framework to get the asynchronous response for the user request and used JavaScript for the validation.
  • Used EJBs in the application and developed Session beans to implement business logic at the middle tier level.
  • Actively involved in writing SQL using SQL Query Builder.
  • Involved in coordinating the on-shore/Off-shore development and mentoring the new team members.
  • Extensively Used Ant tool to build and configure J2EE applications and used Log4J for logging in the application
  • Used JAXB to read and manipulate the xml properties.
  • Used JNI for calling the libraries and other implemented functions in C language.
  • Used prototype MooTools and script.aculo.us for fluid User Interface.
  • Involved in fixing defects and unit testing with test cases using JUnit.
  • Involved in Configured business components, Views, applets, Controls, Menus and other objects to meet the business Requirements

Environment: jdk1.8.0, Java, Servlets, JSP, xml, SQL server, JUnit and Eclipse, Unix, UML, Apache-Maven3.0.3, EJB, Servlets, XSLT, CVS, J2EE, AJAX, Struts, Hibernate, ANT, Tomcat, JMS, Log4J, Oracle 10g, Eclipse, Solaris, JUnit and Windows 7/XP

Hire Now