PROFESSIONAL SUMMARY:

• 6+ years of experience in design, development, production, maintenance of software applications using Java/J2EE technologies and Bigdata related technologies like Hadoop and Spark.
• 4+ years of experienced in Big Data technologies like HDFS, MapReduce, Hive, Pig, Kafka, Flume, Sqoop, HBase, Zookeeper, Oozie, Spark, Spark SQL, SparkStreaming and object - oriented programming and functional programming languages like Java, Python, Scala.
• Built several data pipeline using Spark Architecture on Hadoop using Yarn as a cluster management service.
• Worked on end-to-end data pipelines for use in individual big data analytics applications, providing data in a ready-to-use form to data scientists who are looking to run queries and algorithms against the information for predictive analytics, machine learning and data mining purposes.
• Extensively worked on Spark Streaming and Spark Sql, consuming real-time data from Kafka, and performed various transformations on data and queried using Spark Sql.
• Extensive knowledge on Spark transformations, actions, Dstream operations, Dataframe functions and actions, pair RDD transformations to work on key-value pair datasets.
• Experience in writing MapReduce jobs in Java as well as Python on Hadoop Ecosystem including Hive and Pig .
• Good Exposure on Apache Hadoop MapReduce programming, PIG Scripting and Distribute Application and HDFS.
• Procedural knowledge in cleansing and analyzing data using HiveQL, Pig, and custom MapReduce programs.
• Experienced in Hadoop Development Platform with Cloudera Distribution, Hortonworks, MapR, Amazon web services (AWS) like EMR, EC2 and S3.
• Used Talend ETL tool to perform data integration.
• Experienced in developing custom ETL and data ingestion pipelines in Python using pyspark.
• Consolidate diverse data source by building pipelines and scripts using Python, Hive etc.
• Extensive Experience on importing and exporting data using stream processing platforms like Flume and Kafka.
• Used Flume to channel data from different sources to HDFS.
• Used Cloudera Impala for running Interactive SQL in some of the use cases.
• Extensively performed interactive SQL queries in Impala for Hadoop to get responses faster.
• Worked with Oozie workflow engine to schedule time based jobs to perform multiple actions.
• Experience in understanding the security requirements for Hadoop and integrate with Kerberos authentication and authorization infrastructure.
• Hands on experience in creating interactive shell programs using Scala and Python.
• Expertise in working with Hive data warehouse infrastructure-creating tables, data distribution by implementing partitioning and bucketing, writing, and optimizing the HQL queries.
• Developed advanced aggregation framework using Hive UDFs and UDAFS.
• Experienced with performing real time analytics on NoSQL distributed data bases like Cassandra, HBase and MongoDB.
• Experienced in importing and exporting data from RDBMS into HDFS using Sqoop.
• Built data visualization dashboards using Tableau and Business Objects.
• Created Dashboards, analyzed the data using Tableau and delivered reports in desired formats by using reporting tools Tableau.
• Understanding of Machine Learning algorithms like Regression, Classification, and ML libraries(MLlib).
• Experience of using build tools Sbt, Maven.
• Experience in Test-driven development, Software Development Life Cycle (SDLC)methodologies like Agile and Scrum.
• Expertise in RDBMS namely Oracle, MS-SQL Server, MySQL and DB2 and Stored procedures & Triggers.

TECHNICAL SKILLS:

Big Data/ Data Engineering Technologies: HDFS, MapReduce, Hue, Hive, Spark, Pig, Sqoop, YARN, Oozie, Kafka, Flume, Impala, Zookeeper, Hadoop Distributions (Hortonworks, Cloudera), AWS, EMR, redshift, EC2, S3

Programming Languages: C, Java, Python, Scala, R, JavaScript, Bash Shell-scripting

Java Technologies: Java, J2EE - JSP, Servlets.

Data Stores/Databases: Cassandra, HBase, MongoDB, DynamoDB, Postgre, Vertica, Teradata, MySQL, MS SQL Server, Oracle

Reporting Tools: Tableau 8.X/9.X, Business Objects XI R2, Informatica Power center 8.X, OLAP/OLTP

Web Technologies: JavaScript, jQuery, AJAX, HTML 5, CSS 3, XML.

IDEs: IntelliJ, Pycharm, Anaconda jupyter, Eclipse, R studio

Tools: JIRA, Junit, Jenkins, Maven, sbt.

Version Control Systems: GIT, SVN, github.

PROFESSIONAL EXPERIENCE:

Confidential

Responsibilities:

• Worked on end-to-end data pipelines for use in individual big data analytics applications, providing data in a ready-to-use form to data scientists who are looking to run queries and algorithms against the information for predictive analytics, machine learning and data mining purposes.
• Extensively worked on Spark Streaming and Spark Sql, consuming real-time data from Kafka, and performed various transformations on data and queried using Spark Sql.
• Extensive knowledge on Spark transformations, actions, Dstream operations, Dataframe functions and actions, pair RDD transformations to work on key-value pair datasets.
• Experience in writing MapReduce jobs in Java as well as Python on Hadoop Ecosystem including Hive and Pig .
• Good Exposure on Apache Hadoop MapReduce programming, PIG Scripting and Distribute Application and HDFS.
• Procedural knowledge in cleansing and analyzing data using HiveQL, Pig, and custom MapReduce programs.
• Experienced in Hadoop Development Platform with Cloudera Distribution, Hortonworks, MapR, Amazon web services (AWS) like EMR, EC2 and S3.
• Used Talend ETL tool to perform data integration.
• Experienced in developing custom ETL and data ingestion pipelines in Python using pyspark.
• Consolidate diverse data source by building pipelines and scripts using Python, Hive etc.
• Extensive Experience on importing and exporting data using stream processing platforms like Flume and Kafka.
• Used Flume to channel data from different sources to HDFS.
• Used Cloudera Impala for running Interactive SQL in some of the use cases.
• Extensively performed interactive SQL queries in Impala for Hadoop to get responses faster.
• Worked with Oozie workflow engine to schedule time based jobs to perform multiple actions.
• Experience in understanding the security requirements for Hadoop and integrate with Kerberos authentication and authorization infrastructure.
• Hands on experience in creating interactive shell programs using Scala and Python.
• Expertise in working with Hive data warehouse infrastructure-creating tables, data distribution by implementing partitioning and bucketing, writing, and optimizing the HQL queries.
• Developed advanced aggregation framework using Hive UDFs and UDAFS.
• Experienced with performing real time analytics on NoSQL distributed data bases like Cassandra, HBase and MongoDB.
• Experienced in importing and exporting data from RDBMS into HDFS using Sqoop.
• Built data visualization dashboards using Tableau and Business Objects.
• Created Dashboards, analyzed the data using Tableau and delivered reports in desired formats by using reporting tools Tableau.
• Understanding of Machine Learning algorithms like Regression, Classification, and ML libraries(MLlib).
• Experience of using build tools Sbt, Maven.