SUMMARY

• Experience 8+years of strong experience in IT industry with Complete Software Development Life cycle (SDLC) which includes Business Requirements Gathering, System Analysis & Design, Data Modeling, Development, Testing and Implementation of the projects.
• Experience in developing, implementing and configuring of BigData using Hadoop ecosystemtools like HDFS, MapReduce, Hive, Oozie, Sqoop, Zookeeper, RabbitMQ, Kafka,Cassandra, HBase, Spark and Spark Streaming.
• Experience in installation, configuration, deploying and managing of different Hadoop distributions like Clouderaand HortonworksDistributions.
• Have an experience in importing and exporting data using Sqoop from Hadoop Distributed File Systems to Relational Database Systems and vice versa.
• Experience in handling various file formats like AVRO, Sequential, text, xml and Parquet etc.
• Imported the data from source HDFS intoSparkRDD for in - memory data computation to generate the output response.
• Experience on Spark Architecture including Spark Core, Spark SQL, Data Frames, Spark Streaming and Spark MLlib.
• Expertise in writing SparkRDD transformations, actions, Data Frame's, case classes for the required input data and performed the data transformations using Spark-Core.
• Experience on collection the realtime streaming data pipeline from different source using Kafka and
• Extending HIVE core functionality by using custom User Defined Function's (UDF) and User Defined Aggregating Functions (UDAF) for Hive.
• Knowledge in AWS concepts like EC2,S3, Data pipelines,CloudWatch and CloudFormation.
• Experience in troubleshooting errors in HBase Shell, Pig, Hive and MapReduce.
• Experience in NoSQL Databases HBase, Cassandra and it’s integrated with Hadoopcluster.
• Implemented Cluster for NoSQL tools HBaseas a part of POC to address HBase limitations.
• Exploring withSparkimproving the performance and optimization of the existing algorithms in Hadoop usingSparkcontext,Spark-SQL, Data Frame, pair RDD's,SparkYARN.
• Strong expertise in using ETL ToolInformaticaPower CenterDesigner, Workflow Manager, Repository Manager, Data Quality and ETL concepts.
• Experience of constant information ingestion utilizing Kafka, Spark and different NoSQL databases.
• Knowledge in using NiFi to automate the data movement between different Hadoop systems.
• Implemented Hadoop Security using integrated LDAP store with Kerberos KDC.
• Experience in understanding the security requirements for Hadoop and integrate with Kerberos authentication and authorization infrastructure.
• Experience on cloud integration with Amazon Elastic MapReduce, Amazon Cloud Compute, Amazon's Simple Storage Service and Microsoft Azure.
• Experienced on different Relational Data Base Management Systems likeTeradata, PostgreSQL, Oracle and SQL Server.
• Experienced in scheduling and monitoring the production jobs using Oozie

TECHNICAL SKILLS

Hadoop Ecosystems: Hadoop, HDFS, MapReduce, Hive, YARN, Oozie, Zookeeper, Spark, Impala,Spark SQL, Spark Streaming,Hue, Kafka,Sqoop, NiFi, Elasticsearch.

Cloud Services: Elastic MapReduce, Amazon Cloud Compute, Simple Storage Service and Microsoft Azure.

Languages: Java, Scala, Python, PL/SQL.

Java Technologies: Spring MVC, JDBC, JSON, JDBC, JMS, EJB.

UI Technologies: HTML5, JavaScript, CSS3, XML,JSON.

Development Tools: Eclipse, IntelliJ, Ant, Maven, Postman’s, Scala IDE.

Frameworks/Web Server: Spring, Hibernate, Hadoop, Web Logic, Web Sphere, Tomcat.

SQL/ NoSQL Databases: Teradata, Oracle,HBase, Cassandra, MySQL

Versions/ Building tools: GitHub, BitBucket, SVN, JIRA, Maven

PROFESSIONAL EXPERIENCE

Confidential, WA

Sr. Big Data / Spark Programmer

Responsibilities:

• Involved in installing and configuration of Spark distribution and Kafkain AWS.
• Involved in requirement gathered, narrates the stories, reviewing and merging the codes of development teaminto Dev repositories and follow the agile methodologies of SDLC.
• Importing of data from various data sources and vendors,like REST API and ELASTICSEARCHperformed transformations using Spark, loaded data into HDFS and extracted the data from SQL Server into HDFS using Sqoop.
• Exporting the analyzing data to the Elasticsearchusing Sparkfor visualization and to generate reports for the BI team.
• Worked on different large data sets format like JSON, CSV and did some Data analysis using spark-SQL and getting insights for better understanding about data.
• Worked on analyzing Hadoop cluster using different big data analytic tools including Hive, and MapReduce.
• Worked on batch and streaming data ingestion to Cassandra database.
• Expertise in integrating Kafka with Spark streamingfor high speed data processing.
• Developed Kafka Producers and Consumers for data streaming.
• Worked on implementation of a log producer in Scala that watches for application logs, transform incremental log and sends them to a Kafka and Zookeeper based log collection platform.
• Created Data Pipelines as per the business requirements using spark.
• Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and RDD's.
• Working with Data Modeling for Machine learning and Data Mining using MLlib library using pyspark.
• Developed and performed unit testing using JUnit framework in a Test-Driven environment (TDD).
• Involved in integrating testing and production support.
• Co-ordinate with offshore and onsite team to understand the requirements and prepare High level and Low-level design documents from the requirements specification.

Environment: Spark, Spark SQL, AWS,GitHub, Maven, HDFS,Hive, Elasticsearch, Sqoop, Kafka, ScalaData Modeling (MLlib) using Spark, Machine Learning.

Confidential, TX

Sr. Big Data Hadoop Developer

Responsibilities:

• Involved in installing and configuration of Hadoop distribution systems as a cloudera.
• Involved in requirement gathered, narrates the stories, reviewing and merging the codes of development teaminto Dev repositories and follow the agile methodologies of SDLC.
• Importing of data from various data sources, performed transformations using Hive, loaded data into HDFS and extracted the data from SQL Server into HDFS using Sqoop.
• Exporting the analyzing data to the relational databases using for visualization and to generate reports for the BI team.
• Worked on different data format like AVRO, Parquet and XML.
• Worked on analyzing Hadoop cluster using different big data analytic tools including Pig, Hive, and Map Reduce.
• Expertise in integrating Kafka with Spark streaming for high speed data processing.
• Developed multiple Kafka Producers and Consumers as per the software requirement specifications.
• Worked on implementation of a log producer in Scala that watches for application logs, transform incremental log and sends them to a Kafka and Zookeeper based log collection platform.
• Created Data Pipelines as per the business requirements and scheduled it using Oozie Coordinators.
• Optimizing of existing algorithms in Hadoop using Spark.
• Good understanding of Machine learning and Data Mining.

Environment: Spark, AWS, HDFS,Hive, Bigdata, Oozie, Sqoop, Kafka, HBase, Scala,SQL Server.

Confidential, CA

Sr. Big Data Hadoop Developer

Responsibilities:

• Installed and configured distributed data solution using ClouderaDistribution of Hadoop.
• Involved in complete big data flow of the application data ingestion from upstream to HDFS, processing the data in HDFS and analyzing the data.
• Imported the data from various formats like JSON, Sequential, AVROand Parquetto HDFS cluster.
• Implemented Sqoop for large data transfers from RDMS to HDFS/HBase/Hive and vice-versa.
• Configured Hive and written Hive UDF’s and UDAF’s.
• Implemented advanced procedures like text analytics and processing using the in-memory computing capabilities likespark.
• Importing and exporting data into HDFSand hive using Sqoop and Kafka.
• UsingSpark-Streaming APIs to perform transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time and Persists into HBase.
• Performance analysis of Sparkstreaming and batch jobs by using Spark tuning parameters.
• Enhanced and optimized productSparkcode to aggregate, group and run data mining tasks using theSparkframework.
• Developed Spark scripts by using Python shell commands as per the requirement.
• Using Hive join queries to join multiple tables of a source system and load them to Elastic search tables.
• Experience in managing and reviewing huge Hadoop log files.
• Involved in Cluster maintenance, Cluster Monitoring and Troubleshooting
• Created Data Pipelines as per the business requirements and scheduled it using Oozie Coordinators.
• Maintaining technical documentation for each and every steps of development environment and launching Hadoop clusters.
• Built the automated build and deployment framework using BitBucket,SourceTree and Mavenetc.
• Used IntelliJ IDEA to develop the code and dubbing.
• Worked on BI tools as Tableauto create dashboards like weekly, monthly, daily reports using tableaudesktop and publish them to HDFS cluster.

Environment: Bit Bucket, SourceTree, Maven, Hadoop, Bigdata,HDFS,Hive, Oozie, Sqoop, Spark, Kafka, Elastic Search, Shell Scripting, LINUX, HBase, Scala, Python, Tableau, MySQL.

Confidential - Plano, TX

Big Data/Hadoop Developer

Responsibilities:

• Involved in requirement gathered, narrates the stories and worked with complete Software Development Life Cycle (SDLC) methodologies based on Agile.
• Involved in installation, configuration, supporting and managing HadoopClusters using Hortonworks Distribution (HDP) to Cloudera Distributions Hadoop (CDH).
• Worked on Hadoop MapReduce, HDFS, developed multiple MapReduce jobs in java for data cleaning and preprocessing.
• Experienced in managing and reviewing Hadoop log files and documenting the issues on daily basis to the resolution portal.
• Implemented Dynamic Partitions, Buckets in HIVE.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Scala and Python.
• Experience configuring spouts and bolts in various Storm topologies and validating data in the bolts.
• Experienced in running Hadoop streaming jobs to process terabytes of xml format data.
• Have an experience to load and transform large sets of structured, semi structured and unstructured data, using Sqoop from Hadoop Distributed File Systems to Relational Database Systems and also Relational Database Systems to Hadoop Distributed File Systems.
• Used Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala and databases such as HBase.
• Established/implemented firewall rules, Validated rules with vulnerability scanning tools
• Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java MapReduce Hive, Pig, and Sqoop.
• Implemented Storm builder topologies to perform cleansing operations before moving data into HBase.
• Involved in creating Hive tables, loading with data and writing hive queries which will run internally in map reduce way.
• Developed a custom File System plug in for Hadoop so it can access files on Data Platform.
• Used Spark to create API's in Javaand Python for Big Data analysis.
• Experience in troubleshooting errors in Cassandra, Hive and MapReduce.
• This plugin allows Hadoop MapReduce programs, Cassandra, Pig and Hive to work unmodified and access files directly.
• Used versions controls tools such as Git Hub, SourceTree, to pull data from Upstream to local branch, check conflict, cleaning also reviewing the codes of other developers.
• Involved with development teams to discuss JIRA stories and understand the requirements.
• Actively, involved in complete life cycle of agile methodology to design, develop, deploy and support solutions.

Environment: Hadoop, Hive, Pig,Cassandra, Sqoop,Oozie, Java, Python,Shell Scripting, MapReduce, Java Collection, MySQL, Maven.

Confidential - Austin, TX

Hadoop Developer

Responsibilities:

• Installed and configured Hadoop MapReduce, HDFS and developed multiple MapReduce jobs in Java for datacleansing and preprocessing.
• Involved in loading data from UNIX file system to HDFS.
• Installed and configured Hive and also written Hive UDFs.
• Evaluated business requirements and prepared detailed specifications that follow project guidelines required todevelop written programs.
• Devised procedures that solve complex business problems with due considerations for hardware/softwarecapacity and limitations, operating times and desired results.
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
• Provided quick response to ad hoc internal and external client requests for data and experienced in creating adhoc reports.
• Responsible for building scalable distributed data solutions using Hadoop.
• Responsible for cluster maintenance, adding and removing cluster nodes, cluster monitoring and
• Troubleshooting, manage and review data backups, manage and review Hadoop log files.Worked hands on with ETL process.
• Handled importing of data from various data sources, performed transformations using Hive, MapReduce, andloaded data into HDFS.
• Extracted the data from Teradata into HDFS using Sqoop.
• Analyzed the data by performing Hive queries and running Pig scripts to know user behavior like shoppingenthusiasts, travelers, music lovers etc.
• Created lookup Hive tables, JSON Format Hive Tables.
• Used Maven extensively for building jar files of MapReduce programs and deployed to Cluster.
• Modeled Hive partitions extensively for data separation and faster data processing and followed Pig and Hivebest practices for tuning.
• Exported the patterns analyzed back into Teradata using Sqoop.
• Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.
• Installed Oozie workflow engine to run multiple Hive.

Environment: Java, Oracle, HTML, XML, SQL, J2EE, JUnit, JDBC, JSP, Tomcat, SQL Server, MongoDB, JavaScript, GitHub, SourceTree, NetBeans.