SUMMARY

• 5+ years of overall experience in IT Industry which includes experience in Big Data Technologies and Web Applications in multi - tiered environment using Hadoop, Spark, Hive, Pig, Sqoop, J2EE (Spring, JSP, Servlets), JDBC, HTML, CSS and Java Script (Angular JS).
• 3+ years of comprehensive experience in Big Data Analytics and its ecosystem components in Cent OS and RHEL Linux environment.
• Working knowledge in AWS environment and AWS spark with Strong experience in Cloud computing platforms such as AWS services.
• Demonstrated a full understanding of the Fact/Dimension data warehouse design model, including star and snowflake design methods.
• Designed and developed logical and physical data models that utilize concepts such as Star Schema, Snowflake Schema and Slowly Changing Dimensions.
• Migrated an existing on-premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR.
• Hands On experience on Spark Core, Spark SQL, Spark Streaming and creating the Data Frames handle in SPARK with Scala.
• IntegratedJenkinswith various Build tools to achieve successful implementation ofJenkinsCI/CD pipeline, Developed Scripted pipeline to version control various pipelines and maintain the Source of Truth.
• Implemented CI/CD usingJenkins. Configured security toJenkinsand added multiple nodes for continuous deployments. Deployed applications using build tools like Ant and Maven.
• Strong understanding of build.xml and pom.xml files. Worked on Nexus, Artifactory Repository Managers for Maven builds to upload/download the built Artifacts
• Experience in NoSQL databases and worked on table row key design and to load and retrieve data for real time data processing and performance improvements based on data access patterns.
• Extensive experience in Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, and Map Reduce concepts.
• Experience in building large scale highly available Web Applications. Working knowledge of web services and other integration patterns.
• Developed Simple to complex Map/reduce and Streaming jobs using Java and Scala language.
• Developed Hive scripts for end user / analyst requirements to perform ad hoc analysis.
• EMR with Hive to handle less important bulk ETL jobs.
• Hands Experience with NOSQL Databases like MongoDB.
• Having knowledge about Hadoop architecture and its different components such as HDFS, Job tracker, Task tracker, Resource Manager, Name Node, Data Node and Map Reduce concepts.
• Experience with Java development skills using J2EE, J2SE, Servlets, JSP, EJB, JDBC.
• Implementation knowledge of Enterprise/Web/Client Server using Java, J2EE.
• Experience in analyzing the data using HQL, Pig Latin, and custom Map Reduce programs in Java.
• Handled messaging services using Apache Kafka.
• Experience in integrating Apache Kafka with Apache Storm and created Storm data pipelines for real time processing.
• Established connection from Azure to On-premise datacenter using Azure Express Route for Single and Multi-Subscription
• Experience in Amazon web services (AWS) cloud like S3, EC2 and EMR and in Microsoft Azure.
• Expertise in Azure infrastructure management (Azure Web Roles, Worker Roles, SQL Azure, AzureStorage, Azure AD Licenses, Office365)
• Experience in dealing with Windows Azure IaaS - Virtual Networks, Virtual Machines, Cloud Services, Resource Groups, Express Route, Traffic Manager, VPN, Load Balancing, Application Gateways, Auto-Scaling.
• Exposure on usage of Apache Kafka to develop data pipeline of logs as a stream of messages using producers and consumers.
• Excellent understanding and knowledge of NOSQL databases like Mongo dB and Cassandra.
• Created Cassandra tables to store various data formats of data coming from different sources.
• Extensive knowledge and experience on real time data streaming techniques like Kafka, and Spark Streaming.
• Configured Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS.
• Worked on MongoDB database concepts such as locking, transactions, indexes, replication, schema design.
• Experienced in data formats like JSON, PARQUET, AVRO, RC and ORC formats
• Utilized Flume to analyze log files and write into HDFS.
• Experience in importing and exporting data by Sqoop between HDFS and RDBMS and migrating according to client's requirement
• Having experience in working with data ingestion, storage, processing and analyzing the big data.
• Worked on loading and transforming of large sets of structured, semi structured and unstructured data.
• Developed extraction mappings to load data from Source systems to ODS to Data Warehouse.
• Involved in conceptual, logical and physical datamodeling and used star schema in designing the data Warehouse.
• Used GitHub version control tool to push and pull functions to get the updated code from repository.

TECHNICAL SKILLS

Programming Languages: Java, Scala, Unix Shell Scripting, PL/SQL,Python

J2EE Technologies: Spring, Servlets, JSP, JDBC, Hibernate.

Big Data Ecosystem: HDFS, Map Reduce, Hive, Pig, Spark, Kafka, AWS-EMR, Kinesis, Sqoop, Impala, Oozie, Zookeeper, Flume.

DBMS: Oracle 11g, SQL Server, MySQL, IBM DB2, Snowflake.

Modeling Tools: UML on Rational Rose 4.0

Web Technologies: HTML5, JavaScript, XML, jQuery, Ajax, CSS3.

IDEs: Eclipse, Net beans, WinSCP, Visual Studio and IntelliJ

Operating systems: UNIX, Windows, RHEL, Solaris, Centos.

Version and Source Control: TFS, Git, SVN and IBM Rational Clear Case.

Servers: Apache Tomcat, Web logic and Web Sphere

Frameworks: MVC, Spring, Struts, Log4J, Junit, Maven, ANT

PROFESSIONAL EXPERIENCE

Confidential, Richardson, TX

Big Data/Spark Developer

Responsibilities:

• Experience in Job management using Fair scheduler and Developed job processing scripts using Oozie workflow.
• Used Spark, Hive for implementing the transformations need to join the daily ingested data to historic data.
• Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time.
• Developed Spark scripts by using Scala shell commands as per the requirement.
• Used Spark API over EMR Cluster Hadoop YARN to perform analytics on data in Hive.
• Developed Scala scripts, UDFs using both Data frames/SQL/Data sets and RDD in Spark for Data Aggregation, queries and writing data back into OLTP system through Sqoop.
• Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
• Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD's.Worked with Spark on an EMR cluster along with other Hadoop applications, and it can also leverage the EMR file system (EMRFS) to directly access data in Amazon S3.
• Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark.
• Worked extensively on building Nifi data pipelines in docker container environment in development phase.
• Worked in Spark Scala, improving the performance and optimized of the existing applications running on EMR cluster.
• Good knowledge on Amazon AWS concepts like EMR & EC2 web services which provides fast and efficient processing of Big Data
• Developed reusable transformations to load data from flat files and other data sources to the Data Warehouse.Worked with DevOps team to Clusterize NIFI Pipeline on EC2 nodes integrated with Spark, Kafka, Postgres running on other instances using SSL handshakes in QA and Production Environments
• Assisted operation support team for transactional data loads in developing SQL Loader & Unix scripts
• Implemented Spark SQL queries which intermix the Hive queries with the programmatic data manipulations supported by RDDs and data frames in Scala and python.
• Implemented Python script to call the Cassandra Rest API, performed transformations and loaded the data into Hive.
• Worked on NiFi data Pipeline to process large set of data and configured Lookup’s for Data Validation and Integrity
• Extensively worked on Python and build the custom ingest framework.
• Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself.
• Designed, developed and did maintenance of data integration programs in a Hadoop and RDBMS environment with both traditional and non-traditional source systems as well as RDBMS and NoSQL data stores for data access and analysis.
• Configured Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS.
• Experienced in writing live Real-time Processing using Spark Streaming with Kafka.
• Created Cassandra tables to store various data formats of data coming from different sources.
• Designed, developed data integration programs in a Hadoopenvironment with NoSQL data store Cassandra for data access and analysis.
• Worked extensively with Sqoop for importing metadata from Oracle.
• Involved in creating Hive tables and loading and analyzing data using hive queries.
• Developed Hive queries to process the data and generate the data cubes for visualizing.
• Implemented schema extraction for Parquet and Avro file Formats in Hive.
• Good experience with Talend open studio for designing ETL Jobs for Processing of data.
• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
• Involved in file movements between HDFS and AWS S3.
• Extensively worked with S3 bucket in AWS.
• Used Reporting tools like Tableau to connect with Hive for generating daily reports of data.
• Collaborated with the infrastructure, network, database, application and BI teams to ensure data quality and availability.

Environment: Hadoop YARN, Spark 1.6, Spark Streaming, Spark SQL, Scala, Kafka, Python, Hive, Sqoop 1.4.6, Impala, Tableau, Talend, Oozie, Control-M, Java, AWSS3, Oracle 12c, Linux.

Confidential, Chicago, IL

Big Data/Spark Developer

Responsibilities:

• Involved in complete Bigdata flow of the application starting from data ingestion from upstream to HDFS, processing and analyzing the data in HDFS.
• Responsible for importing data to HDFS using Sqoop from different RDBMS servers and exporting data using Sqoop to the RDBMS servers.
• Developed data pipeline using Sqoop to ingest customer behavioral data and purchase histories into HDFS for analysis.
• Created Partitioned and Bucketed Hive tables in Parquet File Formats with Snappy compression and then loaded data into Parquet hive tables from Avro hive tables.
• Involved in running all the hive scripts through hive. Hive on Spark and some through Spark SQL.
• Involved in importing the real time data to Hadoop using Kafka and implemented the Oozie job for daily imports.
• Troubleshooting the Azure Development, configuration and Performance issues.
• Interacted with multiple teams who are responsible for Azure Platform to fix the Azure Platform Bugs.
• Providing 24/7 support for on-call on Azure configuration and Performance issues.
• Deployed the initial Azure components like Azure Virtual Networks, Azure Application Gateway, Azure Storage and Affinity groups.
• Written Kafka REST API to collect events from front end.
• Extracted feeds form social media sites such as Facebook, Twitter using Python scripts.
• Developed and implemented Apache NIFI across various environments, written QA scripts in Python for tracking files.
• Implemented a prototype for the complete requirements using Splunk, python and Machine learning concepts
• Tested Apache TEZ, an extensible framework for building high performance batch and interactive data processing applications, on Pig and Hive jobs.
• Knowledge on handling Hive queries using Spark SQL that integrate with Spark environment.
• Enhanced and optimized product Spark code to aggregate, group and run data mining tasks using the Spark framework.
• Worked on product positioning and messaging that differentiate Hortonworks in the open source space.
• Experience in design and developing Application leveraging MongoDB.
• Set up Apache NiFi to transfer structured and streaming data into HDFS
• Ingested streaming data with Apache NiFi into Kafka
• Collected the JSON data from HTTP Source and developed Spark APIs that helps to do inserts and updates in Hive tables.
• Developed Spark scripts to import large files from Amazon S3 buckets.
• Developed shell scripts for running Hive scripts in Hive and Impala.
• Used Jira for bug tracking and Bitbucket to check-in and checkout code changes.

Environment: Scala, Azure, HDFS, Yarn, MapReduce, Hive, Sqoop, Flume, Oozie, NIFI, Kafka, Impala, Spark SQL, Spark Streaming, Eclipse, Oracle, Teradata, PL/SQL UNIX Shell Scripting.

Confidential

Hadoop Developer

Responsibilities:

• Extensively worked on Hadoop eco-systems including Hive, Spark Streaming with MapR distribution.
• Collaborated with business users/product owners/developers to contribute to the analysis of functional requirements.
• Worked on analyzing Hadoop cluster using different big data analytic tools including Pig, HBase database and Sqoop.
• Worked on NoSQL support enterprise production and loading data into HBase using Impala and Sqoop.
• Performed multiple MapReduce jobs in Pig and Hive for data cleaning and pre-processing.
• Build Hadoop solutions for big data problems using MR1 and MR2 in YARN.
• Handled importing of data from various data sources, performed transformations using Hive, PIG, and loaded data into HDFS.
• Used JDBC for querying/inserting & data management including triggers and stored procedures in the Java application.
• Worked on data using Sqoop from HDFS to Relational Database Systems and vice-versa. Maintaining and troubleshooting.
• Used Spring/MVC framework to enable the interactions between JSP/View layer and implemented different design patterns with J2EE and XML technology.
• Exploring with Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, Data Frame, pair RDD's.
• Created Hive Tables, loaded claims data from Oracle using Sqoop and loaded the processed data into target database.
• Involved in PL/SQL query optimization to reduce the overall run time of stored procedures.
• Exported data from HDFS to RDBMS via Sqoop for Business Intelligence, visualization and user report generation.
• Used Cloudera Manager for installation and management of Hadoop Cluster.
• Involved in converting HiveQL into Spark transformations using Spark RDD and through Scala programming.
• Integrated Kafka-Spark streaming for high efficiency throughput and reliability
• Worked in tuning Hive & Pig to improve performance and solved performance issues in both scripts.

Environment: Hadoop 3.0, Hive 2.1, J2EE, JDBC, Pig 0.16, HBase 1.1, Sqoop, NoSQL, Impala, Java, Spring, MVC, XML, Spark 1.9, PL/SQL, HDFS, JSON, Hibernate, Bootstrap, JQuery

Confidential

Java Developer

Responsibilities:

• Experience in coding Servlets on the server side, which gets the requests from the client and processes the same by interacting the Oracle database.
• CodedJavaServlets to control and maintain the session state and handle user requests
• GUI development using HTML Forms and Frames and validating the data With JavaScript.
• Used JDBC to connect to the backend database and developed stored procedures.
• Developed code to handle web requests involving Request Handlers, Business Objects, and Data Access Objects.
• Creation of JSP pages including the use of JSP custom tags and other methods ofJavaBeam presentation and all HTML and graphically oriented aspects of the site' s user interface.
• Used XML for mapping the pages and classes and to transfer data universally among different data sources.
• Worked in unit testing and documentation.
• Hands on experience in J2EE framework Struts. Implemented Spring Model View Controller (MVC) Architecture based presentation using JSF framework. Extensively used Core Java API, Spring API in developing the business logic.
• Designed and developed Agile Applications, Light weight solutions, and integrated applications by using and integrating different frameworks like Struts and Spring.

Environment: Eclipse, servlets, Struts, JSPs, HTML, CSS, JavaScript, jQuery, SQL, JDBC.