SUMMARY

• 8 years of experience in design and deployment of Enterprise Application Development and Web Applications which includes 6 plus years of comprehensive experience in big data ecosystem.
• Complete Understanding on Hadoop daemons such as Job Tracker, Task Tracker, Name Node and Data Node that provides client communication, job execution and management, resource scheduling and resource management.
• Expertise on Hadoop architecture and ecosystem such as HDFS, Sqoop, Spark, Ni - Fi, Pig and Oozie.
• Developed ETL pipelines involving transformation of data stored in JSON and parquet files, using GLUE +Pyspark.
• Hands on experience writing Pyspark pipelines in databricks.
• Strong real-life experience in python development especially in Pyspark in AWS cloud environment.
• Good knowledge in MapReduce, that can generate big data sets with a parallel and distributed algorithm.
• Acquired skills using Hive that provides data query and analysis, Sqoop that helps transferring data between relational databases and Hadoop.
• Expertise in ingesting data from external sources to HDFS for data processing using Flume.
• Hands on experience in creating workflows and scheduling jobs using Oozie.
• Expertise in open-source server that provides a distributed configuration service, synchronization service and naming registry using Zookeeper.
• Experience in installation, configuration, management, supporting and monitoring Hadoop cluster using various distributions such as Cloudera, Hortonworks, and various cloud services like AWS, GCP.
• Gathered knowledge in NIFI that can automate the flow of data and learnt how to deliver data to every part of your business with smart data pipelines using Stream Set.
• Expertise in writing custom Kafka consumer code and modifying existing producer code in Python to push data to Spark-streaming jobs.
• Experience with NumPy, a library that supports multi-dimensional arrays and matrices, Matplotlib, a plotting library, Pandas that is used for data manipulation and analysis with less code, and PySpark for writing spark applications using python API’s.
• Ample knowledge on Apache Kafka, Apache Storm to build data platforms, pipelines, and storage systems and search technologies such as Elastic search.
• Worked extensively over semi-structured data (fixed length & delimited files) for data sanitation, report generation and standardization.
• Extensive experience working with AWS Cloud services and AWS SDKs to work with services like AWS API Gateway, Lambda, S3, IAM and EC2.
• Experience in using Kafka and Kafka brokers to initiate spark context and processing live streaming.
• Good at implementing Kafka custom encoders for custom input format to load data in partitions.
• Experienced in providing highly available and fault tolerant applications utilizing orchestration technology on Google Cloud Platform (GCP).
• Experienced on the planning and capacity requirements for the migration path of IBM Big Insights (on-prem) solution to cloud native GCP based solution.This involved tools like DataProc, Dataflow, Cloud Functions, GoogleCloud Storage and Pub/Sub.
• Knowledge in automated deployments leveraging Azure Resource Manager Templates, DevOps and Git repository for Automation and usage of Continuous Integration (CI/CD).
• Experienced in User Defined Functions (UDF) such as Spark, HiveQL, SQL for data processing and analysis.
• Developed expertise in database management system built to handle large amounts of data using Cassandra that is highly scalable.
• Experienced with Google Cloud Platform (GCP) services like compute engine, cloud load balancing, cloud storage, cloud SQL, stack driver monitoring, and cloud deployment manager.
• Gained knowledge and expertise in PostgreSQL that is an open-source object-relational database system, used as the primary data store or data warehouse for many web, mobile, geospatial, and analytics applications.
• Worked with Impala to access/analyze data that is stored in Hadoop data nodes without data movement and provides fast access for the data in HDFS and moved data from the datacenter to the cloud and the edge by using Couchbase.
• Working experience on NoSQL databases like HBase for fault-tolerant way of storing sparse data sets, with functionality and implementation and used built-in functions which are extensions to SQL which are provided by TeraData.
• Extensive experience across both relational databases and non-relational databases such as Oracle for effectively managing data with high performance, authorized access and failure recovery features and worked with PL/SQL, SQL Server, MySQL, and DB2.
• Gained experience in DATA BRICKS which was a simple collaborative environment to run interactive and scheduled data analysis.
• Gained expertise in ELK STACK can provide faster troubleshooting, security analytics and SPLUNK, that can make information searchable, generate alerts, reports, and visualizations.
• Extensive skills in Tableau Software that helps make Big Data small, and small data insightful and actionable.

TECHNICAL SKILLS

PROGRAMMING LANGUAGES: Java, Scala, Python and Shell Scripting, C#

BIG DATA ECOSYSTEM: Spark, Hive, HBase, SQOOP, Oozie, Storm, Flume, Pig, Kafka, NIFI, Zookeeper, MapReduce

CLOUD: AWS EMR, EC2, S3, RDS, Azure Databricks, Azure Data Factory, GCP

DBMS: SQL Server, MySQL, PL/SQL, Oracle, Cassandra, Vertica, Versant

WEB TECHNOLOGIES: HTML, JavaScript, XML, JQuery, Ajax, CSS

WEB SERVICES: Web Logic, Web Sphere

IDEs: Eclipse, IntelliJ, Visual Studio, WinSCP

DevOps: GitHub, Jenkins, Ansible, Chef, Docker, Nagios, Puppet

OPERATING SYSTEMS: Windows, Unix, Linux, Solaris, CentOS

FRAMEWORKS: MVC, Struts, Maven, Junit, Log4J, ANT, Tableau, Splunk, Aqua-data Studio

J2EE TECHNOLOGIES: Spring, Servlets, J2SE, JSP, JDBC

PROFESSIONAL EXPERIENCE

Confidential, FL

Sr.Data Engineer

Responsibilities:

• Used Spark Scala to import customer information data from Oracle database into HDFS for data processing along with minor cleansing.
• Developed MapReduce jobs to calculate the total usage of data by commercial routers in different locations using Horton work distribution.
• Involved in information gathering for new enhancements in Spark Scala, Production support for field issues and label installs for Hive scripts and MapReduce jobs.
• Developed spark applications in python on distributed environment to load massive number of CSV files with different schema in to Hive tables.
• Used Maven to build rpms from source code in Scala checked out from GIT repository, with Jenkins being the Continuous Integration Server and Artifactory as repository manager.
• Responsible for Setting up UNIX/Linux environments for various applications using shell scripting.
• Stored data in AWS S3 like HDFS and performed EMR programs on data stored in S3.
• Used AWS glue ETL service that consumes raw data from S3 bucket and transforms raw data as per the requirement and write the output to s3 bucket in parquet format for data analytics purpose.
• Developed a Python Script to load the CSV files into the S3 buckets and created AWS S3buckets, performed folder management in each bucket, managed logs, and objects within each bucket.
• Worked in cloud formation to automate AWS environment creation along with the ability to deploy AWS using bill scripts (Boto3 and AWS CLI) air.
• Set up scalability for application servers using command line interface for Setting up and administering DNS system in AWS using Route53.
• Write Python scripts to update content in the database and manipulate files. Involved in building database Model, APIs, and Views utilizing Python technologies to build applications.
• Visualize and manipulate the data using various machine learning libraries like NumPy, SciPy and Pandas in Python scripts for the perfect analysis of data.
• Translated customer business requirements into technical design documents, established specific solutions, and leading the efforts including programming in Spark Scala and testing that culminate in client acceptance of the results.
• Expertise in Object-Oriented Design (OOD) and end-to-end software development experience working on Scala coding and implementing mathematical models in Spark Analytics.
• Optimization of Hive queries using best practices and right parameters and using technologies like Hadoop, YARN, Python, Pyspark.
• Worked on reading and writing multiple data formats like JSON, ORC, parquet on HDFS using Pyspark.
• Created Hive external tables on top of datasets loaded in AWS S3 buckets and created various hive scripts to produce series of aggregated datasets for downstream analysis.
• Used AWS Lambda to perform data validation, filtering, sorting, and other transformations for every data change in a HBase table and load the transformed data to RDS.
• Loading data from different servers to S3 bucket and setting appropriate bucket permissions.
• Configured routing to send JMS files to interact with application for real time data using Kafka.
• Managed Zookeeper for cluster co-ordination and Kafka Offset monitoring.
• Optimized legacy queries to extract the customer information from Oracle.
• Reviewed HDFS usage and system design for future scalability and fault tolerance.
• Strong Experience in implementing Data warehouse solutions in Confidential Redshift.
• Worked on various projects to migrate data from on premise databases to Confidential Redshift and RDS

Environment: HDFS, Spark, Scala, Python, Shell Scripting, Hive, HBase, Oracle, MapReduce, Logstash, Jenkins, Versant,Java, Kafka, Horton works, GIT, ClearCase, Zookeeper, Ansible, AWS.

Confidential

Big Data Engineer

Responsibilities:

• Worked as a Data Engineer with Hadoop Ecosystems components like HBase, Sqoop, Zookeeper, Oozie, Hive and Pig with Cloudera Hadoop distribution.
• Involved in Agile development methodology active member in scrum meetings.
• Worked in Azure environment for development and deployment of Custom Hadoop Applications.
• Involved in start to end process of Hadoop jobs that used various technologies such as Sqoop, PIG, Hive, MapReduce, Spark, and Shell scripts.
• Implemented various Azure platforms such as Azure SQL Database, Azure SQL Data Warehouse, Azure Analysis Services, HDInsight, data Lake and Data Factory.
• Extracted and loaded data into Data Lake environment (MS Azure) by using Sqoop which was accessed by business users.
• Manage and support of enterprise Data Warehouse operation, big data advanced predictive application development using Cloud era & Horton works HDP.
• Developed PIG scripts to transform the raw data into intelligent data as specified by business users.
• Installed Hadoop, Map Reduce, HDFS, Azure to develop multiple Map Reduce jobs in PIG and Hive for data cleansing and pre-processing.
• Migrating an entire oracle database to Big Query and using of power BI for reporting.
• Build data pipelines in airflow in GCP for ETL related jobs using different airflow operators.
• Experience in moving data between GCP and Azure using Azure Data Factory.
• Used Cloud shell SDK in GCP to configure the services Data Proc, Storage,, Big Query.
• Used Spark API over Horton works Hadoop YARN to perform analytics on data in Hive.
• Improved the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
• Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
• Developed a Spark job in Java which indexes data into Elastic Search from external Hive tables which are in HDFS.
• Design and develop ETL integration patterns using Python on spark.
• Develop framework for converting existing power center mappings and to PySpark(python and spark)jobs.
• Create Pyspark frame to bring data from DB2 to Amazon S3.
• Optimize the Pyspark jobs to run kubernetes cluster for faster data processing.
• Provide guidance to development team working on Pyspark as ETL process.
• Performed transformations, cleaning and filtering on imported data using Hive, MapReduce, and loaded final data into HDFS.
• Explored with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
• Import the data from different sources like HDFS/HBase into Spark RDD and developed a data pipeline using Kafka and Storm to store data into HDFS.
• Used Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala and NoSQL databases such as HBase and Cassandra.
• Documented the requirements including the available code which should be implemented using Spark, Hive, HDFS, HBase and Elastic Search.
• Performed transformations like event joins, filter boot traffic and some pre-aggregations using Pig.
• Used windows Azure SQL reporting services to create reports with tables, charts, and maps.
• Executed Hive queries on Parquet tables stored in Hive to perform data analysis to meet the business requirements.
• Configured Oozie workflow to run multiple Hive and Pig jobs which run independently with time and data availability.
• Imported and exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.

Environment: Hadoop 3.0, Azure, Sqoop 1.4.6, PIG 0.17, Hive 2.3, MapReduce, Spark 2.2.1, Shells scripts, SQL, Hortonworks, Python, ML Lib, HDFS, YARN, Java, Kafka 1.0, Cassandra 3.11, Oozie, Agile,PostgreSQL.

Confidential

Hadoop Developer

Responsibilities:

• Involved in the project life cycle including the design, development, and implementation of verifying data received in the data lake.
• Design and Develop ETL Processes in AWS Glue to migrate accidents data from external sources like S3, Text Files into AWS Redshift.
• Analyzed impact changes on existing ETL/ELT processes to ensure timely completion and availability of data in data warehouse for reporting use.
• Translated data access, transformation, and movement requirements into functional requirements and mapping designs.
• Used AWS Lambda to perform data validation, filtering, sorting or other transformations for every data change in a HBase table and load the transformed data to another data store.
• Developed, tested, and tuned performance of complex mappings, transforms, aggregations, joins, enrichment, validations for target data underpinnings.
• Used ranking functions (rank, dense rank, percent rank, row number) and aggregation function (sum, min, max) in spark.
• Built real time streaming pipeline utilizing Kafka, Spark Streaming and Redshift.
• Developed logical and physical data flow models for Informatica ETL applications.
• Added support for AWS S3 and RDS to host static /media files and the database into amazon cloud.
• Worked on creation of customer Docker container images, tagging, and pushing of data images.
• Implemented and analyzed SQL query performance issues in databases.
• Responsible for design development of Spark SQL Scripts based on Functional Requirements and Specifications.
• Analysed the sql scripts and designed it by using Pyspark SQL for faster performance.
• Implemented spark programs/applications in Scala using Spark APIs for Data Extraction, Transformation and Aggregation.
• Used Kafka as a message cluster to pull/push messages into spark for data ingestion, processing, and store the resultant data in AWS S3 Buckets.
• Worked on Kafka Publisher integrated in spark job to capture errors from Spark Application and push into Postgres table.
• Intensively used Python, JSON scripts coding to deploy the Stream sets pipelines into the server build pipeline solutions to integrate data from multiple heterogeneous systems using Stream sets data collectors.
• Worked with DevOps team to Clustered Stream sets Pipeline on EC2 nodes integrated with Spark, Kafka, Postgres running on other instances using SSL handshakes in QA and Production Environments.
• Responsible for sending quality data thru secure channel to downstream systems using role base access control and stream sets.
• Responsible for maintaining quality reference data in Oracle by performing operations such as cleaning, transformation and ensuring Integrity in a relational environment
• Created private cloud using Kubernetes that supports DEV, TEST, and PROD environments.
• Managed Kubernetes charts using Helm. Created reproducible builds of the Kubernetes applications, managed Kubernetes manifest files and managed releases of Helm packages.
• Created and executed a Job Stream and added job definitions in Control-M.
• Written shell scripts to extract data from Unix servers into Hadoop HDFS for long-term storage.

Environment: Hadoop, Spark, Hive, Teradata, Tableau, Linux, Python 2.7, Kafka, AWS S3 Buckets, AWS Glue, Stream Sets, PostgreSQL, AWS EC2, Oracle PL/SQL, Development toolkit (JIRA, Bitbucket/Git, Service now etc.,)

Confidential

Java Developer

Responsibilities:

• Professional experience in development and deployment of various Object oriented and web - based Enterprise Applications using Java/J2EE technologies and working on the complete System Development Life Cycle (SDLC).
• Designed and developed the UI of the website using HTML, Spring Boot, React JS, CSS, and JavaScript.
• Proficient in Core Java, J2EE, JDBC, Servlets, JSP, Exception Handling, Multithreading, EJB, XML, HTML5, CSS3,JavaScript, AngularJS.
• Experience in building fast and scalable network applications using Node Js and for quick startup of a production-grade, stand-alone application used Spring Boot.
• Utilized Spring Boot and java as backend and React JS as frontend and MYSQL as database.
• Designed and developed data management system using MySQL. Built application using Spring JPA for database persistence.
• Expertise in application/web servers like IBM Web Sphere, Web Logic Application Servers, JBoss and Tomcat Web Servers.
• In the backend, worked on persisting the data shown on the screen after uploading an excel sheet to the database for a user.
• Created a dashboard for managers to compare Allocation details based on their monthly time.
• Worked on uploading the feature which would allow the user to upload an excel sheet and convert that data into readable format with the help of React JS.
• Created framework with concepts of spring boot using Spring JPA for database persistence.
• Experienced in developing complex MySQL queries, Procedures, Stored Procedures, Packages and Views in MySQL database.
• Ensured availability and security for database in a production environment.
• Configured, tuned, and maintained MySQL Server database servers.
• Implemented monitoring and established best practices around using react libraries.
• Effectively communicated with the external vendors to resolve queries.

Environment: Java, JavaScript, Spring Boot, CSS, SQL, MySQL, React JS, Apache web server, IBM Web Sphere.

Confidential

Java Developer

Responsibilities:

• Worked on both WebLogic Portal 9.2 for portal development and WebLogic 8.1 for Data services programming.
• Developed the presentation layer using JSP, HTML, CSS and client validations using JavaScript.
• Used GWT to send Ajax request to the server and updating data in UI dynamically.
• Developed Hibernate 3.0 in Data Access Layer to access and update information in database.
• Used JDBC, SQL, and PL/SQL programming for storing, retrieving, manipulating the data.
• Involved in designing and development of the ecommerce site using JSP, Servlets, EJBs, JavaScript and JDBC.
• Used Eclipse 6.0 as IDE for application development configured Struts framework to implement MVC design patterns.
• Validated all forms using Struts validation framework and implemented Tiles framework in the presentation layer.
• Designed and developed GUI using JSP, HTML, and CSS. Worked with JMS for messaging interface.
• Used XML for ORM mapping relations with the java classes and the database.
• Used Subversion as the version control system. Extensively used Log4j for logging the log files.

Environment: Java/J2EE, Oracle, SQL, PL/SQL, JSP, EJB, Struts, Hibernate, WebLogic, HTML, CSS, ServletsUML, Junit, Log4j, Eclipse