SUMMARY

• 8+ years of overall experience as Big Data Engineer, Data Analyst and ETL developer, comprises designing, development and implementation of data models for enterprise - level application.
• Good noledge in Technologies on systems which comprises of massive amount of data running in highly distributive mode in Cloudera, Hortonworks Hadoop distributions and Amazon AWS.
• Hands on experience in using Hadoop ecosystem components like Hadoop, Hive, Pig, Sqoop, HBase, Cassandra, Spark, Spark Streaming, Spark SQL, Oozie, Zookeeper, Kafka, Flume, MapReduce framework, Yarn, Scala and Hue.
• Extensive experience in working with NO SQL databases and its integration Dynamo DB, Cosmo DB, Mongo DB, Cassandra and HBase.
• Good Knowledge on architecture and components of Spark, and efficient in working with Spark Core, Spark SQL, Spark streaming and expertise in building PySpark and Spark-Scala applications for interactive analysis, batch processing and stream processing.
• Experience in configuring Spark Streaming to receive real time data from the Apache Kafka and store the stream data to HDFS and expertise in using spark-SQL with various data sources like JSON, Parquet and Hive.
• Extensively used Spark Data Frames API over Cloudera platform to perform analytics on Hive data and also used Spark Data Frame Operations to perform required Validations in the data.
• Proficient in Python Scripting and worked in stats function with NumPy, visualization using Matplotlib and Pandas for organizing data. Involved in loading the structured and semi structured data into spark clusters using SparkSQL and DataFrames Application programming interface (API).
• Accomplished complex HiveQL queries for required data extraction from Hive tables and written Hive User Defined Functions (UDF's) as required.
• Excellent noledge in using Partitions, bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance.
• Proficient in converting Hive/SQL queries into Spark transformations using Data frames and Data sets.
• Experienced in working with Amazon Web Services (AWS) using EC2 for computing and S3 as storage.
• Capable of using AWS utilities such as EMR, S3 and cloud watch to run and monitor Hadoop and spark jobs on Amazon Web Services (AWS).
• Strong noledge in working with Amazon EC2 to provide a complete solution for computing, query processing, and storage across a wide range of applications.
• Capable in using Amazon S3 to support data transfer over SSL and the data gets encrypted automatically once it is uploaded.
• Skilled in using Amazon Redshift to perform large scale database migrations.
• Ingested data into Snowflake cloud data warehouse using SnowPipe. Extensive experience in working with micro batching to ingest millions of files on Snowflake cloud when files arrives to staging area.
• Worked in developing Impala scripts for extraction, transformation, loading of data into data warehouse.
• Extensive noledge in working with Azure cloud platform (HDInsight, DataLake, DataBricks, Blob Storage, Data Factory, Synapse, SQL, SQL DB, DWH and Data Storage Explorer).
• Design and implement database solutions in Azure SQL Data Warehouse, Azure SQL.
• Implement adhoc analysis solutions using Azure Data Lake Analytics/Store, HDInsight.
• Hands on Experience in using Visualization tools like Tableau, Power BI.
• Experience in importing and exporting the data using Sqoop from HDFS to Relational Database Systems and from Relational Database Systems to HDFS.

TECHNICAL SKILLS

Big Data Ecosystem: HDFS, MapReduce, HBase, Pig, Hive, Sqoop, Kafka, Spark Flume, Cassandra, Impala, Oozie, Zookeeper, MapReduce, Amazon Web Services (AWS), EMR

Cloud Technologies: AWS, Azure, Google cloud platform (GCP)

IDE’s: IntelliJ, Eclipse, Spyder, Jupyter.

Operating Systems: Linux, Unix, Windows 8, Windows 7, Windows Server 2008/2003

Programming languages: Python, Scala, Linux shell scripts, Java Scripting, PL/SQL, Java, Pig Latin, HiveQL

Databases: Oracle, MySQL, DB2, MS-SQL Server, MongoDB, HBASE

Web Dev. Technologies: HTML, XML, JSON, CSS, JQUERY, JavaScript

Java Technologies: Core Java, Servlets, JSP, JDBC, Java Beans, J2EE

Business Tools: Tableau, Power BI

PROFESSIONAL EXPERIENCE

Senior AWS Data Engineer

Confidential, Virginia

Responsibilities:

• Implemented a 'serverless' architecture using API Gateway, Lambda, and Dynamo DB and deployed AWS Lambda code from Amazon S3 buckets.
• Created a Lambda Deployment function, and configured it to receive events from you're S3 bucket.
• Designed the data models to be used in data intensive AWS Lambda applications which are aimed to do complex analysis creating analytical reports for end-to-end traceability, lineage, definition of Key Business elements from Aurora.
• Writing code dat optimizes performance of AWS services used by application teams and provide Code level application security for clients (IAM roles, credentials, encryption, etc.).
• Using SonarQube for continuous inspection of code quality and to perform automatic reviews of code to detect bugs. Managing AWS infrastructure and automation with CLI and API.
• Creating AWS Lambda functions using python for deployment management in AWS and designed, investigated and implemented public facing websites on Amazon Web Services and integrated it with other applications infrastructure.
• Creating different AWS Lambda functions and API Gateways, to submit data via API Gateway dat is accessible via Lambda function.
• Responsible for Building Cloud Formation templates for SNS, SQS, Elastic search, Dynamo DB, Lambda, EC2, VPC, RDS, S3, IAM, Cloud Watch services implementation and integrated with Service Catalog.
• Regular monitoring activities in Unix/Linux servers like Log verification, Server CPU usage, Memory check, Load check, Disk space verification, to ensure the application availability and performance by using cloud watch and AWS X-ray. implemented AWS X-Ray service inside Confidential, it allows development teams to visually detect node and edge latency distribution directly from the service map Tools.
• Design and Develop ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, ORC/Parquet/Text Files into AWS Redshift.
• Automate Datadog Dashboards with the stack through Terraform Scripts.
• Developed file cleaners using Python libraries and made it clean.
• Experience in buildingSnow pipe, In-depth noledge ofData Sharingin SnowflakeDatabase, Schema and Tablestructures.
• Exploring DAG's, their dependencies and logs using Airflow pipelines for automation with a creative approach.
• Designed and implemented a fully operational production grade largescale data solution on Snowflake.
• Utilized Python Libraries like Boto3, NumPy for AWS.
• Used Amazon EMR for map reduction jobs and test locally using Jenkins.
• Data Extraction, aggregations and consolidation of Adobe data within AWS Glue using PySpark.
• Create external tables with partitions using Hive, AWS Athena and Redshift.
• Developed the PySprak code for AWS Glue jobs and for EMR. Install and configured Splunk clustered search head and Indexer, Deployment servers, Deployers. Designing and implementing Splunk - based best practice solutions. Designed and Developed ETL jobs to extract data from Salesforce replica and load it in data mart in Redshift.
• Responsible for Designing Logical and Physical data modelling for various data sources on Confidential Redshift.
• Experienced with event-driven and scheduled AWS Lambda functions to trigger various AWS resources.
• Integrated lambda with SQS and DynamoDB with step functions to iterate through list of messages and updated the status into DynamoDB table.

Senior Azure Data Engineer

Confidential, Utah

Responsibilities:

• Analyze, design and build Modern data solutions using Azure PaaS service to support visualization of data.
• Understand current Production state of application and determine the impact of new implementation on existing business processes.
• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics.
• Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
• Experience in buildingSnow pipe, In-depth noledge ofData Sharingin SnowflakeDatabase, Schema and Tablestructures.
• Exploring DAG's, their dependencies and logs using Airflow pipelines for automation with a creative approach.
• Designed and implemented a fully operational production grade largescale data solution on Snowflake.
• Responsible for estimating the cluster size, monitoring and troubleshooting of the Spark databricks cluster.
• Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
• To meet specific business requirements wrote UDF’s in Scala and Pyspark. Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) dat process the data using the Sql Activity.
• Hands-on experience on developing SQL Scripts for automation purpose. Created Build and Release for multiple projects (modules) in production environment using Visual Studio Team Services (VSTS).

Data Engineer

Confidential, Newark, NJ

Responsibilities:

• Experienced in building and architecting multiple Data pipelines, end to end ETL and ELT process for Data Ingestion and transformation in AWS and Spark.
• Leveraged cloud and GPU computing technologies for automated machine learning and analytics pipelines, such as AWS
• Participated in all phases of data mining; data collection, data cleaning, developing models, validation, visualization, performed Gap analysis provide feedback to the business team to improve the software delivery.
• Worked on a project where me have involved in migrating exiting project dat is from Hadoop to big query and migrating the entire project to GCP
• Data Mining with large datasets of Structured and Unstructured data, Data Acquisition, Data Validation, Predictive modeling, Data Visualization on provider, member, claims, and service fund data.
• In this project me mostly worked on application side where we get input data on daily basis sometimes weekly and monthly.
• Analyzed the SQL scripts and designed the solution to implement using Scala.
• Developed analytical component using Scala, Spark, and Spark Stream.
• Involved in Developing a RESTful API's (Microservices) using Python Flask framework dat is packaged in Docker and deployed in Kubernetes using Jenkins Pipelines.
• Experience in building and architecting multiple Data pipelines, end to end ETL, and ELT processes for Data ingestion and transformation in Pyspark.
• Created reusable Rest API's dat exposed data blended from a variety of data sources by reliably gathering requirements from businesses directly.
• Worked on the development of Data Warehouse, a Business Intelligence architecture dat involves data integration and the conversion of data from multiple sources and platforms.
• Responsible for full data loads from production to AWS Redshift staging environment and worked on migrating EDW to AWS using EMR and various other technologies.
• Experience in Creating, Scheduling, and Debugging Spark jobs using Python. Performed Data Analysis, Data Migration, Transformation, Integration, Data Import, and Data Export through Python.
• Gatheird and processed raw data at scale (including writing scripts, web scraping, calling APIs, writing SQL queries, and writing applications).
• Creating reusable Python scripts to ensure data integrity between the source (Teradata/Oracle) and target system.
• Migrated on-premise database structure to Confidential Redshift data warehouse.
• Created data pipelines for different events to load the data from DynamoDB to AWS S3 bucket and then into HDFS and delivered high success metrics.
• Implemented for authoring, scheduling, and monitoring Data Pipelines using Scala and spark.
• Developed and designed system to collect data from multiple platforms using Kafka and then process it using spark.
• Created modules for spark streaming in data into Data Lake using Spark and Worked with different feeds data like JSON, CSV, XML and implemented Data Lake concept.

Hadoop Developer

Confidential

Responsibilities:

• Worked closely with business, transforming business requirements to technical requirements part of Design Reviews & Daily project scrums and wrote custom MapReduce programs by writing Custom input formats.
• Created Sqoop jobs with incremental load to populate Hive External tables.
• Involved in the development of real time streaming applications using PySpark, Kafka on distributed Hadoop Cluster.
• Worked on Partitioning, Bucketing, Join Optimizations, and query optimizations in Hive.
• Worked closely with business, transforming business requirements to technical requirements.
• Designed and developed Hadoop MapReduce programs and algorithms for analysis of cloud scale classified data stored in Cassandra.
• Optimized the Hive tables using optimization techniques like partitioning and bucketing to provide better performance with HiveQL queries.
• Evaluated data import-export capabilities, data analysis performance of Apache Hadoop framework.
• Involved in installation of HDP Hadoop, configuration of the cluster and the eco system components like Sqoop, Pig, Hive, HBase and Oozie.
• Created reports for BI team using Sqoop to export data into HDFS and Hive.
• Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database system and vice versa.
• Created RDD’s in Spark technology.
• Extracted data from data warehouse (Tera Data) on the spark RDD’s.
• Experience on Spark with Scala/Python.
• Working on stateful Transformation in Spark streaming.
• Worked on Batch processing and Real-time data processing and Spark Streaming using Lambda architecture.
• Worked on Spark SQL UDF’s and Hive UDF’s.

Data Engineer (Cloud, Data Analyst)

Confidential

Responsibilities:

• Participated in requirements gathering and worked closely with the architect and SME’s in designing and modeling.
• Handled data ingestions from various data sources, performed transformations using spark, and loaded data into HDFS.
• Involved in converting Hive/SQL queries into Spark Transformations/Actions using PySpark.
• Moved flat files generated from various feeds to HDFS for further processing.
• Developed ETL tool to load the data from a given source to target using python, PySpark, Sqoop, Unix and hive.
• Created Sqoop incremental imports, landed the data in parquet format in HDFS, transformed to ORC format using PySpark.
• Used Dynamic SQL and cursors and their attributes while developing PL/SQL objects.
• Created reusable utilities and programs in python to perform repetitive tasks such as sending emails, comparing data.
• Created and maintained PLSQL procedures to load data sent in XML files into Oracle tables
• Used Sqoop to import the data from RDBMS to Hadoop Distributed File System (HDFS) and later analysed the imported data using HIVE.
• Created UNIX shell scripts to load data from flat files into Oracle tables.
• Created Hive tables to store the processed results in a tabular format.
• Developed the Sqoop scripts to ingest the data from Oracle, Teradata, DB2 into HDFS and Hive, Developed the python and hive scripts for creating the reports from Hive data.

Environment: Hadoop, Hive, Spark, HDFS, MapReduce, Python, PyCharm, PostgreSQL, Oracle 11g, SQL, PL/SQL TOAD, Unix, SharePoint, Teradata, DB2, SVN, Java, Eclipse