Big Data/ Spark Developer Resume
Atlanta, GA
SUMMARY
- Expert level experience in working with broad range of Amazon Web Services (AWS) cloud services and it's features like Autoscaling, AWS Storage, EC2, ELB, VPC, Cloud formation, Security Groups, Access Control Lists (ACL), AWS Glue, RDS, S3, IAM, Cloud Watch.
- Experience in Source/Versioning Controlling tools such as GitHub (GIT), GitLab and Subversion (SVN) for different branching, merging methodologies for Application requirements.
- Extensive experience in building CI/CD pipelines using GitHub, Jenkins, Ansible and Ansible Tower for end - to-end automation for all builds and deployments to various environments.
- Strong experience in creating Docker Images and handled container clusters with Kubernetes to deploy the Docker Images on multiple Nodes in Development, QA, Staging/Pre-Production and Production Environments.
- Working Knowledge on AWS services like AWS Glue, Athena, Lambda, EC2, VPC, IAM, Route53, S3, SNS, Auto Scaling groups, CloudWatch, Elastic Beanstalk.
- Great expertise in SQL databases such as Oracle DB, MySQL, PostgreSQL, and Teradata.
- Good understanding of NoSQL databases and hands-on work experience in writing applications on NoSQL databases like HBase, Cassandra, andMongoDB.
- Experience in writing Spark SQL scripts and improved the performance by implementing techniques like Partitioning and bucketing on Hive tables.
- Experience in migrating the data from RDBMS to HDFS and vice-versa using Sqoop.
- Extensive experience in importing and exporting data using stream processing platforms Kafka.
- Worked in AWS environment for development and deployment of Spark/MapReduce jobs.
- Good knowledge working on components such as Azure Cloud, Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Azure HDInsight.
- Experience in analyzing, designing, developing ETL strategies, processes, and writing ETL specifications.
- Experience in dealing with various data sources like Oracle PL/SQL, T-SQL, SQL Server 2012, Teradata v15/13.
- Experience on Physical & Logical Data Modeling, Dimensional Modeling using Star and Snowflake Schemas, Data marts, OLAP, FACT & Dimensions tables.
- Experience in creating Mappings and Mapplets transformations like Source Qualifier, Aggregator, Expression, Lookup, Filter, Joiner, Union, Router, Rank, Sequence Generator, SQL, HTTP transformations, and Transaction control using Informatica Power Center 8.6.
- Good understanding oflean principles, including DevOps.
- Experienced in using GIT to maintain code versions and have knowledge in CI/CD tools such as Bamboo, Jenkins for code build and deployment in production environments.
- Good exposure with Agile software development process and familiar with Sprint concept through different stages of project life cycle which includes development, testing, debugging, documentation, end-user training, and production support.
- Aptitude for analyzing, identifying problems, and coming with out of the box solutions.
- Ability to achieve organizational integration, assimilate job requirements, employ new ideas, concepts, methods, and technologies.
- Excellent communication, interpersonal and analytical skills.
TECHNICAL SKILLS
Languages: SAS, Python, R, Shell Scripting, Bash scripting, Spark, Java, C++, YAML, JSON, OOPS, Pyspark, Scala, SQL, Database Design in SQL
Query Languages: SQL, PostgreSQL, T-SQL, PL/SQL, NoSQL
Databases: MongoDB, HBase, Cassandra
Tools: R-Studio, Tableau, MS-Office, Jupyter Notebook, Pycharm
Operating Systems: Windows 7/XP/NT/95/98/2000, UNIX and LINUX, Mac iOS
Cloud Services: AWS-EMR, S3, Hadoop, Pig, Hive, Cassandra, H-base(non-SQL), Apache Kafka
Coursework/Other: Statistical Analysis, Machine Learning, Data Visualization, Data Preparation, Advance Data Mining, Text Mining, Parallel and Cloud Computing, Artificial Intelligence, Recommendation Systems, Time Series Analysis and Forecasting, RDBMS, Deep Learning, Natural Language Processing, Data Structure and Problem Solving, Design and Analysis of Algorithms, Distributed, Parallel system and programming, High Performance Computing, Software Design and Development Methodologies, A/B testing
PROFESSIONAL EXPERIENCE
Confidential, Atlanta, GA
Big Data/ Spark Developer
Environment: Spark (PySpark), Spark SQL, Python, AWS, Cloudera, Hive, Sqoop, Impala, HDFS, Linux, Control-M, RDBMS and various file formats
Responsibilities:
- Implemented Sqoop jobs to load data from RDBMS like MySQL and Teradata into HDFS.
- Developed python scripts in Spark to load the data from different file formats such as Text, CSV, JSON, and Avro files into HDFS.
- Developed snow pipes for continuous injection of data using event handler from AWS (S3 bucket).
- Imported data from AWS S3 into Spark DF, performed transformations and actions on DF's.
- Develop Spark SQL applications in PySpark to build analytics on incoming claims data.
- Created Hive tables on the HDFS data in the data lake and performed data validation on different tables using HiveQL.
- Optimized Hive queries and involved in performance tuning using advance techniques like Partitioning (Static/Dynamic), Bucketing and Optimized joins.
- Implemented Spark UDFs in python for data transformations and aggregations.
- Implemented MapReduce programs for data processing and data transformations. Implemented multiple mappers to handle data from multiple sources.
- Experience in manipulating/analyzing large datasets and finding patterns and insights within structured and unstructured data.
- Experienced in building data pipelines, data ingestion flows using Spark and Hadoop MapReduce.
- Implemented pyspark scripts for data cleansing, transformations, pre-processing, and post-processing using Spark.
- Experienced in writing complex MapReduce programs that work with different file formats like Text, Sequence, XML, JSON, Parquet, and Avro.
- Experience in designing and developing applications in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle (RDBMS).
- Strong experience in writing applications using python using different libraries likePandas,NumPy, SciPy, Matplotlib, etc.
- Good Knowledge in Machine Learning algorithms usingPythonand its concepts as data - preprocessing, Regression, Classification, etc and appropriate model selection techniques.
- Experience in Oozie and workflow scheduler to manage Hadoop jobs by Direct Acyclic Graph (DAG) of actions with control flows.
- Transformed old architect of clients’ verification SQL database to NoSQL Hadoop Hbase database.
- Importing, cleaning, merging, Exploratory Data Analysis, built data pipelines in Python.
- Streamlined data ingestion with pandas, shell scripting, bash scripting, unit testing in python.
- Unit testing, Batch and Stream processing, coding in AWS Boto in python, worked on Airflow, AWS Athena, migrated humungous client data to Redshift.
- Experience in Automating, Configuring, and launching EC2, S3, RDS, CLI, creating security groups, creating EMR cluster using Lambda, auto-scaling, load balancers (ELBs), VPC and managing security groups on AWS.
Confidential, Orlando, FL
Data Engineer
Environment: Cloudera Manager, HDFS, Spark, Hive, Pig, Impala, HiveQL, Kafka, SparkSQL, Oozie.
Responsibilities:
- Collaborate with technology leads and architects in conceptualizing objectives and develop new Hadoop applications to process large datasets in agile environment.
- Responsible for data extraction and data ingestion from different data sources into HDFS by creating ETL pipelines
- Involved in designing data warehouse using Hive external tables for Job Control & Data Quality to track job history and successful completion of production jobs.
- Create Hive external tables for consumption and store data in HDFS partitions as Parquet and Text file formats.
- Implement Kafka scripts and create jobs to import data from RDMS (Oracle, SQL) into HBase tables using incremental and full Refresh imports.
- Develop Scala code to convert Hive/SQL queries into Spark transformations using Spark RDDs, Spark SQL and Datasets and perform aggregations on data stored in memory.
- Develop Scala code to gather the data from HBase and implement machine learning solutions using Spark.
- Design and implement complex HiveQL scripts using hive CDC, window functions, joins and partitions to process complex data sets as per business requirements.
- Utilize SparkSQL to extract and process data by parsing using Datasets or RDDs with transformations and actions (map, flatMap, filter, reduce, reduceByKey).
- Write Exporter module in Go Lang to push YARN applications and Scheduler data onto Prometheus time series database to generate reports.
- Develop exporter framework in Scala to get scheduler, job history data from YARN REST APIs and store into HBase tables, generate custom alerts to monitor SLA bound Hadoop applications.
- Manage and schedule jobs by defining Hive, Spark, Sqoop and Python actions on a Hadoop cluster using Oozie workflows and Oozie Coordinator engine
- Involved in Migration of data from legacy systems to cloud instances such as Azure DataBricks.
- Creating Spark clusters and configuring high concurrency clusters using Azure Databricks to speed up the preparation of high-quality data.
- Monitor and troubleshoot performance of applications and take corrective actions in-case of failures and evaluate possible enhancements to meet SLAs.
Confidential, Boston, MA
Data Engineer
Environment: Hadoop, MapReduce, Horton Works Distribution, ORC, Ambari, Pig, HDFS, Sqoop, SQL, HBase, Eclipse, Autosys.
Responsibilities:
- Responsible for building s0calable distributed data solutions using Hadoop and migrate legacy Retail applications ETL to Hadoop.
- Created MapReduce jobs for Log Analysis, Recommendation, and Analytics.
- Handled HDFS data and created external tables using Hive, in order to analyze visitors per day, page views and most purchased products.
- Composed Pig Scripts to create MapReduce jobs and performed ETL procedures on the data in HDFS.
- Utilized MapReduce and Sqoop to load, combine, store and examine web log data from distinctive web servers.
- Reviewed the HDFS usage and system design for future scalability and fault-tolerance.
- Guaranteed the timely completion of the Unit and Integration Testing, testing effort on the project by collaborating with Business SMEs/IT, interface teams and stakeholders.
- Developed a data pipeline using Kafka to store data into HDFS and performed real-time analytics on the incoming data.
- Generated Pig Latin scripts to sort, group, join and filter the enterprise data.
- Widely used Pig for data cleansing.
- Generated Data model for Hive tables.
- Responsible to manage data originating from diverse sources.
- Used AutoSys to automate the jobs for scheduling, monitoring and Reporting.
- Executed tasks like Installation, Configuration, and Upgrade of LINUX operating systems and SQL database.
- Prepared Project maintenance, Test synopsis, Test result, Test case and Go - Live plan documents for project release.
Data Engineer
Confidential
Responsibilities:
- Performed manipulation/analyzing large datasets and finding patterns and insights within structured and unstructured data.
- Built data pipelines, data ingestion flows using Spark and Hadoop MapReduce.
- Experience in designing and developing applications in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle (RDBMS).
- Strong experience in writing applications using python using different libraries likePandas,NumPy, SciPy, Matplotlib, etc.
- Worked on Machine Learning algorithms usingPythonand its concepts as data - preprocessing, Regression, Classification, etc and appropriate model selection techniques.
- Expertise in SQL databases such as Oracle DB, MySQL, PostgreSQL, and Teradata.
- Worked on NoSQL concepts and databases.
- Hands-on work experience in writing applications on NoSQL databases like HBase, Cassandra, andMongoDB.
- Worked in AWS environment for development and deployment of Spark/MapReduce jobs.
- Experience on Physical & Logical Data Modeling, Dimensional Modeling using Star and Snowflake Schemas, Data marts, OLAP, FACT & Dimensions tables.
- Worked using GIT to maintain code versions and have knowledge in CI/CD tools such as Bamboo, Jenkins for code build and deployment in production environments.
- Transformed old architect of clients’ verification SQL database to NoSQL Hadoop Hbase database.
- Importing, cleaning, merging, Exploratory Data Analysis, built data pipelines in Python.
- Streamlined data ingestion with pandas, shell scripting, bash scripting, unit testing in python.
- Unit testing, Batch and Stream processing, coding in AWS Boto in python, worked on Airflow, AWS Athena, migrated humungous client data to Redshift.
- Experience in Automating, Configuring, and launching EC2, S3, RDS, CLI, creating security groups, creating EMR cluster using Lambda, auto-scaling, load balancers (ELBs), VPC and managing security groups on AWS.
Data Analyst
Confidential
Responsibilities:
- Explored traffic data from databases connecting them with transaction data, and presenting as well as writing report for every campaign, providing suggestions for future promotions.
- Extracted data using SQL queries and transferred it to Microsoft Excel and Python for further analysis.
- Data Cleaning, merging, and exporting the dataset was done in Tableau Prep.
- Imported Legacy data from SQL Server and Teradata into Amazon S3.
- Created consumption views on top of metrics to reduce the running time for complex queries.
- Exported Data into Snowflake by creating Staging Tables to load Data of different files from Amazon S3.
- Incorporated predictive modeling (rule engine) to evaluate the Customer/Seller health score using python scripts, performed computations, and integrated with the Tableau viz.
- Worked with stakeholders to communicate campaign results, strategy, issues or needs.
- Analyzed marketing campaigns from various perspectives including CTR, conversion rates, seasonal/geographical trends, search queries, landing page, conversion funnel, quality score, competitors, distribution channel, etc. to achieve maximum ROI for clients.
- Worked with business to identify the gaps in mobile tracking and come up with the solution to solve
- Analyzed click events of Hybrid landing page which includes bounce rate, conversion rate, jump back rate, List/Gallery view, etc. and provide valuable information for landing page optimization.
- Evaluated the traffic and performance of Daily deals PLA ads and compare those items with non-daily deal items to see the possibility of increasing ROI. suggested improvements and modify existing BI components (Reports, Stored Procedures)
- Understood Business requirements to the core and came up with Test Strategy based on Business rules
- Prepared Test Plan to ensure QA and Development phases are in parallel
- Written and executed Test Cases and reviewed with Business & Development Teams.
- Compare the data in a leaf level process from various databases when data transformation or data loading takes place. I need to analyze and look into the data quality when these types of loads are done (To look for any data loss, data corruption).
- As a part of Data Migration, wrote many SQL Scripts for Mismatch of data and worked on loading the history data from Teradata SQL to snowflake.
- Developed SQL scripts to Upload, Retrieve, Manipulate and handle sensitive data (National Provider Identifier Data I.e. Name, Address, SSN, Phone No) in Teradata, SQL Server Management Studio and Snowflake Databases for the Project
- Worked on to retrieve the data from FS to S3 using spark commands
- Built S3 buckets and managed policies for S3 buckets and used S3 bucket and Glacier for storage and backup on AWS
- Created performance dashboards in Tableau/ Excel / Power point for the key stakeholders
- Involved in Functional Testing, Integration testing, Regression Testing, Smoke testing and performance Testing.
- Tested Hadoop Map Reduce developed in python, pig, Hive
- Created Metric tables, End user views in Snowflake to feed data for Tableau refresh.
- Generated Custom SQL to verify the dependency for the daily, Weekly, Monthly jobs.
- Using Nebula Metadata, registered Business and Technical Datasets for corresponding SQL scripts
- Experienced in working with spark ecosystem using Spark SQL and Scala queries on different formats like text file, CSV file.
- Developed spark code and spark-SQL/streaming for faster testing and processing of data.
- Closely involved in scheduling Daily, Monthly jobs with Precondition/Postcondition based on the requirement.
- Data processing and cleaning techniques carried out to reduce text noise, reduce dimensionality in order to improve the analysis.
- Understand the data visualization requirements from the Business Users.
