Senior Data Engineer Resume
Irving, TexaS
PROFESSIONAL SUMMARY:
- Around 8 yearsof IT experience in a variety of industries, which includes hands on experience in Hadoop, MapReduce, Hive, Spark (Pyspark), Sqoop, Airflow, Autosys, Snowflake, Teradata, Oracle, RDMS, Python, Scala, ETL and Data Visualization.
- Around 8 years hands - on experience building ETL pipelines/ Visualizations / Analytics based quality solutions inhouse using AWS / Open-source frameworks.
- Expertise on coding in different technologies i.e., Python, Scala, Java, Unix shell scripting.
- Well versed experience in Amazon Web Services (AWS) Cloud services like EC2, EMR, S3, IAM, CloudWatch,Microsoft Azure and Google cloud platform.
- Experienced in Airflow and Managed Airflow Platform workflows to automate the jobs based on the crontab guru scheduler in Python.
- Good understanding/knowledge of Hadoop Architecture and its components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, Secondary Name node, and MapReduce concepts.
- Exploring with theSparkfor improving the performance and optimization of the existing algorithms in Hadoop usingSparkContext,PySpark SparkSQL, DataFrames, Pair RDD's,SparkYARN, Resource Manager, Memory Tuning, Data Serialization and Memory management.
- Extensive knowledge in programming with Resilient Distributed Datasets (RDDs).
- Experienced in writing custom Hive UDF's to in corporate business logic with Hive queries.
- Experience in process improvement, Normalization/de-Normalization, data extraction, data cleansing, data manipulation on HIVE.
- In-depth understanding of MapReduce Framework andSparkexecution model.
- Good understanding of Partitions, bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance.
- Experience in writing Sqoop command to import data from Relational database to Hive.
- Experienced in performing analytics on structured and unstructured data using Hive queries.
- Experience in handling different file formats like Text files, Sequence files, Avro and Parquet data files using different SerDe's in Hive.
- Mastered in using different columnar file formats like Parquet, RCFile and ORC formats.
- Expert in writing complicated SQL Queries and database analysis for good performance.
- Having experience in SQL Server and Oracle Database and in writing queries.
- Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.
- Extensively used Pig for data cleansing.
- Developed the Pig UDF's to pre-process the data for analysis.
- Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.
- Used Python scripts to build a workflow in Autosys to automate the tasks.
- Closely working with Business team for gathering the requirements and fully understand the business requirements.
- Involved in development of new features like developing Platform Upgrade Testing and Data Snowflake Telemetry as a part of Engineering Excellence.
TECHNICAL SKILLS:
Big data technologies: Hadoop, MapReduce, Hive, Presto,Apache Spark (PySpark), Sqoop,Apache Airflow, Autosys, Snowflake, Teradata, Oracle, RDMS, Python, Scala, ETL and Data Visualization, Kafka.
Database/Datawarehouse: Oracle, SQL Server, Snowflake, Teradata
Programming languages: Python, Scala, C/C++ and Java
Scripting Languages: UNIX shell scripting
CI/CD Tools: Jenkins, GitHub, Jira, Confluence,Tableau
IDE &Tools: Eclipse, PyCharm, Hue, Greenplum, Putty, Autosys, WinSCP, Athena
Cloud Platform: GCP,AWS, Hortonworks, Cloudera,Microsoft Azure.
PROFESSIONAL EXPERIENCE:
Confidential: Irving,Texas
Senior Data Engineer
Responsibilities:
- Worked on large data files using Pyspark (parquet format files).
- Experience in developing and building scripts in Python and shell scripting languages.
- Good hands-on experience on launching the AWS EMR clusters and EC2 instances and understanding the process of AWS IAM properties.
- Experienced in selecting the right nodes, instance types according to the account limit to increase the performance of the ETL.
- Have good understanding of storing data in S3 buckets and perform read, transformations and actions on S3 data using Spark Dataframes and Spark SQL context on PySpark.
- Experience in Spark with Hadoop platform for processing billions of data which uses in-memory data processing.
- Experience in extracting data from HDFS using Hive,presto and performed data analysis using spark with python.
- Experience in creating hive external tables and good understanding the techniques of partitioning, bucketing in hive and perform joins on hive tables.
- Involved in designing Hive schemas, using performance tuning techniques like partitioning, bucketing.
- Build common scripts to load data into our final object in Snowflake by creating External Tables and Manage Table in snowflake which helps our product analysts and reporting developers to read and extract data from Snowflake.
- Experience in real time streaming data using message-based topics between producers and consumers and enabled Kafka in our data lakes.
- Used Airflow scheduler end to end data processing pipelines and scheduling the workflows.
- Experienced working with Snaplogicplatform integration tool.
- Involved in development of new features like developing Platform Upgrade Testing using Cloudtrail data and Data Snowflake Telemetry as a part of Engineering Excellence.
Environment: AWS EMR, EC2, S3, IAM, PySpark, Scala,CloudTail,CloudWatch, Presto,Airflow,Python & Shell scripting, Teradata, GIT,Apache Spark, Jenkins, Athena, Kafka and Snowflake,SQL.
Confidential
Senior Data Engineer
Responsibilities:
- Worked on large data files using Pyspark (parquet format files).
- Experience in developing and building scripts in Python and shell scripting languages.
- Good hands-on experience on launching the AWS EMR clusters and EC2 instances and understanding the process of AWS IAM properties.
- Experienced in selecting the right nodes, instance types according to the account limit to increase the performance of the ETL.
- Have good understanding of storing data in S3 buckets and perform read, transformations and actions on S3 data using Spark Dataframes and Spark SQL context on PySpark.
- Experience in Spark with Hadoop platform for processing billions of data which uses in-memory data processing.
- Experience in creating hive external tables and good understanding the techniques of partitioning, bucketing in hive and perform joins on hive tables.
- Involved in designing Hive schemas, using performance tuning techniques like partitioning, bucketing.
- Build common scripts to load data into our final object in Snowflake by creating External Tables and Manage Table in snowflake which helps our product analysts and reporting developers to read and extract data from Snowflake.
- Experience in real time streaming data using message-based topics between producers and consumers and enabled Kafka in our data lakes.
- Used Airflow scheduler end to end data processing pipelines and scheduling the workflows.
- Built common frameworks for Data Quality Framework to improve no loss of data while performing ETL’s across different platforms and canary checks in ETL’s.
Environment: HBase, Spark-SQL, Spark-Streaming,Redshift,Kinesis, Python, Tableau, AWS, EC2, S3, IAM,GitHub,Pyspark,Glue,Scala and Jenkins.
Confidential
Data Engineer
Responsibilities:
- Responsible for doing requirement analysis, prepare data model.
- Code implementing for importing the data using Sqoop from RDBMS and Spark Scala for loading the files to Hadoop Hive tables and Spark data frames for transforming the data based on business requirements.
- Shell scripting for automating the Sqoop and Spark jobs. Code quality with Sonar Qube
- Converted Sql queries to Scala Data Frames for better performance optimization
- Responsible for DevOps activities such as code maintenance in GIT, Jenkins.
- Spark tuning activities, creating Control-M jobs
- Enhancements to existing code, based on business proposals on changes
- Automated the process for Unit Testing
Environment: Snowflake, Apache Airflow, Spark, Python, presto,Kafka,SQL, Azure Data Factory, Azure Stream Analytics,Terradata,Yarn,GIT.
Confidential
Junior Data Engineer
Responsibilities:
- Responsible in developing this project from scratch using Python, Spark using python (Pyspark) in Agile model.
- Building ETL’s for extracting, cleaning, transforming, and loading data into data warehouse.
- Involved in building the ETL architecture and Source to Target mapping to load data into Data warehouse.
- Experience in hive partitioning, bucketing and perform joins on hive tables.
- Involved in designing Hive schemas, using performance tuning techniques like partitioning, bucketing.
- ConfiguredSparkstreaming to receive real time data from Kafka and store the stream data to HDFS for persistence and Hive for real time reporting.
- Used Sqoop to import the data to Hadoop Distributed File System (HDFS) from RDBMS.
- Used ETL to develop jobs for extracting, cleaning, transforming, and loading data into data warehouse.
- Involved in building the ETL architecture and Source to Target mapping to load data into Data lakes.
- Involved in writing the shell scripts for exporting log files to Hadoop cluster and automated process.
- Used Autosys and Oozie as job scheduler end to end data processing pipelines and for workflows.
- Developed complex logics and generic code for different validations using Spark.
- Worked on Spark SQL for table level validations and fetching meta data from static tables.
- Trained and lead team of associates and colleagues in understanding the functionality of framework and uplifted them technically in Hadoop, Spark and Bigdata technologies.
- Used bit bucket as common repositories for code sharing.
- Developed project using python and spark from the scratch
- Developed complex logic for the framework which automatically generates multiple reports based on metadata
- Audited the report generation status using Hive-HBase tables
- Trained and lead team associates and colleagues in understanding the functionality of framework and uplifted them technically in Hadoop, Spark and Bigdata technologies.
- Played a major role in continuous build, test, and integration process.
Environment: Hadoop, PySpark, Spark MLlib, Hive, Sqoop,Microsoft Azure, SQL-Server, Informatica, HBase, Tableau.
