We provide IT Staff Augmentation Services!

Sr Data Engineer Resume

5.00/5 (Submit Your Rating)

Dallas, TX

SUMMARY

  • 8+ Years of professional IT experience in Analyzing, Designing, Development, Testing, Deployment, Integration, and Maintenance of web based and Client/Server applications using Java and Big Data technologies (Hadoop and Spark).
  • Experience in Big Data and Apache Hadoop ecosystem components like Map - Reduce, HDFS, Sqoop, Flume, Spark, Spark Streaming, Pig, Hive, HBase, Impala, Oozie, Kafka, and Zookeeper.
  • Excellent understanding/ knowledge of Hadoop architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, Resource Manager and Map Reduce.
  • Experience in importing and exporting data using Sqoop from HDFS to Relational database systems and Vice-versa.
  • Worked on data processing and transformations and actions in spark by using Python (Pyspark) language.
  • Experience in Partitioning, Bucketing, Join Optimizations and Query Optimizations in Hive.
  • Experience in integrating HBase in accordance with Hive for real time low latency queries.
  • Hands on experience with Spark Core, Spark SQL and Data Frames/Data Sets/RDDs.
  • Experience in developing Apache Spark programs using Java, Scala.
  • Strong knowledge in Spark Streaming and Kafka.
  • Developed Oozie workflows by integrating all tasks relating to a project and schedule the jobs as per requirements.
  • Experience in storing the data in Hadoop Cluster in Avro, Parquet file formats.
  • Experience in writing Impala scripts for fast access to data.
  • Good experience with Apache Spark, Spark SQL, and Spark streaming with the use of Scala & Pythonprogramming languages.
  • Developed data pipeline using Flume, Sqoop, Pig and Java map reduce to ingest customer behavioral data and financial histories into HDFS for analysis
  • Transform and analyze the data using Pyspark, HIVE, based on ETL mappings
  • Good exposure to CI/CD tools like Jenkins, Gitlab pipelines, Docker, Kubernets
  • Diverse experience in working with variety of Database like Oracle and MySQL.
  • Experience working with NoSQL databases such as HBase and Cassandra to store structured, semi-structured and unstructured data.
  • Experienced in using IDEs and Tools like Eclipse, GitHub, Jenkins, Maven and IntelliJ.
  • Good knowledge on various scripting languages like Linux/Unix shell scripting.
  • Good knowledge on Impala, SparkSQL, Kafka, Hue and AWS.
  • Good understanding ofHadoop YARNArchitecture and its daemons such as Resource manager, Node Manager, Application master, Job Tracker, Task Tracker, Containers, Name Node, Data Node and MapReduce v1 & v2 concepts.
  • Experience data processing like collecting, aggregating, moving from various sources using Apache Flume and Kafka
  • Well versed with different SDLC methodologies with Agile and waterfall models.
  • Experience in Amazon AWS concepts like EMR, S3, EC2, RedShift, Athena.
  • Knowledge of job workflow scheduling and monitoring tools likeOozieandZookeeper.
  • Strong Organizational Skills with ability to work with individuals as well as teams from different backgrounds.

TECHNICAL SKILLS

Big Data: Hadoop, HDFS, MapReduce, Hive, Pig, Sqoop, Flume, Oozie, Avro, Spark, Spark Streaming, Kafka, YARN, Zookeeper, HBase, Impala, ETLHadoop Distributions Cloudera, Hortonworks.

Databases: SQL Server, MySQL, Oracle, Hbase, Cassandra, MongoDB

Languages: Java, HTML, Scala, SQL, Unix Shell Script, Python.

FRAME WORKS: MVC, Struts, Spring.

Build Tools: SBT, Jenkins, Maven.

IDE’s: Eclipse, Intellij.

Cloud Technologies: AWS EMR, S3 EC2, RedShift, Glue

Version Control: GIT, SVN.

Defects Triage: JIRA and Bugzilla

Operating Systems: Windows, UNIX, LINUX, Ubuntu, Cent OS

File Formats: Avro, JSON, Parquet, XML, CSV

PROFESSIONAL EXPERIENCE

Confidential

Sr Data Engineer

Responsibilities:

  • As aData EngineerI have involved to migrate all the services from in-house tocloud (AWS).This includes building adata lakeas acloud-basedsolution inAWSusingAmazonS3and makes it a single source of truth.
  • Gathered data and performed analytics usingAWS stack (EMR, EC2, S3, RDS, Lambda, Redshift).
  • Utilized Apache Spark with Python to develop and execute Big Data Analytics and Machine learning applications, executed machine Learning use cases under Spark ML and Mllib.
  • Developed Spark/Scala, Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for big data resources
  • Transform and analyze the data using Pyspark, HIVE, based on ETL mappings
  • Created data frames in a SPARK SQL from data in HDFS and did transformations analyzed the data and stores back into the HDFS
  • Worked on persist and cache to store the required RDDs on the memory and to use when it is necessary on the other transformations
  • Integrated the spark with Hadoop eco systems and stores the data Hadoop file system(HDFS)
  • Worked extensively on creating combiners, Partitioning, distributed cache to improve the performance of Map Reduce jobs.
  • Worked on Data Serialization formats for converting Complex objects into sequence bits by using AVRO, JSON, CSV formats
  • Developed data pipeline using Flume, Sqoop, Pig and Java map reduce to ingest customer behavioral data and financial histories into HDFS for analysis
  • Migration of data includes various data types likeStreaming data, structured dataandunstructured datafrom various sources and also includes legacy data migration.
  • Migrated on premise database structure toRedshiftdatawarehouse.
  • Migration of Data fromHivetoPresto DBfor faster query execution and data retrieval times.
  • Utilized AWS services with focus on big data analytics, enterprisedata warehouseandbusiness intelligencesolutions to ensure optimalarchitecture,scalability,flexibility
  • Used AWS services like EC2 and S3 for small data sets.
  • Performed CI/CD operations with Gitlab pipelines, Jenkins, Docker, Kubernetes.
  • DesignedAWS architecture,Cloud migration,AWS EMR,Dynamo DB,Redshiftand event processing using lambda function.
  • Involved in converting Hive/SQL queries into Spark transformations using Spark RDD and Pyspark concepts.
  • Wrote variousdata normalizationjobs for newdataingestedintoRedshift.
  • UsedPrestoin-built connectors forRedshiftandHiveto preparedatasetsfor applying advanced analytics (ML) on certain use cases.
  • Implemented performance optimizations on thePresto SQLQueries for improving query retrieval times.
  • Encoded and decoded json objects using PySpark to create and modify the dataframes in Apache Spark.
  • BuiltNoSQLsolution for non-structural data usingAWS Dynamo DB services.
  • UsedAWSgluecatalog with crawler to get the data fromS3and performSqlqueryoperations.
  • Builtdata warehousingsolutions onanalytics/reportingusingAWS Redshift service.
  • DevelopedPythonprograms to consume data fromAPIsas part of several data extraction processes and store the data inAWS S3.
  • Data Extraction, aggregations and consolidation of Adobe data within AWS Glue using PySpark.
  • Create external tables with partitions using Hive, AWS Athena and Redshift.
  • UsedFlumeto collect, aggregate, and store the web log data from different sources like web servers, mobile and network devices and pushed toHDFS.
  • Developed the code for Importing and exporting data intoHDFSandHiveusingSqoop.
  • Developed a data pipeline using Kafka, Spark and Hive to ingest, transform and analyzing data.
  • CreatedHive External tablesto stage data and then move the data from Staging to main tables.
  • WroteHive join queryto fetch info frommultiple tables, writing multiple MapReducejobs to collect output fromHive.
  • Created spark jobs to apply data cleansing/data validation rules on new source files in Inbound bucket and reject records to reject-data S3 bucket.
  • UsedHiveto analyze the partitioned and bucketed data and compute various metrics for reporting on the dashboard.
  • Used Query execution plans inPrestofor tuning thequeriesthat are integrated as data sources forTableaudashboards.
  • Maintained Tableau functional reports based on user requirements.
  • DevelopedSqoopjobs for data ingestion, incremental data loads from RDBMS toSnowflake.
  • Designed ofRedshift Data modeland working on theRedshiftperformance improvements that helps faster query retrieval and also improves the dependentreporting/analytics layers.
  • Created action filters, parameters, and calculated sets for preparing dashboards and worksheets in Tableau.
  • Developed data transition programs fromDynamo DBtoAWS Redshift (ETL Process)usingAWS
  • Lambdaby creating functions inPythonfor the certain events based on use cases.
  • Successful experience in a Fast-paced Startup environment with complete life cycle of product development (SDLC) and Agile/Scrum.

Environment: Hadoop 3.2, Spark, Scala, AWS, RedShift, Glue, Amazon S3, EC2, EMR, Athena, Data Pipeline, Dynamo DB, NoSQL, Spark SQL, Kafka, PySpark, Streaming, GIT, Snowflake, HDFS, Hive 2.3, Sqoop, Map Reduce, Docker, Tableau, ETL andPython.

Confidential, Dallas, TX

Hadoop Developer

Responsibilities:

  • Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase database and SQOOP.
  • Installed Hadoop, Map Reduce, HDFS, and Developed multiple map reduce jobs in PIG and Hive for data cleaning and pre-processing.
  • Developed data pipeline using Flume, Sqoop to ingest cargo data and customer histories into HDFS for analysis.
  • Worked on importing data from HDFS to Oracle database and vice-versa using SQOOP to configure Hive Meta store with MySQL, which stores the metadata for Hive tables.
  • Involved in gathering requirements from client and estimating time line for developing complex queries using HIVE and IMPALA for logistics application.
  • Developed Kafka producer and consumers, HBase clients, Spark and Hadoop MapReduce jobs along with components on HDFS, Hive.
  • Developed Simple to complex Map/reduce Jobs using Scala and Java in Spark.
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team
  • Written Hive and Pig scripts as per requirements to automate the workflow using shell scripts.
  • Participated in Rapid Application Development and Agile processes to deliver new cloud platform services.
  • UsedAWS serviceslikeEC2andS3for small data sets processing andstorage,involvedin Maintaining theHadoopclusteronAWS EMR.
  • Involved in creating Hive tables, loading with data and writing hive queries that will run internally in map reduce way.
  • Created continuous integration and continuous delivery (CI/CD) pipeline on AWS that helps to automate steps in software delivery process.
  • Responsible for design development of Spark SQL Scripts based on Functional Specifications.
  • Exploring with the Spark improving the Performance and Optimization of the existing algorithms in Hadoop.
  • Exported data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose.
  • Responsible for writing Hive Queries for analyzing data in Hive warehouse using Hive Query Language (HQL).
  • Importing and exporting Data from MySQL/Oracle to HiveQL Using SQOOP.

Environment: Hadoop, AWS S3, EC2, EMR, Data Pipeline, Spark, Scala, Kafka, Hive, Zookeeper, Map Reduce, Sqoop, Pig, HCatalog, Unix, Java, Oracle, SQL Server, MYSQL, Oozie, Python.

Confidential

Hadoop Developer

Responsibilities:

  • Worked on loading disparate data sets coming from different sources to BDpaas (HADOOP) environment using SQOOP.
  • Loading datasets from two different sources like Oracle, MySQL to HDFS and Hive respectively.
  • Developed UNIX scripts in creating Batch load for bringing huge amount of data from Relational databases to BIGDATA platform.
  • Importing of data from various data sources, performed transformations, loaded data into HDFS and Extracted the data from MySQL into HDFS using Sqoop
  • Involved in analyzing data coming from various sources and creating Meta-files and control files to ingest the data in to the Data Lake.
  • Involved in configuring batch job to perform ingestion of the source files in to the Data Lake.
  • Developed Pig queries to load data to HBase.
  • Leveraged Hive queries to create ORC tables.
  • Developed HIVE scripts for analyst requirements for analysis.
  • Worked extensively on Hive to create, alter and drop tables and involved in writing hive queries.
  • Created and altered HBase tables on top of data residing in Data Lake.
  • Created Views from Hive Tables on top of data residing in Data Lake.
  • Created Reports with different Selection Criteria from Hive Tables on the data residing in Data Lake.
  • Worked closely with scrum master and team to gather information and perform daily activities.
  • Worked with Systems Analyst and business users to understand requirements Environment: CDH, Hadoop, MapReduce, HDFS, Hive, Sqoop

Environment: Hadoop, MapReduce, Mainframe, Oracle, Linux, Hive, HDFS, DMX-h, Sqoop, Autosys.

We'd love your feedback!