We provide IT Staff Augmentation Services!

Sr Data Engineer Resume

3.00/5 (Submit Your Rating)

Raleigh, NC

SUMMARY

  • 8+ Years of professional IT experience in Analyzing, Designing, Development, Testing, Deployment, Integration, and Maintenance of web based and Client/Server applications using Java and Big Data technologies (Hadoop and Spark).
  • Experience in Big Data and Apache Hadoop ecosystem components like Map - Reduce, HDFS, Sqoop, Flume, Spark, Spark Streaming, Pig, Hive, HBase, Impala, Oozie, Kafka, and Zookeeper.
  • Excellent understanding/ noledge of Hadoop architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, Resource Manager and Map Reduce.
  • Experience in importing and exporting data using Sqoop from HDFS to Relational database systems and Vice-versa.
  • Worked on data processing and transformations and actions in spark by using Python (Pyspark) language.
  • Experience in Partitioning, Bucketing, Join Optimizations and Query Optimizations in Hive.
  • Experience in integrating HBase in accordance with Hive for real time low latency queries.
  • Hands on experience with Spark Core, Spark SQL and Data Frames/Data Sets/RDDs.
  • Experience in developing Apache Spark programs using Java, Scala.
  • Strong noledge in Spark Streaming and Kafka.
  • Developed Oozie workflows by integrating all tasks relating to a project and schedule teh jobs as per requirements.
  • Experience in storing teh data in Hadoop Cluster in Avro, Parquet file formats.
  • Experience in writing Impala scripts for fast access to data.
  • Good experience with Apache Spark, Spark SQL, and Spark streaming with teh use of Scala & Pythonprogramming languages.
  • Developed data pipeline using Flume, Sqoop, Pig and Java map reduce to ingest customer behavioral data and financial histories into HDFS for analysis
  • Transform and analyze teh data using Pyspark, HIVE, based on ETL mappings
  • Good exposure to CI/CD tools like Jenkins, Gitlab pipelines, Docker, Kubernets
  • Diverse experience in working with variety of Database like Oracle and MySQL.
  • Experience working with NoSQL databases such as HBase and Cassandra to store structured, semi-structured and unstructured data.
  • Experienced in using IDEs and Tools like Eclipse, GitHub, Jenkins, Maven and IntelliJ.
  • Good noledge on various scripting languages like Linux/Unix shell scripting.
  • Good noledge on Impala, SparkSQL, Kafka, Hue and AWS.
  • Good understanding ofHadoop YARNArchitecture and its daemons such as Resource manager, Node Manager, Application master, Job Tracker, Task Tracker, Containers, Name Node, Data Node and MapReduce v1 & v2 concepts.
  • Experience data processing like collecting, aggregating, moving from various sources using Apache Flume and Kafka
  • Well versed with different SDLC methodologies with Agile and waterfall models.
  • Experience in Amazon AWS concepts like EMR, S3, EC2, RedShift, Atana.
  • Knowledge of job workflow scheduling and monitoring tools likeOozieandZookeeper.
  • Strong Organizational Skills with ability to work with individuals as well as teams from different backgrounds.

TECHNICAL SKILLS

Big Data: Hadoop, HDFS, MapReduce, Hive, Pig, Sqoop, Flume, Oozie, Avro, Spark, Spark Streaming, Kafka, YARN, Zookeeper, HBase, Impala, ETLHadoop Distributions Cloudera, Hortonworks.

Databases: SQL Server, MySQL, Oracle, Hbase, Cassandra, MongoDB

Languages: Java, HTML, Scala, SQL, Unix Shell Script, Python.

FRAME WORKS: MVC, Struts, Spring.

Build Tools: SBT, Jenkins, Maven.

IDE’s: Eclipse, Intellij.

Cloud Technologies: AWS EMR, S3 EC2, RedShift, Glue

Version Control: GIT, SVN.

Defects Triage: JIRA and Bugzilla

Operating Systems: Windows, UNIX, LINUX, Ubuntu, Cent OS

File Formats: Avro, JSON, Parquet, XML, CSV

PROFESSIONAL EXPERIENCE

Confidential, Raleigh, NC

Sr Data Engineer

Responsibilities:

  • As aData EngineerI have involved to migrate all teh services from in-house tocloud (AWS).dis includes building adata lakeas acloud-basedsolution inAWSusingAmazonS3and makes it a single source of truth.
  • Gathered data and performed analytics usingAWS stack (EMR, EC2, S3, RDS, Lambda, Redshift).
  • Utilized Apache Spark with Python to develop and execute Big Data Analytics and Machine learning applications, executed machine Learning use cases under Spark ML and Mllib.
  • Developed Spark/Scala, Python for regular expression (regex) project in teh Hadoop/Hive environment with Linux/Windows for big data resources
  • Transform and analyze teh data using Pyspark, HIVE, based on ETL mappings
  • Created data frames in a SPARK SQL from data in HDFS and did transformations analyzed teh data and stores back into teh HDFS
  • Worked on persist and cache to store teh required RDDs on teh memory and to use when it is necessary on teh other transformations
  • Integrated teh spark with Hadoop eco systems and stores teh data Hadoop file system(HDFS)
  • Worked extensively on creating combiners, Partitioning, distributed cache to improve teh performance of Map Reduce jobs.
  • Worked on Data Serialization formats for converting Complex objects into sequence bits by using AVRO, JSON, CSV formats
  • Developed data pipeline using Flume, Sqoop, Pig and Java map reduce to ingest customer behavioral data and financial histories into HDFS for analysis
  • Migration of data includes various data types likeStreaming data, structured dataandunstructured datafrom various sources and also includes legacy data migration.
  • Migrated on premise database structure toRedshiftdatawarehouse.
  • Migration of Data fromHivetoPresto DBfor faster query execution and data retrieval times.
  • Utilized AWS services with focus on big data analytics, enterprisedata warehouseandbusiness intelligencesolutions to ensure optimalarchitecture,scalability,flexibility
  • Used AWS services like EC2 and S3 for small data sets.
  • Performed CI/CD operations with Gitlab pipelines, Jenkins, Docker, Kubernetes.
  • DesignedAWS architecture,Cloud migration,AWS EMR,Dynamo DB,Redshiftand event processing using lambda function.
  • Involved in converting Hive/SQL queries into Spark transformations using Spark RDD and Pyspark concepts.
  • Wrote variousdata normalizationjobs for newdataingestedintoRedshift.
  • UsedPrestoin-built connectors forRedshiftandHiveto preparedatasetsfor applying advanced analytics (ML) on certain use cases.
  • Implemented performance optimizations on thePresto SQLQueries for improving query retrieval times.
  • Encoded and decoded json objects using PySpark to create and modify teh dataframes in Apache Spark.
  • BuiltNoSQLsolution for non-structural data usingAWS Dynamo DB services.
  • UsedAWSgluecatalog with crawler to get teh data fromS3and performSqlqueryoperations.
  • Builtdata warehousingsolutions onanalytics/reportingusingAWS Redshift service.
  • DevelopedPythonprograms to consume data fromAPIsas part of several data extraction processes and store teh data inAWS S3.
  • Data Extraction, aggregations and consolidation of Adobe data within AWS Glue using PySpark.
  • Create external tables with partitions using Hive, AWS Atana and Redshift.
  • UsedFlumeto collect, aggregate, and store teh web log data from different sources like web servers, mobile and network devices and pushed toHDFS.
  • Developed teh code for Importing and exporting data intoHDFSandHiveusingSqoop.
  • Developed a data pipeline using Kafka, Spark and Hive to ingest, transform and analyzing data.
  • CreatedHive External tablesto stage data and tan move teh data from Staging to main tables.
  • WroteHive join queryto fetch info frommultiple tables, writing multiple MapReducejobs to collect output fromHive.
  • Created spark jobs to apply data cleansing/data validation rules on new source files in Inbound bucket and reject records to reject-data S3 bucket.
  • UsedHiveto analyze teh partitioned and bucketed data and compute various metrics for reporting on teh dashboard.
  • Used Query execution plans inPrestofor tuning thequeriesthat are integrated as data sources forTableaudashboards.
  • Maintained Tableau functional reports based on user requirements.
  • DevelopedSqoopjobs for data ingestion, incremental data loads from RDBMS toSnowflake.
  • Designed ofRedshift Data modeland working on theRedshiftperformance improvements that halps faster query retrieval and also improves teh dependentreporting/analytics layers.
  • Created action filters, parameters, and calculated sets for preparing dashboards and worksheets in Tableau.
  • Developed data transition programs fromDynamo DBtoAWS Redshift (ETL Process)usingAWS
  • Lambdaby creating functions inPythonfor teh certain events based on use cases.
  • Successful experience in a Fast-paced Startup environment with complete life cycle of product development (SDLC) and Agile/Scrum.

Environment: Hadoop 3.2, Spark, Scala, AWS, RedShift, Glue, Amazon S3, EC2, EMR, Atana, Data Pipeline, Dynamo DB, NoSQL, Spark SQL, Kafka, PySpark, Streaming, GIT, Snowflake, HDFS, Hive 2.3, Sqoop, Map Reduce, Docker, Tableau, ETL andPython.

Confidential, St Louis, MO

Data Engineer

Responsibilities:

  • Involved in completeBigdataflow of teh application starting fromdata ingestionfrom upstream toHDFS,processingandanalyzingthe data inHDFS.
  • FollowedAgile&Scrumprinciples in developing teh project
  • DevelopedSparkAPIto import data intoHDFSfromDB2and createdHive tables.
  • UsedSparkAPIoverHortonworksHadoopYARNto perform analytics on data inHiv.
  • ImplementedSparkusingScalaandSparkSQLfor faster testing and processing of data.
  • Importing Large Data Sets fromDB2toHiveTable usingSqoop
  • UsedImpalafor queryingHDFSdata to achieve better performance.
  • Implemented ApachePIGscriptsto load data from and to store data intoHive.
  • CreatedPartitionedandBucketedHivetables inParquet FileFormats withSnappycompression and tan loaded data intoParquet hive tablesfromAvro hivetables.
  • Involved in running all thehive scriptsthroughhive, Impala, HiveonSparkand some throughSparkSQL.
  • Optimize teh Pyspark jobs to run on Kubernetes Cluster for faster data processing.
  • Imported data fromAWS S3and intoSparkRDDand performed transformations and actions onRDD's.
  • DevelopedSpark scriptsby usingScala Shellcommands as per teh requirement.
  • Worked and learned a great deal fromAmazon Web Services (AWS)Cloud serviceslikeEC2,S3,EBS,EMR, RDSandVPC.
  • Worked onAWSDataPipelineto configure data loads fromS3to intoRedshift.
  • UsedJSONschema to define table and column mapping fromS3data toRedshift.
  • Responsible for implementingETLprocess throughKafka-Spark-HBaseIntegration as per teh requirements of customer facingAPI.
  • Worked on reading and writing multiple data formats like JSON,ORC,Parquet on HDFS using PySpark.
  • UsedSpark-SQLto LoadJSONdata and createSchema RDDand loaded it intoHiveTables and handled Structured data usingSparkSQL.
  • Created action filters, parameters, and calculated sets for preparing dashboards and worksheets in Tableau.
  • Worked onBatch processingandReal-time dataprocessing onSparkStreamingusingLambdaarchitecture.
  • DevelopingSparkcodeinScalaandSparkSQLenvironment for faster testing and processing of data and Loading teh data intoSparkRDDand doingIn-memory computationto generate teh output response with less memory usage.
  • Create external tables with partitions using Hive, AWS Atana and Redshift.
  • Developing and maintaining Workflow Scheduling Jobs inOoziefor importing data fromRDBMStoHive.
  • Created Pyspark frame to bring data from DB2 to Amazon S3.
  • UtilizedSparkCore,Spark Streaming andSparkSQL APIforfaster processingofdatainstead of usingMapReduceinJava.
  • Responsible fordata extractionanddata integrationfrom different data sources intoHadoopData Lakeby creatingETLpipelines UsingSpark, MapReduce, Pig,andHive.
  • Load data into Amazon Redshift and use AWS Cloud Watch to collect and monitor AWS RDS instances within Confidential.
  • Involved in convertingHive/SQLqueries intoSparktransformationsusingSpark DataframesandScala.
  • Involved with teh team of fetching live stream data fromDB2toHbasetable usingSpark StreamingandApache Kafka.
  • Load teh data intoSparkRDDand do in memory data Computation to generate teh Output response.
  • DevelopedSqoopjobs for data ingestion, incremental data loads from RDBMS toSnowflake.
  • UsedSparkfor interactive queries, processing ofstreaming dataand integration with popularNoSQLdatabase for huge volume of data.
  • Performed CI/CD operations with Gitlab pipelines, Jenkins, Docker, Kubernetes.
  • Wrote differentpig scriptsto clean up teh ingested data and created partitions for teh daily data.
  • DevelopedSparkprograms withScala,and applied principles offunctional programmingto process teh complexunstructuredandstructureddata sets
  • Involved in convertingHive/SQLqueries intoSpark transformationsusingSpark RDDs,ScalaandPython.
  • Analyzed theSQL scriptsand designed teh solution to implement usingPySpark.
  • Involved in convertingMapReduceprograms intoSpark transformationsusingSpark RDDinScala.
  • Involved in Agile Scrum methodology, Test Driven Development and CI/CD.
  • Git used as a version control tool and Jenkins as Continuous integration(CI) tool.
  • UsedOozieworkflow to co-ordinate pig andHive Scripts.

Environment: HDFS,MapReduce, Hive, Sqoop, HBase, Oozie, Flume, Impala, Kafka, Zookeeper, SparkSQL, Streaming, NoSQL, Spark Dataframes, Data Pipelines, PySpark, Scala, Amazon AWS S3, EC2, EMR, RedShift, Atana, Tableau, Hadoop, Snowflake, Docker, Python, Java, JSON, SQL Scripting and Linux Shell Scripting, Avro, Parquet, Hortonworks, GIT.

Confidential, IL

BigData Engineer

Responsibilities:

  • In depth understanding/noledge ofHadoopArchitectureand various components such asHDFS, Application master, Node Manager, Resource Manager, Name Node, Datanode and MapReduce concepts.
  • Imported required tables fromRDBMStoHDFSusingSqoopand also usedStormandKafkato get real time streaming of data intoHBase.
  • UsedNoSQLdatabaseHbaseand creatingHbase tablesto load large sets ofsemi structureddata coming from various sources.
  • WroteHiveandPig scriptsasETLtool to dotransformations, event joins, filterboth traffic and some pre-aggregations before storing into theHDFS.
  • Developed data pipeline usingFlume, Sqoop, PigandMapReduceto ingest customer behavioral data and purchase histories intoHDFSfor analysis
  • DevelopedSpark codeusingScalaandSpark-SQLfor faster testing and processing of data.
  • UsedSpark APIoverClouderaHadoopYARNto perform analytics on data inHive.
  • Involved in moving all log files generated from various sources toHDFSfor further processing throughFlume.
  • DevelopedJava codetogenerate,compare&mergeAVROschema files.
  • Developed complexMapReducestreaming jobs usingJava languagethat are implemented UsingHiveandPigand usingMapReducePrograms usingJavato perform variousETL, cleaning and scrubbing tasks.
  • Importing and exporting data intoHDFSandHiveusingSqoop. Writing theHIVE queriesto extract teh data processed
  • Developing and runningMap-ReduceJobs onYARNandHadoopclustersto produce daily and monthly reports as per user's need.
  • Teamed up with Architects to designSpark modelfor teh existingMapReducemodel andMigrated MapReducemodels toSpark ModelsusingScala.
  • Created AWS Code Pipeline, a service that builds, tests, and deploys code every time there is a code change, based on teh release process models
  • Created pipeline that uses AWS Code Deploy to deploy applications from an Amazon S3 bucket and AWS Code Commit repository to Amazon EC2 instances running Amazon Linux.
  • ImplementedSparkusingScalaand utilizingSparkCore, Spark StreamingandSparkSQLAPI for faster processing of data instead ofMapReduceinJava.
  • UsedSpark-SQLto LoadJSONdata and create Schema RDD and loaded it intoHive Tablesand handledStructured datausingSparkSQL.
  • Expertise in implementing DevOps culture through CI/CD tools like Repos, Code Deploy, Code Pipeline, GitHub
  • Handled importing of data from various data sources, performed transformations usingHive, MapReduce, loaded data intoHDFSand Extracted teh data fromMySQLintoHDFSusingSqoop
  • IntegratedApache StormwithKafkato performweb analyticsand to perform click stream data fromKafkatoHDFS.
  • UsedOozieworkflowengine to manage interdependentHadoopjobsand to automate several types of Hadoopjobs such asJava MapReduce Hive, Pig, and Sqoop.
  • Involved in convertingHive/SQLqueries intoSpark transformationsusingSpark RDDs,ScalaandPython.
  • Analyzed large and critical datasets usingCloudera, HDFS, HBase, MapReduce, Hive, Pig, Sqoop, Spark and Zookeeper.
  • Used Git as a version control tool.
  • Expert noledge onMongoDB NoSQLdata modeling, tuning, disaster recovery and backup

Environment: Hadoop, AWS, S3, EC2, EMR, Data Pipeline, Python, HDFS, MapReduce, HBase, Hive, Yarn, Pig, Sqoop, Flume, Zookeeper, Kafka, Scala, Impala, Git, SparkSQL, Spark Core, Spark Streaming, NoSQL, MySQL, Cloudera, ETL, WebLogic, Avro, Cassandra, Oracle, Shell Scripting, Ubuntu.

Confidential, Dallas, TX

Hadoop Developer

Responsibilities:

  • Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase database and SQOOP.
  • Installed Hadoop, Map Reduce, HDFS, and Developed multiple map reduce jobs in PIG and Hive for data cleaning and pre-processing.
  • Developed data pipeline using Flume, Sqoop to ingest cargo data and customer histories into HDFS for analysis.
  • Worked on importing data from HDFS to Oracle database and vice-versa using SQOOP to configure Hive Meta store with MySQL, which stores teh metadata for Hive tables.
  • Involved in gathering requirements from client and estimating time line for developing complex queries using HIVE and IMPALA for logistics application.
  • Developed Kafka producer and consumers, HBase clients, Spark and Hadoop MapReduce jobs along with components on HDFS, Hive.
  • Developed Simple to complex Map/reduce Jobs using Scala and Java in Spark.
  • Exported teh analyzed data to teh relational databases using Sqoop for visualization and to generate reports for teh BI team
  • Written Hive and Pig scripts as per requirements to automate teh workflow using shell scripts.
  • Participated in Rapid Application Development and Agile processes to deliver new cloud platform services.
  • UsedAWS serviceslikeEC2andS3for small data sets processing andstorage,involvedin Maintaining theHadoopclusteronAWS EMR.
  • Involved in creating Hive tables, loading with data and writing hive queries that will run internally in map reduce way.
  • Created continuous integration and continuous delivery (CI/CD) pipeline on AWS that halps to automate steps in software delivery process.
  • Responsible for design development of Spark SQL Scripts based on Functional Specifications.
  • Exploring with teh Spark improving teh Performance and Optimization of teh existing algorithms in Hadoop.
  • Exported data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose.
  • Responsible for writing Hive Queries for analyzing data in Hive warehouse using Hive Query Language (HQL).
  • Importing and exporting Data from MySQL/Oracle to HiveQL Using SQOOP.

Environment: Hadoop, AWS S3, EC2, EMR, Data Pipeline, Spark, Scala, Kafka, Hive, Zookeeper, Map Reduce, Sqoop, Pig, HCatalog, Unix, Java, Oracle, SQL Server, MYSQL, Oozie, Python.

Confidential

Hadoop Developer

Responsibilities:

  • Worked on loading disparate data sets coming from different sources to BDpaas (HADOOP) environment using SQOOP.
  • Loading datasets from two different sources like Oracle, MySQL to HDFS and Hive respectively.
  • Developed UNIX scripts in creating Batch load for bringing huge amount of data from Relational databases to BIGDATA platform.
  • Importing of data from various data sources, performed transformations, loaded data into HDFS and Extracted teh data from MySQL into HDFS using Sqoop
  • Involved in analyzing data coming from various sources and creating Meta-files and control files to ingest teh data in to teh Data Lake.
  • Involved in configuring batch job to perform ingestion of teh source files in to teh Data Lake.
  • Developed Pig queries to load data to HBase.
  • Leveraged Hive queries to create ORC tables.
  • Developed HIVE scripts for analyst requirements for analysis.
  • Worked extensively on Hive to create, alter and drop tables and involved in writing hive queries.
  • Created and altered HBase tables on top of data residing in Data Lake.
  • Created Views from Hive Tables on top of data residing in Data Lake.
  • Created Reports with different Selection Criteria from Hive Tables on teh data residing in Data Lake.
  • Worked closely with scrum master and team to gather information and perform daily activities.
  • Worked with Systems Analyst and business users to understand requirements Environment: CDH, Hadoop, MapReduce, HDFS, Hive, Sqoop

Environment: Hadoop, MapReduce, Mainframe, Oracle, Linux, Hive, HDFS, DMX-h, Sqoop, Autosys.

We'd love your feedback!