We provide IT Staff Augmentation Services!

Sr. Data Engineer Resume

2.00/5 (Submit Your Rating)

Richmond, VA

SUMMARY

  • 8 years of professional experience in IT which includes work experience inBig Data, Hadoop ecosystemrelated technologies.
  • Well versed in Installation, Configuration, Supporting and Managing of Big Data and Underlying infrastructure ofHadoopCluster.
  • Involved in converting Hive/SQL queries into Spark transformations using Spark RDD and Pyspark concepts.
  • Strong experience wif big data processing using Hadoop technologiesMap Reduce, Apache Spark, Apache Crunch, Hive, Pig and Yarn.
  • Good experience wifNoSQLdatabasesHBase, MongoDBand good understanding ofCassandra.
  • Hands on experience in usingApacheSOLR/Lucene.
  • Good knowledge in streaming applications usingApacheKafka.
  • Experience in managing Hadoop clusters usingCloudera Manager tool.
  • Hands on experience wif data acquisition into Hadoop cluster usingSqoopandFlume.
  • Experience in designing both time driven and data driven automated workflows usingOozie.
  • Experience in analyzing data usingSpark SQL,HIVEQL,PIGLatin, Spark/Scalaand customMapReduceprograms in Java.
  • Worked on real time data integration usingKafka,Spark streamingandHBase.
  • Good understanding of cloud configuration in Amazon web services (AWS).
  • ExtendingHIVEandPIGcore functionality by using customUDF’s.
  • Good experience in complete project life cycle (design, development, testing and implementation) of Client Server andWeb applications.
  • Ability to build deployment on AWS, build scripts (Boto 3 & AWS CLI) and automated solutions usingShell and Python.
  • Implemented AWS solutions usingEC2, S3, RDS, EBS, Elastic Load Balancer, Auto scaling groups, AWS CLI.
  • Hands on Experience in application developments likeJava,RDBMSandUNIXShellScripting.
  • Hands on experience inJ2SE, J2EE, JSP, Servlets, EJB, WebLogic, WebSphere, Tomcat, JDBC, PythonandJava Script.
  • Familiarity in working wif popular frameworks likeHibernate,spring, Spring BootandMVC.
  • Good communication skills, work ethics and the ability to work in a team efficiently.

TECHNICAL SKILLS

Hadoop: Hadoop, Map Reduce, HIVE, PIG, Impala SQOOP, HDFS, HBASE, Oozie, Spark, Pyspark, Scala and Mongo DB

Cloud Technologies: AWS Kinesis, Lambda, EMR, EC2, SNS, SQS, Dynamo DB, Step Functions, Glue, Atana, CloudWatch, Functions, Azure SQL Data Warehouse, Databricks and HDInsight

DBMS: Amazon Redshift, Postgres, Oracle 9i, SQL Server, IBM DB2 And Teradata

ETL Tools: Data Stage, Talend and ABInitio

Deployment Tools: Git, Jenkins, Terraform and CloudFormation

Programming Language: Python, Scala, PL/SQL

Scripting: Unix Shell and Bash scripting

PROFESSIONAL EXPERIENCE

Confidential, Richmond, VA

Sr. Data Engineer

Responsibilities:

  • Extracted data using Sqoop Import query from multiple databases and ingest into Hive tables.
  • Nomination QA Application is developed using Scala/spark/dataframes to read data from Hive Tables on YARN Framework.
  • WroteKafkaproducers to stream the data from external rest APIs toKafkatopics.
  • Evaluated and improved application performance wif Spark.
  • Worked wif analyst to determine and understand business requirements.
  • Experienced in developing scripts for doing transformations usingScala.
  • Implementing new dimensions into spark application upon on the business requirements.
  • Integrated various business validation & prep flag rules from campaign managers and business analyst on the data.
  • Hands on experience in AWS Cloud in various AWS services such as Redshift cluster, Route 53 domain configuration.
  • On demand secure EMR launcher wif custom spark submit steps using S3 Event, SNS, KMS and Lambda function.
  • Extensive knowledge of working on NiFi.
  • Migrated an existing on - premises application to AWS.
  • Used AWS services like EC2 and S3 for small data sets.
  • Used Cloud watch logs to move application logs to S3 and create alarms based on a few exceptions raised by applications
  • Implemented Spark Sql to update queries based on the business requirements.
  • Responsible to store processed data into MongoDB.
  • Created multi - tier java based multiple web services to read data from MongoDB.
  • Deployed Spark application and java web services in pivotal cloud foundry.
  • Encoded and decoded json objects using PySpark to create and modify the dataframes in Apache Spark.
  • UsedAUTOMICjob scheduler for scheduling multiple applications in Production.
  • Responsible for designing and developing data ingestion from Kroger using Apache NiFi/Kafka.
  • Handled Prod Deployments and provided production support for fixing the defects.
  • Agile methodology including test-driven and pair-programming concept.
  • Strong communication and analytical skills and a demonstrated ability to handle multiple tasks as well as work independently or in a team.

Environment: Apache Hadoop, Java,Kafka,Scala, Apache Spark,PySpark,NiFi, AWS, Apache Hive, IntelliJ, Maven, Team City, Automic, Apache NiFi and Git Hub.

Confidential - Phoenix, Arizona

Sr. Data Engineer

Responsibilities:

  • Written shell scripts to extract data from Unix servers into Hadoop HDFS for long-term storage.
  • Implemented Micro Services architecture using spring boot framework.
  • DevelopedSparkapplications usingScalautilizingData framesandSpark SQLAPI for faster processing of data
  • Created Messaging queues using RabbitMQ to read data from HDFS to process the data.
  • Written Spark Application to implement Slowly changing dimensions (SCD Type me).
  • Created Oozie workflow in process to automate the spark application.
  • Written pig script to load processed data from HDFS into MongoDB.
  • Built real timedata pipelinesby developingKafkaproducers andSparkstreaming applications for consuming.
  • Involved in file movements betweenHDFSandAWS S3and extensively worked wifS3 bucketinAWS.
  • Used MongoDB to store processed products and commodities data, which can be further down streamed into web application (Green Box/ Zoltar).
  • Consumed the data fromKafkaqueue usingSpark. Configured different topologies forSparkcluster and deployed them on regular basi
  • Deployed Spark application and java web services in pivotal cloud foundry.
  • Agile methodology including test-driven and pair-programming concept.
  • Strong communication and analytical skills and a demonstrated ability to handle multiple tasks as well as work independently or in a team.

Environment: Apache Hadoop,Scala, Apache Spark, Apache Pig, Oozie,Java, MapReduce, Spring Boot,AWS, Kafka, Pivotal Cloud Foundry, JUnit, IntelliJ, Maven, Team City, and Git Hub.

Confidential, Sunnyvale, CA

Big Data Engineer

Responsibilities:

  • DevelopedCrawlersjava ETL framework to extract data from Cerner clients database and Ingest into HDFS &HBase for Long Term Storage.
  • Written data processing pipelines inApache Crunchto standardize and normalize the data and store the normalized data inHBase.
  • Create ETL Pipelines usingApache Crunchto read data from HBase and calculate the KPI metrics and store the data in HDFS, and Bulk load data into HBase.
  • Experience in ETL pipeline from HDFS to Vertica.
  • Working on migrating all the Batch ETL Crunch pipelines to in-memorySpark Pipelinesfor quicker run times of the jobs.
  • CreatedOozie workflowsto manage the execution of the crunch jobs and vertica pipelines.
  • Cluster tuning to improve performance.
  • Create Integration tests to check the validity of the data being crawled.
  • Worked wif other teams to determine and understand business requirement.
  • CreateHivebased reports to support the application metrics (These will be used by UI teams for reports).
  • Implementing new drill to detail dimensions into the data pipeline upon on the business requirements.
  • Hands on experience wifApache SOLRfor indexing HBase tables and querying the indexes.
  • Worked wif multiple teams in resolving production issues.
  • Deployment automation viaChef.
  • UsedsparkSQLfor reading data from external sources and processes the the data usingScalacomputation framework.
  • Handled Prod Deployments and provided production support for fixing the defects.
  • Responsible for creating business layer and its underlying data foundation and connection layer usingInformation Design Tool.
  • Agile methodology including, test-driven and pair-programming concept.
  • Strong communication and analytical skills and a demonstrated ability to handle multiple tasks as well as work independently or in a team.

Environment: Apache Hadoop, Apache Crunch, HBase, Kafka,Apache Hive, Oozie, Chops, HDFS, Apache SOLR, Splunk, Chef, JUnit, Ruby, Eclipse, Maven, Jenkins, and Git Hub.

Confidential, Atlanta, GA

Java/ J2EE/ Hadoop Developer

Responsibilities:

  • Involved in review of functional and non-functional requirements.
  • Installed and configuredHadoopMapReduceandHDFS.
  • Acquired good understanding and experience ofNoSQLdatabases such asHBase and Cassandra.
  • Installed and configured Hive and also implemented various business requirements by writingHiveUDFs.
  • Extensively worked on user interface for few modules usingHTML,JSP’s,JavaScript, PythonandAjax.
  • Generated Business Logic usingservlets, Session beans and deployed them on Web logic server.
  • Created complexSQLqueriesandstoredprocedures.
  • Developed the XML schema and Web services for the data support and structures.
  • Implemented the Web service client for login verification, credit reports and applicant information using Apache Axis 2 web service.
  • Responsible for managing data coming from different sources.
  • Used Hibernate ORM framework wif spring framework for data persistence and transaction management.
  • Usedstrutsvalidation framework for form level validations.
  • Wrote test cases in JUnit for unit testing of classes.
  • Provided technical support for production environments resolving the issues, analyzing the defects, providing and implementing the solution defects.
  • Built and deployed Java application into multiple Unix based environments and produced both unit and functional test results along wif release notes

Environment: Hadoop, HBase, Hive, Java, Eclipse, J2EE 1.4,MapReduce, Struts 1.3, JSP, Servlets 2.5, WebSphere 6.1, HTML, XML, ANT 1.6, Python, JavaScript, Junit 3.8.

Confidential

Java/J2EE Developer

Responsibilities:

  • Assisted in gathering the Customer Requirements and preparingRequirements Document.
  • Selected appropriate J2EE Patterns for implementing business applications.
  • Developed Solution Alternatives, Solution Recommendation, analysis-level as well as design level Package Diagram, Deployment Diagram, Class Diagram, Sequence/Activity diagram, Component Diagram, and non-functional design considerations deliverables.
  • Part of the team, working on thearchitecture and design of the application, stressed on code reuse, maintainability, scalability and high availability.
  • AppliedOOAD/UML, and design patterns during the end-to-end J2EE based design.
  • Developedsession beansthat host Business Logic and Performs DB Access.
  • Developed the Presentation layer using theMVC paradigm, implemented byStruts.
  • Wrote theDAO (Data Access Object) component using JDBCto access the database.
  • Involved in the development of various components of the application.
  • UsedLog4J utilityto log error, info and debug messages.
  • UsedCVS for versioningand concurrent access to the project files.
  • UsedJava scriptfor validation of page data in the JSP pages.
  • Wrote variousSQL Queries, Stored Procedures, triggers, functions.
  • Conducting code inspection for programs developed by team members, whenever required.
  • Implemented test first programming approach usingJUNIT as used JMock.
  • DevelopedCrystal reportsusing Crystal 10 Enterprise.

Environment: Jakarta Struts Framework, JSP, CVS, JUNIT, JMOCK, Tiles, Tag Libraries, HTML, Java Script, Log4J, ANT, Web Logic 8.1, JDBC, Eclipse 3.0.

We'd love your feedback!