Sr. Data Engineer Resume
Richmond, VA
SUMMARY
- 8 years of professional experience in IT which includes work experience inBig Data, Hadoop ecosystemrelated technologies.
- Well versed in Installation, Configuration, Supporting and Managing of Big Data and Underlying infrastructure ofHadoopCluster.
- Involved in converting Hive/SQL queries into Spark transformations using Spark RDD and Pyspark concepts.
- Strong experience wif big data processing using Hadoop technologiesMap Reduce, Apache Spark, Apache Crunch, Hive, Pig and Yarn.
- Good experience wifNoSQLdatabasesHBase, MongoDBand good understanding ofCassandra.
- Hands on experience in usingApacheSOLR/Lucene.
- Good knowledge in streaming applications usingApacheKafka.
- Experience in managing Hadoop clusters usingCloudera Manager tool.
- Hands on experience wif data acquisition into Hadoop cluster usingSqoopandFlume.
- Experience in designing both time driven and data driven automated workflows usingOozie.
- Experience in analyzing data usingSpark SQL,HIVEQL,PIGLatin, Spark/Scalaand customMapReduceprograms in Java.
- Worked on real time data integration usingKafka,Spark streamingandHBase.
- Good understanding of cloud configuration in Amazon web services (AWS).
- ExtendingHIVEandPIGcore functionality by using customUDF’s.
- Good experience in complete project life cycle (design, development, testing and implementation) of Client Server andWeb applications.
- Ability to build deployment on AWS, build scripts (Boto 3 & AWS CLI) and automated solutions usingShell and Python.
- Implemented AWS solutions usingEC2, S3, RDS, EBS, Elastic Load Balancer, Auto scaling groups, AWS CLI.
- Hands on Experience in application developments likeJava,RDBMSandUNIXShellScripting.
- Hands on experience inJ2SE, J2EE, JSP, Servlets, EJB, WebLogic, WebSphere, Tomcat, JDBC, PythonandJava Script.
- Familiarity in working wif popular frameworks likeHibernate,spring, Spring BootandMVC.
- Good communication skills, work ethics and the ability to work in a team efficiently.
TECHNICAL SKILLS
Hadoop: Hadoop, Map Reduce, HIVE, PIG, Impala SQOOP, HDFS, HBASE, Oozie, Spark, Pyspark, Scala and Mongo DB
Cloud Technologies: AWS Kinesis, Lambda, EMR, EC2, SNS, SQS, Dynamo DB, Step Functions, Glue, Atana, CloudWatch, Functions, Azure SQL Data Warehouse, Databricks and HDInsight
DBMS: Amazon Redshift, Postgres, Oracle 9i, SQL Server, IBM DB2 And Teradata
ETL Tools: Data Stage, Talend and ABInitio
Deployment Tools: Git, Jenkins, Terraform and CloudFormation
Programming Language: Python, Scala, PL/SQL
Scripting: Unix Shell and Bash scripting
PROFESSIONAL EXPERIENCE
Confidential, Richmond, VA
Sr. Data Engineer
Responsibilities:
- Extracted data using Sqoop Import query from multiple databases and ingest into Hive tables.
- Nomination QA Application is developed using Scala/spark/dataframes to read data from Hive Tables on YARN Framework.
- WroteKafkaproducers to stream the data from external rest APIs toKafkatopics.
- Evaluated and improved application performance wif Spark.
- Worked wif analyst to determine and understand business requirements.
- Experienced in developing scripts for doing transformations usingScala.
- Implementing new dimensions into spark application upon on the business requirements.
- Integrated various business validation & prep flag rules from campaign managers and business analyst on the data.
- Hands on experience in AWS Cloud in various AWS services such as Redshift cluster, Route 53 domain configuration.
- On demand secure EMR launcher wif custom spark submit steps using S3 Event, SNS, KMS and Lambda function.
- Extensive knowledge of working on NiFi.
- Migrated an existing on - premises application to AWS.
- Used AWS services like EC2 and S3 for small data sets.
- Used Cloud watch logs to move application logs to S3 and create alarms based on a few exceptions raised by applications
- Implemented Spark Sql to update queries based on the business requirements.
- Responsible to store processed data into MongoDB.
- Created multi - tier java based multiple web services to read data from MongoDB.
- Deployed Spark application and java web services in pivotal cloud foundry.
- Encoded and decoded json objects using PySpark to create and modify the dataframes in Apache Spark.
- UsedAUTOMICjob scheduler for scheduling multiple applications in Production.
- Responsible for designing and developing data ingestion from Kroger using Apache NiFi/Kafka.
- Handled Prod Deployments and provided production support for fixing the defects.
- Agile methodology including test-driven and pair-programming concept.
- Strong communication and analytical skills and a demonstrated ability to handle multiple tasks as well as work independently or in a team.
Environment: Apache Hadoop, Java,Kafka,Scala, Apache Spark,PySpark,NiFi, AWS, Apache Hive, IntelliJ, Maven, Team City, Automic, Apache NiFi and Git Hub.
Confidential - Phoenix, Arizona
Sr. Data Engineer
Responsibilities:
- Written shell scripts to extract data from Unix servers into Hadoop HDFS for long-term storage.
- Implemented Micro Services architecture using spring boot framework.
- DevelopedSparkapplications usingScalautilizingData framesandSpark SQLAPI for faster processing of data
- Created Messaging queues using RabbitMQ to read data from HDFS to process the data.
- Written Spark Application to implement Slowly changing dimensions (SCD Type me).
- Created Oozie workflow in process to automate the spark application.
- Written pig script to load processed data from HDFS into MongoDB.
- Built real timedata pipelinesby developingKafkaproducers andSparkstreaming applications for consuming.
- Involved in file movements betweenHDFSandAWS S3and extensively worked wifS3 bucketinAWS.
- Used MongoDB to store processed products and commodities data, which can be further down streamed into web application (Green Box/ Zoltar).
- Consumed the data fromKafkaqueue usingSpark. Configured different topologies forSparkcluster and deployed them on regular basi
- Deployed Spark application and java web services in pivotal cloud foundry.
- Agile methodology including test-driven and pair-programming concept.
- Strong communication and analytical skills and a demonstrated ability to handle multiple tasks as well as work independently or in a team.
Environment: Apache Hadoop,Scala, Apache Spark, Apache Pig, Oozie,Java, MapReduce, Spring Boot,AWS, Kafka, Pivotal Cloud Foundry, JUnit, IntelliJ, Maven, Team City, and Git Hub.
Confidential, Sunnyvale, CA
Big Data Engineer
Responsibilities:
- DevelopedCrawlersjava ETL framework to extract data from Cerner clients database and Ingest into HDFS &HBase for Long Term Storage.
- Written data processing pipelines inApache Crunchto standardize and normalize the data and store the normalized data inHBase.
- Create ETL Pipelines usingApache Crunchto read data from HBase and calculate the KPI metrics and store the data in HDFS, and Bulk load data into HBase.
- Experience in ETL pipeline from HDFS to Vertica.
- Working on migrating all the Batch ETL Crunch pipelines to in-memorySpark Pipelinesfor quicker run times of the jobs.
- CreatedOozie workflowsto manage the execution of the crunch jobs and vertica pipelines.
- Cluster tuning to improve performance.
- Create Integration tests to check the validity of the data being crawled.
- Worked wif other teams to determine and understand business requirement.
- CreateHivebased reports to support the application metrics (These will be used by UI teams for reports).
- Implementing new drill to detail dimensions into the data pipeline upon on the business requirements.
- Hands on experience wifApache SOLRfor indexing HBase tables and querying the indexes.
- Worked wif multiple teams in resolving production issues.
- Deployment automation viaChef.
- UsedsparkSQLfor reading data from external sources and processes the the data usingScalacomputation framework.
- Handled Prod Deployments and provided production support for fixing the defects.
- Responsible for creating business layer and its underlying data foundation and connection layer usingInformation Design Tool.
- Agile methodology including, test-driven and pair-programming concept.
- Strong communication and analytical skills and a demonstrated ability to handle multiple tasks as well as work independently or in a team.
Environment: Apache Hadoop, Apache Crunch, HBase, Kafka,Apache Hive, Oozie, Chops, HDFS, Apache SOLR, Splunk, Chef, JUnit, Ruby, Eclipse, Maven, Jenkins, and Git Hub.
Confidential, Atlanta, GA
Java/ J2EE/ Hadoop Developer
Responsibilities:
- Involved in review of functional and non-functional requirements.
- Installed and configuredHadoopMapReduceandHDFS.
- Acquired good understanding and experience ofNoSQLdatabases such asHBase and Cassandra.
- Installed and configured Hive and also implemented various business requirements by writingHiveUDFs.
- Extensively worked on user interface for few modules usingHTML,JSP’s,JavaScript, PythonandAjax.
- Generated Business Logic usingservlets, Session beans and deployed them on Web logic server.
- Created complexSQLqueriesandstoredprocedures.
- Developed the XML schema and Web services for the data support and structures.
- Implemented the Web service client for login verification, credit reports and applicant information using Apache Axis 2 web service.
- Responsible for managing data coming from different sources.
- Used Hibernate ORM framework wif spring framework for data persistence and transaction management.
- Usedstrutsvalidation framework for form level validations.
- Wrote test cases in JUnit for unit testing of classes.
- Provided technical support for production environments resolving the issues, analyzing the defects, providing and implementing the solution defects.
- Built and deployed Java application into multiple Unix based environments and produced both unit and functional test results along wif release notes
Environment: Hadoop, HBase, Hive, Java, Eclipse, J2EE 1.4,MapReduce, Struts 1.3, JSP, Servlets 2.5, WebSphere 6.1, HTML, XML, ANT 1.6, Python, JavaScript, Junit 3.8.
Confidential
Java/J2EE Developer
Responsibilities:
- Assisted in gathering the Customer Requirements and preparingRequirements Document.
- Selected appropriate J2EE Patterns for implementing business applications.
- Developed Solution Alternatives, Solution Recommendation, analysis-level as well as design level Package Diagram, Deployment Diagram, Class Diagram, Sequence/Activity diagram, Component Diagram, and non-functional design considerations deliverables.
- Part of the team, working on thearchitecture and design of the application, stressed on code reuse, maintainability, scalability and high availability.
- AppliedOOAD/UML, and design patterns during the end-to-end J2EE based design.
- Developedsession beansthat host Business Logic and Performs DB Access.
- Developed the Presentation layer using theMVC paradigm, implemented byStruts.
- Wrote theDAO (Data Access Object) component using JDBCto access the database.
- Involved in the development of various components of the application.
- UsedLog4J utilityto log error, info and debug messages.
- UsedCVS for versioningand concurrent access to the project files.
- UsedJava scriptfor validation of page data in the JSP pages.
- Wrote variousSQL Queries, Stored Procedures, triggers, functions.
- Conducting code inspection for programs developed by team members, whenever required.
- Implemented test first programming approach usingJUNIT as used JMock.
- DevelopedCrystal reportsusing Crystal 10 Enterprise.
Environment: Jakarta Struts Framework, JSP, CVS, JUNIT, JMOCK, Tiles, Tag Libraries, HTML, Java Script, Log4J, ANT, Web Logic 8.1, JDBC, Eclipse 3.0.
