Sr Data Engineer Resume
Arlington, VA
SUMMARY
- Around 8+ years of experience in design and development in technologies domains like BigData, Cloud and automation tools.
- Experience in working with Cloudera (CDH3, CDH4&CDH5) and Horton Works Hadoop Distributions.
- Experienced in AWS and Azure cloud environments for near real time applications.
- Experience indataanalysisand reporting; creating datasets anddatamodels
- Used APP Insights, Document DB, Service Bus, Azure Data Lake Store, Azure Blob Store, Event HUB, Azure Functions.
- Around 5 years of hands on experience in Big Data / Hadoop Ecosystem technologies which includes HDFS, Map Reduce, Pig, Hive, HBase, Spark, Sqoop, Flume, Oozie and Zookeeper.
- Excellent knowledge on Hadoop Ecosystem Architecture and components such as Hadoop Distributed File System (HDFS), MRv1, MRv2, Job Tracker, Task Tracker, Name Node, Data Node, Resource Manager, Node Manager and MapReduce programming.
- Extensive experience in Pharma domain for top clients like Confidential and Confidential Inc.
- Expertise in writingHadoopJobswere using HDFS,Hive, HBase, Pig, Spark, Kafka, Scala, OozieandTalend ETL for analyzing structured and unstructured data using.
- In - depthunderstanding ofSparkArchitecture includingSpark Core, Spark SQL, Data Frames, Spark Streaming.
- Working knowledge in importing and exporting data from RDMS to HDFS using Sqoop and Spark.
- Experience in developing ETL applications using Hive, Spark and Spark streaming.
- Developed bash scripts to launch spark jobs and automated with Control-M.
- Worked withreal-time dataprocessing and streaming techniques usingSpark streaming and Kafka.
- Experience in messaging queue like Apache Kafka.
- Experience in reading and writing data from BigQuery BQ.
- Developed Hadoop Map Reduce program using Java.
- Developed Spark jobs using Java and Scala.
- Experience in job workflow scheduling and monitoring tools like Oozie.
- Good knowledge of Data warehousing conceptsETLandTeradata.
- Experience with Snowflake cloud-based data warehouse
- Working knowledge on NoSQL databases like HBase, MongoDB and Cassandra.
- Experience in building and maintaining spring boot applications built in Java.
- Experience in building CI-CD deployments for various applications using Jenkins.
- Good Knowledge inAmazon AWSconcepts likeEMRandEC2web services.
- Knowledge of Kibana and Grafana for monitoring and analyzing log data.
- Strong experience in RDBMS technologies like SQL, Stored Procedures, Triggers, Functions.
- Expertise working in J2EE, JSP, JDBC, Servlets environments.
- Experience with Agile Methodology, Scrum Methodology and release management.
- Involved in all phases of Software Development Life Cycle (SDLC) in large scale enterprise software using Object Oriented Analysis and Design.
- Experience in working with Unit testing using Junit, Scala Test, Spock and Easy Mock and Mockito
- Knowledge in UNIX Shell Scripting and Perl Scripting.
- Experience in Continuous integration and Continuous deployment using Maven, Jenkins, Docker,Kubernetesand Spring boot.
- Experience in handling defect tracking tools JIRA/ALM.
- Experience with visualization tool like Power BI to create Data Models and build business reports
TECHNICAL SKILLS
Big Data Ecosystems: Hadoop, Map Reduce, HDFS, Zookeeper, Hive, Pig, Sqoop, Flume, Yarn, Spark, SparkSql, Apache Nifi, Kafka, Spark Streaming, AWS Glue, AWS RedShift, Azure Databricks, PySpark
DB Languages: SQL, PL/SQL
Programming Languages: Scala, Java, Python
Scripting Languages: Unix, Python
Web Services: SOAP, Restful
Databases: Oracle, SQL Server, HBASE, Cassandra, Azure Databricks, Teradata
Tools: Eclipse, Intellij, ERWIN, Visio, Putty, WinScp
Platforms: Windows, Linux, Unix
ETL & Visualization: Informatica, Power BI, Tableau
Application Servers: Apache Tomcat, Web Sphere, Web logic
PROFESSIONAL EXPERIENCE
Confidential, Arlington, VA
Sr Data Engineer
Responsibilities:
- Experience in analyzing the relationship between profile and transactional data as per the business use-case.
- Discussed with architects and worked on modeling, design of hive tables.
- Designed and built data processing pipelines using tools and frameworks in the Hadoop Ecosystem to facilitate data analysis.
- Designed and built ETL pipelines to automate ingestion of structured and unstructured data.
- Implemented and configured big data technologies to tune and optimize processes for performance at scale.
- Developed a data model around stated use cases to capture client’s KPIs and data transformations
- Migrated the existing data from Mainframes/Teradata/SQL server and perform ETL operations on it.
- Developed Spark applications for ETL transformations using scala and python.
- Experience in using Azure Data Factory to perform ETL as part of pre-processing
- Used Azure Databricks to run Spark applications.
- Experience with real time processing of data sources using Apache Spark Streaming.
- Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time and Persists into HBase.
- Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, TEMPEffective & efficient Joins, Transformations and other during ingestion process itself.
- Worked on a lot of ETL script developed in Hive and Spark SQL.
- Worked on a product team using Agile Scrum methodology to design, develop, deploy and support solutions that leverage the Client big data platform.
- Developed Spark code using Java, Scala for faster testing and data processing.
- Automated various jobs like hive, spark and shell actions in Apache oozie.
- Ingested real time data from various Kafka topics to Hive and HDFS for analytics purpose.
- Developed reporting applications for generating day to day reports for business.
- Loading data into HBase using Bulk Load and Non-bulk load.
- Involved in performance tuning of Spark Applications for setting right level of Parallelism and memory tuning.
- Used optimization techniques in spark like Data Serialization and Broadcasting.
- Optimizing of existing algorithms in Hadoop using Spark, Spark-SQL and Data Frames.
- Implemented POC in persisting click stream data with Apache Kafka
- Implemented data pipelines to move processed data from Hadoop to MPP, RDBMS and No SQL Databases.
- Followed Agile & Scrum principals in developing the project.
- Developed continuous deployment pipelines in Jenkins for productions and testing.
- Working on Power BI with custom visuals like ESRI Map visualization to show the model output in BNSF track and use DAX code to calculate the additional measures in Power BI as when required.
Environment: Spark, Spark Streaming, Spark- SQL, Azure Databricks, Kafka, Power BI, PySpark, Linux, Subversion, Docker, Jira, ETL, UNIX Shell Scripting, Control-M, Jenkins, HBase, Git, Kafka, Hive.
Confidential, Atlanta, GA
Sr Data Engineer
Responsibilities:
- Developed Spark code using Java, Scala for faster testing and data processing.
- Experience with batch processing of data sources using Apache Spark.
- Experience with real time processing of data sources using Apache Spark Streaming.
- Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time and Persists into HBase.
- Worked on a product team using Agile Scrum methodology to design, develop, deploy and support solutions that leverage the Client big data platform.
- Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, TEMPEffective & efficient Joins, Transformations and other during ingestion process itself.
- Optimized MapReduce Jobs to use HDFS efficiently by using various compression mechanisms.
- Involved in creation and designing of data ingest pipelines using technologies such asApacheSpark and Kafka
- Worked on migrating Map Reduce programs into Spark transformations using Spark and Scala.
- Created External Hive Table on top of parsed data.
- Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
- Automating and scheduling the Sqoop, Map Reduce and Spark jobs in a timely manner using Unix Shell Scripts and Control-M.
- Created Unit Test Documents and performing unit testing.
- Used Jira for bug tracking and Quick Build for continuous Integration.
- Worked on Data serialization formats for converting complex objects into sequence bits by using Avro, Parquet, Json, CSV formats.
- Loading data into HBase using Bulk Load and Non-bulk load.
- Involved in transforming data from Mainframe tables to HDFS, and HBASE tables using Sqoop.
- Hive QL scripts to create, load, and query tables in a Hive.
- Written MapReduce code to process & parsing the data from various sources & storing parsed data intoHBaseandHiveusingHBase-HiveIntegration.
Environment: Sqoop, Spark, Spark Streaming, Java8, Scala, Linux, Maven, Subversion, Jira, ETL, MapReduce, Quick Build, IntelliJ, UNIX Shell Scripting, Control-M, Cloudera, HBase, Git, Kafka, Hive.
Confidential, Westborough, MA
Data Engineer
Responsibilities:
- Developed Sqoop and spark scripts for ingesting data from various RDMS to HDFS and Hive.
- Deployed the HBase cluster in cloud (AWS) environment with scalable nodes as per the incremental business requirement.
- Developed multiple ETL Hive scripts for data cleansing and transformations for data.
- Experience in designing pipelines to Migrate Hive tables to BQ.
- Exported data from BI to Hadoop cluster.
- Converted various Hive scripts in spark scala and python applications in optimized way.
- Ingested data from various sources into Hbase for updating data.
- Exported data from Hive to AWS s3 bucket for further near real time analytics.
- Ingested data in real time from Apache Kafka to Hive and HDFS.
- Used Apache Nifi for loading data from RDBMS to HDFS and HBase.
- Used Apache Solr 5.4 as the Search Engine.
- Developedthe Apache Storm, Kafka,and HDFS integration project to do a real-time data analysis.
- Worked on the Publish component to read the source data, extract metadata and apply transformations to build Solr Documents, index them using SolrJ.
- Designed and Implemented Partitioning (Static, Dynamic), Buckets in Hive.
- Created HBase tables to store variable data formats.
- Continuous integration usingJenkinsand ensured code coverage.
- Used Apache Phoenix tool on top of HBASE to perform CURD operations.
- Used Apache Nifi for loading PDF Documents from Microsoft SharePoint to HDFS. Worked on the Publish component to read the source data, extract metadata and apply transformations to build Solr Documents, index them using SolrJ.
- Developed ETL Processes in AWS Glue to migrate Campaign data from external sources like S3 ORC/Parquet/Text Files into AWS Redshift.
- Data Extraction, aggregations and consolidation of Adobe data within AWS Glue using PySpark.
- Create external tables with partitions using Hive, AWS Athena and Redshift
- Designed External and Managed tables in Hive and processed data to the HDFS using Sqoop.
- Create user defined functions UDF in Redshift.
- Create a complete Roadmap for Data Lake that will stream in data from multiple sources and enable analytics using standard BI analytic tools on the Data Lake.
Environment: AWS, AWS S3, AWS Glue, Hive, Spark, Java, Linux, Redshift, Maven, SQL Server, Git, Jira, ETL, Apache Solr, Apache Tika, Toad 9.6, UNIX Shell Scripting, Scala, Apache Nifi.
Confidential, TN
Hadoop Developer
Responsibilities:
- Involved in the Complete Software development life cycle (SDLC) to develop the application.
- Worked with different feeds data like JSON, CSV, XML and implemented data lake concept.
- Defined UDFs using PIG and Hive in order to capture customer behavior.
- Design and implement MapReduce jobs to support distributed processing using java, Hive and Pig.
- Create Hive external tables on the MapReduce output before partitioning, bucketing is applied.
- Maintenance of data importing scripts using Hive and MapReduce jobs.
- Worked on hive data warehouse modeling to interface with BI tools such as Tableau.
- Extracted the data from Teradata into HDFS using the Sqoop.
- Loaded data into the cluster from dynamically generated files using Flume and from relational database management systems using Sqoop.
- Used Pig as ETL tool to do transformations, event joins, filter bot traffic and some pre-aggregations before storing the data onto HDFS.
- Efficiently put and fetched data to/from HBase by writing MapReduce job.
- Involved in writing Flume and Hive scripts to extract, transform and load the data into Database.
- Very Good experience on UNIX shell scripting, Python and WLST.
- Good experience on developing of ETL Scripts for Data cleansing and Transformation.
- Used Pig as ETL tool to do transformations, event joins, filter bot traffic and some pre-aggregations before storing the data onto HDFS.
- Expertise in designing python scripts to interact with middleware/back end services.
- Worked on python scripts to analyze the data of the customer.
- Used Jira for bug tracking.
- Used Git to check-in and checkout code changes.
Environment: Hadoop, Hortonworks, Linux, Python, HDFS, Pig, Hive, Sqoop, Zookeeper, MapReduce, Restful Service, Teradata, Tableau, Jenkins.
Confidential
Java/PLSQL Developer
Responsibilities:
- Utilized Model-View-Presenter (MVP) design pattern, decoupling view and presenter, in front-end development.
- Extensively used Hibernate in data access layer to access and update information in the database.
- Consumed JSON RESTful Web Services and sent responses with Spring MVC.
- Use Spring MVC with Annotations in integrating different layers within the application.
- Developed the application using Struts MVC 2 architecture.
- Developed JSP custom tags and Struts tags to support custom User Interfaces.
- Developed front-end pages using JSP, HTML and CSS.
- Developed core Java classes for utility classes, business logic, and test cases
- Developed Stored Procedures, Functions, Packages and SQL Scripts using PL/SQL
- Developed OraclePL/SQLcode based on requests raised by changes in business logic, norms and standards.
- Scheduling jobs in control M and handling them.
- Data loaded from legacy systems (ETL Operations) using PL/SQL and SQL*Loader.
- Used IBM Rational ClearCase for version control and source management.
- Used batch scripts to loop from database to database, to perform system-wide functions.
Environment: Oracle 11g/10g,SQLDeveloper, PLSQLDeveloper, Apache Tomcat, Eclipse, Core Java, TOAD, UNIX Shell Scripting, Putty, Control-M, Java, JSP, HTML, JDBC.
