SUMMARY

• Around 8+ years of experience as Software developer with strong emphasis on Data engineering using Big data tools and application development using Java.
• Excellent noledge in building data engineering pipelines, automating, and fine - tuning both batch and real time data pipelines.
• Responsible for creating complex Data lakes both in on-prem and cloud infrastructure.
• Excellent noledge on architecture of Distributed storage and parallel computing frameworks to deal with large datasets in data engineering space.
• Experience in designing and developing production ready data processing applications in Spark using Scala/Python.
• Strong experience creating Spark applications for performing various kinds of data transformations like data cleansing, de-normalization, various kinds of joins, data aggregations etc.,
• Good experience fine-tuning Spark applications utilizing various concepts like Broadcasting, increasing shuffle parallelism, caching/persisting dataframes, sizing executors appropriately to utilize teh available resources in teh cluster effectively etc.,
• Strong experience building real time streaming pipelines using Kafka and Spark Structured streaming.
• Experienced in working with Amazon Web Services (AWS) using S3, EMR, Redshift, Atana, Glue Metastore etc.,
• Strong experience automating data engineering pipelines utilizing proper standards and best practices (right partitioning, right file formats, incremental loads by maintaining previous state etc.,)
• Good Knowledge in productionizing Machine Learning pipelines (Featurization, Learning, Scoring, Evaluation) primarily using Spark ML libraries.
• Good exposure with Agile software development process.
• Experience in manipulating/analyzing large datasets and finding patterns and insights within structured and unstructured data.
• Strong experience on Hadoop distributions like Cloudera, Hortonworks, AWS and Azure Databricks.
• Good understanding of NoSQL databases and hands-on work experience in writing applications on NoSQL databases like HBase, Cassandra and MongoDB.
• Experienced in writing complex MapReduce programs that work with different file formats like Text, JSON, Xml, parquet, and Avro.
• Experience in Oozie and workflow scheduler to manage Hadoop jobs by Direct Acyclic Graph (DAG) of actions with control flows.
• Experience in migrating teh data using Sqoop from HDFS to Relational Database System and vice-versa.
• Extensive Experience on importing and exporting data using stream processing platforms like Flume and Kafka.
• Very good experience in complete project life cycle (design, development, testing and implementation) of Client Server and Web applications.
• Excellent Java development skills using J2EE, J2SE, Servlets, JSP, EJB, JDBC, SOAP and RESTful web services.
• Strong experience in Object-Oriented Design, Analysis, Development, Testing and Maintenance.
• Experienced in using agile approaches, including Extreme Programming, Test-Driven Development and Agile Scrum.
• Worked in large and small teams for systems requirement, design & development.
• Key participant in all phases of software development life cycle with Analysis, Design, Development, Integration, Implementation, Debugging, and Testing of Software Applications in client server environment, Object Oriented Experience in using various IDEs Eclipse, IntelliJ and repositories SVN and Git.
• Experience of using build tools SBT, Maven.

TECHNICAL SKILLS

BigData/ Hadoop Framework: HDFS, MapReduce v1/v2, Yarn, Pig, Hive, Presto, Sqoop, Oozie, ZooKeeper, Flume and HBase, Kafka, Spark

Databases: HP Vertica, Teradata, Netezza, Cassandra and Oracle 9i/10g, Microsoft SQL Server, MySQL

Languages: Java/J2EE, Scala, Spring, Hibernate, Python, Bash, SQL, Pig Latin

BI Tools: Informatica Powercenter 9.x, Business Objects XI, Tableau Desktop, QlikView, R Studio

Operating Systems: Windows XP/7, CentOS, Ubuntu

Development Tools: Intellij IDEA, Eclipse, NetBeans, Visual Studio

Development Methodologies: Six sigma Development methodologies, Agile/Scrum, Waterfall

PROFESSIONAL EXPERIENCE

Data Engineer

Confidential, Pataskala, Ohio

Responsibilities:

• Ingested user behavioral data from external servers such as FTP server and S3 buckets on daily basis using custom Input Adapters.
• Created Sqoop scripts to import/export user profile data from RDBMS to S3 Data Lake.
• Developed various spark applications using Scala to perform various enrichments of user behavioral data (click stream data) merged with user profile data.
• Involved in data cleansing, event enrichment, data aggregation, de-normalization and data preparation needed for downstream model learning and reporting.
• Utilized Spark Scala API to implement batch processing of jobs
• Troubleshooting Spark applications for improved error tolerance.
• Fine-tuning spark applications/jobs to improve teh efficiency and overall processing time for teh pipelines
• Created Kafka producer API to send live-stream data into various Kafka topics.
• Developed Spark-Streaming applications to consume teh data from Kafka topics and to insert teh processed streams to HBase.
• Utilized Spark in Memory capabilities, to handle large datasets.
• Used broadcast variables in spark, effective & efficient Joins, transformations and other capabilities for data processing.
• Experienced in working with EMR cluster and S3 in AWS cloud.
• Creating Hive tables, loading and analyzing data using hive scripts. Implemented Partitioning, Dynamic Partitions, Buckets in Hive.
• Involved in continuous Integration of application using Jenkins.

Environment: AWS EMR, Spark, Hive, HDFS, Sqoop, Kafka, Oozie, HBase, Scala, MapReduce.

Data Engineer

Confidential, Weehawken, NJ

Responsibilities:

• Worked on building centralized Data lake on AWS Cloud utilizing primary services like S3, EMR, Redshift and Atana and Glue.
• Hands on Experience in migrating datasets and ETL workloads with Python from On-prem to AWS Cloud services.
• Built series of Spark Applications and Hive scripts to produce various analytical datasets needed for digital marketing teams.
• Worked extensively on building and automating data ingestion pipelines and moving terabytes of data from existing data warehouses to cloud.
• Worked extensively on fine tuning spark applications and providing production support to various pipelines running in production.
• Worked closely with business teams and data science teams and ensured all teh requirements are translated accurately into our data pipelines.
• Developed PySpark based pipelines using spark data frame operations to load data to EDL using EMR for jobs execution & AWS S3 as storage layer.
• Worked on full spectrum of data engineering pipelines: data ingestion, data transformations and data analysis/consumption.
• Developed AWS lambdas using Python & Step functions to orchestrate data pipelines.
• Worked on automating teh Infrastructure setup, launching and termination EMR clusters etc.
• Created Hive external tables on top of datasets loaded in S3 buckets and created various hive scripts to produce series of aggregated datasets for downstream analysis.
• Build real time streaming pipeline utilizing Kafka, Spark Streaming and Redshift.
• Worked on creating Kafka producers using Kafka Java Producer API for connecting to external Rest live stream application and producing messages to Kafka topic.
• Implemented a Continuous Delivery pipeline with Bitbucket and AWS AMI's.
• Designed, documented operational problems by following standards and procedures using Jira.

Environment: AWS S3, EMR, Lambdas, Redshift, Atana, Glue, Spark, Scala, Python, Java, Hive, Kafka, PySpark, Bitbucket, Jira.

Big Data Engineer

Confidential, Lake Success, NY

Responsibilities:

• Developing and maintaining a Data Lake containing regulatory data for federal reporting with big data technologies such as Hadoop Distributed File System (HDFS), Apache Impala, Apache Hive and Cloudera distribution.
• Developing different ETL jobs to extract data from different data sources like Teradata, SQL Server, transform teh extracted data using Hive Query Language (HQL) and load it into Hadoop Distributed file system (HDFS).
• Involved in importing teh data from different sources into HDFS using sqoop and applying transformations using Hive, spark and tan loading data into Hive tables.
• Primarily involved in Data Migration process using AWS by integrating with Github repository and Jenkins.
• Designed, developed, and did maintenance of data integration programs in a Hadoop and RDBMS environment with both traditional and non-traditional source systems as well as RDBMS and NoSQL data stores for data access and analysis.
• Performed in-memory computing capacity of Spark to perform procedures such as text analysis and processing using Scala.
• Experience working with Spark Streaming and divided data into different branches for batch processing through teh Spark engine.
• Implementing new functionality in teh Data Lake using big data technologies such as Hadoop Distributed File System (HDFS), Apache Impala and Apache Hive based on teh requirements provided by teh client.
• Communicating regularly with teh business teams along with teh project manager to ensure that any gaps between teh client's requirements and project's technical requirements are resolved.
• Developing Python scripts using Hadoop Distributed File System API's to generate Curl commands to migrate data and to prepare different environments within teh project.
• Monitoring production jobs using Control-M on a daily basis.
• Coordinating teh Production releases with teh change management team using Remedy tool.
• Communicating effectively with team members and conducting code reviews.

Environment: Hadoop, Data Lake, AWS, Python, Spark, Hive, Cassandra, ETL Informatica, Cloudera, Teradata, Oozie, Linux.

Big Data Engineer

Confidential

Responsibilities:

• Imported data from different relational data sources like RDBMS, Teradata to HDFS using Sqoop.
• Imported bulk data into HBase using Map Reduce programs.
• Perform analytics on Time Series Data exists in HBase using HBase API.
• Designed and implemented Incremental Imports into Hive tables.
• Used Rest API to Access HBase data to perform analytics.
• Worked in loading and transforming large sets of structured, semi structured, and unstructured data.
• Involved in collecting, aggregating, and moving data from servers to HDFS using Apache Flume.
• Written Hive jobs to parse teh logs and structure them in tabular format to facilitate effective querying on teh log data.
• Involved in creating Hive tables, loading with data, and writing hive queries that will run internally in map reduce way.
• Experienced in managing and reviewing teh Hadoop log files.
• Migrated ETL jobs to Pig scripts do Transformations, even joins and some pre-aggregations before storing teh data onto HDFS.
• Worked with Avro Data Serialization system to work with JSON data formats.
• Worked on different file formats like Sequence files, XML files and Map files using Map Reduce Programs.
• Involved in Unit testing and delivered Unit test plans and results documents using Junit.
• Exported data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose.
• Worked on Oozie workflow engine for job scheduling.
• Created and maintained Technical documentation for launching Hadoop Clusters and for executing Pig Scripts.

Environment: Hadoop, HDFS, Map Reduce, Hive, Oozie, Sqoop, Pig, Java, Rest API, Maven, JUnit.

JAVA Developer

Confidential 

Responsibilities:

• Designed and developed applications using Spring MVC framework with Agile Methodology.
• Developed JSP and HTML pages using CSS and JavaScript as part of teh presentation layer.
• Hibernate framework is used in persistence layer for mapping an object-oriented domain model to database.
• Developed database schema and SQL queries for querying, inserting, and managing database.
• Implemented various design patterns in teh project such as Data Transfer Object, Data Access Object and Singleton.
• Used Git for Source Code Management.
• Used Maven scripts to fetch, build, and deploy application to development environment.
• Created RESTFUL web service interface to Java-based runtime engine.
• Used Git for Source Code Management.
• Used Apache Tomcat for deploying teh application.
• Used Junit for functional and unit testing of code.

Environment: Eclipse IDE, Java/J2EE, Spring, Hibernate, JSP, HTML, CSS, JavaScript, Maven, RESTful Web services, Apache Tomcat, Oracle, JUnit, Git