We provide IT Staff Augmentation Services!

Spark/bigdata Engineer Resume

5.00/5 (Submit Your Rating)

Charlotte, NC

SUMMARY

  • Over 7+ years of experience in IT industry with Configuration Management, Development, System Administration, Support and Maintenance in environments which includes 4+ years of experience as Big data consultant with good knowledge in Hadoop tools.
  • Experience in design, development, maintenance and support of Big Data Analytics using Hadoop Ecosystem components like HDFS, Hive, Impala, Hue, Pig, Sqoop, Flume, MapReduce, Spark, Kafka, Yarn, Zookeeper, Airflow, Nifi and Oozie.
  • Around 2+ years of experience on Cloud tools like Azure, AWS which provides fast and efficient processing of streaming Big Data.
  • Strong experience of leading multiple Azure Big Data and Data transformation implementations in Azure Banking and Financial Services, High Tech and Utilities industries.
  • Created Pipelines in ADF using Linked Services/Datasets/Pipeline/to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write - back Azure tool and backwards.
  • Experience in design, development, and deployment of BI solutions using Azure Data Factory, Azure Data Bricks and Azure Data Lake.
  • Experience with Apache Airflow, Spark, PySpark, AWS, Docker, EMR, Glue and Cloudformation.
  • Hands on experience in data mining process, implementing complex business logic and optimizing the query using HiveQL and controlling the data distribution.
  • Exploring with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame and Pair RDD's.
  • Experience with Hadoop/Big data Horton works and Cloudera distribution of Hadoop.
  • Extensively worked on Sqoop, Hadoop, Hive, PySpark, Cassandra to build ETL and Data Processing systems having various data sources, data targets and data formats
  • Experienced in CI/CD process to optimize, test and debug existing code to enhance performance for maintaining code in development, test and production environments.
  • Experience with container-based deployments using Docker, working with Docker images, Docker Hub and Docker-registries and Kubernetes.
  • Experienced in writingSparkApplications inScalaandPython(Pyspark).
  • Used Kubernetes to orchestrate the deployment, scaling and management of Docker Containers.
  • Have a good Knowledge in ETL and hands on experience in ETL.
  • Developed PL/SQL stored procedures, triggers, complex SQL queries for RDMS databases.
  • Excellent understanding and knowledge of job workflow scheduling and locking tools/services like Oozie, Crontab and Autosys.
  • Well versed with core Java like collections, multithreading and serialization.
  • Experience with Version Control tools like Bitbucket, GitHub and GIT Lab.
  • Excellent communication, interpersonal, and analytical skills. Proven ability to learn and rapidly adapt to new challenges.

TECHNICAL SKILLS

Hadoop: Hive, SparkSQL, Sqoop, Pig, Spark, Kafka, Oozie, Kerberos, MapReduce, Yarn, Impala, Nifi.

Programming: Shell, Perl, Python, PowerShell, SQL, HQL, JAVA, Scala.

Databases: MS Server, MySQL, Redshift, RDF, Helix, RDH, Oracle, DB2, Teradata.

Web Servers: Apache Tomcat, Web Logic, Web SphereVersion Control Systems SVN, GitHub, BitBucket.

Container Technologies: Kubernetes, Docker.

Web Servers: Web Logic, Web Sphere, Apache Tomcat.

CI Tools: Jenkins, Docker, Kubernetes, Ansible, Terraform, Bamboo, Control-M

PROFESSIONAL EXPERIENCE

Confidential, Charlotte, NC

Spark/Bigdata Engineer

Responsibilities:

  • Worked on data ingestion from windows file system into conformed layer data lake.
  • Load the data into curated layer and move the data into SAS tables for data science team and BI applications to generate analytics and reports for insights of data.
  • Responsible to create external secure data pipeline involving diverse systems such Azure SQL DB, Azure SQL DW, ADLS Gen2, Azure Databricks, On-premises compute.
  • Implemented Data Migration and Data Integration using SQL, SQL Azure, Azure Data Lake, and Azure Data Factory, SSIS, PowerShell.
  • Developed a batch processing solution using Azure Data Factory pipelines, Triggers and Integration Runtime and optimize the data pipeline run time.
  • Responsible to deal with support team for deployment process and create the required CR requests for the CICD pipelines creation.
  • Created partitioning, bucketing hive tables and implement map side joins to optimize the hive queries decreased the time of execution.
  • Developed spark applications in Python PySpark on distributed environment to load huge number of CSV files with different schema in to Hive ORC tables.
  • Analysed the sql scripts and designed it by using PySpark SQL for faster performance.
  • Responsible in handling large datasets using Partitions, Spark in-memory capabilities, broadcasts in Spark, efficient joins, Transformations during ingestion process itself.
  • Created customize release framework and implementing continuous integration procedures using Autosys, Jenkins.
  • Run the Ad-hoc jobs in Autosys and Control-M on demand for Parquet, ORC, XML, JSON format files on schedule.
  • Optimize the Pyspark jobs to run on Kubernetes Cluster for faster data processing.
  • Optimization of Hive queries using best practices and right parameters and using technologies like Hadoop, YARN, Python, PySpark.
  • Respond to production issues and coordinate with stakeholders and external teams to arrive at solutions and conduct training sessions and Spark demonstrations. worked in Spark for data and parallel processing centric projects.
  • Worked with Source Code and Version Control systems like SVN, Git, etc.

Environment: HDFS, Hive, Spark, SparkSQL, HQL, SQL, Linux, GitLab, JIIRA, Agile, Autosys, Maven, XML, JSON, Oozie, MapR, Oracle, SAS, PowerShell

Confidential, Charlotte, NC

Spark/AWS/Bigdata Engineer

Responsibilities:

  • Developed pyspark Data pipeline to process the batch data which lands on the partitioned buckets then transform the data and store it into S3 which runs on a transient cluster.
  • Experienced in building Apache spark for CDC data by spinning transitional and spot instance types specs using CloudFormation script.
  • Worked on reading and writing multiple data formats like JSON, ORC, Parquet on HDFS using PySpark.
  • Developed Spark applications using Scala and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
  • I was responsible to build the model on Azure platform using Python and Spark for the model development and Dash Azure by Plotly for visualizations.
  • Responsible to integrate lambda architectures using Azure Data platform capabilities like Azure Data lake, Azure SQL Server, and Power BI.
  • Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the Azure data using the SQL Activity.
  • Integrated Glue jobs with Spark to create the crawlers for each data ingestion on S3 data.
  • Developed Sqoop scripts to import and export data from Oracle and handled incremental and updated changes into HDFS and Hive tables.
  • Involved in converting Hive/SQL queries into Spark transformations using SparkSQL, RDD’s and Data frames using Python.
  • Developed ETL data pipelines using pyspark on AWS EMR
  • Responsible to write the data monitoring scripts using Python for data validation.
  • Responsible to maintain the documentation of each project.
  • Worked on exposing data for further analysis and for generating transforming files from different analytical formats to Parquet, ORC, AVRO, JSON, CSV, Sequence, snappy and Gzip format.
  • Create Pyspark frame to bring data from DB2 to Amazon S3.
  • Used CI/CD tools Jenkins, Bitbucket, Jira and Docker registry/daemon for configuration management and automation using Ansible.
  • Container management using Docker by writing Docker files and set up the automated build on Docker HUB and installed and configured Kubernetes.

Environment: Spark, Databricks, hive, Python, Yarn, Sqoop, PySpark, Azure, Shell, Bamboo, Control-M, JIRA, Docker, Kubernetes

Confidential, Charlotte, NC

Scala/Data Engineer

Responsibilities:

  • Created Partitioning, Bucketing, and Map side Join, Vectorization, Parallel execution for optimizing the hive queries decreased the time of execution from hours to minutes.
  • Responsible to build the Cesium Application deployed on public cloud like AWS, AZURE etc.
  • Developed an automated Sqoop script and developed scripts to extract structured data from MySql and Oracle databases data onto Hive tables directly on daily incremental basis.
  • Developed framework for converting existing PowerCenter mappings and to PySpark(Python and Spark) Jobs.
  • Developed and implemented data integration using, DataBricks Spark, Hadoop Streaming and built the workflows and schedule within Autosys.
  • Responsible for performance tuning of Spark Application for setting right Batch Interval time correct level of parallelism and memory tuning which persists into Hive to HDFS.
  • Responsible in handling large datasets using Partitions, Spark in-memory capabilities, broadcasts in Spark, efficient joins, Transformations during ingestion process itself.
  • Involved in designing and implementing redundant systems, policies, and procedures for disaster recovery and data archiving to ensure effective protection and integrity of Azure data assets.
  • Involved in convertingHive/SQLqueries into Spark transformations usingSpark RDD,ScalaandPython.
  • Experience in implementingSpark RDD'sinScala.
  • Worked on Impala and Hive for exposing data for further analysis and for generating transforming into analytical formats to Parquet, ORC, AVRO snappy compressed format.
  • Developed POC on real time streaming data received by Kafka and processed the data using Spark and this data was further stored into HDFS cluster using Java and Scala.
  • Responsible to work with enterprise architects and IT project managers to enhance existing business applications and facilitate solutions to meet business requirements to end clients.
  • Responsible to develop the Code and Unit test and move the code to UAT and PROD.
  • Work in Agile environment and GitHub as Version control.

Environment: Hive, Sqoop, Impala, Spark, Ansible, SparkSQL, Yarn, Kerberos, Sentry, Cloudera Manger, Pyspark, Jenkins, Python, Shell, MySQL Oracle, GitHub, Bitbucket, Jira, Agile.

Confidential, Charlotte, NC

Hadoop/Scala Developer

Responsibilities:

  • Created Partitioning, Bucketing, and Map side Join, Parallel execution for optimizing the hive queries decreased the time of execution from hours to minutes.
  • Developed Sqoop scripts to import and export data from MySQL and handled incremental and updated changes into HDFS and Hive tables and build Impala analytical queries.
  • Involved in migration of Map Reduce programs into Spark transformations using Spark.
  • Experienced in handling large datasets using partitions, Spark in-memory capabilities, Broadcasts in Spark, Effective & efficient joins, transformations and other during ingestion process itself.
  • Responsible to maintain the documentation of each project from scratch to end.
  • Developed Spark API using Scala and Spark-SQL/Streaming for faster processing of data.
  • Responsible to write the data monitoring scripts using Python for data validation.
  • Written SparkSQL queries for data analysis to meet the business requirements and writing HQL scripts to extract, transform and there by load the data into RDF and Helix database.
  • Developed Data-Models by using Turtle files in the Resource Description Framework (RDF) Database and developed Primitives in RDF database and load them in Linux Server.
  • Developed SPARQL queries to pull the data from RDF database and apply the business rules and manipulate the data as per the requirement.
  • Developed POC for microservice on boarding tools leveraging Python and Jenkins allowing for easy creation and maintenance of build jobs and Kubernetes deploy and services.
  • Written a Scala API called Helix for compare the data in between PAX Database and Cesium UI and must be check them that both the data is matching with each other and without any errors.
  • Generated the clients reports and comparing them building and deploying the code in Helix.
  • Used JIRA for bug tracking and Gitlab and GitHub for version control.
  • Participated in an Agile SDLC to deliver new platform services and components.

Environment: HQL, Impala, Spark, Zookeeper, Jenkins, Pyspark, Maven, SBT, Linux, Shell Script, Autosys, GIT, Jira, Bitbucket, Ansible, Agile, Sqoop, MySQL, RDF

Confidential, San Jose, CA

ETL/Hadoop Developer

Responsibilities:

  • Importing and exporting data into HDFS and Hive using Sqoop and Flume.
  • Developed Pig Scripts, Pig UDFs and Hive Scripts, Hive UDFs to process the data.
  • Implemented ETL pipelines to ingest data from traditional EDW Hadoop.
  • Involved in Design, analysis, Implementation, Testing, and support of ETL Created common reusable objects for the ETL team and overlook coding standards.
  • Used Change Data Capture (CDC) to simplify ETL in data warehouse applications.
  • Involved in loading data from LINUX file system to HDFS.
  • End-to-end performance tuning of Hadoop clusters and Hadoop Map/Reduce routines against very large data sets.
  • Worked on Hive for exposing data for further analysis and for generating transforming files from different analytical formats to text files.
  • Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (like, Pig, Hive, and Sqoop) as well as system specific jobs.
  • Automated all the jobs, for ingesting data from relational databases to load data into Hive tables, using Oozie workflows and enabled.
  • Hands on experience in writing HiveQL queries, Pig Latin Scripts and custom UDF's for Hive and Pig using Java and Python.
  • Developed a POC on transferring data from different data sources into HDFS systems using Kafka producers, consumers and Kafka brokers.
  • Consumed the data from Kafka queue using spark streaming also built spark applications for batch data to deliver downstream cyber security applications like Alert on Search and Instinct which are developed for cyber security and Data Loss Prevention.
  • Proficient in using Cloudera Manager, an end to end tool to manage Hadoop services.

Environment: JDK1.7, HDFS, Hive, Impala, Pig, ETL, Informatica, Sqoop, Spark, Python, Scala, Oozie, Ambari, Oracle11g, MS SQL, Linux, SVN, JIIRA, Agile.

Confidential

J2EE/Hadoop Developer

Responsibilities:

  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
  • Developed complex Hive queries to process the data for visualizing.
  • Used impala to pull the data from Hive tables.
  • Worked with the Data Science team to gather requirements for various data mining projects
  • Written Hive queries for data analysis to meet the business requirements and writing Hive scripts to extract, transform and load the data into Database.
  • Developed Pig Scripts, Pig UDFs and Hive Scripts, Hive UDFs to process the data.
  • Importing and exporting data into hdfs and Hive using Sqoop.
  • Involved in migration of Map Reduce programs into Spark transformations.
  • Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (like, Pig, Hive, and Sqoop) as well as system specific jobs (such as Perl and shell script).
  • Responsible for building scalable distributed data solutions using Hadoop. Written various Hive andPig scripts.
  • Experienced in performance tuning of Spark Application for setting right Batch Interval time, correct level of parallelism and memory tuning which persists into Cassandra.
  • Developed Spark scripts by using Scala shell commands as per the requirement.
  • Developed scripts and Auto sys Jobs to schedule a bundle (group of coordinators) which consists of various Hadoop programs using Oozie.
  • Proficient in using Cloudera Manager, an end-to-end tool to manage Hadoop services.

Environment: JDK1.6, HDFS, Map Reduce, Hive, Pig, Sqoop, Spark, Oozie, Impala, Cloudera, Oracle 11g, My SQL, Linux, SVN, GitHub

Confidential

Java/J2EE Developer

Responsibilities:

  • Responsible to develop class diagrams and sequence diagrams for the modules using UML.
  • Redesigned the web applications by replacing the Custom MVC framework with the Spring MVC module and used spring framework for dependency injection with the help of Configuration files.
  • Implemented Web-Services to integrate between different applications using SOAP and REST web services using Apache-CXF and Developed PL/SQL stored procedures, triggers.
  • Wrote SQL queries and PL/SQL procedures for JDBC.
  • Worked on Struts framework to create the Web application.
  • Designed and developed struts action classes for the controller responsibility.
  • Involved in the integration of Spring for implementing dependency Injection (DI/IOC).
  • Utilized Agile Scrum methodology to manage full life-cycle development of the project.
  • Written deployment scripts to deploy application at client site.
  • Created the stored procedures using Oracle database and accessed through Java JDBC.
  • Developed Web Services using REST and SOAP API’s as a part of unit-testing.
  • Experienced in writing Unix Shell Scripts for file backup.
  • Architecture/Design web-based solution for client by having regular interaction and presenting POC to the clients.

Environment: Core Java, J2EE, WebLogic, JSON, JDBC, Oracle, XML, PL/SQL, Ajax, JavaScript, Bootstrap, Eclipse, Hibernate, Unix, Windows, SVN, Putty

We'd love your feedback!