SUMMARY

• Experience in distributed big data systems wif Hadoop, AWS EMR Hadoop clusters
• Primary technical skills in HDFS, Spark, Kafka, Hive, Sqoop, HBase, Flume, Oozie, Zookeeper, YARN.
• Skilled in Hadoop big data using Cloudera (CDH) and Hortonworks (HDP)
• Experienced team lead providing mentoring to engineers, and liaison for team wif stakeholders, business units, data scientists/analysts and making sure all teams collaborate smoothly.
• Has good experience in extracting and generating data visualizations.
• Facilitation of meetings following Scrum processes such as Sprint Planning, Backlog, Sprint Retrospective, Requirements Gathering and providing planning and documentation for project; ensuring project is on track wif stakeholder wishes.
• In depth understanding/noledge of Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node and MapReduce concepts and experience in working wif
• Used Apache Hadoop for working wif Big Data to analyze large data sets efficiently.
• Hands on experience in working wif Ecosystems like Hive, Sqoop, MapReduce, Oozie.
• Hive's analytical functions, extending Hive core functionality by writing custom SQL queries.
• Experience in importing and exporting Terabytes of data between HDFS and Relational Database Systems using Sqoop.
• ETL from databases such as SQL Server and Oracle to Hadoop HDFS in Data Lake.
• Experience in writing SQL queries, Stored Procedures, Triggers, Cursors and Packages.
• Experience in handling XML files and related technologies.
• Development and Debugging experience on Python, Scala and Java.
• Involved in processes using Spark streaming to receive real time data using Kafka/ on prem and AWS cloud.
• Used Spark Structured Streaming for high performant, scalable, fault - tolerant data processing of real-time data streams by extending core Spark API on prem and AWS.
• Experience wif multiple terabytes of data stored in AWS using Elastic Map Reduce (EMR) and Redshift PostgreSQL.
• Experienced in Amazon Web Services (AWS), and cloud services such as EMR, EC2, S3, EBS and IAM entities, roles, and users.

TECHNICAL SKILLS

Programming & Scripting: Java, Scala, Python, Unix Shell Scripting

Data & File Management: Apache Cassandra, Apache Hbase, Oracle, SQL Server, RDBMS, HDFS, Parquet, Avro, JSON, Snappy, Gzip

Methodologies: Agile, Kanban, Scrum, Jira, Continuous Integration (CI CD), Test-Driven Development, Unit TestingBig Data ETL Tools & Frameworks: Hive, MapReduce, Sqoop, Kafka, Spark, ELK Stack (Elasticsearch, Logstash, Kibana)

Big Data Platforms & Distributions: Hadoop, Cloudera Hadoop (CDH), Hortonworks Hadoop Data Platform (HDP)

Big Data Visualization: Visualization: Kibana, Tableau

Soft Skills: Leadership, Adaptability, Self-Motivation, Mentoring, Communication, Team Lead, Team Work

PROFESSIONAL EXPEREINCE

Big Data Engineer

Confidential, GA

Responsibilities:

• Installation and configuration of teh various Big Data ecosystem tools such as Elastic Search, Logstash, Kibana, Kafka and Cassandra.
• Built Real-Time Streaming Data Pipelines wif Kafka, Spark Streaming and Cassandra.
• Implemented Spark streaming for real-time data processing wif Kafka and handled large amounts of data wif Spark.
• Wrote streaming applications wif Spark Streaming/Kafka.
• Used Spark SQL to perform transformations and actions on data residing in HDFS.
• Responsible for designing and deploying new ELK clusters.
• Worked wif Elasticsearch and Logstash (ELK) performance and configure tuning.
• Created a Kafka producer to connect to different external sources and bring teh data to a Kafka broker.
• Handled schema changes in data stream using Kafka.
• Analyzed and tuned Cassandra data model for multiple internal projects/customers.
• Developed ETL pipeline to process log data from Kafka/HDFS sequence file and output to Hive tables in ORC format.
• Built Jenkins jobs for CI/CD infrastructure from GitHub repos.
• Support for teh clusters, topics on teh Kafka manager.
• Responsible for Kafka operation and monitoring, and handling of messages funneled through Kafka topics.
• Coordinated Kafka operation and monitoring wif dev ops personnel; formulated balancing impact of Kafka producer and Kafka consumer message(topic) consumption.
• Versioning wif Git and set-up a Jenkins CI to manage CICD practices.
• Pulled data and populated teh data in Kibana.
• Kibana dashboard designed over Elasticsearch for visualizing teh data.
• Interacted wif data residing in HDFS using Spark to process teh data.
• Participated in various phases of data processing (collecting, aggregating, moving from various sources) using Apache Spark.
• Handled structured data via Spark SQL then stored into Hive tables for downstream consumption.
• Worked wif Elasticsearch and Logstash (ELK) performance and configure tuning.

Big Data Developer

Confidential, Houston, Tx

Responsibilities:

• Implemented Serverless architecture using AWS Lambda wif AWS S3.
• Expertise in AWS data migration between different database platforms like Local SQL Server to AWS RDS and EMR HIVE.
• Led many critical on-prem data migration to AWS cloud, assisting teh performance tuning and providing successful path towards Redshift Cluster and AWS RDS DB engines.
• Worked on AWS S3 bucket integration for application and development projects.
• Experience in managing and reviewing Hadoop log files in AWS S3.
• Responsible for Designing Logical and Physical data modelling for various data sources
• Migration between different database platforms like Local SQL Server to AWS RDS and EMR HIVE. Redshift.
• Installed, configured, and tested an AWS Lambda function workflow in Python
• Responsible for continuous monitoring and managing Elastic MapReduce (EMR) cluster through AWS console.
• Automated and defined Spark and AWS Best practices for future deployment.
• Worked on Multiple AWS instances, set teh security groups. Auto scaling to design cost TEMPeffective, fault tolerant and highly available systems on AWS.
• Deploy Spark Jobs into AWS EMR.
• Managed AWS Redshift clusters such as launching teh cluster by specifying teh nodes and performing teh data analysis queries.
• Created basic infrastructure of teh pipeline using AWS Cloud Formation.
• Implemented usage of Amazon EMR for processing Big Data across Hadoop Cluster on AWS.
• Managed and monitored AWS EC2 instances through AWS Management Console
• Participated into teh AWS architecture, design and planning from ingestion into reporting.
• Securely controlling AWS users and groups access to AWS services and resources by assigning roles and polices using IAM.
• Configured Elastic search, Log stash and Kibana (ELK) for log analytics, full text search, application monitoring in integration wif AWS Lambda and Cloud Watch.
• Configuring Access for inbound and outbound traffic RDS DB services and EBS volumes to set alarms for notifications or automated actions on AWS.

Hadoop Developer

Confidential, VA

Responsibilities:

• Used Zookeeper and Oozie for coordinating teh cluster and scheduling workflows.
• Worked wif Ambari to monitor workloads, job performance and capacity planning.
• Managing Hadoop clusters via Command Line, and Hortonworks Ambari agent.
• Performed cluster and system performance tuning.
• Monitored Hadoop cluster using tools like Ambari.
• Configure clluster coordination services through Zookeeper and Kafka.
• Configure Yarn capacity scheduler to support various business SLA's.
• Implement and maintain security wif LDAP and Kerberos as designed for cluster.
• Coordinates wif monitors cluster upgrade needs, and monitors cluster health and builds proactive tools to look for anomalous behaviors.
• Worked wif cluster users to ensure efficient resource usage in teh cluster and alleviate multi-tenancy concerns.
• Scheduling batch jobs on a Hadoop Cluster using Oozie.
• Worked on Kafka cluster environment and zookeeper.
• Monitored multiple Hadoop clusters environments using Ambari.
• Experience in configuring, installing and managing Hortonworks (HDP) Distributions.
• Secured teh Kafka cluster communication wif SSL.