Big Data Engineer Resume
PhiladelphiA
SUMMARY
- Over 7 years of experience in developing applications using teh Hadoop ecosystem tools and TEMPhas in depth understanding of Distributed Systems Architecture and Parallel Processing Frameworks.
- Hands on experience wif Microsoft Azure Cloud services, Storage Accounts and Virtual Network
- Knowledge and hands on experience wif many Hadoop ecosystem components like HDFS, Map Reduce, Yarn, Hive, Sqoop, Kafka, Spark Streaming, Spark SQL, Zookeeper and Oozie.
- Experienced working wif various Hadoop Distributions (Cloudera, Hortonworks, Amazon EMR) to fully implement and leverage new Hadoop features.
- Proficient in using Unix based Command Line Interface.
- Experience in moving data into and out of teh HDFS and Relational Database Systems (RDBMS) using Apache Sqoop and Kafka.
- Expertise in working wif Hive data warehouse infrastructure creating tables and implementing Partitioning and Bucketing, and developing and tuning teh HQL queries.
- Experienced in writing custom UDF’s in Hive and custom Input Formats in MapReduce.
- Experience working wif NoSQL database technologies including MongoDB, Cassandra, and HBase.
- Experience in developing Spark Applications using Spark RDD, Spark SQL and Data frame APIs.
- Worked on Spark data transformations for efficient data processing using Spark SQL replacing existing MR jobs and Hive scripts.
- Knowledge of troubleshooting and tuning Spark applications and Hive scripts to achieve optimal performance.
- Strong understanding of real time data processing and streaming techniques using Spark Streaming, Flume and Kafka.
- Good understanding on Machine Learning Methodologies to uncover teh hidden patterns, user behavior and modeling using Spark MLlib.
- Experienced in architecting, designing and operationalization of largescale data and analytics solutions on Azure Cloud Data Warehouse.
- Experience inNoSQLColumn - Oriented Databases likeHBase,Cassandraand its Integration wifHadoopcluster.
- Experience in Cloud (Azure, AWS, GCP), DevOps, Configuration management, Infrastructure automation, Continuous Integration and Delivery (CI/CD).
- Worked on Deep Learning frameworks such as Pytorch and Tensorflow for designing and training deep learning models.
- Well Exposure on Spark SQL, Spark Streaming and using Core Spark API to explore Spark features to build data pipelines.
- Independent worker as well as a dynamic team player wif good communication and presentation skills.
- A fast learner wif strong problem-solving skills, great attention to details and adept at coding.
TECHNICAL SKILLS
Operating Systems: Linux (Ubuntu, CentOS), Windows, Mac OS
Hadoop Ecosystem: Hadoop, MapReduce, Yarn, HDFS, Pig, Zookeeper, Oozie
Big Data Ecosystem: Spark, Spark SQL, Spark Streaming, Spark MLlib, Hive, Impala, Hue
Cloud Ecosystem: AWS, Azure.
Data Ingestion: Sqoop, Flume, Kafka
NOSQL Databases: HBase, MongoDB, Cassandra
Programming Languages: Python, Scala, R, C, C#
Databases: Oracle, PostgreSQL, MySQL, SQL-Server, Teradata
IDE: Eclipse, Visual Studio, Jupyter Notebook, IntelliJ, IDLE
Machine Learning: Pytorch, Tensorflow, Keras, scikit-learn, numpy
Tools: SBT, Putty, Win SCP, Maven, Git, Tableau, Mahout, Toad
Methodologies: Agile, Scrum, Iterative Development, Waterfall Model
PROFESSIONAL EXPERIENCE
Confidential - Philadelphia
Big Data Engineer
Responsibilities:
- Experienced in Provisioning of IAAS & PAAS Virtual Machines and Web\Worker roles on Microsoft Azure Classic and Azure Resource Manager.
- Expertise in building CI/CD on AWS environment using AWS Code Commit, Code Build, Code Deploy and Code Pipeline and experience in using AWS Cloud Formation, API Gateway, and AWS Lambda in automation and securing teh infrastructure on AWS.
- Worked onMicrosoft Azure(Public) Cloud to provide IaaS support to client. Create Virtual Machines throughPower Shell ScriptandAzure Portal.
- Construct teh AWS data pipelines using VPC, EC2, S3, Auto Scaling Groups (ASG), EBS, Snowflake, IAM, Cloud Formation, Route 53, CloudWatch, Cloud Front, Cloud Trail.
- Extensive knowledge onNoSQLdatabases likeHBase, Cassandra, and Mongo DB.
- Develop Shell script dat reads Json files and apply it to Sqoop and Hive.
- Experience wif Azure Micro Services, Azure Functions and azure solutions.
- Work wif PySpark to migrate fixed width, ORC, CSV files, etc.
- Handled Hadoop cluster installations in various environments such as Unix, Linux and Windows
- Assisted in upgrading, configuration and maintenance of various Hadoop infrastructures like Ambari, Spark and Hive.
- Worked on various data formats like AVRO, Sequence File, JSON, Map File, Parquet and ORC.
- Worked on migrating Map Reduce programs into Spark transformations using Spark and Scala.
- Experience in job management using Fair scheduler and developed job processing scripts using Oozie workflow.
- Used Git version control to manage teh source code and integrating Git wif Jenkins to support build automation and integrated wif Jira to monitor teh commits.
- Worked wif NoSQL databases likeHbasein creating Hbase tables to load large sets of semi structured data coming from various sources.
- Working wif data files and develop pipelines using Azure and designing and maintaining temporary and transient tables.
- Planned and created answer for constant information ingestion utilizingKafka, Storm, Sparkspilling and differentNoSQL databases.
- Work extensively on predictive analytics for cutting down teh costs and minimizing risks using machine learning wif Spark MLlib.
- Expert in Setting up Continuous Integration (CI) by configuring Build, Code, Deploy and test automation Jobs in Jenkins for different applications and in creation and deployment using Codeship to automate branch & project creation in Git using (Groovy language) in Jenkins file and automating (using Ansible).
- Design and construct of AWS Data pipelines using various resources in AWS including AWS API Gateway to receives response from aws lambda and retrieve data from snowflake using lambda function and convert teh response into Json format using Database as Snow Flake, DynamoDB, AWS Lambda function and AWS S3.
- Implemented AWS Elastic Container Service (ECS) scheduler to automate application deployment in teh cloud using Docker Automation techniques.
- Generate reports using advanced analytics to meet teh business requirements and gather insights.
- Experienced in working wif Hadoop from Hortonworks Data Platform and running services through Cloudera manager
- Used Agile Scrum methodology and Scrum Alliance for development.
Environment: Hadoop, HDFS, Azure, Scala, MapReduce, YARN, PySpark, Spark MLlib, Hive, MySQL, Kerberos, Maven, Stream sets.
Confidential - Austin, TX
Big Data Developer
Responsibilities:
- Creating Hive external tables to store teh Pig script output and working on them for data analysis in order to meet teh business requirements.
- Contribute to teh support forums (specific to Azure Networking, Azure Virtual Machines, Azure Active Directory, Azure Storage) for Microsoft Developers Network including Partners and MVPs.
- Collect and aggregate large amounts of log data using Apache Flume and stage data in HDFS for further analysis.
- ImplementedApache Airflowfor authoring, scheduling and monitoring Data Pipelines
- Developed data pipelines using Sqoop, Pig and Hive to ingest customer member data, clinical, biometrics, lab and claims data into HDFS to perform data analytics
- Created automation and deployment templates for relational and NOSQL databases
- Obtained good experience wif NOSQL databaseCassandra.
- Provide consulting and cloud architecture for premier customers and internal projects running onMicrosoft Azure platformfor high-availability of services, low operational costs.
- Moving data from Oracle to HDFS and vice-versa using Sqoop.
- Virtualized teh servers using teh Docker for teh test environments and development environment and performed configuration automation using Docker containers. responsible from architecting solutions utilizing Microsoft Azure.
- Implemented partitioning, bucketing in Hive for better organization of teh data.
- Worked wif different file formats and compression techniques to determine standards.
- Developed Map Reduce pipeline jobs to process teh data and create necessary files.
- Involved in loading teh created files into HBase for faster access of teh data.
- Implementation of MapReduce-based large-scale parallel relation-learning system and used Zookeeper for providing coordinating services to teh cluster.
- Written python code for exploratory data analysis using Scikit-learn machine learning python packages- NumPy, Pandas, Matplotlib, Seaborn, stats models, pandas profiling.
- Used Oozie Scheduler system to automate teh pipeline workflow and orchestrate teh map reduces jobs dat extract teh data on a timely manner.
- Migrated existing on-premises application to AWS and used AWS services like EC2 and S3 for small data sets processing and storage.
- Involved in production support, which involved monitoring server and error logs, and foreseeing and preventing potential issues, and escalating issue when necessary.
- Worked on end to end machine learning workflow, written python code for gathering teh data from AWS snowflake, data preprocessing, feature extraction, feature engineering, modeling, evaluating teh model, deployment
- Worked wif NoSQL databases likeHBasein makingHBasetables to load expansive arrangements of semi structured data
- Involved in gathering requirements and responsible for documenting requirements, specs, dataflow, data models and project plan.
Environment: Hadoop, HDFS, Map Reduce, Spark, Hive, Flume, Sqoop, Oracle, MS SQL, Zookeeper, Oozie, AWS
Confidential - San Antonio, TX
Data Engineer
Responsibilities:
- Created Hive tables for loading and analyzing data. Implemented Partitions, Buckets and developed Hive queries to process teh data and generate teh data cubes for visualizing.
- Worked on performing Hive modeling and written many hive scripts to perform various kinds of data preparations dat are needed for running machine learning models.
- Implemented schema extraction for Parquet and Avro file Formats in Hive.
- Used Spark API over Cloudera Hadoop Yarn to perform analytics on data in Hive.
- Created data pipeline for different events of ingestion, aggregation and load teh data in AWS S3 bucket into Hive external tables in HDFS location to serve as feed for tableau dashboards.
- Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD's.
- Lead teh efforts for source system analysis, data discovery, enterprise data modeling, custody systems integration, data quality and automation.
- Worked closely wif teh data science team in automating and applying various models like logistic regression, k-means using Spark MLlib.
- Used Spark Streaming APIs to perform necessary transformations and actions on teh fly for building teh common learner data model which gets teh data from Kafka in near real time and persists into Cassandra.
- Experience in troubleshooting any problems dat arises during any batch data processing jobs.
- Worked wif BI team to create various kinds of reports using Tableau based on teh requirements.
Environment: Hadoop, PySpark, Spark MLlib, Hive, Sqoop, AWS, Oracle, SQL-Server, Kafka, Cassandra, Tableau, Yarn, Cloudera.
Confidential - Atlanta, GA
Data Engineer
Responsibilities:
- Worked wif Business users and Business analysts to understand and analyze ETL and reporting requirements.
- Worked wif homogenous and heterogeneous sources and targets like relational tables, delimited and fixed width flat files and CSV format files while creating mappings.
- Involved in building data warehouse whose primary source of data comes from external vendors and in corporate sources which includes teh data from various source systems like Oracle, Flat files, and Microsoft Excel and XML files.
- Used transformation components like Merge Join, database Join, XML components to load XML files.
- Used Pentaho Import Export utility to Migrate Pentaho Transformations and Job from one environment to others (DEV/QA/PREPROD/PROD).
- Developed complex ETL Procedures to migrate huge amount of data from teh old system to teh new one.
- Worked wif team members to tune teh SQL queries. Created necessary Index/Partition on huge fact tables for getting a better performance in ETL
- Providing Production support to operational team for daily/weekly job failures in Production environments.
Environment: Pentaho BI Server, Pentaho Data Integration (PDI/Kettle), Oracle 10g, PL/SQL, SQL Loader, Oracle SQL Developer, SSIS, Oracle Enterprise Manager, SQL Server
