SUMMARY

• Overall 12+ years of experience in IT industry, including big data environment, Hadoop ecosystem, Java and Design, Developing, Maintenance of various applications.
• Experience in developing custom UDFs for Pig and Hive to in corporate methods and functionality of Python/Java into Pig Latin and HQL (HiveQL).
• Good experience in Tableau for Data Visualization and analysis on large data sets, drawing various conclusions.
• Leveraged and integrated Google Cloud Storage and Big Query applications, which connected to Tableau for end user web - based dashboards and reports.
• Good working experience in Application and web Servers like JBoss and Apache Tomcat.
• Good Knowledge in Amazon Web Service (AWS) concepts like EMR and EC2 web services which provides fast and efficient processing of Teradata Big Data Analytics.
• Expertise in Big Data architecture like Hadoop (Azure, Hortonworks, Cloudera) distributed system, MongoDB, NoSQL
• Development of spark-based application to load streaming data with low latency, using Kafka and Pyspark programming.
• Hands on experience on Hadoop /Big Data related technology experience in Storage, Querying, Processing, and analysis of data.
• Optimized Machine learning algorithms for applications with 2M+ user
• Define overall data architecture and design.
• Responsible for Data Warehousing with help of SSIS services.
• Experience in using PL/SQL to write Stored Procedures, Functions and Triggers.
• Experience in development of Big Data projects using Hadoop, Hive, HDP, Pig, Flume, Storm and MapReduce open-source tools.
• Employed the help of SSRS for its reporting services
• Led and Managed Data Warehousing and Data Integration projects.
• Architected an enterprise-wide consumer Analytics data warehousing to facilitate onboarding, cross-selling and consumer acquisition programs.
• Experience in installation, configuration, supporting and managing Hadoop clusters.
• Experience in working with MapReduce programs using Apache Hadoop for working with Big Data.
• Experience in installation, configuration, supporting and monitoring Hadoop clusters using Apache, Cloudera distributions and AWS.
• Strong hands-on experience with AWS services, including but not limited to EMR, S3, EC2, route53, RDS, ELB, DynamoDB, CloudFormation, etc.
• Excellent working experience in Scrum / Agile framework and Waterfall project execution methodologies.
• Hands on experience in Hadoop ecosystem including Spark, Kafka, HBase, Scala, Pig, Hive, Sqoop, Oozie, Flume, Storm, big data technologies.
• Worked on Sparks, Spark Streaming and using CoreSparkAPI to explore Spark features to build data pipelines.
• Experienced in working with different scripting technologies like Python, UNIX shell scripts.
• Good Knowledge in Amazon Web Service (AWS) concepts like EMR and EC2 web services successfully loaded files to HDFS from Oracle, SQLServer, Teradata and Netezza using Sqoop.
• Excellent Knowledge in understanding Big Data infrastructure, distributed file systems -HDFS, parallel processing - MapReduce framework.
• Extensive knowledge in working with IDE Tools such as My Eclipse, RAD, IntelliJ, NetBeans
• Expert in Amazon EMR, Spark, Kinesis, S3, ECS, Elastic Cache, Dynamo DB, Redshift.
• Experience in installation, configuration, supporting and managing -Cloudera Hadoop platform along with CDH4&CDH5 clusters.
• Strong experience and knowledge of NoSQL databases such as MongoDB and Cassandra.
• Experience in working with different data sources like Flat files, XML files and Databases.
• Experience in database design, entity relationships and database analysis, programming SQL, stored procedures PL/SQL, packages and triggers in Oracle.

TECHNICAL SKILLS

Hadoop/Bigdata: MapReduce, Spark, SparkSQL, Azure, Spark Streaming, Kafka, PySpark, Pig, Hive, HBase, Flume, Flink, Yarn, Oozie, Zookeeper, Hue, Ambari Server

Java Technologies: Core Java, JDBC, JSP, Servlets, Tomcat

Programming Languages: HTML5, DHTML, WSDL, CSS3, C, C++, XML, R/R Studio, SAS Enterprise Guide, SAS, R (Caret, Weka, ggplot), Perl, MATLAB, Mathematica, FORTRAN, DTD, Schemas, Json, Ajax, Java, Scala, Python (NumPy, SciPy, Pandas, Gensim, Keras), Java Script, Shell Scripting

Web Technologies: J2EE, JMS, Web Service

Databases: SQL, NOSQL HBase, MYSQL, Oracle, PL/SQL.

Scripting: HTML, JavaScript, CSS, XML and Ajax

IDE: Eclipse, Microsoft Visual Studio 2008, 2012, Flex Builder

Operating System: Windows, Mac, Linux and UNIX

Protocols: TCP/IP, SSH, HTTP and HTTPS

Version control: Git, SVN, CVS

Tools: FileZilla, Putty, PL/SQL Developer, Junit

PROFESSIONAL EXPERIENCE

Confidential, San Francisco, CA

Sr.Big Data Engineer

Responsibilities:

• Implemented Restful web service to interact with Redis Cache framework.
• Intake happens through Sqoop, and Ingestion happens through Map Reduce, HBASE.
• Extensively worked on Spark Streaming and Apache Kafka to fetch live stream data.
• Interacting with other data scientists and architected custom solutions for data visualization using tools like Tableau, packages in R.
• Developed predictive models using Python & R to predict customers churn and classification of customers.
• Documenting the best practices and target approach for CI/CD pipeline
• Constructed product-usage SDK data and data aggregations by using PySpark, Scala, Spark SQL and Hive context in partitioned Hive external tables maintained in AWS S3 location for reporting, data science dashboarding, and ad-hoc analyses.
• Involved in data processing using an ETL pipeline orchestrated by AWS Data Pipeline using Hive.
• Installed Kafka manager for consumer lags and for monitoring Kafka Metrics also this has been used for adding topics, Partitions etc.
• Handled the importing of data from various data sources, performed transformations using hive,Map-Reduce, loaded data intoHDFSand extracted data fromMySQLintoHDFSusingSqoop. sources, channels and sink by which data is ingested into HDFS
• Responsible for performing various transformations like sort, join, aggregations, filter in-order to retrieve various datasets using apache spark.
• Experience in extracting appropriate features from datasets in-order to handle bad, null, partial records using spark SQL.
• Developed various data loading strategies and performed various transformations for analyzing the datasets by using Hortonworks Distribution for Hadoop ecosystem.
• Wrote Spark RDD transformations, actions, Data Frames, case classes for the required input data ad performed the data transformations using Spark Context to convert RDD to Data frames
• Worked on storing the dataframe into hive as table using Python (PySpark).
• Experienced in ingesting data into HDFS from various Relational databases like Teradata using sqoop and exported data back to Teradata for data storage.
• Experience in developing various spark application using Spark-shell (Scala).
• Involved in creating Hive Tables, loading with data, and writing Hive queries which will invoke and run MapReduce jobs in the backend.
• Designed and implemented Incremental Imports into Hive tables and writing Hive queries to run on TEZ.
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.
• Extracted data from multiple sources, applied transformations, loaded data into HDFS.
• Migrated ETL jobs to Pig scripts do Transformations, even joins and some pre-aggregations before storing the data onto HDFS.
• Hands on experience in developing apache SPARK applications using Spark tools like RDD transformations, Spark core, Spark MLlib, Spark Streaming and Spark SQL.
• Involved in writing optimized Pig Script along with developing and testing Pig Latin Scripts
• Implemented the workflows using Apache Oozie framework to automate tasks
• Worked on different file formats like Sequence files, XML files and Map files using MapReduce Programs.
• Exported data to Cassandra (NoSQL) database from HDFS using sqoop and performed various CQL commands on Cassandra to obtain various datasets as required.
• After performing all the transformations data is stored in MongoDB (NOSQL)using Sqoop.
• Created and imported various collections, documents into MongoDB and performed various actions like query, project, aggregation, sort, limit.
• Involved in Unit testing and delivered Unit test plans and results documents using Junit and MRUnit.

Environment: Hadoop, HDFS, Map Reduce, spark, Sqoop, Oozie, Pig, Kerberos, Hive, Flume, TEZ, LINUX, Java, Eclipse, Cassandra, python, MongoDB.

Confidential, Sacramento, CA

Sr. Big Data Engineer

Responsibilities:

• Performed ETL on data from different formats like JSON, Parquet.
• Analyzed the data by performingHivequeries (HiveQL) and runningPig Scripts(Pig Latin).
• Used Oozie Scheduler systems to automate the pipeline workflow and orchestrate the Spark jobs.
• Worked in Loading and transforming large sets of structured, semi structured, and unstructured data.
• Involved in collecting, aggregating, and moving data from servers to HDFS using Flume.
• Collecting data from various Flume agents that are imported on various servers using Multi-hop Flow.
• Created Hive UDFs and UDAFs using python scripts & Java code based on the given requirement
• Automated all the jobs to pull the data and load into Hive tables, using Oozie workflows
• Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior.
• Knowledge on microservices architecture in spring Boot integrating with various restful webservices.
• Created and maintained technical documentation for launching Hadoop Clusters and for executing Pig Scripts.
• Developed a Python Script to load the CSV files into the S3 buckets and created AWS S3 buckets, performed folder management in each bucket, managed logs and objects within each bucket.
• Developed SQOOP scripts to migrate data from Oracle to Big data Environment.
• Transformed Kafka loaded data using Spark-streaming with Scala and Python.
• Extensively worked with Avro and Parquet files and converted the data from either format Parsed Semi Structured JSON data and converted to Parquet using Data Frames in Spark.
• Transformed Kafka loaded data using Spark-streaming with Scala and Python.
• Used Sci-kit learn, Pandas, Numpy and Tensor flow to determine insights from data and created a trained credit fraud detection model for batch data.
• Created the framework for the dashboard using Tableau and optimized the same using open source Google optimization tools.
• Developed SQOOP scripts to migrate data from Oracle to Big data Environment.
• Created Airflow Scheduling scripts in Python to automate the process of Sqooping wide range of data sets.
• Involved in file movements between HDFS and AWS S3 and extensively worked with S3 bucket in AWS
• Converted all Hadoop jobs to run in EMR by configuring the cluster according to the data size
• Collated Real-time streaming data from credit agencies such as Transunion & Experian, performed data cleaning and fed the data into Kafka.
• Deployed model using RESTful APIs and used Dockers to facilitate multi-environment transition.
• Streaming data was stored using Amazon S3 deployed over EC2 and EMR cluster framework apart from in-house tools.

Environment: Podium Data, Data Lake, HDFS, Hue, AWS S3, Impala, Spark, Scala, Kafka, Looker, AWS EC2 and EMR

Confidential, Tempe, AZ

Big data Engineer

Responsibilities:

• Evaluated business requirements and prepared detailed specifications that follow project guidelines required to develop written programs.
• Modeled Hive partitions extensively for data separation and faster data processing and followed Hive best practices for tuning.
• Responsible for Big data initiatives and engagement including analysis, brainstorming, POC, and architecture.
• Loaded and transformed large sets of structured, semi structured and unstructured data using Hadoop/Big Data concepts.
• Lead the modern Data Architecture practice, deliver Big Data and Cloud Technologies Projects.
• Installed and Configured Apache Hadoop clusters for application development and Hadoop tools.
• Installed and configured Hive and written Hive UDFs and used repository of UDF's for Pig Latin.
• Developed data pipeline using Pig, Sqoop to ingest cargo data and customer histories into HDFS for analysis.
• Migrated the existing on-perm code to AWS EMR cluster.
• Installed and configured Hadoop Ecosystem components and Cloudera manager using CDH distribution.
• Applying machine learning libraries and algorithms optimize existing data.
• Involved in all phases of SDLC using Agile and participated in daily scrum meetings with cross teams.
• Worked on modeling of Dialog process, Business Processes and coding Business Objects, Query Mapper and JUnit files.
• Created automated pipelines in AWS Code Pipeline to deploy Docker containers in AWS ECS using S3.
• Used HBase NoSQL Database for real time and read/write access to huge volumes of data in the use case.
• Extracted Real time feed using Spark streaming and convert it to RDD and process data into Data Frame and load the data into HBase.
• Design and Develop ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, ORC/Parquet/Text Files into AWS Redshift.
• Developed AWS Lambda to invoke glue job as soon as a new file is available in Inbound S3 bucket.
• Created spark jobs to apply data cleansing/data validation rules on new source files in inbound bucket and reject records to reject-data S3 bucket.
• Research and develop state of the art techniques in the field of Machine Learning.
• Created HBase tables to load large sets of semi-structured data coming from various sources.
• Responsible for loading the customer's data and event logs from Kafka into HBase using REST API.
• Created tables along with sort and distribution keys in AWS Redshift.
• Create PySpark frame to bring data from DB2 to Amazon S3.
• Created shell scripts and python scripts to automate our daily tasks (includes our production tasks as well)
• Created, altered, and deleted topics using Kafka Queues when required with varying.
• Used cloud computing on the multi-node cluster and deployed Hadoop application on cloud S3 and used Elastic Map Reduce (EMR) to run a MapReduce.
• Developed analytics enablement layer using ingested data that facilitates faster reporting and dashboards.
• Worked with production support team to provide necessary support for issues with CDH cluster and the data ingestion platform.
• Provide guidance to development team working on PySpark as ETL platform.
• Created Hive External tables to stage data and then move the data from Staging to main tables
• Implemented the Big Data solution using Hadoop, hive and Informatica to pull/load the data into the HDFS system.
• Pulling the data from data lake (HDFS) and massaging the data with various RDD transformations.
• Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.
• Involved in converting Hive/SQL queries into Spark transformations using Spark data frames, Scala and Python.
• Developed and maintained batch data flow using HiveQL and unix scripting.
• Developed and execute data pipeline testing processes and validate business rules and policies.
• Built code for real time data ingestion using MapR-Streams.
• Implemented Spark using Python and Spark SQL for faster processing of data.
• Automation of unit testing using Python. Different testing methodologies like unit testing, Integration testing.
• Used HIVE join queries to join multiple tables of a source system and load them into Elastic Search Tables.
• Involved in writing SQL queries to validate the data between source and target systems
• Implemented different data formatter capabilities and publishing to multiple Kafka Topics.
• Written automated HBase test cases for data quality checks using HBase command line tools.
• Involved in development of Hadoop System and improving multi-node Hadoop Cluster performance.
• Developed and implemented Apache NIFI across various environments, written QA scripts in Python for tracking files.

Environment: Hadoop 3.0, MapReduce, Hive 3.0, Agile, HBase 1.2, PySpark, NoSQL, AWS, Kafka, Pig 0.17, HDFS, Java 8, Hortonworks, Spark, PL/SQL, Python

Confidential, Whippany, NJ

Big data engineer

Responsibilities:

• Evaluated business requirements and prepared detailed specifications that follow project guidelines required to develop written programs.
• Loaded and transformed large sets of structured, semi structured and unstructured data using Hadoop/Big Data concepts.
• Created the automated build and deployment process for application, re-engineering setup for better user experience, and leading up to building a continuous integration system.
• Implemented MapReduce programs to retrieve results from unstructured data set.
• Optimized MapReduce Jobs to use HDFS efficiently by using various compression mechanisms.
• Worked on and designed Big Data analytics platform for processing customer interface preferences and comments using Hadoop, Hive and Pig, Cloudera.
• Importing and exporting data into HDFS and Hive using Sqoop from Oracle and vice versa.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
• Responsible for ETL and Data Architecture projects.
• Worked on reading multiple data formats on HDFS using Python.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Python.
• Installed and configured Pig and also written Pig Latin scripts.
• Developed multiple POCs using Python and deployed on the Yarn cluster, compared the performance of Spark, with Hive and SQL.
• Analyzed the SQL scripts and designed the solution to implement using Python.
• Build data platforms, pipelines, and storage systems using the Apache Kafka, Apache Storm and search technologies such as Elastic search.
• Worked on querying data using Spark SQL on top of PySpark engine.
• Experienced in implementing POC's to migrate iterative MapReduce programs into Spark transformations using Python.
• Developed Spark scripts by using Python shell commands as per the requirement.
• Experienced with batch processing of data sources using Apache Spark, Elastic search.
• Experienced in AWS cloud environment and on S3 storage and EC2 instances
• Developed Spark jobs using Python in test environment for faster data processing and used Spark SQL for querying.
• Configured Spark streaming to receive real time data from the Kafka and store the stream data to HDFS.
• Designed and implemented SOLR indexes for the metadata that enabled internal applications to reference Scopus content.
• Used Spark for Parallel data processing and better performances using Python.
• Extensively used Pig for data cleansing and extract the data from the web server output files to load into HDFS.
• Developed a data pipeline using Kafka and Storm to store data into HDFS.
• Implemented Kafka producers create custom partitions, configured brokers and implemented High level consumers to implement data platform.
• Involved in creating Hive tables, loading with data and writing hive queries which will run internally in MapReduce way.
• Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it using MapReduce programs.
• Developed simple to complex MapReduce streaming jobs using Python.

Environment: Pig 0.17, Hive 2.3, HBase 1.2, Sqoop 1.4, Flume 1.8, zookeeper, AWS, MapReduce, HDFS, Cloudera, Python, Spark 2.3, SQL, Apache Kafka 1.0.1, Apache Storm, Python, Unix.

Confidential, Philadelphia, PA

Big data engineer

Responsibilities:

• Contributing to the development of key data integration and advanced analytics solutions leveraging Apache Hadoop and other big data technologies for leading organizations using major Hadoop Distributions like Hortonworks.
• Involved in Agile methodologies, daily Scrum meetings, Sprint planning.
• Performed Data transformations in HIVE and used partitions, buckets for performance improvements.
• Created Hive external tables on the MapReduce output before partitioning; bucketing is applied on top of it.
• Gathered data requirements and design enhancements to the data warehouse.
• Developed business specific Custom UDF's in Hive.
• Developed end to end architecture design on bigdata solution based on variety of business use cases.
• Worked as a Spark Expert and performance Optimizer.
• Member of Spark COE (Center of Excellence) in Data Simplification project at Cisco
• Experienced with Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark
• Handled Data Skewness in Spark-SQL
• Implemented Spark using Python, Java and utilizing Data frames and Spark SQL API for faster processing of data
• Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data
• Developed a data pipeline using Kafka, HBase, Spark and Hive to ingest, transform and analyzing customer behavioral data also developed Spark jobs and Hive Jobs to summarize and transform data
• Worked extensively with importing metadata into Hive and migrated existing tables and applications to work on Hive and Spark
• Implemented Sqooping from Oracle to Hadoop and load back in parquet format.
• Enhancements to traditional data warehouse based on STAR schema, update data models, perform Data Analytics and Reporting using Tableau.
• Used Spark for interactive queries, processing of streaming data and integration with popular NoSQL database for huge volume of data; Worked under Map Distribution and familiar with HDFS
• Handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive, MapReduce and then loading data into HDFS.
• Collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis.
• Designed and maintained Test workflows to manage the flow of jobs in the cluster.
• Worked with the testing teams to fix bugs and ensured smooth and error-free code
• Preparation of docs like Functional Specification document and Deployment Instruction documents
• Fixed defects during the QA phase, support QA testing, troubleshoot defects and identify the source of defects.
• Involved in installing Hadoop Ecosystem components (Hadoop, MapReduce, Spark, Hive, Sqoop, Flume, Zookeeper and HBase).
• Worked collaboratively with all levels of business stakeholders to architect, implement and test Big Data based analytical solution from disparate sources.

Environment: AWSS3, RDS, EC2, Redshift, Hadoop 3.0, Hive 2.3, Sqoop 1.4.6, Oozie, HBase 1.2, Flume 1.8, Hortonworks, MapReduce, Kafka, HDFS, Oracle 12c, Microsoft, Java, GIS, Spark 2.2, Zookeeper

Confidential

Hadoop Admin/ Developer

Responsibilities:

• Implemented solutions for ingesting data from various sources and processing the Data-at-Rest utilizing Bigdata technologies such as Hadoop, Map Reduce Frameworks, HBase, and Hive.
• Used Sqoop to efficiently transfer data between databases and HDFS and used Flume to stream the log data from servers.
• Developed full SDLC of AWS Hadoop cluster based on client's business need
• Involved in loading and transforming large sets of structured, semi structured and unstructured data from relational databases into HDFS using Sqoop imports.
• Implement enterprise grade platform (mark logic) for ETL from mainframe to NOSQL (Cassandra)
• Responsible for importing log files from various sources into HDFS using Flume
• Analyzed data using HiveQL to generate payer by reports for transmission to payer's form payment summaries.
• Imported millions of structured data from relational databases using Sqoop import to process using Spark and stored the data into HDFS in CSV format.
• Responsible for designing and formulating the architecture, development, and engineering of Big Data solutions of a company.
• Ability to install and deploy the Hadoop cluster, add and remove nodes, monitor tasks and all the critical parts of the cluster, configure name-node, take backups.
• Used Data Frame API in Python for converting the distributed collection of data organized into named columns.
• Performed data profiling and transformation on the raw data using hive, Python, and Java.
• Responsible for keeping the Hadoop clusters running smoothly in production.
• Developed predictive analytic using Apache Spark, Python APIs.
• Involved in working of big data analysis using hive and User defined functions (UDF).
• Created Hive External tables and loaded the data into tables and query data using HQL.
• Implemented Spark Graph application to analyze guest behavior for data science segments.
• Enhancements to traditional data warehouse based on STAR schema, update data models, perform Data Analytics and Reporting using Tableau.
• Involved in migration of data from existing RDBMS (oracle and SQL server) to Hadoop using Sqoop for processing data.
• Developed Shell, Perl and Python scripts to automate and provide Control flow to hive scripts.
• Developed prototype for Big Data analysis using Spark, RDD, Data Frames and Hadoop eco system with CSV, JSON, parquet and HDFS files.
• Developed Hive SQL scripts for performing transformation logic and loading the data from staging zone to landing zone and Semantic zone.
• Involved in creating Oozieworkflow and Coordinator jobs for Hive jobs to kick off the jobs on time for data availability.
• Worked on Oozie scheduler to automate the pipeline workflow and orchestrate the Sqoop, hive jobs that extract the data on a timely manner.
• Exported the generated results to Tableau for testing by connecting to the corresponding Hive tables using Hive ODBC connector.
• Exported data from HDFS to RDBMS via Sqoop for Business Intelligence, visualization and user report generation.
• Managed and lead the development effort with the help of a diverse internal and overseas group.