SUMMARY

• Experience in developing & leading teh end to end implementation of Big Data projects, comprehensive experience in Hadoop Ecosystem like Hadoop, Map Reduce, Hadoop Distributed File System(HDFS), HIVE, PIG, Python, Yarn, Ozie, Flume, Hue, Spark.
• Worked on Importing and exporting data from different databases like Oracle, Teradata into HDFS and Hive using Sqoop.
• Worked on Talend Data Mapper(TDM) for avro schema creation and converting xml,csv file to avro .
• Created complex mappings in Talend 6.4.1/6.3.1 using tMap, tDie, tJoin, tReplicate, tFilterRow, tParallelize, tFixedFlowInput, tAggregateRow, tIterateToFlow etc.
• Created Context Groups, Generic schemas and also context Variables to run jobs against different environments like Dev, Test and Prod.
• Experience in converting Hive/SQL queries into Spark transformations using Talend/Java
• Experience in collecting and storing stream data like log data in HDFS using Apache Flume.
• Wrote Hive and Pig queries for data analysis to meet teh business requirements.
• Involved in Creating tables, partitioning, bucketing of table and creating UDF's in Hive.
• Strong trouble - shooting and problem-solving skills wif a Logical and pragmatic attitude
• Team player wif strong oral and interpersonal skills
• Work wif business to gather requirements and define teh Data Quality solutions for data profiling, standardization and cleansing etc.
• Define and contribute to development of standards, guidelines, design patterns and common development frameworks & components
• Experience in analyzing data using HiveQL, Pig Latin, and custom Map Reduce programs in Java.
• Strong Technical background, excellent analytical ability, team player and goal oriented, wif a commitment toward excellence.
• Highly organized wif teh ability to manage multiple projects and meet deadlines.

TECHNICAL SKILLS

Hadoop ecosystem: Map Reduce, Sqoop, Hive, Impala, Oozie, Hue, PIG, HBase, HDFS, Zookeeper, Flume, Spark

ETL: Talend Big Data Studio7.3.1, 6.4.1,6.3.1

Databases: Teradata (13.0/14.10), Oracle 11g (SQL, PL/SQL Basics), SQL Server 2005, DB2, HBase

Tools: & Utilities: Toad, Microsoft Visio, Winscp, Appworx, Control-M, Remedy, AutoSys

Languages: Java, SQL, PL/SQL, Pig Latin, HiveQL, Unix shell, Python scripting, Scala

Operating System: Windows 2010,7, UNIX, Linux

Domain Knowledge: Finance, Banking, Telecom, Healthcare, Insurance, Manufacturing

Methodologies: Agile, Waterfall

PROFESSIONAL EXPERIENCE

Confidential, Richardson, TX

Big Data Developer

Responsibilities:

• Involved in loading data from UNIX file system to HDFS using command and scripts.
• Attending daily meeting wif Customer to find out teh exact requirement and providing teh technical solution to meet teh customer requirement.
• Gather and analyze business and technical requirements.
• Developed MapReduce programs to parse teh raw data, populate staging tables and store teh refined data in partitioned tables in teh EDW.
• Created Hive queries dat helped analysts spot emerging trends by comparing fresh data wif EDW reference tables and historical metrics.
• Evaluated teh performance of Apache Spark in analyzing genomic data.
• Enabled speedy reviews and first mover advantages by using Oozie to automate data loading into teh Hadoop Distributed File System and PIG to pre-process teh data.
• Prepare Unit Test Plan/ Design (wif direction from customer).
• Experience to prepare HLD, LLD, UTC, Tech Design Document.
• Loading data from different source (database & files) into Hive using Talend tool.
• Data migration from relational (Oracle. Teradata) databases or external data to HDFS using Sqoop and Flume & Spark.
• Used Talend most used components (tMap, tDie, tConvertType, tFlowMeter, tLogCatcher, tRowGenerator, tSetGlobalVar, tHashInput & tHashOutput and many more)
• Utilized Big Data components like tHDFSInput, tHDFSOutput, tPigLoad, tPigFilterRow, tPigFilterColumn, tPigStoreResult, tHiveLoad, tHiveInput, tHbaseInput, tHbaseOutput, tSqoopImport and tSqoopExport.
• Worked on converting Talend big datajobs in to Azure notebooks.
• Analyzed teh data by performing Hive queries and running Pig scripts.
• Designed both Managed and External tables in Hive to optimize performance.
• Regular monitoring of Hadoop Cluster to ensure installed applications are free from errors and warnings.
• Exposure in optimizing Hive queries using Partitioning and Bucketing techniques
• Migrated data on promise datalake to azure synapse
• Used dockers coantainer for Jenkins builds
• Involved in running Hadoop streaming jobs to process terabytes of text data using Flume.
• Implemented Hive UDF's for evaluation, filtering, loading and storing of data.
• Developed HIVE queries for teh analysts.
• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
• Diligently teaming wif teh infrastructure, network, database, application and business intelligence teams to guarantee high data quality and availability.
• Monitor System health and logs and respond accordingly to any warning or failure conditions.

Environment: Hadoop, HDFS, Map Reduce, Scala, Python, Azure Synapse, Hive, ASG Zena,Pig, Sqoop, Spark, Db2, Oracle, Hbase,Teradata, SQL,Eclipse, UNIX,Talend Real-Time Bigdata Platform

Confidential, Lewisville, Texas

Senior Big Data developer

Responsibilities:

• Worked on developing ETL applications.
• Create mapping, sessions and workflows, as per requirement of business to implement teh logic.
• Created a frame work which copy’s data from any Linux based file system to MongoDB.
• Worked on tuples, dictionaries, object-oriented concepts based inheritance features for making algorithms.
• Queried MongoDB database queries from pythonusing pymongo to retrieve information.
• Design reports, dashboards and data visualization using Matplotlib.
• Successfully interpreted data to draw conclusions for managerial action and strategy.
• For larger datasets used Pyspark Data Framesto analyze teh data.
• Modified queries, functions, cursors, triggers and stored procedures for MongoDB to improve performance, while processing data.
• Performed troubleshooting, fixed and deployed many Pythonbug fixes of teh two main applications dat were a main source of data for both customers and internal customer service team.
• Used Pandas to put teh data as time series and tabular format for east timestamp data manipulation and retrieval.
• Cleaned data and processed third party spending data into maneuverable deliverables wifin specific formats wif pythonlibraries. Used TDD (Test driven development) methodology.
• An individual wif excellent interpersonal and communication skills, strong business acumen, creative problem solving skills, technical competency, team-player spirit, and leadership skills.
• Strong oral and written communication, initiation, interpersonal, learning and organizing skills matched wif teh ability to manage time and people TEMPeffectively.

Environment: Python, MongoDB, SFTP, Linux, MapReduce, HDFS, Hive, PyCharm, Jupyter Notebook, UNIX Shell Scripting.

Confidential, Rochester, New York

Big Data / AWS developer

Responsibilities:

• Worked on bigdata infrastructure build out for batch, transactional processing as well as real-time processing.
• Developed systems on Hadoop and Amazon Web Services.
• Brought teh AWS-based system to production in 9 months.
• AWS system built wif Lambda, API Gateway, DynamoDB, and S3.
• Automated big-data processing using EMRand Spark.
• Developed deployment pipelines wif Code Pipeline, Code Build, and Cloud Formation, Terraform.
• Developed components for teh normalization Rest service using Scala.
• Developed Spark scripts by using Scala Shell commands as per teh requirement.
• Good working noledge on AWSIAM service, IAM policies, Roles, Users, Groups, AWSaccess keys and Multi Factor Autantication. And migrated applications to teh AWS Cloud.
• Experience wif AWSCommand line interface and PowerShell for automating administrative tasks. Defined AWSSecurity Groups which acted as virtual firewalls dat controlled teh traffic reaching one or more AWS EC2, LAMBDA instances.
• Hand-On experience in Implementing, Build and Deployment of CI/CD pipelines using Jenkins, managing projects often includes tracking multiple deployments across multiple pipeline stages (Dev, Test/QA staging and production).
• Worked on transferring data from on - perm to dynamo db.
• Designed and developed POCs in Spark using Scala to compare teh performance of Spark wif Hive and dynamo db.
• Hands on Experience in Oozie Job Scheduling.
• Developed Kafka producer and consumers, HBase clients, Sparkand Hadoop MapReduce jobs along wif components on HDFS, Hive.
• Exploring wif teh Sparkimproving teh performance and optimization of teh existing algorithms in Hadoop.
• Involved in converting Hive/SQL queries into Sparktransformations using SparkRDD and Scala.
• Agile methodology was used for development using XP Practices (TDD, Continuous Integration).
• Exposure to burn-up, burn-down charts, dashboards, velocity reporting of sprint and release progress.
• An individual wif excellent interpersonal and communication skills, strong business acumen, creative problem solving skills, technical competency, team-player spirit, and leadership skills.
• Strong oral and written communication, initiation, interpersonal, learning and organizing skills matched wif teh ability to manage time and people TEMPeffectively.

Environment: MapReduce, HDFS, AWS, Hive, Hue, Oozie, Bigdata, Core Java, Eclipse, Hbase, Spark, Scala, Kafka, Jenkins, Cloudera Manager, LINUX, Puppet, IDMS, U

Confidential, Sunnyvale, California

Big Data Developer

Responsibilities:

• Worked on bigdata infrastructure build out for batch processing as well as real-time processing.
• Developed, Installed and configured Hive, Hadoop, Bigdata, hue, Oozie, pig, Sqoop, Kafka, Elastic Search, Java, J2EE, HDFS, XML, PHP and Zookeeper on teh Hadoop cluster.
• Created Hive Tables, loaded retail transactional data from Teradata using Scoop.
• Managed thousands of Hive databases totaling 250+ TBs.
• Developed enhancements to Hive architecture to improve performance and scalability.
• Collaborated wif development teams to define and apply best practices for using Hive.
• Worked on Hadoop, Hive, Oozie, and MySQL customization for batch data platform setup.
• Worked on implementation of a log producer in SCALA dat watches for application logs, transforms incremental logs and sends them to a Kafka and Zookeeper based log collection platform.
• Implemented a data export application to fetch processed data from these platforms to consuming application databases in a scalable manner.
• Extending Hive functionality by writing custom UDF’s and UDTF’s.
• Developed Hive UDTF’s and executed them using spark
• Involved in loading data from Linux file system to HDFS.
• Experience in setting up salt-formulas for centralized configuration management.
• Monitoring Cluster using various tools to see how teh nodes are performing.
• Experience on Oozie workflow scheduling.
• Developed Spark scripts by using Scala Shell commands as per teh requirement.
• Expertise in cluster task like adding Nodes, Removing Nodes wifout any TEMPeffect to running jobs and data.
• Transferred data from Hive tables to HBase via stage tables using Pig and used Impalafor interactive querying of HBase tables.
• Implementation of auditing teh data for accounting by capturing various logs like HDFS Audit logs, YarnsAudit logs, Audit logs.
• Worked on a proof of concept to implement Kafka-Stormbased data pipeline.
• Configured job scheduling in Linux using shell scripts
• Worked on Machine Learning Library (MLib) for clustering and underlying optimization primitives.
• Created custom Solr Query components to enable optimum search matching.
• Utilized teh Solr API to develop custom search jobs and GUI based search applications.
• Also, implemented multiple output formats in teh same program to match teh use cases.
• Developed Hadoop streaming Map/Reduce works using Python.
• Installation of Apache SPARK on Yarnand managing Master and Worker nodes.
• Performed benchmarking of teh No-SQL databases, Cassandra and HBase.
• Created data model for structuring and storing teh data efficiently. Implemented partitioning and bucketing of tables in Cassandra.
• Implemented test scripts to support test driven development and continuous integration.
• Clear understanding of ClouderaManager Enterprise edition.
• Design and developed Microsites, Aggregators and Micro Services.
• Good experience in Hive partitioning, bucketing and perform different types of joins on Hive tables and implementing Hive serdes.
• Working on POC and implementation & integration of Cloudera for multiple clients.
• Good noledge on Creating ETLjobs to load Twitter JSON data into MongoDB and jobs to load data from MongoDB into Data warehouse.
• Design and Implement teh Various ETL Projects using Informatica, Data stage as data integration tool
• Exported teh analyzed data to HBase using Sqoop and to generate reports for teh BI team.
• POC work is going on using Spark and Kafka for real time processing.
• Developed Python scripts to monitor health of Mongodatabases and perform ad-hoc backups using Mongodump and Mongorestore.
• Deployed teh project in Linuxenvironment
• Automated teh apache installation and its components using salt.
• Populated HDFS and Cassandra wif huge amounts of data using Apache Kafka.
• Worked wif NoSQL databases like Cassandra and Mongo DB for POC purpose.
• Implement POC wif Hadoop. Extract data wif Spark into HDFS.

Environment: MapReduce, HDFS, Hive, Pig, Hue, Oozie, Solr, Bigdata, Core Java, Python, Eclipse, Hbase, Flume, Spark, Scala, Kafka, Cloudera Manager, Impala, UNIX RHEL, Cassandra, LINUX

Confidential

Java Developer

Responsibilities:

• Developing new pages for personals.
• Implementing MVC Design pattern for teh Application.
• Used Content Management tool Dynapub for publishing data.
• Implementing AJAX to represent data in friendly and efficient manner.
• Developing and Action Classes.
• Used JMeter for load testing of teh application and captured teh response time of teh application
• Created simple user interface for application's configuration system using MVC design patterns and swing framework.
• Implementing Log4j for logging and debugging.
• Implementing Form based approach for ease of programming team.

Environment: Core Java, Java Swing, Struts, J2EE (JSP/Servlets), XML, AJAX, DB2, My SQL, Tomcat, JMeter.