SUMMARY

• Having 8 plus years of experience in IT experience in software design, development, implementation and support of business applications for Telecom, health and Insurance industries
• 4 plus years of experience in Big data Hadoop, Hadoop Ecosystem components like MapReduce, Sqoop, Flume, Kafka, Pig, Hive, Spark, Storm, Psypark, Partitioning, shuffling, Perquet, CSV, HBase, Oozie, and Zookeeper
• Worked extensively on installing and configuring Hadoop ecosystem components Hive, SQOOP, PIG, HBase, Zookeeper and Flume
• Implemented ETL operations using Big Data platform
• Good Knowledge in Spark and Scala
• Hands of experience on build tools like Maven, Log4j, Junit and Ant
• Working with the data extraction, transformation and load in Hive, Pig and HBase
• Hands on Experience in designing and developing applications in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.
• Experience in understanding the security requirements for Hadoop and integrate with Kerberos authentication and authorization infrastructure.
• Hands on experience on Streaming data ingestion and Processing
• Experience in designing different time driven and data driven automated workflows using Oozie.
• Expertise in writing the Real - time processing application Using spout and bolt in Storm.
• Experience in configuring the Zookeeper to coordinate the servers in clusters and to maintain the data consistency
• Used Python and Django to interface with the jQuery UI and manage the storage and deletion of content.
• Acumen on Data Migration from Relational Database to Hadoop Platform using SQOOP.
• Experienced in migrating ETL transformations using Pig Latin Scripts, transformations, join operations.
• Good understanding of MPP databases such as HP Vertica and Impala.
• Hands on experience in configuring and working with Flume to load the data from multiple sources directly into HDFS
• Experience in using design pattern, Java, JSP, Servlets, JavaScript, HTML, JQuery, Angular JS, Mobile JQuery, JBOSS 4.2.3, XML, Web Logic, SQL, PL/SQL, JUnit, and Apache-Tomcat, Linux.
• Expertise in relational databases like Oracle, My SQL and SQL Server.
• Experience in implementing projects both in Agile and Waterfall methodologies.
• Having Good Experience on Cloud Technologies AWS, Azure and GCP
• Strong Experience on Data Warehousing ETL concepts using Informatica Power Center, OLAP, OLTP and AutoSys.
• Strong analytical and problem solving skills, highly motivated, good team player with very Good communication & interpersonal skill

PROFESSIONAL EXPERIENCE

Confidential

Big Data Engineer

Responsibilities:

• Expertise in designing and deployment of Hadoop cluster and different Big Data analytic tools including Hive, Autosys, Spark, Impala with Cloudera distribution
• Implemented Performance tuning to long running jobs and added Parallelism to jobs
• Created views on top of existing data to transform with HiveQL
• Reviewed Hadoop cluster configuration, application configuration and configure required parameters
• Create distributed datasets from any storage source supported by Hadoop, local file systems, HDFS, Cassendra, HBase and others.
• Perfom computations and tasks on large sets of data by analysing them.
• Optimize the Pyspark jobs to run on kubernetes cluster for faster data processing.
• Perform efficient data storage, retrieval, data compression, encoding schemes and handle complex data in bulk by using Perquet and CSV.
• Distribute data across multiple tables, disks or sites to improve processing performance by using data transformation techniques like Partitioning and shuffling.
• Built processing flows with UC4 scheduler and work on data cleansing routines
• Supports production applications for daily loads and automate them using autosys
• Performed hive query optimization techniques for integration
• Responsible to run Benchmarking and work on fine tuning the applications
• Performed Monthly refresh activities and produce data for MBRs on monthly basis
• Executed SQL statements to execute validation on the application data and update the data layouts
• Experience with collaborating with other teams like Product, Technology, Risk, Compliance, Legal, and Operations
• Worked with teams in different geographical region to deliver solutions
• Managed and Deployed applications as Docker Containers to Aws Cloud Infrastructure
• Deployed Elastic Load balancer for scalable and high available on AWS
• Implemented and managed Continuous automation on AWS Cloud with cloud formation Terraform
• Having Good Knowledge on cloud technologies as AWS, Azure and Google GCP

Confidential, Chicago

Hadoop Developer

Responsibilities:

• Involved in Migrating applications from Cloudera to Hortonworks. Experience in working with various Cloudera distributions (CDH4/CDH5) and have knowledge on Hortonworks
• Expertise in designing and deployment of Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase, Oozie, Sqoop, Flume, Spark, Impala.
• Responsible for data extraction and data ingestion from different data sources into Hadoop Data Lake by creating ETL pipelines using Sqoop and Hive
• Professional Java developer with strong expertise in data engineering and big data technologies.
• Hands on experience in programming using Java, Scala and SQL.
• Analyzed the data using HiveQl to identify the different correlations and used core Java technologies to create Hive/Pig UDFs to use in the project.
• Implemented complex MapReduce programs to perform joins on the Map side using Distributed Cache in Java.
• Experience in performance tuning the Hadoop cluster by gathering and analyzing the existing infrastructure.
• Continuous monitoring and managing the Hadoop cluster using Cloudera Manager and Ambari
• Created multiple Map Reduce Jobs using Java API, Pig and Hive for data extraction
• Wrote ETL jobs to read from web APIs using REST and HTTP calls and loaded into HDFS using java and Talend.
• Create distributed datasets from any storage source supported by Hadoop, local file systems, HDFS, Cassendra, HBase and others.
• Perfom computations and tasks on large sets of data by analysing them.
• Optimize the Pyspark jobs to run on kubernetes cluster for faster data processing.
• Perform efficient data storage, retrieval, data compression, encoding schemes and handle complex data in bulk by using Perquet and CSV.
• Distribute data across multiple tables, disks or sites to improve processing performance by using data transformation techniques like Partitioning and shuffling.
• Operated on Java/J2EE systems with different databases, which include Oracle, MySQL and DB2.
• Developed multiple MapReduce jobs in Java for data cleaning and preprocessing.
• Worked on Talend ETL tool and used features like context variable and database components like input to oracle, output to oracle, tFile compare, tFile copy, to oracle close ETL components
• Created ETL Mapping with Talend Integration Suite to pull data from Source, apply transformations, and load data into target database.
• Worked with D-Series for Scheduling Sqoop jobs
• Used HiveQL for data analysis like creating tables and import the structured data to specified tables for reporting.
• Hands on experience in application development using Java, RDBMS, and Linux shell scripting.
• Designed and Implemented Azure architectures, environments, and resources
• Implemented migrations of applications and databases onto Azure
• Contributed or author blogs, whitepapers, and presentations on Azure technical and strategic topics
• Having experience on creating databases, tables and views in HIVEQL and IMPALA
• Working with data delivery team to setup new Hadoop users, Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig and MapReduce access for the new users on Horton works
• Good knowledge in using Hibernate for mapping Java classes with database and using Hibernate Query Language (HQL).
• Extensive hands on experience in writing complex MapReduce jobs and Hive data modeling.
• Involved in Agile methodologies, daily scrum meetings, spring planning.

Confidential

Hadoop Developer

Responsibilities:

• Developed Schedulers that communicated with the Cloud based services (AWS) to retrieve the data.
• Designed and implemented HIVE queries and functions for evaluation, filtering, loading and storing of data.
• Creating Hive tables and working on them using HiveQL.
• Developed data pipeline using Kafka and Storm to store data into HDFS.
• Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.
• Involved in review of functional and non-functional requirements.
• Implemented Frameworks using Java and python to automate the ingestion flow.
• Developed views and templates with Python and Django's view controller and templating language to create a user-friendly website interface.
• Experience in implementing python alongside using various libraries such as graphs, MySQL db for database connectivity, python-twitter.
• Developed a fully automated continuous integration system using Git, Gerrit, Jenkins, MySQL and custom tools developed in Python and Bash.
• Responsible to manage data coming from different sources.
• Loaded the CDRs from relational DB using Sqoop and other sources to Hadoop cluster by using Flume.
• Experience in processing large volume of data and skills in parallel execution of process using Talend functionality.
• Involved in loading data from UNIX file system and FTP to HDFS.
• Developed Hive queries to analyze the output data.
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.
• Designed Cluster co-ordination services through Zookeeper. Collected the logs data from web servers and integrated in to HDFS using Flume.
• Used HIVE to do transformations, event joins and some pre-aggregations before storing the data onto HDFS.
• Designed and implemented Spark jobs to support distributed data processing.
• Supported the existing MapReduce Programs those are running on the cluster.
• Wrote the shell scripts to monitor the health check of Hadoop daemon services and respond accordingly to any warning or failure conditions.
• Involved in Migrating the Hive Data to Google BigQuery
• Implementing the Automatic workflows with apache Airflow and Integrated the scripts with Jenkins
• Used Jenkins to deploy code to Google Cloud with new namespaces, create Docker images and push them to container registry of Google Cloud.’
• Expertise in documenting and deployment process and high-level preparation of Release notes, Checklists, Quality process docs, Analysis docs, configuration docs with versions.
• Lead many formal and informal sessions to educate the issues of security and the importance of best practices in GCP.
• Expertise in designing the Google Cloud architecture by following the financial regulations from security point of view.
• Expertise in several GCP service focusing on Security, Kubernetes and Biq Query.
• Expertise in automation of the infrastructure using Terraform for both AWS and GCP.
• Created HIVE table to store the processed results in tabular format.
• Involved in Building multitenant solutions using Python and internal tools, delivering complex cloud platforms
• Worked with Spark, improving the performance and optimization of the existing applications in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN
• Monitored Hadoop Jobs and Reviewed Logs of the failed jobs to debug the issues based on the errors
• Fine tune Hadoop applications for high performance and throughput
• Worked on POC on the real time streaming the data using Spark with Kafka.
• Support development with application architecture in both real time and batch big data processing
• Used Spark API over Hadoop YARN as execution engine for data analytics using Hive
• Worked on different file formats (ORCFILE, TEXTFILE) and different Compression Codecs (GZIP, SNAPPY, LZO).

Confidential

Java Developer

Responsibilities:

• Involved in Agile - Sprint methodologies of SDLC for project management design, development,
• Designed and implemented the training and reports modules of the application using Servlets, JSP and Ajax
• Involved in Agile - Sprint methodologies of SDLC for project management design, development
• Experience in using Spring Integration and RabbitMQ for creation of web services and communication.
• Interact with Business Users and Develop Custom Reports based on the criteria defined. Requirement gathering and information collection. Analysis of gathered information so as to prepare a detail work plan and task breakdown structure
• Developed custom JSP tags for the application
• Involved in the phases of SDLC (Software Development Life Cycle) including Requirement collection, Design and analysis of Customer specification, Development and Customization of the application
• Used Postman to trigger HTTP requests making the SOAP and REST based APIs work faster.
• Created and consumed SOAP and REST services using CXF and used Mule ESB to route various calls to do validation of service input and to handle exceptions.
• Used Quartz schedulers to run the jobs in a sequential with in the given time
• Implemented the reports module applications using jasper reports for business intelligence
• Good Experience in Exposure to Writing SQL/Transact-SQL (DDL, DML and DCL), Developing, Creating New Database and Database Objects Such as Tables, Views, Indexes, Complex Stored
• Deployed application on tomcat server for business application in client location
• End-to-End System development and testing of Unit integration and System integration
• Co-ordination activities with Onshore and Offshore team of 10+ members
• Responsible for Effort estimation and timely production deliveries
• Creation and Execution of half yearly and yearly load jobs which updates new rate and discounts etc. for the claim calculations in Database and Files
• Extensively used Java multi-threading to implement batch Jobs with JDK 1.5 features
• Configured the project on Web Logic 10.3 application servers
• ·Implemented the online application using Core Java, JDBC, JSP, Servlets, spring, Hibernate, Web Services, SOAP, and WSD