SUMMARY

• Having around 6+ years of total IT experience with over 4+ years experience in AWS Big Data Hadoop, 2 years of experience in Development and Design of Java based enterprise applications.
• Extensive working experience on Hadoop eco - system components like HDFS, MapReduce, Hive, Sqoop, Flume, Spark, Kafka, Oozie and Zookeeper.
• Implemented performance tuning techniques for Spark-SQLqueries.
• Strong knowledge on Hadoop HDFS architecture, Map-Reduce(MRv1) and YARN(MRv2) framework.
• Strong hands on Experience in publishing teh messages to various Kafka topics using Apache NIFI and consuming teh message to HBase using Spark and Python.
• Worked on creating Spark jobs dat process teh true source files and successful in performing various transformations on teh source data using Spark Dataframe, Spark SQL API's.
• Developed Sqoop scripts to migrate data from Teradata, Oracle to Bigdata Environment.
• Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming teh data to uncover insights into teh customer usage patterns.
• Worked with Hue GUI in scheduling jobs with ease and File browsing, Job browsing, Metastore management.
• Experience in importing and exporting teh data using Sqoop from HDFS to Relational Database systems and vice-versa.
• Hands on experience in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4), Yarn distributions (CDH 5.X).
• Implemented real time data streaming pipeline using AWS Kinesis, Lambda, and Dynamo DBand deployed AWS Lambda codefrom Amazon S3 buckets.
• Integrated AWS DynamoDB using AWS Lambda to store teh value items and backup teh DynamoDB streams.
• Experienced in AWS Elastic Beanstalk for app deployments and worked on AWS Lambda with Amazon Kinesis.
• Developing a Marketing Cloud Service on Amazon AWS. Developed serverless application using AWS Lambda, S3, Redshift and RDS.
• Work on large scale data transfer across different Hadoop clusters, implement new technology stacks on Hadoop clusters using Apache Spark.
• Wrote Python scripts to parse XML documents and load teh data in database.
• Added support for AWS S3 and RDS to host static/media files and teh database into Amazon Cloud.
• Experience in project deployment using Heroku/Jenkins and using web services like Amazon Web Services (AWS) EC2, AWS S3, Autoscaling, CloudWatch and SNS.
• Performed Data scrubbing and processing with Oozie and for workflow automation and coordination.
• Hands on experience in analyzing log files for Hadoop and eco-system services and finding root cause.
• Hands on experience on handling different file formats like AVRO, PARQUET, Sequential files, MAP Files, CSV, xml, log ORC and RC.
• Experience with NoSQL Database HBase, Cassandra, MongoDB.
• Experience with AIX/Linux RHEL, Unix Shell Scripting and SQL Server 2008.
• Worked on data search tool Elastic Search and data collection tool Logstash.
• Strong knowledge in Hadoop cluster installation, capacity planning and performance tuning, benchmarking, disaster recovery plan and application deployment in production cluster.
• Experience in developing stored procedures, triggers using SQL, PL/SQL in relational databases such as MS SQL Server 2005/2008.
• Exposed into methodologies Scrum, Agile and Waterfall.

TECHNICAL SKILLS

Programming Languages: Java, Scala, Python, SQL, and C/C++

Big Data Ecosystem: Hadoop, MapReduce, Kafka, Spark, Pig, Hive, YARN, Flume, Sqoop, Oozie, Zookeeper, Talend.

Hadoop Distributions: Cloudera Enterprise, Data Bricks, Horton Works, EMC Pivotal.

Databases: Oracle, SQL Server, PostgreSQL.

Web Technologies: HTML, XML, JQuery, Ajax, CSS, JavaScript, JSON.

Streaming Tools: Kafka, RabbitMQ

Testing: Hadoop Testing, Hive Testing, MRUnit.

Operating Systems: Linux Red Hat/Ubuntu/CentOS, Windows 10/8.1/7/XP.

Cloud: AWS EMR, Glue, RDS, CloudWatch, S3, Redshift Cluster, Kinesis, DynamoDB.

Technologies and Tools: Servlets, JSP, Spring (Boot, MVC, Batch, Security), Web Services, Hibernate, Maven, GitHub, Bamboo.

Application Servers: Tomcat, JBoss.

IDE’s: Eclipse, Net Beans, IntelliJ.

PROFESSIONAL EXPERIENCE

AWS Big Data Engineer

Confidential, Austin, Texas

RESPONSIBILITIES:

• Designed and developed scalable and cost-TEMPeffective architecture in AWS Big Data services for data life cycle of collection, ingestion, storage, processing, and visualization.
• Involved in creating End-to-End data pipeline within distributed environment using teh Big data tools, Spark framework and Tableau for data visualization.
• Ensure dat application continues to function normally through software maintenance and testing in production environment.
• Leverage Spark features such as In-Memory processing, Distributed Cache, Broadcast, Accumulators, Map side Joins to implement data preprocessing pipelines with minimal latency.
• Implemented real-time solutions for Money Movement and transactional data using Kafka, Spark Streaming, Hbase.
• Teh project also includes a spread of big data tools and programming languages like Sqoop, Scala, Python, Oozie etc.
• Worked on scheduling Oozie workflow engine to run multiple jobs.
• Experience in creating python topology script to generate cloud formation template for creating teh EMR cluster in AWS.
• Good knowledge on AWS Services like EC2, EMR, S3, Service Catalog, and Cloud Watch.
• Experience in using Spark SQL to handle structureddatafrom Hive in AWS EMR Platform (M4.Xlarge, M5.12Xlarge clusters).
• Exploring with Spark, improving performance and optimization of teh existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
• Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, TEMPEffective & efficient Joins, Transformations and other during ingestion process itself.
• Experienced in optimizing Hive queries, joins to handle different data sets.
• Involved in creating Hive tables (Managed tables and External tables), loading and analyzing data using hive queries.
• Actively involved in code review and bug fixing for improving teh performance.
• Good experience in handling data manipulation using python Scripts.
• Involved in development, building, testing, and deploy to Hadoop cluster in distributed mode.
• Created Splunk dashboard to capture teh logs for end to end process of data ingestion.
• Written unit test cases for Spark code for CICD process.
• Good knowledge about teh configuration management tools like BitBucket/Github and Bamboo(CICD).

Environment: AWS EMR, Attunity, Kinesis, DynamoDB, SNS, SQS, CloudWatch, HDFS, Sqoop, LINUX, Oozie, Hive, Spark, Kafka, Spark Streaming, Scala, Python, Tableau, MongoDB, Amazon Web Services, Talend.

Big Data Engineer

Confidential, Ohio

Responsibilities:

• Designed, Developed and Deployed Batch and streaming pipelines using AWS Services.
• Developed data pipelines using cloud and container services like Docker and Kubernetes.
• Developed high performance data pipelines using AWS Glue and PySpark Jobs in EMR cluster.
• Developed multiple ETL pipelines to deliver data to teh Stakeholders.
• Designed and developed monitoring solution using AWS services like AWS CloudWatch, AWS IAM, AWS Glue and AWS QuickSight.
• Used AWS services like Lambda, Glue, EMR, Ec2 and EKS for Data processing.
• Used Spark and Kafka for building batch and streaming pipelines.
• Developed Data Marts, Data Lakes and Data Warehouse using AWS services.
• Extensive experience using AWS storage and querying tools like AWS S3, AWS RDS and AWS Redshift.
• Evaluated and implemented next generation AWS Serverless Architecture.
• Developed UDF s to perform standardization on teh entire dataset.
• Worked on End - to-End Development of teh ingestion Framework using AWS Glue, IAM, CloudFormation, Atana, RESTAPI.
• Worked on teh core logic of masking and creating teh mask utility using teh SHA-2.
• Used Redshift to store teh hashed and un-hashed values for a corresponding PII attribute and to map teh user with his email id or teh oxygen id which is unique for teh user.
• Changed teh entire existing datasets into GDPR Complaint Datasets and pushed it to production.
• Worked on teh successful transformation of teh project in teh PRD among teh users so teh company is completely GDPR complaint.
• Provided Architectural guidance, validation, and Implementation.
• Wrote pipelines for automated hashing and un-hashing of datasets in case of business-critical scenarios.
• Actively responsible for onboarding new technologies by evaluating tools or components which meet with functional and cost needs.
• Created POC projects in DBT and wrote macros to Calculate Formula Fields for Salesforce Data.
• Worked on improving operational efficiency by partnering with Business Partners and Data Scientists.

Hadoop Developer

Confidential

RESPONSIBILITIES:

• Developed real time data processing applications by using Scala and Python and implemented Apache Spark Streaming from various streaming sources like Kafka and JMS.
• Experienced in writing live Real-time Processing and core jobs using Spark Streaming with Kafka as a data pipe-line system.
• Developed Shell, Perl and Python scripts to automate and provide Control flow to Pig scripts.
• Worked on Amazon AWS concepts like EMR and EC2 web services for fast and efficient processing of Big Data.
• Involved in loading data from Linux file systems, servers, java web services using Kafka producers and partitions.
• Applied Kafka custom encoders for custom input format to load data into Kafka Partitions.
• Implement POC with Hadoop. Extract data with Spark into HDFS.
• Used Spark SQL with Scala for creating data frames and performed transformations on data frames.
• Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.
• Developed code to read data stream from Kafka and send it to respective bolts through respective stream.
• Worked on Spark streaming using Apache Kafka for real time data processing.
• Developed Map Reduce jobs using Map Reduce Java API and HIVEQL.
• Developed UDF, UDAF, UDTF functions and implemented it in HIVE Queries.
• Developing Scripts and Batch Job to schedule a bundle (group of coordinators) which consists of various Hadoop Programs using Oozie.
• Experienced in optimizing Hive queries, joins to handle different data sets.
• Involved in ETL, Data Integration and Migrationby writing pig scripts.
• Integrated Hadoop with Solr and implement search algorithms.
• Experience in Storm for handling realtime processing.
• Hands on Experience working in Hortonworks distribution.
• Worked hands on No-SQL databases like MongoDB for POC purpose in storing images and URIs.
• Designed and implemented MongoDB and associated RESTful web service.
• Involved in writing test cases and implement test classes using MRUnit and mocking frameworks.
• Developed Sqoop scripts to extract teh data from MYSQL and load into HDFS..
• Experience in processing large volume of data and skills in parallel execution of process usingTalendfunctionality.
• UsedTalendtool to create workflows for processing data from multiple source systems.

Environment: MapReduce, HDFS, Sqoop, LINUX, Oozie, Hadoop, Pig, Hive, Solr, Spark Streaming, Kafka, Storm, Spark, Scala, Python, MongoDB, Hadoop Cluster, Amazon Web Services, Talend.

Hadoop Developer

Confidential

RESPONSIBILITIES:

• Experience with professional software engineering practices and best practices for teh full software development life cycle including coding standards, code reviews, source control management and build processes.
• Worked on analyzing Hadoop cluster and different big data analytic tools including Map Reduce, Hive.
• Written multiple MapReduce programs for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV & other compressed file formats.
• Worked on Teradata parallel transport (TPT) to load data from databases and files to Teradata.
• Wrote views based on user and/or reporting requirements.
• Wrote Teradata Macros and used various Teradata analytic functions.
• Involved in migration projects to migrate data from data warehouses on Oracle/DB2 and migrated those to Teradata.
• Configured Flume source, sink and memory channel to handle streaming data from server logs and JMS sources.
• Experience in working with Flume to load teh log data from multiple sources directly into HDFS.
• Worked in teh BI team in Big Data Hadoop cluster implementation and data integration in developing large-scale system software.
• Involved in source system analysis, data analysis, data modeling to ETL (Extract, Transform and Load).
• Handling structured and unstructured data and applying ETL processes.
• Worked extensively with Sqoop for importing and exporting teh data from HDFS to Relational Database systems/mainframe and vice-versa. Loading data into HDFS.
• Involved in collecting, aggregating and moving data from servers to HDFS using Flume.
• Implemented logging framework - ELK stack (Elastic Search, LogStash& Kibana) onAWS.
• Developed teh Pig UDF’S to pre-process teh data for analysis.
• Coding complex Oracle stored procedures, functions, packages, and cursors for teh client specific applications.
• Experienced in using Java Rest API to perform CURD operations on HBase data.
• Applied Hive queries to perform data analysis on HBase using Storage Handler to meet teh business requirements
• Writing Hive Queries to Aggregate Data dat needs to be pushed to teh HBase Tables.
• Create/Modify shell scripts for scheduling various data cleansing scripts and ETL loading process.
• Supports and assist QA Engineers in understanding, testing and troubleshooting.

Environment: Hadoop, Hive, Linux, Map Reduce, Sqoop, Storm, HBase, Flume, Eclipse, Maven, Junit, agile methodologies.

Java Developer

Confidential

RESPONSIBILITIES:

• Review teh system requirements and attending requirements meetings with analysts and users.
• Involved in teh life cycle of teh project from documentation to unit testing making development as priority.
• Developed web pages using Struts framework, JSP, XML, JavaScript, HTML/ DHTML and CSS, configure struts application, use tag library.
• Used Apache Struts framework includes teh integrated AJAX.
• Played major role in designing & developing JSP pages and XML reports.
• Developed Servlets and custom tags for JSP pages.
• Developed few module Web pages using Springs IOC and Hibernate.
• Designed and developed dynamic pages using HTML, CSS- layout techniques, Java script.
• Took teh various challenges in teh enhancement and completed them on time.
• Extensive Used Exception handling and Multi-threading for teh optimum performance of teh application.
• Involved in design and implemented (SOA, SOAP) next generation system on distributed platform.
• Extensively used XSL as a XML parsing mechanism for showing Dynamic Web Pages in HTML format.
• Implemented SOAP protocol to get teh requests from teh outside System.
• Used CVS as a source control for code changes.
• Used ANT scripts to build teh project and JUnit to develop unit test cases.
• Developed coding using SQL, PL/SQL, Queries, Joins, Views, Procedures/Functions, Triggers and Packages.
• Provided development support for System Testing, Product Testing, User Acceptance Testing, Data Conversion Testing, Load Testing, and Production.

Environment: Java 1.5, J2EE, AJAX, Servlets, JSP, RUP, Eclipse 3.1, Struts, Spring 2.0, Hibernate, XML, CVS, Java Script, JQuery, ANT, SOAP, Log4J, DB2, Web Sphere server, UNIX, IBM Web Sphere Portal Server