SUMMARY:

• Around 9 years of experience in Software development lifecycle - Software analysis, design, development, testing, deployment and maintenance.
• Working as Big Data Developer for the last 4 years and having strong background of big data stack like Spark, Scala, Hadoop, HDFS, MapReduce, Hive, SQOOP, and PIG.
• Hands-on experience with Apache Spark and its components (Spark core and Spark SQL)
• Experienced in converting HiveQL queries into Spark transformations using Spark RDDs and Scala
• Hands on experience in in-memory data processing with Apache Spark
• Developed Spark scripts by using Scala shell commands as per the requirement
• Hands on experience in using sparkSQL for reading data from external sources and processes the data using Scala computation framework
• Utilized Apache Spark with Python to develop and execute Big Data Analytics
• Experience in designing and developing POCs in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle
• Broad understanding and experience of real-time analytics and batch processing using apache spark.
• Hands on experience in AWS (Amazon Web Services) cloud computing.
• Experience with agile development methodologies like Scrum and Test-Driven Development, Continuous Integration
• Ability to translate business requirements into system design
• Experience in importing and exporting data from HDFS to RDBMS/ non-RDBMS and vice-versa using SQOOP
• Analyzed large amounts of data sets by writing Pig scripts and Hive queries.
• Hands on experience in writing pig Latin scripts and pig commands
• Experience with front end technologies like HTML, CSS and JavaScript
• Experienced in using tools like Eclipse, NetBeans, GIT, Tortoise SVN and TOAD.

TECHNICAL SKILLS:

Big Data: Apache Spark, Scala, Map Reduce, HDFS, HBase, Hive, Pig, SQOOP, PostgreSQL

Databases: Oracle 9i/11g, My SQL, SQL Server 2000/2005

Hadoop distributions: Cloudera, Hortonworks, AWS

DWH (Reporting): OBIEE 10.1.3.2.0/11 g

DWH (ETL): Informatica Power Center 9.6.x

Languages: SQL, PL/SQL, Java

UI: HTML, CSS, JavaScript

Defect Tracking Tools: Quality Center, JIRA

Tools: SQL Tools, TOAD

Version Control: Tortoise SVN, GitHub

Operating Systems: Windows ..., Linux/Unix

PROFESSIONAL EXPERIENCE:

Senior Big Data Developer

Confidential, Cincinnati, OH

Responsibilities:

• Analysed large amounts of data sets to determine optimal way to aggregate and report on it.
• Hands on experience in Spark and Spark Streaming creating RDD's, applying operations -Transformation and Actions.
• Used HIVE to analyze the partitioned and bucketed data and compute various metrics for reporting.
• Handled importing of data from various data sources, performed transformations using Hive, Spark and loaded data into HDFS.
• Developed Spark code and Spark-SQL for faster testing and processing of data.
• Hands on experience in using sparkSQL for reading data from external sources and processes the data using Scala computation framework
• Utilized Apache Spark with Python to develop and execute Big Data Analytics.
• Snapped the cleansed data to the Analytics Cluster for reporting purpose to Business.
• Hands on experience on AWS platform with S3 & EMR.
• Experience on working with different data types like FLATFILES, ORC, AVRO and JSON.
• Automation of Business reports using Bash scripts in UNIX on Data lake by sending them to business owners.
• Provided design recommendations and thought leadership to sponsors/stakeholders that improved review processes and resolved technical problems and suggested some solution.

Environment: Apache Spark, Scala, Spark-Core, Spark-SQL, Python, Hadoop, MapReduce, HDFS, Hive, Pig, MongoDB, Sqoop, Oozie, MySQL, Java (jdk1.7), AWS.

Big Data Developer

Confidential, Urbana, Maryland

Responsibilities:

• Build patterns according to business requirements to help find violations in the market and generate alerts by using Big Data technology (Hive, Tez and Spark) on AWS
• Worked as a Scrum Master, facilitating team productivities and monitoring project progress by applying Agile Methodology Scrum and Kanban on JIRA board to ensure quality of deliverables
• Optimize the long-run pattern by writing shell-scripts and using optimization settings in Hive (e.g. successfully changed 20 hours daily pattern into 7 hours run by figuring out data skew in TB level table, which was adopted company-wise and saved around 50,000 USD per year)
• Migrate on-prem RDBMS (Oracle, Greenplum) code into HiveQL and Spark SQL running on AWS EMR
• Participate in Machine Learning project, including decision tree modeling and feature engineering
• Responsible for ETL and data warehouse process to transfer and register data into AWS S3
• Develop Hive UDF functions with Java and modify framework code with Python

Environment: Apache Spark, Scala, Spark-Core, Spark-Streaming, Spark-SQL, Hadoop, MapReduce, HDFS, Hive, Pig, MongoDB, Sqoop, Oozie, MySQL, Java (jdk1.7), AWS

Big Data Consultant

Confidential, Austin, TX

Responsibilities:

• Gather business requirements for the project by coordinating with Business users and data warehousing (front-end) team members.
• Involved in products data injection into HDFS using Spark
• Created partitioned tables and bucketed data in Hive to improve the performance
• Use Amazon Web Services (AWS), EC2 for computing and S3 as storage mechanism.
• Load data into MongoDB using hive-mongo connection jars for the purpose of reports generation.
• Developed Spark scripts by using Scala shell commands as per the requirement.
• Loaded the data into Spark RDD and do in memory data Computation to generate the output response.
• Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD's.
• Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.
• Worked on migrating Map Reduce programs into Spark transformations using Spark and Scala.
• Handled importing of data from various data sources from Oracle into HDFS vice-versa using Sqoop.
• Involved in converting Map Reduce programs into Spark transformations using Spark RDD's on Scala.
• Migrating various Hive UDF's and queries into Spark SQL for faster requests.
• Involved in creating Hive tables, loading with data and writing hive queries which run internally in MapReduce.

Environment: Apache Spark, Scala, Spark-Core, Spark-Streaming, Spark-SQL, Hadoop, MapReduce, HDFS, Hive, Pig, MongoDB, Sqoop, Oozie, MySQL, Java (jdk1.7), AWS

Big Data Consultant

Confidential

Responsibilities:

• Involved in design and development phases of Software Development Life Cycle (SDLC)
• Developed all parts of modules Artifacts, Reporting, Costing and Admin Modules.
• Designing and developing of the User Interface for the application using JSP and DOJO (for validation and AJAX requests).
• Developed the runtime for the business reports using BIRT runtime tool.
• Followed agile methodology and SCRUM meetings to track, optimize and tailored features to customer needs.
• Developed the middle layer development and using spring framework
• Prepared the document for unit test scripts and testing results.
• Worked on defect resolutions raised at QA and Production level issues.
• Responds promptly and sort the issues raised by both SIT and UAT teams.
• Created users and groups and assign access privileges to users and groups.
• Worked on performance tuning of the complex SQL queries to achieve faster data loads to the data warehouse.

Environment: Environment: Core Java, Spring, JSP, CSS, JDBC, Oracle, SQL Server 2005, Web sphere, Eclipse, XML.

Big Data Consultant

Confidential

Responsibilities:

• Designed, developed, and maintain an internal interface application allowing one application to share data with another.
• Analyzed 90% of all changes and modifications to the interface application.
• Coordinated development work efforts that spanned multiple applications and developers.
• Developed and maintain data models for internal and external interfaces.
• Worked with other Bureaus in the Department of State to implement data sharing interfaces.
• Attended Configuration Management Process Working Group and Configuration Control Board meetings.
• Performed DDL (CREATE, ALTER, DROP, TRUNCATE and RENAME), DML (INSERT, UPDATE, DELETE and SELECT) and DCL (GRANT and REVOKE) operations where permitted.
• Design and develop database applications.
• Design the database structure for an application.
• Estimate storage requirements for an application.
• Specify modifications of the database structures for an application.
• Keep the database administrator informed of required changes.
• Tune the application during development.
• Establish an application's security requirements during development.
• Created Functions, Procedures and Packages as part of the development.
• Assisted the Configuration Management group to design new procedures and processes.
• Lead the Interfaces Team with responsibility to maintain and support both internal and external interfaces.
• Responsible for following all processes and procedures in place for the entire Software Development Life Cycle.
• Wrote documents in support of the SDLC phases. Documents include requirements and analysis reports, design documents, and technical documentation.
• Created MS Project schedules for large work efforts.

Environment: Oracle 9i, Informatica 7.1.x, Control-M, TOAD, Linux/Unix