SUMMARY

• Having 13+ years of IT experience in application development, scripting development, production support and database engineering & design.
• Having 3.5+ years of experience in Big Data Technologies.
• Experience in data science & analytics using PySpark, Python, Pandas.
• Designed and developed data ingestion, processing, transformation and analytic solutions for processing high volume data sets using PySpark, Hive & Spark SQL etc.
• Experience in data validation, data verification and identifying data mismatch.
• Experience with implementing big data solutions on AWS cloud platform.
• Experience in working with AWS - s3, ec2, EMR, lambda, Redshift, RDS, SQS.
• Experience in scripting and automation development using Python/Perl/Shell scripting.
• Experience in database design and development.
• Hands on experience in agile development.
• Experience in various SDLC phases such as requirements gathering, analysis, design, implementation and maintenance of software applications.

TECHNICAL SKILLS

Big Data Technologies: Spark, PySpark, Hive,HDFS

Languages: Python, Pandas, DBI, Perl, Shell Script, PHP, SQL, PL/SQL

Web Technologies: JavaScript, Ajax, jQuery, HTML, CSS, XML, JSON, XLT

Framework: CakePHP

RDBMS: Oracle, MySQL, RDS

NoSQL: MongoDB, Hbase

Cloud Platform: AWS - s3, ec2, EMR, lambda, Redshift, RDS, SQS

Version Control: Git, SVN, CVS

Operating Systems: Windows, UNIX, Linux

Others: Autosys, Remedy, ServiceNow, Jira

PROFESSIONAL EXPERIENCE

Confidential

Consultant - PySpark/Python Developer

Responsibilities:

• Develop PySpark framework to process parquet conversion and stored the parquet files into S3 and load into Hive database.
• Develop Python scripts to process XML to JSON conversion and store the json files into s3.
• Develop PySpark framework to process JSON data into dat files and load the files into Redshift.
• Perform data cleaning and data processing using spark RDDs, spark DataFrames and Spark SQL.
• Develop Unit test cases using Python unittest module.
• Perform data validation from source to target.
• Develop automation test scripts to test the code functionality and for regression testing.

Environment: Spark, PySpark, Python, JSON, XML, Hive, AWS - EMR, S3, lambda, Redshift, RDS, Agile

Confidential

Consultant

Responsibilities:

• Develop PySpark scripts and transformation logic to load data from AWS s3 land bucket to prepare bucket and ingest data into hive tables.
• Perform data cleaning and data processing using spark RDDs, spark DataFrames and Spark SQL.
• Develop utility script to read and write data from AWS s3 locations using Boto3.
• Develop PySpark scripts to convert csv data into parquet form and parquet data into csv.
• Develop hedge accounting optimizer scripts using PySpark, python, pandas, NumPy, SciPy etc.
• Develop python script to generate loan pool data and loan pool confirmation result using python, pandas.
• Develop daily and monthly summary report for hedge accounting.
• Develop Unit test cases using Python unittest module.
• Perform data analysis and validation of source data to generate final data for swap, loan accounting, CCFA, ccfa-loan, last layer loan accounting, krd.
• Develop SQL and Hive query to generate and ingest data into Hive database.
• Develop AWS step functions and lambda functions to execute scripts on AWS.
• Requirement analysis, design and code reviews.
• Develop solution and requirement documents.

Environment: Spark, PySpark, Hive, Python, Pandas, JSON, Boto3, Hive, AWS - EMR, S3, lambda, Agile

Confidential, Reston, VA

Consultant

Responsibilities:

• As a team member, involved in development/deployment of big data applications.
• Involved in Hortonworks clusters installation in POC and dev environment.
• Developed data ingestion solution for moving data from RDBMS to Hadoop using Sqoop.
• Worked on Sqoop jobs for ingesting data from RDBMS to HDFS and created Hive external tables for querying the data.
• Implemented data transformation and analytic solutions using spark, hive, python, Scala etc.
• Developed application for Hadoop DB control to audit privileged user activities using Python, Sqoop, Oozie, Hive etc.
• Involved in scheduling Oozie Workflow Engine to run multiple jobs.
• Involved in data validation, data verification and identifying data mismatch using Hadoop technologies.
• Helped data scientists in data cleaning, data processing using spark RDDs, spark DataFrames and Spark SQL.
• Developed scripts to parse the HDFS, Hive, Hbase, Solr and KMS JSON audit data and load the data into tables.
• Developed scripts to generate traffic for HBASE, HIVE and HDFS commands.
• Identified and Validated HBASE, HIVE, KMS, Solr and HDFS commands for DB control application.
• Developed scripts to capture HBASE, HIVE, KMS, Solr commands executed by Privileged Users from Hadoop Ranger database to PU auditing reports.
• Validated the source and final output data and tested the data.
• Developed script to take HDFS full and incremental backup.
• Developed test plan documents and execute the test cases.
• Develop Solution Specification, implementation and backup plan documents.

Environment: Hortonworks 2.4.2/2.5.3 , Spark, PySpark, HDFS, Hive, HBase, Sqoop, Oozie, Solr, KMS, Python, Pandas, Perl, Shell scripting, Oracle, Ranger, Ambari

Confidential

Senior Software Engineer

Responsibilities:

• Developed, enhanced and maintained applications using Python, Perl, CGI, UNIX Shell scripting, Java, SQL, PL/SQL, Oracle, MySQL, JavaScript, Ajax, JQuery, JSON and HTML.
• Worked on application integration issues which involved analyzing data discrepancy across the system and perform data fixes.
• Developed, enhanced and maintained automation jobs using Perl/Python/Shell script.
• Worked on application performances issues and code bugs.
• Root cause analysis and provided solution for complex issues.
• Used tools like EMAN, ASM for application health monitoring.
• Designed, developed and maintained the database.
• Developed and enhanced SQL and PL/SQL code.
• Monitored application jobs in production.
• Production code deployment and testing.
• Worked on incidents, problem and change management.
• Debugged, tested and fixed the bugs in code.
• Worked with onshore teams and Business Analysts on requirement gathering.
• Designed and developed solution specification, test plan, implementation and backup plan documents.
• Analyzed business and software requirements and designed database solutions, tables, code, and procedures to improve system performance.
• Analyzed and troubleshoot database queries and adjusted indexes to optimize query performance.
• Performed data analysis to scope projects, detect bugs and provide insight into products for users.
• Reviewed project requests and estimated time, cost and feasibility of designs and projects within time and cost constraints.
• Worked on P1/P2 incidents and production support and on call support.
• Worked on Remedy integration with Remedy tool.

Environment: Python, Perl, CGI, Java, UNIX Shell scripting, Oracle, MySQL, SQL, PL/SQL, JavaScript, Ajax, JQuery, JSON, HTML, SVN

Confidential

Senior Software Engineer

Responsibilities:

• Developed, enhanced and maintained the application project.
• Designed, developed and maintained the MySQL database.
• Designed, developed, implemented, and maintained data and database systems, servers, and solutions.
• Developed and maintained cron jobs using Perl/Shell scripts.
• Designed application using object oriented concepts and collaborated with the designers to determine client needs and requirements.
• Reviewed project requests and estimated time, cost and feasibility of designs and projects within time and cost constraints.
• Analyzed business and software requirements and designed database solutions, tables, code, and procedures to improve system performance.
• Write and code logical and physical database descriptions and specify database identifiers.
• Developed batch jobs using Perl/shell scripting and SQL and stored procedures using PL/SQL to load data files, create reports, and performed other custom jobs.
• Monitored application jobs using Perl, Python and shell scripts.
• Developed test plans to test and debug solutions and modified systems, scripts and utilities as needed.
• Analyzed and troubleshoot database queries and adjusted indexes to optimize query performance.
• Performed data analysis to scope projects, detect bugs and provide insight into products for users.
• Developed standards and guidelines for the use and acquisition of software and protect vulnerable information.

Environment: Perl, CGI, PHP, MySQL, Ajax, jQuery, Linux, Apache web server

Confidential

Senior Software Engineer

Responsibilities:

• Developed, enhanced and maintained the application project using PHP, Perl, JSP, Java, and HTML.
• Fixed front-end errors.
• Performed system testing on thick clients and web clients.
• Executed test procedures and conducted code reviews.
• Reviewed project requests and estimated time, cost and feasibility of designs and projects within time and cost constraints.
• Developed best practices and architecture.
• Designed and developed application using object oriented concepts.
• Developed batch jobs using Perl/shell scripting and SQL and stored procedures using PL/SQL to load data files, create reports, and performed other custom jobs.
• Monitored application jobs using Perl and shell scripts.
• Developed test plans to test and debug solutions and modified systems, scripts and utilities as needed.
• Analyzed and troubleshoot database queries and adjusted indexes to optimize query performance.
• Performed data analysis to scope projects, detect bugs and provide insight into products for users.

Environment: Perl, CGI, Java, JSP, DBI, MySQL, Ajax, Java Script, jQuery, JSON, Linux

Confidential

Software Developer

Responsibilities:

• Analyzed and understand the software requirements.
• Designed and developed the software specification documents.
• Designed and developed the software requirements prototype.
• Developed and enhanced web based applications using Perl, CGI, PHP, Oracle, MySQL, JavaScript, Ajax, jQuery, HTML, CSS and UNIX.
• Developed and enhanced the jobs using Perl/shell scripts. Fixed the bugs in the code and tested the code.
• Designed and developed the database.
• Developed store procedures, triggers, and PL/SQL Code. Performed unit testing and integration testing of the system functionality.
• Developed test plans to test and debug solutions and modified systems, scripts and utilities as needed.
• Analyzed and troubleshoot database queries and adjusted indexes to optimize query performance.

Environment: Perl, CGI, PHP, Oracle, MySQL, SQL, PL/SQL, JavaScript, Ajax, JQuery, HTML, CSS, Apache web server, shell script, SVN and UNIX

Confidential

Web Developer

Responsibilities:

• Developed web based applications using PHP, MySQL, SQL, PL/SQL, JavaScript, Ajax and jQuery, HTML and CSS.
• Fixed the errors in the code and tested the code.
• Designed and developed the MySQL database.
• Developed store procedures, triggers, and PL/SQL Code.
• Performed unit testing and integration testing of the system functionality.
• Developed test plans to test and debug solutions and modified systems, scripts and utilities as needed.
• Analyzed and troubleshoot database queries and adjusted indexes to optimize query performance.
• Fixed the performance related issues of the application.
• Fixed front-end errors and tested the code.

Environment: PHP, MySQL, JavaScript, Ajax, JQuery, HTML, CSS, Apache web server

Confidential

Web Developer

Responsibilities:

• Developed and enhanced web based applications using PHP, MySQL, SQL, JavaScript, Ajax and jQuery, HTML, CSS and UNIX.
• Fixed the errors in the code and tested the code.
• Designed and developed the MySQL database.
• Developed store procedures, triggers, and PL/SQL Code.
• Performed unit testing and integration testing of the system functionality.
• Developed test plans to test and debug solutions and modified systems, scripts and utilities as needed.
• Analyzed and troubleshoot database queries and adjusted indexes to optimize query performance.
• Fixed the performance related issues of the application.
• Fixed front-end errors and tested the code.

Environment: PHP, JavaScript, MySQL, HTML, Ajax, Apache web server