SUMMARY

• Over 5+ years of strong experience in Data Analyst, Data mining with large data sets of Structured and Unstructured data, Data Acquisition, Data Validation, Predictive modeling, Statistical modeling, Data modeling, Data Visualization, Web Crawling, Web Scraping. Adept in statistical programming languages like Python, SAAS, Apache Spark, Matlab.
• Experience in multiple programming languages, including Python, CSS, HTML, JavaScript, jQuery, XML, AJAX and JSON and front - end technologies like Angular.js with Django.
• Having experience in Agile Methodologies in a Python based environment with SQL database and other methods to achieve maximum data security.
• Strong expertise in the development of web-based applications using Python, Django, HTML, XML, Angular JS, CSS, REST APIs, JavaScript, JSON,and jQuery.
• Experienced in data manipulation using python for loading and extraction as well as with python libraries such as NumPy, SciPy and Pandas for data analysis and numerical computations.
• Hands on experience working Amazon Web Services (AWS) using Elastic Map Reduce (EMR), Redshift, and EC2 for data processing and
• Experience in SQL and NoSQL database such as Snowflake, Cassandra and MongoDB.
• Skilled in Python wif proven expertise in using new tools and libraries like NumPy, SciPy, matplotlib, PyTest, Pandas etc.
• Experienced in working with various Python Integrated Development Environments like PyCharm, Spyder, PyStudio, PyDev,and Sublime.
• Proficient in writing SQL Queries, Stored procedures, functions, packages, tables, views, triggers using relational databases like Oracle, DB2,andMySQL.
• Experience in data modeling and using scalable warehouse solutions like AWS Lamda, AWS Glue, AWS EC2, AWS Athena and AWS S3 Bucket.
• Responsible for Configuring Kafka Consumer and Producer metrics to visualize the Kafka System performance and monitoring.
• Implemented Docker containers to create images of the applications and dynamically provision slaves to Jenkins CI/CD pipelines.

PROFESSIONAL EXPERIENCE

Confidential

Sr. Python/Data Engineer

Responsibilities:

• Developed rest API's using python with Django framework and done the integration of various data sources including RDBMS, Shell Scripting, Spreadsheets, and Text files.
• Used MongoDB to stored data in JSON format and developed and tested many features of dashboard using Python, Bootstrap, HTML, CSS, and JavaScript.
• Analyzed SQL scripts and designed the solutions to implement using PySpark and used JSON and XML for serialization and de-serialization to load JSON and XML data into Hive tables.
• Developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations.
• Developed Spark Streaming programs to process near real time data from Kafka, and process data with both stateless and state full transformations.
• Used AWS data pipeline for Data Extraction, Transformation and Loading from homogeneous or heterogeneous data sources and built various graphs for business decision-making using Python matplot library.
• Developed Microservices by creating REST APIs and used them to access data from different suppliers and to gather network traffic data from servers.
• Wrote and executed various MYSQL database queries from python using Python-MySQL connector and MySQL dB package.
• Worked on data analysis and data mining algorithms using Teradata. Loaded the data from Teradata to HDFS using Teradata Hadoop connectors.
• Used Pandas API to put the data as time series and tabular format for east timestamp data manipulation and retrieval.
• Used Python and Pandas library, built data analysis graph for documentation and record.
• Worked in development of applications especially in LINUX environment and familiar with all its commands and worked on Jenkins continuous integration tool for deployment of project and deployed the project into Jenkins using GIT version control system.
• Used Spark Streaming APIs to perform transformations and actions on the fly for building common learner data model which gets the data from Kafka in Near real time and persist it to Cassandra.
• Transformed the data using AWS Glue dynamic frames with PySpark; cataloged the transformed the data using Crawlers and scheduled the job and crawler using workflow feature
• Developed a fully automated continuous integration system using Git, Gerrit, Jenkins, MySQL and custom tools developed in Python and Bash
• Used Jenkins pipelines to drive all micro services builds out to the Docker registry and then deployed to Kubernetes, Created Pods and managed using Kubernetes

Confidential

Python/Data Engineer

Responsibilities:

• Managed, developed, and designed a dashboard control panel for customers and Administrators usingDjango, HTML, CSS, JavaScript, Bootstrap, JQueryandAPIcalls.
• Wrote and executed various MySQL database queries from Python using Python-MySQL connector and MySQL dB package.
• Implemented user interface guidelines and standards throughout the development and maintenance of the website using the CSS, HTML, JavaScript, and jQuery.
• Executing Python alongside using various libraries such as matplotlib for charts and graphs, Python -PySide, Pickle, Pandas data frame, and MySQL dB for database.
• Automated the existing scripts for performance calculations using NumPy and SQL alchemy also Designed and maintained databases using Python and developed Python based AP using MySQL
• Scraped and retrieved web data as JSON using Scrapy, presented with Pandas library.
• Worked with Scrapy for webscraping to extract structured data from website to analyze the specific data of a website and work on it.
• Performed data cleaning and feature selection using MLlib package in PySpark and working with deep learning frameworks such as Caffe, Neon.
• Wrote scripts to integrate API's with third party applications. Python using Django based web application, PostgreSQL, and integrations with third party messaging, storage services.
• Design, development and implementation of performant ETL pipelines using python API (pySpark) of Apache Spark on AWS.
• Data modeling using scalable warehouse solutions like AWS Lamda, AWS Glue, AWS EC2 and AWS S3 Bucket and installed the application onAWS EC2instances and also configured the storage on S3 buckets and deployedEC2Instance.
• Worked on implementation of a log producer that watches for application logs, transform incremental log and sends them to a Kafka based log collection platform.
• Configured deployed and maintained multi-node Dev and Test Kafka Clusters.
• Developed Python AWS serverless lambda with concurrent and multi-threading to make the process faster and asynchronously executing the callable and Implemented CloudTrail in order to capture the events related to API calls made to AWS infrastructure.
• Used Jenkins pipelines to drive all micro-services builds out to the Docker registry and then deployed to Kubernetes, Created Pods and managed using Kubernetes.

Confidential

Python Engineer

Responsibilities:

• Created a Python/Django based web application using Python scripting for data processing, MySQL for the database and developed UI using CSS3, HTML5, JavaScript, AngularJS, jQuery, AJAX and JSON.
• Extracted the actual data of HTML5 format and predicted raw data, interpreted and stored predicted one in well-organized JSON files.
• Used Test driven approach for developing the application and Implemented the unit tests using Python Unit test framework.
• Developed entire frontend and backend modules using Python on Django Web Framework and created User Interface (UI) using JavaScript, bootstrap, Cassandra with MySQL.
• Handled end-to-end development from developing APIs in Django and frontend in React to the deployment of various features implemented in client’s system in accounting tools.
• Design and maintain databases using Python and developed Python based API (RESTful Web Service) using Angular, SQL Alchemy and Postgre SQL.
• Develop remote integration with third party platforms by using RESTful web services and Successful implementation of Pyspark and Spark Streaming applications for large scale data.
• Created Terraform scripts for EC2 instances, Elastic Load balancers and S3 buckets.
• Implemented Terraform to manage the AWS infrastructure and managed servers using configuration management tools like Chef and Ansible.
• Used Python and Pandas library, built data analysis graph for documentation and record.