Python Data Engineer Resume
Fort Worth, TX
SUMMARY
- Having 7 years of experience as a Python Data Engineer with a deep understanding of technology trends with expertise in the core of complex technologies.
- Solid experience in analysis, design, and development, testing, implementation, and maintenance of various web applications using Python.
- Experience with Django and Flask high - level Python Web framework.
- Experienced in MVC frameworks like Flask and MVT framework like Django
- Developed Consumer-based custom features and applications using Python, Django, HTML, CSS.
- Proficient in SQL databases MySQL Server, MySQL (RDBMS), Postgres and MongoDB.
- Experience in writing Sub Queries, Stored Procedures, Triggers, Cursors, and Functions on SQL and PostgreSQL database.
- Working experience on JSON based REST Web services and Amazon Web Services (AWS) Such as EC2, Lambda, ELB.
- Sound experience in Object Oriented Programming using concepts like Multi-Threading, Exception Handling and Collections.
- Have experience on Kubernetes and Docker for runtime environment of system to build, test & deploy
- Microservice architecture development using Python and Docker on an Ubuntu Linux platform using HTTP/REST interfaces with deployment into a multi-node Kubernetes environment
- Experience in AWS, DevOps, Continuous Integration, Continuous Deployment and Cloud Implementations.
- Working with backend Python automation, CI pipelines, Docker, and cloud provisioning/automation.
- Worked with Python libraries from Python Package Index (Py PI).
- Experience with Jenkins and Enterprise GitHub. Worked on Jenkins continuous integration tool for deployment of project.
- Followed Agile Methodologies (Scrum) involved in daily stand-up meetings, Hands on experience with Agile project management tools such as JIRA.
- Unit testing for the entire code and fixing bugs in unit testing
- Experienced in working with different operating systems like Windows, Ubuntu Linux, and UNIX.
TECHNICAL SKILLS
Primary Languages: Python 3.6+, Core Java, JavaScript
Frameworks: Pyramid, Fast API, Sprint boot, Django
Testing Frameworks: PyTest, Unit test, Junit
Development Methodology: Agile, Waterfall, BDD (Behavior Driven Development), TDD (Test Driven Development)
Continuous Integration: Jenkins, GitHub actions, Travis, TeamCity, GitLab
Development tools: Jenkins, CloudFormation, Kubernetes, Helm, Argo cd, AWS SAM, Docker
Web Technologies: AngularJS, Angular, Vue Js
Servers: IBM Bluemix, pivotal, Nginx, ASGI servers
Build Tools: Maven, docker
Version Control Tools: GitHub, Bitbucket, GitLab, SVN
Test Management: JIRA, Bugzilla
Additional Programming Langues: Rust, go
Database/ Tools: Postgres, DynamoDB, Redis, MySQL, DBeaver, Sqlite3IDE PyCharm, Eclipse, MS Visual StudioOperating Systems macOS, Linux, Windows
PROFESSIONAL EXPERIENCE
Confidential
Python Data Engineer
Responsibilities:
- Develop a data platform from scratch and took part in requirement gathering and analysis phase of the project in documenting the business requirements
- Worked in designing tables in Hive, MYSQL using SQOOP and processing data like importing and exporting of databases to the HDFS, involved in processing large datasets of different forms including structured, semi-structured and unstructured data
- Worked with Hadoop architecture and the daemons of Hadoop including Name-Node, Data Node, job Tracker, Task Tracker, and Resource Manager.
- Used AWS data pipeline for Data Extraction, Transformation and Loading from homogeneous or heterogeneous data sources and built various graphs for business decision-making using Python matplot library
- Developed scripts to load data to hive from HDFS and involved in ingesting data into Data Warehouse using various data loading techniques.
- Created DDL scripts, database objects like Tables, Views, Indexes, Synonyms and Sequences
- Scheduled jobs using crontab, run deck and control-m.
- Build Cassandra queries for performing various CRUD operations like create, update, read and delete, also used Bootstrap as a mechanism to manage and organize the html page layout
- Developed PL/SQL Procedures, triggers, Functions and Packages and used SQL loader to load data into the database
- Performed tuning and optimizations on the spark jobs and queries for optimal performance
- Worked on Advanced SQL to embed the Stored Procedures into ETL PySpark scripts
- Created and managed external tables in Hive on S3 data
- Migrated the SSRS reports to AWS using Pandas framework
- Converted SOL Server Stored Procedures to Redshift PostgreSQL and embedded them in Python pandas framework
- Optimizing complex ETL PySpark jobs in production on EMR.
- Used partitioning and sub partitioning of tables to improve performance
- Importing and exporting data jobs, to perform operations like copying data from HDFS and to HDFS using Sqoop and developed Spark code and Spark-SQL Streaming for the faster testing and processing of data.
- Analyzed SOL scripts and designed the solutions to implement using PySpark.
- Test case/data preparation. execution and verification of the test results.
- Used JSON and XML SerDe's for serialization and de-serialization to load JSON and XML data into Hive tables.
- Used Spark SQL to load JSON data and create Schema RDD and loaded it into Hive Tables and handled structured data using Spark SQL.
- Developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations
- Added support for Amazon AWS S3 and RDS to host static/media files and the database into Amazon Cloud.
- Worked in development of applications especially in LINUX environment and familiar with all its commands and worked on Jenkins continuous integration tool for deployment of project and deployed the project into Jenkins using GIT version control system
- Managed the imported data from different data sources. Performed transformation using Hive, Pig and Map- Reduce and loaded data in HDFS.
- To achieve Continuous Delivery goal on high scalable environment. Used Docker coupled with load-balancing tool Nginx.
- Used MongoDB to stored data in JSON format and developed and tested many features of dashboard using Python, Bootstrap, CSS, and java Script.
Environment: Hadoop, Hive, Sqoop, Pig, java, Django, Flask, XML, MySQL MS SQL Server, Linux, Shell Scripting, Mongo DB, SQL, Python, Django, HTML5, PyCharm, GIT Linux Shell Scripting, RESTful, Docker, Jenkins, JIRA, jQuery, MySQL, Bootstrap, HTML5, CSS, AWS, EC2, S3.
Confidential, Fort Worth, TX
Python Developer
Responsibilities:
- Responsible for debugging and troubleshooting the web application.
- Writing REST APIs, as part of developing web-based applications for insurance premium calculations, using Flask's REST framework.
- Developed web applications using Object-Oriented Programming (OOP) concepts.
- Outputting the parsed data as JSON/BSON and stored into MongoDB
- Querying data from MongoDB and use them as input for the CSV reports
- Implemented a CI/ CD pipeline with Docker, Jenkins and GitHub by virtualizing the servers using Docker for the Dev and Test environments by achieving needs through configuring automation using Containerization
- Used data structures like directories, tuples, object-oriented class-based inheritance
- Performed Automation Tests for the app using Python Unit Test Framework.
- Efficiently performed all backend tasks from OOPS up to the REST API interface/Portal frontend.
- Wrote ANSIBLE Playbooks with Python, SSH as the Wrapper to Manage Configurations of AWS Nodes and Test Playbooks on AWVS instances using Python. Run Ansible Scripts to provision Dev servers.
- Worked on developing ETL pipelines on S3 parquet files on data lake using AWS Glue
- Used the Bash/Shell scripts to automate the deployment process.
- Used AWS container services with Kubernetes for orchestrating the servers.
- Worked on Jenkins continuous integration tool for deployment of project.
- Automated the continuous integration and deployments (CI-CD) using Jenkins
- Working closely with team members for resolving Critical and high vulnerabilities in Compliance Buddy
- Good knowledge on container management using Docker and created images for managing the application
- Wrote Python normalizations scripts to find duplicate data in different environments.
- Wrote scripts to Import and Export data to CSV, EXCEL formats from different environments using Python and made a Celery action using REST API call.
- Experienced with AWS cloud platform and its features, which includes (EC2, S3, RD5, VPC, IAM, ELk, EMR Cloud watch.
- Worked under Agile/Scrum environment and handled production rollouts and issues.
- Performed data manipulation/storage for incoming test data using lxml/etree libraries.
- Extensively used XLSX reader and writer modules to read, write and analyze data and project the results as per the client request.
- Exposure on Multi-Threading factory to distribute learning process back-testing and the various worker processes.
- Performed efficient delivery of code based on principles of Test-Driven Development (TDD) and continuous integration to keep in line with Agile Software Methodology principles.
- Lock mechanisms were implemented, and the functionality of multithreading has been used.
- Used Pandas library for statistics Analysis.
- Managed large datasets using Panda’s data frames
- Responsible for user validations on client side as well as server side.
Environment: Python 3.7, MongoDB, Informatica, Flask, REST framework, PyUnit, JSON, GitHub, Tivoli Jobs, SOAP/RESTful, Version One, Agile.
Confidential, Fort Worth, TX
Python Developer
Responsibilities:
- Created Python/Postgres back-end for data entry from Flash.
- Exposed to various phases of Software Development Life Cycle using Agile - Scrum Software development methodology.
- Good Experience in ScienceLogic and ServiceNow
- Good experience in developing web applications and implementing Model View Control (MVC) architecture using server-side applications like Django.
- Developed tools to automate some base tasks using Shell Scripting, Python.
- Used Python libraries and MySQL queries/subqueries to create several datasets which produced statistics, tables.
- Strong understanding of Layer 2 and Layer 3 networking concepts
- Good understanding of routers and networking technologies, TCP/IP, IPv4, IPv6, routing protocols, MPLS
- Designing and implementing Docker workflow reduced built and deployment times
- Used Python to write data into JSON files for testing Django Websites. Created scripts for data modelling and data import and export.
- Successfully migrated the Django database from SQLite to PostgreSQL with complete data integrity.
- Maintenance and continues development of the applications in PL/SQL procedures, shell scripting using best practices.
- Worked on CI/CD tool Jenkins to automate the build process from version control tool into testing and production environment, Managed build results in Jenkins and deployed using workflows.
- Deployed the project into Jenkins using GIT version control system.
- Worked with View Sets in Django-REST framework for providing web services and consumed web services performing CRUD operations.
- Strong grasp of operating systems concepts including filesystems, networks, and memory management
- Coordinated with vendors and schedule outages for installation of network devices such as Routers and Switches
- Generated Django forms to record data of online users and used PyTest for writing test cases.
Environment: Python 3.6, Pyramid Framework, Postgres, Django, Flask, XML, HTML, CSS, Angular JS, JavaScript, Windows, Linux.
Confidential, Knoxville, TN
Python Developer
Responsibilities:
- API development for data platforms and providing contract documents for integration with front end.
- Involved in analysis, specification, design, and implementation and testing phases of Software
- Development Life Cycle (SDLC) and used agile methodology for developing application.
- Working as an application developer experienced with controllers, views and models in Django
- Used Salt Stack to configure and manage the infrastructure
- Restful web services using Python REST API Framework.
- Implemented the application using Python Spring IOC (Inversion of Control), Django Framework and handled the security using Python Spring Security.
- Tested entire frontend and backend modules using Python on Django Web Framework
- Responsible for handling the integration of database system.
- Developed Server-side automation using Node JS scripting and connecting different types of SQL and NoSQL stores from Node JS.
- Used object-relational mapping (ORM) solution, technique of mapping data representation from MVC model to Oracle Relational data model with an SQL-based scheme.
- Implemented Performance tuning and improved the Performance of Stored Procedures and Queries.
- Installed and configured py Builder for application builds and deploying it.
- Used Selenium Library to write fully functioning test automation process that allowed the simulation of submitting different we request from multiple browsers to web application.
- Developed and Deployed SOAP based Web Services on Tomcat Server.
- Used Jenkins for continuous integration for code quality inspection and worked on building local repository mirror and source code management using Git hub.
- Used IDE tool to develop the application and JIRA for bug and issue tracking.
- Wrote unit testing codes using unit test, resolving bugs and other defects using Firebug.
- Used JIRA to assign, track, report and audit the issues.
- Used GIT to coordinate team development.
Environment: Python, Django Web Framework, HTML, CSS, NoSQL, JavaScript, JQuery, Sublime Text, Jira, GIT, pyBuilder, unit test, Firebug, Web Services.
Confidential, Columbus, OH
Python Developer
Responsibilities:
- Responsible for gathering requirements, system analysis, design, development, testing, deployment, and Implementation of various stand-alone, client-server enterprise applications.
- Gathered and analyzed the requirements and converted them into User Requirement Specifications and Functional Requirement Specifications for the designers and developers to understand them as per their perspective.
- Worked on object-oriented programming (OOP) concepts using Python, Django and Linux.
- Developed web-based applications using Python, Django, XML, CSS, HTML, JavaScript, Angular JS and JQuery.
- Experience with JSON based REST Web services and Amazon Web Services (AWS).
- Worked on Amazon services like Amazon Cloud EC2.
- Added support for Amazon AWS and RDS to host static/media files and the database into Amazon Cloud.
- Experience in writing Sub Queries, Stored Procedures, Triggers, Cursors, and Functions on MySQL and PostgreSQL database.
- Worked in agile and waterfall methodologies with high quality deliverables delivered on-time.
- Experience with continuous integration and automation using Jenkins.
- Experience with Unit testing/ Test driven Development (TDD), Load Testing.
- Developed the required XML Schema documents and implemented the framework for parsing XML documents.
- Involved in Unit testing and Integration testing.
- Worked on AJAX framework to transform Datasets and Data tables into HTTP-serializable JSON strings.
- Designed Interface using Bootstrap framework.
- Experience with working on multiple environments like development, testing, production. Excellent analytical and problem-solving skills and ability to work on own besides being valuable and contributing team player.
Environment: Python, Django 1.4, MySQL, Amazon EC2, Amazon SQS, S3, Windows, Linux HTML5/CSS3, AJAX, jQuery, JavaScript, Apache.
Confidential
Python Developer
Responsibilities:
- Designed and developed components using Python. Implemented code in Python to retrieve and manipulate data.
- Involved in Continuous Integration (CI) and Continuous Delivery (CD) process implementation using Jenkins along with Shell script.
- Used GIT repository for software configuration management and version control.
- Used JIRA for tracking and Updating Project issue.
- Performed troubleshooting, fixed, and deployed many Python bug fixes of the two main applications that were Maintained main source of data for both customers and internal customer service team.
- Coded test programs and evaluated existing engineering process
- Worked on development of customer support and complains registration system. This is Customer feedback and complains management system.
- Design, develop, test, deploy and maintain the website.
- Coding and execution of scripts in Python/Unix/VB.
- Development of Application using Java and Python.
- Recording of Scripts (Web, Web Services HTML) using Vugen and SoapUI and script validation through co correlations, parameterizations, and other methods. Scripting - web and web services.
- Resolving Complexity in the scripts of the website due to the complex logic and correlations.
- Script validation sometimes becomes challenging as it demanded many web-based logic rather than correlation and parameterization.
- Running load/endurance tests using Vugen, ALM and controller, server monitoring, analysis using Dynatrace, UNIX putty, SQL logs and other tools and reporting the performance. Analyzing errors and exceptions using putty logs (UNIX), etc.
- Testing in citrix protocol with scripts and scenario.
Environment: Python, Django Web Framework, HTML, CSS, NoSQL, JavaScript, JQuery, Sublime Text, Jira, GIT, pyBuilder, unit test, Firebug, Web Services.