We provide IT Staff Augmentation Services!

Data Engineer Resume

3.00/5 (Submit Your Rating)

SUMMARY

  • Professional 7+ years of experience in IT as a Web/Application Developer and coding with analytical programming using Python, Django, Flask and C++.
  • Building and productionizing predictive models on large datasets by utilizing advanced statistical modeling, machine learning, or other data mining techniques
  • Utilizing predictive models to build strategies that can address key issues in growth and operations.
  • Developed intricate algorithms based on deep - dive statistical analysis and predictive data modeling that were used to deepen relationships, strengthen longevity and personalize interactions with customers.
  • Analyzed and processed complex data sets using advanced querying, visualization and analytics tools
  • Experience object-oriented programming (OOP) concepts using Python, C++ and PHP.
  • Experienced in developing web-based applications using Python, DJANGO, GTK, QT, C++, XML, CSS3, HTML5, DHTML, JavaScript and jQuery.
  • Experience converting HiveQL/SQL queries into Spark transformations through Spark RDD and Data frames API in Scala.
  • Extensive experience with Amazon Web Services (AWS), Open stack, Docker Puppet, Chef, Ansible, AWS Cloud Formation, AWS Cloud Front and Experience in using containers like Docker and familiar with Jenkins.
  • Experience with Requests, Pysftp, Gnupg, ReportLab, NumPy, SciPy, Matplotlib, HTTPLib2, Urllib2, Pandas Python libraries during development lifecycle.
  • Python, Flask, and Angular form a great stack to build modern web applications,
  • Worked in Data Warehouse and Business Intelligence Projects along with the team of Informatica (ETL).
  • Used to analyse data using machine learning (MiLB), GraphX and Spark SQL.
  • Loaded events to any data warehouse directly from python application to SQL queries.
  • Very good experience with cloud platforms like Amazon AWS and Google App Engine.
  • Experienced in installing, configuring, modifying, testing and deploying applications with Apache and Siebel.
  • Experienced in MVW frameworks like Django, Angular.js, Java Script, backbone.js, JQuery and Node.js.
  • Experience in building strong websites confirming Web 2.0 standards using Yahoo User Interface (YUI) Framework, jQuery, HTML5, XHTML, DHTML & CSS3 to develop valid code and table-free sites.
  • Python script editor in Power Bi to create amazing visuals in seaborn.
  • Familiar with JSON based REST Web services and Amazon Web services.
  • Experienced in installing, configuring, modifying, testing and deploying applications with Apache.
  • Good knowledge of web services with protocols SOAP and REST.
  • Experienced in WAMP (Windows, Apache, MYSQL and Python/PHP) and LAMP (Linux, Apache, MySQL, Python/PHP) Architecture.
  • Experienced in Python's modules NumPy, matplotlib for generating complex graphical data, creation of histograms etc.
  • Experienced in developing web-based applications using Python, Django, PHP, XML, CSS, HTML, Java Script and JQuery.
  • Expert in developing web-based applications using PHP, XML, CSS3, HTML5, DHTML, XHTML, JavaScript and DOM scripting.
  • Experienced in installing, configuring, modifying, testing and deploying applications with Apache.
  • Experienced in developing Web Services with Python programming language.
  • Experience in writing Sub Queries, Stored Procedures, Triggers, Cursors, and Functions on SQL and PostgreSQL database.
  • Worked in agile and waterfall methodologies with high quality deliverables delivered on-time.
  • Very strong full life cycle application development experience.
  • Experience with continuous integration and automation using Jenkins
  • Installed, Configured, Managed Monitoring Tools such as Splunk, Nagios, Zabbix for Resource Monitoring/network/Monitoring/Log Trace Monitoring.

TECHNICAL SKILLS

Operating Systems: RedHat Enterprise Linux 5/6/7, CentOS, OEL, SUSE Linux 11/12, HP-UX, Solaris, AIX, Windows Server 2008/2012 R2

Databases: MySQL, MongoDB, SQL Server, Cassandra, Couch dB

Scripting/Programming Languages: Python, R

Web/App Server Tools: Apache HTTP, Nginx, JBoss, Oracle WebLogic Server, WebSphere, Tomcat

Monitoring Tool: Zabbix, Splunk, Nagios, Dynatrace, Blue stripe

Ticketing Tools / Project Tracking: Service NOW, JIRA, AOTS

Version Control Tools: GIT, SVN, BitBucket, GitLabs, IBM Rational Clearcase

Cloud Platforms: Amazon Web Services, Microsoft Azure, OpenStack

Bug Tracking tools: JIRA, Service Now, BMC Remedy

PROFESSIONAL EXPERIENCE

Confidential

Data Engineer

Responsibilities:

  • As a Data Engineer , I was responsible for designing, developing, testing, tuning and building a large-scale data processing system for data products that allow Client to improve quality, velocity and monetization of our data assets for both Operational Applications and Analytical needs and also worked along with Data Scientist.
  • Experienced in handling large datasets using Partitions, Python in Memory capabilities, Broadcasts in Python, Effective & efficient Joins, Transformations and other during ingestion process itself.
  • Evaluating statistical models to determine the validity of analyses.
  • Using machine learning to build better predictive algorithms.
  • Testing and continuously improving the accuracy of machine learning models.
  • Building data visualizations to summarize the conclusion of an advanced analysis.
  • Building Data Pipeline by sourcing and collating data from different sources and also APIs for data consumption.
  • Integrating external or new datasets into existing data pipelines.
  • Applying feature transformations for machine learning models on new data.
  • Continuously monitoring and testing the system to ensure optimized performance.
  • Developing C3 types using C3 AI Suite & Python to ingest VIMS data into C3IoT platform.
  • Developing transformations using Python. These transformations help convert JSON into a structured relational data format and apply desired logic or conditions on the same.
  • Load historical machine data from Azure blob to C3IoT platform, the transformed data based on the defined annotations load into S3, PostgreSQL, or Cassandra.
  • Snowflake - data warehouse to consume the data from C3 Platform.
  • Designed and Created C3 Type Data Models, Canonicals, and Transforms to analyze the sensor data.
  • Set up Azure Event Hub to load the LIVE data from different sources into C3 platform, which is received from CAT assets.
  • Create time series data-store to log changes on Tags in an AWS Account into RDS Aurora PostgreSQL.
  • Boto3 API to load the historical oto load, while Cloud Watch logs are used to monitor and capture a tag change.
  • Create Azure Dev Ops build and release pipelines. CI/CD.
  • Analyze data events asynchronously, as data arrive in batches, or on a schedule.
  • Visualize time-series data using Charts, maps, graphs etc.,
  • Loading the data from Amazon S3 buckets and validate/verify the data accuracy in the types created and also involved in deployment using Docker Container on AWS.
  • Leveraged AWS cloud services such as EC2, auto-scaling and VPC to build secure, highly scalable and flexible systems that handled expected and unexpected load bursts.
  • Worked with Requests, Pysftp, NumPy, SciPy, Matplotlib, Beautiful Soup and Pandas python libraries during development lifecycle
  • Designed and developed integration methodologies between client web portals and existing software infrastructure using SOAP API's and vendor specific frameworks.
  • Designed, developed, implemented, and maintained solutions for using Docker, Jenkins, Git, and Puppet for microservices and continuous deployment.
  • Used JIRA for bug tracking and issue tracking and added several options to the application to choose particular algorithm for data and address generation.

Environment: Python 3.7, Django 1.7, CSS, HTML, JavaScript, Bootstrap, Apache Web Server, AJAX, jQuery, Pandas, MySQL, Restful APIs, JavaBeans, Servlets, Agile, Scrum, Heroku, JIRA, GIT.

Confidential

Sr. Python Developer

Responsibilities:

  • The system is a full micro services architecture written in Python utilizing distributed message passing via Kafka with JSON as data exchange formats.
  • Created 6 micro services that has the capability to transfer the input files to machine readable format and pass through respective payment channels.
  • Redis has a client-server architecture and uses a request-response model . This means that you (the client) connect to a Redis server through TCP connection.
  • AngularJS as a frontend for our web application and some of the functions we are using python to do the calculation and get results back.
  • Used Web scraping with Beautiful SOUP in the API of the website .
  • Used BeautifulSoup to parse the HTML content.
  • Managed, developed, and designed a dashboard control panel for customers and Administrators using Django, HTML, CSS, JavaScript, Bootstrap, JQuery and RESTAPI calls.
  • Written Python utilities and scripts to automate tasks in AWS using boto3 and AWS SDK. Automated backups using AWS SDK (boto3) to transfer data into S3 buckets.
  • Analyzed data in Amazon Redshift with pandas.
  • Hands on experience with Hadoop/ Spark Distribution - Hive, HBASE, Oozie, Cloudera, Hortonworks.
  • Experience in implementing Spark with the integration of Hadoop Ecosystem.
  • Experience in designing and developing application in Spark using Scala.
  • Supported development of Web portals, completed Database Modelling in PostgreSQL, front end support in HTML/CSS, JQuery.
  • Debugging the application and following messages in log files, to figure out the error if existing.
  • Designed and maintained databases using Python and developed Python based API (RESTful Web Service) using Flask, SQL Alchemy, PLSQL and PostgreSQL.
  • Developed monitoring application which captures the error related data and store it in database.
  • Involved in scrubbing project, which updates the existing data with hashed values.
  • Used GitHub as a version control.
  • Optimized system performance and managed the API lifecycle.
  • Worked closely with leads in implementation exposure on service based, SOA, RESTful technologies.
  • Involved in debugging the applications monitored on JIRA using agile methodology.

Environment: Python, Drop wizard, Spring Boot, Lagom, Kafka, JSON, GitHub, LINUX, Django, Flask, Varnish, Nginx SOA.

Confidential

Sr. Python Developer

Responsibilities:

  • Implemented Design Patterns like Singleton, Factory, Template and DAO patterns.
  • Used AWS Beanstalk for deploying and scaling web applications and services developed with Java, PHP, Node.js, Python, Ruby, and Docker on familiar servers such as Apache, and IIS.
  • Developed strategy to migrate Dev/Test/Production from an enterprise VMware infrastructure to the IaaSAmazon Web Services (AWS) Cloud environment including runbook processes and procedures.
  • Leveraged AWS cloud services such as EC2, auto-scaling and VPC to build secure, highly scalable and flexible systems that handled expected and unexpected load bursts.
  • Coded various business process methods required for creation of Web Services related to customer information and provider information, using JAX-WS, SOAP, WSDL and JAX-RS, REST web services.
  • Worked with Requests, Pysftp, Gnupg, ReportLab, NumPy, SciPy, Matplotlib, HTTPLib2, Urllib2, Beautiful Soup and Pandas python libraries during development lifecycle
  • Created Data tables utilizing PyQt to display patient and policy information and add, delete, update patient records.
  • Developed high-performance distributed queueing system. Scala, Redis, Akka, closure, MQ messaging, Json Developed new functionality for interactive healthcare system.
  • Implemented user interface guidelines and standards throughout the development and maintenance of the website using the HTML, CSS, JavaScript and JQuery.
  • Used Ansible for automating cloud deployment process.
  • Taken part in entire lifecycle of the projects including Design, Development, and Deployment, Testing and Implementation and support.
  • Python/Django based web application, PostgreSQL DB, and integrations with 3rd party email, messaging, storage services.
  • Developed Restful API's using Python Flask and T-SQL data models as well as ensured code quality by writing unit tests using Pytest.
  • Developed views and templates with Python and Django's view controller and templating language to created user-friendly website interface.
  • Involved in deployment using Docker Container on AWS.
  • Developed Restful API's using Python Flask and SQLAlchemy data models as well as
  • Managed, developed, and designed a dashboard control panel for customers and Administrators using Django, Oracle DB, PostgreSQL, and VMWareAPI calls.
  • Managed datasets using Panda data frames and MySQL, queried MySQL database queries from Python using Python -MySQL connector MySQL dB package to retrieve information.
  • Designed and developed integration methodologies between client web portals and existing software infrastructure using SOAP API's and vendor specific frameworks.
  • Designed, developed, implemented, and maintained solutions for using Docker, Jenkins, Git, and Puppet for microservices and continuous deployment.
  • Utilized PyUnit, the Python Unit test framework, for all Python applications and used Django Database API's to access database objects.
  • Used JIRA for bug tracking and issue tracking and added several options to the application to choose particular algorithm for data and address generation.

Environment: Python 2.7, Django 1.7, CSS, HTML, JavaScript, Bootstrap, Apache Web Server, AJAX, jQuery, Pandas, MySQL, Restful APIs, JavaBeans, Servlets, Agile, Scrum, Heroku, JIRA, GIT.

Confidential

Python Developer

Responsibilities:

  • Participated in requirement gathering and worked closely with the architect in designing and modelling.
  • Developed entire frontend and backend modules using Python on Django Web Framework.
  • Developed tools using Python, Shell scripting, XML to automate some of the menial tasks.
  • Used PyQt for the functionality filtering of columns helping customers to effectively view their transactions and statements. Implemented navigation rules for the application and page. specific focus on Pandas, NumPy, SciPy, Flask, Database Integration(SQLAlchemy) and Automation.
  • Performed testing using Django's Test Module.
  • Used Django configuration to manage URLs and application parameters.
  • Used Python and Django to interface with the jQuery UI and manage the storage and deletion of content.
  • Developed a fully automated continuous integration system using Git, Jenkins, MySQL and custom tools developed in Python and Bash.
  • Designed and managed API system deployment using fast http server and Amazon AWS architecture
  • Creation of Python scripts for data access and analysis (Scripts, Data Feeds, XLS, FIXML) to aid in process and system monitoring, and reporting.
  • Rewrite existing Java application in Python module to deliver certain format of data.
  • Wrote Python scripts to parse XML documents and load the data in database.
  • Also used Bootstrap and Angular.js as a mechanism to manage and organize the HTML page layout.
  • Designed and developed data management system using MySQL.
  • Created a Git repository and added the project to GitHub.
  • Build development environment with JIRA.
  • Creating unit test/regression test framework for working/new code.
  • Responsible for debugging and troubleshooting the web application.

Environment: Python 2.7, Django 1.4, Java, Angular.js, Bootstrap, AWS, Bash, PyQt, XML, Shell Scripting, MySQL, HTML, XHTML, CSS, AJAX, JavaScript, Jenkins, Apache Web Server, Linux.

We'd love your feedback!