Senior Big Data Engineer Resume
Plano, TX
SUMMARY:
- Versatile big data engineering architect and software engineer with strong analytical, communication, and problem solving skills with more than 6 years of professional experience automating tasks, re - engineering and architecting processes with data science related applications, building ETL pipelines, and administering critical databases. My main objective is to utilize my skills and technical knowledge in a progressive environment.
- Excellent work ethic, professionalism, attention to detail, ability to work in a highly collaborative, collegial, and multidisciplinary team environment.
- Excellent analytical and leadership skills with the ability to manage time, priorities, and requests. Excellent communication and interpersonal skills.
- Highly self-motivated, self-starter, and adaptable to learning and understanding new technologies.
TECHNICAL SKILLS:
Languages: Python, Java, SQL, NoSQL, PowerShell, PySpark
Databases: MS SQL Server, MariaDB, MongoDB, MySQL, Oracle 10g/11g
Frameworks: Spring Boot, Hibernate
Tools: Jupyter, Spring Tool Suite, GitHub, GitLab, Jira, Jenkins, Maven
Platforms: Windows, Unix/Linux, Azure, AWS, GCP
Big Data: ETL, NiFi, Python, Java, SSIS, SSAS, Power BI, Data Warehouse, Hadoop, MapReduce, Oozie, Pig, Hive, Sqoop and Floom, Kafka, HBase, Spark
Design Patterns: Factory Pattern, Singleton Pattern, Proxy Pattern, Facade Pattern, MVC
PROFESSIONAL EXPERIENCE:
Confidential, Plano, TX
Senior Big Data Engineer
Responsibilities:
- Developed big data applications using Hadoop framework and its ecosystems like MapReduce, Hive, Pig, Oozie, Spark, Sqoop, Oracle, and SQL Server RDBMS.
- Implemented full data identification, collection, exploration & cleaning for modeling, and participated in model development of ML pipelines.
- Created Continuous Integration and Continuous Development pipeline using Jenkins. Developed multiple custom scripts using python and shell to support the pipeline.
- Proposed and Implemented an automated solution to process metrics data from 50 million files and 3-petabyte data using spark, and optimized the performance of the job by adjusting the system configs and modifying the scripts.
- Worked in Agile development approach, led various development projects.
- Designed and developed data ingestion pipelines to ingest the data in and out of Hadoop using Apache NiFi.
- Experienced in GCP and its services, Dataproc,l Big Query, GCS bucket, Cloud Functions, Cloud Composer, Cloud Dataflow, gsutil tool, BQ command line utilities, Stack driver.
- Scheduled various spark jobs using gcloud and PySpark submit, get it executed in Dataproc cluster.
Technologies Used: Python, SQL Server, MySQL, Jupyter, Hadoop, HDFS, Hive, Kafka, HBase, Apache Spark, PySpark, Oozie, Sqoop, GCS bucket, GCP, Data Proc, Big Query
Confidential
Database Engineer
Responsibilities:
- Administration and engineering support of the client's RDBMS and Non-RDBMS Data and Database Farms such as MS SQL, MySQL, MongoDB, Azure SQL.
- Developed a global inventory RESTful APIs for all servers in-scope, instances, databases, and configurations using Spring Boot.
- Experienced with gathering requirements, logical and physical data modeling, and data architecture to support multiple internal initiatives, while implementing data and system integration projects.
- Data collecting, scraping, cleaning, batch processing, and building scalable data pipeline using PySpark, Spark SQL, and Hive context in partitioned Hive external tables and Big Data Frameworks such as Hadoop, Hbase, and Spark on Cloudera Hadoop platforms.
- Developed different applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation from multiple data sources and file formats such as parquet, CSV, orc, RDBMS.
- Identified and implemented process improvements, troubleshooting, system modifications, and data governance policies while collaborating with engineers, and team leaders.
Technologies Used: Python, SQL, SSIS, SSAS, MongoDB, PowerShell, Jupyter, Hadoop, HDFS, Hive, PySpark, Azure Data Factory, Azure Data Lake
Confidential
Regional IT Officer
Responsibilities:
- Led the development of comprehensive and highly interactive web applications while using the latest technologies.
- Designed and developed End Points and Business Layer using Spring Boot and DAO Layer using Hibernate / JDBC Template.
- Contributed to component development using design patterns such as Data Access Object (DAO), Singleton, and J2EE principles.
- Performed performance related activities including automation, proactive performance monitoring, system tuning, and query optimization.
Technologies Used: Java, SQL Server, MySQL
Confidential
Data Analyst and System Support
Responsibilities:
- Co-developed with team members the SQL server database system to maximize performance benefits for clients.
- Carried out performance assessment tests and tuning on the database system.
- Assisted senior level data engineers in the design of ETL processes, including SSIS packages.
- Analyzed and resolved system integration and performance issues by optimizing complex queries.
Technologies Used: SQL Server, Excel
