Hadoop Engineer Resume
Beaverton, OR
PROFESSIONAL SUMMARY:
- Over 8 years of programming experience in the IT field, including three years of experience in Hadoop Ecosystem and remaining 5 years in Database Development: Analysis, Design, and Implementation of Business Applications using the Oracle Relational Database Management System (RDBMS).
- Involved in all phases of the SDLC (Software Development Life Cycle) from analysis, design, development, testing, implementation and maintenance with timely delivery against aggressive deadlines.
- In depth and extensive knowledge of Bigdata Hadoop architecture and various components.
- Good Exposure on the design and development of applications for Distributed Environment using Hadoop, Java and other Bigdata technologies .
- Involved in Big Data Project Phases like Data Acquisition , Data Pre - Processing , Data Processing, Data View, BI/Analytics/Visualisation.
- Expertise with different tools in Hadoop Environment including Pig, Hive, HDFS, MapReduce, Sqoop, Spark, Kafka, Yarn, Oozie, and Zookeeper.
- Hands on experience working with NoSQL database including MongoDB and HBase.
- Experienced in writing HBase programs using Java API.
- Experienced in writing Map Reduce programs using Java, Python.
- Experienced in writing SPARK SQL programs.
- Extensively used various SPARK RDD Transformations such as map, flatMap, reduce, reduceByKey, group, keyBy, sortByKey, aggregate, aggregateByKey, fold, foldByKey, combineByKey, cogroup etc .
TECHNICAL SKILLS:
- Operating System: Linux, CentOS, Windows, Ubuntu, RedHat
- Programming Languages: Oracle SQL, Oracle PL/SQL, C, Python, Core Java, Scala
- Databases & Tools: Oracle, MySQL, SQL Server, Hbase, MongoDB, Cassandra, PLSQL Developer, TOAD, SQL Developer, SQL Plus
- Hadoop Ecosystem: HDFS, MapReduce, Hive, Pig, Sqoop, Flume, Hbase, Apache Spark, Apache Tika.
- Tools: SQL Developer, PL/SQL Developer, Toad, Informatica, Visual Safe Source (VSS), HP Quality Center (QC), Anaconda/Spyder/Ipython/Ipython Notebook, Eclipse, PyCharm.
PROFESSIONAL EXPERIENCE:
Confidential, Beaverton, OR
Hadoop Engineer
Responsibilities:
- Work closely with various Business and Technical teams to discuss and understand business requirements.
- Involved in ETL strategy and Architecture of the project.
- Recreating existing Teradata objects in Snowflake database.
- Replicating logic from Teradata utilities like Fast load, Multi load to load data into target database.
- Developing Python Scripts and automate Teradata object DDL conversion to Snowflake.
- Create Airflow DAGs and schedule jobs via both Autosys and Apache Airflow.
- Maintain Oracle Database Replicas and perform CDC (Change Data Capture) Using Oracle Triggers.
- Designed and Implemented Oracle Advanced Queues to stream data from Oracle to AWS S3 using Enqueue and Dequeue methods.
- Exported structured data to AWS cloud using SQOOP Jobs.
- Read data from S3 and analyze using Spark/Spark SQL and store results back in S3 for downstream data consumption.
- Integrated Hive with Spark and utilized Spark as computing frame work.
- Created objects in Hive such as Managed Tables, External Tables, Views, Partitioned Tables, and Hive Buckets for better query performance.
- Performed various operations on hive tables including inner join, outer join, full outer join, cross join, union all, distinct, order by, sort by, distribute by, cluster by.
Confidential, Beaverton, OR
Sr. Application Developer and Performance Analyst
Responsibilities:
- Lead performance optimization efforts to reduce long running nightly batch operations in order to run within acceptable time period.
- Introduced best practices to development team, leveraging Oracle tools such as Oracle Enterprise Manager (OEM) , TKPROF , DBMS PROFILER , Hierarchical Profiler along with IPython notebooks and many Python libraries (pandas, matplotlib, etc) for flexible, interactive data exploration and discovery.
- Worked closely with DBAs to fix performance issues caused by Oracle wait events such RC Latch events etc.
- Extensively involved in development efforts in building Supply chain planning application for end users (Planners).
- Involved directly in brainstorming with Business System Analysts to understand new requirements and enhancements to existing application.
- Implemented heavy data transformation logic inside Oracle packages.
- Extensively used Analytic functions - rank, row number, lag, lead - in data transformation.
- Worked extensively with Oracle Collections along with most important performance optimization features like bulk Collect, forall, and table function.
- Created hundreds of procedures, functions, tables, views, materialized views, triggers, sequences, indexes etc.
Confidential, Saint Petersburg, FL
Sr. Oracle Application developer
Responsibilities:
- Took a responsibility of Software Development of Serve Web Apps and Mobile Apps.
- Create new high volume database designs for new modules.
- Creating and maintaining tables, views, procedures, functions, and packages and performing DML operations using insert, update and delete statements
- Translated business requirements into technical requirements and delivered application code that is fully tested and meets the business requirements.
- Created automated scripts to perform actions like Creating new table; adding new column; adding an index etc.
- Constructed the required data involving complex JOIN statements, including outer-joins, intersect and union all.
- Tuned SQL statements using Explain plan, TKPROF for maximum efficiency and performance.
- Identified and fixed application performance issues by providing proven scientific solutions.
- Wrote SQL, PL/SQL programs to retrieve data from data repository using cursors and exceptional handling.
- Created complex SQL queries using Edition views, sub queries, correlated sub queries.
- Extensively used cursors, ref cursors and exceptions in developing packages, procedures and functions.
- Debugging the code and created error log package to record all the bad records along with error codes and error message.
Environment: Oracle 11g, SQL, PL/SQL, Windows, TOAD, SQL Developer, MS-TFS, Project Web App
Confidential, Houston, TX
Sr.PL/SQL developer
Responsibilities:
- Creating and maintaining tables, views, procedures, functions, and packages and performing DML operations using insert, update and delete statements
- Extensively Used Autonomous transaction and triggers to audit the DML operations on tables.
- Translated business requirements into technical requirements and delivered application code that is fully tested and meets the business requirements.
- Handled bulk operation using different types of Collections (Associated arrays, Nested tables, Varrays).
- Created scripts to create database objects like Tables, Indexes, Sequences etc.
- Constructed the required data involving complex JOIN statements, including outer-joins, intersect and union all.
- Worked with Collections and improved the performance of multi-row queries by using Bulk Collect and Bulk binds.
- Tuned SQL statements using Explain plan, TKPROF for maximum efficiency and performance.
- Wrote SQL, PL/SQL programs to retrieve data from data repository using cursors and exceptional handling.
- Avoided mutating table errors using the new functionality in Oracle 11g i.e., Compound triggers.
- Used Global Temporary tables to access the repeated data within a session and improved the performance.
- Coordinated with front end team working on Java and web services and provided them the required procedures and packages and necessary insight into the data.
- Created complex SQL queries using inline views, sub queries, correlated sub queries.
- Extensively used cursors, ref cursors and exceptions in developing packages, procedures and functions.
- Debugging the code and created error log package to record all the bad records along with error codes and error message.
Environment: Oracle 11g, SQL, PL/SQL, UNIX, SQL Developer 3.0.03, Beeline, SVN
Confidential, Reston,VA
Oracle PL/SQL Developer
Responsibilities:
- Physical designing and development of Database
- Worked on PL/SQL in creating complex stored procedures and functions
- Created various database objects like tables, indexes and views
- Used Collections like variable arrays, nested tables extensively
- Writing triggers, stored procedures and functions required to send the Credit Approval Details
- Used various forms of control structures including CASE, DECODE, IF-THEN-ELSE, FOR loops, WHILE loops while developing procedures
- Used composite data types like %ROWTYPE and %TYPE
- Used the advanced features of PL/SQL like Subtypes, Records, Tables, Object types and Dynamic SQL
- Wrote complex SQL queries and generated reports using Reports 6i
- Resolved several hardcore business rules and issues at the client site
Environment: Windows XP, IBM AIX, Oracle 8i, SQL, PL/SQL, SQL*Plus, TOAD, SQL*Loader, Oracle Forms & Reports 6i.
Confidential, Cranston, RI
PL/SQL developer
Responsibilities:
- Coordinated with the front end design team to provide them with the necessary stored procedures and packages and the necessary insight into the data
- Worked on SQL*Loader to load data from flat files obtained from various facilities every day
- Created and modified several UNIX shell Scripts according to the changing needs of the project and client requirements
- Wrote Unix Shell Scripts to process the files on daily basis like renaming the file, extracting date from the file, unzipping the file and remove the junk characters from the file before loading them into the base tables
- Involved in the continuous enhancements and fixing of production problems
- Generated server side PL/SQL scripts for data manipulation and validation and materialized views for remote instances
- Developed PL/SQL triggers and master tables for automatic creation of primary keys
- Created PL/SQL stored procedures, functions and packages for moving the data from staging area to data mart
- Created scripts to create new tables, views, queries for new enhancement in the application using TOAD
- Created indexes on the tables for faster retrieval of the data to enhance database performance
- Involved in data loading using PL/SQL and SQL*Loader and cron jobs calling UNIX scripts to download and manipulate files
- Performed SQL and PL/SQL tuning and Application tuning using various tools like EXPLAIN PLAN, SQL*TRACE, TKPROF, AUTOTRACE
Environment: Oracle 11g, SQL * Plus, TOAD, SQL*Loader, SQL Developer, Shell Scripts, UNIX, Windows XP
Confidential
PL/SQL developer
Responsibilities:
- Involved in the Analysis, Design, Coding and Testing of the application
- Designed Technical design document MD.070 for the business requirements
- Created and Modified PL/SQL Triggers, Procedures, Functions and packages
- Developed ER Diagrams, Data flow diagrams based on the requirement
- Developed SQL scripts to create database objects like tables, views and sequences
- Used SQL*Loader to load bulk data from various flat files and legacy systems
- Developed SQL and PL/ SQL scripts for transfer of data between databases
- Developed complex SQL queries for reports
- Developed complex triggers in reports before/after for validation of user input
Environment: Oracle 10g, SQL, PLSQL, SQL*Loader, PERL Shell script, TOAD, Informatica 8.6.0