We provide IT Staff Augmentation Services!

Lead / Sr. Hadoop Developer Resume

San Jose, CA

SUMMARY

  • 12+ years of experience in software engineering profession in all phases of SDLC including application design, development, production support & maintenance projects.
  • Diversified experience in Energy and utilities, Automobile, Retail and finance business domains.
  • Having 3+ years hands - on expertise wif Big data technologies like Sqoop, Hive, Pig, UNIX, Python, Spark, & Scala.
  • Expertise in Designing Hadoop Applications and recommending teh right solutions and technologies for teh applications.
  • Expertise in designing and building data pipelines of data ingestion, transformation, conversion, and Validation for Tera-bytes of data.
  • Experience in migrating teh data using Sqoop from HDFS to Relational Database System and vice-versa according to client's requirement.
  • Expertise in RDBMS, database Normalization and DE normalization concepts and principals.
  • Extensive knowledge in designing and developing Tableau visualization and dashboard solutions using Tableau.
  • Proficient in developing Entity-Relationship diagrams, Star/Snow Flake Schema Designs, and Expert in modelling Transactional Databases and Data Warehouse.
  • Having sound knowledge in Data warehousing Concepts and Tools like Teradata (Fast Load, Fast Export and Multi Load Utilities) and BASE-SAS 9.1.3.
  • Good hands-on knowledge on Partitions, Bucketing tables in Hive and designed both Managed and External table. Optimizing Hive Queries for better run time.
  • Worked on Teradata understanding and migrating it to Hadoop end to end.
  • Proficiency in importing different types of external files (Excel, CSV, txt, etc.) into SAS library and exporting SAS datasets as external files using Import/Export Wizard, PROC IMPORT, PROC EXPORT, and SAS DATA steps.
  • Implemented projects by using Agile Methodology.

TECHNICAL SKILLS

  • HDP/CDH
  • Oozie
  • Hive
  • Pig
  • Sqoop
  • Impala
  • Spark RDD
  • Spark Data frames
  • Spark Datasets
  • Spark SQL
  • Scala.
  • Core Java
  • LINUX/UNIX
  • Windows
  • ORACLE
  • MySQL
  • DB2 ftp sftp
  • Eclipse
  • SBT
  • Maven
  • Teradata
  • Base SAS
  • Waterfall
  • Agile-Scrum

PROFESSIONAL EXPERIENCE

Confidential, San Jose, CA

Lead / Sr. Hadoop Developer

Responsibilities:

  • Involve in Requirement Analysis, Design, and Development.
  • Reviewing future sprints and backlogs wif Product owners and build project plans based on agile releases.
  • Involve in scrum stand-up meetings for and Bi-weekly sprint cycles and discuss on teh issues, dependencies and roadblocks
  • Involve in Ingesting, enriching and processing large scale of structured and unstructured data into teh HDFS from different data sources.
  • Sourcing incremental volumes to Hadoop Data Lake using Sqoop and UNIX shell scripts.
  • Designing complex ETL work flows to process 15 million transactions (tera-bytes) each day into Hadoop system.
  • Involve in ingesting XML and JSON files to Hive tables.
  • Import data from various data sources, performed transformations using Hive, Spark and loaded data into HDFS.
  • Exploring wif theSpark inimproving teh performance and optimization of teh existing algorithms in Hadoop usingSparkContext,Spark-SQL, Data Frame, Pair RDD's, datasets.
  • Analyse SQL scripts and designed teh solution to implement and Running reports in Pig and Hive.
  • Develop, validate and maintain HiveQL queries.
  • Extensively worked wif Partitions, Bucketing tables in Hive and designed both Managed and External table.
  • Involve in optimizing HiveQL’s for better acceleration.
  • Develop Teradata Fastload, Fastexport and Multiload utilities in bringing data from Teradata to Hadoop and vice-versa.
  • Involve in testing process engine PRESTO, performance compared to mr and Tez.
  • Implement solutions using Hadoop, Hive, Sqoop, Spark and Unix scripts.
  • Involve in creating Hive Summarized tables/views for data visualization team.
  • Involve in Notebook (Ipyhton) integration wif Hadoop and Mysql testing.
  • Involve in payment transaction analysis and generating BI regulatory reports like customer balances, transactional tpv, fund payables and receivables at daily, monthly and quarterly frequencies.

Environment: HDP, HDFS, Python, Hive, Teradata, Spark, Scala, UNIX shell scripting, Tez, mr, Presto, GIT, JIRA.

Confidential, Quincy, MA

Lead / Sr. Hadoop Developer

Responsibilities:

  • Involved in requirements gathering and prepare teh Design.
  • Exported and Imported data into HDFS and Hive using Sqoop.
  • Sourced JSON files from remote servers and create JSON output files for cross platforms.
  • Created sales aggregations per item at product group level on each coupon for analysis using Spark SCALA code.
  • Upgraded Spark programs as per teh unify API’s in Spark 2.0.0.
  • Involved in creating Hive Summarized tables/views for data visualization team.
  • Implemented solutions using Hadoop, Hive, Sqoop, Spark and Unix scripts.

Environment: HDP, HDFS, Python, Hive, Spark, Pig, Scala, UNIX shell scripting, Core Java.

Confidential, Quincy, MA

Lead / Sr. Hadoop Developer

Responsibilities:

  • Supported code/design analysis, strategy development and project planning.
  • Imported data using Sqoop into Hive from existing SQL Server.
  • Created summary table for BI teams in Hive and Impala.
  • Developed multiple MapReduce jobs in Java for data cleaning and pre-processing.
  • Exported and Imported data into HDFS and Hive using Sqoop.
  • Involved in creating Hive tables, loading wif data and writing Hive queries which will run internally in MapReduce way.
  • Worked closely wif teh business and analytics team in gathering teh system requirements.

Environment: HDFS, Core Java, MapReduce, Hive, Sqoop, Oozie, Unix shell scripting.

Confidential, Quincy, MA

Project Lead

Responsibilities:

  • Participated in teh requirements gathering & Analysis.
  • Interacted wif teh Business Analysts to get teh User specifications.
  • Prepared Program Specifications for teh offshore team.
  • Prepared estimates based on WBS techniques.
  • Coded programs in SAS and Easytrieve required for various modules in teh system.
  • Participated in teh Integration Testing, UAT & SIT.
  • Involved in tuning teh batch cycle process.
  • Involved in teh deployment and production support of teh applications.
  • Monitored health of teh project by preparing metrics on weekly basis.
  • Involved in sharing project data to teh Quality Team.
  • Involved in quarterly project reports on time.
  • Provided training to peers in some of my specialized technical skills.

Environment: SAS, Easytrieve, MYSQL, Unix scripting, Dump Master, Isync, Visual Explain, FTP/PGP/TIBCO transfers, Endevor

Confidential

System Analyst

Responsibilities:

  • Participated in teh requirements gathering, analysis & design of teh application.
  • Interacted wif teh Business Analysts to get teh User specifications.
  • Prepared Functional & Program Specifications for teh offshore team.
  • Imported raw data files into new datasets using PROC IMPORT and modified existing datasets using PROC SORT, SET,MERGE, INFORMAT, FORMAT and condition statements.
  • Performed tests on import programs, analyse results, and modify programs.
  • Provided daily progress reports and holds progress meetings.
  • Verified teh accuracy and completeness of work in order to deliver quality solutions that improve of operations.
  • Coded programs in SAS required for various modules in teh system.
  • Optimized SQL queries to enhance teh performance of teh existing modules.
  • Coordinated wif offshore team to ensure teh delivery schedules are met and customer expectations.
  • Participated in teh Integration Testing and UAT.
  • Involved in teh deployment and support of teh applications.
  • Involved in QMS Quality Audits.
  • Provided training to peers in some of my specialized technical skills.

Environment: BASE SAS, MYSQL, MS-ACCESS, BMC, Visual Explain, Test Director, File-Aid, FTP, and Panvalet.

Confidential

Software Engineer

Responsibilities:

  • Involved in coding and testing of teh application components.
  • Involved in streamlining teh batch cycles to support teh online dialogs during teh development & testing phases.
  • Involved in design & coding of teh support systems that ensure implementation of teh Order conversion process.
  • Participated in Integration Testing and UAT.

Environment: MYSQL, MS-ACCESS, BMC, Visual Explain, Test Director, File-Aid, FTP, Panvalet.

Hire Now