Big Data Design Lead Resume
ChicagO
SUMMARY
- Around 8 years of experience in analysis, design, development, implementation and testingof big data projects using different bigdata eco system tools
- Excellent in Hadoop architecture and various components such as HDFS and MapReduce programming paradigm
- Expertise in Hadoop ecosystem components like HBase, Hive and MapReduce, ELK Stack, Spark, Python, Scala,oozie, Impala, Phoenix, Kafka, Sqoop, pig
- Work experience in Amazon Web services
- Excellent in scheduling the jobs using Autosys and Oozie
- Experience in using different file formats(Avro, parquet, Json, xml)
- Worked in different version controlling tools like git, Jenkins, SVN
- Knowledge in Middleware Technology, TIBCO
- Hands on experience in installing, configuring and using Hadoop ecosystem components like HadoopMapReduce, HDFS,HBase, Hive,Sqoop, Pig, Zookeeper,kafka
- Hands on experience in Apache HadoopMap Reduceprogramming,PIG Latin, Hive QL
- Experience in managing and reviewingHadoop log files
- Excellent understanding and knowledge of NOSQL database - HBase
- Experience in importing and exporting data usingSqoopfromHDFSto Relational Database Systems
- Hands on experience inJ2EE and Core Java
- Hands on experience inVPN Putty,winSCP,Tortoise
- Experience inJava, JSP, Servlets, Hibernate, Spring, Apache tomocat,JBoss, Java Script, Ajax,Jquery, XML and HTML
- Industry experience includes projects for Media and Entertainment, Supply Chain, manufacturing, Insurance
- Experienced in coordinating, communicating with outside vendors and managing offshore resources in support of time line and IT project deliverables
- Major job responsibilities are to design and develop Physical and logical data model to data extraction from source to target
- Experience in the assessment of platform architecture frameworks, evaluate and select software and hardware technologies, process definition, reengineering and implementing of data strategy
- Strong experience with Linux/Unix system, shell scripting
- Expertise in designing, development, requirement gathering, system analysis, technical documentation and flow chart, team management, test & data management and product delivery
- Strong experience in understanding the business requirements, technical and existing business process
- Being a good team player and listener has helped building rapport with the teams
- Effective in working independently and collaboratively in teams
- Involved in all phases of the Software Development Life Cycle from analysis, design, development, testing, implementation and maintenance with timely delivery against aggressive deadlines
- Excellent oral and written communication skill
PROFESSIONAL EXPERIENCE
Big data design lead
Confidential, Chicago
Environment: HDFS, Hive, Sqoop, Hbase, Pheonix, Oozie, Pig, SVN, Python, shell script
Responsibilities:
- Understanding the business requirements, current technology challenges & issues.
- Preparing Design Documents ( Hive Mapping Documents), Data Mapping documents.
- Analyzed data from a variety of data sources to present a cohesive view of the data and completed data profiling on various source systems to understand the data data analysis and architect activities for creating conceptual, logical and physical Data model prior to development
- Design the environment to facilitate the core components such as Source data push / pull, transformations on data, loading data into Hive warehouse and archive source data for back-up and recovery
- Developed python scripts to copy SAP files from edge node to HDFS.
- Developed strategy to load data in such a way that it can be reprocessed easily in future.
- Developed python scripts to create tables on the fly with new set of data files
- Developed shell scripts to python scripts
- Scheduled python scripts using autosys
- Created habse tables and phoenix queries to analyze the website data
- Develop hql for duplicate data removal.
- Developed Hive QL and pig latin to analyze the SAP data and website data from hbase
- Exported the analysed data to the relational databases usingSqoopfor visualization and to generate reports for the BI team
- Development of python scripts for the sqooping.
- Scheduled Hadoop jobs using ooziebudles and coordinators to run the jobs at predefined timing
- Developed hive queries for finding the projections for next 60 years, based on the data from oracle and SAP
- Key member for support activities of the projects in Production and troubleshoot the production issues instantly migrating the Projects into new Hadoop platform and experience in Hadoop cluster migration activities
- Coordinating, managing and guiding offshore team during the development
Big data design lead
Confidential, Chicago
Environment: Spark, Hbase, Kafka, scala, python
Responsibilities:
- Designing the application
- Develop the code to stream the data received through kafka using spark streaming
- Develop the code to analyse data and create rowkey based on the input data to load into hbase
- Develop the python scripts to run the spark streaming
Big data design lead
Confidential, Chicago
Environment: HDFS, python, sqoop, Hive, AtScale, Impala
Responsibilities:
- Designing the application
- Performed Import and Export of data into HDFS using Sqoop and managed data coming from different sources
- Developed python scripts to load SAP files into hive external tables
- Loaded and transformed large sets of Structured data
- Involved in loading data from UNIX file system to HDFS
- Involved in creating Hive tables, data loading and writing hive queries which run internally in map reduce programs in a parallel way
- Managed and created Hive Tables on partitions
- Developed python scripts to run HQLs
- Developed python scripts to sqoop data from oracle
- Scheduled python scripts using autosys
- Developed python scripts for impala invalidation
- Stored final result in external hive tables in parquet format for ATSCALE users
Hadoop developer
Confidential
Environment: HDFS, Oozie, Hive, Shell script, python
Responsibilities:
- Involved in incremental updating of latest data for current cycles from warehouse system to HDFS file system.
- Involving in data analysis and architect activities for creating conceptual, logical and physical Data model prior to development.
- Design the environment to facilitate the core components such as Source data push / pull, Transformations on data, loading data into Hive warehouse and archive source data for back-up and recovery.
- Analyzing the requirements and the system to come up with the design and thus discussing the same with the team members and manager
- Development of hive queries which analyses the data in different perspectives such as, delivery, shipment, order processing etc.
- Developed the hive queries for each interfaces based on the mapping provided by the customer.
- DevelopedOozie workflows for scheduling the outbound file creation jobs
- DevelopedCronShell jobs for scheduling the shell scripts which can’t not be scheduled through oozie.
- Development of code to create file after hive action.
- Involved in migration of code to QA and PROD environment and bug fixing after user testing
Hadoop developer
Confidential
Environment: HDFS, Oozie, hive, shell script, Mapreduce
Responsibilities:
- Identified data quality issues by analyzing data in various systems and working with business users in order to implement data cleansing processes.
- Managed program/project roadmap and key milestone regarding criticality, downstream impact if dates are missed and determine alternative/mitigating actions.
- Involved in developing Unix scripts for validating source file, creating transformation and load jobs Develop cross-functional project plans and manages execution of tasks and completion of all deliverables within the business process and technical areas.
- Preparing Deployment and Testing plans for SIT, UAT and Production Migration.
- Analyzed defects raised, provided technical support, Production bug fixings and further enhancements.
- Prepare, maintain and publish ETL documentation, including source-to-target Jobs and business-driven transformation rules.
- Develop dynamic hive queries, which will create queries by itself where static hive queries are not possible to create
- Develop the hive queries according to the ad-hoc requests from clients
- Develop Hive UDF’s for the hive queries to process the data with the functions which are not doable with current Hive built in functions
- Developed the Oozie workflows to schedule whole jobs in such a way that rerunning and adhoc-requests can be handled without frequent changes
- Migrate workflows and code to QA for user acceptance testing and then migration to production and data verification in real time.
Hadoop developer
Confidential
Environment: - HDFS, Hive, java, Mapreduce, AWS
Responsibilities:
- Understanding the business requirements, current technology challenges & issues.
- Preparing Design Documents (Hive Mapping Documents).
- Involved in Dataset creation and transferring the Warehouse data into HDFS file system.
- Involving in the technical architecture and design review assignments for a quality solution implementation of new target Data Warehouse in HDFS.
- Logical and Physical dimensional modeling for Hadoop using Data schema methodologies to have data warehouse tables structure created so that Hadoop can be implemented similar way.
- Transformations on data, loading data into Hive warehouse and archive source data for back-up and recovery.
- Analyzing the requirements and the system to come up with the design and thus discussing the same with the team members and manager
- Development of hive queries to analyze the data stored in HDFS as per the requirement from customer.
- Data analyzed using hive will be stored in hdfs for showing in BI tool.
- Hive query will be analyzing the plan trends of the customers
- Wrote code to Crawl data from twitter about the data/plan of customer to aid the analyzing of data
- Developed the map-reduce code for sentimental analysis using natural language processing to analyze tweets received about the data/plans got from customer and its execution in AWS environment.
- Developed Hive queries and UDF’s to find out aggregation for different scenarios
Hadoop developer
Confidential
Environment: - HDFS, Hbase, hive
Responsibilities:
- Understanding the business requirements, current technology challenges & issues.
- Preparing Design Documents (Hive Mapping Documents), Data Mapping documents.
- Involving in the technical architecture and design review assignments for a quality solution implementation of new target Data Warehouse in HDFS.
- Discussing the project requirement, design and future assignment with the client account team members.
- Analyzing the requirements and the system to come up with the design and thus discussing the same with the account team member.
- Involving in data analysis and architect activities for creating conceptual, logical and physical Data model prior to development.
- Developed the code for data cleansing before loading into HBase based how data will be loaded into HBase for easy retrieval.
- Developed MapReduce code to combine and pull meaningful, needful data from the source
- Developed MapReduce code to do the analysis and wrote the result into hdfs for HBase consumption.
- Designed row-key in such a way that the processed data can be accessed faster and stored efficiently
- Developed the Graphical User Interface to display the engine condition.
