- 5+ years of experience in analysis, design, coding, testing and support in Information Technology industry & & related technologies like Hadoop HDFS, Map Reduce, Apache Pig, Hive, Spark, Sqoop, Hbase, Oozie, NiFi.
- Experienced in analyzing business requirements and translating requirements into functional and technical design specifications.
- Excellent Working experience on designing real time data ingestion using NiFi, Kylo and exporting data to Hive.
- Working knowledge on Spark framework.
- Hands on experience in installing, configuring, monitoring, and using Hadoop ecosystem components like Hadoop Map - Reduce, HDFS, HBase, Hive, Sqoop, Pig, Zookeeper, Horton works, Flume.
- Excellent knowledge of Hadoop architecture and on Hadoop daemons such as Name Node, Data Node.
- Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems(RDBMS) and vice-versa.
- Knowledge of job workflow scheduling and monitoring tools like Oozie and Zookeeper.
- Technical and analytical skills with clear understanding of ETL design and project architecture based on reporting requirements.
- Ability to adapt to evolving technology, strong sense of responsibility and accomplishment.
- Involved in installing and configuring various Hadoop components such as Pig, Hive, Sqoop, Flume, Oozie.
- Processing, cleansing, and verifying the integrity of data used for analysis
- My area of expertise has been on performing duties such as Analytics, Design, Data warehouse Modeling, Development, Implementation, Maintenance, Migration and Production support of large-scale Enterprise Data Warehouses.
- Worked on developing ETL processes to load data from multiple data sources to HDFS using SQOOP, Pig & Oozie. Perform structural modifications using Map-Reduce, HIVE and analyze data using visualization/ reporting tools.
- Ability to work in high-powered, fast-moving, multi-disciplinary working teams and a committed team player with willingness to travel and ability to quickly adapt to new environments and technologies.
Big data versions: Cloudera, Hortonworks.
Big data Tools: Hadoop, Hive, Sqoop, Oozie, HBase, Pig Spark, NiFi and Kylo.
BI Reporting Tool: PowerBI
Data Base: Oracle 11g
Methodologies: Water Fall, Agile Scrum
- Involved in gathering requirements from client and estimating time line.
- Designing real time data ingestion pipeline using NiFi, Kylo.
- Developed complex queries using HIVE & Hue
- Monitored workload, job performance and capacity planning using Cloudera Manager.
- Created and maintained Technical documentation for launching HADOOP Clusters and for executing pig Scripts.
- Create and design visualization reports using BI tools(PowerBI)
- Developed data pipeline using Sqoop, Pig and Oozie to ingest data and patient histories into HDFS for analysis.
- Create user documentation and involve in training user whenever needed.
- Large data sets were analyzed using Pig scripts and Hive queries.
- Involved in developing Pig scripts to transform raw data into data that is useful to gain business insights.
- Have thorough knowledge on spark architecture and how RDD's work internally.
- Involved in processing data from Local files, HDFS and RDBMS sources by creating RDD and optimizing for performance.
- Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior.
- Involved in various POC’s for performance tuning and to find best possible solutions in BIG DATA.
- Designed and Implemented Partitioning, Buckets in HIVE.
- Loaded the aggregated data onto Oracle from Hadoop environment using Sqoop for reporting on the dashboard
Environment: Big data Hadoop, Hive, Pig, Spark, Oracle, Putty, Unix Shell Scripting, Agile.
- Redesigned the existing site and to create new interfaces.
- Used Dreamweaver as Html editor for designing new pages.
- Responsible for content and UI development.
- Developed HTML prototypes and ready XHTML documents with CSS style sheets.
- Worked closely with the programmers and graphic designers for project requirement and analysis.
- Produced GUI prototypes for business logic presentations.
- The new website has made it much easier for the users to access the page and check the required info.
- Created Stored Procedure, Trigger for database access and events.
- Involved in debugging the product using Eclipse and used JIRA for Bug Tracking