- 6+ years of total IT experience in the analysis, design, testing, development and Implementation of Data Warehouse/Data Mart Design, OLAP, Web and Business Intelligence applications on platforms like Windows and Unix.
- Includes 4 years of hands on experience in Big Data technologies and Hands on experience in Hadoop Framework and its ecosystem like Map Reduce Programming, Hive, Sqoop, Nifi, HBase, Impala, and Flume
- Experience in working on Horton works and ClouderaHadoop distributions.
- Experience in analyzing data using HiveQL, and custom Map Reduce programs.
- Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice - versa.
- Extending Hive core functionality by writing custom UDFs, UDTF and UDAFs.
- Collected data from different sources like web servers and social media using Flume for storing in HDFS and analyzing the data using other Hadoop technologies.
- Good knowledge of configuring and maintaining YARN Schedulers.
- Experience in using Zookeeper and Oozie operational services to coordinate clusters and scheduling workflows.
- Hands on experience working on NoSQL databases like HBase
- Experience of semi-structured data processing (XML, JSON, and CSV) in Hive/Impala.
- Good experience in Shell script and Python programming.
- Good knowledge of Java, JDBC, Collections, JSP, JSON, REST, SOAP Web services, and Eclipse.
- Having experience on UNIX commands.
- Developed and maintained web applications running on Apache Web server.
- Experience of working in Agile Software Development environment.
- Exceptional ability to learn new technologies and to deliver outputs in short deadlines.
- Exceptional ability to quickly master new concepts and capable of working in-group as well as independently with excellent communication skills.
Database: Teradata (V13/V12), Oracle (9i/10g/11g), MS SQL Server, DB2, Data Lake.
Web Scripting: Java Script, ETL, CSS
Web services: Soap and Rest
Frameworks: MAP REDUCE, SPARK
Cloud Services: AWS, EC2, S3
Ingestion Tools: Sqoop, NIFI, Flume, Kafka
Languages: C, C++, Java, Visual Basic, COBOL,UNIX Shell Scripting, Hive, Hadoop, Scala
Others: AutoSys, Control-M, PL/SQL, XML
BI/GUI Tools: Tableau, MS Project, Visio, MS office (Word, Excel, PowerPoint, Access), XML
Operating Systems: Confidential AIX, HP UNIX, Solaris, Windows and Sun OS
OTHERS: JIRA(TRACKING), GIT HUB (repository), Jenkins (BUILD), UDEPLOY(deploy)
Confidential, Charlotte, NC
- Creating HIVE entities to create a presentation layer of ingested data (loading data) onto Lake.
- Installed and configured Hadoop Map Reduce, HDFS, Developed multiple Map Reduce jobs in Java for data cleaning and preprocessing.
- Primarily involved in Data Migration process using Azure by integrating with Git Hub repository and Jenkins.
- Built code for real time data ingestion using Shell script and Scala and java.
- Involved in various phases of development analyzed and developed the system going through Agile Scrum methodology.
- Involved in development of Hadoop System and improving multi-node Hadoop Cluster performance.
- Worked on analyzing Hadoop stack and different big data tools including Pig and Hive, Hbase database and Sqoop.
- Developed data pipeline using Nifi, Sqoop and hive to extract the data from weblogs and store in HDFS.
- Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
- Worked with different data sources like Avro data files, XML files, JSON files, SQL server and Oracle to load data into Hive tables.
- Used Spark to create the structured data from large amount of unstructured data from various sources.
- Performed transformations, cleaning and filtering on imported data using Hive, Map Reduce, Impala and loaded final data into HDFS.
- Developed shell scripts to find vulnerabilities with SQL Queries by doing SQL injection.
- Experienced in designing and developing POC's in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.
- Responsible for coding Map Reduce program, Hive queries, testing and debugging the Map Reduce programs.
- Extracted Real time feed using Spark streaming and convert it to RDD and process data into Data Frame and load the data into Cassandra.
- Involved in the process of data acquisition, data pre-processing and data exploration of telecommunication project in Scala.
- Specified the cluster size, allocating Resource pool, Distribution of Hadoop by writing the specification texts in JSON File format.
- Imported weblogs & unstructured data using the Apache Flume and stores the data in Flume channel.
- Exported event weblogs to HDFS by creating a HDFS sink which directly deposits the weblogs in HDFS.
- Utilized XML and XSL Transformation for dynamic web-content and database connectivity.
- Built the automated build and deployment framework using Jenkins, Udeploy etc.
Technology: Hadoop HDFS, Ambari view, Hive, Oozie, Zookeeper, HBase, Spark, Storm, Spark SQL, Sqoop, Nifi, NoSQL, Scala, Kafka, Cassandra, Autosys
- Worked on implementation and data integration in developing large-scale system software experiencing with Hadoop ecosystem components like HBase, Sqoop, Zookeeper, Oozie, and Hive.
- Developed Hive UDF's for extended use and wrote HiveQL for sorting, joining, filtering and grouping the structure data.
- Developed ETL Applications using Hive, Spark, and Impala & Sqoop for Automation using Oozie.
- Automated the process for extraction of data from warehouses and weblogs by developing work-flows and coordinator jobs in Oozie.
- Developed workflow in Oozie to automate the tasks of loading the data into HDFS.
- Creating Hive tables, dynamic partitions, buckets for sampling, and working on them using HiveQL.
- Used Sqoop for importing the data into HBase and Hive, exporting result set from Hive to MySQL using Sqoop export tool for further processing.
- Enumerated Hive queries to do analysis of the data and to generate the end reports to be used by business users.
- Worked on scalable distributed computing systems, software architecture, data structures and algorithms using Hadoop, Apache Spark and Apache Storm.
- Ingested streaming data into Hadoop using Spark, Storm Framework and Scala.
- Implemented POCs with Spark SQL to interpret complex JSON records, Delivery experience on major Hadoop ecosystem Components such as Pig, Hive, Spark, Elastic Search &HBase and monitoring with Cloudera Manager.
- Collection framework used to transfer objects between the different layers of the application.
- Experience in transferring Streaming data, data from different data sources into HDFS and NoSQL databases using Apache Flume.
- Developed Spark jobs written in Scala to perform operations like data aggregation, data processing and data analysis.
- Involved in Spark and Spark Streaming creating RDD's, applying operations -Transformation and Actions.
- Developed Spark code by using Scala and Spark-SQL for faster processing and testing and performed complex HIVEQL queries on HIVE tables
- Used Kafka, Flume for building robust and fault tolerant data Ingestion pipeline between JMS and Spark Streaming Applications for transporting streaming web log data into HDFS.
- Used Spark for series of dependent jobs and for iterative algorithms. Developed a data pipeline using Kafka and Spark Streaming to store data into HDFS.
Technology: Hadoop HDFS, Flume, CDH, Hive, Oozie, Zookeeper, HBase, Spark, Storm, Spark SQL, NoSQL, Scala, Kafka, Cassandra
- Involved in analyzing scope of application, defining relationship within and groups of data using star schema, and snowflake schema.
- Responsible for installation and configuration of HIVE, HBase and Sqoop on the Hadoop cluster and created HIVE tables to store the processed results in a tabular format. Configured Spark Streaming to receive real time data from the Apache Kafka and store the stream data to HDFS using Scala.
- Developed the Sqoop scripts to make the interaction between HIVE and Impala.
- Processed data into HDFS by developing solutions and analyzed the data using Map Reduce, and HIVE to produce summary results from Hadoop to downstream systems.
- Written Map Reduce code to process and parsing the data from various sources and storing parsed data into HBase and HIVE using HBase-HIVE Integration.
- Involved in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analyzed them by running HIVE queries.
- Created Managed tables and External tables in HIVE and loaded data from HDFS
- Developed Spark code by using Scala and Spark-SQL for faster processing and testing and performed complex HIVEQL queries on HIVE tables.
- Scheduled several times based Oozie workflow by developing Python scripts.
- Exporting the data using Sqoop to RDBMS servers and processed that data for ETL operations.
- Designing ETL Data Pipeline flow to ingest the data from RDBMS source to Hadoop using shell script, sqoop, package and MySQL.
- Optimized the HIVE tables using optimization techniques like partitions and bucketing to provide better.
- Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java map-reduce HIVE, Pig, and Sqoop.
- Involved in Spark and Spark Streaming creating RDD's, applying operations -Transformation and Actions.
- Created partitioned tables and loaded data using both static partition and dynamic partition method.
- Developed custom Apache Spark programs in Scala to analyze and transform unstructured data
- Handled importing of data from various data sources, performed transformations using HIVE, Map Reduce, loaded data into HDFS and Extracted the data from Oracle into HDFS using Sqoop
- Using Kafka on publish-subscribe messaging as a distributed commit log, have experienced in its fast, scalable and durability.
- Implemented POC to migrate Map Reduce jobs into Spark RDD transformations using SCALA
- Scheduled map reduces jobs in production environment using Oozie scheduler.
- Managing Amazon Web Services (AWS) infrastructure with automation and orchestration tools such as Chef.
- Proficient in AWS services like VPC, EC2, S3, ELB, IAM, CloudFormation
- Experienced in creating multiple VPC’s and public, private subnets as per requirement and distributed them as groups into various availability zones of the VPC.
- Created NAT gateways and instances to allow communication from the private instances to the internet through bastion hosts.
- Involved in writing Java API for Amazon Lambda to manage some of the AWS services.
- Used security groups, network ACL’s, internet gateways and route tables to ensure a secure zone for organization in AWS public cloud.
- Created and configured elastic load balancers and auto scaling groups to distribute the traffic and to have a cost efficient, fault tolerant and highly available environment.
- Created S3 buckets in the AWS environment to store files, sometimes which are required to serve static content for a web application.
- Used AWS Beanstalk for deploying and scaling web applications and services developed with Java.
- Configured S3 buckets with various life cycle policies to archive the infrequently accessed data to storage classes based on requirement.
- Possess good knowledge in creating and launching EC2 instances using AMI’s of Linux, Ubuntu, RHEL, and Windows and wrote shell scripts to bootstrap instance.
- Used IAM for creating roles, users, groups and also implemented MFA to provide additional security to AWS account and its resources.
- Involved in Cluster maintenance, Cluster Monitoring and Troubleshooting, Manage and review data backups and log files.
- Designed and implemented map reduce jobs to support distributed processing using java, HIVE and Apache Pig.
- Analyzing Hadoop cluster and different Big Data analytic tools including Pig, HIVE, HBase and Sqoop.
- Improved the Performance by tuning of HIVE and map reduce.
- Research, evaluate and utilize modern technologies/tools/frameworks around Hadoop ecosystem.