PROFESSIONAL SUMMARY:

• Around Five years of experience in Big Data and Hadoop technologies with good knowledge of entire hadoop ecosystem
• Experience in ingestion, storage, querying, processing and analysis of Big Data with hands on experience in Apache Spark, Spark SQL, Spark Streaming.
• Experienced in transferring data from different data sources into HDFS systems using Kafka producers, consumers and Kafka brokers.
• Expert in creating and designing data ingest pipelines using technologies such as Apache Storm - Kafka.
• Good in analyzing data using HiveQL, PigLatin and custom MapReduce program in Java.
• Expertise in MapReduce programs in HIVE and PIG to validate and cleanse the data in HDFS, obtained from heterogeneous data sources, to make it suitable for analysis.
• Good in Hive and Impala queries to load and processing data in Hadoop Distributed File system (HDFS).
• Good understanding of NoSQL Data bases and hands on work experience in writing applications on NoSQL databases like Cassandra and MongoDB.
• Hands on Datalake cluster with Hortonworks Ambari on AWS using EC2 and S3.
• Strong knowledge in MongoDB concepts includes CRUD operations and aggregation framework and in document Schema design.
• Experience in storing, processing unstructured data using NoSQL databases like HBase.
• Good in developing web-services using REST, HBase Native API Client to query data from HBase.
• Knowledge of ETL methods for data extraction, transformation and loading in corporate-wide ETL Solutions and Data Warehouse tools for reporting and data analysis.
• Experienced in job workflow scheduling, monitoring tools like Oozie and Zookeeper.
• Experience with Oozie Workflow Engine in running workflow jobs with actions that run Java MapReduce and Pig jobs.
• Experience in managing and reviewing Hadoop Log files generated through YARN.
• Experience in using Apache Solr for search applications.
• Experienced in Java, Spring Boot, Apache Tomcat, Maven, Gradle, Hibernate and open source frameworks/ software's.
• Experience with all stages of the SDLC and Agile Development model right from the requirement gathering to Deployment and production support.
• Proficient on Test driven development (TDD), Agile to produce high quality deliverables.
• Hands on Agile (Scrum), Waterfall model along with automation and enterprise tools like Jenkins, Chef, JIRA, Confluence to develop projects and version control, Git.

TECHNICAL SKILLS:

• Big Data: Hadoop HDFS, MapReduce, HIVE, PIG, HBase, Zookeeper, Sqoop, Oozie, Cassandra, Spark, Scala, Storm, Flume, Kafka and Avro.
• Web Technologies: Core Java, J2EE, Servlets, JSP, JDBC, JBOSS, JSF, XML, AJAX, SOAP, WSDL.
• Methodologies: Agile, SDLC, Waterfall model, UML, Design Patterns, Scrum.
• Frameworks: MVC, Struts 2/1, Hibernate 3, Spring 3/2.5/2.
• Programming Languages: Core Java, C, C++, SQL, Python, Scala, XML, Unix Shell scripting, HTML, CSS, JavaScript.
• Databases: Oracle 11g/10g, IBM DB2, MongoDB, Microsoft SQL Server, MySQL, MS-Access.
• Web Services: Apache Tomcat, JSON-RPC, Web Logic, Web Sphere.
• NOSQL: Cassandra, MongoDB, HBase.
• Monitoring, Reporting Tools: Ganglia, Nagios, Custom Shell Scripts.

PROFESSIONAL EXPERIENCE:

Confidential, TX

Hadoop Developer

Responsibilities:

• Responsible for design and development of analytic models, applications and supporting tools, which enable the teams create algorithms/models in a big data ecosystem.
• Ensures appropriate data testing is completed and meets test plan requirements.
• Engage with business stakeholders to design and own end-to-end solutions to empower data driven decision making.
• Involved in developing code to write canonical model JSON records from numerous input sources to Kafka Queues.
• Experienced on Scala programming as a part of Spark batches and Streaming Data pipelines development.
• Loaded the data into Spark RDD and do in memory data Computation to generate the Output response.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDD’s, and Scala.
• Implemented ELK (Elastic Search, Log stash, Kibana) stack to collect and analyze the logs produced by the spark cluster.
• Collected the logs data from web servers and integrated in to HDFS using Flume.
• Involved in setting QA environment by implementing Pig and Sqoop Scripts.
• Developed Pig Latin Scripts to do operations of sorting, joining and filtering enterprise data.
• Experience in writing storm topology to accept events from Kafka producer and emit to Cassandra.
• Developed Oozie workflow for scheduling ETL process and Hive Scripts.
• Hands on experience in installing, configuring Cloudera.
• Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (like MapReduce, Pig, Hive, Sqoop) as well as system specific jobs.
• Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.
• Have good experience with NOSQL database like Cassandra.
• Analyze Cassandra database and compare it with other open-source NoSQL databases to find which one of them better suites the current requirement.
• Develop, Maintain and support Continuous Integration(CI) framework based on Jenkins.
• Designed Implemented the prototype for the functional comparison using Cassandra vs Solr, and then knowledge sharing with other team members on Cassandra.
• Configured and used FTP by the Informatica Server to access source files.
• Implemented on Hadoop stack and different big data analytic tools, migration from different databases SQL Server2008 R2, Oracle, MYSQL to Hadoop.
• Worked on Hadoop cluster and data querying tools like Hive to store and retrieve data.
• Loading the sstable files to Scylla using the Scylla sstableloader tool + Data validation.
• POC for enabling member and suspect search using Solr.
• Knowledge of ETL methods for data extraction, transformation and loading in corporate-wide ETL Solutions and Data warehouse tools for reporting and data analysis.
• Developed an autonomous continuous integration system by using Git, Gerrit, Jenkins, MySQL and custom tools developed in Python and Bash.
• Designed, developed, implemented, and maintained solutions for using Docker, Jenkins, Git, and Puppet for microservices and continuous deployment.
• Implemented the unit testing by using Python Unit test framework.
• Modified reports and Talend ETL jobs based on the feedback from QA testers and Users in development and staging environments.
• Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems/mainframe and vice-versa.
• Search and fetch the corresponding records using SOLR query.

Environment: Hortonworks, ETL, YARN, HDFS, PIG, Hive, Kafka, Talend, Cassandra, Eclipse, Java, Sqoop, Avro, Spark, Spark API, SparkSQl, Spark Streaming, Scala, Solr, Talend, Linux file system, Linux Shell Scripting, Oozie, Agile.

Confidential

Hadoop Developer

Responsibilities:

• Analyzing Hadoop cluster and different big data analytic tools including Pig, HBase and Sqoop.
• Worked with Linux systems and RDBMS database on a regular basis in order to ingest data using Sqoop.
• Collected and aggregated large amounts of web log data from different sources such as webservers, mobile and network devices using Apache and stored the data into HDFS for analysis.
• Developed Unix shell scripts to load large number of files into HDFS from Linux File System
• Developed Custom Input Formats in MapReduce jobs to handle custom file formats and to convert them into key-value pairs.
• Involved in making Hive tables, stacking information, composing hive inquiries, producing segments and basins for enhancement.
• Created the Hive external tables using Accumulo connector.
• Designed and implemented HIVE queries and functions for evaluation, filtering, loading and storing of data.
• Wrote UNIX shell scripts and automation of the ETL processes using UNIX shell scripting.
• Involved in performance tuning of the ETL process by addressing various performance issues at the extraction and transformation stages.
• Worked with BI teams in generating the reports and designing ETL workflows on Tableau
• Prepared the Technical Specification document for the ETL job development.
• Involved in loading data from UNIX file system and FTP to HDFS
• Used HIVE to do transformations, event joins and some pre-aggregations before storing the data onto HDFS.
• Developed UDF's in java for enhancing functionalities of Pig and Hive scripts.
• Experienced in working with spark eco system using Spark SQL and Scala queries on different formats like Text file, CSV file.
• Implemented Daily Cron jobs that automate parallel tasks of loading the data into HDFS and pre-processing with Pig using Oozie co-coordinator jobs.
• Worked on the Ad hoc queries, Indexing, Replication, Load balancing, Aggregation in MongoDB.
• Experience in managing MongoDB environment from availability, performance and scalability perspectives.
• Developed Spark code using Scala and Spark-SQL for faster testing and data processing.
• Wrote Flume configuration files for importing streaming log data into HBase with Flume.

Environment: Hadoop, HDFS, pig, Hive, HBase, Flume, Sqoop, Oozie, Python, Unix, Shell Scripting, SQL Talend, Tableau, Scala Spark, Spark SQL, Mongo dB.