PROFESSIONAL SUMMARY:

• 7+ years of experience in application development and design using Java /J2EE Technologies, Object Oriented Programming including 3+ years of experience with Apache Hadoop technologies, Hadoop distributed file system (HDFS), MapReduce framework, YARN, Pig, Hive, Sqoop, Spark, Spark SQL, Spark streaming, Hadoop streaming, Kafka, Cassandra and HBase.
• Extensively worked on the Cassandra data modeling and designing of the database
• Extensive experience in developing jars with SBT build tool and also Scala ide using SBT library and maven dependencies for integrating Spark with Kafka, Spark with HDFS and then store the queried data to HDFS, HBase, and Cassandra.
• As part of my assignments, I have been in Requirement Analysis, Development, Application Maintenance, Functional Studies and Unit Testing, Translated complex functional and technical requirements into detailed design, Optimized Map/Reduce Jobs to use HDFS efficiently by using various compression mechanisms, Managed and Reviewed Hadoop Log Files, configured a High Availability in cluster, migrated cluster from one datacenter to another, Implemented Commissioning and Decommissioning of new nodes to existing cluster.
• Implemented the Lambda architecture at enterprise level using technologies like Flume, Kafka. Spark Streaming and Cassandra. The architecture allowed ingestion of data both in the real time and batch setup so that the new incoming records can be used for analytical system built on the batch holders of data
• Excellent knowledge on Hadoop Ecosystem Architecture and components such as Hadoop Distributes File System (HDFS), MRv1, MRv2, Job Tracker, Task Tracker, Name Node, Data Node, Resource Manager, Node Manager and MapReduce programming.
• Experience in developing applications using Map Reduce for analyzing Big Data with different file formats.
• Good involvement in upgrading MapReduce calculations utilizing Mappers, Reducers, Combiners and Partitioners to convey best results for the extensive dataset.
• Led many Data Analysis & Integration efforts involving HADOOP along with ETL.
• Strong expertise on MapReduce programming model with XML, JASON, CSV file formats.
• Experience in importing data from a relational database management system (RDBMS) such as MySQL and Oracle into the Hadoop Distributed File System (HDFS), and exporting the processed data back into RDBMS.
• Excellent understanding and knowledge of NOSQL databases like HBase, and Cassandra.
• Extensive experience in middle - tier development using J2EE technologies like JDBC, JNDI, JSP, Servlets, JSP, JSF, Struts, Spring, Hibernate, JDBC, EJB
• Experienced with various IDEs like My Eclipse, Eclipse.
• Expertise in writing LINUX shell scripting.
• Excellent communication skills, interpersonal skills & problem-solving skills with the ability to interact with individuals at all levels.
• A great team player & ability to effectively communicate with all levels of the organization such as technical, management and customers.
• Highly motivated, committed and a natural team player with the capability to work independently, strong analytical ability coupled with excellent written and verbal communication skills.
• Team player with the ability for working hard and contributing towards of organizational goals.
• Good Interpersonal skills, willing to accept challenges and responsibilities.

TECHNICAL SKILLS:

Big Data Ecosystems: Hadoop, MapReduce, HDFS, HBase, Hive, Pig, Sqoop, Flume, Oozie, Zookeeper, Spark, Kafka, Impala.

Operating Systems: Windows, Linux, UNIX

Languages: C, JSE, JDK1.4/1.5/1.6 (JDK 5/JDK 6), Scala

Java Technologies: JDBC 3.0, Servlets 2.4, JSP 2.0

Web Technologies: HTML, JavaScript, jQuery, AJAX

Shell scripting: UNIX Shell Script.

Frame Works: Spring2.5, Hibernate3.0

Databases: MySQL 4.x/5.x, Oracle-9i, Oracle-10G,Oracle 11G

NoSQL Technologies: HBase, MongoDB, Cassandra

Servers: Tomcat, Web Logic

IDE s: Eclipse, Scala IDE

PROFESSIONAL EXPERIENCE:

Confidential, MAYFIELD, OH

Data Engineer

Responsibilities:

• Implemented the Lambda architecture using Kafka, Spark and Cassandra so as to maintain an ecosystem of data at rest and data in motion (streaming).
• Performed the jobs with Spark core, SparkSQL, Spark Streaming, and Data frames, transformations, actions.
• Configured Kafka cluster in such a way that achieve high availability and no single point of failure.
• Designed Spark schema and data selection queries and Involved in data ingestion process.
• Developed Spark streaming applications to push streaming data coming from Kafka system to Cassandra database.
• Created data frames and tables in Spark for low latency and In-memory querying.
• Achieved performance tuning for Spark Steaming applications e.g. setting right Batch Interval time, correct level of Parallelism, controlling partitions, selecting petitioner, selection of correct Serialization & memory tuning, persisting.
• Extensive experience in troubleshooting and debugging spark applications in testing environment and in production.
• Experience working with multiple main stream data serialization formats such as Parquet, Avro, JSON and Protocol Buffers.
• Pushed data to and fro from Cassandra into spark for quick Data refresh.
• Setup and benchmarked Hadoop / HBase clusters for internal use.
• Connected Spark and Hive tables for interoperability and integrated historical data present in hive tables with Cassandra tables and presented the processed data by generating Dashboards to Business users using Tableau reporting tool.
• Implemented and administered Data center Cassandra Clusters based on the knowledge of architecture and data modeling for Cassandra applications.
• Worked on adding and deleting of nodes along with configuration of Node-to-node and client-to-node encryptions by generating and distributing key store and trust store files to all nodes.
• Worked on performance tuning of cluster using Cassandra Configuration file and JVM Parameters.
• Actively worked in the data modeling and design of tables to satisfy the queries for the application.
• Performed importing data from various sources to the Cassandra cluster using Java and APIs.
• Helped team in Github repositories integration and Migrating to github.
• Extensive experience in developing jars with SBT build tool and also Scala IDE using SBT library or maven dependencies for integrating Spark with Kafka, Spark with Flume, Spark with HDFS and Hive then store the queried data to HDFS and Cassandra.

Environment: Scala, Python, Apache Spark Core, Spark Streaming, Spark SQL, Kafka, Datastax Cassandra, Hive, Mysql, Intellij IDE, SBT, Maven, Github, Agile

Confidential, CA

Spark Developer

Responsibilities:

• Monitoring jobs that used get triggered over a perl wrapper script.
• If they fail look into l the corresponding logs and based on error need to take the decision to re run it or rectify the error before the next run
• Many jobs take the new data and based on a particular unique id for each table we are supposed to deduplicate records over the old data for each id and write it back to HDFS
• Try compress the Rdd or Data Frame using a different compression codec and also try to filter out data that was not needed before the shuffle triggers and try to cut down the shuffle on a large data
• Extensive knowledge of spark core and data frames.
• Changed the codes from Rdd to Data Frames to increase the performance
• Involved in daily scrums.
• Storing the data in the csv format that enables easy load into Cassandra
• Data Modeling for the Cassandra database

Environment: Cloudera, Hadoop, Linux, Scala, HDFS, RDBMS, Spark, RDD, Data Frames, Cassandra, UNIX Shell Scripting

Confidential, PHILADELPHIA, PY

Hadoop Developer

Responsibilities:

• Involved in design and development of CDAS.
• Worked on writing transformer/mapping Map-Reduce pipelines using Java.
• Handling structured and unstructured data and applying ETL processes.
• Collected the logs data from web servers and integrated in to HDFS using Flume.
• Involved in creatingHive Tables, loading with data and writing Hive queries which will invoke and run Map Reduce jobs in the backend.
• Involved in loading data into HBase using HBase Shell, HBase Client API, Pig and Sqoop.
• Designed and implemented Incremental Imports into Hive tables.
• Worked in Loading and transforming large sets of structured, semi structured and unstructured data.
• Extensively used Pig for data cleansing.
• Involvedin collecting, aggregating and moving data from servers to HDFS using Apache Flume.
• Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database system and vice-versa. Loading data into HDFS.
• Involved in creating Hive tables, loading with data and writing hive queries that will run internally in map reduce way.
• Experienced in managing and reviewing theHadooplog files.
• Migrated ETL jobs to Pig scripts do Transformations, even joins and some pre-aggregations before storing the data onto HDFS.
• Worked with Avro Data Serialization system to work with JSON data formats.
• Worked on different file formats like Sequence files, XML files and Map files using Map Reduce Programs.
• Developed scripts and automated data management from end to end and sync up between all the clusters.
• Created and maintained Technical documentation for launching HADOOP Clusters and for executing pig Scripts.

Environment: Hadoop, Big Data, HDFS, Map Reduce, Sqoop, Oozie, Pig, Hive, HBase, Flume, LINUX, Java, Eclipse, Hadoop Distribution of Cloudera., PL/SQL, Windows, UNIX Shell Scripting.

Confidential

Java Developer

Responsibilities:

• Participated in Java and development as a part of cross program.
• Developed the project from scratch.
• Involved in various phases of Software Development Life Cycle (SDLC) as requirement gathering, modeling, analysis, architecture design and development.
• Involved in analyzing the application programs.
• Prepared use cases and designed class diagrams and object models.
• Involved in the creation of SQL tables and indexes and also wrote queries to read/manipulate data.
• Used JDBC to establish connection between the database and the application.
• Extensively used various collection classes like Array List, Hash Map, Hash Table, Hash Set.
• Implemented the queries and stored procedures using PL/SQL, Oracle.
• Implemented controllers layer using servlets and JSPs.
• Implemented view layer using JSPs, JSTL and EL and also made custom JSP tags.
• Created the user interface using HTML, CSS and JavaScript.
• Created/modified shell scripts for scheduling and automating tasks.
• Wrote unit test cases using Junit framework.
• Involved in application production support and weekend DRE activities.

Environment: Java (Jdk 1.5), Servlets, JSPs, HTML, CSS, JavaScript, JQuery, SQL, JDBC, DB2.