- Over 9 years of professional work experience as Big Data/Hadoop Developer& Java Developerin Banking, Insurance,and HealthCaredomains for software application development projects.
- Experience with software development process models like Agile, Scrum, Waterfall methodologies. Strong experience in setting up a full software Development Life Cycle (SDLC) and ability to handle Complete SDLC.
- Expertise in working AWS, Microsoft Azure cloud environments.
- Hands on experience on Cloudera and Hortonworks distributed environments.
- Expertise in developing jobs using Spark framework modules like Spark - Core, Spark-SQL and Spark Streaming using Scala, Python.
- Experienced in migrating map reduce programs into Spark RDD transformations, to improve performance.
- Extensive and progressive experience in Data Ingestion, Transformation, Analytics using Spark framework, Hadoop eco systems componentsand third-party tools like Trifacta.
- Hands on experience in Hadoop components like HDFS, MapReduce, YARN, Hive, HQL,HBase, Pig, Sqoop, Flume, Nifi, Kafka, Spark,Oozie, Zookeeper, Elastic Search, Kibanaand in-depth knowledge on Hadoop architecture.
- Expert in working with Hive data warehouse tool-creating tables, data distribution by implementing partitioning and bucketing, writing and optimizing the HiveQL queries.
- Analyzing Data through HiveQL, Pig Latin & MapReduce programs in Java.Extending HIVE and PIG core functionalities by implementing custom UDF’s.
- Extensively involved in design, development, tuning and maintenance of HBase, CasandraNoSQL databases.
- Having hands on experiencein cluster setup.
- Good knowledge on Integrating the BI tools like Tableau, Kibana with the Hadoop stack and extracting the required Data.
- Experienced in Event Management, Incident Management and Problem Management using Ganglia,Nagios,log4j, JobHistoryServerimplementing Hadoop security using Kerberos.
- Extensive expertise in creating and Automation(Maven) of workflows using Oozie workflow Engine.
- Developing various cross platform products while working with different Hadoop file formats like Sequence File, RC File, ORC, AVRO & Parquet.
- Experience in using Maven to compile, package and deploy to the application servers.
- Experience in using IDEs like Eclipse, IntelliJ, Maven and SBT.
- Hands on experience on writing Queries, Stored procedures, Functions and Triggers by using SQL.
- Good knowledge in DBMS like Oracle, MS SQL Server, Teradata and MYSQL.
- Worked on setting up SSH, SCP, SFTP connectivity between UNIX.
- Expertise in data modelling & deployment strategies in production environment meeting Agile requirements.
Hadoop Technologies and Distributions: Apache Hadoop, Cloudera Hadoop Distribution and Horton works Data Platform (HDP)
Hadoop Ecosystem: HDFS, Map-Reduce, Yarn, Hive, Pig, Sqoop, Oozie, Flume, Zookeeper, Nifi, kafka
Spark Ecosystem: Spark core, Spark SQL, Spark Streaming
NoSql Databases: Hbase, MongoDB
Programming: C, C++, Java, PL/SQL, Scala, Python
RDBMS: ORACLE, MySQL, SQL Server
IDE: Eclipse, NetBeans
Operating Systems: Linux (RedHat, CentOS), Windows XP/7/8/10
Web Servers: Apache Tomcat
Build Tools: Apache Maven,Apache Ant
Scripting: Linux/Unix Shell Scripting
Methodologies Used: Agile(Scrum), Waterfall methodologies, SDLC.
Confidential, Rosemead, CA
- Developing distributed computing Big Data applications using Open Source frameworks like Apache Spark, NIFI and Kafka
- Worked on Spark SQL and Spark Streaming.
- Implemented Spark SQL to access hive tables into Spark for faster processing of data. Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.
- Used Spark SQL with Scala for creating data frames and performed transformations on data frames.
- Implemented Spark SQL to access hive tables into Spark for faster processing of data.
- Worked on Spark streaming using Apache Kafka for real time data processing.
- Experience in creating Kafka producer and Kafka consumer for Spark streaming.
- Used Hive to do transformations, joins, filter and some pre-aggregations after storing the data to HDFS.
- Using Azure cloud to store data in DATA LAKE STORE.
- Analysis the logs data and filter required columns by logstash configuration and send it to elastic search.
- Design the Elasticsearch configuration files based on number of hosts available, naming the cluster and node accordingly.
- Using Kibana illustrate the data with various display dashboard such as matric, graphs, pia-chart, aggregation table.
- Experienced in defining job flows managing and reviewing Hadoop log files
Environment: HADOOP, Cloudera, HDFS, Hive, Impala, Spark, Kafka, Sqoop, Elastic Search, Kibana, Java, Scala, Eclipse, UNIX, and Maven.
Confidential, Lafayette, LA
- Implemented Kafka Custom encoders for custom input format to load data into Kafka Partitions.
- Implemented Spark programming using Scala API for faster testing and processing of data.
- Worked with diligence team to explore whether NIFI was a feasible option to our solution.
- Parse JSon data through Spark core to extract schema for the production data using SparkSQL and Scala.
- Used Spark API over Hadoop YARN to perform analytics on data in Hive.
- Experienced in managing and reviewing the Hadoop log files.
- Used Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data onto HDFS.
- Involved in creating Hive tables, loading with data and writing Hive queries that will run internally in map reduce way.
- Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
- Import data from a relational database management system (SQL Server) into the Hadoop distributed file system using sqoop.
- Create scalable and high-performance Azure web services for data tracking.
- Played a key-role in setting up a 15 node Hadoop cluster on Aws.
- Used Oozie as workflow engine and Falcon for Job scheduling.
- Implemented authentication using Kerberos and authentication using Apache Sentry.
Environment: Hadoop, Cloudera, SQOOP,Hive, Hue, Pig, NIFI, Kafka, Spark, Scala,Talend, Oozie, Zookeeper, Tableau, AWS cloud, Cloudera Manager Kerberos.
Confidential, Pittsburg, PA
- Worked with the advanced analytics team to design fraud detection algorithms and then developed MapReduce programs to efficiently run the algorithm on the huge datasets.
- Run data formatting scripts in python and created terabyte csv files to be consumed by HadoopMapReduce jobs.
- Implemented various hive optimization techniques like Dynamic Partitions, Buckets, Map Joins, Parallel executions in Hive.
- Implemented Hive custom UDF’s to achieve comprehensive data analysis.
- Imported and processed terabytes of data from various structured and unstructured sources into HDFS (AWS cloud) using Sqoop and Flume.
- Involved in Cluster coordination services through Zookeeper.
- Designed and Maintained Oozie workflows to manage the flow of jobs in the cluster.
- Actively updated the upper management with daily updates on the progress of project that include the classification levels that were achieved on the data.
Environment: Hadoop, HDFS, MapReduce, Hive, Pig, Oozie, HBase, Flume,Zookeeper, AWS cloud,Cloudera, SQL, Eclipse.
Confidential, Schaumburg, IL
- Migrated the existing data to Hadoop from RDBMS (SQL Server and Oracle) using Sqoop for processing the data.
- Developed MapReduce programs to cleanse and parse data in HDFS obtained from various data sources and to perform joins on the Map side using distributed cache.
- Used Hive data warehouse tool to analyze the data in HDFS and developed Hive queries.
- Created internal and external tables with properly defined static and dynamic partitions for efficiency.
- Used Pig to develop ad-hoc queries.
- Implemented daily workflow for extraction, processing and analysis of data with Oozie.
Environment: Hadoop, MapReduce, Hive, Pig, Oozie, Sqoop.
- Involved in various stages of the project life cycle mainly design, implementation testing, deployment and enhancement of the application.
- Involved in designing the system based on UML concepts which include Data Flow Diagram, class diagram, sequence diagram, state diagram.
- Worked with custom tag libraries - Logic tags, Bean tags, HTML tags.
- Used Spring IOC- Dependency injection to build service and data access layers.
- Developed J2EE Web Service and SOAP Message components using Sun JAX-RPC and JAXB.
- Developed session beans to implement business logic and message-driven beans for processing messages from JMS.
- Used JDBC for database connectivity with Oracle.
- Devised logging mechanism using Log4J.
- Built AngularJS framework including MVC architectures, different modules, specific controllers, templates, custom directives and custom filters.
- Extensively used WebSphere Studio Application Developer for building, testing and deploying applications.
- Solid experience of web service using AJAX call for processing JSON/XML format files between front-end and back-end.
- Involved in design, development and analysis documents in sharing with Clients.
- Analysis and Design of the Object models using JAVA/J2EE Design Patterns in various tiers of the application.
- Analyzing the Client Requirements and designing the specification document based on the requirements.
- Used application server WebLogic based on the client requirements and project specifications.
- Involved in mapping of all configuration files according to the JSF Framework.
- Testing and production support of core java based multithreading ETL tool for distributed loading XML data into Oracle11g database using JPA/Hibernate.
- Involved in the development of Database Connections and Database Operations using JDBC.
- Wrote Action Form and Action classes and used various HTML tags, Bean, and Logic etc., also configured Struts-Config.xml for global forwards, error forwards & action forwards.
- Developed UI using JSP, JSON and Servlet and server-side code with Java.
- Used Java Mail (JMS) API to send Email Notifications for the users.
- Developed Maven build scripts and involved in deploying the application on WebSphere.
- Designed and developed various stored procedures, functions and triggers in PL/SQL to implement business rules.
Environment: Java/J2EE, JSP, XML, XSLT, Struts, Web Services, SQL, Web Sphere, JUnit, Log4j, MVC, Maven.
Associate Java Developer
- Involved in requirement gathering, analysis, design and development of the application.
- Model View Control (MVC) design pattern is implemented with Spring MVC.
- Developed Dynamic and static web pages using JSP, Custom Tags and HTML.
- Extensively used the Spring MVC, Spring Core for Inversion of Control (IOC)/Dependency Injection, Application Context.
- Implemented persistence layer using Hibernate (Spring + Hibernate integration).
- Exposed the WebServices to the client applications by sharing the WSDL's.
- Transaction management using some Spring frameworks.
- Wrote HQL queries, stored procedures and enhanced performance by running explain plans.
- Used Maven for automated building of projects.
- Involved in preparation of Unit test cases and System test plans for various functionalities using JUnit.
- Used IBMRAD to develop and debug application code.