Sr. Big Data Architect Resume
Bronx, NY
SUMMARY
- 10+ years of IT experience with extensive knowledge in Software Development Life Cycle (SDLC) involving Requirements Gathering, Architect, Design, Analysis, Development, Maintenance, Implementation and Testing.
- Architect, design & develop Big Data Solutions practice including set up Big Data roadmap, build supporting infrastructure and team to provide Big Data.
- Experienced in Big Data Hadoop Eco System including Map Reduce, Map reduce 2, YARN, flume, Sqoop, Hive, Apache Spark, Scala
- Having experience in Hadoop Architect /Developer (with knowledge of Hive, Sqoop, MR, Storm, Pig, HBase, Flume, Spark)
- Experienced in application development using Java, J2EE, JDBC, spring, Junit.
- Architecting and implementing Portfolio Recommendation Analytics Engine using Hadoop MR, Oozie, Spark SQL, Spark Mlib and Cassandra.
- Excellent understanding of Hadoop architecture and underlying framework including storage management.
- Expertise in Big Data Tools like Map Reduce, Hive SQL, Hive PL/SQL, Impala, Pig, Spark Core, YARN, SQOOP etc.
- Expertise in architecting Big data solutions using Data ingestion, Data Storage
- Experienced in Worked on NoSQL databases - Hbase, Cassandra & MongoDB, database performance tuning & data modeling.
- Experience in using PL/SQL to write Stored Procedures, Functions and Triggers.
- Excellent technical and analytical skills with clear understanding of design goals of ER modeling for OLTP and dimension modeling for OLAP.
- Architecting, Solutioning and Modeling DI (Data Integrity) Platforms using sqoop, flume, kafka, Spark Streaming, Spark Mllib, Cassandra.
- Expertise in NOSQL databases like HBase, MongoDB.
- Strong expertise on Amazon AWS EC2, Dynamo DB, S3, Kinesis and other services
- Expertise in data analysis, design and modeling using tools like ErWin.
- Expertise in Big Data architecture like hadoop (Azure, Hortonworks, Cloudera) distributed system, MongoDB, NoSQL
- Hands on experience on Hadoop /Big Data related technology experience in Storage, Querying, Processing and analysis of data.
- Experienced in using various Hadoop infrastructures such as Map Reduce, Hive, Sqoop, and Oozie.
- Expert in AmazonEMR, Spark, Kinesis, S3, Boto3, Bean Stalk, ECS, Cloudwatch, Lambda, ELB, VPC, Elastic Cache, Dynamo DB, Redshit, RDS, Aethna, Zeppelin & Airflow.
- Strong Experience in Front End Technologies like JSP, HTML5, JQuery, JavaScript, CSS3.
- Experienced in testing data in HDFS and Hive for each transaction of data.
- Experienced in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa.
- Strong Experience in working with Databases like Oracle 12c/11g/10g, DB2, SQL Server and MySQL and proficiency in writing complex SQL queries.
- Experienced in using database tools like SQL Navigator, TOAD.
- Experienced with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
- Extensive knowledge in programming with Resilient Distributed Datasets (RDDs).
- Experienced in using Flume to transfer log data files to Hadoop Distributed File System (HDFS)
- Experienced with Akka building high performance and reliable distributed applications in Java and Scala.
- Knowledge and experience in job work-flow scheduling and monitoring tools like Oozie and Zookeeper.
- Knowledge in configuration and managing - Cloudera’s Hadoop platform along with CDH3&4 clusters.
- Experienced in developing web based GUIs using JavaScript, JSP, HTML, JQuery, XMLand CSS.
- Experienced to develop enterprise applications with J2EE/MVC architecture with application servers and Web servers such as, JBoss, and Apache Tomcat 6.0/7.0/8/0.
TECHNICAL SKILLS
Hadoop/Big Data: Map Reduce, HDFS, Hive, Pig, HBase, Zookeeper, Sqoop, Oozie, Flume, Scala, Akka, Kafka, Storm, Mongo DB
Java/J2EE Technologies.: JDBC, Java Script, JSP, Servlets, JQuery
No SQL Databases: Cassandra, mongo DB
Web Technologies: HTML, DHTML, XML, XHTML, JavaScript, CSS, XSLT, AWS, Dynamo DB
Web/Application servers: Apache Tomcat6.0/7.0/8.0, JBoss
Frameworks: MVC, Struts, Spring, Hibernate.
Operating Systems: UNIX, Ubuntu Linux and Windows, Centos, Sun Solaris.
Network protocols: TCP/IP fundamentals, LAN and WAN.
Languages: Java, J2EE, PL/SQL, Pig Latin, HQL, R, Python, XPath, Spark
Databases: Oracle 12c/11g/10g/9i, Microsoft Access, MS SQL
PROFESSIONAL EXPERIENCE
Confidential, Bronx, NY
Sr. Big Data Architect
Responsibilities:
- DevelopedBigDatasolutions focused on pattern matching and predictive modeling.
- Implementation of Big Data eco system (Hive, Impala, Sqoop, Flume, Spark, Lambda) with Cloud Architecture.
- Experience on BI reporting with At Scale OLAP for Big Data.
- Implemented solutions for ingesting data from various sources and processing the Data-at-Rest utilizing Big Data technologies such asHadoop, Map Reduce Frameworks, HBase, Hive
- Designed and Developed Real time Stream processing Application using Spark, Kafka, Scala and Hive to perform Streaming ETL and apply Machine Learning.
- Experience in AWS, implementing solutions using services like (EC2, S3, RDS, Redshift, VPC)
- Worked as a Hadoop consultant on (Map Reduce/Pig/HIVE/Sqoop).
- Worked with Spark and Python.
- Worked using Apache Hadoop ecosystem components like HDFS, Hive, Sqoop, Pig, and Map Reduce.
- Identify query duplication, complexity and dependency to minimize migration efforts
- Exploring with the Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
- Lead architecture and design of data processing, warehousing and analytics initiatives.
- Worked with AWS to implement the client-side encryption as Dynamo DB does not support at rest encryption at this time.
- Used Data Frame API in Scala for converting the distributed collection of data organized into named columns.
- Performed data profiling and transformation on the raw data using Pig, Python, and Java.
- Experienced with batch processing of data sources using Apache Spark.
- Developing predictive analytic using Apache Spark Scala APIs.
- Created Hive External tables and loaded the data into tables and query data using HQL.
- Used Sqoop to efficiently transfer data between databases and HDFS and used Flume to stream the log data from servers.
- Implement enterprise grade platform(mark logic) for ETL from mainframe to NOSQL(cassandra).
- Experience on BI reporting with At Scale OLAP for Big Data.
- Responsible for importing log files from various sources into HDFS using Flume
- Worked on tools Flume, Storm and Spark.
- Expert in performing business analytical scripts using Hive SQL.
- Implemented continuous integration & deployment (CICD) through Jenkins for Hadoop jobs.
- Worked in writing Hadoop Jobs for analyzing data using Hive, Pig accessing Text format files, sequence files, Parquet files.
- Experience in different Hadoop distributions like Cloudera (CDH3 & CDH4) and Horton Works Distributions (HDP) and MapR.
- Experience in integrating oozie logs to kibana dashboard.
- Extracted the data from MySQL, AWS RedShift into HDFS using Sqoop.
- Developed Spark code using Scala and Spark-SQL for faster testing and data processing.
- Imported millions of structured data from relational databases using Sqoop import to process using Spark and stored the data into HDFS in CSV format.
- Developed Spark streaming application to pull data from cloud to Hive table.
- Used Spark SQL to process the huge amount of structured data.
- Assigned name to each of the columns using case class option in Scala.
- Implemented Spark GraphX application to analyze guest behavior for data science segments.
- Enhancements to traditional data warehouse based on STAR schema, update data models, perform Data Analytics and Reporting using Tableau.
Environment: Big Data, Spark, YARN, HIVE, Pig, Scala, Python, Hadoop, AWS, Dynamo DB, Kibana, Cloudera, EMR, JDBC, Redshift, NOSQL, Sqoop, MYSQL.
Confidential, Malvern, PA
Big Data Architect
Responsibilities:
- Architected, Designed and Developed Business applications and Data marts for Marketing and IT department to facilitate departmental reporting.
- Ingest Worked on implementation and maintenance of Cloudera Hadoop cluster.
- Created Hive External tables to stage data and then move the data from Staging to main tables
- Worked in exportingdatafrom Hive 2.0.0 tables into Netezza 7.2.x database.
- Implemented theBigDatasolution using Hadoop, hive and Informatica 9.5.1 to pull/load thedata into the HDFS system.
- Pulling thedatafromdatalake (HDFS) and massaging thedatawith various RDD transformations.
- Active involvement in design, new development and SLA based support tickets ofBigMachines applications.
- Developed Scala scripts, UDF's using bothDataframes/SQL and RDD/MapReduce in Spark 2.0.0 forDataAggregation, queries and writingdataback into RDBMS through Sqoop.
- Developed Spark code using Scala and Spark-SQL/Streaming for faster processing ofdata.
- Developed Oozie 3.1.0 workflow jobs to execute hive 2.0.0, sqoop 1.4.6 and map-reduce actions.
- Provided thought leadership for architecture and the design ofBigDataAnalytics solutions for customers, actively drive Proof of Concept (POC) and Proof of Technology (POT) evaluations and to implement aBigDatasolution.
- Developed numerous MapReduce jobs in Scala 2.10.x forDataCleansing and AnalyzingDatain Impala 2.1.0.
- CreatedDataPipeline using Processor Groups and multiple processors using Apache Nifi for Flat File, RDBMS as part of a POC using Amazon EC2.
- Build Hadoop solutions forbigdataproblems using MR1 and MR2 in YARN.
- Load thedatafrom different sources such as HDFS or HBase into Spark RDD and implement in memorydatacomputation to generate the output response.
- Developed complete end to endBig-dataprocessing in hadoop eco system.
- Objective of this project is to build a data lake as a cloud based solution in AWS using Apache Spark and provide visualization of the ETL orchestration using CDAP tool.
- Implemented Installation and configuration of multi-node cluster on Cloud using Amazon Web Services(AWS) on EC2.
- Proof-of-concept to determine feasibility and product evaluation of Big Data products
- Writing Hive join query to fetch info from multiple tables, writing multiple Map Reduce jobs to collect output from Hive
- Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting on the dashboard.
- AWS Cloud and On-Premise environments with Infrastructure Provisioning / Configuration.
- Worked on writing Perl scripts covering data feed handling, implementingmark logic, communicating with web-services through SOAP Lite module and WSDL.
- UsedHiveto analyze data ingested intoHBaseby usingHive-HBaseintegration and compute various metrics for reporting on the dashboard
- Involved in developing Map-reduce framework, writing queries scheduling map-reduce
- Developed the code for Importing and exporting data into HDFS and Hive using Sqoop
- Installed and configured Hadoop and responsible for maintaining cluster and managing and reviewing Hadoop log files.
- Developed Shell, Perl and Python scripts to automate and provide Control flow to Pig scripts.
- Design of Redshift Data model, Redshift Performance improvements/analysis
- Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.
- Worked on configuring and managing disaster recovery and backup on Cassandra Data.
- Performed File system management and monitoring on Hadoop log files.
- Utilized Oozie workflow to run Pig and Hive Jobs Extracted files from Mongo DB through Sqoop and placed in HDFS and processed.
- Used Flume to collect, aggregate, and store the web log data from different sources like web servers, mobile and network devices and pushed to HDFS.
- Implemented partitioning, dynamic partitions and buckets in HIVE.
- Developed customized classes for serialization and Deserialization in Hadoop
- Implemented a proof of concept deploying this product in Amazon Web Services AWS.
- Involved in migration of data from existing RDBMS (oracle and SQL server) to Hadoop using Sqoop for processing data.
Environment: Pig, Sqoop, Kafka, Apache Cassandra, Oozie, Impala, Cloudera, AWS, AWS EMR, Redshift, Flume, Apache Hadoop, HDFS, Hive, Map Reduce, Cassandra, Zookeeper, MySQL, Eclipse, Dynamo DB, PL/SQL and Python.
Confidential, Arlington, VA
Big Data Engineer
Responsibilities:
- Involved in extractingdatafeatures through Spark ML with Python.
- Designed and developed Enterprise Eligibility business objects and domain objects with Object Relational Mapping framework such as Hibernate.
- Developed the Web Based Rich Internet Application (RIA) using JAVA/J2EE (spring framework).
- Used the light weight container of the Spring Frame work to provide architectural flexibility for inversion of controller (IOC).
- Involved in end to end implementation ofBigdatadesign.
- Developed and Implemented new UI's using Angular JS and HTML.
- Developed Spring Configuration for dependency injection by using Spring IOC, Spring Controllers.
- Implementing Spring MVC and IOC methodologies.
- Used the JNDI for Naming and directory services.
- Involved in the coding and integration of several business critical modules of application using Java, Spring, Hibernate and REST web services on WebSphere application server.
- Experience working withbigdataand real time/near real time analytics andbigdataplatforms like Hadoop, Spark using programming languages like Scala and Java.
- DeliverBigDataProducts including re-platforming Legacy Global Risk Management System with BigDataTechnologies such as Hadoop, Hive and HBase.
- Worked with NoSQL Mongo DB and heavily worked on Hive, Hbase and HDFS
- Developed Restful web services using JAX-RS and used DELETE, PUT, POST, GET HTTP methods in spring 3.0 and OSGI integrated environment.
- Developed optimal strategies for distributing the web log data over the cluster, importing and exporting the stored web log data into HDFS and Hive using Scoop.
- Collected and aggregated large amounts of web log data from different sources such as webservers, mobile and network devices using Apache Flume and stored the data into HDFS for analysis.
- Used Java Messaging Services (JMS) for reliable and asynchronous exchange of important information such as payment status report on IBM WebSphere MQ messaging system.
- Developed presentation layer using Java Server Faces (JSF) MVC framework.
- Participated in JAD meetings to gather the requirements and understand the End Users System.
- Developed user interfaces using JSP, HTML, XML and JavaScript.
- Generated XML Schemas and used XML Beans to parse XML files.
- Modified the existing JSP pages using JSTL.
- Developed web components using JSP, Servlets, and JDBC.
- Developed web pages using JSPs and JSTL to help end user make online submission of rebates. Also used XML Beans for data mapping of XML intoJavaObjects.
- Used Spring JDBC Dao as a data access technology to interact with the database.
- Developed Unit and E2E test cases using Node JS.
Environment: JSP 2.1, Hadoop 1x, Hive, Pig, HBASE, JSTL 1.2, Java, J2EE, Java SE 6, UML, Servlets 2.5, Spring MVC, Hibernate, JSON, Eclipse Kepler-Maven, Serena Dimensions, Unix, JUnit, DB2, Oracle, Restful Web services, Big Data, jQuery, AJAX, Angular Js, JAXB, IRAD Web sphere Integration Developer, Web Sphere 7.0
Confidential, Chicago, IL
Sr. Java/ J2EE Developer
Responsibilities:
- Developing Intranet Web Application using J2EE architecture, using JSP to design the user interfaces, and JSP tag libraries to define custom tags and JDBC for database connectivity.
- Developed and coordinated complex high quality solutions to clients using IBM Products/Tools, Apache Tools, J2SE, J2EE, EJB, Servlets, JSP, HTML, JavaScript, JQuery, JSON and XML.
- Used JPA (Java Persistence API) with Hibernate as Persistence provider for Object Relational mapping.
- Customizing third party vendor information using Web services (SOAP and WSDL).
- Expertise in designing and creating RESTful API's using ApacheSolrand Spring WS Developed and modified database objects as per the requirements.
- Used Struts framework along with JSP, Servlets, HTML, CSS, JQuery and JavaScript to construct dynamic web pages.
- Designed and Developed interactive and dynamic front end web applications using HTML, Bootstrap and CSS.
- Implementing the 1.3 framework for Model View Controller (MVC) pattern.
- Involved in the development of the project using Struts.
- Deployed components on JBoss and Web Sphere application server.
- Used JDBC API to connect to the database and Performed CRUD operations to get and check the data.
- Involved in writing queries, stored procedures on Oracle 10g database.
- Implemented Log4j for the project to compile and package the application, used ANT and MAVEN to automate build and deployment scripts.
- Create SQL and use MySQL and Oracle 10g as Relational Database Management Systems.
- Created POJO classes, java beans, EJBBeans and wrote JUnit test cases to test code as per the acceptance criteria throughout the application during development and testing Phase.
- Used Oracle Coherence for caching scheme, for persistent layer NoSql DB Cassandra is used.
- Used Git as version control tools to maintain the code repository.
- As part ofAngularJSdevelopment have used data-binding and developed controllers, directives, filters and integrated with the backend-services.
- Wrote SQL stored procedures, and used JDBC to connect to database.
- Used Struts-Validator frame-work for all front-end Validations for all the form entries. Configured and used JBoss application server for creating Data source and enabling JNDI for beans and database lookups.
- Worked with CVS Version control.
- Prepared EJB deployment descriptors using XML and Used JAXB components for transferring the objects between the application and the database.
- Used SQL, PL/SQL in indexes, views, stored procedures, functions and relational database models.
- Wrote PL/SQL statements according to the need using Oracle 10g database.
- Used Flume to collect the logdatafrom different resources and transfer thedatatype to hive tables using different SerDe to store in JSON, XML and Sequence file formats.
- Worked with application teams to install operating system,Hadoopupdates, patches, version upgrades as required.
Environment: Java, JSF, Spring, Hibernate, Linux Shell Script, JaxWS, SOAP, WSDL, CSS3, html3, JBOSS, JSF, Rally, Hudson xml, html, Clear Case, Clear Quest, My Eclipse, ANT, Oracle, Linux, Oracle 10g database.
Confidential
Java Consultant
Responsibilities:
- Responsible for design, development, test and maintenance of applications designed on Java technologies.
- Developed UI using HTML, JavaScript, and JSP, and developed Business Logic and Interfacing components using Business Objects, XML, and JDBC.
- Created rapid prototypes of interfaces to be used as blueprints for technical development.
- Developed user stories using Core Java and Spring 3.1 and consumed rest web services exposed from the profit center.
- Used UML diagrams Use Cases, Object, Class, State, Sequence and Collaboration to design the application using Object Oriented analysis and design
- Managed connectivity using JDBC for querying/inserting & data management including triggers and stored procedures.
- Developed JavaScript behavior code for user interaction.
- Created database program in SQL server to manipulate data accumulated by internet transactions.
- Wrote Servlets class to generate dynamic HTML pages.
- Developed SQL queries and Stored Procedures using PL/SQL to retrieve and insert into multiple database schemas.
- Developed the XML Schema and Web services for the data maintenance and structures Wrote test cases in JUnit for unit testing of classes.
- Used DOM and DOM Functions using Firefox and IE Developer Tool bar for IE.
- Debugged the application using Firebug to traverse the documents.
- Involved in developing web pages using HTML and JSP.
- Provided Technical support for production environments resolving the issues, analysing the defects, providing and implementing the solution defects.
- Involved in writing SQL Queries, Stored Procedures and used JDBC for database connectivity with MySQL Server.
- Developed the presentation layer using CSS and HTML taken from bootstrap to develop for browsers.
- Used Oracle 10g as the backend database using UNIX OS.
- Used JSP, HTML, Java Script, Angular JS and CSS3 for content layout and presentation.
- Did core Java coding using JDK 1.3, Eclipse Integrated Development Environment (IDE), clear case, and ANT.
- Used Spring Core and Spring-web framework. Created a lot of classes for backend.
Environment: Java, XML, HTML, JavaScript, JDBC, CSS, SQL, PL/SQL, XML, Web MVC, Eclipse, Ajax, JQuery, spring with Hibernate, Active MQ, Jasper Reports, Ant as build tool and My SQL and Apache Tomcat
