Big Data Architect Resume
San Jose, CA
SUMMARY
- 9+ years of IT experience with gathering and analyzing customer’s technical requirements, development, management, maintenance and production support projects on platforms like Hadoop and Java.
- Extensive working experience on Hadoop eco - system components likeMapReduce (MRv1), Hive, Pig, Sqoop, Flume, Oozie.
- Efficient in writing MapReduce Programs and using Apache Hadoop API for analyzing the structured and unstructured data.
- Experienced in the Hadoop ecosystem components like Hadoop Map Reduce, Cloudera, Hortonworks, HBase, Oozie, Hive, Sqoop, Pig, Flume, and Cassandra.
- Expert in working with Hive data warehouse tool-creating tables, data distribution by implementing partitioning and bucketing, writing and optimizing the HiveQL queries.
- Experience in writing Pig Latin scripts.
- Experience in writing UDFS in java for hive and pig.
- Hands on experience in configuring and working with Flume to load the data from multiple sources directly into HDFS.
- Experience in using Apache Sqoop to import and export data to and from HDFS and Hive.
- Hands on experience in setting up workflow using Apache Oozie workflow engine for managing and scheduling Hadoop jobs.
- Experience in handline messaging services using Apache Kafka.
- Good knowledge of No-SQL databases-Cassandra and HBASE.
- Experience in using Hcatalog for Hive, Pig and Hbase.
- Working experience on Pentaho data integration (PDI) Kettle - Extraction, Transformation, and Loading (ETL) and Tableau visualization.
- Experience in working with BI team and transform Big Data requirements into Hadoop centric technologies.
- Hands on experience in writing MapReduce jobs in Java, Scala and Python
- Strong knowledge in Hadoop cluster installation, capacity planning and performance tuning, benchmarking, disaster recovery plan and application deployment in production cluster.
- Strong knowledge in internals of HDFS and MapReduce framework.
- Experience in developing applications using core Java and web technologies.
- Exposure to Maven/Ant, Git along with Shell Scripting for Build & Deployment Process
- Basic knowledge in application design using Unified Modeling Language (UML), Sequence diagrams, Case diagrams, Entity Relationship Diagrams (ERD) and Data Flow Diagrams (DFD).
- Strong technical and interpersonal skills combined with great commitment towards meeting deadlines.
- Experience working in both team and individual environments. Always eager to learn new technologies and implement them in challenging environment.
TECHNICAL SKILLS
Hadoop Technologies and Distributions: Apache Hadoop, Cloudera Hadoop Distribution CDH3, CDH4, CDH5 and Hortonworks Data Platform (HDP)
Hadoop Ecosystem: HDFS, Map-Reduce, Hive, Kafka, Pig, Sqoop, Oozie, Flume, Zookeeper
NoSql Databases: Cassandra
Programming: C, C++, Java,PL/SQL, ASP .Net
RDBMS: ORACLE, MySQL,SQL Server
Web Development: HTML, JSP, Servlets, JavaScript, CSS, XML
IDE: Rational Rose, Eclipse, NetBeans,Microsoft Visual Studio
Operating Systems: Linux (RedHat, CentOS), Windows XP/7/8
Web Servers: Apache Tomcat
Cluster Management Tools: Cloudera Manager, HortonWorksAmbari
BI Tools: Tableau
PROFESSIONAL EXPERIENCE
Confidential, San Jose, CA
Big Data Architect
Responsibilities:
- Involved in extracting data from various data sources into HDFS. This included data from JIRAP, database and Rest API’s
- Used Sqoop to efficiently transfer data between databases and HDFS and used Flume to stream the JSON data from Rest APIs
- Monitor Hadoop cluster using tools like Hortonworks, Cloudera manager,nagios, Ganglia, and Ambari
- Created Hive tables based on requirement and pushed the data into them for analysis.
- Imported the result of Analysis to Mongo DB from where it got fed into Dashboard where Business would see the result
- Used Apache Kafka for importing real time network log data into HDFS
- Interacted with many teams to get the data sources for ingestion into HDFS.
- Used Sort Merge Buckets(SMB) for reducing run time in the hive analysis.
Confidential, Bellevue, WA
Hadoop Architect
Responsibilities:
- Involved in extracting customer’s Big data from various data sources into Hadoop HDFS. This included data from Excel, ERP systems, databases and also log data from servers.
- Used Sqoop to efficiently transfer data between databases and HDFS and used Flume to stream the log data from servers/sensors.
- Developed a framework with automates the importing data into HDFS and cleanse the data from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis.
- Worked hands on with the DevOps team to meet the specific business requirements for individual customers and proposed Hadoop solutions across multiple verticals.
- Used Hive data warehouse tool to analyze the unified historic data in HDFS to identify issues and behavioral patterns.
- Collected user activity data, log data using Kafka for real time analytics
- Installed and configured multi-nodes fully distributed Hadoop cluster with Hortonworks Ambari
- The Hive tables created as per requirement were internal or external tables defined with appropriate static and dynamic partitions, intended for efficiency.
- Written Multiple Pig UDFS to accommodate business requirements.
- Used Sort Merge Buckets(SMB) for reducing run time in the hive analysis
- Used the RegEx, JSON and Avro SerDe’s for serialization and de-serialization packaged with Hive to parse the contents of streamed log data and implemented Hive custom UDF’s.
- Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java map-reduce, Hive and Sqoop as well as system specific jobs.
- Worked with BI teams in generating the reports and designing ETL workflows.
Confidential, Schaumburg, IL
Hadoop Developer
Responsibilities:
- Involved in installing, configuring and managing Hadoop Ecosystem components like Hive, Pig, Sqoop and Flume.
- Migrated the existing data to Hadoop from RDBMS (SQL Server and Oracle) using Sqoop for processing the data.
- Responsible for loading unstructured and semi-structured data into Hadoop cluster coming from different sources using Flume and managing.
- Developed MapReduce programs to cleanse and parse data in HDFS obtained from various data sources and to perform joins on the Map side using distributed cache.
- Used Hive data warehouse tool to analyze the data in HDFS and developed Hive queries.
- Created internal and external tables with properly defined static and dynamic partitions for efficiency.
- Used the RegEx, JSON and Avro SerDe’s for serialization and de-serialization packaged with Hive to parse the contents of streamed log data.
- Implemented Hive custom UDF’s to achieve comprehensive data analysis.
- Used Pig to develop ad-hoc queries.
- Exported the business required information to RDBMS using Sqoop to make the data available for BI team to generate reports based on data.
- Implemented daily workflow for extraction, processing and analysis of data with Oozie.
- Responsible for troubleshootingMapReduce jobs by reviewing the log files.
- Used PDI (Pentaho Data Integration) to extract, transform and load (ETL) using metadata driven approach.
Confidential
Jr.Java Developer
Responsibilities:
- Worked with requirement analysis team to gather software requirements for application development.
- Designed UML and entity relational diagrams for the process flow and database design.
- Developed java programs to implement the computational logic for the web applications.
- Designed static web user interface with html and css.
- Developed custom packages to connect to standard data sources and retrieve data efficiently eliminating the need for each team to rewrite the same set of code multiple times.
- Worked on product deployment, documentation and support.
- Involved in structuring Wiki and Forums for product documentation
- Involved in R&D, set up and designing Mediawiki, Php and Joomla content management systems.
- Maintained the customer support portal.
