We provide IT Staff Augmentation Services!

Big Data Architect Resume

0/5 (Submit Your Rating)

San Jose, CA

SUMMARY

  • 9+ years of IT experience with gathering and analyzing customer’s technical requirements, development, management, maintenance and production support projects on platforms like Hadoop and Java.
  • Extensive working experience on Hadoop eco - system components likeMapReduce (MRv1), Hive, Pig, Sqoop, Flume, Oozie.
  • Efficient in writing MapReduce Programs and using Apache Hadoop API for analyzing the structured and unstructured data.
  • Experienced in the Hadoop ecosystem components like Hadoop Map Reduce, Cloudera, Hortonworks, HBase, Oozie, Hive, Sqoop, Pig, Flume, and Cassandra.
  • Expert in working with Hive data warehouse tool-creating tables, data distribution by implementing partitioning and bucketing, writing and optimizing the HiveQL queries.
  • Experience in writing Pig Latin scripts.
  • Experience in writing UDFS in java for hive and pig.
  • Hands on experience in configuring and working with Flume to load the data from multiple sources directly into HDFS.
  • Experience in using Apache Sqoop to import and export data to and from HDFS and Hive.
  • Hands on experience in setting up workflow using Apache Oozie workflow engine for managing and scheduling Hadoop jobs.
  • Experience in handline messaging services using Apache Kafka.
  • Good knowledge of No-SQL databases-Cassandra and HBASE.
  • Experience in using Hcatalog for Hive, Pig and Hbase.
  • Working experience on Pentaho data integration (PDI) Kettle - Extraction, Transformation, and Loading (ETL) and Tableau visualization.
  • Experience in working with BI team and transform Big Data requirements into Hadoop centric technologies.
  • Hands on experience in writing MapReduce jobs in Java, Scala and Python
  • Strong knowledge in Hadoop cluster installation, capacity planning and performance tuning, benchmarking, disaster recovery plan and application deployment in production cluster.
  • Strong knowledge in internals of HDFS and MapReduce framework.
  • Experience in developing applications using core Java and web technologies.
  • Exposure to Maven/Ant, Git along with Shell Scripting for Build & Deployment Process
  • Basic knowledge in application design using Unified Modeling Language (UML), Sequence diagrams, Case diagrams, Entity Relationship Diagrams (ERD) and Data Flow Diagrams (DFD).
  • Strong technical and interpersonal skills combined with great commitment towards meeting deadlines.
  • Experience working in both team and individual environments. Always eager to learn new technologies and implement them in challenging environment.

TECHNICAL SKILLS

Hadoop Technologies and Distributions: Apache Hadoop, Cloudera Hadoop Distribution CDH3, CDH4, CDH5 and Hortonworks Data Platform (HDP)

Hadoop Ecosystem: HDFS, Map-Reduce, Hive, Kafka, Pig, Sqoop, Oozie, Flume, Zookeeper

NoSql Databases: Cassandra

Programming: C, C++, Java,PL/SQL, ASP .Net

RDBMS: ORACLE, MySQL,SQL Server

Web Development: HTML, JSP, Servlets, JavaScript, CSS, XML

IDE: Rational Rose, Eclipse, NetBeans,Microsoft Visual Studio

Operating Systems: Linux (RedHat, CentOS), Windows XP/7/8

Web Servers: Apache Tomcat

Cluster Management Tools: Cloudera Manager, HortonWorksAmbari

BI Tools: Tableau

PROFESSIONAL EXPERIENCE

Confidential, San Jose, CA

Big Data Architect

Responsibilities:

  • Involved in extracting data from various data sources into HDFS. This included data from JIRAP, database and Rest API’s
  • Used Sqoop to efficiently transfer data between databases and HDFS and used Flume to stream the JSON data from Rest APIs
  • Monitor Hadoop cluster using tools like Hortonworks, Cloudera manager,nagios, Ganglia, and Ambari
  • Created Hive tables based on requirement and pushed the data into them for analysis.
  • Imported the result of Analysis to Mongo DB from where it got fed into Dashboard where Business would see the result
  • Used Apache Kafka for importing real time network log data into HDFS
  • Interacted with many teams to get the data sources for ingestion into HDFS.
  • Used Sort Merge Buckets(SMB) for reducing run time in the hive analysis.

Confidential, Bellevue, WA

Hadoop Architect

Responsibilities:

  • Involved in extracting customer’s Big data from various data sources into Hadoop HDFS. This included data from Excel, ERP systems, databases and also log data from servers.
  • Used Sqoop to efficiently transfer data between databases and HDFS and used Flume to stream the log data from servers/sensors.
  • Developed a framework with automates the importing data into HDFS and cleanse the data from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis.
  • Worked hands on with the DevOps team to meet the specific business requirements for individual customers and proposed Hadoop solutions across multiple verticals.
  • Used Hive data warehouse tool to analyze the unified historic data in HDFS to identify issues and behavioral patterns.
  • Collected user activity data, log data using Kafka for real time analytics
  • Installed and configured multi-nodes fully distributed Hadoop cluster with Hortonworks Ambari
  • The Hive tables created as per requirement were internal or external tables defined with appropriate static and dynamic partitions, intended for efficiency.
  • Written Multiple Pig UDFS to accommodate business requirements.
  • Used Sort Merge Buckets(SMB) for reducing run time in the hive analysis
  • Used the RegEx, JSON and Avro SerDe’s for serialization and de-serialization packaged with Hive to parse the contents of streamed log data and implemented Hive custom UDF’s.
  • Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java map-reduce, Hive and Sqoop as well as system specific jobs.
  • Worked with BI teams in generating the reports and designing ETL workflows.

Confidential, Schaumburg, IL

Hadoop Developer

Responsibilities:

  • Involved in installing, configuring and managing Hadoop Ecosystem components like Hive, Pig, Sqoop and Flume.
  • Migrated the existing data to Hadoop from RDBMS (SQL Server and Oracle) using Sqoop for processing the data.
  • Responsible for loading unstructured and semi-structured data into Hadoop cluster coming from different sources using Flume and managing.
  • Developed MapReduce programs to cleanse and parse data in HDFS obtained from various data sources and to perform joins on the Map side using distributed cache.
  • Used Hive data warehouse tool to analyze the data in HDFS and developed Hive queries.
  • Created internal and external tables with properly defined static and dynamic partitions for efficiency.
  • Used the RegEx, JSON and Avro SerDe’s for serialization and de-serialization packaged with Hive to parse the contents of streamed log data.
  • Implemented Hive custom UDF’s to achieve comprehensive data analysis.
  • Used Pig to develop ad-hoc queries.
  • Exported the business required information to RDBMS using Sqoop to make the data available for BI team to generate reports based on data.
  • Implemented daily workflow for extraction, processing and analysis of data with Oozie.
  • Responsible for troubleshootingMapReduce jobs by reviewing the log files.
  • Used PDI (Pentaho Data Integration) to extract, transform and load (ETL) using metadata driven approach.

Confidential

Jr.Java Developer

Responsibilities:

  • Worked with requirement analysis team to gather software requirements for application development.
  • Designed UML and entity relational diagrams for the process flow and database design.
  • Developed java programs to implement the computational logic for the web applications.
  • Designed static web user interface with html and css.
  • Developed custom packages to connect to standard data sources and retrieve data efficiently eliminating the need for each team to rewrite the same set of code multiple times.
  • Worked on product deployment, documentation and support.
  • Involved in structuring Wiki and Forums for product documentation
  • Involved in R&D, set up and designing Mediawiki, Php and Joomla content management systems.
  • Maintained the customer support portal.

We'd love your feedback!