We provide IT Staff Augmentation Services!

Sr. Big Data Engineer Resume

5.00/5 (Submit Your Rating)

Sanjose-cA

SUMMARY

  • Hadoop & Spark Developer and analyst with over7+years of overall experience as software developer in design, development, deploying and large scale supporting large scale distributed systems.
  • 5+years of extensive experience as Hadoop and spark engineer and Big Data analyst.
  • Implemented various algorithms for analytics using Cassandra with Spark and Scala.
  • Excellent understanding of Hadoop architecture and underlying framework includingstorage management.
  • Have experience in installing,configuring and administrating Hadoop cluster for major Hadoop distributions likeCDH4, and CDH5.
  • Expertise in using various Hadoop infrastructures such asMap Reduce, Pig, Hive, Zookeeper, Hbase, Sqoop, Oozie, Flume, Drillandsparkfor data storage and analysis.
  • Experience in developing custom UDFsfor Pig and Hive to in corporate methods and functionality of Python/Java intoPig LatinandHQL(HiveQL) and Used UDFs from Piggybank UDF Repository.
  • Experienced in running query - usingImpalaand used BI tools to run ad-hoc queries directly on Hadoop.
  • Good experience inOozieFramework and Automating daily import jobs.
  • Experienced in managing Hadoop clusters and services usingClouderaManager.
  • Experienced in troubleshooting errors in Hbase Shell/API, Pig, Hive and map Reduce.
  • Highly experienced in importing and exporting data betweenHDFSandRelational Database Management systemsusingSqoop.
  • Experienced in Creating Vizboards for data visualization inPlatforafor real - time dashboard on Hadoop.
  • Collected logs data from various sources and integrated in to HDFS usingFlume.
  • AssistedDeploymentteamin setting upHadoop clusterand services.
  • Good experience in Generating Statistics/extracts/reports from the Hadoop.
  • Good understanding ofNoSQLData bases and hands on work experience in writing applications on No SQL data bases likeCassandraandMongo DB.
  • Designed and implemented a product search service using Apache Solr.
  • Good knowledge in querying data fromCassandrafor searching grouping and sorting.
  • Good Knowledge inAmazon AWSconcepts likeEMR and EC2web services which provides fast and efficient processing of Big Data.
  • Strong experience in core Java,Scala, SQL, PL/SQL and Restful web services.
  • Having good knowledge in Benchmarking & Performance Tuning of cluster.
  • Experienced in Identifying improvement areas for systems stability and providing end end high availability architectural solutions.
  • Good experience in GeneratingStatistics and reportsfrom the Hadoop.
  • Determined, committed and hardworking individual with strong communication, interpersonal and organizational skills.

TECHNICAL SKILLS

Hadoop Ecosystem: HDFS, Map Reduce YARN, Hive, Pig, Hbase, Impala, Zookeeper, Sqoop, Oozie, DataStax & Apache Cassandra, Drill, Flume, Spark, Solr and Avro, AWS, Amazon EC2, S3.

Web Technologies: HTML, XML, JDBC, JSP, JavaScript, AJAX

RDBMS: Oracle 10g/11g, MySQL, SQL server, Teradata

No SQL: Hbase, Cassandra

Web/Application servers: Tomcat, LDAP

Methodologies: Agile, UML, Design Patterns (Core Java and J2EE)

Data Bases: Oracle 11g/10g, Teradata, DB2, MS-SQL Server, MySQL, MS-Access

Programming Languages: Scala, Python, SQL, Java, PL/SQL, Linux shell scripts.

Tools: Used: Eclipse, Putty, Cygwin, MS Office

BI Tools: Platfora, Tableau, Pentaho

PROFESSIONAL EXPERIENCE

Confidential, Sanjose-CA

Sr. Big Data Engineer

Responsibilities:

  • Implemented a generic ETL framework withhigh availabilityfor bringing related data for Hadoop & Cassandra from various sources using spark.
  • Experienced in usingPlatforaa data visualization tool specific for Hadoop, and created various Lens and Viz boards for a real-time visualization from hive tables.
  • Queried and analyzed data fromCassandrafor quick searching, sorting and grouping throughCQL.
  • Implemented various Data Modeling techniques forCassandra.
  • Joined various tables in Cassandra usingspark and Scalaand ran analytics on top of them.
  • Participated in various upgradations and troubleshooting activities across enterprise.
  • Knowledge in performancetroubleshooting and tuningHadoop clusters.
  • AppliedSparkadvanced procedures liketext analytics and processingusing thein-memoryprocessing.
  • Implemented ApacheDrillon Hadoop to join data from SQL and No SQL databases and store it in Hadoop.
  • Created architecture stack blueprint for data access with NoSQL DatabaseCassandra;
  • Brought data from various sources in to Hadoop and Cassandra usingKafka.
  • Experienced in usingTidal enterprise scheduler and OozieOperational Services for coordinating the cluster and scheduling workflows.
  • Applied spark streaming for real time data transforming.
  • Created multiple dashboards in tableau for multiple business needs.
  • Installed and configured Hive and written Hive UDFs and used piggy bank a repository of UDF’s for Pig Latin.
  • Implemented Partitioning, Dynamic Partitions and Buckets inHIVEfor efficient data access.
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team UsingTableau.
  • ImplementedCompositeserver for thedata virtualizationneeds and created multiples views for restricted data access using a REST API.
  • Devised and lead the implementation of next generation architecture for more efficient data ingestion and processing.
  • Created and implemented variousshell scriptsfor automating the jobs.
  • ImplementedApache Sentryto restrict the access on the hive tables on a group level.
  • EmployedAVROformat for the entire data ingestion for faster operation and less space utilization.
  • Experienced in managing and reviewing Hadoop log files.
  • Worked inAgile environment,and used rally tool to maintain the user stories and tasks.
  • Worked withEnterprise data supportteams to install Hadoop updates, patches, version upgrades as required and fixed problems, which raised after the upgrades.
  • Implemented test scripts to support test-driven development and continuous integration.
  • Used Spark for Parallel data processing and better performances.

Environment: MapR 5.0.1, Map Reduce, HDFS, Hive, pig, Impala,Cassandra 5.04, spark, Scala,Solr, Java, SQL, Tableau, PIG, Zookeeper, Sqoop, Teradata, CentOS, Pentaho.

Confidential, Louisville, Kentucky

Hadoop Developer

Responsibilities:

  • Installed and configured Hadoop Map Reduce, HDFS, developed multiple Map Reduce jobs in java and Scala for data cleaning and preprocessing.
  • Experienced in installing, configuring and using Hadoop Ecosystem components.
  • Experienced in Importing and exporting data into HDFS and Hive usingSqoop.
  • Participated in development/implementation of Cloudera Hadoop environment.
  • Experienced in running query-usingImpalaand used BI tools to run ad-hoc queries directly on Hadoop.
  • IntegratedCassandraas a distributed persistent metadata store to provide metadata resolution for network entities on the network
  • Involved in various NOSQL databases likeHbase, Cassandrain implementing and integration.
  • Installed and configured Hive and writtenHive UDFsand Used Map Reduce and Junit for unit testing.
  • UsedDataStax Cassandraalong with Pentaho for reporting.
  • Queried and analyzed data fromDataStaxCassandrafor quick searching, sorting and grouping.
  • Experienced in working with various kinds of data sources such as Teradata and Oracle. Successfully loaded files to HDFS from Teradata, and load loaded from HDFS to hive and impala.
  • Designed and implemented a product search service usingApache Solr/Lucene.
  • Worked in installing cluster,commissioning & decommissioning of Data nodes, Name node recovery, capacity planning, and slots configuration.
  • UsedYarn Architecture and Map reduce 2.0in the development cluster for POC.
  • Supported Map Reduce Programs those are running on the cluster. Involved in loading data from UNIX file system to HDFS.
  • Load and transform large sets of structured, semi structured and unstructured data.

Environment: CDH 5.0, 5.1, Map Reduce, HDFS, Hive, pig, Impala, Cassandra, spark, Solr, Java, SQL, Tableau, PIG, Zookeeper, Sqoop, Teradata, CentOS, Pentaho.

Confidential, Oakland-CA

Hadoop Developer

Responsibilities:

  • Acted as a lead resource and build the entire Hadoop platform from scratch.
  • Evaluated suitability of Hadoop and its ecosystem to the above project and implementing / validating with various proof of concept (POC) applications to eventually adopt them to benefit from the Big Data Hadoop initiative.
  • Estimated the Software & Hardware requirements for theName NodeandData Node& planning the cluster.
  • Extracted the needed data from the server into HDFS andBulk Loadedthe cleaned data intoHbase.
  • Lead role in NoSQL column family design, client access software, Cassandra tuning; during migration from Oracle based data stores.
  • Designed, implemented and deployed within a customer’s existingHadoop / Cassandracluster a series of custom parallel algorithms for various customer defined metrics and unsupervised learning models.
  • Using theSparkframeworkEnhancedandoptimizedproductSparkcode toaggregate,groupand rundata miningtasks.
  • Wrote queries UsingDataStax Cassandra CQLto create, alter, insert and delete elements.
  • Written the Map Reduce programs,HiveUDFsin Java.
  • Used Map ReduceJUnitfor unit testing.
  • Deployed an ApacheSolr/Lucenesearch engine server to help speed up the search of financial documents.
  • DevelopHIVEqueries for the analysts.
  • Created ane-mail notificationservice upon completion of job for the particular team which requested for the data.
  • Defined job work flows as per their dependencies inOozie.
  • Played a key role in productionizing the application after testing by BI analysts.
  • Given POC ofFLUMEto handle the real-time log processing for attribution reports.
  • Maintain System integrity of all sub-components related to Hadoop.

Environment: Apache Hadoop, HDFS, Spark, Solr, Hive, DataStax Cassandra, Map Reduce, Pig, Java, Flume, Cloudera CDH4, Oozie, Oracle, MySQL, Amazon S3.

Confidential -Sandiego-CA

Data Analyst

Responsibilities:

  • Successfully Completed Junior Data Analyst Internship in Confidential.
  • Built anExpense Trackerand Zonal Desk.
  • Identifying inconsistencies, correcting them orescalatingthe problems to next level.
  • Assisted in development of interface testing and implementation plans.
  • Analyzing data for data quality and validation issues.
  • Analyzing the websites regularly to ensure site traffic and conversion funnels are performing well.
  • Collaborating with Sales and marketing teams to optimize processes that communicate insights effectively.
  • Creating and maintainingautomated reportsusingSQL.
  • Understood all theHadoop architectureand drove all the meetings
  • Conducted safety check to make sure that my team is feeling safe for the retrospectives
  • Aided in data profiling by examining the source data
  • Extracting features from the given data set and use them to train and evaluate different classifiers that are available in theWEKAtool. Using these features, we differentiate spam messages from legitimate messages.
  • Created numerousSQL queriesto modify data based on data requirements and added enhancements to existing procedures.
  • Implemented statistical modelling techniques in Python.
  • Conducted safety check to make sure that my team is feeling safe for the retrospectives
  • Aided indata profilingby examining the source data
  • Performed data mappings to map the source data to the destination data
  • DevelopedUse Case Diagramsto identify the users involved. Created Activity diagrams and Sequence diagrams to depict the process flows.

Environment: Python, Matlab, Oracle, HTML5, Tableau, MS Excel, Server Services, Informatica Power CenterSQL, Microsoft Test Manager, Adobe Connect, MS Office Suite, LDAP, Hive, Spark, Pig, Oozie.

We'd love your feedback!