Big Data Engineer Resume
Beaverton, OR
SUMMARY
- Over 6+ years of professional IT experience in all phases of Software Development Life Cycle including hands on experience in and Big Data Analytics.
- More than 5+years of work experience in ingestion, storage, querying, processing and analysis of Bigdata with hands on experience in Hadoop Ecosystem development including MapReduce, HDFS, Hive, Pig, Spark, Cloudera Navigator, Mahout, HBase, Zookeeper, Sqoop, Flume, Oozie and AWS.
- 4+ years of expertise in designing and implementing complete end - to-end Hadoop/bigdata solutions using HDFS, MapReduce, HBase, Spark, Yarn, Kafka, Zookeeper, HIVE, Impala.
- Extensive experience working in Teradata, Oracle, Netezza, SQL Server and MySQL database.
- Strong experience working with different Hadoop distributions like Cloudera, Hortonworks, MapReduce and Apache distributions.
- Experience in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH 5.X) distributions and on Amazon web services (AWS).
- Experience in Amazon AWS services such as EMR, EC2, S3, Cloud Formation, RedShift which provides fast and efficient processing of Big Data.
- Strong experience and knowledge of real time data analytics using Storm, Kafka, Flume and Spark.
- Good experience on general data analytics on distributed computing cluster like Hadoop using Apache Spark, Impala, and Scala.
- Experienced in Apache Spark for implementing advanced procedures like text analytics and processing using the in-memory computing capabilities written in Scala.
- Experience in troubleshooting errors in HBase Shell, Pig, Hive and MapReduce.
- Experience in extending HIVE and PIG core functionality by using custom UDF's and UDAF's.
- Debugging MapReduce jobs using Counters and MRUNIT testing.
- Expertise in dealing with semi structured file formats like complex Json and XML file formats.
- Experienced in moving data from different sources using Kafka producers, consumers and pre-process data using Storm topologies.
- Experienced in migrating ETL transformations using Pig Latin Scripts, transformations, join operations.
- Good understanding of MPP databases such as HPVertica, Greenplum and Impala.
- Highly Knowledgeable in streaming data from different data sources like Log files, JMS, applications sources into HDFS using Flume sources.
- Experience with Testing MapReduce programs using MR Unit, Junit.
- Involved in Incident management and change management process. Heavily involve in fully automated CI/CD pipeline process through Github, Jenkins
- Experience working on Version control tools like SVN and Git revision control systems such as GitHub and JIRA/MINGLE to track issues and crucible for code reviews.
- Worked on various Tools and IDEs like Eclipse, IBM Rational, Visio, Apache Ant-Build Tool, MS-Office, PLSQL Developer, SQL*Plus.
- Experience in different application servers like JBoss/Tomcat, WebLogic, and IBM WebSphere.
- Experience in working with Onsite-Offshore model.
- Implemented logging framework - ELK stack (Elastic Search, Logstash & Kibana) on AWS.
TECHNICAL SKILLS
Programming Languages: Ja Python, Scala, Shell Scripting, SQL, PL/SQL.
Bigdata Ecosystem: Hadoop, MapReduce, YARN, HDFS, HBase, Hive, Hue, Pig, Sqoop, HBase, Spark, Impala, Flume, Kafka, Oozie, Zookeeper
Databases: NoSQL, Oracle 10g/11g/12C, SQL Server 2008/2008 R 2/2012/2014/2016/ 2017 , MySQL .
Database Tools: Oracle SQL Developer, MongoDB, TOAD and PLSQL Developer.
Cloud Tools: S3, EMR, Redshift, Redshift Spectrum, Athena, Data pipeline, step functions, Glue and Athena
Operating systems: Windows, UNIX, Linux (Ubuntu), Solaris, Centos, Ubuntu, Windows Server 2003/2006/2008/2009/2012/2013/2016.
Frameworks: MVC, Struts, Log4J, Junit, Maven, ANT, Webservices.
PROFESSIONAL EXPERIENCE
Big Data Engineer
Confidential, Beaverton, OR
Responsibilities:
- Provide support and maintenance for EOC Platform services and its eco-systems includes HDFS, Hive, Flume, Solr, Kafka, HBase.
- Involved in loading data from Oracle database into HDFS using Sqoop queries.
- Loading from disparate data sets using Hadoop stack of ingestion and workflow tools.
- Configure and implementation of Data Marts in Hadoop platform.
- Responsible for importing log files from various sources into HDFS using Flume.
- Imported data using Sqoop to load data from MySQL to HDFS on regular basis.
- Worked on shell scripting in Linux and the Cluster. Used shell scripts to run hive queries from beeline.
- Developed Scripts and automated data management from end to end and sync up between all the clusters.
- Worked with Hue GUI in scheduling jobs with ease and File browsing, Job browsing, meta-store management.
- Involved in creating Hive Tables, loading with data, and writing Hive queries which will invoke and run MapReduce jobs in the backend.
- Developed a detailed project plan and helped manage the data conversion migration from the legacy system to the target snowflake database.
- Design, develop, and test dimensionaldatamodels using Star andSnowflakeschemamethodologies under the Kimball method.
- Developed workflow in Oozie to automate the tasks of loading the data into HDFS and processing with Sqoop and Hive.
- Created Partitions, Buckets based on State to further process using Bucket based Hive joins.
- Loaded multiple NOSQL databases including MongoDB, PostgreSQL, HBase and Cassandra.
- Actively participated in software development lifecycle (scope, design, implement, deploy, test), including design and code reviews, test development, test automation.
- Used Zookeeper for providing coordinating services to the cluster.
- Worked on Kafka Streams which listens to multiple streaming data coming from several producers.
- Implementation of Backup data for Batch processing and for analysis in real time streaming applications.
- Performed loading and transformation of large sets of structured and semi structured data.
- Involved on migrating enterprise application data from Hadoop to AWS S3.
- Perform S3 buckets creation, policies and the IAM role based polices.
- Created monitors, alarms, and notifications for EC2 hosts using Cloud Watch.
- Created Notebook Instances using SageMaker to load the data from S3.
- Used Pyspark to process and analyze the data.
- Worked on Terraform to deploy the applications to AWS.
- Secured API gateway using IAM role and Lambda Authorizer.
- Used Flume to handle streaming data from Kafka and loaded into HBase.
- Involved in complete Data flow of the application starting from data ingestion from different data sources, processing the data into Hadoop EOC Platform.
- Worked on writing Workflow jobs using Oozie and set up multiple programs scheduler system which helped in managing multiple Hadoop jobs.
- Using Sqoop exported Hive external output processed data into MySQL.
- Worked on HBase Indexer which provides the ability to stream events from HBase to Solr for near real time searching.
- Used different file formats like text files, sequence files and Avro.
- Developed the Web services using spring and spring boot.
- Write build scripts to build applications using tools like maven and deploy the code using Jenkins for CI/CD.
- Deployed spring boot applications using PCF.
Environment: Hadoop Framework, HDFS, Hive, Sqoop, Kafka, Oozie, MySQL, Linux, Hue, HDFS, MapReduce, Terraform, Python, Spark, Kafka, Flume, Snowflake, HBase, Active Directory, Maven, Jenkins, Shell Scripting, Spring, Spring Boot, Microservices, AWS, EC2, S3, Lambda, API, Git.
Cloud Data Engineer
Confidential, Oakland CA
Responsibilities:
- Analysed large and critical datasets using Cloudera, HDFS, HBase, MapReduce, Hive, Hive UDF, Pig, Sqoop, Zookeeper and Spark.
- Collaborate with technology leads and architects in conceptualizing objectives and develop new Hadoop applications to process large datasets in agile environment.
- Responsible for designing, implementing and testing data pipelines on the cloud using AWS Services.
- Extensively used Spark to read data from S3 and preprocess it and to store in back S3 again for creating tables using Athena
- Designed and developed Spark application to read data json data from REST API’s.
- Extensively used EMR, S3, Data pipeline and Step Functions for building data pipelines
- Created partitioned tables in Athena, also designed adatawarehouse using Athena external tables and created Athena queries for analysis.
- Designed both 3NF data models for OLTP systems and dimensional data models using star and snowflake Schemas.
- Create Hive external tables for consumption and store data in HDFS with ORC, Parquet and Avro file formats.
- Implement Sqoop scripts and create jobs to import data from RDBMS (Oracle, SQL) into HBase tables using incremental and full Refresh imports.
- Installed and configured apache airflow for workflow management and created workflows in python.
- Develop python framework to convert Hive/SQL queries into Spark transformations using Spark RDDs, Spark SQL and Datasets and perform aggregations on data stored in memory.
- Design data warehouse using Hive external tables for Job Control & Data Quality to track job history and successful completion of production jobs.
- Develop Spark Streaming frameworks to read data from Kafka topics and Messages Queues to write to Kudu and S3.
- Responsible for data extraction and data ingestion from different data sources into HDFS and S3 by creating ETL pipelines.
- Resolve Spark and Yarn resource management issues in Spark including Shuffle issues, Out of Memory issues, heap space errors and schema compatibility.
- Monitor and troubleshoot performance of applications and take corrective actions in-case of failures and evaluate possible enhancements to meet SLAs.
- Experience in fact dimensional modeling (Star schema, Snowflake schema), transactional modeling and SCD (Slowly changing dimension)
- Designed and implemented MapReduce based large-scale parallel relation-learning system.
- Developed custom aggregate functions usingSparkSQL and performed interactive querying.
- Worked on Spark using Scala and Spark SQL for faster testing and processing of data.
- Built Spark Scripts by utilizing Scala shell commands depending on the requirement.
- Worked with Senior Engineer on configuring Kafka for streaming data
- Imported data into HDFS from various SQL databases and files using Sqoop and from streaming systems using Storm into Big Data Lake.
- Involved in scripting (python and shell) to provision and spin up virtualized Hadoop clusters
- Worked with NoSQL databases like HBase to create tables and store the data Collected and aggregated large amounts of log data using Apache Flume and staged data in HDFS for further analysis.
- Wrote Pig scripts to store the data into HBase
- Collect all the logs from source systems into HDFS using Kafka and perform analytics on it.
- Exported the analyzed data to Teradata using Sqoop for visualization and to generate reports for the BI team.
- Extracted files from RDBMS through Sqoop and placed in HDFS and processed.
- Spark Streaming collects this data from Kafka in near-real-time and performs necessary transformations and aggregation on the fly to build the common learner data model and persists the data in NoSQL store (HBase).
- Installed and configured Hive and also written Hive UDFs.
- Loaded data into the cluster from dynamically generated files using FLUME and from RDBMS using Sqoop.
- Developed and ConfiguredKafka brokersto pipeline server logs data into spark streaming
- Sound knowledge in programming Spark using Scala.
- Involved in writing Flume and Hive scripts to extract, transform and load data into Database
- Used HBase as the data storage
- Installed and configured Hadoop Map Reduce, HDFS, developed multiple Map Reduce jobs data cleaning and pre-processing.
- Create interface to convert mainframe data into ASCII.
- Experienced in installing, configuring and using Hadoop Ecosystem components.
- Experienced in Importing and exporting data into HDFS and Hive using Sqoop.
- Installed and configured Hive and also written Hive UDFs and Used Map Reduce and Junit for unit testing.
- Experienced in working with various kinds of data sources such as Teradata and Oracle.
- Successfully loaded files to HDFS from Teradata, and load loaded from HDFS to HIVE and IMPALA.
- Performed POCs on Spark test environment
- Worked on Spark using Scala and Spark SQL for faster testing and processing of data.
- Started using Apache NiFi to copy the data from local file system to HDFS.
- Load and transform large sets of structured, semi structured and unstructured data using Hadoop/Big Data concepts.
- Installed Oozie workflow engine to run multiple Hive and Pig jobs which run independently with time and data availability.
- Developed MapReduce programs to parse the raw data, populate staging tables and store the refined detain partitioned tables in the EDW.
- Developed python code for different tasks, dependencies, SLA watcher and time sensor for each job for workflow management and automation using Airflow tool.
- Monitored and managed the Hadoop cluster using Apache Ambary
- Experience in Administering Installation configuration troubleshooting Security Backup Performance Monitoring and Fine-tuning of Linux Red hat.
- Involved in implementing security on Hortonworks Hadoop Cluster using with Kerberos by working along with operations team to move non-secured cluster to secured cluster.
- Implemented Partitioning, Dynamic Partitions and Buckets in Hive for efficient data access.
Environment: Hadoop, Hive, Pig, Sqoop, Scala, Flume, Airflow, HBase, Athena, EMR, S3, Data pipeline, Step Functions, Sqoop, Spark, Scala, Linux, SQL Server, Data Warehouse and Tableau. Hortonworks, Oracle 10g/11g/12C, Teradata, Cassandra, HDFS, Kafka, Data Lake, Spark, MapReduce, Ambari, Cloudera, Tableau, AWS, Maven, Snappy, Zookeeper, NoSQL, Shell Scripting, Ubuntu, Solar.
Big Data Developer
Confidential
Responsibilities:
- Hands-on experience developing application leveraging Hadoop Ecosystem components (Hadoop, MapReduce, Spark, Pig, Hive and Sqoop).
- Worked extensively on importing data using Sqoop and flume.
- Responsible for creating complex tables using hive and developing Hive queries for the analysts.
- Created partitioned tables in Hive for best performance and faster querying.
- Transportation of data to HBase using pig.
- Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.
- Experience with professional software engineering practices and best practices for the full software development life cycle including coding standards, code reviews, source control management and build processes.
- Worked collaboratively with all levels of business stakeholders to architect, implement and test Big Data based analytical solution from disparate sources
- Involved in source system analysis, data analysis, data modeling to ETL
- Written multiple MapReduce procedures to power data for extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV& other compressed file formats.
- Handling structured and unstructured data and applying ETL processes.
- Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS
- Developed the Pig UDF'S to pre-process the data for analysis.
- Prepare Developer (Unit) Test cases and execute Developer Testing
- Create/Modify shell scripts for scheduling various data cleansing scripts and ETL loading process.
- Supports and assist QA Engineers in understanding, testing, and troubleshooting.
- Written build scripts using ant and participated in the deployment of one or more production systems
- Production Rollout Support which includes monitoring the solution post go-live and resolving any issues that are discovered by the client and client services teams.
- Designed, documented operational problems by following standards and procedures using a software reporting tool JIRA.
- Involved in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analyzed them by running Hive queries and Pig scripts.
- Assisted in Cluster maintenance, Cluster Monitoring and Troubleshooting, Manage and review data backups and log files
- Participated in requirement gathering form the Experts and Business Partners and converting the requirements into technical specifications
Environment: Hadoop, HDFS, Map Reduce, Hive, Sqoop, Kafka, Scala, Pig, Nifi, Hortonworks, Hbase, Spark, Oozie, Cassandra, Python, Shell Scripting, AWS, AWS EMR, EC2. Hadoop, Apache, Sqoop, Hive, Oozie, (jdk1.6), Flat files, Oracle 11g/10g, MySQL, Windows NT, UNIX, Zookeeper, Cloudera, FLUME, CentOS, Maven.
Java Developer
Confidential
Responsibilities:
- Developed the Web Based Rich Internet Application (RIA) using J2EE (Spring framework) and Macromedia/Adobe Flex.
- Extensively worked in designing and implementation of multi-tier applications using JAVA 8, J2EE, JDBC, JSP, HTML5, Spring Boot, Servlets, JavaBeans with Eclipse.
- Used new Java 8 features, such as Lambda expressions, Streams and Functional Interfaces.
- Designed and developed using the Java collection API with Java 8 features and other java processes to fulfill business case requirement, such clients account statements and clients running balance based on each transaction.
- Involved in the design and development of the entire application.
- CreatedUML diagrams(use case, class, sequence, and collaboration) based on the business requirements
- Involved in designing and developing dynamic web pages usingHTMLandJSP withStruts tag libraries
- ImplementedPL/SQLqueries and usedOracle stored procedures, and built-in functions to retrieve and update data from the databases
- UsedHibernateFramework for object relational mapping and persistence.
- Interfaced with the Oracle back-end usingHibernateFramework and XML config files.
- UsedEclipsefor the development environment
- WroteJUnittest cases to test the functionality of each method in theDAOlayer.
- Prepared technical reports and documentation manuals for efficient program development.
Environment: JSP, Servlets, Struts Framework, Struts 1.2, Struts Validator Framework, XML,Hibernate,HQL, Oracle9i, PL/SQL, WebSphere Application server, Eclipse,JBoss Application Server,JUnit, CVS, Rational Clear Quest, Unix. Apache Hadoop, HDFS, MapReduce, Sqoop, Flume, Pig, Hive, HBase, Oozie, Scala, Spark, AWS, Linux.
