We provide IT Staff Augmentation Services!

Sr. Big Data Developer Resume

4.00/5 (Submit Your Rating)

Charlotte, NC

SUMMARY

  • Around 5 years of overall experience in a variety of industries including 3+ years of experience in Big Data Technologies (Apache Hadoop stack and Apache Spark), 4+ years of experience in Java Technologies
  • Hands on experience with various Apache Hadoop Ecosystems such as Hadoop, Spark, HDFS, MapReduce, YARN, TEZ, HBase, Pig, Hive, Sqoop, Flume, Oozie, and Kafka
  • Hands on experience with Cloudera and Hortonworks
  • Strong knowledge of Hadoop Daemons such asHDFS, Job Tracker, Task Tracker, Name None, Data Node and Map Reduce concepts
  • Hands on experience in writing MapReduce jobs in Java and Python
  • Experience in dealing with SQL in Hadoop with Apache Hive
  • Hands on experience in writing Apache Spark SQL and Spark Streaming programming with Scala and Python
  • Designed HIVE queries & Pig scripts to perform data analysis, data transfer and table design
  • Expertise in writing Hive UDF, Generic UDF's to in corporate complex business logic into Hive Queries
  • Experienced in optimizing Hive queries by tuning configuration parameters
  • Implemented SQOOP for large dataset transfer between Hadoop and RDBMS
  • Experience in implementing Real - Time streaming and analytics using Spark Streaming and Kafka
  • Experience in data ingestion using Sqoop from RDBMS to HDFS and Hive and vice-versa
  • Performed data analysis using MySQL, SQL Server Management Studio and Oracle
  • Experience in working with Cloudera (CDH3 & CDH4&CDH5) and Horton works Hadoop Distributions
  • Hands on experience on AWS infrastructure services Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud (Amazon EC2)
  • Worked with Oozie and Zookeeper to manage the flow of jobs and coordination in the cluster
  • Experience in performance tuning, monitoring the Hadoop cluster by gathering and analyzing the existing infrastructure using Cloudera manager
  • Experience with configuration of Hadoop Ecosystem components: Map Reduce, Hive, HBase, Pig, Sqoop, Oozie, Zookeeper, Flume, Storm, Spark, Yarn, Tez
  • Experience with Restful Services and Amazon Web Services
  • Hands on Experience on Amazon’s EC2, EMR and S3
  • Experience in writing Maven and SBT scripts to build and deploy Java and Scala Applications

TECHNICAL SKILLS

Big Data Skills: Hadoop Stack, Spark, R, Hive, Sqoop, Pig, MapReduce, YARN, Flume

Distributions: Cloudera, Hortonworks, AWS EC2 and EMR

Programming languages: Java, Scala, Python, UML, Shell Scripting

File formats: Avro, Parquet, Fixed length, EBCDIC, Text file, XML, JSON, CSV

Compression Techniques: Gzip, LZO, Snappy and Bzip2

Tools: Maven, SBT, Eclipse, IntelliJ, Net beans, PyCharm, Pydev, R Studio, Shiny, Revolution R, Teradata SQL Assistance, TOAD, CA workload, PUTTY, Super PUTTY, MS-Office/Visio/Project, VS, SSMS and BIDS

Operating Systems: Linux, UNIX, Ubuntu, CentOS, Windows and Windows Server

Web: HTML, CSS, AngularJS, JavaScript, JSON and JQuery

Databases: SQL Server, Oracle, Netezza, DB2, Teradata, Exadata, and MySQL

Version Control: SVN, GitHub, TFS

Development Methodologies: Agile/Scrum, Waterfall

PROFESSIONAL EXPERIENCE

Continental, OK

Sr. Big Data Developer

Responsibilities:

  • Build a multi-purpose Hadoop based data management platform as A repository for relational sources spanning both internal and external data sources
  • Ingesting the data from different data sources to HDFS/Amazon S3 with Sqoop, hive and Oozie tools
  • Providing a SQL-like interface for accessing multi-source data from a single interface creating hive/impala/kudu tables to store the data with bounded schema and access the same tables with multiple sql-like interfaces
  • Experience in ingesting incremental updates from structured ERP systems residing on Microsoft SQL server database on to Hadoop data platform using Sqoop
  • Created HBase tables to store various data formats coming from different applications
  • Developed spark programming code in PYTHON on INTELLIJ IDE using SBT tools
  • Performance tuning of Sqoop, Hive and Spark jobs
  • Responsible for modification of ETL data load scripts, scheduling automated jobs and resolving production issues (if any) on time
  • Developed Oozie workflows to automate ETL process by scheduling multiple Sqoop, Hive and Spark jobs
  • Developing analytical software that performs change data capture on ingested sources
  • Design and implement the hashing algorithms in Apache Spark with Python to perform the change data capture
  • Capturing the historical evolution of data change on the ingested sources
  • Designing the patterns to perform the CDC on Cloudera's Hadoop/Spark distribution with Amazon S3 and AWS services
  • Capturing the historical as well as present data in Spark and store them in Hive/Impala/Kudu tables for future analysis
  • Creating master data to fix data quality and integration issues across multiple sources
  • Creating the reference tables with unique mapping in Hive/Impala/Kudu and make them available to access across the application
  • Developing data models as a source for multiple dashboards used by the business users
  • Generating the BI Visualization layer on top of the ETL layer and generating the reports with Spotfire

Environment: Hadoop Stack, HDFS, Sqoop, Hive, PySpark, ATSCALE, Oozie, Microsoft SQL server, NIFI, Kafka, Unix shell scripting, Python, Hue, Avro, Parquet, HBase, YARN, Amazon S3, UNIX Shell Scripting, CRON, Scala, Spark, Maven, SBT, IntelliJ, Tableau.

Confidential, Charlotte, NC

Hadoop/Spark Developer

Responsibilities:

  • Involved in complete Big Data flow of the application starting from data ingestion from upstream to HDFS, analyzing, processing & publishing the data for downstream
  • Created data ingestion pipeline to work with real time streaming data using Flume, Kafka, and Spark Streaming and publish the data to HDFS
  • Published the HDFS/Hive table data to external system using the custom Kafka Producer for continuous updates
  • Involved in developing Sqoop queries for moving data from RDBMS servers to Hadoop.
  • Developed Spark RDD transformations, actions, and Data Frame’s, case classes, Datasets for the required input data and performed the data transformations using Spark-Core
  • Worked on Converting Hive/SQL queries into Spark transformations using Spark RDD with Python and also worked with Spark Context, Spark-SQL, Data Frames, Pair RDD's, and Datasets
  • Created Oozie workflows to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as map-reduce, Hive, pig and Sqoop as well as system specific jobs
  • Imported from several relational databases to HDFS and exported data from HDFS to RDBMS using Sqoop
  • Created Avro & Parquet Hive tables with Snappy compression, loaded data and wrote Hive queries, which will invoke and run MapReduce tasks in the backend
  • Implemented partitioning, dynamic partition, indexing and buckets in Hive and also used custom SerDe’s in hive
  • Created a common workflow to convert from EBCDIC format to ASCII from the Mainframe sources to a delimited file in the Avro format to HDFS, also dealt with text file input formats
  • Developed UDF functions and implemented it in HIVE Queries
  • Created Impala views on top of Hive tables and accessed through HUE/TOAD
  • Worked on performance tuning of various processes/components in Hive, Map Reduce, RDD, Data Frames etc
  • Worked on Mainframe decommissioning - wrote COBOL equivalent programs in Python, Hive, Pig, Spark core, Hive on Spark SQL, and UNIX shell scripting.
  • Developed shell scripts for running HQL files
  • Created reusable components, managed the environments, code migration & maintenance using SVN and Jenkins, as part of System Engineering team

Environment: Cloudera CDH 5.x, HDFS, Spark, MapReduce, Pig, Hive, Impala, Sqoop, Oozie, Scala, Spark SQL, Unix shell scripting, Python, Hue, Avro, Parquet, Autosys, Teradata, DB2, SQL Server, MySQL, Netezza, SVN, Eclipse, Teradata SQL Assistance, TOAD, CA workload, Jenkins, Version One, Agile, PUTTY

Confidential, OH

Responsibilities:

  • Importing and exporting data from different relational databases into HDFS and Hive using Sqoop and performed transformations using MapReduce and Hive and exported back to RDBMS for visualization
  • Involved in creating Hive tables, and loading and analyzing data using hive queries
  • Involved in running Hadoop jobs for processing millions of records of text data
  • Developed multiple MapReduce jobs in java for data cleaning and preprocessing
  • Assisted in exporting analyzed data to relational databases using Sqoop
  • Experienced in analyzing data using HiveQL and custom MapReduce programs in Java
  • Involved in converting Hive/SQL queries into Spark transformations and actions using Spark SQL (RDDs and Dataframes) in Python
  • Implemented Spark SQL queries with Python for faster testing and processing of data
  • Implemented Spark Streaming POC to read real-time data from Kafka in parallel and processed in parallel and save the result as parquet format in Hive
  • Involved in scheduling Oozie workflow engine to run multiple Hive and pig jobs
  • Developed application using Eclipse and used build and deploy tool as Maven
  • Hands on experience inAWSCloud in variousAWSservices such asEC2, EMR, S3, and RDS
  • Ability to spin up differentAWSinstances including EC2-classic and EC2-VPC using Cloud Endure
  • Good in writing Shell scripts to monitor Hadoop job

Environment: Hadoop Stack, Java, Oracle 10g, Ubuntu, YARN, Cloudera, AWS, Teradata, UNIX Shell Scripting, Oozie, Python, Spark, pySpark, R, Maven, GIT, Jenkins, SBT, Zeppelin, Eclipse, IntelliJ, JIRA, PUTTY

Confidential

Data Analyst

Responsibilities:

  • Create interactive/dynamic dashboards of sales data using Tableau and checking Key Performing Indices
  • Develop product fact based summary to improve sourcing decisions
  • Analyzing various statistics of sales data using SQL and spreadsheets
  • Analyze the sales data and forecast the sales using Multiple Regression Analysis
  • Perform exploratory data analysis to uncover important trends and relationships
  • Optimize the production schedule according to the forecasted sales data and inventory
  • Explored the insurance claim data and found the patterns, groups and regions where claims are more and compared them using pie charts and bar graphs
  • Involved in designing and developing predictive model which predicts the false claims using the historical claims data with 85% accuracy using logistic regression, Decision Trees and compare their performances using ROC charts, performed this in Python
  • Designed a model to predict the potential claimant (who claims more than a specific amount) using company’s claims data using Logistic Regression, Decision Trees and Random Forest in R
  • Performed the quality assessment on claims data and improved the quality by missing data imputation, sampling, feature selection, dimensionality reduction and outlier detection. Performed these tasks in both R and Python

Confidential

JR Software Engineer

Responsibilities:

  • Work with team of developers on python applications for RISK management
  • Created SQL queries to pull data from the relational databases
  • Gathered business requirements and converted it into SQL stored procedures for database specific projects
  • Developed the DAO layer for the application using Spring Hibernate Template support
  • Developed UI using HTML, JavaScript, and JSP, and developed Business Logic and Interfacing components using Business Objects, XML, and JDBC.
  • Designed user-interface and checking validations using JavaScript.
  • Managed connectivity using JDBC for querying/inserting & data management including triggers and stored procedures.
  • Developed various EJBs for handling business logic and data manipulations from database.
  • Involved in design of JSP’s and Servlets for navigation among the modules.
  • Designed cascading style sheets and XML part of Order Entry Module & Product Search Module and did client side validations with java script.
  • Developed Tableau visualizations and dashboards using Tableau Desktop
  • Designed and developed data management system using MySQL
  • Wrote python scripts to parse XML documents and load the data in database
  • Expertise in writing Constraints, Indexes, Views, Stored Procedures, Cursors, Triggers and User Defined function
  • Created unit test/regression test framework for working/new code
  • Interfaced with third-party vendors to customize UI/UX solutions
  • Elegantly implemented page designs in standards-compliant dynamic XHTML and CSS

Environment: Python, Django, MySQL, Linux, HTML, XHTML, SVN, CSS, AJAX, Bugzilla, JavaScript, Apache Web Server

We'd love your feedback!