Big Data Developer Resume
4.00/5 (Submit Your Rating)
Portland, OR
SUMMARY
- Highly analytical and solution driven big data developer with over 8+ years of experience in analysis, design, development, and maintenance activities. In depth noledge in different database types; research methodologies and big data capture. Enhance Insights, analytics and business intelligence for process reengineering and corporate growth.
- IT professional with 8+ years of experience in software design, development, deployment and maintenance of data analytic applications in fields of health, insurance, finance and telecommunications sectors.
- 6+ years of experience in building high performance Big Data applications primarily using Hadoop eco - system tools and Spark framework.
- Solid understanding of architecture of Hadoop framework eco-system components Map Reduce, Pig, Hive, HBase, Flume, Sqoop, Hue, Oozie, Spark and Kafka.
- Good handson experiencing working with various hadoop disrtibutions mainly Cloudera (CDH), Hortonworks (HDP), Azure and Amazon EMR.
- Expertise in developing production ready Spark applications utilizing Spark-Core, Data frames, Spark-SQL, Spark-ML and Spark-Streaming API's.
- Strong experience troubleshooting failures in spark applications and fine-tuning for better performance.
- Experience in using D-Streams in spark streaming, accumulators, Broadcast variables, various levels of caching and optimization techniques in spark.
- Strong experience working with both batch and real-time processing using Spark framework.
- Worked extensively on Hive for building complex data analytical applications.
- Deep understanding of performance tuning, partitioning for optimizing spark applications.
- Worked on building real time data workflows using Kafka, Spark streaming and HBase.
- Extensive noledge on NoSQL databases like HBase, Cassandra, KUDU and Mongo DB.
- Strong working experience in extracting, wrangling, ingestion, processing, storing, querying and Analyzing structured, semi-structured and unstructured data.
- Solid understanding of Hadoop MRV1 and Hadoop MRV2 (or) YARN Architecture.
- Solid experience in working with csv, text, sequential, Avro, parquet, orc, JSON formats of data.
- Created Talend Mappings to populate the data into dimensions and fact tables.
- Experience with RDBMS like SQL Server, MySQL, Oracle and data warehouses like Teradata and Netezza.
- Experienced in job workflow scheduling and monitoring tools like Oozie and Airflow
- Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse
- Proficient noledge and hands on experience in writing shell scripts in Linux.
- Expertise in complete Java Package, object oriented design.
- Developed core modules in large cross-platform applications using JAVA, JSP, Servlets, Hibernate, RESTful, JDBC, JavaScript, XML, and HTML.
- Extensive experience in developing and deploying applications using Web Logic, Apache Tomcat and JBOSS.
- Development experience with RDBMS, including writing SQL queries, views, stored procedure, triggers, etc.
- Strong understanding of Software Development Lifecycle (SDLC) and various methodologies (Waterfall, Agile).
TECHNICAL SKILLS
- Hadoop
- HDFS
- Map Reduce
- Hive
- Pig
- Sqoop
- Flume
- Oozie
- Hue
- Ambari
- Zookeeper
- Kafka
- Airflow
- HBase
- Nifi
- SQL Server
- Teradata
- Oracle
- AWS
- AZURE
- Scala
- Python
- Pig Latin
- Hive QL
- Java
- Python
- C language
- SSIS
- SSRS
PROFESSIONAL EXPERIENCE
Big Data Developer
Confidential - Portland, OR
Responsibilities:
- Built S3 buckets and managed policies for S3 buckets and used S3 bucket and Glacier for storage and backup on AWS .
- Work with other teams to halp develop the Puppet infrastructure to conform to various requirements including security and compliance of managed servers.
- Built a VPC, established the site-to- site VPN connection between Data Center and AWS .
- Develop push-button automation for app teams for deployments in multiple environments like Dev, QA, and Production.
- Help with the creation of dev standards for Puppet module development including best practices for version control ( git ).
- Perform troubleshooting and monitoring of the Linux server on AWS using Zabbix, Nagios and Splunk.
- Management and Administration of AWS Services CLI, EC2, VPC, S3, ELB Glacier, Route 53, Cloudtrail, IAM, and Trusted Advisor services.
- Created automated pipelines in AWS CodePipeline to deploy Docker containers in AWS ECS using services like CloudFormation, CodeBuild, CodeDeploy, S3 and puppet .
- Worked on JIRA for defect/issues logging & tracking and documented all my work using CONFLUENCE .
- Integrated services like GitHub, AWS CodePipeline, Jenkins and AWS Elastic Beanstalk to create a deployment pipeline.
- Good Experience in architecting and configuring secure cloud VPC using private and public networks through subnets in AWS.
- Amazon IAM service enabled to grant permissions and resources to users. Managed roles and permissions of users with the halp of AWS IAM .
- Designing and implementing fully automated server build management, monitoring and deployment by using Technologies like Puppet.
- Deployed Puppet, Puppet dashboard for configuration management to existing infrastructure.
- Initiating alarms in CloudWatch service for monitoring the server's performance, CPU Utilization, disk usage etc. to take recommended actions for better performance.
- Configured AWS Multi Factor Autantication in IAM to implement 2 step autantication of user's access using Google Autanticator and AWS Virtual MFA .
- Included security groups, network ACLs, Internet Gateways, and Elastic IP's to ensure a safe area for organization in AWS public cloud.
- Writing UNIX shell scripts to automate the jobs and scheduling cron jobs for job automation using commands with Crontab.
- Wrote Ansible Playbooks with PythonSSH as the Wrapper to Manage Configurations of AWS Nodes and Test Playbooks on AWS instances using Python .Experience with Puppet to more easily manage enterprise Puppet deployments
- Design AWS Cloud Formation templates to create custom sized VPC, subnets, NAT to ensure successful deployment of Web applications and database templates
- Created scripts in Python which integrated with Amazon API to control instance operations.
- Coordinate/assist developers with establishing and applying appropriate branching, labeling /naming conventions using GIT source control.
Data Engineer
Confidential - St.Paul, MN
Responsibilities:
- Analyze, design, and build Modern data solutions using Azure PaaS service to support visualization of data. Understand current Production state of application and determine the impact of new implementation on existing business processes.
- Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics . Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
- Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
- Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
- Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark databricks cluster.
- Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
- To meet specific business requirements wrote UDF's in Scala and Pyspark.
- Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the Sql Activity.
- Hands-on experience on developing SQL Scripts for automation purpose.
- Created Build and Release for multiple projects (modules) in production environment using Visual Studio Team Services (VSTS).
Big Data developer
Confidential - New York, NY
Responsibilities:
- Worked with respective business units in understanding the scope of the analytics requirements.
- Performed core ETL transformations in Spark.
- Automated data pipelines which involve data ingestion, data cleansing, data preparation and data analytics.
- Created end to end Spark applications using Python to perform various data cleansing, validation, transformation, and summarization activities on user behavioral data.
- Converted existing MapReduce jobs into Spark transformations and actions using Spark RDDs, Data frames and Spark SQL API
- Developed end-to-end data pipeline using FTP Adaptor, Spark, Hive, and Impala.
- Used Python to write code for all Spark use cases.
- Implemented design patterns in Scala for the application.
- Implemented Spark using Scala utilized Spark SQL heavily for faster development, and processing of data.
- Exploring with Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, YARN.
- Involved in converting Hive/SQL queries into Spark transformations using Spark SQL and Scala.
- Handled importing other enterprise data from different data sources into HDFS using Sqoop and performing transformations using Hive, Map Reduce and tan loading data into HBase tables.
- Exported the analyzed data to the relational databases using Sqoop, to further visualize and generate reports for the BI team.
- Collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis
- Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
- Created components like Hive UDFs for missing functionality in HIVE for analytics.
- Worked on various performance optimizations like using distributed cache for small datasets, Partition, Bucketing in Hive and Map Side joins.
- Created Oozie workflows and coordinators to automate data pipelines daily, weekly, and monthly.
Hadoop Developer
Confidential - Longwood, FL
Responsibilities:
- Worked with the business team to gather the requirements and participated in the Agile planning meetings to finalize the scope of each development.
- Responsible for building scalable distributed data solutions on CDH.
- Developed multiple MapReduce jobs in Java for data cleaning and preprocessing.
- Implemented data pipelines developing multiple mappers by using Chained Mappers API.
- Developed multiple MapReduce batch jobs in java for loading the data to HDFS in sequential format.
- Ingested structured data from wide array of RDBMS to HDFS as incremental import using Sqoop.
- Involved in writing Pig scripts to wrangle the raw data and store it to HDFS, load the data to Hive tables using HCatalog.
- Configured Flume agents on different data sources to capture the streaming log data from the web servers.
- Implemented Flume (Multiplexing) to steam data from upstream pipes in to HDFS.
- Created Hive external tables with clustering and partitioning on the date for optimizing the performance of ad-hoc queries.
- Involved in writing Hive QL scripts on beeline, impala, hive cli for the consumer data analysis to meet business requirements.
- Exported data from Hive to DWH using Sqoop.
- Worked with different file formats and compression techniques to ensure optimal performance of hive queries.
- Involved in creating Hive tables from wide range of data formats like csv, text, sequential, Avro, parquet, orc, JSON and custom formats.
- Transformed the semi-structured log data to fit into the schema of the Hive tables using Pig.
- Involved in scheduling Oozie workflow engine to run multiple Hive and pig jobs.
- Involved in testing and designing low level and high-level documentation for the business requirement.
Java/ Hadoop Developer
Confidential
Responsibilities:
- Developed web applications by coordinating requirements, user stories, use cases, screen mockups, schedules, and activities.
- Work closely with client business stakeholders on Agile development teams.
- Support users by developing documentation and assistance tools.
- Developed presentation using Spring Framework and used multiple modules in Spring like, Spring MVC, JDBC
- Implemented Web-Services to integrate between different applications components using RESTful using Jersey.
- Developed RESTful Web services for transmission of data in JSON/XML format.
- Involved in writing SQL queries, functions, views, triggers, and stored procedures and also using Oracle relational database.
- Used Sqoop to ingest structured data from Oracle database to HDFS.
- Involved in writing and running Map Reduce batch jobs using java for data wrangling on the cluster.
- Developed map side, reduce side joins using Distributed Cache on various data sets.
- Developed Pig Latin scripts to transform the data according to the business requirement.
- Developed Pig UDFs extending eval, filter functions using java to filter semi structured data.
Java Developer
Confidential
Responsibilities:
- Involved in various phases of Software Development Life Cycle (SDLC) of the application like Requirement gathering, design, development and documentation.
- The application is designed using J2EE design patterns and technologies based on MVC architecture
- Involved in designing the user interfaces using HTML, CSS and JavaScript for client-side validation.
- Developed custom tags, JSTL to support custom User Interfaces.
- Handled business logic as a Model using the halper classes and Servlets to control the flow of application as controller as server-side validations.
- Involved in Servlets, Java Bean programming on the server side for the communication between clients and server.
- Involved in Implemented Web-Services to integrate between different applications (internal and third party components) using Restful services.
- Involved in writing unit testing for doing positive and negative test cases.
- Developed the maven scripts for preparing WAR files used to deploy J2EE components.
- Created tables, views, triggers, stored procedures on MySQL server for data manipulation and retrieval.
- Used JDBC to invoke Stored Procedures and for database connectivity to database server.
- Used Log4J to capture the log that includes runtime exceptions.
- Involved in Bug fixing and functionality enhancements.
- Developed the project using Waterfall model.
