Bigdata Developer / Production Support Resume
Halifax, NS
SUMMARY
- 6+ years of extensive experience and knowledge in IT that includes 2+ experience in Big data ecosystem related technologies.
- Experience in Development, Implementation and testing in various application architectures, including HealthCare, Insurance, Banking, and Telecom.
- Good Knowledge and understanding of Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node and MapReduce programming paradigm.
- Experience installing, configuring and testing Hadoop Ecosystem components like Hadoop MapReduce, HDFS, Pig, Hive, HBase, Spark, Storm, Sqoop, Kafka, Flume, Oozie, Zookeeper and Ambari.
- Good Exposure PIG Scripts and Hive queries.
- Good knowledge on Hadoop cluster architecture and monitoring teh cluster.
- In - depth understanding of Object oriented programming, Data Structures and Algorithms.
- Experience in managing and reviewing Hadoop log files, have good understanding of NoSQL databases MongoDB and Cassandra.
- Experience in importing and exporting data using Sqoop from Relational Database Systems to HDFS and vice-versa.
- Collecting and aggregating large amount of Log data using Apache Flume and storing data in HDFS for further analysis.
- Job/workflow scheduling and monitoring tools like Oozie and Zookeeper.
- Experience in Object Oriented Analysis, Design (OOAD) and development of software using UML Methodology, good knowledge of J2EE design patterns and Core Java design patterns.
- Experience using different file formats like Parquet, ProtoBuff (Protocol Buffer), and Apache Avro, Sequence file, JSON, XML and Flat file.
- Configured and Maintained different topologies in Storm cluster and deployed them on regular basis.
- Experience on Hadoop clusters using major Hadoop distributions like Cloudera (CDH), Hortonworks (HDP) and MapR.
- Expertise in developing and coding web applications using Core Java, J2EE technologies - JSP, Servlets and JDBC.
- Hands on experience in creating various database objects like tables, functions, stored procedures and triggers using SQL, PL/SQL.
- Experience in application development using Java, RDBMS, and Linux shell scripting.
- Experience in all phase of SDLC like Requirement Analysis, Implementation and Maintenance, extensive experience with Agile and SCRUM.
- Experience working on Kafka cluster. Also have experience in working on Spark and Spark streaming.
- Experience in creating event processing data pipelines using Kafka and Spark Streaming.
- Experience in using version controls like Git, tortoise SVN and build tools like Maven.
TECHNICAL SKILLS
Hadoop/ Big Data: HDFS, MapReduce, Sqoop, Kafka, Zookeeper, Ambari, Oozie
Hadoop/ Data Access: Pig, Hive, Solr, Impala, Spark, HBase, Storm
Programming Languages: Java, Python, Shell Scripting, Scala, SQL, C, C++
Web Servers: Tomcat, and Apache
Databases: Oracle 11g, SQL Server, MySQL, NoSQL (Cassandra)
IDE: Sublime Text, IntelliJ, WebStorm, Eclipse, Notepad++
Web Technologies: HTML, XML, JSON, JavaScript
Operating Systems: Windows, Linux, and Mac
PROFESSIONAL EXPERIENCE
Confidential
Bigdata Developer / Production Support
Responsibilities:
- Involved in cluster planning, deployment and monitoring Hadoop cluster with Hortonworks on Redhat Linux, managing with Ambari.
- Used Hive to find correlations between customer’s browser logs in various features of site, analyzed them to build risk profile for such sites.
- Analyzed large data sets by running Hive queries and Pig scripts.
- Involved in creating Hive tables, loading and analyzing data using hive queries.
- Developed simple to complex MapReduce Jobs using Hive and Pig.
- Involved in running Hadoop jobs for processing millions of records of text data.
- Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.
- Responsible for performing extensive data summarization using Hive.
- Worked on RC files, Sequence files, Map side joins, bucketing, partitioning for hive performance enhancement and storage improvement.
- Extracted records from Oracle DB through Sqoop and placed in HDFS and processed into Impala DB.
- Experienced in running Hadoop streaming jobs to process terabytes of text, xml data.
- Involved in submitting and tracking Map Reduce jobs using Job Tracker.
- Knowledge on handling Hive queries using Spark SQL that integrate Spark environment.
- Implemented business logic by writing Pig UDFs in Java and used various UDFs from Piggybanks and other sources.
- Responsible in exporting analyzed data to relational databases using Sqoop.
- Implemented Daily Oozie coordination jobs that automate parallel tasks of loading teh data into HDFS and pre-processing with Pig using Oozie co-coordinator jobs.
- Responsible fortuning Hive and Pig scriptsto improve performance.
- Implemented unit tests with MRUnit and PIGUnit.
- Documented teh technical details of Hadoop cluster management and daily batch pipeline, which includes several jobs of MapReduce, Pig, Hive, Sqoop, Oozie and other scripts.
Environment: Hadoop, HDFS, Pig, Hive, MapReduce, Sqoop, Spark, Oozie, Linux, Grunt, Git/Stash, Jenkins, Java, Oracle SQL DB.
Confidential
Etl/Hadoop Developer
Responsibilities:
- Loaded teh customer profiles data, customer information, billing information onto HDFS using Sqoop and Flume.
- Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, Hive, Hbase and Sqoop.
- Involved in loading data from LINUX file system to HDFS.
- Exported teh analyzed data to teh relational databases using Sqoop for visualization and to generate reports for teh BI team.
- Responsible for importing data into Hive and HDFS using Sqoop.
- Implemented test scripts to support test driven development and continuous integration.
- Installed and configured Hadoop, MapReduce, HDFS (Hadoop Distributed File System).
- Installed and configured Hadoop MapReduce, HDFS, Developed multiple MapReduce jobs in java for data cleansing and preprocessing.
- Created Pig Latin scripts to sort, group, join and filter teh enterprise wise to get transformed data sets.
- Involved in creating Hive tables, loading with data and writing hive queries that will run internally in MapReduce way.
- Worked on tuning teh performance Pig scripts.
- Mentored analyst and test team for writing Hive Queries.
- Installed Oozie workflow engine to run multiple Mapreduce jobs.
- Loaded bulk amounts of data to Hbase using MapReduce.
- Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.
- Created pull request on Git repository for manager approval before merging teh code to teh master branch.
- Worked on aggressive planning and deadlines for completion of teh project.
Environment: Hadoop, HDFS, MapReduce, Hive, Pig, Sqoop, Storm, Linux, Java, Oozie, HBase.
Confidential, Halifax, NS
Consultant
Responsibilities:
- Participated in all phases including Requirement Analysis; Client Interaction; Design, Coding, Testing and Documentation.
- Involved in teh continuous enhancements and fixing of production problems.
- Design and development of UNIX Shell Scripts to handle pre-and post-session processes.
- Developed and scheduled various pre-and post-sessions commands and workflows for all mappings to load data from source files to target tables.
- Analyzed existing system and developed business documentation on changes required.
- Used Informatica Designer to Extract & Transform teh data from various source systems by incorporating various business rules. Also used different transformations, sessions and command tasks.
- Created mappings using different transformations like Aggregator, Expression, Stored Procedure, Filter, Joiner, Lookup, Router, SQL transform and Update Strategy.
- Worked with dynamic parameter file, mapping variable, Mapping parameters and variable functions like Setvariable, Countvariable, Setminvariable and Setmaxvariable.
- Tuned performance of Informatica session for large data files by increasing block size, data cache size, sequence buffer length and target based commit interval.
- Deployed reusable transformation objects such as mapplets to avoid duplication of metadata, reducing teh development time.
- Involved in business analysis and technical design sessions with business and technical members to develop requirements document, and ETL specifications.
- Provided support for testing team, UAT and production issues.
Environment: Informatica Power Center 9.0, Informatica Power Connect, Cognos Flat files, XML Files, Oracle 10g/9i, MS SQL Server 2000, Shell Programming, SQL * Loader, Toad, Excel and Unix scripting, Sun Solaris, Windows XP.
Confidential
Application Developer
Responsibilities:
- Designed web application user interface, interaction scenarios and navigation based on analysts, interpretations of requirement and use cases.
- Closely worked with business system analyst to understand teh requirements to ensure that right set of UI modules been built.
- Wrote application level code to perform client-side validation using jQuery and JavaScript.
- Involved in creating HTML 5 templates.
- Designed teh front-end applications, user interactive (UI) web pages using web technologies like HTML5, XHTML, and CSS3.
- Defined new validations through AngularJS for teh form field validation implemented through HTML5.
- Using advanced level of jQuery, AJAX, JavaScript, CSS and pure CSS.
- Maintain teh code base by frequent updates to revision control using SVN.
- Worked on CSS Background, CSS Positioning, CSS Text, CSS Border, Pseudo classes, Pseudo elements etc.
- Structured teh JavaScript code using jQuery.
- Performed unit testing and requested for code review.
- Identifying and fixing teh browser/performance related issues in teh application.
- Production deployment and validation support.
- Fixed teh post production issues in teh application.
Environment: HTML5, CSS3, JavaScript, AJAX, jQuery, XML, SQL, Python, Bootstrap, Tortoise SVN, Eclipse, EPIC Hyperspace.
Confidential
Programmer Analyst
Responsibilities:
- Developed artifacts including high-level object and data models, interaction diagrams and codes teh solution.
- Implemented java code for all layers of teh application using Struts2 and Hibernate Framework.
- Used Struts2 for developing business logic and presentation layer.
- Developed JSP's for user interface using Struts2 Tag Libraries and Core Tag Libraries.
- Used Java Script and its libraries like jQuery with JSP for User interface.
- Used JSON with Ajax for Data Interchange.
- Wrote JUnit for Unit Testing.
- Good programming experience in SQL Plus and PL/SQL. Wrote Stored Procedures, Functions, Cursors, Indexes, Sequences and Views.
- Good experience with build tool Ant for deployment to teh server and creating different views for releases.
- Involved in Integration testing, System Testing, Bug fixing and Maintenance.
- Hands of work with SVN version controlling system.
- Logger implementation for debugs, info and error messages.
- Responsible for Interfacing with teh business and give them status updates about teh progress of teh enhancements.
- Responsible for prioritizing defects with teh business users and ensuring that they are fixed in teh promised time.
Environment: Java, Struts, Hibernate, JSP, Tiles, HTML, Java script, JUnit, XML, XSL/XSLT, ANT, SVN, Oracle, Tomcat, Eclipse, Weblogic, Toad, Unix, Windows XP.