Hadoop Developer Resume
Dallas, TX
SUMMARY:
- Around 6+ years of IT experience in a variety of industries, which includes hands on experience of 4 years in Big Data Analytics and development.
- Good Knowledge and experience of System Development Life Cycle (SDLC), product development methodologies, database design concepts and system integration strategies.
- Expertise with the tools in Hadoop (HDFS, YARN) Ecosystem including Pig, Hive, Map Reduce, Sqoop, Storm, Spark, Flume, Kafka, Oozie, Impala and Zookeeper and Machine Learning.
- Strong experience in Spark SQL UDFs, Spark SQL Performance, Spark Streaming, Performance Tuning . Experienced in working with input file formats like orc, parquet, Sequence, Json, Avro .
- Experienced in working with Amazon Web Services (AWS) using EMR for computing and S3 as storage.
- Excellent Java development skills using Java, J2EE, Servlets, JSP, EJB, JDBC, SOAP and RESTful web services.
- Worked in large and small teams for Systems requirement, design & development.
- Experienced in using agile approaches, including Extreme Programming, Test - Driven Development and Agile Scrum.
TECHNOLOGIES:
Big Data/Hadoop Technologies: HDFS, YARN, Map Reduce, Hive, Pig, Impala, Sqoop, Flume, Spark, Kafka, Zookeeper, Oozie, Elastic Search
Hadoop Distribution/Monitoring: Cloudera, Horton works, Ambari, Cloudera Manager
NO SQL Databases: HBase, Cassandra, MongoDB
Relational Databases: Microsoft SQL Server, MySQL, Oracle, DB2
Languages : Java, Scala, SQL, PL/SQL, C, C++, Shell Scripting, Python
Java & J2EE Technologies: Core Java, JSP, Servlets, JDBC, JNDI, Hibernate, Spring, Struts, JMS, EJB, Restful, SOAP
Web Technologies: HTML, CSS, XML, Java Script, JQuery
Application Servers: Web Logic, Web Sphere, JBoss, Tomcat
Cloud Computing Tools: Amazon AWS: EC2, S3, IAM, Glacier, CloudFront, EMR
Operating Systems: UNIX, Windows, LINUX
Build Tools: Jenkins, Maven, ANT
ETL Tools: Informatica, Talend
Visualization: Tableau
Development Tools: Eclipse, NetBeans, IntelliJ
Development Methodologies: Agile/Scrum, Waterfall
Version Tools and Testing API: Git, SVN and JUNIT
PROFESSIONAL EXPERIENCE:
Confidential, Dallas, TX
Hadoop Developer
Environment: Scala, Spark, Cloudera, HDFS, Hive, Impala, GitHub, Eclipse, Bamboo, Sqoop, TDCH, Teradata SQL assistant
Responsibilities:
- Worked with Care Evolution to parse clinical documents that come from various data sources in the form of CCDA’s (XML), HL7, ADT messages into SDO (Standard Data Output) files.
- Have written code in Scala using spark and ingested data across different layers in HDFS.
- Have used AVRO tables in Warehouse layer which enables to add/remove column names without affecting the history data.
- Have used parquet format and higher order functions to load the latest and greatest data into outbound layer for reporting purpose.
- Used Akka framework that enables concurrent processing while loading the data lake.
- Ingested data from Teradata to Hadoop using TDCH jobs for information.
- Developed spark code to generate monthly Eligibility extract for Anthem Eligible members.
- Developed code in Scala that identifies Line of Business for every patient having clinical record.
- Have improved job’s performance by using appropriate caching methodology and repartitioning principle.
- Modified all the Scala projects and shell to migrate from unencrypted to encrypted zone. Ensured that the jar/shell is compatible to run in any environment without any modification.
- Have used Control M for job set automation and GIT Version control.
- Developed shell scripts to notify IT team when there is a failure in spark job and maintained logs for job completion.
- Co-ordinated with Hadoop Admin Team, Control-M and ensured that changes have been installed successfully for every release. Responsible for doing post production validation and communicated stake holders.
Confidential, Littleton, CO
Senior Hadoop Developer
Environment: Spark core, Spark SQL, Spark streaming, Spark machine learning, Scala, S3, EMR, Kafka, Hive, Sqoop, GitHub, Web flow, Talend
Responsibilities:
- Load and transform large sets of Structured, Semi Structured and Unstructured data coming from different source systems and a variety of portfolios.
- Wrote Shell Scripts to move batch files from various sources to HDFS, S3 and scheduled using cron tab.
- Worked on incremental Batch Ingestion using Sqoop into HDFS.
- Implemented Streaming data Ingestion using Kafka.
- Used Spark Streaming as a consumer in Kafka for preprocessing real time data.
- Implemented partitions and buckets in Hive for optimizing query performance.
- Used Spark core API and Spark SQL in Scala as per business analysis needs.
- Used AWS S3 for data storage and EMR cluster for processing various jobs.
- Worked on Talend with Hadoop. Worked in migrating from Talend jobs.
- Using Spark Machine learning library to clean data and feed the input stream data to Trained Model.
- Migrated python scikit learn machine learning to data frame-based spark machine learning algorithms
- Worked with Oozie Workflow manager to schedule Hadoop jobs and highly intensive jobs
- Worked with team members in data visualization using Tableau.
Confidential, IL
Hadoop with Spark Developer
Environment: Hadoop-HDFS, Hive, Sqoop, Kafka, Spark, Shell Scripting, HBase, Python, Zookeeper, Maven, Hortonworks, MySQL, Tableau
Responsibilities:
- Experienced in running Hadoop streaming jobs to process terabytes of xml format data.
- Importing and exporting data into HDFS and Hive using Sqoop
- Created Linux shell Scripts to automate the daily ingestion of IVR data
- Good experience in handling data manipulation using python Scripts.
- Consumed the data from Kafka using Apache spark.
- Worked on POC’s with Apache Spark using Scala to implement spark in project.
- Implemented Partitioning, Dynamic Partitions, Buckets in Hive
- Extending Hive core functionality by using custom User Defined Function’s (UDF).
- Executed Hive queries on Parquet tables stored in Hive to perform data analysis to meet the business requirements.
- Responsible for loading data files from various external sources like Oracle, MySQL into staging area in MySQL databases.
- Worked in creating HBase tables to load large sets of semi structured data coming from various sources.
- Exported the aggregated data onto RDBMS using Sqoop for creating dashboards in the Tableau .
- Worked on Installation and configuring of Zookeeper to co-ordinate and monitor the cluster resources.
Confidential
Java Developer
Environment: Java/J2EE, Struts, Hibernate, Web Logic Server, Oracle, HTML, JavaScript, Ant, svn, JUnit
Responsibilities:
- Coordinating with Project Manager for getting the requirements and developing the code to support new applications.
- Developing Web applications using Java, J2EE, Struts and Hibernate.
- Developing Action Form classes, Form beans, and Action classes using struts.
- Using Hibernate for the backend persistence.
- Involved in writing Spring Configuration XML file that contains object declarations and dependencies.
- Implementing MVC, DAO J2EE design patterns as a part of application development.
- Designing the front end using JSP, HTML, CSS and JavaScript as per the requirements that are provided.
- Coding and maintaining Oracle packages, stored procedures and tables.
- Working on Web technologies including Tomcat, Apache, Http, Web service architectures.
- Using SVN for software configuration management and version control.
- Preparing test cases and strategies for unit testing and integration testing.
- Using the LOG4j to log regular Debug and Exception statements.
