Big Data Developer (aws And Cloud Environment) Resume
Bloomington, IllinoiS
PROFESSIONAL SUMMARY:
- Around 6 years of professional IT experience in BigData or HadoopDevelopment and DataAnalytics, ETLDevelopment, Design applications and ETL Reporting tools.
- Have extensively 4 years of work experience on Hadoop Ecosystem, Big Data andSpark in the Software Industry.
- Seeking a challenging position in Softwaredevelopmentindustry that needs innovation, creativity and dedication and enable me to continue to work in a demanding and fast paced environment, leveraging my current knowledge and fostering creativity with many learning opportunities.
- Worked with Hadoop developmentDistributions like Cloudera (CDH4&CDH5),Hortonworks and Amazon Web Services (AWS) in testing big data solutions & alsofor monitoring and managing Hadoop clusters.
- Expertise in installation, configuring and administering clusters ofmajor Hadoop distributions and Hadoop ecosystem components which includesHadoop Frameworks (YARN), Map Reduce, Pig, Hive, Sqoop, H - base, Apache Cassandra, Flume, Kafka, Impala, NiFi, Oozie, Storm& Zookeeper.
- Good knowledge in managing & reviewing Hadooplogfiles and understanding of HDFSDesigns, Daemons and HDFShighavailability (HA).
- Expertise in data warehousing/Data Mart Design, ETL architecture, OLAP, BI, Client/Server applications& also experienced in writing complex SQLQueries like Stored Procedures, triggers, joins and Sub queries.
- Expert in writing Hive, Pig, Sqoop, H-base, Pig Latin, HiveQL, HDFS, Spark&Map Reduce jobs.
- Working with Hive data warehouse tool-creating tables &internal/external Tables/Views, datadistributionby implementing partitioning, bucketing, writing andoptimizing the HiveQL queries usedfor visualization ofbusiness requirement.
- Experience in using Apache Sqoop to import and export data to HDFS and Hive fromRDBMS.
- Experience with Oozie workflow engine to run multiple Hive and Pig jobs independently.
- Used No-SQL databases likeApacheCassandra and H-base for data extraction, analytical operations and storing large volume of data.
- Hands on experience in working with Spark-SQL queries, import data from various Data sources, perform transformations, perform read/write operations, save the results to output directory into HDFS.
- Active participation in tomcat server and Oracle problems (killing instances, debugging server logs& applications logs).
- Experience and Domain knowledge in various industries such as Technology, Insurance, Retail, Health Care, Telecom, Banking andMedia.
- Ability to work with managers and executives to understand the business objectives and deliver as per the businessneeds and a good co-ordinator in team work.
- Ability to perform at a high level, meet deadlines, adaptable to ever changing priorities. Strong Communication skills of written, oral, interpersonal, Problem-Solving, analyticaland presentation.
TECHNICAL SKILLS:
Hadoop Ecosystem: HDFS, Map Reduce, Hive, Pig, Sqoop, Flume, Oozie, Impala, Kafka, H-Base, Zookeeper, NiFi, Strom, Tez, Cloudera Manager, Hortonworks, Spark and Scala.
Spark Components: Spark Core, Spark SQL (Data Frames and Dataset), Scala, Spark- Kafka Streaming.
Programming Languages: SQL, Spark, Scala, Shell, Pig Latin, Hive-QL& Unix/Linux.
Databases: Oracle 12c, 11g, 10g, 9i, Teradata, MySQL, SQL Server 2014/2012, HBase 1.2, NOSQL, Apache Cassandra,ORACE DB2, SQLite, RDBMS, MongoDB 3.2& Netezza.
Database Tools: PL/SQL Developer, Toad, SQL Loader& Erwin.
ETL Tools: Informatica 9.1/9.5, Informatica Power Center, Informatica Power Exchange,Talend, DataStage, Mainframes, IBM AS400, Erwin and Putty.
Reporting Tools: Business Objects, Tableau, Crystal Reports and Excel.
Data Visualization Tools: Qlickview, Tableau9.4/9.2, ggplot2 (R), Apache Zeeplin.
Bigdata Framework: HDFS, MapReduce, Hive, Pig, Sqoop, Oozie, Zookeeper, Flume, HBase, Amazon EC2,( S3 and Red Shift), cloudera (CDH4 & CDH5),Spark, Storm, Impala,Kafka.
GUI Technologies/Tools: HTML, XML, XHTML, CSS, JavaScript and Angular JS.
Operating Systems: LINUX, WINDOWS XP, 10, MS: DOS, UNIX.
Environment: AWS, AZURE, VMWare, Cloud Computing, Cloudera 4.x, Red Hat Linux, Sandbox, Hortonworks etc.
Development Methodologies: Agile, V: Model, Waterfall Model& Scrum.
Web/Application Servers: Web logic 8.0, Web Sphere, JBoss, Maven, Ansible, Puppet& Apache Tomcat.
DWH BI Tools: Data Stage 9.1/ 11.3, Tableau, power BI.
Office Tools: MSOFFICE: Excel, Word and PowerPoint.
Modeling types & tools: Dimensional data modeling, star schema modeling, snow - flake Schema modeling, E-R modeling& Visio.
PROFESSIONAL EXPERIENCE:
Confidential, Bloomington, Illinois
Big Data Developer (AWS and Cloud Environment)
Responsibilities:
- Examined transaction data, identified outliers, inconsistencies and manipulated data to ensure data quality and integration.
- Capturing data from existing databases that provide SQL interfaces using Sqoop and processing stream data using Kafka, Storm, Spark Streaming and Flume.
- Used ApacheFlume for collection, aggregation, moving large amount of data from application server.
- Developed data pipeline using Sqoop, Spark and Hive to ingest, transform and analyse operational data.
- Used Spark SQL with Scala for creating dataframes and performed transformations on data frames.
- Developed Shell and Perl Scripts (Linux amp; UNIX) to execute jobs.
- Good conceptual understanding and experience in cloud computing applications using Amazon EC2, S3, EMR and Amazon Redshiftplatforms.
- Experience in managing multi-tenant Hadoop clusters on public cloud environment - Amazon Web Services (AWS)-EC2.
- Loading data intoSparkRDD& also in Memory DataComputation to generate the Output response.
- Developed performance utilization chartsin-order to perform certain operations as per the business requirements.
- Working closely with Cassandra loading activity on history load and incremental loads from Teradata and Oracle Databases and resolving loading issues & then tuning the loader for optimal performance.
- Wrote Spark-Streaming applications to consume the data from Kafka topics and written the processed streams to HBase.
- Experience in using Apache Sqoopto import and export data to HDFS and Hive fromRDBMS.
- Experience with Oozie workflow engine to run multiple Hive and Pig jobs independently.
- Used No-SQL databases like ApacheCassandra and H-base for data extraction.
- Compiled and build the application using MAVEN and used SVN as version control system.
- Created Partitioned tables & Bucketed tables in Hive.
- Involved in gathering requirements from client and estimating time line for developing complex queries using HIVE and IMPALA for logistics application.
- Prepared technicalreports&documentationmanuals during the program development.
- Agilemethodology was used for development using XP Practices (TDD, Continuous Integration).
- Participated in daily scrummeetings and iterative development.
- Worked using the tools of JIRA and Jenkins within the project & Migrated Hadoop Metadata to Docker Container.
Environment: Hadoop YARN, Spark-Core, Spark Streaming, Spark SQL, Scala, Kafka, Hive, Sqoop, Amazon AWS, HBase, SQL, Maven, Teradata, Talend, Power Center,Perl, storm, Tableau, Apache Flume, SVN, Oozie, Zookeeper, Impala, Oracle 12c/11g, Windows 10,Cassandra, Azure, Amazon-Redshift& Linux/UNIX.
Confidential, Boston, Massachusetts
Big Data Developer/ Big Data Engineer
Responsibilities:
- Converted existing Map Reduce jobs into Sparktransformations and actions using SparkData Frames and SparkSQLAPI’s.
- Developed Sparkprograms for Real time processing.
- Written new Sparkjobs to analyse the data of the customers and sales history.
- Developed and created UserDefinedFunctions and Views.
- Worked on SparkSQL and SparkStreaming.
- Creating end-to-endSpark-Solrapplications using Spark to perform various data cleansing, validation, transformation and summarization activities according to the requirement.
- Developed Sparkscripts by using Scala shell commands as per the requirement.
- Used No-SQL databases like ApacheCassandra and H-base for data extraction, analytical operations and storing large volume of data.
- Hands on experience in working with Spark-SQL queries, import data from various Data sources, perform transformations, perform read/write operations, save the results to output directory into HDFS.
- Active participation in tomcat server and Oracle problems (killing instances, debugging server logs & applications logs).
- Efficiently transferring bulk data between Hadoop Distributed File Systemand structured data stores such as relational databases using ApacheSqoop.
- Developed PIGscripts to transform the data into structuredformat.
- Created Hivetables and working on them using HiveQL. Moreover, Created tables, views in Teradata, according to the requirements.
- Building, publishing customized reports and dashboards, report scheduling using Tableauserver.
- Created action filters, parameters and calculated sets for preparing dashboards and worksheets in Tableau.
- Exported the analysed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
- Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
- Involved in Resolving technical issues during development, deployment and support.
Environment: Hadoop YARN, Spark-Core, Spark Streaming, Spark SQL, Scala, Kafka, Hive, Pig, Sqoop, Amazon Web Services (AWS), HBase, Teradata, Power Center, Tableau server, Oozie, Oracle, MySQL, RDBMS, Impala, JSON, XML, SQLServer, Talend, Cloudera& Linux.
Confidential, Newark, NJ
Hadoop Admin/ Developer
Responsibilities:
- Created HiveTables, loaded transactional data from Teradata using Sqoop.
- Loaded home mortgage data from the existing DWH tables (SQLServer) to HDFS using Sqoop.
- Responsible for Operating system and HadoopCluster monitoring using tools like Nagios and ClouderaManager.
- Worked on POC, implementation & integration of Cloudera&Hortonworks for multiple clients.
- Involved in maintaining clusters on different Hadoopdistributions and also maintained Hadoop administration on Cloudera, Hortonworks and ApacheHadoop1.x &2.x for multiple projects.
- Worked on analyzing Hadoop cluster using different big data analytic tools including Kafka, Pig, Hive, Sqoop and MapReduce.
- Collected and aggregated large amounts of log data using ApacheFlume and staging data in HDFS for further analysis.
- Realtimestreaming the data using Spark with Kafka.
- Configured Sparkstreaming to receive real time data from the Kafka and store the stream data to HDFS using Spark.
- Importing and exporting data into HDFS using Sqoop and Kafka.
- Wrote HiveQueries to have a consolidated view of the mortgage and bank transactional data.
- Loaded the load ready files from mainframes to Hadoop and files were converted to ASCII format.
- Supported team using Talend as ETL tool to transform and load the data from different databases.
- Exposure to burn-up, burn-down charts, dashboards, velocity reporting of sprint and release progress.
Environment: Apache Hadoop, Map Reduce, Cloudera, Hive, Pig Scripts, Kafka, Sqoop, Avro, JSON, Hortonworks, Data Meer, Teradata, SQL Server, IBM Mainframes, SVN, JIRA, Nagios, Talend,Agile, MongoDB, Apache Flume, Maven & Jenkins.
Confidential
ETL Developer
Responsibilities:
- Designed, developed and maintained projects, ETL architecture from multiple environmentsby working with various source systems and fetched data via Flat files, views, databaseconnections in Teradata, Oracle and mainframe etc.
- Created mappings in informatica, SQL loader using UNIX scripting to stage/validate databefore it is to be standardized as per the business requirements which will be leveraged toassemble contracts in later processes.
- Involved in DevelopingETL jobs usingData Stage to load the data into DWH/DM.
- Worked onETL process for bringing the data intoMDMlanding tables.
- Expert level skills on Source, mapping, device configuration, designer transformations,complex and simple mapplets, external call factor, lookups, UDF and several other power center designer objects.
- Created scheduler batches in AutoSys scheduler and automated jobs.
- Analyzed source system’s live data to identify the design fallouts and worked with analysts.
- Extensively used workflow variables, mapping parameters and mapping variables & slowlychanging dimensions’ logics.
- Used ETL methodology for supporting Data Extraction, transformations and loading processing, using Informatica BDE to Hive target in Hadoop.
- Implemented performance tuning logic on targets, sources, mappings and sessions to providemaximum efficiency and performance also by analyzing HIVEscripts that are generated during session runs.
- Written documentation to describe program development, logic, coding, testing, changes and corrections.
Environment: Teradata, Oracle, Mainframes, SQL, PL/SQL, Informatica 9.5, Hive Scripts, SQL Loader, Unix, Data Stage, Unit Testing, ETL Architecture, Informatica Power Center 9.x.
