We provide IT Staff Augmentation Services!

Big Data Developer (aws And Cloud Environment) Resume

3.00/5 (Submit Your Rating)

Bloomington, IllinoiS

PROFESSIONAL SUMMARY:

  • Around 6 years of professional IT experience in BigData or HadoopDevelopment and DataAnalytics, ETLDevelopment, Design applications and ETL Reporting tools.
  • Have extensively 4 years of work experience on Hadoop Ecosystem, Big Data andSpark in the Software Industry.
  • Seeking a challenging position in Softwaredevelopmentindustry that needs innovation, creativity and dedication and enable me to continue to work in a demanding and fast paced environment, leveraging my current knowledge and fostering creativity with many learning opportunities.
  • Worked with Hadoop developmentDistributions like Cloudera (CDH4&CDH5),Hortonworks and Amazon Web Services (AWS) in testing big data solutions & alsofor monitoring and managing Hadoop clusters.
  • Expertise in installation, configuring and administering clusters ofmajor Hadoop distributions and Hadoop ecosystem components which includesHadoop Frameworks (YARN), Map Reduce, Pig, Hive, Sqoop, H - base, Apache Cassandra, Flume, Kafka, Impala, NiFi, Oozie, Storm& Zookeeper.
  • Good knowledge in managing & reviewing Hadooplogfiles and understanding of HDFSDesigns, Daemons and HDFShighavailability (HA).
  • Expertise in data warehousing/Data Mart Design, ETL architecture, OLAP, BI, Client/Server applications& also experienced in writing complex SQLQueries like Stored Procedures, triggers, joins and Sub queries.
  • Expert in writing Hive, Pig, Sqoop, H-base, Pig Latin, HiveQL, HDFS, Spark&Map Reduce jobs.
  • Working with Hive data warehouse tool-creating tables &internal/external Tables/Views, datadistributionby implementing partitioning, bucketing, writing andoptimizing the HiveQL queries usedfor visualization ofbusiness requirement.
  • Experience in using Apache Sqoop to import and export data to HDFS and Hive fromRDBMS.
  • Experience with Oozie workflow engine to run multiple Hive and Pig jobs independently.
  • Used No-SQL databases likeApacheCassandra and H-base for data extraction, analytical operations and storing large volume of data.
  • Hands on experience in working with Spark-SQL queries, import data from various Data sources, perform transformations, perform read/write operations, save the results to output directory into HDFS.
  • Active participation in tomcat server and Oracle problems (killing instances, debugging server logs& applications logs).
  • Experience and Domain knowledge in various industries such as Technology, Insurance, Retail, Health Care, Telecom, Banking andMedia.
  • Ability to work with managers and executives to understand the business objectives and deliver as per the businessneeds and a good co-ordinator in team work.
  • Ability to perform at a high level, meet deadlines, adaptable to ever changing priorities. Strong Communication skills of written, oral, interpersonal, Problem-Solving, analyticaland presentation.

TECHNICAL SKILLS:

Hadoop Ecosystem: HDFS, Map Reduce, Hive, Pig, Sqoop, Flume, Oozie, Impala, Kafka, H-Base, Zookeeper, NiFi, Strom, Tez, Cloudera Manager, Hortonworks, Spark and Scala.

Spark Components: Spark Core, Spark SQL (Data Frames and Dataset), Scala, Spark- Kafka Streaming.

Programming Languages: SQL, Spark, Scala, Shell, Pig Latin, Hive-QL& Unix/Linux.

Databases: Oracle 12c, 11g, 10g, 9i, Teradata, MySQL, SQL Server 2014/2012, HBase 1.2, NOSQL, Apache Cassandra,ORACE DB2, SQLite, RDBMS, MongoDB 3.2& Netezza.

Database Tools: PL/SQL Developer, Toad, SQL Loader& Erwin.

ETL Tools: Informatica 9.1/9.5, Informatica Power Center, Informatica Power Exchange,Talend, DataStage, Mainframes, IBM AS400, Erwin and Putty.

Reporting Tools: Business Objects, Tableau, Crystal Reports and Excel.

Data Visualization Tools: Qlickview, Tableau9.4/9.2, ggplot2 (R), Apache Zeeplin.

Bigdata Framework: HDFS, MapReduce, Hive, Pig, Sqoop, Oozie, Zookeeper, Flume, HBase, Amazon EC2,( S3 and Red Shift), cloudera (CDH4 & CDH5),Spark, Storm, Impala,Kafka.

GUI Technologies/Tools: HTML, XML, XHTML, CSS, JavaScript and Angular JS.

Operating Systems: LINUX, WINDOWS XP, 10, MS: DOS, UNIX.

Environment: AWS, AZURE, VMWare, Cloud Computing, Cloudera 4.x, Red Hat Linux, Sandbox, Hortonworks etc.

Development Methodologies: Agile, V: Model, Waterfall Model& Scrum.

Web/Application Servers: Web logic 8.0, Web Sphere, JBoss, Maven, Ansible, Puppet& Apache Tomcat.

DWH BI Tools: Data Stage 9.1/ 11.3, Tableau, power BI.

Office Tools: MSOFFICE: Excel, Word and PowerPoint.

Modeling types & tools: Dimensional data modeling, star schema modeling, snow - flake Schema modeling, E-R modeling& Visio.

PROFESSIONAL EXPERIENCE:

Confidential, Bloomington, Illinois

Big Data Developer (AWS and Cloud Environment)

Responsibilities:

  • Examined transaction data, identified outliers, inconsistencies and manipulated data to ensure data quality and integration.
  • Capturing data from existing databases that provide SQL interfaces using Sqoop and processing stream data using Kafka, Storm, Spark Streaming and Flume.
  • Used ApacheFlume for collection, aggregation, moving large amount of data from application server.
  • Developed data pipeline using Sqoop, Spark and Hive to ingest, transform and analyse operational data.
  • Used Spark SQL with Scala for creating dataframes and performed transformations on data frames.
  • Developed Shell and Perl Scripts (Linux amp; UNIX) to execute jobs.
  • Good conceptual understanding and experience in cloud computing applications using Amazon EC2, S3, EMR and Amazon Redshiftplatforms.
  • Experience in managing multi-tenant Hadoop clusters on public cloud environment - Amazon Web Services (AWS)-EC2.
  • Loading data intoSparkRDD& also in Memory DataComputation to generate the Output response.
  • Developed performance utilization chartsin-order to perform certain operations as per the business requirements.
  • Working closely with Cassandra loading activity on history load and incremental loads from Teradata and Oracle Databases and resolving loading issues & then tuning the loader for optimal performance.
  • Wrote Spark-Streaming applications to consume the data from Kafka topics and written the processed streams to HBase.
  • Experience in using Apache Sqoopto import and export data to HDFS and Hive fromRDBMS.
  • Experience with Oozie workflow engine to run multiple Hive and Pig jobs independently.
  • Used No-SQL databases like ApacheCassandra and H-base for data extraction.
  • Compiled and build the application using MAVEN and used SVN as version control system.
  • Created Partitioned tables & Bucketed tables in Hive.
  • Involved in gathering requirements from client and estimating time line for developing complex queries using HIVE and IMPALA for logistics application.
  • Prepared technicalreports&documentationmanuals during the program development.
  • Agilemethodology was used for development using XP Practices (TDD, Continuous Integration).
  • Participated in daily scrummeetings and iterative development.
  • Worked using the tools of JIRA and Jenkins within the project & Migrated Hadoop Metadata to Docker Container.

Environment: Hadoop YARN, Spark-Core, Spark Streaming, Spark SQL, Scala, Kafka, Hive, Sqoop, Amazon AWS, HBase, SQL, Maven, Teradata, Talend, Power Center,Perl, storm, Tableau, Apache Flume, SVN, Oozie, Zookeeper, Impala, Oracle 12c/11g, Windows 10,Cassandra, Azure, Amazon-Redshift& Linux/UNIX.

Confidential, Boston, Massachusetts

Big Data Developer/ Big Data Engineer

Responsibilities:

  • Converted existing Map Reduce jobs into Sparktransformations and actions using SparkData Frames and SparkSQLAPI’s.
  • Developed Sparkprograms for Real time processing.
  • Written new Sparkjobs to analyse the data of the customers and sales history.
  • Developed and created UserDefinedFunctions and Views.
  • Worked on SparkSQL and SparkStreaming.
  • Creating end-to-endSpark-Solrapplications using Spark to perform various data cleansing, validation, transformation and summarization activities according to the requirement.
  • Developed Sparkscripts by using Scala shell commands as per the requirement.
  • Used No-SQL databases like ApacheCassandra and H-base for data extraction, analytical operations and storing large volume of data.
  • Hands on experience in working with Spark-SQL queries, import data from various Data sources, perform transformations, perform read/write operations, save the results to output directory into HDFS.
  • Active participation in tomcat server and Oracle problems (killing instances, debugging server logs & applications logs).
  • Efficiently transferring bulk data between Hadoop Distributed File Systemand structured data stores such as relational databases using ApacheSqoop.
  • Developed PIGscripts to transform the data into structuredformat.
  • Created Hivetables and working on them using HiveQL. Moreover, Created tables, views in Teradata, according to the requirements.
  • Building, publishing customized reports and dashboards, report scheduling using Tableauserver.
  • Created action filters, parameters and calculated sets for preparing dashboards and worksheets in Tableau.
  • Exported the analysed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
  • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
  • Involved in Resolving technical issues during development, deployment and support.

Environment: Hadoop YARN, Spark-Core, Spark Streaming, Spark SQL, Scala, Kafka, Hive, Pig, Sqoop, Amazon Web Services (AWS), HBase, Teradata, Power Center, Tableau server, Oozie, Oracle, MySQL, RDBMS, Impala, JSON, XML, SQLServer, Talend, Cloudera& Linux.

Confidential, Newark, NJ

Hadoop Admin/ Developer

Responsibilities:

  • Created HiveTables, loaded transactional data from Teradata using Sqoop.
  • Loaded home mortgage data from the existing DWH tables (SQLServer) to HDFS using Sqoop.
  • Responsible for Operating system and HadoopCluster monitoring using tools like Nagios and ClouderaManager.
  • Worked on POC, implementation & integration of Cloudera&Hortonworks for multiple clients.
  • Involved in maintaining clusters on different Hadoopdistributions and also maintained Hadoop administration on Cloudera, Hortonworks and ApacheHadoop1.x &2.x for multiple projects.
  • Worked on analyzing Hadoop cluster using different big data analytic tools including Kafka, Pig, Hive, Sqoop and MapReduce.
  • Collected and aggregated large amounts of log data using ApacheFlume and staging data in HDFS for further analysis.
  • Realtimestreaming the data using Spark with Kafka.
  • Configured Sparkstreaming to receive real time data from the Kafka and store the stream data to HDFS using Spark.
  • Importing and exporting data into HDFS using Sqoop and Kafka.
  • Wrote HiveQueries to have a consolidated view of the mortgage and bank transactional data.
  • Loaded the load ready files from mainframes to Hadoop and files were converted to ASCII format.
  • Supported team using Talend as ETL tool to transform and load the data from different databases.
  • Exposure to burn-up, burn-down charts, dashboards, velocity reporting of sprint and release progress.

Environment: Apache Hadoop, Map Reduce, Cloudera, Hive, Pig Scripts, Kafka, Sqoop, Avro, JSON, Hortonworks, Data Meer, Teradata, SQL Server, IBM Mainframes, SVN, JIRA, Nagios, Talend,Agile, MongoDB, Apache Flume, Maven & Jenkins.

Confidential

ETL Developer

Responsibilities:

  • Designed, developed and maintained projects, ETL architecture from multiple environmentsby working with various source systems and fetched data via Flat files, views, databaseconnections in Teradata, Oracle and mainframe etc.
  • Created mappings in informatica, SQL loader using UNIX scripting to stage/validate databefore it is to be standardized as per the business requirements which will be leveraged toassemble contracts in later processes.
  • Involved in DevelopingETL jobs usingData Stage to load the data into DWH/DM.
  • Worked onETL process for bringing the data intoMDMlanding tables.
  • Expert level skills on Source, mapping, device configuration, designer transformations,complex and simple mapplets, external call factor, lookups, UDF and several other power center designer objects.
  • Created scheduler batches in AutoSys scheduler and automated jobs.
  • Analyzed source system’s live data to identify the design fallouts and worked with analysts.
  • Extensively used workflow variables, mapping parameters and mapping variables & slowlychanging dimensions’ logics.
  • Used ETL methodology for supporting Data Extraction, transformations and loading processing, using Informatica BDE to Hive target in Hadoop.
  • Implemented performance tuning logic on targets, sources, mappings and sessions to providemaximum efficiency and performance also by analyzing HIVEscripts that are generated during session runs.
  • Written documentation to describe program development, logic, coding, testing, changes and corrections.

Environment: Teradata, Oracle, Mainframes, SQL, PL/SQL, Informatica 9.5, Hive Scripts, SQL Loader, Unix, Data Stage, Unit Testing, ETL Architecture, Informatica Power Center 9.x.

We'd love your feedback!