SUMMARY

• Around 9+ years of professional IT experience in which 8+ years of experience with emphasis on Big Data Technologies, Development and Design of Scala - based enterprise Application.
• 8+ years of IT experience with Data Engineer/Analyst and coding with analytical programming using SQL, Python, Snowflake,Airflow and AWS.
• Good working knowledge in multi-tiered distributed environment, good understanding of Software Development Lifecycle (SDLC)-Agile and Waterfall Methodologies.
• The ability to develop reliable, maintainable, efficient code in most of SQL, Hive and Python.
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Management Systems (RDBMS) and from RDBMS to HDFS.
• Strong knowledge of various data warehousing methodologies and data modeling concepts.
• Experience in the development of ETL processes and frameworks for large-scale, complex datasets.
• Experience with application development on Linux, python, RDBMS, NoSQL and ETL solutions.
• Experience designing and operating very large Data Warehouses.
• Experience in the development of ETL processes and frameworks like matillion for large-scale, complex datasets.
• Good knowledge in BI and data visualization tools such as Tableau.
• Expertise in writing Hadoop Jobs to analyze data using MapReduce, Hive, Pig.
• Gained Knowledge to write cloud formation templates and deployed AWS resourcing.
• Design and build data processing pipelines using tools and frameworks.
• Understanding data transformation and translation requirements and which tools to leverage to get the job done.
• Understanding data pipelines and modern ways of automating data pipeline using cloud-based technique.
• Good understanding and hands on experience in setting up and maintaining NoSQL Databases like MongoDB, and HBase.
• Experience in Designing, Architecting and implementing scalable cloud-based web applications using AWS and GCP.
• Set up a GCP Firewall rules in order to allow or deny traffic to and from the VM’s instances based on specified configuration and used GCP cloud CDN (content delivery network) to deliver content from GCP cache locations drastically improving user experience and latency.
• Familiarity with new advances in the data engineering space such as EMR and NoSQL technologies like DynamoDB.
• Hands-on experience in Hadoop ecosystem components like HDFS, Map Reduce, Yarn, Hive, Spark, Kafka, Sqoop, Oozie.
• Profound Knowledge in using Hadoop cluster in Cloudera’s CDH, Hortonworks HDP.
• Worked with AWS-based Data ingestion and transformations, setting up data inAWSusing S3 bucket and configuring instance backups to S3 bucket.
• Good knowledge of Amazon EMR, EC2, Elastic Map Reduce, Autoscaling, lambda, S3 Buckets.
• Worked on Ingesting data into Snowflake cloud data warehouse.
• Experience in working with Azure cloud platform (Data Lake, Databricks, Blob Storage, Data Factory)
• Implemented AWS solutions usingEC2, S3, RDS, EBS, Elastic Load Balancer, Auto scaling groups, AWS CLI.
• Ability to build deployment on AWS, build scripts (Boto 3 & AWS CLI), and automated solutions usingShell
• Responsible for performing reads and writes in Cassandra from and web application by using java JDBC connectivity.
• Involved in file movements betweenHDFSandAWSS3and extensively worked withS3 bucketinAWS.
• Experienced in the Hadoop ecosystem components like Hadoop Map Reduce, Cloudera, Hortonworks, HBase, Oozie, Hive, Sqoop, Pig, Flume, Kafka, Storm, Spark, Splunk, MongoDB, and Cassandra.
• Involved in developing webservices using REST,HBaseNativeAPI, and BigSQL Client to query data from HBase.
• Extensively used ETL methodology for supporting data Extraction, Transformation, and Loading in a corporate-wide- ETL solution using SAP BW with strong Knowledge on OLAP, OLTP, Extended Star, Star, Snowflake Schema methodologies.
• Experience in analyzing data usingHive QL,Pig Latin,and customMapReduceprograms inJava.
• Good experience in implementing advanced procedures like text analytics and processing the in-memory computing capabilities with Apache Impala.
• Used theSpark- Cassandra Connector to load data to and from Cassandra.
• Extensive Knowledge in architecture design of Extract, Transform and Load environments usingAbinitioSuite of products ACE, BRE, Metadata hub.
• Hands-on practical experience with various Ab Initio components such as PDL, Psets, Vectors, Join, Rollup, Scan, Reformat, Partition by Key, round-robin, gather, merge, Dedup sorted, FTP, etc.
• Developed import rules for importing various sources (EME Datasets, EME Graphs, Database) of metadata into the Metadata hub.
• Experience in working with different cloud infrastructures likeAmazon Web Services (AWS),AzureandGoogle Cloud Platform (GCP)
• Developed KafkaconsumerAPI in Scala for consuming data from Kafka topics and also consumed JSON messages using Kafka and process json files using Spark Streaming to capture UI updates.
• Migrated an existing on-premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR.
• Experience working on AWS lambda to create transformations on data between S3 buckets and EMR.
• Experience using AWS Data pipeline to migrate on premise data to S3 and DynamoDB.
• Expertise in Automating deployment of largeCassandra Clusters on EC2 using EC2 APIs.
• Developed a data pipeline using Kafka, Spark, and Hive to ingest, transform and analyze data. worked on Creating Kafka topics, partitions, writing custom practitioner’s classes.
• Experienced in writing SparkApplications in Scala.
• Imported Avro files using Apache Kafka and did some analytics using Spark Scala.
• Experience in creating RDD, Data frames for the required data, and did transformations using Spark RDD's, Spark SQL.
• Deployed various Microservices like Spark, MongoDB, Cassandra in Kubernetes and Hadoop clusters using Docker.
• Worked on Google cloud platform(GCP),Kubernetes,dataflow,Pub/Sub,Bigquery, etc.,
• DevelopedSparkcode using Scala andSpark-SQL for faster processing and testing.
• Expertise in Scala Programming language and SparkCore. Good experience in analysis using Hive and understanding of SQOOP.
• Managed Terraform pull requests using runatlantis.io and Terraform Enterprise.
• Deployed applications using Jenkins’s framework integrating Git- version control with it.
• Designs, develops, documents, tests, and debugs new and existing configuration management (Terraform) infrastructure as code. Experience with Jenkins, AWS, Terraform and GIT as source control. Documenting operational and security standards
• Automated rusulting scriots and workflow using Apache Airflow and Shell Scripting to ensure daily execution in Production.
• Experience in importing and exporting data using Sqoop from RDBMS to HDFS and vice-versa
• Involved in importing Streaming data using FLUME to HDFS and analyzing using PIG and HIVE
• Expertise in writing Hadoop jobs for Data analysis using MapReduce Hive.
• Experience in using Relational Databases like Oracle, Postgre SQL, MySQL, Microsoft SQL Server, DB2.
• Flexible with Unix/Linux and Windows Environments working with Operating Systems like Centos 5/6, Ubuntu 13/14.
• Performed advanced ETL development activities using Informatica, PL/SQL, Oracle Database tuning, and SQL tuning.
• Analyzed the extraction of data from data partners, loading, cleansing, and validating the data using programming languages like PL/SQL, Shell Scripting, and python, and reported the cleansed data to the clients using Qlik Sense.
• Good knowledge in data visualization tools such as Tableau.
• Ability to work on several python packages like Pandas, NumPy, Matplotlib, Beautiful Soup, Pyspark.
• Hands on experience in working with Continuous Integration and Deployment (CI/CD).
• Defining user stories and driving the agile board in JIRA during project execution, participate in sprint demo and retrospective.
• Install and Configure Apache Airflow for Snowflake data ware house and created dags to run the Airflow.
• Experience in using various version control systems like Git.
• Strong analytical skills with the ability to collect, organize and analyze large amounts of information with attention to detail and accuracy.
• Possess good interpersonal, analytical presentation Skills, ability to work in Self-managed and Team environments

TECHNICAL SKILLS

Hadoop/Big Data ecosystems: HDFS, Map Reduce, Sqoop, Flume, Pig, Hive,Airflow, Oozie, Impala, Zookeeper and Cloudera Manager, Zookeeper, Spark, Scala

NoSQL Database: HBase, Cassandra

Tools: and IDE: Eclipse, NetBeans, Toad, Putty, Maven, DB Visualizer, VS Code, Qlik Sense, Qlik View

Languages: SQL, PL/SQL, JAVA, Scala, Python

Databases: Oracle, SQL Server, MySQL, DB2, PostgreSQL, Teradata

Tracking Tools and Control: SVN, GIT, Maven,Terraform

ETL Tools: OFSAA, IBM DataStage

Cloud Technologies: AWS, Azure, GCP

PROFESSIONAL EXPERIENCE

Confidential, Woonsocket RI

Bigdata Developer

Responsibilities:

• Developed data pipeline using Sqoop,Spark, MapReduce, and Hive to ingest, transform and analyze, customer behavioral data.
• ImplementedSparkusing python andSparkSQL for faster processing of data and algorithms for real-time analysis inSpark.
• Experience in using Splunk, Apache Flume for collecting, aggregation, moving large amounts of data from the application server.
• UsedSparkfor interactive queries, processing of streaming data, and integration with popular NoSQL database for huge volume of data.
• Building a Scala and spark based configurable framework to connect common Data sources like MYSQL, Oracle, Postgres, SQL Server, Bigquery and load it in Bigquery.
• Experienced in implementing Real-Time streaming and analytics using various technologies i.e.Spark Streaming and Kafka.
• Designed and developed batch and streaming data pipelines to load data into Data Lake and Data Warehouse. Using services like AWS S3, RedShift, Glue, EMR, Lambda, Step Function, IAM, QuickSight, RDS, CloudWatch events and Trails, etc.,
• Developed APIs to access enterprise metadata platform for registering datasets and migrated all the data pipeline applications from legacy to new platform.
• Developed and modified all our data application using python
• Extensively used PySpark, Python for building and modifying existing pipelines.
• Worked on creating panda’s data frames and spark data frames where needed in order optimize the performance.
• Created/modified data pipeline applications using spark, python, PySpark, AWS EMR, S3, Lambda, pandas, SparkSQL, Glue, Pestro and Snowflake.
• Extensively worked on EMR and Spark jobs and created IAM roles, S3 buckets policies based on the enterprise request.
• Created complex SQL queries using SparkSQL for all data transformation when loading data into Snowflake
• Experience in working Kafka steaming applications for real time data needs.
• Utilized enterprise scan tool to identify data that is sensitive in all S3 buckets that we own and encrypt the data before downstream can use
• Deploy application to Docker Container using Jenkins and trigger jobs to load data from S3 to Snowflake tables or Snowflake to S3 based on the user requirements
• Collaborated with Data Analysts to understand the data and their end requirements and transform the data using PySpark, SparkSQL and created data frames
• Extensively worked on file formats like Parquet, AVRO, CSV and JSON files
• Experience in using POSTMAN to test the end points for various applications and validate the schema, data types, formats of file etc.
• Monitor applications using PagerDuty, and slunk logs based the data loads daily, weekly and monthly
• Experience working on Docker Images, update docker images as necessary
• Worked on Databricks, created managed clusters and performed data transformations.

Environment: Hadoop, MapReduce, Yarn,Spark, Bigquery, Hive, GCP, Pig, Python, Docker, Databricks, AVRO, PySPark, SparkSQL, Snowflake, Kafka, Bash/Shell Scripting, Flume, EMR, S3, Glue, Pestro.

Confidential, Chicago, IL

Big Data Developer

Responsibilities:

• Research and recommend a suitable technology stack for Hadoop migration considering current enterprise architecture.
• Expertise in writingHadoopJobsfor analyzing structured and unstructured data usingHDFS,Hive, HBase, Pig, Spark, Kafka, Scala, Oozie, andTalend ETL.
• Extensively usedSparkstack to develop preprocessing job which includes RDD, Datasets, and Data frame APIs to transform the data for upstream consumption.
• Build and maintain scalable data pipelines using theHadoop ecosystemand other open-source components likeHive, andHBase.
• Good understanding ofNoSQLDatabases and hands-on work experience in writing applications on No SQL databases likeCassandraandMongo DB.
• Good knowledge in querying data fromCassandrafor searching grouping and sorting.
• Involved in various NoSQL databases likeHBase, Cassandrain implementing and integration.
• Deploy Kubernetes in both AWS and Google cloud. Setup cluster, the replicator. Deploy multiple containers in a pod.
• Using GCP Console, monitor Dataproc cluster and jobs. Stack Driver to monitor Dashboards and do a performance tuning and optimization of jobs which are memory intensive and provide L3 support for the applications in production environment.
• Reviewed and documented business logic in Informatica mapping, work flow to migrated to GCP.
• Experience in google platform architecture & Bigquery & Proficient in designing efficient workflows.
• Worked on scalable distributed data system using Hadoop ecosystem in AWS EMR, MapR distribution.
• Good experience in Administering and configuring Kubernetes.
• Migrated an existing on-premises application to AWS.
• Used AWS services like EC2 and S3 for small data sets.
• UsedZookeeperfor various types of centralized configurations,SVNfor version control, Maven for project management,Jirafor internal bug/defect management,MapReduce
• Experience in analyzing data usingHiveQL,SparkSQL,PostgreSQL, Pig Latin, and customMap Reduceprograms in Java.
• Worked in developing and designing GCP, HBase, Big, Table, Big Query.
• GCP - Stack driver for monitoring, logging, compute engine and Dataproc.
• Involved in analyzing data using google Bigquery to discover information, business value, patterns and trends in support of decision making & Worked on data profiling and data quality analysis
• Involved in migration of Teradata queries into the snowflake Data warehouse queries.
• Involved in developing Spring IOC Inversion of control, DAO, MVC.
• Used spring support for restful webservices to communicate with the host machine for agreement forms.
• Write theGolangcode to pull the data from kinesis and load the data toPrometheuswhich will be used byGrafanafor reporting and visualize
• Involved in modeling different key risk indicators in Splunk and building extensive Hive & SPL quires to understand behavior across the customer life cycle.
• Converted existing Snowflake schema data into Star schema in the hive for building OLAP cubes in Kylin.
• SupportedMapReduceProgramsthat are running on the cluster and also wroteMapReduce jobsusingJavaAPI.
• Optimized Hive QL scripts by using execution engines like Tez, Spark.
• Developed Hive queries to analyze the data in HDFS to identify issues and behavioral patterns.
• Able to use Python Pandas, NumPy modules for Data analysis, Data scraping, and parsing.
• Create common Terraform modules (referencing other Github modules) for use across multiple AWS environments.
• Deployed applications using Jenkins’s framework integrating Git- version control with it.
• Participated in production support regularly to support the Analytics platform
• Used GIT for version control.

Environment: Hadoop, HDFS, AWS, Hive, Bigquery, Tez, Spark SQL, GCP, MapReduce,Sparkstreaming, Abinitio, Sqoop, Oozie, Docker, Kafka,Spark, Scala, Talend, Shell Scripting.

Confidential, Charlotte, NC

Big Data Developer

Responsibilities:

• Performed advanced procedures like text analytics and processing using the in-memory computing capabilities ofSpark.
• DevelopedSparkcode using Scala andSpark-SQL for faster processing and testing.
• Worked onSparkSQL for joining multi hive tables and write them to a final hive table.
• Configured Spark Streaming to receive real-time data from the Apache Kafka and store the stream data toHDFSusingScala.
• Worked on scalable distributed data system using Hadoop ecosystem in AWS EMR and MapR (MapR data platform).
• Developed prototypeSparkapplications usingSpark-Core, Spark SQL, Data Frame APIand developed several custom User-defined functions inHive & Pig using Java & python
• Importing the data intoSparkfromKafkaConsumer group usingSpark Streaming APIs.
• Good Knowledge of reporting and data visualization tools like Oracle Data Visualization Desktop, Tableau, and Grafana.
• Worked on Google Cloud Platform(GCP) Services like Vision API, Instances.
• Used Informatica Cloud Data Integration for global, Data Cloud Architect, Azure, Ansible, Jenkins, Docker, Kubernetes, DevOps, Automation, CI/CD, distributed data warehouse, and analytics projects.
• Utilized Azure Data Factory to create, Data Cloud Architect, Azure, Ansible, Jenkins, Docker, Kubernetes, DevOps, Automation, CI/CD,schedule and manage data pipelines.
• Developed a POC for project migration from on prem Hadoop MapR system to GCP/Snowflake.
• Build cluster on AWS environment using EMR using S3, EC2, and Redshift.
• Built dashboards and visualizations on top of MapR-DB and Hive using Oracle data visualizer desktop. Built real-time visualizations on top of Open TSDB using Grafana.
• Used DataStax Cassandra along with Pentaho for reporting.
• ImplementedAWS EC2, Key Pairs, Security Groups, AutoScaling, ELB, SQS, and SNSusingAWS APIand exposed as theRestful Web services andimplementedReporting, Notification servicesusingAWS API.
• Performed querying of both managed and external tables created by Hive using Impala.
• Developed Impala scripts for end-user/analyst requirements for Adhoc analysis.
• Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.
• UsedSparkAPI over Cloudera Hadoop YARN to perform analytics on data in Hive.
• UsedHTML, CSS, XML, JavaScriptandJSPfor interactive cross browser functionality and complex user interface.
• Experience in writingcustom UDFsfor Hive to in corporate methods and functionality of Java into andHQLHIVESQL.
• Collected data usingSparkStreaming from AWSS3 bucket in near-real-time and performs necessary Transformations and Aggregations to build the data model and persists the data in HDFS.
• Responsible for creating Hive tables, loading with data, and writing Hive queries.
• Optimized Hive QL by using execution engines like Tez,Spark.
• Responsible for creating mappings and workflows to extract and load data from relational databases, flat file sources, and legacy systems using Abinitio.
• Fetch and generate monthly reports, Visualization those reports using Tableau.
• Used Oozie Workflow engine to run multiple Hive jobs.

Environment: Hadoop, Cloudera, Flume, HBase, GCP, CICD, EC2, ELB, SQS, SNS, HQL, HIVESQL, HDFS, MapReduce, YARN, Hive, Sqoop, Oozie, Tableau, Abinitio, JUnit, agile methodologies, UNIX

Confidential

Hadoop/ETL Developer

Responsibilities:

• Implemented ETL Abinitio designs and processes for a load of data from the sources to the target warehouse.
• Responsible for performance tuning at various levels like mapping level, session level, and database level.
• Worked withAbinitioComponents, Parallelism concepts.
• Scheduled various daily and monthly ETL loads using scheduler tools.
• Worked on developing UNIX Scripts.
• Experience migrating data between HDFS and RDBMS using Sqoop and also exporting and importing using streaming platforms Flume and Kafka.
• BTEQ, MLOAD, FLOAD utilities of Teradata and Unix Shell scripts.
• Created tags and save files to migrateAbinitioprojects/objects across the environment.
• Worked with Session Logs and Workflow Logs for Error handling and troubleshooting.
• Developing UDFs in java for hive and pig, worked on reading multiple data formats on HDFS using Scala.
• Perform data masking andETL process using S3, Informatica Cloud, Informatica Power Center, and Informatica Test Data Managementto support Snowflake Data warehousing solution in the cloud.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala.
• Developed multiple POCs using Scala and deployed on the Yarn cluster, compared the performance of Spark, with Hive and SQL/Teradata.
• Analyzed the SQL scripts and designed the solution to implement using Scala
• Pulling data from oracle and dumping on HDFS as Avro format file and then again converting Avro format to parquet, to resolve the performance issue, we are loading data into hive/impala as parquet files.
• Pulling data from web services(HTTP proxy)and do cleansing and keep data on hdfs and then apply some logic through Talend(ETL) and pushing into hdfs to access the data from hive and impala.
• Migrating data from Oracle toData LakeusingSqoop,Spark, and Talend(ETL Tool).
• Use Job Conductor to deploy the job(.zar) files and schedule and monitor the job.
• Migrate the code from development to production environment usingNexus,scheduling job to run in production throughCONTROL-M.
• ImplementedStormbuilder topologies to perform cleansing operations before moving data intoCassandra.
• Partitioning, Dynamic Partitions, Buckets of HIVE.
• Implement HIVE UDF’s for evaluation, filtering, loading, and storing of data.
• Design Managed and External tables in Hive to optimize performance to improve performance
• Using AutoMap join and avoid skew join, optimize limit operator, enable Parallel Execution, enable MapReduce Strict Mode, Single Reduce for Multi-Group BY function.
• Load data from different sources (database and files) into Hive usingTalend tool (standard, Map Reduce, and Spark job), monitor System health and logs, and respond to any warning or failure conditions.
• Build data systems anddata pipelinesthat extract, classify, merge, and deliver new insights.
• Data Ingestion,aggregating, Loading, and transforming large data sets of structured, semi-structured, and unstructured data intoHadoop(data lake).
• CreatedHivequeries that helped market analysts spot emerging trends by comparing fresh data with EDW reference tables and historical metrics.
• Implemented Hive UDF's for evaluation, filtering, loading, and storing of data.
• Partitioning, Dynamic Partitions, Buckets inHIVE.
• Data migration from relational(Oracle, Teradata) databases or external data to HDFS usingSqoop.
• Designed both Managed and External tables in Hive tooptimize performance.To improve performance,we use AutoMap join and avoid skew join, Optimize LIMIT operator,Enable Parallel Execution,Enable MapReduce Strict Mode,Single Reduce for Multi-Group BY.
• Implemented various loads like Daily Loads, Weekly Loads, and Quarterly Loads using Incremental Loading Strategy.
• Responsible for Unit Testing and creating unit test plans and preparing unit test cases.
• Interacting with the system testing team and resolve the issue reported by the testing team.

Environment: Abinitio, Hadoop, Unix, UNIX Scripts, agile methodologies, ETL, AWS, S3, Teradata, Hive, SQL, Shell Script.