We provide IT Staff Augmentation Services!

Sr Tech Lead/cloud Data Engineer Resume

Mason, OH

SUMMARY

  • 16 years of Experience in architectural design and development in Hadoop eco - system along with Java, Scala and Python technologies at Retail,Financial.
  • Extensively having 8 years’ experience at Retail,Insurance and Healthcare Domains.
  • Vast exposure in analyzing in the application bottlenecks or issues and propose the resolutions to get rid of it.
  • Strong in documenting the architecture design / use cases / solution recommendations, etc
  • Worked in building application platforms in the Cloud by leveraging AWS, Azure, Databricks, and open source technologies and with best engineering practices.
  • Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
  • Leveraging Azure Synapse to build analytics ML model data pipeline
  • Robot built using Qualcomm API & Tensor Flow object detection to navigate the robot through tight spaces with Autonomous and TeleOp mode process.
  • Defining implementation plan for migrating the existing legacy applications to adopt PAAS (Auto Scaling, Elastic Beanstalk), Containers (Docker), Containers Orchestration (Kubernetes) and AWS Cloud Serverless architecture.
  • Hands on experience in creating API/Micro service applications using Spring Boot framework by adopting Micro services patterns, AWS Cloud, Hibernate, JWT and Angular JS.
  • Migration of existing Batch and Workflow applications to Serverless applications using AWS Services such as Step Functions, Lambda, APIGateway etc.
  • Deploying highly available and scalable applications using PAAS ( Elastic Beanstalk ) and Container Orchestration, methodologies ( Docker /Kubernetes ) by following the AWS well architecture framework and adopting best practices in Continuous Integration and Delivery process.
  • Integration Architecture design and development of payment system with various other insurance applications using Web Services and Web Sphere MQ /JMS.
  • Worked with Kubernetes to make deployment more robust and to execute ETL pipelines using Spark on Kubernetes.
  • Expertise in Java/J2EE development along with Business Process Management (BPM),SOA,Micro Services
  • Thorough knowledge in core Java concepts like OOP, JDBC, JMS, Multi-Threading, JUnit and advanced Java concepts like JSP, Servlets, HTML, XML, CSS, Hibernate, JPA and Spring.
  • Digital experience to build webapp and mobile app using AngularJS,ExtJS with back end Microservices
  • Very good experience in evaluating, installation, configuration, management and deployme t of Hadoop Cluster and its components along with security (kerberize/ssl/tls/encryption/AD) enabling.
  • Extensive knowledge in design, architecting, analyzing and implementing and building end-to-end ETL pipeline in all the phases of life cycle.
  • Well versed in using software development methodologies like Agile (SCRUM), Waterfall and Test Driven Development and also worked in various fast-paced agile development environments.
  • Demonstrated ability to handle multiple projects/tasks and leading the team. Good self-starter and enthusiastic in learning new technologies and adapt to new environment quickly. Had a good ability to think and work creatively and analytically in a problem-solving and challenging environments
  • Very keen in to doing various POC’s suggest various approaches and also list out pros and cons of approaches along with suggesting right technologies and platforms. Had a very good ability to perform detailed analysis of business problems and technical environments and use this in designing the solution
  • Excellent communication, analytical, presentation and interpersonal skills with the ability to work with leadership and business users/client. Ease in working with team of employees with various disciplines.

TECHNICAL SKILLS

  • Java, Scala, PL/SQL, Python
  • Spark (SQL, Streaming, Machine Library, DataFrame, DataSet, RDD API, GraphX), Flink, Kafka, Sqoop
  • AWS, Azure Cloudera, HortonWorks, MapR,Databricks
  • Docker, Kubernetes
  • Spark , TensorFlow
  • Streamsets, NiFi, CDAP, Cloudera Navigator
  • Akka (Core, Http, Stream, JWT), Scalatest, mockito, playtests, Zookeeper, etc
  • Oracle, MySQL, DB2, SQL Server,
  • Hive, Impala, MongoDB, HBase, Solr
  • Zepplin, Jupyter, Cloudera Data Science Workbench (CDSWS), Livy
  • ELK Logging (Elasticsearch, Logstash, Kibana), Splunk
  • J Query, Angular JS, Ext JS, D3 JS, ReactJS
  • Java / J2EE (JSP, Servlets, EJB, Spring, Struts, ORM - I Batis / Hibernate, Web Services - SOAP / Restful )
  • Websphere, Oracle Apps, WebLogic, Pramati, Tomcat, JBOSS, IIS, Jetty
  • NodeJS, Bash, Python, ANT
  • Git/ Bitbucket /Git lab, Rational Clear Case, CVS and SVN, Tortoise
  • Maven, Gradle, SBT, Jenkins, Autosys, RAD, IntelliJ, Jdeveloper, Eclipse , Bladelogic
  • Windows, Ubuntu, CentOs, RedHat.

PROFESSIONAL EXPERIENCE

Confidential, Mason, OH

Sr Tech Lead/Cloud Data Engineer

Responsibilities:

  • Business needs or requirement discussions with the Business users to come up with effective architectural design and timelines and also managing the development team
  • Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
  • Developed Json Scripts for deploying the Pipeline in Azure Data Factory (ADF) that processes the data using the Cosmos Activity.
  • Leveraging Azure Synapse to build analytics ML model data pipeline
  • Azure Bot Service and Bot Framework provide tools to build, test, deploy, and manage intelligent bots, all in one place.
  • Developed Spark applications using Scala and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
  • Responsible for estimating the cluster size, monitoring and troubleshooting of the Hadoop cluster.
  • Used Zeppelin, Jupyter notebooks and Spark-Shell to develop, test and analyze Spark jobs before Scheduling Customized Spark jobs.
  • Creating development builds and deployables and also helping in setting up the environments like Production, Development, Integration, etc
  • Extensively worked on Spark execution tuning to improve the performance of complete application.
  • Developed various security utilities and also generic utilities to launch jobs for Impala, Spark, etc from API.
  • Heavily usage of Spark Dataframe, Dataset and RDD API.
  • Creating reusable components or Utilities
  • Parsing the Mainframes ebcdic and copybooks from Spark and saving into Hive and HBase
  • Built configurable utility from config file to read the data from Hive or HDFS or apply transformations specified and save the data to HDFS or HBase or Kafka, etc
  • Configurable component to read Kafka data from Flink or Kstream or Flume and apply transformations or ML pipeline and place the data back to HDFS or Kafka or Hive.
  • Created connector to HBase from Spark dataframes to put get and scan functionalities
  • Configuring and launching the Spark jobs from ControlM
  • Involved Spark tuning to improve the Jobs performance based on the Pepper Data monitoring tool metrics.

Technologies: Spark/Scala, PySpark, Spark SQL, Kafka and Hive, Mongo DB, Java, Azure Blob, Azure Synapse, Azure Cosmos DB, Azure Data Factory, Azure Data Lake, Azure Databricks, Azure Event Hubs Azure AI, Azure ML, Bot Framework Emulator

Confidential, Blue Ash, OH

Sr Tech Lead/Cloud Data Engineer

Responsibilities:

  • Business needs or requirement discussions with the Business users to come up with effective architectural design and timelines and also managing the development team
  • Worked in building application platforms in the Cloud by leveraging AWS, Azure Databricks
  • Design and Migration Activities from on-prem to AWS Cloud environment.
  • Responsible for Designing and configuring Network Subnets, Route Tables, Association of Network ACLs to Subnets and Open VPN.
  • Designed AWS Cloud Formation templates to create VPC, subnets, NAT to ensure successful deployment of Web applications and database templates.
  • Enhance the existing product with newly features like ELB, Auto scaling, S3, Cloud Watch, Cloud Trail and RDS-Scheduling.
  • Backup and Recovery, Replication activities from on-prem to cloud.
  • Act as technical liaison between customer and team on all AWS/BigData technical aspects.
  • Creating development builds and deployable and also helping in setting up the environments like Production, Development, Integration, etc
  • Extensively worked on Spark execution tuning to improve the performance of complete application.
  • Configuring and launching the Spark jobs from ControlM
  • Involved Spark tuning to improve the Jobs performance based on the Pepper Data monitoring tool metrics.

Technologies: Cloudera, PySpark, Spark SQL, Scala/Python, Java, Impala, HBase, Hive, Jetty, Kafka, AWS Cloud, Azure Databricks

Confidential, Blue Ash, OH

Sr Tech Lead/Cloud Data Engineer

Responsibilities:

  • Architectural discussions, design, implementation, etc
  • Business needs or requirement discussions with the Business users to come up with effective architectural design and timelines.
  • Interactions with management on the goals and its tracking
  • Discussion with Clients(ember clients) and their needs
  • Azure cloud enablement for on premises Big data platform application into Cloud.
  • Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
  • Managing and guiding the development team along with few implementations and helping them on POC’s like below
  • Develop Spark 2.1/2.4 Scala component to process the business logic and store the computation results of 10 TB data into HBase database to access the downstream web apps using Big SQL db2 database.
  • Worked in building application platforms in the Cloud by leveraging Azure Databricks
  • Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
  • Developed Json Scripts for deploying the Pipeline in Azure Data Factory (ADF) that processes the data using the Cosmos Activity.
  • Used Spark API to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from upstream in near real time and Persists into HBase.
  • Working with different Hive file formats like Text file, Sequence file, ORC file, Parquet and Avro to analyze the data to build data model
  • Worked extensively develop Spark/Scala/BigSQL framework to Process and Persist
  • Customize BIGSQL Load Hadoop component Insert/Update Spark Data Frame/Dataset into HBase.
  • Integrate Spark Jobs with TWS for Jobs scheduling, Develop spark Jobs shell scrips
  • Involved Spark tuning to improve the Jobs performance based on the Pepper Data monitoring tool metrics.
  • Developed and Implement HBase capabilities for Big de-normalized data set and then apply transformation on the de-normalized data set using Spark/Scala
  • Developed and Implement Spark ETL custom component to extract the data from upstream systems and push the data to HDFS and finally store the data in HBase with wide row format.
  • Uploaded Inbound data from Different source s to HDFS and HBase
  • Implement and Develop Order Price component using IBM BigSQL

Technologies: HDP, Azure Cloud, Spark 2.4, Scala, Java, HBase, BigSQL, IBM DB2, Java, Hive, Dynatrace, Pepper Data Tool,SQL,Spring Boot,IntelliJ

Confidential, GA

Sr Cloud Data Engineer

Responsibilities:

  • Architecting, design and development of application and effectively contributing end to end involvement and also managing development teams.
  • Doing various POC’s and propose pros and cons of various approaches.
  • Creating development builds and deployables and also helping in setting up the environments like Production, Development, Integration, etc
  • Extensively worked on Spark execution tuning to improve the performance of complete application.
  • Worked on building the REST custom interface to Spark and other components using LIVY and Databricks for job invocation in Microsoft Azure platform
  • Expert knowledge on Microsoft proprietary SCOPE language specification (de-facto standard for all data related operations) for Microsoft Internal COSMOS platform for most of, much of usage streams such as Azure Compute, Azure Storage, Azure SQL & Azure DocDB.
  • Data Factory for on-perm to Azure SQL and Azure SQL to Microsoft Cloud Internal Implementation COSMOS.
  • Worked on building the custom logging wrapper over Splunk for Analyzing, Alerting and monitoring the jobs

Technologies: PySpark/Spark, CosmosDB, Azure, Azure HDInsight, Databricks, Livy, started mlflow, Scala/Python, spring, SQLServer, Azure Microsoft BI. Splunk etc

Confidential, OH

Sr Cloud Data Engineer

Responsibilities:

  • Architectural discussions, design, implementation, etc
  • Exploring or doing POC’s to suggest on the right approach to Business team, Clients and development teams
  • Business needs or requirement discussions with the Business users to come up with effective architectural design and timelines.
  • Interactions with management on the goals and its tracking
  • Discussion with Clients(ember clients) and their needs
  • Managing and guiding the development team along with few implementations and helping them on POC’s like below
  • Develop Spark 2.1/Scala component to process the business logic and store the computation results of 10 TB data into HBase database to access the downstream web apps using Big SQL db2 database.
  • Used Spark API to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from upstream in near real time and Persists into HBase.
  • Working with different Hive file formats like Text file, Sequence file, ORC file, Parquet and Avro to analyze the data to build data model
  • Worked extensively develop Spark/Scala/BigSQL framework to Process and Persist
  • Customize BIGSQL Load Hadoop component Insert/Update Spark Data Frame/Dataset into HBase.
  • Integrate Spark Jobs with TWS for Jobs scheduling, Develop spark Jobs shell scrips
  • Hands on experience in setting up workflow using Apache Oozie workflow engine for managing and scheduling Hadoop jobs in Talend TAC through wrapper scripts
  • Design and Develop web app using Restful,AngularJS,Spring Boot,Datastax API
  • Build and Design UI screens using AnjularJs, interact with UX team and coordinate with business users
  • Code reviews using GitHub and build, deployment using team city

Technologies: HDP, Spark2.1, Scala, HDFS,YARN,HBase, BigSQL, IBM DB2, Map Reduce, Java, Hive, Dynatrace, Pepper Data Tool,SQL,Spring Boot,IntelliJ,AngularJS 1.6,JAX-RS,HTML 5.0,BootStrap,Git,Team City

Hire Now