SUMMARY

• Around 8 years of professional IT experience involving project development, implementation, deployment and maintenance using Bigdata technologies in designing and implementing complete end - to-end Hadoop based data analytical solutions using HDFS, MapReduce, Spark, Scala, Yarn, Kafka, PIG, HIVE, Sqoop, Flume, Oozie, Impala, HBase.
• Hands - on experience in Azure Cloud Services (PaaS & IaaS), Azure Synapse Analytics, SQL Azure, Data Factory, Azure Analysis services, Application Insights, Azure Monitoring, Key Vault, Azure Data Lake.
• Working experience with Linux line-up like Redhat and CentOS.
• Experience in designing and building Data Management Lifecycle covering Data Ingestion, Data integration, Data consumption, Data delivery, and integration Reporting, Analytics, and System-System integration.
• Proficient in Big Data environment and Hands-on experience in utilizing Hadoop environment components for large-scale data processing including structured and semi-structured data.
• Strong experience with all phases including Requirement Analysis, Design, Coding, Testing, Support, and Documentation.
• Extensive experience with Azure cloud technologies like Azure Data Lake Storage, Azure Data Factory, Azure SQL, Azure Data Warehouse, Azure Synapse Analytical, Azure Analytical Services, Azure HDInsight, and Databricks.
• Solid Knowledge of AWS services like AWS EMR, Redshift, S3, EC2, and concepts, configuring the servers for auto-scaling and elastic load balancing.
• Experience with monitoring the web services using Hadoop and Spark for controlling the applications and analyzing their operation and performance.
• Experienced in Python data manipulation for loading and extraction as well as with Python libraries such as NumPy, Pandas, and SciPy for data analysis and numerical computations.
• Good noledge and experience with NoSQL databases like HBase, Cassandra, and MongoDB and SQL databases like Teradata, Oracle, PostgreSQL, and SQL Server.
• Experience in the development and design of various scalable systems using Hadoop technologies in various environments and analyzing data using MapReduce, Hive, and PIG.
• Hands-on use of Spark and Scala to compare the performance of Spark with Hive and SQL, and Spark SQL to manipulate Data Frames in Scala.
• Strong noledge in working with ETL methods for data extraction, transformation, and loading in corporate- wide ETL Solutions and Data Warehouse tools for reporting and data analysis.
• Hands-on experience in designing and implementing data engineering pipelines and analyzing data using Hadoop ecosystem tools like HDFS, Spark, Sqoop, Hive, Flume, Kafka, Impala, PySpark, Oozie, and HBase.
• Strong experience with all phases including Requirement Analysis, Design, Coding, Testing, Support, and Documentation.
• Experience with different ETL tool environments like SSIS, Informatica, and reporting tool environments like SQL Server Reporting Services, and Business Objects.
• Experience in deployment of applications and scripting using the Unix/Linux Shell scripting.
• Solid noledge of Data Marts, Operational Data Store, OLAP, Dimensional Data Modeling with Star Schema Modeling, Snow Flake Modeling for Dimensions Tables using Analysis Services.
• Extensive experience with various databases like Teradata, MongoDB, Cassandra DB, MySQL, Oracle, and SQL Server.
• Experience in Creating Teradata SQL scripts using OLAP functions like rank and rank over to improve the query performance while pulling the data from large tables.
• Strong Experience in working with Databases like Teradata and proficiency in writing complex SQL, PL/SQL for creating tables, views, indexes, stored procedures, and functions.
• Knowledge and experience with Continuous Integration and Continuous Deployment using containerization technologies like Docker and Jenkins.
• Excellent working experience in Agile/Scrum development and Waterfall project execution methodologies.

TECHNICAL SKILLS

BigData/Hadoop Technologies: MapReduce, Spark, SparkSQL, Azure, Spark Streaming, Kafka, Airflow, PySpark, Pig, Hive, HBase, Flume, Yarn, Oozie, Zookeeper, Hue, Ambari Server

Languages: HTML5,DHTML, WSDL, CSS3, C, C++, XML,R/R Studio, SAS Enterprise Guide, SAS, R (Caret, Weka, ggplot), Perl, MATLAB, Mathematica, FORTRAN, DTD, Schemas, Json, Ajax, Java, Scala, Python (NumPy, SciPy, Pandas, Gensim, Keras), Java Script, Shell Scripting

NO SQL Databases: Cassandra, HBase, MongoDB, MariaDB

Web Design Tools: HTML, CSS, JavaScript, JSP, jQuery, XML

Development Tools: Microsoft SQL Studio, IntelliJ, Azure Databricks, Eclipse, NetBeans.

Public Cloud: EC2, IAM, S3, Autoscaling, CloudWatch, Route53, EMR, RedShift

Development Methodologies: Agile/Scrum, UML, Design Patterns, Waterfall

Build Tools: Jenkins, Toad, SQL Loader, PostgreSQL, Talend, Maven, ANT, RTC, RSA, Control-M, Oziee, Hue, SOAP UI

Reporting Tools: MS Office (Word/Excel/Power Point/ Visio/Outlook), Crystal reports XI, SSRS, Cognos.

Databases: Microsoft SQL Server 2008, 2010/2012, MySQL 4.x/5.x, Oracle 11g, 12c, DB2, Teradata, Netezza, GraphDB

Operating Systems: All versions of Windows, UNIX, LINUX, Macintosh HD, Sun Solaris

PROFESSIONAL EXPERIENCE

Confidential, Illinois

Big Data Engineer/Spark

Responsibilities:

• Evaluating client needs and translating their business requirement to functional specifications thereby onboarding them onto the Hadoop ecosystem.
• Worked with business/user groups for gathering the requirements and working on the creation and development of pipelines.
• Migrated applications from Cassandra DB to Azure Data Lake Storage Gen 1 using Azure Data Factory, created tables, and loading and analyzed data in the Azure cloud.
• Worked on creating Azure Data Factory and managing policies for Data Factory and Utilized Blob storage for storage and backup on Azure.
• Worked on developing the process and ingested the data in Azure cloud from web service and load it to Azure SQL DB.
• Worked with Spark applications in Python for developing the distributed environment to load high volume files using Pyspark with different schema into Pyspark Data frames and process them to reload into Azure SQL DB tables.
• Designed and developed the pipelines using Databricks and automated the pipelines for the ETL processes and further maintenance of the workloads in the process.
• Worked on creating ETL packages using SSIS to extract data from various data sources like Access database, Excel spreadsheet, and flat files, and maintain the data using SQL Server.
• Worked with ETL operations in Azure Databricks by connecting to different relational databases using Kafka and used Informatica for creating, executing, and monitoring sessions and workflows.
• Experience in working with different cloud infrastructures like Amazon Web Services (AWS), Azure and Google Cloud Platform (GCP)
• Worked on automating data ingestion into the Lakehouse and transformed the data, used Apache Spark for leveraging the data, and stored the data in Delta Lake.
• Ensured data quality and integrity of the data using Azure SQL Database and automated ETL deployment and operationalization.
• Used Databricks, Scala, and Spark for creating the data workflows and capturing the data from Delta tables in Delta Lakes.
• Performed Streaming of pipelines using Azure Event Hubs and Stream Analytics to analyze the data from the data-driven workflows.
• Worked with Delta Lakes for consistent unification of Streaming, processed the data, and worked on ACID transactions using Apache Spark.
• Worked with Azure Blob Storage and developed the framework for the implementation of the huge volume of data and the system files.
• Implemented of distributed stream processing platform with low latency and seamless integration, with data and analytics services inside and outside Azure to build you're complete big data pipeline.
• Worked with PowerShell scripting for maintaining and configuring the data. Automated and validated the data using Apache Airflow.
• Worked on optimization of Hive queries using best practices and right parameters and using Hadoop, YARN, Python, and Pyspark.
• Used Sqoop to extract the data from Teradata into HDFS and export the patterns analyzed back to Teradata.
• Worked on Kafka to bring the data from data sources and keep it in HDFS systems for filtering.
• Used Accumulators and Broadcast variables to tune the Spark applications and to monitor the created analytics and jobs.
• Tracked Hadoop cluster job performance and capacity planning and tuning Hadoop performance for high availability and Hadoop cluster recovery.
• Worked with Tableau for generating reports and created Tableau dashboards, pie charts, and heat maps according to the business requirements.
• Worked with all phases of Software Development Life Cycle and used agile methodology for development.

Environment: Hadoop(HDFS,MapReduce),Scala,Databricks,Yarn,IAM,PostgreSql,Spark,Impala,Azure,Hive, Kafka, Airflow, Mongo DB, Pig, Devops, HBase, Oozie, Hue, Sqoop, Flume, GCP, Oracle, NIFI, Git, AWS Services (Lambda, EMR, Autoscaling).Data Modeling Tools: Erwin r9, GraphDB, Erwin r8, Erwin r7.1/7.2, Rational Rose 2000, ER Studio and Oracle Designer. MSSQL 2008.

Confidential, Charlotte, CT

Hadoop developer

Responsibilities:

• Involved in complete Big Data flow of the application starting from data ingestion upstream to HDFS, processing the data in HDFS and analyzing the data and involved.
• Configured Flume to extract the data from the web server output files to load into HDFS.
• Ingested data in mini-batches and performs RDD transformations on those mini-batches of data by using Spark Streaming to perform streaming analytics in Data bricks.
• Used Flume to collect, aggregate and store the web log data from different sources like web servers, mobile and network devices and pushed into HDFS.
• Understood basic business analysts concepts for logical data modeling, data flow processing and data base design.
• Migration of on premise data (Oracle/ SQL Server/ DB2/ MongoDB) to Azure Data Lake Store (ADLS) using Azure Data Factory (ADF V1/V2).
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats.
• Worked on analyzing Hadoop Cluster and different big data analytic tools including Pig, Hive.
• Worked in developing and designing Google Cloud Platform, HBase, Big, Table, and Big Query.
• Worked on airflow as a scheduling and orchestration tool.
• Working experience with data streaming process with Kafka, Apache Spark, Hive.
• Worked with various HDFS file formats like Avro, Sequence File, NiFi, Json and various compression formats like Snappy, bzip2.
• Used Spark-Streaming APIs to perform necessary transformations and actions on the data got from Kafka.
• Migrated ETL code from Talend to Informatica. Involved in development, testing and postproduction for the entire migration project.
• Worked on Migrating SQL database to Azure Data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and Controlling and granting
• Database access and Migrating On premise databases to Azure Data lake store using Azure Data factory.
• Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
• Tuned ETL jobs in the new environment after fully understanding the existing code.
• Analyzed the SQL scripts and designed the solution to implement using Scala.
• Worked on migration of data from On-prem SQL server to Cloud databases (Azure Synapse Analytics (DW) & Azure SQL DB).
• Migrated from Oozie to Apache Airflow. Involved in developing Oozie and Airflow.
• Tested Apache Tez for building high performance batch and interactive data processing applications on Pig and Hive jobs.
• Exploring with Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, PostgreSQL, Scala, Data Frame, Impala, OpenShift, Talend, pair RDD's.
• Have good experience working with Azure BLOB and Data lake storage and loading data into Azure SQL Synapse analytics (DW).
• Designed columnar families in Cassandra and Ingested data from RDBMS, performed transformations and exported the data to Cassandra.
• Leading the testing efforts in support of projects/programs across a large landscape of technologies (UNIX, Angular JS, AWS, sauseLABS, Cucumber JVM, Mongo DB, GITHub, BitBucket, SQL, NoSQL database, API, Java, Jenkins)
• Data warehouse solutions using PolyBase/external table on Azure Synapse/Azure SQL Data warehouse (Azure DW), Using Azure Data Lake as source.
• Rewriting exiting SSAS cubes to Azure Synapse/Azure SQL Data warehouse (Azure DW).
• Designing and maintaining reports in Power BI, built on top of Azure Synapse/Azure Data Warehouse, Azure Data Lake, and Azure SQL.
• Developed workflow in Oozie to automate the tasks of loading the data into Nifi and pre-processing with Pig.
• Worked on Apache NIFI to decompress and move JSON files from local to HDFS.
• Experience on moving raw data between different systems using Apache NIFI.
• Involved in loading data from the UNIX file system to HDFS using Shell Scripting.
• Used Elasticsearch for indexing/full text searching.
• Worked on MSSQL Server Integration Services (SSIS), T-SQL skills, stored procedures, triggers.
• Code and developed a custom Elastic Search java-based wrapper client using the JEST API.
• Hands-on experience in using AWS services like EC2, S3, Mongo DB, Nifi, Talend, Autoscaling and DynamoDB.

Environment: (MapReduce),Hadoop,Databricks,Spark,Talend,Impala,Hive,postgresql,Jenkins,Nifi,Scala,Mongo DB, Cassandra, Python, Pig, Sqoop, Devops, Hibernate, spring, Oozie, GCP, AWS Services EC2, S3, Autoscaling, scala, Azure, Elastic Search, DynamoDB, UNIX Shell Scripting, TEZ.

Confidential, VIRGINIA

Data Engineer

Responsibilities:

• Participated in Data Acquisition with Data Engineer team to extract clinical and imaging data from several data sources like flat file and other databases.
• Participated in the analysis of business requirements and the preparation of comprehensive specifications by project guidelines needed for project development.
• Designed and constructed an ETL framework to load data into Hive from MongoDB and MySQL, and back into MongoDB from Hive.
• Worked extensively on MongoDB, Hadoop-Hive, Spark, SQLs, and PySpark.
• Created Spark applications for data quality assessment using Scala and Python.
• Utilizing load balancers, EC2, S3, and AWS solutions, implementation.
• Configured the storage on S3 buckets and installed the application on AWS EC2 instances.
• Worked on transforming Hive/SQL queries into Spark transformations using Python, Scala, and Spark RDDs.
• Using the Airflow tool, developed Python code for each job's SLA observer, time sensor, dependencies, and various activities for workflow management and automation.
• Performed Data cleaning followed by Exploratory Analysis through visualization on real-time datasets.
• Used various machine learning techniques to build a model that could improve its accuracy in generating better predictions.
• Implemented usage of Amazon EMR for processing Big Data across a Hadoop Cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3).
• Deployed the project on Amazon EMR with S3 connectivity for setting backup storage.
• Worked with Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Spark MLlib, Data Frame, Pair RDD, and Spark YARN.
• Experienced in writing live Real-time Processing and core jobs using Spark Streaming with Kafka as a data pipeline system.
• Migrated an existing on-premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR.
• Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD.
• Worked with ELASTIC MapReduce and set up a Hadoop environment in AWS EC2 Instances.
• Used Kafka functionalities like distribution, partition, and replicated commit log service for messaging systems by maintaining feeds.
• Used Apache Kafka to aggregate web log data from multiple servers and make them available in Downstream systems for Data analysis and engineering type of roles.

Environment: AWS S3, EC2, EMR, ELASTIC MapReduce, MongoDB, Hadoop 2.0,Spark (PySpark, Nifi, Jenkins, SparkSQL, SparkMLlib),Pig 0.15,Python 3.x,MSSQL server, Tableau 10.3,GitHub,AWS EMR/EC2/s# and OpenCV.

Confidential

Data Analyst

Responsibilities:

• Collaborated with Business Analysts, SMEs across departments to gather business requirements, and identify workable items for further development.
• Partnered with ETL developers to ensure that data is well cleaned and the data warehouse is up-to-date for reporting purpose by Pig.
• Selected and generated data into Csv files and stored them into AWS S3 by using AWS EC2 and then structured and stored in AWS Redshift.
• Processed some simple statistical analysis of data profiling like cancel rate, var, skew, kurt of trades, and runs of each stock every day group by 1 min, 5 min, and 15 min.
• Used PySpark and Pandas to calculate the moving average and RSI score of the stocks and generated them into data warehouse.
• Exploring with Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, PostgreSQL, Data Frame, OpenShift, Talend, pair RDD's
• Involved in integration of Hadoop cluster with spark engine to perform BATCH and GRAPHX operations.
• Performed data preprocessing and feature engineering for further predictive analytics using Python Pandas.
• Boosted the performance of regression models by applying polynomial transformation and feature selection and used those methods to select stocks.
• Generated report on predictive analytics using Python and Tableau including visualizing model performance and prediction results.
• Utilized Agile and Scrum methodology for team and project management.
• Designed and Implemented Database auditing solution for Oracle, MSSQL, Sybase and DB2 on 10000+ servers.
• Used Git for version control with colleagues.

Environment: Spark (PySpark, SparkSQL, SparkMLlib), Google Cloud Platform, Oracle, MSSQL server, Python 3.x (Scikit-learn, Numpy, Pandas), Tableau 10.1, GitHub, AWS EMR/EC2/S3/Redshift, and Pig.