SUMMARY

• Around 8 years of professional IT experience involving project development, implementation, deployment and maintenance using Bigdata technologies in designing and implementing complete end - to-end Hadoop based data analytical solutions using HDFS, MapReduce, Spark, Scala, Yarn, Kafka, PIG, HIVE, Sqoop, Flume, Oozie, Impala, HBase.
• Hands - on experience in Azure Cloud Services (PaaS & IaaS), Azure Synapse Analytics, SQL Azure, Data Factory, Azure Analysis services, Application Insights, Azure Monitoring, Key Vault, Azure Data Lake.
• Working experience wif Linux line-up like Redhat and CentOS.
• Experience in designing and building Data Management Lifecycle covering Data Ingestion, Data integration, Data consumption, Data delivery, and integration Reporting, Analytics, and System-System integration.
• Proficient in Big Data environment and Hands-on experience in utilizing Hadoop environment components for large-scale data processing including structured and semi-structured data.
• Strong experience wif all phases including Requirement Analysis, Design, Coding, Testing, Support, and Documentation.
• Extensive experience wif Azure cloud technologies like Azure Data Lake Storage, Azure Data Factory, Azure SQL, Azure Data Warehouse, Azure Synapse Analytical, Azure Analytical Services, Azure HDInsight, and Databricks.
• Solid Knowledge of AWS services like AWS EMR, Redshift, S3, EC2, and concepts, configuring teh servers for auto-scaling and elastic load balancing.
• Experience wif monitoring teh web services using Hadoop and Spark for controlling teh applications and analyzing their operation and performance.
• Experienced in Python data manipulation for loading and extraction as well as wif Python libraries such as NumPy, Pandas, and SciPy for data analysis and numerical computations.
• Good noledge and experience wif NoSQL databases like HBase, Cassandra, and MongoDB and SQL databases like Teradata, Oracle, PostgreSQL, and SQL Server.
• Experience in teh development and design of various scalable systems using Hadoop technologies in various environments and analyzing data using MapReduce, Hive, and PIG.
• Hands-on use of Spark and Scala to compare teh performance of Spark wif Hive and SQL, and Spark SQL to manipulate Data Frames in Scala.
• Strong noledge in working wif ETL methods for data extraction, transformation, and loading in corporate- wide ETL Solutions and Data Warehouse tools for reporting and data analysis.
• Hands-on experience in designing and implementing data engineering pipelines and analyzing data using Hadoop ecosystem tools like HDFS, Spark, Sqoop, Hive, Flume, Kafka, Impala, PySpark, Oozie, and HBase.
• Strong experience wif all phases including Requirement Analysis, Design, Coding, Testing, Support, and Documentation.
• Experience wif different ETL tool environments like SSIS, Informatica, and reporting tool environments like SQL Server Reporting Services, and Business Objects.
• Experience in deployment of applications and scripting using teh Unix/Linux Shell scripting.
• Solid noledge of Data Marts, Operational Data Store, OLAP, Dimensional Data Modeling wif Star Schema Modeling, Snow Flake Modeling for Dimensions Tables using Analysis Services.
• Extensive experience wif various databases like Teradata, MongoDB, Cassandra DB, MySQL, Oracle, and SQL Server.
• Experience in Creating Teradata SQL scripts using OLAP functions like rank and rank over to improve teh query performance while pulling teh data from large tables.
• Strong Experience in working wif Databases like Teradata and proficiency in writing complex SQL, PL/SQL for creating tables, views, indexes, stored procedures, and functions.
• Knowledge and experience wif Continuous Integration and Continuous Deployment using containerization technologies like Docker and Jenkins.
• Excellent working experience in Agile/Scrum development and Waterfall project execution methodologies.

TECHNICAL SKILLS

BigData/Hadoop Technologies: MapReduce, Spark, SparkSQL, Azure, Spark Streaming, Kafka, Airflow, PySpark, Pig, Hive, HBase, Flume, Yarn, Oozie, Zookeeper, Hue, Ambari Server

Languages: HTML5,DHTML, WSDL, CSS3, C, C++, XML,R/R Studio, SAS Enterprise Guide, SAS, R (Caret, Weka, ggplot), Perl, MATLAB, Mathematica, FORTRAN, DTD, Schemas, Json, Ajax, Java, Scala, Python (NumPy, SciPy, Pandas, Gensim, Keras), Java Script, Shell Scripting

NO SQL Databases: Cassandra, HBase, MongoDB, MariaDB

Web Design Tools: HTML, CSS, JavaScript, JSP, jQuery, XML

Development Tools: Microsoft SQL Studio, IntelliJ, Azure Databricks, Eclipse, NetBeans.

Public Cloud: EC2, IAM, S3, Autoscaling, CloudWatch, Route53, EMR, RedShift

Development Methodologies: Agile/Scrum, UML, Design Patterns, Waterfall

Build Tools: Jenkins, Toad, SQL Loader, PostgreSQL, Talend, Maven, ANT, RTC, RSA, Control-M, Oziee, Hue, SOAP UI

Reporting Tools: MS Office (Word/Excel/Power Point/ Visio/Outlook), Crystal reports XI, SSRS, Cognos.

Databases: Microsoft SQL Server 2008, 2010/2012, MySQL 4.x/5.x, Oracle 11g, 12c, DB2, Teradata, Netezza, GraphDB

Operating Systems: All versions of Windows, UNIX, LINUX, Macintosh HD, Sun Solaris

PROFESSIONAL EXPERIENCE

Confidential, Illinois

Big Data Engineer/Spark

Responsibilities:

• Evaluating client needs and translating their business requirement to functional specifications thereby onboarding them onto teh Hadoop ecosystem.
• Worked wif business/user groups for gathering teh requirements and working on teh creation and development of pipelines.
• Migrated applications from Cassandra DB to Azure Data Lake Storage Gen 1 using Azure Data Factory, created tables, and loading and analyzed data in teh Azure cloud.
• Worked on creating Azure Data Factory and managing policies for Data Factory and Utilized Blob storage for storage and backup on Azure.
• Worked on developing teh process and ingested teh data in Azure cloud from web service and load it to Azure SQL DB.
• Worked wif Spark applications in Python for developing teh distributed environment to load high volume files using Pyspark wif different schema into Pyspark Data frames and process them to reload into Azure SQL DB tables.
• Designed and developed teh pipelines using Databricks and automated teh pipelines for teh ETL processes and further maintenance of teh workloads in teh process.
• Worked on creating ETL packages using SSIS to extract data from various data sources like Access database, Excel spreadsheet, and flat files, and maintain teh data using SQL Server.
• Worked wif ETL operations in Azure Databricks by connecting to different relational databases using Kafka and used Informatica for creating, executing, and monitoring sessions and workflows.
• Experience in working wif different cloud infrastructures like Amazon Web Services (AWS), Azure and Google Cloud Platform (GCP)
• Worked on automating data ingestion into teh Lakehouse and transformed teh data, used Apache Spark for leveraging teh data, and stored teh data in Delta Lake.
• Ensured data quality and integrity of teh data using Azure SQL Database and automated ETL deployment and operationalization.
• Used Databricks, Scala, and Spark for creating teh data workflows and capturing teh data from Delta tables in Delta Lakes.
• Performed Streaming of pipelines using Azure Event Hubs and Stream Analytics to analyze teh data from teh data-driven workflows.
• Worked wif Delta Lakes for consistent unification of Streaming, processed teh data, and worked on ACID transactions using Apache Spark.
• Worked wif Azure Blob Storage and developed teh framework for teh implementation of teh huge volume of data and teh system files.
• Implemented of distributed stream processing platform wif low latency and seamless integration, wif data and analytics services inside and outside Azure to build your complete big data pipeline.
• Worked wif PowerShell scripting for maintaining and configuring teh data. Automated and validated teh data using Apache Airflow.
• Worked on optimization of Hive queries using best practices and right parameters and using Hadoop, YARN, Python, and Pyspark.
• Used Sqoop to extract teh data from Teradata into HDFS and export teh patterns analyzed back to Teradata.
• Worked on Kafka to bring teh data from data sources and keep it in HDFS systems for filtering.
• Used Accumulators and Broadcast variables to tune teh Spark applications and to monitor teh created analytics and jobs.
• Tracked Hadoop cluster job performance and capacity planning and tuning Hadoop performance for high availability and Hadoop cluster recovery.
• Worked wif Tableau for generating reports and created Tableau dashboards, pie charts, and heat maps according to teh business requirements.
• Worked wif all phases of Software Development Life Cycle and used agile methodology for development.

Environment: Hadoop(HDFS,MapReduce),Scala,Databricks,Yarn,IAM,PostgreSql,Spark,Impala,Azure,Hive, Kafka, Airflow, Mongo DB, Pig, Devops, HBase, Oozie, Hue, Sqoop, Flume, GCP, Oracle, NIFI, Git, AWS Services (Lambda, EMR, Autoscaling).Data Modeling Tools: Erwin r9, GraphDB, Erwin r8, Erwin r7.1/7.2, Rational Rose 2000, ER Studio and Oracle Designer. MSSQL 2008.

Confidential, Charlotte, CT

Hadoop developer

Responsibilities:

• Involved in complete Big Data flow of teh application starting from data ingestion upstream to HDFS, processing teh data in HDFS and analyzing teh data and involved.
• Configured Flume to extract teh data from teh web server output files to load into HDFS.
• Ingested data in mini-batches and performs RDD transformations on those mini-batches of data by using Spark Streaming to perform streaming analytics in Data bricks.
• Used Flume to collect, aggregate and store teh web log data from different sources like web servers, mobile and network devices and pushed into HDFS.
• Understood basic business analysts concepts for logical data modeling, data flow processing and data base design.
• Migration of on premise data (Oracle/ SQL Server/ DB2/ MongoDB) to Azure Data Lake Store (ADLS) using Azure Data Factory (ADF V1/V2).
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats.
• Worked on analyzing Hadoop Cluster and different big data analytic tools including Pig, Hive.
• Worked in developing and designing Google Cloud Platform, HBase, Big, Table, and Big Query.
• Worked on airflow as a scheduling and orchestration tool.
• Working experience wif data streaming process wif Kafka, Apache Spark, Hive.
• Worked wif various HDFS file formats like Avro, Sequence File, NiFi, Json and various compression formats like Snappy, bzip2.
• Used Spark-Streaming APIs to perform necessary transformations and actions on teh data got from Kafka.
• Migrated ETL code from Talend to Informatica. Involved in development, testing and postproduction for teh entire migration project.
• Worked on Migrating SQL database to Azure Data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and Controlling and granting
• Database access and Migrating On premise databases to Azure Data lake store using Azure Data factory.
• Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
• Tuned ETL jobs in teh new environment after fully understanding teh existing code.
• Analyzed teh SQL scripts and designed teh solution to implement using Scala.
• Worked on migration of data from On-prem SQL server to Cloud databases (Azure Synapse Analytics (DW) & Azure SQL DB).
• Migrated from Oozie to Apache Airflow. Involved in developing Oozie and Airflow.
• Tested Apache Tez for building high performance batch and interactive data processing applications on Pig and Hive jobs.
• Exploring wif Spark to improve teh performance and optimization of teh existing algorithms in Hadoop using Spark context, Spark-SQL, PostgreSQL, Scala, Data Frame, Impala, OpenShift, Talend, pair RDD's.
• Has good experience working wif Azure BLOB and Data lake storage and loading data into Azure SQL Synapse analytics (DW).
• Designed columnar families in Cassandra and Ingested data from RDBMS, performed transformations and exported teh data to Cassandra.
• Leading teh testing efforts in support of projects/programs across a large landscape of technologies (UNIX, Angular JS, AWS, sauseLABS, Cucumber JVM, Mongo DB, GITHub, BitBucket, SQL, NoSQL database, API, Java, Jenkins)
• Data warehouse solutions using PolyBase/external table on Azure Synapse/Azure SQL Data warehouse (Azure DW), Using Azure Data Lake as source.
• Rewriting exiting SSAS cubes to Azure Synapse/Azure SQL Data warehouse (Azure DW).
• Designing and maintaining reports in Power BI, built on top of Azure Synapse/Azure Data Warehouse, Azure Data Lake, and Azure SQL.
• Developed workflow in Oozie to automate teh tasks of loading teh data into Nifi and pre-processing wif Pig.
• Worked on Apache NIFI to decompress and move JSON files from local to HDFS.
• Experience on moving raw data between different systems using Apache NIFI.
• Involved in loading data from teh UNIX file system to HDFS using Shell Scripting.
• Used Elasticsearch for indexing/full text searching.
• Worked on MSSQL Server Integration Services (SSIS), T-SQL skills, stored procedures, triggers.
• Code and developed a custom Elastic Search java-based wrapper client using teh JEST API.
• Hands-on experience in using AWS services like EC2, S3, Mongo DB, Nifi, Talend, Autoscaling and DynamoDB.

Environment: (MapReduce),Hadoop,Databricks,Spark,Talend,Impala,Hive,postgresql,Jenkins,Nifi,Scala,Mongo DB, Cassandra, Python, Pig, Sqoop, Devops, Hibernate, spring, Oozie, GCP, AWS Services EC2, S3, Autoscaling, scala, Azure, Elastic Search, DynamoDB, UNIX Shell Scripting, TEZ.

Confidential, VIRGINIA

Data Engineer

Responsibilities:

• Participated in Data Acquisition wif Data Engineer team to extract clinical and imaging data from several data sources like flat file and other databases.
• Participated in teh analysis of business requirements and teh preparation of comprehensive specifications by project guidelines needed for project development.
• Designed and constructed an ETL framework to load data into Hive from MongoDB and MySQL, and back into MongoDB from Hive.
• Worked extensively on MongoDB, Hadoop-Hive, Spark, SQLs, and PySpark.
• Created Spark applications for data quality assessment using Scala and Python.
• Utilizing load balancers, EC2, S3, and AWS solutions, implementation.
• Configured teh storage on S3 buckets and installed teh application on AWS EC2 instances.
• Worked on transforming Hive/SQL queries into Spark transformations using Python, Scala, and Spark RDDs.
• Using teh Airflow tool, developed Python code for each job's SLA observer, time sensor, dependencies, and various activities for workflow management and automation.
• Performed Data cleaning followed by Exploratory Analysis through visualization on real-time datasets.
• Used various machine learning techniques to build a model that could improve its accuracy in generating better predictions.
• Implemented usage of Amazon EMR for processing Big Data across a Hadoop Cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3).
• Deployed teh project on Amazon EMR wif S3 connectivity for setting backup storage.
• Worked wif Spark for improving performance and optimization of teh existing algorithms in Hadoop using Spark Context, Spark-SQL, Spark MLlib, Data Frame, Pair RDD, and Spark YARN.
• Experienced in writing live Real-time Processing and core jobs using Spark Streaming wif Kafka as a data pipeline system.
• Migrated an existing on-premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining teh Hadoop cluster on AWS EMR.
• Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD.
• Worked wif ELASTIC MapReduce and set up a Hadoop environment in AWS EC2 Instances.
• Used Kafka functionalities like distribution, partition, and replicated commit log service for messaging systems by maintaining feeds.
• Used Apache Kafka to aggregate web log data from multiple servers and make them available in Downstream systems for Data analysis and engineering type of roles.

Environment: AWS S3, EC2, EMR, ELASTIC MapReduce, MongoDB, Hadoop 2.0,Spark (PySpark, Nifi, Jenkins, SparkSQL, SparkMLlib),Pig 0.15,Python 3.x,MSSQL server, Tableau 10.3,GitHub,AWS EMR/EC2/s# and OpenCV.

Confidential

Data Analyst

Responsibilities:

• Collaborated wif Business Analysts, SMEs across departments to gather business requirements, and identify workable items for further development.
• Partnered wif ETL developers to ensure that data is well cleaned and teh data warehouse is up-to-date for reporting purpose by Pig.
• Selected and generated data into Csv files and stored them into AWS S3 by using AWS EC2 and tan structured and stored in AWS Redshift.
• Processed some simple statistical analysis of data profiling like cancel rate, var, skew, kurt of trades, and runs of each stock every day group by 1 min, 5 min, and 15 min.
• Used PySpark and Pandas to calculate teh moving average and RSI score of teh stocks and generated them into data warehouse.
• Exploring wif Spark to improve teh performance and optimization of teh existing algorithms in Hadoop using Spark context, Spark-SQL, PostgreSQL, Data Frame, OpenShift, Talend, pair RDD's
• Involved in integration of Hadoop cluster wif spark engine to perform BATCH and GRAPHX operations.
• Performed data preprocessing and feature engineering for further predictive analytics using Python Pandas.
• Boosted teh performance of regression models by applying polynomial transformation and feature selection and used those methods to select stocks.
• Generated report on predictive analytics using Python and Tableau including visualizing model performance and prediction results.
• Utilized Agile and Scrum methodology for team and project management.
• Designed and Implemented Database auditing solution for Oracle, MSSQL, Sybase and DB2 on 10000+ servers.
• Used Git for version control wif colleagues.

Environment: Spark (PySpark, SparkSQL, SparkMLlib), Google Cloud Platform, Oracle, MSSQL server, Python 3.x (Scikit-learn, Numpy, Pandas), Tableau 10.1, GitHub, AWS EMR/EC2/S3/Redshift, and Pig.