SUMMARY

• Data Engineering professional with solid foundational skills and proven tracks of implementation in a variety of data platforms. Self - motivated with a strong adherence to personal accountability in both individual and team scenarios.
• Around 6 years of experience in Data Engineering, Data Pipeline Design, Development, and Implementation as a Data Engineer/ Big Data Engineer.
• Expertise in transforming business resources and requirements intomanageable data formatsandanalytical models,designing algorithms,building models,developing data miningandreporting solutionsthat scale across a massive volume of structured and unstructured data.
• Knowledge of working with Proof of Concepts (PoC's) and gap analysis and gathered necessary data for analysis from different sources, prepared data for data exploration.
• Experience in developing customized UDF’sin Python to extend Hive and Pig Latin functionality.
• Expertise in designing complex Mappings and have expertise in performance tuning and slowly changing Dimension Tables and Fact tables.
• Good exposure in interacting with clients and solving application environment issues and can communicate effectively with people at different levels including stakeholders, internal teams and the senior management.
• Extensive experieced Big Data - Hadoop developer with varying level of expertise around different Big Data/Hadoop ecosystem projects which include Spark streaming, HDFS,MapReduce, NiFi, HIVE, HBase, Storm, Kafka, Flume, Sqoop, ZooKeeper, Oozie etc.
• Have very strong inter-personal skills and the ability to work independently and with the group, can learn quickly and easily adaptable to the working environment.
• Extensive experience with Real-timestreaming technologies Spark.
• Well experience in Normalization and De-Normalization techniques for optimum performance in relational and dimensional database environments.
• Strong experience in writing scripts using Python API, PySpark API and Spark API for analyzing the data.
• Sustaining the BigQuery, PySpark and Hive code by fixing the bugs and providing the enhancements required by the Business User.
• Experienced in building Automation Regressing Scripts for validation of ETL process between multiple databases like Oracle, SQL Server, Hive, and Mongo DB using Python.
• Skilled in performing data parsing, data ingestion, data manipulation, data architecture, data modelling and data preparation with methods including describe data contents, compute descriptive statistics of data, regex, split and combine, Remap, merge, subset, reindex, melt and reshape.
• Hands-on use of Spark and Scala API's to compare the performance of Spark with Hive and SQL, and Spark SQL to manipulate Data Frames in Scala. Proficiency in SQL across several dialects (we commonly write MySQL, PostgreSQL, SQL Server, and Oracle).
• Expertise in Python and Scala, user-defined functions (UDF) for Hive and Pig using Python.
• Experience in developing Map Reduce Programs using Apache Hadoop for analyzing the big data as per the requirement. Experience in working with Flume and NiFi for loading log files into Hadoop.
• Experience in working with NoSQL databases like HBase and Cassandra. Good Experience in implementing and orchestrating data pipelines using Oozie and Airflow. Experience in designing star schema, Snowflake schema for Data Warehouse, ODS architecture. and dimensional database environments. Skilled in System Analysis, E-R/Dimensional Data Modeling, Database Design and implementing RDBMS specific features. Well experience in Normalization and De-Normalization techniques for optimum performance in relational
• Good knowledge of Data Marts, OLAP, Dimensional Data Modeling with Methodology (Star Schema Modeling, Snow-Flake Modeling for FACT and Dimensions Tables) using Analysis Services.
• Ability to work effectively in cross-functional team environments, excellent communication, and interpersonal skills.

PROFESSIONAL EXPERIENCE

Confidential, Dallas, TX

Big Data Engineer

Responsibilities:

• Performed data analysis and developed analytic solutions. Data investigation to discover correlations trends and the ability to explain them. Worked with Data Engineers, Data Architects, to define back-end requirements for data products (aggregations, materialized views, tables - visualization)
• Developed frameworks and processes to analyze unstructured information. Designing and Developing Oracle PL/SQL and Shell Scripts, Data Import/Export, Data Conversions and Data Cleansing.
• Performing data analysis, statistical analysis, generated reports, listings, and graphs using SAS tools, SAS/Graph, SAS/SQL, SAS/Connect and SAS/Access.
• Developing Spark applications using Scala and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats. Using Kafka and integrating with the Spark Streaming.
• Authoring Python (PySpark) Scripts for custom UDF’s for Row/ Column manipulations, merges, aggregations, stacking, data labeling and for all Cleaning and conforming tasks.
• Migrate data from on-premises to AWS storage buckets. Developed data analysis tools using SQL and Pythoncode.Developed various Mappings with the collection of all Sources, Targets, and Transformations using Informatica Designer.
• Developed Mappings using Transformations like Expression, Filter, Joiner and Lookups for better data messaging and to migrate clean and consistent data.
• Designed and implemented Sqoop for the incremental job to read data from DB2 and load to hive tables and connected to Tableau for generating interactive reports using Hive server2.
• Used Sqoop to channel data from different sources of HDFS and RDBMS. Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats.
• Used Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS using Python and NoSQL databases such as HBase and Cassandra.
• Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS. Used Apache NiFi to copy data from local file system to HDP.
• Worked on Dimensional and Relational Data Modeling using Star and Snowflake Schemas, OLTP/OLAP system, Conceptual, Logical and Physical data modeling using Erwin.
• Automated the data processing with Oozie to automate data loading into the Hadoop Distributed File System.
• Experience in Converting existing AWS Infrastructure to Server less architecture(AWS Lambda, Kinesis),deploying viaTerraformand AWS Cloud Formation templates.
• Architect and design server less application CI/CD by using AWS Server less (Lambda) application model.
• Developedstored procedures/views in Snowflakeand use inTalendfor loading Dimensions and Facts.

Environment: Hadoop, Map Reduce, HDFS, Hive, Spring Boot, Cassandra, Data Lake, Sqoop, Oozie, SQL, Spark, Scala, Java, AWS, GitHub, Big Data Integration.

Confidential, CA

Big Data Engineer

Responsibilities:

• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
• Develop database management systems for easy access, storage, and retrieval of data.
• Perform DB activities such as indexing, performance tuning, and backup and restore.
• Expertise in writing Hadoop Jobs for analyzing data using Hive QL (Queries), Pig Latin (Data flow language), and custom MapReduce programs in Java.
• Exported Data into Snowflake by creating Staging Tables to load Data of different files from Amazon S3.
• Expert in creating Hive UDFs using Java to analyze the data efficiently.
• Responsible for loading the data from BDW Oracle database, Teradata into HDFS using Sqoop.
• Used Oozie scripts for deployment of the application and perforce as the secure versioning software.
• Connected to AWS Redshift through Tableau to extract live data for real time analysis
• Wrote documentation for each report including purpose, data source, column mapping, transformation, and user group.
• Utilized Waterfall methodology for team and project management
• Used Git for version control with Data Engineer team and Data Scientists colleagues. Involved in creating Created Tableau dashboards using stack bars, bar graphs, scattered plots, geographical maps, Gantt charts etc. using show me functionality.Dashboards and stories as needed usingTableauDesktop andTableauServer.
• Created Session Beans and controller Servlets for handling HTTP requests from Talend
• Performed Data Visualization and Designed Dashboards with Tableau and generated complex reports including chars, summaries, and graphs to interpret the findings to the team and stakeholders.
• Responsibilities include gathering business requirements, developing strategy for data cleansing and data migration, writing functional and technical specifications, creating source to target mapping, designing data profiling and data validation jobs in Informatica, and creating ETL jobs in Informatica.
• Import, clean, filter and analyze data using tools such as SQL, HIVE and PIG.
• Used Python& SAS to extract, transform & load source data from transaction systems, generated reports, insights, and key conclusions.
• Wrote data ingestion systems to pull data from traditional RDBMS platforms such as Oracle and Teradata and store it in NoSQL databases such as MongoDB.Involved in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analyzed them by running Hive queries. Processed the image data through the Hadoop distributed system by using Map and Reduce then stored into HDFS.
• Built APIs that will allow customer service representatives to access the data and answer queries.
• Extending the functionality of Hive with custom UDF s and UDAF's.
• The new Business Data Warehouse (BDW) improved query/report performance, reduced the time needed to develop reports and established self-service reporting model in Cognos for business users.
• Implemented Bucketing and Partitioning using hive to assist the users with data analysis.
• Developed story telling dashboards inTableauDesktop and published them on toTableauServer which allowed end users to understand the data on the fly with the usage of quick filters for on demand needed information.

Environment: Cloudera CDH4.3, Hadoop, Pig, Hive, Informatica, HBase, MapReduce, HDFS, Sqoop,SQL, Tableau, Python, SAS, Flume, Oozie.

Confidential, CA

Data Engineer Intern

Responsibilities:

• Created consumption views on top of metrics to reduce the running time for complex queries.
• Compare the data in a leaf level process from various databases when data transformation or data loading takes place.
• Needed to analyze and look into the data quality when these types of loads are done, to look for any data loss, data corruption.
• As a part of Data Migration, wrote SQL Scripts for Mismatch of data and worked on loading the history data from Teradata SQL to snowflake.
• Created Metric tables, End user views in Snowflake to feed data for Tableau refresh.
• Generated Custom SQL to verify the dependency for the daily, Weekly, Monthly jobs.
• Using Nebula Metadata, registered Business and Technical Datasets for corresponding SQL scripts
• Created performance dashboards in Tableau/ Excel / Power point for the key stakeholders
• Incorporated predictive modeling (rule engine) to evaluate the Customer/Seller health score using python scripts, performed computations, and integrated with the Tableau viz.
• Developed SQL scripts to Upload, Retrieve, Manipulate and handle sensitive data (National Provider Identifier Data I.e.
• Name, Address, SSN,Phone No) in Teradata, SQL Server Management Studio and Snowflake Databases for the Project
• Implemented Defect Tracking process using JIRA tool by assigning bugs to Development Team
• Involved in Functional Testing, Integration testing, Regression Testing, Smoke testing and performance Testing. Tested Hadoop Map Reduce developed in python, pig, Hive
• Developed spark code and spark-SQL/streaming for faster testing and processing of data.
• Analyzed marketing campaigns from various perspectives including CTR, conversion rates, seasonal/geographical trends, search queries, landing page, conversion funnel, quality score, competitors, distribution channel, etc. to achieve maximum ROI for clients.
• Evaluated the traffic and performance of Daily deals PLA ads and compare those items with non-daily deal items to see the possibility of increasing ROI. suggested improvements and modify existing BI components (Reports, Stored Procedures)
• Experienced in working with spark ecosystem using Spark SQL and Scala queries on different formats like text file, CSV file.

Environment: Hadoop, MapReduce, Hive,Apache Spark, SqoopSnowflake,, Nebula, Teradata, SQL Server, Python, Pig, GitHub, Teradata, Tableau, MS Excel, MS Power Point.