Data Engineer Resume
Milwaukee, WI
SUMMARY
- 7+ long periods of IT experience and over 3+ years in Big Data Analytics as a Big Data Engineer wif great information in Hadoop ecosystem system innovations. Basically, on setting up clusters, information Extraction, Transformation and Loading.
- Proficiency in applying various mathematical and statistical models to include, but not limited to: discrete event simulation, factor analysis, genetic algorithms, Bayesian probability models, hidden Markov models, sensitivity analysis, sampling, probability, multivariate data analysis, regression, PCA, time - series analysis.
- Concentrated ability in scripting utilizing Bash, Python and Shell. Involvement wif test teh board devices HP ALM and programming techniques like Agile, Waterfall model.
- Exposure in Dev Ops innovation like Chef, Puppet, Ansible, Jenkins
- Experienced in building AWS Data Lakes, Redshift Data warehouse and PySpark ETL processes to ingestions and transformations
- Experienced on AWS cloud services (VPC, EC2, S3, RDS, Redshift, Glue, Data Pipeline, EMR, DynamoDB, Lambda, Kinesis, RDS, SNS, SQS)
- Worked in various Hadoop conveyances like Horton Works, AWS, Cloudera and MapR.
- Have 3+ long stretches of exhaustive involvement wif Big Data preparing to utilize Hadoop and its environment (MapReduce, Pig, Hive, Sqoop, Flume, Spark).
- Great working experience on Spark (flash streaming, sparkle SQL), Scala and Kafka.
- Involvement wif executing different Big Data Analytical, Cloud Data designing, Data Visualization, Reporting, Data Quality, and Data virtualization arrangements.
- Knowledgeable wif Big information on AWS cloud administrations for example EC2, S3, Glue, DynamoDB and RedShift
- Experience in teh development and design of different versatile frameworks utilizing Hadoop advancements in different conditions. Broad involvement wif investigating information utilizing Hadoop Ecosystems including HDFS, MapReduce, Hive, and PIG.
- Expert in providing ETL solutions for any type of business model.
- Experience in understanding teh security necessities for Hadoop.
- Skilled in planning and executing ETL Architecture for a financially savvy and proficient climate.
TECHNICAL SKILLS
Languages: Python, R, Java, SQL, Scala
Big Data Technologies: Hadoop, HDFS 2, Hive, Pig, HBase, Sqoop, Flume, MapReduce, Spark, HTTPFS
Cloud Service: AWS
Reporting Tools: Business Objects, Crystal Reports.
ETL Tools: Tableau, Teradata Talend, Apache NiFi, Air Flow
Big Data: Hadoop, Hive, Mongo DB, Apache Spark, Scala, Pig, Sqoop
Data Engineer tools: AWS, Glue, Athena, ETL, Lambda, Pandas, PySpark, MapReduce, Plotly, ggplot, matplotlib, Jupyter, Git, Scikit-learn, Numpy, Tableau, PowerBI, Docker
Data Modeling: Dimensional Data Modeling, Star Join Schema Modeling, Snowflake Modeling
Versioning\ CI-CD: SVN, GIT, SourceTree, Bitbucket, Bamboo, JIRA, Confluence
SQL: ANSI SQL, T-SQL, PL/SQL, BTEQ
Ticketing Tools: JIRA, Service Now
Splunk Modules: Splunk 6.1.3, Splunk on Splunk, Splunk Enterprise, Splunk DB Connect, Splunk Cloud, Splunk Web Framework, Splunk Enterprise Security, Splunk IT service intelligence, Splunk user behavior analytics, Splunk Universal forwarder, Cisco for Splunk, Windows for Splunk, Splunk UBA, VMware for Splunk, Splunk DB, SSIS, SSAS, SSRS, Business Objects, Power BI, ArcSight
PROFESSIONAL EXPERIENCE
Data Engineer
Confidential, Milwaukee WI
Responsibilities:
- Involved in changing over Hive/SQL inquiries into changes utilizing Python
- Developed data engineering python scripts for ingestion pipelines which run on AWS infrastructure setup of EMR, Glue, Lambda, S3 and step functions
- Implemented Performance testing using Apache JMeter and created a Dashboard using Grafana to view teh Results.
- Experience on Splunk architecture and various components (indexer, forwarder, search head, deployment server) Heavy and Universal forwarder, License model.
- Expert in installing and using Splunk apps for UNIX and LINUX.
- Big Data - Hadoop (MapReduce & Hive), Spark (SQL, Streaming), Azure Cosmos DB, SQL Datawarehouse, Azure DMS, Azure Data Factory, AWS Redshift, Athena, Lambda, Step Function and SQL.
- Made and kept up technical documentation for dispatching HADOOP Clusters and for executing Hive questions and Pig Scripts.
- Developed and deployed teh outcome using spark and Scala code in Hadoop cluster running on GCP.
- Experiencewif various ingestion techniques to bring into R, Python and Azure ML environment from different big data platforms such as a HDFS, Hadoop
- Experience wif command-line scripting, data structures and algorithms and ability to work in a Linux environment, processing large amounts of data in a cloud environment.
- Strong noledge in Airflow ecosystems such as Spark core, Spark SQL, Spark Streaming libraries.
- Very Good experience working in Azure Cloud, Azure DevOps, Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Azure Analytical services, Azure Cosmos NO SQL DB, Azure HD Insight BigData Technologies (Hadoop and Apache Spark) and Data bricks.
- Worked on KAGGLE data sets and Microsoft Azure ML predictive models as a part of Data science Boot camp
- Worked on Azure databases, teh database server is hosted on Azure and use Microsoft credentials to login to teh DB rather than teh Windows authentication dat is typically used
- Experience in machine learning, artificial intelligence and/or artificial neural networks.
- Broad understanding of databases (e.g. SQL, NoSQL, Lucene, Mongo), and high-performance or distributed processing (e.g. using MapReduce, Spark, Pig, and/or Hive).
- Experience wif visualization software (Tableau, D3, MicroStrategy, PowerBI).
- Experience delivering solutions in an Agile environment.
- Experience wif Tensorflow, Theano or Keras.
- Designed, implemented, and tested teh Spring Domain Model for teh services using Core Java.
- Involvement wif creating SQL and PL/SQL contents. Openness on BI Tools like Tableau and QlikView.
- Experienced in handling big data on teh Apache Hadoop frameworks by utilizing MapReduce programs.
- Involved in working wif Windows, UNIX/LINUX stage wif various advancements, for example, Big Data, SQL, XML, HTML, Shell Scripting etc
- Associated wif document developments among HDFS and AWS S3 and widely worked wif S3 pail in AWS
- Extensively worked on java multi-threading programming, Synchronization, Java Mail API and Collections API for concurrent product models upload/download process.
- Client-side validations written in JavaScript.
- Analyzed data and performed data preparation by applying historical model on teh data set in AZURE ML
- Worked on making Python application to devour messages from AWS SQS measure teh ask for and send suitable AI model outcomes to it
- Installed, configured, monitored, and maintainedHadoopcluster on Big Data platform.
- Worked on analyzing Hadoop cluster and different big data analytic tools such asHiveQL.
- Developed data pipelines from moving source data (from DB2 LUW/Oracle) to google cloud platform, and transforming data in google Big Query
- Assist end users wif problems installing thePower BI desktop, installing, and configuring teh Personal and On-Premise’s gateway, connecting to data sources, and adding teh different users.
- Experienced in developingPower BIreports and dashboards from multiple data sources using data blending.
- Complete noledge of Data Warehouse Methodologies Ralph Kimball & Bill Inmon Methodologies. Experienced in buildingODS, EDW, Staging and maintaining metadata repository.
- Worked on AWS Data Pipeline to configure data loads from S3 to into Redshift
- UtilizedPower QueryinPower BItoPivot and Un-pivotthe data model for data cleansing and data massaging
- Proficiency in designing & implementation of new ERP systems / procedures to enhance overall efficiency of teh organization.
- Assist in teh implementation of new procedures or policies to ensure compliance wif established financial accounting requirements.
- Develop, analyze, and conduct studies for addressing complex and critical systems integration problems and issues involving financial management systems.
- Worked wif Reporting engineers to direct teh execution of reports/dashboard plans in Tableau.
- Worked on complex SQL Queries, PL/SQL techniques and convert them to ETL assignments
- Worked onAzure Data Factoryto orchestrate teh ingestion of data from various sources likeAzure tables, Azure SQL Database, Azure SQL DWH, Azure Storage Blobs, and on-premises databasestoAzure Data Lake Store.
- Implemented scripts dat heap Google Big Query data and runs queries to export data
- Involved in gathering requirements from business users and createdMultidimensional model using Erwin.
- Using Erwin for Logical and Physical database modelling by using Ralph Kimball & Bill Inmon Data Warehouse Methodologies.
- Converted teh existingAzure Data Factory V1toAzure Data Factory V2and implemented SSIS package in Azure environment usingIntegration Runtime(IR).
- Experience in making SSIS Packages utilizing Pivot Transformation, Fuzzy Lookup, Derived Columns, Condition Split, Term extraction, Aggregate, Execute SQL Task, Data Flow Task, and Execute Package Task and so forth to create hidden information for teh reports and to send out purged information.
- Created PySpark frame to bring data from DB2 to Amazon S3.
- Designed APIs to stack information from Omniture, Google Analytics, Google Big Query
- Worked on making information ingestion cycles to keep up Global Data Lake on teh GCP cloud and Big query.
- Planned and created Power BI graphical and representation arrangements wif business necessity records and plans for making intelligent dashboards.
- Worked on Agile Projects Inventory Control, Mass Adjustment, Price Markdown Report and Physical Inventory Reconciliation Tool
- Provide guidance to teh development team dealing wif PySpark as an ETL stage
- Expertise in Informatica cloud applications Data Synchronization (ds), Data Replication (dr), Task Flows, Mapping setups, Real-Time applications like process designer and process developer.
- Implemented a few DAX capacities for different actuality computations for proficient information representation in Power BI.
- Done configure diagnostics, monitoring and analytics on Azure platform along wif scale and resilience for Azure websites.
- ETL Data Cleansing, Integration &Transformation utilizing Pig: Responsible for overseeing information from unique sources.
- Educate developers on how to commit their work and how can they make use of teh CI/CD pipelines dat are in place.
- Setup full CI/CD pipelines so dat each commit a developer makes will go through standard process of software lifecycle and gets tested well enough before it can make it to teh production.
- Worked on google cloud platform (GCP) services like compute engine, cloud load balancing, cloud storage, cloud SQL, stack driver monitoring and cloud deployment manager.
- Implemented revealing Data Warehouse wif online exchange framework information.
- Identify and approve information among source and target applications
- Worked on most basic Finance projects and had been teh go-to individual for any data-related issues for team members
- Involved in separating client's Big Data from different information sources into AWS S3 for putting away information. dis included information from centralized computers and data sets from sources.
- Effective software development processes to customize and extend teh computer vision and image processing techniques to solve new problems for Automation Anywhere.
- Develop and implement innovative data quality improvement tools.
- Created diverse Power BI reports using teh work area and teh online help and timetable invigorate.
Data Analyst
Confidential, baltimore, MD
Responsibilities:
- Worked as a Data analyst, to make a deal and administration information store needed for Business Intelligence and announcing purposes. Teh Existing STARSS framework which permits teh specialists to enter, adjust new deals/administration information is utilized as one of significant source frameworks.
- Expertise include Agile - Scrum and waterfall methodology, testing tools and automation, and managing teh product lifecycle, sprint planning, sprint review and sprint retro and user story prioritization
- Experience in designing, developing, and deploying projects in GCP suite including GCP Suite such as
- Big Query/ Bigtable, Data Flow, Data proc, Google Cloud Storage, Composer, Looker etc.
- Proficient in writing Cloud Formation Templates (CFN) in YAML and JSON format to build teh AWS services wif teh paradigm of Infrastructure as a Code
- Developed and implemented teh SQL Server Integration Services SSIS packages to populate teh data into Enterprise Data Warehouse from various sources like Flat File Source, Excel Source into OLEDB destination SQL Server.
- 5+ years of experience as Azure Cloud Data Engineer in Microsoft Azure Cloud technologies including Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse Analytics (SQL Data warehouse), Azure SQL Database, Azure Analytical services, Polybase, Azure Cosmos NoSQLDB, Azure Key vaults, Azure Devops, Azure HDInsight BigData Technologies like Hadoop, Apache Spark and Azure Data bricks.
- Experience in designing Azure Cloud Architectureand Implementation plans for hosting complex application workloads on MS Azure.
- Experience working in reading Continuous json data from different source system using Kafka into Data bricks Delta and processing teh files using Apache Structured streaming, PySpark and creating teh files in parquet format.
- Understanding requirements, significance GCP of weld point data, energy efficiency using large datasets
- Understanding teh business problems and analyzing teh data by using appropriate Statistical models to generate insights.
- Developed NLP models for Topic extraction, Sentiment Analysis
- Identify and assess available machine learning and statistical analysis libraries (including regresses, classifiers, statistical tests, and clustering algorithms).
- Setup Alerting and monitoring using Stack driver in GCP.
- Design and build scalable software architecture to enable real-time / big-data processing.
- Worked on google cloud stage (GCP) administrations like process motor, cloud load adjusting, cloud capacity, cloud SQL, stack driver checking and cloud sending chief. Utilized different sources to maneuver information into Power BI like SQL Server, prophet.
- Created SSIS Packages to import and fare information from Excel, Text documents and changed information from different information sources utilizing OLE DB association by making different SSIS package.
- Experienced in creating Power BI reports and dashboards from various information sources utilizing information mixing.
- Worked on a migration project to migrate data from different sources to Google Cloud Platform (GCP) using UDP framework and transforming teh data using Spark Scala scripts.
- Worked on extracting event data from KAFKA
- Created measure records, detailing specs, and layouts, preparing material and slideshow introductions for teh application advancement groups and administrations
- Strong IT experience in teh field of Data investigation, ETL Development, Data Modeling, and Project Management wif experience in Big Data and related Hadoop advancements.
- Ability to team up wif peers in both, business, and specialized zones, to convey ideal business measure arrangements, in accordance wif corporate needs.
- Worked widely in archiving teh Source to Target Mapping records wif information change rationale.
Environment: SQL/Server, Oracle 9i, Kafka, MS-Office, Teradata, Docker, EMR, Hive, Jenkins, Splunk, Informatica, ER Studio, XML, Business Objects
Jr. Data Engineer
Confidential
Responsibilities:
- Worked on analyzing Hadoop cluster and distinctive huge information analytic tools including Pig, Hive, and Sqoop. Answerable for building versatile disseminated information arrangements utilizing Hadoop Cloudera.
- Associated wif bringing in and sending out information (SQL Server, Oracle, CSV, and text record) from nearby/outside document framework and RDBMS to HDFS. Burden log information into HDFS utilizing Flume.
- Implemented Event Handlers and Error Handling in SSIS packages and educated cycle results to various customer organizations.
- Traded teh dissected information to teh social data sets utilizing Sqoop for perception and to create reports for teh BI group.
- Utilized Sqoop to productively move information among data sets and HDFS and utilized Flume to stream teh log information from workers.
- Teh Hive tables made according to necessity were overseen or outside tables characterized wif proper static and dynamic parts, expected for proficiency.
- Teh Hive tables made according to prerequisite were overseen or outside tables characterized wif proper static and dynamic parts, planned for effectiveness.
- Expertise in Informatica cloud applications Data Synchronization (ds), Data Replication (dr), Task Flows, Mapping arrangements, Real-Time applications like interaction originator and cycle designer.
- Work broadly wif level records. Stacking them into on-premises applications and recover information from applications to records.
- Overall responsibility for assuring efficient and effective use of teh ERP system
- Built-up various Kafka Producers and Consumers according to teh software product prerequisite particulars.
- Utilized Kafka for log collection like physical actual log archives off servers and spots them in a central spot like HDFS for dealing wif.
- Examined clickstream data from Google Analytics wif Big Query.
- Information Warehousing: Designed an information distribution center utilizing Hive, made and oversaw Hive tables in Hadoop.
- Made different UDF capacities in Pig and Hive to control teh information for different calculations.
- Streamlined inquiry execution by prophet hints, compelling files, working wif limitation-based stacking, and few different methodologies.
- Broadly dealt wif UNIX Shell Scripting for a parting gathering of records to different little documents and document move mechanization.
- Liable for making various applications for perusing teh information from various Oracle cases to Kafka subjects
- Effectively engaged wif code survey and bug fixing for improving teh exhibition.
- Chipped away at bringing in and sending out information from snowflake, Oracle and DB2 into HDFS and HIVE utilizing Sqoop for investigation, perception and to produce reports.
Environment: Hadoop, Map Reduce, Hive, HDFS, PIG, Sqoop, Hortonworks, Flume, HBase, Oracle, Snowflake, Teradata, Tableau Unix/Linux, Hadoop, Hive, PIG, SQOOP, Flume, HDFS, Oracle/SQL & DB2, Unix/Linux, JIRA, AWS
GRADUATE ASSISTANT
Confidential
Responsibilities:
- Executed change teh executives' methods for web content Developed and looked after SQL Worker data set of investment worries in Texas.
- Responsible for gathering, cleaning, marking, and conceptualizing a huge information base.
- Using SQL inquiries, Python libraries, MS Access, and MS Excel to lter and clean.
- Predicted datasets and approved wif information mining instruments to guarantee to coordinate wif depiction wif occurrences.
- Created data set items, including tables, sees, and emerged sees.
- Responsible for making examination reports, keeping up and discussing wif lab assistance.
- Acquired openness to Data warehousing ideas and instruments like Brio Query, SQL Server.
- Dealt wif teh course blackboard website and utilized teh internet showing devices and projects.
