Sr Consultant Big Data Engineer Resume
Dallas, TX
PROFESSIONAL SUMMARY:
- 7 years of professional IT experience (Business Analysis, Business Intelligence, Troubleshooting) in with 2 years of Big Data Engineering and Analysis
- Experience with Cloudera Distribution Platform.
- Hands on experience on major components in Hadoop Ecosystem like Hadoop, Map Reduce, HDFS, Impala, Spark (SparkCore, SparkSQL, SparkStreaming), Hive, Hbase Sqoop, Oozie and Flume.
- In depth understanding/knowledge of Hadoop - bases platform and various components such as HDFS, JobTracker, TaskTracker, NameNode, DataNode and MapReduce concepts.
- Experience in implementing complex ETL transformations in Hadoop platform.
- Strong Experience in Unix shell scripting to automate file preparation and database loads.
- Experience in managing and reviewingHadoop Log files.
- Experience with importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa.
- Performance optimization of Sqoop, Hive and Spark.
- Understanding Hive and Pig core functionality by writing custom UDFs.
- Experienced with the integration of various data sources and file types (RDBMS, Json, Parquet, Avro, Text, CSV and 3rd Party feeds).
- Experienced in analyzing data using Scala and Python with HiveQL, RDD, DataFrames, D-streams, Pandas and custom MapReduce programs.
- Experience with Scala and SQL in application development and deployment.
- Experienced in configuring flume and kafka to stream data into HDFS.
- Proficient with NoSQL database platform
- Experience in SDLC - Agile and Waterfall methodology.
- Experience in Web Services using XML and HTML.
- Ability to blend technical expertise with strong Conceptual and Analytical skills to provide quality solutions and result-oriented problem-solving technique and leadership skills.
- Excellent communication skills; creative, conceptual, research-minded, technically competent and result-oriented with problem solving and leadership skills.
- Experience creating embedded Power BI reports using MS Azure and visual studio.
- Experience with Power BI data modelling and DAX programming language.
- Experience with Data Visualization with SSRS, Power BI Desktop, and Power BI Services.
TECHNICAL SKILLS:
Big Data Ecosystem: Hadoop, Spark, HDFS, Map Reduce, Hive, Impala, HBase, Zookeeper,Sqoop, Kafka and Flume
Methodologies: Agile and Waterfall
Programming Languages: Scala, Python, Linux Shell scripting, T-SQL, SQL, DAX, Power Query (M)
Database Systems: MS SQL, MS-Access, MySQL, SSAS TABULAR, Hbase, Cassandra.
Data Warehouse Tools: Management Studio/Enterprise Manager, Query Analyzer, SSIS,SSAS, SQL Profiler.
Operating Systems: Windows 2003/2008/ XP/Vista/7/8/10 Server, UNIX, Linux(Ubuntu), and Mac OS.
Reporting Tools: Power BI, Excel, SQL Server Reporting Services 2012/2016.
Web Technologies: JDBC, XML
Tools: MS SQL Developer, AZURE, Microsoft Office suite desktop applications (e.g. MS Visual Studio, Word, Excel, PowerPoint), MS Power BI, PowerApps, MS Flow, Toad
PROFESSIONAL EXPERIENCE:
Confidential
Sr Consultant Big Data Engineer
Responsibilities:
- Created extract jobs using Sqoop for both history and incremental data to migrate and process large data
- Performed data cleanse and enrichment using Spark with performance optimization.
- Experienced in Hive, created managed and external tables; performed optimization; used partitioning and bucketing; used hive vectorization; used explain plan for query plan and reviewing stages, used various join types.
- Experienced in ingestion and analysis of structured and unstructured data using Spark and SparkSQL.
- Experienced in real time streaming with Kafka, Flume and Spark Streaming. Used streamingcontext with both Basic sources like files /sockets and advance sources like kafka and Flume for live data integration and processing.
- Loaded and transformed large sets of structured, semi structured and unstructured data.
- Performance optimization of Sqoop jobs.
- Defined job flow using Oozie.
- Managed and reviewed Hadoop log files.
- Worked with Columnar NoSQL (HBase and Cassandra) database.
- Supported Map Reduce Programs those are running on the cluster.
- Involved in loading data from UNIX file system to HDFS.
- Installed and configured Hive and written Hive UDFs.
- Worked closely with multiple Data Systems to access information for testing
Environment: Cloudera, Hadoop, Spark, Hive, Linux, HBase, Sqoop, scala, Oozie, SQL Server.
Confidential, Richardson, TX
Big Data/ SQL BI/Power BI Engineer
Responsibilities:
- Designed, engineered and built data platform solutions using Big Data Technologies
- Established and communicated fit for purpose analytical platforms for business prototypes.
- Used Sqoop to dump data from relational database into HDFS for processing.
- Performance optimization of Sqoop jobs.
- Imported and exported full and incremental data into HDFS and Hive using Sqoop.
- Explored, investigated, recommended, and implemented data-centric technologies for the platform by running POCs for streaming data with kafka, flume and sparkstreaming.
- Used Scala and Hive in the analysis of data.
- Experienced in managing and reviewing Hadoop log files.
- Loaded and transformed large sets of structured, semi structured and unstructured data.
- Managed data coming from different sources and loaded data from UNIX file system to HDFS.
- Provided high level technical support in resolving technical issues involving Microsoft PowerBI, SQL Azure and associated products and services.
- Executed and supported processes and jobs to meet deliverables in a timely manner per SLA.
- Developed impressive Power BI visualizations and dashboards using Sparksql/ Impala/ hiveql connectors and managing related issues.
- Performed Power BI desktop installation, gateway installing, and configuration.
- Performed Power BI data modelling, adding measures and calculated columns.
- Created and supported Power BI embedded reports.
- Facilitated delivering of Power BI, MS Flow, SQL Server, BigData and PowerApps program curriculum for new agents SQL Connectivity
- Provide advanced technical support to resolve issues related to SQL Server issues related to SQL Server connectivity, linked servers, data movement, and SQL BI related to SSRS.
- Develop solutions for SQL Server issues related to Installation, Performance and troubleshooting SQL On-Premise environments.
- Assisted customers with configuring Kerberos and Windows authentication.
Confidential, Dallas, Tx
SQL/BI Developer
Responsibilities:
- Built complex T-SQL queries using concepts like joins, case statements and functions to retrieve and analyze data from multiple tables in the database.
- Utilized various SQL Server Constraints (Primary Keys, Foreign Keys, Defaults, Unique keys and Check) to ensure data integrity.
- Extensive use of T-SQL in writing Stored Procedures, Triggers, Tables, User Defined Functions, Views and Indexes.
- Performed extraction, transformation, and loading (ETL) using SSIS
- Created, deployed and managed C-level reports using SSRS
- Backed up database file for each client to keep track of their business data in main database.
- Created ad-hoc queries using T-SQL in SSMS to assist in support of users and everyday tasks.
- Troubleshoot problems and answered questions, installing upgrades and patches
- Assist with documentation for steps to be taken while upgrading the SQL server.
- Assist with the implementation of SQL Logins, Roles and Authentication Modes as a part of Security Policies for various categories of users.
- Assist with checking SQL Log, Activity log and Error Log for troubleshoot the hardware and software issue within specific time frame individually.
Confidential, Dallas, Tx
Business Analyst
Responsibilities:
- Performed regulatory Requirement and Reporting needs.
- Elicitation and analysis of business needs, translating them into functional requirements.
- Interacted with various levels of authority when acting as a liaison between other departments/businesses especially senior leadership at Meridian
- Experienced drafting requirements at all levels in a project (use-cases, functional, and non-functional requirements).
- Facilitated requirements gathering sessions with multiple stakeholders
- User interface prototyping for web-based products
- Ensured reporting design and content conforms to HIPAA standards
- Experienced working in Agile and Waterfall development methodologies
- Performed Data Mapping for Dashboard reports
- Performed Screen Mock ups for Customized solutions
- Documented BRD and FRD
- Assisted with User Acceptance Testing
- Assisted Developers and Testers during various phases of the Project
- Analyze QA defects and assist with Testing Team with seeking Clarification from Business