SUMMARY

• Around 6 years of experience in Data Analysis,Data Modeling,Data Architect,BigData/Hadoop professional with applied information Technology.
• Defined real time data streaming solutions across the cluster using Spark Streaming, Apache Storm, Kafka and Flume.
• Highly skilled and experience in installing, configuring, and using inBigDatacore components and Hadoop Ecosystem, HDFS, MapReduce, YARN, Hive, HBase, Snowflake, Pig, Sqoop, Flume.
• Experience in working with MapReduce programs using Apache Hadoop for working with BigData.
• Strong experience working with conceptual, logical and physicaldatamodeling considering Metadata standards.
• Experience in NoSQL Column - Oriented Databases like HBase and its Integration with Hadoop cluster.
• Working Knowledge in Amazon AWS cloud which includes services like: EC2, S3, EBS, ELB, AMI, IAM, Route53, Autoscaling, CloudFront, CloudWatch, Security Groups.
• Good knowledge in Database with Oracle, Teradata, Netezza, DB2, Snowflake and SQL Server databases.
• In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS, JobTracker, TaskTracker, NameNode, DataNodes and MapReduce concepts.
• Strong knowledge in working with UNIX/LINUX environments, writing shell scripts and PL/SQL Stored Procedures. used Snowflake as our main data warehouse and AWS as our cloud provider
• Knowledge on Airflow and Snowflake bigdata tools, for scheduler for ETL tasksand a Data warehouse tool.
• Hands on experience with popular cloud-based data warehouse platforms such as, Redshift, Snowflake.
• Developed Apache Spark jobs using PySpark in test environment for fasterdataprocessing and used Spark SQL for querying.
• Worked with Oozie workflow engine to schedule time-based jobs to perform multiple actions.
• Experienced in importing and exportingdatafrom RDBMS into HDFS using Sqoop.
• Hands on experience in working with database like Oracle, MySQL and PL/SQL.
• Experienced in developing Web Services with Python programming language.
• Knowledge of continuous integration software (Jenkins) and version control systems (GIT).
• Experienced with Linux based commands of BASH and Shell/Perl Scripting.
• Experience working with Agile and Waterfalldatamodeling methodologies.
• Ability to learn, R&D and adapt new technologies, applications and products, understanding of business workflow and deliver outputs in short deadlines

TECHNICAL SKILLS

Hadoop Ecosystem: HDFS, Hive, MapReduce, Oozie, Pig, Sqoop, Flume, Spark.

SQL Databases: Mysql, Sql Server

NoSQL Databases: Hbase,, Cassandra

Operating Systems: Linux, Windows

Languages: Python, Java, Sql, Pig, Latin, HiveQL, Unix shell scripts

Version control: SVN, Git

Hadoop Distributions: Cloudera, Hortonworks

AWS cloud: EC2, EMR, S3, Redshift, Snowflake

PROFESSIONAL EXPERIENCE

Confidential

Big Data Engineer

Responsibilities:

• Designed and implemented scalable, distributed systems leveraging Big data technologies like Hadoop, Hive, and scripting languages like Python and Shell Script. Discussed various scenarios to find out which solution fits the problem statement in hand throughout the meetings.
• Defined the setup of Hadoop environment and defined the architecture with the provided and available resources. Coordinated the tools which helps in solve the problem statement considering the type of data to deal with.
• Created data pipelines to establish the data flow from various available sources and transform the data then send to the defined targeted areas. Understanding database internals, such as indexes, logs, and transactions.
• Modifying and cleansing the data and convert the data into valuable information using sophisticated methods MapReduce through query language platform like hive. Executed advanced data analytic methods and complex SQL queries.
• Worked with queries to create tables form the available data and to transform them using profound queries to meet the business requirement. Performing exploratory data analysis to determine which questions can be answered effectively with a given dataset.
• Created Syncs for Data Ingestion, transformation, and extraction utilizing Datawarehouse technologies such as Snowflake.
• Written SQL queries against Snowflake Developing scripts Unix, Python to do extract. Load and transform data.
• Develop and automate data quality checks. Perform root cause analysis on internal and external data.
• Articulated pros and cons clearly to provided suggestions and guidance on what data platform techniques are appropriate to solve problems. Understanding their needs of customers and partners and engage with Data Platform.

Environment: Hadoop, HDFS, Pig, Hive, Spark, Scala, HBase, Oozie, Sqoop, Flume, Snowflake, Linux, Java, Maven, Junit, GitHub, Jenkins.

Confidential, OH

Big Data Engineer

Responsibilities:

• Involved in the configuration of Apache Hadoop clusters for application development and Hadoop tools like Hive, HBase, Zookeeper and Sqoop.
• Involved in design, development, integration, deployment/production support & other technical aspects of the development of, modification to the applications
• Created Informatica source-to-target mapping using different transformations to implement business rules to fulfill thedataintegration requirements.
• Prepared DML to perform audit & error handling.
• Provided project estimates, coordinated the development efforts and discussions with the business partners, updated status reports and handled logistics to enable smooth functioning of the project and meet all the deadlines.
• Worked on SQL developer querying the source/target tables to validate the SQL and Lookup override.
• Performing unit testing and responsible for System Integration testing, Regression testing until it moves to production
• Installed and configured Hadoop MapReduce, HDFS
• Importing and exportingdatainto HDFS and Hive using Sqoop.
• Experienced in defining job flows. Experienced in managing and reviewing Hadoop log files.
• Experienced in running Hadoop streaming jobs to process terabytes of xml formatdata.
• Load and transform large sets of structured, semi structured and unstructureddata.
• Supported Map Reduce Programs those are running on the cluster.
• Installed and configured Hive and written Hive UDFs.
• Involved in creating Hive tables, loading withdataand writing Hive queries which will run internally in map reduce way.
• Developing Web Services with Python programming language.
• Involved in building and maintaining test plans, test suites, test cases, defects and test scripts
• Conducted functional, system,dataand regression testing.
• Involved in Bug Review meetings and participated in weekly meetings with management team.

Environment: Hadoop, HDFS, Coludera, Pig, Hive, Spark, SparkSQL, Scala, HBase, Oozie, Sqoop, Flume, Kafka, Linux, Java, Maven, Junit, GitHub, Jenkins.

Confidential, CT

Hadoop developer

Responsibilities:

• Involved in the configuration of Apache Hadoop clusters for application development and Hadoop tools like Hive, HBase, Zookeeper and Sqoop.
• Involved in shell scripts to monitor the health check of Hadoop daemon services and respond accordingly to any warning or failure conditions.
• Involved in collecting and aggregating large amounts of logdatausing Apache Flume and staging datain HBASE/HDFS for further analysis.
• Collected the logsdatafrom web servers and integrated in to HBASE using Flume.
• Real-time streaming thedatausing Spark with Kafka.
• Used Pig & Python scripting for pre-processing thedata.
• Used Sqoop to import and exportdatafrom HDFS to RDBMS and vice-versa.
• Worked on importing and exportingdatafrom Oracle and DB2 into HDFS and HIVE using Sqoop.
• Used Hive in partitioning, bucketing and perform different types of joins on Hive tables and implementing Hive serDes like JSON, Avro and REGEX.
• Pulleddatafrom various databases into Hadoop and automated workflows using shell scripts.
• Developed Sqoop scripts to importdatafrom relational sources and handled incremental loading.
• Help in QA testing and in writing the Python scripts.
• Developed scalable, Hadoop-baseddataprocessing algorithms using HBase, Pig, Hive, Map Reduce and the Hadoop ecosystem.
• Deployed Hadoop Cluster in Pseudo-distributed and Fully Distributed modes.
• Supported in setting up QA environment and updating configurations for implementing scripts with Pig, Hive and Sqoop.
• Involved in writing scripts to automate the process and generate reports.

Environment: Hadoop, MapReduce, Spark, Java, Hive, HDFS, PIG, Sqoop, Kafka, Oozie, Cloudera, Flume, HBase, Zookeeper, CDH4&CDH5, Oracle, PL/SQL, Linux