We provide IT Staff Augmentation Services!

Big Data Engineer Resume

0/5 (Submit Your Rating)

Dallas, TX

SUMMARY:

  • An experienced Big Data - Hadoop developer with varying level of expertise around different Big Data/Hadoop ecosystem projects which include HDFS, MapReduce, HIVE, HBase, Storm, Kafka, Flume, Sqoop, ZooKeeper, Oozie, Pig etc.
  • In depth and extensive knowledge of Hadoop platform.
  • Experience in Hadoop cluster sizing and cluster design.
  • Around 17 years of Software Design, Development, Programming experience on Middleware and Client/Server, Multitier, Web based front end applications in Java, J2EE & web services.
  • Expertise in application development using Java, Servlet, JSP, Struts, Spring, Hibernate, JUnit, Log4J, XML, XSD, XSLT, Web Services, ANT, JavaBeans, EJB, JMS, Maven etc.
  • Experience working with different file formats like Apache Avro, Sequence file, JSON, XML, and Flat text file.
  • Worked closely with business clients and partners to understand and business requirements, develop use case and designed solutions.
  • Extensive experience with successful projects execution in On-site and Off-shore models, leading and managing projects and teams independently.
  • Import data from database to HDFS and vice versa using Sqoop.
  • Knowledge of Real-time streaming technologies Spark, Kafka.
  • Extensive 10+ years of experience in large scale developments, deployments and maintenance of middleware applications in Telecommunications industry with AT&T and Confidential .

TECHNICAL SKILLS:

Big Data/Hadoop Technologies: HDFS, MapReduce, Spark, NiFi, Hive, HBase, ZooKeeper, Oozie, Pig, Sqoop, Flume, Apache Avro, Storm, Kafka

Languages: JAVA, PL/SQL, XML, Groovy, Scala

Java & Web Technologies: JSP, Servlets, JDBC, JNDI, JMS, EJB, SOA, Web Services, XML Schema, XSLT

Reports: Jasper

Frame works: Struts, Spring, Hibernate

Scripting: JavaScript

Design Technologies: OOAD with UML (Rational Rose, Microsoft Visio).

Server Technologies: Web Logic, Web Sphere, Apache Tomcat

Operating Systems: Windows, Unix, Linux

Databases: Oracle 10g, MySQL, SQL Server, MS Access. DB2

Tools: Rational Rose (Clear Quest, Clear Case), Eclipse, XML Spy, TOAD, ANT, Jenkins, Maven, MS Office, JUnit, CVS, SVN, SOAP UI, Putty, WinSCP, File Zilla, Splunk, Nexus

Methodologies: Waterfall, Agile and DevOps

PROFESSIONAL EXPERIENCE:

Confidential, Dallas TX

Big Data Engineer

Responsibilities:

  • Develop Sqoop job to move data from Oracle to Hadoop.
  • Develop spark job to validate and cleansing data and then move to Hive table.
  • Designing and development of HIVE tables and hql scripts for hive tables.
  • Develop shell script and schedule a cron job to move delta csv file from ftp location to staging/HDFS after every day.
  • Nifi - to replace FTP job for better performance and maintenance.
  • Develop spark job to read delta csv file and move that data to Hive table.
  • Design and development Oozie job to run spark job on daily bases.
  • Oozie job uses Status checker to check the hive status table after everyone minute to see if data transfer is completed from FTP location to HDFS via Nifi.
  • Develop Oozie job to send email after Spark job completion with status as success or failure with details.
  • Participate in Agile daily scrum calls, interact and work with QC and Hadoop admin team.

Environment: Hadoop, Hive, Oracle, Spark 1.6, Zookeeper, Scala 2.11.8 Sqoop, Oozie, Shell scripting, WinSCP, HUE, Agile, git, HP ALMQC, Maven, Splunk, Tableau for reports, CDH, HDF (Nifi)

Confidential

Big Data Engineer

Responsibilities:

  • Involved in Cluster sizing discussions and cluster design for various component installation on cluster
  • Develop map reduce programs for schema validator and IVP validation to validate different versions of xml request & responses.
  • Created Sqoop integration scripts to ingest data from HDFS to database and database to HDFS.
  • Sqoop ingest data in hive stage environment. Move to Master data set after cleansing using pig and load key meta data information in database.
  • Writing HIVE script to load the payload in external table from HDFS.
  • Writing scripts for creating, truncating, dropping, altering HBase tables to store the data after execution of map reduce job and to use that for later analytics.
  • Cluster Monitoring and Troubleshooting using Ambari. Helping with troubleshooting issues.
  • Develop shell scripts and invoke through program to process operations on HDFS file system.
  • Design and develop controller to invoke splunk and Nexus adapters for log extraction and kit download to extract schema respectively.
  • Schedule nightly batch jobs using Oozie to perform schema validation and IVP transformation at larger scale to take the advantage of the power of Hadoop.

Environment: Windows 2000, Java, Spring, Angular JS, HDFS, Map Reduce, Storm, Kafka, Hive, HBase, Pig, ZooKeeper, Sqoop, Flume, Oozie, HCatalog, Ambari, Shell scripting.

Confidential

Big Data Engineer

Responsibilities:

  • Implement storm topology job to create data streams to send data to bolts and then to solr.
  • Designed and developed storm spout for handling file data stream which then distributed across all bolts for file comparison.
  • Implement Hazelcast distributed queue messaging which is useful for all cluster members interaction.
  • Integration of Hazelcast with storm spout to handle the incoming data to spout.
  • Implement file level comparison during comparison of two Kits.
  • Develop storm spout to handle file stream data coming from SVN, and direct the data streams to various bolts for processing.
  • Design and develop bolt to analyze and compare files coming from data streams from Strom spout.
  • Developer storm bolt to create Solr document with all metadata information and add that on Solr server.
  • Read the tuples, serialize it, perform data processing for comparison and then store the result into the solr.
  • Fetch the comparison result from solr after job completion.

Environment: Windows 2000, Java, HDFS, MapReduce, Storm, Hazelcast, Kafka, Hive, HBase, ZooKeeper, Hive, Sqoop, Flume, Ambari, Angular JS, Spring, Shell scripting etc..

Confidential

Java Middleware Technical Lead

Responsibilities:

  • Extensive coordination with offshore team and deliveries on schedule time with quality.
  • Communicated project status, including project issues up and down the management chain, including senior management.
  • Communicate the project design to other application designers and team members
  • Ensure Process & SLA Adherence of the projects and deliverables based on AT&T standards (ITUP) and agile methodology.
  • Managed change requests, including assisting the sponsor and stakeholders to understand the impact of the change on schedule or features
  • Involved in planning, creating application designs, validating high level designs (HLDs) to ensure accuracy and completeness against the business requirements and programming the solutions and complete unit testing with unit test plans.
  • Responsible to resolve design issues and develop strategies to make ongoing improvements that support system flexibility, performance and metrics reporting.
  • Extensively worked on Java, J2EE, XML, Schema designs and on Weblogic server.
  • Responsible for release level tasks and involvements in ‘Lessons Learned’ meetings and applied these ideas to improve processes for new projects
  • Design validation strategies and review Test Cases, Test Plans prepared by testing team
  • Handled multiple projects simultaneously and communicate requirements and status effectively
  • Delivered results in assigned timeframes that were sometimes quite short
  • Active involvement in the entire Software Development Life Cycle including Design, Development, Testing, Deployment and Support for Software Systems

Environment: Windows XP, Java 1.6, Tibco Business studio, Contivo, Weblogic 10.3, CVS, SVN, XML, XSD, xml spy, Eclipse, Ant, Maven, Nexus, Jenkins, MS Office 2007 products, Putty (UNIX), J2EE, Web Services, Eclipse, JMS, SVN

Confidential

Java Middleware Technical Lead

Responsibilities:

  • Design for new functionality. Lead the team of developer for both the applications.
  • Provided the solution architecture with different systems involved.
  • Involved in the entire development, integration and validation for other components.
  • Responsible for providing technical requirements based on business scope and vision.
  • Interfacing with business team and providing overall design directions.
  • Responsible for Managing, tracking of project.
  • Served as an escalation point for resolving issues with Java/J2EE.
  • Provide support to QA while testing
  • Provide support during deployment and warranty period.
  • Development of features as per the requirements.
  • Integrate/Unit testing.
  • Help test team to run through regress testing and performance testing

Environment: Java, J2ee, JSP, Servlet, EJB 2.0, Weblogic 10.0, Oracle 10g, CQ

Confidential

Java Lead (Responsible for Design and Development)

Responsibilities:

  • Understanding of ATT Call center domain with understanding of wireless ordering process including ordering, activation and fulfillment.
  • Design and Development of Various products such as DSL, Uverse and Unified Billing such as Titan.
  • Architectural for J2ee based open source projects to be used in the existing applications and new components.
  • Recommend architectural strategies and frameworks to the dev. teams and for new features and projects.
  • Conducted code reviews and design reviews, adhering to various design patterns
  • Creating the base framework for the developers to start coding.
  • Feasibility study on various proposed approaches for improving products scalability and maintainability and other enhancements required by client
  • Creating high level and low-level designs for the development teams
  • Issues resolution and providing technical leadership
  • Release management and software configuration management for the application code.
  • Monitoring performance of individual developers with the aim of keeping their motivational levels high through various measures like skip level meetings, fulfilling needs and other motivational procedures.
  • Track progress via daily huddle meetings with the developers and resolve issues and concerns raised in the daily meetings
  • Code Reviews and code refactoring.
  • Involved in transitioning the work from onsite to offshore

Environment: Java, J2ee, JSP, Servlet, EJB 2.0, Weblogic 10.0, Oracle 10g, CQ

Confidential

Java Technical Lead

Responsibilities:

  • Understanding the architecture of middleware applications (WMIG, SWDT, WMFG, ED) components of FFS program.
  • Help the team in Designing and development of application.
  • Estimation and budget planning in collaboration with customer.
  • Guiding the team with the knowledge about the application in fixing the production bugs.
  • Mentoring and Appraisal of the team.
  • Issue resolution and providing technical leadership
  • Release management and software configuration management for code managed for the production team.
  • Conducts weekly call with team, identification of risks and takes to closure.
  • Ensures that client concern/issues are taken to closure.

Environment: Java, J2ee, JSP, Servlet, EJB 2.0, Weblogic 10.0, Oracle 10g, CQ

Confidential

Java Technical Lead

Responsibilities:

  • As a Designer was involved in the following:
  • Involved in understanding the requirement, preparing the design and helping the offshore team in understanding the requirements/design.
  • Development of application.
  • Status reporting to HCL and AEON management about the progress of the project.
  • Development of the above application.
  • Was involved in testing of above development.
  • AEON Security is developed in J2EE using the WebSphere 5.1, SOAP, JSP and DB2 and MySQL as backend.
  • Track progress via daily meetings with the developers and resolve issues and concerns raised by offshore team members in the daily meetings

Environment: Java, J2ee, JSP, Servlet, EJB 2.0, Weblogic 10.0, Oracle 10g, CQ

Confidential

Sr. Software Engineer

Responsibilities:

  • Involved in the requirement study, design and development
  • Bug fixing and support
  • Testing and Code review
  • Design and development of new modules against fresh requirements. These may be new developments or enhancements to the existing code.
  • Testing and Code Reviews
  • Ensuring reviews and timely client delivery

Environment: APSR was developed in Java, JSP, Weblogic and Oracle as back - end.

Confidential

Developer

Responsibilities:

  • Bug fixing and support
  • Developing new modules within the application to increase the usability
  • Testing and Code review
  • Providing 24 X 7 support to production and test instances
  • Design and development of new modules against fresh requirements. These may be new developments or enhancements to the existing code.
  • Testing and Code Reviews
  • Ensuring reviews and timely client delivery

Environment: VB 5.0/6.0, MS-Access, Crystal Reports.

Confidential

Programmer

Responsibilities:

  • Initial Knowledge transfer at Client location
  • Bug fixing and support
  • Developing code for RFC (Request for changes), TAR (Bugs)
  • Testing and code review

Environment: UNIX, Progress 7.3 CUI.

We'd love your feedback!