I build infrastructure in the cloud using software development principles, with security as a top priority. I multiply my own output by creating tools and frameworks. I help others multiply their output by teaching, mentoring, and reviewing code. I fulfill business needs simply and efficiently by making use of existing systems, and extending them only where necessary. I have contributed to the Linux kernel, Kubernetes, Packer, GoReleaser, and other open source projects.
Confidential, Atlanta, GA
Principal Devops Engineer
- Wrote automation to build and maintain Kubernetes clusters in an air gapped environment using CloudFormation, Ansible, and Go.
- Provided leadership and guidance for the conversion from Mesos to Kubernetes for a payroll platform composed of microservices and their supporting infrastructure applications.
- Planned, implemented, and documented the migration from Jenkins to Concourse CI for multiple development teams writing microservices.
- Wrote coding standards documents to ensure that code for the infrastructure is maintainable, testable, and predictable.
- Created standards for structuring Terraform and Ansible source code, to ensure these tools could be run in an idempotent fashion from CI/CD pipelines whenever triggered by git changes.
- Migrated on - prem hosted tools such as the Jira and Confluence Atlassian suite onto Kubernetes in AWS.
Lead Software Engineer
- Led a team of cloud engineers with the purpose of providing the base infrastructure for migrating the company’s applications into the cloud.
- Worked with AWS professional services to design a strategy for the company’s cloud migration.
- I developed relationships with other teams within the company and supported them in their AWS migrations.
- Created a REST API based on AWS API Gateway, which acted as a data access layer for my team’s internal use, and to provide services for other teams.
- Developed a design and review process for my team to follow when introducing standards and major new features.
- Wrote tools to automate management of the company’s AWS accounts. This included building standardized network infrastructure, IAM roles, and infrastructure for compliance and security monitoring.
- Wrote tools to ensure that the company’s AWS accounts were CIS compliant, resources in the accounts were tagged correctly, and that EC2 instances were built from approved AMIs.
- Built fully automated Jenkins CI/CD pipelines which discovered jobs from Git repositories and ran all jobs in Docker containers on ECS.
- Wrote a secrets management tool in Go which used KMS to manage keys and S3 for storage of artifacts. I helped move a lot of developers away from storing secrets insecurely in git repositories by offering this tool as an alternative, because it was easy to use and required no installation step.
- Created an automated AMI build system with Jenkins and Ansible, which produced CIS compliant images for multiple versions of Linux and Windows.
- Gave a talk at the company’s developer conference on using Ansible to enable infrastructure as code, and how to use it as a general purpose automation front end for full stack deployments, in addition to its more well known purpose as a server provisioning tool.
- Wrote a Python tool which produced CloudFormation templates for standard stack designs based on input parameters, which provided a lot of consistency for deployments. It could build load balanced application stacks, VPC and network stacks, and could be extended with plugins.
- Worked on web services for use by the company to automate tasks within AWS, such as provisioning of accounts with required roles and users created by default, building of VPCs, or creating stacks from a service catalog. These services used Confidential to run many tasks.
- Wrote a web service in Scala to encrypt and store arbitrary data, which could later be decrypted and retrieved if a KMS managed private key was supplied. I also wrote a Python client for use from the command line or as a library. This was “v1” of my secrets management solution which was later rewritten as a simpler client side tool in Go.
- Worked on a team that managed services used by Disney and subsidiaries ESPN and ABC, that included an email and messaging API, a caching proxy, an authorization service, and numerous others.
- Moved our IP geolocation service into AWS to expand capacity, using CloudFormation combined with Chef to configure the entire stack in a fully automated fashion.
- Developed an orchestration and scheduling framework in Go to make it possible to manage jobs from a central location. Jobs were deployed after being checked into a git repository, which gave the ability for the whole team to see what jobs were running and where. Jobs could initiate orchestration tasks by sending messages to nodes in a cluster, and the nodes would respond by running event handlers and sending messages in return.
- Built clusters of Elasticsearch servers for development teams to see logs of their applications. Automation of cluster configuration was handled by Chef, and disk space was monitored and kept low using the orchestration tool I had written.
- As the owner of a cluster of high maintenance SiteScope monitoring servers used to poll Java applications for JMX attributes, I wrote a tool in Java that ran as a daemon on our application servers, collected JMX attributes, and sent the data to some existing Zabbix servers where thresholds and alerts were set. We were able to retire all of the SiteScope servers, which saved money in hardware and licensing, and reduced our maintenance costs.
- Created a monitoring solution that enabled any team to contribute their own monitoring scripts. Contributors would write monitoring scripts that gave specific outputs based on tools that I developed, and would check these scripts into a repository, where my team would send them to production after testing and review.
- Created a rolling deployment mechanism for the monitoring scripts which would stop on the first failure. Upon deployment, all scripts would determine where they should run and when, then automatically report metadata to a centralized location, from which we provided reports about the state of our environments. This was a highly successful project that dramatically increased the uptime for our applications.
- Worked on an agile development team, where I wrote key components of a cloud hosting management system in Java, which took as input a document description, and then built environments for hosting web and database services based upon that description. The system provided service discovery for all of the applications running in it, controls for starting and stopping services, deployment of web applications, and dynamic scaling in the cloud.