I got another map accepted by Valve for Team Fortress 2! I'd like to just showcase some of the work I did for the map here.

The map is called Sharkbay, and the layout was done by Billo. This map has been well liked in the community for a long time, so I was happy to be able to be a part of it!

The map is a basic lobby-style King of the Hill map that takes place in a New York harbor that has become infested with sharks.

Signs

I've recently become sort of obsessed with signs and their ability to be created fairly easily and you can hide jokes and Easter eggs in them while giving your map an extra little polish. All of these were done 100% using Substance Designer, which has quickly become my favorite program.

Another thing I have always tried to make a point to do is to add little calling cards for the mappers involved in the project. Here's the one's I made for myself and Billo:

Models

I was originally brought on to make a single model; the shark cage prop:

This was one of my first bespoke models that I made 100% from scratch, and I was quite happy with how it turned out. Here is a video timeplase of me creating it!

This is incredibly nerdy, but thanks to the help another artist named Tianes, I learned how to do diffuse baking in Blender and unleash the power of my procedural ropes I had designed for TF2. I can basically make any shape rope I want using a spline system in Blender incredibly quickly.

A rather large prop I did that has probably gone unnoticed was the garage door at mid. The brushwork version of this door like really bad so I just had to make it a prop. It's pretty simple, just 2 sliding doors, sort of rusty. The fact that it's gone unnoticed I think is a good thing because it means it fits the space well and reads as a door. 

Soundscape

I have never done a soundscape design before so this was a fun challenge. Basically none of the existing soundscapes matched the environment particularly well. I need a mix of watery, industrial, boat fog horns and seagull sounds. Thankfully all these sounds already existed in TF2's ambient sound files so it was just about mixing them correctly.

I also learned you can specify targets in the soundscape so the sounds come from a particular location in the map. This worked very well for setting direction to the fog horns and seagulls, as well as the faint car honking.

Logic

A lot of people have noticed the shark kill counter, this was one of those, "you know what would be cool?" things we were just spit balling. 

Basically any time someone falls in the water and is killed by a shark, this number increments. This is a bit complicated how it actually works using Source logic but I'll briefly try to explain.

The heavy lifting is done by a material proxy called "ToggleTexture" where you can fire an input to the info_overlay entity and cause it to change to the next frame in the VTF. Shout out to Adam for helping me figure that part out!

So I created 2 VTF's, one with digits 1-9, and a second with blank-9. Then all I need is a single math_counter to bump the second digit every 10 kills, and second math counter to track total kills so we can clamp it to 99.

Next there is an issue of z-fighting between info_overlays. Basically the sign on the brush was fighting with the 2 digits, so my solution was incredibly stupid; I created a 1hu thick func_detail that as a totally blank texture on it, as in 100% white with a 100% alpha channel and applied the 2 digit overlays to that.

Another thing I did was the flying seagulls.

The map has 3 different seagulls that fly. The one sitting on the derrick takes off when the point is capped, as if to imply he is annoyed by the ruckus caused by the point cap that he flies off. Don't worry though, he returns to his perch!

These are actually just retextured Half-Life 2 seagulls, so we get the full benefit of the seagull animations which there is a surprising amount of. Then all I needed to do was set up a tracktrain and a path of them to follow and periodically have them change animations when they pass certain path nodes. I'll say though, aligning the seagull to the one perched on the derrick was really hard because the tracktrain sometimes moves the model in a way that isn't very precise, but it works well enough I suppose.

Menu Photo Render

I've always loved the skeuomorphic aesthetic of a lot of things in TF2; so I wanted to do something spicy for the menu photos for the map. This was done in Blender using a simple orthographic camera and a point light. The newspaper itself was created by Blaholtzen. This turned out great and it's really unique among menu photos in game.

This is just a short rant I guess. Recently, I've wanted to work on some projects requiring me to write a web front-end and every time I revisit doing this, I'm left feeling very frustrated and defeated.

I actually like front-end

Not something you normally hear from people that spend their time writing deploy pipelines and VPC network infrastructure. I've always liked the creative aspect of web design and working on visual mediums. At one point I thought I'd be doing front end dev, but quickly decided to not pursue it after working on front-end stuff at Carfax for about a year. 

I left front-end dev because of just how painful it was to do, and it has always felt so needlessly difficult. Like how many of us have had the experience of "how do I center this fuckin div!". Why is that so hard? Back then we only aligned stuff using floats or tables. I guess in short, the web was just too immature and clunky to my liking. Even today, I feel like you have to be something of a masochist to actually want to do front-end.

As a programmer, I thoroughly enjoy the fact that other people write high quality libraries so that I can just reuse their code and not have to think about those problems and just get going with my project. This is true for most things I do with DevOps and backend libraries in Python. Those libraries / tools work great most of the time. I really wish I could say the same for the frontend...

How it feels to do front-end in 2022

You have a new project and your colleague told you that React is good and an industry standard. So you spend the next hour just trying to choose one out of literally thousands of component libraries to use. 

After you spend an hour waiting for NPM to finish installing the 200mb of needed NodeJS packages, and you finished reading for another hour on Webpack docs so that you can have a constantly running build pipeline locally because this framework uses MochaScript; a way "better" version of JavaScript apparently that has to be compiled. Now you can finally start a local webserver with the convenient and never broken npm start dev command!

If by some miracle you get that working in less than a half day, now you get finally use these components! Wait, the docs don't have copy and passable code? It's no problem; you just have to dig through lots of stack overflow posts to find an example that you can tailor to your needs.

I for one am glad that the frontend community adheres to common conventions. After all; what really is the semantic difference between a Modal, Dialog, Popup and Popover.

So after a while of thoroughly reading the docs and StackOverflow, you finally figured out what to do. You start writing your app, and get pretty far. Ah but your compontent library is missing a component that you need.

Do you...

• Switch to a new component library and rewrite everything from scratch?
• Try to import just the components you need and make a frakenframework of sorts?
• Write your own component?

Writing your own component is sooooo easy. Just use this code generator that will make 30 files for you, and be sure to be up to date on the latest JavaScript, React, JSX and CSS features otherwise you'll be completely lost.

Wait none of that matters now; a new thing came out this week!

The library you using has decided to completely switch underlying tools and dependencies, now you have to decide if you will update to this standard; and they gave you 1 week notice of this change. Most of the code you wrote will have to be significantly changed and the old version of this framework becomes effectively abandoned so you never get updates or security patches. Foolish mortal; you thought MochaScript was the perfect JavaScript? Behold CoffeeScript! Its like MochaScript, but slower and with questionable syntactic sugar.

exampleCode().
  showing(aThing(), thisLibrary.does())
  .justIgnore(
    () -> theSyntax()
);

So readable.

Thank goodness they added these chained call backs, I don't know how people coded before this! I hope I don't have issues compiling back to ES6. Don't sweat it too much though, wait another few months and they'll be using MelonScript or SuperScript, or some new flavor that will solve all it's problems. 

Now your app has been in production for a few months and you need to fix some bugs; requiring you to update your library version. Well now you can't even do that because the package maintainer took the library off the internet and replaced it with bitcoin mining software to protest Climate Change.

Roasting Javascript Aside...

I admit that these issues are not exclusive to JavaScript / Frontend dev. Though it feels especially bad in frontend because of how quickly things are changing. And to what end? Sometimes I really wonder; have we not found the right abstractions to write engaging and pretty user interfaces? Why is it this difficult just to get a basic dynamic page working?

And for fuck sake, can we get a language other than JavaScript to use on the frontend? A lot of these "JavaScript but better" languages that require transpilers have sprung up to attempt to fix the issues with JavaScript itself, which is sort of an interesting way to test new language features and ideas; but good lord JavaScript has the highest density of footgun's of any language I've ever used. I'm sure the reason they don't just make JavaScript not full of foot guns is that they are trying to maintain reverse compatibility.

I know that WebASM is supposed to be at some point allow us to solve this issue of a single language dominating the front-end, but we really haven't seen much practical things from that effort yet; outside of cool OpenGL ports.

I loved the simplicity of the old web. You can super quickly prototype something on a local web page. You didn't need to worry about package managers, CDN's, or running a web server. 

From a DevOps perspective; I will go very very very far out of my way to not have to run a production NodeJs server. I've had numerous horrendous experiences in managing and troubleshooting production NodeJs servers. I'd much rather stick with the devil I know, like NginX. Again I yearn for the simpler times when you just served an HTML page and didn't have so much complexity to deal with. Browser default components don't have any of these issues, why can't we have more of those?

I've been following the evolution of front-end for a long time. I don't think front-end developers realize just how complex they have made their ecosystem. And I'd say, quite needlessly so. For something as huge as Facebook, that level of complexity is probably warranted; but the level of complexity that Facebook uses is the only level. React is React. There is not really a lower level of complexity for smaller teams or hobby projects. One that doesn't require WebPack, NodeJS server, JestJs tests, code generation, tree shaking, CSS precompliation, etc etc. One where you can just get to the heart of what you're actually trying to do, which is make a fucking usable component.

Web Components?

The thing that I think frameworks like React get right is the component encapsulation model and in principal I quite like the React way of doing this, it's just all the ecosystem surrounding it that is really painful and a huge turn off to me and the tools are individually so complex and never work out of the box for me.

When React rose to popularity in ~2015 (which is right around the time when I left Carfax and moved on to DevOps) the W3C had been working to develop and implement a way to do what React does, natively in the browser. 

This is usually referred to as Web Components; but is really the combination of 3 features to the web standard:

• Custom Elements
• Shadow DOM
• HTML Templates and Slots

I quite like the idea of having these things natively. It gives me the thing that I want which is ease of development. I don't need build frameworks, or Webpack or tree shakers or anything like that. All you need to do is include a JavaScript file on your page and you're done. This seems like the level of complexity I'm looking for.

As of 2018, all major browsers support all three of these standards natively which means that native custom components have arrived.

Authoring Web Components

These native API's are a bit clunky to use, or should I say are low level. It makes sense, to a degree, that they would do it that way because it allows for libraries to sit on top that expose different abstractions tailored to certain needs. But I fear this gets us back to the issues from before where these libraries will bring in elaborate build systems and complex bundlers and such.

With that said, I've experimented with a few libraries, as well as 

• StencilJS
• Fast
• Lit (Formerly Polymer)

I quite like Stencil, but am a bit annoyed that it forces you to exclusively use Typescript. It is essentially a toolchain for "compiling" Typescript as well as the HTML and CSS into a native web component. The thing I like the most about it is that it includes the primitives for hydration and minifying bundles. I could use this tool right out of the box. Or put another way; you can experience success in 5 minutes with this tool; which is more than can be said about React. However, in general, I'm quite wary of complex build chains, and indeed if there was some kind of underlying problem with Stencil's use of WebPack, I'd be at a complete loss at how to fix it.

I also quite like Lit, though it doesn't come with all the same nice batteries that Stencil does. Normally this would be a trade off of bundle size but infact Stencil and Lit will produce bundles of basically the same size since they will both produce an artifact that is just the native component with a thin abstraction layer over it. Though all I really care about for small projects is just being able to author a component quickly. Lit is basically just a thin abstraction on top of the native API's. I won't really know which of these I prefer until I really dig into a project using them.

Native Component Libraries

I've searched pretty thoroughly across the web for anyone authoring native web components and here's all the ones I've found:

• Shoelace
• Nord
• Crayons by Freshworks
• Kickstand
• Liquid Oxygen
• Elix
• GoatUI
• Vaadin

I've dabbled with all of these, and the one I like the most is Shoelace. It seems to be the most batteries included, but also customizable (via "design tokens") of the lot.

To me these are basically the next evolution of Bootstrap. Gives you a lot of stuff out of the box, but this much less boilerplate-y to use, and has the styles encapsulated. 

For some upcoming projects; I think I will likely make use of Shoelace and then use Lit or Stencil to author custom components that suit my specific needs, and then glue component functionality together with plain old fashioned no frills JavaScript.

Things do seem to be getting better

With the advent of native web components alongside tons of fantastic new CSS features such as Grid, Flexbox, and variables, it seems like things are heading in a good direction for the web. 

Right now I'm just stuck in a bit of a limbo waiting for Web Components and WebASM to really get mainstream adoption and start bearing fruit. React just has so much momentum and adoption that these new standards will take a while to get large scale adoption.

Though one very nice feature of native components is that you can just use them in React, Vue, Angular, Svelte and basically every React-like framework; so there is a non destruction path to adoption at least.

At least with native web components, we can rely on them to work basically forever. One of my biggest gripes is that with all the frameworks and libraries out there; how many times has someone reinvented the "button" component? There may be times that this needs some specific customized button; but generally its just a box with text. My hope is that native web components will see a lot of reduction of this reinventing the wheel. Just a handful of libraries that give us the tools to customize parts of the element as we need and we're off to the races; or hell, give me high quality, accessible but otherwise unstyled components that I can customize to my brand / needs.

My hope is that one day, the web wont be so painful that I might be able to re-enter the world of the front-end as a professional. For now, I'm fine to watch from the sidelines.

AWS IAM is the global permissions system that AWS uses. People are often mystified by how it works and often end up just giving everything *.* permissions. 

In this post I'll discuss things I learned while managing the IAM configuration at Narrative Science and hopefully allow you can learn from our failures so you can manage your IAM account in a much for sane and scalable way.

IAM Primer

• Roles
• Policies
• Groups
• Users

Roles, Groups and Users can all have policies attached to them. Groups are composed of multiple Users. For example, you could have a db-admin group that has the common permissions needed for that task, and users can be added or removed easily from it.

Users and Roles are the entities that authenticate with AWS. There's a lot of ways this can be done but most commonly it's either an EC2 instance with a Role applied, or an Employee with an access key.

Policies are where the rubber hits the road. When you author a policy, you can say to allow or deny access to a resource (though not allowing something is an implicit deny).

Here is AWS's official documentation on the subject:
https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction_access-management.html

It will also be important to understand ARN's (Amazon Resource Identifiers) as they are used extensively in IAM policy documents:
https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html

Security Objectives

• Least privilege access:
  In short, only granting permissions to the resources and actions an entity will need to perform. The idea here is that having credentials compromised doesn't allow an attacker to do anything they want.
• Easy to administer in a fast changing environment.
  Employee turn over, new projects coming online, etc
• In a multiple account setup, have a single user account for each employee

Naïve Solution #1

One of the first solutions we used at Narrative Science, you just go through all the resources an entity needs access to and which actions they need and grant them. This is a "grant-only" solution, so never use a deny in these policies.

Seems pretty straight forward, why is this naïve?

Well AWS has some annoying limitations:

• Users cannot have more than 10 policies attached.
• Users cannot be a member of more than 10 groups.
• User policy size cannot exceed 2,048 characters.
• Role policy size cannot exceed 10,240 characters.
• Group policy size cannot exceed 5,120 characters.

These might seem like a lot of wiggle room at first but you'll find very quickly that you start running out and have to be overly permissive to fit within these bounds, especially if your team has generalists that perform multiple job functions.

If a user is already maxed out on groups and policies, but needs permission to perform a certain function for a small period of time (privilege escalation), what do you do? Well I guess we just drop one they don't need at that time and revert it later. You'll need some way of tracking that escalation and reminder to deescalate when they're done, which will almost certainly be required for compliance anyway.

At best this is an annoying burden on the admin team managing the user permission granting. At worst this will just lead to that team getting fed up and granting way overly permissive permissions.

Naïve Solution #2

Lets try the opposite; being overly permissive and having explicit deny's on certain resources. On the surface this seems reasonable since its probably easier to state what you don't want to grant access to vs what you do.

Say for example you want engineers to have free access of a lower environment, but want to deny all of production. Well if you have good resource tagging, you can mostly deny by anything with a production tag or having prod somewhere in the ARN.

This also very quickly becomes a mess and doesn't save that many characters in your policy documents after its all said and done. Some resources require you to be able to have read / list access, otherwise the UI is unusable. One example is CloudFormation having to have read / list permissions on all resources but also you have to deny any update access to production stacks specifically. But do you then deny CreateStack access in CloudFormation too? 

There's also a huge wrinkle here; you cannot allow users to create IAM policies because they can just create a role that grants higher access and allow themselves to be able to assume that role. But then how do you allow one to make IAM policies that are deployed with a CloudFormation stack?

A Better Solution

First of all, one of the biggest issues we had was that we were running both our lower environment and production environments in the same AWS account. We thought that it was good to be diligent with tagging and resource naming but that didn't really help us here for the reasons listed in the last section; even though its still really great to be diligent with resource naming and tagging for billing and clarity of communication.

AWS allows an Organization to have multiple accounts associated with it. IAM doesn't do well trying to isolate resources in the same AWS account, so first order of business; isolate your production resources in it's own AWS account, then you can focus on access control to that account specifically.

In our Organization, we setup a security account where all the IAM user accounts are, a non-production account where users were granted almost complete admin access, and a production account where access was extremely locked down.

Before we go further, we need to understand a 2 advanced AWS IAM features:

• Permission Boundary's
  You can basically grant a wider array of access, then with a boundary policy, restrict the access to only the resources in that boundary. Think of it like a Venn diagram of permissions:
  
  
  
  
• Assume Role
  An IAM user can inhabit the permissions of a Role, but only the permissions applied to that role itself.
  Both the UI and CLI are able to do these role assumptions, and can be done in a way that requires MFA (big plus for compliance). This feature is what will enable us to have single IAM accounts per employee but allow them to have permissions in multiple AWS accounts. 

So the idea here is; you create roles with open permissions that you then restrict down with a permission boundary, creating much more strict effective permissions; and then users can inhabit those roles.

You can define these roles any way you want, it will vary greatly depending on your organization and this can obviously be a bit subjective. We decided to define our roles roughly along discrete job functions which allows for an employee to inhabit the role needed to complete a certain task without having to grant full admin (unless that is what is needed).

For our production account, we created the following roles:

• Admin - Complete god permissions only granted to a few, mostly just myself.
• ProdAdmin - Almost the same as god permissions except for it cant change IAM roles, view billing or make account wide changes. Used for team leads / architects
  OnCall - Basically a copy of ProdAdmin except that we wanted to keep this distinct incase these 2 roles diverged. Also its more obvious what it is.
• ProdEngineer - Allows engineers to view the status of deployments, read log files, view alarm states. 
• CustomerSuccess - Allow our customer success team to manage our Serverless SFTP and handle issues relating to customer data delivery. They basically only had access to a few S3 buckets.

If a new job function should arise, then you can of course make new roles to suit those needs. We were working on adding a new one right before we were acquired by Salesforce that would allow some of our engineers to train and maintain production ML models; just as an example.

The advantage of this setup is that not only do you not run into annoying AWS limitations, but also you can grant or remove access to individual users as needed. This can get out of hand to manage manually, so we codified all of our IAM using Git and Terraform. This had the side effect of also allowing engineers to request a role escalation via Git that we could gate with CodeOwners, and have an audit trail for compliance.

Side note: CloudTrail does log when a user assumes a role, and will show that role's session id in any logs done while that user is in that role; so you can audit who did what and when. 

Example Code

Enough of me describing the theory of this; here's some actual code. I'm not allowed to share the exact code (for obvious reasons), however here is an example to demonstrate how we used permission boundaries. Basically think of these 2 policy documents in a Venn diagram. 

Role Permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:Describe*",
                "cloudformation:EstimateTemplateCost",
                "cloudformation:Get*",
                "cloudformation:List*",
                "cloudformation:ValidateTemplate",
                "cloudformation:Detect*"
            ],
            "Resource": "*"
        }
    ]
}    

Role Boundary:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:*"
            ],
            "Resource": [
                "arn:aws:cloudformation:*:ACCOUNT_ID_REDACTED:stack/prod-app-*/*",
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:Describe*",
                "cloudformation:List*",
                "cloudformation:Get*",
                "cloudformation:GetTemplateSummary",
                "cloudformation:CreateChangeSet",
                "cloudformation:ExecuteChangeSet"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}  

Terraform Role Creation Module:

variable "app_admins" {
    type = list(string)
    default = [
        "user1",
        "user2",
        "user3",
        "user4",
        "user5"
    ]
}

resource "aws_iam_role" "app_admin" {
    name = "app_admin"
    path = "/humans/"
    max_session_duration = 3600

    assume_role_policy = data.aws_iam_policy_document.app_admin_trust_policy.json
    permissions_boundary  = aws_iam_policy.app_admin_boundry.arn
}

data "aws_iam_policy_document" "app_admin_trust_policy" {
    statement {
        principals {
            type = "AWS"
            identifiers = [
                for user in var.app_admins :
                "arn:aws:iam::ACCOUNT_ID_REDACTED:user/${user}"
            ]
        }

        actions = [
            "sts:AssumeRole",
            "sts:AssumeRoleWithWebIdentity"
        ]

        condition {
            test = "Bool"
            variable = "aws:MultiFactorAuthPresent"
            values = ["true"]
        }
    }
}

resource "aws_iam_policy" "app_admin" {
    name = "app_admin"
    policy = file("${path.module}/../policies/app_admin.json")
}

resource "aws_iam_policy" "app_admin_boundry" {
    name = "app_admin_boundry"
    policy = file("${path.module}/../permission_boundries/app_boundry.json")
}

resource "aws_iam_role_policy_attachment" "app_admin_attach" {
    role = aws_iam_role.app_admin.name
    policy_arn = aws_iam_policy.app_admin.arn
}        

IAM Scopedown for Service Roles

We never ended up putting this into action at Narrative Science but I stand by the design and idea. 

With services, it can be quite tedious to know all the permissions exhaustively that a service will need ahead of time. Even in earnest, most humans will miss a lot because AWS has just so many services and caveats. But we also need to reach least privilege access for services, is there a way to do this so it takes the burden off the developer and can allow you to achieve least privilege? 

The solution here I think is quite elegant. I call it IAM Scopedown.

Basically, a new project that is not yet in production is allowed to run for a time, say 1 month (but the duration is somewhat arbitrary). Initially you grant a sane but knowingly over permissive set of permissions to the app. CloudTrail will log the actions that project did during that time. Then you can view those logs and see which permissions the app actually needed and edit your IAM policies based on that.

This is ripe for a script isnt it? Well good news, one exists for this very thing! Enter Cloudtracker.

To further on this idea; I would have automation run this script and send IAM scopedown suggestions to the team managing the project; ideally with the exact output needed for an IAM policy document. This is more of a "eventually least privilege" solution; which is far better than what usually happens where people give up and grant it way too much.

It is possible to have this tool automatically scope down service policies live in production, but I'm wary of not having a human involved in something that is likely to brick production in unexpected ways; though with time and developing on this idea further, I think that is feasible.

This scopedown procedure can probably also be applied to IAM for Humans but just because something goes unused doesn't necessarily mean it wont have need in the future. This is why I make the distinction between Services and Humans, since they tend to have big differences in their security needs.

Conclusion

Using a few advanced IAM features and a little bit of clever design, you can make your IAM setup a LOT less tedious to deal with, and reasonably achieve least privilege. 

I have to give some credit to Netflix engineers on their "security pizza" talk. I built on a lot of ideas from this talk:

I'm an engineer, which means I love my tools and there's some tools I hate. I guess that comes with the territory. This is the collection of tools I try to use anytime I can for Cloud Engineering tasks.

Ill format this by category, and list out my likes and dislikes about the tool. Realize that each one of these is just my opinion and I'm not saying they're the only tool anyone should ever use, people have different preferences and needs afterall. Enjoy!

Infrastructure as Code: Terraform

Terraform is a tool that allows you to define your cloud infrastructure as code. It is intended to be declarative rather than programmatic, but it does a good job of balancing between the two. 

Admittedly I am a bit biased because the only other tool in this space I've used is CloudFormation, which works well for small use cases but quickly becomes unmanageable when you have a fast changing dynamic environment like the cloud.

In my mind, the 3 big players for IaC are: CloudFormation, Terraform and Pulumi. The way I like to think about it is a spectrum where CloudFormation being the most "declarative" while Pulumi is almost literally just a library in a programming language. Terraform leans towards being declarative but has much better support for things like conditionals, loops, variables, passing data between modules and other programming language paradigms.

Configuration Management: Ansible

Ansible allows one to codify the steps to provisioning a virtual machine or a Docker image, allowing one to reproduce their configuration whenever needed.

Honestly I feel like I could keep going on and on with this list of pro's.

Personally I've used the task profiling to setup a playbook with parallelization which gave us a dramatic speed up in execution time. Using those 2 features we went from 3 hour runs down to ~25 minutes.

Competitors I've used are Fabric (Python lib) and SaltStack. I've dabbled a bit with Chef but am very unfamiliar with it. Fabric is no longer maintained because of the Python 3 switch, but I do like the concept of managing state with the power of a programming language, but also fear that it can become very unruly very quick if you maintain poor abstractions. I like Ansible because it puts guard rails on those abstractions so things really can't get too complex or hard to understand.

For what Ansible is, it does a great job and is one of the more reliable tools I regularly use at work.

Version Control: Github / Git

Github is a remote code repository and developer platform. It is obviously backed by the company Github itself. It is by far the most popular code repository of it's kind, built by coder for coders.

I have only really used TortiseSVN as an alternative, and it's workflow always felt clunky to me. I was using it right when I started learning programming so perhaps it's not as bad as I remember.

Git is so popular and ingrained in my head that I honestly don't know how I would switch away unless I joined a company that ran another one.

I should also mention that Gitlab is also pretty nice in comparison to Github, though I still prefer Github probably just out of familiarity alone. Both do seem to offer the same things.

Programming Language: Python

Okay before you bite my head off saying its "too slow", remember this post is about the role of a Cloud Engineer, not a HPC or ML engineer. Python has dozens of useful libraries and has a long history of being used as a systems language.

The obvious elephant in the room is Bash. I have a love - hate relationship with Bash, but also feel very inadequate because I've been using daily it for over 10 years and still have to Google basic syntax every time I use it.

The way I think about these 2 "languages" (if you can even call bash a language) is that Bash is great for simple things that you run on your command line, but the second you need more than 1 if statement, you should seriously consider using Python instead. Also bash sucks at handling JSON output and Python defiantly shines there.

I know that Hashicorp was trying to make Ruby the Cloud Engineer language of choice but just doesn't seem to have taken off the same way as Python has.

Just for fun, here's some of my favorite Python libraries:

• boto3
• FastAPI
• Pydantic
• Jinja2
• ormar or beanie
• paramiko
• requests or httpx (async)
• rich
• PromptToolkit
• loguru

Monitoring: Prometheus + Grafana

I've already written about this subject when discussing monitoring for my Team Fortress servers but ill summarize here.

Code Editor: VS Code

VS Code is a code editor created and maintained by Microsoft. It's relatively lightweight and can become very fully featured via their extensions.

Normally I don't like throwing my hat in the ring for things as subjective as a code editor. Everyone has strong opinions about this, however VS Code has a few things I believe seperate it from other editors in it's class.

I've used so many text editors over the years: Brackets, Atom, Notepad++, Sublime, Bluefish, GGTS, Eclipse, Visual Studio, and even just VIM. To be honest, until I started using VS Code, mostly I chose an editor based on if I could find a syntax color theme I liked. 

I will say the #1 thing I love about VS Code is the Remote Developer Pack. Instead of running an SFTP plugin or having just a terminal in another screen, this is a truely seamless experience. You get all the same code searching and features normally only available locally with it. I've been using it to connect to remote servers as well as with Windows WSL. 

VS Code is not as oriented to a specific task as an IDE like Eclipse or Visual Studio. This is actually nice for me since I work on a lot of different tasks and like that I can tailor VS code to my needs, instead of trying to shoe horn stuff into other IDE's.

Cloud Provider: Too difficult to say

Haha didn't expect that one did you?

I feel like this is very dependent on the task. Obviously I like any provider that has integration with other tools I mentioned. So ill just talk briefly about the one's I've used.

AWS

AWS is the only one I've used that has a robust virtual networking offering which is huge for any company trying to do basic network security. And on the subject of security; IAM permissions are a horrendous pain to use, especially if you're not familiar with all the in's and outs of how it works; though no other provider does it any better.

AWS is also by far the most expensive one here. For hobby projects, it's a bit costly unless you really do a good job with cost management. AWS also makes it easy to spend way more than you expect and you need dedicated engineers to pay attention to this kind of thing.

For business uses, I think AWS is the clear choice.

Google Cloud

I had used GCP for a while with my Team Fortress 2 servers, but ultimately ended up moving to AWS because of the lack of features for VM management and their API wasn't quite as simple to integrate with as AWS (GCP has no equivalent to boto3)

GCP's user permissions are also way more byzantine than I expected, and is somehow more confusing to me than AWS IAM. I can't believe I'm going to say this sentence: I'd rather use AWS IAM over GCP's permissions.

They offer a similar suite of features to that of AWS, and is a serious contender for business use cases.

On the subject of cost, GCP has way better cost estimation tools and they tell you the price before you stand something up in their UI; something AWS just simply does not do. Though that said, most of the time I'm standing up infrastructure programmatically so I don't really see this kind of thing often.

Vultr

They obviously don't have nearly as many features as AWS or GCP, but they knock their VM stuff out of the park. Infact you can literally upload your own .iso to it and just run that; opening up the possibility of doing local builds and just sharing an .iso. They also have Windows VM's if you need that kind of thing.

They have recently added an S3-compatible "spaces" offering which I'm using to good effect for TF2maps static sites. It's not fully flushed out though, so I'll hold my judgement on it for now.

It has everything you need for small projects. Built-in DNS, VM's, Block Storage, Firewalls, Load Balancers. It's networking is lacking a bit in my opinion. 

3 features I want from Vultr:

• An AWS Fargate type docker runtime. Basically I just give you a container and you run it
• Lambda's
• Managed databases

A few years ago I had a lot of networking issues hosting game servers, which has turned me off from using them. It appears that this is no longer a problem given we're running almost entirely on Vultr at TF2Maps.

DigitalOcean

If DigitalOcean added the features i mentioned in the Vultr section, I'd happily switch to them. They have the managed DB part down.

Honorable Mentions

• Packer - A simple program for doing Golden image builds. Very modular and simple
• RKT - A Docker-compatible runtime that has a lot more security features vs Docker itself. 
• pgcli, mycli, litecli - Database CLI's that use Prompt-Toolkit. Nice features like syntax highlighting, and context aware tab completion
• tmux - Great for running some services in the background without losing access to the command line. Also great for doing multiple tasks on a VM at the same time
• jq - An attempt to make JSON parsing work in bash without crazy awk, sed and grep pipelines.
• Figma - A free UI prototyping tool. Mostly I've used this as a way to show a more tangible face to some of the backend things I want to do at TF2Maps.
• LucidCharts - Great for making infrastructure diagrams
• CloudTracker - A great tool for creating least priveledge IAM permissions

These are just tools

It's important to remember that just like a hammer is a tool, all of these are just tools for specific tasks. I'm not married to any one of these tools and will switch if another tool comes along that solves my problem better. 

To me it's more important to follow principals of staying organized than to become obsessed with how a certain tool solves a problem. Almost all of these tools I like because they allow me to maintain that kind of simplicity and organization that I want.

One final thought; remember that everyone's use case is different and sometimes a tool is still a better choice even when it has less exciting features because of things like cost and whether engineers at your organization are already familiar with it. There is a cost to onboarding new tools and learning their paradigms.

The most common use of a class in Python is acting as a data container. Throughout the development of our backend libraries at Narrative Science, we’ve seen several generations of code that attempts to make modeling this data easy.

Finding a simple to use abstraction for these type of classes allows us to focus less on writing mundane classes and more on our business logic.

In this post we will outline the evolution of this code and the pros and drawbacks of each approach.

V1: Basic classes

We have all been here; you write a class, and implement all the dunder methods needed by that class. Simple and straight forward.

Example:

class User():
    def __init__(self, fname, lname, groups):
        if not isinstance(groups, list):
            raise ValueError("groups is not a list!")
        if not isinstance(fname, str):
            raise ValueError("fname is not a str!")
        if not isinstance(lname, str):
            raise ValueError("lname is not a str!")

        self.fname = fname
        self.lname = lname
        self.groups = groups
    
    def __repr__(self):
        return f"User({self.fname},{self.lname})"    

Pros:

• This is very explicit, you can see everything this class does
• Basic, most likely taught in beginner Python courses

Cons:

• Type Checking
  Notice all the defensive code needed to handle the cases when we receive arguments with incorrect types. Now imagine this example accepted 20 properties. More than likely the author of this code just wouldn’t even bother to write this kind of type checking since it’s tedious and time consuming. However not doing so can create complex error conditions in downstream code.
  
  
• Database
  Very commonly, these type of objects directly represent data from the database.
  As it is currently written, you would need another translation layer outside the object to do your queries and create the objects as needed.
  
  
• Testing
  You will need to write tests to make sure this code correctly rejects invalid inputs.
  
  
• Instantiating
  In order to create an instance of this object, you need to pass every value into the constructor:
  

  new_user = User("frank", "smith", ["group1", "group2"])
  

  
  It’s worth nothing that it is possible to do some fancy Python magic to create this object out of a dictionary:

  user_data = get_user_data()
  new_user = User(**user_data)

But what if I want to dump the properties of this object back to JSON so we could send it over HTTP? You would have to write a method that takes every property and populates a dictionary.

V2: Database linked classes

The idea here is: what if we created objects that were aware of the database?

Example:

from sqlalchemy import Table, Column, VARCHAR, UUID, JSONB
from ns_database import DatabaseObject

article_table = Table(
    "article",
    Column("id", UUID, primary_key=True),
    Column("name", VARCHAR(45), nullable=False),
    Column("slug", VARCHAR(45), nullable=False),
)

class Article(DatabaseObject):
    table = article_table
    primary_key = app_table.id

new_article = Article.create(id=UUID4(), name="Foo Bar", slug="foo-bar")
new_article.save()

This essentially creates a class that is now able to push and pull its data from a database and functions like a normal object inside of Python.

The way this DatabaseObject class works is that the properties for the class are determined by the SQL Alchemy table.

DatabaseObject also implements a handful of convenient methods to help getting data in and out of the database such as: create, get, save, update, and delete. Each will perform the appropriate SQL query.

Pros:

• We have reduced the amount of tedium from boilerplate code.
• Users have a database aware object without needing to write any SQL of their own.

Cons:

• We still have the issue of having to make sure our types are correct when accepting them as parameters.
• We are unable to set default values for properties, so we must always provide all properties, which is tedious. This is because we’re using SQL alchemy as the source of the property names and don’t have a mechanism for passing other values in.
• What if someone inputs a string longer than 45 characters? We need to write code to check for that.
• There is an assumption that we must know the type in Python that corresponds to the SQLAlchemy type.
• Alongside the tests needed for the DatabaseObject class to ensure the database accessor methods work correctly, we will also need to still have tests that ensure that these classes reject invalid inputs and don’t just throw an exception from SQL alchemy.

V3: Along came dataclasses

Basically, a dataclass is a simple abstraction for classes that store data. It performs automatic type checking, serialization and deserialization, and implements things like __eq__, and gives a standard __repr__. This is basically what we wanted from Generation 1.

There have been many libraries before this to implement the data-class such as attrs, marshmallow, and schematics. It is worth noting that Python 3.7 introduced an official the Dataclass to the core library as well.

The library we chose to use was Pydantic because it worked best with our implementation of FastAPI.

from enum import Enum
from typing import Optional
from pydantic import BaseModel, Field

class User(BaseModel):
    name: str = Field(max_length=45)
    department: Department = Field(default=Department.engineering)
    nickname: Optional[str]

class Department(str, Enum):
    engineering = "engineering"
    finance = "finanace"

class User(BaseModel):
    first_name: str
    last_name: str

    @property
    def email(self):
        return f"{self.firstname[:1]}{self.last_name}@example.com"

    def send_message(self, msg):
        some_email_function(self.email, msg)

...

user = User("John", "Smith")
user.send_message("hey dude!")

• These models are not database aware like Generation 2.

V4: Adding a database to Pydantic

Building on our idea from before about turning an ordinary class into a database aware object; we combined the benefits of Pydantic's input validation and serialization with the SQL Alchemy based database class from Generation 2.

The result is that now we have functional ORM that works with FastAPI!

Example: 

from enum import Enum
from typing import Optional
from pydantic import BaseModel, Field

from ns_database import database_model, Databases

class UserIn(BaseModel):
    name: str = Field(max_length=45)
    department: Department = Field(default=Department.engineering)
    nickname: Optional[str]

class Department(str, Enum):
    engineering = "engineering"
    finance = "finanace"

@database_model(table_name="users", database_name=Databases.main)
class User(UserIn):
    id: Optional[UUID4] = Field(primary_key=True, fetch_on_create=True)
    created_at: Optional[datetime] = Field(fetch_on_create=True)
    updated_at: Optional[datetime] = Field(fetch_on_create=True

We add a class decorator that will use the properties of the model as fields in the database. This means that database models will be 1:1 representations of the database itself.

Using Pydantic, we are able to add additional keyword arguments to the Field function.

Fetch on create / update

We want our database to be the ultimate source of authority, so we rely on it to do the computation for certain fields.

This is done through setting fetch_on_create and fetch_on_update which will defer to the database’s generated value when a record is created or updated respectively.

All that needs to happen is your database is setup to have column defaults such as static values, subroutines, or sequences.

Primary key fields

Since all tables will have a primary key, each database_model object must have a primary key.

Currently we are not able to have compound keys meaning your key must be a single type. This may be a situation where we can create a special type for it to allow it to be more semantic.

This primary key field is also used by our data accessor method to know how to lookup a record.

Pros:

• These database models get all the benefits from Generation 2 and 3!

Cons:

• Lack of foreign key relationships may be misleading to developers. To many developers, this code is their interface to the table schema. Since we use Alembic to manage our database schema and updates; there is actually no need to declare fields inside our database_model to be foreign keys. These are only used by SQL Alchemy in order to create your table schema. This does create a situation where our code and schema may not be in-sync. It is also worth noting that those foreign key constraints are still enforced by the database.
• Database managed fields have to be “optional”. One limitation of Pydantic is that we have to make any field that the database will manage (such as the primary_key) as Optional type. The Pydantic validation code runs before we ever make our fetch from the database. This is syntactically a bit confusing.

Closing thoughts

These new database model classes have greatly improved the readability of our code and reduced the amount of boilerplate we need to write for new objects which ultimately means that we spend more time worrying about our own business logic vs writing all the low level stuff classes usually need.

As with most projects in software engineering, we kept making marginal improvements to the ideas until we got something awesome! I’m excited to see how this idea continues to grow to further help us achieve these same goals.

You can find the full code for this here: https://github.com/NarrativeScience/pynocular