Slices are value types

In practice, while it may appear that slices are pointer types, in actuality they are value types.

package main

import (
	"fmt"
	"unsafe"
)

func main() {
	var s []int
	var p uintptr

	fmt.Printf("slice: %v | pointer: %v", unsafe.Sizeof(s), unsafe.Sizeof(p))
}

$ go run main.go
slice: 24 | pointer: 8

A uintptr, at least on the host machine, is 8 bytes. The slice, []int is three times the size, 24 bytes.

This is due to the fact that a slice is a value type, specifically a struct, that contains three fields: Data, Len, and Cap.

See the SliceHeader documentation in the reflect package for more information. Alternatively, you can look at slice.go from the Go runtime.

This means that when you pass a slice from one function to another, you are copying an entire struct, not just a pointer.

As a bonus, since the map type often gets lumped into the same category as slices when talking about pointer types, a map in Go actually is a pointer type.

package main

import (
	"fmt"
	"unsafe"
)

func main() {
  var m map[int]int
  var p uintptr
  
  fmt.Printf("map: %v | pointer: %v", unsafe.Sizeof(m), unsafe.Sizeof(p))
}

$ go run main.go
map: 8 | pointer: 8

Slices point to arrays

You'll notice that when it comes to growing in size, the behavior of a slice is what you would expect from an array. Hopefully you remember your lectures from data structures!

package main

import (
	"fmt"
)

func main() {
	nums := []int{1}
    
	// NOTE: &nums[0] is used here to show the address of 
    // the first element in the array.
    //
    // This is because the variable `nums` (no indexer) references 
    // the struct of the slice.
    //
	// If the address of nums[0] changes, we know that the array that the 
	// slice points to has changed.
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	nums = append(nums, 2)
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	nums = append(nums, 3)
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	nums = append(nums, 4)
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	nums = append(nums, 5)
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])
}

$ go run main.go
len 1 | cap 1 | addr 0xc00004a1f0
len 2 | cap 2 | addr 0xc0000044a0
len 3 | cap 4 | addr 0xc0000200c0
len 4 | cap 4 | addr 0xc0000200c0
len 5 | cap 8 | addr 0xc00009a000

From the execution above, we can see that the initial creation of the slice sets both the capacity and length to 1.

When we perform an append() operation to add another value to the slice, we see that the capacity and length are incremented to 2.

You'll notice, however, that the address changed after the append. This is because the underlying array reached capacity. In order to be able to store the newly appended value, the slice needs to reference a new, larger array.

To accomplish this, Go will:

• Create a new array that is double the capacity of the original array
• Populate the newly created array with the original values
• Update the pointer on the slice to point to the newly created array

As shown above, we can see all of these changes because append returns the slice with the new capacity and length values (and potentially a new backing array if the original was at capacity).

In the scenario where the slice needs a new backing array, any references to the old array are still valid.

package main

import (
	"fmt"
)

func main() {
	nums := []int{1}

	// Create a copy of the slice that was declared above
	c := nums

	// Sanity check that all of the properties for both slices are identical
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])
	fmt.Printf("len %v | cap %v | addr %p\n", len(c), cap(c), &c[0])

	// Since nums has a capacity of 1 and we are performing an append 
	// operation, the backing array will need to be replaced with 
	// a larger array
	nums = append(nums, 2)

	// After completing the append operation, nums has a new backing array
	// Note however that newslice still references the old array
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])
	fmt.Printf("len %v | cap %v | addr %p\n", len(c), cap(c), &c[0])
}

$ go run main.go
len 1 | cap 1 | addr 0xc0000120d0
len 1 | cap 1 | addr 0xc0000120d0
len 2 | cap 2 | addr 0xc000012120
len 1 | cap 1 | addr 0xc0000120d0

So if slices just reference arrays under the hood, and slices are a slice of an array. What happens if we don't look at the value that is returned from the append?

package main

import (
	"fmt"
)

func main() {
	nums := []int{1}
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	nums = append(nums, 2)
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	nums = append(nums, 3)
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	_ = append(nums, 4)
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])
}

$ go run main.go
len 1 | cap 1 | addr 0xc00004c1e0
len 2 | cap 2 | addr 0xc00005a460
len 3 | cap 4 | addr 0xc000098000
len 3 | cap 4 | addr 0xc000098000

Everything stays the same! Which may not be shocking, but the interesting part is that the underlying array did change. And we can prove that.

package main

import (
	"fmt"
	"reflect"
	"unsafe"
)

func main() {
	nums := []int{1}
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	nums = append(nums, 2)
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	nums = append(nums, 3)
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	// Call the append() function, but DO NOT update the slice 
	// to reflect the changes
	_ = append(nums, 4)
	fmt.Printf("len %v | cap %v | addr %p\n", len(nums), cap(nums), &nums[0])

	// At this point, we already know myslice was unchanged.
	// However, its possible to see the appended value.

	// 1. Get the slice header of myslice (struct with .Data, .Len, and .Cap)
	sliceHeader := (*reflect.SliceHeader)(unsafe.Pointer(&myslice))

	// 2. Using the .Data field in the slice header
	// cast it into a pointer to an array of ints (its actual type).
	array := (*[4]int)(unsafe.Pointer(sliceHeader.Data))

	// REMEMBER: The .Data field in the slice header is the underlying array!
	fmt.Printf("nums len: %v | nums cap: %v\n", len(nums), cap(nums))
	fmt.Printf("last element in the array is: %v", array[3])
}

$ go run main.go
len 1 | cap 1 | addr 0xc0000361f0
len 2 | cap 2 | addr 0xc0000044c0
len 3 | cap 4 | addr 0xc000040080
len 3 | cap 4 | addr 0xc000040080
nums len: 3 | nums cap: 4
last element in the array is: 4

You can see from the output that the while the slice has length 3, we referenced the 4th element in the array (at index 3), and got the last value we appended. The slice remained unchanged, but the underlying array was updated to contain the appended value.

It is worth mentioning that this is a contrived example to showcase some of the internals, and isn't something you'd really see in production code.

In reality, you would just increase the size of the slice so that you were able to interact with more elements in the array. In other words, what the append operation is doing for you when you get its return value.

// nums has a length of 3, so this line 
// would panic with an out of bounds error
fmt.Println(nums[3]) // panic: out of bounds

// Recall, however, that the array itself is still 
// of length 4 and the last appended value is 4

// We can increase the length of nums which lets us 
// access more of the underlying array. 
// This returns a slice with a size and capacity of 4.
nums = nums[:4]

fmt.Println(nums[3]) // prints 4

// NOTE: [:4] means allow the slice to see the 
// 0th index of the underlying array 
// up until the 3rd index.

TL;DR.. I just want to regurgitate stuff

1. Slices are value types. They are not pointers
2. The slice type is a struct which has three private fields: array, len, and cap
3. Slices control what you can and cannot access in its backing array. It's possible to access more of the array by increasing the slice's len property via the append function
4. Slices hold a reference to an array, so it's possible that multiple slices reference the same array

While it would be nice to have an expert driver for every person who needs to get to their destination, it's not a scalable solution. Each person has their own specific timeline and destination.

Our solution as a society, then, is to give each person the ability to handle their own transportation. You take a driver's test to prove that you have a fundamental understanding of the rules of the road and have the ability to actually drive the vehicle. If you pass, you are granted the privileged to join the open road with millions of other human beings. While it sounds like chaos, and honestly it sometimes is, how do we seemingly manage to pull it off day in and day out?

The answer is policies.

To keep drivers from getting distracted while on the road, we've enacted distracted driver policies. These policies include rules that prohibit drivers from using a cellphone while on the road, and bar the use of earbuds when the car is in motion.

We have policies around safety that state that cars must abide by all traffic signs which tell us when we should stop, and how fast we can go.

The key of this approach to enforcement is that it is generic and focuses on what the policy ultimately sets out to do. It doesn't matter if you're driving a Ford, a Chevy, or a Dodge, you're beholden to the same policies as everyone else.

So what's the point of all of this? The same philosophy can be applied within an organization! When policies are enforced within an organization, employees are more empowered to complete their tasks on their own, but are still kept in check by the policies that have been put in place.

You may be thinking, how does a policy differ from permissions?

Permissions for the most part, are relatively easy to wrap our heads around. You either have a permission, or you do not. If you do not have permission perform an action, well, you can't perform that action. This approach is unwavering and not flexible as there are many other aspects that should be considered when someone can or cannot do something.

If you add in roles, we can now reason about what sorts of permissions someone should have given their role in the company. Truthfully, for small systems, this could be completely acceptable. Where roles and permissions fall down, is when you consider larger systems, especially with multi-tenancy.

Let's consider the driving scenario again in the context of the USA.

The United States would be an organization, and the states would be tenants. In this context, it wouldn't be far fetched to have a "allowed to drive" permission per state. Immediately we've introduced 50 permissions, per user, into our system. Not only is that a lot of permissions, we'd be going about securing the roads in the wrong way.

As an organization, we don't necessarily care whether or not you can drive in a specific state. We have driving laws that are applicable everywhere, and really, as long as you abide by them that's all we should care about. This approach gives us an immense amount of flexibility. It would be crazy to have to pass a drivers test for every state you wanted to drive in! So why do we do it in software systems?

The scenario described above, associating a role (or permission) with every specific use case is commonly referred to as role explosion. So what's the alternative?

Attribute-based Access Control

The key difference between ABAC and a role-based permission model is that ABAC considers the context when an authorization request is made. Remember roles and permissions are binary, you can perform the action, or you cannot. Our rules and regulations around driving is a pretty good example of attribute-based access control.

For example, lets say when you pass your driver's test you are added to the Driver role. Then when you're on the road you're constantly being validated by policy checks. Are you going the speed limit? Are you on the correct side of the road? As you drive, we have that context, and the policy engine (police officer) can determine if you are violating the policies that the United States have set for all of its tenants.

If not already obvious, it is worth pointing out that in this scenario we did still use a role. Attribute-based access control can actually use roles as part of its decision making process!

Leveraging policies instead of fine grained permissions gives us a lot of flexibility and lets us focus on what we actually care about. Whereas permission models are often far too restrictive and ignore the context in which the action is being performed in.

Now, we're all well versed in checking specific permissions and roles. One little if() statement can give you a yes or no answer to whether or not the action can be performed. But what does validating a policy actually look like?

Policy Validation with Rego and OPA

I won't dive too deep into Rego as that could be a post in itself, but it is worth introducing Rego as a solution to policy validation for familiarity.

Rego is a policy language drives Open Policy Agent policies. Keeping with the driving theme, an example distracted driving policy would be:

In order to reduce the risks associated with distracted driving, certain conduct is prohibited while driving a company-owned motor vehicle or while driving a personal vehicle while on company business, including: 

• Using cell phones (including hands-free) 
• Operating laptops, tablets, portable media devices, and GPS devices 
• Reading maps or any type of document, printed or electronic

^{Taken from https://emcins.com}

This policy, represented in Rego, might look something like the following:

package distracted

disallowed_conduct = [
    "USING_PHONE",
    "OPERATING_MEDIA_DEVICE",
    "READING_DOCUMENT"
]

deny[msg] {
  input.vehicle.status = "IN_MOTION"
  input.driver.conduct = disallowed_conduct[_]
  
  msg = sprintf("invalid conduct: %v", input.driver.conduct)
}

In this Rego policy, all of the conditions must be met in order for the policy agent to return a deny result for the authorization request.

If the vehicle has a status of IN_MOTION and the operator's conduct is in the disallowed list, we set the msg stating that they are in violation of the policy.

We'd also probably have another policy somewhere, outside of the distracted driving policy, that checks to make sure the operator has the Driver role:

package global

deny[msg] {
  not input.operator.role = "Driver"
  msg = sprintf("Operator must be in the %v role", input.operator.role)
}

Rego input is the JSON we all know and love, so an authorization request in this context could be:

{
  "operator": {
    "role": "Driver",
    "conduct": "USING_PHONE"
  },
  "vehicle": {
    "status": "IN_MOTION"
  }
}

The power of this approach is that because it is so flexible, you can validate policy against almost any data set as long as it has some structure to it.

• You can validate that a Dockerfile only uses FROM on blessed images.
• You can validate that a .tf file from Terraform only uses specific providers.
• You can validate that a Kubernetes manifest contains specific label(s).

.. just to name a few of the possibilities.

Because when you really think about it, these are the things we really care about. We don't necessarily care about a permission, we grant permission because we trust someone to perform an action, because we trust they will enforce the policy.

Here we can just define the policy outright and let anyone* make changes, because we know the policy will always be enforced.

Like everything else in software and in life, it's not a silver bullet. You're still going to use roles and permissions to some degree, and you can't have policies around everything. It simply opens doors to additional approaches to automation via automatic approvals, and explicitly defines what our policies are in code.

If you're interested in seeing some more examples of what types of files you can validate, and what those policies might look like, you can checkout the examples folder for conftest which can be found here: https://github.com/instrumenta/conftest/tree/master/examples

I do help maintain the project so if you do decide to explore it, I'd love to hear your feedback to help make it better.

You may have heard of drifting into failure before. Sidney Dekker published an entire book around the idea, aptly named Drift into Failure. He presents it as the idea that every small problem we ignore (or cover up), gets us closer to failure. What is a problem? What is a failure? That's one of the great things about keeping the idea abstract. It really lets you fill in the blanks and make up your own definitions.

Miss an oil change? Not the end of the world. Keep on putting it off week after week, month after month? Eventually it's going to catch up to you.

Deadlines pushing you to implement a less than ideal solution? Again, probably not the end of the world, but if you keep implementing those less than ideal solutions, you're going to eventually end up with a mess.

You get the picture. The idea here is that failures aren't often born from a single event, but the culmination of a lot of little events that didn't necessarily mean much on their own. We take little shortcuts here and there, but in the end, they will catch up to us in a potentially catastrophic way.

In the context of this post, I want to continue using this idea of drifting into failure, but speak specifically to how it applies to infrastructure.

To begin, you should be aware that there exists an approach to defining infrastructure using code. This is called IaC or Infrastructure as Code. It's a contract that says, make my infrastructure look exactly like this.

One alternative to Infrastructure as Code, is having someone on an operations team make ad hoc changes to the infrastructure. They would be notified that the infrastructure needs to change, so they would login and run some commands.

NOTE: Unfamiliar with the idea of contracts? You can read all about what they are and their benefits in Locked into a Contract? Lucky You!

These are two very different styles of managing the infrastructure. One is an imperative approach, and the other is a declarative approach. These are two very important concepts, so it's worthwhile explaining them.

Imperative and declarative approaches

For me, one of the easiest ways to reason about both of these approaches is that declarative focuses on the what and imperative focuses on the how.

If I wanted you to come to my office, I could say in a declarative way "Come to my office". I don't care how you get here, but my desired state is that you are in my office. Alternatively, I could say in an imperative way, "Leave your office. Go to the elevator. Take the elevator to the 14th floor. Walk into my office".

The end result is the same in both scenarios, but the declarative approach gives you the freedom to do it however you choose to get it done.

It's also important to note that declarative approaches often result in a series of imperative statements. To make that a little more clear, lets continue with the above example. I told you to come to my office, in a declarative manner. But that doesn't mean you just magically teleport in front of me. I made clear what I wanted the end result to be, but I left it up to you to figure out the individual, imperative steps to get there.

Dockerfiles are a great representation of this, in my opinion. A Dockerfile is a series of imperative commands, but they get bundled up into some desired state. The act of using an image from a Dockerfile is declarative (I want exactly this), but the steps required to get there were imperative.

Another important aspect of declarative approaches, is state. In a declarative environment, we need to know what the current state of the infrastructure is. We also need the ability to figure out what needs to be done to get the infrastructure into that desired state. For example, if I tell some process to make my infrastructure look like X, and that process gets interrupted, it should be able to figure out whats remaining to get the infrastructure to look like X when it resumes.

Imperative does not allow for this, as it has no notion of state. While imperative approaches are simpler, the lack of state means that if the process is interrupted we have to start over again, because we don't know where we left off.

This is the power of being declarative and using contracts. They allow us to say exactly what we want the environment to look like with the contract, and leave all of the finer details up to the tooling that is fulfilling the contract.

Declarative infrastructure

You may have figured out by now that Infrastructure as Code is very much a declarative way of defining what your infrastructure should look like. There are a lot of benefits to leveraging Infrastructure as Code. To give you an idea, here are a few that really stand out for me:

Always know what your environment looks like

How many times have you wondered what version of .NET is on the build servers? What was your solution to finding out that information? Most likely it was to call up someone from the Ops team and have them tell you. Using Infrastructure as Code, the configuration of each environment is stored in source control, just like code. If you want to know some properties of an environment, you just need to go to the infrastructure repository and look for yourself.

Push button environments

You'll most likely be hearing this phrase a lot in the days to come. The idea is that we should be able to define exactly what we want an environment to look like, and have some tooling do all of the work to fulfill our contract. Bringing this close to home, think about the time and effort it takes to stand up a new N. It takes a lot of people, a lot of time to make sure everything is just right.

The holy grail for push button environments is you do just that. You push a button, and after some time your environment will materialize before you. There shouldn't be any assumptions about where you are wanting to stand up your environment. Everything required to get that infrastructure up and running should be contained within the contract.

Testability

When we talk about testing, our minds most likely go to testing to make sure our applications work. We have an environment we want to deploy to, and we need to run our unit, integration, <noun> tests to make sure that our application will work when deployed. But where did that environment come from? How can we be sure the environment is correct?

Because Infrastructure as Code is.. code (it's right there in the name!), it gives us the ability to run tests against the infrastructure before it is actually applied. Some tests might be verifying that the docker daemon is installed and running or that some registry values exist. The key here is that we can now test the environment that our applications will be deployed to.

What does any of this have to do with drift?

Glad you asked! One of the key takeaways about contracts and Infrastructure as Code is that you cannot be allowed to make any change to the environment without updating the contract. Doing so, causes drift.

Environment drift occurs whenever someone changes the environment (applies a patch, adds some new registry value, etc) without updating the contract. The whole point of all of this is that we know exactly what the environment looks like, we can easily stand the environment up anywhere because everything about the environment is outlined in a contract, if the environment differs from the contract, then we lose those benefits.

When working with Infrastructure as Code, the same principles and processes we apply to deploying to applications, should also be applied to changes to our infrastructure. We don't just open up some source code on the production server, make a change, and apply it. The same rules and regulations should apply for our infrastructure.

If you want to make a change to the infrastructure, you must make a change to the contract and submit a pull request. If approved, that infrastructure will go through a series of tests, and will eventually be applied to the intended environment. How that change is applied could even mean we completely destroy the old environment and just stand up the result of the pull request.

Hooray immutability!

If you've ever had to sign a contract, you may be able to relate. They can be a pretty big deal. Both parties are explicitly agreeing to terms and conditions that cannot be violated, regardless of what may happen in the future. While contracts may feel like a burden in the real world, they are extremely powerful in software.

So what is a software contract? In practice, it doesn't differ that much from traditional contracts. It's an agreement between systems about what functionality is exposed, the data types of each message, the structure of the resource, etc. This allows us to forego a lot of manual validation that is typical when you're not exactly sure what's going to be sent on the wire or what the environment looks like.

Life without Contracts

Consider two independently deployed web services: PingService and PongService. When PingService sends a message to the PongService, the PongService should return a message, letting the PingService know the request was received.

Immediately we're in a state where there are more questions than answers. We know these services can talk over HTTP.. but that's about it.

• How do we structure the message to send to the PongService? Is it expecting JSON? XML?
• What type of data should we send to the PongService? A Boolean? Integer?

The answer that more than likely comes to mind is to.. look at the API documentation! Which, assuming it was kept up to date, should answer those questions for us.

Ok great, lets assume the documentation is up to date and accurate. We find out that it is indeed JSON and is expecting a string. Apparently the PongService just returns the string that was sent as its response.

So the next steps most likely involve creating some model to represent the result of calling the service, importing Newtonsoft.JSON so we can do some JSON serialization and deserialization, and then ultimately POST to the endpoint (or was it a GET request?). After all of that is implemented, we finally validate whether or not both our request and response was formatted correctly.

Now that seems like a lot of work, manual work, to make a web request. The problem gets worse when you also consider that this process would need to be done for each and every service that wanted to interact with the PongService. That's a lot of code duplication that's scattered throughout the entire ecosystem that each team would need to manage.

And there is.. with Contracts!

Contracts and the tooling that support them go a long way in solving the aforementioned problems. Lets take the same scenario as described above, but walk through it leveraging contracts.

The first thing we need to do is define the contract itself. This can be in any format: JSON, YAML, or in this case, Protocol Buffers. Generally the format that you decide on will be in the format expected by the tooling and technologies you intend to leverage. The key take away here is that you define upfront how messages should be structured and the types of each.

This is a .proto file that explicitly defines how to interact with the PongService. The .proto file is the contract.

Messages are sent to the service using a PongRequest which has a message field, and then messages are received using a PongResponse which also has a message field. There is one endpoint that the service exposes, Get.

Ok, so you might be thinking.. "Cool? Looks a lot like API documentation to me. I'll go start writing some code that structures messages this way."

The beauty of leveraging contracts is that because there is an agreed upon way to interact with the service, it's really easy to automatically generate all of the concerns we spoke about in the previous example.

From the contract, we can automatically generate the PongRequest and PongResponse objects, there is no need for a developer to ever create them. We can automatically generate a Client so that you can connect to the service easily. We can automatically generate fakes, stubs, and mocks for testing. Everything you need to interact with the service can be yours at the press of a few buttons.

This moves the validation and enforcement of the contract to the left, way earlier in the development lifecycle. Remember in the previous example, we didn't really know if things were going to work out until the very end. The code generated from the contract gives us compile time confidence that the data is going to be sent and received a specific way.

We can even take this a step further, and from the same .proto file (contract), generate API documentation

The beauty of all of this is that if you were to change the contract, all of the resources that were generated above would be automatically regenerated during the CI/CD process. No more going into source code and updating client models or firing up some markdown editor to update your API documentation.

Contracts also makes it a lot easier to manage versioning and assess the ramifications of changing the contract when things do inevitably need to change. We can be notified on our local machines if a change that we are introducing breaks the currently defined contract. There's no need to ask ourselves "Is this a breaking change?". We can let the contract validation handle that for us.

Summary and Takeaways

This is an incredibly large topic as it can be applied to almost anything. This post really just scratches the surface. Hopefully, however, it was enough to introduce the idea of contracts and the benefits they can bring when you have a single, unwavering, source of truth that your ecosystem abides by.

Contracts enable the use of code generation with confidence. Why is that important?

• Code generation is your friend. There's no need to write your own client libraries in order to interact with a service.
• Code generation is your friend. There's no need to write your own API documentation.
• Code generation is your friend. There's no need to write your own testing stubs.

This fascination only increased over time. After building websites and learning HTML, I moved onto PHP which opened up a whole other realm of possibilities for what I could put onto the Internet. I sold my first website in the 5th grade. It was just a little website for a band where it listed some information, mostly dates and some background information, but also the ability to send an email to the group with a form.

I can't say for sure when I knew software development was going to be my career, but it could have been at that moment. It has just felt like I've always known. I've considered myself lucky in that regard. Always knowing what I was going to do in life.

Not surprising, but I moved on to major in Computer Science, obtaining my Bachelor's at Michigan Technological University. One short month later, I began my career as a professional software developer.

I cannot adequately explain the excitement I had in the early days of my career. Up until then, my exposure to software development was just my own tinkerings and whatever my professors decided to assign me. I had no notion of what "production" meant, multiple web servers, databases, or deployment pipelines. I just used phpMyAdmin to create some databases and dropped my changes into my FTP client.

I think on my second day I was running into an issue where I couldn't find the source of the data that was displayed on a webpage. I was looking right at the database though. The name of the database was right, the table schema was right, but the rows just weren't there. Hours later I come to find out that there's this thing where you have multiple, identical databases, that are hosted in different locations to address scalability concerns.

WHOA! Oh, OK, so I just need to change this connection string thing so I can connect to the database where the data I'm trying to see actually exists.

This was life as a developer for quite some time. Everyday a new problem. Everyday learning about a new solution.

As time went on however, it got less exciting. There just became a time where I wanted to keep learning, progressing as a developer, but the work itself stayed the same. Company politics and arbitrary deadlines lead to a road of bubble gum and duct tape.

For a long time, a really long time, I didn't feel like I was really making anything. I was just patching the system to keep it running, alone. Collaboration wasn't something that was necessarily seen as a strong need. You are given this task, and you should be able to complete it on your own. Communication just takes up time you could be programming.

Everything was expected to just work, without much regard to the part about coding I cared the most about, the craftsmanship. We were a feature factory.

Even though I was utterly disenchanted for quite some time, I stuck with it. My friends were here, the people were amazing, and the benefits weren't to be taken for granted. It was a strong culture.

To fill in the gaps left by work I started blogging. I started going to every conference I could reasonably attend, and even spoke at some. I went on to get a Master's degree. Anything and everything I could think of to keep my passion in software alive. I figured this is how it had to be, unless I wanted to seek greener pastures elsewhere.

Luckily for me, an opportunity eventually came about. I was invited to join a team whose mission was to re-imagine how we develop at the company. We were to focus on all of the problems that plagued us as a company, and find solutions to them. But not just patch the existing system, and maybe make life slightly better, no. Throw out all restrictions, all previous decisions, and build something from nothing, the right way.

Reimagine how we set up our local environments, no more multiple day setups.

Reimagine how we develop our software, no more READMEs, use tooling.

Reimagine how we test our software, no more deploy to production and pray.

Reimagine the infrastructure our software runs on, no more distributed monolith.

I see it as a project born out of empathy. Most everyone at the company is painfully aware of the problems we face, but lack the time and power to fix anything in a meaningful way. We know what the problems are, we know the situation everyone is in, and it's a painful place to be in. Our true north is bettering the developer experience, and by proxy, bettering the company. If our actions do not reflect that, we've fallen down somewhere.

We've been working on this project for a little bit now, and I can't put into words how reinvigorating it is. Actions have meaning, collaboration is a first class citizen, and all of my teammates share my passion for development and doing it the right way.

It is indeed a welcomed change, and I am ecstatic about the future.

There are fifteen in total so without further ado, lets begin!

1. ASCII Art
2. Busted File
3. Esoteric Stuff
4. Espresso
5. Krafty Kat
6. Dot Nutcracker
7. Capture the Falg
8. Ghost Text
9. The Parrot
10. Stacked Up
11. Unicorn
12. Danger Zone
13. Mrs Robot
14. Cars
15. On Site Challenge

ASCII Art

The first challenge that was unlocked was some ASCII art that spelled out "CODE MASH". It also included this sentence: I love ASCII art. What about you?

Though if you notice, the word CODE has some numbers scattered throughout.

The problem seems to be hinting pretty hard that this problem is relating to ASCII and ASCII codes. So I tried taking each number in the word CODE, assumed it was an ASCII number, and tried to translate it to a character.

cm19-asci-CODE-a1nt-H4RD

Well, that looks like a flag to me!

Busted File

We are given a zip file (busted.zip) that contains a single image (image.png). Though when I tried to extract the image, I got an error saying that the archive is corrupt.

Upon trying to repair the archive using WinRAR, I was given an error about a Corrupt header.

I had no idea what a zip header should look like, so I created my own zip file and opened it up into a hex editor.

You can see here that the newly created (and valid) header contains:

50 4B 03 04 14 00 02 00 08

Opening the corrupt file, you'll notice that the header contains:

DE AD BE EF 14 00 00 00 08

Using the hex editor to replace the corrupted header with the expected header allowed me to successfully unzip the file.

The contents of the zip file contained the following image:

This approach of hiding text within an image is called steganography. There are a couple ways that this could have been done.

1. The flag could be rendered into the image itself. Typical methods of extraction could be to invert the colors, adjust the hue and saturation, etc. Play with the layers and colors to see if the original author put the text into the image itself.
2. Another approach, and the actual solution to this challenge, is to embed the text into the actual file. If you open up this image with Notepad or a hex editor, towards the beginning of the file you'll see the flag in plain text.

Esoteric Stuff

The third challenge contained a large block of periods, exclamation points, and question marks.

....................!?.?...?....
...?...............?............
........?.?.?.?.!!?!.?.?.?.?!!!.
....................!.?.?.......
................................
!.................!.!!!!!!!!!!!!
!!!!!!!!!!!!!..?.?!!!!!!!!!!!!!!
!!!.............................
!.?.?.......!..?.?!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!.?.?.!!!!!!!.
.?.?........................!.!!
!!!!!!!!!!!!!!!!!!!!!...!.?.....
..!.?.!..?.?....................
........!.!!!!!!!!!!!!!!!!!!!!!!
!!!!!...........................
....!.?...........!.?.!..?.?!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!.?.?.......!..?.?..!...!.?.?.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.

It also included this sentence: It's getting a bit esoteric now. Ookay?!

Note the bolded text.

If you're unfamiliar with the term esoteric it means that something is only understood by a small set of individuals. In the world of programming, it refers to esoteric programming languages. These languages are typically just meant for fun and really aren't intended for production use. My goto example of an esoteric language is LOLCODE.

I also noticed that the Ook in Ookay was bolded. I made the conclusion that Ook must be another esoteric programming language and the given text is source code written in Ook.

I attempted to use an Ook compiler to retrieve the flag, and.. success!

Espresso

This challenge only provided a single file, Espresso.class

Luckily, class files are something that you can decompile pretty easily, so the first step I took was to take the class file and put it through a Java decompiler which resulted in the following source:

Looking at the source of the class file, I saw that there is a conditional that checks to see if the user's input was equal to localStringBuffer. If it matches, the program will print out: Flag correct, well done!

That seems like a pretty good starting point, right?

locationStringBuffer is a private variable that is defined and then built up in a for loop. After the for loop completes, the variable doesn't change and should theoretically contain the flag.

To get what localStringBuffer would be after running the for loop, I just went ahead and removed all of the extra code and just focused on getting localStringBuffer to be built.

Running this slimmed down version of Espresso.class yields the flag.

Krafty Kat

RSA encryption.. my dear old friend. This was the first challenge with a difficulty of "hard", and honestly took a decent bit longer than the other ones to crack.

This challenge has a flag.enc file and a key.pem file. The flag is obviously in the flag.enc file, and is encrypted. I knew I needed to use the provided public key in order to gain access to the encrypted file. The public key looked like this:

-----BEGIN PUBLIC KEY-----MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEIAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEcwIDAQAB
-----END PUBLIC KEY-----

Now, how RSA encryption works could be a post all in itself, so I'll just go over each step and describe whats happening.

The above public key is a base64 encoded string that contains what is called the Modulus in RSA. We can use openssl to extract this result like so:

Here we can see the Modulus and the Exponent (65537). Now, the Modulus is currently in hex, but I need to get this number as an integer.

The first step to do this is to strip away all of the colons

800000000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000473

The next step is to convert this into an integer. I used python to cast it, but the method doesn't really matter.

258536048570605626988915097172641057803353415992887757589736645808726151727194085610744190012322229640908553929414833022731988789391928353742297595957524280335918484224592037842886381777837922164216258913020883767452162759055614423529478656392491416399318604362756647059576165428708885769576577464120568116338612287783043168328279883802253392551292712278925133675758884573430117126518137049529478694956180472120065649305066144792948879520988172974401270565515875720597321219813209954571652637983550511681618170361995093036165713586725018263017699754932627926751512891645482818654590419836934782173099306176053375927411

When working with RSA, a really important formula is n = p * q, and the number above (the really big one), is my n value. n is derived from the product of p and q (both of which are prime numbers) resulting in the equation n = p * q. If you know p and q, you can derive the private key.

However, these prime numbers are kept secret and should not be known. It is possible to figure out what they are though if the RSA was implemented poorly (e.g. bad primes were chosen).

So if n is the product of p and q, and both p and q must be prime. How do we figure out what they could possibly be? Brute force is one option, but would take.. a long time. Too much time.

The alternative is to see if any factors of n already exist. I did this through a website called factordb. I plugged in the n value I calculated above and noticed that there are no factors.l

Well.. if n = p * q and p and q must be prime.. but n has no factors.. p must be equal to n and then q must be equal to 1.

At this point we know p, q, e, and n. The last value we need is d or the decryption key. Honestly at this point, d can just be calculated from all of our other values so I just wrote a script in Python to do just that.  

from Crypto.Util.number import inverse
from Crypto.PublicKey import RSA
import gmpy2

#open the provided key file
pub = open("key.pem", "r").read()
pub = RSA.importKey(pub)

#set the values of n and e from the key file
n = int(pub.n)
e = int(pub.e)

#set p and q from factordb
p = 258536048570605626988915097172641057803353415992887757589736645808726151727194085610744190012322229640908553929414833022731988789391928353742297595957524280335918484224592037842886381777837922164216258913020883767452162759055614423529478656392491416399318604362756647059576165428708885769576577464120568116338612287783043168328279883802253392551292712278925133675758884573430117126518137049529478694956180472120065649305066144792948879520988172974401270565515875720597321219813209954571652637983550511681618170361995093036165713586725018263017699754932627926751512891645482818654590419836934782173099306176053375927411
q = 1

#calculate d
d = int(gmpy2.invert(e,n-1))

#use d to create a private key and read the message
key = RSA.construct((n,e,d))
message = key.decrypt(open('flag.enc').read(), '')

print(message)

This will print the contents of the flag.enc, which contains the flag itself!

Dot Nutcracker

In this challenge, there was an executable file that gave you three attempts in order to type in a password. If you got the password correct, the program would output what the flag was.

Now obviously, the three attempts were just fluff, as you could just start the program again. Regardless, the password (and the flag) have to be inside of the executable somewhere.

To get at the source code of the application, I used decompiler called dotPeek which allowed me to view the source of the executable. I even took it a step further and exported the source to a new project. This allowed me to make modifications to the program itself, as well as do any sort of debugging.

Reading through the source code, I came across the primary if check that compared the users input with the stored password.

      while (num > 0)
      {
        Console.WriteLine(string.Format("You have {0}/3 attempt(s) left.", (object) num));
        Console.Write("Enter passphrase: ");
        string str1 = Console.ReadLine();
        string str2 = CryptoHelper.DecryptStringAES("EAAAAH5ZA4kASLVjLUsYmLK3h74KWmkS4BvBS61BuaD4lnyqdz3AO8/xfGO1atVdci0x1g==");
        if (str1 != null && cryptoHelper.Equals(str2, str1))
        {
          Console.WriteLine(string.Format("Decrypted Flag is: {0}", (object) CryptoHelper.DecryptStringAES("EAAAAOlDKPcRaUj/ITV1q9IHN1bAQyUWxZqVob+G1gpmyoIJIPej1O3T4TWnRUndqp4NnA==", CryptoHelper.GetHashString(str2), str1, str1)));
          Console.WriteLine("Press enter to quit");
          Console.ReadLine();
          Environment.Exit(1337);
        }
        --num;
        Console.WriteLine("That's wrong.");
      }

Reading the code above, it looks like str2 contains the password. I did a Console.Writeline of str2 and saw that acrackeradaykeepsthedoctoraway is printed out. Success! Found the password.

However, when attempting to use the password, the program crashes with the following message:

I went through the source code some more and noticed that the original if statement that compares the passwords is actually using CryptoHelper's Equals method and it has the following implementation:

public bool Equals(string a, string b)
{
  if (!b.Equals(a))
    return a.Equals(b + "y");
  return true;
}

This means that both acrackeradaykeepsthedoctoraway and acrackeradaykeepsthedoctorawa should both be correct passwords. Well, if the former causes the application to crash, maybe the second one wont.

It did not!

Capture the Falg

This challenge had the following text:

This falg is easy to catch, isn't it?
criSdmetio1AUdW9dP3n----

and gave this picture:

If you notice, the name of the challenge is Capture the Falg and not Flag. The a and l are jumbled up. The image is also jumbled up. Another interesting notice is that the word falg has 4 letters, and theres also 4 boxes for each piece of the flag.

To render the image correctly, you'd want the 1st image to stay where it's at. Followed by the image directly below it, followed by the top right image, and lastly keep the last image where it's at.

Applying this same approach the word falg, you would get the following:

fa
lg

To read the word flag you need to start from the top left, move down, and then go back to the top in the next column.

Now, if we were to apply this same pattern to the string that we were given (presumably the flag itself), we get the following:

criSd
metio
lAUdW
9dP3n
----

Reading the text just like before will render the flag.

Ghost Text

For this challenge the description was:

Ghost text is ... invisible!

The challenge also provided a text file with this text:

​In​ ​folk​l​or​e​, ​a​ g​hos​t​ is ​the​ ​s​oul​ or​ s​p​ir​it​ ​o​f a​ dea​d​ pe​r​son​ ​or​ ani​m​al ​that ​ca​n​ a​pp​e​ar t​o t​h​e​ l​ivin​g​.​ I​n​ ​gh​o​st​l​o​re, ​gh​o​st​ d​e​s​cr​ipti​o​ns​ vary​ ​wid​el​y,​ from​ ​inv​is​i​bl​e ​pre​sence​s to l​if​el​ike​ vi​si​ons​.

Now, given the challenge description, I assumed there was something hidden inside of the text file that was given. So I opened it with my trusty HexEditor.

You'll notice that there's a lot of jumbled characters intertwined between the text itself, characters we could not see before. These characters are known as zero width characters and are a common approach to hiding messages in text.

To more easily digest how this message really looks, we can use a unicode analyzer. I've provided the link below.

https://www.fontspace.com/unicode/analyzer/?q=%E2%80%8BIn%E2%80%8B+%E2%80%8Bfolk%E2%80%8Bl%E2%80%8Bor%E2%80%8Be%E2%80%8B%2C+%E2%80%8Ba%E2%80%8B+g%E2%80%8Bhos%E2%80%8Bt%E2%80%8B+is+%E2%80%8Bthe%E2%80%8B+%E2%80%8Bs%E2%80%8Boul%E2%80%8B+or%E2%80%8B+s%E2%80%8Bp%E2%80%8Bir%E2%80%8Bit%E2%80%8B+%E2%80%8Bo%E2%80%8Bf+a%E2%80%8B+dea%E2%80%8Bd%E2%80%8B+pe%E2%80%8Br%E2%80%8Bson%E2%80%8B+%E2%80%8Bor%E2%80%8B+ani%E2%80%8Bm%E2%80%8Bal+%E2%80%8Bthat+%E2%80%8Bca%E2%80%8Bn%E2%80%8B+a%E2%80%8Bpp%E2%80%8Be%E2%80%8Bar+t%E2%80%8Bo+t%E2%80%8Bh%E2%80%8Be%E2%80%8B+l%E2%80%8Bivin%E2%80%8Bg%E2%80%8B.%E2%80%8B+I%E2%80%8Bn%E2%80%8B+%E2%80%8Bgh%E2%80%8Bo%E2%80%8Bst%E2%80%8Bl%E2%80%8Bo%E2%80%8Bre%2C+%E2%80%8Bgh%E2%80%8Bo%E2%80%8Bst%E2%80%8B+d%E2%80%8Be%E2%80%8Bs%E2%80%8Bcr%E2%80%8Bipti%E2%80%8Bo%E2%80%8Bns%E2%80%8B+vary%E2%80%8B+%E2%80%8Bwid%E2%80%8Bel%E2%80%8By%2C%E2%80%8B+from%E2%80%8B+%E2%80%8Binv%E2%80%8Bis%E2%80%8Bi%E2%80%8Bbl%E2%80%8Be+%E2%80%8Bpre%E2%80%8Bsence%E2%80%8Bs+to+l%E2%80%8Bif%E2%80%8Bel%E2%80%8Bike%E2%80%8B+vi%E2%80%8Bsi%E2%80%8Bons%E2%80%8B.%E2%80%8B

Using this website, we can see every single character that is contained within the string. You'll notice that there are "ZERO WIDTH SPACE" characters throughout the text. This is important!

This approach to hiding messages in text generally involves the use of binary. Every approach to encryption is going to be different, but most messages end up as some sort of binary representation. Maybe each zero width character is a 1 and each real space is a 0. These 1's and 0's will form a binary string, which can then be translated into readable text.

It's also very common for CTF events to use the same flag structure. In this one, all of the flags (so far) have started with cm19.

With this in mind, I wanted to see if I could extract cm19 in binary from the zero width spaces.

cm19 in binary is: 01100011 01101101 00110001 00111001

If you reference the unicode analyzer page linked previously, you may see a pattern here.

The word "in" is followed by a zero width space.

The single space afterwards is followed by a zero width space.

The word "folklore" has a zero width space after "folk"

If we assume that the zero width spaces are some sort of terminator, it actually starts to fit our assumption of the cm19 binary string.

In would be 01

A single space would be 1

Folk would be 0001

.. and so on. If we combine the above we get 0110001 which is the start of our cm19 binary string!

Now I had no desire to go through this entire sentence by hand and finish out the sequence, so I wrote a quick GOLANG program to do it for me.

package main

import "strings"

func main() {
	input := "​In​ ​folk​l​or​e​, ​a​ g​hos​t​ is ​the​ ​s​oul​ or​ s​p​ir​it​ ​o​f a​ dea​d​ pe​r​son​ ​or​ ani​m​al ​that ​ca​n​ a​pp​e​ar t​o t​h​e​ l​ivin​g​.​ I​n​ ​gh​o​st​l​o​re, ​gh​o​st​ d​e​s​cr​ipti​o​ns​ vary​ ​wid​el​y,​ from​ ​inv​is​i​bl​e ​pre​sence​s to l​if​el​ike​ vi​si​ons​.​"

	result := ""

	for _, v := range input {

		if v == 8203 {
			result = strings.TrimSuffix(result, "0")
			result += "1"

		} else {
			result += "0"
		}

	}

	print(result)
}

Running the application gives the following binary string:

011000110110110100110001001110010010110101110010001100110011010001100100001011010110001001110100011101110110111000101101011101000110100001100101010000110010110101001000010000010101001001010011

Which can then be converted to the flag itself using an ASCII to hex converter.

The Parrot

This challenge had a single pdf file called parrot.pdf. Though when attempting to open the pdf to view it's contents, I was presented with a password dialog.

This is a password protected PDF and I needed a way to get the password so that I can open the PDF in order to see what's inside.

For this, I used a tool called PDFcrack and a collection of commonly used passwords called rockyou.txt. Putting the two together, I got the following result:

You'll notice on the very last line, the program managed to find the password "cracker".

Note: I later found out that if you open the pdf in a text editor and look at the Authors, it actually spells out the password if you look closely:

<< /Authors: charlie romeo alfa charlie kilo echo romeo >>

Moving on!

When the PDF was opened, it contained the following image:

In order to get the image out of the PDF, I used another tool called pdfimages. At this point, I already knew the password, so I ran pdfimages with the password 'cracker'.

pdfimages parrot.pdf -upw cracker .

This command extracted two images in .ppm format. Opening the first image and.. tada!

Stacked Up

The challenge description stated that there was a vulnerable service running in the cloud and that it could be exploited in order to get the flag. A copy of just the binary was given freely for download.

Running the program locally, it just looked like it echo'd back whatever I typed in.

Doesn't seem like a very useful program, does it? I checked out what the hint had to say about this challenge. The hint plainly said:

HINT: What happens if you input a veeeeeeeeeery long string?

Fair enough, I tried entering in a long string.

I didn't think that was incredibly surprising. With input that large I probably caused a buffer overflow.

However, at this point, I went down a pretty large rabbit hole. I spent a lot of time focus'd on the core dump, convinced that there was going to be something valuable inside of the stack trace when the program errored. Ultimately, I did not really get anywhere going down that path.

The next thing I wanted to try, was just to decompile this binary and see if I could make any sense of what was going on inside.

After decompiling the binary using IDA, I noticed a couple really interesting things.

First, the main method had no jumps. There weren't any if statements and all the main() method did was get user input and print it out. I did not expect that at all.

Second, I noticed that there was a method called flag()

This method looked almost identical to the main() method with the exception that this method tried to open a file called flag.txt, and then print its contents.

I knew at this point the goal was going to be to call the flag() method. There must exist a flag.txt file on the remote server where the binary is running, and calling the flag method would read that file and print it out.

In order to do this, the return address of the main() method would need to be overwritten.

When a function is called (in this case main), the return address is stored on the stack and is just above the base pointer. If I overwrite the return address of the main method to point to the flag() method instead of the default return address, the program should call the flag method.

To accomplish this I needed a couple of things.

First, since the return address is right after the base pointer, I needed to know exactly how many characters were required to overflow the program before I started to spill over into the return address.

You can see here in this photo that the return address looks like

0xff021b92 0x00007fff (the last two hex values on the top row)

When I used a lot of A's in order to get the program to overflow, I noticed that the return address starts to get overwritten at the 1033rd character.

See the 41's that are bleeding over into the 3rd column?

Ok, good progress! The 1033rd character is the start of the return address, so we need to fill the buffer with 1032 characters of junk, and then tack on the return address of the flag() method.

Using IDA again, I got the address of the flag() method just by clicking on the starting line of the method (0x400676):

At this point, I have everything needed in order to call the flag() method. To create the input for the application, I just used python to generate all of the junk characters as well as append the address of the flag method to the end of the string

python -c 'print "A"*1032 + p32(0x400676)'

Connecting to the remote service, and using the generated input overwrites the return address, calls the flag() method, and prints the flag!

Unicorn

This challenge was relatively straight forward. A Unicorn.svg file was given, and that's about it.

Typically in CTFs, if you're given a svg, they expect you to manipulate the svg in some way in order to see the flag.

In the case of our cute little Unicorn, I just opened the file into an SVG editor and deleted the Unicorn. The flag was hidden behind it.

Danger Zone

This challenge had the following description:

Can you enter the **danger.zone**, and find the flag?

Service listening on:

whale.hacking-lab.com:**5553**

Note: this is **NOT** about the web site danger.zone!

Like most CTFs, a lot of clues on how to solve the problem are given in the name and the description. For this challenge, it talks about danger.zone and a service that is running on port 5553. The hint itself also really helpful:

HINT: Find out what service is listening on the port. nmap could be helpful for this kind of fingerprinting.

Sounds like a good idea to start off this challenge, so I did just that. Running nmap for port 5553 tells us that the service that is running is a domain service with a version of ISC BIND 9.9.5

With this information, we now know that port 5553 is running a domain service, which means its most likely running a custom DNS service.

For DNS problems, dig can be incredibly helpful. Since I now knew that the service is a DNS service (and is probably vulnerable), I used dig to look up some more information about it while trying to access danger.zone.

I was able to successfully query danger.zone using the DNS service, and on top of that, danger.zone exists as a record in the zone file for the DNS! Which ultimately tells me that I should be able to perform a zone transfer.

After performing the zone transfer, the zone file is printed to the screen, and I can see the flag that was put inside.

Mrs Robot

For this challenge, a webpage that prompted us for a secret word was given.

Viewing the source didn't really bring about anything of interest. Maybe the fact that each word that was typed in just redirected to a webpage that was an md5 hash of the input:

This led me to believe that the flag was an HTML file on the webserver, and the name of the HTML file was the secret word in an md5 hash.

I could use something like DirBuster to find the file for me, hashing common words and checking if the file exists, but I might be there for awhile.

Luckily, the site did have a robots.txt file that contained the following:

User-agent: *
Disallow: /481065fd79b253104aeab5ca5c717cd5.html

Well that looks promising!

Upon landing on the page in the robots.txt file, I was presented with the text:

If you're not already familiar with the Robot Interaction Language, a simple google search for both of these terms will bring up a table with all of the translations necessary to get this text into English.

Leveraging the table I got the secret word, winter, and thus the flag itself.

Cars

For this challenge, a website with a search box and a login page was given. The goal was to login to the page using the admin account.

I was pretty sure this was going to be a SQL injection challenge, and the hint confirmed that:

HINT: Can you find a way to make the search return more data than expected? What you might get, is not the password yet.

I started off by throwing some quotes at the search box as that's a pretty common approach to get some more information about the version of SQL and the query running behind the scenes. Using the following query:

a'

The result that came back was:

You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%'' at line 1

I didn't even use a %, and the behavior of the page did seem to lend itself to being a LIKE query. Inspecting the error more closely, I can see that the trailing SQL query is %'. So, my query of a' must have terminated the query, and %' is what was left over.

With this in mind I felt pretty confident the query was structured as so:

SELECT .. FROM .. WHERE .. LIKE '%' + [input] + '%'

The next step was to start filling in the blanks in order to get some more information about the underlying database.

Using the following query, we can get a dump of all of the underlying tables and their columns

a' union select table_name, column_name FROM information_schema.columns#

The a' is to finish the SELECT statement so that it's valid, and then I can UNION another SELECT statement to pull back information about the schema. Since this is a MySQL database, the pound symbol (#) is used to comment out the rest of the query. It would look something like the following:

SELECT .. FROM .. WHERE .. LIKE '%a' UNION SELECT table_name, column_name FROM information_schema.columns#%'

The hash comments out the trailing %' from the real LIKE query

The results of this query looked like this:

At this point, we have all of the information we need to pull back the data that we're looking for. The users and the password.

Doing another injection using the new tables and columns gives us this query:

a' UNION select username,password_hash from Users#

Resulting in the following records:

There it is! The hash of the admin password.

I was pretty confident the hash was going to be an MD5 hash, so I knew I needed a rainbow table and hope that the hash had been cracked in the past.

I ended up using an online Reverse Hash Calculator which did have the hash already stored.

Using the username of admin and the password of dodge123, revealed the flag.

On Site Challenge

The description for the on site challenge was in the form of a Haiku:

Games are played all night.
This one online, but there, board.
Code is history.

Reading the poem, I saw "but there, board". To me, there is hinting at the location of the flag. It is an on site challenge after all. On site, there is a place where board games are played, so that seemed like a logical conclusion to me.

Upon entering the room, I saw multiple identical QR codes on the wall:

Using a QR code reader, I saw that the QR code is actually a link to:

https://www.owasp.org/index.php/User:Bill_Sempf

Looking at the remainder of the Haiku "Code is history" I eventually figure out that its referring to the "View history" of the wiki which contains a strange entry that was added and then immediately removed about a minute later.

Viewing the revision in which lines were added, I saw what appears to be a Base64 encoded string. Putting that string into a Base64 decoder reveals the flag.

CodeMash is an annual conference that is held at the Kalahari Resorts in Sandusky, Ohio. While the conference itself is only two days long, they do offer what they call "Precompilers" which you can just think of as pre-conference workshops.

First Impressions

It seems like many conferences may focus on specific frameworks such as .NET, or languages like JavaScript, CodeMash definitely makes an effort to be diverse in the content that they provide. I attended sessions discussing security, people skills, SQL, and the list goes on.

While not directly applicable to myself, the other thing I noticed is that CodeMash is incredibly family friendly. Not only was the conference held at an indoor waterpark, it's actually two conferences in one. The second name being KidzMash.

KidzMash is another conference that runs parallel to CodeMash, and provides sessions more geared towards.. you guessed it, kids! Topics included robotics, video game development, and even 3D printing. It made the whole experience feel very wholesome and welcoming.

To add onto that, I felt CodeMash had the most community involvement that I've ever seen in a conference. The conference had its own Slack channel where conference goers could ask questions as the conference was going on. You had people asking if anyone wanted to get together to play games, meetup to discuss another topic, etc. You never really felt alone or out of the loop.

New Faces

It was refreshing to see a lot of newer faces giving presenations in each of the talks. Along with being family focus'd, CodeMash almost seemed to be driven to encourage those who may not have spoken before, to get up on stage and give it a shot. A select number of the speakers at the conference have never spoken before, and the conference explicitly calls out that they like to have those individuals come and present.

For those who it is their first time, CodeMash even pairs them up with a couple mentors that can give feedback on their presentation and pass on their expertise from having been there, and done that.

Not only did you get to see some of the bigger names in the tech world (e.g. Jon Skeet!), you got to introduce yourself to some newer members of the community. These individuals bring a fresh perspective and have their own opinions which I feel disrupts the echo chamber that we may find ourselves lost in from time to time.

Capture the Flag

Speaking of new faces, I met quite a few individuals who were also a part of the CodeMash CTF. For those unfamiliar, CTF (Capture the Flag) is an event where you are given a scenario in which you must obtain the "flag" in order to score points. Scenarios may include: a zip file that has been intentionally corrupted and you must figure out how to get inside to unzip the flag, an image that has a hidden message somewhere inside of it, or even a website that has a web page somewhere on the server and you must do everything possible to find it.

It was one of the best experiences I had at CodeMash, and the people involved were equally enjoyable to converse with.

The organizer, Bill Sempf, also runs the Columbus OWASP group and is an expert lock picker! Luckily one of the "open spaces" at the conference was an introduction to lock picking, so I was able to get some hands on experience picking locks which was a great time.

But I digress.

For the CTF, there's plans for next year to have those who were specifically passionate about the it (this guy) to create our own problems, rather than using problems created by someone else. I'm overly excited to be make some CTF problems coming up here in the near future.

Wrapping Up

Taking everything into consideration, CodeMash is definitely worth it. At only 350$ a ticket, you get two full days of sessions, food, networking opportunities, and an after party. Contrast that with the typical pricing of conferences, it's worth at least giving it a try.

Great content, great speakers, in a great location. I will definitely be going back for CodeMash 2020.

It's close to 32 degrees outside. Wind whipping at your coat. No you're not on a boat. But where are you? Admittedly that's a question with one too many answers, but I bet you wouldn't guess Las Vegas. DEVintersection 2018 was again, held in Las Vegas, Nevada at the MGM grand. And it was cold. Too cold my friends.

On top of that, I'm honestly not much of a gambler, so I spent much of my free time hopping from hotel to hotel, checking out the sites. Indoors. No matter! It was still a fantastic conference, and the theme did slightly change from that of last year.

If you recall my previous post on DEVintersection 2017, the theme was Docker and Microservices. It was a challenge to go to a session and be exposed to one or the other, even if the topic seemed like it was not related at all. While Docker and leveraging microservices are still being talked about, and are definitely not going anywhere, the theme this year definitely did shift.

The Theme

It may not be incredibly surprising to hear that much of the conference was focused around Artificial Intelligence. It's gaining some traction in the tech community.

This year DEVintersection explicitly co-located with the Microsoft Azure + AI Conference. Now, Co-locating is pretty common for DEVintersection, it's why they call it an intersection.. so I guess I shouldn't have been so surprised going into the con that a large portion of it would have AI related content!

The AI keynote highlighted customers that used Microsoft cognitive services to build customer support bots, notably Tobi. They showcased multiple demonstrations where it was near impossible to tell the difference between a two humans interacting with one another and a human interacting with a bot. Going as so far as to have the bot verbally speak. AI is going to have a huge impact in the customer service industry, even moreso than it does today.

Though the highlight of the AI keynote had to be a whiteboard compiler. Now, when they initially announced what they were going to do, I wasn't quite sure at what they were going to be ultimately doing. What the hell is a whiteboard compiler?

In the end, they drew an Azure DevOps pipeline on a whiteboard, took a photo of it, and then it was interpreted by Azure cognitive services to be turned into a usable deployment pipeline. From a whiteboard! A whiteboard I tell you!

So Much Auth

My current area of focus, at work anyway, has to do with identity management. That is, user management within an enterprise including editing users, access to applications, the whole shebang. So unsurprisingly, I spent a large chunk of the conference attending workshops and sessions that closely aligned with identity and access management. Which for the most part means I spent the entire week with Brock Allen and the Solliance team, it's pretty much all they do.

The irony of it all was that after spending two days in a workshop and attending a couple sessions all relating to OAuth and Identity/Access Management, Solliance held a Security After Dark meet and greet to field questions. One of the party goers had asked an OAuth implementation question, and through a number of back and forths the final response was "specifications are suggestions".

Here I was, ready to go back to my office ready to lay down the "obey the spec" hammer, all to be brought back down to earth that it doesn't necessarily have to be that way.

Honestly though, it was good to hear. All to often I get stuck in this mindset of needing to find the perfect answer, the answer that has no equal and is obviously the way to go. Unfortunately in our field, such answers rarely exist. You can't just follow the spec word and word and produce a great product, or guarantee that it's going to be portable. Not everyone is going to follow the spec exactly, and the spec itself in some areas is pretty vague.

Moral of my entire Auth journey at DEVintersection would simply be, don't be afraid to diverge from the spec. At least a little bit.

Why So Fast?

For me, the conference honestly was avoiding the cold and learning the ins and outs of the OAuth spec as well as how to implement it. Though with that said, this time around the conference seemed to go much quicker than the previous year. Maybe the new hotness wore off, or I'm getting more accustomed to conference going. Nevertheless, I'll be back again next year.

We've all seen the joke, time and time again --

"There are only two hard things in Computer Science: off by one errors, cache invalidation, and naming things."

It's true. Naming things can be really hard. There seems to be all of these hidden rules around how we can name our properties and classes.

You: I'm just going to introduce this helper class, FileHelper.cs

The World: No! You can't use Helper in your class! That's just begging to violate the single responsibility principle.

But naming things correctly can save yourself and others a lot of time. I'll share with you an example from earlier this week. Consider this command:

webdriver-manager start

Assuming you had no knowledge of what this command did, I bet you have some guesses. Maybe you would have only one guess, and I'd honestly be Okay with that. You're probably thinking:

Well.. it starts the web driver manager..?

And you'd be right. Almost.

Unfortunately, webdriver-manager start also performs an update. Although the only way you'd be able to figure this out is if you read the documentation (which honestly seems a little buried to me) or if you ran into the same issue I ran into this week.

While not incredibly relevant to the story, if you want to learn more about what webdriver-manager is, you can read the projects npm page or the GitHub repository. The TL;DR is that it is a means to manage E2E tests for Angular projects.

It goes a little something like this...

For most of the things we develop at my company, we put them into Docker containers. This includes our end-to-end (E2E) tests. The Dockerfile for our tests is pretty straightforward, and honestly as to not add too many unnecessary details, the only thing that really matters is that we have the following command in the Dockerfile:

RUN webdriver-manager update

When the image is built, webdriver-manager update will be run, which will install the latest version of the webdriver manager. Docker images are immutable. That is, when they are created, they do not change. This means that the Docker image will be created with the latest version of the web-driver manager.

Now, to start the web-driver manager, we need to run the infamous webdriver-manager start command from within the Docker container.

Though depending on when you created your Docker image and when you started running your container, you're going to get one of two scenarios:

1. The container will start up just fine and run your tests as you expect.
2. The container will error when trying to run the webdriver-manager.

This is due to the fact that, unfortunately, webdriver-manager start not only starts, but attempts to start the latest version. Regardless of what version is installed. So it is possible that a new version has been released, and the Docker image is no longer relevant.

Luckily the solution isn't too bad. We just need to update the Dockerfile to update to a specific version. This forces our Docker image to always have version 3.0.0 installed.

RUN webdriver-manager update --versions.standalone 3.0.0

We then need to change the webdriver-manager start command to also include the same parameter:

webdriver-manager start --versions.standalone 3.0.0

Which in turn, forces the webdriver manager to start a specific version.

A simple solution, but a problem that took a decent time to figure out what was going wrong. Never would I have imagined that start did more than just that. Had the command been called startupdate or just left as start with an optional update parameter, the problem would've been much more apparent.

The biggest takeaway from all of this is that your naming should say exactly what it is doing and..

Have no side effects

Side effects are lies. Your function promises to do one thing, but it also does other hidden things. Sometimes it will have unexpected behavior. They are devious and damaging mistruths that often result in strange temporal couplings and order dependencies.              – Miguel Loureiro

Your code should do what it says it does, and nothing more.

As developers, we perform a lot of code reviews (or at least we should be). They are a great way to not only ensure that the code we intend to deploy to production won't have a negative impact on the system, but also a time to learn and better ourselves.

Unfortunately, I have noticed a trend, especially at work. We tend spend the majority of our cycles verifying whether or not the code conforms to standards. In other words, our first instinct is to scan the code for incorrectly placed braces, improperly named variables, etc. Rather than review the actual quality of the code.

There's a lot of great resources available out there on this subject, so I don't want to regurgitate what's already been published, but I did want to highlight a couple points that this article speaks to.

What a code review is:

• An opportunity to highlight what is being done well. Reaffirm that the developer is taking the correct approach to solving the problem.
• An opportunity to highlight what can be done better. Offer different solutions and approaches and use it as a chance to improve one's skill set.

What a code review is not:

• A witch hunt, a time to point out every little fault. Code reviews are a great opportunity to learn. Developers should be eager to have their code reviewed, not shy away from it for fear of belittlement (even if unintended).

How to conduct a code review:

First and foremost, understand the subject matter. Really familiarize yourself with the problem that the reviewee is trying to solve. Go to the developer's office and immerse yourself in the problem. You can't offer solution suggestions if you don't even know what the problem is.

Focus on how the problem was solved. Not how it's formatted. Think big picture. Is the change the right approach to solve the problem in the grand scheme of things?

Code reviews that just point out every little standard violation aren't beneficial to anyone. Use code reviews as an opportunity to knowledge share, and encourage learning. Not as a platform to demonstrate how well you can enforce formatting guidelines.

I would also encourage you to seek out reviews from other developers, especially if you're even the slightest bit unsure if your solution makes sense. Just because you have the deploy permission, doesn't mean you're stuck in a silo. Go forth and get a second opinion.

If you're looking for additional information, here's a blog series that I completely agree with:

• How to Do Code Reviews Like a Human (Part One)
• How to Do Code Reviews Like a Human (Part Two)

The problem lies within the approach that was proposed, going so far as to say: "We need to ensure that we have at least 80% test coverage."

While the intention is a good one, code coverage is unfortunately useless.

Now, that is a pretty bold statement, so let me clarify a little bit. Code coverage goals are useless. You shouldn't strive for X% coverage on a given codebase. There are a few reasons for this, so let me explain.

It is possible to test enough

Not all code bases are created equal. One could be for an application that sees millions of hits in a day and is grossly complicated. Another could be for a tiny application that services a couple users a day, if that. I always like to envision these different kinds of applications on a risk plane.

Imagine if you will that each dot is an application in our system. The further top-right we go, the more likely that if something were to go wrong it'd be some bad news bears. Whereas the further bottom-left.. eh? Maybe someone would notice.

Now, it would be a silly little to say that every application should have at least 80% code coverage. Why? Opportunity cost. While I am a huge proponent of testing, I don't like to test just because. We should aim to test enough. Test enough so that we have enough confidence that our application will function as we expect it to.

In reality, maybe for our right-winged applications, 80% isn't enough. Maybe that actually should be higher and we should not stop at 80%. On the flip side, our smaller applications in the bottom left probably don't need such a high coverage percentage. The cycles spent adding tests would potentially bring us little to no value and end up just being a waste of time.

_{Note: I feel like at this point some individuals may be a little confused as to how adding tests would be invaluable. There's a whole development methodology called TDD that creates a high level of coverage just by following the red, green, refactor cycle.The points I make here generally refer to going back and adding tests because someone dictated that the code bases coverage percentage was too low. If you're doing TDD to begin with, then setting a target really won't help. It's just a byproduct.}

It's all about context. We can't generalize a percentage of coverage in our code base, because each code base is different.

Fun Fact: Did you know this sort of risk plane chart can be applicable to many different scenarios? Ever wondered what the risk plane for the security guy looks like?

Anyway...

In the same vein, not everything needs a test around it. Let's say we wanted to introduce a new public member into our codebase, something simple

public FirstName { get; set; } 

Introducing this line of code, if not called in any of our tests will drop code coverage. Maybe even below our beloved 80%. The fix?

[Fact]
public void FirstName_ByDefault_CanBeSet()  
{
  var myClass = MyClass();
  myClass.FirstName = "testname";
  Assert.AreEqual("testname", myClass.FirstName)
}

At this point, we're just testing .NET -- something we definitely want to avoid. I tend to only put tests around code that I know could actually have the potential to change in a way that I do not want it to. Logical code.

Code coverage is easy

Just because we have a lot of code coverage, does not necessarily mean that we can have a lot of confidence that our application works as we expect it to. Everything is always more clear with examples, so let's consider the following:

public class Flawless  
{
  public bool IsGuarenteedToWork()
  {
    // some code
  }
}

Now, methods usually have logic that we would normally want to test, right? Conditionals, mathematical operations, you name it. Though, for our example, it doesn't matter! We just want to increase code coverage. That's our goal.

[Fact]
public void IsGuarenteedToWork_ByDefault_Works()  
{
  var flawless = new Flawless();

  var actual = flawless.IsGuarenteedToWork();
}

And there you have it! 100% code coverage. By default, tests that do not have an Assert will be considered passing. Now you're probably thinking.. oh come on, who would actually do this?

People do silly things when incentivized. My go-to example is that of a scenario in which a company tells QA that for every bug they find at the end of the quarter, they will be given a bonus. Seems pretty reasonable right? The flip side of that is the same company tells development that they will receive a bonus based on how few bugs they introduce into the system.

This scenario incentivizes the failure of opposing groups. The development organization doesn't really want to write any code for fear of introducing a bug and wants QA to miss bugs in their analysis. Whereas the QA group wants development to introduce bugs into the system so that they can find them and be rewarded for doing so.

The other thing that we need to keep in mind is that...

Code coverage context matters

Let's consider that our developer wasn't just trying to game the system, and actually put forth an honest effort to obtaining his code coverage goal. Our implementation could be something like the following:

public class Flawless  
{
  public bool IsGuarenteedToWork()
  {
    for(var x = 0; x < int.MaxValue; x++) 
    {
      // Man, this is gonna work. I'll find that solution.. eventually.
    }
  }
}

.. and let's not forget the test.

[Fact]
public void IsGuarenteedToWork_ByDefault_Works()  
{
  var flawless = new Flawless();

  var actual = flawless.IsGuarenteedToWork();

  Assert.True(actual);
}

I hope it was obvious that the example above is far from performant. But in this case, we've reached 100% code coverage and we're actually asserting that the code is working as we intend it to. The implementation works. The test is correct. Everyone is happy. Almost...

When it comes to testing, there are different stakeholders.

Stakeholders are people whose lives you touch - Mark McNeil

This can be broken down further into the types of stakeholders.

1. Primary Stakeholder (who I'm doing it for) _{Example: The customer who requested the feature.}
2. Secondary Stakeholder (others who are directly involved) _{Example: Your boss and/or other developers on the project.}
3. Indirect Stakeholder (those who are impacted otherwise) _{Example: The customers of your customer.}

As programmers, we are writing code to solve problems for other people (sometimes ourselves if we can find the time). The same section of code matters differently to different people. Person A only cares that the answer is correct. Maybe they're notified when it's ready, but they're pretty indifferent to when they receive it. Person B needs the answer soon after requesting it. Our test only completely satisfies Person A.

There can be a lot of stakeholders when it comes to writing code. Unfortunately, we can't say with confidence, even at 100% code coverage, that our code is going to be compatible with everyone's needs.

After all of the harping on why code coverage is useless as a target. I need to wrap up by saying...

Code coverage can actually be useful

I prefer to leverage code coverage as a metric. Coverage is something that we're aware of, something that we can use to make informed decisions about each codebase.

If we notice that one codebase is consistently dropping in coverage, we can take that as a sign to look a little deeper into what's going on. Is the codebase incredibly hard to test? Are the developers just not putting forth the effort to test, even when it makes sense? Maybe it's actually what we would expect from that code base, so everything is gravy.

Coverage can also just let us know if we're doing an adequate amount of testing. If a mission-critical application only has 10% coverage, we should investigate the reasons for that and potentially start a quality initiative and gets some tests strapped on. It allows us to prioritize our testing initiatives without just randomly picking a codebase and start throwing tests at it.

The entire point of all of this is that setting coverage targets will just be counterproductive to your goals. We should be aware of coverage so that we can make informed decisions, but not let it impact the quality of our code just for the sake of coverage attainment.