Adventures in IT

Translating a bit of the idea behind domain driven design into code architecture

noreply@blogger.com (Andrew Beak) — Thu, 22 Jul 2021 08:41:00 +0000

I've often participated in ~~arguments~~ discussions about whether thin models or thin controllers should be preferred.

The wisdom of a thin controller is that if you need to test your controller in isolation then you need to stub the dependencies of your request and response. It also violates the single responsibility principal because the controller could have multiple reasons to change.

Seemingly, the alternative is to settle on having fat models. This results in having domain logic right next to your persistence logic. If you ever want to change your persistence layer you're going to be in for a painful time. That's a bit of a cargo cult argument because honestly who does that, but it's also a violation of the single responsibility principal.

One way to decouple your domain logic from both persistence and controller is to use the "repository pattern".

Here we encapsulate domain logic into a data service. This layer deals exclusively with implementing your domain logic and is entirely reliant on using the repository layer to fetch data.

The repository layer takes care of sorting, aggregating, and filtering your data. It doesn't have intelligence beyond these basic functions. It is unable to make a call directly to the actual data store but rather uses an entity to do so. This keeps the mechanism of retrieving data from storage separate from the logic required to massage it into a shape that is useful to your data service.

In code I usually namespace the domain to keep it clear what I'm trying to do. So I'd end up with namespaces something like this:

\App\Domain\Service
\App\Domain\Repository
\App\Domain\Entity
\App\Domain\Contract

The "Contract" namespace is where I keep my interfaces. I personally like grouping them up like that, but your mileage may vary.

If you're looking for more concrete code examples, I blogged about this many years ago (here). I'm not sure that adding code to the explanation actually helps to improve understanding though. The principal of what we're trying to do is to decouple our domain logic from our persistence mechanism and our controller logic. We want to adhere to SOLID principals and have code that we can test and maintain without swearing.

Grokking PHP monolog context into Elastic

noreply@blogger.com (Andrew Beak) — Mon, 11 Nov 2019 17:34:00 +0000

An indexed and searchable centralized log is one of those tools that once you've had it you'll wonder how you managed without it.

I've experienced a couple of advantages to using a central log - debugging, monitoring performance, and catching unknown problems.

Debugging

Debugging becomes easier because instead of poking around grepping text logs on servers you're able to use a GUI to contrast and compare values between different time ranges.

A ticket will often include sparse information about the problem and observed error, but if you know more or less when a problem occurred then you can check the logs of all your systems at that time.

Problem behaviour in your application can occur as a result of the services you depend on. A database fault will produce errors in your application, for example.

If you log your database errors and your application errors in the same central platform then it's much more convenient to compare behaviour between the systems.

I've often found that knowing the application context makes it a lot easier to debug a problem. I can also more easily reason about the path my code would have taken. If I'm feeling energetic I can make an attempt to reproduce it and play it back in slow motion with my debugger.

Monitoring performance

Indexing your logs lets you use a query language to search them. All of the popular logging platforms include a DSL that lets you perform aggregations and other statistical functions on values contained in your logs.

This makes it a lot easier for your team to ask and answer questions about historical behaviour.

You can very easily ask "find me a list of all endpoints that take longer than 0.5 seconds to respond", or "find me all the requests that took longer than two standard deviations above mean to respond to".

Central logging doesn't enable you to answer questions, you could already do that manually. Central logging makes it easy to answer questions, which encourages you to ask more of them.

Being able to query patterns makes it easier to identify client use patterns that are harmful to your server. This lets you mitigate them on either the server or the client.

You could notice, for example, that a client is not respecting your cache control headers and is performing a repeat call to the API when it should be relying on a local cache. This lets you engage the client developers with alternatives and SDK's to help them take advantage of your caching strategy.

Managing verbosity and unknowns

Anomaly detection is a powerful addition to central logging. Logging is great for analyzing historical performance, but is too verbose to be useful for monitoring in any level of detail.

Machine learning algorithms can analyze your logs and find outliers in message frequency or values. They can do this quite cheaply in real time after an initial learning process. Your central logs can become an early warning system for problems that your customers haven't reported yet.

I'm a fan of Unomaly, but there are lots of other options out there, Splunk and Elastic both include machine learning options for example.

A PHP implementation

So now that you're convinced that central logging will improve your debugging, allow you to identify performance bottlenecks, and make auditing more convenient what is the next step in your PHP project?

Monolog is the de facto standard PHP logging library. Part of its fancy features include the ability to log a contextual array of information, which is provided as a second parameter, like this:

$log->info("Example of something being logged", $visitDetails);

This results in the array being output as a JSON string at the end of your log message and can be very useful indeed.

Some examples of the useful information you can put in the context could be a request id that lets you group log entries from any particular request. You'd assign a request a unique id when you first start processing it, and then consistently include this id in your logs.

Monolog lets you push logs straight into Elastic, but if we push our logs to a local logstash instance then we save ourselves some network I/O and allow for some buffering in the event that Elastic is down.

Logstash also lets you grok your logs and massage them into a form that Elastic can understand.

Actually writing the grok pattern is a lot like writing a regex pattern, but luckily there are some tools that will greatly speed this up for you.

Take a look at:

Oniguruma lets you use regex in the Grok pattern, like this: (?<field_name>the pattern here)

For my out of the box installation of Monolog the logstash configuration looked like this:

The aim here is to use grok to tokenize your log line into fields. You can then use the json filter to further expand any json fields that you've grokked.

In my example the field "context" is grokked from the context that can be added as "extra data". See the Monolog manual for details.

I was able to get Monolog to push logs to Logstash by using a SockerHandler, like this:

When you mix this up with AWS managed Elasticsearch (and their plugin) you can very quickly get up and running with Kibana.

Example repository is at https://github.com/andybeak/monolog-elk-demo

Streaming MPEG-TS to your Raspberry Pi

noreply@blogger.com (Andrew Beak) — Sat, 20 Oct 2018 11:55:00 +0000

Raspberry Penguin
CC by openclipart.org

Usually people focus on getting camera streams off their Pi, but I wanted to set up a test environment that would let me play with some encoding and distribution software.

For the client in this scenario I'm running a Raspberry Pi 3 Model B which I have plugged into an ethernet cable on my home network. I'm using my Ubuntu 18.04 laptop to serve up the multicast stream.

If you're new to multicast the first piece of reading you should do is to read about the reserved IP ranges on Wikipedia. You can't broadcast on just any old address, but you do have a very wide selection to choose from. I decided to broadcast on udp://239.255.0.1 and port 5000, because why not?

I downloaded a video from https://www.stockio.com/free-videos/ and saved it into a folder. From there all you need to do is run ffmpeg on the server, thusly:

ffmpeg -re -i costa.mp4 -bsf:v h264_mp4toannexb -acodec copy -vcodec copy -f mpegts udp://239.255.0.1:5000

There are actually some parameters in there that ffmpeg gave me in its error output when I tried to play the file. If you use the same file as me then you need to include them, otherwise you might not need to.

Once ffmpeg had started streaming the file I quickly opened up VLC on the Pi and tuned to the network stream that my laptop was streaming on. Hurray! Everything is working, my UDP multicast is up and running, now I can play with some more advanced features.

Here is a screenshot from my Pi:

Captured from Raspberry Pi, video from https://www.stockio.com/free-videos/

In case you don't want to have to dash over to your other window to see the stream before the clip ends you should know that ffmpeg is able generate test patterns for you. If you're like me and don't want to spend too much time reading the ffmpeg manual, I found another site that has a good little snippet that you can just copy/paste.

Using Azure Active directory as an OAuth2 provider for Django

noreply@blogger.com (Andrew Beak) — Sat, 29 Sep 2018 21:15:00 +0000

Azure Active Directory is a great product and is invaluable in the enterprise space. In this article we'll be setting it up to provide tokens for the OAuth2 client credentials grant. This authorization flow is useful when you want to authorize server-to-server communication that might not be on behalf of a user.

This diagram, by Microsoft, shows the client credentials grant flow.

From Microsoft documentation

The flow goes like this:

The client sends a request to Azure AD for a token
Azure AD verifies the attached authentication information and issues an access token
The client calls the API with the access token. The API server is able to verify the validity of the token and therefore the identity of the client.
The API responds to the client

Setting up Azure AD as an OAuth2 identity provider

The first step is to create applications in your AD for both your API server and the client. You can find step-by-step instructions on how to register the applications on the Microsoft page at Integrating applications with Azure Active Directory.

We need to generate a key for the client. From your AD blade in the Azure portal click on "app registrations" and then show all of them. Select the client application and then open it's settings. Create a new key and make sure you copy its value after saving.

Now let's add the permissions that our client will ask for. For this article we're just going to use a basic "access" scope. We're not going to add custom scopes to the server application, but this is easily possible by editing the manifest of the server application.

From the settings page of the client application click on "Required permissions". Click "Add" and in the search box type in the name of the server application that you registered in AD. Select it from the list that appears and choose "access" under the delegated permissions.

Now we have a fully fledged OAuth2 service set up and running, which is awesome. Let's call the Azure AD endpoint to get a token:

You can get the application id by opening up the registered application from your Azure AD portal blade.

Azure should respond with a JSON body that looks something like this:

The access_token value will be a JWT token. If you haven't used them before go check out jwt.io. This is the access token that you will present to your Django backend.

If you copy the access token into the decoder on jwt.io then you'll see that the payload looks like this:

Interlude

Let's take a breather and watch a music video. Because, really, 8 bits should be enough for anybody.

Okay, we're back and we're ready to get Django to authenticate the token.

Validating the token in Django

I'm not going to go into the details of setting up your Django project. There is already an excellent tutorial on their site. I'm going to assume that you have an app in your project to handle the API.

What we need to do is write some custom middleware for Django that will get the access token out of the "Authorization" HTTP header, validate the JWT, and check the signature against Azure's published signature. You can find out more about this on Microsoft's website, but this is actually an OAuth2 standard approach so just about any standards compliant documentation will do.

We'll be using standard libraries to decode the JWT, but in order to properly verify it we will need to have the key. We don't want to hardcode the key into our API because Azure rotates their keys on a regular basis. Luckily the OAuth2 standard provides a defined way for an OAuth2 provider to share this sort of information.

Azure publishes it's OAuth2 configuration file at https://login.microsoftonline.com/common/.well-known/openid-configuration. There is also a configuration file that is specific to your tenant at https://login.microsoftonline.com/{tenant}/.well-known/openid-configuration, but the general one is fine for us.

In that document there is a key called "jwks_url" whose value is a url. On that url you will find the keys that Azure is currently using. We need to retrieve that document and find the key that was used in our access token.

Right, so that's the plan. Decode the JWT to find what key was used to sign it and go fetch the key from Microsoft to verify the token. If anybody is able to trick our DNS and serve us content that we think is from Microsoft then they'll be able to pwn us, but otherwise as long as we trust the response from Microsoft we should be able to trust the JWT token the client gave us.

I placed this into settings.py because swapping Azure AD for any other standard implementation of OAuth2 should be possible by just swapping the configuration endpoint. I also set up Memcached while I was busy in the settings, so I ended up with this extra settings information:

Then I made a new file called checkaccesstoken.py and put it into the "middleware" directory of my API app, making sure that I registered it in settings.py

In the __call__ method of the middleware I call out to the functions that are going to check the token:

To get the token we use the Django request object:

We make sure that the token is a Bearer token by defining a class constant called AUTH_TYPE_PREFIX as having the value "Bearer".

Now that we have the access token we need to check it. You could use any JWT library, but I chose to use "pip install pyjwt cryptography". This JWT library will throw a DecodeError if the token is invalid, which we catch and respond with 401 to the user.

The public key is available over the network so we want to cache it to reduce the I/O time. Microsoft suggests that it is safe to cache for about a day.

Now we're done! We can send a bearer access token to our API and be authenticated. Note that we're not doing any authorization, all we're doing here is authenticating. We haven't tied this client application to any form of role in our application. That's for another day :)

References:

I placed the whole middleware file in a Gist
Microsoft article on client credentials flow
Microsoft advice on validating tokens

Standardizing infra - lxd is not Docker

noreply@blogger.com (Andrew Beak) — Thu, 06 Sep 2018 11:43:00 +0000

LXD logo from Ubuntu blog

Selling a software package that is deployed to a customer data-centre can be challenging due to the diversity found in the physical infrastructure.

It is expensive to make (and test) adjustments to the software that allow it to run in a non-standard manner.

Maintaining a swarm of snowflakes is not a scalable business practice. More deployment variations means more documentation and more pairs of hands needed to manage them. Uniformity and automation are what keep our prices competitive.

Opposing the engineering team's desire for uniformity is the practical need to fit our solution into the customers data-centre and budget. We can't define a uniform physical infrastructure that all of our customers must adhere to. Well, I suppose we could, but we would only have a very small number of customers who are willing and able to fit their data-centre around us.

We are therefore on the horns of a dilemma. Specifying standard physical hardware is impractical and limits the sales team. Allowing the sales team to adjust the infrastructure means that we have to tailor our platform to each deployment.

Virtualization is the rational response to this dilemma. We can have a standard virtualized infrastructure and work with the sales team and the customer to make sure that the hardware is able to support this. This way we'll know that our software is always deployed on the same platform.

LXD is a hypervisor for containers and is an expansion of LXC, the Linux container technology behind Docker. It's described by Stéphane Graber, an Ubuntu project engineer,as a "daemon exporting an authenticated representational state transfer application programming interface (REST API) both locally over a unix socket and over the network using https. There are then two clients for this daemon, one is an OpenStack plugin, the other a standalone command line tool."

It's not a replacement for Docker, even though Linux kernel provided containers are at the root of both technologies. So what is LXD for? An LXD container is an alternative to a virtual machine running in a traditional hypervisor.

With virtualization (or LXD containerization), if your customer has limited rack-space or a limited budget you can still sell your software based on a standard platform. You can take what metal is available and install LXD containers to partitition the resources up into separated "machines".

If you like, you can use Docker to manage the deployment of your software into these LXD containers. Docker and LXD are complementary!

In practical terms you can use tools like Ansible to automate the provisioning of your LXD containers on the customers metal. You are able to define in code the infrastructure and platform that your software runs on. And that means your engineering team wins at automation and your sales team wins at fitting the software into the customer data-centre.

Is blockchain "web scale"?

noreply@blogger.com (Andrew Beak) — Tue, 15 May 2018 08:04:00 +0000

For something to be truly awesome, it must be "web scale" right? The rather excellent video just below shows how hype can blind us to the real values of a technology. It's quite a famous trope in some circles and I think is a useful parallel to the excitement about blockchain.

In it an enthusiastic user of a new technology repeats marketing hype despite having no understanding of the technical concerns involved. Watch the video and you'll hear every piece of marketing spin that surrounds NoSQL databases.

I usually get very excited about new technologies but I'm quite underwhelmed by blockchain. In the commercial context I see it as a great solution that really just needs to find a problem to solve.

Lots of companies are looking for the problem that the blockchain solution will solve. Facebook has a blockchain group that is dedicated to studying blockchain and how it can be used in Facebook.

They don't seem to have found a particularly novel use case for it and appear to be set on using it to launch a Facebook crypto-currency. This would let Facebook track your financial transactions (https://cheddar.com/videos/facebook-plans-to-create-its-own-cryptocurrency) which to me sounds like juicy information to sell to advertisers.

I'm convinced that when we finally find that problem blockchain will be a godsend, but right now outside of cryptocurrency I'm struggling to see how the cost and risk of blockchain is worthwhile in the commercial context.

A publicly shared database

Blockchain offers a distributed ledger system, which sounds like something we want right? Everybody can look at the list of transactions at any time, and verify that a particular transaction is genuine.

Using boring traditional data services you'd be forced to have a database and expose an API that lets authenticated users access it in ways that you directly control. How dull is that!

Like communism, decentralizing control of data is a great idea because it promotes equality and encourages distributed effort focused on the greater good. Capitalists and other realists will point out that it's not machine learning algorithms that will be the oil of the future, it's data. Facebook isn't worth billions because it has clever algorithms, but rather because it controls data about people. Data has value, are you sure you want to give it away?

Immutable

Broadly speaking, instead of your company having a private database that it controls access to, you're able to have a shared database that you can only control the contents of if you spend more on hardware than the rest of the world.

Wait, what? Your blockchain consultant didn't explain to you that anybody with a botnet is going to be able to rewrite your public ledger?

Well it's true. The way that blockchain works means that the record is only immutable if there isn't an actor who controls more than 51% of the processing power in the network.

If you have a private blockchain you're going to need to be certain that you can throw enough resources at it to prevent malicious actors from rewriting history. Or you could only allow trusted actors to use your blockchain, which sounds very much like a traditional access method.

Who will mine your blocks?

The most popular use for blockchain (by far) is to provide a ledger for currency, like Bitcoin and the other coins out there. When users mine the blockchain they are rewarded with coins which ultimately they hope to be able to convert into fiat currency or material goods/services at some point. There is a direct incentive to spend money on electricity to crunch the numbers to verify Bitcoin transactions.

If you build your own private blockchain who is going to mine the blocks? What is the value in them doing so and are you going to end up mining the blocks yourself just to get the transactions processed? How is this cheaper than a traditional database?

Given the problems with immutability it should be pretty clear that a private blockchain is a pretty risky way to avoid boring traditional data-sharing approaches like an API. And of course that's assuming that there will be people wanting to mine your blocks.

Digital smart contracts will replace lawyers

Blockchain lets you dispense with traditional signed contracts with suppliers and rather enter into digitally signed contracts. They're touted to replace lawyers (https://blockgeeks.com/guides/smart-contracts/).

Smart contracts cut out the middle-man when it comes to transferring goods and services. They're an alternative to conveyancing costs, credit-card processing fees, escrow costs, and so on. Essentially you place the asset into the trust of the computer program which will then decide where to transfer it depending on whatever conditions you program into it.

Instead of placing your house into escrow with a lawyer who is bound by professional conduct rules and has an insured lawyer's trust fund you can use a smart contract written by anybody. That's the democratizing power of blockchain!

Who can forget about the Ethereum fork which happened because of faulty code in a smart contract? I'm not nearly arrogant enough to assume that I can code tighter code than the guys and girls who created the DAO - are you willing to bet your house that you are?

I am horribly unfair

Maybe it's not fair to use a high value asset as an example for considering a smart chain. What about other use cases touted by blockchain evangelists?

Blockchain supposedly streamlines business decisions by eliminating back-and-forth decision making. A smart contract can simply have the business rules coded into it so instead of seeking approval from humans you can just rely on the contract to grant your request.

For example, if you need to make a business trip it can happen automatically just so long as the coded requirements are met. As long as the contract is coded to be aware of every factor that goes into the decision it can automatically approve your travel request.

That contract won't work for requesting a new monitor for your workstation though. You'll need a different contract for that. Or maybe you can extend the old contract and just add more rules to it?

Given the rate cards for blockchain developers very simple administrative decisions can end up taking a lot more coding effort (and money) than you really need to be dedicating to simple decisions. And what happens if a management decision needs judgment that hasn't been coded?

Banks are already using smart contracts

Indeed they are (https://www.barclayscorporate.com/insight-and-research/technology-and-digital-innovation/what-does-blockchain-do.html).

Barclays sees a future where identity theft is impossible because digital identity is immutable and publicly accessible for them to read and share with law enforcement. My financial records would be available to Barclays (and whoever else can read the blockchain) and they'd be able to make a decision about opening an account quicker than the time it currently takes (about an hour the last time I did it at the branch).

I wouldn't need to take my photo identity and proof of address documents to the bank, I would just need to show that I own the private key associated with the digital identity profile. This will prevent identity theft, according to Barclays, presumably because consumer computers are secure and digital keys can't be stolen.

Trade documents are given as a good example of how digital signing and identity can be accomplished with blockchain.

In blockchain how do you establish identity? The digital identity is established by ownership of a private key, but how do you link that to a physical entity? Surely you need some way to link the digital identity with the physical entity before you ship the goods that your smart contract says have been paid for.

How do I know that wallet address 0xaa8e0d3ded810704c4d8bc879a650aad50f36bc5 is actually Acme Inc trading out of London and not Pirates Ltd trading out of Antigua? Who is responsible for authenticating the digital identities in blockchain?

You can trust the blockchain (as long as the hash power is evenly distributed) but can you trust the digital identities on it?

Digital signing and identity can also be managed through public key cryptography where a recognized and trusted central authority signs keys after verifying the owners identity. This isn't a new arena and blockchain doesn't solve the problems that public key cryptography has.

There are already established digital signing solutions that don't rely on snake-oil. I signed my rent agreement digitally in the UK with my landlords who live in continental Europe. I hardly think that this space is a raison d'être for block chain.

Public, not private blockchains

It seems that my beef is with the impractical nature of using private blockchain where existing solutions are more secure and cheaper. So, what about public blockchains, like Ethereum?

In a traditional blockchain each node on the network needs a full copy of the entire chain in order to be able to verify transactions. Without this public scrutiny the blockchain is no longer secure.

The problem with this is that Ethereum can only process very limited amounts of transactions per seconds. Currently it runs at about 45 transactions per second, which isn't an awful lot when you share it out amongst your company, and all the people speculatively trading Ethereum.

Ethereum is considering a sharding approach where they will decentralize the chain slightly in order to improve transaction speed. A few nodes on the network will have more authority than others. These nodes will need to have explicit trust in each other, and obviously the network will need to follow suit.

As a company do you want to commit to this level of trust? Who are the actors controlling these nodes? Who will they be in three years time? What countries laws are they bound by?

Data you put into the blockchain will be shared with everybody, forever. Governments don't like competition when it comes to spying on people and are passing increasingly strict privacy laws - how will you plan for compliance when you don't control your data?

Blockchain is web-scale!

Component cohesion

noreply@blogger.com (Andrew Beak) — Thu, 28 Dec 2017 12:29:00 +0000

Image: Pixabay

Breaking your application down into components can be a useful approach to a "divide and conquer" methodology. Assigning specific behaviour to a component and then defining interfaces for other components to access it allows you to develop a service driven architecture.

I'm in the process of decomposing a monolithic application into services that will eventually become standalone micro-services. Part of the task ahead lies in determining the service boundaries, which are analogous to software components for my micro-service application.

I want components to be modular to allow them to be developed and deployed as independently as possible. I'm using the approach suggested by Eric Evans in his book on domain driven design where he describes the concept of "bounded contexts". I like to think of a bounded context as being for domain models as a namespace is for classes. These contexts are spaces where a domain model defined in the Ubiquitous Domain Language will have a precise and consistent meaning. Keeping components modular helps to define and maintain these boundaries.

I want my components to be cohesive because I want my architecture to be so simple that people wonder why we need an architect at all. It should be intuitively obvious why a group of classes belong together in a component and what part of my domain logic they're implementing. Cohesion is a good thing and we're all familiar with writing cohesive classes, but what principals are important to consider when looking at grouping up classes into cohesive components?

Robert C Martin discusses three important concepts that govern component cohesion on his website (here)

Release-Reuse equivalency principle (REP) - the granule of release is the granule of reuse
Common Closure principle (CCP) - classes that change together are packaged together
Common Reuse principle (CRP) - Classes that are used together are packaged together

The Release-Reuse equivalence principle (REP) is very simple. It states that classes that are packaged together into a component need to be able to be released together. In practice this boils down to properly versioning your releases and having all of the classes in your component versioned and released together.

The Common Closure principle (CCP) states that you should gather together classes that change for the same reasons and the same times. Conversely you should separate out classes that change for different reasons and at different times.

Remember that the S of SOLID stands for "single responsibility principle" (SRP) where a class should have only one reason to change? The CCP is for components what the SRP is for classes.

We can say that generally stuff that changes together should be kept together.

The Common Reuse principle (CRP) states that you should not force users of a component to depend on things they don't need.

The CRP more strongly suggests that we do not include classes into a component that are not tightly bound to the function of the component. Every time we touch one of those classes we will be forced to retest all of the client applications that are using the component. Our deployment process will be heavier than it needs to be, and crucially we'll be deploying more than we have to.

The CRP is a more general form of the interface segregation principle but suggests that a component should be built from classes that are commonly used together.

Generally speaking, we should avoid depending on things that we don't need.

We've seen three principles that govern how we group up classes into components. The REP and CCP are inclusive about grouping up classes and suggest what classes do belong together. The CRP is more strong about excluding classes from a component. There is therefore a balance to be walked between these principals.

Tim Ottinger suggested a diagram that helps to see the cost of abandoning a principle. The label on an edge is the cost of weakening adherence to the principle on the opposite vertex. So, for example the cost of abandoning CCP is that we have too many components changing at one time.

Diagram suggested by Tim Ottinger illustrating tension between component cohesion principles

Your application will fall somewhere within this triangle as you balance your focus between the principles.

This balance is dynamic and changes over time. Robert C Martin notes that “A good architect finds a position in that tension triangle that meets the _current_ concerns of the development team, but is also aware that those concerns will change over time.”

These principles will govern how I examine my monolith and identify classes that I can group together to form components.

Writing SOLID Laravel code

noreply@blogger.com (Andrew Beak) — Wed, 27 Dec 2017 11:43:00 +0000

Image: Pixabay

SOLID is a mnemonic acronym for five object-oriented design principals that are intended to make software designs more understandable (see Wikipedia). They were promoted by a chap called Robert C Martin who has been programming since before I was born and is an authority on writing clean code.

Laravel is a PHP framework that implements the model-view-controller (MVC) pattern. A lot of people think that their responsibility for OOP design ends with adopting a framework, but actually Laravel is relatively un-opinionated on your OOP design and you still need to think about writing code that is testable and maintainable.

The reason that SOLID principals matter becomes apparent when you work on a single project for a long time. If you're writing throwaway applications for clients that you never expect to work on again (presumably because the client won't hire you again) then the quality of your code doesn't matter. But if you're the guy stuck with managing and implementing change in an application that is actively growing and improving then you're going to want code that is easy to change, easy to test, and easy to deploy.

The most common problem I've seen in Laravel is "fat controllers" that use the ORM to get some data and then push it through to a view. Let's take a look at an example I've made. Imagine that we're writing a payroll program. We might write something like the following controller methods:

This is an unfortunately common Laravel pattern that is taught in countless tutorials. We call the model from the controller, format the data, and then pass it on to the view. This is the easiest way to teach the MVC pattern to beginners but unfortunately it violates the SOLID principals. Let's see why, and how we can improve this code.

The "S" in SOLID stands for single responsibility principal which requires that each module or class should have a single responsibility for the functionality of the application. A more subtle understanding is put forward by Robert C Martin who says that "A class should have only one reason to change".

The thinking behind limiting the reasons for changing a class comes from the observation that software is often developed by teams and that often a team is implementing a feature for a particular actor. In our example the CEO of the company will have different requirements from the CFO, and when either of them requests a change then we want to limit the impact of that change. The actor is the reason for software to change - they request a feature and a team goes ahead and implements it.

In our controller above if the CEO requested a change then that change would definitely affect the CFO. The teams working on the code would need to merge in code from each other. If our code was properly designed then the controller class would be responsible to just one actor.

In this example I've moved the responsibility for calculating the employee pay to its own object. This object will only change if the CFO requests a change to the way that wages are calculated and so adheres to the single responsibility principal. We would similarly have an object that is responsible for counting the hours. I've chosen this way of solving the problem because the Facade pattern is very loaded in Laravel and I think it would just muddy the waters to use it here.

Let's move on to "O" which is the open-closed principal which requires that "A software artifact should be open for extension but closed for modification". It was developed by Betrand Meyer and holds that you should be able to extend on a modules functionality without having to change that module.

The aim of the OCP is to protect important code that implements high level policies from changes made to code that implements low-level details. We want the part of our code that is central to our application to be insulated from changes in other parts of the application.

There is some level of separation in our Laravel application. We can make a change to the View without there being any impact on the Controller, but within the controller above we have no such insulation. If we make a change to the way we read the database then we will be affecting exactly the same function that is responsible for calculating wages!

The open-closed principal seeks to prevent you from changing core functionality as a side-effect of adding new functionality to your application. It works by separating out the application into a hierarchy of dependencies. You can extend functionality from the lower levels of the hierarchy without changing the code in the higher levels.

The "L" in SOLID is named for Barbara Liskov who put forward what is now known as the Liskov substitution principal. The principal holds that "if S is a subtype of T, then objects of type T in a program may be replaced with objects of type S without altering any of the desirable properties of that program".

In the example above I've amended the object to make it inherit from an interface. Both the PermanentEmployeePayCalculator class and the TemporaryEmployeePayCalculator implement this interface and can be substituted for each other. This makes a lot more sense if you consider an LSP violation, such as this one:

This violates the Liskov Substitution Principal because the methods have got different signatures. You cannot substitute the subtypes of PayCalculator each other because they're incompatible. The object that depended on them would need to implement some logic to be able to know how many parameters to pass to the method. Adhering to the Liskov substitution principal removes this need and removes special cases from your code.

Adhering to the Liskov substitution principal

The "I" in SOLID was proposed by Robert C Martin and stands for interface segregation. The idea is that code should not be made to depend on methods that it does not use. By reducing the dependencies between classes you help to decouple your code making it easier to make changes in one section without impacting on others.

Let's imagine that we separated out our controller into classes like the below diagram. We have an Employee data object that is responsible for interacting with the persistence layer and returning results. It has a method that the PayCalculator object uses to determine whether the employee needs to earn their overtime rate and a method that both objects use to fetch the list of hours that an employee has worked (which may or may not violate the single responsibility principal).

Violation of the ISP

The problem here is that the HoursReporter is forced to depend on the isHourOvertime() function. This introduces an additional coupling between the classes that we need to avoid if we want to adhere to the interface segregation principal.

Adhering to the ISP

We can easily solve this problem by declaring an interface for the classes to depend on. The interface for the HoursReporter class excludes the function that we do not want to depend on.

The last letter in SOLID is "D" which stands for the dependency inversion principal which holds that the most flexible modules are those in which source code dependencies refer only to abstractions rather than concretions.

To understand dependency inversion consider two things: flow of control and source code dependency. We want to be able to have our source code dependencies to be independent of how control flows through our application.

Some classes and files in our application are more prone to change than others. They are "volatile" classes. We want to minimise the effect of the changes in these classes to the more stable classes. Ideally we want our business logic to be very stable and highly insulated from changes elsewhere in our system.

In the diagram below I'm illustrating a source code dependency hierarchy. High level classes call functions in lower level classes, but in order to do so they need to depend on that class. This means that your source code dependencies are unavoidably tied to how your flow of control works.

Source code dependency hierarchy

The problem that arises from this dependency is that it becomes difficult to swap functionality. A change to the source code of a low level object means that we need to rebuild all of the files that depend on it; Admittedly rebuilding files in PHP is less of an issue than for statically typed languages that are built in advance, but we are still directly impacting on files other than the one we are touching.

Let's say, for example, we had a class that outputs the Employee wages to screen. In the diagram above we would see the Employee object as the High Level object and perhaps a "ScreenOutput" object as a low-level object. Our Employee object calls the ScreenOutput class directly, and so we have to mention the source code in Employee, like this:

Now our CFO asks us to be able to print out the wages using the black and white printer in her office. Uh-oh, now we need to rewrite our source-code dependency because the "use" statement refers specifically to a concretion.

What happens if we want to make a change to the way that wages are displayed on the screen? We can easily tweak the ScreenOutput object, but can we deploy it separately? What impact is it going to have on all the places that depend on it?

How could we fix this problem and allow ourselves to swap functionality in and out without affecting our source code dependencies? How do we actually decouple these objects?

The answer is to always depend on abstractions rather than concretions. This insulates you from changes in the underlying files and lets you change and deploy parts of your application separately.

Using an interface to implement dependency inversion

In the diagram above the Employee object is calling the ScreenOutput class method through an interface. The class has a source code dependency on the interface file (which shouldn't change often) and any code change in the ScreenOutput class will not affect the Employee object.

The rules to follow for the dependency inversion principal are:

Do not reference volatile concrete classes
Do not derive from volatile concrete classes
Do not override concrete functions
Never mention the name of anything concrete and volatile

One way that you can accomplish this is through using a Factory to instantiate volatile concrete classes. This removes the requirement to have a source code dependency on the class that you're instantiating in the object where you need it.

Laravel approaches dependency inversion by using a "service container". Your code no longer depends on a concrete implementation of a class, but rather requests an instance of an object from the IoC container.

In our controller code above the IoC container returns an instance of the HoursWorked model through the Facade pattern. The controller is not directly dependent on the source code file of the HoursWorked model. So in this particular case we're just lucky to be adhering to a SOLID principal!

How to get Virtualbox VMs to talk to each other

noreply@blogger.com (Andrew Beak) — Fri, 23 Jun 2017 12:30:00 +0000

I'm busy writing an Ansible script and want to test it locally before trying to deploy it anywhere. The easiest way to try and make my local environment as close to my deployment environment was to set up a network of Virtualbox VMs.

The problem was that I've always configured my VM's to use NAT networking. I ssh onto them by setting port forwarding and have never really needed them to have their own address.

The solution to this problem is pretty simple. Just stop the machines and add a new network adapter of type "Host Only". This adapter will handle communication between the guest and host machines.

The trick is that you need to configure the guest OS network interface too.

To do this SSH onto your VM and run "ip add" to list your network adapters. If you're like me and started with NAT before adding "Host Only" as your second adapter the output should look something like this:

You need to identify the adapter that is your "Host Only" network. You can do this by running "ip add" on your host machine and looking for the vboxnet0 network address (assuming you're using the defaults given to you by Virtualbox).

Now you need to edit /etc/network/interfaces and tell Linux (I'm using Ubuntu 16.04) to set up that interface. Add lines like this snippet to your file:

Now your virtual machines will have an ip address (you can grab it with ifconfig) that you can set up in your Ansible inventory.

Is PHP a good fit for an API server?

noreply@blogger.com (Andrew Beak) — Thu, 13 Apr 2017 13:26:00 +0000

Image: Pixabay

Calling PHP a double-claw hammer is a bit of an in-joke in the PHP community. A lot of people bemoan PHP as a language – it's fashionable to do so and it seems to be a way to look clever. The joke came about from a blog post where somebody pointed out all of the problems with PHP (here's a rebuttal - https://blog.codinghorror.com/the-php-singularity/ )

Anyway, PHP is a warty language that sucks in academic circles but it doesn't matter because it's really good at web stuff, there are lots of people who know it (so it's cheap to hire), there are lots of libraries and frameworks (so it's cheap and fast to develop in). The commercial world is willing to overlook the academic warts.

I'm busy helping to improve the performance of an API server. As part of my effort I'm profiling calls to the endpoints. I'm using Blackfire to generate call graphs and also logging the sql queries that the ORM is producing so that I can check the indexes and joins.

Here's a callgraph for a call to the endpoint where we are looking to run a paginated SQL query. We're not applying any business logic or having any side-effects - all we're trying to do is query the database and return a JSON string to the frontend.

Blackfire call graph

That's a pretty substantial call graph for what sounds like a simple task right? All I want to do is route the request to a controller, query the database, and send the results back.

Blackfire tells me that 172 different MySQL queries are being run. The PHP code responsible is using the ORM to build up the joins and so on. I suspect that the problem is that there is pagination being applied and the ORM is not able to optimize the queries it needs to do in order to paginate efficiently.

Okay, so what questions do I have?

Why are we not querying the database more directly? I appreciate that developer productivity is a good reason to use ORM but is it a good reason in this case? 172 queries is an awful lot, especially when a lot of them are related to querying the schema so our ORM can run.

Why on earth does PHP have to spend so much time in disk I/O reading all of those source files when really what we need is request routing, a database query, and a response handler?

Blackfire reports that 304kb of data was transmitted across the wire for this. That seems like a lot of data for the five or six records that I'm returning to the frontend.

The call graph is frustrating – I'm lumbered with a whole lot of black box code and I have no control over the SQL that is being run. How can I improve the performance of this transaction?

So is PHP the best tool for this job?

I have previously had intractable problems with PHP when it comes to memory management. It's pretty complicated and it differs depending on the way that PHP is run but I do not have 100% confidence in PHP's garbage collection.

Circular object references (which I encountered while using an ORM where a model referenced itself as a parent to form a hierarchy) cannot be completely collected by PHP. PHP actually relies on the container the machine runs in to collect this memory.

PHP is not built for being a long-running program. It was never designed for this and it should never be used for this. It was built to handle a request for a page and then terminate.

The application is bootstrapped for every request. How much overhead does this add? Well there's a question that Blackfire is raising for me. Take a look at the timeline for the transaction from before:

Blackfire call timeline

The timeline shows when a PHP function was called in relation to the time taken to generate the response.

My controller function starts at around 750ms into the transaction. The actual time is irrelevant as a benchmark, but the fact that the first time *my* code runs is half-way into the transaction is what is relevant.

Until halfway into my application I've been waiting for PHP to bootstrap my application. You could argue that this is because of the PHP framework I'm using, but actually it is the limitation of PHP not being able to maintain state that requires us to continuously bootstrap the application.

Bootstrapping our application might involve disk I/O (depending on OpCache). It definitely involves network I/O because we have to connect to MySQL and wait for it to authenticate us. I know that there are ways to improve this, like by not using a framework and by tuning OpCache to improve compile time.

I'm concerned about what will happen when the application has 50,000 concurrent users. How much of a strain will it place on my database server to be constantly connecting (and authenticating)?

I think PHP is brilliant at web pages and not so good at being a long-running application that is capable of reusing resources. I'm a huge PHP fan but as an architect I do not want it to be my only tool.

I'm busy learning Elixir and the Phoenix framework (again with the frameworks!) response in microseconds (not milliseconds). I don't think we should using PHP like the hammer we use for everything.

Why is it important to understand how PHP matches a catch block?

noreply@blogger.com (Andrew Beak) — Tue, 21 Mar 2017 12:41:00 +0000

Image: Pixabay

PHP allows you to have multiple catch blocks. The manual says that when an exception occurs PHP will try to find the first matching exception block, but it isn't entirely clear on exactly how PHP matches "catch" blocks when handling exceptions.

Why is it important to understand how PHP matches a catch block?

It really relates to the fact that you are able to extend the base "Exception" class in PHP to make your own classes. Throwing exceptions that are specific is good practice because it allows you to write catch blocks that are focused on solving just one sort of problem at a time. The SPL exceptions are a good example of hierarchical exception inheritance in case you're wanting to see some practical usage.

In PHP a catch block is a match for an exception if the exception class is either the same as or inherits from the class in the catch clause.

Here's a trivial example:

In this example I've set up an exception class that extends the base exception class. I'm throwing an instance of this child exception. If I didn't know that PHP will match it's parent class I would expect the message "Child exception" to be output.

In actual fact if you run this code (here) you will see that the message "Parent exception" is output because PHP has matched MyException with its parent class.

The PHP manual doesn't make this explicit or obvious when discussing catch blocks or exception inheritance but it's one of those things that's always handy to know.

This implies that when we're coding multiple catch blocks we should place the more general parent classes at the bottom of the list. This prevents them from catching exceptions that are meant for their children.

I discuss this and other gotchas in my study guide for the Zend certification exam.

Complying with PCI database requirements in Laravel on AWS

noreply@blogger.com (Andrew Beak) — Mon, 23 Jan 2017 16:03:00 +0000

Image: pexels.com

I'm busy with the self assessment questionnaire for PCI compliance. Part of the database requirements are that cardholder data are encrypted at rest as well as in transit.

I host with Amazon RDS and use Laravel so my life is made pretty easy.

Amazon RDS natively supports encrypted connections and also lets you create a database that is stored on an encrypted backing instance. If you've enabled this option then all that you need to do is make sure that you connect to the database using an encrypted connection.

I'm not getting paid anything for saying that I really enjoy using RDS, but today is another occasion when I'm really happy that I didn't have to sit and install certificates and fiddle with a cluster configuration to enable SSL connections. The "zero config" that comes with RDS saves time and money.

Laravel was really easy to configure to use SSL. All that you need to do is download the RDS certificate chain from https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem and amend your database configuration to use it when connecting.

I saved the pem file into storage/app/mysql_ssl_certificate and then amended my app/database.php to look like this:

Now I can use a command like var_dump(DB::select(DB::raw("SHOW STATUS LIKE 'SSL_CIPHER'"))); to check that I'm connected with SSL.

I also decided that although PCI compliance doesn't actually demand that you encrypt the data in the database I would rather take the performance hit and protect my user details. I don't use them terribly often so the overhead of decrypting them when I do seems worth the effort to avoid the reputational damage my company would suffer if it leaked user details.

Again this is very easy to accomplish with Laravel. I just implemented a trait which I can selectively apply to any of my entities that I want to protect. So now if the application is breached and nasty people run off with my database at least they'll need to have the application key if they hope to read my users details.

Email injection attacks in PHP

noreply@blogger.com (Andrew Beak) — Fri, 20 Jan 2017 15:05:00 +0000

Image: Pixabay

Email injection is one of the topics I cover in my Zend certification guide. You can grab a copy on Leanpub - https://leanpub.com/phpforprogrammers/

It is possible for a user to supply hexadecimal control characters that allow them to change the message body or recipient list.

For example, if your form allows the person to enter their email address as a “from” field for the email then the following string will cause additional recipients to be included as cc and blind carbon copy recipients of the message:

sender@example.com%0ACc:target@email.com%0ABcc:anotherperson@emailexample.com,stranger@shouldhavefiltered.com

It is also possible for the attacker to provide their own body, and even to change the MIME type of the message being sent. This means that your form could be used by spammers to send mail from.

You can protect against this in a couple of ways.

Make sure that you properly filter input that you use when sending mails. The `filter_var()` function provides a number of flags that you can use to make sure that your input data conforms to a desired pattern.

<?php
$from = $_POST["sender"];
$from = filter_var($from, FILTER_SANITIZE_EMAIL);
// send the email

You could also install and use the Suhosin PHP extension. It provides the `suhosin.mail.protect` directive that will guard against this.

You could implement a tarpit to slow bots down or trap them indefinitely. Take a look at msigley/PHP-HTTP-Tarpit on Github as an example of a tarpit.

Just be careful that when setting up your mail server you must make sure it is not configured as an open relay that allows anybody on the Internet to use it to send mail. You should also consider closing port 25 (SMTP) on your firewall so that outside hosts are unable to reach your server.

Traits in PHP

noreply@blogger.com (Andrew Beak) — Thu, 12 Jan 2017 11:14:00 +0000

What are traits?

Image: Pixabay

Let's start by understanding what traits are and how they're useful. We'll move on to code examples straight after that.

Traits are not unique to PHP and are available in other languages too. They provide a way to extend the functionality of a class. A trait will have methods to implement this functionality and make these available as if they had been defined in the class itself.

In other words traits are flattened into a class and it doesn’t matter if a method is defined in the trait or in the class that uses the trait. You could copy and paste the code from the trait into the class and it would be used in the same manner.

The code that is included into a trait is intended to encapsulate reusable properties and methods that

can be applied to multiple classes. Traits group functionality in a fine-grained and consistent way and allow you to reuse this functionality without requiring inheritance.

I mentioned before in the lecture on inheritance that PHP does not support inheriting from multiple classes. Traits offer a way to provide similar functionality and offer a way to avoid the so-called "diamond problem" that is introduced with multiple inheritance.

Defining and using traits

Traits are very easy to use and there are lots of places that they can be useful. I'm going to run through an example that defines a trait that implements singleton functionality.

We define the trait using the `trait` keyword followed by the name we want to use. We can declare variables in the trait as well as functions.

If you look at the code example you'll see that my Singleton trait has got all of the implementation requirements for the singleton pattern.

I've defined a private static variable called $instance in the trait. Although it's private in the trait remember that the code gets flattened when a class uses the trait and so the class would be able to access it. If that's not clear just imagine copying and pasting the code in the trait straight into your class.

The trait also includes a static function called getInstance that returns a singleton instance of the class.

I'm using the trait in the class called MyClass. The `use` keyword indicates that we're using the trait. You might already be familiar with the `use` keyword as a way to import namespaces. When we import a namespace the `use` keyword is outside of the class, if it's inside the class it indicates that we're using a trait.

Now we're able to call the `getInstance()` static method on the class. This method is defined in the trait but we're using the trait in the class so it's available to be used as part of the class.

We're able to overwrite the constructor method in the trait. In this example I've set the constructor to private so that you are not able to instantiate the class but rather have to use the singleton method to get an instance. Trying to create a new instance of MyClass generates a fatal error because `__construct()` is public.

Now we're able to easily give a class all of the functionality it needs to be turned into a singleton without interfering with the inheritance hierarchy.

Namespacing Traits

So far we've seen how to declare and use a trait. What happens if we want to use a name for our trait that has already been using elsewhere in our code? It might happen for example that one of the libraries we use has already defined a trait named Singleton.

Traits, just like classes, can be namespaced. The name of a trait must be unique within its namespace otherwise PHP will throw a fatal error.

Just as with classes if you're wanting to use a trait in a namespace other than your own then you'll need to provide a full reference to it.

Inheritance and Method precedence

Traits may not inherit from other traits or classes but you can nest one trait inside another.

Okay so inheritance is simple, but what about situations where a class and trait have a method with the same name? Which method will take precedence and be run when we try to call it?

Methods declared in a class using a trait take precedence over methods declared in the trait. However, methods in a trait will override methods inherited by a class.

Expressed more simply, precedence in traits and classes is as follows:

CLASS METHODS first, then TRAIT METHODS, and lastly INHERITED METHODS.

Conflict resolution in traits

What happens if your class is using two traits and they both declare a method with the same name?

PHP requires you to explicitly resolve these sort of conflicts and will throw a fatal error if you try to use two traits that each define a method of the same name.

There are two approaches to resolving name conflicts between traits. You can either tell PHP not to use a method that is defined in one of the traits or you can tell PHP to use an alias for a method in one of the traits. Put shorter, you can exclude a method or alias a method.

In order to exclude a method you use the `insteadof` operator which tells PHP to use a method from one trait instead of the method from the other.

In order to alias a method you use the `as` operator to provide an alias to use. The only trick here is that it's not enough to just specify an alias, you must *also* exclude the original method from being used.

Right, so we've got that we can either exclude or alias a method to resolve naming conflicts between traits. Lets look at some code to see how it works.

First we're going to create two traits. We're going to define two methods in each trait and name them such that they'll create naming conflicts.

We're going to use the traits in a class which we're calling DomesticPet. Because the methods in the traits have conflicting names we're going to have to resolve these conflicts.

We resolve the conflicts in a code block that follows the `use` keyword. We have to resolve every conflict so lets step through this now.

Replacing methods with "insteadof"

For the first method, `makeNoise()`, we're going to simply exclude the method from the Dog class. We would do this if we only ever want to use the Cat method and expect never to use the Dog method.

In order to do this we're going to tell the class to use the makeNoise function declared in the Cat trait instead of the one in the Dog trait.

We use the `insteadof` keyword to tell PHP to use the method declared in the Cat trait instead of the one in the dog class.

Whenever the `makeNoise()` function is called PHP will know that you mean the one defined in the Cat trait and you won't be able to access the one from the Dog trait.

Aliasing methods with "as"

For the second method, `wantWalkies()`, we're going to rather alias the method in Cat. We would do this when we want to be able to use both methods in our class.

You may have already noticed in the code sample that there are three statements in the conflict resolution block but only two functions that need to be resolved.

Remember that earlier I said the trick with aliasing methods is that it's not enough to alias by itself and that you must also exclude the method you're aliasing?

We're going to need two statements to alias our method.

First we're going to tell PHP to give the method an alias that we can use to reference it. Then we're going to tell PHP to exclude the original method.

In order to alias the method we use the `as` keyword. We're going to give the `wantWalkies()` method an alias of `kittyWalk()`. By doing this you will be able to call the `kittyWalk()` function in your class. This isn't a new function, it's just an alias for the function in the Cat trait.

Right after we alias the function we go ahead and tell PHP to use the Dog `wantWalkies()` method instead of the Cat trait. If we left out this step we would still have a naming conflict.

The end result

We used the Cat `makeNoise()` function instead of the Dog one. When we call the `makeNoise()` method on the object we are referencing the cat function and so the output is "Purr".

We aliased the Cat `wantWalkies()` method as `kittyWalk()` and so when we call the `kittyWalk()` method on the object we are referencing the aliased Cat method and the output is "No Thanks".

Because we are aliasing the Cat `wantWalkies` method we also had to use the Dog method instead of the Cat method. Therefore when we call the `wantWalkies` method on the object we are referencing the Dog method and the output is "Yes Please".

Visibility

You might want to change the visibility of a method that you're using from a trait. This is done by using the `as` keyword.

Lets look at a simple example before we take a look at the complications.

We can set an aliased method as protected by inserting the keyword `protected` between `as` and the name we're aliasing it as. For example we can say,

Dog::wantWalkies as protected doggyWalk;

Changing the visibility of a replaced method is slightly more complicated.

In order to change the visibility of a replaced method we need to use two steps. First we use `insteadof` to replace the method and then in the next statement we stipulate that it must be protected.

Lets look at this in code.

In the first line inside the trait use block I'm aliasing a function and also changing its visibility. The keyword `protected` appears between `as` and the aliased name. If I were to try and call `doggyWalk()` from outside the class I would receive an error because it is protected.

I've commented out the second line because it's an example of what doesn't work. The rule of thumb seems to be that you can't use both `as` and `insteadof` in one single statement. You need to split them into separate statements as I do in the last two lines of the use block.

Connecting to Elixir web channels from the Angular 2 quickstart application

noreply@blogger.com (Andrew Beak) — Sat, 24 Sep 2016 13:36:00 +0000

I am busy learning Elixir, a language that adds syntactic sugar to the awesomely scalable and concurrent Erlang language. The "go to" framework in Elixir is Phoenix and I'm busy writing my "hello world" application which will serve up data across a web channel.

I followed the Typescript version of the Quickstart guide for Angular 2 (here). I really like what I've seen of Typescript so far. Dependencies are easy to manage and the ability to define interfaces is a good sign of how well structured the language is. I think Satya Nadella should be made an open source hero, if such an award exists.

Anyway, what I wanted to do was get my Angular 2 application to be able to connect to the Elixir server channel and send a request to listen to a particular stream. The idea is to use the Actor concurrency model (explained brilliantly in "The Little Elixir & OTP Book") to start up a new process whenever a request for a stream arrives. This article focuses on setting up the Angular 2 connection.

The official Phoenix javascript client is packaged up and can be installed with "npm install --save phoenix". Once it's installed we need to tell SystemJS where to find it so we amend systemjs.config.js and include it in our map array.

Now we'll be able to import the Phoenix class from it wherever we need to. We'll need it in the service that we're going to use to wrap the Phoenix channel support. Lets take a look…

We import the library using the map name that we set up in our systemjs config to make it available to our class. We then copy the code that Phoenix shows on the channels manual page to actually handle the connection.

This gives us a channel service that we'll be able to inject into components.

Angular 2 uses constructor injection so we'll be passing a channel service into the constructor function. Before we do that though we need to let the component know that our channel service is a provider so that we can properly typecast the injected variable. Once all this is saved run your app with 'npm start' and if you pop over to your browser window you should see error messages in your console log saying that the channel connection was refused, unless of course you're already running your Phoenix server.

Laravel refusing to start up

noreply@blogger.com (Andrew Beak) — Tue, 06 Sep 2016 08:12:00 +0000

I'm very much a fan of the clean implementation of Laravel but really dislike the fact that if there is something wrong with the .env file it refuses to give any meaningful information.

Laravel uses the vlucas/phpdotenv package to manage its environment.

It's pretty well known that if you have a space on either side the = sign in a key value pair then the .env file is invalid, but I had checked for this (and checked again).

Laravel will try to use its standard logging methods before they have actually had a chance to be booted up with the result that you're left with a "reflection error" exception message on the CLI rather than the actual cause of the problem in the dotenv package.

Debugging this is not trivial and I resorted to using strace to try and determine exactly what was going on. Don't do this at home kids! The easier solution is at the end of the article.

I used the following command to generate a trace of the system calls being made by PHP while trying (and failing) to run artisan optimize.

 strace php artisan optimize &> /tmp/strace.txt

That let me walk through the calls and eventually confirm that the first PHP exception was thrown in the package that deals with reading the environment file.

 access("/var/www/raffle.bhf.org.uk.new/vendor/vlucas/phpdotenv/src/Dotenv.php", F_OK) = 0  
 ... more lines loading up more of the package and showing us processing it ....  
 access("/var/www/raffle.bhf.org.uk.new/vendor/vlucas/phpdotenv/src/Exception/InvalidFileException.php", F_OK) = 0

But sadly there was no indication of exactly what the problem is with the file!

I decided that creating a minimal project dedicated to debugging my .env file was going to be faster than anything else.

I created a temporary directory and ran "composer require vlucas/phpdotenv". Then I placed my faulty .env file into the directory and ran the following PHP file:

 <?php  
 require('vendor/autoload.php');  
 $dotenv = new Dotenv\Dotenv(__DIR__);  
 $dotenv->load();

This gave me the actual exception in DotEnv which was that "Dotenv values containing spaces must be surrounded by quotes". So it wasn't a space around the = sign but rather a space in one of my values, which made my life a lot easier! As an extra bonus the first line in the stack showed exactly which key was problematic.

Solving a Docker in VirtualBox DNS issue

noreply@blogger.com (Andrew Beak) — Sun, 21 Aug 2016 15:43:00 +0000

I've recently been playing with Docker on Windows in conjunction with Linux on Windows. I'm really amazed at how cool the stuff coming out of Microsoft is under Satya Nadella.

When I was using Docker Toolbox on Windows my Dockerfiles would build correctly but as soon as I tried to run them in a Virtualbox host they would fail with the error "Something wicked happened resolving 'archive.ubuntu.com:http' (-5 - No address associated with hostname)"

Of course this error wasn't distribution specific and none of the distros I tried were working.

The stack that I am using is Windows Home hosting Ubuntu on VirtualBox which is running Docker. I'm using bash on Linux for Windows because it's easier to do stuff like ssh but it's not relevant to this setup.

I tried setting the DNS in the Docker setup by using RUN steps to update /etc/resolv. This felt a bit hacky and didn't work anyway.

In the end the fix was to go to my Windows shell and run ipconfig to get its DNS servers. I then went and edited /etc/default/docker in my VirtualBox guest server and set the Docker options to use this DNS. After restarting the docker service it was able to resolve DNS properly.

# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 192.168.1.254"

I had installed Docker with the package but I think if you install with binary you would set the DNS when you start up the daemon.

Ironically Docker was easier to use with the Windows toolkit but I'll be deploying to Linux machines so I wanted to get this working under Ubuntu.

Limiting how often you run seeds and migrations in Laravel 5.2 testing

noreply@blogger.com (Andrew Beak) — Tue, 12 Jul 2016 15:05:00 +0000

Image: Pixabay

If integration tests take too long to run then we stop running them because they're just an annoyance. This is obviously not ideal and so getting my tests to run quickly is important.

I'm busy writing a test suite for Laravel and wanted more flexibility than the built-in options that Laravel offers for working with databases between tests.

Using the DataMigrations trait meant that my seed data would be lost after every test. Migrating and seeding my entire database for every test in my suite is very time consuming.

Even when using an in-memory sqlite database doing a complete migration and seed was taking several seconds on my dev box.

If I used the DataTransactions trait then my migrations and seeds would never run and so my tests would fail because the fixture data is missing.

My solution was to use a static variable in the base TestCase class that Laravel supplies. It has to be static so that it retains its values between tests by the way. This variable is a boolean flag and tracks whether we need to run the migrations and seeds. We initialise the class with it set on and so the migrations and seeds will run the first time that any test run.

Now the fact that I'm using Laravel DataTransactions should spare me from affecting the database between tests but if I wanted to be 100% certain I could set the flag and have my database refreshed when the next test runs.

This also means that if the DataTransactions trait is not sufficient (for example I'm using multiple database connections) then I can manually refresh when I want to.

Implementing a very simple "all of the above" checkbox

noreply@blogger.com (Andrew Beak) — Wed, 06 Jul 2016 13:33:00 +0000

Image:Pixabay

Implementing a checkbox that lets the user "select all of the above" is trivial with jQuery.

The idea is that when the user selects the "all" checkbox then all of the other checkboxes are set to the same value. Similarly if they deselect any of the other options we need to turn off the "all" checkbox.

Why am I so late on the Bitcoin train?

noreply@blogger.com (Andrew Beak) — Tue, 28 Jun 2016 09:52:00 +0000

Image: Pixabay

I've been somewhat of a Bitcoin sceptic for quite some time. When it first became a thing I was worried that governments would legislate it out of existence.

It has had a pretty bad rap of being associated with the dark web and it is definitely the choice of currency for malware authors.

In its normal usage Bitcoin is more transparent than cash. If I give you a cash note there is no permanent record of the transaction and the tax man can't get a sniff into our business.

Governments hate transactions they can't tax or police and so in the beginning there was a concern that Bitcoin would be outlawed.

In contrast to cash, if I transfer you Bitcoin then there is a record of the transaction that anybody in the world can inspect. It's possible to trace the coins in your Bitcoin wallet back through the various people who owned them. Anybody in the world can watch the contents of your wallet and see where you spend your money.

This is exactly the sort of thing that governments love.

Of course not everybody wants to share their transactions with the world and so there are Bitcoin laundering services that attempt to anonymise the coins in your wallet. This puts us back to square one with Bitcoin being very convenient for criminals to use in order to evade financial intelligence controls.

I suspect that the process of banning Bitcoin transactions would impinge too much on the freedom of citizens. Some governments are talking about banning cryptography in order to maintain surveillance on their citizens so it's not a stretch to imagine them being displeased with Bitcoin laundering services. *sigh*.

Anyway, back to my killer app for Bitcoin... I've recently emigrated and am still paying debt in South Africa. Sending money back to South Africa costs about £30 and takes 2-4 days if I use the banking system. If I use Bitcoin the process costs around £ 3 and I can have the money in my South African bank account on the very same day.

Bitcoin costs one tenth the price of using banks and is at least twice as fast.

My transaction is not at all anonymous and a government can trace the funds to me on either end of the transaction where copies of my passport are stored with the exchanges. If I wanted to hide this from thieves, the government, spear-phishers, and other people who want to take my money without giving anything in return then I would use a Bitcoin laundry service.

Restarting BOINC automatically

noreply@blogger.com (Andrew Beak) — Fri, 10 Jun 2016 08:21:00 +0000

Image: https://boinc.berkeley.edu, fair use

BOINC is a program curated by the University of Berkeley that allows people around the world to contribute to science projects.

It works by using spare cycles from your computer to perform calculations that help do things like folding proteins to find candidates for cancer treatment, mapping the milky way galaxy, searching for pulsar stars, and improving our understanding of climate change and its effects.

It runs as a background process and is easily configured to only run in certain conditions - like when you haven't used your computer for 10 minutes for example.

It comes with a nifty GUI manager and for most people using it on their desktop this post is not going to be at all relevant. This post deals with the case where a person is running it on a server without the GUI manager.

Anyway, the easiest solution I found to restarting BOINC on a headless server was to use supervisord. It's pretty much the "go to" tool for simple process management and adding the BOINC program was as easy as would be expected:

Here's the program definition from my /etc/supervisord.conf file:

 [program:BOINC]  
 command=sh /root/boinc/startup.sh  
 process_name=boinc  
 exitcodes=0,1,2
 autostart=true  
 autorestart=unexpected

I use a script to restart BOINC because I want to make sure that I get reconnected to my account manager in case something goes wrong.

Here's what /root/boinc/startup.sh script looks like:

 /etc/init.d/boinc-client start  
 sleep 10  
 boinccmd --join_acct_mgr http://bam.boincstats.com <user> <pass>

If BOINC crashes it will automatically get restarted and reconnected to my account manager. This means I don't need to monitor that process on all the servers I install it on.

Associating Vagrant 1.7.2 with an existing VM

noreply@blogger.com (Andrew Beak) — Wed, 01 Jun 2016 12:37:00 +0000

My Vagrant 1.7.2 machine bugged out and when I tried to `vagrant up` it spawned a new box instead of bringing up my existing machine.

Naturally this was a problem because I had made some manual changes to the config that I hadn't had a chance to persist to my puppet config files yet.

To fix the problem I found used the command `VBoxManage list vms` in the directory where my Vagrantfile is. This provided me a list of the machine images it could find.

I then went and edited the file at .vagrant/machines/default/virtualbox/id and replaced the UUID that was in there with the one that the VBoxManage command had output.

Now when I run 'vagrant up' it spins up the correct VM. Happy days.

Redirecting non-www urls to www and http to https in Nginx web server

noreply@blogger.com (Andrew Beak) — Fri, 27 May 2016 12:03:00 +0000

Image: Pixabay

Although I'm currently playing with Elixir and its HTTP servers like Cowboy at the moment Nginx is still my go-to server for production PHP.

If you haven't already swapped your web-server from Apache then you really should consider installing Nginx on a test server and running some stress tests on it. I wrote about stress testing in my book on scaling PHP.

Redirecting non-www traffic to www in nginx is best accomplished by using the "return" verb. You could use a rewrite but the Nginx manual suggests that a return is better in the section on "Taxing Rewrites".

Server blocks are cheap in Nginx and I find it's simplest to have two redirects for the person who arrives on the non-secure non-canonical form of my link. I wouldn't expect many people to reach this link because obviously every link that I create will be properly formatted so being redirected twice will only affect a small minority of people.

Anyway, here's the config:

Logging as a debugging tool

noreply@blogger.com (Andrew Beak) — Wed, 11 May 2016 09:25:00 +0000

Image: https://www.pexels.com

Logging is such an important part of my approach to debugging that I sometimes struggle to understand how programmers avoid including logging in their applications.

Having sufficiently detailed logs enables me to avoid having to make assumptions about variable values and program logic flow.

For example, when a customer wants to know why their credit card was charged twice I want to be able to answer with certainty that we processed the transaction only once and be able to produce the data that I sent to the payment provider.

I have three very simple rules for logging that I follow whenever I'm feeling like being nice to future me. If I hate future me and want him to spend more time answering queries than is needed then I forget these rules:

The first command in any function I write is a debug statement confirming entry into the function
Any time that the script terminates with an error then the error condition is logged, along with the exception message and variable values if applicable.
When I catch an exception then I log that as a debug message in the place that I catch it rather than letting it bubble up the stack. If I'm the one throwing the exception then I log in the place where I throw it.

These rules have grown on me from the experience of debugging code and having to deal with an assortment of customer queries that have been escalated to the development team.

By logging the entry into functions I can go back on my logs and see the path that a request took through the code. Instead of wondering how a particular state of execution came about I have a good trail of functions that led me to that point.

To me errors that fail silently are the worst possible errors. I don't expect that the user needs to be alerted to every error or its details, but if I am unable to send a transactional email then I expect that there should be a log of that fact. That might sound self-evident but I've recently worked on a project where we send a mail and don't check if it was successful or not. This would only occasionally happen and we only noticed something was amiss when a customer complained. I was not able to determine when the problem arose, how often it happened, or to whom I should resend transactional mails along with an apology.

Logging an exception in the place I catch it has consistently proven to be helpful. Having a log as close as possible to the source of the error condition helps to narrow down the stack that it occurred in. This is especially valuable when I rethrow the exception with a user friendly message because I don't lose the technical details of the program state.

Because my logs can get very spammy in production I use the "Fingers Crossed" feature of Monolog. I prefer this to the alternative of increasing the bar for logging to "info" and above because when an error occurs then I have a verbose track of my program state. I've created a gist showing the setup in Laravel 5.1 and 5.2 but the approach will work anywhere that you use Monolog.

Another useful trick I've learned is to integrate my application errors into my log aggregating platform. I use Loggly to aggregate my logs and push application error messages to it. This lets me easily view my application errors in the context of my various server logs so spotting an nginx error or something in syslog that could contribute to the application problem is a lot easier. The gist that I linked above shows my Loggly setup, but you can also read their documentation.

Useful and appropriate logging is an indispensable tool for debugging and if you're not working on developing your own logging style to support your approach to debugging then hop on it!

Are tokens enough to prevent CSRF?

noreply@blogger.com (Andrew Beak) — Fri, 08 Apr 2016 16:37:00 +0000

Image: Pixabay

CSRF attacks exploit the trust that a website has in a client like a web browser. These attacks rely on the website trusting that a request from a client is actually the intention of the person using that client.

An attacker will try to trick the web browser into issuing a request to the server. The server will assume that the request is valid because it trusts the client.

At its most simple a CSRF attack could involve making a malicious form on a webpage that causes the client to send a POST request to a url.

As an example, imagine that a user called Alice is logged into Facebook in one tab and is browsing the internet on another tab. A filthy pirate Bob creates a malicious form in a webpage that submits a POST request to Facebook that sends a person to a link of Rick Astley dancing. Alice arrives on the page we made and Javascript submits the form to Facebook. Facebook trusts Alice's web browser and there is a valid session for her so it processes the request. Before she knows it her Facebook status is a link to Rick Astley (who, by the way, will never give you up).

Of course Facebook is not vulnerable to this, and neither should your code be.

The best way to mitigate CSRF attacks is to generate a very random token which you store in Alice's session. You then make sure that whenever your output a form on your site that you include this token in the form. Alice will send the token whenever she submits the form and you can compare it to the one stored in her session to make sure that the request is originating from your site.

Bob has no way of knowing what the token in Alice's session is and so he can't trick her browser into submitting it to our site. Our site will get a request from Alice's client but because it doesn't have the token we can reject it.

In other words the effect of the token is to stop relying on implicit trust for the client and rather set up a challenge response system whereby the client proves it is trustworthy. If Bob wants to send a request that will be accepted he must find a way to read a token off a form that your site has rendered for Alice. This is not a trivial task but can possibly be done - there are very creative ways (like this attack) to abuse requests.

Another way to prevent CSRF is to rely on multi-factor authentication. We can group ways to authenticate into knowledge (where you know something like a password), possession (where you have something like a USB dongle), or inherent (where you are something).

Instead of just relying on one of these mechanisms we can use two (or more) in order to authenticate. For example we can ask a person for a password and also require that they enter a code sent to the mobile phone which proves they have the mobile phone linked to their account.

CSRF will become much harder for Bob to accomplish if our form is protected with multi-factor authentication (MFA). Of course this comes with a user experience cost so only critical forms need to be protected with MFA. For less critical forms the single authentication method of a CSRF token will suffice.

There is debate around whether it is useful to check whether the referrer header matches your site is helpful in deterring CSRF. It is true that it is trivial to spoof this header in a connection that you control. However it is more difficult to get this level of control in a typical CSRF attack where browsers will rewrite the referrer header in an ajax call (see the specification). By itself it is not sufficient to deter CSRF, but it can raise the difficulty level for attackers.

Cookies should obviously not be used to mitigate CSRF. They are sent along with any request to the domain whether the user intended to make the request or not.

Setting a session timeout window can help a little bit as it will narrow the window that requests will be trusted by your application. This will also improve your session security by making it harder for fixation attacks to be effective.

Tokens are the most convenient way to make CSRF harder to accomplish on your site. When used in conjunction with referrer checks and a narrow session window you can make it significantly harder for an opponent to accomplish a successful attack.

For critically important forms multi-factor authentication are the way to go. They interrupt the user experience and enforce explicit authentication. This has a negative affect on your UX but makes it impossible (I think!) for an automated CSRF attack to be effective.