aneesh.co.uk

Risk neutral pricing a multi-period binomial model

Wed, 24 Apr 2024 00:00:00 GMT

Risk neutral pricing a multi-period binomial model

In this video, we review pages 8 to 15 from Steven Shreve's Stochastic Calculus for Finance part I where we derive the risk-neutral price and delta hedge required for a multi-period binomial model. We revisit the delta hedging formula and risk-neutral pricing formula within a multi-period model to demonstrate how they can influence the fair price of the option at time 0.

Risk-neutral pricing and the delta-hedging ratio

Mon, 01 Apr 2024 00:00:00 GMT

Risk-neutral pricing and the delta-hedging ratio

In this video, we will walk through pages 4 to 8 of Steven Shreve's Stochastic calculus for finance part 1 where we develop the risk-neutral pricing formula and delta hedging ratio for a one-period binomial model.

How to merge data with Panda DataFrames

Fri, 29 Mar 2024 00:00:00 GMT

Merging data with Panda DataFrames - Python Tutorial

Explore some of the ways you can merge DataFrames with the pandas library. This involves handling indexed columns, specifying columns to merge upon and how to merge with an inner, left, right and outer join.

Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/Python File: Pandas 4.ipynb

The one period binomial model

Sat, 23 Mar 2024 00:00:00 GMT

The 1-period binomial model

Walkthrough the first 4 pages of Steven Shreve's Stochastic calculus for finance I, where we introduce the one-period binomial model and how a replicating portfolio can be created for a European call option in the stock and money market.

Combining dataframes with Pandas

Fri, 22 Mar 2024 00:00:00 GMT

Combining DataFrames with Pandas

Walkthrough how you can use the concat method with the pandas library to combine DataFrames both on the x axis and y axis.

Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/Python File: Pandas_3.ipynb

Statistical Methods with Python Pandas

Fri, 15 Mar 2024 00:00:00 GMT

Statistical methods with Pandas - Python Tutorial

A walkthrough of some of the most common statistical methods you can perform out of the box with the Pandas library.

This includes: Sorting data, filtering data, finding correlations amongst fields, replacing values and sampling data.

Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/Python File: Pandas_2.ipynb

Series and DataFrames with Python Pandas

Sat, 24 Feb 2024 00:00:00 GMT

Python Pandas 1

This video serves as an introduction to Python Series and DataFrame structures from the pandas library. We will walk through the key built-in methods that enable you to create the structures, select, filter and transform the data.

How the Electricity Market of Great Britain functions

Sat, 17 Feb 2024 00:00:00 GMT

Key Topics
• The lifecycle of electricity, from generation to consumption.
• The various markets that enable market participants to trade electricity.
• How the National Grid ESO balance the supply and demand of electricity.

Introduction

This blog is going to help you develop an understanding of the mechanisms that enable the electricity market of Great Britain to operate. We will first understand the challenges imposed from the properties of electricity that translate to how it is created, distributed and (not) stored. Following this, we will explore the various physical and financial markets that electricity are traded in. Lastly, we will look at the governance and management of the electricity system that sustain a reliable system.

Generating Electricity

The lifecycle of electricity can be categorised into 3 main steps: generation, transmission, and consumption.

Generation is achieved by dedicated power generators in one of many ways. The most common generators are turbine-driven, they convert mechanical energy into electrical energy by forcing electrons through a circuit. Steam turbines work by burning fuel or biomass to produce pressurised steam. Nuclear power reactors use nuclear fuel rods to produce pressurised steam. Both of these reactors convert the steam into mechanical energy for electricity generation.

A similar type of generator are combustion gas turbines. These generators instead use the combustion of gaseous or liquid fuels to produce pressurised gasses to spin turbines.

And lastly there are alternative, but less commonly used generators such as hydroelectric turbines that use the force of moving water to spin turbines, wind turbines and also solar photovoltaic systems that convert photons from sunlight into electricity.

But what IS electricity?

You can find many definitions of electricity, but at it's very core, it is the flow of electrical power. Electricity represents a flow of electrons, and is often referred to as an energy carrier as it can be converted into other forms of energy such as mechanical energy or heat. This property of electricity, being a flow of electrons, bring us to the main challenge it poses: it cannot be stored at a wholesale level. There is a potential for it to be converted and stored, such as in a battery, however for the wholesale movement of electricity, this would not be feasible or cost effective. As a result, the supply and demand of electricity must be kept in a constant balance, and we will later see how this is achieved at a granular level.

Transmission Networks in GB

The electricity created by a generator is subsequently transmitted around the country via transmission networks. Transmission networks come in 2 forms: large transmission towers which you may see line motorways, and smaller distribution networks that transmit electricity at a local level to individual homes and businesses.

Transmission networks are long distance, high voltage lines transporting electricity from generators to distribution networks. These are considered the 'motorways' of the electricity system. We introduce the term 'voltage' to quantify the pressure which pushes electricity through the wires. Voltage in a transmission network tends to be from 275 to 400 kilovolts.

Distribution networks transmit electricity at a lower voltage from transmission lines to homes and businesses for consumption. This voltage tends to be around 132 kilovolts. There are 14 'grid supply point groups', each are managed by a distribution network operator (DNO). The distribution network is responsible for reducing the voltage of the electricity so it can be safely distributed for consumption within homes and businesses.

Electricity Consumption

The consumption of electricity takes place within homes and businesses, where electricity is used for many reasons, such as keeping your lights on, charging your mobile phone, and watching YouTube videos about Java.

The supply and demand of electricity

We measure the generation of power using watts. Watts capture the rate at which electricity is transferred at a single point of time, and represents the amount of energy required for an item to function. Given the large scale of power generation, we refer to 1000 watts as a kilowatt, 1000 kilowatts as a megawatt, 1000 megawatts as a gigawatt, and 1000 gigawatts as a terawatt.

To put a watt into perspective, an average toaster consumes 1100 watts.

From a supply perspective, consider a wind power generator, which generates on average 3 megawatts of electricity. This fairs in comparison to one rooftop solar system that you find on a residential home, which generates just 4 kilowatts of electricity.

When we talk about the consumption of electricity, however, we use watt hours. Energy is measured in watt hours to capture how much electricity is produced or demanded in a given hour. You can think of this as a car travelling at 30 miles per hour for 1 hour. The speed (watt) would be 30, however the total distance travelled will be 30 miles (watt-hours). For some more perspective: Great Britain consumed 321 terawatt hours of electricity in 2022.

The supply and demand of electricity in Great Britain is based upon watt hours. The parties who are demanding electricity are energy suppliers. These tend to be the privately-owned corporations that you pay bills to, such as British Gas, EDF, E. ON, and Scottish Power. Energy suppliers enter into contracts with power generators, using a forecast for how much energy their customer base will require as the basis for the contract. We will see later on in this blog how the suppliers are better able to meet the finer granularity of demand they face from their customers on an hour-by-hour basis.

Electricity governance and regulation

With hundreds of participants in the electricity market entering into contracts for the purchase and delivery of electricity, there exists a single entity responsible for ensuring the balance of all these contracts in unison. That is the National Grid Electricity System Operator (ESO).

When generators and suppliers of electricity enter into a contract, they will inform the National Grid ESO of their intentions. With a consolidated view of electricity movement across national generators and suppliers, the National Grid can realise where an imbalance lies, whether that is 'long' from too much supply of power, or 'short', due to an inefficient supply.
The National Grid are then able to operate reserve generators to increase or reduce their contribution to the network to maintain a balance.

All participants within Great Britain's energy market have obtained a license from OFGEM. OFGEM are Great Britain's independent gas and electricity regulator. They exist to protect consumers of electricity by enabling market competition, enforcing market rules and promoting innovation. OFGEM are responsible for imposing price caps on electricity suppliers, and imposing fines on generators. You can find an interesting read on one of the larger fines imposed by OFGEM in 2020: here.

Determining the domestic cost of electricity

In this section, we are going to look at how the price you pay for electricity is determined. This price comes down to an understanding of Great Britain's generation stack.

The supply of electricity in Great Britain is made up from a generation stack of different types of generators, all contributing electricity to the National Grid network and at different prices. Electricity from wind turbines will typically be generated at a cheaper rate than nuclear, carbon or gas power stations.

A 'merit order' is defined within the stack to determine which electricity source is consumed first, this will always be the cheapest.

When we consider the total energy that is demanded at a given time, the merit order is used to determine which generators will supply the electricity for that given time. As demand for electricity rises, the number of generators required to meet that demand will increase, and subsequently the more expensive generators down the merit order will be used. The relationship between the merit order and your electricity bill is that the price you pay for electricity at a given time is determined by the most expensive generator used to meet the demand at that time. This is why off-peak electricity, which is generated when demand is low, will be cheaper than when demand is higher and more expensive generators are contributing to the supply of power. This is known as the 'pay-as-you-clear' model.

The merit order also has a role in determining the development of new power generators. When considering the capital expenditure and return a generator can produce, developers will forecast how often the generator will be contribute to the network and generate income.

The Electricity Markets

So far, we have touched the surface of electricity generation, transportation and distributed to your homes. We can understand how the price of electricity can be derived at different times, and the high level challenges the National Grid ESO tackle to maintain a balance of supply and demand in Great Britain. We are now going to take a look at who the different players are within the electricity market, and why they are motivated to buy or sell electricity.

We will begin by dividing the electricity ecosystem into four segments: the capacity market, the wholesale market, ancillary services, and the non-physical market.

The Wholesale Market

The wholesale market is where the majority of electricity is bought and sold. Agreements for electricity occur both bilaterally, known as an over-the-counter deal, and through exchanges. Agreements can vary in time frames from delivery of electricity months in advance, down to delivery in the very same day.

Within a bilateral agreement, the suppliers negotiate a price in MWh for the delivery of electricity at a specific time period with the generators. When a bilateral contract is agreed, the National Grid ESO must be informed so they can manage the supply and demand balance of electricity. The bilateral agreement tend to be long-standing contracts that enable the energy supplier to guarantee the delivery for months at a time. The wholesale market also operates through a power exchange. A power exchange is similar to financial exchanges where suppliers and consumers meet to determine a purchase price. Power exchanges can be used for long-dated contracts as well as short-dated contracts as we will soon discuss.

When buying and selling electricity, both parties will reference specific time periods for delivery. One common time frame is for delivery at every hour within a given day, known as 'baseload'. An energy supplier will use a baseload contract to purchase the least amount of electricity they expect to need in a given day. The baseload therefore ensures the supplier are able to meet the minimum electricity demand they forecast from their customer base throughout a given day.

The wholesale contracts can be traded in large periods ahead of time to secure the delivery and price of electricity. As you may expect, the demand for electricity in a normal household will not be equal at all times. The typical 'peak hours' are from 7am until 7pm. As a result of the higher demand during these hours, the energy suppliers must increase their orders of electricity during these periods, which can be achieved through a 'peakload' contract. The process of matching the specific demand of electricity for all hours of the day is known as 'adding shape'.

In the image below, I have used a red line to illustrate how the forecasted demand of electricity can change within a given day. We can then see how a supplier can meet that demand with individual baseload and peakload contracts. The outstanding gap between the red line and the blocks can be met with more granular agreements in the market.

As the time for electricity to be delivered to consumers approaches, energy suppliers can more accurately forecast their expected levels of electricity required. This is shown as the dashed black lines between the peakload and forecasted demand. Electricity suppliers can work to a granularity of hourly and half-hourly contracts to meet this outstanding demand. The finer requests for electricity supply can be obtained through day-ahead and within-day exchanges, for suppliers in Great Britain, these are the EPEX Spot and Nordpool exchanges.

EPEX and Nordpool offer two types of markets for short-term electricity delivery which can help market participants to . These are the day-ahead (DA) and within-day (WD) markets. As described, the day ahead exchange will deliver electricity for the next day, and within-day is for same-day delivery. Epex and Nordpool work as an auction exchange, where generators and suppliers publish their acceptable price limits to buy and sell electricity. The market will then meet an equilibrium to arrive at a price for electricity at each given time-interval.

The wholesale market operates across huge timelines that enable both a guarantee for the delivery of electricity in the future, but also the granularity of electricity delivery at an hour-by-hour basis. The agreement for electricity delivery can take place at any time up to 'gate closure', which is 1 hour before the electricity is set to be delivered. Upon gate closure, generators will inform the National Grid ESO of their intentions to generate electricity.
Upon gate closure, the National Grid ESO can begin ensuring the supply and demand of electricity is balanced. Balancing is achieved through the use of a balancing mechanism, where generators and industrial users are paid to change their intended demand and supply. This can be a little detailed, so I will write about this in it's own section at the end of this blog.

The Capacity Market

The capacity market can be thought of as an insurance policy for the delivery of electricity to avoid a potential blackout. The capacity market is a standby market of generators that are ready to produce electricity in periods of high stress on the existing system. High stress can be created where the supply of electricity is low, due to factors such as low wind, and the demand for electricity is high. The capacity market therefore benefits electrical generators that can easily start up and turn off on demand. Contracts within this market can range from 1 to 15 years. With long-term contracts, generators expect to exist with longevity in mind, and play an important role in the long-term generation stack that Great Britain uses.

Generators that contribute to the capacity market are paid for the availability they provide the National Grid ESO to turn on at peak times of electricity consumption. In the event of an unexpected sudden surge in electricity requirement, known as a capacity market event, the National Grid ESO will issue a 4-hour warning to these generators to turn on. This is known as a capacity market notice.

The capacity market was first auctioned in 2014 and works in two auctions: T-4 and T-1. The auctions work on a yearly basis, where the 4 year auction can be for a 15 year contract that begins in 4 years time. The T-1 auction market is used by the National Grid to ensure there is sufficient capacity to support the following years winter.

Ancillary Services

Ancillary services capture ways that generators can earn money through services that support the stability and efficiency of the power grid. Ancillary services can be split into 6 responsibilities: frequency response, reactive power, voltage management, inertia services, reserve services and restoration.

The frequency response service exists to ensure transmission towers run at a target frequency of 50 hertz. This frequency of electrical transmission is important to maintain as it can otherwise create blackouts or damage the equipment. The frequency response service will manage frequency on a second-by-second basis to actively manage frequency by ramping power up and down.

Reactive power services exist to ensure voltage levels within the system remain within their target range. Reactive power exist as a by-product of power from AC circuits, you might think of this as the head of a beer when it is poured. It is measured in MVars (megavolt-ampere reactive) and is actively absorbed to reduce voltage or generated to increase voltage on the grid. This balance of reactive power ensures the grid is efficient in delivering power, and not overloaded which can create network volatility.

Voltage management covers the movement of electricity at a specific voltage through the network. It is closely related to reactive power as both of these relate to the rate which power is moved. Each transmission network is designed to manage a level of voltage, such as 400 kilovolts (for national networks) or 275 kilovolts (for distribution networks). Voltage management is achieved through the use of an electrical transformer to optimise the voltage at which electricity is transported. If the voltage is too low, there may be lost value from the transmission networks in how much electricity is being transported, however on the other end, if voltage is too high, this can result in electrical equipment consuming power in its transmission, and even creating an unstable supply of electricity. There is roughly a 5% buffer for which voltage must be managed at in kilovolts.

Inertia services exist to support the reliable frequency of power generation to the grid. In traditional turbine-driven power generators, inertia acts as a softener to frequency changes in electricity as turbines continue to spin even when they are not generating power. This means a sudden change in frequency from a generator will not cause a hiccup in the electricity grid. You can think of inertia as similar to the suspension in a car, which absorbs the shock of a change in the road surface.
The design of wind turbines has reduced the inertia created when the generator stops producing electricity, this means the National Grid must purchase inertia from other generators to maintain the stable level of frequency on the grid.

Reserve services ensure there is sufficient power in reserve in case of a change in generation or denied services.

Lastly, restoration services, formerly known as black start, are responsible for supporting the grid in the event of a complete shutdown. Restoration allows a power station to restart without an external electricity flow to re-energise the grid.

The non-physical market

So far, we have discussed the physical market for electricity where the counterparts involved are engaging in contracts with the intention to either produce or distribute and consume electricity. Non-physical trade differs as it driven by the means of financial speculation or risk management. The scenarios to be discussed are highly simplified for the purpose of the blog.

The non-physical market use a variety of financial derivatives to trade electricity, however the most commonly used derivative is a forward contract.
The forward contract represents an agreement between two parties to buy or sell a quantity of electricity in the future at a price agreed today. The forward contract thus reduces the price risk for the buyer of the electricity who can fix it ahead of time, and the supply risk for the generator who can guarantee a buyer.

Non-physical trading can enable an agent to realise a profit without actually generating or consuming electricity: an agent could enter into a forward contract agreement to purchase a given amount of electricity at a given forward price, K, 1 year from today. As the date for the purchase of electricity approaches, the agent could sell their forward agreement in the market, thus realising a potential profit depending on how the current price of electricity fairs in comparison to the forward price K.

In our scenario, the agent is 'impersonating' a supplier of electricity as they purchase the forward contract from a generator, but when they resell the forward agreement to a supplier, they are 'impersonating' a generator of electricity by selling the contract to a supplier. I put the word 'impersonating' in quotes as the agent are not really impersonating anyone, but are more fairly just taking a position in the market that the generator or supplier would be taking.
An agent can also enter into agreements with suppliers to sell electricity at a given price, thus impersonating a generator. This position would be sold in the market thus granting the contract to generate electricity to an actual generator.

The non-physical market exposes opportunity based on the disparity between a forward price of electricity and the actual price of electricity when the time for delivery approaches. This disparity does not present an arbitrage opportunity. The agents who enter into such deals are speculating upon the price of electricity in the future. If the current price of electricity is £10 MWh, and the forward price for 1 year is £12 MWh, the agent will benefit if the price of electricity rises to £14 MWh as they can receive the power at £12 MWh and sell it in the market at £14 MWh. If the price however falls to £9 MWh, then they are obligated to purchase the electricity at £12 MWh and to sell it at £9 MWh, or effectively sell the forward contract at a loss in the market.

Within the scope of the non-physical market are also asset-backed traders. These are agents that may enter into financial contracts to hedge or speculate upon the price of electricity, however they have the infrastructural means to generate electricity and deliver against the contract, rather than selling their position in the market as the forward date approaches.

Agents are incentivised to realise greater profits from their generation by engaging in the speculative market on potential price, and makes up 90% of the electricity trades that are made in advance of delivery.

For non-physical trading, I have only mentioned the forward contract, which is utilised in bilateral agreements. The futures market, which is effectively a forward contract that is traded through an exchange contributes heavily to the electricity market.

Futures derivatives represent similar contracts, however are more standardised and traded through an exchange. The common exchanges used for trading electricity are ICE for long-dated agreements, and EPEX Spot or Nordpool for day-ahead and within-day trading. Within ICE, the two main prices for Great Britain power are the UBL, the UK Base Electricity Future, and UPL, the UK Peak electricity future. The futures agreements can be traded as far as 5 years into the future, and tend to be for delivery at entire months at a time at the very minimum, this is the standardised nature of a future.

The standardisation of contracts for electricity are largely based on time periods within the day. I have already mentioned baseload and peakload within this blog which are used for long-dated futures with ICE. Nordpool and EPEX Spot are exchanges that are used for short-term electricity delivery and therefore use a much finer granularity of time intervals for delivery. To better understand how a day can be broken down for exchange trading, we can take a look at the EFA day.

The EFA Day

EFA stands for Electricity Forward Agreement. It is used to capture the time-granularity at which electricity can be traded within a day.

The EFA day consists of 6 4-hour time blocks. The time blocks start at 23:00 from the day before, and run until 23:00 of the current day. There are 48 half-hour blocks, running from midnight to midnight. When making an agreement using the settlement blocks, they will therefore apply from previous day.
Each EFA block can be split in half, to give us 1a, 1b, 2a, 2b etc. And EFA blocks can be grouped to provide a descriptive product. These products include:
• Baseload - Blocks 1 to 6.
• Off-Peak - Blocks 1, 2 and 6 during weekdays, and all weekend.
• Peak - Blocks 3, 4, and 5 during weekdays.
• Overnight - Blocks 1 and 2.
• Daytime - Blocks 3, 4, 5, 6.
These blocks are typically traded at at larger time-scales before delivery, whereas smaller blocks of delivery are traded closer to the time of delivery as forecasts for demand are better known from the purchasers of power.

Picture: EFA Blocks from YouTube Video by Modo Energy

Day-ahead and Within-Day Power Trading

The day-ahead and within-day markets provide an opportunity for finer granularity trading at different time periods close to the delivery. These markets use spot markets for price discovery in the here and now. The exchange markets enable energy consumers to finely match their required demand at each hour to the electricity they have incoming from different suppliers as the time for delivery approaches.

As mentioned earlier in this blog, electricity suppliers will often enter into long-dated contracts to fulfil their baseload and peakload requirements using their forecasts. The energy suppliers can then use the Day-ahead (DA) and Within-day (WD) markets to match the outstanding demand at all other times of delivery.

The power exchanges Epex Spot and Nordpool both operate as auction exchanges; they build an order book of the volumes their registered participants need to buy and sell and the prices they are willing to accept for each day. Using the order book, the exchange can calculate the market price at each time interval.

The auctions run everyday, accepting all the bid and offers 1 day prior to the day-ahead auction and calculating the price. The within-day auctions run all the way up to gate closure. Gate closure is 1 hour before the time for delivery of electricity, it is where all participants in the market must submit their intentions for electricity demand and supply to enable the National Grid to ensure the system is balanced and that electricity can be reliably delivered.

The Balancing Mechanism

The balancing mechanism is used by the National Grid to ensure electricity supply and demand are balanced at the time for delivery. The motions for balancing take place after gate closure when generators and suppliers publish details of their intended generation and consumption for the settlement period. From a high level, there are two functions of the balancing mechanism that ensure supply meets demand. The first function comes from designated Balancing Mechanism Units that increase or decrease their intended demand or supply, and the second function is from Net Imbalancing Units that have excess electricity to sell to the grid when required.

The submission from market participants to the National Grid of their intentions is known as a physical notification. The notifications can be updated up until gate closure, when they then become 'final notifications'. The National Grid use final notifications to calculate the imbalance that is present between supply and demand. They then use the balancing mechanism to equate the two.

The Balancing Mechanism operates through registered participants. Participants register themselves with Elexon, the organisation who manage the Balancing and Settlement code. Participants are known as Balancing Mechanism Units (BM Units) and they contribute to the Balancing Mechanism by submitting bid and offer prices for each half-hourly period up to gate closure for what they are willing to pay to reduce or increase their contribution to supply or demand. A bid price is submitted if a generator want to reduce their generation, or if a supplier want to increase their demand. This is known as a flex up. An offer price is made for the opposite, where a generator increases supply or a supplier decreases demand, known as a flex down. The National Grid ESO will decide which bid and offer to accept for balancing using factors such as cost efficiency and volumes from the bid and offers.

These bid and offer prices can also be negative, where the participants are paid to reduce/increase against their original plans. Once the National Grid have accepted a price, they issue a Balancing Offer Acceptance (known as a BOA).

There is a further mechanism beyond the BMU's to ensure the balancing mechanism works, and this is through net imbalance volumes. Once the BMU participants are selected, and the National Grid see a balance between supply and demand, there is still a dependency for all participants to contribute as expected. If there is a shortfall, for example a generator not contributing as expected, or a financial participant has agreed to buy and sell unequal amounts of electricity, Elexon are responsible for meeting the shortfall. Participants that fail to deliver are considered 'out of position' and must therefore pay Elexon what is known as a 'system price' or 'cash out' price for failing to meet what is due. The system price is considered a penalty as the price paid will be greater than the market rate for electricity, whether it is for a difference in supply or demand.

When an imbalance occurs, Elexon will turn to a new participant for the generation of electricity, and these are generators who have excess demand available that they had not assigned to the grid. Generators will receive payment at the system price for the excess electricity that is required, thus possibly realising a greater profit for their electricity than they had at a market price. This unique situation has lead to the term 'NIV Chasing', where generators can deliberately over-generate on the possibility of receiving exposure to the system price.

With the facility from BMU's and generators who contribute to net imbalancing, the National Grid are able to manage the supply and demand of electricity across the grid at all times of the day to minimise the potential for unused electricity while ensuring the expected demand of electricity is met.

Summary

This blog has taken a walk through the electricity markets of Great Britain at a very high level to capture the mechanisms in-place for the electricity lifecycle, physical and financial participation and balancing mechanisms. This simplified view of generators and suppliers enable us to understand where our electricity comes from, the factors that contribute to the price we pay for it and the complexity of the system that enables it to remain stable and efficient.

An introduction to Numpy

Sat, 27 Jan 2024 00:00:00 GMT

Numpy

Follow along in this Python tutorial where I introduce the basic methods and operations that you can perform with numpy. We will take a look at how you can create an array in numpy, reshaping it into different matrix sizes and selecting specific rows and columns from that matrix. We also dive into the different ways you can generate and obtain random numbers with numpy. And lastly we will conclude with some basic mathematical operations you can perform with numpy.

This video will serve as a perfect first stop with getting yourself familiar and confident with the numpy package from Python.

Link to numpy documentation: https://numpy.org/doc/

🕒 Timestamps 🕒

00:00 - Introduction 00:28 - Creating arrays 03:00 - Generating random numbers 05:55 - Reshaping data 06:48 - Indexing numbers 10:45 - Array Operations 12:28 - Mathematical Operations

CORBA 101

Sun, 26 Nov 2023 00:00:00 GMT

Distributed system communication

CORBA - the common object request broker architecture (yes, a mouthful), is a mechanism used within computer networks to allow different machines to identify, manage and invoke operations upon each others Objects remotely. What makes CORBA so useful, is that it has an almost complete language-agnostic design, meaning systems within the network don't need to be written in the same language to interoperate. This is what makes CORBA unique and relevant to engineers, even 30 years after it's inception.

With it's interoperable design, CORBA is a concept I routinely face in my job. What I had thought would be a simple 5-minute google session turned into a multi-day activity, as the CORBA architecture and workflow isn't as straight-forward as it's value proposition.

In this blog, we are going to look into the components that make CORBA work while demystifying the many terms used around CORBA networks. At the end, you should be better positioned to navigate a conversation on CORBA (as I have struggled to do), and to understand why designing interactions within a CORBA network is extremely important.

What CORBA is and what it isn't

Before we dive into CORBA, I will offer a few comparisons of CORBA to help us understand why it still has relevance in engineering, and how it differs from other protocols.

CORBA vs Remote Method Invocation (RMI)
RMI provides a mechanism for which a server and client can communicate with each other, they pass information back and forth and where remote Objects can be created, referenced and invoked upon by the client.
RMI differs from CORBA as it is only used for Java applications, whereas CORBA is language agnostic. RMI uses Java interfaces to define contracts for the type of Objects that can be invoked upon, whereas CORBA uses the interface definition language (IDL), which I will discuss later.
RMI passes Object by value and reference, whereas CORBA uses reference only, via the naming service (also, more on this later).
And lastly, RMI uses the JVM to identify and locate Objects, whereas CORBA uses a Local Object Adaptor (LOA) to do the same.

One way CORBA differs from other translation layers, such as RPC, is that it supports remote Object references. This means you can pass Object references as parameters within your service call. For example, if you had an Object that represented a file, you could pass that Object as a reference to another remote call to then add the file to a database, thus allowing Objects to be passed as parameters.

CORBA vs SOAP
SOAP, the simple Object access protocol, is an xml-based protocol used for distributed members of an application to communicate with each other. SOAP is powerful as it enables language agnostic communication and supports a variety of protocols including HTTP, SMTP and TCP. The main difference between CORBA and SOAP is that CORBA is stateful, whereas SOAP is stateless. With CORBA, you can capture the state of an application from within the instance of the Object. The state of the Object is modified by calling functions upon the Object. SOAP, however, uses a stateless protocol.

SOAP also differs from it's bespoke procedures. Instead of translating requests and responses on both the client and server, the client and server would instead agree to a common interface codified in a language called extended markup language, or XML. This makes SOAP a little more accessible to use and maintain over CORBA. We will later see how translations of requests work in CORBA.

CORBA vs REST
REST is similar to SOAP in how it resolved a communication challenge amongst different systems built with different programming languages. REST provides greater scalability and flexibility in it's design through a dynamic exchange. As a result, REST differs from CORBA in a similar way that SOAP does, through it's stateless design and protocols for communication. Nevertheless, I would say that both REST and SOAP serve different purposes to CORBA in the active management and control of remote Objects.

The CORBA Architecture

Thank you for reading this far, we've opened up quite a few ends from that introduction, but now we can start to look at the real detail of CORBA..

To explain how CORBA works, I will introduce the individual components one by one, and describe how each one interlinks through an interaction between a client and server.
Below is an image of the CORBA architecture to capture the individual elements for a request. (Image by Vinoski via: https://www.dre.vanderbilt.edu/~schmidt/PDF/vinoski.pdf)

Client
A client is responsible for initiating the workflow with CORBA as they invoke operations upon an Object and process their response. The client is made aware of the structure of the Object through it's interface, and is able to experience the behaviour of the Object through invoking it's methods.

Interface Definition Language (IDL)
The IDL is independent of any programming language; it enables systems of all programming languages to access Objects via the advertised interface. The logical processing of the CORBA Object is hidden from the client, however it's structure and methods are made clear through the IDL.

The IDL provides the ability for CORBA clients to invoke actions upon an Object. The IDL is defined for each CORBA Object and is provided to the client for invocation. Any client that wants to invoke upon the Object will use the interface and transmit (by marshalling) the parameters for that request.

Object
An Object represents an implementation of an IDL. Each Object can be identified by a reference which uniquely names the instance across servers (known as an Object reference).

Naming Service
The naming service enables CORBA clients to obtain an Object through it's associated name. The naming service will both create bindings between Objects and their reference names to make them available to a client.

The Internet Inter-ORB Protocol (IIOP)
The CORBA standard uses the IIOP for client-server communication. The IIOP enables a client and server from almost any vendor, computer, OS, language and network to communicate with each other through an Object Request Broker (ORB).

As a result, the client can communicate with the server through it's address (server and port) known as the Interoperable Object Reference (IOR).

Object Request Broker
The ORB, also referred to as an item bus, enables clients to make application calls to the server. It acts as an agent to transmit operation invocations in a distributed environment. The ORB handles communication, marshalling and unmarshalling of parameters with each request to enable CORBA servers to correctly receive request details.

The delivery of requests is performed by the ORB Core using IIOP over TCP.

The ORB also manages the interface repository (IFR), a standardised database of IDL interface definitions. From the client's perspective, the ORB provides interface definitions which enables it to construct invocations against Objects.

The ORB Interface is an abstraction for the client and server to initialise and shutdown the ORB. It can also be used to convert Object references to strings and to create argument lists for requests made through the dynamic invocation interface (DII).

Portable Object Adaptor (POA)
The POA is responsible for assisting the ORB in delivering client requests to Objects. Firstly, it will carry out the activation and deactivation of Object instances when transmitting a request to the server. This is performed for memory conservation. The POA will deactivate and activate as according to a policy that is set by the developer for flexibility in memory requirements.

The POA is also responsible for generating Object references as used by the client. This is then interpreted by the POA when received in a client's invocation request. The POA will also work with the skeleton to invoke operations on the server side.

Stubs and Skeletons A stub and skeleton can be thought of as a local proxy for the client and server of the CORBA Object. They provide a mechanism for invoking an Object reference.

The stub and the skeleton are effectively files that are produced by the IDL Compiler. Both files contain code which is used for operating invocations by the methods specified in the IDL.

On the client's side, the stub will contain code that is generated during the client's build process that is used when sending the request. This compilation of code enables the request to be interpreted once received by the server.

On the server side, the skeleton acts as a pointer for the client's request to the appropriate code in the Object implementation to satisfy the request. The skeleton is therefore connected to both the ORB and the server.

Stubs and Skeletons are used by clients and servers respectfully with the ORB. A stub is used to send requests via the ORB which is then interpreted by the skeleton from the server regardless of the client's programming language or vendor. The interface defines the operations and types the object supports and therefore the requests it handles. The stub and skeleton facilitates the communication between the client and server through the ORB. The stub will marshal application parameters into a common data-level representation. The Skeleton will demarshal the data back to the application-level parameter.

The stub will forward the call to the ORB to be transported to the server. At the server side, the call is routed with an Object adaptor. The Object adaptor passes the call to the skeleton.

Both the stub and the skeleton have dynamic elements. The dynamic invocation interface (DII) for the stub and dynamic skeleton interface (DSI) allow the ORB to deliver requests to the servant that has no compile time knowledge of the IDl interface it is implementing.

Putting it all together

Now that we have touched on each of the components, we can begin to appreciate the many steps taken by our client, ORB, and server when we submit a request upon a remote Object with CORBA:

1. The CORBA Server is responsible for registering Object-name bindings in the naming service. 2. The CORBA Client queries the naming service for an Object reference it can create a request upon. 3. Each Object will contain an IDL that the client can use to understand structure and methods for invocation. The Client submits a request. 3. The IDL Compiler converts the client's request into a stub that is sent via the ORB to the server. 4. The ORB Interface will ensure the ORB is initialised for transport. 5. The ORB Core will deliver the request to the server via IIOP over TCP 6. On the server side, the POA will ensure the Object is activated, and submit the request to the Skeleton to be invoked. 7. The Skeleton demarshals the request for the server to interpret and invoke. 8. On the CORBA server, a response can be returned for the client to receive.

Summary

This blog has taken a look at how CORBA distinguishes itself as one of the most popular middleware options to distributed applications. We have then reviewed the many components that enable it to work. This high level view has demonstrated the complexity of the system and how a single request from a client is dependent on multiple transformations, references and transmissions before the server can invoke and respond with a result.

CORBA's interoperability make it a dynamic solution that can be implemented it a diverse array of systems; however with this comes a heavy technical understanding that is required to extract an optimised solution. I hope you have been able to follow the coverage of this blog well, however I have merely scratched the surface of CORBA where other components such as servants and dynamic invocations can be a good starting point for further research and detail.

Commodity Options: the vanilla and the exotic

Tue, 22 Aug 2023 00:00:00 GMT

Key Takeaways
• Understand what an options contract is and how it can be used.
• Explore exotic options and the various benefits they offer.

Introduction

My previous blog took at a look at forwards and swap derivatives in general with an application to commodity derivatives. This blog will build upon the last blog to look at the different types of option derivatives available within the context of commodities.

An option contract

If we take a look back at the forward contract, upon agreement, the buyer will have an obligation to purchase the exact amount of the commodity upon maturity at the agreed strike price from the seller. While the contract is not often seen to maturity, there remains an outstanding obligation for the contract to be exercised. An options derivative differs as there is no such obligation. A common definition you will see for an option is:
An option provides the right, but not the obligation, to buy or sell a asset at an agreed upon price and quantity at maturity.

Let's take a look at what this means for both sides of the deal.

How option contracts work

An option contract works as such:
1. The contract is purchased by the buyer from the seller for a fee.
2. The contract will specify which asset can be bought/sold, the strike price and volume.
3. The contract will have a maturity date, and depending on the type of contract it is, the buyer can then exercise the contract.

The options contract has a few variables to the buyer, so let's take a look at these:
1. If the contract is to purchase a security, it is known as a 'call' option. If it is to sell a security, it is known as a 'put' option.
2. A 'European' option will only allow the buyer of the option to exercise it at maturity, whereas an 'American' option will allow the buyer to exercise at anytime until maturity. Lastly, a 'Bermudan' option will allow the holder to exercise at a set of pre-agreed dates (such as the 28th of each month).

An option's value will continuously dip and rise in intrinsic value to the holder as the market price of the underlying security changes. For example with a call option, if the price of the security increases beyond the strike price, the option holder can exercise it (if it is American) to make a profit. Exercising the option to buy at a the lower market price will create a profit as they can (in-theory) sell at the higher value in the market. For a put option, the option holder will be able to make money if the value of the security drops, as they are able to then sell the security for a price that is higher than the market price. For both of these scenarios, the option holder are considered 'in the money', where they can profit from exercising their right on the option.

One small point to remember is the fee (aka premium) paid by the buyer to the seller of the option to have the right to buy or sell the asset. As a result of the fee, the option buyer will only be in the money once the strike price has surpassed (for calls) or fallen below (for puts) the market price combined with the option premium. The two graphs below illustrate how a bought call and put option can move in and out of the money as the market price changes.
For the call option, the premium is £10, and the strike is £100, therefore break even is at £110 for the market price.

Why Options are useful

The optionality of option contracts make them useful in multiple ways for hedging risk and taking speculative positions on the market.

The first way an option can be used is to take exposure in an underlying security. If you expect the price of a commodity to rise, you can purchase it in the market with the expectation to sell at a higher value in the future. This exposes you to the complete price movement of the commodity, both upwards and downwards. A future can also be used to obtain exposure. On the other hand, purchasing a call option will provide similar exposure to the commodities price movement, but with a limited downside equal to the premium paid for the option. Therefore, the investor can take a more precautionary position on the commodity while obtaining exposure to it's expected rise in price.

An option can also be used as an asset itself. The premium paid for the option can be reflective of the current market position, expiry and volatility of the commodity. A trader can then sell their right to exercise the option in the market for a greater premium.

Options can be used to hedge the risk of a portfolio should there be adverse movements in the underlying securities price. For example, if a portfolio consisted of copper, they would be completely exposed to a price rise (and fall) of copper. By buying a put option, they will be paying a premium, but guaranteeing a price that they can sell copper should it fall in price.

Exotic Options

Exotic options capture a slightly different, and more complex, way an option can be altered to see a different purpose for the investor and seller.

Binary Option

A binary option, also known as a digital option works where the buyer of the option pays a premium to the seller and receives a fixed pay out if the price of the underlying security reaches a specific value, known as a barrier.

The buyer receives a fixed return if the option is exercised, regardless of the market price of the underlying, but provided it has reached the barrier price.

Barrier Option

Takes deep breathe...

A barrier option itself is pretty complicated, made more confusing as we have just mentioned a 'barrier price' in the binary option. This naming is purely coincidental. Now let's take a look at the barrier option.

Unlike an ordinary option, a barrier option will have a condition attached to it where the option contract is either granted or cancelled when the underlying security reaches a specific price point, known as a trigger.

This means we start with the investor either already owning an option, with the possibility that it will be cancelled when the trigger is reached, or the investor not already owning the option, and the option being granted upon the trigger.
When an option is cancelled, it is known as a 'knock out', and when it is granted, it is known as a 'knock in'.

The placement of the trigger price can be in one of two places: either in the money or out of the money, depending on the option type. A trigger that is out of the money is known as a standard barrier, and one that is in the money is known as a reverse barrier.

So we have 4 different types of barrier options: a standard knock in, a reverse knock in, a standard knock out and a reverse knock out.

A call barrier example

Let's take a call option with the below values:
Strike: £120
Market Price: £100

This means the owner of the call can buy the asset at a price of £120, presumably exercised when the market price is above that.

A standard knock in barrier would place a trigger out of the money, and the option would be granted. Therefore the trigger could be £90. This is known as a 'down and in' barrier, as the price goes down and the option is granted.

Similarly, a reverse knock in would have a trigger in the money, so £125 for example. This is an up and in.

We can also have a standard knock out where the trigger is out of the money, and the option is cancelled, so the trigger would be £95 and this would be down and out. A reverse knock out would see the option cancelled with a trigger of £125 for example.

A put barrier example

The strike values will differ slightly for a put option, which grants the right to sell the option. Let's use the below values:
Strike: £90
Market Price: £100

This contract will mean the owner of the put can sell the underlying security for £90, presumably when the underlying value of the security has fallen below that.

A standard knock in will see the trigger placed out of the money for the option to be granted. So this value would be £105 thus creating an 'up and in' trigger. This is unlike the call option's standard knock in which is down and in.

A reverse knock in would have an in the money trigger (£85) and would therefore be down and in. A standard knock out would have the value at £105 which is an up and out, with a reverse knock out with a trigger of £85 would be down and out.

The additional condition that comes with a barrier option means they are less likely to be exercised as more events (barrier price being reached for knock-in and not being reached for knock out) are required to occur before the option can be exercised. As a result, the barrier option tend to be cheaper than a vanilla option. The customisability of barrier options also mean they are traded over-the-counter and not through an exchange.

Spread Option

A spread option will use two underlying assets and pay off the difference between the two to an agreed strike price. A call option will pay off if the spread is greater than the strike, and a put will pay off if the spread is less than the strike.
Therefore we can use the below formulas to calculate the return on a spread call and put:
Spread call return = max (p1 - p2 - strike, 0).
Spread put return = max (strike - p1 + p2, 0).
Where p1 and p2 are the prices of the 2 assets in the option.

Spread options are popular derivatives within commodities. They provide protection against 2 underlying prices such as the cost of an input (crude oil), and the price of it's output (jet fuel). This can be particularly useful for a refiner who would like to protect the spread they earn from converting the crude oil to jet fuel.
Spread options can offer more variety in their application, where underlying securities can differ by location (thus protecting geographic price ranges) or futures (for protecting against delivery date spreads).

Average rate option

The last exotic option we will look at are average rate options. Also known as an 'avro', the average rate option is the most popular exotic option for commodities. Unlike a vanilla option, an average rate option will use the underlying average price of a security over a time period to determine the market price.

Avros are specifically popular with commodities due to the nature of the contracts that are constructed for physical commodities. Commercial contracts for physical commodities will use average rates. The average rate serves both the buyer and seller from avoiding the highly volatile prices of the commodity, be it the high for the buyer, or the low for the seller.

The relationship that exists between physical and financial commodities derivates is extended towards the payoff on the derivatives that are also used in commercial contracts. Therefore the average rate option is helpful in avoiding a mismatch between the cashflows of the derivative and the physical contract.

The lower price volatility of an average rate option will also reduce the premium required to buy it. Less volatility of price will reduce the opportunity for the contract to be exercised. This is known as a reduced 'implied volatility' of the contract.

Summary

The option contract differentiates itself from other financial derivatives through the optionality of it to be exercised. Further flexibility is provided in it's ability to be used for buying or selling a security, and for the variations of exotic options that exist. This can result in controlled payouts, average price monitoring and using the spread between two securities.

A major opportunity with option contracts lie in the limited downside they require upfront from the investor in the premium, but the potentially unlimited upside they can provide in exposure to the security. This factor makes option contracts popular across asset classes and has thus lead to the creation of exotic types and strategies to incorporate them to a portfolio.

Commodity Derivatives: Futures, Forwards and Swaps

Sat, 12 Aug 2023 00:00:00 GMT

Key Takeaways
• A futures contract allows a trader to guarantee the price and quantity of a commodity for delivery at a date in the future.
• A forward differs to a future by where it is booked and provides greater flexibility in it's terms.
• A swap enables a counterparty to exchange cash flows between different price indexes or a fixed rate.

An intro to financial derivatives

My last blog was the 2nd in my series on commodities where we took a look at the different ways a financial institution can influence commodity markets, and how a commodity can become traded. This blog will extend upon traded commodities to look at the main financial derivatives that can be created and the benefits they present.

This blog will take a look at forwards, futures and swaps; my next blog will look at final piece of the puzzle, option contracts, which are a little more complicated. I will generally use commodities as examples in these explanations, however note that financial derivatives are not limited to just this asset class, so where you see 'commodity', you can replace it with 'asset'.

The future contract

A future is a contractual obligation between two counterparties to purchase or sell a given commodity, at a specific volume and price (known as the futures price) for delivery at a specified date in the future. The buyer of a future contract will benefit as they can guarantee the price of a commodity to be delivered at a date in the future. The seller in the future contract benefits from the guaranteed sale of a specific volume of commodity at a price in the future. Both participants benefit by removing the price volatility for the asset in the sale at the future date.

One example may be an airline, who might like to use a future contract to fix the price of jet fuel for the future. By fixing the price, the airline can better estimate the costs of flights and set prices of tickets. The supplier of jet fuel has a guaranteed sale, and the airline have a guaranteed price that is not impacted by the volatility of market price for jet fuel in the time-frame of the contract.

While a future has many practical benefits, they are not often seen to expiry as a financial derivative. Roughly 95% of futures contracts are settled before expiry, and the delivery of the asset is never made. The two counterparties in the contract will settle a difference in payment between the market rate and the futures price, known as 'marking to market'. For example, if a futures contract is made to buy one ounce of gold at £300 per ounce, and at expiry, the market value of gold is £310 an ounce, the seller of the contract will pay the buyer £10, as they are in theory buying gold at £310 in the market, and selling it (at the futures price) of £300.

Over-the-counter vs exchange trading

A future and a forward contract differ in two main ways: where the trade is made and the terms of the contract. A futures contract is placed through an exchange that uses a central clearing house (CCP). The CCP acts as the buyer to the seller and the seller to the buyer, thus guaranteeing the terms of the trade to both counterparties, even in the event of a default from one of the counterparties. Participants in the contract will pay a small deposit, known as margin, to the clearing house that enables them to guarantee the terms. With the CCP, the market of participants are protected from potential knock-on effects of a single counterparty defaulting, and thus causing other counterparties unable to fulfil their onward financial obligations.

A further difference between a future and forward lies in the terms of the trade, and what can be customised in the transaction. Exchanges operate in a uniform way that specify the terms of the contract for individual commodities. For example, this can control the trading units used for a commodity, along with how a price is quoted (currency), how the trade is settled, which months delivery can take place and margin requirements.

Within commodities, the standardisation of trading can be hugely beneficial. This is because commodities can exist in many different forms to introduce other variables on the commodity such as commodity make-up, refiners, approval of content and delivery volumes.

With a forward contract, the terms of the agreement are made between the two counterparties upon the same factors the exchange standardises. This flexibility allows each party to hedge risk more specifically, or to obtain exposure that otherwise wouldn't ordinarily be available in the exchange.

The margin requirement of the exchange had previously been an outstanding difference between the forward and future, however due to evolving regulatory requirements, the margin requirements between OTC trading are now relatively similar to that of an exchange.

Swaps

A swap contract creates an exchange of cash flows between two parties. The swap is typically made for long periods of time (more than 1 year) and the underpinning cashflows between the two will be based upon different price indexes.

A typical swap will involve the exchange of a fixed rate with that of a variable rate, thus allowing one party to guarantee outgoing cash flows and providing the other party with exposure to a floating rate. The exchange of cash flows will take place at the agreed upon tenor until expiry.

The notional of the trade will determine the payout that is made between the counterparties. For example, bank A and bank B may enter into an interest rate swap. Bank A pays bank B 5% interest and bank B pays the national bank rate. The notional for the swap may be £100 with tenor of 3 months, and expiration in 1 year.
So if after 3 months the bank rate is 4%, bank B will pay bank A 1% (5% - 4%) of £100. If, after 6 months, the bank rate is 7%, bank A will pay bank B 2% (7% - 5%) of £100. These exchanges of cash flows will take place until expiration.

Within commodities, the use of a swap can vary by nature of the commodity. For example, gold has a floating lease rate (the interest rate on a gold-backed loan), and can therefore be swapped with a fixed lease rate. You can also have swaps on the actual floating price of the commodity itself.

Swaps are typically used for hedging risk; if a bank is exposed to the underlying movement of a price index, they can fix their exposure by entering into a swap for a fixed rate. On the contrary, another bank can obtain exposure to the floating rate, possibly for speculative purposes, while limiting their downward exposure against the fixed rate.

Summary

This blog has taken a brief look at the fundamental financial derivatives of futures, forwards and swaps, all of which are used by financial institutions to manage their exposure and to speculate on the underlying price movements of commodities.

My next blog will complete our view of financial derivatives as I look into the options contract, and the various ways it can be used with commodities.

How financial institutions use commodities

Sat, 05 Aug 2023 00:00:00 GMT

Key Takeaways
• Understand the various roles financial institutions play with commodities.
• Understand why some commodities are traded and others are not.

Introduction

In my last blog, I took a high-level view of the various commodity subdivisions and how they are used in the real world. I then touched upon the importance of price discovery within commodity markets and the role of the price reporting agencies. This blog will extend to help us understand the various financial institutions that participate in the commodities market, as well as the requirements necessary for a commodity to become traded.

The role of financial institutions with commodities

Amongst investment banks, there are many other types of financial institutions that play a role within the commodities markets, including: commodity trading houses, hedge funds, and real money accounts.

Investment Banks
Investment banks typically act across a variety of services with commodities. A 'full service' investment bank will exist to deal with financial derivatives, but also to act as a counterparty in the physical trade of the commodity. As a result, an investment bank can play a pivotal role in the delivery of commodities with their cash-rich position and ability to hedge risk at scale. Some examples of how they can be influence the physical commodity market include:
• Owning inventory for refiner participants to reduce their operational capital requirement.
• Purchasing storage facilities or pipelines to transport and store commodities between market participants.
• Provide financial support to supply chain participants.

Commodity Trading Houses (CTH)
A CTH originally existed as a middle-man between consumers and producers of commodities, earning a profit from buying and selling commodities at wholesale. CTH have since evolved to own parts of the supply chains. They therefore solve a similar problem that investment banks do with financing participants and facilitating supply chains to remain operational.

Hedge Funds
Hedge funds operate to serve a smaller pool of investors when compared to investment banks. They tend to use actively managed strategies with the goal of generating a profit in all market conditions. As a result, fees tend to be higher to the investor.

The smaller size and specific goal-driven business of a hedge fund will more often operate with the financial derivatives side of commodities, and will employ event-driven strategies to maximise returns on market-specific events that impact commodity prices.

Real Money Accounts
Some financial institutions do not have a similar capacity to borrow money to finance larger operations like an investment bank. Such accounts include those for insurance and pension funds as well as retail and private banks. They will therefore use commodity derivatives as an investment vehicle for purposes such as hedging risk and seeking diversification in their portfolios to match the goals (risk profile and return) of the fund.

Why not all commodities can be traded

Now that we understand the different types of institutions that can trade commodities and how their functions within the market can differ, let's take a look at the properties of a commodity that enable it to be traded on an exchange.

When we think about a commodity that is traded, it will typically sit in an exchange where buyers and sellers can efficiently receive a price for their desired product. Sometimes, where markets are insufficiently liquid, an investment bank can also step in and act as a market-maker: to be a buyer to the seller and seller to the buyer. Either way, a price for the commodity is transparent and accepted across all market participants of the exchange.

The challenge that commodities introduce to us align with their timelines for delivery and the abundance of grades and varieties that they can exist within. A single commodity (such as gold) can differ by shape, grade, source, transportation and percentage makeup. As a result, a 'spot price' might not exist for each individual commodity subtype, and instead is derived from a reference point price for the commodity instead.

Defining a reference point...

In this section, I am going to talk a little bit about the prices of commodities and the markets they trade in. Before I do that, let's go over a few definitions.
A spot price is the price of an asset, derivative, currency or any product at the current market rate, and for 'immediate' delivery. You can think of a spot price as the price you pay in your supermarket for any item on the shelf. You are paying £1 for a bag of bananas as they are immediately sold and owned by you after checkout.
A spot market, also known as a cash market or physical market, is where assets, derivatives and currencies are traded for immediate delivery. In the spot market, delivery is not going to be instantaneous, but is often made 2 business days later (T+2).
A spot transaction involves a trader buying or selling a product in the spot market for the immediate delivery (T+2).

The unique nature of commodities and their supply chain for extraction, refinement, transportation and delivery, means the spot market for commodities will not often reflect the spot delivery of 2 days time. Instead, the spot price will often reference a point on the futures curve of the commodity, with delivery reflective of that date.

The emergence of a commodity as a traded asset in an exchange is achieved through price discovery in the market. If market participants are able to receive transparency for what price the commodity is traded, they will then be able to confidently use it for their intended purposes.

A further factor to a commodity becoming traded exists with it's requirement to be frequently traded. This may begin to sound like a chicken-and-egg scenario where higher trade frequency of a commodity can come from it's global need (through necessity), but is also facilitated by transparency of it's fair price. As commodities are traded more frequently, the demand for price reporting agencies to provide market-wide price transparency increases, and thus a commodity can establish itself upon an exchange.

As previously mentioned, the heterogeneity of commodities do not place an expectation upon PRAs to provide all price definitions of a commodity. They instead provide specific prices for a commodity that act as a reference point to derive further prices from. For example, a price may be given for a grade of 80% gold, where a 70% grade will be traded at a price that is derived below that.

Following the establishment of a commodity into an exchange, we can see market participants benefit with improved transparency and liquidity of the commodity for trading. This can also lead to standardisation for it's trading in futures and commercial contracts. Lastly, the price confidence across market participants for the commodity will lead to its inclusion with different financial derivatives.

Summary

Financial institutions can play varying roles in the commodities market. While commodities can behave like other asset classes to a bank, by providing a source for speculation and to hedge risk, a bank can also play a vital role in the operational existence of a commodity. The unique physical presence of the commodity asset class mean they present opportunities for specialist financial institutions to play a role in it's creation.

Price reporting agencies will provide price transparency into spot markets for commodities once it's frequency of trading has been established. The PRAs provide price transparency for benchmark grades and variations of a commodity where subsequent prices can be extrapolated.

An intro to commodity derivatives

Wed, 12 Jul 2023 00:00:00 GMT

Key Takeaways
• A small change in my job role and company.
• Commodity derivatives encapsulates both a financial and physical market.
• Price discovery in commodity derivatives plays a major role for the multiple participants in it's supply chain.

A new job and learning chapter..

It's been one year since I last wrote a blog. Since then, I started creating tutorials on the MEAN stack followed by short videos around backend fundamentals. This diversion away from more 'coding' focussed content has come about from a change in job.

In March 2023, I joined the Macquarie group in an opportunity that explores technical infrastructure alongside Java development. This new role will see me work upon the applications used across the bank's commodity desks and through the trade lifecycle.

So with that said, let's take a look at the bare bones of Commodity derivates, before I write deeper upon these topics in the coming months.

Commodity markets

Commodity markets, just like other asset classes, are classified into smaller segments that focus on specific products. What's unique here, is that commodities represent real, tangible products that are used in the world around us. There may be some overlap between markets, however the main ones are as follows:

Energy Markets
Energy markets include the trade of crude oil, refined products, power, natural gas, natural gas liquids and coal.

Crude oil fossil fuels are found between sedimentary rock under the Earth's surface. They are a combination of hydrocarbons (a chemical compound of hydrogen and carbon) formed over millions of years from the remains of animals and plants. Over the years, the remains are covered in layers of sand and rock to create the raw product under heat and pressure.

Crude oil can be transformed into many different refined products after it is extracted. These 'petroleum' products include gasoline, diesel, heating oil, jet fuels, waxes and lubricants.

Power refers to electricity as a tradeable commodity. Electricity is often measured in kilowatt hours, and is interchangeable to how it is generated. For example, electricity can be produced from coal or natural gas, and both of these are measured the same way, through kilowatt hours.

Coal is also a hydrocarbon fossil fuel, found in sedimentary rock. It can be burned to produce fuel or for generating electricity.

Natural gas is a gaseous hydrocarbon, mostly consisting of methane. While it exists in abundance under the Earth's surface, it's gaseous form means it must be transported via pipelines, or converted to a liquid for transportation. This conversion can add to the costs associated with it. Natural gas, once refined, can be used for electrical power generation, residential heating, and industrial production.

Natural gas liquids are components of natural gas that have been separated from the gaseous state. Liquids can be produced in the form of Ethane (used to create plastics), Propane (used for heating), Butane (used for synthetic rubber) and Isobutane (used for aerosols).

Industrial Metals
Industrial metals include copper and aluminium.

Copper is a malleable metal that also acts as an effective conductor of heat and electricity. It is most commonly used for wiring, lighting, motors, cables and electric generators.

Aluminium is one of the most widespread metals on Earth. It is soft and malleable making it useful for a variety of products including foils, window frames, beer kegs and electrical conductors.

Precious Metals
Precious metals include gold and silver. Gold is a durable metal that will not rust or erode with time. Gold is malleable and conductive, making it useful for carrying electrical currents in your smart-phone, but is more commonly used for ornamental objects such as medals and fine jewellery.

Silver is not quite as dense as gold, meaning it will be less durable, but is more malleable. Silver is historically used for photography due to it's ability to reflect natural light. In the modern day, it is used for mirrors, tableware, batteries and alloys amongst many other ornamental objects.

The ownership of these precious metals extend beyond their use to create products, and are also held for hedging against inflation. We will see more on this in a future blog.

Agriculture
Agriculture include 'soft' commodities that are grown, such as grains and coffee, as well as livestock.

Grains are widely consumed and include wheat, corn, soybeans, rice and oats. Livestock includes animals that are grown for meat production, accessory production (such as wool or worm casting) or human consumption (such as dairy products, honey or leather).

Speciality Commodities
The last commodity subdivision captures unique products and derivatives such as forest production, carbon emissions, weather and freight.

Forest production includes pulp and recovered paper. Trees can be used to create wood pulp, that is then used to create paper.

Carbon emissions are government-granted allotments of carbon dioxide outputs that are granted to companies. The carbon credits enable the companies to emit a specific number of emissions, therefore those below their threshold (such as Tesla) are able to sell their credits to enable other companies to manufacture more.

Weather as a derivative is interesting...This can be used to mitigate the risk of unfavourable weather in specific markets. Weather derivatives can be used to value good and bad weather in specific regions. Entities can then trade these to hedge the risk from unfavourable weather to their business.

Lastly, freight derivatives are derived from the current and future value of freight rates. Freight is used to transport commodities and can be used by supply chains to hedge to price volatility risk associated with transport.

What makes commodities unique

The commodities market differentiates itself from other asset classes as they operate in both a physical and financial market. There are financial derivatives for commodities such as options, futures and swaps, however the underlying products represent physical products that must be grown/extracted, transported and stored along an agreed timeline.

The physical market consists of 3 main participants: producers, refiners and consumers. Producers are responsible for sourcing the raw commodity, refiners will transform the commodity into a consumable product (such as transforming oil into gasoline), and the consumers make use of the product. These products, similar to stocks, come in different grades by quality. The supply chain is extended further to include participants for purpose-built storage facilities, and transportation to account for the transfer of ownership for the commodity.

Unlike financial derivatives, commodities cannot be created and placed in a deal book in an instance. There is a timeline for it's creation, production and delivery. This timeline exposes multiple participants to a price risk. For example, the farmer who grows grain will be exposed to future demand and subsequent price volatility. The refiner will be exposed to the cost of delivery and storage to their facility, and the consumer will be exposed to all of these compounded fees.

As a result of the vast number of participants and timeline for product delivery, price discovery plays a major role to all members of the commodities supply-chain.

A common challenge in Commodities

As with all asset classes, commodities presents their own unique challenges for market participants and financial institutions. One challenge I will repeatedly refer to within my blogs are price discovery.

Price discovery is a term that will enable all market participants to understand what a 'fair price' is for the product they are buying. Each participant in the supply chain will be exposed to unique price factors that can influence the value of the commodity as it is extracted, refined and transported. Some factors can include operational costs such as credit protection and transportation and storage fees. As well as the availability of the commodity in the region it is sourced, and also at a grade quality that is expected.

Commodity markets are vast with different qualities of each product, meaning markets are not always liquid enough to provide natural price discovery via an exchange.

A solution to this challenge exists in the form of a price reporting agency (PRA). This is an independent research provider that provides pricing indexes for commodities, thus bringing price transparency to physical commodity markets. The price indexes are available from the agency through a subscription and can be used to understand prices across entire commodity segments.

Popular PRA's include IOSCO and Platts, who are responsible for deriving commodity index markets. These indexes can then be used for commercial contracts and wholesale commodity deals.

Summary

This blog has taken a high level view to the complexities of commodity derivatives; their vast array of products, participants and markets introduces many opportunities for financial services to speculate and hedge risk for their clients and stakeholders.

What makes commodities so valuable to learn about is their representation in real life. The energy sources and raw material that are traded are all around us. This extends to infrastructure, devices, transportation, food and homes; all of which are products of a commodity that has been grown, extracted, refined, transported and stored, all with the help of financial institutions.

I look forward to writing my next blog, where I look at the various roles financial institutions play for commodity market participants, and the difference between traded and non-traded commodities.

Authentication with Mongo DB Part 5

Fri, 27 Jan 2023 00:00:00 GMT

Auto-authentication

This video completes our mongodb authentication functionality by creating logout functionality. We will also explore the local storage capability of our browser that will allow us to persist our web token to enable automatic authentication when a user leaves the web application.

Link to learn more about local storage: https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage

Join along with this tutorial with the source code below: There you will find a before and after folder that contains the code at the stand and end of the video. Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/mongodb-auth-5

Authentication with Mongo DB Part 4

Fri, 20 Jan 2023 00:00:00 GMT

UI Authentication enhancements

This video will enhance the user interface and experience with authentication. Visual components will only become available if a user is authenticated to help the user understand what they can and cannot do. We will also create a route guard to prevent any unwanted navigation from users through the URL.

Authentication with Mongo DB Part 3

Mon, 09 Jan 2023 00:00:00 GMT

Token Verification

This video will look at how we can automatically enrich http requests with a web token once it has been generated in TypeScript. The enriched HTTP request can then be used for to verify user requests with Express JS before they are completed.

Authentication with Mongo DB Part 2

Sat, 31 Dec 2022 00:00:00 GMT

Logging in and creating web tokens

Learn how you can create web tokens with the jwt package in express js. This video will enhance our MEAN application to verify the details of a user when they submit the login form, and return with either a generated web token from the jwt package, or with an exception for an incorrect username or password. This video is part 2 of a series of videos to create authentication with Angular and MongoDB.

Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/mongodb-auth-2

Authentication with Mongo DB Part 1

Mon, 05 Dec 2022 00:00:00 GMT

Authentication with MongoDB Part 1

This video will look at how we can sign up users to our application and to store their details with MongoDB. Our application will allow users to sign up with a new sign up form built within Angular. We will then connect the form with an express endpoint to MongoDB. We will also verify accounts are unique and apply a hash to the password to help protect user information.

Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/mongodb-auth-1

Enhancements with Java 14

Sat, 29 Oct 2022 00:00:00 GMT

Java 14 Enhancements

This video will look at the two key features introduced with Java 14 for the developer. These include the improved syntax for switch statements, and the detailed feedback provided with a null pointer exception.

How to verify static method calls

Sat, 22 Oct 2022 00:00:00 GMT

How to verify static method calls

Learn how you can verify that a static method has been called with PowerMock! This video will look at why Mockito is unable to verify static method calls. We will then review the PowerMock dependencies required, and write a method that verifies if a static method is called and the number of times it is invoked.

Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/VerifyStatic

How to use @RepeatedTest

Sat, 15 Oct 2022 00:00:00 GMT

How to use @RepeatedTest

The RepeatedTest annotation enables you to run the same test multiple times. This video will look at why you may want to use the annotation, and the various ways you can customise it to provide detailed feedback in the console and within the IDE.

Create a full-stack MEAN application

Sat, 01 Oct 2022 00:00:00 GMT

Create a MEAN Application

Build a full-stack mean application from scratch, step-by-step with this video! We will create a web application that can be used for storing, updating and deleting diary entries. This application will use Angular (and bootstrap) for the front-end, node and express JS for the backend server, and mongo db for data persistence!

Link to source code: https://github.com/4neesh/MyDiary

🕒 Timestamps 🕒

00:41 - Setting up Angular 03:22 - Adding Bootstrap 3 04:27 - Creating a HTML Table 05:50 - Angular Shared Services 07:55 - Populating data from an array 09:02 - Adding action buttons to each entry 09:47 - Deleting diary entries 11:14 - Angular Subjects 13:39 - Angular Reactive Forms for adding entries 20:07 - Editing a Diary Entry 26:12 - Splitting the Application 29:11 - Creating a node server 33:12 - Getting started with Express 37:03 - Using next() 37:56 - Registering URI's 38:44 - Migrating Data from Angular to Express 40:41 - Angular HTTP requests 48:15 - Making POST requests 52:22 - Making DELETE requests 56:22 - Making PUT requests 01:00:00 - Setting up MongoDB 01:02:30 - Using Mongoose 01:06:10 - Connection our app to MongoDB 01:08:11 - Saving to MongoDB 01:08:56 - MongoDB Shell 01:10:32 - Fetching data with MongoDB 01:13:33 - Deleting from MongoDB 01:15:43 - Editing data with MongoDB

Creating a MEAN application: Part 5

Sat, 24 Sep 2022 00:00:00 GMT

MEAN Application: Part 5

Part 5 of this MEAN Application series will complete our application with a MongoDB database. We are going to create an account with MongoDB and a cluster to store our data within. We will also use the mongoose package to define a schema for our data, before connecting to MongoDB and persisting the data. We will complete the full-stack functionality by enabling POST, PUT and DELETE functionality with the database and application together.

Link to source code: https://github.com/4neesh/MyDiary

🕒 Timestamps 🕒

00:00 - Introduction 00:26 - Setting up MongoDB 02:54 - Using Mongoose 06:36 - Connection our app to MongoDB 08:38 - Saving to MongoDB 09:22 - MongoDB Shell 10:58 - Fetching data with MongoDB 13:58 - Deleting from MongoDB 16:10 - Editing data with MongoDB

Creating a MEAN application: Part 4

Sat, 17 Sep 2022 00:00:00 GMT

MEAN Application: Part 4

Part 4 of this MEAN Application series will build-out our Express middleware to provide complete backend functionality to our application. We currently use the backend service to obtain the diary entries for our app. This video will expand the functionality so we can perform PUT, POST and DELETE requests to our backend server so we can update, edit and remove entries. We will enhance the data service in the Angular single-page application to make HTTP requests to the Node backend service where data is stored.

Link to source code: https://github.com/4neesh/MyDiary

🕒 Timestamps 🕒

00:00 - Introduction 00:14 - Making POST requests 04:26 - Making DELETE requests 08:24 - Making PUT requests

Creating a MEAN application: Part 3

Sat, 03 Sep 2022 00:00:00 GMT

MEAN Application: Part 3

Part 3 of our MEAN application series will incorporate Express.js into our application. Express will allow us to add valuable middleware into our backend service to register different endpoints for our backend service to provide unique data responses to. We will begin to connect our Angular static webpages to the backend by registering URI's with express and by formatting and returning data. In Angular, we will use the http package to make back-end calls to the node server to retrieve the data. We also dive into CORS policy and what it means to our application.

Link to source code: https://github.com/4neesh/MyDiary

🕒 Timestamps 🕒

00:00 - Introduction 00:22 - Getting started with Express 04:08 - Using next() 05:03 - Registering URI's 05:50 - Migrating Data from Angular to Express 07:48 - Angular HTTP requests

Creating a MEAN application: Part 2

Sun, 28 Aug 2022 00:00:00 GMT

MEAN Application: Part 2

This part of the MEAN series will create the backend server using Node.js. We will start by testing node and then creating a server that will respond to all requests using the methods that come out of the box with the http modules from node.

Link to source code: https://github.com/4neesh/MyDiary

🕒 Timestamps 🕒

00:00 - Introduction 00:15 - Application overview 03:13 - Creating a node server

Creating a MEAN application: Part 1

Sat, 20 Aug 2022 00:00:00 GMT

MEAN Application: Part 1

In this 5-part series we will look at how to create a MEAN application. A MEAN application uses Mongo DB, Express JS, Angular and Node JS to create a full-stack application. This first part of the series will create the bare-bones of our angular application. We will create a personal diary that displays our posts in a table. We will be able to add, delete and update posts within the application and to store the data in an array.

Link to source code: https://github.com/4neesh/MyDiary

🕒 Timestamps 🕒

00:00 - Introduction 00:41 - Setting up Angular 03:22 - Adding Bootstrap 3 04:27 - Creating a HTML Table 05:50 - Angular Shared Services 07:55 - Populating data from an array 09:02 - Adding action buttons to each entry 09:47 - Deleting diary entries 11:14 - Angular Subjects 13:39 - Angular Reactive Forms for adding entries 20:07 - Editing a Diary Entry

From Zero to Hero: Mockito & JUnit Course

Sat, 13 Aug 2022 00:00:00 GMT

From Zero to Hero

This video is a compilation of individual JUnit and Mockito tutorials published on this channel.

Follow the course to learn all about core JUnit and Mockito skills, use the below chapters to navigate to a specific part of the course if required:

Link to source code:

https://github.com/4neesh/YouTube-Channel

Syntax changes from Java 12

Sat, 06 Aug 2022 00:00:00 GMT

Java 12 Syntax Changes

This video covers the main syntax changes that are introduced to Java 12. We will look at how the String class, Files class and Collectors class have all changed to offer new functionality and easier ways to approach unique problems.

How to duplicate and merge a stream with Teeing

Sat, 30 Jul 2022 00:00:00 GMT

Duplicate, collect and merge: How to use Teeing

Take a look at the new method 'Teeing' introduced to the Collectors class with Java 12. Teeing allows us to duplicate a stream, to collect the streams separately, and to merge them anyway we choose to in a single function. We will walk through a demonstration where we use teeing to extract the maximum and minimum values of a collection and to merge them into a map. Teeing is a useful method for specific use cases where we find ourselves streaming a collection multiple times and combining the resultant collection with each other.

Files.mismatch with Java 12

Sat, 23 Jul 2022 00:00:00 GMT

A new way to compare Files with Java 12

Take a look at the new mismatch method introduced to the Files class with Java 12. Files mismatch will help us to determine the exact index position where the content of two files will differ. This video will use an example of two files to see how the method can be used to compare the content of two files, and to see what is returned with identical and different files.

String Method updates with Java 12

Sat, 16 Jul 2022 00:00:00 GMT

Java 12 String enhancements

Take a look at the 2 new methods introduced to the String class with Java 12. String.indent() will allow you to add and remove empty characters to the start of each line of a String. String.transform() will allow you to pass in a functional interface to accept a String and return a String for a convenient way to update a String.

Basics of Angular Routing

Wed, 15 Jun 2022 00:00:00 GMT

Angular Routing

Take a look at how you can configure routes with the Angular Routing Module. We will create multiple components and link them to individual routes. We will then use the RouterLink to create buttons that take us to those routes, before completing the tutorial with a demonstration of how to create nested routes.

Introduction to Repos

Mon, 13 Jun 2022 00:00:00 GMT

Key Takeaways
• What a repo is and how it works.
• The benefit of a repo and its importance to financial markets.
• How central banks are able to control bank reserves through policy and the repo market.

About this blog

I originally expected this blog to be fairly short; while the Repo market is worth trillions of dollars in a global market, it consists of a fairly simple exchange (as we will later see). The elevated importance of Repos as the 'pipelines of the global financial system' will take this blog through a more interesting turn to understanding monetary policy and as a means for central banks to influence the behaviour of the repo market participants.

What is a Repo?

The term 'Repo' is short for repurchase agreement. A repurchase agreement is commonly referred to as a 'pawn shop' for banks, let's look at why...

A repurchase agreement is used by banks to trade debt for cash. With this in mind, this means banks need cash and are happy to trade the high-grade securities they own for it. In a single line, a Repo is a 'short term collateralised loan', where emphasis on 'collateralised' should be taken. The debt traded in a Repo tends to be of high credit quality, such as a government bond. Repurchase agreements are typically traded on an exchange; let's look at an example:
Bank A holds a government bond, Bond A, but needs to raise some short term capital for it's own requirements. One way Bank A can do this is by selling Bond A on the market, but what if Bank A would also like to keep Bond A in it's portfolio for the longer term? Well, Bank A can sell Bond A and then repurchase it from the buyer after an agreed period.

On the other hand, Bank B has excess capital sitting in its reserves, and would like to make a return on that capital rather than have it sit in its books earning nothing. Let's take a look at how Bank A and Bank B can create a mutually beneficial agreement:
Bank A sells Bond A to Bank B. Bank B transfers £100 cash to Bank A and receives Bond A. Bank A only needs the £100 cash overnight, so they agree for Bank B to sell Bond A back to Bank A the next day. Bank B will sell this bond for a pre-agreed price of £101 to realise a £1 profit. Bank A is satisfied that they have ultimately obtained their £100 capital for a cost of £1.

What we have is a repurchase agreement; a short term collateralised loan between two banks. The loan is collateralised because Bank B is effectively lending Bank A the money they require, and in-turn receive Bond A as collateral. If Bank A is unable to repurchase Bond A (and thus default on the repo), Bank B won't face a loss as they can simply sell Bond A in the market.

The percentage difference between the price Bond A is sold for and the price Bond A is repurchased for is known as the repo rate. In our case, Bond A is sold for £100 and bought back for £101, creating a repo rate of 1%. As a repo is effectively a return on a low-risk, short term collateralised loan, a repo rate can be considered a proxy for an overnight risk-free loan.

The Reverse Repo

The reverse repo is used with the intention of purchasing a security and reselling it at a higher price. A central bank may use reverse repos to inject capital into the market. They offer reverse repo's to eligible counterparties that can purchase bonds from them with an agreement to be repurchased at a higher price. The transfer of bonds from the government with cash from the banks will allow the government to increase their capital stores while draining the capital reserves of the banks. We will later see when reverse repos are used in action.

Repo Haircuts

The securities traded within a Repo tend to be of high credit quality and low volatility. Why?
The reliability of a repo trade is that the underlying security does not change value (to any significant degree) for the life of the trade. If the security holds it's value, then the participant purchasing the bond can be confident that in the event of default from the counterparty, they will be able to sell the security to not realise any losses.

The party purchasing the debt in the repo will want to protect themselves from any volatility of the underlying security price, and therefore may request a higher value security is transferred during the trade. As a result, rather than having a £100 bond transferred from Bank A to Bank B, we may have a £108 Bond transferred. This way, if the bond falls in value to £101, Bank B will still be able to realise the same expected profit of £1 from the repo.
The additional value of the security in the trade is known as the haircut.

An interesting story on repo haircuts..
In 2008, Lehman Brothers had found a unique accounting loophole regarding repos that allowed them to hide liabilities. During the life of a repo, the debt is not actually exchanged, but is only used as collateral. Where a haircut on a repo was large enough, Lehman Brothers were able to project the incoming cash as a sale of an asset that can be used to hide liabilities. As we know, all Repos are meant to be repurchased, therefore the 1st leg of the repo transaction (the 1st exchange) should not be treated as a sale, as the security is going to be repurchased during the 2nd leg. This is known as the 105 rule, read about it here.

Repos and monetary policy

So far we understand that repos are used by banks to control their cash balances, and that they typically hold high-grade securities as collateral to a short term loan. The instruments traded are typically government debt as the debt will hold it's value in turbulent markets and therefore requires less of a haircut on the trade.

The role government debt plays in the financial market enables governments to influence the behaviours of banks. For example, a government can issue new debt into the market to enable banks to directly achieve a given return on their excess cash holdings. Issuing new debt can therefore drain cash reserves from the financial markets as it flows from the banks towards the central banks.

Similarly, governments are able to inject capital into the financial market by buying back their debt from banks.

Reference Rates with Repos

While reading about the Repo market, I came across a recent market event that illustrated how the monetary policy of a central bank can shape the repo market. In September 2019, the secured overnight financing rate, also known as SOFR, had a huge spike creating headline news...

SOFR measures the cost of overnight borrowing of dollar-denominated repo transactions. SOFR is one of many overnight reference rates, others include SONIA for Sterling interbank money market trades, and EONIA for the Euro. These reference rates are calculated using real trade data for derivatives and aim to replace reference rates from LIBOR, which are based upon estimates. You can read more about LIBOR in my blog here.

The Repo Rate Hike of 2019

In September 2019 the SOFR rate spiked from 2.2% to 5.25%; the graph below illustrates the significance of the spike to historical trends:

Source: Data available on FRBNY public website.

Why the rate hiked

As we understand, the repo market is used to help banks obtain cash for their high-graded securities, typically overnight. The rate hike occurred due to a sudden rise in the demand for cash, and this is largely considered to be due to the timely occurrence of two events...

The first event were the Q3 taxes that were due for banks in mid September 2019. The tax deadline meant cash from the banks had been planned to flow to the US central bank, the Federal Reserve. As banks needed more cash on hand to pay the taxes, the demand rose.

The second event were the issuance of new investment grade securities by the treasury that also coincided around mid-September to the value of roughly $115 billion. The trading volumes of the new securities meant broker-dealers relied upon the repo market for the cash to fund the volumes. These two events meant cash within the financial markets were in large demand.

Bank reserve frameworks

Bank reserves refer to the amount of cash banks hold that is readily available for use. Governments use a minimum level of reserves target that they would like the banks to hold to reflect the market conditions and prospects. Prior to 2008, the Federal Reserve used a 'scarce reserves' framework where banks held only enough reserves needed for day-to-day activities. Banks were able to borrow from the repo market when cash was required and lend when they had too much. Following the financial crisis, the cash reserves required by banks had grown to a high degree.

From 2008 to 2014, Quantitative Easing had been used as a strategy to introduce a new cash supply within the system to support higher cash reserves. The Federal Reserve implemented Quantitative Easing by purchasing treasury securities from the banks to provide capital in the system.

Since 2014, the banks have applied an 'ample reserves' framework on their balance sheet. The Federal Reserve introduced a new interest rate 'interest on excess reserves' (IOER) where banks were paid a separate interest rate for holding excess amount of reserves in their books. IOER encouraged banks to hold further reserves, but also placed a new rate for the Federal Reserve to control that would influence the amounts of cash reserves the banks held. Increasing IOER would encourage more excess reserves, while the opposite would have the opposite effect.

A further instrument of the ample reserves framework were the overnight reverse repos (ON RRP). A reverse repo will see the Treasury sell a security to an eligible counterparty and repurchase the security at a higher price, implying an interest rate. These transactions were performed overnight for a short-term interest rate gain for the counterparty. The benefit to the counterparty were to protect them from the lower-bounds of the interest on reserves balances rate (IORB). IORB is paid to banks for the reserves they hold, and if that rate fell very low, the ON RRP can be used as a floor to ensure at least the ON RRP % return was possible for the cash reserves. As the ON RRP rate fell, the amount of capital banks were willing to hold increased as they could earn a higher rate than through an ON RRP.

As the reserves in banks fell with the ample reserves framework, the confidence banks had as to what 'ample reserves' would be required became less and less obvious. As a result, banks would hold more cash than they felt required to further drive the demand for cash.

How the Hike impacted banks

The immediate impact of the SOFR rates hike first meant banks were not as able to obtain cash from the repo market as required. A larger repo rate meant a larger cost to the bank for obtaining short-term cash. There would also be banks that benefited from the hike to realise greater profits in their cash reserves if held.

A secondary impact of the SOFR rates hike were the much larger number of parties that used SOFR as a reference rate, specifically to borrow at. This meant their fees for borrowing may have increased in a short term to reflect the sudden SOFR hike.

How the Federal Reserved intervened

The Federal Reserve were able to intervene to the SOFR hike to provide means for reducing the demand for cash. This was achieved by growing their balance sheet (increasing the assets they held) and then offering $75 billion in cash to borrowers on the repo market. The array of collateral that was accepted by the New York Fed increased to include Treasury Securities, agency debt securities and agency mortgage-backed securities. This made it easier for participants to access the cash injection.
Further intervention involved lowering the IEOER rate to 20 basis points below the top of its target range to also discourage excess reserves held by banks. From October 11th 2019, the Fed also injected reserves into the financial system by purchasing Treasury Bills to the value of roughly $60 billion per month through the second quarter of 2020.

Summary

The Repo Market is used to help banks obtain short term cash using highly rated securities as collateral. The Repo Market supports banks to operate with minimal cash reserves they think they require as cash is so easily available and at a low price.

The Repo market uses many reference rates to determine what rate will be returned for exchanging cash overnight, however repos can also be reversed where the party purchasing the security benefit from the resale at a higher price.

The Repo market is one of global magnitude, where reference rates impact both day-to-day operations of a bank and the many parties that use the rate for borrowing and lending. The 2019 SOFR rate hike is one example that illustrates the degree of control the central bank may have in controlling the cash reserves banks hold, and to influence their motivations and behaviour to use the repo market.

The LIBOR/OIS Spread

Sun, 29 May 2022 00:00:00 GMT

Key Takeaways
• An overview of the interbank markets and why they exist.
• Understand the use and differences between LIBOR and OIS.
• Understand how LIBOR-OIS spread can act as an indicator to interbank credit confidence.

Interbank markets

The interbank market differs from the 'open market' as it only enables specific banks to trade with each other. The interbank market often refers to the interbank foreign exchange (FX) market where currency and currency derivatives are traded. There also exists an interbank lending market where short term loans are also traded amongst banks to enable them to better manage their capital reserves and liquidity requirements. This blog will focus on the interbank lending market.

The interbank lending market enables banks to manage their liquidity risk to meet stringent requirements. Amongst many, one reason these requirements are in place are to ensure the banks are able to withstand any potential 'bank runs' by customers. On the other hand, banks with excess levels of capital can lend their reserves to earn an interest income.

Why would banks want to lend to each other?
Imagine if Bank A has £10 million pounds. Bank A will be able to make a nice profit by lending £8 million of that loan to commercial purposes at an interest rate of 2%. But Bank A has not yet found a purpose for the other £2 million on it's books. So rather than letting it sit in the bank earning nothing, they can then lend to other banks in the interbank lending market for a set period of time until they find a more lucrative use for the capital.

Bank B may be a customer to Bank A as they may require extra short-term capital for their own books (perhaps to meet liquidity requirements or to satisfy a new loan opportunity). They are able to borrow capital on the interbank market at an agreed interest rate. Bank A and Bank B will agree a timeframe for which the money is lent; Bank A is happy as they find a use for their extra £2 million and Bank B is happy because they are able to satisfy their immediate capital requirements.

The banks that are participating in the interbank market tend to be of high-credit quality. The interbank market is unique as loans are made without collateral and at a very low interest rate to reflect the credit-quality of the participants.

LIBOR

LIBOR - The London Interbank Offered Rate is a variable interest rate that is used by banks for short-term, unsecured interbank lending, just as Bank A and Bank B would engage in.

LIBOR is a globally accepted benchmark and reflects the cost of lending between banks. LIBOR is calculated on the rate that a consortium of banks lend 5 currencies (USD, EUR, GBP, JPY and CHF) at 7 different maturities (overnight, 1 week and 1, 2, 3, 6, and 12 months) within the interbank market. Despite there being 35 different LIBOR combinations of currency and maturity, the 3-month USD LIBOR rate is most commonly quoted as the 'current LIBOR rate'.

The LIBOR rate is posted on a daily basis. It is calculated by the British Banking Association (BBA) who ask major global banks what they would expect to pay for a short term loan in the interbank market. The BBA then remove the tail-end values and calculate the average. This is known as a 'trimmed average'. The daily LIBOR rate is published by the ICE (Intercontinental Exchange) Benchmark Administration.

The rate that each bank offers for short term lending will change each day to reflect credit risk on the loan. Credit risk is the risk that the original loan amount will not be repaid. As the length of the loan increases, so does the credit risk, as there is a longer period of time where the loanee may default and be unable to repay the loan in full. LIBOR is known as a forward-looking interest rate as it is a determined rate that is set at the beginning of the loan, and is paid at the loan's maturity.

So why would an everyday person care about LIBOR?

As a bank customer, you may also be impacted by LIBOR. LIBOR is often used to determine the interest rate on various products such as credit cards, loans and mortgages. As LIBOR impacts the rate at which a bank can obtain a loan at, it will impact subsequent loans the bank makes to its customers to therefore ensure they remain profitable.

Overnight Index Swap

The overnight index swap is a fixed-floating interest rate swap. The swap uses an overnight index as the floating rate. The overnight index is calculated by compounding an overnight unsecured lending rate to a new maturity. The overnight unsecured index rate is reflective of a single central bank. Indexes include:
• SONIA: Sterling Overnight Index Average
• EONIA: Euro Overnight Index Average
• SARON: Swiss Average Rate Overnight Index
• Mutan Rate: Japanese Overnight Index Rate

The unsecured overnight interest rate is calculated each day using the volume-weighted average of all unsecured loans for that date of a certain value. For example SONIA uses the interest rate applied to all overnight unsecured loans made up to 6pm that day that are greater than 25 million pounds.

LIBOR vs OIS

The key difference between LIBOR and OIS lies in the credit risk they represent. As capital is actually exchanged with a loan using LIBOR, there is a real presence of credit risk that is accounted for in the rate. As the maturity of the loan becomes greater, the credit risk on the loan increases as the opportunity for the counterparty to default increases.
OIS differs as credit risk is virtually eliminated. The overnight rate for a loan accounts for almost no credit risk due to the immediate repayment made on the loan. There is a very small opportunity for the loanee to default on the repayment. The OIS is considered to be risk-free as principal is not exchanged for the swap, but rather the difference in the agreed fixed rate and the floating index rate at maturity.

The LIBOR OIS Spread

The LIBOR-OIS (also known as LOIS) spread compares both the floating rate of LIBOR for a given maturity, and the compounded OIS rate to the same maturity.

As a result, the LIBOR-OIS spread represents the rate of interest that is required within the interbank market to compensate for the additional credit risk that comes with LIBOR due to the longer maturity on loans and capital transferred. As a result, we can use the LOIS to gather an understanding for how banks see credit risk in the interbank market.

You can find the historical LOIS spread here. Notice in March 2020 there had been a huge spike in the spread. A possible cause for this may be the outbreak of Covid-19 and the global lockdown that subsequently followed. A similar spike occurred in 2008 prior to the financial crisis. LOIS can be considered a valuable indicator to the perception of credit risk within the interbank market.

LIBOR scandal

Earlier in the blog, we mentioned how LIBOR was calculated, and how it had been a competitive rate set by specific banks that were averaged out by the British Bankers Association. Unfortunately, this 'competitive' rate submitted by the banks were not always done so with the purest of intentions.

One assumption made within an investment bank is the presence of a 'Chinese wall'. This is a theoretical wall that prevents communication between different divisions of the bank that may have an influence on each other. For example, between the parts of the bank that submit their LIBOR rates to the BBA, and the traders whose derivatives have exposures on LIBOR. The implementation of the Chinese wall may not have been carried out so well with certain banks.

If the LIBOR submitters for a bank are aware that a specifically higher or lower LIBOR can benefit the net position of their traders, they may be motivated to post artificially high or low LIBOR rates to the BBA. It is also believed that up to 20 banks colluded to submitting LIBOR rates that were not strictly competitive to benefit their positions.

A further drive behind collusion in submitting LIBOR rates that were not competitive were to give a false premise for the banks position to borrow and lend in the market. If a bank considered themselves riskier to lend to, they may reduce the LIBOR rate they they think they can lend at. As a result the LIBOR rate for the day may be reduced, thus enabling the bank that submitted the lower rate to borrow more cheaply.

In the aftermath of the scandal, believed to have started since 2003, LIBOR is currently being phased out, expected to be completed by June 30 2023.

Summary

The interbank lending market enables banks to efficiently allocate and obtain capital depending on whether they hold excess amounts or require it for short-term unsecured borrowing. LIBOR is a variable interest rate that determines the rate which banks can lend to each other in 35 different currency/maturity combinations.
OIS is a rate used by banks to borrow overnight, and therefore reflects a reduced presence of credit risk in the pricing.
The LIBOR-OIS spread is a useful indicator to summarise the confidence banks have in lending to each other as it reflects the credit risk they perceive for longer term interbank loans.

Populate multiple Excel columns in Java

Tue, 17 May 2022 00:00:00 GMT

Populate multiple Excel Columns with Java

Use a multi-dimensional Java array pattern to efficiently populate multiple columns in an Excel worksheet with Java. We will first look at how you can manually create cells and populate them with data in a for-loop, then improve our design by populating a multi-dimensional array and filling the day in accordingly. We will conclude the video by looking at an efficient way we can add headers to our worksheet to make the Excel file more organised.

Create an Excel workbook in Java

Tue, 10 May 2022 00:00:00 GMT

Creating an Excel workbook

Use the apache poi and apache poi ooxml libraries to create an Excel workbook in Java. We will add the packages into the maven project and use some of the classes such as XSSFWorkbook, XSSFSheet, Row and Cell to populate the Excel workbook with data!

An Introduction to Yield Curves

Thu, 28 Apr 2022 00:00:00 GMT

Key Takeaways
• How yield curves are used.
• Review the properties of a bond.
• Understand how and why the yield curve can change shape.

Yield Curves - An introduction

This blog is split into multiple sections that will eventually lead to (hopefully) a better understanding of yield curves. First we will look at bonds, what they are and why businesses and investors use them. Then we will look at bonds and interest rates, and why we might consider interest rates when investing in a bond. Then finally we will look at the yield curve, to understand what it looks like, how it can change, and most importantly, what it means.

Yield curves are used within finance to measure the risk appetite of bond investors. A typical yield curve may look like this:

The yield curve has two axis, one for yield (return percentage) and one for time to maturity. We can see that as maturity years change for a bond, the expected yield changes, rising quickly at the start, and then slightly flattening towards the end. But wait, what is a bond?

Bonds

A bond is a fixed income instrument that represents a loan that an investor makes to a borrower (typically a corporation or government entity). The bond will have a few key properties:
• Principal: the amount of money the corporation is borrowing.
• Coupon: a percentage amount of the principal the corporation will pay the investor at fixed intervals (which is typically every 6 months).
• Maturity: when the bond will expire and the principal amount is repaid to the investor by the corporation.

We can describe how a bond works using its properties. The corporation or government entity need to raise short-term capital. They can do this using bonds. An investor can purchase a bond, this involves lending the corporation the principal amount, with the promise of being repaid at the maturity date. The maturity date ranges from weeks to decades. As an incentive to the investor to purchase the bond, the corporation that issues the bond will pay a bi-annual fee to the bond holder, which is a percentage of the principal, known as the coupon rate.

So if I buy a bond with a principal of £100, and a coupon rate of 2% that matures in 3 years, I will receive payments of £2 every 6 months for 3 years before receiving my £100 back from the corporation.

Bonds fall under the type of investment called 'fixed income' as they provide the investor with a fixed stream of income from the investment. This fixed stream is realised in the form of the coupon repayments the investor receives.

Bonds are considered to be a fairly safe investment when compared with equities. This is because bonds are bought with a promise of the issuer to return the face value of the bond at maturity. Equity does not carry such a guarantee. As a bond-holder, you also sit higher in the liquidation hierarchy than a shareholder for a corporation if they were to default.

Bonds and Interest rates

As a bond-holder, you will be also be conscious to the current interest rate and forecasted interest rate for the maturity of the bond. But why?

Bonds act as an alternative low-risk investment option for your money when compared to a savings account. Leaving your money in a savings account will guarantee a return of the interest rate, whereas investing the money in a bond will return the coupon rate for that bond. Buying a bond means your capital is tied up with the corporation until maturity, but in return you receive the bi-annual coupon return for the investment.

Using the example from above, if interest rates were 1%, I would probably prefer to invest my £100 by buying a bond, as the coupon rate is 2% and earn more money over 3 years, than just the 1% interest I receive each year from my bank. This is provided I would not like to access my £100 during that time. The benefit of leaving my money in a savings account means I can withdraw it when required, such a benefit does not exist with a bond, unless I sell it within the market.

When we later look at yield curves, we will be able to reflect upon interest rates as their expected long-term behaviour can influence the expected returns an investor has for a bond.

Different Yield Curves

Not all bonds return the same amount to the investor, nor do they last for the same amount of time, or even pay coupons at the same time. Bonds can differ by the corporation they are bought from, and the credit quality deemed for that corporation. Corporations with a lower credit quality may pay a higher coupon rate that reflect the higher risk faced by the investor. Similarly, maturities for bonds can also be different. If a bond has a longer maturity, it means the principal will be repaid to the investor at a different period of time in the future (and the coupon repayments are received for a longer period of time). We are able to visualise the risk of holding a bond using a yield curve. A yield curve illustrates the expected returns investors have for bonds of a similar credit quality, but at different maturities. This allows us to understand exactly what level of return is expected for a bond where the maturity of the bond is the only variable.

Given any individual yield curve can be reflective of a bond's credit quality, you will find many different yields curves, such as the Treasury Yield curve that reflects the rates of US treasuries, AAA Municipal yield curve that refelct AAA municipal bonds, or even those for a single issuer such as the IBM Yield curve. When you hear about yield curve in the news and in economics, it is likely they are talking about the Treasury Yield Curve.

The standard Yield curve

We have seen what a normal yield curve looks like earlier in this blog:

Let's break it down slightly to understand why it is the shape it is. When an investor commits funds to an bond, they are leaving their capital at risk. If you were to invest £10 for 1 month, and were offered £1 back as profit (so £11 return in 1 month), you probably wouldn't be as happy with an offer of receiving £11 back in 2 months time. Investing money for longer periods of time means your invested capital is at risk (in the event that the principal cannot be repaid) for a longer period, and is therefore at more risk. As a result, investors therefore expect higher returns when the time to repayment (maturity) is longer. Equally, if the money were left in a savings account, it may accrue more interest in a longer period of time.

The yield curve for a bond is not linear, after all we may not expect the exact same yield for 2 months as we do receive for 1 month. We can see in the graph that the yield for 20 year maturity is only slightly less (3.9%) than the maturity for 30 years (4%).

The yield curve typically increases at a steeper rate during the early period, and then flattens towards the longer maturities from 20 years to 30 years.

Interpreting a yield curve

So what does this all mean, and why do yield curves differ? Yield curves are often used to understand how the investors expect long term interest rates to behave. Why? Because if an investor is able to establish an opinion on how they expect interest rates to move, they will therefore have an expectation on what yield to demand for a similar maturity of a bond.

Let's head back to our example from earlier. Let's say we expect interest rates to rise to 5% in 1 years time, and 10% in two years time. All of sudden, the current interest rate of 1% doesn't seem so bad to begin investing with. And all of a sudden, our bond, which pays a 2% coupon every 6 months is not looking so good. If we have good reason to believe interest rates are going to skyrocket in the coming periods, then we can rightfully expect bonds to beat those interest rates for an investment of the same period. Why? Because bonds naturally carry more (be it very little amounts) of risk than a regular savings account. Investors will not be so keen to buy the 2% bond if they expect the interest rates to rise so quickly over the same period of time.

Changing the yield curve

We may see a yield curve begin to flatten when investors expect long term interest rates to fall in comparison with short term interest rates. When this happens, the required rate of return on a bond with long maturity is reduced as the investor does not expect interest rates to be as high in the future. Therefore the right-hand-side of the yield curve may fall, and the curve flattens.

We may also yield curves steepen:

A steepening yield curve will see yields on bonds that mature at dates further into the future return significantly higher yields than those in the short term. A steepening curve reflects a steepening long maturity or a shallowing short maturity. It is indicative of expectations for higher interest rates in the future, so corporations are willing to provide greater returns on their long-term bonds.

Lastly, we can also see an inverted yield curve:

This is a rare case where short term bonds return a greater yield than long term bonds. This means investors expect the interest rates to fall in the future, and are therefore willing to accept smaller returns for the longer maturity bonds.

The yield spread

The term 'yield spread' is used to briefly, and effectively compare two bonds. These bonds can have different maturities, different credit ratings, and different risk levels. It allows analysts to understand how yield is currently represented for a bond, and if, compared to its historical points of reference, is a good investment.

A yield spread is quoted by comparing the yield of one bond against another. If you are comparing a bond yield against the US treasury bond, it is known as a credit spread. The US treasury bond is considered a de-facto standard for bonds. If bond 'A' has a 5-year yield of 3% and the US Treasury bond has a yield of 2%, then the yield spread is 1%, also expressed as 100 basis points. If bond 'A' usually has a yield spread of 0.2% when compared with the US Treasury Bond, then that means it is currently offering a high (or 'wider') spread than usual and is therefore offering a greater return than it historically did.

The yield for corporations that are typically smaller, with poor financial data and backing may be greater than that of a larger organisation or government entity. This is because the bond with the smaller corporation may be perceived as being riskier, thus warranting a larger yield for the bond.

Yield curves today

It can also be interesting to take a look at yield curves today, to understand how investors expect interest rates to change in the future. You can visit worldgovernmentbonds.com to get a picture of different bond yields.

Summary

The yield curve is used to understand the expected returns of an investor for a given bond at a given maturity. The yield curve helps us to understand future expectations of interest rates as investments with bonds are often compared with risk-free savings. Analyst will create historical yield spreads to compare different bonds, and to understand when some bonds are trading at a premium than they previously have against others.

Customise and format REST API Error responses

Mon, 07 Mar 2022 00:00:00 GMT

Customise Error Responses from a Spring REST API

Learn how you can customise the default error response that a Spring REST service can provide to invalid or incomplete API Requests. This video will visit the ResponseEntityExceptionHandler to understand how the error response is created, and will use annotations such as @ControllerAdvice and @ExceptionHandler to build custom responses to specific thrown Exceptions from the REST controller.

Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/REST%20API%20With%20Error%20Formatting/Store

Update a REST Controller with PUT, POST and DELETE endpoints

Sun, 13 Feb 2022 00:00:00 GMT

Make PUT, POST and DELETE requests with a Spring web server

Learn how to create DELETE, PUT and POST endpoints to a RESTful API service built with Spring Boot. We will create these endpoints in the controller layer, and then use the Postman API Platform to make it easier for us to test the endpoints. We then verify the output from our REST calls with the connected MySQL Database.

Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/Store

Connect a Spring Boot REST Server with a MySQL Database

Fri, 11 Feb 2022 00:00:00 GMT

Create a MySQL Database Connection and table

This video will look at how we can connect our Spring Boot Application with a MySQL Database. We will use the Spring data-jpa package to leverage a connection and to provide out-of-the-box methods and functionality to our controller. At the end of this video, we will have created 2 REST endpoints that obtain data from our MySQL database, and present them to the user as a JSON response.

Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/Store

Create a MySQL Database table

Wed, 09 Feb 2022 00:00:00 GMT

Create a MySQL Database Connection and table

Learn how you can download MySQL and the MySQL Workbench to begin storing data with a database. We will walk through where you can find the downloads, and how you can create a connection, schema and table with the MySQL Workbench. We will also explore the various characteristics a field can have within a table, and what they mean. To conclude, we will create the table and populate it with some data.

Link to source code: https://github.com/4neesh/YouTube-Channel/tree/main/Store

Create a Responsive REST Controller

Mon, 07 Feb 2022 00:00:00 GMT

Create a REST Controller with Spring Boot

Get started with the Spring initializer to build a Spring Boot Web Server. The web server will contain a Controller layer that will store endpoints that we interact with through the browser.

We will register some GET endpoints that will return data to the user, and we will explore how JSON data appears within the browser.

Syntax changes with Java 9

Mon, 24 Jan 2022 00:00:00 GMT

Changing default mock behaviour

This video will explore the 3 key syntax changes that are introduced with Java 9 from Java 8. First we will look at how collection factory methods can be written in a neater way for immutable entries. Then we look at how the stream api has been enhanced to introduce methods to apply filters to an ordered collection, and to remove null values from the stream. Lastly we look at how private methods can be used within an interface, and when they may be required.

Java 9 Modularity - Java Tutorial

Mon, 17 Jan 2022 00:00:00 GMT

Changing default mock behaviour

Learn about how Java 9 enables strong encapsulation with modularity. Modules add a further layer of encapsulation to our packages as we are able to specify which packages can be used by other modules, and which modules can have access to them. This video will transform a multi-layered application to use modularity with Java 9 to demonstrate the benefits that can be realised.

The source code to this video can be found on GitHub: https://github.com/4neesh/YouTube-Channel/tree/main/Java9%20Modularity

How to use JShell - Java 9 Tutorial

Mon, 10 Jan 2022 00:00:00 GMT

How to use JShell - Java 9 Tutorial

JShell provides a REPL (Read, evaluate, print - loop) service for Java engineers to test their logic in a convenient way through a terminal. This video will look at how we can define different variables, methods and classes within JShell, how we can save our sessions and load external classes, and some of the most useful keywords from JShell.

Changing default Mock behaviour

Mon, 03 Jan 2022 00:00:00 GMT

Changing default mock behaviour

This video will explore the 3 different ways that a default Mock Object can be returned to the test class from a method call. By understanding how mock Object behaviour can be changed, we can transform Null Object responses into mocked, Spy or real Object representations. As a result, test methods can be more fluent and resistant to empty and null default values.

How to stub void methods

Mon, 27 Dec 2021 00:00:00 GMT

How to stub a void method

This video will look ath the different ways that we can control our system under test when using void methods with Mockito. This video explores how we would want to handle void methods when using a mock or a spy, and how the answer functional interface also presents unique ways to provide custom logic to a void method, and when it would be most appropriate.

ThenReturn vs DoReturn

Mon, 20 Dec 2021 00:00:00 GMT

ThenReturn vs DoReturn

Why would we use thenReturn over doReturn and vice versa with Mockito stubbing? This video compares the two methods to understand exactly how they work, their pros and cons, and when they are most appropriate to use.

Testing the Entity Manager API

Mon, 13 Dec 2021 00:00:00 GMT

Testing the EntityManager

Use Mockito to create stubbed query responses when testing implementations of the Entity Manager API. The Entity Manager API is used to create instances for persistance that fall on the repository layer of a Spring Boot application. The Entity Manager will manage the lifecycle fo the entity instances for persistance, and can be highly important when persisting and managing data for an application.

This video will use Mockito to stub an EntityManager and Query to control the response of an EntityManager for testing a simple select query.

Intro to Mockito Answers

Mon, 06 Dec 2021 00:00:00 GMT

Mockito Answers

Learn how you can achieve dynamic method stubbing with the Answer functional interface and the benefits it can enable over hard-coding stubbed methods. This video will explore the challenges of hard-coded method responses with stubbing and how an Answer implementation can reduce boilerplate code from a test class. #Mockito #SoftwareTesting #JUnit

Test secondary method calls with argument matchers

Mon, 29 Nov 2021 00:00:00 GMT

Using Mockito argument matchers

Learn how to use Mockito argument matchers to verify indirect inputs for secondary method calls. This video will walk through an example of how an indirect input is created, where a secondary method call is made, and how we can generalise these behaviours to be tested with Mockito argument matchers.

Mockito #SoftwareTesting #JUnit

Stubbing consecutive calls with Mockito

Mon, 22 Nov 2021 00:00:00 GMT

Stubbing consecutive methods

Learn how Mockito supports the testing of multiple method outcomes in a concise and readable way with consecutive stubbed calls. This video will explore when we may need to stub and test different method outcomes, and how you can achieve this with Mockito without repeating multiple lines of test code.

Mockito #SoftwareTesting #JUnit

Exploring Collection API Enhancements from Java 8

Mon, 15 Nov 2021 00:00:00 GMT

Java Collection API Enhancements

Learn how and when to use the Collection API enhancements from Java 8 to reduce repetitive and error-prone code when handling and transforming collections. This video will include alternative ways to iterate a list, to navigate and sort a map, and to replace or merge values of a map with new API methods.

Intro to the Java 8 Optional class

Mon, 08 Nov 2021 00:00:00 GMT

Java 8 Optional Class

This video will explore the Optional class and the API methods that are introduced with it in Java 8. The Optional class will enable you to check for null, perform default operations if a value returns null, and also use custom logic as a response on a single line.

No longer will you be required to perform repetitive and cumbersome null pointer checks in your code as you can conveniently handle potential nulls values with the Optional class API.

Using method references with Java 8

Mon, 01 Nov 2021 00:00:00 GMT

Method References in Java 8

Learn how, and when you can use method references to improve the readability of your Lambda functions. Introduced with Java 8, method references can be used to replace lambda expressions that point to a class method, an instance method, and a class constructor. By understanding how and when method references can be used, you will reduce the boilerplate code written for your lambda expressions.

Using the Java 8 Stream API

Mon, 25 Oct 2021 00:00:00 GMT

Java Stream API

This video will take a look at the Java 8 Stream, an incredibly powerful and useful API that enables us to navigate, transform and extract from Collections of Objects into new collections or Objects. We will firstly understand the different ways that we can create a stream and their advantages. Second, we will look at the vast array of functions that allow us to operate upon the Collections, and then lastly explore the various ways that we can aggregate our stream into a new Object or Collection.

Link to the Oracle Stream API documentation: here.

Out of the box functional interfaces with Java 8

Mon, 18 Oct 2021 00:00:00 GMT

Java 8 functional interfaces

Functional interfaces are an important component to Java 8 in it's strive towards more functional programming styles to complement the Object-orientated language.

This video will look at 4 key functional interfaces provided in Java 8 to support Lambda functional writing. First we will look at why they are needed and how they work, then we will explore the extended functional interfaces such as the unary and binary operator.

Creating default methods within interfaces

Mon, 11 Oct 2021 00:00:00 GMT

Interface Default methods

Explore the benefits of the default keyword that has been introduced with interfaces within Java 8. We will look at when and how you can use the default keyword, how interfaces will now compare with the abstract class, and examine the corner case condition where a class may implement two interfaces with the identical default method signatures.

When and how to use Lambda expressions

Mon, 04 Oct 2021 00:00:00 GMT

Lambda Expressions

Lambda expressions are a new feature from Java 8 that allow you to efficiently create instances of a functional interface with a new syntax: () -> {}. This video will take a look at functional interfaces and how we can instantiate them. We will finally look at the lambda expression syntax and how it relates back to the functional interface.

When and how to use the Singleton design pattern

Mon, 27 Sep 2021 00:00:00 GMT

The singleton design pattern

The singleton design pattern is one of the most common design patterns you may come across within object-orientated software development. We use the Singleton pattern when we only want a single instance of our Object to be created and coordinated across our application. The Singleton design pattern is implemented in just 3 simple steps.

Basic file manipulation with awk

Mon, 20 Sep 2021 00:00:00 GMT

An introduction to awk scripting

Learn about the convenient awk scripting language that you can use from the Linux CLI to manipulate, filter and reorder structured files into new reports with a single line command.

Introduction to MockitoBDD

Mon, 13 Sep 2021 00:00:00 GMT

Using MocktioBDD

MockitoBDD provides us with special methods that support the communication of a behaviour-driven-development approach to software development and testing.

In this video, you can integrate MocktioBDD with a given-when-then style of test writing to clearly outline how Objects are mocked and instructed to behave, while asserting against outcomes from the test in a clear manner.

Link to source code: Here.

Comment, like and subscribe for more content on Java and software engineering.

Visit my website to also see my blogs: https://aneesh.co.uk

Options contract and their pricing

Mon, 13 Sep 2021 00:00:00 GMT

Key Takeaways
• Understand what an Options contract is and why it is used.
• Take a brief look into the components of an Options contract that are used to derive the price.

What is an Options contract

An Options contract is a derivative that defines an agreement between two counterparties where the buyer of the contract has the option and NOT the obligation, to buy or sell a particular asset at a particular quantity within a certain time frame.

Using the above description, when we create an Options contract, there are 4 key variables defined:
1. If the buyer of the contract has the option to buy or sell the asset.
2. What the price of the asset will be when bought or sold, known as the Strike price.
3. The quantity of the asset to be bought or sold.
4. The maturity of the Options contract to when it is no longer valid.

The times at when the Options contract can be exercised is determined by its type. 'European' Options can be exercised only at expiry, whereas 'American' Options can be exercised at any time before. 'Bermudan' Options on the other hand will have set intervals when it can be exercised leading up to the maturity (such as once per month).

Buy and Sell contracts

As mentioned above, an Options contract will entitle, but not obligate the buyer to either buy or sell a particular asset at a strike price by the maturity date.
An option contract that entitles the holder to buy an asset is called a Call option. An option contract that entitles the holder to sell an asset is called a Put option.

Option contracts are not limited to only a single market or participant and are traded for a variety of purposes due to their many advantages that we will discuss later. Options contracts exist for a huge range of markets including commodities, stocks, bonds, interest rates, currencies and also swaps, known as 'Swaptions'.

One example of Options contracts traded without an exchange are Dealer options. Dealer options are used to trade commodities over the counter (OTC) without a centralised exchange. The benefit of OTC trading are the personalised terms that can be agreed upon, thus enabling both counterparties to achieve a more bespoke agreement.

The variety of options contract available to investors make them highly personalised and present one key advantage to using them.

Why Options are used

So we can now understand what an Options contract is, what its key properties are and the many forms it can take in the market, but why are they used?

As repeated in this blog, the Options contract provides the right, and not the obligation for a counterparty to transact upon the terms of the derivative.
This property of an Options contract means they can be used with huge flexibility and present ample opportunities for both hedging and speculative incentives.

Hedging strategies
A hedging strategy may see a counterparty use an Options contract to protect themselves from both the upward and downward movement of an interest rate.
For example, if we take a floating point interest rate that is currently 0.16%, counterparty A may agree with counterparty B an Options contract that allows counterparty A to receive payment each time the interest rate exceeds 2% and falls below 1.12%. This hedging strategy is known as a cap and floor, where counterparty A are capping the maximum interest rate they are exposed to at 0.2% and the minimum rate of 0.12%.

Counterparty A will pay a premium to counterparty B for the Options contract, but the benefit to counterparty A is the protection from interest rate volatility if it were to exceed either boundary.

Speculative strategies
On the contrary, the Options contract can enable an investor to benefit from the movement of an underlying asset's price, as they are able to exercise with their right when the price is deemed 'in the money'.
In the money: An options contract that can be exercised to achieve a profit and has 'intrinsic value'.
As a speculator, the benefit of an Options contract can present unlimited upside, with a limited downside (the premium paid for the contract). For example, if an Options contract enables an investor to buy 100kg of wheat for £1000, and the cost of the options contract is just £50, the investor will have a limited downside of £50, but a potential upside determined by the upward price of wheat within the market. If 100kg of wheat rises to £2000, the investor can make a £950 profit from exercising their options contract, whereas if the price of wheat remains below £1000, they will only ever lose the £50 premium paid up-front.

When considering a hedging or speculative strategy, the benefit is realised by the holder of the contract only when it is exercised. At expiry, the Options contract is deemed worthless, and the benefit of the Options contract creation lies with the seller of the contract. With this in mind, you may wonder how the price of an Options contract is derived, and the factors that play into its value throughout it's life.

Options contract pricing

The Black Scholes and Binomial model are popular mathematical models used to calculate the price of an Options contract. The pricing of an Options contract is derived from the probability that the Options contract can be exercised in the money. Each model has its own advantages and approach:

Black Scholes
Developed in 1973, the Black-Scholes-Merton model uses risk factors and time-value to price European Call Options (can only be exercised at expiry).
The analytical model uses 5 variables: strike price, current price, time to expiry, risk free rate of return, and volatility.
The model is based upon the assumption that asset prices have a lognormal distribution (bell curve) with a constant drift and volatility. Using the input variables, the Options price is calculated against the assumptions of constant volatility, unpredictable market movements, and lognormal price distribution.

Binomial Model
The Binomial model is an iterative model that accounts for multiple periods of time for exercise, and is thus an approach used for American Options. While the Binomial model also uses the inputs from the Black Scholes model, it is less complex as it uses a Binomial tree to calculate the value of an Option given two possibilities: underlying price movement goes up or down. The model, shaped like a lattice, can then be split into intervals that represent that time periods for the underlying price movement to happen, thus creating the iterative Options price predictions. The Binomial will build a map of different possible scenarios for the iterative movement of the underlying price of the asset at different time intervals. Each price movement will then represent a present value.

Options value and sensitivities

The Options pricing models make use of multiple variables that represent the multitudes of variance a standard Options contract will be exposed to.

The complexity of Options pricing has lead to the inception of the 'Greeks', four risk measures used to quantify the sensitivity of the price to an external factor.
The 4 measures are:
1. Delta: price sensitivity of the option to the asset price
2. Theta: price sensitivity of the option with time
3. Gamma: the rate of change of Delta
4. Vega/Kappa: rate of change between an option and the asset implied volatility

In my next blog post, I will explore the Greeks and their relationship with 'intrinsic' and 'extrinsic' value to the investor to further understand how an Options price can be so widely influenced.

Conclusion

In this post, we have looked at the Options contract and how it can be used for both speculative and hedging purposes to provide a variety of benefits to both sides of the contract. Not only can Options contract be used for a variety of benefits, but they also come in many shapes and sizes with the right to buy or sell across a variety of markets and with varied opportunities for exercise.

The benefit of 'optionality' from the contract exposes the price of the Options contract to multiple variances within the market and time that can influence their price. As a result, complex mathematical models are used to best estimate the present value of the Options contract within the market. When reviewing Option prices, the sensitivities that are exposed to the price have led to the use of the 'Greeks' a group of well defined variances to the Option price.

Core text file operations with the Linux CLI

Mon, 06 Sep 2021 00:00:00 GMT

Text file operations with the Linux CLI

Take a look at some of the core commands you can use within a Linux terminal to view and navigate files.

Avoid the NullPointerException when using the .equals method.

Mon, 30 Aug 2021 00:00:00 GMT

Avoiding the NullPointerException when comparing Objects

The .equals method is often used within if-statements and to return boolean expressions within an application. By understanding how the structure of the boolean to call the .equals method is made, we are able to design our booleans to avoid any null exceptions when comparing two Objects.

Using the Builder design pattern.

Mon, 23 Aug 2021 00:00:00 GMT

Using the Builder design pattern

The builder design pattern offers a structured and readable way to create complex Objects. Rather than using a long and rigid constructor to instantiate multiple variables and Objects for a class instance, the builder design pattern enables you to set each variable as it is required when creating an instance.
While appearing more verbose, the benefits of the builder design pattern will ensure Objects are created in a responsible way where each property is assigned as expected.

Using Evaluate Expression and Watches in IntelliJ.

Mon, 16 Aug 2021 00:00:00 GMT

Evaluate expression and Watches with IntelliJ

Learn how the evaluate expression and watches tools within IntelliJ can assist with debugging an application. The evaluate expression will allow you to perform operations on an Object that is locally stored. For example, you can stream a List, get a value from a Map, or perform mathematical calculations on a value.
The Watches tool will allow you to perform an operation on an Object, and to monitor that same operation as the debugger runs frame by frame.

How @InjectMocks works with Mockito

Mon, 09 Aug 2021 00:00:00 GMT

How to use @InjectMocks

@InjectMocks will allow you to inject other spies and mocks into a class that requires services when it is being tested. The annotation will scan the class under test and automatically inject all available mocks and spies from within the test class. No more NullPointerExceptions when an unknown service is called!

Customise the IntelliJ test view

Mon, 02 Aug 2021 00:00:00 GMT

Custom test views

Explore some of the on-demand settings that are offered with IntelliJ that enable you to customise your test class outputs that can improve how you navigate, review and debug your test methods.

By the end of the video you will be able to sort and filter the test methods that are returned when running a test class. You will be able to navigate across test methods with a single click, and to the failing assertion with two clicks. And lastly, you will be able to create customised html reports, straight from the IDE for an easy-to-read interface of the test outcomes.

How to use the ArgumentCaptor from Mockito

Mon, 26 Jul 2021 00:00:00 GMT

Using the ArgumentCaptor

This video will explore the conditions that would make us interested in testing the arguments passed around our application methods, and howe we can extract the Objects that are passed using the ArgumentCaptor. With the ArgumentCaptor, we are able to isolate Object arguments away from the stack that calls it, to then test them on their own back in the test class.

An introduction to basis risk

Mon, 26 Jul 2021 00:00:00 GMT

Key Takeaways
• What is basis risk.
• How basis risk is created.
• Why basis risk is so frequently encountered.

Spot prices vs Futures prices

To understand how basis risk, we need to first understand the difference between a spot price and a futures price. A spot price, is the price paid for an commodity, asset or security at a given time. For example, when you go shopping at a supermarket, you're in theory paying the spot price for your groceries.
A futures price is the price for an amount of a specific asset to be delivered at a certain date in the future. For example, a spot price for a StockA might be £2.00 today, but a futures price for it might be £2.03 to be delivered in 5 weeks time. One security can have many different futures prices for it as the quantity and time-frame for when the asset will be delivered will be different.

The futures contract

A futures contract is a derivative that is traded between two counterparties as an agreement to buy or sell an underlying quantity of a commodity, asset or security at a certain price and at a certain point in time in the future. The futures contract is traded through an exchange. A similar kind of derivative is a forward contract, which is traded without an exchange, and is instead traded over the counter.
The futures contract serves both counterparties:

First, the counterparty that sells the contract are fixing a price for which a certain amount of an asset at a point in the future will be delivered and sold.
Second, the counterparty buying the contract are securing the delivery for the amount of the asset at a point in the future.
The time which the exchange happens is known as the expiry. The price of the future contract will represent the price of the asset. For example, I could pay £100 for a future contract that represents the delivery of 10kg of chicken breast in 1 weeks time. The buyer of the futures contract (me) is therefore hedging the risk that the price will increase from when the futures contract is agreed upon (that the price of 10kg chicken breast does not rise beyond £100 in 1 weeks time). As a result, the I am protecting myself from price risk.

The motivation for buying a futures contract can also be used for speculation. For example if an investor believes the price of an asset may rise to more than the market expect, they can purchase a futures contract to secure the delivery of the asset at a cheaper price than they expect it to rise to, and then can make a subsequent profit from the resale of the asset in the market at expiry.

How are futures used in the real world?
A common example I refer to would be any airline company that would like to secure the delivery of jet fuel for the immediate and long-term future. By using a futures contract, they can secure the cost of jet fuel, and better estimate their cash requirements and costs for flights in the future.

A rather interesting example might also be when McDonalds benefitted from a futures contracts that enabled them to create the chicken nugget. You can read the story here.

Basis Risk

As mentioned above, the counterparty buying the futures contract are eliminating 'price risk' on the delivered asset as they secure the cost for its delivery. The price risk is however replaced by a new risk, called 'basis risk'.

Basis risk exists as the price difference between the current price of the underlying asset and the price that is agreed upon in the futures contract. Basis risk is therefore a replacement of price risk as it hedges the investor from the absolute price movement of the asset.

Let's use an example

If a shop owner is worried about the price volatility of milk, where it is £1.40 per gallon today and expected to rise to £1.50 in 6 months, they may like to buy a futures contract to ensure the delivery of milk at a price of £1.50 per gallon for 6 months time, and to protect themselves of any further unexpected price increases. Let's say they're buying 100 gallons worth, the futures contract will be priced at £150 (100 gallons times by £1.50).

The basis risk the shop owner faces is equal to the price agreed within the futures contract and the current price of the asset. This would be 10 pence, where the agreed upon price is £1.50 and the price of milk when the futures contract is bought, is just £1.40.

The price agreed on the futures exchange will edge closer and closer to the actual price of the asset as the time to expiry reduces. As time passes, the movement of the futures price and the underlying asset can however change. For example, the price of milk may rise to £1.60, therefore the basis risk to the shop owner will actually be -10 pence. What we can see here, is that basis risk can actually be positive and negative as the price of the asset changes. As the underlying asset price moves, so do the price of futures contract for the same maturity date that the shop owner had initially agreed upon.

If our shop owner agreed to exchange 6 months from today, which is 1st January 2022, and by December 1st 2021 the price of milk is now £1.60, then the futures price for delivery of milk on the 1st January 2022 is likely to be higher than £1.50 (for those wishing to buy a 1 month maturity future).

The price movements between the underlying asset and the futures market are not always perfectly correlated. There are many factors to consider for a futures contract such as the cost of storing the asset, delivery of the asset and the risk-free rate of return that can all impact price. As a result, the basis risk, which is the difference between the price today for the asset and the agreed upon futures price, can grow and shrink as the market for the asset and its future price move in an uncorrelated fashion. In an ideal world for the shop owner, the price of the milk and the futures price will be well correlated such that basis risk is 0 at expiry.

If markets are perfect, when does basis risk actually occur?

If we use the above example, where basis risk occurs between the difference between the current price and the futures price of an underlying asset, you may be wondering how such a risk can ever be present at maturity. After all, markets should be efficient enough, such that the shop owner is only able to buy at a futures price that very closely reflects the expected price of milk 6 months from today. Perhaps it would take a significant market condition to create such a shock in the milk price that a basis risk becomes prevalent (such as a new cow disease).

The reason basis risk is such a prevalent risk to consider, is because the futures contracts that are bought aren't always perfect. As a buyer of a futures market, you would require a futures contract that specifies all three components perfectly:
1. The exact asset.
2. The exact amount of the asset.
3. The exact maturity required.
4. The exact delivery location of the asset.

Failure to reach all three of these targets in a futures contract will create an imperfect hedging strategy. This imperfect strategy creates a basis risk for the futures contract buyer as the movement of the futures price may not be perfectly correlated to the movement of the true asset they are trying to hedge against.

For this reason, futures contracts are not always held to expiry, and are therefore sold in the market when the price risk for the asset has been reduced against time to a significant degree. The imperfect hedging strategy exposes a counterparty to one of 4 different sub-components to basis risk:
1. Price basis risk: Where the price of the asset does not move as expected towards the futures price.
2. Location basis risk: The delivery location of the asset is different to that desired by the buyer of the contract. For example, they would like delivery of milk to England, however the market can only offer the delivery to France, which would reflect a slightly different market price movement.
3. Calendar basis risk: Where the expiry of the futures contract is different to the desired duration from the contract buyer. If the futures contract expires 1 week beyond the desired delivery of the asset, the buyer may sell the futures contract 1 week early, and realise the outstanding basis risk on the contract.
4. Product quality basis risk: Where the asset on the futures contract does not perfectly reflect the asset hedged against. For example, the futures price for semi-skimmed milk may move slightly different to that of 2% milk.

It is due to these factors that an imperfect hedge may be taken within the market, and thus basis risk is a key exposure faced by the buyer of the contract.

Summary

A futures contract is used to fix the delivery of a specified quantity of a specific asset and at an agreed date in the future. The contract is traded over an exchange.

Basis risk exists at all times within the life of a futures contract where the current spot price for the asset is different to that of the futures price. Therefore basis risk can be both positive and negative.

Perfect markets would mean basis risk is minimised at all expiry dates of all futures contracts. However future contracts are not always perfect for the risk that is hedged by the buyer of the contract. Factors such as the asset, asset quantity, date of delivery, and market pressures, can all mean the futures contract is sold before its expiry, and therefore expose the selling party to the basis risk of the contract.

Testing private methods with Powermock

Mon, 19 Jul 2021 00:00:00 GMT

Testing private methods

Learn how you can test private methods within your application using JUnit. First we explore how the reflection utils from the Java library can be used to control and invoke a private method. Second, we use the Powermock library to condense a large portion of the test to just a single line.

Testing the content and order of collections

Mon, 12 Jul 2021 00:00:00 GMT

Testing collections

Learn about 3 different test methods that will enable you to assert upon the elements contained within a list, the comparator used to sort a list, and the like-for-like elements that are contained within the list to assert upon their order with the AssertJ library.

Exploring stack calls with the debugger

Mon, 05 Jul 2021 00:00:00 GMT

IntelliJ Debugger

Take a look at 3 of the most commonly used tools of the debugger that allow you to navigate your application run with control. A break point will enable you to pause your application each time a specific line of code is executed, from there, we can use the step over tool to execute each line, one by one, and the step into tool to dive into any other method calls made on that line.

Navigating across IntelliJ

Mon, 28 Jun 2021 00:00:00 GMT

IntelliJ Navigation

Discover keyboard shortcuts that can help you to find specific files, methods, classes and instances across your application with higher efficiency with IntelliJ. IntelliJ offers different ways that we can instantly specify where we would like to search and what type of item we are looking for (whether it be a class, file, method, variable and so on.) Lastly, we will look at how you can control-click your way around an application to see where methods are called and what their implementation is.

Using the Rule Annotation

Mon, 21 Jun 2021 00:00:00 GMT

The Rule Annotation

Leverage the Rule annotation to neatly surround your unit tests with bespoke logic for setting up and tearing down components that can be used within each test. Explore the different implementations that come out of the box with JUnit, and dive a little deeper to understand how the TemporaryFolder may be valuable to use when testing with files.

Improve test readability with matchers

Mon, 14 Jun 2021 00:00:00 GMT

Creating a custom matcher

Improve the readability of your unit test assertions with custom assertions, created with the AssertJ library. In this video, we take a look at how we can create a custom assertion to perform custom logic and readability to the assertions we apply within our tests

Improve test readability with matchers

Mon, 07 Jun 2021 00:00:00 GMT

Using Matchers with assertions

Matchers can be used with external libraries such as Hamcrest, FEST and AssertJ to improve the readability of the assertions we write in our test classes. Learn about the different matchers that are available as standard from the AssertJ library as we refactor assertions made in JUnit to be more readable.

Inside Java: why it works everywhere

Mon, 07 Jun 2021 00:00:00 GMT

Key Takeaways
• Understand how the Java engine works to interpret and execute code.
• Review the stages of compilation and execution that support the type safety and memory management properties of Java.

Java - a 30 year old language

Java programming language is one of the most popular programming languages in the world that is prevalent across industries and careers. You can gather an idea of how diverse and important Java has been in shaping peoples lives by searching upon the hashtag #movedByJava on social media platforms such as Twitter.

Until recently, Java had remained as one the two most highly used languages on GitHub since 2012; it has since been overtaken by the languages Python and JavaScript. One of the drivers to the popularity of Python and JavaScript is their syntax design that makes it easier to learn and build with.

As a compiled and statically typed language, Java enables faster runtime speeds than Python and JavaScript, despite this, it is still criticised for its runtime speed in comparison to other object-orientated programming languages such as C++. This blog will look 'under the hood' of Java to understand what happens to code when we run it and why it may have a performance disadvantage to its other object-orientated counterparts.

Compiling vs Interpreting

When we write software code, we are, more-often-than-not, writing in a high-level language that is to be converted and understood by the computer. The high-level code contains easy-to-read syntax that is then compiled into a lower level language (such as machine/binary code) that is to be executed by the CPU. The process of taking the high-level code and translating it into a lower level is often a defining feature of a programming language that will underpin the performance and properties of the language.

It is important to distinguish between the word 'compile' and 'interpret' before we begin exploring the Java engine. 'Compile' involves converting a language into a different level of language (often down). Interpret however involves converting the language into a similar-level language. A good analogy would be an English to French interpreter who simply converts one language into the other at the same level of understanding.

The Java Compiler

Before Java code is processed by the Java engine, it is first compiled with the Java compiler. The Java compiler will first inspect the code to ensure it can be processed by the Java engine. Code inspection includes checking for TypeErrors (such as passing in the incorrect parameter to a method) and Syntax errors. Compile errors are usually captured by a modern IDE such as IntelliJ or Eclipse as it actively analyses code as it is written.

The Java compiler is later used to convert the high-level Java programming language code into a lower level 'bytecode' that is similar to machine code; as a result, the .java files are compiled into .class files of bytecode (see the /target directory of your project after creating a build). The bytecode is later received and processed by the Java engine.

Code Interpretation and Execution

The Java engine that performs the compilation and interpretation of code is known as the Java Virtual Machine (JVM). The JVM resides within the RAM where it receives the bytecode following the first compilation step. The JVM acts as a runtime engine that enables Write Once Run Anywhere (WORA); unlike other languages such as C++, Java code is not converted explicitly for the operating system that is running the code, but for the JVM which will be identically running the application regardless of the operating system.

The JVM steps for processing code is split into 3 key sections: 1. Class loader - where the files are loaded into the JVM to be later stored and used. 2. Runtime data area - where memory area is designated for classes to be stored for execution. 3. Execution Engine - where bytecode is interpreted into machine code using the Just In Time Compiler and is executed.

Class loader

The Classloader is responsible for loading the .class bytecode files from the target directory into the JVM. This process is known as loading.

After loading, a second phase called linking begins. During linking, memory is designated for static variables of each class and the default values are assigned. The processing of linking is divided into three steps: first the type safety of Objects are verified, second, the memory required by the Object is allocated, and the final stage of resolution is the process of replacing the symbolic references of classes with real class references.

The final stage of the class loader is called initialisation. Initialisation involves invoking the initialisation method of the class (use of the 'new' keyword) and the static blocks e.g:

public class MyClass{

    static{

        //static initialisation block

    }
}

Runtime data area

The runtime data area contains the compiled code as well as the methods and properties of classes that are used for execution. The sequential instructions (which piece of code to execute) of the application are followed within the runtime data area as Objects are managed and used accordingly.

The Runtime data area is split into 5 different areas with its own responsibility:
1. Method Area: An area shared across all threads of the runtime environment, the method area stores static variables of each class.
2. Heap Space: Stores the objects and instance variables being created and used at runtime. The area is managed using Garbage collection.
3. Stack Memory: Each thread contains its own stack memory. This is where local variables and memory from method calls are held. When the memory being held on the stack exceeds its limit, a StackOverflowException is thrown.
4. Program Counter Register: Acts as a pointer to the current instruction of the program.
5. Native method stack: Similar to the stack memory, however it is used for executing methods that are not written with Java.

Execution engine

While the runtime data area manages the Objects and properties of our classes at runtime, the execution engine is responsible for executing the logic of the application with the computer, assisted by the various areas of the runtime data area.

There are 3 main components to the engine:
1. Interpreter: Responsible for reading the bytecode into machine code to be executed in a sequential manner.
2. Just in time (JIT) compiler: Provides support to the interpreter. Rather than repeatedly interpreting the same bytecode from repeated method calls, the JIT will use its profiler to identify 'hotspot' areas within the application that are repeatedly executed. The JIT provides a performance enhancement for the interpreter.
3. The garbage collector. Responsible for allocating and removing memory storage for the heap space Objects.

Why Java is perceived as 'slow'

The broadly incorrect reputation Java acquired from its earlier days with 'slow' processing speed came by its comparison with other object-orientated programming languages. The present existence of Java (currently version 16) includes far greater performance optimisation than its first versions. Despite this, Java still retains certain 'slow' features that are embedded by its design and implementation. The 'slow' features include the following:

1. Multiple-compilation
The Java code we write is compiled twice: once with the Java compiler into .class files, and sometimes again within the JVM by the JIT compiler. The JIT compiler acts as an optimizer, preventing the JVM from repeatedly interpreting code again and again. Instead, code is compiled to native machine code that can then be executed. Compilation happens to be a major driver in execution times, and can therefore impact the overall speed of Java.

2. Java encourages code to be written for correctness and readability, rather than performance.
The linking process of the ClassLoader creates a large startup time (known as warm-up time) as it allocates memory and assigns references to Objects for the application to begin. The process of linking occurs each time the application is run, however in other languages such as C, the linking process only occurs once following compilation.

3. The Java String Object
Java stores and handles Strings in a unique manner with their own memory area called the String constant pool. The combination of how Strings are handled as immutable Objects and their use of UTF-16 encoding, contributes to a large memory requirement and inhibition upon performance.

Summary

In this blog, we have taken a look at how the high-level code we write undergoes many transformations and movement within our CPU as it is executed. Java, like all languages, has its own style and design to compiling and interpreting code that provides the developer with unique features such as WORA and static typing.

When comparing Java with other languages, there will always be arguments for and against where the properties of compilation time, execution time, syntax, and portability all play a vital role on a case-by-case basis. By understanding these concepts with reference to Java, you can begin to understand why certain technologies can be more appropriate to use in certain contexts, and when Java can be the technology of choice for you.

Enhanced exceptions testing with catch-exception

Mon, 31 May 2021 00:00:00 GMT

The catch-exception library

Learn about the limitations to conventional exception testing from JUnit and resolve them using the catch-exception library from googlecodes.

The catch-exception library allows us to control a test method to capture an exception that can then be asserted upon with more detail and upon the runtime behaviour of our test.

How to choose the tests to write

Mon, 24 May 2021 00:00:00 GMT

Choosing tests to write for a method

Learn about the different types of data values we might want to consider when testing our methods for the different type of contexts that they may be used in. Following this video, you will understand the differences between expected, boundary and strange values that can support enhance the test cases you write.

Using Mockito Verify

Mon, 17 May 2021 00:00:00 GMT

Mockito Verify

Learn about the when you would want to use Mockito's verify assertion within unit testing and how you would use it.

In this video, I have introduced the concept of the verify assertion using an example to demonstrate how it can be beneficial to covering behaviour cases of an application where there are no return types or method calls made within the method.

Mock, Stub, and Spy with Mockito

Mon, 10 May 2021 00:00:00 GMT

Mock, Stub and Spy

Learn about the differences between a mock, a stub, and a spy for the purposes of creating test doubles with the Mockito framework. In this video, I have used an example to demonstrate when we would want to use each of the different test doubles and the unique benefits they can offer when automation testing.

Getting started with Mockito

Mon, 03 May 2021 00:00:00 GMT

Getting started with Mockito

Learn about what a mocking framework is used for with unit testing, when we it would want to use it, and how we can use the Mockito framework within our tests. This tutorial involves testing a class method in isolation despite its dependency upon a database call. We will understand why mocking is useful and how we can approach it in 3 steps.

Testing for exceptions with JUnit

Mon, 26 Apr 2021 00:00:00 GMT

Testing for Exceptions with JUnit

Learn how you can test for expected exceptions that are thrown from your methods with JUnit.
With just a single line, we can increase our test coverage to expect exceptions to be thrown from our class behaviour.

When and how to use parameterized unit testing

Mon, 19 Apr 2021 00:00:00 GMT

Parameterized unit testing

Learn about how and when you could benefit from creating a test class that uses the JUnitParams package in just 4 steps. The JUnitParams package will enable you to increase test coverage of a method without necessarily creating new test methods for it, instead, we can pass in different parameters that use the method differently and express a different expected result.

Using the @Before Annotation with JUnit

Mon, 12 Apr 2021 00:00:00 GMT

Using the @Before Annotation with JUnit

Learn about how and when you would use the @Before annotation with JUnit by exploring its use with an example. The @Before annotation will enable you to remove repeated code within your test classes, typically when instantiating classes and defining the 'given' phase of unit testing.

Using the different JUnit assertions

Mon, 05 Apr 2021 00:00:00 GMT

JUnit Assertions

Learn about the different assertions that can be used with the unit testing framework: JUnit. Understand when and how you can apply different assertions to optimise and improve the readability of your test cases.

Using the Single Responsibility Principle

Mon, 29 Mar 2021 00:00:00 GMT

Single Responsibility Principle

The single responsibility principle is a software development pattern that asserts each class to have a single purpose. In this tutorial, we use the single responsibility principle by refactoring a multi-purpose class into single-purpose classes and explore the various benefits of this design.

OOP Concepts: Polymorphism

Mon, 22 Mar 2021 00:00:00 GMT

Polymorphism

Polymorphism is one of the four concepts of object-orientated programming. This video is a step-by-step guide on how you can use polymorphism within a Java application. There will be an overview of why polymorphism is appropriate to use and how you can use static and dynamic polymorphism to encourage code reuse with an example within IntelliJ.

Polymorphism is used in our design of how we reuse method names within the same class and with inheritance to create different implementations of the method. The details of how the methods are defined with parameters, and how the class that is calling it is created will ultimately determine which implementation of the method is used at runtime.

OOP Concepts: Abstraction

Mon, 15 Mar 2021 00:00:00 GMT

Abstraction

Abstraction is one of the four concepts of object-orientated programming. This video is a step-by-step guide on how you can use abstraction within a Java application. There will be an overview of why abstraction is appropriate to use and how you can enhance the simplicity of your code with an example within IntelliJ.

Abstraction is used in our design of how we group behaviours of classes together. The details of how the behaviours are used by the class is delegated to the class itself, however abstraction defines a contract that enforces such behaviours to be handled by the class.

OOP Concepts: Inheritance

Mon, 08 Mar 2021 00:00:00 GMT

Inheritance

Inheritance is one of the four concepts of object-orientated programming. This video is a step-by-step tutorial on how you can develop software using inheritance with Java. I use a demonstration within IntelliJ by creating a football team where inheritance is used to make it easier to create and update classes with similar properties and behaviours.

OOP Concepts: Encapsulation

Mon, 01 Mar 2021 00:00:00 GMT

Encapsulation

In this video, I have taken a step-by-step approach to using encapsulation in software development.

Modern portfolio theory and fund diversification

Mon, 15 Feb 2021 00:00:00 GMT

Key Takeaways
• Understand modern portfolio theory and its application to portfolio construction.
• Use statistical measures such as variance, covariance and standard deviation to simulate portfolio risk and return profiles.
• Understand the properties and differences between the asset classes that can be combined to diversify a portfolio.

Modern portfolio theory

Modern portfolio theory (MPT) was first introduced in 1952 by Harry Markowitz to describe how risk-averse investors can achieve optimal returns from their investments at a given level of risk. Harry later received a Nobel prize for MPT, which is still valued to the present day.

MPT involves a process of identifying how the assets within a portfolio of investments contribute to the funds overall profile for risk and return. Through MPT, we are able to understand how the assets can be combined to create different risk and return values, and where an optimal mix of the assets can achieve the highest level of expected return for the fund at a given level of risk.

It is often accepted that a larger appetite for risk may yield larger returns for the investor. Despite this, a portfolio of assets may not always yield the same increments of returns for the additional risk taken, thus the relationship between risk and return is not linear.

In this blog, I will look at the some of the key statistical measures involved with MPT to understand what they tell us about a portfolio of assets, and what their criticisms are. You can find the formulae for all calculations from my Excel sheet available on GitHub. I will also review how the optimal mix of assets can be found for a portfolio. I will conclude the blog by reviewing the different types of assets that can be included within a portfolio and how MPT has since been adapted and applied within industry.

Asset and portfolio risk

We can divide risk into two types: systematic and unsystematic risk. Systematic risk, also known as undiversifiable risk, is implicit with the market; it includes risk from market observables (such as interest rate or inflation). Unsystematic risk, also known as diversifiable risk, is present with the nature of the asset and can be mitigated. One example of an unsystematic risk may be the change in regulations to the airline industry which subsequently caused the likes of Ryanair and EasyJet stock prices to tumble during Covid-19 lockdowns.

MPT calculates the risk of an asset from its variance which is later transformed into the standard deviation. The variance of an asset is used to understand how 'spread out' the values are within the data set. With MPT, variance is used to understand how the average change in stock price differs from the mean change of all the stock prices. In this blog I will use the stock prices of 2 different assets (Company A and Company B) to demonstrate MPT. I will construct the values based upon the past 6 days of their stock price, however in reality a much larger time frame will be used for more meaningful statistical measures of the two assets.

The past 6 stock prices for Company A are: 70.22, 88.21, 84.98, 85.43, 78.81, 88.20.
The past 6 stock prices for Company B are: 122.65, 121.11, 132.56, 124.5, 140.7, 130.5.
For the two assets, the mean values derived represent the average percentage change in the stock price from the previous stock price. For Company A, this is 5.41%, and for Company B, this is 1.58%.

To calculate the variance, I will subtract the percentage change in stock price for each price away from the asset's mean and square the value. I will then take the average of the individual variance values to obtain the variance for the asset.

I have used Excel to calculate the variance values, you will also see the field 'SD' which stands for standard deviation. The variance of Company A and Company B is 0.0149 and 0.0067 respectively; these values, however, can become distorted as the difference between each stock price deviation and the mean is squared.

The standard deviation provides a normalised version of the variance to reduce the impact of an outlier on the final value. The standard deviation is the square root of the variance and provides a contextual value. For Company A and Company B, the standard deviation is 0.122 and 0.082 respectively. The standard deviation tells us that the Company A stock deviates from the mean percentage movement on average by 0.122%, whereas the Company B stock deviates from the mean percentage movement by 0.082%.

Constructing risk measures in a portfolio of assets

The risk profile of a portfolio is derived from the combined risk of the assets along with the respective weight they contribute to portfolio. Before we begin constructing a portfolio of assets, we must first understand how the risk profile of two assets move with respect to each other.

Covariance is a statistical measure used to determine the directional relationship between two assets. Covariance does not capture the strength of a relationship between the assets, but just the direction of the movement. If two asset prices on average move up together, they will have a positive covariance value. Alternatively, if the assets tend to move in different directions to each other, they will have a negative covariance.

The formula for covariance is below:

We describe cov(x,y) as the covariance between x and y. This is the sum of each point of x less its mean, multiplied by each data point of y less its mean, all divided by the sample size less 1.

The covariance of Company A and Company B is calculated below:

The covariance value of -0.007509 tells us that there is a slight negative relationship between Company A and Company B. As a result, the average movement of one of the stocks would create a slight opposite movement in the other stock (if one goes up, the other goes down).

The covariance tells us the portfolio would expect a lower risk profile when the assets are combined than when they are separate. A negative covariance would mean the portfolio value does not exclusively fall to the same amount that an individual asset value would fall; the negative covariance in the other asset would offer a protection to some degree against the fall of one asset as it will theoretically rise in value. The relationships are implied from the historical data, however future behaviour between the assets may not always be so true.

The combination of differing assets within the same portfolio creates diversification. The portfolio becomes more diversified as it adds assets which have contradicting risk exposures, thus reducing the combined risk of the portfolio.

Using MPT to find the optimal mix of assets

In this section, we will look at how the key statistical measures can be combined to create a portfolio with a single risk and return value. The portfolio can be created in many different ways by combining the assets with different weights. For example, creating a portfolio with 10% of Company A stock and 90% Company B stock.

We can simulate many different portfolio combinations in Excel by changing the weights of the assets in the portfolio. The portfolio expected return percentage will be a sum of each asset mean percentage multiplied by the asset weight in the portfolio.

The formula for finding the portfolio variance is slightly more complex:

In the formula, w represents the weight of the asset in the portfolio, and the σ represents the standard deviation of the asset.

Constructing and using the efficient frontier

The 'efficient frontier' is a curve that is drawn upon a graph of the different risk and return portfolios that can be achieved with two assets. In the real world, this may be used with two portfolios rather than assets to capture more complex relationships. From the table which we have created of the weights, expected return and expected standard deviation of the portfolio, we can create a scatter graph to visualise the information:

We can break the graph into different sections to illustrate how a portfolio can be constructed:

The red line represents the efficient frontier, where the optimal mix of assets can be achieved for a given level of risk. It is not possible to construct a portfolio with the two assets to achieve a risk and return profile above the red line, however it is possible to construct one that falls below. If the portfolio risk and return profile is below the line, it is not regarded as optimal with respect to the risk that is taken. The image below illustrates the construction of the portfolio for 100 different weight combinations, rather than the 9 shown before:

We can now see how many more portfolios can be built, and how many more sub-optimal portfolios that are possible below the efficient frontier. By understanding how the two assets move on average against (or with) each other, we can begin to understand how a portfolio of the assets can be created to maximise returns for a certain risk level.

A typical portfolio of assets may not just consist of two assets. Once a portfolio has been constructed, it will have its own risk and return profile that is living with the assets it contains. Further assets may be evaluated before being added to the portfolio by using an understanding of how the risk and return of the asset complements or hinders the expected performance.

When reducing the unsystematic risk from a portfolio, a manager may take an approach of looking at different types of securities, or to look at various asset classes to seek portfolio diversification. An asset allocation fund is a type of fund that creates diversification by balancing a mix of assets. Each asset classes will provide the fund with unique properties from its exposure and return that can be combined to create a well diversified portfolio.

Diversification with asset classes

Depending on where you look, you will find that assets are divided into a varied number classes, typically from 3 to 8. The advent of new derivatives and exotic investment products have introduced new types of asset classes including commodities, real estate, financial derivatives and most recently, cryptocurrencies. The three largest asset classes are equities, fixed income and cash/cash equivalents. In this section, I will reflect upon 5 asset classes to understand their unique properties:
• Equities
• Bonds/fixed income
• Cash and cash equivalents
• Real estate and tangible assets
• Financial Derivatives

Equity
Equity, also known as stocks or shares represent a percentage ownership in a publicly listed company. Equities are often classed by their market capitalisation into small-cap, mid-cap and large-cap. Investors of equities benefit from a potential dividend and the value of the resale of the equity in the market. While equity provides benefits such as company ownership and a right to vote on certain decisions, they are viewed as risky investments due to their large dependence upon company performance.

The payments to an equity holder are less certain than other securities such as those of bonds. Common-stock holders receive payment once all other cash flows to the company have been paid, such as workers, taxes and bondholders. The dividend payments are also not a guaranteed payment as they are reflective of company performance. The value of the equity can also deteriorate under the company performance as the supply and demand of the stock changes within the market.

Bonds and Fixed Income
Bonds represent an issued loan from a company that is repaid over the life of the bond. The bond will include a maturity date - when the principal of the bond is repaid to the investor and a coupon rate - the interest repayment to the investor at set intervals until maturity. Bonds are not always sold at the value of the principal. Bonds can come in many different forms such as zero-coupon bonds, corporate bonds and municipal bonds - all of which offer different levels of return and coupons.

Bonds are often considered less risky than equity for several reasons:
• Bond investments come with a promise to be repaid at maturity.
• The fixed interest repayment on the bond is guaranteed (unlike a dividend payment on equity).
• Bond owners have higher priority during liquidation than equity holders.

Cash and cash equivalents
Cash and cash equivalents are financial securities that have two key properties: high credit quality and high liquidity. Cash equivalents are also known as money market instruments that consist of low risk and short-term securities such as T-bills, certificates of deposit and commercial paper.

The cash and cash equivalents are often shown as a company health indicator as per the balance sheet as it reflects the ability of the company to pay their short-term debts. Cash and cash equivalents may also be used to finance an acquisition, however this is not of relevance for a portfolio of assets.

With respect to a portfolio of assets, cash and cash equivalents will have a low risk and return profile with high liquidity. The T Bills and other securities will carry a low duration due to their short maturity dates thus providing an opportunity to mitigate portfolio exposure to interest rate risk.

Real estate and tangible assets
Real estate and tangible assets reflect securities that derive value from real-life assets that contain an intrinsic value. The link between the security value and the asset means real estate and tangible assets offer a hedge against inflation in the currency for a portfolio. Tangible assets will also typically have very little in common with financial securities to offer a new element of diversification to the portfolio. Real estate and tangible assets offer a stability of payments to the portfolio, however a reduced liquidity due to the time it can take to sell the assets.

A commodity is also considered a tangible asset: an economic good that has a fungibility. Commodities are divided into two types: hard commodities that are mined such as oil and gold, and soft commodities such as harvest (rice, wheat) or livestock (meat, cattle).

FX and other derivatives
The foreign exchange (FX) market is a relatively new asset class since a free-floating exchange had been enabled following the collapse of the Bretton Woods system in the 1970s. FX rates offer a portfolio a hedge against currency risk as the value of other securities can vary with the FX market. FX also offers portfolio growth as speculation in a currency against other currencies can create greater values in securities over others. The FX market is often captured through the derivatives market where they are swapped or secured through contracts.

Derivatives represent a contractual agreement between parties where the value is 'derived' from an underlying security or asset. Common derivatives include a future or forward contract, which is an agreement to pay a certain price for a certain quantity of an asset at a date in the future. Other derivatives include options, which provide a right to buy or sell a security at a point in the future, and swaps, which are exchanges of assets over a period of time.

Derivatives are traded both on an exchange and over-the-counter (OTC) which provide various ways for them to be customised to suit the preference of a portfolio. Derivatives can be used to both hedge a risk (such as with a swap) or to speculate upon a price (such as a call option). Portfolios can therefore benefit from derivatives to achieve bespoke risk and return benefits.

Modern Portfolio Theory today

An understanding of the different asset classes can provide a small amount of colour to the vastly complex world of portfolio construction and assets. MPT has provided a well-respected, yet simplified model at understanding how assets can be combined to create a variety of expected returns at a level of risk. Since its inception, limitations have been identified with MPT in-line with other theoretical models of finance: investors are not always rational, buy and hold strategies are not always optimal, past performance is not indicative of future performance, securities are not always available within the market. Amongst the limitations, there are four common criticisms of MPT that may influence the use and perception of the model.

Assumption: Investments upon the efficient frontier are optimal. The curve created by the efficient frontier does not leave the manager with a complete array of portfolio combinations that are optimal with regards to the level of risk taken for the return achieved. The efficient frontier is to be better utilised by drawing a capital market line that accounts for the risk free rate of return for an investment (typically the return of a government bond) and finding a tangent to the point on the frontier. I will review the capital market line in a separate blog. The key point to note is that the efficient frontier isn't equally efficient in risk/return tradeoff at each point along the curve.

Assumption: Returns on assets are normally distributed. The assumption of a normal distribution of returns is reflected in our expectation that the price of an asset may rise and fall with equal probability. The reality of price movement however is often skewed to reflect the general movement of the asset price, be it upwards or downwards as per the asset classes market expectation at the point in time. You may consider technology-based companies over the past 10 years; we wouldn't expect the prices of their shares to equally move up and down through the years, let alone when considering inflation.

Limitation: Risk is defined as both the upside and downside of price movement. MPT obtains the standard deviation of an asset from both its movement up as well as its movement down. The standard deviation through this calculation has been criticised as it is used for understanding the risk of the asset, which would otherwise be reflected in just the downwards movement of the price, and not the upward movement. Post-modern portfolio theory is a methodology that was introduced in 1991 where the standard deviation of the risk of an asset is defined from its downward movement, rather than all of its movement.

Limitation: Covariance to establish relationships between asset movements.. A limitation of covariance is that is can only be used to measure the directional relationship of two assets. The strength of the relationship is not shown, therefore an understanding to the degree at which two assets move with or against each other cannot be gathered or applied to the efficient frontier. A greater measure of the relationship can be obtained using a correlation coefficient. The correlation will normalise the covariance to provide a signal on the strength of the relationship of a scale of -1 to 1.

Summary

In this blog, we have looked at modern portfolio theory to understand how the risk and return of a portfolio of assets might not always be optimised when grouped together. Amongst MPT, we have also looked at how assets will have their own variance and covariance with other securities. The realisation of a negative covariance can enable us to reduce the variance of returns when combining assets.

The understanding of an asset or portfolio standard deviation will provide an investor an insight into the volatility of the assets together. Diversification can come from many different sources including securities with a variety of companies, sectors and geographical regions, however an extended range of diversification opportunity can be realised across different asset classes, all of which have their own unique properties and underlying characteristics within the market.

MPT has remained resilient with time as a tool to understanding portfolio construction and expectations. The criticisms of MPT have lead to further developments and extensions of the theory. In a future blog, I will write about the capital market line that builds on-top of MPT to develop an understanding of the relationship a portfolio may have with the risk-free rate of return that could be otherwise be obtained with other securities in the market. To reiterate, you can find the excel sheet from this blog on GitHub.

Creating a RESTful web service in Go

Mon, 18 Jan 2021 00:00:00 GMT

Key Takeaways
• Use Go to create a RESTful web service step-by-step.
• Explore some of the key classes and methods from the Go standard library for web services.
• Handle client HTTP requests to perform CRUD operations.
• Persist data with a MySQL database.

Introduction to Go web service

In my previous blog, I reviewed the drivers and key properties of the programming language 'Go'. Go's most defining properties made it a lightweight programming language that is specifically well supported to creating applications with efficient compile time, efficient runtime and with ease of programming.

One of the most valuable use-cases for Go extend to its built-in capacity to create powerful and efficient web services with its 'http' standalone package. This blog will review the process for creating a web service with Go and persisting the data in a MySQL database. I will also pay close attention to the packages, keywords and design patterns used in the source code.

The HTTP communication protocol

Hypertext Transfer Protocol (HTTP) is a client-server protocol during data transfer across the web. When a client requests information from a server, a HTTP Request is made by the client and in return the server will issue a HTTP Response. The Request and Response consist of structured data to inform the server of the request details and the client of the processed outcome. When we design and build Go web services, we are required to specify the request and response details that are encapsulated by HTTP within the source code. Before we create the web services, I will briefly run over the HTTP protocol in a little more detail.

The Go web services use representational state transfer (REST) as the protocol to exchange messages. HTTP in this context is the protocol for transporting the messages. REST is used to respond to incoming HTTP requests where the sender of the request (client) uses an API uniform resource identifier (URI) to obtain and modify the state of the resource. In our case, the resource will be data that would be typically persisted by the application datastore.

HTTP requests are made in a stateless manner within the system meaning they are executed independently from all other requests. This means HTTP transactions are sent, completed, and closed in isolation to support huge scaling as sessions are unrelated to each other. The HTTP message will include a packet of information between the client and server that includes a start-line, header and body.

During a HTTP request, the start line will indicate the HTTP method performed (such as GET, POST, PUT, DELETE) along with information about the target server. The header in the request may include supporting information for the server to process the request such as authentication tokens. The body of a request may include JSON data that is required for processing. The body is used during PUT and POST requests where data is updated or added to the datastore.

A HTTP response will use the start line to define a status code explain the result. Status codes exist from 100 to 599. Codes in the 200s represent successful responses, codes in 300s represent redirected requests, codes in the 400s represent client-side error (such as 404), and codes in the 500s represent server-side errors.
The response header may provide information to support client-side processing such as a cache or cookies. The response body would consist of the requested data (such as for a GET request).

In the following section, we will begin to build a Go web service using HTTP to customise messages from the server.

Creating a Go web service

From a high-level view, creating a web service will include the following steps:
1. Create an entity class for storing the instances of the records.
2. Initialise the data store.
3. Define the web service endpoints.
4. Handle GET requests to the server.
5. Handle POST requests to the server.
6. Handle PUT requests to the server.
7. Run the application.

Before we begin writing the bulk of the source code, we must first create the 'main' file to launch the web service. The file will be named 'main.go', it will define the package as 'main' and will include a method named as 'main':

package main

func main(){

}

1. Creating the entity class

Go defines a 'struct' (short for structure) as a collection of fields for custom user-defined types to be used in the application. The struct is used to hold records that are to be included in the application with certain properties. Similar to a Spring web service by marking a class with @Entity and defining its fields, a struct can be defined with the same purpose as below:

type person struct{
    Id int,
    Name string,
    Age int
}

A struct can be considered as a template that is used for creating data records. The use of a struct is made by using the keyword 'type' followed by the struct name and lastly by 'struct'. In this struct, the fields are accessible outside of their package due to the capitalisation of their first letter (Id, Name, Age).

A struct instance can be created in many ways:

Using direct assignments:

var p Person
p.Id = 1
p.Name = "Alice"
p.Age = 25

Using a code block:

p:= Person {
    Id : 1,
    Name : "Alice",
    Age : 25,	
}

Or in a single line:

p:= Person {1, "Alice",25}

2. Initialise a data store

To initialise a storage of multiple Person instances, we will define a variable as an array of {Person}:

var personList []Person

The in-code data storage will be populated by using a special function within Go called 'init'. Init functions are used for initialising variables, checking states and running other one-time computations. Despite how many times the package is referenced, the 'init' function will only ever be called once:

func init(){

    personJson := `[
        {
            "Id" : 1,
            "Name":"Alice",
            "Age": 25,
        },
        {
            "Id" : 2,
            "Name":"Bob",
            "Age": 22,
        },
        {
            "Id" : 3,
            "Name":"Charlie",
            "Age": 23,
        }
    ]`
    err := json.Unmarshal([]byte(personJson), &personList)
    if err != nil{
        log.Fatal(err)
    }
}

So what have we achieved?
Firstly, the personJson declaration defines a JSON within back ticks.

From lines 3 to 19, the JSON data is used to initialise 3 people into the list with their Id, Name, and Age.

From line 20, a second method call is made that returns an error code 'err'. The method call to json.Unmarshal will transform the JSON into the array of Person instances: personList.

The Unmarshal method parses JSON-encoded data and stores the result within the value pointed to by the second argument. The documentation defines the use of two parameters and to return an error code:

func Unmarshal(data []byte, v interface{}) error

The Unmarshal method performs the reverse of the Marshal method (later visited) by transforming the byte array 'data' into the interface 'v'. From line 20 in the above code snippet, we perform a check upon the error response of the Unmarshal method to verify that it is not equal to Nil. The following code block is frequently used with Go for checking errors:

 if err != nil{
        log.Fatal(err)
    }

3. Handle requests

So far, we have created a data record type (struct) and populated an array of records using the Unmarshal method for parsing JSON. Before we access the data, we must define the endpoints of our web services which are later accessed by the client. The Go web service endpoints are defined in one of two ways:

http.Handle("/home",&Person{Name: "Steve"})
http.HandleFunc("/about", aboutHandler)

The Handle function allows us to register a handler that will service the request for the pattern that matches "/home".
The HandleFunc allows us to assign a function to directly handle in the incoming request when the pattern matches "/about".

The Handle interface from the documentation appears as following:

func Handle(pattern string, handler Handler)

The Handler interface contains a single method: ServeHTTP(ResponseWriter, *Request) which uses a ResponseWriter and a pointer to a Request object to return a response. To define our own implementation of Handler, we will hang the ServeHTTP method on the Person struct by providing an implementation of the method.

func (p *Person) ServeHTTP(w http.ResponseWriter, r *http.Request){

    w.Write([]byte(p.name))
}

On line 1, we first specify a new function with 'func', second, the '(p *Person)' section will specify the Person struct has the method hung from it. By convention, w is used to denote the http.ResponseWriter. The ResponseWriter is used to construct the HTTP response from the server to the client. The HTTP start line, headers, and body can all be defined here. The http.Request instance is conventionally denoted with 'r' and is used to assess the incoming request properties.

The HandleFunc method allows us to handle HTTP methods, however instead of taking a handler as a second argument, it takes a function that has the same signature as ServeHTTP. The ServeHTTP function uses parameters of a ResponseWriter and a pointer to a Request object, therefore our function 'aboutHandler' must do the same:

func main(){
    http.HandleFunc("/about", aboutHandler)
}

func aboutHandler(w http.ResponseWriter, r *http.Request){
    w.Write([]byte("About page"))
}

4. Handle a GET request to the server

The personList can be obtained through the web service by defining a handler to a GET request made on a URL pattern '/people' and returning the personJson array variable to the HTTP body.
First, we will register the request to be handled with HandleFunc:

http.HandleFunc("/people", peopleHandler)

Second, the peopleHandler method is designed to return the peopleJson in the body.

func peopleHandler(w http.ResponseWriter, r *http.Request){

    personJson, err := json.Marshal(personJson)
    if err != nil{
        w.WriteHeader(http.StatusInternalServerError)
        return
    }
    w.Header().Set("Content-Type", "application/json")
    w.Write(personJson)
}

On line 1, the peopleHandler function is defined with the same signature as the ServeHTTP implementation from the Handler interface.

On line 3, the personJson response is defined by calling the json.Marshal method on the initialised personJson array. The json.Marshal interface is defined in its documentations with the below signature:

func Marshal(v interface{}) ([]byte, error)

The Marshal method will do the opposite of the Unmarshal method. The response from Marshal is the JSON data in a byte array and an error code. On lines 4 to 7, the error is handled with a HTTP start line of StatusInternalServerError.

On line 8, a good response is deemed to have been received so we set the header to return content type of JSON and on line 9 we write the HTTP response body to be the JSON returned from the Marshal method.

The GET method will now return the JSON data or an empty body and an internal server error (HTTP code 500).

5. Handle POST requests to the server

For the POST request, we are aiming to add the JSON data into the array of Person instances. We are able to map multiple types of HTTP requests to the same URI to be managed by the handler. In the peopleHandler function, we can decide how to process the request by assessing the request header. For this, a switch statement can address the different types of methods:

func peopleHandler(w http.ResponseWriter, r *http.Request){

    switch r.method{

    case http.MethodGet:
        personJson, err := json.Marshal(personJson)
        if err != nil{
            w.WriteHeader(http.StatusInternalServerError)
            return
        }
        w.Header().Set("Content-Type", "application/json")
        w.Write(personJson)
    case http:MethodPost:
        var newPerson Person
        personDetails, err := ioutil.ReadAll(r.body)
        if err != nil{
            w.WriteHeader(http.StatusBadRequest)
            return
        }
        err = json.Marshal(personDetails, &newPerson)
        if err != nil{
            w.WriteHeader(http.StatusBadRequest)
            return
        }
        if newPerson.Id != 0{
            w.WriteHeader(http.StatusBadRequest)
            return
        }
        newPerson.Id = nextId()
        personList = append(personList, newPerson)
        w.WriteHeader(http.StatusCreated)
        return
    }
}

From line 3, the switch statement begins where the method from the request, r, is cycled upon.

From lines 5 to 12, the GET request from step 4 is captured in its own block for the http:MethodGet case.

From lines 13 to 34, the POST request handler is defined. This method appears fairly long due to the multiple error checks performed at each stage.

On line 13 a new Person instance is defined.

On line 14, the ioutil.ReadAll() method is used to read in the request body and to pass it into a new variable defined as personDetails.

From lines 15 to 18, the error from the ReadAll method call is reviewed before the personDetails JSON from the request body is passed into the newPerson instance using the Marshal method on line 19.

The error returned from the Marshal method is evaluated on lines 20 to 23 before evaluating the Person Id is defaulted to 0. The Id is later assigned using a custom function nextId() to find the highest Id from the list and to increment upon it.

Lastly, the personList is updated to append the newPerson to the list and a response is returned consisting of a 201 code for Status created.

6. Handle PUT requests to the server

Before we can send a PUT request, we need to create a new handler and pattern. When we need to update a user, we must specify exactly which one by a unique identifier (Id). The ID is to be passed into the URL for the PUT request and extracted by the source code. In the main() method, I will create a new HandleFunc with a pattern using a trailing forward-slash:

http.HandleFunc("/people/", singlePersonHandler)

The forward-slash tells Go to use the handler for all requests that define further information in the 'people' endpoint if a more specific pattern is not matched.

The singlePersonHandler method can be split into two parts, firstly to obtain the Id from the URL, and secondly to perform the update to the peopleList accordingly. Part 1 is below:

func singlePersonHandler(w http.ResponseWriter, r *http.Request){

    urlPathSegments := strings.Split(r.URL.path, "people/")
    personId, err := strconv.Atoi(urlPathSegments[len(urlPathSegments) -1])
    if err != nil{
        w.WriteHeader(http.StatusNotFound)
        return
    }
    person, listItemIndex := findPersonById(personId)
    if(person == nil){
        w.WriteHeader(http.StatusNotFound)
        return
    }

}

func findPersonById(personId int) (*Person, int){

    for i, person := range personList{
        if person.Id == personId{
            return &person, i
        }
    }
    return nil, 0

}

On line 3, urlPathSegments is is obtained by splitting the url path from 'people/'.

The personId is then extracted on line 4 by using the strconv.Atoi method to obtain the final parameter from the url. The error is handled accordingly on lines 5 to 8.

On line 9, the person and their index position in the list are obtained through a method call to the function findPersonById. findPersonById will loop through the personList to obtain the person instance and their index position 'i'. The first section concludes by handling a nil person that would be returned by findPersonId if no such Id is found for a person in the personList.

The second section of the function will perform the PUT update:

    var updatedPerson Person
    body, err := ioutil.ReadAll(r.body)
    if err != nil{
        w.WriteHeader(http.StatusBadRequest)
        return
    }
    err = json.Unmarshal(body, &updatedPerson)
    if err != nil{
        w.WriteHeader(http.StatusBadRequest)
        return
    }
    person = &updatedPerson
    personList[listItemIndex] = *person
    w.WriteHeader(http.StatusOK)
    return

The second section will create a new person instance updatedPerson and update the list of people with the new request.

On line 3, the ioutil.ReadAll method is used to read the content of the body from the PUT request. The body of the put request will contain the JSON of the person update. The variable body is therefore populated with the JSON for the updated user.

The error code from ReadAll is handled on lines 4 to 7.

On line 8 the JSON body is extracted into the updatedPerson variables by using the json.Unmarshal method. The error that is returned is then handled on lines 9 to 12.

The second section concludes where the person instance is overwritten with the updatedPerson and the index position in the personList is updated to the new person. A response header of StatusOK is returned.

Section 2 can be converted into a switch statement to satisfy further HTTP methods requests to the url of 'person/' where a GET request would return a single person and a DELETE request would remove the person from the personList:

switch r.Method{

    case http.MethodGet:
        body, err := json.Marshal(person)
        if err != nil{
            w.WriteHeader(http.StatusInternalServerError)
            return
        }
        w.Header().set("Content-type", "application/json")
        return
    case http.MethodPut:
        var updatedPerson Person
        body, err := ioutil.ReadAll(r.body)
        if err != nil{
            w.WriteHeader(http.StatusBadRequest)
            return
        }
        err = json.Unmarshal(body, &updatedPerson)
        if err != nil{
            w.WriteHeader(http.StatusBadRequest)
            return
        }
        person = &updatedPerson
        personList[listItemIndex] = *person
        w.WriteHeader(http.StatusOK)
        return

    case http.MethodDelete:
        personList := remove(personList, listItemIndex)
        w.WriteHeader()

    default:
        w.WriteHeader(http.StatusMethodNotAllowed)
}

func remove(list []Person , index int) []Person{
    list[index] = list[len(list)-1]
    return list[:len(list)-1]
}

7. Run the application

Before the application can be executed, we need to specify a port for which the web service will be run. To do this, the ListenAndServe method from the http package is used to direct the application. ListenAndServe takes two arguments, first a string literal as the address and second a handler. The Handler is often placed as 'Nil' which will cause the handler to use its default 'DefaultServeMux'.

DefaultServeMux is a HTTP multiplexer. The multiplexer is responsible for matching the url of the incoming request to the registered list of patterns that are served.

The web service can be started using the following command from the terminal:

go run .\main.go

If you see an error, use the 'ls' or 'dir' command to ensure you are navigated to the correct directory where the main.go file exists. You can test the web service through the browser or by using an API client such as Postman.

Data persistence

The current web service uses an in-built array that is initialised through the init function to replicate a datastore. When we restart the web service, array is reinitialised and updates are therefore lost. In the following section, we will utilise the database connectivity packages from Go to persist our information in a MySQL database. We will also update the handler functions to interact appropriately with the database.

Creating a MySQL Database

To begin, we will use the MySQL workbench to create a new database and table with the default records of data required using the SQL statement below:

CREATE DATABASE `go_store`
CREATE TABLE `go_store`.`people`(
    `Id` INT NOT NULL AUTO_INCREMENT,
    `Name` VARCHAR(45) NOT NULL,
    `Age` INT NOT NULL,
    PRIMARY KEY (`Id`)
);
INSERT INTO `go_store`.`people` (`Id`, `Name`, `Age`) VALUES (1, "Alice", 25);
INSERT INTO `go_store`.`people` (`Id`, `Name`, `Age`) VALUES (2, "Bob", 22);
INSERT INTO `go_store`.`people` (`Id`, `Name`, `Age`) VALUES (3, "Charlie", 23);

Using MySQL within Go

Go does not include the database driver for mysql in the standard library so we must download it with the following command:

go get -u github.com/go-sql-driver/mysql

The main.go file will import the driver to imply the use of the mysql driver. As the import is not explicitly used within the file, it is marked with an underscore on the import line:

_ "github.com/go-sql-driver/mysql"

We will establish our database connection in a separate package and class: 'database'. The database.go file will use the sql.Open method from the sql package to create a connection. A pool of database connections are defined within the DB type from the sql Go package while the Open method from the sql package is used to open a pool of database connections.

The database.go file will appear as below where a SetupDatabase function is defined:

package database

import (
	"database/sql"
	"log"
)

var DbConn *sql.DB

func SetupDatabase() {
	var err error

	DbConn, err = sql.Open("mysql",
		"root:java@tcp(127.0.0.1:3306)/go_db")

	if err != nil {
		log.Fatal(err)
	}

}

The var DbConn is defined within the package and assigned by the sql.Open method call on line 13. The first argument to the sql.Open defines the driver name. As we are using mysql, it is "mysql", however other driver names can be found here.

The second argument to sql.Open is the data source name. The name is defined using the following pattern: {database username}:{database password}@tcp(127.0.0.1:{port})/{database name}. 'tcp' stands for transmission control protocol which is the protocol used for sending data. The database is hosted locally, thus the 127.0.0.1 IP address.

From the main method, we will call the database SetupDatabase method and run the application to confirm no error has been produced during the connection. Once the connection is established, we can begin to process user requests using SQL queries.

Making GET requests to the database

The first GET request we will need to make will pull in all Person instances from the database:

func getPeople() ([]Person, error) {

	results, err := database.DbConn.Query(`SELECT Id, Name, Age FROM people`)
	if err != nil {
		log.Fatal(err)
	}
	defer results.Close()
	personList := make([]Person, 0)
	for results.Next() {
		var person Person
		results.Scan(&person.Id, &person.Name, &person.Age)
		personList = append(personList, person)

	}
	return personList, nil

}

The getPeople() function will return an array of Person and an error when called.

The Query method from the sql package will return an array of rows from the database and an error following the execution of the query in the argument. The Query statement is a used to return the defined fields in the order as specified by the struct Person. The error is evaluated if the request is not served on lines 4 to 6.

When we reach line 7, we know valid results have been returned (following an error code of Nil on lines 4-6). We then use the Close method from the Rows package to prevent further access to the result set after the getPeople function is executed.

The make([]Person, 0) method is used to create a dynamically-sized array known as a 'Slice'. The personList will be used to store each record as a person instance when it is scanned from the database.

The personList is populated by a for-loop on lines 9 to 13 upon where the database records are iterated upon until there are no further rows.

The Scanmethod from the Rows package will convert the values from each column into the values as specified in the arguments. Therefore the first column will be assigned to the &person.Id, second to &person.Name and third to &person.Age. The getPeople function completes by returning the personList which will consist of an array populated by each record in the database.

After the GET request for ALL people in the database is implemented, we can begin to write a GET request to obtain a single person from the database:

func getPerson(personId int) (*Person, error) {

	row := database.DbConn.QueryRow(`SELECT Id, Name, Age FROM people where Id = ?`, personId)
	person := &Person{}

	err := row.Scan(&person.Id, &person.Name, &person.Age)
	if err == sql.ErrNoRows {
		return nil, nil
	} else if err != nil {
		return nil, err
	}

	return person, nil
}

The getPerson(personId int) function will take in an int value to represent the personId and will perform a search for the user. The method will return a Person instance with an error code.

On line 3, the row item is defined as a single row by using the QueryRow method. The QueryRow method differs from the Query method as it will only return the first result from the database.

On line 4, a new Person is instantiated. The person instance is populated on line 6 with the row.Scan method. The row.Scan method will return the values of the fields as specified by the query. As a result, we pass the values from column 1 into person.Id, column 2 into person.Name and column 3 into person.Age.

The error returned from row.Scan is evaluated against there being 0 records and also against any other errors which are otherwise returned. Lastly, the complete Person instance is returned with an empty error code.

Making POST requests to the database

The insertPerson function is used for making POST requests to the database.

func insertPerson(person Person) (int, error) {
	result, err := database.DbConn.Exec(`INSERT INTO people (Name, Age) 
	VALUES (?, ?);`, person.Name, person.Age)

	if err != nil {
		return 0, err
	}
	insertId, err := result.LastInsertId()
	if err != nil {
		return 0, nil
	}
	return int(insertId), nil
}

The function uses a single Person parameter and returns an int (the personId) along with an error code.

On line 2, the INSERT statement is made using the Exec method without returning any rows. The method will use the person parameter to populate the query to add values to the Name and Age. The database will auto-increment the Id of the person to provide an Id value.

The result from the Exec method will return information such as the number of rows affected. On line 5 to 7, the error code is handled.

On line 8, the lastInsertId method is called to identify the latest Id from the database which will correspond to the new person.

Making DELETE requests to the database

The DELETE function is similar to the insertPerson function where the Exec method is used to call a custom SQL query:

func deletePerson(personId int) error {
	_, err := database.DbConn.Exec(`DELETE FROM people where Id = ?`,
		personId)
	if err != nil {
		return err
	}
	return nil
}

The personId is passed into the method as a parameter and is used to perform the DELETE operation.

Making PUT requests to the database

The PUT request is also similar to the POST and DELETE functions as it also uses the Exec method to perform an UPDATE query upon a user:

func updatePerson(person Person) error {
	_, err := database.DbConn.Exec(`UPDATE people SET Name = ?, Age = ? where Id = ?`,
		person.Name, person.Age, person.Id)
	if err != nil {
		return err
	}
	return nil
}

The query will update the Name and Age of the Person instance where the Id is identified within the database. The handler will define the new Person instance and pass it into the method. An error is returned as per the SQL result.

Updating the handlers for the CRUD requests

The handlers that were previously defined must now be updated to reference the CRUD methods that are defined above. To reflect, there are two handlers to update, one for the "/people" pattern which accesses all records, and one for the "/people/" pattern which accesses an individual person.

The SinglePersonHandler method will be responsible for extracting the final parameter from the URL to make an update to a single row. As the query uses the personId, just 2 HTTP headers are supported: GET and DELETE. The updated method will obtain the Person instance through the new getPerson method:

person, err := getPerson(personId)
	if err != nil {
		w.WriteHeader(http.StatusNotFound)
		return
	}

The switch statement responsible for the GET and DELETE methods are updated as follows:

switch r.Method {
	case http.MethodGet:
		personJson, err := json.Marshal(person)
		if err != nil {
			w.WriteHeader(http.StatusInternalServerError)
			return
		}
		w.Header().Set("Content-Type", "application/json")
		w.Write(personJson)
	case http.MethodDelete:
		err := deletePerson(personId)
		if err != nil {
			w.WriteHeader(http.StatusInternalServerError)
			return
		}
		w.WriteHeader(http.StatusOK)
		return
	default:
		w.WriteHeader(http.StatusMethodNotAllowed)
	}

The GET method will return the JSON of the Person instance as returned from the getPerson method by using json.Marshal. The Delete method will use the personId from the URL to perform the update to the database with the deletePerson method. A default Status Method Not Allowed error will be returned if the HTTP request is not GET or DELETE.

The PersonHandler will be responsible for handling GET, PUT and POST requests under the "/people" pattern from the URL. The GET method is updated to return a list of people using the getPeople() method:

case http.MethodGet:
		personList, err := getPeople()
		if err != nil {
			log.Fatal(err)
		}
		personJson, err := json.Marshal(personList)
		if err != nil {
			w.WriteHeader(http.StatusInternalServerError)
			return
		}

		w.Header().Set("Content-Type", "application/json")
		w.Write(personJson)

The PUT request will use the updatePerson method to perform the update once the request body has been processed and extracted. The person.Id from the request is extracted and performed.

var updatedPerson Person
		bodyBytes, err := ioutil.ReadAll(r.Body)
		if err != nil {
			w.WriteHeader(http.StatusBadRequest)
			return
		}
		err = json.Unmarshal(bodyBytes, &updatedPerson)
		if err != nil {
			w.WriteHeader(http.StatusBadRequest)
			return
		}

		err = updatePerson(updatedPerson)

		if err != nil {
			w.WriteHeader(http.StatusBadRequest)
			return
		}

		w.WriteHeader(http.StatusOK)
		return

The POST request uses the uses the insertPerson method to make the POST request:

var newPerson Person

		personDetails, err := ioutil.ReadAll(r.Body)
		if err != nil {
			w.WriteHeader(http.StatusBadRequest)
			return
		}
		err = json.Unmarshal(personDetails, &newPerson)
		if newPerson.Id != 0 {
			w.WriteHeader((http.StatusBadRequest))
			return
		}

		result, err := insertPerson(newPerson)
		if err != nil {
			log.Fatal(err)
		}
		if result == 0 {
			log.Fatal(err)
		}

		w.WriteHeader(http.StatusCreated)
		return

Once the handler methods are updated, we are able to restart our web service and the send requests to the server to add, update, delete and obtain records from the database. When the web service is shut down, the results are persisted in the database and accessible upon restarting the web service. Use the MySQL workbench or CLI to verify the updates on the database after each client call to see the result of the HTTP requests.

Summary

This blog has taken us through the process of creating a web service using the functions and support from the Go http package and mysql driver package. The blog has also reviewed several keywords of Go to create the CRUD operations using the HTTP protocol. By reviewing the code, line by line, we are able to see patterns for creating endpoints, handling the different types of method requests, and processing the errors and responses into a HTTP response.

If we exclude the SQL driver, we have only used features exclusive to the Go standard library. This out-of-the-box support for web services is what makes Go lightweight to compile and execute. Go also provides comprehensive documentation for the available methods and responses that are possible from the documentation. I would encourage you to review the documentation to understand how Go can be enhanced further with the use of its standard library.

You can find the source code from this blog on GitHub. The non persisted code: here and the source code with MySQL persistance: here.

An Introduction to Go

Mon, 04 Jan 2021 00:00:00 GMT

Key Takeaways
• Understand the drivers for the invention of Go programming language.
• Review the popular use-cases for choosing Go in a project.
• Review the syntax and configuration for Go to begin building applications.

Introduction

Go (often referred to as 'Golang') was created by 3 engineers who worked together at Google during 2009: Ken Thompson, Rob Pike and Robert Griesemer. Go has since been developed into an open source programming language that is used upon many major platforms including Docker, Kubernetes and Terraform. But why is Go used in these cases?

Go came about as a programming language to tackle some of the persistent issues of slow compilation and slow application builds faced by the developers of large, distributed servers at Google. As a result, Go was designed to be the only programming language to serve all 3 sought-after capabilities:
• Efficient compilation.
• Efficient execution.
• Ease of programming.

Languages such as Java and C++ provide efficient compilation and execution, languages such as JavaScript and Python are able to provide ease of programming with efficient compilation, however before Go, there had not been a language to offer the tri-standard.

Go is a statically typed, compiled and lightweight programming language. Go achieves rapid compilation as well as other extensive behaviours to provide developer support with a shallow learning curve, backward compatibility and built-in support for features such as web services and testing.
The extended features of Go can be categorised into 4 key areas:
• Simplicity of implementation.
• Concurrency support.
• Out-of-the-box experience.
• Cross-system performance.

Simplicity of implementation

Go is able to pair rich features with an elegant approach to language design. The framework for testing and profiling Go is provided for multiple types of testing by default. Unit tests can be written in parallel with the code; Go also provides in-built tools to understand test coverage and code documentation.

Go is a multi-paradigm programming language using OOP and procedural code. Go supports the use of types and methods similar to OOP, however it does not use a class hierarchy. Go uses imperative features such as statements and loops to change the state of an Object as well as concurrency for multi-core processing.

The simplicity of Go syntax and minimal nature of its keywords means generics are not supported beyond the built-in data structures of slices and maps. If Go were to add support for custom generics, it will counter the simplicity and clarity objectives while driving up the compile time.

The Go language updates are designed to provide further support for applications that are written in prior versions of the language through backward compatibility. The support for backward compatibility provides confidence to the developers to upgrade with the latest features of Go without risking the functionality of their applications. Other than the exception for security patches and bug-fixes, Go will continue to be completely backward-compatible.

Concurrency

Go was developed during a modern period of computer hardware where the presence of multiple cores to a system were standard by design. Go places the management of multiple cores within its design where other languages may fault around the challenges of maintaining database connections and managing caches. Go achieves its capacity for concurrency through the use of goroutines, a lightweight alternative to individual threads that consequently support a huge scaling capacity of a Go application.

Go features its own network awareness where threads can be executed in a thread-safe manner. Goroutines are able to safely communicate with each other through channels using the Go schedular to assign threads to tasks. Go uses channels to act as a conduit for goroutines to communicate with each other. Channels prevent the potential for race conditions when multiple goroutines access shared memory. Creating a goroutine can be implemented by adding the keyword go before calling a method (line 8 below):

func concatenate(string a, string b) (string){
    return a + b
}

func main(){
    concatenate("hello", "world")

    go concatenate("hello world", "goroutine")
}

In the below example, the <- syntax is used to assign and use a value that is passed to a channel:

    names := make(chan string)

    go func() {
        name <- "John Adams"
    }()

    name :=  <-names

   fmt.Printf("The name we got is: %s", name)

The in-built features for handling multi-thread communication and the go keyword for the creation of goroutines make concurrency easily accessible to the developer. The creation of goroutines has also made concurrency more widely possible without the usual limitation of threads as experienced with Java.

Out of the box experience

Unlike other programming languages, Go does not require extensive libraries and applications to be pulled in to perform the most basic of application designs and uses. The standard library may not be as extensive as other languages, however it focusses on a few core features. Go has been supported by third parties who have built standard libraries to provide support for core components including:
• String manipulation within the application.
• Data exportation to create external archives.
• Web APIs for creating and consuming web services such as the 'net/http' package.
• File handling and operations.
• Testing APIs, a comprehensive in-built test suite.

The Go CLI provides the developer with ample support across the creation of application builds, artifacts and documentation for their source code. From the command line, the developer can execute the following commands to utilise the complete CLI support from Go:

• go build to compile and package a Go application with its dependencies. The resulting executable is named after the first source file.
• go generate to generate Go files by processing source.
• go get to add dependencies to the current module and to install them.
• go test to automate the testing of packages and to print the result to the command line.
• go doc prints the package documentation.

The application profile also performs analysis upon the application consumption of CPU, network, goroutines and requirements.

Cross-system performance

Go contains its own runtime library and is deployed as a single executable binary file. Go eliminates the potential for version or dependency mis-match. The performance advantage of Go is realised from the executable binary file that does not rely on a virtual machine to abstract its execution. Without the VM, Go does not have a write-once-run-everywhere ability like the JVM, however Go builds can be adapted for different operating systems from a single parameter instruction within the build:
For Windows

GOOS=windows go build main.go

For MAC OS

GOOS=darwin go build main.go

The architecture for the platform can also be defined with the GOARCH property.

The benefits of Go as a compiled language can be realised from enhanced execution time where the operating system does not need to interpret the code at runtime. This advantage is also realised from other languages such as C++ over Java where Java uses an interpreter as well as the compiler used by C++.

A final element of Go's multi-dimensional features are its garbage collection feature for heap memory management. This example demonstrates how Go combines the benefits of garbage collection from Java with the compile performance of C++ and an enhanced ease for programming.

Use cases for Go

As previously mentioned, Go provides a unison between some of the most popular languages, taking their best features and combining them into a lightweight language. The features of Go may make it seem appealing for all use-cases, however its standard library may limit its potential. Despite this, there are a few use-cases that really do emphasise the benefits to choosing Go as the language for development to tackle the challenges of compilation, concurrency and simplicity that it addresses.

The primary use cases for Go extend to web services, stand-alone tools and DevOps which lend themselves to the support, scaling and performance attributes of Go.

Distributed services
Distributed services require a vast capacity for concurrency to support communication and data transfer. Go supports distributed services and networks through goroutines and channels. The message delivery and network awareness features of Go and its 'http' package provide extensive support for distributed systems. As a result, Go is a popular language for the creation of APIs and web servers. As a cross-platform language with extensive network features, Go is also popular amongst cloud-native applications.

Stand-alone tools
The single binary output of a Go application make them highly suitable as a utility program. The Go application can quickly launch and perform a service to make them optimal for single-service applications.

DevOps and SRE
Go's 'build once and promote everywhere' principle makes it a valuable language when used for building CICD pipelines and enhancing web scalability and durability. Go-based API builds can also be used for communicating with middleware to take advantage of the networking capability of Go and making them powerful within a CICD process.

Building with Go

The following section of the blog will look at how we can develop a Go project on our machines and the appropriate configuration required. To get started, download Go from https://golang.org.

Configuring a Go project

Before we create a new Go Project, we must navigate to the Go src workspace from where packages and classes are defined. First navigate to the Go directory on your machine. You will find 3 sub directories: pkg, src, and bin. The src folder may not already exist and must therefore be created.

Go workspaces (folders) can be created within the src folder for each project. While it is possible to create a Go workspace at any other directory in your machine, we will later review the benefits of using the Go src workspace directory as we understand how utility packages and class references are made with Go.

Utility packages in Go

Utility packages in Go are similar to creating classes in Java that are later referenced or used as a service by another class. In Go, we can achieve a similar behaviour of different classes that can be referenced from different workspaces.

Packages can contain types, variables and functions that we want to use from other Go packages. Utility packages in Go allow us to separate responsibilities of classes and to create layers to the application. You could think of the layers being similar to a repository-service-controller layer for a web service. Before we can use a utility package, we must first create the class in the Go workspace and install it into the 'pkg' folder of Go using the CLI. The below example demonstrates how a main.go file can access the car.go class as a service:

package main

import "demo/car"

func main(){

    car.MakeSound()
}

package car

import "fmt"

func MakeSound(){
    fmt.Print("vroom")
}

The current main class will show a compile error:

failed to import utility package "demo/car"

We can resolve this problem by installing the car package into the 'pkg' directory on the Go path. The below command will change into the car directory, then install the car package into the 'pkg' folder where it can be referenced.

cd car
go install

The compile error on the main class will no longer appear, and a new folder 'demo/car' will appear within the pkg folder with a car.a file. The package can then be used as a service for other Go applications.

Go syntax

Go is designed with ease of programming as an objective. Amongst the garbage collector for memory management, Go achieves ease through omission of generics, use of types and the use of complete units to express a statement.

I have found learning the Go syntax easiest when using a reference point of Java. The reflection in Java will then allow me to understand and explain the difference in syntax design. The Go syntax was built with a mantra of 'do more with less' to enable an expansive number of actions with a lightweight design. From my initial understanding of Go, I have found the most evident key differences:
• Method names start capitalised (inherited from C#)
• Functions are defined with func(). The function return type comes after the name.
• Go does not handle explicit exceptions, only error values which are returned from a method call.
• Object instances are passed by reference using an ampersand and asterisk syntax.

Creating functions
Functions in Go are marked with the keyword 'func' at the start followed by a name, expected parameters and return type. Unlike Java, Go functions can return more than 1 value, in-fact it is very common for Go methods to return an error as well as a returned value. This is similar to throwing an exception in Java.

An example of a simple function in Go can appear below:

func concatenate(string a, string b) (string){
    return a + b
}

The equivalent in Java would be:

public String concatenate(String a, String b){
    return a + b;
}

Some methods will return an error and a response thus causing two outcomes to be handled by the code. In Java, this is similar to having a try-catch block for a method call, instead with Go, we will express the return as two arguments. The below example demonstrates the Marshal method which will return JSON for a person along with an error from the method call:

json, err := json.Marshal(person)
if err != nil{
    w.WriteHeader(http.StatusInternalServerError)
    return
}

Lines 2 to 5 in the above Go code snippet is a common pattern for handling errors. The error err is checked to be nil before returning a http response (or otherwise returning an error message). The Go function is the equivalent to a try-catch block as below:

try{
    convertToJson(person);
}
catch(Exception e){
    System.out.println(e.getMessage())
}

Naming conventions
Go uses camelCase naming convention for identifiers. The first letter will also determine the visibility of the Object to other packages and classes.
The name provided to Objects, such as 'person', 'Animal' etc, are accessible within and outside their package depending on its naming convention. A capital letter at the start of the name (Animal) will allow the class to be accessed outside of its package. Otherwise a lowercase letter (person) will not be accessible outside its package.

Passing values
Go uses pass-by-value within the source code. This means when a parameter is passed between functions, it will always create a copy of the value and pass it onwards. Go however defines two different types of parameters to be passed: the value or a reference. As Java uses pass by value, where a copy of the Object reference is passed between the methods, the address of the Object is accessed by the calling method:

Person alice = new Person("Alice");
changeName(alice, "Bob");
System.out.println(alice.getName());

prints:

Bob

The reference of alice is passed into the changeName method that updates the name to "Bob" of the Object instance.

In Go, trying to achieve the same objective would not update the instance itself:

func main(){
    person := Person{"Alice"}
    changeName(person)
    fmt.Println(person)
}
func changeName(p Person){
    p.name = "Bob"
}

returns:

Alice

Instead, to pass the actual reference of the instance to the changeName method, we need to use an ampersand (&) to send the reference of the instance, and an asterisk (*) to receive an address of the instance:

func main(){
    person := Person{"Alice"}
    changeName(&person)
    fmt.Println(person)
}
func changeName(p *Person){
    p.name = "Bob"
}

returns:

Bob

The syntax behind the ampersand and asterisk is useful to remember for addressing instances within the application.

Summary

Go is a young programming language that has been developed out of a drive to improve compile time and the ease of programming alongside the enhanced execution time offered by a compiled language. The lightweight design of Go along with its enhanced support for concurrency have made it a bespoke language for various use cases including networking and DevOps. In this blog I have looked at how Go came about, its key use-cases and the key syntax design of the language.

The most valuable features of Go exist from its lightweight design and niche use cases. The Go standard library is designed to offer comprehensive support to networking and concurrency that can make building modern and rapid-fast application performance much easier and simple out-of-the-box. This is an important consideration to make before considering Go for a project and the growth of a project with time.

In my next blog, I will use Go to create a RESTful API service to crete and return JSON data to other services.

Key takeaways from The Phoenix Project

Mon, 21 Dec 2020 00:00:00 GMT

Key Takeaways
• Review the key points and values that can be taken from The Phoenix Project.
• Understand the importance of IT across a business when delivering value to a customer.
• Understand the various elements of DevOps and its importance to creating cohesion between business teams.

The Phoenix Project by Gene Kim, Kevin Behr and George Spafford

The Phoenix Project is an award-winning novel that explores modern-day IT and its inseparable importance to the business operations. The mutual inclusion of business and IT within The Phoenix Project extend the audience of the book beyond IT professionals to provide valuable lessons for all business leaders on delivering value through their IT capability.

The Phoenix Project introduces us to the protagonist Bill Palmer, a newly promoted IT manager at Parts Unlimited who tackles a series of challenges to transform the productivity and chemistry between teams of the same system. The focal challenge of the job regards a major project delivery known as 'Project Phoenix' which acts as milestone to the company and performance review that threatens the IT team to being outsourced.

Amongst the twists and turns of the story, we are introduced to common characters that we find in our own offices that entice us to read further into their professional development. The Phoenix Project introduces the value of DevOps at all corners of the business; unsurprisingly, we will come to find that IT is no longer a business domain to keeping the costs down, but is an interconnected web between all sources of value at Parts Unlimited.

In this blog, I will review the key points and takeaways from the novel that have allowed me to reflect upon the value of DevOps in modern delivery methodologies.

Part 1

Tracking and managing changes
The disorganised state of IT at Parts Unlimited is firstly demonstrated through their handling of an urgent issue where a software change has prevented employee paychecks from being sent. The exact cause of the problem is unbeknownst to anyone amidst the many data sources and their intertwined relationships. There is minimal visibility into the changes made to the system meaning the team were unable to identify who, when or where the critical update was performed. The issue becomes more and more severe as personal identifiable information is at risk thus introducing regulatory pressures to the moral pressure of the company to pay their employees on time.

The insufficient change management process highlighted difficulties in understanding the error which was brashly diagnosed as a 'glitch'. Why were Parts Unlimited unable to identify the source of the problem? The IT team did not utilise an approval chain, communication channel or ticketing system to track and document the updates to the system. As a result, the team were unable to understand where to look for the problem or even when it may have first occurred. As we will later review, the time constraints upon the development team meant the ticketing system along with software testing had been disregarded.

The Three Ways of DevOps
The Phoenix Project introduces three underlying principles of DevOps that are implemented by Parts Unlimited throughout the story. The underlying principles are explored within this blog as a component to maximising workflow from the business to IT, to operations, and lastly the customer.

DevOps Way 1: Systems thinking
The business silos at Parts Unlimited demonstrate their lack of cohesion as marketing publicly communicated a date that was earlier than expected for the deployment of Phoenix. The premature delivery of Phoenix introduced Bill to consequences to breaking the first underlying DevOps way.

Maximising workflow downstream emphasises the importance of the business to act together and not as individual silos. The first way instils an understanding of the system right across the workflow where business silos do not allow a defect to be passed downstream and subsequently take a view of the system and impacts of changes upon others rather than just themselves. Without the systems thinking, the issues experienced at Parts Unlimited before the deployment of Phoenix (such as erratic changes and fixes) were unsurprisingly experienced during the Phoenix deployment.

The 4 types of work
As Bill tries to make sense of the problems at Parts Unlimited, he receives help from an agile 'guru' Erik who uses a plant factory as an analogy to the software delivery at Parts Unlimited. Erik introduces 4 types of work that are constantly worked upon at any business (including Parts Unlimited):
• Planned work - business projects or enhancements
• Internal projects - server migrations and software updates
• Changes - feedback driven updates to the software
• Unplanned work - escalations and fixes to delivered projects.

Amongst the different types of work, unplanned work is the outlier that cannot be accounted for within a delivery plan and must be minimised as much as possible. It is important for Bill to understand the different types of work so new work can be prioritised accordingly. The optimal distribution of work will see the senior members work on planned work while new-joiners undertake the unplanned work.

Constraints within the system
Bill meets Brent, a senior engineer, who is the SME across the IT systems at Parts Unlimited. Brent is soon identified as a constraint to the business who is highly depended upon by multiple work streams as the only person who can resolve their problems. Brent not only has unmanageable levels of work, he is also unable to document his knowledge for others to learn from due to the same time constraint. Brent informs Bill how the ticketing system is also too timely to use thus creating a further dependency upon Brent's expertise in the future.

Brent' responsibilities extend to completing unplanned work thus creating further bottlenecks for multiple work streams that depend upon Brent to fix their problems. The vicious cycle of Brent's knowledge retention and time consumption from fixing problems make the reliance upon him deepen with time. The organisation of the engineers to rely on Brent breaks the optimisation of the team resources where the most senior engineers are to be working solely on enhancing the software and for junior engineers to be learning about the software through unplanned work.

Part 2

DevOps Way 2: Feedback Loops
Part 2 starts on a positive note of leadership bonding followed by a retrospective where Erik emphasises the importance to prioritise and reduce unplanned work. Due-date performance begins to improve however management become anxious to unfreeze their projects. The failure of planned work continues to creates more unplanned work and the business leaders are introduced to the second way of DevOps: increasing feedback loops across the workflow.

The changes within the system are always triggered by failure. Feedback upon changes must therefore be provided across the workflow to reduce the potential for further unplanned work to be created. The leadership are introduced to resilience engineering where the system is designed to withstand failure and the ability to adapt to failure becomes a strength and not a weakness of more unplanned work.

Feedback loops across the workflow are amplified and addressed directly before it can further impact downstream teams and customers. Feedback is supported across more points within the workflow and within non-production environments. The amplified feedback supports software testing for developers before moving changes to production. From a DevOps perspective, feedback loops can be automated across builds and deployments through a continuous integration and delivery pipeline. The CICD pipeline would further prevent the advancement of a software change to further deployment steps if it is detected to fail at any preceding stage.

Utilising test environments
The support for faster feedback at Parts Unlimited came about from a Phoenix change failure that impacted $10 million in sales. The error was traced back to the ineffective testing of a change that had performed as expected in test environments, but differently in production. The need for a preproduction environment, which is a replica of production, became evident to the developers to avoid a similar error in deployment testing. The test environment configuration is therefore important to retain and monitor to support the quality and reliability of testing before deployment to production.

Kanban boards
A Kanban board is an agile project management tool used to visualise work across the business. Kanban boards are first used in part 1 to categorise changes when discussed within the CAB meeting. Changes are grouped by volatility, risk and priority. When Kanban boards were first introduced, some 400 changes were scattered across boards to illustrate the hectic schedule of changes at Parts Unlimited before the deployment of Phoenix.

In part 2, Patty, director of IT service support, begins taking action upon lessons she has picked up from the plant. Kanban boards are used to prioritise changes. The kanban boards allow Patty to visualise the work in progress throughout the organisation and to see how work shifts and does not shift across team boundaries. The visualisation allows Patty to identify where work builds up and to strategically change priorities to focus upon objectives where bottlenecks occur most frequently. As priorities are better aligned with improved visibility across the workflow, the dependency upon Brent is relieved and spread across the teams.

Part 3

DevOps Way 3: Experiment and fail fast
The third way of DevOps requires a culture of continuous experimentation and learning which is fostered through effective support for testing, practice and feedback. Bill explores the concept of 'Takt time'. Takt time is the rate at which a product or feature can be delivered to a customer from its inception. Takt time underpins the agility of Parts Unlimited to adhere to customer demand with their software updates and deliveries. Parts Unlimited set out to achieve the illustrious '10 deploys per day' which requires each team to develop an understanding for where time delays accrue throughout the workflow and how feedback can be retrieved strategically for valuable insight upon deployments.

As the developers review the deployment pipeline, they begin to identify how, step-by-step, it can be automated and optimised from hours to minutes. The developers realise the benefits from an automated pipeline while being encouraged to experiment with new builds that can be tested against prepared environments for feedback.

The benefits of Cloud technology are later raised as Parts Unlimited experience difficulties with their server capacity for testing. The benefit to the developers are practically demonstrated through the release of a software update that prevented 'out of stock' items from being shown on the systems. Further updates including the use of a content delivery network to support greater server usage were considered for enhancing the system thus demonstrating the agile-framed ideas of the team.

Kata
Kata is described within The Phoenix Project to describe the lean processing and 'ironing out' of inefficiencies throughout the delivery of software. A focus upon kata enables Parts Unlimited to achieve the ten deploys per day. The kata of IT at Parts Unlimited demonstrated the presence of IT across the organisation, almost like electricity. The view of IT had shifted from a 'cost to reduce' to a source of competitive advantage. Parts Unlimited evolved their change management process, expectation of IT and care upon decisions for the growth and continued improvement of the company as a whole.

Summary

The Phoenix Project delivers an exciting introduction to DevOps in the form of a novel that navigates the challenges of change across multiple teams within an organisation. The reality of The Phoenix Project extends to the modern day as businesses change their approach to IT and embrace its potential for achieving a competitive advantage or to even maintain cost competitiveness within the market.

While The Phoenix Project is often described to be an exaggeration of reality, it is highly regarded for capturing the most common pain-points for DevOps to drive change within an organisation. Beyond the focal point of DevOps, The Phoenix Project taught me about the challenges of IT before services such as Cloud enabled on-demand compute services. The Phoenix Project offers many lessons to the reader such as to understand how DevOps can transform an IT workforce, or to develop an appreciation for how multi-dimensional the challenges are for IT to support across an organisation. The Phoenix Project in this sense has enabled me to learn backwards about the modern history of IT and its transformation since the start of the millennium to foster the continued growth and adoption across the business.

Related reading to The Phoenix Project
DevOps Guide by Stephen Watts
DevOps by Emily Freeman

Service discovery and load balancing with Spring Eureka

Mon, 07 Dec 2020 00:00:00 GMT

Key Takeaways
• Understand the use of Spring Cloud Netflix Eureka for registering, consuming and load-balancing services.
• Create a Eureka discovery service to register services within a distributed system.
• Register and consumed microservices through the Eureka service.

Netflix Eureka Registration service

Netflix Eureka is a microservice registration tool developed and open sourced by Netflix. Within a microservices environment, microservices will be communicating with each other across different IP addresses and ports thus requiring each service to manage many connection locations that may change with time. Service discovery constitutes the process of registering other services within a distributed architecture. Service discovery supports the horizontal scaling of services; development teams are able to see the instances/CPU that are available for each service from a single point and are able to scale instances of the service to enhance the capability to consume and process requests. Scaling within a monolithic system would have required a vertical scaling across the system to enhance throughput.

The Eureka server is itself a microservice that is used to register and store the location of other microservices within the same system. The services register themselves to the server where they can be identified and located by other microservices when required.

This blog will demonstrate the use of Eureka as a service registry service. The load-balancing feature of the service will be explored as well as the many other benefits of Eureka to report upon health information for services and the system.

What is load-balancing?

Load balancing refers to the efficient distribution of network traffic amongst the available servers, known as a server farm. Within a distributed architecture, individual services are able to communicate with each other, thus constituting network traffic of requests along with external requests made through the system interfaces.

A load-balancer sits in-front of the service instances to monitor the health of the instances and to direct traffic accordingly. If a service instance goes down, the load-balancer will automatically redirect traffic to healthy instances.

Spring Cloud Eureka acts as a load-balancer as well as a discovery service to services within a distributed system.

Creating the registration server

The Eureka server is first created to maintain the pool of services that consistently send a heartbeat of their existence. The discovery service acts as a domain name service or a load balancer as instance health is assessed and requests are routed as appropriate by the registration service for the consumers of the service.

To create the registration service, you must include the spring-cloud-starter-eureka-server dependency to add support for the Eureka server along with Netflix Ribbon, a client-side load balancer.

<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>

The main class of the new Spring project will be annotated with @EnableEurekaServer. The annotation instructs the service to act as a Eureka server:

@SpringBootApplication
@EnableEurekaServer
public class DiscoveryServerApplication {

	public static void main(String[] args) {
		SpringApplication.run(DiscoveryServerApplication.class, args);
	}

}

In the application properties file, the server is configured to a port, and boolean values for registration instructions:

server.port=9001
eureka.client.fetch-registry=false
eureka.client.register-with-eureka=false

The server.port specifies the port the Eureka service is located. By default, it is port 8761.
The fetch-registry will instruct the server not to store the services within its cache. We will look into this with a bit more detail further in the blog.
The register-with-eureka property will instruct the server to not register itself with itself. By default, you will find the server as the only service registered as it will otherwise register itself as a service.

If we navigate to localhost:9001, we can find the server without any instances registered to it:

The 'Environment' at the top of the page is defaulted to 'Test', however can be overridden within the application properties file with eureka.environment key. The section at the top right will show the uptime of the server and the renews threshold and renews. The threshold defines the number of expected heart beats it expects per minute.

In the second half, information including the available memory and CPUs are also shown with the memory usage and percentage for the registered services. As we add instances to the Eureka server, we can review how the general information updates.

Configuring a Discovery client

I will use two services to demonstrate how the Eureka server can be used between microservices to consume each other. The demonstration will include a HR-Service which represents a HR department, and a Phone-book service, which includes a service that returns user names and their phone number. The HR-Service will consume the phone numbers from the Phone-book service to retrieve contact information.

Both clients to the Eureka server will register themselves with the help of the spring-cloud-starter-eureka dependency. The dependency also comes with further properties that can be configured for custom behaviours. The Spring-web dependency is also required for the REST service provided by the client.

<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-web</artifactId>
</dependency>

Adding @EnableDiscoveryClient can be used to activate the discovery client to the server. This is a marker annotation for others to know it is a Eureka client. We do not necessarily have to use it. Alternatively, you can use @EnableEurekaClient for a direct implementation of discovery server. @EnableDiscoveryClient will instead select the implementation (Consul, ZooKeeper, Eureka) from the classpath and assign it.

The properties file of the Phone-book service will be configured accordingly with the Eureka server:

server.port=9002
spring.application.name=phone-book
eureka.client.serviceUrl.defaultZone= http://localhost:9001/eureka/

The application.name defines the name of the service to be used as a key within the Eureka server. The serviceUrl.defaultZone will instruct where the Phone-book service will try to register itself with a Eureka Server.

Both services will include a model class called 'Contact' to store a basic Contact Object with name and number. The Phone-book will later create a list of the Contact Objects and the HR-Service will consume them over a HTTP call.

public class Contact {
	
	private int number;
	private String name;
	
	public Contact(int number, String name) {
		super();
		this.number = number;
		this.name = name;
	}
	public int getNumber() {
		return number;
	}
	public void setNumber(int number) {
		this.number = number;
	}
	public String getName() {
		return name;
	}
	public void setName(String name) {
		this.name = name;
	}
	
}

Within the Phone-book service, the controller class will be used to create and return a list of Contacts:

@RestController
public class PhoneController {
	
    @GetMapping("/contacts")
    public ResponseEntity<?> getContacts() {
    	
	 List<Contact> contactList = new ArrayList<>();
		contactList.add(new Contact(123, "Alice"));
		contactList.add(new Contact(456, "Bob"));
		contactList.add(new Contact(789, "Charlie"));
        	
        return ResponseEntity.ok(contactList);
    }	
}

The above getContacts() method simulates a call to a data store. The ResponseEntity return type of the controller defines a HTTP response that can be consumed and transformed by another service.

If we navigate to the '/contacts' endpoint from the Phone-book service, we can retrieve the list of Contact instances:

We can also check the Eureka service to confirm the Phone-book service is registered. The Eureka service will typically take 30 seconds to show the new service registered as it requires 3 consistent heartbeats from a service for it to be registered.

The number inside the brackets will tell you how many instances of a service with the same name that are running. If we add more services, the number will increase and the status will show the address of each of the services:

Now we can consume and test load-balancing to the instances!

Consuming the Phone-book service

The HR-Service is going to be built out to consume the Phone-book service and to obtain the appropriate Contact instances. The HR-Service will use the RestTemplate class to consume the HTTP body of the Phone-book 'contacts' endpoint. The RestTemplate is a high-level API to the Spring framework that is used for executing synchronous HTTP requests. The RestTemplate will be defined as a Bean within the main class to be later injected:

@Bean
RestTemplate getRestTemplate(){
	return new RestTemplate();
}

The properties file will look similar to the Phone-book service, with the addition of eureka.client.fetch-registry:

server.port=9003
eureka.client.fetchRegistry=true
spring.application.name=hr-service
eureka.client.serviceUrl.defaultZone= http://localhost:9001/eureka/

The fetch-registry property will instruct the service to obtain a list of all services registered to the Eureka server. The services will be stored within the client cache and is updated every 30 seconds.

The HR-Service controller will use the RestTemplate to make a HTTP call to one of the instances of the Phone-book service with the 'contacts' endpoint:

@RestController
public class HrController {

	@Autowired
	RestTemplate restTemplate;
	
	@GetMapping("/request-contacts")
	public Contact[] contacts() {
	        Contact[] result = restTemplate.getForObject("http://localhost:9002/contacts", Contact[].class);
	        return result;
	    }
	
}

So far, this request from the controller doesn't actually need the Eureka server. What happens if port 9002 goes down or is changed? How can we better utilise the available instances of the Phone-book service? In the next section, the Eureka service is used to perform load-balancing and dynamic access to the services.

Enabling microservices to communicate through Eureka

The Eureka server allows our services to communicate without explicitly knowing their location (including port). With the use of the service name as a key, Eureka is able to consolidate all running instances of a server for load-balancing, and also empowers the clients of the server to refer each other upon their name, rather than address. As a result, clients are able to reference each other dynamically, and requests for service consumption can be load-balanced altogether.

The Spring application name, defined the properties of each service, is used as a key within the Eureka server. Before we can allow the RestTemplate to take advantage of the load-balancing provided by Eureka, we must annotate it with @LoadBalanced:

@Bean
@LoadBalanced
RestTemplate getRestTemplate(){
	return new RestTemplate();
}

We use the @LoadBalanced annotation to tell Spring to create a Ribbon-backed RestTemplate. The Ribbon slightly changes the way we interact with the service as Spring no longer expects the physical location of the service to be provided, instead it expects the Eureka service ID (Application name) to be called. Eureka Ribbon will periodically refresh its cache of IP addresses from the Eureka service of service names in an eventually consistent manner every 30 seconds. The Ribbon-enabled RestTemplate parses the URL as the server name key to query Ribbon for a service instance.

If we were to call the existing HR-Service request from the Controller, it will now search for a service within its cache under the name of 'localhost:9002'. The service will throw an error when we try to access the service name. The advantage is that the service will no longer request the exact port of the request, and can now have the request load-balanced to one of the various service instances.

The HR-Service controller is now updated to call the Phone-book service name as the key to the Eureka server:

	@GetMapping("/request-contacts")
	public Contact[] contacts() {
	        Contact[] result = restTemplate.getForObject("http://phone-book/contacts", Contact[].class);
	        return result;
	}

If we make a call, the Eureka server will provide load-balancing for us and return the contacts in JSON as expected from one of the available instances.

Summary

When I first started using Spring Cloud Eureka, I had thought of it as a convenient naming resource that allowed me to reference different services by name, rather than by location. This blog has allowed me to realise the potential of the discovery server for more than just service registration.

The Eureka server can provide useful insight to the health of the many instances running for each service within a distributed system. From the server page, we can see the life of services via the expected and received heartbeats, the capacity of RAM in the system, and also the live update of services as they come up and are brought down.

Spring Cloud Eureka provides load-balancing as a service to the system to support the horizontal scaling of the services as they are required. The Eureka package has also made it intuitive and simple with bootstrapped annotations to implement the discovery server and to also use the discovery server with auto-configured service discovery implementations and configured properties.
You can find the source code from this blog on GitHub here where you launch your own Eureka server to open and consume services through a load-balanced server.

Centralising configuration within a distributed application

Mon, 23 Nov 2020 00:00:00 GMT

Key Takeaways
• Use Spring Cloud Config to create an externalised configuration server in a distributed system.
• Understand how the Config server reads in properties across a filesystem.
• Implement a client server to consume the Config server.

Centralised configuration

In a distributed system, the configuration of the services are often defined at a local level where they are individually updated and maintained by the responsible team. The Spring Cloud framework offers the Config server solution to act as a single-stop solution for managing all configurations to the services within a system. The centralised configuration server will provide transparency and versioning of configurations that can impact multiple or single services at a time.

We use configurations to represent all the changes that might take place between deployments. For example, configurations could include login credentials, connections to databases, hostnames and also logging levels. Configurations for services within a Spring Boot application are often defined within the application.properties file or a similar yml file within the resources directory. With Spring Cloud Config, the distributed services can inject their configurations by scanning the Config server before they are instantiated.

A centralised configuration server offers further benefits such as risk reduction and dependency on local configuration that can otherwise fall out of sync with the other services. Spring Cloud Config enables environment variables to be tracked centrally, where services can even obtain updated properties at runtime without having to restart.

Creating a Config Server

The Config server can use many different medium for storing the configuration files including JDBC, Redis, or AWS S3. The default storage is Git and will be used alongside GitHub in this blog.

When a service is subscribed to the config server, they will be able to use the server as a REST service to obtain their configuration values.

The Config server uses the Spring Cloud Config Server dependency and Spring Web dependency together. The Web dependency provides the REST functionality of the server once all the configuration files are read and stored:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-config-server</artifactId>
</dependency>

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>

The config server must be configured to know where to read the properties files from. The configuration is applied in the application.properties file:

server.port= 8888
spring.cloud.config.server.git.uri = https://github.com/4neesh/DeveloperBlogDemos
spring.cloud.config.server.git.searchPaths= SpringConfig/Config-files*, SpringConfig/Config-files/prod*, SpringConfig/Config-files/dev*

The first line will direct the Config server onto port 8888.
Line 2 will specify where the server will look for the Git configuration files. The value for line 2 will be the home directory of the Git repository (where the .git file is located).
Line 3 defines the search paths to find the appropriate properties files from within the Git repository. The search path will first enter the SpringConfig folder, followed by scanning all files within the Config-files sub-directory, and finally the sub-directory of 'prod' and 'dev' within Config-files. The Config server will register all .properties files found within the search path.

The server is activated through a single annotation in the main class @EnableConfigServer. The annotation will instruct the server to scan the search paths defined and to define the endpoints for reading the configuration files.

@SpringBootApplication
@EnableConfigServer
public class ConfigServerApplication {

	public static void main(String[] args) {
		SpringApplication.run(ConfigServerApplication.class, args);
	}

}

The following list contains all the .properties files that are obtained by the config server. I have manually created these files and placed them within the folders at the Git repository to allow the server to build its key-value pairs of properties:

- /SpringConfig/Config-files/application-prod.properties
- /SpringConfig/Config-files/application-dev.properties
- /SpringConfig/Config-files/application-default.properties
- /SpringConfig/Config-files/prod/drinks-prod.properties
- /SpringConfig/Config-files/prod/food-prod.properties
- /SpringConfig/Config-files/dev/drinks-dev.properties
- /SpringConfig/Config-files/dev/food-dev.properties

The below file directory structure is used:

├── SpringConfig
│   └── Config-files
│         ├── application-prod.properties
│         ├── application-dev.properties
│         ├── application-default.properties
│         └── prod
│         │     ├── drinks-prod.properties
│         │     └── food-prod.properties
│         └── dev
│               ├── drinks-dev.properties
│               └── food-dev.properties

When the config server has started, you will be able to send GET requests to the 8888 port endpoints to obtain the relevant value(s). In the next section, we will look at the exact endpoints that are generated by the config server and how the values returned are determined.

Navigating the Config server

When the Config server is launched, it will use the Spring-web starter to expose endpoints that reflect the properties files which it finds from the saerch paths. A GET request can be made to endpoints using the following name convention:

localhost:8000/{application name}/{active profile}

Each property file is conveniently named to address both the application name and the active profile. For example, 'drink-prod.properties' will have the application name of 'drink' and the active profile of 'prod'. The same goes for all other files. If there is no application name specified, it will be defaulted to 'application' and if there is no profile specified, it will be read as 'default'. Therefore all services that use the config server will obtain the values from the 'application-default.properties' file (if it exists). We are able to define the application name and profile of each service within the application.properties file. The service will then know which properties file to query in the Config server.

You can see the config files on GitHub here. They contain values for a database name, user name and password. The database name will reflect the application name and the profile and the user name and password will either be a service account credential or simple value. They are used for demonstration in this blog, however as we will later visit, they can be made to represent real configuration values.

The following endpoints are exposed from the config server to enable transparency into the values obtained by the services:
/{application}/{profile}
/{application}-{profile}.properties
The first endpoint will return all values that fall under the application and profile together, and also the respective default values as per the application-prod/dev/default.properties files.

The below examples demonstrate how the properties returned from the config server follow a hierarchy of matching from the defined application name and profile:

localhost:8888/food/prod returns:

{
    "name": "food",
    "profiles": [
        "prod"
    ],
    "label": null,
    "version": "f6d1b8edc6e75c5586e369188e6bf6284120f46c",
    "state": null,
    "propertySources": [
        {
            "name": "https://github.com/4neesh/DeveloperBlogDemos/SpringConfig/Config-files/prod/food-prod.properties",
            "source": {
                "Database": "mysql-food-prod",
                "User": "serviceAccount",
                "Password": "serviceAccount"
            }
        },
        {
            "name": "https://github.com/4neesh/DeveloperBlogDemos/SpringConfig/Config-files/application-prod.properties",
            "source": {
                "Database": "mysql-application-prod",
                "User": "serviceAccount",
                "Password": "serviceAccount",
                "Environment": "PROD"
            }
        }
    ]
}

The first tags returns the application name and the profile of the request. The properties from the files on the Config server are presented under 'propertySources'. The first set of values come from the food-prod properties file and the second set of values are from the application-prod properties file. The config server has returned the properties files that are appropriate to the application name and the profile of prod first, followed by the default application-prod properties second.

Another example to demonstrate the config server: localhost:8000/drink/dev:

 "propertySources": [
        {
            "name": "https://github.com/4neesh/DeveloperBlogDemos/SpringConfig/Config-files/dev/drink-dev.properties",
            "source": {
                "Database": "mysql-drink-dev",
                "User": "user",
                "Password": "password"
            }
        },
        {
            "name": "https://github.com/4neesh/DeveloperBlogDemos/SpringConfig/Config-files/application-dev.properties",
            "source": {
                "Database": "mysql-application-dev",
                "User": "user",
                "Password": "password",
                "Environment": "DEV"
            }
        }
    ]

If we request an unknown application name with a 'dev' profile, we will receive the application-dev.properties file:

localhost:8080/cook/dev:

"propertySources": [
        {
            "name": "https://github.com/4neesh/DeveloperBlogDemos/SpringConfig/Config-files/application-dev.properties",
            "source": {
                "Database": "mysql-application-dev",
                "User": "user",
                "Password": "password",
                "Environment": "DEV"
            }
        }
    ]

Consuming the configuration server

We have seen how the config server will return properties as a REST service and how the properties files are returned upon reflection of the application name, profile and defaulted properties files. A client can also perform GET requests to the service to obtain the configuration to be injected at instantiation.

The client service will require the Web-starter dependency to make the GET request and Config-client dependency to register the Config server:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>

<dependency>
  <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-config</artifactId>
</dependency>

The client will need to load the Config server properties before the application uses its local properties by defining the Config server in the bootstrap.properties file. The bootstrap properties file is loaded with precedence before the application has started to override any of the local configurations by default. The bootstrap.properties file is therefore used to load the configuration files from the server to provide configuration to the service.

The bootstrap.properties file will be defined as below to register the config uri, application name and profile:

cloud.config.uri=http://localhost:8888
spring.application.name=drink
spring.profiles.active=prod

The cloud.config.uri will point to the location of the config server. The spring.application.name value will determine the application name for which to obtain the properties file for. This property is defaulted to 'application'. The spring.profiles.active property will tell the application which profile it will assert for the config property search. This property is defaulted to 'default'.

If you check the console for when the application is started, you will see the following line to tell you the config files are being loaded:

Fetching config from server at : http://localhost:8888

When the Spring client has loaded, the config properties from the localhost:8888/drink/prod will be loaded into the application. The following values are pre-loaded into the application:

Database=mysql-drink-prod
User=serviceAccount
Password=serviceAccount
Environment=PROD

The Environment property has been taken from the application-prod.properties file which is read second to the drink-prod.properties file. As the drink-prod file did not contain a value for the key of 'Environment', the value is populated from the application-prod file.

Using the config values within the client

The client can use a RestController to obtain the values from the config server:

@RestController
public class ClientController{

  @Value("${Database}")
  public String database;

  @GetMapping("/database")
  public String getHome(){
    return database;
  }

}

The @Value annotation will obtain the value for the property of 'Database' from the configuration file. The endpoint of '/database' will return the value as a String to the user. Going to the endpoint of '/database' will return 'mysql-drink-prod'.

Obtaining client properties from the Config Server

You may remember that we are importing the config files from the server within the bootstrap.properties file which is initialised before the service iteself. As a result, the config server will theoretically populate items into the application.properties file. To test this, we can update the drink-prod.properties file from the config server to include a further property that is then pushed to the GitHub repository:

server.port=7777

The new property for server.port will instruct the client to be hosted on the 7777 port. The property would only ever work for a single client as only one client can be registered to port 7777. Further requests to the port from different services will return an error that the port is unavailable. If we were to restart the client, we can see it will start on port 7777 from the Spring console output:

Tomcat initialized with port(s): 7777 (http)

By defining the server port through the config server, we have illustrated how the bootstrap.properties file is loaded before the application starts, and how the application name and profile read from the config server is applied. The config server can be taken much further by configuring many other static properties across services such as discovery server locations, JDBC drivers, database locations and non-production credentials.

Obtaining updated properties from the Config server

The current client and Config server work in unison where the client pulls in the config server properties upon instantiation. The properties are read in statically and used for the life of the service. But what happens if we change a property within the config server? The client will not be able to reflect the update as it's properties are static. Instead, we will need to restart the client to obtain the latest information from the config server.

Thankfully, there is an easier way to obtain the latest updates, by using the @RefreshScope annotation from the Spring Cloud Context package. In the RestController, I will add the @RefreshScope annotation to instruct the controller to refresh its beans upon a client refresh. We are able to refresh the client by invoking the 'refresh' endpoint from the Spring actuator with the below dependency:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

By default, not all actuator endpoints are accessible. Therefore in the application.properties file for the client, we are to add the following line to enable the '/refresh' endpoint. Alternatively, you can add the below line to the appropriate file in the Config server.

management.endpoints.web.exposure.include=refresh

The RefreshScope will be tested by updating the property of 'user' in the drink-prod.properties file:

Database=mysql-drink-prod
User=serviceAccount2.0
Password=serviceAccount
server.port=7777

The property update is pushed to the Git repository and the client RefreshScope is activated by sending an empty POST request to the refresh endpoint: localhost:7777/actuator/refresh using Postman. Once the actuator endpoint is called, the RefreshScope will renew its beans to show the updated configuration file reflected in the '/username' endpoint.

So what does RefreshScope actually do?
The RefreshScope annotation will mark the Controller class as 'lazy' where they are initialised when they are used rather than at the application initialisation. The scope of the class will act a cache for storing the initialised values. As a result, the bean can be re-initialised as and when required. It is important to consider which other beans are within the scope of the class as they will also be re-initialised when the actuator refresh endpoint is called. This consideration can have an impact on the efficiency of the refresh and the side-effect upon beans that are refreshed.

Summary

The Spring Cloud Config server is used to act as a REST service for obtaining service configuration at runtime. The centralised configuration offers a transparent and controlled service to all clients within the microservice system. The Config server is configured to scan a file path to obtain the properties files and uses a hierarchy of application name and active profile values to define configurations to be passed in. Each client within the system registers its configuration to the server with the bootstrap.properties file and is able to obtain live updates of the configuration through the RefreshScope annotation.

You can get started with centralised configuration from with the source code from this blog on GitHub here.

The benefits of using Spring Boot

Mon, 09 Nov 2020 00:00:00 GMT

Key Takeaways
• Understand the popularity of the Spring framework to developers.
• Review the core IOC containers provided with Spring.
• Understand how Spring Boot Autoconfiguration enables convention over configuration.

The Spring framework

Spring is an open source application framework used with Java. Spring offers comprehensive support to the developer for building applications with in-built support for testing, data access, web frameworks and messaging support. The popularity of Spring, which is over 18 years old, has been realised as a from its light-weight design and features that include inversion of control (IOC).

Key features of the Spring framework

Spring provides an IOC container which serves as the most valuable feature to the developer. The IOC container is complementary to the final principle of SOLID: 'dependency inversion'. Dependency inversion creates loosely coupled classes within the application design. Loose coupling will provide benefits across unit testing and support for application growth and behaviours. You can read about dependency inversion in more detail from my previous blog.

In addition to the IOC container, the Spring framework also provides out-of-the-box support for further features:

Aspect-orientated programming, also known as AOP, enables cross-cutting concerns to be injected for method calls within the application. With AOP, the process of calling methods can be broken down into sections (such as before the call and after the call) where logic can be injected in a modular fashion. You can read more on AOP in my blog here.

Security is supported by Spring for authentication, authorisation and other aspects. Spring security offers popular security mechanisms that can be efficiently implemented and customised into enterprise applications such as authorisation protocols of Kerberos, OAuth and SAML.

Transaction management by Spring ensures database transactions are made in a timely manner using the ACID components. When an application makes a transaction to a database, the ACID components of atomicity, consistency, isolation and durability are important to meet to ensure transactions are orderly, complete and correct. The Spring framework provides an abstract layer underpinning both local transactions (in a centralised system) and global transactions (in a distributed system). Spring provides an object relational mapping of data where support for integration with Hibernate, Java persistence API and data access object implementations can be configured for the IOC container.

Model-view-controller framework, also known as MVC, is an industry-standard design pattern used for developing front-end interfaces that support scalable applications. MVC defines three components of the application that allow data to be passed through the application for processing: The view, which represents the interface that the user interacts with.
The controller, which accepts an input from the user and converts the request to the model.
The model, the central component which manages the request from the controller and manages the business logic to return a response for the user.

Spring Beans and IOC

Spring achieves IOC by defining a 'container' to store the Objects that are later created and injected to the application. The Objects stored within the container are known as 'beans'. The IOC container can exist with different properties and features, these are largely divided into the BeanFactory and ApplicationContext.

Spring Beans

A Spring Bean contains a set of instructions for creating an Object; by default, the Object is created as a singleton. The bean is later injected as and when it is required. Spring Bean definitions are made dependent to the IOC container that is used. Definitions can be made using XML or annotations. For XML files, beans are defined within a configuration file, and are injected by defining an 'autowire' property to the class configuration or by accessing the bean through its name from the container in the source code. Alternatively, beans can be defined within a class marked with '@Configuration' and injected by using the '@Autowired' annotation. We will later visit how to define the IOC container for the application. An example of the two styles for definitions are below:

XML definition and autowire:

<beans>
    <bean id="car" class="com.f1.Car"/>

    <bean id="race" class="com.f1.Race" autowire="byName">
        <property colour="yellow">
    </bean>
</beans>

Annotation-based:

public class Car{
    String colour;
}

@Configuration
public class ConfigClass{

    @Bean
    public Car getCar(){
        return new Car();
    }
}

@Component
public class Race{

    @Autowired
    Car carToRace;

}

When using the Spring Framework, the IOC container is determined by the developer. The ApplicationContext is significantly more popular as recommended by the documentation: 'use an ApplicationContext unless you have a really good reason for not doing so.' The motivation for using the BeanFactory over the ApplicationContext is limited to only when memory usage is a valuable metric to the application as the BeanFactory can help reduce the memory-load of default ApplicationContext features.

BeanFactory container

The BeanFactory is an interface that is most commonly used through the XmlBeanFactory implementation. The BeanFactory is instantiated within the application using the following dependency:

    <dependency>
	    <groupId>org.springframework</groupId>
	    <artifactId>spring-beans</artifactId>
	    <version>5.2.8.RELEASE</version>
	</dependency>

The BeanFactory instance will take in a Resource instance to locate the configuration file for the beans:

    Resource resource = new ClassPathResource("beans.xml");
    BeanFactory beanFactory = new XmlBeanFactory(resource);

As previously seen, the configuration file will use XML to define beans:

public class Person{
    
    public Person(String name, int age){
        this.name = name;
        this.age = age;
    }

    String name;
    int age;
}

beans.xml file:

<bean id="cricketer" class="com.Person">
    <constructor-arg name="name" value="Ben Stokes"/>
    <constructor-arg name="age" value="29"/>
</bean>

The cricketer bean will create a Person of name "Ben Stokes" and of age 29. The beanFactory can be used to obtain the bean:

    Person person = (Person)beanFactory.getBean("cricketer");

ApplicationContext container

The ApplicationContext is a an advanced Spring container available from Spring 3.0 that offers specific framework-orientated enterprise functionality as well as all BeanFactory features. While the BeanFactory supports only xml-configuration of beans, the ApplicationContext differs as it additionally supports annotation-based configuration that can be used within the application code. The ApplicationContext extends from the BeanFactory to provide further functionality of bean integration with application-layer features such as: * ApplicationEvent support for creating and listening to events during runtime * MessageSource access to support environment-specific configuration and internationalisation * BeanPostProcessor for the creation and use of new beans before formal bean creation

The ApplicationContext container can be implemented similarly to the XmlBeanFactory with the implementation 'ClassPathXmlApplicationContext':

ApplicationContext context = new ClassPathXmlApplicationContext();

The annotation-driven configuration can be defined 'AnnotationConfigApplicationContext':

ApplicationContext context = new AnnotationConfigApplicationContext();

The annotation-based container will perform a component scan to the configuration classes to store the beans that can be later obtained.

Challenges with the Spring framework

While Spring provided developers with some great features to transform their application design and complexity, it introduced some unwanted challenges to the developer. The complexities of Object management and dependency injection was since replaced with verbose xml configuration, most of which was boilerplate code. As applications grew and the relationships between beans became more complicated, the burden of configuring the application became more and more apparent.

One example that captures the laborious configuration required with Spring would be the MVC functionality. Spring MVC requires a few configurations to be made before it can work as expected in the application. This involves the dependency in the pom.xml file and configurations for the DispatcherServlet and InternalResourceViewResolver.

The spring-mvc and java servlet dependencies are first referenced within the pom.xml file:

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-webmvc</artifactId>
    <version>${spring.version}</version>
</dependency>

<dependency>
    <groupId>javax.servlet</groupId>
    <artifactId>javax.servlet-api</artifactId>
    <version>3.1.0</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>javax.servlet</groupId>
    <artifactId>jstl</artifactId>
    <version>1.2</version>
</dependency>

The DispatcherServlet is bootstrapped to the application and is responsible for loading the IOC container. In the WEB-INF/web.xml file, the developer will need to configure the DispatcherServlet:

<servlet>
    <servlet-name>servlet</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
    <init-param>
       <param-name>contextConfigLocation</param-name>
       <param-value>/WEB-INF/spring/appServlet/servlet-context.xml</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
</servlet>

<servlet-mapping>
    <servlet-name>appServlet</servlet-name>
    <url-pattern>/</url-pattern>
</servlet-mapping>

Component scanning will be configured into the app-config.xml file to direct Spring to the packages to be scanned:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
                           http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/context
                           http://www.springframework.org/schema/context/spring-context.xsd ">

    <context:component-scan base-package="com.spring.application" />

</beans>

Lastly, the InternalResourceViewResolver bean is registered which will return the view name as specified by the location of the file:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:mvc="http://www.springframework.org/schema/mvc"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
                           http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/mvc
                           http://www.springframework.org/schema/mvc/spring-mvc.xsd
                           http://www.springframework.org/schema/context
                           http://www.springframework.org/schema/context/spring-context.xsd ">

    <mvc:annotation-driven />
    <context:component-scan base-package="com.spring.application" />

    <bean class="org.springframework.web.servlet.view.InternalResourceViewResolver">
        <property name="viewClass" value="org.springframework.web.servlet.view.JstlView"/>
        <property name="prefix" value="/WEB-INF/view/" />
        <property name="suffix" value=".jsp" />
    </bean>

</beans>

Thankfully, that would be the only configuration required as default for using MVC with the Spring framework. But surely this can be made easier? In April 2014, the advent of Spring Boot came as a solution to resolve the heavy configuration required with the Spring framework...

Introduction to Spring Boot

Spring Boot was introduced with a precendent to convention over configuration: the objective of reducing the number of decisions a developer needs to make when creating an application. But what does this mean?
The design paradigm of convention over configuration will only require the developer to specify unconventional configurations for the application. A sensible default is however defined for all other infrastructure components (which would include DispatcherServlet and component scanning for MVC - yay).

Spring Boot applications can be created out-of-the-box from the 'Spring initializer' at start.spring.io. The Spring initializer can also be used through Spring Tool Suite and allows the developer to specify the dependencies that create the environment.

AutoConfiguration

AutoConfiguration is the core feature from Spring Boot as it resolves the excessive configuration required from the developer for the application purpose.

As previously visited, Spring MVC is one example where default configuration would always be required to set-up the DispatcherServlet and ViewResolver amongst other infrastructure components. The same principle applies for Hibernate where we need to manually configure a data source and entity manager factory to our application. Rather than configuring the xml ourselves, Spring Boot AutoConfiguration will create the beans automatically upon scanning the jar in the class-path and deciding which default beans are required.

Spring AutoConfiguration is used from the spring-boot-autoconfigure jar in the classpath. The AutoConfiguration class, which is run after all other beans are configured, will appear like any usual Spring @Configuration class. The difference however lies in annotations such as @ConditionalOnMissingBean and @ConditionalOnClass which will configure and activate a bean provided certain classes appear within the classpath.

Spring Boot AutoConfiguration also plays a role in the default IOC container that is used by the Spring Boot application. Spring Boot will default to the AnnotationConfigApplicationContext container. If during the Autoconfiguration scan a web starter-project is identified, the IOC container will use the AnnotationConfigServletWebServerApplicationContext container instead. The latter of the ApplicationContext types will create, initialise and run a webserver as required by the dependencies. The selection of the ApplicationContext further extends the functionality of Autoconfiguration to support the 'convention over configuration' framework.

Other key features with Spring Boot

The first main feature from a Spring Boot application is the @SpringBootApplication annotation that is added to the main class as default. The single annotation encapsulates 3 other annotations that are useful for the developer to use the main class for: * @Configuration enables Java-based configuration for the class to enable Beans to be defined * @ComponentScan to enable component scanning across the application package and sub-packages * @EnableAutoConfiguration to enable the Spring Boot AutoConfiguration

Spring Boot introduces 'starter-dependencies' that can be selected during creation of a Spring Boot Application, or injected to the pom.xml file. Starter-dependencies includes the related configurations and beans required by the application thus eliminating the need to add multiple related dependencies in the same pom.xml file. The starter-dependencies extend the convention-over-configuration of Spring Boot by further reducing the configuration required by the developer. If you were to develop a RESTful application, you would require the MVC design pattern which includes Spring MVC, Spring REST and Object to relational mapping. The spring-boot-starter-web dependency encapsulates all these requirements together to include the jar files with the application classpath.

Spring Boot provides an embedded server into the jar executable application as default. Without Spring Boot, the developer would be required to install a web server that the application artifact is later deployed to. Spring Boot offers a default embedded server of Tomcat, however alternatives such as Jetty and Undertow are also available. Apache Tomcat is an open source web server designed to execute Java servlets and to render Java Server Pages.

Spring Boot enables the use of externalised configuration to provide control of the application runtime properties. Beans can use the @Value annotation to obtain their value from an external .properties file. Spring Boot provides a properties file and bootstrap file to priorities configurations that can enable custom application behaviour for different environments. The externalised configuration can be used for changing the URL, username and password accesses, locale for formats and also logging level to the same application in different environments.

Summary

The Spring framework was introduced as a lightweight solution for loose-coupling and management of Java Objects with an IOC container. Within Spring, developers are able to create beans to be stored within an IOC container which represent instructions for creating and managing Java Objects. The ApplicationContext IOC container introduced annotation-based configuration and enhanced IOC bean management features. Following the growth and popularity of Spring, the developer often wrote excessive xml configuration when using the Spring framework thus Spring Boot offered a solution. Spring Boot introduced Autoconfiguration that uses convention over configuration to create default beans and configuration for the application. As a result, Spring Boot enables the developer to create out-of-the-box production-ready applications with minimal configuration required with the flexibility of customisation. You can read more about Spring and its features from the excellent documentation here

Using lambda for functional methods

Mon, 26 Oct 2020 00:00:00 GMT

Key Takeaways
• Understand how functional interfaces are designed and created.
• Use anonymous classes to implement functional interfaces.
• Understand how to create a lambda function for expressive implementations of functional interfaces.

Lambda and functional programming

Lambda functions were introduced in Java 8 as a step for Java towards functional programming. Functional programming enables a developer to pass functions around the application without assigning it to a specific class. Lambda enables Java to use functional programming as functional interface implementations that are dynamically created and executed at runtime.

What are functional interfaces?

A functional interface is defined as an interface with just a single abstract method. A functional interface can also contain many other methods that are either Static or Default, provided there is just a single abstract method. If the interface contains more than one abstract method, it cannot be considered when creating lambda functions.

Prior to Java 8 (and the introduction of lambda functions), functional interfaces would be used through a class implementation where the abstract method is overridden:

//functional interface
public interface CoffeeMachine{
    void makeCoffee();
}

public class EspressoMachine implements CoffeeMachine{

    @Override
    public void makeCoffee(){
        System.out.println("Making an espresso");
    }
}

EspressoMachine machine = new EspressoMachine();
machine.makeCoffee();

Making an espresso

The creation of the EspressoMachine with a single method, and further instantiation of the machine enables an implementation of the CoffeeMachine to be used. Considering what we have achieved, in an implementation of a single method, you may consider the creation and instantiation of the class as being heavy to write. An anonymous class may provide a more straight-forward approach.

Using anonymous classes

Anonymous classes allow us to reduce the boilerplate code of implementing the interface into a class. As the name suggests, an anonymous class does not have a name as it is implemented in the method body of its declaration:

CoffeeMachine espressoMachine = new CoffeeMachine(){

    @Override
    public void makeCoffee(){
        System.out.println("Making an espresso");
    }

};

The anonymous class enables local variables to the class to be defined and for further methods to be used for the espressoMachine:

 CoffeeMachine espresso = new CoffeeMachine() {
    int numberOfCoffees = 1;

    @Override
    public void makeCoffee() {
        this.numberOfCoffees = randomNumberCoffees();
        System.out.println("making " + numberOfCoffees + " coffee(s)");
    }

    int randomNumberCoffees(){
        return (int)(numberOfCoffees * Math.random()*100);
    }
};

Within the body of the anonymous class, the keyword this would refer to the implementation espressoMachine as the anonymous class represents an inner class at compile time.

Functional interfaces as a lambda function

The CoffeeMachine can be implemented as a lambda function as an alternative to an anonymous class. The lambda function ultimately reduces the amount of code to write, however is limited in the complexity of the implementation that can be achieved. Let's look at what a lambda function might look like for the EspressoMachine implementation:

CoffeeMachine espressoMachine = () -> {System.out.println("Making an espresso");};

Is that it? Yes!

The syntax of the lambda function may seem strange and the logic behind it may appear unclear. I will now break down the lambda function and explore how it can be adapted to cater for other types of functional interfaces.

The lambda function is broken down into three areas: parameters, arrow operator, and code block.

The code block can wither exist within curly brackets or it can exist without provided it is a single line.

The parameter is used to take in the arguments as defined by the abstract method. The number of parameters can be any number as long as it reflects the abstract method. The arrow separator is used to instruct a lambda function. The final expression is used as the method body. The implementation of the Lambda function for the CoffeeMachine can exist as below:

CoffeeMachine espresso = () -> System.out.println("Making an espresso");

The lambda function could be replaced to accommodate any number of arguments if the abstract method were to require them:
1 argument (int numberOfCoffee)

CoffeeMachine espresso = (a) -> System.out.println("Making " + a + " espresso(s)");

2 arguments (int numberOfCoffee, String customer)

CoffeeMachine espresso = (a, b) -> System.out.println("Making " + a + " espresso(s) for " + b);

If the lambda expression returns an Object or value, the curly braces must be used with the return statement or none together.

public interface CoffeeMachine{

    String typeOfCoffee();
}

CoffeeMachine latte = () -> {return "Latte";};
CoffeeMachine cappuccino = () -> "Cappuccino";

In the above example, the latte uses both the curly braces and the return keyword. The cappuccino however uses neither.

The relative ease of creating lambda functions is enabled by the JVM as it infers the method call that is represented. There are three key properties to the lambda function that enable it to be inferred without compile error by the JVM:
1. The expected representation of the lambda function must replace a functional interface. Therefore the implemented interface must have only a single abstract method.
2. The abstract methods number of parameters must match the number of parameters and cast as represented by the lambda function.
3. The return type of the abstract method must be equal to the return from the code block of the lambda function.

For the functional interface below, each of the three subsequent lambda functions will fail to compile as they breach of the three rules above:

public interface Vehicle{
    void getTypeOfVehicle();
}

The below lambda will not compile as an implementation of Driver is expected which is not a functional interface (unlike Vehicle).

public String getDriverName(Driver driver){
    return driver.getName();
}

System.out.println(getDriverName(() -> {return "Vehicle is truck";};))

The below lambda will not compile as an argument is passed in when there should be 0 arguments.

(10) -> {System.out.println("Vehicle is a truck");};

The below lambda will not compile as the code block returns a String when the return should be void.

() -> {return "Vehicle is a truck";};

Anonymous classes vs Lambda functions

While Lambda expressions reduce the amount code to write, they may not always be the best solution to implement. The considerations to make for an anonymous class extend to performance, readability and purpose/complexity of the function invoked.

The anonymous class is a class of itself and generates a new '.class' method from compilation that adds to the size of the jar executable file. A lambda function however uses the invokedynamic bytecode instruction to bind the method dynamically to the encapsulating class. Therefore lambda expressions can improve time and performance of the application.

An anonymous class can be used for other implementations of interfaces and not just a functional interface. They are not bounded to the single abstract method, however the anonymous class will become larger as each method is overridden.

The anonymous class tends to be more verbose than the lambda expression. A redundant class body must be defined within the anonymous class and therefore must only be used when a class implementation is valuable. Unlike a lambda function, an anonymous class enables class-local variables to be defined as well as further functions for processing. The anonymous class supports far greater complexity and functionality within the class.

As the differences between lambda and anonymous classes are vast, there are clear and distinct purposes to use one over the other.

Summary

Functional interfaces are interfaces with a single abstract method. Prior to Java 8, a functional interface would require verbose implementation within a class and instantiation to access its behaviour.

Anonymous classes offer a compiled implementation of interfaces within the calling method. The anonymous class is compiled at runtime and enables further complexity of behaviour to be implemented.

Lambda functions introduce functional interface implementations that support behaviours of the application at runtime. The lambda function enables developers to reduce boilerplate code for implementations of functional interfaces using a unique three-part instruction.
Lambda functions can be flexible to support multiple parameters and return types and can be passed around the application and used as arguments to create dynamic and independent behaviours.

How the JVM uses the heap and stack

Mon, 19 Oct 2020 00:00:00 GMT

Key Takeaways
• Understand the differences between the heap space and stack memory.
• Visualise the runtime activity of the stack memory.
• Understand how the heap and stack communicate with each other.

How does the JVM store our Objects?

The JVM divides memory between two locations known as the heap space and stack memory for managing the application runtime and Objects. The two locations differ by their design and properties that effect their function and efficiency of memory use.

In this blog, I am going to review the responsibilities between the heap and stack to the JVM. By understanding the differences between the heap and stack, we may be better positioned to improve our application runtime performance through design.

The heap space

The heap space is created when the JVM starts as a dynamic space for allocating storage of Java Objects. The heap is obtained by the JVM from the operating system as a fixed space for the lifetime of the application. The heap is described as dynamic as it allocates memory during runtime of the application.

When Objects are assigned to the heap, they are globally accessible by all threads of the application, however they are not necessarily thread-safe. Objects can be designed to be thread safe as discussed in my blog here.

The heap space is divided into two 'generations' for storage: old and new. The generational divide is used to determine how Objects are stored, retained and removed from the heap space at runtime. The heap space uses a garbage collector to mark and sweep Objects when the young generation reaches capacity. You can read more about the garbage collector in my blog here.

The impact of the garbage collector upon the JVM performance can be significant. Object management from the heap is therefore a valuable area to inspect and resolve for runtime performance enhancement.
When the heap is full, java will throw a java.lang.OutOfMemoryError exception.

The stack memory

The stack memory contains space for the execution of application threads. The stack is used for temporary memory where variables are stored when methods are invoked. When a method is called, its memory is allocated on top of the stack in a first-in-last-out (FILO) order. Once the method has completed, its memory allocation is cleared from the stack. The diagram below represents 4 different method calls made, where the most recent method call sits at the top of the stack and is currently pointed to by the JVM.

The stack uses static memory allocation where the compiler is able to allocate memory to the application before it has been executed. Once the stack has bounded the memory to locations, it is able to execute the threads efficiently with direct addressing.

The stack will store local variables to the methods and the reference addresses of the Objects within the method. The references will point to the Object location within the heap. When the method is completed, the stack will remove the method from the memory along with the local variables that are referenced by the method.

A key performance advantage of the stack over the heap is the direct addressing of memory. The stack is able to efficiently assign and remove memory unlike the heap. When there is insufficient memory on the stack, the JVM throws a java.lang.StackOverflowError exception.

Using the heap and stack together

The below code sample provides a means to explain how the heap space and stack memory coordinate together:

public class Sport{
    String name;
    int players;

    public Sport(String name, int players){
        this.name = name;
        this.players = players;
    }
}

public class SportBuilder{
    public static Sport Build(String name, int players){
        return new Sport(name, players);
    }
}

public class Main{

    public static void main(String[] args){

        int players = 2;
        String sport = "tennis";
        Sport tennis = null;
        tennis = SportBuilder.build(sport, players);

    }
    
}

The code above defines 3 classes, a Sport entity class, a SportBuilder class with a static method to build a Sport Object and a Main class with the main method.

The main method from the Main class will instantiate the creation of the stack memory with a single thread. On line 3 in the Main class, the main method is defined and added into the stack memory.
On line 5, the primitive int players is added into the main method of the stack memory.
On lines 6 and 7, Objects of Type String and Sport are created in the heap space; their address is referenced by the main method block in the stack.
On line 8, when the SportBuilder method is called, a new block is added to the stack on top of the main method block. The new block will contain a reference to the Main class instance that is calling it, the local int players and the reference to the String "tennis".
In line 3 from the SportBuilder class, a further method is added to the stack as the Sport constructor is called. The third block will contain the primitive int value along with references to the String "tennis". The third block will also contain a reference to the calling Object instance of SportBuilder.
When the Sport Object is created, the third block is swept from the stack, followed by the second block from the SportBuilder class, and finally the Sport Object is populated with the String and primitive value in the heap space.

Referencing Objects from the heap into the stack

The code sample above has demonstrated how the heap and stack work together to store and reference Objects through the lifecycle of a JVM thread. The process for referencing Objects from the stack to the heap can impact the efficiency of the JVM and operations that can be performed upon the Objects. This section will review how the two memory areas communicate.

The pass by reference approach involves passing the address of the Object from the heap space into the stack. In the below example, an instance of Student is created ('alice') and it is stored in memory location 'a' in the heap space. On line 5, the Student is referenced by a method called goToClass() from the stack. The goToClass method will receive the location 'a' of Alice and will set the 'attendance' boolean of the Object to true on line 9.

public static void main(String[] args){
    
    Student alice = new Student("Alice");

    goToClass(alice);
}

private static void goToClass(Student student){
    student.setAttendance(true);
}

The pass by value process involves passing a copy of the Object itself from the heap space into the stack. Using the same example as above, the pass by value movement of the Object would involve creating a new copy of the Object and then passing it into the stack.

In the diagram above, an identical copy of alice is made within the heap space and is then passed into the goToClass method.

How Java passes Objects between the heap space and stack

The JVM uses a combination of pass-by-reference and pass-by-value for managing Objects. Technically, Java uses pass-by-value as the a copy of the Object is passed to the method call, however Java does not make a copy of the Object, it will pass a copy of the Object address from the heap space.

We are able to test how Java passes Objects with the below code snippet. The code will attempt to switch the Person Objects:

class Person{

private String name;

    public Person(String name){
        this.name = name;
    }

    public String getName(){
        return this.name;
    }
}

class Main{

    public static void main(String[] args){
        Person alice = new Person("Alice");
        Person bob = new Person("Bob");
        swap(alice, bob);

        System.out.println(alice.getName());
    }

    public static void swap(Object a, Object b){
        Object c = a;
        a = b;
        b = c;
    }
}

The output from the above code will show:

Alice

The swap() method will not work as expected as we are not actually altering the location of alice and bob, but are altering the location of a copy of the reference:

public static void swap(Object a, Object b){   //(copy of)alice = location a, (copy of)bob = location b
        Object c = a;   //c = location a
        a = b;          //a = location b
        b = c;          //b = location a
        
        //final result: (copy of)alice = location b, (copy of)bob = location a
    }

If the actual reference of the Person was passed into the swap() method, then alice would point to location B and have the name "Bob" and vice versa. However as a copy of the reference was passed, the actual Person objects of Alice and Bob remain unchanged.

The use of the new keyword within Java will allocate a new memory location of an Object. When the Object is then passed around the application, it will be the reference location that is copied and sent across. In the below code snippet, the creation of a new Student Object will result in a new Object being referenced with its own address. Assume the Student Alice has been passed in.

public void changeStudent(Student student){ //student = (copy of) Alice: location a
    student.setName("Alex");                //student = location a
    student = new Student("Charlie");       //student = location c
}

Alice's name is first changed to "Alex". As the reference location is passed into the method, the name will be changed on the Alice instance. Once the student is referenced to a 'new' Object, the memory location of the Object has been changed and all further method calls to the Object 'student' within the method will apply to the the Student instance of Charlie. If you were to query the name of Alice after this method has run, the output would remain as "Alice". The reference of Charlie will be lost after the method as it is no longer referenced within the method.

Conclusion

The heap space and stack memory are two separate areas within the RAM used by the JVM before and during runtime. The two areas work together by using a pass-by-value process where the address of the Objects within the heap space are copied and passed to the stack memory. The pass-by-value process enables the JVM to access and modify Objects without having to return them back to the calling method.

The stack memory is designed with a pointer to quickly navigate methods and remove them once completed. The heap space uses a garbage collector to mark and sweep Objects that are no longer referenced within the application.

Our understanding of the heap and stack can enable us to better understand runtime errors that may arise and how the application can make use of the stack direct addressing and recursive nature to optimise runtime performance.

How continuous delivery supports functional testing

Mon, 12 Oct 2020 00:00:00 GMT

Key Takeaways
• Understand the importance of functional testing before software deployment.
• Identify the challenges that arise with functional testing.
• Review the benefits of using continuous delivery and the Jenkins pipeline.

What is continuous delivery?

Continuous delivery (CD) extends from continuous integration by ensuring the latest software artifact is always in a deployment-ready build to be sent into production. You can read about continuous integration from my previous blog here.

But what does 'deployment-ready' mean? While CI aims to manage the movement of source code branches into new application builds, CD aims to manage the movement of the build through functional testing phases in a safe, sustainable and predictable manner.

The phases of functional testing can include rigorous testing and deployments across environments to ensure the software has been stressed in every feasible way before moving it into production. The diagram below visualises possible continuous delivery processes in a pipeline:

CD encourages frequent delivery of software updates in the same manner that CI encourages the frequent integration of source code. Smaller, but more frequent deployments are less risky to creating a regression in production. CD eliminates the repetitive tasks that may arise for functional testing to return tighter feedback loops for the test team while enabling a faster time to market.
The approach of 'continuous deployment' may often be mistaken with continuous delivery; continuous delivery differs as it requires a single manual approval stage prior to production deployment whereas continuous deployment is fully automated end-to-end.

What are functional requirements?

The functional requirements of a system can be described as the behaviours for inputs and outputs the system is expected to produce. The functional requirements can encapsulate all areas of the system from data processing, data manipulation and algorithms. Functional requirements are captured within use cases and enforced against non-functional requirements such as security and performance metrics.

While unit testing appeals to the developers perspective of the problem, functional testing takes the users perspective. The CD pipeline is responsible for carrying out multiple streams of testing to ensure the latest build adheres to the functional requirements. Functional testing can be divided into further testing domains such as UI testing, load testing, integration testing and API testing.

Integration testing involves testing a combination of software modules in unison. While unit testing involves the testing of individual components on their own, integration testing identifies dependent modules within the application and tests scenarios that occur between them. The main objective from integration testing is to test the interfaces between the modules. The value behind integration testing is to ensure each module can interact with each other as expected and to identify inconsistencies between module implementation.

API testing can fall under the integration testing umbrella where external APIs are tested to perform as expected for functionality and reliability. As modules may communicate with external systems, the API testing will cover the external layer of the system of communication.

Load testing involves stress testing the system capabilities to handle increased levels of traffic and requests than expected. The typical expected usage of the software may be modelled and simulated as well as upper and lower levels of requests. The load testing can provide metrics for the quality of service provided by a system and identify potential bottlenecks or vulnerabilities of the system for certain traffic or use patterns.

Operating System testing will test that the software is able to perform in the operating systems it is designed for as expected.

Release management with continuous delivery

The release management for continuous delivery defines multiple stages of functional testing. The system will be tested within dedicated servers to replicate the production environment to some degree. The separation of a test environment from the production environment enables system changes to be phased into the production environment with confidence that due diligence has been carried out upon each system change.

User acceptance testing (UAT) is a process of verifying the software works for the user. UAT is used to assess if the system can support the day-to-day intended use of the system. The system requirements may be defined within a scope and tested upon both systematically and unsystematically to verify the outcome of the test.

The challenges of continuous delivery

Continuous delivery can constitute a lengthy process where multiple timely tests are completed across multiple test servers. The functional testing process remains largely unchanged despite the size of the system update. As a result, the time consumption for continuous delivery is mostly fixed to the software delivery pipeline.

The complexity of functional testing further extends to the type of tests that are run across environments to verify the software update. Unlike unit tests, integration tests and other functional tests incorporate multiple modules together to cover use cases and behaviours of the system. The tests, as stand alone functions, therefore require greater levels of resources and inspection to verify.

The configuration of the environments and the system to perform checks upon must remain consistent and reliable, however functional testing may alter the server environment and create inconsistencies. While dedicated environments are used for specific testing, it becomes a challenge to 'clean' the environments for testing as and when is appropriate. The responsibility for ensuring tests are performed within a controlled environment introduces further challenges to continuous delivery.

Jenkins as a continuous delivery solution

Jenkins supports CD pipelines to automate the functional testing of builds along with useful tools to resolve the specific challenges related to CD. Amongst the plugins that come with Jenkins, support for multi-threaded testing and stashing archives make Jenkins a popular tool for CD.

Automated delivery pipeline
Similar to the CI pipeline, the Jenkins CD pipeline is an extension of automated procedures to run on the new software update. The CD activities can be lined-up and executed in succession where exceptions are reported back to the appropriate teams.

Concurrent delivery
The master-agent model of the Jenkins pipeline supports parallel processing. As functional testing may take place across multiple servers, the CD pipeline utilise parallel processing to divide tests across the servers at the same time. As a result, the CD pipeline can be streamlined for time efficiency.

Consistent configuration
Jenkins supports the use of stash to save a set of files to be later used in the workspace. Stashed files are present for the duration of the pipeline before being discarded at the end. The stash can support small file saves to a temporary directory to be later unstashed and used to create a consistent environment for testing. Some uses of stash can be for dependencies and configuration files.

Conclusion

The continuous delivery pipeline extends from continuous integration to verify the system upgrade meets functional requirements before being deployed into the production environment. Software testing is performed across different test environments that are designed to support functional testing and to replicate the production environment.

The importance of continuous delivery is to support a reliable, predictable and safe deployment of software into production. The multitude of testing to be performed across environments for verification can create challenges of time consumption and intervention of testing before deploying system updates. Jenkins supports the continuous delivery pipeline by automating the functional testing of the system updates across the appropriate test environments. Jenkins also supports continuous delivery with parallel processing.

Continuous delivery and continuous deployment play a major role in the modern software delivery cycle where code is rigorously integrated, built, tested and deployed to the production environment. The automation of continuous delivery enhances the feedback loop from functional testing to support frequent, incremental and efficient upgrades to software system.

How continuous integration helps developers

Mon, 05 Oct 2020 16:34:00 GMT

Key Takeaways
• Understand the challenges of integrating code between multiple developers.
• Review the phases of an application build and the responsibility of each phase.
• Understand the benefits of creating a continuous integration pipeline for code builds.

What is continuous integration?

Continuous integration is a development practice that automates the combined actions of compiling, building, testing and packaging source code into software. As software is frequently updated, the time cost of integrating code into the latest build can be high to the developer. Furthermore, the coordination between teams of multiple developers and the presence of merge conflicts can make the integration of code even more burdensome.

Merge conflicts arise when multiple developers attempt to integrate code changes that are applied to the same file(s). Multiple developers may have a different idea for the implementation of a module and therefore a merge conflict must be manually reviewed before the code is updated to the single version. Merge conflicts become increasingly likely as larger integrations are performed at a single time.

Continuous integration enables code builds to be performed autonomously and quickly. The process for compiling, testing, building and packaging code is performed through an automated pipeline. As a result, continuous integration encourages the practice of small, daily integrations amongst developers to lower the likelihood and impact of merge conflicts.

Build tools

Build tools automate the individual processes of creating executable builds from the source code. There are individual phases that are used within build tools to separately compile, build, test, package and deploy source code. Some of the popular build tools used are Ant, Maven and Gradle.

The lifecycle of a build

The build contains three different in-built life-cycles that define how a build is constructed: default, clean, and site.
• default: handles project deployment.
• clean: cleans the project by removing the files generated from the previous build.
• site: creates the project's site documentation and reports.

The build lifecycle consists of multiple phases that enable the developer to have control over how the code is compiled:
• validate: validate the project includes all the necessary information.
• compile: compile the source code of the project.
• test: test the compiled source code with the unit testing framework.
• package: package the tested code into a distributable format (such as a war or jar file).
• verify: perform checks upon quality through integration tests.
• install: install the package into the local repository.
• deploy: copy the final package into the the remote repository to be shared.

Each lifecycle phase can be run individually, however the former stages will always be run prior to the phase defined. For example, running the install phase will inadvertently run the validate, compile, test, package and verify phases before install.
The continuous integration phases are only relevant up to the package; these are explained further depth for a java project:

Validate will not perform any actions unless a 'goal' is specified for this phase within the POM.xml file. A simple validation check could reflect the version naming conventions, artifact ids, descriptions and purpose of each class is defined with JavaDocs etc.

Compile will take the source code of the .java files and compile them into bytecode (.class) files for the JVM to later execute at runtime. The JIT Java compiler will then convert the bytecode into machine code to be executed.

Test will compile the unit test code into bytecode and run each of the unit tests for the project. If a test were to fail, the lifecycle will end.

Package will take the compiled code from the .class files and package them into a executable format such as a .jar file. The pacakge phase is to be defined within the POM.xml file. For example:

 <plugin>
    <artifactId>maven-assembly-plugin</artifactId>
    <configuration>
        <descriptorRefs>
            <descriptorRef>jar</descriptorRef>
        </descriptorRefs>
        <archive>
            <manifestFile>src/main/java/META-INF/MANIFEST.MF</manifestFile>
        </archive>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Continuous integration tools

CI tools can be used to streamline the process of compiling, building, testing and packaging software within teams. The tools provide an automated pipeline that can execute each of the phases while provided appropriate feedback, metrics and outcomes of each build as configured. Jenkins is a popular open source automation server that offers the capacity of continuous integration for software systems as well as other convenient plugins that can optimise the pipeline for feedback and diagnostics.

A continuous integration pipeline is used to stitch the various phases of code integration into a production line to be delivered into environments. The goal behind the pipelines is to support the constant flow of software updates from all members of the team, to reduce the time it takes to integrate the code, and thus to encourage the frequent updates of code.

The CI platform is used as a centralised repository of build reports, status' and historical build updates for a project. Whereas before a CI platform was used, the console output and report from a build phase remained local to the developers machine, the console output is now available for review by all members of the team.

Jenkins support for CI

With the use of the scripting language groovy and an abundance of plugins, Jenkins provides the opportunity to customise the output of each build to provide insight and functionality to the pipeline.

Jenkins can offer support into the build times for each phase to diagnose bottlenecks and to understand the implication of pipeline updates to the project. Jenkins supports the use of concurrent builds and can therefore be leveraged to resolve bottlenecks from independent stages.
The graphic below provides a clear visualisation of the individual stages that can be defined within the groovy script.

Project build can be triggered within Jenkins in a variety of ways:

Jenkins can be integrated with Git repositories to initiate builds from commits. You can also set-up periodic builds to run in a more structured manner. As a centralised build tool between teams, the triggers for the builds can be used to share control of the pipeline across all teams to build as frequently, or infrequently, as desired.

Jenkins also supports exception handling within the pipeline. The console provides detailed log messages on each build:

Exceptions can be acted upon with email alerts for build successes and failures to alert the relevant teams and individuals for an end-to-end exception handling process to the software.

Conclusion

Continuous integration is a valuable framework used within DevOps with the objective of reducing the friction behind code updates across teams and streamlining the process for building a new artefact to be taken through delivery. Continuous integration can be defined by a single pipeline that performs each action upon the.

Continuous integration tools can be used to build pipelines that automate the process for integration code updates into builds in a centralised environment.
Jenkins is one such tool that can be used to build interactive reports to obtain an insight to the step-by-step process for builds. Jenkins also provides useful plugins to trigger builds, visualise performance reports and send email notifications upon events.

Comparing the mutual fund and exchange traded fund

Mon, 28 Sep 2020 16:34:00 GMT

Key Takeaways
• Explore the benefits mutual funds and exchange traded funds provide investors.
• Understand the differences between a mutual fund and exchange traded fund.
• Review the creation redemption mechanism for exchange traded funds.

Diversified funds for investors

Mutual funds and exchange traded funds (ETFs) are pooled investment vehicles that collect capital from multiple investors, both retail and institutional, to invest in a variety of securities to satisfy the fund's objectives.
The investment funds offer the investor the opportunity to diversify their investment across securities through a single fund rather than manually selecting securities themselves to invest in. The benefit to the investor is their reduced exposure to the volatility of a single security and to spread their exposure across entire industries, regions, and global markets.

The activity of investing within mutual funds and ETFs fall under the asset management division. Visit my blog here to learn about the different divisions and functions of investment banks. The institution that creates the mutual fund or ETF is known as the investment company. The two types of funds can be further segmented and designed to meet investor objectives for short and long term as well as for risk and return potential. This blog will explore the properties of each fund and the consideration an investor may take when choosing to invest in either.

Below are general definitions of keywords used within this blog:
Investment company: A corporation responsible for pooling capital from investors into securities.
Net asset value: the average value of all assets held within a fund, used to value the fund.
Expense ratio: an annual fee paid by the investor to the investment company as a percentage of the investment they make.

The different types of fund structures

Fund structures define how capital moves into and out of the fund and when shares of the fund are created.

Open ended funds do not set a limit on the amount of capital that it contains, and therefore the number of shares that can be purchased for the fund. If more capital flows into the open ended fund from investors, more capital is invested in the securities and the number of shares for the open ended fund grows at the same price.
If the investors choose to sell their shares, the investment company will act as a market maker to buy-back the shares of the open ended fund. The freedom for the portfolio manager to buy and sell securities within the fund mean the investment mix can change.

Closed ended funds will issue shares to the fund only once through an initial public offering. Once bought, shares from the fund are then traded through the stock exchange and are not bought back by the investment company itself. The price of the shares may be different to the NAV as a premium or discount reflecting the market's expectation for the NAV of the fund to grow or shrink with time.

Unit investment trusts (UIT) are a less commonly traded fund structures that place a timeline upon the life of the fund. The investment company will buy securities with a defined expiration date for when the assets are sold and the shares of the UIT are returned to the investors.
Shares are purchased at the NAV of the securities within the fund. Similar to closed ended funds, UITs will issue a set number of shares (units) that can be bought. The portfolio of investments for the UIT are established at inception and sold at termination. The investment company of the UIT may also act as a market maker to avoid the depletion of the UIT assets.

Introduction to mutual funds

A mutual fund has it's shares traded at the end of each trading day and is often actively managed. Mutual funds are more often open-ended to support the growth of assets within the fund.

The price of the mutual fund is derived from the average value of the investments that it holds. The 'Net Asset Value' is used to define the price of each share from the mutual fund calculated as the market value of all assets less the liabilities divided by the total number of fund shares in circulation.
For example if mutual fund 'M1' contains:
5 shares of A at £3
10 shares of B at £5
2 shares of C at £11
If there are 10 shares of M1 for sale, the NAV = {(5 x 3) + (10 x 5) + (2 x 11)} / 10
= 87 / 10 = £8.70 per share.
If the share price of A rises to £4, the new NAV of the fund will be £9.20.

As an 'actively managed fund', the strategy behind the mutual fund is typically very different to that of the ETF (as we will later review). Actively managed funds describe a fund strategy from the portfolio manager as to try and outperform the market. Outperforming the market requires the portfolio manager to selectively buy undervalued stocks and to sell overvalued stocks to profit from the bid-offer spread on the security. The portfolio manager will be supported by a team of research and credit analysts who offer advice in which securities to act upon. The objective of the actively managed fund is to therefore offer higher returns than the average market growth. While actively managed funds sound great, the investor will have to pay a higher expense ratio provided the additional expertise required to manage the fund, and can also expect a higher level of risk for the fund returns.

Introduction to Exchange Traded Funds

Exchange traded funds differ from mutual funds as they are traded on the exchange during the day at a price that closely reflect it's NAV. The function that enables ETFs to be traded during the day, known as the creation-redemption mechanism, is later reviewed in this blog.

ETFs tend to be passively managed, meaning the fund will invest in a basket of securities from a particular group without trying to select the 'winner' amongst it. Some passive funds, known as index funds, try to replicate the performance of an index. The performance for the fund is defined against how well they track the index and how low their expense ratio is.

While mutual funds tend to be open-ended, ETFs are often open and closed ended. Index funds will aim to track an index such as the S&P 500 or the FTSE 100 to obtain a broad market exposure for the investor. The exposure diversifies the fund and reduces the risk of price volatility to the investor. As the securities within each ETF change throughout the day, the NAV of the fund will change, and the investors within the market will be able to trade against the market price of the fund.

The creation redemption mechanism

One key difference between the ETF and mutual fund is the ability to trade the ETF like a stock during the day. How is the ETF priced? And where does the market liquidity originate from?
The creation-redemption mechanism (CRM) that enables the ETF to be traded like a stock is supported by a third entity in the relationship between the market and the investment company, the authorised participant (AP). An AP can be any large institution with enough buying power to purchase and hold securities to facilitate the liquidity of the ETF.

Creation

When the market requires shares in the ETF that are otherwise not available, the AP will send an order to the investment company requesting new shares of the ETF. The capital from the investor flows to the AP where new securities are bought that reflect the ETF. The AP will swap the securities for new shares of the ETF with the investment company and return the new ETF shares to the investor. The package of ETF shares and securities are known as a creation unit.

Redemption

When the market sells large amounts of the ETF shares, the AP will provide capital to the investor and exchange it for shares in the ETF. The AP will exchange the ETF shares with the investment company and will in-turn receive the underlying securities that the shares represented. The securities can then be sold in the market by the AP.

The creation and redemption mechanism of the ETF enable the shares of the ETF to be traded on the exchange during the day where the AP provides sufficient liquidity to the market for buying and selling shares and underlying securities of the ETF.

Conclusion

Pooled funds offer investors the opportunity to diversify their investments and to reduce their exposure to the volatility of a single security. The mutual fund and exchange traded fund are two popular investment funds that endorse different investment strategies to create varied risk exposure and returns for the investor.

By understanding the differences between fund structures, active and passive management and associate costs, it can become easier to differentiate funds by their objectives and suitability to the investors goals.

The Data Access Object pattern

Mon, 21 Sep 2020 16:34:00 GMT

Key Takeaways
• Understand when to use the DAO pattern and the benefits it brings to software design.
• Explore the use of Spring data JPA as a repository layer.
• Implement the DAO pattern to support multiple repository layer implementations.

The Data Access Object (DAO) pattern

The DAO pattern is used within software development to separate the service layer from the repository layer of an application. The service layer of an application will apply business logic to a user request that is received from a controller and will send the request to the repository layer where it is persisted into a data store. To learn more about the multiple layers of an application, visit a previous blog here.

The DAO pattern uses dependency inversion to loosely couple the service layer from the repository layer. The DAO abstraction enables the service layer to remain agnostic to the implementation of the data store, whether that be a relational database, lightweight directory access protocol, business to business data store or any other type of persistance. The DAO supports the use of the open-closed principle where the service layer can remain unchanged while the underlying data store is dynamic.

A top-down view of the DAO pattern

The DAO pattern is divided between the service layer which intercepts a request from the controller, and the repository layer which is used within the service layer to persist the change to the data store.

The DAO pattern introduces an abstracted relationship between the service layer and the data stores such that the service layer becomes agnostic to the implementations of the data store by the DAO:

Using the DAO design pattern

The DAO pattern in this blog is being used to demonstrate an Estate agent application that will use the repository layer to store properties that are for sale. The application will include a controller API to save, update or delete properties that are stored within the data store. The DAO implementation is used to encapsulate the data store for the service to use, and to access.

DAO Participants

The DAO defines 6 key participants involved with the creation and execution of a request with the persisted data store:
• Business user: The request from the user to obtain or modify data.
• Service: the business logic service to process the request.
• Entity object: instances of the data as an Object representation.
• DAOInterface: interface with repository methods to interact with the data store.
• DAOImpl: individual implementations of DAOInterface to interact with different data storage options.
• Data store: the storage of the data in its persisted form.

Implementing the DAO

The DAO is implemented as an interface for the CRUD operations to be performed upon the data store. The implementations of the interface enable the growth of the application to support further data stores as the DAO interface is passed as a service to the service layer to use.

The DAO interface exists below with descriptive operations to be performed upon the database:

public interface PropertyDao {

	public List<Property> getProperties();

	public void saveProperty(Property property);

	public Property getProperty(int id);

	public void deleteProperty(int id);
	
}

The DAO Implementation uses the Hibernate SessionFactory to create transactions to the data store:

@Repository
public class PropertyDaoImpl implements PropertyDao {

	@Autowired
	SessionFactory sessionFactory;
	
	@Override
	public List<Property> getProperties() {
		Session session = sessionFactory.getCurrentSession();
		Query<Property> query = session.createQuery("from property order by id", Property.class);
		
		List<Property> result = query.getResultList();
		return result;
	}

	@Override
	public void saveProperty(Property property) {
		Session session = sessionFactory.getCurrentSession();
		session.saveOrUpdate(property);
	}

	@Override
	public Property getProperty(int id) {
		Session session = sessionFactory.getCurrentSession();

		Property property = session.get(Property.class, id);
		return property;
	}

	@Override
	public void deleteProperty(int id) {
		Session session = sessionFactory.getCurrentSession();

		Query<Property> query = session.createQuery("delete from Property where id=:id", Property.class);
		query.executeUpdate();
	}

}

Implementing the service layer

The service layer is used to obtain requests from the controller to be passed down to the repository layer.

@Service
public class PropertyService {

	@Autowired
	private PropertyDao repositoryService;
	
    @Transactional
	public void save(Property property) {
		repositoryService.saveProperty(property);		
	}

    @Transactional
	public List<Property> getAllProperties() {
		return repositoryService.getProperties();
	}
	
    @Transactional
	public Property getProperty(int id) {

		Property property = repositoryService.getProperty(id);
		
		return property;
	}	
}

While it may appear to be an additional and unnecessary layer to access the PropertyDaoImpl, the service layer would typically provide additional functionality to the transaction such as logging, authorisation and security. The PropertyDaoImpl is passed in as a service to the layer to provide access to the data store. Each method is marked as @Transactional to assign transactional semantics to the method call. The semantics define the scope for a single database transaction using the Session instance created to synchronize changes that are persisted to the data store.

Using Spring data JPA as a repository layer

In the current application, the service layer creates transactional requests to the repository layer where a session with the data store is used. The DAO pattern is used to define the CRUD methods which are required to the data store.

The use of the Hibernate Session objects in the repository layer requires verbose configuration to create the bean for the Session. The Spring Data JPA abstraction offers an out-of-the-box interface to support the CRUD operations made to a data store thus simplifying the implementation of the repository layer.

The Spring Data JPA package can be obtained within the dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>

The Spring data JPA repository layer is implemented as an extension of the interface which can then be passed into the service layer as a service:

public interface JpaRepositoryImpl extends JpaRepository<Property, Integer>{
	
}

The JpaRepositoryImpl extends the JpaRepository interface and introduces two generic values for the interface. The Property generic is used to identify the entity each record of the data store will coincide with. The Integer identifies the type the primary key is for the table.

The interface is autowired into the service layer and provides the following methods amongst many others to support interaction with the data store:
• findAll() to return a list of all records as Property instances from the data store.
• findById(int id) to return a single record as a Property instance, identified by the primary key.
• save(Property) to update or save a record to the data store.

By implementing Spring data JPA into the application, the DAO is pre-defined to cover all the actions required to the data store. As a result, the extension of behaviour from the interface can be defined where the methods are hidden by the JPA abstraction. We can create a new extension of the interface with a different Entity Object as the Generic to define a new data store for the repository layer.
The Spring data package encapsulates all method calls to the interface with @Transaction behind the scenes. As a result, the service layer will no longer require the @Transactional annotation upon each method.

The updated Service layer will use the Spring data JPA interface:

@Service
public class PropertyService {

	@Autowired
	private JpaRepositoryImpl repositoryService;
	
	
	public void save(Property property) {
		
		repositoryService.save(property);	
	}
	
	public List<Property> getAllProperties() {

		return repositoryService.findAll();
	}
	
	public Property getProperty(int id) {

		Property property = repositoryService.findById(id).get();
		return property;
	}
}

Realising the benefits of DAO

The DAO pattern generalises the transactions that a Service layer can make to the repository layer thus opening the Repository layer to implementations of the transactions. We can use Spring Hibernate to provide a granular approach to defining data stores, sessions and transactions to handle the different implementations of the Repository layer.

As the extension of the repository layer is enabled by the DAO, we can use alternative strategies for the repository layer including the Spring data JPA repository implementation. Provided the DAO supports all the defined methods, the DAO can represent various Repository implementations across data storage medium, transaction definitions and sessions to benefit the application implementation and developer experience.

Conclusion

The data access object pattern is used to separate the service layer from the repository layer. As a result, multiple implementations of the repository layer can be used to represent different data stores and abstractions of the repository layer.
The Hibernate and Spring data packages support multiple designs of the repository layer to provide varying abstractions of the transactions made to the data store. Furthermore, the DAO pattern enables the support of multiple data stores and repository layers.

This blog has used the DAO pattern to support the growth of an application with multiple data stores. Two different strategies have been used to demonstrate how the DAO can be applied to different levels of abstractions of the repository layer. The specific support for database sessions and transactions can therefore be delegated to the implementations of the DAO to enable the repository layer to remain open to extension. You can find the source code from this blog on GitHub here.

Creating a Java GUI

Mon, 14 Sep 2020 16:34:00 GMT

Key Takeaways
• Explore the Java Swing package to create a GUI with Java.
• Understand how the Renderer interface can be used to create custom formatting.
• Use JTable to summarise array data into a table.

The Java Swing package

This blog post will use the Java Swing package to create a GUI from array data. The data will be formatted into a table and will render custom styling according to the values.

The following three classes from the Swing package are used to create the GUI:
• JFrame: a top-level container.
• JTable: a 2-dimensional table.
• TableCellRenderer: custom formatting to the cells of the table.

The final GUI will show the English Championship Football table for the 2019-20 season:

Creating the GUI container

The JFrame class is extended by the class Frame to create the GUI container. The Frame class has optionally used the Singleton design pattern to ensure we always reference the same container within the application.

The Frame class is created with the Singleton design pattern to ensure the application consistently refers to the same GUI instance.

public class Frame extends JFrame {

    private static Frame frame;

    private Frame(){

    }

    public static Frame getFrameInstance(){
        if(frame == null){
            frame = new Frame();
        }
        return frame;
    }
}

The main method is used to create an instance of the Frame class. The JFrame class provides further methods that are called by the main method to define key properties for the container:
• setVisible(boolean) to make the container visible.
• setSize(x, y) to set the size of the frame.
• setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE) to shutdown the application when the frame is closed.
• setTitle() to add a title to the window.
• setResizable(true) to allow the container to be resized by the mouse.

Frame frame = Frame.getFrameInstance();
        frame.setVisible(true);
        frame.setSize(400,400);
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.setTitle("My frame");
        frame.setResizable(true);

An alternative to the setSize(x,y) method is pack() which will automatically adjust the size of the frame so that all table content is at the preferred size.

Adding a Table to the GUI

The JTable class from the Swing package will be used to create the table. The JTable works similarly to a spreadsheet where each cell will have an x and y coordinate which is populated with data.

The class TableBuilder will be used to create a JTable. It will contain a static instance of the table which will populate data through the JTable constructor that accepts (Object[][], Object[]) to define the data and columns respectively:

public class TableBuilder{
    
    private static JTable table;

    public void buildTable(){

        table = new JTable(new String[][]{{"David"},{"Joe"}}, new String[]{"Column 1"});

    }
}

Alternatively, the data and columns can be obtained at runtime and passed into the JTable.

The JTable class offers further methods that can be used to modify the table including:
• addColumn(Object[]) to add in further columns to the table.
• setValueAt(Object value, int row, int column) to update the value of a cell.
• JTable(int row, int column) constructor to create a table with a set number of columns and rows.

Adding the table to the GUI

The JFrame container is used to display 'Components' that render information from the application. After creating the instance of the JTable in the main method through the TableBuilder class, the table can be added into the JFrame container through as a component.
The JScrollPane class provides a scrollable view component for the GUI. From our main method, we will pass our table into a JScrollPane instance which is then added into the JFrame.

    //main method

    JTable table = new TableBuilder().buildTable();
    JScrollPane viewComponent = new JScrollPane(table);
    frame.add(viewComponent);

When we run the application, we will see the following GUI with the rows and columns:

We can further enhance the table to include more descriptive column names and values that are passed into the table. For this example, I will use the final league table of the English Football Championship from 2019-20.

The previous constructor of the JTable took two arguments of Object[][] data and Object[] columns. An alternative constructor can take a DefaultTableModel to build the table. The DefaultTableModel uses a Vector of Vectors to store the cell values that can then be passed into the JTable as content.

    //TableBuilder class, buildTable() method

    String[][] data = {{/*data for table */}}
    String[] columns = {"Pos", "Team", "Won", "Drawn", "Lost", "Points"};

    DefaultTableModel model = new DefaultTableModel(data, columns);
    table = new JTable(model);

    return table;

The current table appears below:

However if we remove the frame.setSize(400, 400) call from the main method and replace it with frame.pack(), the JFrame will automatically adapt to the size of the table:

Custom styling of the GUI

The table can be enhanced by providing styling to the cells with the TableCellRenderer interface from the Java Swing package. The interface is implemented by the TableCellRendererImpl class and will override a single method getTableCellRendererComponent. The getTableCellRendererComponent method is called for upon each cell of the table and can process logic according to the table, cell value, isSelected, hasFocus, row and column arguments. The arguments provide a rich interface for fine-grained control of rendering.

public class TableCellRendererImpl implements TableCellRenderer {

    @Override
    public Component getTableCellRendererComponent(JTable table, Object value, boolean isSelected, boolean hasFocus, int row, int column) {

        return null;
    }
}

The TableCellRendererImpl class will define an instance of the defaultTableCellRenderer which is used to return a styled component back to the table.

    private TableCellRenderer RENDERER = new DefaultTableCellRenderer();

    @Override
    public Component getTableCellRendererComponent(JTable table, Object value, boolean isSelected, boolean hasFocus, int row, int column) {

        Component editor = RENDERER.getTableCellRendererComponent(table, value, isSelected, hasFocus, row, column);
        editor.setBackground(Color.white);
        editor.setForeground(Color.black);

        if(row % 2  == 0){
            editor.setBackground(Color.lightGray);
        }
        boolean topOfLeague = table.getModel().getValueAt(row, 0).equals("1");
        String relegationZone = (String)table.getModel().getValueAt(row, 0);
        boolean relegated = Integer.parseInt(relegationZone) > 21 ? true : false;
        if(topOfLeague){
            editor.setForeground(Color.green.darker());
        }
        if(relegated){
            editor.setForeground(Color.red);
        }

        return editor;
    }

The following business logic is applied to enable the table to reflect the 1st position and final 3 positions in the table: Line 6: The editor is instantiated to define the table contents.
Lines 7 & 8: The default background and foreground colours are set.
Lines 10 to 12: A soft background is applied to all even rows for visual effects.
Lines 13 to 21: The foreground colours (red or green) are defined based upon the value of the "Pos" column of the row.

Lastly, the implementation of TableCellRenderer is passed into the table through the setDefaultRenderer call from the main method. The first argument of 'Object.class' is used to set the default cell renderer for the column class. If the table consisted of an entity class, this could be used as the first argument to provide direct access to the class in the renderer.

    //main method
    table.setDefaultRenderer(Object.class, new TableCellRendererImpl());

The finished GUI will show the table data with custom styling for the first and final 3 rows to highlight the champions and relegated clubs. As a built-in feature, JTable enables the columns to be switched around with a drag and drop to order the fields as desired.

Conclusion

The Java Swing package supports the creation of components into a GUI for a Java application. In this blog, JTable has been used to summarise data into a table with further styling from the custom renderer. The capacity for creating a GUI with Java extends much further than the JFrame and JTable; the Renderer interface supports dynamic styling and the capability for further components such as buttons and mouse-listeners.

Creating a GUI can prove valuable for resolving data analysis and algorithmic solutions; I would encourage you to explore how the capacity for Java as a front-end component can be used to improve user interaction as a stand-alone application. The source code from this blog can be found on GitHub here.

Understanding investment bank services

Mon, 07 Sep 2020 16:34:00 GMT

Key Takeaways
• Understand the difference between retail banks and investment banks.
• Review the divisions that exist within an investment bank.
• Understand the core capability of each division within an investment bank.

My first blog on financial services...

For the past 5 months I have written about the software engineering practices and tools I encounter on a daily basis. I have decided to begin writing about financial services to accompany my technical blogs to reflect upon the context of my job. My bachelors degree and first job aligned to investment management and I continue to find the industry interesting, even more so due to the growing importance that technology plays in the long-term strategy for all the players within the industry.

Introduction to financial services

Financial services can be divided into many sections, in this blog, I have split financial services by the clientele they serve into retail (also known as commercial banking) and investment banking. Both types of banking involve the matching of 'buyers' and 'sellers', however they differ greatly by 'who' they serve.

Retail banking constitutes a service that is provided to individuals and small business owners who would like a loan or mortgage for their livelihoods. The bank will offer a loan to the mortgage buyers with an interest rate of 'r' (e.g 3%) on the repayments. On the other side, the retail bank will also accept deposits from sellers who wish to save their excess funds. The seller is enticed to deposit their money for an interest repayment on their savings of 's' (e.g 2%). The retail bank will earn a profit through the loan repayments where 'r' is greater than 's' (e.g 1%). This is a general idea of retail banking in the public sector, there are many more services provided within retail banking, however the remainder of this post will look into the operations of an investment bank.

An investment bank provides advisory and funding services to customers including corporations, high net worth individuals (HNWI), and government entities. The services provided enable the use of capital markets for the clients and involve much larger volumes of capital. As a result, investment banking can be a more lucrative sector within financial services.

Divisions of an investment bank

The umbrella term 'investment bank' can often take different definitions and meaning. 'Investment bank' must not be mistaken for the 'investment banking division' as the latter is used to describe a group of functions performed by an investment bank. The divisions are used to group the services provided by the bank and are categorised based upon the clients that are dealt with, the services provided, and the style of revenue generation to the bank. The divisions of a bank also supports the regulatory compliance towards information sharing, I will touch upon this topic later.

The image below illustrates the divisions that can exist within an investment bank.

The 'Chinese wall' that splits the investment banking division from the others is a virtual information barrier to separate non-public information and public information. The investment banking division will often have access to non-public information due to their research and client services, and must therefore use to a Chinese wall to prevent insider information from spilling into the other divisions.

Some banks will define themselves as an investment bank when they provide services across multiple divisions, however other banks, which specialise in a certain division, will identify themselves to the division. For example Charles Stanley Wealth Management vs Barclays Investment bank.

Core services of the investment bank

In the following section, I will describe the core services of each division. The terms 'buy side' and 'sell side' are often used to describe the nature of revenue generation for the function, they are briefly defined below:
Buy side: services involved with directly investing capital to purchase securities.
Sell side: services involved with issuing, selling, or trading securities and advisory services.

Investment banking division (IBD)

Underwriting
Underwriting is a primary service within IBD. The process of underwriting sees the bank support a client (business, corporation, government) with raising capital through the sale of stock (equity) or bonds (debt). The investment bank will receive a percentage of the overall value of the underwritten securities as compensation.

The underwriting process can take the form of either an initial public offering (IPO) or a follow-on offerings. An IPO refers to the process of offering the shares of a private corporation to the general public (thus making it a public corporation). The issuance of new stock is used to raise capital for the corporation. A follow-on offering is the issuance of stock after the IPO. During the IPO, the investment bank will make one of three agreements for underwriting:
• Full commitment. The investment bank will buy all the securities and sell them in a secondary market.
• Best efforts. A good faith promise is made to sell as much of the security as possible.
• All or none. The IPO will only go ahead if all the securities are sold.

Market Making
As a market maker, an investment bank can act on both sides of the sale for a security. A bid (buy) and ask (sell) price can be quoted by the investment bank and a profit can be realised from the spread (ask - buy) of the security when it is later sold. The investment bank will take on a risk of holding a security as it may depreciate with time, however their market making services provide liquidity to a security that would otherwise not be sold in the market.

Advisory
Advisory services include research and guidance on M&A (merger and acquisition) and corporate strategy.

The M&A advisory service includes the negotiation and structuring of a merger or acquisition between two corporations. The corporations can be public or private, and the investment bank are able to provide support through their expansive network to find partners and deep-grained industry knowledge on the markets.

Corporate strategy advisory can be used to support the short and long-term strategic direction of corporations in the ways of financing change.

Asset management division

The asset management division is responsible for managing a fund of financial holdings from investors that a risk tolerance and short and long term goals. Investments will be made across instruments such as stocks, bonds, real estate, commodities and derivatives (futures, options, swaps). The client can be HNWI as well as high-end institutions. The relationship between a client and the fund manager is important to maintain as the clients may seek support in reevaluating goals or understanding the opportunities that lie ahead. The service of asset allocation provides advice on how to divide investable assets into growth products (stocks) and fixed income produces (bonds).

Funds will be managed by a portfolio manager who will utilise the research and guidance of their teams to make investment decisions upon the fund. The fund will have a specific criteria to adhere to including the risk tolerance, type of securities that are invested upon, and liquidity.

Capital Markets division

Capital markets are where suppliers with excess capital and those in need of capital meet to satisfy their ambitions. Capital markets in this sense are similar to retail banks with mortgages and savings accounts, however the entities involved are corporations, governments and dedicated individuals who understand and trade upon the market. Securities are exchanged within a primary market of newly issued securities and secondary markets for re-sold securities.

Sales
The sales component of capital markets will be responsible for suggesting trading ideas to their clients and to take orders from them. The sales component will also consist of financial analysts that provide support in research and strategies. Once made, the order will be communicated to the trading desks where they are priced and executed.

Proprietary Trading

'Prop' trading is conducted by a set of traders who invest directly within the capital markets themselves on behalf of the bank, and not the client. The investment bank will directly profit from the market activity as opposed to the margins from client profits they realise.

Structuring
Structured products provide access for retail investors to derivatives by using complex strategies to switch the underlying payment flows of a security with a fixed value that is derived from a derivative. The complexity of a structured product will require a blog itself to explain, however the summary is that it provides access for retail investors to complex derivatives.

Private Banking division

The private banking division is split into two separate services that cater to HNWI: private banking and wealth management.

Private banking provides a personalised and concierge-type service to HNWI for managing their finance. The products offered within private banking are largely identical to those offered within retail banking, however the client to the bank will experience a personalised level of service. Private banks will often use a 6-figure benchmark as a deposit for clients of the bank. The private bank will also offer advisory services for planning, retirement, taxation and inheritance for the client when managing their assets.

Wealth management involves the optimization of a client's portfolio of assets with regards to the individuals risk, goals and asset preference. Wealth management is more closely aligned to the growth and protection of the clients assets and includes services such as estate planning, retirement planning and education planning.

Private Equity division

The private equity division will invest in private institutions with the objective of achieving a mark-up in their value and a return on service fees. Private institutions will not have had an IPO and are therefore not traded on the stock exchange. The private equity division will have many funds that consist of investments from HNWI, corporations, and also loans from banks. The fund is used to purchase multiple private institutions to renovate or provide advisory services to. The growth and enhanced performance for the acquired institution will eventually drive the share price up enabling the private equity fund to realise the profits on the resale of the institution. The private equity division will realise profits from a fee of managing the fund and from the percentage share in the fund they had when the private institutions are sold.

Conclusion

Financial services is a broad industry that incorporates a vast array of services from advisory and research to execution and market making. The differences between these services can depend upon the client being served, the institutions being managed, and the active involvement with investments themselves. Services can be split between 'buy side' and 'sell side', however the overarching principle within financial services is to aid in the movement of money. This blog has only touched upon the functions of each division, however I will use this as a foundation to reflect upon in later posts when I look slightly deeper into how financial services works and why technology has such an influential role in strategy.

The Dependency inversion principle

Mon, 31 Aug 2020 16:34:00 GMT

Key Takeaways
• Understand the use of dependency inversion between high and low level classes.
• Implement dependency inversion with abstraction to decouple class dependencies.
• Compare the open-closed principle and the dependency-inversion principle to each other.

Dependency inversion principle

The dependency inversion principle (DIP) aims to decouple the high level classes from the low level classes that they use. In this post, I will walk through how high and low level classes can appear within and application, how their tightly-coupled relationship can cause problems, and how dependency inversion can be used to loosely couple the classes. We aim to loosely couple high and low level classes to remove the side-effects that can arise within the high level class when a change in the low-level class is made. The dependency inversion principle is closely linked to the Liskov substitution and open-closed principle of the SOLID design principles.

The term 'dependency inversion' must not be confused with the term 'dependency injection'. Dependency injection involves a class demanding the use of another class. Whereas dependency inversion involves one class requesting that the dependency is handled by another class. We will cover this small detail a little bit more in the blog.

High level and low level classes

So what are high level and low level classes? In short, high level classes depend upon or use low level classes to carry out their behaviours. We can see this in action with the following example of a class that is used to check if flights to certain continents exist:

class FlightChecker{

    EuropeFlights europeFlights = new EuropeFlights();
    AsiaFlights asiaFlights = new AsiaFlights();

    public void checkFlights(int flightNumber){

        if(europeFlights.getFlights().contains(flightNumber)){
            System.out.println("Flight to Europe exists");
        }
        else if(asiaFlights.getFlights().contains(flightNumber)){
            System.out.println("Flight to Asia exists");
        }

    }
}

The FlightChecker is tightly coupled to the lower level classes of EuropeFlights and AsiaFlights. As a result of the tight-coupling, if the low level classes were to change (such as changing the signature of the getFlights() method, the FlightChecker would break where the getFlights() method is called. The FlightChecker uses dependency injection as it demands the application to provide an instance.

For this example, we are to assume the EuropeFlights and AsiaFlights classes will pull in all the details of available flights from their respective databases to the FlightChecker class.

Introducing dependency inversion

The use of dependency inversion is often associated to the following two rules:

"High level classes do not depend upon low level classes. They depend upon abstractions."

"Abstractions do not depend upon details; details depend upon abstractions."

The two rules imply the high level classes must use abstractions to obtain behaviours, not through direct links with the low level classes. The second rule asserts that the details upon the low level classes are to be defined by the abstractions. Therefore the low level classes use the abstractions to define their own implementation.

To achieve dependency inversion, the high level classes must be abstracted away from the low level. What we are aiming to achieve is a design where a change in the low level class does not break the high level class.

Abstracting the dependency

To create the abstraction for the high level class, we must ask ourselves what 'contract' we want to enforce upon the low level classes. As we can see from the current state of the example, both the EuropeFlights and AsiaFlights class use the getFlights() method. Therefore I will create a new interface that has a single method getFlights() and I will have the EuropeFlights and AsiaFlights classes implement the interface.

public interface WorldFlights{
    public List<Flights> getFlights();
}

public class EuropeFlights implements WorldFlights{

    List<Flights> flights;

    public List<Flights> getFlights(){
        return this.flights;
    }

     public String toString(){
        return "Europe";
    }

}

public class AsiaFlights implements WorldFlights{

    List<Flights> flights;

    public List<Flights> getFlights(){
        return this.flights;
    }

    public String toString(){
        return "Asia";
    }
}

The abstraction of the WorldFlights interface is now passed into the FlightChecker:

class FlightChecker{

    WorldFlights worldFlights;

    public void checkFlights(int flightNumber){

        if(worldFlights.getClass.equals(EuropeFlights.class) ||
            worldFlights.getClass.equals(AsiaFlights.class)){

            if(worldFlights.getFlights() != null){       
                System.out.println("Flight to " + worldFlights.toString() + " exist.");
            }
        }
        
    }
}

The above example has utilised the worldFlights abstraction for the EuropeFlights and AsiaFlights class to create an abstraction between the low level class and the high level class. The class will check the instance type of the abstraction and return a print of the class. The FlightChecker is now inverting the responsibility of using the EuropeFlight or AsiaFlight class to the abstraction that will provide the implementation.

But isn't this similar to the open-closed principle?

Our use of abstractions with the DIP can sound familiar to the open-closed principle where we use abstractions to close a contract to modifications while leaving it open to extension of new implementations. Similar to the DIP, the OCP places an abstraction between two classes.

If we take our example of the FlightChecker, the abstraction of the WorldFlights class has removed the direct dependency of the high level class (FlightChecker) from the lower-level classes (EuropeFlights and AsiaFlights). But the question remains: has dependency inversion inadvertently fulfilled the OCP, and if so, why are they separate principles?

If we were to introduce a new class: AmericasFlights, which will return the flights to America, the current design will support the extension of behaviours as the AmericaFlights class can implement WorldFlights:

class AmericasFlights implements WorldFlights{

    List<Flights> flights;

    public List<Flights> getFlights(){
        return this.flights;
    }

    public String toString(){
        return "Americas";
    }

}

The dependency inversion principle does not however satisfy the closed for modification principle. The extension of the WorldFlights interface does not naturally fit in with the FlightChecker class. The FlightChecker class requires an update on lines 7 & 8 to check if the instance is of the AmericasFlight class. As a result, the AmericasFlight class will not evaluate to true in the if-statement.

The example I have used deliberately evaluates against each implementation of the extension for WorldClass to illustrate how dependency inversion can be applied without satisfying the open-closed principle.

When we think about dependency inversion, we are distributing the responsibility for detail to the lower level classes and generalising the higher level class. The higher level classes can generalise the requirements they have upon a dependency by delegating the responsibility of detail towards the lower level classes through an interface.

When we think about open-closed principle, we are again using abstractions to enforce certain properties and behaviours to be injected into the higher level class, however we are designing the application to be enhanced with new behaviours that accommodate to the schema for the abstraction. You may find the open-closed principle is supported through dependency inversion, however this would be dependent on the use of the lower level classes within the higher level classes and the requirement for extensibility in the application. For example, what if our lower-level classes used dependencies as well? Then we would need to further abstract behaviours to satisfy the OCP.
The subtle differences between the principles are important to individually evaluate our design upon.

Conclusion

The dependency inversion principle applies to the relationship between a high level class and the low level classes it uses (or depends upon) to perform certain behaviours. The direct relationship between a high and low level class can create a tight coupling where details of the implementation of behaviour can become lost and complicated between classes.

The introduction of an abstraction between the high and low level class creates two effects:
• The higher level class is able to generalise the dependency and behaviours it requires to eliminate any direct relationship with the lower level classes.
• The lower level class details depend upon the contract from the abstraction. They are no longer capable of breaking the higher level class from change.

If you would like to dive deeper into the ways dependency inversion can be applied with Spring, visit my blog here for more information.

The Interface segregation principle

Mon, 24 Aug 2020 16:34:00 GMT

Key Takeaways
• Understand the benefit of abstraction that interfaces bring to design.
• Review the problem of grouping multiple behaviours into a single abstraction.
• Leverage interface segregation to control the behaviours a class must implement from a combination of interfaces.

Interface segregation

Interfaces allow us to achieve an abstraction of behaviours to be implemented by other classes. The abstraction of interfaces enables us to define the behaviours a class must have when they implement an interface 'contract'.

Interfaces have been prevalent throughout the SOLID principles, most notably used with the open-closed principle here.

One of the differences between abstraction and inheritance is that a class can implement multiple interfaces, thus implementing behaviours from multiple classes, however a class can only ever extend a single class.
The interface segregation principle (ISP) encourages well-defined and meaningful interfaces where the behaviours implemented by the class do not pollute its well-defined responsibility.

Much like the Liskov substitution principle, where we aim to inherit classes with the appropriate behaviours, the ISP aims to achieve a similar separation of behaviours for each implementation of an interface.

The contract of interface implementation

The abstract contract of an interface requires the implementing class to take on each behaviour within the interface, and not just selected behaviours. When we design an interface, we must therefore be considerate to the methods we are assigning together as they form the responsibility for the implementing class.

The case for interface segregation arises when classes implement an interface with methods that are not required by the class. As a result, the class obtains excessive behaviours that do not conform to the responsibility of the class. You may think "why not just assign the behaviours without the interface?". This approach will not leverage the benefits of abstraction and can create further issues down the line as the application grows. The Strategy pattern can be valuable to use within our design, therefore we will aim to maintain the use of interfaces while paying consideration to the obstacle of implementing unwanted behaviours.

Interface segregation in action

The below code example demonstrates an interface used for using the DAO design pattern for interacting with a database. The DatabaseOps interface is used for reading and writing to the database and enables the implementations to make both types of requests.

public interface DatabaseOps{

    void testConnection();
    void read();
    void write();
}

There may be many implementations of the DatabaseOps interface, however the implementation of a class to only complete one of the behaviours is not possible. An issue may arise where we would only like to designate the read behaviour to a class, and not the write behaviour as well. The below method illustrates the use of the Strategy pattern where the DatabaseOps interface is a required interface implementation to make calls to the database:

public void makeDatabaseTransaction(DatabaseOps operation){
    operation.read();
}

Segregating the interfaces

We are able to achieve interface segregation by dividing the responsibility of the DatabaseOps interface into two further interfaces that extend from the DatabaseOps interface. The DatabaseOps interface is now split into two further interfaces, ReadDatabaseOps and WriteDatabaseOps:

public interface DatabaseOps{
    void testConnection();
}

public interface ReadDatabaseOps extends DatabaseOps{
    void read();
}

public interface WriteDatabaseOps extends DatabaseOps{
    void write();
}

The image below illustrates the change of the interface design.

The change in interface design means the behaviours of reading and writing are separated, however they can both be contracted to a class that implements both the ReadDatabaseOps and WriteDatabaseOps interfaces.

When to use interface segregation

The interface segregation principle should be considered whenever we create an interface. Questions we may want to ask ourselves when designing an interface include: "do we these behaviours to be grouped together?" or "will these behaviours be required within a class on their own?".

We can use the extension of interfaces to design single-purpose, but related abstract contracts that prevent classes from defining behaviours they do not require.

Conclusion

The interface segregation principle extends upon previous SOLID principles including the single responsibility principle and the liskov substitution principle. When we define interfaces, the implementations of the class are asserted into implementing the behaviours of the interface together. The interface segregation principle aims to break-down the abstract behaviours of interfaces into smaller pieces that can prevent unwanted behaviours being implemented for a class.

The interface segregation principle leverages the inheritance of interfaces to break-down and separate behaviours, while class design enables a class to implement multiple interfaces to loosely regroup the behaviours that are required.

Series link: Go to the next SOLID principle: Dependency Inversion Principle

Liskov substitution principle

Mon, 17 Aug 2020 16:34:00 GMT

Key Takeaways
• Understand the importance of the Liskov substitution principle to inheritance and shared class behaviour.
• Review the rules that ensure the inherited behaviours of subclasses from their parents are as expected.
• Design tests to ensure the behaviour of a parent class is not modified by its subclasses.

What is the Liskov Substitution Principle?

Introduced by Barbara Liskov, the Liskov substitution principle (LSP) is used to decide when it is appropriate to extend a class as opposed to other strategies of abstraction to achieve the goal of shared properties between classes. As we have seen in the previous blog, the open-closed principle can be used to abstract the behaviours of a method; the Liskov substitution principle is used to extend upon the design by contract approach where the behaviour of the class is the primary concern.

The Liskov substitution principle is used to make assertions upon the substitution of behaviours that are implied by a subclass from its parent. When a class is extended, we are in effect implying that the subclass "IS A" extension of the parent class. While the open-closed principle is a focussed upon the structure of a class, LSP promotes strong behaviour sub-typing of classes to the parent class to ensure the expected behaviour of the methods are applicable to the extension.

The below quote is often used to describe the importance of behaviours between inherited classes:
"If it looks like a duck, quacks like a duck, but needs batteries - you probably have the wrong abstraction"

Liskov Substitution by design

When we use inheritance to share class methods and properties, we are implying that the subclass can be used to replace instances of the parent class.
The LSP asserts three rules when applying inheritance with subclasses:
• The subclass cannot enforce stricter rules than the parent class.
• The return value of a subclass method can only differ as a subclass of the return type from the parent.
• The subclass of a parent must be completely capable of substitution for its parent across each behaviour.

Implementing LSP by design

If we reflect upon the rules above, the first two can be validated through the Java compiler and are therefore inherent with Java. A problem can arise with ensuring the behaviour of each method is appropriate for the subclass of the parent.

Demonstration: Is a Penguin a bird?

The below code example demonstrates how easily behaviour can be modified from the intuition that a subclass IS AN extension of a parent class. When we design an application of Birds, Penguins, and Seagull Objects, we may reflect upon the use of class inheritance by saying, out loud, "a Penguin is a Bird, and a Seagull is also a Bird". In our source code, we may therefore create the Seagull and Penguin as subclasses of Bird.

class Bird{
    private boolean hasWings = true;

    public void fly(){

    }
    public boolean getWings(){
        return this.hasWings;
    }    
}

class Penguin extends Bird{

}
class Seagull extends Bird{

}

In our example above, we have confused the properties of the Bird class with the behaviours of the Bird class when creating the Penguin class. The Penguin is a bird because it has the properties of a bird: it has wings, lays hard-shelled eggs etc. However the Penguin does not behave like a bird in the sense that it cannot fly.

The LSP has been breached as the Penguin class is now capable of calling the .fly() method; a behaviour that should not be possible.

Complying with LSP

To resolve the problem, the behaviour of the Bird class can be further split into two: BirdCanFly and BirdCannotFly:

class BirdCanFly{
   private boolean hasWings = true;

    public void fly(){

    }
    public boolean getWings(){
        return this.hasWings;
    } 
}

class BirdCannotFly{

   private boolean hasWings = true;

    public boolean getWings(){
        return this.hasWings;
    } 
}

The Penguin and Seagull can now extend the appropriate class to obtain the Bird properties with and without the flying behaviours:

class Penguin extends BirdCannotFly{

}

class Seagull extends BirdCanFly{

}

The example above illustrates the potential conflict that can arise with the implied substitution from inheritance. The behaviours of flying can be further imposed by the parent class with methods such as "getTopFlightSpeed" and "land" that would imply further irrelevance for the Penguin class without the redefined BirdCannotFly class. The

Conclusion

The LSP is applied to application design when we consider the behaviours of classes that are inherited from other parent classes. Inheritance is one of the most valuable OOP concepts and therefore LSP is an important consideration to make upon child classes whenever a class is extended.

While it may seem natural to extend classes that fit into the "IS A" Object construct, we must consider how the behaviours of a class are inherited, and not just the properties. LSP can be implemented by reviewing the behaviours each class will inherit before adding behaviours to a parent class or extending them, or they can be implemented through unit testing of a parent class with each of their child classes.

Series link: Go to the next SOLID principle: Interface segregation Principle

The Open-closed principle

Mon, 10 Aug 2020 16:34:00 GMT

Key Takeaways
• Understand the importance behind the open-closed principle to code maintainability and growth.
• Identify when the open-closed principle should be used within an application.
• Use the Strategy design pattern to avoid breaking the open-closed principle.

What is the open-closed principle?

The open-closed principle aims to design behaviours that are open to extension, but also closed for modification. The two behaviours can sound like an oxy-moron, however they create two distinct characteristics that can be applied together.

Open for extension describes a method where the runtime execution can be modified when new components are required and added to the application.

Closed for modification design asserts that as new components are added to the application, the dependent components upon the new component do not require modification. The pre-existing behaviours will also work as designed despite new components being added. The new components therefore work themselves around the behaviours of the application.

Why the open-closed principle is important

The open-closed principle is important with respects to supporting the growth and enhancement of an application. As applications grow in size, dependencies and complexities arise which can make it difficult to provide additional functionality without breaking the existing co-dependencies. When we design applications, it is important to consider if certain features are likely to grow or exist in different states. The open-closed principle can be applied to ensure the growth of application behaviours are supported by the design.

The below code sample is an example of a Console class that is able to play different games:

public class Console{

    public void play(CarGame game){
        game.load();
    }
}

public class CarGame{
    public void load(){
        System.out.println("loading car game");
    }
}

The CarGame class is injected into the Console class and the play() method is used to load the game.
The play() method takes the CarGame as an argument, therefore it is tightly coupled to the CarGame class. If a new class, AdventureGame was introduced, the play() method will not be able to use the AdventureGame, and a new method would need to be created. The play() method is not open to extension as an extension of the AdventureGame is not supported, and it is not closed to modification as the argument of CarGame will need to be changed to accommodate for the new AdventureGame.

Introducing the Strategy design pattern

The Strategy design pattern is used to allow the application to change its behaviour at runtime. Strategy enables different behaviours to be interchangeable in the original context by supporting the extension of the application without updating the original method.

The Strategy design pattern introduces "programming by interface", away from the default "programming by implementation". The context of the application becomes dependent upon the behaviours of an interface and not the behaviour of an implementation of the interface.

If we look back at the CarGame class, we could think of it as an implementation of an interface with just the single method load(). The Strategy design pattern encourages the use of an interface which is then referenced by the context so that it accepts any implementation of the interface.

Implementing the Strategy design pattern

Before we make the updates to the Console and CarGame classes, I will review some key terms used within the Strategy design pattern:

•The context class stores the method that is open for extension and closed to modification. The method in the context class must not change as new components are added to the application.
•The strategy is the interface that is passed in as an argument to the method within the context class method.
•The extension of the application is created through different implementations of the strategy.

To begin, I will create the strategy interface which will be used by the context class.

public interface Game{

    void load();

}

The Game interface will be used as the argument in the context class: Console:

public class Console{

    public void play(Game game){
        game.load();
    }
}

The extension of the strategy class will now be implemented in the CarGame class. It will implement the Game interface:

public class CarGame implements Game{

    @Override
    public void load(){
        System.out.println("loading car game");
    }
}

The CarGame class will override the load() method from the Game interface.

When the application requires further functionality, such as an AdventureGame, the extension of the strategy class can be taken further by creating a new implementation:

public class AdventureGame implements Game{

    @Override
    public void load(){
        System.out.println("loading adventure game");
    }
}

The play() method of the Console class is now open to extension as it is designed by implementation of the Game class, and is closed to modification as further components of the application will be accommodated for by the play() method provided they implement the Game interface.

Conclusion

The open-closed principle supports the growth of application behaviours with maintainable and robust code. The strategy pattern enables the open-closed principle to be realised through the creation of interfaces and their placement as a dependency.

When we are designing our applications, it is important to always consider if a behaviour is likely to exist in another manner. The strategy pattern becomes a more and more valuable design to use as the application behaviours and complexity grows with the addition of new behaviours. The three components of the strategy pattern: context, strategy and extension are valuable components to understand and practice as they will appear more and more frequently during application design processes.

Series link: Go to the next SOLID principle: Liskov Substitution Principle

The Single responsibility principle

Mon, 03 Aug 2020 16:34:00 GMT

Key Takeaways
• Introduction to the SOLID principles of design.
• Understand how the principle of Single Responsibility can be applied to individual classes.
• Review the benefits of the SRP.

The SOLID principles of design

This blog post will be the first in a series of 5 posts that cover the SOLID principles of design. The SOLID principles were introduced by Robert C. Martin in his paper Design Principles and Design Patterns. The SOLID acronym was later defined by Michael Feathers.

The SOLID principles encourage the creation of maintainable software that prevents code rot from accumulating within object orientated programming. As software applications scale in size, the design principles from SOLID become more and more valuable to prevent complex issues from arising and to enable a more efficient process for application enhancement and bug fixes.

The SOLID acronym is defined accordingly:
S - Single responsibility principle
O - Open-closed principle
L - Liskov substitution principle
I - Interface segregation principle
D - Dependency inversion principle

This blog post will focus upon the single responsibility principle (SRP).

Single responsibility principle

The single responsibility principle asserts that any one class must have just one responsibility, and therefore just one reason to change.
Classes that use the SRP will be able to track each of their behaviours back to the single responsibility. Alternatively, classes with multiple responsibilities may contain many reasons to change. As we will later discuss, an update to a class with multiple responsibilities may cause unrelated side effects to the function of the class that can cause further problems within a complex application.

The SRP creates classes that are described as being cohesive and robust. A sign that a class has a well defined purpose and singular responsibility can arise when you are naming the class against the responsibility. If you find it difficult to define a name of a class based upon its responsibility, it is likely the class has many responsibilities or is not clear to a single purpose.

Class cohesion

The word cohesion describes the action of forming a united whole. In object orientated programming, classes are cohesive when they can be used together without conflict. Class cohesion is achieved through the design and implementation of a class to a single purpose. Class cohesion facilitates greater potential for class reusability and implementations of the application.

The implementation of class cohesion can be visually demonstrated through the blocks below:

While the blocks are only rotated on the x axis, it is visible that when classes consume just a single responsibility, they become more modular and can be combined with different classes to form different implementations.

Implementing SRP by design

The benefits of SRP will be demonstrated by realising side effects from changing a class with multiple responsibilities.

In our example, the application will be used for renting a car. The class Car exists as an entity to be stored within a database of all cars to be rented. The class will also return properties of the car such as brand, colour and engine size.
The Car class will also include a method called "rent" which sends a request to the database to rent the car.

The Car class has more than one responsibility. It is responsible for holding information about each car instance, but it is also responsible for managing the rental of the car with the database.

public class Car {

    private String brand;
    private BigInteger price;
    private String colour;
    private String key;
    private InventoryService inventoryService;
    
    public void rent(){

        if(inventoryService.getInventory(key) > 0){
            
            ...

        }

    }

}

The rent() method will verify with the Inventory class (not demonstrated in this post) that a car is available before it connects with the database using object to relational mapping to record the update in rental.

The following changes are made to the application:
1. The number of cars has increased in the inventory.
2. The renter must verify their credentials before renting a car.

Increasing the car inventory

The Car class uses the InventoryService to check how many cars are available for rent. When the number of inventory is increased, the Car class, which is responsible for holding information about a car instance, should not be effected. After all, the number of instances of a car does not relate to the properties of a car. As a result, the InventoryService, which is responsible for managing the inventory of the car instances, will be updated. This example demonstrates how SRP creates a single change in a single class (InventoryService) that is responsible for a property (Inventory).

Verify renter credentials before renting

The verification of user credentials is a pre-requisite before enabling a car to be rented. As a result, the Car class must use another service to access the credentials during the rent() method. The Car class, which is meant to be responsible for the car properties information, will now use the RenterDetails class to access information about the renter.

The result of verifying credentials of a renter has created unexpected growth and unrelated dependency of the Car class upon the RenterDetails class. The additional responsibility of the rent() method in the Car class has created two further dependencies on the RenterDetails and InventoryManager classes that otherwise would be completely unrelated to the Car properties.

Reviewing SRP in the Car class

Before the SRP is applied to the Car class, it has a dependency upon the InventoryManager to accommodate for the rent() method. This dependency expands the single responsibility of providing properties of a single car into a further responsibility in renting an instance of the car with the database.

When the application introduces two further requirements, the InventoryManager is not changed, however the Car class must now depend upon the RenterDetails class to verify the person renting the car. The additional dependency breaks the responsibility of the Car class even further:

If the Car class had a single responsibility, it would not contain a rent() method and therefore the InventoryManager would not be injected. The rent() method could belong within a different class, "CarRental", that is responsible for leasing cars. CarRental will use the InventoryManager to obtain inventory levels for each car.

The further requirements of the application can be introduced by updating the inventory in the InventoryManager and by injecting the UserDetails class into the CarRental class. Each class maintains its single responsibility, however the updates made to the classes have not increased their responsibility and kept the functions of the application separate from each other.

The benefits of SRP

In the example above, SRP has been used to separate the responsibilities of the classes. The update and enhancement to the application has been made in expected classes and have not expanded their original responsibility. The SRP version of the application promotes the single reason for change as the enhancements only prompt a single change in the class with the responsibility for handling the inventory and for the user verification.

Debugging
If the application were to experience a bug with inventory handling or user validation, the debugging process would initially begin with the expected classes with the responsibility that handles the functions. Without the SRP, the debugging process would have been slightly more complex as the Car class had the excess dependencies upon the InventoryManager and RenterDetails classes. SRP improved debugging an application as responsibilities are specific and adhered to within each class.

Loose coupling
The Car class that uses the SRP has decoupled the responsibility of rental from the responsibility of obtaining Car details. The individual classes can be reused whenever car details are required or leasing services are needed. They are services for a specific purpose and can be reused more efficiently than a single class with many responsibilities.

Testing
Automated testing (unit testing) is typically designed to test the functionality of a class. Each class will have several tests to ensure it is able to complete it's responsibilities. SRP will naturally decrease the collective number of tests required for the application as there are fewer behaviours that can be combined within a single class.

Conclusion

The single responsibility principle asserts that each class will have a single defined responsibility and therefore just a single reason to change.
An application that complies with the SRP will enable the developer to understand the class dependencies and relationships in a more natural sense. The SRP will facilitate the debugging and enhancement of the code as new responsibilities and functionality is introduced.

Series link: Go to the next SOLID principle: Open-Closed Principle

Git version control fundamentals

Mon, 27 Jul 2020 16:34:00 GMT

Key Takeaways
• Understand how to store and manage files using Git.
• Use Git to manage branches and make pull requests for source code reviews and inclusion.
• Use Git to update branches with the latest updates from the remote repository.

What is Git and version control?

Git is a distributed version-control system that enables you to manage your code and updates in a centralised repository. Git stores snapshots of the source code each time it is saved to enable you to revise historical updates to source code and track changes. We use Git and other version control systems to control the updates and revisions made to our source code in a distributed contribution environment.

Version control is equally valuable to all developer projects regardless of the number of contributors; it formalises the process of creating, updating, and reviewing code before it is submitted to the main application source code.

In this blog post, I will be using GitHub to manage source code. GitHub is a Git repository hosting service that provides a GUI for interacting and collaborating with Git repositories. Git is used to store local source code in a remote (online) location. The remote location is then accessed by all contributors and updated and reviewed in unison.

Creating a repository

A repository can first be created in GitHub from the website by selecting 'New Repository'.

The git repository is mapped into a local directory. Create and navigate to the directory from the command line and initialise the directory as a Git repository with the following command:

git init

Git init will create a .git file in the directory that creates an empty repository. The initilisation makes other git commands available that can be used to add and manage source code within the local and remote repository.

Adding code updates

I have created a new project in the local directory where git init was called to complete the following print function.

public class Application {

    public static void main(String[] args) {

        System.out.println("Hello from the main branch");
    }
}

The remote GitHub repository is currently empty and the our local repository contains source code that is yet to be added. The following command line instructions can be used to move the local source code into GitHub.

git status

git status allows you to inspect a local repository. The command will display the state of the directory against which changes are staged and which files are not tracked by git. The status will inform you which files are staged, unstaged and untracked.
Staged files are files that have been added to the current file history and are to be pushed into the remote repository in GitHub on the next 'push' command (more on 'push' later)
Unstaged files are files that are currently tracked by Git, have been updated, but are not included in the next 'push' command
Untracked files are brand-new files that are not acknowledged by Git at any point.

git add .

git add . is the first instruction that tells Git to save a snapshot of the current project state into the git commit history.
When you later review all the updates to the source code, you will be able to review against each state of the source code each time git add . was called.
The " ." that follows the keyword 'add' instructs git to add all updates within the directory and sub-directories for the local repository. It is possible to replace " ." with specific files for fine-grained updates.
Once git add . is called, the files are moved to the staging area and are ready to be committed to the remote repository.

git add . will ensure all untracked files become tracked and all unstaged changed become staged.

git commit -m "my first commit"

git commit -m"my first commit" is used to create a new revision of the source code that includes the previous version and all the files staged from the previous call(s) to 'git add'.
Each commit command must be made with a comment. -m "my first commit" is the command instruction to include the comment in the speech marks. It is best practice to include a comment that describes the update made, and the problem that the commit is resolving.

git push -u origin <git url.git>

git push will take the current staged commit and push it into the remote repository. The above command includes three further instructions that are required for the very first commit to the repository.
-u instructs git that it is making an upstream push into the origin. 'origin' is the default name for the remote repository and the url for the origin is passed in as a parameter.

Why we branch code

Branching creates a controlled copy of the code to be modified within a safe environment. Branches are used to make granular and specific updates to the code without the risk of creating side-effects or regression to the existing source code. Branches are created and merged into the main source code in a series of events that allow code updates to be reviewed and combined safely.

Branching to make changes

Before we can create a branch from the main remote repository, we need to create a copy of the source code in our personal remote repository. If you own the repository, you can skip this step, however if you are contributing to open source software, the source code will exist in a remote repository that is not your own. Therefore you must create a copy of the remote source code into your own remote location that is linked to the main remote repository. To perform the remote copy, we use git fork.
Forking is the first part of 4 steps to update an existing source code:
1. (If you do not own the central repository) Fork the repository from the remote central location to the remote personal location.
2. Clone the forked repository from the remote personal location to the local location.
3. Apply a fix or update.
4. Submit a pull request for the update to be included in the remote central location.

I will first create a fork of the repository by selecting "Fork" on the GitHub repository page.

The Forked repository will now exist in your own GitHub remote account. Select the repository and copy the URL that appears under the "Code" tab.

In the command line, navigate to the directory where you would like to store the cloned repository and use the following command to create a local copy of the forked repository:

git clone <forked repository URL>

The local repository now includes a copy of the personal remote repository. Use the following command to create a new branch on the local repository:

git checkout -b "firstBranch"

You can see all the available branches by running the following command:

git branch -a

A new branch has now been created called firstBranch.

In the command line, we are able to check the branch we are currently on with 'git status'

on branch firstBranch

The main branch can be accessed by using the following command:

git checkout main

The 'git status' command will now read as "on branch main"

We will navigate back to the firstBranch and apply a single update to the source code on line 6:

public class Application {

    public static void main(String[] args) {

        System.out.println("Hello from the main branch");
        System.out.println("Hello from the first branch");
    }
}

The first branch is updated using the following commands:
• git add .
• git commit -m "first branch print line"

The first branch now has an update that we would like to add to the main branch. You can see the differences between the two branches (main and firstBranch) by using the following command to switch between branches:

git checkout main

git checkout firstBranch

The update from firstBranch can be submitted into the main branch through a pull request:

git push --set-upstream origin firstBranch

The above command uses 3 instructions on the push statement. firstBranch has not specified where it is pushing the latest update to. The upstream target is set to the origin (main) branch and the branch name is specified at the end.

Handling a pull request

The upstream push from the firstBranch has created a new pull request in the main branch. The pull request can be reviewed in GitHub from the main repository page:

The following page allows you to submit the pull request with comments to the maintainers of the main repository. A maintainers of a repository are dedicated individuals who review and approve updates to the main application source code.

If you are the owner or maintainer of the repository, the pull request merge option will be seen as below after any conflicts are reviewed:

Updating branches with the latest change

The main branch from the application has now been updated with the additional print line from the firstBranch. The local copy of the main branch has not reflected the update and only contains a single print line. The latest update to the remote repository can be "pulled" into the local repository using the following command:

git pull

git pull performs two actions:
First, git pull performs a git fetch command to download the latest commits, file, and references from the remote repository locally. Second, git pull will integrate the new data into the working files. The local repository now reflects the latest updates to the source code that is stored remotely.

Conclusion

In this post I have used Git version control to clone a remote repository locally, to create a branch for updates, and to submit a pull request for review and inclusion in the main application source code. The updates to the remote repository are then reflected locally by pulling them from the remote repository using Git.

Git is a popular version control tool used across industries in different forms (such as svn or mercurial). The flows of forking a repository, pulling changes from the remote version and submitting pull requests are important to understand and apply appropriately to standardise and formalise the process for code enhancements and bug fixes.

Working with version control is important within a team and open-source environment; use the techniques from this blog to develop patterns of pulling, updating, and submitting code updates in a safe manner across branches and pull requests in open or even your own closed repositories.

Creating multi-threaded visibility in Java

Mon, 20 Jul 2020 16:34:00 GMT

Key Takeaways
• Understand how a race condition can arise in a multi-threaded application.
• Use Atomic values to create thread-safe variables.
• Use Synchronized blocks to control the accessibility of code blocks to different threads.

What is a race condition in multi-threading?

The race condition is a term that originates from electronics to describe a condition where a systems behaviour is dependent on the timing of uncontrollable events.

In the context of a multi-threaded application, race condition can arise where multiple threads are performing simultaneously upon a stateful session. Stateful sessions contain data that is represented by the current state of the application; the existence of the session variables are dependent on the previous execution of other methods. A stateless session will therefore encounter data that is not impacted by the state or execution of the application.

A static variable is an example of a stateful object. The static variable's existence is dependent on the interactions it has with other instances of the class, whereas a method-local variable is stateless and does not carry such a relationship.

Multi-threading on a shared bank account

To describe the how a race condition may arise, I will use a scenario of a shared bank account. The bank account is not allowed to go into overdraft, therefore when a request is made, it should only be processed if there are sufficient funds in the account.

Each transaction will first check the balance and make a withdrawal if the balance exceeds, or is equal to the withdrawal amount.

A race condition can occur when multiple withdrawal requests are made at a similar time to the bank account using different threads.
Each thread will read the bank balance as containing a sufficient amount, however when both transactions are made, the bank account becomes overdrawn.

The diagram above illustrates how both threads record the balance and make the withdrawal against the balance.
The race condition occurs as the outcome of the transaction is deterministic upon the thread speed and timing between each other.

The possible stateful outcomes

The outcome from the bank account scenario with two threads can be any of following three:
1. Thread 1 can spend the money and Thread 2 is not able to.
2. Thread 2 can spend the money and Thread 1 is not able to.
3. Thread 1 and Thread 2 can spend the money, thus causing the account to become overdrawn.

The race condition is implemented below with a BankAccount class with a withdraw method.
The below code sample enables a race condition to occur:

public class BankAccount {

    private int balance = 50;

    public  void withdraw(int spend){

        int temp = balance;
        System.out.println(Thread.currentThread().getName() + " withdrawing from balance: " + balance);
        balance = temp - spend;
        System.out.println(Thread.currentThread().getName() + " has withdrawn " + spend + ". New balance: " + balance);

    }

    public int getBalance(){
        return this.balance;
    }


}

The BankAccount instance has an initial balance of 50. The withdraw method will first pass the balance into a variable temp before making the withdrawal and printing the balance to the console.

If I were to run the following single-threaded test, the application would perform as expected:

  @Test
    public void bankAccountWithdraws() throws InterruptedException{

        for(int i = 0; i< 5; i++){
            bankAccount.withdraw(10);
        }

        assertEquals(0, bankAccount.getBalance());
    }

main withdrawing from balance: 50
main has withdrawn 10. New balance: 40
main withdrawing from balance: 40
main has withdrawn 10. New balance: 30
main withdrawing from balance: 30
main has withdrawn 10. New balance: 20
main withdrawing from balance: 20
main has withdrawn 10. New balance: 10
main withdrawing from balance: 10
main has withdrawn 10. New balance: 0

Introducing latency and a possible race condition

The current execution of the single thread behaves as expected as the withdraw() method is only called once at any one time. Latency can be introduced into the application by using the wait() method which causes all threads to stop before continuing. The withdraw() method must be marked within a synchronized block to ensure only one thread accesses the method at a single time.

The following update to the withdraw() method will cause the threads to check the balance, wait for each other, then to complete the transaction. As a result, each thread will read in the opening balance before making a transaction against the balance:

   public synchronized void withdraw(int spend) throws InterruptedException{

        int temp = balance;
        System.out.println(Thread.currentThread().getName() + " withdrawing from balance: " + balance);
        wait(100);
        if(balance - spend >=0){
            balance = temp - spend;
            System.out.println(Thread.currentThread().getName() + " has withdrawn " + spend + ". New balance: " + balance);
        }
    }

The single threaded test will continue to behave as expected, however multiple threads can be used to execute the code by using an Executor Service:

  @Test
    public void counterWithConcurrency() throws InterruptedException{

        int numberOfThreads = 5;
        ExecutorService es = Executors.newFixedThreadPool(10);
        CountDownLatch latch = new CountDownLatch(numberOfThreads);

        for(int i = 0; i<numberOfThreads; i++){
           es.execute(() -> {

               try {
                   bankAccount.withdraw(10);
               } catch (InterruptedException e) {
                   e.printStackTrace();
               }

               latch.countDown();
           });
        }
        es.shutdown();
        latch.await();
        assertEquals(0, bankAccount.getBalance());
    }

pool-1-thread-1 withdrawing from balance: 50
pool-1-thread-2 withdrawing from balance: 50
pool-1-thread-3 withdrawing from balance: 50
pool-1-thread-4 withdrawing from balance: 50
pool-1-thread-5 withdrawing from balance: 50
pool-1-thread-1 has withdrawn 10. New balance: 40
pool-1-thread-2 has withdrawn 10. New balance: 30
pool-1-thread-3 has withdrawn 10. New balance: 20
pool-1-thread-4 has withdrawn 10. New balance: 10
pool-1-thread-5 has withdrawn 10. New balance: 0

Each thread reads in the balance as 50. Each thread will then assume the balance remains at 50 and makes the withdrawal of 10.
The class appears to be thread-safe as the balance is reduced as expected, however if we introduce a 6th thread on line 4 of the test, the account becomes overdrawn:

pool-1-thread-1 withdrawing from balance: 50
pool-1-thread-2 withdrawing from balance: 50
pool-1-thread-3 withdrawing from balance: 50
pool-1-thread-4 withdrawing from balance: 50
pool-1-thread-5 withdrawing from balance: 50
pool-1-thread-6 withdrawing from balance: 50
pool-1-thread-1 has withdrawn 10. New balance: 40
pool-1-thread-3 has withdrawn 10. New balance: 30
pool-1-thread-2 has withdrawn 10. New balance: 20
pool-1-thread-4 has withdrawn 10. New balance: 10
pool-1-thread-5 has withdrawn 10. New balance: 0
pool-1-thread-6 has withdrawn 10. New balance: -10

Resolving the race condition

We can resolve the race condition by asserting that only one thread can access the withdraw() method at any one time, or by ensuring all threads are aware of the balance as it is updated. In our BankAccount class so far, we have already utilised one of the keywords used to create thread-safety: synchronized.

Synchronized methods
The Synchronized keyword in Java can be applied to either a method signature or a block of code to ensure that only one thread can access it at a single time. The withdraw() signature is changed to public void synchronized withdraw() to ensure only one thread can obtain the lock for the method at a single time. When the thread reaches the wait() method, it releases the lock allowing another thread to enter the method. The use of the synchronized keyword is not completely effective as it only allows one thread at a time to read the balance. The withdrawal of the fund still remains as a separate action that can be performed by multiple threads at a time.

The problem of the overdrawn account occurs in the following steps:
1. The first thread withdraws the funds.
2. The if statement for all other threads are evaluated to the balance that was read before thread 1 makes a withdrawal.
3. The bank will become overdrawn as the balance from each thread is not updated with the withdrawal from thread 1.

Thread communication and visibility

The withdraw() method may be able to stop multiple threads from accessing the bank balance together, however the wait() method enables withdrawals of the balance to be made by all threads together.. To resolve the remaining race condition problem, all Thread instances must be able to see the updates that are made to the balance as each withdrawal is made by other threads. If the threads are able to see the latest balance update, they will correctly determine if a withdrawal can be made without becoming overdrawn.

Atomic variables

The atomic variables within Java aim to resolve the visibility problem that can occur between multiple threads of an application. The table below illustrates how the threads will obtain the value and perform the operation:

Atomic variables within Java provide variables that are updated across all threads of the application at the same time. An atomic value in this context is useful to ensure all the threads can see the latest balance before making a transaction. This way, the if statement will be applied to the most up-to-date value of the balance. The BankAccount is updated to use AtomicInteger accordingly:

public class BankAccount {

    private volatile AtomicInteger balance = new AtomicInteger(50);

    public synchronized void withdraw(int spend) throws InterruptedException{

        System.out.println("Withdrawing from balance: " + balance);
        wait(100);
        if(balance.get() - spend >= 0) {
            balance.addAndGet(-spend);
            System.out.println(Thread.currentThread().getName() +  " has withdrawn " + spend + ". New balance: " + balance);
        }
        else{
            System.out.println(Thread.currentThread().getName() + " is unable to  withdraw " + spend + " from a balance of " + balance);
        }

    }

    public AtomicInteger getBalance(){
        return this.balance;
    }

}

The shared cache between the threads means the update to balance that is made is visible across all threads. As a result, the second thread is able to see the updated balance value and will evaluate the if statement appropriately. As a result, stage 4 will never be executed as the balance is to become overdrawn.

The AtomicInteger is used to communicate across the threads. When the penultimate thread is executed, the balance is atomically updated to 0 across all threads. Therefore the final thread evaluates the bank account balance to 0 and is no longer able to make the final -10 transaction.

pool-1-thread-1 withdrawing from balance: 50
pool-1-thread-2 withdrawing from balance: 50
pool-1-thread-3 withdrawing from balance: 50
pool-1-thread-4 withdrawing from balance: 50
pool-1-thread-5 withdrawing from balance: 50
pool-1-thread-6 withdrawing from balance: 50
pool-1-thread-2 has withdrawn 10. New balance: 40
pool-1-thread-4 has withdrawn 10. New balance: 30
pool-1-thread-5 has withdrawn 10. New balance: 20
pool-1-thread-6 has withdrawn 10. New balance: 10
pool-1-thread-3 has withdrawn 10. New balance: 0
pool-1-thread-1 is unable to  withdraw 10 from a balance of 0

Conclusion

Multi-threading allows our code to utilise the multiple cores of the CPU. The danger of multi-threading however leads to potential errors of visibility and synchronisation between the threads, one of which is known as race condition.
The Synchronized keyword in Java allows a thread to obtain a key to a code block or method during its execution to ensure no other thread is able to access the code at the same time. The concurrent.atomic package in Java includes different Atomic variables that provide cross-thread visibility to a shared value.

The Atomic package and Synchronized keyword are useful tools that assert an element of determinism about a multi-threaded application. These tools enable execution order to be achieved and cached memory to be shared between threads to reduce non-deterministic behaviour within their domains.

The Spring architecture layers

Mon, 13 Jul 2020 16:34:00 GMT

Key Takeaways
• Compare the properties of the different layers within a Spring application.
• Review the importance of the Spring IoC container, BeanFactory, and ApplicationContext.
• Understand the differences between the beans created from @Component, @Controller, @Service or @Repository.

Spring application architecture

The architecture of a Spring application may reflect upon the intent and organisation of packages and classes.
The Separation of Concern (SoC) design principle aims to create a modular architecture where delegated classes address well-defined functions of the application. SoC supports the long-term maintenance and growth of the application and discourages code rot.

The Spring framework architecture uses 3 layers to offer a degree to separate concerns between classes based upon their responsibility within the application.
Each layer is placed to communicate with the layer above and/or below it. The layers include:

• API/Web layer. Used to accept and authenticate user input and handle HTTP requests. Spring MVC is used to separate the web view, the servlet controller and the model objects.
• Service layer. Interacts with the API layer and model objects to provide authorization and custom business logic processing.
• Data access/integration layer, also known as Repository layer. The basic CRUD operations are exposed in the data access layer to send operations to the database. Object to relational mapping (ORM) is used to map data to and from the database.

Separating layers into a request and response

The diagram below illustrates the layers of a Spring application and how it would handle a request made by a user.

A user request is first intercepted by the API layer. The API layer is used to handle HTTP requests such as GET, PUT, DELETE and POST.

The request can then be processed by the service layer to implement business logic. Such logic may include checking authorisation, transforming the request or logging.

The data access/integration layer transforms the request using ORM to interact with the database. The database is persisted outside the layer and will return a response that is then relayed back to the user.

Spring Inversion of Control Container

The Spring IoC Container is a core Spring Framework component responsible for creating, wiring, managing and configuring objects to be used by the application. The components enable Spring to perform dependency injection as they are wired into classes that require them.

The objects that are configured by the IoC container are known as Spring beans. Each bean contains metadata that includes information for how to create the bean, the bean's dependencies, and the lifecycle details.

There are two types of containers within the Spring Framework:
• BeanFactory.
• ApplicationContext.

BeanFactory
The BeanFactory is responsible for maintaining the registry of different beans and their dependencies.
Beans are injected into the classes that depend upon them with lazy loading and is therefore only used when memory consumption is critical.

ApplicationContext
The ApplicationContext container can perform all the functions of the BeanFactory with additional functionality:
• Load file resources.
• Publish events to event listeners.
• Resolve messages for internationalisation support.
• Support for Annotation-based dependency injection.
• Integration with Spring AOP.

Unlike the lazy-loading of bean from the BeanFactory, the ApplicationContext will load all beans upon instantiation and is therefore considered heavy-weight in comparison to BeanFactory.

Spring class annotations

The Spring class-level annotations are used to register beans into the Spring IoC container. Class-level annotations are detected by the ApplicationContext container during a classpath scan of the application.
The following annotations are applied at the class level according to layer they are responsible within. Each annotation except for @Service contains unique properties to the bean that support their function at the layer.

@Component

The @Component annotation is the parent annotation to the other layer-level annotations @Controller, @Service and @Repository.
Classes marked with @Component will be automatically detected by the Spring framework and added to the ApplicationContext.
The @Controller, @Service, and @Repository annotations all use @Component in their definition, therefore their use also results in the class being added into the ApplicationContext.

@Controller

@Controller is a specialised version of @Component to be used for classes at the API/web layer. When a class is marked with @Controller, the ApplicationContext will register the class as a bean, and the class will be looked upon differently for web-specific tools and services.
When a client request is made to the Spring Application, the dispatcher servlet, which is responsible for forwarding requests to the MVC controllers, will scan all classes marked as @Controller to find the appropriate @RequestMapping annotation to handle the request.

@Service

@Service is a wrapper version of @Component and is used to mark the class as a service. The service class will provide business logic functionality to process requests before passing them to the data access layer for persistence.

The @Service annotation does have include further behaviour that is different to @Component, however service classes should be marked with @Service to mark the intent of the class within the application.

@Repository

The @Repository annotation differs from @Component for the single purpose of catching persistence-specific exceptions.
The @Repository annotation will import the ORM beans into the ApplicationContext container and change the unchecked exceptions eligible for translation into a DataAccessException. The annotation provides enhanced functionality for exception handling.

Conclusion

The Spring IoC container allows the framework to provide dependency injection across the application.
The Spring framework is constructed of 3 layers that separate the responsibility of the class when handling requests and responses.
The ApplicationContext container is a heavy-weight framework for dependency injection that supports annotation-based injection. The @Component annotation is used to register a bean at the class-level into the ApplicationContext. The Spring framework offers three further annotations that extend from @Component to provide unique properties the bean generated to assist with the responsibility of each layer.

Using a RESTful API in Spring

Mon, 06 Jul 2020 16:34:00 GMT

Key Takeaways
• Understand what an API is and how they work.
• Use Spring to make GET requests to the API.
• Explore how the GET request can be broken down and used within a service.

What is an API?

An "application programming interface", abbreviated to "API" is a software intermediary between applications. I like to think of an API as the waiter between you (the client) and your meal (the data) in a restaurant.
APIs allow applications to speak to each other. When an application request is made, it will send the data message to the server where it is interpreted and a response is returned. The application will then interpret the response and display the output to the user.
APIs are all around us in the applications we use; they provide new bounds of opportunities for the information an application can retrieve and interact with.

RESTful APIs and CRUD

The term "REST" in computer systems stands for "Representational state transfer". REST defines a standard to make it easier for computer systems to communicate over the internet. REST-compliant systems are stateless and separate of concerns between client and server.
A RESTful API uses HTTP requests to access and modify the data. As a result, RESTful APIs expose data to the be consumed by our applications.

CRUD operations are the four basic functions of persistent storage. RESTful APIs allow CRUD operations to be performed:

• CREATE: A new data entry can be made with the POST HTTP request.
• READ: Data can be read with a GET HTTP request.
• UPDATE: Data can be updated with a PUT HTTP request.
• DELETE: Data can be deleted with a DELETE HTTP request.

In the following example, I will send a GET request using the Spring framework to access a RESTful API service.

Performing a GET Request with Spring

In this example, I will be using the free RESTful API service by jsonplaceholder.typicode.com to obtain the some sample data of a to do list.

Setting up Spring

The Spring RestTemplate class offers template methods for making HTTP requests of RESTful API services. The RestTemplate class is obtained through the spring-boot-starter-web dependency in a Spring boot application:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

In the main application class, I will create an instance of the RestTemplate in a bean to be later injected:

@SpringBootApplication
public class ToDoGetDemo {
	
	public static void main(String[] args) {
		SpringApplication.run(ToDoGetDemo.class, args);
	}

	@Bean
	public RestTemplate getRestTemplate() {
		return new RestTemplate();
	}
}

The RestTemplate is injected into a class that contains a RestController. The API is then called upon with a GET Mapping:

@RestController
public class APIController {

	@Autowired
	private RestTemplate restTemplate;
	
	@GetMapping("/todo-items")
	public String getPrices(){
		String url = "https://jsonplaceholder.typicode.com/todos";
		return this.restTemplate.getForObject(url, String.class);
	
	}
}

There are quite a few points to take from the class above, so I will break it down line by line:
On line 1, the @RestController annotation is used to mark the class as a request handler. The annotation will be responsible for mapping request data to the defined request handler method.
On line 4 and 5, the RestTemplate is injected into the class. The RestTemplate will be used as a service for making the GET request.
On line 7, the @GetMapping annotation is used to map the request to the "/todo-items" handler. GetMapping was introduced in Spring 4.3 as an amalgamation of "@RequestMapping" and "method=RequestMethod.GET".
On line 9, the URL for the RESTful API is defined. The URL can be tested with an API development tool such as Postman to validate the information we expect to see.
Line 10 uses the RestTemplate instance by passing in the url and mapping the response to an instance of the String class.

Once the class is built, the Spring application can be run and the output from the API can be found on localhost:8080/todo-items. The output will appear as a large String in the browser:

Converting JSON to a POJO

The JSON API content can be converted into a POJO (plain old java object) to allow us to navigate and use the data. Rather than passing the GET request into an item of the String class, we will create our own class to store the values within. The class will then be held in an array of instances.

A new class, "ToDo" is created to store the fields of the API call according to their content type from the request:

public class ToDo{

	private int userId;
    private int id;
    private String title;
    private boolean completed;

    //getters and setters...
}

The RestController is updated to reference an array of the ToDo class as the response mapping to a new handler "todo-objects":

@GetMapping("/todo-objects")
	public ToDo[] getPostAsObject() {
		String url = "https://jsonplaceholder.typicode.com/todos";
		return this.restTemplate.getForObject(url, ToDo[].class);
		
	}

When the "/todo-objects" request is made in the browser, the output is in a JSON format:

Custom deserialisation with Jackson

The current GET request is deserialised by our application into a POJO. One way we may want to refactor the output of the request could be to make the "completed" boolean into a descriptive String.
The JsonDeserializer class from the jackson.databind package enables custom deserialisation of the API response to transform the data.

The custom deserialisation will take place in a new class "ToDoDeserialiser" which extends the JsonDeserialiser class.
The ToDo class will make use of the custom deserialisation class with the @JsonDeserialize annotation where the ToDoDeserialiser is referenced (line 1):

@JsonDeserialize(using = ToDoDeserialiser.class)
public class ToDo{

	private int userId;
    private int id;
    private String title;
    private String completed;

The "completed" field is marked as a String to reflect the new data type we wish to return. Ensure the getter and setter fields, along with the argument-constructor is updated to reflect the change in type.

The ToDoDeserialiser class overrides the deserialize method of the JsonDeserializer class to provide custom logic for deserialisation:

public class ToDoDeserialiser extends JsonDeserializer {

    @Override
    public Object deserialize(JsonParser jsonParser,
				 DeserializationContext deserializationContext) throws IOException {

        JsonNode node = jsonParser.getCodec().readTree(jsonParser);
        int userId = node.get("userId").intValue();
        int id = node.get("id").intValue();
        String title = node.get("title").textValue();
        boolean completed = node.get("completed").asBoolean();
        String isCompleted = completed ? "Is Completed" : "Not Completed";
        return new ToDo(userId, id, title, isCompleted);
    }
    
}

The deserialize method first obtains a new JsonNode node on line 7 from the JsonParser. The JsonParser provides read-only access to the JSON data. Each JsonNode represents an instance of ToDo in the request.
Lines 8 to 11 obtains the values from the node which are later passed into a constructor for the ToDo item that is returned.
Line 12 will convert the boolean "completed" into a descriptive String as per custom logic.
The new String conversion is returned in the ToDo constructor and as a result, the GET mapping translates into a new JSON output where the "completed" value is a String:

Conclusion

RESTful APIs provide a bridge that enables services to interact with other services in a standardised manner. Spring frameworks can be used to interact with RESTful API services that can expand the capabilities of an application. Building microservices that provide RESTful endpoints can also enable you to share data between services.
The JSON format of data can be passed into a POJO that can be used to resolve business logic. The JsonDeserializer class can also be extended to provide custom modification and processing of data from the REST endpoint.
The source code from this blog can be found on GitHub here.

Using dependency injection with Spring

Mon, 29 Jun 2020 16:34:00 GMT

Key Takeaways
• Understand the importance of dependency injection and why it supports modular code design.
• Implement dependency injection with three different techniques.
• Review the benefits and drawbacks to each dependency injection technique.

What is dependency injection?

Dependency injection is a technique used to decouple classes that depend upon each other. Class dependency arises when once class uses the functionality (method) of another class 'as a service'.
Classes are deemed to have a hard dependency if their invocation creates a 'new' instance of another class. Dependency injection falls under the broader design of 'inversion of control'. Inversion of control enforces the fifth principle of SOLID whereby a client's implementation is transferred from the class itself to a framework. In the example below, the client, Student has a hard dependency on the SchoolService. The SchoolService is used to return a List of classes the student would have within the learn() method.

    public class Student{

        SchoolService schoolService;

        public Student(){
            schoolService = new ArtSchool();
        }

        public List<String> learn(){
            
            return schoolService.getClasses();

        }
    }

The Student Object will create the ArtSchool at compile time, however a problem may arise if the we want the Student to instead attend the LiteratureSchool. The class will need to change its dependency implementation accordingly:

public Student(){
    schoolService = new LiteratureSchool();
}

Why use dependency injection?

By using dependency injection, the application will delegate the responsibility of creating Objects away from the individual classes, and into a single framework.
The task execution is decoupled from implementation and a further degree of modularity is achieved.
Further benefits can be seen during unit-testing where components can be mocked and passed with dependency injection for more concise and light-weight tests.

Implementing dependency injection

Dependency injection design can be implemented by refactoring the Student class to reference a pre-existing instance of the SchoolService.

Dependency injection can be achieved in one of many way, including, but not limited to:
• Constructor injection: using the client constructor to take in the service as an argument.
• Spring constructor injection: using a Spring bean to inject the bean as a dependency.
• Spring setter injection: using a setter method within a Spring bean to instantiate the dependency.

Constructor dependency injection

A constructor parameter can take a dependency as an argument to be used by the class. Rather than the Student class defining the type of SchoolService it depends upon, the responsibility is delegated to the class that instantiates the Student instance:

public class Student{

        private SchoolService schoolService;

        public Student(SchoolService schoolServiceArgument){
            this.schoolService = schoolServiceArgument;
        }
    }

From the main method, the SchoolService can be passed in as an argument:

public class Main{

    public static void main(String[] args){

        SchoolService myService = new ArtSchool();
        Student myStudent = new Student(myService);

        myStudent.learn();
    }  
}

Spring Constructor dependency injection

The Spring framework offers annotations that can inject dependencies into the class.
Before using Spring for dependency injection, it is important to ensure the annotation-driven injection is enabled by creating a configuration class:

@Configuration
public class Config {

	@Bean
	public SchoolService schoolService() {
		return new ArtSchool();
	}

    @Bean
	public Student student() {		
		return new Student(schoolService());
	}
	
}

The @Configuration annotation indicates that the class declares bean definitions. The class is therefore processed by the Spring IoC container at runtime to create the beans. The Bean schoolService returns a new instance of ArtSchool. In the Student bean, the SchoolService is injected with the bean definition of ArtSchool. The Spring framework is now responsible for the schoolService injection, and not the Student class itself.

	public class Student {

	private SchoolService schoolService;

	public Student(SchoolService schoolService) {

		this.schoolService = schoolService;

	}
    ...

The @Autowired annotation

The @Autowired annotation is used in Spring to implicitly inject dependencies into a class. It is not necessary to use the @Autowired annotation if the Student class only defines a single constructor. If more than one constructor is defined, @Autowired must be used to instruct the Spring container to inject the dependency into the class.

Spring setter dependency injection

Rather than using the constructor of the Student class to instantiate the schoolService, Spring bean definitions can call a setter method when defining the bean. In the Configuration class, the Student bean is updated to create a Student instance, to call the setter method, and to return the Student.

	
	@Bean
	public Student student() {
		Student student = new Student();
		student.setSchoolService(schoolService());
		
		return student;
	}

The Student class must now define the new no-argument constructor. If the Bean for the Student class was not defined, the dependency injection will no longer work with two constructors. Instead, the @Autowired annotation is necessary to ensure the schoolService is injected with the Bean when a Student is instantiated:

public class Student {

    @Autowired
	private SchoolService schoolService;

	public Student(SchoolService schoolService) {
				
		this.schoolService = schoolService;

	}
	
	public Student() {
		
	}

Choosing a dependency injection technique

The constructor and setter dependency injection achieve the same objective in slightly different ways. The setter method can be seen as more readable against the constructor injection. Setter injections can also resolve a potential ObjectCurrentlyInCreationException where a circular dependency between two Objects arise.

The constructor injection, however, ensures that the Object is not created without the dependency injected. It may still be possible to create a Student class with an incomplete dependency if relying upon the setter injection. The constructor is also not liable to being overridden by child classes, while the setter method can be, introducing an implementation security concern.

Conclusion

Dependency injection is technique used to enable Objects and the services they depend upon to be created as separate entities.
This blog has reviewed three of the most common ways of implementing dependency injection, however there are other annotations such as @Resource and @Inject that can achieve similar outcomes and are worth exploring in Spring.

When using test-driven-development, or any kind of unit test, dependency injection can greatly reduce the weight and complexity of unit tests through mocking frameworks. As well as improving code readability and reducing boilerplate code, dependency injection shifts the implementation of an application away from individual classes, and into a modular framework.
The code sample from this blog can be found on GitHub here.

Understanding Floating point precision

Mon, 22 Jun 2020 16:34:00 GMT

Key Takeaways
• Understand why floating point precision does not always created expected results.
• Understand how the float and double primitives are created in the JVM.
• Use the math.BigDecimal class to resolve floating point precision inaccuracies.

Floating point primitives

The primitive data types float and double offer floating point values to be used for non-whole numbers. The JVM allocates floating point numbers a fixed size of memory: float consumes 4 bytes (32 bits), whereas double consumes 8 bytes (64 bits).
The benefits of using floating point numbers are their speed and efficiency. The JVM is able to very quickly calculate and store the values.

Binary fractions and floating point inaccuracies

The JVM stores floating points values as a binary representation of a fraction and exponent; floating point values are calculated using a Base 2 notation.
In calculus, the value of 0.1 can be calculated as 10^-1. Similarly, the value of 0.55 can be calculated as 55 * 10^-2.
The JVM calculates floating point numbers to the Base of 2. The Base 2 calculation is optimized within computers as their binary representation of 0's and 1's can be efficiently computed. The JVM is able to interpret the values without conversion and therefore enable floating point numbers to realise a benefit of efficiency with computational cost.

The example below demonstrates how the value of 0.01 can be defined in binary code.
The base of 2 is sequentially reduced by 1 from -1. The binary value indicates whether or not the value of Base 2 can increment the accumulative value towards 0.01.

The binary code of 0001100 will continue infinitely as 0's and 1's are added in pairs. The float and double primitive values will however round the value once their memory limit is reached. Some double values can be represented by a binary fraction within a finite length of the primitive data type and therefore does not require rounding. One example may be 0.75, it would be represented as 2^-1 + 2^-2 = 0.5 + 0.25 = 0.75.
As float and double primitives are not able to define the exact value of the decimal, they create unexpected results when applied to operations:

    double a = 0.2;
    double b = 0.19;

    float c = 0.2f;
    float d = 0.19f;

    System.out.println("double answer:  " + (a-b));
    System.out.println("float answer:   " + (c-d));

    double answer:  0.010000000000000009
    float answer:   0.010000005

Resolving floating point precision with math.BigDecimal

The Java math package introduces the BigDecimal class to enable control over the scale and precision of decimal numbers. The following example demonstrates how BigDecimal can resolve the precision inaccuracies of floating point values:

    BigDecimal e = new BigDecimal("0.2");
    BigDecimal f = new BigDecimal("0.19");  
    System.out.println("Big decimal answer: " + e.subtract(f));

    Big decimal answer: 0.01

How BigDecimal works

With the example of 1.23, scale refers to the number of decimal places with a base of 10 (2), and precision refers to the length of arbitrary numbers (3). BigDecimal uses the unscaled value (123) and the scale (10^-2) to create the precision value 1.23
As BigDecimal employs Base 10 calculations for the decimals, it can precisely represent all 2 decimal placed numbers. BigDecimal is typically used within applications that compute with currencies to ensure they are correctly represented.

Constructors

The BigDecimal constructor can take arguments such as BigInteger, String, int, double, float, and char to convert into a BigDecimal. When supplying a BigInteger, the constructor can also take a scale argument to build the decimal:

    int a1 = 1;
    char a2 = 'a';
    double a3 = 2.4;
    String a4 = "2.4";
    BigInteger a5 = new BigInteger("1234");

    System.out.println("BigDecimal (int): " + new BigDecimal(a1));
    System.out.println("BigDecimal (char): " + new BigDecimal(a2));
    System.out.println("BigDecimal (double): " + new BigDecimal(a3));
    System.out.println("BigDecimal (String): " + new BigDecimal(a4));

    System.out.println("BigDecimal (BigInteger): " + new BigDecimal(a5));
    System.out.println("BigDecimal (BigInteger, scale): " + new BigDecimal(a5, 2));

BigDecimal (int):               1
BigDecimal (char):              97
BigDecimal (double):            2.399999999999999911182158029987476766109466552734375
BigDecimal (String):            2.4
BigDecimal (BigInteger):        1234
BigDecimal (BigInteger, scale): 12.34

The example above illustrates several ways a BigDecimal can be instantiated.
The use of double a3 on line 9 prints the binary value of 2.4 as a 32-bit integer. The String constructor on line 10 however prints the expected value of 2.4. Line 12 and 13 demonstrate how BigInteger can be used with a scale operation to build a decimal number.

Good practices: String constructor

The example above demonstrates the successful use of the String constructor with BigDecimal. The String constructor represents good practice as the BigDecimal no longer uses the binary fractions to create the value, and instead employs Base 10.

Methods

The BigDecimal class does not override operators such as + or - used for applying primitive data types to each other. Instead, methods such as add(), subtract(), multiply(), and divide() are implemented for operating with other BigDecimal instances.

BigDecimal Performance

The BigDecimal class introduces precision of decimal values by implementing Base 10 calculations. The final value, derived from the unscaled 32-bit value multiplied by 10^-scale results in a heavy performance cost to the application.
As a result, performance costs from BigDecimal can rise up to 90 times slower than using double. The decision to implement BigDecimal should strongly consider the importance of exact values to the context over speed.

Conclusion

The JVM uses binary functions to compute floating-point primitive values float and double. Their design enables the values to be efficiently calculated and stored.
The Base 2 function for deriving floating point values introduces a degree of precision loss from rounding. BigDecimal is a class within the math package of Java and enables decimal values to be computed against a Base 10. As a result, the decimal value created does not contain an unexpected rounded value.
BigDecimal is a useful class implemented with applications that handle currencies. BigDecimal however introduces performance costs that are to be considered with their use.

Exploring logarithm in computer science

Mon, 15 Jun 2020 16:34:00 GMT

Key Takeaways
• Understand why logarithms are used to calculate the time complexity for certain operations and algorithms.
• Explore how binary trees are designed and used for searching.
• Understand how sorting algorithms may use binary trees.

Logarithms in computer science

This post will explore why logarithms might appear within our calculation of time complexity. The post will then begin to explore the context for why the base of a logarithm is implied as 2 within software data structures. If you are unfamiliar with time complexities, please visit my blog where I review their importance to algorithms here.

Logarithms apply to two different rates of growth within time complexity:
• log (n)
• n log (n)

While there is no distinguished pattern for when log (n) and n log (n) appear within data structures and operations, log (n) most frequently appears when using tree data structures and n log (n) most frequently appears when using sorting algorithms. The Big O cheat sheet, here, provides a breakdown for each occurrence.

Within calculus, logarithms are introduced as a means to inverse a function of exponentiation. 3 to the power of 4, also known as 3 exponent 4, is equal to 81. Consequently, logarithm 81 of base 3 is equal to 4.

Within computer science, logarithms are used to understand the depth of a binary tree. The value, n, is the number of elements within the tree, and the base, b, is set to 2. By setting the base to 2, the logarithm of n will calculate the exponent of 2 to reach n. In other words, we are asking the logarithm how many times the value n can be divided by 2 before it is equal to 2. The value we obtain is rounded up to the nearest integer.
But why do we care about the depth of a binary tree?

Searching within a binary tree

Binary trees split data collections into pairs of nodes from the root node. Each node will be further split into 2 until all elements from the collection are included in the tree. The diagram below illustrates a balanced binary tree for the sorted array of: [0, 4, 6, 12, 7, 5, 9, 3]. The value of each node is identified by finding the middle value of the collection from the node above it and the start or end. Nodes that are greater than the node above it are sent to the right, and nodes that are less are sent to the left:

Calculating the Big O of a binary tree search

If we were to search for the value of 6 within the tree, we would traverse a total of 2 levels. Level 0 is the value 5. As 7 is greater than 5, we search down the right side path. The 1st level shows the value 7. As 6 is less than 7, we now look down the left side of the node. Finally, we find the value 6 on the second level.

When using a binary tree to find a value, we are able to identify the maximum number of operations required to find the value once we know the number of elements, n.
In the example above, where n is equal to 8, the maximum number of moves (or levels) that need to be traversed is 3. The Big O of 3 can also be calculated with log (8) with an implied base of 2.

Merge Sort: n log (n)

Merge sort and quicksort are a popular sorting algorithm that use the divide and conquer strategy. Merge sort performs the following actions to sort the array: [12, 7, 3, 9, 11, 10, 4, 2]:

Step 1 (Divide): The array is split into sub arrays until they are individual elements.

Step 2 (Conquer): Each sub array is sequentially sorted and returned to the array above it.

First the elements are sorted into pairs:

The pairs are sorted into collections of 4:

And finally the two collections are returned to the final array at the top:

Calculating the Big O of a merge sort algorithm

The number of operations required to sort a binary tree is defined within step 2 of the merge sort. the total operations is equal to the number of elements that are sorted at each level multiplied by the number of levels within the tree.
The diagram below illustrates the number of operations performed to sort the sub arrays at each level. Each sub-array except [3, 9] is re-sorted, resulting in a total of 22 operations to be performed:

If sub-array [3, 9] required sorting from the 3rd level, a total of 24 operations would be performed. The Big O for the algorithm is therefore equal to 24, but how can the Big O be calculated?
First, we can see that the maximum number of sorting operations that are performed in the tree is 8 at each level. A total of 8 sorting steps are taken at levels 1 and 2, and only 6 at the third level due to [3, 9] remaining sorted.
From the earlier search example, we know that the number of levels in a binary tree is equal to log (n). Therefore the total number of operations = 8 * log (8), also more generally defined as:
n log (n).

Conclusion

Logarithms are used within data structures to understand the time complexity for binary tree operations and sorting algorithms. By implying a base of 2, we are able to understand how many times a value (number of elements) can be divided by 2, and therefore the depth of its binary tree.
Popular sorting algorithms such as QuickSort and Merge sort implement divide and conquer strategies that split the data into a logarithmic number of levels before implementing the sorting phase.
By understanding how logarithms are used within data structures and algorithms, we are better positioned to understand how time complexity can accrue and where it can be optimized.

Collecting a map in the Stream API

Mon, 08 Jun 2020 16:34:00 GMT

Key Takeaways
• Understand the advantages of using the Stream API.
• Transform a Map into another Map using a stream.
• Explore the 3 overloaded toMap() methods of the Collector class.

Introduction to the Stream API

The Stream API creates a pipeline to process and transform data in bulk operations. Stream introduces functional programming capabilities to Java where data is collectively resolved.

Stream operations are categorised as either terminal or intermediate:

• Intermediate operations will listen to elements being passed within the stream and will perform a function on the data. Together, intermediate operations form a pipeline for which the data is processed.
• Terminal operations will invoke the intermediate operations of a stream to return a single result.
All streams must contain a single terminal operation and zero or more intermediate operations.

Stream API features

The Stream API introduces 4 features for processing data in bulk operations:

Short circuiting allows an infinite size data source be processed within finite boundaries. The following methods demonstrate short circuiting behaviour:
• limit(n) is an intermediate operation to define the number (n) of elements to process.
• findFirst and matchAny operations are terminal and return the value once a condition is satisfied.

Map Reduce enables a data source can be transformed and aggregated within the stream:
• map() is an intermediate operation to transform the data into a new stream.
• filter() is an intermediate operation to remove elements after passing them through a boolean condition.
• reduce() is a terminal operation to evaluate all elements of the stream into a single element.

Parallel operations can be obtained through the Stream API without the complexity of defining threads and operations:
• parallel() is an intermediate operation to utilise the multiple cores that are available.

Lazy evaluation describes each stream as intermediate operations are only used when a terminal operation is called. The stream pipeline will accumulate further streams from intermediate operations and traverse through them only if a terminal operation is called.

Obtaining elements from a Map

Each Map instance crated has three key collections that can be derived from the key-value pairs:
• entrySet() returns the key-value pairs as a Set of Map.Entry casted to the key and value types.
• keySet() returns the keys in a Set casted to the key type.
• values() returns the values as a Collection casted to the value type.

Transforming a Map

The Collectors.toMap() method is a terminal operation that will pass the stream into a Map. The toMap() method is overloaded three ways to support different aggregations of the Map. To demonstrate how the toMap() method can be used, the below Map of United States Presidents and their age at inauguration will be used:

    Map<String, Integer> presidentMap = new HashMap<>();
        presidentMap.put("Washington", 57);
        presidentMap.put("Adams", 61);
        presidentMap.put("Jefferson", 57);
        presidentMap.put("Madison", 57);
        presidentMap.put("Monroe", 58);
        presidentMap.put("Q. Adams", 57);
        presidentMap.put("Jackson", 61);
        presidentMap.put("Van Buren", 54);
        presidentMap.put("Harrison", 68);
        presidentMap.put("Tyler", 51);

toMap(Function key, Function value) takes two arguments that will transform the Collection into a new Map. Functions are used to determine the key and values to be added. A lambda expression can be used to obtain the entry set from the Map: entryInstance -> entryInstance.getKey() will obtain the key from each entry. Similarly, entryInstance -> entryInstance.getValue() can be used to obtain the entry value. It is possible to pass the entryInstance into a method that accepts and returns an int to perform business logic. In the below example, a function is used to double the value and assign it into the Map:

    Map<String, Integer> collectedByKey = presidentMap
        .entrySet()
        .stream()
        .collect(Collectors.toMap(entryInstance -> entryInstance.getKey(), entryInstance -> entryInstance.getValue()*2));

Output:

    {
        Madison=114,
        Harrison=136,
        Adams=122,
        Jefferson=114,
        Monroe=116
        Q. Adams=114,
        Van Buren=108,
        Jackson=122,
        Tyler=102,
        Washington=114
    }

toMap(Function key, Function value, BinaryOperator merge) takes a third argument to merge the values of the Map into a new value. The BinaryOperator functional interface will take two arguments of the same type and return the same type back. In the below example, a new Map is returned where the age (Integer) is the key and the president (String) is the value. As more than one president may map to each key, the BinaryOperator will concatenate the presidents as a value.

    Map<Integer,String> accumulateValues = presidentMap
        .entrySet()
        .stream()
        .collect(Collectors.toMap(entry -> entry.getValue(), entry -> entry.getKey(), (presidentOne, presidentTwo) -> presidentOne + ", " + presidentTwo));

Output:

    {
        51=Tyler,
        68=Harrison,
        54=Van Buren,
        57=Madison, Jefferson, Q. Adams, Washington,
        58=Monroe,
        61=Adams, Jackson
    }

toMap(Function key, Function value, BinaryOperator merge, Supplier mapSupplier) takes a fourth argument, a Supplier, to create a new Map. In this instance, the Supplier creates a LinkedHashMap to maintain the values after they are sorted on line 4. Other implementations of the Map Interface can be used such as a HashTable to ensure concurrent access. In the below example, the presidents are sorted based upon their values and placed into a new LinkedHashMap to maintain the insertion order of each entry.

    Map<String,Integer> sortedByValue2 = presidentMap
        .entrySet()
        .stream()
        .sorted(comparingByValue())
        .collect(Collectors.toMap(Map.Entry::getKey,Map.Entry::getValue,(a,b) -> a,LinkedHashMap::new));

Output:

    {
        Tyler=51,
        Van Buren=54,
        Madison=57,
        Jefferson=57,
        Q. Adams=57,
        Washington=57,
        Monroe=58,
        Adams=61,
        Jackson=61,
        Harrison=68
    }

Conclusion

The Stream API introduces a new paradigm of functional programming that Java did not have prior to Java 8. The use of intermediate operations allow data to be aggregated and transformed before being collected into a new data structure.
The Collectors.toMap() method can been overloaded with 3 implementations to provide different levels of Map transformations. The source code from the example in this blog can be found on GitHub here.

Exploring the String Class

Mon, 01 Jun 2020 16:34:00 GMT

Key Takeaways
• String literals are stored within a dedicated area of the heap area.
• String instantiation can determine how it can be equated to other Strings.
• The StringBuffer and StringBuilder classes offer methods that allow a sequence of characters to be handled as a mutable Object.

Introduction to java.lang.String

The String class is used in Java to create an immutable sequence of unicode characters; each String Object is immutable and final, and therefore thread-safe. The following code sample demonstrates how a String can be instantiated with the new operator or as a literal:

    String stringOne = new String("Hello");
    String stringTwo = "World";

The memory allocation of String Objects is determined by how they are instantiated. The new operator will assign the String into the heap memory, however, a literal String will be stored into the String pool.

String pool
The String pool is a dedicated area within the heap memory for storing String Objects. Before a String literal is created, the JVM will scan the String pool to check if the String value already exists; if the String already exists within the String pool, the String literal will point to the String pool Object.
The String pool is demonstrated using the below example:

    String stringOne = new String("Hello");
    String stringTwo = "World";
    String stringThree = "World";

• When stringOne is created, the String Object is created in the heap space.
• When stringTwo is created, the String is assigned to the String pool.
• When stringThree is created, the literal will point the existing String in the String pool.

The immutable nature of a String Object means new Objects are created when the value of String is updated. If stringTwo and stringThree were assigned to different values, the JVM will create two new Objects providing the new values are not found in the String pool. The String pool will retain the memory address of the value "World", however it will remain unreferenced in the pool:

    String stringTwo = "World";
    String stringThree = "World";
    stringTwo = "Hello";
    stringThree = "Gosling";

Interning Strings

intern() is a public method of the String class that is used to assign a String Object into the String pool. All literal instantiations of a String object call the intern() method by default. When a String is created with the new operator, the intern() method is not called and the String is only stored in the heap area. Calling intern() on a String Object that is instantiated with the new operator will provide additional storage of the String into the String pool. The reference to the String object will now point to the String pool. As a result, twoString can be similarly compared using the == operator as opposed to the .equals() method.
The below example demonstrates the use of intern():
• stringOne is created once: in heap memory
• stringTwo is created twice: in the heap and String pool.

    String stringOne = new String("Hello");
    String stringTwo = new String("World").intern();

Comparing Strings

String Objects can be compared with either == or .equals(). == is a reference comparator and will compare the memory locations of the two Strings. .equals() is a content comparator and will compare the value of the two Strings.
By using the intern() method, the String Object will evaluate to true when using == to compare references with a String literal of the same value. While the == method is a more time efficient comparator, the equality of two String Objects is dependent upon their location reference and not their content. As a result, two Strings with the same value may not evaluate to true if one was created in the heap area. The code sample below demonstrates how different String Objects can be compared using their memory address and value:

    String stringOne = "Hello";
    String stringTwo = new String("Hello");
    String stringThree = new String("Hello").intern();

    System.out.println(stringOne == stringTwo);         //false
    System.out.println(stringOne == stringThree);       //true
    System.out.println(stringOne.equals(stringTwo));    //true
    System.out.println(stringOne.equals(stringThree));  //true
    System.out.println(stringTwo == stringThree);       //false

StringBuffer and StringBuilder

The StringBuffer and StringBuilder classes provide support to using Strings as mutable Objects. StringBuffer and StringBuilder Objects are created within the heap area.

The StringBuffer and StringBuilder classes differ by their thread safety. StringBuffer Objects are accessed through synchronized methods and can therefore only be accessed by a single thread at a single time. The StringBuilder Object is not thread-safe and can be accessed simultaneously by different threads. The StringBuilder class is often preferred due to its latency performance over StringBuffer.

Both StringBuffer and StringBuilder Objects provide functions that enable a String to be transformed and accessed in memory:
• append(String) to concatenate String Objects onto the StringBuffer/Builder
• insert(int, String) to insert the specified String within the StringBuffer/Builder at the int position.
• reverse() to reverse a StringBuffer/Builder.
• delete(int, int) to delete the characters from the start and end index defined by the two arguments.
An additional benefit to using the StringBuffer and StringBuilder aligns to their resemblance of the builder pattern to enhance code readability:

    public String createString(stringArg){
    
        StringBuilder sb = new StringBuilder("hello");

        sb.append(stringArg);           //add argument to sb
        sb.insert(3,"__");              //add "__" at third position
        sb.deleteCharAt(sb.length()-1); //remove the final char
        return sb.toString();           //return sb as a String object

    }

Conclusion

The String class is uniquely handled by the JVM with an optimized pipeline and storage. The immutability of a String means new Objects are created when the String value is modified. A String Object can be instantiated into the heap area which effectively alters how the String Object is equated to other String Objects. Lastly, the JVM offers StringBuffer and StringBuilder classes to provide functionality for String Objects as a mutable Object.

Connecting MySQL to a Spring cloud task

Mon, 25 May 2020 16:34:00 GMT

Key Takeaways
• Use Spring Cloud Task to create a microservice that executes an asynchronous task.
• Connect a Spring Cloud Task to a MySQL database to store runtime execution data.
• Review the information that can be obtained by default from the Spring Cloud Task execution.

What is an asynchronous task?

An asynchronous task is a parallel function that is executed separately from the main application thread. Asynchronous tasks are designed to be stateless and have a distinct start and end time. Without asynchronous tasks, microservices may wait upon other microservices to finish executing their task before they resume processing. As a result, application performance can risk a latency build-up due to a single task execution.

Spring Cloud Task enables asynchronous tasks to be performed within a Spring Boot application. One example of an asynchronous task may involve messaging. Rather than a microservice depending on feedback that a message has been processed and executed, the message can be sent to a bus using Spring Cloud Task. As a result, the dependency of message processing is decoupled from the microservice and the bus can be independently scaled to process messages with event-driven computing.

In this post, Spring Cloud Task will be used to execute simple business logic. The runtime execution of the task generates a log of default fields that can be persisted in a MySQL database.

Creating a Spring Cloud Task

A Spring Cloud Task can be created through the Spring initializer. The dependencies of a MySQL Driver and JDBC API are also necessary for connecting with a MySQL Database:
• Spring Cloud Task.
• MySQL Driver.
• JDBC API.

	<dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-task</artifactId>
    </dependency>

    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
    </dependency>
			
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-jdbc</artifactId>
    </dependency>

The main class is annotated with @EnableTask to bootstrap the functionality of Spring Cloud Task. By default, the annotation will equip the microservice with a log of data from the runtime execution, known as the TaskRepository, stored in-memory.

    @SpringBootApplication
    @EnableTask
    public class AdditionApplication {

        public static void main(String[] args) {
            SpringApplication.run(AdditionApplication.class, args);
        }
    }

A sub-class that implements CommandLineRunner is used to execute the business logic.
The CommandLineRunner interface is one of two interfaces (along with ApplicationRunner) that allows methods to be run once a Spring Boot application has started. The CommandLineRunner interface provides access to the arguments passed into the application by overriding the run(String... args) method.
If the microservice has multiple implementations of CommandLineRunner, the classes can be ordered with the @Order annotation.

	public class Concatenator implements CommandLineRunner{

		@Override
		public void run(String... args) throws Exception {

			if(args.length > 0) {
				StringBuilder concatenation = new StringBuilder();
				for(String argument : args) {
					concatenation = concatenation.append(argument);
				}
				System.out.println("Concatenated arguments: " + concatenation);
			}
			else {
				System.out.println("No arguments made to spring task");
			}	
		}		
	}

A bean is added to the main class to ensure the sub-class and the business logic is processed by Spring at runtime.

    @Bean
	public Concatenator createConcatenator() {
		return new Concatenator();
	}

Creating a MySQL database

A database named 'springCloudTask' is created within the MySQL CLI using the following command:

mysql> create database springCloudTask

The database exists without any tables and can be validated with the following commands:

mysql> use springCloudTask
Database changed
mysql> show tables;
Empty set (0.07 sec)

When the Spring Cloud Task is connected to the database, it will automatically create and populate tables if they do not exist.

Connecting the Spring Cloud Task with the MySQL database

Before the Spring Cloud Task can be run, the application.properties file must be updated to reference the newly created database:

spring.application.name=Concatenate Task
logging.level.org.springframework.cloud.task=DEBUG
spring.datasource.url=jdbc:mysql://localhost:3306/springcloudtask?useSSL=FALSE
spring.datasource.username={ENTER USERNAME}
spring.datasource.password={ENTER PASSWORD}

Line 1 will provide a name to the application for debugging.
Line 2 will provide additional debugging into the log.
Line 3 will connect to the database through the port number.
Lines 4 and 5 are to be updated with the appropriate credentials for database connection.

Before the Spring Cloud Task is executed, update the run configurations to include arguments that are to be concatenated by the business logic. For example, pass in the arguments "run my spring boot app".

Task execution

After running the Spring Cloud Task, the following lines are printed amongst many others to the console:

Initializing task schema for mysql database

Creating: TaskExecution{executionId=0, ..., taskName='Concatenate Task', startTime= {...}, endTime=null, exitMessage='null',..., arguments=[run, my, spring, boot, app]}
Concatenated arguments: runmyspringbootapp

The Spring Cloud Task has executed and shown the start and end time of the task along with the arguments that have been processed.
The execution has also created 5 new tables in the database, found through the MySQL CLI:

mysql> show tables;

+---------------------------+
| Tables_in_springcloudtask |
+---------------------------+
| task_execution            |
| task_execution_params     |
| task_lock                 |
| task_seq                  |
| task_task_batch           |
+---------------------------+
5 rows in set (0.16 sec)

The tables show the following information:
• task_execution: key details about when the task runtime such as id, start time, end time, exit code and exit message.
• task_execution_params: log of the parameters passed into the Spring Cloud Task.
• task_lock: stores information when Spring Cloud Task restrictions are applied.
• task_seq: an incremental Id is stored and obtained as a reference for the next execution.
• task_task_batch: links the task execution with the batch execution.

The task_execution and task_execution_params tables are most relevant to this example as they provide details of the runtime execution. The tables can be viewed from the workbench or the CLI:

Task execution table:

Task execution parameters table:

Conclusion

Spring Cloud Tasks enable tasks with a set start and end time to be decoupled within a microservices architecture. With just a few dependencies and lines in the application.properties file, task execution can be logged into a persistent database.
The Spring Cloud Task can be taken further by deploying it into a Maven repository where it can be called upon by other microservices.
The code sample from this blog can be found on GitHub here.

Implementing chaining with a hash table

Mon, 18 May 2020 16:34:00 GMT

Key Takeaways
• Write functions to enable a hash table to resolve collisions with chaining.
• Write a rehash function to recreate a hash table with a smaller load factor.
• Use variables such as load factor, hash function and buckets to influence hash table performance.

Chaining in a hash table

Hash collisions occur within a hash table or hash map when multiple keys have the same output from a hash function. Chaining, also known as closed addressing, is a technique used to store the value of a key when a hash collision occurs. If you are unfamiliar with chaining, please visit my blog where I explore hash collisions and solutions here.
In this post, I will implement chaining with a hash table. The hash table will resolve collisions that occur within a bucket by storing the collided values into a LinkedList.

Implementing chaining

The hash table design will consist of two classes:
• ChainedHash: the hash table class.
• EntryLinkedList: the LinkedList Object appended to each bucket of the ChainedHash instance.
The diagram below illustrates how the two classes will be used together. The numbers inside the EntryLinkedList Objects represent the values from their key:

The ChainedHash class will use the following methods to implement chaining:
• put(key, value) to place key-value pairs in the hash table.
• hashFunction(key) to calculate the hash value of the key.
• validateLoadFactor() to ensure the load factor of the hash table is monitored and adjusted as new entries are made.

The buckets within the ChainedHash class will be an array of EntryLinkedList Objects. Each EntryListObject will contain the following variables:
• String key to store the key.
• int value to store the value.
• EntryLinkedList nextListEntry to link additional key-value pairs to the bucket.

Class: EntryLinkedList

The EntryLinkedList constructor will instantiate the key, value and nextListEntry. There are getters and setters for the value and nextListEntry, and a getter for the key.

    private String key;
    private int value;
    private EntryLinkedList nextListEntry;
        
    EntryLinkedList(String key, int value) {
        this.key = key;
        this.value = value;
        this.nextListEntry = null;
    }

Class: ChainedHash

The ChainedHash class instantiates its buckets from the constructor. The load factor for the hash table will be evaluated against a static constant: MAX_LOAD_FACTOR. The MAX_LOAD_FACTOR represents the upper-bound value of the load factor. The constructor will take an argument to define the initial number of buckets and it will instantiate all buckets to null.

    private double loadFactor;
    EntryLinkedList[] hashTableBuckets;
    private static int numberOfBuckets;
    private double numberOfEntries = 0;
    public static final double MAX_LOAD_FACTOR = 0.75;

    public ChainedHash(int bucketsToCreate){
        numberOfBuckets = bucketsToCreate;
        hashTableBuckets = new EntryLinkedList[numberOfBuckets];
        for (int i = 0; i < numberOfBuckets; i++) {
            hashTableBuckets[i] = null;
        }
    }
    ...

The hashFunction(String key) method is used to calculate the hash value of the key passed in. The hash function will sum the ascii values of the String (lines 4 to 6) and return the modulo of the sum against the number of buckets (line 9 and 11):

    public static int hashFunction(String key) {

        int asciiValue = 0;
        for (int i = 0; i < key.length(); i++) {
            char c = key.charAt(i);
            asciiValue = asciiValue + (int) c;
        }

        int hashValue = Math.round(asciiValue % numberOfBuckets);

        return hashValue;
    }

The validateLoadFactor() method is called within the put(key, value) method to ensure the load factor of the hash table is not greater than the MAX_LOAD_FACTOR constant. If the load factor is greater than MAX_LOAD_FACTOR, the numberOfBuckets static value is doubled (line 4) and the hash table is replaced by calling the rehash() method (line 6).

    private void validateLoadFactor() {
        if(calculateLoadFactor() > MAX_LOAD_FACTOR){
            System.out.println("Current load factor: " + calculateLoadFactor() );
            numberOfBuckets = numberOfBuckets * 2;
            System.out.println("Buckets raised to: " + numberOfBuckets + " buckets.");
            this.rehash();
        }
    }

The rehash method will recreate the hash table using the updated number of buckets in 4 steps:
1. On line 3, a temporary EntryLinkedList array is created as a replica of the existing array.
2. On line 5, the existing EntryLinkedList is replaced by a new array with the new number of buckets.
3. On line 6, the numberOfEntries static value is reset to ensure the new load factor is correctly recalculated.
4. From line 7, the new hash table is created by looping through the previous entries in the temp array and appending them into the new EntryLinkedList.

Note, the hash function is also adjusted to the new numberOfEntries value and will therefore compute new index positions for each key into the new EntryLinkedList array.

  private void rehash(){

        EntryLinkedList[] temp = hashTableBuckets;

        hashTableBuckets = new EntryLinkedList[numberOfBuckets];
        numberOfEntries = 0;
        for(int i = 0; i< temp.length; i++){
            if(temp[i] != null){
                this.put(temp[i].getKey(), temp[i].getValue());

                while(temp[i].getNextListEntry()!= null){
                    this.put(temp[i].getNextListEntry().getKey(), temp[i].getNextListEntry().getValue());
                    temp[i] = temp[i].getNextListEntry();
                }
            }
        }
    }

The diagram below illustrates the change made to the hash table following a rehash as it doubles in size and realigns each key-value pair.

The put(key, value) method is used to add a new key-value pair into the hash table. The put method consists of 4 key steps:
1. The hashFunction(key) method is used on line 2 to calculate the hash value for the key.
2. On line 5, the load factor is verified against the MAX_LOAD_FACTOR.
3. On line 7, the put method checks if the existing hashFunction value already has an EntryLinkedList Object in the array. If the array position is null, the put method will create a new EntryLinkedList and populate the first position with the value (line 9).
4. If there already exists an EntryLinkedList in the bucket, the put method will loop over the EntryLinkedList array (line 14) to either replace the value of the existing key (line 20) or append the value into the next position on the EntryLinkedList (line 23).

    public void put(String key, int value) {
        int hashValue = hashFunction(key);
        numberOfEntries++;

        validateLoadFactor();

        if (hashTableBuckets[hashValue] == null) {

            hashTableBuckets[hashValue] = new EntryLinkedList(key, value);

        } else {
            EntryLinkedList hashValueLinkedList = hashTableBuckets[hashValue];

            while (hashValueLinkedList.getNextLinkedValue() != null && !hashValueLinkedList.getKey().equals(key)) {
                hashValueLinkedList = hashValueLinkedList.getNextLinkedValue();
            }

            if (hashValueLinkedList.getKey().equals(key)) {
                hashValueLinkedList.setValue(value);
            } else {
                Application.numberOfCollisions++;
                hashValueLinkedList.setNextListEntry(new EntryLinkedList(key, value));
            }
        }
    }

Using the hash table from the main method

The main method will begin by instantiating a ChainedHash instance with 8 buckets. The instance will then call the put method followed by the printEntryLinkedList() method which will print each entry of the hash table to the console.

    ChainedHash hashTable = new ChainedHash(8);

        System.out.println("First entry (Alex, 47)..");
        hashTable.put("Alex", 47);
        hashTable.printEntryLinkedList();

        System.out.println("\nOne more entry (George, 80)..");
        hashTable.put("George", 80);
        hashTable.printEntryLinkedList();

        System.out.println("\nOne more entry (Phillip, 19)..");
        hashTable.put("Phillip", 19);
        hashTable.printEntryLinkedList();

        System.out.println("\nOne more entry (Eliza, 97)..");
        hashTable.put("Eliza", 97);
        hashTable.printEntryLinkedList();

        System.out.println("\nOne more entry (John, 90)..");
        hashTable.put("John", 90);
        hashTable.printEntryLinkedList();

        System.out.println("\nOne more entry (Lin, 16)..");
        hashTable.put("Lin", 16);
        hashTable.printEntryLinkedList();

        System.out.println("\nOne more entry (Miranda, 84)..");
        hashTable.put("Miranda", 84);
        hashTable.printEntryLinkedList();

        System.out.println("\nTotal collisions: " + numberOfCollisions);

Output review

The output includes many debugging lines such as the hash table output and a prompt of what action is being taken for the explanation.

The main method adds key-value pairs as expected up to the third put call (line 11 main method). The key-value pair of (Phillip, 19) maps into index position 2 from the hash function. As a result of the collision, the value of Phillip is appended to the EntryLinkedList instance with (Alex, 47) on line 10 in the output.

When the 7th entry is made on line 28 in the main method (Miranda, 84), the load factor of the hash table is calculated as 0.875 (line 31 in the output). The load factor exceeds the 0.75 MAX_LOAD_FACTOR constant and the hash table is doubled in size to 16 buckets (line 32 in the output). Each entry of the existing hash table is recalculated in the rehash.

The final output of the hash table is shown from lines 33 to 39 in the output. Each index position for the key-value pairs are recalculated and almost all entries move to a different bucket. The new hash table reduces the likelihood of collisions as there are more buckets. The only collision that occurred from the original 8-bucket ChainedHash has been reduced to separate buckets.

The efficiency of the hash table can be evaluated against the number of collisions that are created (as a result of the hash function) and the number of times a rehash is performed. Collisions retain constant time access within the hash table, however rehashing will create a performance cost as each entry must be recalculated and entered.

First entry (Alex, 47).. 
[hash value='2', key='Alex', value=47], null

One more entry (George, 80)..
[hash value='1', key='George', value=80], null
[hash value='2', key='Alex', value=47], null

One more entry (Phillip, 19)..
[hash value='1', key='George', value=80], null
[hash value='2', key='Alex', value=47], [hash value='2', key='Phillip', value=19], null

One more entry (Eliza, 97)..
[hash value='1', key='George', value=80], null
[hash value='2', key='Alex', value=47], [hash value='2', key='Phillip', value=19], null
[hash value='5', key='Eliza', value=97], null

One more entry (John, 90)..
[hash value='1', key='George', value=80], null
[hash value='2', key='Alex', value=47], [hash value='2', key='Phillip', value=19], null
[hash value='5', key='Eliza', value=97], null
[hash value='7', key='John', value=90], null

One more entry (Lin, 16)..
[hash value='1', key='George', value=80], null
[hash value='2', key='Alex', value=47], [hash value='2', key='Phillip', value=19], null
[hash value='3', key='Lin', value=16], null
[hash value='5', key='Eliza', value=97], null
[hash value='7', key='John', value=90], null

One more entry (Miranda, 84)..
Load factor is too high: 0.875
Buckets raised to: 16 buckets.
[hash value='2', key='Phillip', value=19], null
[hash value='3', key='Lin', value=16], null
[hash value='12', key='Miranda', value=84], null
[hash value='5', key='Eliza', value=97], null
[hash value='9', key='George', value=80], null
[hash value='10', key='Alex', value=47], null
[hash value='15', key='John', value=90], null

Total collisions: 1

Conclusion

The implementation of chaining in a hash table can be achieved with just a few methods that cover the load factor evaluation, rehash, and put. By experimenting with the hash function, MAX_LOAD_FACTOR and the number of buckets from instantiation, the performance of the hash table can be altered to reduce the number of rehashes required, the bunching of values in buckets and collision frequency.
Feel free to clone the source code from this blog on GitHub to test the impact of more entries on the performance of the hash table.

How the JVM manages memory

Mon, 11 May 2020 16:34:00 GMT

Key Takeaways
• Memory management plays a vital role in the performance of an application.
• Java uses a garbage collector to automatically manage memory on behalf of the developer.
• There are six different types of garbage collectors.

Automated memory collection in Java

Memory management is the process of assigning memory to Objects created by an application and releasing the memory when the Object is no longer required.
In languages such as C++ and C, the developer will need to manually allocate Objects memory using methods such as malloc() and calloc(). If the developer does not remember to release the Object from the memory, a Memory leak could occur.

Memory leak: when an Object continues to consume memory even after it is needed or when an Object stored in memory cannot be accessed by the programme.

Memory management can be completed with many different strategies to ensure the application efficiently frees and allocates memory during execution. Memory management techniques used within applications are primarily concerned with the performance requirements of the following factors:
• Throughput: how quickly memory can be freed from Objects that are no longer required.
• Stop the world events: a period when the application is completely stopped to allow memory to be freed and assigned to all live Objects in the application.
• Memory fragmentation: when memory is unevenly distributed across a memory block, leaving gaps of unfilled memory slots between Objects. There is a cost-benefit consideration that may arise between the performance gain of leaving memory fragmented against the cost of memory defragmentation to the application runtime.

Garbage collectors

Java makes use of garbage collectors which promise the developer that Objects later referenced within the application will not be deleted from memory. Without such as promise, it is possible that a separate thread may obtain access to an Object reference and remove its memory allocation thus resulting in a null pointer when the Object is referenced from a different thread.

The different types of garbage collectors

There are 6 different types of garbage collectors that can be selectively used together. The different types of collectors allow an application to efficiently manage memory with the relative importance of throughput, memory fragmentation, and stop the world events implemented by design.

Do nothing garbage collector

Do nothing collectors will only guarantee that live Objects are kept alive.

Reference counting garbage collector

Reference counting resolves the problem of having circular references within our code. As the garbage collector is responsible for collecting dead (unreferenced) Objects, a circular reference, where two Objects reference each other, can create a condition whereby neither Object will be collected. Circular referencing exists in the example below:

class One {
    private Two two;
     
    public void setTwo(Two two) {
        this.two = two;
    }
}
 
class Two {
    private One one;
     
    public void setOne(One one) {
        this.one = one;
    }
}
 
public class App {
    public static void main(String[] args) {
        //one instance of each class is created
        One instanceOne = new One();
        Two instanceTwo = new Two();
 
        // A circular reference is made between the two classes
        instanceOne.setTwo(instanceTwo);
        instanceTwo.setOne(instanceOne);
 
        // Although the classes are now null, their reference to each other
        // remains in the other's method, therefore preventing collection
        instanceOne = null;
        instanceTwo = null;
    }
}

Reference counting will resolve the problem of circular references; each Object will own a counter that is incremented and decremented as they are referenced and dereferenced. As a result, the counter assigned to each Object will allow the garbage collector to identify Objects that are truly referenced by the root set.

Root set: the initial set of Objects from which the reachability of all the other Objects can be derived upon.

Mark and sweep garbage collector

Mark and sweep performs three individual phases in garbage collection:
• Mark: identify Objects that are currently in use.
• Sweep: remove unused Objects.
• Compact: remove memory fragmentation that may occur from the sweep phase.

Mark phase

During the mark phase, the Objects that are reachable are identified by following the course of the root set.
In the image below, the identified Objects within the memory are marked as 'live', Objects that reference each other without a root set reference will not be marked as live.

Sweep phase

During the sweep phase, the Objects that were not marked as 'live' are removed from the heap memory. As a result, the memory block contains only live Objects.

Compact phase

The current memory block contains a fragmented distribution of Objects. During the compact phase, the physical memory address of each Object is changed. The Objects are rearranged in the memory block so that they are aligned next to each other. If a compact phase was not executed, Objects would need to fit within the fragmented spaces of the memory block if they are to be stored. Following the compact phase, the memory block has an open space to add Objects as they are assigned to the heap, thus allowing the full memory block to be utilised.

Copying garbage collector

The copying garbage collector uses two memory blocks: 'fromSpace' and 'toSpace'. During each garbage cycle, Objects will be stored in one of the blocks; the block used for storage is known as the 'fromSpace' block. When the 'fromSpace' block becomes full, a mark and sweep will be performed by the garbage collector, moving the live Objects into the 'toSpace' memory block. The 'fromSpace' memory block will now be empty and will be referenced as the 'toSpace' memory block for the next collection cycle.

Incremental garbage collector

An incremental garbage collector does not look at all the Objects during a collection; the cost of the collection is therefore reduced. Similar to the copying garbage collector, the incremental collector will use different memory blocks to divide the Objects that are to be reviewed based upon certain characteristics.

Generational garbage collector

A generational garbage collector is a form of incremental garbage collector. Generational collectors aim to improve the efficiency of a mark and sweep cycle by dividing Objects into sets (generations) according to when they were last used. As an Object survives a mark and sweep cycle, they are promoted into an 'older' generation which are reviewed less frequently for collection. Different generational garbage collectors will implement rules that determine when an Object is promoted and when generations are reviewed by the cycle. Once an Object is moved into the older generation, the younger generation is cleared to allow new Objects to be assigned memory.

How the Java Virtual Machine uses garbage collection

The JVM uses generational, copying, and mark and sweep garbage collectors strategies. The JVM garbage collector consists of two generations: young and old. There are two types of garbage collections: minor collections that occur within the young generation, and full collections that occur across both generations.
The young generation has 3 different memory spaces within it:
• Eden space: where new Objects are directly allocated memory.
• Survivor space 1 and 2: live Objects are assigned into one of the spaces following a minor collection. The two spaces are used for copying garbage collection and memory defragmentation.

Garbage collection cycles within the JVM
The example below shows the movement of Objects within a JVM garbage collection:
1. New Objects are assigned to the Eden space upon creation.
2. When the Eden space is full, a minor collection is run using a mark and sweep.

3. Objects that survive the mark and sweep are assigned into one of the survivor spaces. Below, the marked Objects from the Eden space are moved into the Survivor 1 memory block.

4. When the Eden space becomes full again, the minor collection will be performed on the Eden and Survivor 1 space. The Objects that survive the collection are moved into the survivor space 2.

The Objects are compressed in survivor space 2. They have a number assigned to them which represents the number of cycles they have survived.

5. The JVM can define a number of cycles an Object must survive before it is eligible to be promoted into the old generation. In the example below, 3 cycles are used as the promotion value. When the survivor and eden space are full, Objects that meet the criteria for promotion are moved into the old generation. The old generation Objects will no longer be reviewed by the minor collection.

6. When the old generation eventually becomes full, a full garbage collection is run. The full collection will scan both generations, perform a mark and sweep and remove dead Objects from the young and old generations.

The JVM garbage collector assumes that Objects either live for a short time, or they live forever. By dividing the memory blocks into generations, the memory management system can utilise minor collections to identify and remove short-lived Objects while only using stop the world events sparingly during full collections of the old and young generation.

Conclusion

Garbage collectors introduce different types of mechanisms for allocating and freeing memory within an application. There are many variables that are involved with garbage collection that require consideration. The many types of implementations also allow the garbage collector to leverage different styles to efficiently manage memory. The JVM garbage collector process covered in this blog post is the default serial garbage collector. The multi-threaded capability of Java can enable concurrent garbage collectors to minimise the use of stop the world events. There are three other collectors that can be used in Java which offer various advantages and drawbacks for application behaviour: G1, parallel, and CMS.

Resolving hash collisions

Mon, 04 May 2020 16:34:00 GMT

Key Takeaways
• Hash tables and hash maps are efficient data structures to be used for data that requires dictionary operations.
• Collisions occur when a hash function computes to the same value for different keys.
• Collisions can be resolved with open addressing and closed addressing techniques.

What are hash functions and why do we use them?

Hash functions are used to map data of any arbitrary size into a fixed length. Java uses hash functions in hash maps and hash tables as key-value data structures.
Hash map and hash table are both efficient data structures to use for dictionary operations (insert, search, delete). There are just a few differences between hash map and hash table:
• Hash map data is not synchronised, hash table data is synchronised.
• Due to synchronisation, hash map operations can be performed faster than hash table.
• Hash map allows one null key and any null values, hash table does not allow any null keys or values.

As the differences between a hash maps and hash tables are negligible, the remainder of this post will use hash tables for consistency. The context of hash functions are equally as applicable to hash maps.
Hash tables store data in a key-value pair that use a fast-access location known as a "bucket" for storing the value. Buckets are stored in an array; when an entry is made to a hash table, the position in which the value is stored within the array is calculated through the key of the entry. Hash tables can be designed for "direct-addressing" where each key is an integer and is directly used as the index position for where the corresponding value is stored in the array. Direct-addressing can create a problem as the size of the array used for storing keys can become excessively large. The number of keys stored within the array may be relatively small thus meaning the majority of the array space (and computer memory) assigned to the hash table is not utilised.
Hash functions provide a solution to direct-addressing by reducing the array size for storage, but keeping an appropriate number of open buckets to store values. Each entry will use the key as an input to a hash function which will produce the index position to the bucket that the value is stored in.

The hash function is defined by the public int hashCode() method of the key. The example below demonstrates how a key-value pair of peoples names and age {String, Integer} can be placed within a hash table; H({key}) is read as "hash of {key}" and describes the use of the hash function to obtain the index position.

In other implementations of the hash table, the key and value can be completely different types of Objects.

The benefit of using a hash table

Hash table resolves the time-complexity problem for searching, inserting, or deleting a value within an array when the index position is unknown. If an array has 100 integers, a for-each loop would result in up to 100 searches, and a time-complexity of O(n) - learn more on the time complexity here. Hash tables support the direct searching of a value based upon its key. As a result, the hash table can pinpoint the index of the value instantly, regardless of the number of elements within the hash table. Hash tables support dictionary operations with a constant time O(1).

Hash function collisions

The example above does not explain how the hash function actually derived the Integer value from the String. The below image shows the formula within the hash function:

A collision will occur when the hash function computes two different keys into the same index position of the array.
If a fourth entry with a key-value pair of: {"Phillip" , 19} was added to the hash table, the index position would equal 2 if we follow the same hash function:
1. "Phillip" ASCII value = 722
2. 722 / 8 = 90
3. 90 % 8 = 2
The addition of a new key, "Phillip", will create a collision with "Alex", which is also stored at index 2. Without a collision resolution, {"Alex" , 47} will be replaced by the key-value pair {"Phillip" , 19}.

As the hash table is populated with more entries, the potential for collisions increase.
Load factor is an important statistic used with hash tables, defined as the number of entries divided by the number of buckets. A hash table can implement dynamic resizing where the array will be doubled in size when the load factor has reached a certain value. In practice, a load factor of 75% is deemed the upper-bound limit of populated buckets within the hash table array before it is resized.

Resolving collisions

There are two strategies used for resolving collisions:
• Open addressing: finding alternative locations for storing the collided value, ensuring only one value is stored per bucket.
• Closed addressing (also known as chaining): storing an arbitrary number of values per bucket by using a separate data structure from within the bucket.

Open addressing

Linear probing

Linear probing will re-direct the index to point to the next available bucket in the hash table. For the case of "Phillip", which otherwise would fall into bucket 2, the next available bucket would be bucket 3. As a result, the value of "Phillip" will be stored in bucket 3. If another key is added to the hash table with a hash function of 2, the value will instead be stored in bucket 4.

Cluster effects

Clustering in hash tables refers to the degree that items are "bunched" when stored. A high degree of clustering is likely to increase the chances of collisions, and as a result, can result in less efficient storage of data.

Linear probing can create high amounts of clustering within an array as buckets become populated near each other. Quadratic and double-hash probing algorithms aim to reduce the possibility of high clustering within an array.

Quadratic probing

Quadratic probing resolves collisions by using a quadratic equation to compute the new index away from the original hash function index. The following indices would be checked sequentially for the example of "Phillip" (also assuming a much larger size array):
• 7 + (1*1) = index 8.
• 7 + (2*2) = index 11.
• 7 + (3*3) = index 16.
• 7 + (4*4) = index 23.
• 7 + (5*5) = index 32.
• 7 + (6*6) = index 43.
The quadratic probing collision resolution will result in a less dense cluster of values than linear probing.

Double-hash probing

Double-hash probing resolves collisions by applying a secondary hash to the computed index to create a new random index. A second hash function can be defined and the subsequent index will be computed upon by alternating between hash functions until an empty bucket is found.

Open addressing can enable collisions to be resolved with loose clustering to reduce the chance of repeated collisions. The use of quadratic and double-hashing algorithms, however, increases the compute time of the hash function. When a search operation is performed on a hash table, the computational cost will accumulate through the additional probing algorithms and the degree of clustering within the hash table. Closed addressing offers a resolution to collision hashing which avoids the need for expensive computations to find new index positions.

Closed addressing

Closed addressing offers a more direct approach to collision resolution; rather than finding a new bucket to store the value, each bucket contains a linked list that can be appended upon.
When a hash function identifies a new collision, the value will be placed in the next available position within the linked list of the bucket.
When a search is conducted upon the hash table, the hash function will identify the bucket and list position of the respective value. The addition and retrieval of a value within a linked list is performed at constant time O(1), therefore closed addressing supports the efficient storage of values to the hash table.

Conclusion

Hash functions are algorithms that are used to convert a key into an arbitrary integer. Hash functions enable the hash table to have an efficient computer memory consumption by defining the boundaries of possible index positions in which a value can be stored. Collision resolution techniques are important to understand as they support the use of hash functions by maintaining the integrity and performance of the hash table.
By understanding the differences between open and closed addressing, a hash table can be designed to store collided hash values in an optimised manner.

Packaging and logging a Java application

Mon, 27 Apr 2020 16:34:00 GMT

Key Takeaways
• Maven-assembly-plugin enables us to package Maven dependencies into an executable JAR file.
• We can use Log4J to debug and understand the runtime context of our application as it is run from a server.
• There are many ways that we can configure the format and layout of our logs to enhance readability.

Running a Java application with a JAR

I have recently been developing applications that perform small and simple tasks to eliminate the repetitive work from my daily activity. The applications are packaged into an executable JAR file and run on a rota through a batch job.
Each time I create an application, I repeatedly deploy a similar project structure that allows me to include dependencies within the JAR and external logging of the runtime. The JAR is designed to be run from a server and will provide debugging and context in the form of logging to the runtime execution.

We will begin by running a simple Java class from the command line:

public class Example{

public static void main(String[] args){

    System.out.println("Hello from your Java application");
    }
}

Once we have navigated to the directory of the class, we can compile and execute the main thread with the following lines:

javac Example.java

java Example
Hello from your Java application.

We can achieve the same result if we package our class into an executable JAR and run it from the command line.
Using IntelliJ, you can package your application with the following steps:

1. Select File > Project Structure
2. Select Artifacts > +
3. Select JAR > From modules with dependencies
4. Select Example Class (or class with the main thread) > OK > OK.
5. Select Build > Build Artifacts... > Build

IntelliJ will create a new Out directory in your project structure that will contain the JAR file.
IntelliJ will also create a new folder and file in the src folder: src/META-INF/MANIFEST.MF
The Manifest file contains information about the project structure; the JAR will reference the Manifest file to locate the class with the main thread, thus making the JAR executable.
If we return to the command line, we can now run our class through the JAR file that has been created in the out/artifacts/{project name}_jar/ directory:

java -jar example.jar
Hello from your Java application.

Injecting dependencies into a JAR file

Our current JAR file will compile the classes and run the main thread as per the Manifest file.
I will now introduce a new Class that uses the jackson.databind dependency to convert a JSON file to and from a POJO.
The purpose of this example is to introduce a dependency into our application; the main thread will create a POJO and return the 'firstName' property as per the JSON file.
The pom.xml will include a new dependency:

<dependencies>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.11.0.rc1</version>
    </dependency>
</dependencies>

The main method is also updated to use the dependency:

    public static void main(String[] args) throws IOException {

        ObjectMapper mapper = new ObjectMapper();

        Student student = mapper.readValue(
                new File("data/student_a.json"), Student.class);

        System.out.println("First name: " + student.getFirstName());

    }

If we attempt to rebuild and run our JAR file with the new dependencies, we will receive the following error message from the command line:

no main manifest attribute

The error arises as the JAR file has not been generated through Maven and does not reference the dependencies.
To resolve the problem, we can use maven-assembly-plugin from within our pom.xml file to include the dependencies and reference the location of the Manifest file:

<plugin>
    <artifactId>maven-assembly-plugin</artifactId>
    <configuration>
        <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
        <archive>
            <manifestFile>src/main/java/META-INF/MANIFEST.MF</manifestFile>
        </archive>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Within maven-assembly-plugin, there are a few key values that set the configuration of the JAR file creation:
We define descriptorRef to the value 'jar-with-dependencies'.
The configuration of the descriptorRef will instruct Maven to create a JAR that contains the unpacked dependencies. The descriptorRef will subsequently append 'jar-with-dependencies' to our JAR filename.
The execution block ensures the maven-assembly-plugin is included during the Maven build lifecycle. Therefore, the JAR file will be updated upon each build.
The Manifest file is defined within the manifestFile property of the archive configuration; the JAR can now find and use the Manifest file properties.
Once we have configured our pom.xml file, we can build our new JAR by selecting Maven > Install in the Maven Lifecycle menu on the right side of the IDE.

Maven Install will build a new JAR in the target directory with the dependencies included.
Before we run the new JAR file, it is important to remember that any relative file locations you may specify in the application (such as a properties file) must also be moved relative to where the JAR file is located, otherwise the JAR will not be able to resolve the location.
If we navigate to the target directory at the command line, we can run the JAR file to successfully include the dependencies and print the result:

java -jar example-1.0-SNAPSHOT-jar-with-dependencies.jar
First name: Alice

Introducing logging for our packaged application

Running our packaged application with Maven dependencies is the first step to creating simple and transparent applications, however in a production environment we may find some problems...
• What happens if the application throws an exception?
• How can we verify that the application is working as expected?
• How can we debug our application if any unexpected results occur?
The concerns outlined above can be resolved with logging which will provide us with a view into the runtime activity of our application.
Logging will create a persisted document from the application that can be referenced for debugging or runtime information. The log file can be generated at custom intervals such as each day, each hour, or each execution.

Log4J

Log4J is a popular package that we can implement into our application to provide a detailed logging context.
We will register Log4J to our application and append records to it throughout the life of the application. Each record will include information or debugging context from the application.
We can format the log files to be conveniently stored alongside the JAR file in a separate directory, named with the time and date for ease of access. Log4J will be included in our application with the following Maven dependency in our pom.xml file:

<dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>1.2.17</version>
</dependency>

The runtime configuration for the log is defined in the properties file log4j.properties. Properties may include details such as where we log our files to and the format of the records:

log4j.rootLogger=DEBUG, consoleAppender, fileAppender

log4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender
log4j.appender.consoleAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.consoleAppender.layout.ConversionPattern=[%t] %-5p %c %x - %m%n

log4j.appender.fileAppender=org.apache.log4j.RollingFileAppender
log4j.appender.fileAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.fileAppender.layout.ConversionPattern=[%t] %-5p %c %x - %m%n
log4j.appender.fileAppender.File=log/${current.date.time}_Application.log

The above file makes use of the Log4J interfaces to define and create two different logs.
On line 1, the rootLogger defines variables that we will use to assign different logs.
From line 3 onwards, we use the log4j.appender interface to create and design the layout of our logs:
On lines 3 to 5, we are defining a pattern layout for a log to the console.
On lines 7 to 10, we are creating a File for each log. The format for the log is also defined and the creation of the log will append the current date and time of the machine to the filename.

Once we have the properties configured, we can implement Log4J in our main class with a new static method and two lines to instantiate the logger and reference the properties file:

 static{
        SimpleDateFormat dateFormat = new SimpleDateFormat("yyMMdd-hhmmss");
        System.setProperty("current.date.time", dateFormat.format(new Date()));
    }

    static Logger logger = Logger.getLogger(Example.class);

    public static void main(String[] args){

        PropertyConfigurator.configure("log4j.properties");
    
    ...

The static block at the top of the main class will set the system property we define as current.date.time.
You may recognise the system property from the properties file as it is used to append the date and time for log filenames.
On line 6, we create an instance of the logger. As we may create many different loggers in our application, we name our logger after the class. Using the class name helps us to avoid name duplication and also provides descriptive detail in the log file for which logger within the application is being used.
On line 10 we use the PropertyConfigurator which allows our logger to be configured by an external file.

Now that the logger is successfully configured to the application, we can begin writing logger records that will provide information and debugging context to our log file:

 
 //inside the main method

logger.info("Starting the application");

 ...

 catch(IOException e){
            logger.fatal("Unable to find the json file");
        }

The output of our log file from a successful application run will appear as:

[main]  INFO    Application     - Starting the application

However, if an exception is thrown, we may see more detail:

[main]  INFO    Application     - Starting the application
[main]  FATAL   Application     - Unable to find the JSON file

Conclusion

Packaging and logging are both frequent utilities to implement in our applications; they are two important foundations to any new application we build to enable transparency and ease of use to the application.
The example we have used has only touched upon the various message types and formatting that is available with Log4J. I would encourage you to explore Log4J functionality here to enhance your logging to be succinct and effective.
If you would like to implement logging across your application with enhanced control and structure, please visit my blog on aspect-orientated-programming here.
Finally, you can find the source code from this blog via GitHub here.

Implementing AOP in a Spring Application

Mon, 20 Apr 2020 16:34:00 GMT

Key Takeaways
• AOP enables us to separate cross-cutting concerns from within our application.
• Pointcut expressions enable fine-grained AOP implementations.
• Spring AOP makes use of light-weight AspectJ AOP classes and methods.

What is AOP?

AOP enables us to apply additional functionality to our application to separate cross-cutting concerns without updating the business logic source code.
A cross cutting concern, for example, may include security, logging, or analytics.

AOP introduces several keywords during its best-practice.
We will use the below example to describe a simple business logic application.

public class FootballGame {
	
	public void playGame() {
		System.out.println("Game has started.");
	}
}

The method playGame() is known as the Joinpoint. The Joinpoint is an exact point within the execution of an application that can be defined on the thread. An example of a Joinpoint can be calling a constructor or method, or throwing an Exception.
We are able to define different Joinpoints within our application by using Pointcut expressions. A Pointcut expression is a predicate that can match different Joinpoints. We define Pointcut expressions to pin-point certain Joinpoints within the application that we would like to apply our cross-cutting concerns to. If we defined a Pointcut expression for the above example, it would define the package, class name and method signature exactly.
In AOP, Advice is the action that is taken when a Pointcut expression is satisfied. Both the Pointcut expressions and the Advice are defined together within an Aspect class. The Aspect class is a handy way Spring can identify if, and where, to find possible AOP implementations to be applied in conjunction with a Joinpoint.
You may be wondering when, in relation to the Joinpoint execution, that the Advice method is called. The exact position in relation to the Joinpoint can be defined by the of 5 different types of Advice, illustrated below:

• @Before is called before playGame() is executed.
• @After is called after playGame() is executed.
• @Around is called both before and after playGame() is executed.
• @AfterReturning is called after playGame() is executed and returns a value.
• @AfterThrowing is called after playGame() is executed, and if it throws an Exception.

Implementing AOP step by step

AOP requires two dependencies in our Spring project, we will define them in our POM.xml file:

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.springframework/spring-context -->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring-version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.aspectj/aspectjweaver -->
        <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>1.9.5</version>
        </dependency>
    </dependencies>

We will be using the spring-context dependency for supporting Spring components and aspectj for supporting the Aspect class and Advice implementations.

public static void main(String[] args) {
		AnnotationConfigApplicationContext context
		= new AnnotationConfigApplicationContext(SpringConfig.class);

		FootballGame footballGame = context.getBean("footballGame", FootballGame.class);
		footballGame.playGame();
		context.close();
	}

We will begin by instructing the main thread to call a Joinpoint called playGame().
AnnotationConfigApplicationContext is used to import the Spring Configuration class that defines the package to scan for Spring components.
Context will be used to obtain a Spring bean to which it will later be used to call playGame().

The Spring configuration class exists as below:

@Configuration
@EnableAspectJAutoProxy
@ComponentScan("com.aneesh.aopdemo")
public class SpringConfig {

}

The @Configuration annotation defines the class as a configuration class.
@EnableAspectJAutoProxy enables support for handling Aspect classes.
@ComponentScan("com.aneesh.aopdemo") directs the Spring application on where to search for components.

Business logic is processed in public class FootballGame. This class will begin with a single void method that prints "Game has started.".

@Component
public class FootballGame {

	public void playGame() {
		System.out.println("Game has started.");
	}
	
}

The above class includes the @Component annotation to ensure it is scanned by SpringConfig.

We will now build the Aspect class that will hold the Advice and Pointcut expressions to our business logic:

@Aspect
@Component
public class footballAspect {

	
}

The @Aspect annotation informs Spring that there may be Advice within this class to process.

Defining Pointcut expressions

Before we define the Advice, we need to understand which Joinpoint(s) we would like the Advice to be called upon.
When applying our Pointcut expressions, we use the term "execution". "execution" is the primary method of implementing Pointcut expressions in the following structure:
"execution([optional]{access modifier} {return type} {package} {class} {method name} {arguments}")
By default, the access modifier is 'Public', therefore it can be left undefined from the expression.
For each of the other input values, you can specify custom values or leave it open with an asterisk (*).

"execution(* com.aneesh.aopdemo.* .* () )"
Scan all public access modifiers, all return types for all classes within the com.aneesh.aopdemo package with all method names and no arguments

"execution(* com.aneesh.aopdemo.FootballGame .* () )"
Scan all public access modifiers, all return types for only the FootballGame class within the com.aneesh.aopdemo package with all method names and no arguments

"execution(* com.aneesh.aopdemo.FootballGame .playGame () )"
Scan all public access modifiers, all return types for only the FootballGame class within the com.aneesh.aopdemo package with the method name "playGame" and no arguments

"execution(* com.aneesh.aopdemo.FootballGame .playGame (..) )"
Scan all public access modifiers, all return types for only the FootballGame class within the com.aneesh.aopdemo package with the method name "playGame" and all possible arguments

"execution(private void com.aneesh.aopdemo.FootballGame .playGame (..) )"
Scan all private access modifiers, void return types for only the FootballGame class within the com.aneesh.aopdemo package with the method name "playGame" and all possible arguments

"execution(public void com.aneesh.aopdemo.FootballGame .playGame (String, int) )"
Scan all public access modifiers, void return types for only the FootballGame class within the com.aneesh.aopdemo package with the method name "playGame" and arguments of String followed by int

The examples above illustrate how specific or generic a Pointcut expression can be.
The granularity of Pointcut expressions enable us to define explicitly which Joinpoint(s) we want to be scanned.
We can then use the 5 Advice types to provide further granularity to specify where in the Joinpoint we would like to implement the Advice.
The below example will implement before and after Advice on our playGame() Joinpoint:

Using @Before and @After

@Aspect
@Component
public class footballAspect {

@Before("execution( void com.aneesh.aop.* .playGame (..))")
	public void beforeAspect() {
		
		System.out.println("Make sure players warm up before game.");
	}

@After("execution( void com.aneesh.aop.* .playGame (..))")
	public void afterAspect() {

		System.out.println("Make sure players warm down after a game.");
	}
}

The outcome produced from running the application:

Make sure players warm up before game.
Game has started.
Maintain hydration after the game.

The application has run the Before and After methods in their respective order as expected. As a result, the application has performed some additional behaviours beyond the business source code.

The @Around, @AfterReturning, and @AfterThrowing Advice are implemented slightly differently, introducing a few more variables.
To illustrate the annotations, we will introduce a new method, countPlayers(), that returns the number of players on the pitch if it is equal to 22, otherwise it will throw an Exception.
class FootballGame has been updated (below) and the main class will now call countPlayers() instead of playGame().

public class FootballGame {
	
	int playersOnPitch = 22;

	public void playGame() {
		System.out.println("Game has started.");
	}

	public int countPlayers() throws Exception{

		if (playersOnPitch == 22){
			System.out.println("FootballGame Class has successfully got 22 players.");
			return playersOnPitch;
		}
		else{
			System.out.println("FootballGame Class has not got 22 players and will throw Exception.");
			throw new Exception("FootballGame class Exception: Incorrect number of players.");
		}

	}	

}

public static void main(String[] args) {
		
		AnnotationConfigApplicationContext context
		= new AnnotationConfigApplicationContext(SpringConfig.class);

		FootballGame footballGame = context.getBean("footballGame", FootballGame.class);

		try {
			System.out.println("Main method has counted players value of: " + footballGame.countPlayers());
		}
		catch (Exception e){
			System.out.println("Main method has caught exception: " + e.getMessage());
		}

		context.close();
		
	}

Using @Around

We will create new Advice methods in the Aspect class to process the information from the new Joinpoint. We will include around the execution, and upon returning an int or Exception.

@Around( "execution(  int com.aneesh.aop.* .countPlayers (..))")
	public Object aroundAspect(ProceedingJoinPoint proceedingJoinPoint) throws Throwable {

		System.out.println("Around: Count the players on the pitch...");

		Object result = proceedingJoinPoint.proceed();

		System.out.println("Around: Count is done.");
		return result;
	}

On line 1 we use the @Around type Advice. The Around Advice is very different to Before or After as it is intercepting the method call from the main method to countPlayers(). You will see on line 2 that the method returns an Object.
The business logic expects the call to countPlayers() to return an int. The Around Advice will intercept that method, but it now has the responsibility to retain the business logic and return an int.
The process of receiving, processing, and returning the int is achieved in 3 key sections: {line 4}, {line 6}, and {lines 8 & 9}.
Line 4 will be the business logic processed before countPlayers() is called.
Line 6 will use the ProceedingJoinPoint to execute the actual countPlayers() method.
ProceedingJoinPoint acts as the connection between the Advice and the countPlayers() Joinpoint. By calling .proceed(), we are telling the Advice to continue with executing the Joinpoint in the application.
proceedingJoinPoint.proceed() is a unique method to the @Around Advice as it sits on both sides of the Joinpoint.
By default, the .proceed() method will return an Object. We have defined an Object as the result on line 6, however this can be wrapped in an Integer and returned.
On lines 8 and 9, we process the business logic after the method has returned from countPlayers(). The int is then sent to the main application as it would have expected to be delivered had the Advice not been used. In this example, the int is packaged as an Object.

Using @AfterReturning

The below example uses @AfterReturning:

@AfterReturning(pointcut = "execution( int com.aneesh.aop.* .countPlayers (..))", returning="result")
	public void afterReturningAspect(int result)  {

		System.out.println("AfterReturning has the result: " + result);
	}

On Line 1 we use the @AfterReturning Advice to process information after countPlayers() has returned an Object (otherwise an int).
The argument to the Advice, however, does not only consist of a Pointcut expression. We define the Pointcut to the variable "pointcut", then we also define a String called "return" to the "returning" property.
The value of "returning" is used in the Advice method. The "returning" value is equal to what is returned from countPlayers(). The "returning" value is later back to the main method after the Advice.
We could use the result from countPlayers() in the Advice and return it to the application by adjusting the method signature (on line 2) and returning an int at the end of the method (line 5), however this example will only return void.
On line 2 we can see the method will take an int argument which comes directly from countPlayers(). The business logic is processed on line 4.
The diagram below illustrates the movement of Objects between the countPlayers() and the Advice, before being returned to the main method:

The result from processing the AfterReturning Advice would be:

Around: Count the players on the pitch...
FootballGame Class has successfully got 22 players.
Around: Count is done.
AfterReturning has the result: 22
Main method has counted players value of: 22

Using @AfterThrowing

The current application does not throw an Exception, but if we were to change the playersOnPitch value in class FootballGame, countPlayers() would throw an Exception.
The example below demonstrates how an Exception would be handled with Advice.

@AfterThrowing(pointcut = "execution( int com.aneesh.aop.* .countPlayers (..))",
				 throwing = "thrownExpression")
	public void afterThrowingAspect(Throwable thrownExpression)  {

		System.out.println("AfterThrowing has received a message: " + thrownExpression.getMessage());
	}

On line 2 we have defined the Advice in a similar pattern to the AfterReturning Advice. Instead of using the "returning" parameter, we are using "throwing", and we pass the Exception into the Advice method as a Throwable on line 3.
On line 5, we simply process the message of the Exception before allowing the Exception to be sent back to the main method.
In real business logic, we may log the Exception or transform it before sending it back.

If we change the number of players so an Exception is thrown, the AfterThrowing Advice would return:

Around: Count the players on the pitch...
FootballGame Class has not got 22 players and will throw Exception.
AfterThrowing has received a message: FootballGame class Exception: Incorrect number of players.
Main method has caught exception: FootballGame class Exception: Incorrect number of players.

Notice that the second Around Advice has not been called as an Object was not returned to the Advice. Instead, the thread has thrown an Exception.
The AfterThrowing Advice is acknowledged when the Exception is thrown and run before the Exception is sent back to the main method.

Conclusion

Spring provides simple and effective tools that enable us to implement AOP into our applications.
By defining Pointcut expressions, we are able to specify exact Joinpoints where we would like to add additional processing. We use Advice types to define when we would like the processing to happen, and as a result, we are able to delicately add additional functionality to our application.
By using AOP with object-orientated programming, we are able to drastically improve the maintainability of our source code by grouping additional features and behaviours of our application into modular sections.
Furthermore, we are able to cross boundaries that are defined by packages and classes within our application to provide shared behaviour and functionality that can, incredibly, be defined by just a single method.

Picture: Rio De Janeiro, Brazil by Raphael Nogueira

Calculating the time complexity of algorithms

Mon, 13 Apr 2020 16:34:00 GMT

Key Takeaways
• Time complexity is used to express the efficiency of an algorithm.
• The Big O is a measure that defines the type of CPU growth an algorithm has as the input elements scale upwards.
• The Big O of an algorithm is defined by the operation that is least time efficient.

What is time complexity?

The term ‘time complexity’ is used to describe the efficiency of an algorithm with respect to its number of input elements. Defining the efficiency of an algorithm can be difficult while accounting for variables such as memory, disk, and network usage.
The Big O Notation is used to describe time complexity for the upper-bound runtime; the Big O is expressed for an algorithm's CPU usage alone, therefore discarding the other variables from consideration.

The Big O is part of a family of asymptotic notations which, include Big Theta Θ and Big Omega Ω.
The notations define the upper and lower asymptotic bounds for an algorithm. The Big O is only concerned with the approximate worst-case complexity of an algorithm; in other words, the guaranteed time in which an algorithm will complete.

Big O defines how the resources required (from the operations performed) in the algorithm scale up as the input elements to the algorithm grow.
The Big O does not reflect the performance of an algorithm. The performance of an algorithm would capture how much time or memory is consumed when the algorithm is executed.
The graph below illustrates the 5 classifications an algorithm can have:

• Constant time O(1)

• Logarithmic time O(log n)

• Linear time O(n)

• N logarithmic time O(n log (n))

• Polynomial time O(n²) / Exponential time O(2ⁿ)

Defining the Big O of each operation

We are able to identify the Big O classification of each operation with the help of the Big O Cheat Sheet found here
Below are two examples of algorithms that capture a Big O classification themselves from a single operation:

Constant time O(1) translates to ‘Order of 1’. The resources required to complete the operation will not change as the input size grows. The below example illustrates an O(1) operation:

public void algorithm(List<Integer> values){
    System.out.println(values.get(0));
}

Linear time O(n), defines the linear growth of algorithm complexity in-line with the growth of elements. The below example illustrates an operation of O(n)

public void algorithm(int n){
    for (int i = 0; i<n; i++){
        System.out.println("got the value: " + i);
    }
}

The remaining classifications follow a pattern where the resources required will increase at the respective growth of n.
For example, typical sorting algorithms have a Big O notation of (n log (n)). The sorting algorithm would experience a resource requirement growth of (n log (n)) as the input elements increase by n.

Calculating the Big O of an algorithm

As algorithms will usually consist of multiple operations, we will use the example below to understand how we consider multiple operations into a Big O calculation:

public void calculateMaxValue(List<Integer> values){
    int max = 0;
    for(int i=0; i< values.size(); i++){
        if(values.get(i) > max){
            max = values.get(i);
        }
    }
    System.out.println("Max Value is: " + max);
}

Line 2 is used to initialise a variable. As this line is not influenced by the size of input values, line 2 has a Big O of O(1).
From lines 3 to 7, the function will loop through each member of values and identify if it is the maximum value of the List.
If values = {1,2,3,6,3}, the number of elements, n, will be 5.
The code block will run a total of 5 times in order to validate that the largest value is found within the list.
It will run for int i = 0,1,2,3, and 4 as values.size() = 5. A total of 5 operations for an input size of 5 means the code block has a Big O of O(n).
We can run through the code block again to support the Big O classification. If values = {1,2,3}, then values.size() will equal 3, and the number of operations ran will also be 3 (checking values of 1, 2, and 3).
Lastly, line 8 is used to print the result of the algorithm. This line is not influenced by the number of input values, therefore it is constant time and has a Big O of O(1).

As we account for the three operations within the algorithm, we have accumulated a Big O of: O(1) + O(n) + O(1).
The accumulation can be summarised into: 2 O(1) + O(n).
One defining characteristic of the Big O is that it is only defined by the rate of growth. Big O will not define how tightly an algorithm fits within the classification, but it will define the type of classification it falls within.

As we understand that it is only the type of growth we will consider, we can remove the constants from the accumulated Big O value.
As a result, 2 O(1) + O(n) is reduced to: O(1) + O(n).

The final reduction made to our Big O calculation involves releasing the non-dominant terms, in other words, an algorithm is only as efficient as its least efficient operation.
Working from the least-efficient classification of exponential growth, back to constant growth, we can identify the category the algorithm belongs as the classification that is satisfied first.

• Exponential/Polynomial time: ✘
• N logarithmic time: ✘
• Linear time: ✓
• Logarithmic time: ✘
• Constant time: ✓

We can see from the above checklist, the algorithm satisfies the linear time complexity first (from top-down).
It later satisfies constant time, however this is discounted as it is not the least efficient classification. As a result, the algorithm has a Big O Notation of constant time: O(n).

Conclusion

The Big O notation enables us to better understand how the individual operations within an algorithm can contribute to its upper-bound CPU consumption.
By breaking down an algorithm and understanding how the resource consumption of each operation will scale with the growth of input elements, we are better able to identify where possible inefficiencies lie.
Calculating the Big O will encourage you to explore and utilise the various data structures and methods available to create CPU-optimised algorithms.
The time complexity of our algorithms provide a measurable and reflective purpose for us to review how well each operation within our algorithms can scale with input elements.

Hello World - My first blog

Mon, 06 Apr 2020 16:34:00 GMT

It was roughly 5 years ago when I wrote my first line of code. I had attended a job interview for an internship when I was asked about writing macros to reflect upon my experience with Microsoft Excel. At that point, I had no experience to reflect upon. That evening I wrote my first macro to change the colour of a cell to red with a keyboard shortcut.
This experience taught me that the advent of high-level programming languages could provide a more intuitive experience writing code than I could have thought.

Fast-forward to the present day and I am starting a blog to document the areas of technology that I explore, find interesting, and am challenged by on a daily basis.
My experience with writing a dissertation, where I had to introduce the technologies involved (Big Data and Blockchain) allowed me to appreciate the process and value of understanding a technical concept to its core. My blog will support me as I use the Feynman technique to learn and write bite-sized posts on how and why the technologies I use work.

My expectations for this blog are that it will surround all areas of Java, Spring, and algorithms.
Despite this, the deluge of questions that arrive each day from my search for answers will no doubt broaden the context.
Nevertheless, I am creating this website for my enjoyment and with the hope that you, the reader, may also benefit from one of the many challenges I overcome as an engineer.

Enjoy!