How to write a blog post? I have received a few messages from different readers asking me around this topic.

• How do I find ideas?
• How do I clear my thoughts before writing?
• Do I follow a particular process?
• Do I have any advice that I could share with someone that would like to start blogging?

I still remember the day I started this blog, nearly 13 years ago (as I am writing this, I am realising how prehistoric this sounds). At that time, I had this goal of posting once a month on various technical related topics whatever the idea was originating from, either from work or from personal projects. If it worked pretty well at start, lately, let’s face it, it does not work that well. I feel almost ashamed when I look at the date of the last (previous) article, more than a year ago, this is by far the longest period of inactivity on my blog!

Today, I have decided I will be writing on this particular topic, not only because I have been asked a few times but also because this article represents a milestone. Indeed this is my 100th article so I wanted to write on something slightly different than usual and this feels like the perfect topic.

To be honest, when it’s not part of your full-time job, it takes a fair amount of time and dedication to write articles, and I don’t think there is a magical recipe. Everyone’s mindset works differently, and what can work for me could have the opposite effect on you. In this article, I will try to describe the recipe that works for me. I have applied it to write this article so you will be the best to judge.

Idea

It all starts with an idea (it sounds like a movie trailer I know but it’s true). An idea is a single sentence, no more than a set of keywords, that represents a particular topic, not too specific but not too broad. It can come up while working, while doing tech-related things on my personal computer, while walking, while showering, there is no limit. An idea will usually come up a while before you start to write any content on it, but it’s not always true, and you could find an idea you will be passionate about to write an article in the next hour.

I keep all my ideas as an unordered bulleted list in a note file on my computer. To give you an idea (yes it was an easy joke sorry), I currently have 9 ideas in this list.

Thorough the year, I have this pool of ideas that I can pick up from. As I mentioned previously, there are no rules when picking up an idea, sometimes I write an article on an idea that I just added to the pool (this article is one of them), sometimes I write an article on an idea that I added 3 years ago. Most of the ideas I have now will probably never be transformed into an article, so that’s normal to discard quite a lot of them over time for various reasons, e.g. if the idea is:

• not good enough to make an interesting post ;
• not aligned with the content of my blog, i.e. targeting the wrong audience ;
• about a very niche topic ;
• or, on the opposite, many bloggers already got it covered.

Raw Content

Once I have selected the idea, I find 1 or 2 hours and I start writing. I need some focused time to do this part with as little break as possible. I am talking about writing pure content, a really drafted and raw version of the content, everything I could think of about the idea that I want to talk about, it can be in the form of full sentences, bulleted lists, a few raw links or textual description of the images and diagrams I would like to include, etc.

It depends on the idea, but if it’s a very technical one, I may also do some preparation beforehand and use some notes that I took previously, e.g. it can be a couple of lines of code or other coding examples that I want to include. I can also do some extra bit of research on the Internet to know what is there already, and if there is anything else I did not think of. It’s not a bad thing to get some inspiration from other sources, of course as long as you are not paraphrasing or doing any form of plagiarism!

At this stage, I am not paying attention to spelling, typos and other grammatical mistakes, I am literally writing what comes to my mind and that I would not want to miss out. I also use a simple text editor rather than using the online blog editor to avoid any distraction.

You may not be affected by this, but since my blog is bilingual, available in French and English, I also need to decide in which language I want to write first. In my opinion, it does not make sense to do both languages in parallel, and I prefer to focus on one at a time leaving the translation at the end. Most of the time, I end up writing in English first even if French is my mother tongue. The main reason is because I could write too many complex sentences in French that won’t be easily translatable into English, another reason is the fact I am working in technology in an English environment so English feels simply more natural (I really never think I would write that one day!).

Review, Improve, Polish, Repeat

Now that I got the idea and all the associated information I want to expose and shed light on, you would think I am pretty much at 60% of the writing process. Unfortunately, this is far from that, and realistically I would say I did about 30%.

This is when an iterative, slightly tedious, reviewing & improving process starts. It’s the longest part of writing an article, this is when I think about the structure of the article. I reorganise my thoughts in a logical way so the content will be easy to read and follow by my readers. The return lines, paragraphs, sections, titles and subtitles will naturally appear at that review stage. It’s totally fine to remove some text if you realise it doesn’t fit with the rest, or elaborate on some section if some details were missed, or even rewrite entire paragraphs.

It’s important to adapt the tone of voice and keep it consistent across all your articles. If it’s your own blog, it’s something you won’t even think of, but imagine you are writing an article in a scientific blog that has multiple authors, writing a joke is probably the last thing you would want to do…

Note that I stop using a simple text editor in favour of the online blog editor, so I can use the preview mode and see the final rendering of the article. This is important as this part is not focusing on the text only, but I also think about the images I would like to integrate, the links I would like to include and the formatting I would like to apply.

Visuals

When adding images, don’t overdo it. Adding an image for each section will impact the network bandwidth and cause some distraction to your readers. In most cases, an image at the top of the article will be enough.

To look for the perfect image and if you are not satisfied with your own photos, you could be tempted to use Google Images, but there are other resources available, they will also help you to avoid any form of copyright infringement.

• Unsplash
• Pexels
• The Noun Project

You may need to create a diagram or a schema. I personally use PowerPoint, it may not be the best tool out there but it has a rendering that I like and I am very familiar with Office. In case I need to generate a more technical diagram, I also use PlantUML. It will really depend on what you are writing on and your field of expertise, in short, choose the tool you prefer!

Links

When adding links, don’t overdo it. Linking is the fundamental tool to navigate the Internet, it helps referencing every websites including your own, but adding many links will be distracting and your reader may end up thinking you are advertising rather than providing accurate information.

When adding links, ensure they all have a purpose. Do they help the reader to have more context? Do they let the reader access a tool that you have mentioned? And it’s very important to label your link properly and not use the typical “click here” that does not carry any context.

On the other end of the spectrum, it’s fine if your article does not contain any link or only a few. However, I would suggest to have a section at the bottom of the article with a list of useful links. This can be for reference (also a nice way to thank the authors you may have got some inspiration from) but this can also be useful if the reader wants to carry on more reading on the topic.

Formatting

When adding formatting, don’t overdo it. Consistency and simplicity are key concepts.

You can use multiple levels to differentiate main titles and subtitles in your article. I personally limit myself to 2 levels maximum. If you need more, maybe your article is too complex and you may want to break it down into a series of articles instead.

In general, you should space your text and avoid long and complex sentences. Use bulleted lists where it makes sense, not everywhere.

Some minor text formatting like bold and italic can be useful.

• Bold can emphasise an idea and grab the reader’s attention.
• Italic can be used to introduce a technical term or when quoting a text.

Last Review

At this stage, it’s important to use the preview mode of your blog editor, so you can check that there are no broken links, the images are rendered properly at the right place with the right dimensions, the formatting and spacing is working as expected, etc.

I have to admit I am using some help from Word for my spelling and grammar, what I do is copy the full content to a Word document, and then impact the potential suggested fixes back into the article. A more sophisticated tool like Grammarly or Antidote can also be a great help, and they integrate directly into your browser so you don’t need to transfer the content to an external tool.

If you know someone willing to help reviewing your article, it could be a colleague, a friend or a family member, ask them! It’s always good to have another pair of eyes on your work. Personally, I skip that step and do my own review. But not doing it can be slightly risky, especially if you are posting on behalf of your company for instance. I am pretty sure I have many articles published with typos, this is especially true in the English version of my old articles (hopefully my English is better now). Again, that is not the end of the world, that happens on an individual blog, you already took some time to write and share your knowledge, and your reader won’t judge you for that.

As the title suggests, this is a last review, so don’t review over and over. I know it can be tempting to improve your text again. I think it can be better like that. I think I went into too much details in this part and not enough in that part. The wording is not great here. What if I use this sentence instead? Let’s be honest, there is little chance that you will be fully satisfied.

Since you have already spent hours on your article, it’s time to let it go and click that publish button! Ah, last but not least, don’t forget to find a good title, it needs to be short, catchy and convey the main idea of your article.

Optional Translation

In my case, I am not done and I can’t publish just yet. I also need to translate the article into French before.

Although French is my mother tongue, I take some shortcut to speed up the translation process and I use Google Translate. So I can start from a text already translated that I can review, and potentially reformulate a few sentences here and there. The automatic translation is not perfect, but I have to say that it is pretty good, I would say that about 70% of the content is usually left untouched or with minor changes.

Unless I have to recreate some diagrams in French, the translation is probably the quickest part to do, since I keep the same images, text and formatting, and French is obviously easier for me…

Happy blogging!

The article How to write a blog post? At least my way! was written by Fabian Piau.

In a recent article, Security Magazine stated that 1.3 billion bot attacks were detected in Q3 2020. It’s not a surprise. We are not alone. You are not alone.

Last year, the Opex Excellence team at Expedia Group did a review of the incidents in production that impacted the Hotels.com website. We covered a 12-month period, and bot attacks are one of the most important threats we identified.

Anything that can go wrong will go wrong — Murphy’s law

In this article, I will define and explain what bots and bot attacks are. I hope it will help you in building up some knowledge and getting a better understanding of the threat your website could be facing, so you can come up with a set of strategies and solutions to prevent and mitigate a future bot attack.

Know the enemy and know yourself, in a hundred battles you will never be in peril — Sun Tzu, The Art of War

What are bot attacks?

The first type of bot attack that usually comes to mind are Denial Of Service attacks (DoS) or DDoS (when it’s distributed). A DDoS attack usually impacts the whole website, with failures happening in cascade, making it unavailable. The attacker sends a large number of requests (e.g. perform multiple GET/POST on random URLs) in order to overload the underlying services so much that they cannot handle the traffic anymore. Services will start to respond to requests with timeout or error responses. This obviously also affects the requests originating from legitimate customers — customers that are trying to book their hotel stay, in our case.

Another type of attack, slightly more targeted, are Scraping attacks. A crawler bot can look into various pages in order to extract specific pieces of information. This is also known as Data scraping. They usually target your inventory data, e.g. all the properties details and contact information, or pricing details for specific dates. Sometimes it could even be the whole contents of some pages of your website, in order to replicate them and prepare a phishing attack.

Ideally, the attacker does not want to be seen while scraping a website and will try to span the attack over multiple days to avoid a surge in the traffic that may trigger some alert and investigation on your side. Unless the attack is sophisticated and coming from different machines, you can usually figure out that the website is being scraped when you start to see an abnormal number of requests coming from the same IP address (or range of IPs) at the Edge level. I will give more details about the Edge level in the next part.

If DoS and Scraping attacks are quite general in scope, in the sense they are targeting various pages across the website, we also see attacks against specific pages.

Specialized attacks are much more narrow in scope, and they usually target a specific page or feature of your website. Below are a few real cases that we have been faced with:

• An attack on the Booking page, when the attacker tries to break the coupon field by generating and trying multiple codes, hoping there is an exploitable pattern to the codes. This is using brute force.
• An attack on the Sign in page, when the attacker tries multiple login and password combinations. The idea is to get access to user accounts and confidential data, or what we call Account Take Over (ATO). This is also using brute force.
• An attack on the Mobile App page, when the attacker tries to send large numbers of SMS to random or fake phone numbers. It is quite typical nowadays to prompt users to download an app to their mobile device by sending an SMS with the link to the app on the store. The targeted website will end up paying money as messaging service is usually handled by a third party vendor and each SMS sent has a cost.

General or specific, an attack can be basic or sophisticated. An example of a basic attack is someone trying desperately to find a discount code firing dozen of requests to a coupon service. Basic attacks are difficult to spot but easy to block: you can block the IP address with a WAF (Web Application Firewall) rule. They also have a low risk profile.

On the other hand, with sophisticated attacks, they can be distributed and spanned over thousands of machines located in different countries and using advanced scripting technologies, like headless browser. The level of impact and risk is much higher but they are obviously easier to spot when associated with a traffic surge.

Should we block every bot?

Definitely not! Not all bots are evil; some are even beneficial to us.

• There are the Spider bots (or Crawlers) from popular search engines like Google Search or Microsoft Bing. If we block those, the indexing of the website will be badly impacted, the legitimate traffic will degrade over time, and the website popularity will go down.
• There are also the Commercial bots (e.g. Google AdSense bot), to provide personalized ads to the users, including the ads for our own website.
• There are the Data bots like content aggregators and feeds, and there is also the Archive bot that is building the biggest internet Archive.
• There are the Copyright bots, that look for plagiarism or intellectual property theft. You may have faced one of them if you’ve tried to upload a video on YouTube with your favourite music in background. You certainly quickly realize Google took it down, reminding you politely that you cannot use any protected material.
• There are also the Monitoring bots to make sure your website is healthy, and raise some alert in case it’s not. For example, Akamai, Datadog, and so on use bots to make sure your site is responding properly.

You should not block any of these bots because they are not bad, and usually they are not aggressive, plus they contribute to the Internet. If you feel your legitimate traffic is suffering from them, then instead of blocking, it’s best to have a tarpitting or rate limiting strategy in place to mitigate their impact. More details about this in the next part about the Edge level.

If all these bots are third party, sometimes you can have your own. In our case, an internal bot is regularly checking our landing pages to make sure there are no dead links or unnecessary redirections. So we definitely don’t want to block it!

What can we do?

We can prevent attacks at the Edge level and mitigate them at the Application level.

Use of robots.txt

The first thing that comes to mind when dealing with bots is the robots.txt file. Every website has it and it has been there for ages. This is a text file accessible publicly at the root of your website. It specifies the rules for any bots accessing your site. These rules define which pages the bots can and can’t crawl, and which links they should and shouldn’t follow.

Good bots will follow these rules. For instance, if a website owner doesn’t want a certain page on their site to show up in Google search results, they can write a rule for it, and Google web crawler bots won’t index that page. Although the robots.txt file cannot actually enforce these rules, good bots are programmed to look for that file and follow the rules before they do anything else. It’s based on a code of honor.

Malicious and bad bots will obviously not follow any of your rules. On the opposite, they will often read it to learn what content a website is trying to keep off-limits from them, then access that content. Thus, managing bots requires a more active approach than simply defining the rules for bot behavior in the robots.txt file. This is what we are going to see in the next part.

The robots.txt file can also be used to set up a ‘honeypot’. A honeypot is a fake target for bad actors that, when accessed, exposes the bad actor as malicious. In the case of a bot, a honeypot could be a page on the site that’s forbidden to bots by the robots.txt file. Good bots will read the robots.txt file and avoid that page, some bad bots will crawl the page. By tracking the information of the bots that access the honeypot, bad bots can be identified and blocked. Source: Cloudflare

Advanced Bot Management

The main shield against bad bots is Bot Management at the Edge level. This is much more advanced than the robots.txt file. I took this list from Cloudflare but it will be a similar set of features for any other Edge tool:

• Identify bots vs. human visitors (using behavioral analysis and potentially machine learning)
• Identify bot reputation
• Identify bot origin IP addresses and block based on IP reputation
• Analyze bot behavior
• Add good bots to allowlists
• Add bad bots to blocklists
• Challenge potential bots via a CAPTCHA test, JavaScript injection, or other methods
• Rate limit any potential bot over-using a service
• Tarpit recognized bot requests (see definition below)
• Deny access to certain content or resources for bad bots
• Serve alternative/cached content to bots

‘Tarpitting’ is an interesting feature. It means to add an artificial delay to the request. It is usually much better than blocking because the bot won’t know it has been discovered, but the attack will slow down significantly as fewer requests will reach the Application level, as they could time out at the Edge level. Rate limiting can also be another good strategy you may want to look at.

When deciding about an Advanced Bot manager, you can use a popular third party provider like Akamai or Cloudflare.

Pros

• No impact on the application code.
• A bot rule is relatively quick to deploy with immediate effect.

Cons

• Most of the cons lie in the fact they are third party.
• There is a license cost.
• Bot rules can only be defined against generic and non business parameters like user agent, IP, endpoint, etc.
• They sometimes involve a heavy approval process, e.g. adding a new rule will require people outside of the company plus internal people with special authorization.
• A new rule can have side effects. Unless blocking a unique IP address, it’s very hard to be sure it won’t prevent some legitimate traffic from getting in.
• The maintenance of the rules can be cumbersome over time.
• Partial access and visibility for application teams.

Most of the disadvantages can be mitigated if you are using your own in-house Edge tool. This is particularly interesting when used in addition to a third party Edge tool. It’s obviously not something every company can invest in, but it will give you much more flexibility.

• Ability to set rules related to your business.
• Ability to add some one-off temporary rule to mitigate an attack that you can delete shortly after the attacks passed.
• Ability to centralize the Edge monitoring and make the information available to every team.

Traffic prioritization

The idea here is not about replacing your main Edge tool but to add some bot logic after it. In a nutshell, such a tool will act as a prioritization queue so low value bot requests are deprioritized in favor of real user requests that have higher business value, e.g. ending up with potential booking in our case.

User requests > Internal bot requests > External good bot requests

Netflix is applying some similar concepts that you can read in Keeping Netflix Reliable Using Prioritized Load Shedding.

Caching

We talked about Bot Management but there is something else that the Edge layer can provide for you, and that is the ability to cache content. We usually refer this to a Content Delivery Network (CDN). The idea is to serve cached pages to good bots rather than generate fresh pages.

We did a Proof of Concept last year and it significantly decreased the traffic to our Landing services without affecting the SEO of the website. This year, we are looking at generalizing this approach.

Mitigating at the Application level

Managing bots at the application level means that a bot attack was able to pass through the higher level of protection on the Edge.

The solutions we have at this level are only mitigation solutions in order to be proactive and reduce the burden of an attack.

On a sad note, we know it’s going to happen, and even multiple times, so we need to prepare for it. On a brighter note, what we know is that bot attacks do not last forever. A sophisticated attack costs a significant amount of resources for the attacker, and as resources cost money. As long as we make the attacker spend more value than it is getting, we are certain the attack will be stopped eventually as a bad investment.

There are different actions we can do and most of them are good practices.

• First is when coding the application logic, we can avoid high complexity code, blocking threads, and so on. A good idea is to separate application and management ports. If you use the same port, in case of a bot attack the service will be so overloaded that it won’t be able to respond to health checks and your infrastructure platform will flag it as unhealthy. Even if it does not solve all your issues, having a separate port can mitigate this.
• Having a chaos mindset is important. For critical services, make sure you have load and stress testing in place in your pipeline. This is to ensure your services are resilient enough, and that you have identified potential memory leaks in your code and bottlenecks reaching downstream services or data sources. In case something goes wrong, you still want to serve a degraded response to mitigate the impact to the customer. You could also have some caching mechanism in place.
• Make sure you leverage your infrastructure. If you use Kubernetes, you can take a look at auto-scaling. Be vigilant when enabling it. Ensure the configuration is well thought out and in line with your dependencies. Setting up a high number of pods and consider it as done will be a mistake, as you will also share the load with your downstream dependencies and, if they are not prepared for it, you will basically shift the bottleneck deeper in the stack without solving it. It may also cost you more money, if your infrastructure is hosted on a Cloud provider like AWS. Also make sure your pods are ready to take traffic once they are exposed to the attack. A warm up routine like Mittens will support you, especially for applications that are slow to start up.

There are also other strategies at the application level that are not related to configuration and infrastructure. Some mimic a bot management solution at the Edge level:

• Captcha mechanism. A common way is to display a captcha to the user if there are too many attempts, or to target account-related attacks.
• Authentication mechanism. If your APIs are public, you may want to add some authentication, from ‘Basic Auth’ to ‘CSRF Token’. But you should be aware it will add some complexity to your system, and you have to balance it with the information your API provides, e.g. ask yourself if the content exposed is sensitive enough.
• Caching, Blocking, and Rate limiting mechanism. This may be quite complex to achieve and maintain, especially in a micro-service architecture, but I prefer to mention it because it could be a potential solution if you don’t have any Edge tool or if you are working on a monolithic app.

Observability, Monitoring and Alerting

Last but not least, it’s very important that you have proper observability in place. Any bot attacks that start to get heavy and put high pressure on your system should trigger a set of alerts automatically.

At the Edge level:

• Alert on the bot rules when a rule created recently is blocking too much traffic or, on the opposite, when a rule has not blocked any traffic for the last ‘n’ months and can be reviewed for potential deletion.
• Alert when the bot traffic (good or bad) is much higher than usual.

At the Application and Infrastructure level:

• Alert on auto-scaling and number of instances. E.g. when Kubernetes is spinning up 5 new pods in the last 5 minutes and it’s not Black Friday, there is probably something fishy…
• Alert on response time and status when the service starts to respond slowly and/or with errors. I recommend you to read Creating Monitoring Dashboards, which covers all you need to know about monitoring.
• You can also set up some alerts at the log levels. This can be useful if you are missing some metrics in your application, and in case you are using an advanced log management tool like Splunk.

The last word

I hope this article was useful and you now have better knowledge of bots and bot attacks.

I did not discuss any silver bullet tool here, because there is no such thing as a single, perfect anti-bot tool for everyone. But there is always room for improvement to prevent and mitigate bot attacks.

Ah… And good luck for the next one!

The article Bot Attacks: You are not alone… was written by Fabian Piau.

This is the third article in our series dedicated to Flagger. In a nutshell, Flagger is a progressive delivery tool that automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics and running conformance tests.

Make sure you have a local Kubernetes cluster running with the service mesh Istio. If you don’t, read the first article: Flagger – Get Started with Istio and Kubernetes. You also need to be familiar with Flagger and MHS, you will find all details in the second article: Flagger – Canary deployments on Kubernetes.

In this third guide, we will focus on the installation of Grafana for Flagger and how you can monitor your Canary deployments without having to use Kubernetes (i.e. Kube dashboard or kubectl command line tool).

Installing Grafana

Flagger provides a Grafana dashboard out of the box to monitor all the canary deployments inside your cluster.

Let’s install Grafana in the namespace istio-system with a single command:

Reference: Flagger monitoring
Flagger depends on Istio telemetry and Prometheus (I’ve assumed Istio is installed in the istio-system namespace).

After a few seconds, you should get a message confirming that Grafana for Flagger has been installed. From the Kubernetes dashboard, verify that the Flagger Grafana pod is running in istio-system.

Grafana for Flagger is deployed in your cluster

To expose Grafana, run:

So you can access it with your browser at http://localhost:3000/d/flagger-istio/istio-canary. Use the login and password you specified before (admin / changeme).

Note that as we use Istio, we use the Istio Canary dashboard. Flagger is compatible with other service meshes and there are other dashboards available.

Running the canary deployments

If you have followed the previous article, make sure you select application for the Namespace and associate mhs-primary to Primary and mhs to Canary.

Then try again the different experiments we did in the previous article and monitor the Grafana dashboard at the same time, especially:

• Experiment 1 – MHS v1.1.2 successful deployment
• Experiment 2 – MHS v1.1.3 faulty deployment

Results

Look at this screenshot I took during Experiment 1 when the canary release was successful and the new version rolled out. I annotated it and added explanations, so it’s easier to understand the graphs.

A successful Canary deployment

And this is a similar screenshot for Experiment 2 when the canary release did not succeed and the new version was not rolled out.

A failed Canary deployment

Congratulations, you’ve come to the end of this third tutorial!

Cleaning up resources

You can delete the MHS application and its namespace. We can also remove Istio and Flagger because this is the last article of the series.

You can also stop the Kubernetes cluster by unchecking the box and restarting Docker Desktop.

The article Flagger – Monitor your Canary deployments with Grafana was written by Fabian Piau.

This article is the second one of the series dedicated to Flagger. In a nutshell, Flagger is a progressive delivery tool that automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics and running conformance tests.

Make sure you have a local Kubernetes cluster running with the service mesh Istio. If you don’t, read the first article: Flagger – Get Started with Istio and Kubernetes.

In this second guide, we will focus on the installation of Flagger and run multiple canary deployments of the application Mirror HTTP Server (MHS). Remember that this dummy application can simulate valid and invalid responses based on the request. This is exactly what we need to test the capabilities of Flagger. We will cover both happy (rollout) and unhappy (rollback) scenarios.

Installing Flagger

Let’s install Flagger by running these commands.

We install Flagger in its own namespace flagger-system.

Reference: Flagger Install on Kubernetes
Flagger depends on Istio telemetry and Prometheus (in that case, we assume Istio is installed in the istio-system namespace).
All parameters are available on the Flagger readme file on GitHub.
We don’t specify a version for Flagger, which means it will use the latest available in the repo (1.2.0 at the time of writing).

After a few seconds, you should get a message confirming that Flagger has been installed. From the Kube dashboard, verify that a new namespace has been created flagger-system and the Flagger pod is running.

Flagger is deployed in your cluster

Experiment 0 – Initialize Flagger with MHS v1.1.1

Mirror HTTP Server has multiple versions available. To play with Flagger canary deployment feature, we will switch between version 1.1.1, 1.1.2 and 1.1.3 of MHS (the latest version at the time of writing).

Before deploying MHS, let’s create a new namespace application, we don’t want to use the default one at the root of the cluster (this is good practice). The name is too generic, but sufficient for this tutorial, in general you will use the name of the team or the name of a group of features.

Do not forget to activate Istio on this new namespace:

To deploy MHS via Flagger, I created a Helm chart.

This “canary flavored” chart was created based on the previous chart without Flagger which itself was created with the helm create mhs-chart command, then adapted. In this “canary flavored” chart, I did some extra adaptation to use 2 replicas instead of 1 to make it more realistic and use a fixed version to 1.1.1, I also added the canary resource where the magic happens.

Clone the chart repo:

And install MHS:

After a few moments, if you look at the dashboard, you should see 2 replicas of MHS in the namespace application.

MHS 1.1.1 is deployed in your cluster

It is important to note that no canary analysis has been performed and the version has been automatically promoted. It was not a “real” canary release.
Why? Because Flagger needs to initialize itself the first time we do a canary deployment of the application. So make sure the version you are deploying with Flagger the first time is fully tested and works well!
You could also guess this auto-promotion happened because there was no initial version of the application in the cluster. Although this is obviously a good reason, it’s important to note that, even if we had a previous version before (e.g. 1.1.0), the canary version 1.1.1 would have still been automatically promoted without analysis.

You can still check the canary events with:

You should have a similar output without a canary analysis:

Or you can also directly check the log from Flagger:

If you take a closer look at the Kube dashboard, you should see some mhs and mhs-primary resources:

• mhs-primary are the primary instances (= the non-canary ones). Flagger automatically add the -primary suffix to differentiate them from the canary instances.
• mhs are the canary instances. They exist only during the canary deployment and will disappear once the canary deployment ends. That’s why, in the screenshot above, you don’t see any mhs canary pods (i.e. 0 / 0 pod).

Why this naming convention? I asked Flagger team directly and there is a technical constraint.

Flagger is now initialized properly and MHS is deployed to your cluster. You can use the terminal to confirm MHS is accessible (thanks to the Istio Gateway):

You should receive an HTTP 200 OK response:

And:

should return an HTTP 500 response:

Experiment 1 – MHS v1.1.2 canary deployment

We are going to install a newer version 1.1.2. You need to manually edit the file mhs-canary-chart/mhs/values.yaml and replace tag: 1.1.1 with tag: 1.1.2 (this line).

Then:

While the canary deployment is in progress, it’s very important to generate some traffic to MHS. Without traffic, Flagger will consider that something went wrong with the new version and will rollback automatically to the previous one. Obviously, you don’t need this extra step in a production environment that continuously receives real traffic.

Run this loop command in another terminal to generate artificial traffic:

Check the Kube dashboard, you should see the canary pod with the new version 1.1.2 at some point:

Canary deployment of MHS 1.1.2 in progress in your cluster

Check the canary events with the same command as before:

After a while (about 6 minutes) you should have a similar event output:

The canary release performed successfully. Now you have version 1.1.2 installed on all the primary pods and the canary pod has been removed.

MHS 1.1.2 is deployed in your cluster

Why did this deployment take about 6 minutes? Because it includes a 5 minutes canary analysis. During this analysis, the traffic was routed progressively to the canary pod. The canary traffic increased by steps of 10% every 1 minute until it reached 50% of the global traffic. The analysis is configurable and defined in the canary.yaml file that was added to the chart.

Below is the analysis configuration:

  analysis:
    # stepper schedule interval
    interval: 1m
    # max traffic percentage routed to canary - percentage (0-100)
    maxWeight: 50
    # canary increment step - percentage (0-100)
    stepWeight: 10
    # max number of failed metric checks before rollback (global to all metrics)
    threshold: 5
    metrics:
      - name: request-success-rate
        # percentage before the request success rate metric is considered as failed (0-100)
        thresholdRange:
          min: 99
        # interval for the request success rate metric check
        interval: 30s
      - name: request-duration
        # maximum req duration P99 in milliseconds before the request duration metric is considered as failed
        thresholdRange:
          max: 500
        # interval for the request duration metric check
        interval: 30s

The canary analysis has been covered with the 2 basic metrics that are provided out of the box by Istio / Prometheus (request success rate + duration). It is possible to define your own custom metrics. In that case, they will need to be provided by your application. Your application will need to expose a Prometheus endpoint that includes your custom metrics. And you will be able to update the Flagger analysis configuration to use them with your own PromQL query. Note this goes beyond the scope of this hands-on guide that uses only the built-in metrics.

Experiment 2 – MHS v1.1.3 faulty deployment

Again, you need to manually edit the file mhs-canary-chart/mhs/values.yaml and replace tag: 1.1.2 with tag: 1.1.3.

Then:

We generate some artificial traffic:

This time, we also generate invalid traffic to make sure the request success rate is going down!

Check the canary events with the same command as before:

After a while (about 6 minutes) you should have a similar event output:

And you are still on version 1.1.2.

Flagger decided not to go ahead and propagate version 1.1.3 as it could not perform a successful analysis and the error threshold was reached, i.e. 5 times (indeed, each time, about 50% of the requests were ending up in an HTTP 500 response). Flagger has simply redirected all traffic back to the primary instances and removed the canary pod.

Congratulations, you’ve come to the end of this second tutorial!

Observations

Before we clean up the resources we’ve created, let’s wrap up with a list of observations:

• Deleting a deployment will delete all pods (canary / primary). And we don’t end up with orphan resources.
• Prometheus is required. Without it, the canary analysis won’t work.
• It is not possible to re-trigger a canary deployment of the same version if it has just failed. It forces you to bump up the version (even if it was a configuration and not a code issue).
• Flagger off-boarding process is not as simple as removing the canary resource from the chart and deploy a new version. If you delete the canary resource then Flagger won’t trigger the canary process, it will change the version in mhs and remove mhs-primary but mhs has 0 pods so it will make your service unavailable! You need to be careful and adopt a proper manual off-boarding process. Recently, the Flagger team added a property revertOnDeletion you can enable to avoid this issue. You can read the documentation to know more about this canary finalizer.
• After multiple deployments, it seems that some events can be missing, the Kubernetes describe command is accumulating them (x<int> over <int>m) sometimes the order is not preserved and/or some events are not showing up. You can look at the phase status (terminal status are Initialized, Succeeded and Failed). The best is to look directly at the logs on the Flagger pod as this is always accurate and complete.
• The canary analysis should be configured to run for a short period of time (i.e. no more than 30 minutes) to leverage continuous deployment and avoid releasing a new version while a canary deployment for the previous one is still in progress. If you want to perform canary releases over longer periods, Flagger may not be the best tool.
• Finally, it’s important to remember that the first time you deploy with Flagger (like in experiment 0 above), the tool needs to initialize itself (Initialized status) and will not perform any analysis.

Cleaning up resources

Now the tutorial is complete you can remove the MHS application and its namespace.

We recommend that you leave Flagger and Istio in place to save time in the next tutorial. If however you’d like to remove everything now, then you can run the following commands.

Remove Flagger:

Remove Istio and Prometheus:

What’s next?

The next article will focus on the Grafana dashboard provided out of the box with Flagger which is a nice addition, so you don’t need to manually run any kuberctl commands to check the result of your canary deployments. Stay tuned! In the meantime, you can stop the Kubernetes cluster by unchecking the box and restarting Docker Desktop. Your computer deserves another break.

The article Flagger – Canary deployments on Kubernetes was written by Fabian Piau.

This series of articles is dedicated to Flagger, a tool that integrates with Kubernetes, the popular container orchestration platform. Flagger enables automated deployments and will be one step closer to a continuous deployment process.

This article is the first of the series and also the only one where we won’t use Flagger yet… this article will walk through how you to run a Kubernetes cluster on your local environment and deploy an application which will be accessible via an Istio gateway.

Docker

Install Docker by installing the Docker Desktop for Mac application, you can refer to the official installation guide. For Windows users, the equivalent application “Docker for Windows” exists.

In the next part, we will also use Docker for Mac to set up our local Kubernetes cluster. Note that this tutorial has been tested with Docker for Mac 2.4.0.0 that includes a Kubernetes Cluster in version 1.18.8, this is the latest at the moment of writing.

If you use a different version, technology is moving fast so I cannot guarantee that the commands used in this series will work without any adjustment.

Mirror HTTP Server

First a few words about the application Mirror HTTP Server we will use in this series of articles.

MHS is a very simple JavaScript application based on Node.js using the framework Express which allows you to customize the HTTP response received by setting specific HTTP headers in the request. The Docker image is publicly available on the Docker Hub. You can consult the Github repo of the project to find out more, please note that I am not the author.

This little app is exactly what we need to test the capabilities of Flagger to simulate 200 OK responses and 500 Internal Server Error responses.

Let’s pull the Docker image:

And run a new container that uses it:

Then let’s make sure it is functioning properly:

You should receive an HTTP 200 OK response:

While:

will return an HTTP 500 response:

For simplicity, we use the curl command, but you can use your favourite tool, e.g. Postman.

Kubernetes

Now that you’ve installed Docker for Mac, having a Kubernetes cluster running locally will be a simple formality. You just need to check a box!

Enable Kubernetes with Docker for Mac

If the light is green, then your Kubernetes cluster has successfully started. Please note, this requires a significant amount of resources, so don’t panic if the fan is running at full speed and it takes a bit of time to start…

Kube dashboard

We will install our first application in our Kubernetes cluster.

Kubernetes via Docker does not come with the dashboard by default, you have to install it yourself. This dashboard is very practical and provides a graphical interface of what is going on in your cluster and will save you from having to enter kubectl commands.

The dashboard is protected, but you can use the default user to access it. You can generate a default token via this command:

Copy it.

You will need to re-use this command and /or the token copied if your session has expired, this happens when you don’t interact with the dashboard for a little while.

Finally, create a proxy to access the dashboard from the browser (this command will need to run indefinitely):

If you access http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login and use the token that you copied to authenticate, you should see this screen.

Kube Dashboard

Helm

We use Homebrew for the installation of Helm. Homebrew is a handy package manager available for Mac.

We will use Helm to install Istio and the MHS application in our cluster. Helm is a bit like Homebrew, but for Kubernetes. We are using version 3. Helm will save you from having to enter many kubectl apply commands.

Let’s install Helm 3 with:

To verify that Helm has been installed:

You should have a similar output (note that Helm 3.3.4 is the latest version at the time of writing):

Istio & Prometheus

Now, we are going to install the Istio Service Mesh. For full explanations and the benefits of using a Service Mesh, I invite you to read the official documentation.

First of all, you must increase the memory limits of your Kubernetes via Docker, otherwise you will run into deployment issues. Your laptop’s fans will recover, don’t worry…

Here is my configuration:

Kubernetes Configuration in Docker for Mac for Istio

I followed the Docker Desktop recommendations for Istio.

Let’s go and install Istio 1.7.3 (the latest version at the time of writing). First, download the source:

Add the istioctl client to your path:

Install Istio with the provided client, we use the demo profile:

After a few minutes, you should get a message confirming that Istio has been installed. And voilà!

To install the latest version of Istio, you can simply replace the first line with curl -L https://istio.io/downloadIstio | sh -.

Add Prometheus as it’s required for Flagger:

From the Kube dashboard, verify that a new namespace has been created istio-system and that it contains the Istio tools including Prometheus.

Istio is deployed in your cluster

Why is Prometheus important? Because it is an essential component for Flagger which will provide the metrics to show if the new version of your application is healthy or not, thus it will know when to promote or rollback a version. I will come back to this in detail in the next article.

Deploying Mirror HTTP Server

Before deploying MHS, let’s create a new namespace application, we don’t want to use the default one at the root of the cluster (this is good practice). The name is too generic, but sufficient for this tutorial, in general you will use the name of the team or the name of a group of features.

Do not forget to activate Istio on this new namespace:

To deploy MHS, I created a Helm chart.

This chart was created with the helm create mhs-chart command, then I updated to use the latest image of MHS. I also added a gateway.yaml file to configure the Istio gateway so it can be accessible outside of the cluster.

Clone the chart repo:

And install MHS:

After a few moments, if you look at the dashboard, you should see 1 replica of MHS in the namespace application.

MHS is deployed in your cluster

You now have 1 MHS pod running in your Kubernetes cluster. The pod is exposed to the outside world via an Istio gateway.

To test, use the similar commands that we used against the docker container earlier:

You should receive an HTTP 200 OK response that was handled by Envoy, the proxy used by Istio:

And:

should return an HTTP 500 response:

Congratulations, you’ve come to the end of this first tutorial!

For information, you can also access MHS with your favourite browser if you run a proxy command first to expose the pod:

export POD_NAME=$(kubectl get pods --namespace application -l "app.kubernetes.io/name=mhs,app.kubernetes.io/instance=mhs" -o jsonpath="{.items[0].metadata.name}")

kubectl port-forward --namespace application $POD_NAME 8080:80

Then, navigate to http://localhost:8080/.

You should see a… blank page. This is normal, MHS does not return a body in the response and there is no HTML output!

Cleaning up resources

You can delete the MHS application and its namespace.

We don’t remove Istio / Prometheus because we will need it in the next article, but if you want to free up some resources, you can use these commands:

What’s next?

The next article will focus on the installation of Flagger and use different versions of MHS to try canary deployments. Stay tuned! In the meantime, you can stop the Kubernetes cluster by unchecking the box and restarting Docker Desktop. Your computer deserves a break.

The article Flagger – Get Started with Istio and Kubernetes was written by Fabian Piau.

Last Wednesday, Expedia organized the first CoderDojo in our London office.

CoderDojo is a volunteer-led community group for young people aged 7 to 17 to come together and learn to create cool projects with technology.

It’s free to come along and all experience levels are welcome; whether you’re new to coding or have a project idea that you want to make reality!

For this first session, we had about 10 participants aged between 7 and 13 years old.

As an ice breaker, we did the “two truths and a lie” game mentioning 3 things we had done during that day. I was quite glad that most kids fell into my trap when I said that I played PlayStation at work, and they all thought it was the lie… :)

We organized different activities depending on what the kids were interested in:

• Scratch, using drag & drop blocks instead of traditional code, to make a hippo fly or build more complex projects
• Learn basic algorithm skills based on Star Wars or Flappy Bird games
• Use of the BBC Micro:Bit to create games or simulate robot emotions

CoderDojo @ Expedia London

CoderDojo @ Expedia London

At the end of the activity, the kids were invited to show what they achieved over the last hour. It was nice to see the majority of them really happy & enthusiastic to tell us about what they built.
Sharing knowledge and collaboration are very important soft skills.

In summary, it was fun for the kids but also the mentors!

CoderDojo Mentors: Veli, Alice, Adam, Ana, Fabian, Nicola

After receiving spontaneously good feedback from some participants at the end, we can say this first session was a success. We obviously have room for improvement and already got some new ideas for the next session.

We are planning to have a monthly Dojo, the 3rd Wednesday of each month, which means the next one will be on August, 21st.

Whether you are a kid, a parent or a mentor, we hope to see you soon at the second one! More info and registration at London Angel Expedia CoderDojo.

The article Expedia CoderDojo in London was written by Fabian Piau.

Devoxx4Kids is a spin-off of the Devoxx developer conference, with some differences: the event lasts one day, does not include any conferences, but focuses on workshops and, of course, is designed for kids (ages 10-14).

Last year, I volunteered for Devoxx4Kids London and it was lots of fun. I like IT and I like kids, so it’s a good motivation! Playing with Scratch, making Nao the robot dance & talk, programming some Lego Mindstorms for battles, etc. You almost end up being a kid yourself!

It’s a very good initiative to introduce basic engineering and programming skills to the youngest generation so they discover what IT is in a good and practical way. This experience can also teach them some soft skills like patience, sharing and listening.

Devoxx4Kids 2018 in London - Collage

Devoxx4Kids is also a good opportunity to teach kids that tech is not something only for boys, and girls can take their part too. As they are young, they are much less concerned with stereotypes. As a reader of this blog and probably an adult, you won’t be surprised that someone working in IT can be associated to a bearded nerdy guy typing lines of code in the dark with his coffee & a sexy wallpaper…

Joke apart, Devoxx4kids will take place next Saturday (on May 11th) in London. Unfortunately, I won’t be able to attend and help this year, I will have to give the Sphero BOLT and the Harry Potter coding wand a miss. But don’t hesitate, if you are interested and free that day, I would say it’s never too late to volunteer. You can find all the details on the event page or reach them via Facebook or Twitter.

If you are not living in London or even in UK, there are plenty of Devoxx4Kids around the world, get in touch with the one near your home, I’m sure they will be happy to have some extra help!

Devoxx4Kids 2018 in London - Group photo

The article Volunteering at Devoxx4Kids was written by Fabian Piau.

This post summarizes the work we have achieved within my team to migrate our micro-services from Java 8 to Java 11 for the website Hotels.com.

In summary, for each of the services we own, we have made the following steps:

• Make the code compile with Java 11
• Run the Java 11 compatible service on Java 8
• Run the service on Java 11

In reality, we had some extra steps because when we began the migration, Java 11 was not released yet, we could only use Java 10.
The assumption was if the code compiles on Java 10, there will not be so much work to migrate to Java 11 as the biggest change about modularity was introduced with Java 9 and the Jigsaw project. Thankfully, it was the case!

Star Wars lightsaber upgrade...

1. Make the code compile with Java 11

This was the longest part. Indeed, we had to bump the version up of most of the frameworks and tools we are using. Especially, we had to handle the migration from Spring Boot 1 to 2 and Spring 4 to 5. As these are major versions, we had to fix a couple of breaking changes.

Spring Boot

For Spring Boot 2, the official migration guides for Spring Boot 2.0 and Spring Boot 2.1 are well written and detailed.

• The profile loading has evolved
• The relaxed binding of properties is a bit less relaxed
• Some properties were renamed and others made unavailable (e.g. security.basic.enabled property that has to be replaced with a WebSecurityConfigurerAdapter)
• Some endpoints were renamed (e.g. the actuator healthchecks)
• The bean overriding is now disabled by default, which is something we were using in our integration tests, we had to re-enable it with the new property spring.main.allow-bean-definition-overriding.

Spring

The migration to Spring 5 was quite straightforward from a code point of view with few minor changes. The hard part was related to the fact the project is legacy and we had to deal with complex Spring XML configuration and the migration to Dropwizard Metrics 4.

Misc

Few frameworks are still not compatible with Java 9+ as they are not actively maintained by the community. In our case, we had to find a workaround for Cassandra Unit. We did not want to invest time to change of testing framework as we are planning to move to DynamoDB.

We also had to deal with the Maven dependency hell because some required dependencies were bringing old dependencies not compatible with Java 9+. In most of the cases, adding some exclusions in the POM solved it.

Local environment

Locally, we added a simple set of aliases to our bash profile file to switch between the Java versions.

alias setjava8="export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/"
alias setjava10="export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents/Home/"
alias setjava11="export JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-11.jdk/Contents/Home/"

On IntelliJ, changing of Java version can be done from the project settings (“Project SDK” dropdown).

CI environment

For the Continuous Integration side (we are using Bamboo), we updated the agent to use Java 11.
We noticed that it’s not possible to have different versions of the agent for the branch plans and the master plan as the plan configuration is global. It means updating the agent to Java 11 will break master if someone else was pushing a change to master (e.g. a new feature or a bug fix totally independent from the Java 11 migration).
To mitigate this issue and avoid a red build, it was important for us to make sure the project was compiling with Java 11 and that all tests were passing locally before updating the agent in order to merge the Java migration pull request quickly. Another option would be to set back the agent to Java 8 temporarily once the branch plan was green on Java 11 without forgetting to set it back to Java 11 just before the merge.

2. Run the Java 11 compatible service on Java 8

Once everything was fixed, merged and the master build was green, we had to ensure the Java 11 compatible version was running fine in our test environments. Basically make sure nothing was broken… We had unit, integration and end-to-end tests so our level of confidence was quite high. Just to be safe, we did some extra exploratory and manual testing on the API with some exotic and edge case requests to make sure it was behaving correctly. We also ensure the logs and Grafana dashboards were fine.

The next step was to push the new version in production. The service was still running with Java 8 even if the code was Java 11 compatible (and compiled), we did not want to introduce too many changes at the same time, we don’t like risky releases after all. We handle this release with extra care because of the multiple refactoring and versions bump up. After looking at the Grafana dashboards for a few days, comparing metrics before and after the migration, it shows that all went well.

3. Run the service on Java 11

The ultimate goal was to run the service on Java 11. In theory, it will be as simple as updating the Docker file to use the Java 11 image and push the artifact in production. However, in practice it was not that simple…

First of all, we had to update the Java JVM arguments (the -d64 parameter is deprecated and will prevent the service to start, we also had to update the GC logs argument).

Then, we quickly realized that the service logs had disappeared from Splunk in our on-premises production environment, the logs were actually showing up in the future while it was working fine on AWS. We had to update the logback config to fix this “temporal distortion” ;) by updating the date pattern from %d{ISO8601} to %d{yyyy-MM-dd'T'HH:mm:ss.SSSZ,UTC}.

We had another weird error VerifyError: Bad type on operand stack that arises during the deployment to production, AppDynamics was preventing some instances from starting due to some exotic bytecode manipulation. For some reason, it was fine on prod-canary, then started to fail after a successful deployment on a couple of instances! We had to disable AppDynamics, which was fine as we are not using this tool in our team.

As we were moving to Java 11, we also had to update some of our Grafana dashboards to reflect the use of a new Garbage Collector – G1.

Conclusion

Today, 3 services providing the user notifications using Spring Boot 2, and 1 service providing the website header & footer using Spring 5 are running smoothly on Java 11, it has been several weeks now. They are using the default G1 Garbage Collector and we did not encounter any weird behavior related to memory footprint or any other performance issue. On the other hand, we did not see any improvement in our response time. But now we are using a Java LTS (Long Term Support) release, the migration was a success.

What’s next? Java 12 is going to be released in March 2019. At this time, we still don’t know if we will use this version or wait for the next Java LTS. It will probably depend on which features are included.

The article A Java 11 migration successful story was written by Fabian Piau.

WordPress is one of the most popular CMS (Content Management System). That popularity also means that it is a target of choice for hackers.
In this article, I will give you some tips to keep your website secure and avoid being attacked.

1. Use latest versions

This is true for WordPress itself but also for all your extensions. There are new versions available regularly. If a plugin has not been updated for a while, it is probably not maintained anymore and you might need to remove or replace it. This is also applicable for your theme.
The version of PHP is also important, check with your hosting provider that you are running the latest version of PHP (7.X), especially the versions 5.X won’t be supported by the end of the year.
Also, note that the more extensions you have installed, the more risk you are taking, as your WordPress configuration will rely on more 3rd party code. You should only keep the plugins that you really need. If a plugin is disabled, don’t keep its source code and remove all its associated files.

2. Use secure login details

Never use the default admin user. If you do, disable this account and create your own account with a personalized username.
Choose a strong password. If several users are managing your website, make sure the permissions are valid and avoid giving the admin permission to everyone.

3. Scan your website

This is an easy and quick way to find vulnerabilities and see if one of your plugins is vulnerable or not. You can use these 2 online tools:

• WordPress Security Scan (my favourite with a detailed report)
• WPSec

4. Use .htaccess files to protect your directories

The .htaccess file is a server configuration file. It allows you to define rules for your server to follow.

For example, in /wp-content/uploads, I have created the following .htaccess:

# Deny access to everything by default
Order deny,allow
Deny from all

# Allow access to media files
<FilesMatch '\.(jpg|jpeg|png|gif|bmp|zip|rar|pdf)$'>
    Allow from all
</FilesMatch>

This config ensures only media files are accessible from the browser, any JavaScript, PHP files will be discarded. It is not 100% bulletproof as only the extension is checked, but it is better than nothing.

To avoid execution of malicious PHP in some folder (e.g. in /wp-includes), you can create another .htaccess file with the following content:

<Files *.php>
Order allow,deny
Deny from all
</Files>

5. Review file and directory permission

Make sure the critical files (wp-config.php, php.ini…) are not writable publicly, only readable. Only owners should be able to write.

6. Use security headers

You can check which security headers you currently use with this online tool.

At the root folder, update the .htaccess file and add:

# Extra Security Headers
<IfModule mod_headers.c>
	Header set Strict-Transport-Security 'max-age=31536000; includeSubDomains'
	Header set X-XSS-Protection '1; mode=block'
	Header set X-Frame-Options 'sameorigin'
	Header set X-Content-Type-Options 'nosniff'
	Header unset Server
	Header always unset X-Powered-By
	Header unset X-Powered-By
	Header unset X-CF-Powered-By
	Header unset X-Mod-Pagespeed
	Header unset X-Pingback
</IfModule>

In wp-config.php, add:

/** Extra Security */
header('X-Frame-Options: SAMEORIGIN');
header('X-XSS-Protection: 1; mode=block');
header('X-Content-Type-Options: nosniff');
header('Strict-Transport-Security:max-age=31536000; includeSubdomains; preload');
header('Referrer-Policy: no-referrer-when-downgrade');
header('Content-Security-Policy: upgrade-insecure-requests');
header('Permissions-Policy: autoplay=(), camera=(), encrypted-media=(), fullscreen=(), geolocation=(), microphone=(), midi=(), payment=()');
header_remove('X-Powered-By');
header_remove('Server');
header_remove('X-CF-Powered-By');
header_remove('X-Mod-Pagespeed');
header_remove('X-Pingback');
@ini_set('session.cookie_httponly', true);
@ini_set('session.cookie_secure', true);
@ini_set('session.use_only_cookies', true);

7. Do not expose too much information

At the root folder of your website, in php.ini, add the line:

expose_php = Off

Your current version of PHP will not be exposed.

8. Backup your website regularly

Last but not least! You don’t need a particular software or extra plugin to achieve this.

• With your favourite FTP tool (e.g. Filezilla), save all the files available on your server.
• For the database, use the available MySQL backup feature. Many hosting companies provide access to phpMyAdmin, an online tool.

I recommend doing a backup every month, and keep the history of the last 6 backups somewhere safe. Of course, it depends on the volume of articles you are writing and how critical is your data.

That’s it! If you have done all the above, your website should be more resilient to attacks. In the worst case, you should be able to recover easily.

Happy safe blogging!

The article Tips to make your WordPress website secure was written by Fabian Piau.

This year I attended the 2 days of conference at Devoxx UK taking place in London on 10-11th May. This article is a summary of the notes I took during the second day. You can read the previous article talking about my first day. And if you’re interested in more details in a talk, you can watch the associated video.

Devoxx UK took place in the Business Design Centre in London

Deep Learning: The Future of Artificial Intelligence, with Matthew Renze

In the past, we had to explicitly program a computer step by step to get it to solve a problem (involving if-then statements, for loops and other logical operations). In the future, machines are going to teach themselves how to solve a problem on their own, we just have to provide the data.

What is Deep Learning?

Deep Learning is a form of Artificial Intelligence (AI) that uses a type of Machine Learning (ML) called an Artificial Neural Network with multiple hidden layers in attempt to learn hierarchical representations of the underlying data in order to make predictions given new data.

Machine Learning in essence is the application of Statistics to the problems of Artificial Intelligence. We are teaching machines how to solve problems by identifying statistical patterns in data.

Use (existing) Data -> to learn a Function -> in order to do a Prediction (on new data)

A neural network is a ML algorithm based on a very crude approximation of the way how we used to believe the brain and the neurons are working (the brain is still a black box for scientists).

An artificial neuron takes a set of inputs, applies a function to produce a set of outputs. We represent this neuron mathematically (i.e. inputs and outputs are numbers) so we can use it in a computational model.

A neural network is composed of several neurons organized into different layers: the input layer (the data we feed), 1 or more hidden layers and the output layer (the prediction). A deep neural network contains more than one hidden layer. Adding more than one hidden layer essentially allows us to model much more complex function than a simple single layer.

An example of Deep Learning

An example of deep neural network is a person recognition model based on a picture. The lower hidden layers represents more abstract shapes like geometric primitives (e.g. horizontal, vertical lines) while the intermediate layers are representing more complex features like specific parts of the body (mouth, nose, eye, etc.) and the highest hidden layer will represent the face and ultimately be able to identify the person. The accuracy is increasing as we are approaching the last layer and ultimately make a prediction.

Why are we talking about Deep Learning only now?

After all, the first ML algorithm Perceptron was created in 1957, more than 60 years ago!

• We live in the era of Big Data. In the past 2 years, we have created more data than the entire rest of human history. We never had so much data available, this data is essential to train complex models.
• Computers have never been so powerful: faster CPUs, more memory, solid-state drive. We can leverage the power of GPUs, matrix operations is something needed for video game graphics but also for ML. We also have access to distributed computing technologies where we can share the data processing between multiple machines.

What can we do with Machine Learning?

• Classification. We want to predict a discrete variable that can only take on a certain number of values. Is it a cat or a dog? Is this email a spam or not? Does this person have a cancer or not? What is the category of this article?
• Regression. We want to predict a continuous variable that has an infinite number of possible values. At what price should I sell this house? What is the credit score of this person?
• Text generation. Generate the title of an article, the description of an image based on its content, convert voice to text for automatic subtitles.
• Image generation. Simulate face aging, paint a new Rembrandt that even expert could not identify as a fake, create celebrities that look familiar but don’t exist, create an image based on a description.
• Audio generation. Make an algorithm talk to a human on a phone call (Google Duplex), change your voice based on a text (voice editing) to avoid recording your voice again in case you got it wrong.
• Video generation. Based on rushes of videos and audios from Barack Obama, create a new video with an AI-powered lip sync.

If you have browsed some of the examples above, you will probably find very challenging to differentiate between a real and a computer-generated content. That is quite scary when you think about it and we can barely imagine how it will be in 10 years!

How do I get started if I want to do ML in my company?

Option 1 – Deep Learning as a Service (Google Cloud, AWS, Microsoft Cognitives, IBM Watson…)
It involves a 3rd party company that owns the model and the data. Basically, you can query their API with your new data in order to make a prediction.
This option is good for narrow use cases, if you don’t want to reinvent the wheel and use an already trained model.
Pro: Simple, quick, inexpensive
Cons: Narrow, remote (e.g. you are not protected against network failures or latency), pay-per-use (if your usage increases, so will be the cost)

Option 2 – Deep Learning platform (Microsoft Azure, Cognitive Services…)
This option is good for custom use cases. You upload your data to train the model (transfer learning) then you can query an API to get your predictions.
Pro: Simple, quick, inexpensive
Cons: You need training data to train your model, remote, pay-per-use (per transaction for the prediction but also the transactions for the training)

Option 3 – Do it yourself (TensorFlow, Torch…)
To use if option 1 and 2 fail. You create from scratch your own algorithm, provide the data and host it yourself.
Pro: Custom (you can tune it as you like), local, private (in case you are dealing with sensitive data)
Cons: Complex, labor, expensive

Teaching kids about machine learning, with Dale Lane

Dale has created a website to teach kids about Machine Learning using the visual programming language platform Scratch. Scratch is a way to introduce programming to kids with a simple interface where you can drag and drop blocks representing programming operations (for loops, if-then statements…) and assemble them to create more advanced logic. Dale is using ScratchX (experimental extensions for Scratch) and has created new blocks related to Machine Learning (using IBM Watson under the hood).

He has invented many exercises so the kids can grasp the different concepts of ML using concrete and fun examples. The exercises are mixing text, number and image recognition. Each exercise is usually composed of a phase when you train the model from the web application, then you can evaluate the prediction with Scratch. You can also refine the training set and see the effects on the model predictions.

• Create a chatbot, e.g. an animal that can answer questions about its species.
• Predict the newspaper an article was extracted from based on its title.
• Make Pacman avoid ghosts by playing the game multiple times. The more you play and train the model, the better Pacman gets.
• Categorize images. Uploading images of cups and cars to train the model, so it can classify a new and unknown image to one of these 2 categories. Uploading images of book cover for different categories (sci-fi, romance, thriller…) then the model can predict the category of a new cover. To make it more interactive you can also upload images from the webcam, for example take pictures of your hand for the game rock / paper / scissors and play against the computer. The algorithm will recognize the shape you are doing with your hands if you train it enough.
• Recognize handwritten postcode to figure out which city the mail should be delivered to. That’s a good real-life example.
• Play where is Wally-like games. The computer automatically detects where Scratch the cat is located in a picture.
• And many other exercises.

Dale also mentioned the problems of AI:

• Issue with background, weather conditions. If it is obvious for a human, a computer can make silly mistakes if the training set is incomplete.
• The Russian tank story where the model was trained with high resolution pictures of American tanks but low resolution and blurry pictures of Russian tanks. It performs really poorly to identify tanks in reality.
• Google photo can add automatically text to describe a picture, some black people were categorized as gorillas…
• A model that can advise medicine but is sponsored by a pharmaceutical company. What if the company asks to add more references of its product in the training set, is that something right?

There are a lot of important notions around ethics, bias, model overfitting and the quality of the training set. It’s important that kids understand about these by themselves, and they usually do!

At the end, Dale gave use some links to go further:

• Machine Learning for Kids the website of Dale where you can find all the exercises.
• Teachable Machine from Google to train a model with your webcam.
• Quick, Draw! from Google to recognize a drawing.
• Moral Machine from MIT to judge different scenario involving a self-driving car.

Easy Microservices with JHipster, with Sendil Kumar N

This presentation was a live-coding session. I heard about JHipster a while ago but never had the opportunity to see it in action and the fact the project was initiated by a French developer was another reason to go.

JHipster is making the creation of new applications faster by generating the boiler plate code and dealing with most of the configuration (for example the configuration for Kubernetes, Maven POM files, etc.). It is a command-line tool where you can choose to create a monolith application, a microservice or a UAA (User Accounting and Authorizing service for securing your app using OAuth2). Once you have chosen your type of application, there are many options you can choose for your tech stack: frontend, backend, data source, build, logging, deployment CI/CD, service registry, documentation (e.g. Swagger), testing frameworks.

For the demo, Sendil has created a gateway application that was querying a microservice application. He ran the app locally and then deployed it on Google Cloud Platform (GCP) via Kubernetes, without having to type a single line of code.

Decide the frameworks and technologies you want to use will probably be the tough part as there are so many options. Also you have to keep in mind you need to fine-tune the configuration (e.g. the default config they provide for Kubernetes may not suit your needs).

Troubleshooting & Debugging Production Microservices in Kubernetes, with Ray Tsang

Ray shows us a little Guest book application where a user can post a message with a name (Guestbook service) and also get greetings after posting (Hello service). This is a Spring Boot application deployed on Kubernetes, and made of an UI and 2 microservices (each one with multiple instances). A 500 error page was showing up when he tried to access it indicating null and a 404. This time, it was not a demo effect or an illustration of Murphy’s Law and was part of the presentation. He went through the process of debugging the issue step by step using a variety of tools provided by Google Cloud Platform.

Looking at the logs, he was able to identify 2 instances of the UI service that were having a lot of errors compared to the other instances. Kill/restart them is not the solution as it won’t prevent the error from happening again. So he decided to get one of the faulty instances out of the load balancer with one command in kubectl (a command line interface for running commands against Kubernetes clusters) by changing the serving flag to false. The idea is to isolate it from production traffic and be free to debug it. Note that by disabling the instance, Kubernetes automatically spins up a new instance of the service. Then he configured the port forwarding on this pod to be able to query this specific instance locally. With Stackdriver Trace, he was able to see the call tracing. Filtering on 5xx errors and the particular instance, he then realized an issue occurred at the Hello service level where the hello endpoint is called but responded with 404. It seemed to be the case when the name was missing. To confirm the behaviour, he added some logging on the fly to display the name, and indeed it was empty and the app was failing. Basically, some validation was missing on the form to make the name mandatory.

To recap, we can mention the 4 main debugging tools provided with GCP:

• Browsing and querying the logs with Stackdriver Logging (a Splunk-like tool)
• Tracing all the calls with Stackdriver Trace (a Zipkin-like tool)
• Adding debug logs and breakpoints on the fly to a prod instance with Stackdriver Debug
• Having the history of errors in the logs plus various metrics on the services (response time…) with Stackdriver Monitoring

It was a nice presentation but I can just regret that the title was not mentioning the fact the demo will mainly use Google Cloud commercial tools, I thought it will be more focus on Kubernetes. But still worth it to know what Google is providing. I have to admit that playing with breakpoints and adding logs in the code of a remote prod instance, all from a web interface, was quite impressive.

Fully serverless, a case study, with Stephen Colebourne & Chris Kent

At OpenGamma, they have created a new financial platform built on AWS and decided to use serverless technologies (AWS Lambda).

What is serverless and Lambda?

Serverless is basically when the infrastructure is invisible, you don’t know where the code runs and you don’t have control.
An AWS Lambda is basically made of 2 classes: a simple interface and its implementation, where you implement the handleRequest method. The Lambda terminates once the method completes.
You package the code in a JAR (that includes all the needed dependencies) and upload it in AWS to be run.

How can you trigger a Lambda?

• You can use CloudWatch (CRON-like).
• You can use an event. It can be a REST API call, a file is saved in AWS S3, a message arrives in a queue, a row is added to a table or a call directly from another Lambda.

Limits, limits everywhere!

When working with Lambdas, you have to be very careful with the AWS Lambda Limits.

• A default timeout of execution: 5 minutes
• A maximum amount of memory you can use: 3GB
• A maximum size for a JAR file that you can upload: 50Mb
• A maximum of disk space you can use: 512Mb

For some specific heavy processing, they hit most of these limits and they had to find different ways to handle it:

• Use AWS Batch instead which includes serverless features too, the only downside is that they cannot trigger the process instantly, i.e. when you submit a batch job, it will be processed at some point but you cannot assume it will be instant (can be in 2 or even 10 minutes). This limitation was fine for them, obviously this won’t work for everyone.
• Split and pre-process the data as much as possible before passing it to the Lambda. The Lambda should not do any of this time and memory consuming pre-processing. Also as the data is loaded on demand, it introduces some latency but it was acceptable for them.
• Instead of passing a copy of the data between Lambdas which can be memory/network consuming and costly, they save the data in S3 and then use the reference to it (metadata).

They have ended up with 4 services, formed of a total of 23 Lambdas and 2 Batch.

Lessons learned

The automatic scaling with Lambda is the main benefit. It is totally transparent and managed by AWS. When there is a lot of requests, Lambda are created and share the load. When all requests have been processed, the Lambdas are scaling down (to 0 when idle). However, it’s not a silver bullet because you can have a bottle neck at another level of your system (e.g. your data store is too slow) and you will not be able to fully leverage Lambda scalability.

A downside of the Lambdas is the cold start, it takes time to start. In fact, a Lambda is not really destroyed after it completes, as it can be re-used for the next request, AWS has a keep-alive mechanism, which means Lambdas typically live for few minutes after they execute. They decided to call the Lambda with a bit of code every few minutes to keep it available and “warm”. Obviously, it’s a hack and it’s not guaranteed that it will still work in the future, but Chris told us it was quite common…

For the debugging, things are not easy, you need to deploy to AWS to test, it’s hard to debug. Nowadays there are libraries and frameworks to help you do that.

The logs from the Lambdas are going into AWS CloudWatch Logs. This tool is not great for usability so they have created a Lambda to copy logs from AWS to a dedicated log aggregator tool: Sumo Logic, a Splunk-like tool. For the alerts and monitoring, they use CloudWatch Metrics and they have set up alerts in Sumo Logic too that makes it easier to find the logs.

Building from small simple pieces pushes the complexity elsewhere at the infrastructure level (during build time) and also the interactions between the components (during run time). The monitoring and alerting is also more complex as the system becomes even more fragmented. It is important to keep in mind that a Lambda has very restrictive limits, go above one and your Lambda will be killed. Note that the limits are evolving and are lifted overtime. The technology is still young and the tooling and frameworks available is quite limited, but given the level of excitement around Lambda, it’s clear that it’s going to improve.

On the other hand, you don’t need to think about servers and maintaining them. The scaling is transparent and automatically handles the load. You have large potential cost savings using Lambda, for example, in case your system is country-specific, you know that the traffic will be very limited at night hours. Finally, the programming model is very simple, just a JAR file, just a function.

This second day at Devoxx was great, especially the talks in the morning about Machine Learning. I came out of these 2 days with a lot of innovative ideas. Thanks Devoxx UK and maybe see you next year again!

Closing Keynote - Thanks Devoxx UK!

The article Devoxx UK 2018 – Day 2 was written by Fabian Piau.

This year I attended the 2 days of conference at Devoxx UK taking place in London on 10-11th May. This article is a summary of the notes I took during the first day. If you’re interested in more details in a talk, you can watch the associated video.

Devoxx UK took place in the Business Design Centre in London

A Future without Servers, with Danilo Poccia

How to build the best software with the best user experience?

Work backwards from the customer and before starting any implementation:

1. Write a Press Release
2. Write the FAQ
3. Define the Customer experience
4. Write the User manual

The idea is to get your system simple. A complex system was simple at the beginning, it became complex!

Architecture is changing:

• 10 years ago: we are splitting our monolith applications using XML and SOAP for the communication
• 5 years ago: we are creating micro services architecture using REST/JSON or binary protocol for the communication
• Now: we are building event-driven architecture with ephemeral functions

What about the data?

Data repositories are becoming a source of events. Each event is an immutable information about business.

There is a shift from ACID (Atomic, Consistent, Isolated, Durable) to ACID 2.0 (Associative, Commutative, Idempotent, Distributed).

With event-driven design, we think cause / effect instead of triggers: “Service B is caused by A” instead of “Service A triggers B”. We use notification and acknowledgement mechanism.

So what does the future will look like?

We will write only business logic code!

Let’s Get Lazy: Exploring the Real Power of Streams, with Venkat Subramaniam

Venkat Subramaniam is a very good speaker. It was the first time I was hearing him and I was impressed by his way of presenting, I suggest you to watch one of his videos.

Haskell is a lazy language by default. With Scala, it’s possible by using the keyword “lazy”. But what about Java? The keyword “lazy” doesn’t exist but it’s possible with functional code and streams introduced with Java 8.

Imperative code has high ceremony and has accidental complexity. You tell what to do and how to do it.

Functional code has less ceremony and less complexity. You tell what to do. It is very easy to read from top to bottom.

However, if the code is “cute”, it may not be sustainable. So what about performance? E.g. Do we compute all the collection to take only the first one? FindFirst() is the terminal operation of execution. Until we call it, nothing (i.e. all the intermediate operations) will be executed.

Stream does not execute a function for every object in the collection, instead it executes a collection of functions for every object, but only as needed.

Stream is not a collection of objects, it’s a collection of functions.

Lambda are stateless.

List<Integer> numbers = Arrays.asList(1, 2, 3);
Stream<Integer> stream = numbers.stream()
                            .map (e -> e * 2); // This is a lambda
stream.forEach(System.out::println);

Closure carry immutable state, be very careful when using them.

List<Integer> numbers = Arrays.asList(1, 2, 3);
final int factor = 2;
Stream<Integer> stream = numbers.stream()
                            .map (e -> e * factor); // This is a closure
stream.forEach(System.out::println);

Laziness makes the use of infinite streams possible, otherwise the program below would be an infinite loop.

Stream<Integer> infiniteStream = Stream.iterate(0, e -> e + 1);
List<Integer> numbers = infiniteStream 
                          .limit(5)
                          .collect(Collectors.toList());

Kotlin for Java Programmers, with Venkat Subramaniam

I stayed in the same room as I really like Venkat’s first talk. With the same way of presenting, this second talk was very good as well.

As I never experiment Kotlin, it was a nice introduction for me. This JVM-based language is really getting popular these days, especially since Jetbrain is pushing it to be the main language for Android programming. Venkat gave us a lot of different tricks to make the code concise and suggest us to play with it ourselves using the REPL (kotlinc). No doubt it is less verbose than Java and comes with very nice features (including null safety and the lazy keyword…). I will probably give it a try at some point.

How to use AI and Java to train your application to recognize people by name, with Ruth Yakubu

Ruth introduced us Microsoft Face API running on its cloud computing platform Azure. Face API is one of the “cognitive” services Microsoft provides, e.g. there is a service to recognize speech and process natural language.

She showed us an application written with Spring Boot that is interacting with Face API. First, she uploaded a set of pictures of the actor Matthew McConaughey (if you don’t know him, he was the main character in Interstellar) to train the model. Then she uploaded a new picture of him and his wife that the system did not know yet. The algorithm recognizes that it was the actor with a high precision while it did not know who the woman was but it was able to give an accurate description of her (smiley woman in her thirties, etc.).

It is possible to build your own Machine Learning algorithm with Java, for example using the library DeepLearning4J. When creating a model, it is important to separate the data into 2 groups, training data (80%) and test data (20%) so you can verify your model has a good prediction. It is also important to use GPUs not only CPUs to improve the performances, now the libraries are taking advantage of this, including DL4J.

Building a self-driving RC car, with Tim van Eijndhoven

It was an interesting talk to create a self-driving (toy) car based on a Radio Control (RC) kit. They build this prototype as part of a challenge. The idea is to get the car to drive autonomously and follow an itinerary (a path delimited by 2 white lines) with potential curves and obstacles.

If Tesla can do it, why not us? At our scale, of course…

On top of the RC kit, they added a Raspberry Pi, a power convertor/supply, a camera and a safe shutdown (useful when the car gets out of range of the WiFi so it doesn’t crash somewhere…). The total budget is around 300 euros.

Regarding the technologies they use:

• Vert.x, reactive application on the JVM, event driven and non-blocking
• OpenCV library (Computer Vision) to process the video stream in real-time and make sure the car is following the white lines

There are a lot of things to think about, the environment is probably the most challenging. The algorithm can get lost depending on:

• The surface (patterned carpet, tiles, dark road)
• The weather (sunny, rainy, the program is very sensitive to brightness change)
• And other random things (window reflection, mirror effect)

They have many ideas of improvement for the future:

• It is not needed to analyze all images coming from the video stream (especially on a straight line). Currently, they analyze one image every 100ms (why 100ms? because it takes this amount of time to process one)
• It is not needed to analyze the whole image (some part can be discarded, what is above the horizon is not needed)
• Having the computation done on the car itself instead of a laptop over the WiFi, to avoid network latency (however the computational power of Raspberry Pi can be limited)
• Use AI and deep leaning algorithm so the car gets better at navigating using a training set: videos when the car is remotely controlled by a human (however it can take ages to build many tracks and take many videos)

Cloud Native Java, part deux, with Josh Long

Josh is Spring Developer Advocate at Pivotal, it was the first time I was attending one of his talk. I really enjoyed it and I was very impressed by the speed of speech and coding simultaneously without forgetting multiple jokes. Such a brilliant speaker.

Josh used https://start.spring.io/ to generate a little project to manage reservations using Spring Cloud (built on Spring Boot). Why you should use this online tool to initialize your project? Watch the video to get the answer from Josh, it was hilarious!

He chose Kotlin for the service (because why not?) with some endpoints to get messages and reservations using a MongoDB reactive datastore. The service was loading some data in a react way during the start up.

He chose Java for the client using various technology coming out of the box with Spring (Eureka, Spring security, Hystrix for internal load balancer and fallbacks). The client was able to query the service to retrieve the data.

At the end of the demo, he also shows us some serverless architecture using RIFF, a FaaS for Kubernetes. He wrote a function to make a string uppercase, then using the command line to invoke the function deployed in Kubernetes. He did not have the time to show us the call from a service, but we got the idea.

My first day at Devoxx was great, this year I tried to attend more innovative talks about Serverless and Machine Learning, a mix of live coding and theory. I will post my summary of day 2 soon, so stay tuned!

#DevoxxUK letters in the hall

The article Devoxx UK 2018 – Day 1 was written by Fabian Piau.

For this first article of the year 2018, I will present 3 financial services proven and used by myself for several months, even years. They allow me to make significant savings on my foreign currency transactions by avoiding the high fees of traditional banks. This article is in no way sponsored, but I have allowed myself to include some referral links.

For those who know me or follow me, it’s been a while since I live in England, 3 years already! I also travel abroad to discover new countries, new cultures and incidentally escape from the London Fog…

After arriving in England, I quickly needed to open a local bank account where the currency is, of course, not the Euro, but the Pound Sterling.

To deposit money into this account (while waiting for my first pay), I had to transfer money from my French account to my English account. In the pre-Brexit era when the Pound was very strong, it was a little painful… And like if it was not enough, my bank would also take its share with a rather exorbitant fee on my transfer. To avoid being ripped off twice, I looked on the Internet and browsed some forums looking for advice. I quickly decided to choose Wise (previously known as TransferWise).

The principle is simple and based on common sense. British people sometimes need euros, for example when they travel in Europe; conversely, European people need pounds when they come to England. Wise allows you to link these requests by acting as an intermediary. The company has accounts in different currencies and distributes the amounts between people. For example, for a transfer of 1000 euros to a British account (so about 900 pounds at the moment), the system may need 2 people (one person who wants to convert 400 pounds in euros and another who wants to convert 500 pounds in euros) or 3 people (who want to convert 300 pounds in euros each) or 9 people (who want to convert 100 pounds each). I guess you understand how the system works!

Once the account of Wise is registered with your traditional bank (this may take some time depending on your bank) and the transfer has been made from your original account, you will receive the money quickly in your destination account (usually in 1 day).

You will not be charged because your original account and the Wise account are using the same currency. It’s not a big surprise that Wise will charge a fee, but almost insignificant (e.g. 5 euros for a transfer of 1000 euros). You can do a simulation on their site, the fee is proportional to the amount transferred.

More recently, Wise has set up the borderless account. With this “multi-account” you can receive transfers in different currencies in a transparent way, you just need a click to activate a currency and get your corresponding local bank details (IBAN / BIC) that you can then forward to to the person who owes you money. Since April 2018, you will receive a debit card so you can use it to pay anywhere with your borderless account.

It is important to note that the transfer is executed at the market rate. Since you decide when you want to make the transfer, it is wise to do it when the rate is the most advantageous for you. In my case, it was much more interesting to make pound-to-euro transfers before Brexit.

Also in my example, I mentioned the Pound and Euro exchange, but many currencies are supported: the Swiss Franc, the US Dollar, the Japanese Yen, etc.

Feel free to look for yourself and use my referral link to open your Wise account (and your first transfer will be free).

Although they provide a debit card, Wise works best for money transfers between accounts. How can you manage your expenses on the spot when you are travelling around the world? Who has not already paid a high fee when withdrawing abroad or when paying the bill at a restaurant? Who has not already made a large withdrawal at an ATM to avoid fixed costs, taking the risk to walk around with a large amount of money? If your interest is piqued, then carry on reading…

Opening a Revolut account makes perfect sense. It’s an online bank, which means that you won’t find any physical offices, you manage everything yourself from the application on your smartphone: from changing the PIN code to the deactivation of the card, or the change of your address. Revolut is free, you just have to pay a small fee (5 pounds or equivalent in your currency) to receive your multi-currency debit card at home, unless you use my referral link to avoid the fees to receive a card.

You can convert currencies in advance from the application to ensure your exchange rate (advanced use) or it will be automatically calculated in the country according to the current rate and usage of your card (personally, I find it sufficient). The exchange rate is very low and matches the interbank rate (it is therefore a very low rate close to the real one without extra fee).

Revolut strongly advises its customer to keep an account in a traditional bank in case the card is not accepted, it is a Visa so it should not be a problem, but you will probably be happy to have a backup plan, just in case.

The mobile application is well made with a breakdown of your expenses by category, an instant notification on your smartphone for each expense (useful for a contactless payment to verify the amount), the possibility of refunding another person instantly or share an expense easily.

I take the example of trips abroad, but nothing prevents you from using it every day by topping up your card regularly. You will be able to see your expenses by category, month after month, and refine your budget.

Wondering where is the scam? Well, it’s like Wise, there is not really a catch! But there are withdrawal and card payment limits (daily, weekly and monthly). Frankly, unless you travel for 6 months a year or you are really bad at spreading out your expenses, it should be enough for you. It is still possible to subscribe to the premium option to increase the limits and access additional services.

Revolut is a young, fast-growing company available in several countries. New features are added every month (cryptocurrency exchange, insurance, credit, etc.).

Again, feel free to have a look for yourself and use my referral link to open your Revolut account for free. There is very little chance for you to regret it.

Monzo is also an online bank. It provides a very similar service to Revolut, but you will receive a Mastercard (not Visa) card to make payments in different currencies at the interbank rate.

I must say that I use it for longer and more regularly than Revolut. Unfortunately at that time, the service is only available in UK. Unlike Revolut, there is no charge to receive the card, there is no premium option available, and it’s totally free.

Unlike Revolut, Monzo focuses exclusively on the multi-currency expenses and reporting aspect, you won’t find any insurance, crypto exchange or other services. But what it does, it does it very well! Personally I find the smartphone app a little more convenient. They have limits too, but higher than on Revolut.

In my case, using both services and having 2 debit cards indirectly raise the limits. Also, it is possible that a Revolut card does not work in a place abroad, while it works with Monzo, and vice versa.

Feel free to look for yourself and use my referral link to open your Monzo account and you will get £5 for free.

Last but not least, Curve is also an online bank. The main benefit of this card is that it allows you to group all your cards in one, like a wrapper. If you do not want to take all your cards with you all the time: Revolut, Monzo or any other professional or personal cards, it is possible to use only one: the Curve card. From the application, a simple tap is enough to select the card that will be active. It’s efficient and will make your wallet a little lighter. Note that there may be some limitations in case of disputes, but this will be suitable for everyday use. And since it’s free, why not giving it a try?

You can use my referral link to open your account and get £5 for free. Use code “EKGQQJQN” at sign up.

Once is not custom, I did not look at the technical side, but the post is still about new technologies in the banking system, I hope you’ve found the reading interesting. It also shows that small startups (Fintech) can move the lines of the banking landscape and shape our future. It is critical for traditional banks to constantly innovate to stay in the race, not sure they have all made the shift in time, and the customers of yesterday are no longer the young people of today.

And perhaps you will embrace online banking through your smartphone and even save some money… At least I don’t see any reason not to try!

The article Wise, Revolut and Monzo, a small revolution for travelers and expats was written by Fabian Piau.

Download the file git-completion.bash from Github.

Add this line to your bash profile (e.g. .bashrc), I assume you have copied git-completion.bash at the root of your home folder.

source ~/git-completion.bash

If .bashrc doesn’t exist, create it first.

This line will load git-completion.bash automatically when you open a bash session.

You will then be able to autocomplete all your Git commands (using Tab), and also access advanced completion e.g. look for local and remote branches by starting typing the name.

The article Autocomplete for Git was written by Fabian Piau.

IT projects nowadays are mostly based on a microservice oriented architecture, it is not surprising that the different microservices are developed and maintained by different teams.

Each service provides an API (private or public) that allows the communication with the outside world while guaranteeing its data integrity. Multiple nested calls between microservices make it possible to do more complex processing. It is very important to have an up-to-date documentation for each of your APIs. Who has never heard a developer say to another “This doc has not been updated for months, look at the code directly!”.

Swagger UI complies with this assertion. Simply annotate your code and the framework will semi-automatically generate all your API documentation. I use ‘semi-automatic’ because Swagger cannot guess the business side of your documentation, so you have to write it yourself in a similar way that you are writing Javadoc.

The documentation generated by Swagger is updated at the same time as the code is. For instance, if you add a new parameter to a method that is exposed, it will be automatically taken into account and documented with the appropriate annotations. The documentation is much more than a static HTML file, it allows to act as an HTTP client (there is no need to have Postman installed for example) to test the various methods exposed by the API.

You can take a look at the official sample application (Petstore) and see by yourself the various supported features, for example OAuth2.0 authentication is supported.

Example with Swagger (Petstore)

If you use Spring MVC in your application, the SpringFox project (Swagger wrapper) makes Swagger integration very simple. In less than an hour, you should be able to generate your API documentation. Then you can add optional annotations to make it even more complete.

By adopting Swagger on all your microservices, it is a good bet that the communication between your different teams will improve. You will have a unique entry point and you can forget that dusty wiki page that was updated once in a while… Wishing you good documentation!

The article Swagger, the automated API documentation was written by Fabian Piau.

I’m working on a microservices architecture since quite a long time now and attend many conferences on this topic. I want to gather my experience in this article to give you some feedbacks and also advices. Each title is what you shouldn’t do followed by a description of what you should do instead. Even if you are already doing microservices on your project, it can still be a nice reading (I hope) and refresh your memory. Also, don’t hesitate to comment and share your thoughts!

Adopting a microservices architecture is not that easy. Writing microservices is not only about code, it is also about the teams and the structure organization.

Let’s extract this complex piece of code, it will be more simple in a microservice!

Usually, it is a bit easier when starting a new project from scratch, if it is your case, then you are lucky! If you want to break down a monolithic application into a set of microservices, don’t do everything in one go, the big bang is the best way to blow up everything. Try to gather the similar set of functionality that can be included as part of the same microservice. Make sure that all the features you want to extract are tested before refactoring anything. If the coverage is low, you have to start by writing tests. By doing that, you will improve the code from the monolith, and this code will be reused after in your microservice. Keep in mind you are preparing the extraction, this work won’t be useless.

If you don’t have enough tests, you will do things blindly and you won’t be able to ensure that you are not breaking anything, regression is the last thing you want. To recap, write integration tests around the code you want to extract, then you can start to extract some feature to create a microservice. Your integration tests should still pass, but the plumbing behind will have changed.

Start small and easy at first. And don’t be afraid of duplicating some code. Once your microservice is working fine, you can remove the duplicated code from the monolith. A/B testing can help, so you can gradually redirect the traffic to the new microservice and see how the system reacts.

The ‘micro-monolith-service’ trap

Keep the microservice quite small and not too complex. There is no magical answer to the question how many lines of code a service should have to be considered as a microservice? But it is more about feeling and great design. If a new joiner needs one day to understand the code and what your microservice is doing, then you probably get it wrong. A few hours should be more than enough.

You need to parse an XML file, let me write a microservice for that!

On the other extreme, don’t write a microservice when a library is enough, don’t over engineer things. You don’t want to have a call depth too big (number of nested requests). If you have more than 3 nested calls, you are probably doing nanoservices. A call involves network latency and a possible failure. A microservice will have to be deployed with an API, where a library is directly embedded without ops overhead. It is a balance between DevOps overhead and monolithic complexity.

Sorry, I can’t help you, I never work on this microservice…

It is not possible that all the teams are owning all the microservices. You will have to deal with synchronization and communication between your teams. I heard very often the fact that a team should not own a microservice, and everyone should be able to switch and work on any microservice. Well, in reality, it is not so true, but you can adopt some good practices to make it easier for someone to work on a microservice developed by another team.

In my opinion, a team owns a microservice, which means it is responsible for building and running it. When something wrong happens in production, the team should be the main point of contact. You will have to find the person within the team who will accept a call at 5am… Just kidding ;)

Spring, Dropwizard, Finagle, Clojure, let’s try and mix all!

Try to stick to the same technology stack throughout all your microservices. I know that a nice advantage of microservice is that you can build them with any language you want. Passionate developers will have a tendency to use the latest trendy framework. In the long term and with people turn over, maintaining your microservices will become painful and turns into a nightmare. Switching of microservices between the teams will become very difficult or even impossible. I think it is important to limit to a certain number of technologies.

Many technologies are available to do microservices, you have to use something reliable, well maintained, and so on. Which serialization, text or binary? REST, Thrift, SOAP? An in-house solution, open source framework? There is no best answer. Compare the different technologies, pros and cons, use the most suitable for you.

At the same time, this limitation shouldn’t prevent you to innovate. Don’t stick to what you already use and master if you think there is a better solution on the market. Try it on a new microservice (but non-critical to the system) as a POC, then in the case of success, you can propagate it to the whole system.

What about the data stores?

You can be more flexible about the data store. This is an external dependency, and it can be adapted depending on the needs (relational, noSQL, memory database, read-only access…).

Keep your data stores independent, each microservice is a data keeper. You should never bypass it by linking a microservice to the data store of another microservice. It will be an extremely bad design. In case you are limited to one database for some reasons, it’s fine because you can define different schemas and limit access for each microservice.

If one microservice fails, my whole system is down…

It is important to design a good architecture. Think about scalability, circuit breaker, service discovery from the start, not one month before going to production with thousands of users. You should always keep this in mind from day 0. Some people may argue that point, and sometimes you may have to convince the product owner that doesn’t see the real business value. But trust me, the more you wait, the more work it will be and you won’t be so confident with your architecture until all these technical tasks are complete.

You are dealing with many requests over the network, which is usually not very reliable, you have to imagine a design for failure. This is a mindset the team has to adopt. You have to think about retries (e.g., in the case of failure, you retry on the second instance), idempotent calls and so on. Retry is not as easy as you can explain it, especially when you cannot make the call idempotent. At some point, you will have to fine-tune the retry configuration. E.g. try avoiding retries at the inner level request if there are already retries at the outer level, the timeout at the outer level should be more than the timeout at the inner level, etc. All your microservices should expose a set of health checks URL and other useful metrics that another monitoring tool will use (either on a pull or push model).

It is also important to test the infrastructure. What will happen if this instance of microservice goes down? You should do some Monkey testing (i.e. switch off some microservices randomly), the system should react positively and still be able to handle and serve the requests (in a degraded mode).

All right, let me ‘grep’ the log to see what happened yesterday… Wait… There are 50 log files here!

Having many microservices involve lots of interactions, so lots of logging. It will be difficult to debug in case something went wrong in production. Where was it? Which log file? Which environment? Ops can get lost easily… Try to do accurate logging, and keep a correlation identifier to be able to trace the complete call stack throughout the different log files.

Be proactive and don’t wait for the customer to complain. Track any stacktrace that can occur, raise alerts and constantly fix all problems until your logs are “cleaned” and contains only useful and accurate information. A log monitoring tool (e.g. Splunk, Kibana…) will make sense in a microservices architecture.

We cannot test, the microservice we depend on is still not ready

When writing integration tests, mock all external dependencies of the microservice. Especially, if a team is still developing a microservice you depend on. You don’t want to wait for them to finish. E.g. you can take a look at Wiremock. On the opposite, for the end-to-end tests, try to provide a good and fast tool to run them locally, the use of embedded servers is a good way to achieve that.

I already mention it, deploy often to production, especially at the beginning, and don’t wait 6 months before deploying your microservices. Even if they are not useful (yet) from a business value point of view, you will be more confident when working on them, and then you can implement the business logic without worrying too much about the architecture. The earlier you test in production, the earlier you will see bugs and be able to fix them quickly.

Let’s design the API like that for now, something simple, we will change it later

It is not easy to be agile when designing an API, especially a public one. Once a version is released and start to be used, any big change will involve an API contract change. You will have to think about versioning your API and ultimately maintain different versions. It is important to spend some time at the beginning to define a good API contract that can evolve without breaking changes (ideally). That sounds waterfall to you? Well, that probably is…

Always keep in mind to be backward compatible (even if the calls are internal) and use API versioning. For example, don’t rename a field like that, but follow a longer and safer process, you should add the new field, keep the old one deprecated until all the clients have migrated, then you can delete it.

The ops guys need one day to deploy a couple of microservices, what is wrong with them?!

The ops team has to change its mindset, instead of deploying and monitoring a big and unique application, they will have to manage many little applications and make sure there are no communication failures. From a couple of manual steps, they are ending up with many tedious steps. They will be quickly overwhelmed if they keep the same habits and spirit, especially the number of microservices will grow over time.

They will need to automatize their work and avoid any manual and error-prone step. Adopting a microservices architecture forces you to do “real” DevOps. The developer has to participate in the deployment, it is not about just giving a file anymore and counting on the ops to do the job. The developer has to be involved. Ideally, an ops guy will be dedicated to the team to make sure the network configuration is correct, the health checks are existing and working fine, participate to stand-ups and even seat with the dev team. You will need to start thinking about using a deployment tool such as Ansible.

We have spent 6 months on this architecture, it is working very well… It is just that we have 50 users!

Last advice, don’t try to do everything at once. You have to tackle with so many problematics already. You can put aside some stuff like self-healing, service discovery, circuit breaker, auto scaling, etc. The use of retries and multiple instances will be sufficient at the beginning. But keep in mind, you will have to implement them at some point, especially when your system is successful and used by thousands of users.

The article Microservices architecture – Best practices was written by Fabian Piau.

The article Using Google Forms / Drive / Docs to create an online survey regularly receives comments. I realized that people come across similar issues, so I decided to write this article as a FAQ. I do not consider myself a Google Forms expert and Google can evolve the product at any time, making my articles outdated. However, you can probably find the answer to your question in this FAQ (hopefully).

When a user wants to respond to the form, he / she has to connect to Google first, is it normal?

You must disable the option “Only allow one response per person” that forces people to connect to their Google account. Be aware that without this option on, people can potentially answer the form several times.

How to prevent a person to answer the survey several times?

To have more confidence in the results, you can enable the option “Only allow one response per person”. Note that users will be asked to log in to their Google Account to view and complete the form, but the actual user name will not be recorded.

I don’t want to force users to have a Google account, is there an alternative?

If you do not want to force your users to have a Google account, an alternative is to use pre-populate the form.

For example, here are links to pre-populate the field “name” in a sample survey:

• https://docs.google.com/forms/d/e/1FAIpQLSeC8fIptm9xY8Ai1dADB8JxqbzDbQFv4vULOq_vGCK10NWsUw/viewform?entry.1000000=Fabian
• https://docs.google.com/forms/d/e/1FAIpQLSeC8fIptm9xY8Ai1dADB8JxqbzDbQFv4vULOq_vGCK10NWsUw/viewform?entry.1000000=Caroline

You will notice that the URL changes at the end with an additional parameter. Only one field is pre-populated, but it is very easy to pre-populate several fields (such as name or email). Generate the link from the menu to get the syntax of the parameters and then you can manually change them before sending the link. If you have many links to send, it will take you some time, but it will work.

However, note that it does not prevent the person to modify the pre-populated information (directly from the form, or by changing the URL) or submit the survey multiple times.

It is not possible to hide fields, but there is a workaround! You can add fields you want to hide on a specific section of your form that will never be displayed. You have to use sections in your form and change the navigation to never show the section that contains the hidden fields and go to the next one directly. Read pages Add content to your form to know how to add a section and especially this page Control navigation to sections of a form.
Again, this will not prevent the user to manually change the URL, but many will not pay attention, especially if you use a URL shortener tool.

Is it possible to include the form in my website?

Yes, it is possible to include the form in your website as an iFrame. The option is available from the “File” menu. The menu generates an HTML code snippet that you can then copy / paste into a page of your site. You can also write the code manually, for example:

<iframe src="https://docs.google.com/forms/d/1SEu1y1TvOyPUwkUHcwgC-ky0LIVZXCjjr5ARH_E3mK0/viewform?embedded=true" width="760" height="500" frameborder="0" marginheight="0" marginwidth="0">Loading...</iframe>

You can change the size of the iFrame and the URL to fit your form.

Is it possible to include the form directly in an email?

Yes, it is possible. When sending the email (via the “Send form” menu), make sure the checkbox “Include form in email” is checked.

I would like to get the IP address of each participant. Is it possible?

No, it is not possible to trace the IP address. You can force the users to authenticate before they can respond to the form, but this requires users to log in with their Google Account.

Is it possible to change the answers after having submitted the form?

When building the form, there is an option on the confirmation page settings, “Edit their response: Allows respondents to change their answers to your form”. If this option is checked, after submitting the form the person will have the confirmation page with a unique link to edit his / her answers. This link is displayed only one time, if the person closes the window, he / she will no longer be able to change his / her answers. You can find more information on the official documentation Send your form to respondents.

Is there a maximum number of responses?

There is no particular limit. However, a spreadsheet has a limit of 2 million cells.

Is it possible to redirect the users to another site once the form is submitted?

No, that’s not possible. A good idea may be to change the confirmation page message at the end of your form. Instead of the classic “Your response has been recorded”, replace with another sentence inviting the user to click on a link to continue.

Is it possible to not accept new responses after a specific date?

There is no such feature. But nothing prevents you from adding a free text section at the top of the form to indicate that it will be available until yyyy-mm-dd. And on that date, you close the form. This will have the same effect.

To prevent new answers to be submitted, click the Accepting responses in the toolbar. The button will show “The answers are no longer accepted” To reactivate it, click the button again.
When a form is no longer accepting response, users who visit are informed by a message that their answers will not be collected. To customize this message, change the text that appears under the title “The form has been disabled” in the top of the form.

Is it possible to create a multilingual survey?

It is not possible, but there are some workarounds that can more or less suit your needs.

You can create different forms, the link will be different, so as the summary of responses. It will be impossible to merge the results automatically, you will have to rework the table of responses with a spreadsheet tool like Excel (to consolidate the responses).

You can also add a first question that asks the user his/her language, then you can display the relevant sections in the selected language. Compare to the first solution, the link will be identical for everyone (no risk of error). The downside is that the question in French and the one in English will be on different columns (one cell will be always empty), so the summary of responses won’t match your expectations. Again, you will have to consolidate the responses manually.

Is it possible that a person starts to respond to a form and wants to resume it later?

No, that’s not possible. All responses are recorded once when submitting the form.

The article FAQ – Online survey with Google Forms / Drive / Docs was written by Fabian Piau.

Simon Ritter gave a presentation about JDK 9 at QCon London.

QCon London 2016 - Project Jigsaw in JDK 9 - Modularity comes to Java

Java Compatibility Policy

He started with an interesting fact: Java has a general and longstanding compatibility policy.
If an application uses only supported APIs on version N of Java, it should work on version N+1, even without recompilation.
That’s why, to date there are:

• 23 classes, 18 interfaces and 379 methods that have been deprecated
• And none have been removed.

JDK 9 brings breaking changes

The JDK has been heavier and heavier over the years. With the version 9, things are going to change. For the first time, JDK 9 will bring incompatibilities.

• A small number of supported API will be removed
• The binary structure of the JRE and JDK will change (see details below)
• It is not allowed to use a single underscore _ as a variable name. It is now a reserved keyword.
• There will be a new version scheme so that it is easier to distinguish major, minor, and security-update releases, e.g. Java 9.1.2.1. The old versioning format is difficult to understand, e.g. 8h51 and 8u60 are not very meaningful.

The JDK structure is changing. The JRE folder is removed, tools.jar and rt.jar are no longer present, as there was some duplication between them.

Pre-JDK 9 structure

To have a complete understanding of the pre-JDK 9 structure, you can read JDK 8 and JRE File Structure.

JDK 9 structure

The JDK 9 structure is much cleaner.

JEP 260 proposal is at the heart of JDK 9 and Jigsaw.

The idea is to make most of the JDK’s internal APIs inaccessible by default but leave a few critical, widely-used internal APIs accessible, until supported replacements exist for all or most of their functionality.

The goal is to encapsulate all deprecated API in a module, then eventually getting rid of it in the future (JDK 10?).

Since JDK 8, a command line tool exists to analyze the dependencies of your JAR or class: jdeps. Give it a try! It can be useful to discover if you are using JDK-internal APIs, you should avoid using these dependencies as they may be removed in the future.

Jigsaw and modules

We talk about module just before, but what is it?
Jigsaw brings modularity to the JDK. It makes Java more scalable and flexible, it also improves security, maintainability and performance.
The JDK is split up into modules, and then we take only what we need.

Module

A module is a grouping of code (collection of packages), it can also contain:

• Native code
• Resources
• Configuration data

My first module

To define a module, you need to create a module-info.java file, including (for example):

module com.company.mymodule { 
  requires com.company.myothermodule; 
  requires java.sql; 
}

You can have transitive dependencies (a bit like Maven) thanks to the keyword public.

module java.sql { 
  requires public java.logging; 
}

This means if I have a dependency on the module java.sql, I will have access to all classes included in java.logging (as long as the types are public).

We now have a module dependency graph.

Package visibility

Use of the keyword exports.

module com.company.myothermodule { 
  exports com.company.myothermodule.alpha;  
  exports com.company.myothermodule.beta; 
}

What is exported is visible, by default it’s not!

In Java 9, we have to redefine the keyword public, it comes in different flavours:

• Public to everyone
• Public, but only to specific modules
• Public, only within a module

Public does not mean necessary accessible anymore, that’s a fundamental change.
You can read the Jigsaw specification to know more.

Compilation and execution with module path

Forget about the classpath parameter, there is a new modulepath parameter (or -mp) in the javac / java command line. Its value is one or more directories that contain modules that are required to compile / execute your application.

Also, note there will be default module for the JAR files that are not module-compatible yet (automatic module).

The article QCon London 2016 – Project Jigsaw in JDK 9 – Modularity comes to Java was written by Fabian Piau.

Juergen Hoeller, the co-founder of the Spring framework was doing a presentation last week at QCon London.

QCon London 2016 - Spring Framework 5 - Preview & Roadmap

Some history background first, the Spring framework has been created in 2002 with a first release in 2004 (yes, it was 12 years ago!), quite a long time for a framework which is still widely used and under development. The adoption and the number of projects within Spring has never stopped growing since. At the beginning, Spring was created to manage POJOs in a lightweight container by using dependency injection (following the Inversion of Control or IoC principle), Spring was an alternative to heavy JEE containers such as GlassFish or WebSphere.

Spring has now become a set of projects that are available to fulfill many requirements when building an application. Spring is like a set of Lego bricks, you can pick and assemble these compatible sub-projects to build something bigger.

• Imagine you want to use a database, you can use Spring Data
• You need to secure your application, just use Spring Security
• You want to be able to login to your application through social networks, Spring Social is here for you
• And so on

You can take a look at the full list of Spring projects.

Spring 4.3

Before jumping to Spring 5, Juergen talked about Spring 4. When I write this post, the current version is 4.2.5. A release candidate (RC) of Spring 4.3 will be available in March 2016 followed by a General availability (GA) one in May 2016. This version will include a couple of handy changes:

• In Spring Framework core:
  The @autowired annotation used to inject a dependency will be implicit so will become optional
• In Spring MVC (part of the core):
  The declaration of a MVC controller is refined. Before, value was the attribute name:

  @RequestMapping(value = '/mypath', method = RequestMethod.POST)
  

  After, we use a more meaningful name path:

  @RequestMapping(path = '/mypath', method = RequestMethod.POST)
  

  There are even new annotation shortcuts to avoid specifying the HTTP method attribute. And even the attribute name is optional, so it becomes very simple.

  @PostMapping('/mypath')
  

  And the equivalent for a GET

  @GetMapping('/mypath')
  

Spring 5.0 announced

Then, Juergen announced that Spring 5.0 will be available sometime in 2017 and compatible with JDK 8 and 9. On the roadmap, the GA should be available in March 2017 (at the same time the JDK 9 will be released). In case JDK 9 release is delayed, Spring 5.0 will be probably delayed too.

What are the major updates in Spring 5?

There are 3 key infrastructure themes:

• JDK 9 and Jigsaw modules
• Servlet 4.0 and HTTP/2
• Reactive architecture

JDK 9 and use of Jigsaw

In short, Jigsaw brings modularity to the JDK. The JDK is split up into modules and you can pick up only the ones you need to make your application works.

The JDK will come with a set of modules, but you can also define your owns. To define a module, you need to create a module-info.java file:

module my.app.db {
  requires java.sql;
  requires spring.jdbc;
}

Inside a module, you specify the modules you need with the keyword requires. Modules are managing transitive dependencies, a bit like Maven. If I use the module my.app.db above, I will be able to access any classes coming from the java.sql module. The main benefit is that it makes your application more modular relying on a subset of modules instead of a full JDK.

All the jars for Spring 5 will come with Jigsaw metadata out of the box, the module name will follow the Maven Central jar naming to avoid confusion (spring-context, spring-jdbc, spring-webmvc…). It will be very easy to build your applications by specifying the Spring modules you need.

HTTP/2

HTTP/1.1 is an old protocol (1999) and has some limitations. In the front-end world, we are using some workarounds, but it is not ideal. For example, to avoid overloading the server with multiple requests when a page contains a lot of images, we use CSS sprites (i.e. combine these images into one single file that we then split on the client side via CSS). With HTTP/2, the protocol is faster as it allows concurrency requests and removes the “head-of-line blocking”. There are other benefits that you can read on the HTTP/2 Frequently Asked Questions page.

As Juergen stated, “We need to embrace HTTP/2 in the Java land as well!”

To achieve that, Spring 5 will use the latest version of the Servlet library, Servlet 4.0 that:

• Enforces support for HTTP/2 in servlet containers
• Gives API features for Stream priorization and push resources

Reactive programming

Another big announcement was the creation of a new Spring project called Spring Reactive. It will be a Spring MVC-like endpoint based on a reactive foundation.

• Reuses Spring MVC programming model style
• But accepting and returning reactive streams (e.g. Mono – 1 element, Flux – list of elements)

Spring Reactive will be based on Project Reactor, a set of foundational libraries for building reactive cloud-ready applications on the JVM. This new project is not stand-alone and will be included in the Spring Framework core at some point.

Juergen reminds us that “Reactive is coming”:

• No blocking thread involved
• Reactive datastore drivers become available (PostgreSQL, Mongo, Couchbase…)
• Reactive HTTP client (Netty, Jetty, OkHttp…)

Note for myself and maybe for you as well, it is important to read some materials on Reactive and get familiar with the concepts, it will probably spread in the following years.

Last but not least, Spring 5 will use all the latest features introduced with Java 8:

• Lambdas, method reference, default methods in interfaces
• Ability to expose JDK 8 API types in core interfaces and classes: java.util.Optional, java.util.function, java.util.stream

As a matter of fact, Spring 5 will require Java 8. If your project is still using an older version, you will have to use Spring 4.X. The good news is that Spring 4 will have an extended support until 2020, so you will still have some time to migrate!

Slides of the presentation are available on the QCon website.

The article QCon London 2016 – Spring Framework 5 – Preview & Roadmap was written by Fabian Piau.

In this article, I will give you some useful online tools to ensure your website is efficient, does not contain errors and is optimized for search engines (SEO). These three characteristics are very important because they form a virtuous circle by mutually reinforcing each other. None should be overlooked.

Performance

After analyzing your website, GTMetrix will give you valuable tips to improve its performance.

• Minimize the amount of resource to reduce the number of HTTP requests (using sprites, merging files)
• Reduce the size of your resources. There are very good online compressors for images (TinyPNG), CSS and Javascript files.
• Specify image dimensions, the browser does not need to do this calculation
• Use the browser cache

Search Engine Optimization – SEO

You can check your ranking with SEO SiteCheckup. This tool will give you valuable information to improve your site’s visibility on the Internet.

• Using the basic HTML tags and attributes (title, description, h1, h2, alt…)
• Creation of the sitemap file
• Detection of broken links
• Detection of the social activity linked to your website, your presence on social networks

Technical validators

You should also make sure your site is valid from a technical point of view. I recommend the official developer tools provided by the World Wide Web Consortium (W3C). Each tool is specific for a particular purpose. For example, you will find a tool to validate:

• HTML files
• CSS files
• Links
• RSS newsfeed

It is not necessary to validate every page one by one. The validation of the homepage is usually sufficient.

Email

Your application is sending emails? With Mail tester, you can test the relevance and format of your emails. This tool will let you know if the emails you are currently sending are considered as spam and also give you advice to avoid such a situation.

• The content should adapt to the size of the screen (responsive design)
• An alternative raw text format should be attached for email clients that cannot render HTML
• The authentication of your emails should be properly configured (SPF, DKIM)

To give you an idea of your site’s performance compare to others, most of the tools quoted in this article will give you a score, now you have to increase it!

The article The best free and online tools for testing and optimizing an application or website was written by Fabian Piau.

Google is part of our daily lives

Countless news reports are chronicling the hegemony of the US firm. Google’s success fascinates, irritates and sometimes scares. Like most Internet users, it is difficult to spend a day without using one of the services provided by Google. At some point, you ever :

• Browsed the web with Google Chrome
• Searched the Internet with Google Web Search
• Read your emails with Gmail
• Organized your appointments with Google Calendar
• Managed your photo albums with Picasa
• Chatted with your friends with Google+ or Google Hangouts
• Watched videos with YouTube
• Read books with Google Books
• Managed your documents with Google Docs
• Translated an article with Google Translate
• Planed your moves with Google Maps
• Managed your phone, contacts or alarm clock (and more) with Android
• And even more…

The giant has diversified and is now competing on all fronts (automation, robotics, automotive…). Its strategy is still focusing on individuals and how to improve everyday life. You can read this list of mergers and acquisitions by Google to convince yourself.

One search may hide another

Did you know that Google is gathering and storing all your searches? Google is able to time travel in your search history, to the nearest second. The fact that you have searched for information on your future trip to Thailand on August 10th, 2005 at 10:05 may not be very useful to you, but this information among other represent a goldmine for Google, in order to establish your very detailed profile. It is however possible to disable this feature if you go to your account’s advanced settings. This is not something that an average user does. This is not a coincidence if this setting is enabled by default. A Google search may be trivial at first, but imagine that you’re looking for:

• Cars and brand comparison, Google will determine that you will change your car soon
• Fashion and clothing throughout the year, Google will determine that you are working in this sector
• A disease and related remedies, Google will determine that you have this disease
• Tips to save money, Google will determine that you are barely making ends meet at the end of the month

Jobs, places, favourite musics, movies, cooking, hobbies or political views. These examples are fairly simple, but it can goes very far when overlapping the information. It becomes possible to tell very accurately the events of a lifetime. If you think that Facebook is the biggest player, you’re probably wrong.

You’re the product

Google is continuously collecting huge amount of data from and about its users. It builds and consolidates complete profiles of our lives, and probably knows you better than your own mother or even yourself. Google’s services are free in exchange of your personal data. It is an implicit contract that you agreed (not necessarily read) when creating your account. The data collected can be sold and used for targeted marketing campaigns, statistics, etc. After all, it is fair game!

Smile, you’re on filter

The search engine displays a customized list of results, showing only the results that are relevant to your profile. You can do this simple experiment at home: run several searches with the same set of keywords but using different Google accounts. The list of results will probably be slightly different, especially if people are different (e.g. different gender or generation). The fact that the results are relevant to the connected user is a positive aspect, but it can also be a disadvantage. The primary purpose of the Internet is to make the universal knowledge accessible with a censorship as limited as possible. If you add an intelligent filtering system on your searches, there will be articles or even entire sections of the internet that you’ll never be aware of, simply because Google has decided so. Somehow, the knowledge and access to the information become limited or even incorrect.

I’m not anti-Google

Do not be mistaken, this article is not a pamphlet against Google. I am a very satisfied user for years and it will be probably painful to navigate the web without their services. This is more of a wakeup call. It is worth to spend some time to set up your account rather than letting Google decide for you. Unfortunately, I don’t know any magic tricks to continue using Google while preserving your privacy. However, I can advise you an alternative search engine DuckDuckGo. This search engine has gained great popularity by focusing on the respect of your privacy without neglecting the relevance of the results.

The article Should we be wary of Google? was written by Fabian Piau.