I’m a Cold War kid. I grew up watching news of Pershing II and SS-20 deployments in Europe, the Soviet war in Afghanistan, with some Terminator and Top Gun VHS action on the side. Yugoslavia was trying to play both sides, and for a while it worked like a charm. It all crashed a couple of years after the Berlin Wall came tumbling down, rendering our unaligned prowess pointless.
I admit this is an odd intro for a tech blog, but bear with me; it will start to make sense. Unlike most Europeans, we had good relations with both blocs. We sold tanks to Kuwait and rocket artillery to Saddam, we bought cheap fuel and MiGs from the Soviets, and in return we exported some stuff they couldn’t get directly from the West. I know people who would stay in East Berlin hotels because they were cheaper, then cross the border into West Berlin to work, play and shop, only to cross back via virtually unused border crossings like Checkpoint Charlie, all in a matter of hours.

On one such trip, my dad got me a Commodore C64, which was pressed into service as our Cold War gaming machine. Most 80s video games, and indeed a lot of music and films, were inspired by countless proxy wars and the threat of a nuclear apocalypse. As the Wall came down, a lot of people assumed that would be the end of runaway defence spending and that the world would be a safer place. It didn’t exactly work out that way, did it?

However, the long-term effect of the Cold War on science and technology is more profound than Nena’s 99 Luftbalons, or any Oliver Stone Vietnam flick.

Minuteman: A Cold War Tech Case Study

If you are reading this, you’re already using a technology developed for cold warriors; The Internet. That’s not all. A lot of tech and infrastructure we take for granted was developed, or at least conceived, during these tumultuous decades.

That constellation of GPS satellites orbiting Earth? It wasn’t put up there to geotag selfies or get an Uber ride; it was designed to help the US Strategic Air Command deliver hundreds of megatons worth of instant sunshine on Soviet targets with pinpoint accuracy. Integrated circuits, transistors, solid-state computing? Yep, all developed for the armed forces and paid for by the US taxpayer.

Here is just one example: the sleek and unfathomably deadly LGM-30 Minuteman intercontinental ballistic missile (ICBM). It wasn’t the first ICBM out there, but when it appeared, it was revolutionary. It was a solid fuel missile, which meant it could respond to a threat and launch in a minute without having to be fuelled, hence the name. But solid fuel was only part of the story: Solid-state was a lot more interesting from a geek perspective. Prior to Minuteman, ICBMs relied on analogue computers with mechanical gyros and primitive sensors. Since they were wired to a specific target, the target package could not be changed easily. Minuteman was the first mass implementation of a general purpose digital computer; it integrated an autopilot and missile guidance system in one package, with reliable storage that could take the stress of a silo launch. The computer was also capable of storing multiple targets, and was reprogrammable.

Transistors were nothing new at that point; they were developed years before by Bell Labs. Yes, these primitive transistors were almost exclusively reserved for the military-industrial complex. Uncle Sam was the sole customer for virtually all early computers and chips, burning heaps of money. These early transistors offered a quantum leap over vacuum tubes, but they weren’t perfect. By today’s standards, they were rubbish. The reliability simply wasn’t there, and if you needed to launch a few hundred thermonuclear warheads halfway across the planet, you sort of needed a guidance system that wouldn’t fail as soon as the candle was lit.

So what do you do when you come across a technical problem you can’t solve with money? Simple: You throw more money at it, and that’s exactly what the US Air Force did. They burned millions to make the damn things reliable enough to be used in harsh environments and survive the stress of a high-G ascent to space. This was known as the Minuteman High Reliability (Hi-Rel) programme.

The first truly mobile digital computer was somewhat deadlier than your notebook and iPhone.

It worked, but the USAF got a bit more than they bargained for. In trying to improve a single weapons system, the USAF ended up giving a huge boost to the tech industry in general. Eventually, the Minuteman was upgraded to include a new microchip-based guidance system, with a primitive form of solid-state storage. This Cold War relic has been in service since the Kennedy administration, and the current incarnation has been around for 45 years, receiving multiple hardware and software updates over the years.

So, in outlining the development and evolution of a single strategic weapon delivery system, I have touched on a number of vital technologies we take for granted: reliable transistors, chips, solid-state storage, mass-produced programmable computers and so on. The Minuteman was also the first mobile digital computer.

Some may argue that the legacy of such weapons is that Mutually Assured Destruction (MAD), guaranteed by the nuclear triad, kept superpowers from going to all-out war. It probably did, but in doing so, it also allowed engineers around the world to develop technologies and concepts applicable in various industries and fields of study.
Their real legacy lies in every integrated circuit on the planet.

Capitalist Pioneers Try To Cash In

What could be more capitalist than monetizing instruments of mass murder? The taxpayers paid for their development, not venture capitalists!
Joking aside, it can be argued that the Red Scare of the fifties created Silicon Valley. Most of the money really did come from taxpayers, and most companies that got lucrative defence contracts were quick to make a buck on dual-use technology developed for the military. Remember Bell Labs? A few of their brightest people went on to co-found Fairchild Semiconductor, and eventually created Intel a decade later. The updated Minuteman guidance computer was based on chips from another semiconductor giant: Texas Instruments.

I am not disputing the brilliance of people like Intel co-founders Robert Noyce and Gordon Moore. I have no doubt they would have made their mark on the tech industry even without the biggest arms race in history, but it’s also hard to dispute that the tech industry wouldn’t have developed at nearly the same pace had there been no government funding. Yes, the taxpayers effectively subsidised the tech industry for decades, but in the long run, they’re probably better off. Westinghouse did not need subsidies to develop washing machines and refrigerators, because consumer demand was strong, but in the early days of computing, there was virtually no consumer demand. That’s why governments had to step in.
But what did the taxpayer get?

The Internet, GPS, reliable transistors and chips: Cold War tech made possible by runaway defence spending.

The space and arms race spawned a number of technologies that in turn created countless business opportunities. Even primitive computers had a profound impact on industry. They made energy grids and transportation infrastructure more efficient, helped improve safety in industrial facilities, including sensitive chemical and nuclear facilities, they changed the face of banking, communications, entertainment and so on.
Best of all, we somehow managed not to blow ourselves up with the weapons these technologies made possible, yet at the same time, we turned swords into ploughshares. Back in the fifties, the US and USSR launched initiatives designed to examine civilian uses of nuclear power (including civil engineering nuclear explosives schemes, which went terribly wrong), but they amounted to nothing. It wasn’t the might of the atom that changed the world, it was the humble microchip and ancillary technologies developed for countless defence programmes.

Before they made their mark in science and beat Gary Kasparov at the chess table, supercomputers and their analogue predecessors were used to simulate physical processes vital in the development of thermonuclear weapons. An advantage in sheer computing power could yield advances in countless fields. Computer simulations allowed western navies to develop quieter submarines with new screws, digitally optimised to avoid cavitation. Digital signal processors (DSPs) made sonars far more sensitive, and a couple of decades later, advanced DSPs made music sound better. Computer aided design wasn’t just used to reduce the radar cross-section of airplanes, it also made our buildings and cars cheaper, safer and more energy efficient.

Some of these efforts resulted in a technological dead-end, but most did not. One of my favourite tech duds was Blue Peacock, a British nuclear landmine (yes, landmine, not bomb), weighing in at 7.2 tons. Since it relied on early 50s technology and had to be buried in the German countryside, the engineers quickly realised the cold could kill the electronics inside, so they tried to figure out how to keep circuits warm. Their solution was so outlandish that it was mistaken for an April Fool’s Day joke when the design was declassified on April 1, 2004.

No chickens were harmed in the making of this blog post, or in the Blue Peacock nuclear landmine programme.

A chicken was to be sealed inside the casing, with enough food and water to stay alive for a week. Its body heat would keep the bomb’s electronics operational.
As civilian industries started implementing these cutting edge technologies en masse, our quality of life and productivity shot up exponentially. Our TVs, cars, phones, the clothes we wear, and just about any consumer product we buy: They’re all better thanks to the biggest waste of money in history. Granted, we all have trace amounts of Strontium 90 in our bones, but in the big scheme of things, it’s a small price to pay for the high-tech world we enjoy so much.
Oh yes, we also got video games out of it. Loads and loads of video games.

Kickstarting Game Development

Video games were pioneered on the earliest digital computers (and some analogue ones as well). In fact, Tennis for Two, arguably the first game to use a graphical display, was developed for an analogue computer in 1958. However, not even Bond villains had computers at that point, so the rise of the video game industry had to wait for hardware to mature.

By the mid to late seventies, microchips became cheap enough for mass market applications. Now that we had the hardware, we just needed some software developers and a use-case for cheap chips. Since the average consumer was not interested in expensive and complicated computers that were designed for big business, attention shifted to gaming; arcades, game consoles and inexpensive computers like the ZX and C64.

These humble machines brought programmable computers to millions of households, hooking a generation of kids on digital entertainment, and creating opportunities for game developers. Consoles and cheap computers brought the arcade to the living room, ushering in a new era of video gaming, and creating countless jobs in the industry. Even the Soviets got in on it with Tetris, the first game from behind the iron curtain.

The advent of inexpensive home computers and game consoles created a generation hooked on computing and coding.

It wasn’t just entertainment. Unlike consoles, the ZX and C64 were proper computers, and geeky kids quickly found new uses for them. They started making demos, they started coding. Chances are you know a lot of these kids, and if you’re reading this, you probably work with some of them.

If you’re interested in the development of early video games, and what the Cold War had to do with them, I suggest you check out Nuclear Fruit; a new documentary that’s a must see for all geeks and gamers born in the 70s and early 80s.

These guys and gals went on to develop a new breed of video games, build successful online businesses, create new technologies and revolutionise the digital world, all in the space of a decade. A generation that grew up with the constant threat of nuclear war, enjoying dystopian science fiction, helped make the world a better place. They didn’t develop Skynet, they developed millions of mobile and web apps instead.

So, no Terminators. At least, not yet.

Cold War 2.0 And The Emergence Of New Threats

This is not a geopolitical blog, but if you happen to follow the news, you probably know the world is a messed up place. No, the end of the Cold War didn’t bring an era of peace and stability, and there’s already talk of a “Second Cold War,” or worse, a “hot” war. While most of these worries are nothing more than hype and sensationalism, a number of serious threats remain. The threat of nuclear annihilation is all but gone, but the technology we love so much has created a host of potential threats and issues, ranging from privacy and security, to ethical concerns.
Thankfully, we aren’t likely to see an arms race to rival the one we witnessed in the 20th Century, but we don’t have to. The same technology that makes our lives easier and more productive can also be used against us. The digital infrastructure we rely on for work and play is fragile and can be targeted by criminals, foreign governments, non-state actors, and even lone nutjobs with a grudge.

These new threats include, but are not limited to:

• Cybercrime
• State-sponsored cyber warfare
• Misuse of autonomous vehicle technology
• Privacy breaches
• Mass surveillance abuses
• Use of secure communications for criminal/terrorist activities

All pose a serious challenge and the industry is having trouble keeping up. My argument is simple: We no longer have to develop ground-breaking technology to get an edge in geopolitical struggles, but we will continue to develop technologies and methods of tackling new threats and problems. It’s a vicious circle since these new threats are made possible by our reliance on digital communications and the wide availability of various technologies that can be employed by hostile organisations and individuals.

A new generation of emerging threats is once again rallying industry leaders and governments around a common cause.

Cybercrime is usually associated with identity theft and credit card fraud, but it’s no longer limited to these fields. The advent of secure communication channels has allowed criminals to expand into new niches. The scene has come a long way since the romanticised exploits of phone phreaks like Steve Wozniak. Some offer hacking for hire, others are willing to host all sorts of illicit content, no questions asked. Some groups specialise in money laundering, darknet drug bazaars, and so on. The biggest threat with this new generation of cybercrime is that you no longer have to possess many skills to get involved. As cybercrime matures, different groups specialise in different activities, and they can be hired.

State-sponsored cyberwarfare poses a serious threat to infrastructure, financial systems, and national security. However, there is really not much an individual can do in the face of these threats, so there’s no point in wasting time on them in this post. Another form of economic warfare could be to deprive a nation or region of Internet access. It has happened before, sometimes by accident, sometimes by government decree and enemy action.

Commercial drones don’t have much in common with their military counterparts. Their range and payload are very limited, and while a military drone can usually loiter over an area for hours on end, the endurance of hobbyist drones is limited to minutes rather than hours. This does not mean they cannot be used for crime; they can still invade someone’s privacy, smuggle drugs across a border, or even carry explosives. Autonomous cars are still in their infancy, so I don’t feel the need to discuss the myriad of questions they will raise.

Privacy remains one of the biggest Internet-related concerns expressed by the average person. This is understandable; we have moved so much of our daily lives to the digital sphere, placing our privacy at risk. People don’t even have to be specifically targeted to have their privacy and personal integrity compromised. Most data that makes its way online is released in the form of massive dumps following a security breach affecting many, if not all, users of a particular online service. People will continue to demand more privacy, and in turn clients will demand more security from software engineers (who aren’t miracle workers and can’t guarantee absolute security and privacy).

Mass surveillance is usually performed by governments and should not represent a threat to the average citizen or business. However, it’s still a potential threat as it can be abused by disgruntled workers, foreign governments, or by way of data breaches. The other problem is the sheer cost to the taxpayer; mass surveillance doesn’t come cheap and we will continue to see more of it.

Most governments wouldn’t bother with mass surveillance and metadata programmes if they weren’t facing very real threats. The same technology developed to keep our communications and online activities private can be abused by all sorts of individuals we wouldn’t like to meet in a dark alley. The list includes multinational crime syndicates, terrorists and insurgents. However, not all of this communication needs to be encrypted and secure. The point of propaganda is to make it widely available to anyone, and the Internet has given every crackpot with a smartphone the biggest megaphone in history, with global reach, free of charge. You can use the Internet to rally a million people around a good cause in a matter of days, but the same principles can be applied to a bad cause. If the target audience is people willing to join a death cult with a penchant for black flags, you don’t need a million people; you just need a few dozen.

The Difference Between Science And Science Fiction

For all their brilliance, the science fiction authors who helped shape popular culture in the 20th Century didn’t see the real future coming. They didn’t exactly envision the Internet, let alone its profound impact on society.

Sorry to burst your bubble, but Terminators and Artificial Intelligence (AI) aren’t a threat yet, and won’t be anytime soon. The real threats are more down to earth, but that does not mean we can afford to ignore them. You don’t need a Terminator to create havoc, all you need is a few lines of really nasty code that can disrupt the infrastructure, causing all sorts of problems. You don’t need a super-intelligent automaton from the future to cause damage. Since eBay doesn’t carry Terminators, it’s a lot easier to use an off-the-shelf drone, programmed to deliver a payload to a specific target: drugs to a trafficker, or an explosive charge to a VIP.

But these aren’t the biggest threats, they’re just potential threats: something for a Hollywood script, not a tech blog.
The real threats are criminal in nature, but they tend to stay in the cyber realm. You don’t have to physically move anything to move dirty money and information online. Law enforcement is already having a hard time keeping up with cybercrime, which seems to be getting worse. While it’s true that the crime rate in developed countries is going down, these statistics don’t paint the full picture. A few weeks ago, the British Office for National Statistics (ONS) reported a twofold increase in the crime rate for England and Wales, totalling more than 11.6 million offences. The traditional crime rate continued to fall, but the statistics included 5.1 million online fraud incidents.

The cost of physical crime is going down, but the cost of cybercrime is starting to catch up. I strongly believe the industry will have to do more to bolster security, and governments will have to invest in online security and crime prevention as well.

Just in case you are into dystopian fiction and don’t find criminal threats exciting, another frightening development would be data monopolisation: A process in which industry giants would command such a competitive lead, made possible by their vast user base, as to render competition pointless, thus stifling innovation.

Yes, I am aware that Terminators would make for a more eventful future and interesting blog post, but we’re not there yet.

The smart home technology boom is upon us. Despite lucrative projections for the market, and ever increasing numbers of connected devices, we have yet to witness much social impact from consumer adoption into the home. As a potential tipping point looms, there are several debates surrounding privacy, integration and other technical issues. Yet, there seems to be less speculation regarding why consumers still haven’t bought into the hype, nor how domestic life has improved. Considering how personal the home is, should it be concerning that those advertising these products discuss quality of life less than data, energy and ‘security’? Is the adoption of the Internet of Things into our homes inevitable, or is it already here?

Somewhere in the Near Future

The smart person returns to their certified ‘Internet – of -Things‘ smart home after a long day at work. The smart security system senses that the smart person is alone and initiates the ‘Friday Night In’ sequence. Inside, an intercom with a standardized motherly voice suggests that the smart person might want to order in tonight. The smart person unloads their things in the kitchen where the smart stove displays a selection of take out, rather than it’s default recipe guide. Following the arrival of the food, the smart person retreats to the living room to wind down, and watch some TV in their underwear. The smart TV prepares a selection of Netflix marathons categorized by mood. The smart person chooses: ‘Looking to be cheered up? Comedy Playlist’. Before starting the show, the smart person reviews a set of graphs that display the data from activity and diet throughout the day. A list of tips for smart living is generated at the bottom, one of which reads that based on the amount of consecutive nights that the Smart Person has had alone, they might consider investigating a selection of popular dating sites instead of watching TV tonight. At the slip of a thumb the smart person OKS the request and instantly a set of profiles are displayed, each chosen from a generated list of Smart Person’s tracked preferences. Suddenly, a flurry of pings and messages from other stay-at-home hopefuls fill the screen. The smart home intercom repeats aloud ‘You’ve got mail!’. The smart person fumbles for the remote and – oops” – the TV snaps a selfie in response to the flood of pings. Their image, sitting in their underwear eating noodles appears briefly on the screen before being whisked off into the ether. The flood of messages doubles only to freeze the system, causing the smart home to reboot. The house goes dark. In the now blank screen of the smart TV reflects the image of the smart person again, finally alone.

Home Smart Home

With all the debate and headlines regarding the Internet of Things, and amounts of devices connected and market valuations – is there anyone left to ask about what will happen to the home once Smart Homes take over? The keeping of a home is one of, if not the oldest traditions that we have as humans. Does the Smart Home mean the end of the home as we know it?

The home is the original place where we build our identity and mark our place in the world – the original profile. Each generation has formed its radical dwellings as their respective marks on the world. We can now look back into those past homes as windows into the past lives of those generations, their values and ambitions. What do our Smart Homes tell us about ourselves? Or perhaps instead, what is it telling everyone else?

A Brief History of the Automated Home

The process towards the automated home began almost two centuries ago now. When we first plugged our homes in, the light bulb gave us the night. No longer was man confined to the limits of the sun. The technology offered liberation from the natural hours of the day. Later, appliances replaced tools and everything that moved, or could move, became battery powered. The first generation of the automated home advertised better performance for leisure in exchange. More time for the family, or affording the once confined housewife to pursue her career as well. The automated home liberated us from the need to maintain it.

Now, the technological trend continues to carry us through the next generation into a new domesticity. Although there will always be laments for what has passed, perhaps change isn’t so bad. If there is a new liberation perhaps it is the freedom to stay home. The freedom to sit and allow the world to visit us inside. Freedom from the outdoors. Freedom from each other. Yet, this time, is the freedom coming at a cost?

The home is the original security device – the original firewall. But now, as we allow the entire world to float through our walls and into our homes, have we deflated the entire meaning of our home that has stood for millennia? We speak of security and privacy now in the context of technical systems and hardware. But have we forgotten the origin of what privacy meant? In the spaces where we were once the most intimate, by inviting the world in we are becoming the most exposed. To adopt the Smart Home, must we forfeit the home?

Given the acceptance worldwide that privacy is dead, these thoughts may be obsolete falling on ears of developers and web designers. The point though is to fundamentally question what the Smart Home is offering us in exchange for what we must give it. Effectively, is the trade worth it? Will domestic life improve as it did during the first generations of automation? Or, how do we ensure, especially as the community who may be taking part in that change, that some amount of domesticity is salvaged?

The State of the Internet of Things

The Internet of Things is one of those monster hot topic terms that when we hear it, we know of its significance, and yet may know much less about its tangible effects. We may hear that there are estimates of 200 billion devices being connected in the near future, or that the market’s value is projected at 80 billion dollars. The numbers have similar significance as knowing that the earth is 92.2 million miles away from the sun. They’re very important numbers, yet most of us don’t understand the specifics, or address the numbers daily. All that most people want to know is most cases, is how is this thing going to change, or improve our life?

What is resisting smart home technology from becoming adopted with the same ubiquity as smart phones today? Consider the early phases of smartphone technology. What caused them to make the leap from a niche device, to being fastened to the hip of nearly every person and their grandparent?

Smart phones existed for about a decade before the market saw a significant boom. The release of the iPhone in 2007 generated a major shift as Apple was the first to design and market the device for everyday use. The elements that the iPhone introduced or improved seem to be subtle interface adjustments, and yet were able to catalyze a major shift. Thus, what might be that critical tipping point, or product for smart homes?

The Issue With Niche Products

Everyday there seems to be more smart home products to outfit your home with. Yet with each new application there is a new device, with a new remote that might connect with another new app on your smart phone. No matter how potentially helpful any of these devices might be, they appear to be novel and excessive without being associate with a greater purpose.

Recently, Yves Behar released the designs for a connected garden tracking system called Edyn. The system contains two products, one to monitor the soil, and another to respond to the tracked data and irrigate the garden as necessary. With the data, the app can recommend which plants might be the easiest to grow, and what the produce might need in order to flourish.

Seems pretty handy – but is it necessary enough to become widely adopted, or to really change the way we maintain our yards? Especially considering that for many, gardening is a pleasurable activity. People often garden as a meditative tool. In fact, there was a study recently suggesting that the happier someone is, the more they garden, and the more someone gardens, the happier they are. So, what exactly is Edyn suggesting that they can improve upon in this process that gardening was without before? If gardening becomes easier, what else is to be gained in terms of time in the day? Perhaps now the smart gardener can fit in another Netflix episode, rather than do the watering themselves. Hopefully one day they won’t even have to lift a green thumb at all! Imagine all the Netflix that could be watched.

Now, I don’t mean to pick on Edyn too much, but my point is that this tool is representative of most smart home technology. A lot of it would be nice to have I suppose, but it doesn’t quite seem worth the cost and the trouble.

So What’s Next

The issue with the clunky interaction between multiple apps is now clear to the market, as there is a definite push towards who can develop the ‘hub’ for all smart home devices. Some of the key players so far are SmartThings, Wink, Apple is supposedly generating momentum through AppleTV, and Amazon has the device Echo.

However, there so far is no real front runner, nor have many homes really begun to adopt the technology. What seems to be lacking, as was the case with the iPhone, is the proper interface that can relieve the smart home technology of all of the headaches that get in the way of convenience.

Most of the processes that these technologies are looking to improve are not exactly major burdens – turning of lights, playing music etc. Thus, if the experience is hardly more enjoyable, the new technology won’t be adopted.

One product, Josh.ai, has recognized this need for cohesion between devices, and focuses its manifesto towards interfacing. Josh.ai anticipates the need for programs to develop and build over time along witht the user. Eventually, certain commands become memorized such that Josh.ai will know that every morning it should slowly raise the lights, turn on the morning news, and prepare a cup of coffee. The interface takes any work out of choosing between apps, and instead responses to voice commands to manage all connected devices. Josh.ai is advertised as your home’s best friend. You even talk to Josh.ai as if its your best friend.

Josh.ai bases its product around the use of voice rather than any other interface. This may be a critical move for smart home technology. Josh.ai’s platform essentially asks how can the user program their home with as little work as possible? The hope is that even complex processes like morning routines might be programmed by voice, then memorized for future instances. “Hey Josh, when I get home can you turn up the lights, set the oven to 300, and maybe put on something like Frank Sinatra? I’m bringing a friend home!”

If Josh.ai is one of the more optimistic potential hubs to move the Smart Home trend into the next stage, what does its manifesto tell us about the ambition of what we can expect from the technology in near future?

Very often does the phrase ‘make your life easier and more productive’ occur in many of these manifestos. But, most of what these hubs offer is the autonomous control of small things like lights and music, how much easier will our lives get? Can this at all compare to the transition from brooms to vacuums?

The three essential points of Josh.ai are: thoughtless energy saving, continuous awareness of devices, and a more networked system. How do these new values compare to past technologies that significantly liberated the average person? In the manifestos of most of these hub devices, there is hardly any language that appears human at all.

Although the benefits of some of the products seem underwhelming, the issue is that even the smallest thing poses a significant risk. Take Nest for example, what could be more harmless than a thermostat? Yet already in its relatively short life has Nest been now caught for the sale of information surrounding home fire history to insurance companies.

Even if the Smart Home Hub were your ‘best friend’, do you want your best friend to know everything about your preferences? How much do we even allow our actual best friends to know about ourselves? Do we tell them that we’re lonely? Who’s profiles we look at online? Even if our best friend were really good at keeping secrets, would we tell them? About that one time?

Anytime that someone cries out against the inevitable, they come across as a cranky soap boxer. But this is an odd transition where there seems to be little gained by most of these products, and yet there is such significant risk in what might be lost if the technologies are adopted. Doesn’t it feel strange that in the last century our parents and grandparents marched for privacy, and here we are eagerly handing it back?

Nevermind Game of Thrones, winter is not coming, at least not in the northern hemisphere. It’s summertime, and here at Toptal that usually means many of our freelance developers and designers are either on the road or getting ready to hit the beach.

The below article first appeared on Toptal and is reproduced in it’s entirety.

Life was great for Microsoft developers 10 years ago. Companies were happy with going 100 percent Microsoft for their development projects. With ASP.NET on the frontend, .NET middle-tier and SQL Server on the backend, things worked very well for the most part. When they didn’t, developers just accepted that as something that came with territory. Microsoft was all but running the show. Then, at the end of the last decade, Microsoft’s 800-pound gorilla status started to unravel. Maybe it was due to the introduction of the iPhone and Microsoft missing the shift to mobile, or maybe it was because of the proliferation of open source projects, but things changed, and today those same companies need to be persuaded that going with Microsoft Stack is a good idea. This article presents eight reasons in favor of sticking with the Microsoft software stack.

Reason #1: .NET is Still One of the Best

Introduced more than 10 years ago, .NET Framework is feature-rich and thoroughly battle-tested. While it was commonplace to have to combine native development with managed code in the early days of .NET, the vast majority of development tasks are supported out of the box today. Even companies such as Oracle released components that are 100 percent .NET managed code (i.e. ODP.NET managed driver) to interface with their products. .NET API is consistent, well documented and used by millions.

The knowledge-base available via MSDN, StackOverflow and thousands of forums and blogs is massive. In my years of developing in .NET, I cannot recall an instance where I would get stuck for long on a framework bug; each time, someone had already experienced, researched and posted an answer, not always the answer I was hoping for, but still something that moved me forward. With the upcoming 2015 release, .NET Core will be open-source and available on non-Windows systems.

Reason #2: ASP.NET has Evolved

Looking back at the traditional web-to-database Microsoft stack from 10 years ago, it’s interesting to see which parts survived the test of time and which parts faded away. While the back-end of the Microsoft stack remained pretty much unchanged (we still use the same set of patterns and components, such as Dependency Injection, Tasks, Linq, EF or ADO) the front-end, the ASP.NET piece, saw a fundamental shift from “do it the Microsoft way “ (i.e. Web Forms) to “do it your way and use ASP.NET as a platform.” Today, ASP.NET is an MVC-based framework featuring robust infrastructure for authentication, bundling and routing that integrates with many non-Microsoft technologies such as Bootstrap and AngularJS. ASP.NET sites look nice on a wide range of form-factors, from phones to desktops, and its Web API capabilities make exposing web services a breeze. The framework has been open-source for a number of years, so if you get stuck on a problem, the source is available on GitHub. ASP.NET has changed, and changed for the better.

Reason #3: Simplicity of Web API and Power of WCF

My all-time favorite quote is from Alan Kay who said, “Simple things should be simple; complex things should be possible”. When Windows Communication Foundation (WCF) first came out in 2006, it was anything but simple; behaviors, endpoints, and bindings were overwhelming. So, Microsoft released Web API, an easy-to-use framework that makes exposing HTTP web services a piece of cake. With a few lines of configuration, your API turns into a secure, “industry-standard” web service.

If your use case does not fit the “standard” mold, and you need full control over how your API is exposed over the wire, you can always fall back on WCF. With the myriad of configuration options and hooks, WCF lets you custom-serialize your data, log, intercept, route message, use peer-to-peer and queuing, and much-much more. Web API, together with WCF, delivers on both tenets of Kay’s quote: if you need a simple web service, you are done in minutes with Web API; if your service requirements are complex, “all” is possible with WCF. These two technologies provide a comprehensive coverage of service scenarios and come prepackaged with the .NET framework.

Reason #4: SQL Server is as Solid as Ever

For many years, it seemed like the tidal wave of new development languages, frameworks and patterns came through the front and middle tiers and spared the database back-end. After all, the good old “SELECT” is still as much in use today as it was 20 years ago. I suppose this is due to the fact that many companies view their data as the core of their business, and keeping the integrity of that core far outweighs the excitement of trying “something new” at the database layer.

SQL Server excels at its primary role of a data keeper with a myriad of features for transactions, referential integrity, backups, mirroring and replication, but what sets SQL Server apart from competition is how well it integrates with the rest of the Microsoft stack. For rapid development, there is the Entity Framework, currently in version 6, passed adolescence and delivering well on its promise of streamlining data access. If you need computing power, the .NET Framework is loaded in-process with SQL Server, meaning you can embed .NET code as stored procedures, functions or aggregates without sacrificing performance. Pair that with the fact that SQL Server 2014 comes with in-memory tables, and you can come up with some pretty slick real-time solutions that could not be made fast enough solely with SQL and regular tables. After years in the industry, SQL Server is still on top of my list of RDBMSs.

Reason #5: It’s Easily Testable

So many times, working in corporate IT, I saw software turning into these untouchable black boxes because there were no tests, and nobody wanted to mess with the code for fear of “breaking something else”. Then, I worked on systems that had thousands of tests, and it was a great feeling to be able to tell business that, “yes, we can make these changes,” years after software had been released. The Microsoft stack is designed with testability in mind. ASP.NET MVC has hooks for dependency injection, and in version 5, dependency injection will be included in the framework itself. In the middle tier, it’s a similar story: we use dependency injection to disassociate implementation from the interface, which lets us swap production types with mocks at test time. Even on the database side, there are SQL Server Data Tools that come with templates for testing against the stored procedure layer. Testing is an inseparable part of the software development process today, and the Microsoft stack comes well equipped for this new reality.

When it comes to support, it’s nice to have a range of options, starting with community forums and ending with an actual live human being working on-site on your server. The online ecosystem for Microsoft products is one of the largest in the industry. After all, Microsoft was started by Bill Gates, a software developer himself, who saw the wide adoption by developers as the key to proliferation of Microsoft products. That meant providing these developers with lots and lots of support.

Microsoft was among the first to encourage its employees to blog about the technology they were working on, and while the rest of the industry has certainly caught up, the amount and the quality of instructional videos, guides and articles coming directly from Microsoft today is still very impressive. That layer of quality online content is supplemented by a large number of community-based support ecosystems such as StackOverflow, which are not as consistent when it comes to content quality, but are, nevertheless, far more helpful than not.

Lastly, there is always an option to pick up the phone and call Microsoft support. I rarely had to use it, but there were a handful of production emergencies when having Microsoft devs analyze core dumps saved the day. The range of support options is clearly a factor in favor for going with the Microsoft stack.

Reason #7: Microsoft Sticks to their Products

A few years back, choosing Microsoft Silverlight as the front-end for an application seemed like a valid choice, but that’s no longer the case. With the mobile trend in full swing and JavaScript frameworks dominating the front-end space, Silverlight is no longer a feasible option; nevertheless, it’s still supported by Microsoft through 2021. Microsoft sticks to its guns, which is good for those of us who have to make technology choices without having a magic eight ball to tell us what technology trend will dominate the software landscape in the future. Going with the Microsoft stack ensures that time and money is invested into technology that will be supported even if it falls out of favor with the industry.

Reason #8: Visual Studio Umbrella

A decade ago, I was spending about 50 percent of my time working in Visual Studio and about 50 percent in other tools. Today, the split is overwhelmingly in favor of Visual Studio. Microsoft’s vision for Visual Studio to be a one-stop solution for hosting IDEs is coming to fruition with many Microsoft and non-Microsoft products offering some level of integration with Visual Studio. From database development with SQL Server Data Tools to writing iPad and Android apps with Xamarin, Visual Studio provides a familiar developer experience with a consistent user interface. The same can be said about working with Microsoft Azure, a cloud platform encompassing a variety of services from database hosting to mobile services.

Visual Studio obfuscated the complexities of distributed cloud infrastructure making the experience of developing cloud applications consistent with that of developing applications not hosted in the cloud. All the pieces seem to fit together nicely under the umbrella of Visual Studio, making the overall development process very efficient.

Microsoft Stack – The Best of Both Worlds

Today, there are far more choices for writing quality software compared to 10 years ago. That is certainly a good thing because competition forces big players, such as Google, Apple, Amazon and Microsoft, to continue to innovate and not get complacent. While Microsoft has been pushed from the top of the mountain by the tech evolution of the past decade, the company has shown that it’s willing to adapt and is attuned to realities of the current technological trends. ASP.NET embraced other technologies and methodologies, many of them open source, with the original Web Forms fading into history. The .NET framework continues to evolve, breaking new frontiers with libraries for multi-threading and many-core computing. With the imminent 2015 release, the core of the framework will be open-source and portable to non-Windows platforms, which is a step in the direction of inclusiveness and transparency.

These welcomed improvements come from a company that has long-established processes for releasing software that’s tested, documented and supported. Going with the Microsoft stack brings the excitement of working with modern languages and frameworks plus the stability of being backed by a software giant with decades of experience in the development industry. This is why I am recommending the Microsoft stack today.

We are only human, and one of the traits of being a human is that we make mistakes. On the other hand, we are also self-correcting, meaning we tend to learn from our mistakes and hopefully are thereby able to avoid making the same ones twice. A lot of the mistakes I have made in the WordPress realm originate from trying to save time when implementing solutions. However, these would typically rear their heads down the road when issues would crop up as a result of this approach. Making mistakes is inevitable. However, learning from other people’s oversights (and your own of course!) is a road you should proactively take.

Common Mistake #1: Keeping the Debugging Off

Why should I use debugging when my code is working fine? Debugging is a feature built into WordPress that will cause all PHP errors, warnings, and notices (about deprecated functions, etc.) to be displayed. When debugging is turned off, there may be important warnings or notices being generated that we never see, but which might cause issues later if we don’t deal with them in time. We want our code to play nicely with all the other elements of our site. So, when adding any new custom code to WordPress, you should always do your development work with debugging turned on (but make sure to turn it off before deploying the site to production!).

To enable this feature, you’ll need to edit the wp-config.php file in the root directory of your WordPress install. Here is a snippet of a typical file:

[cc lang=”php”]

// Enable debugging
define(‘WP_DEBUG’, true);

// Log all errors to a text file located at /wp-content/debug.log
define(‘WP_DEBUG_LOG’, true);

// Don’t display error messages write them to the log file /wp-content/debug.log
define(‘WP_DEBUG_DISPLAY’, false);

// Ensure all PHP errors are written to the log file and not displayed on screen
@ini_set(‘display_errors’, 0);

[/cc]
This is not an exhaustive list of configuration options that can be used, but this suggested setup should be sufficient for most debugging needs.

Common Mistake #2: Adding Scripts and Styles Using wp_head Hook

What is wrong with adding the scripts into my header template? WordPress already includes a plethora of popular scripts. Still, many developers will add additional scripts using the wp_head hook. This can result in the same script, but a different version, being loaded multiple times.

Enqueuing here comes to the rescue, which is the WordPress friendly way of adding scripts and styles to our website. We use enqueuing to prevent plugin conflicts and handle any dependencies a script might have. This is achieved by using the inbuilt functions wp_enqueue_script or wp_enqueue_style to enqueue scripts and styles respectively. The main difference between the two functions is that with wp_enqueue_script we have an additional parameter that allows us to move the script into the footer of the page.

[cc lang=”php”]

wp_register_script( $handle, $src, $deps = array(), $ver = false, $in_footer = false )
wp_enqueue_script( $handle, $src = false, $deps = array(), $ver = false, $in_footer = false )

wp_register_style( $handle, $src, $deps = array(), $ver = false, $media = ‘all’ )
wp_enqueue_style( $handle, $src = false, $deps = array(), $ver = false, $media = ‘all’ )

[/cc]
If the script is not required to render content above the fold, we can safely move it to the footer to make sure the content above the fold loads quickly. It’s good practice to register the script first before enqueuing it, as this allows others to deregister your script via the handle in their own plugins, without modifying the core code of your plugin. In addition to this, if the handle of a registered script is listed in the array of dependencies of another script that has been enqueued, that script will automatically be loaded prior to loading that highlighted enqueued script.

Common Mistake #3: Avoiding Child Themes and Modifying WordPress Core Files

Always create a child theme if you plan on modifying a theme. Some developers will make changes to the parent theme files only to discover after an upgrade to the theme that their changes have been overwritten and lost forever.

To create a child theme, place a style.css file in a subdirectory of the child theme’s folder, with the following content:

[cc lang=”php”]

/*
Theme Name: Twenty Sixteen Child
Theme URI: http://example.com/twenty-fifteen-child/
Description: Twenty Fifteen Child Theme
Author: John Doe
Author URI: http://example.com
Template: twentysixteen
Version: 1.0.0
License: GNU General Public License v2 or later
License URI: http://www.gnu.org/licenses/gpl-2.0.html
Tags: light, dark, two-columns, right-sidebar, responsive-layout, accessibility-ready
Text Domain: twenty-sixteen-child
*/

[/cc]
The above example creates a child theme based on the default WordPress theme, Twenty Sixteen. The most important line of this code is the one containing the word “Template” which must match the directory name of the parent theme you are cloning the child from.

The same principles apply to WordPress core files: Don’t take the easy route by modifying the core files. Put in that extra bit of effort by employing WordPress pluggable functions and filters to prevent your changes from being overwritten after a WordPress upgrade. Pluggable functions let you override some core functions, but this method is slowly being phased out and replaced with filters. Filters achieve the same end result and are inserted at the end of WordPress functions to allow their output to be modified. A trick is always to wrap your functions with if ( !function_exists() ) when using pluggable functions since multiple plugins trying to override the same pluggable function without this wrapper will produce a fatal error.

Common Mistake #4: Hardcoding Values

Often it looks quicker to just hardcode a value (such as a URL) somewhere in the code, but the time spent down the road debugging and rectifying issues that arise as a result of this is far greater. By using the corresponding function to generate the desired output dynamically, we greatly simplify subsequent maintenance and debugging of our code. For example, if you migrate your site from a test environment to production with hardcoded URLs, all of a sudden you’ll notice your site it is not working. This is why we should employ functions, like the one listed below, for generating file paths and links:

[cc lang=”php”]

// Get child theme directory uri
stylesheet_directory_uri();
// Get parent theme directory
get_template_directory_uri();
// Retrieves url for the current site
site_url();

[/cc]
Another bad example of hardcoding is when writing custom queries. For example, as a security measure, we change the default WordPress datatable prefix from wp_ to something a little more unique, like wp743_. Our queries will fail if we ever move the WordPress install, as the table prefixes can change between environments. To prevent this from happening, we can reference the table properties of the wpdb class:

[cc lang=”php”]

global $wpdb;
$user_count = $wpdb->get_var( “SELECT COUNT(*) FROM $wpdb->users” );

[/cc]
Notice how I am not using the value wp_users for the table name, but instead, I’m letting WordPress work it out. Using these properties for generating the table names will help ensure that we return the correct results.

Common Mistake #5: Not Stopping Your Site From Being Indexed

Why wouldn’t I want search engines to index my site? Indexing is good, right? Well, when building a website, you don’t want search engines to index your site until you have finished building it and have established a permalink structure. Furthermore, if you have a staging server where you test site upgrades, you don’t want search engines like Google indexing these duplicate pages. When there are multiple pieces of indistinguishable content, it is difficult for search engines to decide which version is more relevant to a search query. Search engines will in such cases penalize sites with duplicate content, and your site will suffer in search rankings as a result of this.

As shown below, WordPress Reading Settings has a checkbox that reads “Discourage search engines from indexing this site”, although this does have an important-to-note underneath stating that “It is up to search engines to honor this request”.

WordPress Reading Settings

Bear in mind that search engines often do not honor this request. Therefore, if you want to reliably prevent search engines from indexing your site, edit your .htaccess file and insert the following line:

[cc lang=”php”]

Header set X-Robots-Tag “noindex, nofollow”
Common Mistake #6: Not Checking if a Plugin is Active

[/cc]

Why should I check if a plugin function exists if my plugin is always switched on? For sure, 99% of the time your plugin will be active. However, what about that 1% of the time when for some reason it has been deactivated? If and when this occurs, your website will probably display some ugly PHP errors. To prevent this, we can check to see if the plugin is active before we call its functions. If the plugin function is being called via the front-end, we need to include the plugin.php library in order to call the function is_plugin_active():

[cc lang=”php”]

include_once( ABSPATH . ‘wp-admin/includes/plugin.php’ );
if ( is_plugin_active( ‘plugin-folder/plugin-main-file.php’ ) ) {
// Run plugin code
}

[/cc]
This technique is usually quite reliable. However, there could be instances where the author has changed the main plugin directory name. A more robust method would be to check for the existence of a class in the plugin:

[cc lang=”php”]

if( class_exists( ‘WooCommerce’ ) ) {
// The plugin WooCommerce is turned on
}

[/cc]
Authors are less likely to change the name of a plugin’s class, so I would generally recommend using this method.

Common Mistake #7: Loading Too Many Resources

Why should we be selective in loading plugin resources for pages? There is no valid reason to load styles and scripts for a plugin if that plugin is not used on the page that the user has navigated to. By only loading plugin files when necessary, we can reduce our page loading time, which will result in an improved end user experience. Take, for example, a WooCommerce site, where we only want the plugin to be loaded on our shopping pages. In such a case, we can selectively remove any files from being loaded on all the other sites pages to reduce bloating. We can add the following code to the theme or plugin’s functions.php file:

[cc lang=”php”]

function load_woo_scripts_styles(){

if( function_exists( ‘is_woocommerce’ ) ){
// Only load styles/scripts on Woocommerce pages
if(! is_woocommerce() && ! is_cart() && ! is_checkout() ) {

// Dequeue scripts.
wp_dequeue_script(‘woocommerce’);
wp_dequeue_script(‘wc-add-to-cart’);
wp_dequeue_script(‘wc-cart-fragments’);

// Dequeue styles.
wp_dequeue_style(‘woocommerce-general’);
wp_dequeue_style(‘woocommerce-layout’);
wp_dequeue_style(‘woocommerce-smallscreen’);

}
}
}

add_action( ‘wp_enqueue_scripts’, ‘load_woo_scripts_styles’);

[/cc]
Scripts can be removed with the function wp_dequeue_script($handle) via the handle with which they were registered. Similarly, wp_dequeue_style($handle) will prevent stylesheets from being loaded. However, if this is too challenging for you to implement, you can install the Plugin Organizer that provides the ability to load plugins selectively based on certain criteria, such as a post type or page name. It’s a good idea to disable any caching plugins, like W3Cache, that you may have switched on to stop you from having to refresh the cache constantly to reflect any changes you have made.

Common Mistake #8: Keeping the Admin Bar

Can’t I just leave the WordPress Admin Bar visible for everyone? Well, yes, you could allow your users access to the admin pages. However, these pages very often do not visually integrate with your chosen theme and don’t provide a seamless integration. If you want your site to look professional, you should disable the Admin Bar and provide a front-end account management page of your own:

[cc lang=”php”]

add_action(‘after_setup_theme’, ‘remove_admin_bar’);

function remove_admin_bar() {
if (!current_user_can(‘administrator’) && !is_admin()) {
show_admin_bar(false);
}
}

[/cc]
The above code, when copied into your theme’s functions.php file will only display the Admin Bar for administrators of the site. You can add any of the WordPress user roles or capabilities into the current_user_can($capability) function to exclude users from seeing the admin bar.

Common Mistake #9: Not Utilizing the GetText Filter

I can use CSS or JavaScript to change the label of a button, what’s wrong with that? Well, yes, you can. However, you’re adding superfluous code and extra time to render the button, when you can instead use one of the handiest filters in WordPress, called gettext. In conjunction with a plugin’s textdomain, a unique identifier that ensures WordPress can distinguish between all loaded translations, we can employ the gettext filter to modify the text before the page is rendered. If you search the source code for the function load_plugin_textdomain($domain), it will give you the domain name we need to override the text in question. Any reputable plugin will ensure that the textdomain for a plugin is set on initialization of the plugin. If it’s some text in a theme that you want to change, search for the load_theme_textdomain($domain) line of code. Using WooCommerce once again as an example, we can change the text that appears for the “Related Products” heading. Insert the following code into your theme’s functions.php file:

[cc lang=”php”]

function translate_string( $translated_text, $untranslated_text, $domain ) {
if ( $translated_text == ‘Related Products’) {
$translated_text = __( ‘Other Great Products’, ‘woocommerce’ );
}
return $translated_text;
}

add_filter( ‘gettext’, ‘translate_string’, 15, 3 );

[/cc]
This filter hook is applied to the translated text by the internationalization functions __() and _e(), as long as the textdomain is set via the aforementioned functions.

[cc lang=”php”]

_e( ‘Related Products’, ‘woocommerce’ );

[/cc]
Search your plugins for these internationalization functions to see what other strings you can customize.

Common Mistake #10: Keeping the Default Permalinks

By default, WordPress uses a query string with the post’s ID to return the specified content. However, this is not user-friendly and users may remove pertinent parts of the URL when copying it. More importantly, these default permalinks do not use a search engine friendly structure. Enabling what we call “pretty” permalinks will ensure our URLs contain relevant keywords from the post title to improve performance in search engine rankings. It can be quite a daunting task having to retrospectively modify your permalinks, especially if your site has been running for a significant period of time, and you’ve got hundreds of posts already indexed by search engines. So after you’ve installed WordPress, ensure you promptly change your permalinks structure to something a little more search engine friendly than just a post ID. I generally use the post name for the majority of sites I build, but you can customize the permalink to whatever format you like using the available permalink structure tags.

WordPress Permalink Settings

Conclusion

This article is by no means an exhaustive list of mistakes made by WordPress developers. If there’s one thing you should take away from this article, though, it’s that you should never take shortcuts (and that’s true in any development platform, not just in WordPress!). Time saved now by poor programming practices will come back to haunt you later. Feel free to share with us some mistakes that you have made in the past – and more importantly any lessons learned – by leaving a comment below.

WordPress is easily the most powerful open source blogging and content management system available online today, and so working knowledge of its intricacies is a boon to any developer or designer resume.

This resource contains a collection of WordPress development best practices and WordPress development tips provided by our Toptal network members. As such, this page will be updated on a regular basis to include additional information and cover emerging techniques. This is a community driven project, so you are encouraged to contribute as well, and we are counting on your feedback.

Check out the Toptal resource pages for additional information on WordPress development; there is a WordPress developer hiring guide and WordPress developer interview questions.

Also, here is a detailed guide on selecting web hosts based on various parameters like customer support, types of web hosting offered, technical reliability and ease of use.

Use Child Themes and Plugins

A lot of newcomers to WordPress dive right in and start modifying their core theme files. This is a definite mistake. All of your changes will disappear right after an upgrade, and since plugins and themes are updated about as often as apps on your phone, this is pretty frequently.

To avoid this, create children of your plugins and themes. Not only will you preserve your changes, you can upgrade on your own time. The same steps used to create a child theme can be applied to creating a child plugin, but let’s use creating a child theme as our example.

To get started making your child theme, create a new folder in your themes folder with a unique name, then create a style.css file in your new folder.

In WordPress, all theme parameters are stored in the style.css file as the first comment block. Open the style.css from your original theme, the parent, to see an example of this.

For your child theme, go ahead and copy that comment block from your original theme’s style.css to the new file and rename it. Adding the template parameter to this header will link your new theme to the original. The Template parameter should point to the folder name of the original theme, like so:

[cc lang=”css”]
/*Theme Name: Twenty Fourteen Child Theme
Description: My new child theme based on Twenty Fourteen
Template: twentyfourteen
Version: 1.0
*/
[/cc]

Now, if you want to modify files of the original theme, copy them from the original theme’s folder and paste them into your child theme folder. WordPress will use original template files unless it sees the same file in your child theme. So if you want to make changes to header.php, copy it from your original theme into your new child theme folder, and make the changes in the copy. For adding new or modified code, you likewise create a new functions file in your child theme and make your changes there.

I am getting old. Back in my day, if you wanted top notch CPU performance, you had to go with a high-end x86 chip, or, if you had deeper pockets, you could get something exotic, like a PowerPC system. The industry’s dependence on x86 processors appeared to be increasing, not declining.

Ten years ago, Apple joined the x86 club, and this prompted many observers to conclude the era of non-x86 processors in the mass market was over. Just a few years later, they had to eat their words, and yet again, Apple had something to do with it. ARM servers are coming, and they could revitilise the server industry.

Rethinking Processor Design

As the paradigm shifted and mainstream users embraced smartphones and tablets, it quickly became apparent that x86 chips from Intel, AMD, and VIA, simply weren’t up to the task. While x86 was the most prolific instruction set on the planet, it wasn’t a good choice for mobile devices for a number of reasons. In fact, Intel’s instruction set still isn’t a popular choice for mobile processors, although this is starting to change thanks to Intel’s foundry tech lead. In any case, when it comes to this market segment, x86 is not as efficient as other CPU architectures out there, namely processors based on ARM’s 32-bit ARMv7 and 64-bit ARMv8 instruction sets.

Over the past decade, and especially over the past five years, ARM processors have come to dominate the smartphone and tablet landscape, and they had a lot going for them. They offered a lot of performance-per-watt, they were cheap to design, produce and deploy. Big vendors could buy the necessary building blocks and design their own processors based on ARMv7 or ARMv8, adding other components according to their needs (high-speed modems and different GPUs to name a couple).

This led some chip designers to take a somewhat different approach and design their own, custom CPU cores. Qualcomm and Apple led the way, both companies became big players on the mobile System-on-Chip (SoC) market, and their development of sophisticated, custom cores, played an instrumental role in their success. However, custom ARM cores were still used in high-end processors, while all other market segments were covered by standard ARM Cortex CPU cores, like the 32-bit Cortex-A8, A9, A7, and A15, followed up by 64-bit designs like the Cortex-A53, A57, and the new A72 core, which is about to start shipping.

The other prerequisite for ARM’s success was Microsoft’s failure.

Windows only ran on x86 processors, so if Microsoft were to gain a foothold in mobile, it would tip the scales in Intel’s favour. However, by the close of the last decade, it became apparent that Redmond had dropped the ball and ceded this lucrative market to Google and Apple. Speaking of balls, long-time Microsoft CEO Steve Ballmer departed the company a couple of years ago, admitting he and his team failed to recognize the potential of smartphone and tablets. Anyway, it’s not Ballmer’s problem anymore: he has other balls on his mind right now, basketballs to be exact.

However, mobile is not the first or only market segment to witness a Microsoft failure of epic proportions. The other is the server market. On the face of it, smartphones and data centres don’t have much in common, but from a technological and business perspective, they have some overlap.

Whether you’re designing a smartphone or a server, you need to emphasise similar aspects of your hardware platform, such as power efficiency, good thermals, performance per dollar, and so on. Most importantly, youdon’t really need an x86-based processor for smartphones and many types of servers. Thanks to Microsoft’s failures, these market segments aren’t dominated by any flavour of Windows. They rely on UNIX-based operating systems instead: Android, iOS, and various Linux distributions.

Microsoft also attempted to leverage the potential of ARM processors, so it tried developing a version of Windows that would run on ARM hardware, which conveniently brings me to the next Microsoft fail: Windows RT. Microsoft eventually pulled the plug on Windows RT, or “Windows on ARM” as it was originally called. Microsoft’s latest Surface tablets employ x64 processors and standard Windows 10. Microsoft’s Lumia smartphone line (née Nokia Lumia) still uses ARM processors from the House of Qualcomm, but Windows Phone is all but dead as a mainstream smartphone platform.

Servers Don’t Have To Cost An ARM And A Leg

Right now, we have a couple of billion smartphone and tablets in the wild, and the vast majority are based on ARM processors. However, ARM chips aren’t making their way into other market segments. There are just a handful high-volume computing platforms based on ARM that don’t fall into the smartphone and tablet category. Google Chromebooks are probably the best-known example. However, ARM chips are used in heaps of other devices: routers, set-top boxes and smart TVs, smartwatches, some gaming devices, automotive infotainment systems and so on.

What about ARM servers?

This is where it gets tricky. I’ve been hearing talk of ARM servers since 2010, but progress has been slow and limited. ARM’s market share in the server segment remains negligible and the ecosystem remains dominated by x86 Xeon and Opteron parts from Intel and AMD respectively. Since AMD is in a world of trouble on the CPU front, Intel has managed to extend its market share lead in recent years.

But why did ARM servers sound like a good idea to begin with?

Money. I could try listing all the geeky points that make ARM a viable alternative to x86 in the server market, but at the end of the day it’s mostly about money, so I will attempt to explain it in a few lines.

• Price/Performance
• Data centre workloads are evolving and changing
• Ability to source processors from various suppliers
• Use of custom designed chips for various niches
• ARM chips are more suitable to some infrastructure applications
• It’s a good way to stick it to Intel and erode its market position (Intel is on the verge of becoming a monopoly in the server space)

We don’t need a huge and expensive Xeon processor for everything. Moreover, using obsolete x86 processors to handle undemanding workloads isn’t a good option due to their power draw. Remember, we are talking about servers, not your MacBook or desktop PC. Servers run around the clock, so every efficiency gain, including relatively small ones, tends to be important. It’s not just about getting a bigger electric bill; data centres have to be cooled and maintained, so processors with a lower Thermal Design Power (TDP) rating are much more valuable to enterprise users than individuals.

Why Use ARM Servers?

So, what sort of enterprise application are ARM processors good for?

Well, ARM expects to get the vast majority of design wins for networking infrastructure applications. Due to their flexibility, small size, efficiency, and low price, ARM processors are a great choice for infrastructure. You can use ARM processors in routers, high-performance storage solutions, and certain types of servers.

However, ARM expects the majority of enterprise growth this decade to come from servers since its other segments are already mature and it has a healthy market share in them. Server workloads are changing as well, and this trend is tied to the growth of cloud services. As a result, servers have to deal with an increasing number of smaller tasks.

Many organisations prefer to keep their options open, so they source hardware from multiple vendors. This is good news for ARM server processors because they could be marketed by a number of different companies. In addition, ARM’s licensing policies and modular approach to processor design can be utilised to design custom processors for specific applications. This is, obviously, something that’s not an option for small companies, but what might happen if big players like Amazon, Facebook or Google start asking for bespoke server processors, designed to excel at one particular application?

As for “sticking it to Intel,” I should note that I don’t mean Intel any harm, and I don’t want to see it fail or be pushed out of various market segments, but at the same time I am concerned that Intel’s domination could end up stifling growth and innovation. More competition should result in lower prices for end-users, and this is what ARM servers are all about.

Only a decade ago, multicore x86 processors were reserved for high-performance computers and servers, but now you can get quad-core x86 chips in $100 tablets.

In the early days of multicore computing, you still needed big CPU cores to get adequate levels of performance. Lots of software was not able to take advantage of these new processors and their extra cores, so good single-thread performance was vital. Things sure have changed; nowadays, we have octa-core smartphones, quad-core Intel tablets and phones, and 16-core x86 server processors.

There is a good reason for this. Building a multi-core processor makes perfect sense from a technological and financial perspective. It’s a lot easier to distribute the load to a few smaller, more efficient CPU cores than to develop a single, huge core capable of running at high frequencies. The multicore approach ensures superior efficiency and chip yields.

ARM has the potential to take the core craze to the next level. ARM CPU cores tend to be smaller than Intel’s so called “big cores” used in server and desktop parts (Intel’s “small core” Atoms are reserved for mobile, although Atom-based server parts are available too). However, this does not mean we will see 128-core or 256-core ARM processors anytime soon, although in theory, they are possible. It depends on how the new crop of ARMv8 server processors handles multithreaded loads. There are some encouraging signs, and chances are that ARM servers will be a good choice for a range of workloads that could benefit from their multicore processors.

Qualcomm’s first server processor has 24 ARMv8 CPU cores, and the chipmaker made it clear that future models will sport even more cores. Remember AMD and its server market woes? Well, the company introduced its long overdue ARM-based Opteron A1100 processor just a couple of weeks ago. Qualcomm made the announcement in October, so both these products will become available over the next few months.

Of course, Intel will not be attending this ARM party, but Qualcomm and AMD are not the only chip outfits working on ARM-based enterprise chips. Chipmakers like Broadcom, Calxeda, Cavium Networks and Huawei HiSilicon have worked on ARM-based server products as well. Nvidia and Samsung, two heavyweights in the SoC and GPU business, also experimented with ARM server parts until a couple of years ago when they decided to halt development. Texas Instruments, Xilinx and Marvell are also exploring ARM server parts.

Some of these companies worked on custom ARM cores, too, but the only non-Apple 64-bit custom ARM core available today is Nvidia’s Denver, which only got a handful of design wins.

What Are ARM Custom Cores?

I know most people can’t be bothered keeping track of all industry niches, including the CPU space, so I think now would be a good time to explain what makes ARM cores different and what custom cores actually are. I will not dissect processors and explain the difference between x86 and ARM instruction sets, but I will outline the differences from a business perspective.

You see, ARM isn’t different just because it uses a different instruction set, although that would make for a quick and geeky explanation, in my opinion, the biggest difference between Intel, AMD and ARM is not the architecture, it’s the business model. Besides, architectures change, new CPU designs are unveiled on a regular basis, but ARM’s approach to marketing and licensing its technology hasn’t changed in years.

Here is a simple example.

An Intel processor is developed by Intel, using Intel instruction sets. It is manufactured in an Intel foundry, packaged and shipped with “Intel Inside” branding. It might sound simple, but let’s not forget the billions that went into R&D over the decades, or the fact that Intel relies on its own fabs for manufacturing (and if you are in the market for a 14nm foundry, make sure you have some spare change on you, because a chip fab costs as much as a nuclear aircraft carrier).

What about ARM products? Well, ARM is not a chipmaker, it’s a chip designer, or a “fabless” chip company, so it doesn’t deal with manufacturing and does not sell own-brand chips. ARM sells something much more interesting: intellectual property. This means ARM clients can choose any of a number of different licensing plans and start making their own designs. Most of them choose ARM’s in-house designs (Cortex series CPUs, Mali series GPUs), so they pay a licensing fee for every CPU/GPU core they produce.

However, a client does not have to license these ready-to-go CPUs; it can license the architecture set instead and develop a custom core based on an ARM instruction set. This is what Apple does. It uses the ARMv8 instruction set to build big and powerful 64-bit CPU cores for its iOS devices. Nvidia’s Denver CPU is similar in this respect, and so are Qualcomm’s custom cores (32-bit Krait and 64-bit Kryo series).

Designing a custom CPU core is not easy. It’s not like you’ll find chip designers out of work and offering to design a custom processor on Craigslist, so this approach is usually reserved for big players who have the necessary technical, financial and human resources to pull it off. Therefore, most companies use off-the-shelf ARM Cortex cores instead (the 64-bit Cortex-A57 core can be employed in a server environment and it’s used by most next-generation ARM server processors).

It is important to note that virtually ARM-based chips are custom designed but the CPU cores used in most are not.

The vast majority of ARM processors rely on standard ARM CPU designs (Cortex CPUs) rather than custom CPU cores. This means chipmakers can choose any of a number of ARM CPU cores, third-party GPUs and other components, and tailor a processor to meet their needs without having to develop a custom CPU core. It’s a cheap way of making the architecture more flexible, and it has more to do with ARM’s licensing policies than engineering.

It is also important to note that these forthcoming ARM servers, based on the latest ARM 64-bit CPU architecture, don’t have much in common with the experimental ARM servers of years gone by. For example, one of our colleagues played around with Scaleway ARM servers, but they are based on ARMv7 processors and have a number of hardware limitations (for example, Scaleway used shared I/O controllers, and the lack of 64-bit support created another set of challenges). The new generation of ARM-based servers won’t suffer from these teething problems; they are much closer to Intel hardware in terms of features and standards.

ARM Server Pros And Cons

The problem with ARM servers is that they tend to be used for small niches, and they are not suitable for small developers who can live with any server. While some big companies find them attractive, the ARM servers that are currently available aren’t suitable for most individual developers.

However, the forthcoming server solutions are different and should appeal to more niches. This is what could make them appeal to a much wider user-base:

• Reduced hardware costs, potentially superior efficiency (performance per dollar, performance per Watt).
• Increasing compatibility and availability of popular ports.
• Support for cutting-edge technology and new industry standards.
• Ability to excel at certain types of workloads (simple but multithreaded loads).
• Potential for more competition and product diversity than in x86 space.

I have to stress that, at this stage, some of these points are theoretical since the hardware isn’t out yet. However, while I can’t categorically claim to know what will happen over the next few quarters, I am confident the new breed of ARM servers will deliver these (and more) benefits. Why am I so confident? Well, if they didn’t have the potential to make a difference, ARM, Qualcomm, AMD, and other companies wouldn’t be wasting their time and burning money on their development.

So, what about ARM server downsides? There are quite a few, and some of them are big. Luckily, the industry is working hard to address them.

• Hit and miss software support
• Availability, potential deployment issues
• ROI concerns
• Tiny ecosystem
• Old habits die hard

Software-related issues will, probably, be the biggest immediate concern. While a lot of popular services will run on ARM servers, software support will be a problem. It’s not enough to merely port stuff to new hardware; we have to make sure everything functions properly so there are no performance hits or failures. In other words, the ported software has to be mature. Nobody will develop and deploy a service built on buggy foundations.

With all the money to be made in the server market, one would expect to see fast progress, but that isn’t always the case. Adopting new hardware and tweaking all the software that runs on it is never easy, and the pace depends on market adoption. The size of the ARM server ecosystem is (very) limited, and I doubt a couple of new processors would make much of a difference in the short run. While influential companies like ARM and Qualcomm have a vested interest in seeing a pickup in demand for ARM servers, there is not much they can do about software. They have next to no influence over software developers, so they can’t force them to add ARM support to existing products.

Long story short: Take a good look at your stack and try to figure out whether everything will run properly on ARM hardware. Given enough time, developers will start adding support for ARM hardware, but this won’t be a fast process. They will have to tweak frameworks and applications to take into account a new architecture, and I suspect, many of them won’t bother until there are enough ARM servers out there (which may take years). Support for legacy software is another obvious problem.

This brings us to the next point: Market availability and potential deployment issues. There aren’t that many ARM servers out there, so choice is limited, and so is availability. A year or two down the road, we could see a number of ARM-based hosting packages on offer, but we won’t see too many. Worse, there’s a good chance these servers will be concentrated in certain parts of the world, making them less attractive to some developers. There are a lot of unknowns related to deployment, so it’s still too early to say how things will pan out.

Slow adoption could create another set of challenges. These aren’t restricted to ARM servers; they apply to most enterprise technology. A lot of organisations are bound to explore the possibility of using ARM servers, but that doesn’t necessarily mean they will actually use them. In order to ensure enough development and consumer demand, market adoption needs to grow steadily. Otherwise, risk-averse people will probably stay away, taking the wait-and-see approach. The other potential problem is economic: If developers aren’t sure the ecosystem is growing fast enough, they might conclude the potential return is simply not worth the effort.

What about these old habits? Well, since server space doesn’t evolve fast, people tend to stick to proven platforms, namely x86 hardware. The motto is simple: If it ain’t broke, don’t fix it. Industry veterans might see ARM servers as an opportunity and take a gamble on them. It would take a fair amount of courage and confidence to tie part of a complex project to what many people still perceive as an untested or immature hardware platform. I fear many people won’t be willing to take the plunge, at least not this early on.

Bright Future And A Pinch Of Hype

I’ve spent the better part of my adult life covering cutting edge silicon, and my personal take on ARM servers is that they have a lot of potential, but, they’re not for everyone. They could play a vital role in the Internet of tomorrow by providing cheap building blocks for infrastructure and handling niche server workloads.

However, at the same time, I cannot escape the feeling that ARM servers tend to be overhyped. Despite this, I don’t see them as a fad. I think they are here to stay, but vendors must carve out a few specific niches that can truly benefit from the new architecture.

In other words, we won’t see a lot of simple LAMP web hosting servers based on ARM, but we could see loads of them in more exotic niches (and some horribly boring ones). ARM processors could be a perfect fit for specific loads, especially those that can take advantage of a large number of small physical CPU cores, stuff that’s not CPU-bound. It might not sound like much, but this actually covers a lot of potential uses: data logging, large volumes of simple queries, certain types of databases, various storage services and so on.

I could go on, listing various use-cases, pros and cons of ARM servers, and potential problems, but at the end of the day, I suspect ARM server adoption will depend on good old cash. Technology aside, ARM servers will only make sense if the economic component checks out. In other words, they will have to offer a lot more bang for buck than x86 processors if they are to justify their existence.

Since this is more or less the whole point of introducing this new architecture to the server industry, I expect attractive pricing, but it will be a few months before we know for sure.

Database tuning can be an incredibly difficult task, particularly when working with large-scale data where even the most minor change can have a dramatic (positive or negative) impact on performance.

In mid-sized and large companies, most SQL database tuning will be handled by a Database Administrator (DBA). But believe me, there are plenty of developers out there who have to perform DBA-like tasks. Further, in many of the companies I’ve seen that do have DBAs, they often struggle to work well with developers—the positions simply require different modes of problem solving, which can lead to disagreement among coworkers.

On top of that, corporate structure can also play a role. Say the DBA team is placed on the 10th floor with all of their databases, while the devs are on the 15th floor, or even in a different building under a completely separate reporting structure—it’s certainly hard to work together smoothly under these conditions.  In this article, I’d like to accomplish two things:

1. Provide developers with some developer-side database tuning techniques.
2. Explain how developers and DBAs can work together effectively.

Database Optimization (in the Codebase)

Indexes

If you’re a complete newcomer to databases and even asking yourself “What is SQL tuning?”, you should know that indexing is an effective way to tune your SQL database that is often neglected during development. In basic terms, an index is a data structure that improves the speed of data retrieval operations on a database table by providing rapid random lookups and efficient access of ordered records. This means that once you’ve created an index, you can select or sort your rows faster than before.

Indexes are also used to define a primary-key or unique index which will guarantee that no other columns have the same values. Of course, indexing is a vast an interesting topic to which I can’t do justice with this brief description (here’s a more detailed write-up).

If you’re new to indexes, I recommend using this diagram when structuring your queries: 

Basically, the goal is to index the major searching and ordering columns.

Note that if your tables are constantly hammered by INSERT, UPDATE, and DELETE, you should be careful when indexing—you could end up decreasing performance as all indexes need to be modified after these operations.

Further, DBAs often drop their indexes before performing million+ row batch inserts to speed up the insertion process. After the batch is inserted, they then recreate the indexes. Remember, however, that dropping indexes will affect every query running in that table; so this approach is only recommended when working with a single, large insertion.

SQL Server Performance Tuning: Execution Plans

By the way: the Execution Plan tool in SQL Server can be useful for creating indexes.

Its main function is to graphically display the data retrieval methods chosen by the SQL Server query optimizer. If you’ve never seen them before, here’s a detailed walkthrough.

To retrieve the execution plan (in SQL Server Management Studio), just click “Include Actual Execution Plan” (CTRL + M) before running your query.

Afterwards, a third tab named “Execution Plan” will appear. You might see a detected missing index. To create it, just right click in the execution plan and choose the “Missing Index Details…”. It’s as simple as that!

Tune SQL query performance by avoiding coding loops

Imagine a scenario in which 1000 queries hammer your database in sequence. Something like:

for (int i = 0; i < 1000; i++)
{
    SqlCommand cmd = new SqlCommand("INSERT INTO TBL (A,B,C) VALUES...");
    cmd.ExecuteNonQuery();
}

You should avoid such loops in your code. For example, we could transform the above snippet by using a unique INSERT or UPDATE statement with multiple rows and values:

INSERT INTO TableName (A,B,C) VALUES (1,2,3),(4,5,6),(7,8,9) -- SQL SERVER 2008

INSERT INTO TableName (A,B,C) SELECT 1,2,3 UNION ALL SELECT 4,5,6 -- SQL SERVER 2005

UPDATE TableName SET A = CASE B
        WHEN 1 THEN 'NEW VALUE'
        WHEN 2 THEN 'NEW VALUE 2'
        WHEN 3 THEN 'NEW VALUE 3'
    END
WHERE B in (1,2,3)

Make sure that your WHERE clause avoids updating the stored value if it matches the existing value. Such a trivial optimization can dramatically increase SQL query performance by updating only hundreds of rows instead thousands. For example:

UPDATE TableName
SET A = @VALUE
WHERE
      B = 'YOUR CONDITION'
            AND A <> @VALUE -- VALIDATION

Avoid correlated subqueries

A correlated subquery is one which uses values from the parent query. This kind of query tends to run row-by-row, once for each row returned by the outer query, and thus decreases SQL query performance. New developers are often caught structuring their queries in this way—because it’s usually the easy route.

Here’s an example of a correlated subquery:

SELECT c.Name, 
       c.City,
       (SELECT CompanyName FROM Company WHERE ID = c.CompanyID) AS CompanyName 
FROM Customer c

In particular, the problem is that the inner query (SELECT CompanyName…) is run for each row returned by the outer query (SELECT c.Name…). But why go over the Company again and again for every row processed by the outer query?

A more efficient performance tuning technique would be to refactor the correlated subquery as a join:

SELECT c.Name, 
       c.City, 
       co.CompanyName 
FROM Customer c 
	LEFT JOIN Company co
		ON c.CompanyID = co.CompanyID

In this case, we go over the Company table just once, at the start, and JOIN it with the Customer table. From then on, we can select the values we need (co.CompanyName) more efficiently.

Select sparingly

One of my favorite SQL optimization tips is to avoid SELECT *! Instead, you should individually include the specific columns that you need. Again, this sounds simple, but I see this error all over the place. Consider a table with hundreds of columns and millions of rows—if your application only really needs a few columns, there’s no sense in querying for all the data. It’s a massive waste of resources. (For more issues, see here.)

For example:

SELECT * FROM Employees

vs.

SELECT FirstName, City, Country FROM Employees

If you really need every column, explicitly list every column. This isn’t so much a rule, but rather, a means of preventing future system errors and additional database performance tuning. For example, if you’re using an INSERT... SELECT... and the source table has changed via the addition of a new column, you might run into issues, even if that column isn’t needed by the destination table, e.g.:

INSERT INTO Employees SELECT * FROM OldEmployees

Msg 213, Level 16, State 1, Line 1
Insert Error: Column name or number of supplied values does not match table definition.

To avoid this kind of error, you should declare each column individually:

INSERT INTO Employees (FirstName, City, Country)
SELECT Name, CityName, CountryName
FROM OldEmployees

Note, however, that there are some situations where the use of SELECT * could be appropriate. For example, with temp tables—which leads us to our next topic.

Temporary tables usually increase a query’s complexity. If your code can be written in a simple, straightforward manner, I’d suggest avoiding temp tables.

But if you have a stored procedure with some data manipulation that cannot be handled with a single query, you can use temp tables as intermediaries to help you to generate a final result.

When you have to join a large table and there are conditions on said table, you can increase database performance by transferring your data in a temp table, and then making a join on that. Your temp table will have fewer rows than the original (large) table, so the join will finish faster!

The decision isn’t always straightforward, but this example will give you a sense for situations in which you might want to use temp tables:

Imagine a customer table with millions of records. You have to make a join on a specific region. You can achieve this by using a SELECT INTO statement and then joining with the temp table:

SELECT * INTO #Temp FROM Customer WHERE RegionID = 5
SELECT r.RegionName, t.Name FROM Region r JOIN #Temp t ON t.RegionID = r.RegionID

(Note: some SQL developers also avoid using SELECT INTO to create temp tables, saying that this command locks the tempdb database, disallowing other users from creating temp tables. Fortunately, this is fixed in 7.0 and later.)

As an alternative to temp tables, you might consider using a subquery as a table:

SELECT r.RegionName, t.Name FROM Region r 
JOIN (SELECT * FROM Customer WHERE RegionID = 5) AS t 
ON t.RegionID = r.RegionID

But wait! There’s a problem with this second query. As described above, we should only be including the columns we need in our subquery (i.e., not using SELECT *). Taking that into account:

SELECT r.RegionName, t.Name FROM Region r 
JOIN (SELECT Name, RegionID FROM Customer WHERE RegionID = 5) AS t 
ON t.RegionID = r.RegionID

All of these snippets will return the same data. But with temp tables, we could, for example, create an index in the temp table to improve performance. There’s some good discussion here on the differences between temporary tables and subqueries.

Finally, when you’re done with your temp table, delete it to clear tempdb resources, rather than just wait for it to be automatically deleted (as it will be when your connection to the database is terminated):

DROP TABLE #temp

Does my record exist?

This SQL opimization technique concerns the use of EXISTS(). If you want to check if a record exists, use EXISTS() instead of COUNT(). While COUNT() scans the entire table, counting up all entries matching your condition, EXISTS() will exit as soon as it sees the result it needs. This will give you better performance and clearer code.

IF (SELECT COUNT(1) FROM EMPLOYEES WHERE FIRSTNAME LIKE '%JOHN%') > 0
 	PRINT 'YES' 

vs.

IF EXISTS(SELECT FIRSTNAME FROM EMPLOYEES WHERE FIRSTNAME LIKE '%JOHN%')
    PRINT 'YES'

Database Optimization (in the Office)

SQL database administrators (DBAs) and developers often clash over data- and non-data-related issues. Drawn from my experience, here are some tips (for both parties) on how to get along and work together effectively.

For developers:

1. If your application stops working suddenly, it may not be a database issue. For example, maybe you have a network problem. Investigate a bit before you accuse a DBA!
2. Even if you’re a ninja SQL data modeler, ask a DBA to help you with your relational diagram. They have a lot to share and offer.
3. DBAs don’t like rapid changes. This is natural: they need to analyze the database as a whole and examine the impact of any changes from all angles. A simple change in a column can take a week to be implemented—but that’s because an error could materialize as huge losses for the company. Be patient!
4. Do not ask SQL DBAs to make data changes in a production environment. If you want access to the production database, you have to be responsible for all your own changes.

For SQL Database Administrators:

1. If you don’t like people asking you about the database, give them a real-time status panel. Developers are always suspicious of a database’s status, and such a panel could save everyone time and energy.
2. Help developers in a test/quality assurance environment. Make it easy to simulate a production server with simple tests on real-data. This will be a significant time-saver for others as well as yourself.
3. Developers spend all day on systems with frequently-changed business logic. Try to understand this world being more flexible, and be able to break some rules in a critical moment.
4. SQL databases evolve. The day will come when you have to migrate your data to a new version. Developers count on significant new functionality with each new version. Instead of refusing to accept their changes, plan ahead and be ready for the migration.

Android devices have a lot of cores, so writing smooth apps is a simple task for anyone, right? Wrong. As everything on Android can be done in a lot of different ways, picking the best option can be tough. If you want to choose the most efficient method, you have to know what’s happening under the hood. Luckily, you don’t have to rely on your feelings or sense of smell, since there’s a lot of tools out there that can help you find bottlenecks by measuring and describing what’s going on. Properly optimized and smooth apps greatly improve the user experience, and also drain less battery.

Let’s see some numbers first to consider how important optimization really is. According to a Nimbledroid post, 86% of users (including me) have uninstalled apps after using them only once due to poor performance. If you’re loading some content, you have less than 11 seconds to show it to the user. Only every third user will give you more time. You might also get a lot of bad reviews on Google Play because of it.

The first thing every user notices over and over is the app’s startup time. According to another Nimbledroid post, out of the 100 top apps, 40 start in under 2 seconds, and 70 start in under 3 seconds. So if possible, you should generally display some content as soon as possible and delay the background checks and updates a bit.

Always remember, premature optimization is the root of all evil. You should also not waste too much time with micro optimization. You will see the most benefit of optimizing code that runs often. For example, this includes the onDraw() function, which runs every frame, ideally 60 times per second. Drawing is the slowest operation out there, so try redrawing only what you have to. More about this will come later.

Performance Tips

Enough theory, here is a list of some of the things you should consider if performance matters to you.

1. String vs StringBuilder

Let’s say that you have a String, and for some reason you want to append more Strings to it 10 thousand times. The code could look something like this.

[ccie_c]

String string = “hello”;
for (int i = 0; i < 10000; i++) {
string += ” world”;
}

[/ccie_c]

You can see on the Android Studio Monitors how inefficient some String concatenation can be. There’s tons of Garbage Collections (GC) going on.

This operation takes around 8 seconds on my fairly good device, which has Android 5.1.1. The more efficient way of achieving the same goal is using a StringBuilder, like this.

[ccie_c]

StringBuilder sb = new StringBuilder(“hello”);

for (int i = 0; i < 10000; i++){ sb.append(” world”);}

String string = sb.toString();

[/ccie_c]

On the same device this happens almost instantly, in less than 5ms. The CPU and Memory visualizations are almost totally flat, so you can imagine how big this improvement is. Notice though, that for achieving this difference, we had to append 10 thousand Strings, which you probably don’t do often. So in case you are adding just a couple Strings once, you will not see any improvement. By the way, if you do:

[ccie_c]

String string = “hello” + ” world”;

[/ccie_c]

It gets internally converted to a StringBuilder, so it will work just fine.

You might be wondering, why is concatenating Strings the first way so slow? It is caused by the fact that Strings are immutable, so once they are created, they cannot be changed. Even if you think you are changing the value of a String, you are actually creating a new String with the new value. In an example like:

[ccie_c]

String myString = “hello”;

myString += ” world”;

[/ccie_c]

What you will get in memory is not 1 String “hello world”, but actually 2 Strings. The String myString will contain “hello world”, as you would expect. However, the original String with the value “hello” is still alive, without any reference to it, waiting to be garbage collected. This is also the reason why you should store passwords in a char array instead of a String. If you store a password as a String, it will stay in the memory in human-readable format until the next GC for an unpredictable length of time. Back to the immutability described above, the String will stay in the memory even if you assign it another value after using it. If you, however, empty the char array after using the password, it will disappear from everywhere.

2. Picking the Correct Data Type

Before you start writing code, you should decide what data types you will use for your collection. For example, should you use a Vector or an ArrayList? Well, it depends on your usecase. If you need a thread-safe collection, which will allow only one thread at once to work with it, you should pick a Vector, as it is synchronized. In other cases you should probably stick to an ArrayList, unless you really have a specific reason to use vectors.

How about the case when you want a collection with unique objects? Well, you should probably pick a Set. They cannot contain duplicates by design, so you will not have to take care of it yourself. There are multiple types of sets, so pick one that fits your use case. For a simple group of unique items, you can use a HashSet. If you want to preserve the order of items in which they were inserted, pick a LinkedHashSet. A TreeSet sorts items automatically, so you will not have to call any sorting methods on it. It should also sort the items efficiently, without you having to think of sorting algorithms.

3. Location Updates

There are a lot of apps out there which collect the user’s location. You should use the Google Location Services API for that purpose, which contains a lot of useful functions. There is a separate article about using it, so I will not repeat it.

I’d just like to stress some important points from a performance perspective.

First of all, use only the most precise location as you need. For example, if you are doing some weather forecasting, you don’t need the most accurate location. Getting just a very rough area based on the network is faster, and more battery efficient. You can achieve it by setting the priority to LocationRequest.PRIORITY_LOW_POWER.

You can also use a function of LocationRequest called setSmallestDisplacement(). Setting this in meters will cause your app to not be notified about location change if it was smaller than the given value. For example, if you have a map with nearby restaurants around you, and you set the smallest displacement to 20 meters, the app will not be making requests for checking restaurants if the user is just walking around in a room. The requests would be useless, as there wouldn’t be any new nearby restaurant anyway.

The second rule is requesting location updates only as often as you need them. This is quite self explanatory. If you are really building that weather forecast app, you do not need to request the location every few seconds, as you probably don’t have such precise forecasts (contact me if you do). You can use the setInterval() function for setting the required interval in which the device will be updating your app about the location. If multiple apps keep requesting the user’s location, every app will be notified at every new location update, even if you have a higher setInterval() set. To prevent your app from being notified too often, make sure to always set a fastest update interval with setFastestInterval().

And finally, the third rule is requesting location updates only if you need them. If you are displaying some nearby objects on the map every x seconds and the app goes in background, you do not need to know the new location. There is no reason to update the map if the user cannot see it anyway. Make sure to stop listening for location updates when appropriate, preferably in onPause(). You can then resume the updates in onResume().

4. Network Requests

There is a high chance that your app is using the internet for downloading or uploading data. If it is, you have several reasons to pay attention to handling network requests. One of them is mobile data, which is very limited to a lot of people and you shouldn’t waste it.

The second one is battery. Both WiFi and mobile networks can consume quite a lot of it if they are used too much. Let’s say that you want to download 1 kb. To make a network request, you have to wake up the cellular or WiFi radio, then you can download your data. However, the radio will not fall asleep immediately after the operation. It will stay in a fairly active state for about 20-40 more seconds, depending on your device and carrier.

So, what can you do about it? Batch. To avoid waking up the radio every couple seconds, prefetch things that the user might need in the upcoming minutes. The proper way of batching is highly dynamic depending on your app, but if it is possible, you should download the data the user might need in the next 3-4 minutes. One could also edit the batch parameters based on the user’s internet type, or charging state. For example, if the user is on WiFi while charging, you can prefetch a lot more data than if the user is on mobile internet with low battery. Taking all these variables into account can be a tough thing, which only few people would do. Luckily though, there is GCM Network Manager to the rescue!

GCM Network Manager is a really helpful class with a lot of customizable attributes. You can easily schedule both repeating and one-off tasks. At repeating tasks you can set the lowest, as well as the highest repeat interval. This will allow batching not only your requests, but also requests from other apps. The radio has to be woken up only once per some period, and while it’s up, all apps in the queue download and upload what they are supposed to. This Manager is also aware of the device’s network type and charging state, so you can adjust accordingly. You can find more details and samples in this article, I urge you to check it out. An example task looks like this:

[ccie_c]

Task task = new OneoffTask.Builder()

.setService(CustomService.class)

.setExecutionWindow(0, 30)

.setTag(LogService.TAG_TASK_ONEOFF_LOG)

.setUpdateCurrent(false)

.setRequiredNetwork(Task.NETWORK_STATE_CONNECTED)

.setRequiresCharging(false)

.build();

[/ccie_c]

By the way, since Android 3.0, if you do a network request on the main thread, you will get a NetworkOnMainThreadException. That will definitely warn you not to do that again.

 5. Reflection

Reflection is the ability of classes and objects to examine their own constructors, fields, methods, and so on. It is used usually for backward compatibility, to check if a given method is available for a particular OS version. If you have to use reflection for that purpose, make sure to cache the response, as using reflection is pretty slow. Some widely used libraries use Reflection too, like Roboguice for dependency injection. That’s the reason why you should prefer Dagger 2. For more details about reflection, you can check a separate post.

6. Autoboxing

Autoboxing and unboxing are processes of converting a primitive type to an Object type, or vice versa. In practice it means converting an int to an Integer. For achieving that, the compiler uses the Integer.valueOf()function internally. Converting is not just slow, Objects also take a lot more memory than their primitive equivalents. Let’s look at some code.

[ccie_c]

Integer total = 0;
for (int i = 0; i < 1000000; i++) {
total += i;
}

[/ccie_c]

While this takes 500ms on average, rewriting it to avoid autoboxing will speed it up drastically.

[ccie_c]

int total = 0; for (int i = 0; i < 1000000; i++) { total += i; }

[/ccie_c]

This solution runs at around 2ms, which is 25 times faster. If you don’t believe me, test it out. The numbers will be obviously different per device, but it should still be a lot faster. And it’s also a really simple step to optimize.

Okay, you probably don’t create a variable of type Integer like this often. But what about the cases when it is more difficult to avoid? Like in a map, where you have to use Objects, like Map<Integer, Integer>? Look at the solution many people use.

[ccie_c]
Map<Integer, Integer> myMap = new HashMap<>(); for (int i = 0; i < 100000; i++) { myMap.put(i, random.nextInt()); }
[/ccie_c]

Inserting 100k random ints in the map takes around 250ms to run. Now look at the solution with SparseIntArray.

[ccie_c]
SparseIntArray myArray = new SparseIntArray(); for (int i = 0; i < 100000; i++) { myArray.put(i, random.nextInt()); }
[/ccie_c]

This takes a lot less, roughly 50ms. It’s also one of the easier methods for improving performance, as nothing complicated has to be done, and the code also stays readable. While running a clear app with the first solution took 13MB of my memory, using primitive ints took something under 7MB, so just the half of it.

SparseIntArray is just one of the cool collections that can help you avoid autoboxing. A map like Map<Integer, Long> could be replaced by SparseLongArray, as the value of the map is of type Long. If you look at the source code of SparseLongArray, you will see something pretty interesting. Under the hood, it is basically just a pair of arrays. You can also use a SparseBooleanArray similarly.

If you read the source code, you might have noticed a note saying that  SparseIntArray can be slower than HashMap . I’ve been experimenting a lot, but for me SparseIntArray was always better both memory and performance wise. I guess it’s still up to you which you choose, experiment with your use cases and see which fits you the most. Definitely have the SparseArrays in your head when using maps.

7. OnDraw

As I’ve said above, when you are optimizing performance, you will probably see the most benefit in optimizing code which runs often. One of the functions running a lot is onDraw(). It may not surprise you that it’s responsible for drawing views on the screen. As the devices usually run at 60 fps, the function is run 60 times per second. Every frame has 16 ms to be fully handled, including its preparation and drawing, so you should really avoid slow functions. Only the main thread can draw on the screen, so you should avoid doing expensive operations on it. If you freeze the main thread for several seconds, you might get the infamous Application Not Responding (ANR) dialog. For resizing images, database work, etc., use a background thread.

One of the things you should avoid in onDraw() is allocating objects like Paint. Prepare everything in the constructor, so it’s ready when drawing. Even if you have onDraw() optimized, you should call it only as often as you have to. What is better than calling an optimized function? Well, not calling any function at all. In case you want to draw text, there is a pretty neat helper function called drawText(), where you can specify things like the text, coordinates, and the text color.

8. ViewHolders

You probably know this one, but I cannot skip it. The Viewholder design pattern is a way of making scrolling lists smoother. It is a kind of view caching, which can seriously reduce the calls to findViewById() and inflating views by storing them. It can look something like this.

[ccie_c]
static class ViewHolder { TextView title; TextView text; public ViewHolder(View view) { title = (TextView) view.findViewById(R.id.title); text = (TextView) view.findViewById(R.id.text); } }
[/ccie_c]

Then, inside the getView() function of your adapter, you can check if you have a useable view. If not, you create one.

[ccie_c]
ViewHolder viewHolder; if (convertView == null) { convertView = inflater.inflate(R.layout.list_item, viewGroup, false); viewHolder = new ViewHolder(convertView); convertView.setTag(viewHolder); } else { viewHolder = (ViewHolder) convertView.getTag(); } viewHolder.title.setText(“Hello World”);
[/ccie_c]

You can find a lot of useable info about this pattern around the internet. It can also be used in cases when your list view has multiple different types of elements in it, like some section headers.

9. Resizing Images

Chances are, your app will contain some images. In case you are downloading some JPGs from the web, they can have really huge resolutions. However, the devices they will be displayed on will be a lot smaller. Even if you take a photo with the camera of your device, it needs to be downsized before displaying as the photo resolution is a lot bigger than the resolution of the display. Resizing images before displaying them is a crucial thing. If you’d try displaying them in full resolutions, you’d run out of memory pretty quickly. There is a whole lot written about displaying bitmaps efficiently in the Android docs, I will try summing it up.

So you have a bitmap, but you don’t know anything about it. There’s a useful flag of Bitmaps called inJustDecodeBounds at your service, which allows you to find out the bitmap’s resolution. Let’s assume that your bitmap is 1024×768, and the ImageView used for displaying it is just 400×300. You should keep dividing the bitmap’s resolution by 2 until it’s still bigger than the given ImageView. If you do, it will downsample the bitmap by a factor of 2, giving you a bitmap of 512×384. The downsampled bitmap uses 4x less memory, which will help you a lot with avoiding the famous OutOfMemory error.

Now that you know how to do it, you should not do it. … At least, not if your app relies on images heavily, and it’s not just 1-2 images. Definitely avoid stuff like resizing and recycling images manually, use some third party libraries for that. The most popular ones are Picasso by Square, Universal Image Loader, Fresco by Facebook, or my favourite, Glide. There is a huge active community of developers around it, so you can find a lot of helpful people at the issues section on GitHub as well.

10. Strict Mode

Strict Mode is a quite useful developer tool that many people don’t know about. It’s usually used for detecting network requests or disk accesses from the main thread. You can set what issues Strict Mode should look for and what penalty it should trigger. A google sample looks like this:

[ccie_c]
public void onCreate() { if (DEVELOPER_MODE) { StrictMode.setThreadPolicy(new StrictMode.ThreadPolicy.Builder() .detectDiskReads() .detectDiskWrites() .detectNetwork() .penaltyLog() .build()); StrictMode.setVmPolicy(new StrictMode.VmPolicy.Builder() .detectLeakedSqlLiteObjects() .detectLeakedClosableObjects() .penaltyLog() .penaltyDeath() .build()); } super.onCreate(); }

[/ccie_c]

If you want to detect every issue Strict Mode can find, you can also use detectAll(). As with many performance tips, you should not blindly try fixing everything Strict Mode reports. Just investigate it, and if you are sure it’s not an issue, leave it alone. Also make sure to use Strict Mode only for debugging, and always have it disabled on production builds.

Debugging Performance: The Pro Way

Let’s now see some tools that can help you find bottlenecks, or at least show that something is wrong.

1. Android Monitor

This is a tool built into Android Studio. By default, you can find the Android Monitor at the bottom left corner, and you can switch between 2 tabs there. Logcat and Monitors. The Monitors section contains 4 different graphs. Network, CPU, GPU, and Memory. They are pretty self explanatory, so I will just quickly go through them. Here is a screenshot of the graphs taken while parsing some JSON as it is downloaded.

The Network part shows the incoming and outgoing traffic in KB/s. The CPU part displays the CPU usage in percent. The GPU monitor displays how much time it takes to render the frames of a UI window. This is the most detailed monitor out of these 4, so if you want more details about it, read this.

Lastly we have the Memory monitor, which you will probably use the most. By default it shows the current amount of Free and Allocated memory. You can force a Garbage Collection with it too, to test if the amount of used memory drops down. It has a useful feature called Dump Java Heap, which will create a HPROF file which can be opened with the HPROF Viewer and Analyzer. That will enable you to see how many objects you have allocated, how much memory is taken by what, and maybe which objects are causing memory leaks. Learning how to use this analyzer is not the simplest task out there, but it is worth it. The next thing you can do with the Memory Monitor is do some timed Allocation Tracking, which you can start and stop as you wish. It could be useful at many cases, for example when scrolling or rotating the device.

2. GPU Overdraw

This is a simple helper tool, which you can activate in Developer Options once you have enabled developer mode. Select Debug GPU overdraw, “Show overdraw areas”, and your screen will get some weird colors. It’s ok, that’s what is supposed to happen. The colors mean how many times a particular area was overdrawn. True color means that there was no overdraw, this is what you should aim for. Blue means one overdraw, green means two, pink three, red four.

While seeing true color is the best, you will always see some overdraws, especially around texts, navigation drawers, dialogs and more. So don’t try getting rid of it fully. If your app is blueish or greenish, that’s probably fine. However, if you see too much red on some simple screens, you should investigate what’s going on. It might be too many fragments stacked onto each other, if you keep adding them instead of replacing. As I’ve mentioned above, drawing is the slowest part of apps, so there is no sense drawing something if there will be more than 3 layers drawn onto it. Feel free to check out your favourite apps with it. You will see that even apps with over a billion downloads have red areas, so just take it easy when you are trying to optimize.

3. GPU Rendering

This is another tool from the Developer options, called Profile GPU rendering. Upon selecting it, pick “On screen as bars”. You will notice some colored bars appearing on your screen. Since every application has separate bars, weirdly the status bar has its own ones, and in case you have software navigation buttons, they have their own bars too. Anyway, the bars get updated as you interact with the screen.

The bars consist of 3-4 colors, and according to the Android docs, their size indeed matters. The smaller, the better. At the bottom you have blue, which represents the time used to create and update the View’s display lists. If this part is too tall, it means that there is a lot of custom view drawing, or a lot of work done in the onDraw() functions. If you have Android 4.0+, you will see a purple bar above the blue one. This represents the time spent transferring resources to the render thread. Then comes the red part, which represents the time spent by Android’s 2D renderer issuing commands to OpenGL to draw and redraw display lists. At the top is the orange bar, which represents the time the CPU is waiting for the GPU to finish its work. If it’s too tall, the app is doing too much work on the GPU.

If you are good enough, there is one more color above the orange. It is a green line representing the 16 ms threshold. As your goal should be running your app at 60 fps, you have 16 ms to draw every frame. If you don’t make it, some frames might be skipped, the app could become jerky, and the user would definitely notice. Pay special attention to animations and scrolling, that’s where the smoothness matters the most. Even though you can detect some skipped frames with this tool, it won’t really help you figuring out where exactly the problem is.

4. Hierarchy Viewer

This is one of my favourite tools out there, as it’s really powerful. You can start it from Android Studio through Tools -> Android -> Android Device Monitor, or it is also in your sdk/tools folder as “monitor”. You can also find a standalone hierarachyviewer executable there, but as it’s deprecated you should open the monitor. However you open the Android Device Monitor, switch to the Hierarchy Viewer perspective. If you don’t see any running apps assigned to your device, there are a couple things you can do to fix it. Also try checking out this issue thread, there are people with all kinds of issues and all kinds of solutions. Something should work for you too.

With Hierarchy Viewer, you can get a really neat overview of your view hierarchies (obviously). If you see every layout in a separate XML, you might easily spot useless views. However, if you keep combining the layouts, it can easily get confusing. A tool like this makes it simple to spot, for example, some RelativeLayout, which has just 1 child, another RelativeLayout. That makes one of them removable.

Avoid calling requestLayout(), as it causes traversing of the entire view hierarchy, to find out how big each view should be. If there is some conflict with the measurements, the hierarchy might be traversed multiple times, which if happens during some animation, it will definitely make some frames be skipped. If you want to find out more about how Android draws its views, you can read this. Let’s look at one view as seen in Hierarchy Viewer.

The top right corner contains a button for maximizing the preview of the particular view in a standalone window. Under it you can also see the actual preview of the view in the app. The next item is a number, which represents how many children the given view has, including the view itself. If you select a node (preferably the root one) and press “Obtain layout times” (3 colored circles), you will have 3 more values filled, together with colored circles appearing labelled measure, layout, and draw. It might not be shocking that the measure phase represents the time it took to measure the given view. The layout phase is about the rendering time, while the drawing is the actual drawing operation. These values and colors are relative to each other. Green one means that the view renders in the top 50% of all views in the tree. Yellow means rendering in the slower 50% of all views in the tree, red means that the given view is one of the slowest. As these values are relative, there will always be red ones. You simply cannot avoid them.

Under the values you have the class name, such as “TextView”, an internal view ID of the object, and the android:id of the view, which you set in the XML files. I urge you to build a habit of adding IDs to all views, even if you don’t reference them in the code. It will make identifying the views in Hierarchy Viewer really simple, and in case you have automated tests in your project, it will also make targeting the elements a lot faster. That will save some time for you and your colleagues writing them. Adding IDs to elements added in XML files is pretty straightforward. But what about the dynamically added elements? Well, it turns out to be really simple too. Just create an ids.xml file inside your values folder and type in the required fields. It can look like this:

[ccie_c]

<resources>

<item name=”item_title” type=”id”/>

<item name=”item_body” type=”id”/>

</resources>

[/ccie_c]

Then in the code, you can use setId(R.id.item_title). It couldn’t be simpler.

There are a couple more things to pay attention to when optimizing UI. You should generally avoid deep hierarchies while preferring shallow, maybe wide ones. Do not use layouts you don’t need. For example, you can probably replace a group of nested LinearLayouts with either a RelativeLayout, or a TableLayout. Feel free to experiment with different layouts, don’t just always use LinearLayout and RelativeLayout. Also try creating some custom views when needed, it can improve the performance significantly if done well. For instance, did you know that Instagram doesn’t use TextViews for displaying comments?

You can find some more info about Hierarchy Viewer on the Android Developers site with descriptions of different panes, using the Pixel Perfect tool, etc. One more thing I would point out is capturing the views in a .psd file, which can be done by the “Capture the window layers” button. Every view will be in a separate layer, so it’s really simple to hide or change it in Photoshop or GIMP. Oh, that’s another reason to add an ID to every view you can. It will make the layers have names that actually make sense.

You will find a lot more debugging tools in Developer options, so I advise you to activate them and see what are they doing. What could possibly go wrong?

The Android developers site contains a set of best practices for performance. They cover a lot of different areas, including memory management, which I haven’t really talked about. I silently ignored it, because handling memory and tracking memory leaks is a whole separate story. Using a third party library for efficiently displaying images will help a lot, but if you still have memory issues, check out Leak canary made by Square, or read this.

Wrapping Up

So, this was the good news. The bad new is, optimizing Android apps is a lot more complicated. There are a lot of ways of doing everything, so you should be familiar with the pros and cons of them. There usually isn’t any silver bullet solution which has only benefits. Only by understanding what’s happening behind the scenes will you be able to pick the solution which is best for you. Just because your favorite developer says that something is good, it doesn’t necessarily mean that it’s the best solution for you. There are a lot more areas to discuss and more profiling tools which are more advanced, so we might get to them next time.

Make sure you learn from the top developers and top companies. You can find a couple hundred engineering blogs at this link. It’s obviously not just Android related stuff, so if you are interested only in Android, you have to filter the particular blog. I would highly recommend the blogs of Facebook and Instagram. Even though the Instagram UI on Android is questionable, their engineering blog has some really cool articles. For me it’s awesome that it’s so easy to see how things are done in companies which are handling hundreds of millions of users daily, so not reading their blogs seems crazy. The world is changing really fast, so if you aren’t constantly trying to improve, learn from others and use new tools, you will be left behind. As Mark Twain said, a person who doesn’t read has no advantage over one who can’t read.

data-recovery-software.review is a website that systematically evaluates the various recovery software commercially available and which one will best fit a user’s needs based on their requirements.

All the evaluated data recovery software on that website uses general common parameters for ratings like :

• Price
• Ease of use
• Refund
• Lifetime updates
• Recovery speed
• User Manual
• Type of file system supported

The test results for these evaluated data recovery software are listed in an easy to understand tabular format with the steps involved in testing them (testbed).

As per the spectrum of data recovery tasks, the website broadly categorizes data recovery tasks into 4 groups :

• General data recovery : This the most common of all data recovery software programs. Simply put, this involves restoring data back to a formatted hard drive after reinstalling OS or recovering a deleted file/folder using any of the data recovery software aimed for home users.

List of software programs reviewed for general data recovery: EaseUS., GetDataBack, ReclaiMe, RecoverMyFiles, R-Studio, Stellar Phoenix, UFS Explorer and ZAR.

• NAS data recovery : NAS (Network Attached Storage) is more inclined for enterprise storage platforms that deals with a lot of data and uses Linux. NAS recovery programs in brief first try to access and read the Linux based file systems and them attempt to restore the data.

List of software programs reviewed for NAS recovery : HomeNAS recovery, ReclaiMe, Runtime NAS recovery, UFS Explorer and ZAR.

• RAID data recovery : RAID (Redundant Array of Independent Disks) recovery can either be a software or hardware based. Again, this is usually an enterprise level storage platform and tools for this come in two main formats : those which just restore RAID configuration and those who along with that recover file systems too. It is explained in much more detail here.

List of software programs reviewed for RAID recovery : ReclaiMe Free RAID Recovery, RAID reconstructor, R-Studio and ZAR.

• Photo data recovery : Photos and image files can be a pain to recover if not backed up properly! This also includes files that are stored on portable drives, memory cards and thumb drives ideally no more than 32GB in size.  Usually as these files are stored on relatively smaller capacity external storage, the data to be recovered is not massive but it still needs to be recovered from the typical FAT file system which these portable devices use for storage.

List of software programs reviewed for photo data recovery : EaseUS, GetDataBack, ReclaiMe, RecoverMyFiles, R-Studio, Recuva, Stellar Phoenix, ZAR.

Overall, as this website groups different types of data recovery tasks and then sorts related programs in a tabular format, it becomes easy to focus on what kind and which program is best suited for purchasing.

Do give it a try!