Whilst at RSA, Mirko from Help Net Security asked me to talk on a 5 minute podcast about Cloud Security from a technical perspective (thanks Mirko!).

This was my last high level presentation on Cloud Security issues - there’s lots of chewy cloud goodness to dive into hence future presentations will be more technical in nature.

P.S If you weren’t at BruCON, you missed an excellent security conference - strong content, excellent organisation and facilities, friendly crowd.  Thanks to Benny & crew for being excellent hosts!

Here’s the abstract:

In a hurry? The short version: learn about cloud security and in the process win a tasty Belgian beer by answering easy questions!

When Amazon CEO Jeff Bezos was photographed standing in front of a vintage 1890s electric generator, it was widely assumed he was paying homage to Nick Carr’s “electric generator” metaphor of utility computing. This was understandable, but quite wrong. Reminiscent of the Bruce Lee movie where the student is chided for failing to look “out there” instead of staring at his own hand, the cloud commentators failed to notice his surroundings. Bezos — and the electric generator — were standing in the middle of a Belgium Brewery! This will be the starting point of our journey through Cloud Security using a fuller flavour metaphor: Belgian beer.

In this presentation I will cover:

• why talking about “cloud” is akin to walking into a Belgian bar and asking for “beer”
• the common cloud architectures and their implications for you - the security dude
• what the beer brewing Trappist Monks can teach us about cloud security
• attacking clouds (aka getting free beer)
• dealing with the hangover: cloud incident response & forensics

I plan to post the presentation online when I get back.  If you will be there, do say hello.

Third-party cloud computing represents the promise of outsourcing as applied to computation. Services, such as Microsoft’s Azure and Amazon’s EC2, allow users to instantiate virtual machines (VMs) on demand and thus purchase precisely the capacity they require when they require it.  In turn, the use of virtualization allows third-party cloud providers to maximize the utilization of their sunk capital costs by multiplexing many customer VMs across a shared physical infrastructure. However, in this paper, we show that this approach can also introduce new vulnerabilities.
Using the Amazon EC2 service as a case study, we show that it is possible to map the internal cloud infrastructure, identify where a particular target VM is likely to reside, and then instantiate new VMs until one is placed co-resident with the target. We explore how such placement can then be used to mount cross-VM side-channel attacks to extract information from a target VM on the same machine.

After introducing the main concepts the threat model of the research is defined:

In our threat model, adversaries are non-provider-affliated malicious parties. Victims are users running confidentiality-requiring services in the cloud. A traditional threat in such a setting is direct compromise, where an attacker attempts remote exploitation of vulnerabilities in the software running on the system. Of course, this threat exists for cloud applications as well. These kinds of attacks (while important) are a known threat and the risks they present are understood. 

We instead focus on where third-party cloud computing gives attackers novel abilities; implicitly expanding the attack surface of the victim. We assume that, like any customer, a malicious party can run and control many instances in the cloud, simply by contracting for them. Further, since the economies offered by third-party compute clouds derive from multiplexing physical infrastructure, we assume (and later validate) that an attacker’s instances might even run on the same physical hardware as potential victims. From this vantage, an attacker might manipulate shared physical resources (e.g., CPU caches, branch target buffers, network queues, etc.) to learn otherwise confidential information.  In this setting, we consider two kinds of attackers: those who cast a wide net and are interested in being able to attack some known hosted service and those focused on attacking a particular victim service. The latter’s task is more expensive and time-consuming than the former’s, but both rely on the same fundamental attack.

Section (5) “Network Probing” lays the foundation for (6) “Cloud Cartography” through describing…

…an empirical measurement study focused on understanding VM placement in the EC2 system and achieving co-resident placement for an adversary. To do this, we make use of network probing both to identify public services hosted on EC2 and to provide evidence of co-residence (that two instances share the same physical server)

They then go on to describe the tools and techniques in:

• “enumerating public EC2-based web servers using external probes” (hping2, nmap, wget)
• “translating responsive public IPs to internal IPs (via DNS queries within the cloud)” (DNS)
• “launching a number of EC2 instances of varying types and surveying the resulting IP address assigned” (I assume this was scripted)

Using data from booting 4,499 (!) EC2 instances across different EC2 accounts under their control (remember that by default, an EC2 account has a 20-instance soft-limit), they develop the following heuristics for mapping EC2 instances:

• All IPs from a /16 are from the same EC2 availability zone (e.g. US).
• A /24 inherits any included sampled instance type (e.g. small, large, x-large etc). If there are multiple instances with distinct types, then we label the /24 with each distinct type (i.e., it is ambiguous).
• A /24 containing a DomO IP address only contains Dom0 IP addresses. We associate to this /24 the type of the DomO’s associated instance (note: DomO is the first domain started by the hypervisor after boot)
• All /24’s between two consecutive Dom0 /24’s inherit the former’s associated type.

They conclude this section with “cartography countermeasures”.  This boils down to making local IP address assignment (from the address pool) random across instance types and availability zones and/or restricting the customers view.

Section (6) describes how to determine co-residence:

Given a set of targets, the EC2 map from the previous section educates choice of instance launch parameters for attempting to achieve placement on the same physical ma- chine. Recall that we refer to instances that are running on the same physical machine as being co-resident.

In this section we describe several easy-to-implement co-residence checks.

Looking ahead, our eventual check of choice will be to compare instances’ Dom0 IP addresses.

We confirm the accuracy of this (and other) co-residence checks by exploit- ing a hard-disk-based covert channel between EC2 instances.

They identify 3 network-based co-residence checks:

• matching Dom0 IP address
• small packet round-trip times
• numerically close internal IP addresses (e.g. within 7).

Now able to determine that a given EC2 instance under their control is on the same physical hardware as the target, section (7) analyses EC2’s VM placement strategy and explores techniques the attacker can use to achieve reliable co-residence with a target VM.   In the context of a brute force placement attack they note that “even a very naive attack strategy can successfully achieve co-residence against a not-so-small fraction of targets”.  Thus if your target set is wide, brute force turns out to be a viable strategy.  For individual or small sets of targets, a more efficient strategy is “instance flooding” (spinning up numerous VMs) immediately after the target has booted to “take advantage of the parallel placement locality exhibited by the EC2 placement algorithms”.  This is where the dynamic nature of cloud comes into play:

But why would we expect that an attacker can launch instances soon after a particular target victim is launched?  Here the dynamic nature of cloud computing plays well into the hands of creative adversaries. Recall that one of the main features of cloud computing is to only run servers when needed. This suggests that servers are often run on instances, terminated when not needed, and later run again.  So for example, an attacker can monitor a server’s state (e.g., via network probing), wait until the instance disappears, and then if it reappears as a new instance, engage in instance flooding. Even more interestingly, an attacker might be able to actively trigger new victim instances due to the use of auto scaling systems. These automatically grow the number of instances used by a service to meet increases in demand. (Examples include scalr [30] and RightGrid [28].  See also [6].) We believe clever adversaries can find many other practical realizations of this attack scenario.

With co-residence achieved, section (8) assesses the practicality of side channel attacks in a VM environment.  As you’d expect, a number of possibilities exist, however the holy grail of cryptographic key extraction does not appear plausible (at this time).  One notable quote:

The side channel attacks we report on in the rest of this section are more coarse-grained than those required to extract cryptographic keys. While this means the attacks extract less bits of information, it also means they are more robust and potentially simpler to implement in noisy environments such as EC2.

Side channel attacks discussed include:

• Denial of Service
• Measure cache usage (measure CPU utilisation on the physical machine; or “how busy are their servers?”)
• Load-based co-residence detection (aka detecting co-residence without relying on sending any network probes)
• Estimating traffic rates (sounds harmless but can be used to deduce targets activity patterns, peak trading times for maximal DoS effect etc)
• Keystroke timing attack (remote keystroke monitoring)

As with each of the other sections, the authors suggest potential countermeasures.

Overall the paper makes a very interested read.  There’s no EC2 “0-day”, but that’s not the intent of the paper.  Rather, we are reminded that cloud platforms and technologies do bring some novel attacks that thus far have not really figured in much of the security conversation to date.  We need more of this type of research to better understand what we are getting ourselves into.

Cloud Security Podcast Episode 1

[If you don't see the player above, turn on Javascript]

Brief show notes:

• Introductions
• Quick recap of what we mean by ‘Cloud Computing’
• Recent news & events (with a focus on FUD)
• Groups developing cloud security guidance: Cloud Security Alliance, Enisa, Jericho
• Wrap-up

Full show notes

As this was our first foray into making our own podcast, we’re seeking your feedback (we know about the audio drop outs).

Tell us what you think…

P.S Submitting to iTunes shortly

For those of us that didn’t make it to Black Hat USA/Defcon, the infosec twitter community gave us the next best thing - a running commentary of the presentations - many of which feature cutting edge security research.

I was particularly interested in following the Sensepost presentation called ‘Clobbering the Cloud’. From the write-up:

Cloud Computing dominates the headlines these days but like most paradigm changes this introduces new risks and new opportunities for us to consider. Some deep technical research has gone into the underlying technologies (like Virtualization) but to some extent this serves only to muddy the waters when considering the overall threat landscape. During this talk, SensePost will attempt to separate fact from fiction while walking through several real-world attacks on “the cloud.” The talk will focus both on attacks against the cloud and on using these platforms as attack tools for general Internet mayhem. For purposes of demonstration we will focus most of our demos and attacks against the big players…

In reverse order, check out the tweets from @GphreakX who was at BH and kindly tweeting proceedings:

Some interesting tweets there for sure!  Hopefully this has whet your appetite for the upcoming cloudsecurity.org interview with Haroon and his Sensepost team…stay tuned.

Google just made three big announcements

Google Chrome Operating System

Perhaps the biggest news is their plan to create a new operating system, based on the Linux kernel, running on X86 and ARM chipsets and targeted at the Netbook/Laptop/Desktop user:

“Google Chrome OS is an open source, lightweight operating system that will initially be targeted at netbooks. Later this year we will open-source its code, and netbooks running Google Chrome OS will be available for consumers in the second half of 2010.”

Talking of their goals:

“Speed, simplicity and security are the key aspects of Google Chrome OS. We’re designing the OS to be fast and lightweight, to start up and get you onto the web in a few seconds. The user interface is minimal to stay out of your way, and most of the user experience takes place on th web.”

And starting from a clean slate (and an obligatory swipe at Microsoft):

“And as we did for the Google Chrome browser, we are going back to the basics and completely redesigning the underlying security architecture of the OS so that users don’t have to deal with viruses, malware and security updates. It should just work.

<snip>

The software architecture is simple — Google Chrome running within a new windowing system on top of a Linux kernel. For application developers, the web is the platform. All web-based applications will automatically work and new applications can be written using your favorite web technologies. And of course, these apps will run not only on Google Chrome OS, but on any standards-based browser on Windows, Mac and Linux thereby giving developers the largest user base of any platform.

Google Chrome OS is a new project, separate from Android. Android was designed from the beginning to work across a variety of devices from phones to set-top boxes to netbooks. Google Chrome OS is being created for people who spend most of their time on the web, and is being designed to power computers ranging from small netbooks to full-size desktop systems. While there are areas where Google Chrome OS and Android overlap, we believe choice will drive innovation for the benefit of everyone, including Google.”

Wow, pretty big announcement with lots of potential market implications.  One way to look at this is they just described a system with “embedded OS” properties running as a mainstream desktop OS with services delivered via the web instead of relying on locally hosted applications.  I suppose in some ways this should come as no real surprise as it is entirely in-line with their cloud based strategy.

Whilst the target market would appear to be consumers, I can see enterprises jumping at a thin OS “that just works”.  Ultimately, this is moving us closer to  an age of disposable computing - low cost devices with low entry software footprints.

Organisations are keen to embrace smaller footprint client computers to cut costs and if the underlying hardware offers enterprise demanded features like full HD encryption (to protect that cached Cloud content), I could see enterprises taking a serious interest.

Do we *really* want to run the dozen endpoint agents we have today for configuration management, NAC, AV, HIPS (pah!) and bear all the costs they bring? With a static client, you won’t need many of these features.  From a security point of view, this could be a very good thing - no AV headaches, significantly less attack surface (enterprise apps often demonstrate “brittle” security) and less PII to lose.

To deliver on a low-update OS, they will need to ship a subset of the Linux kernel that is considered “mature”, otherwise their users will be back on the  “patch treadmill” - which is something they explicitly state they are trying to avoid.

I find it interesting they are designing a new windowing system when there are so many options available today (some with decent security too).  I suspect this is to take advantage of advances in graphical chipsets.  Perhaps they see this as a chance to boost Chrome browser page rendering speed even further.

Perhaps the more fundamental question is whether we want Google owning the last bastion - our desktops.

This brings us to the Chrome browser itself and associated technologies.

Google Native Client

Back in February, Google kicked off a security contest for a “research project” called Google Native Client (NaCl).  First a quick recap on Native Client:

“Native Client is an open-source research technology for running x86 native code in web applications, with the goal of maintaining the browser neutrality, OS portability, and safety that people expect from web apps. We’ve released this project at an early, research stage to get feedback from the security and broader open-source communities. We believe that Native Client technology will someday help web developers to create richer and more dynamic browser-based applications. ”

This is Google’s ambitious attempt to provide a high-speed, browser hosted application alternative to Java or Flash. To do this securely, they designed a new security architecture and NaCl is the implementation.

Announcing the security contest, Henry Bridge from Google wrote:

“Exploits, bugs, vulnerabilities, security holes — for most programmers these terms are synonymous with fire drills and coding all-nighters. However, for the next 10 weeks, the Native Client team is inviting you to bring them on! We’re challenging you to find security exploits in Native Client.”

The judges, led by respected academic Ed Felton (Princeton), assessed the vulnerablities reported by each of the 600 participants based on “a) Quality (Severity, Scope, Reliability and Style) and b) Quantity”.  Participants were limited to reporting on 10 bugs (Google claimed this was to avoid wasting the judges time).

Mark Dowd and Ben Hawkes won the contest, finding the bulk of the best bugs. Mark Dowd is well known in the security community - most often described as a humble genius (or a robot sent back in time :).  I followed along at home and it was great fun reading the bug descriptions as the competition progressed.  As this was a new security design, there were some unique vulnerabilities discovered along with novel exploit avenues.  Despite all the implementation snafus, Google is taking comfort that no underlying architectural weaknesses were found.

“This contest helped us discover implementation errors in Native Client and some areas of our codebase we need to spend more time reviewing. More importantly, that no major architectural flaws were found provides evidence that Native Client can be made safe enough for widespread use. Toward that end, we’re implementing additional security measures, such as an outer sandbox…”

In other posts, Google has indicated the plan to bundle NaCl with the browser, rather than offer as a end-user download.  There is some way to go before this happens, and the security contest is just one step on the journey before NaCl goes live.   The NaCl team also submitted a detailed technical design paper to the IEEE 2009 Symposium on Security and Privacy.  If anyone knows anything on the outcome of the peer review, please leave a comment.

Overall, it has to be said that the NaCl team at Google is doing a solid job trying to flush out security issues before “Primetime”.

Having said that, not all observers agree the architecture is a step in the right direction.  Noted reverse engineer Halvar Flake responded to a post by Dave Aitel on the Daily Dave mailing list remarking that:

“The real beauty in NaCl is that it is certain to defeat DEP [Ed: Data Execution Prevention is an hardware and/or software enabled chipset technology design to throw an exception when an attempt is made to pass off data pages as code pages]. Not that DEP is much of an obstacle in browsers these days, but still. It’ll also almost certainly allow ASLR bypass.

Everyone who has even been to one of my classes has been tortured with the analogy that “writing an exploit is like trying to build a chair out of a number of random parts from the IKEA warehouse: Nothing ever fits, but the more pieces you have, the better your odds of success are.

The power to first execute Javascript to perform [Ed: memory] allocations/dealloctions, coupled with the ability to load arbitrary code into the address space that is only verified under alignment assumptions violated as soon as you can perform a control hijack, does look like a jar of superglue to me. And when you have a sufficiently large jar of superglue, you can essentially build a chair out of wood shavings.”

The point that Halvar is making is that the exploit coder has certain advantages when it comes to exploiting browser based weaknesses. Couple this with the very feature that NaCl introduces - loading Internet hosted native code - and any single implementation weakness makes way for reliable exploitation potential bypassing CPU anti-exploitation features.  This kind of dialogue is very constructive and I look forward to seeing how the thinking around NaCl develops.

Google Apps: Beta Out, Enterprise Features In

Back to the Google announcements, and the day finally came Google dropped ‘Beta’ from Google Apps, Gmail, Google Calendar, Google Docs and Google Talk. This is clearly to please enterprise folks who take the traditional interpretation of “beta == buggy”.  Its hard for a CIO to get buy-in with their own org to adopt a hosted service that has those 4 letters staring back at them (even if they agree with Google’s definition of “beta”.  “Premium beta” anyone? ;-).

Google also added email delegation, retention, DR features to Google Apps, along with “special handling of business users’ data in our data center operations.” If anyone has any details on that last point, do share.  Google is in catch-up mode and ticking the right boxes.

All in all, this was a big day for Google and their evolving Cloud strategy, enterprise security people should take note…

As has been noted before, a blanket ban on legitimate scanning activity by customers of their own infrastructure (whether outsourced or not) undermines security assurance processes and can make regulatory compliance impossible; e.g. PCI DSS mandates network vulnerability scanning as a control.

Vulnerability scanning is a stalwart practice of the Information Security community. Enterprises invest considerable time and money developing vulnerability management programs to help assess IT security risk across applications and infrastructure. Specifically, vulnerability scanners help identify potential security weaknesses at scale; e.g. missing patches, default passwords, coding or configuration weaknesses.

Vulnerability scanning is front of mind for Internet exposed or partner connected infrastructure. However, when said infrastructure is owned and/or operated by a service provider, some of the existing challenges associated with vulnerability scanning are magnified:

• Scans can cause outages.  This can happen if the scanning policy includes Denial of Service checks or the scanning engine is configured with “aggressive” settings; e.g. connection entries in firewall state tables get exhausted.  Its also possible for scans to tickle obscure bugs in the target - or devices enroute to the target.  Even without a full-on outage, poorly configured scans can still negatively impact performance or availability for other customers of shared infrastructure.
• Identifying unauthorised scans. Without a trusted, robust process for “blessing” or approving source IP addresses of customer scan engines, service providers cannot distinguish legitimate scans from scans with the evil bit set.  Sure, they can use whois to determine source network ownership but even if the scan originates from a customer owned network, this does not necessarily mean it is authorised!  Given this, many providers take the stance that all scans are treated as hostile unless pre-agreed.
• Scanning may trigger automated or manual actions by the provider. A common automated response from a provider is to apply traffic shaping to slow down the scan, or simply block the client IP address via an ACL update.  This can lead to false negatives; i.e. vulnerabilities present are not discovered as the scanner IP was automagically identified as a noisy vulnerability scanner and auto-throttled/blocked.  Even half smart attackers can quickly deduces the presence of auto-response mechanisms (”huh, no response now”) so either switches to slow probes from multiple sources or goes for gold with a one-shot exploit.

Enterprise customers on dedicated infrastructure at Tier 1 web hosting providers will either contract the hosting company (or their security partner) to perform vulnerability scans or do it themselves.  Either way, for scanning to happen, agreement will need to be reached on scan scope, types of scans to be run (scanning tools & policies), time windows and source IP addresses used.  Beyond that are the process issues of how results will be communicated, integration with ticketing systems etc.

The provider will limit the scan scope to the dedicated infrastructure allocated to the customer - the scanning of shared infrastructure by the customer is generally a ‘no no’.  This, along with management networks will be scanned by the provider to meet customer compliance mandates or security policies.

With Cloud “Infrastructure as a Service” providers, things get a little more complicated.

• A cloud is multi-tenant; i.e. the cloud platform is shared to multiple customers through software abstraction.  The provider will naturally be concerned with the impact of any scanning activity, particularly if it causes any SLA violations.
• Further, cloud customers can spin up infrastructure on demand.  New virtual servers can be  brought to life automagically to handle increased load.  This increased infrastructure footprint is still subject to the same compliance mandates though; i.e. it must be scanned within some time period of its appearance.  Even if spinning up copies of “known good/secure” virtual machine (VM), you still need to scan them.   New vulnerablities are published all the time, along with corresponding vulnerability checks - hence the need for both regular scans and representative scans.  Further, vulnerbility scanning isn’t just testing the VM, its also helping you verify the security controls outside the VM that are designed to protect it; e.g. a providers’ software firewall.  Picking and choosing which pieces of your hosted infrastructure to scan is a slippery slope to selective exposure if not handled with care.
• Finally, we shouldn’t discount the “Clouding around” factor.  Credit card payments for “instant on” infrastructure changes the dynamic between cloud consumer and cloud provider.  Similar to low end, consumer oriented shared hosting before it, you may never speak with, let alone meet, an employee of your provider before you use their services.  There simply isn’t a conversation about scanning (the “conversation” today is a monologue found in the Terms of Service).  Plus, if the provider fails to meet your needs, you can drop them at a moments notice and switch to another (Cloud baggage permitting…).  In other words, its either not possible, or not convenient to call up your provider to agree the principle and logistics of scanning the services they host on your behalf.  Enterprise customers - or at least their security teams - will be wanting that conversation and can likely strike a deal with a modified ToS to allow scanning of some sort but this seems unncessarily exclusionist to me.

We can address these issues through a mix of provider open-mindedness, policy, process, technology and contract.

For cloud providers to attract certain customers, they may need to soften their policy on vulnerability scanning.  Taking a hardline “no” stance precludes some workloads from ever entering the cloudosphere (with bigger consequences for enterprises seeking a strategic cloud partner).  A preferred scenario has the cloud provider showing some understanding of enterprise prospects assurance needs and defining scanning parameters acceptable to their own operations risk tolerance.

Scanning is not an “unknown” risk, rather its a very well understood activity with quantifiable elements (packet rate, state table usage etc).  Normal rate limiting could be temporarily or permanently loosened for customer approved IP addresses to enable scans against a customers cloud IP addresses (not API endpoints or cloud providers websites!) to complete in a reasonable time window.  Besides, Internet systems are scanned, probed and attacked constantly by script kiddies, Internet surveyors and an assortment of bots and other lifeforms.  So the bad guys get to scan because they don’t care and yet the customer, who wants to do the “right thing”, is not allowed to.  Is that rational?

Assuming a cloud provider with a more measured approach towards vulnerability scanning of customer cloud infrastructure, we now need a simple, mutually trusted mechanism to agree scan sources, rate limits etc.  Something like an “ScanAuth” (Scan Authorize) API call offered by cloud providers that a customer can call with parameters for conveying source IP address(es) that will perform the scanning, and optionally a subset of their Cloud hosted IP addresses, scan start time and/or duration. This request would be signed by the customers API secret/private key as per other privileged API calls. The provider receiving the request can rely on the digital signature as proof that a scan is authorised with the associated parameters. After the provider has processed the scan authorisation request, the provider could return a status code approving or denying the request (with a possible reason code to allow resubmission with more acceptable parameters).  This response can optionally include rate limits which the customer can use to tune the intensity of their scanner.

The provider can now whitelist the customer provided scanner IP(s) for the duration of the requested scanning window such that active countermeasures like anti-DoS controls are not triggered, resulting in a ‘cleaner’ scan (and hence a more accurate report).

Should the scanning activity exceed any specified limits, or communicate with IP addresses not associated with customer virtual machines, the provider could instantly blacklist the scanning IP or apply traffic shaping.

The bottom line: when everyone is clear on the need, approval process, scan parameters and abuse policy, this can be done with very little fuss.

A “ScanAuth” API call empowers the customer (or their nominated 3rd party) to scan their hosted Cloud infrastructure confident in the knowledge they won’t fall foul of the providers Terms of Service. This avoids a situation where either a customers Cloud services are interrupted by an angry provider (availability fail!) or in the worst case, getting kicked off the Cloud entirely.  Clearly, a lose/lose scenario.

What do you think?

The scene goes something like this:

The provider rolls their eyes as yet another customer security team sends in their 500 deeply probing security questions, transmitted in some homegrown template in Word, Excel or $diety forbid, Powerpoint.  The customer security team, naturally suspicious of the provider and irked by managements apparent keenness to outsource the farm, has created the security questionnaire from hell:

• it’s the result of 100 hours of internal team meetings
• it’s gone through 14 drafts, 20 reviewers inboxes, 76 yellow highlighter comment fields and was printed at least 6 times
• it only asks IT security questions (no input from other relevant functions such legal/compliance/audit - HA!)
• it’s laced with a few tricky landmine questions based on potential security issues raised (but not satisfactorily answered) in online forums and provider support forums
• it contains 25 attachments detailing all the company security policies that *must* be followed (huh, Bluetooth policy requirements for a cloud storage provider…interesting)

In the context of cloud providers, they are slammed - a raft of audits in progress right now - with more expected soon.  The provider is experiencing an ADoS (Audit Denial of Service).  Instead of innovating new service offerings (including security!), the talented security professional at the provider is stuck cut and pasting answers from internal cheatsheets to customers questionnaires in the knowledge that the customer likely has no idea how much money it would cost to fulfill some of these security requests.  The sheer number of questions is confusing given that the customer IT team had stated they were only looking to host non-critical, non-sensitive data…

Audits are time consuming, repetitive across customers, costly and generally a motivational drain for everyone involved.  Moreover in the context of Cloud, time consuming audits seriously delays a key benefit of cloud - agility.  Its the “on demand” part of “Infrastructure on demand” that is a primary benefit of cloud.  If the security review process takes 3 months to complete, how much business opportunity has your employer lost?  Don’t like that question?  OK, another one: how much time could you have spent doing something more interesting?

Which leads me to some questions:

• what does the cost/benefit ratio look like of the “questionnaire security review method”? (to be clear, I’m not arguing against the need for security reviews)
• why do we all use different format questionnaires? (note: format)
• why are we asking these questions? (are the bulk of our questions simply an expression of our policy asked in a question format?)
• how many of these questions/policies are predictable and duplicated?  As in, you and I ask some of the same questions…we may differ in the details (e.g. password complexity..eek!) but we both probably ask the same base question even if our thresholds around answers are slightly different.
• what if we were to agree a set of common questions/policy statements?  We don’t all have to subscribe to them, we can pick the ones that reflect our policy…  There could be thousands, you search, pick and mix just like an iTunes playlist (Ed: Genius!)
• for those standard policy questions, could we “digitize” them and express them electronically?  Could the provider host a policy oracle that we could post these questions to?
• for those “uncommon” questions that the providers oracle cannot automagically answer, could we agree a standard way to “ask/transmit” those with some simple agreements about response formats? (um, freetext fields ;-).
• ultimately, could we “digitize” a significant portion of our questions to get near instant answers? (and could we make that multi-lingual…)
• would the provider recognise this as a benefit too?
• would the provider also see the legitimate opportunity this presents to charge for higher assurance services around cloud compute/storage/network based on our policy requirements?  “You want triple cycle, double buffering?  You got it - for an extra 5c per MB”).  Yes, the cost of  your security policies in a pay per drink model are revealed!
• would the provider recognise the opportunity to offer incentives to customers for choosing this low friction path of policy compliance instead of tying up their skilled employees filling out ad-hoc questionnaires?

Is there an existing system/application/protocol whereby I can transmit my policy requirements to a provider, they can respond in real-time with compliance level and any additional costs, with less structured/known requirements responded to by a human (but transmitted the same way)?  In other words, I’m looking for human driven, machine to machine policy exchange/agreement.

I propose that the benefit of quickly ascertaining policy compatibility along with any additional costs involved would reduce the on-ramp to cloud, reduce switching costs, drive a form of policy interoperability and take us closer to where we need to be in the long run: the ability to express security policy for a single unit of compute/storage/network in a cloud.  Ultimately, I want to be able to tie my security policy to the information asset I need to protect and push that to a cloud broker who performs policy reconciliation to determine which of my approved provider(s) can meet my needs without any human intervention (yeah, I can hope ;-).

And before everyone jumps on me and says ‘but the point of an on-site audit/security review is to get assurance that the provider is doing what they claim they are doing” I’d like to point out that policy and assurance are two different things.  Before you and the provider invest time in the optional on-site audit, why not get the bulk of the policy questions out the way in a fast and low cost manner? (i.e. “death to the questoinnaire?”).

If you’re following along thus far, you’ll also see the possibility for trusted 3rd party auditors to digitally ’sign’ individual policy statements made by cloud providers they have audited. That signature could itself reflect the assurance level you need.  This in turn could help drive the nascent cyberinsurance market for cloud…assuming the auditor is open to counterclaims by the insurer ;-).

If you do need to go on-site (and assuming the cloud provider tells you where “on-site” is ;-), you’ll have a list of items the provider categorically stated they do, meaning you can cherry pick the areas where you want to deep dive for assurance.  If upon inspection you find reality does not match stated policy, you can scream bloody murder.  Providers that mislead customers will soon get known.

Thoughts?

On the one hand I totally understand the need for a nation to protect its own interests - particularly where national critical infrastructure is concerned, but on the other, it “feels” a bit strange that an initiative like this is coming from a vendor with a vested interest in Cloud.

Here’s the quote:

Sun’s UK chief technology officer is working with major British public and private organisations to set up a cross-sector forum to resolve cloud-computing security issues.

Cloud-computing systems could become as important as the UK critical national infrastructure, and they need to be secured in an appropriate manner, Wayne Horkan told ZDNet UK on Thursday. The Sun executive said he is working on setting up the forum alongside organisations such as the CBI, Microsoft and Accenture; government departments such as Berr, Dius and the Treasury; and the government’s chief scientific advisor, Professor John Beddington.

“I’m concerned about the security of the supply,” Horkan said at the Cloud Expo Europe conference in London. “If cloud computing becomes a utility, it’s important to me that the UK as a nation state has good security of supply. It’s important that the UK has the appropriate capability in cloud computing.”

He then goes on to cite privacy concerns.

It’s plain to see that the majority of Cloud offerings are from US based companies.  Nearly every briefing I’m invited to is EST or PST.  In fact, I can’t remember even speaking with a UK Cloud provider.   Of the many media requests for comments, all but one were from the US.

I can’t help smelling fear in this effort. As a Brit, I would love to see a UK group coming together to innovate, support and promote the fledgling UK Cloud industry.  Perhaps that will be one of the goals of the group - if so, I don’t think that is ’security’ specific (unless we are talking security innovation).

Development of UK specific Data Privacy guidance in relation to Cloud should be led and enforced by the Information Commissioners Office.

I also feel this will do little to advance security of the Cloud overall. With the positive news yesterday that the UK based Jericho forum and the Cloud Security Alliance (CSA) have formally agreed to “work together”, isn’t this inward looking approach just fragmenting our efforts?  Why not direct the security talent that would comprise this group towards the CSA or ENISA.

Security is a *global* issue.  I’m struggling to see how country specific cloud security interest groups “fit” when we talk about globally distributed systems.  What next - Cloud UN? ;-).

I don’t disagree with the need to protect supply, but I would much prefer to see the UK government driving an initiative like this as part of their critical infrastructure protection strategy.  A strategy around UK Cloud innovation would be nice too ;-).

Perhaps I am being overly pessimistic or missing something. What do you think of a country specific Cloud security group set up by a technology company? A US based technology company no less… ;-).

Launched last month, the founders are security professionals from Cloud customers and Security in the Cloud providers (with sponsorship coming from the latter).  The Technical Adviser is friend and fellow security professional Chris Hoff.

From the Introduction on page 5 of the guidance document:

The Cloud Security Alliance is a grassroots effort to facilitate the mission to create and apply best practices to secure cloud computing. Incorporated as a not-for-profit organization, our efforts will seek to provide a voice for security practitioners. However, recognizing that a secure cloud is a shared responsibility, we will be inclusive of all organizations and points of view to fulfill this mission.
What follows is our initial report, outlining areas of concern and guidance for organizations adopting cloud computing. The intention is to provide security practitioners with a comprehensive roadmap for being proactive in developing positive and secure relationships with cloud providers. Much of this guidance is also quite relevant to the cloud provider to improve the quality and security of their service offerings. As with any initial foray, there will certainly be guidance that we could improve upon. We will quite likely modify the number of domains and change the focus of some areas of concern. We seek your help to improve this guidance to make version 2.0 of this document an even better asset to the security practitioner and cloud provider.

How To Get Involved

This is a real opportunity to shape the future security of Cloud. With sufficient participants, a mature guidance document and strong awareness, I believe a group like this can make a real impact on the future of Cloud Security. Its my view that this advances the Cloud Security conversation which is a major reason why I started this blog and will be contributing as I can.

If you’ve been sitting on the sidelines up to now, I encourage you to get involved and contribute as little or as much as you can.

Getting started is easy:

1. Join the CSA linkedin.com group to become an official member of the group (I’m already a member).

2. Review and give feedback to the CSA guidance document via the CSA Google Group.

Finally, the CSA have a number of  events planned to spread the word, including Gluecon (Denver), ISSA CISO Forum (Chicago) and the Cloud Computing Expo Europe in Prague, Czech Republic.  More info here.

Legal Cloud today announced that several top, international law firms had signed up as early testers of its virtual data center services for the legal market. The Legal Cloud is operating a ‘closed beta’ with select law firms interested in reducing the costs of their existing collocation facilities, finding a way to implement a business continuity program without duplicating private infrastructure or simply planning for their future primary and secondary infrastructure facilities.

What Makes This Different?

From their blurb:

The founders of the Legal Cloud have been working in the legal technology industry for over a decade. We understand that the needs of international law firms are different to other industries. Our data centers are optimized to meet the needs of law firms. Our choice of technologies, performance, data storage, latency, service level agreements, security and features have all been specifically devised to support the needs of the legal industry (source).

Why This Is Important From a Cloud Security Perspective?

• This cloud is designed around the needs of a specific industry:  with a well defined set of clients in mind it can cater to the groups specific operational and security needs
• These are not just “any customers”: international law firms that will have legal, compliance and security requirements over and above your “average” cloud customer today.  This needs to be a cloud with ‘higher assurance’ features to gain the trust and buy-in of legal CIOs
• The security conversation suddenly becomes a lot more focused: we are not talking about a general “one size fits all” cloud anymore and facing the disharmony of varying customers security needs and provider capabilities.  This may sound trivial but security conversations can get painful fast when customer and provider come from different worlds.
• In a view I’ve held for a longtime, its a taste of things to come: banking clouds, healthcare clouds, federal clouds (happening now).  Yes, there are other industry specific clouds (e.g. Salesforce Service Cloud) and they have their own security requirements, but arguably less assurance will be demanded by customers.
• The customers become an important lobby group for future security feature requests: instead of X voices asking for completely different things, the community of Legal Cloud users will state requirements “loud and clear” and if nScaled doesn’t listen, provide an opportunity for “Another Legal Cloud” to steal customers.
• The success of this cloud will be judged by many: if nScaled delivers on their promise, they will benefit from first mover advantage and become the “standard” for legal cloud.  From my UK experience, the legal community is cautious about new technologies and is a pretty tight-nit group, so if sufficient “established” legal firms move its not hard to imagine many more following (well, I’m sure that’s what nScaled hopes ;-).

What Is On Offer?

Legal Cloud is offering the following on a services basis:

• Fully virtualized data centers
• Business Continuity Service
• Active Cloud Servers
• Unlimited Storage
• Snapshot recovery points

And here’s how it looks from a 50,000ft:

What Do They Say About Security?

After a brave headline of “Security Guaranteed” (sure to rile anyone in information security), they go on to state:

The security of your data is of paramount importance. Here is how we guarantee it’s security.

Secure Data Centers

Our data centers are highly secure and redundant precision environments backed by the Fanatical Support of Rackspace. (SAS-70 Compliant)

Secure Virtual Private Networks

We extend your network into the Legal Cloud using VPN (Virtual Private Network) and VLAN (Virtual LAN) technologies. Your data is encrypted during transit with IPsec. Within the Legal Cloud, your data is segregated in logically separate areas from other clients data and attached only to your private networking. This gives each client their own private network and storage in the Cloud.

Data Encryption

Client Data is encrypted from client source servers to target devices using strong encryption protocols.

Not on the public Internet

The legal Cloud is not exposed to the public Internet. It is actually an extention of each clients internal network, each seperated by strong security protocols.

Service Level Agreements

We are working on appropriate SLA’s for our legal customers during the beta period.

Psychologically, I suspect the most significant reassurance for many CISOs will be this one single sentence: “Not on the public Internet”.  Beyond that, use of IPsec will make this feel very much like a standard 3rd party ‘partner’ connection.  I don’t see any mention of storage encryption options as yet, nor any further detail on the logical separation - once I’ve had a briefing and can speak more to the security aspects, I’ll post more.

P.S nScaled have annouced a couple of webinars aimed at their target audience here.

From comments I received afterwards, I got positive feedback despite running out of time (my fault entirely).  I’ve been pleasantly surprised by the number of people asking for copies of the slides, but do bear in mind the slides are somewhat ‘terse’ as they are primarily talking points for me to bounce off of (as it were).

Should anything not be clear, feel free to leave a comment below and I’ll do my best to clarify.

I’d also like to take this chance to thank Jeff Moss, Ping and the rest of the Black Hat crew for doing such a professional job running the conference - it was confidence inspiring to be in such capable hands.

Introducing the slide, I remarked that its important to differentiate the two:

• “Cloud Security”: this refers to the security of “the Cloud”, or more usefully, of a given cloud.  Stepping back, we can use the term to refer to the general security aspects of Cloud Computing.
• “Security in the Cloud”: this is about delivering security services via “the cloud”.

Back in April 2008, when I was naming this blog, I initially planned to call it ‘Security in the Cloud’ but after 30 minutes of Googling and reading, it became evident that I was mistaken as this term had already been adopted to refer to services delivered via the Internet (primarily Security MSSPs).  Hence cloudsecurity.org was born.

Having said all that, I’m now seeing newer “security in the Cloud” providers referring to themselves as ‘the Cloud Security Leader’ which only serves to add to the confusion.

[This post was inspired by "The Real Meaning of Cloud Security Revealed" by Lori MacVittie]

Now’s your chance to express them…

ENISA (the European Network and Information Security Agency) is conducting a security risk assessment of cloud computing.

If ENISA is unfamiliar to you, here’s how they describe themselves:

• Is a Centre of Expertise for the EU Member States and EU Institutions in Network and Information Security, giving expert advice and recommendations
• Is a switchboard of information for best practices
• Facilitates contacts between the EU-institutions, the Members States and the private business & industry actors

For the Cloud Risk Assessment, the group (of which I’m a member) will focus on three scenarios:

1. A user perspective on Cloud Computing (i.e. Small and Medium Enterprises)
2. Cloud Computing in a eGovernment environment (i.e. national health service)
3. Cloud Computing and Resilience

In pursuit of the first scenario, ENISA is seeking feedback:

“…aimed at giving advice to (among others) SME’s on the most important risks in adopting cloud computing technologies, as well as ways to address those risks.

As part of this study, we want to look in detail at the perspective of SME end-users of cloud computing infrastructures and applications (either current users or those considering adoption). As a first step, we have decided to base our study on a survey of the actual needs, requirements and expectations for cloud computing infrastructures.”

Take the 10 minute survey here (results will be shared).

This post is a summary of a very interesting call I had with George Reese, CTO of enStratus and author of the forthcoming “Cloud Application Architectures” book.  Please note: this isn’t a comprehensive review of the full service, rather it reflects the pieces that we delved into based on some of the common concerns we have around Cloud Security (to give you some idea, we spoke for over 90 minutes…).

enStratus offers cloud infrastructure management tools “aimed at the needs of enterprise IT”.  Today, they support Amazon EC2, with support for other clouds to follow.

Their tag line is ‘Confidence in the Cloud’ and their offering focuses on 3 key areas addressing the twin cloud adoption barriers of security and reliability:

• protecting cloud based data through encryption
• offering service levels above that of the underlying cloud provider (99.9999% for EC2)
• achieving Recovery Time and Recovery Point Objectives “in the face of the most extreme disasters”.

George outlined 3 concerns his customers have about cloud providers such as Amazon:

• Amazon controls the physical systems on which the data resides, meaning Amazon malfeasance, Amazon misfeasance, or even 3rd party subpoenas put that data at risk.
• The complexity of resource orchestration in the context of credential management; i.e. when do your credentials need to be in the cloud versus when their presence is just a security risk
• User management, even via traditional identity management systems, can be dysfunctional.

The enStratus Approach to Cloud Key Management

One of my pet peeves with AWS is the “one key to rule them all” security model (the dysfunctional user management George alluded to).  Any disclosure of that key results in an attacker gaining access to all your infrastructure.  But to make privileged API calls, every developer must have a copy of the key…

Its not unknown for AWS users bundling an AMI (creating a virtual machine image) for public consumption to leave their AWS credentials in the AMI itself.  Oops. This is obviously a Bad Thing ™ as a malicious user that opts to use that AMI can steal their access key, gain access to their Amazon hosted infrastructure and run up bills in their name.

One of the things I really like about the enStratus offering, is the relentless focus on controlling the use and hence exposure of a customers’ ‘cloud masterkeys’. Their implementation keeps the keys away from the AMI, and therefore the cloud, PLUS out of the hands of an org’s IT/dev people.

enStratus acts as a trust broker. After signing up for the service, the customer loads their “all powerful” Amazon credentials via a shared enStratus Provisioning Server into a Credentials Server (no direct internet connectivity).

From that point forward, the customers’ IT people access the enStratus service and manage their cloud infrastructure via named user accounts assigned specific privilege levels.

Permissions include;

• administrator
• start/stop servers
• uptime retrieval and
• audit trail review.

Non-administrative users have no direct access to the AWS keys.

Here’s a peek at the architecture of enStratus.

When an authorised enStratus user issues cloud infrastructure management requests via the Web Services and Console server, the Provisioning server issues the cloud API calls on behalf of the users.  This eliminates the need for every user needing a copy of the key to sign requests.  Given they are mediating API requests, adding logging functionality was a no-brainer and means the next time you need to know ‘who spun up that unpatched AMI image with an allow-all security group?’, you can find out.

Its important to note that there is nothing preventing anyone with your AWS key from just making API calls directly to the AWS API endpoint - totally bypassing the enStratus infrastructure.  Therefore, careful key lifecycle management is still necessary; i.e. load fresh AWS credentials straight into enStratus and follow a “no sharing” policy.

I should point out that the EC2 ecosystem players can’t do anything about this as the AWS platform doesn’t offer a mechanism to tie IP level controls to AWS key usage or EC2 (yet).  One way Amazon could implement this (nothing announced) is with their new JSON based Access Policy Language.  Despite this, enStratus can still detect new EC2 instances spun-up by API calls they didn’t mediate, through telemetry used for operational monitoring - they just won’t be able to tell you who started it.

enStratus can help customers build their AMIs, including bundling in HIDS (Host based Intrusion Detection) via ossec, with centralised agent reporting.  Another example of how they protect the AMI key is through error checking in their scripted AMI builds to ensure key material is not left in an AMI accidently.  In addition, users are prevented from accessing partially provisioned AMIs (to eliminate potential key snarfing shenanigans).

Root access to EC2 images is disabled by default (unlike with vanilla EC2).  Privileged access is granted via sudo.

Filesystem Encryption

enStratus offers optional filesystem encryption through a checkbox.  Keys are temporarily passed into the EC2 instance when required; i.e. mounting.

Encrypted filesystem support is implemented via 2 block volumes configured as RAID 0.  2 sets of encryption keys are used.  One for encrypting and remounting the ephemeral drive (this is a “non-persistent store” automagically attached by EC2 to each running AMI).  The second key pair is used to encrypt and mount filesystems attached as Elastic Block Storage (EBS).  EBS is off-instance, persistent storage.  To avoid potential exposure of key material, the 2nd set of keys are stored on the encrypted ephemeral drive during mount.

Worth noting is that in testing, George found that 2 EBS volumes, configured as RAID 0 with an encrypted XFS filesystem  offers similar performance to a single, unencrypted EBS volume with an ext3 filesystem.

George is keen to stress that enStratus is not looking to control both customers data and their keys.  So whilst he recommends and can help customers make use of the EBS snapshot feature to clone/backup storage volumes to Amazon S3 (Simple Storage Service), he isn’t offering a hosted backup service (to avoid a potential conflict).  Of course, an evil and privileged enStratus employee could access your live data as the keys are stored in their Credential server.  Today though, enStratus is a small company so figuring out ‘who dunnit’ would not require the services of Sherlock Holmes.

Futures

Today, the enStratus management infrastructure sits outside of the cloud (at a colo) for cloud monitoring and isolation reasons.  George is exploring the possibility of also offering an on-premise offering for customers wishing for more control.

I offered a few short and medium term suggestions around additional integrity checks, encryption ideas, assurance processes (source code security reviews, penetration testing) and consideration to using a Hardware Security Module (HSM) for key storage to further bolster both security and trust.  George seemed genuinely open and receptive to these ideas and also shared a few interesting customer requests they are actively working on today.  Some of the more expensive line items would become practical if they can attract additional funding.

Overall, I have to say I’m impressed with their approach, technology and attitude.  Definitely worth a hands-on evaluation if current Cloud controls don’t fall within your risk tolerance.

Good luck to George and the rest of the enStratus team as they prepare to present at Under The Radar!

Attending Under The Radar?

As a special offer to cloudsecurity.org readers, the organisors of Under The Radar are offering $100 off the list price for VIP tickets.  To claim yours, click here.

A Quick SQS Reminder

For those unfamiliar with SQS, here’s the elevator pitch from Amazon:

Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed components of their applications that perform different tasks, without losing messages or requiring each component to be always available. Amazon SQS makes it easy to build an automated workflow, working in close conjunction with the Amazon Elastic Compute Cloud EC2 and the other AWS infrastructure web services.

Amazon SQS works by exposing Amazon’s web-scale messaging infrastructure as a web service. Any computer on the Internet can add or read messages without any installed software or special firewall configurations. Components of applications using Amazon SQS can run independently, and do not need to be on the same network, developed with the same technologies, or running at the same time.

So, a very handy data structure that makes perfect sense in distributed programming.  However, access control options were limited…until today.

The New Secret Sauce

AWS is also introducing additional permission features that control access to Amazon SQS and to each of its fundamental actions on a very fine-grained basis. You can exercise this control at two levels:

* At the higher level you can use the new AddPermission and RemovePermission functions to set and remove particular access rights for each queue. Access rights, including the ability to send, receive, or delete messages, change message visibility, or to retrieve queue attributes, can be granted to any AWS user via their AWS account number.
* At the lower level you can use our new Access Policy Language. This expressive language makes its debut as part of this SQS release; over time, we plan to employ this Access Policy Language with our other services. The Access Policy Language enables the creation of complex rules to enable access to queues based on identity (AWS account number), source IP address, date, time, and more.

With this new permission system you can now use Amazon SQS queues to connect non-AWS applications to AWS applications and to connect AWS applications from different organizations. You could use an open, public queue as a drop box, allowing outside applications to submit work items for processing. This could be a fully public drop box, or it could be limited to requests from a single country by using a policy based on an IP address or address range. Communication between organizations can be established based on IP addresses or AWS accounts, as appropriate.

For me, the most significant news is not so much that SQS now has fine grained access control, but that Amazon have introduced a Access Policy Language and they plan to apply it to other AWS services.  This is a very positive development and could be the mechanism they use to overcome some of the longstanding security concerns I blogged about recently.

Architectural Overview

For the visually inclined (source):

Where:

1. You, the resource owner.

2. Your resources (contained within the AWS service; e.g., SQS queues).

3. Your policies.  Typically you have one policy per resource, although you could have multiple. The AWS service itself provides an API you use to upload and manage your policies. For information about the content of the policies, see How to Write a Policy.

4. Requesters and their incoming requests to the AWS service.

5. The access policy language evaluation code.

This is the set of code within the AWS service that evaluates incoming requests against the applicable policies and determines whether the requester is allowed access to the resource. For information about how the service makes the decision, see Evaluation Logic (Ed: note there are soft and hard denials).

An Example

Here’s an example from the developer docs showing a simple IP based control (multiple controls can be applied to a single resource):

The following example policy gives all users permission to use all possible SQS actions that can be shared for the queue named 987654321098/queue1, but only if the request comes from the 192.168.143.0/24 range.

{
  "Version": "2008-10-17",
  "Id": "Queue1_Policy_UUID",
  "Statement":
     {
        "Sid":"Queue1_AnonymousAccess_AllActions_WhitelistIP",
        "Effect": "Allow",
        "Principal": {
           "AWS": "*"
         },
        "Action": "SQS:*",
        "Resource": "/987654321098/queue1",
        "Condition" : {
            "IpAddress" : {
               "SourceIP":"192.168.143.0/24"
            }
     }
}

Conclusion

Notice the values for the ‘Action’ and ‘Resource’ tags.  Now imagine those with different AWS service identifiers and resource types and things start to get really interesting.

Now all we need is an user-friendly, hard-to-shoot-yourself-in-the-foot policy generator/front-end to open this feature up to the masses.

All in all, its great to see the introduction of a consistent policy language from the cloud pioneer.

I’m off to learn more about the language…

Update: in case it isn’t obvious from the example, the policy language is expressed using JSON (thanks @lmacvittie for the prompt)

“You agree not to use the Service in the design, development, production, or use of missiles or the design, development, production, stockpiling, or use of chemical or biological weapons.”

I’m glad they cleared that up - now all the bad guys know to use Amazon AWS or Microsoft Azure.

P.S I can’t help feeling the irony, given the App Engine logo…

In addition to offering Java (with an Eclipse plugin to follow) and a cron facility for scheduled jobs, they now offer a way to integrate your existing enterprise applications into App Engine.

From the announcement of Google Secure Data Connector (SDC) :

SDC provides the following functionality:

1. Secure link.Encrypts connectivity between Google Apps and your network. Google Apps is the only external service that can make requests over this connection.
2. Filters.Limits the scope of the types of requests that can be routed over SDC. You can configure filters to limit which gadgets, spreadsheets, and App Engine applications may access which internal systems. Filters may also be used to control which users can access and share data from your internal systems within Google Apps. You can use the user level filter control to augment the security provided by your internal system for verifying users and originating applications. Filters should not be used as a standalone authentication system.
3. OAuth Signed Fetch.Protects requests that are made through SDC. You can use Signed Fetch to authenticate requests from Google if your applications are OAuth aware. Note that SDC provides connectivity between Google Apps and your internal systems. If you do not use OAuth Signed Fetch, then SDC does not provide authentication.

In case pictures help, this is how it looks:

And these are steps:

1. Google Apps forwards authorized data requests from users who are within the Google Apps domain to the Google tunnel protocol servers.
2. The tunnel servers validate that a user is authorized to make the request to the specified resource. Google tunnel servers are connected by an encrypted tunnel to SDC, which runs within a company’s internal network.
3. The tunnel protocol allows SDC to connect to a Google tunnel server, authenticate, and encrypt the data that flows across the Internet.
4. SDC uses resource rules to validate if a user is authorized to make a request to a specified resource.
5. An optional intranet firewall can be used to provide extra network security.
6. SDC performs a network request to the specified resource or services.
7. The service verifies the signed requests and if the user is authorized, returns the data.

This will enable a whole host of data mashups using an enterprise friendly language (Java).

What I found most interesting (beyond the network security issues of Google having a large tentacle into your network), is the API trust model relies on OAuth.  All the cool kids do OAuth these days, even Twitter recently joined the OAuth party.

Google are offering OAuth as the only mechanism to authenticate Google to your enterprise applications.

Which begs the question: how many of your legacy apps are OAuth enabled?  You don’t know?  Let me help you out here: None.

This is a case of Web 2.0 meeting Enterprise -1989.

How many of your legacy apps could even be OAuth enabled?  OK, take away the word legacy (there is an implicit insult when vendors call all your apps legacy), how many of your newfangled apps could support OAuth?  This question is worth asking as without OAuth support, there is no way to authenticate Google to your applications.   Hmmm, IP level controls OK?

If Google can drive OAuth into the enterprise then frankly, they would have done us a favour.  But how many organisations are going to be willing to re-engineer apps?  Of those that are, how many are just going to say ‘its OK, its no different from our partners accessing our database (hello?)…open the firewall’.

It will be really interesting to observe how enterprise application architects react with CloudVision(tm) react to this.  It will be even more interesting to measure the enterprise uptake of OAuth (I’m not holding my breath).

I’m a big fan of Amazon AWS - they are true innovators and the diversity of their service offerings is astonishing.  There are many use cases for Cloud Computing and not all have high assurance needs. But when touting HIPAA compliance for the processing and storage of medical data, I can’t help feeling there are (amongst other things) glaring API level control gaps that would make me extremely uncomfortable with my medical data stored in the Amazon Cloud today.

Amazon is marketing their Cloud - with its current security controls - as able to support HIPAA compliant applications.  The whitepaper does a good job talking up their existing security controls but omits to mention a number of serious control gaps that decision makers should be aware of prior to committing “other peoples data” medical data to the Amazon Cloud.

I’m not a HIPAA expert and therefore in my ignorance, it may well be that AWS can satisfy covered entities security requirements (if so, it says more about HIPAA than AWS or, more about a specific use case).  Consequently, I’ll step aside from the ‘compliance’ angle for a moment and confine my comments to ’security’ and encourage those with first hand HIPAA compliance experience to comment as they see fit.

There is no customer accessible AWS API call audit log

In other words, you have no way to know if, when and from where (source IP) your AWS key was used to make API calls that may affect the security posture of your AWS resources (an exception is S3, but only if you turn on logging (off by default)). For example; if your key is compromised without your awareness and your EC2 “security groups” (firewall) are changed, there is no customer accessible log record. The only way to detect this is to poll the API on a regular basis (DescribeSecurityGroups) and compare with the output of previous calls or to call up Amazon support on a regular basis and ask them “is all well?”  (j/k).  I can think of much more sinister API calls to invoke, the security groups is just an example.

They also afford no customer visibility over failed authentication attempts at the AWS web interface or at the API.  Again, no visibilty to understand if your infrastructure in the sky is under attack.  Fine for some workloads, not so fine for others.

Oh, and while we are at it, why is there no firewall log for the built in ’security groups’ firewall functionality?  To get visibility today, you have to run a host based firewall with logging enabled but even then you will not see what was dropped by Amazon (when doing network security investigations, this information can be really helpful).  Basic reporting accessible to customers would be a good start.

There is no way to restrict the source IP address from which the sacred AWS API key can be used from

The AWS API interface can be used from any source IP at any time (and as above, you have no audit trail for EC2 API calls).  This is equivalent of exposing your compute and storage management API to the entire planet.  To be clear, they are not exposing their *internal* management plane - the infrastructure that the Amazon AWS team uses to manage the AWS infrastructure, rather its *your* hosted infrastruture management plane that is exposed.

Why can’t the EC2 security group concept (aka firewall) be applied to the AWS API, to limit access from approved management stations? After creating my key I should have the option to limit use to specific subnets of my organisation.  I’d love to hear this option is on the Amazon security roadmap (along with visibility of failed attempts).

Each AWS account is limited to a single key - exposure results in total security failure

I’ve said it before and I’ll say it again - you only get a single key per AWS account.

That makes it really hard to segregate environments.  To do so, you have to create multiple AWS accounts with different credit cards and even then that only gives a very coarse grained access model.

With your master key, why can’t you generate a handful of subkeys for different use cases?

If you could link those subkeys to roles that reflect your security policies, then you’d really have flexibility.  This would also mean you could lock away the all-powerful master key for use only when such power is warranted (thus reducing potential exposure).

The failure mode of a single key is ‘complete and utter’.  If my key is compromised, my entire AWS environment is compromised - all my virtual machines, all my data.  I don’t know about you, but that scares the living daylights out of me in the context of medical data.

Takeaway

Hopefully we all know by now that “compliance” does not equal “security”, but HIPAA compliance not withstanding, would you really want your medical data in a Cloud without some or all of these fundamental control gaps resolved or mitigated?  I can’t find anything in this new whitepaper or the old AWS Security Whitepaper that speaks directly to these issues.  If I missed something, please share below and I’ll update the post.

When medical data is stored at the medical provider, I can believe they will have a firewall that doesn’t allow outsiders to make API calls to their compute and storage management plane from anywhere in the world…  I can believe that leakage of an internal infrastructure management password cannot be used from *outside* the firewall.  At least, not without first exploiting an unrelated security weakness to gain access to the environment in the first place.

I can appreciate that providers are keen to demonstrate use cases but I can’t help feeling that in their eagerness to market ‘compliance’, Amazon is putting the cart before the horse from a security perspective.

What do you think?

This presentation rips apart the hype and “newfangledness” of Cloud Services (IaaS, PaaS, SaaS etc) to expose the ghost in the machine. Just as the human brain has grown, built upon earlier, more primitive brain structures, so it is with the Cloud. With the advent of commercially available, pay-as-you-go public Cloud services, CFOs are casting a weary eye to the CIO in anticipation of joining the great infrastructure linedance in the sky. Meanwhile, vendors are jockeying for position to “enable” the Enterprise Cloud and Cloud brokers are trading excess compute capacity in data centers. What does all this mean from a security point of view? What are the security risks (and benefits)? Are you ready to face the ghosts in the Clouds?

I’m really looking forward to this.  If you’re going to be there, let me know!

Full details of the conference, along with paying 200 Euros less if you register by Wednesday 8th April are here

I’m proud to announce the results of a recent security collaboration with Amazon AWS.

As Jeff Barr announced on the AWS blog today:

Early this morning we launched a brand new cloud computing service. This revolutionary new technology will change the way you think about the cloud.

For a while the cloud was simply a metaphor meaning “a bunch of computers somewhere else.” Until now, somewhere else meant good old terra firma, the Earth itself. After extensive customer research we found that this rigid, antiquated way of thinking just won’t cut it in today’s post-capitalist world. They need locational flexibility, the ability to literally instantiate a cloud where they need it, when they need it.

To solve this problem, we have designed and are now introducing the Floating Amazon Cloud Environment, or FACE for short. Using the latest in airship technology, we’ve created a cloud that can come to you.”

If you’ve been watching Amazons SEC filings, you’ll know they invested heavily in nano. FACE is the first realisation of that investment.

Jeff continues on to implementation details:

The FACE uses durable, unmanned helium-filled blimps with a capacity of 65,536 small EC2 instances, or a proportionate number of larger instances. The top of each blimp is coated in polycrystalline solar cells which supply approximately 40% of the power needed by the servers and the on-board navigation, communication, and defense systems. The remainder of the power is produced by clean, efficient solid oxide fuel cells. There’s enough fuel onboard to last about a month under normal operating conditions. Waste heat from the fuel cells and from the servers is used to generate additional lift.

This is a big win energy wise but presents some interesting communication and security issues.

There are two options for ground communication, WiMAX and laser. The WiMAX option provides low latency and respectable bandwidth. If you have the ground facility and the line of sight access needed to support it, lasers are the way to go. The on-board laser doubles as a defense facility, keeping each FACE safe from harm. Using automated target detectors with human confirmation via the Mechanical Turk, competitors won’t have a chance.

I can now spill the beans on the security aspects of the solution (subject to NDA).

FACE Security

Since FACE is an untethered environment, ensuring cross airspace data transfer compliance was a non-negotiable. It was therefore essential to implement a data privacy ‘hints’ system, whereby the on-board GPS system could be programmed to correlate GPS co-ordinates with terrain specific data privacy laws and issue AMQP style ‘nudge’ messages to the navigation system to counteract potential jurisdictional data drift. The neat thing about this approach was that different FACEs could be deployed to satisfy customers in different regions (much like EC2). Furthermore, should two FACEs need to converage for cross-FACE data transmission, one FACE could draw energy from the other FACE’s solar cells. This turned out to be very useful for availability, particularly for the UK FACE where low cumulus made solar cell charging difficult (thanks to the Portugal team!)

Another security concern was the laser. Amazon legal was naturally concerned about potential liability issues should an attacker compromise a FACE and launch a reverse protocol attack to commandeer ground facilities. If an attacker were able to take over the lasers this would not only be a physical security risk but a PR disaster for Amazon. This led to the development of a novel security protection we nicknamed FACEMASK (original huh?). The idea behind FACEMASK is really simple: treat rapid changes in the solid oxide fuel cells as a potential breach indicator. How so? It turns out that both stack and heap buffer overflow attacks result in a fluctuation of the normally highly stable oxide full cells powering FACE. This isn’t special in itself, however the fingerprint of the energy draw *is*. We developed a catalogue of fingerprints and in testing were able to detect 91% of attacks reliability. No security is perfect and we’ll continue to refine the coverage, however compared to existing signature based defenses, this is orders of magnitude better.

Anyway, I promised I wouldn’t say more, but I hope this gives you a taste of the unique challenges and solutions and how “off our box” thinking can be applied to Cloud Security.

As Jeff hinted in his blog post, this is a limited offer - the doors are likely to close tomorrow. For more information, click here.

“There are composite solutions [to compliance issues]: build the application in the cloud using nothing but anonymous tokens to identify customers… but that is not trivially easy to do,” he said.

“Instead, compliance as a service maybe be offered where [the service provider] acts as an intermediate layer of your application that takes care of a variety of things. They could indemnify the customer against any issues around personally identifiable information crossing boundaries.”

Under such a compliance service, a service provider would accept the burden of knowing the rules, court precedents and regulations which are industry-specific, Coffee said. Responsibility to sanitise data wherever it left the country over a broadband link would move from the customer to the service provider.

“Layers upon layers of new services will emerge representing new layers of expertise and therefore new layers of profitability for those providing services with that kind of value. I think that’s happening now and more so all the time.”

If you consider the cost, complexity, misinterpretations and challenges that organisations face trying to be ‘compliant’ today with their in-house IT, “Compliance as a Service” (CaaS) has to be a Cloud marketeer’s dream!

More seriously, how else can you enforce continued compliance across multiple service providers? This comes back to the notion of packaging security policy along with data, such that in a multi-Cloud provider environment, there is a way to establish automagically who can meet the policy requirements on a dynamic basis.  But would you trust the digital word of the providers?  A provider could accidentally or intentionally affirm compliance with a digitally transmitted policy and go on to accept/process workloads in Clouds that are not suitable/compliant for the data.

Could a 3rd party CaaS inserted “man in the middle” style, act as a trusted arbitrator? If a CaaS provider offers to bear liability for compliance misses and was able to satisfactorily hide the complexity of compliance for the right price, you could foresee such a provider establishing a dominant position in the Cloud-o-sphere.

Right now though, this is all speculation on my part.  Does anyone know of such a service or are you developing one? I’d love to hear about it.

EPIC forwards this complaint on 3 primary fronts:

• the specific ways that Google represents their security controls to consumers (yet disclaims all responsibilities in the Terms of Service)
• the “harm” caused by the recent Google Docs privacy breach
• the claim that Google has “inadequate security”

Secondary arguments include citing a number of other, older vulnerabilities in Google online services and referencing some significant privacy breaches where the FTC acted before.  In my view, these are distractions and inconsistent with the primary argument.  The call for Google to pay 5 million dollars is poorly framed, seemingly an afterthought and potentially devisive.  I suspect EPIC will have lost the goodwill of privacy moderates by making such a demand.  Had they just dropped the number and left the call for a fund, it might have made it more palatable.

Given the complaint is 15 pages long, there is plenty to comment on.  For the sake of brevity, lets contain our analysis to the primary arguments, introduce a potential curveball and go “one step beyond” to examine the implications for Google users should the FTC rule in EPICs favour.

What Google Says About Security

EPIC highlights two specific security claims made by Google.

On the Google Docs homepage

Getting to know Google Docs> Saving your presentation

The complaint then goes on to suggest that Google “encourages users to add personal information to their documents and spreadsheets” and repeats the statement made by Google that “your data is private, unless you grant access to others and/or publish your information”.

Having built their primary argument based on public statements made on Google online properties, they bring out the Google Terms of Service which states in Section 14.1 that the services are provided “as is”, with no warranty and that Google does “not represent or warrant” that [14.2 B] “your use of the Services will be uninterrupted, timely, secure or free from error”.  Section 15 states that Google will not be liable for losses.

The Harm Caused By The Google Docs Privacy Breach

EPIC then attempts to link the Google Docs privacy breach with harm experienced by Google users:

Curious.  2 sentences in a 15 page report where EPIC could have firmly established the ‘harm’ case.  No examples, no quantification, no impact analysis. Perhaps EPIC is playing its hand carefully and is readying a parade of impacted users who can demonstrate they were “harmed” by the privacy snafu.  Failing that, it would mean they have built their case on the morality of a software privacy bug at a popular online service and ultimately, an industry wide disparity between the big print a company uses to market their services (and software!) as trustworthy and the small print where the lawyers spell out the case for the defense.

The Claim That Google has Inadequate Security

The third and final primary argument.  Skipping past the reminder from EPIC to the FTC that they acted in response to other privacy breaches, EPIC goes on to state that Google’s “inadequate security is an unfair business practice” and that Google’s “Inadequate Security” is a deceptive trade practice”.   They argue that the Google Docs privacy breach was a result of inadequate security practices and that Google:

• encourages people to share “sensitive” documentation in promoting their services
• “knew that Cloud Computing Services are susceptible to data breaches”
• “knew that disclosure of personal user data could cause substantial injury to customers”
• was “aware that commonsense security measures, including storing user data in encrypted form, rather than in clear text, could reduce the likelihood and extent of consumer injury”
• “created an unnecessary risk to users’ data by employing unreasonable security practices, including the storage and transmission of personal information on its computer network in clear text”

What I find fascinating about this is that EPIC is drawing a significant conclusion about Google’s security practices based on the fact that Google doesn’t take “commonsense” security measures. In other words, because Google hasn’t implemented a PKI and DRM for document sharing in a (for many) free service, Google is somehow employing unreasonable security practices (!).

This just strikes me as really unreasonable and wholly unrealistic.  If the FTC mandated those level of security protections to “qualify” for accepting data that consumers choose to put in the Cloud, you can say goodnight to *all* of the popular Web 2.0 services.

With Google thoroughly chastised, they draw the following “big picture” conclusion:

First Google, now the Cloud Computing as a whole - I’d better change my domain name fast! ;-).

A Potential Curveball

After I posted my thoughts on how Google Security responded to the Google Docs sharing problem I was contacted by a Google Docs user who stated that he reported a sharing problem to Google in January. He discovered a large number of documents shared with over a hundred people (he gave specific numbers that I’m intentionally not quoting to protect his privacy).  He states he called Google Tech support who initiated a support case.

Assuming this is true (and based on his note, I have no reason to doubt it) it means either:

• Google Docs has suffered another sharing problem that was quietly fixed (no notification?)
• If this is the same sharing problem, it means at least someone in Google knew about it from late January which completely changes how their responsiveness to dealing with this problem will be perceived.

What If EPIC Gets Their Way

If the FTC went as far as forcing Google to suspend its services, we will witness the largest Denial of Legitimate Service (DoLS) attack in history.

Can you imagine how that would play out?  I suspect it would also be the worst PR disaster of all time for EPIC as Google users turn on them in their droves…

In their concern for privacy, one part of security that EPIC seems to have forgotten is availability and the Cloud is all about that.

Why is this useful?  As he states:

Following Terry Matthew’s Sir Terrence Matthews “checkerboard model”, it should be easy going forward to find logical areas that need to be built out. Think about it as “X for the cloud”. For example, identity management from the last era was mostly LAN/WAN-based single-sign on and directory service based. “Identity for the cloud” is a logical hole to fill and sure enough, that is what Symplified is aiming to do.

The map shows the companies along the X axis and the service offerings on the Y axis.

Click the screenshot for the full PDF

As you go through the map, do you spot any security opportunities?  I thought I’d see more security offerings listed but I don’t.  If that doesn’t scream ‘gap in the market‘ I don’t know what does!

Troy is looking for feedback - as he says, its version 1.0.

Let me know what you think in the comments.

The Azure Cloud is currently in Technology Preview mode which means unscheduled developer holidays are pre-baked in.

I haven’t written about Azure before (tut tut), however I have been following along.

In case you somehow missed the announcement from Ray Ozzie, this is the Azure Stack.

And this is what it looked like to the Azure development community for the past 22 hours:

As seems to be the case for Cloud outages, updates are posted to a support forum.  Here is all anyone knows thus far (via Oakleaf Systems):

This was obviously a significant outage and it will be interesting to understand the root cause.  These massively distributed systems are truly an enormous engineering challenge - we’ll wait to see what Microsoft (and others) can learn from this particular incident.

I look forward to exploring the security aspects of Azure with you in the upcoming weeks.

Put simply, these are virtual machine instances that you can pay to have reserved for you to use anytime you want. The benefit should be assured availability and lower cost for machines with heavy uptime requirements.  You still only pay for what you use, in addition to a one time non-refundable payment to make the reservation.

When I first read about this I got quite excited.  I was already thinking of the potential use cases from a security perspective, particularly around business continuance/disaster recovery.

In fact, this is something fellow AWS customers have been asking for:

Also, quite a few customers actually told us something even more interesting: they were interested in using EC2 but needed to make sure that we would have a substantial number of instances available to them at any time in order for them to use EC2 in a DR (Disaster Recovery) scenario. In a scenario like this, you can’t simply hope that your facility has sufficient capacity to accommodate your spot needs; you need to secure a firm resource commitment ahead of time.

This is where it gets interesting.  What does ‘firm resource commitment ahead of time’ mean?

Always Read The Label

I surfed over to the AWS Customer Agreement to read the small print:

8.3. Reserved Instance Pricing. You may designate EC2 instances as subject to the reserved pricing and payment terms (“Reserved Instance Pricing”) set forth on the EC2 detail page on the AWS Website (each designated instance, a “Reserved Instance”). You may designate instances as Reserved Instances solely by calling to the Purchasing API (the “API Call”). In the API Call you must designate an availability zone, instance type and quantity for the applicable Reserved Instances. The Reserved Instances may only be used in the designated availability zone. We may change Reserved Instance Pricing at any time but price changes will not apply to previously designated Reserved Instances. We may terminate the Reserved Instance Pricing program at any time. Notwithstanding anything to the contrary herein, Reserved Instances are nontransferable and all amounts paid in connection with the Reserved Instances are nonrefundable, except that if we terminate this Agreement pursuant to Section 3.3 or terminate the Reserved Instance Pricing program we will refund you a pro rata portion of any up-front fee paid in connection with any previously designated Reserved Instances. In addition to being subject to Reserved Instance Pricing, Reserved Instances are subject to all data transfer and other fees applicable under this Agreement.

Hmm, suddenly I lost that warm, fuzzy feeling.

Obviously a provider has to protect themselves and have a way to drop features/services should there be insufficient demand.  From the above it seems there are two ‘outs’ for Amazon.  The first is contained in Section 3.3 and the other is if they simply terminate the program.  Lets look at Section 3.3

3.3.2. Paid Services (other than Amazon FPS and Amazon DevPay). We may suspend your right and license to use any or all Paid Services (and any associated Amazon Properties) other than Amazon FPS and Amazon DevPay, or terminate this Agreement in its entirety (and, accordingly, cease providing all Services to you), for any reason or for no reason, at our discretion at any time by providing you sixty (60) days’ advance notice in accordance with the notice provisions set forth in Section 15 below.

If you want to use AWS for Disaster Recovery, you better have a plan B as Amazon will only give you 60 days notice if they decide to drop Reserved Instances. And there was I, thinking DR was a plan B…  For some businesses, this may be enough of a commitment, but the more enterprise ’skin’ we put in the Cloud the more we need firmer commitments from providers.  I hope Amazon revises this decision and extends their commitment closer to that of traditional DR providers.

Potential Cost Savings

To give an idea of the cost savings, Geva Perry put together a useful online calculator where you can plug in the compute hours you need and understand the financial savings you’ll see. You can opt for the 1 year or 3 year reservation periods:

Geva goes on to higlight some common scenarios:

The default number I put in the hours column, 8760, is 24×365. So if, for example, you run a Large AMI for the entire year non-stop, you will save $1,153 by using reserved instances.

If you are curious to know the break-even point, it is 4,643 hours annually for the 1-Year fee and 2,381 hours *annually* for the 3-Year fee. In other words, if you expect to run an instance for more than 4,643 hours during the coming 12 months (works out to an average of 12 hours a day), you’re better off with Reserved, otherwise, stick to On-Demand.

This brings AWS pricing down to that of many traditional hosting providers, for those willing to commit upfront.

You might be thinking “what’s the benefit?” over just paying a traditional non-Cloud provider.  In the next post, I’ll cover some security use cases where lower costs and elasticity come into play.

Oh and if anyone from the AWS team thinks I’ve misinterpreted their Customer Agreement feel free to clarify in the comments and I’ll update the post.

However, the website hosted on Mosso’s Cloud, doesn’t actually receive, store, process, transmit any data that falls under the requirements of PCI.

Or to put it another way, its ‘compliance’ through not actually needing to be…

This didn’t deter them from putting a “PCI How To” document together which starts as follows (emphasis mine):

Building a PCI Compliant e-Commerce Solution Using Cloud Sites

Cloud Sites is designed to provide an elastic web hosting environment.  This capability can allow an e-commerce merchant to properly handle the high volume shopping season without carrying extra infrastructure throughout the remainder of the year.  Cloud Sites is not currently designed for the storage or archival of credit card information.  In order to build a PCI compliant e-commerce solution, Cloud Sites needs to be paired up with a payment gateway partner.

They then include the following helpful graphic which I modified to emphasis where the PCI data is NOT received, stored, processed or transmitted.  Everything to the left of the red line is the Mosso Cloud and everything to the right is the Payment Gateway provider.  The middle bit marked ‘API’ is that of the Payment Gateway as called by the merchant.

As they go on to state:

The communication from the Card Processing System to the Web Front End can never contain cardholder data.  Cardholder data includes: primary account number, expiration date, name as it appears on the card, CVV, CVV2 and magnetic stripe.

Yes Cloud Ladies and Gentlemen, this is an implementation of an age-old Internet architecture that involves redirecting customers wishing to pay for the contents of their online basket to an approved and compliant online payment gateway.

This approach follows the advice that RackSpace gives with regard to their dedicated hosting business (non-Cloud):

If you deal with credit cards and are required to meet the PCI DSS, my advice is to find a way to limit the scope of your compliance as much as possible. Rackspace recently concluded a two-year effort to receive our PCI Service Provider Report on Compliance (ROC) as a Compliant Level 1 Service Provider from Visa USA.

Just to be really clear, the PCI certification referred to above is of their dedicated hosting business - not their Cloud (aka Mosso business).  Different technologies and different architectures.

So, is there any PCI angle to this in reality?

The document talks to the PCI requirement as follows (emphasis mine):

By designing your e-commerce site in this manner, PCI compliance is reduced to a Type A SAQ (Self Assessment Questionnaire) for merchants processing less than 6,000,000 annual transactions.  The current version of the Type A SAQ can be obtained at: https://www.pcisecuritystandards.org/saq/instructions_dss.shtml. To achieve compliance when all cardholder information is handled by a partner, you only need to address two of the twelve sections of the complete PCI-DSS (Payment Card Industry – Data Security Standard) and only a subset of the controls in each of those sections.  The two sections are (9) Restrict physical access to cardholder data and (12) Maintain a policy that addresses information security.

The section 9 requirements are designed to protect any cardholder information stored at your office locations. If possible configure the relationship with your payment partner so that it is impossible for you or your employees to obtain complete cardholder information.  When logging into the partner portal you should see at most the last 4 digits of a card number.

The section 12 requirements are designed to ensure you’re working with PCI compliant partners to handle the cardholder information for you and that you have a process in place to ensure those partners remain compliant.  VISA publishes a list of compliant service providers on a monthly basis at: http://usa.visa.com/merchants/risk_management/cisp_service_providers.html

If you’ve followed along this far, you’ll realise that Mosso Cloud Sites is still ‘out of scope’ from PCI requirements as they pertain to the payment process itself, as that is handed off to a 3rd party gateway (the 3rd party must be PCI compliant though).  Section 9 is relevant to the office of the merchant - not the web front end hosting provider (Cloud or not) and section 12 is about your choice of payment gateway, again, nothing to do with Mosso.

Mosso is only relevant when it comes to the PCI requirement that the merchant perimeter is subject to vulnerability scans. In other words, because the merchant has outsourced hosting of an Internet accessible web front-end to Mosso, the merchant website must pass an initial, then four quarterly vulnerability scans to meet the PCI scanning requirement.  But Mosso isn’t responsible for running those scans.  Their contribution was to ‘partner’ with two Approved Scanning Vendors who do the work.

And that brings up two PCI scanning related issues regardless of whether you host on the Cloud or at a traditional hosting provider:

• vulnerability scans must take place after major network changes
• some vulnerability checks rely on banner grabbing to determine software version numbers and some providers (like Mosso) backport security fixes resulting in failed checks as version numbers are not incremented.  This is an age-old problem and a limitation of the scanning technology, not the provider.  The Approved Scanning Vendor will need to liaise with the provider/merchant to create manual exceptions.

So what role does Mosso really play when it comes to PCI compliance today?  They permit the Authorized Scanning Vendor to perform scans and confirm software fixes are in place when vulnerability checks generate false positives.

The Takeaway

The fact that Mosso is seeking ways to help their customers off-load as much PCI compliance requirements to other 3rd parties is fine - it makes business sense for them and their merchant customers.  It’s their positioning of the effort as a “landmark breakthrough” and that they are somehow pioneers which leads to generalisations rooted in misunderstandings that is the problem.

Next time you hear someone say ‘Cloud Provider X is PCI compliant’, ask the golden PCI question: is their Cloud receiving, processing, storing or transmitting Credit Card data (as defined by the PCI DSS)?  If they say ‘No’, you’ll know what that really means…marketecture.

]]> http://cloudsecurity.org/2009/03/14/what-does-pci-compliance-in-the-cloud-really-mean/feed/ http://cloudsecurity.org/2009/03/14/what-does-pci-compliance-in-the-cloud-really-mean/ http://feedproxy.google.com/~r/CloudSecurity/~3/Xp3PgrDyuBs/ http://cloudsecurity.org/2009/03/13/how-much-is-the-reputation-of-your-saas-provider-worth/#comments Fri, 13 Mar 2009 00:30:42 +0000 Craig Balding http://cloudsecurity.org/?p=146 We’re told that SaaS and Cloud providers live and die by their reputation.

If that’s the case, why do some of them give it up so easily?

Baynote just got shamefaced with the discovery of a basic Cross Site Scripting vulnerability in their ‘Social Search’ SaaS offering. Although - as seems to be the trend - you won’t find this out from reading their blog, press releases or other public areas of their website.  Instead, you learn of it from El Reg or from the blog of the security researcher that discovered the bug - Russ McRee:

Following the principles of one flaw to rule them all, a single validation error in the q variable found in http://[Insert customer here].com/socialsearch/query?cn=[customer]&cc=us&q= led to numerous Baynote customers falling prey to cross-site scripting. [VIDEO HERE]

I don’t know if Baynote contacted their clients to explain (a) the ramifications of the flaw and (b) that they were making code changes in the background…but either way, I have a question:

Given that a Cross Site Scripting flaw can be exploited to attack the *users* of a website, where does that leave the visitors of the SaaS clients’ website who would be potentially exposed to the flaw?

Along with the many benefits of SaaS services, you *and your customers* inherit the security bugs too.  From a business perspective we can begin the chorus of wailing and gnashing of teeth as we are reminded that a single vulnerability in a multi-tenant application, exposes all the tenants.  But what about the customers of the tenants?  Surely, the end-user is the real victim!

The positive side of this particular story is that Baynote moved quickly to fix the flaw.

The other angle on this incident is the disparity between the security claims  a SaaS/Cloud provider makes and the reality.

A quick Google site search of Baynote.com for the word ’security’ brings up this:

When a providers primary website property is vulnerable to a basic XSS attack, what do you make of statements like this?

Recently I asked ‘Where are the Cloud Security Evangelists?’. Now I’m asking ‘Where are the Cloud Security startups?’.

I’ve had briefings and Q&A sessions with a few but a recent email from a regular reader over at ‘Under the Radar’, prompted me to think “Yeah, where are they?”.

If you or someone you know are working at a pure-play Cloud Security or SaaS Securty startup, I’d love to hear from you.

First, I’m really curious what security challenge you are addressing.  Second, if you make it past this application form and can get yourself over to San Francisco in April, you get to pitch your offering to a group of CIOs at ‘Under the Radar’.

Ordinarily, the prospect of meeting a room full of CIOs just doesn’t get me all that excited, however, you’ll get to meet CISCO Cloud pin-up James Urquhart!

As its looking highly likely I will be declining my invitation (logistics, logistics), I’d like to make an offer to any security startup that gets accepted to pitch: I will give you up to 2 hours of my personal time (gratis) to grill you on your solution from an enterprise security perspective.  Let me find the security holes before the panel does :-).  If you think that might be valuable, reach out here.

And just to be clear, no-one is paying me for this - I’m just very curious and a firm believer in good karma.

What new Cloud Security startups are you aware of? Hit the comments…

The weakness appears to have been discovered by Richard De Vries.  He retells the story on Slashdot:

I work for a small Dutch company that uses Google Apps. This means that we can share documents with users within our domain (www.deondernemers.nl), as well as @gmail.com accounts or other Apps-domains. About three weeks ago, we discovered that some fifteen documents and spreadsheets were unintentionally shared with a lot of people, some of whom were outside of our domain. We found out that one of us had been wanting to share these documents with a colleague (within our domain). He selected the documents on the documents list and added one user. Google Docs then shared all these documents with everyone who had access to one of the selected documents.

The official Google notification to affected users describes the issue as follows:

We wanted to let you know about a recent issue with your Google Docs account. We’ve identified and fixed a bug which may have caused you to share some of your documents without your knowledge. This inadvertent sharing was limited to people with whom you, or a collaborator with sharing rights, had previously shared a document. The issue only occurred if you, or a collaborator with sharing rights, selected multiple documents and presentations from the documents list and changed the sharing permissions. This issue affected documents and presentations, but not spreadsheets.

To help remedy this issue, we have used an automated process to remove collaborators and viewers from the documents that we identified as being affected. Since the impacted documents are now accessible only to you, you will need to re-share the documents manually. For your reference, we’ve listed below the documents identified as being affected…

I’ll leave others to comment on the obvious privacy issues, but what I find really interesting is how Google handled it. Based on publicly available timeline information, we can attempt to glean insights into how the investigation progressed.

Timeline

February 22nd: Initial report from Richard De Vries.  Richard noted difficulty in knowing where he could report the issue (an issue in itself) so ended up reporting it in a general catch-all bug reporting form.

February 25th: Google Security contact Richard to verify the vulnerability report.  They requested he re-create the sharing scenario.  The 3 day delay between Richared reporting the bug and the response from Google Security will not surprise anyone thats works in a large company.  If security concerns don’t follow your established communication channels (intuitive to customers or not), then a game of pass the parcel ensues until it reaches the internal security team.

March 3rd: Google Security confirms the weakness

March 7th: Google notifies affected customers

Tasks

So in the past 10 days, these are the likely tasks that happened at Google (assume some tasks carried out in parallel and some repeat tasks):

• develop a strategy to handle the weakness and convene a virtual team with representatives from development, Google Security, operations, support, privacy, product managment and PR
• establish a meeting schedule and agree how to collaborate (Google Docs anyone? ;-)
• fully understand the problem and establish the implications of the vulnerability. This is likely to involve a mix of brainstorming, source code review and ad hoc data-mining of the Google Docs database to validate hypothesis
• perform an in-depth security source code review of the affected code and neighbouring code to check for similar logic flaws.  This larger review is a precautionary measure to identify possible weaknesses that could become the subject of attack when word of the original vulnerability gets out
• create necessary fixes and test on a standby system
• go through internal change control processes to deploy and confirm necessary fixes to some portion, or all of the production systems.  When test results match expectations, deploy globally.  Its unknown how fast Google can ripple a functionality change across their numerous data centers (anyone who knows, please share in the comments)
• identify the users affected by the flaw.  They now have to develop, test and run a query against the Google Docs data repository (BigTable?) to identify which documents are affected and in turn, which users own those documents
• having reviewed the results they now run an update to ‘reset permissions’ on affected documents (they could do the above step and this step in one script but splitting into a 2 step approach carries less risk even if it takes longer) so that only the owner has access
• the Google Docs product manager  most likely consulted with the Google privacy and legal teams to understand any potential liability
• create an email template and list for notifying users and get the text of the notification email approved by the product manager, privacy, PR and possibly legal
• the PR team prepare standard responses ready to respond to comments from the media
• prime Google Support ready to handle queries from concerned users
• issue the notification email to affected users.  Note, at time of posting, Google had not posted this issue to either their security blog or the Google Docs blog.  Given their strategy to identify, notify and protect affected users (the “sniper” IR strategy ;-), its arguable they don’t actually need to although I suspect they might
• create Google alerts on phrases in the notification email to see on which web properties it is reported to perform reputation monitoring ;-)
• perform a Root Cause Analysis (RCA) on the coding snafu and implement changes to reduce/eliminate future occurences

Even if they skipped some of the non-essential steps, that’s a *lot* to achieve in the timeframe especially as we don’t know what other issues Google Security may be fielding at the same time (its rare that only person reports an a potential security weakness in widely used software/services, let alone additional security reports that range from completely bogus to potentially valid).

In a nutshell, I’m impressed with the speed Google executed on this.  To put their response in perspective, compare how long it takes some credit card payment processors to react to major data breaches.

Hats off to Google Security on this one…

Information Security magazine just ran a decent piece on Cloud Computing Security called ‘How To Secure Cloud Computing’.

I was asked for my opinion on the security challenges facing enterprises today and what they can start doing about it.

One of the concerns I expressed is the lack of security evangelism around Cloud Computing Security by Cloud Providers.  Attend a Cloud Conference and you’ll see what I mean.  The Cloud Provider Evangelists do a great job turning up at conference after conference explaining the benefits and use cases of their respective Clouds.  But when it comes to a meaningful discussion about security, who can you talk to?  The stock answer is ‘we’ll hook you up with our security team’ or ‘we have a whitepaper about this’ or even ‘we use SSL so its OK’.  Um, what?

Cloud Evangelists quickly get out of their depth when it comes to security. Now, is that bad?  No, I don’t think so.  If I ran a Cloud company, I’d want my evangelists drumming up business, helping build my brand by being highly visible and keeping an eye on my competitors.  But if the sales process is about one thing more than anything else, its about removing barriers to “yes”.  And if survey after survey is telling you that the biggest barrier to Cloud adoption is security, why can’t you find a Cloud Security Evangelist when you need one?

Have the Cloud Security geeks been told they can’t go outside and play with the other geeks?

To me, as someone actively seeking information on this subject, I’m stunned by the lack of attention Cloud companies are paying to their marketing efforts around security.  I’m no Seth Godin or Guy Kawasaki, so if the lack of a security marketing strategy is so blindingly obvious to me, rest assured dear reader, it must be pretty bad.

And just to be really clear: I’m not suggesting that Cloud providers don’t have smart security people.  Its just that their masters don’t seem to realise that invisible security may be the holy grail of usability, but it isn’t when it comes to moving the security conversation forward.

The irony here is that it seems to be the smaller players that actually have people on hand that can directly speak to security.

Can anyone explain this conundrum?  And more importantly, have you ever met a Cloud Security Evangelist?  Perhaps I should start a ’sightings’ page ;-).

The National Institute of Standards and Technology has created a new team to determine the best way to provide security for agencies that want to adopt the emerging technology called cloud computing, said Ron Ross, a senior computer scientist and information security researcher at NIST.

“The team will give our customers a sense of what kinds of risks they may be taking on by moving into that new territory,” Ross said today at the SaaS/Gov 2009 conference produced by the Software and Information Industry Association and market research firm Input.

Googling reveals an earlier presentation titled ‘Perspectives on Cloud Computing and Standards‘ by Peter Mell and Tim Grance from the Information Technology Laboratory at NIST, wherein they announce plans to develop a NIST Special Publication on Cloud Computing Security to cover the following:

I recommend reading the rest of the presentation, particularly around the NIST definition of Cloud (”what you can’t define, you can’t standardize”) and some of the implementation options under consideration.  You can catch the authors giving this presentation in person at the Cloud Computing Interoperability Workshop on March 32rd 23rd in Virginia, US.

It will be interesting to see what this group proposes/mandates.

On a sidenote, this post is the perfect opportunity to draw your attention to fellow Cloud blogger Kevin Jackson.  Kevin focuses on Public Sector Cloud news and developments making his blog a must-read.

One of the security challenges of Cloud Computing - and specifically Infrastructure as a Service (IaaS) is securely connecting your enterprise network to one or more Cloud providers without deploying VPN hardware.  There is also the availability concern - how quickly can you fail over to another Cloud provider when your primary Cloud evaporates.

One company that I voted ‘Hot’ during my somewhat tongue-in-cheek ‘Hot or Not’ quiz at the World Cloud Computing Summit is trying to do something about it.  CohesiveFT released the first version of VPN-Cubed last year and are now seeking beta testers for their next major release.

What is VPN-Cubed?

VPN-Cubed™ is the first commercial solution that enables customer control in a cloud, across multiple clouds, and between private infrastructure and the clouds.

VPN-Cubed provides an overlay network that allows YOU control of addressing, topology, protocols, and encrypted communications for YOUR devices deployed to virtual infrastructure or cloud computing centers.  When using public clouds your corporate assets are going into 3rd party controlled infrastructure.  This could be public clouds like Amazon EC2.  It could be “gated community” clouds from Telcos like BT, ATT and more.  In both cases you are deploying to 3rd party control, yet Enterprise checks and balances require you to exhibit control over your computing infrastructure.  VPN-Cubed gives you flexibility with control in 3rd party environments.

Despite our product having the word “vpn”  (virtual private network) in its name, VPN-Cubed is more than a simple VPN, it is an overlay network that is configured as easily as a traditional VPN.  X-cloud control, administrative simplicity.  To quote one of the leading security bloggers “this is not your father’s VPN”.

Now you can confidently leverage the cloud for redundancy, failover and scalability during critical transitions; whether scaling up to grow the business or scaling down to cut costs.

If you are seeking to leverage IaaS and want to experiment with the only off-the-shelf Cloud network security overlay, you may want to join the beta test program that’s just opened up by emailing vpncubed_beta@cohesiveft.com.

Update: An overview of the beta program with slides showing screenshots of the steps is now available.

P.S The blogger behind the “this is not your father’s VPN” quote is my friend and fellow Information Security Geek Christofer Hoff over at Rational Survivability.  I reference Chris in my resources page - if you are not already subscribed to his blog, I highly recommend you do.

Shortly before the holiday break, I presented my take on Cloud Computing and Security at the IGT2008 World Cloud Computing Summit in Tel Aviv, Israel.

This was a great conference for me personally as it was an opportunity to meet face to face with some very smart people that are passionate about the Cloud.  It also provided an even greater insight into the steamroller that is the Cloud - company after company lining up to either “Clouderize” their current offerings or in most cases, “doing something new”.  I met a few startups looking to solve some tricky problem including a stealth mode security outfit looking to provide enhanced security for SaaS (I can’t say more right now but watch this space).

The main thrust of my talk was that there needs to be a deeper conversation about the security implications of Cloud Computing and Cloud Services in general.  That’s not because I think there is anything innately insecure about Cloud offerings, more that we are venturing into the great unknown with layers of offerings, greater trust transitivity and new (and old) technologies meshed together in ways we frankly don’t understand.  We need to progress the dialogue beyond crying out that the ‘Cloud is insecure’ or just saying ‘the biggest Cloud issue is security’ and get into the nitty gritty details.  But my argument is we can only do that if the providers engage in that conversation.  It’s one of the reasons I encourage Cloud providers to reach out and talk security - most large enterprises have responsibilities that mean they cannot treat the Cloud as a black box.

The 25 minute talk is split into 2 parts:

• after a brief intro - I believe I was the only one there not representing a company - I laid out what I mean by ’security’.  As this wasn’t an information security conference and there was a wide range of people present, I wanted to lay out what I mean by “information security” to provide context for what was to follow.  If you’ve been “doing” enterprise security for years, you can safely skip the first 10 minutes (unless you want to critique me!).
• the second half focused on the need for a new risk model that better represents the ebb and flow of risk in Cloud environments - especially with Cloud Stacks (if someone has a better term, let me know) followed by the Enterprise Cloud Security version of “Hot or Not” - complete with audience voting.  Given that some of the providers I’d included in the game were sitting in the audience, this sparked some decent conversations later that evening.  If you are a Cloud provider featured in the presentation and you didn’t catch my talk, feel free to contact me to discuss your “hotness” ;-).

The videos are now online (IE only), along with the slides.  My talk was on Day 2 in the afternoon (halfway down the right hand side).  I welcome your feedback - feel free to leave comments or ask questions.

You also want to check out the Security Panel on Day 1 hosted by Sam Bercovici.  Professor Barton P. Miller and Alexis Richardson from CohesiveFT and myself.

Recent news stories highlight new features and capabilities, including a SQL-like SELECT API for Amazon SimpleDB and the (significant news) that Cloud Compute is now available in the EU.

More worryingly, however and missing from the Amazon AWS homepage, is that Amazon just rolled out an important security fix to correct a basic, but significant, cryptographic weakness in their request signing code.  The flaw affects their database API (SimpleDB) and compute API (EC2) services.

Colin Percival discovered the weakness and reported it to Amazon back in May (!).  Thats 7.5 months ago.  Here’s the “executive summary” from Colins’ write-up:

AWS signature version 1 is insecure

The important bit first: If you are making Query (aka REST) requests to Amazon SimpleDB, to Amazon Elastic Compute Cloud (EC2), or to Amazon Simple Queue Service (SQS) over HTTP, and there is any way for an attacker to provide you with data which you use to construct your request, switch to HTTPS or start using AWS signature version 2 now. For example, if you allow users to add arbitrary “tags” to documents, and you use SimpleDB to store those tags, this means you. (Amazon Flexible Payments Service (FPS) and Amazon Devpay also use the same insecure signature method, but they already require the use of HTTPS. Amazon S3 and other services use different signature methods.)

So in a nutshell, if you only ever accessed SimpleDB or EC2 services over HTTPS, you’re not impacted.

But if you did use the non-encrypted transport, you were vulnerable to a man-in-the-middle attack (MITM) that could expose your data hosted in SimpleDB and your EC2 compute instances.

Overall, Colin is generous in his praise of the response from the Amazon team - although did express concern at the time it took for the vulnerably to be closed:

I reported this issue to Amazon via an email to Jeff Barr, the “Lead Web Services Evangelist” at Amazon on May 1st, and while it took a long time — 7.5 months — for it to be fixed, I’m happy to say that Amazon took this issue seriously at all times, and the lengthy timeline was simply because of the large amount of work involved. Jeff forwarded my email to someone working on SimpleDB (I’ve been asked not to mention names), who confirmed that they agreed that this was a problem. As part of their review of my findings, Amazon’s security people realized that this also affected EC2 and SQS — in my initial investigation I had only looked at SimpleDB — and at the beginning of July they agreed to send me their planned signature version 2 so that I could review it.

Based on the timeline provided by Colin, it took about 2 months for Amazon to assess the extent of the exposure across their other service APIs and come up with a proposed fix.  That’s a considerable length of time given the simplicity of the weakness.  They then took over 5 months to actually implement the fix.  I don’t care how many client libraries you offer Mr Cloud Provider, thats still a very, very long time to leave your customers exposed while you get a fix into production.

Back to Colin as he describes the vulnerable signing process:

AWS signature version 1 signs an HTTP query string as follows:

1. Split the query string based on ‘&’ and ‘=’ characters into a series of key-value pairs.
2. Sort the pairs based on the keys.
3. Append the keys and values together, in order, to construct one big string (key1 + value1 + key2 + value2 + … ).
4. Sign that string using HMAC-SHA1 and your secret access key.

Did you spot the problem?  Colin did and explains as follows:

When Amazon invented this signature scheme, they forgot about one of the foremost design principles relating to cryptographic signatures: Collisions are BAD! In a well-designed signature system, it should be computationally infeasible to construct two different messages which have the same signature; this prevents substitution attacks where an attacker convinces the key holder to sign a “harmless” message, and then attaches that signature to a different message. Looking at how AWS signature version 1 is computed, it’s easy to see how to construct collisions: Because there are no delimiters between the keys and values, the signature for “foo=bar” is identical to the signature for “foob=ar”; moreover, the signature for “foo=bar&fooble=baz” is the same as the signature for “foo=barfooblebaz”.

Oops.  He then goes on to explain how a man in the middle attack could exploit the weakness.

Taking a step back, what are the concerns with this weakness and the way Amazon handles security vulnerability reports in general?

Vulnerability Reporting

• Amazon doesn’t provide a visible mechanism for researchers/customers to report security vulnerabilities.  You can report ‘abuse’ (i.e. your EC2 image is under attack) via a contact form on the AWS website but unless you know someone at Amazon, you have to guess how to report a security vulnerability.
• Amazon has no published policy on how they handle vulnerability reports.  Nothing to explain their commitment to following up, promptly responding to reports, overall process etc.  Now, it seems they do follow up and overall Colin says good things but that timeline sends shivers up my spine.
• Amazon doesn’t have a dedicated security page, mailing list, RSS feed or pigeon carrier for alerting customers of security issues.  This makes it a PITA for enlightended customers to even know they are at  risk (unless they setup  Google News Alerts) and comes across to me at least as a trifle arrogant.

Vulnerability Notification

• Amazon hasn’t notified their customers yet that their SimpleDB and EC2 instances were vulnerable for the past 7.5 months (I’m a customer and I’ve yet to receive an email).  Obviously the vulnerability itself goes back further than 7.5 months.
• Amazon have not published how customers can determine if the issue was exploited (back to the visibility issue again).   Clearly, Amazon doesn’t have full visibliity of the network between the customer and their data centers, however what analysis has Amazon performed on the logs *only they have access to* that could highlight an issue? (frankly, due to the nature of this issue and the lack of event logs/audit trails I suspect the simple answer is there isn’t much they can do here).

Secure Software Development

• Amazons’ internal code review process failed to detect a basic cryptography flaw.  Note, although the error is pretty basic, errors do happen.  The real problem here is the failure of peer review processes to identify the problem prior to going into production.  Cryptography routines along with other sensitive security routines should be subject to greater security analysis as clearly the stakes are higher.  If they did in fact have this code reviewed for security weaknesses, this is a huge oversight.  So this is now fixed, what else is vulnerable?  It would be great to learn that the lengthy delay in delivering the fix was due to Amazon have all their source code reviewed for security issues…
• Amazon have made no public commitment regarding how security is factored into their software development lifecycle.  Until now they could argue they didn’t need to - they hadn’t messed up (or at least, we don’t think they did).

The geek in me loves Amazon AWS and the promise that it holds, but the Security Professional rooted in the “here and now” is seeing too many red flags (the issues are wider than the handling of this one bug).

If anyone from Amazon reads this and wants to talk security roadmap and go “beyond the whitepaper”, you know where I am.

Internet password managers need to know two things to be secure: Which website is requesting a password? And, to which website is a password being delivered?

Check out the scorecard below:

Test definitions are available here.

Clearly, there is still some way to go.

A few observations:

• The timing of the announcement is interesting.  According to CIS, they notified Google about the issues prior to the release of Chrome.  On December 12th, Google removed the beta label and Chrome is considered production code.  It will be interesting to see if Google comment on this as on the surface this does leave some awkward questions.  Rather than pre-judge now, lets see if/how they respond.
• This issue is compounded with Cloud Computing as the Cloud Management interfaces exposed by the Cloud providers cannot - in most cases - be restricted to specific IP ranges (not a total solution but a significant mitigant).  For example, if an attacker steals your Amazon AWS password, she can directly access the AWS management pages, claim your AccessID and Secret Key and perform a Cloud takeover from any source IP address of her choosing.
• The lack of Cloud visibility means you have no administrative audit trail to (a) pro-actively monitor for misuse of your credentials and (b) perform rapid response.  On that last point, you would need to reach out to your provider and ask them to tell you which source IP accessed your Cloud account (in a future post, I’ll talk about the complete lack of *formal* Incident Response provisions in Cloud SLAs).
• Browser bugs are hardly rare - what makes this batch insidious is the ease with which the weaknesses can be exploited - no special exploit code required.  Expect to see this abused.
• This is a bad week week for browser bugs.  An emergency hotfix from Microsoft to fix an actively exploited heap overflow is released today, along with fixes from Opera rated as highy critical.
• The market for browser security products will continue to grow (duh!).

The most practical recommendation I can give is to recommend all Cloud related administration is carried out in a separate Virtual Machine.  This is good advice for internal IT administrators today - regardless of whether they use Cloud services or not.  The main thrust is to ensure that your email client and “normal” web browser are isolated from your administration infrastructure.  The need for separation of administration environments becomes increasingly vital as organisations adopt Cloud Services.

If you are curious, you can test your browser.

Last month I was interviewed for a podcast with SearchSoftwareQuality.com.

We talked about some of the advantages Cloud Computing could bring to software development and testing.  Notice I say ‘could’ - I continue to see great potential benefits but some of these require us to rethink how we do things as ‘end-users’ and depend on the Cloud Computing ecosystem maturing enough to deliver them (e.g. security monitoring of Cloud API calls).

This was recorded prior to the Microsoft Azure announcement hence the “software + services” model wasn’t covered.

Anyway, the podcast is broken into 3 x 8 minute segments (I think I broke the spoken word count ;-):

• General benefits of cloud computing for software development
• Cloud computing’s impact on agile development practices, software testing, and e-commerce
• Security elements surrounding cloud computing, such as software monitoring, implementing security patches, and the reduction of data leakage.

You can access the podcast segments here.

My thanks to Michelle and Keith over at TechTarget for the opportunity.

What About You?

Apart from general feedback on whether the podcast was helpful or not, I’m interested to hear if you’ve started any Cloud based development projects - please share in the comments.

As part of our ongoing research into Cloud Computing, IDC recently conducted a survey of 244 IT executives/CIOs and their line-of-business (LOB) colleagues about their companies’ use of, and views about, IT Cloud Services. Successful suppliers will need to address both the biggest challenges of cloud services, and the biggest traditional IT user issues.In part 1, we looked at current and future adoption of IT cloud services. In part 2, we looked at users’ views about the key benefits and challenges of IT cloud services.

What is your Cloud Provider doing to address your security concerns?

Richard Stallman - the man that gave us GNU - doesn’t trust Cloud providers with his data and says you shouldn’t either.  Richard believes we should store our private data on our own computers using ‘free’ (as in freedom) software.  The ironic part for Richard is that a significant portion of the Cloud is powered by open source software which he indirectly created (think gcc).

Richard sees it as a question of control.  Control is important but it isn’t the only variable.  Rather, I see it as a question of control, competence and economics.

The quick rebuttal to Richards’ view is this: the average computer user is not as smart as you.  Control is not the same as competence.  Control is about exercising choice, not about requiring everyone in the world to develop sufficient skills to protect complex hardware and software systems (aka their computer) against ever increasing threats.

My view is that privacy is not ‘free’.  It comes at a cost.  Whether you run your own systems or rely on someone else to do it, there is a cost.  There is cost in designing and implementing mechanisms to support privacy.  Beyond upfront costs there are ongoing expenditures to ensure privacy is maintained e.g. maintaining access control lists, testing and applying security patches, data leakage prevention etc.  None of these things are ‘free’.

If we agree that privacy costs money then how much is your privacy worth?

Stop for a second - think of a number…  

Now did we all think of the same number?

The problem with a one size fits all approach to privacy is that we each place a different value on it.

Checking in on the EPIC site, I saw this:  

A new report from Pew Internet and American Life Project indicates that “cloud computing” applications, such as web-based email and other web apps, are raising new privacy concerns. The report Use of Cloud Computing: Applications and Services found that 69% of online Americans use webmail services, store data online, or use software programs such as word processing applications whose functionality is located on the web. At the same time, “users report high levels of concern when presented with scenarios in which companies may put their data to uses of which they may not be aware.” For example, 90% of respondents said that they “would be very concerned if the company at which their data were stored sold it to another party,” 80% say “they would be very concerned if companies used their photos or other data in marketing campaigns,” and 68% of “users of at least one of the six cloud applications say they would be very concerned if companies who provided these services analyzed their information and then displayed ads to them based on their actions.”

What does that tell us?

The average (American) Internet user finds Cloud services convenient but has concerns about how their privacy might be affected by Cloud providers actions (duh!).  The survey identifies a lack of awareness in how private data is used in some consumer based Cloud services (consistent with web advertising awareness surveys).  

Unfortunately, the results of this survey are not very actionable.  The survey doesn’t mention whether these are all ‘free’ Cloud services (we can only assume they are) or ask the respondents what their expectations of privacy are and how much they would be willing to pay for different privacy assurance levels. 

On a sidenote, respondents were not asked if they had actually read the privacy agreement for the services they signed up to.  But the providers know if they did or not…  Or at least, they have the data to figure it out.  At sign up time they can measure the time between displaying the privacy agreement and the user clicking ‘I accept’.  If its just a few seconds then its pretty obvious there was more scrolling than reading going on.  But I think we can probably guess the answer without the data ;-).

I believe we need to be able to link expectation of privacy with cost.

• How much are you willing to pay for privacy?  What level of privacy assurance do you need?
• How much is your Cloud Provider paying to protect your privacy today?  What privacy services could they reasonably offer if they had customers willing to pay?  How might this compare with how you manage your private data on your home computer today?

The cynical view is that we expect privacy but don’t want to pay for it.  Its a bit like uptime - there is a parallel universe out there, where internal IT departments allegedly meet their 99.999% uptime SLAs, but when Gmail goes down, the Sergey Brin witchcraft dolls come out.

From a provider perspective, the “cost” of privacy invariably gets bundled under that line item called ‘Information Security’.  And don’t be fooled, the cost of privacy in reality is more than the salary of the person employed to be the privacy advocate (if there is one).  If we can’t see how much our providers are spending on our privacy then how can we judge if they are spending enough?  And what is enough?  And what can I get if I’m willing to pay a little extra?

Personally, I would rather we get some transparency around privacy costs and assessment of offerings.  However, without a sufficiently sized market of customers willing to pay for privacy assurance and Cloud Providers willing to be more open, I won’t hold my breath.

What about you?  Would you be prepared to pay for privacy?  Should providers be more transparent about what they do and don’t do and how they do it?
 
 

I’ve been sitting on this for a while and I’m glad I can now finally say it…

I’m delighted to announce that I have been invited to present at the World Summit of Cloud Computing, to be held in Israel on 1-2 December 2008.

The event is organised by Avner Algom from the IGT (Israeli Association of Grid Technologies). Putting my invitation to one side, I have to say its a stunning lineup of speakers. Its a who’s who of Cloud players. Avner has clearly done his homework!

Obviously I’ll be talking about the security aspects of Cloud Computing, delving into some of the areas I’ve written about here and some new material that I’m currently working on.

If you work for a company that is consdering future plans and Cloud Computing, you might want to take a look over the agenda. Compared to some other conferences, the ticket prices seem very reasonable to me.

Registration is now open.

If you have any questions, feel free to leave a comment below. I’ll do my best to get them answered. Also, if you know anyone that might benefit from 2 days in a beautiful part of Israel getting up to speed on Cloud Computing, feel free to send them this link.

With all this talk and reporting about security concerns, lets change the channel for a moment and assess the potential security benefits of Cloud Computing.

In my view, there are some strong technical security arguments in favour of Cloud Computing - assuming we can find ways to manage the risks.

With this new paradigm come challenges and opportunities. The challenges are getting plenty of attention - I’m regularly afforded the opportunity to comment on them, plus obviously I cover them on this blog. However, lets not lose sight of the potential upside.

In this post, I walk through seven technical security benefits. Some are immediate, others may arise over time and have conditions attached (some unstated for the sake of brevity). However, I’m including the longer-range benefits now to raise awareness. Some of the outcomes listed are available today without the Cloud, but they are either complex and slow to implement (and thus less likely to happen) or prohibitive for capital cost reasons. I don’t claim this is a definitive list - it reflects where my thinking is today.

Some benefits depend on the Cloud service used and therefore do not apply across the board. For example; I see no solid forensic benefits with SaaS. Also, for space reasons, I’m purposely not including the ‘flip side’ to these benefits, however if you read this blog regularly you should recognise some.

On a sidenote, I believe the Cloud offers Small and Medium Businesses major potential security benefits. Frequently SMBs struggle with limited or non-existent in-house INFOSEC resources and budgets. The caveat is that the Cloud market is still very new - security offerings are somewhat foggy - making selection tricky. Clearly, not all Cloud providers will offer the same security.

Seven Technical Security Benefits of the Cloud

1. Centralised Data

• Reduced Data Leakage: this is the benefit I hear most from Cloud providers - and in my view they are right. How many laptops do we need to lose before we get this? How many backup tapes? The data “landmines” of today could be greatly reduced by the Cloud as thin client technology becomes prevalent. Small, temporary caches on handheld devices or Netbook computers pose less risk than transporting data buckets in the form of laptops. Ask the CISO of any large company if all laptops have company ‘mandated’ controls consistently applied; e.g. full disk encryption. You’ll see the answer by looking at the whites of their eyes. Despite best efforts around asset management and endpoint security we continue to see embarrassing and disturbing misses. And what about SMBs? How many use encryption for sensitive data, or even have a data classification policy in place?
• Monitoring benefits: central storage is easier to control and monitor. The flipside is the nightmare scenario of comprehensive data theft. However, I would rather spend my time as a security professional figuring out smart ways to protect and monitor access to data stored in one place (with the benefit of situational advantage) than trying to figure out all the places where the company data resides across a myriad of thick clients! You can get the benefits of Thin Clients today but Cloud Storage provides a way to centralise the data faster and potentially cheaper. The logistical challenge today is getting Terabytes of data to the Cloud in the first place.

2. Incident Response / Forensics

• Forensic readiness: with Infrastructure as a Service (IaaS) providers, I can build a dedicated forensic server in the same Cloud as my company and place it offline, ready for use when needed. I would only need pay for storage until an incident happens and I need to bring it online. I don’t need to call someone to bring it online or install some kind of remote boot software - I just click a button in the Cloud Providers web interface. If I have multiple incident responders, I can give them a copy of the VM so we can distribute the forensic workload based on the job at hand or as new sources of evidence arise and need analysis. To fully realise this benefit, commercial forensic software vendors would need to move away from archaic, physical dongle based licensing schemes to a network licensing model.
• Decrease evidence acquisition time: if a server in the Cloud gets compromised (i.e. broken into), I can now clone that server at the click of a mouse and make the cloned disks instantly available to my Cloud Forensics server. I didn’t need to “find” storage or have it “ready, waiting and unused” - its just there.
• Eliminate or reduce service downtime: Note that in the above scenario I didn’t have to go tell the COO that the system needs to be taken offline for hours whilst I dig around in the RAID Array hoping that my physical acqusition toolkit is compatible (and that the version of RAID firmware isn’t supported by my forensic software). Abstracting the hardware removes a barrier to even doing forensics in some situations.
• Decrease evidence transfer time: In the same Cloud, bit fot bit copies are super fast - made faster by that replicated, distributed filesystem my Cloud provider engineered for me. From a network traffic perspective, it may even be free to make the copy in the same Cloud. Without the Cloud, I would have to a lot of time consuming and expensive provisioning of physical devices. I only pay for the storage as long as I need the evidence.
• Eliminate forensic image verification time: Some Cloud Storage implementations expose a cryptographic checksum or hash. For example, Amazon S3 generates an MD5 hash automagically when you store an object. In theory you no longer need to generate time-consuming MD5 checksums using external tools - its already there.
• Decrease time to access protected documents: Immense CPU power opens some doors. Did the suspect password protect a document that is relevant to the investigation? You can now test a wider range of candidate passwords in less time to speed investigations.

3. Password assurance testing (aka cracking)

• Decrease password cracking time: if your organisation regularly tests password strength by running password crackers you can use Cloud Compute to decrease crack time and you only pay for what you use. Ironically, your cracking costs go up as people choose better passwords ;-).
• Keep cracking activities to dedicated machines: if today you use a distributed password cracker to spread the load across non-production machines, you can now put those agents in dedicated Compute instances - and thus stop mixing sensitive credentials with other workloads.

4. Logging

• “Unlimited”, pay per drink storage: logging is often an afterthought, consequently insufficient disk space is allocated and logging is either non-existant or minimal. Cloud Storage changes all this - no more ‘guessing’ how much storage you need for standard logs.
• Improve log indexing and search: with your logs in the Cloud you can leverage Cloud Compute to index those logs in real-time and get the benefit of instant search results. What is different here? The Compute instances can be plumbed in and scale as needed based on the logging load - meaning a true real-time view.
• Getting compliant with Extended logging: most modern operating systems offer extended logging in the form of a C2 audit trail. This is rarely enabled for fear of performance degradation and log size. Now you can ‘opt-in’ easily - if you are willing to pay for the enhanced logging, you can do so. Granular logging makes compliance and investigations easier.

5. Improve the state of security software (performance)

• Drive vendors to create more efficient security software: Billable CPU cycles get noticed. More attention will be paid to inefficient processes; e.g. poorly tuned security agents. Process accounting will make a comeback as customers target ‘expensive’ processes. Security vendors that understand how to squeeze the most performance from their software will win.

6. Secure builds

• Pre-hardened, change control builds: this is primarily a benefit of virtualization based Cloud Computing. Now you get a chance to start ’secure’ (by your own definition) - you create your Gold Image VM and clone away. There are ways to do this today with bare-metal OS installs but frequently these require additional 3rd party tools, are time consuming to clone or add yet another agent to each endpoint.
• Reduce exposure through patching offline: Gold images can be kept up securely kept up to date. Offline VMs can be conveniently patched “off” the network.
• Easier to test impact of security changes: this is a big one. Spin up a copy of your production environment, implement a security change and test the impact at low cost, with minimal startup time. This is a big deal and removes a major barrier to ‘doing’ security in production environments.

7. Security Testing

• Reduce cost of testing security: a SaaS provider only passes on a portion of their security testing costs. By sharing the same application as a service, you don’t foot the expensive security code review and/or penetration test. Even with Platform as a Service (PaaS) where your developers get to write code, there are potential cost economies of scale (particularly around use of code scanning tools that sweep source code for security weaknesses).

Your Thoughts?

What benefits do you see that I haven’t included in the above list? Where do you agree/disagree and importantly, why?

Collaboration in the Cloud

Forward thinking companies use collaboration technologies to melt away the physical distance between disparate offices, remote workers and suppliers.  Investments in R&D projects to create the next generation of business collaboration technologies and starting to bear early fruits and are worth paying attention to - especially if you get paid to “do security”.  One major focus area is Virtual Worlds.

Teleporting Virgins

The big news in the Second Life research community is that avatars (”virtual people”) have successfully teleported between distinct virtual worlds.  The virgin teleporters went from a Second Life Preview Grid - an experimental grid completely disconnected from the Main Grid - to a virtual world running IBM OpenSIM.

At this stage there is intentionally no asset transfer going on at all - in other words, you can’t take your “stuff” from one world to another - but that will come in time as the Open Grid Protocol is extended.  Today just login and teleport are supported.  No stealing those trade secret “assets” yet ;-).

Linden Labs speaks to this issue:

Q: How will Linden Lab prevent property from being copied into other virtual worlds?
We’re paying extremely close attention to that question. We will be designing this with the Second Life community to ensure their needs are met. We want to stress that when it does become possible to move avatars between worlds, we will take the utmost care to protect the rights of Second Life property owners and creators. Linden Lab will not design a system that lets people openly violate the permissions of SL goods and take them to other worlds. We recognize that intellectual property is the engine that drives Second Life, and we are completely committed to preserving the qualities that make Second Life the unique, innovative and dynamic place that it is today.

With my “hacker-vision” ™ enabled I see *all kinds* of opportunities for mischief here.  I’m betting we’ll see imaginative attacks as the usual cat and mouse game of vulnerability research and vendor response plays out.  “Sorry boss, someone hijacked my avatar and now I’m stuck on this desert island for who knows how long!”.

Threat Profiling Second Life

Getting back to reality, people are already exploring Virtual World security.  Michael Thumann of ERNW in Germany is a pen-tester and security researcher and in this 10 minute video, Michael shares the result of his security research on Second Life.

He covers:

• In-game cheating
• Identity theft
• Attacking 3rd party servers using Linden Scripting Language (think about the liability issues and the providers ability to track abusers)

For those interested in more detail, the full presentation he gave at BlackHat Europe 2008 in Amsterdam is here (pdf).

Of particular note, Michael applied a formal threat model approach to the research - STRIDE from Microsoft.

In a future post I’ll talk more about threat profiling in the context of Cloud Computing vulnerability research and specific API security vulnerability classes we can expect to see exploited.

“Amazon EC2 public AMIs (Amazon Machine Image) generate unique SSH (Secure Shell) host keys each time you launch an instance. This enables you to get the host SSH keys from the console output and verify the host to which you are connecting.”

Important note: SSH host keys enable clients to verify the server identity (”are you really my server?”) and are separate from SSH user keys that allow the user to prove their identity to the server (”he really is Jeff”).

What does this mean?

It means that EC2 instances created from a public AMI after June 23rd have unique SSH host keys and thus are not vulnerable to a man in the middle attack against the SSH protocol, but only *if* you manually verify the host SSH key during your initial SSH connection.

OK, but I created my AMI before June 23rd - am I vulnerable?

According to Amazon, yes.  Every EC2 instance copied from a public AMI will have the same SSH host keys as the original AMI.  The only exception to this is if the original AMI creator spotted this problem and used a hook to force SSH host key regeneration upon first boot. This means that an attacker who say, uses a DNS cache poisoning attack, can intercept the communication between your SSH client and your AMI.

How can I fix my pre-June 23rd AMIs?

Regenerate the SSH host key.  The exact commands will depend on your operating system (hint: ssh-keygen).

Who is to blame?

Either the creators of the original AMI or Amazon - depends how you look at it.  If Amazon created the public AMI then it could be argued they are responsible.  However, anyone can submit a public AMI and Amazon makes no guarantee they are fit for use (Amazon do review the AMI listing according to their documentation).

Amazon can in fact make the argument they are acting in the interests of their users by implementing a shared solution to key regeneration (rather than requiring each user to manually regenerate the ssh host keys after booting an image).   That’s fine going forward but what of potential exposure to customers using the pre-June 23rd public AMI copies?

Just to be clear, its not the fault of SSH - ’secure channels’ require proper key management and the need for unique host keys is well documented.

Are there any mitigating factors?

Yes, if you have used security groups to limit SSH access to your AMI from IP ranges you trust (rather than the entire Internet).  You’ll still want to regenerate the ssh host keys sooner than later.

Is the Amazon environment vulnerable to Man-in-the-middle attacks?

I don’t know.  But that isn’t the real question - is the path between you and your AMI immune to MITM attacks and the answer is most definitely no.  If SSH on your AMI is only accessible from another AMI then its a fair question but its unlikely Amazon are going to show you their network diagrams ;-).  From experience performing MITM attacks, I would assume most networks are vulnerable (one of the reasons why we use SSH).

Why Didn’t Amazon Tell Me I’m Vulnerable?  They know from their logs what AMIs I use!

Didn’t they?  Whoops - naughty Amazon :P.

But seriously, Amazon are not responsible for the configuration of the public AMIs you use.  Its important not to confuse the AMI selection and cloning mechanism that Amazon provides, with the content of an AMI itself.

Does Amazon have a mailing list for customers to learn about new security problems (even if its not Amazon’s fault).

Not that I know of.   Right now you have to search forum posts and monitor documentation updates - which is time consuming and makes it easy to miss something.  I also can’t find an area on the AWS website where they collect security related items together (e.g. best practices, advisories, key management).   In my view, this is a shame as it probably undermines the effort that Amazon are putting into their security  (for some customers, if they don’t “see it”, it doesn’t “exist”).

A ‘Security’ link on the main AWS homepage pointing to those resources would go a long way to improving the visibility of the AWS security related information.

A quick post to say a very warm welcome to IMI Tech Talk / KFNX listeners!

I was recently approached to take part in an interview about Cloud Computing and Security on IMI Tech Talk, broadcast on KFNX News Talk Radio.  KFNX is a US based radio station based out of Phoenix, Arizona.  More in-depth than the previous opportunity, a range of Cloud Computing technologies were discussed in the 30 minute segment:

• Who am I?
• What is cloud computing? (*that* question!).
• Introduction to virtualization.
• Examples of cloud computing services that exist today.
• Barriers to entry.
• Security issues of processing or storing data in the cloud
• cloudsecurity.org

I did mention I would provide links to useful Cloud Computing resources (as my mind went totally blank during the interview!) - watch for a post next week covering the blogs I read regularly.

Cloudsecurity.org was born as I couldn’t find any dedicated web resource discussing Cloud Computing and Security.  If there are subjects you want to see covered, feel free to leave a suggestion in the Skribit sidebar to the right.

I do welcome comments in response to blog posts on the blog itself - don’t be shy :-).

For private communications I can be reached at craig.balding@gmail.com.

My thanks to the IMI Tech Talk team, particularly Tom and Eric.

Enjoy the blog,

Craig

In this interview, cloudsecurity.org talks to Guido van Rossum about Python, Google App Engine and security.

Guido is the creator of the Python programming language and more recently, Google App Engine team member.  His involvement with the App Engine project was pretty late - the code “was almost ready for release” when he get involved.  The security architect of App Engine was primarily project lead, Kevin Gibbs, supported by the rest of the App Engine crew and the Google Security Team.

The Interview

cloudsecurity.org: What security principles did you follow for App Engine?

GvR: While I can’t share any specifics on what we’re doing to secure App Engine, I can say that the main principle we’ve followed could be called “defense in depth”. We’re not relying exclusively on a secure interpreter, or any other single security layer, to protect our users.

cloudsecurity.org: Please provide some examples of how those principles played out in terms of the current implementation?

GvR: Sorry, we don’t divulge such information.

cloudsecurity.org: What criteria did you apply to Python module selection?

GvR: We first looked for modules that were useful and straightforward to audit. If a module was large or complex, we’d only audit it (fixing things we found) if it was deemed essential or at least useful for a large number of users; otherwise we’d exclude it.

cloudsecurity.org: What do you see as the security risks inherent in exposing an interpreter runtime in a shared environment?

GvR: I presume you’re asking about risks to users, like providing accidental access to data belonging to another app. We’ve taken extensive measures to isolate different apps from each other. For example, each app runs in a separate process, and the datastore prevents an app from accessing data belonging to other apps.

cloudsecurity.org: I recently attended a fascinating talk by Justin Ferguson (a Seattle based security consultant) at eusecwest in London.  He gave a great talk exploring security vulnerabilities in language interpreters and specifically highlighted some security weaknesses in Python App Engine.  What are your thoughts on his research and specifically the Python issues he highlighted?  When do you anticipate they will get fixed?

GvR: We’ve anticipated all of the possibilities raised in Justin’s talk, and took measures to protect our users. Justin highlighted weaknesses in Python, but not in App Engine. Furthermore, our security model does not rely solely upon protections within the Python interpreter; there are additional protections that these external analyses have missed.

cloudsecurity.org: How do you contain an attacker that exploits bugs in App Engine from exploiting the underlying OS and potentially interfering with other users processes or attacking backend systems?

GvR: You are correct that there are strong measures in place, but I’m not at liberty to discuss details.

cloudsecurity.org: Python was the first language to get the App Engine treatment, what language is next and what are some of the language specific security challenges the team has had to deal with?

GvR: Although I can’t comment on what language is next, we are working on this, and have gotten a lot of great feedback from our developers. As far as language-specific security challenges, they stemmed mostly from the complexity of the Python interpreter. We spent a lot of time auditing this, and did a great deal more than just identifying buffer overflows.  I can also add that Google is actively researching the security of interpreted languages.  Google engineers routinely contribute security fixes to open source projects, including but not limited to Python.

cloudsecurity.org: How does the team decide when ‘enough is enough’ in terms of hardening the interpreter?

GvR: That’s not really how we approach it. We realize that security is an ongoing effort, and try to stay ahead of threats through continuous monitoring and testing.

cloudsecurity.org: Some commentators have suggested that perhaps the difficulty of auditing the implementation led to some modules being more heavily restricted than perhaps necessary.  What are your thoughts on that and what plans, if any, are there to bring back code objects/functions that were eliminated in the initial release?  (with the benefit of hindsight).

GvR: The only thing we are likely to put back is the _ast module, which was not audited based upon an underestimation of its usefulness (see my answer to question #3 above).  We will also put back some dummy functions and other objects whose absence currently prevents some popular frameworks from being loaded without modifications. For example, some harmless functionality in the imp module will come back. We’re also looking into making urllib2 work (to some extent), though that’s not really a security issue but merely a matter of API adjustment.

cloudsecurity.org: It is reported that Google encourages small groups to go off and create.  How involved were the Google security team with App Engine in terms of design and implementation review/testing?  Given the dynamics, is it possible to have a meaningful security process that shadows the development process?

GvR: The Google Security team is involved in everything we do. They have been extremely helpful.

cloudsecurity.org: How can people report security weaknesses they discover in App Engine?  What commitment does Google give in terms of dealing vulnerability reports?

GvR: There is a standard process for submitting security issues. See http://www.google.com/corporate/security.html. Google moves very fast to protect its users when a verifiable security vulnerability is reported.

cloudsecurity.org: One concern is the potential misuse of App Engine to exploit security vulnerabilities in visitors browsers.  This is not a new problem per se, shared hosting providers know all about this.  But with Google and other Cloud providers, the scalability potential is much higher.  What are your thoughts on this and what pro-active steps is Google taking to detect and terminate evil apps?

GvR: This is high on our list of concerns. We deal with this through a combination of restrictions on what you can do (e.g. certain HTTP headers and ports are off-limits) and, again, monitoring.

cloudsecurity.org: Beyond App Engine, what role do you think Python will play in the Cloud both now and in the future?

GvR: Sorry, I’m not prone to philosophizing about the future.

cloudsecurity.org: Trust is often cited as a barrier to enterprise adoption of Cloud Computing.  What role do you personally think Google can play in building that trust?

GvR: I think trust is built up over a long period of experience. Our actions in terms of being open to our users will be the most important factor in establishing trust. Of course, Google’s reputation also helps: everybody understands that Google doesn’t want its name associated with a bad product.

cloudsecurity.org: Looking at the Cloud Computing landscape beyond Google, what are your thoughts on the current state of Cloud Computing and Security?

GvR: It’s obvious that Cloud Computing is only just taking off. The next few years will be very exciting.

cloudsecurity.org: Lastly, what are some of your favourite App Engine apps?

GvR: There are too many to enumerate. If you insist on a highlight, well, I like Rietveld (http://codereview.appspot.com), a tool for collaborative code review which I (largely) wrote myself. It is open source and includes some essential components from Mondrian, a similar internal tool which I created before I joined the App Engine team.

Thanks

My thanks to Guido for his time and sharing his views.

This happened recently to some Amazon S3 customers. There were complaints in the AWS forums about ‘S3 Corruption’. The first post in the forum was recorded at Jun 22, 2008 5:05 PM PDT (although in subsequent posts some people reported emailing Amazon prior to this):

we are having some serious S3 issues.

all data we store on S3 has gone through the same code path for months. starting a couple days ago a small percentage of the objects we are retrieving are not checksumming to the correct values. we hash and store objects by checksum and rehash the objects when we retrieve to ensure there is no data corruption. all the objects we’re having issues with were uploaded at approximately the same time period a few days ago.

we’ve stored 10’s of millions of objects in S3 and never encountered such problems. please let me know ASAP if you have any idea what could be going on here. thanks.

Amazon responded 6 minutes later (!) and started investigating. To troubleshoot they asked customers to email aws@amazon.com with the ‘Bucket-Name and few keys that you believe are having issues’.

Others weighed in reporting similar problems. Amazon provided status updates and on Monday Jun 23rd at 6:10pm PDT, provided the following explanation:

We’ve isolated this issue to a single load balancer that was brought into service at 10:55pm PDT on Friday, 6/20. It was taken out of service at 11am PDT Sunday, 6/22. While it was in service it handled a small fraction of Amazon S3’s total requests in the US. Intermittently, under load, it was corrupting single bytes in the byte stream. When the requests reached Amazon S3, if the Content-MD5 header was specified, Amazon S3 returned an error indicating the object did not match the MD5 supplied. When no MD5 is specified, we are unable to determine if transmission errors occurred, and Amazon S3 must assume that the object has been correctly transmitted. Based on our investigation with both internal and external customers, the small amount of traffic received by this particular load balancer, and the intermittent nature of the above issue on this one load balancer, this appears to have impacted a very small portion of PUTs during this time frame.

What are some of the takeaways?

• If you are directly using the AWS S3 API, make sure to calculate and send MD5 checksums along with actual data. Check status return codes - an HTTP 400 error code means ’something went wrong’ - respond appropriately.
• If you are relying on 3rd party tools to access S3, be sure to check with your software vendor that they are following the advice from Amazon to use MD5. If they are not then your data can get silently corrupted…
• Downloads, aka HTTP GETs, can also be affected. The thread in the forum continues and questions are asked as to whether the corruption caused by the loadbalancer was affecting both incoming and outgoing traffic. The conclusion was yes. If you are hosting media on S3, and the browser is using partial GET requests (to download in chunks) then the corruption will not be automatically detectable.
• If your business relies on Cloud Storage, are you prepared to wait a 36 hours for a resolution? This isn’t a swipe at Amazon, this is true for any provider. Check your SLA’s, check the trouble ticket resolution times, ask about availability of experts for troubleshooting etc.
• Cloud Providers will increasingly need to instrument their services such that they can ‘early detect’ negative operational events. In this case, Amazon has stated plans to use better logging and analysis to automate detection of unusual error patterns (i.e. anomoly detection).
• This incident - caused by an Amazon malfunctioning loadbalancer - did not make it onto the AWS status page at http://status.aws.amazon.com/. Taking Amazon at face value, this incident only affected a small number of transfers, relative to the total number of S3 transfers. But this begs the question, what level of outage or service problem needs to happen before Amazon will flag the issue on their status page? On a sidenote, based on the timestamps, 31 hours passed between the loadbalancer being taken out of service and Amazon providing the explanation on the forum.
• When Amazon update their S3 API documentation, it would be useful to have entries in the S3 API index for ‘checksum’, ‘MD5′, ‘integrity’ and ‘corruption’.
• Stepping back, will customers hold Cloud Service Providers to a higher standard than their own internal IT teams?

I’m sure there are more takeaways I didn’t cover. What say you?

###

Kudos for the heads-up on the S3 issue goes to my friend and colleague Jason Harper - network supremo and crypto-head. Thanks Jason!

Cloud Computing is starting to escape the technical and business press.

The proof?

I was invited to talk about Cloud Computing and Security on NPR “Morning Edition”.

NPR - National Public Radio - is a US based, non-commercial radio station covering news, talk and current affairs. British readers may find it similar to BBC Radio 4.

Every Monday, the “Morning Edition” has a technology theme. The Cloud Computing segment was high level and aimed primarily at a non-tech audience. I always find it hard to answer the question ‘what is Cloud Computing?’ as there are so many different definitions. Regardless, it was a great chance to talk about an exciting technology and highlight the need for a real security conversation between the providers and people interested in IT security - the primary reason why I created cloudsecurity.org.

The show boasts a very impressive audience - around 13 million! I’ve never before had the opportunity to confuse that many people in one shot ;-).

If you would like to listen (its short - 3.5 mins), click here.

I’d like to publicly thank Nina at NPR for reaching out and extend a warm ‘Welcome’ to any NPR listeners who have dropped by. Feel free to leave a message below or email me if you have any comments or questions.

With in-house IT, you pay your upfront capital costs and maintenance fees and you get whatever compute power you paid for. If you over-specify, you have excess computer power or disk - you are wasting money.  If you under-specify, you may be forced to raid your ‘rainy day’ budget and order new hardware.

A primary selling point of Cloud Computing is the ‘pay by the drink’ billing model - you only pay for the CPU cycles and storage you use - that’s it.

If you run any IT security tools at all, Cloud Computing may impact the way you calculate your IT security budgets.

Assessing The Cost of Runtime Security

Security costs can be overt or hidden:

• budget items spread across infrastructure, security, compliance, midrange.
• the runtime security costs of security tools that execute on the systems.

How many organisations know their runtime security compute costs?  My guess is not many.  Under the traditional IT billing model, you mostly don’t need to figure this stuff out. As long as your security tools don’t chew up the CPU unnecessarily or fill the disk, everyone is happy.

The performance of security products varies greatly.  On the negative side, poor design or implementation are problems only the vendor can address.   Site specific issues arise through all kinds of madness - customers failing to “read the label” and provision properly, insufficiently trained people making poor configuration choices or simply relying on the default settings in a very non-default environment!

The negative side effects of in-line security tools hit home as system load increases.  Access checks, logging and other ‘in-line’ security operations may perform fine under normal load fail to scale as load increases past a certain threshold.  This can lead to CPU spikes or poor disk access patterns.

Switch Off Or Pay Up?

To bring this closer to home, lets explore how the impact of security tools plays out today under traditional IT and tomorrow, under Cloud Computing.  Lets eavesdrop on a fictitious conversation between Oscar the ORACLE DBA and Simon the Security Dude.

Oscar: Hey Simon, your Security Agents are killing system performance again. Anna in accounts called up to say they can’t do the Quarterly close, the jobs are getting killed before they finish.

Simon: Hi Simon, I understand but we can’t just disable all the security!

Oscar: Well, we need to do something if we are going to finish posting our numbers this quarter. Are you volunteering to explain to our CEO why we didn’t?

Simon: Hmm. Let me check the agent logs, perhaps there is a problem.

Oscar: I already checked them, no errors reported.

Simon: Hmm. I’ll log a call with the Premium International Support Service.

Oscar: You did that last time and the support guy stuck to the party line that the security agent takes 5-10% of CPU. We know those numbers are wrong from our benchmarking - sometimes it takes 20% of CPU and always a lot more during quarter close.

Simon: Hmm. Are there any other processes running on the system we can disable for a while?

Oscar: Nope - we’re running a tight a ship as we can here. I’ve already told Steve from sourcing he is going to have to wait for his reports.

Simon: Hmm. Bugger. OK, I’ll disable the agents - but you must tell me as soon as the quarter close completes so I can start them up again.

Oscar: Thanks - will do.

A classic conversation under the ‘old regime’. Simon is forced into an operational security decision due to an under-specified system or an over indulgent security agent. His only option in this scenario is to disable the poorly scaling security tool. He can’t just scream “Need more power!” and additional CPUs appear.

Now lets see how this plays out with Cloud Computing, where the change in paradigm will remove the compute limits and make your on the spot risk decisions link directly to your costs and security tool efficiencies:

Simon the Security Dude receives an auto-generated email from the Cloud Provider:

A virtual CPU was auto-inserted on virtual machine image FINANCE1 at 10:30am as Runtime Security Compute usage exceeded the agreed threshold in the SLA.   Please note, you have now reached your soft credit limit - please click the link below to authorize an increase. You currently have 4USD left in your account.

So what does Simon do now? He already tapped into his security compute budget five times this week and he’s running low. The silver lining is that at least he gets to make the decision now - he isn’t forced to ’switch off security’. If he has the cash, he can attempt to buy his way out of the problem. The obvious negative is “death by a thousand costs” - he’s running out of budget.

The root cause of the problem is that prior to moving to the Cloud, Simon didn’t have a handle on how much runtime security was *really* costing. He didn’t know (a) his runtime security costs or (b) how much of that cost was unnecessary - caused by security tool inefficiency.  He wasn’t the one paying, so most of the time he didn’t have to care. Even if he had found a way to calculate his costs, he’d still have to figure out how performance differences of Cloud Computing would skew his numbers.

And therein lies the rub: if you don’t know your security runtime costs are today - and where the waste is - how will you cope “tomorrow” when it’s always your turn to pay for drinks at the Cloud Bar?

What are the telltale signs that your company is already Computing in the Cloud?

Is it when the CIO makes a big announcement at the monthly IT meeting?

Is it when the IT newsletter drops a reference to pilot testing of some ‘web based’ software?

Or, is it when the secretary whips out the boss’s Corporate Credit Card and signs up to a Cloud Service?

Here are 12 indicators that your company is *already* part of the Cloud:

1. Your internal helpdesk reports fewer password resets.
2. Finance contacts you to confirm all the DVD readers are disabled - they are puzzled by the number of recurring credit card charges for Amazon (are the secretaries spreading out their orders for “Lost” DVDs again?).
3. You are asked to authorise a network change ticket that modifies the LAN routing policy.  All traffic will be sent directly to the Internet proxy (for performance reasons).  From the accompanying diagram, the data center appears to have been cut and pasted on the wrong side of the firewall (idiots!).
4. You walk into the Data Center and it feels cooler than usual.
5. When the builders next door accidentally saw through the company Internet connection, people complain there must be a DoS attack going on as they can’t get to their files.
6. During physical inspections, you notice unexplained gaps in server cabinets.
7. Login failures go down, in fact login “attempts” in general go down but the company car park is full.
8. As you walk through the office, you notice all the “Security Awareness” posters have been replaced with pictures of Jeff Bezos (!)
9. You are asked to authorise a visit from the local environment group. Fearing protesters, you are surprised to learn that your company has won a prize for reducing its Carbon Footprint
10. Your Intrusion Prevention System is preventing the call center from uploading contracts stored as GIF files.
11. You detect the presence of ‘malware’ in the form of unexplained ‘Machine Images’ on IT’s desktops.
12. You stop finding Windows passwords under keyboards, instead you find random hex digits next to the words ‘Access Key’ and ‘Secret Key’. You sigh, but at least they are setting difficult to guess passwords now!

If you are charged with IT security in your company, you may want to start checking your web proxy logs for telltale signs that people are talking to the Cloud…or just talk to finance.