Aptimize is a New Zealand-based company that has developed software to automatically do the most high value front end optimizations (image spriting, CSS/JS combination and minification, etc.).  We predict it’ll be big.  On a site like ours, going back and doing all this across hundreds of apps will never happen – we can engineer new ones and important ones better, but something like this which can benefit apps by the handful is great.

I got some good info from the MySpace people.  We’ve been talking about whether to run our back end as Linux/Apache/Java or Windows/IIS/.NET for some of our newer stuff.  In the first workshop, I was impressed when the guy asked who all runs .NET and only one guy raised his hand.   MySpace is one of the big .NET sites, but when I talked with them about what they felt the advantage was, they looked at each other and said “Well…  It was the most expeditious choice at the time…”  That’s damning with faint praise, so I asked about what they saw the main disadvantage being, and they cited remote administration – even with the new PowerShell stuff it’s just still not as easy as remote admin/CM of Linux.  That’s top of my list too, but often Microsoft apologists will say “You just don’t understand because you don’t run it…”  But apparently running it doesn’t necessarily sell you either.

Our friends from Opnet were there.  It was probably a tough show for them, as many of these shops are of the “I never pay for software” camp.  However, you end up wasting far more in skilled personnel time if you don’t have the right tools for the job.  We use the heck out of their Panorama tool – it pulls metrics from all tiers of your system, including deep in the JVM, and does dynamic baselining, correlation and deviation.  If all your programmers are 3l33t maybe you don’t need it, but if you’re unsurprised when one of them says “Uhhh… What’s a thread leak?” then it’s money.

ControlTier is nice, they’re a commercial open source CM tool for app deploys – it works at a higher level than chef/puppet, more like capistrano.

EngineYard was a really nice cloud provisioning solution (sits on top of Amazon or whatever).  The reality of cloud computing as provided by the base IaaS vendors isn’t really the “machines dynamically spinning up and down and automatically scaling your app” they say it is without something like this (or lots of custom work).  Their solution is, sadly, Rails only right now.  But it is slick, very close to the blue-sky vision of what cloud computing can enable.

And also, I joined the EFF!  Cyber rights now!

You can see most of the official proceedings from the conference (for free!):

• Conference Videos
• Velocity Photos
• Presentation Slides

And then – we were lame and just turned in.  I’m getting old, can’t party every night like I used to.  (I don’t know what Peco’s excuse is!)

• Online apps are complex systems
• A siloed approach of deciding to improve midtier vs CDN vs front end engineering results in suboptimal experience to the end user – have to take holistic view.  I totally agree with this, in our own caching project we took special care to do an analysis project first where we evaluated impact and benefit of each of these items not only in isolation but together so we’d know where we should expend effort.
• Use top level/end user metrics, not system metrics, to measure performance.
• There are other metrics that correlate to your performance – “key indicators.”
• It’s hard to take low level metrics and take them “up” into a meaningful picture of user experience.

He’s covering good stuff but it’s nothing I don’t know.  We see the differences and benefits in point in time tools, Passive RUM, tagging RUM, synthetic monitoring, end user/last mile synthetic monitoring…  If you don’t, read the presentation, it’s good.  As for me, it’s off to the scaling session.

I hopped into this session a half hour late.  It’s Scalable Internet Architectures (again, go get the presentation) by Theo Schlossnagle, CEO of OmniTI and author of the similarly named book.

I like his talk, it starts by getting to the heart of what Web Operations – what we call “Web Admin” hereabouts – is.  It kinda confuses architecture and operations initially but maybe that’s because I came in late.

He talks about knowledge, tools, experience, and discipline, and mentions that discipline is the most lacking element in the field. Like him, I’m a “real engineer” who went into IT so I agree vigorously.

What specifically should you do?

• Use version control
• Monitor
• Serve static content using a CDN, and behind that a reverse proxy and behind that peer based HA.  Distribute DNS for global distribution.
• Dynamic content – now it’s time for optimization.

Optimizing Dynamic Content

Don’t pay to generate the same content twice – use caching.  Generate content only when things change and break the system into components so you can cache appropriately.

example: a php news site – articles are in oracle, personalization on each page, top new forum posts in a sidebar.

Why abuse oracle by hitting it every page view?  updates are controlled.  The page should pull user prefs from a cookie.  (p.s. rewrite your query strings)
But it’s still slow to pull from the db vs hardcoding it.
All blog sw does this, for example
Check for a hardcoded php page – if it’s not there, run something that puts it there.  Still dynamically puts in user personalization from the cookie.  In the preso he provides details on how to do this.
Do cache invalidation on content change, use a message queuing system like openAMQ for async writes.
Apache is now the bottleneck – use APC (alternative php cache)
or use memcached – he says no timeouts!  Or… be careful about them!  Or something.

Scaling Databases

1. shard them
2. shoot yourself

Sharding, or breaking your data up by range across many databases, means you throw away relational constraints and that’s sad.  Get over it.

You may not need relations – use files fool!  Or other options like couchdb, etc.  Or hadoop, from the previous workshop!

Vertically scale first by:

• not hitting the damn db!
• run a good db.  postgres!  not mySQL boo-yah!

When you have to go horizontal, partition right – more than one shard shouldn’t answer an oltp question.   If that’s not possible, consider duplication.

IM example.  Store messages sharded by recipient.  But then the sender wants to see them too and that’s an expensive operation – so just store them twice!!!

But if it’s not that simple, partitioning can hose you.

Do math and simulate it before you do it fool!   Be an engineer!

Multi-master replication doesn’t work right.  But it’s getting closer.

Networking

The network’s part of it, can’t forget it.

Of course if you’re using Ruby on Rails the network will never make your app suck more.  Heh, the random drive-by disses rile the crowd up.

A single machine can push a gig.  More isn’t hard with aggregated ports.  Apache too, serving static files.  Load balancers too.  How to get to 10 or 20 Gbps though?  All the drivers and firmware suck.  Buy an expensive LB?

Use routing.  It supports naive LB’ing.  Or routing protocol on front end cache/LBs talking to your edge router.  Use hashed routes upstream.  User caches use same IP.  Fault tolerant, distributed load, free.

Use isolation for floods.  Set up a surge net.  Route out based on MAC.  Used vs DDoSes.

Service Decoupling

One of the most overlooked techniques for scalable systems.  Why do now what you can postpone till later?

Break transaction into parts.  Queue info.  Process queues behind the scenes.  Messaging!  There’s different options – AMQP, Spread, JMS.  Specifically good message queuing options are:

• ActiveMQ (Java)
• OpenAMQ (C)
• RabbitMQ (erlang)

Most common – STOMP, sucks but universal.

Combine a queue and a job dispatcher to make this happen.  Side note – Gearman, while cool, doesn’t do this – it dispatches work but it doesn’t decouple action from outcome – should be used to scale work that can’t be decoupled.  (Yes it does, says dude in crowd.)

Scalability Problems

It often boils down to “don’t be an idiot.”  His words not mine.  I like this guy. Performance is easier than scaling.  Extremely high perf systems tend to be easier to scale because they don’t have to scale as much.

e.g. An email marketing campaign with an URL not ending in a trailing slash.  Guess what, you just doubled your hits.  Use the damn trailing slash to avoid 302s.

How do you stop everyone from being an idiot though?  Every person who sends a mass email from your company?  That’s our problem  – with more than fifty programmers and business people generating apps and content for our Web site, there is always a weakest link.

Caching should be controlled not prevented in nearly any circumstance.

Understand the problem.  going from 100k to 10MM users – don’t just bucketize in small chunks and assume it will scale.  Allow for margin for error.  Designing for 100x or 1000x requires a profound understanding of the problem.

Example – I plan for a traffic spike of 3000 new visitors/sec.  My page is about 300k.  CPU bound.  8ms service time.  Calculate servers needed.  If I varnish the static assets, the calculation says I need 3-4 machines.  But do the math and it’s 8 GB/sec of throughput.  No way.  At 1.5MM packets/sec – the firewall dies.  You have to keep the whole system in mind.

So spread out static resources across multiple datacenters, agg’d pipes.
The rest is only 350 Mbps, 75k packets per second, doable – except the 302 adds 50% overage in packets per sec.

Last bonus thought – use zfs/dtrace for dbs, so run them on solaris!

Other good references -
book: “Hadoop: The Definitive Guide”
preso: hadoop cluster management from USENIX 2009

Hadoop is an Apache project inspired by Google’s infrastructure; it’s software for programming warehouse-scale computers.

It has recently been split into three main subprojects – HDFS, MapReduce, and Hadoop Common – and sports an ecosystem of various smaller subprojects (hive, etc.).

Usually a hadoop cluster is a mess of stock 1 RU servers with 4×1TB SATA disks in them.  “I like my servers like I like my women – cheap and dirty,” Jeff did not say.

HDFS:

• Pools servers into a single hierarchical namespace
• It’s designed for large files, written once/read many times
• It does checksumming, replication, compression
• Access is from from Java, C, command line, etc.  Not usually mounted at the OS level.

MapReduce:

• Is a fault tolerant data layer and API for parallel data processing
• Has a key/value pair model
• Access is via Java, C++, streaming (for scripts), SQL (Hive), etc
• Pushes work out to the data

Subprojects:

• Avro (serialization)
• HBase (like Google BigTable)
• Hive (SQL interface)
• Pig (language for dataflow programming)
• zookeeper (coordination for distrib. systems)

Facebook used scribe (log aggregation tool) to pull a big wad of info into hadoop, published it out to mysql for user dash, to oracle rac for internal…
Yahoo! uses it too.

Sample projects hadoop would be good for – log/message warehouse, database archival store, search team projects (autocomplete), targeted web crawls…
As boxes you can use unused desktops, retired db servers, amazon ec2…

Tools they use to make hadoop include subversion/jira/ant/ivy/junit/hudson/javadoc/forrest
It uses an Apache 2.0 license

Good configs for hadoop:

• use 7200 rpm sata, ecc ram, 1U servers
• use linux, ext3 or maybe xfs filesystem, with noatime
• JBOD disk config, no raid
• java6_14+

To manage it -

unix utes: sar, iostat, iftop, vmstat, nfsstat, strace, dmesg, friends

java utes: jps, jstack, jconsole
Get the rpm!  www.cloudera.com/hadoop

config: my.cloudera.com
modes – standalong, pseudo-distrib, distrib
“It’s nice to use dsh, cfengine/puppet/bcfg2/chef for config managment across a cluster; maybe use scribe for centralized logging”

I love hearing what tools people are using, that’s mainly how I find out about new ones!

Common hadoop problems:

• “It’s almost always DNS” – use hostnames
• open ports
• distrib ssh keys (expect)
• write permissions
• make sure you’re using all the disks
• don’t share NFS mounts for large clusters
• set JAVA_HOME to new jvm (stick to sun’s)

HDFS In Depth

1.  NameNode (master)
VERSION file shows data structs, filesystem image (in memory) and edit log (persisted) – if they change, painful upgrade

2.  Secondary NameNode (aka checkpoint node) – checkpoints the FS image and then truncates edit log, usually run on a sep node
New backup node in .21 removes need for NFS mount write for HA

3.  DataNode (workers)
stores data in local fs
stored data into blk_<id> files, round robins through dirs
heartbeat to namenode
raw socket to serve to client

4.  Client (Java HDFS lib)
other stuff (libhdfs) more unstable

hdfs operator utilities

• safe mode – when it starts up
• fsck – hadoop version
• dfsadmin
• block scanner – runs every 3 wks, has web interface
• balancer – examines ratio of used to total capacity across the cluster
• har (like tar) archive – bunch up smaller files
• distcp – parallel copy utility (uses mapreduce) for big loads
• quotas

has users, groups, permissions – including x but there is no execution, but used for dirs
hadoop has some access trust issues – used through gateway cluster or in trusted env
audit logs – turn on in log4j.properties

has loads of Web UIs – on namenode go to /metrics, /logLevel, /stacks
non-hdfs access – HDFS proxy to http, or thriftfs
has trash (.Trash in home dir) – turn it on

includes benchmarks – testdfsio, nnbench

Common HDFS problems

• disk capacity, esp due to log file sizes – crank up reserved space
• slow but not dead disks and flapping NICS to slow mode
• checkpointing and backing up metadata – monitor that it happens hourly
• losing write pipeline for long lived writes – redo every hour is recommended
• upgrades
• many small files

MapReduce

use Fair Share or Capacity scheduler
distributed cache
jobcontrol for ordering

Monitoring – They use ganglia, jconsole, nagios and canary jobs for functionality

Question – how much admin resource would you need for hadoop?  Answer – Facebook ops team had 20% of 2 guys hadooping, estimate you can use 1 person/100 nodes

He also notes that this preso and maybe more are on slideshare under “jhammerb.”

I thought this presentation was very complete and bad ass, and I may have some use cases that hadoop would be good for coming up!

Why the cloud is a problem for security

• Poor understanding of cloud taxonomies and definitions
• A generic term, frequently misused to refer to anything on the Internet
• Lack of visibility into cloud deployments
• Organic consumption

Couldn’t have talked about this stuff 6 months ago because nobody knew about it and it wasn’t discussed.

Security Implications

• Variable control
• Variable visibility
• Variable simplicity/complexity
• Variable resources

Control, visibility, and resources goes down as simplicity and management goes up

Is the cloud more or less secure than we are now?  It depends.  Something are more secure and some things are less secure because of all of the variability.

Saas

• Most constrained
• Most security managed by your provider
• Least flexible

PaaS

• Less constrained
• Security varies tremendously based on provider and application-shared responsibility
• Security responsibility

IaaS

• Most flexible
• Most security managed by your developers

Specific Issues

• Spillage and data security
• Reliability/availability
• Capability to apply traditional security controls in a dynamic environment
• Lack of visibility into cloud usage
• Changing development patterns/cycles

How do you use your static and dynamic analysis testing tools in the cloud?

Where do you roll your cloud when it fails?

Your Top 2 Cloud Security Defenses

• SLA
• Contracts

Understand Your SLAs

• Are there security-specific SLAs?
• Can you audit against those SLAs?
• Are there contractual penalties for non-compliance?
• Do your SLAs meet your risk tolerance requirements?

Suggested SLAs

• Availability
• Security audits – including third party
• Data security/encryption
• Personal security
• Security controls (depend based on service)
• User account management
• Infrastructure changes

Understand Your Cloud

• What security controls are in your cloud?
• How can you manage and integrate with the controls?
• What security documentation is available?
• What contingency plans are available?

Cloud Security Controls to Look For

• Data encryption/security (key management)
• Perimeter defenses
• Auditing/logging
• Authentication
• Segregation
• Compliance

Cloud Security Macro Layers

• Network
• Service
• User
• Transaction
• Data

Don’t Trust

• SAS70 Audits
• Documentation without verification
• Non-contractual SLAs

What to Do

• Educate yourself
• Engage with developers
• Develop cloud security requirements

Phil: Little difference between outsources of the past and today’s Cloud Computing.  All of that stuff is sitting outside of your environment and we’ve been evolving toward that for a long time.

Rich: My impression is that there are benefits to outsourced hosting, but there are clearly areas that make sense and areas that don’t.  This is fundamentally different from shared computing resources.  Very different applications for this.  Complexity goes up very quickly very quickly for security controls.  Where do you see the most value today?  Where do people need to be most cautious?

Jim: Internal virtualization is almost necessary, but it impacts almost every IT process.  Technology is still evolving and is far from advanced state.  Be pragmatic and find particular applications with a good ROI.

Josh: Understand what you are putting into a cloud environment.  Have a good understanding of what a provider can offer you in terms of sensitive data.  Otherwise you’re putting yourself in a very bad situation.  A lot of promise.  Great for social networking and web development.  Not appropriate with enterprises with large amounts of IP and sensitive data.

Jim: We’ll get there in 4-5 years.

Phil: Let supply chain experts do it for you and then interact with them.  Access their enviornment from anywhere.  Use a secure URL with a federated identity.  Your business will come back to you and say “We need to do this” and IT will be unable to assist them.  Use it as an opportunity to mobilize compliance and InfoSec and get involved.  It’s going to come to use and we’re just going to have to deal with it.  There’s a long line of people with a “right to audit”.  Don’t think that someone is doing the right thing in this space, you have to ask.

Audience: What is the most likely channel for standards?

Phil: Cloud Security Alliance is a step in the right direction.  Want to come up with PCI DSS like checklists.  CSA is working with IEEE and NIST to work along with them.  Goal is to be able to feed the standards process, not become a standards body.

Rich: The market is anti-standards based.  If we get standardized, then all of the providers are only competing based on cost.

Jim: I think it’ll happen.  We will see ISO groups for standards on cloud quality.

Audience: Moving data between multiple clouds.  How do you determine who gets paid?

Jim: There are proposals for doing that.  All of the resource parameters.

Phil: Should see standards based on federated identity.  Who is doing what and where.  That’s where I’ve seen the most movement.  There is no ISO for SaaS.  Remapping how 27001 and 27002 apply to us as a software provider.

Audience: Two things that drive standards.  The market or monopoly (BetaMax).

Rich: We will have monopolistic ones and then 3rd parties that say they use those standards.

Audience: How can you really have an objective body create standards without being completely embedded in the technology?

Jim: You create a reference standard and the market drives that.

Phil: Gravity pulls us to things that work.  Uses SAML as an example.  It’s the way the internet has always worked.  The strongest will survive and the right standards will manifest themselves.

Rich: What are some of things that you’re dealing with internally (as consumers and providers) and the top suggestions for people stuck in this situation?

Jim: People who don’t have all of the  requirements do public clouds.  If what you want is available (salesforce.com), it may be irresistible.

Josh: Solution needs to be appropriate to the need.  Consult with your attorney to make sure you contract is in line with what you’re leveraging the provider for.  It’s really about what you agree to with that provider and their responsibilities.

Phil: The hurricane is coming.  You can’t scream into the wind, you gotta learn to run for cover.  Find the safe spot.

Audience: What industries do you see using this?  I don’t see it with healthcare.

Phil: Mostly providers for us.  Outsourcing service desks.  Government.  Large states/local.

Josh: Small and medium retail businesses.  Get products out there at a significantly reduced cost.

Jim: Lots of financial institutions looking for ways to cut costs.  Healthcare industry as well (Mayo Clinic).  Broad interest across the whole market, but especially anywhere they’re under extreme cost measures.

Rich: I run a small business that picked an elastic provider that couldn’t pay for a full virtual hosting provider.  Doing shared hosting right now, but capable of growing to a virtual private server.  Have redundancy.  Able to go full-colocation if they need it.  Able to support growth, but start with the same instance to get there.

Audience: How does 3rd party transparency factor into financial uses?

Jim: Almost exclusively private clouds.  There are use cases playing out right now that will be repeatable patterns.  Use cases.

Phil: When the volume isn’t there, offload to someone like Rackspace and they’ll help you to grow.

Audience: Are there guidelines to contracts to make sure information doesn’t just get outsourced to yet another party?

Phil: Your largest partners/vendors steal their contracts.  Use them as templates.

Audience: What recourse do you have that an audit is used to verify that security is not an issue?

Rich: Contracts.

Phil: Third party assessment (ie. the right to audit).  It’s in our interest to verify they are secure.  It’s a trend and we now have a long list of people looking to audit against us as a provider.  Hoping for an ISO to come up truly for the cloud.

Audience: Is cloud computing just outsourcing?

Rich: It’s more than that.  For example, companies have internal clouds that aren’t outsourced at all.

Josh: Most of the time it’s leveraging resources more efficiently at hopefully a reduced cost.

Audience: How do I know you’re telling me the truth about the resources I’m using?  What if I’m a bad guy who wants to exploit a competitor using the cloud?

Josh: We’ve seen guys create botnets using stolen credit cards.  What you’re billed for is in your contract.

Jim: We’ve had this solved for decades on mainframes.  Precious resources propagated amongst users.  There’s no technical reason we’re not doing it today.

Rich: It depends what type of cloud you’re using.  Some will tell you.

Josh: If you’re worried about someone abusing you, why are you there in the first place?

Phil: For our service desk we meter this by how many calls, by location.  Monitor servers that were accessed/patched/etc.  Different service providers will have different levels.

Audience: Seeing some core issues at the heart of this.  For businesses, an assessment of core competencies.  Can you build a better data center with the cloud?  Second issue involves risk assessment.  Can you do a technical audit?  Can you pay for it legally?  How much market presence does the vendor have?  Who has responsibility for what?  Notion of transparency of control.  Seems like it distills down to those core basics.

Jim: I agree.

Rich: Well said.

Phil: Yes, yes, yes.

Audience: How do you write a contract for failed nation states, volatility, etc?  Do we say you can’t put our stuff in these countries?

Phil: This is the white elephant in the room.  How can you ensure that my data is being protected the way I’d protect it myself.  It’s amazing what other people do when they get a hold of that stuff.  This is the underlying problem that we have to solve.  “Moving from a single-family home to a multi-tenant condo.  How do we build that now?

Rich: You need to be comfortable with what you’re putting out there.

Audience: To what extent is the military or federal government using cloud computing?

Jim: They’re interested in finding ways, but they don’t talk about how they’re using it.

Audience – Vern: They’re doing cloud computing using an internal private cloud already.  They bill back to the appropriate agency based on use.

Phil: Government is very wary of what’s going on.

Temple Inland Implementation – Stage 1

Overcome Hurdles

• Management skeptical of Windows virtualization

Don’t Fear the Virtual World

• First year:
  • Built out development only environment
  • Trained staff
  • Developed support processes
  • Showed hard dollar savings

Temple Inland – Stage 2

• Build QA environment
• Improve processes
• Develop rapid provisioning
• Demonstrate advanced functions
  • Vmotion
  • P2V Conversions

Temple Inland – Stage 3

First production environment

Temple-Inland Implementation

• Prior to VMWare. Typical remote facility
  • Physical domain controller
  • Physical application/file server
  • Physical tape drive
• New architecture
  • Single VMWare server
  • No tape drive

• Desktops
  • Virtualize desktops through VMWare
  • No application issues like Citrix Metaframe
  • Quick deployment and repair

How Virtualization Affects Datacenter Security

• Abstraction and Consolidation
  • +Capital and Operational Cost Savings
  • -New infrastructure layer to be secured
  • -Greater impact of attack or misconfiguration
• Collapse of Switches and servers into one device
  • +Flexibility
  • +Cost-savings
  • -Lack of virtual network visibility
  • -No separation-by-default of administration

Temple-Inland split the teams so that there was a virtual network administration team within the server administration team.

How Virtualization Affects Datacenter Security

• Faster deployment of servers
  • + IT responsiveness
  • -Lack of adequate planning
  • -Incomplete knowledge of current state of infrastructure
• VM Mobility
  • +Improved Service Levels
  • -Identity divorced from physical location
• VM Encapsulation
  • +Ease of business continuity
  • +Consistency of deployment
  • +Hardware Independence
  • -Outdated offline systems

Build anti-virus, client firewalls, etc into the offline images so that servers are up-to-date right when they are installed.

If something happens to a system, you can’t just pull the plug anymore.  You have to have policies and processes in place.

With virtualization you can have a true “gold image” instead of having different images for all of the different types of hardware.

Security Advantages of Virtualization

• Allows automation of many manual error prone processes
• Cleaner and easier disaster recovery/business continuity
• Better forensics capabilities
• Faster recovery after an attack
• Patching is safer and more effective
• Better control over desktop resources
• More cost effective security devices
• App virtualization allows de-privileging of end users
• Better lifecycle controls
• Future: Security through VM Introspection

Gartner: “Like their physical counterparts, most security vulnerabilities will be introduced through misconfiguration”

What Not to Worry About

• Hypervisor Attacks
  • ALL theoretical, highly complex attacks
  • Widely recognized by security community as being only of academic interest
• Irrelevant Architectures
  • Apply only to hosted architecture (ie. Workstation) not bare-metal (ie. ESX)
  • Hosted architecture generally suitable only when you can trust the guest VM
• Contrived Scenarios
  • Involved exploits where best practices around hardening, lockdown, desgin, for virtualization etc not followed or
  • Poor general IT infrastructure security is assumed

Are there any Hypervisor Attack Vectors?

There are currently no known hypervisor attack vectors to date that have lead to “VM Escape”

• Architecture Vulnerability
  • Designed specifically with isolation in mind
• Software Vulnerability – Possible like with any code written by humans
  • Mitigating Circumstances:
    • Small Code Footprint of Hypervisor (~21MB) is easier to audit
    • If a software vulnerability is found, exploit difficulty will be very high
      • Purpose build for virtualization only
      • Non-interactive environment
      • Less code for hackers to leverage
  • Ultimately depends on VMWare security response and patching

Concern: Virtualizing the DMZ/Mixing Trust Zones

Three Primary Configurations

• Physical separation of trust zones
• Virtual separation of trust zones with physical security devices
• Fully collapsing all servers and security devices into a VI3 infrastructure

Also applies to PCI requirement

Physical Separation of Trust Zones

Advantages

• Simpler, less complex configuration
• Less change to physical environment
• Little change to separation of duties
• Less change in staff knowledge requirements
• Smaller chance of misconfiguration

Disadvantages

• Lower consolidation and utilization of resources
• Higher cost

Virtual Separation of Trust Zones with Physical Security Devices

Advantages

• Better utilization of resources
• Take full advantage of virtualization benefits
• Lower cost

Disadvantages (can be mitigated)

• More complexity
• Greater chance of misconfiguration

Getting more toward “the cloud” where web zone, app zone, and DB zone are all virtualized on the same system, but still using physical firewalls.

Fully Collapsed Trust Zones Including Security Devices

Advantages

• Full utilization of resources, replacing physical security devices with virtual
• Lowest-cost option
• Management of entire DMZ and network from a single management workstation

Disadvantages

• Greatest complexity, which in turn creates highest chance of misconfiguration
• Requirement for explicit configuration to define separation of duties to help mitigate risk of misconfiguration; also requires regualar audits of configurations
• Potential loss of certain functionality, such as VMotion (being mitigated by vendors and VMsafe)

How do we secure our Virtual Infrastructure?

Use the principles of Information Security

• Hardening and lockdown
• Defense in depth
• Authorization, authentication, and accounting
• Separation of duties and least privileges
• Administrative controls

Protect your management interfaces (VCenter)!  They are the keys to the kingdom.

Fundamental Design Principles

• Isolate all management networks
• Disable all unneeded services
• Tightly regualte all administrative access

Summary

• Define requirements and ensure vendor/product can deliver
  • Consider culture, capability, maturity, architecture and security needs
• Implement under controlled conditions using a defined methodology
  • Use the opportunity to improve control deficiencies in existing physical server areas if possible
  • Implement processes for review and validation of controls to prevent the introduction of weaknesses
• Round corners where your control environment allows
  • Sustain sound practices that maintain required controls
  • Leverage the technology to achieve efficiency and improve scale

Agenda

• About the Cloud Security Alliance
• Getting Involved
• Guidance 1.0
• Call to Action

About the Cloud Security Alliance

• Not-for-profit organization
• Inclusive membership, supporting broad spectrum of subject matter expertise: cloud experts, security, legal, compliance, virtualization, etc
• We believe in Cloud Computing, we want to make it better

Getting Involved

• Individual membership (free)
  • Subject matter experts for research
  • Interested in learning about the topic
  • Administrative & organizational help
• Corporate Sponsorship
  • Help fund outreach, events
• Affiliated Organizations (free)
  • Joint projects in the community interest
• Contact information on website

Download version 1.0 of the Security Guidance at http://www.cloudsecurityalliance.org/guidance

Overview of Guidance

• 15 domains
• #1 is Architecture & Framework
• Covers Governing in the Cloud (2-7) and Operating in the Cloud (8-15) as well

Assumptions & Objectives

• Trying to bridge gap between cloud adopters and security practitioners
• Broad “security program” view of the problem

Architecture Framework

• Not “One Cloud”: Nuanced definition critical to understanding risks & mitigation
• 5 principal characteristics (abstration, sharing, SOA, elasticity, consumption/allocation)
• 3 delivery models
  • Infrastructure as a Service
  • Platform as a Service
  • Software as a Service
• 4 deployment models: Public, Private, Managed, Hybrid

Governance & ERM

• A portion of cloud cost savings must be invested into provider security
• Third party transparency of cloud provider
• Financial viability of cloud provider
• Alignment of key performance indicators
• PII best suited in private/hybrid cloud outside of significant due diligence of public cloud provider
• Increased frequency of 3rd party risk assessments

Important thing to consider is the financial viability of your provider.  You never want to have your data held hostage in a court battle.

Legal

• Contracts must have flexible structure for dynamic cloud relationships
• Plan for both an expected and unexpected termination of the relationship and an orderly return of your assets
• Find conflicts between the laws the cloud provider must comply with and those governing the cloud customer

Compliance & Audit

• Classify data and systems to understand compliance requirements
• Understand data locations, copies

Information Lifecycle Management

• Understand the logical segregation of information and protective controls imnplemented in storage, transfers, backups

Summary

• Cloud Computing is real and transformational
• Cloud Computing can and will be secured
• Broad governance approach needed
• Tactical fixes needed
• Combination of updating existing best practices and creating completely new best practices
• Common sense is not optional

Call to Action

• Join us, help make our work better
• www.cloudsecurityalliance.org
• info@cloudsecurityalliance.org
• Twitter: @cloudsa, #csaguide

5 Key Cloud Characteristics

• On-demand self-service
• Ubiquitous network access
• Location independent resource pooling
• Rapid elasticity
• Pay per use

3 Cloud Delivery Models

• Software as a Service (SaaS): Providers applications over a network
• Platform as a Service (PaaS): Deploy customer-created apps to a cloud
• Infrastructure as a Service (IaaS): Rent processing, storage, etc

4 Cloud Deployment Models

• Private cloud: Enterprise owned or leased
• Community cloud: Shared infrastructure for a specific community
• Public cloud: Sold to the public, Mega-scale infrastructure
• Hybrid cloud: Composition of two or more clouds

• Two types: internal and external
• http://csrc.nist.com/groups/SNS/cloud-computing/index.html

Common Cloud Characteristics

• Massive scale
• Virtualization
• Free software
• Autonomic computing
• Multi-tenancy
• Geographically distributed systems
• Advanced security technologies
• Service oriented software

Pros

• Lower central processing unit (CPU) density
• Flexible use of resources
• Rapid deployment of new servers
• Simplified recovery
• Virtual network connections

Cons

• Complexity
• Potential impact of a single component failure
• Hypervisor security issues
• Keeping virtual machine (VM) images current
• Virtual network connections

Virtualization Security Concerns

• Protecting the virtual fabric
• Patching off-line VM images
• Configuration Management
• Firewall configurations
• Complicating Audit and Forensics

You can get the work files from git://github.com/reductivelabs/velocity_puppet_workshop_2009.git, and the presentation’s available here.

I saw Luke’s Puppet talk last year at Velocity 2008, but am more ready to start uptaking some conf management back home.  Our UNIX admins use cfengine, and puppet is supposed to be a better-newer cfengine.  Now there’s also an (allegedly) better-newer one called chef I read about lately.  So this should be interesting in helping to orient me to the space.  At lunch, we sat with Luke and found out that Reductive just got their second round funding and were quite happy, though got nervous and prickly when there was too much discussion of whether they were all buying Teslas now.  Congrats Reductive!

Now, to work along, you git the bundle and use it with puppet.  Luke assumes we all have laptops, all have git installed on our laptops, and know how to sync his bundle of goodness down.  And have puppet or can quickly install it.  Bah.  I reckon I’ll just follow along.

You can get puppet support via IRC, or the puppet-users google group.

First we exercise “ralsh”, the resource abstraction layer shell, which can interact with resources like packages, hosts, and users.  Check em, add em, modify em.

You define abstraction packages.  Like “ssh means ssh on debian, openssh on solaris…”  It requires less redundancy of config than cfengine.

“puppet”  consists of several executables – puppet, ralsh, puppetd, puppetmasterd, and puppetca.

As an aside, cft is a neat config file snapshot thing in red hat.

Anyway, you should use puppet not ralsh directly.  Anyway the syntax is similar.  Here’s an example invocation:

puppet -e 'file { "/tmp/eh": ensure => present }'

There’s a file backup, or “bucket”, functionality when you change/delete files.

You make a repository and can either distribute it or run it all from a server.

There is reporting.

There’s a gepetto addon that helps you build a central repo.

A repo has (or should have) modules, which are basically functional groupings.  Modules have “code.”  The code can be a class definition.  init.pp is the top/special one.   There’s a modulepath setting for puppet.  Load the file, include the class, it runs all the stuff in the class.

It has “nodes” but he scoffs at them.  Put them in manifests/site.pp.  default, or hostname specific (can inherit default).   But you should use a different application, not puppet, to do this.

You have to be able to completely and correctly describe a task for puppet to do it.  This is a feature not a bug.

Puppet uses a client-server pull architecure.  You start a puppetmasterd on a server.  Use the SSH defaults because that’s complicated and will hose you eventually.  Then start a puppetd on a client and it’ll pull changes from the server.

This is disjointed.  Sorry about that.  The session is really just reading the slide equivalent of man pages while flipping back and forth to a command prompt to run basic examples.  I don’t feel like this session gave enough of an intro to puppet, it was just “launch into the man pages and then run individual commands, many of which he tells you to never do.”  I don’t feel like I’m a lot more informed on puppet than when I started, which makes me sad.  I’m not sure what the target audience for this is.  If it’s people totally new to puppet, like me, it starts in the weeds too much.  If it’s for someone whohas used puppet, it didn’t seem to have many pro tips or design considerations, it was basic command execution.  Anyway, he ran out of time and flipped through the last ten slides in as many seconds.  I’m out!