The backdoor leaves no traces of compromised hosts on the hard drive other than its modified httpd binary. This is where CloudPassage Halo’s File Integrity Monitoring Capabilities help ensure that critical system binaries, among other files, remain unchanged and in full working order. Should any person or entity try to change your web server binary, associated configuration files, or modify file metadata, the change will automatically be reported by Halo as it will fail to match the approved and expected baseline.

If for some reason you do not have an up-to-date baseline of your system, or you suspect that your only baselines include the malicious file in question, you may find value in using the VicRail.

The script, named in honor of Victory (‘Vic’) Rail who died of the first known reported case of the Hendra virus, is used to easily send the cryptographic checksum of a suspected compromised file to Virus Total, Shadowserver, and Team Cymru for comparison with other reported cases of known malware.

VicRail can be download from the CloudPassage GitHub page at https://github.com/cloudpassage/VicRail. To use the tool, the uirusu Ruby Gem is required and you also require a free public api key from virustotal.com that will allow you to compare the hash values of your suspect files against the Virus Total database.

Usage

# ruby vicrail.rb /path/to/file1 /path/to/file2 /path/to/file3 ... /path/to/file234

e.g.

$ ruby vicrail.rb apache2 eicar.com
==== VirusTotal - www.virustotal.com ====
apache2 - sha1
Hash identified in the database...
91e70da15f5d08793d2ab8258eece06331232959: Scanner: TotalDefense Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: MicroWorld-eScan Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: nProtect Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: CAT-QuickHeal Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: McAfee Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Malwarebytes Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: K7AntiVirus Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: K7GW Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: TheHacker Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: NANO-Antivirus Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: F-Prot Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Symantec Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Norman Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: ByteHero Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: TrendMicro-HouseCall Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Avast Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: eSafe Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: ClamAV Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Kaspersky Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: BitDefender Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Agnitum Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: ViRobot Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Sophos Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Comodo Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: F-Secure Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: DrWeb Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: VIPRE Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: AntiVir Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: TrendMicro Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: McAfee-GW-Edition Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Emsisoft Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Jiangmin Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Antiy-AVL Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Kingsoft Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Microsoft Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: SUPERAntiSpyware Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: GData Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Commtouch Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: AhnLab-V3 Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: VBA32 Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: PCTools Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: ESET-NOD32 Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Ikarus Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Fortinet Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: AVG Result: Nothing detected
91e70da15f5d08793d2ab8258eece06331232959: Scanner: Panda Result: Nothing detected

eicar.com - sha1
Hash identified in the database...
3395856ce81f2b7382dee72602f798b642f14140 1259633424 83

Using VicRail and CloudPassage Halo together, you can investigate potentially malicious files quickly and effectively. Halo can report on changed files and VicRail can generate the hash and send it off for comparison. Just another tool to add to your incident responder tool belt from your friends at CloudPassage.

Photo Credit: blakespot via Compfight cc

The problem

Systems that allow remote login present an ongoing problem: how can we tell if the user logging in is actually the person we expected?  If the only required authentication is a password, couldn’t that password be captured with a keylogger on that person’s laptop and used by an attacker at a different IP address?

One of the ways to detect that this may have happened is to look at the IP addresses from which the logins originated.  If I look at a particular user and see that they’ve logged in 30 times from IP address 5.6.7.8 and once from 4.4.6.6, that would raise a small flag for me.  If 5.6.7.8 was at one ISP and 4.4.6.6 was at a different ISP, the flag would go up a little higher.  If all but one of the attempts from 4.4.6.6 failed, it would move up even more.  And if they were in different countries – and I’m pretty sure that user hasn’t been recently traveling – I’d get on the phone with them right away.

Where Are They Now? (watn)

where-are-they-now.rb is a Ruby program that summarizes the logins and attempts the Halo Portal has seen from each user.  Here’s a sample report:

You can quickly see your users’ Portal login addresses.  Anne Smith has only logged in from 13.14.15.176, which is in dsl1.net according to DNS.  She’s logged in twice successfully in this time period, with no login failures.  Once I’ve confirmed that that is her address, I’ll add it to /etc/verified-client-ips and future reports will move this to the “Verified login IPs” column.

Jeremy Parker has done most of his logins (10 out of 11) from 3.5.5.102, also from dsl1.net, and I’ve already confirmed with him that this is his home dsl address.  He does have a login from the country of Wadiya, though, and I don’t remember him traveling anytime since the beginning of February.  When I hover my mouse over the 1.8.7.34 address, a hover box shows that the address was used on Feb 21st, 18:11 UCT.  That helps me narrow down my search to see why the account was used from an unexpected location.

The next line (jparker-yubikey) shows a secondary account for Jeremy with Yubikey authentication.  Since Jeremy has confirmed that 6.3.3.205 is one of his legitimate addresses, I’ve added it to /etc/verified-client-ips and now it shows up in the “Verified Login IPs” column.  Since I know that address changes frequently, I’ve added “6.3.3.0/24” to /etc/verified-client-ips as well; that brings 4 more addresses over to the “Verified CIDR blocks” column.  They should be individually checked at some point, but that’s sufficient verification for now.

The “Don the daemon” icon next to 7.6.5.109 means that’s a machine that’s currently managed by Halo, as you can see down at the bottom of the report.  Jeremy probably started a Remote Desktop Protocol (RDP) session to that machine to run a web browser as part of installing the daemon.  The icon gives me one more clue that this is probably a legitimate source address.

Using it yourself

The program and install instructions can be found at both  https://github.com/cloudpassage/cloudpassage_tools and http://www.stearns.org/watn/.  Once installed it can be run on demand or nightly from cron.  You can specify a starting date on the command line to only look at logins and attempts since that date, or leave that option off to see all login events.

I hope that this is useful to you too, and that each report can be reviewed long before you finish your first cup.

I won’t go into detail on setting up a free Hosted Chef account, but it’s not that hard. Sign up for a free account here. Here’s a link to their QuickStart Guide on their learnchef.opscode.com site which covers it a bit more. For this post I’ll use the same workstation I used in the workshop, but any Chef supported workstation should work.

1. Install chef on Ubuntu 12.04 Desktop as a chef-workstation.

This will leverage your free Hosted Chef-server account. In a terminal, run the following commands to install chef.

• curl -L https://www.opscode.com/chef/install.sh | sudo bash
• I had problems installing the next requirement. The fix, was to run “sudo su -” and to run:
• apt-get install g++ (may not be necessary/available on ubuntu server)
  • apt-get install build-essential
  • apt-get install libxslt-dev
  • apt-get install libxml2-dev
  • Also add “/opt/chef/embedded/bin:/opt/chef/bin” to your user and root’s $PATH environment
• gem install knife-windows
• Download the <org>-validator.pem, <your-name>.pem and knife.rb (part 2/Chef Repo in the learnchef “QuickStart Guide”)
• Download the .zip or clone the chef-repo using git from github. unzip, move and rename it to ~/chef-repo
• Move the .pem files and knife.rb into the local repo
• mv ~/Downloads/*.pem ~/chef-repo/.chef
• mv ~/Downloads/knife.rb ~/chef-repo/.chef
• To verify it’s setup correctly, run the command: knife client list
  • You should see something like: <org>-validator
  • In the screenshot below, you’ll also see the servers I managed with chef during the work

2. Download the powershell cookbook which we’ll be using as a Halo dependency.

• wget -O ~/chef-repo/cookbooks/powershell.zip https://github.com/opscode-cookbooks/powershell/archive/master.zip
• unzip ~/chef-repo/cookbooks/powershell.zip -d ~/chef-repo/cookbooks
• mv ~/chef-repo/cookbooks/powershell-master ~/chef-repo/cookbooks/powershell

3. Download the cloudpassage_windows cookbook which contains our recipe.

It’s a subdirectory within our cloudpassage_tools repo.

• wget -O ~/chef-repo/cloudpassage_tools.zip https://github.com/cloudpassage/cloudpassage_tools/archive/master.zip
• unzip ~/chef-repo/cloudpassage_tools.zip -d ~/chef-repo
• cp -r ~/chef-repo/cloudpassage_tools-master/chef/cloudpassage_windows/ ~/chef-repo/cookbooks/

4. Edit ~/chef-repo/cookbooks/cloudpassage_windows/attributes/default.rb to add your specific Halo account daemon-key, current Halo version, and serverGroup tag

• The Daemon-key is account specific and can be found under Settings > Site Administration > Daemon Settings

• The current Windows version is: cphalo-2.7.8-win64.exe
• The serverGroup tag will automatically move the Server into the associated serverGroup and apply all Cloud Firewall, Configuration Security, or Intrusion Detection policies assigned to the serverGroup. (For more information see: Dynamic Security)

The attributes.rb file should look something like this:

5. Upload the cookbooks to your chef-server account by running this command:

• knife cookbook upload -a

6. Spin up a Windows Server 2008 or 2012 instance.

The catch is that Windows Remote Management needs to be running and listening for connections. Knife’s bootstrap command offers two mechanisms to install chef and run recipes on the Windows server. It supports ssh or winrm.

For this example we’ll be using winrm. ChefConf 2013 provided EC2 instances, but here’s a preconfigured public instance I found: Windows_Server-2008-R2_SP1-English-64Bit-Base-WinRM-2012.04.11

• Even though this AMI has winRM enabled, Opscode recommends specific winrm settings. They have complete instructions and references to Microsoft KB articles here. Open a cmd.exe prompt as an Administrator and run the following commands:
• winrm quickconfig -q
• winrm set winrm/config/winrs @{MaxMemoryPerShellMB=”300″}
• winrm set winrm/config @{MaxTimeoutms=”1800000″}
• winrm set winrm/config/service @{AllowUnencrypted=”true”}
• winrm set winrm/config/service/auth @{Basic=”true”}
  • These settings are designed for development and test purposes only

7. Here’s the magic.

Bootstrap the running Windows server with Halo by passing in the run_list parameter and include both the powershell and cloudpassage_windows cookbooks.

• knife bootstrap windows winrm <your EC2 instance FQDN> -x <administrator user_name> -P “<user_passwd>” -r “powershell,cloudpassage_windows”

You should see some similar output which shows that the recipe was successfully deployed!

You will also see the new server in the portal dashboard. For this post, the server name is AMAZONA-P0RI2H1

Good luck and happy cooking!

Amazon EC2-Classic gained popularity because it was so easy to use (all you need to get started is a credit card), and allowed users the flexibility to spin up environments on a whim.  However, some customers felt more comfortable using Amazon’s VPC platform, which has more built-in network security control than EC2’s Security Groups. VPC allows you to create a logically private network – you can create a VM that only has a private IP.  This is a great feature to allow for the ability to mimic a typical datacenter network structure where you can segregate servers based on function (ie. Webservers, Application Servers and Database servers).  VPC also includes the ability to add rules for outbound traffic as part of a set of rules in Security Groups.  You can also create Network Access Control Lists as a second layer of defense. Network ACLs let you define “allow” and “deny” rules, where with Security Groups, a user can only define “allow” rules. The differences between Amazon EC2-Classic and Amazon EC2-VPC / Amazon VPC are outlined here.

Though VPC security features do offer more control over network access to cloud servers, network access control is not enough to call your cloud servers “secure”. In fact, Amazon outlines other considerations that customers must be aware of in its white paper describing security processes:

Moving IT infrastructure to AWS creates a shared responsibility model between the customer and AWS. This shared model can reduce your operational burden as AWS operates, manages, and controls the components from the host operating system and virtualization layer down to the physical security of the facilities in which the services operate. In turn, you assume responsibility and management of the guest operating system (including updates and security patches), other associated application software, as well as the configuration of the AWS-provided security group firewall… It is possible for you to enhance security and/or meet more stringent compliance requirements by leveraging technology such as host-based firewalls, host-based intrusion detection/prevention, and encryption.

– (AWS Overview of Security Processes – pdf)

Put simply, AWS will handle the security of the hypervisor down; this would include all the hardware and the datacenter.  You, as the Amazon user, are responsible for security from the Guest OS and up.

Awareness of the shared responsibility for security is the first step. Many people just assume that since they are buying a product from Amazon that security is built into the product. So where do you go from here? Since you have complete control over the Guest OS it would make sense that you add as many security measures in the VM.

Securing via host-based firewalls is a first step.  One of the biggest advantages of using host-based firewalls in the cloud is that the traffic can be logged (logs are unavailable in Security Groups), so you have the ability to see what IP and what type of traffic is being denied and/or accepted.  The solution you choose should also have the ability to manage IPs dynamically, as VPC’s public IPs may change. Being able to group your servers logically into areas like Web Servers, Application Servers, and Database Servers and having the ability to apply the appropriate host firewall rules automatically as a server starts up into a group ensures communication between servers.

IDS is the next big step in securing your VPC servers. You want to be automatically informed if any vital files or settings have been modified either accidentally or maliciously. This would include any applications you are using like Apache, MySQL, Postgres, etc. Watching ownership of files, directories and what processes are running is an important piece of your security solution. Once you have set a baseline of what your system should look like, any unexpected deviation should be considered a breach of security.

Moving up in the stack, you need to be able to handle User Access Management. Auditing which users accounts exist / have root access / have passwords and being able to add/remove accounts as required is a must. Monitoring accounts is a fundamental step in ensuring that VPC servers are secure and unauthorized users don’t have access to your systems.

As security holes are found in various applications, software scanning becomes a must have. Closing these holes as patches become available minimizes the risk that your servers will be compromised. This all starts with being informed that the vulnerability exists.

Another thing to consider is how much CPU the solution is using to perform its scans. In AWS you are paying per hour for your Virtual Machine, so the resources that your security solution consumes become a financial concern. Traditional security solutions that are designed for the datacenter can consume a huge chunk of the machine’s processing power; however, in the datacenter there is minimal cost to eating up the CPU. In AWS the dollar amount can add up quickly.

Lastly, but perhaps the most important point, is automation. With the self-service nature of the cloud you need to make sure that security is baked into the Virtual Machine so that as environments are being spun up you do not have to worry if the VM is susceptible to attacks.

Moving to the Cloud provides many advantages, but requires a shift in thinking when it comes to security. Security in the traditional datacenter where you typically push security measures out to the perimeter is no longer possible in the Cloud. As you can see AWS EC2-VPC provides customers with some great security features, but by themselves are not enough to protect your Cloud servers. You need to add security where you have broad control and that is at the Guest OS level. When you look for the solution to secure your VPC you need to ask these questions to make sure you’re taking advantage of the Cloud:

• Is the security automated?
• Will the security scale?
• Will the security solution be portable?
• Will the security solution secure the different layers of the Virtual Machine?

The Halo Event Connector enables you to pull security event logs from Halo into your Sumo Logic dashboard, including alerts from your configuration, file integrity, and software vulnerability scans. Halo can also deliver unprecedented visibility of your cloud servers, directly into your log management console. You can track server events such as your server rebooting, shutting down, changing IP addresses, and much more.

The purpose of the Halo event script is to retrieve event data from a CloudPassage Halo account and import it into Sumo Logic for indexing or processing. It is designed to execute repeatedly, keeping the Sumo Collector up-to-date with Halo events as time passes and new events occur.

Using the scripts and documentation posted on Github, you can quickly and easily add Halo as a “source” to your Sumo Collector, so events generated by Halo will feed into your log management system, giving you centralized, and more complete visibility across your server environment.

Halo Event Connector is free to use, and will work with any Halo subscription.  To get started integrating Halo events into Sumo Logic, make sure you have set up accounts for CloudPassage Halo and Sumo Logic.

Then, generate an API key in your CloudPassage Halo portal.  Once you have an API key, follow the steps provided in the Halo – Sumo Logic Documentation, using the scripts provided on Github. The documentation available with those files on GitHub walks you through the process of testing the Halo Event Connector script.

Once you have tested the script, you will then add the output as a “Source” by selecting “Script” in Sumo Logic (see below).

When you have finished adding the new data source that integrates the Halo Event Connector with Sumo Logic (as detailed in the .pdf documentation), you will be taken back to the “Collectors” tab where the newly added Script source will be listed.

Once the Connector runs successfully and is importing event data into Sumo Logic, you will see Halo events such as the following appear in your Sumo Logic searches:

Try it out today – we are eager to hear your feedback! We hope that integrating these two tools make your server security automation even more powerful.

After reading Ken Pryor’s excellent NBDServer blog post on Wednesday, April 10th, and while preparing for my SOURCE Boston 2013 talk entitled Facilitating Fluffy Forensics, I found myself wondering if the tool might help with investigations in public cloud environments.

In fact, I contacted Ken via a Direct Message on Twitter to share my idea (and thank him for the inspiration):

I also had a conversation with Windows forensics expert, Harlan Carvey, about his Forensic Scanner application and potentially using NBDServer in place of the commercial acquisition tool, F-Response, to mount the drive.

Of course, a native Windows nbd-client does not exist so I put my efforts into detailing a configuration that would allow a nbd-client server, running on Linux, to act as a relay for Read-Only file system access. For the sake of brevity, and due to the detailed nature of this configuration, CloudPassage Halo policies to secure the documented process will be discussed in a later blog post.

Network Diagram

The Configuration

Step 1) Prepare your nbd-client server

For the purpose of this example we used an Ubuntu Server 12.10 32-bit micro instance image to act as the nbd-client instance on Amazon EC2. After launch, the first thing you should do is update your server’s packages:

ubuntu@ip-10-50-201-39:~# sudo apt-get update && sudo apt-get upgrade –y

You must install a few packages to satisfy the undocumented dependencies:

ubuntu@ip-10-50-201-39:~# sudo apt-get install libglib2.0-dev pkg-config build-essential -y

And then downloaded and extracted the nbd-3.3.tar.gz from SourceForge.net:

ubuntu@ip-10-50-201-39:~# wget http://sourceforge.net/projects/nbd/files/nbd/3.3/nbd-3.3.tar.gz
ubuntu@ip-10-50-201-39:~# tar zxvf nbd-3.3.tar.gz ubuntu@ip-10-50-201-39:~# cd nbd-3.3

You must then configure, make, and make install the application:

ubuntu@ip-10-50-201-39:~/nbd-3.3# sudo ./configure
ubuntu@ip-10-50-201-39:~/nbd-3.3# sudo make && sudo make install

Step 2) Preparing your Windows nbd-server target

Using a Windows Server 2008 R2 system on Amazon EC2, I downloaded and installed the NBDServer from Jeff Bryner’s GitHub repository. You can either use the browser to download the file or use the following quick PowerShell command:

PS C:UsersAdministratorDownloads> Invoke-WebRequest https://github.com/jeffbryner/NBDServer/archive/master.zip -OutFile master.zip

Change to the NBDServer-master directory and launch the NBDServer:

PS C:UsersAdministratorDownloads> cd NBDServer-master
PS C:UsersAdministratorDownloadsNBDServer-masterNBDServer-master> .NBDServer.exe -c 10.50.201.39 -f .PHYSICALDRIVE0 -n0

This instructs the server to connect to the private client IP (10.50.201.39) and serve up the first partition on drive 0.

Step 3) Ensuring communications

By default, the Windows Firewall will block communications between the nbd-client server and the target running the nbd-server. Using Halo, you can adjust your firewall policy to allow communications between your nbd-client and nbd-server instances via their private IP addresses. A Windows firewall policy must be created for the nbd-server target that:

• Only allows authorized incident responders to access the server,
• Allows nbd-client connections, on 60000/tcp, from the private IP address of your nbd-client system, and
• Allows outbound connections to GitHub, on port 443, for the downloading of the NBDServer executable.

A Linux firewall policy must also be created for the nbd-client server that:

• Only allows authorized authenticated incident responders to access the server,
• Allows outbound nbd-client connections, on 60000/tcp, from the private IP address of your nbd-client system to the nbd-server, and
• Allows outbound connections to Ubuntu repositories, on port 443, for the downloading of updates.

Note, a full Halo policy for performing the above firewall configurations will be discussed in a follow-on blog post.

Finally, the Amazon EC2 Security Groups must be configured to allow nbd communications between servers via TCP port 60000. Even though Amazon utilizes 10.0.0.0/8 for its private addressing scheme, each network is chopped up into /24 networks. As such, it’s impossible to guarantee that your nbd-client and nbd-server target will be in the same network – unless of course you launch everything within the same VPC. Note: Samba rules will also need to be added to the local firewall policy and the EC2 Security Groups if the nbd-client is going to be relaying the Read-Only share to additional servers.

Step 4) Connecting the client

To connect the nbd-client to the nbd-server, you must first load the nbd module into the kernel:

ubuntu@ip-10-50-201-39:~/nbd-3.3# sudo modprobe nbd

You can then launch the client:

ubuntu@ip-10-50-201-39:~/nbd-3.3# sudo ./nbd-client 10.198.86.84 60000 /dev/nbd0

Where 10.198.86.84 is the target Windows Server IP address, 60000 is the default port for the nbd-client to connect on, and /dev/nbd0 is the first nbd block device to connect the remote target as.

Upon successful connection you should see something similar to the following lines within your Linux terminal:

Negotiation: ..size = 30718MB bs=1024, sz=32210157568 bytes

And a connection message within your Windows terminal:

[+] Connection made with: 10.50.201.39

Step 5) Mounting the target drive locally

You must first create a directory against which you will mount the target drive:

ubuntu@ip-10-50-201-39:~/# mkdir /mnt/target10-198-86-84
ubuntu@ip-10-50-201-39:~/# sudo mount -t ntfs-3g -o ro,show_sys_files,streams_interface=windows /dev/nbd0 /mnt/target10-198-86-84

You might notice that the directory was mounted with the ro setting (for Read-Only) configured. This actually doesn’t matter as the target’s drive is shared as Read-Only. You can mount using the rw settings if you wish, but it will still not allow you to modify the contents or structure of the target server’s file system.

Step 6) Sharing the target’s drive

You may want to share the target’s Read-Only drive to additional systems. This is useful when you need to run Windows-based tools, such as Forensic Scanner or a collection of malware scanners, against the target system. Perhaps the easiest way to share the target’s drive is to use Samba. The first thing you must do is install the Samba packages on your nbd-client server:

ubuntu@ip-10-50-201-39:~/# sudo apt-get install samba –y

Once installed, edit the /etc/samba/smb.conf file. Ensure that the security = user setting is enabled (i.e. the hash character is removed) and that the following is added to the bottom of the file:

[share]
comment = Forensic Share
path = /mnt/target10-198-86-84
browsable = yes
guest ok = yes
read only = yes
create mask = 0755

Save and exit the file. In order for the changes to work, you must restart the smbd and nmbd daemons:

ubuntu@ip-10-50-201-39:~/etc/samba# sudo restart smbd && sudo restart nmbd

Conclusion

The steps mentioned above help you configure the servers and the NBDServer client/server application. Using nbd-client, nbd-server, Samba, and Amazon EC2 micro instances, you should now be able to mount the share from your server without the possibility of tainting the file system or any forensic artifacts of the nbd-server target. In my next blog post I’ll expand on using NBDServer, cloud server instances, and CloudPassage Halo to facilitate secure cloud forensics. Topics that will be discussed include how to:

• Automate the isolation of the target server,
• Restrict remote access to incident responders,
• Dynamically restrict, and selectively allow, inter-server communications,
• Generate logs that prove file system integrity of all servers and applications throughout the investigation, and
• Validate the configuration settings of required services (e.g. Samba).

References:

• Windows Network Block Device Server by Jeff Bryner – https://github.com/jeffbryner/NBDServer/
• Ken Pryor’s April 2013 Digital Forensics Blog post on NBDServer – http://digiforensics.blogspot.com/2013/04/nbdserver.html
• Harlan Carvey’s October 2012 Windows Incident Response blog post on Forensic Scanner – http://windowsir.blogspot.com/2012/10/forensic-scanner.html

Something’s not right.

Identical systems

With servers built from the same process or server image and maintained in parallel, there are very few legitimate reasons why they should differ from each other.  At the mundane end of the scale, another admin installed a package or started up a daemon to do some troubleshooting.  Other reasons include problems with the build or patching process or a server break-in.

We can use any differences as starting points for investigation; just why is this extra package installed on 2 of these servers, but not the rest?  To get to that point, though, we need a way to identify the differences first.

OOTT (One Of These Things is not like the others…)

While it sounds like the noise made by a cartoon character, OOTT is a Ruby program that collects information about your Halo-managed systems and presents HTML views of each group of machines (and one final report summarizing all of them).  For a simple aspect of a system (such as “Account tjames exists”), it shows you how many systems have it:

To find out which servers have the asmith and tjames accounts, click on the number of servers and you get a list of server names as well:

When the servers can have different observed values, such as installed package versions, we summarize those as well:

The report also shows any Configuration policy rules and checks that fail as well:

The beauty here is that the report automatically summarizes all of the Configuration Policy rules and checks that need attention.  Given the wide range of Configuration checks one can perform, this turns out to be a remarkably rich source of information.

And finally, I’m glad to report that these 2 servers do match in some of their aspects:

The connecting IP parent domain can quickly identify machines that are running in a network you hadn’t expected.

How does this help?

The real benefit of a report like this shows up when you manage more than a few systems.  OOTT does the work of summarizing a large number of server aspects; you can look at a report like this and ask “Why do 10 of my main servers have postfix 2.5.4 installed, and 2 of them have postfix 2.3.1?”  If you double the number of machines in a group, the report stays just as easy to review.  This means less manual effort to identify outliers.

The pages are organized from highest priority to lowest, so you know where to start.  The top half of the report holds the server aspects where they disagree.  The bottom half holds the ones where they all agree.  Inside both of those we start with critical+bad, then bad, indeterminate, and finally good.

As you work your way down the list, finding and fixing discrepancies, you can ask the Portal to run a new scan on the systems you’ve updated, then rerun the OOTT report.  The aspects that have been fixed will fall off so you can focus on the next issues.

Giving it a try

OOTT and its associated readme and library are available at both the CloudPassage Toolbox on Github and http://www.stearns.org/halo-api/ .  The install takes just a few minutes and requires Ruby and a few Ruby support libraries.

We hope it’s useful to you.  Let us know if you have suggestions for improving it!

While we normally think of our computers as having a single router leading to the Internet, it’s possible to have 2 or more routers that would be willing to carry our packets for redundancy.  In that case, we’d create extra routing table entries, one per gateway.  The operating system on the machine would take responsibility for deciding which one to use at any given time.

If an attacker wanted to read our network traffic, she might also create a second default gateway routing table entry, sending our traffic through a machine she controls, say 172.16.1.5.  That gives her a chance to sniff and inspect the traffic coming from this machine.  Unlike the redundancy example above, this malicious default gateway entry is a silent way for an attacker to gain intelligence about us from the raw packets.

How would we detect that a second default route had been created?

Since there are 4 billion IPv4 IP addresses alone, we can’t put in a rule for each one saying

/proc/net/route Does not contain “^eth0s+00000000s+00000000s.*s00000000s”

/proc/net/route Does not contain “^eth0s+00000000s+01000000s.*s00000000s”

/proc/net/route Does not contain “^eth0s+00000000s+02000000s.*s00000000s”

/proc/net/route Does not contain “^eth0s+00000000s+03000000s.*s00000000s”

…

We have to get sneakier.  :-)

The alert I want is this: tell me if there’s a default route line (“^eth0s+00000000s+{something}s.*s00000000s”) that has something other than 010110AC in the default gateway column.  Instead of saying “I need the default gateway to be 010110AC”, I want to be warned if those characters are anything except 010110AC.

Our Search Expressions have something that looks promising; negating sets of characters with the caret symbol.  If I use paired brackets (“[{characters}]”), I match any character in between the brackets.  But if I place a caret right after the left square bracket, that says match any character except the ones in this set.  For example, “[0-9ABCDEF]” will match any digit or letter between uppercase A and uppercase F.  But if I put a caret right after the left square bracket: “[^0-9ABCDEF]”, that matches anything that isn’t an uppercase hexadecimal character ( %, &, j, d, H, L, etc.).

OK, so let’s try negating each character in the default gateway and see if that works:

 #Does this work?

/proc/net/route Does not contain “^eth0s+00000000s+[^0][^1][^0][^1][^1][^0][^A][^C]s.*s00000000s”

To test this, let’s assume we have a second default gateway line in /proc/net/route:

# cat /proc/net/route

Iface   Destination     Gateway         Flags   RefCnt  Use     Metric  Mask   MTU      Window  IRTT

eth0    00000000        010110AC        0003    0       0       0       000000000       0       0

eth0    00000000        FEDCBA09        0003    0       0       0       000000000       0       0

eth0    000110AC        00000000        0001    0       0       0       00FFFFFF0       0       0

Our second bogus default gateway is FEDCBA09.  That matches each of the above sets (F is a character other than 0, E is a character other than 1, D is a character other than 0, C is a character other than 1, etc.).  So we have a line that matches the Search Expression.  Since we specifically didn’t want any lines that match, that will set off an alert and the planet is safe again.

Not so fast.

There’s a problem there; in our example, every single character in FEDCBA09 was different from its corresponding character in 010110AC, and since each one had to be different for the alert to go off, the alert went off.  What happens if the second bogus default gateway is different but has some characters that do match?

For simplicity, let’s say that we get a second default route through 172.16.1.255.  The reverse hex of that is FF0110AC.  Now see if that matches “^eth0s+00000000s+[^0][^1][^0][^1][^1][^0][^A][^C]s.*s00000000s”.

F is not 0, so we match there.  The second F is not 1, so we match there.  0 in our default gateway does equal 0, so we don’t match “[^0]”.  Oops, FF0110AC does not match [^0][^1][^0][^1][^1][^0][^A][^C], and since we only alert if there is a match, we won’t get an alert.

Is your head spinning like mine is?  Trust me; I had to both stop and think about this very carefully (especially because of the double negatives) and double check my logic with colleagues Apurva and Christian to make sure I got this right (thanks guys!).

So there’s the problem; we alert in the first case because all 8 characters we pulled (“FEDCBA09”) were different from their counterparts in the search expression ([^0][^1][^0][^1][^1][^0][^A][^C]).  But we didn’t alert in the second case because some of the characters in “FF0110AC” were different from that search expression and some were the same as their counterparts.  So this Search Expression doesn’t alert on every gateway except 010110AC.  :-(

—–

OK, back to the drawing board.  What are we really trying to check anyways?

What we really want to know is this:  “Alert me if any of the hex characters of the default gateway are different than the ones I expected (“010110AC”).  We already tried checking them all at once and crashed and burned.  How about checking them one at a time?

Let’s put in the following 8 rules, each of which checks one character at a time:

/proc/net/route Does not contain “^eth0s+00000000s+[^0]…….s.*s00000000s”

/proc/net/route Does not contain “^eth0s+00000000s+.[^1]……s.*s00000000s”

/proc/net/route Does not contain “^eth0s+00000000s+..[^0]…..s.*s00000000s”

/proc/net/route Does not contain “^eth0s+00000000s+…[^1]….s.*s00000000s”

/proc/net/route Does not contain “^eth0s+00000000s+….[^1]…s.*s00000000s”

/proc/net/route Does not contain “^eth0s+00000000s+…..[^0]..s.*s00000000s”

/proc/net/route Does not contain “^eth0s+00000000s+……[^A].s.*s00000000s”

/proc/net/route Does not contain “^eth0s+00000000s+…….[^C]s.*s00000000s”

Notice how we use periods as placeholder; each one stands in for a character, but I don’t really care what that character is.  That allows us to focus on one character at at time; the first rule checks the first character of the gateway address, the second rule focuses on the second character, etc.

From the top, these rules will respectively alert me if:

- The first character of the default gateway is anything other than a 0

- The second character of the default gateway is anything other than a 1

- The third character of the default gateway is anything other than a 0

- The fourth character of the default gateway is anything other than a 1

…

- The eighth character of the default gateway is anything other than a C

So how does that do with our three test cases?

1) If we compare our legitimate router (“010110AC”) to each of those expressions, it doesn’t match any of them because each character matches its counterpart in the rules.  That means we won’t get any alerts, and that’s a good thing.

2) When we compare the default gateway that has completely different characters (“FEDCBA09”), rule number 1 alerts because “F” is in the set [^0] .  Rule number 2 alerts because “E” is in the set [^1] .  Rule number three alerts because “D” is in the set [^0] , and so on.  Because all 8 characters are different from their legitimate counterparts, we’ll end up with 8 alerts.  It’s arguably noisy, but it will certainly get the point across.

3) Now we compare the final bogus gateway (“FF0110AC”) to our 8 rules.  The first rule will alert because “F” is in [^0] .  The second rule alerts because the second “F” is in [^1] .  The third rule does not alert because 0 is not in [^0] .  The fourth rule does not alert because 1 is not in [^1] .  In fact, all the rules from the third through the eighth do not alert because the characters in the second bogus gateway do match their counterparts in the rules.  We end up with just 2 alerts, one for each character that does not match the legitimate gateway.  And that’s enough to show us there’s a bogus gateway line.

You might not be interested in checking /proc/net/route, but the same techniques can be used to both look for a correct line in a file and look for any other incorrect lines in a file.

A big part of my job here at CloudPassage, in addition to educating people about Halo, is to try and help everyone understand that security is far less effective as an afterthought than if transparently folded in as part of the continuous integration process.

Have you ever run so fast that you lose control, trip yourself up, and fall flat on your face? Not only is it embarrassing but…it hurts quite a bit. The same can be said with regards to deploying applications and servers at a breakneck pace without considering security.

Sure, you’ll get there fast but eventually, you’ll trip up and fall on your face. In this new cloud world, it may not be a single fall but rather a whole slew of falls if all of your cloud servers were deployed in the same fashion. This brings an entirely new meaning to the term fail-fast, doesn’t it?

The house that cloud built

A lot of people use the analogy of building a house on an unstable foundation when discussing the security of software and servers. That analogy is fine when we’re in control of the physical infrastructure but when we’re talking about deploying servers and applications in IaaS (Infrastructure-as-a-Service) cloud environments, the foundation (architecture) is poured and the main floor (hypervisor) is already built. We have the ability to build on the existing architecture (e.g. walls, fixtures, decorations) but cannot alter the underlying structure (e.g. gas, water, load bearing walls).

If we’re stuck building within the constraints of the foundation, why not create easily reproducible server and application images? If we borrow a page from how mobile homes are architected, we can design rugged and highly portable server images that can be reused time and time again.

A mobile home is built to be, well, mobile.

Mobile homes, also known as trailers or caravans (as our friends across the pond call them), have all of the amenities – well, most of the amenities — that one might need to enjoy indoor living. Perhaps their biggest advantage, however, is their ability to be moved from campsite to campsite with as little effort as possible. You can interpret this analogy in a number of different ways. As a server’s ability to be moved between hypervisors, cloud providers or even between public, private and hybrid deployments. Another way to think about it is in terms of cloning servers and spinning up new instances based on the trusted ‘gold standard’ image.

Putting the effort into creating a secure server baseline enables organizations to deploy trusted servers at a moments notice. The bottom line is that we should all be creating servers with a minimal attack surface area exposed to potential attackers or other malicious entities.

Check yourself, before you wreck yourself

This is certainly not a new concept. The basic strategies of attack surface reduction are to reduce the amount of code running, reduce entry points available to untrusted users, and eliminate services requested by relatively few users (source: http://en.wikipedia.org/wiki/Attack_surface). Another way to think about it is to consider why football players wear a helmet, shoulder pads, gloves, shoes, thigh pads, knee pads, neck rolls, elbow pads, mouth guards, hip pads, tailbone pads, rib pads, and other equipment? That’s right…to reduce their attack surface area. A football player knows that if they don’t wear at least the basic padding, they leave themselves open to attack from other players and even exploitation (e.g. targeting a weak knee, exposed hand or unprotected thigh muscle).

Here are 5 easy steps to help organizations start on their creation of a secure server baseline:

1. Disable unnecessary services
2. Remove unneeded packages
3. Restrict access to sensitive files & directories
4. Remove insecure/default configurations
5. Allow administrative access ONLY from trusted servers/clients

You might notice that the above suggestions are all configuration-related items and not related to the installation of security or other mitigating tools. That’s because I believe that systems administrators and DevOps teams need to get back to protecting their servers by first eliminating the potential avenues for exploitation or compromise before even starting to think about third-party tools to protect their servers. That’s not to say that security tools aren’t required. All I’m saying is that you’ll have a far easier time securing your servers if you eliminate services and configurations that don’t explicitly map to the operational requirements of your server.

The chat went very well, with a lot of good conversation! (So good, in fact, that 30 minutes seemed way too short.  We may in the future extend the chat to an hour.)  We plan on having similar chats using the #cloudsec hashtag every month or so – hope you can join in the next one!

The general attitude towards the PCI Cloud SIG standards was that they are a good start, but not the end-all of what it takes to secure a cloud environment.  Many people highlighted that compliance does not necessarily equal security, and didn’t agree with the premise that the standards would do much to remove roadblocks for companies hesitant to move processes into the cloud. John Strand captured the general sentiment pretty concisely:

Good security is good security regardless of PCI or anyone else #CloudSec

— strandjs (@strandjs) March 5, 2013

We’ve got some highlights down below, but you can search #cloudsec on twitter and see the whole stream.

Question 1: What do you think about the new PCI Cloud SIG Guidance?

andrewsmhay, sec_prof, and johnlkinsella said that they see the new guidance as a good start, but expressed that it doesn’t cover all of the requirements; selenakyle says the principles of good security are larger than the specific architecture:

@andrewsmhay clarifying evaluation/audit criteria always useful but principles of good security should be omni-architecture #pci #cloudsec

— selenakyle(@selenakyle) March 5, 2013

ken5m1th said that the new guidance does not include any new info for those that already know PCI.

Others like jaysonstreet and jack_daniel pointed out that more compliance policies don’t necessarily make the world a safer place.

My fear about PCI "guidance" is that it is years behind, early adopters have already figured it out and are left to react #cloudsec

— Jack Daniel (@jack_daniel) March 5, 2013

...while the late adopters will think the PCI cloud guides are <shudders> best practices #CloudSec

— Jack Daniel (@jack_daniel) March 5, 2013

carsonsweet, johnlkinsella and iiamit pointed out that even though compliance and security are not equivalent, compliance guidance can help with implementing security on the practical side.

Also, we all know checklists!=security. You might not see the value of both, but both have their purposes #CloudSec

— John Kinsella (@johnlkinsella) March 5, 2013

Question 2: Will the guidance remove PCI-related roadblocks for cloud adoption?

sec_prof and jack_daniel expressed little confidence that the new guidelines will remove obstacles for wider cloud adoption, while Shpantzer expressed concern that it would result in some vendors will make unwarranted “cloud ready” claims. On the other hand, andrewsmhay suggested that some organizations have been waiting for something like this to move forward.

A2: No. I think that it will crack the door open, but until larger adoption happens, folks will still use it an an excuse #CloudSec

— Phil Cox (@sec_prof) March 5, 2013

Question 3: What would you change, if anything about the guidance?

andrewsmhay pointed out that more collaboration w/ the Cloud Security Alliance could raise the bar, and sec_prof would have liked to see more specifics in the guidelines, but acknowledged the challenges of being both specific and vendor-neutral.

@cloudpassage Well, for one I’d collaborate more with the @cloudsa to merge ideas and raise the bar #CloudSec

— Andrew Hay (@andrewsmhay) March 5, 2013