Stephen Reesehttp://www.rsreese.comMy Dark Cloud Kung Fuen-USMon, 20 May 2013 10:45:48 PDThttp://wordpress.org/?v=3.5.1hourly130.092346-81.60274rsreesehttp://feedburner.google.comPassive DNS collection and analysis using YaF and Mediatorhttp://feedproxy.google.com/~r/rsreese/~3/8j_C3k672Eg/securityanalysisnetflowpassive dnsStephen ReeseMon, 20 May 2013 04:12:14 PDThttp://www.rsreese.com/?p=1289Passive DNS is a useful tool for any analysts teams toolbox, I have noted several public sensors here but they only see data (queries and responses) that transverse their sensors. I have been working on setting up passive DNS using Yet another Flowmeter (YaF) and Mediator (YaF to MySQL) to fill the gap where third-party sensors may not be providing the coverage I would like. Passive DNS can provide tremendous insight and analytics upon DNS queries that users and/or malware may be performing. A few items of interest:

  • Hostnames that have a large number of IP addresses associated with them in a short time period and they have only been visited by very few hosts host on the network.
  • Tertiary name usage associated with a specific domain?
  • When was the domain first resolved on the network and further, how often is it being resolved and by whom?
  • A recently accessed/registered domain with short time to live (TTL’s) often associated with new IP addresses may indicate malicious activity, or a CDN.
  • Queries for TLD’s that you typically do not interact with may be worth looking into.
  • Users using non-approved DNS servers

Passive DNS may be also helpful in tracking infections using Fast-flux which make blocking the C2 difficult as the attackers will create algorithms to rotate the IP addresses and even the hostnames in the case of double-flux. (TorPig) The list goes on but in a nutshell, I wanted to be able to perform this activity without having to rely on having all of the DNS server logs in a centralized location, especially since users may reconfigure their DNS settings to use non-approved servers, e.g. BYOD.

This entry demonstrates how to build and setup YaF and Mediator both of which are available from the CERT NetSA site and should be considered complementary to the documentation the NetSA team have already provided for each of the respective tools. This setup was tested on CentOS 6.4 but most Linux distributions should work fine.

1. Have site reconfigure interfaces on all hosts. eth0 should be management interface and eth1 should be the tap OR whatever makes sense, this need to happen every time the host comes up, i.e.

sudo ifconfig eth1 up promisc

2. Ensure development libraries/dependencies are installed. Some may require enabling the optional software channel

sudo yum install glib2-devel lzo  gcc-c++ libpcap-devel pcre-devel

3. Install libfixbuf

cd libfixbuf-1.3.0
./configure
make
sudo make install

4. Install YaF

cd yaf-2.3.3
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
./configure --with-libpcap --enable-applabel --enable-plugins
make
sudo make install

5. Edit ld

sudo echo "/usr/local/lib" >> /etc/ld.so.conf
sudo /sbin/ldconfig
sudo /sbin/ldconfig -v | grep libzmq # should rebuild the cache including zmq too.

OR

export PATH=$PATH:/usr/local/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

6. Configure cmake

cd cmake-2.8.10.2
./configure
gmake

6. Optionally, configure YaF to File output for testing purposes.

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
cd yaf_file_mediator-1.1.0/
./configure
../cmake-2.8.10.2/bin/cmake .
make

7. Configure YaF to MySQL

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
cd yaf_silk_mysql_mediator-1.4.0
../cmake-2.8.10.2/bin/cmake .
./configure --with-mysql
make

Next, populate create a database and respective tables:

./yafMySQL -o localhost -n username -p password -d eflows

8. Setup YaF to start capturing. Here we are only capture DNS traffic and rotating the files written to disk after 5 minutes. Originally set to 10 minutes but yaf_silk_mysql_mediator would segmentation fault because MySQL would close the connection before all of the data would insert. We have a continuous method that works a little better which we should a little later. We lock the file so that another process cannot take the file that is currently being written to.

sudo /usr/local/bin/yaf --live pcap --in eth1 --out /data/ipfix/ --rotate 600 --filter="port 53" --applabel --applabel-rules=/usr/local/etc/yafApplabelRules.conf --max-payload=1000 --plugin-name=/usr/local/lib/yaf/dpacketplugin.la --plugin-opts="53" --lock --become-user=nobody --become-group=nobody &

9. Testing the output of a YaF

yaf_file_mediator-1.1.0/yaf_file_mediator --input /data/ipfix/filename.yaf --output test.txt

After a few minutes, you should be able to parse the filename.yaf that was first written (in this case 5 minutes). The contents of test.txt should be similar to the following:

-------------------------------
Template ID is 45841
Application Label: 53
Source IP: 192.168.0.5
Destination IP: 8.8.8.8
Source Port: 53855
Dest Port: 53
Flow Attributes: 1
Rev Flow Attributes: 0
flowStartTime: 2013-04-24 23:53:43
flowEndTime: 2013-04-24 23:58:02
flowEndReason: 1
Protocol: 17
Octet Total Count: 120
Rev Octet count: 244
Packet Total Count: 2
Rev Packet Total Count: 2
DNS ID: 32852 Type: 28 RR Section: 0 TTL: 0 Query: www.google.com.
DNS ID: 32852 Type: 28 RR Section: 1 TTL: 204 RRName: www.google.com. AAAA: 2607:f8b0:400c:0c04::0069

-------------------------------
Template ID is 45841
Application Label: 53
Source IP: 192.168.0.5
Destination IP: 8.8.8.8
Source Port: 50845
Dest Port: 53
flowStartTime: 2013-04-24 23:58:02
flowEndTime: 2013-04-24 23:58:02
flowEndReason: 1
Protocol: 17
Octet Total Count: 60
Rev Octet count: 156
Packet Total Count: 1
Rev Packet Total Count: 1
DNS ID: 21141 Type: 1 RR Section: 0 TTL: 0 Query: www.google.com.
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.103
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.99
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.105
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.104
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.106
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.147

10. After you have confirmed that your YaF entries contain records, add a little automation. This will scoop up the files in the directory where the YaF files are being written, place them in the MySQL DBMS and delete the file. Note, if you start seeing “Segmentation Fault” then MySQL may be closing the connection before all of the records from the YaF file could be written to the DBMS. You can try modifying MySQL parameters or reduce the the size of YaF files being written to disk in order to try mitigating this symptom if it occurs in your environment.

for i in $( ls /data/ipfix/*.yaf ); do /home/user/silk-installs/yaf_silk_mysql_mediator-1.4.0/yaf_silk_mysql_mediator --in-file $i --mysql-host localhost --name username --pass password --database eflows && sudo rm $i; done

Here is our first query, lets see who has recently made requests for www.google.com.

mysql> SELECT rrname,rrval,srcip4,dstip4,flowStartMilliseconds FROM dns d, flows f WHERE f.id = d.id AND rrname LIKE "www.google.com." GROUP by rrval ORDER BY f.id DESC LIMIT 50;
+-----------------+---------------------------+------------+-----------+-----------------------+
| rrname          | rrval                     | srcip4     | dstip4    | flowStartMilliseconds |
+-----------------+---------------------------+------------+-----------+-----------------------+
| www.google.com. | 2001:4860:4001:0802::1012 | 3232235525 | 134744072 | 2013-05-03 17:47:24   |
| www.google.com. | 2001:4860:4001:0801::1014 | 3232235525 | 134744072 | 2013-05-03 15:35:32   |
| www.google.com. | 2001:4860:4001:0802::1014 | 3232235525 | 134744072 | 2013-05-03 11:28:42   |
| www.google.com. | 2001:4860:4001:0801::1010 | 3232235525 | 134744072 | 2013-05-02 16:48:31   |
| www.google.com. | 2001:4860:4001:0802::1011 | 3232235525 | 134744072 | 2013-05-02 13:33:57   |
| www.google.com. | 2001:4860:4001:0803::1010 | 3232235525 | 134744072 | 2013-05-02 12:01:56   |
| www.google.com. | 2607:f8b0:4004:0801::1012 | 3232235525 | 134744072 | 2013-05-01 21:36:55   |
| www.google.com. | 2001:4860:4001:0802::1010 | 3232235525 | 134744072 | 2013-05-01 12:44:52   |
| www.google.com. | 74.125.239.80             | 3232235525 | 134744072 | 2013-05-01 10:45:04   |
| www.google.com. | 74.125.239.83             | 3232235525 | 134744072 | 2013-05-01 10:45:04   |
| www.google.com. | 74.125.239.82             | 3232235525 | 134744072 | 2013-05-01 10:45:04   |
| www.google.com. | 74.125.239.81             | 3232235525 | 134744072 | 2013-05-01 10:45:04   |
| www.google.com. | 74.125.239.84             | 3232235525 | 134744072 | 2013-05-01 10:45:04   |
| www.google.com. | 2607:f8b0:4004:0802::1010 | 3232235525 | 134744072 | 2013-04-29 19:54:00   |
| www.google.com. | 2607:f8b0:4005:0802::1010 | 3232235525 | 134744072 | 2013-04-28 15:52:00   |
| www.google.com. | 2607:f8b0:4004:0803::1013 | 3232235525 | 134744072 | 2013-04-28 15:05:53   |
| www.google.com. | 2607:f8b0:4005:0802::1011 | 3232235525 | 134744072 | 2013-04-27 14:45:35   |
| www.google.com. | 2607:f8b0:4004:0801::1013 | 3232235525 | 134744072 | 2013-04-26 18:53:45   |
| www.google.com. | 2607:f8b0:4005:0802::1012 | 3232235525 | 134744072 | 2013-04-26 13:55:51   |
| www.google.com. | 2607:f8b0:4005:0802::1013 | 3232235525 | 134744072 | 2013-04-26 12:35:18   |
| www.google.com. | 74.125.239.145            | 3232235525 | 134744072 | 2013-04-26 12:03:10   |
| www.google.com. | 74.125.239.148            | 3232235525 | 134744072 | 2013-04-26 12:03:10   |
| www.google.com. | 74.125.239.146            | 3232235525 | 134744072 | 2013-04-26 12:03:10   |
| www.google.com. | 74.125.239.147            | 3232235525 | 134744072 | 2013-04-26 12:03:10   |
| www.google.com. | 74.125.239.144            | 3232235525 | 134744072 | 2013-04-26 12:03:10   |
| www.google.com. | 2607:f8b0:4005:0802::1014 | 3232235525 | 134744072 | 2013-04-26 11:31:59   |
| www.google.com. | 74.125.228.112            | 3232235525 | 134744072 | 2013-04-25 16:25:39   |
| www.google.com. | 74.125.228.114            | 3232235525 | 134744072 | 2013-04-25 16:25:39   |
| www.google.com. | 74.125.228.113            | 3232235525 | 134744072 | 2013-04-25 16:25:39   |
| www.google.com. | 74.125.228.115            | 3232235525 | 134744072 | 2013-04-25 16:25:39   |
| www.google.com. | 74.125.228.116            | 3232235525 | 134744072 | 2013-04-25 16:25:39   |
| www.google.com. | 2607:f8b0:4004:0802::1012 | 3232235525 | 134744072 | 2013-04-25 11:29:45   |
| www.google.com. | 2607:f8b0:4004:0803::1014 | 3232235525 | 134744072 | 2013-04-24 20:33:42   |
| www.google.com. | 2607:f8b0:400e:0c04::006a | 3232235525 | 134744072 | 2013-04-24 18:04:19   |
| www.google.com. | 2607:f8b0:400e:0c02::006a | 3232235525 | 134744072 | 2013-04-24 15:26:22   |
| www.google.com. | 74.125.228.20             | 3232235525 | 134744072 | 2013-04-24 12:05:43   |
| www.google.com. | 74.125.228.16             | 3232235525 | 134744072 | 2013-04-24 12:05:43   |
| www.google.com. | 74.125.228.18             | 3232235525 | 134744072 | 2013-04-24 12:05:43   |
| www.google.com. | 74.125.228.19             | 3232235525 | 134744072 | 2013-04-24 12:05:43   |
| www.google.com. | 74.125.228.17             | 3232235525 | 134744072 | 2013-04-24 12:05:43   |
| www.google.com. | 2607:f8b0:4004:0801::1014 | 3232235525 | 134744072 | 2013-04-23 20:43:26   |
| www.google.com. | 74.125.228.50             | 3232235525 | 134744072 | 2013-04-23 20:38:43   |
| www.google.com. | 74.125.228.51             | 3232235525 | 134744072 | 2013-04-23 20:38:43   |
| www.google.com. | 74.125.228.52             | 3232235525 | 134744072 | 2013-04-23 20:38:43   |
| www.google.com. | 74.125.228.48             | 3232235525 | 134744072 | 2013-04-23 20:38:43   |
| www.google.com. | 74.125.228.49             | 3232235525 | 134744072 | 2013-04-23 20:38:43   |
| www.google.com. | 2607:f8b0:4004:0801::1011 | 3232235525 | 134744072 | 2013-04-23 18:38:52   |
| www.google.com. | 2607:f8b0:400e:0c01::0067 | 3232235525 | 134744072 | 2013-04-23 15:57:45   |
| www.google.com. | 2607:f8b0:4004:0801::1010 | 3232235525 | 134744072 | 2013-04-23 15:07:59   |
| www.google.com. | 2607:f8b0:400e:0c01::0069 | 3232235525 | 134744072 | 2013-04-23 12:30:28   |
+-----------------+---------------------------+------------+-----------+-----------------------+

Here is a similar query but we want to see any tertiary youtube.com domains and sort by the lookup returned.

mysql> SELECT qr,type,auth,nx,ttl,rrname,rrval from dns WHERE rrname LIKE "%.youtube.com." GROUP BY rrval LIMIT 50;
+------+------+------+------+------+--------------------------------+----------------+
| qr   | type | auth | nx   | ttl  | rrname                         | rrval          |
+------+------+------+------+------+--------------------------------+----------------+
|    0 |    1 |    0 |    0 |    0 | www.youtube.com.               |                |
|    1 |    1 |    0 |    0 |  300 | v17.lscache2.c.youtube.com.    | 12.216.80.12   |
|    1 |    1 |    0 |    0 | 1800 | r2.sn-5uu-vgqe.c.youtube.com.  | 12.216.80.13   |
|    1 |    1 |    0 |    0 | 1800 | r3.sn-5uu-vgqe.c.youtube.com.  | 12.216.80.14   |
|    1 |    1 |    0 |    0 | 1800 | r4.att-ord1.c.youtube.com.     | 12.216.80.15   |
|    1 |    1 |    0 |    0 | 1800 | r6.sn-5uu-vgqe.c.youtube.com.  | 12.216.80.17   |
|    1 |    1 |    0 |    0 | 1714 | r8.sn-5uu-vgqe.c.youtube.com.  | 12.216.80.19   |
|    1 |    1 |    0 |    0 | 1741 | r1.sn-5uu-vgql.c.youtube.com.  | 12.216.80.44   |
|    1 |    1 |    0 |    0 | 1800 | r2.sn-5uu-vgql.c.youtube.com.  | 12.216.80.45   |
|    1 |    1 |    0 |    0 | 1800 | r3.sn-5uu-vgql.c.youtube.com.  | 12.216.80.46   |
|    1 |    1 |    0 |    0 | 1279 | r4.sn-5uu-vgql.c.youtube.com.  | 12.216.80.47   |
|    1 |    1 |    0 |    0 | 1800 | r6.sn-5uu-vgql.c.youtube.com.  | 12.216.80.49   |
|    1 |    1 |    0 |    0 | 1800 | r7.sn-5uu-vgql.c.youtube.com.  | 12.216.80.50   |
|    1 |    1 |    0 |    0 | 1739 | r8.sn-5uu-vgql.c.youtube.com.  | 12.216.80.51   |
|    1 |    1 |    0 |    0 | 1800 | r12.sn-hp576nes.c.youtube.com. | 173.194.17.17  |
|    1 |    1 |    0 |    0 | 1800 | r20.sn-hp576nes.c.youtube.com. | 173.194.17.25  |
|    1 |    1 |    0 |    0 | 1800 | r6.sn-q4f7dnel.c.youtube.com.  | 173.194.24.11  |
|    1 |    1 |    0 |    0 | 1800 | r1.dfw06s08.c.youtube.com.     | 173.194.24.134 |
|    1 |    1 |    0 |    0 | 1800 | r15.sn-q4f7dn7r.c.youtube.com. | 173.194.24.148 |
|    1 |    1 |    0 |    0 | 1800 | r18.sn-hp576n7d.c.youtube.com. | 173.194.29.119 |
|    1 |    1 |    0 |    0 | 1800 | r9.sn-hp576n7z.c.youtube.com.  | 173.194.29.46  |
|    1 |    1 |    0 |    0 | 1800 | r5.sn-ab5e6ner.c.youtube.com.  | 173.194.31.10  |
|    1 |    1 |    0 |    0 | 1800 | r1.sn-ab5e6nle.c.youtube.com.  | 173.194.31.102 |
|    1 |    1 |    0 |    0 |  640 | r2.sn-ab5e6nle.c.youtube.com.  | 173.194.31.103 |
|    1 |    1 |    0 |    0 | 1800 | r3.sn-ab5e6nle.c.youtube.com.  | 173.194.31.104 |
|    1 |    1 |    0 |    0 | 1800 | r4.sn-ab5e6nle.c.youtube.com.  | 173.194.31.105 |
|    1 |    1 |    0 |    0 | 1800 | r5.sn-ab5e6nle.c.youtube.com.  | 173.194.31.106 |
|    1 |    1 |    0 |    0 | 1800 | r6.sn-ab5e6nle.c.youtube.com.  | 173.194.31.107 |
|    1 |    1 |    0 |    0 | 1800 | r7.sn-ab5e6nle.c.youtube.com.  | 173.194.31.108 |
|    1 |    1 |    0 |    0 | 1800 | r8.sn-ab5e6nle.c.youtube.com.  | 173.194.31.109 |
|    1 |    1 |    0 |    0 |  705 | r6.sn-ab5e6ner.c.youtube.com.  | 173.194.31.11  |
|    1 |    1 |    0 |    0 | 1800 | r9.sn-ab5e6nle.c.youtube.com.  | 173.194.31.110 |
|    1 |    1 |    0 |    0 | 1800 | r10.sn-ab5e6nle.c.youtube.com. | 173.194.31.111 |
|    1 |    1 |    0 |    0 |  292 | r11.sn-ab5e6nle.c.youtube.com. | 173.194.31.112 |
|    1 |    1 |    0 |    0 | 1800 | r12.sn-ab5e6nle.c.youtube.com. | 173.194.31.113 |
|    1 |    1 |    0 |    0 |  178 | r13.sn-ab5e6nle.c.youtube.com. | 173.194.31.114 |
|    1 |    1 |    0 |    0 | 1800 | r14.sn-ab5e6nle.c.youtube.com. | 173.194.31.115 |
|    1 |    1 |    0 |    0 | 1800 | r15.sn-ab5e6nle.c.youtube.com. | 173.194.31.116 |
|    1 |    1 |    0 |    0 | 1800 | r16.sn-ab5e6nle.c.youtube.com. | 173.194.31.117 |
|    1 |    1 |    0 |    0 | 1800 | r17.sn-ab5e6nle.c.youtube.com. | 173.194.31.118 |
|    1 |    1 |    0 |    0 | 1800 | r18.sn-ab5e6nle.c.youtube.com. | 173.194.31.119 |
|    1 |    1 |    0 |    0 | 1653 | r7.sn-ab5e6ner.c.youtube.com.  | 173.194.31.12  |
|    1 |    1 |    0 |    0 | 1800 | r19.sn-ab5e6nle.c.youtube.com. | 173.194.31.120 |
|    1 |    1 |    0 |    0 | 1800 | r20.sn-ab5e6nle.c.youtube.com. | 173.194.31.121 |
|    1 |    1 |    0 |    0 | 1800 | r8.sn-ab5e6ner.c.youtube.com.  | 173.194.31.13  |
|    1 |    1 |    0 |    0 |   81 | r1.sn-ab5e6nll.c.youtube.com.  | 173.194.31.134 |
|    1 |    1 |    0 |    0 | 1800 | r2.sn-ab5e6nll.c.youtube.com.  | 173.194.31.135 |
|    1 |    1 |    0 |    0 | 1800 | r3.sn-ab5e6nll.c.youtube.com.  | 173.194.31.136 |
|    1 |    1 |    0 |    0 | 1800 | r4.sn-ab5e6nll.c.youtube.com.  | 173.194.31.137 |
|    1 |    1 |    0 |    0 | 1800 | r5.lga15s22.c.youtube.com.     | 173.194.31.138 |
+------+------+------+------+------+--------------------------------+----------------+
50 rows in set (22.17 sec)

An alternative method is to write YaF records directly to mediator, and further the MySQL DBMS rather then writing files to disk although you can still do this with the appropriate toggles. Here is example usage to start the processes:

$ ./silk-installs/yaf_silk_mysql_mediator-1.4.0/yaf_silk_mysql_mediator --in-host=127.0.0.1 --in-port=18000 --mysql-host=localhost --name=username --pass password --database eflows
$ sudo /usr/local/bin/yaf --live pcap --in eth1 --out 127.0.0.1 --ipfix-port=18000 --ipfix tcp --log=/var/log/yaf.log --filter="port 53" --applabel --applabel-rules=/usr/local/etc/yafApplabelRules.conf --max-payload=1000 --plugin-name=/usr/local/lib/yaf/dpacketplugin.la --plugin-opts="53" &

Ensure YaF and mediator are connected:

$ sudo netstat -tupan|grep yaf
tcp        0      0 127.0.0.1:18000             0.0.0.0:*                   LISTEN      6497/yaf_silk_mysql
tcp        0      0 127.0.0.1:47417             127.0.0.1:18000             ESTABLISHED 6513/yaf
tcp        0      0 127.0.0.1:18000             127.0.0.1:47417             ESTABLISHED 6497/yaf_silk_mysql

You may use the following MySQL query to see when the table was last updated to ensure records are being inserted on a regular basis:

mysql> SHOW TABLE STATUS in eflows;

After a few minutes of collection, query a domain that has been recently resolved and you should see it in the DBMS.

mysql> SELECT rrname,rrval from dns WHERE rrname LIKE "%rsreese.com." GROUP BY rrval LIMIT 10;
+--------------+--------------------------------+
| rrname       | rrval                          |
+--------------+--------------------------------+
| rsreese.com. |                                |
| rsreese.com. | 2600:3c02::f03c:91ff:fe96:f7bd |
| rsreese.com. | 74.207.234.79                  |
| rsreese.com. | ns1.linode.com.                |
| rsreese.com. | ns2.linode.com.                |
| rsreese.com. | ns3.linode.com.                |
| rsreese.com. | ns4.linode.com.                |
| rsreese.com. | ns5.linode.com.                |
+--------------+--------------------------------+
8 rows in set (18.26 sec)

There are a number of different fields available for query so I leave it to you to come up with whatever is most useful for you. Further, think of how you could write a shiny front-end for analysts to use rather then having to use the MySQL command line interface. Hope you found this useful and leave a comment if you did or have any questions.

]]>Passive DNS is a useful tool for any analysts teams toolbox, I have noted several public sensors here but they only see data (queries and responses) that transverse their sensors. I have been working on setting up passive DNS using &#8230; <a href="http://www.rsreese.com/passive-dns-collection-and-analysis-using-yaf-and-mediator/">Continue reading <span class="meta-nav">&#8594;</span></a>http://www.rsreese.com/passive-dns-collection-and-analysis-using-yaf-and-mediator/feed/0http://www.rsreese.com/passive-dns-collection-and-analysis-using-yaf-and-mediator/Online Information Security Analysis Tools and Resourceshttp://feedproxy.google.com/~r/rsreese/~3/BGenUSUKiWM/securityanalysisdefenseindicatorStephen ReeseSun, 31 Mar 2013 07:17:30 PDThttp://www.rsreese.com/?p=1111Security_Panda

Panda courtesy of www.xen.org.

Here are a list of sites that analysts may find useful in their day-to-day analysis of indicators and threats. While verifying and searching for new sources, I came across Links and resources for malware samples and Free Online Tools for Looking Up Potentially Malicious Websites which may also be helpful. This page may be considered a work-in-progress but if you feel something is missing or broken, leave a comment or contact me. Entries with an asterisk (*) require an account.

IP/ISP/Domain, and WHOIS look-ups

IP and Domain analysis for malware or web-based threats

Open-source Threat Reports, IP and Domain Blacklists

Malware Binary Analysis

Malware Samples

HTTP Agent sniffers, Decode De-Obfuscate JavaScript and Base 64

BotNet Tracking

Site History

Google Hacking

]]>Here are a list of sites that analysts may find useful in their day-to-day analysis of indicators and threats. While verifying and searching for new sources, I came across Links and resources for malware samples and Free Online Tools for Looking &#8230; <a href="http://www.rsreese.com/online-information-security-analysis-tools-and-resources/">Continue reading <span class="meta-nav">&#8594;</span></a>http://www.rsreese.com/online-information-security-analysis-tools-and-resources/feed/4http://www.rsreese.com/online-information-security-analysis-tools-and-resources/Running Molochhttp://feedproxy.google.com/~r/rsreese/~3/--6lkpYOHg0/networksecurityjavamolochnetwork defenseStephen ReeseSat, 16 Mar 2013 11:19:30 PDThttp://www.rsreese.com/?p=1150This is an overview of installing and running Moloch on a single host. After seeing the 2013 ShmooCon presentation, I have been looking forward to giving the tool a test-drive. Per the documentation, “Moloch is a open source large scale IPv4 full PCAP capturing, indexing and database system”. It is fast and has a pretty nice interface to boot. Although it does not contain the same feature-set as some commercial over the shelf (COTS) products, I see Moloch fitting into a similar space where COTS products such might sit. When analysts are made aware of anomaly-based alerts from signature/misuse based intrusion detection systems (IDS), e.g. Snort, or anomalous activity from network flow, e.g. SiLK, the analyst can obtain packet capture (PCAP) for further investigation. The existing commercial tool suites are expensive PCAP indexing tools if that is all they are being used for, especially if you are locked into their storage mechanism. A budget conscious security operation center (SOC) can setup Moloch for a fraction of the maintenance cost of commercial offerings and instead use the funds for additional hardware (longer retention), maintenance, and even some Moloch development contribution.

Although the developers have provided a script to get Moloch going, I had a few hiccups so I figured I would document them in the event they help someone else out. I used a CentOS release 6.4 (Final) x86_64 base bare-metal install. I imagine you could run it in a virtual environment for testing purposes. After you get the operating system (OS) installed and patched, pull down the latest Oracle Java for your distribution. Untar the package and create a symbolic in a directory that Moloch will be able to find.

$ sudo cp -R jre1.7.0_17/ /usr/bin/
$ sudo  ln -s /usr/bin/jre1.7.0_17/bin/java /usr/bin/java

Next, pull down the latest moloch build. I just grabbed the ZIP but it’s hosted on GitHub. You might want to take a look at the install script to see if everything is ideal for you. Run the easy installer which should pull down the prerequisites needed, build and install.

$ cd moloch-master/
$ sudo ./easybutton-singlehost.sh

If everything went smoothly, the script will try starting the three Moloch components being elasticsearch, capture, and viewer. The latter process did not start and this was probably for the better as I required me to take a closer look at what the install script was doing and the default configuration files (config.ini and elaseticsearch.yml). The configuration files are located in:

# ls -l /data/moloch/etc/
total 4680
-rw-r--r--. 1 root root    6766 Mar 14 17:21 config.ini
-rw-r--r--. 1 root root    6551 Mar 13 22:30 config.ini.template
-rw-r--r--. 1 root root   12545 Mar 14 22:54 elasticsearch.yml
-rw-r--r--. 1 root root 3360134 Mar  6 15:10 GeoIPASNum.dat
-rw-r--r--. 1 root root 1358092 Mar  5 21:48 GeoIP.dat
-rw-r--r--. 1 root root    1249 Mar 13 22:31 moloch.crt
-rw-r--r--. 1 root root    1029 Mar 13 22:31 moloch.csr
-rw-r--r--. 1 root root    1704 Mar 13 22:31 moloch.key
-rw-r--r--. 1 root root   10875 Mar 13 22:31 openssl.cnf
-rw-r--r--. 1 root root   10909 Mar 13 22:30 openssl.cnf.template

First, I had to sort out what was preventing the viewer from starting so I took a look at the viewer.log. 

Mar 13 23:13:04 http.c:245 moloch_http_connect(): Connecting 0x7f6e0d19b010
Mar 13 23:13:04 http.c:276 moloch_http_connect(): 0x7f6e0d19b010: Error: Error connecting: Address family not supported by protocol
Couldn't connect to elastic search at 'localhost:9200'

Log files are located in:

# ls -l /data/moloch/logs/
total 6047776
-rw-r--r--. 1 root root 6180585472 Mar 15 23:44 capture.log
-rw-r--r--. 1 root root   12062720 Mar 14 17:22 capture.log.old
-rw-r--r--. 1 root root          0 Mar 13 22:31 Moloch_index_indexing_slowlog.log
-rw-r--r--. 1 root root          0 Mar 13 22:31 Moloch_index_search_slowlog.log
-rw-r--r--. 1 root root        163 Mar 15 20:00 Moloch.log
-rw-r--r--. 1 root root       2943 Mar 13 23:27 Moloch.log.2013-03-13
-rw-r--r--. 1 root root      35410 Mar 14 23:34 Moloch.log.2013-03-14
-rw-r--r--. 1 root root     208487 Mar 15 23:06 viewer.log
-rw-r--r--. 1 root root       1668 Mar 15 09:06 viewer.log.old

I had to change the directive in the config.ini from localhost to 127.0.0.1, otherwise the viewer would not connect to the elasticsearch instance in CentOS. Probably due to the initial IPv6 look-up, just a guess. Also added a Berkley packet filter (BPF) to prevent the capture and indexing of internal-to-internal traffic.

elasticsearch=127.0.0.1:9200
bpf=not src net (10.0.0.0/8) and dst net (10.0.0.0/8)

While I was adjusting the configuration, I decided to adjust the elasticsearch memory usage from what I originally specified in the installer script. You might want to take a look at their hardware requirements but I was able to run with a less powerful node:

$ sudo vim /data/moloch/bin/run_es.sh

ES_HEAP_SIZE=2G bin/elasticsearch -Des.config=${TDIR}/etc/elasticsearch.yml

The viewer would now start (the capture and viewer process were already running but had gracefully killed them). Here are the commands to start each process based on the default installation criteria.

$ sudo nohup /data/moloch/bin/run_es.sh
$ sudo nohup /data/moloch/bin/run_capture.sh &
$ sudo nohup /data/moloch/bin/run_viewer.sh &

Sessions page screen-shot after capturing some traffic, not including session listing:

moloch-graph-thumb

Stats page screen-shot:

moloch-stats-thumb

I noticed the mention of two plugins to keep tabs on the elasticsearch memory usage and to maintain session data. This is pretty important as I determined if you remove PCAP and the session data remained, think metadata, users that attempted to drill-down on the aforementioned session data for the missing PCAP would cause the viewer process to die. In my case, I setup Putty to tunnel my connection to the locally listening plug-in interfaces and delete the offending session data:

moloch-putty

ElasticSearch maintenance screenshot located at http://127.0.0.1:9200/_plugin/head/ after tunneling via Putty. I was able to drop the session via this interface.

moloch-head-thumb

Node statistics screen-shot accessed at http://127.0.0.1:9200/_plugin/bigdesk/ after correctly configuring Putty. Note that we want to keep an eye on the heap memory to ensure it does not approach the maximum specified value. There are many more statistics not shown in this screen-shot.

moloch-bigdesk-thumb

Here’s a Youtube video featuring Moloch in actions. As usual, if you have trouble installing or running Moloch, please leave a comment below, and do not forget to check out the Moloch FAQ.

]]>This is an overview of installing and running Moloch on a single host. After seeing the 2013 ShmooCon presentation, I have been looking forward to giving the tool a test-drive. Per the documentation, &#8220;Moloch is a open source large scale &#8230; <a href="http://www.rsreese.com/running-moloch/">Continue reading <span class="meta-nav">&#8594;</span></a>http://www.rsreese.com/running-moloch/feed/0http://www.rsreese.com/running-moloch/Increment IP packet timestamphttp://feedproxy.google.com/~r/rsreese/~3/mpwfXxcB0H0/networkpythonscapyStephen ReeseTue, 12 Mar 2013 19:48:35 PDThttp://www.rsreese.com/?p=1143I recently had a need to specify and increment the IP timestamp values of packets in a PCAP. In this example, the starting second value is specified and we increment the microsecond value. This requires the use of Scapy. If you have any questions or recommendations for improvement, please leave a comment below.

#! /usr/bin/python

# Script to parse a PCAP and modify timestamps
# Requires Scapy
# 0.1 - 03012012
# Stephen Reese

from scapy.all import *
import sys

# Get input and output files from command line
if len(sys.argv) < 2:
        print "Usage: rewritetimestamp.py inputpcapfile"
        sys.exit(1)

# Assign variable names for input and output files
infile = sys.argv[1]

def process_packets():
    pkts = rdpcap(infile)
    cooked=[]
    timestamp = 1234567890.000000
    for p in pkts:
        p.time = timestamp
        timestamp += 0.000001
        pmod=p
        p.time
        cooked.append(pmod)

    wrpcap("out.pcap", cooked)

process_packets()
]]>I recently had a need to specify and increment the IP timestamp values of packets in a PCAP. In this example, the starting second value is specified and we increment the microsecond value. This requires the use of Scapy. If &#8230; <a href="http://www.rsreese.com/increment-ip-packet-timestamp/">Continue reading <span class="meta-nav">&#8594;</span></a>http://www.rsreese.com/increment-ip-packet-timestamp/feed/0http://www.rsreese.com/increment-ip-packet-timestamp/Running SnortADhttp://feedproxy.google.com/~r/rsreese/~3/Jj1ZwzkvsN0/networksecurityanomaly detectionStephen ReeseWed, 09 Jan 2013 19:00:46 PSThttp://www.rsreese.com/?p=1063I recently fired up a Snort Anomaly Detection instance provided by the SnortAD project and wanted to share my experience for those who might be interested in trying it on your network. SnortAD is the third generation anomaly detection preprocessor for Snort and is a little different than its predecessors but don’t take my word for it, check out their site.

First you need to create a log file based on your network, the log file will contain a profile of your network traffics characteristics. Although a log file has been provided with the SnortAD virtual machine (VM) that contains null entries it will not do you much good aside from alerting on everything. In order to characterize your network, you will need to create a log file with enough data to be statistically relevant. For the impatient, you can create a day or two worth of data and duplicate the data. Duplicating the data will have adverse effects though. Think about a university in which a majority of classes occur on Monday and Wednesday. If you only create a profile for Monday and duplicate it for the rest of the week, you can quickly understand how your results might be skewed.

To get going, use the snort.conf included on SnortAD VM and begin creating a log file but remember to backup or remove the original log file in the event you need it for reference. Also, always backup your configuration files before making changes for good measure.

Configure the snort.conf file to log. Something like the following should work fine:

preprocessor AnomalyDetection: LogPath /var/log/snort log time 60

Next, run Snort to generate log data. As mentioned, you should create enough data to make it statistically relevant. The evaluator script expects three weeks. As an alternate, you might be able to use tcpreplay to replay existing PCAP if you have enough data.

$ sudo /usr/local/bin/snort -c /etc/snort.conf -i eth0

You should start seeing messages to stdout that look like the following:

Loged transfer between 06-01-13 15:33:52 - 06-01-13 15:34:52
Loged transfer between 06-01-13 15:34:52 - 06-01-13 15:35:52

Now you should have a log with a number of entries saved in /var/log/snort. The profile generation script is next run. In this example we specify a week rather than opt for the three week default but again, YMMV and you made need to adjust these values. Also, make sure you check the help of the profile generator as there are other algorithms, five to be specific: Moving average (default), Naive method, Autoregressive time series model, Holt-Winters model, and HW model with Brutlag's confidence band.

/usr/local/src/profilegenerator/ad_profilegenerator.r -m AVG --avg 'WEEKLY,1' -l Log_Data.txt -p profile.txt -e evaluator.txt -P pattern.txt

The previous command creates the profile.txt file which is a CSV file, i.e. you could respectively name it profile.csv. The CSV file will be used by your updated snort.conf file. In order to enable anomaly detection, we need to download or create a few Snort configuration files:

$ ls -l /etc/snort
total 4200
-rw-r--r--. 1 root root    3621 Jan  5 15:35 classification.config
-rw-r--r--. 1 root root   29596 Jan  5 15:35 gen-msg.map
-rw-r--r--. 1 root root    7897 Jan  5 15:35 preprocessor.rules
-rw-r--r--. 1 root root 1484013 Jan  5 15:35 profile.csv
-rw-r--r--. 1 root root     746 Jan  5 15:35 reference.config
-rw-r--r--. 1 root root 2696705 Jan  5 15:35 sid-msg.map
-rw-r--r--. 1 root root     255 Jan  5 15:35 snort.conf
-rw-r--r--. 1 root root    2556 Jan  5 15:35 threshold.conf
-rw-r--r--. 1 root root   53841 Jan  5 15:35 unicode.map

I found it simplest to pull down the latest Snort signature as they have the additional required files that are not included in the provide SnortAD build. You can pull down the needed preprocessor.rules from one of the authors bitbucket. The snort.conf was populated with the following contents:

include classification.config
include reference.config
include preprocessor.rules
preprocessor AnomalyDetection: ProfilePath /etc/snort/profile.csv LogPath /var/log/snort alert log time 60

If you have everything in the /etc/snort directory, you should be able to run Snort and see alerts when anomalies are detected:

$ sudo /usr/local/bin/snort -c /etc/snort/snort.conf -i eth0

Here are some sample alerts from some early testing. It will probably take some tuning to begin seeing useful alerts:

[**] [1000100:1000101:1] AD_UNUSUALLY_HIGH_TCP_TRAFFIC [**]
[Classification: Potentially Bad Traffic] [Priority: 2]
01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8
ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF
Type:8  Code:0  ID:30537   Seq:1  ECHO

[**] [1000100:1000107:1] AD_HIGH_LAN_TCP_TRAFFIC [**]
[Classification: Potentially Bad Traffic] [Priority: 2]
01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8
ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF
Type:8  Code:0  ID:30537   Seq:1  ECHO

[**] [1000100:1000108:1] AD_UNUSUALLY_LOW_UDP_TRAFFIC [**]
[Classification: Potentially Bad Traffic] [Priority: 2]
01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8
ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF
Type:8  Code:0  ID:30537   Seq:1  ECHO

[**] [1000100:1000114:1] AD_LOW_LAN_UDP_TRAFFIC [**]
[Classification: Potentially Bad Traffic] [Priority: 2]
01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8
ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF
Type:8  Code:0  ID:30537   Seq:1  ECHO

[**] [1000100:1000134:1] AD_LOW_ARP_REQUEST_NUMBER [**]
[Classification: Potentially Bad Traffic] [Priority: 2]
01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8
ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF
Type:8  Code:0  ID:30537   Seq:1  ECHO

[**] [1000100:1000138:1] AD_LOW_NOT_TCP_IP_TRAFFIC [**]
[Classification: Potentially Bad Traffic] [Priority: 2]
01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8
ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF
Type:8  Code:0  ID:30537   Seq:1  ECHO

[**] [1000100:1000140:1] AD_LOW_OVERALL_PACKET_NUMBER [**]
[Classification: Potentially Bad Traffic] [Priority: 2]
01/06-20:59:04.308505 10.0.0.116 -> 8.8.8.8
ICMP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:84 DF
Type:8  Code:0  ID:30537   Seq:1  ECHO

If you have any questions, leave a comment and/or check out the authors Readme.txt for some additional usage insight.

]]>I recently fired up a Snort Anomaly Detection instance provided by the SnortAD project and wanted to share my experience for those who might be interested in trying it on your network. SnortAD is the third generation anomaly detection preprocessor &#8230; <a href="http://www.rsreese.com/running-snortad/">Continue reading <span class="meta-nav">&#8594;</span></a>http://www.rsreese.com/running-snortad/feed/4http://www.rsreese.com/running-snortad/Mailing Listshttp://feedproxy.google.com/~r/rsreese/~3/W6avuJrMjKQ/networksecuritymailing listsStephen ReeseFri, 09 Nov 2012 20:27:11 PSThttp://www.rsreese.com/?p=1100Here are a few technology and information security related mailing-lists that I subscribe to in no particular order. Leave a comment if you think I missed one.

asterisk-users.lists.digium.com
beginners.perl.org
snort-users.lists.sourceforge.net
nessus.list.nessus.org
pauldotcom.mail.pauldotcom.com
samurai-devel.lists.sourceforge.net
ptk-forensics-mail.lists.sourceforge.net
gcfa.lists.sans.org
framework-hackers.spool.metasploit.com
framework.spool.metasploit.com
secureideas-base-user.lists.sourceforge.net
python-list.python.org
nexpose-users.lists.rapid7.com
winquisitor-beta.googlegroups.com
securitybsides.googlegroups.com
datarecoverycertification.googlegroups.com
full-disclosure.lists.grok.org.uk
scap_interest.ietf.org
cipp.news.infracritical.com
scadasec.news.infracritical.com
debian-security-announce.lists.debian.org
bugtraq.list-id.securityfocus.com
ietf.ietf.org
dfir.lists.sans.org
webappsec.list-id.securityfocus.com
sleuthkit-users.lists.sourceforge.net
vol-users.volatilesystems.com
emerging-sigs.emergingthreats.net

]]>Here are a few technology and information security related mailing-lists that I subscribe to in no particular order. Leave a comment if you think I missed one. asterisk-users.lists.digium.com beginners.perl.org snort-users.lists.sourceforge.net nessus.list.nessus.org pauldotcom.mail.pauldotcom.com samurai-devel.lists.sourceforge.net ptk-forensics-mail.lists.sourceforge.net gcfa.lists.sans.org framework-hackers.spool.metasploit.com framework.spool.metasploit.com secureideas-base-user.lists.sourceforge.net python-list.python.org nexpose-users.lists.rapid7.com &#8230; <a href="http://www.rsreese.com/mailing-lists/">Continue reading <span class="meta-nav">&#8594;</span></a>http://www.rsreese.com/mailing-lists/feed/0http://www.rsreese.com/mailing-lists/Podcastshttp://feedproxy.google.com/~r/rsreese/~3/mIaZhmTrReA/networksecuritypodcastStephen ReeseSun, 09 Sep 2012 21:29:24 PDThttp://www.rsreese.com/?p=1103Here’s a list of information technology and security podcasts. Some are technical, others are higher level so YMMV. Essentially a source of information to keep me up to date on what’s going on in the information technology realm. If you think of something I have missed, leave a commment. *Note some of these may be explicit so please use discretion and they are in no particular order.

Wired Features
ArcSight Podcasts
The CyberJungle
Tech Talk
The Digita
l Underground Podcast
Forensic 4cast » Forensic 4cast
Imperva Data Security Podcast
The Linux Action Show! MP3
AudioParasitics – The Official Podcast of Mc
Afee Labs
McAfee SaaS Security Buzz
Packet Pushers Podcast
The Silver Bullet Security Podcast
Social-Engineer.Org PodCast
The Southern Fried Security Podcast
Internet Storm Center Threat Update
RB2
Security Now!
NPR Topics: Technology Podcast
Network Security Podcast
PaulDotCom Security Weekly
Sophos Podcasts
Tenable Network Security Podcast
Risky Business
Exotic Liability
Eurotrash Security Podcast: Security with funny accents
SecuraBit
Security Justice
The Security Catalyst
CERT’s Podcast Series: Security for Business Leaders
My Hard Drive Died – w/Scott Moulton
CyberSpeak’s Podcast
OWASP Security Podcast
Crypto-Gram Security Podcast
Cisco TAC Security Podcast Series
Off The Hook: high-bitrate MP3 feed
Security Wire Weekly
InfoSec Daily Podcast
An Information Security Place Podcast
Security Insider – Podcast Edition
Wall Street
Journal Tech News Briefing
The 404 (MP3)
Click
Security.Exe powered by The CISO Group with Alan Shimel
FLOSS Weekly
The Stack Exchange Podcast
Down the Security Rabbithole
AuditCasts with David Hoelzer
CERIAS Security Seminar Podcast

]]>Here&#8217;s a list of information technology and security podcasts. Some are technical, others are higher level so YMMV. Essentially a source of information to keep me up to date on what&#8217;s going on in the information technology realm. If you &#8230; <a href="http://www.rsreese.com/podcasts/">Continue reading <span class="meta-nav">&#8594;</span></a>http://www.rsreese.com/podcasts/feed/0http://www.rsreese.com/podcasts/Decoding XOR payload using first few bytes as keyhttp://feedproxy.google.com/~r/rsreese/~3/fVmTSvsbmmA/networksecurityobfuscationpythonStephen ReeseMon, 23 Jul 2012 21:07:20 PDThttp://www.rsreese.com/?p=973I recently came across the need to decode an exclusive or (XOR) payload. In my case, the key to de-obfuscating the traffic was the first three bytes of each packets payload. While it is trivial to decode each payload, it was not reasonable for a large number of packets.

For testing purposes, create a packet:

$ scapy
Welcome to Scapy (2.1.0)
>>> p = (IP(ttl=10)/TCP(sport=1024,dport=443,flags="S")/"   WHATSTHESECRET0000ABCD0000ABCD0000ABCD")
>>> wrpcap("p.pcap", p)
>>> quit()

Should see something similar to this:

04:29:31.255470 IP 127.0.0.1.1024 > 127.0.0.1.443: Flags [S], seq 0:41, win 8192, length 41
        0x0000:  4500 0051 0001 0000 0a06 b2a4 7f00 0001  E..Q............
        0x0010:  7f00 0001 0400 01bb 0000 0000 0000 0000  ................
        0x0020:  5002 2000 751d 0000 2020 2057 4841 5453  P...u......WHATS
        0x0030:  5448 4553 4543 5245 5430 3030 3041 4243  THESECRET0000ABC
        0x0040:  4430 3030 3041 4243 4430 3030 3041 4243  D0000ABCD0000ABC
        0x0050:  44                                       D

Next, the payload is XOR’d using the first three bytes of the payload for the entire payload. If you note the first tcpdump, the three bytes of the payload were left empty, here I am placing the key that will be used to XOR the rest of the payload within the first three bytes of the payload.

The payload has been obfuscated using the key ‘the’.

Next we can use the script below or here to decode all of the packets. The script is not intelligent enough to know which need to be de-obfuscated so it is best to probably filter these into a new PCAP. Secondly, the script requires Scapy to be installed.

#! /usr/bin/python

# Script to parse a PCAP and XOR data based on a byte offset
# Requires Scapy
# 0.1 - 07172012
# Default is two bytes, change at line 35
# Stephen Reese and Chris Gragsone
#
# todo: add two more args, offset length and static offset option

from scapy.all import *
import sys

# Get input and output files from command line
if len(sys.argv) < 2:
        print "Usage: decodexorpayload.py [input pcap file]"
        sys.exit(1)

# Assign variable names for input and output files
infile = sys.argv[1]

def many_byte_xor(buf, key):
    buf = bytearray(buf)
    key = bytearray(key)
    key_len = len(key)
    for i, bufbyte in enumerate(buf):
        buf[i] = bufbyte ^ key[i % key_len]
    return str(buf)

def process_packets():
    pkts = rdpcap(infile)
    cooked=[]
    for p in pkts:
        # You may have to adjust the payload depth here:
        # i.e. p.payload.payload.payload
        pkt_payload = str(p.payload.payload)
        pkt_offset = str(p.payload.payload)[:3]
        if pkt_payload and pkt_offset:
              pmod=p
              # You may have to adjust the payload depth here:
              p.payload.payload=many_byte_xor(pkt_payload, pkt_offset)
              cooked.append(pmod)

    wrpcap("dump.pcap", cooked)

process_packets()

After script completion, viewing the packet does indeed show the de-obfuscated packet:

reading from file dump.pcap, link-type RAW (Raw IP)
04:24:44.415262 IP 127.0.0.1.1024 > 127.0.0.1.443: Flags [S], seq 0:41, win 8192, length 41
        0x0000:  4500 0051 0001 0000 0a06 b2a4 7f00 0001  E..Q............
        0x0010:  7f00 0001 0400 01bb 0000 0000 0000 0000  ................
        0x0020:  5002 2000 751d 0000 0000 0057 4841 5453  P...u......WHATS
        0x0030:  5448 4553 4543 5245 5430 3030 3041 4243  THESECRET0000ABC
        0x0040:  4430 3030 3041 4243 4430 3030 3041 4243  D0000ABCD0000ABC
        0x0050:  44                                       D

There are a number of features that could be added and of course the code can probably be improved upon. Have some ideas? Leave a comment below.

]]>I recently came across the need to decode an exclusive or (XOR) payload. In my case, the key to de-obfuscating the traffic was the first three bytes of each packets payload. While it is trivial to decode each payload, it &#8230; <a href="http://www.rsreese.com/decoding-xor-payload-using-first-few-bytes-as-key/">Continue reading <span class="meta-nav">&#8594;</span></a>http://www.rsreese.com/decoding-xor-payload-using-first-few-bytes-as-key/feed/4http://www.rsreese.com/decoding-xor-payload-using-first-few-bytes-as-key/World IPv6 Dayhttp://feedproxy.google.com/~r/rsreese/~3/Sh1KnwYkaog/networkipv6Stephen ReeseWed, 06 Jun 2012 18:29:09 PDThttp://www.rsreese.com/?p=957World IPv6 Day (June 8th 2012) is rapidly approaching. It is an exciting and scary reality. For my personal assets, there was a small investment on my part to get everything up to par. My internet provider Comcast is dual-stack ready which is nice because I experienced some serious latency from time to time when using a tunnel-broker (note that other factors probably contributed). You can see more information about the Comcast IPv6 trial and preparation here. First, I had to invest in a new cable-modem as my old Motorola SB1000 was not up to the task. Comcast has created a hardware compatibility list. From the list I decided to go with the Motorola SB6121 as I have had pretty good success with their modems in the past. Secondly you need a device that is capable of filtering and distributing addresses to your internal devices. I am not going into details here, but a Cisco ASA5500 or a home-brew Linux device usually will work quite nicely. The most important part to read into is that you are also filtering v6 IP traffic along with the v4 so you do not have evil-doers sneaker-netting into your network. Your network devices will not hide behind network address translation (NAT). Lastly, keep the images, firmware, or distributions patched and monitor your traffic from time to time. Kind of like a cavity, you usually do not know you have one until it is too late.

My blog has also moved to a dual-stack (Linode awesome service and support) from a tunnel-broker! This was really straightforward to implement as Linode provides some great documentation in their library. As with any setup, you need to filter unwanted traffic from entering/exiting your node(s), Iptables makes quick work of this. In this scenario, I am going with a deny-by-default posture and log everything that is dropped. This is by no means definitive but just a place to get started.

*filter
# Drop everything
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]

# Allow the loopback
-A INPUT -i lo -j ACCEPT
-A INPUT -d ::1/128 ! -i lo -j REJECT --reject-with icmp6-port-unreachable

# All returning connections
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Let the web server respond
-A INPUT -p tcp --sport 1024:65535 --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -p tcp --sport 1024:65535 --dport 443 -m state --state NEW -j ACCEPT

# All SSH session but limit attempt, also see fail2ban
-A INPUT -p tcp --sport 1024:65535 --dport 22 --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -m limit --limit 1/min --limit-burst 3 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 --tcp-flags FIN,SYN,RST,ACK SYN -j DROP

# Allow ICMP but need to restrict based on type
-A INPUT -p ipv6-icmp -j ACCEPT

# Drop everything else and log it
-A INPUT -m limit --limit 3/min -j LOG --log-prefix "ipv6 input denied: " --log-level 7

# Respective outbound rules
-A OUTPUT -p ipv6-icmp -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m limit --limit 3/min -j LOG --log-prefix "ipv6 output denied: " --log-level 7
COMMIT
]]>World IPv6 Day (June 8th 2012) is rapidly approaching. It is an exciting and scary reality. For my personal assets, there was a small investment on my part to get everything up to par. My internet provider Comcast is dual-stack &#8230; <a href="http://www.rsreese.com/world-ipv6-day/">Continue reading <span class="meta-nav">&#8594;</span></a>http://www.rsreese.com/world-ipv6-day/feed/3http://www.rsreese.com/world-ipv6-day/How-to setup an Upside-Down-Ternethttp://feedproxy.google.com/~r/rsreese/~3/fraHjIDbMQE/internetdebianiptablessquid proxyStephen ReeseFri, 10 Feb 2012 19:07:15 PSThttp://www.rsreese.com/?p=901In an effort to replicate the amusing idea of a transparent proxy that manipulates traffic in a fun way found here and made even better with some great scripts that you can pull down from here. A Debian box was stood up with two network cards; one connects to the internal LAN and the other connected to an access-point which your guests connect to. I chose to post this how-to as the initial idea did not provide a complete reference on how to setup the needed components.

First, we are using an access-point we take care of the DHCP and DNS duties but the access-point or another host could perform these duties if they support said services. I choose to install the following DHCP service:

$ sudo apt-get install isc-dhcp-server

The following configuration provides the scope for the clients. We only define a scope for the client side which will use a 192.168.0.0 network for the example purposes.

$ grep ^[^#] /etc/dhcp/dhcpd.conf
ddns-update-style none;
default-lease-time 600;
max-lease-time 7200;
log-facility local7;
subnet 192.168.0.0 netmask 255.255.255.0 {
  range 192.168.0.100 192.168.0.200;
  option domain-name-servers 192.168.0.1;
  option domain-name "kittenwar.com";
  option routers 192.168.0.1;
  option broadcast-address 192.168.0.255;
  default-lease-time 600;
  max-lease-time 7200;
}

Secondly, the guests are going to need some resolution, rather than have their queries pass through the network, lets setup a simple resolver for them using BIND:

$ sudo apt-get install bind9

Setup some forwarders and the interface we want to listen on, for example sake, the same subnet servicing the clients:

$ grep ^[^#] /etc/bind/named.conf.options
options {
        directory "/var/cache/bind";
        version "tbd";
        forwarders { 8.8.8.8; 8.8.4.4; };
        auth-nxdomain no;    # conform to RFC1035
        listen-on-v6 { none; };
        listen-on { 192.168.0.1; 127.0.0.1; };
};

Some of the fun scripts require a HTTP service to serve up flipped images and all sorts of other goodness so Apache and ImageMagick are needed:

$ sudo apt-get install apache2
$sudo apt-get -y install imagemagick

The last service is Squid caching proxy. Install version 3 was installed from the repositories:

$ sudo apt-get install squid3

Edit the Squid configuration, this is a default configuration but the acl for the clients has been enabled along with interception mode (read transparent) and finally call the script via url_rewrite_program:

$ grep ^[^#] /etc/squid3/squid.conf
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access deny all
http_port 3128 intercept
hierarchy_stoplist cgi-bin ?
coredump_dir /var/spool/squid3
url_rewrite_program /home/us3r/squidScripts/flipImages.pl
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

Execute the following to create some protection from the subnet being advertised and furthermore forces all of the web request to use the Squid cache. The rule-set is by no means perfect or definitive, feel free to tailor to your needs and provide feedback.

$ grep ^[^#] fw-script
PATH=/sbin
iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -X
iptables --policy INPUT DROP
iptables --policy OUTPUT DROP
iptables --policy FORWARD DROP
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A INPUT -i eth2 -p tcp --dport 3128 -j ACCEPT
iptables -A INPUT -i eth2 -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -i eth2 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth2 -p udp --dport 53 -j ACCEPT
iptables -A INPUT -i eth2 -p udp --dport 67 -j ACCEPT
iptables -A OUTPUT -o eth2 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i eth1 -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -i eth1 -p tcp --dport 8000 -j ACCEPT
iptables -A INPUT -i eth1 -p udp --dport 68 -j ACCEPT
iptables -A INPUT -i eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -o eth1 -p tcp --dport 80 -j ACCEPT
iptables -A OUTPUT -o eth1 -p udp --dport 67 -j ACCEPT
iptables -A OUTPUT -o eth1 -p udp --dport 53 -j ACCEPT
iptables -A OUTPUT -o eth1 -p udp --dport 443 -j ACCEPT
iptables -A OUTPUT -o eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -I INPUT -j LOG --log-prefix "iptables denied: " --log-level 7
iptables -I OUTPUT -j LOG --log-prefix "iptables denied: " --log-level 7
iptables -t nat -A PREROUTING -i eth2 -p tcp --dport 80 -j DNAT --to-destination 192.168.0.1:3128
iptables -t nat -A POSTROUTING -o eth2 -s 192.168.0.0/24 -d 192.168.0.1 -j SNAT --to 192.168.0.1
iptables -t nat -A PREROUTING -i eth2 -p tcp --dport 80 -j REDIRECT --to-port 3128

You can down pull down a script from the Google code repository mentioned above which you have referenced in the Squid configuration. There are variables in the top of the scripts that you downloaded earlier. The variables need to be updated to reflect your system. A few Perl module prerequisites are also listed in the top of said scripts, access CPAN and install them:

$ sudo perl -MCPAN -e shell

After the required Perl modules are installed, you should be able to place a client on the guest network and they will retrieve sites, although it will not take long for to notice that in this case all of the images are inverted. Do not forget to checkout the other scripts.

ternet-pinterest-scaled

Lots of fun! If I missed something or you have some feedback, use the comment form below.

]]>In an effort to replicate the amusing idea of a transparent proxy that manipulates traffic in a fun way found here and made even better with some great scripts that you can pull down from here. A Debian box was &#8230; <a href="http://www.rsreese.com/how-to-setup-an-upside-down-ternet/">Continue reading <span class="meta-nav">&#8594;</span></a>http://www.rsreese.com/how-to-setup-an-upside-down-ternet/feed/0http://www.rsreese.com/how-to-setup-an-upside-down-ternet/